2024-09-16T04:52:52.087 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T04:52:52.108 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:52:52.198 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095 branch: squid description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906095' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 - volumes: size: 50 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 filestore queue throttle high multiple: 2 filestore queue throttle max multiple: 10 journal throttle high multiple: 2 journal throttle max multiple: 10 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 9 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/Gfz2O9oiXtJclWQQHD9rMbvmOwjYXTFp29zxR4E+WVqdmR6d2kZcfqwkT9e0yweDLPmsKDpi4h+Z9PG+qmIw= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrcAheY7xESDsiaKoxy1uiTgrVGFAgwJvkL7+kEbKqcoFfa1pIHeS+dL0r2SRxJnjB9BQWraT5Wi/ZaHN557XQ= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDLo+EPoEq9WFW5hBk7x0n8zfFLmqmNCjGgcnoqJK+2QK7gRpE7zIhIUU2T20GiHMphvD/3nswbrIjCMsvD0KFc= tasks: - install: branch: quincy exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 3 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.2 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T04:52:52.198 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T04:52:52.199 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T04:52:52.199 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T04:52:52.200 INFO:teuthology.task.internal:Saving configuration 2024-09-16T04:52:52.215 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T04:52:52.222 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T04:52:52.247 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:47:51.899745', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/Gfz2O9oiXtJclWQQHD9rMbvmOwjYXTFp29zxR4E+WVqdmR6d2kZcfqwkT9e0yweDLPmsKDpi4h+Z9PG+qmIw='} 2024-09-16T04:52:52.268 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:47:51.897258', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrcAheY7xESDsiaKoxy1uiTgrVGFAgwJvkL7+kEbKqcoFfa1pIHeS+dL0r2SRxJnjB9BQWraT5Wi/ZaHN557XQ='} 2024-09-16T04:52:52.289 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 04:47:51.898685', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDLo+EPoEq9WFW5hBk7x0n8zfFLmqmNCjGgcnoqJK+2QK7gRpE7zIhIUU2T20GiHMphvD/3nswbrIjCMsvD0KFc='} 2024-09-16T04:52:52.289 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T04:52:52.296 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-16T04:52:52.296 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-16T04:52:52.296 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-09-16T04:52:52.296 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T04:52:52.381 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcb8ff7fac0>, signals=[15]) 2024-09-16T04:52:52.381 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T04:52:52.387 INFO:teuthology.task.internal:Opening connections... 2024-09-16T04:52:52.388 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-09-16T04:52:52.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:52:52.465 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-09-16T04:52:52.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:52:52.541 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T04:52:52.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:52:52.627 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T04:52:52.634 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-09-16T04:52:52.653 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-09-16T04:52:52.653 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-09-16T04:52:52.711 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-09-16T04:52:52.711 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-09-16T04:52:52.711 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-09-16T04:52:52.711 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-09-16T04:52:52.711 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-09-16T04:52:52.711 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-09-16T04:52:52.712 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T04:52:52.712 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-09-16T04:52:52.712 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-09-16T04:52:52.712 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T04:52:52.712 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-09-16T04:52:52.712 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T04:52:52.712 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T04:52:52.712 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T04:52:52.713 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-09-16T04:52:52.737 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-09-16T04:52:52.755 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-09-16T04:52:52.756 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-09-16T04:52:52.814 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-09-16T04:52:52.815 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T04:52:52.815 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-09-16T04:52:52.815 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T04:52:52.815 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T04:52:52.815 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T04:52:52.815 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-09-16T04:52:52.837 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-09-16T04:52:52.856 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-09-16T04:52:52.856 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-09-16T04:52:52.914 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-09-16T04:52:52.914 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-09-16T04:52:52.914 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-09-16T04:52:52.914 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T04:52:52.915 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T04:52:52.915 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-09-16T04:52:52.937 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T04:52:52.947 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T04:52:52.953 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T04:52:52.953 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:52:52.955 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:52:52.958 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-09-16T04:52:52.975 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T04:52:52.980 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T04:52:52.981 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:52:53.013 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:52:53.016 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-09-16T04:52:53.123 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T04:52:53.206 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T04:52:53.224 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T04:52:53.224 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T04:52:53.224 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:52:53.225 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:52:53.225 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T04:52:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:52:53.242 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-16T04:52:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:52:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:52:53.244 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-09-16T04:52:53.244 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-16T04:52:53.301 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-503.el9.x86_64 2024-09-16T04:52:53.301 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-503.el9.x86_64 2024-09-16T04:52:53.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-09-16T04:52:53.303 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-503.el9.x86_64 2024-09-16T04:52:53.303 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-503.el9.x86_64 2024-09-16T04:52:53.303 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.14.0-503.el9.x86_64 2024-09-16T04:52:53.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-09-16T04:52:53.303 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-503.el9.x86_64 2024-09-16T04:52:53.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-09-16T04:52:55.745 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-16T04:52:55.758 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-16T04:52:56.840 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.2 MB 00:01 2024-09-16T04:52:57.629 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-16T04:52:57.875 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-16T04:52:58.924 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-16T04:53:02.134 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-16T04:53:03.383 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-16T04:53:03.634 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-16T04:53:04.480 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.5 MB 00:03 2024-09-16T04:53:04.688 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-16T04:53:04.884 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 19 kB 00:00 2024-09-16T04:53:05.947 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 19 kB 00:00 2024-09-16T04:53:06.122 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T04:53:06.935 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-16T04:53:10.094 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T04:53:11.803 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-16T04:53:12.005 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:01 ago on Mon 16 Sep 2024 04:53:11 AM UTC. 2024-09-16T04:53:12.084 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T04:53:12.084 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T04:53:12.172 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T04:53:12.178 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T04:53:12.178 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-16T04:53:12.178 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T04:53:12.178 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T04:53:12.178 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T04:53:12.178 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-16T04:53:12.178 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T04:53:12.178 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T04:53:12.179 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T04:53:12.179 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T04:53:12.179 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T04:53:12.179 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T04:53:12.179 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-09-16T04:53:12.179 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T04:53:12.179 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 79 M 2024-09-16T04:53:12.179 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 126 M 2024-09-16T04:53:12.180 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T04:53:12.378 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 249 kB/s | 23 kB 00:00 2024-09-16T04:53:12.440 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-16T04:53:12.762 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-16T04:53:13.146 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 37 MB/s | 29 MB 00:00 2024-09-16T04:53:13.446 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-09-16T04:53:13.446 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:53:13.446 INFO:teuthology.orchestra.run.smithi196.stdout:Total 62 MB/s | 79 MB 00:01 2024-09-16T04:53:13.918 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T04:53:13.918 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T04:53:14.010 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T04:53:14.017 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T04:53:14.017 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-09-16T04:53:14.017 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-09-16T04:53:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T04:53:14.019 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 79 M 2024-09-16T04:53:14.019 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 126 M 2024-09-16T04:53:14.019 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T04:53:14.126 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T04:53:14.196 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 183 kB/s | 23 kB 00:00 2024-09-16T04:53:14.439 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T04:53:14.439 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T04:53:14.493 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T04:53:14.493 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T04:53:14.523 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T04:53:14.529 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T04:53:14.529 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-16T04:53:14.529 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T04:53:14.529 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T04:53:14.529 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-09-16T04:53:14.530 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T04:53:14.531 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 79 M 2024-09-16T04:53:14.531 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 126 M 2024-09-16T04:53:14.531 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T04:53:14.717 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 342 kB/s | 23 kB 00:00 2024-09-16T04:53:14.988 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-09-16T04:53:15.367 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 48 MB/s | 35 MB 00:00 2024-09-16T04:53:15.488 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T04:53:15.488 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T04:53:15.621 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 22 MB/s | 35 MB 00:01 2024-09-16T04:53:15.868 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-16T04:53:16.226 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 19 MB/s | 29 MB 00:01 2024-09-16T04:53:16.227 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:53:16.227 INFO:teuthology.orchestra.run.smithi204.stdout:Total 46 MB/s | 79 MB 00:01 2024-09-16T04:53:16.472 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-16T04:53:16.472 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:53:16.472 INFO:teuthology.orchestra.run.smithi032.stdout:Total 32 MB/s | 79 MB 00:02 2024-09-16T04:53:16.907 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T04:53:17.180 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T04:53:17.253 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T04:53:17.253 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T04:53:17.560 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T04:53:17.560 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T04:53:17.966 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T04:53:18.234 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T04:53:18.235 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T04:53:18.512 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T04:53:18.512 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T04:53:18.657 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:53:18.771 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:53:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T04:53:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T04:53:21.385 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:53:21.477 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:53:21.558 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:53:21.691 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:53:21.740 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:53:21.823 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:53:24.270 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:53:24.408 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:53:24.604 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:53:24.736 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:53:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:53:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:53:29.223 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:53:29.350 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:53:29.866 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:53:29.912 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:53:31.376 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:53:34.189 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:53:35.058 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:09.641 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:09.685 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:09.948 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:09.948 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:54:09.949 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:54:09.949 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:54:10.792 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:10.792 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T04:54:10.792 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T04:54:10.792 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:10.792 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T04:54:10.792 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T04:54:10.792 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T04:54:10.793 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T04:54:10.793 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T04:54:11.150 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:54:11.243 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:11.243 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:54:11.243 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T04:54:11.243 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:54:11.243 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:54:11.244 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T04:54:11.244 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:54:11.244 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:54:11.303 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:11.303 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:54:11.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-09-16T04:54:11.344 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-09-16T04:54:11.344 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-09-16T04:54:11.344 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-09-16T04:54:11.345 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 82.el9 2024-09-16T04:54:11.345 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-09-16T04:54:11.345 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T04:54:11.345 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-09-16T04:54:11.345 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8274670 2024-09-16T04:54:11.345 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-09-16T04:54:11.345 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T04:54:11.346 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-09-16T04:54:11.347 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T04:54:11.347 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T04:54:11.347 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T04:54:11.348 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-505.el9.x86_64 2024-09-16T04:54:11.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T04:54:12.025 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-09-16T04:54:12.550 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:12.598 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:54:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:54:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:54:13.312 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T04:54:13.328 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-09-16T04:54:13.330 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-09-16T04:54:13.346 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.JPEmxEdlcx 2024-09-16T04:54:13.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JPEmxEdlcx 2024-09-16T04:54:13.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.JPEmxEdlcx 2024-09-16T04:54:13.552 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.JPEmxEdlcx is 6KB 2024-09-16T04:54:13.601 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.JPEmxEdlcx 2024-09-16T04:54:13.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T04:54:13.686 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T04:54:13.686 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T04:54:13.686 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T04:54:13.686 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T04:54:13.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T04:54:13.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-09-16T04:54:13.866 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:13.867 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T04:54:13.867 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T04:54:13.867 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:13.867 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T04:54:13.867 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T04:54:13.867 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T04:54:13.867 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T04:54:13.867 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T04:54:14.194 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:54:14.236 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:14.278 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:14.315 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:14.315 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:54:14.315 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T04:54:14.315 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:54:14.315 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:54:14.316 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-16T04:54:14.316 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:54:14.316 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:54:14.374 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:14.374 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:54:14.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 82.el9 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8274670 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-09-16T04:54:14.410 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-09-16T04:54:14.411 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T04:54:14.412 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T04:54:14.412 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T04:54:14.413 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-505.el9.x86_64 2024-09-16T04:54:14.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T04:54:14.582 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T04:54:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T04:54:14.583 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T04:54:15.026 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T04:54:15.364 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T04:54:15.705 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:54:15.791 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:15.791 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:54:15.792 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T04:54:15.792 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T04:54:15.792 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T04:54:15.792 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T04:54:15.792 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T04:54:15.792 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T04:54:15.851 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T04:54:15.851 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T04:54:15.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-09-16T04:54:15.888 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-09-16T04:54:15.888 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-09-16T04:54:15.888 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-09-16T04:54:15.888 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 82.el9 2024-09-16T04:54:15.888 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-09-16T04:54:15.888 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T04:54:15.888 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8274670 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T04:54:15.889 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T04:54:15.890 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T04:54:15.890 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-09-16T04:54:15.890 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T04:54:15.890 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T04:54:15.891 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T04:54:15.891 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 5.14.0-505.el9.x86_64 2024-09-16T04:54:15.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T04:54:16.385 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T04:54:16.416 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-09-16T04:54:16.418 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-09-16T04:54:16.434 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.K379Qy3ltN 2024-09-16T04:54:16.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.K379Qy3ltN 2024-09-16T04:54:16.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.K379Qy3ltN 2024-09-16T04:54:16.566 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-09-16T04:54:16.643 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.K379Qy3ltN is 6KB 2024-09-16T04:54:16.691 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.K379Qy3ltN 2024-09-16T04:54:16.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T04:54:16.769 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T04:54:16.769 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T04:54:16.769 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T04:54:16.769 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T04:54:16.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T04:54:16.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-09-16T04:54:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T04:54:17.937 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-09-16T04:54:17.939 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-09-16T04:54:17.955 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.HswlvLWAqB 2024-09-16T04:54:17.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HswlvLWAqB 2024-09-16T04:54:18.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.HswlvLWAqB 2024-09-16T04:54:18.173 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.HswlvLWAqB is 6KB 2024-09-16T04:54:18.223 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.HswlvLWAqB 2024-09-16T04:54:18.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T04:54:18.299 INFO:teuthology.orchestra.run.smithi032.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T04:54:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T04:54:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T04:54:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T04:54:18.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T04:54:18.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-09-16T04:54:43.885 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T04:54:43.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-16T04:54:43.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:54:46.933 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-16T04:54:46.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T04:54:46.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:54:48.467 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T04:54:48.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-16T04:54:48.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:55:02.205 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-09-16T04:55:05.277 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-16T04:55:06.813 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-09-16T04:55:11.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-16T04:55:11.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:55:14.269 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-09-16T04:55:14.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T04:55:14.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:55:15.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-16T04:55:15.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:55:17.345 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-16T04:55:26.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-16T04:55:26.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:55:29.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T04:55:29.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:55:34.205 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-09-16T04:55:44.701 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-09-16T04:55:46.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-16T04:55:46.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:55:47.127 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-09-16T04:55:47.594 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-16T04:55:47.594 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T04:55:47.595 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-09-16T04:55:47.609 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-505.el9.x86_64 2024-09-16T04:55:47.610 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T04:55:47.610 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:55:47.610 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T04:55:47.773 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-16T04:55:48.611 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T04:55:48.611 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:55:48.693 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-09-16T04:55:48.724 DEBUG:teuthology.parallel:result is None 2024-09-16T04:55:59.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-16T04:55:59.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:56:00.088 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-16T04:56:00.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-16T04:56:00.446 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T04:56:00.446 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-16T04:56:00.461 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-505.el9.x86_64 2024-09-16T04:56:00.461 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T04:56:00.461 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:56:00.461 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T04:56:01.462 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T04:56:01.463 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:56:01.574 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-09-16T04:56:01.609 DEBUG:teuthology.parallel:result is None 2024-09-16T04:56:02.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T04:56:02.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:56:03.151 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-16T04:56:03.570 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T04:56:03.571 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-505.el9.x86_64"... 2024-09-16T04:56:03.571 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-16T04:56:03.585 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-505.el9.x86_64 2024-09-16T04:56:03.585 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T04:56:03.585 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:56:03.585 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T04:56:04.587 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-16T04:56:04.587 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:56:04.712 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-09-16T04:56:04.737 DEBUG:teuthology.parallel:result is None 2024-09-16T04:56:04.737 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T04:56:04.744 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T04:56:04.745 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:56:04.748 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:56:04.751 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:56:04.767 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T04:56:04.819 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T04:56:04.825 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T04:56:04.825 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:56:04.828 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:56:04.830 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:56:04.900 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T04:56:04.907 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T04:56:04.908 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:56:04.910 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:56:04.913 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:56:04.941 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:56:04.953 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:56:04.968 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:56:04.981 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:56:05.163 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:56:05.174 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:56:05.176 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T04:56:05.183 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T04:56:05.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:56:05.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:56:05.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:56:05.215 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T04:56:05.226 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T04:56:05.226 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:56:05.257 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:56:05.259 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:56:05.274 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:56:05.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:56:05.418 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:56:05.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:56:05.549 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T04:56:05.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:56:05.618 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:56:05.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:56:05.706 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:56:05.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:56:05.827 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T04:56:05.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:56:05.892 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:56:05.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:56:06.056 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:56:06.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:56:06.191 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T04:56:06.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:56:06.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-09-16T04:56:06.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-09-16T04:56:06.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-09-16T04:56:06.297 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:56:06.300 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:56:06.360 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:56:06.777 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T04:56:06.787 INFO:teuthology.task.internal:Starting timer... 2024-09-16T04:56:06.788 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T04:56:06.804 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T04:56:06.812 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T04:56:06.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-09-16T04:56:06.892 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-09-16T04:56:06.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-09-16T04:56:06.966 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-09-16T04:56:06.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-09-16T04:56:07.126 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-09-16T04:56:07.128 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T04:56:07.129 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-09-16T04:56:07.160 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-09-16T04:56:07.160 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-09-16T04:56:07.183 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-09-16T04:56:07.184 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-09-16T04:56:07.219 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-09-16T04:56:07.219 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-09-16T04:56:07.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:56:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:56:07.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:56:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:56:07.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:56:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:56:07.306 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T04:56:07.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:56:07.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:56:07.374 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:56:07.403 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T04:56:07.413 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T04:56:07.424 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T04:56:07.426 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi196.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T04:59:56.431 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-09-16T04:59:56.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-16T04:59:56.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:59:56.511 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-09-16T04:59:56.593 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-09-16T04:59:56.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-16T04:59:56.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:59:56.671 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-16T04:59:56.748 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-16T04:59:56.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T04:59:56.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:59:56.826 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-16T04:59:56.912 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T04:59:56.913 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T04:59:56.923 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T04:59:56.924 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:59:56.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:59:56.927 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:59:56.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:59:56.929 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:59:56.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:59:56.957 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:59:56.958 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:59:56.972 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:59:56.975 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:59:56.985 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:59:56.996 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:59:56.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-09-16T04:59:57.003 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-09-16T04:59:57.008 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-09-16T04:59:57.016 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-09-16T04:59:57.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-09-16T04:59:57.022 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:59:57.031 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-09-16T04:59:57.031 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:59:57.035 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:59:57.045 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:59:57.047 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:59:57.059 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:59:57.090 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:59:57.093 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:59:57.093 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-16T04:59:57.093 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.094 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.094 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.094 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.109 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:59:57.113 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:59:57.113 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-09-16T04:59:57.113 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.113 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.113 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.113 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.119 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:59:57.123 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:59:57.123 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-16T04:59:57.123 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.123 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.124 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.124 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:59:57.125 INFO:teuthology.run_tasks:Running task install... 2024-09-16T04:59:57.133 DEBUG:teuthology.task.install:project ceph 2024-09-16T04:59:57.133 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T04:59:57.133 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T04:59:57.134 INFO:teuthology.task.install:Using flavor: default 2024-09-16T04:59:57.139 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T04:59:57.140 INFO:teuthology.task.install:extra packages: [] 2024-09-16T04:59:57.141 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:59:57.141 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:59:57.141 INFO:teuthology.packaging:ref: None 2024-09-16T04:59:57.141 INFO:teuthology.packaging:tag: None 2024-09-16T04:59:57.141 INFO:teuthology.packaging:branch: quincy 2024-09-16T04:59:57.141 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:59:57.142 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-16T04:59:57.144 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:59:57.144 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:59:57.144 INFO:teuthology.packaging:ref: None 2024-09-16T04:59:57.144 INFO:teuthology.packaging:tag: None 2024-09-16T04:59:57.144 INFO:teuthology.packaging:branch: quincy 2024-09-16T04:59:57.144 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:59:57.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-16T04:59:57.145 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:59:57.145 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:59:57.146 INFO:teuthology.packaging:ref: None 2024-09-16T04:59:57.146 INFO:teuthology.packaging:tag: None 2024-09-16T04:59:57.146 INFO:teuthology.packaging:branch: quincy 2024-09-16T04:59:57.146 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:59:57.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-16T04:59:57.372 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/ 2024-09-16T04:59:57.372 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1633.g6d025bc2 2024-09-16T04:59:57.376 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/ 2024-09-16T04:59:57.377 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1633.g6d025bc2 2024-09-16T04:59:57.387 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/ 2024-09-16T04:59:57.388 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1633.g6d025bc2 2024-09-16T04:59:57.550 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:59:57.551 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T04:59:57.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:59:57.562 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:59:57.562 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T04:59:57.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:59:57.571 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/6d025bc26f3176c910707f8cb4c8d8e076640615/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:59:57.571 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T04:59:57.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:59:57.576 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:59:57.577 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:59:57.577 INFO:teuthology.packaging:ref: None 2024-09-16T04:59:57.577 INFO:teuthology.packaging:tag: None 2024-09-16T04:59:57.577 INFO:teuthology.packaging:branch: quincy 2024-09-16T04:59:57.577 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:59:57.577 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:59:57.587 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:59:57.587 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:59:57.588 INFO:teuthology.packaging:ref: None 2024-09-16T04:59:57.588 INFO:teuthology.packaging:tag: None 2024-09-16T04:59:57.588 INFO:teuthology.packaging:branch: quincy 2024-09-16T04:59:57.588 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:59:57.588 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:59:57.600 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:59:57.600 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T04:59:57.600 INFO:teuthology.packaging:ref: None 2024-09-16T04:59:57.600 INFO:teuthology.packaging:tag: None 2024-09-16T04:59:57.600 INFO:teuthology.packaging:branch: quincy 2024-09-16T04:59:57.600 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:59:57.601 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:59:57.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:59:57.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:59:57.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:59:57.720 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:59:57.728 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:59:57.747 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:59:57.785 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-09-16T04:59:57.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-09-16T04:59:57.794 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-09-16T04:59:57.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-09-16T04:59:57.816 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-09-16T04:59:57.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-09-16T04:59:58.015 INFO:teuthology.orchestra.run.smithi204.stdout:45 files removed 2024-09-16T04:59:58.023 INFO:teuthology.orchestra.run.smithi196.stdout:45 files removed 2024-09-16T04:59:58.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw 2024-09-16T04:59:58.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-09-16T04:59:58.054 INFO:teuthology.orchestra.run.smithi032.stdout:45 files removed 2024-09-16T04:59:58.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw 2024-09-16T04:59:58.605 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 278 kB/s | 77 kB 00:00 2024-09-16T04:59:58.619 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 274 kB/s | 77 kB 00:00 2024-09-16T04:59:58.653 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 283 kB/s | 77 kB 00:00 2024-09-16T04:59:58.890 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-16T04:59:58.903 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-16T04:59:58.937 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-16T04:59:59.165 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-16T04:59:59.204 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 6.8 kB/s | 1.8 kB 00:00 2024-09-16T04:59:59.213 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 7.6 kB/s | 1.8 kB 00:00 2024-09-16T04:59:59.721 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-16T05:00:01.837 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.2 MB 00:02 2024-09-16T05:00:01.982 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.2 MB 00:02 2024-09-16T05:00:04.231 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 20 MB 00:03 2024-09-16T05:00:04.266 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-09-16T05:00:07.671 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 20 MB 00:04 2024-09-16T05:00:08.965 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.5 MB 00:00 2024-09-16T05:00:09.359 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.5 MB 00:01 2024-09-16T05:00:10.707 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-16T05:00:10.776 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 19 kB 00:00 2024-09-16T05:00:12.386 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-16T05:00:12.405 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-16T05:00:13.438 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-16T05:00:15.106 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-16T05:00:16.477 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-16T05:00:17.895 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-09-16T05:00:18.000 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-09-16T05:00:20.273 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:00:20.280 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:20.280 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T05:00:20.280 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:20.280 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:00:20.280 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 14 M 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 3.1 M 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.9 M 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 5.0 M 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 22 M 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 25 k 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 637 k 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T05:00:20.281 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 457 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 4.9 M 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 45 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 120 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 156 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 320 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 295 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 99 k 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:20.282 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:00:20.283 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:20.283 INFO:teuthology.orchestra.run.smithi204.stdout:Install 22 Packages 2024-09-16T05:00:20.283 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 2 Packages 2024-09-16T05:00:20.283 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:20.283 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 58 M 2024-09-16T05:00:20.283 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:00:20.461 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:00:20.464 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-16T05:00:20.464 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:20.464 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:00:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 14 M 2024-09-16T05:00:20.464 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 3.1 M 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.9 M 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 5.0 M 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 22 M 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 25 k 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 637 k 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 457 k 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T05:00:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 4.9 M 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 45 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 120 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 156 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 320 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 295 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 99 k 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:00:20.466 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:20.467 INFO:teuthology.orchestra.run.smithi032.stdout:Install 22 Packages 2024-09-16T05:00:20.467 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages 2024-09-16T05:00:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:20.467 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 58 M 2024-09-16T05:00:20.467 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:00:21.622 INFO:teuthology.orchestra.run.smithi204.stdout:(1/24): ceph-base-17.2.7-1633.g6d025bc2.el9.x86 4.7 MB/s | 5.0 MB 00:01 2024-09-16T05:00:21.660 INFO:teuthology.orchestra.run.smithi032.stdout:(1/24): ceph-base-17.2.7-1633.g6d025bc2.el9.x86 6.3 MB/s | 5.0 MB 00:00 2024-09-16T05:00:21.672 INFO:teuthology.orchestra.run.smithi204.stdout:(2/24): ceph-selinux-17.2.7-1633.g6d025bc2.el9. 510 kB/s | 25 kB 00:00 2024-09-16T05:00:21.718 INFO:teuthology.orchestra.run.smithi032.stdout:(2/24): ceph-selinux-17.2.7-1633.g6d025bc2.el9. 432 kB/s | 25 kB 00:00 2024-09-16T05:00:21.827 INFO:teuthology.orchestra.run.smithi032.stdout:(3/24): libcephfs2-17.2.7-1633.g6d025bc2.el9.x8 5.7 MB/s | 637 kB 00:00 2024-09-16T05:00:21.830 INFO:teuthology.orchestra.run.smithi204.stdout:(3/24): libcephfs2-17.2.7-1633.g6d025bc2.el9.x8 3.9 MB/s | 637 kB 00:00 2024-09-16T05:00:21.927 INFO:teuthology.orchestra.run.smithi032.stdout:(4/24): libradosstriper1-17.2.7-1633.g6d025bc2. 4.5 MB/s | 457 kB 00:00 2024-09-16T05:00:21.989 INFO:teuthology.orchestra.run.smithi204.stdout:(4/24): libradosstriper1-17.2.7-1633.g6d025bc2. 2.8 MB/s | 457 kB 00:00 2024-09-16T05:00:22.074 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-16T05:00:22.328 INFO:teuthology.orchestra.run.smithi032.stdout:(5/24): ceph-radosgw-17.2.7-1633.g6d025bc2.el9. 9.9 MB/s | 14 MB 00:01 2024-09-16T05:00:22.570 INFO:teuthology.orchestra.run.smithi032.stdout:(6/24): ceph-common-17.2.7-1633.g6d025bc2.el9.x 13 MB/s | 22 MB 00:01 2024-09-16T05:00:22.596 INFO:teuthology.orchestra.run.smithi032.stdout:(7/24): python3-ceph-argparse-17.2.7-1633.g6d02 169 kB/s | 45 kB 00:00 2024-09-16T05:00:22.662 INFO:teuthology.orchestra.run.smithi032.stdout:(8/24): librgw2-17.2.7-1633.g6d025bc2.el9.x86_6 6.7 MB/s | 4.9 MB 00:00 2024-09-16T05:00:22.687 INFO:teuthology.orchestra.run.smithi032.stdout:(9/24): python3-ceph-common-17.2.7-1633.g6d025b 1.0 MB/s | 120 kB 00:00 2024-09-16T05:00:22.712 INFO:teuthology.orchestra.run.smithi032.stdout:(10/24): python3-cephfs-17.2.7-1633.g6d025bc2.e 1.3 MB/s | 156 kB 00:00 2024-09-16T05:00:22.737 INFO:teuthology.orchestra.run.smithi032.stdout:(11/24): python3-rados-17.2.7-1633.g6d025bc2.el 4.2 MB/s | 320 kB 00:00 2024-09-16T05:00:22.756 INFO:teuthology.orchestra.run.smithi204.stdout:(5/24): ceph-radosgw-17.2.7-1633.g6d025bc2.el9. 6.6 MB/s | 14 MB 00:02 2024-09-16T05:00:22.762 INFO:teuthology.orchestra.run.smithi032.stdout:(12/24): python3-rbd-17.2.7-1633.g6d025bc2.el9. 3.8 MB/s | 295 kB 00:00 2024-09-16T05:00:22.788 INFO:teuthology.orchestra.run.smithi032.stdout:(13/24): python3-rgw-17.2.7-1633.g6d025bc2.el9. 1.3 MB/s | 99 kB 00:00 2024-09-16T05:00:22.831 INFO:teuthology.orchestra.run.smithi204.stdout:(6/24): librgw2-17.2.7-1633.g6d025bc2.el9.x86_6 5.8 MB/s | 4.9 MB 00:00 2024-09-16T05:00:22.856 INFO:teuthology.orchestra.run.smithi204.stdout:(7/24): python3-ceph-argparse-17.2.7-1633.g6d02 451 kB/s | 45 kB 00:00 2024-09-16T05:00:22.881 INFO:teuthology.orchestra.run.smithi204.stdout:(8/24): python3-ceph-common-17.2.7-1633.g6d025b 2.3 MB/s | 120 kB 00:00 2024-09-16T05:00:22.907 INFO:teuthology.orchestra.run.smithi204.stdout:(9/24): python3-cephfs-17.2.7-1633.g6d025bc2.el 3.0 MB/s | 156 kB 00:00 2024-09-16T05:00:22.930 INFO:teuthology.orchestra.run.smithi032.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 271 kB/s | 45 kB 00:00 2024-09-16T05:00:22.932 INFO:teuthology.orchestra.run.smithi204.stdout:(10/24): python3-rados-17.2.7-1633.g6d025bc2.el 6.2 MB/s | 320 kB 00:00 2024-09-16T05:00:22.955 INFO:teuthology.orchestra.run.smithi032.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 834 kB/s | 160 kB 00:00 2024-09-16T05:00:22.957 INFO:teuthology.orchestra.run.smithi204.stdout:(11/24): python3-rbd-17.2.7-1633.g6d025bc2.el9. 5.7 MB/s | 295 kB 00:00 2024-09-16T05:00:22.983 INFO:teuthology.orchestra.run.smithi204.stdout:(12/24): python3-rgw-17.2.7-1633.g6d025bc2.el9. 1.9 MB/s | 99 kB 00:00 2024-09-16T05:00:23.006 INFO:teuthology.orchestra.run.smithi032.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-09-16T05:00:23.047 INFO:teuthology.orchestra.run.smithi032.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-16T05:00:23.089 INFO:teuthology.orchestra.run.smithi032.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-16T05:00:23.140 INFO:teuthology.orchestra.run.smithi032.stdout:(19/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-16T05:00:23.165 INFO:teuthology.orchestra.run.smithi032.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 421 kB/s | 49 kB 00:00 2024-09-16T05:00:23.215 INFO:teuthology.orchestra.run.smithi032.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 572 kB/s | 67 kB 00:00 2024-09-16T05:00:23.249 INFO:teuthology.orchestra.run.smithi204.stdout:(13/24): ceph-common-17.2.7-1633.g6d025bc2.el9. 8.4 MB/s | 22 MB 00:02 2024-09-16T05:00:23.266 INFO:teuthology.orchestra.run.smithi204.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 160 kB/s | 45 kB 00:00 2024-09-16T05:00:23.274 INFO:teuthology.orchestra.run.smithi032.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-16T05:00:23.325 INFO:teuthology.orchestra.run.smithi204.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 436 kB/s | 160 kB 00:00 2024-09-16T05:00:23.367 INFO:teuthology.orchestra.run.smithi204.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T05:00:23.420 INFO:teuthology.orchestra.run.smithi204.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-16T05:00:23.424 INFO:teuthology.orchestra.run.smithi032.stdout:(23/24): librados2-17.2.7-1633.g6d025bc2.el9.x8 12 MB/s | 3.1 MB 00:00 2024-09-16T05:00:23.442 INFO:teuthology.orchestra.run.smithi204.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T05:00:23.468 INFO:teuthology.orchestra.run.smithi204.stdout:(19/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-16T05:00:23.482 INFO:teuthology.orchestra.run.smithi032.stdout:(24/24): librbd1-17.2.7-1633.g6d025bc2.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-09-16T05:00:23.488 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:00:23.488 INFO:teuthology.orchestra.run.smithi032.stdout:Total 19 MB/s | 58 MB 00:03 2024-09-16T05:00:23.493 INFO:teuthology.orchestra.run.smithi204.stdout:(20/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-09-16T05:00:23.527 INFO:teuthology.orchestra.run.smithi204.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 802 kB/s | 67 kB 00:00 2024-09-16T05:00:23.557 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:00:23.593 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:00:23.593 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:00:23.602 INFO:teuthology.orchestra.run.smithi204.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-16T05:00:23.761 INFO:teuthology.orchestra.run.smithi204.stdout:(23/24): librbd1-17.2.7-1633.g6d025bc2.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-16T05:00:23.894 INFO:teuthology.orchestra.run.smithi204.stdout:(24/24): librados2-17.2.7-1633.g6d025bc2.el9.x8 7.8 MB/s | 3.1 MB 00:00 2024-09-16T05:00:23.898 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:00:23.898 INFO:teuthology.orchestra.run.smithi204.stdout:Total 16 MB/s | 58 MB 00:03 2024-09-16T05:00:23.959 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:00:23.992 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:00:23.992 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:00:24.265 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:00:24.265 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:00:24.467 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:00:24.497 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:00:24.497 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-16T05:00:24.497 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:00:24.497 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:00:24.497 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 14 M 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 3.1 M 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.9 M 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 5.0 M 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 22 M 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 25 k 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 637 k 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T05:00:24.498 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 457 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 4.9 M 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 45 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 120 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 156 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 320 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 295 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 99 k 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T05:00:24.499 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:24.500 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:00:24.500 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:00:24.500 INFO:teuthology.orchestra.run.smithi196.stdout:Install 22 Packages 2024-09-16T05:00:24.500 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-09-16T05:00:24.500 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:24.500 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 58 M 2024-09-16T05:00:24.500 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:00:24.644 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:00:24.645 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:00:25.059 INFO:teuthology.orchestra.run.smithi196.stdout:(1/24): ceph-base-17.2.7-1633.g6d025bc2.el9.x86 11 MB/s | 5.0 MB 00:00 2024-09-16T05:00:25.117 INFO:teuthology.orchestra.run.smithi196.stdout:(2/24): ceph-selinux-17.2.7-1633.g6d025bc2.el9. 433 kB/s | 25 kB 00:00 2024-09-16T05:00:25.217 INFO:teuthology.orchestra.run.smithi196.stdout:(3/24): libcephfs2-17.2.7-1633.g6d025bc2.el9.x8 6.2 MB/s | 637 kB 00:00 2024-09-16T05:00:25.293 INFO:teuthology.orchestra.run.smithi196.stdout:(4/24): libradosstriper1-17.2.7-1633.g6d025bc2. 5.9 MB/s | 457 kB 00:00 2024-09-16T05:00:25.523 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:00:25.643 INFO:teuthology.orchestra.run.smithi196.stdout:(5/24): ceph-radosgw-17.2.7-1633.g6d025bc2.el9. 14 MB/s | 14 MB 00:01 2024-09-16T05:00:25.718 INFO:teuthology.orchestra.run.smithi196.stdout:(6/24): librgw2-17.2.7-1633.g6d025bc2.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-09-16T05:00:25.743 INFO:teuthology.orchestra.run.smithi196.stdout:(7/24): python3-ceph-argparse-17.2.7-1633.g6d02 450 kB/s | 45 kB 00:00 2024-09-16T05:00:25.786 INFO:teuthology.orchestra.run.smithi196.stdout:(8/24): python3-ceph-common-17.2.7-1633.g6d025b 1.7 MB/s | 120 kB 00:00 2024-09-16T05:00:25.811 INFO:teuthology.orchestra.run.smithi196.stdout:(9/24): python3-cephfs-17.2.7-1633.g6d025bc2.el 2.3 MB/s | 156 kB 00:00 2024-09-16T05:00:25.836 INFO:teuthology.orchestra.run.smithi196.stdout:(10/24): python3-rados-17.2.7-1633.g6d025bc2.el 6.3 MB/s | 320 kB 00:00 2024-09-16T05:00:25.861 INFO:teuthology.orchestra.run.smithi196.stdout:(11/24): python3-rbd-17.2.7-1633.g6d025bc2.el9. 5.7 MB/s | 295 kB 00:00 2024-09-16T05:00:25.874 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:00:25.895 INFO:teuthology.orchestra.run.smithi196.stdout:(12/24): python3-rgw-17.2.7-1633.g6d025bc2.el9. 1.6 MB/s | 99 kB 00:00 2024-09-16T05:00:25.924 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T05:00:25.977 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:26.121 INFO:teuthology.orchestra.run.smithi196.stdout:(13/24): ceph-common-17.2.7-1633.g6d025bc2.el9. 15 MB/s | 22 MB 00:01 2024-09-16T05:00:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/26 2024-09-16T05:00:26.187 INFO:teuthology.orchestra.run.smithi196.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 492 kB/s | 160 kB 00:00 2024-09-16T05:00:26.197 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:26.212 INFO:teuthology.orchestra.run.smithi196.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 143 kB/s | 45 kB 00:00 2024-09-16T05:00:26.254 INFO:teuthology.orchestra.run.smithi196.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T05:00:26.261 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T05:00:26.285 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:26.288 INFO:teuthology.orchestra.run.smithi196.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-09-16T05:00:26.322 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:26.330 INFO:teuthology.orchestra.run.smithi196.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-16T05:00:26.337 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T05:00:26.380 INFO:teuthology.orchestra.run.smithi196.stdout:(19/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-16T05:00:26.383 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T05:00:26.405 INFO:teuthology.orchestra.run.smithi196.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 442 kB/s | 49 kB 00:00 2024-09-16T05:00:26.429 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:26.431 INFO:teuthology.orchestra.run.smithi196.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-09-16T05:00:26.509 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/26 2024-09-16T05:00:26.514 INFO:teuthology.orchestra.run.smithi196.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-16T05:00:26.531 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:26.590 INFO:teuthology.orchestra.run.smithi196.stdout:(23/24): librados2-17.2.7-1633.g6d025bc2.el9.x8 17 MB/s | 3.1 MB 00:00 2024-09-16T05:00:26.620 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:26.656 INFO:teuthology.orchestra.run.smithi196.stdout:(24/24): librbd1-17.2.7-1633.g6d025bc2.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-16T05:00:26.661 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:00:26.662 INFO:teuthology.orchestra.run.smithi196.stdout:Total 27 MB/s | 58 MB 00:02 2024-09-16T05:00:26.665 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T05:00:26.695 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T05:00:26.724 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:00:26.757 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:00:26.757 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:00:26.847 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T05:00:26.873 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 8/26 2024-09-16T05:00:26.936 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 8/26 2024-09-16T05:00:26.973 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T05:00:27.019 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9. 10/26 2024-09-16T05:00:27.061 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 11/26 2024-09-16T05:00:27.125 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 12/26 2024-09-16T05:00:27.133 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T05:00:27.151 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 13/26 2024-09-16T05:00:27.160 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 8/26 2024-09-16T05:00:27.206 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 8/26 2024-09-16T05:00:27.220 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 13/26 2024-09-16T05:00:27.251 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T05:00:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T05:00:27.304 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9. 10/26 2024-09-16T05:00:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T05:00:27.339 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 11/26 2024-09-16T05:00:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T05:00:27.401 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 12/26 2024-09-16T05:00:27.422 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:00:27.422 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:00:27.429 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 13/26 2024-09-16T05:00:27.495 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 13/26 2024-09-16T05:00:27.534 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T05:00:27.571 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T05:00:27.596 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T05:00:27.658 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T05:00:27.704 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 18/26 2024-09-16T05:00:27.795 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 18/26 2024-09-16T05:00:27.871 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 19/26 2024-09-16T05:00:27.898 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x8 20/26 2024-09-16T05:00:27.900 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T05:00:27.950 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 18/26 2024-09-16T05:00:28.039 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 18/26 2024-09-16T05:00:28.114 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 19/26 2024-09-16T05:00:28.144 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x8 20/26 2024-09-16T05:00:28.696 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:00:29.088 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T05:00:29.142 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:29.234 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:29.320 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/26 2024-09-16T05:00:29.369 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:29.458 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:29.516 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T05:00:29.549 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T05:00:29.995 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T05:00:30.030 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 8/26 2024-09-16T05:00:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 8/26 2024-09-16T05:00:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T05:00:30.176 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9. 10/26 2024-09-16T05:00:30.218 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 11/26 2024-09-16T05:00:30.281 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 12/26 2024-09-16T05:00:30.308 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 13/26 2024-09-16T05:00:30.374 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 13/26 2024-09-16T05:00:30.413 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T05:00:30.450 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T05:00:30.452 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:30.484 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T05:00:30.502 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:30.503 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:30.540 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:30.786 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T05:00:30.837 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 18/26 2024-09-16T05:00:30.927 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 18/26 2024-09-16T05:00:31.001 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 19/26 2024-09-16T05:00:31.032 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x8 20/26 2024-09-16T05:00:31.055 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:31.071 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:31.098 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 22/26 2024-09-16T05:00:31.119 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 22/26 2024-09-16T05:00:31.199 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 22/26 2024-09-16T05:00:31.199 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T05:00:31.199 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T05:00:31.199 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:31.217 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 22/26 2024-09-16T05:00:31.217 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T05:00:31.217 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T05:00:31.217 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:31.241 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:31.245 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:33.959 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 21/26 2024-09-16T05:00:33.998 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 22/26 2024-09-16T05:00:34.096 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 22/26 2024-09-16T05:00:34.097 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T05:00:34.097 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T05:00:34.097 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:34.133 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:41.161 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:41.161 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-09-16T05:00:41.162 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-09-16T05:00:41.162 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-09-16T05:00:41.162 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-09-16T05:00:41.162 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-09-16T05:00:41.162 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-09-16T05:00:41.162 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-09-16T05:00:41.162 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:41.713 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:41.713 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-09-16T05:00:41.713 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-09-16T05:00:41.713 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-09-16T05:00:41.713 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-09-16T05:00:41.714 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-09-16T05:00:41.714 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-09-16T05:00:41.714 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-09-16T05:00:41.714 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 24/26 2024-09-16T05:00:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 24/26 2024-09-16T05:00:42.595 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:00:42.595 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T05:00:42.595 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:00:42.595 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:00:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T05:00:42.686 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T05:00:42.686 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:43.070 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 24/26 2024-09-16T05:00:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 24/26 2024-09-16T05:00:43.147 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:00:43.147 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T05:00:43.147 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:00:43.147 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:00:43.147 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:43.149 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T05:00:43.240 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T05:00:43.240 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:43.951 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 5/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 6/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 7/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9. 8/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x8 9/26 2024-09-16T05:00:43.952 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 10/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 11/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 12/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 13/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T05:00:43.953 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T05:00:43.954 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:43.954 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T05:00:43.954 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 25/26 2024-09-16T05:00:44.106 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:44.106 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-09-16T05:00:44.106 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-09-16T05:00:44.106 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-09-16T05:00:44.106 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-09-16T05:00:44.106 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-09-16T05:00:44.106 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-09-16T05:00:44.106 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-09-16T05:00:44.107 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:44.636 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/26 2024-09-16T05:00:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/26 2024-09-16T05:00:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 5/26 2024-09-16T05:00:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 6/26 2024-09-16T05:00:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 7/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9. 8/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x8 9/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 10/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 11/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 12/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 13/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T05:00:44.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T05:00:44.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T05:00:44.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:44.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T05:00:44.639 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 25/26 2024-09-16T05:00:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:45.004 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T05:00:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T05:00:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T05:00:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:45.007 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:00:45.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test 2024-09-16T05:00:45.443 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 24/26 2024-09-16T05:00:45.508 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:45.509 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-09-16T05:00:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.509 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:00:45.509 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T05:00:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T05:00:45.510 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T05:00:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T05:00:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T05:00:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T05:00:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T05:00:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.511 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T05:00:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.512 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T05:00:45.513 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:45.513 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:00:45.531 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 24/26 2024-09-16T05:00:45.531 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:00:45.531 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T05:00:45.532 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:00:45.532 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:00:45.532 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:45.533 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T05:00:45.622 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T05:00:45.622 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:45.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test 2024-09-16T05:00:45.879 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:00:46.019 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:00:46.019 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:46.019 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 48 M 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 163 k 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-09-16T05:00:46.020 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:46.021 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 48 M 2024-09-16T05:00:46.021 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 193 M 2024-09-16T05:00:46.021 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:00:46.166 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-16T05:00:46.208 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T05:00:46.266 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): libcephsqlite-17.2.7-1633.g6d025bc2.el9. 732 kB/s | 163 kB 00:00 2024-09-16T05:00:46.458 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:29 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:00:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:00:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:46.606 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 48 M 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 163 k 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-09-16T05:00:46.607 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 48 M 2024-09-16T05:00:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 193 M 2024-09-16T05:00:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:00:46.880 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.7 MB/s | 304 kB 00:00 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 5/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_6 6/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 7/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9. 8/26 2024-09-16T05:00:46.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x8 9/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 10/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 11/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 12/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 13/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T05:00:46.907 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T05:00:46.908 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 23/26 2024-09-16T05:00:46.908 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T05:00:46.908 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 25/26 2024-09-16T05:00:46.947 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): libcephsqlite-17.2.7-1633.g6d025bc2.el9. 684 kB/s | 163 kB 00:00 2024-09-16T05:00:46.972 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 695 kB/s | 64 kB 00:00 2024-09-16T05:00:47.689 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T05:00:47.689 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:47.689 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.690 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.691 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.692 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:47.692 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T05:00:47.692 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:47.692 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:00:48.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-09-16T05:00:48.630 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:26 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:00:48.771 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 48 M 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 163 k 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:00:48.772 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:00:48.773 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-09-16T05:00:48.773 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:48.773 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 48 M 2024-09-16T05:00:48.773 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 193 M 2024-09-16T05:00:48.773 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:00:49.064 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): ceph-test-17.2.7-1633.g6d025bc2.el9.x86_ 20 MB/s | 48 MB 00:02 2024-09-16T05:00:49.066 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:00:49.066 INFO:teuthology.orchestra.run.smithi032.stdout:Total 20 MB/s | 48 MB 00:02 2024-09-16T05:00:49.094 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-16T05:00:49.095 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:00:49.116 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:00:49.116 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:00:49.161 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): libcephsqlite-17.2.7-1633.g6d025bc2.el9. 620 kB/s | 163 kB 00:00 2024-09-16T05:00:49.261 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 381 kB/s | 64 kB 00:00 2024-09-16T05:00:49.502 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:00:49.502 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:00:49.550 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): ceph-test-17.2.7-1633.g6d025bc2.el9.x86_ 14 MB/s | 48 MB 00:03 2024-09-16T05:00:49.552 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:00:49.553 INFO:teuthology.orchestra.run.smithi204.stdout:Total 14 MB/s | 48 MB 00:03 2024-09-16T05:00:49.581 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:00:49.600 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:00:49.600 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:00:49.965 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:00:49.966 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:00:50.057 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:00:50.142 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T05:00:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T05:00:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/4 2024-09-16T05:00:50.492 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:00:50.574 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T05:00:50.625 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T05:00:50.652 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/4 2024-09-16T05:00:51.461 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): ceph-test-17.2.7-1633.g6d025bc2.el9.x86_ 19 MB/s | 48 MB 00:02 2024-09-16T05:00:51.463 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:00:51.464 INFO:teuthology.orchestra.run.smithi196.stdout:Total 18 MB/s | 48 MB 00:02 2024-09-16T05:00:51.492 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:00:51.511 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:00:51.511 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:00:51.876 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:00:51.876 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:00:52.397 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:00:52.495 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T05:00:52.540 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T05:00:52.571 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/4 2024-09-16T05:00:54.865 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/4 2024-09-16T05:00:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/4 2024-09-16T05:00:55.024 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/4 2024-09-16T05:00:55.091 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/4 2024-09-16T05:00:56.984 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/4 2024-09-16T05:00:57.060 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/4 2024-09-16T05:00:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/4 2024-09-16T05:00:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/4 2024-09-16T05:00:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/4 2024-09-16T05:00:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T05:00:57.427 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/4 2024-09-16T05:00:57.427 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/4 2024-09-16T05:00:57.427 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/4 2024-09-16T05:00:57.428 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T05:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T05:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T05:00:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T05:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:57.618 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:00:57.764 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T05:00:57.764 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:57.765 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:00:57.765 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:57.765 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:57.765 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T05:00:57.765 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T05:00:57.765 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:57.765 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:00:57.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph 2024-09-16T05:00:58.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph 2024-09-16T05:00:58.470 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:41 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:00:58.617 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:00:58.621 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:41 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:00:58.621 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:58.621 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 6.5 k 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.1 M 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 1.4 M 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1633.g6d025bc2.el9 ceph-noarch 240 k 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 3.9 M 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 15 M 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T05:00:58.622 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T05:00:58.623 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T05:00:58.624 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:58.625 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:00:58.625 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:00:58.625 INFO:teuthology.orchestra.run.smithi032.stdout:Install 34 Packages 2024-09-16T05:00:58.625 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:00:58.625 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 27 M 2024-09-16T05:00:58.625 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 96 M 2024-09-16T05:00:58.625 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:00:58.757 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:00:58.759 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:58.759 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T05:00:58.759 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:58.759 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:00:58.759 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 6.5 k 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.1 M 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 1.4 M 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1633.g6d025bc2.el9 ceph-noarch 240 k 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 3.9 M 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 15 M 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T05:00:58.760 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T05:00:58.761 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:00:58.762 INFO:teuthology.orchestra.run.smithi204.stdout:Install 34 Packages 2024-09-16T05:00:58.763 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:00:58.763 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 27 M 2024-09-16T05:00:58.763 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 96 M 2024-09-16T05:00:58.764 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:00:59.093 INFO:teuthology.orchestra.run.smithi204.stdout:(1/34): ceph-17.2.7-1633.g6d025bc2.el9.x86_64.r 32 kB/s | 6.5 kB 00:00 2024-09-16T05:00:59.307 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/4 2024-09-16T05:00:59.307 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/4 2024-09-16T05:00:59.307 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/4 2024-09-16T05:00:59.307 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T05:00:59.440 INFO:teuthology.orchestra.run.smithi032.stdout:(1/34): ceph-17.2.7-1633.g6d025bc2.el9.x86_64.r 54 kB/s | 6.5 kB 00:00 2024-09-16T05:00:59.560 INFO:teuthology.orchestra.run.smithi204.stdout:(2/34): ceph-mgr-17.2.7-1633.g6d025bc2.el9.x86_ 2.1 MB/s | 1.4 MB 00:00 2024-09-16T05:00:59.629 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T05:00:59.630 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:59.630 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:00:59.630 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:59.630 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:00:59.630 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T05:00:59.630 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T05:00:59.630 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:00:59.630 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:00:59.694 INFO:teuthology.orchestra.run.smithi204.stdout:(3/34): ceph-mds-17.2.7-1633.g6d025bc2.el9.x86_ 2.6 MB/s | 2.1 MB 00:00 2024-09-16T05:00:59.761 INFO:teuthology.orchestra.run.smithi204.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1633.g6d02 3.5 MB/s | 240 kB 00:00 2024-09-16T05:00:59.849 INFO:teuthology.orchestra.run.smithi032.stdout:(2/34): ceph-mgr-17.2.7-1633.g6d025bc2.el9.x86_ 2.7 MB/s | 1.4 MB 00:00 2024-09-16T05:00:59.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-09-16T05:00:59.969 INFO:teuthology.orchestra.run.smithi204.stdout:(5/34): ceph-mon-17.2.7-1633.g6d025bc2.el9.x86_ 4.5 MB/s | 3.9 MB 00:00 2024-09-16T05:00:59.982 INFO:teuthology.orchestra.run.smithi032.stdout:(3/34): ceph-mds-17.2.7-1633.g6d025bc2.el9.x86_ 3.1 MB/s | 2.1 MB 00:00 2024-09-16T05:01:00.049 INFO:teuthology.orchestra.run.smithi032.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1633.g6d02 3.5 MB/s | 240 kB 00:00 2024-09-16T05:01:00.249 INFO:teuthology.orchestra.run.smithi032.stdout:(5/34): ceph-mon-17.2.7-1633.g6d025bc2.el9.x86_ 4.8 MB/s | 3.9 MB 00:00 2024-09-16T05:01:00.283 INFO:teuthology.orchestra.run.smithi032.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-16T05:01:00.333 INFO:teuthology.orchestra.run.smithi032.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-16T05:01:00.383 INFO:teuthology.orchestra.run.smithi032.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-16T05:01:00.442 INFO:teuthology.orchestra.run.smithi032.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-16T05:01:00.500 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:38 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:00.500 INFO:teuthology.orchestra.run.smithi032.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T05:01:00.610 INFO:teuthology.orchestra.run.smithi032.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 3.4 MB/s | 1.2 MB 00:00 2024-09-16T05:01:00.639 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:00.641 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:00.641 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 6.5 k 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.1 M 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 1.4 M 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1633.g6d025bc2.el9 ceph-noarch 240 k 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 3.9 M 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 15 M 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T05:01:00.642 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T05:01:00.643 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T05:01:00.643 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T05:01:00.643 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T05:01:00.643 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections 2024-09-16T05:01:00.643 INFO:teuthology.orchestra.run.smithi196.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T05:01:00.643 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T05:01:00.643 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T05:01:00.643 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T05:01:00.644 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T05:01:00.644 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T05:01:00.644 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T05:01:00.644 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T05:01:00.644 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T05:01:00.645 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T05:01:00.645 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T05:01:00.645 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T05:01:00.645 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T05:01:00.646 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T05:01:00.646 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T05:01:00.646 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T05:01:00.646 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T05:01:00.646 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T05:01:00.647 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T05:01:00.647 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T05:01:00.647 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T05:01:00.647 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:00.647 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:00.647 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:00.648 INFO:teuthology.orchestra.run.smithi196.stdout:Install 34 Packages 2024-09-16T05:01:00.649 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:00.650 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 27 M 2024-09-16T05:01:00.650 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 96 M 2024-09-16T05:01:00.650 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:00.651 INFO:teuthology.orchestra.run.smithi032.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T05:01:00.662 INFO:teuthology.orchestra.run.smithi204.stdout:(6/34): ceph-osd-17.2.7-1633.g6d025bc2.el9.x86_ 14 MB/s | 15 MB 00:01 2024-09-16T05:01:00.693 INFO:teuthology.orchestra.run.smithi032.stdout:(13/34): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-16T05:01:00.695 INFO:teuthology.orchestra.run.smithi204.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 271 kB/s | 253 kB 00:00 2024-09-16T05:01:00.719 INFO:teuthology.orchestra.run.smithi032.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-16T05:01:00.745 INFO:teuthology.orchestra.run.smithi032.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T05:01:00.777 INFO:teuthology.orchestra.run.smithi032.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 713 kB/s | 42 kB 00:00 2024-09-16T05:01:00.811 INFO:teuthology.orchestra.run.smithi032.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 651 kB/s | 43 kB 00:00 2024-09-16T05:01:00.853 INFO:teuthology.orchestra.run.smithi032.stdout:(18/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.3 MB/s | 172 kB 00:00 2024-09-16T05:01:00.879 INFO:teuthology.orchestra.run.smithi032.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 433 kB/s | 11 kB 00:00 2024-09-16T05:01:00.904 INFO:teuthology.orchestra.run.smithi032.stdout:(20/34): python3-cherrypy-18.6.1-2.el9.noarch.r 3.9 MB/s | 358 kB 00:00 2024-09-16T05:01:00.930 INFO:teuthology.orchestra.run.smithi032.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-16T05:01:00.957 INFO:teuthology.orchestra.run.smithi032.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 695 kB/s | 19 kB 00:00 2024-09-16T05:01:00.982 INFO:teuthology.orchestra.run.smithi032.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 435 kB/s | 23 kB 00:00 2024-09-16T05:01:00.987 INFO:teuthology.orchestra.run.smithi204.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 1.2 MB/s | 1.2 MB 00:01 2024-09-16T05:01:01.008 INFO:teuthology.orchestra.run.smithi032.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-16T05:01:01.033 INFO:teuthology.orchestra.run.smithi032.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-16T05:01:01.058 INFO:teuthology.orchestra.run.smithi032.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T05:01:01.084 INFO:teuthology.orchestra.run.smithi032.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 664 kB/s | 16 kB 00:00 2024-09-16T05:01:01.117 INFO:teuthology.orchestra.run.smithi032.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-09-16T05:01:01.142 INFO:teuthology.orchestra.run.smithi032.stdout:(29/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-09-16T05:01:01.159 INFO:teuthology.orchestra.run.smithi032.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 858 kB/s | 36 kB 00:00 2024-09-16T05:01:01.193 INFO:teuthology.orchestra.run.smithi032.stdout:(31/34): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T05:01:01.211 INFO:teuthology.orchestra.run.smithi032.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-16T05:01:01.221 INFO:teuthology.orchestra.run.smithi196.stdout:(1/34): ceph-17.2.7-1633.g6d025bc2.el9.x86_64.r 46 kB/s | 6.5 kB 00:00 2024-09-16T05:01:01.244 INFO:teuthology.orchestra.run.smithi032.stdout:(33/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-16T05:01:01.396 INFO:teuthology.orchestra.run.smithi204.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 193 kB/s | 135 kB 00:00 2024-09-16T05:01:01.410 INFO:teuthology.orchestra.run.smithi032.stdout:(34/34): ceph-osd-17.2.7-1633.g6d025bc2.el9.x86 9.8 MB/s | 15 MB 00:01 2024-09-16T05:01:01.414 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:01.415 INFO:teuthology.orchestra.run.smithi032.stdout:Total 9.7 MB/s | 27 MB 00:02 2024-09-16T05:01:01.421 INFO:teuthology.orchestra.run.smithi204.stdout:(10/34): python3-ply-3.11-14.el9.noarch.rpm 140 kB/s | 106 kB 00:00 2024-09-16T05:01:01.546 INFO:teuthology.orchestra.run.smithi204.stdout:(11/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T05:01:01.551 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:01.592 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:01.592 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:01.605 INFO:teuthology.orchestra.run.smithi196.stdout:(2/34): ceph-mgr-17.2.7-1633.g6d025bc2.el9.x86_ 2.7 MB/s | 1.4 MB 00:00 2024-09-16T05:01:01.613 INFO:teuthology.orchestra.run.smithi204.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 523 kB/s | 35 kB 00:00 2024-09-16T05:01:01.680 INFO:teuthology.orchestra.run.smithi204.stdout:(13/34): python3-packaging-20.9-5.el9.noarch.rp 1.1 MB/s | 77 kB 00:00 2024-09-16T05:01:01.713 INFO:teuthology.orchestra.run.smithi196.stdout:(3/34): ceph-mds-17.2.7-1633.g6d025bc2.el9.x86_ 3.3 MB/s | 2.1 MB 00:00 2024-09-16T05:01:01.722 INFO:teuthology.orchestra.run.smithi204.stdout:(14/34): python3-requests-2.25.1-8.el9.noarch.r 170 kB/s | 125 kB 00:00 2024-09-16T05:01:01.756 INFO:teuthology.orchestra.run.smithi204.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-09-16T05:01:01.780 INFO:teuthology.orchestra.run.smithi196.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1633.g6d02 3.5 MB/s | 240 kB 00:00 2024-09-16T05:01:01.806 INFO:teuthology.orchestra.run.smithi204.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 871 kB/s | 43 kB 00:00 2024-09-16T05:01:01.840 INFO:teuthology.orchestra.run.smithi204.stdout:(17/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.1 MB/s | 172 kB 00:00 2024-09-16T05:01:01.865 INFO:teuthology.orchestra.run.smithi204.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 295 kB/s | 42 kB 00:00 2024-09-16T05:01:01.899 INFO:teuthology.orchestra.run.smithi204.stdout:(19/34): python3-cherrypy-18.6.1-2.el9.noarch.r 6.0 MB/s | 358 kB 00:00 2024-09-16T05:01:01.924 INFO:teuthology.orchestra.run.smithi204.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 770 kB/s | 18 kB 00:00 2024-09-16T05:01:01.947 INFO:teuthology.orchestra.run.smithi196.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-16T05:01:01.958 INFO:teuthology.orchestra.run.smithi204.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-09-16T05:01:01.983 INFO:teuthology.orchestra.run.smithi204.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 393 kB/s | 23 kB 00:00 2024-09-16T05:01:02.009 INFO:teuthology.orchestra.run.smithi204.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-16T05:01:02.031 INFO:teuthology.orchestra.run.smithi196.stdout:(6/34): ceph-mon-17.2.7-1633.g6d025bc2.el9.x86_ 4.8 MB/s | 3.9 MB 00:00 2024-09-16T05:01:02.033 INFO:teuthology.orchestra.run.smithi204.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-16T05:01:02.034 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:02.035 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:02.059 INFO:teuthology.orchestra.run.smithi204.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-16T05:01:02.084 INFO:teuthology.orchestra.run.smithi204.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T05:01:02.098 INFO:teuthology.orchestra.run.smithi196.stdout:(7/34): python3-cryptography-36.0.1-4.el9.x86_6 8.2 MB/s | 1.2 MB 00:00 2024-09-16T05:01:02.109 INFO:teuthology.orchestra.run.smithi204.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 658 kB/s | 16 kB 00:00 2024-09-16T05:01:02.135 INFO:teuthology.orchestra.run.smithi204.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-16T05:01:02.148 INFO:teuthology.orchestra.run.smithi196.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-16T05:01:02.160 INFO:teuthology.orchestra.run.smithi204.stdout:(29/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-16T05:01:02.174 INFO:teuthology.orchestra.run.smithi196.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 748 kB/s | 106 kB 00:00 2024-09-16T05:01:02.186 INFO:teuthology.orchestra.run.smithi204.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-16T05:01:02.199 INFO:teuthology.orchestra.run.smithi196.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-16T05:01:02.211 INFO:teuthology.orchestra.run.smithi204.stdout:(31/34): python3-urllib3-1.26.5-6.el9.noarch.rp 264 kB/s | 215 kB 00:00 2024-09-16T05:01:02.245 INFO:teuthology.orchestra.run.smithi204.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-16T05:01:02.266 INFO:teuthology.orchestra.run.smithi196.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-16T05:01:02.270 INFO:teuthology.orchestra.run.smithi204.stdout:(33/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-16T05:01:02.295 INFO:teuthology.orchestra.run.smithi204.stdout:(34/34): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-09-16T05:01:02.301 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:02.301 INFO:teuthology.orchestra.run.smithi204.stdout:Total 7.7 MB/s | 27 MB 00:03 2024-09-16T05:01:02.316 INFO:teuthology.orchestra.run.smithi196.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-16T05:01:02.341 INFO:teuthology.orchestra.run.smithi196.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 462 kB/s | 35 kB 00:00 2024-09-16T05:01:02.366 INFO:teuthology.orchestra.run.smithi196.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-16T05:01:02.392 INFO:teuthology.orchestra.run.smithi196.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T05:01:02.417 INFO:teuthology.orchestra.run.smithi196.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 827 kB/s | 42 kB 00:00 2024-09-16T05:01:02.435 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:02.473 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:02.473 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:02.476 INFO:teuthology.orchestra.run.smithi196.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 520 kB/s | 43 kB 00:00 2024-09-16T05:01:02.521 INFO:teuthology.orchestra.run.smithi196.stdout:(18/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-16T05:01:02.543 INFO:teuthology.orchestra.run.smithi196.stdout:(19/34): python3-cherrypy-18.6.1-2.el9.noarch.r 5.4 MB/s | 358 kB 00:00 2024-09-16T05:01:02.568 INFO:teuthology.orchestra.run.smithi196.stdout:(20/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-16T05:01:02.594 INFO:teuthology.orchestra.run.smithi196.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-16T05:01:02.619 INFO:teuthology.orchestra.run.smithi196.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-16T05:01:02.645 INFO:teuthology.orchestra.run.smithi196.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-16T05:01:02.670 INFO:teuthology.orchestra.run.smithi196.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T05:01:02.697 INFO:teuthology.orchestra.run.smithi196.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-16T05:01:02.720 INFO:teuthology.orchestra.run.smithi196.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T05:01:02.754 INFO:teuthology.orchestra.run.smithi196.stdout:(27/34): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-09-16T05:01:02.771 INFO:teuthology.orchestra.run.smithi196.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T05:01:02.796 INFO:teuthology.orchestra.run.smithi196.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T05:01:02.813 INFO:teuthology.orchestra.run.smithi196.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 854 kB/s | 36 kB 00:00 2024-09-16T05:01:02.838 INFO:teuthology.orchestra.run.smithi196.stdout:(31/34): python3-webob-1.8.8-2.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-09-16T05:01:02.855 INFO:teuthology.orchestra.run.smithi196.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-16T05:01:02.884 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:02.884 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:02.889 INFO:teuthology.orchestra.run.smithi196.stdout:(33/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-16T05:01:03.039 INFO:teuthology.orchestra.run.smithi196.stdout:(34/34): ceph-osd-17.2.7-1633.g6d025bc2.el9.x86 11 MB/s | 15 MB 00:01 2024-09-16T05:01:03.044 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:03.044 INFO:teuthology.orchestra.run.smithi196.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-16T05:01:03.171 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:03.208 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:03.208 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:03.420 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:03.505 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T05:01:03.555 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T05:01:03.587 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T05:01:03.642 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:03.643 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:03.684 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T05:01:03.768 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T05:01:03.816 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T05:01:03.858 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T05:01:03.889 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T05:01:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T05:01:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T05:01:04.005 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T05:01:04.065 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T05:01:04.100 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T05:01:04.122 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:04.152 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T05:01:04.181 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T05:01:04.188 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T05:01:04.214 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T05:01:04.255 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T05:01:04.264 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T05:01:04.357 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T05:01:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T05:01:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T05:01:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T05:01:04.532 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T05:01:04.551 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T05:01:04.586 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T05:01:04.615 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T05:01:04.619 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T05:01:04.660 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T05:01:04.667 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T05:01:04.695 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T05:01:04.712 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T05:01:04.729 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T05:01:04.770 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T05:01:04.786 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T05:01:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T05:01:04.834 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T05:01:04.881 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T05:01:04.954 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T05:01:04.986 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T05:01:05.019 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T05:01:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T05:01:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T05:01:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T05:01:05.110 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T05:01:05.135 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T05:01:05.181 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T05:01:05.181 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T05:01:05.209 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T05:01:05.231 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T05:01:05.252 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T05:01:05.274 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T05:01:05.277 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T05:01:05.309 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T05:01:05.312 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T05:01:05.332 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T05:01:05.355 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T05:01:05.364 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T05:01:05.396 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T05:01:05.437 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T05:01:05.452 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T05:01:05.472 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T05:01:05.507 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T05:01:05.553 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T05:01:05.554 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T05:01:05.587 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T05:01:05.610 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9. 29/34 2024-09-16T05:01:05.614 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T05:01:05.633 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 30/34 2024-09-16T05:01:05.660 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T05:01:05.661 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 30/34 2024-09-16T05:01:05.662 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:05.662 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T05:01:05.662 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T05:01:05.662 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T05:01:05.662 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:05.693 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T05:01:05.791 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T05:01:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T05:01:05.841 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T05:01:05.879 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T05:01:05.940 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T05:01:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T05:01:05.978 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T05:01:06.027 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T05:01:06.083 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T05:01:06.118 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9. 29/34 2024-09-16T05:01:06.143 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 30/34 2024-09-16T05:01:06.170 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 30/34 2024-09-16T05:01:06.170 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:06.170 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T05:01:06.170 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T05:01:06.170 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T05:01:06.171 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:06.270 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T05:01:06.319 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T05:01:06.388 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T05:01:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T05:01:06.537 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T05:01:06.658 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T05:01:06.830 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9. 29/34 2024-09-16T05:01:06.855 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 30/34 2024-09-16T05:01:06.881 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 30/34 2024-09-16T05:01:06.881 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:06.881 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T05:01:06.881 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T05:01:06.881 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T05:01:06.881 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:07.051 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 31/34 2024-09-16T05:01:07.080 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 31/34 2024-09-16T05:01:07.080 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:07.080 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T05:01:07.080 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T05:01:07.080 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T05:01:07.080 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:07.477 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 32/34 2024-09-16T05:01:07.486 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 31/34 2024-09-16T05:01:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 32/34 2024-09-16T05:01:07.502 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:07.502 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T05:01:07.502 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T05:01:07.502 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T05:01:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:07.513 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 31/34 2024-09-16T05:01:07.513 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:07.513 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T05:01:07.514 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T05:01:07.514 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T05:01:07.514 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:07.719 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 33/34 2024-09-16T05:01:07.744 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 33/34 2024-09-16T05:01:07.744 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:07.744 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T05:01:07.744 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T05:01:07.744 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T05:01:07.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:07.818 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 34/34 2024-09-16T05:01:07.903 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 32/34 2024-09-16T05:01:07.927 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 32/34 2024-09-16T05:01:07.928 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:07.928 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T05:01:07.928 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T05:01:07.928 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T05:01:07.928 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:08.137 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 33/34 2024-09-16T05:01:08.161 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 33/34 2024-09-16T05:01:08.161 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:08.161 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T05:01:08.161 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T05:01:08.161 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T05:01:08.162 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:08.189 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 31/34 2024-09-16T05:01:08.216 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 31/34 2024-09-16T05:01:08.216 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:08.216 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T05:01:08.216 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T05:01:08.216 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T05:01:08.216 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:08.225 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 34/34 2024-09-16T05:01:08.590 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 32/34 2024-09-16T05:01:08.613 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 32/34 2024-09-16T05:01:08.614 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:08.614 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T05:01:08.614 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T05:01:08.614 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T05:01:08.614 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:08.816 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 33/34 2024-09-16T05:01:08.839 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 33/34 2024-09-16T05:01:08.839 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:08.839 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T05:01:08.839 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T05:01:08.839 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T05:01:08.840 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:08.903 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 34/34 2024-09-16T05:01:09.598 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 34/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 5/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9. 6/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T05:01:09.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T05:01:09.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T05:01:09.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T05:01:09.967 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 34/34 2024-09-16T05:01:09.967 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 5/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9. 6/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T05:01:09.968 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T05:01:09.969 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T05:01:09.970 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T05:01:10.472 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T05:01:10.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:10.472 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:10.472 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9.noarch 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T05:01:10.473 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T05:01:10.474 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T05:01:10.475 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T05:01:10.475 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T05:01:10.475 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T05:01:10.475 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T05:01:10.475 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T05:01:10.475 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T05:01:10.475 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:10.475 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 34/34 2024-09-16T05:01:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/34 2024-09-16T05:01:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/34 2024-09-16T05:01:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 3/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 4/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 5/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9. 6/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T05:01:10.538 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T05:01:10.539 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T05:01:10.539 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T05:01:10.539 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T05:01:10.539 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T05:01:10.539 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T05:01:10.539 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T05:01:10.539 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T05:01:10.539 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T05:01:10.541 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T05:01:10.541 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T05:01:10.541 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T05:01:10.541 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T05:01:10.542 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T05:01:10.542 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T05:01:10.542 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T05:01:10.542 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T05:01:10.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base 2024-09-16T05:01:10.951 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T05:01:10.951 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:10.951 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:10.951 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.951 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.951 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9.noarch 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T05:01:10.952 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T05:01:10.953 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T05:01:10.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T05:01:10.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T05:01:10.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T05:01:10.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T05:01:10.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T05:01:10.954 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:10.954 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:11.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base 2024-09-16T05:01:11.383 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:17.2.7-1633.g6d025bc2.el9.noarch 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T05:01:11.384 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T05:01:11.385 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:11.386 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:11.442 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:11.536 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:11.580 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:11.581 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:11.581 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:11.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install cephadm 2024-09-16T05:01:11.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-09-16T05:01:11.865 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:11.954 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:11.996 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:11.997 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:11.997 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:12.055 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install cephadm 2024-09-16T05:01:12.252 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:12.311 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:50 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:12.390 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:12.391 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:12.391 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-16T05:01:12.391 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:12.391 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:01:12.391 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:17.2.7-1633.g6d025bc2.el9 ceph-noarch 83 k 2024-09-16T05:01:12.391 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:12.391 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:01:12.391 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:12.392 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-16T05:01:12.392 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:12.392 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 83 k 2024-09-16T05:01:12.392 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 375 k 2024-09-16T05:01:12.392 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:01:12.402 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:12.443 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:12.445 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:12.445 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:12.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-09-16T05:01:12.584 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm-17.2.7-1633.g6d025bc2.el9.noarch.rpm 431 kB/s | 83 kB 00:00 2024-09-16T05:01:12.584 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:12.585 INFO:teuthology.orchestra.run.smithi032.stdout:Total 427 kB/s | 83 kB 00:00 2024-09-16T05:01:12.585 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:12.591 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:12.591 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:12.592 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:12.593 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:12.619 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:12.668 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:12.751 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:17.2.7-1633.g6d025bc2.el9 ceph-noarch 83 k 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:12.752 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 83 k 2024-09-16T05:01:12.753 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 375 k 2024-09-16T05:01:12.753 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:01:12.939 INFO:teuthology.orchestra.run.smithi204.stdout:cephadm-17.2.7-1633.g6d025bc2.el9.noarch.rpm 445 kB/s | 83 kB 00:00 2024-09-16T05:01:12.939 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:12.940 INFO:teuthology.orchestra.run.smithi204.stdout:Total 440 kB/s | 83 kB 00:00 2024-09-16T05:01:12.940 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:12.945 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:12.945 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:12.947 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:12.947 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:13.082 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:51 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:13.128 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:13.213 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:13.214 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:13.214 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-16T05:01:13.214 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:13.214 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:17.2.7-1633.g6d025bc2.el9 ceph-noarch 83 k 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 83 k 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 375 k 2024-09-16T05:01:13.215 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:13.249 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:13.385 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-17.2.7-1633.g6d025bc2.el9.noarch.rpm 497 kB/s | 83 kB 00:00 2024-09-16T05:01:13.385 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:13.385 INFO:teuthology.orchestra.run.smithi196.stdout:Total 493 kB/s | 83 kB 00:00 2024-09-16T05:01:13.385 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:13.388 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:13.388 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:13.389 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:13.390 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:13.464 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:13.478 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:13.568 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:13.589 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:13.854 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:13.854 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:13.854 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:13.854 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 2024-09-16T05:01:13.855 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:13.855 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:13.913 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:13.941 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:14.020 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:14.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T05:01:14.192 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:14.192 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:14.192 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:14.192 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 2024-09-16T05:01:14.193 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:14.193 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:14.376 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:14.383 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T05:01:14.653 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:14.660 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 1/1 2024-09-16T05:01:14.660 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:14.660 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:14.660 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:17.2.7-1633.g6d025bc2.el9.noarch 2024-09-16T05:01:14.661 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:14.661 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:14.789 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:14.791 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:14.791 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-09-16T05:01:14.791 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:14.791 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:01:14.791 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 137 k 2024-09-16T05:01:14.791 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:14.792 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:01:14.792 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:14.792 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-16T05:01:14.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:14.792 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 137 k 2024-09-16T05:01:14.793 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 379 k 2024-09-16T05:01:14.793 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:01:14.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T05:01:14.964 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:15.001 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-17.2.7-1633.g6d025b 655 kB/s | 137 kB 00:00 2024-09-16T05:01:15.001 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:15.002 INFO:teuthology.orchestra.run.smithi032.stdout:Total 650 kB/s | 137 kB 00:00 2024-09-16T05:01:15.002 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:15.010 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:15.010 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:15.070 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:15.070 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:15.099 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 137 k 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T05:01:15.101 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:15.102 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 137 k 2024-09-16T05:01:15.102 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 379 k 2024-09-16T05:01:15.102 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:01:15.229 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:15.303 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-17.2.7-1633.g6d025b 681 kB/s | 137 kB 00:00 2024-09-16T05:01:15.304 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:15.304 INFO:teuthology.orchestra.run.smithi204.stdout:Total 674 kB/s | 137 kB 00:00 2024-09-16T05:01:15.304 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:15.310 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:15.312 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:15.312 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:15.337 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:15.337 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:15.337 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T05:01:15.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:15.368 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:15.368 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:15.412 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:15.524 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:15.545 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:15.546 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-16T05:01:15.546 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:15.546 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 137 k 2024-09-16T05:01:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:15.546 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:15.547 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:15.547 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-16T05:01:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:15.547 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 137 k 2024-09-16T05:01:15.547 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 379 k 2024-09-16T05:01:15.547 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:15.621 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:15.645 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:15.646 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:15.646 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T05:01:15.646 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:15.763 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-17.2.7-1633.g6d025b 637 kB/s | 137 kB 00:00 2024-09-16T05:01:15.763 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:15.764 INFO:teuthology.orchestra.run.smithi196.stdout:Total 631 kB/s | 137 kB 00:00 2024-09-16T05:01:15.764 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:15.772 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:15.772 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:15.830 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:15.830 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:15.968 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:16.081 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:16.081 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T05:01:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:16.224 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:16.224 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:16.224 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:16.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr 2024-09-16T05:01:16.618 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:16.618 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:16.618 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:16.618 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:16.618 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:16.618 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:16.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr 2024-09-16T05:01:17.020 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2. 1/1 2024-09-16T05:01:17.021 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:17.021 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:17.021 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:17.021 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:17.021 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:17.027 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:17.122 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:17.166 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:17.168 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:17.168 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:17.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse 2024-09-16T05:01:17.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-09-16T05:01:17.400 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:17.492 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:17.534 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:17.535 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:17.535 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:17.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse 2024-09-16T05:01:17.825 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:17.832 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:17.928 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:17.966 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:17.966 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-09-16T05:01:17.966 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:17.966 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:01:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 814 k 2024-09-16T05:01:17.966 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-09-16T05:01:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T05:01:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:17.967 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:01:17.967 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:17.967 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-09-16T05:01:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:17.967 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 893 k 2024-09-16T05:01:17.968 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 2.6 M 2024-09-16T05:01:17.968 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:01:17.969 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:17.970 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:17.971 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:18.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-09-16T05:01:18.160 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:18.188 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 795 kB/s | 80 kB 00:00 2024-09-16T05:01:18.294 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:18.294 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:18.294 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 814 k 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:18.295 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-09-16T05:01:18.296 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:18.296 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 893 k 2024-09-16T05:01:18.296 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.6 M 2024-09-16T05:01:18.296 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:01:18.388 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): ceph-fuse-17.2.7-1633.g6d025bc2.el9.x86_ 2.6 MB/s | 814 kB 00:00 2024-09-16T05:01:18.389 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:18.390 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.1 MB/s | 893 kB 00:00 2024-09-16T05:01:18.401 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:18.424 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:18.424 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:18.612 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:18.700 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): ceph-fuse-17.2.7-1633.g6d025bc2.el9.x86_ 2.8 MB/s | 814 kB 00:00 2024-09-16T05:01:18.703 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:18.743 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:18.744 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:18.744 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-16T05:01:18.744 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:18.744 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:18.744 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 814 k 2024-09-16T05:01:18.744 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-16T05:01:18.744 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T05:01:18.745 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:18.745 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:18.745 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:18.745 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-09-16T05:01:18.745 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:18.745 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 893 k 2024-09-16T05:01:18.745 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.6 M 2024-09-16T05:01:18.746 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T05:01:18.866 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 176 kB/s | 80 kB 00:00 2024-09-16T05:01:18.867 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:18.867 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-09-16T05:01:18.879 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:18.904 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:18.904 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:18.930 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/2 2024-09-16T05:01:18.975 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:18.975 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:19.051 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 311 kB/s | 80 kB 00:00 2024-09-16T05:01:19.076 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-17.2.7-1633.g6d025bc2.el9.x86_ 2.8 MB/s | 814 kB 00:00 2024-09-16T05:01:19.076 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:19.077 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.6 MB/s | 893 kB 00:00 2024-09-16T05:01:19.088 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:19.110 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:19.110 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:19.171 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:19.182 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:19.183 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:19.323 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T05:01:19.383 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:19.397 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/2 2024-09-16T05:01:19.528 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T05:01:19.597 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/2 2024-09-16T05:01:19.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/2 2024-09-16T05:01:19.598 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/2 2024-09-16T05:01:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T05:01:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:19.849 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T05:01:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:19.850 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:20.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel 2024-09-16T05:01:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/2 2024-09-16T05:01:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/2 2024-09-16T05:01:20.272 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2/2 2024-09-16T05:01:20.272 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/2 2024-09-16T05:01:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T05:01:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:20.393 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T05:01:20.393 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:20.394 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:20.535 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T05:01:20.536 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:20.536 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:20.536 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T05:01:20.536 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:20.536 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:20.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel 2024-09-16T05:01:20.677 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:03 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:20.816 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:20.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-09-16T05:01:20.817 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-09-16T05:01:20.817 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:20.817 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:01:20.817 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 126 k 2024-09-16T05:01:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:20.818 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:01:20.818 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:20.818 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-16T05:01:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:20.818 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 126 k 2024-09-16T05:01:20.818 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 448 k 2024-09-16T05:01:20.818 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:01:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-17.2.7-1633.g6d025bc2.el9.x86_64 581 kB/s | 126 kB 00:00 2024-09-16T05:01:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:21.035 INFO:teuthology.orchestra.run.smithi032.stdout:Total 577 kB/s | 126 kB 00:00 2024-09-16T05:01:21.036 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:21.042 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:21.042 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:21.103 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:21.103 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:21.211 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:04 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:21.345 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 126 k 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 126 k 2024-09-16T05:01:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 448 k 2024-09-16T05:01:21.347 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:01:21.358 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:21.385 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:21.520 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:21.521 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:21.521 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-16T05:01:21.521 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:21.521 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 126 k 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 126 k 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 448 k 2024-09-16T05:01:21.522 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:21.538 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-17.2.7-1633.g6d025bc2.el9.x86_64 659 kB/s | 126 kB 00:00 2024-09-16T05:01:21.539 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:21.539 INFO:teuthology.orchestra.run.smithi204.stdout:Total 654 kB/s | 126 kB 00:00 2024-09-16T05:01:21.539 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:21.544 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:21.545 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:21.601 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:21.601 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:21.706 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-17.2.7-1633.g6d025bc2.el9.x86_64 687 kB/s | 126 kB 00:00 2024-09-16T05:01:21.707 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:21.707 INFO:teuthology.orchestra.run.smithi196.stdout:Total 682 kB/s | 126 kB 00:00 2024-09-16T05:01:21.707 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:21.712 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:21.712 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:21.769 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:21.769 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:21.824 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:21.851 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:22.002 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:22.055 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:22.055 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:22.055 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:22.055 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:22.056 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:22.056 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:22.250 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:22.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2 2024-09-16T05:01:22.400 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:22.491 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:22.491 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:22.659 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:22.660 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:22.660 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:22.660 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:22.660 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:22.660 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:22.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2 2024-09-16T05:01:22.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-09-16T05:01:22.893 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:22.986 INFO:teuthology.orchestra.run.smithi032.stdout:Package libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:23.030 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:23.031 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:23.031 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:23.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs-devel 2024-09-16T05:01:23.302 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:23.394 INFO:teuthology.orchestra.run.smithi204.stdout:Package libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:23.436 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:23.437 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:23.437 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:23.446 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:23.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs-devel 2024-09-16T05:01:23.539 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:23.583 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:23.584 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:23.584 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:23.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-09-16T05:01:23.699 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:23.835 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:23.835 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:23.835 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-09-16T05:01:23.836 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:23.836 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:01:23.836 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 25 k 2024-09-16T05:01:23.836 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:23.836 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:01:23.836 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:23.836 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-16T05:01:23.836 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:23.837 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 25 k 2024-09-16T05:01:23.837 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 110 k 2024-09-16T05:01:23.837 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:01:23.987 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs-devel-17.2.7-1633.g6d025bc2.el9.x86_6 165 kB/s | 25 kB 00:00 2024-09-16T05:01:23.988 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:23.988 INFO:teuthology.orchestra.run.smithi032.stdout:Total 164 kB/s | 25 kB 00:00 2024-09-16T05:01:23.988 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:23.992 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:23.992 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:23.994 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:23.994 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:24.057 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:07 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:24.084 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:24.188 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 25 k 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:24.189 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 25 k 2024-09-16T05:01:24.190 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 110 k 2024-09-16T05:01:24.190 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:01:24.194 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:24.217 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:02 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:24.343 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs-devel-17.2.7-1633.g6d025bc2.el9.x86_6 163 kB/s | 25 kB 00:00 2024-09-16T05:01:24.343 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:24.343 INFO:teuthology.orchestra.run.smithi204.stdout:Total 161 kB/s | 25 kB 00:00 2024-09-16T05:01:24.344 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:24.347 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:24.347 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:24.348 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:24.349 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:24.349 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 25 k 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:24.350 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 25 k 2024-09-16T05:01:24.351 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 110 k 2024-09-16T05:01:24.351 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:24.425 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:24.510 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-17.2.7-1633.g6d025bc2.el9.x86_6 157 kB/s | 25 kB 00:00 2024-09-16T05:01:24.511 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:24.511 INFO:teuthology.orchestra.run.smithi196.stdout:Total 155 kB/s | 25 kB 00:00 2024-09-16T05:01:24.511 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:24.514 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:24.514 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:24.516 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:24.516 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:24.537 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:24.542 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:24.602 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:24.712 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:24.763 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:24.764 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:24.764 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:24.764 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:24.764 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:24.764 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:24.894 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:24.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2 2024-09-16T05:01:25.041 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:25.115 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:25.115 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:25.115 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:25.115 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:25.115 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:25.115 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:25.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2 2024-09-16T05:01:25.275 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:25.275 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:25.275 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:25.275 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:25.275 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:25.275 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:25.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-09-16T05:01:25.555 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:08 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:25.656 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:25.701 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:25.702 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:25.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1 2024-09-16T05:01:25.839 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:08 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:26.022 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:04 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:26.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1 2024-09-16T05:01:26.116 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:26.161 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:26.162 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:26.162 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:26.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-09-16T05:01:26.372 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:09 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:26.467 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:26.513 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:26.514 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:26.514 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:26.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados 2024-09-16T05:01:26.611 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:09 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:26.702 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:26.744 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:26.745 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:26.745 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:26.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados 2024-09-16T05:01:26.833 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:04 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:26.925 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:26.969 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:26.970 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:26.970 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:27.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-09-16T05:01:27.162 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:10 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:27.257 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:27.303 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:27.304 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:27.304 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:27.375 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:10 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:27.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw 2024-09-16T05:01:27.466 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:27.508 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:27.509 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:27.509 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:27.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw 2024-09-16T05:01:27.609 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:27.702 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:27.743 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:27.744 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:27.744 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:27.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-09-16T05:01:27.963 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:10 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:28.055 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:28.099 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:28.100 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:28.101 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:28.132 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:11 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:28.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs 2024-09-16T05:01:28.223 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:28.266 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:28.266 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:28.266 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:28.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs 2024-09-16T05:01:28.368 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:28.459 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:28.500 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:28.501 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:28.501 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:28.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-09-16T05:01:28.753 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:11 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:28.844 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:28.888 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:28.889 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:11 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:28.889 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:28.889 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:28.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd 2024-09-16T05:01:28.978 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:29.020 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:29.021 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:29.021 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:29.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd 2024-09-16T05:01:29.137 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:07 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:29.225 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:29.267 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:29.269 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:29.269 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:29.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-09-16T05:01:29.539 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:12 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:29.634 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:29.653 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:12 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:29.679 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:29.680 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-09-16T05:01:29.680 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:29.745 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:29.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse 2024-09-16T05:01:29.786 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:29.787 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T05:01:29.787 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:29.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse 2024-09-16T05:01:29.912 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:07 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:30.003 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 is already installed. 2024-09-16T05:01:30.044 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:30.045 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-16T05:01:30.045 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:30.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-09-16T05:01:30.339 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:13 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:30.423 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:13 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:30.476 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 82 k 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 82 k 2024-09-16T05:01:30.477 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 222 k 2024-09-16T05:01:30.478 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:01:30.557 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 82 k 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T05:01:30.558 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:30.559 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 82 k 2024-09-16T05:01:30.559 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 222 k 2024-09-16T05:01:30.559 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:01:30.692 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 382 kB/s | 82 kB 00:00 2024-09-16T05:01:30.692 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:30.693 INFO:teuthology.orchestra.run.smithi032.stdout:Total 379 kB/s | 82 kB 00:00 2024-09-16T05:01:30.693 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:30.697 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:08 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:30.700 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:30.700 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:30.759 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:30.759 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:30.787 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 357 kB/s | 82 kB 00:00 2024-09-16T05:01:30.787 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:30.788 INFO:teuthology.orchestra.run.smithi204.stdout:Total 355 kB/s | 82 kB 00:00 2024-09-16T05:01:30.788 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:30.795 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:30.795 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:30.830 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 82 k 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:30.831 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 82 k 2024-09-16T05:01:30.832 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 222 k 2024-09-16T05:01:30.832 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:30.852 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:30.853 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:30.903 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:30.992 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:31.005 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 471 kB/s | 82 kB 00:00 2024-09-16T05:01:31.006 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.006 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:31.007 INFO:teuthology.orchestra.run.smithi196.stdout:Total 467 kB/s | 82 kB 00:00 2024-09-16T05:01:31.007 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:31.013 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:31.013 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:31.069 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:31.070 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:31.099 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.202 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:31.300 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.459 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.643 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.643 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:31.643 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:31.643 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:31.691 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.701 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.701 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:31.701 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:31.701 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:31.701 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:31.701 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:31.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror 2024-09-16T05:01:31.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2024-09-16T05:01:31.950 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:31.950 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:31.950 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:31.950 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:31.950 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:31.950 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:32.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-09-16T05:01:32.488 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:15 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:32.489 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:15 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:32.625 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:32.626 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-16T05:01:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:01:32.626 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.9 M 2024-09-16T05:01:32.626 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:01:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:32.627 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T05:01:32.627 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:32.627 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.9 M 2024-09-16T05:01:32.627 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 12 M 2024-09-16T05:01:32.627 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.9 M 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 2.9 M 2024-09-16T05:01:32.629 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 12 M 2024-09-16T05:01:32.629 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:01:32.768 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:10 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:32.906 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 2.9 M 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-16T05:01:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:32.908 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.9 M 2024-09-16T05:01:32.908 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 12 M 2024-09-16T05:01:32.908 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:32.959 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 8.9 MB/s | 2.9 MB 00:00 2024-09-16T05:01:32.959 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:32.959 INFO:teuthology.orchestra.run.smithi204.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-09-16T05:01:32.960 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:32.968 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:32.968 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:32.970 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 8.6 MB/s | 2.9 MB 00:00 2024-09-16T05:01:32.970 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:32.971 INFO:teuthology.orchestra.run.smithi032.stdout:Total 8.6 MB/s | 2.9 MB 00:00 2024-09-16T05:01:32.971 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:32.981 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:32.981 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:33.041 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:33.041 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:33.058 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:33.058 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:33.209 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 9.7 MB/s | 2.9 MB 00:00 2024-09-16T05:01:33.210 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:33.210 INFO:teuthology.orchestra.run.smithi196.stdout:Total 9.7 MB/s | 2.9 MB 00:00 2024-09-16T05:01:33.210 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:33.219 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:33.219 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:33.293 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:33.293 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:33.448 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:33.480 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:33.518 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:33.543 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:33.543 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:33.543 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T05:01:33.543 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T05:01:33.544 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T05:01:33.544 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:33.573 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:33.573 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:33.573 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T05:01:33.574 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T05:01:33.574 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T05:01:33.574 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:33.698 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:33.752 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:33.778 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:33.778 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:01:33.779 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T05:01:33.779 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T05:01:33.779 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T05:01:33.779 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:34.618 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:34.618 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:34.618 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:34.618 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:34.618 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:34.618 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:34.638 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:34.638 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:34.638 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:34.638 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:34.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd 2024-09-16T05:01:34.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:34.866 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:34.866 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:34.866 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:34.866 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:34.866 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:34.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-09-16T05:01:35.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-09-16T05:01:35.417 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:18 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:35.460 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:18 ago on Mon 16 Sep 2024 05:00:17 AM UTC. 2024-09-16T05:01:35.551 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T05:01:35.552 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:35.552 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-16T05:01:35.552 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:35.552 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 165 k 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 165 k 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 477 k 2024-09-16T05:01:35.553 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T05:01:35.596 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-09-16T05:01:35.596 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 165 k 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 165 k 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 477 k 2024-09-16T05:01:35.597 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-09-16T05:01:35.686 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:13 ago on Mon 16 Sep 2024 05:00:22 AM UTC. 2024-09-16T05:01:35.763 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 783 kB/s | 165 kB 00:00 2024-09-16T05:01:35.763 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:35.763 INFO:teuthology.orchestra.run.smithi204.stdout:Total 778 kB/s | 165 kB 00:00 2024-09-16T05:01:35.764 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T05:01:35.771 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T05:01:35.772 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T05:01:35.819 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-16T05:01:35.820 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:35.820 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-16T05:01:35.820 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:35.821 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-16T05:01:35.821 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:17.2.7-1633.g6d025bc2.el9 ceph 165 k 2024-09-16T05:01:35.821 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:35.821 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-16T05:01:35.821 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-16T05:01:35.821 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-16T05:01:35.821 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:35.822 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 165 k 2024-09-16T05:01:35.822 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 477 k 2024-09-16T05:01:35.822 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-16T05:01:35.823 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 732 kB/s | 165 kB 00:00 2024-09-16T05:01:35.823 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:35.824 INFO:teuthology.orchestra.run.smithi032.stdout:Total 727 kB/s | 165 kB 00:00 2024-09-16T05:01:35.824 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-09-16T05:01:35.828 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T05:01:35.829 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T05:01:35.832 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-09-16T05:01:35.832 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-09-16T05:01:35.893 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-09-16T05:01:35.893 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-09-16T05:01:35.986 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T05:01:36.046 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-17.2.7-1633.g6d025bc2.el9.x86_64.rpm 732 kB/s | 165 kB 00:00 2024-09-16T05:01:36.047 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:01:36.047 INFO:teuthology.orchestra.run.smithi196.stdout:Total 727 kB/s | 165 kB 00:00 2024-09-16T05:01:36.047 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-16T05:01:36.055 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-16T05:01:36.055 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-16T05:01:36.055 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-09-16T05:01:36.083 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:36.111 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-16T05:01:36.111 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-16T05:01:36.171 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:36.259 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-16T05:01:36.350 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:36.515 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:36.575 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:36.778 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:36.778 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:36.778 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T05:01:36.779 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:36.779 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:01:36.779 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T05:01:36.788 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:36.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:36.788 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-09-16T05:01:36.788 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:36.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:01:36.789 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-09-16T05:01:36.791 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:37.001 DEBUG:teuthology.parallel:result is None 2024-09-16T05:01:37.030 DEBUG:teuthology.parallel:result is None 2024-09-16T05:01:37.058 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 1/1 2024-09-16T05:01:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:37.059 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-16T05:01:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:17.2.7-1633.g6d025bc2.el9.x86_64 2024-09-16T05:01:37.059 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:01:37.059 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-16T05:01:37.317 DEBUG:teuthology.parallel:result is None 2024-09-16T05:01:37.318 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T05:01:37.318 INFO:teuthology.packaging:ref: None 2024-09-16T05:01:37.318 INFO:teuthology.packaging:tag: None 2024-09-16T05:01:37.318 INFO:teuthology.packaging:branch: quincy 2024-09-16T05:01:37.318 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:01:37.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-16T05:01:37.525 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T05:01:37.552 INFO:teuthology.orchestra.run.smithi032.stdout:17.2.7-1633.g6d025bc2.el9 2024-09-16T05:01:37.553 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1633.g6d025bc2.el9 2024-09-16T05:01:37.553 INFO:teuthology.task.install:The correct ceph version 17.2.7-1633.g6d025bc2 is installed. 2024-09-16T05:01:37.554 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T05:01:37.554 INFO:teuthology.packaging:ref: None 2024-09-16T05:01:37.555 INFO:teuthology.packaging:tag: None 2024-09-16T05:01:37.555 INFO:teuthology.packaging:branch: quincy 2024-09-16T05:01:37.555 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:01:37.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-16T05:01:37.757 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T05:01:37.782 INFO:teuthology.orchestra.run.smithi196.stdout:17.2.7-1633.g6d025bc2.el9 2024-09-16T05:01:37.782 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1633.g6d025bc2.el9 2024-09-16T05:01:37.783 INFO:teuthology.task.install:The correct ceph version 17.2.7-1633.g6d025bc2 is installed. 2024-09-16T05:01:37.785 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T05:01:37.785 INFO:teuthology.packaging:ref: None 2024-09-16T05:01:37.785 INFO:teuthology.packaging:tag: None 2024-09-16T05:01:37.785 INFO:teuthology.packaging:branch: quincy 2024-09-16T05:01:37.785 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:01:37.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-16T05:01:37.976 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T05:01:38.001 INFO:teuthology.orchestra.run.smithi204.stdout:17.2.7-1633.g6d025bc2.el9 2024-09-16T05:01:38.002 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1633.g6d025bc2.el9 2024-09-16T05:01:38.002 INFO:teuthology.task.install:The correct ceph version 17.2.7-1633.g6d025bc2 is installed. 2024-09-16T05:01:38.004 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T05:01:38.004 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:01:38.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T05:01:38.036 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:01:38.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T05:01:38.067 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:01:38.068 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T05:01:38.098 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T05:01:38.099 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:01:38.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T05:01:38.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T05:01:38.201 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:01:38.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T05:01:38.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T05:01:38.301 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:01:38.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T05:01:38.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T05:01:38.401 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T05:01:38.401 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:01:38.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T05:01:38.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T05:01:38.504 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:01:38.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T05:01:38.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T05:01:38.603 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:01:38.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T05:01:38.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T05:01:38.704 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T05:01:38.704 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:01:38.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T05:01:38.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T05:01:38.807 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:01:38.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T05:01:38.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T05:01:38.921 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:01:38.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T05:01:38.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T05:01:39.020 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-16T05:01:39.123 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T05:01:39.123 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:01:39.124 INFO:tasks.cephadm:Cluster fsid is c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:01:39.124 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-16T05:01:39.124 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.32', 'mon.b': '172.21.15.196', 'mon.c': '172.21.15.204'} 2024-09-16T05:01:39.124 INFO:tasks.cephadm:First mon is mon.a on smithi032 2024-09-16T05:01:39.124 INFO:tasks.cephadm:First mgr is y 2024-09-16T05:01:39.124 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-16T05:01:39.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-09-16T05:01:39.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-09-16T05:01:39.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-09-16T05:01:39.216 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-16T05:01:39.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:01:39.359 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56081/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-326-g8921c8ef', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-326.g8921c8ef'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-15 16:06:42.385367', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-16T05:01:39.498 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 8921c8ef947807b717db041d85f0761b7914d5fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:01:39.499 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-16T05:01:39.500 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-16T05:01:39.500 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:39.886 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 05:01 /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:39.887 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:40.285 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 05:01 /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:40.285 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:40.672 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 05:01 /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:40.673 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:40.693 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:40.713 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T05:01:40.748 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc on all hosts... 2024-09-16T05:01:40.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T05:01:40.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T05:01:40.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T05:01:42.287 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T05:01:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T05:01:42.482 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T05:02:18.374 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:02:18.375 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T05:02:18.375 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T05:02:18.375 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-09-16T05:02:18.375 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T05:02:18.375 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:02:18.375 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:02:26.415 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:02:26.415 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T05:02:26.415 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T05:02:26.415 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-09-16T05:02:26.415 INFO:teuthology.orchestra.run.smithi196.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T05:02:26.415 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:02:26.416 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:02:26.689 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:02:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T05:02:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T05:02:26.689 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-09-16T05:02:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T05:02:26.690 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:02:26.690 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:02:26.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-09-16T05:02:26.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-09-16T05:02:26.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-09-16T05:02:26.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-09-16T05:02:26.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-09-16T05:02:26.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-09-16T05:02:26.911 INFO:tasks.cephadm:Writing seed config... 2024-09-16T05:02:26.912 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-09-16T05:02:26.912 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-16T05:02:26.912 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-16T05:02:26.912 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-16T05:02:26.912 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-09-16T05:02:26.912 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-09-16T05:02:26.912 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [global] ms inject delay max = 1 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [global] ms inject delay probability = 0.005 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [global] ms inject delay type = osd 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [global] ms inject internal delays = 0.002 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 2 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [osd] filestore queue throttle high multiple = 2 2024-09-16T05:02:26.913 INFO:tasks.cephadm: override: [osd] filestore queue throttle max multiple = 10 2024-09-16T05:02:26.914 INFO:tasks.cephadm: override: [osd] journal throttle high multiple = 2 2024-09-16T05:02:26.914 INFO:tasks.cephadm: override: [osd] journal throttle max multiple = 10 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd max backfills = 9 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-09-16T05:02:26.916 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-09-16T05:02:26.917 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-16T05:02:26.917 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-16T05:02:26.917 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-16T05:02:26.917 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:02:26.917 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-16T05:02:26.934 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c1b4ea6a-73e8-11ef-bceb-c7b262605968 mon client directed command retry = 5 mon election default strategy = 1 ms inject delay max = 1 ms inject delay probability = 0.005 ms inject delay type = osd ms inject internal delays = 0.002 ms inject socket failures = 2500 osd_pool_default_min_size = 2 osd_pool_default_size = 2 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 filestore queue throttle high multiple = 2 filestore queue throttle max multiple = 10 journal throttle high multiple = 2 journal throttle max multiple = 10 osd blocked scrub grace period = 3600 osd max backfills = 9 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon scrub interval = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-16T05:02:26.934 DEBUG:teuthology.orchestra.run.smithi032:mon.a> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.a.service 2024-09-16T05:02:26.977 DEBUG:teuthology.orchestra.run.smithi032:mgr.y> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.y.service 2024-09-16T05:02:27.020 INFO:tasks.cephadm:Bootstrapping... 2024-09-16T05:02:27.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc -v bootstrap --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.32 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:02:27.229 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:02:27.229 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc', '-v', 'bootstrap', '--fsid', 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.32', '--skip-admin-label'] 2024-09-16T05:02:27.230 INFO:teuthology.orchestra.run.smithi032.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-16T05:02:27.230 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying podman|docker is present... 2024-09-16T05:02:27.254 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 5.2.2 2024-09-16T05:02:27.254 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying lvm2 is present... 2024-09-16T05:02:27.254 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying time synchronization is in place... 2024-09-16T05:02:27.262 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T05:02:27.262 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T05:02:27.268 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T05:02:27.268 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-09-16T05:02:27.275 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-09-16T05:02:27.282 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-09-16T05:02:27.282 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-09-16T05:02:27.282 INFO:teuthology.orchestra.run.smithi032.stdout:Repeating the final host check... 2024-09-16T05:02:27.309 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 5.2.2 2024-09-16T05:02:27.309 INFO:teuthology.orchestra.run.smithi032.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-16T05:02:27.309 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl is present 2024-09-16T05:02:27.309 INFO:teuthology.orchestra.run.smithi032.stdout:lvcreate is present 2024-09-16T05:02:27.316 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T05:02:27.316 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T05:02:27.322 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T05:02:27.322 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-09-16T05:02:27.329 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-09-16T05:02:27.336 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-09-16T05:02:27.336 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-09-16T05:02:27.336 INFO:teuthology.orchestra.run.smithi032.stdout:Host looks OK 2024-09-16T05:02:27.336 INFO:teuthology.orchestra.run.smithi032.stdout:Cluster fsid: c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:27.336 INFO:teuthology.orchestra.run.smithi032.stdout:Acquiring lock 139825993076448 on /run/cephadm/c1b4ea6a-73e8-11ef-bceb-c7b262605968.lock 2024-09-16T05:02:27.336 INFO:teuthology.orchestra.run.smithi032.stdout:Lock 139825993076448 acquired on /run/cephadm/c1b4ea6a-73e8-11ef-bceb-c7b262605968.lock 2024-09-16T05:02:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 3300 ... 2024-09-16T05:02:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 6789 ... 2024-09-16T05:02:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:Base mon IP(s) is [172.21.15.32:3300, 172.21.15.32:6789], mon addrv is [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-09-16T05:02:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.32 metric 100 2024-09-16T05:02:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.32 metric 100 2024-09-16T05:02:27.343 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-16T05:02:27.343 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-16T05:02:27.343 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-09-16T05:02:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-16T05:02:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-16T05:02:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T05:02:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-16T05:02:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2602/64 scope link 2024-09-16T05:02:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T05:02:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-09-16T05:02:27.347 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-09-16T05:02:27.347 INFO:teuthology.orchestra.run.smithi032.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-16T05:02:27.347 INFO:teuthology.orchestra.run.smithi032.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-16T05:02:27.347 INFO:teuthology.orchestra.run.smithi032.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T05:02:27.810 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-16T05:02:27.810 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T05:02:27.810 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Getting image source signatures 2024-09-16T05:02:27.810 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:c0574da0c3174f3dc49daa274908f97d06456bd6a3cfab87e94022c1369a83a6 2024-09-16T05:02:27.810 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-16T05:02:27.810 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying config sha256:f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-16T05:02:27.810 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-16T05:02:28.689 INFO:teuthology.orchestra.run.smithi032.stdout:ceph: stdout ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T05:02:28.689 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph version: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T05:02:28.689 INFO:teuthology.orchestra.run.smithi032.stdout:Extracting ceph user uid/gid from container image... 2024-09-16T05:02:29.693 INFO:teuthology.orchestra.run.smithi032.stdout:stat: stdout 167 167 2024-09-16T05:02:29.693 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial keys... 2024-09-16T05:02:30.526 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQDmu+dmznfAAhAA7dLkp3x4pyW+0cCGIHtvVQ== 2024-09-16T05:02:31.414 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQDmu+dmJx5MMxAABo9evhOxQpsaYKljWltcNA== 2024-09-16T05:02:32.270 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQDnu+dmSFqfKhAAWCW9lMbmsAtaROF3WBTFRQ== 2024-09-16T05:02:32.271 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial monmap... 2024-09-16T05:02:33.151 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool for a [v2:172.21.15.32:3300,v1:172.21.15.32:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: set fsid to c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:02:33.152 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mon... 2024-09-16T05:02:34.050 INFO:teuthology.orchestra.run.smithi032.stdout:create mon.a on 2024-09-16T05:02:34.264 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-16T05:02:34.408 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-16T05:02:34.601 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968.target. 2024-09-16T05:02:34.601 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968.target. 2024-09-16T05:02:34.908 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.a 2024-09-16T05:02:34.908 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-16T05:02:35.070 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968.target.wants/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@.service. 2024-09-16T05:02:35.618 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:35 smithi032 ceph-mon[27018]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T05:02:35.618 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:35 smithi032 bash[26995]: c718b1e4efa245357d4a8ddcf9ecf67d5745cc7c8a4436cf76476e0b45cd2122 2024-09-16T05:02:35.633 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T05:02:35.634 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-09-16T05:02:35.641 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T05:02:35.641 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-09-16T05:02:35.642 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-09-16T05:02:35.642 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T05:02:35.642 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon to start... 2024-09-16T05:02:35.642 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon... 2024-09-16T05:02:35.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:35 smithi032 systemd[1]: Started Ceph mon.a for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: monmap epoch 1 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: last_changed 2024-09-16T05:02:32.609764+0000 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: min_mon_release 19 (squid) 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: election_strategy: 1 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: fsmap 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: osdmap e1: 0 total, 0 up, 0 in 2024-09-16T05:02:36.767 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: mgrmap e1: no daemons active 2024-09-16T05:02:36.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:36 smithi032 ceph-mon[27018]: from='client.? 172.21.15.32:0/2783959104' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-16T05:02:36.770 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout cluster: 2024-09-16T05:02:36.770 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout id: c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:36.770 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout services: 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.546417s) 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout data: 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pgs: 2024-09-16T05:02:36.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:36.772 INFO:teuthology.orchestra.run.smithi032.stdout:mon is available 2024-09-16T05:02:36.772 INFO:teuthology.orchestra.run.smithi032.stdout:Assimilating anything we can from ceph.conf... 2024-09-16T05:02:37.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:37.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T05:02:37.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:37.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-09-16T05:02:37.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T05:02:37.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T05:02:37.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T05:02:37.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_high_multiple = 2 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_max_multiple = 10 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T05:02:37.777 INFO:teuthology.orchestra.run.smithi032.stdout:Generating new minimal ceph.conf... 2024-09-16T05:02:37.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:37 smithi032 ceph-mon[27018]: from='client.? 172.21.15.32:0/400299012' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-16T05:02:37.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:37 smithi032 ceph-mon[27018]: from='client.? 172.21.15.32:0/400299012' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-16T05:02:38.773 INFO:teuthology.orchestra.run.smithi032.stdout:Restarting the monitor... 2024-09-16T05:02:38.788 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:38 smithi032 ceph-mon[27018]: from='client.? 172.21.15.32:0/1391339639' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:02:39.049 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:38 smithi032 systemd[1]: Stopping Ceph mon.a for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:02:39.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:39 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27009]: 2024-09-16T05:02:39.047+0000 7f6a50225640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:02:39.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:39 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27009]: 2024-09-16T05:02:39.047+0000 7f6a50225640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-16T05:02:39.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:39 smithi032 podman[27267]: 2024-09-16 05:02:39.273094673 +0000 UTC m=+0.337014803 container died c718b1e4efa245357d4a8ddcf9ecf67d5745cc7c8a4436cf76476e0b45cd2122 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-16T05:02:39.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:39 smithi032 podman[27267]: 2024-09-16 05:02:39.64687779 +0000 UTC m=+0.710797909 container cleanup c718b1e4efa245357d4a8ddcf9ecf67d5745cc7c8a4436cf76476e0b45cd2122 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:02:39.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:39 smithi032 bash[27267]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a 2024-09-16T05:02:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:39 smithi032 podman[27281]: 2024-09-16 05:02:39.905968286 +0000 UTC m=+0.622755322 container remove c718b1e4efa245357d4a8ddcf9ecf67d5745cc7c8a4436cf76476e0b45cd2122 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:02:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 systemd[1]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-16T05:02:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 systemd[1]: Stopped Ceph mon.a for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:02:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 systemd[1]: Starting Ceph mon.a for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:02:40.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 podman[27369]: 2024-09-16 05:02:40.146520571 +0000 UTC m=+0.016409445 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:02:40.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 podman[27369]: 2024-09-16 05:02:40.265191235 +0000 UTC m=+0.135080096 container create d9caed178edf300992f3eec70594794e53439b7571d4cc9a28ad07a27b8c35a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-16T05:02:40.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 podman[27369]: 2024-09-16 05:02:40.517590713 +0000 UTC m=+0.387479572 container init d9caed178edf300992f3eec70594794e53439b7571d4cc9a28ad07a27b8c35a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-16T05:02:40.614 INFO:teuthology.orchestra.run.smithi032.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-16T05:02:40.843 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 podman[27369]: 2024-09-16 05:02:40.520788419 +0000 UTC m=+0.390677278 container start d9caed178edf300992f3eec70594794e53439b7571d4cc9a28ad07a27b8c35a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-16T05:02:40.843 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T05:02:40.843 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: pidfile_write: ignore empty --pid-file 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: load: jerasure load: lrc 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: RocksDB version: 7.9.2 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Git sha 0 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: DB SUMMARY 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: DB Session ID: TG9JZNXNWF00S6G85AB6 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: CURRENT file: CURRENT 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97771 ; 2024-09-16T05:02:40.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.error_if_exists: 0 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.create_if_missing: 0 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.paranoid_checks: 1 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.env: 0x55873c2c7940 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.info_log: 0x55873db2f6e0 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.statistics: (nil) 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.use_fsync: 0 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_log_file_size: 0 2024-09-16T05:02:40.845 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T05:02:40.846 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T05:02:40.846 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T05:02:40.846 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T05:02:40.846 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.allow_fallocate: 1 2024-09-16T05:02:40.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T05:02:40.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T05:02:40.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.use_direct_reads: 0 2024-09-16T05:02:40.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.db_log_dir: 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.wal_dir: 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T05:02:40.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.write_buffer_manager: 0x55873db3edc0 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.unordered_write: 0 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T05:02:40.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.row_cache: None 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.wal_filter: None 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.two_write_queues: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.wal_compression: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.atomic_flush: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.log_readahead_size: 0 2024-09-16T05:02:40.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_background_jobs: 2 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_background_compactions: -1 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_subcompactions: 1 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T05:02:40.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_open_files: -1 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_background_flushes: -1 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Compression algorithms supported: 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: kZSTD supported: 0 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: kXpressCompression supported: 0 2024-09-16T05:02:40.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: kBZip2Compression supported: 0 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: kLZ4Compression supported: 1 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: kZlibCompression supported: 1 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: kSnappyCompression supported: 1 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.merge_operator: 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_filter: None 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T05:02:40.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55873db2f320) 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_type: 0 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: data_block_index_type: 0 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_shortening: 1 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: checksum: 4 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: no_block_cache: 0 2024-09-16T05:02:40.854 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache: 0x55873db231f0 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_options: 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: capacity : 536870912 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: num_shard_bits : 4 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: strict_capacity_limit : 0 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_compressed: (nil) 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: persistent_cache: (nil) 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_size: 4096 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_size_deviation: 10 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_restart_interval: 16 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_block_restart_interval: 1 2024-09-16T05:02:40.855 INFO:journalctl@ceph.mon.a.smithi032.stdout: metadata_block_size: 4096 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: partition_filters: 0 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: use_delta_encoding: 1 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: filter_policy: bloomfilter 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: whole_key_filtering: 1 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: verify_compression: 0 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: format_version: 5 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: enable_index_compression: 1 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_align: 0 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: max_auto_readahead_size: 262144 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: prepopulate_block_cache: 0 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: initial_auto_readahead_size: 8192 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T05:02:40.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression: NoCompression 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.num_levels: 7 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T05:02:40.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T05:02:40.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T05:02:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T05:02:40.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.inplace_update_support: 0 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.bloom_locality: 0 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.max_successive_merges: 0 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T05:02:40.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.ttl: 2592000 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.enable_blob_files: false 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.min_blob_size: 0 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T05:02:40.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 38feac50-d3f7-48d1-a127-340c7d09e829 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726462960547480, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726462960548540, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93479, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 292, "table_properties": {"data_size": 91519, "index_size": 280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12452, "raw_average_key_size": 49, "raw_value_size": 84661, "raw_average_value_size": 337, "num_data_blocks": 12, "num_entries": 251, "num_filter_entries": 251, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726462960, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "38feac50-d3f7-48d1-a127-340c7d09e829", "db_session_id": "TG9JZNXNWF00S6G85AB6", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726462960548637, "job": 1, "event": "recovery_finished"} 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55873db50e00 2024-09-16T05:02:40.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: DB pointer 0x55873dc4c000 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** DB Stats ** 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** Compaction Stats [default] ** 2024-09-16T05:02:40.864 INFO:journalctl@ceph.mon.a.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: L0 2/0 93.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: Sum 2/0 93.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** Compaction Stats [default] ** 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 128.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-09-16T05:02:40.865 INFO:journalctl@ceph.mon.a.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative compaction: 0.00 GB write, 27.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval compaction: 0.00 GB write, 27.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: Block cache BinnedLRUCache@0x55873db231f0#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: starting mon.a rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:40.866 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a@-1(???) e1 preinit fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a@-1(???).mds e1 new map 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a@-1(???).mds e1 print_map 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout: e1 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout: btime 2024-09-16T05:02:35:583250+0000 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout: legacy client fscid: -1 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout: No filesystems configured 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-16T05:02:40.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: monmap epoch 1 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: last_changed 2024-09-16T05:02:32.609764+0000 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: min_mon_release 19 (squid) 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: election_strategy: 1 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: fsmap 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: osdmap e1: 0 total, 0 up, 0 in 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 ceph-mon[27388]: mgrmap e1: no daemons active 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 bash[27369]: d9caed178edf300992f3eec70594794e53439b7571d4cc9a28ad07a27b8c35a0 2024-09-16T05:02:40.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:40 smithi032 systemd[1]: Started Ceph mon.a for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:02:41.594 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-16T05:02:41.607 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:02:41.607 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mgr... 2024-09-16T05:02:41.607 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-16T05:02:41.608 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-16T05:02:41.866 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.y 2024-09-16T05:02:41.866 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-16T05:02:42.023 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968.target.wants/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@.service. 2024-09-16T05:02:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1527495400' entity='client.admin' 2024-09-16T05:02:42.529 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:42 smithi032 podman[27599]: 2024-09-16 05:02:42.447831793 +0000 UTC m=+0.300249406 container start 2ac258de4f2a37583f9abb2b352e44a61993c23b52422454cb103077a92edd35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-16T05:02:42.529 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:42 smithi032 bash[27599]: 2ac258de4f2a37583f9abb2b352e44a61993c23b52422454cb103077a92edd35 2024-09-16T05:02:42.544 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T05:02:42.544 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-09-16T05:02:42.551 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T05:02:42.551 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-09-16T05:02:42.551 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-09-16T05:02:42.551 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T05:02:42.560 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T05:02:42.560 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-09-16T05:02:42.568 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T05:02:42.568 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-09-16T05:02:42.568 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-09-16T05:02:42.569 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-16T05:02:42.569 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr to start... 2024-09-16T05:02:42.569 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr... 2024-09-16T05:02:42.797 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:42 smithi032 systemd[1]: Started Ceph mgr.y for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:02:42.798 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:42 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:42.578+0000 7f11b543f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T05:02:42.798 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:42 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:42.647+0000 7f11b543f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T05:02:43.170 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:42 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:42.806+0000 7f11b543f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T05:02:43.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:42 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:42.876+0000 7f11b543f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T05:02:43.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:43 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:43.030+0000 7f11b543f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:02:43.508 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:43 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:43.177+0000 7f11b543f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T05:02:43.508 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:43 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:43.507+0000 7f11b543f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T05:02:43.781 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:43 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:43.570+0000 7f11b543f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T05:02:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1703655711' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-16T05:02:44.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:43 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:43.779+0000 7f11b543f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T05:02:44.920 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:44 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:44.732+0000 7f11b543f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T05:02:44.920 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:44 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:44.804+0000 7f11b543f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T05:02:45.842 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:45 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:45.565+0000 7f11b543f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T05:02:45.842 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:45 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:45.686+0000 7f11b543f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T05:02:46.132 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:45 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:45.837+0000 7f11b543f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:02:46.132 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:45 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:45.901+0000 7f11b543f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T05:02:46.132 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.059+0000 7f11b543f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T05:02:46.132 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.130+0000 7f11b543f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T05:02:46.421 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.193+0000 7f11b543f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T05:02:46.421 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.362+0000 7f11b543f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T05:02:46.717 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.432+0000 7f11b543f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T05:02:46.717 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.503+0000 7f11b543f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T05:02:46.717 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.590+0000 7f11b543f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T05:02:46.717 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.653+0000 7f11b543f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T05:02:46.717 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:46.715+0000 7f11b543f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T05:02:47.920 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:47.548+0000 7f11b543f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T05:02:47.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: Activating manager daemon y 2024-09-16T05:02:47.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: mgrmap e2: y(active, starting, since 0.00322744s) 2024-09-16T05:02:47.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: Manager daemon y is now available 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' 2024-09-16T05:02:47.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:47 smithi032 ceph-mon[27388]: from='mgr.14100 172.21.15.32:0/104952083' entity='mgr.y' 2024-09-16T05:02:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:49 smithi032 ceph-mon[27388]: mgrmap e3: y(active, since 1.00687s) 2024-09-16T05:02:50.073 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T05:02:50.080 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T05:02:50.081 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T05:02:50.082 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T05:02:50.082 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:50.082 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T05:02:50.082 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T05:02:50.082 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T05:02:50.082 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T05:02:50.082 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T05:02:50.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T05:02:35:583250+0000", 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:50.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T05:02:50.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T05:02:50.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T05:02:35.583893+0000", 2024-09-16T05:02:50.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T05:02:50.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:50.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T05:02:50.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-09-16T05:02:50.086 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (1/15)... 2024-09-16T05:02:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:50 smithi032 ceph-mon[27388]: mgrmap e4: y(active, since 2s) 2024-09-16T05:02:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1905320302' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-16T05:02:53.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:53.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-09-16T05:02:53.421 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-09-16T05:02:53.422 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T05:02:53.423 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T05:02:53.427 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T05:02:35:583250+0000", 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T05:02:53.428 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T05:02:35.583893+0000", 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-09-16T05:02:53.429 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T05:02:53.430 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-09-16T05:02:53.430 INFO:teuthology.orchestra.run.smithi032.stdout:mgr is available 2024-09-16T05:02:54.563 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1713847966' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-16T05:02:54.835 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:54.835 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T05:02:54.836 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T05:02:54.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-09-16T05:02:54.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T05:02:54.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_high_multiple = 2 2024-09-16T05:02:54.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout filestore_queue_throttle_max_multiple = 10 2024-09-16T05:02:54.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T05:02:54.837 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T05:02:54.837 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling cephadm module... 2024-09-16T05:02:56.081 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2764139558' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-16T05:02:56.082 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:55 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: ignoring --setuser ceph since I am not root 2024-09-16T05:02:56.082 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:55 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: ignoring --setgroup ceph since I am not root 2024-09-16T05:02:56.082 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:55 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:55.864+0000 7fae112a7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T05:02:56.082 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:55 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:55.924+0000 7fae112a7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T05:02:56.082 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:56 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:56.079+0000 7fae112a7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T05:02:56.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:56 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:56.148+0000 7fae112a7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T05:02:56.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:56 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:56.302+0000 7fae112a7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:02:56.724 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:56 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:56.449+0000 7fae112a7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T05:02:57.019 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2764139558' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-16T05:02:57.019 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:02:56 smithi032 ceph-mon[27388]: mgrmap e5: y(active, since 8s) 2024-09-16T05:02:57.020 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:56 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:56.778+0000 7fae112a7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T05:02:57.020 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:56 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:56.844+0000 7fae112a7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T05:02:57.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:57 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:57.042+0000 7fae112a7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T05:02:58.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:57 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:57.984+0000 7fae112a7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T05:02:58.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:58 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:58.056+0000 7fae112a7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T05:02:58.920 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:58 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:58.805+0000 7fae112a7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T05:02:59.298 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:58 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:58.926+0000 7fae112a7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T05:02:59.298 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.075+0000 7fae112a7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:02:59.298 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.140+0000 7fae112a7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T05:02:59.298 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.296+0000 7fae112a7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T05:02:59.599 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.368+0000 7fae112a7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T05:02:59.599 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.431+0000 7fae112a7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T05:02:59.599 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.598+0000 7fae112a7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T05:02:59.887 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.666+0000 7fae112a7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T05:02:59.888 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.736+0000 7fae112a7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T05:02:59.888 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.824+0000 7fae112a7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T05:02:59.888 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.886+0000 7fae112a7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T05:03:00.170 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:02:59 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:02:59.953+0000 7fae112a7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T05:03:00.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-09-16T05:02:55.433+0000 7fce0bfff640 0 --2- 172.21.15.32:0/2764139558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce0c1071d0 0x7fce0c19a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:03:01.170 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:00.798+0000 7fae112a7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: Active manager daemon y restarted 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: Activating manager daemon y 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: osdmap e2: 0 total, 0 up, 0 in 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: mgrmap e6: y(active, starting, since 0.00344116s) 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: Manager daemon y is now available 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:01.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:01.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:00 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:01.458 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-09-16T05:03:01.458 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 6, 2024-09-16T05:03:01.458 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T05:03:01.458 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-16T05:03:01.458 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T05:03:01.458 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-09-16T05:03:01.458 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-09-16T05:03:01.458 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 6... 2024-09-16T05:03:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:01 smithi032 ceph-mon[27388]: Found migration_current of "None". Setting to last migration. 2024-09-16T05:03:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:01 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-16T05:03:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:01 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-16T05:03:02.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4084249393' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-16T05:03:02.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:01 smithi032 ceph-mon[27388]: mgrmap e7: y(active, since 1.00535s) 2024-09-16T05:03:02.439 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-09-16T05:03:02.439 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-16T05:03:02.439 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T05:03:02.439 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-09-16T05:03:02.439 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 6 is available 2024-09-16T05:03:02.439 INFO:teuthology.orchestra.run.smithi032.stdout:Setting orchestrator backend to cephadm... 2024-09-16T05:03:03.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-mon[27388]: from='client.14127 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-16T05:03:03.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-mon[27388]: from='client.14127 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-16T05:03:03.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:03.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:04.001 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:03.733+0000 7fad785c1640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:05:03:03] ENGINE Error in HTTPServer.serve 2024-09-16T05:03:04.001 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: Traceback (most recent call last): 2024-09-16T05:03:04.001 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: self._connections.run(self.expiration_interval) 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: self._run(expiration_interval) 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: s = self.context.wrap_socket( 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: return self.sslsocket_class._create( 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T05:03:04.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: self.do_handshake() 2024-09-16T05:03:04.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T05:03:04.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: self._sslobj.do_handshake() 2024-09-16T05:03:04.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T05:03:04.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:03 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:04.320 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: mgrmap e8: y(active, since 2s) 2024-09-16T05:03:04.320 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: from='client.14129 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:04.320 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:03] ENGINE Bus STARTING 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:03] ENGINE Serving on http://172.21.15.32:8765 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:03] ENGINE Serving on https://172.21.15.32:7150 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:03] ENGINE Bus STARTED 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:04 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:03] ENGINE Error in HTTPServer.serve 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout: Traceback (most recent call last): 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout: self._connections.run(self.expiration_interval) 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T05:03:04.321 INFO:journalctl@ceph.mon.a.smithi032.stdout: self._run(expiration_interval) 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: s = self.context.wrap_socket( 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: return self.sslsocket_class._create( 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: self.do_handshake() 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: self._sslobj.do_handshake() 2024-09-16T05:03:04.322 INFO:journalctl@ceph.mon.a.smithi032.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T05:03:05.215 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-mon[27388]: from='client.14131 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:05.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-16T05:03:05.218 INFO:teuthology.orchestra.run.smithi032.stdout:Generating ssh key... 2024-09-16T05:03:06.288 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: Generating public/private rsa key pair. 2024-09-16T05:03:06.288 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: Your identification has been saved in /tmp/tmpttzaabg5/key 2024-09-16T05:03:06.288 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: Your public key has been saved in /tmp/tmpttzaabg5/key.pub 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: The key fingerprint is: 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: SHA256:+hf28WWLkLYnlV6vZ4HMlAhIyEKyW44jhV3JbLlZdWk ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: The key's randomart image is: 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: +---[RSA 3072]----+ 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: | ..+ooooo .. | 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: | ooo*o.. oE | 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: |..oo.+ .. . . | 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: | .= o . o | 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: |.+ . S = o | 2024-09-16T05:03:06.289 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: |. . . o+.* oo| 2024-09-16T05:03:06.290 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: | . ..o=ooo+| 2024-09-16T05:03:06.290 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: | . .o.+..+| 2024-09-16T05:03:06.290 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: | .. o .+ | 2024-09-16T05:03:06.290 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: +----[SHA256]-----+ 2024-09-16T05:03:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:06 smithi032 ceph-mon[27388]: from='client.14133 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:06 smithi032 ceph-mon[27388]: Generating ssh key... 2024-09-16T05:03:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:06 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:06 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:07.722 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDIQDAbUKlq4BqEpobspuZ9gEwDdmZJCqvSwHJf94lgpIJy30iVidkAvuTUW9TMtD4SCyPHUaffPshFdXzeb+5li1lXnjn7ty3DZOFKGRDIgt41P/7EVv5DwZqe/da4k2nu/u5/bgy9yxM8U6touu9g6nsjYNol94dDqMRxQP5ApjfYDeVD924UDF/j9OlvuE5rL1V1aE+fAspJYP/W4/abP3jttLpgrlNMov/STvIMdFEH26ubceb1X5ks1q8yGfLs9PfAxOL+8x1rh0SuhTKI86K+trzOgx9Uht3LDZtahYhVGuK9Ul0QVlSZyuFSzMZJKimYkxEHOIndOuTqA/VGXHrwfqYs3ohb/H9w1P1wK0Odr49mNvwBT1XDOIngJVPT38JQ7OtPPM2gaaM6giCxep/pCIY+T8VHvDRTw0mzSMphKiKmBrT4hxfsLtOJgefAlJAIaxBT5RbcUjXIJVCNnrDobu7gfKdPU64UY4Ds+spE4bu0+p3lxwM6WCQcDc= ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:03:07.723 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-16T05:03:07.723 INFO:teuthology.orchestra.run.smithi032.stdout:Adding key to root@localhost authorized_keys... 2024-09-16T05:03:07.733 INFO:teuthology.orchestra.run.smithi032.stdout:Adding host smithi032... 2024-09-16T05:03:08.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:07 smithi032 ceph-mon[27388]: from='client.14135 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:09.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:08 smithi032 ceph-mon[27388]: from='client.14137 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:10.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:09 smithi032 ceph-mon[27388]: Deploying cephadm binary to smithi032 2024-09-16T05:03:12.939 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Added host 'smithi032' with addr '172.21.15.32' 2024-09-16T05:03:12.939 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mon service... 2024-09-16T05:03:13.596 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:13 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:13.596 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:13 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:13.596 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:13 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:13.596 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:13 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:03:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:14 smithi032 ceph-mon[27388]: from='client.14139 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:14 smithi032 ceph-mon[27388]: Saving service mon spec with placement count:5 2024-09-16T05:03:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:14 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:15.137 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-16T05:03:15.137 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mgr service... 2024-09-16T05:03:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:15 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:15 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:17.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:16 smithi032 ceph-mon[27388]: from='client.14141 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:17.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:16 smithi032 ceph-mon[27388]: Saving service mgr spec with placement count:2 2024-09-16T05:03:17.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:16 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:17.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:16 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:17.030 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-16T05:03:18.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:18 smithi032 ceph-mon[27388]: from='mgr.14117 172.21.15.32:0/1588383825' entity='mgr.y' 2024-09-16T05:03:18.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2989635198' entity='client.admin' 2024-09-16T05:03:19.445 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling the dashboard module... 2024-09-16T05:03:20.088 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1096943410' entity='client.admin' 2024-09-16T05:03:20.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2005989811' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-16T05:03:21.170 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:20 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: ignoring --setuser ceph since I am not root 2024-09-16T05:03:21.170 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:20 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: ignoring --setgroup ceph since I am not root 2024-09-16T05:03:21.170 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:21.000+0000 7fe0f9a28040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T05:03:21.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:21.060+0000 7fe0f9a28040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T05:03:21.446 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:21.219+0000 7fe0f9a28040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T05:03:21.446 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:21.288+0000 7fe0f9a28040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T05:03:21.446 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:21.444+0000 7fe0f9a28040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:03:21.732 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:21.600+0000 7fe0f9a28040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T05:03:22.004 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:21.936+0000 7fe0f9a28040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T05:03:22.005 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2005989811' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-16T05:03:22.005 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:21 smithi032 ceph-mon[27388]: mgrmap e9: y(active, since 20s) 2024-09-16T05:03:22.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:22 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:22.003+0000 7fe0f9a28040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T05:03:22.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:22 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:22.199+0000 7fe0f9a28040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T05:03:22.763 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-09-16T05:03:22.763 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-16T05:03:22.763 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T05:03:22.764 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-16T05:03:22.764 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T05:03:22.764 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-09-16T05:03:22.764 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-09-16T05:03:22.764 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 9... 2024-09-16T05:03:23.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1146097894' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-16T05:03:23.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:23 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:23.141+0000 7fe0f9a28040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T05:03:23.420 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:23 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:23.214+0000 7fe0f9a28040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T05:03:24.235 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:23 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:23.963+0000 7fe0f9a28040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T05:03:24.235 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.083+0000 7fe0f9a28040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T05:03:24.525 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.233+0000 7fe0f9a28040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:03:24.526 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.297+0000 7fe0f9a28040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T05:03:24.526 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.453+0000 7fe0f9a28040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T05:03:24.526 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.524+0000 7fe0f9a28040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T05:03:24.822 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.586+0000 7fe0f9a28040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T05:03:24.822 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.753+0000 7fe0f9a28040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T05:03:24.823 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.821+0000 7fe0f9a28040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T05:03:25.102 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.890+0000 7fe0f9a28040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T05:03:25.102 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:24 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:24.977+0000 7fe0f9a28040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T05:03:25.102 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:25.039+0000 7fe0f9a28040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T05:03:25.102 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:25.100+0000 7fe0f9a28040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T05:03:26.201 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:25.945+0000 7fe0f9a28040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T05:03:26.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: Active manager daemon y restarted 2024-09-16T05:03:26.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: Activating manager daemon y 2024-09-16T05:03:26.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: osdmap e3: 0 total, 0 up, 0 in 2024-09-16T05:03:26.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: mgrmap e10: y(active, starting, since 0.00345381s) 2024-09-16T05:03:26.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:03:26.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T05:03:26.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T05:03:26.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T05:03:26.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T05:03:26.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: Manager daemon y is now available 2024-09-16T05:03:26.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-16T05:03:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-16T05:03:27.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-mon[27388]: mgrmap e11: y(active, since 1.0059s) 2024-09-16T05:03:27.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-09-16T05:03:27.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-16T05:03:27.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T05:03:27.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-09-16T05:03:27.924 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 9 is available 2024-09-16T05:03:27.924 INFO:teuthology.orchestra.run.smithi032.stdout:Generating a dashboard self-signed certificate... 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:27.912+0000 7fe064d3c640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:05:03:27] ENGINE Error in HTTPServer.serve 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: Traceback (most recent call last): 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: self._connections.run(self.expiration_interval) 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: self._run(expiration_interval) 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T05:03:28.171 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: s = self.context.wrap_socket( 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: return self.sslsocket_class._create( 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: self.do_handshake() 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: self._sslobj.do_handshake() 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T05:03:28.172 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:03:27 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:03:28.624 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:28.625 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:28 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:27] ENGINE Bus STARTING 2024-09-16T05:03:29.255 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-16T05:03:29.255 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial admin user... 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:27] ENGINE Serving on https://172.21.15.32:7150 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:27] ENGINE Error in HTTPServer.serve 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout: Traceback (most recent call last): 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout: self._connections.run(self.expiration_interval) 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout: self._run(expiration_interval) 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T05:03:29.597 INFO:journalctl@ceph.mon.a.smithi032.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: s = self.context.wrap_socket( 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: return self.sslsocket_class._create( 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: self.do_handshake() 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: self._sslobj.do_handshake() 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:28] ENGINE Serving on http://172.21.15.32:8765 2024-09-16T05:03:29.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: [16/Sep/2024:05:03:28] ENGINE Bus STARTED 2024-09-16T05:03:29.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: mgrmap e12: y(active, since 2s) 2024-09-16T05:03:29.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:29.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:29.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:29.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:30.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:30 smithi032 ceph-mon[27388]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:30.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:30.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xV8ubkJWiOA3ccn.IoEdW.ji2MMQc6wk8hlhLL/v.mrWWNzz..Gb.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726463010, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-16T05:03:30.781 INFO:teuthology.orchestra.run.smithi032.stdout:Fetching dashboard port number... 2024-09-16T05:03:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3250646293' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-16T05:03:32.301 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 8443 2024-09-16T05:03:32.312 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T05:03:32.312 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-09-16T05:03:32.320 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T05:03:32.320 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-09-16T05:03:32.321 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-09-16T05:03:32.321 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-16T05:03:32.322 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph Dashboard is now available at: 2024-09-16T05:03:32.322 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:32.322 INFO:teuthology.orchestra.run.smithi032.stdout: URL: https://smithi032.front.sepia.ceph.com:8443/ 2024-09-16T05:03:32.322 INFO:teuthology.orchestra.run.smithi032.stdout: User: admin 2024-09-16T05:03:32.323 INFO:teuthology.orchestra.run.smithi032.stdout: Password: rbrbv2lrmh 2024-09-16T05:03:32.323 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:32.323 INFO:teuthology.orchestra.run.smithi032.stdout:Saving cluster configuration to /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config directory 2024-09-16T05:03:33.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:33 smithi032 ceph-mon[27388]: mgrmap e13: y(active, since 6s) 2024-09-16T05:03:33.597 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1369989874' entity='client.admin' 2024-09-16T05:03:33.599 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-16T05:03:33.599 INFO:teuthology.orchestra.run.smithi032.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:Or, if you are only running a single cluster on this host: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: ceph telemetry on 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout:For more information see: 2024-09-16T05:03:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:33.601 INFO:teuthology.orchestra.run.smithi032.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-16T05:03:33.601 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:33.601 INFO:teuthology.orchestra.run.smithi032.stdout:Bootstrap complete. 2024-09-16T05:03:33.644 INFO:tasks.cephadm:Fetching config... 2024-09-16T05:03:33.644 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:03:33.644 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-16T05:03:33.662 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-16T05:03:33.662 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:03:33.662 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-16T05:03:33.719 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-16T05:03:33.719 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:03:33.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-16T05:03:33.786 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-16T05:03:33.786 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:03:33.786 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-16T05:03:33.842 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-16T05:03:33.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDIQDAbUKlq4BqEpobspuZ9gEwDdmZJCqvSwHJf94lgpIJy30iVidkAvuTUW9TMtD4SCyPHUaffPshFdXzeb+5li1lXnjn7ty3DZOFKGRDIgt41P/7EVv5DwZqe/da4k2nu/u5/bgy9yxM8U6touu9g6nsjYNol94dDqMRxQP5ApjfYDeVD924UDF/j9OlvuE5rL1V1aE+fAspJYP/W4/abP3jttLpgrlNMov/STvIMdFEH26ubceb1X5ks1q8yGfLs9PfAxOL+8x1rh0SuhTKI86K+trzOgx9Uht3LDZtahYhVGuK9Ul0QVlSZyuFSzMZJKimYkxEHOIndOuTqA/VGXHrwfqYs3ohb/H9w1P1wK0Odr49mNvwBT1XDOIngJVPT38JQ7OtPPM2gaaM6giCxep/pCIY+T8VHvDRTw0mzSMphKiKmBrT4hxfsLtOJgefAlJAIaxBT5RbcUjXIJVCNnrDobu7gfKdPU64UY4Ds+spE4bu0+p3lxwM6WCQcDc= ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T05:03:33.925 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDIQDAbUKlq4BqEpobspuZ9gEwDdmZJCqvSwHJf94lgpIJy30iVidkAvuTUW9TMtD4SCyPHUaffPshFdXzeb+5li1lXnjn7ty3DZOFKGRDIgt41P/7EVv5DwZqe/da4k2nu/u5/bgy9yxM8U6touu9g6nsjYNol94dDqMRxQP5ApjfYDeVD924UDF/j9OlvuE5rL1V1aE+fAspJYP/W4/abP3jttLpgrlNMov/STvIMdFEH26ubceb1X5ks1q8yGfLs9PfAxOL+8x1rh0SuhTKI86K+trzOgx9Uht3LDZtahYhVGuK9Ul0QVlSZyuFSzMZJKimYkxEHOIndOuTqA/VGXHrwfqYs3ohb/H9w1P1wK0Odr49mNvwBT1XDOIngJVPT38JQ7OtPPM2gaaM6giCxep/pCIY+T8VHvDRTw0mzSMphKiKmBrT4hxfsLtOJgefAlJAIaxBT5RbcUjXIJVCNnrDobu7gfKdPU64UY4Ds+spE4bu0+p3lxwM6WCQcDc= ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:03:33.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDIQDAbUKlq4BqEpobspuZ9gEwDdmZJCqvSwHJf94lgpIJy30iVidkAvuTUW9TMtD4SCyPHUaffPshFdXzeb+5li1lXnjn7ty3DZOFKGRDIgt41P/7EVv5DwZqe/da4k2nu/u5/bgy9yxM8U6touu9g6nsjYNol94dDqMRxQP5ApjfYDeVD924UDF/j9OlvuE5rL1V1aE+fAspJYP/W4/abP3jttLpgrlNMov/STvIMdFEH26ubceb1X5ks1q8yGfLs9PfAxOL+8x1rh0SuhTKI86K+trzOgx9Uht3LDZtahYhVGuK9Ul0QVlSZyuFSzMZJKimYkxEHOIndOuTqA/VGXHrwfqYs3ohb/H9w1P1wK0Odr49mNvwBT1XDOIngJVPT38JQ7OtPPM2gaaM6giCxep/pCIY+T8VHvDRTw0mzSMphKiKmBrT4hxfsLtOJgefAlJAIaxBT5RbcUjXIJVCNnrDobu7gfKdPU64UY4Ds+spE4bu0+p3lxwM6WCQcDc= ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T05:03:33.995 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDIQDAbUKlq4BqEpobspuZ9gEwDdmZJCqvSwHJf94lgpIJy30iVidkAvuTUW9TMtD4SCyPHUaffPshFdXzeb+5li1lXnjn7ty3DZOFKGRDIgt41P/7EVv5DwZqe/da4k2nu/u5/bgy9yxM8U6touu9g6nsjYNol94dDqMRxQP5ApjfYDeVD924UDF/j9OlvuE5rL1V1aE+fAspJYP/W4/abP3jttLpgrlNMov/STvIMdFEH26ubceb1X5ks1q8yGfLs9PfAxOL+8x1rh0SuhTKI86K+trzOgx9Uht3LDZtahYhVGuK9Ul0QVlSZyuFSzMZJKimYkxEHOIndOuTqA/VGXHrwfqYs3ohb/H9w1P1wK0Odr49mNvwBT1XDOIngJVPT38JQ7OtPPM2gaaM6giCxep/pCIY+T8VHvDRTw0mzSMphKiKmBrT4hxfsLtOJgefAlJAIaxBT5RbcUjXIJVCNnrDobu7gfKdPU64UY4Ds+spE4bu0+p3lxwM6WCQcDc= ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:03:34.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDIQDAbUKlq4BqEpobspuZ9gEwDdmZJCqvSwHJf94lgpIJy30iVidkAvuTUW9TMtD4SCyPHUaffPshFdXzeb+5li1lXnjn7ty3DZOFKGRDIgt41P/7EVv5DwZqe/da4k2nu/u5/bgy9yxM8U6touu9g6nsjYNol94dDqMRxQP5ApjfYDeVD924UDF/j9OlvuE5rL1V1aE+fAspJYP/W4/abP3jttLpgrlNMov/STvIMdFEH26ubceb1X5ks1q8yGfLs9PfAxOL+8x1rh0SuhTKI86K+trzOgx9Uht3LDZtahYhVGuK9Ul0QVlSZyuFSzMZJKimYkxEHOIndOuTqA/VGXHrwfqYs3ohb/H9w1P1wK0Odr49mNvwBT1XDOIngJVPT38JQ7OtPPM2gaaM6giCxep/pCIY+T8VHvDRTw0mzSMphKiKmBrT4hxfsLtOJgefAlJAIaxBT5RbcUjXIJVCNnrDobu7gfKdPU64UY4Ds+spE4bu0+p3lxwM6WCQcDc= ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T05:03:34.057 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDIQDAbUKlq4BqEpobspuZ9gEwDdmZJCqvSwHJf94lgpIJy30iVidkAvuTUW9TMtD4SCyPHUaffPshFdXzeb+5li1lXnjn7ty3DZOFKGRDIgt41P/7EVv5DwZqe/da4k2nu/u5/bgy9yxM8U6touu9g6nsjYNol94dDqMRxQP5ApjfYDeVD924UDF/j9OlvuE5rL1V1aE+fAspJYP/W4/abP3jttLpgrlNMov/STvIMdFEH26ubceb1X5ks1q8yGfLs9PfAxOL+8x1rh0SuhTKI86K+trzOgx9Uht3LDZtahYhVGuK9Ul0QVlSZyuFSzMZJKimYkxEHOIndOuTqA/VGXHrwfqYs3ohb/H9w1P1wK0Odr49mNvwBT1XDOIngJVPT38JQ7OtPPM2gaaM6giCxep/pCIY+T8VHvDRTw0mzSMphKiKmBrT4hxfsLtOJgefAlJAIaxBT5RbcUjXIJVCNnrDobu7gfKdPU64UY4Ds+spE4bu0+p3lxwM6WCQcDc= ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:03:34.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-16T05:03:34.326 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:03:35.598 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-16T05:03:35.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-16T05:03:36.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4029310423' entity='client.admin' 2024-09-16T05:03:36.436 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:03:37.709 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-09-16T05:03:37.709 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:03:37.709 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-09-16T05:03:37.728 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:03:37.728 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:03:37.787 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-09-16T05:03:37.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch host add smithi196 2024-09-16T05:03:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:03:38.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:38 smithi032 ceph-mon[27388]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:38.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:38 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:39.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: Updating smithi032:/etc/ceph/ceph.conf 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: Updating smithi032:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: Updating smithi032:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:39.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:39.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:03:39.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:03:39.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:40.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:40 smithi032 ceph-mon[27388]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:41.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:41 smithi032 ceph-mon[27388]: Deploying cephadm binary to smithi196 2024-09-16T05:03:42.905 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-09-16T05:03:43.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-16T05:03:43.757 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:03:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:43 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:43 smithi032 ceph-mon[27388]: Added host smithi196 2024-09-16T05:03:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:43 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:43 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:43 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:44.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:44.377 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-09-16T05:03:44.976 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-09-16T05:03:44.976 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:03:44.976 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-09-16T05:03:44.995 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:03:44.995 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:03:45.053 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-09-16T05:03:45.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch host add smithi204 2024-09-16T05:03:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:03:45.524 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:45.524 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:45 smithi032 ceph-mon[27388]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:03:47.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:47 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:03:49.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:48 smithi032 ceph-mon[27388]: Updating smithi196:/etc/ceph/ceph.conf 2024-09-16T05:03:49.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:48 smithi032 ceph-mon[27388]: Deploying cephadm binary to smithi204 2024-09-16T05:03:49.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:48 smithi032 ceph-mon[27388]: Updating smithi196:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:03:49.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:48 smithi032 ceph-mon[27388]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:03:49.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:48 smithi032 ceph-mon[27388]: Updating smithi196:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T05:03:49.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:48 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:49.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:48 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:49.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:48 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:49.864 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-09-16T05:03:50.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:49 smithi032 ceph-mon[27388]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:03:50.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:50.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:03:50.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-16T05:03:50.676 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:03:51.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:50 smithi032 ceph-mon[27388]: Added host smithi204 2024-09-16T05:03:51.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:51.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:51.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:03:51.312 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-09-16T05:03:51.909 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-16T05:03:51.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-16T05:03:52.160 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:03:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:51 smithi032 ceph-mon[27388]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:03:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:51 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:52.874 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-09-16T05:03:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:52 smithi032 ceph-mon[27388]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:03:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2993443072' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-16T05:03:53.511 INFO:tasks.cephadm:Adding mon.a on smithi032 2024-09-16T05:03:53.511 INFO:tasks.cephadm:Adding mon.b on smithi196 2024-09-16T05:03:53.511 INFO:tasks.cephadm:Adding mon.c on smithi204 2024-09-16T05:03:53.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi032:172.21.15.32=a;smithi196:172.21.15.196=b;smithi204:172.21.15.204=c' 2024-09-16T05:03:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:53 smithi032 ceph-mon[27388]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:03:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2993443072' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T05:03:54.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:53 smithi032 ceph-mon[27388]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:03:54.177 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T05:03:54.223 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T05:03:54.935 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:03:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: Updating smithi204:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: Updating smithi204:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:03:55.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:03:55.587 DEBUG:teuthology.orchestra.run.smithi196:mon.b> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.b.service 2024-09-16T05:03:55.590 DEBUG:teuthology.orchestra.run.smithi204:mon.c> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.c.service 2024-09-16T05:03:55.592 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T05:03:55.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T05:03:56.324 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:03:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:56 smithi032 ceph-mon[27388]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi196:172.21.15.196=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:03:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:56 smithi032 ceph-mon[27388]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi196:172.21.15.196=b;smithi204:172.21.15.204=c;count:3 2024-09-16T05:03:56.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:56 smithi032 ceph-mon[27388]: Deploying daemon mon.c on smithi204 2024-09-16T05:03:57.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:57 smithi032 ceph-mon[27388]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:03:57.501 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:03:57.501 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","modified":"2024-09-16T05:02:32.609764Z","created":"2024-09-16T05:02:32.609764Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-16T05:03:57.502 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-09-16T05:03:58.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/4150411813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:03:59.207 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T05:03:59.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T05:03:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:03:59 smithi032 ceph-mon[27388]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:03:59.494 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:04:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 podman[30899]: 2024-09-16 05:03:59.98481941 +0000 UTC m=+0.016116206 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:04:00.183 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:04:00.183 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","modified":"2024-09-16T05:02:32.609764Z","created":"2024-09-16T05:02:32.609764Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-16T05:04:00.183 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-09-16T05:04:00.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 podman[30899]: 2024-09-16 05:04:00.431876854 +0000 UTC m=+0.463173649 container init 66b7fb61ff0fa0e3a38eb8e65afeaf6a8848d009fba38cbc5cb17d88ce625111 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 podman[30899]: 2024-09-16 05:04:00.435613997 +0000 UTC m=+0.466910791 container start 66b7fb61ff0fa0e3a38eb8e65afeaf6a8848d009fba38cbc5cb17d88ce625111 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-c, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: pidfile_write: ignore empty --pid-file 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: load: jerasure load: lrc 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: RocksDB version: 7.9.2 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Git sha 0 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: DB SUMMARY 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: DB Session ID: JU4CW974XKWJI7UZJSSO 2024-09-16T05:04:00.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: CURRENT file: CURRENT 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.error_if_exists: 0 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.create_if_missing: 0 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.paranoid_checks: 1 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T05:04:00.852 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.env: 0x55701e221940 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.info_log: 0x55702019f6e0 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.statistics: (nil) 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.use_fsync: 0 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_log_file_size: 0 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.allow_fallocate: 1 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T05:04:00.853 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.use_direct_reads: 0 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.db_log_dir: 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.wal_dir: 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T05:04:00.854 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.write_buffer_manager: 0x5570201aedc0 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.unordered_write: 0 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T05:04:00.855 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.row_cache: None 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.wal_filter: None 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.two_write_queues: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.wal_compression: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.atomic_flush: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.log_readahead_size: 0 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T05:04:00.856 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_background_jobs: 2 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_background_compactions: -1 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_subcompactions: 1 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T05:04:00.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_open_files: -1 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_background_flushes: -1 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Compression algorithms supported: 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: kZSTD supported: 0 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: kXpressCompression supported: 0 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: kBZip2Compression supported: 0 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T05:04:00.858 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: kLZ4Compression supported: 1 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: kZlibCompression supported: 1 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: kSnappyCompression supported: 1 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.merge_operator: 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_filter: None 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T05:04:00.859 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55702019f320) 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks: 1 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_top_level_index_and_filter: 1 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_type: 0 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_index_type: 0 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_shortening: 1 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: checksum: 4 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: no_block_cache: 0 2024-09-16T05:04:00.860 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache: 0x5570201931f0 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_name: BinnedLRUCache 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_options: 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: capacity : 536870912 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_shard_bits : 4 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: strict_capacity_limit : 0 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: high_pri_pool_ratio: 0.000 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_compressed: (nil) 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: persistent_cache: (nil) 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size: 4096 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size_deviation: 10 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_restart_interval: 16 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_block_restart_interval: 1 2024-09-16T05:04:00.861 INFO:journalctl@ceph.mon.c.smithi204.stdout: metadata_block_size: 4096 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: partition_filters: 0 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: use_delta_encoding: 1 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: filter_policy: bloomfilter 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: whole_key_filtering: 1 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: verify_compression: 0 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: read_amp_bytes_per_bit: 0 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: format_version: 5 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_index_compression: 1 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_align: 0 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: max_auto_readahead_size: 262144 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: prepopulate_block_cache: 0 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: initial_auto_readahead_size: 8192 2024-09-16T05:04:00.862 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression: NoCompression 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.num_levels: 7 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T05:04:00.863 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T05:04:00.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T05:04:00.865 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T05:04:00.865 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T05:04:00.865 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T05:04:00.865 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T05:04:00.865 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T05:04:00.865 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T05:04:00.865 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T05:04:00.865 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T05:04:00.866 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T05:04:00.866 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T05:04:00.866 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T05:04:00.866 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T05:04:00.867 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T05:04:00.868 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.inplace_update_support: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.bloom_locality: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.max_successive_merges: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.ttl: 2592000 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T05:04:00.869 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.enable_blob_files: false 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.min_blob_size: 0 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T05:04:00.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 736b51a6-3a99-44ef-a401-235c8123bc3b 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726463040463185, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726463040463621, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726463040, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "736b51a6-3a99-44ef-a401-235c8123bc3b", "db_session_id": "JU4CW974XKWJI7UZJSSO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726463040463730, "job": 1, "event": "recovery_finished"} 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5570201c0e00 2024-09-16T05:04:00.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: DB pointer 0x5570202bc000 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** DB Stats ** 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T05:04:00.872 INFO:journalctl@ceph.mon.c.smithi204.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T05:04:00.873 INFO:journalctl@ceph.mon.c.smithi204.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache BinnedLRUCache@0x5570201931f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: using public_addr v2:172.21.15.204:0/0 -> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-09-16T05:04:00.874 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: starting mon.c rank -1 at public addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] at bind addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(???) e0 preinit fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).mds e1 new map 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout: e1 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout: btime 2024-09-16T05:02:35:583250+0000 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout: legacy client fscid: -1 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout: No filesystems configured 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-16T05:04:00.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2993443072' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2993443072' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T05:04:00.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: Updating smithi204:/etc/ceph/ceph.conf 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: Updating smithi204:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: Updating smithi204:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi196:172.21.15.196=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi196:172.21.15.196=b;smithi204:172.21.15.204=c;count:3 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: Deploying daemon mon.c on smithi204 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/4150411813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 ceph-mon[30933]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 bash[30899]: 66b7fb61ff0fa0e3a38eb8e65afeaf6a8848d009fba38cbc5cb17d88ce625111 2024-09-16T05:04:00.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:00 smithi204 systemd[1]: Started Ceph mon.c for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:04:02.156 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T05:04:02.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T05:04:02.412 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:04:05.874 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:05.874 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: mon.a calling monitor election 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: mon.c calling monitor election 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: monmap epoch 2 2024-09-16T05:04:05.875 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: last_changed 2024-09-16T05:04:00.492828+0000 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: min_mon_release 19 (squid) 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: election_strategy: 1 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: fsmap 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: mgrmap e13: y(active, since 39s) 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: overall HEALTH_OK 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:05.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:05.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: mon.a calling monitor election 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: mon.c calling monitor election 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: monmap epoch 2 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: last_changed 2024-09-16T05:04:00.492828+0000 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: min_mon_release 19 (squid) 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: election_strategy: 1 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: fsmap 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: mgrmap e13: y(active, since 39s) 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: overall HEALTH_OK 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:05.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:05.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:05.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:06.084 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:04:06.084 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","modified":"2024-09-16T05:04:00.492828Z","created":"2024-09-16T05:02:32.609764Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T05:04:06.085 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-09-16T05:04:06.640 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:06 smithi204 ceph-mon[30933]: Deploying daemon mon.b on smithi196 2024-09-16T05:04:06.640 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/2673897175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:04:06.640 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:06 smithi032 ceph-mon[27388]: Deploying daemon mon.b on smithi196 2024-09-16T05:04:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/2673897175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:04:06.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:07.668 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T05:04:07.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T05:04:07.918 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:04:07.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:07 smithi032 ceph-mon[27388]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:07.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:07 smithi204 ceph-mon[30933]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:08.580 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:04:08.580 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","modified":"2024-09-16T05:04:00.492828Z","created":"2024-09-16T05:02:32.609764Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T05:04:08.580 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-09-16T05:04:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:09 smithi196 systemd[1]: Started Ceph mon.b for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:04:10.149 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T05:04:10.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T05:04:10.398 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:04:14.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:04:14.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: mon.c calling monitor election 2024-09-16T05:04:14.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: mon.a calling monitor election 2024-09-16T05:04:14.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:14.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:14.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: monmap epoch 3 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: last_changed 2024-09-16T05:04:09.059648+0000 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: min_mon_release 19 (squid) 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: election_strategy: 1 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:04:14.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: fsmap 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: mgrmap e13: y(active, since 48s) 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: mon.b (rank 2) addr [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] is down (out of quorum) 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:14.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: mon.c calling monitor election 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: mon.a calling monitor election 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:14.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: monmap epoch 3 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: last_changed 2024-09-16T05:04:09.059648+0000 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: min_mon_release 19 (squid) 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: election_strategy: 1 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: fsmap 2024-09-16T05:04:14.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: mgrmap e13: y(active, since 48s) 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: mon.b (rank 2) addr [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] is down (out of quorum) 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:14.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:14.706 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:04:14.706 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","modified":"2024-09-16T05:04:09.059648Z","created":"2024-09-16T05:02:32.609764Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T05:04:14.707 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-09-16T05:04:15.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:15 smithi204 ceph-mon[30933]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:15.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:15.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/173501491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:04:15.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:15.340 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-16T05:04:15.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-16T05:04:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:15 smithi032 ceph-mon[27388]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/173501491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:04:15.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:15.608 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:04:16.296 INFO:teuthology.orchestra.run.smithi032.stdout:# minimal ceph.conf for c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:16.296 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-09-16T05:04:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:16.296 INFO:teuthology.orchestra.run.smithi032.stdout: mon_host = [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-09-16T05:04:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:04:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: mon.c calling monitor election 2024-09-16T05:04:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: mon.a calling monitor election 2024-09-16T05:04:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: monmap epoch 3 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: last_changed 2024-09-16T05:04:09.059648+0000 2024-09-16T05:04:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: min_mon_release 19 (squid) 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: election_strategy: 1 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: fsmap 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: mgrmap e13: y(active, since 48s) 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: mon.b (rank 2) addr [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] is down (out of quorum) 2024-09-16T05:04:16.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:16.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:16.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:16.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:16.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:16.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:16.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/173501491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:04:16.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:16.891 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-16T05:04:16.891 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:04:16.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T05:04:16.920 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:04:16.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:04:16.986 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:04:16.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T05:04:17.016 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:04:17.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:04:17.082 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:04:17.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T05:04:17.111 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:04:17.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T05:04:17.177 INFO:tasks.cephadm:Adding mgr.y on smithi032 2024-09-16T05:04:17.177 INFO:tasks.cephadm:Adding mgr.x on smithi196 2024-09-16T05:04:17.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi032=y;smithi196=x' 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: mon.b calling monitor election 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: Updating smithi204:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: Updating smithi196:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: Updating smithi032:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: mon.b calling monitor election 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: mon.c calling monitor election 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: mon.a calling monitor election 2024-09-16T05:04:17.275 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: monmap epoch 3 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: last_changed 2024-09-16T05:04:09.059648+0000 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: min_mon_release 19 (squid) 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: election_strategy: 1 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: fsmap 2024-09-16T05:04:17.276 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: mgrmap e13: y(active, since 50s) 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: overall HEALTH_OK 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3043837921' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:17.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:16 smithi032 ceph-mon[27388]: Reconfiguring daemon mon.a on smithi032 2024-09-16T05:04:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: mon.b calling monitor election 2024-09-16T05:04:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: Updating smithi204:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: Updating smithi196:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: Updating smithi032:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: mon.b calling monitor election 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: mon.c calling monitor election 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: mon.a calling monitor election 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: monmap epoch 3 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: last_changed 2024-09-16T05:04:09.059648+0000 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: min_mon_release 19 (squid) 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: election_strategy: 1 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-09-16T05:04:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: fsmap 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: mgrmap e13: y(active, since 50s) 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: overall HEALTH_OK 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3043837921' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:17.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:16 smithi204 ceph-mon[30933]: Reconfiguring daemon mon.a on smithi032 2024-09-16T05:04:17.454 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:04:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: mon.b calling monitor election 2024-09-16T05:04:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: Updating smithi204:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: Updating smithi196:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: Updating smithi032:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: mon.b calling monitor election 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: mon.c calling monitor election 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: mon.a calling monitor election 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: monmap epoch 3 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: last_changed 2024-09-16T05:04:09.059648+0000 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: created 2024-09-16T05:02:32.609764+0000 2024-09-16T05:04:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: min_mon_release 19 (squid) 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: election_strategy: 1 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: fsmap 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: mgrmap e13: y(active, since 50s) 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: overall HEALTH_OK 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3043837921' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: Reconfiguring daemon mon.a on smithi032 2024-09-16T05:04:18.080 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-09-16T05:04:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:18.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:18.421 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:04:18 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:04:18.063+0000 7fe098731640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T05:04:18.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:18.619 DEBUG:teuthology.orchestra.run.smithi196:mgr.x> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.x.service 2024-09-16T05:04:18.631 INFO:tasks.cephadm:Deploying OSDs... 2024-09-16T05:04:18.631 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:04:18.632 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T05:04:18.681 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T05:04:18.681 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-09-16T05:04:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T05:04:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:18.761 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T05:04:18.761 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:18.761 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:18.761 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-09-16 05:03:36.764081269 +0000 2024-09-16T05:04:18.761 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-09-16 04:59:36.144200029 +0000 2024-09-16T05:04:18.762 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-09-16 04:59:36.144200029 +0000 2024-09-16T05:04:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-09-16 04:59:36.144200029 +0000 2024-09-16T05:04:18.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T05:04:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-09-16T05:04:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-09-16T05:04:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000133741 s, 3.8 MB/s 2024-09-16T05:04:18.855 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T05:04:18.953 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-09-16T05:04:19.038 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T05:04:19.038 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:19.038 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T05:04:19.038 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:19.038 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:19.038 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-09-16 05:03:36.792080686 +0000 2024-09-16T05:04:19.038 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-09-16 04:59:36.476192928 +0000 2024-09-16T05:04:19.039 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-09-16 04:59:36.476192928 +0000 2024-09-16T05:04:19.039 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-09-16 04:59:36.476192928 +0000 2024-09-16T05:04:19.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T05:04:19.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: Reconfiguring mon.b (monmap changed)... 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: Reconfiguring daemon mon.b on smithi196 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: Saving service mgr spec with placement smithi032=y;smithi196=x;count:2 2024-09-16T05:04:19.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.106 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-09-16T05:04:19.107 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-09-16T05:04:19.107 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000125687 s, 4.1 MB/s 2024-09-16T05:04:19.108 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T05:04:19.176 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-09-16T05:04:19.237 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T05:04:19.237 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:19.237 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T05:04:19.237 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:19.237 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:19.238 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-09-16 05:03:36.821080083 +0000 2024-09-16T05:04:19.238 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-09-16 04:59:36.806185870 +0000 2024-09-16T05:04:19.238 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-09-16 04:59:36.806185870 +0000 2024-09-16T05:04:19.238 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-09-16 04:59:36.806185870 +0000 2024-09-16T05:04:19.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T05:04:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-09-16T05:04:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-09-16T05:04:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000121246 s, 4.2 MB/s 2024-09-16T05:04:19.308 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T05:04:19.370 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: Reconfiguring mon.b (monmap changed)... 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: Reconfiguring daemon mon.b on smithi196 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:19.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: Saving service mgr spec with placement smithi032=y;smithi196=x;count:2 2024-09-16T05:04:19.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: Reconfiguring mon.b (monmap changed)... 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: Reconfiguring daemon mon.b on smithi196 2024-09-16T05:04:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:19.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: Saving service mgr spec with placement smithi032=y;smithi196=x;count:2 2024-09-16T05:04:19.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:19.432 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T05:04:19.432 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:19.432 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T05:04:19.433 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:19.433 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:19.433 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-09-16 05:03:36.850079479 +0000 2024-09-16T05:04:19.433 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-09-16 04:59:37.128178983 +0000 2024-09-16T05:04:19.433 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-09-16 04:59:37.128178983 +0000 2024-09-16T05:04:19.433 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-09-16 04:59:37.128178983 +0000 2024-09-16T05:04:19.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T05:04:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-09-16T05:04:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-09-16T05:04:19.501 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000118673 s, 4.3 MB/s 2024-09-16T05:04:19.502 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T05:04:19.562 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:04:19.562 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T05:04:19.579 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T05:04:19.580 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-09-16T05:04:19.636 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T05:04:19.636 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:19.636 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T05:04:19.636 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:19.636 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:19.637 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-16 05:04:08.520776289 +0000 2024-09-16T05:04:19.637 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-16 04:59:35.158831166 +0000 2024-09-16T05:04:19.637 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-16 04:59:35.158831166 +0000 2024-09-16T05:04:19.637 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-16 04:59:35.158831166 +0000 2024-09-16T05:04:19.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T05:04:19.700 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-16T05:04:19.701 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-16T05:04:19.701 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000235371 s, 2.2 MB/s 2024-09-16T05:04:19.702 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T05:04:19.759 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-09-16T05:04:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T05:04:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:19.814 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T05:04:19.814 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:19.814 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:19.814 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-16 05:04:08.521776267 +0000 2024-09-16T05:04:19.815 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-16 04:59:35.471824042 +0000 2024-09-16T05:04:19.815 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-16 04:59:35.471824042 +0000 2024-09-16T05:04:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-16 04:59:35.471824042 +0000 2024-09-16T05:04:19.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T05:04:19.877 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-16T05:04:19.877 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-16T05:04:19.877 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000198012 s, 2.6 MB/s 2024-09-16T05:04:19.878 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T05:04:19.936 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-09-16T05:04:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T05:04:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:19.991 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T05:04:19.991 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:19.992 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:19.992 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-16 05:04:08.521776267 +0000 2024-09-16T05:04:19.992 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-16 04:59:35.793816712 +0000 2024-09-16T05:04:19.992 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-16 04:59:35.793816712 +0000 2024-09-16T05:04:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-16 04:59:35.793816712 +0000 2024-09-16T05:04:19.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T05:04:20.056 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-16T05:04:20.056 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-16T05:04:20.056 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000215945 s, 2.4 MB/s 2024-09-16T05:04:20.057 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T05:04:20.114 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-16 05:04:08.521776267 +0000 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-16 04:59:36.134808950 +0000 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-16 04:59:36.134808950 +0000 2024-09-16T05:04:20.170 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-16 04:59:36.134808950 +0000 2024-09-16T05:04:20.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T05:04:20.233 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-16T05:04:20.233 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-16T05:04:20.233 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000221688 s, 2.3 MB/s 2024-09-16T05:04:20.234 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T05:04:20.291 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:04:20.291 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T05:04:20.308 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T05:04:20.308 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-16 05:03:59.651120494 +0000 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-16 04:59:35.144708270 +0000 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-16 04:59:35.144708270 +0000 2024-09-16T05:04:20.364 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-16 04:59:35.144708270 +0000 2024-09-16T05:04:20.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T05:04:20.431 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-16T05:04:20.431 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-16T05:04:20.431 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000209066 s, 2.4 MB/s 2024-09-16T05:04:20.432 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T05:04:20.489 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-09-16T05:04:20.545 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T05:04:20.545 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:20.546 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T05:04:20.546 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:20.546 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:20.546 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-16 05:03:59.651120494 +0000 2024-09-16T05:04:20.546 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-16 04:59:35.474703654 +0000 2024-09-16T05:04:20.546 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-16 04:59:35.474703654 +0000 2024-09-16T05:04:20.546 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-16 04:59:35.474703654 +0000 2024-09-16T05:04:20.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T05:04:20.609 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-16T05:04:20.610 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-16T05:04:20.610 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000212893 s, 2.4 MB/s 2024-09-16T05:04:20.611 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T05:04:20.667 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-09-16T05:04:20.724 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T05:04:20.724 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:20.724 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-16T05:04:20.725 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:20.725 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:20.725 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-16 05:03:59.652120480 +0000 2024-09-16T05:04:20.725 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-16 04:59:35.787699276 +0000 2024-09-16T05:04:20.725 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-16 04:59:35.787699276 +0000 2024-09-16T05:04:20.725 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-16 04:59:35.787699276 +0000 2024-09-16T05:04:20.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T05:04:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:20 smithi196 ceph-mon[30802]: Reconfiguring mon.c (monmap changed)... 2024-09-16T05:04:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:20 smithi196 ceph-mon[30802]: Reconfiguring daemon mon.c on smithi204 2024-09-16T05:04:20.786 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:20.786 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:20.786 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:20 smithi204 ceph-mon[30933]: Reconfiguring mon.c (monmap changed)... 2024-09-16T05:04:20.786 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:20.786 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:20.786 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:20.786 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:20 smithi204 ceph-mon[30933]: Reconfiguring daemon mon.c on smithi204 2024-09-16T05:04:20.788 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-16T05:04:20.788 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-16T05:04:20.788 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00020131 s, 2.5 MB/s 2024-09-16T05:04:20.790 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T05:04:20.847 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-09-16T05:04:20.902 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T05:04:20.902 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:04:20.903 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T05:04:20.903 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:04:20.903 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T05:04:20.903 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-16 05:03:59.652120480 +0000 2024-09-16T05:04:20.903 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-16 04:59:36.120694618 +0000 2024-09-16T05:04:20.903 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-16 04:59:36.120694618 +0000 2024-09-16T05:04:20.903 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-16 04:59:36.120694618 +0000 2024-09-16T05:04:20.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T05:04:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:20 smithi032 ceph-mon[27388]: Reconfiguring mon.c (monmap changed)... 2024-09-16T05:04:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T05:04:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T05:04:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:20 smithi032 ceph-mon[27388]: Reconfiguring daemon mon.c on smithi204 2024-09-16T05:04:20.966 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-16T05:04:20.967 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-16T05:04:20.967 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000198159 s, 2.6 MB/s 2024-09-16T05:04:20.968 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T05:04:21.024 INFO:tasks.cephadm:Deploying osd.0 on smithi032 with /dev/vg_nvme/lv_4... 2024-09-16T05:04:21.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T05:04:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:04:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:21.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:21.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:21.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T05:04:21.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T05:04:21.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T05:04:21.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:21.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:21 smithi196 ceph-mon[30802]: Deploying daemon mgr.x on smithi196 2024-09-16T05:04:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T05:04:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:21 smithi032 ceph-mon[27388]: Deploying daemon mgr.x on smithi196 2024-09-16T05:04:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:22.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T05:04:22.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T05:04:22.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T05:04:22.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:22.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:21 smithi204 ceph-mon[30933]: Deploying daemon mgr.x on smithi196 2024-09-16T05:04:22.738 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:22 smithi196 systemd[1]: Starting Ceph mgr.x for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:04:23.086 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:22 smithi196 podman[32150]: 2024-09-16 05:04:22.726007892 +0000 UTC m=+0.016069884 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:04:23.087 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:22 smithi196 podman[32150]: 2024-09-16 05:04:22.858861836 +0000 UTC m=+0.148923814 container create 935883dfbbc79b26b857ffc55c73e28c20c41edd93222bebe7d969bf25e59ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:04:23.087 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 podman[32150]: 2024-09-16 05:04:23.086182399 +0000 UTC m=+0.376244378 container init 935883dfbbc79b26b857ffc55c73e28c20c41edd93222bebe7d969bf25e59ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-16T05:04:23.437 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 podman[32150]: 2024-09-16 05:04:23.089026697 +0000 UTC m=+0.379088676 container start 935883dfbbc79b26b857ffc55c73e28c20c41edd93222bebe7d969bf25e59ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:04:23.437 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-mgr[32170]: -- 172.21.15.196:0/2924523201 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2337bd0e0 con 0x55d23379a800 2024-09-16T05:04:23.437 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 bash[32150]: 935883dfbbc79b26b857ffc55c73e28c20c41edd93222bebe7d969bf25e59ece 2024-09-16T05:04:23.437 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 systemd[1]: Started Ceph mgr.x for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:04:23.437 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:23.220+0000 7f15037d7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T05:04:23.437 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:23.279+0000 7f15037d7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T05:04:23.437 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:23.435+0000 7f15037d7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T05:04:23.604 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:04:23.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_4 2024-09-16T05:04:23.738 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:23.504+0000 7f15037d7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T05:04:23.738 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:23.666+0000 7f15037d7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:04:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:04:24.109 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:23.828+0000 7f15037d7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T05:04:24.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-mon[30802]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:24.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.109 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:24.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:23 smithi032 ceph-mon[27388]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:24.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:23 smithi204 ceph-mon[30933]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:24.409 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:24 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:24.151+0000 7f15037d7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T05:04:24.409 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:24 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:24.214+0000 7f15037d7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T05:04:24.738 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:24 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:24.408+0000 7f15037d7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T05:04:25.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:25.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.601 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.602 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:25 smithi032 ceph-mon[27388]: Reconfiguring daemon mgr.y on smithi032 2024-09-16T05:04:25.738 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:25.295+0000 7f15037d7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T05:04:25.738 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:25.362+0000 7f15037d7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T05:04:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:25 smithi196 ceph-mon[30802]: Reconfiguring daemon mgr.y on smithi032 2024-09-16T05:04:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: Reconfiguring mgr.y (unknown last config time)... 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:25 smithi204 ceph-mon[30933]: Reconfiguring daemon mgr.y on smithi032 2024-09-16T05:04:26.330 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.073+0000 7f15037d7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T05:04:26.330 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.188+0000 7f15037d7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T05:04:26.331 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.329+0000 7f15037d7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:04:26.606 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.389+0000 7f15037d7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T05:04:26.606 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.537+0000 7f15037d7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T05:04:26.606 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.605+0000 7f15037d7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T05:04:26.892 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.664+0000 7f15037d7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T05:04:26.892 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.826+0000 7f15037d7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T05:04:26.892 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.890+0000 7f15037d7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T05:04:27.160 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:26.958+0000 7f15037d7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T05:04:27.160 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:27.042+0000 7f15037d7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T05:04:27.161 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:27.101+0000 7f15037d7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T05:04:27.161 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:27.159+0000 7f15037d7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T05:04:27.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-mon[30802]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:27.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:27.162 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:27.162 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:27.162 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.303 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:26 smithi032 ceph-mon[27388]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:27.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:27.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:27.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:27.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:26 smithi204 ceph-mon[30933]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:27.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:27.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:27.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:27.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:27.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:28.238 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:04:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x[32166]: 2024-09-16T05:04:27.943+0000 7f15037d7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T05:04:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:28 smithi196 ceph-mon[30802]: Standby manager daemon x started 2024-09-16T05:04:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:28 smithi196 ceph-mon[30802]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-16T05:04:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:28 smithi196 ceph-mon[30802]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T05:04:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:28 smithi196 ceph-mon[30802]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-16T05:04:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:28 smithi196 ceph-mon[30802]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T05:04:28.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:28 smithi032 ceph-mon[27388]: Standby manager daemon x started 2024-09-16T05:04:28.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:28 smithi032 ceph-mon[27388]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-16T05:04:28.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:28 smithi032 ceph-mon[27388]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T05:04:28.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:28 smithi032 ceph-mon[27388]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-16T05:04:28.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:28 smithi032 ceph-mon[27388]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T05:04:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:28 smithi204 ceph-mon[30933]: Standby manager daemon x started 2024-09-16T05:04:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:28 smithi204 ceph-mon[30933]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-16T05:04:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:28 smithi204 ceph-mon[30933]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T05:04:28.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:28 smithi204 ceph-mon[30933]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-16T05:04:28.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:28 smithi204 ceph-mon[30933]: from='mgr.? 172.21.15.196:0/2200790615' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T05:04:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:29 smithi032 ceph-mon[27388]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:29 smithi032 ceph-mon[27388]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-16T05:04:29.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T05:04:29.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/467065221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88"}]: dispatch 2024-09-16T05:04:29.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/467065221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88"}]': finished 2024-09-16T05:04:29.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:29 smithi032 ceph-mon[27388]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T05:04:29.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:29.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3149492122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:04:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:29 smithi204 ceph-mon[30933]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:29 smithi204 ceph-mon[30933]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-16T05:04:29.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T05:04:29.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/467065221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88"}]: dispatch 2024-09-16T05:04:29.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/467065221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88"}]': finished 2024-09-16T05:04:29.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:29 smithi204 ceph-mon[30933]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T05:04:29.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:29.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3149492122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:04:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:29 smithi196 ceph-mon[30802]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:29 smithi196 ceph-mon[30802]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-16T05:04:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T05:04:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/467065221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88"}]: dispatch 2024-09-16T05:04:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/467065221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88"}]': finished 2024-09-16T05:04:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:29 smithi196 ceph-mon[30802]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T05:04:29.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:29.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3149492122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:04:31.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:31 smithi032 ceph-mon[27388]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:31 smithi204 ceph-mon[30933]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:31 smithi196 ceph-mon[30802]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:33.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:33 smithi032 ceph-mon[27388]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:33.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:33 smithi204 ceph-mon[30933]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:33.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:33 smithi196 ceph-mon[30802]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:35.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:35 smithi196 ceph-mon[30802]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:35 smithi032 ceph-mon[27388]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:35.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:35 smithi204 ceph-mon[30933]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T05:04:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:36 smithi032 ceph-mon[27388]: Deploying daemon osd.0 on smithi032 2024-09-16T05:04:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T05:04:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:36 smithi204 ceph-mon[30933]: Deploying daemon osd.0 on smithi032 2024-09-16T05:04:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T05:04:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:36 smithi196 ceph-mon[30802]: Deploying daemon osd.0 on smithi032 2024-09-16T05:04:37.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:37 smithi032 ceph-mon[27388]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:37 smithi204 ceph-mon[30933]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:37.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:37 smithi196 ceph-mon[30802]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:39.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:39 smithi032 ceph-mon[27388]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:39.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:39 smithi204 ceph-mon[30933]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:39.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:39 smithi196 ceph-mon[30802]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:41.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:41 smithi032 ceph-mon[27388]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:41 smithi204 ceph-mon[30933]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:41.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:41 smithi196 ceph-mon[30802]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:42.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:42.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:42.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:43 smithi032 ceph-mon[27388]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:43 smithi204 ceph-mon[30933]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:43 smithi196 ceph-mon[30802]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:45.315 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:45 smithi032 ceph-mon[27388]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:45.316 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:45.316 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:45.316 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:45.316 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:45.316 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:45.353 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 0 on host 'smithi032' 2024-09-16T05:04:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:45 smithi204 ceph-mon[30933]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:45 smithi196 ceph-mon[30802]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:46.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:46.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:46.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:46.106 DEBUG:teuthology.orchestra.run.smithi032:osd.0> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.0.service 2024-09-16T05:04:46.109 INFO:tasks.cephadm:Deploying osd.1 on smithi032 with /dev/vg_nvme/lv_3... 2024-09-16T05:04:46.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T05:04:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:46.920 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:04:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0[40713]: 2024-09-16T05:04:46.877+0000 7f603b5f0640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:04:47.543 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:47 smithi032 ceph-mon[27388]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:47.543 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:47 smithi032 ceph-mon[27388]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T05:04:47.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:47 smithi204 ceph-mon[30933]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:47.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:47 smithi204 ceph-mon[30933]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T05:04:47.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:47 smithi196 ceph-mon[30802]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:47.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:47 smithi196 ceph-mon[30802]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T05:04:47.879 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:04:48.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:48 smithi032 ceph-mon[27388]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T05:04:48.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:48 smithi032 ceph-mon[27388]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T05:04:48.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:48 smithi032 ceph-mon[27388]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:04:48.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:48 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:48.636 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:04:48 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0[40713]: 2024-09-16T05:04:48.629+0000 7f6037d87640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T05:04:48.637 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:04:48 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0[40713]: 2024-09-16T05:04:48.632+0000 7f602f393640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:04:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:48 smithi204 ceph-mon[30933]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T05:04:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:48 smithi204 ceph-mon[30933]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T05:04:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:48 smithi204 ceph-mon[30933]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:04:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:48 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:48.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:48 smithi196 ceph-mon[30802]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T05:04:48.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:48 smithi196 ceph-mon[30802]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T05:04:48.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:48 smithi196 ceph-mon[30802]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:04:48.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:48 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:49.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi032 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:04:49.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi032 to 17092M 2024-09-16T05:04:49.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:49.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:49.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:04:49.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:49.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:49.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:49.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:49.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:49.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:49.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:04:49.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T05:04:49.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:49.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:49.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' 2024-09-16T05:04:49.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi032 2024-09-16T05:04:49.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:04:49.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi032 to 17092M 2024-09-16T05:04:49.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:49.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:49.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:04:49.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:49.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:49.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:49.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:49.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:49.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.744 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384]' entity='osd.0' 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi032 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:04:49.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi032 to 17092M 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:04:49.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:04:50.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:50 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:04:50.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:50 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:04:50.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:50.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:50 smithi204 ceph-mon[30933]: osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384] boot 2024-09-16T05:04:50.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:50 smithi204 ceph-mon[30933]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T05:04:50.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:50.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:50 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:04:50.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:50 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:04:50.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:50.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:50 smithi032 ceph-mon[27388]: osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384] boot 2024-09-16T05:04:50.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:50 smithi032 ceph-mon[27388]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T05:04:50.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:50.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:50 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:04:50.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:50 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:04:50.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:50.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:50 smithi196 ceph-mon[30802]: osd.0 [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384] boot 2024-09-16T05:04:50.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:50 smithi196 ceph-mon[30802]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T05:04:50.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T05:04:51.028 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:04:51.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_3 2024-09-16T05:04:51.322 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:04:51.743 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:51 smithi032 ceph-mon[27388]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:51.744 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:51 smithi032 ceph-mon[27388]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T05:04:51.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:51 smithi204 ceph-mon[30933]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:51.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:51 smithi204 ceph-mon[30933]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T05:04:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:51 smithi196 ceph-mon[30802]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T05:04:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:51 smithi196 ceph-mon[30802]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T05:04:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:04:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:04:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:04:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:04:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:04:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:04:52.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:04:53.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:53 smithi032 ceph-mon[27388]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:53.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:53 smithi032 ceph-mon[27388]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:53.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:53 smithi204 ceph-mon[30933]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:53.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:53 smithi204 ceph-mon[30933]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:53 smithi196 ceph-mon[30802]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:04:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:53 smithi196 ceph-mon[30802]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2127571332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4"}]: dispatch 2024-09-16T05:04:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2127571332' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4"}]': finished 2024-09-16T05:04:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:54 smithi032 ceph-mon[27388]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T05:04:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:54 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:04:54.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2127571332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4"}]: dispatch 2024-09-16T05:04:54.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2127571332' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4"}]': finished 2024-09-16T05:04:54.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:54 smithi204 ceph-mon[30933]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T05:04:54.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:54 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:04:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2127571332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4"}]: dispatch 2024-09-16T05:04:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2127571332' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4"}]': finished 2024-09-16T05:04:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:54 smithi196 ceph-mon[30802]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T05:04:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:54 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:04:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:55 smithi032 ceph-mon[27388]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/626885170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:04:55.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:55 smithi204 ceph-mon[30933]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:55.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/626885170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:04:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:55 smithi196 ceph-mon[30802]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/626885170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:04:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:57 smithi032 ceph-mon[27388]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:57.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:57 smithi204 ceph-mon[30933]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:57 smithi196 ceph-mon[30802]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:04:59 smithi032 ceph-mon[27388]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:04:59 smithi204 ceph-mon[30933]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:04:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:04:59 smithi196 ceph-mon[30802]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:01 smithi204 ceph-mon[30933]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T05:05:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:01.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:01 smithi032 ceph-mon[27388]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:01.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T05:05:01.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:01 smithi196 ceph-mon[30802]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T05:05:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:02.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:02 smithi032 ceph-mon[27388]: Deploying daemon osd.1 on smithi032 2024-09-16T05:05:02.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:02 smithi196 ceph-mon[30802]: Deploying daemon osd.1 on smithi032 2024-09-16T05:05:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:02 smithi204 ceph-mon[30933]: Deploying daemon osd.1 on smithi032 2024-09-16T05:05:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:03 smithi032 ceph-mon[27388]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:03 smithi204 ceph-mon[30933]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:03 smithi196 ceph-mon[30802]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:05 smithi196 ceph-mon[30802]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:06.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:05 smithi032 ceph-mon[27388]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:05 smithi204 ceph-mon[30933]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:07.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:07 smithi032 ceph-mon[27388]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:07.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:07 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:07.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:07 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:07.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:07 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:07.941 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:07 smithi204 ceph-mon[30933]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:07.941 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:07 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:07.941 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:07 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:07.941 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:07 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:07 smithi196 ceph-mon[30802]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:07 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:07 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:07 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:09 smithi032 ceph-mon[27388]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:09 smithi196 ceph-mon[30802]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:09 smithi204 ceph-mon[30933]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:11.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:10 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:10 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:10 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:11.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:10 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:11.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:10 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.084 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 1 on host 'smithi032' 2024-09-16T05:05:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:10 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:10 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:10 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:10 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:10 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:10 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:10 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:10 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:10 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:10 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.680 DEBUG:teuthology.orchestra.run.smithi032:osd.1> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.1.service 2024-09-16T05:05:11.682 INFO:tasks.cephadm:Deploying osd.2 on smithi032 with /dev/vg_nvme/lv_2... 2024-09-16T05:05:11.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T05:05:11.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:11 smithi032 ceph-mon[27388]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:11.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:11 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:11.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:11 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:11 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:11.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:11 smithi032 ceph-mon[27388]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T05:05:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:11 smithi204 ceph-mon[30933]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:11 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:11 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:11 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:11 smithi204 ceph-mon[30933]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T05:05:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:11 smithi196 ceph-mon[30802]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:11 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:11 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:11 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:11 smithi196 ceph-mon[30802]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T05:05:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:05:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:13 smithi032 ceph-mon[27388]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:13 smithi032 ceph-mon[27388]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T05:05:13.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:13 smithi032 ceph-mon[27388]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T05:05:13.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:13 smithi032 ceph-mon[27388]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:05:13.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:13 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:13.171 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:13 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[46290]: 2024-09-16T05:05:13.087+0000 7f11156ad640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T05:05:13.171 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:13 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[46290]: 2024-09-16T05:05:13.090+0000 7f11114d5640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:05:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:13 smithi204 ceph-mon[30933]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:13 smithi204 ceph-mon[30933]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T05:05:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:13 smithi204 ceph-mon[30933]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T05:05:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:13 smithi204 ceph-mon[30933]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:05:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:13 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:13 smithi196 ceph-mon[30802]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:13 smithi196 ceph-mon[30802]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T05:05:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:13 smithi196 ceph-mon[30802]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T05:05:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:13 smithi196 ceph-mon[30802]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:05:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:13 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:14.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:14 smithi032 ceph-mon[27388]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:05:14.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:14 smithi032 ceph-mon[27388]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T05:05:14.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:14.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:14 smithi204 ceph-mon[30933]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:05:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:14 smithi204 ceph-mon[30933]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T05:05:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:14 smithi196 ceph-mon[30802]: from='osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:05:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:14 smithi196 ceph-mon[30802]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T05:05:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:15.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:15 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:05:15.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:15 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:05:15.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:15 smithi032 ceph-mon[27388]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:15.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:15.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:15 smithi032 ceph-mon[27388]: osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951] boot 2024-09-16T05:05:15.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:15 smithi032 ceph-mon[27388]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T05:05:15.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:15 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:05:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:15 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:05:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:15 smithi204 ceph-mon[30933]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:15 smithi204 ceph-mon[30933]: osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951] boot 2024-09-16T05:05:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:15 smithi204 ceph-mon[30933]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T05:05:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:15 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:05:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:15 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:05:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:15 smithi196 ceph-mon[30802]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T05:05:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:15 smithi196 ceph-mon[30802]: osd.1 [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951] boot 2024-09-16T05:05:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:15 smithi196 ceph-mon[30802]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T05:05:15.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:05:16.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi032 to 8546M 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:16.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:16.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:16.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:16.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:16.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:16.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi032 to 8546M 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T05:05:16.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:16.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:16.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi032 to 8546M 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:16.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:17.009 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:05:17.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_2 2024-09-16T05:05:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:05:17.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:17 smithi032 ceph-mon[27388]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:17.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:05:17.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T05:05:17.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:17 smithi032 ceph-mon[27388]: osdmap e15: 2 total, 2 up, 2 in 2024-09-16T05:05:17.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:05:17.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:17 smithi196 ceph-mon[30802]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:17.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:05:17.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T05:05:17.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:17 smithi196 ceph-mon[30802]: osdmap e15: 2 total, 2 up, 2 in 2024-09-16T05:05:17.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:05:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:17 smithi204 ceph-mon[30933]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:05:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T05:05:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:17 smithi204 ceph-mon[30933]: osdmap e15: 2 total, 2 up, 2 in 2024-09-16T05:05:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:05:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T05:05:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:18 smithi032 ceph-mon[27388]: osdmap e16: 2 total, 2 up, 2 in 2024-09-16T05:05:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:05:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:05:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T05:05:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:18 smithi204 ceph-mon[30933]: osdmap e16: 2 total, 2 up, 2 in 2024-09-16T05:05:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:05:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:05:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T05:05:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:18 smithi196 ceph-mon[30802]: osdmap e16: 2 total, 2 up, 2 in 2024-09-16T05:05:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:05:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:05:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:19.591 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:05:19 smithi032 sudo[50525]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T05:05:19.592 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:05:19 smithi032 sudo[50525]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:19.592 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:05:19 smithi032 sudo[50525]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:19.592 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:05:19 smithi032 sudo[50525]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:19 smithi032 ceph-mon[27388]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:05:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:19 smithi032 ceph-mon[27388]: pgmap v62: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:19 smithi032 ceph-mon[27388]: osdmap e17: 2 total, 2 up, 2 in 2024-09-16T05:05:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:19 smithi204 ceph-mon[30933]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:05:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:19 smithi204 ceph-mon[30933]: pgmap v62: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:19 smithi204 ceph-mon[30933]: osdmap e17: 2 total, 2 up, 2 in 2024-09-16T05:05:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:19 smithi196 ceph-mon[30802]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:05:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:19 smithi196 ceph-mon[30802]: pgmap v62: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:19 smithi196 ceph-mon[30802]: osdmap e17: 2 total, 2 up, 2 in 2024-09-16T05:05:20.420 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:05:19 smithi032 sudo[50545]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T05:05:20.420 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:05:19 smithi032 sudo[50545]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:20.420 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:05:19 smithi032 sudo[50545]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:20.420 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:05:19 smithi032 sudo[50545]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:21.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50633]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T05:05:21.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50633]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:21.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50633]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:21.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50633]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4232608942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70"}]: dispatch 2024-09-16T05:05:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4232608942' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70"}]': finished 2024-09-16T05:05:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:20 smithi032 ceph-mon[27388]: osdmap e18: 3 total, 2 up, 3 in 2024-09-16T05:05:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4232608942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70"}]: dispatch 2024-09-16T05:05:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4232608942' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70"}]': finished 2024-09-16T05:05:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:20 smithi204 ceph-mon[30933]: osdmap e18: 3 total, 2 up, 3 in 2024-09-16T05:05:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4232608942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70"}]: dispatch 2024-09-16T05:05:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4232608942' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70"}]': finished 2024-09-16T05:05:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:20 smithi196 ceph-mon[30802]: osdmap e18: 3 total, 2 up, 3 in 2024-09-16T05:05:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:21.900 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50660]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T05:05:21.900 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50660]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:21.900 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50660]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:21.900 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50660]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50666]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T05:05:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50666]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50666]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:21 smithi032 sudo[50666]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:22.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 ceph-mon[30933]: pgmap v64: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:22.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 ceph-mon[30933]: mgrmap e15: y(active, since 114s), standbys: x 2024-09-16T05:05:22.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/582647716' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:05:22.625 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:22.625 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 sudo[32860]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T05:05:22.625 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 sudo[32860]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:22.625 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 sudo[32860]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:22.625 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 sudo[32860]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:22.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:22 smithi032 sudo[51035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T05:05:22.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:22 smithi032 sudo[51035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:22.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:22 smithi032 sudo[51035]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:22.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:22 smithi032 sudo[51035]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:22.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:22 smithi032 ceph-mon[27388]: pgmap v64: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:22.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:22 smithi032 ceph-mon[27388]: mgrmap e15: y(active, since 114s), standbys: x 2024-09-16T05:05:22.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/582647716' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:05:22.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:22 smithi032 ceph-mon[27388]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:22.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:22 smithi196 ceph-mon[30802]: pgmap v64: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:22.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:22 smithi196 ceph-mon[30802]: mgrmap e15: y(active, since 114s), standbys: x 2024-09-16T05:05:22.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/582647716' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:05:22.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:22 smithi196 ceph-mon[30802]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:22.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 sudo[32863]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T05:05:22.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 sudo[32863]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:22.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 sudo[32863]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:22.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:22 smithi204 sudo[32863]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:23.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 sudo[32838]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T05:05:23.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 sudo[32838]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:23.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 sudo[32838]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:23.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 sudo[32838]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:23.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:23.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:23.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:23.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:23.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:23.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:23.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:23.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:23.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:23.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 sudo[32841]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T05:05:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 sudo[32841]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T05:05:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 sudo[32841]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T05:05:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 sudo[32841]: pam_unix(sudo:session): session closed for user root 2024-09-16T05:05:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:23.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:23.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:23.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:23.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:23.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:24.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:24 smithi032 ceph-mon[27388]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:24.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:24 smithi032 ceph-mon[27388]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:24.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:24 smithi204 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:24 smithi204 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:24.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:24 smithi196 ceph-mon[30802]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T05:05:24.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:24 smithi196 ceph-mon[30802]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T05:05:24.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:25 smithi196 ceph-mon[30802]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:25.744 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:25 smithi032 ceph-mon[27388]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:25 smithi204 ceph-mon[30933]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:27.614 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:27 smithi032 ceph-mon[27388]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 5.1 KiB/s rd, 231 KiB/s wr, 16 op/s 2024-09-16T05:05:27.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:27 smithi196 ceph-mon[30802]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 5.1 KiB/s rd, 231 KiB/s wr, 16 op/s 2024-09-16T05:05:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:27 smithi204 ceph-mon[30933]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 5.1 KiB/s rd, 231 KiB/s wr, 16 op/s 2024-09-16T05:05:28.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T05:05:28.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:28.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:28.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:28.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: Deploying daemon osd.2 on smithi032 2024-09-16T05:05:28.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:28.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:28.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:28.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T05:05:28.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:28.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:28.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:28.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: Deploying daemon osd.2 on smithi032 2024-09-16T05:05:28.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:28.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:28.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:28.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T05:05:28.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:28.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:28.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:28.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: Deploying daemon osd.2 on smithi032 2024-09-16T05:05:28.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:28.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:28.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:28.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:29 smithi032 ceph-mon[27388]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 3.8 KiB/s rd, 171 KiB/s wr, 12 op/s 2024-09-16T05:05:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:29 smithi196 ceph-mon[30802]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 3.8 KiB/s rd, 171 KiB/s wr, 12 op/s 2024-09-16T05:05:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:29 smithi204 ceph-mon[30933]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 3.8 KiB/s rd, 171 KiB/s wr, 12 op/s 2024-09-16T05:05:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:31 smithi196 ceph-mon[30802]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 3 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-09-16T05:05:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:31 smithi032 ceph-mon[27388]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 3 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-09-16T05:05:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:31 smithi204 ceph-mon[30933]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 3 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-09-16T05:05:33.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:33 smithi032 ceph-mon[27388]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.5 KiB/s rd, 112 KiB/s wr, 7 op/s 2024-09-16T05:05:33.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:33 smithi196 ceph-mon[30802]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.5 KiB/s rd, 112 KiB/s wr, 7 op/s 2024-09-16T05:05:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:33 smithi204 ceph-mon[30933]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.5 KiB/s rd, 112 KiB/s wr, 7 op/s 2024-09-16T05:05:34.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:34.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:34.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:34.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:34.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:34.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:35.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:35.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:35.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:35.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:35 smithi032 ceph-mon[27388]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.4 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-16T05:05:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:35 smithi196 ceph-mon[30802]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.4 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-16T05:05:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:35 smithi204 ceph-mon[30933]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.4 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-16T05:05:36.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:36.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:36.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:36.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:36.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:36.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:36.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:36.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:36.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:36.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:36.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:36.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:37.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:37.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:37.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:05:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:05:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:05:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.312 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 2 on host 'smithi032' 2024-09-16T05:05:37.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:37 smithi032 ceph-mon[27388]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.4 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-16T05:05:37.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:37 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:37.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:37 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:37 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.929 DEBUG:teuthology.orchestra.run.smithi032:osd.2> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.2.service 2024-09-16T05:05:37.932 INFO:tasks.cephadm:Deploying osd.3 on smithi032 with /dev/vg_nvme/lv_1... 2024-09-16T05:05:37.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T05:05:37.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:37 smithi196 ceph-mon[30802]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.4 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-16T05:05:37.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:37 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:37.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:37 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:37.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:37 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:37 smithi204 ceph-mon[30933]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail; 2.4 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-16T05:05:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:37 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:05:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:37 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:37 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:38.573 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:05:38.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:38 smithi032 ceph-mon[27388]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T05:05:38.917 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:05:38 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[52308]: 2024-09-16T05:05:38.500+0000 7f5c7cb21640 -1 osd.2 0 log_to_monitors true 2024-09-16T05:05:38.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:38 smithi196 ceph-mon[30802]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T05:05:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:38 smithi204 ceph-mon[30933]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T05:05:39.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:39 smithi032 ceph-mon[27388]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:39.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:39 smithi032 ceph-mon[27388]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T05:05:39.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:39 smithi032 ceph-mon[27388]: osdmap e19: 3 total, 2 up, 3 in 2024-09-16T05:05:39.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:39 smithi032 ceph-mon[27388]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:05:39.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:39 smithi196 ceph-mon[30802]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:39 smithi196 ceph-mon[30802]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T05:05:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:39 smithi196 ceph-mon[30802]: osdmap e19: 3 total, 2 up, 3 in 2024-09-16T05:05:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:39 smithi196 ceph-mon[30802]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:05:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:39 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:40.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:39 smithi204 ceph-mon[30933]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:39 smithi204 ceph-mon[30933]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T05:05:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:39 smithi204 ceph-mon[30933]: osdmap e19: 3 total, 2 up, 3 in 2024-09-16T05:05:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:39 smithi204 ceph-mon[30933]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:05:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:39 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:40.206 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:05:39 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[52308]: 2024-09-16T05:05:39.995+0000 7f5c78aa4640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T05:05:40.206 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:05:39 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[52308]: 2024-09-16T05:05:39.997+0000 7f5c708c4640 -1 osd.2 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:05:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:40 smithi032 ceph-mon[27388]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:05:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:40 smithi032 ceph-mon[27388]: osdmap e20: 3 total, 2 up, 3 in 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:40 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:40 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:40 smithi032 ceph-mon[27388]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:40 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:40 smithi204 ceph-mon[30933]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:40 smithi204 ceph-mon[30933]: osdmap e20: 3 total, 2 up, 3 in 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:40 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:40 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:40 smithi204 ceph-mon[30933]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' 2024-09-16T05:05:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:40 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:40 smithi196 ceph-mon[30802]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:05:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:40 smithi196 ceph-mon[30802]: osdmap e20: 3 total, 2 up, 3 in 2024-09-16T05:05:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:40 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:40 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:40 smithi196 ceph-mon[30802]: from='osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664]' entity='osd.2' 2024-09-16T05:05:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:40 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:05:41.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:05:41.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:41.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664] boot 2024-09-16T05:05:41.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: osdmap e21: 3 total, 3 up, 3 in 2024-09-16T05:05:41.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:41.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:41.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:41.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:41.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:41.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:41.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:41.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:41.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:41.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:05:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:05:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664] boot 2024-09-16T05:05:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: osdmap e21: 3 total, 3 up, 3 in 2024-09-16T05:05:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:05:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:05:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: osd.2 [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664] boot 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: osdmap e21: 3 total, 3 up, 3 in 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:05:42.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:05:42.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:42 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi032 2024-09-16T05:05:42.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:42 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi032 to 5697M 2024-09-16T05:05:42.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:42 smithi032 ceph-mon[27388]: osdmap e22: 3 total, 3 up, 3 in 2024-09-16T05:05:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:05:42.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_1 2024-09-16T05:05:43.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:42 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi032 2024-09-16T05:05:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:42 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi032 to 5697M 2024-09-16T05:05:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:42 smithi204 ceph-mon[30933]: osdmap e22: 3 total, 3 up, 3 in 2024-09-16T05:05:43.205 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:05:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:42 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi032 2024-09-16T05:05:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:42 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi032 to 5697M 2024-09-16T05:05:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:42 smithi196 ceph-mon[30802]: osdmap e22: 3 total, 3 up, 3 in 2024-09-16T05:05:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:43 smithi032 ceph-mon[27388]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:43 smithi204 ceph-mon[30933]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:43 smithi196 ceph-mon[30802]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:44.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:44 smithi032 ceph-mon[27388]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:05:44.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:05:44.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:05:44.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:44 smithi204 ceph-mon[30933]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:05:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:05:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:05:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:44 smithi196 ceph-mon[30802]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:05:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:05:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:05:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:45 smithi032 ceph-mon[27388]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:45 smithi204 ceph-mon[30933]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:45 smithi196 ceph-mon[30802]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1858551008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71705a06-9102-4b97-be6a-4989306666d2"}]: dispatch 2024-09-16T05:05:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1858551008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71705a06-9102-4b97-be6a-4989306666d2"}]': finished 2024-09-16T05:05:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:46 smithi196 ceph-mon[30802]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T05:05:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:05:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1858551008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71705a06-9102-4b97-be6a-4989306666d2"}]: dispatch 2024-09-16T05:05:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1858551008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71705a06-9102-4b97-be6a-4989306666d2"}]': finished 2024-09-16T05:05:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:46 smithi032 ceph-mon[27388]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T05:05:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:05:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1858551008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71705a06-9102-4b97-be6a-4989306666d2"}]: dispatch 2024-09-16T05:05:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1858551008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71705a06-9102-4b97-be6a-4989306666d2"}]': finished 2024-09-16T05:05:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:46 smithi204 ceph-mon[30933]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T05:05:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:05:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:47 smithi032 ceph-mon[27388]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3395942419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:05:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:47 smithi204 ceph-mon[30933]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3395942419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:05:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:47 smithi196 ceph-mon[30802]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3395942419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:05:50.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:49 smithi032 ceph-mon[27388]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:49 smithi204 ceph-mon[30933]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:49 smithi196 ceph-mon[30802]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:51 smithi032 ceph-mon[27388]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:51 smithi204 ceph-mon[30933]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:51 smithi196 ceph-mon[30802]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:53.792 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:53 smithi032 ceph-mon[27388]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:53.793 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T05:05:53.793 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:53 smithi204 ceph-mon[30933]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T05:05:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:53 smithi196 ceph-mon[30802]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T05:05:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:05:55.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:54 smithi032 ceph-mon[27388]: Deploying daemon osd.3 on smithi032 2024-09-16T05:05:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:54 smithi204 ceph-mon[30933]: Deploying daemon osd.3 on smithi032 2024-09-16T05:05:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:54 smithi196 ceph-mon[30802]: Deploying daemon osd.3 on smithi032 2024-09-16T05:05:56.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:55 smithi032 ceph-mon[27388]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:55 smithi204 ceph-mon[30933]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:55 smithi196 ceph-mon[30802]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:58.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:57 smithi032 ceph-mon[27388]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:57 smithi204 ceph-mon[30933]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:05:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:57 smithi196 ceph-mon[30802]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:00.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:59 smithi032 ceph-mon[27388]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:00.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:59 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:00.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:59 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:00.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:05:59 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:59 smithi204 ceph-mon[30933]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:59 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:59 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:05:59 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:59 smithi196 ceph-mon[30802]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:59 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:59 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:05:59 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:02.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:01 smithi032 ceph-mon[27388]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:02.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:02.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:02.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:01 smithi196 ceph-mon[30802]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:02.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:02.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:02.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:01 smithi204 ceph-mon[30933]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:02.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:02.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:02.439 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 3 on host 'smithi032' 2024-09-16T05:06:03.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:03.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:03.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:03.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.057 DEBUG:teuthology.orchestra.run.smithi032:osd.3> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.3.service 2024-09-16T05:06:03.060 INFO:tasks.cephadm:Deploying osd.4 on smithi196 with /dev/vg_nvme/lv_4... 2024-09-16T05:06:03.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T05:06:03.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:03.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:03.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:03.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:03.312 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:06:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:03 smithi032 ceph-mon[27388]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:03 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:06:03.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:03 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:06:03.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:03 smithi032 ceph-mon[27388]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T05:06:03.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:03 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:04.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:03 smithi196 ceph-mon[30802]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:04.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:03 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:06:04.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:03 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:06:04.135 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:03 smithi196 ceph-mon[30802]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T05:06:04.136 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:03 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:03 smithi204 ceph-mon[30933]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:03 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:06:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:03 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:06:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:03 smithi204 ceph-mon[30933]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T05:06:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:03 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:04.350 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:06:04 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:06:04.004+0000 7f5a3399d640 -1 osd.3 0 log_to_monitors true 2024-09-16T05:06:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:04 smithi032 ceph-mon[27388]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T05:06:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:04 smithi196 ceph-mon[30802]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T05:06:05.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:04 smithi204 ceph-mon[30933]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T05:06:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:06:05.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-09-16T05:06:06.109 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:06:06.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:05 smithi032 ceph-mon[27388]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:06.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:05 smithi032 ceph-mon[27388]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T05:06:06.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:05 smithi032 ceph-mon[27388]: osdmap e25: 4 total, 3 up, 4 in 2024-09-16T05:06:06.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:05 smithi032 ceph-mon[27388]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:06:06.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:05 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:06.171 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:06:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:06:05.803+0000 7f5a2f920640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T05:06:06.171 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:06:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:06:05.810+0000 7f5a2b748640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:06:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:05 smithi204 ceph-mon[30933]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:05 smithi204 ceph-mon[30933]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T05:06:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:05 smithi204 ceph-mon[30933]: osdmap e25: 4 total, 3 up, 4 in 2024-09-16T05:06:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:05 smithi204 ceph-mon[30933]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:06:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:05 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:06.227 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:05 smithi196 ceph-mon[30802]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:06.227 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:05 smithi196 ceph-mon[30802]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T05:06:06.228 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:05 smithi196 ceph-mon[30802]: osdmap e25: 4 total, 3 up, 4 in 2024-09-16T05:06:06.228 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:05 smithi196 ceph-mon[30802]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:06:06.228 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:05 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:06:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: osdmap e26: 4 total, 3 up, 4 in 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:06:06.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi032 to 4273M 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:06.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: osdmap e26: 4 total, 3 up, 4 in 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi032 to 4273M 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:07.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:07.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:07.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:07.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:07.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:07.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: osdmap e26: 4 total, 3 up, 4 in 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi032 to 4273M 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:07.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:07.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:07.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:07.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:07.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:07.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:08.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:07 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:06:08.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:07 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:06:08.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:07 smithi196 ceph-mon[30802]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:08.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:07 smithi196 ceph-mon[30802]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:06:08.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:07 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:08.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:07 smithi196 ceph-mon[30802]: osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358] boot 2024-09-16T05:06:08.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:07 smithi196 ceph-mon[30802]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T05:06:08.134 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:07 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:07 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:06:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:07 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:06:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:07 smithi032 ceph-mon[27388]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:07 smithi032 ceph-mon[27388]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:06:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:07 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:07 smithi032 ceph-mon[27388]: osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358] boot 2024-09-16T05:06:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:07 smithi032 ceph-mon[27388]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T05:06:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:07 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:07 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:06:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:07 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:06:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:07 smithi204 ceph-mon[30933]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-16T05:06:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:07 smithi204 ceph-mon[30933]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:06:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:07 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:07 smithi204 ceph-mon[30933]: osd.3 [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358] boot 2024-09-16T05:06:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:07 smithi204 ceph-mon[30933]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T05:06:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:07 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T05:06:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:08 smithi032 ceph-mon[27388]: osdmap e28: 4 total, 4 up, 4 in 2024-09-16T05:06:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:08 smithi204 ceph-mon[30933]: osdmap e28: 4 total, 4 up, 4 in 2024-09-16T05:06:09.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:08 smithi196 ceph-mon[30802]: osdmap e28: 4 total, 4 up, 4 in 2024-09-16T05:06:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:09 smithi032 ceph-mon[27388]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/2215005541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]: dispatch 2024-09-16T05:06:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:09 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]: dispatch 2024-09-16T05:06:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:09 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]': finished 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:09 smithi032 ceph-mon[27388]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/1823656977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:09 smithi204 ceph-mon[30933]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/2215005541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]: dispatch 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:09 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]: dispatch 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:09 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]': finished 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:09 smithi204 ceph-mon[30933]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/1823656977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:06:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:09 smithi196 ceph-mon[30802]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/2215005541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]: dispatch 2024-09-16T05:06:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:09 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]: dispatch 2024-09-16T05:06:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:09 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4"}]': finished 2024-09-16T05:06:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:09 smithi196 ceph-mon[30802]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T05:06:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/1823656977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:06:12.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:11 smithi032 ceph-mon[27388]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:11 smithi204 ceph-mon[30933]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:11 smithi196 ceph-mon[30802]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:14.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:13 smithi032 ceph-mon[27388]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:13 smithi204 ceph-mon[30933]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:14.212 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:13 smithi196 ceph-mon[30802]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:15.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:14 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:06:15.134 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:14 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:06:15.134 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:14 smithi196 ceph-mon[30802]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T05:06:15.134 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:14 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:06:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:14 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:06:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:14 smithi032 ceph-mon[27388]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T05:06:15.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:14 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:06:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:14 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:06:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:14 smithi204 ceph-mon[30933]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T05:06:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:15 smithi196 ceph-mon[30802]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:15 smithi032 ceph-mon[27388]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:16.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:15 smithi204 ceph-mon[30933]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:17.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T05:06:17.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T05:06:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T05:06:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:18.090 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:17 smithi196 ceph-mon[30802]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:18.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:17 smithi196 ceph-mon[30802]: Deploying daemon osd.4 on smithi196 2024-09-16T05:06:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:17 smithi032 ceph-mon[27388]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:17 smithi032 ceph-mon[27388]: Deploying daemon osd.4 on smithi196 2024-09-16T05:06:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:17 smithi204 ceph-mon[30933]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:17 smithi204 ceph-mon[30933]: Deploying daemon osd.4 on smithi196 2024-09-16T05:06:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:19 smithi196 ceph-mon[30802]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:20.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:19 smithi032 ceph-mon[27388]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:19 smithi204 ceph-mon[30933]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:21 smithi196 ceph-mon[30802]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:21 smithi032 ceph-mon[27388]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:21 smithi204 ceph-mon[30933]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:23.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:23.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:23.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:23.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:23.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:23 smithi196 ceph-mon[30802]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:23 smithi032 ceph-mon[27388]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:23 smithi204 ceph-mon[30933]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:25.834 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 4 on host 'smithi196' 2024-09-16T05:06:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:26.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:26.449 DEBUG:teuthology.orchestra.run.smithi196:osd.4> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.4.service 2024-09-16T05:06:26.451 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_3... 2024-09-16T05:06:26.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T05:06:27.488 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:06:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:06:27.175+0000 7fe42f8ed640 -1 osd.4 0 log_to_monitors true 2024-09-16T05:06:27.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:27 smithi196 ceph-mon[30802]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:27.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:27 smithi196 ceph-mon[30802]: from='osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T05:06:27.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:27 smithi196 ceph-mon[30802]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T05:06:28.097 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:06:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:27 smithi032 ceph-mon[27388]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:27 smithi032 ceph-mon[27388]: from='osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T05:06:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:27 smithi032 ceph-mon[27388]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T05:06:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:27 smithi204 ceph-mon[30933]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:27 smithi204 ceph-mon[30933]: from='osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T05:06:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:27 smithi204 ceph-mon[30933]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T05:06:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:28 smithi032 ceph-mon[27388]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T05:06:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:28 smithi032 ceph-mon[27388]: osdmap e31: 5 total, 4 up, 5 in 2024-09-16T05:06:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:28 smithi032 ceph-mon[27388]: from='osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:28 smithi032 ceph-mon[27388]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:28 smithi204 ceph-mon[30933]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T05:06:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:28 smithi204 ceph-mon[30933]: osdmap e31: 5 total, 4 up, 5 in 2024-09-16T05:06:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:28 smithi204 ceph-mon[30933]: from='osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:28 smithi204 ceph-mon[30933]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:29.238 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:06:28 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:06:28.880+0000 7fe42b870640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T05:06:29.238 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:06:28 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:06:28.883+0000 7fe427698640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:06:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:28 smithi196 ceph-mon[30802]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T05:06:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:28 smithi196 ceph-mon[30802]: osdmap e31: 5 total, 4 up, 5 in 2024-09-16T05:06:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:28 smithi196 ceph-mon[30802]: from='osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:29.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:28 smithi196 ceph-mon[30802]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:29.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:29.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:06:29.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: osdmap e32: 5 total, 4 up, 5 in 2024-09-16T05:06:29.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:29.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:29.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:06:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: osdmap e32: 5 total, 4 up, 5 in 2024-09-16T05:06:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: osdmap e32: 5 total, 4 up, 5 in 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:30.986 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:06:31.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:30 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:06:31.010 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:30 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:06:31.010 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:30 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi196 2024-09-16T05:06:31.010 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:30 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-16T05:06:31.010 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:30 smithi196 ceph-mon[30802]: osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655] boot 2024-09-16T05:06:31.010 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:30 smithi196 ceph-mon[30802]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T05:06:31.010 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:31.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-09-16T05:06:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:30 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:06:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:30 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:06:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:30 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi196 2024-09-16T05:06:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:30 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:30 smithi032 ceph-mon[27388]: osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655] boot 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:30 smithi032 ceph-mon[27388]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:30 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:30 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:30 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi196 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:30 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi196 to 17088M 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:30 smithi204 ceph-mon[30933]: osd.4 [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655] boot 2024-09-16T05:06:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:30 smithi204 ceph-mon[30933]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T05:06:31.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T05:06:31.252 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:06:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:31 smithi196 ceph-mon[30802]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:31 smithi196 ceph-mon[30802]: osdmap e34: 5 total, 5 up, 5 in 2024-09-16T05:06:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:31 smithi032 ceph-mon[27388]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:31 smithi032 ceph-mon[27388]: osdmap e34: 5 total, 5 up, 5 in 2024-09-16T05:06:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:32.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:32.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:31 smithi204 ceph-mon[30933]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-16T05:06:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:31 smithi204 ceph-mon[30933]: osdmap e34: 5 total, 5 up, 5 in 2024-09-16T05:06:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:32 smithi032 ceph-mon[27388]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:06:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:32 smithi204 ceph-mon[30933]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:06:33.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:32 smithi196 ceph-mon[30802]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:06:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:33 smithi032 ceph-mon[27388]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:33 smithi204 ceph-mon[30933]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:33 smithi196 ceph-mon[30802]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/2317559762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]: dispatch 2024-09-16T05:06:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:34 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]: dispatch 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:34 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]': finished 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:34 smithi032 ceph-mon[27388]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/2052559673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/2317559762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]: dispatch 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:34 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]: dispatch 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:34 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]': finished 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:34 smithi204 ceph-mon[30933]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T05:06:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/2052559673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:06:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/2317559762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]: dispatch 2024-09-16T05:06:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:34 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]: dispatch 2024-09-16T05:06:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:34 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59"}]': finished 2024-09-16T05:06:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:34 smithi196 ceph-mon[30802]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T05:06:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/2052559673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:06:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:35 smithi196 ceph-mon[30802]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:35 smithi032 ceph-mon[27388]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:35 smithi204 ceph-mon[30933]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:37 smithi204 ceph-mon[30933]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:37 smithi032 ceph-mon[27388]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:37 smithi196 ceph-mon[30802]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:39 smithi196 ceph-mon[30802]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:39 smithi032 ceph-mon[27388]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:39 smithi204 ceph-mon[30933]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:41 smithi032 ceph-mon[27388]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:41 smithi032 ceph-mon[27388]: 1.0 deep-scrub starts 2024-09-16T05:06:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:41 smithi032 ceph-mon[27388]: 1.0 deep-scrub ok 2024-09-16T05:06:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T05:06:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:41 smithi204 ceph-mon[30933]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:41 smithi204 ceph-mon[30933]: 1.0 deep-scrub starts 2024-09-16T05:06:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:41 smithi204 ceph-mon[30933]: 1.0 deep-scrub ok 2024-09-16T05:06:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T05:06:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:41 smithi196 ceph-mon[30802]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:41 smithi196 ceph-mon[30802]: 1.0 deep-scrub starts 2024-09-16T05:06:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:41 smithi196 ceph-mon[30802]: 1.0 deep-scrub ok 2024-09-16T05:06:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T05:06:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:43.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:42 smithi196 ceph-mon[30802]: Deploying daemon osd.5 on smithi196 2024-09-16T05:06:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:42 smithi032 ceph-mon[27388]: Deploying daemon osd.5 on smithi196 2024-09-16T05:06:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:42 smithi204 ceph-mon[30933]: Deploying daemon osd.5 on smithi196 2024-09-16T05:06:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:43 smithi032 ceph-mon[27388]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:43 smithi204 ceph-mon[30933]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:43 smithi196 ceph-mon[30802]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:45 smithi196 ceph-mon[30802]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:46.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:45 smithi032 ceph-mon[27388]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:45 smithi204 ceph-mon[30933]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:48.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:47 smithi196 ceph-mon[30802]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:48.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:47 smithi032 ceph-mon[27388]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:48.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:47 smithi204 ceph-mon[30933]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:49.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:48 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:49.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:48 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:49.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:48 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:49.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:48 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:49.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:48 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:49.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:48 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:48 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:48 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:48 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:49 smithi196 ceph-mon[30802]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:50.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:49 smithi032 ceph-mon[27388]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:49 smithi204 ceph-mon[30933]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:51.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:51 smithi196 ceph-mon[30802]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:51.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:51 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:51.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:51 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:51.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:51 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:51.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:51 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:51.861 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:51 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:51.900 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-09-16T05:06:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:51 smithi032 ceph-mon[27388]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:51.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:51 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:51.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:51 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:51.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:51 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:51.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:51 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:51.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:51 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:51.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:51 smithi204 ceph-mon[30933]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:51.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:51 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:51.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:51 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:51.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:51 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:51.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:51 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:51.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:51 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:52.469 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.5.service 2024-09-16T05:06:52.472 INFO:tasks.cephadm:Deploying osd.6 on smithi196 with /dev/vg_nvme/lv_2... 2024-09-16T05:06:52.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T05:06:52.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:52.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:52.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:06:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:53.577 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:06:53 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:06:53.265+0000 7f72706ba640 -1 osd.5 0 log_to_monitors true 2024-09-16T05:06:53.632 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:06:54.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:53 smithi196 ceph-mon[30802]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:54.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:53 smithi196 ceph-mon[30802]: from='osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T05:06:54.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:53 smithi196 ceph-mon[30802]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T05:06:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:53 smithi032 ceph-mon[27388]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:53 smithi032 ceph-mon[27388]: from='osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T05:06:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:53 smithi032 ceph-mon[27388]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T05:06:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:53 smithi204 ceph-mon[30933]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:53 smithi204 ceph-mon[30933]: from='osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T05:06:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:53 smithi204 ceph-mon[30933]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T05:06:55.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:54 smithi196 ceph-mon[30802]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T05:06:55.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:54 smithi196 ceph-mon[30802]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T05:06:55.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:54 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:55.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:54 smithi196 ceph-mon[30802]: from='osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:55.071 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:54 smithi196 ceph-mon[30802]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:55.072 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:06:54 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:06:54.915+0000 7f726c63d640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T05:06:55.072 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:06:54 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:06:54.919+0000 7f7268465640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:06:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:54 smithi032 ceph-mon[27388]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T05:06:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:54 smithi032 ceph-mon[27388]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T05:06:55.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:54 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:55.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:54 smithi032 ceph-mon[27388]: from='osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:55.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:54 smithi032 ceph-mon[27388]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:54 smithi204 ceph-mon[30933]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T05:06:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:54 smithi204 ceph-mon[30933]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T05:06:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:54 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:54 smithi204 ceph-mon[30933]: from='osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:54 smithi204 ceph-mon[30933]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:06:56.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:56.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: osdmap e37: 6 total, 5 up, 6 in 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128] boot 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:56.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:06:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: osdmap e37: 6 total, 5 up, 6 in 2024-09-16T05:06:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128] boot 2024-09-16T05:06:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-16T05:06:56.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:06:56.210 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: osdmap e37: 6 total, 5 up, 6 in 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: osd.5 [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128] boot 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:06:56.211 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:06:56.983 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:06:57.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-09-16T05:06:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:56 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:06:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:56 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:06:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:56 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi196 2024-09-16T05:06:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:56 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-16T05:06:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:56 smithi196 ceph-mon[30802]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T05:06:57.253 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:06:57.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:56 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:56 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:56 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi196 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:56 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:56 smithi032 ceph-mon[27388]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:56 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:56 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:56 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi196 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:56 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi196 to 8544M 2024-09-16T05:06:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:56 smithi204 ceph-mon[30933]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T05:06:57.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:57 smithi196 ceph-mon[30802]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:06:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:57 smithi196 ceph-mon[30802]: osdmap e40: 6 total, 6 up, 6 in 2024-09-16T05:06:58.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:57 smithi032 ceph-mon[27388]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:06:58.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:57 smithi032 ceph-mon[27388]: osdmap e40: 6 total, 6 up, 6 in 2024-09-16T05:06:58.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:57 smithi204 ceph-mon[30933]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:06:58.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:57 smithi204 ceph-mon[30933]: osdmap e40: 6 total, 6 up, 6 in 2024-09-16T05:06:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:58 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:06:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:58 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:06:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:58 smithi196 ceph-mon[30802]: osdmap e41: 6 total, 6 up, 6 in 2024-09-16T05:06:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:58 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:06:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:58 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:06:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:58 smithi032 ceph-mon[27388]: osdmap e41: 6 total, 6 up, 6 in 2024-09-16T05:06:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:06:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:06:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:06:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:58 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:06:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:58 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:06:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:58 smithi204 ceph-mon[30933]: osdmap e41: 6 total, 6 up, 6 in 2024-09-16T05:07:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:59 smithi196 ceph-mon[30802]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:59 smithi196 ceph-mon[30802]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:06:59 smithi196 ceph-mon[30802]: osdmap e42: 6 total, 6 up, 6 in 2024-09-16T05:07:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:59 smithi032 ceph-mon[27388]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:59 smithi032 ceph-mon[27388]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:06:59 smithi032 ceph-mon[27388]: osdmap e42: 6 total, 6 up, 6 in 2024-09-16T05:07:00.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:59 smithi204 ceph-mon[30933]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:00.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:59 smithi204 ceph-mon[30933]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:00.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:06:59 smithi204 ceph-mon[30933]: osdmap e42: 6 total, 6 up, 6 in 2024-09-16T05:07:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/978195812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]: dispatch 2024-09-16T05:07:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:00 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]: dispatch 2024-09-16T05:07:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:00 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]': finished 2024-09-16T05:07:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:00 smithi196 ceph-mon[30802]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T05:07:01.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:00 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:01.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/1295772569' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/978195812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]: dispatch 2024-09-16T05:07:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:00 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]: dispatch 2024-09-16T05:07:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:00 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]': finished 2024-09-16T05:07:01.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:00 smithi032 ceph-mon[27388]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T05:07:01.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:00 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:01.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/1295772569' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/978195812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]: dispatch 2024-09-16T05:07:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:00 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]: dispatch 2024-09-16T05:07:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:00 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca"}]': finished 2024-09-16T05:07:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:00 smithi204 ceph-mon[30933]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T05:07:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:01.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/1295772569' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:01 smithi196 ceph-mon[30802]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:02.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:01 smithi032 ceph-mon[27388]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:02.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:01 smithi204 ceph-mon[30933]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:03 smithi196 ceph-mon[30802]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:03 smithi032 ceph-mon[27388]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:03 smithi204 ceph-mon[30933]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:04 smithi196 ceph-mon[30802]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:05.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:04 smithi204 ceph-mon[30933]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:05.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:04 smithi032 ceph-mon[27388]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:07.393 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:07 smithi196 ceph-mon[30802]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:07.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:07 smithi032 ceph-mon[27388]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:07.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:07 smithi204 ceph-mon[30933]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:08.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:08 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T05:07:08.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:08 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:08.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:08 smithi196 ceph-mon[30802]: Deploying daemon osd.6 on smithi196 2024-09-16T05:07:08.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:08 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T05:07:08.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:08 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:08.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:08 smithi032 ceph-mon[27388]: Deploying daemon osd.6 on smithi196 2024-09-16T05:07:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:08 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T05:07:08.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:08 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:08.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:08 smithi204 ceph-mon[30933]: Deploying daemon osd.6 on smithi196 2024-09-16T05:07:09.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:09 smithi196 ceph-mon[30802]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:09.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:09 smithi032 ceph-mon[27388]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:09 smithi204 ceph-mon[30933]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:11.189 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:11 smithi196 ceph-mon[30802]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:11.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:11 smithi032 ceph-mon[27388]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:11.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:11 smithi204 ceph-mon[30933]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:12 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:07:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:12 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:07:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:12 smithi032 ceph-mon[27388]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T05:07:12.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:12 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:12 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:07:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:12 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:07:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:12 smithi204 ceph-mon[30933]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T05:07:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:12 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:12.451 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:12 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:07:12.451 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:12 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:07:12.452 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:12 smithi196 ceph-mon[30802]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T05:07:12.452 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:12 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:13.360 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:13 smithi196 ceph-mon[30802]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:13 smithi032 ceph-mon[27388]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:13.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:13 smithi204 ceph-mon[30933]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:14.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:14.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:14.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:14.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:15 smithi032 ceph-mon[27388]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:15.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:15 smithi204 ceph-mon[30933]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:15 smithi196 ceph-mon[30802]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:17.421 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 6 on host 'smithi196' 2024-09-16T05:07:17.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:17.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:17.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:17.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:17.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.719 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:17.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:17.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:17.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:18.020 DEBUG:teuthology.orchestra.run.smithi196:osd.6> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.6.service 2024-09-16T05:07:18.023 INFO:tasks.cephadm:Deploying osd.7 on smithi196 with /dev/vg_nvme/lv_1... 2024-09-16T05:07:18.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T05:07:18.857 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:07:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:18 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:07:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:18 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:07:18.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:18 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:07:18.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:18 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:07:18.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:18 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:07:18.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:18 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:07:18.983 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:07:18 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:07:18.629+0000 7f6afc530640 -1 osd.6 0 log_to_monitors true 2024-09-16T05:07:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:19 smithi032 ceph-mon[27388]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:19 smithi032 ceph-mon[27388]: osdmap e45: 7 total, 6 up, 7 in 2024-09-16T05:07:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:19 smithi032 ceph-mon[27388]: from='osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T05:07:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:19 smithi032 ceph-mon[27388]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T05:07:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:19 smithi204 ceph-mon[30933]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:19 smithi204 ceph-mon[30933]: osdmap e45: 7 total, 6 up, 7 in 2024-09-16T05:07:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:19 smithi204 ceph-mon[30933]: from='osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T05:07:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:19 smithi204 ceph-mon[30933]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T05:07:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:19 smithi196 ceph-mon[30802]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:19 smithi196 ceph-mon[30802]: osdmap e45: 7 total, 6 up, 7 in 2024-09-16T05:07:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:19 smithi196 ceph-mon[30802]: from='osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T05:07:19.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:19 smithi196 ceph-mon[30802]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T05:07:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:20 smithi196 ceph-mon[30802]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T05:07:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:20 smithi196 ceph-mon[30802]: osdmap e46: 7 total, 6 up, 7 in 2024-09-16T05:07:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:20 smithi196 ceph-mon[30802]: from='osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:20 smithi196 ceph-mon[30802]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:20 smithi196 ceph-mon[30802]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:07:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:20 smithi032 ceph-mon[27388]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T05:07:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:20 smithi032 ceph-mon[27388]: osdmap e46: 7 total, 6 up, 7 in 2024-09-16T05:07:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:20 smithi032 ceph-mon[27388]: from='osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:20 smithi032 ceph-mon[27388]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:20 smithi032 ceph-mon[27388]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:20 smithi204 ceph-mon[30933]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:20 smithi204 ceph-mon[30933]: osdmap e46: 7 total, 6 up, 7 in 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:20 smithi204 ceph-mon[30933]: from='osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:20 smithi204 ceph-mon[30933]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:20 smithi204 ceph-mon[30933]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:07:21.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:07:21.007+0000 7f6af84b3640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T05:07:21.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:07:21.014+0000 7f6af02d3640 -1 osd.6 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:07:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-mon[30802]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-mon[30802]: osdmap e47: 7 total, 6 up, 7 in 2024-09-16T05:07:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-mon[30802]: osdmap e48: 7 total, 6 up, 7 in 2024-09-16T05:07:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:21 smithi196 ceph-mon[30802]: from='osd.6 ' entity='osd.6' 2024-09-16T05:07:21.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:21 smithi032 ceph-mon[27388]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:21.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:21 smithi032 ceph-mon[27388]: osdmap e47: 7 total, 6 up, 7 in 2024-09-16T05:07:21.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:21 smithi032 ceph-mon[27388]: osdmap e48: 7 total, 6 up, 7 in 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:21 smithi032 ceph-mon[27388]: from='osd.6 ' entity='osd.6' 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:21 smithi204 ceph-mon[30933]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:21 smithi204 ceph-mon[30933]: osdmap e47: 7 total, 6 up, 7 in 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:21 smithi204 ceph-mon[30933]: osdmap e48: 7 total, 6 up, 7 in 2024-09-16T05:07:21.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:21.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:21 smithi204 ceph-mon[30933]: from='osd.6 ' entity='osd.6' 2024-09-16T05:07:22.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:07:22.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:07:22.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:22.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi196 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861] boot 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: osdmap e49: 7 total, 7 up, 7 in 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:22.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:22.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:07:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi196 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861] boot 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: osdmap e49: 7 total, 7 up, 7 in 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi196 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: osd.6 [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861] boot 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: osdmap e49: 7 total, 7 up, 7 in 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi196 to 5696M 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:22.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:23.015 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-16T05:07:23.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_1 2024-09-16T05:07:23.290 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:07:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:23 smithi196 ceph-mon[30802]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:23 smithi196 ceph-mon[30802]: osdmap e50: 7 total, 7 up, 7 in 2024-09-16T05:07:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:23 smithi032 ceph-mon[27388]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:23 smithi032 ceph-mon[27388]: osdmap e50: 7 total, 7 up, 7 in 2024-09-16T05:07:23.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:23 smithi204 ceph-mon[30933]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-16T05:07:23.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:23 smithi204 ceph-mon[30933]: osdmap e50: 7 total, 7 up, 7 in 2024-09-16T05:07:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:24 smithi032 ceph-mon[27388]: osdmap e51: 7 total, 7 up, 7 in 2024-09-16T05:07:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:24 smithi032 ceph-mon[27388]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:07:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:07:24.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:24 smithi204 ceph-mon[30933]: osdmap e51: 7 total, 7 up, 7 in 2024-09-16T05:07:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:24 smithi204 ceph-mon[30933]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:07:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:07:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:24.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:24 smithi196 ceph-mon[30802]: osdmap e51: 7 total, 7 up, 7 in 2024-09-16T05:07:24.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:24 smithi196 ceph-mon[30802]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:24.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:07:24.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:07:24.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:25 smithi196 ceph-mon[30802]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:25 smithi196 ceph-mon[30802]: osdmap e52: 7 total, 7 up, 7 in 2024-09-16T05:07:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:25 smithi032 ceph-mon[27388]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:25 smithi032 ceph-mon[27388]: osdmap e52: 7 total, 7 up, 7 in 2024-09-16T05:07:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:25 smithi204 ceph-mon[30933]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:25 smithi204 ceph-mon[30933]: osdmap e52: 7 total, 7 up, 7 in 2024-09-16T05:07:26.912 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:26 smithi196 ceph-mon[30802]: osdmap e53: 7 total, 7 up, 7 in 2024-09-16T05:07:26.912 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:26 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:07:26.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:26 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:07:26.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/142477493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]: dispatch 2024-09-16T05:07:26.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:26 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]: dispatch 2024-09-16T05:07:26.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:26 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]': finished 2024-09-16T05:07:26.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:26 smithi196 ceph-mon[30802]: osdmap e54: 8 total, 7 up, 8 in 2024-09-16T05:07:26.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:27.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:26 smithi032 ceph-mon[27388]: osdmap e53: 7 total, 7 up, 7 in 2024-09-16T05:07:27.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:26 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:07:27.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:26 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:07:27.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/142477493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]: dispatch 2024-09-16T05:07:27.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:26 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]: dispatch 2024-09-16T05:07:27.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:26 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]': finished 2024-09-16T05:07:27.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:26 smithi032 ceph-mon[27388]: osdmap e54: 8 total, 7 up, 8 in 2024-09-16T05:07:27.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:26 smithi204 ceph-mon[30933]: osdmap e53: 7 total, 7 up, 7 in 2024-09-16T05:07:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:26 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:07:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:26 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:07:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/142477493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]: dispatch 2024-09-16T05:07:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:26 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]: dispatch 2024-09-16T05:07:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:26 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2"}]': finished 2024-09-16T05:07:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:26 smithi204 ceph-mon[30933]: osdmap e54: 8 total, 7 up, 8 in 2024-09-16T05:07:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:27 smithi196 ceph-mon[30802]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/3819940146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:28.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:27 smithi032 ceph-mon[27388]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:28.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/3819940146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:27 smithi204 ceph-mon[30933]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/3819940146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:29.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:29 smithi196 ceph-mon[30802]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-16T05:07:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:29 smithi032 ceph-mon[27388]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-16T05:07:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:29 smithi204 ceph-mon[30933]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-16T05:07:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:31 smithi196 ceph-mon[30802]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-16T05:07:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:31 smithi032 ceph-mon[27388]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-16T05:07:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:31 smithi204 ceph-mon[30933]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-16T05:07:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:33 smithi196 ceph-mon[30802]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-16T05:07:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:33 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T05:07:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:33 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:33 smithi032 ceph-mon[27388]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-16T05:07:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:33 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T05:07:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:33 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:33 smithi204 ceph-mon[30933]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-16T05:07:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:33 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T05:07:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:33 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:34.855 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:34 smithi196 ceph-mon[30802]: Deploying daemon osd.7 on smithi196 2024-09-16T05:07:35.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:34 smithi204 ceph-mon[30933]: Deploying daemon osd.7 on smithi196 2024-09-16T05:07:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:34 smithi032 ceph-mon[27388]: Deploying daemon osd.7 on smithi196 2024-09-16T05:07:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:35 smithi196 ceph-mon[30802]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T05:07:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:35 smithi032 ceph-mon[27388]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T05:07:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:35 smithi204 ceph-mon[30933]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-16T05:07:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:37 smithi032 ceph-mon[27388]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-16T05:07:37.934 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:37 smithi196 ceph-mon[30802]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-16T05:07:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:37 smithi204 ceph-mon[30933]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-16T05:07:39.958 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:39 smithi196 ceph-mon[30802]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-09-16T05:07:39.958 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:39 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:39.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:39 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:39.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:39 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:39 smithi032 ceph-mon[27388]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-09-16T05:07:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:39 smithi204 ceph-mon[30933]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-09-16T05:07:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:39 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:39 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:39 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:40 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:07:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:40 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:07:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:40 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:07:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:40 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:07:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:40 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:07:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:40 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:07:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:41 smithi196 ceph-mon[30802]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:41 smithi196 ceph-mon[30802]: osdmap e55: 8 total, 7 up, 8 in 2024-09-16T05:07:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:41 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:41 smithi032 ceph-mon[27388]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:41 smithi032 ceph-mon[27388]: osdmap e55: 8 total, 7 up, 8 in 2024-09-16T05:07:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:41 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:41 smithi204 ceph-mon[30933]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:41 smithi204 ceph-mon[30933]: osdmap e55: 8 total, 7 up, 8 in 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:41 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:42.427 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 7 on host 'smithi196' 2024-09-16T05:07:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:43.162 DEBUG:teuthology.orchestra.run.smithi196:osd.7> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.7.service 2024-09-16T05:07:43.165 INFO:tasks.cephadm:Deploying osd.8 on smithi204 with /dev/vg_nvme/lv_4... 2024-09-16T05:07:43.165 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T05:07:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:07:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:43.415 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:07:43.814 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:43 smithi204 ceph-mon[30933]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:43 smithi196 ceph-mon[30802]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:43 smithi032 ceph-mon[27388]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:44.375 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:07:44 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[52338]: 2024-09-16T05:07:44.006+0000 7fc26ce84640 -1 osd.7 0 log_to_monitors true 2024-09-16T05:07:44.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:44 smithi196 ceph-mon[30802]: from='osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:07:44.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:44 smithi196 ceph-mon[30802]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:07:45.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:44 smithi204 ceph-mon[30933]: from='osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:07:45.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:44 smithi204 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:07:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:44 smithi032 ceph-mon[27388]: from='osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:07:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:44 smithi032 ceph-mon[27388]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:07:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:07:45.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_4 2024-09-16T05:07:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:45 smithi196 ceph-mon[30802]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:45 smithi196 ceph-mon[30802]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T05:07:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:45 smithi196 ceph-mon[30802]: osdmap e56: 8 total, 7 up, 8 in 2024-09-16T05:07:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:45 smithi196 ceph-mon[30802]: from='osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:45 smithi196 ceph-mon[30802]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:46.087 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:07:46.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:45 smithi032 ceph-mon[27388]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:46.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:45 smithi032 ceph-mon[27388]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:45 smithi032 ceph-mon[27388]: osdmap e56: 8 total, 7 up, 8 in 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:45 smithi032 ceph-mon[27388]: from='osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:45 smithi032 ceph-mon[27388]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:45 smithi204 ceph-mon[30933]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:45 smithi204 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:45 smithi204 ceph-mon[30933]: osdmap e56: 8 total, 7 up, 8 in 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:45 smithi204 ceph-mon[30933]: from='osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:46.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:45 smithi204 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:07:46.488 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[52338]: 2024-09-16T05:07:46.166+0000 7fc268e07640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T05:07:46.488 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[52338]: 2024-09-16T05:07:46.174+0000 7fc260c27640 -1 osd.7 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:07:47.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:07:47.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: osdmap e57: 8 total, 7 up, 8 in 2024-09-16T05:07:47.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='osd.7 ' entity='osd.7' 2024-09-16T05:07:47.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846] boot 2024-09-16T05:07:47.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T05:07:47.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:07:47.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:07:47.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: osdmap e57: 8 total, 7 up, 8 in 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:47.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='osd.7 ' entity='osd.7' 2024-09-16T05:07:47.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846] boot 2024-09-16T05:07:47.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T05:07:47.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:07:47.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:07:47.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-09-16T05:07:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: osdmap e57: 8 total, 7 up, 8 in 2024-09-16T05:07:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='osd.7 ' entity='osd.7' 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: osd.7 [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846] boot 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: osdmap e58: 8 total, 8 up, 8 in 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:07:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:48.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:47 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:07:48.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:47 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:07:48.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:47 smithi204 ceph-mon[30933]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:48.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:47 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi196 2024-09-16T05:07:48.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:47 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-16T05:07:48.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:47 smithi204 ceph-mon[30933]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:48.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:47 smithi204 ceph-mon[30933]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T05:07:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:47 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:07:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:47 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:07:48.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:47 smithi032 ceph-mon[27388]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:48.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:47 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi196 2024-09-16T05:07:48.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:47 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-16T05:07:48.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:47 smithi032 ceph-mon[27388]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:48.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:47 smithi032 ceph-mon[27388]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T05:07:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:47 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:07:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:47 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:07:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:47 smithi196 ceph-mon[30802]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-16T05:07:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:47 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi196 2024-09-16T05:07:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:47 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi196 to 4272M 2024-09-16T05:07:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:47 smithi196 ceph-mon[30802]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:07:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:47 smithi196 ceph-mon[30802]: osdmap e59: 8 total, 8 up, 8 in 2024-09-16T05:07:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:49 smithi204 ceph-mon[30933]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:07:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:49 smithi204 ceph-mon[30933]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T05:07:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/400844750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]: dispatch 2024-09-16T05:07:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:49 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]: dispatch 2024-09-16T05:07:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:49 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]': finished 2024-09-16T05:07:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:49 smithi204 ceph-mon[30933]: osdmap e61: 9 total, 8 up, 9 in 2024-09-16T05:07:49.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:49.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/1892895750' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:50.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:49 smithi032 ceph-mon[27388]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:07:50.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:49 smithi032 ceph-mon[27388]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T05:07:50.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/400844750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]: dispatch 2024-09-16T05:07:50.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:49 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]: dispatch 2024-09-16T05:07:50.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:49 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]': finished 2024-09-16T05:07:50.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:49 smithi032 ceph-mon[27388]: osdmap e61: 9 total, 8 up, 9 in 2024-09-16T05:07:50.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:50.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/1892895750' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:49 smithi196 ceph-mon[30802]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:07:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:49 smithi196 ceph-mon[30802]: osdmap e60: 8 total, 8 up, 8 in 2024-09-16T05:07:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/400844750' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]: dispatch 2024-09-16T05:07:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:49 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]: dispatch 2024-09-16T05:07:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:49 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090"}]': finished 2024-09-16T05:07:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:49 smithi196 ceph-mon[30802]: osdmap e61: 9 total, 8 up, 9 in 2024-09-16T05:07:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/1892895750' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:07:51.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:51 smithi032 ceph-mon[27388]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:07:51.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:51 smithi032 ceph-mon[27388]: osdmap e62: 9 total, 8 up, 9 in 2024-09-16T05:07:51.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:51 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:51 smithi204 ceph-mon[30933]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:07:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:51 smithi204 ceph-mon[30933]: osdmap e62: 9 total, 8 up, 9 in 2024-09-16T05:07:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:51 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:51.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:51 smithi196 ceph-mon[30802]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:07:51.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:51 smithi196 ceph-mon[30802]: osdmap e62: 9 total, 8 up, 9 in 2024-09-16T05:07:51.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:51 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:52.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:52 smithi032 ceph-mon[27388]: osdmap e63: 9 total, 8 up, 9 in 2024-09-16T05:07:52.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:52 smithi204 ceph-mon[30933]: osdmap e63: 9 total, 8 up, 9 in 2024-09-16T05:07:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:52 smithi196 ceph-mon[30802]: osdmap e63: 9 total, 8 up, 9 in 2024-09-16T05:07:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:07:53.160 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:53 smithi204 ceph-mon[30933]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-09-16T05:07:53.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:53 smithi032 ceph-mon[27388]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-09-16T05:07:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:53 smithi196 ceph-mon[30802]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-09-16T05:07:54.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:54 smithi204 ceph-mon[30933]: 1.0 deep-scrub starts 2024-09-16T05:07:54.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:54 smithi204 ceph-mon[30933]: 1.0 deep-scrub ok 2024-09-16T05:07:54.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:54 smithi032 ceph-mon[27388]: 1.0 deep-scrub starts 2024-09-16T05:07:54.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:54 smithi032 ceph-mon[27388]: 1.0 deep-scrub ok 2024-09-16T05:07:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:54 smithi196 ceph-mon[30802]: 1.0 deep-scrub starts 2024-09-16T05:07:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:54 smithi196 ceph-mon[30802]: 1.0 deep-scrub ok 2024-09-16T05:07:55.231 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:55 smithi204 ceph-mon[30933]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-16T05:07:55.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:55 smithi032 ceph-mon[27388]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-16T05:07:55.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:55 smithi196 ceph-mon[30802]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-16T05:07:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:56 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T05:07:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:56 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:56 smithi204 ceph-mon[30933]: Deploying daemon osd.8 on smithi204 2024-09-16T05:07:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:56 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T05:07:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:56 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:56 smithi032 ceph-mon[27388]: Deploying daemon osd.8 on smithi204 2024-09-16T05:07:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:56 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T05:07:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:56 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:07:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:56 smithi196 ceph-mon[30802]: Deploying daemon osd.8 on smithi204 2024-09-16T05:07:57.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:57 smithi204 ceph-mon[30933]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T05:07:57.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:57 smithi032 ceph-mon[27388]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T05:07:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:57 smithi196 ceph-mon[30802]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T05:07:59.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:07:59 smithi204 ceph-mon[30933]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-16T05:07:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:07:59 smithi032 ceph-mon[27388]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-16T05:07:59.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:07:59 smithi196 ceph-mon[30802]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-16T05:08:01.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:01 smithi204 ceph-mon[30933]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-16T05:08:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:01 smithi032 ceph-mon[27388]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-16T05:08:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:01 smithi196 ceph-mon[30802]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-16T05:08:02.325 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:02.325 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:02.325 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:02 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:02.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:02.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:02.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:02 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:02 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:04.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:03 smithi204 ceph-mon[30933]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:08:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:03 smithi032 ceph-mon[27388]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:08:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:03 smithi196 ceph-mon[30802]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:08:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:05.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:05.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:05.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:05.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:05.979 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 8 on host 'smithi204' 2024-09-16T05:08:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:05 smithi196 ceph-mon[30802]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:06.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:05 smithi204 ceph-mon[30933]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:06.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:05 smithi032 ceph-mon[27388]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:06.621 DEBUG:teuthology.orchestra.run.smithi204:osd.8> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.8.service 2024-09-16T05:08:06.624 INFO:tasks.cephadm:Deploying osd.9 on smithi204 with /dev/vg_nvme/lv_3... 2024-09-16T05:08:06.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T05:08:06.838 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:08:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:08:06.680+0000 7f7354bd5640 -1 osd.8 0 log_to_monitors true 2024-09-16T05:08:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:06 smithi204 ceph-mon[30933]: from='osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T05:08:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:06 smithi204 ceph-mon[30933]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T05:08:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:06 smithi196 ceph-mon[30802]: from='osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T05:08:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:06 smithi196 ceph-mon[30802]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T05:08:07.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:07.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:07.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:07.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:06 smithi032 ceph-mon[27388]: from='osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T05:08:07.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:06 smithi032 ceph-mon[27388]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T05:08:08.144 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:08:08.170 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:08:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:08:07.984+0000 7f735136c640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T05:08:08.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:08:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:08:07.991+0000 7f734c980640 -1 osd.8 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:08:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:07 smithi204 ceph-mon[30933]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:07 smithi204 ceph-mon[30933]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T05:08:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:07 smithi204 ceph-mon[30933]: from='osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:07 smithi204 ceph-mon[30933]: osdmap e64: 9 total, 8 up, 9 in 2024-09-16T05:08:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:07 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:07 smithi204 ceph-mon[30933]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:07 smithi196 ceph-mon[30802]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:07 smithi196 ceph-mon[30802]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T05:08:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:07 smithi196 ceph-mon[30802]: from='osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:07 smithi196 ceph-mon[30802]: osdmap e64: 9 total, 8 up, 9 in 2024-09-16T05:08:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:07 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:07 smithi196 ceph-mon[30802]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:08.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:07 smithi032 ceph-mon[27388]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:08.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:07 smithi032 ceph-mon[27388]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T05:08:08.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:07 smithi032 ceph-mon[27388]: from='osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:08.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:07 smithi032 ceph-mon[27388]: osdmap e64: 9 total, 8 up, 9 in 2024-09-16T05:08:08.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:07 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:08.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:07 smithi032 ceph-mon[27388]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:09.290 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:08 smithi204 ceph-mon[30933]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:09.290 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:08 smithi204 ceph-mon[30933]: osdmap e65: 9 total, 8 up, 9 in 2024-09-16T05:08:09.290 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:08 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:09.290 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:08 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:09.290 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:08 smithi204 ceph-mon[30933]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:09.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:08 smithi032 ceph-mon[27388]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:09.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:08 smithi032 ceph-mon[27388]: osdmap e65: 9 total, 8 up, 9 in 2024-09-16T05:08:09.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:08 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:09.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:08 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:09.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:08 smithi032 ceph-mon[27388]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:08 smithi196 ceph-mon[30802]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:08 smithi196 ceph-mon[30802]: osdmap e65: 9 total, 8 up, 9 in 2024-09-16T05:08:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:08 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:08 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:08 smithi196 ceph-mon[30802]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-16T05:08:10.369 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:10.369 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:10.369 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812] boot 2024-09-16T05:08:10.369 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: osdmap e66: 9 total, 9 up, 9 in 2024-09-16T05:08:10.369 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi204 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi204 to 21184M 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:10.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:09 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812] boot 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: osdmap e66: 9 total, 9 up, 9 in 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi204 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi204 to 21184M 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:10.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:09 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:10.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:10.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: osd.8 [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812] boot 2024-09-16T05:08:10.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: osdmap e66: 9 total, 9 up, 9 in 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi204 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi204 to 21184M 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:10.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:09 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:11.036 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:08:11.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_3 2024-09-16T05:08:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:11 smithi204 ceph-mon[30933]: osdmap e67: 9 total, 9 up, 9 in 2024-09-16T05:08:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:11 smithi204 ceph-mon[30933]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:11.302 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:08:11.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:11 smithi032 ceph-mon[27388]: osdmap e67: 9 total, 9 up, 9 in 2024-09-16T05:08:11.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:11 smithi032 ceph-mon[27388]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:11.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:11 smithi196 ceph-mon[30802]: osdmap e67: 9 total, 9 up, 9 in 2024-09-16T05:08:11.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:11 smithi196 ceph-mon[30802]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:12.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:12 smithi204 ceph-mon[30933]: osdmap e68: 9 total, 9 up, 9 in 2024-09-16T05:08:12.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:12 smithi204 ceph-mon[30933]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:08:12.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:12 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:08:12.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:12 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:08:12.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:12 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:12 smithi032 ceph-mon[27388]: osdmap e68: 9 total, 9 up, 9 in 2024-09-16T05:08:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:12 smithi032 ceph-mon[27388]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:08:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:12 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:08:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:12 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:08:12.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:12 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:12 smithi196 ceph-mon[30802]: osdmap e68: 9 total, 9 up, 9 in 2024-09-16T05:08:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:12 smithi196 ceph-mon[30802]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:08:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:12 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:08:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:12 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:08:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:12 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:13.296 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:13 smithi204 ceph-mon[30933]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-16T05:08:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:13 smithi032 ceph-mon[27388]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-16T05:08:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:13 smithi196 ceph-mon[30802]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-16T05:08:14.318 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:14 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:14.318 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:14 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:14.318 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:14 smithi204 ceph-mon[30933]: osdmap e69: 9 total, 9 up, 9 in 2024-09-16T05:08:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:14 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:14 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:14.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:14 smithi032 ceph-mon[27388]: osdmap e69: 9 total, 9 up, 9 in 2024-09-16T05:08:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:14 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:14 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:14 smithi196 ceph-mon[30802]: osdmap e69: 9 total, 9 up, 9 in 2024-09-16T05:08:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-16T05:08:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/3009574927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]: dispatch 2024-09-16T05:08:15.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]: dispatch 2024-09-16T05:08:15.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]': finished 2024-09-16T05:08:15.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: osdmap e70: 10 total, 9 up, 10 in 2024-09-16T05:08:15.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:15.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:15.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:15.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/1734120624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:08:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-16T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/3009574927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]: dispatch 2024-09-16T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]: dispatch 2024-09-16T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]': finished 2024-09-16T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: osdmap e70: 10 total, 9 up, 10 in 2024-09-16T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/1734120624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:08:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-16T05:08:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/3009574927' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]: dispatch 2024-09-16T05:08:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]: dispatch 2024-09-16T05:08:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a"}]': finished 2024-09-16T05:08:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: osdmap e70: 10 total, 9 up, 10 in 2024-09-16T05:08:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:15.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:15.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:15.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/1734120624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:08:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:16 smithi204 ceph-mon[30933]: osdmap e71: 10 total, 9 up, 10 in 2024-09-16T05:08:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:16.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:16 smithi196 ceph-mon[30802]: osdmap e71: 10 total, 9 up, 10 in 2024-09-16T05:08:16.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:16.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:16 smithi032 ceph-mon[27388]: osdmap e71: 10 total, 9 up, 10 in 2024-09-16T05:08:16.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:17.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:17 smithi032 ceph-mon[27388]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-16T05:08:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:17 smithi204 ceph-mon[30933]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-16T05:08:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:17 smithi196 ceph-mon[30802]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-16T05:08:19.413 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:19 smithi204 ceph-mon[30933]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:19 smithi032 ceph-mon[27388]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:19 smithi196 ceph-mon[30802]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:21.207 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:21 smithi204 ceph-mon[30933]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:21.208 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:21 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:21.208 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:21 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:21.208 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:21 smithi204 ceph-mon[30933]: osdmap e72: 10 total, 9 up, 10 in 2024-09-16T05:08:21.208 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:21.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:21 smithi032 ceph-mon[27388]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:21.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:21 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:21.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:21 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:21.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:21 smithi032 ceph-mon[27388]: osdmap e72: 10 total, 9 up, 10 in 2024-09-16T05:08:21.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:21 smithi196 ceph-mon[30802]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:21 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:21 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:21 smithi196 ceph-mon[30802]: osdmap e72: 10 total, 9 up, 10 in 2024-09-16T05:08:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T05:08:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:22.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:22 smithi032 ceph-mon[27388]: Deploying daemon osd.9 on smithi204 2024-09-16T05:08:22.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:22 smithi032 ceph-mon[27388]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:22.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:08:22.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:08:22.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:08:22.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T05:08:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:22 smithi204 ceph-mon[30933]: Deploying daemon osd.9 on smithi204 2024-09-16T05:08:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:22 smithi204 ceph-mon[30933]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:08:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:08:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:08:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T05:08:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:22 smithi196 ceph-mon[30802]: Deploying daemon osd.9 on smithi204 2024-09-16T05:08:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:22 smithi196 ceph-mon[30802]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:08:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:08:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:08:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:23.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:23.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:23.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:25.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:25 smithi032 ceph-mon[27388]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:25 smithi204 ceph-mon[30933]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:25 smithi196 ceph-mon[30802]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:27.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:27 smithi032 ceph-mon[27388]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:27 smithi204 ceph-mon[30933]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:27 smithi196 ceph-mon[30802]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:28.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:28.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:28.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:29.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:29 smithi196 ceph-mon[30802]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:29.488 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:29 smithi204 ceph-mon[30933]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:29.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:29 smithi032 ceph-mon[27388]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:30.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:30.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:30.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:30.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:30.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:30.946 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 9 on host 'smithi204' 2024-09-16T05:08:31.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:31 smithi204 ceph-mon[30933]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:31.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:31.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:31.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:31.557 DEBUG:teuthology.orchestra.run.smithi204:osd.9> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.9.service 2024-09-16T05:08:31.560 INFO:tasks.cephadm:Deploying osd.10 on smithi204 with /dev/vg_nvme/lv_2... 2024-09-16T05:08:31.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T05:08:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:31 smithi196 ceph-mon[30802]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:31.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:31 smithi032 ceph-mon[27388]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:31.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:31.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:32.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:08:31 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:08:31.987+0000 7f9fdd63c640 -1 osd.9 0 log_to_monitors true 2024-09-16T05:08:32.361 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:08:32.720 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:32 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:32.720 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:32 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:32.720 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:32 smithi204 ceph-mon[30933]: osdmap e73: 10 total, 9 up, 10 in 2024-09-16T05:08:32.720 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:32 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:32.721 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:32 smithi204 ceph-mon[30933]: from='osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T05:08:32.721 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:32 smithi204 ceph-mon[30933]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T05:08:32.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:32 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:32.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:32 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:32.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:32 smithi196 ceph-mon[30802]: osdmap e73: 10 total, 9 up, 10 in 2024-09-16T05:08:32.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:32 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:32.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:32 smithi196 ceph-mon[30802]: from='osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T05:08:32.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:32 smithi196 ceph-mon[30802]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T05:08:32.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:32 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:32.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:32 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:32 smithi032 ceph-mon[27388]: osdmap e73: 10 total, 9 up, 10 in 2024-09-16T05:08:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:32 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:32 smithi032 ceph-mon[27388]: from='osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T05:08:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:32 smithi032 ceph-mon[27388]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T05:08:33.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:33 smithi204 ceph-mon[30933]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:33.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:33 smithi204 ceph-mon[30933]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T05:08:33.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:33 smithi204 ceph-mon[30933]: from='osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:33.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:33 smithi204 ceph-mon[30933]: osdmap e74: 10 total, 9 up, 10 in 2024-09-16T05:08:33.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:33 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:33.602 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:33 smithi204 ceph-mon[30933]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:33 smithi196 ceph-mon[30802]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:33 smithi196 ceph-mon[30802]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T05:08:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:33 smithi196 ceph-mon[30802]: from='osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:33 smithi196 ceph-mon[30802]: osdmap e74: 10 total, 9 up, 10 in 2024-09-16T05:08:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:33 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:33.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:33 smithi196 ceph-mon[30802]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:33.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:33 smithi032 ceph-mon[27388]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:33.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:33 smithi032 ceph-mon[27388]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T05:08:33.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:33 smithi032 ceph-mon[27388]: from='osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:33.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:33 smithi032 ceph-mon[27388]: osdmap e74: 10 total, 9 up, 10 in 2024-09-16T05:08:33.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:33 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:33.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:33 smithi032 ceph-mon[27388]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:34.131 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:08:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:08:33.853+0000 7f9fd95bf640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T05:08:34.131 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:08:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:08:33.861+0000 7f9fd13df640 -1 osd.9 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: osdmap e75: 10 total, 9 up, 10 in 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='osd.9 ' entity='osd.9' 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:34.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:34.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:34.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:34.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: osdmap e75: 10 total, 9 up, 10 in 2024-09-16T05:08:34.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:34.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:34.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='osd.9 ' entity='osd.9' 2024-09-16T05:08:34.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:34.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:34.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:34.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:34.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: osdmap e75: 10 total, 9 up, 10 in 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='osd.9 ' entity='osd.9' 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:34.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:35.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:35.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:35.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:35.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi204 2024-09-16T05:08:35.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi204 to 10592M 2024-09-16T05:08:35.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:35.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345] boot 2024-09-16T05:08:35.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: osdmap e76: 10 total, 10 up, 10 in 2024-09-16T05:08:35.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:35 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi204 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi204 to 10592M 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345] boot 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: osdmap e76: 10 total, 10 up, 10 in 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:35 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:35.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-16T05:08:35.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi204 2024-09-16T05:08:35.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi204 to 10592M 2024-09-16T05:08:35.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:35.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: osd.9 [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345] boot 2024-09-16T05:08:35.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: osdmap e76: 10 total, 10 up, 10 in 2024-09-16T05:08:35.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:35 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T05:08:36.170 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:08:36.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_2 2024-09-16T05:08:36.442 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:08:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:36 smithi032 ceph-mon[27388]: osdmap e77: 10 total, 10 up, 10 in 2024-09-16T05:08:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:36 smithi204 ceph-mon[30933]: osdmap e77: 10 total, 10 up, 10 in 2024-09-16T05:08:37.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:36 smithi196 ceph-mon[30802]: osdmap e77: 10 total, 10 up, 10 in 2024-09-16T05:08:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:37 smithi032 ceph-mon[27388]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:37 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:08:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:37 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:08:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:37 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:37 smithi204 ceph-mon[30933]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:37 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:08:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:37 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:08:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:37 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:37 smithi196 ceph-mon[30802]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:37 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:08:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:37 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:08:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:37 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:39.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:38 smithi204 ceph-mon[30933]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:08:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:38 smithi032 ceph-mon[27388]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:08:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:38 smithi196 ceph-mon[30802]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:08:39.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:39 smithi204 ceph-mon[30933]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:39.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/2796348620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]: dispatch 2024-09-16T05:08:39.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:39 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]: dispatch 2024-09-16T05:08:39.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:39 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]': finished 2024-09-16T05:08:39.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:39 smithi204 ceph-mon[30933]: osdmap e78: 11 total, 10 up, 11 in 2024-09-16T05:08:39.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:39 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:39 smithi032 ceph-mon[27388]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/2796348620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]: dispatch 2024-09-16T05:08:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:39 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]: dispatch 2024-09-16T05:08:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:39 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]': finished 2024-09-16T05:08:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:39 smithi032 ceph-mon[27388]: osdmap e78: 11 total, 10 up, 11 in 2024-09-16T05:08:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:39 smithi196 ceph-mon[30802]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/2796348620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]: dispatch 2024-09-16T05:08:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:39 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]: dispatch 2024-09-16T05:08:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:39 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846"}]': finished 2024-09-16T05:08:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:39 smithi196 ceph-mon[30802]: osdmap e78: 11 total, 10 up, 11 in 2024-09-16T05:08:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:39 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/1246532340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:08:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/1246532340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:08:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/1246532340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:08:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:41 smithi032 ceph-mon[27388]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:41 smithi204 ceph-mon[30933]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:41 smithi196 ceph-mon[30802]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:42 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:42 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:42 smithi032 ceph-mon[27388]: osdmap e79: 11 total, 10 up, 11 in 2024-09-16T05:08:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:42 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:42 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:42 smithi204 ceph-mon[30933]: osdmap e79: 11 total, 10 up, 11 in 2024-09-16T05:08:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:42 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:42 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:42 smithi196 ceph-mon[30802]: osdmap e79: 11 total, 10 up, 11 in 2024-09-16T05:08:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:43 smithi032 ceph-mon[27388]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:43 smithi204 ceph-mon[30933]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:43 smithi196 ceph-mon[30802]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:44 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:44 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:44 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:44 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:44 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:44 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:46.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:45 smithi204 ceph-mon[30933]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:46.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:45 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:08:46.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:45 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:08:46.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:45 smithi204 ceph-mon[30933]: osdmap e80: 11 total, 10 up, 11 in 2024-09-16T05:08:46.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:45 smithi032 ceph-mon[27388]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:45 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:08:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:45 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:08:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:45 smithi032 ceph-mon[27388]: osdmap e80: 11 total, 10 up, 11 in 2024-09-16T05:08:46.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:45 smithi196 ceph-mon[30802]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:45 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:08:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:45 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:08:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:45 smithi196 ceph-mon[30802]: osdmap e80: 11 total, 10 up, 11 in 2024-09-16T05:08:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T05:08:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T05:08:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T05:08:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:47 smithi032 ceph-mon[27388]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:47 smithi032 ceph-mon[27388]: Deploying daemon osd.10 on smithi204 2024-09-16T05:08:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:47 smithi204 ceph-mon[30933]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:47 smithi204 ceph-mon[30933]: Deploying daemon osd.10 on smithi204 2024-09-16T05:08:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:47 smithi196 ceph-mon[30802]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:47 smithi196 ceph-mon[30802]: Deploying daemon osd.10 on smithi204 2024-09-16T05:08:50.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:49 smithi032 ceph-mon[27388]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:49 smithi204 ceph-mon[30933]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:49 smithi196 ceph-mon[30802]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:51 smithi032 ceph-mon[27388]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:51 smithi204 ceph-mon[30933]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:51 smithi196 ceph-mon[30802]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:53 smithi032 ceph-mon[27388]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:53 smithi204 ceph-mon[30933]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:53 smithi196 ceph-mon[30802]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:55 smithi204 ceph-mon[30933]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:55 smithi196 ceph-mon[30802]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:55 smithi032 ceph-mon[27388]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:08:56.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:08:56.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:56.621 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 10 on host 'smithi204' 2024-09-16T05:08:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:56 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:56 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:56 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:57.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:56 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:57.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:56 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:57.309 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:56 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:57.335 DEBUG:teuthology.orchestra.run.smithi204:osd.10> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.10.service 2024-09-16T05:08:57.338 INFO:tasks.cephadm:Deploying osd.11 on smithi204 with /dev/vg_nvme/lv_1... 2024-09-16T05:08:57.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T05:08:57.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:56 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:08:57.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:56 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:57.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:56 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:08:57.560 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:08:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:08:57.534+0000 7f6c79b91640 -1 osd.10 0 log_to_monitors true 2024-09-16T05:08:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:57 smithi204 ceph-mon[30933]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:57 smithi204 ceph-mon[30933]: from='osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T05:08:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:57 smithi204 ceph-mon[30933]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T05:08:58.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:57 smithi196 ceph-mon[30802]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:58.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:57 smithi196 ceph-mon[30802]: from='osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T05:08:58.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:57 smithi196 ceph-mon[30802]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T05:08:58.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:57 smithi032 ceph-mon[27388]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:08:58.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:57 smithi032 ceph-mon[27388]: from='osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T05:08:58.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:57 smithi032 ceph-mon[27388]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T05:08:58.534 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:08:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:58 smithi204 ceph-mon[30933]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T05:08:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:58 smithi204 ceph-mon[30933]: osdmap e81: 11 total, 10 up, 11 in 2024-09-16T05:08:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:58 smithi204 ceph-mon[30933]: from='osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:58 smithi204 ceph-mon[30933]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:58 smithi204 ceph-mon[30933]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:58 smithi204 ceph-mon[30933]: osdmap e82: 11 total, 10 up, 11 in 2024-09-16T05:08:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:58 smithi196 ceph-mon[30802]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T05:08:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:58 smithi196 ceph-mon[30802]: osdmap e81: 11 total, 10 up, 11 in 2024-09-16T05:08:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:58 smithi196 ceph-mon[30802]: from='osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:58 smithi196 ceph-mon[30802]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:58 smithi196 ceph-mon[30802]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:58 smithi196 ceph-mon[30802]: osdmap e82: 11 total, 10 up, 11 in 2024-09-16T05:08:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:58 smithi032 ceph-mon[27388]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T05:08:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:58 smithi032 ceph-mon[27388]: osdmap e81: 11 total, 10 up, 11 in 2024-09-16T05:08:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:58 smithi032 ceph-mon[27388]: from='osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:58 smithi032 ceph-mon[27388]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:08:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:58 smithi032 ceph-mon[27388]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:08:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:58 smithi032 ceph-mon[27388]: osdmap e82: 11 total, 10 up, 11 in 2024-09-16T05:08:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:08:59.548 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:08:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:08:59.387+0000 7f6c76328640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T05:08:59.548 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:08:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:08:59.397+0000 7f6c6d934640 -1 osd.10 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:09:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:59 smithi204 ceph-mon[30933]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:09:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:59 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:59 smithi204 ceph-mon[30933]: from='osd.10 ' entity='osd.10' 2024-09-16T05:09:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:08:59 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:59 smithi196 ceph-mon[30802]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:09:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:59 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:59 smithi196 ceph-mon[30802]: from='osd.10 ' entity='osd.10' 2024-09-16T05:09:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:08:59 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:59 smithi032 ceph-mon[27388]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:09:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:59 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:59 smithi032 ceph-mon[27388]: from='osd.10 ' entity='osd.10' 2024-09-16T05:09:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:08:59 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:01.557 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436] boot 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: osdmap e83: 11 total, 11 up, 11 in 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.558 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:01.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:01.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:01 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436] boot 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: osdmap e83: 11 total, 11 up, 11 in 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:01.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:01.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:01 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:01.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:01.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-09-16T05:09:01.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: osd.10 [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436] boot 2024-09-16T05:09:01.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: osdmap e83: 11 total, 11 up, 11 in 2024-09-16T05:09:01.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T05:09:01.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:01.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:01.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:01.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:01.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:01 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:02.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:02 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:02.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:02 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:02.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:02 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi204 2024-09-16T05:09:02.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:02 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi204 to 7061M 2024-09-16T05:09:02.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:02 smithi204 ceph-mon[30933]: osdmap e84: 11 total, 11 up, 11 in 2024-09-16T05:09:02.608 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T05:09:02.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_1 2024-09-16T05:09:02.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:02 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:02.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:02 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:02.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:02 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi204 2024-09-16T05:09:02.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:02 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi204 to 7061M 2024-09-16T05:09:02.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:02 smithi032 ceph-mon[27388]: osdmap e84: 11 total, 11 up, 11 in 2024-09-16T05:09:02.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:02 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:02.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:02 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:02.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:02 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi204 2024-09-16T05:09:02.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:02 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi204 to 7061M 2024-09-16T05:09:02.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:02 smithi196 ceph-mon[30802]: osdmap e84: 11 total, 11 up, 11 in 2024-09-16T05:09:02.881 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:09:03.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:03 smithi032 ceph-mon[27388]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:03.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:03 smithi204 ceph-mon[30933]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:03.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:03 smithi196 ceph-mon[30802]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:04 smithi204 ceph-mon[30933]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:09:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:09:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:09:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:04 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:04.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:04 smithi032 ceph-mon[27388]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:09:04.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:09:04.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:09:04.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:04 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:04 smithi196 ceph-mon[30802]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:09:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T05:09:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T05:09:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:04 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:05.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:05 smithi032 ceph-mon[27388]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:05.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:05 smithi204 ceph-mon[30933]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:05.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:05 smithi196 ceph-mon[30802]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/3891249311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]: dispatch 2024-09-16T05:09:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:06 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]: dispatch 2024-09-16T05:09:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:06 smithi032 ceph-mon[27388]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]': finished 2024-09-16T05:09:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:06 smithi032 ceph-mon[27388]: osdmap e85: 12 total, 11 up, 12 in 2024-09-16T05:09:06.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:06 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/3891249311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]: dispatch 2024-09-16T05:09:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:06 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]: dispatch 2024-09-16T05:09:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:06 smithi204 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]': finished 2024-09-16T05:09:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:06 smithi204 ceph-mon[30933]: osdmap e85: 12 total, 11 up, 12 in 2024-09-16T05:09:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:06 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/3891249311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]: dispatch 2024-09-16T05:09:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:06 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]: dispatch 2024-09-16T05:09:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:06 smithi196 ceph-mon[30802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3"}]': finished 2024-09-16T05:09:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:06 smithi196 ceph-mon[30802]: osdmap e85: 12 total, 11 up, 12 in 2024-09-16T05:09:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:06 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:07.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:07 smithi032 ceph-mon[27388]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:07.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/295823572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:09:07.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:07 smithi204 ceph-mon[30933]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/295823572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:09:07.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:07 smithi196 ceph-mon[30802]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:07.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/295823572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T05:09:09.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:09 smithi032 ceph-mon[27388]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:09.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:09 smithi204 ceph-mon[30933]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:09.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:09 smithi196 ceph-mon[30802]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:11 smithi196 ceph-mon[30802]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:11.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:11 smithi032 ceph-mon[27388]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:11.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:11 smithi204 ceph-mon[30933]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:13.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:13 smithi204 ceph-mon[30933]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:13.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:13 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T05:09:13.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:13 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:13.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:13 smithi196 ceph-mon[30802]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:13.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:13 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T05:09:13.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:13 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:13.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:13 smithi032 ceph-mon[27388]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:13.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:13 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T05:09:13.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:13 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:14.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:14 smithi204 ceph-mon[30933]: Deploying daemon osd.11 on smithi204 2024-09-16T05:09:14.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:14 smithi196 ceph-mon[30802]: Deploying daemon osd.11 on smithi204 2024-09-16T05:09:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:14 smithi032 ceph-mon[27388]: Deploying daemon osd.11 on smithi204 2024-09-16T05:09:15.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:15 smithi204 ceph-mon[30933]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:15.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:15 smithi196 ceph-mon[30802]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:15 smithi032 ceph-mon[27388]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:16.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=4106213186,config=2788817630,health=3897446751,kv=822144132}) 2024-09-16T05:09:16.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=197125943}) 2024-09-16T05:09:16.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=77,logm=23} crc {kv=3223005659,logm=485059369}) 2024-09-16T05:09:16.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851714808}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273431482}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=498890668}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4280004769}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270086492}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4011445249}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=493482871}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085697295}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195277165}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753883167}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270693802}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768997742}) 2024-09-16T05:09:16.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350345647}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469076642}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2198343564}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=55,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=21} crc {logm=665092542,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=503714355}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2831918342}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=44,mon_config_key=56} crc {mgrstat=2636593926,mon_config_key=394777526}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3453223737}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2252506558}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=75} crc {mon_config_key=1139659332,monmap=2256977890,osd_metadata=1015557338,osd_pg_creating=2000802784,osdmap=4124408805}) 2024-09-16T05:09:16.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=925685842}) 2024-09-16T05:09:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=4106213186,config=2788817630,health=3897446751,kv=822144132}) 2024-09-16T05:09:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=197125943}) 2024-09-16T05:09:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=77,logm=23} crc {kv=3223005659,logm=485059369}) 2024-09-16T05:09:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851714808}) 2024-09-16T05:09:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273431482}) 2024-09-16T05:09:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=498890668}) 2024-09-16T05:09:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4280004769}) 2024-09-16T05:09:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270086492}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4011445249}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=493482871}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085697295}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195277165}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753883167}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270693802}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768997742}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350345647}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469076642}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2198343564}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=55,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=21} crc {logm=665092542,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=503714355}) 2024-09-16T05:09:16.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2831918342}) 2024-09-16T05:09:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=44,mon_config_key=56} crc {mgrstat=2636593926,mon_config_key=394777526}) 2024-09-16T05:09:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3453223737}) 2024-09-16T05:09:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2252506558}) 2024-09-16T05:09:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=75} crc {mon_config_key=1139659332,monmap=2256977890,osd_metadata=1015557338,osd_pg_creating=2000802784,osdmap=4124408805}) 2024-09-16T05:09:16.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=925685842}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=4106213186,config=2788817630,health=3897446751,kv=822144132}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=197125943}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=77,logm=23} crc {kv=3223005659,logm=485059369}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3851714808}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273431482}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=498890668}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4280004769}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270086492}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4011445249}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=493482871}) 2024-09-16T05:09:16.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085697295}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195277165}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753883167}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1270693802}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1768997742}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350345647}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469076642}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2198343564}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=55,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=21} crc {logm=665092542,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=503714355}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=2831918342}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=44,mon_config_key=56} crc {mgrstat=2636593926,mon_config_key=394777526}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3453223737}) 2024-09-16T05:09:16.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2252506558}) 2024-09-16T05:09:16.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=75} crc {mon_config_key=1139659332,monmap=2256977890,osd_metadata=1015557338,osd_pg_creating=2000802784,osdmap=4124408805}) 2024-09-16T05:09:16.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=98} crc {osdmap=925685842}) 2024-09-16T05:09:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:17 smithi204 ceph-mon[30933]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:17.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:17 smithi196 ceph-mon[30802]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:17 smithi032 ceph-mon[27388]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:19.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:19 smithi204 ceph-mon[30933]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:19.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:09:19.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:19.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:19.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:19 smithi196 ceph-mon[30802]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:19.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:09:19.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:19.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:19 smithi032 ceph-mon[27388]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:09:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:20.708 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:20 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:20.708 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:20 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:20.709 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:20 smithi204 ceph-mon[30933]: osdmap e86: 12 total, 11 up, 12 in 2024-09-16T05:09:20.709 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:20 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:20 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:20 smithi196 ceph-mon[30802]: osdmap e86: 12 total, 11 up, 12 in 2024-09-16T05:09:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:20 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:20 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:20 smithi032 ceph-mon[27388]: osdmap e86: 12 total, 11 up, 12 in 2024-09-16T05:09:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:21.541 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:21 smithi204 ceph-mon[30933]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:21.541 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:21 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:09:21.541 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:21 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:09:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:21 smithi196 ceph-mon[30802]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:21 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:09:21.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:21 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:09:21.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:21 smithi032 ceph-mon[27388]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:21.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:21 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:09:21.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:21 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:09:22.470 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 11 on host 'smithi204' 2024-09-16T05:09:23.162 DEBUG:teuthology.orchestra.run.smithi204:osd.11> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.11.service 2024-09-16T05:09:23.165 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-16T05:09:23.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T05:09:23.433 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:23 smithi196 ceph-mon[30802]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.807 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:23 smithi032 ceph-mon[27388]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:23.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:23 smithi204 ceph-mon[30933]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:23.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:23.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:24.209 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:09:24.470 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:09:24 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:09:24.179+0000 7fed196a9640 -1 osd.11 0 log_to_monitors true 2024-09-16T05:09:24.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:24 smithi196 ceph-mon[30802]: from='osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T05:09:24.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:24 smithi196 ceph-mon[30802]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T05:09:24.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1849501737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T05:09:24.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:24 smithi204 ceph-mon[30933]: from='osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T05:09:24.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:24 smithi204 ceph-mon[30933]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T05:09:24.747 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1849501737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T05:09:24.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:24 smithi032 ceph-mon[27388]: from='osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T05:09:24.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:24 smithi032 ceph-mon[27388]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T05:09:24.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1849501737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T05:09:24.922 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":86,"num_osds":12,"num_up_osds":11,"osd_up_since":1726463340,"num_in_osds":12,"osd_in_since":1726463345,"num_remapped_pgs":0} 2024-09-16T05:09:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:25 smithi196 ceph-mon[30802]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:25 smithi196 ceph-mon[30802]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T05:09:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:25 smithi196 ceph-mon[30802]: osdmap e87: 12 total, 11 up, 12 in 2024-09-16T05:09:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:25 smithi196 ceph-mon[30802]: from='osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:09:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:25.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:25 smithi196 ceph-mon[30802]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:09:25.849 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:25 smithi204 ceph-mon[30933]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:25.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:25 smithi204 ceph-mon[30933]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T05:09:25.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:25 smithi204 ceph-mon[30933]: osdmap e87: 12 total, 11 up, 12 in 2024-09-16T05:09:25.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:25 smithi204 ceph-mon[30933]: from='osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:09:25.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:25.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:25 smithi204 ceph-mon[30933]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:09:25.850 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:09:25 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:09:25.475+0000 7fed1562c640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T05:09:25.850 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:09:25 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:09:25.487+0000 7fed11454640 -1 osd.11 88 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:09:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:25 smithi032 ceph-mon[27388]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:25 smithi032 ceph-mon[27388]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T05:09:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:25 smithi032 ceph-mon[27388]: osdmap e87: 12 total, 11 up, 12 in 2024-09-16T05:09:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:25 smithi032 ceph-mon[27388]: from='osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:09:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:25 smithi032 ceph-mon[27388]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-09-16T05:09:25.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T05:09:26.197 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:26.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:09:26.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: osdmap e88: 12 total, 11 up, 12 in 2024-09-16T05:09:26.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:26.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:26.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.577 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:09:26.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:09:26.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: osdmap e88: 12 total, 11 up, 12 in 2024-09-16T05:09:26.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:26.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:26.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:09:26.852 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:09:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-09-16T05:09:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: osdmap e88: 12 total, 11 up, 12 in 2024-09-16T05:09:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:26.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:26.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T05:09:26.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:26.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:26.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:26.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:09:27.479 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":89,"num_osds":12,"num_up_osds":12,"osd_up_since":1726463366,"num_in_osds":12,"osd_in_since":1726463345,"num_remapped_pgs":1} 2024-09-16T05:09:27.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T05:09:27.737 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:27.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:27.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:27.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:27.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: Detected new or changed devices on smithi204 2024-09-16T05:09:27.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: Adjusting osd_memory_target on smithi204 to 5296M 2024-09-16T05:09:27.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191] boot 2024-09-16T05:09:27.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T05:09:27.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:27.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:27.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:27.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:27.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2585154674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: Detected new or changed devices on smithi204 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: Adjusting osd_memory_target on smithi204 to 5296M 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191] boot 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:27.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:27.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:27.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2585154674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T05:09:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:09:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: Detected new or changed devices on smithi204 2024-09-16T05:09:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: Adjusting osd_memory_target on smithi204 to 5296M 2024-09-16T05:09:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: osd.11 [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191] boot 2024-09-16T05:09:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T05:09:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T05:09:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:09:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:09:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:09:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2585154674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T05:09:28.385 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:09:28.386 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":90,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","created":"2024-09-16T05:02:35.583426+0000","modified":"2024-09-16T05:09:27.482751+0000","last_up_change":"2024-09-16T05:09:26.475677+0000","last_in_change":"2024-09-16T05:09:05.828775+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:05:16.445728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f3cfb49b-8223-4649-b211-a2e254248d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6803","nonce":406159384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6805","nonce":406159384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6809","nonce":406159384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6807","nonce":406159384}]},"public_addr":"172.21.15.32:6803/406159384","cluster_addr":"172.21.15.32:6805/406159384","heartbeat_back_addr":"172.21.15.32:6809/406159384","heartbeat_front_addr":"172.21.15.32:6807/406159384","state":["exists","up"]},{"osd":1,"uuid":"cfb43840-55b6-4d39-a325-2c5416ed97b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6811","nonce":976643951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6813","nonce":976643951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6817","nonce":976643951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6815","nonce":976643951}]},"public_addr":"172.21.15.32:6811/976643951","cluster_addr":"172.21.15.32:6813/976643951","heartbeat_back_addr":"172.21.15.32:6817/976643951","heartbeat_front_addr":"172.21.15.32:6815/976643951","state":["exists","up"]},{"osd":2,"uuid":"3158b0cc-849c-4383-8d4d-585cf0723e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6819","nonce":3508564664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6821","nonce":3508564664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6825","nonce":3508564664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6823","nonce":3508564664}]},"public_addr":"172.21.15.32:6819/3508564664","cluster_addr":"172.21.15.32:6821/3508564664","heartbeat_back_addr":"172.21.15.32:6825/3508564664","heartbeat_front_addr":"172.21.15.32:6823/3508564664","state":["exists","up"]},{"osd":3,"uuid":"71705a06-9102-4b97-be6a-4989306666d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6827","nonce":3040098358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6829","nonce":3040098358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6833","nonce":3040098358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6831","nonce":3040098358}]},"public_addr":"172.21.15.32:6827/3040098358","cluster_addr":"172.21.15.32:6829/3040098358","heartbeat_back_addr":"172.21.15.32:6833/3040098358","heartbeat_front_addr":"172.21.15.32:6831/3040098358","state":["exists","up"]},{"osd":4,"uuid":"ca8f47c4-c13f-4bde-b162-0e3b57bad1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6801","nonce":3219469655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6803","nonce":3219469655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6807","nonce":3219469655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6805","nonce":3219469655}]},"public_addr":"172.21.15.196:6801/3219469655","cluster_addr":"172.21.15.196:6803/3219469655","heartbeat_back_addr":"172.21.15.196:6807/3219469655","heartbeat_front_addr":"172.21.15.196:6805/3219469655","state":["exists","up"]},{"osd":5,"uuid":"b5b76271-0539-42dd-a9e7-a95afded2d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6809","nonce":3274041128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6811","nonce":3274041128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6815","nonce":3274041128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6813","nonce":3274041128}]},"public_addr":"172.21.15.196:6809/3274041128","cluster_addr":"172.21.15.196:6811/3274041128","heartbeat_back_addr":"172.21.15.196:6815/3274041128","heartbeat_front_addr":"172.21.15.196:6813/3274041128","state":["exists","up"]},{"osd":6,"uuid":"d3f2312b-cc04-48e4-993a-f08a948ba3ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6817","nonce":2509099861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6819","nonce":2509099861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6823","nonce":2509099861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6821","nonce":2509099861}]},"public_addr":"172.21.15.196:6817/2509099861","cluster_addr":"172.21.15.196:6819/2509099861","heartbeat_back_addr":"172.21.15.196:6823/2509099861","heartbeat_front_addr":"172.21.15.196:6821/2509099861","state":["exists","up"]},{"osd":7,"uuid":"fc7515ee-0a55-4b10-825e-8a4efa96d0e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6825","nonce":3304789846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6827","nonce":3304789846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6831","nonce":3304789846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6829","nonce":3304789846}]},"public_addr":"172.21.15.196:6825/3304789846","cluster_addr":"172.21.15.196:6827/3304789846","heartbeat_back_addr":"172.21.15.196:6831/3304789846","heartbeat_front_addr":"172.21.15.196:6829/3304789846","state":["exists","up"]},{"osd":8,"uuid":"e116a9c9-3c07-4b81-8b23-f2ba84916090","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6801","nonce":3463685812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6803","nonce":3463685812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6807","nonce":3463685812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6805","nonce":3463685812}]},"public_addr":"172.21.15.204:6801/3463685812","cluster_addr":"172.21.15.204:6803/3463685812","heartbeat_back_addr":"172.21.15.204:6807/3463685812","heartbeat_front_addr":"172.21.15.204:6805/3463685812","state":["exists","up"]},{"osd":9,"uuid":"06573a93-d7be-426b-97c4-4e088b2bfa1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6809","nonce":1139381345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6811","nonce":1139381345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6815","nonce":1139381345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6813","nonce":1139381345}]},"public_addr":"172.21.15.204:6809/1139381345","cluster_addr":"172.21.15.204:6811/1139381345","heartbeat_back_addr":"172.21.15.204:6815/1139381345","heartbeat_front_addr":"172.21.15.204:6813/1139381345","state":["exists","up"]},{"osd":10,"uuid":"388cc928-3439-4f37-a2dc-1f2a494f9846","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6817","nonce":726811436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6819","nonce":726811436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6823","nonce":726811436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6821","nonce":726811436}]},"public_addr":"172.21.15.204:6817/726811436","cluster_addr":"172.21.15.204:6819/726811436","heartbeat_back_addr":"172.21.15.204:6823/726811436","heartbeat_front_addr":"172.21.15.204:6821/726811436","state":["exists","up"]},{"osd":11,"uuid":"49911d7e-fe03-41a6-a25a-11a11c08f8b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6825","nonce":1668426191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6827","nonce":1668426191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6831","nonce":1668426191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6829","nonce":1668426191}]},"public_addr":"172.21.15.204:6825/1668426191","cluster_addr":"172.21.15.204:6827/1668426191","heartbeat_back_addr":"172.21.15.204:6831/1668426191","heartbeat_front_addr":"172.21.15.204:6829/1668426191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:20.017390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:44.000265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:12.882869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:44.536362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:41.236494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:14.243536+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:30.792659+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:01.019036+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:19.251545+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:33.037587+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:58.580094+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:25.147837+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2865075910":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6801/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/194694811":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/1290699598":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/2769196199":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/4153012034":"2024-09-17T05:03:00.800739+0000","172.21.15.32:6801/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/1722832924":"2024-09-17T05:03:00.800739+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:09:28.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:28 smithi032 ceph-mon[27388]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T05:09:28.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2039485233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:28.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:28 smithi204 ceph-mon[30933]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T05:09:28.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2039485233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:28.924 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T05:09:28.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-16T05:09:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:28 smithi196 ceph-mon[30802]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T05:09:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2039485233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:29.179 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:29.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:29 smithi032 ceph-mon[27388]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:29.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:29 smithi032 ceph-mon[27388]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T05:09:29.786 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-09-16T05:09:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:29 smithi204 ceph-mon[30933]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:29 smithi204 ceph-mon[30933]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T05:09:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:29 smithi196 ceph-mon[30802]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:29 smithi196 ceph-mon[30802]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T05:09:30.376 INFO:tasks.cephadm:Setting up client nodes... 2024-09-16T05:09:30.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T05:09:30.635 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:30.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:30 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:30.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:30 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:30.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:30 smithi032 ceph-mon[27388]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T05:09:30.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:30 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:30.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:30 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:30.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3358677518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T05:09:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:30 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:30 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:30 smithi204 ceph-mon[30933]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T05:09:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:30 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:30 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3358677518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T05:09:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:30 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:30 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:30 smithi196 ceph-mon[30802]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T05:09:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:30 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:30 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3358677518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T05:09:31.361 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-09-16T05:09:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCLvedmAYhrFRAAS30WoD8OHFPieEQtrIDlMA== 2024-09-16T05:09:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:31 smithi032 ceph-mon[27388]: pgmap v264: 1 pgs: 1 remapped+peering; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:31 smithi032 ceph-mon[27388]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T05:09:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3356361191' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3356361191' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:31 smithi204 ceph-mon[30933]: pgmap v264: 1 pgs: 1 remapped+peering; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:31 smithi204 ceph-mon[30933]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T05:09:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3356361191' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3356361191' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:31 smithi196 ceph-mon[30802]: pgmap v264: 1 pgs: 1 remapped+peering; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:31 smithi196 ceph-mon[30802]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T05:09:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3356361191' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3356361191' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:32.132 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-09-16T05:09:32.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-16T05:09:32.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T05:09:32.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T05:09:32.429 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:09:33.122 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-09-16T05:09:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCNvedmWuv9BhAAIlPaJhUx5zieUgqBefti9w== 2024-09-16T05:09:33.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:33 smithi196 ceph-mon[30802]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s, 0 objects/s recovering 2024-09-16T05:09:33.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.196:0/2592727267' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:33.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:33 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:33.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:33 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:33.825 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-16T05:09:33.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-16T05:09:33.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T05:09:33.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T05:09:33.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:33 smithi204 ceph-mon[30933]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s, 0 objects/s recovering 2024-09-16T05:09:33.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.196:0/2592727267' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:33.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:33 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:33.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:33 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:33.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:33 smithi032 ceph-mon[27388]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s, 0 objects/s recovering 2024-09-16T05:09:33.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.196:0/2592727267' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:33.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:33 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:33.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:33 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:34.120 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:09:34.858 INFO:teuthology.orchestra.run.smithi204.stdout:[client.2] 2024-09-16T05:09:34.859 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQCOvedmk8nfMhAA4I0UTEgvlgXA0XGUcUc7GA== 2024-09-16T05:09:35.519 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T05:09:35.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-09-16T05:09:35.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-16T05:09:35.596 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T05:09:35.597 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-16T05:09:35.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-16T05:09:35.860 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:35 smithi032 ceph-mon[27388]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-16T05:09:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/2656071457' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:35 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:35 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:35 smithi204 ceph-mon[30933]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-16T05:09:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/2656071457' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:35 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:35 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:35 smithi196 ceph-mon[30802]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-16T05:09:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/2656071457' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:35 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T05:09:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:35 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T05:09:36.588 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:09:37.284 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"flags":0,"active_gid":14149,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2705336016},{"type":"v1","addr":"172.21.15.32:6801","nonce":2705336016}]},"active_addr":"172.21.15.32:6801/2705336016","active_change":"2024-09-16T05:03:25.948144+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.32:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":4057034797}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":1326813644}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":3138182630}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":725655923}]}]} 2024-09-16T05:09:37.290 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-16T05:09:37.290 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-16T05:09:37.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T05:09:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:37.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:37 smithi032 ceph-mon[27388]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-16T05:09:37.768 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2560146919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T05:09:37.800 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:37 smithi204 ceph-mon[30933]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-16T05:09:37.800 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2560146919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T05:09:37.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:37 smithi196 ceph-mon[30802]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-16T05:09:37.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2560146919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T05:09:38.161 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:09:38.161 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":93,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","created":"2024-09-16T05:02:35.583426+0000","modified":"2024-09-16T05:09:30.500310+0000","last_up_change":"2024-09-16T05:09:26.475677+0000","last_in_change":"2024-09-16T05:09:05.828775+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:05:16.445728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f3cfb49b-8223-4649-b211-a2e254248d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6803","nonce":406159384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6805","nonce":406159384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6809","nonce":406159384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6807","nonce":406159384}]},"public_addr":"172.21.15.32:6803/406159384","cluster_addr":"172.21.15.32:6805/406159384","heartbeat_back_addr":"172.21.15.32:6809/406159384","heartbeat_front_addr":"172.21.15.32:6807/406159384","state":["exists","up"]},{"osd":1,"uuid":"cfb43840-55b6-4d39-a325-2c5416ed97b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6811","nonce":976643951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6813","nonce":976643951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6817","nonce":976643951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6815","nonce":976643951}]},"public_addr":"172.21.15.32:6811/976643951","cluster_addr":"172.21.15.32:6813/976643951","heartbeat_back_addr":"172.21.15.32:6817/976643951","heartbeat_front_addr":"172.21.15.32:6815/976643951","state":["exists","up"]},{"osd":2,"uuid":"3158b0cc-849c-4383-8d4d-585cf0723e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6819","nonce":3508564664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6821","nonce":3508564664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6825","nonce":3508564664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6823","nonce":3508564664}]},"public_addr":"172.21.15.32:6819/3508564664","cluster_addr":"172.21.15.32:6821/3508564664","heartbeat_back_addr":"172.21.15.32:6825/3508564664","heartbeat_front_addr":"172.21.15.32:6823/3508564664","state":["exists","up"]},{"osd":3,"uuid":"71705a06-9102-4b97-be6a-4989306666d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6827","nonce":3040098358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6829","nonce":3040098358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6833","nonce":3040098358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6831","nonce":3040098358}]},"public_addr":"172.21.15.32:6827/3040098358","cluster_addr":"172.21.15.32:6829/3040098358","heartbeat_back_addr":"172.21.15.32:6833/3040098358","heartbeat_front_addr":"172.21.15.32:6831/3040098358","state":["exists","up"]},{"osd":4,"uuid":"ca8f47c4-c13f-4bde-b162-0e3b57bad1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6801","nonce":3219469655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6803","nonce":3219469655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6807","nonce":3219469655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6805","nonce":3219469655}]},"public_addr":"172.21.15.196:6801/3219469655","cluster_addr":"172.21.15.196:6803/3219469655","heartbeat_back_addr":"172.21.15.196:6807/3219469655","heartbeat_front_addr":"172.21.15.196:6805/3219469655","state":["exists","up"]},{"osd":5,"uuid":"b5b76271-0539-42dd-a9e7-a95afded2d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6809","nonce":3274041128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6811","nonce":3274041128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6815","nonce":3274041128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6813","nonce":3274041128}]},"public_addr":"172.21.15.196:6809/3274041128","cluster_addr":"172.21.15.196:6811/3274041128","heartbeat_back_addr":"172.21.15.196:6815/3274041128","heartbeat_front_addr":"172.21.15.196:6813/3274041128","state":["exists","up"]},{"osd":6,"uuid":"d3f2312b-cc04-48e4-993a-f08a948ba3ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6817","nonce":2509099861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6819","nonce":2509099861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6823","nonce":2509099861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6821","nonce":2509099861}]},"public_addr":"172.21.15.196:6817/2509099861","cluster_addr":"172.21.15.196:6819/2509099861","heartbeat_back_addr":"172.21.15.196:6823/2509099861","heartbeat_front_addr":"172.21.15.196:6821/2509099861","state":["exists","up"]},{"osd":7,"uuid":"fc7515ee-0a55-4b10-825e-8a4efa96d0e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6825","nonce":3304789846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6827","nonce":3304789846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6831","nonce":3304789846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6829","nonce":3304789846}]},"public_addr":"172.21.15.196:6825/3304789846","cluster_addr":"172.21.15.196:6827/3304789846","heartbeat_back_addr":"172.21.15.196:6831/3304789846","heartbeat_front_addr":"172.21.15.196:6829/3304789846","state":["exists","up"]},{"osd":8,"uuid":"e116a9c9-3c07-4b81-8b23-f2ba84916090","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6801","nonce":3463685812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6803","nonce":3463685812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6807","nonce":3463685812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6805","nonce":3463685812}]},"public_addr":"172.21.15.204:6801/3463685812","cluster_addr":"172.21.15.204:6803/3463685812","heartbeat_back_addr":"172.21.15.204:6807/3463685812","heartbeat_front_addr":"172.21.15.204:6805/3463685812","state":["exists","up"]},{"osd":9,"uuid":"06573a93-d7be-426b-97c4-4e088b2bfa1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6809","nonce":1139381345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6811","nonce":1139381345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6815","nonce":1139381345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6813","nonce":1139381345}]},"public_addr":"172.21.15.204:6809/1139381345","cluster_addr":"172.21.15.204:6811/1139381345","heartbeat_back_addr":"172.21.15.204:6815/1139381345","heartbeat_front_addr":"172.21.15.204:6813/1139381345","state":["exists","up"]},{"osd":10,"uuid":"388cc928-3439-4f37-a2dc-1f2a494f9846","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6817","nonce":726811436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6819","nonce":726811436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6823","nonce":726811436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6821","nonce":726811436}]},"public_addr":"172.21.15.204:6817/726811436","cluster_addr":"172.21.15.204:6819/726811436","heartbeat_back_addr":"172.21.15.204:6823/726811436","heartbeat_front_addr":"172.21.15.204:6821/726811436","state":["exists","up"]},{"osd":11,"uuid":"49911d7e-fe03-41a6-a25a-11a11c08f8b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6825","nonce":1668426191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6827","nonce":1668426191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6831","nonce":1668426191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6829","nonce":1668426191}]},"public_addr":"172.21.15.204:6825/1668426191","cluster_addr":"172.21.15.204:6827/1668426191","heartbeat_back_addr":"172.21.15.204:6831/1668426191","heartbeat_front_addr":"172.21.15.204:6829/1668426191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:29.755221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:44.000265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:29.046992+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:44.536362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:41.236494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:14.243536+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:30.792659+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:01.019036+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:19.251545+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:33.037587+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:58.580094+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:25.147837+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2865075910":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6801/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/194694811":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/1290699598":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/2769196199":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/4153012034":"2024-09-17T05:03:00.800739+0000","172.21.15.32:6801/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/1722832924":"2024-09-17T05:03:00.800739+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:09:38.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2069604799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:38.901 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-16T05:09:38.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T05:09:38.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2069604799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:38.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2069604799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:39.161 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:39.773 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:09:40.747 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":94,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","created":"2024-09-16T05:02:35.583426+0000","modified":"2024-09-16T05:09:39.519126+0000","last_up_change":"2024-09-16T05:09:26.475677+0000","last_in_change":"2024-09-16T05:09:05.828775+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:05:16.445728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f3cfb49b-8223-4649-b211-a2e254248d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6803","nonce":406159384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6805","nonce":406159384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6809","nonce":406159384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6807","nonce":406159384}]},"public_addr":"172.21.15.32:6803/406159384","cluster_addr":"172.21.15.32:6805/406159384","heartbeat_back_addr":"172.21.15.32:6809/406159384","heartbeat_front_addr":"172.21.15.32:6807/406159384","state":["exists","up"]},{"osd":1,"uuid":"cfb43840-55b6-4d39-a325-2c5416ed97b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6811","nonce":976643951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6813","nonce":976643951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6817","nonce":976643951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":976643951},{"type":"v1","addr":"172.21.15.32:6815","nonce":976643951}]},"public_addr":"172.21.15.32:6811/976643951","cluster_addr":"172.21.15.32:6813/976643951","heartbeat_back_addr":"172.21.15.32:6817/976643951","heartbeat_front_addr":"172.21.15.32:6815/976643951","state":["exists","up"]},{"osd":2,"uuid":"3158b0cc-849c-4383-8d4d-585cf0723e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6819","nonce":3508564664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6821","nonce":3508564664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6825","nonce":3508564664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3508564664},{"type":"v1","addr":"172.21.15.32:6823","nonce":3508564664}]},"public_addr":"172.21.15.32:6819/3508564664","cluster_addr":"172.21.15.32:6821/3508564664","heartbeat_back_addr":"172.21.15.32:6825/3508564664","heartbeat_front_addr":"172.21.15.32:6823/3508564664","state":["exists","up"]},{"osd":3,"uuid":"71705a06-9102-4b97-be6a-4989306666d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6827","nonce":3040098358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6829","nonce":3040098358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6833","nonce":3040098358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6831","nonce":3040098358}]},"public_addr":"172.21.15.32:6827/3040098358","cluster_addr":"172.21.15.32:6829/3040098358","heartbeat_back_addr":"172.21.15.32:6833/3040098358","heartbeat_front_addr":"172.21.15.32:6831/3040098358","state":["exists","up"]},{"osd":4,"uuid":"ca8f47c4-c13f-4bde-b162-0e3b57bad1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6801","nonce":3219469655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6803","nonce":3219469655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6807","nonce":3219469655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6805","nonce":3219469655}]},"public_addr":"172.21.15.196:6801/3219469655","cluster_addr":"172.21.15.196:6803/3219469655","heartbeat_back_addr":"172.21.15.196:6807/3219469655","heartbeat_front_addr":"172.21.15.196:6805/3219469655","state":["exists","up"]},{"osd":5,"uuid":"b5b76271-0539-42dd-a9e7-a95afded2d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6809","nonce":3274041128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6811","nonce":3274041128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6815","nonce":3274041128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6813","nonce":3274041128}]},"public_addr":"172.21.15.196:6809/3274041128","cluster_addr":"172.21.15.196:6811/3274041128","heartbeat_back_addr":"172.21.15.196:6815/3274041128","heartbeat_front_addr":"172.21.15.196:6813/3274041128","state":["exists","up"]},{"osd":6,"uuid":"d3f2312b-cc04-48e4-993a-f08a948ba3ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6817","nonce":2509099861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6819","nonce":2509099861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6823","nonce":2509099861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6821","nonce":2509099861}]},"public_addr":"172.21.15.196:6817/2509099861","cluster_addr":"172.21.15.196:6819/2509099861","heartbeat_back_addr":"172.21.15.196:6823/2509099861","heartbeat_front_addr":"172.21.15.196:6821/2509099861","state":["exists","up"]},{"osd":7,"uuid":"fc7515ee-0a55-4b10-825e-8a4efa96d0e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6825","nonce":3304789846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6827","nonce":3304789846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6831","nonce":3304789846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3304789846},{"type":"v1","addr":"172.21.15.196:6829","nonce":3304789846}]},"public_addr":"172.21.15.196:6825/3304789846","cluster_addr":"172.21.15.196:6827/3304789846","heartbeat_back_addr":"172.21.15.196:6831/3304789846","heartbeat_front_addr":"172.21.15.196:6829/3304789846","state":["exists","up"]},{"osd":8,"uuid":"e116a9c9-3c07-4b81-8b23-f2ba84916090","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6801","nonce":3463685812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6803","nonce":3463685812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6807","nonce":3463685812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6805","nonce":3463685812}]},"public_addr":"172.21.15.204:6801/3463685812","cluster_addr":"172.21.15.204:6803/3463685812","heartbeat_back_addr":"172.21.15.204:6807/3463685812","heartbeat_front_addr":"172.21.15.204:6805/3463685812","state":["exists","up"]},{"osd":9,"uuid":"06573a93-d7be-426b-97c4-4e088b2bfa1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6809","nonce":1139381345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6811","nonce":1139381345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6815","nonce":1139381345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6813","nonce":1139381345}]},"public_addr":"172.21.15.204:6809/1139381345","cluster_addr":"172.21.15.204:6811/1139381345","heartbeat_back_addr":"172.21.15.204:6815/1139381345","heartbeat_front_addr":"172.21.15.204:6813/1139381345","state":["exists","up"]},{"osd":10,"uuid":"388cc928-3439-4f37-a2dc-1f2a494f9846","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6817","nonce":726811436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6819","nonce":726811436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6823","nonce":726811436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6821","nonce":726811436}]},"public_addr":"172.21.15.204:6817/726811436","cluster_addr":"172.21.15.204:6819/726811436","heartbeat_back_addr":"172.21.15.204:6823/726811436","heartbeat_front_addr":"172.21.15.204:6821/726811436","state":["exists","up"]},{"osd":11,"uuid":"49911d7e-fe03-41a6-a25a-11a11c08f8b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6825","nonce":1668426191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6827","nonce":1668426191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6831","nonce":1668426191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6829","nonce":1668426191}]},"public_addr":"172.21.15.204:6825/1668426191","cluster_addr":"172.21.15.204:6827/1668426191","heartbeat_back_addr":"172.21.15.204:6831/1668426191","heartbeat_front_addr":"172.21.15.204:6829/1668426191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:29.755221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:44.000265+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:29.046992+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:44.536362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:41.236494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:39.367915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:30.792659+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:01.019036+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:19.251545+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:33.037587+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:08:58.580094+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:09:25.147837+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2865075910":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6801/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/194694811":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/1290699598":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/2769196199":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/4153012034":"2024-09-17T05:03:00.800739+0000","172.21.15.32:6801/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/1722832924":"2024-09-17T05:03:00.800739+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:09:40.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:39 smithi196 ceph-mon[30802]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-16T05:09:40.748 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:39 smithi204 ceph-mon[30933]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-16T05:09:40.748 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:39 smithi032 ceph-mon[27388]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-16T05:09:40.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-16T05:09:40.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-16T05:09:40.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-16T05:09:40.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-16T05:09:40.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-16T05:09:40.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-16T05:09:40.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-16T05:09:40.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-16T05:09:40.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-16T05:09:40.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-16T05:09:40.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-16T05:09:40.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-16T05:09:40.801 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:40 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:40.802 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:40 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:40.802 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:40 smithi032 ceph-mon[27388]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T05:09:40.802 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1193806674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:40 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:40 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:40 smithi204 ceph-mon[30933]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T05:09:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1193806674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:40 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:40 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:40 smithi196 ceph-mon[30802]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T05:09:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1193806674' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:09:41.464 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:41.466 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:41.842 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:41 smithi032 ceph-mon[27388]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-16T05:09:41.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:41 smithi204 ceph-mon[30933]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-16T05:09:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:41 smithi196 ceph-mon[30802]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-16T05:09:42.621 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.622 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.624 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.624 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.624 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.628 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.628 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:42.631 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:43.833 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:43 smithi032 ceph-mon[27388]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:43.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:43 smithi204 ceph-mon[30933]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:43 smithi196 ceph-mon[30802]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:44.454 INFO:teuthology.orchestra.run.smithi032.stdout:249108103194 2024-09-16T05:09:44.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-16T05:09:44.587 INFO:teuthology.orchestra.run.smithi032.stdout:55834574905 2024-09-16T05:09:44.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-16T05:09:45.668 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:45 smithi032 ceph-mon[27388]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:45.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:45 smithi204 ceph-mon[30933]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:45 smithi196 ceph-mon[30802]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:46.065 INFO:teuthology.orchestra.run.smithi032.stdout:326417514513 2024-09-16T05:09:46.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-16T05:09:47.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:47 smithi204 ceph-mon[30933]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:47.802 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:47 smithi032 ceph-mon[27388]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:47.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:47 smithi196 ceph-mon[30802]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:48.402 INFO:teuthology.orchestra.run.smithi032.stdout:163208757285 2024-09-16T05:09:48.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-16T05:09:49.623 INFO:teuthology.orchestra.run.smithi032.stdout:115964117040 2024-09-16T05:09:49.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-16T05:09:49.870 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:49 smithi032 ceph-mon[27388]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:49 smithi204 ceph-mon[30933]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:49 smithi196 ceph-mon[30802]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:50 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:50 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:50 smithi032 ceph-mon[27388]: osdmap e95: 12 total, 12 up, 12 in 2024-09-16T05:09:50.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:50 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:50.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:50 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:50.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:50 smithi204 ceph-mon[30933]: osdmap e95: 12 total, 12 up, 12 in 2024-09-16T05:09:50.979 INFO:teuthology.orchestra.run.smithi032.stdout:210453397538 2024-09-16T05:09:50.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-16T05:09:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:50 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:50 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:50 smithi196 ceph-mon[30802]: osdmap e95: 12 total, 12 up, 12 in 2024-09-16T05:09:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:141733920811 2024-09-16T05:09:51.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-16T05:09:51.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:51 smithi032 ceph-mon[27388]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:51.884 INFO:teuthology.orchestra.run.smithi032.stdout:90194313269 2024-09-16T05:09:51.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-16T05:09:51.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:51 smithi204 ceph-mon[30933]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:51 smithi196 ceph-mon[30802]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:52.429 INFO:teuthology.orchestra.run.smithi032.stdout:382252089352 2024-09-16T05:09:52.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-16T05:09:53.224 INFO:teuthology.orchestra.run.smithi032.stdout:356482285580 2024-09-16T05:09:53.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-16T05:09:53.649 INFO:teuthology.orchestra.run.smithi032.stdout:34359738430 2024-09-16T05:09:53.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-16T05:09:53.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:53 smithi032 ceph-mon[27388]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:53.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:53 smithi204 ceph-mon[30933]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:53 smithi196 ceph-mon[30802]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:54.281 INFO:teuthology.orchestra.run.smithi032.stdout:283467841558 2024-09-16T05:09:54.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-16T05:09:54.469 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.475 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.479 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.479 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.483 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.488 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.489 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.489 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.494 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:54.495 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:09:55.371 INFO:teuthology.orchestra.run.smithi032.stdout:326417514515 2024-09-16T05:09:55.701 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:55 smithi032 ceph-mon[27388]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:55.701 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2047876685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T05:09:55.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:55 smithi204 ceph-mon[30933]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:55.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2047876685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T05:09:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:55 smithi196 ceph-mon[30802]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2047876685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T05:09:56.359 INFO:teuthology.orchestra.run.smithi032.stdout:356482285581 2024-09-16T05:09:56.359 INFO:teuthology.orchestra.run.smithi032.stdout:90194313270 2024-09-16T05:09:56.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1202108868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T05:09:56.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1327168424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T05:09:56.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1202108868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T05:09:56.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1327168424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T05:09:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1202108868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T05:09:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1327168424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T05:09:57.385 INFO:teuthology.orchestra.run.smithi032.stdout:141733920813 2024-09-16T05:09:57.457 INFO:teuthology.orchestra.run.smithi032.stdout:34359738433 2024-09-16T05:09:57.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:57 smithi032 ceph-mon[27388]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:57.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3928970309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T05:09:57.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2377525032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T05:09:57.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:57 smithi204 ceph-mon[30933]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:57.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3928970309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T05:09:57.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2377525032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T05:09:57.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:57 smithi196 ceph-mon[30802]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:57.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3928970309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T05:09:57.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2377525032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T05:09:58.476 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514513 got 326417514515 for osd.9 2024-09-16T05:09:58.477 DEBUG:teuthology.parallel:result is None 2024-09-16T05:09:58.818 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920811 got 141733920813 for osd.4 2024-09-16T05:09:58.818 DEBUG:teuthology.parallel:result is None 2024-09-16T05:09:59.205 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738430 got 34359738433 for osd.0 2024-09-16T05:09:59.206 DEBUG:teuthology.parallel:result is None 2024-09-16T05:09:59.558 INFO:teuthology.orchestra.run.smithi032.stdout:210453397540 2024-09-16T05:09:59.799 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:59 smithi032 ceph-mon[27388]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:59.799 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:59 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:09:59.799 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:59 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:09:59.799 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:09:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/308782626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T05:09:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:59 smithi204 ceph-mon[30933]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:59 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:09:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:59 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:09:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:09:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/308782626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T05:09:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:59 smithi196 ceph-mon[30802]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:09:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:59 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:09:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:59 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:09:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:09:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/308782626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T05:10:00.673 INFO:teuthology.orchestra.run.smithi032.stdout:55834574908 2024-09-16T05:10:00.758 INFO:teuthology.orchestra.run.smithi032.stdout:163208757287 2024-09-16T05:10:00.898 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:00 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:00.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:00 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:00.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:00 smithi032 ceph-mon[27388]: osdmap e96: 12 total, 12 up, 12 in 2024-09-16T05:10:00.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:00 smithi032 ceph-mon[27388]: overall HEALTH_OK 2024-09-16T05:10:00.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:00 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:00.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:00 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:00 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:00 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:00 smithi204 ceph-mon[30933]: osdmap e96: 12 total, 12 up, 12 in 2024-09-16T05:10:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:00 smithi204 ceph-mon[30933]: overall HEALTH_OK 2024-09-16T05:10:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:00 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:00 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:00 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:00 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:00 smithi196 ceph-mon[30802]: osdmap e96: 12 total, 12 up, 12 in 2024-09-16T05:10:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:00 smithi196 ceph-mon[30802]: overall HEALTH_OK 2024-09-16T05:10:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:00 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:00 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:01.579 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757285 got 163208757287 for osd.5 2024-09-16T05:10:01.580 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:01 smithi032 ceph-mon[27388]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:01 smithi032 ceph-mon[27388]: osdmap e97: 12 total, 12 up, 12 in 2024-09-16T05:10:01.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/698892409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T05:10:01.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/820568880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T05:10:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:01 smithi204 ceph-mon[30933]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:01 smithi204 ceph-mon[30933]: osdmap e97: 12 total, 12 up, 12 in 2024-09-16T05:10:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/698892409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T05:10:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/820568880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T05:10:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:01 smithi196 ceph-mon[30802]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:01 smithi196 ceph-mon[30802]: osdmap e97: 12 total, 12 up, 12 in 2024-09-16T05:10:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/698892409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T05:10:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/820568880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T05:10:02.007 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285580 got 356482285581 for osd.10 2024-09-16T05:10:02.007 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:02.625 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574905 got 55834574908 for osd.1 2024-09-16T05:10:02.625 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:02.981 INFO:teuthology.orchestra.run.smithi032.stdout:382252089354 2024-09-16T05:10:03.176 INFO:teuthology.orchestra.run.smithi032.stdout:115964117042 2024-09-16T05:10:03.903 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:03 smithi032 ceph-mon[27388]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:03.903 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:03 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:03.903 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:03 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:03.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:03 smithi032 ceph-mon[27388]: osdmap e98: 12 total, 12 up, 12 in 2024-09-16T05:10:03.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3452956243' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T05:10:03.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2027198807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T05:10:03.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:03 smithi204 ceph-mon[30933]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:03 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:03 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:03 smithi204 ceph-mon[30933]: osdmap e98: 12 total, 12 up, 12 in 2024-09-16T05:10:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3452956243' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T05:10:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2027198807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T05:10:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:03 smithi196 ceph-mon[30802]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:03 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:03 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:03 smithi196 ceph-mon[30802]: osdmap e98: 12 total, 12 up, 12 in 2024-09-16T05:10:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3452956243' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T05:10:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2027198807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T05:10:04.065 INFO:tasks.cephadm.ceph_manager.ceph:need seq 382252089352 got 382252089354 for osd.11 2024-09-16T05:10:04.065 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:04.655 INFO:tasks.cephadm.ceph_manager.ceph:need seq 90194313269 got 90194313270 for osd.2 2024-09-16T05:10:04.655 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:04.723 INFO:teuthology.orchestra.run.smithi032.stdout:249108103198 2024-09-16T05:10:04.723 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:04 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:04.724 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:04 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:04.724 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:04 smithi032 ceph-mon[27388]: osdmap e99: 12 total, 12 up, 12 in 2024-09-16T05:10:04.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:04 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:04.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:04 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:04.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:04 smithi204 ceph-mon[30933]: osdmap e99: 12 total, 12 up, 12 in 2024-09-16T05:10:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:04 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:04 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:04 smithi196 ceph-mon[30802]: osdmap e99: 12 total, 12 up, 12 in 2024-09-16T05:10:05.402 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103194 got 249108103198 for osd.7 2024-09-16T05:10:05.402 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:05.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:05 smithi032 ceph-mon[27388]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:05.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1722424783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T05:10:05.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:05 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:05.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:05 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:05.840 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117040 got 115964117042 for osd.3 2024-09-16T05:10:05.840 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:05 smithi196 ceph-mon[30802]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1722424783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T05:10:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:05 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:05.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:05 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:05.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:05 smithi204 ceph-mon[30933]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1722424783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T05:10:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:05 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:05 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:06.229 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397538 got 210453397540 for osd.6 2024-09-16T05:10:06.229 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:06.233 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:06.909 INFO:teuthology.orchestra.run.smithi032.stdout:283467841561 2024-09-16T05:10:06.909 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:06 smithi032 ceph-mon[27388]: osdmap e100: 12 total, 12 up, 12 in 2024-09-16T05:10:06.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:06 smithi204 ceph-mon[30933]: osdmap e100: 12 total, 12 up, 12 in 2024-09-16T05:10:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:06 smithi196 ceph-mon[30802]: osdmap e100: 12 total, 12 up, 12 in 2024-09-16T05:10:07.580 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841558 got 283467841561 for osd.8 2024-09-16T05:10:07.580 DEBUG:teuthology.parallel:result is None 2024-09-16T05:10:07.580 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T05:10:07.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:10:07.874 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:07.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:07 smithi204 ceph-mon[30933]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:07.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3844006360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T05:10:07.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:07 smithi032 ceph-mon[27388]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:07.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3844006360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T05:10:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:07 smithi196 ceph-mon[30802]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3844006360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T05:10:08.506 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:10:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:10:09.099 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-16T05:10:08.029103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":346392,"kb_used_data":13688,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124595432,"statfs":{"total":1151940427776,"available":1151585722368,"internally_reserved":0,"allocated":14016512,"data_stored":7178942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001361"},"pg_stats":[{"pgid":"1.0","version":"35'139","reported_seq":379,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-16T05:10:06.433578+0000","last_change":"2024-09-16T05:09:29.571554+0000","last_active":"2024-09-16T05:10:06.433578+0000","last_peered":"2024-09-16T05:10:06.433578+0000","last_clean":"2024-09-16T05:10:06.433578+0000","last_became_active":"2024-09-16T05:09:29.279282+0000","last_became_peered":"2024-09-16T05:09:29.279282+0000","last_unstale":"2024-09-16T05:10:06.433578+0000","last_undegraded":"2024-09-16T05:10:06.433578+0000","last_fullsized":"2024-09-16T05:10:06.433578+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:09:20.060218+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:09:20.060218+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:10:48.724228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1622016,"data_stored":1045166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28084,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28088,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841561,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28068,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1064960,"data_stored":501484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103199,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1609728,"data_stored":1037789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28092,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28084,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28088,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28092,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32244,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32248,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:10:09.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:10:09.357 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:09.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:09 smithi032 ceph-mon[27388]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:09.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:09 smithi032 ceph-mon[27388]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:09.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:09 smithi204 ceph-mon[30933]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:09.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:09 smithi204 ceph-mon[30933]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:09 smithi196 ceph-mon[30802]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:09 smithi196 ceph-mon[30802]: from='client.14649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:10.024 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:10:10.024 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:10:10.743 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-16T05:10:08.029103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":346392,"kb_used_data":13688,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124595432,"statfs":{"total":1151940427776,"available":1151585722368,"internally_reserved":0,"allocated":14016512,"data_stored":7178942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001361"},"pg_stats":[{"pgid":"1.0","version":"35'139","reported_seq":379,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-16T05:10:06.433578+0000","last_change":"2024-09-16T05:09:29.571554+0000","last_active":"2024-09-16T05:10:06.433578+0000","last_peered":"2024-09-16T05:10:06.433578+0000","last_clean":"2024-09-16T05:10:06.433578+0000","last_became_active":"2024-09-16T05:09:29.279282+0000","last_became_peered":"2024-09-16T05:09:29.279282+0000","last_unstale":"2024-09-16T05:10:06.433578+0000","last_undegraded":"2024-09-16T05:10:06.433578+0000","last_fullsized":"2024-09-16T05:10:06.433578+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:09:20.060218+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:09:20.060218+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:10:48.724228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1622016,"data_stored":1045166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28084,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28088,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841561,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28068,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1064960,"data_stored":501484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103199,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1609728,"data_stored":1037789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397542,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28092,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28084,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28088,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28092,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32244,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32248,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1077248,"data_stored":508861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:10:10.744 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T05:10:10.744 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T05:10:10.744 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-16T05:10:10.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-16T05:10:11.004 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:11.679 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:10:11.679 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T05:10:11.679 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:11 smithi032 ceph-mon[27388]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:11.679 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:11 smithi032 ceph-mon[27388]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:11.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:11 smithi204 ceph-mon[30933]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:11.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:11 smithi204 ceph-mon[30933]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:11.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:11 smithi196 ceph-mon[30802]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:11.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:11 smithi196 ceph-mon[30802]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:12.441 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-16T05:10:12.441 INFO:tasks.cephadm:Setup complete, yielding 2024-09-16T05:10:12.442 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T05:10:12.453 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T05:10:12.453 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-09-16T05:10:12.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-09-16T05:10:12.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3098661888' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout: "active": true, 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout: "last_optimize_duration": "0:00:00.000313", 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout: "last_optimize_started": "Mon Sep 16 05:09:25 2024", 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": "upmap", 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout: "no_optimization_needed": true, 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout: "plans": [] 2024-09-16T05:10:12.749 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:12.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-09-16T05:10:12.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3098661888' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T05:10:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3098661888' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T05:10:13.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-09-16T05:10:13.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:13 smithi032 ceph-mon[27388]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:13.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:13 smithi032 ceph-mon[27388]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:13.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:13 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-09-16T05:10:13.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:13 smithi204 ceph-mon[30933]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:13.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:13 smithi204 ceph-mon[30933]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:13.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:13 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-09-16T05:10:13.970 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T05:10:13.980 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 3, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T05:10:13.981 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore queue throttle high multiple': 2, 'filestore queue throttle max multiple': 10, 'journal throttle high multiple': 2, 'journal throttle max multiple': 10, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 9, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}} 2024-09-16T05:10:13.981 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 3, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T05:10:13.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T05:10:13.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:13 smithi196 ceph-mon[30802]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:13 smithi196 ceph-mon[30802]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:13 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-09-16T05:10:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:14.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:14.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-09-16T05:10:14.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: osdmap e101: 12 total, 12 up, 12 in 2024-09-16T05:10:14.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-09-16T05:10:14.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:14.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:10:14.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:14.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:10:14.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:10:14.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:14.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:14.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-09-16T05:10:14.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: osdmap e101: 12 total, 12 up, 12 in 2024-09-16T05:10:14.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-09-16T05:10:14.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:14.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:10:14.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:14.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:10:14.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:10:14.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: osdmap e101: 12 total, 12 up, 12 in 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:10:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:10:14.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:15.624 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T05:10:15.624 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T05:10:15.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:10:15.882 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:16.132 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:15 smithi032 ceph-mon[27388]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:16.132 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3986798675' entity='client.admin' 2024-09-16T05:10:16.132 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:10:16.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:10:16.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:10:16.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:15 smithi204 ceph-mon[30933]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3986798675' entity='client.admin' 2024-09-16T05:10:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:10:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:10:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:10:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:15 smithi196 ceph-mon[30802]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3986798675' entity='client.admin' 2024-09-16T05:10:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:10:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:10:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:10:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:10:16.542 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:10:16.542 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:10:17.281 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T05:10:16.030479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":346540,"kb_used_data":13880,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124595284,"statfs":{"total":1151940427776,"available":1151585570816,"internally_reserved":0,"allocated":14213120,"data_stored":7307030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001828"},"pg_stats":[{"pgid":"1.0","version":"35'139","reported_seq":379,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-16T05:10:06.433578+0000","last_change":"2024-09-16T05:09:29.571554+0000","last_active":"2024-09-16T05:10:06.433578+0000","last_peered":"2024-09-16T05:10:06.433578+0000","last_clean":"2024-09-16T05:10:06.433578+0000","last_became_active":"2024-09-16T05:09:29.279282+0000","last_became_peered":"2024-09-16T05:09:29.279282+0000","last_unstale":"2024-09-16T05:10:06.433578+0000","last_undegraded":"2024-09-16T05:10:06.433578+0000","last_fullsized":"2024-09-16T05:10:06.433578+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:09:20.060218+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:09:20.060218+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:10:48.724228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1622016,"data_stored":1045166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514521,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841563,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28092,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103200,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1622016,"data_stored":1045166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28092,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28092,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313274,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32252,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:10:17.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:10:17.541 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:17 smithi032 ceph-mon[27388]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:17.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:17 smithi032 ceph-mon[27388]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:17 smithi204 ceph-mon[30933]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:17 smithi204 ceph-mon[30933]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:10:18.193 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:10:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:17 smithi196 ceph-mon[30802]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:17 smithi196 ceph-mon[30802]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:18.858 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-16T05:10:18.030907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":346596,"kb_used_data":13940,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124595228,"statfs":{"total":1151940427776,"available":1151585513472,"internally_reserved":0,"allocated":14274560,"data_stored":7352295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001804"},"pg_stats":[{"pgid":"1.0","version":"35'139","reported_seq":381,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T05:10:13.634296+0000","last_change":"2024-09-16T05:09:29.571554+0000","last_active":"2024-09-16T05:10:13.634296+0000","last_peered":"2024-09-16T05:10:13.634296+0000","last_clean":"2024-09-16T05:10:13.634296+0000","last_became_active":"2024-09-16T05:09:29.279282+0000","last_became_peered":"2024-09-16T05:09:29.279282+0000","last_unstale":"2024-09-16T05:10:13.634296+0000","last_undegraded":"2024-09-16T05:10:13.634296+0000","last_fullsized":"2024-09-16T05:10:13.634296+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:09:20.060218+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:09:20.060218+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:10:48.724228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089358,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1634304,"data_stored":1054219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514521,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841563,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28092,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1089536,"data_stored":516238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103201,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1634304,"data_stored":1054219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117046,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:10:18.858 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T05:10:18.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-16T05:10:19.112 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:19.727 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 101 2024-09-16T05:10:19.727 INFO:teuthology.orchestra.run.smithi032.stdout:fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-09-16T05:02:35.583426+0000 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-09-16T05:10:13.630435+0000 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:crush_version 27 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:full_ratio 0.95 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:backfillfull_ratio 0.9 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:nearfull_ratio 0.85 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:require_min_compat_client luminous 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:min_compat_client jewel 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:require_osd_release squid 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:stretch_mode_enabled false 2024-09-16T05:10:19.728 INFO:teuthology.orchestra.run.smithi032.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:max_osd 12 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 up in weight 1 up_from 8 up_thru 61 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384] [v2:172.21.15.32:6804/406159384,v1:172.21.15.32:6805/406159384] exists,up f3cfb49b-8223-4649-b211-a2e254248d88 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951] [v2:172.21.15.32:6812/976643951,v1:172.21.15.32:6813/976643951] exists,up cfb43840-55b6-4d39-a325-2c5416ed97b4 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664] [v2:172.21.15.32:6820/3508564664,v1:172.21.15.32:6821/3508564664] exists,up 3158b0cc-849c-4383-8d4d-585cf0723e70 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358] [v2:172.21.15.32:6828/3040098358,v1:172.21.15.32:6829/3040098358] exists,up 71705a06-9102-4b97-be6a-4989306666d2 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655] [v2:172.21.15.196:6802/3219469655,v1:172.21.15.196:6803/3219469655] exists,up ca8f47c4-c13f-4bde-b162-0e3b57bad1b4 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 up in weight 1 up_from 38 up_thru 41 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128] [v2:172.21.15.196:6810/3274041128,v1:172.21.15.196:6811/3274041128] exists,up b5b76271-0539-42dd-a9e7-a95afded2d59 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 up in weight 1 up_from 49 up_thru 52 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861] [v2:172.21.15.196:6818/2509099861,v1:172.21.15.196:6819/2509099861] exists,up d3f2312b-cc04-48e4-993a-f08a948ba3ca 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 up in weight 1 up_from 58 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846] [v2:172.21.15.196:6826/3304789846,v1:172.21.15.196:6827/3304789846] exists,up fc7515ee-0a55-4b10-825e-8a4efa96d0e2 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812] [v2:172.21.15.204:6802/3463685812,v1:172.21.15.204:6803/3463685812] exists,up e116a9c9-3c07-4b81-8b23-f2ba84916090 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345] [v2:172.21.15.204:6810/1139381345,v1:172.21.15.204:6811/1139381345] exists,up 06573a93-d7be-426b-97c4-4e088b2bfa1a 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436] [v2:172.21.15.204:6818/726811436,v1:172.21.15.204:6819/726811436] exists,up 388cc928-3439-4f37-a2dc-1f2a494f9846 2024-09-16T05:10:19.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11 up in weight 1 up_from 89 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191] [v2:172.21.15.204:6826/1668426191,v1:172.21.15.204:6827/1668426191] exists,up 49911d7e-fe03-41a6-a25a-11a11c08f8b3 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/2865075910 expires 2024-09-17T05:03:25.948027+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6801/1180926382 expires 2024-09-17T05:03:25.948027+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6800/1180926382 expires 2024-09-17T05:03:25.948027+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/194694811 expires 2024-09-17T05:03:25.948027+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/1290699598 expires 2024-09-17T05:03:25.948027+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6800/178109651 expires 2024-09-17T05:03:00.800739+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/2769196199 expires 2024-09-17T05:03:00.800739+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/4153012034 expires 2024-09-17T05:03:00.800739+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6801/178109651 expires 2024-09-17T05:03:00.800739+0000 2024-09-16T05:10:19.730 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/1722832924 expires 2024-09-17T05:03:00.800739+0000 2024-09-16T05:10:20.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:19 smithi032 ceph-mon[27388]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:20.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:19 smithi032 ceph-mon[27388]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:20.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1614472176' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T05:10:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:19 smithi204 ceph-mon[30933]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:19 smithi204 ceph-mon[30933]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1614472176' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T05:10:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:19 smithi196 ceph-mon[30802]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:19 smithi196 ceph-mon[30802]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1614472176' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T05:10:20.390 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 61 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6802/406159384,v1:172.21.15.32:6803/406159384] [v2:172.21.15.32:6804/406159384,v1:172.21.15.32:6805/406159384] exists,up f3cfb49b-8223-4649-b211-a2e254248d88', 'osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6810/976643951,v1:172.21.15.32:6811/976643951] [v2:172.21.15.32:6812/976643951,v1:172.21.15.32:6813/976643951] exists,up cfb43840-55b6-4d39-a325-2c5416ed97b4', 'osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6818/3508564664,v1:172.21.15.32:6819/3508564664] [v2:172.21.15.32:6820/3508564664,v1:172.21.15.32:6821/3508564664] exists,up 3158b0cc-849c-4383-8d4d-585cf0723e70', 'osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6826/3040098358,v1:172.21.15.32:6827/3040098358] [v2:172.21.15.32:6828/3040098358,v1:172.21.15.32:6829/3040098358] exists,up 71705a06-9102-4b97-be6a-4989306666d2', 'osd.4 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6800/3219469655,v1:172.21.15.196:6801/3219469655] [v2:172.21.15.196:6802/3219469655,v1:172.21.15.196:6803/3219469655] exists,up ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'osd.5 up in weight 1 up_from 38 up_thru 41 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6808/3274041128,v1:172.21.15.196:6809/3274041128] [v2:172.21.15.196:6810/3274041128,v1:172.21.15.196:6811/3274041128] exists,up b5b76271-0539-42dd-a9e7-a95afded2d59', 'osd.6 up in weight 1 up_from 49 up_thru 52 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6816/2509099861,v1:172.21.15.196:6817/2509099861] [v2:172.21.15.196:6818/2509099861,v1:172.21.15.196:6819/2509099861] exists,up d3f2312b-cc04-48e4-993a-f08a948ba3ca', 'osd.7 up in weight 1 up_from 58 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6824/3304789846,v1:172.21.15.196:6825/3304789846] [v2:172.21.15.196:6826/3304789846,v1:172.21.15.196:6827/3304789846] exists,up fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'osd.8 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6800/3463685812,v1:172.21.15.204:6801/3463685812] [v2:172.21.15.204:6802/3463685812,v1:172.21.15.204:6803/3463685812] exists,up e116a9c9-3c07-4b81-8b23-f2ba84916090', 'osd.9 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6808/1139381345,v1:172.21.15.204:6809/1139381345] [v2:172.21.15.204:6810/1139381345,v1:172.21.15.204:6811/1139381345] exists,up 06573a93-d7be-426b-97c4-4e088b2bfa1a', 'osd.10 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6816/726811436,v1:172.21.15.204:6817/726811436] [v2:172.21.15.204:6818/726811436,v1:172.21.15.204:6819/726811436] exists,up 388cc928-3439-4f37-a2dc-1f2a494f9846', 'osd.11 up in weight 1 up_from 89 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6824/1668426191,v1:172.21.15.204:6825/1668426191] [v2:172.21.15.204:6826/1668426191,v1:172.21.15.204:6827/1668426191] exists,up 49911d7e-fe03-41a6-a25a-11a11c08f8b3'] 2024-09-16T05:10:20.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-16T05:10:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:21.126 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mon.a", 2024-09-16T05:10:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster": "ceph", 2024-09-16T05:10:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T05:10:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "admin_socket_mode": "", 2024-09-16T05:10:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T05:10:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_client_required": "cephx, none", 2024-09-16T05:10:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_cluster_required": "cephx", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_debug": "true", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_service_required": "cephx", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_supported": "", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_aio": "true", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_aio_reap_max": "16", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_async_discard": "false", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_block_size": "4096", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_debug_aio": "false", 2024-09-16T05:10:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_enable_discard": "false", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_flock_retry": "3", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_inject_crash": "0", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_ioring": "false", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_ioring_hipri": "false", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T05:10:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bdev_type": "", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_allocator": "hybrid", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_buffered_io": "true", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T05:10:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_replay_recovery": "false", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_sync_write": "false", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T05:10:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_allocator": "hybrid", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_create": "true", 2024-09-16T05:10:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_db_create": "false", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_db_path": "", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_db_size": "0", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_path": "", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_size": "107374182400", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_wal_create": "false", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_wal_path": "", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bluefs": "true", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T05:10:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_autotune": "true", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_size": "0", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T05:10:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T05:10:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T05:10:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T05:10:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_type": "2q", 2024-09-16T05:10:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_clone_cow": "true", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_mode": "none", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T05:10:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_freelist": "false", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T05:10:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T05:10:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fail_eio": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T05:10:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T05:10:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_blob_size": "0", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T05:10:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_spdk_mem": "512", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T05:10:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_tracing": "false", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T05:10:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T05:10:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_cluster_require_version": "2", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_require_signatures": "false", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_require_version": "2", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_service_require_signatures": "false", 2024-09-16T05:10:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_service_require_version": "2", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "cephx_sign_messages": "true", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "chdir": "", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_acl_type": "", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_asio_thread_count": "2", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_cache_mid": "0.750000", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_cache_size": "16384", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_caps_release_delay": "5", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_check_pool_perm": "true", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_debug_force_sync_read": "false", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_debug_getattr_caps": "false", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_debug_inject_features": "", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T05:10:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_die_on_failed_remount": "false", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_dirsize_rbytes": "true", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_force_lazyio": "false", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_fs": "", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_inject_release_failure": "false", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_max_inline_size": "4096", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mds_namespace": "", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_metadata": "", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mount_gid": "-1", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mount_timeout": "300", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mount_uid": "-1", 2024-09-16T05:10:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_mountpoint": "/", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_notify_timeout": "10", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc": "true", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_max_objects": "1000", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_size": "209715200", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_permissions": "true", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_quota": "true", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_quota_df": "true", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_readahead_max_bytes": "0", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_readahead_max_periods": "4", 2024-09-16T05:10:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "client_readahead_min": "131072", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "client_reconnect_stale": "false", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "client_shutdown_timeout": "30", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "client_snapdir": ".snap", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "client_tick_interval": "1", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "client_trace": "", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "client_use_faked_inos": "false", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "client_use_random_mds": "false", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_graylog": "false", 2024-09-16T05:10:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_graylog_port": "12201", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_monitors": "default=true", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_syslog": "false", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "clog_to_syslog_level": "info", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "-", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_network": "", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_network_interface": "", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "compressor_zlib_isal": "false", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "compressor_zlib_level": "5", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "compressor_zstd_level": "1", 2024-09-16T05:10:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_location": "", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_location_hook": "", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_location_hook_timeout": "10", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "daemonize": "false", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "daos_pool": "tank", 2024-09-16T05:10:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_alienstore": "0/5", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_asok": "1/5", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_asok_assert_abort": "false", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_auth": "1/5", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_bdev": "1/3", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_bluefs": "1/5", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_bluestore": "1/5", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_buffer": "0/1", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T05:10:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_cephsqlite": "0/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_client": "0/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_compressor": "1/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_context": "0/1", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_crush": "1/1", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_crypto": "1/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_cyanstore": "0/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_dpdk": "1/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_eventtrace": "1/5", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_filer": "0/1", 2024-09-16T05:10:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_filestore": "1/3", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_finisher": "1/1", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_fuse": "1/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_javaclient": "1/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_journal": "1/3", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_journaler": "0/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_kstore": "1/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_lockdep": "0/1", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mclock": "1/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds": "1/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_balancer": "1/5", 2024-09-16T05:10:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_locker": "1/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_log": "1/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_migrator": "1/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_memstore": "1/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mgr": "2/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mgrc": "1/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_mon": "20/20", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_monc": "0/10", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_ms": "1/1", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_none": "0/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_objclass": "0/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_objectcacher": "0/5", 2024-09-16T05:10:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_objecter": "0/1", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_optracker": "0/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_osd": "1/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_paxos": "20/20", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_perfcounter": "1/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_prioritycache": "1/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rados": "0/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rbd": "0/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rbd_replay": "0/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_refs": "0/0", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_reserver": "1/1", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw": "1/5", 2024-09-16T05:10:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw_access": "1/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw_flight": "1/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rgw_sync": "1/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_rocksdb": "4/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore": "0/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_backref": "0/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_cache": "0/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T05:10:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_device": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_epm": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_journal": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_lba": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_odata": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_omap": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_onode": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_t": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_seastore_tm": "0/5", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_striper": "0/1", 2024-09-16T05:10:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_test": "0/5", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_throttle": "1/1", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_timer": "0/1", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_tp": "0/5", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "debug_trace": "1/5", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "device_failure_prediction_mode": "none", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "err_to_graylog": "false", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "err_to_journald": "false", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "err_to_stderr": "true", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "err_to_syslog": "false", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event_tracing": "false", 2024-09-16T05:10:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_http_port": "9926", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_prio_limit": "5", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_sort_metrics": "true", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "exporter_stats_period": "5", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "fake_statfs_for_testing": "0", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "fatal_signal_handlers": "true", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "filer_max_purge_ops": "10", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "filer_max_truncate_ops": "128", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_blackhole": "false", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T05:10:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_btrfs_snap": "true", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_caller_concurrency": "10", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_debug_omap_check": "false", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_debug_verify_split": "false", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_dump_file": "", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fadvise": "true", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fail_eio": "true", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T05:10:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fd_cache_size": "128", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fiemap": "false", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_inject_stall": "0", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_journal_parallel": "false", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_journal_trailing": "false", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_journal_writeahead": "false", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_kill_at": "0", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T05:10:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T05:10:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_merge_threshold": "-10", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_odsync_write": "false", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_omap_backend_path": "", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_op_threads": "2", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_punch_hole": "false", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T05:10:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_queue_max_ops": "50", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_seek_data_hole": "false", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_sloppy_crc": "false", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_splice": "false", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_split_multiple": "2", 2024-09-16T05:10:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_split_rand_factor": "20", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_update_to": "1000", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T05:10:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_xfs_extsize": "false", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "filestore_zfs_snap": "false", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fio_dir": "/tmp/fio", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_allow_other": "true", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_big_writes": "true", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_debug": "false", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_default_permissions": "false", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_disable_pagecache": "false", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_max_write": "0", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_multithreaded": "true", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_require_active_mds": "true", 2024-09-16T05:10:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_set_user_groups": "true", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_splice_move": "true", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_splice_read": "true", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_splice_write": "true", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "gss_target_name": "ceph", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_file": "", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_inject_failure": "0", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_interval": "5", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "host": "smithi032", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T05:10:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "inject_early_sigterm": "false", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "jaeger_agent_port": "6799", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "jaeger_tracing_enable": "false", 2024-09-16T05:10:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_aio": "true", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_align_min_size": "65536", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_block_align": "true", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_block_size": "4096", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_dio": "true", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_discard": "false", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_force_aio": "false", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_ignore_corruption": "false", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_max_write_entries": "100", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_replay_from": "0", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T05:10:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_write_header_frequency": "0", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "journal_zero_on_create": "false", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "journaler_prefetch_periods": "10", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "journaler_prezero_periods": "5", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "journaler_write_head_interval": "15", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "keyfile": "", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_backend": "rocksdb", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T05:10:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_max_bytes": "67108864", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_max_ops": "512", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_onode_map_size": "1024", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "kstore_sync_transaction": "false", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "librados_thread_count": "2", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "lockdep": "false", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "lockdep_force_backtrace": "false", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "log_coarse_timestamps": "true", 2024-09-16T05:10:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_flush_on_exit": "false", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_graylog_port": "12201", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_max_new": "1000", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_max_recent": "10000", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_stderr_prefix": "", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": "true", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_graylog": "false", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_journald": "false", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_stderr": "false", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_syslog": "false", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T05:10:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_action_on_write_error": "1", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_alternate_name_max": "8192", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_asio_thread_count": "2", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_export_pin": "true", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_interval": "10", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_max": "-1", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_max_until": "-1", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_merge_size": "50", 2024-09-16T05:10:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_mode": "0", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_split_bits": "3", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_split_size": "10000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T05:10:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_mid": "0.700000", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T05:10:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_trim_interval": "1", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_debug_auth_pins": "false", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_debug_frag": "false", 2024-09-16T05:10:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_debug_scatterstat": "false", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_debug_subtrees": "false", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_default_dir_hash": "2", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_defer_session_stale": "true", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dir_max_entries": "0", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dir_prefetch": "true", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T05:10:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_early_reply": "true", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_enable_op_tracker": "true", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_enforce_unique_name": "true", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T05:10:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_health_dummy": "false", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T05:10:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_join_fs": "", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_journal_format": "1", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_create_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_export_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_import_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_journal_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_link_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_openc_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_rename_at": "0", 2024-09-16T05:10:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_max_events": "-1", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_max_segments": "128", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_pause": "false", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_segment_size": "0", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T05:10:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_trim_threshold": "128", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_completed_requests": "100000", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_export_size": "20971520", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_file_recover": "32", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_purge_files": "64", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_purge_ops": "8192", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T05:10:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_metrics_update_interval": "2", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_min_caps_per_client": "100", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_numa_node": "-1", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_history_duration": "600", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_history_size": "20", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_op_log_threshold": "5", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_ping_grace": "15", 2024-09-16T05:10:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_ping_interval": "5", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_max_caps": "30000", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_replay_interval": "1.000000", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_root_ino_gid": "0", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_root_ino_uid": "0", 2024-09-16T05:10:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_shutdown_check": "0", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_skip_ino": "0", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T05:10:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_snap_min_uid": "0", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_snap_rstat": "false", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_symlink_recovery": "true", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_thrash_exports": "0", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_thrash_fragments": "0", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_tick_interval": "5.000000", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_valgrind_exit": "false", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_verify_backtrace": "1", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_verify_scatter": "false", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_wipe_sessions": "false", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool_debug": "false", 2024-09-16T05:10:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "memstore_page_set": "false", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "memstore_page_size": "65536", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_client_bytes": "134217728", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_client_messages": "512", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_disabled_modules": "", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T05:10:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_mds_messages": "128", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_mon_messages": "128", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_op_history_duration": "600", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_op_history_size": "20", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_op_log_threshold": "5", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_osd_messages": "8192", 2024-09-16T05:10:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_pool": "true", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_standby_modules": "true", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_stats_period": "5", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_stats_threshold": "5", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_tick_period": "2", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_allow_pool_delete": "true", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_allow_pool_size_one": "false", 2024-09-16T05:10:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_bytes": "104857600", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_directed_command_retry": "5", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T05:10:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_client_target_rank": "-1", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_file_level": "debug", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T05:10:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_journald": "true", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_compact_on_start": "false", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_compact_on_trim": "true", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_cpu_threads": "4", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T05:10:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_data_avail_crit": "5", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_data_avail_warn": "5", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_dump_json": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_extra_checks": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T05:10:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_down_mkfs_grace": "60", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_down_uptime_grace": "60", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_election_default_strategy": "1", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_election_timeout": "5.000000", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_enable_op_tracker": "true", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_fake_pool_delete": "false", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_force_quorum_join": "false", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T05:10:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_detail_to_clog": "true", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_log_update_period": "5", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_max_detail": "50", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_to_clog": "true", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_host": "[v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0]", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_host_override": "", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_initial_members": "", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T05:10:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_lease": "5.000000", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_log_full_interval": "50", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_log_max": "10000", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_log_max_summary": "50", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_log_epochs": "500", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_osd": "10000", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T05:10:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_memory_autotune": "true", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_memory_target": "2147483648", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_digest_period": "5", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T05:10:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_complaint_time": "30", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_history_duration": "600", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_history_size": "20", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_op_log_threshold": "5", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T05:10:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T05:10:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_cache_size": "500", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T05:10:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T05:10:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T05:10:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T05:10:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_report_timeout": "900", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T05:10:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_reweight_max_osds": "4", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T05:10:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_interval": "300", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_max_keys": "100", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_scrub_timeout": "300", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_session_timeout": "300", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_smart_report_timeout": "5", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stretch_pool_size": "4", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_debug": "false", 2024-09-16T05:10:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_tick_interval": "5", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T05:10:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_older_version": "true", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T05:10:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "monmap": "", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T05:10:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "motr_tracing_enabled": "false", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_op_threads": "3", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_cm": "false", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_device_name": "", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T05:10:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T05:10:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T05:10:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T05:10:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T05:10:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T05:10:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_sl": "3", 2024-09-16T05:10:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T05:10:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_rdma_type": "ib", 2024-09-16T05:10:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_async_reap_threshold": "5", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_before_connect": "false", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_ipv4": "true", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_ipv6": "false", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_msgr1": "true", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_msgr2": "true", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_port_max": "7568", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_port_min": "6800", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_retry_count": "3", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_bind_retry_delay": "5", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_client": "false", 2024-09-16T05:10:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_mds": "false", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_mgr": "false", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_mon": "false", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_blackhole_osd": "false", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_client_mode": "crc secure", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_cluster_type": "", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_compress_secure": "false", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_crc_data": "true", 2024-09-16T05:10:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_crc_header": "true", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_bug": "true", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_old_message": "true", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T05:10:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_hugepages": "", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_lro": "true", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_pmd": "", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_port_id": "0", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T05:10:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_dump_on_send": "false", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_delay_type": "osd", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_network_congestion": "0", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_inject_socket_failures": "2500", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_max_accept_failures": "4", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_max_backoff": "15.000000", 2024-09-16T05:10:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_osd_compress_mode": "none", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_pq_min_cost": "65536", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_public_type": "", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_service_mode": "crc secure", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_tcp_nodelay": "true", 2024-09-16T05:10:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "ms_type": "async+posix", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "no_config_file": "false", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_inflight_ops": "1024", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter_timeout": "10.000000", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objectstore_blackhole": "false", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T05:10:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "openssl_engine_opts": "", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_max_ops": "4", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_agent_slop": "0.020000", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_asio_thread_count": "2", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T05:10:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backfill_scan_max": "512", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backfill_scan_min": "64", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backoff_on_peering": "false", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_beacon_report_interval": "300", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bench_duration": "30", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T05:10:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_blkin_trace_all": "false", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_class_update_on_start": "true", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_client_message_cap": "256", 2024-09-16T05:10:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_client_op_priority": "63", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_client_watch_timeout": "30", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_command_max_records": "256", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_compact_on_start": "false", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_crush_update_on_start": "true", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T05:10:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_op_order": "false", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T05:10:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_shutdown": "false", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_trim_objects": "false", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T05:10:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_default_notify_timeout": "30", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_enable_op_tracker": "true", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T05:10:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_info": "true", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_shutdown": "true", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_function_tracing": "false", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_grace": "20", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_interval": "6", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T05:10:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_stale": "600", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_journal_size": "5120", 2024-09-16T05:10:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_kill_backfill_at": "0", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_cache_size": "50", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_dedup": "true", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_message_max": "40", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_attr_name_len": "100", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_attr_size": "0", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_backfills": "1", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_markdown_count": "5", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_markdown_period": "600", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_object_name_len": "2048", 2024-09-16T05:10:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_object_size": "134217728", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_pgls": "1024", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_push_cost": "8388608", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_push_objects": "10", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_scrubs": "3", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T05:10:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_max_write_size": "90", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_profile": "balanced", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T05:10:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_base": "805306368", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T05:10:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_target": "4294967296", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_target_autotune": "false", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_min_recovery_priority": "0", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_report_interval": "5", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_num_cache_shards": "32", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T05:10:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_numa_node": "-1", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_objecter_finishers": "1", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_objectstore": "bluestore", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_objectstore_fuse": "false", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_objectstore_tracing": "false", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_history_duration": "600", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_history_size": "20", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_log_threshold": "5", 2024-09-16T05:10:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_shards": "0", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_op_thread_timeout": "15", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_open_classes_on_start": "true", 2024-09-16T05:10:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_os_flags": "0", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_peering_op_priority": "255", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_delete_priority": "5", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T05:10:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_crimson": "false", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T05:10:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_flags": "0", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_min_size": "2", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T05:10:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_size": "2", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_default_type": "replicated", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_cost": "20971520", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_active": "0", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T05:10:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_op_priority": "3", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_priority": "5", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_repair_during_recovery": "false", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T05:10:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_cost": "52428800", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_end_hour": "0", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T05:10:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_priority": "5", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_skip_data_digest": "false", 2024-09-16T05:10:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_smart_report_timeout": "5", 2024-09-16T05:10:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T05:10:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_priority": "5", 2024-09-16T05:10:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T05:10:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T05:10:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T05:10:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_target_transaction_size": "30", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T05:10:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_tracing": "false", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_use_stale_snap": "false", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_kill_at": "0", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_max_join_drift": "10", 2024-09-16T05:10:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_min": "500", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_min_wait": "0.050000", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_service_trim_max": "500", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_service_trim_min": "250", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_stash_full_interval": "25", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_trim_max": "500", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "paxos_trim_min": "250", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "perf": "true", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "pid_file": "", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T05:10:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "-", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrv": "", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "public_bind_addr": "-", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "public_network": "", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "public_network_interface": "", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "qat_compressor_enabled": "false", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "rados_mon_op_timeout": "0", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "rados_osd_op_timeout": "0", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "rados_tracing": "false", 2024-09-16T05:10:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_atime_update_interval": "60", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache": "true", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T05:10:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_policy": "writearound", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_size": "33554432", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_compression_hint": "none", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_clone_format": "auto", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_data_pool": "", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_features": "61", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_format": "2", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_map_options": "", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_order": "22", 2024-09-16T05:10:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_pool": "rbd", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_stripe_count": "0", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_io_scheduler": "simple", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T05:10:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_order": "24", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_pool": "", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_journal_splay_width": "4", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T05:10:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T05:10:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T05:10:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T05:10:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T05:10:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T05:10:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T05:10:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_op_threads": "1", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_plugins": "", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T05:10:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T05:10:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T05:10:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_tracing": "false", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_validate_names": "true", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_validate_pool": "true", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "restapi_base_url": "", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "restapi_log_level": "", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_admin_entry": "admin", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_backend_store": "rados", 2024-09-16T05:10:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_barbican_url": "", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_beast_enable_async": "true", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cache_enabled": "true", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T05:10:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_config_store": "rados", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_content_length_compat": "false", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_cross_domain_policy": "", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T05:10:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T05:10:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T05:10:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T05:10:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_d4n_port": "6379", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_log_window": "30", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T05:10:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T05:10:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dns_name": "", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dns_s3website_name": "", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T05:10:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_ops_log": "false", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_static_website": "false", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enable_usage_log": "false", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_expose_bucket": "false", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_extended_http_attrs": "", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_filter": "none", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T05:10:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_objs": "32", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T05:10:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_init_timeout": "300", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_password": "", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_project": "", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T05:10:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_token": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_admin_user": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_api_version": "2", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_url": "", 2024-09-16T05:10:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_max_objs": "32", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_max_worker": "3", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T05:10:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ldap_uri": "", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_log_http_headers": "", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_attr_size": "0", 2024-09-16T05:10:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_listing_results": "1000", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_notify_retries": "10", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T05:10:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T05:10:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_num_control_oids": "8", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_numa_node": "-1", 2024-09-16T05:10:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_op_tracing": "false", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_opa_token": "", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_opa_url": "", 2024-09-16T05:10:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ops_log_rados": "false", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T05:10:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_print_continue": "true", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_rados_tracing": "false", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_realm": "", 2024-09-16T05:10:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_realm_id": "", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_region": "", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_request_uri": "", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_resolve_cname": "false", 2024-09-16T05:10:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_run_sync_thread": "true", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_script_uri": "", 2024-09-16T05:10:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_service_provider_name": "", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_client_id": "", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_client_secret": "", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_entry": "sts", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_key": "sts", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_auth_url": "", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_custom_header": "", 2024-09-16T05:10:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_need_stats": "true", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_tenant_name": "", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_url": "", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_lease_period": "120", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T05:10:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_thread_pool_size": "512", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_comment": "", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_createby": "", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_encoding": "", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_flag": "false", 2024-09-16T05:10:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_origin": "", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_torrent_tracker": "", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_usage_max_shards": "32", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_use_opa_authz": "false", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_counters_cache": "false", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T05:10:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_user_unique_email": "true", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_verify_ssl": "true", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zone": "", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zone_id": "", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zonegroup": "", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zonegroup_id": "", 2024-09-16T05:10:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_block_size": "4096", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T05:10:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_partition_filters": "false", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_perf": "false", 2024-09-16T05:10:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "run_dir": "/var/run/ceph", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_block_create": "true", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_device_size": "53687091200", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_full_integrity_check": "false", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T05:10:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_main_device_type": "SSD", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "seastore_segment_size": "67108864", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "set_keepcaps": "false", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "setgroup": "ceph", 2024-09-16T05:10:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "setuser": "ceph", 2024-09-16T05:10:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "setuser_match_path": "", 2024-09-16T05:10:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T05:10:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "thp": "false", 2024-09-16T05:10:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "threadpool_default_timeout": "60", 2024-09-16T05:10:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T05:10:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "throttler_perf_counter": "true" 2024-09-16T05:10:21.291 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:21.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T05:10:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:21 smithi032 ceph-mon[27388]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:21 smithi204 ceph-mon[30933]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:22.235 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:21 smithi196 ceph-mon[30802]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:22.705 INFO:teuthology.orchestra.run.smithi032.stdout:mon.a: {} 2024-09-16T05:10:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T05:10:22.913 INFO:teuthology.orchestra.run.smithi032.stdout:mon.c: {} 2024-09-16T05:10:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T05:10:22.928 INFO:teuthology.orchestra.run.smithi032.stdout:mon.b: {} 2024-09-16T05:10:22.928 INFO:teuthology.orchestra.run.smithi032.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T05:10:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3922659660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:10:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3922659660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:10:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3922659660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:10:23.449 INFO:teuthology.run_tasks:Running task rados... 2024-09-16T05:10:23.462 INFO:tasks.rados:Beginning rados... 2024-09-16T05:10:23.463 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-16T05:10:23.479 INFO:tasks.rados:joining rados 2024-09-16T05:10:23.479 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T05:10:23.479 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:10:23.479 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:10:23.480 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-16T05:10:23.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:10:23.481 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T05:10:23.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:23.481 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T05:10:23.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:10:23.482 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T05:10:23.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:10:23.483 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T05:10:23.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:10:23.483 INFO:tasks.rados:clients are ['client.0', 'client.1', 'client.2'] 2024-09-16T05:10:23.484 INFO:tasks.rados:starting run 0 out of 1 2024-09-16T05:10:23.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:23.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:23.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:23.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:23.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:23.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:23.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:23.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:23.789 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:23.804 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:23.804 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:23.806 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:23.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:23.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:23.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:23.904 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:24.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:24.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:24.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:24.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:24.156 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:24.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:23 smithi032 ceph-mon[27388]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:23 smithi204 ceph-mon[30933]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:23 smithi196 ceph-mon[30802]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:24.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:24.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:24.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:24.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:24.408 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:24.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:24.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:24.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:24.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:24.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:24.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:24.665 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:24.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:24.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:24.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:24.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:24.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:24.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:25.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:25.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:25.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:25.049 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:10:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:10:25.050 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4293636003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:25.116 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:10:25.117 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:10:25.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:25.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4293636003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:25.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:25.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4293636003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:25.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:25.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:25.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:10:25.370 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:10:25.391 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:10:25.392 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:10:25.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:25.412 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:10:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:10:25.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:25.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:25.435 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:10:25.435 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:10:25.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:10:25.457 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:10:25.479 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:10:25.479 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:10:25.501 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:10:25.501 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:10:25.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:25.524 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:10:25.524 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:10:25.546 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:10:25.546 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:10:25.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:25.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:25.568 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:10:25.569 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:10:25.592 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:10:25.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:10:25.614 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:10:25.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:25.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:25.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:25.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:25.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:25.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:25.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:25.930 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:25.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:26.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:26.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:26.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:25 smithi032 ceph-mon[27388]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:25 smithi204 ceph-mon[30933]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:26.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:25 smithi196 ceph-mon[30802]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:26.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:26.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:26.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:26.430 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:26.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:26.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:26.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:26.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:26.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:26.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:26.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:26.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:26.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:26.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:10:26.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:26.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:26.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:27.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:26 smithi032 ceph-mon[27388]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:27.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3846257445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:10:27.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:27.067 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:27.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:27.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:26 smithi204 ceph-mon[30933]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:27.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3846257445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:10:27.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:27.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:26 smithi196 ceph-mon[30802]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:10:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3846257445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:10:27.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:27.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:27.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:27.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:10:27.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:27.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:27.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:27.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:27.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:27.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:27.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:27.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:27.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:27.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:27.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:27.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:27.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:27.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:28.012 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-16T05:10:24.031967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":346608,"kb_used_data":13952,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124595216,"statfs":{"total":1151940427776,"available":1151585501184,"internally_reserved":0,"allocated":14286848,"data_stored":7361348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.399044"},"pg_stats":[{"pgid":"1.0","version":"35'139","reported_seq":381,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T05:10:13.634296+0000","last_change":"2024-09-16T05:09:29.571554+0000","last_active":"2024-09-16T05:10:13.634296+0000","last_peered":"2024-09-16T05:10:13.634296+0000","last_clean":"2024-09-16T05:10:13.634296+0000","last_became_active":"2024-09-16T05:09:29.279282+0000","last_became_peered":"2024-09-16T05:09:29.279282+0000","last_unstale":"2024-09-16T05:10:13.634296+0000","last_undegraded":"2024-09-16T05:10:13.634296+0000","last_fullsized":"2024-09-16T05:10:13.634296+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:09:20.060218+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:09:20.060218+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:10:48.724228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089359,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1634304,"data_stored":1054219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285587,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514522,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841565,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103202,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1634304,"data_stored":1054219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1101824,"data_stored":525291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:10:28.012 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:10:28.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-16T05:10:28.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:27 smithi032 ceph-mon[27388]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:28.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:27 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:28.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3846257445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:10:28.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:27 smithi032 ceph-mon[27388]: osdmap e102: 12 total, 12 up, 12 in 2024-09-16T05:10:28.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:28.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:28.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:28.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:27 smithi204 ceph-mon[30933]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:27 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3846257445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:10:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:27 smithi204 ceph-mon[30933]: osdmap e102: 12 total, 12 up, 12 in 2024-09-16T05:10:28.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:28.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:27 smithi196 ceph-mon[30802]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:28.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:27 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:28.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3846257445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:10:28.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:27 smithi196 ceph-mon[30802]: osdmap e102: 12 total, 12 up, 12 in 2024-09-16T05:10:28.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:28.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:28.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:28.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:28.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:28.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:28.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:28.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:28.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:28.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:28.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:10:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:28.717 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:28.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:28.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:28.840 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:28.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:28.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:28.967 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:28.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:29.046 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:29.057 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:10:29.057 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:10:29.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:29.093 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:29.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:29.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:29.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:29.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:29.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:29.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:29.346 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:29.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:29.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:29.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:29.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:29.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:29.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:29.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:29.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:29.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:29.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:29.742 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:29.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:10:29.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:29.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:29.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:29.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:29.977 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:29 smithi032 ceph-mon[27388]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:29.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:29.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:30.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:30.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:30.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:29 smithi204 ceph-mon[30933]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:30.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:29 smithi196 ceph-mon[30802]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:30.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:30.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:30.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:30.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:30.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:30.436 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:10:30.436 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:10:30.458 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:10:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:10:30.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:30.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:10:30.478 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:10:30.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:30.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:30.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:10:30.499 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:10:30.520 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:10:30.521 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:10:30.542 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:10:30.543 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:10:30.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:10:30.564 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:10:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:10:30.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:30.623 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:30.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:30.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:30.685 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:10:30.686 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:10:30.722 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:10:30.722 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:10:30.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:30.755 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:10:30.756 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:10:30.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:30.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:30.778 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:10:30.779 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:10:30.800 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:10:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:10:30.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:30.886 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:30.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:30.917 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 1 pg_num to 11 2024-09-16T05:10:30.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:31.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:31.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:31.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:31.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:31.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2175929241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1002236050' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-16T05:10:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2175929241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1002236050' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-16T05:10:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2175929241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1002236050' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-16T05:10:31.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:31.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:31.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:31.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:31.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:31.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:31.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:31.523 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:31.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:31.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:31.649 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:31.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:31.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:31.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:31.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:16 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'6, mlcod=16'4)", 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 313.18208436399999, 2024-09-16T05:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29879279399999997, 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661725+0000", 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0800000000000003e-07 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661731+0000", 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4880000000000004e-06 2024-09-16T05:10:31.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960116+0000", 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29838472799999999 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960124+0000", 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960207+0000", 2024-09-16T05:10:31.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3122999999999997e-05 2024-09-16T05:10:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960225+0000", 2024-09-16T05:10:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8099000000000001e-05 2024-09-16T05:10:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960487+0000", 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026233000000000002 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960517+0000", 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9149000000000002e-05 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.857 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:21 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'11, mlcod=17'8)", 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:18.964051+0000", 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 312.87975682400003, 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000644562, 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964051+0000", 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964051+0000", 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964052+0000", 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7199999999999998e-07 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964101+0000", 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8618000000000001e-05 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964135+0000", 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4479e-05 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964142+0000", 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7530000000000004e-06 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964176+0000", 2024-09-16T05:10:31.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3735999999999997e-05 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964191+0000", 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5692000000000002e-05 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964675+0000", 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048355300000000001 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.964696+0000", 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1058999999999999e-05 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:57 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'35, mlcod=17'33)", 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:18.984513+0000", 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 312.85929543100002, 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074220500000000001, 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.862 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984513+0000", 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984513+0000", 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984514+0000", 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8899999999999998e-07 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984564+0000", 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0793e-05 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984592+0000", 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7954999999999999e-05 2024-09-16T05:10:31.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984597+0000", 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5220000000000001e-06 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984645+0000", 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7719999999999997e-05 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984672+0000", 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7776999999999998e-05 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.985209+0000", 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053635399999999995 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.985255+0000", 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6195000000000001e-05 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.866 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:58 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'36, mlcod=17'33)", 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:18.984892+0000", 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 312.858916126, 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070573200000000004, 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984892+0000", 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984892+0000", 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984893+0000", 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5700000000000002e-07 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984952+0000", 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9379999999999997e-05 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984988+0000", 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5765000000000001e-05 2024-09-16T05:10:31.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.984995+0000", 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0430000000000002e-06 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.985044+0000", 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9437000000000001e-05 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.985068+0000", 2024-09-16T05:10:31.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3609999999999999e-05 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.985557+0000", 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048905300000000004 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.985598+0000", 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0787000000000001e-05 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:71 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'45, mlcod=17'43)", 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:18.994202+0000", 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 312.849606145, 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062411300000000003, 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994202+0000", 2024-09-16T05:10:31.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994202+0000", 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994203+0000", 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9699999999999996e-07 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994222+0000", 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9286000000000001e-05 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994250+0000", 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8442999999999999e-05 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994255+0000", 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.656e-06 2024-09-16T05:10:31.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994301+0000", 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6180000000000002e-05 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994322+0000", 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.105e-05 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994781+0000", 2024-09-16T05:10:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00045871099999999999 2024-09-16T05:10:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.994826+0000", 2024-09-16T05:10:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5189999999999999e-05 2024-09-16T05:10:31.876 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:89 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'59, mlcod=17'57)", 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:19.004416+0000", 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 312.83939238900001, 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00065399800000000004, 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.877 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.004416+0000", 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.004416+0000", 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.004417+0000", 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7899999999999997e-07 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.004437+0000", 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0809e-05 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.004470+0000", 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2511000000000001e-05 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.004477+0000", 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7340000000000002e-06 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.004527+0000", 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0364000000000001e-05 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.004546+0000", 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9009999999999999e-05 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.005041+0000", 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00049461899999999996 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.005070+0000", 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9172000000000001e-05 2024-09-16T05:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:104 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'66, mlcod=17'64)", 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:19.942031+0000", 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 311.901776832, 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079861899999999998, 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942031+0000", 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942031+0000", 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942032+0000", 2024-09-16T05:10:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.001e-06 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942045+0000", 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2463000000000001e-05 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942076+0000", 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1256e-05 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942081+0000", 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3109999999999996e-06 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942147+0000", 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5916999999999999e-05 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942170+0000", 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2912e-05 2024-09-16T05:10:31.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942793+0000", 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062313099999999997 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.942830+0000", 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6628000000000002e-05 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:108 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'70, mlcod=17'67)", 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:19.944207+0000", 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 311.899601201, 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00064565200000000003, 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944207+0000", 2024-09-16T05:10:31.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944207+0000", 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944208+0000", 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0999999999999998e-07 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944220+0000", 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1861e-05 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944248+0000", 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7991999999999999e-05 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944254+0000", 2024-09-16T05:10:31.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4740000000000002e-06 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944298+0000", 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3761999999999999e-05 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944318+0000", 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0199000000000001e-05 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944725+0000", 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040723000000000002 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944853+0000", 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012742400000000001 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.891 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:109 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'71, mlcod=17'68)", 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:19.944667+0000", 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 311.89914080599999, 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068000600000000003, 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944667+0000", 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944667+0000", 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944668+0000", 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1100000000000005e-07 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944719+0000", 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0594999999999999e-05 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944753+0000", 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4141999999999999e-05 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944838+0000", 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5555000000000003e-05 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944894+0000", 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5087999999999998e-05 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.944919+0000", 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5519000000000001e-05 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.945302+0000", 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000382915 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.945347+0000", 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5380999999999998e-05 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:119 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'81, mlcod=17'78)", 2024-09-16T05:10:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:19.950357+0000", 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 311.89345143100002, 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00065829199999999999, 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950357+0000", 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950357+0000", 2024-09-16T05:10:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950358+0000", 2024-09-16T05:10:31.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1299999999999999e-07 2024-09-16T05:10:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950395+0000", 2024-09-16T05:10:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7744999999999999e-05 2024-09-16T05:10:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950422+0000", 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6548000000000001e-05 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950428+0000", 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8619999999999999e-06 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950499+0000", 2024-09-16T05:10:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1353999999999995e-05 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950517+0000", 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8202999999999998e-05 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.950988+0000", 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00047061999999999999 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:19.951015+0000", 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7146999999999999e-05 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:130 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'88, mlcod=18'85)", 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:22.881411+0000", 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.96239735099999, 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071295400000000002, 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.881411+0000", 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.881411+0000", 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.881412+0000", 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9699999999999995e-07 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.881418+0000", 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8930000000000002e-06 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.881474+0000", 2024-09-16T05:10:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6057999999999998e-05 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.881481+0000", 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7619999999999995e-06 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.881537+0000", 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5506e-05 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.881556+0000", 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9347999999999998e-05 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.882079+0000", 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052291000000000002 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.882124+0000", 2024-09-16T05:10:31.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4679999999999999e-05 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:140 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'98, mlcod=18'94)", 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:22.887235+0000", 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.95657359400002, 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069572999999999998, 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887235+0000", 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887235+0000", 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887235+0000", 2024-09-16T05:10:31.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1399999999999996e-07 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887256+0000", 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1039e-05 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887288+0000", 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1319000000000003e-05 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887295+0000", 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9469999999999997e-06 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887337+0000", 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2318999999999999e-05 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887349+0000", 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1946e-05 2024-09-16T05:10:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887881+0000", 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053198700000000004 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:22.887930+0000", 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9459000000000002e-05 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:154 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'108, mlcod=18'105)", 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:23.407805+0000", 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.43600307299999, 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080574399999999995, 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.407805+0000", 2024-09-16T05:10:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.407805+0000", 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.407806+0000", 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4900000000000004e-07 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.407813+0000", 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4409999999999998e-06 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.407850+0000", 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7506999999999999e-05 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.407857+0000", 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9380000000000003e-06 2024-09-16T05:10:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.407921+0000", 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3569000000000001e-05 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.407937+0000", 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6682000000000001e-05 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408576+0000", 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063904599999999999 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408611+0000", 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4612e-05 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:155 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'109, mlcod=18'105)", 2024-09-16T05:10:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:23.408174+0000", 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.43563432100001, 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083376200000000004, 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408174+0000", 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408174+0000", 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408175+0000", 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1459999999999999e-06 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408206+0000", 2024-09-16T05:10:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1372999999999999e-05 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408246+0000", 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0009000000000003e-05 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408254+0000", 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8169999999999999e-06 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408322+0000", 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7494000000000001e-05 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408359+0000", 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6962999999999998e-05 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.408938+0000", 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00057977199999999999 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.409008+0000", 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9188000000000002e-05 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:158 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'112, mlcod=18'109)", 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:23.410264+0000", 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.43354468199999, 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079641399999999995, 2024-09-16T05:10:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.410264+0000", 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.410264+0000", 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.410264+0000", 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8199999999999999e-07 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.410311+0000", 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6519e-05 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.410349+0000", 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7978000000000001e-05 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.410357+0000", 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8609999999999998e-06 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.410403+0000", 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5889000000000001e-05 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.410422+0000", 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9361999999999999e-05 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411022+0000", 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00059992499999999996 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411060+0000", 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8098e-05 2024-09-16T05:10:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:161 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'115, mlcod=18'110)", 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:23.411698+0000", 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.43211006000001, 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061963999999999997, 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411698+0000", 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411698+0000", 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411699+0000", 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9100000000000003e-07 2024-09-16T05:10:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411763+0000", 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4473000000000004e-05 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411814+0000", 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0105000000000002e-05 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411823+0000", 2024-09-16T05:10:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2210000000000003e-06 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411865+0000", 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.18e-05 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.411883+0000", 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8638999999999999e-05 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.412274+0000", 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039121400000000003 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.412318+0000", 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3396999999999998e-05 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:164 1.0 e18/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'118, mlcod=18'114)", 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:23.413388+0000", 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.430420309, 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00064543100000000002, 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.413388+0000", 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.413388+0000", 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.413389+0000", 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1399999999999996e-07 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.413422+0000", 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3714999999999998e-05 2024-09-16T05:10:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.413464+0000", 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.193e-05 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.413472+0000", 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3019999999999997e-06 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.413522+0000", 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0139000000000002e-05 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.413546+0000", 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3748e-05 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.414001+0000", 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000455358 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.414033+0000", 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2425000000000002e-05 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:166 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'120, mlcod=18'118)", 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:23.415153+0000", 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.428655295, 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000740386, 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415153+0000", 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415153+0000", 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415154+0000", 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3500000000000005e-07 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415182+0000", 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8561999999999999e-05 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415228+0000", 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6044999999999997e-05 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415236+0000", 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4610000000000002e-06 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415278+0000", 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2009999999999999e-05 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415298+0000", 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0301000000000001e-05 2024-09-16T05:10:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415864+0000", 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056558599999999998 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415893+0000", 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9485999999999999e-05 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:168 1.0 e18/15 1:30306672:devicehealth::main.db.0000000000000000:head v 18'122, mlcod=18'118)", 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:23.415998+0000", 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 308.42780972000003, 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00067177100000000004, 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415998+0000", 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415998+0000", 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.415999+0000", 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9599999999999999e-07 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.416025+0000", 2024-09-16T05:10:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5774000000000001e-05 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.416056+0000", 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1232999999999997e-05 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.416063+0000", 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1389999999999998e-06 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.416127+0000", 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3394999999999999e-05 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.416149+0000", 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2255999999999998e-05 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.416641+0000", 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000492425 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:23.416670+0000", 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8952999999999999e-05 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.1.0:137 1.0 e35/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 35'138, mlcod=18'136)", 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:06:40.782135+0000", 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 231.061673469, 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071199000000000002, 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782135+0000", 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782135+0000", 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:31.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782136+0000", 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.23e-07 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782145+0000", 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9210000000000004e-06 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782178+0000", 2024-09-16T05:10:31.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.218e-05 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782190+0000", 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1914e-05 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782244+0000", 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4249999999999997e-05 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782260+0000", 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6232e-05 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782793+0000", 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053341 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:06:40.782847+0000", 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3260000000000002e-05 2024-09-16T05:10:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:31.952 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:31.953 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:31.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:32.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:32.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:32.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:32.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:31 smithi032 ceph-mon[27388]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:32.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1002236050' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T05:10:32.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:31 smithi032 ceph-mon[27388]: osdmap e103: 12 total, 12 up, 12 in 2024-09-16T05:10:32.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:31 smithi204 ceph-mon[30933]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1002236050' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T05:10:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:31 smithi204 ceph-mon[30933]: osdmap e103: 12 total, 12 up, 12 in 2024-09-16T05:10:32.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:32.203 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:32.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:31 smithi196 ceph-mon[30802]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1002236050' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-16T05:10:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:31 smithi196 ceph-mon[30802]: osdmap e103: 12 total, 12 up, 12 in 2024-09-16T05:10:32.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:32.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:32.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:32.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:32.452 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:32.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:32.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:32.582 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:32.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:32.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:32.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:32.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:32.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:10:32.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:32.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:32.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:32.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:32.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:33.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:33.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:32 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-16T05:10:33.089 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:33.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:33.117 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-16T05:10:33.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-16T05:10:33.167 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:32 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-16T05:10:33.177 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:33.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:33.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:33.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:33.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:32 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-16T05:10:33.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:33.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:33.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:33.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:33.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:33.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:33.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:10:33.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:33.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:33.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:33.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:33.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:33.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:33.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:33.842 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:33.843 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:10:33.843 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:10:33.843 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:33.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:33.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:33.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:33.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:33.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:33.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:33 smithi032 ceph-mon[27388]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:33.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:33 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T05:10:33.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:33 smithi032 ceph-mon[27388]: osdmap e104: 12 total, 12 up, 12 in 2024-09-16T05:10:34.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:34.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:34.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:34.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:34.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:34.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:34.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:33 smithi196 ceph-mon[30802]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:33 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T05:10:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:33 smithi196 ceph-mon[30802]: osdmap e104: 12 total, 12 up, 12 in 2024-09-16T05:10:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:34.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:34.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:34.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:34.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:33 smithi204 ceph-mon[30933]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:34.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:33 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-16T05:10:34.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:33 smithi204 ceph-mon[30933]: osdmap e104: 12 total, 12 up, 12 in 2024-09-16T05:10:34.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:34.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:34.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:34.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:34.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:34.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:34.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:34.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:34.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:34.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:34.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:34.870 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:34.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:34.940 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:10:34.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:34.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:34.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:35.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:34 smithi032 ceph-mon[27388]: osdmap e105: 12 total, 12 up, 12 in 2024-09-16T05:10:35.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-16T05:10:35.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1909307810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:35.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:35.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:35.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:35.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:34 smithi196 ceph-mon[30802]: osdmap e105: 12 total, 12 up, 12 in 2024-09-16T05:10:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-16T05:10:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1909307810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:35.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:35.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:35.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:34 smithi204 ceph-mon[30933]: osdmap e105: 12 total, 12 up, 12 in 2024-09-16T05:10:35.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-16T05:10:35.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1909307810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:35.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:35.375 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:35.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:35.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:35.499 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:35.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:35.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:35.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:35.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:35.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:35.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:35.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:35.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:35.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:35.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'unique_pool_0' created 2024-09-16T05:10:35.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:36.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:36.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:36.111 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:36.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:36.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:36.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:35 smithi032 ceph-mon[27388]: pgmap v309: 11 pgs: 10 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:36.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:35 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:36.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:35 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T05:10:36.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1909307810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:10:36.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:35 smithi032 ceph-mon[27388]: osdmap e106: 12 total, 12 up, 12 in 2024-09-16T05:10:36.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3924567020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-16T05:10:36.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2531890717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:36.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:10:36.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:35 smithi196 ceph-mon[30802]: pgmap v309: 11 pgs: 10 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:35 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:35 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T05:10:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1909307810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:10:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:35 smithi196 ceph-mon[30802]: osdmap e106: 12 total, 12 up, 12 in 2024-09-16T05:10:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3924567020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-16T05:10:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2531890717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:36.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:36.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:36.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:36.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:36.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:35 smithi204 ceph-mon[30933]: pgmap v309: 11 pgs: 10 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:35 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:35 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-16T05:10:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1909307810' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:10:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:35 smithi204 ceph-mon[30933]: osdmap e106: 12 total, 12 up, 12 in 2024-09-16T05:10:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3924567020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-16T05:10:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2531890717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:36.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:36.517 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:36.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:36.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:36.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:36.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:36.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:36.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:36.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:36.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T05:10:36.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:36.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:36.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:36.940 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:10:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:10:36.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:10:36.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:10:36.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:10:36.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:10:36.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:10:36.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:10:36.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:10:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:10:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:10:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:10:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:10:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:10:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:10:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:10:37.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:37.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:37.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:37.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:37.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:37.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:36 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:36 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3924567020' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-16T05:10:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:36 smithi032 ceph-mon[27388]: osdmap e107: 12 total, 12 up, 12 in 2024-09-16T05:10:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:36 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:36 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3924567020' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-16T05:10:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:36 smithi196 ceph-mon[30802]: osdmap e107: 12 total, 12 up, 12 in 2024-09-16T05:10:37.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:37.277 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:37.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:37.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:37.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:37.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:37.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:36 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:37.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:36 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3924567020' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-16T05:10:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:36 smithi204 ceph-mon[30933]: osdmap e107: 12 total, 12 up, 12 in 2024-09-16T05:10:37.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:37.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:37.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:37.586 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:37.625 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:37.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:37.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:37.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:37.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:37.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:10:37.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:37.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:37.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:37.907 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:37.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:38.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:38.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:38.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:38.118 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:10:38.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:10:38.118 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-09-16T05:10:38.119 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:10:38.119 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:10:38.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-09-16T05:10:38.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:38.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:37 smithi032 ceph-mon[27388]: pgmap v312: 27 pgs: 26 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:38.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:37 smithi032 ceph-mon[27388]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T05:10:38.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:37 smithi032 ceph-mon[27388]: osdmap e108: 12 total, 12 up, 12 in 2024-09-16T05:10:38.158 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:10:38.158 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:10:38.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:38.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:37 smithi196 ceph-mon[30802]: pgmap v312: 27 pgs: 26 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:37 smithi196 ceph-mon[30802]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T05:10:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:37 smithi196 ceph-mon[30802]: osdmap e108: 12 total, 12 up, 12 in 2024-09-16T05:10:38.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:38.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:38.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:38.373 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:10:38.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:38.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:37 smithi204 ceph-mon[30933]: pgmap v312: 27 pgs: 26 unknown, 1 active+clean; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:10:38.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:37 smithi204 ceph-mon[30933]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T05:10:38.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:37 smithi204 ceph-mon[30933]: osdmap e108: 12 total, 12 up, 12 in 2024-09-16T05:10:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:38.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:38.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:38.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:38.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:38.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:38.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:38.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:38.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:38.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:38.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:38.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:38.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:10:38.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:38.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:38.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:39.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:39.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:39.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:39.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:38 smithi204 ceph-mon[30933]: osdmap e109: 12 total, 12 up, 12 in 2024-09-16T05:10:39.104 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:10:39.104 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T05:10:39.104 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:10:39.156 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:39.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:38 smithi032 ceph-mon[27388]: osdmap e109: 12 total, 12 up, 12 in 2024-09-16T05:10:39.189 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:39.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:39.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:39.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:38 smithi196 ceph-mon[30802]: osdmap e109: 12 total, 12 up, 12 in 2024-09-16T05:10:39.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:39.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:39.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:39.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:39.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:39.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:39.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:39.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:39.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:39.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:10:39.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:39.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:39.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:39.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:39.822 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:39.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:39.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:39.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:39.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T05:10:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:10:40.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:40.052 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:10:40.053 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:10:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:40.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:40.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:39 smithi032 ceph-mon[27388]: pgmap v315: 27 pgs: 4 creating+peering, 7 active+clean, 7 remapped+peering, 9 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T05:10:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:39 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:39 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:39 smithi032 ceph-mon[27388]: osdmap e110: 12 total, 12 up, 12 in 2024-09-16T05:10:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3602355876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:10:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2918169860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:10:40.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:40.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:40.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:39 smithi196 ceph-mon[30802]: pgmap v315: 27 pgs: 4 creating+peering, 7 active+clean, 7 remapped+peering, 9 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T05:10:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:39 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:39 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:39 smithi196 ceph-mon[30802]: osdmap e110: 12 total, 12 up, 12 in 2024-09-16T05:10:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3602355876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:10:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2918169860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:10:40.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:40.330 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:40.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:39 smithi204 ceph-mon[30933]: pgmap v315: 27 pgs: 4 creating+peering, 7 active+clean, 7 remapped+peering, 9 unknown; 577 KiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T05:10:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:39 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:39 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:39 smithi204 ceph-mon[30933]: osdmap e110: 12 total, 12 up, 12 in 2024-09-16T05:10:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3602355876' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T05:10:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2918169860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:10:40.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:40.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:40.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:40.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:40.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:40.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:40.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:40.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:40.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:40.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:40.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:40.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:40.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:40.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:40.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:40.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:41.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:41.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:41.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:41.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:41.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:40 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3602355876' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T05:10:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2918169860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:10:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:40 smithi196 ceph-mon[30802]: osdmap e111: 12 total, 12 up, 12 in 2024-09-16T05:10:41.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:41.558 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:40 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:41.559 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3602355876' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T05:10:41.559 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2918169860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:10:41.559 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:40 smithi032 ceph-mon[27388]: osdmap e111: 12 total, 12 up, 12 in 2024-09-16T05:10:41.560 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:40 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:41.560 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3602355876' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-16T05:10:41.560 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2918169860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:10:41.560 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:40 smithi204 ceph-mon[30933]: osdmap e111: 12 total, 12 up, 12 in 2024-09-16T05:10:41.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:41.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:41.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:41.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:41.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:41.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:10:41.803 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:41.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:41.843 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:41.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:41.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:41.967 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:41.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:42.066 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:41 smithi032 ceph-mon[27388]: pgmap v318: 27 pgs: 4 creating+peering, 10 active+clean, 7 remapped+peering, 6 unknown; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T05:10:42.066 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:41 smithi032 ceph-mon[27388]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T05:10:42.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:42.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:42.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:42.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:42.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:42.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:42.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:41 smithi196 ceph-mon[30802]: pgmap v318: 27 pgs: 4 creating+peering, 10 active+clean, 7 remapped+peering, 6 unknown; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T05:10:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:41 smithi196 ceph-mon[30802]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T05:10:42.293 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:10:42.293 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:10:42.309 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:42.315 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:10:42.315 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:10:42.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:42.343 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:10:42.343 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:10:42.351 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:42.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:42.365 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:10:42.365 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:10:42.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:10:42.387 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:10:42.409 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:10:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:10:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:41 smithi204 ceph-mon[30933]: pgmap v318: 27 pgs: 4 creating+peering, 10 active+clean, 7 remapped+peering, 6 unknown; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-16T05:10:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:41 smithi204 ceph-mon[30933]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T05:10:42.430 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:10:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:10:42.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:42.453 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:10:42.453 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:10:42.474 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:10:42.474 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:10:42.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:42.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:42.496 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:10:42.496 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:10:42.518 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:10:42.519 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:10:42.540 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:10:42.540 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:10:42.561 DEBUG:teuthology.orchestra.run.smithi204:> CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-09-16T05:10:42.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:42.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:42.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:adding op weight read -> 100 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:adding op weight write -> 50 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:adding op weight delete -> 50 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:adding op weight snap_create -> 50 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:adding op weight snap_remove -> 50 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:adding op weight rollback -> 50 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:adding op weight copy_from -> 50 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:adding op weight write_excl -> 50 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:ceph version 17.2.7-1633-g6d025bc2 (6d025bc26f3176c910707f8cb4c8d8e076640615) quincy (stable) 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout:Configuration: 2024-09-16T05:10:42.647 INFO:tasks.rados.rados.2.smithi204.stdout: Number of operations: 4000 2024-09-16T05:10:42.648 INFO:tasks.rados.rados.2.smithi204.stdout: Number of objects: 50 2024-09-16T05:10:42.648 INFO:tasks.rados.rados.2.smithi204.stdout: Max in flight operations: 16 2024-09-16T05:10:42.648 INFO:tasks.rados.rados.2.smithi204.stdout: Object size (in bytes): 4000000 2024-09-16T05:10:42.648 INFO:tasks.rados.rados.2.smithi204.stdout: Write stride min: 400000 2024-09-16T05:10:42.648 INFO:tasks.rados.rados.2.smithi204.stdout: Write stride max: 800000 2024-09-16T05:10:42.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:42.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:42.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:42.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:42.857 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:42.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:42.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:42.972 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:10:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:10:42.972 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:42.983 INFO:tasks.rados.rados.2.smithi204.stdout:1: write initial oid 1 2024-09-16T05:10:42.983 INFO:tasks.rados.rados.2.smithi204.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-16T05:10:42.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:42.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:42.988 INFO:tasks.rados.rados.2.smithi204.stdout:1: writing smithi20457400-1 from 416807 to 892056 tid 1 2024-09-16T05:10:42.994 INFO:tasks.rados.rados.2.smithi204.stdout:1: writing smithi20457400-1 from 1542129 to 2016807 tid 2 2024-09-16T05:10:42.995 INFO:tasks.rados.rados.2.smithi204.stdout:2: write initial oid 2 2024-09-16T05:10:42.995 INFO:tasks.rados.rados.2.smithi204.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-16T05:10:42.999 INFO:tasks.rados.rados.2.smithi204.stdout:2: writing smithi20457400-2 from 416807 to 892056 tid 1 2024-09-16T05:10:43.006 INFO:tasks.rados.rados.2.smithi204.stdout:2: writing smithi20457400-2 from 1542129 to 2016807 tid 2 2024-09-16T05:10:43.006 INFO:tasks.rados.rados.2.smithi204.stdout:3: write initial oid 3 2024-09-16T05:10:43.006 INFO:tasks.rados.rados.2.smithi204.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-16T05:10:43.011 INFO:tasks.rados.rados.2.smithi204.stdout:3: writing smithi20457400-3 from 433614 to 984112 tid 1 2024-09-16T05:10:43.017 INFO:tasks.rados.rados.2.smithi204.stdout:3: writing smithi20457400-3 from 1600611 to 2033614 tid 2 2024-09-16T05:10:43.017 INFO:tasks.rados.rados.2.smithi204.stdout:4: write initial oid 4 2024-09-16T05:10:43.017 INFO:tasks.rados.rados.2.smithi204.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-16T05:10:43.023 INFO:tasks.rados.rados.2.smithi204.stdout:4: writing smithi20457400-4 from 450421 to 1076168 tid 1 2024-09-16T05:10:43.029 INFO:tasks.rados.rados.2.smithi204.stdout:4: writing smithi20457400-4 from 1659093 to 2050421 tid 2 2024-09-16T05:10:43.029 INFO:tasks.rados.rados.2.smithi204.stdout:5: write initial oid 5 2024-09-16T05:10:43.029 INFO:tasks.rados.rados.2.smithi204.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-16T05:10:43.035 INFO:tasks.rados.rados.2.smithi204.stdout:5: writing smithi20457400-5 from 467228 to 1168224 tid 1 2024-09-16T05:10:43.040 INFO:tasks.rados.rados.2.smithi204.stdout:5: writing smithi20457400-5 from 1717575 to 2067228 tid 2 2024-09-16T05:10:43.040 INFO:tasks.rados.rados.2.smithi204.stdout:6: write initial oid 6 2024-09-16T05:10:43.040 INFO:tasks.rados.rados.2.smithi204.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-16T05:10:43.047 INFO:tasks.rados.rados.2.smithi204.stdout:6: writing smithi20457400-6 from 484035 to 1260280 tid 1 2024-09-16T05:10:43.052 INFO:tasks.rados.rados.2.smithi204.stdout:6: writing smithi20457400-6 from 2059704 to 2084035 tid 2 2024-09-16T05:10:43.052 INFO:tasks.rados.rados.2.smithi204.stdout:7: write initial oid 7 2024-09-16T05:10:43.052 INFO:tasks.rados.rados.2.smithi204.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-16T05:10:43.057 INFO:tasks.rados.rados.2.smithi204.stdout:7: writing smithi20457400-7 from 500842 to 952336 tid 1 2024-09-16T05:10:43.065 INFO:tasks.rados.rados.2.smithi204.stdout:7: writing smithi20457400-7 from 1718186 to 2100842 tid 2 2024-09-16T05:10:43.065 INFO:tasks.rados.rados.2.smithi204.stdout:8: write initial oid 8 2024-09-16T05:10:43.065 INFO:tasks.rados.rados.2.smithi204.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-16T05:10:43.069 INFO:tasks.rados.rados.2.smithi204.stdout:8: writing smithi20457400-8 from 517649 to 1044392 tid 1 2024-09-16T05:10:43.075 INFO:tasks.rados.rados.2.smithi204.stdout:8: writing smithi20457400-8 from 1776668 to 2117649 tid 2 2024-09-16T05:10:43.075 INFO:tasks.rados.rados.2.smithi204.stdout:9: write initial oid 9 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:1: finishing write tid 1 to smithi20457400-1 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:1: finishing write tid 2 to smithi20457400-1 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:1: finishing write tid 3 to smithi20457400-1 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:1: finishing write tid 4 to smithi20457400-1 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:2: finishing write tid 1 to smithi20457400-2 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:2: finishing write tid 2 to smithi20457400-2 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:2: finishing write tid 3 to smithi20457400-2 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:2: finishing write tid 4 to smithi20457400-2 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-16T05:10:43.076 INFO:tasks.rados.rados.2.smithi204.stdout:3: finishing write tid 1 to smithi20457400-3 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:3: finishing write tid 2 to smithi20457400-3 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:3: finishing write tid 3 to smithi20457400-3 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:3: finishing write tid 4 to smithi20457400-3 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:4: finishing write tid 1 to smithi20457400-4 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:4: finishing write tid 2 to smithi20457400-4 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:4: finishing write tid 3 to smithi20457400-4 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:4: finishing write tid 4 to smithi20457400-4 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:5: finishing write tid 1 to smithi20457400-5 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:5: finishing write tid 2 to smithi20457400-5 2024-09-16T05:10:43.077 INFO:tasks.rados.rados.2.smithi204.stdout:6: finishing write tid 1 to smithi20457400-6 2024-09-16T05:10:43.078 INFO:tasks.rados.rados.2.smithi204.stdout:6: finishing write tid 2 to smithi20457400-6 2024-09-16T05:10:43.078 INFO:tasks.rados.rados.2.smithi204.stdout:7: finishing write tid 1 to smithi20457400-7 2024-09-16T05:10:43.078 INFO:tasks.rados.rados.2.smithi204.stdout:7: finishing write tid 2 to smithi20457400-7 2024-09-16T05:10:43.078 INFO:tasks.rados.rados.2.smithi204.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-16T05:10:43.081 INFO:tasks.rados.rados.2.smithi204.stdout:9: writing smithi20457400-9 from 534456 to 1252801 tid 1 2024-09-16T05:10:43.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:43.087 INFO:tasks.rados.rados.2.smithi204.stdout:9: writing smithi20457400-9 from 1951503 to 2134456 tid 2 2024-09-16T05:10:43.087 INFO:tasks.rados.rados.2.smithi204.stdout:1: done (8 left) 2024-09-16T05:10:43.087 INFO:tasks.rados.rados.2.smithi204.stdout:2: done (7 left) 2024-09-16T05:10:43.087 INFO:tasks.rados.rados.2.smithi204.stdout:3: done (6 left) 2024-09-16T05:10:43.087 INFO:tasks.rados.rados.2.smithi204.stdout:4: done (5 left) 2024-09-16T05:10:43.087 INFO:tasks.rados.rados.2.smithi204.stdout:10: write initial oid 10 2024-09-16T05:10:43.087 INFO:tasks.rados.rados.2.smithi204.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-16T05:10:43.093 INFO:tasks.rados.rados.2.smithi204.stdout:10: writing smithi20457400-10 from 551263 to 1344857 tid 1 2024-09-16T05:10:43.098 INFO:tasks.rados.rados.2.smithi204.stdout:10: writing smithi20457400-10 from 1893632 to 2151263 tid 2 2024-09-16T05:10:43.098 INFO:tasks.rados.rados.2.smithi204.stdout:11: write initial oid 11 2024-09-16T05:10:43.098 INFO:tasks.rados.rados.2.smithi204.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-16T05:10:43.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:10:43.103 INFO:tasks.rados.rados.2.smithi204.stdout:11: writing smithi20457400-11 from 568070 to 1036913 tid 1 2024-09-16T05:10:43.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:43.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:43.110 INFO:tasks.rados.rados.2.smithi204.stdout:11: writing smithi20457400-11 from 1552114 to 2168070 tid 2 2024-09-16T05:10:43.110 INFO:tasks.rados.rados.2.smithi204.stdout:12: write initial oid 12 2024-09-16T05:10:43.110 INFO:tasks.rados.rados.2.smithi204.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-16T05:10:43.116 INFO:tasks.rados.rados.2.smithi204.stdout:12: writing smithi20457400-12 from 584877 to 1128969 tid 1 2024-09-16T05:10:43.122 INFO:tasks.rados.rados.2.smithi204.stdout:12: writing smithi20457400-12 from 1610596 to 2172599 tid 2 2024-09-16T05:10:43.122 INFO:tasks.rados.rados.2.smithi204.stdout:13: write initial oid 13 2024-09-16T05:10:43.122 INFO:tasks.rados.rados.2.smithi204.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-16T05:10:43.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:42 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:10:43.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:42 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:10:43.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:42 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:10:43.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:42 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:10:43.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2867496412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:43.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.204:0/662043117' entity='client.2' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-16T05:10:43.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:42 smithi032 ceph-mon[27388]: from='client.? ' entity='client.2' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-16T05:10:43.128 INFO:tasks.rados.rados.2.smithi204.stdout:13: writing smithi20457400-13 from 601684 to 1221025 tid 1 2024-09-16T05:10:43.134 INFO:tasks.rados.rados.2.smithi204.stdout:13: writing smithi20457400-13 from 1669078 to 2201684 tid 2 2024-09-16T05:10:43.134 INFO:tasks.rados.rados.2.smithi204.stdout:14: write initial oid 14 2024-09-16T05:10:43.134 INFO:tasks.rados.rados.2.smithi204.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-16T05:10:43.141 INFO:tasks.rados.rados.2.smithi204.stdout:14: writing smithi20457400-14 from 618491 to 1313081 tid 1 2024-09-16T05:10:43.147 INFO:tasks.rados.rados.2.smithi204.stdout:14: writing smithi20457400-14 from 2011207 to 2218491 tid 2 2024-09-16T05:10:43.147 INFO:tasks.rados.rados.2.smithi204.stdout:15: write initial oid 15 2024-09-16T05:10:43.147 INFO:tasks.rados.rados.2.smithi204.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-16T05:10:43.154 INFO:tasks.rados.rados.2.smithi204.stdout:15: writing smithi20457400-15 from 635298 to 1405137 tid 1 2024-09-16T05:10:43.161 INFO:tasks.rados.rados.2.smithi204.stdout:15: writing smithi20457400-15 from 2069689 to 2235298 tid 2 2024-09-16T05:10:43.161 INFO:tasks.rados.rados.2.smithi204.stdout:16: write initial oid 16 2024-09-16T05:10:43.161 INFO:tasks.rados.rados.2.smithi204.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-16T05:10:43.164 INFO:tasks.rados.rados.2.smithi204.stdout:16: writing smithi20457400-16 from 652105 to 1097193 tid 1 2024-09-16T05:10:43.170 INFO:tasks.rados.rados.2.smithi204.stdout:16: writing smithi20457400-16 from 1728171 to 2181159 tid 2 2024-09-16T05:10:43.170 INFO:tasks.rados.rados.2.smithi204.stdout:17: write initial oid 17 2024-09-16T05:10:43.170 INFO:tasks.rados.rados.2.smithi204.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-16T05:10:43.177 INFO:tasks.rados.rados.2.smithi204.stdout:17: writing smithi20457400-17 from 668912 to 1305602 tid 1 2024-09-16T05:10:43.182 INFO:tasks.rados.rados.2.smithi204.stdout:17: writing smithi20457400-17 from 1903006 to 2268912 tid 2 2024-09-16T05:10:43.182 INFO:tasks.rados.rados.2.smithi204.stdout:18: write initial oid 18 2024-09-16T05:10:43.182 INFO:tasks.rados.rados.2.smithi204.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-16T05:10:43.189 INFO:tasks.rados.rados.2.smithi204.stdout:18: writing smithi20457400-18 from 685719 to 1397658 tid 1 2024-09-16T05:10:43.194 INFO:tasks.rados.rados.2.smithi204.stdout:18: writing smithi20457400-18 from 1845135 to 2285719 tid 2 2024-09-16T05:10:43.194 INFO:tasks.rados.rados.2.smithi204.stdout:19: write initial oid 19 2024-09-16T05:10:43.194 INFO:tasks.rados.rados.2.smithi204.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-16T05:10:43.202 INFO:tasks.rados.rados.2.smithi204.stdout:19: writing smithi20457400-19 from 702526 to 1489714 tid 1 2024-09-16T05:10:43.206 INFO:tasks.rados.rados.2.smithi204.stdout:19: writing smithi20457400-19 from 1903617 to 2302526 tid 2 2024-09-16T05:10:43.207 INFO:tasks.rados.rados.2.smithi204.stdout:20: write initial oid 20 2024-09-16T05:10:43.207 INFO:tasks.rados.rados.2.smithi204.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-16T05:10:43.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:43.213 INFO:tasks.rados.rados.2.smithi204.stdout:20: writing smithi20457400-20 from 719333 to 1181770 tid 1 2024-09-16T05:10:43.219 INFO:tasks.rados.rados.2.smithi204.stdout:20: writing smithi20457400-20 from 1962099 to 2319333 tid 2 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout:7: finishing write tid 3 to smithi20457400-7 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout:7: finishing write tid 4 to smithi20457400-7 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout:8: finishing write tid 1 to smithi20457400-8 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout:7: done (15 left) 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout:21: write initial oid 21 2024-09-16T05:10:43.220 INFO:tasks.rados.rados.2.smithi204.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-16T05:10:43.226 INFO:tasks.rados.rados.2.smithi204.stdout:21: writing smithi20457400-21 from 736140 to 1273826 tid 1 2024-09-16T05:10:43.232 INFO:tasks.rados.rados.2.smithi204.stdout:21: writing smithi20457400-21 from 2020581 to 2336140 tid 2 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:8: finishing write tid 2 to smithi20457400-8 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:8: finishing write tid 3 to smithi20457400-8 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:8: finishing write tid 4 to smithi20457400-8 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:9: finishing write tid 1 to smithi20457400-9 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:9: finishing write tid 2 to smithi20457400-9 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:9: finishing write tid 3 to smithi20457400-9 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:9: finishing write tid 4 to smithi20457400-9 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T05:10:43.233 INFO:tasks.rados.rados.2.smithi204.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-16T05:10:43.234 INFO:tasks.rados.rados.2.smithi204.stdout:10: finishing write tid 1 to smithi20457400-10 2024-09-16T05:10:43.234 INFO:tasks.rados.rados.2.smithi204.stdout:10: finishing write tid 2 to smithi20457400-10 2024-09-16T05:10:43.234 INFO:tasks.rados.rados.2.smithi204.stdout:10: finishing write tid 3 to smithi20457400-10 2024-09-16T05:10:43.234 INFO:tasks.rados.rados.2.smithi204.stdout:8: done (15 left) 2024-09-16T05:10:43.234 INFO:tasks.rados.rados.2.smithi204.stdout:9: done (14 left) 2024-09-16T05:10:43.234 INFO:tasks.rados.rados.2.smithi204.stdout:22: write initial oid 22 2024-09-16T05:10:43.234 INFO:tasks.rados.rados.2.smithi204.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-16T05:10:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:43.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:42 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:10:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:42 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:10:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:42 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:10:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:42 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:10:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2867496412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.204:0/662043117' entity='client.2' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-16T05:10:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:42 smithi196 ceph-mon[30802]: from='client.? ' entity='client.2' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-16T05:10:43.239 INFO:tasks.rados.rados.2.smithi204.stdout:22: writing smithi20457400-22 from 752947 to 1365882 tid 1 2024-09-16T05:10:43.245 INFO:tasks.rados.rados.2.smithi204.stdout:22: writing smithi20457400-22 from 1962710 to 2352947 tid 2 2024-09-16T05:10:43.245 INFO:tasks.rados.rados.2.smithi204.stdout:23: write initial oid 23 2024-09-16T05:10:43.245 INFO:tasks.rados.rados.2.smithi204.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-16T05:10:43.252 INFO:tasks.rados.rados.2.smithi204.stdout:23: writing smithi20457400-23 from 769754 to 1457938 tid 1 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout:23: writing smithi20457400-23 from 2021192 to 2369754 tid 2 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout:11: finishing write tid 1 to smithi20457400-11 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout:11: finishing write tid 2 to smithi20457400-11 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout:11: finishing write tid 3 to smithi20457400-11 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout:11: finishing write tid 4 to smithi20457400-11 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-16T05:10:43.258 INFO:tasks.rados.rados.2.smithi204.stdout:12: finishing write tid 1 to smithi20457400-12 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:12: finishing write tid 2 to smithi20457400-12 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:12: finishing write tid 3 to smithi20457400-12 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:12: finishing write tid 4 to smithi20457400-12 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:13: finishing write tid 1 to smithi20457400-13 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:13: finishing write tid 2 to smithi20457400-13 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:13: finishing write tid 3 to smithi20457400-13 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:13: finishing write tid 4 to smithi20457400-13 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 9 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-16T05:10:43.259 INFO:tasks.rados.rados.2.smithi204.stdout:11: done (15 left) 2024-09-16T05:10:43.260 INFO:tasks.rados.rados.2.smithi204.stdout:12: done (14 left) 2024-09-16T05:10:43.260 INFO:tasks.rados.rados.2.smithi204.stdout:13: done (13 left) 2024-09-16T05:10:43.260 INFO:tasks.rados.rados.2.smithi204.stdout:24: write initial oid 24 2024-09-16T05:10:43.260 INFO:tasks.rados.rados.2.smithi204.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-16T05:10:43.264 INFO:tasks.rados.rados.2.smithi204.stdout:24: writing smithi20457400-24 from 786561 to 1266347 tid 1 2024-09-16T05:10:43.270 INFO:tasks.rados.rados.2.smithi204.stdout:24: writing smithi20457400-24 from 1796027 to 2263691 tid 2 2024-09-16T05:10:43.270 INFO:tasks.rados.rados.2.smithi204.stdout:25: write initial oid 25 2024-09-16T05:10:43.270 INFO:tasks.rados.rados.2.smithi204.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-16T05:10:43.275 INFO:tasks.rados.rados.2.smithi204.stdout:25: writing smithi20457400-25 from 403368 to 958403 tid 1 2024-09-16T05:10:43.282 INFO:tasks.rados.rados.2.smithi204.stdout:25: writing smithi20457400-25 from 1454509 to 2182184 tid 2 2024-09-16T05:10:43.282 INFO:tasks.rados.rados.2.smithi204.stdout:26: write initial oid 26 2024-09-16T05:10:43.282 INFO:tasks.rados.rados.2.smithi204.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-16T05:10:43.288 INFO:tasks.rados.rados.2.smithi204.stdout:26: writing smithi20457400-26 from 420175 to 1050459 tid 1 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:26: writing smithi20457400-26 from 1796638 to 2267971 tid 2 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:14: finishing write tid 1 to smithi20457400-14 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:14: finishing write tid 2 to smithi20457400-14 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:14: finishing write tid 3 to smithi20457400-14 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:14: finishing write tid 4 to smithi20457400-14 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:6: finishing write tid 3 to smithi20457400-6 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:6: finishing write tid 4 to smithi20457400-6 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T05:10:43.296 INFO:tasks.rados.rados.2.smithi204.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:15: finishing write tid 1 to smithi20457400-15 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:15: finishing write tid 2 to smithi20457400-15 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:15: finishing write tid 3 to smithi20457400-15 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:15: finishing write tid 4 to smithi20457400-15 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:16: finishing write tid 1 to smithi20457400-16 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:16: finishing write tid 2 to smithi20457400-16 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:16: finishing write tid 3 to smithi20457400-16 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:16: finishing write tid 4 to smithi20457400-16 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 12 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-16T05:10:43.297 INFO:tasks.rados.rados.2.smithi204.stdout:17: finishing write tid 1 to smithi20457400-17 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:17: finishing write tid 2 to smithi20457400-17 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:17: finishing write tid 3 to smithi20457400-17 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:17: finishing write tid 4 to smithi20457400-17 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:6: done (15 left) 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:14: done (14 left) 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:15: done (13 left) 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:16: done (12 left) 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:17: done (11 left) 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:27: write initial oid 27 2024-09-16T05:10:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-16T05:10:43.301 INFO:tasks.rados.rados.2.smithi204.stdout:27: writing smithi20457400-27 from 436982 to 1142515 tid 1 2024-09-16T05:10:43.308 INFO:tasks.rados.rados.2.smithi204.stdout:27: writing smithi20457400-27 from 1855120 to 2436982 tid 2 2024-09-16T05:10:43.308 INFO:tasks.rados.rados.2.smithi204.stdout:28: write initial oid 28 2024-09-16T05:10:43.308 INFO:tasks.rados.rados.2.smithi204.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-16T05:10:43.314 INFO:tasks.rados.rados.2.smithi204.stdout:28: writing smithi20457400-28 from 453789 to 1234571 tid 1 2024-09-16T05:10:43.321 INFO:tasks.rados.rados.2.smithi204.stdout:28: writing smithi20457400-28 from 1913602 to 2388604 tid 2 2024-09-16T05:10:43.321 INFO:tasks.rados.rados.2.smithi204.stdout:29: write initial oid 29 2024-09-16T05:10:43.321 INFO:tasks.rados.rados.2.smithi204.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-16T05:10:43.325 INFO:tasks.rados.rados.2.smithi204.stdout:29: writing smithi20457400-29 from 470596 to 926627 tid 1 2024-09-16T05:10:43.333 INFO:tasks.rados.rados.2.smithi204.stdout:29: writing smithi20457400-29 from 1572084 to 2190744 tid 2 2024-09-16T05:10:43.333 INFO:tasks.rados.rados.2.smithi204.stdout:30: write initial oid 30 2024-09-16T05:10:43.333 INFO:tasks.rados.rados.2.smithi204.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-16T05:10:43.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:43.339 INFO:tasks.rados.rados.2.smithi204.stdout:30: writing smithi20457400-30 from 487403 to 1018683 tid 1 2024-09-16T05:10:43.344 INFO:tasks.rados.rados.2.smithi204.stdout:30: writing smithi20457400-30 from 1514213 to 1992884 tid 2 2024-09-16T05:10:43.344 INFO:tasks.rados.rados.2.smithi204.stdout:31: write initial oid 31 2024-09-16T05:10:43.344 INFO:tasks.rados.rados.2.smithi204.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-16T05:10:43.349 INFO:tasks.rados.rados.2.smithi204.stdout:31: writing smithi20457400-31 from 504210 to 1110739 tid 1 2024-09-16T05:10:43.356 INFO:tasks.rados.rados.2.smithi204.stdout:31: writing smithi20457400-31 from 1572695 to 2195024 tid 2 2024-09-16T05:10:43.356 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.356 INFO:tasks.rados.rados.2.smithi204.stdout:18: finishing write tid 1 to smithi20457400-18 2024-09-16T05:10:43.356 INFO:tasks.rados.rados.2.smithi204.stdout:18: finishing write tid 2 to smithi20457400-18 2024-09-16T05:10:43.356 INFO:tasks.rados.rados.2.smithi204.stdout:18: finishing write tid 3 to smithi20457400-18 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:18: finishing write tid 4 to smithi20457400-18 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:19: finishing write tid 1 to smithi20457400-19 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:19: finishing write tid 2 to smithi20457400-19 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:19: finishing write tid 3 to smithi20457400-19 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:18: done (15 left) 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:32: write initial oid 32 2024-09-16T05:10:43.357 INFO:tasks.rados.rados.2.smithi204.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-16T05:10:43.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:43.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:43.363 INFO:tasks.rados.rados.2.smithi204.stdout:32: writing smithi20457400-32 from 521017 to 1319148 tid 1 2024-09-16T05:10:43.369 INFO:tasks.rados.rados.2.smithi204.stdout:32: writing smithi20457400-32 from 1747530 to 2229870 tid 2 2024-09-16T05:10:43.369 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.369 INFO:tasks.rados.rados.2.smithi204.stdout:20: finishing write tid 1 to smithi20457400-20 2024-09-16T05:10:43.369 INFO:tasks.rados.rados.2.smithi204.stdout:20: finishing write tid 2 to smithi20457400-20 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:20: finishing write tid 3 to smithi20457400-20 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:20: finishing write tid 4 to smithi20457400-20 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 15 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:10: finishing write tid 4 to smithi20457400-10 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 9 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:21: finishing write tid 1 to smithi20457400-21 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:10: done (15 left) 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:20: done (14 left) 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:33: write initial oid 33 2024-09-16T05:10:43.370 INFO:tasks.rados.rados.2.smithi204.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-16T05:10:43.374 INFO:tasks.rados.rados.2.smithi204.stdout:33: writing smithi20457400-33 from 537824 to 1011204 tid 1 2024-09-16T05:10:43.382 INFO:tasks.rados.rados.2.smithi204.stdout:33: writing smithi20457400-33 from 1806012 to 2432010 tid 2 2024-09-16T05:10:43.382 INFO:tasks.rados.rados.2.smithi204.stdout:34: write initial oid 34 2024-09-16T05:10:43.382 INFO:tasks.rados.rados.2.smithi204.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-16T05:10:43.388 INFO:tasks.rados.rados.2.smithi204.stdout:34: writing smithi20457400-34 from 554631 to 1103260 tid 1 2024-09-16T05:10:43.395 INFO:tasks.rados.rados.2.smithi204.stdout:34: writing smithi20457400-34 from 1748141 to 2234150 tid 2 2024-09-16T05:10:43.395 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.395 INFO:tasks.rados.rados.2.smithi204.stdout:21: finishing write tid 2 to smithi20457400-21 2024-09-16T05:10:43.395 INFO:tasks.rados.rados.2.smithi204.stdout:21: finishing write tid 3 to smithi20457400-21 2024-09-16T05:10:43.395 INFO:tasks.rados.rados.2.smithi204.stdout:5: finishing write tid 3 to smithi20457400-5 2024-09-16T05:10:43.395 INFO:tasks.rados.rados.2.smithi204.stdout:5: finishing write tid 4 to smithi20457400-5 2024-09-16T05:10:43.395 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:19: finishing write tid 4 to smithi20457400-19 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:22: finishing write tid 1 to smithi20457400-22 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:22: finishing write tid 2 to smithi20457400-22 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:5: done (15 left) 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:19: done (14 left) 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:35: write initial oid 35 2024-09-16T05:10:43.396 INFO:tasks.rados.rados.2.smithi204.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-16T05:10:43.401 INFO:tasks.rados.rados.2.smithi204.stdout:35: writing smithi20457400-35 from 571438 to 1195316 tid 1 2024-09-16T05:10:43.408 INFO:tasks.rados.rados.2.smithi204.stdout:35: writing smithi20457400-35 from 1806623 to 2436290 tid 2 2024-09-16T05:10:43.408 INFO:tasks.rados.rados.2.smithi204.stdout:36: write initial oid 36 2024-09-16T05:10:43.408 INFO:tasks.rados.rados.2.smithi204.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-16T05:10:43.415 INFO:tasks.rados.rados.2.smithi204.stdout:36: writing smithi20457400-36 from 588245 to 1287372 tid 1 2024-09-16T05:10:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:42 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:10:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:42 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:10:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:42 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:10:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:42 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:10:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2867496412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.204:0/662043117' entity='client.2' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-16T05:10:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:42 smithi204 ceph-mon[30933]: from='client.? ' entity='client.2' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-16T05:10:43.422 INFO:tasks.rados.rados.2.smithi204.stdout:36: writing smithi20457400-36 from 1865105 to 2354783 tid 2 2024-09-16T05:10:43.422 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:22: finishing write tid 3 to smithi20457400-22 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:23: finishing write tid 1 to smithi20457400-23 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:23: finishing write tid 2 to smithi20457400-23 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:23: finishing write tid 3 to smithi20457400-23 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:23: finishing write tid 4 to smithi20457400-23 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:22: finishing write tid 4 to smithi20457400-22 2024-09-16T05:10:43.423 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:24: finishing write tid 1 to smithi20457400-24 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:24: finishing write tid 2 to smithi20457400-24 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:24: finishing write tid 3 to smithi20457400-24 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:25: finishing write tid 1 to smithi20457400-25 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:25: finishing write tid 2 to smithi20457400-25 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:25: finishing write tid 3 to smithi20457400-25 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:25: finishing write tid 4 to smithi20457400-25 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:26: finishing write tid 1 to smithi20457400-26 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:22: done (15 left) 2024-09-16T05:10:43.424 INFO:tasks.rados.rados.2.smithi204.stdout:23: done (14 left) 2024-09-16T05:10:43.425 INFO:tasks.rados.rados.2.smithi204.stdout:25: done (13 left) 2024-09-16T05:10:43.426 INFO:tasks.rados.rados.2.smithi204.stdout:37: write initial oid 37 2024-09-16T05:10:43.426 INFO:tasks.rados.rados.2.smithi204.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-16T05:10:43.429 INFO:tasks.rados.rados.2.smithi204.stdout:37: writing smithi20457400-37 from 605052 to 1379428 tid 1 2024-09-16T05:10:43.436 INFO:tasks.rados.rados.2.smithi204.stdout:37: writing smithi20457400-37 from 1923587 to 2556923 tid 2 2024-09-16T05:10:43.436 INFO:tasks.rados.rados.2.smithi204.stdout:38: write initial oid 38 2024-09-16T05:10:43.436 INFO:tasks.rados.rados.2.smithi204.stdout:26: finishing write tid 2 to smithi20457400-26 2024-09-16T05:10:43.436 INFO:tasks.rados.rados.2.smithi204.stdout:26: finishing write tid 3 to smithi20457400-26 2024-09-16T05:10:43.436 INFO:tasks.rados.rados.2.smithi204.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-16T05:10:43.441 INFO:tasks.rados.rados.2.smithi204.stdout:38: writing smithi20457400-38 from 621859 to 1071484 tid 1 2024-09-16T05:10:43.450 INFO:tasks.rados.rados.2.smithi204.stdout:38: writing smithi20457400-38 from 1865716 to 2621859 tid 2 2024-09-16T05:10:43.450 INFO:tasks.rados.rados.2.smithi204.stdout:39: write initial oid 39 2024-09-16T05:10:43.450 INFO:tasks.rados.rados.2.smithi204.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-16T05:10:43.455 INFO:tasks.rados.rados.2.smithi204.stdout:39: writing smithi20457400-39 from 638666 to 1163540 tid 1 2024-09-16T05:10:43.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout:39: writing smithi20457400-39 from 1924198 to 2561203 tid 2 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout:26: finishing write tid 4 to smithi20457400-26 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout:27: finishing write tid 1 to smithi20457400-27 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout:26: done (15 left) 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout:40: write initial oid 40 2024-09-16T05:10:43.464 INFO:tasks.rados.rados.2.smithi204.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-16T05:10:43.471 INFO:tasks.rados.rados.2.smithi204.stdout:40: writing smithi20457400-40 from 655473 to 1371949 tid 1 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout:40: writing smithi20457400-40 from 2099033 to 2655473 tid 2 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout:27: finishing write tid 2 to smithi20457400-27 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout:27: finishing write tid 3 to smithi20457400-27 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout:27: finishing write tid 4 to smithi20457400-27 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout:28: finishing write tid 1 to smithi20457400-28 2024-09-16T05:10:43.480 INFO:tasks.rados.rados.2.smithi204.stdout:28: finishing write tid 2 to smithi20457400-28 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:28: finishing write tid 3 to smithi20457400-28 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:24: finishing write tid 4 to smithi20457400-24 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:29: finishing write tid 1 to smithi20457400-29 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:30: finishing write tid 1 to smithi20457400-30 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:30: finishing write tid 2 to smithi20457400-30 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:30: finishing write tid 3 to smithi20457400-30 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:30: finishing write tid 4 to smithi20457400-30 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:31: finishing write tid 1 to smithi20457400-31 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:31: finishing write tid 2 to smithi20457400-31 2024-09-16T05:10:43.481 INFO:tasks.rados.rados.2.smithi204.stdout:31: finishing write tid 3 to smithi20457400-31 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:31: finishing write tid 4 to smithi20457400-31 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:29: finishing write tid 2 to smithi20457400-29 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:29: finishing write tid 3 to smithi20457400-29 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:24: done (15 left) 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:27: done (14 left) 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:30: done (13 left) 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:31: done (12 left) 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:41: write initial oid 41 2024-09-16T05:10:43.482 INFO:tasks.rados.rados.2.smithi204.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-16T05:10:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:43.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:43.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:43.486 INFO:tasks.rados.rados.2.smithi204.stdout:41: writing smithi20457400-41 from 672280 to 1464005 tid 1 2024-09-16T05:10:43.493 INFO:tasks.rados.rados.2.smithi204.stdout:41: writing smithi20457400-41 from 2157515 to 2672280 tid 2 2024-09-16T05:10:43.494 INFO:tasks.rados.rados.2.smithi204.stdout:42: write initial oid 42 2024-09-16T05:10:43.494 INFO:tasks.rados.rados.2.smithi204.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-16T05:10:43.499 INFO:tasks.rados.rados.2.smithi204.stdout:42: writing smithi20457400-42 from 689087 to 1156061 tid 1 2024-09-16T05:10:43.507 INFO:tasks.rados.rados.2.smithi204.stdout:42: writing smithi20457400-42 from 1699644 to 2483976 tid 2 2024-09-16T05:10:43.507 INFO:tasks.rados.rados.2.smithi204.stdout:43: write initial oid 43 2024-09-16T05:10:43.507 INFO:tasks.rados.rados.2.smithi204.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-16T05:10:43.513 INFO:tasks.rados.rados.2.smithi204.stdout:43: writing smithi20457400-43 from 705894 to 1248117 tid 1 2024-09-16T05:10:43.519 INFO:tasks.rados.rados.2.smithi204.stdout:43: writing smithi20457400-43 from 1758126 to 2402469 tid 2 2024-09-16T05:10:43.520 INFO:tasks.rados.rados.2.smithi204.stdout:44: write initial oid 44 2024-09-16T05:10:43.520 INFO:tasks.rados.rados.2.smithi204.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-16T05:10:43.527 INFO:tasks.rados.rados.2.smithi204.stdout:44: writing smithi20457400-44 from 722701 to 1340173 tid 1 2024-09-16T05:10:43.535 INFO:tasks.rados.rados.2.smithi204.stdout:44: writing smithi20457400-44 from 1816608 to 2604609 tid 2 2024-09-16T05:10:43.535 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.535 INFO:tasks.rados.rados.2.smithi204.stdout:29: finishing write tid 4 to smithi20457400-29 2024-09-16T05:10:43.535 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T05:10:43.536 INFO:tasks.rados.rados.2.smithi204.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-16T05:10:43.536 INFO:tasks.rados.rados.2.smithi204.stdout:32: finishing write tid 1 to smithi20457400-32 2024-09-16T05:10:43.536 INFO:tasks.rados.rados.2.smithi204.stdout:29: done (15 left) 2024-09-16T05:10:43.536 INFO:tasks.rados.rados.2.smithi204.stdout:45: write initial oid 45 2024-09-16T05:10:43.536 INFO:tasks.rados.rados.2.smithi204.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-16T05:10:43.541 INFO:tasks.rados.rados.2.smithi204.stdout:45: writing smithi20457400-45 from 739508 to 1432229 tid 1 2024-09-16T05:10:43.549 INFO:tasks.rados.rados.2.smithi204.stdout:45: writing smithi20457400-45 from 1875090 to 2523102 tid 2 2024-09-16T05:10:43.549 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.549 INFO:tasks.rados.rados.2.smithi204.stdout:33: finishing write tid 1 to smithi20457400-33 2024-09-16T05:10:43.549 INFO:tasks.rados.rados.2.smithi204.stdout:33: finishing write tid 2 to smithi20457400-33 2024-09-16T05:10:43.549 INFO:tasks.rados.rados.2.smithi204.stdout:33: finishing write tid 3 to smithi20457400-33 2024-09-16T05:10:43.549 INFO:tasks.rados.rados.2.smithi204.stdout:33: finishing write tid 4 to smithi20457400-33 2024-09-16T05:10:43.549 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 18 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T05:10:43.550 INFO:tasks.rados.rados.2.smithi204.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-16T05:10:43.550 INFO:tasks.rados.rados.2.smithi204.stdout:32: finishing write tid 2 to smithi20457400-32 2024-09-16T05:10:43.550 INFO:tasks.rados.rados.2.smithi204.stdout:32: finishing write tid 3 to smithi20457400-32 2024-09-16T05:10:43.550 INFO:tasks.rados.rados.2.smithi204.stdout:32: finishing write tid 4 to smithi20457400-32 2024-09-16T05:10:43.550 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T05:10:43.550 INFO:tasks.rados.rados.2.smithi204.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-16T05:10:43.550 INFO:tasks.rados.rados.2.smithi204.stdout:34: finishing write tid 1 to smithi20457400-34 2024-09-16T05:10:43.550 INFO:tasks.rados.rados.2.smithi204.stdout:32: done (15 left) 2024-09-16T05:10:43.551 INFO:tasks.rados.rados.2.smithi204.stdout:33: done (14 left) 2024-09-16T05:10:43.551 INFO:tasks.rados.rados.2.smithi204.stdout:46: write initial oid 46 2024-09-16T05:10:43.551 INFO:tasks.rados.rados.2.smithi204.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-16T05:10:43.556 INFO:tasks.rados.rados.2.smithi204.stdout:46: writing smithi20457400-46 from 756315 to 1524285 tid 1 2024-09-16T05:10:43.564 INFO:tasks.rados.rados.2.smithi204.stdout:46: writing smithi20457400-46 from 1933572 to 2725242 tid 2 2024-09-16T05:10:43.565 INFO:tasks.rados.rados.2.smithi204.stdout:47: write initial oid 47 2024-09-16T05:10:43.565 INFO:tasks.rados.rados.2.smithi204.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-16T05:10:43.573 INFO:tasks.rados.rados.2.smithi204.stdout:47: writing smithi20457400-47 from 773122 to 1332694 tid 1 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:47: writing smithi20457400-47 from 1992054 to 2643735 tid 2 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:34: finishing write tid 2 to smithi20457400-34 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:34: finishing write tid 3 to smithi20457400-34 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:34: finishing write tid 4 to smithi20457400-34 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:35: finishing write tid 1 to smithi20457400-35 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:35: finishing write tid 2 to smithi20457400-35 2024-09-16T05:10:43.578 INFO:tasks.rados.rados.2.smithi204.stdout:35: finishing write tid 3 to smithi20457400-35 2024-09-16T05:10:43.579 INFO:tasks.rados.rados.2.smithi204.stdout:35: finishing write tid 4 to smithi20457400-35 2024-09-16T05:10:43.579 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 15 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T05:10:43.579 INFO:tasks.rados.rados.2.smithi204.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-16T05:10:43.579 INFO:tasks.rados.rados.2.smithi204.stdout:36: finishing write tid 1 to smithi20457400-36 2024-09-16T05:10:43.579 INFO:tasks.rados.rados.2.smithi204.stdout:34: done (15 left) 2024-09-16T05:10:43.579 INFO:tasks.rados.rados.2.smithi204.stdout:35: done (14 left) 2024-09-16T05:10:43.579 INFO:tasks.rados.rados.2.smithi204.stdout:48: write initial oid 48 2024-09-16T05:10:43.579 INFO:tasks.rados.rados.2.smithi204.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-16T05:10:43.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:43.585 INFO:tasks.rados.rados.2.smithi204.stdout:48: writing smithi20457400-48 from 789929 to 1424750 tid 1 2024-09-16T05:10:43.593 INFO:tasks.rados.rados.2.smithi204.stdout:48: writing smithi20457400-48 from 2050536 to 2789929 tid 2 2024-09-16T05:10:43.593 INFO:tasks.rados.rados.2.smithi204.stdout:49: write initial oid 49 2024-09-16T05:10:43.593 INFO:tasks.rados.rados.2.smithi204.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-16T05:10:43.599 INFO:tasks.rados.rados.2.smithi204.stdout:49: writing smithi20457400-49 from 406736 to 1116806 tid 1 2024-09-16T05:10:43.608 INFO:tasks.rados.rados.2.smithi204.stdout:49: writing smithi20457400-49 from 1709018 to 2364368 tid 2 2024-09-16T05:10:43.608 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.608 INFO:tasks.rados.rados.2.smithi204.stdout:36: finishing write tid 2 to smithi20457400-36 2024-09-16T05:10:43.608 INFO:tasks.rados.rados.2.smithi204.stdout:36: finishing write tid 3 to smithi20457400-36 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:37: finishing write tid 1 to smithi20457400-37 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:37: finishing write tid 2 to smithi20457400-37 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:37: finishing write tid 3 to smithi20457400-37 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:37: finishing write tid 4 to smithi20457400-37 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 3 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:38: finishing write tid 1 to smithi20457400-38 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:38: finishing write tid 2 to smithi20457400-38 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:37: done (15 left) 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:50: write initial oid 50 2024-09-16T05:10:43.609 INFO:tasks.rados.rados.2.smithi204.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-16T05:10:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:43.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:43.613 INFO:tasks.rados.rados.2.smithi204.stdout:50: writing smithi20457400-50 from 423543 to 1208862 tid 1 2024-09-16T05:10:43.621 INFO:tasks.rados.rados.2.smithi204.stdout:50: writing smithi20457400-50 from 1767500 to 2566508 tid 2 2024-09-16T05:10:43.621 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:38: finishing write tid 3 to smithi20457400-38 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:38: finishing write tid 4 to smithi20457400-38 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 18 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:39: finishing write tid 1 to smithi20457400-39 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:39: finishing write tid 2 to smithi20457400-39 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:38: done (15 left) 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:51: delete oid 24 current snap is 0 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:39: finishing write tid 3 to smithi20457400-39 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:39: finishing write tid 4 to smithi20457400-39 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 21 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T05:10:43.622 INFO:tasks.rados.rados.2.smithi204.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-16T05:10:43.623 INFO:tasks.rados.rados.2.smithi204.stdout:28: finishing write tid 4 to smithi20457400-28 2024-09-16T05:10:43.623 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:10:43.623 INFO:tasks.rados.rados.2.smithi204.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-16T05:10:43.623 INFO:tasks.rados.rados.2.smithi204.stdout:40: finishing write tid 1 to smithi20457400-40 2024-09-16T05:10:43.623 INFO:tasks.rados.rados.2.smithi204.stdout:40: finishing write tid 2 to smithi20457400-40 2024-09-16T05:10:43.623 INFO:tasks.rados.rados.2.smithi204.stdout:40: finishing write tid 3 to smithi20457400-40 2024-09-16T05:10:43.623 INFO:tasks.rados.rados.2.smithi204.stdout:21: finishing write tid 4 to smithi20457400-21 2024-09-16T05:10:43.624 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T05:10:43.624 INFO:tasks.rados.rados.2.smithi204.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:41: finishing write tid 1 to smithi20457400-41 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:42: finishing write tid 1 to smithi20457400-42 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:42: finishing write tid 2 to smithi20457400-42 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:42: finishing write tid 3 to smithi20457400-42 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:42: finishing write tid 4 to smithi20457400-42 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 21 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:43: finishing write tid 1 to smithi20457400-43 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:43: finishing write tid 2 to smithi20457400-43 2024-09-16T05:10:43.626 INFO:tasks.rados.rados.2.smithi204.stdout:43: finishing write tid 3 to smithi20457400-43 2024-09-16T05:10:43.627 INFO:tasks.rados.rados.2.smithi204.stdout:43: finishing write tid 4 to smithi20457400-43 2024-09-16T05:10:43.627 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T05:10:43.627 INFO:tasks.rados.rados.2.smithi204.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-16T05:10:43.627 INFO:tasks.rados.rados.2.smithi204.stdout:44: finishing write tid 1 to smithi20457400-44 2024-09-16T05:10:43.627 INFO:tasks.rados.rados.2.smithi204.stdout:44: finishing write tid 2 to smithi20457400-44 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:44: finishing write tid 3 to smithi20457400-44 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:44: finishing write tid 4 to smithi20457400-44 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:45: finishing write tid 1 to smithi20457400-45 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:45: finishing write tid 2 to smithi20457400-45 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:45: finishing write tid 3 to smithi20457400-45 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:45: finishing write tid 4 to smithi20457400-45 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T05:10:43.628 INFO:tasks.rados.rados.2.smithi204.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:41: finishing write tid 2 to smithi20457400-41 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:46: finishing write tid 1 to smithi20457400-46 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:46: finishing write tid 2 to smithi20457400-46 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:46: finishing write tid 3 to smithi20457400-46 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:47: finishing write tid 1 to smithi20457400-47 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:47: finishing write tid 2 to smithi20457400-47 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:47: finishing write tid 3 to smithi20457400-47 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:47: finishing write tid 4 to smithi20457400-47 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:36: finishing write tid 4 to smithi20457400-36 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T05:10:43.629 INFO:tasks.rados.rados.2.smithi204.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:48: finishing write tid 1 to smithi20457400-48 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:48: finishing write tid 2 to smithi20457400-48 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:48: finishing write tid 3 to smithi20457400-48 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:49: finishing write tid 1 to smithi20457400-49 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:49: finishing write tid 2 to smithi20457400-49 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:49: finishing write tid 3 to smithi20457400-49 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:49: finishing write tid 4 to smithi20457400-49 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:46: finishing write tid 4 to smithi20457400-46 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 3 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-16T05:10:43.630 INFO:tasks.rados.rados.2.smithi204.stdout:48: finishing write tid 4 to smithi20457400-48 2024-09-16T05:10:43.631 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T05:10:43.631 INFO:tasks.rados.rados.2.smithi204.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-16T05:10:43.631 INFO:tasks.rados.rados.2.smithi204.stdout:50: finishing write tid 1 to smithi20457400-50 2024-09-16T05:10:43.631 INFO:tasks.rados.rados.2.smithi204.stdout:50: finishing write tid 2 to smithi20457400-50 2024-09-16T05:10:43.631 INFO:tasks.rados.rados.2.smithi204.stdout:50: finishing write tid 3 to smithi20457400-50 2024-09-16T05:10:43.670 INFO:tasks.rados.rados.2.smithi204.stdout:50: finishing write tid 4 to smithi20457400-50 2024-09-16T05:10:43.670 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T05:10:43.670 INFO:tasks.rados.rados.2.smithi204.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-16T05:10:43.670 INFO:tasks.rados.rados.2.smithi204.stdout:41: finishing write tid 3 to smithi20457400-41 2024-09-16T05:10:43.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:10:43.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:43.720 INFO:tasks.rados.rados.2.smithi204.stdout:40: finishing write tid 4 to smithi20457400-40 2024-09-16T05:10:43.720 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T05:10:43.720 INFO:tasks.rados.rados.2.smithi204.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-16T05:10:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:43.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:43.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:43.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:43.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:43.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:43.886 INFO:tasks.rados.rados.2.smithi204.stdout:41: finishing write tid 4 to smithi20457400-41 2024-09-16T05:10:43.887 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 12 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T05:10:43.887 INFO:tasks.rados.rados.2.smithi204.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:10:43.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:44.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:44.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:44.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:21: done (15 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:28: done (14 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:36: done (13 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:39: done (12 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:40: done (11 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:41: done (10 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:42: done (9 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:43: done (8 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:44: done (7 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:45: done (6 left) 2024-09-16T05:10:44.127 INFO:tasks.rados.rados.2.smithi204.stdout:46: done (5 left) 2024-09-16T05:10:44.128 INFO:tasks.rados.rados.2.smithi204.stdout:47: done (4 left) 2024-09-16T05:10:44.128 INFO:tasks.rados.rados.2.smithi204.stdout:48: done (3 left) 2024-09-16T05:10:44.128 INFO:tasks.rados.rados.2.smithi204.stdout:49: done (2 left) 2024-09-16T05:10:44.128 INFO:tasks.rados.rados.2.smithi204.stdout:50: done (1 left) 2024-09-16T05:10:44.128 INFO:tasks.rados.rados.2.smithi204.stdout:51: done (0 left) 2024-09-16T05:10:44.128 INFO:tasks.rados.rados.2.smithi204.stdout:52: read oid 19 snap -1 2024-09-16T05:10:44.128 INFO:tasks.rados.rados.2.smithi204.stdout:52: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-16T05:10:44.128 INFO:tasks.rados.rados.2.smithi204.stdout:53: snap_create 2024-09-16T05:10:44.132 INFO:tasks.rados.rados.2.smithi204.stdout:52: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-16T05:10:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:44.135 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:44.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:44.236 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:44.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:43 smithi196 ceph-mon[30802]: pgmap v319: 27 pgs: 26 active+clean, 1 remapped+peering; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 52 KiB/s wr, 5 op/s; 91 KiB/s, 0 objects/s recovering 2024-09-16T05:10:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:43 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:10:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:43 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:10:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:43 smithi196 ceph-mon[30802]: from='client.? ' entity='client.2' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-16T05:10:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:43 smithi196 ceph-mon[30802]: osdmap e112: 12 total, 12 up, 12 in 2024-09-16T05:10:44.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:44.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:44.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:43 smithi032 ceph-mon[27388]: pgmap v319: 27 pgs: 26 active+clean, 1 remapped+peering; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 52 KiB/s wr, 5 op/s; 91 KiB/s, 0 objects/s recovering 2024-09-16T05:10:44.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:43 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:10:44.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:43 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:10:44.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:43 smithi032 ceph-mon[27388]: from='client.? ' entity='client.2' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-16T05:10:44.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:43 smithi032 ceph-mon[27388]: osdmap e112: 12 total, 12 up, 12 in 2024-09-16T05:10:44.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:44.385 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:44.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:44.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:43 smithi204 ceph-mon[30933]: pgmap v319: 27 pgs: 26 active+clean, 1 remapped+peering; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 52 KiB/s wr, 5 op/s; 91 KiB/s, 0 objects/s recovering 2024-09-16T05:10:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:43 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:10:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:43 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:10:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:43 smithi204 ceph-mon[30933]: from='client.? ' entity='client.2' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-16T05:10:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:43 smithi204 ceph-mon[30933]: osdmap e112: 12 total, 12 up, 12 in 2024-09-16T05:10:44.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:44.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:44.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:44.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:44.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:44.701 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:44.726 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:10:44.726 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:10:44.727 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T05:10:44.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:10:44.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:44.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:44.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:44.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:44.892 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:44.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:10:44.903+0000 7f9249d38640 0 --2- 172.21.15.32:0/3948514654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9244102d30 0x7f9244103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:10:44.989 INFO:tasks.rados.rados.2.smithi204.stdout:52: done (1 left) 2024-09-16T05:10:44.989 INFO:tasks.rados.rados.2.smithi204.stdout:53: done (0 left) 2024-09-16T05:10:44.989 INFO:tasks.rados.rados.2.smithi204.stdout:54: write oid 46 current snap is 1 2024-09-16T05:10:44.989 INFO:tasks.rados.rados.2.smithi204.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-16T05:10:44.994 INFO:tasks.rados.rados.2.smithi204.stdout:54: writing smithi20457400-46 from 457157 to 992974 tid 1 2024-09-16T05:10:44.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:45.003 INFO:tasks.rados.rados.2.smithi204.stdout:54: writing smithi20457400-46 from 1768111 to 2170788 tid 2 2024-09-16T05:10:45.003 INFO:tasks.rados.rados.2.smithi204.stdout:55: write oid 3 current snap is 1 2024-09-16T05:10:45.004 INFO:tasks.rados.rados.2.smithi204.stdout:55: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-09-16T05:10:45.006 INFO:tasks.rados.rados.2.smithi204.stdout:55: writing smithi20457400-3 from 473964 to 1085030 tid 1 2024-09-16T05:10:45.014 INFO:tasks.rados.rados.2.smithi204.stdout:55: writing smithi20457400-3 from 1826593 to 2372928 tid 2 2024-09-16T05:10:45.017 INFO:tasks.rados.rados.2.smithi204.stdout:55: writing smithi20457400-3 from 2778819 to 2873964 tid 3 2024-09-16T05:10:45.017 INFO:tasks.rados.rados.2.smithi204.stdout:56: read oid 45 snap 0 2024-09-16T05:10:45.017 INFO:tasks.rados.rados.2.smithi204.stdout:56: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-16T05:10:45.017 INFO:tasks.rados.rados.2.smithi204.stdout:57: rollback oid 50 current snap is 1 2024-09-16T05:10:45.017 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 0 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:54: finishing write tid 1 to smithi20457400-46 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:55: finishing write tid 1 to smithi20457400-3 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:54: finishing write tid 2 to smithi20457400-46 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:58: snap_create 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:54: finishing write tid 3 to smithi20457400-46 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:54: finishing write tid 4 to smithi20457400-46 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 7 (ObjNum 51 snap 1 seq_num 51) dirty exists 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:54: left oid 46 (ObjNum 51 snap 1 seq_num 51) 2024-09-16T05:10:45.018 INFO:tasks.rados.rados.2.smithi204.stdout:55: finishing write tid 2 to smithi20457400-3 2024-09-16T05:10:45.019 INFO:tasks.rados.rados.2.smithi204.stdout:55: finishing write tid 3 to smithi20457400-3 2024-09-16T05:10:45.019 INFO:tasks.rados.rados.2.smithi204.stdout:57: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:10:45.019 INFO:tasks.rados.rados.2.smithi204.stdout:57: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:10:45.020 INFO:tasks.rados.rados.2.smithi204.stdout:55: finishing write tid 4 to smithi20457400-3 2024-09-16T05:10:45.020 INFO:tasks.rados.rados.2.smithi204.stdout:55: finishing write tid 5 to smithi20457400-3 2024-09-16T05:10:45.020 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 26 (ObjNum 52 snap 1 seq_num 52) dirty exists 2024-09-16T05:10:45.020 INFO:tasks.rados.rados.2.smithi204.stdout:55: left oid 3 (ObjNum 52 snap 1 seq_num 52) 2024-09-16T05:10:45.020 INFO:tasks.rados.rados.2.smithi204.stdout:57: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:10:45.020 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 13 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T05:10:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:45.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:45.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:45.022 INFO:tasks.rados.rados.2.smithi204.stdout:56: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-16T05:10:45.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:45.145 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:45.145 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:10:45.145 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:10:45.145 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:10:45.146 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:45.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:45.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:45.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:44 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:10:45.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:44 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:10:45.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:44 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:10:45.158 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:44 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:10:45.159 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:44 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:10:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:44 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:10:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:44 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:10:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:44 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:10:45.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:45.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:45.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:45.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:45.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:45.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:45.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:45.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:44 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:10:45.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:44 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:10:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:44 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:10:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:44 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:10:45.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:45.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:45.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:45.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:45.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:45.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:45.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:45.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:45.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:45.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:10:45.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:45.905 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:45.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:45.991 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:10:45.994 INFO:tasks.rados.rados.2.smithi204.stdout:54: done (4 left) 2024-09-16T05:10:45.994 INFO:tasks.rados.rados.2.smithi204.stdout:55: done (3 left) 2024-09-16T05:10:45.994 INFO:tasks.rados.rados.2.smithi204.stdout:56: done (2 left) 2024-09-16T05:10:45.994 INFO:tasks.rados.rados.2.smithi204.stdout:57: done (1 left) 2024-09-16T05:10:45.995 INFO:tasks.rados.rados.2.smithi204.stdout:58: done (0 left) 2024-09-16T05:10:45.995 INFO:tasks.rados.rados.2.smithi204.stdout:59: write oid 40 current snap is 2 2024-09-16T05:10:45.995 INFO:tasks.rados.rados.2.smithi204.stdout:59: seq_num 53 ranges {490771=686315,1885075=406346} 2024-09-16T05:10:46.000 INFO:tasks.rados.rados.2.smithi204.stdout:59: writing smithi20457400-40 from 490771 to 1177086 tid 1 2024-09-16T05:10:46.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:46.006 INFO:tasks.rados.rados.2.smithi204.stdout:59: writing smithi20457400-40 from 1885075 to 2291421 tid 2 2024-09-16T05:10:46.006 INFO:tasks.rados.rados.2.smithi204.stdout:60: snap_remove snap 0 2024-09-16T05:10:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:46.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:46.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:46.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:45 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:10:46.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:45 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:10:46.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:45 smithi032 ceph-mon[27388]: pgmap v321: 27 pgs: 27 active+clean; 36 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-16T05:10:46.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:45 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:10:46.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:45 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:10:46.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:45 smithi032 ceph-mon[27388]: osdmap e113: 12 total, 12 up, 12 in 2024-09-16T05:10:46.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3804110357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:46.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:46.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:10:46.157 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:46.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:46.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:46.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:46.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:46.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:46.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:46.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:46.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:46.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:45 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:10:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:45 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:10:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:45 smithi204 ceph-mon[30933]: pgmap v321: 27 pgs: 27 active+clean; 36 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-16T05:10:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:45 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:10:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:45 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:10:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:45 smithi204 ceph-mon[30933]: osdmap e113: 12 total, 12 up, 12 in 2024-09-16T05:10:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3804110357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:45 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:10:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:45 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:10:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:45 smithi196 ceph-mon[30802]: pgmap v321: 27 pgs: 27 active+clean; 36 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2.7 MiB/s wr, 10 op/s 2024-09-16T05:10:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:45 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:10:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:45 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:10:46.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:45 smithi196 ceph-mon[30802]: osdmap e113: 12 total, 12 up, 12 in 2024-09-16T05:10:46.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3804110357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:46.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:46.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:46.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:46.661 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:46.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:46.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:46.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:46.785 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:46.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:46.887 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:46.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:46.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:46.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:47.003 INFO:tasks.rados.rados.2.smithi204.stdout:60: done (1 left) 2024-09-16T05:10:47.003 INFO:tasks.rados.rados.2.smithi204.stdout:61: delete oid 44 current snap is 2 2024-09-16T05:10:47.003 INFO:tasks.rados.rados.2.smithi204.stdout:59: finishing write tid 1 to smithi20457400-40 2024-09-16T05:10:47.003 INFO:tasks.rados.rados.2.smithi204.stdout:59: finishing write tid 2 to smithi20457400-40 2024-09-16T05:10:47.003 INFO:tasks.rados.rados.2.smithi204.stdout:59: finishing write tid 3 to smithi20457400-40 2024-09-16T05:10:47.003 INFO:tasks.rados.rados.2.smithi204.stdout:59: finishing write tid 4 to smithi20457400-40 2024-09-16T05:10:47.003 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 16 (ObjNum 53 snap 2 seq_num 53) dirty exists 2024-09-16T05:10:47.003 INFO:tasks.rados.rados.2.smithi204.stdout:59: left oid 40 (ObjNum 53 snap 2 seq_num 53) 2024-09-16T05:10:47.005 INFO:tasks.rados.rados.2.smithi204.stdout:59: done (1 left) 2024-09-16T05:10:47.005 INFO:tasks.rados.rados.2.smithi204.stdout:61: done (0 left) 2024-09-16T05:10:47.005 INFO:tasks.rados.rados.2.smithi204.stdout:62: write (excl) oid 36 current snap is 2 2024-09-16T05:10:47.005 INFO:tasks.rados.rados.2.smithi204.stdout:62: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-16T05:10:47.010 INFO:tasks.rados.rados.2.smithi204.stdout:62: writing smithi20457400-36 from 524385 to 1077551 tid 1 2024-09-16T05:10:47.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:47.015 INFO:tasks.rados.rados.2.smithi204.stdout:62: writing smithi20457400-36 from 1602039 to 2012054 tid 2 2024-09-16T05:10:47.021 INFO:tasks.rados.rados.2.smithi204.stdout:62: writing smithi20457400-36 from 2577441 to 2924385 tid 3 2024-09-16T05:10:47.021 INFO:tasks.rados.rados.2.smithi204.stdout:63: delete oid 29 current snap is 2 2024-09-16T05:10:47.021 INFO:tasks.rados.rados.2.smithi204.stdout:62: finishing write tid 1 to smithi20457400-36 2024-09-16T05:10:47.021 INFO:tasks.rados.rados.2.smithi204.stdout:62: finishing write tid 2 to smithi20457400-36 2024-09-16T05:10:47.024 INFO:tasks.rados.rados.2.smithi204.stdout:62: finishing write tid 3 to smithi20457400-36 2024-09-16T05:10:47.024 INFO:tasks.rados.rados.2.smithi204.stdout:62: finishing write tid 4 to smithi20457400-36 2024-09-16T05:10:47.024 INFO:tasks.rados.rados.2.smithi204.stdout:62: finishing write tid 5 to smithi20457400-36 2024-09-16T05:10:47.024 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 23 (ObjNum 55 snap 2 seq_num 55) dirty exists 2024-09-16T05:10:47.024 INFO:tasks.rados.rados.2.smithi204.stdout:62: left oid 36 (ObjNum 55 snap 2 seq_num 55) 2024-09-16T05:10:47.024 INFO:tasks.rados.rados.2.smithi204.stdout:62: done (1 left) 2024-09-16T05:10:47.024 INFO:tasks.rados.rados.2.smithi204.stdout:63: done (0 left) 2024-09-16T05:10:47.025 INFO:tasks.rados.rados.2.smithi204.stdout:64: read oid 3 snap -1 2024-09-16T05:10:47.025 INFO:tasks.rados.rados.2.smithi204.stdout:64: expect (ObjNum 52 snap 1 seq_num 52) 2024-09-16T05:10:47.025 INFO:tasks.rados.rados.2.smithi204.stdout:65: write oid 39 current snap is 2 2024-09-16T05:10:47.025 INFO:tasks.rados.rados.2.smithi204.stdout:65: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-16T05:10:47.031 INFO:tasks.rados.rados.2.smithi204.stdout:65: writing smithi20457400-39 from 557999 to 1261663 tid 1 2024-09-16T05:10:47.036 INFO:tasks.rados.rados.2.smithi204.stdout:65: writing smithi20457400-39 from 1719003 to 2132687 tid 2 2024-09-16T05:10:47.037 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 115, 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:10:46.991384+0000", 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:09:26.475677+0000", 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:09:05.828775+0000", 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2024-09-16T05:10:47.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 27, 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:10:47.039 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:10:47.040 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 11, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 11, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 11, 2024-09-16T05:10:47.041 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 11, 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 11, 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "106", 2024-09-16T05:10:47.042 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "104", 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:10:47.043 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:10:47.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:10:47.045 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:10:47.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.048 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 3.2899999618530273, 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 3.2899999618530273, 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.82999998331069946, 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.7300000190734863, 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.7300000190734863, 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 1, 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-16T05:10:47.049 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:10:47.050 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 16, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 16, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 16, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 16, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 16, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:10:47.051 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "115", 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 3, 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 115, 2024-09-16T05:10:47.052 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:10:47.053 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:10:47.054 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.25, 2024-09-16T05:10:47.055 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.25, 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.25, 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.25, 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 1, 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:10:47.056 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 61, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.057 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:47.060 INFO:tasks.rados.rados.2.smithi204.stdout:65: writing smithi20457400-39 from 2632287 to 2957999 tid 3 2024-09-16T05:10:47.060 INFO:tasks.rados.rados.2.smithi204.stdout:66: snap_remove snap 1 2024-09-16T05:10:47.061 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:10:47.061 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:47.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.061 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.061 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:10:47.061 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:10:47.062 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.065 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.066 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.067 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:10:47.070 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 21, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.071 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.072 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3508564664", 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3508564664", 2024-09-16T05:10:47.075 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3508564664", 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3508564664", 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.076 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:47.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.081 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-09-16T05:10:47.082 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.083 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:47.084 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:47.085 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:47.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.087 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.089 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.090 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.091 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.093 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:10:47.096 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:47.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:10:47.100 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.101 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.101 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.101 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.101 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.101 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 108, 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.102 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.106 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.107 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:47.109 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:47.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:10:47.112 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.113 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.114 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.115 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:10:47.116 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:10:47.118 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:10:47.119 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:10:47.120 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:10:47.123 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.127 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:47.128 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.129 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.130 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.131 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:47.132 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:47.133 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:10:47.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.134 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:10:47.134 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.134 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:29.755221+0000", 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.135 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:02.790003+0000", 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.136 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:29.046992+0000", 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:04.735259+0000", 2024-09-16T05:10:47.137 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:00.052444+0000", 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:10:47.138 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:39.367915+0000", 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.139 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:49.025968+0000", 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.140 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:02.089795+0000", 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:35.890683+0000", 2024-09-16T05:10:47.141 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.142 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:10:47.142 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.142 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.142 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.142 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.142 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:47.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:47.143 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:59.269066+0000", 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.144 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:59.495208+0000", 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:38.259572+0000", 2024-09-16T05:10:47.145 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:47.146 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:10:47.147 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.148 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 2, 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [ 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.149 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 2, 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:10:47.150 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:10:47.151 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:10:47.151 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:10:47.151 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:10:47.151 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:10:47.151 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:10:47.151 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:47.151 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:47.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:10:47.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:10:47.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:10:47.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:10:47.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:10:47.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:10:47.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:10:47.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:10:47.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3804110357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:10:47.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:46 smithi032 ceph-mon[27388]: osdmap e114: 12 total, 12 up, 12 in 2024-09-16T05:10:47.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:47.266 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:47.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:47.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:47.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:47.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:47.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3804110357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:10:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:46 smithi204 ceph-mon[30933]: osdmap e114: 12 total, 12 up, 12 in 2024-09-16T05:10:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:10:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:10:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:10:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:10:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:10:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:10:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:10:47.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:10:47.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3804110357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:10:47.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:46 smithi196 ceph-mon[30802]: osdmap e114: 12 total, 12 up, 12 in 2024-09-16T05:10:47.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:47.522 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:47.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:47.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:47.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:47.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:47.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:47.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:47.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:47.898 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:47.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:48.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:48.005 INFO:tasks.rados.rados.2.smithi204.stdout:64: expect (ObjNum 52 snap 1 seq_num 52) 2024-09-16T05:10:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:48.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:48.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:48.093 INFO:tasks.thrashosds.thrasher:j is {'epoch': 115, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:10:46.991384+0000', 'last_up_change': '2024-09-16T05:09:26.475677+0000', 'last_in_change': '2024-09-16T05:09:05.828775+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '104', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2899999618530273, 'score_stable': 3.2899999618530273, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.7300000190734863, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 115, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3508564664}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3508564664}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3508564664}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3508564664}]}, 'public_addr': '172.21.15.32:6819/3508564664', 'cluster_addr': '172.21.15.32:6821/3508564664', 'heartbeat_back_addr': '172.21.15.32:6825/3508564664', 'heartbeat_front_addr': '172.21.15.32:6823/3508564664', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 108, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:29.755221+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:02.790003+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:29.046992+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:04.735259+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:00.052444+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:39.367915+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:49.025968+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:02.089795+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:35.890683+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:59.269066+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:59.495208+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:38.259572+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:10:48.093 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T05:10:48.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:48.134 INFO:tasks.rados.rados.2.smithi204.stdout:65: finishing write tid 1 to smithi20457400-39 2024-09-16T05:10:48.134 INFO:tasks.rados.rados.2.smithi204.stdout:65: finishing write tid 2 to smithi20457400-39 2024-09-16T05:10:48.134 INFO:tasks.rados.rados.2.smithi204.stdout:65: finishing write tid 3 to smithi20457400-39 2024-09-16T05:10:48.134 INFO:tasks.rados.rados.2.smithi204.stdout:65: finishing write tid 4 to smithi20457400-39 2024-09-16T05:10:48.134 INFO:tasks.rados.rados.2.smithi204.stdout:65: finishing write tid 5 to smithi20457400-39 2024-09-16T05:10:48.134 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 33 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:10:48.134 INFO:tasks.rados.rados.2.smithi204.stdout:65: left oid 39 (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:10:48.135 INFO:tasks.rados.rados.2.smithi204.stdout:64: done (2 left) 2024-09-16T05:10:48.135 INFO:tasks.rados.rados.2.smithi204.stdout:65: done (1 left) 2024-09-16T05:10:48.135 INFO:tasks.rados.rados.2.smithi204.stdout:66: done (0 left) 2024-09-16T05:10:48.135 INFO:tasks.rados.rados.2.smithi204.stdout:67: delete oid 10 current snap is 2 2024-09-16T05:10:48.136 INFO:tasks.rados.rados.2.smithi204.stdout:67: done (0 left) 2024-09-16T05:10:48.136 INFO:tasks.rados.rados.2.smithi204.stdout:68: write oid 33 current snap is 2 2024-09-16T05:10:48.136 INFO:tasks.rados.rados.2.smithi204.stdout:68: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-16T05:10:48.142 INFO:tasks.rados.rados.2.smithi204.stdout:68: writing smithi20457400-33 from 591613 to 1045775 tid 1 2024-09-16T05:10:48.148 INFO:tasks.rados.rados.2.smithi204.stdout:68: writing smithi20457400-33 from 1719614 to 2136967 tid 2 2024-09-16T05:10:48.153 INFO:tasks.rados.rados.2.smithi204.stdout:68: writing smithi20457400-33 from 2570780 to 2991613 tid 3 2024-09-16T05:10:48.153 INFO:tasks.rados.rados.2.smithi204.stdout:69: write (excl) oid 4 current snap is 2 2024-09-16T05:10:48.154 INFO:tasks.rados.rados.2.smithi204.stdout:68: finishing write tid 1 to smithi20457400-33 2024-09-16T05:10:48.154 INFO:tasks.rados.rados.2.smithi204.stdout:68: finishing write tid 2 to smithi20457400-33 2024-09-16T05:10:48.154 INFO:tasks.rados.rados.2.smithi204.stdout:69: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-16T05:10:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:48.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:48.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:48.159 INFO:tasks.rados.rados.2.smithi204.stdout:69: writing smithi20457400-4 from 608420 to 1137831 tid 1 2024-09-16T05:10:48.165 INFO:tasks.rados.rados.2.smithi204.stdout:69: writing smithi20457400-4 from 1778096 to 2339107 tid 2 2024-09-16T05:10:48.169 INFO:tasks.rados.rados.2.smithi204.stdout:69: writing smithi20457400-4 from 2881850 to 3008420 tid 3 2024-09-16T05:10:48.169 INFO:tasks.rados.rados.2.smithi204.stdout:70: rollback oid 48 current snap is 2 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:68: finishing write tid 3 to smithi20457400-33 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:68: finishing write tid 4 to smithi20457400-33 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:68: finishing write tid 5 to smithi20457400-33 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 25 (ObjNum 59 snap 2 seq_num 59) dirty exists 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:68: left oid 33 (ObjNum 59 snap 2 seq_num 59) 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:69: finishing write tid 1 to smithi20457400-4 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:69: finishing write tid 2 to smithi20457400-4 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:68: done (2 left) 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:70: done (1 left) 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:71: rollback oid 17 current snap is 2 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:71: done (1 left) 2024-09-16T05:10:48.170 INFO:tasks.rados.rados.2.smithi204.stdout:72: delete oid 32 current snap is 2 2024-09-16T05:10:48.171 INFO:tasks.rados.rados.2.smithi204.stdout:72: done (1 left) 2024-09-16T05:10:48.171 INFO:tasks.rados.rados.2.smithi204.stdout:73: read oid 41 snap -1 2024-09-16T05:10:48.171 INFO:tasks.rados.rados.2.smithi204.stdout:73: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:10:48.171 INFO:tasks.rados.rados.2.smithi204.stdout:74: write (excl) oid 23 current snap is 2 2024-09-16T05:10:48.171 INFO:tasks.rados.rados.2.smithi204.stdout:74: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-16T05:10:48.178 INFO:tasks.rados.rados.2.smithi204.stdout:74: writing smithi20457400-23 from 642034 to 1438296 tid 1 2024-09-16T05:10:48.184 INFO:tasks.rados.rados.2.smithi204.stdout:74: writing smithi20457400-23 from 1895060 to 2459740 tid 2 2024-09-16T05:10:48.184 INFO:tasks.rados.rados.2.smithi204.stdout:75: read oid 3 snap -1 2024-09-16T05:10:48.184 INFO:tasks.rados.rados.2.smithi204.stdout:75: expect (ObjNum 52 snap 1 seq_num 52) 2024-09-16T05:10:48.184 INFO:tasks.rados.rados.2.smithi204.stdout:76: read oid 30 snap -1 2024-09-16T05:10:48.185 INFO:tasks.rados.rados.2.smithi204.stdout:76: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T05:10:48.185 INFO:tasks.rados.rados.2.smithi204.stdout:77: snap_create 2024-09-16T05:10:48.185 INFO:tasks.rados.rados.2.smithi204.stdout:69: finishing write tid 3 to smithi20457400-4 2024-09-16T05:10:48.185 INFO:tasks.rados.rados.2.smithi204.stdout:69: finishing write tid 4 to smithi20457400-4 2024-09-16T05:10:48.185 INFO:tasks.rados.rados.2.smithi204.stdout:69: finishing write tid 5 to smithi20457400-4 2024-09-16T05:10:48.185 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 22 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-09-16T05:10:48.185 INFO:tasks.rados.rados.2.smithi204.stdout:69: left oid 4 (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:10:48.185 INFO:tasks.rados.rados.2.smithi204.stdout:73: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:10:48.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:48.271 INFO:tasks.rados.rados.2.smithi204.stdout:74: finishing write tid 1 to smithi20457400-23 2024-09-16T05:10:48.271 INFO:tasks.rados.rados.2.smithi204.stdout:74: finishing write tid 2 to smithi20457400-23 2024-09-16T05:10:48.271 INFO:tasks.rados.rados.2.smithi204.stdout:74: finishing write tid 3 to smithi20457400-23 2024-09-16T05:10:48.271 INFO:tasks.rados.rados.2.smithi204.stdout:75: expect (ObjNum 52 snap 1 seq_num 52) 2024-09-16T05:10:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:48.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:48.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:48.296 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:47 smithi032 ceph-mon[27388]: pgmap v324: 27 pgs: 27 active+clean; 76 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 5.7 MiB/s wr, 9 op/s 2024-09-16T05:10:48.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:47 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:10:48.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:47 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:10:48.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:47 smithi032 ceph-mon[27388]: osdmap e115: 12 total, 12 up, 12 in 2024-09-16T05:10:48.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3705317788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:10:48.302 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:48.308 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:48.308 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5 2.1 2:8c1ba486:::smithi20457400-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.004952+0000", 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.3016002049999997, 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004144816, 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.310 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.311 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.311 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5 2024-09-16T05:10:48.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.311 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.004952+0000", 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.004952+0000", 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.004954+0000", 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6529999999999998e-06 2024-09-16T05:10:48.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.005584+0000", 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062926699999999998 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.005597+0000", 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3271999999999999e-05 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.005605+0000", 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7800000000000001e-06 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.005668+0000", 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2937999999999999e-05 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.005676+0000", 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1200000000000002e-06 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006778+0000", 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001102638 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006871+0000", 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2709999999999998e-05 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006981+0000", 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010986 2024-09-16T05:10:48.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008620+0000", 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016384959999999999 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008987+0000", 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036797199999999999 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009043+0000", 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5791999999999997e-05 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009097+0000", 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3318e-05 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:6 2.1 2:8c1ba486:::smithi20457400-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.317 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.006181+0000", 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.300370655, 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032165280000000002, 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 6 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006181+0000", 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006181+0000", 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006182+0000", 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7300000000000005e-07 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006634+0000", 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000451447 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006646+0000", 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2232000000000001e-05 2024-09-16T05:10:48.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006654+0000", 2024-09-16T05:10:48.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9279999999999993e-06 2024-09-16T05:10:48.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007803+0000", 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011488799999999999 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007848+0000", 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5692000000000002e-05 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007936+0000", 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7754e-05 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008626+0000", 2024-09-16T05:10:48.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068980799999999998 2024-09-16T05:10:48.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009300+0000", 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00067451700000000002 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009354+0000", 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3853000000000001e-05 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009398+0000", 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3643999999999998e-05 2024-09-16T05:10:48.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7 2.1 2:8c1ba486:::smithi20457400-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.006679+0000", 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.2998732679999998, 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028325999999999998, 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.325 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.326 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.326 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.326 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.326 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.326 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7 2024-09-16T05:10:48.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006679+0000", 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006679+0000", 2024-09-16T05:10:48.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006680+0000", 2024-09-16T05:10:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2399999999999996e-07 2024-09-16T05:10:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006684+0000", 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4730000000000002e-06 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006692+0000", 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4970000000000004e-06 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006699+0000", 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9480000000000002e-06 2024-09-16T05:10:48.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008312+0000", 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001613628 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008358+0000", 2024-09-16T05:10:48.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5580000000000001e-05 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008450+0000", 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2038000000000002e-05 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008721+0000", 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00027111799999999999 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009433+0000", 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071226700000000004 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009464+0000", 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1204999999999997e-05 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009511+0000", 2024-09-16T05:10:48.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6922000000000002e-05 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8 2.1 2:8c1ba486:::smithi20457400-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.006705+0000", 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.2998466339999997, 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030133289999999999, 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:10:48.333 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006705+0000", 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006705+0000", 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006706+0000", 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6300000000000001e-07 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006713+0000", 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4150000000000002e-06 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006718+0000", 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0130000000000001e-06 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.006722+0000", 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6040000000000001e-06 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008646+0000", 2024-09-16T05:10:48.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019243089999999999 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008687+0000", 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1060999999999999e-05 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009554+0000", 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00086672699999999997 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009628+0000", 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4552e-05 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.009719+0000", 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0284999999999994e-05 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.338 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:13 2.e 2:7e4e65a4:::smithi20457400-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.023840+0000", 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.2827117059999997, 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022592049999999998, 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.339 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 13 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.023840+0000", 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.023840+0000", 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.023842+0000", 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3200000000000001e-06 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.024404+0000", 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000562723 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.024418+0000", 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3614e-05 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.024424+0000", 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7960000000000001e-06 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.024487+0000", 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3502000000000003e-05 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.024564+0000", 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7194999999999995e-05 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.024659+0000", 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4532999999999998e-05 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.025820+0000", 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011614100000000001 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.026036+0000", 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021600499999999999 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.026069+0000", 2024-09-16T05:10:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2424e-05 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.026099+0000", 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0682999999999997e-05 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:65 2.e 2:75e143e9:::smithi20457400-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.177344+0000", 2024-09-16T05:10:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.1292082570000002, 2024-09-16T05:10:48.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028805800000000002, 2024-09-16T05:10:48.346 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.346 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.346 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.346 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 65 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.177344+0000", 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.177344+0000", 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.177346+0000", 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9690000000000001e-06 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.177946+0000", 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060003699999999999 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.177965+0000", 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9021e-05 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.177974+0000", 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4010000000000003e-06 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178041+0000", 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7318999999999997e-05 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178189+0000", 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014756799999999999 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178361+0000", 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017193899999999999 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.179844+0000", 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001482698 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.180110+0000", 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026602700000000001 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.180171+0000", 2024-09-16T05:10:48.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1197000000000004e-05 2024-09-16T05:10:48.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.180224+0000", 2024-09-16T05:10:48.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3403999999999999e-05 2024-09-16T05:10:48.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.352 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:66 2.e 2:75e143e9:::smithi20457400-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.182335+0000", 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.1242165689999997, 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018299259999999999, 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 66 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182335+0000", 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182335+0000", 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182337+0000", 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8369999999999999e-06 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182726+0000", 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00038921099999999998 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182745+0000", 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8046e-05 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182754+0000", 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0159999999999997e-06 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182805+0000", 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1771999999999998e-05 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182868+0000", 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.224e-05 2024-09-16T05:10:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.182968+0000", 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100064 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.183698+0000", 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073085999999999995 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.184092+0000", 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000393447 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.184121+0000", 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9142e-05 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.184165+0000", 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4291e-05 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:69 2.3 e112/107 2:c018a787:::smithi20457400-18:head v 112'1, mlcod=0'0)", 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.190759+0000", 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.1157930980000001, 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017512179999999999, 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.190759+0000", 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.190759+0000", 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.190759+0000", 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6400000000000001e-07 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.191595+0000", 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083558199999999997 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.191619+0000", 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4511e-05 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.191629+0000", 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0208e-05 2024-09-16T05:10:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.191691+0000", 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1270000000000001e-05 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.191713+0000", 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2657e-05 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.192475+0000", 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076178800000000001 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.192510+0000", 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5037999999999999e-05 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:109 2.e 2:75dc7feb:::smithi20457400-28:head [write 453789~780782,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.315124+0000", 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.9914281020000004, 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028520059999999998, 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 109 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.315124+0000", 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.315124+0000", 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.315127+0000", 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6910000000000002e-06 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.315921+0000", 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000794702 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.315946+0000", 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4606000000000001e-05 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.315957+0000", 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0812000000000001e-05 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316031+0000", 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4023999999999995e-05 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316129+0000", 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8661999999999998e-05 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316230+0000", 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010035300000000001 2024-09-16T05:10:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.317687+0000", 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014573730000000001 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.317911+0000", 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022435399999999999 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.317943+0000", 2024-09-16T05:10:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1968000000000003e-05 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.317976+0000", 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2461e-05 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:110 2.e 2:75dc7feb:::smithi20457400-28:head [write 1913602~475002] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.321098+0000", 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.9854539390000001, 2024-09-16T05:10:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019220489999999999, 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 110 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321098+0000", 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321098+0000", 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321099+0000", 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8400000000000002e-07 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321530+0000", 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043127999999999998 2024-09-16T05:10:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321542+0000", 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1953000000000001e-05 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321551+0000", 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3459999999999994e-06 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321610+0000", 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9741999999999998e-05 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321702+0000", 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1717000000000003e-05 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321838+0000", 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013552799999999999 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.322668+0000", 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000830096 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.322921+0000", 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025289300000000002 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.322974+0000", 2024-09-16T05:10:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3050999999999997e-05 2024-09-16T05:10:48.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.323020+0000", 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6458999999999997e-05 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:112 2.e 2:75dc7feb:::smithi20457400-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.321601+0000", 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.9849512840000001, 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017461849999999999, 2024-09-16T05:10:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 112 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321601+0000", 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321601+0000", 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321601+0000", 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321602+0000", 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4750000000000001e-06 2024-09-16T05:10:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321607+0000", 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6979999999999997e-06 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.321611+0000", 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3969999999999998e-06 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.322529+0000", 2024-09-16T05:10:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091839999999999999 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.322612+0000", 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3267000000000001e-05 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.323177+0000", 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056452 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.323253+0000", 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6087999999999994e-05 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.323347+0000", 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4048000000000005e-05 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:157 2.e 2:7b642727:::smithi20457400-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.471440+0000", 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.8351119889999996, 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028812740000000001, 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 157 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.471440+0000", 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.471440+0000", 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.471442+0000", 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.562e-06 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472057+0000", 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061575699999999998 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472075+0000", 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7272999999999999e-05 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472081+0000", 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4949999999999996e-06 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472148+0000", 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6462000000000002e-05 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472290+0000", 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014264900000000001 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472454+0000", 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016404399999999999 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.474105+0000", 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016508969999999999 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.474214+0000", 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010836900000000001 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.474275+0000", 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1449000000000001e-05 2024-09-16T05:10:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.474321+0000", 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6316999999999997e-05 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:158 2.e 2:7b642727:::smithi20457400-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.478678+0000", 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.8278737400000002, 2024-09-16T05:10:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002018078, 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 158 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.478678+0000", 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.478678+0000", 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.478680+0000", 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.967e-06 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479118+0000", 2024-09-16T05:10:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043741299999999999 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479129+0000", 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1841000000000001e-05 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479141+0000", 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1585e-05 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479195+0000", 2024-09-16T05:10:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3770000000000002e-05 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479291+0000", 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6618000000000002e-05 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479433+0000", 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014171100000000001 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.480391+0000", 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095734699999999999 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.480599+0000", 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020806199999999999 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.480657+0000", 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8349e-05 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.480696+0000", 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9415000000000001e-05 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:160 2.e 2:7b642727:::smithi20457400-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.479194+0000", 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.8273579089999998, 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001820293, 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 160 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479194+0000", 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479194+0000", 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479195+0000", 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7800000000000001e-07 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479197+0000", 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.661e-06 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479205+0000", 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4830000000000002e-06 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.479210+0000", 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.933e-06 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.480163+0000", 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095342599999999995 2024-09-16T05:10:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.480240+0000", 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6984999999999996e-05 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.480912+0000", 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00067199800000000004 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.480959+0000", 2024-09-16T05:10:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6591999999999997e-05 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.481014+0000", 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5537e-05 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:222 2.e 2:7b642727:::smithi20457400-40:head [write 490771~686315] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:46.001086+0000", 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.3054659709999998, 2024-09-16T05:10:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034724320000000001, 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 222 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.001086+0000", 2024-09-16T05:10:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.001086+0000", 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.001088+0000", 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9460000000000001e-06 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.001797+0000", 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070916700000000002 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.001822+0000", 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5137e-05 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.001832+0000", 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5740000000000002e-06 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.001922+0000", 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0227000000000002e-05 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002033+0000", 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111104 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002208+0000", 2024-09-16T05:10:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017458799999999999 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.004139+0000", 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001930925 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.004461+0000", 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00032251699999999998 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.004519+0000", 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7428e-05 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.004558+0000", 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9818999999999999e-05 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:223 2.e 2:7b642727:::smithi20457400-40:head [write 1885075~406346] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T05:10:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:46.006950+0000", 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.299602084, 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022826560000000001, 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 223 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.006950+0000", 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.006950+0000", 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.006951+0000", 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.393e-06 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007316+0000", 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036482799999999998 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007337+0000", 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0894999999999999e-05 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007348+0000", 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0733999999999999e-05 2024-09-16T05:10:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007421+0000", 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2873000000000005e-05 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007532+0000", 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011145399999999999 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007690+0000", 2024-09-16T05:10:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015778499999999999 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.008916+0000", 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001226614 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009161+0000", 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024404400000000001 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009200+0000", 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9097999999999997e-05 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009233+0000", 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2938000000000002e-05 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:224 2.e 2:7b642727:::smithi20457400-40:head [setxattr _header (56) in=7b,truncate 2890771] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:46.007374+0000", 2024-09-16T05:10:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.299178301, 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002022664, 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 224 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007374+0000", 2024-09-16T05:10:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007374+0000", 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007375+0000", 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.85e-07 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007380+0000", 2024-09-16T05:10:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5339999999999999e-06 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007389+0000", 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.268e-06 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007395+0000", 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5090000000000001e-06 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.008359+0000", 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096369299999999995 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.008459+0000", 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100251 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.008572+0000", 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000112704 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.008919+0000", 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000347583 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009281+0000", 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036145699999999998 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009337+0000", 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6297e-05 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009396+0000", 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9483e-05 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:225 2.e 2:7b642727:::smithi20457400-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e114)", 2024-09-16T05:10:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:46.007506+0000", 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.2990462150000002, 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020113800000000001, 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 225 2024-09-16T05:10:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007506+0000", 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007506+0000", 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007506+0000", 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2799999999999996e-07 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007510+0000", 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9600000000000002e-06 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007520+0000", 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3840000000000003e-06 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.007527+0000", 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2699999999999999e-06 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.008847+0000", 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001319651 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.008907+0000", 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0322000000000003e-05 2024-09-16T05:10:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009422+0000", 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00051553700000000005 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009464+0000", 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1149999999999997e-05 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.009517+0000", 2024-09-16T05:10:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3578000000000001e-05 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:249 2.e 2:7e4e65a4:::smithi20457400-4:head [write 608420~529411,stat] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:48.159727+0000", 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.14682529, 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022843949999999998, 2024-09-16T05:10:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 249 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.159727+0000", 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.159727+0000", 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.159731+0000", 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4870000000000004e-06 2024-09-16T05:10:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.160272+0000", 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054075399999999995 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.160284+0000", 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2371000000000001e-05 2024-09-16T05:10:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.160290+0000", 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7250000000000002e-06 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.160337+0000", 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6989e-05 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.160387+0000", 2024-09-16T05:10:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0451000000000002e-05 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.160475+0000", 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7835000000000002e-05 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.161712+0000", 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012365830000000001 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.161952+0000", 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000240125 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.161985+0000", 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3420000000000002e-05 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.162011+0000", 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5655000000000001e-05 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:250 2.e 2:7e4e65a4:::smithi20457400-4:head [write 1778096~561011] snapc 3=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:48.166031+0000", 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.14052113799999999, 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002257869, 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 250 2024-09-16T05:10:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166031+0000", 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166031+0000", 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166040+0000", 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4569999999999994e-06 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166552+0000", 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000511485 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166564+0000", 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2048000000000001e-05 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166569+0000", 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4380000000000003e-06 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166611+0000", 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1736999999999997e-05 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166662+0000", 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1449999999999997e-05 2024-09-16T05:10:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 3", 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166739+0000", 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6180999999999994e-05 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.167981+0000", 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012423149999999999 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.168225+0000", 2024-09-16T05:10:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000244288 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.168260+0000", 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4660999999999999e-05 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.168289+0000", 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8809000000000001e-05 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:48.436 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:48.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:48.437 INFO:tasks.rados.rados.2.smithi204.stdout:74: finishing write tid 4 to smithi20457400-23 2024-09-16T05:10:48.437 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 15 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-09-16T05:10:48.437 INFO:tasks.rados.rados.2.smithi204.stdout:74: left oid 23 (ObjNum 62 snap 2 seq_num 62) 2024-09-16T05:10:48.437 INFO:tasks.rados.rados.2.smithi204.stdout:76: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T05:10:48.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:47 smithi204 ceph-mon[30933]: pgmap v324: 27 pgs: 27 active+clean; 76 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 5.7 MiB/s wr, 9 op/s 2024-09-16T05:10:48.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:47 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:10:48.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:47 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:10:48.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:47 smithi204 ceph-mon[30933]: osdmap e115: 12 total, 12 up, 12 in 2024-09-16T05:10:48.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3705317788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:10:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:48.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:48.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:47 smithi196 ceph-mon[30802]: pgmap v324: 27 pgs: 27 active+clean; 76 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 5.7 MiB/s wr, 9 op/s 2024-09-16T05:10:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:47 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:10:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:47 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:10:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:47 smithi196 ceph-mon[30802]: osdmap e115: 12 total, 12 up, 12 in 2024-09-16T05:10:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3705317788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:10:48.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:48.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:48.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:48.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:10:48.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:48.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:48.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:48.832 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:48.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:48.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:48.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:48.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:49.008 INFO:tasks.rados.rados.2.smithi204.stdout:69: done (5 left) 2024-09-16T05:10:49.008 INFO:tasks.rados.rados.2.smithi204.stdout:73: done (4 left) 2024-09-16T05:10:49.009 INFO:tasks.rados.rados.2.smithi204.stdout:74: done (3 left) 2024-09-16T05:10:49.009 INFO:tasks.rados.rados.2.smithi204.stdout:75: done (2 left) 2024-09-16T05:10:49.009 INFO:tasks.rados.rados.2.smithi204.stdout:76: done (1 left) 2024-09-16T05:10:49.009 INFO:tasks.rados.rados.2.smithi204.stdout:77: done (0 left) 2024-09-16T05:10:49.009 INFO:tasks.rados.rados.2.smithi204.stdout:78: read oid 11 snap -1 2024-09-16T05:10:49.009 INFO:tasks.rados.rados.2.smithi204.stdout:78: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T05:10:49.010 INFO:tasks.rados.rados.2.smithi204.stdout:79: write oid 5 current snap is 3 2024-09-16T05:10:49.010 INFO:tasks.rados.rados.2.smithi204.stdout:79: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-16T05:10:49.015 INFO:tasks.rados.rados.2.smithi204.stdout:79: writing smithi20457400-5 from 658841 to 1130352 tid 1 2024-09-16T05:10:49.022 INFO:tasks.rados.rados.2.smithi204.stdout:79: writing smithi20457400-5 from 1553542 to 2261880 tid 2 2024-09-16T05:10:49.025 INFO:tasks.rados.rados.2.smithi204.stdout:79: writing smithi20457400-5 from 2964119 to 3058841 tid 3 2024-09-16T05:10:49.026 INFO:tasks.rados.rados.2.smithi204.stdout:80: copy_from oid 12 from oid 32 current snap is 3 2024-09-16T05:10:49.026 INFO:tasks.rados.rados.2.smithi204.stdout:78: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T05:10:49.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:49.082 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:49.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:49.095 INFO:tasks.rados.rados.2.smithi204.stdout:79: finishing write tid 1 to smithi20457400-5 2024-09-16T05:10:49.095 INFO:tasks.rados.rados.2.smithi204.stdout:79: finishing write tid 2 to smithi20457400-5 2024-09-16T05:10:49.095 INFO:tasks.rados.rados.2.smithi204.stdout:78: done (2 left) 2024-09-16T05:10:49.095 INFO:tasks.rados.rados.2.smithi204.stdout:81: delete oid 25 current snap is 3 2024-09-16T05:10:49.098 INFO:tasks.rados.rados.2.smithi204.stdout:81: done (2 left) 2024-09-16T05:10:49.098 INFO:tasks.rados.rados.2.smithi204.stdout:82: read oid 14 snap -1 2024-09-16T05:10:49.098 INFO:tasks.rados.rados.2.smithi204.stdout:82: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T05:10:49.098 INFO:tasks.rados.rados.2.smithi204.stdout:83: read oid 44 snap 2 2024-09-16T05:10:49.098 INFO:tasks.rados.rados.2.smithi204.stdout:83: expect deleted 2024-09-16T05:10:49.098 INFO:tasks.rados.rados.2.smithi204.stdout:84: delete oid 49 current snap is 3 2024-09-16T05:10:49.098 INFO:tasks.rados.rados.2.smithi204.stdout:79: finishing write tid 3 to smithi20457400-5 2024-09-16T05:10:49.100 INFO:tasks.rados.rados.2.smithi204.stdout:82: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T05:10:49.175 INFO:tasks.rados.rados.2.smithi204.stdout:79: finishing write tid 4 to smithi20457400-5 2024-09-16T05:10:49.175 INFO:tasks.rados.rados.2.smithi204.stdout:79: finishing write tid 5 to smithi20457400-5 2024-09-16T05:10:49.175 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 20 (ObjNum 63 snap 3 seq_num 63) dirty exists 2024-09-16T05:10:49.175 INFO:tasks.rados.rados.2.smithi204.stdout:79: left oid 5 (ObjNum 63 snap 3 seq_num 63) 2024-09-16T05:10:49.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:49.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:49.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:49.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:49.334 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:49.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:49.340 INFO:tasks.rados.rados.2.smithi204.stdout:79: done (4 left) 2024-09-16T05:10:49.340 INFO:tasks.rados.rados.2.smithi204.stdout:82: done (3 left) 2024-09-16T05:10:49.340 INFO:tasks.rados.rados.2.smithi204.stdout:83: done (2 left) 2024-09-16T05:10:49.341 INFO:tasks.rados.rados.2.smithi204.stdout:84: done (1 left) 2024-09-16T05:10:49.341 INFO:tasks.rados.rados.2.smithi204.stdout:85: read oid 35 snap -1 2024-09-16T05:10:49.341 INFO:tasks.rados.rados.2.smithi204.stdout:85: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T05:10:49.341 INFO:tasks.rados.rados.2.smithi204.stdout:86: copy_from oid 31 from oid 50 current snap is 3 2024-09-16T05:10:49.341 INFO:tasks.rados.rados.2.smithi204.stdout:87: copy_from oid 3 from oid 8 current snap is 3 2024-09-16T05:10:49.341 INFO:tasks.rados.rados.2.smithi204.stdout:88: snap_remove snap 2 2024-09-16T05:10:49.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:49 smithi032 ceph-mon[27388]: osdmap e116: 12 total, 12 up, 12 in 2024-09-16T05:10:49.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:49 smithi032 ceph-mon[27388]: pgmap v327: 27 pgs: 27 active+clean; 117 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 6 op/s 2024-09-16T05:10:49.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:49 smithi204 ceph-mon[30933]: osdmap e116: 12 total, 12 up, 12 in 2024-09-16T05:10:49.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:49 smithi204 ceph-mon[30933]: pgmap v327: 27 pgs: 27 active+clean; 117 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 6 op/s 2024-09-16T05:10:49.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:49.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:49.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:49.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:49 smithi196 ceph-mon[30802]: osdmap e116: 12 total, 12 up, 12 in 2024-09-16T05:10:49.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:49 smithi196 ceph-mon[30802]: pgmap v327: 27 pgs: 27 active+clean; 117 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 6 op/s 2024-09-16T05:10:49.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:10:49.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:49.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:49.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:49.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:49.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:49.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:49.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:49.724 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:10:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:10:49.733 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:49.747 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:10:49.747 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:10:49.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:10:49.775 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:10:49.795 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:10:49.795 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:10:49.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:49.817 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:10:49.817 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:10:49.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:49.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:49.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:10:49.839 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:10:49.862 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:10:49.862 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:10:49.884 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:10:49.885 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:10:49.906 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:10:49.906 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:10:49.929 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:10:49.929 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:10:49.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:49.949 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:10:49.949 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:10:49.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:49.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:49.971 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:10:49.971 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:10:50.015 INFO:tasks.rados.rados.2.smithi204.stdout:88: done (4 left) 2024-09-16T05:10:50.016 INFO:tasks.rados.rados.2.smithi204.stdout:89: read oid 47 snap -1 2024-09-16T05:10:50.016 INFO:tasks.rados.rados.2.smithi204.stdout:89: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-16T05:10:50.016 INFO:tasks.rados.rados.2.smithi204.stdout:90: snap_create 2024-09-16T05:10:50.016 INFO:tasks.rados.rados.2.smithi204.stdout:85: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T05:10:50.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:50.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:87: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:86: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:87: finishing copy_from to smithi20457400-3 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:86: finishing copy_from to smithi20457400-31 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 13 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:80: finishing copy_from to smithi20457400-12 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:80: got expected ENOENT (src dne) 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:80: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:10:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:89: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-16T05:10:50.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:50.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:50.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:50.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:50.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:50 smithi032 ceph-mon[27388]: osdmap e117: 12 total, 12 up, 12 in 2024-09-16T05:10:50.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3281737123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:50.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:50.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:50 smithi204 ceph-mon[30933]: osdmap e117: 12 total, 12 up, 12 in 2024-09-16T05:10:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3281737123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:50.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:50.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:50.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:50.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:50 smithi196 ceph-mon[30802]: osdmap e117: 12 total, 12 up, 12 in 2024-09-16T05:10:50.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3281737123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:50.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:50.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:50.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:50.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:50.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:50.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:50.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:50.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:50.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:50.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:50.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:51.017 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:10:51.021 INFO:tasks.rados.rados.2.smithi204.stdout:80: done (5 left) 2024-09-16T05:10:51.021 INFO:tasks.rados.rados.2.smithi204.stdout:85: done (4 left) 2024-09-16T05:10:51.021 INFO:tasks.rados.rados.2.smithi204.stdout:86: done (3 left) 2024-09-16T05:10:51.021 INFO:tasks.rados.rados.2.smithi204.stdout:87: done (2 left) 2024-09-16T05:10:51.021 INFO:tasks.rados.rados.2.smithi204.stdout:89: done (1 left) 2024-09-16T05:10:51.021 INFO:tasks.rados.rados.2.smithi204.stdout:90: done (0 left) 2024-09-16T05:10:51.022 INFO:tasks.rados.rados.2.smithi204.stdout:91: copy_from oid 37 from oid 11 current snap is 4 2024-09-16T05:10:51.022 INFO:tasks.rados.rados.2.smithi204.stdout:92: write (excl) oid 25 current snap is 4 2024-09-16T05:10:51.022 INFO:tasks.rados.rados.2.smithi204.stdout:92: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-16T05:10:51.029 INFO:tasks.rados.rados.2.smithi204.stdout:92: writing smithi20457400-25 from 709262 to 1406520 tid 1 2024-09-16T05:10:51.035 INFO:tasks.rados.rados.2.smithi204.stdout:92: writing smithi20457400-25 from 2012635 to 2584653 tid 2 2024-09-16T05:10:51.039 INFO:tasks.rados.rados.2.smithi204.stdout:92: writing smithi20457400-25 from 3046388 to 3109262 tid 3 2024-09-16T05:10:51.039 INFO:tasks.rados.rados.2.smithi204.stdout:93: read oid 9 snap 3 2024-09-16T05:10:51.039 INFO:tasks.rados.rados.2.smithi204.stdout:93: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T05:10:51.039 INFO:tasks.rados.rados.2.smithi204.stdout:94: write (excl) oid 7 current snap is 4 2024-09-16T05:10:51.039 INFO:tasks.rados.rados.2.smithi204.stdout:91: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:10:51.039 INFO:tasks.rados.rados.2.smithi204.stdout:91: finishing copy_from to smithi20457400-37 2024-09-16T05:10:51.039 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T05:10:51.040 INFO:tasks.rados.rados.2.smithi204.stdout:92: finishing write tid 1 to smithi20457400-25 2024-09-16T05:10:51.040 INFO:tasks.rados.rados.2.smithi204.stdout:94: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-16T05:10:51.047 INFO:tasks.rados.rados.2.smithi204.stdout:94: writing smithi20457400-7 from 726069 to 1498576 tid 1 2024-09-16T05:10:51.053 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:51 smithi032 ceph-mon[27388]: osdmap e118: 12 total, 12 up, 12 in 2024-09-16T05:10:51.053 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:51 smithi032 ceph-mon[27388]: pgmap v330: 27 pgs: 27 active+clean; 116 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 18 op/s 2024-09-16T05:10:51.053 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2033758738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:10:51.054 INFO:tasks.rados.rados.2.smithi204.stdout:94: writing smithi20457400-7 from 2071117 to 2786793 tid 2 2024-09-16T05:10:51.054 INFO:tasks.rados.rados.2.smithi204.stdout:91: done (3 left) 2024-09-16T05:10:51.054 INFO:tasks.rados.rados.2.smithi204.stdout:95: read oid 15 snap 3 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:95: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:96: read oid 27 snap -1 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:96: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:97: copy_from oid 2 from oid 23 current snap is 4 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:92: finishing write tid 2 to smithi20457400-25 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:92: finishing write tid 3 to smithi20457400-25 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:92: finishing write tid 4 to smithi20457400-25 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:92: finishing write tid 5 to smithi20457400-25 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 37 (ObjNum 66 snap 4 seq_num 66) dirty exists 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:92: left oid 25 (ObjNum 66 snap 4 seq_num 66) 2024-09-16T05:10:51.055 INFO:tasks.rados.rados.2.smithi204.stdout:92: done (5 left) 2024-09-16T05:10:51.056 INFO:tasks.rados.rados.2.smithi204.stdout:98: write oid 6 current snap is 4 2024-09-16T05:10:51.056 INFO:tasks.rados.rados.2.smithi204.stdout:93: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T05:10:51.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:51.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:51.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:51.124 INFO:tasks.rados.rados.2.smithi204.stdout:94: finishing write tid 1 to smithi20457400-7 2024-09-16T05:10:51.124 INFO:tasks.rados.rados.2.smithi204.stdout:94: finishing write tid 2 to smithi20457400-7 2024-09-16T05:10:51.125 INFO:tasks.rados.rados.2.smithi204.stdout:94: finishing write tid 3 to smithi20457400-7 2024-09-16T05:10:51.125 INFO:tasks.rados.rados.2.smithi204.stdout:94: finishing write tid 4 to smithi20457400-7 2024-09-16T05:10:51.125 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 19 (ObjNum 67 snap 4 seq_num 67) dirty exists 2024-09-16T05:10:51.125 INFO:tasks.rados.rados.2.smithi204.stdout:94: left oid 7 (ObjNum 67 snap 4 seq_num 67) 2024-09-16T05:10:51.125 INFO:tasks.rados.rados.2.smithi204.stdout:98: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-16T05:10:51.130 INFO:tasks.rados.rados.2.smithi204.stdout:98: writing smithi20457400-6 from 742876 to 1190632 tid 1 2024-09-16T05:10:51.137 INFO:tasks.rados.rados.2.smithi204.stdout:98: writing smithi20457400-6 from 1729599 to 2305286 tid 2 2024-09-16T05:10:51.141 INFO:tasks.rados.rados.2.smithi204.stdout:98: writing smithi20457400-6 from 3101234 to 3142876 tid 3 2024-09-16T05:10:51.142 INFO:tasks.rados.rados.2.smithi204.stdout:93: done (5 left) 2024-09-16T05:10:51.142 INFO:tasks.rados.rados.2.smithi204.stdout:94: done (4 left) 2024-09-16T05:10:51.142 INFO:tasks.rados.rados.2.smithi204.stdout:99: copy_from oid 25 from oid 44 current snap is 4 2024-09-16T05:10:51.142 INFO:tasks.rados.rados.2.smithi204.stdout:95: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T05:10:51.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:51.215 INFO:tasks.rados.rados.2.smithi204.stdout:96: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T05:10:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:51.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:51.292 INFO:tasks.rados.rados.2.smithi204.stdout:97: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:10:51.292 INFO:tasks.rados.rados.2.smithi204.stdout:97: finishing copy_from to smithi20457400-2 2024-09-16T05:10:51.292 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 15 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-09-16T05:10:51.292 INFO:tasks.rados.rados.2.smithi204.stdout:98: finishing write tid 1 to smithi20457400-6 2024-09-16T05:10:51.292 INFO:tasks.rados.rados.2.smithi204.stdout:98: finishing write tid 2 to smithi20457400-6 2024-09-16T05:10:51.292 INFO:tasks.rados.rados.2.smithi204.stdout:98: finishing write tid 3 to smithi20457400-6 2024-09-16T05:10:51.292 INFO:tasks.rados.rados.2.smithi204.stdout:98: finishing write tid 4 to smithi20457400-6 2024-09-16T05:10:51.292 INFO:tasks.rados.rados.2.smithi204.stdout:98: finishing write tid 5 to smithi20457400-6 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 8 (ObjNum 68 snap 4 seq_num 68) dirty exists 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:98: left oid 6 (ObjNum 68 snap 4 seq_num 68) 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:99: finishing copy_from to smithi20457400-25 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:99: got expected ENOENT (src dne) 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:99: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:95: done (4 left) 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:96: done (3 left) 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:97: done (2 left) 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:98: done (1 left) 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:99: done (0 left) 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:100: write (excl) oid 1 current snap is 4 2024-09-16T05:10:51.293 INFO:tasks.rados.rados.2.smithi204.stdout:100: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-16T05:10:51.298 INFO:tasks.rados.rados.2.smithi204.stdout:100: writing smithi20457400-1 from 759683 to 1399041 tid 1 2024-09-16T05:10:51.305 INFO:tasks.rados.rados.2.smithi204.stdout:100: writing smithi20457400-1 from 1904434 to 2623779 tid 2 2024-09-16T05:10:51.310 INFO:tasks.rados.rados.2.smithi204.stdout:100: writing smithi20457400-1 from 3128657 to 3159683 tid 3 2024-09-16T05:10:51.310 INFO:tasks.rados.rados.2.smithi204.stdout:101: delete oid 25 current snap is 4 2024-09-16T05:10:51.310 INFO:tasks.rados.rados.2.smithi204.stdout:100: finishing write tid 1 to smithi20457400-1 2024-09-16T05:10:51.310 INFO:tasks.rados.rados.2.smithi204.stdout:100: finishing write tid 2 to smithi20457400-1 2024-09-16T05:10:51.310 INFO:tasks.rados.rados.2.smithi204.stdout:101: done (1 left) 2024-09-16T05:10:51.310 INFO:tasks.rados.rados.2.smithi204.stdout:102: read oid 4 snap -1 2024-09-16T05:10:51.311 INFO:tasks.rados.rados.2.smithi204.stdout:102: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:10:51.311 INFO:tasks.rados.rados.2.smithi204.stdout:103: rollback oid 44 current snap is 4 2024-09-16T05:10:51.311 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 3 2024-09-16T05:10:51.311 INFO:tasks.rados.rados.2.smithi204.stdout:104: write (excl) oid 23 current snap is 4 2024-09-16T05:10:51.311 INFO:tasks.rados.rados.2.smithi204.stdout:104: seq_num 71 ranges {793297=789856,2305045=723014} 2024-09-16T05:10:51.318 INFO:tasks.rados.rados.2.smithi204.stdout:104: writing smithi20457400-23 from 793297 to 1583153 tid 1 2024-09-16T05:10:51.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:51.326 INFO:tasks.rados.rados.2.smithi204.stdout:104: writing smithi20457400-23 from 2305045 to 3028059 tid 2 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:105: read oid 2 snap -1 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:105: expect (ObjNum 62 snap 2 seq_num 62) 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:106: read oid 40 snap -1 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:106: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:107: snap_create 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:100: finishing write tid 3 to smithi20457400-1 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:100: finishing write tid 4 to smithi20457400-1 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:103: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:100: finishing write tid 5 to smithi20457400-1 2024-09-16T05:10:51.327 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 11 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-09-16T05:10:51.328 INFO:tasks.rados.rados.2.smithi204.stdout:100: left oid 1 (ObjNum 69 snap 4 seq_num 69) 2024-09-16T05:10:51.328 INFO:tasks.rados.rados.2.smithi204.stdout:102: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:10:51.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:51.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:51.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:51 smithi204 ceph-mon[30933]: osdmap e118: 12 total, 12 up, 12 in 2024-09-16T05:10:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:51 smithi204 ceph-mon[30933]: pgmap v330: 27 pgs: 27 active+clean; 116 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 18 op/s 2024-09-16T05:10:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2033758738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:10:51.446 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:10:51.446 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:10:51.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:51.460 INFO:tasks.rados.rados.2.smithi204.stdout:104: finishing write tid 1 to smithi20457400-23 2024-09-16T05:10:51.460 INFO:tasks.rados.rados.2.smithi204.stdout:104: finishing write tid 2 to smithi20457400-23 2024-09-16T05:10:51.460 INFO:tasks.rados.rados.2.smithi204.stdout:104: finishing write tid 3 to smithi20457400-23 2024-09-16T05:10:51.460 INFO:tasks.rados.rados.2.smithi204.stdout:104: finishing write tid 4 to smithi20457400-23 2024-09-16T05:10:51.460 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 30 (ObjNum 71 snap 4 seq_num 71) dirty exists 2024-09-16T05:10:51.460 INFO:tasks.rados.rados.2.smithi204.stdout:104: left oid 23 (ObjNum 71 snap 4 seq_num 71) 2024-09-16T05:10:51.460 INFO:tasks.rados.rados.2.smithi204.stdout:105: expect (ObjNum 62 snap 2 seq_num 62) 2024-09-16T05:10:51.469 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:51.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:51.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:51 smithi196 ceph-mon[30802]: osdmap e118: 12 total, 12 up, 12 in 2024-09-16T05:10:51.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:51 smithi196 ceph-mon[30802]: pgmap v330: 27 pgs: 27 active+clean; 116 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.0 MiB/s wr, 18 op/s 2024-09-16T05:10:51.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2033758738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:10:51.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:51.596 INFO:tasks.rados.rados.2.smithi204.stdout:106: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-16T05:10:51.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:51.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:51.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:51.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:51.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:51.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:51.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:51.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:51.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:51.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:51.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:51.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:52.029 INFO:tasks.rados.rados.2.smithi204.stdout:100: done (6 left) 2024-09-16T05:10:52.029 INFO:tasks.rados.rados.2.smithi204.stdout:102: done (5 left) 2024-09-16T05:10:52.030 INFO:tasks.rados.rados.2.smithi204.stdout:103: done (4 left) 2024-09-16T05:10:52.030 INFO:tasks.rados.rados.2.smithi204.stdout:104: done (3 left) 2024-09-16T05:10:52.030 INFO:tasks.rados.rados.2.smithi204.stdout:105: done (2 left) 2024-09-16T05:10:52.030 INFO:tasks.rados.rados.2.smithi204.stdout:106: done (1 left) 2024-09-16T05:10:52.030 INFO:tasks.rados.rados.2.smithi204.stdout:107: done (0 left) 2024-09-16T05:10:52.030 INFO:tasks.rados.rados.2.smithi204.stdout:108: delete oid 8 current snap is 5 2024-09-16T05:10:52.032 INFO:tasks.rados.rados.2.smithi204.stdout:108: done (0 left) 2024-09-16T05:10:52.032 INFO:tasks.rados.rados.2.smithi204.stdout:109: read oid 30 snap -1 2024-09-16T05:10:52.032 INFO:tasks.rados.rados.2.smithi204.stdout:109: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T05:10:52.032 INFO:tasks.rados.rados.2.smithi204.stdout:110: read oid 21 snap -1 2024-09-16T05:10:52.032 INFO:tasks.rados.rados.2.smithi204.stdout:110: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T05:10:52.032 INFO:tasks.rados.rados.2.smithi204.stdout:111: snap_remove snap 3 2024-09-16T05:10:52.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:52.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:52.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:52 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2033758738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:10:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:52 smithi032 ceph-mon[27388]: osdmap e119: 12 total, 12 up, 12 in 2024-09-16T05:10:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:52 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:52 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:52.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:10:52.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:10:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:52.222 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:52.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:10:52.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:52.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:52.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:52.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:52.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:52 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:52.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2033758738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:10:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:52 smithi204 ceph-mon[30933]: osdmap e119: 12 total, 12 up, 12 in 2024-09-16T05:10:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:52 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:52 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:52.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:52.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:52.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:52 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:10:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2033758738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:10:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:52 smithi196 ceph-mon[30802]: osdmap e119: 12 total, 12 up, 12 in 2024-09-16T05:10:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:52 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:52 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:52.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:52.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:52.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:52.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:52.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:52.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:52.790 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:52.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:52.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:10:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:52.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:52.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:52.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:52.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:53.038 INFO:tasks.rados.rados.2.smithi204.stdout:111: done (2 left) 2024-09-16T05:10:53.039 INFO:tasks.rados.rados.2.smithi204.stdout:112: delete oid 17 current snap is 5 2024-09-16T05:10:53.039 INFO:tasks.rados.rados.2.smithi204.stdout:109: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T05:10:53.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:53.093 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:10:53.094 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:10:53.094 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T05:10:53.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:10:53.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:53.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:53.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:53.122 INFO:tasks.rados.rados.2.smithi204.stdout:110: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T05:10:53.200 INFO:tasks.rados.rados.2.smithi204.stdout:109: done (2 left) 2024-09-16T05:10:53.201 INFO:tasks.rados.rados.2.smithi204.stdout:110: done (1 left) 2024-09-16T05:10:53.201 INFO:tasks.rados.rados.2.smithi204.stdout:112: done (0 left) 2024-09-16T05:10:53.201 INFO:tasks.rados.rados.2.smithi204.stdout:113: write (excl) oid 10 current snap is 5 2024-09-16T05:10:53.201 INFO:tasks.rados.rados.2.smithi204.stdout:113: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-09-16T05:10:53.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:53.206 INFO:tasks.rados.rados.2.smithi204.stdout:113: writing smithi20457400-10 from 443718 to 1059321 tid 1 2024-09-16T05:10:53.211 INFO:tasks.rados.rados.2.smithi204.stdout:113: writing smithi20457400-10 from 1564138 to 2034479 tid 2 2024-09-16T05:10:53.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:53 smithi032 ceph-mon[27388]: osdmap e120: 12 total, 12 up, 12 in 2024-09-16T05:10:53.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:53 smithi032 ceph-mon[27388]: pgmap v333: 27 pgs: 27 active+clean; 122 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2024-09-16T05:10:53.217 INFO:tasks.rados.rados.2.smithi204.stdout:113: writing smithi20457400-10 from 2633066 to 3243718 tid 3 2024-09-16T05:10:53.218 INFO:tasks.rados.rados.2.smithi204.stdout:114: write (excl) oid 26 current snap is 5 2024-09-16T05:10:53.218 INFO:tasks.rados.rados.2.smithi204.stdout:113: finishing write tid 1 to smithi20457400-10 2024-09-16T05:10:53.218 INFO:tasks.rados.rados.2.smithi204.stdout:113: finishing write tid 2 to smithi20457400-10 2024-09-16T05:10:53.218 INFO:tasks.rados.rados.2.smithi204.stdout:114: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-16T05:10:53.224 INFO:tasks.rados.rados.2.smithi204.stdout:114: writing smithi20457400-26 from 460525 to 1151377 tid 1 2024-09-16T05:10:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:53.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:53.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:53.230 INFO:tasks.rados.rados.2.smithi204.stdout:114: writing smithi20457400-26 from 1622620 to 2352972 tid 2 2024-09-16T05:10:53.236 INFO:tasks.rados.rados.2.smithi204.stdout:114: writing smithi20457400-26 from 3060489 to 3260525 tid 3 2024-09-16T05:10:53.236 INFO:tasks.rados.rados.2.smithi204.stdout:115: snap_create 2024-09-16T05:10:53.236 INFO:tasks.rados.rados.2.smithi204.stdout:113: finishing write tid 3 to smithi20457400-10 2024-09-16T05:10:53.236 INFO:tasks.rados.rados.2.smithi204.stdout:113: finishing write tid 4 to smithi20457400-10 2024-09-16T05:10:53.236 INFO:tasks.rados.rados.2.smithi204.stdout:113: finishing write tid 5 to smithi20457400-10 2024-09-16T05:10:53.236 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 21 (ObjNum 74 snap 5 seq_num 74) dirty exists 2024-09-16T05:10:53.236 INFO:tasks.rados.rados.2.smithi204.stdout:113: left oid 10 (ObjNum 74 snap 5 seq_num 74) 2024-09-16T05:10:53.236 INFO:tasks.rados.rados.2.smithi204.stdout:114: finishing write tid 1 to smithi20457400-26 2024-09-16T05:10:53.237 INFO:tasks.rados.rados.2.smithi204.stdout:114: finishing write tid 2 to smithi20457400-26 2024-09-16T05:10:53.238 INFO:tasks.rados.rados.2.smithi204.stdout:114: finishing write tid 3 to smithi20457400-26 2024-09-16T05:10:53.238 INFO:tasks.rados.rados.2.smithi204.stdout:114: finishing write tid 4 to smithi20457400-26 2024-09-16T05:10:53.238 INFO:tasks.rados.rados.2.smithi204.stdout:114: finishing write tid 5 to smithi20457400-26 2024-09-16T05:10:53.238 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 43 (ObjNum 75 snap 5 seq_num 75) dirty exists 2024-09-16T05:10:53.238 INFO:tasks.rados.rados.2.smithi204.stdout:114: left oid 26 (ObjNum 75 snap 5 seq_num 75) 2024-09-16T05:10:53.320 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:10:53.320 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:10:53.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:53.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:53.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:53 smithi204 ceph-mon[30933]: osdmap e120: 12 total, 12 up, 12 in 2024-09-16T05:10:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:53 smithi204 ceph-mon[30933]: pgmap v333: 27 pgs: 27 active+clean; 122 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2024-09-16T05:10:53.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:53.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:53.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:53.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:53 smithi196 ceph-mon[30802]: osdmap e120: 12 total, 12 up, 12 in 2024-09-16T05:10:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:53 smithi196 ceph-mon[30802]: pgmap v333: 27 pgs: 27 active+clean; 122 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2024-09-16T05:10:53.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:53.604 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:53.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:53.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:53.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:53.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:53.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:53.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:53.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:53.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:10:53.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:53.972 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:53.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:53.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:54.004 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:54.035 INFO:tasks.rados.rados.2.smithi204.stdout:113: done (2 left) 2024-09-16T05:10:54.036 INFO:tasks.rados.rados.2.smithi204.stdout:114: done (1 left) 2024-09-16T05:10:54.036 INFO:tasks.rados.rados.2.smithi204.stdout:115: done (0 left) 2024-09-16T05:10:54.036 INFO:tasks.rados.rados.2.smithi204.stdout:116: read oid 44 snap 5 2024-09-16T05:10:54.036 INFO:tasks.rados.rados.2.smithi204.stdout:116: expect deleted 2024-09-16T05:10:54.036 INFO:tasks.rados.rados.2.smithi204.stdout:117: read oid 12 snap 5 2024-09-16T05:10:54.036 INFO:tasks.rados.rados.2.smithi204.stdout:117: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T05:10:54.036 INFO:tasks.rados.rados.2.smithi204.stdout:118: write (excl) oid 14 current snap is 6 2024-09-16T05:10:54.036 INFO:tasks.rados.rados.2.smithi204.stdout:118: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-16T05:10:54.042 INFO:tasks.rados.rados.2.smithi204.stdout:118: writing smithi20457400-14 from 477332 to 1243433 tid 1 2024-09-16T05:10:54.047 INFO:tasks.rados.rados.2.smithi204.stdout:118: writing smithi20457400-14 from 1681102 to 2155112 tid 2 2024-09-16T05:10:54.054 INFO:tasks.rados.rados.2.smithi204.stdout:118: writing smithi20457400-14 from 2687912 to 3277332 tid 3 2024-09-16T05:10:54.054 INFO:tasks.rados.rados.2.smithi204.stdout:119: snap_remove snap 4 2024-09-16T05:10:54.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:54.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:54.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:54 smithi032 ceph-mon[27388]: osdmap e121: 12 total, 12 up, 12 in 2024-09-16T05:10:54.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:54.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:54.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:54.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:54.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:54.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:54 smithi204 ceph-mon[30933]: osdmap e121: 12 total, 12 up, 12 in 2024-09-16T05:10:54.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:54.483 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:54.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:54.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:54 smithi196 ceph-mon[30802]: osdmap e121: 12 total, 12 up, 12 in 2024-09-16T05:10:54.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:54.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:54.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:54.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:54.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:10:54.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:54.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:54.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:54.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:54.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:54.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:10:54.814+0000 7f62e9558640 0 --2- 172.21.15.32:0/2967904541 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62e41082a0 0x7f62e410ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:10:54.871 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:10:54.871 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:10:54.904 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:10:54.905 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:10:54.907 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:54.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 122, 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:10:54.027715+0000", 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:09:26.475677+0000", 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:09:05.828775+0000", 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5801984, 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "noscrub", 2024-09-16T05:10:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 27, 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:10:54.918 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:10:54.919 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 11, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 11, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 11, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 11, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 11, 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:10:54.920 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "106", 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "104", 2024-09-16T05:10:54.921 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:10:54.922 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:10:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:10:54.924 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 3.2899999618530273, 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 3.2899999618530273, 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.82999998331069946, 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.7300000190734863, 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.7300000190734863, 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 1, 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:10:54.926 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 16, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 16, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 16, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 16, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 16, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:10:54.927 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "122", 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 7, 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 122, 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:10:54.928 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:10:54.929 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:10:54.930 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.25, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.25, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.25, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.25, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-16T05:10:54.931 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 1, 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 1 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.932 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 61, 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.935 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:54.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:10:54.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.938 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:10:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 21, 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3508564664", 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3508564664", 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3508564664", 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3508564664", 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.949 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-09-16T05:10:54.950 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:54.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:54.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:54.953 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:10:54.954 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.955 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:10:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:10:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:10:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.972 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.972 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:10:54.972 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:10:54.973 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 108, 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.975 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.976 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.980 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.983 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.985 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:10:54.986 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:10:54.987 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:10:54.988 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:54.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.990 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:10:54.991 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:54.992 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:54.993 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:54.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:10:54.995 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.996 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:10:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:10:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.002 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:51.488968+0000", 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:02.790003+0000", 2024-09-16T05:10:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:29.046992+0000", 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:10:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:04.735259+0000", 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:00.052444+0000", 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:39.367915+0000", 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:49.025968+0000", 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:02.089795+0000", 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:35.890683+0000", 2024-09-16T05:10:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:59.269066+0000", 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:10:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:59.495208+0000", 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:10:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:38.259572+0000", 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:55.011 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:10:55.012 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 2, 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 4 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:10:55.013 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:55.014 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:55.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:10:55.015 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:10:55.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:10:55.015 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:10:55.016 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:10:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:10:55.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:55.030 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:10:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:10:55.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:55.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:55.045 INFO:tasks.rados.rados.2.smithi204.stdout:119: done (3 left) 2024-09-16T05:10:55.045 INFO:tasks.rados.rados.2.smithi204.stdout:120: read oid 41 snap 5 2024-09-16T05:10:55.045 INFO:tasks.rados.rados.2.smithi204.stdout:120: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:10:55.045 INFO:tasks.rados.rados.2.smithi204.stdout:121: rollback oid 8 current snap is 6 2024-09-16T05:10:55.045 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 8 to 5 2024-09-16T05:10:55.045 INFO:tasks.rados.rados.2.smithi204.stdout:117: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T05:10:55.056 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:10:55.056 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:10:55.077 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:10:55.078 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:10:55.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:55 smithi032 ceph-mon[27388]: osdmap e122: 12 total, 12 up, 12 in 2024-09-16T05:10:55.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:55 smithi032 ceph-mon[27388]: pgmap v336: 27 pgs: 27 active+clean; 114 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 17 op/s 2024-09-16T05:10:55.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1411261724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:55.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1240391636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:10:55.099 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:10:55.099 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:10:55.113 INFO:tasks.rados.rados.2.smithi204.stdout:118: finishing write tid 1 to smithi20457400-14 2024-09-16T05:10:55.113 INFO:tasks.rados.rados.2.smithi204.stdout:118: finishing write tid 2 to smithi20457400-14 2024-09-16T05:10:55.113 INFO:tasks.rados.rados.2.smithi204.stdout:118: finishing write tid 3 to smithi20457400-14 2024-09-16T05:10:55.113 INFO:tasks.rados.rados.2.smithi204.stdout:118: finishing write tid 4 to smithi20457400-14 2024-09-16T05:10:55.114 INFO:tasks.rados.rados.2.smithi204.stdout:118: finishing write tid 5 to smithi20457400-14 2024-09-16T05:10:55.114 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 48 (ObjNum 76 snap 6 seq_num 76) dirty exists 2024-09-16T05:10:55.114 INFO:tasks.rados.rados.2.smithi204.stdout:118: left oid 14 (ObjNum 76 snap 6 seq_num 76) 2024-09-16T05:10:55.114 INFO:tasks.rados.rados.2.smithi204.stdout:116: done (4 left) 2024-09-16T05:10:55.114 INFO:tasks.rados.rados.2.smithi204.stdout:117: done (3 left) 2024-09-16T05:10:55.114 INFO:tasks.rados.rados.2.smithi204.stdout:118: done (2 left) 2024-09-16T05:10:55.114 INFO:tasks.rados.rados.2.smithi204.stdout:122: delete oid 20 current snap is 6 2024-09-16T05:10:55.114 INFO:tasks.rados.rados.2.smithi204.stdout:120: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:10:55.121 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:10:55.121 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:10:55.143 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:10:55.143 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:10:55.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:55.169 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:55.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:55.200 INFO:tasks.rados.rados.2.smithi204.stdout:121: finishing rollback tid 1 to smithi20457400-8 2024-09-16T05:10:55.200 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 15 (ObjNum 129 snap 0 seq_num 4283294696) dirty dne 2024-09-16T05:10:55.200 INFO:tasks.rados.rados.2.smithi204.stdout:120: done (2 left) 2024-09-16T05:10:55.200 INFO:tasks.rados.rados.2.smithi204.stdout:121: done (1 left) 2024-09-16T05:10:55.200 INFO:tasks.rados.rados.2.smithi204.stdout:122: done (0 left) 2024-09-16T05:10:55.200 INFO:tasks.rados.rados.2.smithi204.stdout:123: read oid 46 snap -1 2024-09-16T05:10:55.200 INFO:tasks.rados.rados.2.smithi204.stdout:123: expect (ObjNum 51 snap 1 seq_num 51) 2024-09-16T05:10:55.200 INFO:tasks.rados.rados.2.smithi204.stdout:124: delete oid 25 current snap is 6 2024-09-16T05:10:55.201 INFO:tasks.rados.rados.2.smithi204.stdout:124: done (1 left) 2024-09-16T05:10:55.201 INFO:tasks.rados.rados.2.smithi204.stdout:125: snap_create 2024-09-16T05:10:55.208 INFO:tasks.rados.rados.2.smithi204.stdout:123: expect (ObjNum 51 snap 1 seq_num 51) 2024-09-16T05:10:55.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:55.293 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:55.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:55.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:55.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:55 smithi204 ceph-mon[30933]: osdmap e122: 12 total, 12 up, 12 in 2024-09-16T05:10:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:55 smithi204 ceph-mon[30933]: pgmap v336: 27 pgs: 27 active+clean; 114 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 17 op/s 2024-09-16T05:10:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1411261724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1240391636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:10:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:55.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:55.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:55 smithi196 ceph-mon[30802]: osdmap e122: 12 total, 12 up, 12 in 2024-09-16T05:10:55.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:55 smithi196 ceph-mon[30802]: pgmap v336: 27 pgs: 27 active+clean; 114 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 17 op/s 2024-09-16T05:10:55.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1411261724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:55.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1240391636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:10:55.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:55.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:55.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:55.636 INFO:tasks.rados.rados.2.smithi204.stdout:123: done (1 left) 2024-09-16T05:10:55.637 INFO:tasks.rados.rados.2.smithi204.stdout:125: done (0 left) 2024-09-16T05:10:55.637 INFO:tasks.rados.rados.2.smithi204.stdout:126: copy_from oid 27 from oid 31 current snap is 7 2024-09-16T05:10:55.637 INFO:tasks.rados.rados.2.smithi204.stdout:127: snap_remove snap 6 2024-09-16T05:10:55.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:55.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:55.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:55.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:55.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:55.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:55.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:55.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:55.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:56.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:56.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:56.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:56.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:56.179 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:56.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:56.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:56.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:56.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:56.309 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:56 smithi032 ceph-mon[27388]: osdmap e123: 12 total, 12 up, 12 in 2024-09-16T05:10:56.309 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:56 smithi032 ceph-mon[27388]: osdmap e124: 12 total, 12 up, 12 in 2024-09-16T05:10:56.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:56.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:56 smithi204 ceph-mon[30933]: osdmap e123: 12 total, 12 up, 12 in 2024-09-16T05:10:56.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:56 smithi204 ceph-mon[30933]: osdmap e124: 12 total, 12 up, 12 in 2024-09-16T05:10:56.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:56.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:56 smithi196 ceph-mon[30802]: osdmap e123: 12 total, 12 up, 12 in 2024-09-16T05:10:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:56 smithi196 ceph-mon[30802]: osdmap e124: 12 total, 12 up, 12 in 2024-09-16T05:10:56.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:56.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:56.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:56.639 INFO:tasks.rados.rados.2.smithi204.stdout:127: done (1 left) 2024-09-16T05:10:56.639 INFO:tasks.rados.rados.2.smithi204.stdout:128: delete oid 47 current snap is 7 2024-09-16T05:10:56.639 INFO:tasks.rados.rados.2.smithi204.stdout:126: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:10:56.639 INFO:tasks.rados.rados.2.smithi204.stdout:126: finishing copy_from to smithi20457400-27 2024-09-16T05:10:56.639 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 13 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T05:10:56.641 INFO:tasks.rados.rados.2.smithi204.stdout:126: done (1 left) 2024-09-16T05:10:56.641 INFO:tasks.rados.rados.2.smithi204.stdout:128: done (0 left) 2024-09-16T05:10:56.642 INFO:tasks.rados.rados.2.smithi204.stdout:129: write oid 26 current snap is 7 2024-09-16T05:10:56.642 INFO:tasks.rados.rados.2.smithi204.stdout:129: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-16T05:10:56.649 INFO:tasks.rados.rados.2.smithi204.stdout:129: writing smithi20457400-26 from 544560 to 1328010 tid 1 2024-09-16T05:10:56.654 INFO:tasks.rados.rados.2.smithi204.stdout:129: writing smithi20457400-26 from 1915030 to 2396378 tid 2 2024-09-16T05:10:56.659 INFO:tasks.rados.rados.2.smithi204.stdout:129: writing smithi20457400-26 from 2797604 to 3277365 tid 3 2024-09-16T05:10:56.660 INFO:tasks.rados.rados.2.smithi204.stdout:130: snap_remove snap 5 2024-09-16T05:10:56.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:56.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:56.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:56.752 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5 2.1 e112/107 2:8c1ba486:::smithi20457400-2:head v 112'1, mlcod=0'0)", 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.007090+0000", 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.734430547000001, 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001761501, 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007090+0000", 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007090+0000", 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007090+0000", 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3099999999999999e-07 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007391+0000", 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030055400000000002 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007415+0000", 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4556999999999999e-05 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007424+0000", 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2619999999999998e-06 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007469+0000", 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5532999999999999e-05 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007492+0000", 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2305000000000001e-05 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008819+0000", 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013269239999999999 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008852+0000", 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3135e-05 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.757 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:45 2.a e112/107 2:5827f14d:::smithi20457400-12:head v 112'1, mlcod=0'0)", 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.121669+0000", 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.619852030000001, 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001560564, 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.121669+0000", 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.121669+0000", 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.121669+0000", 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1799999999999999e-07 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.122266+0000", 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000597311 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.122290+0000", 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3399000000000001e-05 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.122299+0000", 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7790000000000007e-06 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.122386+0000", 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6217999999999993e-05 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.122403+0000", 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7733000000000001e-05 2024-09-16T05:10:56.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.123193+0000", 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078982499999999997 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.123229+0000", 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6081000000000002e-05 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:46 2.a e112/107 2:5827f14d:::smithi20457400-12:head v 112'2, mlcod=0'0)", 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.122643+0000", 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.618877213999999, 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001413826, 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.122643+0000", 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.122643+0000", 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.122644+0000", 2024-09-16T05:10:56.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.34e-07 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.123197+0000", 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00055332999999999999 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.123221+0000", 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3261e-05 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.123228+0000", 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0720000000000002e-06 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.123277+0000", 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9217e-05 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.123297+0000", 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0658000000000001e-05 2024-09-16T05:10:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.124025+0000", 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00072722200000000002 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.124057+0000", 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2632000000000002e-05 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:65 2.e e112/107 2:75e143e9:::smithi20457400-17:head v 112'4, mlcod=112'2)", 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.178458+0000", 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.563062863000001, 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014900810000000001, 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178458+0000", 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178458+0000", 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178459+0000", 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2100000000000001e-06 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178639+0000", 2024-09-16T05:10:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000180025 2024-09-16T05:10:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178679+0000", 2024-09-16T05:10:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9959999999999997e-05 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178690+0000", 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0723999999999999e-05 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178763+0000", 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3022999999999995e-05 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.178789+0000", 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6386000000000001e-05 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.179904+0000", 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011152969999999999 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.179948+0000", 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3455999999999999e-05 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.773 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.773 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:109 2.e e112/107 2:75dc7feb:::smithi20457400-28:head v 112'7, mlcod=112'5)", 2024-09-16T05:10:56.773 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.316348+0000", 2024-09-16T05:10:56.773 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.425172952, 2024-09-16T05:10:56.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014548919999999999, 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316348+0000", 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316348+0000", 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316349+0000", 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2580000000000001e-06 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316509+0000", 2024-09-16T05:10:56.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016035800000000001 2024-09-16T05:10:56.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316546+0000", 2024-09-16T05:10:56.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6659000000000003e-05 2024-09-16T05:10:56.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316554+0000", 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7859999999999997e-06 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316618+0000", 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4294000000000005e-05 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.316648+0000", 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9818000000000001e-05 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.317766+0000", 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011183619999999999 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.317803+0000", 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6356999999999998e-05 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.778 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:114 2.a e112/107 2:580e4d0e:::smithi20457400-29:head v 112'5, mlcod=112'3)", 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.333649+0000", 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.407871367, 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013234200000000001, 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.333649+0000", 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.333649+0000", 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.333650+0000", 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6300000000000005e-07 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334092+0000", 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00044221199999999999 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334117+0000", 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.497e-05 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334126+0000", 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4149999999999999e-06 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334201+0000", 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5326000000000004e-05 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334217+0000", 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6056e-05 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334945+0000", 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00072805899999999998 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334973+0000", 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7719e-05 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:115 2.a e112/107 2:580e4d0e:::smithi20457400-29:head v 112'6, mlcod=112'3)", 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.407176271000001, 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0311933, 2024-09-16T05:10:56.783 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334345+0000", 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334350+0000", 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.887e-06 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365213+0000", 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030863313999999999 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365220+0000", 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7870000000000004e-06 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365291+0000", 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1397999999999996e-05 2024-09-16T05:10:56.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365307+0000", 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5990000000000001e-05 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365507+0000", 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000199221 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365538+0000", 2024-09-16T05:10:56.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1167000000000002e-05 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:125 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'7, mlcod=112'4)", 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.377006142000001, 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006511045, 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.788 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365198+0000", 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068299699999999997 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369743+0000", 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045446979999999998 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369750+0000", 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1940000000000001e-06 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369826+0000", 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6160000000000003e-05 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369841+0000", 2024-09-16T05:10:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4895000000000001e-05 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.370997+0000", 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001155866 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.371026+0000", 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8564999999999998e-05 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:126 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'8, mlcod=112'5)", 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.369441+0000", 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.372079816999999, 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001659998, 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369441+0000", 2024-09-16T05:10:56.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369441+0000", 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369443+0000", 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6160000000000001e-06 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369722+0000", 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00027996199999999997 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369771+0000", 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8041999999999998e-05 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369778+0000", 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5850000000000002e-06 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.370368+0000", 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00058986300000000004 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.370384+0000", 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6195999999999999e-05 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.371077+0000", 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069290200000000001 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.371101+0000", 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3832000000000002e-05 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:127 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'9, mlcod=112'5)", 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.371795233, 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020225074999999999, 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369726+0000", 2024-09-16T05:10:56.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-09-16T05:10:56.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369728+0000", 2024-09-16T05:10:56.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9410000000000001e-06 2024-09-16T05:10:56.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389627+0000", 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019899751 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389632+0000", 2024-09-16T05:10:56.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389704+0000", 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1418e-05 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389718+0000", 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.433e-05 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389922+0000", 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020421600000000001 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389951+0000", 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8140000000000002e-05 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:133 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'10, mlcod=112'7)", 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:10:56.803 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.352261305000001, 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074633729999999997, 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389261+0000", 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.973e-06 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389613+0000", 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035121600000000001 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395726+0000", 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006113851 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395731+0000", 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0359999999999997e-06 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395819+0000", 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7128999999999999e-05 2024-09-16T05:10:56.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395836+0000", 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7768999999999999e-05 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396693+0000", 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085685999999999998 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396723+0000", 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9539e-05 2024-09-16T05:10:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:134 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'11, mlcod=112'8)", 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.395287+0000", 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.346233621, 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014980690000000001, 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395287+0000", 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395287+0000", 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395288+0000", 2024-09-16T05:10:56.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0299999999999998e-07 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395712+0000", 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042417399999999998 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395785+0000", 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3208999999999997e-05 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395789+0000", 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8850000000000001e-06 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396160+0000", 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000370878 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396178+0000", 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8346999999999999e-05 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396752+0000", 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000573541 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396785+0000", 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3232000000000003e-05 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.812 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:135 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'12, mlcod=112'8)", 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.34575033, 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069274319999999999, 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395771+0000", 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4499999999999997e-07 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395775+0000", 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6509999999999999e-06 2024-09-16T05:10:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402319+0000", 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065439469999999996 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402326+0000", 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5769999999999996e-06 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402424+0000", 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8171000000000006e-05 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402439+0000", 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4844999999999999e-05 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402660+0000", 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022083700000000001 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402698+0000", 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7858999999999998e-05 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:137 2.a e112/107 2:53edbf55:::smithi20457400-35:head v 112'13, mlcod=112'10)", 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.409611+0000", 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.331910003999999, 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001839961, 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.409611+0000", 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.409611+0000", 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.409611+0000", 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7700000000000005e-07 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410294+0000", 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068301500000000003 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410336+0000", 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1542999999999998e-05 2024-09-16T05:10:56.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410343+0000", 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9029999999999998e-06 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410406+0000", 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3474999999999998e-05 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410421+0000", 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.451e-05 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411424+0000", 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00100332 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411451+0000", 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6718000000000001e-05 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.821 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:138 2.a e112/107 2:53edbf55:::smithi20457400-35:head v 112'14, mlcod=112'10)", 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.410298+0000", 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.331223095, 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014477730000000001, 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410298+0000", 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410298+0000", 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410298+0000", 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9499999999999998e-07 2024-09-16T05:10:56.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410999+0000", 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070145799999999996 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411029+0000", 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9536999999999999e-05 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411034+0000", 2024-09-16T05:10:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7899999999999999e-06 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411071+0000", 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7163999999999999e-05 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411084+0000", 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3389e-05 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411715+0000", 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063097399999999997 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411745+0000", 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0066e-05 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:141 2.a e112/107 2:5089ebe1:::smithi20457400-36:head v 112'16, mlcod=112'14)", 2024-09-16T05:10:56.826 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.416198+0000", 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.32532224, 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001436973, 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.416198+0000", 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.416198+0000", 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.416199+0000", 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0299999999999999e-07 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.416833+0000", 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063366400000000004 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.416856+0000", 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2955e-05 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.416860+0000", 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8570000000000003e-06 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.416922+0000", 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1206999999999999e-05 2024-09-16T05:10:56.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.416936+0000", 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4574e-05 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.417609+0000", 2024-09-16T05:10:56.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000672553 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.417635+0000", 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6659999999999999e-05 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:157 2.e e112/107 2:7b642727:::smithi20457400-40:head v 112'10, mlcod=112'8)", 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.472536+0000", 2024-09-16T05:10:56.831 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 13.268984941999999, 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015513739999999999, 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472536+0000", 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472536+0000", 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472537+0000", 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2979999999999999e-06 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472686+0000", 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014899800000000001 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472720+0000", 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3587999999999998e-05 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472728+0000", 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7620000000000001e-06 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472798+0000", 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9371999999999999e-05 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.472825+0000", 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7253000000000001e-05 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.474047+0000", 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012218800000000001 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.474087+0000", 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0222999999999998e-05 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.835 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:222 2.e e114/107 2:7b642727:::smithi20457400-40:head v 114'14, mlcod=112'11)", 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:46.002342+0000", 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 10.739178281000001, 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019311160000000001, 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002342+0000", 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002342+0000", 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002343+0000", 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6199999999999999e-07 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002788+0000", 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000445236 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002845+0000", 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7271000000000001e-05 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002857+0000", 2024-09-16T05:10:56.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1888999999999999e-05 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002939+0000", 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1420000000000003e-05 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002967+0000", 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8272999999999998e-05 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.004225+0000", 2024-09-16T05:10:56.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001257932 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.004274+0000", 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8733000000000002e-05 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:228 2.a e115/107 2:5089ebe1:::smithi20457400-36:head v 115'20, mlcod=112'17)", 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:47.013127+0000", 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 9.7283940970000007, 2024-09-16T05:10:56.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020438700000000001, 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013127+0000", 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013127+0000", 2024-09-16T05:10:56.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013127+0000", 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9900000000000002e-07 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013827+0000", 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069999700000000005 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013859+0000", 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1606000000000002e-05 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013868+0000", 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013934+0000", 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6019999999999995e-05 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013953+0000", 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8374999999999999e-05 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.015118+0000", 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011652279999999999 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.015170+0000", 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2571000000000002e-05 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.844 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:250 2.e e116/107 2:7e4e65a4:::smithi20457400-4:head v 116'20, mlcod=116'18)", 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:48.166805+0000", 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 8.574715759, 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013039709999999999, 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:10:56.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166805+0000", 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166805+0000", 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.166806+0000", 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1900000000000001e-07 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.167135+0000", 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00032871899999999999 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.167163+0000", 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8427000000000001e-05 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.167169+0000", 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5890000000000002e-06 2024-09-16T05:10:56.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.167223+0000", 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4623999999999997e-05 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.167250+0000", 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6567000000000002e-05 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:10:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.168074+0000", 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00082406099999999996 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:48.168109+0000", 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5064999999999997e-05 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:10:56.849 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:10:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:56.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:56.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:56.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:10:56.978 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:56.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:10:57.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:57.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:57.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:57.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:57.219 INFO:tasks.thrashosds.thrasher:j is {'epoch': 122, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:10:54.027715+0000', 'last_up_change': '2024-09-16T05:09:26.475677+0000', 'last_in_change': '2024-09-16T05:09:05.828775+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '104', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2899999618530273, 'score_stable': 3.2899999618530273, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.7300000190734863, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 122, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3508564664}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3508564664}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3508564664}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3508564664}]}, 'public_addr': '172.21.15.32:6819/3508564664', 'cluster_addr': '172.21.15.32:6821/3508564664', 'heartbeat_back_addr': '172.21.15.32:6825/3508564664', 'heartbeat_front_addr': '172.21.15.32:6823/3508564664', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 108, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:51.488968+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:02.790003+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:29.046992+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:04.735259+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:00.052444+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:39.367915+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:49.025968+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:02.089795+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:35.890683+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:59.269066+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:59.495208+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:38.259572+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 2, 'length': 4}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:10:57.219 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T05:10:57.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:57.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:57.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:57.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:57.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:57.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:57.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:57.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:57.580 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:57.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:57.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:57.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:57.641 INFO:tasks.rados.rados.2.smithi204.stdout:130: done (1 left) 2024-09-16T05:10:57.641 INFO:tasks.rados.rados.2.smithi204.stdout:131: write (excl) oid 34 current snap is 7 2024-09-16T05:10:57.641 INFO:tasks.rados.rados.2.smithi204.stdout:129: finishing write tid 1 to smithi20457400-26 2024-09-16T05:10:57.641 INFO:tasks.rados.rados.2.smithi204.stdout:129: finishing write tid 2 to smithi20457400-26 2024-09-16T05:10:57.642 INFO:tasks.rados.rados.2.smithi204.stdout:129: finishing write tid 3 to smithi20457400-26 2024-09-16T05:10:57.642 INFO:tasks.rados.rados.2.smithi204.stdout:131: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-16T05:10:57.646 INFO:tasks.rados.rados.2.smithi204.stdout:131: writing smithi20457400-34 from 561367 to 1020066 tid 1 2024-09-16T05:10:57.654 INFO:tasks.rados.rados.2.smithi204.stdout:131: writing smithi20457400-34 from 1573512 to 2314871 tid 2 2024-09-16T05:10:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:131: writing smithi20457400-34 from 2941380 to 3361367 tid 3 2024-09-16T05:10:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:132: rollback oid 32 current snap is 7 2024-09-16T05:10:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:132: done (2 left) 2024-09-16T05:10:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:133: write oid 10 current snap is 7 2024-09-16T05:10:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:133: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-16T05:10:57.666 INFO:tasks.rados.rados.2.smithi204.stdout:133: writing smithi20457400-10 from 578174 to 1112122 tid 1 2024-09-16T05:10:57.671 INFO:tasks.rados.rados.2.smithi204.stdout:133: writing smithi20457400-10 from 1515641 to 2000658 tid 2 2024-09-16T05:10:57.678 INFO:tasks.rados.rados.2.smithi204.stdout:133: writing smithi20457400-10 from 2736097 to 3238402 tid 3 2024-09-16T05:10:57.678 INFO:tasks.rados.rados.2.smithi204.stdout:134: write (excl) oid 16 current snap is 7 2024-09-16T05:10:57.678 INFO:tasks.rados.rados.2.smithi204.stdout:129: finishing write tid 4 to smithi20457400-26 2024-09-16T05:10:57.678 INFO:tasks.rados.rados.2.smithi204.stdout:129: finishing write tid 5 to smithi20457400-26 2024-09-16T05:10:57.678 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 58 (ObjNum 80 snap 7 seq_num 80) dirty exists 2024-09-16T05:10:57.678 INFO:tasks.rados.rados.2.smithi204.stdout:129: left oid 26 (ObjNum 80 snap 7 seq_num 80) 2024-09-16T05:10:57.679 INFO:tasks.rados.rados.2.smithi204.stdout:134: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-16T05:10:57.684 INFO:tasks.rados.rados.2.smithi204.stdout:134: writing smithi20457400-16 from 594981 to 1204178 tid 1 2024-09-16T05:10:57.693 INFO:tasks.rados.rados.2.smithi204.stdout:134: writing smithi20457400-16 from 1974123 to 2719151 tid 2 2024-09-16T05:10:57.697 INFO:tasks.rados.rados.2.smithi204.stdout:134: writing smithi20457400-16 from 3279873 to 3394981 tid 3 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:129: done (3 left) 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:135: delete oid 12 current snap is 7 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:131: finishing write tid 1 to smithi20457400-34 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:131: finishing write tid 2 to smithi20457400-34 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:131: finishing write tid 3 to smithi20457400-34 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:131: finishing write tid 4 to smithi20457400-34 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:131: finishing write tid 5 to smithi20457400-34 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 34 (ObjNum 81 snap 7 seq_num 81) dirty exists 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:131: left oid 34 (ObjNum 81 snap 7 seq_num 81) 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:133: finishing write tid 1 to smithi20457400-10 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:133: finishing write tid 2 to smithi20457400-10 2024-09-16T05:10:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:133: finishing write tid 3 to smithi20457400-10 2024-09-16T05:10:57.702 INFO:tasks.rados.rados.2.smithi204.stdout:134: finishing write tid 1 to smithi20457400-16 2024-09-16T05:10:57.702 INFO:tasks.rados.rados.2.smithi204.stdout:134: finishing write tid 2 to smithi20457400-16 2024-09-16T05:10:57.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:57 smithi032 ceph-mon[27388]: pgmap v339: 27 pgs: 27 active+clean; 114 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-16T05:10:57.703 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:57 smithi032 ceph-mon[27388]: osdmap e125: 12 total, 12 up, 12 in 2024-09-16T05:10:57.703 INFO:tasks.rados.rados.2.smithi204.stdout:131: done (3 left) 2024-09-16T05:10:57.703 INFO:tasks.rados.rados.2.smithi204.stdout:135: done (2 left) 2024-09-16T05:10:57.704 INFO:tasks.rados.rados.2.smithi204.stdout:136: read oid 30 snap -1 2024-09-16T05:10:57.704 INFO:tasks.rados.rados.2.smithi204.stdout:136: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T05:10:57.704 INFO:tasks.rados.rados.2.smithi204.stdout:137: write (excl) oid 32 current snap is 7 2024-09-16T05:10:57.704 INFO:tasks.rados.rados.2.smithi204.stdout:137: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-16T05:10:57.704 INFO:tasks.rados.rados.2.smithi204.stdout:137: writing smithi20457400-32 from 628595 to 1104643 tid 1 2024-09-16T05:10:57.711 INFO:tasks.rados.rados.2.smithi204.stdout:137: writing smithi20457400-32 from 1807440 to 2439784 tid 2 2024-09-16T05:10:57.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:57.716 INFO:tasks.rados.rados.2.smithi204.stdout:137: writing smithi20457400-32 from 2934719 to 3387193 tid 3 2024-09-16T05:10:57.717 INFO:tasks.rados.rados.2.smithi204.stdout:138: write (excl) oid 50 current snap is 7 2024-09-16T05:10:57.717 INFO:tasks.rados.rados.2.smithi204.stdout:138: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-16T05:10:57.723 INFO:tasks.rados.rados.2.smithi204.stdout:138: writing smithi20457400-50 from 645402 to 1196699 tid 1 2024-09-16T05:10:57.728 INFO:tasks.rados.rados.2.smithi204.stdout:138: writing smithi20457400-50 from 1749569 to 2241924 tid 2 2024-09-16T05:10:57.735 INFO:tasks.rados.rados.2.smithi204.stdout:138: writing smithi20457400-50 from 2845789 to 3445402 tid 3 2024-09-16T05:10:57.735 INFO:tasks.rados.rados.2.smithi204.stdout:139: read oid 9 snap -1 2024-09-16T05:10:57.735 INFO:tasks.rados.rados.2.smithi204.stdout:139: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T05:10:57.735 INFO:tasks.rados.rados.2.smithi204.stdout:140: write (excl) oid 19 current snap is 7 2024-09-16T05:10:57.736 INFO:tasks.rados.rados.2.smithi204.stdout:140: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-16T05:10:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:57.742 INFO:tasks.rados.rados.2.smithi204.stdout:140: writing smithi20457400-19 from 662209 to 1288755 tid 1 2024-09-16T05:10:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:57.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:57.749 INFO:tasks.rados.rados.2.smithi204.stdout:140: writing smithi20457400-19 from 1808051 to 2444064 tid 2 2024-09-16T05:10:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:140: writing smithi20457400-19 from 2873212 to 3462209 tid 3 2024-09-16T05:10:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:141: snap_create 2024-09-16T05:10:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:134: finishing write tid 3 to smithi20457400-16 2024-09-16T05:10:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:136: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:137: finishing write tid 1 to smithi20457400-32 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:137: finishing write tid 2 to smithi20457400-32 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:137: finishing write tid 3 to smithi20457400-32 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:137: finishing write tid 4 to smithi20457400-32 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:137: finishing write tid 5 to smithi20457400-32 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 39 (ObjNum 85 snap 7 seq_num 85) dirty exists 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:137: left oid 32 (ObjNum 85 snap 7 seq_num 85) 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:138: finishing write tid 1 to smithi20457400-50 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:138: finishing write tid 2 to smithi20457400-50 2024-09-16T05:10:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:139: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T05:10:57.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:57.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:57.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:57.911 INFO:tasks.rados.rados.2.smithi204.stdout:133: finishing write tid 4 to smithi20457400-10 2024-09-16T05:10:57.911 INFO:tasks.rados.rados.2.smithi204.stdout:133: finishing write tid 5 to smithi20457400-10 2024-09-16T05:10:57.911 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 28 (ObjNum 82 snap 7 seq_num 82) dirty exists 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:133: left oid 10 (ObjNum 82 snap 7 seq_num 82) 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:138: finishing write tid 3 to smithi20457400-50 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:138: finishing write tid 4 to smithi20457400-50 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:138: finishing write tid 5 to smithi20457400-50 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 25 (ObjNum 86 snap 7 seq_num 86) dirty exists 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:138: left oid 50 (ObjNum 86 snap 7 seq_num 86) 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:140: finishing write tid 1 to smithi20457400-19 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:140: finishing write tid 2 to smithi20457400-19 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:140: finishing write tid 3 to smithi20457400-19 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:140: finishing write tid 4 to smithi20457400-19 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:140: finishing write tid 5 to smithi20457400-19 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 17 (ObjNum 87 snap 7 seq_num 87) dirty exists 2024-09-16T05:10:57.912 INFO:tasks.rados.rados.2.smithi204.stdout:140: left oid 19 (ObjNum 87 snap 7 seq_num 87) 2024-09-16T05:10:57.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:57 smithi204 ceph-mon[30933]: pgmap v339: 27 pgs: 27 active+clean; 114 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-16T05:10:57.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:57 smithi204 ceph-mon[30933]: osdmap e125: 12 total, 12 up, 12 in 2024-09-16T05:10:57.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:57 smithi196 ceph-mon[30802]: pgmap v339: 27 pgs: 27 active+clean; 114 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2024-09-16T05:10:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:57 smithi196 ceph-mon[30802]: osdmap e125: 12 total, 12 up, 12 in 2024-09-16T05:10:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:57.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:57.992 INFO:tasks.rados.rados.2.smithi204.stdout:134: finishing write tid 4 to smithi20457400-16 2024-09-16T05:10:57.992 INFO:tasks.rados.rados.2.smithi204.stdout:134: finishing write tid 5 to smithi20457400-16 2024-09-16T05:10:57.992 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 47 (ObjNum 83 snap 7 seq_num 83) dirty exists 2024-09-16T05:10:57.992 INFO:tasks.rados.rados.2.smithi204.stdout:134: left oid 16 (ObjNum 83 snap 7 seq_num 83) 2024-09-16T05:10:58.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:10:58.114 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:58.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:10:58.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:10:58.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:58.247 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:58.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:58.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:10:58.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:58.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:10:58.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:10:58.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:58.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:58.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:58.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:58.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:58.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:58.649 INFO:tasks.rados.rados.2.smithi204.stdout:133: done (7 left) 2024-09-16T05:10:58.649 INFO:tasks.rados.rados.2.smithi204.stdout:134: done (6 left) 2024-09-16T05:10:58.650 INFO:tasks.rados.rados.2.smithi204.stdout:136: done (5 left) 2024-09-16T05:10:58.650 INFO:tasks.rados.rados.2.smithi204.stdout:137: done (4 left) 2024-09-16T05:10:58.650 INFO:tasks.rados.rados.2.smithi204.stdout:138: done (3 left) 2024-09-16T05:10:58.650 INFO:tasks.rados.rados.2.smithi204.stdout:139: done (2 left) 2024-09-16T05:10:58.650 INFO:tasks.rados.rados.2.smithi204.stdout:140: done (1 left) 2024-09-16T05:10:58.650 INFO:tasks.rados.rados.2.smithi204.stdout:141: done (0 left) 2024-09-16T05:10:58.650 INFO:tasks.rados.rados.2.smithi204.stdout:142: write (excl) oid 26 current snap is 8 2024-09-16T05:10:58.650 INFO:tasks.rados.rados.2.smithi204.stdout:142: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-16T05:10:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:142: writing smithi20457400-26 from 679016 to 1380811 tid 1 2024-09-16T05:10:58.662 INFO:tasks.rados.rados.2.smithi204.stdout:142: writing smithi20457400-26 from 1866533 to 2362557 tid 2 2024-09-16T05:10:58.668 INFO:tasks.rados.rados.2.smithi204.stdout:142: writing smithi20457400-26 from 2900635 to 3303278 tid 3 2024-09-16T05:10:58.668 INFO:tasks.rados.rados.2.smithi204.stdout:143: rollback oid 49 current snap is 8 2024-09-16T05:10:58.668 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 7 2024-09-16T05:10:58.668 INFO:tasks.rados.rados.2.smithi204.stdout:142: finishing write tid 1 to smithi20457400-26 2024-09-16T05:10:58.668 INFO:tasks.rados.rados.2.smithi204.stdout:142: finishing write tid 2 to smithi20457400-26 2024-09-16T05:10:58.668 INFO:tasks.rados.rados.2.smithi204.stdout:144: write (excl) oid 7 current snap is 8 2024-09-16T05:10:58.668 INFO:tasks.rados.rados.2.smithi204.stdout:144: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-09-16T05:10:58.670 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:10:58.675 INFO:tasks.rados.rados.2.smithi204.stdout:144: writing smithi20457400-7 from 695823 to 1472867 tid 1 2024-09-16T05:10:58.682 INFO:tasks.rados.rados.2.smithi204.stdout:144: writing smithi20457400-7 from 1925015 to 2564697 tid 2 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:144: writing smithi20457400-7 from 3328058 to 3495823 tid 3 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:145: delete oid 42 current snap is 8 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:143: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 129 snap 0 seq_num 1891624216) dirty dne 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:142: finishing write tid 3 to smithi20457400-26 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:142: finishing write tid 4 to smithi20457400-26 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:142: finishing write tid 5 to smithi20457400-26 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 67 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:10:58.687 INFO:tasks.rados.rados.2.smithi204.stdout:142: left oid 26 (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:10:58.688 INFO:tasks.rados.rados.2.smithi204.stdout:144: finishing write tid 1 to smithi20457400-7 2024-09-16T05:10:58.688 INFO:tasks.rados.rados.2.smithi204.stdout:144: finishing write tid 2 to smithi20457400-7 2024-09-16T05:10:58.689 INFO:tasks.rados.rados.2.smithi204.stdout:144: finishing write tid 3 to smithi20457400-7 2024-09-16T05:10:58.689 INFO:tasks.rados.rados.2.smithi204.stdout:144: finishing write tid 4 to smithi20457400-7 2024-09-16T05:10:58.689 INFO:tasks.rados.rados.2.smithi204.stdout:142: done (3 left) 2024-09-16T05:10:58.690 INFO:tasks.rados.rados.2.smithi204.stdout:143: done (2 left) 2024-09-16T05:10:58.690 INFO:tasks.rados.rados.2.smithi204.stdout:145: done (1 left) 2024-09-16T05:10:58.690 INFO:tasks.rados.rados.2.smithi204.stdout:146: read oid 11 snap -1 2024-09-16T05:10:58.690 INFO:tasks.rados.rados.2.smithi204.stdout:146: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T05:10:58.690 INFO:tasks.rados.rados.2.smithi204.stdout:147: copy_from oid 19 from oid 39 current snap is 8 2024-09-16T05:10:58.690 INFO:tasks.rados.rados.2.smithi204.stdout:148: snap_create 2024-09-16T05:10:58.691 INFO:tasks.rados.rados.2.smithi204.stdout:144: finishing write tid 5 to smithi20457400-7 2024-09-16T05:10:58.691 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 30 (ObjNum 89 snap 8 seq_num 89) dirty exists 2024-09-16T05:10:58.691 INFO:tasks.rados.rados.2.smithi204.stdout:144: left oid 7 (ObjNum 89 snap 8 seq_num 89) 2024-09-16T05:10:58.696 INFO:tasks.rados.rados.2.smithi204.stdout:146: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T05:10:58.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:10:58.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:58 smithi196 ceph-mon[30802]: osdmap e126: 12 total, 12 up, 12 in 2024-09-16T05:10:58.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:58 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:10:58.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:58 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:10:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:58.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:10:58.766 INFO:tasks.rados.rados.2.smithi204.stdout:147: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:10:58.766 INFO:tasks.rados.rados.2.smithi204.stdout:147: finishing copy_from to smithi20457400-19 2024-09-16T05:10:58.766 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 33 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:10:58.774 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:10:58.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:58.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:58.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:58.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:58 smithi032 ceph-mon[27388]: osdmap e126: 12 total, 12 up, 12 in 2024-09-16T05:10:58.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:58 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:10:58.893 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:58 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:10:58.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:58 smithi204 ceph-mon[30933]: osdmap e126: 12 total, 12 up, 12 in 2024-09-16T05:10:58.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:58 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:10:58.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:58 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:10:58.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:10:59.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:59.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:10:59.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:10:59.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:59.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:10:59.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:59.237 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:10:59.237 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:10:59.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:59.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:59.259 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:10:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:10:59.279 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:10:59.280 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:10:59.292 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:10:59.293 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:10:59.293 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:10:59.293 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:10:59.300 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:10:59.300 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:10:59.322 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:10:59.322 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:10:59.344 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:10:59.344 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:10:59.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:10:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:59.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:10:59.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:10:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:59.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:10:59.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:10:59.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:10:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:10:59.588 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:10:59.589 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:10:59.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:10:59.610 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:10:59.610 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:10:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:59.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:10:59.631 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:10:59.632 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:10:59.652 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:10:59.654 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:10:59.654 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:10:59.655 INFO:tasks.rados.rados.2.smithi204.stdout:144: done (3 left) 2024-09-16T05:10:59.655 INFO:tasks.rados.rados.2.smithi204.stdout:146: done (2 left) 2024-09-16T05:10:59.656 INFO:tasks.rados.rados.2.smithi204.stdout:147: done (1 left) 2024-09-16T05:10:59.656 INFO:tasks.rados.rados.2.smithi204.stdout:148: done (0 left) 2024-09-16T05:10:59.656 INFO:tasks.rados.rados.2.smithi204.stdout:149: delete oid 16 current snap is 9 2024-09-16T05:10:59.659 INFO:tasks.rados.rados.2.smithi204.stdout:149: done (0 left) 2024-09-16T05:10:59.659 INFO:tasks.rados.rados.2.smithi204.stdout:150: read oid 39 snap -1 2024-09-16T05:10:59.659 INFO:tasks.rados.rados.2.smithi204.stdout:150: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:10:59.659 INFO:tasks.rados.rados.2.smithi204.stdout:151: write oid 12 current snap is 9 2024-09-16T05:10:59.659 INFO:tasks.rados.rados.2.smithi204.stdout:151: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-16T05:10:59.667 INFO:tasks.rados.rados.2.smithi204.stdout:151: writing smithi20457400-12 from 746244 to 1465388 tid 1 2024-09-16T05:10:59.673 INFO:tasks.rados.rados.2.smithi204.stdout:151: writing smithi20457400-12 from 2100461 to 2603823 tid 2 2024-09-16T05:10:59.678 INFO:tasks.rados.rados.2.smithi204.stdout:151: writing smithi20457400-12 from 3126680 to 3546244 tid 3 2024-09-16T05:10:59.678 INFO:tasks.rados.rados.2.smithi204.stdout:152: write oid 40 current snap is 9 2024-09-16T05:10:59.678 INFO:tasks.rados.rados.2.smithi204.stdout:152: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-16T05:10:59.680 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:10:59.680 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:10:59.687 INFO:tasks.rados.rados.2.smithi204.stdout:152: writing smithi20457400-40 from 763051 to 1557444 tid 1 2024-09-16T05:10:59.693 INFO:tasks.rados.rados.2.smithi204.stdout:152: writing smithi20457400-40 from 2158943 to 2805963 tid 2 2024-09-16T05:10:59.698 INFO:tasks.rados.rados.2.smithi204.stdout:152: writing smithi20457400-40 from 3437750 to 3563051 tid 3 2024-09-16T05:10:59.698 INFO:tasks.rados.rados.2.smithi204.stdout:153: copy_from oid 2 from oid 45 current snap is 9 2024-09-16T05:10:59.698 INFO:tasks.rados.rados.2.smithi204.stdout:150: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:10:59.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:10:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:59.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:10:59.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:151: finishing write tid 1 to smithi20457400-12 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:151: finishing write tid 2 to smithi20457400-12 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:151: finishing write tid 3 to smithi20457400-12 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:151: finishing write tid 4 to smithi20457400-12 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:151: finishing write tid 5 to smithi20457400-12 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 43 (ObjNum 92 snap 9 seq_num 92) dirty exists 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:151: left oid 12 (ObjNum 92 snap 9 seq_num 92) 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:152: finishing write tid 1 to smithi20457400-40 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:150: done (3 left) 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:151: done (2 left) 2024-09-16T05:10:59.831 INFO:tasks.rados.rados.2.smithi204.stdout:154: snap_create 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:152: finishing write tid 2 to smithi20457400-40 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:152: finishing write tid 3 to smithi20457400-40 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:152: finishing write tid 4 to smithi20457400-40 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:152: finishing write tid 5 to smithi20457400-40 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 31 (ObjNum 93 snap 9 seq_num 93) dirty exists 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:152: left oid 40 (ObjNum 93 snap 9 seq_num 93) 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:153: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:153: finishing copy_from to smithi20457400-2 2024-09-16T05:10:59.832 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T05:10:59.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:10:59.871 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:59 smithi032 ceph-mon[27388]: pgmap v342: 27 pgs: 27 active+clean; 112 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2024-09-16T05:10:59.871 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:59 smithi032 ceph-mon[27388]: osdmap e127: 12 total, 12 up, 12 in 2024-09-16T05:10:59.871 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1071009793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:59.871 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:10:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1995491034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:59.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:10:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:10:59.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:10:59.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:59 smithi196 ceph-mon[30802]: pgmap v342: 27 pgs: 27 active+clean; 112 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2024-09-16T05:10:59.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:59 smithi196 ceph-mon[30802]: osdmap e127: 12 total, 12 up, 12 in 2024-09-16T05:10:59.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1071009793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:59.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:10:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1995491034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:59 smithi204 ceph-mon[30933]: pgmap v342: 27 pgs: 27 active+clean; 112 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2024-09-16T05:10:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:59 smithi204 ceph-mon[30933]: osdmap e127: 12 total, 12 up, 12 in 2024-09-16T05:10:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1071009793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:10:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:10:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1995491034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:10:59.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:10:59.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:00.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:00.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:00.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:00.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:00.249 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:00.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:00.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:00.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.498 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:00.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:00.571 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:00.571 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:00.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.622 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:00.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:00.638 INFO:tasks.rados.rados.2.smithi204.stdout:152: done (2 left) 2024-09-16T05:11:00.638 INFO:tasks.rados.rados.2.smithi204.stdout:153: done (1 left) 2024-09-16T05:11:00.638 INFO:tasks.rados.rados.2.smithi204.stdout:154: done (0 left) 2024-09-16T05:11:00.638 INFO:tasks.rados.rados.2.smithi204.stdout:155: rollback oid 25 current snap is 10 2024-09-16T05:11:00.638 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 9 2024-09-16T05:11:00.638 INFO:tasks.rados.rados.2.smithi204.stdout:156: snap_create 2024-09-16T05:11:00.640 INFO:tasks.rados.rados.2.smithi204.stdout:155: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:11:00.640 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 129 snap 0 seq_num 2671885633) dirty dne 2024-09-16T05:11:00.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:00.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:00.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:00.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:00.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:00 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:00.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:00 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:00.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:00 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:00.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1071009793' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:00.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:00 smithi196 ceph-mon[30802]: osdmap e128: 12 total, 12 up, 12 in 2024-09-16T05:11:00.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:00 smithi196 ceph-mon[30802]: osdmap e129: 12 total, 12 up, 12 in 2024-09-16T05:11:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:00 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:00 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:00 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1071009793' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:00 smithi032 ceph-mon[27388]: osdmap e128: 12 total, 12 up, 12 in 2024-09-16T05:11:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:00 smithi032 ceph-mon[27388]: osdmap e129: 12 total, 12 up, 12 in 2024-09-16T05:11:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:00 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:00 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:00 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1071009793' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:00 smithi204 ceph-mon[30933]: osdmap e128: 12 total, 12 up, 12 in 2024-09-16T05:11:00.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:00 smithi204 ceph-mon[30933]: osdmap e129: 12 total, 12 up, 12 in 2024-09-16T05:11:00.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:00.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:00.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:01.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:01.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:01.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:01.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:01.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:01.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:01.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:11:01.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:01.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:01.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:01.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:01.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:01.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:01.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:01.535 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:01.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:01.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:01.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:01.644 INFO:tasks.rados.rados.2.smithi204.stdout:155: done (1 left) 2024-09-16T05:11:01.645 INFO:tasks.rados.rados.2.smithi204.stdout:156: done (0 left) 2024-09-16T05:11:01.645 INFO:tasks.rados.rados.2.smithi204.stdout:157: read oid 36 snap -1 2024-09-16T05:11:01.645 INFO:tasks.rados.rados.2.smithi204.stdout:157: expect (ObjNum 55 snap 2 seq_num 55) 2024-09-16T05:11:01.645 INFO:tasks.rados.rados.2.smithi204.stdout:158: copy_from oid 32 from oid 46 current snap is 11 2024-09-16T05:11:01.645 INFO:tasks.rados.rados.2.smithi204.stdout:159: delete oid 7 current snap is 11 2024-09-16T05:11:01.647 INFO:tasks.rados.rados.2.smithi204.stdout:159: done (2 left) 2024-09-16T05:11:01.648 INFO:tasks.rados.rados.2.smithi204.stdout:160: snap_create 2024-09-16T05:11:01.652 INFO:tasks.rados.rados.2.smithi204.stdout:157: expect (ObjNum 55 snap 2 seq_num 55) 2024-09-16T05:11:01.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:01.757 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:01.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:01.779 INFO:tasks.rados.rados.2.smithi204.stdout:158: finishing copy_from racing read to smithi20457400-32 2024-09-16T05:11:01.779 INFO:tasks.rados.rados.2.smithi204.stdout:158: finishing copy_from to smithi20457400-32 2024-09-16T05:11:01.779 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 7 (ObjNum 51 snap 1 seq_num 51) dirty exists 2024-09-16T05:11:01.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:01.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:01.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:01 smithi032 ceph-mon[27388]: pgmap v345: 27 pgs: 27 active+clean; 108 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-16T05:11:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:01 smithi032 ceph-mon[27388]: osdmap e130: 12 total, 12 up, 12 in 2024-09-16T05:11:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:01 smithi204 ceph-mon[30933]: pgmap v345: 27 pgs: 27 active+clean; 108 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-16T05:11:01.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:01 smithi204 ceph-mon[30933]: osdmap e130: 12 total, 12 up, 12 in 2024-09-16T05:11:01.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:01 smithi196 ceph-mon[30802]: pgmap v345: 27 pgs: 27 active+clean; 108 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-16T05:11:01.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:01 smithi196 ceph-mon[30802]: osdmap e130: 12 total, 12 up, 12 in 2024-09-16T05:11:01.970 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:01.971 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:01.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:02.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:02.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:02.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:02.140 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:02.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:02.220 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:11:02.220 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:11:02.220 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-16T05:11:02.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:11:02.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:02.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:02.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:02.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:02.413 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:02.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:02.486 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:02.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:02.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:11:02.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:11:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:02.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:02.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:02.641 INFO:tasks.rados.rados.2.smithi204.stdout:157: done (2 left) 2024-09-16T05:11:02.641 INFO:tasks.rados.rados.2.smithi204.stdout:158: done (1 left) 2024-09-16T05:11:02.641 INFO:tasks.rados.rados.2.smithi204.stdout:160: done (0 left) 2024-09-16T05:11:02.641 INFO:tasks.rados.rados.2.smithi204.stdout:161: read oid 44 snap -1 2024-09-16T05:11:02.642 INFO:tasks.rados.rados.2.smithi204.stdout:161: expect deleted 2024-09-16T05:11:02.642 INFO:tasks.rados.rados.2.smithi204.stdout:162: delete oid 2 current snap is 12 2024-09-16T05:11:02.645 INFO:tasks.rados.rados.2.smithi204.stdout:161: done (1 left) 2024-09-16T05:11:02.645 INFO:tasks.rados.rados.2.smithi204.stdout:162: done (0 left) 2024-09-16T05:11:02.645 INFO:tasks.rados.rados.2.smithi204.stdout:163: rollback oid 41 current snap is 12 2024-09-16T05:11:02.645 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 41 to 7 2024-09-16T05:11:02.645 INFO:tasks.rados.rados.2.smithi204.stdout:164: rollback oid 21 current snap is 12 2024-09-16T05:11:02.645 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 21 to 11 2024-09-16T05:11:02.645 INFO:tasks.rados.rados.2.smithi204.stdout:165: copy_from oid 19 from oid 8 current snap is 12 2024-09-16T05:11:02.645 INFO:tasks.rados.rados.2.smithi204.stdout:166: snap_remove snap 8 2024-09-16T05:11:02.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:02.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:02.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:02.781 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:02.789 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:02.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:02.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:02.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:02.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:02.977 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:03.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:03.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:11:03.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:03.138 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:11:03.138 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:11:03.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:03.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:03.227 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:03.227 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:03.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:03.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.411 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:03.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:03.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:03.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:03.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:03.647 INFO:tasks.rados.rados.2.smithi204.stdout:166: done (3 left) 2024-09-16T05:11:03.648 INFO:tasks.rados.rados.2.smithi204.stdout:167: copy_from oid 38 from oid 28 current snap is 12 2024-09-16T05:11:03.648 INFO:tasks.rados.rados.2.smithi204.stdout:168: write (excl) oid 22 current snap is 12 2024-09-16T05:11:03.648 INFO:tasks.rados.rados.2.smithi204.stdout:168: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-09-16T05:11:03.653 INFO:tasks.rados.rados.2.smithi204.stdout:168: writing smithi20457400-22 from 413472 to 1033612 tid 1 2024-09-16T05:11:03.661 INFO:tasks.rados.rados.2.smithi204.stdout:168: writing smithi20457400-22 from 1818036 to 2328736 tid 2 2024-09-16T05:11:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:03.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:168: writing smithi20457400-22 from 3120019 to 3613472 tid 3 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:169: snap_create 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:164: finishing rollback tid 0 to smithi20457400-21 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:164: finishing rollback tid 1 to smithi20457400-21 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:163: finishing rollback tid 0 to smithi20457400-41 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:163: finishing rollback tid 1 to smithi20457400-41 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:164: finishing rollback tid 2 to smithi20457400-21 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 36 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T05:11:03.669 INFO:tasks.rados.rados.2.smithi204.stdout:163: finishing rollback tid 2 to smithi20457400-41 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 32 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:165: finishing copy_from to smithi20457400-19 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:165: got expected ENOENT (src dne) 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:165: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:168: finishing write tid 1 to smithi20457400-22 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:167: finishing copy_from racing read to smithi20457400-38 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:167: finishing copy_from to smithi20457400-38 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:11:03.670 INFO:tasks.rados.rados.2.smithi204.stdout:168: finishing write tid 2 to smithi20457400-22 2024-09-16T05:11:03.671 INFO:tasks.rados.rados.2.smithi204.stdout:168: finishing write tid 3 to smithi20457400-22 2024-09-16T05:11:03.671 INFO:tasks.rados.rados.2.smithi204.stdout:168: finishing write tid 4 to smithi20457400-22 2024-09-16T05:11:03.671 INFO:tasks.rados.rados.2.smithi204.stdout:168: finishing write tid 5 to smithi20457400-22 2024-09-16T05:11:03.671 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 37 (ObjNum 96 snap 12 seq_num 96) dirty exists 2024-09-16T05:11:03.671 INFO:tasks.rados.rados.2.smithi204.stdout:168: left oid 22 (ObjNum 96 snap 12 seq_num 96) 2024-09-16T05:11:03.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:03.788 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:03 smithi032 ceph-mon[27388]: pgmap v348: 27 pgs: 27 active+clean; 113 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-16T05:11:03.788 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:03 smithi032 ceph-mon[27388]: osdmap e131: 12 total, 12 up, 12 in 2024-09-16T05:11:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:03.817 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-16T05:11:02.638010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131150438,"num_objects":50,"num_object_clones":7,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":1,"num_read":370,"num_read_kb":222085,"num_write":359,"num_write_kb":115981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1907,"ondisk_log_size":1907,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":490920,"kb_used_data":152464,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124450904,"statfs":{"total":1151940427776,"available":1151437725696,"internally_reserved":0,"allocated":156123136,"data_stored":150026788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14159814,"num_objects":4,"num_object_clones":5,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":65,"num_read_kb":51890,"num_write":62,"num_write_kb":25758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.992334"},"pg_stats":[{"pgid":"2.f","version":"126'24","reported_seq":126,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T05:10:58.647513+0000","last_change":"2024-09-16T05:10:57.641117+0000","last_active":"2024-09-16T05:10:58.647513+0000","last_peered":"2024-09-16T05:10:58.647513+0000","last_clean":"2024-09-16T05:10:58.647513+0000","last_became_active":"2024-09-16T05:10:36.956705+0000","last_became_peered":"2024-09-16T05:10:36.956705+0000","last_unstale":"2024-09-16T05:10:58.647513+0000","last_undegraded":"2024-09-16T05:10:58.647513+0000","last_fullsized":"2024-09-16T05:10:58.647513+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:50.320929+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0024589780000000001,"stat_sum":{"num_bytes":4907578,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":15768,"num_write":19,"num_write_kb":7361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.e","version":"128'31","reported_seq":141,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-16T05:11:01.637780+0000","last_change":"2024-09-16T05:10:57.747721+0000","last_active":"2024-09-16T05:11:01.637780+0000","last_peered":"2024-09-16T05:11:01.637780+0000","last_clean":"2024-09-16T05:11:01.637780+0000","last_became_active":"2024-09-16T05:10:37.001925+0000","last_became_peered":"2024-09-16T05:10:37.001925+0000","last_unstale":"2024-09-16T05:11:01.637780+0000","last_undegraded":"2024-09-16T05:11:01.637780+0000","last_fullsized":"2024-09-16T05:11:01.637780+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:38.491880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027872900000000002,"stat_sum":{"num_bytes":10466673,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":22100,"num_write":31,"num_write_kb":8235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"2.d","version":"127'69","reported_seq":180,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T05:10:58.674281+0000","last_change":"2024-09-16T05:10:57.643265+0000","last_active":"2024-09-16T05:10:58.674281+0000","last_peered":"2024-09-16T05:10:58.674281+0000","last_clean":"2024-09-16T05:10:58.674281+0000","last_became_active":"2024-09-16T05:10:36.957439+0000","last_became_peered":"2024-09-16T05:10:36.957439+0000","last_unstale":"2024-09-16T05:10:58.674281+0000","last_undegraded":"2024-09-16T05:10:58.674281+0000","last_fullsized":"2024-09-16T05:10:58.674281+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:05.362791+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0029665120000000001,"stat_sum":{"num_bytes":19348456,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":60,"num_read_kb":30500,"num_write":62,"num_write_kb":19666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.c","version":"127'19","reported_seq":129,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-16T05:11:01.645673+0000","last_change":"2024-09-16T05:10:57.649569+0000","last_active":"2024-09-16T05:11:01.645673+0000","last_peered":"2024-09-16T05:11:01.645673+0000","last_clean":"2024-09-16T05:11:01.645673+0000","last_became_active":"2024-09-16T05:10:36.957767+0000","last_became_peered":"2024-09-16T05:10:36.957767+0000","last_unstale":"2024-09-16T05:11:01.645673+0000","last_undegraded":"2024-09-16T05:11:01.645673+0000","last_fullsized":"2024-09-16T05:11:01.645673+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:04.052240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048105799999999997,"stat_sum":{"num_bytes":11563084,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":13503,"num_write":23,"num_write_kb":9108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"2.8","version":"126'19","reported_seq":119,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.699893+0000","last_change":"2024-09-16T05:10:57.671346+0000","last_active":"2024-09-16T05:11:00.699893+0000","last_peered":"2024-09-16T05:11:00.699893+0000","last_clean":"2024-09-16T05:11:00.699893+0000","last_became_active":"2024-09-16T05:10:36.959606+0000","last_became_peered":"2024-09-16T05:10:36.959606+0000","last_unstale":"2024-09-16T05:11:00.699893+0000","last_undegraded":"2024-09-16T05:11:00.699893+0000","last_fullsized":"2024-09-16T05:11:00.699893+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:04.209518+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0015852399999999999,"stat_sum":{"num_bytes":5983226,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":7283,"num_write":14,"num_write_kb":6304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"2.3","version":"112'3","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.919624+0000","last_change":"2024-09-16T05:10:57.730349+0000","last_active":"2024-09-16T05:11:00.919624+0000","last_peered":"2024-09-16T05:11:00.919624+0000","last_clean":"2024-09-16T05:11:00.919624+0000","last_became_active":"2024-09-16T05:10:36.956582+0000","last_became_peered":"2024-09-16T05:10:36.956582+0000","last_unstale":"2024-09-16T05:11:00.919624+0000","last_undegraded":"2024-09-16T05:11:00.919624+0000","last_fullsized":"2024-09-16T05:11:00.919624+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:43.610432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041476799999999998,"stat_sum":{"num_bytes":2285719,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"1.0","version":"35'139","reported_seq":451,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.700002+0000","last_change":"2024-09-16T05:10:41.789086+0000","last_active":"2024-09-16T05:11:00.700002+0000","last_peered":"2024-09-16T05:11:00.700002+0000","last_clean":"2024-09-16T05:11:00.700002+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:11:00.700002+0000","last_undegraded":"2024-09-16T05:11:00.700002+0000","last_fullsized":"2024-09-16T05:11:00.700002+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:41.788991+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:41.788991+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:41.897583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"112'3","reported_seq":102,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.919621+0000","last_change":"2024-09-16T05:10:57.730275+0000","last_active":"2024-09-16T05:11:00.919621+0000","last_peered":"2024-09-16T05:11:00.919621+0000","last_clean":"2024-09-16T05:11:00.919621+0000","last_became_active":"2024-09-16T05:10:36.956451+0000","last_became_peered":"2024-09-16T05:10:36.956451+0000","last_unstale":"2024-09-16T05:11:00.919621+0000","last_undegraded":"2024-09-16T05:11:00.919621+0000","last_fullsized":"2024-09-16T05:11:00.919621+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:53.257579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037732299999999999,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":2747,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"1.1","version":"35'139","reported_seq":453,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.732580+0000","last_change":"2024-09-16T05:10:46.732985+0000","last_active":"2024-09-16T05:11:00.732580+0000","last_peered":"2024-09-16T05:11:00.732580+0000","last_clean":"2024-09-16T05:11:00.732580+0000","last_became_active":"2024-09-16T05:10:38.972381+0000","last_became_peered":"2024-09-16T05:10:38.972381+0000","last_unstale":"2024-09-16T05:11:00.732580+0000","last_undegraded":"2024-09-16T05:11:00.732580+0000","last_fullsized":"2024-09-16T05:11:00.732580+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:46.732896+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:46.732896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:47.973041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":455,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.699742+0000","last_change":"2024-09-16T05:10:42.783065+0000","last_active":"2024-09-16T05:11:00.699742+0000","last_peered":"2024-09-16T05:11:00.699742+0000","last_clean":"2024-09-16T05:11:00.699742+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:11:00.699742+0000","last_undegraded":"2024-09-16T05:11:00.699742+0000","last_fullsized":"2024-09-16T05:11:00.699742+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:42.782976+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:42.782976+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:44.473801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"128'9","reported_seq":113,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-16T05:11:01.637775+0000","last_change":"2024-09-16T05:10:57.747665+0000","last_active":"2024-09-16T05:11:01.637775+0000","last_peered":"2024-09-16T05:11:01.637775+0000","last_clean":"2024-09-16T05:11:01.637775+0000","last_became_active":"2024-09-16T05:10:37.001937+0000","last_became_peered":"2024-09-16T05:10:37.001937+0000","last_unstale":"2024-09-16T05:11:01.637775+0000","last_undegraded":"2024-09-16T05:11:01.637775+0000","last_fullsized":"2024-09-16T05:11:01.637775+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:43.289635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026862300000000001,"stat_sum":{"num_bytes":5781542,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":4613,"num_write":6,"num_write_kb":6576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"1.3","version":"35'139","reported_seq":451,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T05:10:59.653421+0000","last_change":"2024-09-16T05:10:45.849284+0000","last_active":"2024-09-16T05:10:59.653421+0000","last_peered":"2024-09-16T05:10:59.653421+0000","last_clean":"2024-09-16T05:10:59.653421+0000","last_became_active":"2024-09-16T05:10:38.966861+0000","last_became_peered":"2024-09-16T05:10:38.966861+0000","last_unstale":"2024-09-16T05:10:59.653421+0000","last_undegraded":"2024-09-16T05:10:59.653421+0000","last_fullsized":"2024-09-16T05:10:59.653421+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.849175+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.849175+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:04.083168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"121'10","reported_seq":106,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.699885+0000","last_change":"2024-09-16T05:10:57.670194+0000","last_active":"2024-09-16T05:11:00.699885+0000","last_peered":"2024-09-16T05:11:00.699885+0000","last_clean":"2024-09-16T05:11:00.699885+0000","last_became_active":"2024-09-16T05:10:36.983391+0000","last_became_peered":"2024-09-16T05:10:36.983391+0000","last_unstale":"2024-09-16T05:11:00.699885+0000","last_undegraded":"2024-09-16T05:11:00.699885+0000","last_fullsized":"2024-09-16T05:11:00.699885+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:48.637780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041933999999999998,"stat_sum":{"num_bytes":3142876,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":1825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"2.7","version":"0'0","reported_seq":96,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-16T05:11:02.209738+0000","last_change":"2024-09-16T05:10:57.730216+0000","last_active":"2024-09-16T05:11:02.209738+0000","last_peered":"2024-09-16T05:11:02.209738+0000","last_clean":"2024-09-16T05:11:02.209738+0000","last_became_active":"2024-09-16T05:10:36.983959+0000","last_became_peered":"2024-09-16T05:10:36.983959+0000","last_unstale":"2024-09-16T05:11:02.209738+0000","last_undegraded":"2024-09-16T05:11:02.209738+0000","last_fullsized":"2024-09-16T05:11:02.209738+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:41.318547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029564899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"1.4","version":"35'139","reported_seq":415,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-16T05:11:01.645632+0000","last_change":"2024-09-16T05:10:41.421609+0000","last_active":"2024-09-16T05:11:01.645632+0000","last_peered":"2024-09-16T05:11:01.645632+0000","last_clean":"2024-09-16T05:11:01.645632+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:11:01.645632+0000","last_undegraded":"2024-09-16T05:11:01.645632+0000","last_fullsized":"2024-09-16T05:11:01.645632+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:41.421501+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:41.421501+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:51.416537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":107,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T05:10:59.653521+0000","last_change":"2024-09-16T05:10:57.668386+0000","last_active":"2024-09-16T05:10:59.653521+0000","last_peered":"2024-09-16T05:10:59.653521+0000","last_clean":"2024-09-16T05:10:59.653521+0000","last_became_active":"2024-09-16T05:10:36.957473+0000","last_became_peered":"2024-09-16T05:10:36.957473+0000","last_unstale":"2024-09-16T05:10:59.653521+0000","last_undegraded":"2024-09-16T05:10:59.653521+0000","last_fullsized":"2024-09-16T05:10:59.653521+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:42.344508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029218499999999998,"stat_sum":{"num_bytes":2857157,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13147,"num_write":8,"num_write_kb":2442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.5","version":"35'139","reported_seq":454,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.732600+0000","last_change":"2024-09-16T05:10:44.788217+0000","last_active":"2024-09-16T05:11:00.732600+0000","last_peered":"2024-09-16T05:11:00.732600+0000","last_clean":"2024-09-16T05:11:00.732600+0000","last_became_active":"2024-09-16T05:10:38.972150+0000","last_became_peered":"2024-09-16T05:10:38.972150+0000","last_unstale":"2024-09-16T05:11:00.732600+0000","last_undegraded":"2024-09-16T05:11:00.732600+0000","last_fullsized":"2024-09-16T05:11:00.732600+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.788114+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.788114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:57.373836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"126'47","reported_seq":161,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T05:10:58.691448+0000","last_change":"2024-09-16T05:10:57.640720+0000","last_active":"2024-09-16T05:10:58.691448+0000","last_peered":"2024-09-16T05:10:58.691448+0000","last_clean":"2024-09-16T05:10:58.691448+0000","last_became_active":"2024-09-16T05:10:36.953485+0000","last_became_peered":"2024-09-16T05:10:36.953485+0000","last_unstale":"2024-09-16T05:10:58.691448+0000","last_undegraded":"2024-09-16T05:10:58.691448+0000","last_fullsized":"2024-09-16T05:10:58.691448+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:01.717442+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0019527240000000001,"stat_sum":{"num_bytes":15462242,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":37208,"num_write":45,"num_write_kb":13281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.6","version":"35'139","reported_seq":454,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.732600+0000","last_change":"2024-09-16T05:10:45.767820+0000","last_active":"2024-09-16T05:11:00.732600+0000","last_peered":"2024-09-16T05:11:00.732600+0000","last_clean":"2024-09-16T05:11:00.732600+0000","last_became_active":"2024-09-16T05:10:38.972286+0000","last_became_peered":"2024-09-16T05:10:38.972286+0000","last_unstale":"2024-09-16T05:11:00.732600+0000","last_undegraded":"2024-09-16T05:11:00.732600+0000","last_fullsized":"2024-09-16T05:11:00.732600+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.767717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.767717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:08.779630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"127'30","reported_seq":131,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T05:10:59.653510+0000","last_change":"2024-09-16T05:10:57.668590+0000","last_active":"2024-09-16T05:10:59.653510+0000","last_peered":"2024-09-16T05:10:59.653510+0000","last_clean":"2024-09-16T05:10:59.653510+0000","last_became_active":"2024-09-16T05:10:36.956394+0000","last_became_peered":"2024-09-16T05:10:36.956394+0000","last_unstale":"2024-09-16T05:10:59.653510+0000","last_undegraded":"2024-09-16T05:10:59.653510+0000","last_fullsized":"2024-09-16T05:10:59.653510+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:36.619908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044924400000000001,"stat_sum":{"num_bytes":10694091,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":6242,"num_write":28,"num_write_kb":7806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.7","version":"35'139","reported_seq":453,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-16T05:11:02.209746+0000","last_change":"2024-09-16T05:10:44.115485+0000","last_active":"2024-09-16T05:11:02.209746+0000","last_peered":"2024-09-16T05:11:02.209746+0000","last_clean":"2024-09-16T05:11:02.209746+0000","last_became_active":"2024-09-16T05:10:38.969713+0000","last_became_peered":"2024-09-16T05:10:38.969713+0000","last_unstale":"2024-09-16T05:11:02.209746+0000","last_undegraded":"2024-09-16T05:11:02.209746+0000","last_fullsized":"2024-09-16T05:11:02.209746+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.115392+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.115392+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:47.509310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"121'32","reported_seq":130,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T05:10:58.647574+0000","last_change":"2024-09-16T05:10:57.639123+0000","last_active":"2024-09-16T05:10:58.647574+0000","last_peered":"2024-09-16T05:10:58.647574+0000","last_clean":"2024-09-16T05:10:58.647574+0000","last_became_active":"2024-09-16T05:10:36.956652+0000","last_became_peered":"2024-09-16T05:10:36.956652+0000","last_unstale":"2024-09-16T05:10:58.647574+0000","last_undegraded":"2024-09-16T05:10:58.647574+0000","last_fullsized":"2024-09-16T05:10:58.647574+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:38.790810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045959800000000003,"stat_sum":{"num_bytes":8958032,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":30,"num_write_kb":8590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.8","version":"35'139","reported_seq":452,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.919541+0000","last_change":"2024-09-16T05:10:45.437354+0000","last_active":"2024-09-16T05:11:00.919541+0000","last_peered":"2024-09-16T05:11:00.919541+0000","last_clean":"2024-09-16T05:11:00.919541+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:11:00.919541+0000","last_undegraded":"2024-09-16T05:11:00.919541+0000","last_fullsized":"2024-09-16T05:11:00.919541+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.437261+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.437261+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:05.011089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"130'45","reported_seq":164,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-09-16T05:11:01.663077+0000","last_change":"2024-09-16T05:10:57.650819+0000","last_active":"2024-09-16T05:11:01.663077+0000","last_peered":"2024-09-16T05:11:01.663077+0000","last_clean":"2024-09-16T05:11:01.663077+0000","last_became_active":"2024-09-16T05:10:37.002200+0000","last_became_peered":"2024-09-16T05:10:37.002200+0000","last_unstale":"2024-09-16T05:11:01.663077+0000","last_undegraded":"2024-09-16T05:11:01.663077+0000","last_fullsized":"2024-09-16T05:11:01.663077+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:36.310233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017399480000000001,"stat_sum":{"num_bytes":18689186,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":30357,"num_write":48,"num_write_kb":15567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"6","length":"3"}]},{"pgid":"1.9","version":"35'139","reported_seq":455,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T05:11:00.699810+0000","last_change":"2024-09-16T05:10:43.759254+0000","last_active":"2024-09-16T05:11:00.699810+0000","last_peered":"2024-09-16T05:11:00.699810+0000","last_clean":"2024-09-16T05:11:00.699810+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:11:00.699810+0000","last_undegraded":"2024-09-16T05:11:00.699810+0000","last_fullsized":"2024-09-16T05:11:00.699810+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:43.759130+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:43.759130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:55.539739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"126'28","reported_seq":140,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T05:10:59.653495+0000","last_change":"2024-09-16T05:10:57.669314+0000","last_active":"2024-09-16T05:10:59.653495+0000","last_peered":"2024-09-16T05:10:59.653495+0000","last_clean":"2024-09-16T05:10:59.653495+0000","last_became_active":"2024-09-16T05:10:36.958530+0000","last_became_peered":"2024-09-16T05:10:36.958530+0000","last_unstale":"2024-09-16T05:10:59.653495+0000","last_undegraded":"2024-09-16T05:10:59.653495+0000","last_fullsized":"2024-09-16T05:10:59.653495+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:47.915669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012568309999999999,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":38607,"num_write":28,"num_write_kb":7179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"1.a","version":"35'139","reported_seq":451,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T05:10:58.658576+0000","last_change":"2024-09-16T05:10:44.010384+0000","last_active":"2024-09-16T05:10:58.658576+0000","last_peered":"2024-09-16T05:10:58.658576+0000","last_clean":"2024-09-16T05:10:58.658576+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:10:58.658576+0000","last_undegraded":"2024-09-16T05:10:58.658576+0000","last_fullsized":"2024-09-16T05:10:58.658576+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.010311+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.010311+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:55.319454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":130560070,"num_objects":48,"num_object_clones":7,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":1,"num_read":370,"num_read_kb":222085,"num_write":359,"num_write_kb":115981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139010048,"data_stored":141138696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":378,"ondisk_log_size":378,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089367,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49884,"kb_used_data":22852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93695268,"statfs":{"total":95995035648,"available":95943954432,"internally_reserved":0,"allocated":23400448,"data_stored":22584960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285595,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29812,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93715340,"statfs":{"total":95995035648,"available":95964508160,"internally_reserved":0,"allocated":2842624,"data_stored":2141667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514530,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45908,"kb_used_data":18880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93699244,"statfs":{"total":95995035648,"available":95948025856,"internally_reserved":0,"allocated":19333120,"data_stored":19165757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841572,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45952,"kb_used_data":18712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93699200,"statfs":{"total":95995035648,"available":95947980800,"internally_reserved":0,"allocated":19161088,"data_stored":18341171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103210,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37552,"kb_used_data":10520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93707600,"statfs":{"total":95995035648,"available":95956582400,"internally_reserved":0,"allocated":10772480,"data_stored":10020107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397553,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32604,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712548,"statfs":{"total":95995035648,"available":95961649152,"internally_reserved":0,"allocated":1445888,"data_stored":756469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757300,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":30676,"kb_used_data":3644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93714476,"statfs":{"total":95995035648,"available":95963623424,"internally_reserved":0,"allocated":3731456,"data_stored":3028942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39652,"kb_used_data":12620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93705500,"statfs":{"total":95995035648,"available":95954432000,"internally_reserved":0,"allocated":12922880,"data_stored":12783982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117054,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39432,"kb_used_data":12404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93705720,"statfs":{"total":95995035648,"available":95954657280,"internally_reserved":0,"allocated":12701696,"data_stored":11954148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313285,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42760,"kb_used_data":11568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93702392,"statfs":{"total":95995035648,"available":95951249408,"internally_reserved":0,"allocated":11845632,"data_stored":11639112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43284,"kb_used_data":14864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93701868,"statfs":{"total":95995035648,"available":95950712832,"internally_reserved":0,"allocated":15220736,"data_stored":15069045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53404,"kb_used_data":22212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93691748,"statfs":{"total":95995035648,"available":95940349952,"internally_reserved":0,"allocated":22745088,"data_stored":22541428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21311488,"data_stored":21793335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13787136,"data_stored":14320952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10194944,"data_stored":10681395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12435456,"data_stored":12361279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11489280,"data_stored":12035889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2105344,"data_stored":2087971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9326592,"data_stored":9263638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":17424384,"data_stored":17282138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":17911808,"data_stored":18426034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1409024,"data_stored":1393574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":21614592,"data_stored":21492491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:11:03.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-09-16T05:11:03.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:03.904 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:03 smithi196 ceph-mon[30802]: pgmap v348: 27 pgs: 27 active+clean; 113 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-16T05:11:03.905 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:03 smithi196 ceph-mon[30802]: osdmap e131: 12 total, 12 up, 12 in 2024-09-16T05:11:03.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:03 smithi204 ceph-mon[30933]: pgmap v348: 27 pgs: 27 active+clean; 113 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-16T05:11:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:03 smithi204 ceph-mon[30933]: osdmap e131: 12 total, 12 up, 12 in 2024-09-16T05:11:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:03.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:11:04.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:04.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:04.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:04.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:04.172 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:04.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:04.215 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:04.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:04.288 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:04.300 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:04.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:04.306 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:04.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:04.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:04.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:04.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:04.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:04.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:04.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:163: done (5 left) 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:164: done (4 left) 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:165: done (3 left) 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:167: done (2 left) 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:168: done (1 left) 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:169: done (0 left) 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:170: rollback oid 48 current snap is 13 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 48 to 7 2024-09-16T05:11:04.654 INFO:tasks.rados.rados.2.smithi204.stdout:171: snap_remove snap 11 2024-09-16T05:11:04.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:04.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:04 smithi032 ceph-mon[27388]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:04.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:04 smithi032 ceph-mon[27388]: osdmap e132: 12 total, 12 up, 12 in 2024-09-16T05:11:04.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2182484492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:04.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:04.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:04.759 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:04.759 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:04.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:04.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:04.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:04.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:04 smithi204 ceph-mon[30933]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:04.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:04 smithi204 ceph-mon[30933]: osdmap e132: 12 total, 12 up, 12 in 2024-09-16T05:11:04.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2182484492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:04.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:04 smithi196 ceph-mon[30802]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:04 smithi196 ceph-mon[30802]: osdmap e132: 12 total, 12 up, 12 in 2024-09-16T05:11:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2182484492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:05.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:05.053 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:05.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:05.097 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:05.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:05.119 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:05.139 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:05.140 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:05.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:05.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:05.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:05.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:05.182 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:05.182 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:05.204 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:05.204 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:05.225 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:05.248 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:05.249 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:05.271 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:05.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:05.293 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:05.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:05.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:05.314 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:05.314 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:05.336 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:05.337 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:05.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:05.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:05.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:05.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:05.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:05.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:05.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:11:05.636 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pgp_num to 16 2024-09-16T05:11:05.636 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:11:05.640 INFO:tasks.rados.rados.2.smithi204.stdout:171: done (1 left) 2024-09-16T05:11:05.640 INFO:tasks.rados.rados.2.smithi204.stdout:172: snap_create 2024-09-16T05:11:05.640 INFO:tasks.rados.rados.2.smithi204.stdout:170: finishing rollback tid 0 to smithi20457400-48 2024-09-16T05:11:05.640 INFO:tasks.rados.rados.2.smithi204.stdout:170: finishing rollback tid 1 to smithi20457400-48 2024-09-16T05:11:05.640 INFO:tasks.rados.rados.2.smithi204.stdout:170: finishing rollback tid 2 to smithi20457400-48 2024-09-16T05:11:05.640 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 40 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T05:11:05.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:05.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:05.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:05.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:05.804 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:05.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:05.872 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:05.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:05 smithi032 ceph-mon[27388]: pgmap v351: 27 pgs: 27 active+clean; 127 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 30 op/s 2024-09-16T05:11:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:05 smithi032 ceph-mon[27388]: osdmap e133: 12 total, 12 up, 12 in 2024-09-16T05:11:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/31916287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/726524328' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-16T05:11:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:05 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/31916287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/726524328' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T05:11:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:05 smithi032 ceph-mon[27388]: osdmap e134: 12 total, 12 up, 12 in 2024-09-16T05:11:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:05 smithi204 ceph-mon[30933]: pgmap v351: 27 pgs: 27 active+clean; 127 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 30 op/s 2024-09-16T05:11:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:05 smithi204 ceph-mon[30933]: osdmap e133: 12 total, 12 up, 12 in 2024-09-16T05:11:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/31916287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/726524328' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-16T05:11:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:05 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/31916287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/726524328' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T05:11:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:05 smithi204 ceph-mon[30933]: osdmap e134: 12 total, 12 up, 12 in 2024-09-16T05:11:05.930 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:05.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:05 smithi196 ceph-mon[30802]: pgmap v351: 27 pgs: 27 active+clean; 127 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.5 MiB/s wr, 30 op/s 2024-09-16T05:11:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:05 smithi196 ceph-mon[30802]: osdmap e133: 12 total, 12 up, 12 in 2024-09-16T05:11:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/31916287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/726524328' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-16T05:11:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:05 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/31916287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/726524328' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-16T05:11:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:05 smithi196 ceph-mon[30802]: osdmap e134: 12 total, 12 up, 12 in 2024-09-16T05:11:06.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:06.055 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:06.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:06.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:06.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:06.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:06.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:06.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:06.364 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:11:06.364 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:11:06.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:11:06.365 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:11:06.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.365 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGPush(1.0 40/39 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 35'138, data_included: [0~4096,65536~197152], data_size: 201248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@35'138, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T05:11:06.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:06:57.712705+0000", 2024-09-16T05:11:06.365 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 248.647766507, 2024-09-16T05:11:06.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00067402100000000002, 2024-09-16T05:11:06.366 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.366 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.366 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.366 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.712705+0000", 2024-09-16T05:11:06.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.712705+0000", 2024-09-16T05:11:06.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.712706+0000", 2024-09-16T05:11:06.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.06e-07 2024-09-16T05:11:06.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.712994+0000", 2024-09-16T05:11:06.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028854299999999999 2024-09-16T05:11:06.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.713021+0000", 2024-09-16T05:11:06.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6525000000000001e-05 2024-09-16T05:11:06.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.713039+0000", 2024-09-16T05:11:06.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8224e-05 2024-09-16T05:11:06.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.713105+0000", 2024-09-16T05:11:06.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5610000000000004e-05 2024-09-16T05:11:06.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.713118+0000", 2024-09-16T05:11:06.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3063e-05 2024-09-16T05:11:06.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.713379+0000", 2024-09-16T05:11:06.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026144999999999999 2024-09-16T05:11:06.373 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.373 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.373 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.374 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGPush(1.0 40/39 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 35'139, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@35'139, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T05:11:06.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:06:57.714501+0000", 2024-09-16T05:11:06.374 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 248.64597011699999, 2024-09-16T05:11:06.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00074829400000000004, 2024-09-16T05:11:06.374 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.374 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.374 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.375 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.714501+0000", 2024-09-16T05:11:06.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.714501+0000", 2024-09-16T05:11:06.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.714502+0000", 2024-09-16T05:11:06.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2300000000000002e-07 2024-09-16T05:11:06.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.714811+0000", 2024-09-16T05:11:06.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030892199999999999 2024-09-16T05:11:06.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.714834+0000", 2024-09-16T05:11:06.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3050000000000001e-05 2024-09-16T05:11:06.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.714848+0000", 2024-09-16T05:11:06.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3969e-05 2024-09-16T05:11:06.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.714899+0000", 2024-09-16T05:11:06.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1301999999999998e-05 2024-09-16T05:11:06.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.714906+0000", 2024-09-16T05:11:06.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4869999999999996e-06 2024-09-16T05:11:06.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:06:57.715250+0000", 2024-09-16T05:11:06.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034324099999999999 2024-09-16T05:11:06.382 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.382 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.382 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.382 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:57 2.2 2:4e7d172c:::smithi20457400-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:06.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:11:06.383 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.20143401, 2024-09-16T05:11:06.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045779080000000003, 2024-09-16T05:11:06.383 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.383 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 57 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159039+0000", 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6640000000000001e-06 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159883+0000", 2024-09-16T05:11:06.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084362100000000002 2024-09-16T05:11:06.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159895+0000", 2024-09-16T05:11:06.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2632999999999999e-05 2024-09-16T05:11:06.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159903+0000", 2024-09-16T05:11:06.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8900000000000007e-06 2024-09-16T05:11:06.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159978+0000", 2024-09-16T05:11:06.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5143999999999997e-05 2024-09-16T05:11:06.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:11:06.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159988+0000", 2024-09-16T05:11:06.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.431e-06 2024-09-16T05:11:06.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161080+0000", 2024-09-16T05:11:06.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010917990000000001 2024-09-16T05:11:06.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161168+0000", 2024-09-16T05:11:06.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8140000000000007e-05 2024-09-16T05:11:06.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:11:06.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161283+0000", 2024-09-16T05:11:06.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115513 2024-09-16T05:11:06.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:06.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163508+0000", 2024-09-16T05:11:06.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022243050000000002 2024-09-16T05:11:06.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:06.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163545+0000", 2024-09-16T05:11:06.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7033000000000003e-05 2024-09-16T05:11:06.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:06.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163578+0000", 2024-09-16T05:11:06.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2804e-05 2024-09-16T05:11:06.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163615+0000", 2024-09-16T05:11:06.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7931e-05 2024-09-16T05:11:06.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.395 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.396 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:58 2.2 2:4e7d172c:::smithi20457400-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:06.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:11:06.396 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.200551352000002, 2024-09-16T05:11:06.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003854761, 2024-09-16T05:11:06.397 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.397 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.397 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:06.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:06.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:06.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:06.398 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58 2024-09-16T05:11:06.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.398 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:11:06.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:11:06.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159921+0000", 2024-09-16T05:11:06.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2699999999999999e-07 2024-09-16T05:11:06.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160047+0000", 2024-09-16T05:11:06.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012650399999999999 2024-09-16T05:11:06.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160055+0000", 2024-09-16T05:11:06.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6539999999999999e-06 2024-09-16T05:11:06.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160059+0000", 2024-09-16T05:11:06.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5059999999999997e-06 2024-09-16T05:11:06.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160100+0000", 2024-09-16T05:11:06.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0153e-05 2024-09-16T05:11:06.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:06.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160104+0000", 2024-09-16T05:11:06.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0600000000000001e-06 2024-09-16T05:11:06.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162622+0000", 2024-09-16T05:11:06.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025179540000000002 2024-09-16T05:11:06.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162679+0000", 2024-09-16T05:11:06.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7163999999999997e-05 2024-09-16T05:11:06.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:11:06.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162793+0000", 2024-09-16T05:11:06.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114398 2024-09-16T05:11:06.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:06.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163515+0000", 2024-09-16T05:11:06.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072178499999999996 2024-09-16T05:11:06.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:06.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163692+0000", 2024-09-16T05:11:06.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017674699999999999 2024-09-16T05:11:06.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:06.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163727+0000", 2024-09-16T05:11:06.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5337999999999999e-05 2024-09-16T05:11:06.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163775+0000", 2024-09-16T05:11:06.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7970999999999999e-05 2024-09-16T05:11:06.410 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.410 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.411 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.411 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:59 2.2 2:4e7d172c:::smithi20457400-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:06.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:11:06.411 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.200402458999999, 2024-09-16T05:11:06.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038110150000000001, 2024-09-16T05:11:06.412 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.412 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.412 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:06.412 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:06.412 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:06.413 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:06.413 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 59 2024-09-16T05:11:06.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.413 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:11:06.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:11:06.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160070+0000", 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4999999999999998e-07 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160072+0000", 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.88e-06 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160078+0000", 2024-09-16T05:11:06.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.152e-06 2024-09-16T05:11:06.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160081+0000", 2024-09-16T05:11:06.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.422e-06 2024-09-16T05:11:06.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160135+0000", 2024-09-16T05:11:06.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3808999999999999e-05 2024-09-16T05:11:06.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:06.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160138+0000", 2024-09-16T05:11:06.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5590000000000001e-06 2024-09-16T05:11:06.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163139+0000", 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030010689999999999 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163192+0000", 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2398000000000002e-05 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163275+0000", 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2825999999999996e-05 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163518+0000", 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024361200000000001 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163815+0000", 2024-09-16T05:11:06.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029653299999999997 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163845+0000", 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0409e-05 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163880+0000", 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4895999999999999e-05 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.423 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:60 2.2 2:4e7d172c:::smithi20457400-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:06.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:11:06.423 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.200372589000001, 2024-09-16T05:11:06.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039136570000000001, 2024-09-16T05:11:06.423 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.423 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.423 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.424 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:06.424 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:06.424 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:06.424 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 60 2024-09-16T05:11:06.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.425 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:11:06.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6100000000000001e-07 2024-09-16T05:11:06.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160103+0000", 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.162e-06 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160108+0000", 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3279999999999996e-06 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160112+0000", 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3479999999999997e-06 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160157+0000", 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5141e-05 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160161+0000", 2024-09-16T05:11:06.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8589999999999998e-06 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163451+0000", 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032897220000000001 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163500+0000", 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8665000000000002e-05 2024-09-16T05:11:06.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163911+0000", 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041112700000000001 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163953+0000", 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2228000000000003e-05 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.164013+0000", 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9616e-05 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:57 2.2 e112/107 ondisk, result = 0)", 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.163415+0000", 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.197056614000001, 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020953200000000001, 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163415+0000", 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163415+0000", 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163416+0000", 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.283e-06 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163421+0000", 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0100000000000003e-06 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163452+0000", 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.095e-05 2024-09-16T05:11:06.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163459+0000", 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2239999999999998e-06 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163535+0000", 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6019999999999994e-05 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163540+0000", 2024-09-16T05:11:06.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7740000000000004e-06 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163624+0000", 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4270999999999993e-05 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:58 2.2 e112/107 ondisk, result = 0)", 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.163606+0000", 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.196865189, 2024-09-16T05:11:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000177663, 2024-09-16T05:11:06.436 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.436 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.436 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.436 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163606+0000", 2024-09-16T05:11:06.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163606+0000", 2024-09-16T05:11:06.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163607+0000", 2024-09-16T05:11:06.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6599999999999996e-07 2024-09-16T05:11:06.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163611+0000", 2024-09-16T05:11:06.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2479999999999998e-06 2024-09-16T05:11:06.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163639+0000", 2024-09-16T05:11:06.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7943999999999999e-05 2024-09-16T05:11:06.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163644+0000", 2024-09-16T05:11:06.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0520000000000001e-06 2024-09-16T05:11:06.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163683+0000", 2024-09-16T05:11:06.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8733000000000003e-05 2024-09-16T05:11:06.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163689+0000", 2024-09-16T05:11:06.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7200000000000003e-06 2024-09-16T05:11:06.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163784+0000", 2024-09-16T05:11:06.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5400000000000001e-05 2024-09-16T05:11:06.445 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.445 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.445 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.446 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:59 2.2 e112/107 ondisk, result = 0)", 2024-09-16T05:11:06.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.163692+0000", 2024-09-16T05:11:06.446 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.196779514999999, 2024-09-16T05:11:06.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019399, 2024-09-16T05:11:06.446 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.446 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.447 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.447 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163692+0000", 2024-09-16T05:11:06.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163692+0000", 2024-09-16T05:11:06.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163693+0000", 2024-09-16T05:11:06.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1799999999999995e-07 2024-09-16T05:11:06.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163696+0000", 2024-09-16T05:11:06.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.247e-06 2024-09-16T05:11:06.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163714+0000", 2024-09-16T05:11:06.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8485e-05 2024-09-16T05:11:06.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163719+0000", 2024-09-16T05:11:06.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3100000000000002e-06 2024-09-16T05:11:06.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163807+0000", 2024-09-16T05:11:06.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8802999999999997e-05 2024-09-16T05:11:06.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163812+0000", 2024-09-16T05:11:06.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.369e-06 2024-09-16T05:11:06.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163886+0000", 2024-09-16T05:11:06.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4258000000000006e-05 2024-09-16T05:11:06.455 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:69 2.3 2:c018a787:::smithi20457400-18:head [write 685719~711939,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.170682003, 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002893503, 2024-09-16T05:11:06.456 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 69 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189792+0000", 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1299999999999999e-06 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190294+0000", 2024-09-16T05:11:06.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050236900000000001 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190307+0000", 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3315e-05 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190329+0000", 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2116999999999998e-05 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190394+0000", 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4177999999999994e-05 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190467+0000", 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3795999999999996e-05 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190551+0000", 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3119000000000002e-05 2024-09-16T05:11:06.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.191468+0000", 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091758400000000004 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192622+0000", 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011534869999999999 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192656+0000", 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4520000000000002e-05 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192683+0000", 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6888000000000001e-05 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:69 2.3 e112/107 ondisk, result = 0)", 2024-09-16T05:11:06.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.192550+0000", 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.167921270000001, 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000140853, 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192550+0000", 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192550+0000", 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192551+0000", 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8400000000000003e-07 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192554+0000", 2024-09-16T05:11:06.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8899999999999999e-06 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192574+0000", 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9870000000000001e-05 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192579+0000", 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1209999999999998e-06 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192614+0000", 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4977000000000001e-05 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192619+0000", 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7809999999999996e-06 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192691+0000", 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2329999999999994e-05 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.466 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:70 2.3 2:c018a787:::smithi20457400-18:head [write 1845135~440584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.194603+0000", 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.165868566, 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001788261, 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:06.467 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 70 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194603+0000", 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194603+0000", 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194605+0000", 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9570000000000001e-06 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194857+0000", 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025245399999999999 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194871+0000", 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3179999999999999e-05 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194877+0000", 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.212e-06 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194922+0000", 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5527e-05 2024-09-16T05:11:06.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194975+0000", 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2370999999999997e-05 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195052+0000", 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7354999999999994e-05 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195911+0000", 2024-09-16T05:11:06.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00085943599999999999 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196327+0000", 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041503200000000002 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196364+0000", 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196391+0000", 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.692e-05 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:71 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (56) in=7b,truncate 2285719] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.194889+0000", 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.165582126, 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016096109999999999, 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.473 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 71 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194889+0000", 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194889+0000", 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194890+0000", 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3099999999999998e-07 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194893+0000", 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7300000000000001e-06 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194897+0000", 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.189e-06 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194900+0000", 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.45e-06 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195512+0000", 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00061151900000000004 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195605+0000", 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3039000000000001e-05 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195681+0000", 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5918000000000001e-05 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195914+0000", 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00023352299999999999 2024-09-16T05:11:06.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196423+0000", 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050919699999999997 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196455+0000", 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1452000000000003e-05 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196499+0000", 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4162999999999997e-05 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:72 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.194919+0000", 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.165552253000001, 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001705549, 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194919+0000", 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194919+0000", 2024-09-16T05:11:06.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194920+0000", 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4400000000000001e-07 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194921+0000", 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7260000000000001e-06 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194928+0000", 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4370000000000004e-06 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194932+0000", 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0300000000000004e-06 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195850+0000", 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091856800000000003 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195900+0000", 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9851000000000001e-05 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196527+0000", 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062624600000000001 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196569+0000", 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2281999999999999e-05 2024-09-16T05:11:06.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196625+0000", 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6064999999999999e-05 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:70 2.3 e112/107 ondisk, result = 0)", 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.196223+0000", 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.164248615999998, 2024-09-16T05:11:06.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000175639, 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196223+0000", 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196223+0000", 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196224+0000", 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2500000000000005e-07 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196228+0000", 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.301e-06 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196266+0000", 2024-09-16T05:11:06.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7599999999999999e-05 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196271+0000", 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.259e-06 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196318+0000", 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7429999999999998e-05 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196323+0000", 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0760000000000002e-06 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196399+0000", 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5247999999999995e-05 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.488 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:71 2.3 e112/107 ondisk, result = 0)", 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.196231+0000", 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.164240581000001, 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028081900000000002, 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196231+0000", 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196231+0000", 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196231+0000", 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4000000000000001e-07 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196241+0000", 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7389999999999994e-06 2024-09-16T05:11:06.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196288+0000", 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7299999999999998e-05 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196292+0000", 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3790000000000002e-06 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196415+0000", 2024-09-16T05:11:06.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012321500000000001 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196420+0000", 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4600000000000002e-06 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196512+0000", 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1586e-05 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.492 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(1.8,GRANT,e:113,from:2,reservation_nonce:0)", 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:45.435522+0000", 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.924949409, 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030893499999999998, 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435522+0000", 2024-09-16T05:11:06.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435522+0000", 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435524+0000", 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4670000000000001e-06 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435533+0000", 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6129999999999993e-06 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435566+0000", 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2910999999999997e-05 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435575+0000", 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.02e-06 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435649+0000", 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3967999999999994e-05 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435666+0000", 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6657e-05 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.435831+0000", 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016529899999999999 2024-09-16T05:11:06.496 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(1.8 e109 from shard 2)", 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:45.436879+0000", 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.923592200000002, 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116542, 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:06.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436879+0000", 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436879+0000", 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:06.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436880+0000", 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9899999999999999e-07 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436890+0000", 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3589999999999997e-06 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436920+0000", 2024-09-16T05:11:06.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9992999999999999e-05 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436925+0000", 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4280000000000004e-06 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436962+0000", 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7270000000000001e-05 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436969+0000", 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1709999999999996e-06 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:45.436996+0000", 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6522000000000001e-05 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:06.501 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:11:06.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:06.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:06.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:06.641 INFO:tasks.rados.rados.2.smithi204.stdout:170: done (1 left) 2024-09-16T05:11:06.641 INFO:tasks.rados.rados.2.smithi204.stdout:172: done (0 left) 2024-09-16T05:11:06.641 INFO:tasks.rados.rados.2.smithi204.stdout:173: snap_create 2024-09-16T05:11:06.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:06.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:06.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:06.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:06.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:06.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:06.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:06.910 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:06.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:07.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:07.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:07.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:07.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:07.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:07.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:07.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:07.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:11:07.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:07.286 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:07.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:07.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:07.405 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:07.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:07.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:07.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:07.540 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:07.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:07.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:07.645 INFO:tasks.rados.rados.2.smithi204.stdout:173: done (0 left) 2024-09-16T05:11:07.645 INFO:tasks.rados.rados.2.smithi204.stdout:174: delete oid 23 current snap is 15 2024-09-16T05:11:07.647 INFO:tasks.rados.rados.2.smithi204.stdout:174: done (0 left) 2024-09-16T05:11:07.648 INFO:tasks.rados.rados.2.smithi204.stdout:175: write (excl) oid 44 current snap is 15 2024-09-16T05:11:07.648 INFO:tasks.rados.rados.2.smithi204.stdout:175: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-16T05:11:07.654 INFO:tasks.rados.rados.2.smithi204.stdout:175: writing smithi20457400-44 from 447086 to 1217724 tid 1 2024-09-16T05:11:07.662 INFO:tasks.rados.rados.2.smithi204.stdout:175: writing smithi20457400-44 from 1935000 to 2733016 tid 2 2024-09-16T05:11:07.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:07.668 INFO:tasks.rados.rados.2.smithi204.stdout:175: writing smithi20457400-44 from 3458512 to 3647086 tid 3 2024-09-16T05:11:07.668 INFO:tasks.rados.rados.2.smithi204.stdout:176: delete oid 2 current snap is 15 2024-09-16T05:11:07.668 INFO:tasks.rados.rados.2.smithi204.stdout:175: finishing write tid 1 to smithi20457400-44 2024-09-16T05:11:07.668 INFO:tasks.rados.rados.2.smithi204.stdout:175: finishing write tid 2 to smithi20457400-44 2024-09-16T05:11:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:07.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:07.670 INFO:tasks.rados.rados.2.smithi204.stdout:175: finishing write tid 3 to smithi20457400-44 2024-09-16T05:11:07.670 INFO:tasks.rados.rados.2.smithi204.stdout:175: finishing write tid 4 to smithi20457400-44 2024-09-16T05:11:07.670 INFO:tasks.rados.rados.2.smithi204.stdout:175: finishing write tid 5 to smithi20457400-44 2024-09-16T05:11:07.670 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 28 (ObjNum 98 snap 15 seq_num 98) dirty exists 2024-09-16T05:11:07.671 INFO:tasks.rados.rados.2.smithi204.stdout:175: left oid 44 (ObjNum 98 snap 15 seq_num 98) 2024-09-16T05:11:07.671 INFO:tasks.rados.rados.2.smithi204.stdout:175: done (1 left) 2024-09-16T05:11:07.671 INFO:tasks.rados.rados.2.smithi204.stdout:176: done (0 left) 2024-09-16T05:11:07.671 INFO:tasks.rados.rados.2.smithi204.stdout:177: rollback oid 48 current snap is 15 2024-09-16T05:11:07.671 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 48 to 7 2024-09-16T05:11:07.671 INFO:tasks.rados.rados.2.smithi204.stdout:178: snap_create 2024-09-16T05:11:07.672 INFO:tasks.rados.rados.2.smithi204.stdout:177: finishing rollback tid 0 to smithi20457400-48 2024-09-16T05:11:07.672 INFO:tasks.rados.rados.2.smithi204.stdout:177: finishing rollback tid 1 to smithi20457400-48 2024-09-16T05:11:07.672 INFO:tasks.rados.rados.2.smithi204.stdout:177: finishing rollback tid 2 to smithi20457400-48 2024-09-16T05:11:07.672 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 43 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T05:11:07.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:07.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:07.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:07.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:07.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:07 smithi032 ceph-mon[27388]: pgmap v354: 27 pgs: 27 active+clean; 127 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2024-09-16T05:11:07.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:07 smithi032 ceph-mon[27388]: osdmap e135: 12 total, 12 up, 12 in 2024-09-16T05:11:07.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:07 smithi204 ceph-mon[30933]: pgmap v354: 27 pgs: 27 active+clean; 127 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2024-09-16T05:11:07.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:07 smithi204 ceph-mon[30933]: osdmap e135: 12 total, 12 up, 12 in 2024-09-16T05:11:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:07.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:07.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:07.934 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:07 smithi196 ceph-mon[30802]: pgmap v354: 27 pgs: 27 active+clean; 127 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2024-09-16T05:11:07.934 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:07 smithi196 ceph-mon[30802]: osdmap e135: 12 total, 12 up, 12 in 2024-09-16T05:11:07.945 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:11:07.945 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:11:07.945 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:11:07.945 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:11:08.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:08.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:08.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:08.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:11:08.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:08.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:08.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:08.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:08.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:08.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:08.331 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:08.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:08.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:08.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:08.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:08.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:08.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:08.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:11:08.667 INFO:tasks.rados.rados.2.smithi204.stdout:177: done (1 left) 2024-09-16T05:11:08.667 INFO:tasks.rados.rados.2.smithi204.stdout:178: done (0 left) 2024-09-16T05:11:08.667 INFO:tasks.rados.rados.2.smithi204.stdout:179: write (excl) oid 11 current snap is 16 2024-09-16T05:11:08.667 INFO:tasks.rados.rados.2.smithi204.stdout:179: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-16T05:11:08.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:08.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:08.673 INFO:tasks.rados.rados.2.smithi204.stdout:179: writing smithi20457400-11 from 480700 to 1118189 tid 1 2024-09-16T05:11:08.678 INFO:tasks.rados.rados.2.smithi204.stdout:179: writing smithi20457400-11 from 1651964 to 2053649 tid 2 2024-09-16T05:11:08.687 INFO:tasks.rados.rados.2.smithi204.stdout:179: writing smithi20457400-11 from 2713358 to 3483971 tid 3 2024-09-16T05:11:08.687 INFO:tasks.rados.rados.2.smithi204.stdout:180: snap_remove snap 10 2024-09-16T05:11:08.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:08 smithi032 ceph-mon[27388]: osdmap e136: 12 total, 12 up, 12 in 2024-09-16T05:11:08.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:08.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:08.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:08.865 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:08.865 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:08.886 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:08.886 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:08.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:08.901 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:08.907 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:08.907 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:08.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:08 smithi204 ceph-mon[30933]: osdmap e136: 12 total, 12 up, 12 in 2024-09-16T05:11:08.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:08.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:08.927 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:08 smithi196 ceph-mon[30802]: osdmap e136: 12 total, 12 up, 12 in 2024-09-16T05:11:09.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:09.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:09.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:09.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:09.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:09.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:09.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:09.210 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:09.232 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:09.232 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:09.253 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:09.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:09.274 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:09.296 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:09.296 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:09.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:09.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:09.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:11:09.318 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:09.318 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:09.343 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:09.344 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:09.374 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:09.374 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:09.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:09.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:09.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:09.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:09.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:09.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:09.656 INFO:tasks.rados.rados.2.smithi204.stdout:180: done (1 left) 2024-09-16T05:11:09.657 INFO:tasks.rados.rados.2.smithi204.stdout:181: read oid 4 snap -1 2024-09-16T05:11:09.657 INFO:tasks.rados.rados.2.smithi204.stdout:181: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:11:09.657 INFO:tasks.rados.rados.2.smithi204.stdout:182: copy_from oid 2 from oid 18 current snap is 16 2024-09-16T05:11:09.657 INFO:tasks.rados.rados.2.smithi204.stdout:183: write oid 20 current snap is 16 2024-09-16T05:11:09.657 INFO:tasks.rados.rados.2.smithi204.stdout:183: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-16T05:11:09.663 INFO:tasks.rados.rados.2.smithi204.stdout:183: writing smithi20457400-20 from 497507 to 1210245 tid 1 2024-09-16T05:11:09.670 INFO:tasks.rados.rados.2.smithi204.stdout:183: writing smithi20457400-20 from 1710446 to 2372142 tid 2 2024-09-16T05:11:09.677 INFO:tasks.rados.rados.2.smithi204.stdout:183: writing smithi20457400-20 from 3140781 to 3697507 tid 3 2024-09-16T05:11:09.677 INFO:tasks.rados.rados.2.smithi204.stdout:184: snap_create 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:179: finishing write tid 1 to smithi20457400-11 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:179: finishing write tid 2 to smithi20457400-11 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:179: finishing write tid 3 to smithi20457400-11 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:179: finishing write tid 4 to smithi20457400-11 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:179: finishing write tid 5 to smithi20457400-11 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 60 (ObjNum 100 snap 16 seq_num 100) dirty exists 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:179: left oid 11 (ObjNum 100 snap 16 seq_num 100) 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:183: finishing write tid 1 to smithi20457400-20 2024-09-16T05:11:09.678 INFO:tasks.rados.rados.2.smithi204.stdout:181: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:11:09.679 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:09.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:09.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:09.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:09.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:182: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:182: finishing copy_from to smithi20457400-2 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:183: finishing write tid 2 to smithi20457400-20 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:183: finishing write tid 3 to smithi20457400-20 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:183: finishing write tid 4 to smithi20457400-20 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:183: finishing write tid 5 to smithi20457400-20 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 68 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-16T05:11:09.811 INFO:tasks.rados.rados.2.smithi204.stdout:183: left oid 20 (ObjNum 101 snap 16 seq_num 101) 2024-09-16T05:11:09.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:09.909 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:09 smithi032 ceph-mon[27388]: pgmap v357: 27 pgs: 27 active+clean; 130 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 397 KiB/s wr, 4 op/s 2024-09-16T05:11:09.909 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:09 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:09.909 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:09 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:09.909 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:09 smithi032 ceph-mon[27388]: osdmap e137: 12 total, 12 up, 12 in 2024-09-16T05:11:09.909 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/22564657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:09.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:09 smithi204 ceph-mon[30933]: pgmap v357: 27 pgs: 27 active+clean; 130 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 397 KiB/s wr, 4 op/s 2024-09-16T05:11:09.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:09 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:09.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:09 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:09.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:09 smithi204 ceph-mon[30933]: osdmap e137: 12 total, 12 up, 12 in 2024-09-16T05:11:09.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/22564657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:09.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:09.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:09.944 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:09 smithi196 ceph-mon[30802]: pgmap v357: 27 pgs: 27 active+clean; 130 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 397 KiB/s wr, 4 op/s 2024-09-16T05:11:09.944 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:09 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:09.944 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:09 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:09.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:09 smithi196 ceph-mon[30802]: osdmap e137: 12 total, 12 up, 12 in 2024-09-16T05:11:09.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/22564657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:09.958 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:09.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:11:10.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:10.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:10.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:10.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:10.254 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:10.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:10.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:10.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:10.428 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:10.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:10.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:10.552 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:10.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:10.637 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:179: done (4 left) 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:181: done (3 left) 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:182: done (2 left) 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:183: done (1 left) 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:184: done (0 left) 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:185: read oid 4 snap 12 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:185: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:186: write (excl) oid 13 current snap is 17 2024-09-16T05:11:10.641 INFO:tasks.rados.rados.2.smithi204.stdout:186: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-09-16T05:11:10.648 INFO:tasks.rados.rados.2.smithi204.stdout:186: writing smithi20457400-13 from 514314 to 1302301 tid 1 2024-09-16T05:11:10.652 INFO:tasks.rados.rados.2.smithi204.stdout:186: writing smithi20457400-13 from 1768928 to 2174282 tid 2 2024-09-16T05:11:10.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:10.659 INFO:tasks.rados.rados.2.smithi204.stdout:186: writing smithi20457400-13 from 2768204 to 3277714 tid 3 2024-09-16T05:11:10.659 INFO:tasks.rados.rados.2.smithi204.stdout:187: write (excl) oid 35 current snap is 17 2024-09-16T05:11:10.659 INFO:tasks.rados.rados.2.smithi204.stdout:187: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-16T05:11:10.664 INFO:tasks.rados.rados.2.smithi204.stdout:187: writing smithi20457400-35 from 531121 to 994357 tid 1 2024-09-16T05:11:10.672 INFO:tasks.rados.rados.2.smithi204.stdout:187: writing smithi20457400-35 from 1711057 to 2376422 tid 2 2024-09-16T05:11:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:10.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:10.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:10.680 INFO:tasks.rados.rados.2.smithi204.stdout:187: writing smithi20457400-35 from 3079274 to 3731121 tid 3 2024-09-16T05:11:10.680 INFO:tasks.rados.rados.2.smithi204.stdout:188: read oid 14 snap -1 2024-09-16T05:11:10.680 INFO:tasks.rados.rados.2.smithi204.stdout:188: expect (ObjNum 76 snap 6 seq_num 76) 2024-09-16T05:11:10.680 INFO:tasks.rados.rados.2.smithi204.stdout:189: rollback oid 39 current snap is 17 2024-09-16T05:11:10.680 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 39 to 7 2024-09-16T05:11:10.680 INFO:tasks.rados.rados.2.smithi204.stdout:185: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:11:10.718 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:10.718 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:10.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:10.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:10.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:10.813 INFO:tasks.rados.rados.2.smithi204.stdout:186: finishing write tid 1 to smithi20457400-13 2024-09-16T05:11:10.813 INFO:tasks.rados.rados.2.smithi204.stdout:186: finishing write tid 2 to smithi20457400-13 2024-09-16T05:11:10.813 INFO:tasks.rados.rados.2.smithi204.stdout:186: finishing write tid 3 to smithi20457400-13 2024-09-16T05:11:10.813 INFO:tasks.rados.rados.2.smithi204.stdout:185: done (4 left) 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:190: copy_from oid 40 from oid 36 current snap is 17 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:186: finishing write tid 4 to smithi20457400-13 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:186: finishing write tid 5 to smithi20457400-13 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 73 (ObjNum 102 snap 17 seq_num 102) dirty exists 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:186: left oid 13 (ObjNum 102 snap 17 seq_num 102) 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:187: finishing write tid 1 to smithi20457400-35 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:187: finishing write tid 2 to smithi20457400-35 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:189: finishing rollback tid 0 to smithi20457400-39 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:189: finishing rollback tid 1 to smithi20457400-39 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:189: finishing rollback tid 2 to smithi20457400-39 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 77 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:187: finishing write tid 3 to smithi20457400-35 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:187: finishing write tid 4 to smithi20457400-35 2024-09-16T05:11:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:187: finishing write tid 5 to smithi20457400-35 2024-09-16T05:11:10.815 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 54 (ObjNum 103 snap 17 seq_num 103) dirty exists 2024-09-16T05:11:10.815 INFO:tasks.rados.rados.2.smithi204.stdout:187: left oid 35 (ObjNum 103 snap 17 seq_num 103) 2024-09-16T05:11:10.815 INFO:tasks.rados.rados.2.smithi204.stdout:188: expect (ObjNum 76 snap 6 seq_num 76) 2024-09-16T05:11:10.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:10 smithi032 ceph-mon[27388]: osdmap e138: 12 total, 12 up, 12 in 2024-09-16T05:11:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4248215943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:10 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:10 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:11:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4248215943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:10 smithi032 ceph-mon[27388]: osdmap e139: 12 total, 12 up, 12 in 2024-09-16T05:11:10.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:10 smithi204 ceph-mon[30933]: osdmap e138: 12 total, 12 up, 12 in 2024-09-16T05:11:10.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4248215943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:10.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:10 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:10.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:10 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:11:10.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4248215943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:10.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:10 smithi204 ceph-mon[30933]: osdmap e139: 12 total, 12 up, 12 in 2024-09-16T05:11:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:10.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:10.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:10.949 INFO:tasks.rados.rados.2.smithi204.stdout:190: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:11:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:190: finishing copy_from to smithi20457400-40 2024-09-16T05:11:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 23 (ObjNum 55 snap 2 seq_num 55) dirty exists 2024-09-16T05:11:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:186: done (4 left) 2024-09-16T05:11:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:187: done (3 left) 2024-09-16T05:11:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:188: done (2 left) 2024-09-16T05:11:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:189: done (1 left) 2024-09-16T05:11:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:190: done (0 left) 2024-09-16T05:11:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:191: snap_create 2024-09-16T05:11:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:10 smithi196 ceph-mon[30802]: osdmap e138: 12 total, 12 up, 12 in 2024-09-16T05:11:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4248215943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:10 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:10 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:11:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4248215943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:10 smithi196 ceph-mon[30802]: osdmap e139: 12 total, 12 up, 12 in 2024-09-16T05:11:11.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:11.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:11.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:11.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:11.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:11.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:11.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:11.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:11:11.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:11.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:11.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:11.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:11.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:11.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:11.614 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:11.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:11.668 INFO:tasks.rados.rados.2.smithi204.stdout:191: done (0 left) 2024-09-16T05:11:11.669 INFO:tasks.rados.rados.2.smithi204.stdout:192: snap_remove snap 7 2024-09-16T05:11:11.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:11.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:11.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:11.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:11.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:11.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:11.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:11 smithi032 ceph-mon[27388]: pgmap v360: 27 pgs: 27 active+clean; 136 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-16T05:11:11.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:11 smithi204 ceph-mon[30933]: pgmap v360: 27 pgs: 27 active+clean; 136 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-16T05:11:11.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:11.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:11.945 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:11:11.945 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:11:11.945 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-16T05:11:11.945 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:11:11.945 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:11:11.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-16T05:11:11.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:11:11.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:11 smithi196 ceph-mon[30802]: pgmap v360: 27 pgs: 27 active+clean; 136 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-16T05:11:12.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:12.333 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:12.333 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:12.333 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:12.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:12.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:12.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:12.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:12.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:12.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:12.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:12.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:12.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:12.684 INFO:tasks.rados.rados.2.smithi204.stdout:192: done (0 left) 2024-09-16T05:11:12.684 INFO:tasks.rados.rados.2.smithi204.stdout:193: delete oid 23 current snap is 18 2024-09-16T05:11:12.685 INFO:tasks.rados.rados.2.smithi204.stdout:193: done (0 left) 2024-09-16T05:11:12.685 INFO:tasks.rados.rados.2.smithi204.stdout:194: write oid 38 current snap is 18 2024-09-16T05:11:12.685 INFO:tasks.rados.rados.2.smithi204.stdout:194: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-16T05:11:12.691 INFO:tasks.rados.rados.2.smithi204.stdout:194: writing smithi20457400-38 from 564735 to 1178469 tid 1 2024-09-16T05:11:12.698 INFO:tasks.rados.rados.2.smithi204.stdout:194: writing smithi20457400-38 from 1828021 to 2497055 tid 2 2024-09-16T05:11:12.699 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:12.706 INFO:tasks.rados.rados.2.smithi204.stdout:194: writing smithi20457400-38 from 3134120 to 3764735 tid 3 2024-09-16T05:11:12.706 INFO:tasks.rados.rados.2.smithi204.stdout:195: copy_from oid 31 from oid 14 current snap is 18 2024-09-16T05:11:12.707 INFO:tasks.rados.rados.2.smithi204.stdout:194: finishing write tid 1 to smithi20457400-38 2024-09-16T05:11:12.707 INFO:tasks.rados.rados.2.smithi204.stdout:194: finishing write tid 2 to smithi20457400-38 2024-09-16T05:11:12.707 INFO:tasks.rados.rados.2.smithi204.stdout:196: rollback oid 17 current snap is 18 2024-09-16T05:11:12.707 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 17 to 12 2024-09-16T05:11:12.707 INFO:tasks.rados.rados.2.smithi204.stdout:197: write (excl) oid 34 current snap is 18 2024-09-16T05:11:12.707 INFO:tasks.rados.rados.2.smithi204.stdout:197: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-16T05:11:12.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:12.713 INFO:tasks.rados.rados.2.smithi204.stdout:197: writing smithi20457400-34 from 581542 to 1270525 tid 1 2024-09-16T05:11:12.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:11:12.719 INFO:tasks.rados.rados.2.smithi204.stdout:197: writing smithi20457400-34 from 1886503 to 2299195 tid 2 2024-09-16T05:11:12.722 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:197: writing smithi20457400-34 from 2761543 to 3432494 tid 3 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:198: delete oid 27 current snap is 18 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:196: finishing rollback tid 1 to smithi20457400-17 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 129 snap 0 seq_num 2671885346) dirty dne 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:194: finishing write tid 3 to smithi20457400-38 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:194: finishing write tid 4 to smithi20457400-38 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:194: finishing write tid 5 to smithi20457400-38 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 92 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:194: left oid 38 (ObjNum 105 snap 18 seq_num 105) 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:197: finishing write tid 1 to smithi20457400-34 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:195: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:197: finishing write tid 2 to smithi20457400-34 2024-09-16T05:11:12.726 INFO:tasks.rados.rados.2.smithi204.stdout:195: finishing copy_from to smithi20457400-31 2024-09-16T05:11:12.727 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 48 (ObjNum 76 snap 6 seq_num 76) dirty exists 2024-09-16T05:11:12.728 INFO:tasks.rados.rados.2.smithi204.stdout:197: finishing write tid 3 to smithi20457400-34 2024-09-16T05:11:12.728 INFO:tasks.rados.rados.2.smithi204.stdout:197: finishing write tid 4 to smithi20457400-34 2024-09-16T05:11:12.728 INFO:tasks.rados.rados.2.smithi204.stdout:194: done (4 left) 2024-09-16T05:11:12.729 INFO:tasks.rados.rados.2.smithi204.stdout:195: done (3 left) 2024-09-16T05:11:12.729 INFO:tasks.rados.rados.2.smithi204.stdout:196: done (2 left) 2024-09-16T05:11:12.729 INFO:tasks.rados.rados.2.smithi204.stdout:198: done (1 left) 2024-09-16T05:11:12.729 INFO:tasks.rados.rados.2.smithi204.stdout:199: read oid 15 snap 17 2024-09-16T05:11:12.729 INFO:tasks.rados.rados.2.smithi204.stdout:199: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T05:11:12.729 INFO:tasks.rados.rados.2.smithi204.stdout:200: snap_remove snap 14 2024-09-16T05:11:12.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:12.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:12.739 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:12.743 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:12.743 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:12.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:12.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:12.857 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:12.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:12 smithi032 ceph-mon[27388]: osdmap e140: 12 total, 12 up, 12 in 2024-09-16T05:11:12.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:12 smithi204 ceph-mon[30933]: osdmap e140: 12 total, 12 up, 12 in 2024-09-16T05:11:12.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:12.965 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:12.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:12.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:12.986 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:12.987 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:12.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:12 smithi196 ceph-mon[30802]: osdmap e140: 12 total, 12 up, 12 in 2024-09-16T05:11:13.009 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:13.009 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:13.044 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:13.045 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:13.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:13.083 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:13.084 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:13.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:13.105 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:13.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:13.127 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:13.127 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:13.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:13.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:13.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:13.236 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:11:13.236 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T05:11:13.237 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:11:13.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:11:13.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:13.350 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:13.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:13.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:13.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:13.372 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:13.372 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:13.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:13.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:13.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:13.484 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:13.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:13.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:13.684 INFO:tasks.rados.rados.2.smithi204.stdout:200: done (2 left) 2024-09-16T05:11:13.684 INFO:tasks.rados.rados.2.smithi204.stdout:201: rollback oid 42 current snap is 18 2024-09-16T05:11:13.684 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 13 2024-09-16T05:11:13.684 INFO:tasks.rados.rados.2.smithi204.stdout:197: finishing write tid 5 to smithi20457400-34 2024-09-16T05:11:13.684 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 63 (ObjNum 106 snap 18 seq_num 106) dirty exists 2024-09-16T05:11:13.684 INFO:tasks.rados.rados.2.smithi204.stdout:197: left oid 34 (ObjNum 106 snap 18 seq_num 106) 2024-09-16T05:11:13.684 INFO:tasks.rados.rados.2.smithi204.stdout:199: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T05:11:13.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:13.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:13.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:13.759 INFO:tasks.rados.rados.2.smithi204.stdout:201: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:11:13.759 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:11:13.759 INFO:tasks.rados.rados.2.smithi204.stdout:197: done (2 left) 2024-09-16T05:11:13.759 INFO:tasks.rados.rados.2.smithi204.stdout:199: done (1 left) 2024-09-16T05:11:13.760 INFO:tasks.rados.rados.2.smithi204.stdout:201: done (0 left) 2024-09-16T05:11:13.760 INFO:tasks.rados.rados.2.smithi204.stdout:202: delete oid 36 current snap is 18 2024-09-16T05:11:13.761 INFO:tasks.rados.rados.2.smithi204.stdout:202: done (0 left) 2024-09-16T05:11:13.761 INFO:tasks.rados.rados.2.smithi204.stdout:203: write oid 36 current snap is 18 2024-09-16T05:11:13.761 INFO:tasks.rados.rados.2.smithi204.stdout:203: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-09-16T05:11:13.768 INFO:tasks.rados.rados.2.smithi204.stdout:203: writing smithi20457400-36 from 631963 to 1263046 tid 1 2024-09-16T05:11:13.775 INFO:tasks.rados.rados.2.smithi204.stdout:203: writing smithi20457400-36 from 2061949 to 2621968 tid 2 2024-09-16T05:11:13.782 INFO:tasks.rados.rados.2.smithi204.stdout:203: writing smithi20457400-36 from 3243812 to 3748579 tid 3 2024-09-16T05:11:13.782 INFO:tasks.rados.rados.2.smithi204.stdout:204: copy_from oid 34 from oid 47 current snap is 18 2024-09-16T05:11:13.782 INFO:tasks.rados.rados.2.smithi204.stdout:203: finishing write tid 1 to smithi20457400-36 2024-09-16T05:11:13.782 INFO:tasks.rados.rados.2.smithi204.stdout:203: finishing write tid 2 to smithi20457400-36 2024-09-16T05:11:13.782 INFO:tasks.rados.rados.2.smithi204.stdout:205: write oid 45 current snap is 18 2024-09-16T05:11:13.782 INFO:tasks.rados.rados.2.smithi204.stdout:205: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-16T05:11:13.789 INFO:tasks.rados.rados.2.smithi204.stdout:205: writing smithi20457400-45 from 648770 to 1355102 tid 1 2024-09-16T05:11:13.796 INFO:tasks.rados.rados.2.smithi204.stdout:205: writing smithi20457400-45 from 2120431 to 2540461 tid 2 2024-09-16T05:11:13.803 INFO:tasks.rados.rados.2.smithi204.stdout:205: writing smithi20457400-45 from 3271235 to 3703627 tid 3 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:206: read oid 11 snap -1 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:206: expect (ObjNum 100 snap 16 seq_num 100) 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:207: copy_from oid 10 from oid 23 current snap is 18 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:203: finishing write tid 3 to smithi20457400-36 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:203: finishing write tid 4 to smithi20457400-36 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:203: finishing write tid 5 to smithi20457400-36 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 73 (ObjNum 109 snap 18 seq_num 109) dirty exists 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:203: left oid 36 (ObjNum 109 snap 18 seq_num 109) 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:204: finishing copy_from to smithi20457400-34 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:204: got expected ENOENT (src dne) 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:204: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:11:13.804 INFO:tasks.rados.rados.2.smithi204.stdout:205: finishing write tid 1 to smithi20457400-45 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:203: done (4 left) 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:204: done (3 left) 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:208: copy_from oid 13 from oid 49 current snap is 18 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:205: finishing write tid 2 to smithi20457400-45 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:209: read oid 38 snap -1 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:209: expect (ObjNum 105 snap 18 seq_num 105) 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:210: rollback oid 42 current snap is 18 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 13 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:211: copy_from oid 33 from oid 5 current snap is 18 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:212: copy_from oid 43 from oid 9 current snap is 18 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:213: snap_create 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:210: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:11:13.805 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:11:13.806 INFO:tasks.rados.rados.2.smithi204.stdout:205: finishing write tid 3 to smithi20457400-45 2024-09-16T05:11:13.806 INFO:tasks.rados.rados.2.smithi204.stdout:205: finishing write tid 4 to smithi20457400-45 2024-09-16T05:11:13.806 INFO:tasks.rados.rados.2.smithi204.stdout:205: finishing write tid 5 to smithi20457400-45 2024-09-16T05:11:13.806 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 33 (ObjNum 110 snap 18 seq_num 110) dirty exists 2024-09-16T05:11:13.807 INFO:tasks.rados.rados.2.smithi204.stdout:205: left oid 45 (ObjNum 110 snap 18 seq_num 110) 2024-09-16T05:11:13.809 INFO:tasks.rados.rados.2.smithi204.stdout:206: expect (ObjNum 100 snap 16 seq_num 100) 2024-09-16T05:11:13.826 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:13.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:13.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:13.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:13.958 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:13 smithi032 ceph-mon[27388]: pgmap v363: 27 pgs: 27 active+clean; 141 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2024-09-16T05:11:13.958 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3236736093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:13.958 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:13 smithi032 ceph-mon[27388]: osdmap e141: 12 total, 12 up, 12 in 2024-09-16T05:11:13.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:13 smithi196 ceph-mon[30802]: pgmap v363: 27 pgs: 27 active+clean; 141 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2024-09-16T05:11:13.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3236736093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:13.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:13 smithi196 ceph-mon[30802]: osdmap e141: 12 total, 12 up, 12 in 2024-09-16T05:11:13.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:13.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:13.983 INFO:tasks.rados.rados.2.smithi204.stdout:207: finishing copy_from to smithi20457400-10 2024-09-16T05:11:13.983 INFO:tasks.rados.rados.2.smithi204.stdout:207: got expected ENOENT (src dne) 2024-09-16T05:11:13.983 INFO:tasks.rados.rados.2.smithi204.stdout:207: finishing copy_from racing read to smithi20457400-10 2024-09-16T05:11:13.983 INFO:tasks.rados.rados.2.smithi204.stdout:212: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:11:13.983 INFO:tasks.rados.rados.2.smithi204.stdout:209: expect (ObjNum 105 snap 18 seq_num 105) 2024-09-16T05:11:14.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:14.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:14.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:14.107 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:14.141 INFO:tasks.rados.rados.2.smithi204.stdout:208: finishing copy_from to smithi20457400-13 2024-09-16T05:11:14.141 INFO:tasks.rados.rados.2.smithi204.stdout:208: got expected ENOENT (src dne) 2024-09-16T05:11:14.141 INFO:tasks.rados.rados.2.smithi204.stdout:208: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:11:14.141 INFO:tasks.rados.rados.2.smithi204.stdout:212: finishing copy_from to smithi20457400-43 2024-09-16T05:11:14.141 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T05:11:14.141 INFO:tasks.rados.rados.2.smithi204.stdout:211: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:11:14.141 INFO:tasks.rados.rados.2.smithi204.stdout:211: finishing copy_from to smithi20457400-33 2024-09-16T05:11:14.141 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 20 (ObjNum 63 snap 3 seq_num 63) dirty exists 2024-09-16T05:11:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:13 smithi204 ceph-mon[30933]: pgmap v363: 27 pgs: 27 active+clean; 141 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.1 MiB/s wr, 9 op/s 2024-09-16T05:11:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3236736093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:13 smithi204 ceph-mon[30933]: osdmap e141: 12 total, 12 up, 12 in 2024-09-16T05:11:14.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:14.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:14.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:14.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:14.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:14.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:14.458 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:11:14.458 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:11:14.458 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:11:14.458 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:11:14.458 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:11:14.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:14.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:14.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:14.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:14.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:14.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:14.685 INFO:tasks.rados.rados.2.smithi204.stdout:205: done (8 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:206: done (7 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:207: done (6 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:208: done (5 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:209: done (4 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:210: done (3 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:211: done (2 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:212: done (1 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:213: done (0 left) 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:214: read oid 25 snap -1 2024-09-16T05:11:14.686 INFO:tasks.rados.rados.2.smithi204.stdout:214: expect deleted 2024-09-16T05:11:14.687 INFO:tasks.rados.rados.2.smithi204.stdout:215: snap_remove snap 12 2024-09-16T05:11:14.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:14.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:14.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:14.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:14.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:14.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: osdmap e142: 12 total, 12 up, 12 in 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872389664879905]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08736669557508858]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08838852827187324]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08736669557508858]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872389664879905]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08711123740089242]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:14.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08707474337600725]}]: dispatch 2024-09-16T05:11:14.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08711123740089242]}]: dispatch 2024-09-16T05:11:14.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:14.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:14.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:14.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: osdmap e142: 12 total, 12 up, 12 in 2024-09-16T05:11:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872389664879905]}]: dispatch 2024-09-16T05:11:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08736669557508858]}]: dispatch 2024-09-16T05:11:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08838852827187324]}]: dispatch 2024-09-16T05:11:14.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08736669557508858]}]: dispatch 2024-09-16T05:11:14.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872389664879905]}]: dispatch 2024-09-16T05:11:14.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08711123740089242]}]: dispatch 2024-09-16T05:11:14.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:14.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08707474337600725]}]: dispatch 2024-09-16T05:11:14.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08711123740089242]}]: dispatch 2024-09-16T05:11:14.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:15.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:15.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:15.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: osdmap e142: 12 total, 12 up, 12 in 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872389664879905]}]: dispatch 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08736669557508858]}]: dispatch 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08838852827187324]}]: dispatch 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08736669557508858]}]: dispatch 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872389664879905]}]: dispatch 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08711123740089242]}]: dispatch 2024-09-16T05:11:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08707474337600725]}]: dispatch 2024-09-16T05:11:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08711123740089242]}]: dispatch 2024-09-16T05:11:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716232903573165]}]: dispatch 2024-09-16T05:11:15.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:15.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:15.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:15.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:11:15.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:15.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:15.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:15.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:15.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:15.522 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:11:15.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:15.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:15.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:11:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:215: done (1 left) 2024-09-16T05:11:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:216: copy_from oid 3 from oid 4 current snap is 19 2024-09-16T05:11:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:214: done (1 left) 2024-09-16T05:11:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:217: write oid 44 current snap is 19 2024-09-16T05:11:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:217: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-16T05:11:15.650 INFO:tasks.rados.rados.2.smithi204.stdout:217: writing smithi20457400-44 from 665577 to 1447158 tid 1 2024-09-16T05:11:15.657 INFO:tasks.rados.rados.2.smithi204.stdout:217: writing smithi20457400-44 from 2062560 to 2626248 tid 2 2024-09-16T05:11:15.664 INFO:tasks.rados.rados.2.smithi204.stdout:217: writing smithi20457400-44 from 3182305 to 3825969 tid 3 2024-09-16T05:11:15.664 INFO:tasks.rados.rados.2.smithi204.stdout:218: snap_create 2024-09-16T05:11:15.664 INFO:tasks.rados.rados.2.smithi204.stdout:216: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:11:15.664 INFO:tasks.rados.rados.2.smithi204.stdout:217: finishing write tid 1 to smithi20457400-44 2024-09-16T05:11:15.664 INFO:tasks.rados.rados.2.smithi204.stdout:216: finishing copy_from to smithi20457400-3 2024-09-16T05:11:15.664 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 22 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-09-16T05:11:15.664 INFO:tasks.rados.rados.2.smithi204.stdout:217: finishing write tid 2 to smithi20457400-44 2024-09-16T05:11:15.666 INFO:tasks.rados.rados.2.smithi204.stdout:217: finishing write tid 3 to smithi20457400-44 2024-09-16T05:11:15.666 INFO:tasks.rados.rados.2.smithi204.stdout:217: finishing write tid 4 to smithi20457400-44 2024-09-16T05:11:15.666 INFO:tasks.rados.rados.2.smithi204.stdout:217: finishing write tid 5 to smithi20457400-44 2024-09-16T05:11:15.666 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 40 (ObjNum 111 snap 19 seq_num 111) dirty exists 2024-09-16T05:11:15.666 INFO:tasks.rados.rados.2.smithi204.stdout:217: left oid 44 (ObjNum 111 snap 19 seq_num 111) 2024-09-16T05:11:15.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:15.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:15.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:15.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:15.830 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:15.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:15.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: pgmap v366: 27 pgs: 27 active+clean; 152 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-16T05:11:15.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:15.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:15.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872389664879905]}]': finished 2024-09-16T05:11:15.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:15.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:15.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08736669557508858]}]': finished 2024-09-16T05:11:15.901 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08838852827187324]}]': finished 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08736669557508858]}]': finished 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872389664879905]}]': finished 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08711123740089242]}]': finished 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08707474337600725]}]': finished 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08711123740089242]}]': finished 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: osdmap e143: 12 total, 12 up, 12 in 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2035305225' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:11:15.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:11:15.903 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:15.903 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2035305225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:15.903 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:15 smithi196 ceph-mon[30802]: osdmap e144: 12 total, 12 up, 12 in 2024-09-16T05:11:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: pgmap v366: 27 pgs: 27 active+clean; 152 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872389664879905]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08736669557508858]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08838852827187324]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08736669557508858]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872389664879905]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08711123740089242]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:15.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08707474337600725]}]': finished 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08711123740089242]}]': finished 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: osdmap e143: 12 total, 12 up, 12 in 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2035305225' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2035305225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:15.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:15 smithi032 ceph-mon[27388]: osdmap e144: 12 total, 12 up, 12 in 2024-09-16T05:11:15.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:15.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:15.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:15.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:11:16.001 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:11:16.001 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:11:16.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:11:16.001 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:11:16.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.001 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1 2.8 2:12e92f6c:::smithi20457400-1:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:42.993477+0000", 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 33.005500275999999, 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041395950000000003, 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.993477+0000", 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.993477+0000", 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.993479+0000", 2024-09-16T05:11:16.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6470000000000001e-06 2024-09-16T05:11:16.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994189+0000", 2024-09-16T05:11:16.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00071080099999999995 2024-09-16T05:11:16.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994203+0000", 2024-09-16T05:11:16.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.375e-05 2024-09-16T05:11:16.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994211+0000", 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9140000000000008e-06 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994291+0000", 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9931000000000004e-05 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994390+0000", 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9234999999999994e-05 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994507+0000", 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011694200000000001 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997077+0000", 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025702979999999999 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997514+0000", 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00043703900000000001 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997568+0000", 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3687000000000003e-05 2024-09-16T05:11:16.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997616+0000", 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8350999999999998e-05 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.009 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2 2.8 2:12e92f6c:::smithi20457400-1:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:42.994842+0000", 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 33.004134788999998, 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028772350000000001, 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2 2024-09-16T05:11:16.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.011 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994842+0000", 2024-09-16T05:11:16.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994842+0000", 2024-09-16T05:11:16.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994844+0000", 2024-09-16T05:11:16.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.407e-06 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995544+0000", 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00070042100000000001 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995559+0000", 2024-09-16T05:11:16.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5140000000000001e-05 2024-09-16T05:11:16.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995570+0000", 2024-09-16T05:11:16.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0453e-05 2024-09-16T05:11:16.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995635+0000", 2024-09-16T05:11:16.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5143999999999998e-05 2024-09-16T05:11:16.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995713+0000", 2024-09-16T05:11:16.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8573999999999997e-05 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995835+0000", 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121388 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997080+0000", 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001245468 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997656+0000", 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057615900000000002 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997691+0000", 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4520000000000002e-05 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997720+0000", 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8561e-05 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3 2.8 2:12e92f6c:::smithi20457400-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:42.995596+0000", 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 33.003380851999999, 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022823549999999998, 2024-09-16T05:11:16.019 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995596+0000", 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995596+0000", 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995597+0000", 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7300000000000004e-07 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995602+0000", 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7360000000000001e-06 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995612+0000", 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5389999999999996e-06 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995618+0000", 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5889999999999999e-06 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996445+0000", 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082694400000000003 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996496+0000", 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0559000000000002e-05 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996638+0000", 2024-09-16T05:11:16.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014271800000000001 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997143+0000", 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050490099999999998 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997771+0000", 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062767599999999997 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997827+0000", 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6069999999999997e-05 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997879+0000", 2024-09-16T05:11:16.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1650000000000002e-05 2024-09-16T05:11:16.026 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.026 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.026 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.026 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4 2.8 2:12e92f6c:::smithi20457400-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2024-09-16T05:11:16.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:42.995645+0000", 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 33.003332452999999, 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023934989999999999, 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995645+0000", 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995645+0000", 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995645+0000", 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7899999999999997e-07 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995649+0000", 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1999999999999999e-06 2024-09-16T05:11:16.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995658+0000", 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7040000000000006e-06 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995665+0000", 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4470000000000003e-06 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996856+0000", 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011909139999999999 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996933+0000", 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6829999999999995e-05 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997932+0000", 2024-09-16T05:11:16.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099908099999999997 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997979+0000", 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7485999999999999e-05 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.998038+0000", 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9058000000000003e-05 2024-09-16T05:11:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:21 2.0 2:06458cf1:::smithi20457400-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.047722+0000", 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.951254912000003, 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065493820000000003, 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.033 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.034 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.034 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 21 2024-09-16T05:11:16.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.034 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.047722+0000", 2024-09-16T05:11:16.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.047722+0000", 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.047723+0000", 2024-09-16T05:11:16.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.232e-06 2024-09-16T05:11:16.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048355+0000", 2024-09-16T05:11:16.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063158099999999998 2024-09-16T05:11:16.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048369+0000", 2024-09-16T05:11:16.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.418e-05 2024-09-16T05:11:16.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048377+0000", 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6690000000000006e-06 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048466+0000", 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8831000000000004e-05 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048543+0000", 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6995000000000004e-05 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048632+0000", 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9104000000000006e-05 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.050003+0000", 2024-09-16T05:11:16.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001370916 2024-09-16T05:11:16.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.041 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054207+0000", 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042047680000000002 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054241+0000", 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3565000000000001e-05 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054272+0000", 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0540999999999998e-05 2024-09-16T05:11:16.042 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:22 2.0 2:06458cf1:::smithi20457400-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052091+0000", 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.946886153999998, 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002289555, 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.043 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 22 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052091+0000", 2024-09-16T05:11:16.044 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052091+0000", 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052092+0000", 2024-09-16T05:11:16.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6099999999999999e-07 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052159+0000", 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7067e-05 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052169+0000", 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0241000000000001e-05 2024-09-16T05:11:16.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052174+0000", 2024-09-16T05:11:16.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8180000000000003e-06 2024-09-16T05:11:16.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052260+0000", 2024-09-16T05:11:16.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5952000000000006e-05 2024-09-16T05:11:16.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.048 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052315+0000", 2024-09-16T05:11:16.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4678000000000001e-05 2024-09-16T05:11:16.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.049 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:11:16.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052400+0000", 2024-09-16T05:11:16.049 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4915999999999998e-05 2024-09-16T05:11:16.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053195+0000", 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079554599999999995 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054316+0000", 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011208749999999999 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054352+0000", 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5837999999999998e-05 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054381+0000", 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8663e-05 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.051 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:23 2.0 2:06458cf1:::smithi20457400-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.946791607999998, 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103108687, 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 23 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:11:16.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3599999999999999e-07 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052188+0000", 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4169999999999999e-06 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052214+0000", 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5327e-05 2024-09-16T05:11:16.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052217+0000", 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1810000000000001e-06 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052730+0000", 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051270700000000005 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052818+0000", 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8641999999999996e-05 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052937+0000", 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119009 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053259+0000", 2024-09-16T05:11:16.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032151800000000002 2024-09-16T05:11:16.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155207+0000", 2024-09-16T05:11:16.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101948155 2024-09-16T05:11:16.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155248+0000", 2024-09-16T05:11:16.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1115999999999997e-05 2024-09-16T05:11:16.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155294+0000", 2024-09-16T05:11:16.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6179e-05 2024-09-16T05:11:16.060 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.060 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.060 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.061 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:24 2.0 2:06458cf1:::smithi20457400-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:16.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:11:16.061 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.946755547000002, 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103238884, 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 24 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:11:16.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:11:16.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:11:16.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8799999999999998e-07 2024-09-16T05:11:16.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052223+0000", 2024-09-16T05:11:16.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2950000000000001e-06 2024-09-16T05:11:16.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052227+0000", 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5240000000000001e-06 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052230+0000", 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2160000000000002e-06 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053127+0000", 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089734199999999997 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053182+0000", 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4935999999999997e-05 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155331+0000", 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.102149077 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155389+0000", 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7691e-05 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155460+0000", 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1514999999999995e-05 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:29 2.9 e112/107 2:94d33c56:::smithi20457400-8:head v 112'1, mlcod=0'0)", 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.070372+0000", 2024-09-16T05:11:16.069 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.928605439999998, 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001444477, 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.070372+0000", 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.070372+0000", 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.070372+0000", 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-07 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.071005+0000", 2024-09-16T05:11:16.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063298200000000003 2024-09-16T05:11:16.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.071032+0000", 2024-09-16T05:11:16.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6933000000000001e-05 2024-09-16T05:11:16.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.071039+0000", 2024-09-16T05:11:16.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5679999999999996e-06 2024-09-16T05:11:16.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.071092+0000", 2024-09-16T05:11:16.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2947e-05 2024-09-16T05:11:16.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.071108+0000", 2024-09-16T05:11:16.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5994999999999998e-05 2024-09-16T05:11:16.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.071775+0000", 2024-09-16T05:11:16.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00066748100000000004 2024-09-16T05:11:16.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.071816+0000", 2024-09-16T05:11:16.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1057999999999999e-05 2024-09-16T05:11:16.078 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.078 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.078 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.079 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:33 2.9 e112/107 2:9662c29c:::smithi20457400-9:head v 112'4, mlcod=112'2)", 2024-09-16T05:11:16.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.082840+0000", 2024-09-16T05:11:16.079 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.916137405999997, 2024-09-16T05:11:16.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001547559, 2024-09-16T05:11:16.079 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.079 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.080 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.080 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.082840+0000", 2024-09-16T05:11:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.082840+0000", 2024-09-16T05:11:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.082840+0000", 2024-09-16T05:11:16.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1999999999999999e-07 2024-09-16T05:11:16.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.083517+0000", 2024-09-16T05:11:16.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00067679700000000003 2024-09-16T05:11:16.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.083545+0000", 2024-09-16T05:11:16.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7518999999999999e-05 2024-09-16T05:11:16.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.083550+0000", 2024-09-16T05:11:16.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4500000000000003e-06 2024-09-16T05:11:16.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.083585+0000", 2024-09-16T05:11:16.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4526000000000001e-05 2024-09-16T05:11:16.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.083596+0000", 2024-09-16T05:11:16.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1408e-05 2024-09-16T05:11:16.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.084348+0000", 2024-09-16T05:11:16.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075188100000000003 2024-09-16T05:11:16.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.084387+0000", 2024-09-16T05:11:16.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9258000000000003e-05 2024-09-16T05:11:16.087 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.088 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.088 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.088 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:37 2.9 e112/107 2:96dc17ef:::smithi20457400-10:head v 112'7, mlcod=112'5)", 2024-09-16T05:11:16.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.095120+0000", 2024-09-16T05:11:16.089 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.903857582999997, 2024-09-16T05:11:16.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016065299999999999, 2024-09-16T05:11:16.089 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.089 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.089 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.089 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.095120+0000", 2024-09-16T05:11:16.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.095120+0000", 2024-09-16T05:11:16.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.095120+0000", 2024-09-16T05:11:16.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0400000000000005e-07 2024-09-16T05:11:16.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.095808+0000", 2024-09-16T05:11:16.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00068800600000000001 2024-09-16T05:11:16.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.095838+0000", 2024-09-16T05:11:16.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9079000000000001e-05 2024-09-16T05:11:16.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.095844+0000", 2024-09-16T05:11:16.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0159999999999997e-06 2024-09-16T05:11:16.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.095892+0000", 2024-09-16T05:11:16.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8653999999999999e-05 2024-09-16T05:11:16.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.095907+0000", 2024-09-16T05:11:16.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4637999999999999e-05 2024-09-16T05:11:16.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.096689+0000", 2024-09-16T05:11:16.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00078241200000000002 2024-09-16T05:11:16.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.096726+0000", 2024-09-16T05:11:16.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6820999999999999e-05 2024-09-16T05:11:16.098 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.098 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.098 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.098 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:105 2.8 2:1030fb5b:::smithi20457400-27:head [write 436982~705533,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:16.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.301386+0000", 2024-09-16T05:11:16.098 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.697590873999999, 2024-09-16T05:11:16.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030539830000000001, 2024-09-16T05:11:16.099 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.099 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.099 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.099 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.099 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.100 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 105 2024-09-16T05:11:16.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.100 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301386+0000", 2024-09-16T05:11:16.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301386+0000", 2024-09-16T05:11:16.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301389+0000", 2024-09-16T05:11:16.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1370000000000002e-06 2024-09-16T05:11:16.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301943+0000", 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055360100000000003 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301963+0000", 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0398999999999999e-05 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301971+0000", 2024-09-16T05:11:16.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0700000000000007e-06 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.302040+0000", 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8230000000000002e-05 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.302155+0000", 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114864 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.302262+0000", 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010745 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.303571+0000", 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013085670000000001 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.304372+0000", 2024-09-16T05:11:16.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080121800000000003 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.304407+0000", 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4842000000000003e-05 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.304440+0000", 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3605000000000001e-05 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:106 2.8 2:1030fb5b:::smithi20457400-27:head [write 1855120~581862] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.308375+0000", 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.690602241000001, 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002390011, 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 106 2024-09-16T05:11:16.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308375+0000", 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308375+0000", 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308376+0000", 2024-09-16T05:11:16.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2619999999999999e-06 2024-09-16T05:11:16.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308837+0000", 2024-09-16T05:11:16.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046054599999999999 2024-09-16T05:11:16.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308851+0000", 2024-09-16T05:11:16.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3913999999999999e-05 2024-09-16T05:11:16.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308857+0000", 2024-09-16T05:11:16.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1979999999999998e-06 2024-09-16T05:11:16.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308912+0000", 2024-09-16T05:11:16.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.541e-05 2024-09-16T05:11:16.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308968+0000", 2024-09-16T05:11:16.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5939000000000001e-05 2024-09-16T05:11:16.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:11:16.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309054+0000", 2024-09-16T05:11:16.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6149000000000005e-05 2024-09-16T05:11:16.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309844+0000", 2024-09-16T05:11:16.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00078919799999999996 2024-09-16T05:11:16.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310692+0000", 2024-09-16T05:11:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084889399999999999 2024-09-16T05:11:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310728+0000", 2024-09-16T05:11:16.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5842e-05 2024-09-16T05:11:16.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310765+0000", 2024-09-16T05:11:16.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6659000000000003e-05 2024-09-16T05:11:16.117 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.117 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.117 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.117 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:107 2.8 2:1030fb5b:::smithi20457400-27:head [setxattr _header (56) in=7b,truncate 2436982] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:16.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.308872+0000", 2024-09-16T05:11:16.117 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.690104773000002, 2024-09-16T05:11:16.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020057790000000001, 2024-09-16T05:11:16.118 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.118 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.118 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.118 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.118 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.118 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.118 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 107 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308872+0000", 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308872+0000", 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308873+0000", 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9899999999999999e-07 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308879+0000", 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3639999999999998e-06 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308896+0000", 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7422e-05 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308904+0000", 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8390000000000007e-06 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309514+0000", 2024-09-16T05:11:16.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000610298 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309582+0000", 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7837999999999997e-05 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309683+0000", 2024-09-16T05:11:16.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010155299999999999 2024-09-16T05:11:16.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309981+0000", 2024-09-16T05:11:16.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029724500000000002 2024-09-16T05:11:16.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310807+0000", 2024-09-16T05:11:16.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082646200000000003 2024-09-16T05:11:16.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310841+0000", 2024-09-16T05:11:16.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3793e-05 2024-09-16T05:11:16.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310878+0000", 2024-09-16T05:11:16.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7166000000000003e-05 2024-09-16T05:11:16.126 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.126 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.126 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.127 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:108 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:16.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.308916+0000", 2024-09-16T05:11:16.127 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.690061303, 2024-09-16T05:11:16.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021015669999999999, 2024-09-16T05:11:16.127 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.128 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.128 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:16.128 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.128 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.128 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.129 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108 2024-09-16T05:11:16.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.129 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308916+0000", 2024-09-16T05:11:16.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308916+0000", 2024-09-16T05:11:16.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308916+0000", 2024-09-16T05:11:16.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3399999999999999e-07 2024-09-16T05:11:16.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308920+0000", 2024-09-16T05:11:16.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.343e-06 2024-09-16T05:11:16.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308928+0000", 2024-09-16T05:11:16.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9459999999999998e-06 2024-09-16T05:11:16.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308934+0000", 2024-09-16T05:11:16.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1310000000000003e-06 2024-09-16T05:11:16.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309871+0000", 2024-09-16T05:11:16.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00093735499999999996 2024-09-16T05:11:16.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:16.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309921+0000", 2024-09-16T05:11:16.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9323999999999997e-05 2024-09-16T05:11:16.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310907+0000", 2024-09-16T05:11:16.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0009860750000000001 2024-09-16T05:11:16.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310954+0000", 2024-09-16T05:11:16.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7760999999999999e-05 2024-09-16T05:11:16.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.311017+0000", 2024-09-16T05:11:16.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2997999999999996e-05 2024-09-16T05:11:16.139 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.139 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.139 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.140 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:161 2.9 e112/107 2:9423b7dc:::smithi20457400-41:head v 112'10, mlcod=112'8)", 2024-09-16T05:11:16.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.487995+0000", 2024-09-16T05:11:16.140 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.510982351000003, 2024-09-16T05:11:16.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019256189999999999, 2024-09-16T05:11:16.140 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.140 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.140 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.141 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.487995+0000", 2024-09-16T05:11:16.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.487995+0000", 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.488002+0000", 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6820000000000003e-06 2024-09-16T05:11:16.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.488711+0000", 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00070838100000000001 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.488779+0000", 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8589000000000003e-05 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.488788+0000", 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9789999999999999e-06 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.488859+0000", 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0531000000000006e-05 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.488878+0000", 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9504000000000002e-05 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.489873+0000", 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099483300000000009 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.489920+0000", 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7120000000000003e-05 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.145 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:391 2.8 2:1030fb5b:::smithi20457400-27:head [copy-from ver 13] snapc 8=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.362190226999999, 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021838053999999999, 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.146 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 391 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636789+0000", 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3829999999999999e-06 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636801+0000", 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1459e-05 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636822+0000", 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1719e-05 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636833+0000", 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0366e-05 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636912+0000", 2024-09-16T05:11:16.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9060999999999994e-05 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636918+0000", 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0900000000000001e-06 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638224+0000", 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00130613 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638314+0000", 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9771000000000005e-05 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.652220+0000", 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013906112 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.656458+0000", 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004237673 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658553+0000", 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002095609 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658592+0000", 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8627000000000001e-05 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658625+0000", 2024-09-16T05:11:16.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3053999999999999e-05 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:392 2.8 2:1030fb5b:::smithi20457400-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.362119005, 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019221338000000001, 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:11:16.153 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 392 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636859+0000", 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0699999999999998e-07 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636861+0000", 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5129999999999999e-06 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636867+0000", 2024-09-16T05:11:16.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6030000000000004e-06 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636870+0000", 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.551e-06 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636972+0000", 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010187999999999999 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:16.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636978+0000", 2024-09-16T05:11:16.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6269999999999997e-06 2024-09-16T05:11:16.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638477+0000", 2024-09-16T05:11:16.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014995049999999999 2024-09-16T05:11:16.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638531+0000", 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3387999999999998e-05 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.655845+0000", 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017313963000000002 2024-09-16T05:11:16.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.655956+0000", 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011097799999999999 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.656079+0000", 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012392300000000001 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.159 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:406 2.9 e126/107 2:96dc17ef:::smithi20457400-10:head v 126'25, mlcod=123'22)", 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:57.668655+0000", 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.330321896000001, 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015910130000000001, 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.668655+0000", 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.668655+0000", 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.668656+0000", 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2e-07 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.669222+0000", 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056613500000000005 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.669247+0000", 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4655e-05 2024-09-16T05:11:16.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.669256+0000", 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1779999999999992e-06 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.669315+0000", 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8835999999999997e-05 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.669330+0000", 2024-09-16T05:11:16.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5500000000000001e-05 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.670202+0000", 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087222500000000002 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.670246+0000", 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3964000000000001e-05 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.164 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:408 2.9 e126/107 2:96dc17ef:::smithi20457400-10:head v 126'27, mlcod=123'24)", 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:57.679332+0000", 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.319645221999998, 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014568649999999999, 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.679332+0000", 2024-09-16T05:11:16.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.679332+0000", 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.679333+0000", 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9700000000000005e-07 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.679719+0000", 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00038621800000000002 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.679748+0000", 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9150999999999999e-05 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.679764+0000", 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5671999999999998e-05 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.679846+0000", 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2316000000000003e-05 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.679861+0000", 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4881e-05 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.680746+0000", 2024-09-16T05:11:16.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00088509400000000001 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:57.680789+0000", 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2636000000000003e-05 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:11:16.169 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:11:16.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: pgmap v366: 27 pgs: 27 active+clean; 152 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.0872389664879905]}]': finished 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08736669557508858]}]': finished 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08838852827187324]}]': finished 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08736669557508858]}]': finished 2024-09-16T05:11:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0872389664879905]}]': finished 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08711123740089242]}]': finished 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08707474337600725]}]': finished 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08711123740089242]}]': finished 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716232903573165]}]': finished 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: osdmap e143: 12 total, 12 up, 12 in 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2035305225' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2035305225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:15 smithi204 ceph-mon[30933]: osdmap e144: 12 total, 12 up, 12 in 2024-09-16T05:11:16.192 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:16.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:16.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:16.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:16.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:16.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:16.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:16.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:16.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:16.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:16.644 INFO:tasks.rados.rados.2.smithi204.stdout:216: done (2 left) 2024-09-16T05:11:16.644 INFO:tasks.rados.rados.2.smithi204.stdout:217: done (1 left) 2024-09-16T05:11:16.644 INFO:tasks.rados.rados.2.smithi204.stdout:218: done (0 left) 2024-09-16T05:11:16.644 INFO:tasks.rados.rados.2.smithi204.stdout:219: write (excl) oid 50 current snap is 20 2024-09-16T05:11:16.644 INFO:tasks.rados.rados.2.smithi204.stdout:219: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-16T05:11:16.649 INFO:tasks.rados.rados.2.smithi204.stdout:219: writing smithi20457400-50 from 682384 to 1139214 tid 1 2024-09-16T05:11:16.655 INFO:tasks.rados.rados.2.smithi204.stdout:219: writing smithi20457400-50 from 1721042 to 2144741 tid 2 2024-09-16T05:11:16.661 INFO:tasks.rados.rados.2.smithi204.stdout:219: writing smithi20457400-50 from 2809728 to 3264664 tid 3 2024-09-16T05:11:16.664 INFO:tasks.rados.rados.2.smithi204.stdout:219: writing smithi20457400-50 from 3731357 to 3882384 tid 4 2024-09-16T05:11:16.665 INFO:tasks.rados.rados.2.smithi204.stdout:220: rollback oid 47 current snap is 20 2024-09-16T05:11:16.665 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 47 to 18 2024-09-16T05:11:16.665 INFO:tasks.rados.rados.2.smithi204.stdout:219: finishing write tid 1 to smithi20457400-50 2024-09-16T05:11:16.665 INFO:tasks.rados.rados.2.smithi204.stdout:219: finishing write tid 2 to smithi20457400-50 2024-09-16T05:11:16.665 INFO:tasks.rados.rados.2.smithi204.stdout:219: finishing write tid 3 to smithi20457400-50 2024-09-16T05:11:16.665 INFO:tasks.rados.rados.2.smithi204.stdout:221: snap_remove snap 15 2024-09-16T05:11:16.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:16.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:16.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:16.776 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:16.777 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:16.785 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:16.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:16.807 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:16.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:16.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:16.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:16.829 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:16.850 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:16.851 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:16.871 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:16.872 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:16.892 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:16.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:16.914 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:16.914 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:16.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:16.935 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:16.936 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:16.938 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:16.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:16.956 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:16.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:16.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:16.978 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:17.000 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:17.001 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:17.002 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:11:17.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:17.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:17.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:17.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:17.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:17.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:17.242 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:17.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:17.314 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:17.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:17.416 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:17.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:17.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:17.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:17.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:17.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:17.652 INFO:tasks.rados.rados.2.smithi204.stdout:221: done (2 left) 2024-09-16T05:11:17.653 INFO:tasks.rados.rados.2.smithi204.stdout:222: write (excl) oid 28 current snap is 20 2024-09-16T05:11:17.653 INFO:tasks.rados.rados.2.smithi204.stdout:220: finishing rollback tid 1 to smithi20457400-47 2024-09-16T05:11:17.653 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 129 snap 0 seq_num 4282058624) dirty dne 2024-09-16T05:11:17.653 INFO:tasks.rados.rados.2.smithi204.stdout:219: finishing write tid 4 to smithi20457400-50 2024-09-16T05:11:17.653 INFO:tasks.rados.rados.2.smithi204.stdout:219: finishing write tid 5 to smithi20457400-50 2024-09-16T05:11:17.654 INFO:tasks.rados.rados.2.smithi204.stdout:219: finishing write tid 6 to smithi20457400-50 2024-09-16T05:11:17.654 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 56 (ObjNum 112 snap 20 seq_num 112) dirty exists 2024-09-16T05:11:17.654 INFO:tasks.rados.rados.2.smithi204.stdout:219: left oid 50 (ObjNum 112 snap 20 seq_num 112) 2024-09-16T05:11:17.654 INFO:tasks.rados.rados.2.smithi204.stdout:222: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-16T05:11:17.660 INFO:tasks.rados.rados.2.smithi204.stdout:222: writing smithi20457400-28 from 699191 to 1231270 tid 1 2024-09-16T05:11:17.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:17.667 INFO:tasks.rados.rados.2.smithi204.stdout:222: writing smithi20457400-28 from 1779524 to 2346881 tid 2 2024-09-16T05:11:17.675 INFO:tasks.rados.rados.2.smithi204.stdout:222: writing smithi20457400-28 from 2837151 to 3503359 tid 3 2024-09-16T05:11:17.675 INFO:tasks.rados.rados.2.smithi204.stdout:219: done (2 left) 2024-09-16T05:11:17.675 INFO:tasks.rados.rados.2.smithi204.stdout:220: done (1 left) 2024-09-16T05:11:17.675 INFO:tasks.rados.rados.2.smithi204.stdout:223: snap_remove snap 17 2024-09-16T05:11:17.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:17.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:17.734 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:17.734 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:17.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:17 smithi196 ceph-mon[30802]: pgmap v369: 27 pgs: 27 active+clean; 153 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2024-09-16T05:11:17.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:17 smithi196 ceph-mon[30802]: osdmap e145: 12 total, 12 up, 12 in 2024-09-16T05:11:17.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2212384799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:17.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:17.814 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:17.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:17.892 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:17 smithi204 ceph-mon[30933]: pgmap v369: 27 pgs: 27 active+clean; 153 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2024-09-16T05:11:17.892 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:17 smithi204 ceph-mon[30933]: osdmap e145: 12 total, 12 up, 12 in 2024-09-16T05:11:17.892 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2212384799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:17.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:17 smithi032 ceph-mon[27388]: pgmap v369: 27 pgs: 27 active+clean; 153 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2024-09-16T05:11:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:17 smithi032 ceph-mon[27388]: osdmap e145: 12 total, 12 up, 12 in 2024-09-16T05:11:17.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2212384799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:17.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:17.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:18.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:18.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:18.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:18.165 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:18.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:18.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:18.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:11:18.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:18.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:18.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:18.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:11:18.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:18.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:18.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:18.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:18.514 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:18.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:18.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:18.636 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:18.657 INFO:tasks.rados.rados.2.smithi204.stdout:223: done (1 left) 2024-09-16T05:11:18.657 INFO:tasks.rados.rados.2.smithi204.stdout:224: snap_create 2024-09-16T05:11:18.658 INFO:tasks.rados.rados.2.smithi204.stdout:222: finishing write tid 1 to smithi20457400-28 2024-09-16T05:11:18.658 INFO:tasks.rados.rados.2.smithi204.stdout:222: finishing write tid 2 to smithi20457400-28 2024-09-16T05:11:18.658 INFO:tasks.rados.rados.2.smithi204.stdout:222: finishing write tid 3 to smithi20457400-28 2024-09-16T05:11:18.658 INFO:tasks.rados.rados.2.smithi204.stdout:222: finishing write tid 4 to smithi20457400-28 2024-09-16T05:11:18.658 INFO:tasks.rados.rados.2.smithi204.stdout:222: finishing write tid 5 to smithi20457400-28 2024-09-16T05:11:18.658 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 48 (ObjNum 113 snap 20 seq_num 113) dirty exists 2024-09-16T05:11:18.658 INFO:tasks.rados.rados.2.smithi204.stdout:222: left oid 28 (ObjNum 113 snap 20 seq_num 113) 2024-09-16T05:11:18.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:18.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:18.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:18.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:18.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:18.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:18.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:18 smithi032 ceph-mon[27388]: osdmap e146: 12 total, 12 up, 12 in 2024-09-16T05:11:18.903 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:18 smithi204 ceph-mon[30933]: osdmap e146: 12 total, 12 up, 12 in 2024-09-16T05:11:18.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:18.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:18.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:18.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:18 smithi196 ceph-mon[30802]: osdmap e146: 12 total, 12 up, 12 in 2024-09-16T05:11:18.992 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:11:18.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:11:18.992 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T05:11:18.993 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.254329 2024-09-16T05:11:18.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 0.25432873116435295 2024-09-16T05:11:19.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:19.060 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:19.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:19.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:19.185 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:19.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:19.196 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:19.196 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:19.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:19.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:19.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:19.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:19.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:19.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:19.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:19.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:19.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:11:19.654 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:11:19.657 INFO:tasks.rados.rados.2.smithi204.stdout:222: done (1 left) 2024-09-16T05:11:19.657 INFO:tasks.rados.rados.2.smithi204.stdout:224: done (0 left) 2024-09-16T05:11:19.657 INFO:tasks.rados.rados.2.smithi204.stdout:225: read oid 2 snap -1 2024-09-16T05:11:19.658 INFO:tasks.rados.rados.2.smithi204.stdout:225: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T05:11:19.658 INFO:tasks.rados.rados.2.smithi204.stdout:226: snap_create 2024-09-16T05:11:19.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:19.665 INFO:tasks.rados.rados.2.smithi204.stdout:225: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T05:11:19.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:19.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:19.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:19.813 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:19.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:19.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:11:19.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:19 smithi032 ceph-mon[27388]: pgmap v372: 27 pgs: 1 peering, 26 active+clean; 151 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2024-09-16T05:11:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:19 smithi032 ceph-mon[27388]: osdmap e147: 12 total, 12 up, 12 in 2024-09-16T05:11:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4238302035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:19 smithi204 ceph-mon[30933]: pgmap v372: 27 pgs: 1 peering, 26 active+clean; 151 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2024-09-16T05:11:19.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:19 smithi204 ceph-mon[30933]: osdmap e147: 12 total, 12 up, 12 in 2024-09-16T05:11:19.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4238302035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:19.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:19.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:19 smithi196 ceph-mon[30802]: pgmap v372: 27 pgs: 1 peering, 26 active+clean; 151 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2024-09-16T05:11:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:19 smithi196 ceph-mon[30802]: osdmap e147: 12 total, 12 up, 12 in 2024-09-16T05:11:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4238302035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:20.020 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:20.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:20.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:20.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:20.117 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:20.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:20.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:20.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:20.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:20.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:20.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:20.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:20.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:20.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:20.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:20.577 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:20.577 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:20.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:20.640 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.6 primary-affinity to 0.254329 (8166672) 2024-09-16T05:11:20.643 INFO:tasks.rados.rados.2.smithi204.stdout:225: done (1 left) 2024-09-16T05:11:20.643 INFO:tasks.rados.rados.2.smithi204.stdout:226: done (0 left) 2024-09-16T05:11:20.643 INFO:tasks.rados.rados.2.smithi204.stdout:227: read oid 42 snap -1 2024-09-16T05:11:20.643 INFO:tasks.rados.rados.2.smithi204.stdout:227: expect deleted 2024-09-16T05:11:20.644 INFO:tasks.rados.rados.2.smithi204.stdout:228: write (excl) oid 17 current snap is 22 2024-09-16T05:11:20.644 INFO:tasks.rados.rados.2.smithi204.stdout:228: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-16T05:11:20.650 INFO:tasks.rados.rados.2.smithi204.stdout:228: writing smithi20457400-17 from 715998 to 1323326 tid 1 2024-09-16T05:11:20.655 INFO:tasks.rados.rados.2.smithi204.stdout:228: writing smithi20457400-17 from 1838006 to 2265374 tid 2 2024-09-16T05:11:20.661 INFO:tasks.rados.rados.2.smithi204.stdout:228: writing smithi20457400-17 from 2864574 to 3342054 tid 3 2024-09-16T05:11:20.665 INFO:tasks.rados.rados.2.smithi204.stdout:228: writing smithi20457400-17 from 3863835 to 3915998 tid 4 2024-09-16T05:11:20.665 INFO:tasks.rados.rados.2.smithi204.stdout:229: rollback oid 32 current snap is 22 2024-09-16T05:11:20.665 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 19 2024-09-16T05:11:20.665 INFO:tasks.rados.rados.2.smithi204.stdout:228: finishing write tid 1 to smithi20457400-17 2024-09-16T05:11:20.665 INFO:tasks.rados.rados.2.smithi204.stdout:228: finishing write tid 2 to smithi20457400-17 2024-09-16T05:11:20.665 INFO:tasks.rados.rados.2.smithi204.stdout:227: done (2 left) 2024-09-16T05:11:20.665 INFO:tasks.rados.rados.2.smithi204.stdout:230: read oid 42 snap 18 2024-09-16T05:11:20.665 INFO:tasks.rados.rados.2.smithi204.stdout:230: expect deleted 2024-09-16T05:11:20.666 INFO:tasks.rados.rados.2.smithi204.stdout:231: read oid 5 snap 20 2024-09-16T05:11:20.666 INFO:tasks.rados.rados.2.smithi204.stdout:231: expect (ObjNum 63 snap 3 seq_num 63) 2024-09-16T05:11:20.666 INFO:tasks.rados.rados.2.smithi204.stdout:232: snap_remove snap 21 2024-09-16T05:11:20.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:20.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:20.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:20.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:20.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:20.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4238302035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:20 smithi032 ceph-mon[27388]: osdmap e148: 12 total, 12 up, 12 in 2024-09-16T05:11:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4232839013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.25432873116435295}]: dispatch 2024-09-16T05:11:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:20 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4232839013' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.25432873116435295}]': finished 2024-09-16T05:11:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:20 smithi032 ceph-mon[27388]: osdmap e149: 12 total, 12 up, 12 in 2024-09-16T05:11:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4238302035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:20 smithi204 ceph-mon[30933]: osdmap e148: 12 total, 12 up, 12 in 2024-09-16T05:11:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4232839013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.25432873116435295}]: dispatch 2024-09-16T05:11:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:20 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4232839013' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.25432873116435295}]': finished 2024-09-16T05:11:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:20 smithi204 ceph-mon[30933]: osdmap e149: 12 total, 12 up, 12 in 2024-09-16T05:11:20.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:20.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:20.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4238302035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:20 smithi196 ceph-mon[30802]: osdmap e148: 12 total, 12 up, 12 in 2024-09-16T05:11:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4232839013' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.25432873116435295}]: dispatch 2024-09-16T05:11:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:20 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4232839013' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.25432873116435295}]': finished 2024-09-16T05:11:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:20 smithi196 ceph-mon[30802]: osdmap e149: 12 total, 12 up, 12 in 2024-09-16T05:11:21.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:21.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:21.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:21.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:11:21.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:21.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:21.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:21.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:21.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:21.434 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:21.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:21.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:21.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:21.536 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:21.536 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:21.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:21.557 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:21.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:21.578 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:21.578 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:21.582 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:21.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:21.599 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:21.599 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:21.621 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:21.621 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:21.643 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:21.644 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:21.665 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:21.665 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:21.673 INFO:tasks.rados.rados.2.smithi204.stdout:232: done (4 left) 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:233: read oid 37 snap -1 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:233: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:234: delete oid 18 current snap is 22 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:228: finishing write tid 3 to smithi20457400-17 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:228: finishing write tid 4 to smithi20457400-17 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:228: finishing write tid 5 to smithi20457400-17 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:229: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:229: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:229: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 93 (ObjNum 51 snap 1 seq_num 51) dirty exists 2024-09-16T05:11:21.674 INFO:tasks.rados.rados.2.smithi204.stdout:231: expect (ObjNum 63 snap 3 seq_num 63) 2024-09-16T05:11:21.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:21.688 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:21.688 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:21.707 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:21.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:21.728 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:21.728 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:21.785 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:21.785 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:21.806 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:21.806 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:21.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:21.821 INFO:tasks.rados.rados.2.smithi204.stdout:228: finishing write tid 6 to smithi20457400-17 2024-09-16T05:11:21.821 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 55 (ObjNum 114 snap 22 seq_num 114) dirty exists 2024-09-16T05:11:21.821 INFO:tasks.rados.rados.2.smithi204.stdout:228: left oid 17 (ObjNum 114 snap 22 seq_num 114) 2024-09-16T05:11:21.821 INFO:tasks.rados.rados.2.smithi204.stdout:233: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-16T05:11:21.826 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:21 smithi204 ceph-mon[30933]: pgmap v375: 27 pgs: 1 peering, 26 active+clean; 156 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:11:21.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:21 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:21.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:21 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:21.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3396185711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:21.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:21.827 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:21.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:21.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:21.894 INFO:tasks.rados.rados.2.smithi204.stdout:228: done (5 left) 2024-09-16T05:11:21.894 INFO:tasks.rados.rados.2.smithi204.stdout:229: done (4 left) 2024-09-16T05:11:21.894 INFO:tasks.rados.rados.2.smithi204.stdout:230: done (3 left) 2024-09-16T05:11:21.894 INFO:tasks.rados.rados.2.smithi204.stdout:231: done (2 left) 2024-09-16T05:11:21.894 INFO:tasks.rados.rados.2.smithi204.stdout:233: done (1 left) 2024-09-16T05:11:21.894 INFO:tasks.rados.rados.2.smithi204.stdout:234: done (0 left) 2024-09-16T05:11:21.894 INFO:tasks.rados.rados.2.smithi204.stdout:235: read oid 36 snap 20 2024-09-16T05:11:21.895 INFO:tasks.rados.rados.2.smithi204.stdout:235: expect (ObjNum 109 snap 18 seq_num 109) 2024-09-16T05:11:21.895 INFO:tasks.rados.rados.2.smithi204.stdout:236: read oid 21 snap -1 2024-09-16T05:11:21.895 INFO:tasks.rados.rados.2.smithi204.stdout:236: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T05:11:21.895 INFO:tasks.rados.rados.2.smithi204.stdout:237: read oid 5 snap -1 2024-09-16T05:11:21.895 INFO:tasks.rados.rados.2.smithi204.stdout:237: expect (ObjNum 63 snap 3 seq_num 63) 2024-09-16T05:11:21.895 INFO:tasks.rados.rados.2.smithi204.stdout:238: snap_create 2024-09-16T05:11:21.899 INFO:tasks.rados.rados.2.smithi204.stdout:236: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T05:11:21.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:21.935 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:11:21.935 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:11:21.935 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:11:21.935 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:11:21.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:21.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:21.984 INFO:tasks.rados.rados.2.smithi204.stdout:237: expect (ObjNum 63 snap 3 seq_num 63) 2024-09-16T05:11:21.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:21 smithi196 ceph-mon[30802]: pgmap v375: 27 pgs: 1 peering, 26 active+clean; 156 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:11:21.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:21 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:21 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3396185711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:22.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:22.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:22.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:22.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:22.120 INFO:tasks.rados.rados.2.smithi204.stdout:235: expect (ObjNum 109 snap 18 seq_num 109) 2024-09-16T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:21 smithi032 ceph-mon[27388]: pgmap v375: 27 pgs: 1 peering, 26 active+clean; 156 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:21 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:21 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:22.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3396185711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:22.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:22.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:22.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:22.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:22.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:22.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:22.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:11:22.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:22.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:22.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:22.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:11:22.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:22.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:22.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:22.666 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:22.689 INFO:tasks.rados.rados.2.smithi204.stdout:235: done (3 left) 2024-09-16T05:11:22.689 INFO:tasks.rados.rados.2.smithi204.stdout:236: done (2 left) 2024-09-16T05:11:22.690 INFO:tasks.rados.rados.2.smithi204.stdout:237: done (1 left) 2024-09-16T05:11:22.690 INFO:tasks.rados.rados.2.smithi204.stdout:238: done (0 left) 2024-09-16T05:11:22.690 INFO:tasks.rados.rados.2.smithi204.stdout:239: snap_create 2024-09-16T05:11:22.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:22.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:22.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:22.788 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:22.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:22.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:22.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:22.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:22 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:22.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:22 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:22.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:22 smithi204 ceph-mon[30933]: osdmap e150: 12 total, 12 up, 12 in 2024-09-16T05:11:22.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:22.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:22.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:22 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:22 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:22 smithi196 ceph-mon[30802]: osdmap e150: 12 total, 12 up, 12 in 2024-09-16T05:11:22.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:22 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:22.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:22 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:22.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:22 smithi032 ceph-mon[27388]: osdmap e150: 12 total, 12 up, 12 in 2024-09-16T05:11:23.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:23.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:23.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:23.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:23.226 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:11:23.226 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:11:23.226 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:11:23.226 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:11:23.226 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:11:23.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:23.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:23.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:23.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:23.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:23.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:23.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:23.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:23.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:23.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:23.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:11:23.691 INFO:tasks.rados.rados.2.smithi204.stdout:239: done (0 left) 2024-09-16T05:11:23.691 INFO:tasks.rados.rados.2.smithi204.stdout:240: read oid 18 snap 22 2024-09-16T05:11:23.691 INFO:tasks.rados.rados.2.smithi204.stdout:240: expect deleted 2024-09-16T05:11:23.691 INFO:tasks.rados.rados.2.smithi204.stdout:241: rollback oid 14 current snap is 24 2024-09-16T05:11:23.691 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 22 2024-09-16T05:11:23.691 INFO:tasks.rados.rados.2.smithi204.stdout:242: snap_remove snap 9 2024-09-16T05:11:23.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:23.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:23.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:23.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:23.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:23.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:23.966 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:23 smithi204 ceph-mon[30933]: pgmap v378: 27 pgs: 3 peering, 24 active+clean; 164 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:11:23.966 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:23 smithi204 ceph-mon[30933]: osdmap e151: 12 total, 12 up, 12 in 2024-09-16T05:11:23.966 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4289594716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:23.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:23 smithi196 ceph-mon[30802]: pgmap v378: 27 pgs: 3 peering, 24 active+clean; 164 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:11:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:23 smithi196 ceph-mon[30802]: osdmap e151: 12 total, 12 up, 12 in 2024-09-16T05:11:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4289594716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:23.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:11:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:23.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:23.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:24.051 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:23 smithi032 ceph-mon[27388]: pgmap v378: 27 pgs: 3 peering, 24 active+clean; 164 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:11:24.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:23 smithi032 ceph-mon[27388]: osdmap e151: 12 total, 12 up, 12 in 2024-09-16T05:11:24.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4289594716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:24.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:24.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:24.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:24.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:24.235 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:24.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:24.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:24.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:24.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:24.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:24.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:11:24.467 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:24.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:24.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:24.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:24.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:24.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:24.677 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:11:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:11:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:53 2.d 2:ba21f1a9:::smithi20457400-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.146158+0000", 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.528530484000001, 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035468079999999998, 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 53 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146158+0000", 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146158+0000", 2024-09-16T05:11:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146159+0000", 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.525e-06 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146709+0000", 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00055025899999999999 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146719+0000", 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0860000000000008e-06 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146735+0000", 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6504e-05 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146801+0000", 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5666000000000005e-05 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146908+0000", 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010762799999999999 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.147041+0000", 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013216700000000001 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.148575+0000", 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001534344 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149619+0000", 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010436359999999999 2024-09-16T05:11:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149659+0000", 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0689000000000003e-05 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149705+0000", 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5303999999999999e-05 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:54 2.d 2:ba21f1a9:::smithi20457400-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.146754+0000", 2024-09-16T05:11:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.527934352000003, 2024-09-16T05:11:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031139599999999998, 2024-09-16T05:11:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 54 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146754+0000", 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146754+0000", 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146754+0000", 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9799999999999999e-07 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146796+0000", 2024-09-16T05:11:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1289999999999999e-05 2024-09-16T05:11:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146801+0000", 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146804+0000", 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5439999999999999e-06 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.147836+0000", 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001031828 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.147899+0000", 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2843999999999997e-05 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.147989+0000", 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9713e-05 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.148581+0000", 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059255899999999999 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149772+0000", 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011905119999999999 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149829+0000", 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7182000000000002e-05 2024-09-16T05:11:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149868+0000", 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8673000000000001e-05 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:55 2.d 2:ba21f1a9:::smithi20457400-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.146813+0000", 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.527875391999999, 2024-09-16T05:11:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031733680000000002, 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 55 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146813+0000", 2024-09-16T05:11:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146813+0000", 2024-09-16T05:11:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.695 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146813+0000", 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5999999999999999e-07 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146815+0000", 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8360000000000001e-06 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146824+0000", 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1279999999999997e-06 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146828+0000", 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4510000000000001e-06 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.148325+0000", 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014971559999999999 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.148372+0000", 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7738000000000003e-05 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.148453+0000", 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.085e-05 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.148684+0000", 2024-09-16T05:11:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023069200000000001 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149909+0000", 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00122455 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149938+0000", 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9416000000000001e-05 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.149986+0000", 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8191e-05 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:56 2.d 2:ba21f1a9:::smithi20457400-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.146832+0000", 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.527855977999998, 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033484360000000002, 2024-09-16T05:11:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 56 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146832+0000", 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146832+0000", 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146832+0000", 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6300000000000001e-07 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146834+0000", 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.827e-06 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146838+0000", 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.946e-06 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.146841+0000", 2024-09-16T05:11:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9950000000000001e-06 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.148614+0000", 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00177308 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.148660+0000", 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5277e-05 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.150023+0000", 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001363487 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.150096+0000", 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3056e-05 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.150181+0000", 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4505000000000005e-05 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:265 2.d 2:ba754fcb:::smithi20457400-30:head [read 0~2487403,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:48.184500+0000", 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.490187982999998, 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032531589999999998, 2024-09-16T05:11:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 265 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184500+0000", 2024-09-16T05:11:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184500+0000", 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184502+0000", 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0789999999999999e-06 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184512+0000", 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5699999999999999e-06 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184531+0000", 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9303000000000001e-05 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184541+0000", 2024-09-16T05:11:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0101999999999999e-05 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184599+0000", 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8038999999999998e-05 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184688+0000", 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8634999999999994e-05 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.187753+0000", 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003065431 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:266 2.d 2:ba754fcb:::smithi20457400-30:head [read 0~2487403,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:48.184566+0000", 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.490122521000004, 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043037020000000004, 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 266 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184566+0000", 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184566+0000", 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184566+0000", 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9999999999999998e-07 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184570+0000", 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3400000000000002e-06 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184577+0000", 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0280000000000003e-06 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184582+0000", 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7529999999999998e-06 2024-09-16T05:11:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.187786+0000", 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003203685 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.187848+0000", 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2249000000000007e-05 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.188869+0000", 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001021147 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:267 2.d 2:ba754fcb:::smithi20457400-30:head [sparse-read 0~1009951] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:48.184590+0000", 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.490097691000003, 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045450029999999997, 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 267 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184590+0000", 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184590+0000", 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184591+0000", 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3599999999999999e-07 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184593+0000", 2024-09-16T05:11:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0150000000000002e-06 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184598+0000", 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1839999999999998e-06 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.184602+0000", 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.25e-06 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.188902+0000", 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004299855 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.188990+0000", 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7862999999999995e-05 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:48.189135+0000", 2024-09-16T05:11:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014540000000000001 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:290 2.d 2:b98a79db:::smithi20457400-31:head [copy-from ver 13] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.334223946999998, 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020074727000000001, 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 290 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340466+0000", 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4300000000000001e-06 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340473+0000", 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3339999999999996e-06 2024-09-16T05:11:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340486+0000", 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.279e-05 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340495+0000", 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4890000000000001e-06 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340559+0000", 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3962999999999997e-05 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340653+0000", 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3468999999999996e-05 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.353864+0000", 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01321114 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357823+0000", 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039590759999999997 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360478+0000", 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026554489999999998 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360511+0000", 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3052999999999997e-05 2024-09-16T05:11:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360539+0000", 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7534000000000001e-05 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:291 2.d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.334168063999996, 2024-09-16T05:11:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016626765000000002, 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 291 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:11:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340521+0000", 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4199999999999995e-07 2024-09-16T05:11:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340524+0000", 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8590000000000001e-06 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340532+0000", 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4090000000000003e-06 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340538+0000", 2024-09-16T05:11:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.835e-06 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340822+0000", 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028415500000000002 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340874+0000", 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1558999999999999e-05 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357014+0000", 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016140379999999999 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357072+0000", 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7887999999999999e-05 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357147+0000", 2024-09-16T05:11:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5037999999999995e-05 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.7.58:1 2.d 2:b98a79db:::smithi20457400-31:head [assert-version v13,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:55.643390+0000", 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.031297952999999, 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048006359999999996, 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.643390+0000", 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.643390+0000", 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.643392+0000", 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2670000000000001e-06 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.643404+0000", 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1202999999999999e-05 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.643432+0000", 2024-09-16T05:11:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8469000000000001e-05 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.643448+0000", 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5673000000000001e-05 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.643572+0000", 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012448200000000001 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.643597+0000", 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4816000000000001e-05 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.644716+0000", 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00111867 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.644814+0000", 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8443000000000006e-05 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.648191+0000", 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033766130000000001 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:391 2.8 e124/107 2:1030fb5b:::smithi20457400-27:head v 124'15, mlcod=119'11)", 2024-09-16T05:11:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:55.652649+0000", 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.022039583000002, 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056997660000000002, 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.652649+0000", 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.652649+0000", 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.652649+0000", 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3000000000000001e-07 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.656404+0000", 2024-09-16T05:11:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037554200000000002 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.656430+0000", 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5530000000000001e-05 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.656437+0000", 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7479999999999996e-06 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.656483+0000", 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6174000000000002e-05 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.656501+0000", 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8595000000000001e-05 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.658308+0000", 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001806385 2024-09-16T05:11:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:55.658348+0000", 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0484000000000001e-05 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:395 2.d 2:ba7e58cf:::smithi20457400-26:head [write 544560~783450] snapc 8=[7] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:56.649398+0000", 2024-09-16T05:11:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.025290271999999, 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003836913, 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 395 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.649398+0000", 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.649398+0000", 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.649400+0000", 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8199999999999999e-06 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.650011+0000", 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061156199999999996 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.650027+0000", 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.650034+0000", 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7870000000000004e-06 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.650094+0000", 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9951000000000002e-05 2024-09-16T05:11:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.650162+0000", 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8385000000000003e-05 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.650289+0000", 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012718000000000001 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.651911+0000", 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00162127 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.653160+0000", 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012488359999999999 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.653199+0000", 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9304999999999997e-05 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:56.653235+0000", 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5936000000000003e-05 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:439 2.d 2:ba7e58cf:::smithi20457400-26:head [write 679016~701795,stat] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:58.657187+0000", 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.017500885, 2024-09-16T05:11:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043578820000000004, 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 439 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.657187+0000", 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.657187+0000", 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.657189+0000", 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.792e-06 2024-09-16T05:11:24.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.657552+0000", 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036261099999999999 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.657571+0000", 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9650999999999999e-05 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.657582+0000", 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0473e-05 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.657645+0000", 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3285000000000001e-05 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.657650+0000", 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6600000000000003e-06 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.658743+0000", 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010933270000000001 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.658804+0000", 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0646999999999997e-05 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.658927+0000", 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000123752 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.660425+0000", 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014974960000000001 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.661473+0000", 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010481030000000001 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.661513+0000", 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9443000000000002e-05 2024-09-16T05:11:24.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.661545+0000", 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2641999999999997e-05 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:442 2.d 2:ba7e58cf:::smithi20457400-26:head [setxattr _header (56) in=7b,truncate 3479016] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:58.667973+0000", 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.006715115999999, 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061931820000000002, 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 442 2024-09-16T05:11:24.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.667973+0000", 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.667973+0000", 2024-09-16T05:11:24.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.667974+0000", 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8599999999999998e-07 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.667978+0000", 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9400000000000004e-06 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.667985+0000", 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3320000000000003e-06 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.667990+0000", 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6199999999999998e-06 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.668843+0000", 2024-09-16T05:11:24.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000853019 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.668924+0000", 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1129000000000002e-05 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.669013+0000", 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9122999999999994e-05 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.669301+0000", 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000288392 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.674050+0000", 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047491069999999998 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.674107+0000", 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6233999999999997e-05 2024-09-16T05:11:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.674166+0000", 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9700000000000001e-05 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:443 2.d 2:ba7e58cf:::smithi20457400-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e127)", 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:58.668009+0000", 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.006679639000001, 2024-09-16T05:11:24.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062994679999999999, 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 443 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.668009+0000", 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.668009+0000", 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.668009+0000", 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7e-07 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.668012+0000", 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2899999999999998e-06 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.668019+0000", 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8820000000000003e-06 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.668025+0000", 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5570000000000003e-06 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.669181+0000", 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001156168 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.669254+0000", 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3501e-05 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.674201+0000", 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049466429999999997 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.674247+0000", 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5757000000000003e-05 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.674308+0000", 2024-09-16T05:11:24.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1199999999999997e-05 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:605 2.d 2:b1061e13:::smithi20457400-33:head [copy-from ver 20] snapc 13=[13,12,11,f,e,b] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 10.871737067, 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017106367000000001, 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 605 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802952+0000", 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9999999999999998e-07 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802955+0000", 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1379999999999999e-06 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.774 INFO:tasks.rados.rados.2.smithi204.stdout:242: done (2 left) 2024-09-16T05:11:24.774 INFO:tasks.rados.rados.2.smithi204.stdout:243: snap_create 2024-09-16T05:11:24.774 INFO:tasks.rados.rados.2.smithi204.stdout:241: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:11:24.774 INFO:tasks.rados.rados.2.smithi204.stdout:241: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:11:24.774 INFO:tasks.rados.rados.2.smithi204.stdout:241: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:11:24.774 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 97 (ObjNum 76 snap 6 seq_num 76) dirty exists 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802961+0000", 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3820000000000001e-06 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802966+0000", 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1370000000000001e-06 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803384+0000", 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00041787299999999997 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803568+0000", 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018420200000000001 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.814402+0000", 2024-09-16T05:11:24.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010833617 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818577+0000", 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041754770000000004 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.819989+0000", 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001411818 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.820026+0000", 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6613000000000003e-05 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.820057+0000", 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1609999999999997e-05 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.778 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:606 2.d 2:b1061e13:::smithi20457400-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 10.871712273, 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015239317, 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 606 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.63e-07 2024-09-16T05:11:24.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802980+0000", 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.309e-06 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802993+0000", 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3574000000000001e-05 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802999+0000", 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5949999999999998e-06 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803711+0000", 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071251099999999998 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803769+0000", 2024-09-16T05:11:24.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7649000000000003e-05 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818011+0000", 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014242184 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818110+0000", 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8938999999999996e-05 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818215+0000", 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105093 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:678 2.d 2:ba21f1a9:::smithi20457400-14:head [append 0~0] snapc 19=[19,18,16,15,14,12,f,b] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:23.691212+0000", 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.983475981, 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033082139999999999, 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.784 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 678 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691212+0000", 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691212+0000", 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691214+0000", 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.747e-06 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691223+0000", 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7399999999999993e-06 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691242+0000", 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9264999999999999e-05 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691251+0000", 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0040000000000005e-06 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691311+0000", 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0225e-05 2024-09-16T05:11:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691316+0000", 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.006e-06 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.692550+0000", 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001233622 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.692616+0000", 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6394000000000002e-05 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.692751+0000", 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013497699999999999 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694082+0000", 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001330473 2024-09-16T05:11:24.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694240+0000", 2024-09-16T05:11:24.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015799 2024-09-16T05:11:24.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694480+0000", 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024078399999999999 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694520+0000", 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9987000000000002e-05 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.792 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:679 2.d 2:ba21f1a9:::smithi20457400-14:head [rollback 18] snapc 19=[19,18,16,15,14,12,f,b] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:23.691275+0000", 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.98341314800000001, 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003371101, 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 679 2024-09-16T05:11:24.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691275+0000", 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691275+0000", 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691276+0000", 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6899999999999997e-07 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691281+0000", 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5480000000000001e-06 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691289+0000", 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6469999999999998e-06 2024-09-16T05:11:24.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691295+0000", 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0050000000000001e-06 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691364+0000", 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8744000000000005e-05 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691367+0000", 2024-09-16T05:11:24.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8399999999999997e-06 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.693212+0000", 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018443999999999999 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.693265+0000", 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3226000000000002e-05 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.693381+0000", 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116403 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694090+0000", 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00070861800000000005 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694555+0000", 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046516 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694598+0000", 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3019999999999998e-05 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694646+0000", 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7920999999999997e-05 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.799 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:680 2.d 2:ba21f1a9:::smithi20457400-14:head [append 0~0] snapc 19=[19,18,16,15,14,12,f,b] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:23.691314+0000", 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.98337403599999995, 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034174880000000002, 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 680 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691314+0000", 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691314+0000", 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691315+0000", 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8200000000000001e-07 2024-09-16T05:11:24.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691317+0000", 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0729999999999999e-06 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691322+0000", 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3179999999999997e-06 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691325+0000", 2024-09-16T05:11:24.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3909999999999998e-06 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691383+0000", 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8119999999999999e-05 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.691386+0000", 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9610000000000001e-06 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.693802+0000", 2024-09-16T05:11:24.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024158230000000001 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.693853+0000", 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0331000000000003e-05 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.693941+0000", 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8070000000000002e-05 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694170+0000", 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022938400000000001 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694674+0000", 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050364699999999997 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694703+0000", 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9179999999999998e-05 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:24.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:23.694732+0000", 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8807999999999999e-05 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:24.806 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:11:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:24.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:24.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:24.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:24.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4289594716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:24.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:24 smithi204 ceph-mon[30933]: osdmap e152: 12 total, 12 up, 12 in 2024-09-16T05:11:24.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:24.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:24.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4289594716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:24.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:24 smithi032 ceph-mon[27388]: osdmap e152: 12 total, 12 up, 12 in 2024-09-16T05:11:24.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4289594716' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:24.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:24 smithi196 ceph-mon[30802]: osdmap e152: 12 total, 12 up, 12 in 2024-09-16T05:11:25.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:25.057 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:25.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:25.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:25.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:25.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:25.200 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:25.200 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:25.219 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:25.219 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:25.239 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:25.240 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:25.260 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:25.260 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:25.281 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:25.282 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:25.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:25.304 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:25.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:25.326 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:25.327 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:25.348 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:25.370 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:25.392 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:25.392 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:25.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:25.413 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:25.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:25.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:25.433 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:25.434 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:25.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:25.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:11:25.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:25.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:25.645 INFO:tasks.rados.rados.2.smithi204.stdout:240: done (2 left) 2024-09-16T05:11:25.645 INFO:tasks.rados.rados.2.smithi204.stdout:241: done (1 left) 2024-09-16T05:11:25.645 INFO:tasks.rados.rados.2.smithi204.stdout:243: done (0 left) 2024-09-16T05:11:25.645 INFO:tasks.rados.rados.2.smithi204.stdout:244: write oid 13 current snap is 25 2024-09-16T05:11:25.645 INFO:tasks.rados.rados.2.smithi204.stdout:244: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-09-16T05:11:25.652 INFO:tasks.rados.rados.2.smithi204.stdout:244: writing smithi20457400-13 from 749612 to 1223791 tid 1 2024-09-16T05:11:25.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:25.659 INFO:tasks.rados.rados.2.smithi204.stdout:244: writing smithi20457400-13 from 1954970 to 2386007 tid 2 2024-09-16T05:11:25.668 INFO:tasks.rados.rados.2.smithi204.stdout:244: writing smithi20457400-13 from 2919420 to 3535797 tid 3 2024-09-16T05:11:25.668 INFO:tasks.rados.rados.2.smithi204.stdout:245: rollback oid 42 current snap is 25 2024-09-16T05:11:25.668 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 20 2024-09-16T05:11:25.668 INFO:tasks.rados.rados.2.smithi204.stdout:244: finishing write tid 1 to smithi20457400-13 2024-09-16T05:11:25.668 INFO:tasks.rados.rados.2.smithi204.stdout:244: finishing write tid 2 to smithi20457400-13 2024-09-16T05:11:25.668 INFO:tasks.rados.rados.2.smithi204.stdout:246: rollback oid 14 current snap is 25 2024-09-16T05:11:25.668 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 23 2024-09-16T05:11:25.669 INFO:tasks.rados.rados.2.smithi204.stdout:247: delete oid 33 current snap is 25 2024-09-16T05:11:25.670 INFO:tasks.rados.rados.2.smithi204.stdout:244: finishing write tid 3 to smithi20457400-13 2024-09-16T05:11:25.670 INFO:tasks.rados.rados.2.smithi204.stdout:244: finishing write tid 4 to smithi20457400-13 2024-09-16T05:11:25.671 INFO:tasks.rados.rados.2.smithi204.stdout:244: finishing write tid 5 to smithi20457400-13 2024-09-16T05:11:25.671 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 139 (ObjNum 116 snap 25 seq_num 116) dirty exists 2024-09-16T05:11:25.671 INFO:tasks.rados.rados.2.smithi204.stdout:244: left oid 13 (ObjNum 116 snap 25 seq_num 116) 2024-09-16T05:11:25.671 INFO:tasks.rados.rados.2.smithi204.stdout:245: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:11:25.671 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 129 snap 0 seq_num 4282278336) dirty dne 2024-09-16T05:11:25.671 INFO:tasks.rados.rados.2.smithi204.stdout:246: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:246: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:246: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 108 (ObjNum 76 snap 6 seq_num 76) dirty exists 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:244: done (3 left) 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:245: done (2 left) 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:246: done (1 left) 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:247: done (0 left) 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:248: write oid 2 current snap is 25 2024-09-16T05:11:25.673 INFO:tasks.rados.rados.2.smithi204.stdout:248: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-16T05:11:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:25.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:25.681 INFO:tasks.rados.rados.2.smithi204.stdout:248: writing smithi20457400-2 from 783226 to 1407903 tid 1 2024-09-16T05:11:25.689 INFO:tasks.rados.rados.2.smithi204.stdout:248: writing smithi20457400-2 from 2071934 to 2506640 tid 2 2024-09-16T05:11:25.696 INFO:tasks.rados.rados.2.smithi204.stdout:248: writing smithi20457400-2 from 2974266 to 3613187 tid 3 2024-09-16T05:11:25.696 INFO:tasks.rados.rados.2.smithi204.stdout:249: read oid 23 snap 22 2024-09-16T05:11:25.696 INFO:tasks.rados.rados.2.smithi204.stdout:249: expect deleted 2024-09-16T05:11:25.697 INFO:tasks.rados.rados.2.smithi204.stdout:250: snap_remove snap 18 2024-09-16T05:11:25.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:25.799 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:25.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:25.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:25.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:25.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:25 smithi204 ceph-mon[30933]: pgmap v381: 27 pgs: 2 peering, 25 active+clean; 164 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 23 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:11:25.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:25 smithi204 ceph-mon[30933]: osdmap e153: 12 total, 12 up, 12 in 2024-09-16T05:11:25.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2831743094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:25.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:25 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:25.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:25 smithi204 ceph-mon[30933]: osdmap e154: 12 total, 12 up, 12 in 2024-09-16T05:11:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:25 smithi032 ceph-mon[27388]: pgmap v381: 27 pgs: 2 peering, 25 active+clean; 164 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 23 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:11:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:25 smithi032 ceph-mon[27388]: osdmap e153: 12 total, 12 up, 12 in 2024-09-16T05:11:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2831743094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:25 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:25 smithi032 ceph-mon[27388]: osdmap e154: 12 total, 12 up, 12 in 2024-09-16T05:11:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:25.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:25.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:25 smithi196 ceph-mon[30802]: pgmap v381: 27 pgs: 2 peering, 25 active+clean; 164 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.2 MiB/s wr, 23 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:11:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:25 smithi196 ceph-mon[30802]: osdmap e153: 12 total, 12 up, 12 in 2024-09-16T05:11:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2831743094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:25 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:25 smithi196 ceph-mon[30802]: osdmap e154: 12 total, 12 up, 12 in 2024-09-16T05:11:26.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:26.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:26.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:26.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:26.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:26.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:26.269 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:11:26.269 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:11:26.270 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T05:11:26.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:11:26.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:26.292 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:26.292 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:26.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:26.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:26.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:11:26.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:26.433 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:26.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:26.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:26.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:26.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:26.649 INFO:tasks.rados.rados.2.smithi204.stdout:250: done (2 left) 2024-09-16T05:11:26.649 INFO:tasks.rados.rados.2.smithi204.stdout:251: write oid 1 current snap is 25 2024-09-16T05:11:26.649 INFO:tasks.rados.rados.2.smithi204.stdout:248: finishing write tid 1 to smithi20457400-2 2024-09-16T05:11:26.650 INFO:tasks.rados.rados.2.smithi204.stdout:251: seq_num 119 ranges {400033=699926,1614063=385970} 2024-09-16T05:11:26.655 INFO:tasks.rados.rados.2.smithi204.stdout:251: writing smithi20457400-1 from 400033 to 1099959 tid 1 2024-09-16T05:11:26.657 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:26.660 INFO:tasks.rados.rados.2.smithi204.stdout:251: writing smithi20457400-1 from 1614063 to 2000033 tid 2 2024-09-16T05:11:26.660 INFO:tasks.rados.rados.2.smithi204.stdout:252: read oid 26 snap -1 2024-09-16T05:11:26.661 INFO:tasks.rados.rados.2.smithi204.stdout:252: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:11:26.661 INFO:tasks.rados.rados.2.smithi204.stdout:253: write (excl) oid 30 current snap is 25 2024-09-16T05:11:26.661 INFO:tasks.rados.rados.2.smithi204.stdout:253: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-16T05:11:26.667 INFO:tasks.rados.rados.2.smithi204.stdout:253: writing smithi20457400-30 from 416840 to 1192015 tid 1 2024-09-16T05:11:26.672 INFO:tasks.rados.rados.2.smithi204.stdout:253: writing smithi20457400-30 from 1672545 to 2016840 tid 2 2024-09-16T05:11:26.672 INFO:tasks.rados.rados.2.smithi204.stdout:254: write oid 5 current snap is 25 2024-09-16T05:11:26.672 INFO:tasks.rados.rados.2.smithi204.stdout:254: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-16T05:11:26.676 INFO:tasks.rados.rados.2.smithi204.stdout:254: writing smithi20457400-5 from 433647 to 884071 tid 1 2024-09-16T05:11:26.682 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:26.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:26.684 INFO:tasks.rados.rados.2.smithi204.stdout:254: writing smithi20457400-5 from 1331027 to 1913060 tid 2 2024-09-16T05:11:26.684 INFO:tasks.rados.rados.2.smithi204.stdout:255: copy_from oid 20 from oid 25 current snap is 25 2024-09-16T05:11:26.684 INFO:tasks.rados.rados.2.smithi204.stdout:248: finishing write tid 2 to smithi20457400-2 2024-09-16T05:11:26.684 INFO:tasks.rados.rados.2.smithi204.stdout:248: finishing write tid 3 to smithi20457400-2 2024-09-16T05:11:26.684 INFO:tasks.rados.rados.2.smithi204.stdout:248: finishing write tid 4 to smithi20457400-2 2024-09-16T05:11:26.684 INFO:tasks.rados.rados.2.smithi204.stdout:248: finishing write tid 5 to smithi20457400-2 2024-09-16T05:11:26.684 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 28 (ObjNum 118 snap 25 seq_num 118) dirty exists 2024-09-16T05:11:26.684 INFO:tasks.rados.rados.2.smithi204.stdout:248: left oid 2 (ObjNum 118 snap 25 seq_num 118) 2024-09-16T05:11:26.685 INFO:tasks.rados.rados.2.smithi204.stdout:248: done (6 left) 2024-09-16T05:11:26.685 INFO:tasks.rados.rados.2.smithi204.stdout:249: done (5 left) 2024-09-16T05:11:26.685 INFO:tasks.rados.rados.2.smithi204.stdout:256: write (excl) oid 14 current snap is 25 2024-09-16T05:11:26.685 INFO:tasks.rados.rados.2.smithi204.stdout:251: finishing write tid 1 to smithi20457400-1 2024-09-16T05:11:26.685 INFO:tasks.rados.rados.2.smithi204.stdout:256: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-16T05:11:26.688 INFO:tasks.rados.rados.2.smithi204.stdout:256: writing smithi20457400-14 from 450454 to 1092480 tid 1 2024-09-16T05:11:26.694 INFO:tasks.rados.rados.2.smithi204.stdout:256: writing smithi20457400-14 from 1505862 to 2050454 tid 2 2024-09-16T05:11:26.694 INFO:tasks.rados.rados.2.smithi204.stdout:257: snap_create 2024-09-16T05:11:26.694 INFO:tasks.rados.rados.2.smithi204.stdout:251: finishing write tid 2 to smithi20457400-1 2024-09-16T05:11:26.694 INFO:tasks.rados.rados.2.smithi204.stdout:251: finishing write tid 3 to smithi20457400-1 2024-09-16T05:11:26.694 INFO:tasks.rados.rados.2.smithi204.stdout:251: finishing write tid 4 to smithi20457400-1 2024-09-16T05:11:26.694 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 35 (ObjNum 119 snap 25 seq_num 119) dirty exists 2024-09-16T05:11:26.694 INFO:tasks.rados.rados.2.smithi204.stdout:251: left oid 1 (ObjNum 119 snap 25 seq_num 119) 2024-09-16T05:11:26.695 INFO:tasks.rados.rados.2.smithi204.stdout:252: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:11:26.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:26.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:26.809 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:26.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:26.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:11:26.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:26.945 INFO:tasks.rados.rados.2.smithi204.stdout:253: finishing write tid 1 to smithi20457400-30 2024-09-16T05:11:26.945 INFO:tasks.rados.rados.2.smithi204.stdout:253: finishing write tid 2 to smithi20457400-30 2024-09-16T05:11:26.945 INFO:tasks.rados.rados.2.smithi204.stdout:253: finishing write tid 3 to smithi20457400-30 2024-09-16T05:11:26.945 INFO:tasks.rados.rados.2.smithi204.stdout:253: finishing write tid 4 to smithi20457400-30 2024-09-16T05:11:26.945 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 118 (ObjNum 120 snap 25 seq_num 120) dirty exists 2024-09-16T05:11:26.945 INFO:tasks.rados.rados.2.smithi204.stdout:253: left oid 30 (ObjNum 120 snap 25 seq_num 120) 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:254: finishing write tid 1 to smithi20457400-5 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:254: finishing write tid 2 to smithi20457400-5 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:254: finishing write tid 3 to smithi20457400-5 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:254: finishing write tid 4 to smithi20457400-5 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 59 (ObjNum 121 snap 25 seq_num 121) dirty exists 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:254: left oid 5 (ObjNum 121 snap 25 seq_num 121) 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:255: finishing copy_from to smithi20457400-20 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:255: got expected ENOENT (src dne) 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:255: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:11:26.946 INFO:tasks.rados.rados.2.smithi204.stdout:256: finishing write tid 1 to smithi20457400-14 2024-09-16T05:11:26.947 INFO:tasks.rados.rados.2.smithi204.stdout:256: finishing write tid 2 to smithi20457400-14 2024-09-16T05:11:26.947 INFO:tasks.rados.rados.2.smithi204.stdout:256: finishing write tid 3 to smithi20457400-14 2024-09-16T05:11:26.947 INFO:tasks.rados.rados.2.smithi204.stdout:256: finishing write tid 4 to smithi20457400-14 2024-09-16T05:11:26.947 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 121 (ObjNum 122 snap 25 seq_num 122) dirty exists 2024-09-16T05:11:26.947 INFO:tasks.rados.rados.2.smithi204.stdout:256: left oid 14 (ObjNum 122 snap 25 seq_num 122) 2024-09-16T05:11:26.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:26.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:27.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:27.077 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:27.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:27.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:27.180 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:27.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:27.212 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:27.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:27.252 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 155, 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:11:26.641248+0000", 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:09:26.475677+0000", 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:09:05.828775+0000", 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5804032, 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-09-16T05:11:27.253 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 28, 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:11:27.254 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:11:27.255 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 11, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 11, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 11, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 11, 2024-09-16T05:11:27.256 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 11, 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "106", 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:11:27.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "104", 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:11:27.258 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:11:27.259 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:11:27.260 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:11:27.260 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:11:27.260 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:11:27.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:11:27.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.260 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:11:27.260 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:11:27.260 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:11:27.261 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 4.3899998664855957, 2024-09-16T05:11:27.262 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 4.3899998664855957, 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.82999998331069946, 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 3.6400001049041748, 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 3.6400001049041748, 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:11:27.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:11:27.264 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 16, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 16, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 16, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 16, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 16, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:11:27.265 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "155", 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 26, 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 155, 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:11:27.266 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:11:27.267 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:11:27.268 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.25, 2024-09-16T05:11:27.269 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.25, 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.25, 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.25, 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:11:27.270 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 61, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.271 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.272 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.273 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.276 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 145, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.277 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.278 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:11:27.281 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:11:27.282 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:11:27.282 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:11:27.282 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.283 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.283 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.283 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.283 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:11:27.283 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:11:27.283 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.284 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.284 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.284 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.284 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.284 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.284 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 21, 2024-09-16T05:11:27.284 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:11:27.285 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.285 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.285 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.285 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.285 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.285 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:11:27.286 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:11:27.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.286 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:11:27.286 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:11:27.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.287 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.287 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.287 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.287 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:11:27.287 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:11:27.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.288 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:11:27.288 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:11:27.288 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.288 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:11:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:11:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:11:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:11:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:11:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:11:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.292 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.292 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:11:27.292 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:11:27.292 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.292 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.292 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3508564664", 2024-09-16T05:11:27.293 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3508564664", 2024-09-16T05:11:27.293 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3508564664", 2024-09-16T05:11:27.293 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3508564664", 2024-09-16T05:11:27.293 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.293 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.293 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.293 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.294 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:11:27.294 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:11:27.294 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.294 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.294 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.294 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.294 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.295 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.295 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:11:27.295 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 146, 2024-09-16T05:11:27.295 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.295 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.295 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.295 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.296 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:11:27.296 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:11:27.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.296 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:11:27.297 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:11:27.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.297 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.297 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.297 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.298 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:11:27.298 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:11:27.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.298 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:11:27.298 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:11:27.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.299 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.299 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.299 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.299 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.301 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:11:27.301 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:11:27.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.302 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:11:27.302 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:11:27.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.302 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.302 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.302 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.303 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:11:27.303 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:11:27.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.304 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:11:27.304 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:11:27.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.304 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.304 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:11:27.304 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:11:27.305 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:11:27.305 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:11:27.305 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.305 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.305 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.305 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.306 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:11:27.306 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:11:27.306 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.306 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.306 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.306 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.306 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.306 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.307 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:11:27.307 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-09-16T05:11:27.307 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.307 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.307 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.307 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.307 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.308 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:11:27.308 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:11:27.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.308 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.308 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:11:27.308 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:11:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.309 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:11:27.310 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:11:27.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.310 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:11:27.310 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:11:27.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.311 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.311 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.311 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.311 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.311 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:11:27.312 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:11:27.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.312 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.312 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:11:27.312 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:11:27.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.313 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.313 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.313 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.314 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:11:27.314 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:11:27.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.314 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.314 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:11:27.314 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:11:27.315 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.315 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.315 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:11:27.315 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:11:27.315 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:11:27.315 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:11:27.316 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.316 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.316 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.316 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.316 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:11:27.317 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:11:27.317 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.317 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.317 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.317 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.317 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.317 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.318 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:11:27.318 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:11:27.318 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.318 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.318 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.318 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.319 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.319 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:11:27.319 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:11:27.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.320 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.320 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:11:27.320 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:11:27.320 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.320 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.321 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.321 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.321 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.321 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:11:27.322 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:11:27.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.322 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.322 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:11:27.322 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:11:27.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.323 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.323 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.323 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.324 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:11:27.324 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:11:27.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.324 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.324 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:11:27.325 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:11:27.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.325 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.325 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.325 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.326 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.326 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:11:27.326 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:11:27.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.327 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.327 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:11:27.327 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:11:27.327 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.327 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.328 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:11:27.328 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:11:27.328 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:11:27.328 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:11:27.328 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.328 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.329 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.329 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.329 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:11:27.329 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:11:27.329 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.330 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.330 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.330 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:11:27.330 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.330 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.330 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:11:27.331 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:11:27.331 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.331 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.331 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.331 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.331 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:11:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:11:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:11:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:11:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:11:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 108, 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:11:27.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.342 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:11:27.343 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:11:27.344 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:11:27.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:11:27.346 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:11:27.347 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:11:27.348 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 109, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.349 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.351 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:11:27.353 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.354 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 149, 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:11:27.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:11:27.356 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:11:27.357 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.358 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:11:27.359 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:11:27.360 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:11:27.361 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:11:27.362 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.363 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:11:27.365 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.366 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:51.488968+0000", 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:14.594921+0000", 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:58.222328+0000", 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.368 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:04.735259+0000", 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.369 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:21.139048+0000", 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:08.506447+0000", 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:58.893839+0000", 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.371 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:02.089795+0000", 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:35.890683+0000", 2024-09-16T05:11:27.372 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:21.438658+0000", 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:11:27.373 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:09:59.495208+0000", 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:11:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:38.259572+0000", 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:11:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:11:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 11, 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 20, 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 23, 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [ 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:11:27.378 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 20, 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:11:27.379 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:11:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:11:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:11:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:11:27.380 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:27.380 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:11:27.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:27.402 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:27.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:27.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:27.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:27.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:27.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:27.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:27.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:27.657 INFO:tasks.rados.rados.2.smithi204.stdout:251: done (6 left) 2024-09-16T05:11:27.657 INFO:tasks.rados.rados.2.smithi204.stdout:252: done (5 left) 2024-09-16T05:11:27.657 INFO:tasks.rados.rados.2.smithi204.stdout:253: done (4 left) 2024-09-16T05:11:27.657 INFO:tasks.rados.rados.2.smithi204.stdout:254: done (3 left) 2024-09-16T05:11:27.657 INFO:tasks.rados.rados.2.smithi204.stdout:255: done (2 left) 2024-09-16T05:11:27.657 INFO:tasks.rados.rados.2.smithi204.stdout:256: done (1 left) 2024-09-16T05:11:27.657 INFO:tasks.rados.rados.2.smithi204.stdout:257: done (0 left) 2024-09-16T05:11:27.657 INFO:tasks.rados.rados.2.smithi204.stdout:258: write oid 38 current snap is 26 2024-09-16T05:11:27.658 INFO:tasks.rados.rados.2.smithi204.stdout:258: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-16T05:11:27.663 INFO:tasks.rados.rados.2.smithi204.stdout:258: writing smithi20457400-38 from 467261 to 1184536 tid 1 2024-09-16T05:11:27.669 INFO:tasks.rados.rados.2.smithi204.stdout:258: writing smithi20457400-38 from 1847991 to 2067261 tid 2 2024-09-16T05:11:27.669 INFO:tasks.rados.rados.2.smithi204.stdout:259: read oid 40 snap 16 2024-09-16T05:11:27.669 INFO:tasks.rados.rados.2.smithi204.stdout:259: expect (ObjNum 93 snap 9 seq_num 93) 2024-09-16T05:11:27.669 INFO:tasks.rados.rados.2.smithi204.stdout:260: write (excl) oid 31 current snap is 26 2024-09-16T05:11:27.669 INFO:tasks.rados.rados.2.smithi204.stdout:260: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-16T05:11:27.675 INFO:tasks.rados.rados.2.smithi204.stdout:260: writing smithi20457400-31 from 484068 to 1276592 tid 1 2024-09-16T05:11:27.680 INFO:tasks.rados.rados.2.smithi204.stdout:260: writing smithi20457400-31 from 1906473 to 2084068 tid 2 2024-09-16T05:11:27.680 INFO:tasks.rados.rados.2.smithi204.stdout:261: delete oid 37 current snap is 26 2024-09-16T05:11:27.680 INFO:tasks.rados.rados.2.smithi204.stdout:258: finishing write tid 1 to smithi20457400-38 2024-09-16T05:11:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:258: finishing write tid 2 to smithi20457400-38 2024-09-16T05:11:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:258: finishing write tid 3 to smithi20457400-38 2024-09-16T05:11:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:258: finishing write tid 4 to smithi20457400-38 2024-09-16T05:11:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 147 (ObjNum 123 snap 26 seq_num 123) dirty exists 2024-09-16T05:11:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:258: left oid 38 (ObjNum 123 snap 26 seq_num 123) 2024-09-16T05:11:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:259: expect (ObjNum 93 snap 9 seq_num 93) 2024-09-16T05:11:27.719 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:27.719 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:27.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:27.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:27.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:27 smithi032 ceph-mon[27388]: pgmap v384: 27 pgs: 27 active+clean; 160 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T05:11:27.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:27 smithi032 ceph-mon[27388]: osdmap e155: 12 total, 12 up, 12 in 2024-09-16T05:11:27.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:27 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:27.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:27 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:27.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3176858410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:11:27.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:27 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:27.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:27 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:27.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:260: finishing write tid 1 to smithi20457400-31 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:260: finishing write tid 2 to smithi20457400-31 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:260: finishing write tid 3 to smithi20457400-31 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:260: finishing write tid 4 to smithi20457400-31 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 125 (ObjNum 124 snap 26 seq_num 124) dirty exists 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:260: left oid 31 (ObjNum 124 snap 26 seq_num 124) 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:258: done (3 left) 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:259: done (2 left) 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:260: done (1 left) 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:261: done (0 left) 2024-09-16T05:11:27.897 INFO:tasks.rados.rados.2.smithi204.stdout:262: snap_create 2024-09-16T05:11:27.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:27.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:27 smithi204 ceph-mon[30933]: pgmap v384: 27 pgs: 27 active+clean; 160 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T05:11:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:27 smithi204 ceph-mon[30933]: osdmap e155: 12 total, 12 up, 12 in 2024-09-16T05:11:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:27 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:27 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3176858410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:11:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:27 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:27 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:27 smithi196 ceph-mon[30802]: pgmap v384: 27 pgs: 27 active+clean; 160 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 0 B/s wr, 11 op/s 2024-09-16T05:11:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:27 smithi196 ceph-mon[30802]: osdmap e155: 12 total, 12 up, 12 in 2024-09-16T05:11:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:27 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:27 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3176858410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:11:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:27 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:27 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:28.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:28.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:28.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:28.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:11:28.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:28.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:28.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:28.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:28.303 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:28.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:28.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:11:28.362 INFO:tasks.thrashosds.thrasher:j is {'epoch': 155, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:11:26.641248+0000', 'last_up_change': '2024-09-16T05:09:26.475677+0000', 'last_in_change': '2024-09-16T05:09:05.828775+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '104', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.389999866485596, 'score_stable': 4.389999866485596, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 3.640000104904175, 'raw_score_stable': 3.640000104904175, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 26, 'snap_epoch': 155, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 145, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3508564664}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3508564664}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3508564664}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3508564664}]}, 'public_addr': '172.21.15.32:6819/3508564664', 'cluster_addr': '172.21.15.32:6821/3508564664', 'heartbeat_back_addr': '172.21.15.32:6825/3508564664', 'heartbeat_front_addr': '172.21.15.32:6823/3508564664', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 108, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:51.488968+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:14.594921+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:58.222328+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:04.735259+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:21.139048+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:08.506447+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:58.893839+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:02.089795+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:35.890683+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:21.438658+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:09:59.495208+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:38.259572+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 11, 'length': 1}, {'begin': 20, 'length': 1}, {'begin': 23, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:11:28.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:11:28.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:28.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:28.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:28.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:28.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:28.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:28.569 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:28.597 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:28.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:28.657 INFO:tasks.rados.rados.2.smithi204.stdout:262: done (0 left) 2024-09-16T05:11:28.657 INFO:tasks.rados.rados.2.smithi204.stdout:263: delete oid 19 current snap is 27 2024-09-16T05:11:28.660 INFO:tasks.rados.rados.2.smithi204.stdout:263: done (0 left) 2024-09-16T05:11:28.660 INFO:tasks.rados.rados.2.smithi204.stdout:264: read oid 14 snap -1 2024-09-16T05:11:28.660 INFO:tasks.rados.rados.2.smithi204.stdout:264: expect (ObjNum 122 snap 25 seq_num 122) 2024-09-16T05:11:28.660 INFO:tasks.rados.rados.2.smithi204.stdout:265: write oid 29 current snap is 27 2024-09-16T05:11:28.660 INFO:tasks.rados.rados.2.smithi204.stdout:265: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-16T05:11:28.666 INFO:tasks.rados.rados.2.smithi204.stdout:265: writing smithi20457400-29 from 534489 to 1152760 tid 1 2024-09-16T05:11:28.672 INFO:tasks.rados.rados.2.smithi204.stdout:265: writing smithi20457400-29 from 1565566 to 2134489 tid 2 2024-09-16T05:11:28.672 INFO:tasks.rados.rados.2.smithi204.stdout:266: rollback oid 3 current snap is 27 2024-09-16T05:11:28.672 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 24 2024-09-16T05:11:28.672 INFO:tasks.rados.rados.2.smithi204.stdout:264: expect (ObjNum 122 snap 25 seq_num 122) 2024-09-16T05:11:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:28.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:28.774 INFO:tasks.rados.rados.2.smithi204.stdout:265: finishing write tid 1 to smithi20457400-29 2024-09-16T05:11:28.774 INFO:tasks.rados.rados.2.smithi204.stdout:266: finishing rollback tid 0 to smithi20457400-3 2024-09-16T05:11:28.774 INFO:tasks.rados.rados.2.smithi204.stdout:266: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:266: finishing rollback tid 2 to smithi20457400-3 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 151 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:265: finishing write tid 2 to smithi20457400-29 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:265: finishing write tid 3 to smithi20457400-29 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:265: finishing write tid 4 to smithi20457400-29 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 108 (ObjNum 127 snap 27 seq_num 127) dirty exists 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:265: left oid 29 (ObjNum 127 snap 27 seq_num 127) 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:264: done (2 left) 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:265: done (1 left) 2024-09-16T05:11:28.775 INFO:tasks.rados.rados.2.smithi204.stdout:266: done (0 left) 2024-09-16T05:11:28.776 INFO:tasks.rados.rados.2.smithi204.stdout:267: copy_from oid 5 from oid 39 current snap is 27 2024-09-16T05:11:28.776 INFO:tasks.rados.rados.2.smithi204.stdout:268: read oid 4 snap -1 2024-09-16T05:11:28.776 INFO:tasks.rados.rados.2.smithi204.stdout:268: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:11:28.776 INFO:tasks.rados.rados.2.smithi204.stdout:269: copy_from oid 26 from oid 33 current snap is 27 2024-09-16T05:11:28.776 INFO:tasks.rados.rados.2.smithi204.stdout:270: rollback oid 38 current snap is 27 2024-09-16T05:11:28.776 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 38 to 20 2024-09-16T05:11:28.776 INFO:tasks.rados.rados.2.smithi204.stdout:271: write oid 48 current snap is 27 2024-09-16T05:11:28.776 INFO:tasks.rados.rados.2.smithi204.stdout:271: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-16T05:11:28.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:28.781 INFO:tasks.rados.rados.2.smithi204.stdout:271: writing smithi20457400-48 from 551296 to 1244816 tid 1 2024-09-16T05:11:28.786 INFO:tasks.rados.rados.2.smithi204.stdout:271: writing smithi20457400-48 from 2024048 to 2151296 tid 2 2024-09-16T05:11:28.786 INFO:tasks.rados.rados.2.smithi204.stdout:272: read oid 44 snap -1 2024-09-16T05:11:28.786 INFO:tasks.rados.rados.2.smithi204.stdout:272: expect (ObjNum 111 snap 19 seq_num 111) 2024-09-16T05:11:28.786 INFO:tasks.rados.rados.2.smithi204.stdout:273: snap_remove snap 24 2024-09-16T05:11:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:28.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:28.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:28.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:28.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:28 smithi032 ceph-mon[27388]: osdmap e156: 12 total, 12 up, 12 in 2024-09-16T05:11:28.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:28 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:11:28.911 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:28 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:11:28.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:28 smithi204 ceph-mon[30933]: osdmap e156: 12 total, 12 up, 12 in 2024-09-16T05:11:28.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:28 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:11:28.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:28 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:11:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:28.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:28.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:28 smithi196 ceph-mon[30802]: osdmap e156: 12 total, 12 up, 12 in 2024-09-16T05:11:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:28 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:11:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:28 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:11:29.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:29.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:29.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:29.077 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:29.077 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:29.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:29.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:29.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:29.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:29.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:29.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:29.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:29.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:29.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:29.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:29.561 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:29.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:29.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:11:29.570+0000 7f5f1e39b640 0 --2- 172.21.15.32:0/2673767127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f181008d0 0x7f5f18197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:11:29.659 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:11:29.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:29.663 INFO:tasks.rados.rados.2.smithi204.stdout:273: done (6 left) 2024-09-16T05:11:29.663 INFO:tasks.rados.rados.2.smithi204.stdout:274: write oid 37 current snap is 27 2024-09-16T05:11:29.663 INFO:tasks.rados.rados.2.smithi204.stdout:274: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-16T05:11:29.671 INFO:tasks.rados.rados.2.smithi204.stdout:274: writing smithi20457400-37 from 568103 to 1336872 tid 1 2024-09-16T05:11:29.675 INFO:tasks.rados.rados.2.smithi204.stdout:274: writing smithi20457400-37 from 2082530 to 2168103 tid 2 2024-09-16T05:11:29.675 INFO:tasks.rados.rados.2.smithi204.stdout:275: snap_remove snap 26 2024-09-16T05:11:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:29.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:29.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:29.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:29 smithi204 ceph-mon[30933]: pgmap v387: 27 pgs: 27 active+clean; 157 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 680 KiB/s wr, 5 op/s 2024-09-16T05:11:29.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:29 smithi204 ceph-mon[30933]: osdmap e157: 12 total, 12 up, 12 in 2024-09-16T05:11:29.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/62033061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:29.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2673767127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:29.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:11:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:11:29.755 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:11:29.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:29 smithi032 ceph-mon[27388]: pgmap v387: 27 pgs: 27 active+clean; 157 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 680 KiB/s wr, 5 op/s 2024-09-16T05:11:29.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:29 smithi032 ceph-mon[27388]: osdmap e157: 12 total, 12 up, 12 in 2024-09-16T05:11:29.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/62033061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:29.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2673767127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:29.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:29.800 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:29.800 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:29.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:29.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:29.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:29.822 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:29.842 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:29.842 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:29.862 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:29.863 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:29.884 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:29.884 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:29.907 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:29.907 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:29.912 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:29.933 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:29.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:29.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:29.955 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:29.969 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:29.976 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:29.976 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:29 smithi196 ceph-mon[30802]: pgmap v387: 27 pgs: 27 active+clean; 157 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 680 KiB/s wr, 5 op/s 2024-09-16T05:11:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:29 smithi196 ceph-mon[30802]: osdmap e157: 12 total, 12 up, 12 in 2024-09-16T05:11:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/62033061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2673767127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:29.998 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:29.998 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:30.020 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:30.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:30.041 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:30.042 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:30.064 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:30.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:30.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:30.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:30.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:30.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:30.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:30.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:30.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:30.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:30.433 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-16T05:11:29.659783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":171106839,"num_objects":74,"num_object_clones":31,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":4,"num_read":547,"num_read_kb":391185,"num_write":503,"num_write_kb":167772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2320,"ondisk_log_size":2320,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":558984,"kb_used_data":213792,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124382840,"statfs":{"total":1151940427776,"available":1151368028160,"internally_reserved":0,"allocated":218923008,"data_stored":236975469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3018974,"num_objects":5,"num_object_clones":5,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":15,"num_read_kb":12329,"num_write":29,"num_write_kb":9022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018233"},"pg_stats":[{"pgid":"2.f","version":"156'54","reported_seq":296,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T05:11:28.791983+0000","last_change":"2024-09-16T05:11:26.648670+0000","last_active":"2024-09-16T05:11:28.791983+0000","last_peered":"2024-09-16T05:11:28.791983+0000","last_clean":"2024-09-16T05:11:28.791983+0000","last_became_active":"2024-09-16T05:10:36.956705+0000","last_became_peered":"2024-09-16T05:10:36.956705+0000","last_unstale":"2024-09-16T05:11:28.791983+0000","last_undegraded":"2024-09-16T05:11:28.791983+0000","last_fullsized":"2024-09-16T05:11:28.791983+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:01.688434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015559689999999999,"stat_sum":{"num_bytes":12818829,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":45,"num_read_kb":31573,"num_write":35,"num_write_kb":12547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"b","length":"1"},{"start":"14","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.e","version":"153'59","reported_seq":288,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.684041+0000","last_change":"2024-09-16T05:11:24.852086+0000","last_active":"2024-09-16T05:11:25.684041+0000","last_peered":"2024-09-16T05:11:25.684041+0000","last_clean":"2024-09-16T05:11:25.684041+0000","last_became_active":"2024-09-16T05:10:37.001925+0000","last_became_peered":"2024-09-16T05:10:37.001925+0000","last_unstale":"2024-09-16T05:11:25.684041+0000","last_undegraded":"2024-09-16T05:11:25.684041+0000","last_fullsized":"2024-09-16T05:11:25.684041+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:04.415528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001869342,"stat_sum":{"num_bytes":18410481,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":48532,"num_write":43,"num_write_kb":14347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.d","version":"153'103","reported_seq":329,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T05:11:24.707263+0000","last_change":"2024-09-16T05:11:24.707441+0000","last_active":"2024-09-16T05:11:24.707263+0000","last_peered":"2024-09-16T05:11:24.707263+0000","last_clean":"2024-09-16T05:11:24.707263+0000","last_became_active":"2024-09-16T05:10:36.957439+0000","last_became_peered":"2024-09-16T05:10:36.957439+0000","last_unstale":"2024-09-16T05:11:24.707263+0000","last_undegraded":"2024-09-16T05:11:24.707263+0000","last_fullsized":"2024-09-16T05:11:24.707263+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:42.328911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045618569999999999,"stat_sum":{"num_bytes":21395080,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":45225,"num_write":66,"num_write_kb":25855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.c","version":"153'54","reported_seq":278,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.679274+0000","last_change":"2024-09-16T05:11:26.679415+0000","last_active":"2024-09-16T05:11:26.679274+0000","last_peered":"2024-09-16T05:11:26.679274+0000","last_clean":"2024-09-16T05:11:26.679274+0000","last_became_active":"2024-09-16T05:10:36.957767+0000","last_became_peered":"2024-09-16T05:10:36.957767+0000","last_unstale":"2024-09-16T05:11:26.679274+0000","last_undegraded":"2024-09-16T05:11:26.679274+0000","last_fullsized":"2024-09-16T05:11:26.679274+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:41.118048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404485,"stat_sum":{"num_bytes":9361400,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":13504,"num_write":32,"num_write_kb":10695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"14","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.8","version":"155'35","reported_seq":256,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.663339+0000","last_change":"2024-09-16T05:11:26.645133+0000","last_active":"2024-09-16T05:11:26.663339+0000","last_peered":"2024-09-16T05:11:26.663339+0000","last_clean":"2024-09-16T05:11:26.663339+0000","last_became_active":"2024-09-16T05:10:36.959606+0000","last_became_peered":"2024-09-16T05:10:36.959606+0000","last_unstale":"2024-09-16T05:11:26.663339+0000","last_undegraded":"2024-09-16T05:11:26.663339+0000","last_fullsized":"2024-09-16T05:11:26.663339+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:46.254425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044980500000000001,"stat_sum":{"num_bytes":7069122,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":7284,"num_write":19,"num_write_kb":7365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b","length":"1"},{"start":"14","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.3","version":"153'7","reported_seq":218,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.856205+0000","last_change":"2024-09-16T05:11:24.851779+0000","last_active":"2024-09-16T05:11:25.856205+0000","last_peered":"2024-09-16T05:11:25.856205+0000","last_clean":"2024-09-16T05:11:25.856205+0000","last_became_active":"2024-09-16T05:10:36.956582+0000","last_became_peered":"2024-09-16T05:10:36.956582+0000","last_unstale":"2024-09-16T05:11:25.856205+0000","last_undegraded":"2024-09-16T05:11:25.856205+0000","last_fullsized":"2024-09-16T05:11:25.856205+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:50.350105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019543680000000002,"stat_sum":{"num_bytes":2285719,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":503,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.644864+0000","last_change":"2024-09-16T05:10:41.789086+0000","last_active":"2024-09-16T05:11:26.644864+0000","last_peered":"2024-09-16T05:11:26.644864+0000","last_clean":"2024-09-16T05:11:26.644864+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:11:26.644864+0000","last_undegraded":"2024-09-16T05:11:26.644864+0000","last_fullsized":"2024-09-16T05:11:26.644864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:41.788991+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:41.788991+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:41.897583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"112'3","reported_seq":220,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.856172+0000","last_change":"2024-09-16T05:11:24.850192+0000","last_active":"2024-09-16T05:11:25.856172+0000","last_peered":"2024-09-16T05:11:25.856172+0000","last_clean":"2024-09-16T05:11:25.856172+0000","last_became_active":"2024-09-16T05:10:36.956451+0000","last_became_peered":"2024-09-16T05:10:36.956451+0000","last_unstale":"2024-09-16T05:11:25.856172+0000","last_undegraded":"2024-09-16T05:11:25.856172+0000","last_fullsized":"2024-09-16T05:11:25.856172+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:55.575592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038276500000000003,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":12397,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":503,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.856143+0000","last_change":"2024-09-16T05:10:46.732985+0000","last_active":"2024-09-16T05:11:25.856143+0000","last_peered":"2024-09-16T05:11:25.856143+0000","last_clean":"2024-09-16T05:11:25.856143+0000","last_became_active":"2024-09-16T05:10:38.972381+0000","last_became_peered":"2024-09-16T05:10:38.972381+0000","last_unstale":"2024-09-16T05:11:25.856143+0000","last_undegraded":"2024-09-16T05:11:25.856143+0000","last_fullsized":"2024-09-16T05:11:25.856143+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:46.732896+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:46.732896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:47.973041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":507,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.644615+0000","last_change":"2024-09-16T05:10:42.783065+0000","last_active":"2024-09-16T05:11:26.644615+0000","last_peered":"2024-09-16T05:11:26.644615+0000","last_clean":"2024-09-16T05:11:26.644615+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:11:26.644615+0000","last_undegraded":"2024-09-16T05:11:26.644615+0000","last_fullsized":"2024-09-16T05:11:26.644615+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:42.782976+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:42.782976+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:44.473801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"154'28","reported_seq":245,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.948651+0000","last_change":"2024-09-16T05:11:24.851880+0000","last_active":"2024-09-16T05:11:25.948651+0000","last_peered":"2024-09-16T05:11:25.948651+0000","last_clean":"2024-09-16T05:11:25.948651+0000","last_became_active":"2024-09-16T05:10:37.001937+0000","last_became_peered":"2024-09-16T05:10:37.001937+0000","last_unstale":"2024-09-16T05:11:25.948651+0000","last_undegraded":"2024-09-16T05:11:25.948651+0000","last_fullsized":"2024-09-16T05:11:25.948651+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:45.313919+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.001676273,"stat_sum":{"num_bytes":4821688,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":11502,"num_write":13,"num_write_kb":10469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":508,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.680478+0000","last_change":"2024-09-16T05:11:19.419444+0000","last_active":"2024-09-16T05:11:26.680478+0000","last_peered":"2024-09-16T05:11:26.680478+0000","last_clean":"2024-09-16T05:11:26.680478+0000","last_became_active":"2024-09-16T05:11:19.419149+0000","last_became_peered":"2024-09-16T05:11:19.419149+0000","last_unstale":"2024-09-16T05:11:26.680478+0000","last_undegraded":"2024-09-16T05:11:26.680478+0000","last_fullsized":"2024-09-16T05:11:26.680478+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.849175+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.849175+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:52.137143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"121'10","reported_seq":228,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.644984+0000","last_change":"2024-09-16T05:11:26.645188+0000","last_active":"2024-09-16T05:11:26.644984+0000","last_peered":"2024-09-16T05:11:26.644984+0000","last_clean":"2024-09-16T05:11:26.644984+0000","last_became_active":"2024-09-16T05:10:36.983391+0000","last_became_peered":"2024-09-16T05:10:36.983391+0000","last_unstale":"2024-09-16T05:11:26.644984+0000","last_undegraded":"2024-09-16T05:11:26.644984+0000","last_fullsized":"2024-09-16T05:11:26.644984+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:37.431249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051184699999999995,"stat_sum":{"num_bytes":3142876,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":1825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b","length":"1"},{"start":"14","length":"1"},{"start":"17","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.856431+0000","last_change":"2024-09-16T05:11:24.704111+0000","last_active":"2024-09-16T05:11:25.856431+0000","last_peered":"2024-09-16T05:11:25.856431+0000","last_clean":"2024-09-16T05:11:25.856431+0000","last_became_active":"2024-09-16T05:11:21.676078+0000","last_became_peered":"2024-09-16T05:11:21.676078+0000","last_unstale":"2024-09-16T05:11:25.856431+0000","last_undegraded":"2024-09-16T05:11:25.856431+0000","last_fullsized":"2024-09-16T05:11:25.856431+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:42.231191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000340804,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"b","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":465,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.679061+0000","last_change":"2024-09-16T05:10:41.421609+0000","last_active":"2024-09-16T05:11:26.679061+0000","last_peered":"2024-09-16T05:11:26.679061+0000","last_clean":"2024-09-16T05:11:26.679061+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:11:26.679061+0000","last_undegraded":"2024-09-16T05:11:26.679061+0000","last_fullsized":"2024-09-16T05:11:26.679061+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:41.421501+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:41.421501+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:51.416537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":223,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T05:11:24.698599+0000","last_change":"2024-09-16T05:11:24.698802+0000","last_active":"2024-09-16T05:11:24.698599+0000","last_peered":"2024-09-16T05:11:24.698599+0000","last_clean":"2024-09-16T05:11:24.698599+0000","last_became_active":"2024-09-16T05:10:36.957473+0000","last_became_peered":"2024-09-16T05:10:36.957473+0000","last_unstale":"2024-09-16T05:11:24.698599+0000","last_undegraded":"2024-09-16T05:11:24.698599+0000","last_fullsized":"2024-09-16T05:11:24.698599+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:05.678441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039616599999999999,"stat_sum":{"num_bytes":2857157,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13147,"num_write":8,"num_write_kb":2442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":504,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.856170+0000","last_change":"2024-09-16T05:10:44.788217+0000","last_active":"2024-09-16T05:11:25.856170+0000","last_peered":"2024-09-16T05:11:25.856170+0000","last_clean":"2024-09-16T05:11:25.856170+0000","last_became_active":"2024-09-16T05:10:38.972150+0000","last_became_peered":"2024-09-16T05:10:38.972150+0000","last_unstale":"2024-09-16T05:11:25.856170+0000","last_undegraded":"2024-09-16T05:11:25.856170+0000","last_fullsized":"2024-09-16T05:11:25.856170+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.788114+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.788114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:57.373836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"157'155","reported_seq":399,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T05:11:28.777788+0000","last_change":"2024-09-16T05:11:26.648799+0000","last_active":"2024-09-16T05:11:28.777788+0000","last_peered":"2024-09-16T05:11:28.777788+0000","last_clean":"2024-09-16T05:11:28.777788+0000","last_became_active":"2024-09-16T05:10:36.953485+0000","last_became_peered":"2024-09-16T05:10:36.953485+0000","last_unstale":"2024-09-16T05:11:28.777788+0000","last_undegraded":"2024-09-16T05:11:28.777788+0000","last_fullsized":"2024-09-16T05:11:28.777788+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:44.721517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017064350000000001,"stat_sum":{"num_bytes":31740796,"num_objects":15,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":92,"num_read_kb":78564,"num_write":83,"num_write_kb":28210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"b","length":"1"},{"start":"14","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":504,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.856170+0000","last_change":"2024-09-16T05:10:45.767820+0000","last_active":"2024-09-16T05:11:25.856170+0000","last_peered":"2024-09-16T05:11:25.856170+0000","last_clean":"2024-09-16T05:11:25.856170+0000","last_became_active":"2024-09-16T05:10:38.972286+0000","last_became_peered":"2024-09-16T05:10:38.972286+0000","last_unstale":"2024-09-16T05:11:25.856170+0000","last_undegraded":"2024-09-16T05:11:25.856170+0000","last_fullsized":"2024-09-16T05:11:25.856170+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.767717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.767717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:08.779630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"153'66","reported_seq":279,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T05:11:24.702165+0000","last_change":"2024-09-16T05:11:24.702320+0000","last_active":"2024-09-16T05:11:24.702165+0000","last_peered":"2024-09-16T05:11:24.702165+0000","last_clean":"2024-09-16T05:11:24.702165+0000","last_became_active":"2024-09-16T05:10:36.956394+0000","last_became_peered":"2024-09-16T05:10:36.956394+0000","last_unstale":"2024-09-16T05:11:24.702165+0000","last_undegraded":"2024-09-16T05:11:24.702165+0000","last_fullsized":"2024-09-16T05:11:24.702165+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:57.370219+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.0039173790000000003,"stat_sum":{"num_bytes":7553989,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":12479,"num_write":37,"num_write_kb":9260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":416,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.856266+0000","last_change":"2024-09-16T05:11:21.676392+0000","last_active":"2024-09-16T05:11:25.856266+0000","last_peered":"2024-09-16T05:11:25.856266+0000","last_clean":"2024-09-16T05:11:25.856266+0000","last_became_active":"2024-09-16T05:11:21.676107+0000","last_became_peered":"2024-09-16T05:11:21.676107+0000","last_unstale":"2024-09-16T05:11:25.856266+0000","last_undegraded":"2024-09-16T05:11:25.856266+0000","last_fullsized":"2024-09-16T05:11:25.856266+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.115392+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.115392+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:26.765525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"157'61","reported_seq":297,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T05:11:28.785924+0000","last_change":"2024-09-16T05:11:26.647350+0000","last_active":"2024-09-16T05:11:28.785924+0000","last_peered":"2024-09-16T05:11:28.785924+0000","last_clean":"2024-09-16T05:11:28.785924+0000","last_became_active":"2024-09-16T05:10:36.956652+0000","last_became_peered":"2024-09-16T05:10:36.956652+0000","last_unstale":"2024-09-16T05:11:28.785924+0000","last_undegraded":"2024-09-16T05:11:28.785924+0000","last_fullsized":"2024-09-16T05:11:28.785924+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:45.945792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026069400000000001,"stat_sum":{"num_bytes":15082944,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":35,"num_read_kb":31365,"num_write":37,"num_write_kb":14573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"b","length":"1"},{"start":"14","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":502,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T05:11:25.856049+0000","last_change":"2024-09-16T05:10:45.437354+0000","last_active":"2024-09-16T05:11:25.856049+0000","last_peered":"2024-09-16T05:11:25.856049+0000","last_clean":"2024-09-16T05:11:25.856049+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:11:25.856049+0000","last_undegraded":"2024-09-16T05:11:25.856049+0000","last_fullsized":"2024-09-16T05:11:25.856049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.437261+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.437261+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:05.011089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"155'105","reported_seq":340,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.680511+0000","last_change":"2024-09-16T05:11:26.680633+0000","last_active":"2024-09-16T05:11:26.680511+0000","last_peered":"2024-09-16T05:11:26.680511+0000","last_clean":"2024-09-16T05:11:26.680511+0000","last_became_active":"2024-09-16T05:10:37.002200+0000","last_became_peered":"2024-09-16T05:10:37.002200+0000","last_unstale":"2024-09-16T05:11:26.680511+0000","last_undegraded":"2024-09-16T05:11:26.680511+0000","last_fullsized":"2024-09-16T05:11:26.680511+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:45.952428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001627297,"stat_sum":{"num_bytes":23502512,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":46993,"num_write":66,"num_write_kb":20697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"b","length":"1"},{"start":"14","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":507,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T05:11:26.644633+0000","last_change":"2024-09-16T05:10:43.759254+0000","last_active":"2024-09-16T05:11:26.644633+0000","last_peered":"2024-09-16T05:11:26.644633+0000","last_clean":"2024-09-16T05:11:26.644633+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:11:26.644633+0000","last_undegraded":"2024-09-16T05:11:26.644633+0000","last_fullsized":"2024-09-16T05:11:26.644633+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:43.759130+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:43.759130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:55.539739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"153'42","reported_seq":270,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T05:11:24.701047+0000","last_change":"2024-09-16T05:11:24.701607+0000","last_active":"2024-09-16T05:11:24.701047+0000","last_peered":"2024-09-16T05:11:24.701047+0000","last_clean":"2024-09-16T05:11:24.701047+0000","last_became_active":"2024-09-16T05:10:36.958530+0000","last_became_peered":"2024-09-16T05:10:36.958530+0000","last_unstale":"2024-09-16T05:11:24.701047+0000","last_undegraded":"2024-09-16T05:11:24.701047+0000","last_fullsized":"2024-09-16T05:11:24.701047+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:03.542315+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0031954800000000001,"stat_sum":{"num_bytes":8184910,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":38607,"num_write":30,"num_write_kb":7179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"1"},{"start":"e","length":"1"},{"start":"11","length":"1"},{"start":"13","length":"1"},{"start":"17","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":502,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T05:11:24.702397+0000","last_change":"2024-09-16T05:10:44.010384+0000","last_active":"2024-09-16T05:11:24.702397+0000","last_peered":"2024-09-16T05:11:24.702397+0000","last_clean":"2024-09-16T05:11:24.702397+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:11:24.702397+0000","last_undegraded":"2024-09-16T05:11:24.702397+0000","last_fullsized":"2024-09-16T05:11:24.702397+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.010311+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.010311+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:55.319454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":170462801,"num_objects":72,"num_object_clones":31,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":4,"num_read":536,"num_read_kb":391175,"num_write":487,"num_write_kb":167505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192528384,"data_stored":221121617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54804,"kb_used_data":27768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93690348,"statfs":{"total":95995035648,"available":95938916352,"internally_reserved":0,"allocated":28434432,"data_stored":32086351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285600,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":30108,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93715044,"statfs":{"total":95995035648,"available":95964205056,"internally_reserved":0,"allocated":3149824,"data_stored":2353246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514535,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56524,"kb_used_data":25328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688628,"statfs":{"total":95995035648,"available":95937155072,"internally_reserved":0,"allocated":25935872,"data_stored":27730539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841578,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63132,"kb_used_data":36096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93682020,"statfs":{"total":95995035648,"available":95930388480,"internally_reserved":0,"allocated":36962304,"data_stored":40457827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103215,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43092,"kb_used_data":11900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93702060,"statfs":{"total":95995035648,"available":95950909440,"internally_reserved":0,"allocated":12185600,"data_stored":11697182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32900,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712252,"statfs":{"total":95995035648,"available":95961346048,"internally_reserved":0,"allocated":1753088,"data_stored":968176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757305,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":30984,"kb_used_data":3956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93714168,"statfs":{"total":95995035648,"available":95963308032,"internally_reserved":0,"allocated":4050944,"data_stored":3248891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40868,"kb_used_data":13832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93704284,"statfs":{"total":95995035648,"available":95953186816,"internally_reserved":0,"allocated":14163968,"data_stored":15260192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117060,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55304,"kb_used_data":28276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93689848,"statfs":{"total":95995035648,"available":95938404352,"internally_reserved":0,"allocated":28954624,"data_stored":31525215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313290,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47096,"kb_used_data":15900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93698056,"statfs":{"total":95995035648,"available":95946809344,"internally_reserved":0,"allocated":16281600,"data_stored":16080469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42256,"kb_used_data":15224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93702896,"statfs":{"total":95995035648,"available":95951765504,"internally_reserved":0,"allocated":15589376,"data_stored":17774600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61916,"kb_used_data":30724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683236,"statfs":{"total":95995035648,"available":95931633664,"internally_reserved":0,"allocated":31461376,"data_stored":37792781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25608192,"data_stored":30171092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11612160,"data_stored":12437991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14323712,"data_stored":14911045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27189248,"data_stored":30548583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12423168,"data_stored":16263078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2105344,"data_stored":2087971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10420224,"data_stored":10720550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":36900864,"data_stored":45034827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24195072,"data_stored":26770867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1409024,"data_stored":1393574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26341376,"data_stored":30782039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:11:30.435 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [4, 6, 9, 3] 2024-09-16T05:11:30.435 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.0', '4', '6', '9', '3'] 2024-09-16T05:11:30.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 1.0 4 6 9 3 2024-09-16T05:11:30.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:30.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:30.509 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:11:30.509 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:11:30.509 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:11:30.509 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:11:30.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:30.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:30.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:275: done (7 left) 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:276: copy_from oid 8 from oid 18 current snap is 27 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:269: finishing copy_from to smithi20457400-26 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:269: got expected ENOENT (src dne) 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:269: finishing copy_from racing read to smithi20457400-26 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:270: finishing rollback tid 0 to smithi20457400-38 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:270: finishing rollback tid 1 to smithi20457400-38 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:270: finishing rollback tid 2 to smithi20457400-38 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 155 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-16T05:11:30.647 INFO:tasks.rados.rados.2.smithi204.stdout:268: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T05:11:30.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:30 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:30 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:30 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:30 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:11:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/62033061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:30 smithi204 ceph-mon[30933]: osdmap e158: 12 total, 12 up, 12 in 2024-09-16T05:11:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:30 smithi204 ceph-mon[30933]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:30 smithi204 ceph-mon[30933]: osdmap e159: 12 total, 12 up, 12 in 2024-09-16T05:11:30.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:30.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:271: finishing write tid 1 to smithi20457400-48 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:267: finishing copy_from racing read to smithi20457400-5 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:271: finishing write tid 2 to smithi20457400-48 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:268: done (7 left) 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:269: done (6 left) 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:270: done (5 left) 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:277: copy_from oid 2 from oid 34 current snap is 27 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:271: finishing write tid 3 to smithi20457400-48 2024-09-16T05:11:30.784 INFO:tasks.rados.rados.2.smithi204.stdout:271: finishing write tid 4 to smithi20457400-48 2024-09-16T05:11:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 60 (ObjNum 128 snap 27 seq_num 128) dirty exists 2024-09-16T05:11:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:271: left oid 48 (ObjNum 128 snap 27 seq_num 128) 2024-09-16T05:11:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:267: finishing copy_from to smithi20457400-5 2024-09-16T05:11:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 77 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:11:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:272: expect (ObjNum 111 snap 19 seq_num 111) 2024-09-16T05:11:30.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:30.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:30.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:30.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:30 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:30 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:30 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:30 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:11:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/62033061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:30 smithi032 ceph-mon[27388]: osdmap e158: 12 total, 12 up, 12 in 2024-09-16T05:11:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:30 smithi032 ceph-mon[27388]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:30 smithi032 ceph-mon[27388]: osdmap e159: 12 total, 12 up, 12 in 2024-09-16T05:11:30.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:30.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:30.961 INFO:tasks.rados.rados.2.smithi204.stdout:274: finishing write tid 1 to smithi20457400-37 2024-09-16T05:11:30.961 INFO:tasks.rados.rados.2.smithi204.stdout:274: finishing write tid 2 to smithi20457400-37 2024-09-16T05:11:30.961 INFO:tasks.rados.rados.2.smithi204.stdout:267: done (5 left) 2024-09-16T05:11:30.962 INFO:tasks.rados.rados.2.smithi204.stdout:271: done (4 left) 2024-09-16T05:11:30.962 INFO:tasks.rados.rados.2.smithi204.stdout:272: done (3 left) 2024-09-16T05:11:30.962 INFO:tasks.rados.rados.2.smithi204.stdout:278: write (excl) oid 40 current snap is 27 2024-09-16T05:11:30.962 INFO:tasks.rados.rados.2.smithi204.stdout:274: finishing write tid 3 to smithi20457400-37 2024-09-16T05:11:30.963 INFO:tasks.rados.rados.2.smithi204.stdout:278: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-16T05:11:30.967 INFO:tasks.rados.rados.2.smithi204.stdout:278: writing smithi20457400-40 from 584910 to 1145281 tid 1 2024-09-16T05:11:30.973 INFO:tasks.rados.rados.2.smithi204.stdout:278: writing smithi20457400-40 from 1857365 to 2184910 tid 2 2024-09-16T05:11:30.973 INFO:tasks.rados.rados.2.smithi204.stdout:279: write oid 48 current snap is 27 2024-09-16T05:11:30.973 INFO:tasks.rados.rados.2.smithi204.stdout:279: seq_num 131 ranges {601717=635620,1799494=402223} 2024-09-16T05:11:30.979 INFO:tasks.rados.rados.2.smithi204.stdout:279: writing smithi20457400-48 from 601717 to 1237337 tid 1 2024-09-16T05:11:30.985 INFO:tasks.rados.rados.2.smithi204.stdout:279: writing smithi20457400-48 from 1799494 to 2201717 tid 2 2024-09-16T05:11:30.985 INFO:tasks.rados.rados.2.smithi204.stdout:280: snap_remove snap 22 2024-09-16T05:11:30.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:30 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:30.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:30 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:30 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:30 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:11:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/62033061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:30 smithi196 ceph-mon[30802]: osdmap e158: 12 total, 12 up, 12 in 2024-09-16T05:11:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:30 smithi196 ceph-mon[30802]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:30 smithi196 ceph-mon[30802]: osdmap e159: 12 total, 12 up, 12 in 2024-09-16T05:11:31.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:31.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:31.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:31.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:31.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:11:31.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:31.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:31.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:31.317 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:31.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:31.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:31.422 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:31.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:31.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:31.530 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:31.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:31.568 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:31.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:31.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:280: done (5 left) 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:281: snap_create 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:274: finishing write tid 4 to smithi20457400-37 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 59 (ObjNum 129 snap 27 seq_num 129) dirty exists 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:274: left oid 37 (ObjNum 129 snap 27 seq_num 129) 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:276: finishing copy_from to smithi20457400-8 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:276: got expected ENOENT (src dne) 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:276: finishing copy_from racing read to smithi20457400-8 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:277: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:11:31.673 INFO:tasks.rados.rados.2.smithi204.stdout:277: finishing copy_from to smithi20457400-2 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 63 (ObjNum 106 snap 18 seq_num 106) dirty exists 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:278: finishing write tid 1 to smithi20457400-40 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:278: finishing write tid 2 to smithi20457400-40 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:278: finishing write tid 3 to smithi20457400-40 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:278: finishing write tid 4 to smithi20457400-40 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 65 (ObjNum 130 snap 27 seq_num 130) dirty exists 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:278: left oid 40 (ObjNum 130 snap 27 seq_num 130) 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:279: finishing write tid 1 to smithi20457400-48 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:279: finishing write tid 2 to smithi20457400-48 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:279: finishing write tid 3 to smithi20457400-48 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:279: finishing write tid 4 to smithi20457400-48 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 71 (ObjNum 131 snap 27 seq_num 131) dirty exists 2024-09-16T05:11:31.674 INFO:tasks.rados.rados.2.smithi204.stdout:279: left oid 48 (ObjNum 131 snap 27 seq_num 131) 2024-09-16T05:11:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:31.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:31.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:31.786 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:31 smithi032 ceph-mon[27388]: pgmap v390: 27 pgs: 27 active+clean; 163 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-16T05:11:31.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:31.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:31.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:31.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:31 smithi204 ceph-mon[30933]: pgmap v390: 27 pgs: 27 active+clean; 163 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-16T05:11:31.869 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:11:31.870 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:11:31.870 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:11:31.870 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:11:31.870 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:11:31.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:31.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:31.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:31.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:31 smithi196 ceph-mon[30802]: pgmap v390: 27 pgs: 27 active+clean; 163 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-16T05:11:32.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:32.073 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:32.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:32.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:32.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:32.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:32.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:32.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:32.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:32.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:32.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:32.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:32.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:11:32.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:32.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:32.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:32.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:set 1.0 pg_upmap_items mapping to [4->6,9->3] 2024-09-16T05:11:32.678 INFO:tasks.rados.rados.2.smithi204.stdout:274: done (5 left) 2024-09-16T05:11:32.678 INFO:tasks.rados.rados.2.smithi204.stdout:276: done (4 left) 2024-09-16T05:11:32.678 INFO:tasks.rados.rados.2.smithi204.stdout:277: done (3 left) 2024-09-16T05:11:32.678 INFO:tasks.rados.rados.2.smithi204.stdout:278: done (2 left) 2024-09-16T05:11:32.679 INFO:tasks.rados.rados.2.smithi204.stdout:279: done (1 left) 2024-09-16T05:11:32.679 INFO:tasks.rados.rados.2.smithi204.stdout:281: done (0 left) 2024-09-16T05:11:32.679 INFO:tasks.rados.rados.2.smithi204.stdout:282: snap_create 2024-09-16T05:11:32.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:32.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:32.751 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:32.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:32.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:32.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:32.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:32 smithi204 ceph-mon[30933]: osdmap e160: 12 total, 12 up, 12 in 2024-09-16T05:11:32.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:32 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2899144652' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [4, 6, 9, 3]}]: dispatch 2024-09-16T05:11:32.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:11:32.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:32 smithi032 ceph-mon[27388]: osdmap e160: 12 total, 12 up, 12 in 2024-09-16T05:11:32.928 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:32 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2899144652' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [4, 6, 9, 3]}]: dispatch 2024-09-16T05:11:32.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:32.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:32.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:32.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:32 smithi196 ceph-mon[30802]: osdmap e160: 12 total, 12 up, 12 in 2024-09-16T05:11:32.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:32 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2899144652' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [4, 6, 9, 3]}]: dispatch 2024-09-16T05:11:33.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:33.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:33.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:33.205 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:33.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:33.223 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:19 2.b e112/107 2:d7c71ae7:::smithi20457400-5:head v 112'3, mlcod=0'0)", 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 50.156636333000002, 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19579059400000001, 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041248+0000", 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5799999999999999e-07 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041253+0000", 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4500000000000003e-06 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236743+0000", 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19548974399999999 2024-09-16T05:11:33.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236749+0000", 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6780000000000003e-06 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236795+0000", 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5164999999999999e-05 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236811+0000", 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6453999999999999e-05 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237007+0000", 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019572799999999999 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237037+0000", 2024-09-16T05:11:33.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0716999999999997e-05 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:115 2.a 2:580e4d0e:::smithi20457400-29:head [setxattr _header (56) in=7b,truncate 2470596] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.864691385999997, 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032560466000000003, 2024-09-16T05:11:33.229 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:11:33.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.77e-07 2024-09-16T05:11:33.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333194+0000", 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9580000000000001e-06 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333200+0000", 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3130000000000004e-06 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333204+0000", 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2710000000000001e-06 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334135+0000", 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093147499999999997 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334189+0000", 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3374000000000001e-05 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334274+0000", 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4987999999999993e-05 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:33.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334496+0000", 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022271300000000001 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365655+0000", 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031158857000000002 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365694+0000", 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9187999999999998e-05 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365752+0000", 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:116 2.a 2:580e4d0e:::smithi20457400-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.86467408, 2024-09-16T05:11:33.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032766964000000003, 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:11:33.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6e-07 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333211+0000", 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.221e-06 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333214+0000", 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.844e-06 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333218+0000", 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3050000000000001e-06 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334432+0000", 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012139379999999999 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334485+0000", 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2865000000000003e-05 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365804+0000", 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031319393000000001 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365881+0000", 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6759000000000001e-05 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365976+0000", 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5378999999999996e-05 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.241 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:127 2.a 2:5ae26c04:::smithi20457400-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.369006+0000", 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.828877433000002, 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021134482, 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.242 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 127 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369006+0000", 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369006+0000", 2024-09-16T05:11:33.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369006+0000", 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3399999999999999e-07 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369010+0000", 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1540000000000002e-06 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369015+0000", 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7950000000000004e-06 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369019+0000", 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.258e-06 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369520+0000", 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050143400000000004 2024-09-16T05:11:33.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369568+0000", 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8072000000000003e-05 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369647+0000", 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8594e-05 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369876+0000", 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022953599999999999 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390070+0000", 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02019425 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390104+0000", 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3599999999999997e-05 2024-09-16T05:11:33.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390140+0000", 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6155000000000002e-05 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:128 2.a 2:5ae26c04:::smithi20457400-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.828856373999997, 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021253916000000001, 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.248 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 128 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3099999999999998e-07 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369029+0000", 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9209999999999999e-06 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369036+0000", 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3309999999999998e-06 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369040+0000", 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0069999999999999e-06 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369803+0000", 2024-09-16T05:11:33.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076296199999999995 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369852+0000", 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8396000000000002e-05 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390172+0000", 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020319918999999999 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390220+0000", 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7936000000000003e-05 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390281+0000", 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1013e-05 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.253 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:133 2.a 2:5f916a06:::smithi20457400-34:head [write 554631~548629,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.388544+0000", 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.809339414, 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0083446630000000004, 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 133 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.254 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.388544+0000", 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.388544+0000", 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.388545+0000", 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.136e-06 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.388785+0000", 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000239788 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.388800+0000", 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5274e-05 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.388807+0000", 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0539999999999997e-06 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.388858+0000", 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0943000000000003e-05 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.388947+0000", 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8603999999999994e-05 2024-09-16T05:11:33.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.389037+0000", 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0094999999999997e-05 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.389993+0000", 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00095666700000000002 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.396834+0000", 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068403309999999998 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.396865+0000", 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1535000000000002e-05 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.396888+0000", 2024-09-16T05:11:33.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3235999999999999e-05 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:135 2.a 2:5f916a06:::smithi20457400-34:head [setxattr _header (56) in=7b,truncate 2554631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.395029+0000", 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.802854578000002, 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015111792000000001, 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.260 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 135 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395029+0000", 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395029+0000", 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395029+0000", 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3900000000000005e-07 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395033+0000", 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7720000000000001e-06 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395040+0000", 2024-09-16T05:11:33.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9369999999999998e-06 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395047+0000", 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5660000000000001e-06 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395556+0000", 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050895400000000005 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395609+0000", 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2451000000000003e-05 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395697+0000", 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8098999999999998e-05 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395915+0000", 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000217934 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410037+0000", 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014121984000000001 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410081+0000", 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4338999999999997e-05 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410140+0000", 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9216999999999999e-05 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:136 2.a 2:5f916a06:::smithi20457400-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.395062+0000", 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.802821365, 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015201183, 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.266 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 136 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395062+0000", 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395062+0000", 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395062+0000", 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9200000000000001e-07 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395064+0000", 2024-09-16T05:11:33.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1749999999999999e-06 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395072+0000", 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5460000000000003e-06 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395077+0000", 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8169999999999998e-06 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395859+0000", 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00078203300000000001 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395903+0000", 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4517000000000001e-05 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410164+0000", 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014260291 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410202+0000", 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8124000000000003e-05 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410263+0000", 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1187999999999998e-05 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:137 2.a 2:53edbf55:::smithi20457400-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.401462+0000", 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.796420802999997, 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010163180000000001, 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 137 2024-09-16T05:11:33.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401462+0000", 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401462+0000", 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401463+0000", 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4099999999999997e-07 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401585+0000", 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012222599999999999 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401597+0000", 2024-09-16T05:11:33.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1457999999999999e-05 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401602+0000", 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3900000000000001e-06 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401642+0000", 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9591000000000001e-05 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401712+0000", 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9979000000000002e-05 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.401808+0000", 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6106000000000004e-05 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.402544+0000", 2024-09-16T05:11:33.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00073649999999999996 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.411558+0000", 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0090137800000000007 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.411590+0000", 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2143e-05 2024-09-16T05:11:33.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.411626+0000", 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5166000000000002e-05 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:277 2.a 2:5827f14d:::smithi20457400-12:head [copy-from ver 0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:11:33.278 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.171990774999998, 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38105511399999997, 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 277 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025894+0000", 2024-09-16T05:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.683e-06 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025900+0000", 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7740000000000001e-06 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025917+0000", 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6742999999999999e-05 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025927+0000", 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.978e-06 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025988+0000", 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1574000000000004e-05 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026078+0000", 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0326000000000002e-05 2024-09-16T05:11:33.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.406948+0000", 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38086903599999999 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:278 2.a 2:5827f14d:::smithi20457400-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.171935367000003, 2024-09-16T05:11:33.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38122975799999997, 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 278 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5899999999999996e-07 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025951+0000", 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7819999999999998e-06 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025959+0000", 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1019999999999997e-06 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025966+0000", 2024-09-16T05:11:33.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.409e-06 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026245+0000", 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027952800000000002 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026298+0000", 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2747999999999997e-05 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407054+0000", 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.380756119 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407114+0000", 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0350999999999999e-05 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407178+0000", 2024-09-16T05:11:33.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3159999999999998e-05 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:275 2.b e117/107 2:d7c71ae7:::smithi20457400-5:head v 117'20, mlcod=116'15)", 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.171239696999997, 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072246237000000005, 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.289 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026644+0000", 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6000000000000003e-07 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026651+0000", 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0210000000000002e-06 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098551+0000", 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.071899669999999999 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098559+0000", 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.028e-06 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098612+0000", 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3485e-05 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098628+0000", 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5206e-05 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098860+0000", 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023204700000000001 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098890+0000", 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.012e-05 2024-09-16T05:11:33.293 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:286 2.b e117/107 2:da4f6049:::smithi20457400-49:head v 117'22, mlcod=117'17)", 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.099362005000003, 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.24158528100000001, 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.294 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098523+0000", 2024-09-16T05:11:33.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.508e-06 2024-09-16T05:11:33.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098530+0000", 2024-09-16T05:11:33.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.075e-06 2024-09-16T05:11:33.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339416+0000", 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.240886668 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339422+0000", 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0680000000000002e-06 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339502+0000", 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9698999999999997e-05 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339527+0000", 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4780999999999999e-05 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340073+0000", 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054627699999999996 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340106+0000", 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3204999999999998e-05 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:403 2.a 2:5f916a06:::smithi20457400-34:head [write 2941380~419987] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.660237+0000", 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.537646234999997, 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098209409999999997, 2024-09-16T05:11:33.299 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 403 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660237+0000", 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660237+0000", 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660238+0000", 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.271e-06 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660383+0000", 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014456899999999999 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660398+0000", 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5065999999999999e-05 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660406+0000", 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7100000000000007e-06 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660468+0000", 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2000000000000003e-05 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660535+0000", 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7453000000000006e-05 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660634+0000", 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9414999999999996e-05 2024-09-16T05:11:33.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.661495+0000", 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00086088899999999999 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.669977+0000", 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084813990000000006 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.670024+0000", 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7191999999999998e-05 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.670058+0000", 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3976999999999997e-05 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:404 2.a 2:5f916a06:::smithi20457400-34:head [setxattr _header (56) in=7b,truncate 3361367] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.660421+0000", 2024-09-16T05:11:33.305 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.537462067999996, 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0097542940000000002, 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 404 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660421+0000", 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660421+0000", 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660422+0000", 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0699999999999997e-07 2024-09-16T05:11:33.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660425+0000", 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5310000000000002e-06 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660434+0000", 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6349999999999995e-06 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660439+0000", 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9350000000000002e-06 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.661193+0000", 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075392500000000002 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.661261+0000", 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8087e-05 2024-09-16T05:11:33.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.661354+0000", 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3018999999999998e-05 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.661700+0000", 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000346017 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.670099+0000", 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0083987690000000004 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.670135+0000", 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6152000000000002e-05 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.670175+0000", 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0617000000000001e-05 2024-09-16T05:11:33.311 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:405 2.a 2:5f916a06:::smithi20457400-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.660460+0000", 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.537422915000001, 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0099105489999999994, 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.312 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 405 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660460+0000", 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660460+0000", 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660461+0000", 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4999999999999998e-07 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660463+0000", 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2699999999999999e-06 2024-09-16T05:11:33.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660470+0000", 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0890000000000002e-06 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.660475+0000", 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0470000000000001e-06 2024-09-16T05:11:33.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:33.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.661527+0000", 2024-09-16T05:11:33.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001051838 2024-09-16T05:11:33.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:11:33.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.661578+0000", 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1517999999999997e-05 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.670211+0000", 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086326979999999994 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.670265+0000", 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3378999999999998e-05 2024-09-16T05:11:33.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.670371+0000", 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010626 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:454 2.c 2:33100188:::smithi20457400-19:head [copy-from ver 33] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:58.688638+0000", 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.509245022999998, 2024-09-16T05:11:33.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017328989999999999, 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 454 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688638+0000", 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688638+0000", 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688640+0000", 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1430000000000001e-06 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688651+0000", 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0501e-05 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688670+0000", 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8974000000000002e-05 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688680+0000", 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9879999999999999e-06 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688761+0000", 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1339000000000002e-05 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688768+0000", 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.221e-06 2024-09-16T05:11:33.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.689852+0000", 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010839249999999999 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.689917+0000", 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4380000000000004e-05 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:11:33.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.699999+0000", 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010082749 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.703541+0000", 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035421210000000001 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.705852+0000", 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023108349999999998 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.705916+0000", 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3625000000000002e-05 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.705967+0000", 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1189000000000001e-05 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:455 2.c 2:33100188:::smithi20457400-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e127)", 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:58.688733+0000", 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.509150040000002, 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014215077, 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 455 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.327 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688733+0000", 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688733+0000", 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688734+0000", 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8899999999999999e-07 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688738+0000", 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7670000000000001e-06 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688747+0000", 2024-09-16T05:11:33.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5149999999999995e-06 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688753+0000", 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3740000000000003e-06 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688828+0000", 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4164000000000003e-05 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688833+0000", 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7599999999999999e-06 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.690047+0000", 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012134730000000001 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.690095+0000", 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7821999999999997e-05 2024-09-16T05:11:33.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.702811+0000", 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012715842999999999 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.702874+0000", 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3428000000000003e-05 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.702948+0000", 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4241999999999998e-05 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:665 2.a 2:5089ebe1:::smithi20457400-36:16 [read 0~3831963,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:21.894361+0000", 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 11.303521757, 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081235430000000004, 2024-09-16T05:11:33.333 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 665 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894361+0000", 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894361+0000", 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894362+0000", 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.06e-07 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894366+0000", 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2089999999999999e-06 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894375+0000", 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0219999999999993e-06 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894382+0000", 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6930000000000001e-06 2024-09-16T05:11:33.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894484+0000", 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102247 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894489+0000", 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0839999999999999e-06 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.900648+0000", 2024-09-16T05:11:33.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061588260000000001 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.900788+0000", 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014025600000000001 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.902485+0000", 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016967 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.338 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:666 2.a 2:5089ebe1:::smithi20457400-36:16 [sparse-read 0~1695869] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:21.894402+0000", 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 11.303480839000001, 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084927790000000006, 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 666 2024-09-16T05:11:33.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894402+0000", 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894402+0000", 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894403+0000", 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7700000000000005e-07 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894417+0000", 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4474000000000001e-05 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894424+0000", 2024-09-16T05:11:33.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6420000000000001e-06 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894428+0000", 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.76e-06 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894516+0000", 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7989000000000001e-05 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:11:33.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.894520+0000", 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9230000000000004e-06 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.902548+0000", 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0080288110000000003 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.902692+0000", 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014370299999999999 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:11:33.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:21.902895+0000", 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000203 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:11:33.344 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:11:33.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:33.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:33.351 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:33.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:33.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:11:33.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:33.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:33.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:33.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:33.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:33.678 INFO:tasks.rados.rados.2.smithi204.stdout:282: done (0 left) 2024-09-16T05:11:33.678 INFO:tasks.rados.rados.2.smithi204.stdout:283: delete oid 43 current snap is 29 2024-09-16T05:11:33.680 INFO:tasks.rados.rados.2.smithi204.stdout:283: done (0 left) 2024-09-16T05:11:33.680 INFO:tasks.rados.rados.2.smithi204.stdout:284: delete oid 40 current snap is 29 2024-09-16T05:11:33.690 INFO:tasks.rados.rados.2.smithi204.stdout:284: done (0 left) 2024-09-16T05:11:33.690 INFO:tasks.rados.rados.2.smithi204.stdout:285: write oid 43 current snap is 29 2024-09-16T05:11:33.690 INFO:tasks.rados.rados.2.smithi204.stdout:285: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-16T05:11:33.696 INFO:tasks.rados.rados.2.smithi204.stdout:285: writing smithi20457400-43 from 652138 to 1113505 tid 1 2024-09-16T05:11:33.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:33.702 INFO:tasks.rados.rados.2.smithi204.stdout:285: writing smithi20457400-43 from 1574940 to 2252138 tid 2 2024-09-16T05:11:33.702 INFO:tasks.rados.rados.2.smithi204.stdout:286: write (excl) oid 40 current snap is 29 2024-09-16T05:11:33.702 INFO:tasks.rados.rados.2.smithi204.stdout:285: finishing write tid 1 to smithi20457400-43 2024-09-16T05:11:33.702 INFO:tasks.rados.rados.2.smithi204.stdout:286: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-16T05:11:33.709 INFO:tasks.rados.rados.2.smithi204.stdout:286: writing smithi20457400-40 from 668945 to 1205561 tid 1 2024-09-16T05:11:33.714 INFO:tasks.rados.rados.2.smithi204.stdout:286: writing smithi20457400-40 from 1633422 to 2124785 tid 2 2024-09-16T05:11:33.714 INFO:tasks.rados.rados.2.smithi204.stdout:287: snap_remove snap 20 2024-09-16T05:11:33.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:33.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:33.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:33.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:33.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:33.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:33 smithi032 ceph-mon[27388]: pgmap v393: 27 pgs: 27 active+clean; 171 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-16T05:11:33.878 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2899144652' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [4, 6, 9, 3]}]': finished 2024-09-16T05:11:33.878 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:33 smithi032 ceph-mon[27388]: osdmap e161: 12 total, 12 up, 12 in 2024-09-16T05:11:33.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:33 smithi204 ceph-mon[30933]: pgmap v393: 27 pgs: 27 active+clean; 171 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-16T05:11:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2899144652' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [4, 6, 9, 3]}]': finished 2024-09-16T05:11:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:33 smithi204 ceph-mon[30933]: osdmap e161: 12 total, 12 up, 12 in 2024-09-16T05:11:33.946 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:33.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:33.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:33 smithi196 ceph-mon[30802]: pgmap v393: 27 pgs: 27 active+clean; 171 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-16T05:11:33.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2899144652' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [4, 6, 9, 3]}]': finished 2024-09-16T05:11:33.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:33 smithi196 ceph-mon[30802]: osdmap e161: 12 total, 12 up, 12 in 2024-09-16T05:11:34.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:34.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:34.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:34.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:11:34.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:34.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:34.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:34.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:34.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:34.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:34.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:34.470 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:34.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:34.489 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:34.489 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:34.497 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:34.511 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:34.511 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:34.536 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:34.537 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:34.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:34.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:34.578 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:34.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:34.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:34.601 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:34.601 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:34.623 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:34.623 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:34.645 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:34.666 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:34.667 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:34.684 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:11:34.691 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:34.691 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:34.693 INFO:tasks.rados.rados.2.smithi204.stdout:287: done (2 left) 2024-09-16T05:11:34.693 INFO:tasks.rados.rados.2.smithi204.stdout:288: snap_remove snap 27 2024-09-16T05:11:34.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:34.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:34.714 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:34.714 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:34 smithi032 ceph-mon[27388]: osdmap e162: 12 total, 12 up, 12 in 2024-09-16T05:11:34.714 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3416572397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:34.714 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2706650089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:34.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:34.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:34.735 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:34.735 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:34.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:34.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:34.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:34.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:34.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:34.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:34.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:34 smithi196 ceph-mon[30802]: osdmap e162: 12 total, 12 up, 12 in 2024-09-16T05:11:34.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3416572397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:34.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2706650089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:35.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:35.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:35.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:35.113 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:35.113 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:34 smithi204 ceph-mon[30933]: osdmap e162: 12 total, 12 up, 12 in 2024-09-16T05:11:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3416572397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2706650089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:35.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:35.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:35.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:35.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:35.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:35.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:35.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:35.478 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:35.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:35.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:35.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:35.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:35.648 INFO:tasks.rados.rados.2.smithi204.stdout:288: done (2 left) 2024-09-16T05:11:35.648 INFO:tasks.rados.rados.2.smithi204.stdout:289: copy_from oid 49 from oid 39 current snap is 29 2024-09-16T05:11:35.648 INFO:tasks.rados.rados.2.smithi204.stdout:285: finishing write tid 2 to smithi20457400-43 2024-09-16T05:11:35.648 INFO:tasks.rados.rados.2.smithi204.stdout:285: finishing write tid 3 to smithi20457400-43 2024-09-16T05:11:35.648 INFO:tasks.rados.rados.2.smithi204.stdout:285: finishing write tid 4 to smithi20457400-43 2024-09-16T05:11:35.648 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 81 (ObjNum 134 snap 29 seq_num 134) dirty exists 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:285: left oid 43 (ObjNum 134 snap 29 seq_num 134) 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:286: finishing write tid 1 to smithi20457400-40 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:286: finishing write tid 2 to smithi20457400-40 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:286: finishing write tid 3 to smithi20457400-40 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:286: finishing write tid 4 to smithi20457400-40 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 72 (ObjNum 135 snap 29 seq_num 135) dirty exists 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:286: left oid 40 (ObjNum 135 snap 29 seq_num 135) 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:285: done (2 left) 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:286: done (1 left) 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:290: rollback oid 25 current snap is 29 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 23 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:291: rollback oid 28 current snap is 29 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 28 to 19 2024-09-16T05:11:35.649 INFO:tasks.rados.rados.2.smithi204.stdout:292: delete oid 32 current snap is 29 2024-09-16T05:11:35.650 INFO:tasks.rados.rados.2.smithi204.stdout:290: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:11:35.650 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:11:35.651 INFO:tasks.rados.rados.2.smithi204.stdout:290: done (3 left) 2024-09-16T05:11:35.651 INFO:tasks.rados.rados.2.smithi204.stdout:292: done (2 left) 2024-09-16T05:11:35.651 INFO:tasks.rados.rados.2.smithi204.stdout:293: rollback oid 2 current snap is 29 2024-09-16T05:11:35.651 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 2 to 16 2024-09-16T05:11:35.651 INFO:tasks.rados.rados.2.smithi204.stdout:294: rollback oid 7 current snap is 29 2024-09-16T05:11:35.651 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 7 to 23 2024-09-16T05:11:35.651 INFO:tasks.rados.rados.2.smithi204.stdout:295: read oid 8 snap 25 2024-09-16T05:11:35.651 INFO:tasks.rados.rados.2.smithi204.stdout:295: expect deleted 2024-09-16T05:11:35.652 INFO:tasks.rados.rados.2.smithi204.stdout:296: read oid 21 snap -1 2024-09-16T05:11:35.652 INFO:tasks.rados.rados.2.smithi204.stdout:296: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T05:11:35.652 INFO:tasks.rados.rados.2.smithi204.stdout:297: snap_remove snap 28 2024-09-16T05:11:35.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:35.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:35.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:35.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:35.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:35.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:35 smithi032 ceph-mon[27388]: pgmap v396: 27 pgs: 27 active+clean; 177 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 15 op/s 2024-09-16T05:11:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:35 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3416572397' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:35 smithi032 ceph-mon[27388]: osdmap e163: 12 total, 12 up, 12 in 2024-09-16T05:11:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:35 smithi032 ceph-mon[27388]: osdmap e164: 12 total, 12 up, 12 in 2024-09-16T05:11:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:35 smithi204 ceph-mon[30933]: pgmap v396: 27 pgs: 27 active+clean; 177 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 15 op/s 2024-09-16T05:11:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:35 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3416572397' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:35 smithi204 ceph-mon[30933]: osdmap e163: 12 total, 12 up, 12 in 2024-09-16T05:11:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:35 smithi204 ceph-mon[30933]: osdmap e164: 12 total, 12 up, 12 in 2024-09-16T05:11:35.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:35.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:35.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:35.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:35.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:35 smithi196 ceph-mon[30802]: pgmap v396: 27 pgs: 27 active+clean; 177 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 15 op/s 2024-09-16T05:11:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:35 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3416572397' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:35 smithi196 ceph-mon[30802]: osdmap e163: 12 total, 12 up, 12 in 2024-09-16T05:11:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:35 smithi196 ceph-mon[30802]: osdmap e164: 12 total, 12 up, 12 in 2024-09-16T05:11:36.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:36.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:36.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:36.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:36.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:36.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:36.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:36.350 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:36.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:36.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:36.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:36.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:36.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:36.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:36.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:36.651 INFO:tasks.rados.rados.2.smithi204.stdout:297: done (6 left) 2024-09-16T05:11:36.651 INFO:tasks.rados.rados.2.smithi204.stdout:298: read oid 44 snap 16 2024-09-16T05:11:36.651 INFO:tasks.rados.rados.2.smithi204.stdout:298: expect (ObjNum 98 snap 15 seq_num 98) 2024-09-16T05:11:36.651 INFO:tasks.rados.rados.2.smithi204.stdout:299: delete oid 50 current snap is 29 2024-09-16T05:11:36.652 INFO:tasks.rados.rados.2.smithi204.stdout:291: finishing rollback tid 0 to smithi20457400-28 2024-09-16T05:11:36.652 INFO:tasks.rados.rados.2.smithi204.stdout:291: finishing rollback tid 1 to smithi20457400-28 2024-09-16T05:11:36.652 INFO:tasks.rados.rados.2.smithi204.stdout:291: finishing rollback tid 2 to smithi20457400-28 2024-09-16T05:11:36.652 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 78 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:11:36.652 INFO:tasks.rados.rados.2.smithi204.stdout:293: finishing rollback tid 0 to smithi20457400-2 2024-09-16T05:11:36.652 INFO:tasks.rados.rados.2.smithi204.stdout:293: finishing rollback tid 1 to smithi20457400-2 2024-09-16T05:11:36.652 INFO:tasks.rados.rados.2.smithi204.stdout:294: finishing rollback tid 1 to smithi20457400-7 2024-09-16T05:11:36.653 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 129 snap 0 seq_num 4282439528) dirty dne 2024-09-16T05:11:36.653 INFO:tasks.rados.rados.2.smithi204.stdout:293: finishing rollback tid 2 to smithi20457400-2 2024-09-16T05:11:36.653 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 66 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T05:11:36.653 INFO:tasks.rados.rados.2.smithi204.stdout:289: finishing copy_from racing read to smithi20457400-49 2024-09-16T05:11:36.653 INFO:tasks.rados.rados.2.smithi204.stdout:289: finishing copy_from to smithi20457400-49 2024-09-16T05:11:36.653 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 77 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:11:36.653 INFO:tasks.rados.rados.2.smithi204.stdout:296: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T05:11:36.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:36.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:36.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:36.732 INFO:tasks.rados.rados.2.smithi204.stdout:298: expect (ObjNum 98 snap 15 seq_num 98) 2024-09-16T05:11:36.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:11:36.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:36.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:36.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:36.852 INFO:tasks.rados.rados.2.smithi204.stdout:289: done (7 left) 2024-09-16T05:11:36.852 INFO:tasks.rados.rados.2.smithi204.stdout:291: done (6 left) 2024-09-16T05:11:36.852 INFO:tasks.rados.rados.2.smithi204.stdout:293: done (5 left) 2024-09-16T05:11:36.852 INFO:tasks.rados.rados.2.smithi204.stdout:294: done (4 left) 2024-09-16T05:11:36.852 INFO:tasks.rados.rados.2.smithi204.stdout:295: done (3 left) 2024-09-16T05:11:36.852 INFO:tasks.rados.rados.2.smithi204.stdout:296: done (2 left) 2024-09-16T05:11:36.853 INFO:tasks.rados.rados.2.smithi204.stdout:298: done (1 left) 2024-09-16T05:11:36.853 INFO:tasks.rados.rados.2.smithi204.stdout:299: done (0 left) 2024-09-16T05:11:36.853 INFO:tasks.rados.rados.2.smithi204.stdout:300: delete oid 17 current snap is 29 2024-09-16T05:11:36.854 INFO:tasks.rados.rados.2.smithi204.stdout:300: done (0 left) 2024-09-16T05:11:36.854 INFO:tasks.rados.rados.2.smithi204.stdout:301: rollback oid 27 current snap is 29 2024-09-16T05:11:36.854 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 27 to 16 2024-09-16T05:11:36.854 INFO:tasks.rados.rados.2.smithi204.stdout:302: read oid 24 snap 19 2024-09-16T05:11:36.854 INFO:tasks.rados.rados.2.smithi204.stdout:302: expect deleted 2024-09-16T05:11:36.854 INFO:tasks.rados.rados.2.smithi204.stdout:303: write oid 32 current snap is 29 2024-09-16T05:11:36.855 INFO:tasks.rados.rados.2.smithi204.stdout:303: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-16T05:11:36.861 INFO:tasks.rados.rados.2.smithi204.stdout:303: writing smithi20457400-32 from 736173 to 1290138 tid 1 2024-09-16T05:11:36.867 INFO:tasks.rados.rados.2.smithi204.stdout:303: writing smithi20457400-32 from 1867350 to 2336173 tid 2 2024-09-16T05:11:36.867 INFO:tasks.rados.rados.2.smithi204.stdout:304: write (excl) oid 39 current snap is 29 2024-09-16T05:11:36.867 INFO:tasks.rados.rados.2.smithi204.stdout:304: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-16T05:11:36.874 INFO:tasks.rados.rados.2.smithi204.stdout:304: writing smithi20457400-39 from 752980 to 1382194 tid 1 2024-09-16T05:11:36.880 INFO:tasks.rados.rados.2.smithi204.stdout:304: writing smithi20457400-39 from 1809479 to 2352980 tid 2 2024-09-16T05:11:36.880 INFO:tasks.rados.rados.2.smithi204.stdout:305: write oid 42 current snap is 29 2024-09-16T05:11:36.880 INFO:tasks.rados.rados.2.smithi204.stdout:305: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-16T05:11:36.887 INFO:tasks.rados.rados.2.smithi204.stdout:305: writing smithi20457400-42 from 769787 to 1474250 tid 1 2024-09-16T05:11:36.893 INFO:tasks.rados.rados.2.smithi204.stdout:305: writing smithi20457400-42 from 2267961 to 2369787 tid 2 2024-09-16T05:11:36.893 INFO:tasks.rados.rados.2.smithi204.stdout:306: rollback oid 19 current snap is 29 2024-09-16T05:11:36.893 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 19 to 13 2024-09-16T05:11:36.893 INFO:tasks.rados.rados.2.smithi204.stdout:301: finishing rollback tid 1 to smithi20457400-27 2024-09-16T05:11:36.893 INFO:tasks.rados.rados.2.smithi204.stdout:301: finishing rollback tid 2 to smithi20457400-27 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 43 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:303: finishing write tid 1 to smithi20457400-32 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:303: finishing write tid 2 to smithi20457400-32 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:301: done (5 left) 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:302: done (4 left) 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:307: copy_from oid 36 from oid 38 current snap is 29 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:303: finishing write tid 3 to smithi20457400-32 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:303: finishing write tid 4 to smithi20457400-32 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 117 (ObjNum 139 snap 29 seq_num 139) dirty exists 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:303: left oid 32 (ObjNum 139 snap 29 seq_num 139) 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:304: finishing write tid 1 to smithi20457400-39 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:303: done (4 left) 2024-09-16T05:11:36.894 INFO:tasks.rados.rados.2.smithi204.stdout:308: write oid 32 current snap is 29 2024-09-16T05:11:36.895 INFO:tasks.rados.rados.2.smithi204.stdout:304: finishing write tid 2 to smithi20457400-39 2024-09-16T05:11:36.895 INFO:tasks.rados.rados.2.smithi204.stdout:304: finishing write tid 3 to smithi20457400-39 2024-09-16T05:11:36.895 INFO:tasks.rados.rados.2.smithi204.stdout:304: finishing write tid 4 to smithi20457400-39 2024-09-16T05:11:36.895 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 181 (ObjNum 140 snap 29 seq_num 140) dirty exists 2024-09-16T05:11:36.895 INFO:tasks.rados.rados.2.smithi204.stdout:304: left oid 39 (ObjNum 140 snap 29 seq_num 140) 2024-09-16T05:11:36.895 INFO:tasks.rados.rados.2.smithi204.stdout:305: finishing write tid 1 to smithi20457400-42 2024-09-16T05:11:36.895 INFO:tasks.rados.rados.2.smithi204.stdout:308: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-16T05:11:36.901 INFO:tasks.rados.rados.2.smithi204.stdout:308: writing smithi20457400-32 from 786594 to 1566306 tid 1 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:308: writing smithi20457400-32 from 2326443 to 2386594 tid 2 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:304: done (4 left) 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:309: read oid 34 snap -1 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:309: expect (ObjNum 106 snap 18 seq_num 106) 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:310: copy_from oid 43 from oid 12 current snap is 29 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:306: finishing rollback tid 1 to smithi20457400-19 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:305: finishing write tid 2 to smithi20457400-42 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:305: finishing write tid 3 to smithi20457400-42 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:305: finishing write tid 4 to smithi20457400-42 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 153 (ObjNum 141 snap 29 seq_num 141) dirty exists 2024-09-16T05:11:36.907 INFO:tasks.rados.rados.2.smithi204.stdout:305: left oid 42 (ObjNum 141 snap 29 seq_num 141) 2024-09-16T05:11:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:305: done (5 left) 2024-09-16T05:11:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:311: write (excl) oid 11 current snap is 29 2024-09-16T05:11:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:307: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:11:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:307: finishing copy_from to smithi20457400-36 2024-09-16T05:11:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 155 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-16T05:11:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:308: finishing write tid 1 to smithi20457400-32 2024-09-16T05:11:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:311: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-16T05:11:36.911 INFO:tasks.rados.rados.2.smithi204.stdout:311: writing smithi20457400-11 from 403401 to 858362 tid 1 2024-09-16T05:11:36.918 INFO:tasks.rados.rados.2.smithi204.stdout:311: writing smithi20457400-11 from 1584925 to 2090964 tid 2 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:307: done (5 left) 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:312: rollback oid 42 current snap is 29 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 16 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:308: finishing write tid 2 to smithi20457400-32 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:313: snap_create 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:311: finishing write tid 1 to smithi20457400-11 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:308: finishing write tid 3 to smithi20457400-32 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:308: finishing write tid 4 to smithi20457400-32 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 158 (ObjNum 142 snap 29 seq_num 142) dirty exists 2024-09-16T05:11:36.919 INFO:tasks.rados.rados.2.smithi204.stdout:308: left oid 32 (ObjNum 142 snap 29 seq_num 142) 2024-09-16T05:11:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:312: finishing rollback tid 0 to smithi20457400-42 2024-09-16T05:11:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:312: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:11:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 155 (ObjNum 273 snap 0 seq_num 4282292192) dirty dne 2024-09-16T05:11:36.921 INFO:tasks.rados.rados.2.smithi204.stdout:311: finishing write tid 2 to smithi20457400-11 2024-09-16T05:11:36.921 INFO:tasks.rados.rados.2.smithi204.stdout:311: finishing write tid 3 to smithi20457400-11 2024-09-16T05:11:36.921 INFO:tasks.rados.rados.2.smithi204.stdout:311: finishing write tid 4 to smithi20457400-11 2024-09-16T05:11:36.921 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 185 (ObjNum 143 snap 29 seq_num 143) dirty exists 2024-09-16T05:11:36.921 INFO:tasks.rados.rados.2.smithi204.stdout:311: left oid 11 (ObjNum 143 snap 29 seq_num 143) 2024-09-16T05:11:36.922 INFO:tasks.rados.rados.2.smithi204.stdout:310: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:11:36.925 INFO:tasks.rados.rados.2.smithi204.stdout:310: finishing copy_from to smithi20457400-43 2024-09-16T05:11:36.925 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 43 (ObjNum 92 snap 9 seq_num 92) dirty exists 2024-09-16T05:11:36.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:36.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:36.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:37.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:37.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:37.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:37.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:37.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:37.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:37.258 INFO:tasks.rados.rados.2.smithi204.stdout:309: expect (ObjNum 106 snap 18 seq_num 106) 2024-09-16T05:11:37.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:37.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:37.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:37.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:37.468 INFO:tasks.rados.rados.2.smithi204.stdout:306: finishing rollback tid 2 to smithi20457400-19 2024-09-16T05:11:37.468 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 81 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:11:37.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:11:37.501 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:37.501 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:37.569 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:37.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:37.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:37.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:37 smithi032 ceph-mon[27388]: pgmap v399: 27 pgs: 27 active+clean; 177 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-16T05:11:37.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:37 smithi032 ceph-mon[27388]: osdmap e165: 12 total, 12 up, 12 in 2024-09-16T05:11:37.691 INFO:tasks.rados.rados.2.smithi204.stdout:306: done (6 left) 2024-09-16T05:11:37.692 INFO:tasks.rados.rados.2.smithi204.stdout:308: done (5 left) 2024-09-16T05:11:37.692 INFO:tasks.rados.rados.2.smithi204.stdout:309: done (4 left) 2024-09-16T05:11:37.692 INFO:tasks.rados.rados.2.smithi204.stdout:310: done (3 left) 2024-09-16T05:11:37.692 INFO:tasks.rados.rados.2.smithi204.stdout:311: done (2 left) 2024-09-16T05:11:37.692 INFO:tasks.rados.rados.2.smithi204.stdout:312: done (1 left) 2024-09-16T05:11:37.692 INFO:tasks.rados.rados.2.smithi204.stdout:313: done (0 left) 2024-09-16T05:11:37.692 INFO:tasks.rados.rados.2.smithi204.stdout:314: snap_create 2024-09-16T05:11:37.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:37.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:37.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:37.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:37.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:37.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:37.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:37 smithi204 ceph-mon[30933]: pgmap v399: 27 pgs: 27 active+clean; 177 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-16T05:11:37.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:37 smithi204 ceph-mon[30933]: osdmap e165: 12 total, 12 up, 12 in 2024-09-16T05:11:37.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:37.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:37.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:37 smithi196 ceph-mon[30802]: pgmap v399: 27 pgs: 27 active+clean; 177 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1.2 MiB/s wr, 6 op/s 2024-09-16T05:11:37.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:37 smithi196 ceph-mon[30802]: osdmap e165: 12 total, 12 up, 12 in 2024-09-16T05:11:38.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:38.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:11:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:38.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:38.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:38.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:38.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:11:38.207 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:38.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:38.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:38.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:38.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:38.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:38.346 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:11:38.346 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:11:38.347 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-09-16T05:11:38.347 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:11:38.347 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:11:38.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-09-16T05:11:38.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:38.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:38.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:38.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:38.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:38.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:38.597 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:11:38.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:38.696 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:38 smithi032 ceph-mon[27388]: osdmap e166: 12 total, 12 up, 12 in 2024-09-16T05:11:38.696 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1827297384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:38.698 INFO:tasks.rados.rados.2.smithi204.stdout:314: done (0 left) 2024-09-16T05:11:38.699 INFO:tasks.rados.rados.2.smithi204.stdout:315: snap_create 2024-09-16T05:11:38.704 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:38.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:38.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:38.727 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:38.727 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:38.728 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:38.730 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:38.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:38.775 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:38.775 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:38.797 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:38.818 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:38.819 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:38.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:38.840 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:38.841 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:38.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:38.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:38.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:38.862 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:38.862 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:38.884 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:38.884 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:38.905 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:38.925 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:38.946 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:38.946 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:38.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:38.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:38.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:38 smithi196 ceph-mon[30802]: osdmap e166: 12 total, 12 up, 12 in 2024-09-16T05:11:38.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1827297384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:38.994 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:38 smithi204 ceph-mon[30933]: osdmap e166: 12 total, 12 up, 12 in 2024-09-16T05:11:38.995 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1827297384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:39.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:39.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:39.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:39.170 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:11:39.170 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T05:11:39.170 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:11:39.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:39.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:39.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:39.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:39.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:39.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:39.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:39.481 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:39.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:39.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:39.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:39.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:39.698 INFO:tasks.rados.rados.2.smithi204.stdout:315: done (0 left) 2024-09-16T05:11:39.698 INFO:tasks.rados.rados.2.smithi204.stdout:316: delete oid 8 current snap is 32 2024-09-16T05:11:39.701 INFO:tasks.rados.rados.2.smithi204.stdout:316: done (0 left) 2024-09-16T05:11:39.702 INFO:tasks.rados.rados.2.smithi204.stdout:317: read oid 6 snap -1 2024-09-16T05:11:39.702 INFO:tasks.rados.rados.2.smithi204.stdout:317: expect (ObjNum 68 snap 4 seq_num 68) 2024-09-16T05:11:39.702 INFO:tasks.rados.rados.2.smithi204.stdout:318: read oid 5 snap -1 2024-09-16T05:11:39.702 INFO:tasks.rados.rados.2.smithi204.stdout:318: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:11:39.702 INFO:tasks.rados.rados.2.smithi204.stdout:319: copy_from oid 49 from oid 43 current snap is 32 2024-09-16T05:11:39.702 INFO:tasks.rados.rados.2.smithi204.stdout:320: write oid 15 current snap is 32 2024-09-16T05:11:39.702 INFO:tasks.rados.rados.2.smithi204.stdout:320: seq_num 145 ranges {437015=721812,1701889=509708} 2024-09-16T05:11:39.708 INFO:tasks.rados.rados.2.smithi204.stdout:320: writing smithi20457400-15 from 437015 to 1158827 tid 1 2024-09-16T05:11:39.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:39.714 INFO:tasks.rados.rados.2.smithi204.stdout:320: writing smithi20457400-15 from 1701889 to 2211597 tid 2 2024-09-16T05:11:39.715 INFO:tasks.rados.rados.2.smithi204.stdout:321: snap_create 2024-09-16T05:11:39.715 INFO:tasks.rados.rados.2.smithi204.stdout:318: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:11:39.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:39.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:39.798 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:11:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:11:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:11:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:11:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:16 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'6, mlcod=16'4)", 2024-09-16T05:11:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:11:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 380.936494235, 2024-09-16T05:11:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29879279399999997, 2024-09-16T05:11:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:11:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:11:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661725+0000", 2024-09-16T05:11:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0800000000000003e-07 2024-09-16T05:11:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661731+0000", 2024-09-16T05:11:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4880000000000004e-06 2024-09-16T05:11:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960116+0000", 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29838472799999999 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960124+0000", 2024-09-16T05:11:39.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960207+0000", 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3122999999999997e-05 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960225+0000", 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8099000000000001e-05 2024-09-16T05:11:39.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960487+0000", 2024-09-16T05:11:39.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026233000000000002 2024-09-16T05:11:39.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960517+0000", 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9149000000000002e-05 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.806 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:9 2.5 e112/107 2:a1d8c6d9:::smithi20457400-3:head v 112'1, mlcod=0'0)", 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.018230+0000", 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.579988063999998, 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021039539999999999, 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.018230+0000", 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.018230+0000", 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.018231+0000", 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9599999999999999e-07 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.019073+0000", 2024-09-16T05:11:39.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00084265700000000002 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.019256+0000", 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00018243799999999999 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.019267+0000", 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1739999999999999e-05 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.019313+0000", 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6068e-05 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.019325+0000", 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1864e-05 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.020304+0000", 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097850299999999992 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.020334+0000", 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0188e-05 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:73 2.c e112/107 2:33100188:::smithi20457400-19:head v 112'1, mlcod=0'0)", 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.203026+0000", 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.395192430999998, 2024-09-16T05:11:39.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018772789999999999, 2024-09-16T05:11:39.812 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.812 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.812 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.203026+0000", 2024-09-16T05:11:39.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.203026+0000", 2024-09-16T05:11:39.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.203026+0000", 2024-09-16T05:11:39.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9000000000000001e-07 2024-09-16T05:11:39.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.203819+0000", 2024-09-16T05:11:39.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079346099999999997 2024-09-16T05:11:39.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.203842+0000", 2024-09-16T05:11:39.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2665999999999999e-05 2024-09-16T05:11:39.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.203851+0000", 2024-09-16T05:11:39.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2790000000000004e-06 2024-09-16T05:11:39.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.203920+0000", 2024-09-16T05:11:39.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8696999999999996e-05 2024-09-16T05:11:39.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.203940+0000", 2024-09-16T05:11:39.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0064e-05 2024-09-16T05:11:39.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.820 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.204852+0000", 2024-09-16T05:11:39.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091163099999999999 2024-09-16T05:11:39.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.820 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.204903+0000", 2024-09-16T05:11:39.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1291000000000001e-05 2024-09-16T05:11:39.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.821 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.822 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:190 2.c e112/107 2:3a3554ba:::smithi20457400-48:head v 112'11, mlcod=112'9)", 2024-09-16T05:11:39.822 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.593699+0000", 2024-09-16T05:11:39.822 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.004518818000001, 2024-09-16T05:11:39.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001916769, 2024-09-16T05:11:39.822 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.823 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.823 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.823 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.593699+0000", 2024-09-16T05:11:39.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.593699+0000", 2024-09-16T05:11:39.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.593700+0000", 2024-09-16T05:11:39.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3200000000000003e-07 2024-09-16T05:11:39.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.594658+0000", 2024-09-16T05:11:39.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095824099999999998 2024-09-16T05:11:39.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.594695+0000", 2024-09-16T05:11:39.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6158999999999997e-05 2024-09-16T05:11:39.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.594702+0000", 2024-09-16T05:11:39.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1579999999999999e-06 2024-09-16T05:11:39.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.594759+0000", 2024-09-16T05:11:39.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7491999999999998e-05 2024-09-16T05:11:39.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.594777+0000", 2024-09-16T05:11:39.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7404000000000001e-05 2024-09-16T05:11:39.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.595566+0000", 2024-09-16T05:11:39.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078945100000000004 2024-09-16T05:11:39.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.832 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.832 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.595616+0000", 2024-09-16T05:11:39.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9932000000000002e-05 2024-09-16T05:11:39.832 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.833 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.833 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.833 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:210 2.5 e113/107 2:a1d8c6d9:::smithi20457400-3:head v 113'23, mlcod=112'20)", 2024-09-16T05:11:39.834 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:45.007743+0000", 2024-09-16T05:11:39.834 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 54.590474913000001, 2024-09-16T05:11:39.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001757717, 2024-09-16T05:11:39.834 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.834 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.834 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.835 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.007743+0000", 2024-09-16T05:11:39.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.007743+0000", 2024-09-16T05:11:39.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.007743+0000", 2024-09-16T05:11:39.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:11:39.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.008400+0000", 2024-09-16T05:11:39.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00065702800000000004 2024-09-16T05:11:39.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.839 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.008425+0000", 2024-09-16T05:11:39.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4249000000000001e-05 2024-09-16T05:11:39.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.839 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.008432+0000", 2024-09-16T05:11:39.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9929999999999998e-06 2024-09-16T05:11:39.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.008496+0000", 2024-09-16T05:11:39.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4610999999999995e-05 2024-09-16T05:11:39.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.008519+0000", 2024-09-16T05:11:39.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2966e-05 2024-09-16T05:11:39.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.009453+0000", 2024-09-16T05:11:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00093346399999999995 2024-09-16T05:11:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.009501+0000", 2024-09-16T05:11:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8229999999999997e-05 2024-09-16T05:11:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:237 2.5 e115/107 2:a49174eb:::smithi20457400-39:head v 115'30, mlcod=113'26)", 2024-09-16T05:11:39.844 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:47.033389+0000", 2024-09-16T05:11:39.844 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 52.564829424999999, 2024-09-16T05:11:39.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018957539999999999, 2024-09-16T05:11:39.844 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.844 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.844 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.033389+0000", 2024-09-16T05:11:39.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.033389+0000", 2024-09-16T05:11:39.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.033390+0000", 2024-09-16T05:11:39.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1790000000000001e-06 2024-09-16T05:11:39.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.034190+0000", 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079982899999999995 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.034216+0000", 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6787999999999998e-05 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.034223+0000", 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1630000000000001e-06 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.034266+0000", 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3161999999999998e-05 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.034281+0000", 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5162e-05 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.035252+0000", 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097077600000000002 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.035284+0000", 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2694999999999998e-05 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.850 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:238 2.5 e115/107 2:a49174eb:::smithi20457400-39:head v 115'31, mlcod=115'28)", 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:47.037228+0000", 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 52.560990132999997, 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033756519999999998, 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.037228+0000", 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.037228+0000", 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.037229+0000", 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2500000000000001e-06 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039475+0000", 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022458930000000001 2024-09-16T05:11:39.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039524+0000", 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8851999999999999e-05 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039537+0000", 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2761e-05 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039629+0000", 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2516000000000007e-05 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039658+0000", 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9122e-05 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.040548+0000", 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088925199999999999 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.040604+0000", 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6005999999999999e-05 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:292 2.5 e117/107 2:a1d8c6d9:::smithi20457400-3:head v 117'37, mlcod=115'33)", 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:49.353041+0000", 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.245177247999997, 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047142110000000003, 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.353041+0000", 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.353041+0000", 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.353046+0000", 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6350000000000003e-06 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356065+0000", 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030184589999999998 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356114+0000", 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9397999999999997e-05 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356125+0000", 2024-09-16T05:11:39.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0574000000000001e-05 2024-09-16T05:11:39.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356200+0000", 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5099999999999996e-05 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356229+0000", 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8750000000000001e-05 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.357714+0000", 2024-09-16T05:11:39.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014852979999999999 2024-09-16T05:11:39.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.357755+0000", 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0997000000000001e-05 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:425 2.4 e126/107 2:2cffa4d6:::smithi20457400-50:head v 126'22, mlcod=119'19)", 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.724633+0000", 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 41.873585313, 2024-09-16T05:11:39.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001742375, 2024-09-16T05:11:39.862 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.862 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.862 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.724633+0000", 2024-09-16T05:11:39.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.724633+0000", 2024-09-16T05:11:39.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.864 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.864 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.724633+0000", 2024-09-16T05:11:39.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.39e-07 2024-09-16T05:11:39.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.724773+0000", 2024-09-16T05:11:39.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000140047 2024-09-16T05:11:39.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.724873+0000", 2024-09-16T05:11:39.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100009 2024-09-16T05:11:39.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.724882+0000", 2024-09-16T05:11:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4980000000000006e-06 2024-09-16T05:11:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.724965+0000", 2024-09-16T05:11:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2943999999999997e-05 2024-09-16T05:11:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.724991+0000", 2024-09-16T05:11:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6312000000000002e-05 2024-09-16T05:11:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.726321+0000", 2024-09-16T05:11:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013300199999999999 2024-09-16T05:11:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.726375+0000", 2024-09-16T05:11:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4206000000000002e-05 2024-09-16T05:11:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.871 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:427 2.4 e126/107 2:2cffa4d6:::smithi20457400-50:head v 126'24, mlcod=126'22)", 2024-09-16T05:11:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.736734+0000", 2024-09-16T05:11:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 41.861484003999998, 2024-09-16T05:11:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024208680000000001, 2024-09-16T05:11:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736734+0000", 2024-09-16T05:11:39.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736734+0000", 2024-09-16T05:11:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736735+0000", 2024-09-16T05:11:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2920000000000001e-06 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.737927+0000", 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001191356 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.737991+0000", 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4702000000000005e-05 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738002+0000", 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0693000000000001e-05 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738097+0000", 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5340000000000005e-05 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738122+0000", 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4709999999999999e-05 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.739110+0000", 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00098828200000000005 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.739155+0000", 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4493000000000002e-05 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:433 2.c e126/107 2:33100188:::smithi20457400-19:head v 126'14, mlcod=112'12)", 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.743576+0000", 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 41.854642310000003, 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017242889999999999, 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.743576+0000", 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.743576+0000", 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.743577+0000", 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1510000000000001e-06 2024-09-16T05:11:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.744329+0000", 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00075190200000000004 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.744356+0000", 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7586999999999999e-05 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.880 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.744365+0000", 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9139999999999997e-06 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.744428+0000", 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2818999999999996e-05 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.744454+0000", 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5885999999999999e-05 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.745270+0000", 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00081608600000000005 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.745300+0000", 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9944e-05 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.882 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:445 2.4 e127/107 2:2d3f13f8:::smithi20457400-7:head v 127'27, mlcod=126'24)", 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:58.678392+0000", 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.919825824999997, 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018960540000000001, 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.678392+0000", 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.678392+0000", 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.678392+0000", 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6e-07 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.678761+0000", 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036828300000000001 2024-09-16T05:11:39.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.678802+0000", 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1208999999999998e-05 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.678812+0000", 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.026e-05 2024-09-16T05:11:39.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.678877+0000", 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4930000000000003e-05 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.678906+0000", 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9278999999999999e-05 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.680241+0000", 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001334773 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.680288+0000", 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7160000000000002e-05 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.887 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:454 2.c e127/107 2:33100188:::smithi20457400-19:head v 127'19, mlcod=126'16)", 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.898071496, 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054909060000000003, 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:11:39.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:11:39.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7800000000000001e-07 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703134+0000", 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002986954 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703165+0000", 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1201000000000002e-05 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703173+0000", 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.32e-06 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703255+0000", 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1730000000000005e-05 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703282+0000", 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7389000000000002e-05 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.705600+0000", 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023176360000000001 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.705637+0000", 2024-09-16T05:11:39.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7497999999999999e-05 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:523 2.5 e137/107 2:a5bc1608:::smithi20457400-11:head v 137'59, mlcod=137'57)", 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:08.687551+0000", 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.910667081, 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017842520000000001, 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.893 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.687551+0000", 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.687551+0000", 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.687553+0000", 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7099999999999999e-06 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.688372+0000", 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00081925599999999998 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.688399+0000", 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6795999999999999e-05 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.688404+0000", 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2680000000000002e-06 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.688448+0000", 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4186e-05 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.688463+0000", 2024-09-16T05:11:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5098e-05 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.689302+0000", 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000838414 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:08.689335+0000", 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3524e-05 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:116 2.5 e141/107 2:ad4a88eb:::smithi20457400-16:a v 141'87, mlcod=141'79)", 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.903209966999999, 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006093261, 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695009+0000", 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6499999999999999e-07 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695014+0000", 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8350000000000003e-06 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700654+0000", 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056407499999999999 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700660+0000", 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4879999999999998e-06 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700700+0000", 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0583000000000001e-05 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700714+0000", 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.322e-05 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701062+0000", 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00034833999999999999 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701101+0000", 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9480000000000001e-05 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.902 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:563 2.5 e141/107 2:ad7afe2b:::smithi20457400-38:head v 141'88, mlcod=141'83)", 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.700032+0000", 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.898186106000001, 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024023, 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700032+0000", 2024-09-16T05:11:39.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700032+0000", 2024-09-16T05:11:39.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700033+0000", 2024-09-16T05:11:39.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8899999999999998e-07 2024-09-16T05:11:39.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700638+0000", 2024-09-16T05:11:39.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060523200000000004 2024-09-16T05:11:39.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701617+0000", 2024-09-16T05:11:39.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097897799999999992 2024-09-16T05:11:39.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701621+0000", 2024-09-16T05:11:39.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2400000000000001e-06 2024-09-16T05:11:39.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701655+0000", 2024-09-16T05:11:39.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3940999999999999e-05 2024-09-16T05:11:39.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701669+0000", 2024-09-16T05:11:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.375e-05 2024-09-16T05:11:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.702407+0000", 2024-09-16T05:11:39.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000737921 2024-09-16T05:11:39.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.702434+0000", 2024-09-16T05:11:39.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7449000000000001e-05 2024-09-16T05:11:39.912 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.912 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.912 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.913 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:118 2.5 e141/107 2:ad7afe2b:::smithi20457400-38:d v 141'90, mlcod=141'85)", 2024-09-16T05:11:39.913 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:11:39.913 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.896615643000001, 2024-09-16T05:11:39.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066456359999999999, 2024-09-16T05:11:39.913 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.914 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.914 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.914 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:11:39.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:11:39.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701603+0000", 2024-09-16T05:11:39.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:11:39.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701606+0000", 2024-09-16T05:11:39.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3890000000000001e-06 2024-09-16T05:11:39.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707421+0000", 2024-09-16T05:11:39.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005814601 2024-09-16T05:11:39.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707428+0000", 2024-09-16T05:11:39.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4839999999999998e-06 2024-09-16T05:11:39.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707497+0000", 2024-09-16T05:11:39.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8441000000000004e-05 2024-09-16T05:11:39.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707520+0000", 2024-09-16T05:11:39.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3513999999999999e-05 2024-09-16T05:11:39.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708185+0000", 2024-09-16T05:11:39.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066482599999999998 2024-09-16T05:11:39.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708248+0000", 2024-09-16T05:11:39.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2959000000000005e-05 2024-09-16T05:11:39.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.923 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.924 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:564 2.5 e141/107 2:ad7afe2b:::smithi20457400-38:head v 141'91, mlcod=141'87)", 2024-09-16T05:11:39.924 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.706595+0000", 2024-09-16T05:11:39.924 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.891622791, 2024-09-16T05:11:39.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002056142, 2024-09-16T05:11:39.924 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.706595+0000", 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.706595+0000", 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.706596+0000", 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.16e-07 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707399+0000", 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080258 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707501+0000", 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102323 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707507+0000", 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2099999999999998e-06 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707750+0000", 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024271300000000001 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707768+0000", 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7706e-05 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708607+0000", 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083984999999999995 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708651+0000", 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3943999999999998e-05 2024-09-16T05:11:39.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:565 2.5 e141/107 2:ad7afe2b:::smithi20457400-38:head v 141'92, mlcod=141'87)", 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.707480+0000", 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.890738083999999, 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001798579, 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.929 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707480+0000", 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707480+0000", 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707481+0000", 2024-09-16T05:11:39.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1699999999999998e-07 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707487+0000", 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5740000000000001e-06 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708866+0000", 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013783249999999999 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708874+0000", 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8079999999999994e-06 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708946+0000", 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2181000000000006e-05 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708973+0000", 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6109e-05 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.709234+0000", 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026189300000000002 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.709279+0000", 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4072000000000001e-05 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:614 2.5 e144/107 2:a1d8c6d9:::smithi20457400-3:head v 144'110, mlcod=144'106)", 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 23.947649878, 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054521470000000001, 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650569+0000", 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7299999999999995e-07 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653581+0000", 2024-09-16T05:11:39.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003012617 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653627+0000", 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5649999999999998e-05 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653639+0000", 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2126e-05 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:11:39.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653715+0000", 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5485e-05 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653748+0000", 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3373e-05 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.655964+0000", 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002215875 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:11:39.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.656020+0000", 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6348000000000003e-05 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:11:39.938 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:11:39.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:39.940 INFO:tasks.rados.rados.2.smithi204.stdout:317: expect (ObjNum 68 snap 4 seq_num 68) 2024-09-16T05:11:39.940 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:39 smithi032 ceph-mon[27388]: pgmap v402: 27 pgs: 27 active+clean; 187 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-16T05:11:39.940 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:39 smithi032 ceph-mon[27388]: osdmap e167: 12 total, 12 up, 12 in 2024-09-16T05:11:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:39.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:39.985 INFO:tasks.rados.rados.2.smithi204.stdout:320: finishing write tid 1 to smithi20457400-15 2024-09-16T05:11:39.985 INFO:tasks.rados.rados.2.smithi204.stdout:319: finishing copy_from racing read to smithi20457400-49 2024-09-16T05:11:39.985 INFO:tasks.rados.rados.2.smithi204.stdout:319: finishing copy_from to smithi20457400-49 2024-09-16T05:11:39.985 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 43 (ObjNum 92 snap 9 seq_num 92) dirty exists 2024-09-16T05:11:39.985 INFO:tasks.rados.rados.2.smithi204.stdout:320: finishing write tid 2 to smithi20457400-15 2024-09-16T05:11:39.986 INFO:tasks.rados.rados.2.smithi204.stdout:320: finishing write tid 3 to smithi20457400-15 2024-09-16T05:11:39.986 INFO:tasks.rados.rados.2.smithi204.stdout:320: finishing write tid 4 to smithi20457400-15 2024-09-16T05:11:39.986 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 7 (ObjNum 145 snap 32 seq_num 145) dirty exists 2024-09-16T05:11:39.986 INFO:tasks.rados.rados.2.smithi204.stdout:320: left oid 15 (ObjNum 145 snap 32 seq_num 145) 2024-09-16T05:11:39.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:39 smithi196 ceph-mon[30802]: pgmap v402: 27 pgs: 27 active+clean; 187 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-16T05:11:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:39 smithi196 ceph-mon[30802]: osdmap e167: 12 total, 12 up, 12 in 2024-09-16T05:11:40.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:40.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:40.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:39 smithi204 ceph-mon[30933]: pgmap v402: 27 pgs: 27 active+clean; 187 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2024-09-16T05:11:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:39 smithi204 ceph-mon[30933]: osdmap e167: 12 total, 12 up, 12 in 2024-09-16T05:11:40.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:40.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:40.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:40.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:40.342 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:40.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:40.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:40.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:40.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:40.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:40.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:40.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:40.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:11:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:11:40.648 INFO:tasks.rados.rados.2.smithi204.stdout:317: done (4 left) 2024-09-16T05:11:40.648 INFO:tasks.rados.rados.2.smithi204.stdout:318: done (3 left) 2024-09-16T05:11:40.648 INFO:tasks.rados.rados.2.smithi204.stdout:319: done (2 left) 2024-09-16T05:11:40.649 INFO:tasks.rados.rados.2.smithi204.stdout:320: done (1 left) 2024-09-16T05:11:40.649 INFO:tasks.rados.rados.2.smithi204.stdout:321: done (0 left) 2024-09-16T05:11:40.649 INFO:tasks.rados.rados.2.smithi204.stdout:322: read oid 45 snap -1 2024-09-16T05:11:40.649 INFO:tasks.rados.rados.2.smithi204.stdout:322: expect (ObjNum 110 snap 18 seq_num 110) 2024-09-16T05:11:40.649 INFO:tasks.rados.rados.2.smithi204.stdout:323: write (excl) oid 47 current snap is 33 2024-09-16T05:11:40.649 INFO:tasks.rados.rados.2.smithi204.stdout:323: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-16T05:11:40.655 INFO:tasks.rados.rados.2.smithi204.stdout:323: writing smithi20457400-47 from 453822 to 1250883 tid 1 2024-09-16T05:11:40.662 INFO:tasks.rados.rados.2.smithi204.stdout:323: writing smithi20457400-47 from 1760371 to 2413737 tid 2 2024-09-16T05:11:40.662 INFO:tasks.rados.rados.2.smithi204.stdout:324: snap_create 2024-09-16T05:11:40.662 INFO:tasks.rados.rados.2.smithi204.stdout:322: expect (ObjNum 110 snap 18 seq_num 110) 2024-09-16T05:11:40.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:40.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:40.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:40.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:40.842 INFO:tasks.rados.rados.2.smithi204.stdout:323: finishing write tid 1 to smithi20457400-47 2024-09-16T05:11:40.843 INFO:tasks.rados.rados.2.smithi204.stdout:323: finishing write tid 2 to smithi20457400-47 2024-09-16T05:11:40.843 INFO:tasks.rados.rados.2.smithi204.stdout:323: finishing write tid 3 to smithi20457400-47 2024-09-16T05:11:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:40.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:40.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:40 smithi032 ceph-mon[27388]: osdmap e168: 12 total, 12 up, 12 in 2024-09-16T05:11:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1004451275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:40 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1004451275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:40 smithi032 ceph-mon[27388]: osdmap e169: 12 total, 12 up, 12 in 2024-09-16T05:11:40.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:40.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:40.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:40.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:40 smithi196 ceph-mon[30802]: osdmap e168: 12 total, 12 up, 12 in 2024-09-16T05:11:40.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1004451275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:40 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1004451275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:40 smithi196 ceph-mon[30802]: osdmap e169: 12 total, 12 up, 12 in 2024-09-16T05:11:41.002 INFO:tasks.rados.rados.2.smithi204.stdout:323: finishing write tid 4 to smithi20457400-47 2024-09-16T05:11:41.002 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 66 (ObjNum 146 snap 33 seq_num 146) dirty exists 2024-09-16T05:11:41.002 INFO:tasks.rados.rados.2.smithi204.stdout:323: left oid 47 (ObjNum 146 snap 33 seq_num 146) 2024-09-16T05:11:41.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:41.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:41.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:41.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:41.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:40 smithi204 ceph-mon[30933]: osdmap e168: 12 total, 12 up, 12 in 2024-09-16T05:11:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1004451275' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:40 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1004451275' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:40 smithi204 ceph-mon[30933]: osdmap e169: 12 total, 12 up, 12 in 2024-09-16T05:11:41.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:41.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:41.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:41.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:41.356 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:41.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:41.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:41.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:41.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:41.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:41.604 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:41.605 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:41.650 INFO:tasks.rados.rados.2.smithi204.stdout:322: done (2 left) 2024-09-16T05:11:41.650 INFO:tasks.rados.rados.2.smithi204.stdout:323: done (1 left) 2024-09-16T05:11:41.650 INFO:tasks.rados.rados.2.smithi204.stdout:324: done (0 left) 2024-09-16T05:11:41.650 INFO:tasks.rados.rados.2.smithi204.stdout:325: write (excl) oid 43 current snap is 34 2024-09-16T05:11:41.650 INFO:tasks.rados.rados.2.smithi204.stdout:325: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-16T05:11:41.654 INFO:tasks.rados.rados.2.smithi204.stdout:325: writing smithi20457400-43 from 470629 to 942939 tid 1 2024-09-16T05:11:41.660 INFO:tasks.rados.rados.2.smithi204.stdout:325: writing smithi20457400-43 from 1418853 to 1932230 tid 2 2024-09-16T05:11:41.660 INFO:tasks.rados.rados.2.smithi204.stdout:326: read oid 25 snap -1 2024-09-16T05:11:41.660 INFO:tasks.rados.rados.2.smithi204.stdout:326: expect deleted 2024-09-16T05:11:41.660 INFO:tasks.rados.rados.2.smithi204.stdout:327: delete oid 1 current snap is 34 2024-09-16T05:11:41.662 INFO:tasks.rados.rados.2.smithi204.stdout:325: finishing write tid 1 to smithi20457400-43 2024-09-16T05:11:41.663 INFO:tasks.rados.rados.2.smithi204.stdout:325: finishing write tid 2 to smithi20457400-43 2024-09-16T05:11:41.663 INFO:tasks.rados.rados.2.smithi204.stdout:326: done (2 left) 2024-09-16T05:11:41.663 INFO:tasks.rados.rados.2.smithi204.stdout:327: done (1 left) 2024-09-16T05:11:41.663 INFO:tasks.rados.rados.2.smithi204.stdout:328: snap_remove snap 25 2024-09-16T05:11:41.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:41.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:41.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:41.742 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:41.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:41.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:41.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:41 smithi032 ceph-mon[27388]: pgmap v405: 27 pgs: 27 active+clean; 193 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2024-09-16T05:11:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:41 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:41 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:41 smithi032 ceph-mon[27388]: osdmap e170: 12 total, 12 up, 12 in 2024-09-16T05:11:41.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:41.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:41.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:41.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:41 smithi196 ceph-mon[30802]: pgmap v405: 27 pgs: 27 active+clean; 193 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2024-09-16T05:11:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:41 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:41 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:41.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:41 smithi196 ceph-mon[30802]: osdmap e170: 12 total, 12 up, 12 in 2024-09-16T05:11:42.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:42.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:42.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:41 smithi204 ceph-mon[30933]: pgmap v405: 27 pgs: 27 active+clean; 193 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2024-09-16T05:11:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:41 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:41 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:41 smithi204 ceph-mon[30933]: osdmap e170: 12 total, 12 up, 12 in 2024-09-16T05:11:42.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:42.208 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:11:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:11:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:11:42.208 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:11:42.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:42.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:42.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:42.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:42.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:42.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:42.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:42.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:42.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:42.606 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:42.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:42.653 INFO:tasks.rados.rados.2.smithi204.stdout:328: done (1 left) 2024-09-16T05:11:42.653 INFO:tasks.rados.rados.2.smithi204.stdout:329: copy_from oid 36 from oid 37 current snap is 34 2024-09-16T05:11:42.701 INFO:tasks.rados.rados.2.smithi204.stdout:325: finishing write tid 3 to smithi20457400-43 2024-09-16T05:11:42.701 INFO:tasks.rados.rados.2.smithi204.stdout:325: finishing write tid 4 to smithi20457400-43 2024-09-16T05:11:42.701 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 88 (ObjNum 147 snap 34 seq_num 147) dirty exists 2024-09-16T05:11:42.701 INFO:tasks.rados.rados.2.smithi204.stdout:325: left oid 43 (ObjNum 147 snap 34 seq_num 147) 2024-09-16T05:11:42.701 INFO:tasks.rados.rados.2.smithi204.stdout:325: done (1 left) 2024-09-16T05:11:42.701 INFO:tasks.rados.rados.2.smithi204.stdout:330: snap_create 2024-09-16T05:11:42.702 INFO:tasks.rados.rados.2.smithi204.stdout:329: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:11:42.702 INFO:tasks.rados.rados.2.smithi204.stdout:329: finishing copy_from to smithi20457400-36 2024-09-16T05:11:42.702 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 59 (ObjNum 129 snap 27 seq_num 129) dirty exists 2024-09-16T05:11:42.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:42.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:42.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:11:42.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:42.893 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:42.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:42.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:43.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:43.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:43.081 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:43.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:43.143 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:43.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:43.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:11:43.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:43.272 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:43.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:43.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:43.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:43.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:43.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:43.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:43.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:43.542 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:11:43.542 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:11:43.542 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:11:43.542 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:11:43.542 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:11:43.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:43.656 INFO:tasks.rados.rados.2.smithi204.stdout:329: done (1 left) 2024-09-16T05:11:43.657 INFO:tasks.rados.rados.2.smithi204.stdout:330: done (0 left) 2024-09-16T05:11:43.657 INFO:tasks.rados.rados.2.smithi204.stdout:331: rollback oid 10 current snap is 35 2024-09-16T05:11:43.657 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 13 2024-09-16T05:11:43.657 INFO:tasks.rados.rados.2.smithi204.stdout:332: snap_remove snap 16 2024-09-16T05:11:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:43.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:43.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:43.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:11:43.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:43.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:43.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:43.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:43 smithi032 ceph-mon[27388]: pgmap v408: 27 pgs: 27 active+clean; 197 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-16T05:11:43.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:43 smithi032 ceph-mon[27388]: osdmap e171: 12 total, 12 up, 12 in 2024-09-16T05:11:43.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:43.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:43.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:43.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:43 smithi204 ceph-mon[30933]: pgmap v408: 27 pgs: 27 active+clean; 197 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-16T05:11:43.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:43 smithi204 ceph-mon[30933]: osdmap e171: 12 total, 12 up, 12 in 2024-09-16T05:11:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:43 smithi196 ceph-mon[30802]: pgmap v408: 27 pgs: 27 active+clean; 197 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-16T05:11:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:43 smithi196 ceph-mon[30802]: osdmap e171: 12 total, 12 up, 12 in 2024-09-16T05:11:44.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:44.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:44.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:44.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:44.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:44.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:44.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:11:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:44.233 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:44.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:44.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:44.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:44.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:44.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:44.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:44.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:44.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:44.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:44.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:44.664 INFO:tasks.rados.rados.2.smithi204.stdout:332: done (1 left) 2024-09-16T05:11:44.665 INFO:tasks.rados.rados.2.smithi204.stdout:333: write oid 12 current snap is 35 2024-09-16T05:11:44.665 INFO:tasks.rados.rados.2.smithi204.stdout:331: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:11:44.665 INFO:tasks.rados.rados.2.smithi204.stdout:331: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:11:44.665 INFO:tasks.rados.rados.2.smithi204.stdout:331: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:11:44.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 47 (ObjNum 82 snap 7 seq_num 82) dirty exists 2024-09-16T05:11:44.665 INFO:tasks.rados.rados.2.smithi204.stdout:333: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-16T05:11:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:44.672 INFO:tasks.rados.rados.2.smithi204.stdout:333: writing smithi20457400-12 from 504243 to 1127051 tid 1 2024-09-16T05:11:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:44.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:44.678 INFO:tasks.rados.rados.2.smithi204.stdout:333: writing smithi20457400-12 from 1819464 to 2336510 tid 2 2024-09-16T05:11:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:331: done (1 left) 2024-09-16T05:11:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:334: rollback oid 7 current snap is 35 2024-09-16T05:11:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 7 to 30 2024-09-16T05:11:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:333: finishing write tid 1 to smithi20457400-12 2024-09-16T05:11:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:335: read oid 25 snap -1 2024-09-16T05:11:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:335: expect deleted 2024-09-16T05:11:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:336: snap_create 2024-09-16T05:11:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:334: finishing rollback tid 1 to smithi20457400-7 2024-09-16T05:11:44.680 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 193 snap 0 seq_num 4282056880) dirty dne 2024-09-16T05:11:44.681 INFO:tasks.rados.rados.2.smithi204.stdout:333: finishing write tid 2 to smithi20457400-12 2024-09-16T05:11:44.681 INFO:tasks.rados.rados.2.smithi204.stdout:333: finishing write tid 3 to smithi20457400-12 2024-09-16T05:11:44.681 INFO:tasks.rados.rados.2.smithi204.stdout:333: finishing write tid 4 to smithi20457400-12 2024-09-16T05:11:44.681 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 161 (ObjNum 149 snap 35 seq_num 149) dirty exists 2024-09-16T05:11:44.681 INFO:tasks.rados.rados.2.smithi204.stdout:333: left oid 12 (ObjNum 149 snap 35 seq_num 149) 2024-09-16T05:11:44.771 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:44 smithi032 ceph-mon[27388]: osdmap e172: 12 total, 12 up, 12 in 2024-09-16T05:11:44.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:44.790 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:11:44.791 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:11:44.791 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T05:11:44.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:11:44.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:44.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:44.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:44.883 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:44.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:44.905 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:44.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:44 smithi204 ceph-mon[30933]: osdmap e172: 12 total, 12 up, 12 in 2024-09-16T05:11:44.925 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:44.925 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:44.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:44.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:44.946 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:44.946 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:44.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:44.968 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:44.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:44 smithi196 ceph-mon[30802]: osdmap e172: 12 total, 12 up, 12 in 2024-09-16T05:11:44.989 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:44.990 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:45.011 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:45.012 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:45.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:45.035 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:45.035 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:45.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:45.057 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:45.079 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:45.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:45.102 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:45.124 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:45.124 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:45.154 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:45.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:45.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:45.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:45.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:45.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:45.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:45.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:45.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:45.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:45.561 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:45.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:45.647 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:11:45.651 INFO:tasks.rados.rados.2.smithi204.stdout:333: done (3 left) 2024-09-16T05:11:45.651 INFO:tasks.rados.rados.2.smithi204.stdout:334: done (2 left) 2024-09-16T05:11:45.651 INFO:tasks.rados.rados.2.smithi204.stdout:335: done (1 left) 2024-09-16T05:11:45.651 INFO:tasks.rados.rados.2.smithi204.stdout:336: done (0 left) 2024-09-16T05:11:45.651 INFO:tasks.rados.rados.2.smithi204.stdout:337: rollback oid 50 current snap is 36 2024-09-16T05:11:45.651 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 29 2024-09-16T05:11:45.651 INFO:tasks.rados.rados.2.smithi204.stdout:338: read oid 39 snap 35 2024-09-16T05:11:45.651 INFO:tasks.rados.rados.2.smithi204.stdout:338: expect (ObjNum 140 snap 29 seq_num 140) 2024-09-16T05:11:45.652 INFO:tasks.rados.rados.2.smithi204.stdout:339: copy_from oid 8 from oid 37 current snap is 36 2024-09-16T05:11:45.652 INFO:tasks.rados.rados.2.smithi204.stdout:340: write (excl) oid 40 current snap is 36 2024-09-16T05:11:45.652 INFO:tasks.rados.rados.2.smithi204.stdout:340: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-16T05:11:45.657 INFO:tasks.rados.rados.2.smithi204.stdout:340: writing smithi20457400-40 from 521050 to 1219107 tid 1 2024-09-16T05:11:45.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:45.665 INFO:tasks.rados.rados.2.smithi204.stdout:340: writing smithi20457400-40 from 1877946 to 2521050 tid 2 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:341: read oid 26 snap -1 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:341: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:342: rollback oid 36 current snap is 36 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 36 to 33 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:337: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 70 (ObjNum 129 snap 0 seq_num 4282009224) dirty dne 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:337: done (5 left) 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:343: snap_create 2024-09-16T05:11:45.666 INFO:tasks.rados.rados.2.smithi204.stdout:338: expect (ObjNum 140 snap 29 seq_num 140) 2024-09-16T05:11:45.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:45.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:339: finishing copy_from racing read to smithi20457400-8 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:339: finishing copy_from to smithi20457400-8 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 59 (ObjNum 129 snap 27 seq_num 129) dirty exists 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:342: finishing rollback tid 0 to smithi20457400-36 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:342: finishing rollback tid 1 to smithi20457400-36 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:342: finishing rollback tid 2 to smithi20457400-36 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 164 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:340: finishing write tid 1 to smithi20457400-40 2024-09-16T05:11:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:340: finishing write tid 2 to smithi20457400-40 2024-09-16T05:11:45.768 INFO:tasks.rados.rados.2.smithi204.stdout:340: finishing write tid 3 to smithi20457400-40 2024-09-16T05:11:45.768 INFO:tasks.rados.rados.2.smithi204.stdout:340: finishing write tid 4 to smithi20457400-40 2024-09-16T05:11:45.768 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 100 (ObjNum 150 snap 36 seq_num 150) dirty exists 2024-09-16T05:11:45.768 INFO:tasks.rados.rados.2.smithi204.stdout:340: left oid 40 (ObjNum 150 snap 36 seq_num 150) 2024-09-16T05:11:45.768 INFO:tasks.rados.rados.2.smithi204.stdout:341: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:11:45.790 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:45.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:45.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:45.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:45 smithi032 ceph-mon[27388]: pgmap v411: 27 pgs: 27 active+clean; 203 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2024-09-16T05:11:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:45 smithi032 ceph-mon[27388]: osdmap e173: 12 total, 12 up, 12 in 2024-09-16T05:11:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1550110470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1284904389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:45 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1284904389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:45 smithi032 ceph-mon[27388]: osdmap e174: 12 total, 12 up, 12 in 2024-09-16T05:11:45.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:45 smithi204 ceph-mon[30933]: pgmap v411: 27 pgs: 27 active+clean; 203 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2024-09-16T05:11:45.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:45 smithi204 ceph-mon[30933]: osdmap e173: 12 total, 12 up, 12 in 2024-09-16T05:11:45.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1550110470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:45.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1284904389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:45.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:45 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:45.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1284904389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:45.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:45 smithi204 ceph-mon[30933]: osdmap e174: 12 total, 12 up, 12 in 2024-09-16T05:11:45.939 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:45.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:45 smithi196 ceph-mon[30802]: pgmap v411: 27 pgs: 27 active+clean; 203 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2024-09-16T05:11:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:45 smithi196 ceph-mon[30802]: osdmap e173: 12 total, 12 up, 12 in 2024-09-16T05:11:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1550110470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1284904389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:11:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:45 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1284904389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:11:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:45 smithi196 ceph-mon[30802]: osdmap e174: 12 total, 12 up, 12 in 2024-09-16T05:11:46.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:46.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:46.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:46.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:46.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:46.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:46.267 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:46.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:46.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:46.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:46.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:46.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:46.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:46.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:46.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:46.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:46.651 INFO:tasks.rados.rados.2.smithi204.stdout:338: done (5 left) 2024-09-16T05:11:46.652 INFO:tasks.rados.rados.2.smithi204.stdout:339: done (4 left) 2024-09-16T05:11:46.652 INFO:tasks.rados.rados.2.smithi204.stdout:340: done (3 left) 2024-09-16T05:11:46.652 INFO:tasks.rados.rados.2.smithi204.stdout:341: done (2 left) 2024-09-16T05:11:46.652 INFO:tasks.rados.rados.2.smithi204.stdout:342: done (1 left) 2024-09-16T05:11:46.652 INFO:tasks.rados.rados.2.smithi204.stdout:343: done (0 left) 2024-09-16T05:11:46.652 INFO:tasks.rados.rados.2.smithi204.stdout:344: write (excl) oid 38 current snap is 37 2024-09-16T05:11:46.652 INFO:tasks.rados.rados.2.smithi204.stdout:344: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-16T05:11:46.658 INFO:tasks.rados.rados.2.smithi204.stdout:344: writing smithi20457400-38 from 537857 to 1311163 tid 1 2024-09-16T05:11:46.665 INFO:tasks.rados.rados.2.smithi204.stdout:344: writing smithi20457400-38 from 1936428 to 2457143 tid 2 2024-09-16T05:11:46.665 INFO:tasks.rados.rados.2.smithi204.stdout:345: rollback oid 22 current snap is 37 2024-09-16T05:11:46.665 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 30 2024-09-16T05:11:46.665 INFO:tasks.rados.rados.2.smithi204.stdout:344: finishing write tid 1 to smithi20457400-38 2024-09-16T05:11:46.665 INFO:tasks.rados.rados.2.smithi204.stdout:346: snap_create 2024-09-16T05:11:46.667 INFO:tasks.rados.rados.2.smithi204.stdout:344: finishing write tid 2 to smithi20457400-38 2024-09-16T05:11:46.667 INFO:tasks.rados.rados.2.smithi204.stdout:344: finishing write tid 3 to smithi20457400-38 2024-09-16T05:11:46.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:344: finishing write tid 4 to smithi20457400-38 2024-09-16T05:11:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 207 (ObjNum 151 snap 37 seq_num 151) dirty exists 2024-09-16T05:11:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:344: left oid 38 (ObjNum 151 snap 37 seq_num 151) 2024-09-16T05:11:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:345: finishing rollback tid 0 to smithi20457400-22 2024-09-16T05:11:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:345: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:11:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:345: finishing rollback tid 2 to smithi20457400-22 2024-09-16T05:11:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 93 (ObjNum 96 snap 12 seq_num 96) dirty exists 2024-09-16T05:11:46.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:46.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:46.729 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:46.729 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:46.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:46.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:46.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:46.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:46 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:11:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:46 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:11:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:46 smithi032 ceph-mon[27388]: osdmap e175: 12 total, 12 up, 12 in 2024-09-16T05:11:46.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:46.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:46.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:46.946 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:46 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:11:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:46 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:11:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:46 smithi196 ceph-mon[30802]: osdmap e175: 12 total, 12 up, 12 in 2024-09-16T05:11:47.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:47.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:47.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:47.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:46 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:11:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:46 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:11:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:46 smithi204 ceph-mon[30933]: osdmap e175: 12 total, 12 up, 12 in 2024-09-16T05:11:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:47.189 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:47.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:47.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:47.313 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:47.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:47.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:11:47.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:47.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:47.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:47.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:47.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:47.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:47.656 INFO:tasks.rados.rados.2.smithi204.stdout:344: done (2 left) 2024-09-16T05:11:47.656 INFO:tasks.rados.rados.2.smithi204.stdout:345: done (1 left) 2024-09-16T05:11:47.656 INFO:tasks.rados.rados.2.smithi204.stdout:346: done (0 left) 2024-09-16T05:11:47.656 INFO:tasks.rados.rados.2.smithi204.stdout:347: delete oid 33 current snap is 38 2024-09-16T05:11:47.659 INFO:tasks.rados.rados.2.smithi204.stdout:347: done (0 left) 2024-09-16T05:11:47.659 INFO:tasks.rados.rados.2.smithi204.stdout:348: rollback oid 5 current snap is 38 2024-09-16T05:11:47.659 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 5 to 31 2024-09-16T05:11:47.659 INFO:tasks.rados.rados.2.smithi204.stdout:349: copy_from oid 32 from oid 39 current snap is 38 2024-09-16T05:11:47.659 INFO:tasks.rados.rados.2.smithi204.stdout:350: snap_create 2024-09-16T05:11:47.661 INFO:tasks.rados.rados.2.smithi204.stdout:348: finishing rollback tid 0 to smithi20457400-5 2024-09-16T05:11:47.661 INFO:tasks.rados.rados.2.smithi204.stdout:348: finishing rollback tid 1 to smithi20457400-5 2024-09-16T05:11:47.661 INFO:tasks.rados.rados.2.smithi204.stdout:348: finishing rollback tid 2 to smithi20457400-5 2024-09-16T05:11:47.661 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 100 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:11:47.666 INFO:tasks.rados.rados.2.smithi204.stdout:349: finishing copy_from racing read to smithi20457400-32 2024-09-16T05:11:47.668 INFO:tasks.rados.rados.2.smithi204.stdout:349: finishing copy_from to smithi20457400-32 2024-09-16T05:11:47.668 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 181 (ObjNum 140 snap 29 seq_num 140) dirty exists 2024-09-16T05:11:47.670 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:47.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:47.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:47.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:11:47.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:47.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:47.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: pgmap v414: 27 pgs: 27 active+clean; 203 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:11:47.828 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:47 smithi032 ceph-mon[27388]: osdmap e176: 12 total, 12 up, 12 in 2024-09-16T05:11:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:47.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:47.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:47.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: pgmap v414: 27 pgs: 27 active+clean; 203 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-16T05:11:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:11:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:11:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:11:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:11:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:11:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:11:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:11:47.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:47 smithi196 ceph-mon[30802]: osdmap e176: 12 total, 12 up, 12 in 2024-09-16T05:11:48.016 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:11:48.017 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:11:48.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:48.072 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:48.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: pgmap v414: 27 pgs: 27 active+clean; 203 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-16T05:11:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:11:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:11:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:11:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:11:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:11:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:11:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:11:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:47 smithi204 ceph-mon[30933]: osdmap e176: 12 total, 12 up, 12 in 2024-09-16T05:11:48.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:48.194 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:48.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:48.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:11:48.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:48.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:48.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:48.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:48.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:48.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:48.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:48.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:48.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:48.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:48.657 INFO:tasks.rados.rados.2.smithi204.stdout:348: done (2 left) 2024-09-16T05:11:48.657 INFO:tasks.rados.rados.2.smithi204.stdout:349: done (1 left) 2024-09-16T05:11:48.657 INFO:tasks.rados.rados.2.smithi204.stdout:350: done (0 left) 2024-09-16T05:11:48.657 INFO:tasks.rados.rados.2.smithi204.stdout:351: write (excl) oid 40 current snap is 39 2024-09-16T05:11:48.657 INFO:tasks.rados.rados.2.smithi204.stdout:351: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-16T05:11:48.663 INFO:tasks.rados.rados.2.smithi204.stdout:351: writing smithi20457400-40 from 571471 to 1211628 tid 1 2024-09-16T05:11:48.668 INFO:tasks.rados.rados.2.smithi204.stdout:351: writing smithi20457400-40 from 1653392 to 2177776 tid 2 2024-09-16T05:11:48.669 INFO:tasks.rados.rados.2.smithi204.stdout:352: snap_remove snap 23 2024-09-16T05:11:48.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:48.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:48.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:48.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:48.816 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-16T05:11:47.650063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218280512,"num_objects":99,"num_object_clones":53,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":6,"num_read":659,"num_read_kb":505600,"num_write":597,"num_write_kb":207514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2650,"ondisk_log_size":2650,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":608908,"kb_used_data":259560,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124332916,"statfs":{"total":1151940427776,"available":1151316905984,"internally_reserved":0,"allocated":265789440,"data_stored":307815929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11831180,"num_objects":7,"num_object_clones":6,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":27,"num_read_kb":28540,"num_write":24,"num_write_kb":10389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000093"},"pg_stats":[{"pgid":"2.f","version":"171'68","reported_seq":392,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T05:11:43.651444+0000","last_change":"2024-09-16T05:11:42.657870+0000","last_active":"2024-09-16T05:11:43.651444+0000","last_peered":"2024-09-16T05:11:43.651444+0000","last_clean":"2024-09-16T05:11:43.651444+0000","last_became_active":"2024-09-16T05:10:36.956705+0000","last_became_peered":"2024-09-16T05:10:36.956705+0000","last_unstale":"2024-09-16T05:11:43.651444+0000","last_undegraded":"2024-09-16T05:11:43.651444+0000","last_fullsized":"2024-09-16T05:11:43.651444+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:45.709752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074290270000000004,"stat_sum":{"num_bytes":17440754,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":65,"num_read_kb":58742,"num_write":43,"num_write_kb":14800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"2.e","version":"174'100","reported_seq":426,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.671413+0000","last_change":"2024-09-16T05:11:45.029868+0000","last_active":"2024-09-16T05:11:45.671413+0000","last_peered":"2024-09-16T05:11:45.671413+0000","last_clean":"2024-09-16T05:11:45.671413+0000","last_became_active":"2024-09-16T05:10:37.001925+0000","last_became_peered":"2024-09-16T05:10:37.001925+0000","last_unstale":"2024-09-16T05:11:45.671413+0000","last_undegraded":"2024-09-16T05:11:45.671413+0000","last_fullsized":"2024-09-16T05:11:45.671413+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:51.193766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002391958,"stat_sum":{"num_bytes":23120023,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":77,"num_read_kb":73321,"num_write":59,"num_write_kb":17531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"2.d","version":"173'165","reported_seq":483,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-16T05:11:44.670392+0000","last_change":"2024-09-16T05:11:44.670650+0000","last_active":"2024-09-16T05:11:44.670392+0000","last_peered":"2024-09-16T05:11:44.670392+0000","last_clean":"2024-09-16T05:11:44.670392+0000","last_became_active":"2024-09-16T05:10:36.957439+0000","last_became_peered":"2024-09-16T05:10:36.957439+0000","last_unstale":"2024-09-16T05:11:44.670392+0000","last_undegraded":"2024-09-16T05:11:44.670392+0000","last_fullsized":"2024-09-16T05:11:44.670392+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:56.364997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006480609,"stat_sum":{"num_bytes":24671287,"num_objects":11,"num_object_clones":6,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":95,"num_read_kb":63165,"num_write":87,"num_write_kb":29845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"2.c","version":"175'93","reported_seq":402,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T05:11:46.668299+0000","last_change":"2024-09-16T05:11:44.667440+0000","last_active":"2024-09-16T05:11:46.668299+0000","last_peered":"2024-09-16T05:11:46.668299+0000","last_clean":"2024-09-16T05:11:46.668299+0000","last_became_active":"2024-09-16T05:10:36.957767+0000","last_became_peered":"2024-09-16T05:10:36.957767+0000","last_unstale":"2024-09-16T05:11:46.668299+0000","last_undegraded":"2024-09-16T05:11:46.668299+0000","last_fullsized":"2024-09-16T05:11:46.668299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036194069999999998,"stat_sum":{"num_bytes":13415142,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":13506,"num_write":44,"num_write_kb":12512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"2.8","version":"173'51","reported_seq":362,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T05:11:46.754032+0000","last_change":"2024-09-16T05:11:44.663917+0000","last_active":"2024-09-16T05:11:46.754032+0000","last_peered":"2024-09-16T05:11:46.754032+0000","last_clean":"2024-09-16T05:11:46.754032+0000","last_became_active":"2024-09-16T05:10:36.959606+0000","last_became_peered":"2024-09-16T05:10:36.959606+0000","last_unstale":"2024-09-16T05:11:46.754032+0000","last_undegraded":"2024-09-16T05:11:46.754032+0000","last_fullsized":"2024-09-16T05:11:46.754032+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023773610000000001,"stat_sum":{"num_bytes":9892665,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":17,"num_read_kb":7284,"num_write":21,"num_write_kb":7365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"2.3","version":"173'13","reported_seq":315,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.028873+0000","last_change":"2024-09-16T05:11:45.029036+0000","last_active":"2024-09-16T05:11:45.028873+0000","last_peered":"2024-09-16T05:11:45.028873+0000","last_clean":"2024-09-16T05:11:45.028873+0000","last_became_active":"2024-09-16T05:10:36.956582+0000","last_became_peered":"2024-09-16T05:10:36.956582+0000","last_unstale":"2024-09-16T05:11:45.028873+0000","last_undegraded":"2024-09-16T05:11:45.028873+0000","last_fullsized":"2024-09-16T05:11:45.028873+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:47.838999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001497373,"stat_sum":{"num_bytes":2285719,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":543,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T05:11:46.754114+0000","last_change":"2024-09-16T05:10:41.789086+0000","last_active":"2024-09-16T05:11:46.754114+0000","last_peered":"2024-09-16T05:11:46.754114+0000","last_clean":"2024-09-16T05:11:46.754114+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:11:46.754114+0000","last_undegraded":"2024-09-16T05:11:46.754114+0000","last_fullsized":"2024-09-16T05:11:46.754114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:41.788991+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:41.788991+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:49.806632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"173'11","reported_seq":319,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.028857+0000","last_change":"2024-09-16T05:11:45.029044+0000","last_active":"2024-09-16T05:11:45.028857+0000","last_peered":"2024-09-16T05:11:45.028857+0000","last_clean":"2024-09-16T05:11:45.028857+0000","last_became_active":"2024-09-16T05:10:36.956451+0000","last_became_peered":"2024-09-16T05:10:36.956451+0000","last_unstale":"2024-09-16T05:11:45.028857+0000","last_undegraded":"2024-09-16T05:11:45.028857+0000","last_fullsized":"2024-09-16T05:11:45.028857+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:59.771036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015319330000000001,"stat_sum":{"num_bytes":3668535,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":12398,"num_write":8,"num_write_kb":2117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":543,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.716322+0000","last_change":"2024-09-16T05:10:46.732985+0000","last_active":"2024-09-16T05:11:45.716322+0000","last_peered":"2024-09-16T05:11:45.716322+0000","last_clean":"2024-09-16T05:11:45.716322+0000","last_became_active":"2024-09-16T05:10:38.972381+0000","last_became_peered":"2024-09-16T05:10:38.972381+0000","last_unstale":"2024-09-16T05:11:45.716322+0000","last_undegraded":"2024-09-16T05:11:45.716322+0000","last_fullsized":"2024-09-16T05:11:45.716322+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:46.732896+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:46.732896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:47.973041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":547,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T05:11:46.754042+0000","last_change":"2024-09-16T05:10:42.783065+0000","last_active":"2024-09-16T05:11:46.754042+0000","last_peered":"2024-09-16T05:11:46.754042+0000","last_clean":"2024-09-16T05:11:46.754042+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:11:46.754042+0000","last_undegraded":"2024-09-16T05:11:46.754042+0000","last_fullsized":"2024-09-16T05:11:46.754042+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:42.782976+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:42.782976+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:49.806652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"173'48","reported_seq":360,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.666565+0000","last_change":"2024-09-16T05:11:45.029856+0000","last_active":"2024-09-16T05:11:45.666565+0000","last_peered":"2024-09-16T05:11:45.666565+0000","last_clean":"2024-09-16T05:11:45.666565+0000","last_became_active":"2024-09-16T05:10:37.001937+0000","last_became_peered":"2024-09-16T05:10:37.001937+0000","last_unstale":"2024-09-16T05:11:45.666565+0000","last_undegraded":"2024-09-16T05:11:45.666565+0000","last_fullsized":"2024-09-16T05:11:45.666565+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:51.276055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024364730000000002,"stat_sum":{"num_bytes":4571438,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":11502,"num_write":16,"num_write_kb":14162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":540,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-16T05:11:42.654621+0000","last_change":"2024-09-16T05:11:19.419444+0000","last_active":"2024-09-16T05:11:42.654621+0000","last_peered":"2024-09-16T05:11:42.654621+0000","last_clean":"2024-09-16T05:11:42.654621+0000","last_became_active":"2024-09-16T05:11:19.419149+0000","last_became_peered":"2024-09-16T05:11:19.419149+0000","last_unstale":"2024-09-16T05:11:42.654621+0000","last_undegraded":"2024-09-16T05:11:42.654621+0000","last_fullsized":"2024-09-16T05:11:42.654621+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.849175+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.849175+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:52.137143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"121'10","reported_seq":331,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T05:11:46.786813+0000","last_change":"2024-09-16T05:11:46.786839+0000","last_active":"2024-09-16T05:11:46.786813+0000","last_peered":"2024-09-16T05:11:46.786813+0000","last_clean":"2024-09-16T05:11:46.786813+0000","last_became_active":"2024-09-16T05:10:36.983391+0000","last_became_peered":"2024-09-16T05:10:36.983391+0000","last_unstale":"2024-09-16T05:11:46.786813+0000","last_undegraded":"2024-09-16T05:11:46.786813+0000","last_fullsized":"2024-09-16T05:11:46.786813+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'10","last_scrub_stamp":"2024-09-16T05:11:46.786746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.786746+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:07.705012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043346200000000002,"stat_sum":{"num_bytes":3142876,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":4089,"num_write":9,"num_write_kb":1825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":127,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.716468+0000","last_change":"2024-09-16T05:11:44.703409+0000","last_active":"2024-09-16T05:11:45.716468+0000","last_peered":"2024-09-16T05:11:45.716468+0000","last_clean":"2024-09-16T05:11:45.716468+0000","last_became_active":"2024-09-16T05:11:21.676078+0000","last_became_peered":"2024-09-16T05:11:21.676078+0000","last_unstale":"2024-09-16T05:11:45.716468+0000","last_undegraded":"2024-09-16T05:11:45.716468+0000","last_fullsized":"2024-09-16T05:11:45.716468+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028234000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":505,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T05:11:46.666289+0000","last_change":"2024-09-16T05:10:41.421609+0000","last_active":"2024-09-16T05:11:46.666289+0000","last_peered":"2024-09-16T05:11:46.666289+0000","last_clean":"2024-09-16T05:11:46.666289+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:11:46.666289+0000","last_undegraded":"2024-09-16T05:11:46.666289+0000","last_fullsized":"2024-09-16T05:11:46.666289+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:41.421501+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:41.421501+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:51.416537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":319,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-16T05:11:44.660538+0000","last_change":"2024-09-16T05:11:44.660882+0000","last_active":"2024-09-16T05:11:44.660538+0000","last_peered":"2024-09-16T05:11:44.660538+0000","last_clean":"2024-09-16T05:11:44.660538+0000","last_became_active":"2024-09-16T05:10:36.957473+0000","last_became_peered":"2024-09-16T05:10:36.957473+0000","last_unstale":"2024-09-16T05:11:44.660538+0000","last_undegraded":"2024-09-16T05:11:44.660538+0000","last_fullsized":"2024-09-16T05:11:44.660538+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:02.461110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020250960000000001,"stat_sum":{"num_bytes":2857157,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13147,"num_write":8,"num_write_kb":2442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":544,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.716362+0000","last_change":"2024-09-16T05:10:44.788217+0000","last_active":"2024-09-16T05:11:45.716362+0000","last_peered":"2024-09-16T05:11:45.716362+0000","last_clean":"2024-09-16T05:11:45.716362+0000","last_became_active":"2024-09-16T05:10:38.972150+0000","last_became_peered":"2024-09-16T05:10:38.972150+0000","last_unstale":"2024-09-16T05:11:45.716362+0000","last_undegraded":"2024-09-16T05:11:45.716362+0000","last_fullsized":"2024-09-16T05:11:45.716362+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.788114+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.788114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:57.373836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"171'193","reported_seq":506,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T05:11:43.651429+0000","last_change":"2024-09-16T05:11:42.655304+0000","last_active":"2024-09-16T05:11:43.651429+0000","last_peered":"2024-09-16T05:11:43.651429+0000","last_clean":"2024-09-16T05:11:43.651429+0000","last_became_active":"2024-09-16T05:10:36.953485+0000","last_became_peered":"2024-09-16T05:10:36.953485+0000","last_unstale":"2024-09-16T05:11:43.651429+0000","last_undegraded":"2024-09-16T05:11:43.651429+0000","last_fullsized":"2024-09-16T05:11:43.651429+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:55.702577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048865940000000002,"stat_sum":{"num_bytes":32937966,"num_objects":16,"num_object_clones":10,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":96,"num_read_kb":78566,"num_write":91,"num_write_kb":30296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":544,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.716322+0000","last_change":"2024-09-16T05:10:45.767820+0000","last_active":"2024-09-16T05:11:45.716322+0000","last_peered":"2024-09-16T05:11:45.716322+0000","last_clean":"2024-09-16T05:11:45.716322+0000","last_became_active":"2024-09-16T05:10:38.972286+0000","last_became_peered":"2024-09-16T05:10:38.972286+0000","last_unstale":"2024-09-16T05:11:45.716322+0000","last_undegraded":"2024-09-16T05:11:45.716322+0000","last_fullsized":"2024-09-16T05:11:45.716322+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.767717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.767717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:08.779630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"173'75","reported_seq":387,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-16T05:11:44.660584+0000","last_change":"2024-09-16T05:11:44.660584+0000","last_active":"2024-09-16T05:11:44.660584+0000","last_peered":"2024-09-16T05:11:44.660584+0000","last_clean":"2024-09-16T05:11:44.660584+0000","last_became_active":"2024-09-16T05:10:36.956394+0000","last_became_peered":"2024-09-16T05:10:36.956394+0000","last_unstale":"2024-09-16T05:11:44.660584+0000","last_undegraded":"2024-09-16T05:11:44.660584+0000","last_fullsized":"2024-09-16T05:11:44.660584+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:46.958753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001640772,"stat_sum":{"num_bytes":7553989,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":40,"num_read_kb":18715,"num_write":38,"num_write_kb":9260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":464,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.716351+0000","last_change":"2024-09-16T05:11:27.759078+0000","last_active":"2024-09-16T05:11:45.716351+0000","last_peered":"2024-09-16T05:11:45.716351+0000","last_clean":"2024-09-16T05:11:45.716351+0000","last_became_active":"2024-09-16T05:11:21.676107+0000","last_became_peered":"2024-09-16T05:11:21.676107+0000","last_unstale":"2024-09-16T05:11:45.716351+0000","last_undegraded":"2024-09-16T05:11:45.716351+0000","last_fullsized":"2024-09-16T05:11:45.716351+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:27.759006+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:27.759006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:36.703164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"171'92","reported_seq":405,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T05:11:43.651443+0000","last_change":"2024-09-16T05:11:42.653531+0000","last_active":"2024-09-16T05:11:43.651443+0000","last_peered":"2024-09-16T05:11:43.651443+0000","last_clean":"2024-09-16T05:11:43.651443+0000","last_became_active":"2024-09-16T05:10:36.956652+0000","last_became_peered":"2024-09-16T05:10:36.956652+0000","last_unstale":"2024-09-16T05:11:43.651443+0000","last_undegraded":"2024-09-16T05:11:43.651443+0000","last_fullsized":"2024-09-16T05:11:43.651443+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0030219130000000002,"stat_sum":{"num_bytes":25086661,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":51,"num_read_kb":48705,"num_write":49,"num_write_kb":26467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":539,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-16T05:11:45.027436+0000","last_change":"2024-09-16T05:10:45.437354+0000","last_active":"2024-09-16T05:11:45.027436+0000","last_peered":"2024-09-16T05:11:45.027436+0000","last_clean":"2024-09-16T05:11:45.027436+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:11:45.027436+0000","last_undegraded":"2024-09-16T05:11:45.027436+0000","last_fullsized":"2024-09-16T05:11:45.027436+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.437261+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.437261+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:05.011089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"174'144","reported_seq":482,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T05:11:46.666331+0000","last_change":"2024-09-16T05:11:46.247494+0000","last_active":"2024-09-16T05:11:46.666331+0000","last_peered":"2024-09-16T05:11:46.666331+0000","last_clean":"2024-09-16T05:11:46.666331+0000","last_became_active":"2024-09-16T05:10:37.002200+0000","last_became_peered":"2024-09-16T05:10:37.002200+0000","last_unstale":"2024-09-16T05:11:46.666331+0000","last_undegraded":"2024-09-16T05:11:46.666331+0000","last_fullsized":"2024-09-16T05:11:46.666331+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'144","last_scrub_stamp":"2024-09-16T05:11:46.247408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.247408+0000","objects_scrubbed":15,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:54.954900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0050220050000000004,"stat_sum":{"num_bytes":38861022,"num_objects":15,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":78,"num_read_kb":63852,"num_write":87,"num_write_kb":30586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"12","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":547,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T05:11:46.754030+0000","last_change":"2024-09-16T05:10:43.759254+0000","last_active":"2024-09-16T05:11:46.754030+0000","last_peered":"2024-09-16T05:11:46.754030+0000","last_clean":"2024-09-16T05:11:46.754030+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:11:46.754030+0000","last_undegraded":"2024-09-16T05:11:46.754030+0000","last_fullsized":"2024-09-16T05:11:46.754030+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:43.759130+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:43.759130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:55.539739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"173'49","reported_seq":372,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-16T05:11:44.660454+0000","last_change":"2024-09-16T05:11:44.660771+0000","last_active":"2024-09-16T05:11:44.660454+0000","last_peered":"2024-09-16T05:11:44.660454+0000","last_clean":"2024-09-16T05:11:44.660454+0000","last_became_active":"2024-09-16T05:10:36.958530+0000","last_became_peered":"2024-09-16T05:10:36.958530+0000","last_unstale":"2024-09-16T05:11:44.660454+0000","last_undegraded":"2024-09-16T05:11:44.660454+0000","last_fullsized":"2024-09-16T05:11:44.660454+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:47.700338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019361910000000001,"stat_sum":{"num_bytes":8184910,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":49,"num_read_kb":38607,"num_write":32,"num_write_kb":7179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12","length":"1"},{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":541,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-09-16T05:11:44.663909+0000","last_change":"2024-09-16T05:10:44.010384+0000","last_active":"2024-09-16T05:11:44.663909+0000","last_peered":"2024-09-16T05:11:44.663909+0000","last_clean":"2024-09-16T05:11:44.663909+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:11:44.663909+0000","last_undegraded":"2024-09-16T05:11:44.663909+0000","last_fullsized":"2024-09-16T05:11:44.663909+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.010311+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.010311+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:11:55.319454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":217690144,"num_objects":97,"num_object_clones":53,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":6,"num_read":659,"num_read_kb":505600,"num_write":597,"num_write_kb":207514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":240181248,"data_stored":292110035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1121,"ondisk_log_size":1121,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089376,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70388,"kb_used_data":43356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93674764,"statfs":{"total":95995035648,"available":95922958336,"internally_reserved":0,"allocated":44396544,"data_stored":55401773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285604,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34516,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93710636,"statfs":{"total":95995035648,"available":95959691264,"internally_reserved":0,"allocated":3395584,"data_stored":2522575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514539,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62048,"kb_used_data":30856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683104,"statfs":{"total":95995035648,"available":95931498496,"internally_reserved":0,"allocated":31596544,"data_stored":33759428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841581,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74184,"kb_used_data":47148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93670968,"statfs":{"total":95995035648,"available":95919071232,"internally_reserved":0,"allocated":48279552,"data_stored":56469033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103219,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44296,"kb_used_data":13108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93700856,"statfs":{"total":95995035648,"available":95949676544,"internally_reserved":0,"allocated":13422592,"data_stored":16422228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397562,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33152,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712000,"statfs":{"total":95995035648,"available":95961088000,"internally_reserved":0,"allocated":2011136,"data_stored":1145857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757309,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32432,"kb_used_data":5404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712720,"statfs":{"total":95995035648,"available":95961825280,"internally_reserved":0,"allocated":5533696,"data_stored":4919751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44328,"kb_used_data":17296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93700824,"statfs":{"total":95995035648,"available":95949643776,"internally_reserved":0,"allocated":17711104,"data_stored":17515156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117063,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61264,"kb_used_data":34236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93683888,"statfs":{"total":95995035648,"available":95932301312,"internally_reserved":0,"allocated":35057664,"data_stored":40032676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313294,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47992,"kb_used_data":16800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93697160,"statfs":{"total":95995035648,"available":95945891840,"internally_reserved":0,"allocated":17203200,"data_stored":19259372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41508,"kb_used_data":14476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93703644,"statfs":{"total":95995035648,"available":95952531456,"internally_reserved":0,"allocated":14823424,"data_stored":16857945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62800,"kb_used_data":31600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682352,"statfs":{"total":95995035648,"available":95930728448,"internally_reserved":0,"allocated":32358400,"data_stored":43510135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30396416,"data_stored":42398006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12849152,"data_stored":15737416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14999552,"data_stored":17920651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32391168,"data_stored":37820473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15724544,"data_stored":16386155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3342336,"data_stored":3589502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11411456,"data_stored":15276371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45989888,"data_stored":55029224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":29609984,"data_stored":32630427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1409024,"data_stored":1393574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":42057728,"data_stored":53928236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:11:48.818 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:11:48.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-16T05:11:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:48.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:48.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:48.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:48.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:48.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:49.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:49.081 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:49.081 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:49.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:49.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:49.189 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:49.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:49.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:49.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:49.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:49.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:49.350 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:49.351 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:49.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:49.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:49.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:49.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:49.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:49.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:352: done (1 left) 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:353: copy_from oid 7 from oid 41 current snap is 39 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:351: finishing write tid 1 to smithi20457400-40 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:351: finishing write tid 2 to smithi20457400-40 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:351: finishing write tid 3 to smithi20457400-40 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:351: finishing write tid 4 to smithi20457400-40 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 104 (ObjNum 153 snap 39 seq_num 153) dirty exists 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:351: left oid 40 (ObjNum 153 snap 39 seq_num 153) 2024-09-16T05:11:49.660 INFO:tasks.rados.rados.2.smithi204.stdout:351: done (1 left) 2024-09-16T05:11:49.661 INFO:tasks.rados.rados.2.smithi204.stdout:354: write (excl) oid 39 current snap is 39 2024-09-16T05:11:49.661 INFO:tasks.rados.rados.2.smithi204.stdout:354: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-16T05:11:49.667 INFO:tasks.rados.rados.2.smithi204.stdout:354: writing smithi20457400-39 from 588278 to 1303684 tid 1 2024-09-16T05:11:49.674 INFO:tasks.rados.rados.2.smithi204.stdout:354: writing smithi20457400-39 from 1711874 to 2379916 tid 2 2024-09-16T05:11:49.674 INFO:tasks.rados.rados.2.smithi204.stdout:355: read oid 10 snap -1 2024-09-16T05:11:49.674 INFO:tasks.rados.rados.2.smithi204.stdout:355: expect (ObjNum 82 snap 7 seq_num 82) 2024-09-16T05:11:49.674 INFO:tasks.rados.rados.2.smithi204.stdout:356: read oid 36 snap -1 2024-09-16T05:11:49.675 INFO:tasks.rados.rados.2.smithi204.stdout:356: expect (ObjNum 105 snap 18 seq_num 105) 2024-09-16T05:11:49.675 INFO:tasks.rados.rados.2.smithi204.stdout:357: write oid 45 current snap is 39 2024-09-16T05:11:49.675 INFO:tasks.rados.rados.2.smithi204.stdout:357: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-16T05:11:49.681 INFO:tasks.rados.rados.2.smithi204.stdout:357: writing smithi20457400-45 from 605085 to 1395740 tid 1 2024-09-16T05:11:49.688 INFO:tasks.rados.rados.2.smithi204.stdout:357: writing smithi20457400-45 from 2170356 to 2605085 tid 2 2024-09-16T05:11:49.688 INFO:tasks.rados.rados.2.smithi204.stdout:358: write oid 25 current snap is 39 2024-09-16T05:11:49.688 INFO:tasks.rados.rados.2.smithi204.stdout:354: finishing write tid 1 to smithi20457400-39 2024-09-16T05:11:49.688 INFO:tasks.rados.rados.2.smithi204.stdout:353: finishing copy_from racing read to smithi20457400-7 2024-09-16T05:11:49.688 INFO:tasks.rados.rados.2.smithi204.stdout:353: finishing copy_from to smithi20457400-7 2024-09-16T05:11:49.689 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 32 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T05:11:49.689 INFO:tasks.rados.rados.2.smithi204.stdout:354: finishing write tid 2 to smithi20457400-39 2024-09-16T05:11:49.689 INFO:tasks.rados.rados.2.smithi204.stdout:358: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-16T05:11:49.694 INFO:tasks.rados.rados.2.smithi204.stdout:358: writing smithi20457400-25 from 621892 to 1087796 tid 1 2024-09-16T05:11:49.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:49.702 INFO:tasks.rados.rados.2.smithi204.stdout:358: writing smithi20457400-25 from 1712485 to 2384196 tid 2 2024-09-16T05:11:49.702 INFO:tasks.rados.rados.2.smithi204.stdout:353: done (5 left) 2024-09-16T05:11:49.702 INFO:tasks.rados.rados.2.smithi204.stdout:359: snap_remove snap 36 2024-09-16T05:11:49.712 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:49 smithi032 ceph-mon[27388]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:49.713 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:49 smithi032 ceph-mon[27388]: pgmap v417: 27 pgs: 27 active+clean; 208 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-16T05:11:49.713 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:49 smithi032 ceph-mon[27388]: osdmap e177: 12 total, 12 up, 12 in 2024-09-16T05:11:49.721 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:49.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:49.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:11:49.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:49.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:49.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:49.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:49 smithi204 ceph-mon[30933]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:49 smithi204 ceph-mon[30933]: pgmap v417: 27 pgs: 27 active+clean; 208 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-16T05:11:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:49 smithi204 ceph-mon[30933]: osdmap e177: 12 total, 12 up, 12 in 2024-09-16T05:11:49.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:49.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:49.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:49.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:49 smithi196 ceph-mon[30802]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:11:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:49 smithi196 ceph-mon[30802]: pgmap v417: 27 pgs: 27 active+clean; 208 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-16T05:11:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:49 smithi196 ceph-mon[30802]: osdmap e177: 12 total, 12 up, 12 in 2024-09-16T05:11:50.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:50.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:50.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:50.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:50.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:50.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:50.350 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:50.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:50.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:50.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:50.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:50.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:50.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:50.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:50.620 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:50.642 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:50.642 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:50.652 INFO:tasks.rados.rados.2.smithi204.stdout:359: done (5 left) 2024-09-16T05:11:50.652 INFO:tasks.rados.rados.2.smithi204.stdout:360: delete oid 6 current snap is 39 2024-09-16T05:11:50.652 INFO:tasks.rados.rados.2.smithi204.stdout:354: finishing write tid 3 to smithi20457400-39 2024-09-16T05:11:50.652 INFO:tasks.rados.rados.2.smithi204.stdout:354: finishing write tid 4 to smithi20457400-39 2024-09-16T05:11:50.652 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 221 (ObjNum 154 snap 39 seq_num 154) dirty exists 2024-09-16T05:11:50.652 INFO:tasks.rados.rados.2.smithi204.stdout:354: left oid 39 (ObjNum 154 snap 39 seq_num 154) 2024-09-16T05:11:50.652 INFO:tasks.rados.rados.2.smithi204.stdout:356: expect (ObjNum 105 snap 18 seq_num 105) 2024-09-16T05:11:50.664 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:50.665 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:50.685 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:50.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:50.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:50.707 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:50.707 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:50.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:50.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:50.729 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:50.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:50.750 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:50.750 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:50.809 INFO:tasks.rados.rados.2.smithi204.stdout:357: finishing write tid 1 to smithi20457400-45 2024-09-16T05:11:50.809 INFO:tasks.rados.rados.2.smithi204.stdout:357: finishing write tid 2 to smithi20457400-45 2024-09-16T05:11:50.809 INFO:tasks.rados.rados.2.smithi204.stdout:357: finishing write tid 3 to smithi20457400-45 2024-09-16T05:11:50.809 INFO:tasks.rados.rados.2.smithi204.stdout:357: finishing write tid 4 to smithi20457400-45 2024-09-16T05:11:50.809 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 80 (ObjNum 155 snap 39 seq_num 155) dirty exists 2024-09-16T05:11:50.809 INFO:tasks.rados.rados.2.smithi204.stdout:357: left oid 45 (ObjNum 155 snap 39 seq_num 155) 2024-09-16T05:11:50.809 INFO:tasks.rados.rados.2.smithi204.stdout:358: finishing write tid 1 to smithi20457400-25 2024-09-16T05:11:50.810 INFO:tasks.rados.rados.2.smithi204.stdout:358: finishing write tid 2 to smithi20457400-25 2024-09-16T05:11:50.810 INFO:tasks.rados.rados.2.smithi204.stdout:358: finishing write tid 3 to smithi20457400-25 2024-09-16T05:11:50.810 INFO:tasks.rados.rados.2.smithi204.stdout:358: finishing write tid 4 to smithi20457400-25 2024-09-16T05:11:50.810 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 177 (ObjNum 156 snap 39 seq_num 156) dirty exists 2024-09-16T05:11:50.810 INFO:tasks.rados.rados.2.smithi204.stdout:358: left oid 25 (ObjNum 156 snap 39 seq_num 156) 2024-09-16T05:11:50.810 INFO:tasks.rados.rados.2.smithi204.stdout:355: expect (ObjNum 82 snap 7 seq_num 82) 2024-09-16T05:11:50.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:50 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:11:50.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:50 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:11:50.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:50 smithi032 ceph-mon[27388]: osdmap e178: 12 total, 12 up, 12 in 2024-09-16T05:11:50.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:50 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:11:50.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:50 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:11:50.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1584411270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:50.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:50 smithi032 ceph-mon[27388]: osdmap e179: 12 total, 12 up, 12 in 2024-09-16T05:11:50.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:50.836 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:50.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:50.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:50.868 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:50.868 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:50.870 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:50.894 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:50.915 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:50.916 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:50.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:50 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:11:50.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:50 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:11:50.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:50 smithi204 ceph-mon[30933]: osdmap e178: 12 total, 12 up, 12 in 2024-09-16T05:11:50.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:50 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:11:50.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:50 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:11:50.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1584411270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:50.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:50 smithi204 ceph-mon[30933]: osdmap e179: 12 total, 12 up, 12 in 2024-09-16T05:11:50.936 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:50.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:50.958 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:50.971 INFO:tasks.rados.rados.2.smithi204.stdout:354: done (5 left) 2024-09-16T05:11:50.971 INFO:tasks.rados.rados.2.smithi204.stdout:355: done (4 left) 2024-09-16T05:11:50.971 INFO:tasks.rados.rados.2.smithi204.stdout:356: done (3 left) 2024-09-16T05:11:50.971 INFO:tasks.rados.rados.2.smithi204.stdout:357: done (2 left) 2024-09-16T05:11:50.971 INFO:tasks.rados.rados.2.smithi204.stdout:358: done (1 left) 2024-09-16T05:11:50.971 INFO:tasks.rados.rados.2.smithi204.stdout:360: done (0 left) 2024-09-16T05:11:50.971 INFO:tasks.rados.rados.2.smithi204.stdout:361: write oid 39 current snap is 39 2024-09-16T05:11:50.971 INFO:tasks.rados.rados.2.smithi204.stdout:361: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-16T05:11:50.977 INFO:tasks.rados.rados.2.smithi204.stdout:361: writing smithi20457400-39 from 655506 to 1271908 tid 1 2024-09-16T05:11:50.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:50.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:50.984 INFO:tasks.rados.rados.2.smithi204.stdout:361: writing smithi20457400-39 from 1829449 to 2504829 tid 2 2024-09-16T05:11:50.984 INFO:tasks.rados.rados.2.smithi204.stdout:362: write (excl) oid 19 current snap is 39 2024-09-16T05:11:50.984 INFO:tasks.rados.rados.2.smithi204.stdout:362: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-16T05:11:50.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:50 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:11:50.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:50 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:11:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:50 smithi196 ceph-mon[30802]: osdmap e178: 12 total, 12 up, 12 in 2024-09-16T05:11:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:50 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:11:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:50 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:11:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1584411270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:50 smithi196 ceph-mon[30802]: osdmap e179: 12 total, 12 up, 12 in 2024-09-16T05:11:50.992 INFO:tasks.rados.rados.2.smithi204.stdout:362: writing smithi20457400-19 from 672313 to 1363964 tid 1 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:362: writing smithi20457400-19 from 1887931 to 2306969 tid 2 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:363: copy_from oid 12 from oid 31 current snap is 39 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:361: finishing write tid 1 to smithi20457400-39 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:361: finishing write tid 2 to smithi20457400-39 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:361: finishing write tid 3 to smithi20457400-39 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:361: finishing write tid 4 to smithi20457400-39 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 228 (ObjNum 158 snap 39 seq_num 158) dirty exists 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:361: left oid 39 (ObjNum 158 snap 39 seq_num 158) 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:362: finishing write tid 1 to smithi20457400-19 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:361: done (2 left) 2024-09-16T05:11:50.998 INFO:tasks.rados.rados.2.smithi204.stdout:364: read oid 50 snap -1 2024-09-16T05:11:50.999 INFO:tasks.rados.rados.2.smithi204.stdout:364: expect deleted 2024-09-16T05:11:50.999 INFO:tasks.rados.rados.2.smithi204.stdout:365: copy_from oid 21 from oid 44 current snap is 39 2024-09-16T05:11:50.999 INFO:tasks.rados.rados.2.smithi204.stdout:366: rollback oid 38 current snap is 39 2024-09-16T05:11:50.999 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 38 to 34 2024-09-16T05:11:50.999 INFO:tasks.rados.rados.2.smithi204.stdout:367: read oid 32 snap -1 2024-09-16T05:11:51.000 INFO:tasks.rados.rados.2.smithi204.stdout:367: expect (ObjNum 140 snap 29 seq_num 140) 2024-09-16T05:11:51.000 INFO:tasks.rados.rados.2.smithi204.stdout:368: rollback oid 40 current snap is 39 2024-09-16T05:11:51.000 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 40 to 30 2024-09-16T05:11:51.000 INFO:tasks.rados.rados.2.smithi204.stdout:369: read oid 48 snap 29 2024-09-16T05:11:51.000 INFO:tasks.rados.rados.2.smithi204.stdout:369: expect (ObjNum 131 snap 27 seq_num 131) 2024-09-16T05:11:51.000 INFO:tasks.rados.rados.2.smithi204.stdout:370: delete oid 11 current snap is 39 2024-09-16T05:11:51.001 INFO:tasks.rados.rados.2.smithi204.stdout:366: finishing rollback tid 0 to smithi20457400-38 2024-09-16T05:11:51.002 INFO:tasks.rados.rados.2.smithi204.stdout:366: finishing rollback tid 1 to smithi20457400-38 2024-09-16T05:11:51.002 INFO:tasks.rados.rados.2.smithi204.stdout:366: finishing rollback tid 2 to smithi20457400-38 2024-09-16T05:11:51.002 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 232 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-16T05:11:51.002 INFO:tasks.rados.rados.2.smithi204.stdout:364: done (8 left) 2024-09-16T05:11:51.002 INFO:tasks.rados.rados.2.smithi204.stdout:366: done (7 left) 2024-09-16T05:11:51.002 INFO:tasks.rados.rados.2.smithi204.stdout:370: done (6 left) 2024-09-16T05:11:51.002 INFO:tasks.rados.rados.2.smithi204.stdout:371: snap_create 2024-09-16T05:11:51.002 INFO:tasks.rados.rados.2.smithi204.stdout:368: finishing rollback tid 0 to smithi20457400-40 2024-09-16T05:11:51.003 INFO:tasks.rados.rados.2.smithi204.stdout:368: finishing rollback tid 1 to smithi20457400-40 2024-09-16T05:11:51.004 INFO:tasks.rados.rados.2.smithi204.stdout:368: finishing rollback tid 2 to smithi20457400-40 2024-09-16T05:11:51.004 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 113 (ObjNum 135 snap 29 seq_num 135) dirty exists 2024-09-16T05:11:51.008 INFO:tasks.rados.rados.2.smithi204.stdout:367: expect (ObjNum 140 snap 29 seq_num 140) 2024-09-16T05:11:51.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:51.105 INFO:tasks.rados.rados.2.smithi204.stdout:369: expect (ObjNum 131 snap 27 seq_num 131) 2024-09-16T05:11:51.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:51.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:51.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:51.213 INFO:tasks.rados.rados.2.smithi204.stdout:362: finishing write tid 2 to smithi20457400-19 2024-09-16T05:11:51.213 INFO:tasks.rados.rados.2.smithi204.stdout:363: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:11:51.213 INFO:tasks.rados.rados.2.smithi204.stdout:365: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:11:51.213 INFO:tasks.rados.rados.2.smithi204.stdout:363: finishing copy_from to smithi20457400-12 2024-09-16T05:11:51.214 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 125 (ObjNum 124 snap 26 seq_num 124) dirty exists 2024-09-16T05:11:51.214 INFO:tasks.rados.rados.2.smithi204.stdout:365: finishing copy_from to smithi20457400-21 2024-09-16T05:11:51.214 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 40 (ObjNum 111 snap 19 seq_num 111) dirty exists 2024-09-16T05:11:51.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:51.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:51.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:51.362 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:51.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:51.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:51.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:51.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:51.536 INFO:tasks.rados.rados.2.smithi204.stdout:362: finishing write tid 3 to smithi20457400-19 2024-09-16T05:11:51.536 INFO:tasks.rados.rados.2.smithi204.stdout:362: finishing write tid 4 to smithi20457400-19 2024-09-16T05:11:51.536 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 105 (ObjNum 159 snap 39 seq_num 159) dirty exists 2024-09-16T05:11:51.536 INFO:tasks.rados.rados.2.smithi204.stdout:362: left oid 19 (ObjNum 159 snap 39 seq_num 159) 2024-09-16T05:11:51.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:51.613 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:51.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:362: done (6 left) 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:363: done (5 left) 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:365: done (4 left) 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:367: done (3 left) 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:368: done (2 left) 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:369: done (1 left) 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:371: done (0 left) 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:372: read oid 9 snap 19 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:372: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T05:11:51.669 INFO:tasks.rados.rados.2.smithi204.stdout:373: delete oid 24 current snap is 40 2024-09-16T05:11:51.670 INFO:tasks.rados.rados.2.smithi204.stdout:373: done (1 left) 2024-09-16T05:11:51.670 INFO:tasks.rados.rados.2.smithi204.stdout:374: write (excl) oid 10 current snap is 40 2024-09-16T05:11:51.670 INFO:tasks.rados.rados.2.smithi204.stdout:374: seq_num 162 ranges {722734=633751,2063377=659357} 2024-09-16T05:11:51.677 INFO:tasks.rados.rados.2.smithi204.stdout:374: writing smithi20457400-10 from 722734 to 1356485 tid 1 2024-09-16T05:11:51.681 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:51 smithi032 ceph-mon[27388]: pgmap v420: 27 pgs: 27 active+clean; 208 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-16T05:11:51.681 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:51 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:11:51.681 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:51 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:11:51.681 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:51 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:11:51.681 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:51 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:11:51.681 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1551656030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:51.685 INFO:tasks.rados.rados.2.smithi204.stdout:374: writing smithi20457400-10 from 2063377 to 2722734 tid 2 2024-09-16T05:11:51.685 INFO:tasks.rados.rados.2.smithi204.stdout:375: snap_remove snap 13 2024-09-16T05:11:51.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:51.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:51.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:51.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:51.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:51.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:51.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:51 smithi204 ceph-mon[30933]: pgmap v420: 27 pgs: 27 active+clean; 208 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-16T05:11:51.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:51 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:11:51.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:51 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:11:51.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:51 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:11:51.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:51 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:11:51.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1551656030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:51.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:51 smithi196 ceph-mon[30802]: pgmap v420: 27 pgs: 27 active+clean; 208 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2024-09-16T05:11:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:51 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:11:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:51 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:11:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:51 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:11:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:51 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:11:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1551656030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:11:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:51.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:52.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:52.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:52.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:52.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:52.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:52.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:52.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:52.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:52.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:52.467 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:52.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:52.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:52.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:52.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:52.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:52.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:52.681 INFO:tasks.rados.rados.2.smithi204.stdout:372: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-16T05:11:52.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:52.723 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:11:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:11:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:11:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:11:52.723 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:11:52.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:52.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:52.751 INFO:tasks.rados.rados.2.smithi204.stdout:374: finishing write tid 1 to smithi20457400-10 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:374: finishing write tid 2 to smithi20457400-10 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:374: finishing write tid 3 to smithi20457400-10 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:374: finishing write tid 4 to smithi20457400-10 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 62 (ObjNum 162 snap 40 seq_num 162) dirty exists 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:374: left oid 10 (ObjNum 162 snap 40 seq_num 162) 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:372: done (2 left) 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:374: done (1 left) 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:375: done (0 left) 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:376: copy_from oid 21 from oid 40 current snap is 40 2024-09-16T05:11:52.752 INFO:tasks.rados.rados.2.smithi204.stdout:377: snap_remove snap 38 2024-09-16T05:11:52.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:52.872 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:52.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:52 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:11:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:52 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:11:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:52 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:52 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:11:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1551656030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:52.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:52 smithi032 ceph-mon[27388]: osdmap e180: 12 total, 12 up, 12 in 2024-09-16T05:11:52.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:52 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:11:52.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:52 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:11:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:52 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:11:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:52 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:11:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:52 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:52.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:52 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:11:52.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1551656030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:52.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:52 smithi204 ceph-mon[30933]: osdmap e180: 12 total, 12 up, 12 in 2024-09-16T05:11:52.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:52 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:11:52.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:52 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:11:52.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:52 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:11:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:52 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:11:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:52 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:11:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:52 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:11:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1551656030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:11:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:52 smithi196 ceph-mon[30802]: osdmap e180: 12 total, 12 up, 12 in 2024-09-16T05:11:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:52 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:11:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:52 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:11:52.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:52.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:52.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:53.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:53.120 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:53.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:53.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:53.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:53.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:53.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:53.370 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:53.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:53.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:53.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:53.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:53.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:11:53.596 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:53.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:53.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:53.683 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pg_num to 26 2024-09-16T05:11:53.690 INFO:tasks.rados.rados.2.smithi204.stdout:377: done (1 left) 2024-09-16T05:11:53.690 INFO:tasks.rados.rados.2.smithi204.stdout:378: rollback oid 37 current snap is 40 2024-09-16T05:11:53.690 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 34 2024-09-16T05:11:53.690 INFO:tasks.rados.rados.2.smithi204.stdout:376: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:11:53.690 INFO:tasks.rados.rados.2.smithi204.stdout:376: finishing copy_from to smithi20457400-21 2024-09-16T05:11:53.690 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 113 (ObjNum 135 snap 29 seq_num 135) dirty exists 2024-09-16T05:11:53.690 INFO:tasks.rados.rados.2.smithi204.stdout:376: done (1 left) 2024-09-16T05:11:53.690 INFO:tasks.rados.rados.2.smithi204.stdout:379: delete oid 34 current snap is 40 2024-09-16T05:11:53.693 INFO:tasks.rados.rados.2.smithi204.stdout:378: finishing rollback tid 0 to smithi20457400-37 2024-09-16T05:11:53.693 INFO:tasks.rados.rados.2.smithi204.stdout:378: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:11:53.693 INFO:tasks.rados.rados.2.smithi204.stdout:378: finishing rollback tid 2 to smithi20457400-37 2024-09-16T05:11:53.693 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 92 (ObjNum 129 snap 27 seq_num 129) dirty exists 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:378: done (1 left) 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:379: done (0 left) 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:380: rollback oid 5 current snap is 40 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 5 to 33 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:381: read oid 17 snap -1 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:381: expect deleted 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:382: copy_from oid 3 from oid 47 current snap is 40 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:383: read oid 50 snap -1 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:383: expect deleted 2024-09-16T05:11:53.694 INFO:tasks.rados.rados.2.smithi204.stdout:384: delete oid 38 current snap is 40 2024-09-16T05:11:53.695 INFO:tasks.rados.rados.2.smithi204.stdout:380: finishing rollback tid 0 to smithi20457400-5 2024-09-16T05:11:53.695 INFO:tasks.rados.rados.2.smithi204.stdout:384: done (4 left) 2024-09-16T05:11:53.696 INFO:tasks.rados.rados.2.smithi204.stdout:385: write (excl) oid 15 current snap is 40 2024-09-16T05:11:53.696 INFO:tasks.rados.rados.2.smithi204.stdout:385: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-16T05:11:53.697 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:53 smithi032 ceph-mon[27388]: pgmap v423: 27 pgs: 1 active+clean+snaptrim, 26 active+clean; 208 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-16T05:11:53.697 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:53 smithi032 ceph-mon[27388]: osdmap e181: 12 total, 12 up, 12 in 2024-09-16T05:11:53.697 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:53 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:11:53.698 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:53 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:11:53.698 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2538512693' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-16T05:11:53.705 INFO:tasks.rados.rados.2.smithi204.stdout:385: writing smithi20457400-15 from 773155 to 1232653 tid 1 2024-09-16T05:11:53.708 INFO:tasks.rados.rados.2.smithi204.stdout:385: writing smithi20457400-15 from 1722470 to 2152515 tid 2 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:385: writing smithi20457400-15 from 2648676 to 2773155 tid 3 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:386: rollback oid 28 current snap is 40 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 28 to 31 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:380: finishing rollback tid 1 to smithi20457400-5 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:380: finishing rollback tid 2 to smithi20457400-5 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 116 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:380: done (5 left) 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:381: done (4 left) 2024-09-16T05:11:53.712 INFO:tasks.rados.rados.2.smithi204.stdout:387: snap_remove snap 19 2024-09-16T05:11:53.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:11:53.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:53.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:53.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:53.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:53.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:53.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:53.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:53 smithi196 ceph-mon[30802]: pgmap v423: 27 pgs: 1 active+clean+snaptrim, 26 active+clean; 208 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-16T05:11:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:53 smithi196 ceph-mon[30802]: osdmap e181: 12 total, 12 up, 12 in 2024-09-16T05:11:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:53 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:11:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:53 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:11:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2538512693' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-16T05:11:53.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:53.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:54.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:54.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:54.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:53 smithi204 ceph-mon[30933]: pgmap v423: 27 pgs: 1 active+clean+snaptrim, 26 active+clean; 208 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-16T05:11:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:53 smithi204 ceph-mon[30933]: osdmap e181: 12 total, 12 up, 12 in 2024-09-16T05:11:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:53 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:11:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:53 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:11:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2538512693' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-16T05:11:54.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:54.247 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:54.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:54.302 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:54.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:11:54.343 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:54.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:54.372 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:54.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:54.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:54.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:54.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:54.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:54.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:54.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:54.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:54.694 INFO:tasks.rados.rados.2.smithi204.stdout:387: done (4 left) 2024-09-16T05:11:54.694 INFO:tasks.rados.rados.2.smithi204.stdout:388: write (excl) oid 41 current snap is 40 2024-09-16T05:11:54.694 INFO:tasks.rados.rados.2.smithi204.stdout:388: seq_num 166 ranges {789962=534747,1780952=690056} 2024-09-16T05:11:54.702 INFO:tasks.rados.rados.2.smithi204.stdout:388: writing smithi20457400-41 from 789962 to 1324709 tid 1 2024-09-16T05:11:54.707 INFO:tasks.rados.rados.2.smithi204.stdout:388: writing smithi20457400-41 from 1780952 to 2471008 tid 2 2024-09-16T05:11:54.708 INFO:tasks.rados.rados.2.smithi204.stdout:389: snap_remove snap 33 2024-09-16T05:11:54.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:54.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:54.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:54.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:54.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:54 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:11:54.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:54 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:11:54.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2538512693' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-16T05:11:54.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:54 smithi032 ceph-mon[27388]: osdmap e182: 12 total, 12 up, 12 in 2024-09-16T05:11:54.868 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:54 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-16T05:11:54.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:54.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:54.894 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:54.894 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:54.894 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:54.894 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:54.913 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:54.913 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:54.929 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:54.936 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:54.937 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:54.957 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:54.958 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:54.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:54.981 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:11:54.981 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:11:54.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:54 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:11:54.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:54 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:11:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2538512693' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-16T05:11:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:54 smithi196 ceph-mon[30802]: osdmap e182: 12 total, 12 up, 12 in 2024-09-16T05:11:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:54 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-16T05:11:55.002 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:11:55.002 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:11:55.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:55.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:55.023 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:11:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:11:55.047 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:11:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:11:55.068 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:11:55.069 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:11:55.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:11:55.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:11:55.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:55.111 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:11:55.112 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:11:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:55.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:55.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:54 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:11:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:54 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:11:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2538512693' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-16T05:11:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:54 smithi204 ceph-mon[30933]: osdmap e182: 12 total, 12 up, 12 in 2024-09-16T05:11:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:54 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-16T05:11:55.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:55.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:55.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:55.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:11:55.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:55.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:11:55.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:55.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:55.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:55.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:55.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:55.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:55.653 INFO:tasks.rados.rados.2.smithi204.stdout:389: done (5 left) 2024-09-16T05:11:55.653 INFO:tasks.rados.rados.2.smithi204.stdout:390: delete oid 48 current snap is 40 2024-09-16T05:11:55.653 INFO:tasks.rados.rados.2.smithi204.stdout:382: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:11:55.653 INFO:tasks.rados.rados.2.smithi204.stdout:382: finishing copy_from to smithi20457400-3 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 66 (ObjNum 146 snap 33 seq_num 146) dirty exists 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:385: finishing write tid 1 to smithi20457400-15 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:385: finishing write tid 2 to smithi20457400-15 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:385: finishing write tid 3 to smithi20457400-15 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:385: finishing write tid 4 to smithi20457400-15 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:385: finishing write tid 5 to smithi20457400-15 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 20 (ObjNum 165 snap 40 seq_num 165) dirty exists 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:385: left oid 15 (ObjNum 165 snap 40 seq_num 165) 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:386: finishing rollback tid 0 to smithi20457400-28 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:386: finishing rollback tid 1 to smithi20457400-28 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:386: finishing rollback tid 2 to smithi20457400-28 2024-09-16T05:11:55.654 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 125 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:11:55.663 INFO:tasks.rados.rados.2.smithi204.stdout:382: done (5 left) 2024-09-16T05:11:55.663 INFO:tasks.rados.rados.2.smithi204.stdout:383: done (4 left) 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:385: done (3 left) 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:386: done (2 left) 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:390: done (1 left) 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:391: rollback oid 18 current snap is 40 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 18 to 37 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:392: read oid 13 snap -1 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:392: expect (ObjNum 116 snap 25 seq_num 116) 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:393: rollback oid 35 current snap is 40 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 35 to 32 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:394: snap_create 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:391: finishing rollback tid 1 to smithi20457400-18 2024-09-16T05:11:55.664 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 129 snap 0 seq_num 4282317968) dirty dne 2024-09-16T05:11:55.669 INFO:tasks.rados.rados.2.smithi204.stdout:393: finishing rollback tid 0 to smithi20457400-35 2024-09-16T05:11:55.669 INFO:tasks.rados.rados.2.smithi204.stdout:393: finishing rollback tid 1 to smithi20457400-35 2024-09-16T05:11:55.669 INFO:tasks.rados.rados.2.smithi204.stdout:393: finishing rollback tid 2 to smithi20457400-35 2024-09-16T05:11:55.669 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 190 (ObjNum 103 snap 17 seq_num 103) dirty exists 2024-09-16T05:11:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:388: finishing write tid 1 to smithi20457400-41 2024-09-16T05:11:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:388: finishing write tid 2 to smithi20457400-41 2024-09-16T05:11:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:388: finishing write tid 3 to smithi20457400-41 2024-09-16T05:11:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:388: finishing write tid 4 to smithi20457400-41 2024-09-16T05:11:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 65 (ObjNum 166 snap 40 seq_num 166) dirty exists 2024-09-16T05:11:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:388: left oid 41 (ObjNum 166 snap 40 seq_num 166) 2024-09-16T05:11:55.672 INFO:tasks.rados.rados.2.smithi204.stdout:392: expect (ObjNum 116 snap 25 seq_num 116) 2024-09-16T05:11:55.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:55.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:55.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:55.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: pgmap v426: 27 pgs: 1 active+clean+snaptrim, 26 active+clean; 209 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2024-09-16T05:11:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:11:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:11:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-16T05:11:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: osdmap e183: 12 total, 12 up, 12 in 2024-09-16T05:11:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3818866194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:11:55.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:11:55.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:55 smithi032 ceph-mon[27388]: osdmap e184: 12 total, 12 up, 12 in 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: pgmap v426: 27 pgs: 1 active+clean+snaptrim, 26 active+clean; 209 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: osdmap e183: 12 total, 12 up, 12 in 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3818866194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:11:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:55 smithi196 ceph-mon[30802]: osdmap e184: 12 total, 12 up, 12 in 2024-09-16T05:11:55.998 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:56.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:56.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:56.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:56.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:56.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: pgmap v426: 27 pgs: 1 active+clean+snaptrim, 26 active+clean; 209 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: osdmap e183: 12 total, 12 up, 12 in 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3818866194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:11:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:55 smithi204 ceph-mon[30933]: osdmap e184: 12 total, 12 up, 12 in 2024-09-16T05:11:56.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:56.282 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:56.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:56.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:11:56.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:56.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:11:56.449 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:56.449 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:56.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:56.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:56.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:56.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:56.650 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:11:56.657 INFO:tasks.rados.rados.2.smithi204.stdout:388: done (4 left) 2024-09-16T05:11:56.657 INFO:tasks.rados.rados.2.smithi204.stdout:391: done (3 left) 2024-09-16T05:11:56.658 INFO:tasks.rados.rados.2.smithi204.stdout:392: done (2 left) 2024-09-16T05:11:56.658 INFO:tasks.rados.rados.2.smithi204.stdout:393: done (1 left) 2024-09-16T05:11:56.658 INFO:tasks.rados.rados.2.smithi204.stdout:394: done (0 left) 2024-09-16T05:11:56.658 INFO:tasks.rados.rados.2.smithi204.stdout:395: write oid 11 current snap is 41 2024-09-16T05:11:56.658 INFO:tasks.rados.rados.2.smithi204.stdout:395: seq_num 168 ranges {423576=401598,1497916=693725} 2024-09-16T05:11:56.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:56.662 INFO:tasks.rados.rados.2.smithi204.stdout:395: writing smithi20457400-11 from 423576 to 825174 tid 1 2024-09-16T05:11:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:56.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:56.670 INFO:tasks.rados.rados.2.smithi204.stdout:395: writing smithi20457400-11 from 1497916 to 2191641 tid 2 2024-09-16T05:11:56.670 INFO:tasks.rados.rados.2.smithi204.stdout:396: read oid 32 snap -1 2024-09-16T05:11:56.670 INFO:tasks.rados.rados.2.smithi204.stdout:396: expect (ObjNum 140 snap 29 seq_num 140) 2024-09-16T05:11:56.671 INFO:tasks.rados.rados.2.smithi204.stdout:397: rollback oid 20 current snap is 41 2024-09-16T05:11:56.671 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 20 to 37 2024-09-16T05:11:56.671 INFO:tasks.rados.rados.2.smithi204.stdout:395: finishing write tid 1 to smithi20457400-11 2024-09-16T05:11:56.671 INFO:tasks.rados.rados.2.smithi204.stdout:398: read oid 21 snap 40 2024-09-16T05:11:56.671 INFO:tasks.rados.rados.2.smithi204.stdout:398: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-16T05:11:56.671 INFO:tasks.rados.rados.2.smithi204.stdout:399: write (excl) oid 19 current snap is 41 2024-09-16T05:11:56.671 INFO:tasks.rados.rados.2.smithi204.stdout:399: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-16T05:11:56.675 INFO:tasks.rados.rados.2.smithi204.stdout:399: writing smithi20457400-19 from 440383 to 917230 tid 1 2024-09-16T05:11:56.681 INFO:tasks.rados.rados.2.smithi204.stdout:399: writing smithi20457400-19 from 1556398 to 1993781 tid 2 2024-09-16T05:11:56.686 INFO:tasks.rados.rados.2.smithi204.stdout:399: writing smithi20457400-19 from 2474721 to 2840383 tid 3 2024-09-16T05:11:56.686 INFO:tasks.rados.rados.2.smithi204.stdout:400: write (excl) oid 45 current snap is 41 2024-09-16T05:11:56.686 INFO:tasks.rados.rados.2.smithi204.stdout:400: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-16T05:11:56.691 INFO:tasks.rados.rados.2.smithi204.stdout:400: writing smithi20457400-45 from 457190 to 1009286 tid 1 2024-09-16T05:11:56.697 INFO:tasks.rados.rados.2.smithi204.stdout:400: writing smithi20457400-45 from 1614880 to 2195921 tid 2 2024-09-16T05:11:56.702 INFO:tasks.rados.rados.2.smithi204.stdout:400: writing smithi20457400-45 from 2785791 to 2857190 tid 3 2024-09-16T05:11:56.702 INFO:tasks.rados.rados.2.smithi204.stdout:401: snap_create 2024-09-16T05:11:56.702 INFO:tasks.rados.rados.2.smithi204.stdout:395: finishing write tid 2 to smithi20457400-11 2024-09-16T05:11:56.702 INFO:tasks.rados.rados.2.smithi204.stdout:395: finishing write tid 3 to smithi20457400-11 2024-09-16T05:11:56.702 INFO:tasks.rados.rados.2.smithi204.stdout:395: finishing write tid 4 to smithi20457400-11 2024-09-16T05:11:56.702 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 269 (ObjNum 168 snap 41 seq_num 168) dirty exists 2024-09-16T05:11:56.702 INFO:tasks.rados.rados.2.smithi204.stdout:395: left oid 11 (ObjNum 168 snap 41 seq_num 168) 2024-09-16T05:11:56.703 INFO:tasks.rados.rados.2.smithi204.stdout:396: expect (ObjNum 140 snap 29 seq_num 140) 2024-09-16T05:11:56.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:56.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:56.800 INFO:tasks.rados.rados.2.smithi204.stdout:397: finishing rollback tid 0 to smithi20457400-20 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:397: finishing rollback tid 1 to smithi20457400-20 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:399: finishing write tid 1 to smithi20457400-19 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:399: finishing write tid 2 to smithi20457400-19 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:399: finishing write tid 3 to smithi20457400-19 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:399: finishing write tid 4 to smithi20457400-19 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:399: finishing write tid 5 to smithi20457400-19 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 131 (ObjNum 169 snap 41 seq_num 169) dirty exists 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:399: left oid 19 (ObjNum 169 snap 41 seq_num 169) 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:400: finishing write tid 1 to smithi20457400-45 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:400: finishing write tid 2 to smithi20457400-45 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:400: finishing write tid 3 to smithi20457400-45 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:400: finishing write tid 4 to smithi20457400-45 2024-09-16T05:11:56.801 INFO:tasks.rados.rados.2.smithi204.stdout:400: finishing write tid 5 to smithi20457400-45 2024-09-16T05:11:56.802 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 105 (ObjNum 170 snap 41 seq_num 170) dirty exists 2024-09-16T05:11:56.802 INFO:tasks.rados.rados.2.smithi204.stdout:400: left oid 45 (ObjNum 170 snap 41 seq_num 170) 2024-09-16T05:11:56.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:56.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:56.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:56.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3220558035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:56.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:11:56.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3220558035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-16T05:11:56.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:56 smithi032 ceph-mon[27388]: osdmap e185: 12 total, 12 up, 12 in 2024-09-16T05:11:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3220558035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:11:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:11:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:11:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:11:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-16T05:11:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:11:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:11:56.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:56.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3220558035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:56.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-16T05:11:56.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:56 smithi196 ceph-mon[30802]: osdmap e185: 12 total, 12 up, 12 in 2024-09-16T05:11:57.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:57.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:57.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:57.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:57.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:57.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3220558035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:11:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:11:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3220558035' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-16T05:11:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:56 smithi204 ceph-mon[30933]: osdmap e185: 12 total, 12 up, 12 in 2024-09-16T05:11:57.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:57.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:57.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:57.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:57.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:57.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:57.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:57.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:57.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:57.622 INFO:tasks.rados.rados.2.smithi204.stdout:397: finishing rollback tid 2 to smithi20457400-20 2024-09-16T05:11:57.623 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 273 (ObjNum 101 snap 16 seq_num 101) dirty exists 2024-09-16T05:11:57.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:395: done (6 left) 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:396: done (5 left) 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:397: done (4 left) 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:399: done (3 left) 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:400: done (2 left) 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:401: done (1 left) 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:402: read oid 47 snap 31 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:402: expect deleted 2024-09-16T05:11:57.657 INFO:tasks.rados.rados.2.smithi204.stdout:403: delete oid 31 current snap is 42 2024-09-16T05:11:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:402: done (2 left) 2024-09-16T05:11:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:403: done (1 left) 2024-09-16T05:11:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:404: read oid 48 snap -1 2024-09-16T05:11:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:404: expect deleted 2024-09-16T05:11:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:405: copy_from oid 3 from oid 36 current snap is 42 2024-09-16T05:11:57.660 INFO:tasks.rados.rados.2.smithi204.stdout:406: write oid 31 current snap is 42 2024-09-16T05:11:57.661 INFO:tasks.rados.rados.2.smithi204.stdout:406: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-16T05:11:57.667 INFO:tasks.rados.rados.2.smithi204.stdout:406: writing smithi20457400-31 from 490804 to 1193398 tid 1 2024-09-16T05:11:57.672 INFO:tasks.rados.rados.2.smithi204.stdout:406: writing smithi20457400-31 from 1615491 to 2200201 tid 2 2024-09-16T05:11:57.677 INFO:tasks.rados.rados.2.smithi204.stdout:406: writing smithi20457400-31 from 2724284 to 2890804 tid 3 2024-09-16T05:11:57.677 INFO:tasks.rados.rados.2.smithi204.stdout:407: snap_create 2024-09-16T05:11:57.677 INFO:tasks.rados.rados.2.smithi204.stdout:398: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-16T05:11:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:57.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:57.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:11:57.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:11:57.754 INFO:tasks.rados.rados.2.smithi204.stdout:406: finishing write tid 1 to smithi20457400-31 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:405: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:406: finishing write tid 2 to smithi20457400-31 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:405: finishing copy_from to smithi20457400-3 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 164 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:406: finishing write tid 3 to smithi20457400-31 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:406: finishing write tid 4 to smithi20457400-31 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:406: finishing write tid 5 to smithi20457400-31 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 199 (ObjNum 172 snap 42 seq_num 172) dirty exists 2024-09-16T05:11:57.755 INFO:tasks.rados.rados.2.smithi204.stdout:406: left oid 31 (ObjNum 172 snap 42 seq_num 172) 2024-09-16T05:11:57.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:57.801 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:57.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:57.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:57 smithi032 ceph-mon[27388]: pgmap v429: 37 pgs: 1 active+clean+scrubbing, 3 peering, 7 unknown, 26 active+clean; 202 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2024-09-16T05:11:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:57 smithi032 ceph-mon[27388]: 2.9 deep-scrub starts 2024-09-16T05:11:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:57 smithi032 ceph-mon[27388]: 2.9 deep-scrub ok 2024-09-16T05:11:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:57 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:11:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:57 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:11:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:57 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:11:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:57 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:11:57.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:57 smithi032 ceph-mon[27388]: osdmap e186: 12 total, 12 up, 12 in 2024-09-16T05:11:57.929 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:57.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:57.948 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:57 smithi196 ceph-mon[30802]: pgmap v429: 37 pgs: 1 active+clean+scrubbing, 3 peering, 7 unknown, 26 active+clean; 202 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2024-09-16T05:11:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:57 smithi196 ceph-mon[30802]: 2.9 deep-scrub starts 2024-09-16T05:11:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:57 smithi196 ceph-mon[30802]: 2.9 deep-scrub ok 2024-09-16T05:11:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:57 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:11:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:57 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:11:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:57 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:11:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:57 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:11:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:57 smithi196 ceph-mon[30802]: osdmap e186: 12 total, 12 up, 12 in 2024-09-16T05:11:58.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:58.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:58.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:58.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:57 smithi204 ceph-mon[30933]: pgmap v429: 37 pgs: 1 active+clean+scrubbing, 3 peering, 7 unknown, 26 active+clean; 202 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2024-09-16T05:11:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:57 smithi204 ceph-mon[30933]: 2.9 deep-scrub starts 2024-09-16T05:11:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:57 smithi204 ceph-mon[30933]: 2.9 deep-scrub ok 2024-09-16T05:11:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:57 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:11:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:57 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:11:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:57 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:11:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:57 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:11:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:57 smithi204 ceph-mon[30933]: osdmap e186: 12 total, 12 up, 12 in 2024-09-16T05:11:58.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:58.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:58.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:11:58.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:58.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:58.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:58.375 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:11:58.375 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:11:58.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:58.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:58.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:58.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:11:58.560 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:58.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:11:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:398: done (4 left) 2024-09-16T05:11:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:404: done (3 left) 2024-09-16T05:11:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:405: done (2 left) 2024-09-16T05:11:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:406: done (1 left) 2024-09-16T05:11:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:407: done (0 left) 2024-09-16T05:11:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:408: read oid 39 snap -1 2024-09-16T05:11:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:408: expect (ObjNum 158 snap 39 seq_num 158) 2024-09-16T05:11:58.657 INFO:tasks.rados.rados.2.smithi204.stdout:409: delete oid 31 current snap is 43 2024-09-16T05:11:58.661 INFO:tasks.rados.rados.2.smithi204.stdout:408: expect (ObjNum 158 snap 39 seq_num 158) 2024-09-16T05:11:58.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:11:58.683 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:58.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:58.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:11:58.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:11:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:58.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:58.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:11:58.837 INFO:tasks.rados.rados.2.smithi204.stdout:408: done (1 left) 2024-09-16T05:11:58.837 INFO:tasks.rados.rados.2.smithi204.stdout:409: done (0 left) 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:410: copy_from oid 24 from oid 42 current snap is 43 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:411: read oid 37 snap -1 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:411: expect (ObjNum 129 snap 27 seq_num 129) 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:412: read oid 5 snap -1 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:412: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:413: read oid 39 snap -1 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:413: expect (ObjNum 158 snap 39 seq_num 158) 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:414: write oid 22 current snap is 43 2024-09-16T05:11:58.838 INFO:tasks.rados.rados.2.smithi204.stdout:414: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-16T05:11:58.842 INFO:tasks.rados.rados.2.smithi204.stdout:414: writing smithi20457400-22 from 524418 to 977510 tid 1 2024-09-16T05:11:58.850 INFO:tasks.rados.rados.2.smithi204.stdout:414: writing smithi20457400-22 from 1732455 to 2320834 tid 2 2024-09-16T05:11:58.853 INFO:tasks.rados.rados.2.smithi204.stdout:414: writing smithi20457400-22 from 2779130 to 2924418 tid 3 2024-09-16T05:11:58.853 INFO:tasks.rados.rados.2.smithi204.stdout:415: snap_remove snap 35 2024-09-16T05:11:58.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:11:58.931 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:58.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:11:59.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:59.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:59.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:59.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:11:59.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:11:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:59.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:11:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:59.228 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:11:59.228 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:11:59.229 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T05:11:59.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:11:59.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:59.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:11:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:59.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:59.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:59.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:59.436 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:59.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:59.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:11:59.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:59.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:59.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:11:59.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:11:59.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:11:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:59.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:11:59.700 INFO:tasks.rados.rados.2.smithi204.stdout:415: done (5 left) 2024-09-16T05:11:59.700 INFO:tasks.rados.rados.2.smithi204.stdout:416: rollback oid 2 current snap is 43 2024-09-16T05:11:59.700 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 2 to 39 2024-09-16T05:11:59.701 INFO:tasks.rados.rados.2.smithi204.stdout:417: write (excl) oid 41 current snap is 43 2024-09-16T05:11:59.701 INFO:tasks.rados.rados.2.smithi204.stdout:417: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-16T05:11:59.707 INFO:tasks.rados.rados.2.smithi204.stdout:417: writing smithi20457400-41 from 541225 to 1185919 tid 1 2024-09-16T05:11:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:417: writing smithi20457400-41 from 1907290 to 2355680 tid 2 2024-09-16T05:11:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:418: snap_create 2024-09-16T05:11:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:410: finishing copy_from to smithi20457400-24 2024-09-16T05:11:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:410: got expected ENOENT (src dne) 2024-09-16T05:11:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:410: finishing copy_from racing read to smithi20457400-24 2024-09-16T05:11:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:413: expect (ObjNum 158 snap 39 seq_num 158) 2024-09-16T05:11:59.735 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:11:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:11:59.756 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:59.756 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:11:59.761 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:11:59.761 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:11:59.791 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:11:59.791 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:11:59.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:11:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:59.816 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:11:59.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:11:59.816 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:11:59.817 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:11:59.838 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:11:59.838 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:11:59.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:59 smithi032 ceph-mon[27388]: pgmap v432: 37 pgs: 1 active+clean+scrubbing, 5 peering, 5 unknown, 26 active+clean; 167 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 0 op/s 2024-09-16T05:11:59.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:59 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:11:59.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:59 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:11:59.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:11:59 smithi032 ceph-mon[27388]: osdmap e187: 12 total, 12 up, 12 in 2024-09-16T05:11:59.883 INFO:tasks.rados.rados.2.smithi204.stdout:411: expect (ObjNum 129 snap 27 seq_num 129) 2024-09-16T05:11:59.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:11:59.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:59 smithi204 ceph-mon[30933]: pgmap v432: 37 pgs: 1 active+clean+scrubbing, 5 peering, 5 unknown, 26 active+clean; 167 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 0 op/s 2024-09-16T05:11:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:59 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:11:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:59 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:11:59.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:11:59 smithi204 ceph-mon[30933]: osdmap e187: 12 total, 12 up, 12 in 2024-09-16T05:11:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:59.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:11:59.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:11:59.957 INFO:tasks.rados.rados.2.smithi204.stdout:412: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:11:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:59 smithi196 ceph-mon[30802]: pgmap v432: 37 pgs: 1 active+clean+scrubbing, 5 peering, 5 unknown, 26 active+clean; 167 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 0 op/s 2024-09-16T05:11:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:59 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:11:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:59 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:11:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:11:59 smithi196 ceph-mon[30802]: osdmap e187: 12 total, 12 up, 12 in 2024-09-16T05:12:00.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:00.061 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:00.062 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:00.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:00.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:00.083 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:00.087 INFO:tasks.rados.rados.2.smithi204.stdout:414: finishing write tid 1 to smithi20457400-22 2024-09-16T05:12:00.087 INFO:tasks.rados.rados.2.smithi204.stdout:414: finishing write tid 2 to smithi20457400-22 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:414: finishing write tid 3 to smithi20457400-22 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:414: finishing write tid 4 to smithi20457400-22 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:414: finishing write tid 5 to smithi20457400-22 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 136 (ObjNum 174 snap 43 seq_num 174) dirty exists 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:414: left oid 22 (ObjNum 174 snap 43 seq_num 174) 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:416: finishing rollback tid 0 to smithi20457400-2 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:416: finishing rollback tid 1 to smithi20457400-2 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:416: finishing rollback tid 2 to smithi20457400-2 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 69 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:417: finishing write tid 1 to smithi20457400-41 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:417: finishing write tid 2 to smithi20457400-41 2024-09-16T05:12:00.088 INFO:tasks.rados.rados.2.smithi204.stdout:417: finishing write tid 3 to smithi20457400-41 2024-09-16T05:12:00.089 INFO:tasks.rados.rados.2.smithi204.stdout:417: finishing write tid 4 to smithi20457400-41 2024-09-16T05:12:00.089 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 78 (ObjNum 175 snap 43 seq_num 175) dirty exists 2024-09-16T05:12:00.089 INFO:tasks.rados.rados.2.smithi204.stdout:417: left oid 41 (ObjNum 175 snap 43 seq_num 175) 2024-09-16T05:12:00.104 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:00.126 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:00.126 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:00.147 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:00.148 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:00.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:00.168 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:00.168 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:00.187 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:00.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:00.190 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:00.190 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:00.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:00.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:00.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:00.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:00.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:00.487 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:12:00.487 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 188, 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:11:59.695412+0000", 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:09:26.475677+0000", 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:09:05.828775+0000", 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5801984, 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "noscrub", 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:12:00.488 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 28, 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:12:00.489 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:12:00.490 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 11, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 11, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 11, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 11, 2024-09-16T05:12:00.491 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 11, 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "106", 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:12:00.492 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "104", 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:12:00.493 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:12:00.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:12:00.495 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 4.3899998664855957, 2024-09-16T05:12:00.496 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 4.3899998664855957, 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.82999998331069946, 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 3.6400001049041748, 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 3.6400001049041748, 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.497 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:12:00.498 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 26, 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 26, 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 26, 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 26, 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 26, 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:12:00.499 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "188", 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "183", 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 44, 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 188, 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:12:00.500 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:12:00.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:12:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.309999942779541, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.309999942779541, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.309999942779541, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.309999942779541, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:12:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.504 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 187, 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.509 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:12:00.510 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 183, 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:12:00.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 21, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 183, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.516 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3508564664 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3508564664", 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3508564664", 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3508564664", 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3508564664", 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 185, 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.528 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.529 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.530 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 0, 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.534 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:12:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 185, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.538 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:12:00.539 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:12:00.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:12:00.541 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:12:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:12:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 185, 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:12:00.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:12:00.548 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.549 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 183, 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.550 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.551 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:12:00.554 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:12:00.555 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 185, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.558 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.559 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 185, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:12:00.566 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.567 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 185, 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:12:00.568 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:12:00.569 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:12:00.570 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.571 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:12:00.572 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:12:00.573 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 107, 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:12:00.574 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:12:00.577 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:12:00.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.580 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:12:00.580 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.580 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.580 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.580 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:59.105660+0000", 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.581 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:14.594921+0000", 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.582 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:58.222328+0000", 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:27.014040+0000", 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.583 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:21.139048+0000", 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.584 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:08.506447+0000", 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:10:58.893839+0000", 2024-09-16T05:12:00.585 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:29.286641+0000", 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:12:00.586 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:58.589883+0000", 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.587 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:21.438658+0000", 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:27.324418+0000", 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:11:40.265420+0000", 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-09-16T05:12:00.589 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:12:00.590 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:12:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 15, 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 21, 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 25, 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:12:00.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 35, 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 37, 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 2 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 40, 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [ 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 37, 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:12:00.594 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:00.595 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:12:00.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:00.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:00.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:410: done (7 left) 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:411: done (6 left) 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:412: done (5 left) 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:413: done (4 left) 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:414: done (3 left) 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:416: done (2 left) 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:417: done (1 left) 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:418: done (0 left) 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:419: read oid 21 snap -1 2024-09-16T05:12:00.712 INFO:tasks.rados.rados.2.smithi204.stdout:419: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-16T05:12:00.713 INFO:tasks.rados.rados.2.smithi204.stdout:420: write oid 36 current snap is 44 2024-09-16T05:12:00.713 INFO:tasks.rados.rados.2.smithi204.stdout:420: seq_num 176 ranges {558032=719943,1849419=592048} 2024-09-16T05:12:00.719 INFO:tasks.rados.rados.2.smithi204.stdout:420: writing smithi20457400-36 from 558032 to 1277975 tid 1 2024-09-16T05:12:00.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:00.726 INFO:tasks.rados.rados.2.smithi204.stdout:420: writing smithi20457400-36 from 1849419 to 2441467 tid 2 2024-09-16T05:12:00.726 INFO:tasks.rados.rados.2.smithi204.stdout:421: snap_create 2024-09-16T05:12:00.726 INFO:tasks.rados.rados.2.smithi204.stdout:419: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-16T05:12:00.746 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:00.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:00.802 INFO:tasks.rados.rados.2.smithi204.stdout:420: finishing write tid 1 to smithi20457400-36 2024-09-16T05:12:00.802 INFO:tasks.rados.rados.2.smithi204.stdout:420: finishing write tid 2 to smithi20457400-36 2024-09-16T05:12:00.802 INFO:tasks.rados.rados.2.smithi204.stdout:420: finishing write tid 3 to smithi20457400-36 2024-09-16T05:12:00.803 INFO:tasks.rados.rados.2.smithi204.stdout:420: finishing write tid 4 to smithi20457400-36 2024-09-16T05:12:00.803 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 204 (ObjNum 176 snap 44 seq_num 176) dirty exists 2024-09-16T05:12:00.803 INFO:tasks.rados.rados.2.smithi204.stdout:420: left oid 36 (ObjNum 176 snap 44 seq_num 176) 2024-09-16T05:12:00.810 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:00.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:00.870 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:00.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:00 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:00 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:00 smithi032 ceph-mon[27388]: osdmap e188: 12 total, 12 up, 12 in 2024-09-16T05:12:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3183382711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:00.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1360368828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:12:00.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:00 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:00 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:00 smithi196 ceph-mon[30802]: osdmap e188: 12 total, 12 up, 12 in 2024-09-16T05:12:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3183382711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1360368828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:12:00.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:00.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:01.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:01.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:01.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:01.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:00 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:00 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:00 smithi204 ceph-mon[30933]: osdmap e188: 12 total, 12 up, 12 in 2024-09-16T05:12:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3183382711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1360368828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:12:01.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:01.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:01.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:01.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:01.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:01.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:01.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:01.489 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:12:01.489 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:12:01.489 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:12:01.489 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:12:01.489 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:12:01.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:01.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:01.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:01.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:01.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:01.716 INFO:tasks.rados.rados.2.smithi204.stdout:419: done (2 left) 2024-09-16T05:12:01.716 INFO:tasks.rados.rados.2.smithi204.stdout:420: done (1 left) 2024-09-16T05:12:01.716 INFO:tasks.rados.rados.2.smithi204.stdout:421: done (0 left) 2024-09-16T05:12:01.716 INFO:tasks.rados.rados.2.smithi204.stdout:422: write (excl) oid 30 current snap is 45 2024-09-16T05:12:01.716 INFO:tasks.rados.rados.2.smithi204.stdout:422: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-16T05:12:01.723 INFO:tasks.rados.rados.2.smithi204.stdout:422: writing smithi20457400-30 from 574839 to 1370031 tid 1 2024-09-16T05:12:01.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:01.729 INFO:tasks.rados.rados.2.smithi204.stdout:422: writing smithi20457400-30 from 1907901 to 2359960 tid 2 2024-09-16T05:12:01.730 INFO:tasks.rados.rados.2.smithi204.stdout:423: read oid 38 snap -1 2024-09-16T05:12:01.730 INFO:tasks.rados.rados.2.smithi204.stdout:423: expect deleted 2024-09-16T05:12:01.730 INFO:tasks.rados.rados.2.smithi204.stdout:422: finishing write tid 1 to smithi20457400-30 2024-09-16T05:12:01.730 INFO:tasks.rados.rados.2.smithi204.stdout:424: read oid 1 snap 42 2024-09-16T05:12:01.730 INFO:tasks.rados.rados.2.smithi204.stdout:424: expect deleted 2024-09-16T05:12:01.730 INFO:tasks.rados.rados.2.smithi204.stdout:425: snap_create 2024-09-16T05:12:01.738 INFO:tasks.rados.rados.2.smithi204.stdout:422: finishing write tid 2 to smithi20457400-30 2024-09-16T05:12:01.738 INFO:tasks.rados.rados.2.smithi204.stdout:422: finishing write tid 3 to smithi20457400-30 2024-09-16T05:12:01.738 INFO:tasks.rados.rados.2.smithi204.stdout:422: finishing write tid 4 to smithi20457400-30 2024-09-16T05:12:01.738 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 207 (ObjNum 177 snap 45 seq_num 177) dirty exists 2024-09-16T05:12:01.738 INFO:tasks.rados.rados.2.smithi204.stdout:422: left oid 30 (ObjNum 177 snap 45 seq_num 177) 2024-09-16T05:12:01.748 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:01.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:01.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:01.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:01.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:01.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:01.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:01 smithi196 ceph-mon[30802]: pgmap v435: 37 pgs: 1 active+clean+scrubbing, 5 peering, 4 unknown, 27 active+clean; 150 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:01 smithi196 ceph-mon[30802]: osdmap e189: 12 total, 12 up, 12 in 2024-09-16T05:12:01.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:02.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:02.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:01 smithi032 ceph-mon[27388]: pgmap v435: 37 pgs: 1 active+clean+scrubbing, 5 peering, 4 unknown, 27 active+clean; 150 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:02.042 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:01 smithi032 ceph-mon[27388]: osdmap e189: 12 total, 12 up, 12 in 2024-09-16T05:12:02.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:02.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:02.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:01 smithi204 ceph-mon[30933]: pgmap v435: 37 pgs: 1 active+clean+scrubbing, 5 peering, 4 unknown, 27 active+clean; 150 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:01 smithi204 ceph-mon[30933]: osdmap e189: 12 total, 12 up, 12 in 2024-09-16T05:12:02.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:02.248 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:02.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:02.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:02.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:02.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:02.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:12:02.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:02.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:02.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:02.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:02.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:02.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:02.726 INFO:tasks.rados.rados.2.smithi204.stdout:422: done (3 left) 2024-09-16T05:12:02.727 INFO:tasks.rados.rados.2.smithi204.stdout:423: done (2 left) 2024-09-16T05:12:02.727 INFO:tasks.rados.rados.2.smithi204.stdout:424: done (1 left) 2024-09-16T05:12:02.727 INFO:tasks.rados.rados.2.smithi204.stdout:425: done (0 left) 2024-09-16T05:12:02.727 INFO:tasks.rados.rados.2.smithi204.stdout:426: write (excl) oid 32 current snap is 46 2024-09-16T05:12:02.727 INFO:tasks.rados.rados.2.smithi204.stdout:426: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-16T05:12:02.731 INFO:tasks.rados.rados.2.smithi204.stdout:426: writing smithi20457400-32 from 591646 to 1062087 tid 1 2024-09-16T05:12:02.737 INFO:tasks.rados.rados.2.smithi204.stdout:426: writing smithi20457400-32 from 1566383 to 2162100 tid 2 2024-09-16T05:12:02.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:02.743 INFO:tasks.rados.rados.2.smithi204.stdout:426: writing smithi20457400-32 from 2888822 to 2991646 tid 3 2024-09-16T05:12:02.743 INFO:tasks.rados.rados.2.smithi204.stdout:427: read oid 18 snap -1 2024-09-16T05:12:02.743 INFO:tasks.rados.rados.2.smithi204.stdout:427: expect deleted 2024-09-16T05:12:02.743 INFO:tasks.rados.rados.2.smithi204.stdout:428: read oid 33 snap -1 2024-09-16T05:12:02.743 INFO:tasks.rados.rados.2.smithi204.stdout:428: expect deleted 2024-09-16T05:12:02.743 INFO:tasks.rados.rados.2.smithi204.stdout:429: snap_remove snap 34 2024-09-16T05:12:02.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:02.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:02.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:02.889 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:02.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:02.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:02 smithi196 ceph-mon[30802]: osdmap e190: 12 total, 12 up, 12 in 2024-09-16T05:12:02.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:03.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:03.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:03.024 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:02 smithi032 ceph-mon[27388]: osdmap e190: 12 total, 12 up, 12 in 2024-09-16T05:12:03.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:03.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:03.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:02 smithi204 ceph-mon[30933]: osdmap e190: 12 total, 12 up, 12 in 2024-09-16T05:12:03.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:03.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:03.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:03.318 INFO:tasks.thrashosds.thrasher:j is {'epoch': 188, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:11:59.695412+0000', 'last_up_change': '2024-09-16T05:09:26.475677+0000', 'last_in_change': '2024-09-16T05:09:05.828775+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '104', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.389999866485596, 'score_stable': 4.389999866485596, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 3.640000104904175, 'raw_score_stable': 3.640000104904175, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '183', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 44, 'snap_epoch': 188, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.309999942779541, 'score_stable': 2.309999942779541, 'optimal_score': 1, 'raw_score_acting': 2.309999942779541, 'raw_score_stable': 2.309999942779541, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 187, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3508564664}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3508564664}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3508564664}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3508564664}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3508564664}]}, 'public_addr': '172.21.15.32:6819/3508564664', 'cluster_addr': '172.21.15.32:6821/3508564664', 'heartbeat_back_addr': '172.21.15.32:6825/3508564664', 'heartbeat_front_addr': '172.21.15.32:6823/3508564664', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:59.105660+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:14.594921+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:58.222328+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:27.014040+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:21.139048+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:08.506447+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:10:58.893839+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:29.286641+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:58.589883+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:21.438658+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:27.324418+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:11:40.265420+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 15, 'length': 1}, {'begin': 21, 'length': 1}, {'begin': 25, 'length': 1}, {'begin': 35, 'length': 1}, {'begin': 37, 'length': 2}, {'begin': 40, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 37, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:12:03.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:12:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:03.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:03.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:03.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:03.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:03.521 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:03.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:03.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:03.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:03.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:03.728 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:12:03.735 INFO:tasks.rados.rados.2.smithi204.stdout:429: done (3 left) 2024-09-16T05:12:03.735 INFO:tasks.rados.rados.2.smithi204.stdout:430: rollback oid 24 current snap is 46 2024-09-16T05:12:03.735 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 37 2024-09-16T05:12:03.735 INFO:tasks.rados.rados.2.smithi204.stdout:431: copy_from oid 19 from oid 43 current snap is 46 2024-09-16T05:12:03.735 INFO:tasks.rados.rados.2.smithi204.stdout:426: finishing write tid 1 to smithi20457400-32 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:426: finishing write tid 2 to smithi20457400-32 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:426: finishing write tid 3 to smithi20457400-32 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:426: finishing write tid 4 to smithi20457400-32 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:426: finishing write tid 5 to smithi20457400-32 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 209 (ObjNum 178 snap 46 seq_num 178) dirty exists 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:426: left oid 32 (ObjNum 178 snap 46 seq_num 178) 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:426: done (4 left) 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:427: done (3 left) 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:428: done (2 left) 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:432: rollback oid 37 current snap is 46 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 31 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:433: write oid 20 current snap is 46 2024-09-16T05:12:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:433: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-16T05:12:03.741 INFO:tasks.rados.rados.2.smithi204.stdout:433: writing smithi20457400-20 from 608453 to 1154143 tid 1 2024-09-16T05:12:03.746 INFO:tasks.rados.rados.2.smithi204.stdout:433: writing smithi20457400-20 from 1624865 to 2080593 tid 2 2024-09-16T05:12:03.751 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:433: writing smithi20457400-20 from 2632598 to 3008453 tid 3 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:430: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:432: finishing rollback tid 0 to smithi20457400-37 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:432: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:430: done (3 left) 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:434: write (excl) oid 14 current snap is 46 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:432: finishing rollback tid 2 to smithi20457400-37 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 117 (ObjNum 129 snap 27 seq_num 129) dirty exists 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:433: finishing write tid 1 to smithi20457400-20 2024-09-16T05:12:03.753 INFO:tasks.rados.rados.2.smithi204.stdout:431: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:12:03.754 INFO:tasks.rados.rados.2.smithi204.stdout:434: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-16T05:12:03.759 INFO:tasks.rados.rados.2.smithi204.stdout:434: writing smithi20457400-14 from 625260 to 1246199 tid 1 2024-09-16T05:12:03.764 INFO:tasks.rados.rados.2.smithi204.stdout:434: writing smithi20457400-14 from 1683347 to 2282733 tid 2 2024-09-16T05:12:03.769 INFO:tasks.rados.rados.2.smithi204.stdout:434: writing smithi20457400-14 from 2943668 to 3025260 tid 3 2024-09-16T05:12:03.769 INFO:tasks.rados.rados.2.smithi204.stdout:432: done (3 left) 2024-09-16T05:12:03.769 INFO:tasks.rados.rados.2.smithi204.stdout:435: copy_from oid 22 from oid 5 current snap is 46 2024-09-16T05:12:03.769 INFO:tasks.rados.rados.2.smithi204.stdout:431: finishing copy_from to smithi20457400-19 2024-09-16T05:12:03.769 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 88 (ObjNum 147 snap 34 seq_num 147) dirty exists 2024-09-16T05:12:03.769 INFO:tasks.rados.rados.2.smithi204.stdout:431: done (3 left) 2024-09-16T05:12:03.770 INFO:tasks.rados.rados.2.smithi204.stdout:436: write (excl) oid 4 current snap is 46 2024-09-16T05:12:03.770 INFO:tasks.rados.rados.2.smithi204.stdout:433: finishing write tid 2 to smithi20457400-20 2024-09-16T05:12:03.770 INFO:tasks.rados.rados.2.smithi204.stdout:433: finishing write tid 3 to smithi20457400-20 2024-09-16T05:12:03.770 INFO:tasks.rados.rados.2.smithi204.stdout:436: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-16T05:12:03.776 INFO:tasks.rados.rados.2.smithi204.stdout:436: writing smithi20457400-4 from 642067 to 1338255 tid 1 2024-09-16T05:12:03.777 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:03.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:03.782 INFO:tasks.rados.rados.2.smithi204.stdout:436: writing smithi20457400-4 from 2025476 to 2484873 tid 2 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:436: writing smithi20457400-4 from 2971091 to 3042067 tid 3 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:437: write oid 21 current snap is 46 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:433: finishing write tid 4 to smithi20457400-20 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:433: finishing write tid 5 to smithi20457400-20 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 296 (ObjNum 179 snap 46 seq_num 179) dirty exists 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:433: left oid 20 (ObjNum 179 snap 46 seq_num 179) 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:434: finishing write tid 1 to smithi20457400-14 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:434: finishing write tid 2 to smithi20457400-14 2024-09-16T05:12:03.786 INFO:tasks.rados.rados.2.smithi204.stdout:437: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-16T05:12:03.793 INFO:tasks.rados.rados.2.smithi204.stdout:437: writing smithi20457400-21 from 658874 to 1430311 tid 1 2024-09-16T05:12:03.801 INFO:tasks.rados.rados.2.smithi204.stdout:437: writing smithi20457400-21 from 2083958 to 2687013 tid 2 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:433: done (4 left) 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:438: read oid 19 snap 39 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:438: expect (ObjNum 159 snap 39 seq_num 159) 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:439: delete oid 23 current snap is 46 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:434: finishing write tid 3 to smithi20457400-14 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:434: finishing write tid 4 to smithi20457400-14 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:434: finishing write tid 5 to smithi20457400-14 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 216 (ObjNum 180 snap 46 seq_num 180) dirty exists 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:434: left oid 14 (ObjNum 180 snap 46 seq_num 180) 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:435: finishing copy_from racing read to smithi20457400-22 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:435: finishing copy_from to smithi20457400-22 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 116 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:436: finishing write tid 1 to smithi20457400-4 2024-09-16T05:12:03.802 INFO:tasks.rados.rados.2.smithi204.stdout:436: finishing write tid 2 to smithi20457400-4 2024-09-16T05:12:03.803 INFO:tasks.rados.rados.2.smithi204.stdout:436: finishing write tid 3 to smithi20457400-4 2024-09-16T05:12:03.803 INFO:tasks.rados.rados.2.smithi204.stdout:436: finishing write tid 4 to smithi20457400-4 2024-09-16T05:12:03.803 INFO:tasks.rados.rados.2.smithi204.stdout:436: finishing write tid 5 to smithi20457400-4 2024-09-16T05:12:03.803 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 146 (ObjNum 181 snap 46 seq_num 181) dirty exists 2024-09-16T05:12:03.803 INFO:tasks.rados.rados.2.smithi204.stdout:436: left oid 4 (ObjNum 181 snap 46 seq_num 181) 2024-09-16T05:12:03.803 INFO:tasks.rados.rados.2.smithi204.stdout:437: finishing write tid 1 to smithi20457400-21 2024-09-16T05:12:03.804 INFO:tasks.rados.rados.2.smithi204.stdout:434: done (5 left) 2024-09-16T05:12:03.804 INFO:tasks.rados.rados.2.smithi204.stdout:435: done (4 left) 2024-09-16T05:12:03.804 INFO:tasks.rados.rados.2.smithi204.stdout:436: done (3 left) 2024-09-16T05:12:03.804 INFO:tasks.rados.rados.2.smithi204.stdout:439: done (2 left) 2024-09-16T05:12:03.804 INFO:tasks.rados.rados.2.smithi204.stdout:440: write oid 16 current snap is 46 2024-09-16T05:12:03.804 INFO:tasks.rados.rados.2.smithi204.stdout:440: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-16T05:12:03.809 INFO:tasks.rados.rados.2.smithi204.stdout:440: writing smithi20457400-16 from 692488 to 1330776 tid 1 2024-09-16T05:12:03.816 INFO:tasks.rados.rados.2.smithi204.stdout:440: writing smithi20457400-16 from 1917275 to 2523999 tid 2 2024-09-16T05:12:03.816 INFO:tasks.rados.rados.2.smithi204.stdout:441: write oid 13 current snap is 46 2024-09-16T05:12:03.816 INFO:tasks.rados.rados.2.smithi204.stdout:437: finishing write tid 2 to smithi20457400-21 2024-09-16T05:12:03.817 INFO:tasks.rados.rados.2.smithi204.stdout:437: finishing write tid 3 to smithi20457400-21 2024-09-16T05:12:03.817 INFO:tasks.rados.rados.2.smithi204.stdout:437: finishing write tid 4 to smithi20457400-21 2024-09-16T05:12:03.817 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 116 (ObjNum 182 snap 46 seq_num 182) dirty exists 2024-09-16T05:12:03.817 INFO:tasks.rados.rados.2.smithi204.stdout:437: left oid 21 (ObjNum 182 snap 46 seq_num 182) 2024-09-16T05:12:03.817 INFO:tasks.rados.rados.2.smithi204.stdout:438: expect (ObjNum 159 snap 39 seq_num 159) 2024-09-16T05:12:03.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:03 smithi032 ceph-mon[27388]: pgmap v438: 37 pgs: 3 peering, 34 active+clean; 172 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s, 7 objects/s recovering 2024-09-16T05:12:03.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:03 smithi032 ceph-mon[27388]: osdmap e191: 12 total, 12 up, 12 in 2024-09-16T05:12:03.877 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3242907329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:03.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5 2.1 e112/107 2:8c1ba486:::smithi20457400-2:head v 112'1, mlcod=0'0)", 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.007090+0000", 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 80.868647155000005, 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001761501, 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007090+0000", 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007090+0000", 2024-09-16T05:12:03.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007090+0000", 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3099999999999999e-07 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007391+0000", 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030055400000000002 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007415+0000", 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4556999999999999e-05 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007424+0000", 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2619999999999998e-06 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007469+0000", 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5532999999999999e-05 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.007492+0000", 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2305000000000001e-05 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008819+0000", 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013269239999999999 2024-09-16T05:12:03.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.008852+0000", 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3135e-05 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:115 2.a e112/107 2:580e4d0e:::smithi20457400-29:head v 112'6, mlcod=112'3)", 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 80.541392879, 2024-09-16T05:12:03.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0311933, 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:12:03.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334345+0000", 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334350+0000", 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.887e-06 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365213+0000", 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030863313999999999 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365220+0000", 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7870000000000004e-06 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365291+0000", 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1397999999999996e-05 2024-09-16T05:12:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365307+0000", 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5990000000000001e-05 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365507+0000", 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000199221 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365538+0000", 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1167000000000002e-05 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:125 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'7, mlcod=112'4)", 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 80.511222750000002, 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006511045, 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.906 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365198+0000", 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068299699999999997 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369743+0000", 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045446979999999998 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369750+0000", 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1940000000000001e-06 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369826+0000", 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6160000000000003e-05 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369841+0000", 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4895000000000001e-05 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.370997+0000", 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001155866 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.371026+0000", 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8564999999999998e-05 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:127 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'9, mlcod=112'5)", 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 80.506011841000003, 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020225074999999999, 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369726+0000", 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-09-16T05:12:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369728+0000", 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9410000000000001e-06 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389627+0000", 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019899751 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389632+0000", 2024-09-16T05:12:03.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389704+0000", 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1418e-05 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389718+0000", 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.433e-05 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389922+0000", 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020421600000000001 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389951+0000", 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8140000000000002e-05 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:133 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'10, mlcod=112'7)", 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 80.486477913000002, 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074633729999999997, 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389261+0000", 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.973e-06 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389613+0000", 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035121600000000001 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395726+0000", 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006113851 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395731+0000", 2024-09-16T05:12:03.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0359999999999997e-06 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395819+0000", 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7128999999999999e-05 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395836+0000", 2024-09-16T05:12:03.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7768999999999999e-05 2024-09-16T05:12:03.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396693+0000", 2024-09-16T05:12:03.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085685999999999998 2024-09-16T05:12:03.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396723+0000", 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9539e-05 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:135 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'12, mlcod=112'8)", 2024-09-16T05:12:03.921 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 80.479966938000004, 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069274319999999999, 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395771+0000", 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4499999999999997e-07 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395775+0000", 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6509999999999999e-06 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402319+0000", 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065439469999999996 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402326+0000", 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5769999999999996e-06 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402424+0000", 2024-09-16T05:12:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8171000000000006e-05 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402439+0000", 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4844999999999999e-05 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402660+0000", 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022083700000000001 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402698+0000", 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7858999999999998e-05 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:137 2.a e112/107 2:53edbf55:::smithi20457400-35:head v 112'13, mlcod=112'10)", 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.409611+0000", 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 80.466126611999996, 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001839961, 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.926 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.409611+0000", 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.409611+0000", 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.409611+0000", 2024-09-16T05:12:03.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7700000000000005e-07 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410294+0000", 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068301500000000003 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410336+0000", 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1542999999999998e-05 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410343+0000", 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9029999999999998e-06 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410406+0000", 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3474999999999998e-05 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.410421+0000", 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.451e-05 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411424+0000", 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00100332 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.411451+0000", 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6718000000000001e-05 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:222 2.e e114/107 2:7b642727:::smithi20457400-40:head v 114'14, mlcod=112'11)", 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:46.002342+0000", 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.873394888999997, 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019311160000000001, 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002342+0000", 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002342+0000", 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002343+0000", 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6199999999999999e-07 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002788+0000", 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000445236 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002845+0000", 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7271000000000001e-05 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002857+0000", 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1888999999999999e-05 2024-09-16T05:12:03.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002939+0000", 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1420000000000003e-05 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.002967+0000", 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8272999999999998e-05 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.004225+0000", 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001257932 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:46.004274+0000", 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8733000000000002e-05 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:228 2.a e115/107 2:5089ebe1:::smithi20457400-36:head v 115'20, mlcod=112'17)", 2024-09-16T05:12:03.935 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:47.013127+0000", 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 76.862610704999994, 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020438700000000001, 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013127+0000", 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013127+0000", 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013127+0000", 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9900000000000002e-07 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013827+0000", 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069999700000000005 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013859+0000", 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1606000000000002e-05 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013868+0000", 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013934+0000", 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6019999999999995e-05 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.013953+0000", 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8374999999999999e-05 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.015118+0000", 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011652279999999999 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.015170+0000", 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2571000000000002e-05 2024-09-16T05:12:03.939 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:582 2.a e142/107 2:5089ebe1:::smithi20457400-36:head v 142'70, mlcod=142'67)", 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:13.769619+0000", 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 50.106117849999997, 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017663760000000001, 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.769619+0000", 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.769619+0000", 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.769626+0000", 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2169999999999999e-06 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.770512+0000", 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088603 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.770539+0000", 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7586e-05 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.770545+0000", 2024-09-16T05:12:03.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5330000000000002e-06 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.770602+0000", 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7200999999999997e-05 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.770629+0000", 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6619e-05 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.771339+0000", 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071014500000000003 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.771386+0000", 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7045000000000001e-05 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:773 2.1 e159/107 2:8c1ba486:::smithi20457400-2:head v 159'34, mlcod=155'30)", 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:12:03.944 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 33.082562875000001, 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060955719999999996, 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793175+0000", 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795522+0000", 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023476920000000002 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795574+0000", 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1459999999999999e-05 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795584+0000", 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0274e-05 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795649+0000", 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4901999999999996e-05 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795679+0000", 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9887e-05 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.799228+0000", 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035493679999999998 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.799270+0000", 2024-09-16T05:12:03.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1813000000000001e-05 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:891 2.a e171/107 2:5089ebe1:::smithi20457400-36:head v 171'128, mlcod=171'124)", 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.215984584000001, 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003806771, 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1699999999999998e-07 2024-09-16T05:12:03.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662145+0000", 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023911589999999999 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662170+0000", 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100999999999999e-05 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662177+0000", 2024-09-16T05:12:03.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5120000000000002e-06 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662246+0000", 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9264000000000006e-05 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662262+0000", 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5128e-05 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.663526+0000", 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001264311 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.663559+0000", 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3679000000000001e-05 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.953 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:898 2.a e173/107 2:5827f14d:::smithi20457400-12:head v 173'138, mlcod=173'134)", 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:44.673351+0000", 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 19.202386624999999, 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00193917, 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.673351+0000", 2024-09-16T05:12:03.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.673351+0000", 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.673352+0000", 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0359999999999999e-06 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.674052+0000", 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070030699999999995 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.674091+0000", 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9011000000000003e-05 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.674099+0000", 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0709999999999995e-06 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.674155+0000", 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6137999999999997e-05 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.674173+0000", 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7535999999999999e-05 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.675241+0000", 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010679260000000001 2024-09-16T05:12:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:44.675290+0000", 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9144999999999998e-05 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:974 2.a e179/107 2:5827f14d:::smithi20457400-12:head v 179'156, mlcod=178'152)", 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 12.868305253000001, 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068838909999999996, 2024-09-16T05:12:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007433+0000", 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.421e-06 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009914+0000", 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024808030000000002 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009966+0000", 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1456999999999999e-05 2024-09-16T05:12:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009975+0000", 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4010000000000003e-06 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010042+0000", 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010064+0000", 2024-09-16T05:12:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2282000000000001e-05 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014259+0000", 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041942020000000002 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014316+0000", 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7377999999999999e-05 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1117 2.a e189/107 2:5089ebe1:::smithi20457400-36:head v 189'202, mlcod=188'198)", 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:00.720288+0000", 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.155449242, 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001990206, 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.720288+0000", 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.720288+0000", 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.720288+0000", 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8300000000000001e-07 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.721111+0000", 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00082249299999999999 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.721132+0000", 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0979e-05 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.721139+0000", 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1600000000000001e-06 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.721173+0000", 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4474000000000002e-05 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.721189+0000", 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5526e-05 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.722242+0000", 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010533820000000001 2024-09-16T05:12:03.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:00.722278+0000", 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6009e-05 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1133 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'206, mlcod=189'203)", 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.141510644, 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021225035, 2024-09-16T05:12:03.967 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.85e-07 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734829+0000", 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060196800000000003 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734866+0000", 2024-09-16T05:12:03.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7469000000000003e-05 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734881+0000", 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5058999999999999e-05 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734970+0000", 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9004000000000004e-05 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734998+0000", 2024-09-16T05:12:03.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7606e-05 2024-09-16T05:12:03.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755384+0000", 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02038587 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755452+0000", 2024-09-16T05:12:03.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7873999999999994e-05 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1134 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'207, mlcod=189'203)", 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.1367255219999999, 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016477325000000001, 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.973 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739014+0000", 2024-09-16T05:12:03.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2819999999999999e-06 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739797+0000", 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078265599999999996 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739844+0000", 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7608000000000002e-05 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739854+0000", 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4239999999999999e-06 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739935+0000", 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1593000000000003e-05 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739963+0000", 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7492e-05 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755459+0000", 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015496565 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755489+0000", 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9705000000000001e-05 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.977 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1135 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'208, mlcod=189'203)", 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.132302685, 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012094011, 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743436+0000", 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.584e-06 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743535+0000", 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9110000000000005e-05 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743568+0000", 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2335000000000001e-05 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743576+0000", 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.191e-06 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743646+0000", 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0481000000000005e-05 2024-09-16T05:12:03.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743676+0000", 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9620000000000001e-05 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755492+0000", 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011815888 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755529+0000", 2024-09-16T05:12:03.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6801999999999998e-05 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1136 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'209, mlcod=189'203)", 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.131695127, 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011564019, 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744043+0000", 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3600000000000003e-07 2024-09-16T05:12:03.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744049+0000", 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8250000000000001e-06 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744069+0000", 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0460000000000001e-05 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744074+0000", 2024-09-16T05:12:03.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4730000000000002e-06 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744119+0000", 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744143+0000", 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4085999999999999e-05 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755570+0000", 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011427299 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755606+0000", 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146000000000003e-05 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1163 2.e e192/107 2:7e4e65a4:::smithi20457400-4:head v 192'139, mlcod=188'135)", 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.779192+0000", 2024-09-16T05:12:03.986 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.096545493999999996, 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001922024, 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.779192+0000", 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.779192+0000", 2024-09-16T05:12:03.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.779192+0000", 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9000000000000001e-07 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.779547+0000", 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035509499999999999 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:03.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.779624+0000", 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6835999999999994e-05 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.779630+0000", 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0449999999999997e-06 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.779682+0000", 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2444000000000001e-05 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.779703+0000", 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0615000000000002e-05 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.781076+0000", 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013729390000000001 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.781114+0000", 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.786e-05 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.990 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:03.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:03.991 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:12:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:03.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:03.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:03.992 INFO:tasks.rados.rados.2.smithi204.stdout:440: finishing write tid 1 to smithi20457400-16 2024-09-16T05:12:03.992 INFO:tasks.rados.rados.2.smithi204.stdout:440: finishing write tid 2 to smithi20457400-16 2024-09-16T05:12:03.993 INFO:tasks.rados.rados.2.smithi204.stdout:440: finishing write tid 3 to smithi20457400-16 2024-09-16T05:12:03.993 INFO:tasks.rados.rados.2.smithi204.stdout:440: finishing write tid 4 to smithi20457400-16 2024-09-16T05:12:03.993 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 267 (ObjNum 184 snap 46 seq_num 184) dirty exists 2024-09-16T05:12:03.993 INFO:tasks.rados.rados.2.smithi204.stdout:440: left oid 16 (ObjNum 184 snap 46 seq_num 184) 2024-09-16T05:12:03.993 INFO:tasks.rados.rados.2.smithi204.stdout:441: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-16T05:12:03.993 INFO:tasks.rados.rados.2.smithi204.stdout:441: writing smithi20457400-13 from 709295 to 1422832 tid 1 2024-09-16T05:12:03.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:03 smithi196 ceph-mon[30802]: pgmap v438: 37 pgs: 3 peering, 34 active+clean; 172 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s, 7 objects/s recovering 2024-09-16T05:12:03.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:03 smithi196 ceph-mon[30802]: osdmap e191: 12 total, 12 up, 12 in 2024-09-16T05:12:03.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3242907329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:03.995 INFO:tasks.rados.rados.2.smithi204.stdout:441: writing smithi20457400-13 from 1859404 to 2609786 tid 2 2024-09-16T05:12:03.996 INFO:tasks.rados.rados.2.smithi204.stdout:437: done (3 left) 2024-09-16T05:12:03.996 INFO:tasks.rados.rados.2.smithi204.stdout:438: done (2 left) 2024-09-16T05:12:03.996 INFO:tasks.rados.rados.2.smithi204.stdout:440: done (1 left) 2024-09-16T05:12:03.996 INFO:tasks.rados.rados.2.smithi204.stdout:442: read oid 27 snap -1 2024-09-16T05:12:03.996 INFO:tasks.rados.rados.2.smithi204.stdout:442: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-16T05:12:03.996 INFO:tasks.rados.rados.2.smithi204.stdout:443: snap_create 2024-09-16T05:12:03.996 INFO:tasks.rados.rados.2.smithi204.stdout:441: finishing write tid 1 to smithi20457400-13 2024-09-16T05:12:03.997 INFO:tasks.rados.rados.2.smithi204.stdout:441: finishing write tid 2 to smithi20457400-13 2024-09-16T05:12:04.004 INFO:tasks.rados.rados.2.smithi204.stdout:442: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-16T05:12:04.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:12:04.091 INFO:tasks.rados.rados.2.smithi204.stdout:441: finishing write tid 3 to smithi20457400-13 2024-09-16T05:12:04.091 INFO:tasks.rados.rados.2.smithi204.stdout:441: finishing write tid 4 to smithi20457400-13 2024-09-16T05:12:04.091 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 271 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:12:04.091 INFO:tasks.rados.rados.2.smithi204.stdout:441: left oid 13 (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:12:04.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:04.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:04.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:04.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:03 smithi204 ceph-mon[30933]: pgmap v438: 37 pgs: 3 peering, 34 active+clean; 172 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s, 7 objects/s recovering 2024-09-16T05:12:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:03 smithi204 ceph-mon[30933]: osdmap e191: 12 total, 12 up, 12 in 2024-09-16T05:12:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3242907329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:04.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:04.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:04.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:04.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:04.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:04.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:04.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:04.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:04.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:04.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:04.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:04.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:04.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:12:04.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:04.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:04.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:04.746 INFO:tasks.rados.rados.2.smithi204.stdout:441: done (2 left) 2024-09-16T05:12:04.746 INFO:tasks.rados.rados.2.smithi204.stdout:442: done (1 left) 2024-09-16T05:12:04.746 INFO:tasks.rados.rados.2.smithi204.stdout:443: done (0 left) 2024-09-16T05:12:04.746 INFO:tasks.rados.rados.2.smithi204.stdout:444: write (excl) oid 43 current snap is 47 2024-09-16T05:12:04.746 INFO:tasks.rados.rados.2.smithi204.stdout:444: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-16T05:12:04.754 INFO:tasks.rados.rados.2.smithi204.stdout:444: writing smithi20457400-43 from 726102 to 1514888 tid 1 2024-09-16T05:12:04.759 INFO:tasks.rados.rados.2.smithi204.stdout:444: writing smithi20457400-43 from 1917886 to 2528279 tid 2 2024-09-16T05:12:04.763 INFO:tasks.rados.rados.2.smithi204.stdout:444: writing smithi20457400-43 from 3108206 to 3126102 tid 3 2024-09-16T05:12:04.763 INFO:tasks.rados.rados.2.smithi204.stdout:445: read oid 7 snap -1 2024-09-16T05:12:04.763 INFO:tasks.rados.rados.2.smithi204.stdout:445: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:12:04.763 INFO:tasks.rados.rados.2.smithi204.stdout:446: snap_create 2024-09-16T05:12:04.763 INFO:tasks.rados.rados.2.smithi204.stdout:444: finishing write tid 1 to smithi20457400-43 2024-09-16T05:12:04.764 INFO:tasks.rados.rados.2.smithi204.stdout:444: finishing write tid 2 to smithi20457400-43 2024-09-16T05:12:04.764 INFO:tasks.rados.rados.2.smithi204.stdout:444: finishing write tid 3 to smithi20457400-43 2024-09-16T05:12:04.765 INFO:tasks.rados.rados.2.smithi204.stdout:444: finishing write tid 4 to smithi20457400-43 2024-09-16T05:12:04.765 INFO:tasks.rados.rados.2.smithi204.stdout:444: finishing write tid 5 to smithi20457400-43 2024-09-16T05:12:04.765 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 134 (ObjNum 186 snap 47 seq_num 186) dirty exists 2024-09-16T05:12:04.765 INFO:tasks.rados.rados.2.smithi204.stdout:444: left oid 43 (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:12:04.768 INFO:tasks.rados.rados.2.smithi204.stdout:445: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:12:04.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:04.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:04.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:04.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:04.911 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:04 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3242907329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:04 smithi032 ceph-mon[27388]: osdmap e192: 12 total, 12 up, 12 in 2024-09-16T05:12:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:04 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:04 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:04.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:04.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:04.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:04.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:04 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:04.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3242907329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:04 smithi196 ceph-mon[30802]: osdmap e192: 12 total, 12 up, 12 in 2024-09-16T05:12:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:04 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:04 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:05.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:04 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:05.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3242907329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:05.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:04 smithi204 ceph-mon[30933]: osdmap e192: 12 total, 12 up, 12 in 2024-09-16T05:12:05.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:04 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:05.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:04 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:05.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:05.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:05.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:05.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:05.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:05.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:05.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:05.361 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:05.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:05.408 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:05.461 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:12:05.462 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:12:05.462 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:12:05.462 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:12:05.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:05.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:05.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:05.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:05.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:05.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:05.656 INFO:tasks.rados.rados.2.smithi204.stdout:444: done (2 left) 2024-09-16T05:12:05.656 INFO:tasks.rados.rados.2.smithi204.stdout:445: done (1 left) 2024-09-16T05:12:05.656 INFO:tasks.rados.rados.2.smithi204.stdout:446: done (0 left) 2024-09-16T05:12:05.656 INFO:tasks.rados.rados.2.smithi204.stdout:447: write (excl) oid 14 current snap is 48 2024-09-16T05:12:05.656 INFO:tasks.rados.rados.2.smithi204.stdout:447: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-16T05:12:05.663 INFO:tasks.rados.rados.2.smithi204.stdout:447: writing smithi20457400-14 from 742909 to 1206944 tid 1 2024-09-16T05:12:05.674 INFO:tasks.rados.rados.2.smithi204.stdout:447: writing smithi20457400-14 from 1976368 to 2730419 tid 2 2024-09-16T05:12:05.674 INFO:tasks.rados.rados.2.smithi204.stdout:448: copy_from oid 11 from oid 44 current snap is 48 2024-09-16T05:12:05.674 INFO:tasks.rados.rados.2.smithi204.stdout:447: finishing write tid 1 to smithi20457400-14 2024-09-16T05:12:05.674 INFO:tasks.rados.rados.2.smithi204.stdout:449: write oid 50 current snap is 48 2024-09-16T05:12:05.674 INFO:tasks.rados.rados.2.smithi204.stdout:449: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-16T05:12:05.681 INFO:tasks.rados.rados.2.smithi204.stdout:449: writing smithi20457400-50 from 759716 to 1299000 tid 1 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:449: writing smithi20457400-50 from 2034850 to 2648912 tid 2 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:450: delete oid 38 current snap is 48 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:447: finishing write tid 2 to smithi20457400-14 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:447: finishing write tid 3 to smithi20457400-14 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:447: finishing write tid 4 to smithi20457400-14 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 220 (ObjNum 187 snap 48 seq_num 187) dirty exists 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:447: left oid 14 (ObjNum 187 snap 48 seq_num 187) 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:448: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:449: finishing write tid 1 to smithi20457400-50 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:448: finishing copy_from to smithi20457400-11 2024-09-16T05:12:05.691 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 40 (ObjNum 111 snap 19 seq_num 111) dirty exists 2024-09-16T05:12:05.692 INFO:tasks.rados.rados.2.smithi204.stdout:447: done (3 left) 2024-09-16T05:12:05.692 INFO:tasks.rados.rados.2.smithi204.stdout:448: done (2 left) 2024-09-16T05:12:05.692 INFO:tasks.rados.rados.2.smithi204.stdout:450: done (1 left) 2024-09-16T05:12:05.693 INFO:tasks.rados.rados.2.smithi204.stdout:451: read oid 25 snap 44 2024-09-16T05:12:05.693 INFO:tasks.rados.rados.2.smithi204.stdout:451: expect (ObjNum 156 snap 39 seq_num 156) 2024-09-16T05:12:05.693 INFO:tasks.rados.rados.2.smithi204.stdout:452: read oid 17 snap -1 2024-09-16T05:12:05.693 INFO:tasks.rados.rados.2.smithi204.stdout:452: expect deleted 2024-09-16T05:12:05.693 INFO:tasks.rados.rados.2.smithi204.stdout:453: read oid 43 snap -1 2024-09-16T05:12:05.693 INFO:tasks.rados.rados.2.smithi204.stdout:453: expect (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:12:05.693 INFO:tasks.rados.rados.2.smithi204.stdout:454: delete oid 19 current snap is 48 2024-09-16T05:12:05.693 INFO:tasks.rados.rados.2.smithi204.stdout:449: finishing write tid 2 to smithi20457400-50 2024-09-16T05:12:05.694 INFO:tasks.rados.rados.2.smithi204.stdout:449: finishing write tid 3 to smithi20457400-50 2024-09-16T05:12:05.694 INFO:tasks.rados.rados.2.smithi204.stdout:449: finishing write tid 4 to smithi20457400-50 2024-09-16T05:12:05.694 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 119 (ObjNum 188 snap 48 seq_num 188) dirty exists 2024-09-16T05:12:05.694 INFO:tasks.rados.rados.2.smithi204.stdout:449: left oid 50 (ObjNum 188 snap 48 seq_num 188) 2024-09-16T05:12:05.694 INFO:tasks.rados.rados.2.smithi204.stdout:449: done (4 left) 2024-09-16T05:12:05.695 INFO:tasks.rados.rados.2.smithi204.stdout:454: done (3 left) 2024-09-16T05:12:05.695 INFO:tasks.rados.rados.2.smithi204.stdout:455: delete oid 2 current snap is 48 2024-09-16T05:12:05.696 INFO:tasks.rados.rados.2.smithi204.stdout:452: done (3 left) 2024-09-16T05:12:05.696 INFO:tasks.rados.rados.2.smithi204.stdout:455: done (2 left) 2024-09-16T05:12:05.696 INFO:tasks.rados.rados.2.smithi204.stdout:456: read oid 13 snap -1 2024-09-16T05:12:05.696 INFO:tasks.rados.rados.2.smithi204.stdout:456: expect (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:12:05.696 INFO:tasks.rados.rados.2.smithi204.stdout:457: delete oid 27 current snap is 48 2024-09-16T05:12:05.698 INFO:tasks.rados.rados.2.smithi204.stdout:457: done (3 left) 2024-09-16T05:12:05.699 INFO:tasks.rados.rados.2.smithi204.stdout:458: write (excl) oid 33 current snap is 48 2024-09-16T05:12:05.699 INFO:tasks.rados.rados.2.smithi204.stdout:458: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-16T05:12:05.705 INFO:tasks.rados.rados.2.smithi204.stdout:458: writing smithi20457400-33 from 443751 to 1075633 tid 1 2024-09-16T05:12:05.713 INFO:tasks.rados.rados.2.smithi204.stdout:458: writing smithi20457400-33 from 1810907 to 2575965 tid 2 2024-09-16T05:12:05.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:05.717 INFO:tasks.rados.rados.2.smithi204.stdout:458: writing smithi20457400-33 from 3067461 to 3243751 tid 3 2024-09-16T05:12:05.717 INFO:tasks.rados.rados.2.smithi204.stdout:459: read oid 1 snap -1 2024-09-16T05:12:05.717 INFO:tasks.rados.rados.2.smithi204.stdout:459: expect deleted 2024-09-16T05:12:05.717 INFO:tasks.rados.rados.2.smithi204.stdout:460: delete oid 23 current snap is 48 2024-09-16T05:12:05.717 INFO:tasks.rados.rados.2.smithi204.stdout:453: expect (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:12:05.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:05.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:05.833 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:05 smithi032 ceph-mon[27388]: pgmap v441: 37 pgs: 1 peering, 36 active+clean; 175 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.2 MiB/s wr, 27 op/s; 5.0 MiB/s, 9 objects/s recovering 2024-09-16T05:12:05.833 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:05 smithi032 ceph-mon[27388]: osdmap e193: 12 total, 12 up, 12 in 2024-09-16T05:12:05.833 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:05 smithi032 ceph-mon[27388]: osdmap e194: 12 total, 12 up, 12 in 2024-09-16T05:12:05.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:05.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:05.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:05.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:05.864 INFO:tasks.rados.rados.2.smithi204.stdout:451: expect (ObjNum 156 snap 39 seq_num 156) 2024-09-16T05:12:05.904 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:05.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:05 smithi204 ceph-mon[30933]: pgmap v441: 37 pgs: 1 peering, 36 active+clean; 175 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.2 MiB/s wr, 27 op/s; 5.0 MiB/s, 9 objects/s recovering 2024-09-16T05:12:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:05 smithi204 ceph-mon[30933]: osdmap e193: 12 total, 12 up, 12 in 2024-09-16T05:12:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:05 smithi204 ceph-mon[30933]: osdmap e194: 12 total, 12 up, 12 in 2024-09-16T05:12:05.954 INFO:tasks.rados.rados.2.smithi204.stdout:456: expect (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:12:05.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:05.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:05.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:05.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:05 smithi196 ceph-mon[30802]: pgmap v441: 37 pgs: 1 peering, 36 active+clean; 175 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.2 MiB/s wr, 27 op/s; 5.0 MiB/s, 9 objects/s recovering 2024-09-16T05:12:05.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:05 smithi196 ceph-mon[30802]: osdmap e193: 12 total, 12 up, 12 in 2024-09-16T05:12:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:05 smithi196 ceph-mon[30802]: osdmap e194: 12 total, 12 up, 12 in 2024-09-16T05:12:06.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:06.109 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:06.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:06.114 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:12:06.114 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:12:06.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:12:06.188 INFO:tasks.rados.rados.2.smithi204.stdout:458: finishing write tid 1 to smithi20457400-33 2024-09-16T05:12:06.189 INFO:tasks.rados.rados.2.smithi204.stdout:458: finishing write tid 2 to smithi20457400-33 2024-09-16T05:12:06.189 INFO:tasks.rados.rados.2.smithi204.stdout:451: done (5 left) 2024-09-16T05:12:06.189 INFO:tasks.rados.rados.2.smithi204.stdout:453: done (4 left) 2024-09-16T05:12:06.189 INFO:tasks.rados.rados.2.smithi204.stdout:456: done (3 left) 2024-09-16T05:12:06.189 INFO:tasks.rados.rados.2.smithi204.stdout:460: done (2 left) 2024-09-16T05:12:06.189 INFO:tasks.rados.rados.2.smithi204.stdout:461: snap_remove snap 42 2024-09-16T05:12:06.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:06.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:06.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:06.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:06.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:06.401 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:06.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:06.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:06.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:06.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:06.567 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:06.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:06.593 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:06.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:06.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:06.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:06.614 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:06.634 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:06.634 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:06.656 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:06.656 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:06.677 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:06.677 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:06.699 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:06.699 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:06.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:06.720 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:06.720 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:06.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:06.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:06.743 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:06.743 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:06.747 INFO:tasks.rados.rados.2.smithi204.stdout:461: done (2 left) 2024-09-16T05:12:06.747 INFO:tasks.rados.rados.2.smithi204.stdout:462: read oid 38 snap -1 2024-09-16T05:12:06.747 INFO:tasks.rados.rados.2.smithi204.stdout:462: expect deleted 2024-09-16T05:12:06.747 INFO:tasks.rados.rados.2.smithi204.stdout:463: write (excl) oid 6 current snap is 48 2024-09-16T05:12:06.747 INFO:tasks.rados.rados.2.smithi204.stdout:463: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-16T05:12:06.755 INFO:tasks.rados.rados.2.smithi204.stdout:463: writing smithi20457400-6 from 477365 to 1259745 tid 1 2024-09-16T05:12:06.762 INFO:tasks.rados.rados.2.smithi204.stdout:463: writing smithi20457400-6 from 1927871 to 2696598 tid 2 2024-09-16T05:12:06.766 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:06.766 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:463: writing smithi20457400-6 from 3122307 to 3277365 tid 3 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:464: read oid 21 snap -1 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:464: expect (ObjNum 182 snap 46 seq_num 182) 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:465: snap_create 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:458: finishing write tid 3 to smithi20457400-33 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:458: finishing write tid 4 to smithi20457400-33 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:458: finishing write tid 5 to smithi20457400-33 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 224 (ObjNum 193 snap 48 seq_num 193) dirty exists 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:458: left oid 33 (ObjNum 193 snap 48 seq_num 193) 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:463: finishing write tid 1 to smithi20457400-6 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:463: finishing write tid 2 to smithi20457400-6 2024-09-16T05:12:06.768 INFO:tasks.rados.rados.2.smithi204.stdout:463: finishing write tid 3 to smithi20457400-6 2024-09-16T05:12:06.769 INFO:tasks.rados.rados.2.smithi204.stdout:463: finishing write tid 4 to smithi20457400-6 2024-09-16T05:12:06.769 INFO:tasks.rados.rados.2.smithi204.stdout:463: finishing write tid 5 to smithi20457400-6 2024-09-16T05:12:06.769 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 26 (ObjNum 195 snap 48 seq_num 195) dirty exists 2024-09-16T05:12:06.769 INFO:tasks.rados.rados.2.smithi204.stdout:463: left oid 6 (ObjNum 195 snap 48 seq_num 195) 2024-09-16T05:12:06.771 INFO:tasks.rados.rados.2.smithi204.stdout:464: expect (ObjNum 182 snap 46 seq_num 182) 2024-09-16T05:12:06.789 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:06.798 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1085852731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:06.810 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:06.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:06.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:06.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:06.906 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:06.906 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:06.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:06.980 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:06.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:06.999 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1085852731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:07.026 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-16T05:12:06.051973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":193912316,"num_objects":100,"num_object_clones":53,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":8,"num_read":721,"num_read_kb":587064,"num_write":636,"num_write_kb":222458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":20774053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3731,"ondisk_log_size":3731,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":607084,"kb_used_data":249428,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124334740,"statfs":{"total":1151940427776,"available":1151318773760,"internally_reserved":0,"allocated":255414272,"data_stored":301515950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14003765,"num_objects":10,"num_object_clones":9,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":-7,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":26,"num_read_kb":15306,"num_write":39,"num_write_kb":10272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1237143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000708"},"pg_stats":[{"pgid":"2.19","version":"182'60","reported_seq":29,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-16T05:12:02.735186+0000","last_change":"2024-09-16T05:11:59.699876+0000","last_active":"2024-09-16T05:12:02.735186+0000","last_peered":"2024-09-16T05:12:02.735186+0000","last_clean":"2024-09-16T05:12:02.735186+0000","last_became_active":"2024-09-16T05:11:57.658646+0000","last_became_peered":"2024-09-16T05:11:57.658646+0000","last_unstale":"2024-09-16T05:12:02.735186+0000","last_undegraded":"2024-09-16T05:12:02.735186+0000","last_fullsized":"2024-09-16T05:12:02.735186+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"174'50","last_scrub_stamp":"2024-09-16T05:11:49.167247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:49.167247+0000","objects_scrubbed":5,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00046797499999999998,"stat_sum":{"num_bytes":5823061,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":32171,"num_write":19,"num_write_kb":5280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":432,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:04.764589+0000","last_change":"2024-09-16T05:12:03.734185+0000","last_active":"2024-09-16T05:12:04.764589+0000","last_peered":"2024-09-16T05:12:04.764589+0000","last_clean":"2024-09-16T05:12:04.764589+0000","last_became_active":"2024-09-16T05:11:57.660350+0000","last_became_peered":"2024-09-16T05:11:57.660350+0000","last_unstale":"2024-09-16T05:12:04.764589+0000","last_undegraded":"2024-09-16T05:12:04.764589+0000","last_fullsized":"2024-09-16T05:12:04.764589+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:11:52.869870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:52.869870+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014946829999999999,"stat_sum":{"num_bytes":3534561,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":3642,"num_write":11,"num_write_kb":3683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.17","version":"0'0","reported_seq":208,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.856598+0000","last_change":"2024-09-16T05:11:59.707103+0000","last_active":"2024-09-16T05:12:01.856598+0000","last_peered":"2024-09-16T05:12:01.856598+0000","last_clean":"2024-09-16T05:12:01.856598+0000","last_became_active":"2024-09-16T05:11:57.657217+0000","last_became_peered":"2024-09-16T05:11:57.657217+0000","last_unstale":"2024-09-16T05:12:01.856598+0000","last_undegraded":"2024-09-16T05:12:01.856598+0000","last_fullsized":"2024-09-16T05:12:01.856598+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:11:46.760515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.760515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000674261,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]},{"pgid":"2.16","version":"116'9","reported_seq":48,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.134967+0000","last_change":"2024-09-16T05:12:03.730684+0000","last_active":"2024-09-16T05:12:05.134967+0000","last_peered":"2024-09-16T05:12:05.134967+0000","last_clean":"2024-09-16T05:12:05.134967+0000","last_became_active":"2024-09-16T05:11:57.656772+0000","last_became_peered":"2024-09-16T05:11:57.656772+0000","last_unstale":"2024-09-16T05:12:05.134967+0000","last_undegraded":"2024-09-16T05:12:05.134967+0000","last_fullsized":"2024-09-16T05:12:05.134967+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:11:50.181661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:50.181661+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00038304399999999997,"stat_sum":{"num_bytes":1428579,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6574,"num_write":4,"num_write_kb":1221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2024259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.15","version":"192'271","reported_seq":678,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:04.764755+0000","last_change":"2024-09-16T05:12:03.740877+0000","last_active":"2024-09-16T05:12:04.764755+0000","last_peered":"2024-09-16T05:12:04.764755+0000","last_clean":"2024-09-16T05:12:04.764755+0000","last_became_active":"2024-09-16T05:12:00.711890+0000","last_became_peered":"2024-09-16T05:12:00.711890+0000","last_unstale":"2024-09-16T05:12:04.764755+0000","last_undegraded":"2024-09-16T05:12:04.764755+0000","last_fullsized":"2024-09-16T05:12:04.764755+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067421529999999999,"stat_sum":{"num_bytes":18038835,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":66,"num_read_kb":56757,"num_write":62,"num_write_kb":20933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":7576804,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.14","version":"192'101","reported_seq":95,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:04.766737+0000","last_change":"2024-09-16T05:12:03.741367+0000","last_active":"2024-09-16T05:12:04.766737+0000","last_peered":"2024-09-16T05:12:04.766737+0000","last_clean":"2024-09-16T05:12:04.766737+0000","last_became_active":"2024-09-16T05:11:57.658852+0000","last_became_peered":"2024-09-16T05:11:57.658852+0000","last_unstale":"2024-09-16T05:12:04.766737+0000","last_undegraded":"2024-09-16T05:12:04.766737+0000","last_fullsized":"2024-09-16T05:12:04.766737+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073907820000000003,"stat_sum":{"num_bytes":5595123,"num_objects":2,"num_object_clones":2,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":31397,"num_write":25,"num_write_kb":10274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6142869,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.13","version":"181'15","reported_seq":39,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:04.764618+0000","last_change":"2024-09-16T05:12:03.734385+0000","last_active":"2024-09-16T05:12:04.764618+0000","last_peered":"2024-09-16T05:12:04.764618+0000","last_clean":"2024-09-16T05:12:04.764618+0000","last_became_active":"2024-09-16T05:11:57.660401+0000","last_became_peered":"2024-09-16T05:11:57.660401+0000","last_unstale":"2024-09-16T05:12:04.764618+0000","last_undegraded":"2024-09-16T05:12:04.764618+0000","last_fullsized":"2024-09-16T05:12:04.764618+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:11:54.342283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:54.342283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012049999999999999,"stat_sum":{"num_bytes":1142860,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.12","version":"192'28","reported_seq":55,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.068271+0000","last_change":"2024-09-16T05:12:03.744771+0000","last_active":"2024-09-16T05:12:05.068271+0000","last_peered":"2024-09-16T05:12:05.068271+0000","last_clean":"2024-09-16T05:12:05.068271+0000","last_became_active":"2024-09-16T05:11:57.658975+0000","last_became_peered":"2024-09-16T05:11:57.658975+0000","last_unstale":"2024-09-16T05:12:05.068271+0000","last_undegraded":"2024-09-16T05:12:05.068271+0000","last_fullsized":"2024-09-16T05:12:05.068271+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026626380000000002,"stat_sum":{"num_bytes":2447109,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6199,"num_write":7,"num_write_kb":1554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3792978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":581,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:04.764730+0000","last_change":"2024-09-16T05:10:44.010384+0000","last_active":"2024-09-16T05:12:04.764730+0000","last_peered":"2024-09-16T05:12:04.764730+0000","last_clean":"2024-09-16T05:12:04.764730+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:12:04.764730+0000","last_undegraded":"2024-09-16T05:12:04.764730+0000","last_fullsized":"2024-09-16T05:12:04.764730+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.010311+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.010311+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"192'82","reported_seq":520,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-16T05:12:03.739490+0000","last_change":"2024-09-16T05:12:03.739885+0000","last_active":"2024-09-16T05:12:03.739490+0000","last_peered":"2024-09-16T05:12:03.739490+0000","last_clean":"2024-09-16T05:12:03.739490+0000","last_became_active":"2024-09-16T05:11:55.657137+0000","last_became_peered":"2024-09-16T05:11:55.657137+0000","last_unstale":"2024-09-16T05:12:03.739490+0000","last_undegraded":"2024-09-16T05:12:03.739490+0000","last_fullsized":"2024-09-16T05:12:03.739490+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'70","last_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.116608+0000","objects_scrubbed":7,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:12.690183+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.0030566149999999999,"stat_sum":{"num_bytes":14232953,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":577,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.714690+0000","last_change":"2024-09-16T05:10:43.759254+0000","last_active":"2024-09-16T05:12:01.714690+0000","last_peered":"2024-09-16T05:12:01.714690+0000","last_clean":"2024-09-16T05:12:01.714690+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:12:01.714690+0000","last_undegraded":"2024-09-16T05:12:01.714690+0000","last_fullsized":"2024-09-16T05:12:01.714690+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:43.759130+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:43.759130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:01.857687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"189'204","reported_seq":614,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.856595+0000","last_change":"2024-09-16T05:11:59.707279+0000","last_active":"2024-09-16T05:12:01.856595+0000","last_peered":"2024-09-16T05:12:01.856595+0000","last_clean":"2024-09-16T05:12:01.856595+0000","last_became_active":"2024-09-16T05:10:37.002200+0000","last_became_peered":"2024-09-16T05:10:37.002200+0000","last_unstale":"2024-09-16T05:12:01.856595+0000","last_undegraded":"2024-09-16T05:12:01.856595+0000","last_fullsized":"2024-09-16T05:12:01.856595+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'144","last_scrub_stamp":"2024-09-16T05:11:46.247408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.247408+0000","objects_scrubbed":15,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:57.222144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0074781750000000001,"stat_sum":{"num_bytes":32166456,"num_objects":14,"num_object_clones":8,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":108,"num_read_kb":100267,"num_write":96,"num_write_kb":36203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":578,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.134873+0000","last_change":"2024-09-16T05:10:45.437354+0000","last_active":"2024-09-16T05:12:05.134873+0000","last_peered":"2024-09-16T05:12:05.134873+0000","last_clean":"2024-09-16T05:12:05.134873+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:12:05.134873+0000","last_undegraded":"2024-09-16T05:12:05.134873+0000","last_fullsized":"2024-09-16T05:12:05.134873+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.437261+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.437261+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"192'129","reported_seq":544,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-16T05:12:03.769818+0000","last_change":"2024-09-16T05:12:03.732965+0000","last_active":"2024-09-16T05:12:03.769818+0000","last_peered":"2024-09-16T05:12:03.769818+0000","last_clean":"2024-09-16T05:12:03.769818+0000","last_became_active":"2024-09-16T05:10:36.956652+0000","last_became_peered":"2024-09-16T05:10:36.956652+0000","last_unstale":"2024-09-16T05:12:03.769818+0000","last_undegraded":"2024-09-16T05:12:03.769818+0000","last_fullsized":"2024-09-16T05:12:03.769818+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'106","last_scrub_stamp":"2024-09-16T05:11:51.494420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:51.494420+0000","objects_scrubbed":11,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:54.551252+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026423520000000002,"stat_sum":{"num_bytes":13994173,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":58473,"num_write":53,"num_write_kb":26467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":502,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.068251+0000","last_change":"2024-09-16T05:11:27.759078+0000","last_active":"2024-09-16T05:12:05.068251+0000","last_peered":"2024-09-16T05:12:05.068251+0000","last_clean":"2024-09-16T05:12:05.068251+0000","last_became_active":"2024-09-16T05:11:21.676107+0000","last_became_peered":"2024-09-16T05:11:21.676107+0000","last_unstale":"2024-09-16T05:12:05.068251+0000","last_undegraded":"2024-09-16T05:12:05.068251+0000","last_fullsized":"2024-09-16T05:12:05.068251+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:27.759006+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:27.759006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:36.703164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":493,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-16T05:12:03.736862+0000","last_change":"2024-09-16T05:12:03.737137+0000","last_active":"2024-09-16T05:12:03.736862+0000","last_peered":"2024-09-16T05:12:03.736862+0000","last_clean":"2024-09-16T05:12:03.736862+0000","last_became_active":"2024-09-16T05:11:55.665450+0000","last_became_peered":"2024-09-16T05:11:55.665450+0000","last_unstale":"2024-09-16T05:12:03.736862+0000","last_undegraded":"2024-09-16T05:12:03.736862+0000","last_fullsized":"2024-09-16T05:12:03.736862+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045412499999999999,"stat_sum":{"num_bytes":7294172,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":9357,"num_write":20,"num_write_kb":8930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":582,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.068226+0000","last_change":"2024-09-16T05:10:45.767820+0000","last_active":"2024-09-16T05:12:05.068226+0000","last_peered":"2024-09-16T05:12:05.068226+0000","last_clean":"2024-09-16T05:12:05.068226+0000","last_became_active":"2024-09-16T05:10:38.972286+0000","last_became_peered":"2024-09-16T05:10:38.972286+0000","last_unstale":"2024-09-16T05:12:05.068226+0000","last_undegraded":"2024-09-16T05:12:05.068226+0000","last_fullsized":"2024-09-16T05:12:05.068226+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.767717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.767717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:08.779630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"192'296","reported_seq":711,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-16T05:12:03.754132+0000","last_change":"2024-09-16T05:12:03.744669+0000","last_active":"2024-09-16T05:12:03.754132+0000","last_peered":"2024-09-16T05:12:03.754132+0000","last_clean":"2024-09-16T05:12:03.754132+0000","last_became_active":"2024-09-16T05:11:55.663098+0000","last_became_peered":"2024-09-16T05:11:55.663098+0000","last_unstale":"2024-09-16T05:12:03.754132+0000","last_undegraded":"2024-09-16T05:12:03.754132+0000","last_fullsized":"2024-09-16T05:12:03.754132+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'266","last_scrub_stamp":"2024-09-16T05:11:56.487686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.487686+0000","objects_scrubbed":7,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:02.453456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.014236212,"stat_sum":{"num_bytes":23540752,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":21,"num_read_kb":23922,"num_write":12,"num_write_kb":6095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":582,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.068190+0000","last_change":"2024-09-16T05:10:44.788217+0000","last_active":"2024-09-16T05:12:05.068190+0000","last_peered":"2024-09-16T05:12:05.068190+0000","last_clean":"2024-09-16T05:12:05.068190+0000","last_became_active":"2024-09-16T05:10:38.972150+0000","last_became_peered":"2024-09-16T05:10:38.972150+0000","last_unstale":"2024-09-16T05:12:05.068190+0000","last_undegraded":"2024-09-16T05:12:05.068190+0000","last_fullsized":"2024-09-16T05:12:05.068190+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.788114+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.788114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":419,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-16T05:12:03.736797+0000","last_change":"2024-09-16T05:12:03.737083+0000","last_active":"2024-09-16T05:12:03.736797+0000","last_peered":"2024-09-16T05:12:03.736797+0000","last_clean":"2024-09-16T05:12:03.736797+0000","last_became_active":"2024-09-16T05:11:55.657738+0000","last_became_peered":"2024-09-16T05:11:55.657738+0000","last_unstale":"2024-09-16T05:12:03.736797+0000","last_undegraded":"2024-09-16T05:12:03.736797+0000","last_fullsized":"2024-09-16T05:12:03.736797+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:11:50.181661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:50.181661+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046387099999999998,"stat_sum":{"num_bytes":1428578,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6573,"num_write":4,"num_write_kb":1221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":535,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.856773+0000","last_change":"2024-09-16T05:10:41.421609+0000","last_active":"2024-09-16T05:12:01.856773+0000","last_peered":"2024-09-16T05:12:01.856773+0000","last_clean":"2024-09-16T05:12:01.856773+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:12:01.856773+0000","last_undegraded":"2024-09-16T05:12:01.856773+0000","last_fullsized":"2024-09-16T05:12:01.856773+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:41.421501+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:41.421501+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:05.337075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":235,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.068387+0000","last_change":"2024-09-16T05:12:03.743336+0000","last_active":"2024-09-16T05:12:05.068387+0000","last_peered":"2024-09-16T05:12:05.068387+0000","last_clean":"2024-09-16T05:12:05.068387+0000","last_became_active":"2024-09-16T05:11:55.659156+0000","last_became_peered":"2024-09-16T05:11:55.659156+0000","last_unstale":"2024-09-16T05:12:05.068387+0000","last_undegraded":"2024-09-16T05:12:05.068387+0000","last_fullsized":"2024-09-16T05:12:05.068387+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:11:56.378387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.378387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:18.560154+0000","scrub_duration":686,"objects_trimmed":0,"snaptrim_duration":0.00083998799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.0","version":"188'20","reported_seq":425,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.714705+0000","last_change":"2024-09-16T05:11:59.706648+0000","last_active":"2024-09-16T05:12:01.714705+0000","last_peered":"2024-09-16T05:12:01.714705+0000","last_clean":"2024-09-16T05:12:01.714705+0000","last_became_active":"2024-09-16T05:11:55.656232+0000","last_became_peered":"2024-09-16T05:11:55.656232+0000","last_unstale":"2024-09-16T05:12:01.714705+0000","last_undegraded":"2024-09-16T05:12:01.714705+0000","last_fullsized":"2024-09-16T05:12:01.714705+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'18","last_scrub_stamp":"2024-09-16T05:11:55.862199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:55.862199+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:10.759217+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002279845,"stat_sum":{"num_bytes":3142876,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":588,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-16T05:12:02.735106+0000","last_change":"2024-09-16T05:11:53.500880+0000","last_active":"2024-09-16T05:12:02.735106+0000","last_peered":"2024-09-16T05:12:02.735106+0000","last_clean":"2024-09-16T05:12:02.735106+0000","last_became_active":"2024-09-16T05:11:19.419149+0000","last_became_peered":"2024-09-16T05:11:19.419149+0000","last_unstale":"2024-09-16T05:12:02.735106+0000","last_undegraded":"2024-09-16T05:12:02.735106+0000","last_fullsized":"2024-09-16T05:12:02.735106+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:53.500734+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:53.500734+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:08.214137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":454,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-16T05:12:00.960305+0000","last_change":"2024-09-16T05:11:59.707085+0000","last_active":"2024-09-16T05:12:00.960305+0000","last_peered":"2024-09-16T05:12:00.960305+0000","last_clean":"2024-09-16T05:12:00.960305+0000","last_became_active":"2024-09-16T05:11:55.656596+0000","last_became_peered":"2024-09-16T05:11:55.656596+0000","last_unstale":"2024-09-16T05:12:00.960305+0000","last_undegraded":"2024-09-16T05:12:00.960305+0000","last_fullsized":"2024-09-16T05:12:00.960305+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:11:56.000487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.000487+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:02.575949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073958100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]},{"pgid":"1.2","version":"35'139","reported_seq":577,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.714658+0000","last_change":"2024-09-16T05:10:42.783065+0000","last_active":"2024-09-16T05:12:01.714658+0000","last_peered":"2024-09-16T05:12:01.714658+0000","last_clean":"2024-09-16T05:12:01.714658+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:12:01.714658+0000","last_undegraded":"2024-09-16T05:12:01.714658+0000","last_fullsized":"2024-09-16T05:12:01.714658+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:42.782976+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:42.782976+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:01.857667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":589,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.068188+0000","last_change":"2024-09-16T05:11:51.723968+0000","last_active":"2024-09-16T05:12:05.068188+0000","last_peered":"2024-09-16T05:12:05.068188+0000","last_clean":"2024-09-16T05:12:05.068188+0000","last_became_active":"2024-09-16T05:10:38.972381+0000","last_became_peered":"2024-09-16T05:10:38.972381+0000","last_unstale":"2024-09-16T05:12:05.068188+0000","last_undegraded":"2024-09-16T05:12:05.068188+0000","last_fullsized":"2024-09-16T05:12:05.068188+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:51.723892+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:51.723892+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:12.858953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":429,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.135015+0000","last_change":"2024-09-16T05:12:03.730591+0000","last_active":"2024-09-16T05:12:05.135015+0000","last_peered":"2024-09-16T05:12:05.135015+0000","last_clean":"2024-09-16T05:12:05.135015+0000","last_became_active":"2024-09-16T05:11:55.657334+0000","last_became_peered":"2024-09-16T05:11:55.657334+0000","last_unstale":"2024-09-16T05:12:05.135015+0000","last_undegraded":"2024-09-16T05:12:05.135015+0000","last_fullsized":"2024-09-16T05:12:05.135015+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:11:56.310771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.310771+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:21.279858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032890000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":582,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.714811+0000","last_change":"2024-09-16T05:11:54.869494+0000","last_active":"2024-09-16T05:12:01.714811+0000","last_peered":"2024-09-16T05:12:01.714811+0000","last_clean":"2024-09-16T05:12:01.714811+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:12:01.714811+0000","last_undegraded":"2024-09-16T05:12:01.714811+0000","last_fullsized":"2024-09-16T05:12:01.714811+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:54.869407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:54.869407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:21.879716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"184'17","reported_seq":422,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:05.135078+0000","last_change":"2024-09-16T05:12:03.730854+0000","last_active":"2024-09-16T05:12:05.135078+0000","last_peered":"2024-09-16T05:12:05.135078+0000","last_clean":"2024-09-16T05:12:05.135078+0000","last_became_active":"2024-09-16T05:11:55.661748+0000","last_became_peered":"2024-09-16T05:11:55.661748+0000","last_unstale":"2024-09-16T05:12:05.135078+0000","last_undegraded":"2024-09-16T05:12:05.135078+0000","last_fullsized":"2024-09-16T05:12:05.135078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:11:54.342283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:54.342283+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000563639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.8","version":"184'61","reported_seq":456,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.714722+0000","last_change":"2024-09-16T05:11:59.704904+0000","last_active":"2024-09-16T05:12:01.714722+0000","last_peered":"2024-09-16T05:12:01.714722+0000","last_clean":"2024-09-16T05:12:01.714722+0000","last_became_active":"2024-09-16T05:11:55.658383+0000","last_became_peered":"2024-09-16T05:11:55.658383+0000","last_unstale":"2024-09-16T05:12:01.714722+0000","last_undegraded":"2024-09-16T05:12:01.714722+0000","last_fullsized":"2024-09-16T05:12:01.714722+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:11:52.869870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:52.869870+0000","objects_scrubbed":4,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052807600000000004,"stat_sum":{"num_bytes":1289015,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3642,"num_write":10,"num_write_kb":3682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]},{"pgid":"2.c","version":"188'142","reported_seq":529,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.856870+0000","last_change":"2024-09-16T05:11:59.706124+0000","last_active":"2024-09-16T05:12:01.856870+0000","last_peered":"2024-09-16T05:12:01.856870+0000","last_clean":"2024-09-16T05:12:01.856870+0000","last_became_active":"2024-09-16T05:10:36.957767+0000","last_became_peered":"2024-09-16T05:10:36.957767+0000","last_unstale":"2024-09-16T05:12:01.856870+0000","last_undegraded":"2024-09-16T05:12:01.856870+0000","last_fullsized":"2024-09-16T05:12:01.856870+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'99","last_scrub_stamp":"2024-09-16T05:11:50.223175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:50.223175+0000","objects_scrubbed":7,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:01.532860+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0062479170000000004,"stat_sum":{"num_bytes":12350528,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":40,"num_read_kb":23363,"num_write":59,"num_write_kb":16010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]},{"pgid":"2.d","version":"192'216","reported_seq":631,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-16T05:12:04.764930+0000","last_change":"2024-09-16T05:12:03.744991+0000","last_active":"2024-09-16T05:12:04.764930+0000","last_peered":"2024-09-16T05:12:04.764930+0000","last_clean":"2024-09-16T05:12:04.764930+0000","last_became_active":"2024-09-16T05:10:36.957439+0000","last_became_peered":"2024-09-16T05:10:36.957439+0000","last_unstale":"2024-09-16T05:12:04.764930+0000","last_undegraded":"2024-09-16T05:12:04.764930+0000","last_fullsized":"2024-09-16T05:12:04.764930+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'165","last_scrub_stamp":"2024-09-16T05:11:45.953349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:45.953349+0000","objects_scrubbed":11,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:48.939252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010560786000000001,"stat_sum":{"num_bytes":18968056,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":110,"num_read_kb":78953,"num_write":107,"num_write_kb":34870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.e","version":"188'137","reported_seq":531,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-16T05:12:00.960285+0000","last_change":"2024-09-16T05:11:59.710377+0000","last_active":"2024-09-16T05:12:00.960285+0000","last_peered":"2024-09-16T05:12:00.960285+0000","last_clean":"2024-09-16T05:12:00.960285+0000","last_became_active":"2024-09-16T05:10:37.001925+0000","last_became_peered":"2024-09-16T05:10:37.001925+0000","last_unstale":"2024-09-16T05:12:00.960285+0000","last_undegraded":"2024-09-16T05:12:00.960285+0000","last_fullsized":"2024-09-16T05:12:00.960285+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:01.453903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037410989999999999,"stat_sum":{"num_bytes":10385925,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":79,"num_read_kb":73322,"num_write":67,"num_write_kb":18670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]},{"pgid":"2.f","version":"192'117","reported_seq":544,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-16T05:12:03.742199+0000","last_change":"2024-09-16T05:12:03.738139+0000","last_active":"2024-09-16T05:12:03.742199+0000","last_peered":"2024-09-16T05:12:03.742199+0000","last_clean":"2024-09-16T05:12:03.742199+0000","last_became_active":"2024-09-16T05:10:36.956705+0000","last_became_peered":"2024-09-16T05:10:36.956705+0000","last_unstale":"2024-09-16T05:12:03.742199+0000","last_undegraded":"2024-09-16T05:12:03.742199+0000","last_fullsized":"2024-09-16T05:12:03.742199+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'74","last_scrub_stamp":"2024-09-16T05:11:46.374423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.374423+0000","objects_scrubbed":6,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:56.586504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0077688150000000001,"stat_sum":{"num_bytes":14946898,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":76,"num_read_kb":64654,"num_write":56,"num_write_kb":17176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":36,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-16T05:12:03.730530+0000","last_change":"2024-09-16T05:12:03.730854+0000","last_active":"2024-09-16T05:12:03.730530+0000","last_peered":"2024-09-16T05:12:03.730530+0000","last_clean":"2024-09-16T05:12:03.730530+0000","last_became_active":"2024-09-16T05:11:57.658263+0000","last_became_peered":"2024-09-16T05:11:57.658263+0000","last_unstale":"2024-09-16T05:12:03.730530+0000","last_undegraded":"2024-09-16T05:12:03.730530+0000","last_fullsized":"2024-09-16T05:12:03.730530+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"121'10","last_scrub_stamp":"2024-09-16T05:11:46.786746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.786746+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051952300000000003,"stat_sum":{"num_bytes":1571438,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":2045,"num_write":5,"num_write_kb":913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"4"},{"start":"28","length":"1"}]},{"pgid":"2.11","version":"188'56","reported_seq":53,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-16T05:12:01.856622+0000","last_change":"2024-09-16T05:11:59.709430+0000","last_active":"2024-09-16T05:12:01.856622+0000","last_peered":"2024-09-16T05:12:01.856622+0000","last_clean":"2024-09-16T05:12:01.856622+0000","last_became_active":"2024-09-16T05:11:57.662582+0000","last_became_peered":"2024-09-16T05:11:57.662582+0000","last_unstale":"2024-09-16T05:12:01.856622+0000","last_undegraded":"2024-09-16T05:12:01.856622+0000","last_fullsized":"2024-09-16T05:12:01.856622+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:11:50.024084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:50.024084+0000","objects_scrubbed":2,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002908445,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":5751,"num_write":10,"num_write_kb":7081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1237143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"1"},{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"23","length":"1"},{"start":"25","length":"2"},{"start":"28","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":193321948,"num_objects":98,"num_object_clones":53,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":8,"num_read":721,"num_read_kb":587064,"num_write":636,"num_write_kb":222458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":20774053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233205760,"data_stored":293301399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2202,"ondisk_log_size":2202,"up":52,"acting":52,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089379,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63404,"kb_used_data":36372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93681748,"statfs":{"total":95995035648,"available":95930109952,"internally_reserved":0,"allocated":37244928,"data_stored":50433479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285609,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38480,"kb_used_data":7288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93706672,"statfs":{"total":95995035648,"available":95955632128,"internally_reserved":0,"allocated":7462912,"data_stored":6755078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514543,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74228,"kb_used_data":43040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93670924,"statfs":{"total":95995035648,"available":95919026176,"internally_reserved":0,"allocated":44072960,"data_stored":48484858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841585,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59640,"kb_used_data":32608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93685512,"statfs":{"total":95995035648,"available":95933964288,"internally_reserved":0,"allocated":33390592,"data_stored":41258446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103222,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46316,"kb_used_data":15124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93698836,"statfs":{"total":95995035648,"available":95947608064,"internally_reserved":0,"allocated":15486976,"data_stored":18256305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397565,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42536,"kb_used_data":11340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93702616,"statfs":{"total":95995035648,"available":95951478784,"internally_reserved":0,"allocated":11612160,"data_stored":11906063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31400,"kb_used_data":4368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713752,"statfs":{"total":95995035648,"available":95962882048,"internally_reserved":0,"allocated":4472832,"data_stored":3524022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41712,"kb_used_data":14684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93703440,"statfs":{"total":95995035648,"available":95952322560,"internally_reserved":0,"allocated":15036416,"data_stored":16172529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117067,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54692,"kb_used_data":27652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93690460,"statfs":{"total":95995035648,"available":95939031040,"internally_reserved":0,"allocated":28315648,"data_stored":35253862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38560,"kb_used_data":7368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93706592,"statfs":{"total":95995035648,"available":95955550208,"internally_reserved":0,"allocated":7544832,"data_stored":9113102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51908,"kb_used_data":20724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93693244,"statfs":{"total":95995035648,"available":95941881856,"internally_reserved":0,"allocated":21221376,"data_stored":25268240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64208,"kb_used_data":28860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93680944,"statfs":{"total":95995035648,"available":95929286656,"internally_reserved":0,"allocated":29552640,"data_stored":35089966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27320320,"data_stored":33791451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19001344,"data_stored":23978207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5156864,"data_stored":7647205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27299840,"data_stored":35787690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15609856,"data_stored":18616020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2035712,"data_stored":2024259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9428992,"data_stored":10641414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13316096,"data_stored":17000112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32243712,"data_stored":42339433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":41840640,"data_stored":47186343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5230592,"data_stored":5456563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":34721792,"data_stored":48832702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:12:07.029 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [9, 11] 2024-09-16T05:12:07.029 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.5', '9', '11'] 2024-09-16T05:12:07.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 2.5 9 11 2024-09-16T05:12:07.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:07.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:07.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1085852731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:07.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:07.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:07.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:07.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:07.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:12:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:07.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:07.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:07.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:07.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:07.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:07.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:07.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:12:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:07.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:07.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:07.607 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:07.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:07.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:07.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:07.749 INFO:tasks.rados.rados.2.smithi204.stdout:458: done (5 left) 2024-09-16T05:12:07.749 INFO:tasks.rados.rados.2.smithi204.stdout:459: done (4 left) 2024-09-16T05:12:07.749 INFO:tasks.rados.rados.2.smithi204.stdout:462: done (3 left) 2024-09-16T05:12:07.749 INFO:tasks.rados.rados.2.smithi204.stdout:463: done (2 left) 2024-09-16T05:12:07.749 INFO:tasks.rados.rados.2.smithi204.stdout:464: done (1 left) 2024-09-16T05:12:07.749 INFO:tasks.rados.rados.2.smithi204.stdout:465: done (0 left) 2024-09-16T05:12:07.749 INFO:tasks.rados.rados.2.smithi204.stdout:466: snap_remove snap 40 2024-09-16T05:12:07.824 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:07.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:07.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:07.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:07.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:07 smithi196 ceph-mon[30802]: pgmap v444: 37 pgs: 37 active+clean; 185 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 16 op/s; 302 KiB/s, 1 objects/s recovering 2024-09-16T05:12:07.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:07 smithi196 ceph-mon[30802]: from='client.24754 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:07.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:07 smithi196 ceph-mon[30802]: osdmap e195: 12 total, 12 up, 12 in 2024-09-16T05:12:07.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:07.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:07.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:07.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:07 smithi032 ceph-mon[27388]: pgmap v444: 37 pgs: 37 active+clean; 185 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 16 op/s; 302 KiB/s, 1 objects/s recovering 2024-09-16T05:12:07.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:07 smithi032 ceph-mon[27388]: from='client.24754 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:07.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:07 smithi032 ceph-mon[27388]: osdmap e195: 12 total, 12 up, 12 in 2024-09-16T05:12:08.061 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:08.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:08.100 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:08.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:07 smithi204 ceph-mon[30933]: pgmap v444: 37 pgs: 37 active+clean; 185 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 16 op/s; 302 KiB/s, 1 objects/s recovering 2024-09-16T05:12:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:07 smithi204 ceph-mon[30933]: from='client.24754 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:07 smithi204 ceph-mon[30933]: osdmap e195: 12 total, 12 up, 12 in 2024-09-16T05:12:08.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:08.222 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:08.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:08.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:08.335 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:08.335 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:08.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:08.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:08.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:08.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:08.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:08.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:08.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:08.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:08.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:08.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:08.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:08.755 INFO:teuthology.orchestra.run.smithi032.stderr:set 2.5 pg_upmap mapping to [9,11] 2024-09-16T05:12:08.755 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:12:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:466: done (0 left) 2024-09-16T05:12:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:467: write (excl) oid 10 current snap is 49 2024-09-16T05:12:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:467: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-16T05:12:08.765 INFO:tasks.rados.rados.2.smithi204.stdout:467: writing smithi20457400-10 from 494172 to 951801 tid 1 2024-09-16T05:12:08.770 INFO:tasks.rados.rados.2.smithi204.stdout:467: writing smithi20457400-10 from 1586353 to 2098738 tid 2 2024-09-16T05:12:08.777 INFO:tasks.rados.rados.2.smithi204.stdout:467: writing smithi20457400-10 from 2749730 to 3294172 tid 3 2024-09-16T05:12:08.777 INFO:tasks.rados.rados.2.smithi204.stdout:468: copy_from oid 32 from oid 13 current snap is 49 2024-09-16T05:12:08.777 INFO:tasks.rados.rados.2.smithi204.stdout:467: finishing write tid 1 to smithi20457400-10 2024-09-16T05:12:08.777 INFO:tasks.rados.rados.2.smithi204.stdout:467: finishing write tid 2 to smithi20457400-10 2024-09-16T05:12:08.777 INFO:tasks.rados.rados.2.smithi204.stdout:469: snap_create 2024-09-16T05:12:08.780 INFO:tasks.rados.rados.2.smithi204.stdout:467: finishing write tid 3 to smithi20457400-10 2024-09-16T05:12:08.781 INFO:tasks.rados.rados.2.smithi204.stdout:467: finishing write tid 4 to smithi20457400-10 2024-09-16T05:12:08.781 INFO:tasks.rados.rados.2.smithi204.stdout:467: finishing write tid 5 to smithi20457400-10 2024-09-16T05:12:08.781 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 91 (ObjNum 196 snap 49 seq_num 196) dirty exists 2024-09-16T05:12:08.781 INFO:tasks.rados.rados.2.smithi204.stdout:467: left oid 10 (ObjNum 196 snap 49 seq_num 196) 2024-09-16T05:12:08.784 INFO:tasks.rados.rados.2.smithi204.stdout:468: finishing copy_from racing read to smithi20457400-32 2024-09-16T05:12:08.787 INFO:tasks.rados.rados.2.smithi204.stdout:468: finishing copy_from to smithi20457400-32 2024-09-16T05:12:08.787 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 271 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:12:08.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:08.845 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:08.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:08.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:08.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:08.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:08 smithi196 ceph-mon[30802]: osdmap e196: 12 total, 12 up, 12 in 2024-09-16T05:12:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1565282760' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/287395901' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.5", "id": [9, 11]}]: dispatch 2024-09-16T05:12:09.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:09.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:12:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:09.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:09.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:08 smithi032 ceph-mon[27388]: osdmap e196: 12 total, 12 up, 12 in 2024-09-16T05:12:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1565282760' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/287395901' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.5", "id": [9, 11]}]: dispatch 2024-09-16T05:12:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:08 smithi204 ceph-mon[30933]: osdmap e196: 12 total, 12 up, 12 in 2024-09-16T05:12:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1565282760' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/287395901' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.5", "id": [9, 11]}]: dispatch 2024-09-16T05:12:09.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:09.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:09.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:09.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:09.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:09.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:09.357 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:09.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:09.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:09.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:09.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:12:09.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:09.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:09.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:09.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:09.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:09.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:09.761 INFO:tasks.rados.rados.2.smithi204.stdout:467: done (2 left) 2024-09-16T05:12:09.761 INFO:tasks.rados.rados.2.smithi204.stdout:468: done (1 left) 2024-09-16T05:12:09.761 INFO:tasks.rados.rados.2.smithi204.stdout:469: done (0 left) 2024-09-16T05:12:09.761 INFO:tasks.rados.rados.2.smithi204.stdout:470: rollback oid 34 current snap is 50 2024-09-16T05:12:09.761 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 39 2024-09-16T05:12:09.761 INFO:tasks.rados.rados.2.smithi204.stdout:471: delete oid 44 current snap is 50 2024-09-16T05:12:09.763 INFO:tasks.rados.rados.2.smithi204.stdout:470: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:12:09.763 INFO:tasks.rados.rados.2.smithi204.stdout:470: finishing rollback tid 2 to smithi20457400-34 2024-09-16T05:12:09.763 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 273 (ObjNum 106 snap 18 seq_num 106) dirty exists 2024-09-16T05:12:09.763 INFO:tasks.rados.rados.2.smithi204.stdout:470: done (1 left) 2024-09-16T05:12:09.764 INFO:tasks.rados.rados.2.smithi204.stdout:471: done (0 left) 2024-09-16T05:12:09.764 INFO:tasks.rados.rados.2.smithi204.stdout:472: snap_create 2024-09-16T05:12:09.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:09.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:09.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:09.863 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:09.863 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:09.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: pgmap v447: 37 pgs: 37 active+clean; 194 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.9 MiB/s wr, 22 op/s 2024-09-16T05:12:09.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:09.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1565282760' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:09.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/287395901' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.5", "id": [9, 11]}]': finished 2024-09-16T05:12:09.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: osdmap e197: 12 total, 12 up, 12 in 2024-09-16T05:12:09.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:12:09.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:12:09.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:12:09.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:12:09.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:12:09.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:09 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:12:09.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:09.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:09.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:10.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:10.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:10.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: pgmap v447: 37 pgs: 37 active+clean; 194 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.9 MiB/s wr, 22 op/s 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1565282760' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/287395901' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.5", "id": [9, 11]}]': finished 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: osdmap e197: 12 total, 12 up, 12 in 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:12:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:09 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:12:10.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:10.231 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:10.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:10.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: pgmap v447: 37 pgs: 37 active+clean; 194 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.9 MiB/s wr, 22 op/s 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1565282760' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/287395901' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.5", "id": [9, 11]}]': finished 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: osdmap e197: 12 total, 12 up, 12 in 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:12:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:09 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:12:10.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:10.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:10.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:10.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:10.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:10.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:10.512 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:10.513 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:10.543 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:10.543 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:10.564 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:10.584 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:10.585 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:10.586 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:10.605 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:10.606 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:10.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:12:10.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:10.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:10.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:10.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:10.627 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:10.648 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:10.648 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:10.656 INFO:tasks.rados.rados.2.smithi204.stdout:472: done (0 left) 2024-09-16T05:12:10.656 INFO:tasks.rados.rados.2.smithi204.stdout:473: snap_create 2024-09-16T05:12:10.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:10.739 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:10.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:10.767 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:10.767 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:10.788 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:10.789 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:10.810 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:10.811 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:10.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:10.832 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:10.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:10.853 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:10.853 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:10.855 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:10.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:10.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:10.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:10.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:12:11.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: osdmap e198: 12 total, 12 up, 12 in 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2397699507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:11.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:10 smithi196 ceph-mon[30802]: osdmap e199: 12 total, 12 up, 12 in 2024-09-16T05:12:11.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:11.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:11.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:12:11.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:12:11.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:12:11.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:12:11.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:12:11.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:12:11.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:12:11.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:12:11.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:12:11.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:12:11.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:12:11.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:12:11.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: osdmap e198: 12 total, 12 up, 12 in 2024-09-16T05:12:11.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:12:11.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:12:11.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:12:11.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:12:11.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:12:11.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:12:11.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2397699507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:11.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:10 smithi032 ceph-mon[27388]: osdmap e199: 12 total, 12 up, 12 in 2024-09-16T05:12:11.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:12:11.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:12:11.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:12:11.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:12:11.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:12:11.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:12:11.177 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:12:11.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:12:11.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:12:11.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:12:11.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:12:11.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:12:11.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: osdmap e198: 12 total, 12 up, 12 in 2024-09-16T05:12:11.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:12:11.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:12:11.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:12:11.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:12:11.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:12:11.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:12:11.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2397699507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:11.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:10 smithi204 ceph-mon[30933]: osdmap e199: 12 total, 12 up, 12 in 2024-09-16T05:12:11.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:11.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:11.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:11.276 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:12:11.276 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:12:11.276 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:12:11.276 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:12:11.276 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:12:11.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:11.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:11.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:11.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:11.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:11.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:11.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:11.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:12:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:11.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:11.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:11.657 INFO:tasks.rados.rados.2.smithi204.stdout:473: done (0 left) 2024-09-16T05:12:11.657 INFO:tasks.rados.rados.2.smithi204.stdout:474: snap_remove snap 44 2024-09-16T05:12:11.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:11.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:11.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:11.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:11.854 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:11.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:11.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:11.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:11 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:12:11.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:11 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:12:11.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:11 smithi196 ceph-mon[30802]: pgmap v450: 37 pgs: 37 active+clean; 196 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-16T05:12:11.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:11 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:12:11.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:11 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:12:11.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:11 smithi196 ceph-mon[30802]: osdmap e200: 12 total, 12 up, 12 in 2024-09-16T05:12:11.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:12:11.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:11.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:11.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:12.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:12.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:11 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:12:12.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:11 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:12:12.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:11 smithi032 ceph-mon[27388]: pgmap v450: 37 pgs: 37 active+clean; 196 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-16T05:12:12.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:11 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:12:12.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:11 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:12:12.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:11 smithi032 ceph-mon[27388]: osdmap e200: 12 total, 12 up, 12 in 2024-09-16T05:12:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:12.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:12.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:11 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:12:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:11 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:12:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:11 smithi204 ceph-mon[30933]: pgmap v450: 37 pgs: 37 active+clean; 196 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-16T05:12:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:11 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:12:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:11 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:12:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:11 smithi204 ceph-mon[30933]: osdmap e200: 12 total, 12 up, 12 in 2024-09-16T05:12:12.190 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:12.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:12.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:12.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:12.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:12.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:12.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:12.467 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:12.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:12.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:12.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:12.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:12.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:12.655 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:12:12.660 INFO:tasks.rados.rados.2.smithi204.stdout:474: done (0 left) 2024-09-16T05:12:12.660 INFO:tasks.rados.rados.2.smithi204.stdout:475: snap_create 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:57 2.2 2:4e7d172c:::smithi20457400-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 89.517636734000007, 2024-09-16T05:12:12.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045779080000000003, 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 57 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159039+0000", 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6640000000000001e-06 2024-09-16T05:12:12.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159883+0000", 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084362100000000002 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159895+0000", 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2632999999999999e-05 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159903+0000", 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8900000000000007e-06 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159978+0000", 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5143999999999997e-05 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.683 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159988+0000", 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.431e-06 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161080+0000", 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010917990000000001 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161168+0000", 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8140000000000007e-05 2024-09-16T05:12:12.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161283+0000", 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115513 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163508+0000", 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022243050000000002 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163545+0000", 2024-09-16T05:12:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7033000000000003e-05 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163578+0000", 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2804e-05 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163615+0000", 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7931e-05 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.686 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:58 2.2 2:4e7d172c:::smithi20457400-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 89.516754075999998, 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003854761, 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58 2024-09-16T05:12:12.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159921+0000", 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2699999999999999e-07 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160047+0000", 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012650399999999999 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160055+0000", 2024-09-16T05:12:12.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6539999999999999e-06 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160059+0000", 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5059999999999997e-06 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160100+0000", 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0153e-05 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160104+0000", 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0600000000000001e-06 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162622+0000", 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025179540000000002 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162679+0000", 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7163999999999997e-05 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162793+0000", 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114398 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163515+0000", 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072178499999999996 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163692+0000", 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017674699999999999 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163727+0000", 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5337999999999999e-05 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163775+0000", 2024-09-16T05:12:12.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7970999999999999e-05 2024-09-16T05:12:12.694 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.694 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.694 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.694 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:59 2.2 2:4e7d172c:::smithi20457400-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:12.695 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:12:12.695 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 89.516605182999996, 2024-09-16T05:12:12.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038110150000000001, 2024-09-16T05:12:12.695 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.695 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 59 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160070+0000", 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4999999999999998e-07 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160072+0000", 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.88e-06 2024-09-16T05:12:12.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160078+0000", 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.152e-06 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160081+0000", 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.422e-06 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160135+0000", 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3808999999999999e-05 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160138+0000", 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5590000000000001e-06 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163139+0000", 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030010689999999999 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163192+0000", 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2398000000000002e-05 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163275+0000", 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2825999999999996e-05 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163518+0000", 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024361200000000001 2024-09-16T05:12:12.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163815+0000", 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029653299999999997 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163845+0000", 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0409e-05 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163880+0000", 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4895999999999999e-05 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:60 2.2 2:4e7d172c:::smithi20457400-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 89.516575313000004, 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039136570000000001, 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.702 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 60 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6100000000000001e-07 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160103+0000", 2024-09-16T05:12:12.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.162e-06 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160108+0000", 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3279999999999996e-06 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160112+0000", 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3479999999999997e-06 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160157+0000", 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5141e-05 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160161+0000", 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8589999999999998e-06 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163451+0000", 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032897220000000001 2024-09-16T05:12:12.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163500+0000", 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8665000000000002e-05 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163911+0000", 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041112700000000001 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163953+0000", 2024-09-16T05:12:12.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2228000000000003e-05 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.164013+0000", 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9616e-05 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:69 2.3 2:c018a787:::smithi20457400-18:head [write 685719~711939,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 89.486884727000003, 2024-09-16T05:12:12.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002893503, 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 69 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.709 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189792+0000", 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1299999999999999e-06 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190294+0000", 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050236900000000001 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190307+0000", 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3315e-05 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190329+0000", 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2116999999999998e-05 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190394+0000", 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4177999999999994e-05 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190467+0000", 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3795999999999996e-05 2024-09-16T05:12:12.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190551+0000", 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3119000000000002e-05 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.191468+0000", 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091758400000000004 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192622+0000", 2024-09-16T05:12:12.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011534869999999999 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192656+0000", 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4520000000000002e-05 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192683+0000", 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6888000000000001e-05 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:70 2.3 2:c018a787:::smithi20457400-18:head [write 1845135~440584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.194603+0000", 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 89.482071289999993, 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001788261, 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.715 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 70 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194603+0000", 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194603+0000", 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194605+0000", 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9570000000000001e-06 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194857+0000", 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025245399999999999 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194871+0000", 2024-09-16T05:12:12.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3179999999999999e-05 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194877+0000", 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.212e-06 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194922+0000", 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5527e-05 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194975+0000", 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2370999999999997e-05 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195052+0000", 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7354999999999994e-05 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195911+0000", 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00085943599999999999 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196327+0000", 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041503200000000002 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196364+0000", 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196391+0000", 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.692e-05 2024-09-16T05:12:12.720 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:72 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.194919+0000", 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 89.481754976999994, 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001705549, 2024-09-16T05:12:12.721 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 72 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194919+0000", 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194919+0000", 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194920+0000", 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4400000000000001e-07 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194921+0000", 2024-09-16T05:12:12.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7260000000000001e-06 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194928+0000", 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4370000000000004e-06 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.194932+0000", 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0300000000000004e-06 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195850+0000", 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091856800000000003 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.195900+0000", 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9851000000000001e-05 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196527+0000", 2024-09-16T05:12:12.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062624600000000001 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196569+0000", 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2281999999999999e-05 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.196625+0000", 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6064999999999999e-05 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.726 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:663 2.3 2:c018a787:::smithi20457400-18:head [delete] snapc 17=[16,15,14,12,f,b] ondisk+write+known_if_redirected+full_force+supports_pool_eio e150)", 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:21.673892+0000", 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 51.002781855000002, 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003266751, 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.727 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 663 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673892+0000", 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673892+0000", 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673894+0000", 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6190000000000001e-06 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673903+0000", 2024-09-16T05:12:12.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9649999999999997e-06 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673922+0000", 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9352e-05 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673932+0000", 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.73e-06 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.674009+0000", 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6707999999999998e-05 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.674017+0000", 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0630000000000006e-06 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.675338+0000", 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001321053 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.675446+0000", 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108467 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.675657+0000", 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021090099999999999 2024-09-16T05:12:12.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.676435+0000", 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077779900000000003 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.677042+0000", 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00060720499999999998 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.677106+0000", 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3399999999999996e-05 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.677159+0000", 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3489000000000002e-05 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.735 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:675 2.3 2:c018a787:::smithi20457400-18:18 [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e152)", 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:23.691201+0000", 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 48.985472839000003, 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002197634, 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 675 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691201+0000", 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691201+0000", 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691204+0000", 2024-09-16T05:12:12.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.492e-06 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691218+0000", 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4409999999999999e-05 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691242+0000", 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3547e-05 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691254+0000", 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2132e-05 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691333+0000", 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9136999999999998e-05 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691342+0000", 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1989999999999995e-06 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.693229+0000", 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001887073 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.693399+0000", 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016964399999999999 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.740 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:676 2.3 2:c018a787:::smithi20457400-18:18 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e152)", 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:23.691275+0000", 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 48.985398912999997, 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022351210000000001, 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 676 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691275+0000", 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691275+0000", 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691276+0000", 2024-09-16T05:12:12.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3500000000000002e-07 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691278+0000", 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2670000000000001e-06 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691284+0000", 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0449999999999997e-06 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691288+0000", 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7459999999999998e-06 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691418+0000", 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013056100000000001 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691425+0000", 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.46e-06 2024-09-16T05:12:12.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.693422+0000", 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019968939999999999 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.693510+0000", 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8713000000000003e-05 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:677 2.3 2:c018a787:::smithi20457400-18:18 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e152)", 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:23.691322+0000", 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 48.985351809000001, 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002335171, 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 677 2024-09-16T05:12:12.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691322+0000", 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691322+0000", 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691323+0000", 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.15e-07 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691325+0000", 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.742e-06 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691330+0000", 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6890000000000001e-06 2024-09-16T05:12:12.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691334+0000", 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8020000000000002e-06 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691457+0000", 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012282999999999999 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691463+0000", 2024-09-16T05:12:12.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7610000000000004e-06 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.693543+0000", 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020804539999999998 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.693658+0000", 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114478 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.750 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(osd.1.13:2 2.3 2:c018a787:::smithi20457400-18:head [copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:30.653613+0000", 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 42.023061642000002, 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022538160000000001, 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653613+0000", 2024-09-16T05:12:12.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653613+0000", 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653615+0000", 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6630000000000002e-06 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653630+0000", 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.485e-05 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653663+0000", 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2796000000000002e-05 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653679+0000", 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6175999999999999e-05 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653825+0000", 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000145959 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653832+0000", 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6989999999999997e-06 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.655674+0000", 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001842209 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.655866+0000", 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019246399999999999 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:869 2.2 2:4e7d172c:::smithi20457400-15:head [write 437015~721812] snapc 21=[21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-16T05:12:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:12:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.968057125000001, 2024-09-16T05:12:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053129060000000001, 2024-09-16T05:12:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.756 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 869 2024-09-16T05:12:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708620+0000", 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0369999999999999e-06 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709599+0000", 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00097878799999999997 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709622+0000", 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2609999999999999e-05 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709633+0000", 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1654999999999999e-05 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709764+0000", 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131023 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709772+0000", 2024-09-16T05:12:12.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6109999999999996e-06 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711031+0000", 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001258753 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711097+0000", 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6699000000000006e-05 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711246+0000", 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000148629 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.712976+0000", 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001729721 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713857+0000", 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00088113600000000003 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713898+0000", 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0843000000000002e-05 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713930+0000", 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2400999999999997e-05 2024-09-16T05:12:12.764 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:870 2.2 2:4e7d172c:::smithi20457400-15:head [write 1701889~509708] snapc 21=[21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:39.715008+0000", 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.961666346000001, 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022077680000000001, 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.765 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 870 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715008+0000", 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715008+0000", 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715010+0000", 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3099999999999999e-06 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715445+0000", 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00043461599999999997 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715457+0000", 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2411e-05 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715465+0000", 2024-09-16T05:12:12.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8099999999999998e-06 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715556+0000", 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1160000000000001e-05 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715619+0000", 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2918999999999998e-05 2024-09-16T05:12:12.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715723+0000", 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103713 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.716554+0000", 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00083093399999999997 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717141+0000", 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00058706800000000005 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717183+0000", 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2589999999999997e-05 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717216+0000", 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2237000000000003e-05 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.771 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:871 2.2 2:4e7d172c:::smithi20457400-15:head [setxattr _header (57) in=7b,truncate 2437015] snapc 21=[21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:39.715482+0000", 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.961192519999997, 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018490360000000001, 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.772 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 871 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715482+0000", 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715482+0000", 2024-09-16T05:12:12.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715482+0000", 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5199999999999996e-07 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715504+0000", 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1257e-05 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715511+0000", 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3289999999999996e-06 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715517+0000", 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8150000000000002e-06 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.716247+0000", 2024-09-16T05:12:12.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072972200000000003 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.716317+0000", 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0537999999999994e-05 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.716422+0000", 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104725 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.716848+0000", 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042613500000000001 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717255+0000", 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040753799999999999 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717292+0000", 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6075000000000003e-05 2024-09-16T05:12:12.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717331+0000", 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9249999999999999e-05 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:872 2.2 2:4e7d172c:::smithi20457400-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e168)", 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:39.715549+0000", 2024-09-16T05:12:12.778 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.961125320000001, 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019471720000000001, 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 872 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715549+0000", 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715549+0000", 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715550+0000", 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5000000000000002e-07 2024-09-16T05:12:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715553+0000", 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8909999999999997e-06 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715562+0000", 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4810000000000006e-06 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.715568+0000", 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4300000000000003e-06 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.716579+0000", 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010104140000000001 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.716625+0000", 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6032000000000002e-05 2024-09-16T05:12:12.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717366+0000", 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00074071399999999996 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717415+0000", 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9051000000000002e-05 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.717496+0000", 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1508999999999995e-05 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1023 2.2 2:4e7d172c:::smithi20457400-15:head [write 773155~459498,stat] snapc 29=[29,27,25,24,23,22,21,20,1f,15] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:53.703726+0000", 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.972948314, 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045541829999999998, 2024-09-16T05:12:12.784 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1023 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.703726+0000", 2024-09-16T05:12:12.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.703726+0000", 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.703728+0000", 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.097e-06 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704236+0000", 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050769499999999996 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704257+0000", 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1362e-05 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704265+0000", 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0199999999999994e-06 2024-09-16T05:12:12.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704366+0000", 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100938 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704373+0000", 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7530000000000004e-06 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.705649+0000", 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001276304 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.705772+0000", 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000122509 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.705912+0000", 2024-09-16T05:12:12.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014048700000000001 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.707325+0000", 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001412459 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708200+0000", 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087534099999999997 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708245+0000", 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5024000000000001e-05 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708280+0000", 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5194000000000002e-05 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1024 2.2 2:4e7d172c:::smithi20457400-15:head [write 1722470~430045] snapc 29=[29,27,25,24,23,22,21,20,1f,15] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:53.708629+0000", 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.968045325999999, 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002116288, 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1024 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.792 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708629+0000", 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708629+0000", 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708630+0000", 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.372e-06 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.709003+0000", 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000372992 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.709018+0000", 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4450999999999999e-05 2024-09-16T05:12:12.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.709025+0000", 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9149999999999999e-06 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.709065+0000", 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0697e-05 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.709130+0000", 2024-09-16T05:12:12.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5075999999999998e-05 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.709237+0000", 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106873 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.709921+0000", 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00068402800000000004 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.710675+0000", 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075372500000000001 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.710712+0000", 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6875999999999997e-05 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.710745+0000", 2024-09-16T05:12:12.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3283e-05 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1025 2.2 2:4e7d172c:::smithi20457400-15:head [write 2648676~124479] snapc 29=[29,27,25,24,23,22,21,20,1f,15] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:53.712037+0000", 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.964637504999999, 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016684969999999999, 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.798 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1025 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712037+0000", 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712037+0000", 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712038+0000", 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.161e-06 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712146+0000", 2024-09-16T05:12:12.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107612 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712159+0000", 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3774000000000001e-05 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712183+0000", 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4056000000000001e-05 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712220+0000", 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6328999999999997e-05 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712275+0000", 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5470000000000003e-05 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712371+0000", 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6001000000000005e-05 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713068+0000", 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00069724200000000004 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713632+0000", 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056372799999999999 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713671+0000", 2024-09-16T05:12:12.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8661000000000001e-05 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713705+0000", 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4462999999999999e-05 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1027 2.2 2:4e7d172c:::smithi20457400-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:53.712215+0000", 2024-09-16T05:12:12.804 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 18.964458933, 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017734980000000001, 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1027 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712215+0000", 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712215+0000", 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712216+0000", 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.96e-07 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712218+0000", 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3190000000000002e-06 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712222+0000", 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2189999999999998e-06 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.712226+0000", 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4989999999999999e-06 2024-09-16T05:12:12.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713095+0000", 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000869215 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713179+0000", 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4308000000000007e-05 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:12.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713867+0000", 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00068789299999999997 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713920+0000", 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2720999999999999e-05 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.713989+0000", 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9028000000000004e-05 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.809 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.810 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.810 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:12.810 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:12.810 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:12:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:12.811 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:12.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:12.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:12:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:12:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:12:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:12:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3482789044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:12.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:12:12.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3482789044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:12.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:12 smithi032 ceph-mon[27388]: osdmap e201: 12 total, 12 up, 12 in 2024-09-16T05:12:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:12.937 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:12.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3482789044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3482789044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:12.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:12 smithi196 ceph-mon[30802]: osdmap e201: 12 total, 12 up, 12 in 2024-09-16T05:12:13.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:13.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:13.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:13.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:12:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:12:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:12:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:12:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3482789044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:12:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3482789044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:12 smithi204 ceph-mon[30933]: osdmap e201: 12 total, 12 up, 12 in 2024-09-16T05:12:13.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:13.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:13.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:13.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:13.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:13.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:13.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:13.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:13.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:12:13.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:13.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:13.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:13.660 INFO:tasks.rados.rados.2.smithi204.stdout:475: done (0 left) 2024-09-16T05:12:13.660 INFO:tasks.rados.rados.2.smithi204.stdout:476: snap_remove snap 39 2024-09-16T05:12:13.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:13.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:13.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:13.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:12:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:13.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:13.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:13.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:13.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:13.949 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:13.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:13.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:13 smithi196 ceph-mon[30802]: pgmap v453: 37 pgs: 1 remapped+peering, 36 active+clean; 200 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:13 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:12:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:13 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:12:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:13 smithi196 ceph-mon[30802]: osdmap e202: 12 total, 12 up, 12 in 2024-09-16T05:12:14.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:13 smithi032 ceph-mon[27388]: pgmap v453: 37 pgs: 1 remapped+peering, 36 active+clean; 200 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:14.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:13 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:12:14.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:13 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:12:14.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:13 smithi032 ceph-mon[27388]: osdmap e202: 12 total, 12 up, 12 in 2024-09-16T05:12:14.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:14.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:14.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:13 smithi204 ceph-mon[30933]: pgmap v453: 37 pgs: 1 remapped+peering, 36 active+clean; 200 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:13 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:12:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:13 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:12:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:13 smithi204 ceph-mon[30933]: osdmap e202: 12 total, 12 up, 12 in 2024-09-16T05:12:14.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:14.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:14.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:14.214 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:14.215 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:14.235 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:14.256 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:14.256 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:14.277 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:14.299 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:14.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:14.326 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:14.326 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:14.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:14.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:14.347 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:14.347 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:14.368 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:14.369 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:14.390 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:14.391 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:14.412 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:14.412 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:14.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:14.434 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:14.434 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:14.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:14.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:14.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:14.456 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:14.457 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:14.465 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:14.465 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:14.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:14.576 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:14.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:14.664 INFO:tasks.rados.rados.2.smithi204.stdout:476: done (0 left) 2024-09-16T05:12:14.664 INFO:tasks.rados.rados.2.smithi204.stdout:477: write oid 13 current snap is 53 2024-09-16T05:12:14.664 INFO:tasks.rados.rados.2.smithi204.stdout:477: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-09-16T05:12:14.670 INFO:tasks.rados.rados.2.smithi204.stdout:477: writing smithi20457400-13 from 527786 to 1252266 tid 1 2024-09-16T05:12:14.676 INFO:tasks.rados.rados.2.smithi204.stdout:477: writing smithi20457400-13 from 1703317 to 2219371 tid 2 2024-09-16T05:12:14.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:14.681 INFO:tasks.rados.rados.2.smithi204.stdout:477: writing smithi20457400-13 from 2804576 to 3239611 tid 3 2024-09-16T05:12:14.681 INFO:tasks.rados.rados.2.smithi204.stdout:478: write (excl) oid 36 current snap is 53 2024-09-16T05:12:14.681 INFO:tasks.rados.rados.2.smithi204.stdout:478: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-09-16T05:12:14.688 INFO:tasks.rados.rados.2.smithi204.stdout:478: writing smithi20457400-36 from 544593 to 1344322 tid 1 2024-09-16T05:12:14.695 INFO:tasks.rados.rados.2.smithi204.stdout:478: writing smithi20457400-36 from 1761799 to 2537864 tid 2 2024-09-16T05:12:14.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:14.700 INFO:tasks.rados.rados.2.smithi204.stdout:478: writing smithi20457400-36 from 2948352 to 3344593 tid 3 2024-09-16T05:12:14.700 INFO:tasks.rados.rados.2.smithi204.stdout:479: snap_remove snap 50 2024-09-16T05:12:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:14.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:14.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:14.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:14.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:14.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:14.949 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:14.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:14.978 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:12:14.978 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:12:14.978 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T05:12:14.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:12:15.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:14 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:12:15.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:14 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:12:15.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:14 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:12:15.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:14 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:12:15.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:14 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:12:15.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:14 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:12:15.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3751844425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:15.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:14 smithi032 ceph-mon[27388]: osdmap e203: 12 total, 12 up, 12 in 2024-09-16T05:12:15.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:15.079 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:15.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:15.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:14 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:12:15.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:14 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:12:15.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:14 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:12:15.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:14 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:12:15.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:14 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:12:15.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:14 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:12:15.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3751844425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:15.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:14 smithi196 ceph-mon[30802]: osdmap e203: 12 total, 12 up, 12 in 2024-09-16T05:12:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:14 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:12:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:14 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:12:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:14 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:12:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:14 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:12:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:14 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:12:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:14 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:12:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3751844425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:14 smithi204 ceph-mon[30933]: osdmap e203: 12 total, 12 up, 12 in 2024-09-16T05:12:15.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:15.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:12:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:15.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:15.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:15.246 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:15.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:15.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:15.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:15.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:12:15.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:15.448 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:15.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:15.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:15.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:15.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:15.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:15.659 INFO:tasks.rados.rados.2.smithi204.stdout:479: done (2 left) 2024-09-16T05:12:15.659 INFO:tasks.rados.rados.2.smithi204.stdout:480: read oid 12 snap -1 2024-09-16T05:12:15.659 INFO:tasks.rados.rados.2.smithi204.stdout:480: expect (ObjNum 124 snap 26 seq_num 124) 2024-09-16T05:12:15.659 INFO:tasks.rados.rados.2.smithi204.stdout:481: write oid 6 current snap is 53 2024-09-16T05:12:15.659 INFO:tasks.rados.rados.2.smithi204.stdout:477: finishing write tid 1 to smithi20457400-13 2024-09-16T05:12:15.659 INFO:tasks.rados.rados.2.smithi204.stdout:481: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-16T05:12:15.665 INFO:tasks.rados.rados.2.smithi204.stdout:481: writing smithi20457400-6 from 561400 to 1036378 tid 1 2024-09-16T05:12:15.672 INFO:tasks.rados.rados.2.smithi204.stdout:481: writing smithi20457400-6 from 1820281 to 2340004 tid 2 2024-09-16T05:12:15.677 INFO:tasks.rados.rados.2.smithi204.stdout:481: writing smithi20457400-6 from 2859422 to 3317001 tid 3 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:482: delete oid 9 current snap is 53 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:477: finishing write tid 2 to smithi20457400-13 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:477: finishing write tid 3 to smithi20457400-13 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:477: finishing write tid 4 to smithi20457400-13 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:477: finishing write tid 5 to smithi20457400-13 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 287 (ObjNum 198 snap 53 seq_num 198) dirty exists 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:477: left oid 13 (ObjNum 198 snap 53 seq_num 198) 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:478: finishing write tid 1 to smithi20457400-36 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:478: finishing write tid 2 to smithi20457400-36 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:478: finishing write tid 3 to smithi20457400-36 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:478: finishing write tid 4 to smithi20457400-36 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:478: finishing write tid 5 to smithi20457400-36 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 277 (ObjNum 199 snap 53 seq_num 199) dirty exists 2024-09-16T05:12:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:478: left oid 36 (ObjNum 199 snap 53 seq_num 199) 2024-09-16T05:12:15.679 INFO:tasks.rados.rados.2.smithi204.stdout:480: expect (ObjNum 124 snap 26 seq_num 124) 2024-09-16T05:12:15.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:15.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:15.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:15.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:15.806 INFO:tasks.rados.rados.2.smithi204.stdout:481: finishing write tid 1 to smithi20457400-6 2024-09-16T05:12:15.807 INFO:tasks.rados.rados.2.smithi204.stdout:481: finishing write tid 2 to smithi20457400-6 2024-09-16T05:12:15.807 INFO:tasks.rados.rados.2.smithi204.stdout:477: done (4 left) 2024-09-16T05:12:15.807 INFO:tasks.rados.rados.2.smithi204.stdout:478: done (3 left) 2024-09-16T05:12:15.807 INFO:tasks.rados.rados.2.smithi204.stdout:480: done (2 left) 2024-09-16T05:12:15.807 INFO:tasks.rados.rados.2.smithi204.stdout:482: done (1 left) 2024-09-16T05:12:15.807 INFO:tasks.rados.rados.2.smithi204.stdout:483: snap_remove snap 52 2024-09-16T05:12:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:15.827 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:15.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:15.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:15 smithi196 ceph-mon[30802]: pgmap v456: 37 pgs: 1 remapped+peering, 36 active+clean; 202 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:15.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:12:15.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:15 smithi196 ceph-mon[30802]: osdmap e204: 12 total, 12 up, 12 in 2024-09-16T05:12:15.874 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:12:15.875 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:12:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:15 smithi032 ceph-mon[27388]: pgmap v456: 37 pgs: 1 remapped+peering, 36 active+clean; 202 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:12:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:15 smithi032 ceph-mon[27388]: osdmap e204: 12 total, 12 up, 12 in 2024-09-16T05:12:15.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:15.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:15.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:15.978 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:15.978 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:16.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:16.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:16.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:16.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:15 smithi204 ceph-mon[30933]: pgmap v456: 37 pgs: 1 remapped+peering, 36 active+clean; 202 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:12:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:15 smithi204 ceph-mon[30933]: osdmap e204: 12 total, 12 up, 12 in 2024-09-16T05:12:16.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:16.200 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:16.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:16.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:16.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:16.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:16.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:16.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:16.451 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:16.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:16.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:16.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:12:16.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:16.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:16.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:16.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:16.770 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:12:16.773 INFO:tasks.rados.rados.2.smithi204.stdout:483: done (1 left) 2024-09-16T05:12:16.773 INFO:tasks.rados.rados.2.smithi204.stdout:484: read oid 13 snap 41 2024-09-16T05:12:16.774 INFO:tasks.rados.rados.2.smithi204.stdout:484: expect (ObjNum 116 snap 25 seq_num 116) 2024-09-16T05:12:16.774 INFO:tasks.rados.rados.2.smithi204.stdout:485: snap_create 2024-09-16T05:12:16.774 INFO:tasks.rados.rados.2.smithi204.stdout:481: finishing write tid 3 to smithi20457400-6 2024-09-16T05:12:16.774 INFO:tasks.rados.rados.2.smithi204.stdout:481: finishing write tid 4 to smithi20457400-6 2024-09-16T05:12:16.774 INFO:tasks.rados.rados.2.smithi204.stdout:481: finishing write tid 5 to smithi20457400-6 2024-09-16T05:12:16.774 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 31 (ObjNum 200 snap 53 seq_num 200) dirty exists 2024-09-16T05:12:16.774 INFO:tasks.rados.rados.2.smithi204.stdout:481: left oid 6 (ObjNum 200 snap 53 seq_num 200) 2024-09-16T05:12:16.784 INFO:tasks.rados.rados.2.smithi204.stdout:484: expect (ObjNum 116 snap 25 seq_num 116) 2024-09-16T05:12:16.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:16.818 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:16.825 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:16.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:16.848 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-16T05:12:15.654655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":212144433,"num_objects":107,"num_object_clones":60,"num_object_copies":214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":8,"num_read":541,"num_read_kb":460775,"num_write":500,"num_write_kb":179916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3926,"ondisk_log_size":3926,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":642292,"kb_used_data":280464,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124299532,"statfs":{"total":1151940427776,"available":1151282720768,"internally_reserved":0,"allocated":287195136,"data_stored":346172998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6034897,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":-197,"num_read_kb":-164700,"num_write":-129,"num_write_kb":-44616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-32,"num_bytes_recovered":-14956816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000347"},"pg_stats":[{"pgid":"2.19","version":"182'60","reported_seq":83,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T05:12:12.658835+0000","last_change":"2024-09-16T05:12:12.659138+0000","last_active":"2024-09-16T05:12:12.658835+0000","last_peered":"2024-09-16T05:12:12.658835+0000","last_clean":"2024-09-16T05:12:12.658835+0000","last_became_active":"2024-09-16T05:11:57.658646+0000","last_became_peered":"2024-09-16T05:11:57.658646+0000","last_unstale":"2024-09-16T05:12:12.658835+0000","last_undegraded":"2024-09-16T05:12:12.658835+0000","last_fullsized":"2024-09-16T05:12:12.658835+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:12:09.470099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.470099+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:32.434898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050470400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":486,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.660900+0000","last_change":"2024-09-16T05:12:14.661168+0000","last_active":"2024-09-16T05:12:14.660900+0000","last_peered":"2024-09-16T05:12:14.660900+0000","last_clean":"2024-09-16T05:12:14.660900+0000","last_became_active":"2024-09-16T05:11:57.660350+0000","last_became_peered":"2024-09-16T05:11:57.660350+0000","last_unstale":"2024-09-16T05:12:14.660900+0000","last_undegraded":"2024-09-16T05:12:14.660900+0000","last_fullsized":"2024-09-16T05:12:14.660900+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:12:08.969861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.969861+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:09.971038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048928899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.17","version":"0'0","reported_seq":258,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-16T05:12:12.058794+0000","last_change":"2024-09-16T05:12:09.877178+0000","last_active":"2024-09-16T05:12:12.058794+0000","last_peered":"2024-09-16T05:12:12.058794+0000","last_clean":"2024-09-16T05:12:12.058794+0000","last_became_active":"2024-09-16T05:11:57.657217+0000","last_became_peered":"2024-09-16T05:11:57.657217+0000","last_unstale":"2024-09-16T05:12:12.058794+0000","last_undegraded":"2024-09-16T05:12:12.058794+0000","last_fullsized":"2024-09-16T05:12:12.058794+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:12:09.877069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.877069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:28.448773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000283868,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"2.16","version":"116'9","reported_seq":77,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.305504+0000","last_change":"2024-09-16T05:12:09.331947+0000","last_active":"2024-09-16T05:12:10.305504+0000","last_peered":"2024-09-16T05:12:10.305504+0000","last_clean":"2024-09-16T05:12:10.305504+0000","last_became_active":"2024-09-16T05:11:57.656772+0000","last_became_peered":"2024-09-16T05:11:57.656772+0000","last_unstale":"2024-09-16T05:12:10.305504+0000","last_undegraded":"2024-09-16T05:12:10.305504+0000","last_fullsized":"2024-09-16T05:12:10.305504+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:12:09.331857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.331857+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:38.420377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00026249500000000001,"stat_sum":{"num_bytes":2857157,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"2"},{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"2.15","version":"203'287","reported_seq":751,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.683718+0000","last_change":"2024-09-16T05:12:14.666677+0000","last_active":"2024-09-16T05:12:14.683718+0000","last_peered":"2024-09-16T05:12:14.683718+0000","last_clean":"2024-09-16T05:12:14.683718+0000","last_became_active":"2024-09-16T05:12:00.711890+0000","last_became_peered":"2024-09-16T05:12:00.711890+0000","last_unstale":"2024-09-16T05:12:14.683718+0000","last_undegraded":"2024-09-16T05:12:14.683718+0000","last_fullsized":"2024-09-16T05:12:14.683718+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'276","last_scrub_stamp":"2024-09-16T05:12:11.920515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:11.920515+0000","objects_scrubbed":7,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:19.366003+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0056553180000000003,"stat_sum":{"num_bytes":15328519,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.14","version":"203'112","reported_seq":164,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.662884+0000","last_change":"2024-09-16T05:12:14.663289+0000","last_active":"2024-09-16T05:12:14.662884+0000","last_peered":"2024-09-16T05:12:14.662884+0000","last_clean":"2024-09-16T05:12:14.662884+0000","last_became_active":"2024-09-16T05:11:57.658852+0000","last_became_peered":"2024-09-16T05:11:57.658852+0000","last_unstale":"2024-09-16T05:12:14.662884+0000","last_undegraded":"2024-09-16T05:12:14.662884+0000","last_fullsized":"2024-09-16T05:12:14.662884+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'108","last_scrub_stamp":"2024-09-16T05:12:10.924782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.924782+0000","objects_scrubbed":6,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:14.952554+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0024980470000000002,"stat_sum":{"num_bytes":12183434,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.13","version":"181'15","reported_seq":93,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.660963+0000","last_change":"2024-09-16T05:12:14.661295+0000","last_active":"2024-09-16T05:12:14.660963+0000","last_peered":"2024-09-16T05:12:14.660963+0000","last_clean":"2024-09-16T05:12:14.660963+0000","last_became_active":"2024-09-16T05:11:57.660401+0000","last_became_peered":"2024-09-16T05:11:57.660401+0000","last_unstale":"2024-09-16T05:12:14.660963+0000","last_undegraded":"2024-09-16T05:12:14.660963+0000","last_fullsized":"2024-09-16T05:12:14.660963+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:12:09.952097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.952097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:20.629801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000530621,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.12","version":"192'28","reported_seq":89,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.670115+0000","last_change":"2024-09-16T05:12:09.669191+0000","last_active":"2024-09-16T05:12:10.670115+0000","last_peered":"2024-09-16T05:12:10.670115+0000","last_clean":"2024-09-16T05:12:10.670115+0000","last_became_active":"2024-09-16T05:11:57.658975+0000","last_became_peered":"2024-09-16T05:11:57.658975+0000","last_unstale":"2024-09-16T05:12:10.670115+0000","last_undegraded":"2024-09-16T05:12:10.670115+0000","last_fullsized":"2024-09-16T05:12:10.670115+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'28","last_scrub_stamp":"2024-09-16T05:12:09.669066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.669066+0000","objects_scrubbed":3,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:12.814663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00034816799999999999,"stat_sum":{"num_bytes":4894218,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":609,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.660700+0000","last_change":"2024-09-16T05:12:13.922480+0000","last_active":"2024-09-16T05:12:14.660700+0000","last_peered":"2024-09-16T05:12:14.660700+0000","last_clean":"2024-09-16T05:12:14.660700+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:12:14.660700+0000","last_undegraded":"2024-09-16T05:12:14.660700+0000","last_fullsized":"2024-09-16T05:12:14.660700+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.922401+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.922401+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:15.772375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"203'97","reported_seq":583,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.667555+0000","last_change":"2024-09-16T05:12:14.667791+0000","last_active":"2024-09-16T05:12:14.667555+0000","last_peered":"2024-09-16T05:12:14.667555+0000","last_clean":"2024-09-16T05:12:14.667555+0000","last_became_active":"2024-09-16T05:11:55.657137+0000","last_became_peered":"2024-09-16T05:11:55.657137+0000","last_unstale":"2024-09-16T05:12:14.667555+0000","last_undegraded":"2024-09-16T05:12:14.667555+0000","last_fullsized":"2024-09-16T05:12:14.667555+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'70","last_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.116608+0000","objects_scrubbed":7,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:20.439730+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.0025968839999999998,"stat_sum":{"num_bytes":15774405,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":595,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.340135+0000","last_change":"2024-09-16T05:10:43.759254+0000","last_active":"2024-09-16T05:12:11.340135+0000","last_peered":"2024-09-16T05:12:11.340135+0000","last_clean":"2024-09-16T05:12:11.340135+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:12:11.340135+0000","last_undegraded":"2024-09-16T05:12:11.340135+0000","last_fullsized":"2024-09-16T05:12:11.340135+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:43.759130+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:43.759130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"198'231","reported_seq":677,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.657839+0000","last_change":"2024-09-16T05:12:08.763489+0000","last_active":"2024-09-16T05:12:11.657839+0000","last_peered":"2024-09-16T05:12:11.657839+0000","last_clean":"2024-09-16T05:12:11.657839+0000","last_became_active":"2024-09-16T05:10:37.002200+0000","last_became_peered":"2024-09-16T05:10:37.002200+0000","last_unstale":"2024-09-16T05:12:11.657839+0000","last_undegraded":"2024-09-16T05:12:11.657839+0000","last_fullsized":"2024-09-16T05:12:11.657839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'144","last_scrub_stamp":"2024-09-16T05:11:46.247408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.247408+0000","objects_scrubbed":15,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:52.538166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044808249999999999,"stat_sum":{"num_bytes":38594014,"num_objects":15,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":111,"num_read_kb":100268,"num_write":103,"num_write_kb":40383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":587,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.305409+0000","last_change":"2024-09-16T05:10:45.437354+0000","last_active":"2024-09-16T05:12:10.305409+0000","last_peered":"2024-09-16T05:12:10.305409+0000","last_clean":"2024-09-16T05:12:10.305409+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:12:10.305409+0000","last_undegraded":"2024-09-16T05:12:10.305409+0000","last_fullsized":"2024-09-16T05:12:10.305409+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.437261+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.437261+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"201'141","reported_seq":598,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T05:12:12.660264+0000","last_change":"2024-09-16T05:12:12.660522+0000","last_active":"2024-09-16T05:12:12.660264+0000","last_peered":"2024-09-16T05:12:12.660264+0000","last_clean":"2024-09-16T05:12:12.660264+0000","last_became_active":"2024-09-16T05:10:36.956652+0000","last_became_peered":"2024-09-16T05:10:36.956652+0000","last_unstale":"2024-09-16T05:12:12.660264+0000","last_undegraded":"2024-09-16T05:12:12.660264+0000","last_fullsized":"2024-09-16T05:12:12.660264+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'106","last_scrub_stamp":"2024-09-16T05:11:51.494420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:51.494420+0000","objects_scrubbed":11,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:06.990943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030202670000000001,"stat_sum":{"num_bytes":15991175,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":72717,"num_write":58,"num_write_kb":27853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":514,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.670073+0000","last_change":"2024-09-16T05:11:27.759078+0000","last_active":"2024-09-16T05:12:10.670073+0000","last_peered":"2024-09-16T05:12:10.670073+0000","last_clean":"2024-09-16T05:12:10.670073+0000","last_became_active":"2024-09-16T05:11:21.676107+0000","last_became_peered":"2024-09-16T05:11:21.676107+0000","last_unstale":"2024-09-16T05:12:10.670073+0000","last_undegraded":"2024-09-16T05:12:10.670073+0000","last_fullsized":"2024-09-16T05:12:10.670073+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:27.759006+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:27.759006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:36.703164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":550,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.665386+0000","last_change":"2024-09-16T05:12:14.666120+0000","last_active":"2024-09-16T05:12:14.665386+0000","last_peered":"2024-09-16T05:12:14.665386+0000","last_clean":"2024-09-16T05:12:14.665386+0000","last_became_active":"2024-09-16T05:11:55.665450+0000","last_became_peered":"2024-09-16T05:11:55.665450+0000","last_unstale":"2024-09-16T05:12:14.665386+0000","last_undegraded":"2024-09-16T05:12:14.665386+0000","last_fullsized":"2024-09-16T05:12:14.665386+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:12:10.025043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.025043+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:32.418787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093617600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":594,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.669973+0000","last_change":"2024-09-16T05:10:45.767820+0000","last_active":"2024-09-16T05:12:10.669973+0000","last_peered":"2024-09-16T05:12:10.669973+0000","last_clean":"2024-09-16T05:12:10.669973+0000","last_became_active":"2024-09-16T05:10:38.972286+0000","last_became_peered":"2024-09-16T05:10:38.972286+0000","last_unstale":"2024-09-16T05:12:10.669973+0000","last_undegraded":"2024-09-16T05:12:10.669973+0000","last_fullsized":"2024-09-16T05:12:10.669973+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:45.767717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:45.767717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"203'314","reported_seq":765,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.664201+0000","last_change":"2024-09-16T05:12:14.664201+0000","last_active":"2024-09-16T05:12:14.664201+0000","last_peered":"2024-09-16T05:12:14.664201+0000","last_clean":"2024-09-16T05:12:14.664201+0000","last_became_active":"2024-09-16T05:12:12.663643+0000","last_became_peered":"2024-09-16T05:12:12.663643+0000","last_unstale":"2024-09-16T05:12:14.664201+0000","last_undegraded":"2024-09-16T05:12:14.664201+0000","last_fullsized":"2024-09-16T05:12:14.664201+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'266","last_scrub_stamp":"2024-09-16T05:11:56.487686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.487686+0000","objects_scrubbed":7,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:04.922625+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032980240000000001,"stat_sum":{"num_bytes":27406329,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":22,"num_read_kb":23922,"num_write":13,"num_write_kb":9870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":594,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.669994+0000","last_change":"2024-09-16T05:10:44.788217+0000","last_active":"2024-09-16T05:12:10.669994+0000","last_peered":"2024-09-16T05:12:10.669994+0000","last_clean":"2024-09-16T05:12:10.669994+0000","last_became_active":"2024-09-16T05:10:38.972150+0000","last_became_peered":"2024-09-16T05:10:38.972150+0000","last_unstale":"2024-09-16T05:12:10.669994+0000","last_undegraded":"2024-09-16T05:12:10.669994+0000","last_fullsized":"2024-09-16T05:12:10.669994+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:44.788114+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:44.788114+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:11.681342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":476,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.665326+0000","last_change":"2024-09-16T05:12:14.665907+0000","last_active":"2024-09-16T05:12:14.665326+0000","last_peered":"2024-09-16T05:12:14.665326+0000","last_clean":"2024-09-16T05:12:14.665326+0000","last_became_active":"2024-09-16T05:11:55.657738+0000","last_became_peered":"2024-09-16T05:11:55.657738+0000","last_unstale":"2024-09-16T05:12:14.665326+0000","last_undegraded":"2024-09-16T05:12:14.665326+0000","last_fullsized":"2024-09-16T05:12:14.665326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:12:08.997285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.997285+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:27.694342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000749221,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":562,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.657904+0000","last_change":"2024-09-16T05:12:09.317163+0000","last_active":"2024-09-16T05:12:11.657904+0000","last_peered":"2024-09-16T05:12:11.657904+0000","last_clean":"2024-09-16T05:12:11.657904+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:12:11.657904+0000","last_undegraded":"2024-09-16T05:12:11.657904+0000","last_fullsized":"2024-09-16T05:12:11.657904+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:09.317055+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.317055+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:18.430868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":261,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.670156+0000","last_change":"2024-09-16T05:12:09.040184+0000","last_active":"2024-09-16T05:12:10.670156+0000","last_peered":"2024-09-16T05:12:10.670156+0000","last_clean":"2024-09-16T05:12:10.670156+0000","last_became_active":"2024-09-16T05:11:55.659156+0000","last_became_peered":"2024-09-16T05:11:55.659156+0000","last_unstale":"2024-09-16T05:12:10.670156+0000","last_undegraded":"2024-09-16T05:12:10.670156+0000","last_fullsized":"2024-09-16T05:12:10.670156+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:11:56.378387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.378387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:24.956070+0000","scrub_duration":686,"objects_trimmed":0,"snaptrim_duration":0.00029330699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"2.0","version":"195'26","reported_seq":471,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.340326+0000","last_change":"2024-09-16T05:12:08.756025+0000","last_active":"2024-09-16T05:12:11.340326+0000","last_peered":"2024-09-16T05:12:11.340326+0000","last_clean":"2024-09-16T05:12:11.340326+0000","last_became_active":"2024-09-16T05:11:55.656232+0000","last_became_peered":"2024-09-16T05:11:55.656232+0000","last_unstale":"2024-09-16T05:12:11.340326+0000","last_undegraded":"2024-09-16T05:12:11.340326+0000","last_fullsized":"2024-09-16T05:12:11.340326+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'18","last_scrub_stamp":"2024-09-16T05:11:55.862199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:55.862199+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:04.015970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000327786,"stat_sum":{"num_bytes":6420241,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":608,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T05:12:12.658072+0000","last_change":"2024-09-16T05:11:53.500880+0000","last_active":"2024-09-16T05:12:12.658072+0000","last_peered":"2024-09-16T05:12:12.658072+0000","last_clean":"2024-09-16T05:12:12.658072+0000","last_became_active":"2024-09-16T05:11:19.419149+0000","last_became_peered":"2024-09-16T05:11:19.419149+0000","last_unstale":"2024-09-16T05:12:12.658072+0000","last_undegraded":"2024-09-16T05:12:12.658072+0000","last_fullsized":"2024-09-16T05:12:12.658072+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:53.500734+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:53.500734+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:08.214137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":497,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.655404+0000","last_change":"2024-09-16T05:12:09.119321+0000","last_active":"2024-09-16T05:12:11.655404+0000","last_peered":"2024-09-16T05:12:11.655404+0000","last_clean":"2024-09-16T05:12:11.655404+0000","last_became_active":"2024-09-16T05:11:55.656596+0000","last_became_peered":"2024-09-16T05:11:55.656596+0000","last_unstale":"2024-09-16T05:12:11.655404+0000","last_undegraded":"2024-09-16T05:12:11.655404+0000","last_fullsized":"2024-09-16T05:12:11.655404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:11:56.000487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.000487+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:16.106618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040549599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"1.2","version":"35'139","reported_seq":595,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.339945+0000","last_change":"2024-09-16T05:10:42.783065+0000","last_active":"2024-09-16T05:12:11.339945+0000","last_peered":"2024-09-16T05:12:11.339945+0000","last_clean":"2024-09-16T05:12:11.339945+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:12:11.339945+0000","last_undegraded":"2024-09-16T05:12:11.339945+0000","last_fullsized":"2024-09-16T05:12:11.339945+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:42.782976+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:42.782976+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":601,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.669955+0000","last_change":"2024-09-16T05:11:51.723968+0000","last_active":"2024-09-16T05:12:10.669955+0000","last_peered":"2024-09-16T05:12:10.669955+0000","last_clean":"2024-09-16T05:12:10.669955+0000","last_became_active":"2024-09-16T05:10:38.972381+0000","last_became_peered":"2024-09-16T05:10:38.972381+0000","last_unstale":"2024-09-16T05:12:10.669955+0000","last_undegraded":"2024-09-16T05:12:10.669955+0000","last_fullsized":"2024-09-16T05:12:10.669955+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:51.723892+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:51.723892+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:12.858953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":450,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.305455+0000","last_change":"2024-09-16T05:12:09.039770+0000","last_active":"2024-09-16T05:12:10.305455+0000","last_peered":"2024-09-16T05:12:10.305455+0000","last_clean":"2024-09-16T05:12:10.305455+0000","last_became_active":"2024-09-16T05:11:55.657334+0000","last_became_peered":"2024-09-16T05:11:55.657334+0000","last_unstale":"2024-09-16T05:12:10.305455+0000","last_undegraded":"2024-09-16T05:12:10.305455+0000","last_fullsized":"2024-09-16T05:12:10.305455+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:11:56.310771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.310771+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:12.190084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000238311,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"2"},{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":600,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.340301+0000","last_change":"2024-09-16T05:11:54.869494+0000","last_active":"2024-09-16T05:12:11.340301+0000","last_peered":"2024-09-16T05:12:11.340301+0000","last_clean":"2024-09-16T05:12:11.340301+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:12:11.340301+0000","last_undegraded":"2024-09-16T05:12:11.340301+0000","last_fullsized":"2024-09-16T05:12:11.340301+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:54.869407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:54.869407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:21.879716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"184'17","reported_seq":451,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-16T05:12:10.305948+0000","last_change":"2024-09-16T05:12:10.305973+0000","last_active":"2024-09-16T05:12:10.305948+0000","last_peered":"2024-09-16T05:12:10.305948+0000","last_clean":"2024-09-16T05:12:10.305948+0000","last_became_active":"2024-09-16T05:11:55.661748+0000","last_became_peered":"2024-09-16T05:11:55.661748+0000","last_unstale":"2024-09-16T05:12:10.305948+0000","last_undegraded":"2024-09-16T05:12:10.305948+0000","last_fullsized":"2024-09-16T05:12:10.305948+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:12:10.305897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.305897+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:37.407601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035453999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"24","length":"2"},{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"2.8","version":"197'67","reported_seq":511,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.340160+0000","last_change":"2024-09-16T05:12:09.039266+0000","last_active":"2024-09-16T05:12:11.340160+0000","last_peered":"2024-09-16T05:12:11.340160+0000","last_clean":"2024-09-16T05:12:11.340160+0000","last_became_active":"2024-09-16T05:11:55.658383+0000","last_became_peered":"2024-09-16T05:11:55.658383+0000","last_unstale":"2024-09-16T05:12:11.340160+0000","last_undegraded":"2024-09-16T05:12:11.340160+0000","last_fullsized":"2024-09-16T05:12:11.340160+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'67","last_scrub_stamp":"2024-09-16T05:12:09.039188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.039188+0000","objects_scrubbed":4,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:17.774410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024929370000000002,"stat_sum":{"num_bytes":4823576,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"2.c","version":"197'163","reported_seq":589,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.657914+0000","last_change":"2024-09-16T05:12:08.762816+0000","last_active":"2024-09-16T05:12:11.657914+0000","last_peered":"2024-09-16T05:12:11.657914+0000","last_clean":"2024-09-16T05:12:11.657914+0000","last_became_active":"2024-09-16T05:10:36.957767+0000","last_became_peered":"2024-09-16T05:10:36.957767+0000","last_unstale":"2024-09-16T05:12:11.657914+0000","last_undegraded":"2024-09-16T05:12:11.657914+0000","last_fullsized":"2024-09-16T05:12:11.657914+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'99","last_scrub_stamp":"2024-09-16T05:11:50.223175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:50.223175+0000","objects_scrubbed":7,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:04.824381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003626116,"stat_sum":{"num_bytes":17779156,"num_objects":11,"num_object_clones":8,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":51,"num_read_kb":35573,"num_write":62,"num_write_kb":21312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"2.d","version":"203'246","reported_seq":707,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-16T05:12:14.666965+0000","last_change":"2024-09-16T05:12:14.667123+0000","last_active":"2024-09-16T05:12:14.666965+0000","last_peered":"2024-09-16T05:12:14.666965+0000","last_clean":"2024-09-16T05:12:14.666965+0000","last_became_active":"2024-09-16T05:10:36.957439+0000","last_became_peered":"2024-09-16T05:10:36.957439+0000","last_unstale":"2024-09-16T05:12:14.666965+0000","last_undegraded":"2024-09-16T05:12:14.666965+0000","last_fullsized":"2024-09-16T05:12:14.666965+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'165","last_scrub_stamp":"2024-09-16T05:11:45.953349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:45.953349+0000","objects_scrubbed":11,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:47.764666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060107839999999999,"stat_sum":{"num_bytes":20656738,"num_objects":10,"num_object_clones":4,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":123,"num_read_kb":90314,"num_write":116,"num_write_kb":37600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"29","length":"2"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.e","version":"197'150","reported_seq":586,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-16T05:12:11.655478+0000","last_change":"2024-09-16T05:12:09.120318+0000","last_active":"2024-09-16T05:12:11.655478+0000","last_peered":"2024-09-16T05:12:11.655478+0000","last_clean":"2024-09-16T05:12:11.655478+0000","last_became_active":"2024-09-16T05:10:37.001925+0000","last_became_peered":"2024-09-16T05:10:37.001925+0000","last_unstale":"2024-09-16T05:12:11.655478+0000","last_undegraded":"2024-09-16T05:12:11.655478+0000","last_fullsized":"2024-09-16T05:12:11.655478+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:10:35.939787+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013919469999999999,"stat_sum":{"num_bytes":11612486,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":81,"num_read_kb":73323,"num_write":72,"num_write_kb":19869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]},{"pgid":"2.f","version":"201'129","reported_seq":593,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T05:12:12.661674+0000","last_change":"2024-09-16T05:12:12.661926+0000","last_active":"2024-09-16T05:12:12.661674+0000","last_peered":"2024-09-16T05:12:12.661674+0000","last_clean":"2024-09-16T05:12:12.661674+0000","last_became_active":"2024-09-16T05:10:36.956705+0000","last_became_peered":"2024-09-16T05:10:36.956705+0000","last_unstale":"2024-09-16T05:12:12.661674+0000","last_undegraded":"2024-09-16T05:12:12.661674+0000","last_fullsized":"2024-09-16T05:12:12.661674+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'74","last_scrub_stamp":"2024-09-16T05:11:46.374423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.374423+0000","objects_scrubbed":6,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:09.245884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0044182149999999996,"stat_sum":{"num_bytes":14946898,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":76,"num_read_kb":64654,"num_write":57,"num_write_kb":17176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":83,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-16T05:12:12.657664+0000","last_change":"2024-09-16T05:12:12.658006+0000","last_active":"2024-09-16T05:12:12.657664+0000","last_peered":"2024-09-16T05:12:12.657664+0000","last_clean":"2024-09-16T05:12:12.657664+0000","last_became_active":"2024-09-16T05:11:57.658263+0000","last_became_peered":"2024-09-16T05:11:57.658263+0000","last_unstale":"2024-09-16T05:12:12.657664+0000","last_undegraded":"2024-09-16T05:12:12.657664+0000","last_fullsized":"2024-09-16T05:12:12.657664+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:12:09.468817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.468817+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:30.697491+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053940399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"},{"start":"2e","length":"1"}]},{"pgid":"2.11","version":"197'64","reported_seq":110,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-16T05:12:12.058740+0000","last_change":"2024-09-16T05:12:08.859693+0000","last_active":"2024-09-16T05:12:12.058740+0000","last_peered":"2024-09-16T05:12:12.058740+0000","last_clean":"2024-09-16T05:12:12.058740+0000","last_became_active":"2024-09-16T05:11:57.662582+0000","last_became_peered":"2024-09-16T05:11:57.662582+0000","last_unstale":"2024-09-16T05:12:12.058740+0000","last_undegraded":"2024-09-16T05:12:12.058740+0000","last_fullsized":"2024-09-16T05:12:12.058740+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'64","last_scrub_stamp":"2024-09-16T05:12:08.859612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.859612+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:37.543596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00130817,"stat_sum":{"num_bytes":2285719,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2a","length":"1"},{"start":"2c","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":211554065,"num_objects":105,"num_object_clones":60,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":8,"num_read":541,"num_read_kb":460775,"num_write":500,"num_write_kb":179916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258334720,"data_stored":328837545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2397,"ondisk_log_size":2397,"up":52,"acting":52,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089381,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87928,"kb_used_data":60892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93657224,"statfs":{"total":95995035648,"available":95904997376,"internally_reserved":0,"allocated":62353408,"data_stored":81326147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285610,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40220,"kb_used_data":9028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93704932,"statfs":{"total":95995035648,"available":95953850368,"internally_reserved":0,"allocated":9244672,"data_stored":8503660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514545,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90712,"kb_used_data":59520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93654440,"statfs":{"total":95995035648,"available":95902146560,"internally_reserved":0,"allocated":60948480,"data_stored":69038796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841587,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52564,"kb_used_data":21368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93692588,"statfs":{"total":95995035648,"available":95941210112,"internally_reserved":0,"allocated":21880832,"data_stored":29250410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103224,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49612,"kb_used_data":18416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93695540,"statfs":{"total":95995035648,"available":95944232960,"internally_reserved":0,"allocated":18857984,"data_stored":22711511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397567,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45140,"kb_used_data":13952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93700012,"statfs":{"total":95995035648,"available":95948812288,"internally_reserved":0,"allocated":14286848,"data_stored":15033298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31444,"kb_used_data":4416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713708,"statfs":{"total":95995035648,"available":95962836992,"internally_reserved":0,"allocated":4521984,"data_stored":3557944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43084,"kb_used_data":16052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93702068,"statfs":{"total":95995035648,"available":95950917632,"internally_reserved":0,"allocated":16437248,"data_stored":18503128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117069,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58468,"kb_used_data":31436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93686684,"statfs":{"total":95995035648,"available":95935164416,"internally_reserved":0,"allocated":32190464,"data_stored":42482013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313299,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39876,"kb_used_data":8684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93705276,"statfs":{"total":95995035648,"available":95954202624,"internally_reserved":0,"allocated":8892416,"data_stored":11063216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53364,"kb_used_data":22172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93691788,"statfs":{"total":95995035648,"available":95940390912,"internally_reserved":0,"allocated":22704128,"data_stored":28194812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49880,"kb_used_data":14528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93695272,"statfs":{"total":95995035648,"available":95943958528,"internally_reserved":0,"allocated":14876672,"data_stored":16508063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12521472,"data_stored":15124776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20348928,"data_stored":26811525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6393856,"data_stored":9521036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29872128,"data_stored":41124133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14143488,"data_stored":17162196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2035712,"data_stored":2024259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11980800,"data_stored":13683860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16564224,"data_stored":21370579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":19222528,"data_stored":27556357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":58593280,"data_stored":67655509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6950912,"data_stored":7162728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":59707392,"data_stored":79640587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:12:16.850 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:12:16.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 36 2024-09-16T05:12:16.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:16.939 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:16 smithi196 ceph-mon[30802]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:16.939 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2286286717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:16.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:16.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:17.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:17.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:17.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:17.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:12:17.114 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:17.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:16 smithi032 ceph-mon[27388]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:17.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2286286717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:17.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:16 smithi204 ceph-mon[30933]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2286286717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:17.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:17.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:17.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:17.293 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:17.293 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:17.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:17.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:17.423 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:17.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:17.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:17.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:17.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:17.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:17.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:17.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:17.774 INFO:tasks.rados.rados.2.smithi204.stdout:481: done (2 left) 2024-09-16T05:12:17.774 INFO:tasks.rados.rados.2.smithi204.stdout:484: done (1 left) 2024-09-16T05:12:17.774 INFO:tasks.rados.rados.2.smithi204.stdout:485: done (0 left) 2024-09-16T05:12:17.774 INFO:tasks.rados.rados.2.smithi204.stdout:486: snap_remove snap 46 2024-09-16T05:12:17.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:17.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:17.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:17.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:17 smithi032 ceph-mon[27388]: pgmap v459: 37 pgs: 37 active+clean; 202 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:17.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:17 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:17.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2286286717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:17.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:17 smithi032 ceph-mon[27388]: osdmap e205: 12 total, 12 up, 12 in 2024-09-16T05:12:17.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:12:17.903 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:12:17.903 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:12:17.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:17.941 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:17.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:17 smithi196 ceph-mon[30802]: pgmap v459: 37 pgs: 37 active+clean; 202 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:17 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2286286717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:17 smithi196 ceph-mon[30802]: osdmap e205: 12 total, 12 up, 12 in 2024-09-16T05:12:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:12:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:12:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:12:18.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:18.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:18.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:18.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:12:18.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:17 smithi204 ceph-mon[30933]: pgmap v459: 37 pgs: 37 active+clean; 202 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:18.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:17 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:18.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2286286717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:18.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:17 smithi204 ceph-mon[30933]: osdmap e205: 12 total, 12 up, 12 in 2024-09-16T05:12:18.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:12:18.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:12:18.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:12:18.190 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:18.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:18.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:18.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:18.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:18.364 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:18.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:18.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:18.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:18.542 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:18.542 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:18.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:18.563 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:18.563 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:18.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:18.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:18.586 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:18.586 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:18.604 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:18.625 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:18.658 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:18.658 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:18.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:18.679 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:18.679 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:18.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:18.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:18.700 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:18.700 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:18.721 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:18.742 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:18.742 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:18.764 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:18.764 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:18.779 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pg_num to 36 2024-09-16T05:12:18.782 INFO:tasks.rados.rados.2.smithi204.stdout:486: done (0 left) 2024-09-16T05:12:18.782 INFO:tasks.rados.rados.2.smithi204.stdout:487: read oid 22 snap -1 2024-09-16T05:12:18.783 INFO:tasks.rados.rados.2.smithi204.stdout:487: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:12:18.783 INFO:tasks.rados.rados.2.smithi204.stdout:488: read oid 13 snap 29 2024-09-16T05:12:18.783 INFO:tasks.rados.rados.2.smithi204.stdout:488: expect (ObjNum 116 snap 25 seq_num 116) 2024-09-16T05:12:18.783 INFO:tasks.rados.rados.2.smithi204.stdout:489: rollback oid 42 current snap is 54 2024-09-16T05:12:18.784 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 30 2024-09-16T05:12:18.784 INFO:tasks.rados.rados.2.smithi204.stdout:490: snap_remove snap 41 2024-09-16T05:12:18.787 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:18.787 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:18.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:18.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:18.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:18.885 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:12:18.886 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:12:18.886 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:12:18.886 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:12:18.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:18.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:18.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:19.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:19.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:19.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:19.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:18 smithi032 ceph-mon[27388]: osdmap e206: 12 total, 12 up, 12 in 2024-09-16T05:12:19.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1656514264' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-16T05:12:19.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/942882519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:19.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:18 smithi204 ceph-mon[30933]: osdmap e206: 12 total, 12 up, 12 in 2024-09-16T05:12:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1656514264' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-16T05:12:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/942882519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:19.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:18 smithi196 ceph-mon[30802]: osdmap e206: 12 total, 12 up, 12 in 2024-09-16T05:12:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1656514264' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-16T05:12:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/942882519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:19.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:19.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:19.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:19.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:19.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:19.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:19.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:19.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:19.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:12:19.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:19.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:19.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:12:19.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:19.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:19.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:19.790 INFO:tasks.rados.rados.2.smithi204.stdout:490: done (3 left) 2024-09-16T05:12:19.790 INFO:tasks.rados.rados.2.smithi204.stdout:491: read oid 8 snap -1 2024-09-16T05:12:19.790 INFO:tasks.rados.rados.2.smithi204.stdout:491: expect (ObjNum 129 snap 27 seq_num 129) 2024-09-16T05:12:19.790 INFO:tasks.rados.rados.2.smithi204.stdout:492: rollback oid 14 current snap is 54 2024-09-16T05:12:19.790 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 32 2024-09-16T05:12:19.790 INFO:tasks.rados.rados.2.smithi204.stdout:493: write oid 47 current snap is 54 2024-09-16T05:12:19.791 INFO:tasks.rados.rados.2.smithi204.stdout:493: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-16T05:12:19.797 INFO:tasks.rados.rados.2.smithi204.stdout:493: writing smithi20457400-47 from 595014 to 1220490 tid 1 2024-09-16T05:12:19.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:19.803 INFO:tasks.rados.rados.2.smithi204.stdout:493: writing smithi20457400-47 from 1820892 to 2344284 tid 2 2024-09-16T05:12:19.809 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:19.810 INFO:tasks.rados.rados.2.smithi204.stdout:493: writing smithi20457400-47 from 2797915 to 3394391 tid 3 2024-09-16T05:12:19.810 INFO:tasks.rados.rados.2.smithi204.stdout:494: snap_remove snap 48 2024-09-16T05:12:19.823 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:19.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:19.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:19.949 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:19.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:20.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:19 smithi032 ceph-mon[27388]: pgmap v462: 37 pgs: 37 active+clean; 205 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:20.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1656514264' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-16T05:12:20.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:19 smithi032 ceph-mon[27388]: osdmap e207: 12 total, 12 up, 12 in 2024-09-16T05:12:20.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:20.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:20.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:20.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:19 smithi196 ceph-mon[30802]: pgmap v462: 37 pgs: 37 active+clean; 205 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:20.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1656514264' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-16T05:12:20.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:19 smithi196 ceph-mon[30802]: osdmap e207: 12 total, 12 up, 12 in 2024-09-16T05:12:20.122 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:19 smithi204 ceph-mon[30933]: pgmap v462: 37 pgs: 37 active+clean; 205 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1656514264' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-16T05:12:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:19 smithi204 ceph-mon[30933]: osdmap e207: 12 total, 12 up, 12 in 2024-09-16T05:12:20.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:20.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:20.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:20.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:20.301 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:12:20.301 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:12:20.301 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:12:20.301 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:12:20.301 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:12:20.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:20.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:20.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:20.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:20.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:20.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:20.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:20.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:20.660 INFO:tasks.rados.rados.2.smithi204.stdout:494: done (6 left) 2024-09-16T05:12:20.660 INFO:tasks.rados.rados.2.smithi204.stdout:495: read oid 28 snap -1 2024-09-16T05:12:20.660 INFO:tasks.rados.rados.2.smithi204.stdout:495: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T05:12:20.660 INFO:tasks.rados.rados.2.smithi204.stdout:496: snap_create 2024-09-16T05:12:20.661 INFO:tasks.rados.rados.2.smithi204.stdout:487: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T05:12:20.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:20.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:20.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:20.792 INFO:tasks.rados.rados.2.smithi204.stdout:488: expect (ObjNum 116 snap 25 seq_num 116) 2024-09-16T05:12:20.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:20.827 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:20.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:20 smithi032 ceph-mon[27388]: osdmap e208: 12 total, 12 up, 12 in 2024-09-16T05:12:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-16T05:12:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-16T05:12:20.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:20 smithi032 ceph-mon[27388]: osdmap e209: 12 total, 12 up, 12 in 2024-09-16T05:12:20.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:20.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:20.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:20 smithi196 ceph-mon[30802]: osdmap e208: 12 total, 12 up, 12 in 2024-09-16T05:12:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-16T05:12:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-16T05:12:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:20 smithi196 ceph-mon[30802]: osdmap e209: 12 total, 12 up, 12 in 2024-09-16T05:12:21.027 INFO:tasks.rados.rados.2.smithi204.stdout:489: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:12:21.027 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T05:12:21.027 INFO:tasks.rados.rados.2.smithi204.stdout:492: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:12:21.027 INFO:tasks.rados.rados.2.smithi204.stdout:492: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:12:21.027 INFO:tasks.rados.rados.2.smithi204.stdout:492: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:12:21.028 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 253 (ObjNum 122 snap 25 seq_num 122) dirty exists 2024-09-16T05:12:21.028 INFO:tasks.rados.rados.2.smithi204.stdout:491: expect (ObjNum 129 snap 27 seq_num 129) 2024-09-16T05:12:21.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:21.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:12:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:21.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:21.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:21.101 INFO:tasks.rados.rados.2.smithi204.stdout:493: finishing write tid 1 to smithi20457400-47 2024-09-16T05:12:21.101 INFO:tasks.rados.rados.2.smithi204.stdout:493: finishing write tid 2 to smithi20457400-47 2024-09-16T05:12:21.101 INFO:tasks.rados.rados.2.smithi204.stdout:493: finishing write tid 3 to smithi20457400-47 2024-09-16T05:12:21.101 INFO:tasks.rados.rados.2.smithi204.stdout:493: finishing write tid 4 to smithi20457400-47 2024-09-16T05:12:21.101 INFO:tasks.rados.rados.2.smithi204.stdout:493: finishing write tid 5 to smithi20457400-47 2024-09-16T05:12:21.101 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 146 (ObjNum 202 snap 54 seq_num 202) dirty exists 2024-09-16T05:12:21.101 INFO:tasks.rados.rados.2.smithi204.stdout:493: left oid 47 (ObjNum 202 snap 54 seq_num 202) 2024-09-16T05:12:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:20 smithi204 ceph-mon[30933]: osdmap e208: 12 total, 12 up, 12 in 2024-09-16T05:12:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-16T05:12:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-16T05:12:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:20 smithi204 ceph-mon[30933]: osdmap e209: 12 total, 12 up, 12 in 2024-09-16T05:12:21.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:21.199 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:21.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:21.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:21.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:21.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:21.358 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:12:21.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:21.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:21.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:21.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:21.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:21.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:21.657 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:12:21.665 INFO:tasks.rados.rados.2.smithi204.stdout:487: done (7 left) 2024-09-16T05:12:21.665 INFO:tasks.rados.rados.2.smithi204.stdout:488: done (6 left) 2024-09-16T05:12:21.665 INFO:tasks.rados.rados.2.smithi204.stdout:489: done (5 left) 2024-09-16T05:12:21.665 INFO:tasks.rados.rados.2.smithi204.stdout:491: done (4 left) 2024-09-16T05:12:21.665 INFO:tasks.rados.rados.2.smithi204.stdout:492: done (3 left) 2024-09-16T05:12:21.665 INFO:tasks.rados.rados.2.smithi204.stdout:493: done (2 left) 2024-09-16T05:12:21.665 INFO:tasks.rados.rados.2.smithi204.stdout:496: done (1 left) 2024-09-16T05:12:21.666 INFO:tasks.rados.rados.2.smithi204.stdout:497: write (excl) oid 32 current snap is 55 2024-09-16T05:12:21.666 INFO:tasks.rados.rados.2.smithi204.stdout:497: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-16T05:12:21.672 INFO:tasks.rados.rados.2.smithi204.stdout:497: writing smithi20457400-32 from 611821 to 1312546 tid 1 2024-09-16T05:12:21.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:21.681 INFO:tasks.rados.rados.2.smithi204.stdout:497: writing smithi20457400-32 from 1879374 to 2662777 tid 2 2024-09-16T05:12:21.684 INFO:tasks.rados.rados.2.smithi204.stdout:497: writing smithi20457400-32 from 3341691 to 3411821 tid 3 2024-09-16T05:12:21.684 INFO:tasks.rados.rados.2.smithi204.stdout:498: read oid 49 snap -1 2024-09-16T05:12:21.684 INFO:tasks.rados.rados.2.smithi204.stdout:498: expect (ObjNum 92 snap 9 seq_num 92) 2024-09-16T05:12:21.684 INFO:tasks.rados.rados.2.smithi204.stdout:499: write (excl) oid 5 current snap is 55 2024-09-16T05:12:21.684 INFO:tasks.rados.rados.2.smithi204.stdout:499: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-16T05:12:21.692 INFO:tasks.rados.rados.2.smithi204.stdout:499: writing smithi20457400-5 from 628628 to 1404602 tid 1 2024-09-16T05:12:21.698 INFO:tasks.rados.rados.2.smithi204.stdout:499: writing smithi20457400-5 from 1937856 to 2464917 tid 2 2024-09-16T05:12:21.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:21.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:21.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:21.704 INFO:tasks.rados.rados.2.smithi204.stdout:499: writing smithi20457400-5 from 3252761 to 3428628 tid 3 2024-09-16T05:12:21.704 INFO:tasks.rados.rados.2.smithi204.stdout:500: read oid 34 snap -1 2024-09-16T05:12:21.704 INFO:tasks.rados.rados.2.smithi204.stdout:500: expect (ObjNum 106 snap 18 seq_num 106) 2024-09-16T05:12:21.704 INFO:tasks.rados.rados.2.smithi204.stdout:501: delete oid 9 current snap is 55 2024-09-16T05:12:21.704 INFO:tasks.rados.rados.2.smithi204.stdout:495: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T05:12:21.783 INFO:tasks.rados.rados.2.smithi204.stdout:497: finishing write tid 1 to smithi20457400-32 2024-09-16T05:12:21.783 INFO:tasks.rados.rados.2.smithi204.stdout:495: done (5 left) 2024-09-16T05:12:21.783 INFO:tasks.rados.rados.2.smithi204.stdout:501: done (4 left) 2024-09-16T05:12:21.783 INFO:tasks.rados.rados.2.smithi204.stdout:502: read oid 31 snap -1 2024-09-16T05:12:21.783 INFO:tasks.rados.rados.2.smithi204.stdout:502: expect deleted 2024-09-16T05:12:21.783 INFO:tasks.rados.rados.2.smithi204.stdout:503: write (excl) oid 40 current snap is 55 2024-09-16T05:12:21.784 INFO:tasks.rados.rados.2.smithi204.stdout:497: finishing write tid 2 to smithi20457400-32 2024-09-16T05:12:21.784 INFO:tasks.rados.rados.2.smithi204.stdout:503: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-09-16T05:12:21.790 INFO:tasks.rados.rados.2.smithi204.stdout:503: writing smithi20457400-40 from 662242 to 1305067 tid 1 2024-09-16T05:12:21.797 INFO:tasks.rados.rados.2.smithi204.stdout:503: writing smithi20457400-40 from 2054820 to 2585550 tid 2 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:503: writing smithi20457400-40 from 3307607 to 3462242 tid 3 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:504: read oid 36 snap 47 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:504: expect (ObjNum 176 snap 44 seq_num 176) 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:505: delete oid 44 current snap is 55 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:497: finishing write tid 3 to smithi20457400-32 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:497: finishing write tid 4 to smithi20457400-32 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:497: finishing write tid 5 to smithi20457400-32 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 281 (ObjNum 203 snap 55 seq_num 203) dirty exists 2024-09-16T05:12:21.802 INFO:tasks.rados.rados.2.smithi204.stdout:497: left oid 32 (ObjNum 203 snap 55 seq_num 203) 2024-09-16T05:12:21.803 INFO:tasks.rados.rados.2.smithi204.stdout:498: expect (ObjNum 92 snap 9 seq_num 92) 2024-09-16T05:12:21.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:21.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:21.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:21.868 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:12:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:12:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:12:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:12:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1 2.8 2:12e92f6c:::smithi20457400-1:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T05:12:21.868 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:42.993477+0000", 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.872415973000003, 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041395950000000003, 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.993477+0000", 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.993477+0000", 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.993479+0000", 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6470000000000001e-06 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994189+0000", 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00071080099999999995 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994203+0000", 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.375e-05 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994211+0000", 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9140000000000008e-06 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994291+0000", 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9931000000000004e-05 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994390+0000", 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9234999999999994e-05 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994507+0000", 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011694200000000001 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997077+0000", 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025702979999999999 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997514+0000", 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00043703900000000001 2024-09-16T05:12:21.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997568+0000", 2024-09-16T05:12:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3687000000000003e-05 2024-09-16T05:12:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997616+0000", 2024-09-16T05:12:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8350999999999998e-05 2024-09-16T05:12:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.875 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2 2.8 2:12e92f6c:::smithi20457400-1:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:42.994842+0000", 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.871050486000001, 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028772350000000001, 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.876 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994842+0000", 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994842+0000", 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.994844+0000", 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.407e-06 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995544+0000", 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00070042100000000001 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995559+0000", 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5140000000000001e-05 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995570+0000", 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0453e-05 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995635+0000", 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5143999999999998e-05 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995713+0000", 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8573999999999997e-05 2024-09-16T05:12:21.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995835+0000", 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121388 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997080+0000", 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001245468 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997656+0000", 2024-09-16T05:12:21.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057615900000000002 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997691+0000", 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4520000000000002e-05 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997720+0000", 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8561e-05 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3 2.8 2:12e92f6c:::smithi20457400-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:42.995596+0000", 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.870296549000003, 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022823549999999998, 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.882 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995596+0000", 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995596+0000", 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995597+0000", 2024-09-16T05:12:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7300000000000004e-07 2024-09-16T05:12:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995602+0000", 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7360000000000001e-06 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995612+0000", 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5389999999999996e-06 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995618+0000", 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5889999999999999e-06 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996445+0000", 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082694400000000003 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996496+0000", 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0559000000000002e-05 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996638+0000", 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014271800000000001 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997143+0000", 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050490099999999998 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997771+0000", 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062767599999999997 2024-09-16T05:12:21.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997827+0000", 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6069999999999997e-05 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997879+0000", 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1650000000000002e-05 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4 2.8 2:12e92f6c:::smithi20457400-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:42.995645+0000", 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.870248149999995, 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023934989999999999, 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.890 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995645+0000", 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995645+0000", 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995645+0000", 2024-09-16T05:12:21.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7899999999999997e-07 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995649+0000", 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1999999999999999e-06 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995658+0000", 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7040000000000006e-06 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.995665+0000", 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4470000000000003e-06 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996856+0000", 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011909139999999999 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.996933+0000", 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6829999999999995e-05 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997932+0000", 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099908099999999997 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.997979+0000", 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7485999999999999e-05 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:42.998038+0000", 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9058000000000003e-05 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.894 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:21 2.0 2:06458cf1:::smithi20457400-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.047722+0000", 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.818170609000006, 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065493820000000003, 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.895 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 21 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.047722+0000", 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.047722+0000", 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.047723+0000", 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.232e-06 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048355+0000", 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063158099999999998 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048369+0000", 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.418e-05 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048377+0000", 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6690000000000006e-06 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048466+0000", 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8831000000000004e-05 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048543+0000", 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6995000000000004e-05 2024-09-16T05:12:21.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.048632+0000", 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9104000000000006e-05 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.050003+0000", 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001370916 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054207+0000", 2024-09-16T05:12:21.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042047680000000002 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054241+0000", 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3565000000000001e-05 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054272+0000", 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0540999999999998e-05 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.900 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:22 2.0 2:06458cf1:::smithi20457400-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052091+0000", 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.813801850999994, 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002289555, 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 22 2024-09-16T05:12:21.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052091+0000", 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052091+0000", 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052092+0000", 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6099999999999999e-07 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052159+0000", 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7067e-05 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052169+0000", 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0241000000000001e-05 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052174+0000", 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8180000000000003e-06 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052260+0000", 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5952000000000006e-05 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052315+0000", 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4678000000000001e-05 2024-09-16T05:12:21.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052400+0000", 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4915999999999998e-05 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053195+0000", 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079554599999999995 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054316+0000", 2024-09-16T05:12:21.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011208749999999999 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054352+0000", 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5837999999999998e-05 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.054381+0000", 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8663e-05 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.906 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:23 2.0 2:06458cf1:::smithi20457400-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.813707304999994, 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103108687, 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 23 2024-09-16T05:12:21.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3599999999999999e-07 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052188+0000", 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4169999999999999e-06 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052214+0000", 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5327e-05 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052217+0000", 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1810000000000001e-06 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052730+0000", 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051270700000000005 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052818+0000", 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8641999999999996e-05 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052937+0000", 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119009 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053259+0000", 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032151800000000002 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155207+0000", 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101948155 2024-09-16T05:12:21.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155248+0000", 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1115999999999997e-05 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155294+0000", 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6179e-05 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:24 2.0 2:06458cf1:::smithi20457400-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.813671244000005, 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103238884, 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 24 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:12:21.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8799999999999998e-07 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052223+0000", 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2950000000000001e-06 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052227+0000", 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5240000000000001e-06 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052230+0000", 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2160000000000002e-06 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053127+0000", 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089734199999999997 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053182+0000", 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4935999999999997e-05 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155331+0000", 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.102149077 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155389+0000", 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7691e-05 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155460+0000", 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1514999999999995e-05 2024-09-16T05:12:21.917 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:105 2.8 2:1030fb5b:::smithi20457400-27:head [write 436982~705533,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.301386+0000", 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.564506570999995, 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030539830000000001, 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 105 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301386+0000", 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301386+0000", 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301389+0000", 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1370000000000002e-06 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301943+0000", 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055360100000000003 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301963+0000", 2024-09-16T05:12:21.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0398999999999999e-05 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.301971+0000", 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0700000000000007e-06 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.302040+0000", 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8230000000000002e-05 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.302155+0000", 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114864 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.302262+0000", 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010745 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.303571+0000", 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013085670000000001 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.304372+0000", 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080121800000000003 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.304407+0000", 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4842000000000003e-05 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.304440+0000", 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3605000000000001e-05 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.923 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:106 2.8 2:1030fb5b:::smithi20457400-27:head [write 1855120~581862] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.308375+0000", 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.557517938000004, 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002390011, 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.924 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 106 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308375+0000", 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308375+0000", 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308376+0000", 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2619999999999999e-06 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308837+0000", 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046054599999999999 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308851+0000", 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3913999999999999e-05 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308857+0000", 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1979999999999998e-06 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308912+0000", 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.541e-05 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308968+0000", 2024-09-16T05:12:21.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5939000000000001e-05 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309054+0000", 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6149000000000005e-05 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309844+0000", 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00078919799999999996 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310692+0000", 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084889399999999999 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310728+0000", 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5842e-05 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310765+0000", 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6659000000000003e-05 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.929 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:108 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.308916+0000", 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.556977000000003, 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021015669999999999, 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.930 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308916+0000", 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308916+0000", 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308916+0000", 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3399999999999999e-07 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308920+0000", 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.343e-06 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308928+0000", 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9459999999999998e-06 2024-09-16T05:12:21.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.308934+0000", 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1310000000000003e-06 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309871+0000", 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00093735499999999996 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.309921+0000", 2024-09-16T05:12:21.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9323999999999997e-05 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310907+0000", 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0009860750000000001 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.310954+0000", 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7760999999999999e-05 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.311017+0000", 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2997999999999996e-05 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:391 2.8 2:1030fb5b:::smithi20457400-27:head [copy-from ver 13] snapc 8=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 86.229105923999995, 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021838053999999999, 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.935 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 391 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636789+0000", 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3829999999999999e-06 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636801+0000", 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1459e-05 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636822+0000", 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1719e-05 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636833+0000", 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0366e-05 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636912+0000", 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9060999999999994e-05 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636918+0000", 2024-09-16T05:12:21.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0900000000000001e-06 2024-09-16T05:12:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638224+0000", 2024-09-16T05:12:21.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00130613 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638314+0000", 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9771000000000005e-05 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.652220+0000", 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013906112 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.656458+0000", 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004237673 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658553+0000", 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002095609 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658592+0000", 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8627000000000001e-05 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:21.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:21 smithi032 ceph-mon[27388]: pgmap v465: 37 pgs: 37 active+clean; 205 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 836 KiB/s wr, 5 op/s 2024-09-16T05:12:21.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4058845480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:21.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4058845480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:21.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:21 smithi032 ceph-mon[27388]: osdmap e210: 12 total, 12 up, 12 in 2024-09-16T05:12:21.944 INFO:tasks.rados.rados.2.smithi204.stdout:499: finishing write tid 1 to smithi20457400-5 2024-09-16T05:12:21.944 INFO:tasks.rados.rados.2.smithi204.stdout:499: finishing write tid 2 to smithi20457400-5 2024-09-16T05:12:21.944 INFO:tasks.rados.rados.2.smithi204.stdout:499: finishing write tid 3 to smithi20457400-5 2024-09-16T05:12:21.944 INFO:tasks.rados.rados.2.smithi204.stdout:499: finishing write tid 4 to smithi20457400-5 2024-09-16T05:12:21.945 INFO:tasks.rados.rados.2.smithi204.stdout:499: finishing write tid 5 to smithi20457400-5 2024-09-16T05:12:21.945 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 154 (ObjNum 204 snap 55 seq_num 204) dirty exists 2024-09-16T05:12:21.945 INFO:tasks.rados.rados.2.smithi204.stdout:499: left oid 5 (ObjNum 204 snap 55 seq_num 204) 2024-09-16T05:12:21.945 INFO:tasks.rados.rados.2.smithi204.stdout:500: expect (ObjNum 106 snap 18 seq_num 106) 2024-09-16T05:12:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658625+0000", 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3053999999999999e-05 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:392 2.8 2:1030fb5b:::smithi20457400-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 86.229034702000007, 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019221338000000001, 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.946 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 392 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636859+0000", 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0699999999999998e-07 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636861+0000", 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5129999999999999e-06 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636867+0000", 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6030000000000004e-06 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636870+0000", 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.551e-06 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636972+0000", 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010187999999999999 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636978+0000", 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6269999999999997e-06 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638477+0000", 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014995049999999999 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638531+0000", 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3387999999999998e-05 2024-09-16T05:12:21.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.655845+0000", 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017313963000000002 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.655956+0000", 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011097799999999999 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.656079+0000", 2024-09-16T05:12:21.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012392300000000001 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:889 2.8 2:12e92f6c:::smithi20457400-1:head [delete] snapc 23=[23,22,21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+full_force+supports_pool_eio e170)", 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:41.660516+0000", 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.205376690999998, 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028080850000000001, 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.952 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 889 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.660516+0000", 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.660516+0000", 2024-09-16T05:12:21.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.660520+0000", 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5930000000000002e-06 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.660532+0000", 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2310000000000001e-05 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.660567+0000", 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4773000000000001e-05 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.660590+0000", 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3553e-05 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.660790+0000", 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000199686 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:21.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.660802+0000", 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1549000000000001e-05 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.662024+0000", 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012225820000000001 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.662092+0000", 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8017999999999998e-05 2024-09-16T05:12:21.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.662232+0000", 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139498 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.662887+0000", 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000655469 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.663251+0000", 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036379099999999997 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.663292+0000", 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0877000000000002e-05 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:41.663324+0000", 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2385999999999998e-05 2024-09-16T05:12:21.958 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:965 2.0 2:06458cf1:::smithi20457400-6:head [delete] snapc 28=[28,27,25,24,23,22,21,20,1f,15,f] ondisk+write+known_if_redirected+full_force+supports_pool_eio e179)", 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:50.809357+0000", 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 31.056536120000001, 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032163740000000001, 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.959 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 965 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.809357+0000", 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.809357+0000", 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.809359+0000", 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1299999999999999e-06 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.809369+0000", 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0380000000000001e-05 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.809400+0000", 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1223000000000002e-05 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.809426+0000", 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5309000000000001e-05 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.809606+0000", 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018021000000000001 2024-09-16T05:12:21.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.809618+0000", 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.223e-05 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.811251+0000", 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001632613 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.811322+0000", 2024-09-16T05:12:21.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1230999999999996e-05 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.811485+0000", 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016294500000000001 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.812243+0000", 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000758069 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.812465+0000", 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000222401 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.812525+0000", 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9932999999999997e-05 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:50.812573+0000", 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7700000000000001e-05 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:999 2.9 e180/107 2:96dc17ef:::smithi20457400-10:head v 180'54, mlcod=174'50)", 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:51.679499+0000", 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 30.186393766999998, 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024945079999999999, 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.966 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.679499+0000", 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.679499+0000", 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.679499+0000", 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.79e-07 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.680676+0000", 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001177176 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.680704+0000", 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7651e-05 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.680715+0000", 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0808999999999999e-05 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.680777+0000", 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2383000000000002e-05 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.680805+0000", 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7883999999999999e-05 2024-09-16T05:12:21.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.681940+0000", 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011349070000000001 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:51.681994+0000", 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3519e-05 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1184 2.8 2:1030fb5b:::smithi20457400-27:head [sparse-read 0~1769472,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:03.995880+0000", 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 17.870012736, 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020507519999999999, 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1184 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995880+0000", 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995880+0000", 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995883+0000", 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0000000000000001e-06 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995895+0000", 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2006e-05 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995927+0000", 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1612999999999997e-05 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995948+0000", 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1573000000000002e-05 2024-09-16T05:12:21.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996093+0000", 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014503699999999999 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996252+0000", 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000158527 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.997931+0000", 2024-09-16T05:12:21.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016789960000000001 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1185 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~2823543,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:03.995994+0000", 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 17.869898331999998, 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003154858, 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.975 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1185 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995994+0000", 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995994+0000", 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.995996+0000", 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5030000000000001e-06 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996006+0000", 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0477e-05 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996038+0000", 2024-09-16T05:12:21.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1322999999999998e-05 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996055+0000", 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6866000000000001e-05 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.997985+0000", 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019302080000000001 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.998049+0000", 2024-09-16T05:12:21.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4382999999999997e-05 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.999149+0000", 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011000980000000001 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1186 2.8 2:1030fb5b:::smithi20457400-27:head [sparse-read 0~1769472] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:03.996087+0000", 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 17.869805614000001, 2024-09-16T05:12:21.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034276279999999998, 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1186 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996087+0000", 2024-09-16T05:12:21.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996087+0000", 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996089+0000", 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.31e-06 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996094+0000", 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.947e-06 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996114+0000", 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9069999999999999e-05 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.996127+0000", 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3672999999999999e-05 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.999194+0000", 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030671779999999998 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.999308+0000", 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011372200000000001 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.999515+0000", 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020672799999999999 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1222 2.8 2:1030fb5b:::smithi20457400-27:head [stat,delete] snapc 31=[31,30,2f,2e,2d,2c,2b,2a,29,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:05.695394+0000", 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.170498872, 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034040770000000001, 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1222 2024-09-16T05:12:21.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.695394+0000", 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.695394+0000", 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:12:21.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.695396+0000", 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.685e-06 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.695402+0000", 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3829999999999997e-06 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.695418+0000", 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5651e-05 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.695425+0000", 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0790000000000003e-06 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.695478+0000", 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3650000000000003e-05 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.695483+0000", 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6460000000000001e-06 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.696563+0000", 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010797859999999999 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.696611+0000", 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8211999999999999e-05 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.696709+0000", 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8338999999999995e-05 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.697311+0000", 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00060137899999999998 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.698719+0000", 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001408147 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.698759+0000", 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0151999999999998e-05 2024-09-16T05:12:21.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:05.698798+0000", 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8967999999999997e-05 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:12:21.990 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:12:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:21.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:21.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:22.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:12:22.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:22.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:22.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:22.134 INFO:tasks.rados.rados.2.smithi204.stdout:503: finishing write tid 1 to smithi20457400-40 2024-09-16T05:12:22.134 INFO:tasks.rados.rados.2.smithi204.stdout:503: finishing write tid 2 to smithi20457400-40 2024-09-16T05:12:22.134 INFO:tasks.rados.rados.2.smithi204.stdout:503: finishing write tid 3 to smithi20457400-40 2024-09-16T05:12:22.134 INFO:tasks.rados.rados.2.smithi204.stdout:503: finishing write tid 4 to smithi20457400-40 2024-09-16T05:12:22.134 INFO:tasks.rados.rados.2.smithi204.stdout:503: finishing write tid 5 to smithi20457400-40 2024-09-16T05:12:22.134 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 163 (ObjNum 206 snap 55 seq_num 206) dirty exists 2024-09-16T05:12:22.134 INFO:tasks.rados.rados.2.smithi204.stdout:503: left oid 40 (ObjNum 206 snap 55 seq_num 206) 2024-09-16T05:12:22.134 INFO:tasks.rados.rados.2.smithi204.stdout:504: expect (ObjNum 176 snap 44 seq_num 176) 2024-09-16T05:12:22.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:21 smithi196 ceph-mon[30802]: pgmap v465: 37 pgs: 37 active+clean; 205 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 836 KiB/s wr, 5 op/s 2024-09-16T05:12:22.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4058845480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:22.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4058845480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:22.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:21 smithi196 ceph-mon[30802]: osdmap e210: 12 total, 12 up, 12 in 2024-09-16T05:12:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:21 smithi204 ceph-mon[30933]: pgmap v465: 37 pgs: 37 active+clean; 205 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 836 KiB/s wr, 5 op/s 2024-09-16T05:12:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4058845480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4058845480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:21 smithi204 ceph-mon[30933]: osdmap e210: 12 total, 12 up, 12 in 2024-09-16T05:12:22.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:22.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:22.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:22.306 INFO:tasks.rados.rados.2.smithi204.stdout:497: done (7 left) 2024-09-16T05:12:22.307 INFO:tasks.rados.rados.2.smithi204.stdout:498: done (6 left) 2024-09-16T05:12:22.307 INFO:tasks.rados.rados.2.smithi204.stdout:499: done (5 left) 2024-09-16T05:12:22.307 INFO:tasks.rados.rados.2.smithi204.stdout:500: done (4 left) 2024-09-16T05:12:22.307 INFO:tasks.rados.rados.2.smithi204.stdout:503: done (3 left) 2024-09-16T05:12:22.307 INFO:tasks.rados.rados.2.smithi204.stdout:504: done (2 left) 2024-09-16T05:12:22.307 INFO:tasks.rados.rados.2.smithi204.stdout:505: done (1 left) 2024-09-16T05:12:22.307 INFO:tasks.rados.rados.2.smithi204.stdout:506: write (excl) oid 48 current snap is 55 2024-09-16T05:12:22.307 INFO:tasks.rados.rados.2.smithi204.stdout:506: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-09-16T05:12:22.314 INFO:tasks.rados.rados.2.smithi204.stdout:506: writing smithi20457400-48 from 695856 to 1489179 tid 1 2024-09-16T05:12:22.320 INFO:tasks.rados.rados.2.smithi204.stdout:506: writing smithi20457400-48 from 2171784 to 2706183 tid 2 2024-09-16T05:12:22.325 INFO:tasks.rados.rados.2.smithi204.stdout:506: writing smithi20457400-48 from 3362453 to 3495856 tid 3 2024-09-16T05:12:22.325 INFO:tasks.rados.rados.2.smithi204.stdout:507: read oid 21 snap 53 2024-09-16T05:12:22.325 INFO:tasks.rados.rados.2.smithi204.stdout:507: expect (ObjNum 182 snap 46 seq_num 182) 2024-09-16T05:12:22.325 INFO:tasks.rados.rados.2.smithi204.stdout:508: delete oid 34 current snap is 55 2024-09-16T05:12:22.325 INFO:tasks.rados.rados.2.smithi204.stdout:506: finishing write tid 1 to smithi20457400-48 2024-09-16T05:12:22.325 INFO:tasks.rados.rados.2.smithi204.stdout:506: finishing write tid 2 to smithi20457400-48 2024-09-16T05:12:22.327 INFO:tasks.rados.rados.2.smithi204.stdout:506: finishing write tid 3 to smithi20457400-48 2024-09-16T05:12:22.327 INFO:tasks.rados.rados.2.smithi204.stdout:508: done (3 left) 2024-09-16T05:12:22.327 INFO:tasks.rados.rados.2.smithi204.stdout:509: rollback oid 49 current snap is 55 2024-09-16T05:12:22.327 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 54 2024-09-16T05:12:22.327 INFO:tasks.rados.rados.2.smithi204.stdout:506: finishing write tid 4 to smithi20457400-48 2024-09-16T05:12:22.328 INFO:tasks.rados.rados.2.smithi204.stdout:510: rollback oid 5 current snap is 55 2024-09-16T05:12:22.328 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 5 to 29 2024-09-16T05:12:22.328 INFO:tasks.rados.rados.2.smithi204.stdout:511: write (excl) oid 22 current snap is 55 2024-09-16T05:12:22.328 INFO:tasks.rados.rados.2.smithi204.stdout:511: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-09-16T05:12:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:22.334 INFO:tasks.rados.rados.2.smithi204.stdout:511: writing smithi20457400-22 from 729470 to 1273291 tid 1 2024-09-16T05:12:22.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:22.340 INFO:tasks.rados.rados.2.smithi204.stdout:511: writing smithi20457400-22 from 1772395 to 2310463 tid 2 2024-09-16T05:12:22.345 INFO:tasks.rados.rados.2.smithi204.stdout:511: writing smithi20457400-22 from 2900946 to 3303951 tid 3 2024-09-16T05:12:22.345 INFO:tasks.rados.rados.2.smithi204.stdout:512: copy_from oid 13 from oid 32 current snap is 55 2024-09-16T05:12:22.345 INFO:tasks.rados.rados.2.smithi204.stdout:506: finishing write tid 5 to smithi20457400-48 2024-09-16T05:12:22.346 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 183 (ObjNum 208 snap 55 seq_num 208) dirty exists 2024-09-16T05:12:22.346 INFO:tasks.rados.rados.2.smithi204.stdout:506: left oid 48 (ObjNum 208 snap 55 seq_num 208) 2024-09-16T05:12:22.346 INFO:tasks.rados.rados.2.smithi204.stdout:507: expect (ObjNum 182 snap 46 seq_num 182) 2024-09-16T05:12:22.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:22.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:22.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:22.482 INFO:tasks.rados.rados.2.smithi204.stdout:509: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:12:22.482 INFO:tasks.rados.rados.2.smithi204.stdout:509: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:12:22.482 INFO:tasks.rados.rados.2.smithi204.stdout:509: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:12:22.482 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 153 (ObjNum 92 snap 9 seq_num 92) dirty exists 2024-09-16T05:12:22.482 INFO:tasks.rados.rados.2.smithi204.stdout:510: finishing rollback tid 0 to smithi20457400-5 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:510: finishing rollback tid 1 to smithi20457400-5 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:510: finishing rollback tid 2 to smithi20457400-5 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 157 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:511: finishing write tid 1 to smithi20457400-22 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:511: finishing write tid 2 to smithi20457400-22 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:506: done (6 left) 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:507: done (5 left) 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:509: done (4 left) 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:510: done (3 left) 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:513: copy_from oid 27 from oid 17 current snap is 55 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:511: finishing write tid 3 to smithi20457400-22 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:511: finishing write tid 4 to smithi20457400-22 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:511: finishing write tid 5 to smithi20457400-22 2024-09-16T05:12:22.483 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 188 (ObjNum 210 snap 55 seq_num 210) dirty exists 2024-09-16T05:12:22.484 INFO:tasks.rados.rados.2.smithi204.stdout:511: left oid 22 (ObjNum 210 snap 55 seq_num 210) 2024-09-16T05:12:22.484 INFO:tasks.rados.rados.2.smithi204.stdout:512: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:12:22.484 INFO:tasks.rados.rados.2.smithi204.stdout:512: finishing copy_from to smithi20457400-13 2024-09-16T05:12:22.484 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 281 (ObjNum 203 snap 55 seq_num 203) dirty exists 2024-09-16T05:12:22.484 INFO:tasks.rados.rados.2.smithi204.stdout:511: done (3 left) 2024-09-16T05:12:22.484 INFO:tasks.rados.rados.2.smithi204.stdout:512: done (2 left) 2024-09-16T05:12:22.484 INFO:tasks.rados.rados.2.smithi204.stdout:514: delete oid 41 current snap is 55 2024-09-16T05:12:22.485 INFO:tasks.rados.rados.2.smithi204.stdout:514: done (2 left) 2024-09-16T05:12:22.485 INFO:tasks.rados.rados.2.smithi204.stdout:515: read oid 7 snap -1 2024-09-16T05:12:22.485 INFO:tasks.rados.rados.2.smithi204.stdout:515: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:12:22.485 INFO:tasks.rados.rados.2.smithi204.stdout:516: write oid 22 current snap is 55 2024-09-16T05:12:22.485 INFO:tasks.rados.rados.2.smithi204.stdout:516: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-16T05:12:22.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:22.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:22.492 INFO:tasks.rados.rados.2.smithi204.stdout:516: writing smithi20457400-22 from 763084 to 1457403 tid 1 2024-09-16T05:12:22.497 INFO:tasks.rados.rados.2.smithi204.stdout:516: writing smithi20457400-22 from 1889359 to 2431096 tid 2 2024-09-16T05:12:22.503 INFO:tasks.rados.rados.2.smithi204.stdout:516: writing smithi20457400-22 from 2955792 to 3497694 tid 3 2024-09-16T05:12:22.503 INFO:tasks.rados.rados.2.smithi204.stdout:517: delete oid 47 current snap is 55 2024-09-16T05:12:22.503 INFO:tasks.rados.rados.2.smithi204.stdout:515: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:12:22.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:22.590 INFO:tasks.rados.rados.2.smithi204.stdout:513: finishing copy_from to smithi20457400-27 2024-09-16T05:12:22.590 INFO:tasks.rados.rados.2.smithi204.stdout:513: got expected ENOENT (src dne) 2024-09-16T05:12:22.590 INFO:tasks.rados.rados.2.smithi204.stdout:513: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:12:22.590 INFO:tasks.rados.rados.2.smithi204.stdout:516: finishing write tid 1 to smithi20457400-22 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:516: finishing write tid 2 to smithi20457400-22 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:513: done (4 left) 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:515: done (3 left) 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:517: done (2 left) 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:518: read oid 36 snap -1 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:518: expect (ObjNum 199 snap 53 seq_num 199) 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:519: copy_from oid 18 from oid 30 current snap is 55 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:516: finishing write tid 3 to smithi20457400-22 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:516: finishing write tid 4 to smithi20457400-22 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:516: finishing write tid 5 to smithi20457400-22 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 192 (ObjNum 212 snap 55 seq_num 212) dirty exists 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:516: left oid 22 (ObjNum 212 snap 55 seq_num 212) 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:516: done (3 left) 2024-09-16T05:12:22.591 INFO:tasks.rados.rados.2.smithi204.stdout:520: rollback oid 32 current snap is 55 2024-09-16T05:12:22.592 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 53 2024-09-16T05:12:22.592 INFO:tasks.rados.rados.2.smithi204.stdout:521: delete oid 42 current snap is 55 2024-09-16T05:12:22.594 INFO:tasks.rados.rados.2.smithi204.stdout:520: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:12:22.595 INFO:tasks.rados.rados.2.smithi204.stdout:502: done (4 left) 2024-09-16T05:12:22.595 INFO:tasks.rados.rados.2.smithi204.stdout:521: done (3 left) 2024-09-16T05:12:22.595 INFO:tasks.rados.rados.2.smithi204.stdout:522: rollback oid 20 current snap is 55 2024-09-16T05:12:22.595 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 20 to 51 2024-09-16T05:12:22.595 INFO:tasks.rados.rados.2.smithi204.stdout:523: copy_from oid 11 from oid 17 current snap is 55 2024-09-16T05:12:22.595 INFO:tasks.rados.rados.2.smithi204.stdout:524: copy_from oid 31 from oid 15 current snap is 55 2024-09-16T05:12:22.595 INFO:tasks.rados.rados.2.smithi204.stdout:525: snap_create 2024-09-16T05:12:22.595 INFO:tasks.rados.rados.2.smithi204.stdout:518: expect (ObjNum 199 snap 53 seq_num 199) 2024-09-16T05:12:22.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:22.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:22.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:22.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:22.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:22.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:520: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:12:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:520: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:12:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 285 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:12:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:522: finishing rollback tid 0 to smithi20457400-20 2024-09-16T05:12:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:522: finishing rollback tid 1 to smithi20457400-20 2024-09-16T05:12:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:522: finishing rollback tid 2 to smithi20457400-20 2024-09-16T05:12:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 326 (ObjNum 179 snap 46 seq_num 179) dirty exists 2024-09-16T05:12:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:523: finishing copy_from to smithi20457400-11 2024-09-16T05:12:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:523: got expected ENOENT (src dne) 2024-09-16T05:12:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:523: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:12:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:519: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:12:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:519: finishing copy_from to smithi20457400-18 2024-09-16T05:12:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 207 (ObjNum 177 snap 45 seq_num 177) dirty exists 2024-09-16T05:12:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:524: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:12:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:524: finishing copy_from to smithi20457400-31 2024-09-16T05:12:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 20 (ObjNum 165 snap 40 seq_num 165) dirty exists 2024-09-16T05:12:22.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:518: done (6 left) 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:519: done (5 left) 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:520: done (4 left) 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:522: done (3 left) 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:523: done (2 left) 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:524: done (1 left) 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:525: done (0 left) 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:526: read oid 21 snap 54 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:526: expect (ObjNum 182 snap 46 seq_num 182) 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:527: read oid 43 snap -1 2024-09-16T05:12:22.811 INFO:tasks.rados.rados.2.smithi204.stdout:527: expect (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:12:22.812 INFO:tasks.rados.rados.2.smithi204.stdout:528: read oid 17 snap -1 2024-09-16T05:12:22.812 INFO:tasks.rados.rados.2.smithi204.stdout:528: expect deleted 2024-09-16T05:12:22.812 INFO:tasks.rados.rados.2.smithi204.stdout:529: write (excl) oid 27 current snap is 56 2024-09-16T05:12:22.812 INFO:tasks.rados.rados.2.smithi204.stdout:529: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-16T05:12:22.812 INFO:tasks.rados.rados.2.smithi204.stdout:529: writing smithi20457400-27 from 413505 to 1049924 tid 1 2024-09-16T05:12:22.819 INFO:tasks.rados.rados.2.smithi204.stdout:529: writing smithi20457400-27 from 1664805 to 2353869 tid 2 2024-09-16T05:12:22.826 INFO:tasks.rados.rados.2.smithi204.stdout:529: writing smithi20457400-27 from 3038061 to 3530132 tid 3 2024-09-16T05:12:22.826 INFO:tasks.rados.rados.2.smithi204.stdout:530: copy_from oid 18 from oid 2 current snap is 56 2024-09-16T05:12:22.826 INFO:tasks.rados.rados.2.smithi204.stdout:526: expect (ObjNum 182 snap 46 seq_num 182) 2024-09-16T05:12:22.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:22.831 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:22.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:22.852 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:22.852 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:22.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:22.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:22.872 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:22.873 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:22.893 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:22.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:22.914 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:22.936 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:22.936 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:22.957 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:22.958 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:22.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:22.967 INFO:tasks.rados.rados.2.smithi204.stdout:529: finishing write tid 1 to smithi20457400-27 2024-09-16T05:12:22.967 INFO:tasks.rados.rados.2.smithi204.stdout:529: finishing write tid 2 to smithi20457400-27 2024-09-16T05:12:22.967 INFO:tasks.rados.rados.2.smithi204.stdout:526: done (4 left) 2024-09-16T05:12:22.967 INFO:tasks.rados.rados.2.smithi204.stdout:528: done (3 left) 2024-09-16T05:12:22.967 INFO:tasks.rados.rados.2.smithi204.stdout:531: write oid 49 current snap is 56 2024-09-16T05:12:22.967 INFO:tasks.rados.rados.2.smithi204.stdout:529: finishing write tid 3 to smithi20457400-27 2024-09-16T05:12:22.967 INFO:tasks.rados.rados.2.smithi204.stdout:531: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-09-16T05:12:22.973 INFO:tasks.rados.rados.2.smithi204.stdout:531: writing smithi20457400-49 from 430312 to 1141980 tid 1 2024-09-16T05:12:22.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:22.979 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:22.979 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-16T05:12:22.979 INFO:tasks.rados.rados.2.smithi204.stdout:531: writing smithi20457400-49 from 1723287 to 2272362 tid 2 2024-09-16T05:12:22.986 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:22.987 INFO:tasks.rados.rados.2.smithi204.stdout:531: writing smithi20457400-49 from 2781837 to 3485180 tid 3 2024-09-16T05:12:22.987 INFO:tasks.rados.rados.2.smithi204.stdout:532: delete oid 5 current snap is 56 2024-09-16T05:12:22.987 INFO:tasks.rados.rados.2.smithi204.stdout:529: finishing write tid 4 to smithi20457400-27 2024-09-16T05:12:22.987 INFO:tasks.rados.rados.2.smithi204.stdout:529: finishing write tid 5 to smithi20457400-27 2024-09-16T05:12:22.987 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 79 (ObjNum 215 snap 56 seq_num 215) dirty exists 2024-09-16T05:12:22.987 INFO:tasks.rados.rados.2.smithi204.stdout:529: left oid 27 (ObjNum 215 snap 56 seq_num 215) 2024-09-16T05:12:22.987 INFO:tasks.rados.rados.2.smithi204.stdout:530: finishing copy_from to smithi20457400-18 2024-09-16T05:12:22.987 INFO:tasks.rados.rados.2.smithi204.stdout:530: got expected ENOENT (src dne) 2024-09-16T05:12:22.988 INFO:tasks.rados.rados.2.smithi204.stdout:530: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:12:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:22.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:22.989 INFO:tasks.rados.rados.2.smithi204.stdout:529: done (4 left) 2024-09-16T05:12:22.989 INFO:tasks.rados.rados.2.smithi204.stdout:530: done (3 left) 2024-09-16T05:12:22.989 INFO:tasks.rados.rados.2.smithi204.stdout:532: done (2 left) 2024-09-16T05:12:22.989 INFO:tasks.rados.rados.2.smithi204.stdout:533: delete oid 13 current snap is 56 2024-09-16T05:12:22.990 INFO:tasks.rados.rados.2.smithi204.stdout:533: done (2 left) 2024-09-16T05:12:22.990 INFO:tasks.rados.rados.2.smithi204.stdout:534: rollback oid 38 current snap is 56 2024-09-16T05:12:22.990 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 38 to 54 2024-09-16T05:12:22.990 INFO:tasks.rados.rados.2.smithi204.stdout:535: read oid 4 snap -1 2024-09-16T05:12:22.990 INFO:tasks.rados.rados.2.smithi204.stdout:535: expect (ObjNum 181 snap 46 seq_num 181) 2024-09-16T05:12:22.990 INFO:tasks.rados.rados.2.smithi204.stdout:536: rollback oid 30 current snap is 56 2024-09-16T05:12:22.990 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 30 to 30 2024-09-16T05:12:22.990 INFO:tasks.rados.rados.2.smithi204.stdout:537: snap_create 2024-09-16T05:12:22.991 INFO:tasks.rados.rados.2.smithi204.stdout:534: finishing rollback tid 1 to smithi20457400-38 2024-09-16T05:12:22.991 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 252 (ObjNum 49 snap 0 seq_num 4282349344) dirty dne 2024-09-16T05:12:23.000 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:23.000 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:23.022 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:23.040 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:23.043 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:23.065 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:23.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:23.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:23.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:23.161 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-16T05:12:23.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:23.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:23.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:23.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:23.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-16T05:12:23.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:23.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:23.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:23.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:23.484 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:23.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:23.499 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:23.500 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:23.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:23.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:23.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:23.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:23.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:23.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:23.803 INFO:tasks.rados.rados.2.smithi204.stdout:534: done (5 left) 2024-09-16T05:12:23.803 INFO:tasks.rados.rados.2.smithi204.stdout:537: done (4 left) 2024-09-16T05:12:23.803 INFO:tasks.rados.rados.2.smithi204.stdout:538: write oid 2 current snap is 57 2024-09-16T05:12:23.803 INFO:tasks.rados.rados.2.smithi204.stdout:538: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-16T05:12:23.813 INFO:tasks.rados.rados.2.smithi204.stdout:538: writing smithi20457400-2 from 480733 to 1018148 tid 1 2024-09-16T05:12:23.818 INFO:tasks.rados.rados.2.smithi204.stdout:538: writing smithi20457400-2 from 1782380 to 2478782 tid 2 2024-09-16T05:12:23.825 INFO:tasks.rados.rados.2.smithi204.stdout:538: writing smithi20457400-2 from 3031400 to 3568559 tid 3 2024-09-16T05:12:23.825 INFO:tasks.rados.rados.2.smithi204.stdout:539: read oid 46 snap -1 2024-09-16T05:12:23.826 INFO:tasks.rados.rados.2.smithi204.stdout:539: expect (ObjNum 51 snap 1 seq_num 51) 2024-09-16T05:12:23.826 INFO:tasks.rados.rados.2.smithi204.stdout:540: write oid 44 current snap is 57 2024-09-16T05:12:23.826 INFO:tasks.rados.rados.2.smithi204.stdout:540: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-16T05:12:23.831 INFO:tasks.rados.rados.2.smithi204.stdout:540: writing smithi20457400-44 from 497540 to 1110204 tid 1 2024-09-16T05:12:23.838 INFO:tasks.rados.rados.2.smithi204.stdout:540: writing smithi20457400-44 from 1840862 to 2280922 tid 2 2024-09-16T05:12:23.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:23.845 INFO:tasks.rados.rados.2.smithi204.stdout:540: writing smithi20457400-44 from 3058823 to 3523607 tid 3 2024-09-16T05:12:23.845 INFO:tasks.rados.rados.2.smithi204.stdout:541: delete oid 40 current snap is 57 2024-09-16T05:12:23.845 INFO:tasks.rados.rados.2.smithi204.stdout:535: expect (ObjNum 181 snap 46 seq_num 181) 2024-09-16T05:12:23.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:23.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:23 smithi032 ceph-mon[27388]: pgmap v468: 47 pgs: 2 peering, 7 unknown, 38 active+clean; 203 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T05:12:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-16T05:12:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:23 smithi032 ceph-mon[27388]: osdmap e211: 12 total, 12 up, 12 in 2024-09-16T05:12:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2346063024' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:23.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:23.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:23.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:24.038 INFO:tasks.rados.rados.2.smithi204.stdout:538: finishing write tid 1 to smithi20457400-2 2024-09-16T05:12:24.038 INFO:tasks.rados.rados.2.smithi204.stdout:527: expect (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:12:24.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:23 smithi204 ceph-mon[30933]: pgmap v468: 47 pgs: 2 peering, 7 unknown, 38 active+clean; 203 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T05:12:24.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-16T05:12:24.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:23 smithi204 ceph-mon[30933]: osdmap e211: 12 total, 12 up, 12 in 2024-09-16T05:12:24.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2346063024' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:24.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:24.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:24.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:24.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:538: finishing write tid 2 to smithi20457400-2 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:531: finishing write tid 1 to smithi20457400-49 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:531: finishing write tid 2 to smithi20457400-49 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:538: finishing write tid 3 to smithi20457400-2 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:538: finishing write tid 4 to smithi20457400-2 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:538: finishing write tid 5 to smithi20457400-2 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 76 (ObjNum 219 snap 57 seq_num 219) dirty exists 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:538: left oid 2 (ObjNum 219 snap 57 seq_num 219) 2024-09-16T05:12:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:531: finishing write tid 3 to smithi20457400-49 2024-09-16T05:12:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:531: finishing write tid 4 to smithi20457400-49 2024-09-16T05:12:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:531: finishing write tid 5 to smithi20457400-49 2024-09-16T05:12:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 158 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:12:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:531: left oid 49 (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:12:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:540: finishing write tid 1 to smithi20457400-44 2024-09-16T05:12:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:539: expect (ObjNum 51 snap 1 seq_num 51) 2024-09-16T05:12:24.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:23 smithi196 ceph-mon[30802]: pgmap v468: 47 pgs: 2 peering, 7 unknown, 38 active+clean; 203 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T05:12:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-16T05:12:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:23 smithi196 ceph-mon[30802]: osdmap e211: 12 total, 12 up, 12 in 2024-09-16T05:12:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2346063024' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:24.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:24.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:540: finishing write tid 2 to smithi20457400-44 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:540: finishing write tid 3 to smithi20457400-44 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:527: done (7 left) 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:531: done (6 left) 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:535: done (5 left) 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:538: done (4 left) 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:539: done (3 left) 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:541: done (2 left) 2024-09-16T05:12:24.317 INFO:tasks.rados.rados.2.smithi204.stdout:542: rollback oid 22 current snap is 57 2024-09-16T05:12:24.318 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 49 2024-09-16T05:12:24.318 INFO:tasks.rados.rados.2.smithi204.stdout:540: finishing write tid 4 to smithi20457400-44 2024-09-16T05:12:24.318 INFO:tasks.rados.rados.2.smithi204.stdout:540: finishing write tid 5 to smithi20457400-44 2024-09-16T05:12:24.318 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 157 (ObjNum 220 snap 57 seq_num 220) dirty exists 2024-09-16T05:12:24.318 INFO:tasks.rados.rados.2.smithi204.stdout:540: left oid 44 (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:12:24.318 INFO:tasks.rados.rados.2.smithi204.stdout:540: done (2 left) 2024-09-16T05:12:24.318 INFO:tasks.rados.rados.2.smithi204.stdout:543: snap_create 2024-09-16T05:12:24.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:24.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:24.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:24.379 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:24.466 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:12:24.466 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:12:24.467 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-16T05:12:24.467 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:12:24.467 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:12:24.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-16T05:12:24.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:24.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:24.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:24.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:24.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:24.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:24.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:24.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:24.810 INFO:tasks.rados.rados.2.smithi204.stdout:543: done (2 left) 2024-09-16T05:12:24.810 INFO:tasks.rados.rados.2.smithi204.stdout:544: read oid 13 snap -1 2024-09-16T05:12:24.810 INFO:tasks.rados.rados.2.smithi204.stdout:544: expect deleted 2024-09-16T05:12:24.810 INFO:tasks.rados.rados.2.smithi204.stdout:545: copy_from oid 18 from oid 26 current snap is 58 2024-09-16T05:12:24.810 INFO:tasks.rados.rados.2.smithi204.stdout:546: write oid 35 current snap is 58 2024-09-16T05:12:24.810 INFO:tasks.rados.rados.2.smithi204.stdout:546: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-16T05:12:24.815 INFO:tasks.rados.rados.2.smithi204.stdout:546: writing smithi20457400-35 from 531154 to 1010669 tid 1 2024-09-16T05:12:24.820 INFO:tasks.rados.rados.2.smithi204.stdout:546: writing smithi20457400-35 from 1557826 to 2001555 tid 2 2024-09-16T05:12:24.827 INFO:tasks.rados.rados.2.smithi204.stdout:546: writing smithi20457400-35 from 2713669 to 3200997 tid 3 2024-09-16T05:12:24.830 INFO:tasks.rados.rados.2.smithi204.stdout:546: writing smithi20457400-35 from 3679295 to 3731154 tid 4 2024-09-16T05:12:24.831 INFO:tasks.rados.rados.2.smithi204.stdout:547: snap_remove snap 32 2024-09-16T05:12:24.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:24.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:24.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:24.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:24.868 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:24.868 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:24.957 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:24 smithi032 ceph-mon[27388]: osdmap e212: 12 total, 12 up, 12 in 2024-09-16T05:12:24.957 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:24 smithi032 ceph-mon[27388]: osdmap e213: 12 total, 12 up, 12 in 2024-09-16T05:12:24.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:24.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:24.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:24.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:25.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:25.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:25.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:24 smithi204 ceph-mon[30933]: osdmap e212: 12 total, 12 up, 12 in 2024-09-16T05:12:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:24 smithi204 ceph-mon[30933]: osdmap e213: 12 total, 12 up, 12 in 2024-09-16T05:12:25.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:24 smithi196 ceph-mon[30802]: osdmap e212: 12 total, 12 up, 12 in 2024-09-16T05:12:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:24 smithi196 ceph-mon[30802]: osdmap e213: 12 total, 12 up, 12 in 2024-09-16T05:12:25.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:25.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:25.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:25.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:25.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:25.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:25.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:25.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:25.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:12:25.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:25.615 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:12:25.615 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T05:12:25.615 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:12:25.622 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:25.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:25.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:12:25.658 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:12:25.663 INFO:tasks.rados.rados.2.smithi204.stdout:547: done (5 left) 2024-09-16T05:12:25.664 INFO:tasks.rados.rados.2.smithi204.stdout:548: copy_from oid 33 from oid 37 current snap is 58 2024-09-16T05:12:25.664 INFO:tasks.rados.rados.2.smithi204.stdout:549: read oid 15 snap 30 2024-09-16T05:12:25.664 INFO:tasks.rados.rados.2.smithi204.stdout:549: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T05:12:25.664 INFO:tasks.rados.rados.2.smithi204.stdout:550: snap_remove snap 47 2024-09-16T05:12:25.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:25.747 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:25.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:25.809 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:25.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:25.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:25.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:25.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:25.995 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:25.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:26.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:25 smithi204 ceph-mon[30933]: pgmap v471: 47 pgs: 4 peering, 1 remapped+peering, 3 unknown, 39 active+clean; 211 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 14 op/s 2024-09-16T05:12:26.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2909821099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:26.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:25 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:26.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:25 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:26.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:25 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:26.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2909821099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:26.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:25 smithi204 ceph-mon[30933]: osdmap e214: 12 total, 12 up, 12 in 2024-09-16T05:12:26.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:26.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:26.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:25 smithi032 ceph-mon[27388]: pgmap v471: 47 pgs: 4 peering, 1 remapped+peering, 3 unknown, 39 active+clean; 211 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 14 op/s 2024-09-16T05:12:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2909821099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:25 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:25 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:25 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2909821099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:25 smithi032 ceph-mon[27388]: osdmap e214: 12 total, 12 up, 12 in 2024-09-16T05:12:26.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:25 smithi196 ceph-mon[30802]: pgmap v471: 47 pgs: 4 peering, 1 remapped+peering, 3 unknown, 39 active+clean; 211 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 14 op/s 2024-09-16T05:12:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2909821099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:25 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:25 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:25 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2909821099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:25 smithi196 ceph-mon[30802]: osdmap e214: 12 total, 12 up, 12 in 2024-09-16T05:12:26.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:26.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:26.293 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:26.293 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:26.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:26.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:26.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:26.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:26.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:26.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:26.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:26.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:26.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:26.664 INFO:tasks.rados.rados.2.smithi204.stdout:550: done (7 left) 2024-09-16T05:12:26.664 INFO:tasks.rados.rados.2.smithi204.stdout:551: snap_create 2024-09-16T05:12:26.664 INFO:tasks.rados.rados.2.smithi204.stdout:546: finishing write tid 1 to smithi20457400-35 2024-09-16T05:12:26.664 INFO:tasks.rados.rados.2.smithi204.stdout:546: finishing write tid 2 to smithi20457400-35 2024-09-16T05:12:26.664 INFO:tasks.rados.rados.2.smithi204.stdout:546: finishing write tid 3 to smithi20457400-35 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:546: finishing write tid 4 to smithi20457400-35 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:546: finishing write tid 5 to smithi20457400-35 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:546: finishing write tid 6 to smithi20457400-35 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 282 (ObjNum 222 snap 58 seq_num 222) dirty exists 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:546: left oid 35 (ObjNum 222 snap 58 seq_num 222) 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:545: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:545: finishing copy_from to smithi20457400-18 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 67 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:536: finishing rollback tid 0 to smithi20457400-30 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:536: finishing rollback tid 1 to smithi20457400-30 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:536: finishing rollback tid 2 to smithi20457400-30 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 266 (ObjNum 120 snap 25 seq_num 120) dirty exists 2024-09-16T05:12:26.665 INFO:tasks.rados.rados.2.smithi204.stdout:549: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T05:12:26.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:26.739 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:26.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:26.740 INFO:tasks.rados.rados.2.smithi204.stdout:548: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:12:26.740 INFO:tasks.rados.rados.2.smithi204.stdout:548: finishing copy_from to smithi20457400-33 2024-09-16T05:12:26.741 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 117 (ObjNum 129 snap 27 seq_num 129) dirty exists 2024-09-16T05:12:26.741 INFO:tasks.rados.rados.2.smithi204.stdout:542: finishing rollback tid 0 to smithi20457400-22 2024-09-16T05:12:26.741 INFO:tasks.rados.rados.2.smithi204.stdout:542: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:12:26.741 INFO:tasks.rados.rados.2.smithi204.stdout:542: finishing rollback tid 2 to smithi20457400-22 2024-09-16T05:12:26.741 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 196 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T05:12:26.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:26.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:26.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:12:26.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:26 smithi204 ceph-mon[30933]: osdmap e215: 12 total, 12 up, 12 in 2024-09-16T05:12:26.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:12:26.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:26.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:26.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:26.998 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:27.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:27.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:27.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:27.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:27.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:27.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:12:27.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:12:27.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:12:27.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:12:27.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:12:27.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:12:27.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:26 smithi032 ceph-mon[27388]: osdmap e215: 12 total, 12 up, 12 in 2024-09-16T05:12:27.182 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:27.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:27.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:27.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:12:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:26 smithi196 ceph-mon[30802]: osdmap e215: 12 total, 12 up, 12 in 2024-09-16T05:12:27.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:27.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:27.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:27.433 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:27.433 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:27.454 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:27.454 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:27.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:27.474 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:27.475 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:27.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:27.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:27.495 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:27.495 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:27.516 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:27.539 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:27.560 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:27.561 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:27.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:27.582 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:27.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:27.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:27.604 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:27.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:27.624 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:27.625 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:27.646 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:27.646 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:27.664 INFO:tasks.rados.rados.2.smithi204.stdout:536: done (7 left) 2024-09-16T05:12:27.664 INFO:tasks.rados.rados.2.smithi204.stdout:542: done (6 left) 2024-09-16T05:12:27.664 INFO:tasks.rados.rados.2.smithi204.stdout:544: done (5 left) 2024-09-16T05:12:27.664 INFO:tasks.rados.rados.2.smithi204.stdout:545: done (4 left) 2024-09-16T05:12:27.664 INFO:tasks.rados.rados.2.smithi204.stdout:546: done (3 left) 2024-09-16T05:12:27.664 INFO:tasks.rados.rados.2.smithi204.stdout:548: done (2 left) 2024-09-16T05:12:27.665 INFO:tasks.rados.rados.2.smithi204.stdout:549: done (1 left) 2024-09-16T05:12:27.665 INFO:tasks.rados.rados.2.smithi204.stdout:551: done (0 left) 2024-09-16T05:12:27.665 INFO:tasks.rados.rados.2.smithi204.stdout:552: delete oid 35 current snap is 59 2024-09-16T05:12:27.668 INFO:tasks.rados.rados.2.smithi204.stdout:552: done (0 left) 2024-09-16T05:12:27.668 INFO:tasks.rados.rados.2.smithi204.stdout:553: read oid 6 snap 54 2024-09-16T05:12:27.669 INFO:tasks.rados.rados.2.smithi204.stdout:553: expect (ObjNum 200 snap 53 seq_num 200) 2024-09-16T05:12:27.669 INFO:tasks.rados.rados.2.smithi204.stdout:554: snap_create 2024-09-16T05:12:27.669 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:27.669 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:553: expect (ObjNum 200 snap 53 seq_num 200) 2024-09-16T05:12:27.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:27.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:27.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:27.794 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:12:27.794 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:12:27.794 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:12:27.794 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:12:27.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:27.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:27.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: pgmap v474: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 2 peering, 1 remapped+peering, 1 unknown, 41 active+clean; 225 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 21 op/s; 12/236 objects degraded (5.085%); 13/236 objects misplaced (5.508%); 2.9 MiB/s, 4 objects/s recovering 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: Health check failed: Degraded data redundancy: 12/236 objects degraded (5.085%), 1 pg degraded (PG_DEGRADED) 2024-09-16T05:12:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2517066416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:27.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:12:27.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:12:27.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:27 smithi032 ceph-mon[27388]: osdmap e216: 12 total, 12 up, 12 in 2024-09-16T05:12:27.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: pgmap v474: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 2 peering, 1 remapped+peering, 1 unknown, 41 active+clean; 225 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 21 op/s; 12/236 objects degraded (5.085%); 13/236 objects misplaced (5.508%); 2.9 MiB/s, 4 objects/s recovering 2024-09-16T05:12:27.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:12:27.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:12:27.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: Health check failed: Degraded data redundancy: 12/236 objects degraded (5.085%), 1 pg degraded (PG_DEGRADED) 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2517066416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:12:27.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:27 smithi204 ceph-mon[30933]: osdmap e216: 12 total, 12 up, 12 in 2024-09-16T05:12:27.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:27.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:28.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:28.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:28.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:28.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:28.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:28.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:28.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:28.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: pgmap v474: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 2 peering, 1 remapped+peering, 1 unknown, 41 active+clean; 225 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 21 op/s; 12/236 objects degraded (5.085%); 13/236 objects misplaced (5.508%); 2.9 MiB/s, 4 objects/s recovering 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: Health check failed: Degraded data redundancy: 12/236 objects degraded (5.085%), 1 pg degraded (PG_DEGRADED) 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2517066416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:12:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:12:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:27 smithi196 ceph-mon[30802]: osdmap e216: 12 total, 12 up, 12 in 2024-09-16T05:12:28.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:28.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:28.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:28.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:12:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:28.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:28.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:28.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:28.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:28.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:28.665 INFO:tasks.rados.rados.2.smithi204.stdout:553: done (1 left) 2024-09-16T05:12:28.665 INFO:tasks.rados.rados.2.smithi204.stdout:554: done (0 left) 2024-09-16T05:12:28.665 INFO:tasks.rados.rados.2.smithi204.stdout:555: rollback oid 23 current snap is 60 2024-09-16T05:12:28.666 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 23 to 56 2024-09-16T05:12:28.666 INFO:tasks.rados.rados.2.smithi204.stdout:556: write (excl) oid 25 current snap is 60 2024-09-16T05:12:28.666 INFO:tasks.rados.rados.2.smithi204.stdout:556: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-16T05:12:28.672 INFO:tasks.rados.rados.2.smithi204.stdout:556: writing smithi20457400-25 from 564768 to 1194781 tid 1 2024-09-16T05:12:28.677 INFO:tasks.rados.rados.2.smithi204.stdout:556: writing smithi20457400-25 from 1674790 to 2122188 tid 2 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:556: writing smithi20457400-25 from 2768515 to 3394740 tid 3 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:557: copy_from oid 21 from oid 49 current snap is 60 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:555: finishing rollback tid 1 to smithi20457400-23 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:556: finishing write tid 1 to smithi20457400-25 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:556: finishing write tid 2 to smithi20457400-25 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:555: done (2 left) 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:558: copy_from oid 11 from oid 23 current snap is 60 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:559: read oid 43 snap -1 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:559: expect (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:12:28.685 INFO:tasks.rados.rados.2.smithi204.stdout:560: write oid 40 current snap is 60 2024-09-16T05:12:28.686 INFO:tasks.rados.rados.2.smithi204.stdout:560: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-16T05:12:28.692 INFO:tasks.rados.rados.2.smithi204.stdout:560: writing smithi20457400-40 from 581575 to 1286837 tid 1 2024-09-16T05:12:28.699 INFO:tasks.rados.rados.2.smithi204.stdout:560: writing smithi20457400-40 from 1733272 to 2440681 tid 2 2024-09-16T05:12:28.706 INFO:tasks.rados.rados.2.smithi204.stdout:560: writing smithi20457400-40 from 3195938 to 3633435 tid 3 2024-09-16T05:12:28.706 INFO:tasks.rados.rados.2.smithi204.stdout:561: snap_create 2024-09-16T05:12:28.706 INFO:tasks.rados.rados.2.smithi204.stdout:556: finishing write tid 3 to smithi20457400-25 2024-09-16T05:12:28.706 INFO:tasks.rados.rados.2.smithi204.stdout:556: finishing write tid 4 to smithi20457400-25 2024-09-16T05:12:28.706 INFO:tasks.rados.rados.2.smithi204.stdout:556: finishing write tid 5 to smithi20457400-25 2024-09-16T05:12:28.706 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 267 (ObjNum 224 snap 60 seq_num 224) dirty exists 2024-09-16T05:12:28.706 INFO:tasks.rados.rados.2.smithi204.stdout:556: left oid 25 (ObjNum 224 snap 60 seq_num 224) 2024-09-16T05:12:28.706 INFO:tasks.rados.rados.2.smithi204.stdout:559: expect (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:12:28.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:28.725 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:28.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:28.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:28.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:28.849 INFO:tasks.rados.rados.2.smithi204.stdout:558: finishing copy_from to smithi20457400-11 2024-09-16T05:12:28.849 INFO:tasks.rados.rados.2.smithi204.stdout:558: got expected ENOENT (src dne) 2024-09-16T05:12:28.849 INFO:tasks.rados.rados.2.smithi204.stdout:557: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:12:28.849 INFO:tasks.rados.rados.2.smithi204.stdout:558: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:12:28.849 INFO:tasks.rados.rados.2.smithi204.stdout:560: finishing write tid 1 to smithi20457400-40 2024-09-16T05:12:28.849 INFO:tasks.rados.rados.2.smithi204.stdout:557: finishing copy_from to smithi20457400-21 2024-09-16T05:12:28.850 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 158 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:12:28.850 INFO:tasks.rados.rados.2.smithi204.stdout:560: finishing write tid 2 to smithi20457400-40 2024-09-16T05:12:28.850 INFO:tasks.rados.rados.2.smithi204.stdout:560: finishing write tid 3 to smithi20457400-40 2024-09-16T05:12:28.850 INFO:tasks.rados.rados.2.smithi204.stdout:560: finishing write tid 4 to smithi20457400-40 2024-09-16T05:12:28.850 INFO:tasks.rados.rados.2.smithi204.stdout:560: finishing write tid 5 to smithi20457400-40 2024-09-16T05:12:28.850 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 175 (ObjNum 225 snap 60 seq_num 225) dirty exists 2024-09-16T05:12:28.850 INFO:tasks.rados.rados.2.smithi204.stdout:560: left oid 40 (ObjNum 225 snap 60 seq_num 225) 2024-09-16T05:12:28.859 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:28.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:28.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:28.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:12:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:28.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:28.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:29.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:12:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:12:29.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:12:29.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:12:29.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:12:29.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:12:29.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:12:29.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:12:29.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:12:29.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:28 smithi204 ceph-mon[30933]: osdmap e217: 12 total, 12 up, 12 in 2024-09-16T05:12:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:29.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:12:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:12:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:12:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:12:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:12:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:12:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:12:29.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:12:29.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:12:29.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:12:29.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:12:29.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:12:29.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:12:29.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:12:29.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:12:29.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:12:29.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:28 smithi032 ceph-mon[27388]: osdmap e217: 12 total, 12 up, 12 in 2024-09-16T05:12:29.201 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:12:29.201 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:12:29.201 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:12:29.202 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:12:29.202 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:12:29.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:12:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:12:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:12:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:12:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:12:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:12:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:12:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:12:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:12:29.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:12:29.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:12:29.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:12:29.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:12:29.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:12:29.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:12:29.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:12:29.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:28 smithi196 ceph-mon[30802]: osdmap e217: 12 total, 12 up, 12 in 2024-09-16T05:12:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:29.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:29.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:29.251 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:29.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:29.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:29.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:29.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:29.491 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:29.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:29.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:29.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:29.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:29.666 INFO:tasks.rados.rados.2.smithi204.stdout:556: done (5 left) 2024-09-16T05:12:29.666 INFO:tasks.rados.rados.2.smithi204.stdout:557: done (4 left) 2024-09-16T05:12:29.666 INFO:tasks.rados.rados.2.smithi204.stdout:558: done (3 left) 2024-09-16T05:12:29.666 INFO:tasks.rados.rados.2.smithi204.stdout:559: done (2 left) 2024-09-16T05:12:29.666 INFO:tasks.rados.rados.2.smithi204.stdout:560: done (1 left) 2024-09-16T05:12:29.666 INFO:tasks.rados.rados.2.smithi204.stdout:561: done (0 left) 2024-09-16T05:12:29.666 INFO:tasks.rados.rados.2.smithi204.stdout:562: read oid 2 snap 53 2024-09-16T05:12:29.667 INFO:tasks.rados.rados.2.smithi204.stdout:562: expect deleted 2024-09-16T05:12:29.667 INFO:tasks.rados.rados.2.smithi204.stdout:563: write (excl) oid 31 current snap is 61 2024-09-16T05:12:29.667 INFO:tasks.rados.rados.2.smithi204.stdout:563: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-16T05:12:29.673 INFO:tasks.rados.rados.2.smithi204.stdout:563: writing smithi20457400-31 from 598382 to 1378893 tid 1 2024-09-16T05:12:29.679 INFO:tasks.rados.rados.2.smithi204.stdout:563: writing smithi20457400-31 from 2075401 to 2526468 tid 2 2024-09-16T05:12:29.686 INFO:tasks.rados.rados.2.smithi204.stdout:563: writing smithi20457400-31 from 3107008 to 3755777 tid 3 2024-09-16T05:12:29.687 INFO:tasks.rados.rados.2.smithi204.stdout:564: read oid 28 snap -1 2024-09-16T05:12:29.687 INFO:tasks.rados.rados.2.smithi204.stdout:564: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T05:12:29.687 INFO:tasks.rados.rados.2.smithi204.stdout:565: delete oid 25 current snap is 61 2024-09-16T05:12:29.687 INFO:tasks.rados.rados.2.smithi204.stdout:563: finishing write tid 1 to smithi20457400-31 2024-09-16T05:12:29.687 INFO:tasks.rados.rados.2.smithi204.stdout:563: finishing write tid 2 to smithi20457400-31 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:563: finishing write tid 3 to smithi20457400-31 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:563: finishing write tid 4 to smithi20457400-31 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:563: finishing write tid 5 to smithi20457400-31 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 281 (ObjNum 226 snap 61 seq_num 226) dirty exists 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:563: left oid 31 (ObjNum 226 snap 61 seq_num 226) 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:562: done (3 left) 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:563: done (2 left) 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:565: done (1 left) 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:566: read oid 46 snap 29 2024-09-16T05:12:29.690 INFO:tasks.rados.rados.2.smithi204.stdout:566: expect (ObjNum 51 snap 1 seq_num 51) 2024-09-16T05:12:29.691 INFO:tasks.rados.rados.2.smithi204.stdout:567: copy_from oid 8 from oid 32 current snap is 61 2024-09-16T05:12:29.691 INFO:tasks.rados.rados.2.smithi204.stdout:568: delete oid 25 current snap is 61 2024-09-16T05:12:29.691 INFO:tasks.rados.rados.2.smithi204.stdout:568: done (3 left) 2024-09-16T05:12:29.691 INFO:tasks.rados.rados.2.smithi204.stdout:569: write oid 10 current snap is 61 2024-09-16T05:12:29.691 INFO:tasks.rados.rados.2.smithi204.stdout:569: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-16T05:12:29.699 INFO:tasks.rados.rados.2.smithi204.stdout:569: writing smithi20457400-10 from 648803 to 1371414 tid 1 2024-09-16T05:12:29.706 INFO:tasks.rados.rados.2.smithi204.stdout:569: writing smithi20457400-10 from 1967200 to 2681947 tid 2 2024-09-16T05:12:29.713 INFO:tasks.rados.rados.2.smithi204.stdout:569: writing smithi20457400-10 from 3421983 to 3848803 tid 3 2024-09-16T05:12:29.713 INFO:tasks.rados.rados.2.smithi204.stdout:570: rollback oid 13 current snap is 61 2024-09-16T05:12:29.713 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 13 to 56 2024-09-16T05:12:29.713 INFO:tasks.rados.rados.2.smithi204.stdout:571: read oid 12 snap -1 2024-09-16T05:12:29.713 INFO:tasks.rados.rados.2.smithi204.stdout:571: expect (ObjNum 124 snap 26 seq_num 124) 2024-09-16T05:12:29.713 INFO:tasks.rados.rados.2.smithi204.stdout:572: copy_from oid 47 from oid 35 current snap is 61 2024-09-16T05:12:29.713 INFO:tasks.rados.rados.2.smithi204.stdout:564: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T05:12:29.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:29.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:29.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:29.793 INFO:tasks.rados.rados.2.smithi204.stdout:566: expect (ObjNum 51 snap 1 seq_num 51) 2024-09-16T05:12:29.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:29.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:29.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:29.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:12:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:29 smithi032 ceph-mon[27388]: pgmap v477: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 1 remapped+peering, 44 active+clean; 247 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 12/252 objects degraded (4.762%); 13/252 objects misplaced (5.159%); 3.9 MiB/s, 5 objects/s recovering 2024-09-16T05:12:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:29 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:12:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:29 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:12:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:29 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:12:29.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:29 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:12:29.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:29 smithi032 ceph-mon[27388]: osdmap e218: 12 total, 12 up, 12 in 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:567: finishing copy_from racing read to smithi20457400-8 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:567: finishing copy_from to smithi20457400-8 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 285 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:569: finishing write tid 1 to smithi20457400-10 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:569: finishing write tid 2 to smithi20457400-10 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:564: done (6 left) 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:566: done (5 left) 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:567: done (4 left) 2024-09-16T05:12:29.925 INFO:tasks.rados.rados.2.smithi204.stdout:573: snap_remove snap 30 2024-09-16T05:12:29.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:29.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:30.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:30.116 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:30.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:30.166 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:29 smithi204 ceph-mon[30933]: pgmap v477: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 1 remapped+peering, 44 active+clean; 247 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 12/252 objects degraded (4.762%); 13/252 objects misplaced (5.159%); 3.9 MiB/s, 5 objects/s recovering 2024-09-16T05:12:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:29 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:12:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:29 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:12:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:29 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:12:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:29 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:12:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:29 smithi204 ceph-mon[30933]: osdmap e218: 12 total, 12 up, 12 in 2024-09-16T05:12:30.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:30.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:29 smithi196 ceph-mon[30802]: pgmap v477: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 1 remapped+peering, 44 active+clean; 247 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 12/252 objects degraded (4.762%); 13/252 objects misplaced (5.159%); 3.9 MiB/s, 5 objects/s recovering 2024-09-16T05:12:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:29 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:12:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:29 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:12:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:29 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:12:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:29 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:12:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:29 smithi196 ceph-mon[30802]: osdmap e218: 12 total, 12 up, 12 in 2024-09-16T05:12:30.239 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:30.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:30.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:30.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:12:30.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:30.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:30.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:30.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:30.488 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:30.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:30.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:30.536 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:30.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:290 2.d 2:b98a79db:::smithi20457400-31:head [copy-from ver 13] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 101.259659052, 2024-09-16T05:12:30.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020074727000000001, 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 290 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:12:30.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340466+0000", 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4300000000000001e-06 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340473+0000", 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3339999999999996e-06 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340486+0000", 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.279e-05 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340495+0000", 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4890000000000001e-06 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340559+0000", 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3962999999999997e-05 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340653+0000", 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3468999999999996e-05 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.353864+0000", 2024-09-16T05:12:30.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01321114 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357823+0000", 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039590759999999997 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360478+0000", 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026554489999999998 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360511+0000", 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3052999999999997e-05 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360539+0000", 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7534000000000001e-05 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.608 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:291 2.d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 101.259603169, 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016626765000000002, 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 291 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340521+0000", 2024-09-16T05:12:30.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4199999999999995e-07 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340524+0000", 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8590000000000001e-06 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340532+0000", 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4090000000000003e-06 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340538+0000", 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.835e-06 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340822+0000", 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028415500000000002 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340874+0000", 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1558999999999999e-05 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357014+0000", 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016140379999999999 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357072+0000", 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7887999999999999e-05 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357147+0000", 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5037999999999995e-05 2024-09-16T05:12:30.613 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:605 2.d 2:b1061e13:::smithi20457400-33:head [copy-from ver 20] snapc 13=[13,12,11,f,e,b] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 76.797172172000003, 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017106367000000001, 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 605 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802952+0000", 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9999999999999998e-07 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802955+0000", 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1379999999999999e-06 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:30.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802961+0000", 2024-09-16T05:12:30.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3820000000000001e-06 2024-09-16T05:12:30.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802966+0000", 2024-09-16T05:12:30.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1370000000000001e-06 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803384+0000", 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00041787299999999997 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803568+0000", 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018420200000000001 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.814402+0000", 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010833617 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818577+0000", 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041754770000000004 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.819989+0000", 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001411818 2024-09-16T05:12:30.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.820026+0000", 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6613000000000003e-05 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.820057+0000", 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1609999999999997e-05 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:606 2.d 2:b1061e13:::smithi20457400-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 76.797147378000005, 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015239317, 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 606 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.622 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.63e-07 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802980+0000", 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.309e-06 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802993+0000", 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3574000000000001e-05 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802999+0000", 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5949999999999998e-06 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803711+0000", 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071251099999999998 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803769+0000", 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7649000000000003e-05 2024-09-16T05:12:30.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818011+0000", 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014242184 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818110+0000", 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8938999999999996e-05 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818215+0000", 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105093 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1125 2.d 2:ba754fcb:::smithi20457400-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e190)", 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:01.729008+0000", 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.871115578000001, 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0094147350000000005, 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.628 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1125 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.729008+0000", 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.729008+0000", 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.729008+0000", 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3200000000000005e-07 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.729011+0000", 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7580000000000001e-06 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.729016+0000", 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9169999999999997e-06 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.729019+0000", 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5190000000000001e-06 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.730086+0000", 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010664979999999999 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.730140+0000", 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4103e-05 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.738312+0000", 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081717600000000001 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.738359+0000", 2024-09-16T05:12:30.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7119000000000001e-05 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:01.738422+0000", 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3528999999999994e-05 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1182 2.15 2:a94680a3:::smithi20457400-13:head [setxattr _header (57) in=7b,truncate 3109295] snapc 2f=[2f,2e,2d,2c,2b,2a,29,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:12:30.633 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.604148366, 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016329251999999999, 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1182 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:12:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9999999999999998e-07 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995978+0000", 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2589999999999999e-06 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995982+0000", 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3059999999999999e-06 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995985+0000", 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1829999999999998e-06 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996680+0000", 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069462300000000001 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996787+0000", 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107268 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996940+0000", 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000152826 2024-09-16T05:12:30.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997265+0000", 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032523600000000001 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012161+0000", 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014896243999999999 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012231+0000", 2024-09-16T05:12:30.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9796999999999996e-05 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012304+0000", 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3109999999999996e-05 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1183 2.15 2:a94680a3:::smithi20457400-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.60412311, 2024-09-16T05:12:30.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016473326999999999, 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1183 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996001+0000", 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4200000000000002e-07 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996003+0000", 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1679999999999998e-06 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996009+0000", 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3890000000000002e-06 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996014+0000", 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7049999999999998e-06 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997177+0000", 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00116278 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997224+0000", 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7601e-05 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012338+0000", 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015113685999999999 2024-09-16T05:12:30.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012400+0000", 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2199999999999994e-05 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012474+0000", 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3455999999999996e-05 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1329 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.8170863829999995, 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80822626200000003, 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1329 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.645 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783039+0000", 2024-09-16T05:12:30.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.674e-06 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783047+0000", 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6619999999999994e-06 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783063+0000", 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6058000000000001e-05 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783071+0000", 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8399999999999995e-06 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783134+0000", 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2496999999999995e-05 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783142+0000", 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1219999999999995e-06 2024-09-16T05:12:30.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591169+0000", 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80802694100000005 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591263+0000", 2024-09-16T05:12:30.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4468000000000004e-05 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1330 2.1d 2:b98a79db:::smithi20457400-31:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.8170354209999999, 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80866329400000003, 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.650 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1330 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783092+0000", 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5149999999999998e-06 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783104+0000", 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2412000000000001e-05 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783117+0000", 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2379e-05 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783170+0000", 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3727999999999998e-05 2024-09-16T05:12:30.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783178+0000", 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3479999999999998e-06 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591633+0000", 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.808455339 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591751+0000", 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118087 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1331 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.8169946550000002, 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80878027500000005, 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:12:30.655 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1331 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5400000000000001e-07 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783132+0000", 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689999999999999e-06 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783139+0000", 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7920000000000004e-06 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783144+0000", 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783192+0000", 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8304999999999999e-05 2024-09-16T05:12:30.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783196+0000", 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8480000000000003e-06 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591842+0000", 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80864587700000001 2024-09-16T05:12:30.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591909+0000", 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7195999999999998e-05 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1361 2.15 2:a94680a3:::smithi20457400-13:head [copy-from ver 281] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2544787280000005, 2024-09-16T05:12:30.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017774172000000001, 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1361 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345646+0000", 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.646e-06 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345656+0000", 2024-09-16T05:12:30.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3170000000000008e-06 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345672+0000", 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6821e-05 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345679+0000", 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0840000000000003e-06 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345741+0000", 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1345000000000003e-05 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345803+0000", 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2003999999999998e-05 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.354995+0000", 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091920379999999996 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.359012+0000", 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040172389999999997 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363338+0000", 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043261180000000003 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363376+0000", 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7613000000000001e-05 2024-09-16T05:12:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363419+0000", 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2947000000000001e-05 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1362 2.15 2:a94680a3:::smithi20457400-13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.345699+0000", 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2544244219999996, 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012587882, 2024-09-16T05:12:30.666 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1362 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345699+0000", 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345699+0000", 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345699+0000", 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5600000000000001e-07 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345701+0000", 2024-09-16T05:12:30.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2330000000000001e-06 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345706+0000", 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7700000000000001e-06 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345709+0000", 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1980000000000001e-06 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345941+0000", 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000231499 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345981+0000", 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0451000000000003e-05 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.358149+0000", 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01216767 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.358216+0000", 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7422000000000006e-05 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.358287+0000", 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0283000000000005e-05 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.671 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.9.0:220 2.1d e210/209 ondisk, result = 0)", 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.0071172130000008, 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018128111999999998, 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593007+0000", 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3480000000000001e-06 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593019+0000", 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1963999999999999e-05 2024-09-16T05:12:30.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610880+0000", 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017860157000000002 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610890+0000", 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.093e-05 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610951+0000", 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0375999999999999e-05 2024-09-16T05:12:30.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610963+0000", 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1861e-05 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.611134+0000", 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000171476 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1389 2.1d 2:b98a79db:::smithi20457400-31:head [copy-from ver 20] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.0046733969999995, 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.062531666999999999, 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1389 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.676 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:30.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:30.677 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:12:30.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0499999999999999e-07 2024-09-16T05:12:30.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595453+0000", 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9699999999999999e-06 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595467+0000", 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3725e-05 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595472+0000", 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.028e-06 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.597983+0000", 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002510759 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.598071+0000", 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8158000000000005e-05 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.605944+0000", 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078733620000000001 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608849+0000", 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029044190000000001 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657852+0000", 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.049003110000000002 2024-09-16T05:12:30.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657923+0000", 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1388000000000002e-05 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657982+0000", 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8343e-05 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1390 2.1d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.595478+0000", 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.0046451279999999, 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012924761, 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1390 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595478+0000", 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595478+0000", 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595479+0000", 2024-09-16T05:12:30.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1899999999999998e-07 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595482+0000", 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.72e-06 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595488+0000", 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999999e-06 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595493+0000", 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6739999999999996e-06 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.598209+0000", 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027159250000000001 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.598266+0000", 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997000000000003e-05 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608189+0000", 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0099233610000000003 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608291+0000", 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101686 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608403+0000", 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111869 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.688 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1389 2.1d e210/209 ondisk, result = 0)", 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 7.9892744740000001, 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.047139519999999997, 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610850+0000", 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5600000000000001e-06 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610858+0000", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2099999999999996e-06 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657761+0000", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.046903037000000002 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657768+0000", 2024-09-16T05:12:30.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4080000000000001e-06 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657828+0000", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0241999999999997e-05 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657843+0000", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5043e-05 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657988+0000", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014501999999999999 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1471 2.14 2:2ace617f:::smithi20457400-21:head [copy-from ver 158] snapc 3d=[3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.9150454400000001, 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015385315, 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.693 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1471 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9299999999999998e-07 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685082+0000", 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3960000000000002e-06 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685089+0000", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.323e-06 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685094+0000", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0200000000000002e-06 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685163+0000", 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8861000000000004e-05 2024-09-16T05:12:30.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685259+0000", 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6365000000000003e-05 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.692105+0000", 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068456330000000003 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.696821+0000", 2024-09-16T05:12:30.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047159250000000002 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700376+0000", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035550970000000001 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700421+0000", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5158999999999999e-05 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700463+0000", 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2042999999999997e-05 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1472 2.14 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:28.685112+0000", 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.9150114039999999, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010871023, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.699 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1472 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685112+0000", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685112+0000", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685112+0000", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9399999999999995e-07 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685115+0000", 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.435e-06 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685130+0000", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5034999999999999e-05 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685136+0000", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.925e-06 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685456+0000", 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000320525 2024-09-16T05:12:30.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685504+0000", 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7787999999999997e-05 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.695783+0000", 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010279082 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.695867+0000", 2024-09-16T05:12:30.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3354000000000002e-05 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.695983+0000", 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116385 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1473 2.5 2:a5bc1608:::smithi20457400-11:head [copy-from ver 0] snapc 3d=[3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:28.685146+0000", 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.914977141, 2024-09-16T05:12:30.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010657663, 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1473 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685146+0000", 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685146+0000", 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685147+0000", 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4400000000000001e-07 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685160+0000", 2024-09-16T05:12:30.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3084999999999999e-05 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685167+0000", 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8220000000000001e-06 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685172+0000", 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3090000000000003e-06 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.687051+0000", 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00187941 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.687189+0000", 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000138164 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.695804+0000", 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086144289999999998 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.708 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1474 2.5 2:a5bc1608:::smithi20457400-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:28.685180+0000", 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.914943308, 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010838211, 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:30.709 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1474 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685180+0000", 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685180+0000", 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685181+0000", 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5899999999999996e-07 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685183+0000", 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.227e-06 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685189+0000", 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.246e-06 2024-09-16T05:12:30.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685194+0000", 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1100000000000002e-06 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.687340+0000", 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021459069999999998 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.687430+0000", 2024-09-16T05:12:30.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0367999999999999e-05 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.695872+0000", 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084418610000000002 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.695950+0000", 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7350999999999999e-05 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:30.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.696018+0000", 2024-09-16T05:12:30.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8582000000000002e-05 2024-09-16T05:12:30.714 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.714 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.714 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.714 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:30.714 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:30.714 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:12:30.714 INFO:tasks.rados.rados.2.smithi204.stdout:573: done (4 left) 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:574: rollback oid 24 current snap is 61 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 45 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:570: finishing rollback tid 1 to smithi20457400-13 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 301 (ObjNum 33 snap 0 seq_num 4282414272) dirty dne 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:569: finishing write tid 3 to smithi20457400-10 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:572: finishing copy_from to smithi20457400-47 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:572: got expected ENOENT (src dne) 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:569: finishing write tid 4 to smithi20457400-10 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:572: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:569: finishing write tid 5 to smithi20457400-10 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 289 (ObjNum 229 snap 61 seq_num 229) dirty exists 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:569: left oid 10 (ObjNum 229 snap 61 seq_num 229) 2024-09-16T05:12:30.715 INFO:tasks.rados.rados.2.smithi204.stdout:571: expect (ObjNum 124 snap 26 seq_num 124) 2024-09-16T05:12:30.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:574: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:12:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 433 snap 0 seq_num 2944376448) dirty dne 2024-09-16T05:12:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:569: done (4 left) 2024-09-16T05:12:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:570: done (3 left) 2024-09-16T05:12:30.785 INFO:tasks.rados.rados.2.smithi204.stdout:571: done (2 left) 2024-09-16T05:12:30.786 INFO:tasks.rados.rados.2.smithi204.stdout:572: done (1 left) 2024-09-16T05:12:30.786 INFO:tasks.rados.rados.2.smithi204.stdout:574: done (0 left) 2024-09-16T05:12:30.786 INFO:tasks.rados.rados.2.smithi204.stdout:575: write (excl) oid 24 current snap is 61 2024-09-16T05:12:30.786 INFO:tasks.rados.rados.2.smithi204.stdout:575: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-16T05:12:30.792 INFO:tasks.rados.rados.2.smithi204.stdout:575: writing smithi20457400-24 from 665610 to 1463470 tid 1 2024-09-16T05:12:30.797 INFO:tasks.rados.rados.2.smithi204.stdout:575: writing smithi20457400-24 from 1909329 to 2367734 tid 2 2024-09-16T05:12:30.800 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:30.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:30.803 INFO:tasks.rados.rados.2.smithi204.stdout:575: writing smithi20457400-24 from 2816700 to 3226910 tid 3 2024-09-16T05:12:30.803 INFO:tasks.rados.rados.2.smithi204.stdout:576: copy_from oid 17 from oid 14 current snap is 61 2024-09-16T05:12:30.803 INFO:tasks.rados.rados.2.smithi204.stdout:575: finishing write tid 1 to smithi20457400-24 2024-09-16T05:12:30.803 INFO:tasks.rados.rados.2.smithi204.stdout:575: finishing write tid 2 to smithi20457400-24 2024-09-16T05:12:30.803 INFO:tasks.rados.rados.2.smithi204.stdout:577: rollback oid 49 current snap is 61 2024-09-16T05:12:30.803 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 49 2024-09-16T05:12:30.803 INFO:tasks.rados.rados.2.smithi204.stdout:578: snap_remove snap 29 2024-09-16T05:12:30.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:12:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:12:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/266492778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/266492778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:30 smithi032 ceph-mon[27388]: osdmap e219: 12 total, 12 up, 12 in 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/266492778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/266492778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:30.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:30 smithi204 ceph-mon[30933]: osdmap e219: 12 total, 12 up, 12 in 2024-09-16T05:12:30.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:30.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:31.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:31.053 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:31.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:31.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:31.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/266492778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/266492778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:30 smithi196 ceph-mon[30802]: osdmap e219: 12 total, 12 up, 12 in 2024-09-16T05:12:31.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:31.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:31.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:31.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:31.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:31.426 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:31.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:12:31.486 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:12:31.487 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:12:31.487 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-16T05:12:31.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:12:31.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:31.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:31.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:31.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:31.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:31.571 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:31.571 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:31.592 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:31.612 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:31.613 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:31.634 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:31.656 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:31.656 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:31.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:31.664 INFO:tasks.rados.rados.2.smithi204.stdout:578: done (3 left) 2024-09-16T05:12:31.664 INFO:tasks.rados.rados.2.smithi204.stdout:579: snap_create 2024-09-16T05:12:31.664 INFO:tasks.rados.rados.2.smithi204.stdout:577: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:12:31.664 INFO:tasks.rados.rados.2.smithi204.stdout:577: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:12:31.664 INFO:tasks.rados.rados.2.smithi204.stdout:577: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 173 (ObjNum 92 snap 9 seq_num 92) dirty exists 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:575: finishing write tid 3 to smithi20457400-24 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:575: finishing write tid 4 to smithi20457400-24 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:575: finishing write tid 5 to smithi20457400-24 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 211 (ObjNum 230 snap 61 seq_num 230) dirty exists 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:575: left oid 24 (ObjNum 230 snap 61 seq_num 230) 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:576: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:576: finishing copy_from to smithi20457400-17 2024-09-16T05:12:31.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 253 (ObjNum 122 snap 25 seq_num 122) dirty exists 2024-09-16T05:12:31.682 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:31.682 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:31.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:31.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:31.704 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:31.704 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:31.727 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:31.728 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:31.749 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:31.750 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:31.750 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:31.771 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:31.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:31.793 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:31.794 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:31.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:31.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:31.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:31.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:31.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:32.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:32.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:32.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:32.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:31 smithi204 ceph-mon[30933]: pgmap v480: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 1 remapped+peering, 44 active+clean; 248 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 12/258 objects degraded (4.651%); 13/258 objects misplaced (5.039%); 1.1 MiB/s, 0 objects/s recovering 2024-09-16T05:12:32.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3463305983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:32.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:31 smithi204 ceph-mon[30933]: osdmap e220: 12 total, 12 up, 12 in 2024-09-16T05:12:32.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:31 smithi032 ceph-mon[27388]: pgmap v480: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 1 remapped+peering, 44 active+clean; 248 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 12/258 objects degraded (4.651%); 13/258 objects misplaced (5.039%); 1.1 MiB/s, 0 objects/s recovering 2024-09-16T05:12:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3463305983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:31 smithi032 ceph-mon[27388]: osdmap e220: 12 total, 12 up, 12 in 2024-09-16T05:12:32.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:32.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:32.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:31 smithi196 ceph-mon[30802]: pgmap v480: 47 pgs: 1 active+recovering+remapped, 1 activating+degraded+remapped, 1 remapped+peering, 44 active+clean; 248 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 12/258 objects degraded (4.651%); 13/258 objects misplaced (5.039%); 1.1 MiB/s, 0 objects/s recovering 2024-09-16T05:12:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3463305983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:31 smithi196 ceph-mon[30802]: osdmap e220: 12 total, 12 up, 12 in 2024-09-16T05:12:32.262 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:32.262 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:32.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:32.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:32.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:32.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:32.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:32.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:32.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:32.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:32.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:32.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:12:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:12:32.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:32.668 INFO:tasks.rados.rados.2.smithi204.stdout:575: done (3 left) 2024-09-16T05:12:32.668 INFO:tasks.rados.rados.2.smithi204.stdout:576: done (2 left) 2024-09-16T05:12:32.668 INFO:tasks.rados.rados.2.smithi204.stdout:577: done (1 left) 2024-09-16T05:12:32.668 INFO:tasks.rados.rados.2.smithi204.stdout:579: done (0 left) 2024-09-16T05:12:32.668 INFO:tasks.rados.rados.2.smithi204.stdout:580: write oid 46 current snap is 62 2024-09-16T05:12:32.668 INFO:tasks.rados.rados.2.smithi204.stdout:580: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-16T05:12:32.674 INFO:tasks.rados.rados.2.smithi204.stdout:580: writing smithi20457400-46 from 682417 to 1155526 tid 1 2024-09-16T05:12:32.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:32.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:32.680 INFO:tasks.rados.rados.2.smithi204.stdout:580: writing smithi20457400-46 from 1567811 to 2169874 tid 2 2024-09-16T05:12:32.688 INFO:tasks.rados.rados.2.smithi204.stdout:580: writing smithi20457400-46 from 2844123 to 3465605 tid 3 2024-09-16T05:12:32.688 INFO:tasks.rados.rados.2.smithi204.stdout:581: copy_from oid 35 from oid 14 current snap is 62 2024-09-16T05:12:32.688 INFO:tasks.rados.rados.2.smithi204.stdout:580: finishing write tid 1 to smithi20457400-46 2024-09-16T05:12:32.688 INFO:tasks.rados.rados.2.smithi204.stdout:580: finishing write tid 2 to smithi20457400-46 2024-09-16T05:12:32.688 INFO:tasks.rados.rados.2.smithi204.stdout:582: snap_create 2024-09-16T05:12:32.690 INFO:tasks.rados.rados.2.smithi204.stdout:580: finishing write tid 3 to smithi20457400-46 2024-09-16T05:12:32.690 INFO:tasks.rados.rados.2.smithi204.stdout:580: finishing write tid 4 to smithi20457400-46 2024-09-16T05:12:32.691 INFO:tasks.rados.rados.2.smithi204.stdout:580: finishing write tid 5 to smithi20457400-46 2024-09-16T05:12:32.691 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 14 (ObjNum 231 snap 62 seq_num 231) dirty exists 2024-09-16T05:12:32.691 INFO:tasks.rados.rados.2.smithi204.stdout:580: left oid 46 (ObjNum 231 snap 62 seq_num 231) 2024-09-16T05:12:32.698 INFO:tasks.rados.rados.2.smithi204.stdout:581: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:12:32.700 INFO:tasks.rados.rados.2.smithi204.stdout:581: finishing copy_from to smithi20457400-35 2024-09-16T05:12:32.700 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 253 (ObjNum 122 snap 25 seq_num 122) dirty exists 2024-09-16T05:12:32.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:32.803 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:32.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:32.838 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:32 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:12:32.838 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:32 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:12:32.838 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:32 smithi032 ceph-mon[27388]: osdmap e221: 12 total, 12 up, 12 in 2024-09-16T05:12:32.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:32.927 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:32.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:32.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:33.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:12:33.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:33.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:33.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:33.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:32 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:12:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:32 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:12:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:32 smithi204 ceph-mon[30933]: osdmap e221: 12 total, 12 up, 12 in 2024-09-16T05:12:33.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:33.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:32 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:12:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:32 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:12:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:32 smithi196 ceph-mon[30802]: osdmap e221: 12 total, 12 up, 12 in 2024-09-16T05:12:33.249 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:33.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:33.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:12:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:33.300 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:33.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:33.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:33.423 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:33.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:33.483 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-16T05:12:32.057598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":276121324,"num_objects":134,"num_object_clones":83,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":11,"num_read":218,"num_read_kb":237002,"num_write":127,"num_write_kb":59017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5623,"ondisk_log_size":5623,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":767972,"kb_used_data":396676,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124173852,"statfs":{"total":1151940427776,"available":1151154024448,"internally_reserved":0,"allocated":406196224,"data_stored":491283274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16881668,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-12,"num_objects_misplaced":-13,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":-365,"num_read_kb":-291988,"num_write":-359,"num_write_kb":-118558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-23,"num_bytes_recovered":-16566440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000808"},"pg_stats":[{"pgid":"2.23","version":"184'17","reported_seq":38,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-16T05:12:27.668189+0000","last_change":"2024-09-16T05:12:26.663186+0000","last_active":"2024-09-16T05:12:27.668189+0000","last_peered":"2024-09-16T05:12:27.668189+0000","last_clean":"2024-09-16T05:12:27.668189+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:12:27.668189+0000","last_undegraded":"2024-09-16T05:12:27.668189+0000","last_fullsized":"2024-09-16T05:12:27.668189+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:12:26.533655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.533655+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:27.131192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073283599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":51,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.674456+0000","last_change":"2024-09-16T05:12:30.674705+0000","last_active":"2024-09-16T05:12:30.674456+0000","last_peered":"2024-09-16T05:12:30.674456+0000","last_clean":"2024-09-16T05:12:30.674456+0000","last_became_active":"2024-09-16T05:12:23.806375+0000","last_became_peered":"2024-09-16T05:12:23.806375+0000","last_unstale":"2024-09-16T05:12:30.674456+0000","last_undegraded":"2024-09-16T05:12:30.674456+0000","last_fullsized":"2024-09-16T05:12:30.674456+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:26.634729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.634729+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:30.220357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027077049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":613,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.662289+0000","last_change":"2024-09-16T05:12:31.662712+0000","last_active":"2024-09-16T05:12:31.662289+0000","last_peered":"2024-09-16T05:12:31.662289+0000","last_clean":"2024-09-16T05:12:31.662289+0000","last_became_active":"2024-09-16T05:12:23.806187+0000","last_became_peered":"2024-09-16T05:12:23.806187+0000","last_unstale":"2024-09-16T05:12:31.662289+0000","last_undegraded":"2024-09-16T05:12:31.662289+0000","last_fullsized":"2024-09-16T05:12:31.662289+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:26.747969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.747969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:38.302169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047617500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"2"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.20","version":"220'41","reported_seq":621,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.671322+0000","last_change":"2024-09-16T05:12:31.671486+0000","last_active":"2024-09-16T05:12:31.671322+0000","last_peered":"2024-09-16T05:12:31.671322+0000","last_clean":"2024-09-16T05:12:31.671322+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:12:31.671322+0000","last_undegraded":"2024-09-16T05:12:31.671322+0000","last_fullsized":"2024-09-16T05:12:31.671322+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"214'37","last_scrub_stamp":"2024-09-16T05:12:27.064860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.064860+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:37.147166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022931869999999999,"stat_sum":{"num_bytes":7916920,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":15203,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1f","length":"2"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.1f","version":"215'148","reported_seq":702,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.910608+0000","last_change":"2024-09-16T05:12:27.189349+0000","last_active":"2024-09-16T05:12:29.910608+0000","last_peered":"2024-09-16T05:12:29.910608+0000","last_clean":"2024-09-16T05:12:29.910608+0000","last_became_active":"2024-09-16T05:12:26.664097+0000","last_became_peered":"2024-09-16T05:12:26.664097+0000","last_unstale":"2024-09-16T05:12:29.910608+0000","last_undegraded":"2024-09-16T05:12:29.910608+0000","last_fullsized":"2024-09-16T05:12:29.910608+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'148","last_scrub_stamp":"2024-09-16T05:12:27.189264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.189264+0000","objects_scrubbed":2,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:34.479443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002527737,"stat_sum":{"num_bytes":6029630,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.1e","version":"215'171","reported_seq":734,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-16T05:12:27.668069+0000","last_change":"2024-09-16T05:12:27.537355+0000","last_active":"2024-09-16T05:12:27.668069+0000","last_peered":"2024-09-16T05:12:27.668069+0000","last_clean":"2024-09-16T05:12:27.668069+0000","last_became_active":"2024-09-16T05:12:23.805930+0000","last_became_peered":"2024-09-16T05:12:23.805930+0000","last_unstale":"2024-09-16T05:12:27.668069+0000","last_undegraded":"2024-09-16T05:12:27.668069+0000","last_fullsized":"2024-09-16T05:12:27.668069+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'171","last_scrub_stamp":"2024-09-16T05:12:27.537258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.537258+0000","objects_scrubbed":7,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.560153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024187700000000002,"stat_sum":{"num_bytes":11208944,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.1d","version":"219'287","reported_seq":846,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.809979+0000","last_change":"2024-09-16T05:12:30.675244+0000","last_active":"2024-09-16T05:12:30.809979+0000","last_peered":"2024-09-16T05:12:30.809979+0000","last_clean":"2024-09-16T05:12:30.809979+0000","last_became_active":"2024-09-16T05:12:27.665044+0000","last_became_peered":"2024-09-16T05:12:27.665044+0000","last_unstale":"2024-09-16T05:12:30.809979+0000","last_undegraded":"2024-09-16T05:12:30.809979+0000","last_fullsized":"2024-09-16T05:12:30.809979+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'276","last_scrub_stamp":"2024-09-16T05:12:29.635450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:29.635450+0000","objects_scrubbed":9,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:54.144142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041690269999999996,"stat_sum":{"num_bytes":20495146,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.1c","version":"219'211","reported_seq":764,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.809148+0000","last_change":"2024-09-16T05:12:30.674385+0000","last_active":"2024-09-16T05:12:30.809148+0000","last_peered":"2024-09-16T05:12:30.809148+0000","last_clean":"2024-09-16T05:12:30.809148+0000","last_became_active":"2024-09-16T05:12:27.664976+0000","last_became_peered":"2024-09-16T05:12:27.664976+0000","last_unstale":"2024-09-16T05:12:30.809148+0000","last_undegraded":"2024-09-16T05:12:30.809148+0000","last_fullsized":"2024-09-16T05:12:30.809148+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'202","last_scrub_stamp":"2024-09-16T05:12:28.627965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.627965+0000","objects_scrubbed":8,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:45.066238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003048355,"stat_sum":{"num_bytes":20813872,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.1b","version":"215'164","reported_seq":736,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-16T05:12:26.669110+0000","last_change":"2024-09-16T05:12:26.669110+0000","last_active":"2024-09-16T05:12:26.669110+0000","last_peered":"2024-09-16T05:12:26.669110+0000","last_clean":"2024-09-16T05:12:26.669110+0000","last_became_active":"2024-09-16T05:12:23.807872+0000","last_became_peered":"2024-09-16T05:12:23.807872+0000","last_unstale":"2024-09-16T05:12:26.669110+0000","last_undegraded":"2024-09-16T05:12:26.669110+0000","last_fullsized":"2024-09-16T05:12:26.669110+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"214'162","last_scrub_stamp":"2024-09-16T05:12:26.326323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.326323+0000","objects_scrubbed":7,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:48.934418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032257660000000001,"stat_sum":{"num_bytes":15081330,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.1a","version":"216'278","reported_seq":814,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-16T05:12:28.451691+0000","last_change":"2024-09-16T05:12:28.451691+0000","last_active":"2024-09-16T05:12:28.451691+0000","last_peered":"2024-09-16T05:12:28.451691+0000","last_clean":"2024-09-16T05:12:28.451691+0000","last_became_active":"2024-09-16T05:12:27.664055+0000","last_became_peered":"2024-09-16T05:12:27.664055+0000","last_unstale":"2024-09-16T05:12:28.451691+0000","last_undegraded":"2024-09-16T05:12:28.451691+0000","last_fullsized":"2024-09-16T05:12:28.451691+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'278","last_scrub_stamp":"2024-09-16T05:12:28.451582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.451582+0000","objects_scrubbed":11,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:35.525627+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.269340843,"stat_sum":{"num_bytes":25800736,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":160,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-16T05:12:27.668031+0000","last_change":"2024-09-16T05:12:26.663641+0000","last_active":"2024-09-16T05:12:27.668031+0000","last_peered":"2024-09-16T05:12:27.668031+0000","last_clean":"2024-09-16T05:12:27.668031+0000","last_became_active":"2024-09-16T05:11:57.658646+0000","last_became_peered":"2024-09-16T05:11:57.658646+0000","last_unstale":"2024-09-16T05:12:27.668031+0000","last_undegraded":"2024-09-16T05:12:27.668031+0000","last_fullsized":"2024-09-16T05:12:27.668031+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:12:09.470099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.470099+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.990181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001085073,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":559,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.687955+0000","last_change":"2024-09-16T05:12:26.664634+0000","last_active":"2024-09-16T05:12:29.687955+0000","last_peered":"2024-09-16T05:12:29.687955+0000","last_clean":"2024-09-16T05:12:29.687955+0000","last_became_active":"2024-09-16T05:11:57.660350+0000","last_became_peered":"2024-09-16T05:11:57.660350+0000","last_unstale":"2024-09-16T05:12:29.687955+0000","last_undegraded":"2024-09-16T05:12:29.687955+0000","last_fullsized":"2024-09-16T05:12:29.687955+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:12:08.969861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.969861+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:16.343042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052395499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.17","version":"0'0","reported_seq":339,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-16T05:12:26.663151+0000","last_change":"2024-09-16T05:12:26.663482+0000","last_active":"2024-09-16T05:12:26.663151+0000","last_peered":"2024-09-16T05:12:26.663151+0000","last_clean":"2024-09-16T05:12:26.663151+0000","last_became_active":"2024-09-16T05:11:57.657217+0000","last_became_peered":"2024-09-16T05:11:57.657217+0000","last_unstale":"2024-09-16T05:12:26.663151+0000","last_undegraded":"2024-09-16T05:12:26.663151+0000","last_fullsized":"2024-09-16T05:12:26.663151+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:12:09.877069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.877069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:35.060431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059807100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.16","version":"116'9","reported_seq":186,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.670571+0000","last_change":"2024-09-16T05:12:30.670901+0000","last_active":"2024-09-16T05:12:30.670571+0000","last_peered":"2024-09-16T05:12:30.670571+0000","last_clean":"2024-09-16T05:12:30.670571+0000","last_became_active":"2024-09-16T05:11:57.656772+0000","last_became_peered":"2024-09-16T05:11:57.656772+0000","last_unstale":"2024-09-16T05:12:30.670571+0000","last_undegraded":"2024-09-16T05:12:30.670571+0000","last_fullsized":"2024-09-16T05:12:30.670571+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:12:09.331857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.331857+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:31.550720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079200400000000002,"stat_sum":{"num_bytes":2857157,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":26290,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.15","version":"218'309","reported_seq":850,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.688109+0000","last_change":"2024-09-16T05:12:26.668521+0000","last_active":"2024-09-16T05:12:29.688109+0000","last_peered":"2024-09-16T05:12:29.688109+0000","last_clean":"2024-09-16T05:12:29.688109+0000","last_became_active":"2024-09-16T05:12:00.711890+0000","last_became_peered":"2024-09-16T05:12:00.711890+0000","last_unstale":"2024-09-16T05:12:29.688109+0000","last_undegraded":"2024-09-16T05:12:29.688109+0000","last_fullsized":"2024-09-16T05:12:29.688109+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'276","last_scrub_stamp":"2024-09-16T05:12:11.920515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:11.920515+0000","objects_scrubbed":7,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:31.866852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027592179999999999,"stat_sum":{"num_bytes":18740340,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":21,"num_read_kb":40809,"num_write":7,"num_write_kb":4969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.14","version":"217'118","reported_seq":252,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.688096+0000","last_change":"2024-09-16T05:12:26.665055+0000","last_active":"2024-09-16T05:12:29.688096+0000","last_peered":"2024-09-16T05:12:29.688096+0000","last_clean":"2024-09-16T05:12:29.688096+0000","last_became_active":"2024-09-16T05:11:57.658852+0000","last_became_peered":"2024-09-16T05:11:57.658852+0000","last_unstale":"2024-09-16T05:12:29.688096+0000","last_undegraded":"2024-09-16T05:12:29.688096+0000","last_fullsized":"2024-09-16T05:12:29.688096+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'108","last_scrub_stamp":"2024-09-16T05:12:10.924782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.924782+0000","objects_scrubbed":6,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:24.021513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00047423000000000002,"stat_sum":{"num_bytes":15813746,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":31483,"num_write":1,"num_write_kb":3546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.13","version":"181'15","reported_seq":166,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.688098+0000","last_change":"2024-09-16T05:12:26.665285+0000","last_active":"2024-09-16T05:12:29.688098+0000","last_peered":"2024-09-16T05:12:29.688098+0000","last_clean":"2024-09-16T05:12:29.688098+0000","last_became_active":"2024-09-16T05:11:57.660401+0000","last_became_peered":"2024-09-16T05:11:57.660401+0000","last_unstale":"2024-09-16T05:12:29.688098+0000","last_undegraded":"2024-09-16T05:12:29.688098+0000","last_fullsized":"2024-09-16T05:12:29.688098+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:12:09.952097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.952097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:34.150574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00067967600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.12","version":"219'34","reported_seq":199,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.676213+0000","last_change":"2024-09-16T05:12:30.676455+0000","last_active":"2024-09-16T05:12:30.676213+0000","last_peered":"2024-09-16T05:12:30.676213+0000","last_clean":"2024-09-16T05:12:30.676213+0000","last_became_active":"2024-09-16T05:11:57.658975+0000","last_became_peered":"2024-09-16T05:11:57.658975+0000","last_unstale":"2024-09-16T05:12:30.676213+0000","last_undegraded":"2024-09-16T05:12:30.676213+0000","last_fullsized":"2024-09-16T05:12:30.676213+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'28","last_scrub_stamp":"2024-09-16T05:12:09.669066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.669066+0000","objects_scrubbed":3,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:31.104804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0042118340000000002,"stat_sum":{"num_bytes":4894218,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":6198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":639,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.687887+0000","last_change":"2024-09-16T05:12:13.922480+0000","last_active":"2024-09-16T05:12:29.687887+0000","last_peered":"2024-09-16T05:12:29.687887+0000","last_clean":"2024-09-16T05:12:29.687887+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:12:29.687887+0000","last_undegraded":"2024-09-16T05:12:29.687887+0000","last_fullsized":"2024-09-16T05:12:29.687887+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.922401+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.922401+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:15.772375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"218'137","reported_seq":692,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.715038+0000","last_change":"2024-09-16T05:12:26.664912+0000","last_active":"2024-09-16T05:12:29.715038+0000","last_peered":"2024-09-16T05:12:29.715038+0000","last_clean":"2024-09-16T05:12:29.715038+0000","last_became_active":"2024-09-16T05:11:55.657137+0000","last_became_peered":"2024-09-16T05:11:55.657137+0000","last_unstale":"2024-09-16T05:12:29.715038+0000","last_undegraded":"2024-09-16T05:12:29.715038+0000","last_fullsized":"2024-09-16T05:12:29.715038+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'70","last_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.116608+0000","objects_scrubbed":7,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:05.599539+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.0044142160000000003,"stat_sum":{"num_bytes":20031342,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":14,"num_read_kb":6175,"num_write":17,"num_write_kb":7406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":637,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.669594+0000","last_change":"2024-09-16T05:10:43.759254+0000","last_active":"2024-09-16T05:12:31.669594+0000","last_peered":"2024-09-16T05:12:31.669594+0000","last_clean":"2024-09-16T05:12:31.669594+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:12:31.669594+0000","last_undegraded":"2024-09-16T05:12:31.669594+0000","last_fullsized":"2024-09-16T05:12:31.669594+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:43.759130+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:43.759130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"215'274","reported_seq":808,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-16T05:12:26.670915+0000","last_change":"2024-09-16T05:12:26.671074+0000","last_active":"2024-09-16T05:12:26.670915+0000","last_peered":"2024-09-16T05:12:26.670915+0000","last_clean":"2024-09-16T05:12:26.670915+0000","last_became_active":"2024-09-16T05:12:21.665466+0000","last_became_peered":"2024-09-16T05:12:21.665466+0000","last_unstale":"2024-09-16T05:12:26.670915+0000","last_undegraded":"2024-09-16T05:12:26.670915+0000","last_fullsized":"2024-09-16T05:12:26.670915+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'144","last_scrub_stamp":"2024-09-16T05:11:46.247408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:46.247408+0000","objects_scrubbed":15,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0042280599999999996,"stat_sum":{"num_bytes":21745455,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":83232,"num_write":60,"num_write_kb":22584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":637,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.670035+0000","last_change":"2024-09-16T05:12:13.374972+0000","last_active":"2024-09-16T05:12:30.670035+0000","last_peered":"2024-09-16T05:12:30.670035+0000","last_clean":"2024-09-16T05:12:30.670035+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:12:30.670035+0000","last_undegraded":"2024-09-16T05:12:30.670035+0000","last_fullsized":"2024-09-16T05:12:30.670035+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.374887+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.374887+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:41.588892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"215'163","reported_seq":707,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-16T05:12:27.662007+0000","last_change":"2024-09-16T05:12:26.665388+0000","last_active":"2024-09-16T05:12:27.662007+0000","last_peered":"2024-09-16T05:12:27.662007+0000","last_clean":"2024-09-16T05:12:27.662007+0000","last_became_active":"2024-09-16T05:12:21.661312+0000","last_became_peered":"2024-09-16T05:12:21.661312+0000","last_unstale":"2024-09-16T05:12:27.662007+0000","last_undegraded":"2024-09-16T05:12:27.662007+0000","last_fullsized":"2024-09-16T05:12:27.662007+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'161","last_scrub_stamp":"2024-09-16T05:12:26.506811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.506811+0000","objects_scrubbed":4,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:41.416776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.004625194,"stat_sum":{"num_bytes":4261034,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":554,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.671080+0000","last_change":"2024-09-16T05:11:27.759078+0000","last_active":"2024-09-16T05:12:30.671080+0000","last_peered":"2024-09-16T05:12:30.671080+0000","last_clean":"2024-09-16T05:12:30.671080+0000","last_became_active":"2024-09-16T05:11:21.676107+0000","last_became_peered":"2024-09-16T05:11:21.676107+0000","last_unstale":"2024-09-16T05:12:30.671080+0000","last_undegraded":"2024-09-16T05:12:30.671080+0000","last_fullsized":"2024-09-16T05:12:30.671080+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:27.759006+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:27.759006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:12:36.703164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":623,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.664316+0000","last_change":"2024-09-16T05:12:26.661125+0000","last_active":"2024-09-16T05:12:29.664316+0000","last_peered":"2024-09-16T05:12:29.664316+0000","last_clean":"2024-09-16T05:12:29.664316+0000","last_became_active":"2024-09-16T05:11:55.665450+0000","last_became_peered":"2024-09-16T05:11:55.665450+0000","last_unstale":"2024-09-16T05:12:29.664316+0000","last_undegraded":"2024-09-16T05:12:29.664316+0000","last_fullsized":"2024-09-16T05:12:29.664316+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:12:10.025043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.025043+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:24.800973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061671399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":642,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.671731+0000","last_change":"2024-09-16T05:12:12.696846+0000","last_active":"2024-09-16T05:12:30.671731+0000","last_peered":"2024-09-16T05:12:30.671731+0000","last_clean":"2024-09-16T05:12:30.671731+0000","last_became_active":"2024-09-16T05:10:38.972286+0000","last_became_peered":"2024-09-16T05:10:38.972286+0000","last_unstale":"2024-09-16T05:12:30.671731+0000","last_undegraded":"2024-09-16T05:12:30.671731+0000","last_fullsized":"2024-09-16T05:12:30.671731+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:12.696764+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:12.696764+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:35.404312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"215'338","reported_seq":859,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.687990+0000","last_change":"2024-09-16T05:12:26.669234+0000","last_active":"2024-09-16T05:12:29.687990+0000","last_peered":"2024-09-16T05:12:29.687990+0000","last_clean":"2024-09-16T05:12:29.687990+0000","last_became_active":"2024-09-16T05:12:12.663643+0000","last_became_peered":"2024-09-16T05:12:12.663643+0000","last_unstale":"2024-09-16T05:12:29.687990+0000","last_undegraded":"2024-09-16T05:12:29.687990+0000","last_fullsized":"2024-09-16T05:12:29.687990+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'266","last_scrub_stamp":"2024-09-16T05:11:56.487686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.487686+0000","objects_scrubbed":7,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:14.002645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038186169999999998,"stat_sum":{"num_bytes":24952507,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":24,"num_read_kb":23922,"num_write":15,"num_write_kb":9870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":642,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.670817+0000","last_change":"2024-09-16T05:12:11.705573+0000","last_active":"2024-09-16T05:12:30.670817+0000","last_peered":"2024-09-16T05:12:30.670817+0000","last_clean":"2024-09-16T05:12:30.670817+0000","last_became_active":"2024-09-16T05:10:38.972150+0000","last_became_peered":"2024-09-16T05:10:38.972150+0000","last_unstale":"2024-09-16T05:12:30.670817+0000","last_undegraded":"2024-09-16T05:12:30.670817+0000","last_fullsized":"2024-09-16T05:12:30.670817+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:11.705505+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:11.705505+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:26.546601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":549,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.664395+0000","last_change":"2024-09-16T05:12:26.661706+0000","last_active":"2024-09-16T05:12:29.664395+0000","last_peered":"2024-09-16T05:12:29.664395+0000","last_clean":"2024-09-16T05:12:29.664395+0000","last_became_active":"2024-09-16T05:11:55.657738+0000","last_became_peered":"2024-09-16T05:11:55.657738+0000","last_unstale":"2024-09-16T05:12:29.664395+0000","last_undegraded":"2024-09-16T05:12:29.664395+0000","last_fullsized":"2024-09-16T05:12:29.664395+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:12:08.997285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.997285+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:30.047632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011749830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":592,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-16T05:12:26.666212+0000","last_change":"2024-09-16T05:12:09.317163+0000","last_active":"2024-09-16T05:12:26.666212+0000","last_peered":"2024-09-16T05:12:26.666212+0000","last_clean":"2024-09-16T05:12:26.666212+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:12:26.666212+0000","last_undegraded":"2024-09-16T05:12:26.666212+0000","last_fullsized":"2024-09-16T05:12:26.666212+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:09.317055+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.317055+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:18.430868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":361,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.671541+0000","last_change":"2024-09-16T05:12:30.671989+0000","last_active":"2024-09-16T05:12:30.671541+0000","last_peered":"2024-09-16T05:12:30.671541+0000","last_clean":"2024-09-16T05:12:30.671541+0000","last_became_active":"2024-09-16T05:11:55.659156+0000","last_became_peered":"2024-09-16T05:11:55.659156+0000","last_unstale":"2024-09-16T05:12:30.671541+0000","last_undegraded":"2024-09-16T05:12:30.671541+0000","last_fullsized":"2024-09-16T05:12:30.671541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:11:56.378387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.378387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:11.413763+0000","scrub_duration":686,"objects_trimmed":0,"snaptrim_duration":0.00067977499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.0","version":"205'33","reported_seq":596,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.671552+0000","last_change":"2024-09-16T05:12:31.671868+0000","last_active":"2024-09-16T05:12:31.671552+0000","last_peered":"2024-09-16T05:12:31.671552+0000","last_clean":"2024-09-16T05:12:31.671552+0000","last_became_active":"2024-09-16T05:12:21.667216+0000","last_became_peered":"2024-09-16T05:12:21.667216+0000","last_unstale":"2024-09-16T05:12:31.671552+0000","last_undegraded":"2024-09-16T05:12:31.671552+0000","last_fullsized":"2024-09-16T05:12:31.671552+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:12:28.044321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.044321+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.406425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002140571,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1f","length":"2"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":638,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-16T05:12:27.667997+0000","last_change":"2024-09-16T05:11:53.500880+0000","last_active":"2024-09-16T05:12:27.667997+0000","last_peered":"2024-09-16T05:12:27.667997+0000","last_clean":"2024-09-16T05:12:27.667997+0000","last_became_active":"2024-09-16T05:11:19.419149+0000","last_became_peered":"2024-09-16T05:11:19.419149+0000","last_unstale":"2024-09-16T05:12:27.667997+0000","last_undegraded":"2024-09-16T05:12:27.667997+0000","last_fullsized":"2024-09-16T05:12:27.667997+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:53.500734+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:53.500734+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:08.214137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":610,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.662233+0000","last_change":"2024-09-16T05:12:31.662706+0000","last_active":"2024-09-16T05:12:31.662233+0000","last_peered":"2024-09-16T05:12:31.662233+0000","last_clean":"2024-09-16T05:12:31.662233+0000","last_became_active":"2024-09-16T05:12:21.666300+0000","last_became_peered":"2024-09-16T05:12:21.666300+0000","last_unstale":"2024-09-16T05:12:31.662233+0000","last_undegraded":"2024-09-16T05:12:31.662233+0000","last_fullsized":"2024-09-16T05:12:31.662233+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:25.777434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:25.777434+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:26.761099+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000441056,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"2"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"1.2","version":"35'139","reported_seq":645,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.669143+0000","last_change":"2024-09-16T05:12:31.125618+0000","last_active":"2024-09-16T05:12:31.669143+0000","last_peered":"2024-09-16T05:12:31.669143+0000","last_clean":"2024-09-16T05:12:31.669143+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:12:31.669143+0000","last_undegraded":"2024-09-16T05:12:31.669143+0000","last_fullsized":"2024-09-16T05:12:31.669143+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:31.125520+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:31.125520+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.623317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":641,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.671615+0000","last_change":"2024-09-16T05:11:51.723968+0000","last_active":"2024-09-16T05:12:30.671615+0000","last_peered":"2024-09-16T05:12:30.671615+0000","last_clean":"2024-09-16T05:12:30.671615+0000","last_became_active":"2024-09-16T05:10:38.972381+0000","last_became_peered":"2024-09-16T05:10:38.972381+0000","last_unstale":"2024-09-16T05:12:30.671615+0000","last_undegraded":"2024-09-16T05:12:30.671615+0000","last_fullsized":"2024-09-16T05:12:30.671615+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:51.723892+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:51.723892+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:12.858953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":564,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.670851+0000","last_change":"2024-09-16T05:12:30.671059+0000","last_active":"2024-09-16T05:12:30.670851+0000","last_peered":"2024-09-16T05:12:30.670851+0000","last_clean":"2024-09-16T05:12:30.670851+0000","last_became_active":"2024-09-16T05:12:21.660508+0000","last_became_peered":"2024-09-16T05:12:21.660508+0000","last_unstale":"2024-09-16T05:12:30.670851+0000","last_undegraded":"2024-09-16T05:12:30.670851+0000","last_fullsized":"2024-09-16T05:12:30.670851+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:27.335469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.335469+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.755970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071066599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":642,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.669696+0000","last_change":"2024-09-16T05:11:54.869494+0000","last_active":"2024-09-16T05:12:31.669696+0000","last_peered":"2024-09-16T05:12:31.669696+0000","last_clean":"2024-09-16T05:12:31.669696+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:12:31.669696+0000","last_undegraded":"2024-09-16T05:12:31.669696+0000","last_fullsized":"2024-09-16T05:12:31.669696+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:54.869407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:54.869407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:21.879716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"213'20","reported_seq":571,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-16T05:12:30.670508+0000","last_change":"2024-09-16T05:12:30.670871+0000","last_active":"2024-09-16T05:12:30.670508+0000","last_peered":"2024-09-16T05:12:30.670508+0000","last_clean":"2024-09-16T05:12:30.670508+0000","last_became_active":"2024-09-16T05:12:21.663580+0000","last_became_peered":"2024-09-16T05:12:21.663580+0000","last_unstale":"2024-09-16T05:12:30.670508+0000","last_undegraded":"2024-09-16T05:12:30.670508+0000","last_fullsized":"2024-09-16T05:12:30.670508+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-09-16T05:12:26.380594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.380594+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:32.830258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000671073,"stat_sum":{"num_bytes":6453855,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"20","length":"1"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.8","version":"220'93","reported_seq":642,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.671373+0000","last_change":"2024-09-16T05:12:31.671670+0000","last_active":"2024-09-16T05:12:31.671373+0000","last_peered":"2024-09-16T05:12:31.671373+0000","last_clean":"2024-09-16T05:12:31.671373+0000","last_became_active":"2024-09-16T05:11:55.658383+0000","last_became_peered":"2024-09-16T05:11:55.658383+0000","last_unstale":"2024-09-16T05:12:31.671373+0000","last_undegraded":"2024-09-16T05:12:31.671373+0000","last_fullsized":"2024-09-16T05:12:31.671373+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'67","last_scrub_stamp":"2024-09-16T05:12:09.039188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.039188+0000","objects_scrubbed":4,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:30.826978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002081781,"stat_sum":{"num_bytes":8437081,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1f","length":"2"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.c","version":"215'183","reported_seq":702,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-16T05:12:27.313546+0000","last_change":"2024-09-16T05:12:27.313580+0000","last_active":"2024-09-16T05:12:27.313546+0000","last_peered":"2024-09-16T05:12:27.313546+0000","last_clean":"2024-09-16T05:12:27.313546+0000","last_became_active":"2024-09-16T05:12:21.664367+0000","last_became_peered":"2024-09-16T05:12:21.664367+0000","last_unstale":"2024-09-16T05:12:27.313546+0000","last_undegraded":"2024-09-16T05:12:27.313546+0000","last_fullsized":"2024-09-16T05:12:27.313546+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'183","last_scrub_stamp":"2024-09-16T05:12:27.313488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.313488+0000","objects_scrubbed":3,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:31.342817+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0041352660000000003,"stat_sum":{"num_bytes":4997813,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.d","version":"218'270","reported_seq":813,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-16T05:12:29.689877+0000","last_change":"2024-09-16T05:12:26.668995+0000","last_active":"2024-09-16T05:12:29.689877+0000","last_peered":"2024-09-16T05:12:29.689877+0000","last_clean":"2024-09-16T05:12:29.689877+0000","last_became_active":"2024-09-16T05:12:21.707118+0000","last_became_peered":"2024-09-16T05:12:21.707118+0000","last_unstale":"2024-09-16T05:12:29.689877+0000","last_undegraded":"2024-09-16T05:12:29.689877+0000","last_fullsized":"2024-09-16T05:12:29.689877+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'262","last_scrub_stamp":"2024-09-16T05:12:25.983777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:25.983777+0000","objects_scrubbed":3,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:42.142334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030170570000000001,"stat_sum":{"num_bytes":10254033,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.e","version":"220'165","reported_seq":729,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-16T05:12:31.664271+0000","last_change":"2024-09-16T05:12:31.664773+0000","last_active":"2024-09-16T05:12:31.664271+0000","last_peered":"2024-09-16T05:12:31.664271+0000","last_clean":"2024-09-16T05:12:31.664271+0000","last_became_active":"2024-09-16T05:12:21.665948+0000","last_became_peered":"2024-09-16T05:12:21.665948+0000","last_unstale":"2024-09-16T05:12:31.664271+0000","last_undegraded":"2024-09-16T05:12:31.664271+0000","last_fullsized":"2024-09-16T05:12:31.664271+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'160","last_scrub_stamp":"2024-09-16T05:12:27.786933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.786933+0000","objects_scrubbed":2,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.875915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002682472,"stat_sum":{"num_bytes":4504243,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":3685,"num_write":1,"num_write_kb":2003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f","length":"2"},{"start":"22","length":"1"},{"start":"31","length":"1"}]},{"pgid":"2.f","version":"215'165","reported_seq":714,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-16T05:12:27.662219+0000","last_change":"2024-09-16T05:12:27.494380+0000","last_active":"2024-09-16T05:12:27.662219+0000","last_peered":"2024-09-16T05:12:27.662219+0000","last_clean":"2024-09-16T05:12:27.662219+0000","last_became_active":"2024-09-16T05:12:21.662019+0000","last_became_peered":"2024-09-16T05:12:21.662019+0000","last_unstale":"2024-09-16T05:12:27.662219+0000","last_undegraded":"2024-09-16T05:12:27.662219+0000","last_fullsized":"2024-09-16T05:12:27.662219+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'165","last_scrub_stamp":"2024-09-16T05:12:27.494268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.494268+0000","objects_scrubbed":8,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:32.570372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00540165,"stat_sum":{"num_bytes":14275102,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":160,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-16T05:12:27.661900+0000","last_change":"2024-09-16T05:12:26.661528+0000","last_active":"2024-09-16T05:12:27.661900+0000","last_peered":"2024-09-16T05:12:27.661900+0000","last_clean":"2024-09-16T05:12:27.661900+0000","last_became_active":"2024-09-16T05:11:57.658263+0000","last_became_peered":"2024-09-16T05:11:57.658263+0000","last_unstale":"2024-09-16T05:12:27.661900+0000","last_undegraded":"2024-09-16T05:12:27.661900+0000","last_fullsized":"2024-09-16T05:12:27.661900+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:12:09.468817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.468817+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:17.515086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077265700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.11","version":"215'80","reported_seq":208,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-09-16T05:12:26.664321+0000","last_change":"2024-09-16T05:12:26.664496+0000","last_active":"2024-09-16T05:12:26.664321+0000","last_peered":"2024-09-16T05:12:26.664321+0000","last_clean":"2024-09-16T05:12:26.664321+0000","last_became_active":"2024-09-16T05:11:57.662582+0000","last_became_peered":"2024-09-16T05:11:57.662582+0000","last_unstale":"2024-09-16T05:12:26.664321+0000","last_undegraded":"2024-09-16T05:12:26.664321+0000","last_fullsized":"2024-09-16T05:12:26.664321+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'64","last_scrub_stamp":"2024-09-16T05:12:08.859612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.859612+0000","objects_scrubbed":3,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:15.462259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016533959999999999,"stat_sum":{"num_bytes":5966452,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"1"},{"start":"29","length":"1"},{"start":"2b","length":"1"},{"start":"30","length":"3"},{"start":"34","length":"1"},{"start":"36","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":36,"stat_sum":{"num_bytes":275530956,"num_objects":132,"num_object_clones":83,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":11,"num_read":218,"num_read_kb":237002,"num_write":127,"num_write_kb":59017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362487808,"data_stored":454355475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4094,"ondisk_log_size":4094,"up":72,"acting":72,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089384,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97176,"kb_used_data":65980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93647976,"statfs":{"total":95995035648,"available":95895527424,"internally_reserved":0,"allocated":67563520,"data_stored":93691898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285614,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62600,"kb_used_data":31408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682552,"statfs":{"total":95995035648,"available":95930933248,"internally_reserved":0,"allocated":32161792,"data_stored":38332878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514548,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97820,"kb_used_data":66628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93647332,"statfs":{"total":95995035648,"available":95894867968,"internally_reserved":0,"allocated":68227072,"data_stored":79713133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841590,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60980,"kb_used_data":28632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93684172,"statfs":{"total":95995035648,"available":95932592128,"internally_reserved":0,"allocated":29319168,"data_stored":35619957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103228,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86996,"kb_used_data":55812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93658156,"statfs":{"total":95995035648,"available":95905951744,"internally_reserved":0,"allocated":57151488,"data_stored":68913250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397570,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61576,"kb_used_data":30384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683576,"statfs":{"total":95995035648,"available":95931981824,"internally_reserved":0,"allocated":31113216,"data_stored":33721514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40468,"kb_used_data":13432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93704684,"statfs":{"total":95995035648,"available":95953596416,"internally_reserved":0,"allocated":13754368,"data_stored":13344145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37776,"kb_used_data":10744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93707376,"statfs":{"total":95995035648,"available":95956353024,"internally_reserved":0,"allocated":11001856,"data_stored":11421917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117072,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67496,"kb_used_data":36308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93677656,"statfs":{"total":95995035648,"available":95925919744,"internally_reserved":0,"allocated":37179392,"data_stored":51384394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313303,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40976,"kb_used_data":9784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93704176,"statfs":{"total":95995035648,"available":95953076224,"internally_reserved":0,"allocated":10018816,"data_stored":10224377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59320,"kb_used_data":28128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93685832,"statfs":{"total":95995035648,"available":95934291968,"internally_reserved":0,"allocated":28803072,"data_stored":34628950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54788,"kb_used_data":19436,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690364,"statfs":{"total":95995035648,"available":95938932736,"internally_reserved":0,"allocated":19902464,"data_stored":20286861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17362944,"data_stored":18775150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26263552,"data_stored":33117239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7249920,"data_stored":8494182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20729856,"data_stored":30434836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8462336,"data_stored":9910206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11010048,"data_stored":11631186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28610560,"data_stored":32235300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":54599680,"data_stored":67392788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":26476544,"data_stored":33797520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":65687552,"data_stored":78201422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":31301632,"data_stored":38487642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":64733184,"data_stored":91878004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:12:33.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-16T05:12:33.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:33.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:33.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:33.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:33.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:33.667 INFO:tasks.rados.rados.2.smithi204.stdout:580: done (2 left) 2024-09-16T05:12:33.667 INFO:tasks.rados.rados.2.smithi204.stdout:581: done (1 left) 2024-09-16T05:12:33.667 INFO:tasks.rados.rados.2.smithi204.stdout:582: done (0 left) 2024-09-16T05:12:33.667 INFO:tasks.rados.rados.2.smithi204.stdout:583: snap_create 2024-09-16T05:12:33.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:33.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:33.702 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:33.702 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:33.782 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:33.804 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:33.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:33.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:33.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:33.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:34.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: pgmap v483: 47 pgs: 47 active+clean; 263 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/258 objects degraded (4.651%), 1 pg degraded) 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:34.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:33 smithi204 ceph-mon[30933]: osdmap e222: 12 total, 12 up, 12 in 2024-09-16T05:12:34.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:34.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:34.072 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:34.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: pgmap v483: 47 pgs: 47 active+clean; 263 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/258 objects degraded (4.651%), 1 pg degraded) 2024-09-16T05:12:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:12:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:33 smithi032 ceph-mon[27388]: osdmap e222: 12 total, 12 up, 12 in 2024-09-16T05:12:34.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:34.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: pgmap v483: 47 pgs: 47 active+clean; 263 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/258 objects degraded (4.651%), 1 pg degraded) 2024-09-16T05:12:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:12:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:33 smithi196 ceph-mon[30802]: osdmap e222: 12 total, 12 up, 12 in 2024-09-16T05:12:34.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:34.313 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:34.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:34.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:34.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:12:34.416 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:34.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:34.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:34.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:34.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:34.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:34.598 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:34.665 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:12:34.668 INFO:tasks.rados.rados.2.smithi204.stdout:583: done (0 left) 2024-09-16T05:12:34.668 INFO:tasks.rados.rados.2.smithi204.stdout:584: read oid 23 snap -1 2024-09-16T05:12:34.668 INFO:tasks.rados.rados.2.smithi204.stdout:584: expect deleted 2024-09-16T05:12:34.668 INFO:tasks.rados.rados.2.smithi204.stdout:585: snap_remove snap 57 2024-09-16T05:12:34.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:34.696 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:34.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:34.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:34.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:34.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:34.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:12:34.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:34.949 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:34.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:35.027 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1501880260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:35.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:34 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:35.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1501880260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:35.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:34 smithi204 ceph-mon[30933]: osdmap e223: 12 total, 12 up, 12 in 2024-09-16T05:12:35.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2257379272' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-16T05:12:35.033 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:35.033 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:35.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:35.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1501880260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:34 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1501880260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:34 smithi196 ceph-mon[30802]: osdmap e223: 12 total, 12 up, 12 in 2024-09-16T05:12:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2257379272' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-16T05:12:35.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1501880260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:35.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:34 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:35.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1501880260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:35.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:34 smithi032 ceph-mon[27388]: osdmap e223: 12 total, 12 up, 12 in 2024-09-16T05:12:35.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2257379272' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-16T05:12:35.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:35.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:35.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:35.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:35.328 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:35.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:35.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:35.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:35.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:35.549 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:35.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:35.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:35.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:35.661 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 1 pgp_num to 11 2024-09-16T05:12:35.664 INFO:tasks.rados.rados.2.smithi204.stdout:585: done (1 left) 2024-09-16T05:12:35.664 INFO:tasks.rados.rados.2.smithi204.stdout:586: write (excl) oid 17 current snap is 64 2024-09-16T05:12:35.664 INFO:tasks.rados.rados.2.smithi204.stdout:586: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-16T05:12:35.671 INFO:tasks.rados.rados.2.smithi204.stdout:586: writing smithi20457400-17 from 699224 to 1247582 tid 1 2024-09-16T05:12:35.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:35.678 INFO:tasks.rados.rados.2.smithi204.stdout:586: writing smithi20457400-17 from 2026293 to 2488367 tid 2 2024-09-16T05:12:35.685 INFO:tasks.rados.rados.2.smithi204.stdout:586: writing smithi20457400-17 from 3271546 to 3704300 tid 3 2024-09-16T05:12:35.685 INFO:tasks.rados.rados.2.smithi204.stdout:587: read oid 47 snap -1 2024-09-16T05:12:35.685 INFO:tasks.rados.rados.2.smithi204.stdout:587: expect deleted 2024-09-16T05:12:35.685 INFO:tasks.rados.rados.2.smithi204.stdout:588: write oid 3 current snap is 64 2024-09-16T05:12:35.685 INFO:tasks.rados.rados.2.smithi204.stdout:588: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-09-16T05:12:35.692 INFO:tasks.rados.rados.2.smithi204.stdout:588: writing smithi20457400-3 from 716031 to 1339638 tid 1 2024-09-16T05:12:35.701 INFO:tasks.rados.rados.2.smithi204.stdout:588: writing smithi20457400-3 from 2084775 to 2690507 tid 2 2024-09-16T05:12:35.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:35.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:588: writing smithi20457400-3 from 3298969 to 3916031 tid 3 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:589: rollback oid 4 current snap is 64 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 53 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:586: finishing write tid 1 to smithi20457400-17 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:586: finishing write tid 2 to smithi20457400-17 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:586: finishing write tid 3 to smithi20457400-17 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:586: finishing write tid 4 to smithi20457400-17 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:586: finishing write tid 5 to smithi20457400-17 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 257 (ObjNum 232 snap 64 seq_num 232) dirty exists 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:586: left oid 17 (ObjNum 232 snap 64 seq_num 232) 2024-09-16T05:12:35.709 INFO:tasks.rados.rados.2.smithi204.stdout:588: finishing write tid 1 to smithi20457400-3 2024-09-16T05:12:35.710 INFO:tasks.rados.rados.2.smithi204.stdout:588: finishing write tid 2 to smithi20457400-3 2024-09-16T05:12:35.710 INFO:tasks.rados.rados.2.smithi204.stdout:584: done (4 left) 2024-09-16T05:12:35.710 INFO:tasks.rados.rados.2.smithi204.stdout:586: done (3 left) 2024-09-16T05:12:35.710 INFO:tasks.rados.rados.2.smithi204.stdout:587: done (2 left) 2024-09-16T05:12:35.710 INFO:tasks.rados.rados.2.smithi204.stdout:590: snap_create 2024-09-16T05:12:35.711 INFO:tasks.rados.rados.2.smithi204.stdout:588: finishing write tid 3 to smithi20457400-3 2024-09-16T05:12:35.712 INFO:tasks.rados.rados.2.smithi204.stdout:588: finishing write tid 4 to smithi20457400-3 2024-09-16T05:12:35.712 INFO:tasks.rados.rados.2.smithi204.stdout:589: finishing rollback tid 0 to smithi20457400-4 2024-09-16T05:12:35.712 INFO:tasks.rados.rados.2.smithi204.stdout:589: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:12:35.712 INFO:tasks.rados.rados.2.smithi204.stdout:588: finishing write tid 5 to smithi20457400-3 2024-09-16T05:12:35.712 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 359 (ObjNum 233 snap 64 seq_num 233) dirty exists 2024-09-16T05:12:35.712 INFO:tasks.rados.rados.2.smithi204.stdout:588: left oid 3 (ObjNum 233 snap 64 seq_num 233) 2024-09-16T05:12:35.712 INFO:tasks.rados.rados.2.smithi204.stdout:589: finishing rollback tid 2 to smithi20457400-4 2024-09-16T05:12:35.712 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 187 (ObjNum 181 snap 46 seq_num 181) dirty exists 2024-09-16T05:12:35.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:12:35.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:35.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:35.855 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:35.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:35 smithi032 ceph-mon[27388]: pgmap v486: 47 pgs: 47 active+clean; 263 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2257379272' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-16T05:12:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:35 smithi032 ceph-mon[27388]: osdmap e224: 12 total, 12 up, 12 in 2024-09-16T05:12:35.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:35.979 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:35.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:36.015 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:36.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:36.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:36.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:36.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:35 smithi204 ceph-mon[30933]: pgmap v486: 47 pgs: 47 active+clean; 263 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:36.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2257379272' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-16T05:12:36.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:35 smithi204 ceph-mon[30933]: osdmap e224: 12 total, 12 up, 12 in 2024-09-16T05:12:36.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:36.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:35 smithi196 ceph-mon[30802]: pgmap v486: 47 pgs: 47 active+clean; 263 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2257379272' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-16T05:12:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:35 smithi196 ceph-mon[30802]: osdmap e224: 12 total, 12 up, 12 in 2024-09-16T05:12:36.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:36.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:36.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:36.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:36.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:36.503 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:36.503 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:36.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:36.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:36.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:36.654 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:36.665 INFO:tasks.rados.rados.2.smithi204.stdout:588: done (2 left) 2024-09-16T05:12:36.665 INFO:tasks.rados.rados.2.smithi204.stdout:589: done (1 left) 2024-09-16T05:12:36.665 INFO:tasks.rados.rados.2.smithi204.stdout:590: done (0 left) 2024-09-16T05:12:36.665 INFO:tasks.rados.rados.2.smithi204.stdout:591: write (excl) oid 45 current snap is 65 2024-09-16T05:12:36.665 INFO:tasks.rados.rados.2.smithi204.stdout:591: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-16T05:12:36.672 INFO:tasks.rados.rados.2.smithi204.stdout:591: writing smithi20457400-45 from 732838 to 1431694 tid 1 2024-09-16T05:12:36.676 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:36.677 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:36.678 INFO:tasks.rados.rados.2.smithi204.stdout:591: writing smithi20457400-45 from 2026904 to 2492647 tid 2 2024-09-16T05:12:36.686 INFO:tasks.rados.rados.2.smithi204.stdout:591: writing smithi20457400-45 from 3210039 to 3781690 tid 3 2024-09-16T05:12:36.686 INFO:tasks.rados.rados.2.smithi204.stdout:592: snap_create 2024-09-16T05:12:36.686 INFO:tasks.rados.rados.2.smithi204.stdout:591: finishing write tid 1 to smithi20457400-45 2024-09-16T05:12:36.686 INFO:tasks.rados.rados.2.smithi204.stdout:591: finishing write tid 2 to smithi20457400-45 2024-09-16T05:12:36.688 INFO:tasks.rados.rados.2.smithi204.stdout:591: finishing write tid 3 to smithi20457400-45 2024-09-16T05:12:36.688 INFO:tasks.rados.rados.2.smithi204.stdout:591: finishing write tid 4 to smithi20457400-45 2024-09-16T05:12:36.688 INFO:tasks.rados.rados.2.smithi204.stdout:591: finishing write tid 5 to smithi20457400-45 2024-09-16T05:12:36.688 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 157 (ObjNum 234 snap 65 seq_num 234) dirty exists 2024-09-16T05:12:36.688 INFO:tasks.rados.rados.2.smithi204.stdout:591: left oid 45 (ObjNum 234 snap 65 seq_num 234) 2024-09-16T05:12:36.698 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:36.698 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:36.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:36.718 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:36.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:36.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:36.740 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:36.761 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:36.761 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:36.783 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:36.805 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:36.805 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:36.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:36.826 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:36.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:36.848 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:36.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:36.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:36.873 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:36.891 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:36.891 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:36.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2595748484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:36.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:36 smithi032 ceph-mon[27388]: osdmap e225: 12 total, 12 up, 12 in 2024-09-16T05:12:36.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:36.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:36.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:37.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:37.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:37.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:37.140 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2595748484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:37.140 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:36 smithi204 ceph-mon[30933]: osdmap e225: 12 total, 12 up, 12 in 2024-09-16T05:12:37.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:12:37.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:37.220 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:37.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2595748484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:36 smithi196 ceph-mon[30802]: osdmap e225: 12 total, 12 up, 12 in 2024-09-16T05:12:37.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:37.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:37.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:37.408 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:37.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:12:37.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:37.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:37.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:37.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:37.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:37.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:37.667 INFO:tasks.rados.rados.2.smithi204.stdout:591: done (1 left) 2024-09-16T05:12:37.667 INFO:tasks.rados.rados.2.smithi204.stdout:592: done (0 left) 2024-09-16T05:12:37.667 INFO:tasks.rados.rados.2.smithi204.stdout:593: write oid 31 current snap is 66 2024-09-16T05:12:37.667 INFO:tasks.rados.rados.2.smithi204.stdout:593: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-16T05:12:37.674 INFO:tasks.rados.rados.2.smithi204.stdout:593: writing smithi20457400-31 from 749645 to 1523750 tid 1 2024-09-16T05:12:37.681 INFO:tasks.rados.rados.2.smithi204.stdout:593: writing smithi20457400-31 from 2085386 to 2694787 tid 2 2024-09-16T05:12:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:37.689 INFO:tasks.rados.rados.2.smithi204.stdout:593: writing smithi20457400-31 from 3237462 to 3949645 tid 3 2024-09-16T05:12:37.689 INFO:tasks.rados.rados.2.smithi204.stdout:594: snap_remove snap 45 2024-09-16T05:12:37.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:37.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:37.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:37.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:37.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:37.903 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:12:37.903 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:12:37.903 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:12:37.903 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:12:37.903 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:12:37.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:37.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:37.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:38.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:37 smithi032 ceph-mon[27388]: pgmap v489: 47 pgs: 47 active+clean; 265 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:38.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:37 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:38.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:37 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:38.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:37 smithi032 ceph-mon[27388]: osdmap e226: 12 total, 12 up, 12 in 2024-09-16T05:12:38.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:38.121 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:38.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:37 smithi204 ceph-mon[30933]: pgmap v489: 47 pgs: 47 active+clean; 265 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:37 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:37 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:37 smithi204 ceph-mon[30933]: osdmap e226: 12 total, 12 up, 12 in 2024-09-16T05:12:38.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:37 smithi196 ceph-mon[30802]: pgmap v489: 47 pgs: 47 active+clean; 265 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:12:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:37 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:37 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:37 smithi196 ceph-mon[30802]: osdmap e226: 12 total, 12 up, 12 in 2024-09-16T05:12:38.248 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:38.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:38.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:38.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:38.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:38.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:38.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:38.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:38.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:12:38.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:38.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:38.665 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:12:38.670 INFO:tasks.rados.rados.2.smithi204.stdout:594: done (1 left) 2024-09-16T05:12:38.670 INFO:tasks.rados.rados.2.smithi204.stdout:595: read oid 13 snap 53 2024-09-16T05:12:38.670 INFO:tasks.rados.rados.2.smithi204.stdout:595: expect (ObjNum 198 snap 53 seq_num 198) 2024-09-16T05:12:38.670 INFO:tasks.rados.rados.2.smithi204.stdout:596: write (excl) oid 11 current snap is 66 2024-09-16T05:12:38.670 INFO:tasks.rados.rados.2.smithi204.stdout:593: finishing write tid 1 to smithi20457400-31 2024-09-16T05:12:38.670 INFO:tasks.rados.rados.2.smithi204.stdout:593: finishing write tid 2 to smithi20457400-31 2024-09-16T05:12:38.670 INFO:tasks.rados.rados.2.smithi204.stdout:593: finishing write tid 3 to smithi20457400-31 2024-09-16T05:12:38.671 INFO:tasks.rados.rados.2.smithi204.stdout:593: finishing write tid 4 to smithi20457400-31 2024-09-16T05:12:38.671 INFO:tasks.rados.rados.2.smithi204.stdout:593: finishing write tid 5 to smithi20457400-31 2024-09-16T05:12:38.671 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 300 (ObjNum 235 snap 66 seq_num 235) dirty exists 2024-09-16T05:12:38.671 INFO:tasks.rados.rados.2.smithi204.stdout:593: left oid 31 (ObjNum 235 snap 66 seq_num 235) 2024-09-16T05:12:38.671 INFO:tasks.rados.rados.2.smithi204.stdout:596: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-16T05:12:38.678 INFO:tasks.rados.rados.2.smithi204.stdout:596: writing smithi20457400-11 from 766452 to 1332159 tid 1 2024-09-16T05:12:38.683 INFO:tasks.rados.rados.2.smithi204.stdout:596: writing smithi20457400-11 from 1860221 to 2329633 tid 2 2024-09-16T05:12:38.690 INFO:tasks.rados.rados.2.smithi204.stdout:596: writing smithi20457400-11 from 2981238 to 3575433 tid 3 2024-09-16T05:12:38.690 INFO:tasks.rados.rados.2.smithi204.stdout:593: done (2 left) 2024-09-16T05:12:38.690 INFO:tasks.rados.rados.2.smithi204.stdout:597: write oid 17 current snap is 66 2024-09-16T05:12:38.691 INFO:tasks.rados.rados.2.smithi204.stdout:597: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-16T05:12:38.698 INFO:tasks.rados.rados.2.smithi204.stdout:597: writing smithi20457400-17 from 783259 to 1424215 tid 1 2024-09-16T05:12:38.704 INFO:tasks.rados.rados.2.smithi204.stdout:597: writing smithi20457400-17 from 1918703 to 2531773 tid 2 2024-09-16T05:12:38.709 INFO:tasks.rados.rados.2.smithi204.stdout:597: writing smithi20457400-17 from 3008661 to 3414128 tid 3 2024-09-16T05:12:38.712 INFO:tasks.rados.rados.2.smithi204.stdout:597: writing smithi20457400-17 from 3821939 to 3983259 tid 4 2024-09-16T05:12:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:598: read oid 35 snap 31 2024-09-16T05:12:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:598: expect (ObjNum 103 snap 17 seq_num 103) 2024-09-16T05:12:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:599: read oid 19 snap 49 2024-09-16T05:12:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:599: expect deleted 2024-09-16T05:12:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:600: copy_from oid 28 from oid 8 current snap is 66 2024-09-16T05:12:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:595: expect (ObjNum 198 snap 53 seq_num 198) 2024-09-16T05:12:38.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:38.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:38.802 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:12:38.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:38.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:38.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:38.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/20190426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:38.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/20190426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:38.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:38 smithi032 ceph-mon[27388]: osdmap e227: 12 total, 12 up, 12 in 2024-09-16T05:12:38.956 INFO:tasks.rados.rados.2.smithi204.stdout:596: finishing write tid 1 to smithi20457400-11 2024-09-16T05:12:38.956 INFO:tasks.rados.rados.2.smithi204.stdout:595: done (5 left) 2024-09-16T05:12:38.957 INFO:tasks.rados.rados.2.smithi204.stdout:601: write (excl) oid 46 current snap is 66 2024-09-16T05:12:38.957 INFO:tasks.rados.rados.2.smithi204.stdout:596: finishing write tid 2 to smithi20457400-11 2024-09-16T05:12:38.957 INFO:tasks.rados.rados.2.smithi204.stdout:601: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-16T05:12:38.963 INFO:tasks.rados.rados.2.smithi204.stdout:601: writing smithi20457400-46 from 400066 to 1116271 tid 1 2024-09-16T05:12:38.969 INFO:tasks.rados.rados.2.smithi204.stdout:601: writing smithi20457400-46 from 1860832 to 2000066 tid 2 2024-09-16T05:12:38.969 INFO:tasks.rados.rados.2.smithi204.stdout:602: snap_create 2024-09-16T05:12:38.969 INFO:tasks.rados.rados.2.smithi204.stdout:596: finishing write tid 3 to smithi20457400-11 2024-09-16T05:12:38.969 INFO:tasks.rados.rados.2.smithi204.stdout:597: finishing write tid 1 to smithi20457400-17 2024-09-16T05:12:38.969 INFO:tasks.rados.rados.2.smithi204.stdout:597: finishing write tid 2 to smithi20457400-17 2024-09-16T05:12:38.970 INFO:tasks.rados.rados.2.smithi204.stdout:597: finishing write tid 3 to smithi20457400-17 2024-09-16T05:12:38.970 INFO:tasks.rados.rados.2.smithi204.stdout:597: finishing write tid 4 to smithi20457400-17 2024-09-16T05:12:38.970 INFO:tasks.rados.rados.2.smithi204.stdout:597: finishing write tid 5 to smithi20457400-17 2024-09-16T05:12:38.970 INFO:tasks.rados.rados.2.smithi204.stdout:597: finishing write tid 6 to smithi20457400-17 2024-09-16T05:12:38.970 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 262 (ObjNum 237 snap 66 seq_num 237) dirty exists 2024-09-16T05:12:38.970 INFO:tasks.rados.rados.2.smithi204.stdout:597: left oid 17 (ObjNum 237 snap 66 seq_num 237) 2024-09-16T05:12:38.970 INFO:tasks.rados.rados.2.smithi204.stdout:598: expect (ObjNum 103 snap 17 seq_num 103) 2024-09-16T05:12:38.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:39.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:39.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:39.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:39.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:39.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:600: finishing copy_from racing read to smithi20457400-28 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:600: finishing copy_from to smithi20457400-28 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 285 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:601: finishing write tid 1 to smithi20457400-46 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:601: finishing write tid 2 to smithi20457400-46 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:601: finishing write tid 3 to smithi20457400-46 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:601: finishing write tid 4 to smithi20457400-46 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 22 (ObjNum 238 snap 66 seq_num 238) dirty exists 2024-09-16T05:12:39.130 INFO:tasks.rados.rados.2.smithi204.stdout:601: left oid 46 (ObjNum 238 snap 66 seq_num 238) 2024-09-16T05:12:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/20190426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/20190426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:38 smithi204 ceph-mon[30933]: osdmap e227: 12 total, 12 up, 12 in 2024-09-16T05:12:39.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:39.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/20190426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:39.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/20190426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:38 smithi196 ceph-mon[30802]: osdmap e227: 12 total, 12 up, 12 in 2024-09-16T05:12:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:39.252 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:39.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:39.304 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:12:39.304 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:19 2.b e112/107 2:d7c71ae7:::smithi20457400-5:head v 112'3, mlcod=0'0)", 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 116.260561404, 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19579059400000001, 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041248+0000", 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5799999999999999e-07 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041253+0000", 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4500000000000003e-06 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236743+0000", 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19548974399999999 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236749+0000", 2024-09-16T05:12:39.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6780000000000003e-06 2024-09-16T05:12:39.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236795+0000", 2024-09-16T05:12:39.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5164999999999999e-05 2024-09-16T05:12:39.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236811+0000", 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6453999999999999e-05 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237007+0000", 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019572799999999999 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237037+0000", 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0716999999999997e-05 2024-09-16T05:12:39.309 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:115 2.a 2:580e4d0e:::smithi20457400-29:head [setxattr _header (56) in=7b,truncate 2470596] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 115.968616457, 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032560466000000003, 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.310 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.77e-07 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333194+0000", 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9580000000000001e-06 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333200+0000", 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3130000000000004e-06 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333204+0000", 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2710000000000001e-06 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334135+0000", 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093147499999999997 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334189+0000", 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3374000000000001e-05 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334274+0000", 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4987999999999993e-05 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334496+0000", 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022271300000000001 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365655+0000", 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031158857000000002 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365694+0000", 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9187999999999998e-05 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365752+0000", 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:116 2.a 2:580e4d0e:::smithi20457400-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 115.96859915100001, 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032766964000000003, 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.316 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6e-07 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333211+0000", 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.221e-06 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333214+0000", 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.844e-06 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333218+0000", 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3050000000000001e-06 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334432+0000", 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012139379999999999 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334485+0000", 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2865000000000003e-05 2024-09-16T05:12:39.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365804+0000", 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031319393000000001 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365881+0000", 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6759000000000001e-05 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365976+0000", 2024-09-16T05:12:39.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5378999999999996e-05 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:127 2.a 2:5ae26c04:::smithi20457400-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.369006+0000", 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 115.93280250399999, 2024-09-16T05:12:39.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021134482, 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 127 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369006+0000", 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369006+0000", 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369006+0000", 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3399999999999999e-07 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369010+0000", 2024-09-16T05:12:39.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1540000000000002e-06 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369015+0000", 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7950000000000004e-06 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369019+0000", 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.258e-06 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369520+0000", 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050143400000000004 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369568+0000", 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8072000000000003e-05 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:12:39.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369647+0000", 2024-09-16T05:12:39.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8594e-05 2024-09-16T05:12:39.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369876+0000", 2024-09-16T05:12:39.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022953599999999999 2024-09-16T05:12:39.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390070+0000", 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02019425 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390104+0000", 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3599999999999997e-05 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390140+0000", 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6155000000000002e-05 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:128 2.a 2:5ae26c04:::smithi20457400-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 115.932781445, 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021253916000000001, 2024-09-16T05:12:39.333 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 128 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3099999999999998e-07 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369029+0000", 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9209999999999999e-06 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369036+0000", 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3309999999999998e-06 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369040+0000", 2024-09-16T05:12:39.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0069999999999999e-06 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369803+0000", 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076296199999999995 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369852+0000", 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8396000000000002e-05 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390172+0000", 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020319918999999999 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390220+0000", 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7936000000000003e-05 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390281+0000", 2024-09-16T05:12:39.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1013e-05 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:135 2.a 2:5f916a06:::smithi20457400-34:head [setxattr _header (56) in=7b,truncate 2554631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.395029+0000", 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 115.906779649, 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015111792000000001, 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.339 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.340 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.340 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.342 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.342 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 135 2024-09-16T05:12:39.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.342 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395029+0000", 2024-09-16T05:12:39.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395029+0000", 2024-09-16T05:12:39.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395029+0000", 2024-09-16T05:12:39.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3900000000000005e-07 2024-09-16T05:12:39.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395033+0000", 2024-09-16T05:12:39.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7720000000000001e-06 2024-09-16T05:12:39.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395040+0000", 2024-09-16T05:12:39.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9369999999999998e-06 2024-09-16T05:12:39.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395047+0000", 2024-09-16T05:12:39.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5660000000000001e-06 2024-09-16T05:12:39.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395556+0000", 2024-09-16T05:12:39.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050895400000000005 2024-09-16T05:12:39.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395609+0000", 2024-09-16T05:12:39.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2451000000000003e-05 2024-09-16T05:12:39.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:12:39.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395697+0000", 2024-09-16T05:12:39.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8098999999999998e-05 2024-09-16T05:12:39.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395915+0000", 2024-09-16T05:12:39.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000217934 2024-09-16T05:12:39.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410037+0000", 2024-09-16T05:12:39.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014121984000000001 2024-09-16T05:12:39.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410081+0000", 2024-09-16T05:12:39.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4338999999999997e-05 2024-09-16T05:12:39.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410140+0000", 2024-09-16T05:12:39.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9216999999999999e-05 2024-09-16T05:12:39.351 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.352 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.352 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:136 2.a 2:5f916a06:::smithi20457400-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.395062+0000", 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 115.90674643600001, 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015201183, 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.353 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 136 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395062+0000", 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395062+0000", 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395062+0000", 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9200000000000001e-07 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395064+0000", 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1749999999999999e-06 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395072+0000", 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5460000000000003e-06 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395077+0000", 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8169999999999998e-06 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395859+0000", 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00078203300000000001 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.395903+0000", 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4517000000000001e-05 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410164+0000", 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014260291 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410202+0000", 2024-09-16T05:12:39.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8124000000000003e-05 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.410263+0000", 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1187999999999998e-05 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:277 2.a 2:5827f14d:::smithi20457400-12:head [copy-from ver 0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:12:39.358 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.275915846, 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38105511399999997, 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 277 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:12:39.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025894+0000", 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.683e-06 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025900+0000", 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7740000000000001e-06 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025917+0000", 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6742999999999999e-05 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025927+0000", 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.978e-06 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025988+0000", 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1574000000000004e-05 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026078+0000", 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0326000000000002e-05 2024-09-16T05:12:39.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.406948+0000", 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38086903599999999 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:278 2.a 2:5827f14d:::smithi20457400-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.275860438, 2024-09-16T05:12:39.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38122975799999997, 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 278 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:12:39.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5899999999999996e-07 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025951+0000", 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7819999999999998e-06 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025959+0000", 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1019999999999997e-06 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025966+0000", 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.409e-06 2024-09-16T05:12:39.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026245+0000", 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027952800000000002 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026298+0000", 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2747999999999997e-05 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407054+0000", 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.380756119 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407114+0000", 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0350999999999999e-05 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407178+0000", 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3159999999999998e-05 2024-09-16T05:12:39.368 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:275 2.b e117/107 2:d7c71ae7:::smithi20457400-5:head v 117'20, mlcod=116'15)", 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.275164768, 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072246237000000005, 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.369 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:12:39.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026644+0000", 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6000000000000003e-07 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026651+0000", 2024-09-16T05:12:39.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0210000000000002e-06 2024-09-16T05:12:39.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098551+0000", 2024-09-16T05:12:39.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.071899669999999999 2024-09-16T05:12:39.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098559+0000", 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.028e-06 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098612+0000", 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3485e-05 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098628+0000", 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5206e-05 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098860+0000", 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023204700000000001 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098890+0000", 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.012e-05 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:286 2.b e117/107 2:da4f6049:::smithi20457400-49:head v 117'22, mlcod=117'17)", 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.203287076, 2024-09-16T05:12:39.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.24158528100000001, 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:12:39.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098523+0000", 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.508e-06 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098530+0000", 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.075e-06 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339416+0000", 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.240886668 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339422+0000", 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0680000000000002e-06 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339502+0000", 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9698999999999997e-05 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339527+0000", 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4780999999999999e-05 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340073+0000", 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054627699999999996 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340106+0000", 2024-09-16T05:12:39.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3204999999999998e-05 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:454 2.c 2:33100188:::smithi20457400-19:head [copy-from ver 33] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:58.688638+0000", 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 100.613170094, 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017328989999999999, 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.380 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 454 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688638+0000", 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688638+0000", 2024-09-16T05:12:39.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688640+0000", 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1430000000000001e-06 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688651+0000", 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0501e-05 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688670+0000", 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8974000000000002e-05 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688680+0000", 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9879999999999999e-06 2024-09-16T05:12:39.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688761+0000", 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1339000000000002e-05 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688768+0000", 2024-09-16T05:12:39.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.221e-06 2024-09-16T05:12:39.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.689852+0000", 2024-09-16T05:12:39.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010839249999999999 2024-09-16T05:12:39.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.689917+0000", 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4380000000000004e-05 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.699999+0000", 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010082749 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.703541+0000", 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035421210000000001 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.705852+0000", 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023108349999999998 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.705916+0000", 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3625000000000002e-05 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.705967+0000", 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1189000000000001e-05 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.388 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:455 2.c 2:33100188:::smithi20457400-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e127)", 2024-09-16T05:12:39.389 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:58.688733+0000", 2024-09-16T05:12:39.389 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 100.613075111, 2024-09-16T05:12:39.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014215077, 2024-09-16T05:12:39.389 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.389 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 455 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688733+0000", 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688733+0000", 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688734+0000", 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8899999999999999e-07 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688738+0000", 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7670000000000001e-06 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688747+0000", 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5149999999999995e-06 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688753+0000", 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3740000000000003e-06 2024-09-16T05:12:39.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688828+0000", 2024-09-16T05:12:39.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4164000000000003e-05 2024-09-16T05:12:39.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:12:39.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.688833+0000", 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7599999999999999e-06 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.690047+0000", 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012134730000000001 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.690095+0000", 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7821999999999997e-05 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.702811+0000", 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012715842999999999 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.702874+0000", 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3428000000000003e-05 2024-09-16T05:12:39.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:58.702948+0000", 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4241999999999998e-05 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:884 2.b e170/107 2:d8680303:::smithi20457400-43:head v 170'88, mlcod=165'82)", 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.662116+0000", 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 57.639692580000002, 2024-09-16T05:12:39.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67911829000000001, 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662116+0000", 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662116+0000", 2024-09-16T05:12:39.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662117+0000", 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.111e-06 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662125+0000", 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9410000000000007e-06 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.340626+0000", 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67850096299999996 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.340633+0000", 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.08e-06 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.340718+0000", 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5235000000000006e-05 2024-09-16T05:12:39.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.340766+0000", 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8031999999999997e-05 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341203+0000", 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000436759 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341234+0000", 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1168999999999999e-05 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:972 2.c 2:33100188:::smithi20457400-19:head [setxattr _header (57) in=7b,truncate 2672313] snapc 28=[28,27,25,24,23,22,21,20,1f,15,f] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:50.997840+0000", 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.303968126000001, 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53798918100000004, 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.401 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 972 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997840+0000", 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997840+0000", 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997841+0000", 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0399999999999996e-07 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997844+0000", 2024-09-16T05:12:39.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4809999999999998e-06 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997853+0000", 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2010000000000005e-06 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997859+0000", 2024-09-16T05:12:39.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9920000000000004e-06 2024-09-16T05:12:39.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.998655+0000", 2024-09-16T05:12:39.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00079606800000000004 2024-09-16T05:12:39.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.998775+0000", 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119679 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.998945+0000", 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016966099999999999 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.000857+0000", 2024-09-16T05:12:39.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019118410000000001 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535713+0000", 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53485633799999999 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535763+0000", 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9926999999999998e-05 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535829+0000", 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6389000000000004e-05 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:973 2.c 2:33100188:::smithi20457400-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:50.997896+0000", 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.303912625000002, 2024-09-16T05:12:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53811316099999995, 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 973 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997896+0000", 2024-09-16T05:12:39.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997896+0000", 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997896+0000", 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8999999999999996e-07 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997899+0000", 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.627e-06 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997906+0000", 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.622e-06 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997911+0000", 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.989e-06 2024-09-16T05:12:39.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.999224+0000", 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001312183 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.999334+0000", 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109947 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535872+0000", 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53653808800000002 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535938+0000", 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6586000000000002e-05 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.536009+0000", 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0529000000000002e-05 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.414 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:974 2.a 2:5827f14d:::smithi20457400-12:head [copy-from ver 125] snapc 28=[28,27,25,24,23,22,21,20,1f,15,f] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:50.997917+0000", 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.303891176999997, 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016628899999999999, 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.415 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 974 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997917+0000", 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997917+0000", 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997917+0000", 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2099999999999998e-07 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997919+0000", 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0360000000000001e-06 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997923+0000", 2024-09-16T05:12:39.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.749e-06 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997926+0000", 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.942e-06 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997974+0000", 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8102000000000001e-05 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.998065+0000", 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0554000000000001e-05 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.007282+0000", 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.009217233 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.009903+0000", 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026210920000000002 2024-09-16T05:12:39.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.014462+0000", 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045590149999999996 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.014505+0000", 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2889000000000002e-05 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.014546+0000", 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0967000000000003e-05 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1133 2.a 2:5ae26c04:::smithi20457400-32:head [write 591646~470441,stat] snapc 2f=[2f,2e,2d,2c,2b,2a,29,27,24,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:02.732148+0000", 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.569660051, 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023703246000000001, 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.421 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1133 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732148+0000", 2024-09-16T05:12:39.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732148+0000", 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732151+0000", 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.554e-06 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732426+0000", 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027569299999999998 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732451+0000", 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4324e-05 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732462+0000", 2024-09-16T05:12:39.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1099e-05 2024-09-16T05:12:39.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732528+0000", 2024-09-16T05:12:39.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6525000000000004e-05 2024-09-16T05:12:39.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732536+0000", 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4839999999999998e-06 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.733720+0000", 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011842000000000001 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.733818+0000", 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8364999999999998e-05 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.733981+0000", 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016235800000000001 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.735390+0000", 2024-09-16T05:12:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014088200000000001 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.755705+0000", 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020315054999999999 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.755785+0000", 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0304999999999998e-05 2024-09-16T05:12:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.755851+0000", 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6464000000000006e-05 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.430 INFO:tasks.rados.rados.2.smithi204.stdout:596: finishing write tid 4 to smithi20457400-11 2024-09-16T05:12:39.430 INFO:tasks.rados.rados.2.smithi204.stdout:596: finishing write tid 5 to smithi20457400-11 2024-09-16T05:12:39.430 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 370 (ObjNum 236 snap 66 seq_num 236) dirty exists 2024-09-16T05:12:39.430 INFO:tasks.rados.rados.2.smithi204.stdout:596: left oid 11 (ObjNum 236 snap 66 seq_num 236) 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1134 2.a 2:5ae26c04:::smithi20457400-32:head [write 1566383~595717] snapc 2f=[2f,2e,2d,2c,2b,2a,29,27,24,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:02.738120+0000", 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.563687989000002, 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017881484999999999, 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1134 2024-09-16T05:12:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738120+0000", 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738120+0000", 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738122+0000", 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.451e-06 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738476+0000", 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00035429100000000001 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738491+0000", 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.525e-05 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738499+0000", 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738555+0000", 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5946999999999998e-05 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738661+0000", 2024-09-16T05:12:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105468 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.738844+0000", 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018353099999999999 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.740183+0000", 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001338576 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.755912+0000", 2024-09-16T05:12:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015729528 2024-09-16T05:12:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.755960+0000", 2024-09-16T05:12:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8078999999999998e-05 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.756002+0000", 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1454e-05 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1146 2.c 2:33100188:::smithi20457400-19:head [copy-from ver 88] snapc 2f=[2f,2e,2d,2c,2b,2a,29,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:12:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:03.734197+0000", 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.567611603000003, 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014781183, 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1146 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.734197+0000", 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.734197+0000", 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.734197+0000", 2024-09-16T05:12:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5000000000000002e-07 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.734201+0000", 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1060000000000002e-06 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.734209+0000", 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6329999999999996e-06 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.734215+0000", 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7069999999999997e-06 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.735767+0000", 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015523379999999999 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.735880+0000", 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113183 2024-09-16T05:12:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.742656+0000", 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067757310000000001 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.747263+0000", 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004606728 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.748839+0000", 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00157609 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.748905+0000", 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6378999999999996e-05 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.748978+0000", 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2638000000000005e-05 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:12:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:12:39.444 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:12:39.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:39.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:39.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:39.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:12:39.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:39.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:39.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:39.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:39.720 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:39.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:39.770 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:39.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:39.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:39.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:39.857 INFO:tasks.rados.rados.2.smithi204.stdout:596: done (6 left) 2024-09-16T05:12:39.857 INFO:tasks.rados.rados.2.smithi204.stdout:597: done (5 left) 2024-09-16T05:12:39.858 INFO:tasks.rados.rados.2.smithi204.stdout:598: done (4 left) 2024-09-16T05:12:39.858 INFO:tasks.rados.rados.2.smithi204.stdout:599: done (3 left) 2024-09-16T05:12:39.858 INFO:tasks.rados.rados.2.smithi204.stdout:600: done (2 left) 2024-09-16T05:12:39.858 INFO:tasks.rados.rados.2.smithi204.stdout:601: done (1 left) 2024-09-16T05:12:39.858 INFO:tasks.rados.rados.2.smithi204.stdout:602: done (0 left) 2024-09-16T05:12:39.858 INFO:tasks.rados.rados.2.smithi204.stdout:603: copy_from oid 38 from oid 48 current snap is 67 2024-09-16T05:12:39.858 INFO:tasks.rados.rados.2.smithi204.stdout:604: write oid 33 current snap is 67 2024-09-16T05:12:39.858 INFO:tasks.rados.rados.2.smithi204.stdout:604: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-16T05:12:39.864 INFO:tasks.rados.rados.2.smithi204.stdout:604: writing smithi20457400-33 from 416873 to 1208327 tid 1 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:604: writing smithi20457400-33 from 1919314 to 2016873 tid 2 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:605: read oid 11 snap 31 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:605: expect (ObjNum 143 snap 29 seq_num 143) 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:606: copy_from oid 37 from oid 12 current snap is 67 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:603: finishing copy_from racing read to smithi20457400-38 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:604: finishing write tid 1 to smithi20457400-33 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:607: rollback oid 4 current snap is 67 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 56 2024-09-16T05:12:39.870 INFO:tasks.rados.rados.2.smithi204.stdout:608: snap_remove snap 54 2024-09-16T05:12:39.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:39 smithi204 ceph-mon[30933]: pgmap v492: 47 pgs: 47 active+clean; 270 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.2 MiB/s wr, 4 op/s 2024-09-16T05:12:39.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:39.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:40.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:40.097 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:40.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:40.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:39 smithi032 ceph-mon[27388]: pgmap v492: 47 pgs: 47 active+clean; 270 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.2 MiB/s wr, 4 op/s 2024-09-16T05:12:40.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:12:40.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:39 smithi196 ceph-mon[30802]: pgmap v492: 47 pgs: 47 active+clean; 270 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.2 MiB/s wr, 4 op/s 2024-09-16T05:12:40.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:40.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:40.339 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:40.339 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:40.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:40.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:40.361 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:40.382 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:40.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:40.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:40.404 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:40.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:40.427 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:40.447 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:40.451 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:40.451 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:40.474 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:40.474 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:40.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:40.496 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:40.496 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:40.507 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:40.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:40.517 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:40.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:40.540 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:40.540 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:40.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:40.583 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:40.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:40.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:40.668 INFO:tasks.rados.rados.2.smithi204.stdout:607: finishing rollback tid 0 to smithi20457400-4 2024-09-16T05:12:40.668 INFO:tasks.rados.rados.2.smithi204.stdout:607: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:12:40.668 INFO:tasks.rados.rados.2.smithi204.stdout:607: finishing rollback tid 2 to smithi20457400-4 2024-09-16T05:12:40.668 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 195 (ObjNum 181 snap 46 seq_num 181) dirty exists 2024-09-16T05:12:40.668 INFO:tasks.rados.rados.2.smithi204.stdout:606: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:12:40.668 INFO:tasks.rados.rados.2.smithi204.stdout:606: finishing copy_from to smithi20457400-37 2024-09-16T05:12:40.669 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 125 (ObjNum 124 snap 26 seq_num 124) dirty exists 2024-09-16T05:12:40.669 INFO:tasks.rados.rados.2.smithi204.stdout:603: finishing copy_from to smithi20457400-38 2024-09-16T05:12:40.669 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 183 (ObjNum 208 snap 55 seq_num 208) dirty exists 2024-09-16T05:12:40.669 INFO:tasks.rados.rados.2.smithi204.stdout:604: finishing write tid 2 to smithi20457400-33 2024-09-16T05:12:40.669 INFO:tasks.rados.rados.2.smithi204.stdout:605: expect (ObjNum 143 snap 29 seq_num 143) 2024-09-16T05:12:40.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:40.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:40.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:40.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:604: finishing write tid 3 to smithi20457400-33 2024-09-16T05:12:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:604: finishing write tid 4 to smithi20457400-33 2024-09-16T05:12:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 280 (ObjNum 239 snap 67 seq_num 239) dirty exists 2024-09-16T05:12:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:604: left oid 33 (ObjNum 239 snap 67 seq_num 239) 2024-09-16T05:12:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:603: done (5 left) 2024-09-16T05:12:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:604: done (4 left) 2024-09-16T05:12:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:605: done (3 left) 2024-09-16T05:12:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:606: done (2 left) 2024-09-16T05:12:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:607: done (1 left) 2024-09-16T05:12:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:608: done (0 left) 2024-09-16T05:12:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:609: snap_remove snap 43 2024-09-16T05:12:40.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:40.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:40.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:40.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:41.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:41.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:41.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:41.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:40 smithi032 ceph-mon[27388]: osdmap e228: 12 total, 12 up, 12 in 2024-09-16T05:12:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/278558401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:40 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:41.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:40 smithi032 ceph-mon[27388]: osdmap e229: 12 total, 12 up, 12 in 2024-09-16T05:12:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:40 smithi204 ceph-mon[30933]: osdmap e228: 12 total, 12 up, 12 in 2024-09-16T05:12:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/278558401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:40 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:40 smithi204 ceph-mon[30933]: osdmap e229: 12 total, 12 up, 12 in 2024-09-16T05:12:41.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:40 smithi196 ceph-mon[30802]: osdmap e228: 12 total, 12 up, 12 in 2024-09-16T05:12:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/278558401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:40 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:40 smithi196 ceph-mon[30802]: osdmap e229: 12 total, 12 up, 12 in 2024-09-16T05:12:41.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:41.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:41.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:12:41.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:41.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:41.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:41.430 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:12:41.430 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:12:41.430 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:12:41.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd out 10 2024-09-16T05:12:41.484 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:41.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:41.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:41.611 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:41.611 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:41.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:41.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:41.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:41.668 INFO:tasks.rados.rados.2.smithi204.stdout:609: done (0 left) 2024-09-16T05:12:41.668 INFO:tasks.rados.rados.2.smithi204.stdout:610: rollback oid 12 current snap is 67 2024-09-16T05:12:41.668 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 49 2024-09-16T05:12:41.668 INFO:tasks.rados.rados.2.smithi204.stdout:611: snap_remove snap 65 2024-09-16T05:12:41.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:41.763 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:41.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:41.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:41.889 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:41.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:41.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:42.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:42.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:42.139 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:42.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:41 smithi032 ceph-mon[27388]: pgmap v495: 47 pgs: 47 active+clean; 270 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 853 KiB/s wr, 2 op/s 2024-09-16T05:12:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:41 smithi032 ceph-mon[27388]: osdmap e230: 12 total, 12 up, 12 in 2024-09-16T05:12:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:41 smithi204 ceph-mon[30933]: pgmap v495: 47 pgs: 47 active+clean; 270 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 853 KiB/s wr, 2 op/s 2024-09-16T05:12:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:41 smithi204 ceph-mon[30933]: osdmap e230: 12 total, 12 up, 12 in 2024-09-16T05:12:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:41 smithi196 ceph-mon[30802]: pgmap v495: 47 pgs: 47 active+clean; 270 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 853 KiB/s wr, 2 op/s 2024-09-16T05:12:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:41 smithi196 ceph-mon[30802]: osdmap e230: 12 total, 12 up, 12 in 2024-09-16T05:12:42.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:42.264 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:42.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:42.285 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:42.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:12:42.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:42.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:42.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:42.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:42.516 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:42.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:42.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:42.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:42.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:42.672 INFO:tasks.rados.rados.2.smithi204.stdout:611: done (1 left) 2024-09-16T05:12:42.672 INFO:tasks.rados.rados.2.smithi204.stdout:612: read oid 2 snap 31 2024-09-16T05:12:42.672 INFO:tasks.rados.rados.2.smithi204.stdout:612: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T05:12:42.672 INFO:tasks.rados.rados.2.smithi204.stdout:613: write (excl) oid 22 current snap is 67 2024-09-16T05:12:42.672 INFO:tasks.rados.rados.2.smithi204.stdout:613: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-16T05:12:42.677 INFO:tasks.rados.rados.2.smithi204.stdout:613: writing smithi20457400-22 from 433680 to 900383 tid 1 2024-09-16T05:12:42.684 INFO:tasks.rados.rados.2.smithi204.stdout:613: writing smithi20457400-22 from 1577796 to 2033680 tid 2 2024-09-16T05:12:42.684 INFO:tasks.rados.rados.2.smithi204.stdout:614: write oid 48 current snap is 67 2024-09-16T05:12:42.684 INFO:tasks.rados.rados.2.smithi204.stdout:610: finishing rollback tid 0 to smithi20457400-12 2024-09-16T05:12:42.684 INFO:tasks.rados.rados.2.smithi204.stdout:614: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-16T05:12:42.689 INFO:tasks.rados.rados.2.smithi204.stdout:614: writing smithi20457400-48 from 450487 to 992439 tid 1 2024-09-16T05:12:42.695 INFO:tasks.rados.rados.2.smithi204.stdout:614: writing smithi20457400-48 from 1636278 to 2050487 tid 2 2024-09-16T05:12:42.695 INFO:tasks.rados.rados.2.smithi204.stdout:615: write (excl) oid 41 current snap is 67 2024-09-16T05:12:42.695 INFO:tasks.rados.rados.2.smithi204.stdout:610: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:12:42.695 INFO:tasks.rados.rados.2.smithi204.stdout:610: finishing rollback tid 2 to smithi20457400-12 2024-09-16T05:12:42.695 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 302 (ObjNum 124 snap 26 seq_num 124) dirty exists 2024-09-16T05:12:42.695 INFO:tasks.rados.rados.2.smithi204.stdout:615: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-16T05:12:42.701 INFO:tasks.rados.rados.2.smithi204.stdout:615: writing smithi20457400-41 from 467294 to 1084495 tid 1 2024-09-16T05:12:42.706 INFO:tasks.rados.rados.2.smithi204.stdout:615: writing smithi20457400-41 from 1578407 to 2067294 tid 2 2024-09-16T05:12:42.706 INFO:tasks.rados.rados.2.smithi204.stdout:610: done (4 left) 2024-09-16T05:12:42.707 INFO:tasks.rados.rados.2.smithi204.stdout:616: delete oid 35 current snap is 67 2024-09-16T05:12:42.707 INFO:tasks.rados.rados.2.smithi204.stdout:612: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T05:12:42.742 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:42.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:42.755 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:42.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:42.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:42.779 INFO:tasks.rados.rados.2.smithi204.stdout:613: finishing write tid 1 to smithi20457400-22 2024-09-16T05:12:42.779 INFO:tasks.rados.rados.2.smithi204.stdout:612: done (4 left) 2024-09-16T05:12:42.779 INFO:tasks.rados.rados.2.smithi204.stdout:616: done (3 left) 2024-09-16T05:12:42.779 INFO:tasks.rados.rados.2.smithi204.stdout:617: write oid 16 current snap is 67 2024-09-16T05:12:42.779 INFO:tasks.rados.rados.2.smithi204.stdout:613: finishing write tid 2 to smithi20457400-22 2024-09-16T05:12:42.780 INFO:tasks.rados.rados.2.smithi204.stdout:613: finishing write tid 3 to smithi20457400-22 2024-09-16T05:12:42.780 INFO:tasks.rados.rados.2.smithi204.stdout:613: finishing write tid 4 to smithi20457400-22 2024-09-16T05:12:42.780 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 225 (ObjNum 240 snap 67 seq_num 240) dirty exists 2024-09-16T05:12:42.780 INFO:tasks.rados.rados.2.smithi204.stdout:613: left oid 22 (ObjNum 240 snap 67 seq_num 240) 2024-09-16T05:12:42.780 INFO:tasks.rados.rados.2.smithi204.stdout:614: finishing write tid 1 to smithi20457400-48 2024-09-16T05:12:42.780 INFO:tasks.rados.rados.2.smithi204.stdout:617: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-16T05:12:42.784 INFO:tasks.rados.rados.2.smithi204.stdout:617: writing smithi20457400-16 from 500908 to 984960 tid 1 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:617: writing smithi20457400-16 from 1411724 to 2100908 tid 2 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:613: done (3 left) 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:618: delete oid 32 current snap is 67 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:614: finishing write tid 2 to smithi20457400-48 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:614: finishing write tid 3 to smithi20457400-48 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:614: finishing write tid 4 to smithi20457400-48 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 229 (ObjNum 241 snap 67 seq_num 241) dirty exists 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:614: left oid 48 (ObjNum 241 snap 67 seq_num 241) 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:615: finishing write tid 1 to smithi20457400-41 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:615: finishing write tid 2 to smithi20457400-41 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:615: finishing write tid 3 to smithi20457400-41 2024-09-16T05:12:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:615: finishing write tid 4 to smithi20457400-41 2024-09-16T05:12:42.792 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 292 (ObjNum 242 snap 67 seq_num 242) dirty exists 2024-09-16T05:12:42.792 INFO:tasks.rados.rados.2.smithi204.stdout:615: left oid 41 (ObjNum 242 snap 67 seq_num 242) 2024-09-16T05:12:42.792 INFO:tasks.rados.rados.2.smithi204.stdout:617: finishing write tid 1 to smithi20457400-16 2024-09-16T05:12:42.793 INFO:tasks.rados.rados.2.smithi204.stdout:614: done (3 left) 2024-09-16T05:12:42.793 INFO:tasks.rados.rados.2.smithi204.stdout:615: done (2 left) 2024-09-16T05:12:42.793 INFO:tasks.rados.rados.2.smithi204.stdout:618: done (1 left) 2024-09-16T05:12:42.793 INFO:tasks.rados.rados.2.smithi204.stdout:619: rollback oid 38 current snap is 67 2024-09-16T05:12:42.793 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 38 to 51 2024-09-16T05:12:42.793 INFO:tasks.rados.rados.2.smithi204.stdout:620: copy_from oid 36 from oid 8 current snap is 67 2024-09-16T05:12:42.793 INFO:tasks.rados.rados.2.smithi204.stdout:621: snap_remove snap 31 2024-09-16T05:12:42.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:42.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:42.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:42.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:43.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:43.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:43.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:43.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:43.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:43.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:12:43.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:43.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:43.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:43.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:43.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:43.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:43.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:43.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:43.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:43.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:43.656 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:43.657 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:43.666 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:43 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:12:43.405+0000 7f7c3f992640 -1 mon.a@0(leader).osd e231 definitely_dead 0 2024-09-16T05:12:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:marked out osd.10. 2024-09-16T05:12:43.670 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:12:43.675 INFO:tasks.rados.rados.2.smithi204.stdout:621: done (3 left) 2024-09-16T05:12:43.675 INFO:tasks.rados.rados.2.smithi204.stdout:622: rollback oid 24 current snap is 67 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 51 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:617: finishing write tid 2 to smithi20457400-16 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:617: finishing write tid 3 to smithi20457400-16 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:617: finishing write tid 4 to smithi20457400-16 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 328 (ObjNum 244 snap 67 seq_num 244) dirty exists 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:617: left oid 16 (ObjNum 244 snap 67 seq_num 244) 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:619: finishing rollback tid 0 to smithi20457400-38 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:619: finishing rollback tid 1 to smithi20457400-38 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 330 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:620: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:620: finishing copy_from to smithi20457400-36 2024-09-16T05:12:43.676 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 285 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:12:43.677 INFO:tasks.rados.rados.2.smithi204.stdout:617: done (3 left) 2024-09-16T05:12:43.677 INFO:tasks.rados.rados.2.smithi204.stdout:619: done (2 left) 2024-09-16T05:12:43.677 INFO:tasks.rados.rados.2.smithi204.stdout:620: done (1 left) 2024-09-16T05:12:43.677 INFO:tasks.rados.rados.2.smithi204.stdout:623: read oid 18 snap -1 2024-09-16T05:12:43.677 INFO:tasks.rados.rados.2.smithi204.stdout:623: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:12:43.677 INFO:tasks.rados.rados.2.smithi204.stdout:624: snap_create 2024-09-16T05:12:43.690 INFO:tasks.rados.rados.2.smithi204.stdout:623: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:12:43.745 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:43.746 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:43.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:43.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:43.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:43.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:43.907 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:43.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:43 smithi032 ceph-mon[27388]: pgmap v498: 47 pgs: 47 active+clean; 287 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 16 op/s 2024-09-16T05:12:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:43 smithi032 ceph-mon[27388]: osdmap e231: 12 total, 12 up, 12 in 2024-09-16T05:12:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/448186965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/744956388' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-09-16T05:12:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:43 smithi032 ceph-mon[27388]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-16T05:12:43.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:43 smithi204 ceph-mon[30933]: pgmap v498: 47 pgs: 47 active+clean; 287 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 16 op/s 2024-09-16T05:12:43.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:43 smithi204 ceph-mon[30933]: osdmap e231: 12 total, 12 up, 12 in 2024-09-16T05:12:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/448186965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/744956388' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-09-16T05:12:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:43 smithi204 ceph-mon[30933]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-16T05:12:43.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:43 smithi196 ceph-mon[30802]: pgmap v498: 47 pgs: 47 active+clean; 287 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 16 op/s 2024-09-16T05:12:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:43 smithi196 ceph-mon[30802]: osdmap e231: 12 total, 12 up, 12 in 2024-09-16T05:12:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/448186965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:12:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/744956388' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-09-16T05:12:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:43 smithi196 ceph-mon[30802]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-16T05:12:44.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:44.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:44.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:44.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:44.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:44.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:44.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:44.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:44.281 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:44.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:44.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:44.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:44.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:44.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:44.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:44.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:44.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:44.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:623: done (2 left) 2024-09-16T05:12:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:624: done (1 left) 2024-09-16T05:12:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:625: copy_from oid 2 from oid 21 current snap is 68 2024-09-16T05:12:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:626: write oid 22 current snap is 68 2024-09-16T05:12:44.679 INFO:tasks.rados.rados.2.smithi204.stdout:626: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-16T05:12:44.685 INFO:tasks.rados.rados.2.smithi204.stdout:626: writing smithi20457400-22 from 534522 to 1169072 tid 1 2024-09-16T05:12:44.691 INFO:tasks.rados.rados.2.smithi204.stdout:626: writing smithi20457400-22 from 1812335 to 2134522 tid 2 2024-09-16T05:12:44.691 INFO:tasks.rados.rados.2.smithi204.stdout:627: rollback oid 3 current snap is 68 2024-09-16T05:12:44.692 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 49 2024-09-16T05:12:44.692 INFO:tasks.rados.rados.2.smithi204.stdout:628: copy_from oid 11 from oid 44 current snap is 68 2024-09-16T05:12:44.692 INFO:tasks.rados.rados.2.smithi204.stdout:629: read oid 33 snap -1 2024-09-16T05:12:44.692 INFO:tasks.rados.rados.2.smithi204.stdout:629: expect (ObjNum 239 snap 67 seq_num 239) 2024-09-16T05:12:44.692 INFO:tasks.rados.rados.2.smithi204.stdout:630: read oid 41 snap 51 2024-09-16T05:12:44.692 INFO:tasks.rados.rados.2.smithi204.stdout:630: expect (ObjNum 175 snap 43 seq_num 175) 2024-09-16T05:12:44.692 INFO:tasks.rados.rados.2.smithi204.stdout:631: snap_create 2024-09-16T05:12:44.703 INFO:tasks.rados.rados.2.smithi204.stdout:630: expect (ObjNum 175 snap 43 seq_num 175) 2024-09-16T05:12:44.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:44.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:44.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:44.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/448186965' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:44.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/744956388' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-16T05:12:44.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:44 smithi032 ceph-mon[27388]: osdmap e232: 12 total, 12 up, 11 in 2024-09-16T05:12:44.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:44 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:44.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:44 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:44.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:12:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:622: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:12:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:622: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:12:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 232 (ObjNum 129 snap 0 seq_num 4282058672) dirty dne 2024-09-16T05:12:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:627: finishing rollback tid 0 to smithi20457400-3 2024-09-16T05:12:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:627: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:12:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:627: finishing rollback tid 2 to smithi20457400-3 2024-09-16T05:12:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 396 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-16T05:12:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:629: expect (ObjNum 239 snap 67 seq_num 239) 2024-09-16T05:12:44.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:44.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:44.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:44.969 INFO:tasks.rados.rados.2.smithi204.stdout:626: finishing write tid 1 to smithi20457400-22 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:626: finishing write tid 2 to smithi20457400-22 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:626: finishing write tid 3 to smithi20457400-22 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:626: finishing write tid 4 to smithi20457400-22 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 236 (ObjNum 246 snap 68 seq_num 246) dirty exists 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:626: left oid 22 (ObjNum 246 snap 68 seq_num 246) 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:625: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:628: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:628: finishing copy_from to smithi20457400-11 2024-09-16T05:12:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 157 (ObjNum 220 snap 57 seq_num 220) dirty exists 2024-09-16T05:12:44.971 INFO:tasks.rados.rados.2.smithi204.stdout:625: finishing copy_from to smithi20457400-2 2024-09-16T05:12:44.971 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 158 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:12:44.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/448186965' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:44.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/744956388' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-16T05:12:44.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:44 smithi196 ceph-mon[30802]: osdmap e232: 12 total, 12 up, 11 in 2024-09-16T05:12:44.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:44 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:44.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:44 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:45.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:45.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:45.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:45.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:45.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:45.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/448186965' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:12:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/744956388' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-16T05:12:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:44 smithi204 ceph-mon[30933]: osdmap e232: 12 total, 12 up, 11 in 2024-09-16T05:12:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:44 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:44 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:45.240 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:45.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:45.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:45.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:45.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:45.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:45.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:45.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:45.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:45.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:45.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:45.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:45.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:45.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:45.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:622: done (7 left) 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:625: done (6 left) 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:626: done (5 left) 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:627: done (4 left) 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:628: done (3 left) 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:629: done (2 left) 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:630: done (1 left) 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:631: done (0 left) 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:632: read oid 4 snap -1 2024-09-16T05:12:45.671 INFO:tasks.rados.rados.2.smithi204.stdout:632: expect (ObjNum 181 snap 46 seq_num 181) 2024-09-16T05:12:45.672 INFO:tasks.rados.rados.2.smithi204.stdout:633: rollback oid 26 current snap is 69 2024-09-16T05:12:45.672 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 63 2024-09-16T05:12:45.672 INFO:tasks.rados.rados.2.smithi204.stdout:634: snap_remove snap 59 2024-09-16T05:12:45.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:45.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:45.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:45.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:45.918 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:45.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:45 smithi032 ceph-mon[27388]: pgmap v501: 47 pgs: 47 active+clean; 281 MiB data, 688 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 20 op/s 2024-09-16T05:12:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:45 smithi032 ceph-mon[27388]: osdmap e233: 12 total, 12 up, 11 in 2024-09-16T05:12:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:45 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:12:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:45 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:12:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:45 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:45 smithi032 ceph-mon[27388]: osdmap e234: 12 total, 12 up, 11 in 2024-09-16T05:12:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:45 smithi196 ceph-mon[30802]: pgmap v501: 47 pgs: 47 active+clean; 281 MiB data, 688 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 20 op/s 2024-09-16T05:12:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:45 smithi196 ceph-mon[30802]: osdmap e233: 12 total, 12 up, 11 in 2024-09-16T05:12:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:45 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:12:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:45 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:12:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:45 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:45 smithi196 ceph-mon[30802]: osdmap e234: 12 total, 12 up, 11 in 2024-09-16T05:12:46.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:46.042 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:46.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:46.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:46.160 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:46.161 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:45 smithi204 ceph-mon[30933]: pgmap v501: 47 pgs: 47 active+clean; 281 MiB data, 688 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 20 op/s 2024-09-16T05:12:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:45 smithi204 ceph-mon[30933]: osdmap e233: 12 total, 12 up, 11 in 2024-09-16T05:12:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:45 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:12:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:45 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:12:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:45 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:45 smithi204 ceph-mon[30933]: osdmap e234: 12 total, 12 up, 11 in 2024-09-16T05:12:46.172 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:46.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:46.184 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:46.184 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:46.189 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:46.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:46.206 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:46.228 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:46.228 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:46.251 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:46.251 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:46.273 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:46.273 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:46.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:46.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:46.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:46.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:46.316 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:46.316 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:46.339 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:46.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:46.361 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:46.382 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:46.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:46.404 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:46.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:46.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:46.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:46.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:46.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:46.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:634: done (2 left) 2024-09-16T05:12:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:635: write oid 22 current snap is 69 2024-09-16T05:12:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:633: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:12:46.669 INFO:tasks.rados.rados.2.smithi204.stdout:632: expect (ObjNum 181 snap 46 seq_num 181) 2024-09-16T05:12:46.679 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:46.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:46.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:46 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:12:46.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:46 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:12:46.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:46 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:12:46.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:46 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:12:46.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:46 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:12:46.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:46 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:12:46.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2553370039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:46.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:46 smithi196 ceph-mon[30802]: osdmap e235: 12 total, 12 up, 11 in 2024-09-16T05:12:46.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:46.804 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:46.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:46.847 INFO:tasks.rados.rados.2.smithi204.stdout:633: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:12:46.847 INFO:tasks.rados.rados.2.smithi204.stdout:633: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:12:46.847 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 322 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:12:46.847 INFO:tasks.rados.rados.2.smithi204.stdout:635: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-16T05:12:46.854 INFO:tasks.rados.rados.2.smithi204.stdout:635: writing smithi20457400-22 from 551329 to 1261128 tid 1 2024-09-16T05:12:46.859 INFO:tasks.rados.rados.2.smithi204.stdout:635: writing smithi20457400-22 from 1870817 to 2151329 tid 2 2024-09-16T05:12:46.859 INFO:tasks.rados.rados.2.smithi204.stdout:632: done (2 left) 2024-09-16T05:12:46.859 INFO:tasks.rados.rados.2.smithi204.stdout:633: done (1 left) 2024-09-16T05:12:46.859 INFO:tasks.rados.rados.2.smithi204.stdout:636: write oid 32 current snap is 69 2024-09-16T05:12:46.859 INFO:tasks.rados.rados.2.smithi204.stdout:636: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-16T05:12:46.866 INFO:tasks.rados.rados.2.smithi204.stdout:636: writing smithi20457400-32 from 568136 to 1353184 tid 1 2024-09-16T05:12:46.871 INFO:tasks.rados.rados.2.smithi204.stdout:636: writing smithi20457400-32 from 1929299 to 2168136 tid 2 2024-09-16T05:12:46.871 INFO:tasks.rados.rados.2.smithi204.stdout:637: snap_create 2024-09-16T05:12:46.871 INFO:tasks.rados.rados.2.smithi204.stdout:635: finishing write tid 1 to smithi20457400-22 2024-09-16T05:12:46.871 INFO:tasks.rados.rados.2.smithi204.stdout:635: finishing write tid 2 to smithi20457400-22 2024-09-16T05:12:46.871 INFO:tasks.rados.rados.2.smithi204.stdout:636: finishing write tid 1 to smithi20457400-32 2024-09-16T05:12:46.872 INFO:tasks.rados.rados.2.smithi204.stdout:636: finishing write tid 2 to smithi20457400-32 2024-09-16T05:12:46.873 INFO:tasks.rados.rados.2.smithi204.stdout:636: finishing write tid 3 to smithi20457400-32 2024-09-16T05:12:46.873 INFO:tasks.rados.rados.2.smithi204.stdout:636: finishing write tid 4 to smithi20457400-32 2024-09-16T05:12:46.873 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 319 (ObjNum 248 snap 69 seq_num 248) dirty exists 2024-09-16T05:12:46.873 INFO:tasks.rados.rados.2.smithi204.stdout:636: left oid 32 (ObjNum 248 snap 69 seq_num 248) 2024-09-16T05:12:46.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:46.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:47.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:47.053 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:47.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:47.087 INFO:tasks.rados.rados.2.smithi204.stdout:635: finishing write tid 3 to smithi20457400-22 2024-09-16T05:12:47.087 INFO:tasks.rados.rados.2.smithi204.stdout:635: finishing write tid 4 to smithi20457400-22 2024-09-16T05:12:47.087 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 248 (ObjNum 247 snap 69 seq_num 247) dirty exists 2024-09-16T05:12:47.087 INFO:tasks.rados.rados.2.smithi204.stdout:635: left oid 22 (ObjNum 247 snap 69 seq_num 247) 2024-09-16T05:12:47.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:46 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:12:47.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:46 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:12:47.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:46 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:12:47.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:46 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:12:47.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:46 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:12:47.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:46 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:12:47.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2553370039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:47.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:46 smithi032 ceph-mon[27388]: osdmap e235: 12 total, 12 up, 11 in 2024-09-16T05:12:47.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:46 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:12:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:46 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:12:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:46 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:12:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:46 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:12:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:46 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:12:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:46 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:12:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2553370039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:46 smithi204 ceph-mon[30933]: osdmap e235: 12 total, 12 up, 11 in 2024-09-16T05:12:47.178 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:47.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:47.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:47.303 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:47.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:47.376 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.14155.0:16 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'6, mlcod=16'4)", 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 448.713263176, 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29879279399999997, 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.377 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661724+0000", 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661725+0000", 2024-09-16T05:12:47.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0800000000000003e-07 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661731+0000", 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4880000000000004e-06 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960116+0000", 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29838472799999999 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960124+0000", 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9689999999999994e-06 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960207+0000", 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3122999999999997e-05 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960225+0000", 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8099000000000001e-05 2024-09-16T05:12:47.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960487+0000", 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026233000000000002 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960517+0000", 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9149000000000002e-05 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:238 2.5 e115/107 2:a49174eb:::smithi20457400-39:head v 115'31, mlcod=115'28)", 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:47.037228+0000", 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 120.337759074, 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033756519999999998, 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.037228+0000", 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.037228+0000", 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.037229+0000", 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2500000000000001e-06 2024-09-16T05:12:47.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039475+0000", 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022458930000000001 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039524+0000", 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8851999999999999e-05 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039537+0000", 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2761e-05 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039629+0000", 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2516000000000007e-05 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.039658+0000", 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9122e-05 2024-09-16T05:12:47.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.040548+0000", 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088925199999999999 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:47.040604+0000", 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6005999999999999e-05 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:292 2.5 e117/107 2:a1d8c6d9:::smithi20457400-3:head v 117'37, mlcod=115'33)", 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:49.353041+0000", 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 118.021946189, 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047142110000000003, 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.353041+0000", 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.353041+0000", 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.353046+0000", 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6350000000000003e-06 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356065+0000", 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030184589999999998 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356114+0000", 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9397999999999997e-05 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356125+0000", 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0574000000000001e-05 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356200+0000", 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5099999999999996e-05 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.356229+0000", 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8750000000000001e-05 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.357714+0000", 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014852979999999999 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:49.357755+0000", 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0997000000000001e-05 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:454 2.c e127/107 2:33100188:::smithi20457400-19:head v 127'19, mlcod=126'16)", 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 108.674840437, 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054909060000000003, 2024-09-16T05:12:47.391 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7800000000000001e-07 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703134+0000", 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002986954 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703165+0000", 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1201000000000002e-05 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703173+0000", 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.32e-06 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703255+0000", 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1730000000000005e-05 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703282+0000", 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7389000000000002e-05 2024-09-16T05:12:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.705600+0000", 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023176360000000001 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.705637+0000", 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7497999999999999e-05 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:116 2.5 e141/107 2:ad4a88eb:::smithi20457400-16:a v 141'87, mlcod=141'79)", 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 94.679978907999995, 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006093261, 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695009+0000", 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6499999999999999e-07 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695014+0000", 2024-09-16T05:12:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8350000000000003e-06 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700654+0000", 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056407499999999999 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700660+0000", 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4879999999999998e-06 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700700+0000", 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0583000000000001e-05 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700714+0000", 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.322e-05 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701062+0000", 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00034833999999999999 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701101+0000", 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9480000000000001e-05 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:118 2.5 e141/107 2:ad7afe2b:::smithi20457400-38:d v 141'90, mlcod=141'85)", 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 94.673384584000004, 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066456359999999999, 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701603+0000", 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701606+0000", 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3890000000000001e-06 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707421+0000", 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005814601 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707428+0000", 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4839999999999998e-06 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707497+0000", 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8441000000000004e-05 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707520+0000", 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3513999999999999e-05 2024-09-16T05:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708185+0000", 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066482599999999998 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708248+0000", 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2959000000000005e-05 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:614 2.5 e144/107 2:a1d8c6d9:::smithi20457400-3:head v 144'110, mlcod=144'106)", 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 91.724418818999993, 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054521470000000001, 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650569+0000", 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7299999999999995e-07 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653581+0000", 2024-09-16T05:12:47.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003012617 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653627+0000", 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5649999999999998e-05 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653639+0000", 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2126e-05 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653715+0000", 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5485e-05 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653748+0000", 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3373e-05 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.655964+0000", 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002215875 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.656020+0000", 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6348000000000003e-05 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:172 2.c e179/107 2:3d4bedc8:::smithi20457400-22:26 v 179'101, mlcod=178'97)", 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.722261420999999, 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34174038899999998, 2024-09-16T05:12:47.410 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4999999999999999e-07 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652730+0000", 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8789999999999996e-06 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993974+0000", 2024-09-16T05:12:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34124383699999999 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993980+0000", 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4139999999999999e-06 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994036+0000", 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5850999999999998e-05 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994054+0000", 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8182999999999999e-05 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994414+0000", 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036047800000000001 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994466+0000", 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1496999999999998e-05 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:970 2.c e179/107 2:33100188:::smithi20457400-19:head v 179'103, mlcod=178'97)", 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.993178+0000", 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.381808536999998, 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026927880000000002, 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993178+0000", 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993178+0000", 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993180+0000", 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0780000000000001e-06 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993954+0000", 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077436400000000004 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994894+0000", 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000939933 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994905+0000", 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0651e-05 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994934+0000", 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9759000000000001e-05 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994947+0000", 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3158e-05 2024-09-16T05:12:47.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.995827+0000", 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088005799999999997 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.995871+0000", 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3786999999999999e-05 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:971 2.c e179/107 2:33100188:::smithi20457400-19:head v 179'104, mlcod=179'101)", 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.998237+0000", 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.376750203, 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028420379999999999, 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998237+0000", 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998237+0000", 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998238+0000", 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8000000000000004e-07 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998671+0000", 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043326900000000002 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999104+0000", 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000432738 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999113+0000", 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0839999999999998e-06 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999906+0000", 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079338599999999998 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999927+0000", 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0559000000000001e-05 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.001035+0000", 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001108541 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.001079+0000", 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3581000000000002e-05 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:972 2.c e179/107 2:33100188:::smithi20457400-19:head v 179'105, mlcod=179'101)", 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.375917520000002, 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.53639116399999998, 2024-09-16T05:12:47.424 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.425 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.425 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.425 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:12:47.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999070+0000", 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0999999999999997e-07 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999087+0000", 2024-09-16T05:12:47.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7014999999999999e-05 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534916+0000", 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.53582909000000001 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534923+0000", 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0099999999999998e-06 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534975+0000", 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1938999999999999e-05 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534994+0000", 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8504000000000001e-05 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.535427+0000", 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043298700000000002 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.535461+0000", 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3809000000000001e-05 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:979 2.4 e179/107 2:2ace617f:::smithi20457400-21:head v 179'81, mlcod=178'78)", 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.368117163999997, 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076005070000000003, 2024-09-16T05:12:47.429 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009810+0000", 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002939929 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009892+0000", 2024-09-16T05:12:47.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2349999999999996e-05 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009916+0000", 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3200000000000001e-05 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009983+0000", 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7577999999999996e-05 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010005+0000", 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2337999999999999e-05 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014425+0000", 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044196119999999998 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014470+0000", 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5297999999999999e-05 2024-09-16T05:12:47.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1004 2.4 e181/107 2:2ace617f:::smithi20457400-21:head v 181'87, mlcod=181'83)", 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.761359+0000", 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 54.613627749000003, 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003231448, 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.761359+0000", 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.761359+0000", 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.761361+0000", 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.463e-06 2024-09-16T05:12:47.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.762467+0000", 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011063469999999999 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.762514+0000", 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6993000000000002e-05 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.762524+0000", 2024-09-16T05:12:47.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9879999999999999e-06 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.762582+0000", 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8085999999999999e-05 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.762601+0000", 2024-09-16T05:12:47.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8822999999999999e-05 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764551+0000", 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019501449999999999 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764591+0000", 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9603e-05 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1017 2.5 e182/107 2:a1d8c6d9:::smithi20457400-3:head v 182'254, mlcod=182'250)", 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 53.676262516000001, 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067341900000000001, 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:12:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698726+0000", 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0410000000000001e-06 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703134+0000", 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044088850000000004 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703168+0000", 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3195999999999999e-05 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703175+0000", 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703234+0000", 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8859e-05 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703262+0000", 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7650000000000001e-05 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.705387+0000", 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021252670000000001 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.705459+0000", 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1598000000000001e-05 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:191 2.c e183/107 2:3a3554ba:::smithi20457400-48:1c v 183'116, mlcod=181'112)", 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:54.693394+0000", 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 52.681592682999998, 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002635933, 2024-09-16T05:12:47.444 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693394+0000", 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693394+0000", 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693395+0000", 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0699999999999999e-07 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693402+0000", 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4189999999999996e-06 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693480+0000", 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7714999999999997e-05 2024-09-16T05:12:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693491+0000", 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1229999999999999e-05 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693574+0000", 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3393999999999995e-05 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693607+0000", 2024-09-16T05:12:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2762999999999998e-05 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.695983+0000", 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002375622 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.696030+0000", 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7583000000000002e-05 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:192 2.c e183/107 2:3d4bedc8:::smithi20457400-22:26 v 183'118, mlcod=181'112)", 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:54.693404+0000", 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 52.681583123000003, 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002663548, 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693404+0000", 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693404+0000", 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693404+0000", 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8099999999999999e-07 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693410+0000", 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3770000000000001e-06 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693523+0000", 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000112859 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.693529+0000", 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3619999999999997e-06 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.694010+0000", 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048125099999999998 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.694033+0000", 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3315999999999998e-05 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.696034+0000", 2024-09-16T05:12:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020009060000000002 2024-09-16T05:12:47.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:54.696067+0000", 2024-09-16T05:12:47.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3296000000000001e-05 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1146 2.c e192/107 2:33100188:::smithi20457400-19:head v 192'151, mlcod=192'147)", 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 43.63202768, 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057288909999999998, 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742960+0000", 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3399999999999999e-07 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746015+0000", 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030547809999999999 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746040+0000", 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5075999999999998e-05 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746048+0000", 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9030000000000004e-06 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746099+0000", 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1004999999999997e-05 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746117+0000", 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8189000000000002e-05 2024-09-16T05:12:47.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.748630+0000", 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025129549999999999 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.748688+0000", 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8448e-05 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1161 2.c e192/107 2:3d4bedc8:::smithi20457400-22:head v 192'153, mlcod=192'149)", 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 43.603542132000001, 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055212639999999997, 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771447+0000", 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3360000000000002e-06 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774126+0000", 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026786790000000002 2024-09-16T05:12:47.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774176+0000", 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9669999999999997e-05 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774185+0000", 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4530000000000008e-06 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774260+0000", 2024-09-16T05:12:47.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5395999999999994e-05 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774291+0000", 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0561000000000001e-05 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.776739+0000", 2024-09-16T05:12:47.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002447968 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.776966+0000", 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022720100000000001 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1201 2.5 e194/183 2:a5bc1608:::smithi20457400-11:head v 194'298, mlcod=192'295)", 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 41.696044164, 2024-09-16T05:12:47.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080910559999999992, 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:12:47.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683786+0000", 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048424469999999997 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683833+0000", 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7540999999999998e-05 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683844+0000", 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1354e-05 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683920+0000", 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5760999999999995e-05 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683940+0000", 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9715999999999999e-05 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.686996+0000", 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003055954 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.687034+0000", 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8031999999999998e-05 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.470 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1557 2.1f 2:fc733d2f:::smithi20457400-45:head [write 732838~698856,stat] snapc 42=[42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2f,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:36.673137+0000", 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 10.701849710999999, 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036354960000000002, 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:12:47.471 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1557 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.673137+0000", 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.673137+0000", 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.673139+0000", 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.466e-06 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.673884+0000", 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074492899999999997 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.673910+0000", 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6744e-05 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.673918+0000", 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8599999999999993e-06 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.673978+0000", 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0009000000000001e-05 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.674051+0000", 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2917000000000007e-05 2024-09-16T05:12:47.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.674194+0000", 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000142791 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.675781+0000", 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001586788 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.676667+0000", 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00088639800000000005 2024-09-16T05:12:47.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.676722+0000", 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4778000000000003e-05 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:36.676773+0000", 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0816000000000003e-05 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:12:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:12:47.477 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:12:47.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:47.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:47.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:47.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:47.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:47.670 INFO:tasks.rados.rados.2.smithi204.stdout:635: done (2 left) 2024-09-16T05:12:47.670 INFO:tasks.rados.rados.2.smithi204.stdout:636: done (1 left) 2024-09-16T05:12:47.670 INFO:tasks.rados.rados.2.smithi204.stdout:637: done (0 left) 2024-09-16T05:12:47.670 INFO:tasks.rados.rados.2.smithi204.stdout:638: delete oid 12 current snap is 70 2024-09-16T05:12:47.672 INFO:tasks.rados.rados.2.smithi204.stdout:638: done (0 left) 2024-09-16T05:12:47.672 INFO:tasks.rados.rados.2.smithi204.stdout:639: read oid 12 snap -1 2024-09-16T05:12:47.672 INFO:tasks.rados.rados.2.smithi204.stdout:639: expect deleted 2024-09-16T05:12:47.672 INFO:tasks.rados.rados.2.smithi204.stdout:640: write (excl) oid 50 current snap is 70 2024-09-16T05:12:47.672 INFO:tasks.rados.rados.2.smithi204.stdout:640: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-16T05:12:47.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:12:47.678 INFO:tasks.rados.rados.2.smithi204.stdout:640: writing smithi20457400-50 from 601750 to 1137296 tid 1 2024-09-16T05:12:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:640: writing smithi20457400-50 from 1929910 to 2201750 tid 2 2024-09-16T05:12:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:641: read oid 44 snap -1 2024-09-16T05:12:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:641: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:12:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:642: rollback oid 10 current snap is 70 2024-09-16T05:12:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 62 2024-09-16T05:12:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:640: finishing write tid 1 to smithi20457400-50 2024-09-16T05:12:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:639: done (3 left) 2024-09-16T05:12:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:643: delete oid 28 current snap is 70 2024-09-16T05:12:47.687 INFO:tasks.rados.rados.2.smithi204.stdout:640: finishing write tid 2 to smithi20457400-50 2024-09-16T05:12:47.687 INFO:tasks.rados.rados.2.smithi204.stdout:640: finishing write tid 3 to smithi20457400-50 2024-09-16T05:12:47.687 INFO:tasks.rados.rados.2.smithi204.stdout:640: finishing write tid 4 to smithi20457400-50 2024-09-16T05:12:47.687 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 161 (ObjNum 250 snap 70 seq_num 250) dirty exists 2024-09-16T05:12:47.687 INFO:tasks.rados.rados.2.smithi204.stdout:640: left oid 50 (ObjNum 250 snap 70 seq_num 250) 2024-09-16T05:12:47.688 INFO:tasks.rados.rados.2.smithi204.stdout:640: done (3 left) 2024-09-16T05:12:47.688 INFO:tasks.rados.rados.2.smithi204.stdout:643: done (2 left) 2024-09-16T05:12:47.688 INFO:tasks.rados.rados.2.smithi204.stdout:644: write oid 24 current snap is 70 2024-09-16T05:12:47.688 INFO:tasks.rados.rados.2.smithi204.stdout:644: seq_num 252 ranges {635364=402397,1763227=472137} 2024-09-16T05:12:47.693 INFO:tasks.rados.rados.2.smithi204.stdout:644: writing smithi20457400-24 from 635364 to 1037761 tid 1 2024-09-16T05:12:47.700 INFO:tasks.rados.rados.2.smithi204.stdout:644: writing smithi20457400-24 from 1763227 to 2235364 tid 2 2024-09-16T05:12:47.700 INFO:tasks.rados.rados.2.smithi204.stdout:645: delete oid 46 current snap is 70 2024-09-16T05:12:47.700 INFO:tasks.rados.rados.2.smithi204.stdout:642: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:12:47.700 INFO:tasks.rados.rados.2.smithi204.stdout:642: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:12:47.701 INFO:tasks.rados.rados.2.smithi204.stdout:642: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:12:47.701 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 295 (ObjNum 229 snap 61 seq_num 229) dirty exists 2024-09-16T05:12:47.701 INFO:tasks.rados.rados.2.smithi204.stdout:641: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:12:47.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:47.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:47.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:47.821 INFO:tasks.rados.rados.2.smithi204.stdout:644: finishing write tid 1 to smithi20457400-24 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:644: finishing write tid 2 to smithi20457400-24 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:644: finishing write tid 3 to smithi20457400-24 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:644: finishing write tid 4 to smithi20457400-24 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 251 (ObjNum 252 snap 70 seq_num 252) dirty exists 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:644: left oid 24 (ObjNum 252 snap 70 seq_num 252) 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:641: done (3 left) 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:642: done (2 left) 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:644: done (1 left) 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:645: done (0 left) 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:646: write oid 47 current snap is 70 2024-09-16T05:12:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:646: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-16T05:12:47.828 INFO:tasks.rados.rados.2.smithi204.stdout:646: writing smithi20457400-47 from 668978 to 1221873 tid 1 2024-09-16T05:12:47.833 INFO:tasks.rados.rados.2.smithi204.stdout:646: writing smithi20457400-47 from 1763838 to 2268978 tid 2 2024-09-16T05:12:47.834 INFO:tasks.rados.rados.2.smithi204.stdout:647: snap_create 2024-09-16T05:12:47.834 INFO:tasks.rados.rados.2.smithi204.stdout:646: finishing write tid 1 to smithi20457400-47 2024-09-16T05:12:47.835 INFO:tasks.rados.rados.2.smithi204.stdout:646: finishing write tid 2 to smithi20457400-47 2024-09-16T05:12:47.842 INFO:tasks.rados.rados.2.smithi204.stdout:646: finishing write tid 3 to smithi20457400-47 2024-09-16T05:12:47.842 INFO:tasks.rados.rados.2.smithi204.stdout:646: finishing write tid 4 to smithi20457400-47 2024-09-16T05:12:47.842 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 202 (ObjNum 254 snap 70 seq_num 254) dirty exists 2024-09-16T05:12:47.842 INFO:tasks.rados.rados.2.smithi204.stdout:646: left oid 47 (ObjNum 254 snap 70 seq_num 254) 2024-09-16T05:12:47.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:47.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:47.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:47.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:47.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:47 smithi196 ceph-mon[30802]: pgmap v504: 47 pgs: 47 active+clean; 279 MiB data, 678 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 22 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:12:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:47 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:12:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:47 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:12:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:47 smithi196 ceph-mon[30802]: osdmap e236: 12 total, 12 up, 11 in 2024-09-16T05:12:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:47.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:48.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:48.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:47 smithi032 ceph-mon[27388]: pgmap v504: 47 pgs: 47 active+clean; 279 MiB data, 678 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 22 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:12:48.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:47 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:12:48.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:47 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:12:48.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:47 smithi032 ceph-mon[27388]: osdmap e236: 12 total, 12 up, 11 in 2024-09-16T05:12:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:48.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:48.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:48.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:12:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:47 smithi204 ceph-mon[30933]: pgmap v504: 47 pgs: 47 active+clean; 279 MiB data, 678 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 22 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:12:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:47 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:12:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:47 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:12:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:47 smithi204 ceph-mon[30933]: osdmap e236: 12 total, 12 up, 11 in 2024-09-16T05:12:48.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:48.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:48.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:48.264 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:48.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:48.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:48.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:48.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:48.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:48.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:48.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:48.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:48.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:48.690 INFO:tasks.rados.rados.2.smithi204.stdout:646: done (1 left) 2024-09-16T05:12:48.690 INFO:tasks.rados.rados.2.smithi204.stdout:647: done (0 left) 2024-09-16T05:12:48.690 INFO:tasks.rados.rados.2.smithi204.stdout:648: rollback oid 21 current snap is 71 2024-09-16T05:12:48.690 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 21 to 56 2024-09-16T05:12:48.690 INFO:tasks.rados.rados.2.smithi204.stdout:649: snap_create 2024-09-16T05:12:48.693 INFO:tasks.rados.rados.2.smithi204.stdout:648: finishing rollback tid 0 to smithi20457400-21 2024-09-16T05:12:48.694 INFO:tasks.rados.rados.2.smithi204.stdout:648: finishing rollback tid 1 to smithi20457400-21 2024-09-16T05:12:48.694 INFO:tasks.rados.rados.2.smithi204.stdout:648: finishing rollback tid 2 to smithi20457400-21 2024-09-16T05:12:48.694 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 164 (ObjNum 182 snap 46 seq_num 182) dirty exists 2024-09-16T05:12:48.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:48.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:48.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:48.790 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:48.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:48.884 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:48.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:48.911 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:48 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:12:48.912 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:48 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:12:48.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:48.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:48 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:12:48.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:48 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:12:49.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:49.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:49.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:49.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:49.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:49.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:12:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:48 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:12:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:48 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:12:49.222 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:12:49.223 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:12:49.223 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:12:49.223 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:12:49.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:49.267 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:49.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:49.296 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:12:49.296 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:12:49.297 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T05:12:49.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:12:49.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:49.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:49.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:49.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:49.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:49.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:49.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:49.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:49.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:49.689 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:12:49.692 INFO:tasks.rados.rados.2.smithi204.stdout:648: done (1 left) 2024-09-16T05:12:49.692 INFO:tasks.rados.rados.2.smithi204.stdout:649: done (0 left) 2024-09-16T05:12:49.692 INFO:tasks.rados.rados.2.smithi204.stdout:650: snap_create 2024-09-16T05:12:49.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:49.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:49.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:49.797 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:49 smithi032 ceph-mon[27388]: pgmap v507: 47 pgs: 47 active+clean; 275 MiB data, 704 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 28 op/s; 8.1 MiB/s, 10 objects/s recovering 2024-09-16T05:12:49.797 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:49 smithi032 ceph-mon[27388]: osdmap e237: 12 total, 12 up, 11 in 2024-09-16T05:12:49.797 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3873259977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:49.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:12:49.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:49.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:49.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:49 smithi196 ceph-mon[30802]: pgmap v507: 47 pgs: 47 active+clean; 275 MiB data, 704 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 28 op/s; 8.1 MiB/s, 10 objects/s recovering 2024-09-16T05:12:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:49 smithi196 ceph-mon[30802]: osdmap e237: 12 total, 12 up, 11 in 2024-09-16T05:12:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3873259977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:49.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:50.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:50.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:50.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:50.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:50.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:50.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:49 smithi204 ceph-mon[30933]: pgmap v507: 47 pgs: 47 active+clean; 275 MiB data, 704 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 28 op/s; 8.1 MiB/s, 10 objects/s recovering 2024-09-16T05:12:50.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:49 smithi204 ceph-mon[30933]: osdmap e237: 12 total, 12 up, 11 in 2024-09-16T05:12:50.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3873259977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:12:50.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:50.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:50.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:50.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:50.403 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:50.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:50.499 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:50.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:50.515 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:50.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:50.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:50.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:50.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:50.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:50.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:50.668 INFO:tasks.rados.rados.2.smithi204.stdout:650: done (0 left) 2024-09-16T05:12:50.669 INFO:tasks.rados.rados.2.smithi204.stdout:651: write (excl) oid 49 current snap is 73 2024-09-16T05:12:50.669 INFO:tasks.rados.rados.2.smithi204.stdout:651: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-16T05:12:50.675 INFO:tasks.rados.rados.2.smithi204.stdout:651: writing smithi20457400-49 from 685785 to 1313929 tid 1 2024-09-16T05:12:50.681 INFO:tasks.rados.rados.2.smithi204.stdout:651: writing smithi20457400-49 from 1822320 to 2285785 tid 2 2024-09-16T05:12:50.681 INFO:tasks.rados.rados.2.smithi204.stdout:652: rollback oid 10 current snap is 73 2024-09-16T05:12:50.682 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 66 2024-09-16T05:12:50.682 INFO:tasks.rados.rados.2.smithi204.stdout:651: finishing write tid 1 to smithi20457400-49 2024-09-16T05:12:50.682 INFO:tasks.rados.rados.2.smithi204.stdout:653: write (excl) oid 37 current snap is 73 2024-09-16T05:12:50.682 INFO:tasks.rados.rados.2.smithi204.stdout:653: seq_num 256 ranges {702592=703393,1880802=421790} 2024-09-16T05:12:50.689 INFO:tasks.rados.rados.2.smithi204.stdout:653: writing smithi20457400-37 from 702592 to 1405985 tid 1 2024-09-16T05:12:50.695 INFO:tasks.rados.rados.2.smithi204.stdout:653: writing smithi20457400-37 from 1880802 to 2302592 tid 2 2024-09-16T05:12:50.696 INFO:tasks.rados.rados.2.smithi204.stdout:654: snap_remove snap 62 2024-09-16T05:12:50.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:50.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:50.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:50.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:50.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:50.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3873259977' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: osdmap e238: 12 total, 12 up, 11 in 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:12:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:12:50.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:12:50.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:50 smithi032 ceph-mon[27388]: osdmap e239: 12 total, 12 up, 11 in 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3873259977' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: osdmap e238: 12 total, 12 up, 11 in 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:12:50.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:12:50.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:50 smithi196 ceph-mon[30802]: osdmap e239: 12 total, 12 up, 11 in 2024-09-16T05:12:51.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:51.032 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:51.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:51.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:51.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:51.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:12:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:12:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3873259977' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:12:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: osdmap e238: 12 total, 12 up, 11 in 2024-09-16T05:12:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:12:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:12:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:12:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:12:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:12:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:12:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:12:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:50 smithi204 ceph-mon[30933]: osdmap e239: 12 total, 12 up, 11 in 2024-09-16T05:12:51.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:51.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:51.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:51.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:51.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:51.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:51.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:51.378 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:51.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:51.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:51.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:51.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:51.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:51.378 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:51.379 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:51.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:51.400 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:51.400 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:51.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:51.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:51.422 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:51.422 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:51.444 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:51.444 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:51.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:51.465 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:51.489 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:51.489 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:51.509 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:51.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:51.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:51.545 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:12:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:12:51.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:51.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:51.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:51.699 INFO:tasks.rados.rados.2.smithi204.stdout:654: done (3 left) 2024-09-16T05:12:51.699 INFO:tasks.rados.rados.2.smithi204.stdout:655: snap_remove snap 68 2024-09-16T05:12:51.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:51 smithi032 ceph-mon[27388]: pgmap v510: 47 pgs: 47 active+clean; 279 MiB data, 721 MiB used, 983 GiB / 983 GiB avail; 6.3 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 8.0 MiB/s, 10 objects/s recovering 2024-09-16T05:12:51.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:51 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:12:51.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:51 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:12:51.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3442696650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:51.772 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:51 smithi032 ceph-mon[27388]: osdmap e240: 12 total, 12 up, 11 in 2024-09-16T05:12:51.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:51.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:51.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:51.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:51.918 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:51.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:51 smithi196 ceph-mon[30802]: pgmap v510: 47 pgs: 47 active+clean; 279 MiB data, 721 MiB used, 983 GiB / 983 GiB avail; 6.3 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 8.0 MiB/s, 10 objects/s recovering 2024-09-16T05:12:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:51 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:12:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:51 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:12:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3442696650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:51.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:51 smithi196 ceph-mon[30802]: osdmap e240: 12 total, 12 up, 11 in 2024-09-16T05:12:52.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:52.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:52.146 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:52.146 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:52.168 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:51 smithi204 ceph-mon[30933]: pgmap v510: 47 pgs: 47 active+clean; 279 MiB data, 721 MiB used, 983 GiB / 983 GiB avail; 6.3 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 8.0 MiB/s, 10 objects/s recovering 2024-09-16T05:12:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:51 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:12:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:51 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:12:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3442696650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:51 smithi204 ceph-mon[30933]: osdmap e240: 12 total, 12 up, 11 in 2024-09-16T05:12:52.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:52.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:52.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:52.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:52.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:12:52.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:52.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:52.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:52.678 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:52.701 INFO:tasks.rados.rados.2.smithi204.stdout:655: done (3 left) 2024-09-16T05:12:52.701 INFO:tasks.rados.rados.2.smithi204.stdout:656: read oid 11 snap 71 2024-09-16T05:12:52.701 INFO:tasks.rados.rados.2.smithi204.stdout:656: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:12:52.701 INFO:tasks.rados.rados.2.smithi204.stdout:657: copy_from oid 27 from oid 1 current snap is 73 2024-09-16T05:12:52.701 INFO:tasks.rados.rados.2.smithi204.stdout:651: finishing write tid 2 to smithi20457400-49 2024-09-16T05:12:52.701 INFO:tasks.rados.rados.2.smithi204.stdout:651: finishing write tid 3 to smithi20457400-49 2024-09-16T05:12:52.701 INFO:tasks.rados.rados.2.smithi204.stdout:651: finishing write tid 4 to smithi20457400-49 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 199 (ObjNum 255 snap 73 seq_num 255) dirty exists 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:651: left oid 49 (ObjNum 255 snap 73 seq_num 255) 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:652: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:652: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:652: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 298 (ObjNum 229 snap 61 seq_num 229) dirty exists 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:653: finishing write tid 1 to smithi20457400-37 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:653: finishing write tid 2 to smithi20457400-37 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:653: finishing write tid 3 to smithi20457400-37 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:653: finishing write tid 4 to smithi20457400-37 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 206 (ObjNum 256 snap 73 seq_num 256) dirty exists 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:653: left oid 37 (ObjNum 256 snap 73 seq_num 256) 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:651: done (4 left) 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:652: done (3 left) 2024-09-16T05:12:52.702 INFO:tasks.rados.rados.2.smithi204.stdout:653: done (2 left) 2024-09-16T05:12:52.703 INFO:tasks.rados.rados.2.smithi204.stdout:658: delete oid 37 current snap is 73 2024-09-16T05:12:52.703 INFO:tasks.rados.rados.2.smithi204.stdout:658: done (2 left) 2024-09-16T05:12:52.703 INFO:tasks.rados.rados.2.smithi204.stdout:659: copy_from oid 19 from oid 33 current snap is 73 2024-09-16T05:12:52.703 INFO:tasks.rados.rados.2.smithi204.stdout:660: snap_remove snap 61 2024-09-16T05:12:52.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:52.802 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:52.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:52.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:52.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:52.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:52 smithi196 ceph-mon[30802]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:52 smithi196 ceph-mon[30802]: osdmap e241: 12 total, 12 up, 11 in 2024-09-16T05:12:53.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:53.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:53.097 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:52 smithi032 ceph-mon[27388]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:53.097 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:52 smithi032 ceph-mon[27388]: osdmap e241: 12 total, 12 up, 11 in 2024-09-16T05:12:53.125 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-16T05:12:50.665029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":296375142,"num_objects":143,"num_object_clones":94,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":10,"num_read":265,"num_read_kb":280845,"num_write":178,"num_write_kb":80444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":33595633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6186,"ondisk_log_size":6186,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":740188,"kb_used_data":400580,"kb_used_omap":17,"kb_used_meta":338670,"kb_avail":1030456484,"statfs":{"total":1055945392128,"available":1055187439616,"internally_reserved":0,"allocated":410193920,"data_stored":497111444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":346798969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3533392,"num_objects":2,"num_object_clones":4,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":25824,"num_write":19,"num_write_kb":11911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":33403295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000092"},"pg_stats":[{"pgid":"2.23","version":"184'17","reported_seq":134,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T05:12:47.669589+0000","last_change":"2024-09-16T05:12:46.671347+0000","last_active":"2024-09-16T05:12:47.669589+0000","last_peered":"2024-09-16T05:12:47.669589+0000","last_clean":"2024-09-16T05:12:47.669589+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:12:47.669589+0000","last_undegraded":"2024-09-16T05:12:47.669589+0000","last_fullsized":"2024-09-16T05:12:47.669589+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:12:26.533655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.533655+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:29.099105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048295199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":35,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T05:12:47.669635+0000","last_change":"2024-09-16T05:12:46.674003+0000","last_active":"2024-09-16T05:12:47.669635+0000","last_peered":"2024-09-16T05:12:47.669635+0000","last_clean":"2024-09-16T05:12:47.669635+0000","last_became_active":"2024-09-16T05:12:44.679281+0000","last_became_peered":"2024-09-16T05:12:44.679281+0000","last_unstale":"2024-09-16T05:12:47.669635+0000","last_undegraded":"2024-09-16T05:12:47.669635+0000","last_fullsized":"2024-09-16T05:12:47.669635+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:45.564449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.564449+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:06.705690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002004039,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":677,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.767424+0000","last_change":"2024-09-16T05:12:43.681638+0000","last_active":"2024-09-16T05:12:45.767424+0000","last_peered":"2024-09-16T05:12:45.767424+0000","last_clean":"2024-09-16T05:12:45.767424+0000","last_became_active":"2024-09-16T05:12:23.806187+0000","last_became_peered":"2024-09-16T05:12:23.806187+0000","last_unstale":"2024-09-16T05:12:45.767424+0000","last_undegraded":"2024-09-16T05:12:45.767424+0000","last_fullsized":"2024-09-16T05:12:45.767424+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:26.747969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.747969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:43.442027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045934099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.20","version":"232'43","reported_seq":696,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.667647+0000","last_change":"2024-09-16T05:12:46.667987+0000","last_active":"2024-09-16T05:12:46.667647+0000","last_peered":"2024-09-16T05:12:46.667647+0000","last_clean":"2024-09-16T05:12:46.667647+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:12:46.667647+0000","last_undegraded":"2024-09-16T05:12:46.667647+0000","last_fullsized":"2024-09-16T05:12:46.667647+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"214'37","last_scrub_stamp":"2024-09-16T05:12:27.064860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.064860+0000","objects_scrubbed":3,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:29.338332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0005932,"stat_sum":{"num_bytes":7916920,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":15203,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.1f","version":"235'167","reported_seq":817,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.745098+0000","last_change":"2024-09-16T05:12:46.668115+0000","last_active":"2024-09-16T05:12:49.745098+0000","last_peered":"2024-09-16T05:12:49.745098+0000","last_clean":"2024-09-16T05:12:49.745098+0000","last_became_active":"2024-09-16T05:12:26.664097+0000","last_became_peered":"2024-09-16T05:12:26.664097+0000","last_unstale":"2024-09-16T05:12:49.745098+0000","last_undegraded":"2024-09-16T05:12:49.745098+0000","last_fullsized":"2024-09-16T05:12:49.745098+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'148","last_scrub_stamp":"2024-09-16T05:12:27.189264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.189264+0000","objects_scrubbed":2,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:28.960700+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023622809999999999,"stat_sum":{"num_bytes":8269877,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.1e","version":"235'211","reported_seq":866,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T05:12:47.669718+0000","last_change":"2024-09-16T05:12:46.674189+0000","last_active":"2024-09-16T05:12:47.669718+0000","last_peered":"2024-09-16T05:12:47.669718+0000","last_clean":"2024-09-16T05:12:47.669718+0000","last_became_active":"2024-09-16T05:12:23.805930+0000","last_became_peered":"2024-09-16T05:12:23.805930+0000","last_unstale":"2024-09-16T05:12:47.669718+0000","last_undegraded":"2024-09-16T05:12:47.669718+0000","last_fullsized":"2024-09-16T05:12:47.669718+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'171","last_scrub_stamp":"2024-09-16T05:12:27.537258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.537258+0000","objects_scrubbed":7,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:48.325885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0033274400000000001,"stat_sum":{"num_bytes":13901463,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":5495,"num_write":9,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.1d","version":"235'326","reported_seq":1009,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.673010+0000","last_change":"2024-09-16T05:12:46.673010+0000","last_active":"2024-09-16T05:12:46.673010+0000","last_peered":"2024-09-16T05:12:46.673010+0000","last_clean":"2024-09-16T05:12:46.673010+0000","last_became_active":"2024-09-16T05:12:44.698111+0000","last_became_peered":"2024-09-16T05:12:44.698111+0000","last_unstale":"2024-09-16T05:12:46.673010+0000","last_undegraded":"2024-09-16T05:12:46.673010+0000","last_fullsized":"2024-09-16T05:12:46.673010+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'276","last_scrub_stamp":"2024-09-16T05:12:29.635450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:29.635450+0000","objects_scrubbed":9,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.838413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0054185400000000003,"stat_sum":{"num_bytes":22590835,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":3886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":16830963,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.1c","version":"235'244","reported_seq":907,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.670982+0000","last_change":"2024-09-16T05:12:46.670982+0000","last_active":"2024-09-16T05:12:46.670982+0000","last_peered":"2024-09-16T05:12:46.670982+0000","last_clean":"2024-09-16T05:12:46.670982+0000","last_became_active":"2024-09-16T05:12:44.684658+0000","last_became_peered":"2024-09-16T05:12:44.684658+0000","last_unstale":"2024-09-16T05:12:46.670982+0000","last_undegraded":"2024-09-16T05:12:46.670982+0000","last_fullsized":"2024-09-16T05:12:46.670982+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'202","last_scrub_stamp":"2024-09-16T05:12:28.627965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.627965+0000","objects_scrubbed":8,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.385495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0033928109999999999,"stat_sum":{"num_bytes":22462598,"num_objects":11,"num_object_clones":8,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":5,"num_read_kb":3,"num_write":19,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13583680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.1b","version":"235'195","reported_seq":861,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.673220+0000","last_change":"2024-09-16T05:12:46.673555+0000","last_active":"2024-09-16T05:12:46.673220+0000","last_peered":"2024-09-16T05:12:46.673220+0000","last_clean":"2024-09-16T05:12:46.673220+0000","last_became_active":"2024-09-16T05:12:23.807872+0000","last_became_peered":"2024-09-16T05:12:23.807872+0000","last_unstale":"2024-09-16T05:12:46.673220+0000","last_undegraded":"2024-09-16T05:12:46.673220+0000","last_fullsized":"2024-09-16T05:12:46.673220+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"214'162","last_scrub_stamp":"2024-09-16T05:12:26.326323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.326323+0000","objects_scrubbed":7,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.795396+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026288769999999999,"stat_sum":{"num_bytes":12026115,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":10164,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.1a","version":"236'321","reported_seq":952,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-16T05:12:48.688067+0000","last_change":"2024-09-16T05:12:46.670340+0000","last_active":"2024-09-16T05:12:48.688067+0000","last_peered":"2024-09-16T05:12:48.688067+0000","last_clean":"2024-09-16T05:12:48.688067+0000","last_became_active":"2024-09-16T05:12:27.664055+0000","last_became_peered":"2024-09-16T05:12:27.664055+0000","last_unstale":"2024-09-16T05:12:48.688067+0000","last_undegraded":"2024-09-16T05:12:48.688067+0000","last_fullsized":"2024-09-16T05:12:48.688067+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'278","last_scrub_stamp":"2024-09-16T05:12:28.451582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.451582+0000","objects_scrubbed":11,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:48.147074+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0039204399999999999,"stat_sum":{"num_bytes":23434037,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":10,"num_read_kb":9481,"num_write":8,"num_write_kb":1001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":256,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T05:12:47.669497+0000","last_change":"2024-09-16T05:12:46.671814+0000","last_active":"2024-09-16T05:12:47.669497+0000","last_peered":"2024-09-16T05:12:47.669497+0000","last_clean":"2024-09-16T05:12:47.669497+0000","last_became_active":"2024-09-16T05:11:57.658646+0000","last_became_peered":"2024-09-16T05:11:57.658646+0000","last_unstale":"2024-09-16T05:12:47.669497+0000","last_undegraded":"2024-09-16T05:12:47.669497+0000","last_fullsized":"2024-09-16T05:12:47.669497+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:12:09.470099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.470099+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:19.442626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086837399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":655,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.837893+0000","last_change":"2024-09-16T05:12:46.669710+0000","last_active":"2024-09-16T05:12:49.837893+0000","last_peered":"2024-09-16T05:12:49.837893+0000","last_clean":"2024-09-16T05:12:49.837893+0000","last_became_active":"2024-09-16T05:11:57.660350+0000","last_became_peered":"2024-09-16T05:11:57.660350+0000","last_unstale":"2024-09-16T05:12:49.837893+0000","last_undegraded":"2024-09-16T05:12:49.837893+0000","last_fullsized":"2024-09-16T05:12:49.837893+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:12:08.969861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.969861+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:26.813860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056382199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.17","version":"0'0","reported_seq":438,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.669479+0000","last_change":"2024-09-16T05:12:46.669806+0000","last_active":"2024-09-16T05:12:46.669479+0000","last_peered":"2024-09-16T05:12:46.669479+0000","last_clean":"2024-09-16T05:12:46.669479+0000","last_became_active":"2024-09-16T05:12:44.679525+0000","last_became_peered":"2024-09-16T05:12:44.679525+0000","last_unstale":"2024-09-16T05:12:46.669479+0000","last_undegraded":"2024-09-16T05:12:46.669479+0000","last_fullsized":"2024-09-16T05:12:46.669479+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:12:09.877069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.877069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:38.402556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077799499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.16","version":"232'30","reported_seq":280,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.768014+0000","last_change":"2024-09-16T05:12:43.678982+0000","last_active":"2024-09-16T05:12:45.768014+0000","last_peered":"2024-09-16T05:12:45.768014+0000","last_clean":"2024-09-16T05:12:45.768014+0000","last_became_active":"2024-09-16T05:11:57.656772+0000","last_became_peered":"2024-09-16T05:11:57.656772+0000","last_unstale":"2024-09-16T05:12:45.768014+0000","last_undegraded":"2024-09-16T05:12:45.768014+0000","last_fullsized":"2024-09-16T05:12:45.768014+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:12:09.331857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.331857+0000","objects_scrubbed":1,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:13.321193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023706780000000002,"stat_sum":{"num_bytes":5826062,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":26292,"num_write":9,"num_write_kb":2494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.15","version":"235'336","reported_seq":974,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.838080+0000","last_change":"2024-09-16T05:12:46.675574+0000","last_active":"2024-09-16T05:12:49.838080+0000","last_peered":"2024-09-16T05:12:49.838080+0000","last_clean":"2024-09-16T05:12:49.838080+0000","last_became_active":"2024-09-16T05:12:00.711890+0000","last_became_peered":"2024-09-16T05:12:00.711890+0000","last_unstale":"2024-09-16T05:12:49.838080+0000","last_undegraded":"2024-09-16T05:12:49.838080+0000","last_fullsized":"2024-09-16T05:12:49.838080+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'276","last_scrub_stamp":"2024-09-16T05:12:11.920515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:11.920515+0000","objects_scrubbed":7,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.234266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044031859999999999,"stat_sum":{"num_bytes":17392677,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":32,"num_read_kb":56081,"num_write":14,"num_write_kb":9530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.14","version":"237'142","reported_seq":372,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.837963+0000","last_change":"2024-09-16T05:12:46.672990+0000","last_active":"2024-09-16T05:12:49.837963+0000","last_peered":"2024-09-16T05:12:49.837963+0000","last_clean":"2024-09-16T05:12:49.837963+0000","last_became_active":"2024-09-16T05:11:57.658852+0000","last_became_peered":"2024-09-16T05:11:57.658852+0000","last_unstale":"2024-09-16T05:12:49.837963+0000","last_undegraded":"2024-09-16T05:12:49.837963+0000","last_fullsized":"2024-09-16T05:12:49.837963+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'108","last_scrub_stamp":"2024-09-16T05:12:10.924782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.924782+0000","objects_scrubbed":6,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:20.836560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003073452,"stat_sum":{"num_bytes":18723582,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":28,"num_read_kb":31484,"num_write":7,"num_write_kb":4335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.13","version":"181'15","reported_seq":262,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.838072+0000","last_change":"2024-09-16T05:12:46.671743+0000","last_active":"2024-09-16T05:12:49.838072+0000","last_peered":"2024-09-16T05:12:49.838072+0000","last_clean":"2024-09-16T05:12:49.838072+0000","last_became_active":"2024-09-16T05:11:57.660401+0000","last_became_peered":"2024-09-16T05:11:57.660401+0000","last_unstale":"2024-09-16T05:12:49.838072+0000","last_undegraded":"2024-09-16T05:12:49.838072+0000","last_fullsized":"2024-09-16T05:12:49.838072+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:12:09.952097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.952097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:20.599982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00093207700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.12","version":"233'38","reported_seq":242,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T05:12:47.669752+0000","last_change":"2024-09-16T05:12:46.673212+0000","last_active":"2024-09-16T05:12:47.669752+0000","last_peered":"2024-09-16T05:12:47.669752+0000","last_clean":"2024-09-16T05:12:47.669752+0000","last_became_active":"2024-09-16T05:12:44.682320+0000","last_became_peered":"2024-09-16T05:12:44.682320+0000","last_unstale":"2024-09-16T05:12:47.669752+0000","last_undegraded":"2024-09-16T05:12:47.669752+0000","last_fullsized":"2024-09-16T05:12:47.669752+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'28","last_scrub_stamp":"2024-09-16T05:12:09.669066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.669066+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:24.634917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001482101,"stat_sum":{"num_bytes":3662698,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3180990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":681,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.837845+0000","last_change":"2024-09-16T05:12:13.922480+0000","last_active":"2024-09-16T05:12:49.837845+0000","last_peered":"2024-09-16T05:12:49.837845+0000","last_clean":"2024-09-16T05:12:49.837845+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:12:49.837845+0000","last_undegraded":"2024-09-16T05:12:49.837845+0000","last_fullsized":"2024-09-16T05:12:49.837845+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.922401+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.922401+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.949519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"236'192","reported_seq":839,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.837957+0000","last_change":"2024-09-16T05:12:46.678373+0000","last_active":"2024-09-16T05:12:49.837957+0000","last_peered":"2024-09-16T05:12:49.837957+0000","last_clean":"2024-09-16T05:12:49.837957+0000","last_became_active":"2024-09-16T05:11:55.657137+0000","last_became_peered":"2024-09-16T05:11:55.657137+0000","last_unstale":"2024-09-16T05:12:49.837957+0000","last_undegraded":"2024-09-16T05:12:49.837957+0000","last_fullsized":"2024-09-16T05:12:49.837957+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'70","last_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.116608+0000","objects_scrubbed":7,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:19.031506+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.002918852,"stat_sum":{"num_bytes":21834208,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":24,"num_read_kb":15470,"num_write":23,"num_write_kb":8487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":669,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.667444+0000","last_change":"2024-09-16T05:10:43.759254+0000","last_active":"2024-09-16T05:12:46.667444+0000","last_peered":"2024-09-16T05:12:46.667444+0000","last_clean":"2024-09-16T05:12:46.667444+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:12:46.667444+0000","last_undegraded":"2024-09-16T05:12:46.667444+0000","last_fullsized":"2024-09-16T05:12:46.667444+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:10:43.759130+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:10:43.759130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"235'307","reported_seq":943,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.674932+0000","last_change":"2024-09-16T05:12:46.674932+0000","last_active":"2024-09-16T05:12:46.674932+0000","last_peered":"2024-09-16T05:12:46.674932+0000","last_clean":"2024-09-16T05:12:46.674932+0000","last_became_active":"2024-09-16T05:12:21.665466+0000","last_became_peered":"2024-09-16T05:12:21.665466+0000","last_unstale":"2024-09-16T05:12:46.674932+0000","last_undegraded":"2024-09-16T05:12:46.674932+0000","last_fullsized":"2024-09-16T05:12:46.674932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'276","last_scrub_stamp":"2024-09-16T05:12:28.342526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.342526+0000","objects_scrubbed":8,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:50.621756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0040477739999999996,"stat_sum":{"num_bytes":17402062,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":13,"num_read_kb":21868,"num_write":3,"num_write_kb":5040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":669,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.768016+0000","last_change":"2024-09-16T05:12:13.374972+0000","last_active":"2024-09-16T05:12:45.768016+0000","last_peered":"2024-09-16T05:12:45.768016+0000","last_clean":"2024-09-16T05:12:45.768016+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:12:45.768016+0000","last_undegraded":"2024-09-16T05:12:45.768016+0000","last_fullsized":"2024-09-16T05:12:45.768016+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.374887+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.374887+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:25.238853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"232'175","reported_seq":816,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-16T05:12:48.687895+0000","last_change":"2024-09-16T05:12:46.667039+0000","last_active":"2024-09-16T05:12:48.687895+0000","last_peered":"2024-09-16T05:12:48.687895+0000","last_clean":"2024-09-16T05:12:48.687895+0000","last_became_active":"2024-09-16T05:12:21.661312+0000","last_became_peered":"2024-09-16T05:12:21.661312+0000","last_unstale":"2024-09-16T05:12:48.687895+0000","last_undegraded":"2024-09-16T05:12:48.687895+0000","last_fullsized":"2024-09-16T05:12:48.687895+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'161","last_scrub_stamp":"2024-09-16T05:12:26.506811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.506811+0000","objects_scrubbed":4,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.033134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063733400000000003,"stat_sum":{"num_bytes":4261034,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":501,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T05:12:47.672620+0000","last_change":"2024-09-16T05:12:47.672398+0000","last_active":"2024-09-16T05:12:47.672620+0000","last_peered":"2024-09-16T05:12:47.672620+0000","last_clean":"2024-09-16T05:12:47.672620+0000","last_became_active":"2024-09-16T05:12:47.672164+0000","last_became_peered":"2024-09-16T05:12:47.672164+0000","last_unstale":"2024-09-16T05:12:47.672620+0000","last_undegraded":"2024-09-16T05:12:47.672620+0000","last_fullsized":"2024-09-16T05:12:47.672620+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:27.759006+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:27.759006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":720,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.838042+0000","last_change":"2024-09-16T05:12:46.676692+0000","last_active":"2024-09-16T05:12:49.838042+0000","last_peered":"2024-09-16T05:12:49.838042+0000","last_clean":"2024-09-16T05:12:49.838042+0000","last_became_active":"2024-09-16T05:11:55.665450+0000","last_became_peered":"2024-09-16T05:11:55.665450+0000","last_unstale":"2024-09-16T05:12:49.838042+0000","last_undegraded":"2024-09-16T05:12:49.838042+0000","last_fullsized":"2024-09-16T05:12:49.838042+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:12:10.025043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.025043+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:35.446952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011717279999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":426,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.837943+0000","last_change":"2024-09-16T05:12:45.101784+0000","last_active":"2024-09-16T05:12:49.837943+0000","last_peered":"2024-09-16T05:12:49.837943+0000","last_clean":"2024-09-16T05:12:49.837943+0000","last_became_active":"2024-09-16T05:12:44.692856+0000","last_became_peered":"2024-09-16T05:12:44.692856+0000","last_unstale":"2024-09-16T05:12:49.837943+0000","last_undegraded":"2024-09-16T05:12:49.837943+0000","last_fullsized":"2024-09-16T05:12:49.837943+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:45.101705+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.101705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:51.855025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"235'402","reported_seq":1011,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.838027+0000","last_change":"2024-09-16T05:12:46.674576+0000","last_active":"2024-09-16T05:12:49.838027+0000","last_peered":"2024-09-16T05:12:49.838027+0000","last_clean":"2024-09-16T05:12:49.838027+0000","last_became_active":"2024-09-16T05:12:12.663643+0000","last_became_peered":"2024-09-16T05:12:12.663643+0000","last_unstale":"2024-09-16T05:12:49.838027+0000","last_undegraded":"2024-09-16T05:12:49.838027+0000","last_fullsized":"2024-09-16T05:12:49.838027+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'266","last_scrub_stamp":"2024-09-16T05:11:56.487686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:11:56.487686+0000","objects_scrubbed":7,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:24.437472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003655036,"stat_sum":{"num_bytes":31098166,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":31224,"num_write":28,"num_write_kb":16878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":418,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.767462+0000","last_change":"2024-09-16T05:12:44.751434+0000","last_active":"2024-09-16T05:12:45.767462+0000","last_peered":"2024-09-16T05:12:45.767462+0000","last_clean":"2024-09-16T05:12:45.767462+0000","last_became_active":"2024-09-16T05:12:44.684578+0000","last_became_peered":"2024-09-16T05:12:44.684578+0000","last_unstale":"2024-09-16T05:12:45.767462+0000","last_undegraded":"2024-09-16T05:12:45.767462+0000","last_fullsized":"2024-09-16T05:12:45.767462+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:44.751365+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:44.751365+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:58.531322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":646,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.838181+0000","last_change":"2024-09-16T05:12:46.676091+0000","last_active":"2024-09-16T05:12:49.838181+0000","last_peered":"2024-09-16T05:12:49.838181+0000","last_clean":"2024-09-16T05:12:49.838181+0000","last_became_active":"2024-09-16T05:11:55.657738+0000","last_became_peered":"2024-09-16T05:11:55.657738+0000","last_unstale":"2024-09-16T05:12:49.838181+0000","last_undegraded":"2024-09-16T05:12:49.838181+0000","last_fullsized":"2024-09-16T05:12:49.838181+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:12:08.997285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.997285+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:27.103879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067972400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":634,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.670487+0000","last_change":"2024-09-16T05:12:09.317163+0000","last_active":"2024-09-16T05:12:46.670487+0000","last_peered":"2024-09-16T05:12:46.670487+0000","last_clean":"2024-09-16T05:12:46.670487+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:12:46.670487+0000","last_undegraded":"2024-09-16T05:12:46.670487+0000","last_fullsized":"2024-09-16T05:12:46.670487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:09.317055+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.317055+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.261087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.781494+0000","last_change":"2024-09-16T05:12:45.781494+0000","last_active":"2024-09-16T05:12:45.781494+0000","last_peered":"2024-09-16T05:12:45.781494+0000","last_clean":"2024-09-16T05:12:45.781494+0000","last_became_active":"2024-09-16T05:12:44.684566+0000","last_became_peered":"2024-09-16T05:12:44.684566+0000","last_unstale":"2024-09-16T05:12:45.781494+0000","last_undegraded":"2024-09-16T05:12:45.781494+0000","last_fullsized":"2024-09-16T05:12:45.781494+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:12:45.781416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.781416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:03.429864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001796944,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.0","version":"205'33","reported_seq":673,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.671061+0000","last_change":"2024-09-16T05:12:46.671241+0000","last_active":"2024-09-16T05:12:46.671061+0000","last_peered":"2024-09-16T05:12:46.671061+0000","last_clean":"2024-09-16T05:12:46.671061+0000","last_became_active":"2024-09-16T05:12:44.681562+0000","last_became_peered":"2024-09-16T05:12:44.681562+0000","last_unstale":"2024-09-16T05:12:46.671061+0000","last_undegraded":"2024-09-16T05:12:46.671061+0000","last_fullsized":"2024-09-16T05:12:46.671061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:12:28.044321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.044321+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:57.862521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003467754,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":679,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T05:12:47.669597+0000","last_change":"2024-09-16T05:11:53.500880+0000","last_active":"2024-09-16T05:12:47.669597+0000","last_peered":"2024-09-16T05:12:47.669597+0000","last_clean":"2024-09-16T05:12:47.669597+0000","last_became_active":"2024-09-16T05:11:19.419149+0000","last_became_peered":"2024-09-16T05:11:19.419149+0000","last_unstale":"2024-09-16T05:12:47.669597+0000","last_undegraded":"2024-09-16T05:12:47.669597+0000","last_fullsized":"2024-09-16T05:12:47.669597+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:53.500734+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:53.500734+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:10.037163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":675,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.767771+0000","last_change":"2024-09-16T05:12:43.681910+0000","last_active":"2024-09-16T05:12:45.767771+0000","last_peered":"2024-09-16T05:12:45.767771+0000","last_clean":"2024-09-16T05:12:45.767771+0000","last_became_active":"2024-09-16T05:12:21.666300+0000","last_became_peered":"2024-09-16T05:12:21.666300+0000","last_unstale":"2024-09-16T05:12:45.767771+0000","last_undegraded":"2024-09-16T05:12:45.767771+0000","last_fullsized":"2024-09-16T05:12:45.767771+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:25.777434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:25.777434+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:34.019695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070610900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.2","version":"35'139","reported_seq":677,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.667389+0000","last_change":"2024-09-16T05:12:31.125618+0000","last_active":"2024-09-16T05:12:46.667389+0000","last_peered":"2024-09-16T05:12:46.667389+0000","last_clean":"2024-09-16T05:12:46.667389+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:12:46.667389+0000","last_undegraded":"2024-09-16T05:12:46.667389+0000","last_fullsized":"2024-09-16T05:12:46.667389+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:31.125520+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:31.125520+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.820470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":425,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.837767+0000","last_change":"2024-09-16T05:12:46.116747+0000","last_active":"2024-09-16T05:12:49.837767+0000","last_peered":"2024-09-16T05:12:49.837767+0000","last_clean":"2024-09-16T05:12:49.837767+0000","last_became_active":"2024-09-16T05:12:44.814612+0000","last_became_peered":"2024-09-16T05:12:44.814612+0000","last_unstale":"2024-09-16T05:12:49.837767+0000","last_undegraded":"2024-09-16T05:12:49.837767+0000","last_fullsized":"2024-09-16T05:12:49.837767+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:46.116628+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:46.116628+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.926144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":638,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.768061+0000","last_change":"2024-09-16T05:12:43.680423+0000","last_active":"2024-09-16T05:12:45.768061+0000","last_peered":"2024-09-16T05:12:45.768061+0000","last_clean":"2024-09-16T05:12:45.768061+0000","last_became_active":"2024-09-16T05:12:21.660508+0000","last_became_peered":"2024-09-16T05:12:21.660508+0000","last_unstale":"2024-09-16T05:12:45.768061+0000","last_undegraded":"2024-09-16T05:12:45.768061+0000","last_fullsized":"2024-09-16T05:12:45.768061+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:27.335469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.335469+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:48.857884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038119270000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"43","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":673,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.667645+0000","last_change":"2024-09-16T05:11:54.869494+0000","last_active":"2024-09-16T05:12:46.667645+0000","last_peered":"2024-09-16T05:12:46.667645+0000","last_clean":"2024-09-16T05:12:46.667645+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:12:46.667645+0000","last_undegraded":"2024-09-16T05:12:46.667645+0000","last_fullsized":"2024-09-16T05:12:46.667645+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:54.869407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:54.869407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:18.625896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"224'22","reported_seq":650,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.768317+0000","last_change":"2024-09-16T05:12:43.684811+0000","last_active":"2024-09-16T05:12:45.768317+0000","last_peered":"2024-09-16T05:12:45.768317+0000","last_clean":"2024-09-16T05:12:45.768317+0000","last_became_active":"2024-09-16T05:12:21.663580+0000","last_became_peered":"2024-09-16T05:12:21.663580+0000","last_unstale":"2024-09-16T05:12:45.768317+0000","last_undegraded":"2024-09-16T05:12:45.768317+0000","last_fullsized":"2024-09-16T05:12:45.768317+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-09-16T05:12:26.380594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.380594+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:48.199823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.008244665,"stat_sum":{"num_bytes":6453855,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":16028,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.8","version":"232'101","reported_seq":723,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.668292+0000","last_change":"2024-09-16T05:12:46.668292+0000","last_active":"2024-09-16T05:12:46.668292+0000","last_peered":"2024-09-16T05:12:46.668292+0000","last_clean":"2024-09-16T05:12:46.668292+0000","last_became_active":"2024-09-16T05:11:55.658383+0000","last_became_peered":"2024-09-16T05:11:55.658383+0000","last_unstale":"2024-09-16T05:12:46.668292+0000","last_undegraded":"2024-09-16T05:12:46.668292+0000","last_fullsized":"2024-09-16T05:12:46.668292+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'67","last_scrub_stamp":"2024-09-16T05:12:09.039188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.039188+0000","objects_scrubbed":4,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:33.560805+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00080218499999999996,"stat_sum":{"num_bytes":6437048,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.c","version":"232'193","reported_seq":808,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.670987+0000","last_change":"2024-09-16T05:12:46.671564+0000","last_active":"2024-09-16T05:12:46.670987+0000","last_peered":"2024-09-16T05:12:46.670987+0000","last_clean":"2024-09-16T05:12:46.670987+0000","last_became_active":"2024-09-16T05:12:21.664367+0000","last_became_peered":"2024-09-16T05:12:21.664367+0000","last_unstale":"2024-09-16T05:12:46.670987+0000","last_undegraded":"2024-09-16T05:12:46.670987+0000","last_fullsized":"2024-09-16T05:12:46.670987+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'183","last_scrub_stamp":"2024-09-16T05:12:27.313488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.313488+0000","objects_scrubbed":3,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:34.971784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001083337,"stat_sum":{"num_bytes":2957999,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.d","version":"235'292","reported_seq":931,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-16T05:12:49.838172+0000","last_change":"2024-09-16T05:12:46.676425+0000","last_active":"2024-09-16T05:12:49.838172+0000","last_peered":"2024-09-16T05:12:49.838172+0000","last_clean":"2024-09-16T05:12:49.838172+0000","last_became_active":"2024-09-16T05:12:21.707118+0000","last_became_peered":"2024-09-16T05:12:21.707118+0000","last_unstale":"2024-09-16T05:12:49.838172+0000","last_undegraded":"2024-09-16T05:12:49.838172+0000","last_fullsized":"2024-09-16T05:12:49.838172+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'262","last_scrub_stamp":"2024-09-16T05:12:25.983777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:25.983777+0000","objects_scrubbed":3,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.080202+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0050091709999999998,"stat_sum":{"num_bytes":11143046,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":14,"num_read_kb":11826,"num_write":10,"num_write_kb":2534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.e","version":"232'188","reported_seq":815,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T05:12:45.767665+0000","last_change":"2024-09-16T05:12:43.684351+0000","last_active":"2024-09-16T05:12:45.767665+0000","last_peered":"2024-09-16T05:12:45.767665+0000","last_clean":"2024-09-16T05:12:45.767665+0000","last_became_active":"2024-09-16T05:12:21.665948+0000","last_became_peered":"2024-09-16T05:12:21.665948+0000","last_unstale":"2024-09-16T05:12:45.767665+0000","last_undegraded":"2024-09-16T05:12:45.767665+0000","last_fullsized":"2024-09-16T05:12:45.767665+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'160","last_scrub_stamp":"2024-09-16T05:12:27.786933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.786933+0000","objects_scrubbed":2,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:41.564813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029203940000000002,"stat_sum":{"num_bytes":11855640,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":3687,"num_write":13,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.f","version":"236'202","reported_seq":847,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-16T05:12:48.687974+0000","last_change":"2024-09-16T05:12:46.669348+0000","last_active":"2024-09-16T05:12:48.687974+0000","last_peered":"2024-09-16T05:12:48.687974+0000","last_clean":"2024-09-16T05:12:48.687974+0000","last_became_active":"2024-09-16T05:12:21.662019+0000","last_became_peered":"2024-09-16T05:12:21.662019+0000","last_unstale":"2024-09-16T05:12:48.687974+0000","last_undegraded":"2024-09-16T05:12:48.687974+0000","last_fullsized":"2024-09-16T05:12:48.687974+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'165","last_scrub_stamp":"2024-09-16T05:12:27.494268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.494268+0000","objects_scrubbed":8,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:30.526139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029318790000000001,"stat_sum":{"num_bytes":16538088,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":10,"num_read_kb":10191,"num_write":5,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":257,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-16T05:12:48.687850+0000","last_change":"2024-09-16T05:12:46.667248+0000","last_active":"2024-09-16T05:12:48.687850+0000","last_peered":"2024-09-16T05:12:48.687850+0000","last_clean":"2024-09-16T05:12:48.687850+0000","last_became_active":"2024-09-16T05:11:57.658263+0000","last_became_peered":"2024-09-16T05:11:57.658263+0000","last_unstale":"2024-09-16T05:12:48.687850+0000","last_undegraded":"2024-09-16T05:12:48.687850+0000","last_fullsized":"2024-09-16T05:12:48.687850+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:12:09.468817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.468817+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:28.166999+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00083885700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.11","version":"235'94","reported_seq":321,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-16T05:12:46.670826+0000","last_change":"2024-09-16T05:12:46.671016+0000","last_active":"2024-09-16T05:12:46.670826+0000","last_peered":"2024-09-16T05:12:46.670826+0000","last_clean":"2024-09-16T05:12:46.670826+0000","last_became_active":"2024-09-16T05:11:57.662582+0000","last_became_peered":"2024-09-16T05:11:57.662582+0000","last_unstale":"2024-09-16T05:12:46.670826+0000","last_undegraded":"2024-09-16T05:12:46.670826+0000","last_fullsized":"2024-09-16T05:12:46.670826+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'64","last_scrub_stamp":"2024-09-16T05:12:08.859612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.859612+0000","objects_scrubbed":3,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:25.021112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015379949999999999,"stat_sum":{"num_bytes":9596764,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":10146,"num_write":6,"num_write_kb":5277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"21","length":"1"},{"start":"2d","length":"1"},{"start":"38","length":"1"},{"start":"3d","length":"1"},{"start":"43","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":36,"stat_sum":{"num_bytes":295784774,"num_objects":141,"num_object_clones":94,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":10,"num_read":265,"num_read_kb":280845,"num_write":178,"num_write_kb":80444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":33595633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":412278784,"data_stored":519403942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4657,"ondisk_log_size":4657,"up":72,"acting":72,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089388,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93412,"kb_used_data":62224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93651740,"statfs":{"total":95995035648,"available":95899381760,"internally_reserved":0,"allocated":63717376,"data_stored":87958238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":356482285617,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514552,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98676,"kb_used_data":67488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93646476,"statfs":{"total":95995035648,"available":95893991424,"internally_reserved":0,"allocated":69107712,"data_stored":84535309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841594,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75212,"kb_used_data":44016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93669940,"statfs":{"total":95995035648,"available":95918018560,"internally_reserved":0,"allocated":45072384,"data_stored":54225855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103231,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95100,"kb_used_data":63252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93650052,"statfs":{"total":95995035648,"available":95897653248,"internally_reserved":0,"allocated":64770048,"data_stored":76406191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397574,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66512,"kb_used_data":35324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93678640,"statfs":{"total":95995035648,"available":95926927360,"internally_reserved":0,"allocated":36171776,"data_stored":39016879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43164,"kb_used_data":16128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93701988,"statfs":{"total":95995035648,"available":95950835712,"internally_reserved":0,"allocated":16515072,"data_stored":17875006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41652,"kb_used_data":14620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93703500,"statfs":{"total":95995035648,"available":95952384000,"internally_reserved":0,"allocated":14970880,"data_stored":14253646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117076,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66172,"kb_used_data":34976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93678980,"statfs":{"total":95995035648,"available":95927275520,"internally_reserved":0,"allocated":35815424,"data_stored":46672907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313306,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46628,"kb_used_data":15440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93698524,"statfs":{"total":95995035648,"available":95947288576,"internally_reserved":0,"allocated":15810560,"data_stored":18107228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59704,"kb_used_data":28508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93685448,"statfs":{"total":95995035648,"available":95933898752,"internally_reserved":0,"allocated":29192192,"data_stored":34482060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53956,"kb_used_data":18604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93691196,"statfs":{"total":95995035648,"available":95939784704,"internally_reserved":0,"allocated":19050496,"data_stored":23578125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16265216,"data_stored":21895561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26419200,"data_stored":32807957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12869632,"data_stored":16257353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33067008,"data_stored":45015734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12247040,"data_stored":12613589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13586432,"data_stored":16033701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":33423360,"data_stored":37359706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":62033920,"data_stored":74757564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":41984000,"data_stored":52232533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":66322432,"data_stored":82852745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":33406976,"data_stored":41595568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":60653568,"data_stored":85981931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:12:53.128 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:12:53.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 46 2024-09-16T05:12:53.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:52 smithi204 ceph-mon[30933]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:12:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:52 smithi204 ceph-mon[30933]: osdmap e241: 12 total, 12 up, 11 in 2024-09-16T05:12:53.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:53.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:53.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:53.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.431 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:53.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:53.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:53.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:53.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:53.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:53.705 INFO:tasks.rados.rados.2.smithi204.stdout:660: done (3 left) 2024-09-16T05:12:53.705 INFO:tasks.rados.rados.2.smithi204.stdout:661: snap_remove snap 58 2024-09-16T05:12:53.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:53.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:12:53.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:53.851 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:53.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:53.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:53.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:53.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:53 smithi032 ceph-mon[27388]: pgmap v513: 47 pgs: 47 active+clean; 286 MiB data, 728 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:12:53.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:53 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:12:53.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:53 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:12:53.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:53 smithi032 ceph-mon[27388]: osdmap e242: 12 total, 12 up, 11 in 2024-09-16T05:12:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:53 smithi196 ceph-mon[30802]: pgmap v513: 47 pgs: 47 active+clean; 286 MiB data, 728 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:12:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:53 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:12:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:53 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:12:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:53 smithi196 ceph-mon[30802]: osdmap e242: 12 total, 12 up, 11 in 2024-09-16T05:12:54.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:54.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:54.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:54.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:53 smithi204 ceph-mon[30933]: pgmap v513: 47 pgs: 47 active+clean; 286 MiB data, 728 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:12:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:53 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:12:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:53 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:12:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:53 smithi204 ceph-mon[30933]: osdmap e242: 12 total, 12 up, 11 in 2024-09-16T05:12:54.193 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:54.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:54.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:54.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:54.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:54.361 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:54.402 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:54.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:54.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:54.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:54.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:54.569 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:54.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:54.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:54.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:54.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:54.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:54.699 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:12:54.704 INFO:tasks.rados.rados.2.smithi204.stdout:661: done (3 left) 2024-09-16T05:12:54.704 INFO:tasks.rados.rados.2.smithi204.stdout:662: read oid 2 snap 69 2024-09-16T05:12:54.705 INFO:tasks.rados.rados.2.smithi204.stdout:662: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:12:54.705 INFO:tasks.rados.rados.2.smithi204.stdout:663: copy_from oid 26 from oid 45 current snap is 73 2024-09-16T05:12:54.705 INFO:tasks.rados.rados.2.smithi204.stdout:657: finishing copy_from to smithi20457400-27 2024-09-16T05:12:54.705 INFO:tasks.rados.rados.2.smithi204.stdout:657: got expected ENOENT (src dne) 2024-09-16T05:12:54.705 INFO:tasks.rados.rados.2.smithi204.stdout:657: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:12:54.705 INFO:tasks.rados.rados.2.smithi204.stdout:656: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:12:54.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:12:54.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:54.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:54.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:659: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:12:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:659: finishing copy_from to smithi20457400-19 2024-09-16T05:12:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 280 (ObjNum 239 snap 67 seq_num 239) dirty exists 2024-09-16T05:12:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:662: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:12:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3387739886' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:54 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3387739886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:54 smithi032 ceph-mon[27388]: osdmap e243: 12 total, 12 up, 11 in 2024-09-16T05:12:54.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:54.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:54.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3387739886' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:54 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3387739886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:54 smithi196 ceph-mon[30802]: osdmap e243: 12 total, 12 up, 11 in 2024-09-16T05:12:54.993 INFO:tasks.rados.rados.2.smithi204.stdout:663: finishing copy_from racing read to smithi20457400-26 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:663: finishing copy_from to smithi20457400-26 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 157 (ObjNum 234 snap 65 seq_num 234) dirty exists 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:656: done (4 left) 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:657: done (3 left) 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:659: done (2 left) 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:662: done (1 left) 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:663: done (0 left) 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:664: write oid 15 current snap is 73 2024-09-16T05:12:54.994 INFO:tasks.rados.rados.2.smithi204.stdout:664: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-16T05:12:55.000 INFO:tasks.rados.rados.2.smithi204.stdout:664: writing smithi20457400-15 from 736206 to 1190097 tid 1 2024-09-16T05:12:55.006 INFO:tasks.rados.rados.2.smithi204.stdout:664: writing smithi20457400-15 from 1881413 to 2336206 tid 2 2024-09-16T05:12:55.006 INFO:tasks.rados.rados.2.smithi204.stdout:665: read oid 48 snap -1 2024-09-16T05:12:55.007 INFO:tasks.rados.rados.2.smithi204.stdout:665: expect (ObjNum 241 snap 67 seq_num 241) 2024-09-16T05:12:55.007 INFO:tasks.rados.rados.2.smithi204.stdout:666: read oid 45 snap -1 2024-09-16T05:12:55.007 INFO:tasks.rados.rados.2.smithi204.stdout:666: expect (ObjNum 234 snap 65 seq_num 234) 2024-09-16T05:12:55.007 INFO:tasks.rados.rados.2.smithi204.stdout:667: snap_remove snap 49 2024-09-16T05:12:55.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:55.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3387739886' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:12:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:54 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:12:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3387739886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:12:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:54 smithi204 ceph-mon[30933]: osdmap e243: 12 total, 12 up, 11 in 2024-09-16T05:12:55.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:55.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:55.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:55.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:55.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:55.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:55.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:55.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:55.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:55.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:55.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:55.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:55.670 INFO:tasks.rados.rados.2.smithi204.stdout:667: done (3 left) 2024-09-16T05:12:55.670 INFO:tasks.rados.rados.2.smithi204.stdout:668: read oid 5 snap -1 2024-09-16T05:12:55.670 INFO:tasks.rados.rados.2.smithi204.stdout:668: expect deleted 2024-09-16T05:12:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:669: delete oid 16 current snap is 73 2024-09-16T05:12:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:664: finishing write tid 1 to smithi20457400-15 2024-09-16T05:12:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:664: finishing write tid 2 to smithi20457400-15 2024-09-16T05:12:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:664: finishing write tid 3 to smithi20457400-15 2024-09-16T05:12:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:664: finishing write tid 4 to smithi20457400-15 2024-09-16T05:12:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 42 (ObjNum 258 snap 73 seq_num 258) dirty exists 2024-09-16T05:12:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:664: left oid 15 (ObjNum 258 snap 73 seq_num 258) 2024-09-16T05:12:55.671 INFO:tasks.rados.rados.2.smithi204.stdout:666: expect (ObjNum 234 snap 65 seq_num 234) 2024-09-16T05:12:55.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:55.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:55.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:55.823 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:55.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:55.852 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:55.864 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:12:55.864 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:12:55.911 INFO:tasks.rados.rados.2.smithi204.stdout:665: expect (ObjNum 241 snap 67 seq_num 241) 2024-09-16T05:12:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:55 smithi032 ceph-mon[27388]: pgmap v516: 47 pgs: 47 active+clean; 287 MiB data, 730 MiB used, 983 GiB / 983 GiB avail; 1.1 MiB/s wr, 2 op/s 2024-09-16T05:12:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:55 smithi032 ceph-mon[27388]: osdmap e244: 12 total, 12 up, 11 in 2024-09-16T05:12:55.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:55.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:55.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:55 smithi196 ceph-mon[30802]: pgmap v516: 47 pgs: 47 active+clean; 287 MiB data, 730 MiB used, 983 GiB / 983 GiB avail; 1.1 MiB/s wr, 2 op/s 2024-09-16T05:12:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:55 smithi196 ceph-mon[30802]: osdmap e244: 12 total, 12 up, 11 in 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:664: done (4 left) 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:665: done (3 left) 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:666: done (2 left) 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:669: done (1 left) 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:670: read oid 19 snap 55 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:670: expect deleted 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:671: copy_from oid 9 from oid 25 current snap is 73 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:672: write oid 16 current snap is 73 2024-09-16T05:12:55.997 INFO:tasks.rados.rados.2.smithi204.stdout:672: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-16T05:12:56.004 INFO:tasks.rados.rados.2.smithi204.stdout:672: writing smithi20457400-16 from 769820 to 1490562 tid 1 2024-09-16T05:12:56.009 INFO:tasks.rados.rados.2.smithi204.stdout:672: writing smithi20457400-16 from 2114730 to 2369820 tid 2 2024-09-16T05:12:56.009 INFO:tasks.rados.rados.2.smithi204.stdout:673: write oid 26 current snap is 73 2024-09-16T05:12:56.009 INFO:tasks.rados.rados.2.smithi204.stdout:673: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-16T05:12:56.018 INFO:tasks.rados.rados.2.smithi204.stdout:673: writing smithi20457400-26 from 786627 to 1582618 tid 1 2024-09-16T05:12:56.022 INFO:tasks.rados.rados.2.smithi204.stdout:673: writing smithi20457400-26 from 2173212 to 2386627 tid 2 2024-09-16T05:12:56.022 INFO:tasks.rados.rados.2.smithi204.stdout:674: delete oid 14 current snap is 73 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:671: finishing copy_from to smithi20457400-9 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:671: got expected ENOENT (src dne) 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:671: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:672: finishing write tid 1 to smithi20457400-16 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:672: finishing write tid 2 to smithi20457400-16 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:672: finishing write tid 3 to smithi20457400-16 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:672: finishing write tid 4 to smithi20457400-16 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 351 (ObjNum 260 snap 73 seq_num 260) dirty exists 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:672: left oid 16 (ObjNum 260 snap 73 seq_num 260) 2024-09-16T05:12:56.023 INFO:tasks.rados.rados.2.smithi204.stdout:673: finishing write tid 1 to smithi20457400-26 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:673: finishing write tid 2 to smithi20457400-26 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:673: finishing write tid 3 to smithi20457400-26 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:670: done (5 left) 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:671: done (4 left) 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:672: done (3 left) 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:674: done (2 left) 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:675: copy_from oid 21 from oid 18 current snap is 73 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:676: read oid 45 snap -1 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:676: expect (ObjNum 234 snap 65 seq_num 234) 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:677: copy_from oid 3 from oid 36 current snap is 73 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:678: rollback oid 28 current snap is 73 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 28 to 37 2024-09-16T05:12:56.027 INFO:tasks.rados.rados.2.smithi204.stdout:679: write oid 16 current snap is 73 2024-09-16T05:12:56.028 INFO:tasks.rados.rados.2.smithi204.stdout:679: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-16T05:12:56.033 INFO:tasks.rados.rados.2.smithi204.stdout:679: writing smithi20457400-16 from 420241 to 966730 tid 1 2024-09-16T05:12:56.039 INFO:tasks.rados.rados.2.smithi204.stdout:679: writing smithi20457400-16 from 1373823 to 1918237 tid 2 2024-09-16T05:12:56.039 INFO:tasks.rados.rados.2.smithi204.stdout:680: delete oid 49 current snap is 73 2024-09-16T05:12:56.039 INFO:tasks.rados.rados.2.smithi204.stdout:673: finishing write tid 4 to smithi20457400-26 2024-09-16T05:12:56.039 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 351 (ObjNum 261 snap 73 seq_num 261) dirty exists 2024-09-16T05:12:56.040 INFO:tasks.rados.rados.2.smithi204.stdout:673: left oid 26 (ObjNum 261 snap 73 seq_num 261) 2024-09-16T05:12:56.040 INFO:tasks.rados.rados.2.smithi204.stdout:678: finishing rollback tid 1 to smithi20457400-28 2024-09-16T05:12:56.040 INFO:tasks.rados.rados.2.smithi204.stdout:678: finishing rollback tid 2 to smithi20457400-28 2024-09-16T05:12:56.040 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 288 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:12:56.040 INFO:tasks.rados.rados.2.smithi204.stdout:677: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:12:56.040 INFO:tasks.rados.rados.2.smithi204.stdout:679: finishing write tid 1 to smithi20457400-16 2024-09-16T05:12:56.040 INFO:tasks.rados.rados.2.smithi204.stdout:677: finishing copy_from to smithi20457400-3 2024-09-16T05:12:56.040 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 285 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:12:56.041 INFO:tasks.rados.rados.2.smithi204.stdout:679: finishing write tid 2 to smithi20457400-16 2024-09-16T05:12:56.041 INFO:tasks.rados.rados.2.smithi204.stdout:679: finishing write tid 3 to smithi20457400-16 2024-09-16T05:12:56.041 INFO:tasks.rados.rados.2.smithi204.stdout:679: finishing write tid 4 to smithi20457400-16 2024-09-16T05:12:56.041 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 354 (ObjNum 263 snap 73 seq_num 263) dirty exists 2024-09-16T05:12:56.041 INFO:tasks.rados.rados.2.smithi204.stdout:679: left oid 16 (ObjNum 263 snap 73 seq_num 263) 2024-09-16T05:12:56.043 INFO:tasks.rados.rados.2.smithi204.stdout:675: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:12:56.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:56.048 INFO:tasks.rados.rados.2.smithi204.stdout:676: expect (ObjNum 234 snap 65 seq_num 234) 2024-09-16T05:12:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:56.071 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:56.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:55 smithi204 ceph-mon[30933]: pgmap v516: 47 pgs: 47 active+clean; 287 MiB data, 730 MiB used, 983 GiB / 983 GiB avail; 1.1 MiB/s wr, 2 op/s 2024-09-16T05:12:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:55 smithi204 ceph-mon[30933]: osdmap e244: 12 total, 12 up, 11 in 2024-09-16T05:12:56.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:56.193 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:56.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:56.290 INFO:tasks.rados.rados.2.smithi204.stdout:675: finishing copy_from to smithi20457400-21 2024-09-16T05:12:56.290 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 67 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:12:56.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:56.317 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:56.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:56.388 INFO:tasks.rados.rados.2.smithi204.stdout:673: done (7 left) 2024-09-16T05:12:56.388 INFO:tasks.rados.rados.2.smithi204.stdout:675: done (6 left) 2024-09-16T05:12:56.388 INFO:tasks.rados.rados.2.smithi204.stdout:676: done (5 left) 2024-09-16T05:12:56.388 INFO:tasks.rados.rados.2.smithi204.stdout:677: done (4 left) 2024-09-16T05:12:56.388 INFO:tasks.rados.rados.2.smithi204.stdout:678: done (3 left) 2024-09-16T05:12:56.388 INFO:tasks.rados.rados.2.smithi204.stdout:679: done (2 left) 2024-09-16T05:12:56.388 INFO:tasks.rados.rados.2.smithi204.stdout:680: done (1 left) 2024-09-16T05:12:56.388 INFO:tasks.rados.rados.2.smithi204.stdout:681: copy_from oid 40 from oid 24 current snap is 73 2024-09-16T05:12:56.389 INFO:tasks.rados.rados.2.smithi204.stdout:682: rollback oid 18 current snap is 73 2024-09-16T05:12:56.389 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 18 to 72 2024-09-16T05:12:56.389 INFO:tasks.rados.rados.2.smithi204.stdout:683: rollback oid 41 current snap is 73 2024-09-16T05:12:56.389 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 41 to 37 2024-09-16T05:12:56.389 INFO:tasks.rados.rados.2.smithi204.stdout:684: snap_remove snap 63 2024-09-16T05:12:56.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:56.441 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:56.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:56.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:12:56.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:56.565 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:56.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:56.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:56.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:56.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:56.730 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pg_num to 46 2024-09-16T05:12:56.733 INFO:tasks.rados.rados.2.smithi204.stdout:683: finishing rollback tid 0 to smithi20457400-41 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:683: finishing rollback tid 1 to smithi20457400-41 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:683: finishing rollback tid 2 to smithi20457400-41 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 301 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:682: finishing rollback tid 0 to smithi20457400-18 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:682: finishing rollback tid 1 to smithi20457400-18 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:682: finishing rollback tid 2 to smithi20457400-18 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 210 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:681: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:681: finishing copy_from to smithi20457400-40 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 251 (ObjNum 252 snap 70 seq_num 252) dirty exists 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:668: done (4 left) 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:681: done (3 left) 2024-09-16T05:12:56.734 INFO:tasks.rados.rados.2.smithi204.stdout:682: done (2 left) 2024-09-16T05:12:56.735 INFO:tasks.rados.rados.2.smithi204.stdout:683: done (1 left) 2024-09-16T05:12:56.735 INFO:tasks.rados.rados.2.smithi204.stdout:684: done (0 left) 2024-09-16T05:12:56.735 INFO:tasks.rados.rados.2.smithi204.stdout:685: write (excl) oid 27 current snap is 73 2024-09-16T05:12:56.735 INFO:tasks.rados.rados.2.smithi204.stdout:685: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-16T05:12:56.740 INFO:tasks.rados.rados.2.smithi204.stdout:685: writing smithi20457400-27 from 453855 to 1150842 tid 1 2024-09-16T05:12:56.747 INFO:tasks.rados.rados.2.smithi204.stdout:685: writing smithi20457400-27 from 1890787 to 2438870 tid 2 2024-09-16T05:12:56.747 INFO:tasks.rados.rados.2.smithi204.stdout:686: copy_from oid 13 from oid 19 current snap is 73 2024-09-16T05:12:56.748 INFO:tasks.rados.rados.2.smithi204.stdout:685: finishing write tid 1 to smithi20457400-27 2024-09-16T05:12:56.748 INFO:tasks.rados.rados.2.smithi204.stdout:687: snap_create 2024-09-16T05:12:56.752 INFO:tasks.rados.rados.2.smithi204.stdout:686: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:12:56.755 INFO:tasks.rados.rados.2.smithi204.stdout:685: finishing write tid 2 to smithi20457400-27 2024-09-16T05:12:56.755 INFO:tasks.rados.rados.2.smithi204.stdout:685: finishing write tid 3 to smithi20457400-27 2024-09-16T05:12:56.756 INFO:tasks.rados.rados.2.smithi204.stdout:686: finishing copy_from to smithi20457400-13 2024-09-16T05:12:56.756 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 280 (ObjNum 239 snap 67 seq_num 239) dirty exists 2024-09-16T05:12:56.756 INFO:tasks.rados.rados.2.smithi204.stdout:685: finishing write tid 4 to smithi20457400-27 2024-09-16T05:12:56.756 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 105 (ObjNum 265 snap 73 seq_num 265) dirty exists 2024-09-16T05:12:56.756 INFO:tasks.rados.rados.2.smithi204.stdout:685: left oid 27 (ObjNum 265 snap 73 seq_num 265) 2024-09-16T05:12:56.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:56.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:56.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:56.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:56.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:56.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:56.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3499532532' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-16T05:12:57.000 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3499532532' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-16T05:12:57.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:57.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:57.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:57.109 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:12:57.110 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:12:57.133 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:12:57.133 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:12:57.143 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:57.154 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:12:57.155 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:12:57.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3499532532' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-16T05:12:57.176 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:12:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:12:57.191 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:57.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:57.198 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:12:57.198 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:12:57.220 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:12:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:12:57.242 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:12:57.242 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:12:57.264 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:12:57.264 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:12:57.285 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:12:57.286 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:12:57.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:57.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:12:57.308 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:12:57.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:57.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:57.329 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:12:57.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:12:57.351 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:12:57.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:12:57.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:12:57.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:57.439 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:57.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:57.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:12:57.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:57.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:12:57.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:57.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:57.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:685: done (2 left) 2024-09-16T05:12:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:686: done (1 left) 2024-09-16T05:12:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:687: done (0 left) 2024-09-16T05:12:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:688: rollback oid 4 current snap is 74 2024-09-16T05:12:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 73 2024-09-16T05:12:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:689: snap_create 2024-09-16T05:12:57.742 INFO:tasks.rados.rados.2.smithi204.stdout:688: finishing rollback tid 0 to smithi20457400-4 2024-09-16T05:12:57.742 INFO:tasks.rados.rados.2.smithi204.stdout:688: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:12:57.742 INFO:tasks.rados.rados.2.smithi204.stdout:688: finishing rollback tid 2 to smithi20457400-4 2024-09-16T05:12:57.743 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 254 (ObjNum 181 snap 46 seq_num 181) dirty exists 2024-09-16T05:12:57.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:57.820 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:57.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:57.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:57.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:57.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:57 smithi196 ceph-mon[30802]: pgmap v519: 47 pgs: 47 active+clean; 287 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 9.4 MiB/s rd, 541 KiB/s wr, 9 op/s 2024-09-16T05:12:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3499532532' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-16T05:12:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:57 smithi196 ceph-mon[30802]: osdmap e245: 12 total, 12 up, 11 in 2024-09-16T05:12:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3506596708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:58.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:57 smithi032 ceph-mon[27388]: pgmap v519: 47 pgs: 47 active+clean; 287 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 9.4 MiB/s rd, 541 KiB/s wr, 9 op/s 2024-09-16T05:12:58.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3499532532' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-16T05:12:58.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:57 smithi032 ceph-mon[27388]: osdmap e245: 12 total, 12 up, 11 in 2024-09-16T05:12:58.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3506596708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:58.047 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:58.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:58.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:57 smithi204 ceph-mon[30933]: pgmap v519: 47 pgs: 47 active+clean; 287 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 9.4 MiB/s rd, 541 KiB/s wr, 9 op/s 2024-09-16T05:12:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3499532532' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-16T05:12:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:57 smithi204 ceph-mon[30933]: osdmap e245: 12 total, 12 up, 11 in 2024-09-16T05:12:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3506596708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:12:58.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:12:58.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:58.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:12:58.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:58.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:58.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:58.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:58.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:58.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:58.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:12:58.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:58.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:12:58.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:12:58.709 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:58.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:12:58.746 INFO:tasks.rados.rados.2.smithi204.stdout:688: done (1 left) 2024-09-16T05:12:58.746 INFO:tasks.rados.rados.2.smithi204.stdout:689: done (0 left) 2024-09-16T05:12:58.746 INFO:tasks.rados.rados.2.smithi204.stdout:690: read oid 30 snap 55 2024-09-16T05:12:58.747 INFO:tasks.rados.rados.2.smithi204.stdout:690: expect (ObjNum 177 snap 45 seq_num 177) 2024-09-16T05:12:58.747 INFO:tasks.rados.rados.2.smithi204.stdout:691: write (excl) oid 29 current snap is 75 2024-09-16T05:12:58.747 INFO:tasks.rados.rados.2.smithi204.stdout:691: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-16T05:12:58.753 INFO:tasks.rados.rados.2.smithi204.stdout:691: writing smithi20457400-29 from 470662 to 1242898 tid 1 2024-09-16T05:12:58.759 INFO:tasks.rados.rados.2.smithi204.stdout:691: writing smithi20457400-29 from 1832916 to 2241010 tid 2 2024-09-16T05:12:58.760 INFO:tasks.rados.rados.2.smithi204.stdout:692: write oid 4 current snap is 75 2024-09-16T05:12:58.760 INFO:tasks.rados.rados.2.smithi204.stdout:692: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-16T05:12:58.765 INFO:tasks.rados.rados.2.smithi204.stdout:692: writing smithi20457400-4 from 487469 to 1051307 tid 1 2024-09-16T05:12:58.773 INFO:tasks.rados.rados.2.smithi204.stdout:692: writing smithi20457400-4 from 1607751 to 2159503 tid 2 2024-09-16T05:12:58.774 INFO:tasks.rados.rados.2.smithi204.stdout:693: read oid 20 snap 69 2024-09-16T05:12:58.774 INFO:tasks.rados.rados.2.smithi204.stdout:693: expect (ObjNum 179 snap 46 seq_num 179) 2024-09-16T05:12:58.774 INFO:tasks.rados.rados.2.smithi204.stdout:694: read oid 21 snap -1 2024-09-16T05:12:58.774 INFO:tasks.rados.rados.2.smithi204.stdout:694: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:12:58.774 INFO:tasks.rados.rados.2.smithi204.stdout:695: write oid 43 current snap is 75 2024-09-16T05:12:58.774 INFO:tasks.rados.rados.2.smithi204.stdout:695: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-16T05:12:58.777 INFO:tasks.rados.rados.2.smithi204.stdout:695: writing smithi20457400-43 from 504276 to 1143363 tid 1 2024-09-16T05:12:58.784 INFO:tasks.rados.rados.2.smithi204.stdout:695: writing smithi20457400-43 from 1666233 to 2361643 tid 2 2024-09-16T05:12:58.784 INFO:tasks.rados.rados.2.smithi204.stdout:696: write (excl) oid 27 current snap is 75 2024-09-16T05:12:58.785 INFO:tasks.rados.rados.2.smithi204.stdout:691: finishing write tid 1 to smithi20457400-29 2024-09-16T05:12:58.785 INFO:tasks.rados.rados.2.smithi204.stdout:690: expect (ObjNum 177 snap 45 seq_num 177) 2024-09-16T05:12:58.795 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:12:58.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:12:58.795 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:12:58.795 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:12:58.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:58.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:58.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:58.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:12:58.938 INFO:tasks.rados.rados.2.smithi204.stdout:691: finishing write tid 2 to smithi20457400-29 2024-09-16T05:12:58.938 INFO:tasks.rados.rados.2.smithi204.stdout:691: finishing write tid 3 to smithi20457400-29 2024-09-16T05:12:58.938 INFO:tasks.rados.rados.2.smithi204.stdout:691: finishing write tid 4 to smithi20457400-29 2024-09-16T05:12:58.938 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 347 (ObjNum 266 snap 75 seq_num 266) dirty exists 2024-09-16T05:12:58.939 INFO:tasks.rados.rados.2.smithi204.stdout:691: left oid 29 (ObjNum 266 snap 75 seq_num 266) 2024-09-16T05:12:58.939 INFO:tasks.rados.rados.2.smithi204.stdout:692: finishing write tid 1 to smithi20457400-4 2024-09-16T05:12:58.939 INFO:tasks.rados.rados.2.smithi204.stdout:692: finishing write tid 2 to smithi20457400-4 2024-09-16T05:12:58.939 INFO:tasks.rados.rados.2.smithi204.stdout:692: finishing write tid 3 to smithi20457400-4 2024-09-16T05:12:58.939 INFO:tasks.rados.rados.2.smithi204.stdout:696: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-16T05:12:58.944 INFO:tasks.rados.rados.2.smithi204.stdout:696: writing smithi20457400-27 from 521083 to 1235419 tid 1 2024-09-16T05:12:58.950 INFO:tasks.rados.rados.2.smithi204.stdout:696: writing smithi20457400-27 from 1724715 to 2280136 tid 2 2024-09-16T05:12:58.951 INFO:tasks.rados.rados.2.smithi204.stdout:690: done (6 left) 2024-09-16T05:12:58.951 INFO:tasks.rados.rados.2.smithi204.stdout:691: done (5 left) 2024-09-16T05:12:58.951 INFO:tasks.rados.rados.2.smithi204.stdout:697: copy_from oid 24 from oid 15 current snap is 75 2024-09-16T05:12:58.951 INFO:tasks.rados.rados.2.smithi204.stdout:692: finishing write tid 4 to smithi20457400-4 2024-09-16T05:12:58.951 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 257 (ObjNum 267 snap 75 seq_num 267) dirty exists 2024-09-16T05:12:58.951 INFO:tasks.rados.rados.2.smithi204.stdout:692: left oid 4 (ObjNum 267 snap 75 seq_num 267) 2024-09-16T05:12:58.951 INFO:tasks.rados.rados.2.smithi204.stdout:694: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:12:58.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:58.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:12:59.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:12:59.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:59.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:12:59.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:58 smithi032 ceph-mon[27388]: osdmap e246: 12 total, 12 up, 11 in 2024-09-16T05:12:59.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-16T05:12:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:58 smithi204 ceph-mon[30933]: osdmap e246: 12 total, 12 up, 11 in 2024-09-16T05:12:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-16T05:12:59.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:12:59.196 INFO:tasks.rados.rados.2.smithi204.stdout:695: finishing write tid 1 to smithi20457400-43 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:695: finishing write tid 2 to smithi20457400-43 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:695: finishing write tid 3 to smithi20457400-43 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:695: finishing write tid 4 to smithi20457400-43 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 216 (ObjNum 268 snap 75 seq_num 268) dirty exists 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:695: left oid 43 (ObjNum 268 snap 75 seq_num 268) 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:697: finishing copy_from racing read to smithi20457400-24 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:697: finishing copy_from to smithi20457400-24 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 42 (ObjNum 258 snap 73 seq_num 258) dirty exists 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:692: done (5 left) 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:694: done (4 left) 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:695: done (3 left) 2024-09-16T05:12:59.197 INFO:tasks.rados.rados.2.smithi204.stdout:697: done (2 left) 2024-09-16T05:12:59.198 INFO:tasks.rados.rados.2.smithi204.stdout:698: delete oid 43 current snap is 75 2024-09-16T05:12:59.198 INFO:tasks.rados.rados.2.smithi204.stdout:698: done (2 left) 2024-09-16T05:12:59.198 INFO:tasks.rados.rados.2.smithi204.stdout:699: read oid 2 snap 37 2024-09-16T05:12:59.198 INFO:tasks.rados.rados.2.smithi204.stdout:699: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T05:12:59.198 INFO:tasks.rados.rados.2.smithi204.stdout:700: delete oid 1 current snap is 75 2024-09-16T05:12:59.203 INFO:tasks.rados.rados.2.smithi204.stdout:699: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T05:12:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:59.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:12:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:58 smithi196 ceph-mon[30802]: osdmap e246: 12 total, 12 up, 11 in 2024-09-16T05:12:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-16T05:12:59.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:12:59.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:12:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:59.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:12:59.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:12:59.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:59.446 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:12:59.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:59.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:59.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:12:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:59.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:59.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:12:59.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:12:59.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:12:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:59.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:12:59.756 INFO:tasks.rados.rados.2.smithi204.stdout:696: finishing write tid 1 to smithi20457400-27 2024-09-16T05:12:59.756 INFO:tasks.rados.rados.2.smithi204.stdout:696: finishing write tid 2 to smithi20457400-27 2024-09-16T05:12:59.756 INFO:tasks.rados.rados.2.smithi204.stdout:696: finishing write tid 3 to smithi20457400-27 2024-09-16T05:12:59.757 INFO:tasks.rados.rados.2.smithi204.stdout:696: finishing write tid 4 to smithi20457400-27 2024-09-16T05:12:59.757 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 109 (ObjNum 269 snap 75 seq_num 269) dirty exists 2024-09-16T05:12:59.757 INFO:tasks.rados.rados.2.smithi204.stdout:696: left oid 27 (ObjNum 269 snap 75 seq_num 269) 2024-09-16T05:12:59.757 INFO:tasks.rados.rados.2.smithi204.stdout:696: done (3 left) 2024-09-16T05:12:59.757 INFO:tasks.rados.rados.2.smithi204.stdout:699: done (2 left) 2024-09-16T05:12:59.757 INFO:tasks.rados.rados.2.smithi204.stdout:700: done (1 left) 2024-09-16T05:12:59.757 INFO:tasks.rados.rados.2.smithi204.stdout:701: write oid 35 current snap is 75 2024-09-16T05:12:59.757 INFO:tasks.rados.rados.2.smithi204.stdout:701: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-16T05:12:59.762 INFO:tasks.rados.rados.2.smithi204.stdout:701: writing smithi20457400-35 from 571504 to 1111587 tid 1 2024-09-16T05:12:59.770 INFO:tasks.rados.rados.2.smithi204.stdout:701: writing smithi20457400-35 from 1783808 to 2486556 tid 2 2024-09-16T05:12:59.770 INFO:tasks.rados.rados.2.smithi204.stdout:702: read oid 9 snap -1 2024-09-16T05:12:59.770 INFO:tasks.rados.rados.2.smithi204.stdout:702: expect deleted 2024-09-16T05:12:59.770 INFO:tasks.rados.rados.2.smithi204.stdout:703: delete oid 2 current snap is 75 2024-09-16T05:12:59.770 INFO:tasks.rados.rados.2.smithi204.stdout:701: finishing write tid 1 to smithi20457400-35 2024-09-16T05:12:59.772 INFO:tasks.rados.rados.2.smithi204.stdout:702: done (3 left) 2024-09-16T05:12:59.772 INFO:tasks.rados.rados.2.smithi204.stdout:703: done (2 left) 2024-09-16T05:12:59.772 INFO:tasks.rados.rados.2.smithi204.stdout:704: write (excl) oid 30 current snap is 75 2024-09-16T05:12:59.772 INFO:tasks.rados.rados.2.smithi204.stdout:704: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-16T05:12:59.777 INFO:tasks.rados.rados.2.smithi204.stdout:704: writing smithi20457400-30 from 605118 to 1012052 tid 1 2024-09-16T05:12:59.785 INFO:tasks.rados.rados.2.smithi204.stdout:704: writing smithi20457400-30 from 1617125 to 2323542 tid 2 2024-09-16T05:12:59.785 INFO:tasks.rados.rados.2.smithi204.stdout:705: rollback oid 23 current snap is 75 2024-09-16T05:12:59.785 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 23 to 53 2024-09-16T05:12:59.785 INFO:tasks.rados.rados.2.smithi204.stdout:701: finishing write tid 2 to smithi20457400-35 2024-09-16T05:12:59.785 INFO:tasks.rados.rados.2.smithi204.stdout:701: finishing write tid 3 to smithi20457400-35 2024-09-16T05:12:59.785 INFO:tasks.rados.rados.2.smithi204.stdout:701: finishing write tid 4 to smithi20457400-35 2024-09-16T05:12:59.785 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 328 (ObjNum 272 snap 75 seq_num 272) dirty exists 2024-09-16T05:12:59.785 INFO:tasks.rados.rados.2.smithi204.stdout:701: left oid 35 (ObjNum 272 snap 75 seq_num 272) 2024-09-16T05:12:59.786 INFO:tasks.rados.rados.2.smithi204.stdout:704: finishing write tid 1 to smithi20457400-30 2024-09-16T05:12:59.786 INFO:tasks.rados.rados.2.smithi204.stdout:701: done (3 left) 2024-09-16T05:12:59.786 INFO:tasks.rados.rados.2.smithi204.stdout:706: write oid 8 current snap is 75 2024-09-16T05:12:59.786 INFO:tasks.rados.rados.2.smithi204.stdout:706: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-16T05:12:59.791 INFO:tasks.rados.rados.2.smithi204.stdout:706: writing smithi20457400-8 from 621925 to 1104108 tid 1 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:706: writing smithi20457400-8 from 1559254 to 2125682 tid 2 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:707: delete oid 22 current snap is 75 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:705: finishing rollback tid 1 to smithi20457400-23 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:704: finishing write tid 2 to smithi20457400-30 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:704: finishing write tid 3 to smithi20457400-30 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:704: finishing write tid 4 to smithi20457400-30 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 363 (ObjNum 274 snap 75 seq_num 274) dirty exists 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:704: left oid 30 (ObjNum 274 snap 75 seq_num 274) 2024-09-16T05:12:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:706: finishing write tid 1 to smithi20457400-8 2024-09-16T05:12:59.799 INFO:tasks.rados.rados.2.smithi204.stdout:704: done (4 left) 2024-09-16T05:12:59.799 INFO:tasks.rados.rados.2.smithi204.stdout:705: done (3 left) 2024-09-16T05:12:59.799 INFO:tasks.rados.rados.2.smithi204.stdout:707: done (2 left) 2024-09-16T05:12:59.799 INFO:tasks.rados.rados.2.smithi204.stdout:708: read oid 5 snap -1 2024-09-16T05:12:59.799 INFO:tasks.rados.rados.2.smithi204.stdout:708: expect deleted 2024-09-16T05:12:59.799 INFO:tasks.rados.rados.2.smithi204.stdout:709: delete oid 31 current snap is 75 2024-09-16T05:12:59.799 INFO:tasks.rados.rados.2.smithi204.stdout:706: finishing write tid 2 to smithi20457400-8 2024-09-16T05:12:59.799 INFO:tasks.rados.rados.2.smithi204.stdout:706: finishing write tid 3 to smithi20457400-8 2024-09-16T05:12:59.800 INFO:tasks.rados.rados.2.smithi204.stdout:706: finishing write tid 4 to smithi20457400-8 2024-09-16T05:12:59.800 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 304 (ObjNum 275 snap 75 seq_num 275) dirty exists 2024-09-16T05:12:59.800 INFO:tasks.rados.rados.2.smithi204.stdout:706: left oid 8 (ObjNum 275 snap 75 seq_num 275) 2024-09-16T05:12:59.800 INFO:tasks.rados.rados.2.smithi204.stdout:706: done (3 left) 2024-09-16T05:12:59.800 INFO:tasks.rados.rados.2.smithi204.stdout:708: done (2 left) 2024-09-16T05:12:59.800 INFO:tasks.rados.rados.2.smithi204.stdout:709: done (1 left) 2024-09-16T05:12:59.800 INFO:tasks.rados.rados.2.smithi204.stdout:710: snap_remove snap 53 2024-09-16T05:12:59.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:12:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:59.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:59.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:12:59.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:12:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:12:59.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:12:59.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:12:59.974 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:00.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:00.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:13:00.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:59 smithi032 ceph-mon[27388]: pgmap v522: 47 pgs: 47 active+clean; 294 MiB data, 743 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-16T05:13:00.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:59 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-16T05:13:00.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:12:59 smithi032 ceph-mon[27388]: osdmap e247: 12 total, 12 up, 11 in 2024-09-16T05:13:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:00.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:59 smithi204 ceph-mon[30933]: pgmap v522: 47 pgs: 47 active+clean; 294 MiB data, 743 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-16T05:13:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:59 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-16T05:13:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:12:59 smithi204 ceph-mon[30933]: osdmap e247: 12 total, 12 up, 11 in 2024-09-16T05:13:00.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:00.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:00.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:00.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:59 smithi196 ceph-mon[30802]: pgmap v522: 47 pgs: 47 active+clean; 294 MiB data, 743 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-16T05:13:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:59 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-16T05:13:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:12:59 smithi196 ceph-mon[30802]: osdmap e247: 12 total, 12 up, 11 in 2024-09-16T05:13:00.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:00.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:00.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:00.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:00.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:00.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:00.575 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:13:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:13:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:13:00.575 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:13:00.575 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:13:00.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:00.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:00.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:00.667 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:13:00.671 INFO:tasks.rados.rados.2.smithi204.stdout:710: done (1 left) 2024-09-16T05:13:00.671 INFO:tasks.rados.rados.2.smithi204.stdout:711: delete oid 46 current snap is 75 2024-09-16T05:13:00.682 INFO:tasks.rados.rados.2.smithi204.stdout:711: done (1 left) 2024-09-16T05:13:00.682 INFO:tasks.rados.rados.2.smithi204.stdout:712: delete oid 10 current snap is 75 2024-09-16T05:13:00.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:00.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:00.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:00.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:00.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:00.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:00.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:00.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:00.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:00.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:01.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:00 smithi032 ceph-mon[27388]: osdmap e248: 12 total, 12 up, 11 in 2024-09-16T05:13:01.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:00 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-16T05:13:01.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/767618939' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:01.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:00 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:01.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:00 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-16T05:13:01.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/767618939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:01.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:00 smithi032 ceph-mon[27388]: osdmap e249: 12 total, 12 up, 11 in 2024-09-16T05:13:01.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:01.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:01.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:00 smithi204 ceph-mon[30933]: osdmap e248: 12 total, 12 up, 11 in 2024-09-16T05:13:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-16T05:13:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/767618939' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:00 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:00 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-16T05:13:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/767618939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:00 smithi204 ceph-mon[30933]: osdmap e249: 12 total, 12 up, 11 in 2024-09-16T05:13:01.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:01.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:00 smithi196 ceph-mon[30802]: osdmap e248: 12 total, 12 up, 11 in 2024-09-16T05:13:01.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:00 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-16T05:13:01.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/767618939' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:01.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:00 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:01.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:00 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-16T05:13:01.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/767618939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:01.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:00 smithi196 ceph-mon[30802]: osdmap e249: 12 total, 12 up, 11 in 2024-09-16T05:13:01.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:01.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:01.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:01.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:01.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:01.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:01.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:01.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:01.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:01.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:01.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:01.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:01.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:13:01.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:01.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:01.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:01.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:01.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:01.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:01.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:01.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:02.006 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:02.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:02.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:01 smithi032 ceph-mon[27388]: pgmap v525: 57 pgs: 1 peering, 10 unknown, 46 active+clean; 295 MiB data, 743 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 12 op/s 2024-09-16T05:13:02.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:01 smithi032 ceph-mon[27388]: osdmap e250: 12 total, 12 up, 11 in 2024-09-16T05:13:02.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:02.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:02.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:01 smithi204 ceph-mon[30933]: pgmap v525: 57 pgs: 1 peering, 10 unknown, 46 active+clean; 295 MiB data, 743 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 12 op/s 2024-09-16T05:13:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:01 smithi204 ceph-mon[30933]: osdmap e250: 12 total, 12 up, 11 in 2024-09-16T05:13:02.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:02.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:01 smithi196 ceph-mon[30802]: pgmap v525: 57 pgs: 1 peering, 10 unknown, 46 active+clean; 295 MiB data, 743 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 12 op/s 2024-09-16T05:13:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:01 smithi196 ceph-mon[30802]: osdmap e250: 12 total, 12 up, 11 in 2024-09-16T05:13:02.260 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:02.261 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:02.262 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:02.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:02.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:02.284 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:02.284 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:02.306 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:02.306 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:02.326 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:02.349 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:02.349 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:02.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:02.370 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:02.371 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:02.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:02.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:02.392 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:02.393 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:02.414 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:02.414 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:02.435 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:02.435 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:02.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:02.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:02.479 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:02.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:02.503 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:02.503 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:02.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:02.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:02.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:02.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:02.687 INFO:tasks.rados.rados.2.smithi204.stdout:693: expect (ObjNum 179 snap 46 seq_num 179) 2024-09-16T05:13:02.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:02.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:02.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:02.788 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/150949570' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:02.788 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:02 smithi032 ceph-mon[27388]: osdmap e251: 12 total, 12 up, 11 in 2024-09-16T05:13:02.868 INFO:tasks.rados.rados.2.smithi204.stdout:693: done (1 left) 2024-09-16T05:13:02.868 INFO:tasks.rados.rados.2.smithi204.stdout:712: done (0 left) 2024-09-16T05:13:02.868 INFO:tasks.rados.rados.2.smithi204.stdout:713: write oid 18 current snap is 75 2024-09-16T05:13:02.868 INFO:tasks.rados.rados.2.smithi204.stdout:713: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-16T05:13:02.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:02.874 INFO:tasks.rados.rados.2.smithi204.stdout:713: writing smithi20457400-18 from 705960 to 1164388 tid 1 2024-09-16T05:13:02.881 INFO:tasks.rados.rados.2.smithi204.stdout:713: writing smithi20457400-18 from 1735311 to 2452735 tid 2 2024-09-16T05:13:02.881 INFO:tasks.rados.rados.2.smithi204.stdout:714: snap_create 2024-09-16T05:13:02.881 INFO:tasks.rados.rados.2.smithi204.stdout:713: finishing write tid 1 to smithi20457400-18 2024-09-16T05:13:02.884 INFO:tasks.rados.rados.2.smithi204.stdout:713: finishing write tid 2 to smithi20457400-18 2024-09-16T05:13:02.884 INFO:tasks.rados.rados.2.smithi204.stdout:713: finishing write tid 3 to smithi20457400-18 2024-09-16T05:13:02.884 INFO:tasks.rados.rados.2.smithi204.stdout:713: finishing write tid 4 to smithi20457400-18 2024-09-16T05:13:02.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 213 (ObjNum 280 snap 75 seq_num 280) dirty exists 2024-09-16T05:13:02.885 INFO:tasks.rados.rados.2.smithi204.stdout:713: left oid 18 (ObjNum 280 snap 75 seq_num 280) 2024-09-16T05:13:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:02.893 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:02.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:02.898 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:02.898 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:02.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:03.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:03.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:03.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:03.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:03.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:03.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/150949570' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:02 smithi204 ceph-mon[30933]: osdmap e251: 12 total, 12 up, 11 in 2024-09-16T05:13:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/150949570' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:02 smithi196 ceph-mon[30802]: osdmap e251: 12 total, 12 up, 11 in 2024-09-16T05:13:03.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:03.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:03.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:03.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:03.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:03.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:03.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:03.427 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:13:03.427 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:13:03.428 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T05:13:03.428 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:13:03.428 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:13:03.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-16T05:13:03.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:03.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:03.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:03.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:03.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:03.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:03.677 INFO:tasks.rados.rados.2.smithi204.stdout:713: done (1 left) 2024-09-16T05:13:03.677 INFO:tasks.rados.rados.2.smithi204.stdout:714: done (0 left) 2024-09-16T05:13:03.677 INFO:tasks.rados.rados.2.smithi204.stdout:715: write (excl) oid 41 current snap is 76 2024-09-16T05:13:03.677 INFO:tasks.rados.rados.2.smithi204.stdout:715: seq_num 281 ranges {722767=533677,1793793=461082} 2024-09-16T05:13:03.683 INFO:tasks.rados.rados.2.smithi204.stdout:715: writing smithi20457400-41 from 722767 to 1256444 tid 1 2024-09-16T05:13:03.689 INFO:tasks.rados.rados.2.smithi204.stdout:715: writing smithi20457400-41 from 1793793 to 2254875 tid 2 2024-09-16T05:13:03.689 INFO:tasks.rados.rados.2.smithi204.stdout:716: snap_remove snap 73 2024-09-16T05:13:03.751 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:03.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:03.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:03.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:13:03.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:03.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:03.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:13:03.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:03 smithi032 ceph-mon[27388]: pgmap v528: 57 pgs: 1 peering, 1 remapped+peering, 4 unknown, 51 active+clean; 298 MiB data, 764 MiB used, 983 GiB / 983 GiB avail; 9.3 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2024-09-16T05:13:03.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:03 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:03.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:03 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:03.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:03 smithi032 ceph-mon[27388]: osdmap e252: 12 total, 12 up, 11 in 2024-09-16T05:13:04.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:04.031 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:04.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:04.035 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:04.072 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:04.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:04.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:04.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:04.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:03 smithi204 ceph-mon[30933]: pgmap v528: 57 pgs: 1 peering, 1 remapped+peering, 4 unknown, 51 active+clean; 298 MiB data, 764 MiB used, 983 GiB / 983 GiB avail; 9.3 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2024-09-16T05:13:04.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:03 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:04.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:03 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:04.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:03 smithi204 ceph-mon[30933]: osdmap e252: 12 total, 12 up, 11 in 2024-09-16T05:13:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:03 smithi196 ceph-mon[30802]: pgmap v528: 57 pgs: 1 peering, 1 remapped+peering, 4 unknown, 51 active+clean; 298 MiB data, 764 MiB used, 983 GiB / 983 GiB avail; 9.3 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2024-09-16T05:13:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:03 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:03 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:03 smithi196 ceph-mon[30802]: osdmap e252: 12 total, 12 up, 11 in 2024-09-16T05:13:04.261 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:04.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:04.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:04.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:04.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:04.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:04.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:04.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:04.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:04.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:04.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:04.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:04.677 INFO:tasks.rados.rados.2.smithi204.stdout:716: done (1 left) 2024-09-16T05:13:04.678 INFO:tasks.rados.rados.2.smithi204.stdout:717: snap_remove snap 64 2024-09-16T05:13:04.705 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:13:04.705 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T05:13:04.705 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:13:04.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:04.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:04.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:04.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:04.918 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:04.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:05.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:05.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:05.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:05.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:05.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:05.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:13:05.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:05.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:05.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:05.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:05.452 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:05.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:05.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:05.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:05.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:05.679 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:13:05.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:05.682 INFO:tasks.rados.rados.2.smithi204.stdout:717: done (1 left) 2024-09-16T05:13:05.682 INFO:tasks.rados.rados.2.smithi204.stdout:718: snap_create 2024-09-16T05:13:05.683 INFO:tasks.rados.rados.2.smithi204.stdout:715: finishing write tid 1 to smithi20457400-41 2024-09-16T05:13:05.683 INFO:tasks.rados.rados.2.smithi204.stdout:715: finishing write tid 2 to smithi20457400-41 2024-09-16T05:13:05.683 INFO:tasks.rados.rados.2.smithi204.stdout:715: finishing write tid 3 to smithi20457400-41 2024-09-16T05:13:05.683 INFO:tasks.rados.rados.2.smithi204.stdout:715: finishing write tid 4 to smithi20457400-41 2024-09-16T05:13:05.683 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 308 (ObjNum 281 snap 76 seq_num 281) dirty exists 2024-09-16T05:13:05.683 INFO:tasks.rados.rados.2.smithi204.stdout:715: left oid 41 (ObjNum 281 snap 76 seq_num 281) 2024-09-16T05:13:05.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:05.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:05.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:05.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:05 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:05 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:05 smithi032 ceph-mon[27388]: pgmap v531: 57 pgs: 2 peering, 1 active+remapped, 1 remapped+peering, 1 unknown, 52 active+clean; 317 MiB data, 784 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 4.8 MiB/s wr, 26 op/s; 3.2 MiB/s, 4 objects/s recovering 2024-09-16T05:13:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:05 smithi032 ceph-mon[27388]: osdmap e253: 12 total, 12 up, 11 in 2024-09-16T05:13:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3992802247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:05.934 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:05.934 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:05.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:05.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:05.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:05.973 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:05 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:05.973 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:05 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:05.973 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:05 smithi204 ceph-mon[30933]: pgmap v531: 57 pgs: 2 peering, 1 active+remapped, 1 remapped+peering, 1 unknown, 52 active+clean; 317 MiB data, 784 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 4.8 MiB/s wr, 26 op/s; 3.2 MiB/s, 4 objects/s recovering 2024-09-16T05:13:05.973 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:05 smithi204 ceph-mon[30933]: osdmap e253: 12 total, 12 up, 11 in 2024-09-16T05:13:05.973 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3992802247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:05 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:05 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:05 smithi196 ceph-mon[30802]: pgmap v531: 57 pgs: 2 peering, 1 active+remapped, 1 remapped+peering, 1 unknown, 52 active+clean; 317 MiB data, 784 MiB used, 983 GiB / 983 GiB avail; 19 MiB/s rd, 4.8 MiB/s wr, 26 op/s; 3.2 MiB/s, 4 objects/s recovering 2024-09-16T05:13:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:05 smithi196 ceph-mon[30802]: osdmap e253: 12 total, 12 up, 11 in 2024-09-16T05:13:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3992802247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:06.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:06.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:06.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:06.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:06.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:06.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:06.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:06.338 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:06.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:06.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:06.462 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:06.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:06.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:06.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:06.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:06.685 INFO:tasks.rados.rados.2.smithi204.stdout:715: done (1 left) 2024-09-16T05:13:06.685 INFO:tasks.rados.rados.2.smithi204.stdout:718: done (0 left) 2024-09-16T05:13:06.685 INFO:tasks.rados.rados.2.smithi204.stdout:719: read oid 27 snap 76 2024-09-16T05:13:06.686 INFO:tasks.rados.rados.2.smithi204.stdout:719: expect (ObjNum 269 snap 75 seq_num 269) 2024-09-16T05:13:06.686 INFO:tasks.rados.rados.2.smithi204.stdout:720: read oid 36 snap -1 2024-09-16T05:13:06.686 INFO:tasks.rados.rados.2.smithi204.stdout:720: expect (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:13:06.686 INFO:tasks.rados.rados.2.smithi204.stdout:721: snap_create 2024-09-16T05:13:06.689 INFO:tasks.rados.rados.2.smithi204.stdout:720: expect (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:13:06.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:06.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:06.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:06.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:06.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:06.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:06.886 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:06.886 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:06.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:06 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:06.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3992802247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:06.894 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:06 smithi032 ceph-mon[27388]: osdmap e254: 12 total, 12 up, 11 in 2024-09-16T05:13:06.895 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:06 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:13:06.895 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:06 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:13:06.922 INFO:tasks.rados.rados.2.smithi204.stdout:719: expect (ObjNum 269 snap 75 seq_num 269) 2024-09-16T05:13:06.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:06.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:06.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:06.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:06 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3992802247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:06 smithi196 ceph-mon[30802]: osdmap e254: 12 total, 12 up, 11 in 2024-09-16T05:13:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:06 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:13:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:06 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:13:07.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:07.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:07.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:06 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3992802247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:06 smithi204 ceph-mon[30933]: osdmap e254: 12 total, 12 up, 11 in 2024-09-16T05:13:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:06 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:13:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:06 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:13:07.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:07.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:07.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:07.211 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:07.211 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:07.232 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:07.233 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:07.253 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:07.273 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:07.273 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:07.295 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:07.295 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:07.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:07.316 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:07.317 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:07.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:07.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:07.337 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:07.338 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:07.359 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:07.359 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:07.382 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:07.382 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:07.403 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:07.403 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:07.424 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:07.425 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:07.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:07.446 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:07.446 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:07.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:07.458 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:07.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:07.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:07.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:07.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:07.686 INFO:tasks.rados.rados.2.smithi204.stdout:719: done (2 left) 2024-09-16T05:13:07.686 INFO:tasks.rados.rados.2.smithi204.stdout:720: done (1 left) 2024-09-16T05:13:07.686 INFO:tasks.rados.rados.2.smithi204.stdout:721: done (0 left) 2024-09-16T05:13:07.686 INFO:tasks.rados.rados.2.smithi204.stdout:722: snap_create 2024-09-16T05:13:07.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:07.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:07.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:07.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:07.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:07.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:07.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:07.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:13:07.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.29 scrub ok 2024-09-16T05:13:07.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.24 scrub starts 2024-09-16T05:13:07.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.24 scrub ok 2024-09-16T05:13:07.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: pgmap v534: 57 pgs: 2 peering, 1 active+remapped, 1 remapped+peering, 1 unknown, 52 active+clean; 319 MiB data, 773 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 24 op/s; 3.2 MiB/s, 4 objects/s recovering 2024-09-16T05:13:07.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: osdmap e255: 12 total, 12 up, 11 in 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:13:07.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:13:07.858 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3857984299' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:07.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:13:07.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:07.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:07.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:13:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.29 scrub ok 2024-09-16T05:13:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.24 scrub starts 2024-09-16T05:13:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.24 scrub ok 2024-09-16T05:13:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: pgmap v534: 57 pgs: 2 peering, 1 active+remapped, 1 remapped+peering, 1 unknown, 52 active+clean; 319 MiB data, 773 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 24 op/s; 3.2 MiB/s, 4 objects/s recovering 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:13:07.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: osdmap e255: 12 total, 12 up, 11 in 2024-09-16T05:13:07.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:13:07.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:13:07.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3857984299' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:08.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:08.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:08.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:13:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.29 scrub ok 2024-09-16T05:13:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.24 scrub starts 2024-09-16T05:13:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.24 scrub ok 2024-09-16T05:13:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: pgmap v534: 57 pgs: 2 peering, 1 active+remapped, 1 remapped+peering, 1 unknown, 52 active+clean; 319 MiB data, 773 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 24 op/s; 3.2 MiB/s, 4 objects/s recovering 2024-09-16T05:13:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: osdmap e255: 12 total, 12 up, 11 in 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:13:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3857984299' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:08.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:08.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:08.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:08.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:08.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:13:08.324 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:08.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:08.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:08.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:08.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:08.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:08.575 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:08.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:08.605 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:08.613 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:08.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:08.696 INFO:tasks.rados.rados.2.smithi204.stdout:722: done (0 left) 2024-09-16T05:13:08.696 INFO:tasks.rados.rados.2.smithi204.stdout:723: rollback oid 2 current snap is 79 2024-09-16T05:13:08.696 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 2 to 51 2024-09-16T05:13:08.696 INFO:tasks.rados.rados.2.smithi204.stdout:724: delete oid 39 current snap is 79 2024-09-16T05:13:08.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:08.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:08.701 INFO:tasks.rados.rados.2.smithi204.stdout:723: finishing rollback tid 1 to smithi20457400-2 2024-09-16T05:13:08.701 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 159 (ObjNum 8177 snap 0 seq_num 2944377712) dirty dne 2024-09-16T05:13:08.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:08.824 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:08.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:08.866 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.26 scrub starts 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.26 scrub ok 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: osdmap e256: 12 total, 12 up, 11 in 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:08 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:13:08.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:08.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:08.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.26 scrub starts 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.26 scrub ok 2024-09-16T05:13:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:13:08.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:13:08.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: osdmap e256: 12 total, 12 up, 11 in 2024-09-16T05:13:08.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:13:08.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:08 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:13:09.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:723: done (1 left) 2024-09-16T05:13:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:724: done (0 left) 2024-09-16T05:13:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:725: rollback oid 33 current snap is 79 2024-09-16T05:13:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 33 to 71 2024-09-16T05:13:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:726: rollback oid 5 current snap is 79 2024-09-16T05:13:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 5 to 77 2024-09-16T05:13:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:727: read oid 43 snap 77 2024-09-16T05:13:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:727: expect deleted 2024-09-16T05:13:09.074 INFO:tasks.rados.rados.2.smithi204.stdout:728: snap_create 2024-09-16T05:13:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:725: finishing rollback tid 0 to smithi20457400-33 2024-09-16T05:13:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:725: finishing rollback tid 1 to smithi20457400-33 2024-09-16T05:13:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:725: finishing rollback tid 2 to smithi20457400-33 2024-09-16T05:13:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 316 (ObjNum 239 snap 67 seq_num 239) dirty exists 2024-09-16T05:13:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:726: finishing rollback tid 1 to smithi20457400-5 2024-09-16T05:13:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 159 (ObjNum 257 snap 0 seq_num 4283257936) dirty dne 2024-09-16T05:13:09.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:09.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:13:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.26 scrub starts 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.26 scrub ok 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: osdmap e256: 12 total, 12 up, 11 in 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:13:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:08 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:13:09.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:09.203 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:09.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:09.294 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:13:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:13:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:13:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:13:09.295 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:13:09.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:09.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:09.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:09.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:09.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:09.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:09.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:09.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:09.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:09.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:09.710 INFO:tasks.rados.rados.2.smithi204.stdout:725: done (3 left) 2024-09-16T05:13:09.710 INFO:tasks.rados.rados.2.smithi204.stdout:726: done (2 left) 2024-09-16T05:13:09.710 INFO:tasks.rados.rados.2.smithi204.stdout:727: done (1 left) 2024-09-16T05:13:09.710 INFO:tasks.rados.rados.2.smithi204.stdout:728: done (0 left) 2024-09-16T05:13:09.710 INFO:tasks.rados.rados.2.smithi204.stdout:729: rollback oid 45 current snap is 80 2024-09-16T05:13:09.711 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 55 2024-09-16T05:13:09.711 INFO:tasks.rados.rados.2.smithi204.stdout:730: write (excl) oid 33 current snap is 80 2024-09-16T05:13:09.711 INFO:tasks.rados.rados.2.smithi204.stdout:730: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-16T05:13:09.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:09.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:09.716 INFO:tasks.rados.rados.2.smithi204.stdout:730: writing smithi20457400-33 from 756381 to 1156909 tid 1 2024-09-16T05:13:09.724 INFO:tasks.rados.rados.2.smithi204.stdout:730: writing smithi20457400-33 from 1910757 to 2375508 tid 2 2024-09-16T05:13:09.724 INFO:tasks.rados.rados.2.smithi204.stdout:731: read oid 26 snap -1 2024-09-16T05:13:09.724 INFO:tasks.rados.rados.2.smithi204.stdout:731: expect (ObjNum 261 snap 73 seq_num 261) 2024-09-16T05:13:09.724 INFO:tasks.rados.rados.2.smithi204.stdout:732: copy_from oid 37 from oid 22 current snap is 80 2024-09-16T05:13:09.724 INFO:tasks.rados.rados.2.smithi204.stdout:729: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:13:09.724 INFO:tasks.rados.rados.2.smithi204.stdout:729: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:13:09.724 INFO:tasks.rados.rados.2.smithi204.stdout:729: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:13:09.724 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 185 (ObjNum 170 snap 41 seq_num 170) dirty exists 2024-09-16T05:13:09.725 INFO:tasks.rados.rados.2.smithi204.stdout:729: done (3 left) 2024-09-16T05:13:09.725 INFO:tasks.rados.rados.2.smithi204.stdout:733: copy_from oid 49 from oid 21 current snap is 80 2024-09-16T05:13:09.725 INFO:tasks.rados.rados.2.smithi204.stdout:734: read oid 44 snap -1 2024-09-16T05:13:09.725 INFO:tasks.rados.rados.2.smithi204.stdout:734: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:13:09.725 INFO:tasks.rados.rados.2.smithi204.stdout:735: snap_remove snap 66 2024-09-16T05:13:09.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:09.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:09.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:09.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:09.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:09 smithi032 ceph-mon[27388]: pgmap v537: 57 pgs: 2 peering, 55 active+clean; 336 MiB data, 790 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-16T05:13:09.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:09 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:13:09.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:09 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:13:09.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:09 smithi032 ceph-mon[27388]: 2.2a scrub starts 2024-09-16T05:13:09.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:09 smithi032 ceph-mon[27388]: 2.2a scrub ok 2024-09-16T05:13:09.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:09 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:13:09.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:09 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:13:09.904 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:09 smithi032 ceph-mon[27388]: osdmap e257: 12 total, 12 up, 11 in 2024-09-16T05:13:09.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:13:09.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:09.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:09 smithi196 ceph-mon[30802]: pgmap v537: 57 pgs: 2 peering, 55 active+clean; 336 MiB data, 790 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-16T05:13:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:09 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:13:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:09 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:13:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:09 smithi196 ceph-mon[30802]: 2.2a scrub starts 2024-09-16T05:13:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:09 smithi196 ceph-mon[30802]: 2.2a scrub ok 2024-09-16T05:13:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:09 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:13:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:09 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:13:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:09 smithi196 ceph-mon[30802]: osdmap e257: 12 total, 12 up, 11 in 2024-09-16T05:13:10.001 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:10.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:10.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:10.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:10.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:09 smithi204 ceph-mon[30933]: pgmap v537: 57 pgs: 2 peering, 55 active+clean; 336 MiB data, 790 MiB used, 983 GiB / 983 GiB avail; 2.1 MiB/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-16T05:13:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:09 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:13:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:09 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:13:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:09 smithi204 ceph-mon[30933]: 2.2a scrub starts 2024-09-16T05:13:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:09 smithi204 ceph-mon[30933]: 2.2a scrub ok 2024-09-16T05:13:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:09 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:13:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:09 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:13:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:09 smithi204 ceph-mon[30933]: osdmap e257: 12 total, 12 up, 11 in 2024-09-16T05:13:10.197 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:10.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:10.254 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:10.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:10.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:10.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:10.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:10.508 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:10.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:10.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:13:10.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:10.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:10.654 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:13:10.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:13:10.654 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:13:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:10.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:10.670 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:13:10.676 INFO:tasks.rados.rados.2.smithi204.stdout:735: done (5 left) 2024-09-16T05:13:10.677 INFO:tasks.rados.rados.2.smithi204.stdout:736: delete oid 40 current snap is 80 2024-09-16T05:13:10.677 INFO:tasks.rados.rados.2.smithi204.stdout:730: finishing write tid 1 to smithi20457400-33 2024-09-16T05:13:10.677 INFO:tasks.rados.rados.2.smithi204.stdout:730: finishing write tid 2 to smithi20457400-33 2024-09-16T05:13:10.677 INFO:tasks.rados.rados.2.smithi204.stdout:730: finishing write tid 3 to smithi20457400-33 2024-09-16T05:13:10.677 INFO:tasks.rados.rados.2.smithi204.stdout:730: finishing write tid 4 to smithi20457400-33 2024-09-16T05:13:10.677 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 320 (ObjNum 283 snap 80 seq_num 283) dirty exists 2024-09-16T05:13:10.677 INFO:tasks.rados.rados.2.smithi204.stdout:730: left oid 33 (ObjNum 283 snap 80 seq_num 283) 2024-09-16T05:13:10.677 INFO:tasks.rados.rados.2.smithi204.stdout:734: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:115 2.a e112/107 2:580e4d0e:::smithi20457400-29:head v 112'6, mlcod=112'3)", 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 147.345595965, 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0311933, 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334345+0000", 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334350+0000", 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.887e-06 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365213+0000", 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030863313999999999 2024-09-16T05:13:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365220+0000", 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7870000000000004e-06 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365291+0000", 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1397999999999996e-05 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365307+0000", 2024-09-16T05:13:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5990000000000001e-05 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365507+0000", 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000199221 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365538+0000", 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1167000000000002e-05 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:125 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'7, mlcod=112'4)", 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 147.315425836, 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006511045, 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365198+0000", 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068299699999999997 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369743+0000", 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045446979999999998 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369750+0000", 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1940000000000001e-06 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369826+0000", 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6160000000000003e-05 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369841+0000", 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4895000000000001e-05 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.370997+0000", 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001155866 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.371026+0000", 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8564999999999998e-05 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:127 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'9, mlcod=112'5)", 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 147.310214927, 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020225074999999999, 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369726+0000", 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369728+0000", 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9410000000000001e-06 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389627+0000", 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019899751 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389632+0000", 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389704+0000", 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1418e-05 2024-09-16T05:13:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389718+0000", 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.433e-05 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389922+0000", 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020421600000000001 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389951+0000", 2024-09-16T05:13:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8140000000000002e-05 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:133 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'10, mlcod=112'7)", 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 147.29068099899999, 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074633729999999997, 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389261+0000", 2024-09-16T05:13:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.973e-06 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389613+0000", 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035121600000000001 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395726+0000", 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006113851 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395731+0000", 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0359999999999997e-06 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395819+0000", 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7128999999999999e-05 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395836+0000", 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7768999999999999e-05 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396693+0000", 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085685999999999998 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396723+0000", 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9539e-05 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:135 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'12, mlcod=112'8)", 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 147.28417002399999, 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069274319999999999, 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395771+0000", 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4499999999999997e-07 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395775+0000", 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6509999999999999e-06 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402319+0000", 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065439469999999996 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402326+0000", 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5769999999999996e-06 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402424+0000", 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8171000000000006e-05 2024-09-16T05:13:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402439+0000", 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4844999999999999e-05 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402660+0000", 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022083700000000001 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402698+0000", 2024-09-16T05:13:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7858999999999998e-05 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:773 2.1 e159/107 2:8c1ba486:::smithi20457400-2:head v 159'34, mlcod=155'30)", 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 99.886765960999995, 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060955719999999996, 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793175+0000", 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:13:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795522+0000", 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023476920000000002 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795574+0000", 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1459999999999999e-05 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795584+0000", 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0274e-05 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795649+0000", 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4901999999999996e-05 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795679+0000", 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9887e-05 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.799228+0000", 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035493679999999998 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.799270+0000", 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1813000000000001e-05 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:891 2.a e171/107 2:5089ebe1:::smithi20457400-36:head v 171'128, mlcod=171'124)", 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 88.020187669999999, 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003806771, 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1699999999999998e-07 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662145+0000", 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023911589999999999 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662170+0000", 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100999999999999e-05 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662177+0000", 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5120000000000002e-06 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662246+0000", 2024-09-16T05:13:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9264000000000006e-05 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662262+0000", 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5128e-05 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.663526+0000", 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001264311 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.663559+0000", 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3679000000000001e-05 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:974 2.a e179/107 2:5827f14d:::smithi20457400-12:head v 179'156, mlcod=178'152)", 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 79.672508339000004, 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068838909999999996, 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007433+0000", 2024-09-16T05:13:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.421e-06 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009914+0000", 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024808030000000002 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009966+0000", 2024-09-16T05:13:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1456999999999999e-05 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009975+0000", 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4010000000000003e-06 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010042+0000", 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010064+0000", 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2282000000000001e-05 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014259+0000", 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041942020000000002 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014316+0000", 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7377999999999999e-05 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1133 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'206, mlcod=189'203)", 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.945713729999994, 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021225035, 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:13:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.85e-07 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734829+0000", 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060196800000000003 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734866+0000", 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7469000000000003e-05 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734881+0000", 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5058999999999999e-05 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734970+0000", 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9004000000000004e-05 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734998+0000", 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7606e-05 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755384+0000", 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02038587 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755452+0000", 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7873999999999994e-05 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1134 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'207, mlcod=189'203)", 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.940928607999993, 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016477325000000001, 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739014+0000", 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2819999999999999e-06 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739797+0000", 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078265599999999996 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739844+0000", 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7608000000000002e-05 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739854+0000", 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4239999999999999e-06 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739935+0000", 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1593000000000003e-05 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739963+0000", 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7492e-05 2024-09-16T05:13:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755459+0000", 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015496565 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755489+0000", 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9705000000000001e-05 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1135 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'208, mlcod=189'203)", 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.936505771, 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012094011, 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743436+0000", 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.584e-06 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743535+0000", 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9110000000000005e-05 2024-09-16T05:13:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743568+0000", 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2335000000000001e-05 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743576+0000", 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.191e-06 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743646+0000", 2024-09-16T05:13:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0481000000000005e-05 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743676+0000", 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9620000000000001e-05 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755492+0000", 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011815888 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755529+0000", 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6801999999999998e-05 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1136 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'209, mlcod=189'203)", 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.935898213000002, 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011564019, 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744043+0000", 2024-09-16T05:13:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3600000000000003e-07 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744049+0000", 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8250000000000001e-06 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744069+0000", 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0460000000000001e-05 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744074+0000", 2024-09-16T05:13:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4730000000000002e-06 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744119+0000", 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744143+0000", 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4085999999999999e-05 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755570+0000", 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011427299 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755606+0000", 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146000000000003e-05 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1312 2.1a e210/209 2:5ae26c04:::smithi20457400-32:head v 210'262, mlcod=208'258)", 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:21.673576+0000", 2024-09-16T05:13:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 49.006364427000001, 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035088160000000001, 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.673576+0000", 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.673576+0000", 2024-09-16T05:13:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.673577+0000", 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.046e-06 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.675597+0000", 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020199089999999999 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.675644+0000", 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7024000000000002e-05 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.675656+0000", 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2189e-05 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.675759+0000", 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103006 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.675784+0000", 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4362000000000002e-05 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.677039+0000", 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001255496 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:21.677085+0000", 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5784000000000001e-05 2024-09-16T05:13:10.776 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1478 2.1e 2:7b642727:::smithi20457400-40:head [write 581575~705262] snapc 3d=[3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:28.693708+0000", 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 41.986232246999997, 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059374299999999996, 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:13:10.777 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1478 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.693708+0000", 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.693708+0000", 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.693711+0000", 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9699999999999999e-06 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695661+0000", 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019495020000000001 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695679+0000", 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8553999999999999e-05 2024-09-16T05:13:10.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695688+0000", 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0070000000000003e-06 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695790+0000", 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000101944 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:13:10.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695799+0000", 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4749999999999993e-06 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.697065+0000", 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012665860000000001 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.697162+0000", 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7158999999999996e-05 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.697311+0000", 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000148639 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699078+0000", 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001766847 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699537+0000", 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000459328 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699602+0000", 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5288000000000002e-05 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699646+0000", 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3130999999999998e-05 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.783 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1479 2.1e 2:7b642727:::smithi20457400-40:head [write 1733272~707409] snapc 3d=[3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:28.699922+0000", 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 41.980018424000001, 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034777879999999999, 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1479 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699922+0000", 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699922+0000", 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699924+0000", 2024-09-16T05:13:10.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5999999999999999e-06 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.700956+0000", 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010327820000000001 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.700972+0000", 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5846e-05 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.700982+0000", 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4129999999999995e-06 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.701036+0000", 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4562999999999999e-05 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.701126+0000", 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9577e-05 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.701263+0000", 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000137656 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.702366+0000", 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011028259999999999 2024-09-16T05:13:10.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.703308+0000", 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094160299999999995 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.703360+0000", 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2102999999999999e-05 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.703400+0000", 2024-09-16T05:13:10.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9818999999999999e-05 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1520 2.e e219/209 2:75e143e9:::smithi20457400-17:head v 219'163, mlcod=214'160)", 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:30.813845+0000", 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 39.866095358000003, 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003377279, 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.813845+0000", 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.813845+0000", 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.813846+0000", 2024-09-16T05:13:10.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.074e-06 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.814915+0000", 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010693429999999999 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.814958+0000", 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2682000000000002e-05 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.814969+0000", 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1212999999999999e-05 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.815031+0000", 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1241000000000005e-05 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.815050+0000", 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9690999999999998e-05 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.817169+0000", 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002119035 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:30.817222+0000", 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3000000000000001e-05 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.794 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1589 2.e e227/209 2:75dc7feb:::smithi20457400-28:head v 227'178, mlcod=227'175)", 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:38.722921+0000", 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 31.957018898000001, 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051262199999999999, 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.722921+0000", 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.722921+0000", 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.722923+0000", 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4130000000000001e-06 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724645+0000", 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001722131 2024-09-16T05:13:10.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724699+0000", 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3522e-05 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724711+0000", 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2376e-05 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724779+0000", 2024-09-16T05:13:10.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8391000000000003e-05 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724805+0000", 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5177e-05 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.728005+0000", 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032002689999999999 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.798 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.728048+0000", 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2941000000000001e-05 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1799 2.1e 2:7e4e65a4:::smithi20457400-4:head [append 0~0] snapc 4b=[4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:57.739968+0000", 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 12.93997257, 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027208359999999999, 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1799 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.739968+0000", 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.739968+0000", 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.739970+0000", 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.841e-06 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.739979+0000", 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.798e-06 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740000+0000", 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0995000000000001e-05 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740012+0000", 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1382e-05 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740108+0000", 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5660000000000002e-05 2024-09-16T05:13:10.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740322+0000", 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021425099999999999 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740598+0000", 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00027585799999999997 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742550+0000", 2024-09-16T05:13:10.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001952372 2024-09-16T05:13:10.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:10.805 INFO:tasks.rados.rados.2.smithi204.stdout:732: finishing copy_from to smithi20457400-37 2024-09-16T05:13:10.805 INFO:tasks.rados.rados.2.smithi204.stdout:732: got expected ENOENT (src dne) 2024-09-16T05:13:10.805 INFO:tasks.rados.rados.2.smithi204.stdout:732: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:13:10.805 INFO:tasks.rados.rados.2.smithi204.stdout:733: finishing copy_from racing read to smithi20457400-49 2024-09-16T05:13:10.805 INFO:tasks.rados.rados.2.smithi204.stdout:731: expect (ObjNum 261 snap 73 seq_num 261) 2024-09-16T05:13:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742610+0000", 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0262e-05 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742652+0000", 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1495999999999997e-05 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742689+0000", 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6921000000000001e-05 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.806 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1801 2.1e 2:7e4e65a4:::smithi20457400-4:head [append 0~0] snapc 4b=[4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:57.740129+0000", 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 12.939811296, 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028408539999999999, 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:10.807 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1801 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740129+0000", 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740129+0000", 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740130+0000", 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7000000000000003e-07 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740134+0000", 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4909999999999999e-06 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740145+0000", 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0971999999999999e-05 2024-09-16T05:13:10.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.740153+0000", 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6839999999999996e-06 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742074+0000", 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019211930000000001 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742185+0000", 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111183 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742312+0000", 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012613300000000001 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742785+0000", 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00047340700000000002 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.811 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742891+0000", 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105815 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742938+0000", 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7148999999999999e-05 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:57.742970+0000", 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1956999999999999e-05 2024-09-16T05:13:10.812 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1810 2.1e 2:7e4e65a4:::smithi20457400-4:head [write 487469~563838] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:58.765846+0000", 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 11.914093998, 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031111310000000001, 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:13:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1810 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.765846+0000", 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.765846+0000", 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.765848+0000", 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9879999999999999e-06 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.766486+0000", 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063722200000000001 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.766510+0000", 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3930999999999998e-05 2024-09-16T05:13:10.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.766523+0000", 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3093e-05 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.766600+0000", 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7292999999999993e-05 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.766693+0000", 2024-09-16T05:13:10.816 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3169000000000002e-05 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.766857+0000", 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000163833 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.768455+0000", 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015977980000000001 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:10.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.768799+0000", 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00034464800000000002 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.768878+0000", 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8677999999999994e-05 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:58.768958+0000", 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9478e-05 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.818 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.819 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:13:10.819 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:13:10.819 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:13:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:10.841 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:10.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:10.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:10 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:13:10.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:10 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:13:10.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:10 smithi032 ceph-mon[27388]: osdmap e258: 12 total, 12 up, 11 in 2024-09-16T05:13:10.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3086459006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:10.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:10 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:10.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:10 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:13:10.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3086459006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:10.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:10 smithi032 ceph-mon[27388]: osdmap e259: 12 total, 12 up, 11 in 2024-09-16T05:13:10.934 INFO:tasks.rados.rados.2.smithi204.stdout:733: finishing copy_from to smithi20457400-49 2024-09-16T05:13:10.934 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 67 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:13:10.934 INFO:tasks.rados.rados.2.smithi204.stdout:730: done (5 left) 2024-09-16T05:13:10.934 INFO:tasks.rados.rados.2.smithi204.stdout:731: done (4 left) 2024-09-16T05:13:10.934 INFO:tasks.rados.rados.2.smithi204.stdout:732: done (3 left) 2024-09-16T05:13:10.935 INFO:tasks.rados.rados.2.smithi204.stdout:733: done (2 left) 2024-09-16T05:13:10.935 INFO:tasks.rados.rados.2.smithi204.stdout:734: done (1 left) 2024-09-16T05:13:10.935 INFO:tasks.rados.rados.2.smithi204.stdout:736: done (0 left) 2024-09-16T05:13:10.935 INFO:tasks.rados.rados.2.smithi204.stdout:737: write (excl) oid 12 current snap is 80 2024-09-16T05:13:10.935 INFO:tasks.rados.rados.2.smithi204.stdout:737: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-16T05:13:10.941 INFO:tasks.rados.rados.2.smithi204.stdout:737: writing smithi20457400-12 from 789995 to 1341021 tid 1 2024-09-16T05:13:10.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:10.947 INFO:tasks.rados.rados.2.smithi204.stdout:737: writing smithi20457400-12 from 2027721 to 2496141 tid 2 2024-09-16T05:13:10.947 INFO:tasks.rados.rados.2.smithi204.stdout:738: delete oid 15 current snap is 80 2024-09-16T05:13:10.947 INFO:tasks.rados.rados.2.smithi204.stdout:737: finishing write tid 1 to smithi20457400-12 2024-09-16T05:13:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:737: finishing write tid 2 to smithi20457400-12 2024-09-16T05:13:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:737: finishing write tid 3 to smithi20457400-12 2024-09-16T05:13:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:737: finishing write tid 4 to smithi20457400-12 2024-09-16T05:13:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 372 (ObjNum 285 snap 80 seq_num 285) dirty exists 2024-09-16T05:13:10.950 INFO:tasks.rados.rados.2.smithi204.stdout:737: left oid 12 (ObjNum 285 snap 80 seq_num 285) 2024-09-16T05:13:10.951 INFO:tasks.rados.rados.2.smithi204.stdout:737: done (1 left) 2024-09-16T05:13:10.951 INFO:tasks.rados.rados.2.smithi204.stdout:738: done (0 left) 2024-09-16T05:13:10.951 INFO:tasks.rados.rados.2.smithi204.stdout:739: copy_from oid 1 from oid 38 current snap is 80 2024-09-16T05:13:10.951 INFO:tasks.rados.rados.2.smithi204.stdout:740: read oid 11 snap -1 2024-09-16T05:13:10.951 INFO:tasks.rados.rados.2.smithi204.stdout:740: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:13:10.951 INFO:tasks.rados.rados.2.smithi204.stdout:741: snap_remove snap 55 2024-09-16T05:13:10.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:10.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:10.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:10 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:13:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:10 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:13:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:10 smithi196 ceph-mon[30802]: osdmap e258: 12 total, 12 up, 11 in 2024-09-16T05:13:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3086459006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:10 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:10.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:10 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:13:10.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3086459006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:10.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:10 smithi196 ceph-mon[30802]: osdmap e259: 12 total, 12 up, 11 in 2024-09-16T05:13:11.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:11.092 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:11.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:10 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:13:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:10 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:13:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:10 smithi204 ceph-mon[30933]: osdmap e258: 12 total, 12 up, 11 in 2024-09-16T05:13:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3086459006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:10 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:10 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:13:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3086459006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:10 smithi204 ceph-mon[30933]: osdmap e259: 12 total, 12 up, 11 in 2024-09-16T05:13:11.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:11.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:11.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:11.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:11.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:11.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:11.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:11.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:11.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:11.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:11.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:11.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:11.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:11.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:11.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:11.723 INFO:tasks.rados.rados.2.smithi204.stdout:741: done (2 left) 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:742: read oid 22 snap -1 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:742: expect deleted 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:743: read oid 6 snap 51 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:743: expect (ObjNum 195 snap 48 seq_num 195) 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:744: read oid 12 snap 69 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:744: expect (ObjNum 124 snap 26 seq_num 124) 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:745: read oid 32 snap -1 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:745: expect (ObjNum 248 snap 69 seq_num 248) 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:746: delete oid 13 current snap is 80 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:739: finishing copy_from to smithi20457400-1 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:739: got expected ENOENT (src dne) 2024-09-16T05:13:11.724 INFO:tasks.rados.rados.2.smithi204.stdout:739: finishing copy_from racing read to smithi20457400-1 2024-09-16T05:13:11.725 INFO:tasks.rados.rados.2.smithi204.stdout:740: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:13:11.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:11.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:11.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:11.850 INFO:tasks.rados.rados.2.smithi204.stdout:739: done (6 left) 2024-09-16T05:13:11.851 INFO:tasks.rados.rados.2.smithi204.stdout:740: done (5 left) 2024-09-16T05:13:11.851 INFO:tasks.rados.rados.2.smithi204.stdout:742: done (4 left) 2024-09-16T05:13:11.851 INFO:tasks.rados.rados.2.smithi204.stdout:746: done (3 left) 2024-09-16T05:13:11.851 INFO:tasks.rados.rados.2.smithi204.stdout:747: snap_remove snap 78 2024-09-16T05:13:11.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:11.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:11.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:11.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:11 smithi196 ceph-mon[30802]: pgmap v540: 57 pgs: 57 active+clean; 335 MiB data, 788 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:11.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:11 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:13:11.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:11 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:13:11.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:11 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:13:11.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:11 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:13:12.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:12.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:12.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:12.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:11 smithi032 ceph-mon[27388]: pgmap v540: 57 pgs: 57 active+clean; 335 MiB data, 788 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:12.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:11 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:13:12.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:11 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:13:12.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:11 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:13:12.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:11 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:13:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:11 smithi204 ceph-mon[30933]: pgmap v540: 57 pgs: 57 active+clean; 335 MiB data, 788 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:11 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:13:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:11 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:13:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:11 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:13:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:11 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:13:12.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:12.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:13:12.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:12.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:12.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:12.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:12.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:12.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:12.448 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:12.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:12.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:12.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:12.584 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:12.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:12.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:12.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:12.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:12.724 INFO:tasks.rados.rados.2.smithi204.stdout:747: done (3 left) 2024-09-16T05:13:12.725 INFO:tasks.rados.rados.2.smithi204.stdout:748: read oid 43 snap 51 2024-09-16T05:13:12.725 INFO:tasks.rados.rados.2.smithi204.stdout:748: expect (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:13:12.725 INFO:tasks.rados.rados.2.smithi204.stdout:749: snap_remove snap 79 2024-09-16T05:13:12.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:12.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:12.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:12 smithi196 ceph-mon[30802]: osdmap e260: 12 total, 12 up, 11 in 2024-09-16T05:13:12.882 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:13:12.882 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:13:12.882 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:13:12.882 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:13:12.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:12.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:12.985 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:12 smithi032 ceph-mon[27388]: osdmap e260: 12 total, 12 up, 11 in 2024-09-16T05:13:13.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:13.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:13.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:12 smithi204 ceph-mon[30933]: osdmap e260: 12 total, 12 up, 11 in 2024-09-16T05:13:13.184 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:13.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:13.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:13.261 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:13:13.262 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:13:13.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:13.330 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:13.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:13.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:13.454 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:13.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:13.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:13.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:13.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:13.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:13:13.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:13.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:13.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:13.726 INFO:tasks.rados.rados.2.smithi204.stdout:749: done (4 left) 2024-09-16T05:13:13.726 INFO:tasks.rados.rados.2.smithi204.stdout:750: snap_create 2024-09-16T05:13:13.726 INFO:tasks.rados.rados.2.smithi204.stdout:744: expect (ObjNum 124 snap 26 seq_num 124) 2024-09-16T05:13:13.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:13:13.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:13.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:13.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:13.840 INFO:tasks.rados.rados.2.smithi204.stdout:745: expect (ObjNum 248 snap 69 seq_num 248) 2024-09-16T05:13:13.893 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:13.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:13 smithi032 ceph-mon[27388]: 2.18 deep-scrub starts 2024-09-16T05:13:13.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:13 smithi032 ceph-mon[27388]: 2.18 deep-scrub ok 2024-09-16T05:13:13.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:13 smithi032 ceph-mon[27388]: pgmap v543: 57 pgs: 57 active+clean; 308 MiB data, 780 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:13.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:13 smithi032 ceph-mon[27388]: osdmap e261: 12 total, 12 up, 11 in 2024-09-16T05:13:13.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1769255834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:13.910 INFO:tasks.rados.rados.2.smithi204.stdout:743: expect (ObjNum 195 snap 48 seq_num 195) 2024-09-16T05:13:13.927 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":544,"stamp":"2024-09-16T05:13:12.721622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":326821140,"num_objects":165,"num_object_clones":116,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":14,"num_read":307,"num_read_kb":353770,"num_write":190,"num_write_kb":77318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":33595633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8679,"ondisk_log_size":8679,"up":114,"acting":114,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":800380,"kb_used_data":448936,"kb_used_omap":17,"kb_used_meta":351150,"kb_avail":1030396292,"statfs":{"total":1055945392128,"available":1055125803008,"internally_reserved":0,"allocated":459710464,"data_stored":567233234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":359578489},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25199478,"num_objects":-13,"num_object_clones":-9,"num_object_copies":-26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13,"num_whiteouts":-2,"num_read":-79,"num_read_kb":-70704,"num_write":-56,"num_write_kb":-31403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-19,"num_bytes_recovered":-13556304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030404"},"pg_stats":[{"pgid":"2.2d","version":"245'306","reported_seq":1049,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.726565+0000","last_change":"2024-09-16T05:13:11.726851+0000","last_active":"2024-09-16T05:13:11.726565+0000","last_peered":"2024-09-16T05:13:11.726565+0000","last_clean":"2024-09-16T05:13:11.726565+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:13:11.726565+0000","last_undegraded":"2024-09-16T05:13:11.726565+0000","last_fullsized":"2024-09-16T05:13:11.726565+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'306","last_scrub_stamp":"2024-09-16T05:13:10.471980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:10.471980+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:14.792442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001233647,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":875,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T05:13:08.694443+0000","last_change":"2024-09-16T05:13:08.484612+0000","last_active":"2024-09-16T05:13:08.694443+0000","last_peered":"2024-09-16T05:13:08.694443+0000","last_clean":"2024-09-16T05:13:08.694443+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:13:08.694443+0000","last_undegraded":"2024-09-16T05:13:08.694443+0000","last_fullsized":"2024-09-16T05:13:08.694443+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:13:08.484521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.484521+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:16.361434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0062201519999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.2b","version":"260'182","reported_seq":946,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.722870+0000","last_change":"2024-09-16T05:13:11.723145+0000","last_active":"2024-09-16T05:13:11.722870+0000","last_peered":"2024-09-16T05:13:11.722870+0000","last_clean":"2024-09-16T05:13:11.722870+0000","last_became_active":"2024-09-16T05:13:04.678973+0000","last_became_peered":"2024-09-16T05:13:04.678973+0000","last_unstale":"2024-09-16T05:13:11.722870+0000","last_undegraded":"2024-09-16T05:13:11.722870+0000","last_fullsized":"2024-09-16T05:13:11.722870+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"253'179","last_scrub_stamp":"2024-09-16T05:13:06.865580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.865580+0000","objects_scrubbed":4,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:09.523073+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0042347929999999997,"stat_sum":{"num_bytes":2957999,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1069,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.730537+0000","last_change":"2024-09-16T05:13:11.730809+0000","last_active":"2024-09-16T05:13:11.730537+0000","last_peered":"2024-09-16T05:13:11.730537+0000","last_clean":"2024-09-16T05:13:11.730537+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:13:11.730537+0000","last_undegraded":"2024-09-16T05:13:11.730537+0000","last_fullsized":"2024-09-16T05:13:11.730537+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:13:08.516105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.516105+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:35.882379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.004708528,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.29","version":"260'252","reported_seq":1070,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.727863+0000","last_change":"2024-09-16T05:13:11.728263+0000","last_active":"2024-09-16T05:13:11.727863+0000","last_peered":"2024-09-16T05:13:11.727863+0000","last_clean":"2024-09-16T05:13:11.727863+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:13:11.727863+0000","last_undegraded":"2024-09-16T05:13:11.727863+0000","last_fullsized":"2024-09-16T05:13:11.727863+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"254'244","last_scrub_stamp":"2024-09-16T05:13:06.019617+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.019617+0000","objects_scrubbed":16,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:32.867558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0036671270000000001,"stat_sum":{"num_bytes":26600345,"num_objects":16,"num_object_clones":12,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":833,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T05:13:07.367356+0000","last_change":"2024-09-16T05:13:06.496211+0000","last_active":"2024-09-16T05:13:07.367356+0000","last_peered":"2024-09-16T05:13:07.367356+0000","last_clean":"2024-09-16T05:13:07.367356+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:13:07.367356+0000","last_undegraded":"2024-09-16T05:13:07.367356+0000","last_fullsized":"2024-09-16T05:13:07.367356+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:13:06.496116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.496116+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:29.943542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085741599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":79,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.726223+0000","last_change":"2024-09-16T05:13:11.726470+0000","last_active":"2024-09-16T05:13:11.726223+0000","last_peered":"2024-09-16T05:13:11.726223+0000","last_clean":"2024-09-16T05:13:11.726223+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:13:11.726223+0000","last_undegraded":"2024-09-16T05:13:11.726223+0000","last_fullsized":"2024-09-16T05:13:11.726223+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:09.483428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:09.483428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:20.362033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086116800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":67,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.730172+0000","last_change":"2024-09-16T05:13:11.730461+0000","last_active":"2024-09-16T05:13:11.730172+0000","last_peered":"2024-09-16T05:13:11.730172+0000","last_clean":"2024-09-16T05:13:11.730172+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:13:11.730172+0000","last_undegraded":"2024-09-16T05:13:11.730172+0000","last_fullsized":"2024-09-16T05:13:11.730172+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:13:07.538943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.538943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:32.190035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.004565253,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.25","version":"254'432","reported_seq":1133,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T05:13:08.694611+0000","last_change":"2024-09-16T05:13:07.464546+0000","last_active":"2024-09-16T05:13:08.694611+0000","last_peered":"2024-09-16T05:13:08.694611+0000","last_clean":"2024-09-16T05:13:08.694611+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:13:08.694611+0000","last_undegraded":"2024-09-16T05:13:08.694611+0000","last_fullsized":"2024-09-16T05:13:08.694611+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"254'432","last_scrub_stamp":"2024-09-16T05:13:07.464459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.464459+0000","objects_scrubbed":9,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:33.606785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032186889999999998,"stat_sum":{"num_bytes":20783505,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":53,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.719092+0000","last_change":"2024-09-16T05:13:06.032613+0000","last_active":"2024-09-16T05:13:09.719092+0000","last_peered":"2024-09-16T05:13:09.719092+0000","last_clean":"2024-09-16T05:13:09.719092+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:13:09.719092+0000","last_undegraded":"2024-09-16T05:13:09.719092+0000","last_fullsized":"2024-09-16T05:13:09.719092+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:13:06.032520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.032520+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:34.935602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001224192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":224,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T05:13:07.367601+0000","last_change":"2024-09-16T05:13:05.684236+0000","last_active":"2024-09-16T05:13:07.367601+0000","last_peered":"2024-09-16T05:13:07.367601+0000","last_clean":"2024-09-16T05:13:07.367601+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:13:07.367601+0000","last_undegraded":"2024-09-16T05:13:07.367601+0000","last_fullsized":"2024-09-16T05:13:07.367601+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:12:26.533655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.533655+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:35.546145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001275481,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":125,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T05:13:07.367336+0000","last_change":"2024-09-16T05:13:05.683756+0000","last_active":"2024-09-16T05:13:07.367336+0000","last_peered":"2024-09-16T05:13:07.367336+0000","last_clean":"2024-09-16T05:13:07.367336+0000","last_became_active":"2024-09-16T05:12:44.679281+0000","last_became_peered":"2024-09-16T05:12:44.679281+0000","last_unstale":"2024-09-16T05:13:07.367336+0000","last_undegraded":"2024-09-16T05:13:07.367336+0000","last_fullsized":"2024-09-16T05:13:07.367336+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:45.564449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.564449+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:08.033484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086376300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":800,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.721794+0000","last_change":"2024-09-16T05:13:11.722654+0000","last_active":"2024-09-16T05:13:11.721794+0000","last_peered":"2024-09-16T05:13:11.721794+0000","last_clean":"2024-09-16T05:13:11.721794+0000","last_became_active":"2024-09-16T05:12:23.806187+0000","last_became_peered":"2024-09-16T05:12:23.806187+0000","last_unstale":"2024-09-16T05:13:11.721794+0000","last_undegraded":"2024-09-16T05:13:11.721794+0000","last_fullsized":"2024-09-16T05:13:11.721794+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:26.747969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.747969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.777999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030569490000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.20","version":"244'45","reported_seq":817,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.729827+0000","last_change":"2024-09-16T05:13:11.730110+0000","last_active":"2024-09-16T05:13:11.729827+0000","last_peered":"2024-09-16T05:13:11.729827+0000","last_clean":"2024-09-16T05:13:11.729827+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:13:11.729827+0000","last_undegraded":"2024-09-16T05:13:11.729827+0000","last_fullsized":"2024-09-16T05:13:11.729827+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"214'37","last_scrub_stamp":"2024-09-16T05:12:27.064860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.064860+0000","objects_scrubbed":3,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:27.495386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.004630856,"stat_sum":{"num_bytes":7916920,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":24943,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.1f","version":"258'185","reported_seq":938,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.718973+0000","last_change":"2024-09-16T05:13:05.690655+0000","last_active":"2024-09-16T05:13:09.718973+0000","last_peered":"2024-09-16T05:13:09.718973+0000","last_clean":"2024-09-16T05:13:09.718973+0000","last_became_active":"2024-09-16T05:12:26.664097+0000","last_became_peered":"2024-09-16T05:12:26.664097+0000","last_unstale":"2024-09-16T05:13:09.718973+0000","last_undegraded":"2024-09-16T05:13:09.718973+0000","last_fullsized":"2024-09-16T05:13:09.718973+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'148","last_scrub_stamp":"2024-09-16T05:12:27.189264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.189264+0000","objects_scrubbed":2,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:41.023693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040464120000000001,"stat_sum":{"num_bytes":9434476,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":53749,"num_write":7,"num_write_kb":1697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.1e","version":"254'245","reported_seq":985,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T05:13:07.367493+0000","last_change":"2024-09-16T05:13:05.688740+0000","last_active":"2024-09-16T05:13:07.367493+0000","last_peered":"2024-09-16T05:13:07.367493+0000","last_clean":"2024-09-16T05:13:07.367493+0000","last_became_active":"2024-09-16T05:12:23.805930+0000","last_became_peered":"2024-09-16T05:12:23.805930+0000","last_unstale":"2024-09-16T05:13:07.367493+0000","last_undegraded":"2024-09-16T05:13:07.367493+0000","last_fullsized":"2024-09-16T05:13:07.367493+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'171","last_scrub_stamp":"2024-09-16T05:12:27.537258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.537258+0000","objects_scrubbed":7,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:41.494204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0057721919999999998,"stat_sum":{"num_bytes":17252417,"num_objects":11,"num_object_clones":9,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10,"num_read_kb":5496,"num_write":16,"num_write_kb":5081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.1d","version":"260'399","reported_seq":1190,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.734335+0000","last_change":"2024-09-16T05:13:11.734548+0000","last_active":"2024-09-16T05:13:11.734335+0000","last_peered":"2024-09-16T05:13:11.734335+0000","last_clean":"2024-09-16T05:13:11.734335+0000","last_became_active":"2024-09-16T05:12:44.698111+0000","last_became_peered":"2024-09-16T05:12:44.698111+0000","last_unstale":"2024-09-16T05:13:11.734335+0000","last_undegraded":"2024-09-16T05:13:11.734335+0000","last_fullsized":"2024-09-16T05:13:11.734335+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'276","last_scrub_stamp":"2024-09-16T05:12:29.635450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:29.635450+0000","objects_scrubbed":9,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:34.348938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0086318950000000005,"stat_sum":{"num_bytes":24741139,"num_objects":15,"num_object_clones":11,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":24,"num_read_kb":23289,"num_write":23,"num_write_kb":9802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":16830963,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.1c","version":"260'303","reported_seq":1078,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.733287+0000","last_change":"2024-09-16T05:13:11.733641+0000","last_active":"2024-09-16T05:13:11.733287+0000","last_peered":"2024-09-16T05:13:11.733287+0000","last_clean":"2024-09-16T05:13:11.733287+0000","last_became_active":"2024-09-16T05:12:44.684658+0000","last_became_peered":"2024-09-16T05:12:44.684658+0000","last_unstale":"2024-09-16T05:13:11.733287+0000","last_undegraded":"2024-09-16T05:13:11.733287+0000","last_fullsized":"2024-09-16T05:13:11.733287+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'202","last_scrub_stamp":"2024-09-16T05:12:28.627965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.627965+0000","objects_scrubbed":8,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:31.631483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0068312160000000002,"stat_sum":{"num_bytes":23277421,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":18,"num_read_kb":9440,"num_write":29,"num_write_kb":8506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13583680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.1b","version":"260'237","reported_seq":1016,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.724185+0000","last_change":"2024-09-16T05:13:11.724499+0000","last_active":"2024-09-16T05:13:11.724185+0000","last_peered":"2024-09-16T05:13:11.724185+0000","last_clean":"2024-09-16T05:13:11.724185+0000","last_became_active":"2024-09-16T05:12:23.807872+0000","last_became_peered":"2024-09-16T05:12:23.807872+0000","last_unstale":"2024-09-16T05:13:11.724185+0000","last_undegraded":"2024-09-16T05:13:11.724185+0000","last_fullsized":"2024-09-16T05:13:11.724185+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"214'162","last_scrub_stamp":"2024-09-16T05:12:26.326323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.326323+0000","objects_scrubbed":7,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:51.172111+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0037531109999999999,"stat_sum":{"num_bytes":15505131,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":13,"num_read_kb":10166,"num_write":13,"num_write_kb":5770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.1a","version":"254'363","reported_seq":1079,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T05:13:08.694436+0000","last_change":"2024-09-16T05:13:06.104443+0000","last_active":"2024-09-16T05:13:08.694436+0000","last_peered":"2024-09-16T05:13:08.694436+0000","last_clean":"2024-09-16T05:13:08.694436+0000","last_became_active":"2024-09-16T05:12:27.664055+0000","last_became_peered":"2024-09-16T05:12:27.664055+0000","last_unstale":"2024-09-16T05:13:08.694436+0000","last_undegraded":"2024-09-16T05:13:08.694436+0000","last_fullsized":"2024-09-16T05:13:08.694436+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'278","last_scrub_stamp":"2024-09-16T05:12:28.451582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.451582+0000","objects_scrubbed":11,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:33.469627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.42404124399999998,"stat_sum":{"num_bytes":24844019,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":12,"num_read_kb":9482,"num_write":12,"num_write_kb":2155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":346,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T05:13:07.367733+0000","last_change":"2024-09-16T05:13:05.687222+0000","last_active":"2024-09-16T05:13:07.367733+0000","last_peered":"2024-09-16T05:13:07.367733+0000","last_clean":"2024-09-16T05:13:07.367733+0000","last_became_active":"2024-09-16T05:11:57.658646+0000","last_became_peered":"2024-09-16T05:11:57.658646+0000","last_unstale":"2024-09-16T05:13:07.367733+0000","last_undegraded":"2024-09-16T05:13:07.367733+0000","last_fullsized":"2024-09-16T05:13:07.367733+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:12:09.470099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.470099+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.391049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039168370000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":748,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.718893+0000","last_change":"2024-09-16T05:13:05.686454+0000","last_active":"2024-09-16T05:13:09.718893+0000","last_peered":"2024-09-16T05:13:09.718893+0000","last_clean":"2024-09-16T05:13:09.718893+0000","last_became_active":"2024-09-16T05:11:57.660350+0000","last_became_peered":"2024-09-16T05:11:57.660350+0000","last_unstale":"2024-09-16T05:13:09.718893+0000","last_undegraded":"2024-09-16T05:13:09.718893+0000","last_fullsized":"2024-09-16T05:13:09.718893+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:12:08.969861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.969861+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:25.467645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077301900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.17","version":"0'0","reported_seq":554,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.725075+0000","last_change":"2024-09-16T05:13:11.725583+0000","last_active":"2024-09-16T05:13:11.725075+0000","last_peered":"2024-09-16T05:13:11.725075+0000","last_clean":"2024-09-16T05:13:11.725075+0000","last_became_active":"2024-09-16T05:12:44.679525+0000","last_became_peered":"2024-09-16T05:12:44.679525+0000","last_unstale":"2024-09-16T05:13:11.725075+0000","last_undegraded":"2024-09-16T05:13:11.725075+0000","last_fullsized":"2024-09-16T05:13:11.725075+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:12:09.877069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.877069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:33.923821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092708600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.16","version":"259'53","reported_seq":418,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:13:10.870038+0000","last_change":"2024-09-16T05:13:10.870281+0000","last_active":"2024-09-16T05:13:10.870038+0000","last_peered":"2024-09-16T05:13:10.870038+0000","last_clean":"2024-09-16T05:13:10.870038+0000","last_became_active":"2024-09-16T05:11:57.656772+0000","last_became_peered":"2024-09-16T05:11:57.656772+0000","last_unstale":"2024-09-16T05:13:10.870038+0000","last_undegraded":"2024-09-16T05:13:10.870038+0000","last_fullsized":"2024-09-16T05:13:10.870038+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:12:09.331857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.331857+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:27.631824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020355880000000001,"stat_sum":{"num_bytes":4044872,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":21,"num_read_kb":26292,"num_write":10,"num_write_kb":2494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"37","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.15","version":"254'363","reported_seq":1094,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.719277+0000","last_change":"2024-09-16T05:13:05.690664+0000","last_active":"2024-09-16T05:13:09.719277+0000","last_peered":"2024-09-16T05:13:09.719277+0000","last_clean":"2024-09-16T05:13:09.719277+0000","last_became_active":"2024-09-16T05:12:00.711890+0000","last_became_peered":"2024-09-16T05:12:00.711890+0000","last_unstale":"2024-09-16T05:13:09.719277+0000","last_undegraded":"2024-09-16T05:13:09.719277+0000","last_fullsized":"2024-09-16T05:13:09.719277+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'276","last_scrub_stamp":"2024-09-16T05:12:11.920515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:11.920515+0000","objects_scrubbed":7,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:22.277180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0045069259999999996,"stat_sum":{"num_bytes":20066047,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":36,"num_read_kb":56083,"num_write":24,"num_write_kb":13520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.14","version":"254'176","reported_seq":495,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.719219+0000","last_change":"2024-09-16T05:13:05.689198+0000","last_active":"2024-09-16T05:13:09.719219+0000","last_peered":"2024-09-16T05:13:09.719219+0000","last_clean":"2024-09-16T05:13:09.719219+0000","last_became_active":"2024-09-16T05:11:57.658852+0000","last_became_peered":"2024-09-16T05:11:57.658852+0000","last_unstale":"2024-09-16T05:13:09.719219+0000","last_undegraded":"2024-09-16T05:13:09.719219+0000","last_fullsized":"2024-09-16T05:13:09.719219+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'108","last_scrub_stamp":"2024-09-16T05:12:10.924782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.924782+0000","objects_scrubbed":6,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:23.622184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00344306,"stat_sum":{"num_bytes":22202598,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":47512,"num_write":8,"num_write_kb":7733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.13","version":"181'15","reported_seq":355,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.719070+0000","last_change":"2024-09-16T05:13:05.686448+0000","last_active":"2024-09-16T05:13:09.719070+0000","last_peered":"2024-09-16T05:13:09.719070+0000","last_clean":"2024-09-16T05:13:09.719070+0000","last_became_active":"2024-09-16T05:11:57.660401+0000","last_became_peered":"2024-09-16T05:11:57.660401+0000","last_unstale":"2024-09-16T05:13:09.719070+0000","last_undegraded":"2024-09-16T05:13:09.719070+0000","last_fullsized":"2024-09-16T05:13:09.719070+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:12:09.952097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.952097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:25.466494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071533899999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.12","version":"254'50","reported_seq":345,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T05:13:07.367602+0000","last_change":"2024-09-16T05:13:05.689018+0000","last_active":"2024-09-16T05:13:07.367602+0000","last_peered":"2024-09-16T05:13:07.367602+0000","last_clean":"2024-09-16T05:13:07.367602+0000","last_became_active":"2024-09-16T05:12:44.682320+0000","last_became_peered":"2024-09-16T05:12:44.682320+0000","last_unstale":"2024-09-16T05:13:07.367602+0000","last_undegraded":"2024-09-16T05:13:07.367602+0000","last_fullsized":"2024-09-16T05:13:07.367602+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'28","last_scrub_stamp":"2024-09-16T05:12:09.669066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.669066+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:26.371216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0057148169999999996,"stat_sum":{"num_bytes":4571382,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6199,"num_write":4,"num_write_kb":889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3180990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":721,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.718865+0000","last_change":"2024-09-16T05:12:13.922480+0000","last_active":"2024-09-16T05:13:09.718865+0000","last_peered":"2024-09-16T05:13:09.718865+0000","last_clean":"2024-09-16T05:13:09.718865+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:13:09.718865+0000","last_undegraded":"2024-09-16T05:13:09.718865+0000","last_fullsized":"2024-09-16T05:13:09.718865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.922401+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.922401+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.949519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":967,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.792643+0000","last_change":"2024-09-16T05:13:08.517375+0000","last_active":"2024-09-16T05:13:09.792643+0000","last_peered":"2024-09-16T05:13:09.792643+0000","last_clean":"2024-09-16T05:13:09.792643+0000","last_became_active":"2024-09-16T05:12:59.750592+0000","last_became_peered":"2024-09-16T05:12:59.750592+0000","last_unstale":"2024-09-16T05:13:09.792643+0000","last_undegraded":"2024-09-16T05:13:09.792643+0000","last_fullsized":"2024-09-16T05:13:09.792643+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:13:08.517287+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.517287+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:11.623191+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0.00052414099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":727,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.725622+0000","last_change":"2024-09-16T05:12:47.244164+0000","last_active":"2024-09-16T05:13:11.725622+0000","last_peered":"2024-09-16T05:13:11.725622+0000","last_clean":"2024-09-16T05:13:11.725622+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:13:11.725622+0000","last_undegraded":"2024-09-16T05:13:11.725622+0000","last_fullsized":"2024-09-16T05:13:11.725622+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:47.244082+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:47.244082+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:53.753278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"260'344","reported_seq":1106,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.726196+0000","last_change":"2024-09-16T05:13:11.726396+0000","last_active":"2024-09-16T05:13:11.726196+0000","last_peered":"2024-09-16T05:13:11.726196+0000","last_clean":"2024-09-16T05:13:11.726196+0000","last_became_active":"2024-09-16T05:12:59.751132+0000","last_became_peered":"2024-09-16T05:12:59.751132+0000","last_unstale":"2024-09-16T05:13:11.726196+0000","last_undegraded":"2024-09-16T05:13:11.726196+0000","last_fullsized":"2024-09-16T05:13:11.726196+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'336","last_scrub_stamp":"2024-09-16T05:13:06.619984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.619984+0000","objects_scrubbed":8,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:28.641723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0054068859999999996,"stat_sum":{"num_bytes":16460681,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":14421,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":719,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:13:10.868355+0000","last_change":"2024-09-16T05:12:13.374972+0000","last_active":"2024-09-16T05:13:10.868355+0000","last_peered":"2024-09-16T05:13:10.868355+0000","last_clean":"2024-09-16T05:13:10.868355+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:13:10.868355+0000","last_undegraded":"2024-09-16T05:13:10.868355+0000","last_fullsized":"2024-09-16T05:13:10.868355+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.374887+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.374887+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:25.238853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":924,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T05:13:08.694427+0000","last_change":"2024-09-16T05:13:06.465458+0000","last_active":"2024-09-16T05:13:08.694427+0000","last_peered":"2024-09-16T05:13:08.694427+0000","last_clean":"2024-09-16T05:13:08.694427+0000","last_became_active":"2024-09-16T05:12:59.748431+0000","last_became_peered":"2024-09-16T05:12:59.748431+0000","last_unstale":"2024-09-16T05:13:08.694427+0000","last_undegraded":"2024-09-16T05:13:08.694427+0000","last_fullsized":"2024-09-16T05:13:08.694427+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:13:06.465328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.465328+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:08.915368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001036997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":547,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T05:13:07.367264+0000","last_change":"2024-09-16T05:12:50.665960+0000","last_active":"2024-09-16T05:13:07.367264+0000","last_peered":"2024-09-16T05:13:07.367264+0000","last_clean":"2024-09-16T05:13:07.367264+0000","last_became_active":"2024-09-16T05:12:47.672164+0000","last_became_peered":"2024-09-16T05:12:47.672164+0000","last_unstale":"2024-09-16T05:13:07.367264+0000","last_undegraded":"2024-09-16T05:13:07.367264+0000","last_fullsized":"2024-09-16T05:13:07.367264+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:50.665816+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:50.665816+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:00.976998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":825,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.792668+0000","last_change":"2024-09-16T05:13:06.313093+0000","last_active":"2024-09-16T05:13:09.792668+0000","last_peered":"2024-09-16T05:13:09.792668+0000","last_clean":"2024-09-16T05:13:09.792668+0000","last_became_active":"2024-09-16T05:12:59.749293+0000","last_became_peered":"2024-09-16T05:12:59.749293+0000","last_unstale":"2024-09-16T05:13:09.792668+0000","last_undegraded":"2024-09-16T05:13:09.792668+0000","last_fullsized":"2024-09-16T05:13:09.792668+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:13:06.313006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.313006+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:28.528284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000519039,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":466,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.719142+0000","last_change":"2024-09-16T05:12:45.101784+0000","last_active":"2024-09-16T05:13:09.719142+0000","last_peered":"2024-09-16T05:13:09.719142+0000","last_clean":"2024-09-16T05:13:09.719142+0000","last_became_active":"2024-09-16T05:12:44.692856+0000","last_became_peered":"2024-09-16T05:12:44.692856+0000","last_unstale":"2024-09-16T05:13:09.719142+0000","last_undegraded":"2024-09-16T05:13:09.719142+0000","last_fullsized":"2024-09-16T05:13:09.719142+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:45.101705+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.101705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:51.855025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"254'430","reported_seq":1143,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.719161+0000","last_change":"2024-09-16T05:13:07.051341+0000","last_active":"2024-09-16T05:13:09.719161+0000","last_peered":"2024-09-16T05:13:09.719161+0000","last_clean":"2024-09-16T05:13:09.719161+0000","last_became_active":"2024-09-16T05:12:59.751828+0000","last_became_peered":"2024-09-16T05:12:59.751828+0000","last_unstale":"2024-09-16T05:13:09.719161+0000","last_undegraded":"2024-09-16T05:13:09.719161+0000","last_fullsized":"2024-09-16T05:13:09.719161+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'430","last_scrub_stamp":"2024-09-16T05:13:07.051261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.051261+0000","objects_scrubbed":5,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:24.142973+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026547680000000001,"stat_sum":{"num_bytes":13423956,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":470,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.718877+0000","last_change":"2024-09-16T05:12:44.751434+0000","last_active":"2024-09-16T05:13:11.718877+0000","last_peered":"2024-09-16T05:13:11.718877+0000","last_clean":"2024-09-16T05:13:11.718877+0000","last_became_active":"2024-09-16T05:12:44.684578+0000","last_became_peered":"2024-09-16T05:12:44.684578+0000","last_unstale":"2024-09-16T05:13:11.718877+0000","last_undegraded":"2024-09-16T05:13:11.718877+0000","last_fullsized":"2024-09-16T05:13:11.718877+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:44.751365+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:44.751365+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:58.531322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":751,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.792856+0000","last_change":"2024-09-16T05:13:07.299328+0000","last_active":"2024-09-16T05:13:09.792856+0000","last_peered":"2024-09-16T05:13:09.792856+0000","last_clean":"2024-09-16T05:13:09.792856+0000","last_became_active":"2024-09-16T05:12:59.759329+0000","last_became_peered":"2024-09-16T05:12:59.759329+0000","last_unstale":"2024-09-16T05:13:09.792856+0000","last_undegraded":"2024-09-16T05:13:09.792856+0000","last_fullsized":"2024-09-16T05:13:09.792856+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:13:07.299230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.299230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:25.387342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00046599100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":684,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.720859+0000","last_change":"2024-09-16T05:12:09.317163+0000","last_active":"2024-09-16T05:13:11.720859+0000","last_peered":"2024-09-16T05:13:11.720859+0000","last_clean":"2024-09-16T05:13:11.720859+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:13:11.720859+0000","last_undegraded":"2024-09-16T05:13:11.720859+0000","last_fullsized":"2024-09-16T05:13:11.720859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:09.317055+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.317055+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.261087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":180,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.722268+0000","last_change":"2024-09-16T05:13:11.722611+0000","last_active":"2024-09-16T05:13:11.722268+0000","last_peered":"2024-09-16T05:13:11.722268+0000","last_clean":"2024-09-16T05:13:11.722268+0000","last_became_active":"2024-09-16T05:12:59.746906+0000","last_became_peered":"2024-09-16T05:12:59.746906+0000","last_unstale":"2024-09-16T05:13:11.722268+0000","last_undegraded":"2024-09-16T05:13:11.722268+0000","last_fullsized":"2024-09-16T05:13:11.722268+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:05.895744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:05.895744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:11.740568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034076829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.0","version":"205'33","reported_seq":789,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.730894+0000","last_change":"2024-09-16T05:13:11.731174+0000","last_active":"2024-09-16T05:13:11.730894+0000","last_peered":"2024-09-16T05:13:11.730894+0000","last_clean":"2024-09-16T05:13:11.730894+0000","last_became_active":"2024-09-16T05:12:44.681562+0000","last_became_peered":"2024-09-16T05:12:44.681562+0000","last_unstale":"2024-09-16T05:13:11.730894+0000","last_undegraded":"2024-09-16T05:13:11.730894+0000","last_fullsized":"2024-09-16T05:13:11.730894+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:12:28.044321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.044321+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:44.181043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0049740679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":717,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T05:13:07.367533+0000","last_change":"2024-09-16T05:11:53.500880+0000","last_active":"2024-09-16T05:13:07.367533+0000","last_peered":"2024-09-16T05:13:07.367533+0000","last_clean":"2024-09-16T05:13:07.367533+0000","last_became_active":"2024-09-16T05:11:19.419149+0000","last_became_peered":"2024-09-16T05:11:19.419149+0000","last_unstale":"2024-09-16T05:13:07.367533+0000","last_undegraded":"2024-09-16T05:13:07.367533+0000","last_fullsized":"2024-09-16T05:13:07.367533+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:53.500734+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:53.500734+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:10.037163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":798,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.719143+0000","last_change":"2024-09-16T05:13:11.719436+0000","last_active":"2024-09-16T05:13:11.719143+0000","last_peered":"2024-09-16T05:13:11.719143+0000","last_clean":"2024-09-16T05:13:11.719143+0000","last_became_active":"2024-09-16T05:12:21.666300+0000","last_became_peered":"2024-09-16T05:12:21.666300+0000","last_unstale":"2024-09-16T05:13:11.719143+0000","last_undegraded":"2024-09-16T05:13:11.719143+0000","last_fullsized":"2024-09-16T05:13:11.719143+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:25.777434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:25.777434+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:48.741373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053326999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"1.2","version":"35'139","reported_seq":727,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.726333+0000","last_change":"2024-09-16T05:12:31.125618+0000","last_active":"2024-09-16T05:13:11.726333+0000","last_peered":"2024-09-16T05:13:11.726333+0000","last_clean":"2024-09-16T05:13:11.726333+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:13:11.726333+0000","last_undegraded":"2024-09-16T05:13:11.726333+0000","last_fullsized":"2024-09-16T05:13:11.726333+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:31.125520+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:31.125520+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.820470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":465,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.718843+0000","last_change":"2024-09-16T05:12:46.116747+0000","last_active":"2024-09-16T05:13:09.718843+0000","last_peered":"2024-09-16T05:13:09.718843+0000","last_clean":"2024-09-16T05:13:09.718843+0000","last_became_active":"2024-09-16T05:12:44.814612+0000","last_became_peered":"2024-09-16T05:12:44.814612+0000","last_unstale":"2024-09-16T05:13:09.718843+0000","last_undegraded":"2024-09-16T05:13:09.718843+0000","last_fullsized":"2024-09-16T05:13:09.718843+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:46.116628+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:46.116628+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.926144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":754,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:13:10.868502+0000","last_change":"2024-09-16T05:13:10.868797+0000","last_active":"2024-09-16T05:13:10.868502+0000","last_peered":"2024-09-16T05:13:10.868502+0000","last_clean":"2024-09-16T05:13:10.868502+0000","last_became_active":"2024-09-16T05:12:21.660508+0000","last_became_peered":"2024-09-16T05:12:21.660508+0000","last_unstale":"2024-09-16T05:13:10.868502+0000","last_undegraded":"2024-09-16T05:13:10.868502+0000","last_fullsized":"2024-09-16T05:13:10.868502+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:27.335469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.335469+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:38.209982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000554577,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"37","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":723,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.725944+0000","last_change":"2024-09-16T05:11:54.869494+0000","last_active":"2024-09-16T05:13:11.725944+0000","last_peered":"2024-09-16T05:13:11.725944+0000","last_clean":"2024-09-16T05:13:11.725944+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:13:11.725944+0000","last_undegraded":"2024-09-16T05:13:11.725944+0000","last_fullsized":"2024-09-16T05:13:11.725944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:54.869407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:54.869407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:18.625896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"259'38","reported_seq":782,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:13:10.869912+0000","last_change":"2024-09-16T05:13:10.870104+0000","last_active":"2024-09-16T05:13:10.869912+0000","last_peered":"2024-09-16T05:13:10.869912+0000","last_clean":"2024-09-16T05:13:10.869912+0000","last_became_active":"2024-09-16T05:12:21.663580+0000","last_became_peered":"2024-09-16T05:12:21.663580+0000","last_unstale":"2024-09-16T05:13:10.869912+0000","last_undegraded":"2024-09-16T05:13:10.869912+0000","last_fullsized":"2024-09-16T05:13:10.869912+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-09-16T05:12:26.380594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.380594+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.827865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019188740000000001,"stat_sum":{"num_bytes":7629707,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":16029,"num_write":6,"num_write_kb":1149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"37","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.8","version":"259'116","reported_seq":871,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.726973+0000","last_change":"2024-09-16T05:13:11.730092+0000","last_active":"2024-09-16T05:13:11.726973+0000","last_peered":"2024-09-16T05:13:11.726973+0000","last_clean":"2024-09-16T05:13:11.726973+0000","last_became_active":"2024-09-16T05:12:59.751630+0000","last_became_peered":"2024-09-16T05:12:59.751630+0000","last_unstale":"2024-09-16T05:13:11.726973+0000","last_undegraded":"2024-09-16T05:13:11.726973+0000","last_fullsized":"2024-09-16T05:13:11.726973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'114","last_scrub_stamp":"2024-09-16T05:13:06.516389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.516389+0000","objects_scrubbed":4,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:17.952674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0043097430000000004,"stat_sum":{"num_bytes":9019103,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":8046,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.c","version":"241'194","reported_seq":941,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.721042+0000","last_change":"2024-09-16T05:13:11.721290+0000","last_active":"2024-09-16T05:13:11.721042+0000","last_peered":"2024-09-16T05:13:11.721042+0000","last_clean":"2024-09-16T05:13:11.721042+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:13:11.721042+0000","last_undegraded":"2024-09-16T05:13:11.721042+0000","last_fullsized":"2024-09-16T05:13:11.721042+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:13:07.600154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.600154+0000","objects_scrubbed":2,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:22.856864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047691200000000002,"stat_sum":{"num_bytes":4974872,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.d","version":"258'320","reported_seq":1063,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-16T05:13:09.727434+0000","last_change":"2024-09-16T05:13:08.021942+0000","last_active":"2024-09-16T05:13:09.727434+0000","last_peered":"2024-09-16T05:13:09.727434+0000","last_clean":"2024-09-16T05:13:09.727434+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:13:09.727434+0000","last_undegraded":"2024-09-16T05:13:09.727434+0000","last_fullsized":"2024-09-16T05:13:09.727434+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'312","last_scrub_stamp":"2024-09-16T05:13:08.021858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.021858+0000","objects_scrubbed":6,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:20.164741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0048563770000000003,"stat_sum":{"num_bytes":12389198,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.e","version":"260'222","reported_seq":968,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.721371+0000","last_change":"2024-09-16T05:13:11.721674+0000","last_active":"2024-09-16T05:13:11.721371+0000","last_peered":"2024-09-16T05:13:11.721371+0000","last_clean":"2024-09-16T05:13:11.721371+0000","last_became_active":"2024-09-16T05:12:21.665948+0000","last_became_peered":"2024-09-16T05:12:21.665948+0000","last_unstale":"2024-09-16T05:13:11.721371+0000","last_undegraded":"2024-09-16T05:13:11.721371+0000","last_fullsized":"2024-09-16T05:13:11.721371+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'160","last_scrub_stamp":"2024-09-16T05:12:27.786933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.786933+0000","objects_scrubbed":2,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.524201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027014080000000002,"stat_sum":{"num_bytes":12000132,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":3687,"num_write":15,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"2.f","version":"254'229","reported_seq":966,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T05:13:08.694599+0000","last_change":"2024-09-16T05:13:05.685901+0000","last_active":"2024-09-16T05:13:08.694599+0000","last_peered":"2024-09-16T05:13:08.694599+0000","last_clean":"2024-09-16T05:13:08.694599+0000","last_became_active":"2024-09-16T05:12:21.662019+0000","last_became_peered":"2024-09-16T05:12:21.662019+0000","last_unstale":"2024-09-16T05:13:08.694599+0000","last_undegraded":"2024-09-16T05:13:08.694599+0000","last_fullsized":"2024-09-16T05:13:08.694599+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'165","last_scrub_stamp":"2024-09-16T05:12:27.494268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.494268+0000","objects_scrubbed":8,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.431028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.005372,"stat_sum":{"num_bytes":16538088,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":12,"num_read_kb":10192,"num_write":10,"num_write_kb":4169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":350,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T05:13:08.694428+0000","last_change":"2024-09-16T05:13:05.681152+0000","last_active":"2024-09-16T05:13:08.694428+0000","last_peered":"2024-09-16T05:13:08.694428+0000","last_clean":"2024-09-16T05:13:08.694428+0000","last_became_active":"2024-09-16T05:11:57.658263+0000","last_became_peered":"2024-09-16T05:11:57.658263+0000","last_unstale":"2024-09-16T05:13:08.694428+0000","last_undegraded":"2024-09-16T05:13:08.694428+0000","last_fullsized":"2024-09-16T05:13:08.694428+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:12:09.468817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.468817+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:22.244438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073192699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3c","length":"1"},{"start":"41","length":"2"},{"start":"4b","length":"1"}]},{"pgid":"2.11","version":"259'111","reported_seq":459,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-16T05:13:11.725680+0000","last_change":"2024-09-16T05:13:11.727367+0000","last_active":"2024-09-16T05:13:11.725680+0000","last_peered":"2024-09-16T05:13:11.725680+0000","last_clean":"2024-09-16T05:13:11.725680+0000","last_became_active":"2024-09-16T05:11:57.662582+0000","last_became_peered":"2024-09-16T05:11:57.662582+0000","last_unstale":"2024-09-16T05:13:11.725680+0000","last_undegraded":"2024-09-16T05:13:11.725680+0000","last_fullsized":"2024-09-16T05:13:11.725680+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'64","last_scrub_stamp":"2024-09-16T05:12:08.859612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.859612+0000","objects_scrubbed":3,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:27.512391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002628465,"stat_sum":{"num_bytes":9596764,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":27,"num_read_kb":28743,"num_write":7,"num_write_kb":5277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":46,"stat_sum":{"num_bytes":326230772,"num_objects":163,"num_object_clones":116,"num_object_copies":326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":14,"num_read":307,"num_read_kb":353770,"num_write":190,"num_write_kb":77318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":33595633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":457535488,"data_stored":583592356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7150,"ondisk_log_size":7150,"up":92,"acting":92,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089393,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97228,"kb_used_data":66040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93647924,"statfs":{"total":95995035648,"available":95895474176,"internally_reserved":0,"allocated":67624960,"data_stored":85893638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":356482285622,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514556,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101388,"kb_used_data":66032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93643764,"statfs":{"total":95995035648,"available":95891214336,"internally_reserved":0,"allocated":67616768,"data_stored":80911203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841598,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88760,"kb_used_data":57568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93656392,"statfs":{"total":95995035648,"available":95904145408,"internally_reserved":0,"allocated":58949632,"data_stored":67252560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103236,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86596,"kb_used_data":55408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93658556,"statfs":{"total":95995035648,"available":95906361344,"internally_reserved":0,"allocated":56737792,"data_stored":69169653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397579,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89292,"kb_used_data":58096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93655860,"statfs":{"total":95995035648,"available":95903600640,"internally_reserved":0,"allocated":59490304,"data_stored":71440585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757326,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43280,"kb_used_data":16248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93701872,"statfs":{"total":95995035648,"available":95950716928,"internally_reserved":0,"allocated":16637952,"data_stored":19660242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920852,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50148,"kb_used_data":18952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93695004,"statfs":{"total":95995035648,"available":95943684096,"internally_reserved":0,"allocated":19406848,"data_stored":23068669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85724,"kb_used_data":54536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93659428,"statfs":{"total":95995035648,"available":95907254272,"internally_reserved":0,"allocated":55844864,"data_stored":76200255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313311,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53580,"kb_used_data":18228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93691572,"statfs":{"total":95995035648,"available":95940169728,"internally_reserved":0,"allocated":18665472,"data_stored":26053133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47040,"kb_used_data":15844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93698112,"statfs":{"total":95995035648,"available":95946866688,"internally_reserved":0,"allocated":16224256,"data_stored":16070237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57344,"kb_used_data":21984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93687808,"statfs":{"total":95995035648,"available":95936315392,"internally_reserved":0,"allocated":22511616,"data_stored":31513059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19480576,"data_stored":29660540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13205504,"data_stored":14226179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14229504,"data_stored":21958871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52850688,"data_stored":74373119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16375808,"data_stored":21216150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13402112,"data_stored":17606475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":56434688,"data_stored":69570928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53694464,"data_stored":67308513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":55615488,"data_stored":65089283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":64585728,"data_stored":79058684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":33406976,"data_stored":41595568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":64253952,"data_stored":81928046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:13:13.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:13.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:13.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:13 smithi196 ceph-mon[30802]: 2.18 deep-scrub starts 2024-09-16T05:13:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:13 smithi196 ceph-mon[30802]: 2.18 deep-scrub ok 2024-09-16T05:13:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:13 smithi196 ceph-mon[30802]: pgmap v543: 57 pgs: 57 active+clean; 308 MiB data, 780 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:13 smithi196 ceph-mon[30802]: osdmap e261: 12 total, 12 up, 11 in 2024-09-16T05:13:13.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1769255834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:14.006 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:14.012 INFO:tasks.rados.rados.2.smithi204.stdout:748: expect (ObjNum 186 snap 47 seq_num 186) 2024-09-16T05:13:14.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:14.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:14.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:14.159 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:14.159 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:13 smithi204 ceph-mon[30933]: 2.18 deep-scrub starts 2024-09-16T05:13:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:13 smithi204 ceph-mon[30933]: 2.18 deep-scrub ok 2024-09-16T05:13:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:13 smithi204 ceph-mon[30933]: pgmap v543: 57 pgs: 57 active+clean; 308 MiB data, 780 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:13 smithi204 ceph-mon[30933]: osdmap e261: 12 total, 12 up, 11 in 2024-09-16T05:13:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1769255834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:14.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:14.180 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:14.181 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:14.201 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:14.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:14.202 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:14.202 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:14.222 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:14.243 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:14.266 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:14.267 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:14.295 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:14.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:14.305 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:14.305 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:14.324 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:14.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:14.356 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:14.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:14.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:14.401 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:14.401 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:14.423 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:14.423 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:14.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:14.444 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:14.444 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:14.448 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:14.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:14.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:14.572 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:14.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:14.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:14.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:14.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:13:14.739 INFO:tasks.rados.rados.2.smithi204.stdout:743: done (4 left) 2024-09-16T05:13:14.739 INFO:tasks.rados.rados.2.smithi204.stdout:744: done (3 left) 2024-09-16T05:13:14.739 INFO:tasks.rados.rados.2.smithi204.stdout:745: done (2 left) 2024-09-16T05:13:14.739 INFO:tasks.rados.rados.2.smithi204.stdout:748: done (1 left) 2024-09-16T05:13:14.740 INFO:tasks.rados.rados.2.smithi204.stdout:750: done (0 left) 2024-09-16T05:13:14.740 INFO:tasks.rados.rados.2.smithi204.stdout:751: snap_remove snap 77 2024-09-16T05:13:14.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:14.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:14.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:14.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:14.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:14.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:14.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:13:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:13:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: osdmap e262: 12 total, 12 up, 11 in 2024-09-16T05:13:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:13:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:13:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3981280413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09567810220094734]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0953200589489482]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09510666421346688]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0948953576507067]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09693328108236611]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09669195303398699]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09450645494740906]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0943576779499407]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09479627187094859]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09432461639494771]}]: dispatch 2024-09-16T05:13:14.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08709716796875]}]: dispatch 2024-09-16T05:13:14.986 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:14 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09493594842508182]}]: dispatch 2024-09-16T05:13:15.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:13:15.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:13:15.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:15.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: osdmap e262: 12 total, 12 up, 11 in 2024-09-16T05:13:15.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:13:15.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3981280413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09567810220094734]}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0953200589489482]}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09510666421346688]}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0948953576507067]}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09693328108236611]}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09669195303398699]}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09450645494740906]}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0943576779499407]}]: dispatch 2024-09-16T05:13:15.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09479627187094859]}]: dispatch 2024-09-16T05:13:15.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09432461639494771]}]: dispatch 2024-09-16T05:13:15.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08709716796875]}]: dispatch 2024-09-16T05:13:15.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:14 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09493594842508182]}]: dispatch 2024-09-16T05:13:15.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:13:15.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:15.067 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:15.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:15.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: osdmap e262: 12 total, 12 up, 11 in 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3981280413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09567810220094734]}]: dispatch 2024-09-16T05:13:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0953200589489482]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09510666421346688]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0948953576507067]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09693328108236611]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09669195303398699]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09450645494740906]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0943576779499407]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09479627187094859]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09432461639494771]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08709716796875]}]: dispatch 2024-09-16T05:13:15.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:14 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09493594842508182]}]: dispatch 2024-09-16T05:13:15.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:15.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:15.249 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:15.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:15.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:15.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:15.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:15.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:15.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:15.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:15.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:15.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:15.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:15.677 INFO:tasks.rados.rados.2.smithi204.stdout:751: done (0 left) 2024-09-16T05:13:15.677 INFO:tasks.rados.rados.2.smithi204.stdout:752: read oid 7 snap 51 2024-09-16T05:13:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:752: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:13:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:753: read oid 39 snap -1 2024-09-16T05:13:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:753: expect deleted 2024-09-16T05:13:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:754: copy_from oid 11 from oid 35 current snap is 81 2024-09-16T05:13:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:755: write (excl) oid 18 current snap is 81 2024-09-16T05:13:15.678 INFO:tasks.rados.rados.2.smithi204.stdout:755: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-16T05:13:15.685 INFO:tasks.rados.rados.2.smithi204.stdout:755: writing smithi20457400-18 from 440416 to 1217189 tid 1 2024-09-16T05:13:15.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:15.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:15.692 INFO:tasks.rados.rados.2.smithi204.stdout:755: writing smithi20457400-18 from 1686814 to 2418914 tid 2 2024-09-16T05:13:15.692 INFO:tasks.rados.rados.2.smithi204.stdout:756: read oid 40 snap -1 2024-09-16T05:13:15.692 INFO:tasks.rados.rados.2.smithi204.stdout:756: expect deleted 2024-09-16T05:13:15.693 INFO:tasks.rados.rados.2.smithi204.stdout:757: delete oid 38 current snap is 81 2024-09-16T05:13:15.693 INFO:tasks.rados.rados.2.smithi204.stdout:754: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:13:15.693 INFO:tasks.rados.rados.2.smithi204.stdout:754: finishing copy_from to smithi20457400-11 2024-09-16T05:13:15.693 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 328 (ObjNum 272 snap 75 seq_num 272) dirty exists 2024-09-16T05:13:15.693 INFO:tasks.rados.rados.2.smithi204.stdout:755: finishing write tid 1 to smithi20457400-18 2024-09-16T05:13:15.693 INFO:tasks.rados.rados.2.smithi204.stdout:752: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T05:13:15.698 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:15.698 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:15.780 INFO:tasks.rados.rados.2.smithi204.stdout:755: finishing write tid 2 to smithi20457400-18 2024-09-16T05:13:15.780 INFO:tasks.rados.rados.2.smithi204.stdout:755: finishing write tid 3 to smithi20457400-18 2024-09-16T05:13:15.780 INFO:tasks.rados.rados.2.smithi204.stdout:752: done (5 left) 2024-09-16T05:13:15.780 INFO:tasks.rados.rados.2.smithi204.stdout:753: done (4 left) 2024-09-16T05:13:15.780 INFO:tasks.rados.rados.2.smithi204.stdout:754: done (3 left) 2024-09-16T05:13:15.780 INFO:tasks.rados.rados.2.smithi204.stdout:756: done (2 left) 2024-09-16T05:13:15.780 INFO:tasks.rados.rados.2.smithi204.stdout:757: done (1 left) 2024-09-16T05:13:15.780 INFO:tasks.rados.rados.2.smithi204.stdout:758: snap_remove snap 72 2024-09-16T05:13:15.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:15.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:15.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:15.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:15.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: pgmap v546: 57 pgs: 57 active+clean; 314 MiB data, 780 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3981280413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09567810220094734]}]': finished 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0953200589489482]}]': finished 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09510666421346688]}]': finished 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0948953576507067]}]': finished 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09693328108236611]}]': finished 2024-09-16T05:13:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09669195303398699]}]': finished 2024-09-16T05:13:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09450645494740906]}]': finished 2024-09-16T05:13:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0943576779499407]}]': finished 2024-09-16T05:13:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09479627187094859]}]': finished 2024-09-16T05:13:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09432461639494771]}]': finished 2024-09-16T05:13:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08709716796875]}]': finished 2024-09-16T05:13:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09493594842508182]}]': finished 2024-09-16T05:13:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: osdmap e263: 12 total, 12 up, 11 in 2024-09-16T05:13:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:15 smithi196 ceph-mon[30802]: osdmap e264: 12 total, 12 up, 11 in 2024-09-16T05:13:16.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:16.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:16.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:16.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: pgmap v546: 57 pgs: 57 active+clean; 314 MiB data, 780 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3981280413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09567810220094734]}]': finished 2024-09-16T05:13:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0953200589489482]}]': finished 2024-09-16T05:13:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09510666421346688]}]': finished 2024-09-16T05:13:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0948953576507067]}]': finished 2024-09-16T05:13:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09693328108236611]}]': finished 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09669195303398699]}]': finished 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09450645494740906]}]': finished 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0943576779499407]}]': finished 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09479627187094859]}]': finished 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09432461639494771]}]': finished 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08709716796875]}]': finished 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09493594842508182]}]': finished 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: osdmap e263: 12 total, 12 up, 11 in 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:15 smithi032 ceph-mon[27388]: osdmap e264: 12 total, 12 up, 11 in 2024-09-16T05:13:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: pgmap v546: 57 pgs: 57 active+clean; 314 MiB data, 780 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3981280413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09567810220094734]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0953200589489482]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09510666421346688]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0948953576507067]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09693328108236611]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09669195303398699]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.09450645494740906]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0943576779499407]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09479627187094859]}]': finished 2024-09-16T05:13:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.09432461639494771]}]': finished 2024-09-16T05:13:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08709716796875]}]': finished 2024-09-16T05:13:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09493594842508182]}]': finished 2024-09-16T05:13:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: osdmap e263: 12 total, 12 up, 11 in 2024-09-16T05:13:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:15 smithi204 ceph-mon[30933]: osdmap e264: 12 total, 12 up, 11 in 2024-09-16T05:13:16.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:16.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:16.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:16.312 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:16.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:16.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:13:16.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:16.437 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:16.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:16.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:16.563 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:16.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:16.576 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:16.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:16.677 INFO:tasks.rados.rados.2.smithi204.stdout:758: done (1 left) 2024-09-16T05:13:16.677 INFO:tasks.rados.rados.2.smithi204.stdout:759: delete oid 15 current snap is 81 2024-09-16T05:13:16.678 INFO:tasks.rados.rados.2.smithi204.stdout:755: finishing write tid 4 to smithi20457400-18 2024-09-16T05:13:16.678 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 216 (ObjNum 288 snap 81 seq_num 288) dirty exists 2024-09-16T05:13:16.678 INFO:tasks.rados.rados.2.smithi204.stdout:755: left oid 18 (ObjNum 288 snap 81 seq_num 288) 2024-09-16T05:13:16.679 INFO:tasks.rados.rados.2.smithi204.stdout:755: done (1 left) 2024-09-16T05:13:16.679 INFO:tasks.rados.rados.2.smithi204.stdout:759: done (0 left) 2024-09-16T05:13:16.679 INFO:tasks.rados.rados.2.smithi204.stdout:760: copy_from oid 20 from oid 21 current snap is 81 2024-09-16T05:13:16.679 INFO:tasks.rados.rados.2.smithi204.stdout:761: snap_remove snap 76 2024-09-16T05:13:16.691 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:16.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:16.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:16.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:16.816 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:16.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:16.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:16.943 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:16.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:17.043 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:17.043 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:17.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:17.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:17.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:17.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:17.196 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:17.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:17.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:17.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:17.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:17.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:13:17.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:17.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:17.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:17.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:17.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:17.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:17.602 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:17.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:17.679 INFO:tasks.rados.rados.2.smithi204.stdout:761: done (1 left) 2024-09-16T05:13:17.679 INFO:tasks.rados.rados.2.smithi204.stdout:762: write oid 6 current snap is 81 2024-09-16T05:13:17.679 INFO:tasks.rados.rados.2.smithi204.stdout:760: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:13:17.679 INFO:tasks.rados.rados.2.smithi204.stdout:762: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-16T05:13:17.685 INFO:tasks.rados.rados.2.smithi204.stdout:762: writing smithi20457400-6 from 490837 to 1209710 tid 1 2024-09-16T05:13:17.692 INFO:tasks.rados.rados.2.smithi204.stdout:762: writing smithi20457400-6 from 1862260 to 2341687 tid 2 2024-09-16T05:13:17.695 INFO:tasks.rados.rados.2.smithi204.stdout:762: writing smithi20457400-6 from 2875032 to 2890837 tid 3 2024-09-16T05:13:17.695 INFO:tasks.rados.rados.2.smithi204.stdout:763: delete oid 31 current snap is 81 2024-09-16T05:13:17.695 INFO:tasks.rados.rados.2.smithi204.stdout:760: finishing copy_from to smithi20457400-20 2024-09-16T05:13:17.695 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 67 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:13:17.695 INFO:tasks.rados.rados.2.smithi204.stdout:762: finishing write tid 1 to smithi20457400-6 2024-09-16T05:13:17.695 INFO:tasks.rados.rados.2.smithi204.stdout:762: finishing write tid 2 to smithi20457400-6 2024-09-16T05:13:17.696 INFO:tasks.rados.rados.2.smithi204.stdout:762: finishing write tid 3 to smithi20457400-6 2024-09-16T05:13:17.696 INFO:tasks.rados.rados.2.smithi204.stdout:760: done (2 left) 2024-09-16T05:13:17.696 INFO:tasks.rados.rados.2.smithi204.stdout:763: done (1 left) 2024-09-16T05:13:17.697 INFO:tasks.rados.rados.2.smithi204.stdout:764: snap_create 2024-09-16T05:13:17.697 INFO:tasks.rados.rados.2.smithi204.stdout:762: finishing write tid 4 to smithi20457400-6 2024-09-16T05:13:17.697 INFO:tasks.rados.rados.2.smithi204.stdout:762: finishing write tid 5 to smithi20457400-6 2024-09-16T05:13:17.697 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 50 (ObjNum 291 snap 81 seq_num 291) dirty exists 2024-09-16T05:13:17.697 INFO:tasks.rados.rados.2.smithi204.stdout:762: left oid 6 (ObjNum 291 snap 81 seq_num 291) 2024-09-16T05:13:17.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:17.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:17.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:13:17.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:17.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:17.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:17.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:13:17.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:17.926 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:17 smithi032 ceph-mon[27388]: pgmap v549: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 313 MiB data, 774 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-16T05:13:17.926 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:17 smithi032 ceph-mon[27388]: osdmap e265: 12 total, 12 up, 11 in 2024-09-16T05:13:17.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:17.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:17.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:17 smithi196 ceph-mon[30802]: pgmap v549: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 313 MiB data, 774 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-16T05:13:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:17 smithi196 ceph-mon[30802]: osdmap e265: 12 total, 12 up, 11 in 2024-09-16T05:13:18.022 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:18.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:18.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:18.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:18.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:17 smithi204 ceph-mon[30933]: pgmap v549: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 313 MiB data, 774 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-09-16T05:13:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:17 smithi204 ceph-mon[30933]: osdmap e265: 12 total, 12 up, 11 in 2024-09-16T05:13:18.191 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:18.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:18.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:18.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:18.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:18.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:18.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:18.480 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:18.480 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:18.541 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:18.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:18.564 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:18.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:18.682 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:13:18.686 INFO:tasks.rados.rados.2.smithi204.stdout:762: done (1 left) 2024-09-16T05:13:18.686 INFO:tasks.rados.rados.2.smithi204.stdout:764: done (0 left) 2024-09-16T05:13:18.686 INFO:tasks.rados.rados.2.smithi204.stdout:765: write oid 30 current snap is 82 2024-09-16T05:13:18.686 INFO:tasks.rados.rados.2.smithi204.stdout:765: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-09-16T05:13:18.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:18.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:18.691 INFO:tasks.rados.rados.2.smithi204.stdout:765: writing smithi20457400-30 from 524451 to 993822 tid 1 2024-09-16T05:13:18.698 INFO:tasks.rados.rados.2.smithi204.stdout:765: writing smithi20457400-30 from 1579224 to 2062320 tid 2 2024-09-16T05:13:18.703 INFO:tasks.rados.rados.2.smithi204.stdout:765: writing smithi20457400-30 from 2529878 to 2924451 tid 3 2024-09-16T05:13:18.703 INFO:tasks.rados.rados.2.smithi204.stdout:766: read oid 44 snap -1 2024-09-16T05:13:18.703 INFO:tasks.rados.rados.2.smithi204.stdout:766: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:13:18.703 INFO:tasks.rados.rados.2.smithi204.stdout:767: snap_create 2024-09-16T05:13:18.703 INFO:tasks.rados.rados.2.smithi204.stdout:765: finishing write tid 1 to smithi20457400-30 2024-09-16T05:13:18.703 INFO:tasks.rados.rados.2.smithi204.stdout:765: finishing write tid 2 to smithi20457400-30 2024-09-16T05:13:18.705 INFO:tasks.rados.rados.2.smithi204.stdout:765: finishing write tid 3 to smithi20457400-30 2024-09-16T05:13:18.705 INFO:tasks.rados.rados.2.smithi204.stdout:765: finishing write tid 4 to smithi20457400-30 2024-09-16T05:13:18.705 INFO:tasks.rados.rados.2.smithi204.stdout:765: finishing write tid 5 to smithi20457400-30 2024-09-16T05:13:18.705 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 413 (ObjNum 293 snap 82 seq_num 293) dirty exists 2024-09-16T05:13:18.705 INFO:tasks.rados.rados.2.smithi204.stdout:765: left oid 30 (ObjNum 293 snap 82 seq_num 293) 2024-09-16T05:13:18.708 INFO:tasks.rados.rados.2.smithi204.stdout:766: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:13:18.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:18 smithi196 ceph-mon[30802]: osdmap e266: 12 total, 12 up, 11 in 2024-09-16T05:13:18.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2063766181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:18.766 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:18.766 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:18.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:18.789 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:18.789 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:18.809 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:18.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:18.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:18.830 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:18.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:18.872 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:18.873 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:18.894 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:18.894 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:18.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:18.915 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:18.915 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:18.916 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:18 smithi032 ceph-mon[27388]: osdmap e266: 12 total, 12 up, 11 in 2024-09-16T05:13:18.916 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2063766181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:18.931 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:13:18.931 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:13:18.931 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T05:13:18.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:13:18.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:18.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:18.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:18.959 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:18.959 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:18.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:18.981 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:19.002 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:19.002 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:19.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:19.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:19.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:19.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:13:19.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:18 smithi204 ceph-mon[30933]: osdmap e266: 12 total, 12 up, 11 in 2024-09-16T05:13:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2063766181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:19.178 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:19.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:19.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:19.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:19.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:19.356 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:19.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:19.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:19.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:19.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:19.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:19.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:19.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:19.689 INFO:tasks.rados.rados.2.smithi204.stdout:765: done (2 left) 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:766: done (1 left) 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:767: done (0 left) 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:768: read oid 14 snap -1 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:768: expect deleted 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:769: read oid 4 snap -1 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:769: expect (ObjNum 267 snap 75 seq_num 267) 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:770: copy_from oid 22 from oid 23 current snap is 83 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:771: write oid 40 current snap is 83 2024-09-16T05:13:19.690 INFO:tasks.rados.rados.2.smithi204.stdout:771: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-16T05:13:19.695 INFO:tasks.rados.rados.2.smithi204.stdout:771: writing smithi20457400-40 from 541258 to 1085878 tid 1 2024-09-16T05:13:19.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:19.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:19.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:19.701 INFO:tasks.rados.rados.2.smithi204.stdout:771: writing smithi20457400-40 from 1637706 to 2264460 tid 2 2024-09-16T05:13:19.705 INFO:tasks.rados.rados.2.smithi204.stdout:771: writing smithi20457400-40 from 2840948 to 2941258 tid 3 2024-09-16T05:13:19.705 INFO:tasks.rados.rados.2.smithi204.stdout:772: write (excl) oid 5 current snap is 83 2024-09-16T05:13:19.705 INFO:tasks.rados.rados.2.smithi204.stdout:769: expect (ObjNum 267 snap 75 seq_num 267) 2024-09-16T05:13:19.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:19 smithi196 ceph-mon[30802]: pgmap v552: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 313 MiB data, 782 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 368 KiB/s wr, 3 op/s 2024-09-16T05:13:19.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2063766181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:19.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:19 smithi196 ceph-mon[30802]: osdmap e267: 12 total, 12 up, 11 in 2024-09-16T05:13:19.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2666427459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:19.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:19.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:19.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:19.853 INFO:tasks.rados.rados.2.smithi204.stdout:770: finishing copy_from to smithi20457400-22 2024-09-16T05:13:19.853 INFO:tasks.rados.rados.2.smithi204.stdout:770: got expected ENOENT (src dne) 2024-09-16T05:13:19.853 INFO:tasks.rados.rados.2.smithi204.stdout:770: finishing copy_from racing read to smithi20457400-22 2024-09-16T05:13:19.853 INFO:tasks.rados.rados.2.smithi204.stdout:771: finishing write tid 1 to smithi20457400-40 2024-09-16T05:13:19.853 INFO:tasks.rados.rados.2.smithi204.stdout:771: finishing write tid 2 to smithi20457400-40 2024-09-16T05:13:19.854 INFO:tasks.rados.rados.2.smithi204.stdout:772: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-16T05:13:19.859 INFO:tasks.rados.rados.2.smithi204.stdout:772: writing smithi20457400-5 from 558065 to 1177934 tid 1 2024-09-16T05:13:19.864 INFO:tasks.rados.rados.2.smithi204.stdout:772: writing smithi20457400-5 from 1579835 to 2066600 tid 2 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:772: writing smithi20457400-5 from 2468371 to 2958065 tid 3 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:768: done (4 left) 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:769: done (3 left) 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:770: done (2 left) 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:773: rollback oid 27 current snap is 83 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 27 to 80 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:771: finishing write tid 3 to smithi20457400-40 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:771: finishing write tid 4 to smithi20457400-40 2024-09-16T05:13:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:771: finishing write tid 5 to smithi20457400-40 2024-09-16T05:13:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 271 (ObjNum 294 snap 83 seq_num 294) dirty exists 2024-09-16T05:13:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:771: left oid 40 (ObjNum 294 snap 83 seq_num 294) 2024-09-16T05:13:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:772: finishing write tid 1 to smithi20457400-5 2024-09-16T05:13:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:772: finishing write tid 2 to smithi20457400-5 2024-09-16T05:13:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:771: done (2 left) 2024-09-16T05:13:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:774: write oid 24 current snap is 83 2024-09-16T05:13:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:774: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-16T05:13:19.875 INFO:tasks.rados.rados.2.smithi204.stdout:774: writing smithi20457400-24 from 574872 to 1269990 tid 1 2024-09-16T05:13:19.884 INFO:tasks.rados.rados.2.smithi204.stdout:774: writing smithi20457400-24 from 2038317 to 2668740 tid 2 2024-09-16T05:13:19.884 INFO:tasks.rados.rados.2.smithi204.stdout:775: read oid 18 snap 71 2024-09-16T05:13:19.884 INFO:tasks.rados.rados.2.smithi204.stdout:775: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:776: copy_from oid 31 from oid 15 current snap is 83 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:773: finishing rollback tid 0 to smithi20457400-27 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:777: read oid 8 snap -1 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:777: expect (ObjNum 275 snap 75 seq_num 275) 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:778: copy_from oid 45 from oid 32 current snap is 83 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:773: finishing rollback tid 1 to smithi20457400-27 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:773: finishing rollback tid 2 to smithi20457400-27 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 122 (ObjNum 269 snap 75 seq_num 269) dirty exists 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:772: finishing write tid 3 to smithi20457400-5 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:773: done (6 left) 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:779: delete oid 3 current snap is 83 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:772: finishing write tid 4 to smithi20457400-5 2024-09-16T05:13:19.885 INFO:tasks.rados.rados.2.smithi204.stdout:772: finishing write tid 5 to smithi20457400-5 2024-09-16T05:13:19.886 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 186 (ObjNum 295 snap 83 seq_num 295) dirty exists 2024-09-16T05:13:19.886 INFO:tasks.rados.rados.2.smithi204.stdout:772: left oid 5 (ObjNum 295 snap 83 seq_num 295) 2024-09-16T05:13:19.886 INFO:tasks.rados.rados.2.smithi204.stdout:774: finishing write tid 1 to smithi20457400-24 2024-09-16T05:13:19.886 INFO:tasks.rados.rados.2.smithi204.stdout:772: done (6 left) 2024-09-16T05:13:19.886 INFO:tasks.rados.rados.2.smithi204.stdout:779: done (5 left) 2024-09-16T05:13:19.886 INFO:tasks.rados.rados.2.smithi204.stdout:780: rollback oid 17 current snap is 83 2024-09-16T05:13:19.886 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 17 to 37 2024-09-16T05:13:19.886 INFO:tasks.rados.rados.2.smithi204.stdout:781: snap_remove snap 81 2024-09-16T05:13:19.895 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:57 2.2 2:4e7d172c:::smithi20457400-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 156.73423858300001, 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045779080000000003, 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 57 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159038+0000", 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159039+0000", 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6640000000000001e-06 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159883+0000", 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084362100000000002 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159895+0000", 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2632999999999999e-05 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159903+0000", 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8900000000000007e-06 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159978+0000", 2024-09-16T05:13:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5143999999999997e-05 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159988+0000", 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.431e-06 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161080+0000", 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010917990000000001 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161168+0000", 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8140000000000007e-05 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.161283+0000", 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115513 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163508+0000", 2024-09-16T05:13:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022243050000000002 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163545+0000", 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7033000000000003e-05 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163578+0000", 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2804e-05 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163615+0000", 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7931e-05 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:58 2.2 2:4e7d172c:::smithi20457400-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 156.73335592500001, 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003854761, 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159920+0000", 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.159921+0000", 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2699999999999999e-07 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160047+0000", 2024-09-16T05:13:19.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012650399999999999 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160055+0000", 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6539999999999999e-06 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160059+0000", 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5059999999999997e-06 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160100+0000", 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0153e-05 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160104+0000", 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0600000000000001e-06 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162622+0000", 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025179540000000002 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162679+0000", 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7163999999999997e-05 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.162793+0000", 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114398 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163515+0000", 2024-09-16T05:13:19.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072178499999999996 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163692+0000", 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017674699999999999 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163727+0000", 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5337999999999999e-05 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163775+0000", 2024-09-16T05:13:19.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7970999999999999e-05 2024-09-16T05:13:19.910 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.910 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.910 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.910 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:59 2.2 2:4e7d172c:::smithi20457400-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:13:19.911 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:13:19.911 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 156.733207032, 2024-09-16T05:13:19.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038110150000000001, 2024-09-16T05:13:19.911 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.911 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.911 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.911 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 59 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160069+0000", 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160070+0000", 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4999999999999998e-07 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160072+0000", 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.88e-06 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160078+0000", 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.152e-06 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160081+0000", 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.422e-06 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160135+0000", 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3808999999999999e-05 2024-09-16T05:13:19.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160138+0000", 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5590000000000001e-06 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163139+0000", 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030010689999999999 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163192+0000", 2024-09-16T05:13:19.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2398000000000002e-05 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163275+0000", 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2825999999999996e-05 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163518+0000", 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024361200000000001 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163815+0000", 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029653299999999997 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163845+0000", 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0409e-05 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163880+0000", 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4895999999999999e-05 2024-09-16T05:13:19.917 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:60 2.2 2:4e7d172c:::smithi20457400-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 156.733177162, 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039136570000000001, 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.918 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 60 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160099+0000", 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6100000000000001e-07 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160103+0000", 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.162e-06 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160108+0000", 2024-09-16T05:13:19.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3279999999999996e-06 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160112+0000", 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3479999999999997e-06 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160157+0000", 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5141e-05 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:19.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.160161+0000", 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8589999999999998e-06 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163451+0000", 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032897220000000001 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163500+0000", 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8665000000000002e-05 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163911+0000", 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041112700000000001 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.163953+0000", 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2228000000000003e-05 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.164013+0000", 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9616e-05 2024-09-16T05:13:19.923 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:69 2.3 2:c018a787:::smithi20457400-18:head [write 685719~711939,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 156.70348657599999, 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002893503, 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.924 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 69 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189790+0000", 2024-09-16T05:13:19.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.189792+0000", 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1299999999999999e-06 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190294+0000", 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050236900000000001 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190307+0000", 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3315e-05 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190329+0000", 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2116999999999998e-05 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190394+0000", 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4177999999999994e-05 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190467+0000", 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3795999999999996e-05 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.190551+0000", 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3119000000000002e-05 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.191468+0000", 2024-09-16T05:13:19.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091758400000000004 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192622+0000", 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011534869999999999 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192656+0000", 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4520000000000002e-05 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.192683+0000", 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6888000000000001e-05 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:663 2.3 2:c018a787:::smithi20457400-18:head [delete] snapc 17=[16,15,14,12,f,b] ondisk+write+known_if_redirected+full_force+supports_pool_eio e150)", 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:21.673892+0000", 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 118.21938370399999, 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003266751, 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.930 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 663 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673892+0000", 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673892+0000", 2024-09-16T05:13:19.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673894+0000", 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6190000000000001e-06 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673903+0000", 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9649999999999997e-06 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673922+0000", 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9352e-05 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.673932+0000", 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.73e-06 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.674009+0000", 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6707999999999998e-05 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.674017+0000", 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0630000000000006e-06 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.675338+0000", 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001321053 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.675446+0000", 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108467 2024-09-16T05:13:19.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.675657+0000", 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021090099999999999 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.676435+0000", 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077779900000000003 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.677042+0000", 2024-09-16T05:13:19.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00060720499999999998 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.677106+0000", 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3399999999999996e-05 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:21.677159+0000", 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3489000000000002e-05 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:677 2.3 2:c018a787:::smithi20457400-18:18 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e152)", 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:23.691322+0000", 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 116.20195365799999, 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002335171, 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 677 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.937 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691322+0000", 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691322+0000", 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691323+0000", 2024-09-16T05:13:19.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.15e-07 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691325+0000", 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.742e-06 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691330+0000", 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6890000000000001e-06 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691334+0000", 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8020000000000002e-06 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691457+0000", 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012282999999999999 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.691463+0000", 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7610000000000004e-06 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.693543+0000", 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020804539999999998 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:23.693658+0000", 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114478 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.941 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(osd.1.13:2 2.3 2:c018a787:::smithi20457400-18:head [copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e159)", 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:30.653613+0000", 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.239663491, 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022538160000000001, 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653613+0000", 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653613+0000", 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653615+0000", 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6630000000000002e-06 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653630+0000", 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.485e-05 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653663+0000", 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2796000000000002e-05 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653679+0000", 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6175999999999999e-05 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653825+0000", 2024-09-16T05:13:19.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000145959 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.653832+0000", 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6989999999999997e-06 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.655674+0000", 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001842209 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:30.655866+0000", 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019246399999999999 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:869 2.2 2:4e7d172c:::smithi20457400-15:head [write 437015~721812] snapc 21=[21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 100.184658974, 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053129060000000001, 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.946 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 869 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708620+0000", 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0369999999999999e-06 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709599+0000", 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00097878799999999997 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709622+0000", 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2609999999999999e-05 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709633+0000", 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1654999999999999e-05 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709764+0000", 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131023 2024-09-16T05:13:19.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709772+0000", 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6109999999999996e-06 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711031+0000", 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001258753 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711097+0000", 2024-09-16T05:13:19.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6699000000000006e-05 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711246+0000", 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000148629 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.712976+0000", 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001729721 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713857+0000", 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00088113600000000003 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713898+0000", 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0843000000000002e-05 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713930+0000", 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2400999999999997e-05 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.952 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1023 2.2 2:4e7d172c:::smithi20457400-15:head [write 773155~459498,stat] snapc 29=[29,27,25,24,23,22,21,20,1f,15] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:53.703726+0000", 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 86.189550163000007, 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045541829999999998, 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.953 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1023 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.703726+0000", 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.703726+0000", 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.703728+0000", 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.097e-06 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704236+0000", 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050769499999999996 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704257+0000", 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1362e-05 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704265+0000", 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0199999999999994e-06 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704366+0000", 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100938 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.704373+0000", 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7530000000000004e-06 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.705649+0000", 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001276304 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.705772+0000", 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000122509 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.705912+0000", 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014048700000000001 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.707325+0000", 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001412459 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708200+0000", 2024-09-16T05:13:19.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087534099999999997 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708245+0000", 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5024000000000001e-05 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:53.708280+0000", 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5194000000000002e-05 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1378 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 207] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 57.303759808999999, 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018067625, 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1378 2024-09-16T05:13:19.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589519+0000", 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3460000000000001e-06 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589547+0000", 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8572999999999999e-05 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589577+0000", 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9822e-05 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589592+0000", 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4630000000000001e-05 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589745+0000", 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015316700000000001 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589921+0000", 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000175995 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.602330+0000", 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012408878 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605723+0000", 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033928000000000001 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607502+0000", 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017799140000000001 2024-09-16T05:13:19.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607546+0000", 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3130999999999998e-05 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607584+0000", 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8368999999999998e-05 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1379 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 57.303653005000001, 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015638332000000001, 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1379 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589624+0000", 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4e-07 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:19.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589629+0000", 2024-09-16T05:13:19.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.366e-06 2024-09-16T05:13:19.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589642+0000", 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2114e-05 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589660+0000", 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8039000000000001e-05 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.590086+0000", 2024-09-16T05:13:19.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042614600000000002 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.590119+0000", 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3395999999999997e-05 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605083+0000", 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014963611999999999 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605167+0000", 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4444999999999995e-05 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605261+0000", 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4273999999999999e-05 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1406 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 0] snapc 39=[39,38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-16T05:13:19.972 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.826720+0000", 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 57.066556192, 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051871499999999997, 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1406 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826720+0000", 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826720+0000", 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826721+0000", 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3289999999999999e-06 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826727+0000", 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9490000000000001e-06 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826741+0000", 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3592e-05 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826777+0000", 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6019000000000002e-05 2024-09-16T05:13:19.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826831+0000", 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4574000000000003e-05 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826895+0000", 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3492999999999997e-05 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.831907+0000", 2024-09-16T05:13:19.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050121940000000002 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1407 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e211)", 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.826794+0000", 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 57.066482276000002, 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052833339999999998, 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.977 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1407 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826794+0000", 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826794+0000", 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826794+0000", 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3599999999999999e-07 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826796+0000", 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8840000000000001e-06 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826802+0000", 2024-09-16T05:13:19.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0159999999999997e-06 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826806+0000", 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6569999999999999e-06 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.827022+0000", 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021612899999999999 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.827072+0000", 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9971e-05 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.831969+0000", 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048975010000000003 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.832017+0000", 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7790000000000002e-05 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.832077+0000", 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0050000000000003e-05 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1496 2.16 2:6be67188:::smithi20457400-46:1f [read 0~2857157,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:29.690192+0000", 2024-09-16T05:13:19.982 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 50.203084220999997, 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025524469999999998, 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1496 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690192+0000", 2024-09-16T05:13:19.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690192+0000", 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690194+0000", 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.716e-06 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690213+0000", 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.978e-05 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690233+0000", 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9235000000000001e-05 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690259+0000", 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5891999999999999e-05 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690328+0000", 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9937000000000005e-05 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690335+0000", 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5250000000000002e-06 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.691618+0000", 2024-09-16T05:13:19.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012828970000000001 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.691695+0000", 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7474999999999999e-05 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.692744+0000", 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010489900000000001 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1497 2.16 2:6be67188:::smithi20457400-46:1f [sparse-read 0~2024259] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:29.690282+0000", 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 50.202994422000003, 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002669243, 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1497 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.988 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690282+0000", 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690282+0000", 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690282+0000", 2024-09-16T05:13:19.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7400000000000002e-07 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690286+0000", 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1530000000000001e-06 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690297+0000", 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1232e-05 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690304+0000", 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3429999999999998e-06 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690401+0000", 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6612000000000003e-05 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690405+0000", 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3819999999999997e-06 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.692772+0000", 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023672720000000001 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.692836+0000", 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3151999999999994e-05 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.692951+0000", 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011532299999999999 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.992 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1498 2.16 2:6be67188:::smithi20457400-46:1f [read 0~2857157,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:29.690314+0000", 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 50.202962178999996, 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034889309999999998, 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.993 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1498 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690314+0000", 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690314+0000", 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690314+0000", 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3300000000000002e-07 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690318+0000", 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8340000000000001e-06 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690325+0000", 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6320000000000002e-06 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690331+0000", 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5940000000000001e-06 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690426+0000", 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5268999999999999e-05 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.690430+0000", 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0709999999999996e-06 2024-09-16T05:13:19.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.692976+0000", 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025461490000000002 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.693048+0000", 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2188999999999996e-05 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.693803+0000", 2024-09-16T05:13:19.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075465999999999999 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1591 2.16 2:6be67188:::smithi20457400-46:head [write 400066~716205,stat] snapc 43=[43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:38.963369+0000", 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.929907178000001, 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00362123, 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:19.998 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1591 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.963369+0000", 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.963369+0000", 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:19.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.963371+0000", 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.069e-06 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964201+0000", 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000829497 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:20.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964216+0000", 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5658000000000002e-05 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964225+0000", 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0110000000000006e-06 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964284+0000", 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9255000000000001e-05 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:20.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964365+0000", 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0859999999999995e-05 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964528+0000", 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016263599999999999 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.966462+0000", 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001934531 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.966900+0000", 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00043739199999999999 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.966943+0000", 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3251999999999999e-05 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.966990+0000", 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7068999999999999e-05 2024-09-16T05:13:20.003 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1701 2.16 2:6be67188:::smithi20457400-46:head [delete] snapc 47=[47,46,45,44,42,41,40,3f,3e,3c,3a,39,37,35,33,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e236)", 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:47.700668+0000", 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.192608139000001, 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030615180000000001, 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:20.004 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1701 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.700668+0000", 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.700668+0000", 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.700671+0000", 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2830000000000002e-06 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.700684+0000", 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2636999999999999e-05 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.700720+0000", 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6498999999999997e-05 2024-09-16T05:13:20.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.700738+0000", 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7883000000000001e-05 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.700909+0000", 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017081999999999999 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.700932+0000", 2024-09-16T05:13:20.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3054e-05 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.702209+0000", 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001276445 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.702314+0000", 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105251 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:13:20.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.702534+0000", 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022057700000000001 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.703294+0000", 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075993800000000004 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.703657+0000", 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036285199999999999 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.703694+0000", 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7156000000000001e-05 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.703730+0000", 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5123000000000002e-05 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1858 2.16 2:6be67188:::smithi20457400-46:head [delete] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e249)", 2024-09-16T05:13:20.010 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:00.671909+0000", 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.221367411999999, 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010133186000000001, 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1858 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671909+0000", 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671909+0000", 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671911+0000", 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9700000000000002e-06 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671929+0000", 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7847e-05 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671948+0000", 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9117999999999999e-05 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671957+0000", 2024-09-16T05:13:20.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4539999999999996e-06 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.672037+0000", 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9628000000000004e-05 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.672046+0000", 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6150000000000003e-06 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.679299+0000", 2024-09-16T05:13:20.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072529539999999998 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.679400+0000", 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100228 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682042+0000", 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002642372 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:20.015 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:20.016 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:20.016 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:13:20.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:19 smithi032 ceph-mon[27388]: pgmap v552: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 313 MiB data, 782 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 368 KiB/s wr, 3 op/s 2024-09-16T05:13:20.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2063766181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:20.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:19 smithi032 ceph-mon[27388]: osdmap e267: 12 total, 12 up, 11 in 2024-09-16T05:13:20.017 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2666427459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:20.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:20.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:20.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:20.160 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:20.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:20.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:19 smithi204 ceph-mon[30933]: pgmap v552: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 313 MiB data, 782 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 368 KiB/s wr, 3 op/s 2024-09-16T05:13:20.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2063766181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:19 smithi204 ceph-mon[30933]: osdmap e267: 12 total, 12 up, 11 in 2024-09-16T05:13:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2666427459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:20.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:20.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:20.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:20.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:20.409 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:20.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:20.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:20.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:20.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:20.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:20.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:20.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:20.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:13:20.678 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:13:20.695 INFO:tasks.rados.rados.2.smithi204.stdout:781: done (6 left) 2024-09-16T05:13:20.695 INFO:tasks.rados.rados.2.smithi204.stdout:782: write oid 7 current snap is 83 2024-09-16T05:13:20.695 INFO:tasks.rados.rados.2.smithi204.stdout:776: finishing copy_from to smithi20457400-31 2024-09-16T05:13:20.695 INFO:tasks.rados.rados.2.smithi204.stdout:776: got expected ENOENT (src dne) 2024-09-16T05:13:20.695 INFO:tasks.rados.rados.2.smithi204.stdout:782: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-16T05:13:20.701 INFO:tasks.rados.rados.2.smithi204.stdout:782: writing smithi20457400-7 from 608486 to 1170455 tid 1 2024-09-16T05:13:20.709 INFO:tasks.rados.rados.2.smithi204.stdout:782: writing smithi20457400-7 from 1871634 to 2505726 tid 2 2024-09-16T05:13:20.712 INFO:tasks.rados.rados.2.smithi204.stdout:782: writing smithi20457400-7 from 2950640 to 3008486 tid 3 2024-09-16T05:13:20.712 INFO:tasks.rados.rados.2.smithi204.stdout:783: read oid 12 snap -1 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:783: expect (ObjNum 285 snap 80 seq_num 285) 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:784: delete oid 47 current snap is 83 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:776: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:780: finishing rollback tid 0 to smithi20457400-17 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:780: finishing rollback tid 1 to smithi20457400-17 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 290 (ObjNum 1345 snap 0 seq_num 2965905320) dirty dne 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:774: finishing write tid 2 to smithi20457400-24 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:774: finishing write tid 3 to smithi20457400-24 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:774: finishing write tid 4 to smithi20457400-24 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 311 (ObjNum 296 snap 83 seq_num 296) dirty exists 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:774: left oid 24 (ObjNum 296 snap 83 seq_num 296) 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:778: finishing copy_from racing read to smithi20457400-45 2024-09-16T05:13:20.713 INFO:tasks.rados.rados.2.smithi204.stdout:778: finishing copy_from to smithi20457400-45 2024-09-16T05:13:20.714 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 319 (ObjNum 248 snap 69 seq_num 248) dirty exists 2024-09-16T05:13:20.714 INFO:tasks.rados.rados.2.smithi204.stdout:777: expect (ObjNum 275 snap 75 seq_num 275) 2024-09-16T05:13:20.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:13:20.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:20.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:20.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:20.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:20.882 INFO:tasks.rados.rados.2.smithi204.stdout:775: expect (ObjNum 88 snap 8 seq_num 88) 2024-09-16T05:13:20.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:20.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:20.973 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:20.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:20 smithi196 ceph-mon[30802]: osdmap e268: 12 total, 12 up, 11 in 2024-09-16T05:13:20.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:20 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:21.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:20 smithi032 ceph-mon[27388]: osdmap e268: 12 total, 12 up, 11 in 2024-09-16T05:13:21.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:20 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:21.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:21.030 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:21.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:21.128 INFO:tasks.rados.rados.2.smithi204.stdout:782: finishing write tid 1 to smithi20457400-7 2024-09-16T05:13:21.129 INFO:tasks.rados.rados.2.smithi204.stdout:782: finishing write tid 2 to smithi20457400-7 2024-09-16T05:13:21.129 INFO:tasks.rados.rados.2.smithi204.stdout:782: finishing write tid 3 to smithi20457400-7 2024-09-16T05:13:21.129 INFO:tasks.rados.rados.2.smithi204.stdout:782: finishing write tid 4 to smithi20457400-7 2024-09-16T05:13:21.129 INFO:tasks.rados.rados.2.smithi204.stdout:782: finishing write tid 5 to smithi20457400-7 2024-09-16T05:13:21.129 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 191 (ObjNum 298 snap 83 seq_num 298) dirty exists 2024-09-16T05:13:21.129 INFO:tasks.rados.rados.2.smithi204.stdout:782: left oid 7 (ObjNum 298 snap 83 seq_num 298) 2024-09-16T05:13:21.129 INFO:tasks.rados.rados.2.smithi204.stdout:783: expect (ObjNum 285 snap 80 seq_num 285) 2024-09-16T05:13:21.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:21.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:21.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:21.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:20 smithi204 ceph-mon[30933]: osdmap e268: 12 total, 12 up, 11 in 2024-09-16T05:13:21.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:20 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:774: done (8 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:775: done (7 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:776: done (6 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:777: done (5 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:778: done (4 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:780: done (3 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:782: done (2 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:783: done (1 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:784: done (0 left) 2024-09-16T05:13:21.222 INFO:tasks.rados.rados.2.smithi204.stdout:785: delete oid 13 current snap is 83 2024-09-16T05:13:21.223 INFO:tasks.rados.rados.2.smithi204.stdout:785: done (0 left) 2024-09-16T05:13:21.223 INFO:tasks.rados.rados.2.smithi204.stdout:786: write (excl) oid 31 current snap is 83 2024-09-16T05:13:21.223 INFO:tasks.rados.rados.2.smithi204.stdout:786: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-16T05:13:21.230 INFO:tasks.rados.rados.2.smithi204.stdout:786: writing smithi20457400-31 from 658907 to 1446623 tid 1 2024-09-16T05:13:21.236 INFO:tasks.rados.rados.2.smithi204.stdout:786: writing smithi20457400-31 from 1930727 to 2428499 tid 2 2024-09-16T05:13:21.239 INFO:tasks.rados.rados.2.smithi204.stdout:786: writing smithi20457400-31 from 3032909 to 3058907 tid 3 2024-09-16T05:13:21.239 INFO:tasks.rados.rados.2.smithi204.stdout:787: write oid 45 current snap is 83 2024-09-16T05:13:21.240 INFO:tasks.rados.rados.2.smithi204.stdout:787: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-16T05:13:21.245 INFO:tasks.rados.rados.2.smithi204.stdout:787: writing smithi20457400-45 from 675714 to 1138679 tid 1 2024-09-16T05:13:21.251 INFO:tasks.rados.rados.2.smithi204.stdout:787: writing smithi20457400-45 from 1589209 to 2230639 tid 2 2024-09-16T05:13:21.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:787: writing smithi20457400-45 from 2943979 to 3075714 tid 3 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:788: snap_create 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:786: finishing write tid 1 to smithi20457400-31 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:786: finishing write tid 2 to smithi20457400-31 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:786: finishing write tid 3 to smithi20457400-31 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:786: finishing write tid 4 to smithi20457400-31 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:786: finishing write tid 5 to smithi20457400-31 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 419 (ObjNum 301 snap 83 seq_num 301) dirty exists 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:786: left oid 31 (ObjNum 301 snap 83 seq_num 301) 2024-09-16T05:13:21.257 INFO:tasks.rados.rados.2.smithi204.stdout:787: finishing write tid 1 to smithi20457400-45 2024-09-16T05:13:21.258 INFO:tasks.rados.rados.2.smithi204.stdout:787: finishing write tid 2 to smithi20457400-45 2024-09-16T05:13:21.259 INFO:tasks.rados.rados.2.smithi204.stdout:787: finishing write tid 3 to smithi20457400-45 2024-09-16T05:13:21.259 INFO:tasks.rados.rados.2.smithi204.stdout:787: finishing write tid 4 to smithi20457400-45 2024-09-16T05:13:21.259 INFO:tasks.rados.rados.2.smithi204.stdout:787: finishing write tid 5 to smithi20457400-45 2024-09-16T05:13:21.259 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 323 (ObjNum 302 snap 83 seq_num 302) dirty exists 2024-09-16T05:13:21.259 INFO:tasks.rados.rados.2.smithi204.stdout:787: left oid 45 (ObjNum 302 snap 83 seq_num 302) 2024-09-16T05:13:21.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:21.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:21.315 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":555,"stamp":"2024-09-16T05:13:20.067746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":333893163,"num_objects":169,"num_object_clones":120,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":17,"num_read":367,"num_read_kb":427564,"num_write":204,"num_write_kb":85699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":33595633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8706,"ondisk_log_size":8706,"up":114,"acting":114,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":804688,"kb_used_data":449100,"kb_used_omap":17,"kb_used_meta":355310,"kb_avail":1030391984,"statfs":{"total":1055945392128,"available":1055121391616,"internally_reserved":0,"allocated":459878400,"data_stored":556486591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":363838329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6050520,"num_objects":2,"num_object_clones":2,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":20354,"num_write":2,"num_write_kb":5910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000781"},"pg_stats":[{"pgid":"2.2d","version":"245'306","reported_seq":1080,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.679114+0000","last_change":"2024-09-16T05:13:16.679289+0000","last_active":"2024-09-16T05:13:16.679114+0000","last_peered":"2024-09-16T05:13:16.679114+0000","last_clean":"2024-09-16T05:13:16.679114+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:13:16.679114+0000","last_undegraded":"2024-09-16T05:13:16.679114+0000","last_fullsized":"2024-09-16T05:13:16.679114+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'306","last_scrub_stamp":"2024-09-16T05:13:10.471980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:10.471980+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:29.711188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0051002929999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.2c","version":"241'194","reported_seq":933,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-16T05:13:18.704349+0000","last_change":"2024-09-16T05:13:18.046049+0000","last_active":"2024-09-16T05:13:18.704349+0000","last_peered":"2024-09-16T05:13:18.704349+0000","last_clean":"2024-09-16T05:13:18.704349+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:13:18.704349+0000","last_undegraded":"2024-09-16T05:13:18.704349+0000","last_fullsized":"2024-09-16T05:13:18.704349+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:13:08.484521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.484521+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:13.899605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069742299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.2b","version":"260'182","reported_seq":970,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.689545+0000","last_change":"2024-09-16T05:13:15.689758+0000","last_active":"2024-09-16T05:13:15.689545+0000","last_peered":"2024-09-16T05:13:15.689545+0000","last_clean":"2024-09-16T05:13:15.689545+0000","last_became_active":"2024-09-16T05:13:04.678973+0000","last_became_peered":"2024-09-16T05:13:04.678973+0000","last_unstale":"2024-09-16T05:13:15.689545+0000","last_undegraded":"2024-09-16T05:13:15.689545+0000","last_fullsized":"2024-09-16T05:13:15.689545+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"253'179","last_scrub_stamp":"2024-09-16T05:13:06.865580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.865580+0000","objects_scrubbed":4,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:22.496223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058701700000000001,"stat_sum":{"num_bytes":2957999,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1100,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.674991+0000","last_change":"2024-09-16T05:13:16.675342+0000","last_active":"2024-09-16T05:13:16.674991+0000","last_peered":"2024-09-16T05:13:16.674991+0000","last_clean":"2024-09-16T05:13:16.674991+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:13:16.674991+0000","last_undegraded":"2024-09-16T05:13:16.674991+0000","last_fullsized":"2024-09-16T05:13:16.674991+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:13:08.516105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.516105+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:09.936070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013826299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.29","version":"265'258","reported_seq":1106,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.685890+0000","last_change":"2024-09-16T05:13:16.686066+0000","last_active":"2024-09-16T05:13:16.685890+0000","last_peered":"2024-09-16T05:13:16.685890+0000","last_clean":"2024-09-16T05:13:16.685890+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:13:16.685890+0000","last_undegraded":"2024-09-16T05:13:16.685890+0000","last_fullsized":"2024-09-16T05:13:16.685890+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"254'244","last_scrub_stamp":"2024-09-16T05:13:06.019617+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.019617+0000","objects_scrubbed":16,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:09.350594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0097838019999999994,"stat_sum":{"num_bytes":26600345,"num_objects":16,"num_object_clones":12,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.28","version":"245'105","reported_seq":892,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-16T05:13:17.682145+0000","last_change":"2024-09-16T05:13:17.682352+0000","last_active":"2024-09-16T05:13:17.682145+0000","last_peered":"2024-09-16T05:13:17.682145+0000","last_clean":"2024-09-16T05:13:17.682145+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:13:17.682145+0000","last_undegraded":"2024-09-16T05:13:17.682145+0000","last_fullsized":"2024-09-16T05:13:17.682145+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:13:06.496116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.496116+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:19.963956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002531789,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.27","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.678831+0000","last_change":"2024-09-16T05:13:16.679029+0000","last_active":"2024-09-16T05:13:16.678831+0000","last_peered":"2024-09-16T05:13:16.678831+0000","last_clean":"2024-09-16T05:13:16.678831+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:13:16.678831+0000","last_undegraded":"2024-09-16T05:13:16.678831+0000","last_fullsized":"2024-09-16T05:13:16.678831+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:09.483428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:09.483428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:24.701868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0048709649999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.26","version":"116'9","reported_seq":98,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.674521+0000","last_change":"2024-09-16T05:13:16.674947+0000","last_active":"2024-09-16T05:13:16.674521+0000","last_peered":"2024-09-16T05:13:16.674521+0000","last_clean":"2024-09-16T05:13:16.674521+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:13:16.674521+0000","last_undegraded":"2024-09-16T05:13:16.674521+0000","last_fullsized":"2024-09-16T05:13:16.674521+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:13:07.538943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.538943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:09.545848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001024383,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.25","version":"266'460","reported_seq":1217,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-16T05:13:18.704135+0000","last_change":"2024-09-16T05:13:18.048694+0000","last_active":"2024-09-16T05:13:18.704135+0000","last_peered":"2024-09-16T05:13:18.704135+0000","last_clean":"2024-09-16T05:13:18.704135+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:13:18.704135+0000","last_undegraded":"2024-09-16T05:13:18.704135+0000","last_fullsized":"2024-09-16T05:13:18.704135+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"254'432","last_scrub_stamp":"2024-09-16T05:13:07.464459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.464459+0000","objects_scrubbed":9,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.467186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0031041609999999998,"stat_sum":{"num_bytes":26834025,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":11,"num_read_kb":16048,"num_write":3,"num_write_kb":5910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.24","version":"182'89","reported_seq":84,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.738201+0000","last_change":"2024-09-16T05:13:14.057713+0000","last_active":"2024-09-16T05:13:14.738201+0000","last_peered":"2024-09-16T05:13:14.738201+0000","last_clean":"2024-09-16T05:13:14.738201+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:13:14.738201+0000","last_undegraded":"2024-09-16T05:13:14.738201+0000","last_fullsized":"2024-09-16T05:13:14.738201+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:13:06.032520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.032520+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:20.998946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044362399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":283,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-16T05:13:17.680385+0000","last_change":"2024-09-16T05:13:17.680633+0000","last_active":"2024-09-16T05:13:17.680385+0000","last_peered":"2024-09-16T05:13:17.680385+0000","last_clean":"2024-09-16T05:13:17.680385+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:13:17.680385+0000","last_undegraded":"2024-09-16T05:13:17.680385+0000","last_fullsized":"2024-09-16T05:13:17.680385+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:12:26.533655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.533655+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.983798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073755000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.22","version":"182'20","reported_seq":184,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-16T05:13:17.680042+0000","last_change":"2024-09-16T05:13:17.680324+0000","last_active":"2024-09-16T05:13:17.680042+0000","last_peered":"2024-09-16T05:13:17.680042+0000","last_clean":"2024-09-16T05:13:17.680042+0000","last_became_active":"2024-09-16T05:12:44.679281+0000","last_became_peered":"2024-09-16T05:12:44.679281+0000","last_unstale":"2024-09-16T05:13:17.680042+0000","last_undegraded":"2024-09-16T05:13:17.680042+0000","last_fullsized":"2024-09-16T05:13:17.680042+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:45.564449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.564449+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:54.053896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050531999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.21","version":"178'50","reported_seq":824,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.689916+0000","last_change":"2024-09-16T05:13:15.690599+0000","last_active":"2024-09-16T05:13:15.689916+0000","last_peered":"2024-09-16T05:13:15.689916+0000","last_clean":"2024-09-16T05:13:15.689916+0000","last_became_active":"2024-09-16T05:12:23.806187+0000","last_became_peered":"2024-09-16T05:12:23.806187+0000","last_unstale":"2024-09-16T05:13:15.689916+0000","last_undegraded":"2024-09-16T05:13:15.689916+0000","last_fullsized":"2024-09-16T05:13:15.689916+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:26.747969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.747969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:43.150514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00117424,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.20","version":"244'45","reported_seq":848,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.674070+0000","last_change":"2024-09-16T05:13:16.674457+0000","last_active":"2024-09-16T05:13:16.674070+0000","last_peered":"2024-09-16T05:13:16.674070+0000","last_clean":"2024-09-16T05:13:16.674070+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:13:16.674070+0000","last_undegraded":"2024-09-16T05:13:16.674070+0000","last_fullsized":"2024-09-16T05:13:16.674070+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"214'37","last_scrub_stamp":"2024-09-16T05:12:27.064860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.064860+0000","objects_scrubbed":3,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:50.277436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059264599999999995,"stat_sum":{"num_bytes":7916920,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":24943,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.1f","version":"262'193","reported_seq":977,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.871565+0000","last_change":"2024-09-16T05:13:13.750032+0000","last_active":"2024-09-16T05:13:14.871565+0000","last_peered":"2024-09-16T05:13:14.871565+0000","last_clean":"2024-09-16T05:13:14.871565+0000","last_became_active":"2024-09-16T05:12:26.664097+0000","last_became_peered":"2024-09-16T05:12:26.664097+0000","last_unstale":"2024-09-16T05:13:14.871565+0000","last_undegraded":"2024-09-16T05:13:14.871565+0000","last_fullsized":"2024-09-16T05:13:14.871565+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'148","last_scrub_stamp":"2024-09-16T05:12:27.189264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.189264+0000","objects_scrubbed":2,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:31.966595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015795379999999999,"stat_sum":{"num_bytes":9434476,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":53749,"num_write":7,"num_write_kb":1697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"2.1e","version":"266'267","reported_seq":1062,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-16T05:13:17.681623+0000","last_change":"2024-09-16T05:13:17.682070+0000","last_active":"2024-09-16T05:13:17.681623+0000","last_peered":"2024-09-16T05:13:17.681623+0000","last_clean":"2024-09-16T05:13:17.681623+0000","last_became_active":"2024-09-16T05:12:23.805930+0000","last_became_peered":"2024-09-16T05:12:23.805930+0000","last_unstale":"2024-09-16T05:13:17.681623+0000","last_undegraded":"2024-09-16T05:13:17.681623+0000","last_fullsized":"2024-09-16T05:13:17.681623+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'171","last_scrub_stamp":"2024-09-16T05:12:27.537258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.537258+0000","objects_scrubbed":7,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:33.286725+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002182963,"stat_sum":{"num_bytes":17252417,"num_objects":11,"num_object_clones":9,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":10,"num_read_kb":5496,"num_write":17,"num_write_kb":5081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.1d","version":"265'407","reported_seq":1227,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.688653+0000","last_change":"2024-09-16T05:13:16.688899+0000","last_active":"2024-09-16T05:13:16.688653+0000","last_peered":"2024-09-16T05:13:16.688653+0000","last_clean":"2024-09-16T05:13:16.688653+0000","last_became_active":"2024-09-16T05:12:44.698111+0000","last_became_peered":"2024-09-16T05:12:44.698111+0000","last_unstale":"2024-09-16T05:13:16.688653+0000","last_undegraded":"2024-09-16T05:13:16.688653+0000","last_fullsized":"2024-09-16T05:13:16.688653+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'276","last_scrub_stamp":"2024-09-16T05:12:29.635450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:29.635450+0000","objects_scrubbed":9,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:59.480363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014539504,"stat_sum":{"num_bytes":24741139,"num_objects":15,"num_object_clones":11,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":24,"num_read_kb":23289,"num_write":23,"num_write_kb":9802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":16830963,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.1c","version":"265'307","reported_seq":1112,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.680302+0000","last_change":"2024-09-16T05:13:16.680694+0000","last_active":"2024-09-16T05:13:16.680302+0000","last_peered":"2024-09-16T05:13:16.680302+0000","last_clean":"2024-09-16T05:13:16.680302+0000","last_became_active":"2024-09-16T05:12:44.684658+0000","last_became_peered":"2024-09-16T05:12:44.684658+0000","last_unstale":"2024-09-16T05:13:16.680302+0000","last_undegraded":"2024-09-16T05:13:16.680302+0000","last_fullsized":"2024-09-16T05:13:16.680302+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'202","last_scrub_stamp":"2024-09-16T05:12:28.627965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.627965+0000","objects_scrubbed":8,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.961011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0065720120000000003,"stat_sum":{"num_bytes":23277421,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":18,"num_read_kb":9440,"num_write":29,"num_write_kb":8506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13583680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.1b","version":"265'241","reported_seq":1053,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.679310+0000","last_change":"2024-09-16T05:13:16.679681+0000","last_active":"2024-09-16T05:13:16.679310+0000","last_peered":"2024-09-16T05:13:16.679310+0000","last_clean":"2024-09-16T05:13:16.679310+0000","last_became_active":"2024-09-16T05:12:23.807872+0000","last_became_peered":"2024-09-16T05:12:23.807872+0000","last_unstale":"2024-09-16T05:13:16.679310+0000","last_undegraded":"2024-09-16T05:13:16.679310+0000","last_fullsized":"2024-09-16T05:13:16.679310+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"214'162","last_scrub_stamp":"2024-09-16T05:12:26.326323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.326323+0000","objects_scrubbed":7,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:45.971514+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0052229310000000001,"stat_sum":{"num_bytes":15505131,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":21,"num_read_kb":20330,"num_write":13,"num_write_kb":5770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.1a","version":"265'380","reported_seq":1159,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-16T05:13:18.704293+0000","last_change":"2024-09-16T05:13:18.046415+0000","last_active":"2024-09-16T05:13:18.704293+0000","last_peered":"2024-09-16T05:13:18.704293+0000","last_clean":"2024-09-16T05:13:18.704293+0000","last_became_active":"2024-09-16T05:12:27.664055+0000","last_became_peered":"2024-09-16T05:12:27.664055+0000","last_unstale":"2024-09-16T05:13:18.704293+0000","last_undegraded":"2024-09-16T05:13:18.704293+0000","last_fullsized":"2024-09-16T05:13:18.704293+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'278","last_scrub_stamp":"2024-09-16T05:12:28.451582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.451582+0000","objects_scrubbed":11,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:32.446574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00094264100000000003,"stat_sum":{"num_bytes":27634014,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":31,"num_read_kb":25203,"num_write":16,"num_write_kb":3152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.19","version":"182'60","reported_seq":405,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-16T05:13:17.682562+0000","last_change":"2024-09-16T05:13:17.683090+0000","last_active":"2024-09-16T05:13:17.682562+0000","last_peered":"2024-09-16T05:13:17.682562+0000","last_clean":"2024-09-16T05:13:17.682562+0000","last_became_active":"2024-09-16T05:11:57.658646+0000","last_became_peered":"2024-09-16T05:11:57.658646+0000","last_unstale":"2024-09-16T05:13:17.682562+0000","last_undegraded":"2024-09-16T05:13:17.682562+0000","last_fullsized":"2024-09-16T05:13:17.682562+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:12:09.470099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.470099+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002895052,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.18","version":"181'57","reported_seq":787,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.737970+0000","last_change":"2024-09-16T05:13:14.058164+0000","last_active":"2024-09-16T05:13:14.737970+0000","last_peered":"2024-09-16T05:13:14.737970+0000","last_clean":"2024-09-16T05:13:14.737970+0000","last_became_active":"2024-09-16T05:11:57.660350+0000","last_became_peered":"2024-09-16T05:11:57.660350+0000","last_unstale":"2024-09-16T05:13:14.737970+0000","last_undegraded":"2024-09-16T05:13:14.737970+0000","last_fullsized":"2024-09-16T05:13:14.737970+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:13:12.039899+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:21.760193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079443000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"2.17","version":"0'0","reported_seq":585,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.676434+0000","last_change":"2024-09-16T05:13:16.676722+0000","last_active":"2024-09-16T05:13:16.676434+0000","last_peered":"2024-09-16T05:13:16.676434+0000","last_clean":"2024-09-16T05:13:16.676434+0000","last_became_active":"2024-09-16T05:12:44.679525+0000","last_became_peered":"2024-09-16T05:12:44.679525+0000","last_unstale":"2024-09-16T05:13:16.676434+0000","last_undegraded":"2024-09-16T05:13:16.676434+0000","last_fullsized":"2024-09-16T05:13:16.676434+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:12:09.877069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.877069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:25.854249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051755200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.16","version":"260'55","reported_seq":459,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.687490+0000","last_change":"2024-09-16T05:13:15.687633+0000","last_active":"2024-09-16T05:13:15.687490+0000","last_peered":"2024-09-16T05:13:15.687490+0000","last_clean":"2024-09-16T05:13:15.687490+0000","last_became_active":"2024-09-16T05:11:57.656772+0000","last_became_peered":"2024-09-16T05:11:57.656772+0000","last_unstale":"2024-09-16T05:13:15.687490+0000","last_undegraded":"2024-09-16T05:13:15.687490+0000","last_fullsized":"2024-09-16T05:13:15.687490+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"260'55","last_scrub_stamp":"2024-09-16T05:13:13.991451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:13.991451+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:17.444305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049024599999999995,"stat_sum":{"num_bytes":4044872,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":21,"num_read_kb":26292,"num_write":10,"num_write_kb":2494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.15","version":"262'375","reported_seq":1135,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.738399+0000","last_change":"2024-09-16T05:13:14.059843+0000","last_active":"2024-09-16T05:13:14.738399+0000","last_peered":"2024-09-16T05:13:14.738399+0000","last_clean":"2024-09-16T05:13:14.738399+0000","last_became_active":"2024-09-16T05:12:00.711890+0000","last_became_peered":"2024-09-16T05:12:00.711890+0000","last_unstale":"2024-09-16T05:13:14.738399+0000","last_undegraded":"2024-09-16T05:13:14.738399+0000","last_fullsized":"2024-09-16T05:13:14.738399+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'276","last_scrub_stamp":"2024-09-16T05:12:11.920515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:11.920515+0000","objects_scrubbed":7,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.170025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022098450000000002,"stat_sum":{"num_bytes":16654226,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":37,"num_read_kb":56083,"num_write":25,"num_write_kb":13520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"2.14","version":"260'184","reported_seq":532,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.738304+0000","last_change":"2024-09-16T05:13:14.058065+0000","last_active":"2024-09-16T05:13:14.738304+0000","last_peered":"2024-09-16T05:13:14.738304+0000","last_clean":"2024-09-16T05:13:14.738304+0000","last_became_active":"2024-09-16T05:11:57.658852+0000","last_became_peered":"2024-09-16T05:11:57.658852+0000","last_unstale":"2024-09-16T05:13:14.738304+0000","last_undegraded":"2024-09-16T05:13:14.738304+0000","last_fullsized":"2024-09-16T05:13:14.738304+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'108","last_scrub_stamp":"2024-09-16T05:12:10.924782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:10.924782+0000","objects_scrubbed":6,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:39.081546+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00072459399999999995,"stat_sum":{"num_bytes":22202598,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":47512,"num_write":8,"num_write_kb":7733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"2.13","version":"181'15","reported_seq":394,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.738215+0000","last_change":"2024-09-16T05:13:14.058404+0000","last_active":"2024-09-16T05:13:14.738215+0000","last_peered":"2024-09-16T05:13:14.738215+0000","last_clean":"2024-09-16T05:13:14.738215+0000","last_became_active":"2024-09-16T05:11:57.660401+0000","last_became_peered":"2024-09-16T05:11:57.660401+0000","last_unstale":"2024-09-16T05:13:14.738215+0000","last_undegraded":"2024-09-16T05:13:14.738215+0000","last_fullsized":"2024-09-16T05:13:14.738215+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:13:13.052418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:13.052418+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:18.771555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001023585,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"2.12","version":"266'67","reported_seq":420,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-16T05:13:17.682816+0000","last_change":"2024-09-16T05:13:17.683040+0000","last_active":"2024-09-16T05:13:17.682816+0000","last_peered":"2024-09-16T05:13:17.682816+0000","last_clean":"2024-09-16T05:13:17.682816+0000","last_became_active":"2024-09-16T05:12:44.682320+0000","last_became_peered":"2024-09-16T05:12:44.682320+0000","last_unstale":"2024-09-16T05:13:17.682816+0000","last_undegraded":"2024-09-16T05:13:17.682816+0000","last_fullsized":"2024-09-16T05:13:17.682816+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'28","last_scrub_stamp":"2024-09-16T05:12:09.669066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.669066+0000","objects_scrubbed":3,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:32.744579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031304200000000001,"stat_sum":{"num_bytes":4571382,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":6199,"num_write":5,"num_write_kb":889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3180990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"1.a","version":"35'139","reported_seq":731,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.737932+0000","last_change":"2024-09-16T05:12:13.922480+0000","last_active":"2024-09-16T05:13:14.737932+0000","last_peered":"2024-09-16T05:13:14.737932+0000","last_clean":"2024-09-16T05:13:14.737932+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:13:14.737932+0000","last_undegraded":"2024-09-16T05:13:14.737932+0000","last_fullsized":"2024-09-16T05:13:14.737932+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.922401+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.922401+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.949519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":998,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.734786+0000","last_change":"2024-09-16T05:13:14.060089+0000","last_active":"2024-09-16T05:13:14.734786+0000","last_peered":"2024-09-16T05:13:14.734786+0000","last_clean":"2024-09-16T05:13:14.734786+0000","last_became_active":"2024-09-16T05:12:59.750592+0000","last_became_peered":"2024-09-16T05:12:59.750592+0000","last_unstale":"2024-09-16T05:13:14.734786+0000","last_undegraded":"2024-09-16T05:13:14.734786+0000","last_fullsized":"2024-09-16T05:13:14.734786+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:13:08.517287+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.517287+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:14.853343+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0.000376731,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"1.9","version":"35'139","reported_seq":737,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.673759+0000","last_change":"2024-09-16T05:12:47.244164+0000","last_active":"2024-09-16T05:13:16.673759+0000","last_peered":"2024-09-16T05:13:16.673759+0000","last_clean":"2024-09-16T05:13:16.673759+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:13:16.673759+0000","last_undegraded":"2024-09-16T05:13:16.673759+0000","last_fullsized":"2024-09-16T05:13:16.673759+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:47.244082+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:47.244082+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:53.753278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"260'344","reported_seq":1138,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.675378+0000","last_change":"2024-09-16T05:13:16.675661+0000","last_active":"2024-09-16T05:13:16.675378+0000","last_peered":"2024-09-16T05:13:16.675378+0000","last_clean":"2024-09-16T05:13:16.675378+0000","last_became_active":"2024-09-16T05:12:59.751132+0000","last_became_peered":"2024-09-16T05:12:59.751132+0000","last_unstale":"2024-09-16T05:13:16.675378+0000","last_undegraded":"2024-09-16T05:13:16.675378+0000","last_fullsized":"2024-09-16T05:13:16.675378+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'336","last_scrub_stamp":"2024-09-16T05:13:06.619984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.619984+0000","objects_scrubbed":8,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:26.467176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011227769999999999,"stat_sum":{"num_bytes":16460681,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":14421,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"1.8","version":"35'139","reported_seq":729,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.687167+0000","last_change":"2024-09-16T05:12:13.374972+0000","last_active":"2024-09-16T05:13:15.687167+0000","last_peered":"2024-09-16T05:13:15.687167+0000","last_clean":"2024-09-16T05:13:15.687167+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:13:15.687167+0000","last_undegraded":"2024-09-16T05:13:15.687167+0000","last_fullsized":"2024-09-16T05:13:15.687167+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.374887+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.374887+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:25.238853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":982,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-16T05:13:18.704129+0000","last_change":"2024-09-16T05:13:18.045772+0000","last_active":"2024-09-16T05:13:18.704129+0000","last_peered":"2024-09-16T05:13:18.704129+0000","last_clean":"2024-09-16T05:13:18.704129+0000","last_became_active":"2024-09-16T05:12:59.748431+0000","last_became_peered":"2024-09-16T05:12:59.748431+0000","last_unstale":"2024-09-16T05:13:18.704129+0000","last_undegraded":"2024-09-16T05:13:18.704129+0000","last_fullsized":"2024-09-16T05:13:18.704129+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:13:06.465328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.465328+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:14.630122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043768399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"1.7","version":"35'139","reported_seq":568,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-16T05:13:17.679773+0000","last_change":"2024-09-16T05:12:50.665960+0000","last_active":"2024-09-16T05:13:17.679773+0000","last_peered":"2024-09-16T05:13:17.679773+0000","last_clean":"2024-09-16T05:13:17.679773+0000","last_became_active":"2024-09-16T05:12:47.672164+0000","last_became_peered":"2024-09-16T05:12:47.672164+0000","last_unstale":"2024-09-16T05:13:17.679773+0000","last_undegraded":"2024-09-16T05:13:17.679773+0000","last_fullsized":"2024-09-16T05:13:17.679773+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:50.665816+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:50.665816+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:00.976998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":856,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.734990+0000","last_change":"2024-09-16T05:13:14.060124+0000","last_active":"2024-09-16T05:13:14.734990+0000","last_peered":"2024-09-16T05:13:14.734990+0000","last_clean":"2024-09-16T05:13:14.734990+0000","last_became_active":"2024-09-16T05:12:59.749293+0000","last_became_peered":"2024-09-16T05:12:59.749293+0000","last_unstale":"2024-09-16T05:13:14.734990+0000","last_undegraded":"2024-09-16T05:13:14.734990+0000","last_fullsized":"2024-09-16T05:13:14.734990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:13:06.313006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.313006+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:24.102703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00038964400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":476,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.738196+0000","last_change":"2024-09-16T05:12:45.101784+0000","last_active":"2024-09-16T05:13:14.738196+0000","last_peered":"2024-09-16T05:13:14.738196+0000","last_clean":"2024-09-16T05:13:14.738196+0000","last_became_active":"2024-09-16T05:12:44.692856+0000","last_became_peered":"2024-09-16T05:12:44.692856+0000","last_unstale":"2024-09-16T05:13:14.738196+0000","last_undegraded":"2024-09-16T05:13:14.738196+0000","last_fullsized":"2024-09-16T05:13:14.738196+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:45.101705+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.101705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:51.855025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"260'434","reported_seq":1178,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.738219+0000","last_change":"2024-09-16T05:13:14.057862+0000","last_active":"2024-09-16T05:13:14.738219+0000","last_peered":"2024-09-16T05:13:14.738219+0000","last_clean":"2024-09-16T05:13:14.738219+0000","last_became_active":"2024-09-16T05:12:59.751828+0000","last_became_peered":"2024-09-16T05:12:59.751828+0000","last_unstale":"2024-09-16T05:13:14.738219+0000","last_undegraded":"2024-09-16T05:13:14.738219+0000","last_fullsized":"2024-09-16T05:13:14.738219+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'430","last_scrub_stamp":"2024-09-16T05:13:07.051261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.051261+0000","objects_scrubbed":5,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.061933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056202699999999999,"stat_sum":{"num_bytes":13423956,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":478,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.688956+0000","last_change":"2024-09-16T05:12:44.751434+0000","last_active":"2024-09-16T05:13:15.688956+0000","last_peered":"2024-09-16T05:13:15.688956+0000","last_clean":"2024-09-16T05:13:15.688956+0000","last_became_active":"2024-09-16T05:12:44.684578+0000","last_became_peered":"2024-09-16T05:12:44.684578+0000","last_unstale":"2024-09-16T05:13:15.688956+0000","last_undegraded":"2024-09-16T05:13:15.688956+0000","last_fullsized":"2024-09-16T05:13:15.688956+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:44.751365+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:44.751365+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:58.531322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":782,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.735133+0000","last_change":"2024-09-16T05:13:14.060343+0000","last_active":"2024-09-16T05:13:14.735133+0000","last_peered":"2024-09-16T05:13:14.735133+0000","last_clean":"2024-09-16T05:13:14.735133+0000","last_became_active":"2024-09-16T05:12:59.759329+0000","last_became_peered":"2024-09-16T05:12:59.759329+0000","last_unstale":"2024-09-16T05:13:14.735133+0000","last_undegraded":"2024-09-16T05:13:14.735133+0000","last_fullsized":"2024-09-16T05:13:14.735133+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:13:07.299230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.299230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:31.441077+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00054675800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"1.4","version":"35'139","reported_seq":694,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.674298+0000","last_change":"2024-09-16T05:12:09.317163+0000","last_active":"2024-09-16T05:13:16.674298+0000","last_peered":"2024-09-16T05:13:16.674298+0000","last_clean":"2024-09-16T05:13:16.674298+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:13:16.674298+0000","last_undegraded":"2024-09-16T05:13:16.674298+0000","last_fullsized":"2024-09-16T05:13:16.674298+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:09.317055+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.317055+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.261087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2,"reported_epoch":263,"state":"peering","last_fresh":"2024-09-16T05:13:13.724101+0000","last_change":"2024-09-16T05:13:15.473860+0000","last_active":"2024-09-16T05:13:15.473860+0000","last_peered":"2024-09-16T05:13:13.724101+0000","last_clean":"2024-09-16T05:13:13.724101+0000","last_became_active":"2024-09-16T05:12:59.746906+0000","last_became_peered":"2024-09-16T05:12:59.746906+0000","last_unstale":"2024-09-16T05:13:13.724101+0000","last_undegraded":"2024-09-16T05:13:13.724101+0000","last_fullsized":"2024-09-16T05:13:13.724101+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:05.895744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:05.895744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037818699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":820,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.678405+0000","last_change":"2024-09-16T05:13:16.678742+0000","last_active":"2024-09-16T05:13:16.678405+0000","last_peered":"2024-09-16T05:13:16.678405+0000","last_clean":"2024-09-16T05:13:16.678405+0000","last_became_active":"2024-09-16T05:12:44.681562+0000","last_became_peered":"2024-09-16T05:12:44.681562+0000","last_unstale":"2024-09-16T05:13:16.678405+0000","last_undegraded":"2024-09-16T05:13:16.678405+0000","last_fullsized":"2024-09-16T05:13:16.678405+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:12:28.044321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.044321+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:34.204889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0045994190000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"1.3","version":"0'0","reported_seq":1,"reported_epoch":263,"state":"unknown","last_fresh":"2024-09-16T05:13:14.734448+0000","last_change":"2024-09-16T05:13:15.130980+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T05:13:14.734448+0000","last_undegraded":"2024-09-16T05:13:14.734448+0000","last_fullsized":"2024-09-16T05:13:14.734448+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:10.386111+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:10.386111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":822,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.689813+0000","last_change":"2024-09-16T05:13:15.690565+0000","last_active":"2024-09-16T05:13:15.689813+0000","last_peered":"2024-09-16T05:13:15.689813+0000","last_clean":"2024-09-16T05:13:15.689813+0000","last_became_active":"2024-09-16T05:12:21.666300+0000","last_became_peered":"2024-09-16T05:12:21.666300+0000","last_unstale":"2024-09-16T05:13:15.689813+0000","last_undegraded":"2024-09-16T05:13:15.689813+0000","last_fullsized":"2024-09-16T05:13:15.689813+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:25.777434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:25.777434+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:42.099303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001276629,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"1.2","version":"35'139","reported_seq":737,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.673895+0000","last_change":"2024-09-16T05:12:31.125618+0000","last_active":"2024-09-16T05:13:16.673895+0000","last_peered":"2024-09-16T05:13:16.673895+0000","last_clean":"2024-09-16T05:13:16.673895+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:13:16.673895+0000","last_undegraded":"2024-09-16T05:13:16.673895+0000","last_fullsized":"2024-09-16T05:13:16.673895+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:31.125520+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:31.125520+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.820470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":475,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.737943+0000","last_change":"2024-09-16T05:12:46.116747+0000","last_active":"2024-09-16T05:13:14.737943+0000","last_peered":"2024-09-16T05:13:14.737943+0000","last_clean":"2024-09-16T05:13:14.737943+0000","last_became_active":"2024-09-16T05:12:44.814612+0000","last_became_peered":"2024-09-16T05:12:44.814612+0000","last_unstale":"2024-09-16T05:13:14.737943+0000","last_undegraded":"2024-09-16T05:13:14.737943+0000","last_fullsized":"2024-09-16T05:13:14.737943+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:46.116628+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:46.116628+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.926144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":785,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.687427+0000","last_change":"2024-09-16T05:13:15.687595+0000","last_active":"2024-09-16T05:13:15.687427+0000","last_peered":"2024-09-16T05:13:15.687427+0000","last_clean":"2024-09-16T05:13:15.687427+0000","last_became_active":"2024-09-16T05:12:21.660508+0000","last_became_peered":"2024-09-16T05:12:21.660508+0000","last_unstale":"2024-09-16T05:13:15.687427+0000","last_undegraded":"2024-09-16T05:13:15.687427+0000","last_fullsized":"2024-09-16T05:13:15.687427+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:27.335469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.335469+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:27.767915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044849899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"1.0","version":"35'139","reported_seq":733,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.674176+0000","last_change":"2024-09-16T05:11:54.869494+0000","last_active":"2024-09-16T05:13:16.674176+0000","last_peered":"2024-09-16T05:13:16.674176+0000","last_clean":"2024-09-16T05:13:16.674176+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:13:16.674176+0000","last_undegraded":"2024-09-16T05:13:16.674176+0000","last_fullsized":"2024-09-16T05:13:16.674176+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:11:54.869407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:11:54.869407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:18.625896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"264'44","reported_seq":819,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.695150+0000","last_change":"2024-09-16T05:13:15.689618+0000","last_active":"2024-09-16T05:13:15.695150+0000","last_peered":"2024-09-16T05:13:15.695150+0000","last_clean":"2024-09-16T05:13:15.695150+0000","last_became_active":"2024-09-16T05:12:21.663580+0000","last_became_peered":"2024-09-16T05:12:21.663580+0000","last_unstale":"2024-09-16T05:13:15.695150+0000","last_undegraded":"2024-09-16T05:13:15.695150+0000","last_fullsized":"2024-09-16T05:13:15.695150+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-09-16T05:12:26.380594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.380594+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:31.503416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022837410000000002,"stat_sum":{"num_bytes":9273036,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":16030,"num_write":10,"num_write_kb":2623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.8","version":"265'118","reported_seq":904,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.678148+0000","last_change":"2024-09-16T05:13:16.678340+0000","last_active":"2024-09-16T05:13:16.678148+0000","last_peered":"2024-09-16T05:13:16.678148+0000","last_clean":"2024-09-16T05:13:16.678148+0000","last_became_active":"2024-09-16T05:12:59.751630+0000","last_became_peered":"2024-09-16T05:12:59.751630+0000","last_unstale":"2024-09-16T05:13:16.678148+0000","last_undegraded":"2024-09-16T05:13:16.678148+0000","last_fullsized":"2024-09-16T05:13:16.678148+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'114","last_scrub_stamp":"2024-09-16T05:13:06.516389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.516389+0000","objects_scrubbed":4,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:18.391325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0044504779999999999,"stat_sum":{"num_bytes":9019103,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":8046,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.c","version":"241'194","reported_seq":972,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:16.674497+0000","last_change":"2024-09-16T05:13:16.674852+0000","last_active":"2024-09-16T05:13:16.674497+0000","last_peered":"2024-09-16T05:13:16.674497+0000","last_clean":"2024-09-16T05:13:16.674497+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:13:16.674497+0000","last_undegraded":"2024-09-16T05:13:16.674497+0000","last_fullsized":"2024-09-16T05:13:16.674497+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:13:07.600154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.600154+0000","objects_scrubbed":2,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:18.255198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000563129,"stat_sum":{"num_bytes":4974872,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.d","version":"262'330","reported_seq":1103,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-16T05:13:14.738406+0000","last_change":"2024-09-16T05:13:14.059625+0000","last_active":"2024-09-16T05:13:14.738406+0000","last_peered":"2024-09-16T05:13:14.738406+0000","last_clean":"2024-09-16T05:13:14.738406+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:13:14.738406+0000","last_undegraded":"2024-09-16T05:13:14.738406+0000","last_fullsized":"2024-09-16T05:13:14.738406+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'312","last_scrub_stamp":"2024-09-16T05:13:08.021858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.021858+0000","objects_scrubbed":6,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:20.503181+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0020151129999999998,"stat_sum":{"num_bytes":12389198,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"37","length":"1"},{"start":"39","length":"1"},{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4b","length":"1"},{"start":"50","length":"2"}]},{"pgid":"2.e","version":"260'222","reported_seq":992,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-16T05:13:15.689482+0000","last_change":"2024-09-16T05:13:15.689852+0000","last_active":"2024-09-16T05:13:15.689482+0000","last_peered":"2024-09-16T05:13:15.689482+0000","last_clean":"2024-09-16T05:13:15.689482+0000","last_became_active":"2024-09-16T05:12:21.665948+0000","last_became_peered":"2024-09-16T05:12:21.665948+0000","last_unstale":"2024-09-16T05:13:15.689482+0000","last_undegraded":"2024-09-16T05:13:15.689482+0000","last_fullsized":"2024-09-16T05:13:15.689482+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'160","last_scrub_stamp":"2024-09-16T05:12:27.786933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.786933+0000","objects_scrubbed":2,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:56.577639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078137499999999995,"stat_sum":{"num_bytes":12000132,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":3687,"num_write":15,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4f","length":"3"}]},{"pgid":"2.f","version":"265'235","reported_seq":1036,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-16T05:13:18.704901+0000","last_change":"2024-09-16T05:13:18.046173+0000","last_active":"2024-09-16T05:13:18.704901+0000","last_peered":"2024-09-16T05:13:18.704901+0000","last_clean":"2024-09-16T05:13:18.704901+0000","last_became_active":"2024-09-16T05:12:21.662019+0000","last_became_peered":"2024-09-16T05:12:21.662019+0000","last_unstale":"2024-09-16T05:13:18.704901+0000","last_undegraded":"2024-09-16T05:13:18.704901+0000","last_fullsized":"2024-09-16T05:13:18.704901+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'165","last_scrub_stamp":"2024-09-16T05:12:27.494268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.494268+0000","objects_scrubbed":8,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:50.028821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000740462,"stat_sum":{"num_bytes":16538088,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":30,"num_read_kb":42052,"num_write":10,"num_write_kb":4169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.10","version":"182'16","reported_seq":416,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-16T05:13:18.704116+0000","last_change":"2024-09-16T05:13:18.045766+0000","last_active":"2024-09-16T05:13:18.704116+0000","last_peered":"2024-09-16T05:13:18.704116+0000","last_clean":"2024-09-16T05:13:18.704116+0000","last_became_active":"2024-09-16T05:11:57.658263+0000","last_became_peered":"2024-09-16T05:11:57.658263+0000","last_unstale":"2024-09-16T05:13:18.704116+0000","last_undegraded":"2024-09-16T05:13:18.704116+0000","last_fullsized":"2024-09-16T05:13:18.704116+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:13:14.638370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:14.638370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:23.876454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044639499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"4"}]},{"pgid":"2.11","version":"265'113","reported_seq":492,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-16T05:13:17.336691+0000","last_change":"2024-09-16T05:13:17.336924+0000","last_active":"2024-09-16T05:13:17.336691+0000","last_peered":"2024-09-16T05:13:17.336691+0000","last_clean":"2024-09-16T05:13:17.336691+0000","last_became_active":"2024-09-16T05:11:57.662582+0000","last_became_peered":"2024-09-16T05:11:57.662582+0000","last_unstale":"2024-09-16T05:13:17.336691+0000","last_undegraded":"2024-09-16T05:13:17.336691+0000","last_fullsized":"2024-09-16T05:13:17.336691+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'64","last_scrub_stamp":"2024-09-16T05:12:08.859612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:08.859612+0000","objects_scrubbed":3,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.66075134599999996,"stat_sum":{"num_bytes":9596764,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":27,"num_read_kb":28743,"num_write":7,"num_write_kb":5277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"44","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":46,"stat_sum":{"num_bytes":333302795,"num_objects":167,"num_object_clones":120,"num_object_copies":334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":17,"num_read":367,"num_read_kb":427564,"num_write":204,"num_write_kb":85699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":33595633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":459612160,"data_stored":578014176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7316,"ondisk_log_size":7316,"up":92,"acting":92,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1390,"ondisk_log_size":1390,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089394,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101012,"kb_used_data":69824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93644140,"statfs":{"total":95995035648,"available":95891599360,"internally_reserved":0,"allocated":71499776,"data_stored":87970739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":356482285623,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514557,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98944,"kb_used_data":63592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93646208,"statfs":{"total":95995035648,"available":95893716992,"internally_reserved":0,"allocated":65118208,"data_stored":75561825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841600,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91960,"kb_used_data":60764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93653192,"statfs":{"total":95995035648,"available":95900868608,"internally_reserved":0,"allocated":62222336,"data_stored":70493477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103237,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90824,"kb_used_data":55468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93654328,"statfs":{"total":95995035648,"available":95902031872,"internally_reserved":0,"allocated":56799232,"data_stored":69213900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397580,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89348,"kb_used_data":58156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93655804,"statfs":{"total":95995035648,"available":95903543296,"internally_reserved":0,"allocated":59551744,"data_stored":71484853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43344,"kb_used_data":16308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93701808,"statfs":{"total":95995035648,"available":95950651392,"internally_reserved":0,"allocated":16699392,"data_stored":19704431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49032,"kb_used_data":17840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93696120,"statfs":{"total":95995035648,"available":95944826880,"internally_reserved":0,"allocated":18268160,"data_stored":20015421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117082,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84288,"kb_used_data":53100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93660864,"statfs":{"total":95995035648,"available":95908724736,"internally_reserved":0,"allocated":54374400,"data_stored":72821511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313312,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53960,"kb_used_data":18612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93691192,"statfs":{"total":95995035648,"available":95939780608,"internally_reserved":0,"allocated":19058688,"data_stored":25993953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44596,"kb_used_data":13404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93700556,"statfs":{"total":95995035648,"available":95949369344,"internally_reserved":0,"allocated":13725696,"data_stored":13553049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57380,"kb_used_data":22032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93687772,"statfs":{"total":95995035648,"available":95936278528,"internally_reserved":0,"allocated":22560768,"data_stored":29673432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19480576,"data_stored":27786709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10645504,"data_stored":11666326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15749120,"data_stored":23887318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51257344,"data_stored":70907586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15175680,"data_stored":18118713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14921728,"data_stored":19534922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":56434688,"data_stored":69570928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53694464,"data_stored":67308513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":58765312,"data_stored":68243342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":62025728,"data_stored":73665117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":33406976,"data_stored":41595568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":68055040,"data_stored":85729134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:13:21.318 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:13:21.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 56 2024-09-16T05:13:21.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:13:21.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:21.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:21.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:21.474 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:13:21.474 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:13:21.474 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:13:21.474 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:13:21.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:21.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:21.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:21.585 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:21.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:21.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:21.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:21.698 INFO:tasks.rados.rados.2.smithi204.stdout:786: done (2 left) 2024-09-16T05:13:21.698 INFO:tasks.rados.rados.2.smithi204.stdout:787: done (1 left) 2024-09-16T05:13:21.698 INFO:tasks.rados.rados.2.smithi204.stdout:788: done (0 left) 2024-09-16T05:13:21.698 INFO:tasks.rados.rados.2.smithi204.stdout:789: delete oid 8 current snap is 84 2024-09-16T05:13:21.701 INFO:tasks.rados.rados.2.smithi204.stdout:789: done (0 left) 2024-09-16T05:13:21.701 INFO:tasks.rados.rados.2.smithi204.stdout:790: write (excl) oid 46 current snap is 84 2024-09-16T05:13:21.701 INFO:tasks.rados.rados.2.smithi204.stdout:790: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-16T05:13:21.709 INFO:tasks.rados.rados.2.smithi204.stdout:790: writing smithi20457400-46 from 709328 to 1322791 tid 1 2024-09-16T05:13:21.716 INFO:tasks.rados.rados.2.smithi204.stdout:790: writing smithi20457400-46 from 1989820 to 2634919 tid 2 2024-09-16T05:13:21.716 INFO:tasks.rados.rados.2.smithi204.stdout:791: write oid 35 current snap is 84 2024-09-16T05:13:21.716 INFO:tasks.rados.rados.2.smithi204.stdout:791: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-16T05:13:21.722 INFO:tasks.rados.rados.2.smithi204.stdout:791: writing smithi20457400-35 from 726135 to 1131200 tid 1 2024-09-16T05:13:21.729 INFO:tasks.rados.rados.2.smithi204.stdout:791: writing smithi20457400-35 from 1764655 to 2553412 tid 2 2024-09-16T05:13:21.733 INFO:tasks.rados.rados.2.smithi204.stdout:791: writing smithi20457400-35 from 3026248 to 3126135 tid 3 2024-09-16T05:13:21.733 INFO:tasks.rados.rados.2.smithi204.stdout:792: copy_from oid 24 from oid 14 current snap is 84 2024-09-16T05:13:21.733 INFO:tasks.rados.rados.2.smithi204.stdout:790: finishing write tid 1 to smithi20457400-46 2024-09-16T05:13:21.733 INFO:tasks.rados.rados.2.smithi204.stdout:790: finishing write tid 2 to smithi20457400-46 2024-09-16T05:13:21.733 INFO:tasks.rados.rados.2.smithi204.stdout:790: finishing write tid 3 to smithi20457400-46 2024-09-16T05:13:21.733 INFO:tasks.rados.rados.2.smithi204.stdout:790: finishing write tid 4 to smithi20457400-46 2024-09-16T05:13:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 58 (ObjNum 304 snap 84 seq_num 304) dirty exists 2024-09-16T05:13:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:790: left oid 46 (ObjNum 304 snap 84 seq_num 304) 2024-09-16T05:13:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:791: finishing write tid 1 to smithi20457400-35 2024-09-16T05:13:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:791: finishing write tid 2 to smithi20457400-35 2024-09-16T05:13:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:790: done (2 left) 2024-09-16T05:13:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:793: snap_create 2024-09-16T05:13:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:791: finishing write tid 3 to smithi20457400-35 2024-09-16T05:13:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:791: finishing write tid 4 to smithi20457400-35 2024-09-16T05:13:21.735 INFO:tasks.rados.rados.2.smithi204.stdout:791: finishing write tid 5 to smithi20457400-35 2024-09-16T05:13:21.735 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 349 (ObjNum 305 snap 84 seq_num 305) dirty exists 2024-09-16T05:13:21.735 INFO:tasks.rados.rados.2.smithi204.stdout:791: left oid 35 (ObjNum 305 snap 84 seq_num 305) 2024-09-16T05:13:21.735 INFO:tasks.rados.rados.2.smithi204.stdout:792: finishing copy_from to smithi20457400-24 2024-09-16T05:13:21.735 INFO:tasks.rados.rados.2.smithi204.stdout:792: got expected ENOENT (src dne) 2024-09-16T05:13:21.735 INFO:tasks.rados.rados.2.smithi204.stdout:792: finishing copy_from racing read to smithi20457400-24 2024-09-16T05:13:21.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:21.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:21.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:21.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:21.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:21.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:21 smithi196 ceph-mon[30802]: pgmap v555: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 318 MiB data, 786 MiB used, 983 GiB / 983 GiB avail; 5.0 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2024-09-16T05:13:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:21 smithi196 ceph-mon[30802]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:21 smithi196 ceph-mon[30802]: osdmap e269: 12 total, 12 up, 11 in 2024-09-16T05:13:21.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:22.019 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:22.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:22.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:13:22.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:21 smithi032 ceph-mon[27388]: pgmap v555: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 318 MiB data, 786 MiB used, 983 GiB / 983 GiB avail; 5.0 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2024-09-16T05:13:22.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:21 smithi032 ceph-mon[27388]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:22.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:21 smithi032 ceph-mon[27388]: osdmap e269: 12 total, 12 up, 11 in 2024-09-16T05:13:22.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:13:22.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:22.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:22.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:21 smithi204 ceph-mon[30933]: pgmap v555: 57 pgs: 1 peering, 1 unknown, 55 active+clean; 318 MiB data, 786 MiB used, 983 GiB / 983 GiB avail; 5.0 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2024-09-16T05:13:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:21 smithi204 ceph-mon[30933]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:21 smithi204 ceph-mon[30933]: osdmap e269: 12 total, 12 up, 11 in 2024-09-16T05:13:22.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:22.271 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:22.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:22.326 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:22.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:22.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:22.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:22.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:22.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:22.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:22.643 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:22.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:22.697 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pg_num to 56 2024-09-16T05:13:22.699 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:22.701 INFO:tasks.rados.rados.2.smithi204.stdout:791: done (2 left) 2024-09-16T05:13:22.701 INFO:tasks.rados.rados.2.smithi204.stdout:792: done (1 left) 2024-09-16T05:13:22.701 INFO:tasks.rados.rados.2.smithi204.stdout:793: done (0 left) 2024-09-16T05:13:22.701 INFO:tasks.rados.rados.2.smithi204.stdout:794: snap_remove snap 69 2024-09-16T05:13:22.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:22.764 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:13:22.764 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:13:22.764 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:13:22.765 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:13:22.765 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:13:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:22.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:22.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:22.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:22.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:22 smithi032 ceph-mon[27388]: osdmap e270: 12 total, 12 up, 11 in 2024-09-16T05:13:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1373575552' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]: dispatch 2024-09-16T05:13:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:22 smithi196 ceph-mon[30802]: osdmap e270: 12 total, 12 up, 11 in 2024-09-16T05:13:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1373575552' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]: dispatch 2024-09-16T05:13:22.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:23.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:23.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:23.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:23.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:23.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:23.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:22 smithi204 ceph-mon[30933]: osdmap e270: 12 total, 12 up, 11 in 2024-09-16T05:13:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1373575552' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]: dispatch 2024-09-16T05:13:23.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:23.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:23.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:23.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:23.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:23.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:23.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:13:23.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:23.521 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:23.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:23.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:23.643 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:23.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:23.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:23.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:13:23.711 INFO:tasks.rados.rados.2.smithi204.stdout:794: done (0 left) 2024-09-16T05:13:23.712 INFO:tasks.rados.rados.2.smithi204.stdout:795: write oid 24 current snap is 85 2024-09-16T05:13:23.712 INFO:tasks.rados.rados.2.smithi204.stdout:795: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-16T05:13:23.719 INFO:tasks.rados.rados.2.smithi204.stdout:795: writing smithi20457400-24 from 742942 to 1223256 tid 1 2024-09-16T05:13:23.726 INFO:tasks.rados.rados.2.smithi204.stdout:795: writing smithi20457400-24 from 1823137 to 2471905 tid 2 2024-09-16T05:13:23.727 INFO:tasks.rados.rados.2.smithi204.stdout:796: copy_from oid 16 from oid 28 current snap is 85 2024-09-16T05:13:23.727 INFO:tasks.rados.rados.2.smithi204.stdout:795: finishing write tid 1 to smithi20457400-24 2024-09-16T05:13:23.727 INFO:tasks.rados.rados.2.smithi204.stdout:797: copy_from oid 17 from oid 48 current snap is 85 2024-09-16T05:13:23.727 INFO:tasks.rados.rados.2.smithi204.stdout:798: write oid 36 current snap is 85 2024-09-16T05:13:23.727 INFO:tasks.rados.rados.2.smithi204.stdout:798: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-16T05:13:23.733 INFO:tasks.rados.rados.2.smithi204.stdout:798: writing smithi20457400-36 from 759749 to 1315312 tid 1 2024-09-16T05:13:23.740 INFO:tasks.rados.rados.2.smithi204.stdout:798: writing smithi20457400-36 from 1765266 to 2557692 tid 2 2024-09-16T05:13:23.745 INFO:tasks.rados.rados.2.smithi204.stdout:798: writing smithi20457400-36 from 2964741 to 3159749 tid 3 2024-09-16T05:13:23.745 INFO:tasks.rados.rados.2.smithi204.stdout:799: snap_create 2024-09-16T05:13:23.745 INFO:tasks.rados.rados.2.smithi204.stdout:795: finishing write tid 2 to smithi20457400-24 2024-09-16T05:13:23.745 INFO:tasks.rados.rados.2.smithi204.stdout:795: finishing write tid 3 to smithi20457400-24 2024-09-16T05:13:23.745 INFO:tasks.rados.rados.2.smithi204.stdout:795: finishing write tid 4 to smithi20457400-24 2024-09-16T05:13:23.745 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 319 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:13:23.745 INFO:tasks.rados.rados.2.smithi204.stdout:795: left oid 24 (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:13:23.746 INFO:tasks.rados.rados.2.smithi204.stdout:796: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:13:23.746 INFO:tasks.rados.rados.2.smithi204.stdout:796: finishing copy_from to smithi20457400-16 2024-09-16T05:13:23.746 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 288 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:13:23.746 INFO:tasks.rados.rados.2.smithi204.stdout:798: finishing write tid 1 to smithi20457400-36 2024-09-16T05:13:23.746 INFO:tasks.rados.rados.2.smithi204.stdout:797: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:13:23.746 INFO:tasks.rados.rados.2.smithi204.stdout:797: finishing copy_from to smithi20457400-17 2024-09-16T05:13:23.746 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 229 (ObjNum 241 snap 67 seq_num 241) dirty exists 2024-09-16T05:13:23.746 INFO:tasks.rados.rados.2.smithi204.stdout:798: finishing write tid 2 to smithi20457400-36 2024-09-16T05:13:23.747 INFO:tasks.rados.rados.2.smithi204.stdout:798: finishing write tid 3 to smithi20457400-36 2024-09-16T05:13:23.747 INFO:tasks.rados.rados.2.smithi204.stdout:798: finishing write tid 4 to smithi20457400-36 2024-09-16T05:13:23.747 INFO:tasks.rados.rados.2.smithi204.stdout:798: finishing write tid 5 to smithi20457400-36 2024-09-16T05:13:23.747 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 354 (ObjNum 307 snap 85 seq_num 307) dirty exists 2024-09-16T05:13:23.747 INFO:tasks.rados.rados.2.smithi204.stdout:798: left oid 36 (ObjNum 307 snap 85 seq_num 307) 2024-09-16T05:13:23.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:23.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:23.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:23.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:23.896 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:23.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:23 smithi032 ceph-mon[27388]: pgmap v558: 57 pgs: 57 active+clean; 330 MiB data, 800 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2024-09-16T05:13:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1373575552' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]': finished 2024-09-16T05:13:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:23 smithi032 ceph-mon[27388]: osdmap e271: 12 total, 12 up, 11 in 2024-09-16T05:13:23.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1734976248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:23 smithi196 ceph-mon[30802]: pgmap v558: 57 pgs: 57 active+clean; 330 MiB data, 800 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2024-09-16T05:13:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1373575552' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]': finished 2024-09-16T05:13:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:23 smithi196 ceph-mon[30802]: osdmap e271: 12 total, 12 up, 11 in 2024-09-16T05:13:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1734976248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:23.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:24.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:24.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:24.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(1.a,RELEASE,e:112,from:9,reservation_nonce:0)", 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 160.109469252, 2024-09-16T05:13:24.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.42149128000000002, 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010445+0000", 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5999999999999999e-07 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010447+0000", 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.227e-06 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431780+0000", 2024-09-16T05:13:24.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.42133351800000002 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431791+0000", 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0773e-05 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431896+0000", 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104955 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431918+0000", 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2096e-05 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431936+0000", 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7351e-05 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1168 2.14 e192/185 2:2ace617f:::smithi20457400-21:head v 192'99, mlcod=188'95)", 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:03.794630+0000", 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 80.325283755000001, 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023315050000000002, 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.794630+0000", 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.794630+0000", 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.794640+0000", 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7690000000000008e-06 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.795545+0000", 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00090524500000000005 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.795581+0000", 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5777e-05 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.795589+0000", 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.089e-06 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.795674+0000", 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5748000000000006e-05 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.795703+0000", 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8311999999999999e-05 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.796911+0000", 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012081990000000001 2024-09-16T05:13:24.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:03.796961+0000", 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0365999999999999e-05 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.1a,RELEASE,e:216,from:8,reservation_nonce:0)", 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 55.668120537999997, 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065133569000000002, 2024-09-16T05:13:24.140 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2499999999999999e-07 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451795+0000", 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.674e-06 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516800+0000", 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065004988 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516812+0000", 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2265e-05 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516889+0000", 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6662999999999994e-05 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516910+0000", 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0718999999999999e-05 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516927+0000", 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7034999999999999e-05 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1471 2.14 e217/185 2:2ace617f:::smithi20457400-21:head v 217'118, mlcod=208'114)", 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:28.692382+0000", 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 55.427531213000002, 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077658060000000001, 2024-09-16T05:13:24.144 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.692382+0000", 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.692382+0000", 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.692383+0000", 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3399999999999999e-07 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696469+0000", 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040866189999999997 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696505+0000", 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5453000000000001e-05 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696513+0000", 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6329999999999996e-06 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696606+0000", 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3894000000000006e-05 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696633+0000", 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6516000000000002e-05 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.700102+0000", 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034693390000000001 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.700148+0000", 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5818000000000001e-05 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.148 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1484 2.11 2:8c1ba486:::smithi20457400-2:37 [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:29.666238+0000", 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 54.453675855999997, 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021428250000000001, 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1484 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666238+0000", 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666238+0000", 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666240+0000", 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.249e-06 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666249+0000", 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8009999999999993e-06 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666268+0000", 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.891e-05 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666278+0000", 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8800000000000003e-06 2024-09-16T05:13:24.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666341+0000", 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3787000000000004e-05 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666349+0000", 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3590000000000002e-06 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.668305+0000", 2024-09-16T05:13:24.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019561240000000001 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.668381+0000", 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5715000000000003e-05 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1485 2.11 2:8c1ba486:::smithi20457400-2:37 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:29.666300+0000", 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 54.453613519999998, 2024-09-16T05:13:24.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021783560000000002, 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1485 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666300+0000", 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666300+0000", 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666301+0000", 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6499999999999999e-07 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666305+0000", 2024-09-16T05:13:24.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7570000000000004e-06 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666314+0000", 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8089999999999999e-06 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666322+0000", 2024-09-16T05:13:24.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1510000000000004e-06 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666422+0000", 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010005699999999999 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666431+0000", 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6130000000000004e-06 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.668406+0000", 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00197503 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.668478+0000", 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2274000000000006e-05 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1486 2.11 2:8c1ba486:::smithi20457400-2:37 [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:29.666347+0000", 2024-09-16T05:13:24.158 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 54.453566670999997, 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002247073, 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1486 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666347+0000", 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666347+0000", 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666347+0000", 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666351+0000", 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9709999999999998e-06 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666360+0000", 2024-09-16T05:13:24.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6389999999999998e-06 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666365+0000", 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4489999999999998e-06 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666454+0000", 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8962999999999995e-05 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:24.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.666458+0000", 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5719999999999999e-06 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.668520+0000", 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020620790000000001 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:29.668594+0000", 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3875000000000007e-05 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.163 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.8.0:422 2.1a e227/215 2:5ae26c04:::smithi20457400-32:2f v 227'292, mlcod=220'288)", 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:38.667900+0000", 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 45.452013446999999, 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030168, 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.667900+0000", 2024-09-16T05:13:24.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.667900+0000", 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.667900+0000", 2024-09-16T05:13:24.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4499999999999998e-07 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.667936+0000", 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5587000000000003e-05 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.667977+0000", 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0828999999999998e-05 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.667985+0000", 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8050000000000007e-06 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.668087+0000", 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102229 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.668120+0000", 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3670000000000001e-05 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.670850+0000", 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027298299999999999 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.670917+0000", 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6605000000000004e-05 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.168 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.9.0:267 2.14 e227/185 2:2ace617f:::smithi20457400-21:2f v 227'126, mlcod=220'122)", 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:38.668856+0000", 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 45.451057466999998, 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023301609999999999, 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.668856+0000", 2024-09-16T05:13:24.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.668856+0000", 2024-09-16T05:13:24.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.668856+0000", 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8900000000000001e-07 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.668869+0000", 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2948999999999999e-05 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.668923+0000", 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3322000000000002e-05 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.668931+0000", 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5669999999999995e-06 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.669046+0000", 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115245 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.669068+0000", 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1991999999999999e-05 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.671128+0000", 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020591400000000001 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.671186+0000", 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8656999999999998e-05 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1591 2.16 e227/185 2:6be67188:::smithi20457400-46:head v 227'20, mlcod=224'16)", 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:38.964627+0000", 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 45.155286785999998, 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002107261, 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964627+0000", 2024-09-16T05:13:24.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964627+0000", 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.964628+0000", 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.246e-06 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.965302+0000", 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00067447500000000001 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.965342+0000", 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9393e-05 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.965350+0000", 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4800000000000001e-06 2024-09-16T05:13:24.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.965414+0000", 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3639000000000005e-05 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.965433+0000", 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8553999999999999e-05 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.966691+0000", 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001258094 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:38.966734+0000", 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3380000000000001e-05 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1806 2.1a e247/215 2:580e4d0e:::smithi20457400-29:head v 247'345, mlcod=245'341)", 2024-09-16T05:13:24.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.755521+0000", 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.364392287000001, 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002712337, 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.755521+0000", 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.755521+0000", 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.755521+0000", 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6999999999999999e-07 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.756515+0000", 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099368200000000007 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.756549+0000", 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3917000000000001e-05 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.756560+0000", 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1321000000000001e-05 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.756677+0000", 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011671 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.756703+0000", 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5745000000000002e-05 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.758185+0000", 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001482548 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.758234+0000", 2024-09-16T05:13:24.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8244000000000001e-05 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1833 2.11 2:8c1ba486:::smithi20457400-2:27 [read 0~2285719,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:59.198332+0000", 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.921581073999999, 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024373440000000001, 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.185 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1833 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198332+0000", 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198332+0000", 2024-09-16T05:13:24.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198333+0000", 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2300000000000002e-07 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198337+0000", 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6489999999999999e-06 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198341+0000", 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8489999999999999e-06 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198345+0000", 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.275e-06 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.199892+0000", 2024-09-16T05:13:24.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015470289999999999 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.199952+0000", 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0674999999999998e-05 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.200770+0000", 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00081754400000000002 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.8.0:490 2.1a e249/215 2:5f916a06:::smithi20457400-34:38 v 249'355, mlcod=249'349)", 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:00.675124+0000", 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.444789176, 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058030440000000003, 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.675124+0000", 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.675124+0000", 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.675125+0000", 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6600000000000002e-07 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.675132+0000", 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0550000000000002e-06 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.675207+0000", 2024-09-16T05:13:24.193 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:23 smithi204 ceph-mon[30933]: pgmap v558: 57 pgs: 57 active+clean; 330 MiB data, 800 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 19 op/s 2024-09-16T05:13:24.193 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1373575552' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "56"}]': finished 2024-09-16T05:13:24.193 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:23 smithi204 ceph-mon[30933]: osdmap e271: 12 total, 12 up, 11 in 2024-09-16T05:13:24.193 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1734976248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5360999999999999e-05 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.675212+0000", 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.259e-06 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.675669+0000", 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045650399999999999 2024-09-16T05:13:24.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.675690+0000", 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0917e-05 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.680881+0000", 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051912690000000001 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.680927+0000", 2024-09-16T05:13:24.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6313000000000002e-05 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1859 2.29 2.f7e83b69 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.437569221, 2024-09-16T05:13:24.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98895853899999997, 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1859 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:13:24.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682346+0000", 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9640000000000002e-06 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682353+0000", 2024-09-16T05:13:24.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2330000000000003e-06 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682369+0000", 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6721000000000001e-05 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682383+0000", 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4197e-05 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682449+0000", 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5660999999999994e-05 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682458+0000", 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8000000000000004e-06 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:01.671285+0000", 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98882742000000001 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:01.671303+0000", 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7543000000000001e-05 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1880 2.11 2:8c1ba486:::smithi20457400-2:head [rollback 35] snapc 50=[50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:08.696558+0000", 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.423355407000001, 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046736460000000001, 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:24.201 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1880 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.696558+0000", 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.696558+0000", 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.696560+0000", 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0310000000000001e-06 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.696569+0000", 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6750000000000008e-06 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.696586+0000", 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7524e-05 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.696595+0000", 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2290000000000003e-06 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.696664+0000", 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9610000000000006e-05 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.696672+0000", 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3640000000000001e-06 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.698635+0000", 2024-09-16T05:13:24.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019634470000000001 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.698767+0000", 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013203800000000001 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.701232+0000", 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024647279999999998 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2007 2.29 2:94d33c56:::smithi20457400-8:head [read 0~2621925,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:19.884091+0000", 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.2358222550000004, 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061480550000000004, 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2007 2024-09-16T05:13:24.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884091+0000", 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884091+0000", 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884093+0000", 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0229999999999999e-06 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884105+0000", 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.187e-05 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884123+0000", 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7430000000000001e-05 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884130+0000", 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4730000000000003e-06 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884200+0000", 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9444999999999997e-05 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884208+0000", 2024-09-16T05:13:24.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3669999999999997e-06 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.885780+0000", 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015716599999999999 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.885866+0000", 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6267000000000006e-05 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890239+0000", 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043735199999999997 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2008 2.29 2:94d33c56:::smithi20457400-8:head [sparse-read 0~2045226] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:19.884159+0000", 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.2357547719999999, 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00628205, 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.212 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2008 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884159+0000", 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884159+0000", 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884159+0000", 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1999999999999999e-07 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884162+0000", 2024-09-16T05:13:24.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8619999999999999e-06 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884178+0000", 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6234000000000001e-05 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884184+0000", 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8490000000000002e-06 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884270+0000", 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5259000000000005e-05 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884275+0000", 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0699999999999997e-06 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890284+0000", 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060089059999999996 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890346+0000", 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2331999999999999e-05 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890441+0000", 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4918000000000002e-05 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2009 2.29 2:94d33c56:::smithi20457400-8:head [sparse-read 0~2045226] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:19.884195+0000", 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.235718855, 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065356520000000003, 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2009 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.218 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884195+0000", 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884195+0000", 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884195+0000", 2024-09-16T05:13:24.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9100000000000004e-07 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884199+0000", 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8469999999999998e-06 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884205+0000", 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2360000000000001e-06 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884210+0000", 2024-09-16T05:13:24.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0540000000000002e-06 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884303+0000", 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2717e-05 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884308+0000", 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4650000000000002e-06 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890476+0000", 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006167802 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890576+0000", 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100146 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890730+0000", 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015389399999999999 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.222 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2016 2.14 e269/185 2:2d3f13f8:::smithi20457400-7:head v 269'188, mlcod=260'184)", 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:20.704375+0000", 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.415538126, 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025250310000000001, 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.704375+0000", 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.704375+0000", 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.704376+0000", 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8699999999999995e-07 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.705012+0000", 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063577399999999997 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.705046+0000", 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4e-05 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.705057+0000", 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0849000000000001e-05 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.705159+0000", 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102048 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.705182+0000", 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3383999999999998e-05 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.706855+0000", 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001673515 2024-09-16T05:13:24.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:20.706900+0000", 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4974e-05 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2037 2.29 2:94d33c56:::smithi20457400-8:head [delete] snapc 55=[55,54,52,4d,4c,49,48,47,45,3e,3a,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e270)", 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:21.698684+0000", 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 2.4212293369999998, 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029316960000000001, 2024-09-16T05:13:24.227 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2037 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.698684+0000", 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.698684+0000", 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.698686+0000", 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.728e-06 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.698695+0000", 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3479999999999993e-06 2024-09-16T05:13:24.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.698716+0000", 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.052e-05 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.698729+0000", 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3562e-05 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.698822+0000", 2024-09-16T05:13:24.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2573999999999998e-05 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.698827+0000", 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3920000000000002e-06 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.699911+0000", 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010840400000000001 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:13:24.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.699976+0000", 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4559000000000003e-05 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.700090+0000", 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114444 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.700861+0000", 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077055299999999999 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:24.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.701509+0000", 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00064817100000000001 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.701571+0000", 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2306999999999998e-05 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:21.701616+0000", 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4498e-05 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.233 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:13:24.234 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:13:24.234 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:13:24.234 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:24.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:24.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:24.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:24.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:24.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:24.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:24.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:24.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:24.617 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:24.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:24.633 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:24.633 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:24.655 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:24.655 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:24.676 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:24.676 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:24.696 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:795: done (4 left) 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:796: done (3 left) 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:797: done (2 left) 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:798: done (1 left) 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:799: done (0 left) 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:800: read oid 2 snap -1 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:800: expect deleted 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:801: read oid 9 snap 71 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:801: expect deleted 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:802: read oid 33 snap -1 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:802: expect (ObjNum 283 snap 80 seq_num 283) 2024-09-16T05:13:24.718 INFO:tasks.rados.rados.2.smithi204.stdout:803: write (excl) oid 3 current snap is 86 2024-09-16T05:13:24.719 INFO:tasks.rados.rados.2.smithi204.stdout:803: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2024-09-16T05:13:24.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:24.724 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:24.724 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:24.726 INFO:tasks.rados.rados.2.smithi204.stdout:803: writing smithi20457400-3 from 776556 to 1407368 tid 1 2024-09-16T05:13:24.732 INFO:tasks.rados.rados.2.smithi204.stdout:803: writing smithi20457400-3 from 1823748 to 2476185 tid 2 2024-09-16T05:13:24.736 INFO:tasks.rados.rados.2.smithi204.stdout:803: writing smithi20457400-3 from 3108517 to 3176556 tid 3 2024-09-16T05:13:24.736 INFO:tasks.rados.rados.2.smithi204.stdout:804: delete oid 31 current snap is 86 2024-09-16T05:13:24.736 INFO:tasks.rados.rados.2.smithi204.stdout:802: expect (ObjNum 283 snap 80 seq_num 283) 2024-09-16T05:13:24.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:24.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:24.746 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:24.746 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:24.767 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:24.767 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:24.788 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:24.788 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:24.811 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:24.811 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:24.837 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:24.837 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:24.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:24.858 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:24.858 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:24.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:24.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:24.880 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:24.880 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:24.887 INFO:tasks.rados.rados.2.smithi204.stdout:803: finishing write tid 1 to smithi20457400-3 2024-09-16T05:13:24.887 INFO:tasks.rados.rados.2.smithi204.stdout:803: finishing write tid 2 to smithi20457400-3 2024-09-16T05:13:24.887 INFO:tasks.rados.rados.2.smithi204.stdout:803: finishing write tid 3 to smithi20457400-3 2024-09-16T05:13:24.887 INFO:tasks.rados.rados.2.smithi204.stdout:803: finishing write tid 4 to smithi20457400-3 2024-09-16T05:13:24.887 INFO:tasks.rados.rados.2.smithi204.stdout:803: finishing write tid 5 to smithi20457400-3 2024-09-16T05:13:24.887 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 446 (ObjNum 308 snap 86 seq_num 308) dirty exists 2024-09-16T05:13:24.887 INFO:tasks.rados.rados.2.smithi204.stdout:803: left oid 3 (ObjNum 308 snap 86 seq_num 308) 2024-09-16T05:13:24.887 INFO:tasks.rados.rados.2.smithi204.stdout:801: done (4 left) 2024-09-16T05:13:24.888 INFO:tasks.rados.rados.2.smithi204.stdout:802: done (3 left) 2024-09-16T05:13:24.888 INFO:tasks.rados.rados.2.smithi204.stdout:803: done (2 left) 2024-09-16T05:13:24.888 INFO:tasks.rados.rados.2.smithi204.stdout:804: done (1 left) 2024-09-16T05:13:24.888 INFO:tasks.rados.rados.2.smithi204.stdout:805: write oid 44 current snap is 86 2024-09-16T05:13:24.888 INFO:tasks.rados.rados.2.smithi204.stdout:805: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-16T05:13:24.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1734976248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:24 smithi032 ceph-mon[27388]: osdmap e272: 12 total, 12 up, 11 in 2024-09-16T05:13:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1081882843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:24 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:13:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:24 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:13:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]: dispatch 2024-09-16T05:13:24.892 INFO:tasks.rados.rados.2.smithi204.stdout:805: writing smithi20457400-44 from 410170 to 1191480 tid 1 2024-09-16T05:13:24.900 INFO:tasks.rados.rados.2.smithi204.stdout:805: writing smithi20457400-44 from 1940712 to 2596818 tid 2 2024-09-16T05:13:24.904 INFO:tasks.rados.rados.2.smithi204.stdout:805: writing smithi20457400-44 from 3163363 to 3210170 tid 3 2024-09-16T05:13:24.904 INFO:tasks.rados.rados.2.smithi204.stdout:806: snap_remove snap 82 2024-09-16T05:13:24.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:13:24.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1734976248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:24.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:24 smithi196 ceph-mon[30802]: osdmap e272: 12 total, 12 up, 11 in 2024-09-16T05:13:24.967 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1081882843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:24.967 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:24 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:13:24.967 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:24 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:13:24.967 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]: dispatch 2024-09-16T05:13:24.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:24.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:24.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:25.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:25.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:25.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:25.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1734976248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:24 smithi204 ceph-mon[30933]: osdmap e272: 12 total, 12 up, 11 in 2024-09-16T05:13:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1081882843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:24 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:13:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:24 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:13:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]: dispatch 2024-09-16T05:13:25.184 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:25.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:25.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:25.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:25.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:25.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:25.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:25.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:25.489 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:25.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:25.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:25.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:25.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:25.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:25.653 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:25.654 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:25.680 INFO:tasks.rados.rados.2.smithi204.stdout:806: done (2 left) 2024-09-16T05:13:25.680 INFO:tasks.rados.rados.2.smithi204.stdout:807: delete oid 42 current snap is 86 2024-09-16T05:13:25.686 INFO:tasks.rados.rados.2.smithi204.stdout:805: finishing write tid 1 to smithi20457400-44 2024-09-16T05:13:25.686 INFO:tasks.rados.rados.2.smithi204.stdout:800: done (2 left) 2024-09-16T05:13:25.686 INFO:tasks.rados.rados.2.smithi204.stdout:807: done (1 left) 2024-09-16T05:13:25.686 INFO:tasks.rados.rados.2.smithi204.stdout:808: write (excl) oid 35 current snap is 86 2024-09-16T05:13:25.687 INFO:tasks.rados.rados.2.smithi204.stdout:808: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-09-16T05:13:25.692 INFO:tasks.rados.rados.2.smithi204.stdout:808: writing smithi20457400-35 from 443784 to 1091945 tid 1 2024-09-16T05:13:25.699 INFO:tasks.rados.rados.2.smithi204.stdout:808: writing smithi20457400-35 from 1657676 to 2317451 tid 2 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:808: writing smithi20457400-35 from 2818209 to 3243784 tid 3 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:809: rollback oid 14 current snap is 86 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 85 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:805: finishing write tid 2 to smithi20457400-44 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:805: finishing write tid 3 to smithi20457400-44 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:805: finishing write tid 4 to smithi20457400-44 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:805: finishing write tid 5 to smithi20457400-44 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 244 (ObjNum 310 snap 86 seq_num 310) dirty exists 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:805: left oid 44 (ObjNum 310 snap 86 seq_num 310) 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:808: finishing write tid 1 to smithi20457400-35 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:805: done (2 left) 2024-09-16T05:13:25.705 INFO:tasks.rados.rados.2.smithi204.stdout:810: snap_remove snap 84 2024-09-16T05:13:25.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:25.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:25.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:25.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:25.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:25.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:25 smithi032 ceph-mon[27388]: pgmap v561: 57 pgs: 57 active+clean; 335 MiB data, 811 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-16T05:13:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2024-09-16T05:13:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:25 smithi032 ceph-mon[27388]: osdmap e273: 12 total, 12 up, 11 in 2024-09-16T05:13:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:25 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:25 smithi032 ceph-mon[27388]: osdmap e274: 12 total, 12 up, 11 in 2024-09-16T05:13:25.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:25.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:25 smithi196 ceph-mon[30802]: pgmap v561: 57 pgs: 57 active+clean; 335 MiB data, 811 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-16T05:13:25.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2024-09-16T05:13:25.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:25 smithi196 ceph-mon[30802]: osdmap e273: 12 total, 12 up, 11 in 2024-09-16T05:13:25.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:25 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:25.975 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:25 smithi196 ceph-mon[30802]: osdmap e274: 12 total, 12 up, 11 in 2024-09-16T05:13:25.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:25.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:26.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:26.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:26.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:25 smithi204 ceph-mon[30933]: pgmap v561: 57 pgs: 57 active+clean; 335 MiB data, 811 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-16T05:13:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2024-09-16T05:13:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:25 smithi204 ceph-mon[30933]: osdmap e273: 12 total, 12 up, 11 in 2024-09-16T05:13:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:25 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:25 smithi204 ceph-mon[30933]: osdmap e274: 12 total, 12 up, 11 in 2024-09-16T05:13:26.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:26.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:26.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:26.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:13:26.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:13:26.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:26.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:26.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:26.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:26.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:26.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:26.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:26.558 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:26.574 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:26.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:26.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:26.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:26.681 INFO:tasks.rados.rados.2.smithi204.stdout:810: done (2 left) 2024-09-16T05:13:26.682 INFO:tasks.rados.rados.2.smithi204.stdout:811: copy_from oid 27 from oid 19 current snap is 86 2024-09-16T05:13:26.682 INFO:tasks.rados.rados.2.smithi204.stdout:812: write (excl) oid 10 current snap is 86 2024-09-16T05:13:26.682 INFO:tasks.rados.rados.2.smithi204.stdout:812: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-16T05:13:26.687 INFO:tasks.rados.rados.2.smithi204.stdout:812: writing smithi20457400-10 from 460591 to 1184001 tid 1 2024-09-16T05:13:26.692 INFO:tasks.rados.rados.2.smithi204.stdout:812: writing smithi20457400-10 from 1716158 to 2119591 tid 2 2024-09-16T05:13:26.699 INFO:tasks.rados.rados.2.smithi204.stdout:812: writing smithi20457400-10 from 2729279 to 3169419 tid 3 2024-09-16T05:13:26.699 INFO:tasks.rados.rados.2.smithi204.stdout:813: rollback oid 33 current snap is 86 2024-09-16T05:13:26.699 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 33 to 83 2024-09-16T05:13:26.699 INFO:tasks.rados.rados.2.smithi204.stdout:808: finishing write tid 2 to smithi20457400-35 2024-09-16T05:13:26.699 INFO:tasks.rados.rados.2.smithi204.stdout:809: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:13:26.699 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 353 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:13:26.699 INFO:tasks.rados.rados.2.smithi204.stdout:808: finishing write tid 3 to smithi20457400-35 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:808: finishing write tid 4 to smithi20457400-35 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:808: finishing write tid 5 to smithi20457400-35 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 363 (ObjNum 312 snap 86 seq_num 312) dirty exists 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:808: left oid 35 (ObjNum 312 snap 86 seq_num 312) 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:811: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:812: finishing write tid 1 to smithi20457400-10 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:811: finishing copy_from to smithi20457400-27 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 280 (ObjNum 239 snap 67 seq_num 239) dirty exists 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:812: finishing write tid 2 to smithi20457400-10 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:808: done (4 left) 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:809: done (3 left) 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:811: done (2 left) 2024-09-16T05:13:26.700 INFO:tasks.rados.rados.2.smithi204.stdout:814: rollback oid 36 current snap is 86 2024-09-16T05:13:26.701 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 36 to 71 2024-09-16T05:13:26.701 INFO:tasks.rados.rados.2.smithi204.stdout:815: delete oid 35 current snap is 86 2024-09-16T05:13:26.702 INFO:tasks.rados.rados.2.smithi204.stdout:812: finishing write tid 3 to smithi20457400-10 2024-09-16T05:13:26.702 INFO:tasks.rados.rados.2.smithi204.stdout:812: finishing write tid 4 to smithi20457400-10 2024-09-16T05:13:26.702 INFO:tasks.rados.rados.2.smithi204.stdout:813: finishing rollback tid 0 to smithi20457400-33 2024-09-16T05:13:26.702 INFO:tasks.rados.rados.2.smithi204.stdout:812: finishing write tid 5 to smithi20457400-10 2024-09-16T05:13:26.702 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 313 (ObjNum 313 snap 86 seq_num 313) dirty exists 2024-09-16T05:13:26.702 INFO:tasks.rados.rados.2.smithi204.stdout:812: left oid 10 (ObjNum 313 snap 86 seq_num 313) 2024-09-16T05:13:26.702 INFO:tasks.rados.rados.2.smithi204.stdout:813: finishing rollback tid 1 to smithi20457400-33 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:813: finishing rollback tid 2 to smithi20457400-33 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 342 (ObjNum 283 snap 80 seq_num 283) dirty exists 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:814: finishing rollback tid 0 to smithi20457400-36 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:814: finishing rollback tid 1 to smithi20457400-36 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:814: finishing rollback tid 2 to smithi20457400-36 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 371 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:812: done (3 left) 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:813: done (2 left) 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:814: done (1 left) 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:815: done (0 left) 2024-09-16T05:13:26.703 INFO:tasks.rados.rados.2.smithi204.stdout:816: snap_remove snap 83 2024-09-16T05:13:26.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:26.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:26.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:26.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:26.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:26.879 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:26.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:26.947 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:13:26.947 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.36 scrub starts 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.36 scrub ok 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:13:26.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]: dispatch 2024-09-16T05:13:26.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]': finished 2024-09-16T05:13:26.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:26 smithi032 ceph-mon[27388]: osdmap e275: 12 total, 12 up, 11 in 2024-09-16T05:13:26.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:13:26.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.36 scrub starts 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.36 scrub ok 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]: dispatch 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]': finished 2024-09-16T05:13:26.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:26 smithi196 ceph-mon[30802]: osdmap e275: 12 total, 12 up, 11 in 2024-09-16T05:13:26.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:27.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:27.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:27.109 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:27.109 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:27.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:27.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:27.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:27.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.36 scrub starts 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.36 scrub ok 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]: dispatch 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "56"}]': finished 2024-09-16T05:13:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:26 smithi204 ceph-mon[30933]: osdmap e275: 12 total, 12 up, 11 in 2024-09-16T05:13:27.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:27.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:27.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:27.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:27.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:27.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:27.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:27.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:27.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:27.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:13:27.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:27.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:27.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:27.676 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:13:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:816: done (0 left) 2024-09-16T05:13:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:817: read oid 37 snap -1 2024-09-16T05:13:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:817: expect deleted 2024-09-16T05:13:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:818: copy_from oid 46 from oid 24 current snap is 86 2024-09-16T05:13:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:819: write (excl) oid 14 current snap is 86 2024-09-16T05:13:27.681 INFO:tasks.rados.rados.2.smithi204.stdout:819: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-16T05:13:27.687 INFO:tasks.rados.rados.2.smithi204.stdout:819: writing smithi20457400-14 from 494205 to 968113 tid 1 2024-09-16T05:13:27.693 INFO:tasks.rados.rados.2.smithi204.stdout:819: writing smithi20457400-14 from 1433122 to 1840224 tid 2 2024-09-16T05:13:27.698 INFO:tasks.rados.rados.2.smithi204.stdout:819: writing smithi20457400-14 from 2384125 to 2846809 tid 3 2024-09-16T05:13:27.698 INFO:tasks.rados.rados.2.smithi204.stdout:820: snap_remove snap 70 2024-09-16T05:13:27.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:27.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:13:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:27.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:27.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:27.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:27.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:27.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: pgmap v564: 67 pgs: 3 peering, 9 unknown, 55 active+clean; 342 MiB data, 819 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-16T05:13:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:13:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:13:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1858509331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:13:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1858509331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:27.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:27 smithi032 ceph-mon[27388]: osdmap e276: 12 total, 12 up, 11 in 2024-09-16T05:13:27.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: pgmap v564: 67 pgs: 3 peering, 9 unknown, 55 active+clean; 342 MiB data, 819 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:13:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:13:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:13:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:13:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:13:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1858509331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:13:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1858509331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:27 smithi196 ceph-mon[30802]: osdmap e276: 12 total, 12 up, 11 in 2024-09-16T05:13:27.989 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:13:27.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:28.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:28.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:28.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:28.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: pgmap v564: 67 pgs: 3 peering, 9 unknown, 55 active+clean; 342 MiB data, 819 MiB used, 983 GiB / 983 GiB avail; 5.1 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2024-09-16T05:13:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:13:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:13:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:13:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:13:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:13:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:13:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1858509331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1858509331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:27 smithi204 ceph-mon[30933]: osdmap e276: 12 total, 12 up, 11 in 2024-09-16T05:13:28.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:28.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:28.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:28.294 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:13:28.294 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:13:28.294 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-16T05:13:28.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd in 10 2024-09-16T05:13:28.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:28.394 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:28.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:28.451 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:28.460 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:28.460 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:28.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:28.519 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:28.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:28.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:28.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:28.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:819: finishing write tid 1 to smithi20457400-14 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:819: finishing write tid 2 to smithi20457400-14 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:818: finishing copy_from racing read to smithi20457400-46 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:818: finishing copy_from to smithi20457400-46 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 319 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:819: finishing write tid 3 to smithi20457400-14 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:819: finishing write tid 4 to smithi20457400-14 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:819: finishing write tid 5 to smithi20457400-14 2024-09-16T05:13:28.683 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 438 (ObjNum 315 snap 86 seq_num 315) dirty exists 2024-09-16T05:13:28.684 INFO:tasks.rados.rados.2.smithi204.stdout:819: left oid 14 (ObjNum 315 snap 86 seq_num 315) 2024-09-16T05:13:28.684 INFO:tasks.rados.rados.2.smithi204.stdout:817: done (3 left) 2024-09-16T05:13:28.684 INFO:tasks.rados.rados.2.smithi204.stdout:818: done (2 left) 2024-09-16T05:13:28.684 INFO:tasks.rados.rados.2.smithi204.stdout:819: done (1 left) 2024-09-16T05:13:28.684 INFO:tasks.rados.rados.2.smithi204.stdout:820: done (0 left) 2024-09-16T05:13:28.684 INFO:tasks.rados.rados.2.smithi204.stdout:821: write (excl) oid 23 current snap is 86 2024-09-16T05:13:28.684 INFO:tasks.rados.rados.2.smithi204.stdout:821: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-09-16T05:13:28.689 INFO:tasks.rados.rados.2.smithi204.stdout:821: writing smithi20457400-23 from 511012 to 1060169 tid 1 2024-09-16T05:13:28.696 INFO:tasks.rados.rados.2.smithi204.stdout:821: writing smithi20457400-23 from 1775251 to 2326011 tid 2 2024-09-16T05:13:28.702 INFO:tasks.rados.rados.2.smithi204.stdout:821: writing smithi20457400-23 from 3095195 to 3311012 tid 3 2024-09-16T05:13:28.702 INFO:tasks.rados.rados.2.smithi204.stdout:822: rollback oid 50 current snap is 86 2024-09-16T05:13:28.702 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 80 2024-09-16T05:13:28.702 INFO:tasks.rados.rados.2.smithi204.stdout:821: finishing write tid 1 to smithi20457400-23 2024-09-16T05:13:28.702 INFO:tasks.rados.rados.2.smithi204.stdout:821: finishing write tid 2 to smithi20457400-23 2024-09-16T05:13:28.702 INFO:tasks.rados.rados.2.smithi204.stdout:823: write (excl) oid 13 current snap is 86 2024-09-16T05:13:28.702 INFO:tasks.rados.rados.2.smithi204.stdout:823: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-09-16T05:13:28.708 INFO:tasks.rados.rados.2.smithi204.stdout:823: writing smithi20457400-13 from 527819 to 1152225 tid 1 2024-09-16T05:13:28.714 INFO:tasks.rados.rados.2.smithi204.stdout:823: writing smithi20457400-13 from 1833733 to 2244504 tid 2 2024-09-16T05:13:28.721 INFO:tasks.rados.rados.2.smithi204.stdout:823: writing smithi20457400-13 from 2722618 to 3324199 tid 3 2024-09-16T05:13:28.721 INFO:tasks.rados.rados.2.smithi204.stdout:824: read oid 29 snap -1 2024-09-16T05:13:28.721 INFO:tasks.rados.rados.2.smithi204.stdout:824: expect (ObjNum 266 snap 75 seq_num 266) 2024-09-16T05:13:28.721 INFO:tasks.rados.rados.2.smithi204.stdout:825: copy_from oid 47 from oid 16 current snap is 86 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:821: finishing write tid 3 to smithi20457400-23 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:821: finishing write tid 4 to smithi20457400-23 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:821: finishing write tid 5 to smithi20457400-23 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 253 (ObjNum 316 snap 86 seq_num 316) dirty exists 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:821: left oid 23 (ObjNum 316 snap 86 seq_num 316) 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:822: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:822: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:822: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 203 (ObjNum 250 snap 70 seq_num 250) dirty exists 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:823: finishing write tid 1 to smithi20457400-13 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:823: finishing write tid 2 to smithi20457400-13 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:821: done (4 left) 2024-09-16T05:13:28.722 INFO:tasks.rados.rados.2.smithi204.stdout:822: done (3 left) 2024-09-16T05:13:28.723 INFO:tasks.rados.rados.2.smithi204.stdout:826: snap_remove snap 74 2024-09-16T05:13:28.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:28.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:28.829 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:28 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:13:28.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:28 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:13:28.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:28 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:13:28.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:28 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:13:28.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:28 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:13:28.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:28 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:13:28.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:28 smithi032 ceph-mon[27388]: osdmap e277: 12 total, 12 up, 11 in 2024-09-16T05:13:28.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:28.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:28.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:28.941 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:28.941 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:28.962 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:28.962 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:28.985 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:28.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:28 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:13:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:28 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:13:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:28 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:13:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:28 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:13:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:28 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:13:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:28 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:13:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:28 smithi196 ceph-mon[30802]: osdmap e277: 12 total, 12 up, 11 in 2024-09-16T05:13:28.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:28.999 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:29.007 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:29.007 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:29.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:29.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:29.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:29.036 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:29.057 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:29.057 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:29.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:13:29.078 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:29.079 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:29.099 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:29.100 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:28 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:13:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:28 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:13:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:28 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:13:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:28 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:13:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:28 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:13:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:28 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:13:29.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:28 smithi204 ceph-mon[30933]: osdmap e277: 12 total, 12 up, 11 in 2024-09-16T05:13:29.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:29.121 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:29.122 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:29.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:29.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:29.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:29.146 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:29.166 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:29.166 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:29.189 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:29.189 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:29.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:29.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:29.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:29.330 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:29.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:29.391 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:29.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:29.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:29.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:29.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:29.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:29.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:29.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:29.738 INFO:tasks.rados.rados.2.smithi204.stdout:826: done (3 left) 2024-09-16T05:13:29.738 INFO:tasks.rados.rados.2.smithi204.stdout:827: snap_remove snap 56 2024-09-16T05:13:29.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:29.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:29.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:29.802 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:13:29.802 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:13:29.802 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:13:29.802 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:13:29.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:13:29.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:13:29.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: pgmap v567: 67 pgs: 5 peering, 7 unknown, 55 active+clean; 329 MiB data, 830 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 3 op/s 2024-09-16T05:13:29.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: 2.35 deep-scrub starts 2024-09-16T05:13:29.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: 2.35 deep-scrub ok 2024-09-16T05:13:29.819 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:13:29.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:13:29.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:13:29.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:13:29.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:13:29.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1044109424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:29.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:29.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:29.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:29.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:29.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: pgmap v567: 67 pgs: 5 peering, 7 unknown, 55 active+clean; 329 MiB data, 830 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 3 op/s 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: 2.35 deep-scrub starts 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: 2.35 deep-scrub ok 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:13:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1044109424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:29.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:29.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:29 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:29.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:30.016 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:30.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: pgmap v567: 67 pgs: 5 peering, 7 unknown, 55 active+clean; 329 MiB data, 830 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 3 op/s 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: 2.35 deep-scrub starts 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: 2.35 deep-scrub ok 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:13:30.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:13:30.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1044109424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:30.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:30.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:29 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:30.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:30.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:30.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:29 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:13:29.817+0000 7f7c3f992640 -1 mon.a@0(leader).osd e278 definitely_dead 0 2024-09-16T05:13:30.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:30.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:30.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:30.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:13:30.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:30.390 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:30.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:30.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:30.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:13:30.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:30.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:30.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:30.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:30.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:marked in osd.10. 2024-09-16T05:13:30.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:30.747 INFO:tasks.rados.rados.2.smithi204.stdout:827: done (3 left) 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:828: rollback oid 49 current snap is 86 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 60 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:829: copy_from oid 5 from oid 6 current snap is 86 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:830: rollback oid 8 current snap is 86 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 8 to 67 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:823: finishing write tid 3 to smithi20457400-13 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:831: snap_create 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:823: finishing write tid 4 to smithi20457400-13 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:823: finishing write tid 5 to smithi20457400-13 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 391 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:823: left oid 13 (ObjNum 317 snap 86 seq_num 317) 2024-09-16T05:13:30.748 INFO:tasks.rados.rados.2.smithi204.stdout:824: expect (ObjNum 266 snap 75 seq_num 266) 2024-09-16T05:13:30.768 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:30.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:30.772 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:30.868 INFO:tasks.rados.rados.2.smithi204.stdout:825: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:13:30.868 INFO:tasks.rados.rados.2.smithi204.stdout:825: finishing copy_from to smithi20457400-47 2024-09-16T05:13:30.868 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 288 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:13:30.868 INFO:tasks.rados.rados.2.smithi204.stdout:830: finishing rollback tid 1 to smithi20457400-8 2024-09-16T05:13:30.868 INFO:tasks.rados.rados.2.smithi204.stdout:830: finishing rollback tid 2 to smithi20457400-8 2024-09-16T05:13:30.868 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 315 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-16T05:13:30.868 INFO:tasks.rados.rados.2.smithi204.stdout:828: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:13:30.869 INFO:tasks.rados.rados.2.smithi204.stdout:828: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:13:30.869 INFO:tasks.rados.rados.2.smithi204.stdout:828: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:13:30.869 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 263 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:13:30.869 INFO:tasks.rados.rados.2.smithi204.stdout:829: finishing copy_from racing read to smithi20457400-5 2024-09-16T05:13:30.869 INFO:tasks.rados.rados.2.smithi204.stdout:829: finishing copy_from to smithi20457400-5 2024-09-16T05:13:30.869 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 50 (ObjNum 291 snap 81 seq_num 291) dirty exists 2024-09-16T05:13:30.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:30.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:30.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:30 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:13:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:30 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:13:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:30 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:13:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:30 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:13:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:30 smithi196 ceph-mon[30802]: osdmap e278: 12 total, 12 up, 11 in 2024-09-16T05:13:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/545916263' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-09-16T05:13:30.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:31.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:30 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:13:31.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:30 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:13:31.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:30 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:13:31.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:30 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:13:31.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:30 smithi204 ceph-mon[30933]: osdmap e278: 12 total, 12 up, 11 in 2024-09-16T05:13:31.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/545916263' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-09-16T05:13:31.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:31.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:31.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:31.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:30 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:13:31.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:30 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:13:31.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:30 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:13:31.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:30 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:13:31.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:30 smithi032 ceph-mon[27388]: osdmap e278: 12 total, 12 up, 11 in 2024-09-16T05:13:31.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/545916263' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-09-16T05:13:31.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:31.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:31.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:31.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:31.262 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:13:31.262 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:13:31.262 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:13:31.262 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:13:31.262 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:13:31.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:31.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:31.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:31.399 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-16T05:13:31.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:31.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:31.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:31.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:31.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:31.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:31.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:31.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:31.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:31.749 INFO:tasks.rados.rados.2.smithi204.stdout:823: done (6 left) 2024-09-16T05:13:31.749 INFO:tasks.rados.rados.2.smithi204.stdout:824: done (5 left) 2024-09-16T05:13:31.749 INFO:tasks.rados.rados.2.smithi204.stdout:825: done (4 left) 2024-09-16T05:13:31.749 INFO:tasks.rados.rados.2.smithi204.stdout:828: done (3 left) 2024-09-16T05:13:31.749 INFO:tasks.rados.rados.2.smithi204.stdout:829: done (2 left) 2024-09-16T05:13:31.749 INFO:tasks.rados.rados.2.smithi204.stdout:830: done (1 left) 2024-09-16T05:13:31.750 INFO:tasks.rados.rados.2.smithi204.stdout:831: done (0 left) 2024-09-16T05:13:31.750 INFO:tasks.rados.rados.2.smithi204.stdout:832: read oid 50 snap 71 2024-09-16T05:13:31.750 INFO:tasks.rados.rados.2.smithi204.stdout:832: expect (ObjNum 250 snap 70 seq_num 250) 2024-09-16T05:13:31.750 INFO:tasks.rados.rados.2.smithi204.stdout:833: read oid 2 snap -1 2024-09-16T05:13:31.750 INFO:tasks.rados.rados.2.smithi204.stdout:833: expect deleted 2024-09-16T05:13:31.750 INFO:tasks.rados.rados.2.smithi204.stdout:834: snap_remove snap 60 2024-09-16T05:13:31.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:31.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:31.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:31.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:31.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:31.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:31 smithi032 ceph-mon[27388]: pgmap v570: 67 pgs: 5 peering, 7 unknown, 55 active+clean; 322 MiB data, 827 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:31.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:31 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:13:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:31 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:13:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/545916263' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-16T05:13:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:31 smithi032 ceph-mon[27388]: osdmap e279: 12 total, 12 up, 12 in 2024-09-16T05:13:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:31 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:13:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:31 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:13:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:31 smithi204 ceph-mon[30933]: pgmap v570: 67 pgs: 5 peering, 7 unknown, 55 active+clean; 322 MiB data, 827 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:31 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:13:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:31 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:13:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/545916263' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-16T05:13:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:31 smithi204 ceph-mon[30933]: osdmap e279: 12 total, 12 up, 12 in 2024-09-16T05:13:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:31 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:13:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:31 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:13:31.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:13:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:31 smithi196 ceph-mon[30802]: pgmap v570: 67 pgs: 5 peering, 7 unknown, 55 active+clean; 322 MiB data, 827 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:13:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:31 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:13:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:31 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:13:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/545916263' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-16T05:13:31.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:31 smithi196 ceph-mon[30802]: osdmap e279: 12 total, 12 up, 12 in 2024-09-16T05:13:31.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:31 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:13:31.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:31 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:13:32.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:32.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:32.032 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:32.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:32.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:13:32.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:32.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:32.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:32.244 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:32.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:32.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:32.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:32.326 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:32.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:32.408 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:32.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:32.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:32.532 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:32.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:32.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:32.656 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:32.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:32.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:32.695 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:13:32.695 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:13:32.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:13:32.695 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:13:32.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.695 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:19 2.b 2:d7c71ae7:::smithi20457400-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 169.652660217, 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19676892500000001, 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 19 2024-09-16T05:13:32.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040519+0000", 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7300000000000001e-07 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040541+0000", 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2320999999999999e-05 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040547+0000", 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3060000000000003e-06 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040553+0000", 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2750000000000003e-06 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041020+0000", 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046726200000000003 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041092+0000", 2024-09-16T05:13:32.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2275999999999997e-05 2024-09-16T05:13:32.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:13:32.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041183+0000", 2024-09-16T05:13:32.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1071999999999998e-05 2024-09-16T05:13:32.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041456+0000", 2024-09-16T05:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027275299999999998 2024-09-16T05:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237175+0000", 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19571920200000001 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237230+0000", 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4855000000000002e-05 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237287+0000", 2024-09-16T05:13:32.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7129999999999997e-05 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:20 2.b 2:d7c71ae7:::smithi20457400-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 169.652618483, 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19685694000000001, 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.703 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 20 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:13:32.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4400000000000001e-07 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040587+0000", 2024-09-16T05:13:32.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6444e-05 2024-09-16T05:13:32.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040593+0000", 2024-09-16T05:13:32.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5320000000000003e-06 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040600+0000", 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5139999999999998e-06 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041356+0000", 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075660300000000001 2024-09-16T05:13:32.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041400+0000", 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3936000000000001e-05 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237316+0000", 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.195915805 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237361+0000", 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4548000000000001e-05 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237417+0000", 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6214000000000001e-05 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:274 2.b 2:d7c71ae7:::smithi20457400-5:head [write 2964119~94722] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 163.66734064299999, 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073226163999999996, 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.710 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 274 2024-09-16T05:13:32.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.711 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:13:32.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025839+0000", 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.141e-06 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025876+0000", 2024-09-16T05:13:32.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6724999999999998e-05 2024-09-16T05:13:32.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025887+0000", 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1413999999999999e-05 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2739999999999998e-06 2024-09-16T05:13:32.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025932+0000", 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.943e-05 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026014+0000", 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2540000000000006e-05 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026154+0000", 2024-09-16T05:13:32.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013913200000000001 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026818+0000", 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00066437899999999999 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099005+0000", 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072186933999999994 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099037+0000", 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2011000000000002e-05 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099064+0000", 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7183999999999999e-05 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:275 2.b 2:d7c71ae7:::smithi20457400-5:head [setxattr _header (55) in=7b,truncate 3058841] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 163.66727524800001, 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13101358900000001, 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 275 2024-09-16T05:13:32.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025904+0000", 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.34e-07 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025907+0000", 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9610000000000001e-06 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025910+0000", 2024-09-16T05:13:32.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8680000000000001e-06 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025913+0000", 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9960000000000002e-06 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026440+0000", 2024-09-16T05:13:32.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00052629399999999996 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026504+0000", 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3934000000000001e-05 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026588+0000", 2024-09-16T05:13:32.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4745000000000003e-05 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026821+0000", 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000232301 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156835+0000", 2024-09-16T05:13:32.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13001446 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156875+0000", 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9360999999999998e-05 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156917+0000", 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2335e-05 2024-09-16T05:13:32.724 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.725 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.725 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.725 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:276 2.b 2:d7c71ae7:::smithi20457400-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:13:32.725 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:13:32.725 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 163.667250182, 2024-09-16T05:13:32.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.131149614, 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 276 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.726 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025929+0000", 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.39e-07 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025931+0000", 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5940000000000001e-06 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025939+0000", 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1829999999999997e-06 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025944+0000", 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4770000000000002e-06 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026741+0000", 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000797326 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026805+0000", 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3409000000000002e-05 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156955+0000", 2024-09-16T05:13:32.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13015033600000001 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.157016+0000", 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0795999999999999e-05 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.157078+0000", 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2053999999999999e-05 2024-09-16T05:13:32.731 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:274 2.b e117/107 ondisk, result = 0)", 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.026832+0000", 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 163.666346809, 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072250941999999999, 2024-09-16T05:13:32.732 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026832+0000", 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026832+0000", 2024-09-16T05:13:32.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026832+0000", 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0200000000000002e-07 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026836+0000", 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7570000000000002e-06 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098946+0000", 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072109698 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098953+0000", 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.041e-06 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098996+0000", 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3041999999999999e-05 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099001+0000", 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6370000000000004e-06 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099083+0000", 2024-09-16T05:13:32.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1165e-05 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:286 2.b 2:da4f6049:::smithi20457400-49:head [delete] snapc 4=[4] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117)", 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 163.595049813, 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.242160297, 2024-09-16T05:13:32.737 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 286 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6699999999999999e-07 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098131+0000", 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1289999999999999e-06 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098138+0000", 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5660000000000003e-06 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098143+0000", 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098194+0000", 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1368e-05 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098273+0000", 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8170999999999994e-05 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098449+0000", 2024-09-16T05:13:32.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176171 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099087+0000", 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00063836000000000003 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340229+0000", 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.241141684 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340263+0000", 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3813000000000003e-05 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340289+0000", 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6403000000000001e-05 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.743 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:275 2.b e117/107 ondisk, result = 0)", 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098921+0000", 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 163.59425707700001, 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.058002900000000003, 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098921+0000", 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098921+0000", 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098922+0000", 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1200000000000002e-07 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098926+0000", 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6839999999999998e-06 2024-09-16T05:13:32.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156740+0000", 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.057813937000000003 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156750+0000", 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7829999999999993e-06 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156824+0000", 2024-09-16T05:13:32.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4012999999999998e-05 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156831+0000", 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7219999999999999e-06 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156924+0000", 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3048999999999996e-05 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.747 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:292 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 3] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340479+0000", 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 163.35269930600001, 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017522666999999999, 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.748 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 292 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340479+0000", 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340479+0000", 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340481+0000", 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2920000000000001e-06 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340486+0000", 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.609e-06 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340498+0000", 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1953999999999999e-05 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340505+0000", 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8979999999999999e-06 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340559+0000", 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4061000000000003e-05 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340651+0000", 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1849999999999996e-05 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.352854+0000", 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012202927000000001 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.355587+0000", 2024-09-16T05:13:32.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027334159999999998 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357931+0000", 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023442409999999999 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357969+0000", 2024-09-16T05:13:32.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7585e-05 2024-09-16T05:13:32.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.358002+0000", 2024-09-16T05:13:32.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2833999999999998e-05 2024-09-16T05:13:32.755 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:414 2.5 2:ad4a88eb:::smithi20457400-16:head [setxattr _header (56) in=7b,truncate 3394981] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 154.996259294, 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29453131100000002, 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.756 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 414 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:13:32.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696920+0000", 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999996e-07 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696923+0000", 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.92e-06 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696937+0000", 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3892e-05 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696941+0000", 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3819999999999997e-06 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697487+0000", 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054579200000000002 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697548+0000", 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1167000000000006e-05 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697638+0000", 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0548000000000001e-05 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697936+0000", 2024-09-16T05:13:32.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029720499999999998 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991363+0000", 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29342759099999999 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991404+0000", 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1013999999999997e-05 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991451+0000", 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6303e-05 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:415 2.5 2:ad4a88eb:::smithi20457400-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 154.99623116199999, 2024-09-16T05:13:32.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29466997700000003, 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 415 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696948+0000", 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696952+0000", 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6500000000000002e-06 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696958+0000", 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9149999999999999e-06 2024-09-16T05:13:32.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696963+0000", 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9439999999999996e-06 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697806+0000", 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084291800000000001 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697861+0000", 2024-09-16T05:13:32.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4725999999999997e-05 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991493+0000", 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29363172500000001 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991551+0000", 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8622000000000002e-05 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991617+0000", 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5977999999999997e-05 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:414 2.5 e126/107 ondisk, result = 0)", 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 154.99506958699999, 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29334887300000001, 2024-09-16T05:13:32.769 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.772 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.772 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.772 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:13:32.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.06e-07 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698112+0000", 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6520000000000002e-06 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991214+0000", 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29310221600000003 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991228+0000", 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.411e-05 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991351+0000", 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012277999999999999 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991359+0000", 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0430000000000008e-06 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991458+0000", 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8566000000000005e-05 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.775 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:884 2.b 2:d8680303:::smithi20457400-43:head [setxattr _header (57) in=7b,truncate 2470629] snapc 23=[23,22,21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 111.03250355599999, 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68080432499999999, 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.776 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 884 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660676+0000", 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2099999999999996e-07 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660679+0000", 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1769999999999998e-06 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660685+0000", 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6529999999999997e-06 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660691+0000", 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2530000000000004e-06 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.661798+0000", 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011070520000000001 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.661889+0000", 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0785999999999994e-05 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662038+0000", 2024-09-16T05:13:32.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000149777 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662482+0000", 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00044356499999999999 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341385+0000", 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67890330499999996 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341430+0000", 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4613999999999997e-05 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341479+0000", 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9521999999999998e-05 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:885 2.b 2:d8680303:::smithi20457400-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 111.032479486, 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68095269999999997, 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.783 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 885 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999999e-07 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660702+0000", 2024-09-16T05:13:32.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3099999999999999e-06 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660709+0000", 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0299999999999996e-06 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660714+0000", 2024-09-16T05:13:32.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1780000000000002e-06 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662345+0000", 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016309390000000001 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662461+0000", 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011565600000000001 2024-09-16T05:13:32.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341517+0000", 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67905638400000001 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341582+0000", 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4746999999999995e-05 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341652+0000", 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0026999999999999e-05 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:405 2.1a e216/215 ondisk, result = 0)", 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:13:32.789 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 65.026672625000003, 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26443446100000001, 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666507+0000", 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8700000000000005e-07 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666511+0000", 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1130000000000003e-06 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930764+0000", 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26425358900000001 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930771+0000", 2024-09-16T05:13:32.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7329999999999997e-06 2024-09-16T05:13:32.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930824+0000", 2024-09-16T05:13:32.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3436000000000001e-05 2024-09-16T05:13:32.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930834+0000", 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1570000000000006e-06 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930940+0000", 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106746 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.794 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1814 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.921404224, 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898298185, 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:13:32.795 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1814 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771776+0000", 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5149999999999999e-06 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771783+0000", 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0180000000000004e-06 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771805+0000", 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2076999999999999e-05 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771824+0000", 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8813000000000001e-05 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747524+0000", 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97570031700000004 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747560+0000", 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5549000000000001e-05 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670057+0000", 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92249772500000005 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670072+0000", 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5170999999999999e-05 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.800 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1815 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:13:32.801 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:13:32.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:32.802 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:13:32.802 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.921302324999999, 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8982342750000001, 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1815 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771877+0000", 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9599999999999999e-07 2024-09-16T05:13:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771881+0000", 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6880000000000001e-06 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771889+0000", 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6789999999999994e-06 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771896+0000", 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747584+0000", 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97568814000000004 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747590+0000", 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7440000000000004e-06 2024-09-16T05:13:32.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670102+0000", 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92251251499999998 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670110+0000", 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2740000000000007e-06 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.807 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1816 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.921220335999998, 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898192844, 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:13:32.808 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1816 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771959+0000", 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1999999999999998e-07 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771964+0000", 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771971+0000", 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.576e-06 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771977+0000", 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2960000000000004e-06 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747609+0000", 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97563200299999997 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747615+0000", 2024-09-16T05:13:32.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2820000000000004e-06 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670144+0000", 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92252909100000002 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670151+0000", 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2259999999999999e-06 2024-09-16T05:13:32.813 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1881 2.25 2:a49174eb:::smithi20457400-39:head [delete] snapc 50=[50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e257)", 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.996631035, 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37621878800000003, 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:32.814 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1881 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696549+0000", 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696556+0000", 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8090000000000004e-06 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696568+0000", 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2636000000000001e-05 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696576+0000", 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3459999999999996e-06 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696625+0000", 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9830000000000002e-05 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696720+0000", 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4816999999999997e-05 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696869+0000", 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014893999999999999 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.697931+0000", 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010616759999999999 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:32.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072669+0000", 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37473788800000002 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072715+0000", 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6371e-05 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072766+0000", 2024-09-16T05:13:32.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1220000000000001e-05 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1881 2.25 e257/252 ondisk, result = 0)", 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.995102786, 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37469565799999999, 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:32.821 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:13:32.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698077+0000", 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5099999999999998e-07 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698082+0000", 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072556+0000", 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37447454499999999 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072564+0000", 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4120000000000004e-06 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072654+0000", 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9878999999999998e-05 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072663+0000", 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072771+0000", 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108267 2024-09-16T05:13:32.825 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.826 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.826 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.826 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:32.826 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:32.826 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:834: done (2 left) 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:835: write (excl) oid 15 current snap is 87 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:835: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:835: writing smithi20457400-15 from 544626 to 1244281 tid 1 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:835: writing smithi20457400-15 from 1892215 to 2446644 tid 2 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:835: writing smithi20457400-15 from 3150041 to 3344626 tid 3 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:836: read oid 22 snap -1 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:836: expect deleted 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:837: snap_create 2024-09-16T05:13:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:832: expect (ObjNum 250 snap 70 seq_num 250) 2024-09-16T05:13:32.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:32.829 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:32.829 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:32.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:32.843 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:32.843 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:32.863 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:32.864 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:835: finishing write tid 1 to smithi20457400-15 2024-09-16T05:13:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:835: finishing write tid 2 to smithi20457400-15 2024-09-16T05:13:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:835: finishing write tid 3 to smithi20457400-15 2024-09-16T05:13:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:835: finishing write tid 4 to smithi20457400-15 2024-09-16T05:13:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:835: finishing write tid 5 to smithi20457400-15 2024-09-16T05:13:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 81 (ObjNum 318 snap 87 seq_num 318) dirty exists 2024-09-16T05:13:32.880 INFO:tasks.rados.rados.2.smithi204.stdout:835: left oid 15 (ObjNum 318 snap 87 seq_num 318) 2024-09-16T05:13:32.884 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:32.884 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:32.905 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:32.905 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:32.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: 2.2e scrub starts 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: 2.2e scrub ok 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: osdmap e280: 12 total, 12 up, 12 in 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/112152523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: 2.31 deep-scrub starts 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: 2.31 deep-scrub ok 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: 2.37 deep-scrub starts 2024-09-16T05:13:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:32 smithi204 ceph-mon[30933]: 2.37 deep-scrub ok 2024-09-16T05:13:32.926 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:32.927 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:32.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:32.948 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:32.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:32.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:32.969 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: 2.2e scrub starts 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: 2.2e scrub ok 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: osdmap e280: 12 total, 12 up, 12 in 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/112152523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: 2.31 deep-scrub starts 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: 2.31 deep-scrub ok 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: 2.37 deep-scrub starts 2024-09-16T05:13:32.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:32 smithi032 ceph-mon[27388]: 2.37 deep-scrub ok 2024-09-16T05:13:32.990 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:32.990 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:33.012 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:33.012 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:33.033 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:33.034 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:33.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:33.055 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:33.055 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:33.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:33.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:33.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:33.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: 2.2e scrub starts 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: 2.2e scrub ok 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: osdmap e280: 12 total, 12 up, 12 in 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/112152523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: 2.31 deep-scrub starts 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: 2.31 deep-scrub ok 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: 2.37 deep-scrub starts 2024-09-16T05:13:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:32 smithi196 ceph-mon[30802]: 2.37 deep-scrub ok 2024-09-16T05:13:33.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:33.327 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:33.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:33.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:33.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:33.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:33.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:33.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:33.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:33.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:33.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:33.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:33.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:33.755 INFO:tasks.rados.rados.2.smithi204.stdout:832: done (4 left) 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:833: done (3 left) 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:835: done (2 left) 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:836: done (1 left) 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:837: done (0 left) 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:838: read oid 17 snap -1 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:838: expect (ObjNum 241 snap 67 seq_num 241) 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:839: copy_from oid 35 from oid 13 current snap is 88 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:840: write oid 49 current snap is 88 2024-09-16T05:13:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:840: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-16T05:13:33.763 INFO:tasks.rados.rados.2.smithi204.stdout:840: writing smithi20457400-49 from 561433 to 1336337 tid 1 2024-09-16T05:13:33.769 INFO:tasks.rados.rados.2.smithi204.stdout:840: writing smithi20457400-49 from 1950697 to 2365137 tid 2 2024-09-16T05:13:33.775 INFO:tasks.rados.rados.2.smithi204.stdout:840: writing smithi20457400-49 from 2777464 to 3361433 tid 3 2024-09-16T05:13:33.775 INFO:tasks.rados.rados.2.smithi204.stdout:841: delete oid 44 current snap is 88 2024-09-16T05:13:33.775 INFO:tasks.rados.rados.2.smithi204.stdout:838: expect (ObjNum 241 snap 67 seq_num 241) 2024-09-16T05:13:33.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:33.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:13:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:33.824 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:33.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:33.863 INFO:tasks.rados.rados.2.smithi204.stdout:840: finishing write tid 1 to smithi20457400-49 2024-09-16T05:13:33.863 INFO:tasks.rados.rados.2.smithi204.stdout:839: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:13:33.863 INFO:tasks.rados.rados.2.smithi204.stdout:840: finishing write tid 2 to smithi20457400-49 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:839: finishing copy_from to smithi20457400-35 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 391 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:840: finishing write tid 3 to smithi20457400-49 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:840: finishing write tid 4 to smithi20457400-49 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:838: done (3 left) 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:839: done (2 left) 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:841: done (1 left) 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:842: write oid 39 current snap is 88 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:840: finishing write tid 5 to smithi20457400-49 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 272 (ObjNum 319 snap 88 seq_num 319) dirty exists 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:840: left oid 49 (ObjNum 319 snap 88 seq_num 319) 2024-09-16T05:13:33.864 INFO:tasks.rados.rados.2.smithi204.stdout:842: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-16T05:13:33.869 INFO:tasks.rados.rados.2.smithi204.stdout:842: writing smithi20457400-39 from 595047 to 1236802 tid 1 2024-09-16T05:13:33.874 INFO:tasks.rados.rados.2.smithi204.stdout:842: writing smithi20457400-39 from 1667661 to 2085770 tid 2 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:842: writing smithi20457400-39 from 2548663 to 3311685 tid 3 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:840: done (1 left) 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:843: read oid 38 snap -1 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:843: expect deleted 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:842: finishing write tid 1 to smithi20457400-39 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:842: finishing write tid 2 to smithi20457400-39 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:844: read oid 29 snap -1 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:844: expect (ObjNum 266 snap 75 seq_num 266) 2024-09-16T05:13:33.882 INFO:tasks.rados.rados.2.smithi204.stdout:845: delete oid 2 current snap is 88 2024-09-16T05:13:33.883 INFO:tasks.rados.rados.2.smithi204.stdout:845: done (3 left) 2024-09-16T05:13:33.883 INFO:tasks.rados.rados.2.smithi204.stdout:846: rollback oid 1 current snap is 88 2024-09-16T05:13:33.883 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 1 to 67 2024-09-16T05:13:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:847: delete oid 5 current snap is 88 2024-09-16T05:13:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:844: expect (ObjNum 266 snap 75 seq_num 266) 2024-09-16T05:13:33.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:33.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:33.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:846: finishing rollback tid 1 to smithi20457400-1 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 369 snap 0 seq_num 2944376384) dirty dne 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:842: finishing write tid 3 to smithi20457400-39 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:842: finishing write tid 4 to smithi20457400-39 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:842: finishing write tid 5 to smithi20457400-39 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 494 (ObjNum 321 snap 88 seq_num 321) dirty exists 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:842: left oid 39 (ObjNum 321 snap 88 seq_num 321) 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:842: done (4 left) 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:843: done (3 left) 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:844: done (2 left) 2024-09-16T05:13:34.005 INFO:tasks.rados.rados.2.smithi204.stdout:846: done (1 left) 2024-09-16T05:13:34.006 INFO:tasks.rados.rados.2.smithi204.stdout:847: done (0 left) 2024-09-16T05:13:34.006 INFO:tasks.rados.rados.2.smithi204.stdout:848: copy_from oid 35 from oid 13 current snap is 88 2024-09-16T05:13:34.006 INFO:tasks.rados.rados.2.smithi204.stdout:849: write (excl) oid 14 current snap is 88 2024-09-16T05:13:34.006 INFO:tasks.rados.rados.2.smithi204.stdout:849: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-16T05:13:34.008 INFO:tasks.rados.rados.2.smithi204.stdout:849: writing smithi20457400-14 from 645468 to 1112970 tid 1 2024-09-16T05:13:34.015 INFO:tasks.rados.rados.2.smithi204.stdout:849: writing smithi20457400-14 from 1726754 to 2292190 tid 2 2024-09-16T05:13:34.022 INFO:tasks.rados.rados.2.smithi204.stdout:849: writing smithi20457400-14 from 2798226 to 3395064 tid 3 2024-09-16T05:13:34.022 INFO:tasks.rados.rados.2.smithi204.stdout:850: snap_create 2024-09-16T05:13:34.022 INFO:tasks.rados.rados.2.smithi204.stdout:848: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:13:34.022 INFO:tasks.rados.rados.2.smithi204.stdout:848: finishing copy_from to smithi20457400-35 2024-09-16T05:13:34.022 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 391 (ObjNum 317 snap 86 seq_num 317) dirty exists 2024-09-16T05:13:34.048 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: pgmap v573: 67 pgs: 1 active+clean+scrubbing, 6 peering, 6 unknown, 54 active+clean; 309 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:34.048 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:13:34.048 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:13:34.048 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:13:34.048 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:13:34.049 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:34.049 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/112152523' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:34.049 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: osdmap e281: 12 total, 12 up, 12 in 2024-09-16T05:13:34.049 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1521548032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:34.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:34.071 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:34.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: pgmap v573: 67 pgs: 1 active+clean+scrubbing, 6 peering, 6 unknown, 54 active+clean; 309 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:13:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:13:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:13:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:13:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/112152523' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: osdmap e281: 12 total, 12 up, 12 in 2024-09-16T05:13:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1521548032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:34.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:34.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:34.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: pgmap v573: 67 pgs: 1 active+clean+scrubbing, 6 peering, 6 unknown, 54 active+clean; 309 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:13:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:13:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:13:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:13:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/112152523' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: osdmap e281: 12 total, 12 up, 12 in 2024-09-16T05:13:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1521548032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:34.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:34.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:34.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:34.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:34.456 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:34.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:34.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:34.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:34.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:34.624 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:34.624 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:34.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:34.713 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:34.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:34.769 INFO:tasks.rados.rados.2.smithi204.stdout:848: done (2 left) 2024-09-16T05:13:34.769 INFO:tasks.rados.rados.2.smithi204.stdout:850: done (1 left) 2024-09-16T05:13:34.770 INFO:tasks.rados.rados.2.smithi204.stdout:851: read oid 27 snap -1 2024-09-16T05:13:34.770 INFO:tasks.rados.rados.2.smithi204.stdout:851: expect (ObjNum 239 snap 67 seq_num 239) 2024-09-16T05:13:34.770 INFO:tasks.rados.rados.2.smithi204.stdout:852: delete oid 48 current snap is 89 2024-09-16T05:13:34.774 INFO:tasks.rados.rados.2.smithi204.stdout:852: done (2 left) 2024-09-16T05:13:34.774 INFO:tasks.rados.rados.2.smithi204.stdout:853: read oid 35 snap -1 2024-09-16T05:13:34.774 INFO:tasks.rados.rados.2.smithi204.stdout:853: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-16T05:13:34.774 INFO:tasks.rados.rados.2.smithi204.stdout:854: snap_remove snap 37 2024-09-16T05:13:34.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:34.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:34.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:34.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:34 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:34.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:34 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:34.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:34 smithi204 ceph-mon[30933]: osdmap e282: 12 total, 12 up, 12 in 2024-09-16T05:13:34.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:34.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:34.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:35.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:35.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:35.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:35.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:34 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:34 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:35.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:34 smithi196 ceph-mon[30802]: osdmap e282: 12 total, 12 up, 12 in 2024-09-16T05:13:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:34 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:34 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:34 smithi032 ceph-mon[27388]: osdmap e282: 12 total, 12 up, 12 in 2024-09-16T05:13:35.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:35.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:35.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:35.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:35.321 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:35.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:13:35.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:35.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:35.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:13:35.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:35.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:35.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:35.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:35.589 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:35.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:35.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:35.689 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:35.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:35.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:35.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:35.764 INFO:tasks.rados.rados.2.smithi204.stdout:854: done (3 left) 2024-09-16T05:13:35.764 INFO:tasks.rados.rados.2.smithi204.stdout:855: delete oid 17 current snap is 89 2024-09-16T05:13:35.764 INFO:tasks.rados.rados.2.smithi204.stdout:851: expect (ObjNum 239 snap 67 seq_num 239) 2024-09-16T05:13:35.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:35.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:35.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:849: finishing write tid 1 to smithi20457400-14 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:849: finishing write tid 2 to smithi20457400-14 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:849: finishing write tid 3 to smithi20457400-14 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:849: finishing write tid 4 to smithi20457400-14 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:849: finishing write tid 5 to smithi20457400-14 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 463 (ObjNum 324 snap 88 seq_num 324) dirty exists 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:849: left oid 14 (ObjNum 324 snap 88 seq_num 324) 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:849: done (3 left) 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:851: done (2 left) 2024-09-16T05:13:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:855: done (1 left) 2024-09-16T05:13:35.856 INFO:tasks.rados.rados.2.smithi204.stdout:856: read oid 25 snap -1 2024-09-16T05:13:35.856 INFO:tasks.rados.rados.2.smithi204.stdout:856: expect deleted 2024-09-16T05:13:35.856 INFO:tasks.rados.rados.2.smithi204.stdout:857: delete oid 10 current snap is 89 2024-09-16T05:13:35.856 INFO:tasks.rados.rados.2.smithi204.stdout:853: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-16T05:13:35.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:35.957 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:35 smithi204 ceph-mon[30933]: pgmap v576: 67 pgs: 1 active+clean+scrubbing, 1 active+remapped, 5 peering, 5 unknown, 55 active+clean; 319 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:35.957 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:35 smithi204 ceph-mon[30933]: osdmap e283: 12 total, 12 up, 12 in 2024-09-16T05:13:35.964 INFO:tasks.rados.rados.2.smithi204.stdout:853: done (2 left) 2024-09-16T05:13:35.965 INFO:tasks.rados.rados.2.smithi204.stdout:856: done (1 left) 2024-09-16T05:13:35.965 INFO:tasks.rados.rados.2.smithi204.stdout:857: done (0 left) 2024-09-16T05:13:35.965 INFO:tasks.rados.rados.2.smithi204.stdout:858: copy_from oid 19 from oid 40 current snap is 89 2024-09-16T05:13:35.965 INFO:tasks.rados.rados.2.smithi204.stdout:859: snap_create 2024-09-16T05:13:35.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:35.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:35.978 INFO:tasks.rados.rados.2.smithi204.stdout:858: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:13:35.980 INFO:tasks.rados.rados.2.smithi204.stdout:858: finishing copy_from to smithi20457400-19 2024-09-16T05:13:35.980 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 271 (ObjNum 294 snap 83 seq_num 294) dirty exists 2024-09-16T05:13:35.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:35 smithi196 ceph-mon[30802]: pgmap v576: 67 pgs: 1 active+clean+scrubbing, 1 active+remapped, 5 peering, 5 unknown, 55 active+clean; 319 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:35.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:35 smithi196 ceph-mon[30802]: osdmap e283: 12 total, 12 up, 12 in 2024-09-16T05:13:36.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:35 smithi032 ceph-mon[27388]: pgmap v576: 67 pgs: 1 active+clean+scrubbing, 1 active+remapped, 5 peering, 5 unknown, 55 active+clean; 319 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:36.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:35 smithi032 ceph-mon[27388]: osdmap e283: 12 total, 12 up, 12 in 2024-09-16T05:13:36.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:36.073 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:36.073 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:36.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:13:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:36.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:36.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:36.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:36.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:36.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:36.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:36.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:36.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:36.344 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:36.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:13:36.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:13:36.401 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T05:13:36.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:13:36.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:36.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:36.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:36.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:36.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:36.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:36.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:36.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:36.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:36.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:13:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:13:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:858: done (1 left) 2024-09-16T05:13:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:859: done (0 left) 2024-09-16T05:13:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:860: snap_remove snap 89 2024-09-16T05:13:36.811 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:36.811 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:36.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:36.832 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:36.832 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:36.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:36.838 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:36.853 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:36.853 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:36.874 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:36.894 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:36.894 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:36.915 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:36.936 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:36.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:36.957 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:36.958 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:36.958 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:36 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:36.958 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:36 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:36.958 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:36 smithi032 ceph-mon[27388]: osdmap e284: 12 total, 12 up, 12 in 2024-09-16T05:13:36.958 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2859337311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:36.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:36.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:36.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:37.001 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:37.001 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:37.013 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:37.022 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:37.022 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:37.048 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:37.048 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:37.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:37.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:37.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:37.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:36 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:37.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:36 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:37.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:36 smithi204 ceph-mon[30933]: osdmap e284: 12 total, 12 up, 12 in 2024-09-16T05:13:37.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2859337311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:37.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:37.208 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:37.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:37.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:37.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:36 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:36 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:36 smithi196 ceph-mon[30802]: osdmap e284: 12 total, 12 up, 12 in 2024-09-16T05:13:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2859337311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:37.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:37.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:37.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:37.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:37.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:37.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:37.574 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:13:37.574 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:13:37.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:13:37.574 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:13:37.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.574 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:290 2.d 2:b98a79db:::smithi20457400-31:head [copy-from ver 13] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 168.23184736300001, 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020074727000000001, 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 290 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340466+0000", 2024-09-16T05:13:37.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4300000000000001e-06 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340473+0000", 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3339999999999996e-06 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340486+0000", 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.279e-05 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340495+0000", 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4890000000000001e-06 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340559+0000", 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3962999999999997e-05 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340653+0000", 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3468999999999996e-05 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.353864+0000", 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01321114 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357823+0000", 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039590759999999997 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360478+0000", 2024-09-16T05:13:37.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026554489999999998 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360511+0000", 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3052999999999997e-05 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360539+0000", 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7534000000000001e-05 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:291 2.d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 168.23179148, 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016626765000000002, 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.581 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 291 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:13:37.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340521+0000", 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4199999999999995e-07 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340524+0000", 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8590000000000001e-06 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340532+0000", 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4090000000000003e-06 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340538+0000", 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.835e-06 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340822+0000", 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028415500000000002 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340874+0000", 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1558999999999999e-05 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357014+0000", 2024-09-16T05:13:37.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016140379999999999 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357072+0000", 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7887999999999999e-05 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357147+0000", 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5037999999999995e-05 2024-09-16T05:13:37.586 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:605 2.d 2:b1061e13:::smithi20457400-33:head [copy-from ver 20] snapc 13=[13,12,11,f,e,b] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 143.76936048300001, 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017106367000000001, 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.587 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 605 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802952+0000", 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9999999999999998e-07 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802955+0000", 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1379999999999999e-06 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802961+0000", 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3820000000000001e-06 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802966+0000", 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1370000000000001e-06 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803384+0000", 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00041787299999999997 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803568+0000", 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018420200000000001 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.814402+0000", 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010833617 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818577+0000", 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041754770000000004 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.819989+0000", 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001411818 2024-09-16T05:13:37.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.820026+0000", 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6613000000000003e-05 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.820057+0000", 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1609999999999997e-05 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:606 2.d 2:b1061e13:::smithi20457400-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 143.769335689, 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015239317, 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 606 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.594 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802976+0000", 2024-09-16T05:13:37.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.63e-07 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802980+0000", 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.309e-06 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802993+0000", 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3574000000000001e-05 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802999+0000", 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5949999999999998e-06 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803711+0000", 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071251099999999998 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803769+0000", 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7649000000000003e-05 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818011+0000", 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014242184 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818110+0000", 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8938999999999996e-05 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818215+0000", 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000105093 2024-09-16T05:13:37.598 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1182 2.15 2:a94680a3:::smithi20457400-13:head [setxattr _header (57) in=7b,truncate 3109295] snapc 2f=[2f,2e,2d,2c,2b,2a,29,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 93.576336677, 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016329251999999999, 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.599 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1182 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9999999999999998e-07 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995978+0000", 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2589999999999999e-06 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995982+0000", 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3059999999999999e-06 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995985+0000", 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1829999999999998e-06 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996680+0000", 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069462300000000001 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996787+0000", 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107268 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996940+0000", 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000152826 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997265+0000", 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032523600000000001 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012161+0000", 2024-09-16T05:13:37.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014896243999999999 2024-09-16T05:13:37.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012231+0000", 2024-09-16T05:13:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:37.606 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:37.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9796999999999996e-05 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012304+0000", 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3109999999999996e-05 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1183 2.15 2:a94680a3:::smithi20457400-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:13:37.607 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:13:37.608 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 93.576311421, 2024-09-16T05:13:37.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016473326999999999, 2024-09-16T05:13:37.608 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.608 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.608 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:37.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.608 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1183 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996001+0000", 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4200000000000002e-07 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996003+0000", 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1679999999999998e-06 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996009+0000", 2024-09-16T05:13:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3890000000000002e-06 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996014+0000", 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7049999999999998e-06 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997177+0000", 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00116278 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997224+0000", 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7601e-05 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012338+0000", 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015113685999999999 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012400+0000", 2024-09-16T05:13:37.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2199999999999994e-05 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012474+0000", 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3455999999999996e-05 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1329 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:13:37.613 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 75.789274693999999, 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80822626200000003, 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1329 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:13:37.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783039+0000", 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.674e-06 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783047+0000", 2024-09-16T05:13:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6619999999999994e-06 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783063+0000", 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6058000000000001e-05 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783071+0000", 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8399999999999995e-06 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783134+0000", 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2496999999999995e-05 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783142+0000", 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1219999999999995e-06 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591169+0000", 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80802694100000005 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591263+0000", 2024-09-16T05:13:37.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4468000000000004e-05 2024-09-16T05:13:37.618 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.618 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.618 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1330 2.1d 2:b98a79db:::smithi20457400-31:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 75.789223731999996, 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80866329400000003, 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1330 2024-09-16T05:13:37.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783092+0000", 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5149999999999998e-06 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783104+0000", 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2412000000000001e-05 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783117+0000", 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2379e-05 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783170+0000", 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3727999999999998e-05 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783178+0000", 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3479999999999998e-06 2024-09-16T05:13:37.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591633+0000", 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.808455339 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591751+0000", 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118087 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1331 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 75.789182965999998, 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80878027500000005, 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1331 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.624 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5400000000000001e-07 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783132+0000", 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689999999999999e-06 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783139+0000", 2024-09-16T05:13:37.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7920000000000004e-06 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783144+0000", 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783192+0000", 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8304999999999999e-05 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:13:37.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783196+0000", 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8480000000000003e-06 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591842+0000", 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80864587700000001 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591909+0000", 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7195999999999998e-05 2024-09-16T05:13:37.628 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1361 2.15 2:a94680a3:::smithi20457400-13:head [copy-from ver 281] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 75.226667039000006, 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017774172000000001, 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.629 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1361 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345646+0000", 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.646e-06 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345656+0000", 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3170000000000008e-06 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345672+0000", 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6821e-05 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345679+0000", 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0840000000000003e-06 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345741+0000", 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1345000000000003e-05 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345803+0000", 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2003999999999998e-05 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.354995+0000", 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091920379999999996 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.359012+0000", 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040172389999999997 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363338+0000", 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043261180000000003 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363376+0000", 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7613000000000001e-05 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363419+0000", 2024-09-16T05:13:37.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2947000000000001e-05 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1362 2.15 2:a94680a3:::smithi20457400-13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.345699+0000", 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 75.226612732999996, 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012587882, 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:37.635 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1362 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345699+0000", 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345699+0000", 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345699+0000", 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5600000000000001e-07 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345701+0000", 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2330000000000001e-06 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345706+0000", 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7700000000000001e-06 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345709+0000", 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1980000000000001e-06 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345941+0000", 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000231499 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345981+0000", 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0451000000000003e-05 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.358149+0000", 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01216767 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.358216+0000", 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7422000000000006e-05 2024-09-16T05:13:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.358287+0000", 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0283000000000005e-05 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.9.0:220 2.1d e210/209 ondisk, result = 0)", 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 74.979305523999997, 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018128111999999998, 2024-09-16T05:13:37.640 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593007+0000", 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3480000000000001e-06 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593019+0000", 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1963999999999999e-05 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610880+0000", 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017860157000000002 2024-09-16T05:13:37.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610890+0000", 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.093e-05 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610951+0000", 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0375999999999999e-05 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610963+0000", 2024-09-16T05:13:37.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1861e-05 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.611134+0000", 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000171476 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1389 2.1d 2:b98a79db:::smithi20457400-31:head [copy-from ver 20] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:13:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 74.976861708000001, 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.062531666999999999, 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1389 2024-09-16T05:13:37.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:13:37.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0499999999999999e-07 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595453+0000", 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9699999999999999e-06 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595467+0000", 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3725e-05 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595472+0000", 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.028e-06 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.597983+0000", 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002510759 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.598071+0000", 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8158000000000005e-05 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.605944+0000", 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078733620000000001 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608849+0000", 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029044190000000001 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657852+0000", 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.049003110000000002 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657923+0000", 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1388000000000002e-05 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657982+0000", 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8343e-05 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.650 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1390 2.1d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.595478+0000", 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 74.976833439000004, 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012924761, 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1390 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595478+0000", 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595478+0000", 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595479+0000", 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1899999999999998e-07 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595482+0000", 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.72e-06 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595488+0000", 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999999e-06 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595493+0000", 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6739999999999996e-06 2024-09-16T05:13:37.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.598209+0000", 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027159250000000001 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.598266+0000", 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997000000000003e-05 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608189+0000", 2024-09-16T05:13:37.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0099233610000000003 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608291+0000", 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101686 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608403+0000", 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111869 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1389 2.1d e210/209 ondisk, result = 0)", 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 74.961462784999995, 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.047139519999999997, 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610850+0000", 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5600000000000001e-06 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610858+0000", 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2099999999999996e-06 2024-09-16T05:13:37.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657761+0000", 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.046903037000000002 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657768+0000", 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4080000000000001e-06 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657828+0000", 2024-09-16T05:13:37.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0241999999999997e-05 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657843+0000", 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5043e-05 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657988+0000", 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014501999999999999 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1471 2.14 2:2ace617f:::smithi20457400-21:head [copy-from ver 158] snapc 3d=[3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 68.887233750999997, 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015385315, 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1471 2024-09-16T05:13:37.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9299999999999998e-07 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685082+0000", 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3960000000000002e-06 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685089+0000", 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.323e-06 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685094+0000", 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0200000000000002e-06 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685163+0000", 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8861000000000004e-05 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685259+0000", 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6365000000000003e-05 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.692105+0000", 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068456330000000003 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.696821+0000", 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047159250000000002 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700376+0000", 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035550970000000001 2024-09-16T05:13:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700421+0000", 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5158999999999999e-05 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700463+0000", 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2042999999999997e-05 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1575 2.5 2:a5bc1608:::smithi20457400-11:head [setxattr _header (57) in=7b,truncate 3966452] snapc 43=[43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 58.881136945000002, 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64249167500000004, 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1575 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5200000000000002e-07 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691178+0000", 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.734e-06 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691184+0000", 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2940000000000002e-06 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691188+0000", 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.642e-06 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692349+0000", 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001161444 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692452+0000", 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102916 2024-09-16T05:13:37.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692596+0000", 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000143552 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692970+0000", 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037462699999999999 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333552+0000", 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64058177999999999 2024-09-16T05:13:37.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333609+0000", 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6728999999999998e-05 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333666+0000", 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7504999999999999e-05 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1576 2.5 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 58.881118209999997, 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64272760100000004, 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1576 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.672 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691194+0000", 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.65e-07 2024-09-16T05:13:37.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691195+0000", 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-06 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691199+0000", 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0060000000000003e-06 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691202+0000", 2024-09-16T05:13:37.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9239999999999999e-06 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692872+0000", 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001669467 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692954+0000", 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2361000000000006e-05 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333715+0000", 2024-09-16T05:13:37.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64076073300000003 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333817+0000", 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010184200000000001 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333921+0000", 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104249 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1596 2.15 2:ad7afe2b:::smithi20457400-38:head [copy-from ver 183] snapc 44=[44,43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:39.857462+0000", 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 57.714849260000001, 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017269851999999999, 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1596 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.677 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857462+0000", 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857462+0000", 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857464+0000", 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9420000000000002e-06 2024-09-16T05:13:37.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857474+0000", 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0135999999999999e-05 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857494+0000", 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0030999999999998e-05 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857506+0000", 2024-09-16T05:13:37.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1586999999999999e-05 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857574+0000", 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8201999999999995e-05 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857581+0000", 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2990000000000002e-06 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858652+0000", 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010710990000000001 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858713+0000", 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1587999999999994e-05 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.864612+0000", 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058987079999999999 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.868201+0000", 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035895290000000002 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.874670+0000", 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064688539999999996 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.874700+0000", 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9419000000000001e-05 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.874732+0000", 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2458e-05 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2055 2.15 2:ad4a88eb:::smithi20457400-16:head [copy-from ver 288] snapc 56=[56,55,54,52,4d,4c,49,48,45,3e,3a,35,27] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:23.726594+0000", 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 13.845717602000001, 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011944943, 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:37.683 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2055 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.726594+0000", 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.726594+0000", 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.726596+0000", 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.897e-06 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.726603+0000", 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7120000000000003e-06 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.726618+0000", 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5328999999999998e-05 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.726624+0000", 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.494e-06 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:37.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.726702+0000", 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7541999999999997e-05 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.726825+0000", 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000123201 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.732247+0000", 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054222269999999999 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.736118+0000", 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038703600000000002 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.738452+0000", 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023345179999999998 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.738498+0000", 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5781000000000001e-05 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:23.738539+0000", 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0881999999999999e-05 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.690 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.691 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.691 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:37.691 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:37.691 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:13:37.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:860: done (0 left) 2024-09-16T05:13:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:861: rollback oid 16 current snap is 90 2024-09-16T05:13:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 16 to 86 2024-09-16T05:13:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:862: rollback oid 28 current snap is 90 2024-09-16T05:13:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 28 to 51 2024-09-16T05:13:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:863: snap_create 2024-09-16T05:13:37.777 INFO:tasks.rados.rados.2.smithi204.stdout:861: finishing rollback tid 0 to smithi20457400-16 2024-09-16T05:13:37.777 INFO:tasks.rados.rados.2.smithi204.stdout:862: finishing rollback tid 0 to smithi20457400-28 2024-09-16T05:13:37.777 INFO:tasks.rados.rados.2.smithi204.stdout:861: finishing rollback tid 1 to smithi20457400-16 2024-09-16T05:13:37.778 INFO:tasks.rados.rados.2.smithi204.stdout:862: finishing rollback tid 1 to smithi20457400-28 2024-09-16T05:13:37.778 INFO:tasks.rados.rados.2.smithi204.stdout:861: finishing rollback tid 2 to smithi20457400-16 2024-09-16T05:13:37.778 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 411 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:13:37.778 INFO:tasks.rados.rados.2.smithi204.stdout:862: finishing rollback tid 2 to smithi20457400-28 2024-09-16T05:13:37.778 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 294 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:13:37.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:37.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:37.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:37.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:37.938 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:37.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:38.029 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:37 smithi204 ceph-mon[30933]: pgmap v579: 67 pgs: 67 active+clean; 360 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 5.4 MiB/s wr, 40 op/s; 15 MiB/s, 18 objects/s recovering 2024-09-16T05:13:38.029 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2859337311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:38.029 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:37 smithi204 ceph-mon[30933]: osdmap e285: 12 total, 12 up, 12 in 2024-09-16T05:13:38.029 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1023231454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:38.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:37 smithi032 ceph-mon[27388]: pgmap v579: 67 pgs: 67 active+clean; 360 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 5.4 MiB/s wr, 40 op/s; 15 MiB/s, 18 objects/s recovering 2024-09-16T05:13:38.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2859337311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:38.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:37 smithi032 ceph-mon[27388]: osdmap e285: 12 total, 12 up, 12 in 2024-09-16T05:13:38.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1023231454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:38.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:38.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:38.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:38.115 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:38.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:38.189 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:38.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:37 smithi196 ceph-mon[30802]: pgmap v579: 67 pgs: 67 active+clean; 360 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 5.4 MiB/s wr, 40 op/s; 15 MiB/s, 18 objects/s recovering 2024-09-16T05:13:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2859337311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:37 smithi196 ceph-mon[30802]: osdmap e285: 12 total, 12 up, 12 in 2024-09-16T05:13:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1023231454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:38.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:38.312 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:38.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:38.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:38.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:38.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:38.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:38.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:38.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:38.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:38.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:38.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:38.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:13:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:13:38.704 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:861: done (2 left) 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:862: done (1 left) 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:863: done (0 left) 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:864: rollback oid 44 current snap is 91 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 67 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:865: read oid 25 snap -1 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:865: expect deleted 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:866: read oid 18 snap -1 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:866: expect (ObjNum 288 snap 81 seq_num 288) 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:867: write (excl) oid 23 current snap is 91 2024-09-16T05:13:38.782 INFO:tasks.rados.rados.2.smithi204.stdout:867: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-16T05:13:38.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:38.789 INFO:tasks.rados.rados.2.smithi204.stdout:867: writing smithi20457400-23 from 712696 to 1197547 tid 1 2024-09-16T05:13:38.797 INFO:tasks.rados.rados.2.smithi204.stdout:867: writing smithi20457400-23 from 1960682 to 2533456 tid 2 2024-09-16T05:13:38.804 INFO:tasks.rados.rados.2.smithi204.stdout:867: writing smithi20457400-23 from 3307918 to 3512696 tid 3 2024-09-16T05:13:38.804 INFO:tasks.rados.rados.2.smithi204.stdout:868: rollback oid 22 current snap is 91 2024-09-16T05:13:38.804 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 71 2024-09-16T05:13:38.804 INFO:tasks.rados.rados.2.smithi204.stdout:864: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:13:38.804 INFO:tasks.rados.rados.2.smithi204.stdout:864: finishing rollback tid 2 to smithi20457400-44 2024-09-16T05:13:38.804 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 260 (ObjNum 220 snap 57 seq_num 220) dirty exists 2024-09-16T05:13:38.804 INFO:tasks.rados.rados.2.smithi204.stdout:867: finishing write tid 1 to smithi20457400-23 2024-09-16T05:13:38.805 INFO:tasks.rados.rados.2.smithi204.stdout:866: expect (ObjNum 288 snap 81 seq_num 288) 2024-09-16T05:13:38.809 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:38.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:38.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:38.935 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:38.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:38.956 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:39.030 INFO:tasks.rados.rados.2.smithi204.stdout:867: finishing write tid 2 to smithi20457400-23 2024-09-16T05:13:39.030 INFO:tasks.rados.rados.2.smithi204.stdout:864: done (4 left) 2024-09-16T05:13:39.030 INFO:tasks.rados.rados.2.smithi204.stdout:865: done (3 left) 2024-09-16T05:13:39.030 INFO:tasks.rados.rados.2.smithi204.stdout:866: done (2 left) 2024-09-16T05:13:39.030 INFO:tasks.rados.rados.2.smithi204.stdout:869: snap_create 2024-09-16T05:13:39.030 INFO:tasks.rados.rados.2.smithi204.stdout:867: finishing write tid 3 to smithi20457400-23 2024-09-16T05:13:39.030 INFO:tasks.rados.rados.2.smithi204.stdout:867: finishing write tid 4 to smithi20457400-23 2024-09-16T05:13:39.030 INFO:tasks.rados.rados.2.smithi204.stdout:867: finishing write tid 5 to smithi20457400-23 2024-09-16T05:13:39.031 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 281 (ObjNum 328 snap 91 seq_num 328) dirty exists 2024-09-16T05:13:39.031 INFO:tasks.rados.rados.2.smithi204.stdout:867: left oid 23 (ObjNum 328 snap 91 seq_num 328) 2024-09-16T05:13:39.031 INFO:tasks.rados.rados.2.smithi204.stdout:868: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:13:39.031 INFO:tasks.rados.rados.2.smithi204.stdout:868: finishing rollback tid 2 to smithi20457400-22 2024-09-16T05:13:39.031 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 347 (ObjNum 247 snap 69 seq_num 247) dirty exists 2024-09-16T05:13:39.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:39.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:39.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:39.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:38 smithi204 ceph-mon[30933]: osdmap e286: 12 total, 12 up, 12 in 2024-09-16T05:13:39.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:38 smithi032 ceph-mon[27388]: osdmap e286: 12 total, 12 up, 12 in 2024-09-16T05:13:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:39.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:39.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:38 smithi196 ceph-mon[30802]: osdmap e286: 12 total, 12 up, 12 in 2024-09-16T05:13:39.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:39.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:39.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:39.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:39.364 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":582,"stamp":"2024-09-16T05:13:38.071675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":364458391,"num_objects":169,"num_object_clones":118,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":14,"num_read":350,"num_read_kb":326658,"num_write":280,"num_write_kb":91521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":70295060,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10804,"ondisk_log_size":10804,"up":134,"acting":134,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":868588,"kb_used_data":468600,"kb_used_omap":18,"kb_used_meta":394797,"kb_avail":1124073236,"statfs":{"total":1151940427776,"available":1151050993664,"internally_reserved":0,"allocated":479846400,"data_stored":560159279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":404272441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29807325,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":86,"num_read_kb":31408,"num_write":124,"num_write_kb":33934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":63434267,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000782"},"pg_stats":[{"pgid":"2.37","version":"0'0","reported_seq":70,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.770577+0000","last_change":"2024-09-16T05:13:35.770805+0000","last_active":"2024-09-16T05:13:35.770577+0000","last_peered":"2024-09-16T05:13:35.770577+0000","last_clean":"2024-09-16T05:13:35.770577+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:13:35.770577+0000","last_undegraded":"2024-09-16T05:13:35.770577+0000","last_fullsized":"2024-09-16T05:13:35.770577+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:32.155153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:13:32.155153+0000","last_clean_scrub_stamp":"2024-09-16T05:13:32.155153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:39.812018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063989400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.36","version":"272'60","reported_seq":52,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.883290+0000","last_change":"2024-09-16T05:13:32.748607+0000","last_active":"2024-09-16T05:13:33.883290+0000","last_peered":"2024-09-16T05:13:33.883290+0000","last_clean":"2024-09-16T05:13:33.883290+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:13:33.883290+0000","last_undegraded":"2024-09-16T05:13:33.883290+0000","last_fullsized":"2024-09-16T05:13:33.883290+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:13:25.869308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.869308+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:38.009635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010300439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.35","version":"281'405","reported_seq":1311,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.883306+0000","last_change":"2024-09-16T05:13:32.749895+0000","last_active":"2024-09-16T05:13:33.883306+0000","last_peered":"2024-09-16T05:13:33.883306+0000","last_clean":"2024-09-16T05:13:33.883306+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:13:33.883306+0000","last_undegraded":"2024-09-16T05:13:33.883306+0000","last_fullsized":"2024-09-16T05:13:33.883306+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"276'397","last_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:13:28.279556+0000","objects_scrubbed":6,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:48.364950+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.002358686,"stat_sum":{"num_bytes":11677611,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.34","version":"281'213","reported_seq":714,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.766539+0000","last_change":"2024-09-16T05:13:35.768036+0000","last_active":"2024-09-16T05:13:35.766539+0000","last_peered":"2024-09-16T05:13:35.766539+0000","last_clean":"2024-09-16T05:13:35.766539+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:13:35.766539+0000","last_undegraded":"2024-09-16T05:13:35.766539+0000","last_fullsized":"2024-09-16T05:13:35.766539+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"279'209","last_scrub_stamp":"2024-09-16T05:13:31.216492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:31.216492+0000","objects_scrubbed":5,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:33.106789+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0.0024074230000000001,"stat_sum":{"num_bytes":8171649,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":9555,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.33","version":"181'15","reported_seq":70,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.775934+0000","last_change":"2024-09-16T05:13:35.776155+0000","last_active":"2024-09-16T05:13:35.775934+0000","last_peered":"2024-09-16T05:13:35.775934+0000","last_clean":"2024-09-16T05:13:35.775934+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:13:35.775934+0000","last_undegraded":"2024-09-16T05:13:35.775934+0000","last_fullsized":"2024-09-16T05:13:35.775934+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:13:28.549292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:28.549292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:33.977792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013146102,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.32","version":"284'83","reported_seq":103,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.769466+0000","last_change":"2024-09-16T05:13:35.769655+0000","last_active":"2024-09-16T05:13:35.769466+0000","last_peered":"2024-09-16T05:13:35.769466+0000","last_clean":"2024-09-16T05:13:35.769466+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:13:35.769466+0000","last_undegraded":"2024-09-16T05:13:35.769466+0000","last_fullsized":"2024-09-16T05:13:35.769466+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:13:27.771535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:27.771535+0000","objects_scrubbed":4,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:34.227505+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.004654877,"stat_sum":{"num_bytes":7026465,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.31","version":"282'126","reported_seq":633,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.757961+0000","last_change":"2024-09-16T05:13:32.757746+0000","last_active":"2024-09-16T05:13:34.757961+0000","last_peered":"2024-09-16T05:13:34.757961+0000","last_clean":"2024-09-16T05:13:34.757961+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:13:34.757961+0000","last_undegraded":"2024-09-16T05:13:34.757961+0000","last_fullsized":"2024-09-16T05:13:34.757961+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"280'123","last_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:13:31.958196+0000","objects_scrubbed":4,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:55.621602+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0052959239999999996,"stat_sum":{"num_bytes":9596764,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.30","version":"182'16","reported_seq":34,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.770263+0000","last_change":"2024-09-16T05:13:35.770521+0000","last_active":"2024-09-16T05:13:35.770263+0000","last_peered":"2024-09-16T05:13:35.770263+0000","last_clean":"2024-09-16T05:13:35.770263+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:13:35.770263+0000","last_undegraded":"2024-09-16T05:13:35.770263+0000","last_fullsized":"2024-09-16T05:13:35.770263+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:13:27.778999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:27.778999+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:44.087321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058823000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.11","version":"272'115","reported_seq":606,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.767390+0000","last_change":"2024-09-16T05:13:35.768081+0000","last_active":"2024-09-16T05:13:35.767390+0000","last_peered":"2024-09-16T05:13:35.767390+0000","last_clean":"2024-09-16T05:13:35.767390+0000","last_became_active":"2024-09-16T05:13:25.679267+0000","last_became_peered":"2024-09-16T05:13:25.679267+0000","last_unstale":"2024-09-16T05:13:35.767390+0000","last_undegraded":"2024-09-16T05:13:35.767390+0000","last_fullsized":"2024-09-16T05:13:35.767390+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:13:25.954175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.954175+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:36.499099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049740899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.10","version":"182'16","reported_seq":508,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.753777+0000","last_change":"2024-09-16T05:13:32.752017+0000","last_active":"2024-09-16T05:13:33.753777+0000","last_peered":"2024-09-16T05:13:33.753777+0000","last_clean":"2024-09-16T05:13:33.753777+0000","last_became_active":"2024-09-16T05:13:25.680445+0000","last_became_peered":"2024-09-16T05:13:25.680445+0000","last_unstale":"2024-09-16T05:13:33.753777+0000","last_undegraded":"2024-09-16T05:13:33.753777+0000","last_fullsized":"2024-09-16T05:13:33.753777+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:13:27.647121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:27.647121+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:54.569793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011083899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.f","version":"282'256","reported_seq":1148,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.754707+0000","last_change":"2024-09-16T05:13:32.763477+0000","last_active":"2024-09-16T05:13:33.754707+0000","last_peered":"2024-09-16T05:13:33.754707+0000","last_clean":"2024-09-16T05:13:33.754707+0000","last_became_active":"2024-09-16T05:13:25.680481+0000","last_became_peered":"2024-09-16T05:13:25.680481+0000","last_unstale":"2024-09-16T05:13:33.754707+0000","last_undegraded":"2024-09-16T05:13:33.754707+0000","last_fullsized":"2024-09-16T05:13:33.754707+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'248","last_scrub_stamp":"2024-09-16T05:13:28.609859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:28.609859+0000","objects_scrubbed":3,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:57.005783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011358982,"stat_sum":{"num_bytes":9047340,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.e","version":"272'232","reported_seq":1121,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.762775+0000","last_change":"2024-09-16T05:13:35.763077+0000","last_active":"2024-09-16T05:13:35.762775+0000","last_peered":"2024-09-16T05:13:35.762775+0000","last_clean":"2024-09-16T05:13:35.762775+0000","last_became_active":"2024-09-16T05:13:25.680369+0000","last_became_peered":"2024-09-16T05:13:25.680369+0000","last_unstale":"2024-09-16T05:13:35.762775+0000","last_undegraded":"2024-09-16T05:13:35.762775+0000","last_fullsized":"2024-09-16T05:13:35.762775+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:13:25.818442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.818442+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:32.054368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073016900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.d","version":"281'356","reported_seq":1237,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.762248+0000","last_change":"2024-09-16T05:13:32.763507+0000","last_active":"2024-09-16T05:13:34.762248+0000","last_peered":"2024-09-16T05:13:34.762248+0000","last_clean":"2024-09-16T05:13:34.762248+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:13:34.762248+0000","last_undegraded":"2024-09-16T05:13:34.762248+0000","last_fullsized":"2024-09-16T05:13:34.762248+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'312","last_scrub_stamp":"2024-09-16T05:13:08.021858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.021858+0000","objects_scrubbed":6,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:14.233648+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0096944530000000004,"stat_sum":{"num_bytes":9128668,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":12164,"num_write":8,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.c","version":"284'198","reported_seq":1078,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.978139+0000","last_change":"2024-09-16T05:13:35.768660+0000","last_active":"2024-09-16T05:13:35.978139+0000","last_peered":"2024-09-16T05:13:35.978139+0000","last_clean":"2024-09-16T05:13:35.978139+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:13:35.978139+0000","last_undegraded":"2024-09-16T05:13:35.978139+0000","last_fullsized":"2024-09-16T05:13:35.978139+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:13:07.600154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.600154+0000","objects_scrubbed":2,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:10.883546+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0039372360000000002,"stat_sum":{"num_bytes":4958131,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.8","version":"284'143","reported_seq":1032,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:36.457871+0000","last_change":"2024-09-16T05:13:36.458136+0000","last_active":"2024-09-16T05:13:36.457871+0000","last_peered":"2024-09-16T05:13:36.457871+0000","last_clean":"2024-09-16T05:13:36.457871+0000","last_became_active":"2024-09-16T05:12:59.751630+0000","last_became_peered":"2024-09-16T05:12:59.751630+0000","last_unstale":"2024-09-16T05:13:36.457871+0000","last_undegraded":"2024-09-16T05:13:36.457871+0000","last_fullsized":"2024-09-16T05:13:36.457871+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'114","last_scrub_stamp":"2024-09-16T05:13:06.516389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.516389+0000","objects_scrubbed":4,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:08.815279+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.69528911100000002,"stat_sum":{"num_bytes":7193325,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":14138,"num_write":3,"num_write_kb":1970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.3","version":"281'58","reported_seq":944,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.766241+0000","last_change":"2024-09-16T05:13:35.766476+0000","last_active":"2024-09-16T05:13:35.766241+0000","last_peered":"2024-09-16T05:13:35.766241+0000","last_clean":"2024-09-16T05:13:35.766241+0000","last_became_active":"2024-09-16T05:12:21.663580+0000","last_became_peered":"2024-09-16T05:12:21.663580+0000","last_unstale":"2024-09-16T05:13:35.766241+0000","last_undegraded":"2024-09-16T05:13:35.766241+0000","last_fullsized":"2024-09-16T05:13:35.766241+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-09-16T05:12:26.380594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.380594+0000","objects_scrubbed":2,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000934066,"stat_sum":{"num_bytes":6298197,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":32058,"num_write":10,"num_write_kb":2623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.0","version":"35'139","reported_seq":779,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.763284+0000","last_change":"2024-09-16T05:13:26.541877+0000","last_active":"2024-09-16T05:13:35.763284+0000","last_peered":"2024-09-16T05:13:35.763284+0000","last_clean":"2024-09-16T05:13:35.763284+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:13:35.763284+0000","last_undegraded":"2024-09-16T05:13:35.763284+0000","last_fullsized":"2024-09-16T05:13:35.763284+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:26.541796+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:26.541796+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:52.178618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":893,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.765641+0000","last_change":"2024-09-16T05:13:35.766162+0000","last_active":"2024-09-16T05:13:35.765641+0000","last_peered":"2024-09-16T05:13:35.765641+0000","last_clean":"2024-09-16T05:13:35.765641+0000","last_became_active":"2024-09-16T05:12:21.660508+0000","last_became_peered":"2024-09-16T05:12:21.660508+0000","last_unstale":"2024-09-16T05:13:35.765641+0000","last_undegraded":"2024-09-16T05:13:35.765641+0000","last_fullsized":"2024-09-16T05:13:35.765641+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:27.335469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.335469+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:45.822521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071138600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.1","version":"35'139","reported_seq":413,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.769998+0000","last_change":"2024-09-16T05:13:34.762822+0000","last_active":"2024-09-16T05:13:35.769998+0000","last_peered":"2024-09-16T05:13:35.769998+0000","last_clean":"2024-09-16T05:13:35.769998+0000","last_became_active":"2024-09-16T05:13:34.762556+0000","last_became_peered":"2024-09-16T05:13:34.762556+0000","last_unstale":"2024-09-16T05:13:35.769998+0000","last_undegraded":"2024-09-16T05:13:35.769998+0000","last_fullsized":"2024-09-16T05:13:35.769998+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:46.116628+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:46.116628+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.206966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":775,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.762918+0000","last_change":"2024-09-16T05:12:31.125618+0000","last_active":"2024-09-16T05:13:35.762918+0000","last_peered":"2024-09-16T05:13:35.762918+0000","last_clean":"2024-09-16T05:13:35.762918+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:13:35.762918+0000","last_undegraded":"2024-09-16T05:13:35.762918+0000","last_fullsized":"2024-09-16T05:13:35.762918+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:31.125520+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:31.125520+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.820470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":930,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.762506+0000","last_change":"2024-09-16T05:13:35.762708+0000","last_active":"2024-09-16T05:13:35.762506+0000","last_peered":"2024-09-16T05:13:35.762506+0000","last_clean":"2024-09-16T05:13:35.762506+0000","last_became_active":"2024-09-16T05:12:21.666300+0000","last_became_peered":"2024-09-16T05:12:21.666300+0000","last_unstale":"2024-09-16T05:13:35.762506+0000","last_undegraded":"2024-09-16T05:13:35.762506+0000","last_fullsized":"2024-09-16T05:13:35.762506+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:12:25.777434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:25.777434+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:41.627865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00039704300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.3","version":"35'139","reported_seq":780,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.883054+0000","last_change":"2024-09-16T05:13:18.689030+0000","last_active":"2024-09-16T05:13:33.883054+0000","last_peered":"2024-09-16T05:13:33.883054+0000","last_clean":"2024-09-16T05:13:33.883054+0000","last_became_active":"2024-09-16T05:13:18.688801+0000","last_became_peered":"2024-09-16T05:13:18.688801+0000","last_unstale":"2024-09-16T05:13:33.883054+0000","last_undegraded":"2024-09-16T05:13:33.883054+0000","last_fullsized":"2024-09-16T05:13:33.883054+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:10.386111+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:10.386111+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:36.719242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":924,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.776536+0000","last_change":"2024-09-16T05:13:35.776742+0000","last_active":"2024-09-16T05:13:35.776536+0000","last_peered":"2024-09-16T05:13:35.776536+0000","last_clean":"2024-09-16T05:13:35.776536+0000","last_became_active":"2024-09-16T05:13:31.746704+0000","last_became_peered":"2024-09-16T05:13:31.746704+0000","last_unstale":"2024-09-16T05:13:35.776536+0000","last_undegraded":"2024-09-16T05:13:35.776536+0000","last_fullsized":"2024-09-16T05:13:35.776536+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:12:28.044321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.044321+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:47.939909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.013675156000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":131,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.762091+0000","last_change":"2024-09-16T05:13:32.764062+0000","last_active":"2024-09-16T05:13:34.762091+0000","last_peered":"2024-09-16T05:13:34.762091+0000","last_clean":"2024-09-16T05:13:34.762091+0000","last_became_active":"2024-09-16T05:13:15.680699+0000","last_became_peered":"2024-09-16T05:13:15.680699+0000","last_unstale":"2024-09-16T05:13:34.762091+0000","last_undegraded":"2024-09-16T05:13:34.762091+0000","last_fullsized":"2024-09-16T05:13:34.762091+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:05.895744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:05.895744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:31.209062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099781279999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.4","version":"35'139","reported_seq":732,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.764532+0000","last_change":"2024-09-16T05:12:09.317163+0000","last_active":"2024-09-16T05:13:35.764532+0000","last_peered":"2024-09-16T05:13:35.764532+0000","last_clean":"2024-09-16T05:13:35.764532+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:13:35.764532+0000","last_undegraded":"2024-09-16T05:13:35.764532+0000","last_fullsized":"2024-09-16T05:13:35.764532+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:09.317055+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:09.317055+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:36.261087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":887,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.883319+0000","last_change":"2024-09-16T05:13:32.749164+0000","last_active":"2024-09-16T05:13:33.883319+0000","last_peered":"2024-09-16T05:13:33.883319+0000","last_clean":"2024-09-16T05:13:33.883319+0000","last_became_active":"2024-09-16T05:12:59.759329+0000","last_became_peered":"2024-09-16T05:12:59.759329+0000","last_unstale":"2024-09-16T05:13:33.883319+0000","last_undegraded":"2024-09-16T05:13:33.883319+0000","last_fullsized":"2024-09-16T05:13:33.883319+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:13:07.299230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.299230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:31.110600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001437621,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.5","version":"35'139","reported_seq":519,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.770353+0000","last_change":"2024-09-16T05:13:34.764400+0000","last_active":"2024-09-16T05:13:35.770353+0000","last_peered":"2024-09-16T05:13:35.770353+0000","last_clean":"2024-09-16T05:13:35.770353+0000","last_became_active":"2024-09-16T05:13:34.764118+0000","last_became_peered":"2024-09-16T05:13:34.764118+0000","last_unstale":"2024-09-16T05:13:35.770353+0000","last_undegraded":"2024-09-16T05:13:35.770353+0000","last_fullsized":"2024-09-16T05:13:35.770353+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:44.751365+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:44.751365+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:49.794121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"281'456","reported_seq":1308,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.762030+0000","last_change":"2024-09-16T05:13:32.758093+0000","last_active":"2024-09-16T05:13:34.762030+0000","last_peered":"2024-09-16T05:13:34.762030+0000","last_clean":"2024-09-16T05:13:34.762030+0000","last_became_active":"2024-09-16T05:12:59.751828+0000","last_became_peered":"2024-09-16T05:12:59.751828+0000","last_unstale":"2024-09-16T05:13:34.762030+0000","last_undegraded":"2024-09-16T05:13:34.762030+0000","last_fullsized":"2024-09-16T05:13:34.762030+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'430","last_scrub_stamp":"2024-09-16T05:13:07.051261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.051261+0000","objects_scrubbed":5,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:10.199247+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0047416189999999999,"stat_sum":{"num_bytes":16600512,"num_objects":6,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.6","version":"35'139","reported_seq":414,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.769813+0000","last_change":"2024-09-16T05:13:34.764496+0000","last_active":"2024-09-16T05:13:35.769813+0000","last_peered":"2024-09-16T05:13:35.769813+0000","last_clean":"2024-09-16T05:13:35.769813+0000","last_became_active":"2024-09-16T05:13:34.764187+0000","last_became_peered":"2024-09-16T05:13:34.764187+0000","last_unstale":"2024-09-16T05:13:35.769813+0000","last_undegraded":"2024-09-16T05:13:35.769813+0000","last_fullsized":"2024-09-16T05:13:35.769813+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:45.101705+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.101705+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.207000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":192337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":961,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.883203+0000","last_change":"2024-09-16T05:13:32.748108+0000","last_active":"2024-09-16T05:13:33.883203+0000","last_peered":"2024-09-16T05:13:33.883203+0000","last_clean":"2024-09-16T05:13:33.883203+0000","last_became_active":"2024-09-16T05:12:59.749293+0000","last_became_peered":"2024-09-16T05:12:59.749293+0000","last_unstale":"2024-09-16T05:13:33.883203+0000","last_undegraded":"2024-09-16T05:13:33.883203+0000","last_fullsized":"2024-09-16T05:13:33.883203+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:13:06.313006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.313006+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:06.925901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078394900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.7","version":"35'139","reported_seq":606,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.770072+0000","last_change":"2024-09-16T05:13:34.764429+0000","last_active":"2024-09-16T05:13:35.770072+0000","last_peered":"2024-09-16T05:13:35.770072+0000","last_clean":"2024-09-16T05:13:35.770072+0000","last_became_active":"2024-09-16T05:13:34.764109+0000","last_became_peered":"2024-09-16T05:13:34.764109+0000","last_unstale":"2024-09-16T05:13:35.770072+0000","last_undegraded":"2024-09-16T05:13:35.770072+0000","last_fullsized":"2024-09-16T05:13:35.770072+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:50.665816+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:50.665816+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:19.290266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":1063,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.753774+0000","last_change":"2024-09-16T05:13:32.751938+0000","last_active":"2024-09-16T05:13:33.753774+0000","last_peered":"2024-09-16T05:13:33.753774+0000","last_clean":"2024-09-16T05:13:33.753774+0000","last_became_active":"2024-09-16T05:12:59.748431+0000","last_became_peered":"2024-09-16T05:12:59.748431+0000","last_unstale":"2024-09-16T05:13:33.753774+0000","last_undegraded":"2024-09-16T05:13:33.753774+0000","last_fullsized":"2024-09-16T05:13:33.753774+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:13:06.465328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.465328+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:13.494059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001325842,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.8","version":"35'139","reported_seq":769,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.765363+0000","last_change":"2024-09-16T05:12:13.374972+0000","last_active":"2024-09-16T05:13:35.765363+0000","last_peered":"2024-09-16T05:13:35.765363+0000","last_clean":"2024-09-16T05:13:35.765363+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:10:38.967671+0000","last_unstale":"2024-09-16T05:13:35.765363+0000","last_undegraded":"2024-09-16T05:13:35.765363+0000","last_fullsized":"2024-09-16T05:13:35.765363+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.374887+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.374887+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:37.895703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"284'392","reported_seq":1286,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.770761+0000","last_change":"2024-09-16T05:13:35.771168+0000","last_active":"2024-09-16T05:13:35.770761+0000","last_peered":"2024-09-16T05:13:35.770761+0000","last_clean":"2024-09-16T05:13:35.770761+0000","last_became_active":"2024-09-16T05:12:59.751132+0000","last_became_peered":"2024-09-16T05:12:59.751132+0000","last_unstale":"2024-09-16T05:13:35.770761+0000","last_undegraded":"2024-09-16T05:13:35.770761+0000","last_fullsized":"2024-09-16T05:13:35.770761+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'336","last_scrub_stamp":"2024-09-16T05:13:06.619984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.619984+0000","objects_scrubbed":8,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:13.386654+0000","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.0061520760000000002,"stat_sum":{"num_bytes":20481764,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":29204,"num_write":20,"num_write_kb":10967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.9","version":"35'139","reported_seq":775,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.762791+0000","last_change":"2024-09-16T05:12:47.244164+0000","last_active":"2024-09-16T05:13:35.762791+0000","last_peered":"2024-09-16T05:13:35.762791+0000","last_clean":"2024-09-16T05:13:35.762791+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:13:35.762791+0000","last_undegraded":"2024-09-16T05:13:35.762791+0000","last_fullsized":"2024-09-16T05:13:35.762791+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:47.244082+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:47.244082+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:53.753278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1103,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.883113+0000","last_change":"2024-09-16T05:13:32.748109+0000","last_active":"2024-09-16T05:13:33.883113+0000","last_peered":"2024-09-16T05:13:33.883113+0000","last_clean":"2024-09-16T05:13:33.883113+0000","last_became_active":"2024-09-16T05:12:59.750592+0000","last_became_peered":"2024-09-16T05:12:59.750592+0000","last_unstale":"2024-09-16T05:13:33.883113+0000","last_undegraded":"2024-09-16T05:13:33.883113+0000","last_fullsized":"2024-09-16T05:13:33.883113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:13:08.517287+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.517287+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:34.557596+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0.00080544500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"1.a","version":"35'139","reported_seq":771,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.761829+0000","last_change":"2024-09-16T05:12:13.922480+0000","last_active":"2024-09-16T05:13:34.761829+0000","last_peered":"2024-09-16T05:13:34.761829+0000","last_clean":"2024-09-16T05:13:34.761829+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:13:34.761829+0000","last_undegraded":"2024-09-16T05:13:34.761829+0000","last_fullsized":"2024-09-16T05:13:34.761829+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:13.922401+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:13.922401+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:40.949519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":34,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.770198+0000","last_change":"2024-09-16T05:13:35.770437+0000","last_active":"2024-09-16T05:13:35.770198+0000","last_peered":"2024-09-16T05:13:35.770198+0000","last_clean":"2024-09-16T05:13:35.770198+0000","last_became_active":"2024-09-16T05:13:31.747644+0000","last_became_peered":"2024-09-16T05:13:31.747644+0000","last_unstale":"2024-09-16T05:13:35.770198+0000","last_undegraded":"2024-09-16T05:13:35.770198+0000","last_fullsized":"2024-09-16T05:13:35.770198+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:13:26.508703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:26.508703+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:46.553632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055144600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.13","version":"181'15","reported_seq":513,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.761944+0000","last_change":"2024-09-16T05:13:32.755966+0000","last_active":"2024-09-16T05:13:34.761944+0000","last_peered":"2024-09-16T05:13:34.761944+0000","last_clean":"2024-09-16T05:13:34.761944+0000","last_became_active":"2024-09-16T05:13:25.681297+0000","last_became_peered":"2024-09-16T05:13:25.681297+0000","last_unstale":"2024-09-16T05:13:34.761944+0000","last_undegraded":"2024-09-16T05:13:34.761944+0000","last_fullsized":"2024-09-16T05:13:34.761944+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:13:27.005361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:27.005361+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:38.168442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025125730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.14","version":"281'203","reported_seq":685,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.761868+0000","last_change":"2024-09-16T05:13:32.761191+0000","last_active":"2024-09-16T05:13:34.761868+0000","last_peered":"2024-09-16T05:13:34.761868+0000","last_clean":"2024-09-16T05:13:34.761868+0000","last_became_active":"2024-09-16T05:13:25.684888+0000","last_became_peered":"2024-09-16T05:13:25.684888+0000","last_unstale":"2024-09-16T05:13:34.761868+0000","last_undegraded":"2024-09-16T05:13:34.761868+0000","last_fullsized":"2024-09-16T05:13:34.761868+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'197","last_scrub_stamp":"2024-09-16T05:13:28.044263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:28.044263+0000","objects_scrubbed":5,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:32.308793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0075187800000000001,"stat_sum":{"num_bytes":15563216,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.15","version":"278'393","reported_seq":1276,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.762045+0000","last_change":"2024-09-16T05:13:32.756775+0000","last_active":"2024-09-16T05:13:34.762045+0000","last_peered":"2024-09-16T05:13:34.762045+0000","last_clean":"2024-09-16T05:13:34.762045+0000","last_became_active":"2024-09-16T05:13:25.680026+0000","last_became_peered":"2024-09-16T05:13:25.680026+0000","last_unstale":"2024-09-16T05:13:34.762045+0000","last_undegraded":"2024-09-16T05:13:34.762045+0000","last_fullsized":"2024-09-16T05:13:34.762045+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'387","last_scrub_stamp":"2024-09-16T05:13:25.974552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.974552+0000","objects_scrubbed":4,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:34.418199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031585799999999998,"stat_sum":{"num_bytes":10758223,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.16","version":"284'68","reported_seq":593,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.768310+0000","last_change":"2024-09-16T05:13:35.768543+0000","last_active":"2024-09-16T05:13:35.768310+0000","last_peered":"2024-09-16T05:13:35.768310+0000","last_clean":"2024-09-16T05:13:35.768310+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:13:35.768310+0000","last_undegraded":"2024-09-16T05:13:35.768310+0000","last_fullsized":"2024-09-16T05:13:35.768310+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:13:26.848310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:26.848310+0000","objects_scrubbed":3,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:37.800233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030847840000000001,"stat_sum":{"num_bytes":10297142,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.17","version":"0'0","reported_seq":700,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.768026+0000","last_change":"2024-09-16T05:13:35.768567+0000","last_active":"2024-09-16T05:13:35.768026+0000","last_peered":"2024-09-16T05:13:35.768026+0000","last_clean":"2024-09-16T05:13:35.768026+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:13:35.768026+0000","last_undegraded":"2024-09-16T05:13:35.768026+0000","last_fullsized":"2024-09-16T05:13:35.768026+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:26.947197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:26.947197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:42.829799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010417709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.18","version":"181'57","reported_seq":895,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.761750+0000","last_change":"2024-09-16T05:13:32.764117+0000","last_active":"2024-09-16T05:13:34.761750+0000","last_peered":"2024-09-16T05:13:34.761750+0000","last_clean":"2024-09-16T05:13:34.761750+0000","last_became_active":"2024-09-16T05:11:57.660350+0000","last_became_peered":"2024-09-16T05:11:57.660350+0000","last_unstale":"2024-09-16T05:13:34.761750+0000","last_undegraded":"2024-09-16T05:13:34.761750+0000","last_fullsized":"2024-09-16T05:13:34.761750+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:13:12.039899+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:41.344676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010200430999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.19","version":"182'60","reported_seq":517,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-16T05:13:37.777345+0000","last_change":"2024-09-16T05:13:37.777503+0000","last_active":"2024-09-16T05:13:37.777345+0000","last_peered":"2024-09-16T05:13:37.777345+0000","last_clean":"2024-09-16T05:13:37.777345+0000","last_became_active":"2024-09-16T05:11:57.658646+0000","last_became_peered":"2024-09-16T05:11:57.658646+0000","last_unstale":"2024-09-16T05:13:37.777345+0000","last_undegraded":"2024-09-16T05:13:37.777345+0000","last_fullsized":"2024-09-16T05:13:37.777345+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:13:25.467220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.467220+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:27.068023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087573399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"},{"start":"5b","length":"1"}]},{"pgid":"2.1a","version":"281'400","reported_seq":1267,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.883425+0000","last_change":"2024-09-16T05:13:32.760216+0000","last_active":"2024-09-16T05:13:33.883425+0000","last_peered":"2024-09-16T05:13:33.883425+0000","last_clean":"2024-09-16T05:13:33.883425+0000","last_became_active":"2024-09-16T05:12:27.664055+0000","last_became_peered":"2024-09-16T05:12:27.664055+0000","last_unstale":"2024-09-16T05:13:33.883425+0000","last_undegraded":"2024-09-16T05:13:33.883425+0000","last_fullsized":"2024-09-16T05:13:33.883425+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'278","last_scrub_stamp":"2024-09-16T05:12:28.451582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.451582+0000","objects_scrubbed":11,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:41.585580+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0082626720000000004,"stat_sum":{"num_bytes":26149886,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":57,"num_read_kb":54789,"num_write":16,"num_write_kb":3152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.1b","version":"284'276","reported_seq":1184,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.767158+0000","last_change":"2024-09-16T05:13:35.767833+0000","last_active":"2024-09-16T05:13:35.767158+0000","last_peered":"2024-09-16T05:13:35.767158+0000","last_clean":"2024-09-16T05:13:35.767158+0000","last_became_active":"2024-09-16T05:12:23.807872+0000","last_became_peered":"2024-09-16T05:12:23.807872+0000","last_unstale":"2024-09-16T05:13:35.767158+0000","last_undegraded":"2024-09-16T05:13:35.767158+0000","last_fullsized":"2024-09-16T05:13:35.767158+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"214'162","last_scrub_stamp":"2024-09-16T05:12:26.326323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:26.326323+0000","objects_scrubbed":7,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0031973209999999999,"stat_sum":{"num_bytes":20830085,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":24,"num_read_kb":20332,"num_write":25,"num_write_kb":8789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.1c","version":"284'345","reported_seq":1295,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.772473+0000","last_change":"2024-09-16T05:13:35.772756+0000","last_active":"2024-09-16T05:13:35.772473+0000","last_peered":"2024-09-16T05:13:35.772473+0000","last_clean":"2024-09-16T05:13:35.772473+0000","last_became_active":"2024-09-16T05:13:34.764188+0000","last_became_peered":"2024-09-16T05:13:34.764188+0000","last_unstale":"2024-09-16T05:13:35.772473+0000","last_undegraded":"2024-09-16T05:13:35.772473+0000","last_fullsized":"2024-09-16T05:13:35.772473+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'202","last_scrub_stamp":"2024-09-16T05:12:28.627965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:28.627965+0000","objects_scrubbed":8,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:48.253025+0000","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.0024452089999999998,"stat_sum":{"num_bytes":14140186,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":21,"num_read_kb":9442,"num_write":38,"num_write_kb":10905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":30728122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.1d","version":"284'471","reported_seq":1441,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.774281+0000","last_change":"2024-09-16T05:13:35.774508+0000","last_active":"2024-09-16T05:13:35.774281+0000","last_peered":"2024-09-16T05:13:35.774281+0000","last_clean":"2024-09-16T05:13:35.774281+0000","last_became_active":"2024-09-16T05:13:34.772983+0000","last_became_peered":"2024-09-16T05:13:34.772983+0000","last_unstale":"2024-09-16T05:13:35.774281+0000","last_undegraded":"2024-09-16T05:13:35.774281+0000","last_fullsized":"2024-09-16T05:13:35.774281+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"216'276","last_scrub_stamp":"2024-09-16T05:12:29.635450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:29.635450+0000","objects_scrubbed":9,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:56.121715+0000","scrub_duration":3,"objects_trimmed":3,"snaptrim_duration":0.0040669069999999998,"stat_sum":{"num_bytes":28600472,"num_objects":14,"num_object_clones":10,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":32,"num_read_kb":23293,"num_write":44,"num_write_kb":15308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":39182263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.1e","version":"284'295","reported_seq":1193,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-16T05:13:37.777270+0000","last_change":"2024-09-16T05:13:37.777479+0000","last_active":"2024-09-16T05:13:37.777270+0000","last_peered":"2024-09-16T05:13:37.777270+0000","last_clean":"2024-09-16T05:13:37.777270+0000","last_became_active":"2024-09-16T05:12:23.805930+0000","last_became_peered":"2024-09-16T05:12:23.805930+0000","last_unstale":"2024-09-16T05:13:37.777270+0000","last_undegraded":"2024-09-16T05:13:37.777270+0000","last_fullsized":"2024-09-16T05:13:37.777270+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'171","last_scrub_stamp":"2024-09-16T05:12:27.537258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.537258+0000","objects_scrubbed":7,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:45.329592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001129054,"stat_sum":{"num_bytes":15384799,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":10673,"num_write":22,"num_write_kb":6324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"},{"start":"5b","length":"1"}]},{"pgid":"2.1f","version":"281'219","reported_seq":1112,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.758055+0000","last_change":"2024-09-16T05:13:32.756829+0000","last_active":"2024-09-16T05:13:34.758055+0000","last_peered":"2024-09-16T05:13:34.758055+0000","last_clean":"2024-09-16T05:13:34.758055+0000","last_became_active":"2024-09-16T05:12:26.664097+0000","last_became_peered":"2024-09-16T05:12:26.664097+0000","last_unstale":"2024-09-16T05:13:34.758055+0000","last_undegraded":"2024-09-16T05:13:34.758055+0000","last_fullsized":"2024-09-16T05:13:34.758055+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'148","last_scrub_stamp":"2024-09-16T05:12:27.189264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.189264+0000","objects_scrubbed":2,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:49.299084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0048641550000000002,"stat_sum":{"num_bytes":12510190,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":53750,"num_write":13,"num_write_kb":5024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.20","version":"284'62","reported_seq":967,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.775617+0000","last_change":"2024-09-16T05:13:35.775872+0000","last_active":"2024-09-16T05:13:35.775617+0000","last_peered":"2024-09-16T05:13:35.775617+0000","last_clean":"2024-09-16T05:13:35.775617+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:13:35.775617+0000","last_undegraded":"2024-09-16T05:13:35.775617+0000","last_fullsized":"2024-09-16T05:13:35.775617+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"214'37","last_scrub_stamp":"2024-09-16T05:12:27.064860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:27.064860+0000","objects_scrubbed":3,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:56.912443+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.013025307999999999,"stat_sum":{"num_bytes":5988149,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":24944,"num_write":5,"num_write_kb":1188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.21","version":"178'50","reported_seq":940,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.762575+0000","last_change":"2024-09-16T05:13:35.762864+0000","last_active":"2024-09-16T05:13:35.762575+0000","last_peered":"2024-09-16T05:13:35.762575+0000","last_clean":"2024-09-16T05:13:35.762575+0000","last_became_active":"2024-09-16T05:12:23.806187+0000","last_became_peered":"2024-09-16T05:12:23.806187+0000","last_unstale":"2024-09-16T05:13:35.762575+0000","last_undegraded":"2024-09-16T05:13:35.762575+0000","last_fullsized":"2024-09-16T05:13:35.762575+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:13:31.784297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:31.784297+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:39.141773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053347799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.22","version":"182'20","reported_seq":34,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.770495+0000","last_change":"2024-09-16T05:13:35.770732+0000","last_active":"2024-09-16T05:13:35.770495+0000","last_peered":"2024-09-16T05:13:35.770495+0000","last_clean":"2024-09-16T05:13:35.770495+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:13:35.770495+0000","last_undegraded":"2024-09-16T05:13:35.770495+0000","last_fullsized":"2024-09-16T05:13:35.770495+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:12:45.564449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:12:45.564449+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:13:57.189551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059086100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.23","version":"184'17","reported_seq":395,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-16T05:13:37.776483+0000","last_change":"2024-09-16T05:13:37.776692+0000","last_active":"2024-09-16T05:13:37.776483+0000","last_peered":"2024-09-16T05:13:37.776483+0000","last_clean":"2024-09-16T05:13:37.776483+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:13:37.776483+0000","last_undegraded":"2024-09-16T05:13:37.776483+0000","last_fullsized":"2024-09-16T05:13:37.776483+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:13:32.510259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:32.510259+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:57.754363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000477207,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"},{"start":"5b","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":192,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.761846+0000","last_change":"2024-09-16T05:13:32.756460+0000","last_active":"2024-09-16T05:13:34.761846+0000","last_peered":"2024-09-16T05:13:34.761846+0000","last_clean":"2024-09-16T05:13:34.761846+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:13:34.761846+0000","last_undegraded":"2024-09-16T05:13:34.761846+0000","last_fullsized":"2024-09-16T05:13:34.761846+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:13:06.032520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.032520+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:10.731289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002989738,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.25","version":"282'494","reported_seq":1328,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.885791+0000","last_change":"2024-09-16T05:13:32.758563+0000","last_active":"2024-09-16T05:13:33.885791+0000","last_peered":"2024-09-16T05:13:33.885791+0000","last_clean":"2024-09-16T05:13:33.885791+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:13:33.885791+0000","last_undegraded":"2024-09-16T05:13:33.885791+0000","last_fullsized":"2024-09-16T05:13:33.885791+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"254'432","last_scrub_stamp":"2024-09-16T05:13:07.464459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.464459+0000","objects_scrubbed":9,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:26.867477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0065008369999999998,"stat_sum":{"num_bytes":30229072,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":12,"num_read_kb":16049,"num_write":8,"num_write_kb":7692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.26","version":"116'9","reported_seq":199,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.777445+0000","last_change":"2024-09-16T05:13:35.777653+0000","last_active":"2024-09-16T05:13:35.777445+0000","last_peered":"2024-09-16T05:13:35.777445+0000","last_clean":"2024-09-16T05:13:35.777445+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:13:35.777445+0000","last_undegraded":"2024-09-16T05:13:35.777445+0000","last_fullsized":"2024-09-16T05:13:35.777445+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:13:07.538943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.538943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:11.628680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.013610397,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.27","version":"0'0","reported_seq":211,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.776815+0000","last_change":"2024-09-16T05:13:35.777060+0000","last_active":"2024-09-16T05:13:35.776815+0000","last_peered":"2024-09-16T05:13:35.776815+0000","last_clean":"2024-09-16T05:13:35.776815+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:13:35.776815+0000","last_undegraded":"2024-09-16T05:13:35.776815+0000","last_fullsized":"2024-09-16T05:13:35.776815+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:09.483428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:09.483428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:20.617417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.013844927,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.28","version":"245'105","reported_seq":996,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-16T05:13:37.776399+0000","last_change":"2024-09-16T05:13:37.776621+0000","last_active":"2024-09-16T05:13:37.776399+0000","last_peered":"2024-09-16T05:13:37.776399+0000","last_clean":"2024-09-16T05:13:37.776399+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:13:37.776399+0000","last_undegraded":"2024-09-16T05:13:37.776399+0000","last_fullsized":"2024-09-16T05:13:37.776399+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:13:06.496116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.496116+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:29.525340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044323000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"},{"start":"5b","length":"1"}]},{"pgid":"2.29","version":"284'306","reported_seq":1250,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.771534+0000","last_change":"2024-09-16T05:13:35.771534+0000","last_active":"2024-09-16T05:13:35.771534+0000","last_peered":"2024-09-16T05:13:35.771534+0000","last_clean":"2024-09-16T05:13:35.771534+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:13:35.771534+0000","last_undegraded":"2024-09-16T05:13:35.771534+0000","last_fullsized":"2024-09-16T05:13:35.771534+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"254'244","last_scrub_stamp":"2024-09-16T05:13:06.019617+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.019617+0000","objects_scrubbed":16,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:07.963291+0000","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.0046099000000000001,"stat_sum":{"num_bytes":28842557,"num_objects":15,"num_object_clones":11,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":10,"num_read_kb":9123,"num_write":8,"num_write_kb":1531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1201,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.776215+0000","last_change":"2024-09-16T05:13:35.776495+0000","last_active":"2024-09-16T05:13:35.776215+0000","last_peered":"2024-09-16T05:13:35.776215+0000","last_clean":"2024-09-16T05:13:35.776215+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:13:35.776215+0000","last_undegraded":"2024-09-16T05:13:35.776215+0000","last_fullsized":"2024-09-16T05:13:35.776215+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:13:08.516105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.516105+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:10.135603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.01347682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.2b","version":"284'192","reported_seq":1088,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.766977+0000","last_change":"2024-09-16T05:13:35.767298+0000","last_active":"2024-09-16T05:13:35.766977+0000","last_peered":"2024-09-16T05:13:35.766977+0000","last_clean":"2024-09-16T05:13:35.766977+0000","last_became_active":"2024-09-16T05:13:04.678973+0000","last_became_peered":"2024-09-16T05:13:04.678973+0000","last_unstale":"2024-09-16T05:13:35.766977+0000","last_undegraded":"2024-09-16T05:13:35.766977+0000","last_fullsized":"2024-09-16T05:13:35.766977+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"253'179","last_scrub_stamp":"2024-09-16T05:13:06.865580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.865580+0000","objects_scrubbed":4,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:14.438813+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0049519519999999999,"stat_sum":{"num_bytes":8806901,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":7,"num_write_kb":4385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.2c","version":"241'194","reported_seq":1014,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-16T05:13:33.753775+0000","last_change":"2024-09-16T05:13:32.760434+0000","last_active":"2024-09-16T05:13:33.753775+0000","last_peered":"2024-09-16T05:13:33.753775+0000","last_clean":"2024-09-16T05:13:33.753775+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:13:33.753775+0000","last_undegraded":"2024-09-16T05:13:33.753775+0000","last_fullsized":"2024-09-16T05:13:33.753775+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:13:08.484521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.484521+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:32.747657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0082476939999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.2d","version":"274'307","reported_seq":1182,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.777130+0000","last_change":"2024-09-16T05:13:35.777358+0000","last_active":"2024-09-16T05:13:35.777130+0000","last_peered":"2024-09-16T05:13:35.777130+0000","last_clean":"2024-09-16T05:13:35.777130+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:13:35.777130+0000","last_undegraded":"2024-09-16T05:13:35.777130+0000","last_fullsized":"2024-09-16T05:13:35.777130+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'306","last_scrub_stamp":"2024-09-16T05:13:10.471980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:10.471980+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:22.827316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014001111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.2e","version":"284'244","reported_seq":1132,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-16T05:13:35.778774+0000","last_change":"2024-09-16T05:13:35.778989+0000","last_active":"2024-09-16T05:13:35.778774+0000","last_peered":"2024-09-16T05:13:35.778774+0000","last_clean":"2024-09-16T05:13:35.778774+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:13:35.778774+0000","last_undegraded":"2024-09-16T05:13:35.778774+0000","last_fullsized":"2024-09-16T05:13:35.778774+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"279'238","last_scrub_stamp":"2024-09-16T05:13:31.497779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:31.497779+0000","objects_scrubbed":6,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:41.650808+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.014715705000000001,"stat_sum":{"num_bytes":14050619,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":9,"num_read_kb":7120,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"1"},{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]},{"pgid":"2.2f","version":"281'252","reported_seq":1173,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-16T05:13:34.762070+0000","last_change":"2024-09-16T05:13:32.763502+0000","last_active":"2024-09-16T05:13:34.762070+0000","last_peered":"2024-09-16T05:13:34.762070+0000","last_clean":"2024-09-16T05:13:34.762070+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:13:34.762070+0000","last_undegraded":"2024-09-16T05:13:34.762070+0000","last_fullsized":"2024-09-16T05:13:34.762070+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"278'248","last_scrub_stamp":"2024-09-16T05:13:30.085727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:30.085727+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:48.224039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0096193500000000005,"stat_sum":{"num_bytes":11428760,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"3a","length":"1"},{"start":"3e","length":"1"},{"start":"47","length":"2"},{"start":"4c","length":"1"},{"start":"53","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":56,"stat_sum":{"num_bytes":363760683,"num_objects":166,"num_object_clones":118,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":14,"num_read":327,"num_read_kb":326638,"num_write":247,"num_write_kb":90986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":69910385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":432021504,"data_stored":525240844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9275,"ondisk_log_size":9275,"up":112,"acting":112,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697708,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20,"num_write":33,"num_write_kb":535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":384675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089398,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97980,"kb_used_data":66788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93647172,"statfs":{"total":95995035648,"available":95894704128,"internally_reserved":0,"allocated":68390912,"data_stored":80731121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285627,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80636,"kb_used_data":44572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93664516,"statfs":{"total":95995035648,"available":95912464384,"internally_reserved":0,"allocated":45641728,"data_stored":51183236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514561,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96584,"kb_used_data":61236,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93648568,"statfs":{"total":95995035648,"available":95896133632,"internally_reserved":0,"allocated":62705664,"data_stored":75661645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841603,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72200,"kb_used_data":41004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93672952,"statfs":{"total":95995035648,"available":95921102848,"internally_reserved":0,"allocated":41988096,"data_stored":46024978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103241,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91280,"kb_used_data":55928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93653872,"statfs":{"total":95995035648,"available":95901564928,"internally_reserved":0,"allocated":57270272,"data_stored":67307244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397584,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79476,"kb_used_data":44120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93665676,"statfs":{"total":95995035648,"available":95913652224,"internally_reserved":0,"allocated":45178880,"data_stored":54942211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757331,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55064,"kb_used_data":23868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690088,"statfs":{"total":95995035648,"available":95938650112,"internally_reserved":0,"allocated":24440832,"data_stored":27350472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920857,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49256,"kb_used_data":18068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93695896,"statfs":{"total":95995035648,"available":95944597504,"internally_reserved":0,"allocated":18501632,"data_stored":22448242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117086,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84032,"kb_used_data":52836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93661120,"statfs":{"total":95995035648,"available":95908986880,"internally_reserved":0,"allocated":54104064,"data_stored":68580467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313316,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49344,"kb_used_data":13988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93695808,"statfs":{"total":95995035648,"available":95944507392,"internally_reserved":0,"allocated":14323712,"data_stored":14170136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47796,"kb_used_data":16600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93697356,"statfs":{"total":95995035648,"available":95946092544,"internally_reserved":0,"allocated":16998400,"data_stored":16739294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64940,"kb_used_data":29592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93680212,"statfs":{"total":95995035648,"available":95928537088,"internally_reserved":0,"allocated":30302208,"data_stored":35020233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26963968,"data_stored":32951810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13672448,"data_stored":14679340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10768384,"data_stored":11891882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50741248,"data_stored":67750768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14282752,"data_stored":16837793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20885504,"data_stored":25072218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":41828352,"data_stored":52865205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53919744,"data_stored":65230238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40177664,"data_stored":45468907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":59367424,"data_stored":73593222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":34701312,"data_stored":40573026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":64712704,"data_stored":78326435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:13:39.368 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:13:39.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 66 2024-09-16T05:13:39.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:13:39.409 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:13:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:13:39.409 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:13:39.409 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:13:39.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:39.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:39.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:39.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:39.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:39.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:39.632 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:39.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:39.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:39.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:39.786 INFO:tasks.rados.rados.2.smithi204.stdout:867: done (2 left) 2024-09-16T05:13:39.786 INFO:tasks.rados.rados.2.smithi204.stdout:868: done (1 left) 2024-09-16T05:13:39.786 INFO:tasks.rados.rados.2.smithi204.stdout:869: done (0 left) 2024-09-16T05:13:39.786 INFO:tasks.rados.rados.2.smithi204.stdout:870: write oid 49 current snap is 92 2024-09-16T05:13:39.786 INFO:tasks.rados.rados.2.smithi204.stdout:870: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-16T05:13:39.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:39.792 INFO:tasks.rados.rados.2.smithi204.stdout:870: writing smithi20457400-49 from 729503 to 1289603 tid 1 2024-09-16T05:13:39.800 INFO:tasks.rados.rados.2.smithi204.stdout:870: writing smithi20457400-49 from 2019164 to 2735596 tid 2 2024-09-16T05:13:39.805 INFO:tasks.rados.rados.2.smithi204.stdout:870: writing smithi20457400-49 from 3335341 to 3529503 tid 3 2024-09-16T05:13:39.805 INFO:tasks.rados.rados.2.smithi204.stdout:871: write (excl) oid 22 current snap is 92 2024-09-16T05:13:39.805 INFO:tasks.rados.rados.2.smithi204.stdout:870: finishing write tid 1 to smithi20457400-49 2024-09-16T05:13:39.805 INFO:tasks.rados.rados.2.smithi204.stdout:871: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-16T05:13:39.812 INFO:tasks.rados.rados.2.smithi204.stdout:871: writing smithi20457400-22 from 746310 to 1381659 tid 1 2024-09-16T05:13:39.814 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:39.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:39.820 INFO:tasks.rados.rados.2.smithi204.stdout:871: writing smithi20457400-22 from 2077646 to 2654089 tid 2 2024-09-16T05:13:39.825 INFO:tasks.rados.rados.2.smithi204.stdout:871: writing smithi20457400-22 from 3362764 to 3546310 tid 3 2024-09-16T05:13:39.825 INFO:tasks.rados.rados.2.smithi204.stdout:872: delete oid 26 current snap is 92 2024-09-16T05:13:39.825 INFO:tasks.rados.rados.2.smithi204.stdout:870: finishing write tid 2 to smithi20457400-49 2024-09-16T05:13:39.825 INFO:tasks.rados.rados.2.smithi204.stdout:870: finishing write tid 3 to smithi20457400-49 2024-09-16T05:13:39.825 INFO:tasks.rados.rados.2.smithi204.stdout:870: finishing write tid 4 to smithi20457400-49 2024-09-16T05:13:39.826 INFO:tasks.rados.rados.2.smithi204.stdout:870: finishing write tid 5 to smithi20457400-49 2024-09-16T05:13:39.826 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 286 (ObjNum 329 snap 92 seq_num 329) dirty exists 2024-09-16T05:13:39.826 INFO:tasks.rados.rados.2.smithi204.stdout:870: left oid 49 (ObjNum 329 snap 92 seq_num 329) 2024-09-16T05:13:39.826 INFO:tasks.rados.rados.2.smithi204.stdout:871: finishing write tid 1 to smithi20457400-22 2024-09-16T05:13:39.826 INFO:tasks.rados.rados.2.smithi204.stdout:871: finishing write tid 2 to smithi20457400-22 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:871: finishing write tid 3 to smithi20457400-22 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:871: finishing write tid 4 to smithi20457400-22 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:870: done (2 left) 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:872: done (1 left) 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:873: copy_from oid 3 from oid 9 current snap is 92 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:871: finishing write tid 5 to smithi20457400-22 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 352 (ObjNum 330 snap 92 seq_num 330) dirty exists 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:871: left oid 22 (ObjNum 330 snap 92 seq_num 330) 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:871: done (1 left) 2024-09-16T05:13:39.827 INFO:tasks.rados.rados.2.smithi204.stdout:874: delete oid 20 current snap is 92 2024-09-16T05:13:39.828 INFO:tasks.rados.rados.2.smithi204.stdout:874: done (1 left) 2024-09-16T05:13:39.828 INFO:tasks.rados.rados.2.smithi204.stdout:875: snap_create 2024-09-16T05:13:39.834 INFO:tasks.rados.rados.2.smithi204.stdout:873: finishing copy_from to smithi20457400-3 2024-09-16T05:13:39.834 INFO:tasks.rados.rados.2.smithi204.stdout:873: got expected ENOENT (src dne) 2024-09-16T05:13:39.834 INFO:tasks.rados.rados.2.smithi204.stdout:873: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:13:39.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:39.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:39.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:39.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:13:40.023 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:39 smithi032 ceph-mon[27388]: pgmap v582: 67 pgs: 67 active+clean; 348 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.3 MiB/s wr, 52 op/s; 15 MiB/s, 19 objects/s recovering 2024-09-16T05:13:40.023 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:39 smithi032 ceph-mon[27388]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:40.023 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:39 smithi032 ceph-mon[27388]: osdmap e287: 12 total, 12 up, 12 in 2024-09-16T05:13:40.024 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:39 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:40.024 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:39 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:40.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:40.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:13:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:40.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:40.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:40.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:39 smithi204 ceph-mon[30933]: pgmap v582: 67 pgs: 67 active+clean; 348 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.3 MiB/s wr, 52 op/s; 15 MiB/s, 19 objects/s recovering 2024-09-16T05:13:40.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:39 smithi204 ceph-mon[30933]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:40.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:39 smithi204 ceph-mon[30933]: osdmap e287: 12 total, 12 up, 12 in 2024-09-16T05:13:40.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:39 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:40.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:39 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:40.129 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:40.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:40.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:40.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:40.231 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:39 smithi196 ceph-mon[30802]: pgmap v582: 67 pgs: 67 active+clean; 348 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.3 MiB/s wr, 52 op/s; 15 MiB/s, 19 objects/s recovering 2024-09-16T05:13:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:39 smithi196 ceph-mon[30802]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:13:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:39 smithi196 ceph-mon[30802]: osdmap e287: 12 total, 12 up, 12 in 2024-09-16T05:13:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:39 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:39 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:40.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:40.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:40.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:40.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:40.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:40.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:40.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:40.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:40.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:40.605 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:40.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:40.677 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pg_num to 66 2024-09-16T05:13:40.680 INFO:tasks.rados.rados.2.smithi204.stdout:873: done (1 left) 2024-09-16T05:13:40.680 INFO:tasks.rados.rados.2.smithi204.stdout:875: done (0 left) 2024-09-16T05:13:40.680 INFO:tasks.rados.rados.2.smithi204.stdout:876: snap_create 2024-09-16T05:13:40.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:40.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:40.700 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:13:40.700 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:13:40.700 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:13:40.701 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:13:40.701 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:13:40.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:40.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:40.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:40.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:40 smithi204 ceph-mon[30933]: osdmap e288: 12 total, 12 up, 12 in 2024-09-16T05:13:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2974176690' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]: dispatch 2024-09-16T05:13:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:40 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2974176690' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]': finished 2024-09-16T05:13:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:40 smithi204 ceph-mon[30933]: osdmap e289: 12 total, 12 up, 12 in 2024-09-16T05:13:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1972971731' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:40.941 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:40.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:41.043 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:41.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:41.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:41.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:40 smithi032 ceph-mon[27388]: osdmap e288: 12 total, 12 up, 12 in 2024-09-16T05:13:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2974176690' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]: dispatch 2024-09-16T05:13:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:40 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2974176690' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]': finished 2024-09-16T05:13:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:40 smithi032 ceph-mon[27388]: osdmap e289: 12 total, 12 up, 12 in 2024-09-16T05:13:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1972971731' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:41.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:41.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:40 smithi196 ceph-mon[30802]: osdmap e288: 12 total, 12 up, 12 in 2024-09-16T05:13:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2974176690' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]: dispatch 2024-09-16T05:13:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:40 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2974176690' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "66"}]': finished 2024-09-16T05:13:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:40 smithi196 ceph-mon[30802]: osdmap e289: 12 total, 12 up, 12 in 2024-09-16T05:13:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1972971731' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:13:41.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:41.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:41.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:41.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:13:41.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:41.446 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:41.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:41.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:41.573 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:41.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:41.631 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:41.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:41.678 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:13:41.681 INFO:tasks.rados.rados.2.smithi204.stdout:876: done (0 left) 2024-09-16T05:13:41.681 INFO:tasks.rados.rados.2.smithi204.stdout:877: rollback oid 30 current snap is 94 2024-09-16T05:13:41.681 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 30 to 93 2024-09-16T05:13:41.681 INFO:tasks.rados.rados.2.smithi204.stdout:878: copy_from oid 50 from oid 46 current snap is 94 2024-09-16T05:13:41.681 INFO:tasks.rados.rados.2.smithi204.stdout:879: write (excl) oid 9 current snap is 94 2024-09-16T05:13:41.682 INFO:tasks.rados.rados.2.smithi204.stdout:879: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-16T05:13:41.688 INFO:tasks.rados.rados.2.smithi204.stdout:879: writing smithi20457400-9 from 796731 to 1257827 tid 1 2024-09-16T05:13:41.694 INFO:tasks.rados.rados.2.smithi204.stdout:879: writing smithi20457400-9 from 1736739 to 2460509 tid 2 2024-09-16T05:13:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:41.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:41.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:41.701 INFO:tasks.rados.rados.2.smithi204.stdout:879: writing smithi20457400-9 from 2928680 to 3596731 tid 3 2024-09-16T05:13:41.701 INFO:tasks.rados.rados.2.smithi204.stdout:880: read oid 8 snap -1 2024-09-16T05:13:41.701 INFO:tasks.rados.rados.2.smithi204.stdout:880: expect (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:13:41.701 INFO:tasks.rados.rados.2.smithi204.stdout:881: snap_remove snap 92 2024-09-16T05:13:41.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:41.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:41.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:41 smithi032 ceph-mon[27388]: pgmap v585: 67 pgs: 67 active+clean; 334 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-16T05:13:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:41 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:41 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2549213704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1972971731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:41.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:41 smithi032 ceph-mon[27388]: osdmap e290: 12 total, 12 up, 12 in 2024-09-16T05:13:41.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:41.944 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:41.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:42.009 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:41 smithi204 ceph-mon[30933]: pgmap v585: 67 pgs: 67 active+clean; 334 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-16T05:13:42.009 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:41 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:42.009 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:41 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:42.009 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2549213704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:42.009 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1972971731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:42.009 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:41 smithi204 ceph-mon[30933]: osdmap e290: 12 total, 12 up, 12 in 2024-09-16T05:13:42.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:42.068 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:42.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:42.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:42.109 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:13:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:13:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:21 2.0 e112/107 2:06458cf1:::smithi20457400-6:head v 112'1, mlcod=0'0)", 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.048847+0000", 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.05802863, 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0052069860000000003, 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.048847+0000", 2024-09-16T05:13:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.048847+0000", 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.048847+0000", 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.67e-07 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.049807+0000", 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00095999799999999999 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.052607+0000", 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002799925 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.052615+0000", 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2490000000000004e-06 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.052675+0000", 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0433999999999998e-05 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.052716+0000", 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0527e-05 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.054017+0000", 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013014260000000001 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.054054+0000", 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7259999999999999e-05 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:23 2.0 e112/107 2:06458cf1:::smithi20457400-6:head v 112'3, mlcod=0'0)", 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.053813991, 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.102016024, 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053064+0000", 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8470000000000001e-06 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053068+0000", 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9519999999999996e-06 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154742+0000", 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.101674714 2024-09-16T05:13:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154751+0000", 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9020000000000005e-06 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154811+0000", 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9329000000000001e-05 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154832+0000", 2024-09-16T05:13:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1174000000000001e-05 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.155048+0000", 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021618599999999999 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.155078+0000", 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9920000000000002e-05 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1487 2.1d 2:b98a79db:::smithi20457400-31:head [write 598382~780511,stat] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:29.673785+0000", 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.433090578999995, 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0052092350000000004, 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1487 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.673785+0000", 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.673785+0000", 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.673788+0000", 2024-09-16T05:13:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.277e-06 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.674256+0000", 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046860199999999998 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.674269+0000", 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.24e-05 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.674277+0000", 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0960000000000001e-06 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.674351+0000", 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3906999999999996e-05 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.674356+0000", 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3480000000000003e-06 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.675463+0000", 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011068779999999999 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.675570+0000", 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107692 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.675784+0000", 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000213062 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.677290+0000", 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001506292 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.678925+0000", 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001635421 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.678965+0000", 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9474999999999997e-05 2024-09-16T05:13:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.678995+0000", 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9785e-05 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1489 2.1d 2:b98a79db:::smithi20457400-31:head [write 3107008~648769] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:29.686838+0000", 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.420037492000006, 2024-09-16T05:13:42.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027239500000000002, 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1489 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.686838+0000", 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.686838+0000", 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.686839+0000", 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4300000000000002e-07 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.687370+0000", 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053054399999999998 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.687381+0000", 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1387e-05 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.687388+0000", 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1999999999999997e-06 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.687433+0000", 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5049000000000002e-05 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.687491+0000", 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7871000000000002e-05 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.687595+0000", 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010385799999999999 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.688672+0000", 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010768889999999999 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.689435+0000", 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076278700000000001 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.689485+0000", 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0188999999999997e-05 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:29.689562+0000", 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7433000000000002e-05 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1514 2.1c 2:3edd0db8:::smithi20457400-24:head [rollback 2f] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:30.665515+0000", 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 71.441360996, 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084191119999999994, 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.131 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1514 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665515+0000", 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665515+0000", 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665517+0000", 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9429999999999999e-06 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665524+0000", 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.267e-06 2024-09-16T05:13:42.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665546+0000", 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1546e-05 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665555+0000", 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7359999999999996e-06 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665650+0000", 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4549000000000006e-05 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665657+0000", 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5449999999999998e-06 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.671508+0000", 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058509449999999998 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.671640+0000", 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013188100000000001 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.673934+0000", 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022937000000000001 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1515 2.1c 2:3edd0db8:::smithi20457400-24:head [write 665610~797860,stat] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:30.793060+0000", 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 71.313815830999999, 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035762839999999999, 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.136 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.137 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.137 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.138 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.138 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.138 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1515 2024-09-16T05:13:42.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.138 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.793060+0000", 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.793060+0000", 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.793063+0000", 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9780000000000001e-06 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.793733+0000", 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00067024900000000004 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.793764+0000", 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0712e-05 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.793778+0000", 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3597999999999999e-05 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.793877+0000", 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9764000000000002e-05 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.793948+0000", 2024-09-16T05:13:42.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0506000000000006e-05 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.794065+0000", 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117076 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.795219+0000", 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001153755 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.796560+0000", 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001341255 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.796600+0000", 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9978000000000002e-05 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.796636+0000", 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6412999999999998e-05 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.143 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1517 2.1c 2:3edd0db8:::smithi20457400-24:head [write 2816700~410210] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:30.802761+0000", 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 71.304115173, 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058433460000000001, 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.144 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1517 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.802761+0000", 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.802761+0000", 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.802762+0000", 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1200000000000001e-06 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.802984+0000", 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022203899999999999 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.802996+0000", 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.201e-05 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803001+0000", 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3480000000000003e-06 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803055+0000", 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4184000000000002e-05 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803144+0000", 2024-09-16T05:13:42.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8487999999999997e-05 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803284+0000", 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000139698 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804207+0000", 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00092351900000000003 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.808509+0000", 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043015659999999997 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.808576+0000", 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7088999999999994e-05 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.808604+0000", 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8285000000000001e-05 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.149 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1518 2.1c 2:3edd0db8:::smithi20457400-24:head [setxattr _header (57) in=7b,truncate 3865610] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:30.803015+0000", 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 71.303860733999997, 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060158909999999998, 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.150 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1518 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803015+0000", 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803015+0000", 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803016+0000", 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8899999999999999e-07 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803019+0000", 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.179e-06 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803025+0000", 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.28e-06 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803029+0000", 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7299999999999999e-06 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803830+0000", 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00080152399999999996 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803906+0000", 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5210000000000007e-05 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804014+0000", 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010878399999999999 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804346+0000", 2024-09-16T05:13:42.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033128699999999999 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.808909+0000", 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004563564 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.808971+0000", 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1913000000000002e-05 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.809031+0000", 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9830999999999997e-05 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1519 2.1c 2:3edd0db8:::smithi20457400-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:30.803051+0000", 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 71.303824930999994, 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006120154, 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1519 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803051+0000", 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803051+0000", 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803051+0000", 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4400000000000001e-07 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803054+0000", 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.261e-06 2024-09-16T05:13:42.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803061+0000", 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3629999999999996e-06 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803066+0000", 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8860000000000003e-06 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804243+0000", 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011766470000000001 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804327+0000", 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4711999999999998e-05 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.809067+0000", 2024-09-16T05:13:42.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047402640000000001 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.809112+0000", 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4477999999999997e-05 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.809171+0000", 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9098999999999998e-05 2024-09-16T05:13:42.161 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1563 2.1d 2:b98a79db:::smithi20457400-31:head [write 749645~774105] snapc 43=[43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2f,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:37.675152+0000", 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 64.431723731999995, 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044947449999999996, 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.162 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1563 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.675152+0000", 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.675152+0000", 2024-09-16T05:13:42.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.675153+0000", 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.375e-06 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.675493+0000", 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033948399999999998 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.675507+0000", 2024-09-16T05:13:42.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3977e-05 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.675515+0000", 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3890000000000005e-06 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.675583+0000", 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7990000000000005e-05 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:13:42.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.675591+0000", 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4919999999999996e-06 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.676794+0000", 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012033359999999999 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.676894+0000", 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9759000000000005e-05 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.677076+0000", 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018160100000000001 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.678784+0000", 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017089640000000001 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.679537+0000", 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075265100000000001 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.679600+0000", 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2947999999999994e-05 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:37.679647+0000", 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6779000000000001e-05 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.9.76:12 2.1c 2:3a3554ba:::smithi20457400-48:head [assert-version v183,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e228)", 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:39.858827+0000", 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 62.248049170000002, 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031498279999999999, 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858827+0000", 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858827+0000", 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858828+0000", 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5099999999999999e-06 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858838+0000", 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8230000000000006e-06 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858861+0000", 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2586e-05 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858869+0000", 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0690000000000002e-06 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858921+0000", 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1956999999999997e-05 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858925+0000", 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.335e-06 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.859920+0000", 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00099493699999999999 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.859969+0000", 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9055999999999999e-05 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.861977+0000", 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020075549999999998 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPush(2.1d 281/280 [PushOp(2:ba21f1a9:::smithi20457400-14:37, version: 249'366, data_included: [2110984~619435,2943668~81592], data_size: 701027, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba21f1a9:::smithi20457400-14:37@249'366, size: 3142909, copy_subset: [(625260, 620939), (1683347, 1459562)], clone_subset: {2:ba21f1a9:::smithi20457400-14:2f: [(0, 625260), (1246199, 437148)]}, snapset: 57=[]:{2f: [27], 37: [35], 4a: [49, 45, 3e, 3a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3142909, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2110984, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:32.810757+0000", 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2961189560000008, 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00300009, 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.810757+0000", 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.810757+0000", 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.810758+0000", 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0299999999999999e-07 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.811425+0000", 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00066701300000000005 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.811452+0000", 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7531000000000001e-05 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.811465+0000", 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2696e-05 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.811565+0000", 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9922999999999999e-05 2024-09-16T05:13:42.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.811579+0000", 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4187e-05 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.813757+0000", 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021781370000000001 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.177 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPush(2.1d 281/280 [PushOp(2:ba754fcb:::smithi20457400-30:39, version: 260'396, data_included: [2161285~198675], data_size: 198675, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba754fcb:::smithi20457400-30:39@260'396, size: 2974839, copy_subset: [(574839, 795192), (1907901, 1066938)], clone_subset: {2:ba754fcb:::smithi20457400-30:2e: [(0, 574839), (1370031, 537870)]}, snapset: 53=[]:{2e: [27], 39: [35], 4c: [49, 45, 3e, 3a], 53: [52, 4d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2974839, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2161285, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:32.827926+0000", 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2789502959999997, 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028007129999999998, 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.827926+0000", 2024-09-16T05:13:42.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.827926+0000", 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.827926+0000", 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.828128+0000", 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000201936 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.828150+0000", 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2154000000000002e-05 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.828160+0000", 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0139000000000001e-05 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.828214+0000", 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3625999999999998e-05 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.828223+0000", 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9039999999999998e-06 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.830726+0000", 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025035000000000001 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.181 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPush(2.1d 281/280 [PushOp(2:ba754fcb:::smithi20457400-30:53, version: 269'414, data_included: [2048525~275017], data_size: 275017, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba754fcb:::smithi20457400-30:53@269'414, size: 2605118, copy_subset: [(574839, 795192), (1617125, 987993)], clone_subset: {2:ba754fcb:::smithi20457400-30:39: [(0, 574839), (1370031, 247094)]}, snapset: 53=[]:{2e: [27], 39: [35], 4c: [49, 45, 3e, 3a], 53: [52, 4d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605118, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2048525, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:32.841495+0000", 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2653807599999993, 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002874883, 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.841495+0000", 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.841495+0000", 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.841496+0000", 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7999999999999996e-07 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.841769+0000", 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027303499999999998 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.841794+0000", 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5758e-05 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.841805+0000", 2024-09-16T05:13:42.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0795000000000001e-05 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.841866+0000", 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0615999999999997e-05 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.841875+0000", 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1929999999999999e-06 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.844370+0000", 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024950060000000001 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPush(2.1c 281/280 [PushOp(2:3d4bedc8:::smithi20457400-22:4c, version: 278'332, data_included: [2134305~17024], data_size: 17024, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3d4bedc8:::smithi20457400-22:4c@278'332, size: 2151329, copy_subset: [(534522, 726606), (1812335, 338994)], clone_subset: {2:3d4bedc8:::smithi20457400-22:45: [(0, 534522), (1261128, 551207)]}, snapset: 4c=[]:{2c: [27], 38: [35], 3a: [3a], 44: [3e], 45: [45], 4c: [49]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2151329, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134305, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:32.849859+0000", 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2570165279999994, 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003596803, 2024-09-16T05:13:42.186 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.188 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.188 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:13:42.189 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.849859+0000", 2024-09-16T05:13:42.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.849859+0000", 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.849860+0000", 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4700000000000002e-07 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.849887+0000", 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6984000000000001e-05 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.849907+0000", 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0309000000000002e-05 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.849917+0000", 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0293e-05 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.850011+0000", 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3344999999999995e-05 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.850026+0000", 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5546999999999999e-05 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:32.853456+0000", 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034299780000000002 2024-09-16T05:13:42.192 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2234 2.1c 2:3d4bedc8:::smithi20457400-22:head [rollback 49] snapc 5c=[5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:38.804351+0000", 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 3.3025245330000002, 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034857170000000002, 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.193 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2234 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804351+0000", 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804351+0000", 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804354+0000", 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3020000000000002e-06 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804361+0000", 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6820000000000003e-06 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804382+0000", 2024-09-16T05:13:42.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0679e-05 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804393+0000", 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0719e-05 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804470+0000", 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7620000000000006e-05 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:13:42.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804479+0000", 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2179999999999999e-06 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.806075+0000", 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015969490000000001 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.806172+0000", 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6553999999999997e-05 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.806407+0000", 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023461200000000001 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.807197+0000", 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00079025099999999995 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.807769+0000", 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057247600000000004 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.807806+0000", 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6646000000000001e-05 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.807837+0000", 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1009000000000001e-05 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2235 2.1c 2:3d4bedc8:::smithi20457400-22:head [append 0~0] snapc 5c=[5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:38.804418+0000", 2024-09-16T05:13:42.199 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 3.3024579040000002, 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035920259999999999, 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2235 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.200 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804418+0000", 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804418+0000", 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804419+0000", 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804422+0000", 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0010000000000002e-06 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804431+0000", 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0180000000000007e-06 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804436+0000", 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5949999999999998e-06 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804547+0000", 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111168 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.804554+0000", 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4529999999999999e-06 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.806794+0000", 2024-09-16T05:13:42.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022402210000000001 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.806890+0000", 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5736000000000006e-05 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.807014+0000", 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012463200000000001 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.807300+0000", 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028606500000000001 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.807942+0000", 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064135899999999998 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.807975+0000", 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3191000000000001e-05 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:38.808010+0000", 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5030999999999997e-05 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2242 2.1c 2:3d4bedc8:::smithi20457400-22:head [write 746310~635349,stat] snapc 5d=[5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:39.813233+0000", 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.293642599, 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045763499999999999, 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.206 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2242 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.813233+0000", 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.207 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.813233+0000", 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.813235+0000", 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9870000000000002e-06 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.813658+0000", 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042235599999999998 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.208 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.813677+0000", 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9758e-05 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.813704+0000", 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6403000000000001e-05 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.209 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.813772+0000", 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8089000000000005e-05 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.813777+0000", 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.656e-06 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.815189+0000", 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014123759999999999 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.210 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.815246+0000", 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6987999999999997e-05 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.815372+0000", 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012662599999999999 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.816673+0000", 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001300524 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.817736+0000", 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001062847 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.817776+0000", 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0120000000000002e-05 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:39.817810+0000", 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.362e-05 2024-09-16T05:13:42.212 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2253 2.1d 2:ba754fcb:::smithi20457400-30:head [append 0~0] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:41.681580+0000", 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.42529588400000001, 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037634109999999999, 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.213 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2253 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681580+0000", 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681580+0000", 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681582+0000", 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.322e-06 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681593+0000", 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0783e-05 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681614+0000", 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0808000000000001e-05 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681625+0000", 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1367e-05 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681683+0000", 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911999999999997e-05 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681688+0000", 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9019999999999998e-06 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.683068+0000", 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013796909999999999 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.683163+0000", 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4920999999999995e-05 2024-09-16T05:13:42.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.683283+0000", 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012043499999999999 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684688+0000", 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001405 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.685251+0000", 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00056309100000000005 2024-09-16T05:13:42.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.685289+0000", 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7379999999999998e-05 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.685343+0000", 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4799000000000002e-05 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2254 2.1d 2:ba754fcb:::smithi20457400-30:head [rollback 5f] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:41.681650+0000", 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.42522539100000001, 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038358020000000001, 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2254 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:13:42.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:13:42.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681650+0000", 2024-09-16T05:13:42.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681650+0000", 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681651+0000", 2024-09-16T05:13:42.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4000000000000001e-07 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681654+0000", 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8990000000000001e-06 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681662+0000", 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7130000000000005e-06 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:13:42.223 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681671+0000", 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7199999999999995e-06 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681753+0000", 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2295999999999999e-05 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681757+0000", 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.777e-06 2024-09-16T05:13:42.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.683806+0000", 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020496339999999998 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.683888+0000", 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1589999999999996e-05 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684028+0000", 2024-09-16T05:13:42.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013982799999999999 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684693+0000", 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00066508099999999998 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.685402+0000", 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00070881399999999999 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:13:42.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.685449+0000", 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7428000000000001e-05 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.685486+0000", 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7282e-05 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:13:42.227 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:13:42.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:41 smithi196 ceph-mon[30802]: pgmap v585: 67 pgs: 67 active+clean; 334 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-16T05:13:42.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:41 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:41 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2549213704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1972971731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:13:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:41 smithi196 ceph-mon[30802]: osdmap e290: 12 total, 12 up, 12 in 2024-09-16T05:13:42.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:42.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:42.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:42.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:42.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:42.408 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:42.408 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:42.429 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:42.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:42.450 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:42.470 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:42.471 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:42.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:42.492 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:42.493 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:42.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:42.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:42.514 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:42.514 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:42.536 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:42.536 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:42.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:42.581 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:42.581 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:42.602 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:42.602 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:42.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:42.623 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:42.624 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:42.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:42.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:42.644 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:42.645 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:42.686 INFO:tasks.rados.rados.2.smithi204.stdout:881: done (4 left) 2024-09-16T05:13:42.686 INFO:tasks.rados.rados.2.smithi204.stdout:882: read oid 4 snap -1 2024-09-16T05:13:42.686 INFO:tasks.rados.rados.2.smithi204.stdout:882: expect (ObjNum 267 snap 75 seq_num 267) 2024-09-16T05:13:42.686 INFO:tasks.rados.rados.2.smithi204.stdout:883: delete oid 44 current snap is 94 2024-09-16T05:13:42.686 INFO:tasks.rados.rados.2.smithi204.stdout:877: finishing rollback tid 0 to smithi20457400-30 2024-09-16T05:13:42.686 INFO:tasks.rados.rados.2.smithi204.stdout:877: finishing rollback tid 1 to smithi20457400-30 2024-09-16T05:13:42.686 INFO:tasks.rados.rados.2.smithi204.stdout:879: finishing write tid 1 to smithi20457400-9 2024-09-16T05:13:42.686 INFO:tasks.rados.rados.2.smithi204.stdout:878: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:879: finishing write tid 2 to smithi20457400-9 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:878: finishing copy_from to smithi20457400-50 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 319 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:879: finishing write tid 3 to smithi20457400-9 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:879: finishing write tid 4 to smithi20457400-9 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:879: finishing write tid 5 to smithi20457400-9 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 320 (ObjNum 333 snap 94 seq_num 333) dirty exists 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:879: left oid 9 (ObjNum 333 snap 94 seq_num 333) 2024-09-16T05:13:42.687 INFO:tasks.rados.rados.2.smithi204.stdout:880: expect (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:13:42.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:42.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:42.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:42.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:42.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:42.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]: dispatch 2024-09-16T05:13:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:42 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:13:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:42 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:13:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:42 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]': finished 2024-09-16T05:13:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:42 smithi032 ceph-mon[27388]: osdmap e291: 12 total, 12 up, 12 in 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:877: finishing rollback tid 2 to smithi20457400-30 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 477 (ObjNum 293 snap 82 seq_num 293) dirty exists 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:877: done (5 left) 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:878: done (4 left) 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:879: done (3 left) 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:880: done (2 left) 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:883: done (1 left) 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:884: read oid 13 snap -1 2024-09-16T05:13:42.922 INFO:tasks.rados.rados.2.smithi204.stdout:884: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-16T05:13:42.923 INFO:tasks.rados.rados.2.smithi204.stdout:885: rollback oid 21 current snap is 94 2024-09-16T05:13:42.923 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 21 to 87 2024-09-16T05:13:42.923 INFO:tasks.rados.rados.2.smithi204.stdout:886: snap_create 2024-09-16T05:13:42.925 INFO:tasks.rados.rados.2.smithi204.stdout:885: finishing rollback tid 0 to smithi20457400-21 2024-09-16T05:13:42.925 INFO:tasks.rados.rados.2.smithi204.stdout:885: finishing rollback tid 1 to smithi20457400-21 2024-09-16T05:13:42.925 INFO:tasks.rados.rados.2.smithi204.stdout:885: finishing rollback tid 2 to smithi20457400-21 2024-09-16T05:13:42.925 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 209 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:13:42.927 INFO:tasks.rados.rados.2.smithi204.stdout:884: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-16T05:13:42.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:13:42.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:13:42.997 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:42.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:13:43.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:43.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:43.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:43.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]: dispatch 2024-09-16T05:13:43.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:42 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:13:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:42 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:13:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:42 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]': finished 2024-09-16T05:13:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:42 smithi204 ceph-mon[30933]: osdmap e291: 12 total, 12 up, 12 in 2024-09-16T05:13:43.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]: dispatch 2024-09-16T05:13:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:42 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:13:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:42 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:13:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:42 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "66"}]': finished 2024-09-16T05:13:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:42 smithi196 ceph-mon[30802]: osdmap e291: 12 total, 12 up, 12 in 2024-09-16T05:13:43.238 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:43.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:43.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:43.377 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:43.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:43.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:43.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:43.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:43.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:43.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:43.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:43.668 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:43.668 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:43.687 INFO:tasks.rados.rados.2.smithi204.stdout:884: done (3 left) 2024-09-16T05:13:43.687 INFO:tasks.rados.rados.2.smithi204.stdout:885: done (2 left) 2024-09-16T05:13:43.687 INFO:tasks.rados.rados.2.smithi204.stdout:886: done (1 left) 2024-09-16T05:13:43.687 INFO:tasks.rados.rados.2.smithi204.stdout:887: delete oid 34 current snap is 95 2024-09-16T05:13:43.689 INFO:tasks.rados.rados.2.smithi204.stdout:887: done (1 left) 2024-09-16T05:13:43.690 INFO:tasks.rados.rados.2.smithi204.stdout:888: write oid 41 current snap is 95 2024-09-16T05:13:43.690 INFO:tasks.rados.rados.2.smithi204.stdout:888: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-16T05:13:43.695 INFO:tasks.rados.rados.2.smithi204.stdout:888: writing smithi20457400-41 from 447152 to 850348 tid 1 2024-09-16T05:13:43.702 INFO:tasks.rados.rados.2.smithi204.stdout:888: writing smithi20457400-41 from 1512185 to 2099635 tid 2 2024-09-16T05:13:43.709 INFO:tasks.rados.rados.2.smithi204.stdout:888: writing smithi20457400-41 from 2727302 to 3408463 tid 3 2024-09-16T05:13:43.709 INFO:tasks.rados.rados.2.smithi204.stdout:889: snap_remove snap 51 2024-09-16T05:13:43.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:43.749 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:43.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:43.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:43.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:43.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:43.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:43 smithi204 ceph-mon[30933]: pgmap v588: 67 pgs: 67 active+clean; 325 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 341 KiB/s wr, 6 op/s 2024-09-16T05:13:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:43 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:13:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:43 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:13:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:43 smithi204 ceph-mon[30933]: osdmap e292: 12 total, 12 up, 12 in 2024-09-16T05:13:43.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:43.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:43.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:44.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:44.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:44.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:44.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:43 smithi032 ceph-mon[27388]: pgmap v588: 67 pgs: 67 active+clean; 325 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 341 KiB/s wr, 6 op/s 2024-09-16T05:13:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:43 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:13:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:43 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:13:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:43 smithi032 ceph-mon[27388]: osdmap e292: 12 total, 12 up, 12 in 2024-09-16T05:13:44.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:44.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:43 smithi196 ceph-mon[30802]: pgmap v588: 67 pgs: 67 active+clean; 325 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 341 KiB/s wr, 6 op/s 2024-09-16T05:13:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:43 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:13:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:43 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:13:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:43 smithi196 ceph-mon[30802]: osdmap e292: 12 total, 12 up, 12 in 2024-09-16T05:13:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:44.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:44.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:44.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:44.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:44.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:13:44.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:13:44.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:44.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:44.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:44.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:44.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:44.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:44.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:44.640 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:44.676 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:44.687 INFO:tasks.rados.rados.2.smithi204.stdout:889: done (2 left) 2024-09-16T05:13:44.687 INFO:tasks.rados.rados.2.smithi204.stdout:890: read oid 34 snap -1 2024-09-16T05:13:44.687 INFO:tasks.rados.rados.2.smithi204.stdout:890: expect deleted 2024-09-16T05:13:44.688 INFO:tasks.rados.rados.2.smithi204.stdout:891: rollback oid 49 current snap is 95 2024-09-16T05:13:44.688 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 71 2024-09-16T05:13:44.688 INFO:tasks.rados.rados.2.smithi204.stdout:892: write oid 17 current snap is 95 2024-09-16T05:13:44.688 INFO:tasks.rados.rados.2.smithi204.stdout:892: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-16T05:13:44.693 INFO:tasks.rados.rados.2.smithi204.stdout:892: writing smithi20457400-17 from 463959 to 942404 tid 1 2024-09-16T05:13:44.703 INFO:tasks.rados.rados.2.smithi204.stdout:892: writing smithi20457400-17 from 1570667 to 2301775 tid 2 2024-09-16T05:13:44.710 INFO:tasks.rados.rados.2.smithi204.stdout:892: writing smithi20457400-17 from 3038372 to 3530805 tid 3 2024-09-16T05:13:44.710 INFO:tasks.rados.rados.2.smithi204.stdout:893: snap_create 2024-09-16T05:13:44.710 INFO:tasks.rados.rados.2.smithi204.stdout:888: finishing write tid 1 to smithi20457400-41 2024-09-16T05:13:44.710 INFO:tasks.rados.rados.2.smithi204.stdout:882: expect (ObjNum 267 snap 75 seq_num 267) 2024-09-16T05:13:44.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:44.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:44.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:44.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:888: finishing write tid 2 to smithi20457400-41 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:888: finishing write tid 3 to smithi20457400-41 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:888: finishing write tid 4 to smithi20457400-41 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:888: finishing write tid 5 to smithi20457400-41 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 324 (ObjNum 336 snap 95 seq_num 336) dirty exists 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:888: left oid 41 (ObjNum 336 snap 95 seq_num 336) 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:892: finishing write tid 1 to smithi20457400-17 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:891: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:891: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:13:44.868 INFO:tasks.rados.rados.2.smithi204.stdout:891: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:13:44.869 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 290 (ObjNum 92 snap 9 seq_num 92) dirty exists 2024-09-16T05:13:44.869 INFO:tasks.rados.rados.2.smithi204.stdout:892: finishing write tid 2 to smithi20457400-17 2024-09-16T05:13:44.869 INFO:tasks.rados.rados.2.smithi204.stdout:892: finishing write tid 3 to smithi20457400-17 2024-09-16T05:13:44.869 INFO:tasks.rados.rados.2.smithi204.stdout:892: finishing write tid 4 to smithi20457400-17 2024-09-16T05:13:44.869 INFO:tasks.rados.rados.2.smithi204.stdout:892: finishing write tid 5 to smithi20457400-17 2024-09-16T05:13:44.869 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 298 (ObjNum 337 snap 95 seq_num 337) dirty exists 2024-09-16T05:13:44.869 INFO:tasks.rados.rados.2.smithi204.stdout:892: left oid 17 (ObjNum 337 snap 95 seq_num 337) 2024-09-16T05:13:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:44.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:44.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:44.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]: dispatch 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:13:45.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]': finished 2024-09-16T05:13:45.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:44 smithi032 ceph-mon[27388]: osdmap e293: 12 total, 12 up, 12 in 2024-09-16T05:13:45.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:45.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:13:45.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:13:45.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:13:45.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:13:45.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:13:45.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:13:45.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]: dispatch 2024-09-16T05:13:45.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:13:45.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:13:45.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:13:45.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:13:45.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]': finished 2024-09-16T05:13:45.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:44 smithi204 ceph-mon[30933]: osdmap e293: 12 total, 12 up, 12 in 2024-09-16T05:13:45.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:45.203 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:45.204 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:45.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:45.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]: dispatch 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "66"}]': finished 2024-09-16T05:13:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:44 smithi196 ceph-mon[30802]: osdmap e293: 12 total, 12 up, 12 in 2024-09-16T05:13:45.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:45.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:13:45.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:45.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:45.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:45.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:45.525 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:45.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:45.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:45.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:13:45.682 INFO:tasks.rados.rados.2.smithi204.stdout:882: done (5 left) 2024-09-16T05:13:45.682 INFO:tasks.rados.rados.2.smithi204.stdout:888: done (4 left) 2024-09-16T05:13:45.682 INFO:tasks.rados.rados.2.smithi204.stdout:891: done (3 left) 2024-09-16T05:13:45.682 INFO:tasks.rados.rados.2.smithi204.stdout:892: done (2 left) 2024-09-16T05:13:45.683 INFO:tasks.rados.rados.2.smithi204.stdout:893: done (1 left) 2024-09-16T05:13:45.683 INFO:tasks.rados.rados.2.smithi204.stdout:894: rollback oid 24 current snap is 96 2024-09-16T05:13:45.683 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 80 2024-09-16T05:13:45.683 INFO:tasks.rados.rados.2.smithi204.stdout:895: write (excl) oid 4 current snap is 96 2024-09-16T05:13:45.683 INFO:tasks.rados.rados.2.smithi204.stdout:895: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-16T05:13:45.688 INFO:tasks.rados.rados.2.smithi204.stdout:895: writing smithi20457400-4 from 480766 to 1034460 tid 1 2024-09-16T05:13:45.695 INFO:tasks.rados.rados.2.smithi204.stdout:895: writing smithi20457400-4 from 1629149 to 2220268 tid 2 2024-09-16T05:13:45.700 INFO:tasks.rados.rados.2.smithi204.stdout:895: writing smithi20457400-4 from 2782148 to 3202206 tid 3 2024-09-16T05:13:45.701 INFO:tasks.rados.rados.2.smithi204.stdout:896: read oid 18 snap -1 2024-09-16T05:13:45.701 INFO:tasks.rados.rados.2.smithi204.stdout:896: expect (ObjNum 288 snap 81 seq_num 288) 2024-09-16T05:13:45.701 INFO:tasks.rados.rados.2.smithi204.stdout:897: copy_from oid 49 from oid 37 current snap is 96 2024-09-16T05:13:45.701 INFO:tasks.rados.rados.2.smithi204.stdout:894: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:13:45.701 INFO:tasks.rados.rados.2.smithi204.stdout:898: snap_remove snap 95 2024-09-16T05:13:45.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:45.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:45.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:45.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:13:45.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:45.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:13:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3737513227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3737513227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:45.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:45 smithi032 ceph-mon[27388]: osdmap e294: 12 total, 12 up, 12 in 2024-09-16T05:13:45.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:45.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:45.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:46.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:46.107 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:13:46.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:46.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:46.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:13:46.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:13:46.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3737513227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3737513227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:46.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:45 smithi204 ceph-mon[30933]: osdmap e294: 12 total, 12 up, 12 in 2024-09-16T05:13:46.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:46.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:46.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:13:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:13:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:13:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:13:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:13:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:13:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:13:46.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3737513227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:46.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:46.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:13:46.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:13:46.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3737513227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:46.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:45 smithi196 ceph-mon[30802]: osdmap e294: 12 total, 12 up, 12 in 2024-09-16T05:13:46.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:46.361 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:46.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:46.381 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:13:46.382 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:13:46.382 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:13:46.382 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-16T05:13:46.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.2 2024-09-16T05:13:46.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:46.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:46.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:46.578 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:46.629 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:46.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:46.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:46.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:898: done (5 left) 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:899: copy_from oid 36 from oid 40 current snap is 96 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:897: finishing copy_from to smithi20457400-49 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:897: got expected ENOENT (src dne) 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:897: finishing copy_from racing read to smithi20457400-49 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:894: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:894: finishing rollback tid 2 to smithi20457400-24 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 356 (ObjNum 258 snap 73 seq_num 258) dirty exists 2024-09-16T05:13:46.686 INFO:tasks.rados.rados.2.smithi204.stdout:896: expect (ObjNum 288 snap 81 seq_num 288) 2024-09-16T05:13:46.696 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:46 smithi032 systemd[1]: Stopping Ceph osd.2 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:13:46.708 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:46.708 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:46.733 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:46.733 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:46.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:46.754 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:13:46.754 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:13:46.775 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:46.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:46.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:46.797 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:46.797 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:46.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:46.843 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:46.843 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:46.867 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:46.867 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:46.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:46.888 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:46.889 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:46.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:46.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:46.911 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:895: finishing write tid 1 to smithi20457400-4 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:895: finishing write tid 2 to smithi20457400-4 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:895: finishing write tid 3 to smithi20457400-4 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:890: done (5 left) 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:894: done (4 left) 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:896: done (3 left) 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:897: done (2 left) 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:900: rollback oid 25 current snap is 96 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 85 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:899: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:13:46.912 INFO:tasks.rados.rados.2.smithi204.stdout:895: finishing write tid 4 to smithi20457400-4 2024-09-16T05:13:46.913 INFO:tasks.rados.rados.2.smithi204.stdout:895: finishing write tid 5 to smithi20457400-4 2024-09-16T05:13:46.913 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 300 (ObjNum 338 snap 96 seq_num 338) dirty exists 2024-09-16T05:13:46.913 INFO:tasks.rados.rados.2.smithi204.stdout:895: left oid 4 (ObjNum 338 snap 96 seq_num 338) 2024-09-16T05:13:46.913 INFO:tasks.rados.rados.2.smithi204.stdout:899: finishing copy_from to smithi20457400-36 2024-09-16T05:13:46.913 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 271 (ObjNum 294 snap 83 seq_num 294) dirty exists 2024-09-16T05:13:46.913 INFO:tasks.rados.rados.2.smithi204.stdout:895: done (2 left) 2024-09-16T05:13:46.913 INFO:tasks.rados.rados.2.smithi204.stdout:899: done (1 left) 2024-09-16T05:13:46.913 INFO:tasks.rados.rados.2.smithi204.stdout:901: snap_remove snap 90 2024-09-16T05:13:46.932 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:46.953 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:46.954 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:46.954 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[52308]: 2024-09-16T05:13:46.941+0000 7f5c79ab9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:13:46.954 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[52308]: 2024-09-16T05:13:46.941+0000 7f5c79ab9640 -1 osd.2 295 *** Got signal Terminated *** 2024-09-16T05:13:46.954 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[52308]: 2024-09-16T05:13:46.941+0000 7f5c79ab9640 -1 osd.2 295 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:13:46.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:13:46.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2963045285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:46.955 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:46 smithi032 ceph-mon[27388]: osdmap e295: 12 total, 12 up, 12 in 2024-09-16T05:13:47.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:47.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:47.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:47.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:47.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:47.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2963045285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:46 smithi204 ceph-mon[30933]: osdmap e295: 12 total, 12 up, 12 in 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:13:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:13:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:13:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:13:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2963045285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:46 smithi196 ceph-mon[30802]: osdmap e295: 12 total, 12 up, 12 in 2024-09-16T05:13:47.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:47.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:47.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:47.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:13:47.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:47.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:47.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:47.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:47.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:47.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:47.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:47.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:47.685 INFO:tasks.rados.rados.2.smithi204.stdout:901: done (1 left) 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:902: read oid 50 snap -1 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:902: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:903: read oid 5 snap -1 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:903: expect deleted 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:904: rollback oid 24 current snap is 96 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 67 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:900: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:900: done (3 left) 2024-09-16T05:13:47.686 INFO:tasks.rados.rados.2.smithi204.stdout:905: delete oid 49 current snap is 96 2024-09-16T05:13:47.689 INFO:tasks.rados.rados.2.smithi204.stdout:904: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:13:47.689 INFO:tasks.rados.rados.2.smithi204.stdout:904: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:13:47.689 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 362 (ObjNum 129 snap 0 seq_num 2671800642) dirty dne 2024-09-16T05:13:47.692 INFO:tasks.rados.rados.2.smithi204.stdout:904: done (3 left) 2024-09-16T05:13:47.692 INFO:tasks.rados.rados.2.smithi204.stdout:905: done (2 left) 2024-09-16T05:13:47.692 INFO:tasks.rados.rados.2.smithi204.stdout:906: rollback oid 21 current snap is 96 2024-09-16T05:13:47.692 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 21 to 87 2024-09-16T05:13:47.692 INFO:tasks.rados.rados.2.smithi204.stdout:907: write (excl) oid 1 current snap is 96 2024-09-16T05:13:47.692 INFO:tasks.rados.rados.2.smithi204.stdout:907: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-16T05:13:47.696 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:47.700 INFO:tasks.rados.rados.2.smithi204.stdout:907: writing smithi20457400-1 from 514380 to 1218572 tid 1 2024-09-16T05:13:47.705 INFO:tasks.rados.rados.2.smithi204.stdout:907: writing smithi20457400-1 from 1629760 to 2108195 tid 2 2024-09-16T05:13:47.713 INFO:tasks.rados.rados.2.smithi204.stdout:907: writing smithi20457400-1 from 2604288 to 3046890 tid 3 2024-09-16T05:13:47.713 INFO:tasks.rados.rados.2.smithi204.stdout:908: snap_remove snap 93 2024-09-16T05:13:47.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:47.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:47.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:47.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:47.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:47.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:47.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:47 smithi032 ceph-mon[27388]: pgmap v594: 77 pgs: 2 peering, 8 unknown, 67 active+clean; 302 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:47.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:47 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:13:47.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:47 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:13:47.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:47 smithi032 ceph-mon[27388]: osd.2 marked itself down and dead 2024-09-16T05:13:47.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:47 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:13:47.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:47 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:13:47.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:47 smithi032 ceph-mon[27388]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T05:13:47.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:47 smithi032 ceph-mon[27388]: osdmap e296: 12 total, 11 up, 12 in 2024-09-16T05:13:47.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:48.013 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:48.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:48.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:48.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:48.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:48.158 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:13:48.158 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:13:48.158 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:13:48.158 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:13:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:47 smithi204 ceph-mon[30933]: pgmap v594: 77 pgs: 2 peering, 8 unknown, 67 active+clean; 302 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:47 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:13:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:47 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:13:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:47 smithi204 ceph-mon[30933]: osd.2 marked itself down and dead 2024-09-16T05:13:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:47 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:13:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:47 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:13:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:47 smithi204 ceph-mon[30933]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T05:13:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:47 smithi204 ceph-mon[30933]: osdmap e296: 12 total, 11 up, 12 in 2024-09-16T05:13:48.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:47 smithi196 ceph-mon[30802]: pgmap v594: 77 pgs: 2 peering, 8 unknown, 67 active+clean; 302 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:47 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:13:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:47 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:13:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:47 smithi196 ceph-mon[30802]: osd.2 marked itself down and dead 2024-09-16T05:13:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:47 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:13:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:47 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:13:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:47 smithi196 ceph-mon[30802]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T05:13:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:47 smithi196 ceph-mon[30802]: osdmap e296: 12 total, 11 up, 12 in 2024-09-16T05:13:48.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:48.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:48.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:48.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:13:48.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:48.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:48.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:48.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:48 smithi032 podman[94917]: 2024-09-16 05:13:48.343049549 +0000 UTC m=+1.508261389 container died 5fc47068d60a754a24a67ded828ce6df4d6ceb4a24df9989f9707fbcabb197b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, ceph=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-16T05:13:48.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:48.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:48.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:48.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:48.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:48.687 INFO:tasks.rados.rados.2.smithi204.stdout:908: done (4 left) 2024-09-16T05:13:48.687 INFO:tasks.rados.rados.2.smithi204.stdout:909: write (excl) oid 29 current snap is 96 2024-09-16T05:13:48.687 INFO:tasks.rados.rados.2.smithi204.stdout:909: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-16T05:13:48.695 INFO:tasks.rados.rados.2.smithi204.stdout:909: writing smithi20457400-29 from 531187 to 1310628 tid 1 2024-09-16T05:13:48.703 INFO:tasks.rados.rados.2.smithi204.stdout:909: writing smithi20457400-29 from 2088242 to 2826688 tid 2 2024-09-16T05:13:48.708 INFO:tasks.rados.rados.2.smithi204.stdout:909: writing smithi20457400-29 from 3431711 to 3731187 tid 3 2024-09-16T05:13:48.708 INFO:tasks.rados.rados.2.smithi204.stdout:910: snap_create 2024-09-16T05:13:48.709 INFO:tasks.rados.rados.2.smithi204.stdout:906: finishing rollback tid 0 to smithi20457400-21 2024-09-16T05:13:48.709 INFO:tasks.rados.rados.2.smithi204.stdout:906: finishing rollback tid 1 to smithi20457400-21 2024-09-16T05:13:48.709 INFO:tasks.rados.rados.2.smithi204.stdout:906: finishing rollback tid 2 to smithi20457400-21 2024-09-16T05:13:48.709 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 217 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-09-16T05:13:48.709 INFO:tasks.rados.rados.2.smithi204.stdout:902: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:13:48.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:48.764 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:48.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:48.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:48.875 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:48 smithi032 podman[94917]: 2024-09-16 05:13:48.715163616 +0000 UTC m=+1.880375453 container cleanup 5fc47068d60a754a24a67ded828ce6df4d6ceb4a24df9989f9707fbcabb197b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:13:48.875 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:48 smithi032 bash[94917]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2 2024-09-16T05:13:48.875 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:48 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:13:48.875 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:48 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:13:48.875 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:48 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:13:48.875 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:48 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:13:48.875 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:48 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:13:48.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:48 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:13:48.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:48 smithi032 ceph-mon[27388]: osdmap e297: 12 total, 11 up, 12 in 2024-09-16T05:13:48.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:48.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:907: finishing write tid 1 to smithi20457400-1 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:907: finishing write tid 2 to smithi20457400-1 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:907: finishing write tid 3 to smithi20457400-1 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:907: finishing write tid 4 to smithi20457400-1 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:907: finishing write tid 5 to smithi20457400-1 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 284 (ObjNum 340 snap 96 seq_num 340) dirty exists 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:907: left oid 1 (ObjNum 340 snap 96 seq_num 340) 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:909: finishing write tid 1 to smithi20457400-29 2024-09-16T05:13:48.972 INFO:tasks.rados.rados.2.smithi204.stdout:909: finishing write tid 2 to smithi20457400-29 2024-09-16T05:13:48.973 INFO:tasks.rados.rados.2.smithi204.stdout:909: finishing write tid 3 to smithi20457400-29 2024-09-16T05:13:48.973 INFO:tasks.rados.rados.2.smithi204.stdout:909: finishing write tid 4 to smithi20457400-29 2024-09-16T05:13:48.973 INFO:tasks.rados.rados.2.smithi204.stdout:909: finishing write tid 5 to smithi20457400-29 2024-09-16T05:13:48.973 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 419 (ObjNum 341 snap 96 seq_num 341) dirty exists 2024-09-16T05:13:48.973 INFO:tasks.rados.rados.2.smithi204.stdout:909: left oid 29 (ObjNum 341 snap 96 seq_num 341) 2024-09-16T05:13:48.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:49.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:49.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:49.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:49.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:49.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:49.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:49.170 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:48 smithi032 podman[95066]: 2024-09-16 05:13:48.87488298 +0000 UTC m=+0.653561716 container remove 5fc47068d60a754a24a67ded828ce6df4d6ceb4a24df9989f9707fbcabb197b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-16T05:13:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:48 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:13:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:48 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:13:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:48 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:13:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:48 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:13:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:48 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:13:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:48 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:13:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:48 smithi204 ceph-mon[30933]: osdmap e297: 12 total, 11 up, 12 in 2024-09-16T05:13:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:48 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:13:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:48 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:13:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:48 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:13:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:48 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:13:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:48 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:13:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:48 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:13:49.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:48 smithi196 ceph-mon[30802]: osdmap e297: 12 total, 11 up, 12 in 2024-09-16T05:13:49.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:49.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:13:49.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:49.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:49.344 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:49.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:49.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:49.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:49.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:49.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:49.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:49.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:49.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:49.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:13:49.686 INFO:tasks.rados.rados.2.smithi204.stdout:902: done (5 left) 2024-09-16T05:13:49.687 INFO:tasks.rados.rados.2.smithi204.stdout:906: done (4 left) 2024-09-16T05:13:49.687 INFO:tasks.rados.rados.2.smithi204.stdout:907: done (3 left) 2024-09-16T05:13:49.687 INFO:tasks.rados.rados.2.smithi204.stdout:909: done (2 left) 2024-09-16T05:13:49.687 INFO:tasks.rados.rados.2.smithi204.stdout:910: done (1 left) 2024-09-16T05:13:49.687 INFO:tasks.rados.rados.2.smithi204.stdout:911: rollback oid 25 current snap is 97 2024-09-16T05:13:49.687 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 86 2024-09-16T05:13:49.687 INFO:tasks.rados.rados.2.smithi204.stdout:912: write oid 15 current snap is 97 2024-09-16T05:13:49.687 INFO:tasks.rados.rados.2.smithi204.stdout:912: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-16T05:13:49.692 INFO:tasks.rados.rados.2.smithi204.stdout:912: writing smithi20457400-15 from 547994 to 1002684 tid 1 2024-09-16T05:13:49.699 INFO:tasks.rados.rados.2.smithi204.stdout:912: writing smithi20457400-15 from 1746724 to 2228828 tid 2 2024-09-16T05:13:49.703 INFO:tasks.rados.rados.2.smithi204.stdout:912: writing smithi20457400-15 from 2659134 to 3124280 tid 3 2024-09-16T05:13:49.707 INFO:tasks.rados.rados.2.smithi204.stdout:912: writing smithi20457400-15 from 3605976 to 3747994 tid 4 2024-09-16T05:13:49.707 INFO:tasks.rados.rados.2.smithi204.stdout:913: write (excl) oid 8 current snap is 97 2024-09-16T05:13:49.707 INFO:tasks.rados.rados.2.smithi204.stdout:911: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:13:49.708 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 225 snap 0 seq_num 3363517365) dirty dne 2024-09-16T05:13:49.708 INFO:tasks.rados.rados.2.smithi204.stdout:912: finishing write tid 1 to smithi20457400-15 2024-09-16T05:13:49.708 INFO:tasks.rados.rados.2.smithi204.stdout:912: finishing write tid 2 to smithi20457400-15 2024-09-16T05:13:49.708 INFO:tasks.rados.rados.2.smithi204.stdout:913: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-16T05:13:49.713 INFO:tasks.rados.rados.2.smithi204.stdout:913: writing smithi20457400-8 from 564801 to 1211093 tid 1 2024-09-16T05:13:49.721 INFO:tasks.rados.rados.2.smithi204.stdout:913: writing smithi20457400-8 from 1921559 to 2663674 tid 2 2024-09-16T05:13:49.727 INFO:tasks.rados.rados.2.smithi204.stdout:913: writing smithi20457400-8 from 3202910 to 3764801 tid 3 2024-09-16T05:13:49.727 INFO:tasks.rados.rados.2.smithi204.stdout:911: done (3 left) 2024-09-16T05:13:49.727 INFO:tasks.rados.rados.2.smithi204.stdout:914: snap_create 2024-09-16T05:13:49.727 INFO:tasks.rados.rados.2.smithi204.stdout:912: finishing write tid 3 to smithi20457400-15 2024-09-16T05:13:49.727 INFO:tasks.rados.rados.2.smithi204.stdout:912: finishing write tid 4 to smithi20457400-15 2024-09-16T05:13:49.727 INFO:tasks.rados.rados.2.smithi204.stdout:912: finishing write tid 5 to smithi20457400-15 2024-09-16T05:13:49.728 INFO:tasks.rados.rados.2.smithi204.stdout:912: finishing write tid 6 to smithi20457400-15 2024-09-16T05:13:49.728 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 91 (ObjNum 342 snap 97 seq_num 342) dirty exists 2024-09-16T05:13:49.728 INFO:tasks.rados.rados.2.smithi204.stdout:912: left oid 15 (ObjNum 342 snap 97 seq_num 342) 2024-09-16T05:13:49.728 INFO:tasks.rados.rados.2.smithi204.stdout:913: finishing write tid 1 to smithi20457400-8 2024-09-16T05:13:49.728 INFO:tasks.rados.rados.2.smithi204.stdout:913: finishing write tid 2 to smithi20457400-8 2024-09-16T05:13:49.730 INFO:tasks.rados.rados.2.smithi204.stdout:913: finishing write tid 3 to smithi20457400-8 2024-09-16T05:13:49.730 INFO:tasks.rados.rados.2.smithi204.stdout:913: finishing write tid 4 to smithi20457400-8 2024-09-16T05:13:49.730 INFO:tasks.rados.rados.2.smithi204.stdout:913: finishing write tid 5 to smithi20457400-8 2024-09-16T05:13:49.730 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 332 (ObjNum 343 snap 97 seq_num 343) dirty exists 2024-09-16T05:13:49.730 INFO:tasks.rados.rados.2.smithi204.stdout:913: left oid 8 (ObjNum 343 snap 97 seq_num 343) 2024-09-16T05:13:49.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:49.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:49.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:49.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:49.813 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:49.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:49.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:49.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: pgmap v597: 77 pgs: 2 stale+active+clean, 2 active+remapped, 3 peering, 3 unknown, 67 active+clean; 302 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: 2.3c scrub starts 2024-09-16T05:13:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: 2.3c scrub ok 2024-09-16T05:13:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: 2.3b scrub starts 2024-09-16T05:13:49.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: 2.3b scrub ok 2024-09-16T05:13:49.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:13:49.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:13:49.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:13:49.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: 2.3d scrub starts 2024-09-16T05:13:49.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: 2.3d scrub ok 2024-09-16T05:13:49.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:49 smithi032 ceph-mon[27388]: osdmap e298: 12 total, 11 up, 12 in 2024-09-16T05:13:49.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:50.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:50.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:50.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:50.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:50.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: pgmap v597: 77 pgs: 2 stale+active+clean, 2 active+remapped, 3 peering, 3 unknown, 67 active+clean; 302 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: 2.3c scrub starts 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: 2.3c scrub ok 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: 2.3b scrub starts 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: 2.3b scrub ok 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: 2.3d scrub starts 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: 2.3d scrub ok 2024-09-16T05:13:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:49 smithi204 ceph-mon[30933]: osdmap e298: 12 total, 11 up, 12 in 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: pgmap v597: 77 pgs: 2 stale+active+clean, 2 active+remapped, 3 peering, 3 unknown, 67 active+clean; 302 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: 2.3c scrub starts 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: 2.3c scrub ok 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: 2.3b scrub starts 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: 2.3b scrub ok 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: 2.3d scrub starts 2024-09-16T05:13:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: 2.3d scrub ok 2024-09-16T05:13:50.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:49 smithi196 ceph-mon[30802]: osdmap e298: 12 total, 11 up, 12 in 2024-09-16T05:13:50.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:50.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:50.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:50.353 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:13:50.353 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:13:50.353 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:13:50.353 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:13:50.354 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:13:50.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:50.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:50.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:50.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:50.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:50.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:50.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:50.638 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:50.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:912: done (3 left) 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:913: done (2 left) 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:914: done (1 left) 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:915: copy_from oid 31 from oid 46 current snap is 98 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:916: rollback oid 49 current snap is 98 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 86 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:917: rollback oid 32 current snap is 98 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 80 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:918: read oid 30 snap 88 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:918: expect (ObjNum 293 snap 82 seq_num 293) 2024-09-16T05:13:50.691 INFO:tasks.rados.rados.2.smithi204.stdout:919: snap_remove snap 87 2024-09-16T05:13:50.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:50.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:50.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:50.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:50.890 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:50.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:50.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:51.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:51.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:51.109 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:50 smithi032 podman[95406]: 2024-09-16 05:13:50.576885149 +0000 UTC m=+0.450888875 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:13:51.109 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:50 smithi032 podman[95406]: 2024-09-16 05:13:50.677812371 +0000 UTC m=+0.551816085 container create 8fe949d4c57866f06283169ba489509f59c6eaf9778131b8877e530fd10dca38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:13:51.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:50 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:13:51.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:50 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:13:51.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:50 smithi032 ceph-mon[27388]: 2.3f scrub starts 2024-09-16T05:13:51.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:50 smithi032 ceph-mon[27388]: 2.3f scrub ok 2024-09-16T05:13:51.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/555647127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:51.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:50 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:51.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/555647127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:51.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:50 smithi032 ceph-mon[27388]: osdmap e299: 12 total, 11 up, 12 in 2024-09-16T05:13:51.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:51.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:51.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:50 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:13:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:50 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:13:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:50 smithi204 ceph-mon[30933]: 2.3f scrub starts 2024-09-16T05:13:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:50 smithi204 ceph-mon[30933]: 2.3f scrub ok 2024-09-16T05:13:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/555647127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:50 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/555647127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:50 smithi204 ceph-mon[30933]: osdmap e299: 12 total, 11 up, 12 in 2024-09-16T05:13:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:50 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:13:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:50 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:13:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:50 smithi196 ceph-mon[30802]: 2.3f scrub starts 2024-09-16T05:13:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:50 smithi196 ceph-mon[30802]: 2.3f scrub ok 2024-09-16T05:13:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/555647127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:13:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:50 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/555647127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:13:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:50 smithi196 ceph-mon[30802]: osdmap e299: 12 total, 11 up, 12 in 2024-09-16T05:13:51.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:51.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:51.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:51.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:51.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:51.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:51.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:51.526 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:51.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:51.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:51.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:51.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:919: done (5 left) 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:920: snap_create 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:917: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:917: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:917: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 423 (ObjNum 248 snap 69 seq_num 248) dirty exists 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:916: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:916: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 301 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:13:51.691 INFO:tasks.rados.rados.2.smithi204.stdout:918: expect (ObjNum 293 snap 82 seq_num 293) 2024-09-16T05:13:51.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:51.780 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:51.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:51.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:13:51.878 INFO:tasks.rados.rados.2.smithi204.stdout:915: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:13:51.878 INFO:tasks.rados.rados.2.smithi204.stdout:915: finishing copy_from to smithi20457400-31 2024-09-16T05:13:51.878 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 319 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:13:51.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:51.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:51.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:52.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:52.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:52.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:52.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:52.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:52.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:51 smithi032 ceph-mon[27388]: pgmap v600: 77 pgs: 2 stale+active+clean, 2 active+remapped, 3 peering, 2 unknown, 68 active+clean; 304 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s; 13 MiB/s, 18 objects/s recovering 2024-09-16T05:13:52.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:51 smithi032 ceph-mon[27388]: 2.3a deep-scrub starts 2024-09-16T05:13:52.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:51 smithi032 ceph-mon[27388]: 2.3a deep-scrub ok 2024-09-16T05:13:52.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:51 smithi032 ceph-mon[27388]: osdmap e300: 12 total, 11 up, 12 in 2024-09-16T05:13:52.160 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:51 smithi032 podman[95406]: 2024-09-16 05:13:51.954122157 +0000 UTC m=+1.828125885 container init 8fe949d4c57866f06283169ba489509f59c6eaf9778131b8877e530fd10dca38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-16T05:13:52.160 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:51 smithi032 podman[95406]: 2024-09-16 05:13:51.95802639 +0000 UTC m=+1.832030111 container start 8fe949d4c57866f06283169ba489509f59c6eaf9778131b8877e530fd10dca38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-16T05:13:52.160 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:52 smithi032 podman[95406]: 2024-09-16 05:13:52.024179642 +0000 UTC m=+1.898183362 container attach 8fe949d4c57866f06283169ba489509f59c6eaf9778131b8877e530fd10dca38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-16T05:13:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:51 smithi204 ceph-mon[30933]: pgmap v600: 77 pgs: 2 stale+active+clean, 2 active+remapped, 3 peering, 2 unknown, 68 active+clean; 304 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s; 13 MiB/s, 18 objects/s recovering 2024-09-16T05:13:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:51 smithi204 ceph-mon[30933]: 2.3a deep-scrub starts 2024-09-16T05:13:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:51 smithi204 ceph-mon[30933]: 2.3a deep-scrub ok 2024-09-16T05:13:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:51 smithi204 ceph-mon[30933]: osdmap e300: 12 total, 11 up, 12 in 2024-09-16T05:13:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:51 smithi196 ceph-mon[30802]: pgmap v600: 77 pgs: 2 stale+active+clean, 2 active+remapped, 3 peering, 2 unknown, 68 active+clean; 304 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s; 13 MiB/s, 18 objects/s recovering 2024-09-16T05:13:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:51 smithi196 ceph-mon[30802]: 2.3a deep-scrub starts 2024-09-16T05:13:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:51 smithi196 ceph-mon[30802]: 2.3a deep-scrub ok 2024-09-16T05:13:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:51 smithi196 ceph-mon[30802]: osdmap e300: 12 total, 11 up, 12 in 2024-09-16T05:13:52.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:52.278 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:52.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:52.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:52.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:52.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:52.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:13:52 smithi032 podman[95406]: 2024-09-16 05:13:52.27442257 +0000 UTC m=+2.148426301 container died 8fe949d4c57866f06283169ba489509f59c6eaf9778131b8877e530fd10dca38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-16T05:13:52.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:52.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:52.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:52.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:52.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:52.692 INFO:tasks.rados.rados.2.smithi204.stdout:915: done (5 left) 2024-09-16T05:13:52.692 INFO:tasks.rados.rados.2.smithi204.stdout:916: done (4 left) 2024-09-16T05:13:52.692 INFO:tasks.rados.rados.2.smithi204.stdout:917: done (3 left) 2024-09-16T05:13:52.693 INFO:tasks.rados.rados.2.smithi204.stdout:918: done (2 left) 2024-09-16T05:13:52.693 INFO:tasks.rados.rados.2.smithi204.stdout:920: done (1 left) 2024-09-16T05:13:52.693 INFO:tasks.rados.rados.2.smithi204.stdout:921: read oid 46 snap -1 2024-09-16T05:13:52.693 INFO:tasks.rados.rados.2.smithi204.stdout:921: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:13:52.693 INFO:tasks.rados.rados.2.smithi204.stdout:922: rollback oid 11 current snap is 99 2024-09-16T05:13:52.693 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 11 to 71 2024-09-16T05:13:52.693 INFO:tasks.rados.rados.2.smithi204.stdout:923: write oid 19 current snap is 99 2024-09-16T05:13:52.694 INFO:tasks.rados.rados.2.smithi204.stdout:923: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-16T05:13:52.699 INFO:tasks.rados.rados.2.smithi204.stdout:923: writing smithi20457400-19 from 581608 to 1303149 tid 1 2024-09-16T05:13:52.705 INFO:tasks.rados.rados.2.smithi204.stdout:923: writing smithi20457400-19 from 1863688 to 2349461 tid 2 2024-09-16T05:13:52.712 INFO:tasks.rados.rados.2.smithi204.stdout:923: writing smithi20457400-19 from 3113980 to 3718023 tid 3 2024-09-16T05:13:52.713 INFO:tasks.rados.rados.2.smithi204.stdout:924: snap_create 2024-09-16T05:13:52.713 INFO:tasks.rados.rados.2.smithi204.stdout:922: finishing rollback tid 0 to smithi20457400-11 2024-09-16T05:13:52.713 INFO:tasks.rados.rados.2.smithi204.stdout:922: finishing rollback tid 1 to smithi20457400-11 2024-09-16T05:13:52.713 INFO:tasks.rados.rados.2.smithi204.stdout:922: finishing rollback tid 2 to smithi20457400-11 2024-09-16T05:13:52.713 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 516 (ObjNum 220 snap 57 seq_num 220) dirty exists 2024-09-16T05:13:52.713 INFO:tasks.rados.rados.2.smithi204.stdout:921: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:13:52.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:52.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:52.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:52.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:52.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:52.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:52.973 INFO:tasks.rados.rados.2.smithi204.stdout:923: finishing write tid 1 to smithi20457400-19 2024-09-16T05:13:52.973 INFO:tasks.rados.rados.2.smithi204.stdout:923: finishing write tid 2 to smithi20457400-19 2024-09-16T05:13:52.973 INFO:tasks.rados.rados.2.smithi204.stdout:923: finishing write tid 3 to smithi20457400-19 2024-09-16T05:13:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:923: finishing write tid 4 to smithi20457400-19 2024-09-16T05:13:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:923: finishing write tid 5 to smithi20457400-19 2024-09-16T05:13:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 284 (ObjNum 344 snap 99 seq_num 344) dirty exists 2024-09-16T05:13:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:923: left oid 19 (ObjNum 344 snap 99 seq_num 344) 2024-09-16T05:13:53.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:53.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:53.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:53.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:53.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:53.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:53.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.2.service' 2024-09-16T05:13:53.254 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:53.264 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:13:53.264 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-16T05:13:53.264 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-09-16T05:13:53.264 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi032.front.sepia.ceph.com 2024-09-16T05:13:53.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.2 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-09-16T05:13:53.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:53.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:53.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:53.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:53.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:53.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:53.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:13:53.503 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.2/config 2024-09-16T05:13:53.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:53.543 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:53.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:53.610 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:13:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:13:53.632 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:13:53.633 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:13:53.633 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:13:53.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:53.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:13:53.653 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:13:53.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:53.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:53.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:53.697 INFO:tasks.rados.rados.2.smithi204.stdout:921: done (4 left) 2024-09-16T05:13:53.697 INFO:tasks.rados.rados.2.smithi204.stdout:922: done (3 left) 2024-09-16T05:13:53.698 INFO:tasks.rados.rados.2.smithi204.stdout:923: done (2 left) 2024-09-16T05:13:53.698 INFO:tasks.rados.rados.2.smithi204.stdout:924: done (1 left) 2024-09-16T05:13:53.698 INFO:tasks.rados.rados.2.smithi204.stdout:925: copy_from oid 6 from oid 45 current snap is 100 2024-09-16T05:13:53.698 INFO:tasks.rados.rados.2.smithi204.stdout:926: snap_remove snap 85 2024-09-16T05:13:53.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:53.776 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:53 smithi032 ceph-mon[27388]: pgmap v603: 77 pgs: 1 undersized+peered, 1 active+clean+remapped, 1 stale+active+clean, 2 peering, 1 unknown, 3 undersized+degraded+peered, 68 active+clean; 303 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 7/288 objects degraded (2.431%) 2024-09-16T05:13:53.776 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:53 smithi032 ceph-mon[27388]: osdmap e301: 12 total, 11 up, 12 in 2024-09-16T05:13:53.776 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2460648046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:53.776 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:53 smithi032 ceph-mon[27388]: Health check failed: Degraded data redundancy: 7/288 objects degraded (2.431%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:13:53.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:53.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:53.876 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:13:53.877 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:13:53.898 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:13:53.898 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:13:53.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:53.921 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:13:53.921 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:13:53.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:53.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:53.943 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:13:53.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:13:53.966 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:13:53.966 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:13:53.987 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:13:53.988 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:13:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:53 smithi196 ceph-mon[30802]: pgmap v603: 77 pgs: 1 undersized+peered, 1 active+clean+remapped, 1 stale+active+clean, 2 peering, 1 unknown, 3 undersized+degraded+peered, 68 active+clean; 303 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 7/288 objects degraded (2.431%) 2024-09-16T05:13:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:53 smithi196 ceph-mon[30802]: osdmap e301: 12 total, 11 up, 12 in 2024-09-16T05:13:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2460648046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:53 smithi196 ceph-mon[30802]: Health check failed: Degraded data redundancy: 7/288 objects degraded (2.431%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:13:54.009 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:13:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:13:54.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:54.030 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:13:54.031 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:13:54.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:54.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:54.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:53 smithi204 ceph-mon[30933]: pgmap v603: 77 pgs: 1 undersized+peered, 1 active+clean+remapped, 1 stale+active+clean, 2 peering, 1 unknown, 3 undersized+degraded+peered, 68 active+clean; 303 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 7/288 objects degraded (2.431%) 2024-09-16T05:13:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:53 smithi204 ceph-mon[30933]: osdmap e301: 12 total, 11 up, 12 in 2024-09-16T05:13:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2460648046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:13:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:53 smithi204 ceph-mon[30933]: Health check failed: Degraded data redundancy: 7/288 objects degraded (2.431%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:13:54.176 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:54.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:54.278 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:54.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:54.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:54.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:54.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:54.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:54.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:54.557 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:54.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:54.656 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:13:54.659 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:54.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:54.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:54.705 INFO:tasks.rados.rados.2.smithi204.stdout:926: done (2 left) 2024-09-16T05:13:54.705 INFO:tasks.rados.rados.2.smithi204.stdout:927: delete oid 31 current snap is 100 2024-09-16T05:13:54.706 INFO:tasks.rados.rados.2.smithi204.stdout:925: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:13:54.706 INFO:tasks.rados.rados.2.smithi204.stdout:925: finishing copy_from to smithi20457400-6 2024-09-16T05:13:54.706 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 323 (ObjNum 302 snap 83 seq_num 302) dirty exists 2024-09-16T05:13:54.713 INFO:tasks.rados.rados.2.smithi204.stdout:925: done (2 left) 2024-09-16T05:13:54.713 INFO:tasks.rados.rados.2.smithi204.stdout:927: done (1 left) 2024-09-16T05:13:54.713 INFO:tasks.rados.rados.2.smithi204.stdout:928: snap_remove snap 67 2024-09-16T05:13:54.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:54.808 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:54.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:54.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:54 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:54 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:54 smithi032 ceph-mon[27388]: osdmap e302: 12 total, 11 up, 12 in 2024-09-16T05:13:54.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:54.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:54.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:54 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:54 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:54 smithi196 ceph-mon[30802]: osdmap e302: 12 total, 11 up, 12 in 2024-09-16T05:13:55.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:55.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:55.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:55.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:54 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:54 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:54 smithi204 ceph-mon[30933]: osdmap e302: 12 total, 11 up, 12 in 2024-09-16T05:13:55.190 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:55.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:13:55.239 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:13:55.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:55.315 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:55.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:55.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:55.439 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:55.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:55.473 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:55.473 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:55.478 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:55.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:55.568 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:55.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:55.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:55.685 INFO:tasks.rados.rados.2.smithi204.stdout:928: done (1 left) 2024-09-16T05:13:55.685 INFO:tasks.rados.rados.2.smithi204.stdout:929: read oid 22 snap -1 2024-09-16T05:13:55.685 INFO:tasks.rados.rados.2.smithi204.stdout:929: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-16T05:13:55.685 INFO:tasks.rados.rados.2.smithi204.stdout:930: snap_remove snap 75 2024-09-16T05:13:55.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:55.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:55.803 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:55 smithi032 ceph-mon[27388]: pgmap v606: 77 pgs: 3 undersized+peered, 1 active+clean+remapped, 4 undersized+degraded+peered, 69 active+clean; 309 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 10/294 objects degraded (3.401%) 2024-09-16T05:13:55.803 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:55 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:13:55.804 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:55 smithi032 ceph-mon[27388]: osdmap e303: 12 total, 11 up, 12 in 2024-09-16T05:13:55.804 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:55 smithi032 ceph-mon[27388]: osdmap e304: 12 total, 11 up, 12 in 2024-09-16T05:13:55.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:55.829 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:55.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:55.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:55.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:55.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:55 smithi196 ceph-mon[30802]: pgmap v606: 77 pgs: 3 undersized+peered, 1 active+clean+remapped, 4 undersized+degraded+peered, 69 active+clean; 309 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 10/294 objects degraded (3.401%) 2024-09-16T05:13:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:55 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:13:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:55 smithi196 ceph-mon[30802]: osdmap e303: 12 total, 11 up, 12 in 2024-09-16T05:13:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:55 smithi196 ceph-mon[30802]: osdmap e304: 12 total, 11 up, 12 in 2024-09-16T05:13:56.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:56.081 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:56.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:55 smithi204 ceph-mon[30933]: pgmap v606: 77 pgs: 3 undersized+peered, 1 active+clean+remapped, 4 undersized+degraded+peered, 69 active+clean; 309 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 10/294 objects degraded (3.401%) 2024-09-16T05:13:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:55 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:13:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:55 smithi204 ceph-mon[30933]: osdmap e303: 12 total, 11 up, 12 in 2024-09-16T05:13:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:55 smithi204 ceph-mon[30933]: osdmap e304: 12 total, 11 up, 12 in 2024-09-16T05:13:56.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:56.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:56.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:56.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:56.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:56.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:56.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:56.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:56.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:56.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:56.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:56.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:56.669 INFO:teuthology.orchestra.run.smithi032.stdout:2.e 2024-09-16T05:13:56.670 INFO:teuthology.orchestra.run.smithi032.stdout:2.3a 2024-09-16T05:13:56.670 INFO:teuthology.orchestra.run.smithi032.stdout:2.3 2024-09-16T05:13:56.670 INFO:teuthology.orchestra.run.smithi032.stdout:2.2b 2024-09-16T05:13:56.670 INFO:teuthology.orchestra.run.smithi032.stdout:2.21 2024-09-16T05:13:56.670 INFO:teuthology.orchestra.run.smithi032.stdout:2.1 2024-09-16T05:13:56.670 INFO:teuthology.orchestra.run.smithi032.stdout:1.8 2024-09-16T05:13:56.670 INFO:teuthology.orchestra.run.smithi032.stdout:1.5 2024-09-16T05:13:56.689 INFO:tasks.rados.rados.2.smithi204.stdout:930: done (2 left) 2024-09-16T05:13:56.689 INFO:tasks.rados.rados.2.smithi204.stdout:931: read oid 45 snap 91 2024-09-16T05:13:56.689 INFO:tasks.rados.rados.2.smithi204.stdout:931: expect (ObjNum 302 snap 83 seq_num 302) 2024-09-16T05:13:56.689 INFO:tasks.rados.rados.2.smithi204.stdout:932: copy_from oid 40 from oid 7 current snap is 100 2024-09-16T05:13:56.689 INFO:tasks.rados.rados.2.smithi204.stdout:933: read oid 19 snap 91 2024-09-16T05:13:56.689 INFO:tasks.rados.rados.2.smithi204.stdout:933: expect (ObjNum 294 snap 83 seq_num 294) 2024-09-16T05:13:56.690 INFO:tasks.rados.rados.2.smithi204.stdout:934: write (excl) oid 24 current snap is 100 2024-09-16T05:13:56.690 INFO:tasks.rados.rados.2.smithi204.stdout:934: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-16T05:13:56.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:56.695 INFO:tasks.rados.rados.2.smithi204.stdout:934: writing smithi20457400-24 from 615222 to 1087261 tid 1 2024-09-16T05:13:56.701 INFO:tasks.rados.rados.2.smithi204.stdout:934: writing smithi20457400-24 from 1580652 to 2070094 tid 2 2024-09-16T05:13:56.709 INFO:tasks.rados.rados.2.smithi204.stdout:934: writing smithi20457400-24 from 2768826 to 3395413 tid 3 2024-09-16T05:13:56.709 INFO:tasks.rados.rados.2.smithi204.stdout:935: read oid 11 snap -1 2024-09-16T05:13:56.709 INFO:tasks.rados.rados.2.smithi204.stdout:935: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:13:56.709 INFO:tasks.rados.rados.2.smithi204.stdout:936: snap_create 2024-09-16T05:13:56.710 INFO:tasks.rados.rados.2.smithi204.stdout:929: expect (ObjNum 330 snap 92 seq_num 330) 2024-09-16T05:13:56.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:56.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:56.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:56.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:13:56.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:56.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:56.923 INFO:tasks.rados.rados.2.smithi204.stdout:931: expect (ObjNum 302 snap 83 seq_num 302) 2024-09-16T05:13:56.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:56.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:56.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:56.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:57.058 INFO:tasks.rados.rados.2.smithi204.stdout:934: finishing write tid 1 to smithi20457400-24 2024-09-16T05:13:57.059 INFO:tasks.rados.rados.2.smithi204.stdout:933: expect (ObjNum 294 snap 83 seq_num 294) 2024-09-16T05:13:57.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:57.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:57.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:13:57.158 INFO:tasks.rados.rados.2.smithi204.stdout:934: finishing write tid 2 to smithi20457400-24 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:932: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:932: finishing copy_from to smithi20457400-40 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 191 (ObjNum 298 snap 83 seq_num 298) dirty exists 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:934: finishing write tid 3 to smithi20457400-24 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:934: finishing write tid 4 to smithi20457400-24 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:934: finishing write tid 5 to smithi20457400-24 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 378 (ObjNum 346 snap 100 seq_num 346) dirty exists 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:934: left oid 24 (ObjNum 346 snap 100 seq_num 346) 2024-09-16T05:13:57.159 INFO:tasks.rados.rados.2.smithi204.stdout:935: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:13:57.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:57.229 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:57.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:57.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:13:57.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:13:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:57.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:57.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:13:57.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:57.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:57.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:57.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:57.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:57.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:57.706 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:13:57.719 INFO:tasks.rados.rados.2.smithi204.stdout:929: done (7 left) 2024-09-16T05:13:57.719 INFO:tasks.rados.rados.2.smithi204.stdout:931: done (6 left) 2024-09-16T05:13:57.719 INFO:tasks.rados.rados.2.smithi204.stdout:932: done (5 left) 2024-09-16T05:13:57.719 INFO:tasks.rados.rados.2.smithi204.stdout:933: done (4 left) 2024-09-16T05:13:57.719 INFO:tasks.rados.rados.2.smithi204.stdout:934: done (3 left) 2024-09-16T05:13:57.719 INFO:tasks.rados.rados.2.smithi204.stdout:935: done (2 left) 2024-09-16T05:13:57.719 INFO:tasks.rados.rados.2.smithi204.stdout:936: done (1 left) 2024-09-16T05:13:57.720 INFO:tasks.rados.rados.2.smithi204.stdout:937: read oid 3 snap -1 2024-09-16T05:13:57.720 INFO:tasks.rados.rados.2.smithi204.stdout:937: expect (ObjNum 308 snap 86 seq_num 308) 2024-09-16T05:13:57.720 INFO:tasks.rados.rados.2.smithi204.stdout:938: write oid 13 current snap is 101 2024-09-16T05:13:57.720 INFO:tasks.rados.rados.2.smithi204.stdout:938: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-16T05:13:57.725 INFO:tasks.rados.rados.2.smithi204.stdout:938: writing smithi20457400-13 from 632029 to 1179317 tid 1 2024-09-16T05:13:57.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:57.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:57.732 INFO:tasks.rados.rados.2.smithi204.stdout:938: writing smithi20457400-13 from 1639134 to 2388587 tid 2 2024-09-16T05:13:57.737 INFO:tasks.rados.rados.2.smithi204.stdout:938: writing smithi20457400-13 from 2796249 to 3234108 tid 3 2024-09-16T05:13:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:938: writing smithi20457400-13 from 3653524 to 3832029 tid 4 2024-09-16T05:13:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:939: delete oid 9 current snap is 101 2024-09-16T05:13:57.740 INFO:tasks.rados.rados.2.smithi204.stdout:937: expect (ObjNum 308 snap 86 seq_num 308) 2024-09-16T05:13:57.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:57.846 INFO:tasks.rados.rados.2.smithi204.stdout:938: finishing write tid 1 to smithi20457400-13 2024-09-16T05:13:57.847 INFO:tasks.rados.rados.2.smithi204.stdout:938: finishing write tid 2 to smithi20457400-13 2024-09-16T05:13:57.847 INFO:tasks.rados.rados.2.smithi204.stdout:938: finishing write tid 3 to smithi20457400-13 2024-09-16T05:13:57.847 INFO:tasks.rados.rados.2.smithi204.stdout:938: finishing write tid 4 to smithi20457400-13 2024-09-16T05:13:57.847 INFO:tasks.rados.rados.2.smithi204.stdout:938: finishing write tid 5 to smithi20457400-13 2024-09-16T05:13:57.847 INFO:tasks.rados.rados.2.smithi204.stdout:938: finishing write tid 6 to smithi20457400-13 2024-09-16T05:13:57.847 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 405 (ObjNum 347 snap 101 seq_num 347) dirty exists 2024-09-16T05:13:57.848 INFO:tasks.rados.rados.2.smithi204.stdout:938: left oid 13 (ObjNum 347 snap 101 seq_num 347) 2024-09-16T05:13:57.848 INFO:tasks.rados.rados.2.smithi204.stdout:937: done (3 left) 2024-09-16T05:13:57.848 INFO:tasks.rados.rados.2.smithi204.stdout:938: done (2 left) 2024-09-16T05:13:57.848 INFO:tasks.rados.rados.2.smithi204.stdout:939: done (1 left) 2024-09-16T05:13:57.848 INFO:tasks.rados.rados.2.smithi204.stdout:940: rollback oid 4 current snap is 101 2024-09-16T05:13:57.848 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 98 2024-09-16T05:13:57.848 INFO:tasks.rados.rados.2.smithi204.stdout:941: write oid 10 current snap is 101 2024-09-16T05:13:57.849 INFO:tasks.rados.rados.2.smithi204.stdout:941: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-09-16T05:13:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:57.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:57.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:57.854 INFO:tasks.rados.rados.2.smithi204.stdout:941: writing smithi20457400-10 from 665643 to 1363429 tid 1 2024-09-16T05:13:57.861 INFO:tasks.rados.rados.2.smithi204.stdout:941: writing smithi20457400-10 from 2039745 to 2792867 tid 2 2024-09-16T05:13:57.867 INFO:tasks.rados.rados.2.smithi204.stdout:941: writing smithi20457400-10 from 3534742 to 3865643 tid 3 2024-09-16T05:13:57.867 INFO:tasks.rados.rados.2.smithi204.stdout:942: snap_create 2024-09-16T05:13:57.867 INFO:tasks.rados.rados.2.smithi204.stdout:940: finishing rollback tid 0 to smithi20457400-4 2024-09-16T05:13:57.868 INFO:tasks.rados.rados.2.smithi204.stdout:940: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:13:57.868 INFO:tasks.rados.rados.2.smithi204.stdout:940: finishing rollback tid 2 to smithi20457400-4 2024-09-16T05:13:57.868 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 316 (ObjNum 338 snap 96 seq_num 338) dirty exists 2024-09-16T05:13:57.868 INFO:tasks.rados.rados.2.smithi204.stdout:941: finishing write tid 1 to smithi20457400-10 2024-09-16T05:13:57.868 INFO:tasks.rados.rados.2.smithi204.stdout:941: finishing write tid 2 to smithi20457400-10 2024-09-16T05:13:57.869 INFO:tasks.rados.rados.2.smithi204.stdout:941: finishing write tid 3 to smithi20457400-10 2024-09-16T05:13:57.869 INFO:tasks.rados.rados.2.smithi204.stdout:941: finishing write tid 4 to smithi20457400-10 2024-09-16T05:13:57.869 INFO:tasks.rados.rados.2.smithi204.stdout:941: finishing write tid 5 to smithi20457400-10 2024-09-16T05:13:57.869 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 356 (ObjNum 349 snap 101 seq_num 349) dirty exists 2024-09-16T05:13:57.870 INFO:tasks.rados.rados.2.smithi204.stdout:941: left oid 10 (ObjNum 349 snap 101 seq_num 349) 2024-09-16T05:13:57.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:57.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:57.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:57.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:57.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: osdmap e305: 12 total, 11 up, 12 in 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:57.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:13:57.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/86169045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:58.044 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: osdmap e305: 12 total, 11 up, 12 in 2024-09-16T05:13:58.044 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:13:58.044 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:58.045 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/86169045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:58.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.2 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 1.8 --file /var/log/ceph/exp.1.8.2 2024-09-16T05:13:58.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:58.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:58.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: osdmap e305: 12 total, 11 up, 12 in 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:13:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:13:58.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:13:58.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:13:58.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:13:58.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/86169045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:13:58.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:13:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:58.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:58.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:13:58.268 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.2/config 2024-09-16T05:13:58.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:58.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:58.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:58.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:13:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:58.483 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:58.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:13:58.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:13:58.609 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:58.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:13:58.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:58.712 INFO:tasks.rados.rados.2.smithi204.stdout:940: done (3 left) 2024-09-16T05:13:58.712 INFO:tasks.rados.rados.2.smithi204.stdout:941: done (2 left) 2024-09-16T05:13:58.712 INFO:tasks.rados.rados.2.smithi204.stdout:942: done (1 left) 2024-09-16T05:13:58.712 INFO:tasks.rados.rados.2.smithi204.stdout:943: write oid 45 current snap is 102 2024-09-16T05:13:58.712 INFO:tasks.rados.rados.2.smithi204.stdout:943: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-16T05:13:58.717 INFO:tasks.rados.rados.2.smithi204.stdout:943: writing smithi20457400-45 from 682450 to 1171838 tid 1 2024-09-16T05:13:58.723 INFO:tasks.rados.rados.2.smithi204.stdout:943: writing smithi20457400-45 from 1814580 to 2311360 tid 2 2024-09-16T05:13:58.730 INFO:tasks.rados.rados.2.smithi204.stdout:943: writing smithi20457400-45 from 2878518 to 3666546 tid 3 2024-09-16T05:13:58.730 INFO:tasks.rados.rados.2.smithi204.stdout:944: copy_from oid 34 from oid 3 current snap is 102 2024-09-16T05:13:58.731 INFO:tasks.rados.rados.2.smithi204.stdout:943: finishing write tid 1 to smithi20457400-45 2024-09-16T05:13:58.731 INFO:tasks.rados.rados.2.smithi204.stdout:943: finishing write tid 2 to smithi20457400-45 2024-09-16T05:13:58.731 INFO:tasks.rados.rados.2.smithi204.stdout:945: rollback oid 22 current snap is 102 2024-09-16T05:13:58.731 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 86 2024-09-16T05:13:58.731 INFO:tasks.rados.rados.2.smithi204.stdout:946: read oid 9 snap -1 2024-09-16T05:13:58.731 INFO:tasks.rados.rados.2.smithi204.stdout:946: expect deleted 2024-09-16T05:13:58.731 INFO:tasks.rados.rados.2.smithi204.stdout:947: snap_remove snap 99 2024-09-16T05:13:58.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:58.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:58.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:13:58.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:58.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:13:58.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:58.968 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:58 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:58.968 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/86169045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:58.968 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:58 smithi032 ceph-mon[27388]: osdmap e306: 12 total, 11 up, 12 in 2024-09-16T05:13:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:58.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:58.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:13:58.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:58 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:58.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/86169045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:58.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:58 smithi196 ceph-mon[30802]: osdmap e306: 12 total, 11 up, 12 in 2024-09-16T05:13:59.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:13:59.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:13:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:59.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:13:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:58 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:13:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/86169045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:13:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:58 smithi204 ceph-mon[30933]: osdmap e306: 12 total, 11 up, 12 in 2024-09-16T05:13:59.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:13:59.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:59.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:13:59.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:59.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:59.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:59.373 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:13:59.373 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:13:59.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:59.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:59.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:59.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:59.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:13:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:59.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:59.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:13:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:947: done (5 left) 2024-09-16T05:13:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:948: rollback oid 33 current snap is 102 2024-09-16T05:13:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 33 to 96 2024-09-16T05:13:59.714 INFO:tasks.rados.rados.2.smithi204.stdout:945: finishing rollback tid 0 to smithi20457400-22 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:945: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 381 (ObjNum 65 snap 0 seq_num 0) dirty dne 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:943: finishing write tid 3 to smithi20457400-45 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:943: finishing write tid 4 to smithi20457400-45 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:943: finishing write tid 5 to smithi20457400-45 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 327 (ObjNum 350 snap 102 seq_num 350) dirty exists 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:943: left oid 45 (ObjNum 350 snap 102 seq_num 350) 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:944: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:943: done (5 left) 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:945: done (4 left) 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:946: done (3 left) 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:949: snap_create 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:944: finishing copy_from to smithi20457400-34 2024-09-16T05:13:59.715 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 446 (ObjNum 308 snap 86 seq_num 308) dirty exists 2024-09-16T05:13:59.718 INFO:tasks.rados.rados.2.smithi204.stdout:948: finishing rollback tid 0 to smithi20457400-33 2024-09-16T05:13:59.718 INFO:tasks.rados.rados.2.smithi204.stdout:948: finishing rollback tid 1 to smithi20457400-33 2024-09-16T05:13:59.719 INFO:tasks.rados.rados.2.smithi204.stdout:948: finishing rollback tid 2 to smithi20457400-33 2024-09-16T05:13:59.719 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 371 (ObjNum 283 snap 80 seq_num 283) dirty exists 2024-09-16T05:13:59.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:13:59.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:13:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:59.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:13:59.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:13:59.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:13:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:13:59.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:13:59.956 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:59 smithi032 ceph-mon[27388]: pgmap v612: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 300 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 10/274 objects degraded (3.650%) 2024-09-16T05:13:59.956 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:59 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 10/274 objects degraded (3.650%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:13:59.956 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:59 smithi032 ceph-mon[27388]: osdmap e307: 12 total, 11 up, 12 in 2024-09-16T05:13:59.956 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:59 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:13:59.957 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:13:59 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:13:59.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:13:59.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:13:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:59 smithi196 ceph-mon[30802]: pgmap v612: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 300 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 10/274 objects degraded (3.650%) 2024-09-16T05:13:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:59 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 10/274 objects degraded (3.650%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:13:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:59 smithi196 ceph-mon[30802]: osdmap e307: 12 total, 11 up, 12 in 2024-09-16T05:13:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:59 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:13:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:13:59 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:14:00.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:00.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:00.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:00.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:00.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:59 smithi204 ceph-mon[30933]: pgmap v612: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 300 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 10/274 objects degraded (3.650%) 2024-09-16T05:14:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:59 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 10/274 objects degraded (3.650%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:59 smithi204 ceph-mon[30933]: osdmap e307: 12 total, 11 up, 12 in 2024-09-16T05:14:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:59 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:14:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:13:59 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:14:00.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:00.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:00.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:00.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:00.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:00.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:00.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:14:00.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:00.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:00.524 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:00.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:00.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:14:00.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:00.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:00.724 INFO:tasks.rados.rados.2.smithi204.stdout:944: done (3 left) 2024-09-16T05:14:00.724 INFO:tasks.rados.rados.2.smithi204.stdout:948: done (2 left) 2024-09-16T05:14:00.724 INFO:tasks.rados.rados.2.smithi204.stdout:949: done (1 left) 2024-09-16T05:14:00.724 INFO:tasks.rados.rados.2.smithi204.stdout:950: copy_from oid 22 from oid 47 current snap is 103 2024-09-16T05:14:00.724 INFO:tasks.rados.rados.2.smithi204.stdout:951: write oid 36 current snap is 103 2024-09-16T05:14:00.724 INFO:tasks.rados.rados.2.smithi204.stdout:951: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-16T05:14:00.731 INFO:tasks.rados.rados.2.smithi204.stdout:951: writing smithi20457400-36 from 699257 to 1263894 tid 1 2024-09-16T05:14:00.738 INFO:tasks.rados.rados.2.smithi204.stdout:951: writing smithi20457400-36 from 1873062 to 2513500 tid 2 2024-09-16T05:14:00.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:00.746 INFO:tasks.rados.rados.2.smithi204.stdout:951: writing smithi20457400-36 from 3305941 to 3899257 tid 3 2024-09-16T05:14:00.746 INFO:tasks.rados.rados.2.smithi204.stdout:952: read oid 29 snap -1 2024-09-16T05:14:00.746 INFO:tasks.rados.rados.2.smithi204.stdout:952: expect (ObjNum 341 snap 96 seq_num 341) 2024-09-16T05:14:00.746 INFO:tasks.rados.rados.2.smithi204.stdout:953: delete oid 31 current snap is 103 2024-09-16T05:14:00.746 INFO:tasks.rados.rados.2.smithi204.stdout:950: finishing copy_from racing read to smithi20457400-22 2024-09-16T05:14:00.746 INFO:tasks.rados.rados.2.smithi204.stdout:951: finishing write tid 1 to smithi20457400-36 2024-09-16T05:14:00.746 INFO:tasks.rados.rados.2.smithi204.stdout:950: finishing copy_from to smithi20457400-22 2024-09-16T05:14:00.747 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 288 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:14:00.747 INFO:tasks.rados.rados.2.smithi204.stdout:951: finishing write tid 2 to smithi20457400-36 2024-09-16T05:14:00.749 INFO:tasks.rados.rados.2.smithi204.stdout:951: finishing write tid 3 to smithi20457400-36 2024-09-16T05:14:00.749 INFO:tasks.rados.rados.2.smithi204.stdout:951: finishing write tid 4 to smithi20457400-36 2024-09-16T05:14:00.750 INFO:tasks.rados.rados.2.smithi204.stdout:951: finishing write tid 5 to smithi20457400-36 2024-09-16T05:14:00.750 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 419 (ObjNum 351 snap 103 seq_num 351) dirty exists 2024-09-16T05:14:00.750 INFO:tasks.rados.rados.2.smithi204.stdout:951: left oid 36 (ObjNum 351 snap 103 seq_num 351) 2024-09-16T05:14:00.750 INFO:tasks.rados.rados.2.smithi204.stdout:950: done (4 left) 2024-09-16T05:14:00.750 INFO:tasks.rados.rados.2.smithi204.stdout:951: done (3 left) 2024-09-16T05:14:00.750 INFO:tasks.rados.rados.2.smithi204.stdout:953: done (2 left) 2024-09-16T05:14:00.750 INFO:tasks.rados.rados.2.smithi204.stdout:954: snap_create 2024-09-16T05:14:00.753 INFO:tasks.rados.rados.2.smithi204.stdout:952: expect (ObjNum 341 snap 96 seq_num 341) 2024-09-16T05:14:00.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:00.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:00.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:00.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:00.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:00.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:00 smithi032 ceph-mon[27388]: osdmap e308: 12 total, 11 up, 12 in 2024-09-16T05:14:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:00 smithi196 ceph-mon[30802]: osdmap e308: 12 total, 11 up, 12 in 2024-09-16T05:14:00.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:01.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:01.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:01.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:01.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:01.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:00 smithi204 ceph-mon[30933]: osdmap e308: 12 total, 11 up, 12 in 2024-09-16T05:14:01.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:01.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:01.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:01.370 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:01.372 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:01.377 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:01.377 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:01.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:01.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:01.399 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:01.400 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:01.400 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:14:01.421 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:01.422 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:01.443 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:01.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:01.488 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:01.488 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:01.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:01.509 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:01.510 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:01.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:01.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:01.534 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:01.534 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:01.556 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:01.556 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:01.577 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:01.577 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:01.598 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:01.598 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:01.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:01.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:01.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:01.724 INFO:tasks.rados.rados.2.smithi204.stdout:952: done (2 left) 2024-09-16T05:14:01.725 INFO:tasks.rados.rados.2.smithi204.stdout:954: done (1 left) 2024-09-16T05:14:01.725 INFO:tasks.rados.rados.2.smithi204.stdout:955: copy_from oid 12 from oid 7 current snap is 104 2024-09-16T05:14:01.725 INFO:tasks.rados.rados.2.smithi204.stdout:956: read oid 2 snap -1 2024-09-16T05:14:01.725 INFO:tasks.rados.rados.2.smithi204.stdout:956: expect deleted 2024-09-16T05:14:01.725 INFO:tasks.rados.rados.2.smithi204.stdout:957: write oid 25 current snap is 104 2024-09-16T05:14:01.725 INFO:tasks.rados.rados.2.smithi204.stdout:957: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-16T05:14:01.732 INFO:tasks.rados.rados.2.smithi204.stdout:957: writing smithi20457400-25 from 732871 to 1448006 tid 1 2024-09-16T05:14:01.738 INFO:tasks.rados.rados.2.smithi204.stdout:957: writing smithi20457400-25 from 1873673 to 2517780 tid 2 2024-09-16T05:14:01.746 INFO:tasks.rados.rados.2.smithi204.stdout:957: writing smithi20457400-25 from 3244434 to 3932871 tid 3 2024-09-16T05:14:01.746 INFO:tasks.rados.rados.2.smithi204.stdout:958: delete oid 13 current snap is 104 2024-09-16T05:14:01.746 INFO:tasks.rados.rados.2.smithi204.stdout:957: finishing write tid 1 to smithi20457400-25 2024-09-16T05:14:01.746 INFO:tasks.rados.rados.2.smithi204.stdout:955: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:14:01.746 INFO:tasks.rados.rados.2.smithi204.stdout:957: finishing write tid 2 to smithi20457400-25 2024-09-16T05:14:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:955: finishing copy_from to smithi20457400-12 2024-09-16T05:14:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 191 (ObjNum 298 snap 83 seq_num 298) dirty exists 2024-09-16T05:14:01.749 INFO:tasks.rados.rados.2.smithi204.stdout:957: finishing write tid 3 to smithi20457400-25 2024-09-16T05:14:01.749 INFO:tasks.rados.rados.2.smithi204.stdout:957: finishing write tid 4 to smithi20457400-25 2024-09-16T05:14:01.749 INFO:tasks.rados.rados.2.smithi204.stdout:957: finishing write tid 5 to smithi20457400-25 2024-09-16T05:14:01.749 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 375 (ObjNum 353 snap 104 seq_num 353) dirty exists 2024-09-16T05:14:01.749 INFO:tasks.rados.rados.2.smithi204.stdout:957: left oid 25 (ObjNum 353 snap 104 seq_num 353) 2024-09-16T05:14:01.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:01.750 INFO:tasks.rados.rados.2.smithi204.stdout:955: done (4 left) 2024-09-16T05:14:01.750 INFO:tasks.rados.rados.2.smithi204.stdout:956: done (3 left) 2024-09-16T05:14:01.750 INFO:tasks.rados.rados.2.smithi204.stdout:957: done (2 left) 2024-09-16T05:14:01.751 INFO:tasks.rados.rados.2.smithi204.stdout:958: done (1 left) 2024-09-16T05:14:01.751 INFO:tasks.rados.rados.2.smithi204.stdout:959: write oid 9 current snap is 104 2024-09-16T05:14:01.751 INFO:tasks.rados.rados.2.smithi204.stdout:959: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-09-16T05:14:01.756 INFO:tasks.rados.rados.2.smithi204.stdout:959: writing smithi20457400-9 from 766485 to 1232118 tid 1 2024-09-16T05:14:01.763 INFO:tasks.rados.rados.2.smithi204.stdout:959: writing smithi20457400-9 from 1990637 to 2638413 tid 2 2024-09-16T05:14:01.770 INFO:tasks.rados.rados.2.smithi204.stdout:959: writing smithi20457400-9 from 3299280 to 3966485 tid 3 2024-09-16T05:14:01.771 INFO:tasks.rados.rados.2.smithi204.stdout:960: read oid 7 snap -1 2024-09-16T05:14:01.771 INFO:tasks.rados.rados.2.smithi204.stdout:960: expect (ObjNum 298 snap 83 seq_num 298) 2024-09-16T05:14:01.771 INFO:tasks.rados.rados.2.smithi204.stdout:961: read oid 31 snap -1 2024-09-16T05:14:01.771 INFO:tasks.rados.rados.2.smithi204.stdout:961: expect deleted 2024-09-16T05:14:01.771 INFO:tasks.rados.rados.2.smithi204.stdout:962: write oid 20 current snap is 104 2024-09-16T05:14:01.771 INFO:tasks.rados.rados.2.smithi204.stdout:959: finishing write tid 1 to smithi20457400-9 2024-09-16T05:14:01.771 INFO:tasks.rados.rados.2.smithi204.stdout:959: finishing write tid 2 to smithi20457400-9 2024-09-16T05:14:01.771 INFO:tasks.rados.rados.2.smithi204.stdout:962: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-09-16T05:14:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:01.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:01.778 INFO:tasks.rados.rados.2.smithi204.stdout:962: writing smithi20457400-20 from 783292 to 1324174 tid 1 2024-09-16T05:14:01.784 INFO:tasks.rados.rados.2.smithi204.stdout:962: writing smithi20457400-20 from 1932766 to 2440553 tid 2 2024-09-16T05:14:01.791 INFO:tasks.rados.rados.2.smithi204.stdout:962: writing smithi20457400-20 from 3210350 to 3782363 tid 3 2024-09-16T05:14:01.792 INFO:tasks.rados.rados.2.smithi204.stdout:963: write oid 34 current snap is 104 2024-09-16T05:14:01.792 INFO:tasks.rados.rados.2.smithi204.stdout:963: seq_num 357 ranges {400099=616131,1591248=408851} 2024-09-16T05:14:01.797 INFO:tasks.rados.rados.2.smithi204.stdout:963: writing smithi20457400-34 from 400099 to 1016230 tid 1 2024-09-16T05:14:01.802 INFO:tasks.rados.rados.2.smithi204.stdout:963: writing smithi20457400-34 from 1591248 to 2000099 tid 2 2024-09-16T05:14:01.802 INFO:tasks.rados.rados.2.smithi204.stdout:964: write oid 45 current snap is 104 2024-09-16T05:14:01.803 INFO:tasks.rados.rados.2.smithi204.stdout:964: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-16T05:14:01.807 INFO:tasks.rados.rados.2.smithi204.stdout:964: writing smithi20457400-45 from 416906 to 824639 tid 1 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:964: writing smithi20457400-45 from 1366083 to 1877539 tid 2 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:965: read oid 6 snap -1 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:965: expect (ObjNum 302 snap 83 seq_num 302) 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:966: snap_create 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:959: finishing write tid 3 to smithi20457400-9 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:959: finishing write tid 4 to smithi20457400-9 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:959: finishing write tid 5 to smithi20457400-9 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 362 (ObjNum 355 snap 104 seq_num 355) dirty exists 2024-09-16T05:14:01.813 INFO:tasks.rados.rados.2.smithi204.stdout:959: left oid 9 (ObjNum 355 snap 104 seq_num 355) 2024-09-16T05:14:01.814 INFO:tasks.rados.rados.2.smithi204.stdout:960: expect (ObjNum 298 snap 83 seq_num 298) 2024-09-16T05:14:01.875 INFO:teuthology.orchestra.run.smithi032.stderr:Exporting 1.8 info 1.8( v 35'139 (0'0,35'139] local-lis/les=109/110 n=0 ec=104/15 lis/c=109/109 les/c/f=110/110/0 sis=109) 2024-09-16T05:14:01.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:01 smithi032 ceph-mon[27388]: pgmap v615: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 294 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 10/270 objects degraded (3.704%) 2024-09-16T05:14:01.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:01 smithi032 ceph-mon[27388]: osdmap e309: 12 total, 11 up, 12 in 2024-09-16T05:14:01.876 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/422018644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:01.876 INFO:teuthology.orchestra.run.smithi032.stderr:Read #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-09-16T05:14:01.876 INFO:teuthology.orchestra.run.smithi032.stderr:Export successful 2024-09-16T05:14:01.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:01.898 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:01.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:01.948 INFO:tasks.rados.rados.2.smithi204.stdout:962: finishing write tid 1 to smithi20457400-20 2024-09-16T05:14:01.948 INFO:tasks.rados.rados.2.smithi204.stdout:962: finishing write tid 2 to smithi20457400-20 2024-09-16T05:14:01.948 INFO:tasks.rados.rados.2.smithi204.stdout:962: finishing write tid 3 to smithi20457400-20 2024-09-16T05:14:01.948 INFO:tasks.rados.rados.2.smithi204.stdout:962: finishing write tid 4 to smithi20457400-20 2024-09-16T05:14:01.948 INFO:tasks.rados.rados.2.smithi204.stdout:962: finishing write tid 5 to smithi20457400-20 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 536 (ObjNum 356 snap 104 seq_num 356) dirty exists 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:962: left oid 20 (ObjNum 356 snap 104 seq_num 356) 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:963: finishing write tid 1 to smithi20457400-34 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:963: finishing write tid 2 to smithi20457400-34 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:963: finishing write tid 3 to smithi20457400-34 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:963: finishing write tid 4 to smithi20457400-34 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 449 (ObjNum 357 snap 104 seq_num 357) dirty exists 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:963: left oid 34 (ObjNum 357 snap 104 seq_num 357) 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:964: finishing write tid 1 to smithi20457400-45 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:964: finishing write tid 2 to smithi20457400-45 2024-09-16T05:14:01.949 INFO:tasks.rados.rados.2.smithi204.stdout:964: finishing write tid 3 to smithi20457400-45 2024-09-16T05:14:01.950 INFO:tasks.rados.rados.2.smithi204.stdout:964: finishing write tid 4 to smithi20457400-45 2024-09-16T05:14:01.950 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 330 (ObjNum 358 snap 104 seq_num 358) dirty exists 2024-09-16T05:14:01.950 INFO:tasks.rados.rados.2.smithi204.stdout:964: left oid 45 (ObjNum 358 snap 104 seq_num 358) 2024-09-16T05:14:01.950 INFO:tasks.rados.rados.2.smithi204.stdout:965: expect (ObjNum 302 snap 83 seq_num 302) 2024-09-16T05:14:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:01 smithi196 ceph-mon[30802]: pgmap v615: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 294 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 10/270 objects degraded (3.704%) 2024-09-16T05:14:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:01 smithi196 ceph-mon[30802]: osdmap e309: 12 total, 11 up, 12 in 2024-09-16T05:14:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/422018644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:02.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:02.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:02.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:02.121 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:14:02.121 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:14:02.122 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:14:02.122 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:14:02.122 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:14:02.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:02.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:02.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:01 smithi204 ceph-mon[30933]: pgmap v615: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 294 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 10/270 objects degraded (3.704%) 2024-09-16T05:14:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:01 smithi204 ceph-mon[30933]: osdmap e309: 12 total, 11 up, 12 in 2024-09-16T05:14:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/422018644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:02.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:02.274 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:02.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:02.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:02.400 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:02.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:02.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:02.524 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:02.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:02.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:02.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:14:02.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:02.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:02.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:02.731 INFO:tasks.rados.rados.2.smithi204.stdout:959: done (8 left) 2024-09-16T05:14:02.731 INFO:tasks.rados.rados.2.smithi204.stdout:960: done (7 left) 2024-09-16T05:14:02.731 INFO:tasks.rados.rados.2.smithi204.stdout:961: done (6 left) 2024-09-16T05:14:02.731 INFO:tasks.rados.rados.2.smithi204.stdout:962: done (5 left) 2024-09-16T05:14:02.732 INFO:tasks.rados.rados.2.smithi204.stdout:963: done (4 left) 2024-09-16T05:14:02.732 INFO:tasks.rados.rados.2.smithi204.stdout:964: done (3 left) 2024-09-16T05:14:02.732 INFO:tasks.rados.rados.2.smithi204.stdout:965: done (2 left) 2024-09-16T05:14:02.732 INFO:tasks.rados.rados.2.smithi204.stdout:966: done (1 left) 2024-09-16T05:14:02.732 INFO:tasks.rados.rados.2.smithi204.stdout:967: delete oid 47 current snap is 105 2024-09-16T05:14:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:967: done (1 left) 2024-09-16T05:14:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:968: read oid 43 snap -1 2024-09-16T05:14:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:968: expect deleted 2024-09-16T05:14:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:969: copy_from oid 21 from oid 46 current snap is 105 2024-09-16T05:14:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:970: read oid 3 snap 80 2024-09-16T05:14:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:970: expect (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:14:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:971: delete oid 12 current snap is 105 2024-09-16T05:14:02.738 INFO:tasks.rados.rados.2.smithi204.stdout:968: done (4 left) 2024-09-16T05:14:02.738 INFO:tasks.rados.rados.2.smithi204.stdout:971: done (3 left) 2024-09-16T05:14:02.738 INFO:tasks.rados.rados.2.smithi204.stdout:972: write oid 19 current snap is 105 2024-09-16T05:14:02.738 INFO:tasks.rados.rados.2.smithi204.stdout:972: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-16T05:14:02.744 INFO:tasks.rados.rados.2.smithi204.stdout:972: writing smithi20457400-19 from 467327 to 1100807 tid 1 2024-09-16T05:14:02.750 INFO:tasks.rados.rados.2.smithi204.stdout:972: writing smithi20457400-19 from 1825176 to 2067327 tid 2 2024-09-16T05:14:02.750 INFO:tasks.rados.rados.2.smithi204.stdout:973: write oid 30 current snap is 105 2024-09-16T05:14:02.750 INFO:tasks.rados.rados.2.smithi204.stdout:973: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-16T05:14:02.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:02.757 INFO:tasks.rados.rados.2.smithi204.stdout:973: writing smithi20457400-30 from 484134 to 1192863 tid 1 2024-09-16T05:14:02.763 INFO:tasks.rados.rados.2.smithi204.stdout:973: writing smithi20457400-30 from 1883658 to 2084134 tid 2 2024-09-16T05:14:02.763 INFO:tasks.rados.rados.2.smithi204.stdout:974: snap_create 2024-09-16T05:14:02.763 INFO:tasks.rados.rados.2.smithi204.stdout:970: expect (ObjNum 185 snap 46 seq_num 185) 2024-09-16T05:14:02.776 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:02.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:02.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:02.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:02.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:02.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:02 smithi196 ceph-mon[30802]: osdmap e310: 12 total, 11 up, 12 in 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:972: finishing write tid 1 to smithi20457400-19 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:972: finishing write tid 2 to smithi20457400-19 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:972: finishing write tid 3 to smithi20457400-19 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:972: finishing write tid 4 to smithi20457400-19 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 287 (ObjNum 361 snap 105 seq_num 361) dirty exists 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:972: left oid 19 (ObjNum 361 snap 105 seq_num 361) 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:969: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:969: finishing copy_from to smithi20457400-21 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 319 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:973: finishing write tid 1 to smithi20457400-30 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:973: finishing write tid 2 to smithi20457400-30 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:973: finishing write tid 3 to smithi20457400-30 2024-09-16T05:14:02.991 INFO:tasks.rados.rados.2.smithi204.stdout:973: finishing write tid 4 to smithi20457400-30 2024-09-16T05:14:02.992 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 525 (ObjNum 362 snap 105 seq_num 362) dirty exists 2024-09-16T05:14:02.992 INFO:tasks.rados.rados.2.smithi204.stdout:973: left oid 30 (ObjNum 362 snap 105 seq_num 362) 2024-09-16T05:14:03.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:03.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:03.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:03.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:03.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:03.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:02 smithi032 ceph-mon[27388]: osdmap e310: 12 total, 11 up, 12 in 2024-09-16T05:14:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:02 smithi204 ceph-mon[30933]: osdmap e310: 12 total, 11 up, 12 in 2024-09-16T05:14:03.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.2 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 1.8 2024-09-16T05:14:03.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:03.313 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:03.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:03.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:03.438 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:03.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:03.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:03.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:03.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:03.624 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.2/config 2024-09-16T05:14:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:14:03.634 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:14:03.639 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:03.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:03.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:03.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:03.736 INFO:tasks.rados.rados.2.smithi204.stdout:969: done (5 left) 2024-09-16T05:14:03.737 INFO:tasks.rados.rados.2.smithi204.stdout:970: done (4 left) 2024-09-16T05:14:03.737 INFO:tasks.rados.rados.2.smithi204.stdout:972: done (3 left) 2024-09-16T05:14:03.737 INFO:tasks.rados.rados.2.smithi204.stdout:973: done (2 left) 2024-09-16T05:14:03.737 INFO:tasks.rados.rados.2.smithi204.stdout:974: done (1 left) 2024-09-16T05:14:03.737 INFO:tasks.rados.rados.2.smithi204.stdout:975: read oid 35 snap -1 2024-09-16T05:14:03.737 INFO:tasks.rados.rados.2.smithi204.stdout:975: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-16T05:14:03.737 INFO:tasks.rados.rados.2.smithi204.stdout:976: delete oid 18 current snap is 106 2024-09-16T05:14:03.740 INFO:tasks.rados.rados.2.smithi204.stdout:975: expect (ObjNum 317 snap 86 seq_num 317) 2024-09-16T05:14:03.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:03.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:03.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:03.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:03.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:03 smithi032 ceph-mon[27388]: pgmap v618: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 293 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.2 MiB/s wr, 9 op/s; 10/272 objects degraded (3.676%); 1/272 objects misplaced (0.368%) 2024-09-16T05:14:03.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:03 smithi032 ceph-mon[27388]: osdmap e311: 12 total, 11 up, 12 in 2024-09-16T05:14:03.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:03.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:03 smithi196 ceph-mon[30802]: pgmap v618: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 293 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.2 MiB/s wr, 9 op/s; 10/272 objects degraded (3.676%); 1/272 objects misplaced (0.368%) 2024-09-16T05:14:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:03 smithi196 ceph-mon[30802]: osdmap e311: 12 total, 11 up, 12 in 2024-09-16T05:14:04.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:04.072 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:04.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:04.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:03 smithi204 ceph-mon[30933]: pgmap v618: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 293 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.2 MiB/s wr, 9 op/s; 10/272 objects degraded (3.676%); 1/272 objects misplaced (0.368%) 2024-09-16T05:14:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:03 smithi204 ceph-mon[30933]: osdmap e311: 12 total, 11 up, 12 in 2024-09-16T05:14:04.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:04.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:04.198 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:04.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:04.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:04.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:04.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:04.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:04.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:04.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:04.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:04.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:04.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:04.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:04.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:04.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:04.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:04.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:04.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:04.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:04 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:14:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:04 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:14:04.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:04 smithi032 ceph-mon[27388]: osdmap e312: 12 total, 11 up, 12 in 2024-09-16T05:14:04.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:04.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:04.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:05.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:05.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:04 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:14:05.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:04 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:14:05.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:04 smithi204 ceph-mon[30933]: osdmap e312: 12 total, 11 up, 12 in 2024-09-16T05:14:05.075 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:14:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:05.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:05.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:05.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:05.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:05.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:04 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:14:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:04 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:14:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:04 smithi196 ceph-mon[30802]: osdmap e312: 12 total, 11 up, 12 in 2024-09-16T05:14:05.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:05.342 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:05.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:05.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:05.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:05.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:05.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:05.517 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:05.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:05.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:05.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:05.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:14:05.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:05.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:05.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:05.744 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:14:05.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:05.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:05.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:05 smithi032 ceph-mon[27388]: pgmap v621: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 309 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 10/282 objects degraded (3.546%); 1/282 objects misplaced (0.355%) 2024-09-16T05:14:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:05 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 10/282 objects degraded (3.546%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1578399976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:05.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:05.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:06.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:06.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:06.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:05 smithi196 ceph-mon[30802]: pgmap v621: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 309 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 10/282 objects degraded (3.546%); 1/282 objects misplaced (0.355%) 2024-09-16T05:14:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:05 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 10/282 objects degraded (3.546%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:06.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1578399976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:06.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:05 smithi204 ceph-mon[30933]: pgmap v621: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 309 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 10/282 objects degraded (3.546%); 1/282 objects misplaced (0.355%) 2024-09-16T05:14:06.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:05 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 10/282 objects degraded (3.546%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:06.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1578399976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:06.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:06.217 INFO:teuthology.orchestra.run.smithi032.stdout: marking collection for removal 2024-09-16T05:14:06.217 INFO:teuthology.orchestra.run.smithi032.stdout:setting '_remove' omap key 2024-09-16T05:14:06.217 INFO:teuthology.orchestra.run.smithi032.stdout:finish_remove_pgs 1.8_head removing 1.8 2024-09-16T05:14:06.217 INFO:teuthology.orchestra.run.smithi032.stdout:Remove successful 2024-09-16T05:14:06.219 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:06.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:06.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:06.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:06.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:06.468 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:06.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:06.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:06.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:06.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:06.695 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:06.716 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:06.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:06.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:06.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:06.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:06.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:06.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:06.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:07.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:06 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:14:07.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:06 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:14:07.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:06 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:07.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1578399976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:07.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:06 smithi032 ceph-mon[27388]: osdmap e313: 12 total, 11 up, 12 in 2024-09-16T05:14:07.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:07.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:07.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:07.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:06 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:14:07.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:06 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:14:07.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:06 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:07.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1578399976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:07.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:06 smithi204 ceph-mon[30933]: osdmap e313: 12 total, 11 up, 12 in 2024-09-16T05:14:07.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:07.211 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:07.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:06 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:14:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:06 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:14:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:06 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1578399976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:07.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:06 smithi196 ceph-mon[30802]: osdmap e313: 12 total, 11 up, 12 in 2024-09-16T05:14:07.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:07.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:07.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.2 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.8.2 2024-09-16T05:14:07.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:07.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:07.459 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:07.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:07.481 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.14155.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:17.800064+0000", 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 529.67934986900002, 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26367868, 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.482 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.14155", 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.32:0/1326813644", 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:17.800064+0000", 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.483 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:17.800064+0000", 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:17.800065+0000", 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1969999999999999e-06 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:17.800072+0000", 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8839999999999997e-06 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:17.800082+0000", 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6199999999999994e-06 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:17.800089+0000", 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7929999999999998e-06 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:17.800146+0000", 2024-09-16T05:14:07.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6416000000000003e-05 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:17.800157+0000", 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0849000000000001e-05 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.062752+0000", 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26259586899999998 2024-09-16T05:14:07.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.062879+0000", 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000126437 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.063012+0000", 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132707 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.063427+0000", 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00041554100000000001 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.063684+0000", 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025669000000000001 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.063718+0000", 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4450999999999999e-05 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.063743+0000", 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4226000000000001e-05 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.14155.0:16 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.allocated (16) in=17b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:05:18.661503+0000", 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 528.81791121699996, 2024-09-16T05:14:07.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.29930417300000001, 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.14155", 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.32:0/1326813644", 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 16 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661503+0000", 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661503+0000", 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661503+0000", 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0500000000000003e-07 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661507+0000", 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9759999999999997e-06 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661516+0000", 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7439999999999996e-06 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661522+0000", 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4289999999999998e-06 2024-09-16T05:14:07.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661554+0000", 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2035000000000001e-05 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661595+0000", 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1142e-05 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:14:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661661+0000", 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5137999999999999e-05 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.661875+0000", 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021466599999999999 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960726+0000", 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.298850538 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960760+0000", 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4610999999999998e-05 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:05:18.960807+0000", 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6288999999999997e-05 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.495 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:163 2.9 2:9423b7dc:::smithi20457400-41:head [setxattr _header (56) in=7b,truncate 2672280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:14:07.496 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 203.98566933999999, 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.176412596, 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 163 2024-09-16T05:14:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493745+0000", 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1899999999999998e-07 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493747+0000", 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1600000000000001e-06 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493751+0000", 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8260000000000003e-06 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493754+0000", 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8779999999999998e-06 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494256+0000", 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050281900000000005 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.500 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494314+0000", 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7592e-05 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494393+0000", 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8799000000000002e-05 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494718+0000", 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00032548400000000002 2024-09-16T05:14:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670063+0000", 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17534508600000001 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670109+0000", 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5427999999999999e-05 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670157+0000", 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8105000000000001e-05 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:164 2.9 2:9423b7dc:::smithi20457400-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 203.98565410500001, 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17655259500000001, 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.503 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 164 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.504 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7000000000000001e-07 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493762+0000", 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0660000000000002e-06 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493766+0000", 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.732e-06 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493769+0000", 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6799999999999999e-06 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494530+0000", 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076053399999999999 2024-09-16T05:14:07.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494606+0000", 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6538999999999993e-05 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670197+0000", 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17559084699999999 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.507 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670253+0000", 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5893999999999997e-05 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670312+0000", 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9033000000000002e-05 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:163 2.9 e112/107 ondisk, result = 0)", 2024-09-16T05:14:07.508 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 203.98349711200001, 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17425236199999999, 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.14e-07 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495918+0000", 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5519999999999999e-06 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.510 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.669955+0000", 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17403649900000001 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.669969+0000", 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4064e-05 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670051+0000", 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2313999999999998e-05 2024-09-16T05:14:07.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670059+0000", 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1240000000000005e-06 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670169+0000", 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000109695 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:207 2.6 2:6be67188:::smithi20457400-46:head [write 1768111~402677] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 202.478349081, 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092996950000000002, 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.513 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 207 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.514 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001066+0000", 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4900000000000004e-07 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001419+0000", 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035348299999999998 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001430+0000", 2024-09-16T05:14:07.515 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0499000000000001e-05 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001435+0000", 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.22e-06 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001488+0000", 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3588999999999998e-05 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001556+0000", 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7358000000000001e-05 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001638+0000", 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2319999999999998e-05 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.002547+0000", 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090899100000000005 2024-09-16T05:14:07.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010283+0000", 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077356680000000002 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010323+0000", 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0389000000000002e-05 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.518 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010364+0000", 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1229000000000001e-05 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:209 2.6 2:6be67188:::smithi20457400-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:45.001480+0000", 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 202.477933648, 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0091486060000000001, 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 209 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001480+0000", 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001480+0000", 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001481+0000", 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7899999999999999e-07 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001484+0000", 2024-09-16T05:14:07.521 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9310000000000001e-06 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001492+0000", 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4159999999999998e-06 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001497+0000", 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4049999999999999e-06 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.522 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.002447+0000", 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00094984099999999999 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.002532+0000", 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4809999999999996e-05 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010516+0000", 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079837069999999996 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010569+0000", 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3335999999999999e-05 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010629+0000", 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9681e-05 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.524 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:407 2.9 2:96dc17ef:::smithi20457400-10:head [write 1515641~485017] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 189.807934918, 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095319989999999993, 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 407 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671480+0000", 2024-09-16T05:14:07.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6199999999999996e-07 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671908+0000", 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042804200000000001 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671917+0000", 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6870000000000006e-06 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671923+0000", 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0499999999999997e-06 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671969+0000", 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5461999999999999e-05 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672023+0000", 2024-09-16T05:14:07.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4215000000000002e-05 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672096+0000", 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2574e-05 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672934+0000", 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083847200000000005 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680951+0000", 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080164480000000007 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680984+0000", 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3399000000000003e-05 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.681011+0000", 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6787999999999998e-05 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.530 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:409 2.9 2:96dc17ef:::smithi20457400-10:head [setxattr _header (56) in=7b,truncate 3378174] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 189.800459969, 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059654601000000002, 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.531 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 409 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7899999999999999e-07 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678957+0000", 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1109999999999999e-06 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678961+0000", 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9639999999999997e-06 2024-09-16T05:14:07.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678969+0000", 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8839999999999994e-06 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.679871+0000", 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090153499999999997 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.679945+0000", 2024-09-16T05:14:07.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4518999999999996e-05 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680024+0000", 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9295999999999994e-05 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680263+0000", 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000238994 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738545+0000", 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058281865000000002 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738576+0000", 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0349999999999999e-05 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738608+0000", 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2703999999999997e-05 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:410 2.9 2:96dc17ef:::smithi20457400-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 189.8004402, 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059752424999999998, 2024-09-16T05:14:07.537 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 410 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9400000000000001e-07 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678975+0000", 2024-09-16T05:14:07.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1960000000000001e-06 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678979+0000", 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7479999999999999e-06 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678982+0000", 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8820000000000001e-06 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680163+0000", 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011814880000000001 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680245+0000", 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1411999999999999e-05 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738635+0000", 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058390240000000003 2024-09-16T05:14:07.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738674+0000", 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9119999999999998e-05 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738726+0000", 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2045000000000001e-05 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:409 2.9 e126/107 ondisk, result = 0)", 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 189.79852902299999, 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057729559, 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3799999999999999e-07 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680887+0000", 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7689999999999999e-06 2024-09-16T05:14:07.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736934+0000", 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.056047685999999999 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736942+0000", 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4479999999999997e-06 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738537+0000", 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015953720000000001 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738542+0000", 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7190000000000001e-06 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738614+0000", 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2427000000000003e-05 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:597 2.9 2:96dc17ef:::smithi20457400-10:head [copy-from ver 0] snapc 13=[13,12,11,f,e,b] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 173.67645744800001, 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010909965000000001, 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.547 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 597 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802959+0000", 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4540000000000001e-06 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802992+0000", 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3025000000000003e-05 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803026+0000", 2024-09-16T05:14:07.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3903999999999999e-05 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803041+0000", 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5299e-05 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803197+0000", 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000156389 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803202+0000", 2024-09-16T05:14:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.758e-06 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805000+0000", 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017979859999999999 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805054+0000", 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4249999999999997e-05 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813866+0000", 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088118999999999992 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:598 2.9 2:96dc17ef:::smithi20457400-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 173.676340283, 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010979131, 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.552 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 598 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803074+0000", 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.47e-07 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803078+0000", 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1030000000000004e-06 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803090+0000", 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1875999999999999e-05 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803099+0000", 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2199999999999992e-06 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803245+0000", 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014687899999999999 2024-09-16T05:14:07.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803249+0000", 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0520000000000002e-06 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805228+0000", 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019794729999999998 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805254+0000", 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6016e-05 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813951+0000", 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086966319999999993 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813999+0000", 2024-09-16T05:14:07.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8263999999999997e-05 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.814053+0000", 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3668999999999997e-05 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:772 2.9 2:94d33c56:::smithi20457400-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e159)", 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:14:07.559 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 156.832451166, 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092047459999999998, 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 772 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:14:07.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7100000000000002e-07 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646975+0000", 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1861999999999999e-05 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646984+0000", 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2550000000000003e-06 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646991+0000", 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6340000000000003e-06 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.647057+0000", 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6353000000000007e-05 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.647061+0000", 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2720000000000002e-06 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.648424+0000", 2024-09-16T05:14:07.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013634470000000001 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.648472+0000", 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8093999999999997e-05 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.656107+0000", 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076354400000000003 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.656167+0000", 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9818000000000002e-05 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:979 2.4 2:2ace617f:::smithi20457400-21:head [copy-from ver 40] snapc 28=[28,27,25,24,23,22,21,20,1f,15,f] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.481549613, 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016837338, 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.565 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 979 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997865+0000", 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8899999999999999e-07 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997872+0000", 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0639999999999996e-06 2024-09-16T05:14:07.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997880+0000", 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4039999999999995e-06 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997886+0000", 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4600000000000002e-06 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998030+0000", 2024-09-16T05:14:07.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014470800000000001 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998036+0000", 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6620000000000002e-06 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000325+0000", 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022892450000000001 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000434+0000", 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010908699999999999 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006744+0000", 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063091240000000002 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014271+0000", 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0075269869999999997 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014635+0000", 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036412499999999997 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014670+0000", 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5799000000000001e-05 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014701+0000", 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0984e-05 2024-09-16T05:14:07.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:980 2.4 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.48151907799999, 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013613798, 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.572 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 980 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997899+0000", 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7319999999999998e-06 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997902+0000", 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.819e-06 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997913+0000", 2024-09-16T05:14:07.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.083e-05 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997919+0000", 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5570000000000003e-06 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998065+0000", 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000146012 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998069+0000", 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8550000000000001e-06 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000716+0000", 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026463070000000001 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000762+0000", 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6443999999999998e-05 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011363+0000", 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010600559000000001 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011432+0000", 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8955000000000006e-05 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011508+0000", 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6728000000000001e-05 2024-09-16T05:14:07.577 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.578 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.578 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.578 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1004 2.4 2:2ace617f:::smithi20457400-21:head [copy-from ver 113] snapc 29=[29,28,27,25,24,23,22,21,20,1f,15] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-16T05:14:07.578 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:14:07.578 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 134.72729530999999, 2024-09-16T05:14:07.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012670931, 2024-09-16T05:14:07.580 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.580 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.580 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.580 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1004 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752120+0000", 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.996e-06 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752132+0000", 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1389000000000001e-05 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752150+0000", 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8533999999999999e-05 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752159+0000", 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9959999999999999e-06 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752236+0000", 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6211000000000006e-05 2024-09-16T05:14:07.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752247+0000", 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1127e-05 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753383+0000", 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011362830000000001 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753447+0000", 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4009000000000003e-05 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.761264+0000", 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078170510000000002 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764658+0000", 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033938449999999999 2024-09-16T05:14:07.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764722+0000", 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4177999999999994e-05 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764759+0000", 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7363000000000001e-05 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764789+0000", 2024-09-16T05:14:07.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9949000000000001e-05 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1005 2.4 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e181)", 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 134.72723090400001, 2024-09-16T05:14:07.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011596623, 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1005 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0600000000000001e-07 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752185+0000", 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2419999999999999e-06 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752190+0000", 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7049999999999998e-06 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752193+0000", 2024-09-16T05:14:07.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2739999999999999e-06 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752304+0000", 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110301 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752307+0000", 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6559999999999998e-06 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753639+0000", 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001331201 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753690+0000", 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1826999999999997e-05 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763643+0000", 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0099530030000000002 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763709+0000", 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5469999999999995e-05 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763779+0000", 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0537999999999994e-05 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1499 2.9 2:94d33c56:::smithi20457400-8:head [copy-from ver 285] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 97.789187100999996, 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016013924999999998, 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1499 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690228+0000", 2024-09-16T05:14:07.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6419999999999999e-06 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690237+0000", 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8929999999999994e-06 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690270+0000", 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2679999999999999e-05 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690281+0000", 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1446e-05 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690342+0000", 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0681999999999999e-05 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690408+0000", 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6173000000000005e-05 2024-09-16T05:14:07.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.698590+0000", 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081814639999999994 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.704358+0000", 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057679389999999997 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706113+0000", 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001755097 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706183+0000", 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9991000000000001e-05 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706241+0000", 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7917999999999997e-05 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1500 2.9 2:94d33c56:::smithi20457400-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 97.789107313000002, 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012373189, 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:07.600 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1500 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690307+0000", 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7599999999999997e-07 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:07.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690310+0000", 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2610000000000002e-06 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690337+0000", 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6733999999999998e-05 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690344+0000", 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2779999999999996e-06 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690524+0000", 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017976900000000001 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690560+0000", 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5762000000000001e-05 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702528+0000", 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011967976 2024-09-16T05:14:07.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702599+0000", 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1429999999999999e-05 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702680+0000", 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0402999999999996e-05 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:07.605 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:07.606 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:14:07.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:07.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:07.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:07.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:07.727 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.2/config 2024-09-16T05:14:07.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:07.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:07.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:07.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:07.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:07.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:07.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:07.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:08.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:08.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:08.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:07 smithi032 ceph-mon[27388]: pgmap v623: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 318 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 10/294 objects degraded (3.401%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:07 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:14:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:07 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:14:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:07 smithi204 ceph-mon[30933]: pgmap v623: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 318 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 10/294 objects degraded (3.401%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:07 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:14:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:07 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:14:08.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:08.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:07 smithi196 ceph-mon[30802]: pgmap v623: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 318 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 10/294 objects degraded (3.401%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:07 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:14:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:07 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:14:08.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:08.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:08.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:08.268 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:08.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:14:08.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:08.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:08.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:08.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:14:08.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:08.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:08.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:08.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:08.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:08.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:08.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:08.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:08.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:08.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:08.885 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:08.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:08.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:08 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:14:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:08 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:14:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:08 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:14:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:08 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:14:08.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:09.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:09.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:09.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:09.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:09.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:08 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:14:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:08 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:14:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:08 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:14:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:08 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:14:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:08 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:14:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:08 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:14:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:08 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:14:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:08 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:14:09.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:09.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:09.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:09.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:09.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:09.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:09.497 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:09.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:09.518 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:09.519 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:14:09.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:09.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:09.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:09.539 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:09.540 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:09.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:09.583 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:09.583 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:09.604 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:09.605 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:09.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:09.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:09.627 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:09.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:09.648 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:09.649 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:09.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:09.669 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:09.669 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:09.691 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:09.691 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:09.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:09.713 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:09.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:09.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:09.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:09.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:09.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:09.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:09.982 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:14:10.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:10.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:10.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:10.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:10.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:09 smithi032 ceph-mon[27388]: pgmap v624: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 17 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:10.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:09 smithi032 ceph-mon[27388]: osdmap e314: 12 total, 11 up, 12 in 2024-09-16T05:14:10.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1988915869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:10.162 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:10.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:09 smithi204 ceph-mon[30933]: pgmap v624: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 17 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:09 smithi204 ceph-mon[30933]: osdmap e314: 12 total, 11 up, 12 in 2024-09-16T05:14:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1988915869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:10.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:09 smithi196 ceph-mon[30802]: pgmap v624: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 17 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:09 smithi196 ceph-mon[30802]: osdmap e314: 12 total, 11 up, 12 in 2024-09-16T05:14:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1988915869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:10.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:10.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:10.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:10.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:10.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:10.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:10.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:10.542 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:10.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:10.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:10.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:10.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:10.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:14:10.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:14:10.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:10.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:10.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:10.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:11.012 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:11.012 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:11.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:11.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:11.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:11.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:10 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:14:11.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:10 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:14:11.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:10 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 10/298 objects degraded (3.356%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:11.154 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:10 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:14:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:10 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:14:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:10 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 10/298 objects degraded (3.356%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:11.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:10 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:14:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:10 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:14:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:10 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 10/298 objects degraded (3.356%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:11.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:11.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:11.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:11.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:11.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:11.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:11.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:11.534 INFO:teuthology.orchestra.run.smithi032.stdout:get_pg_num_history pg_num_history pg_num_history(e295 pg_nums {1={15=1,104=11},2={107=16,183=26,209=36,247=46,273=56,291=66}} deleted_pools ) 2024-09-16T05:14:11.534 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pgid 1.8 2024-09-16T05:14:11.534 INFO:teuthology.orchestra.run.smithi032.stdout:Write #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-09-16T05:14:11.534 INFO:teuthology.orchestra.run.smithi032.stdout:snapset 0=[]:{} 2024-09-16T05:14:11.534 INFO:teuthology.orchestra.run.smithi032.stdout:write_pg epoch 293 info 1.8( v 35'139 (0'0,35'139] local-lis/les=109/110 n=0 ec=104/15 lis/c=109/109 les/c/f=110/110/0 sis=109) 2024-09-16T05:14:11.535 INFO:teuthology.orchestra.run.smithi032.stdout:Import successful 2024-09-16T05:14:11.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:11.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:11.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:11.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:11.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:11.762 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:14:11.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:11.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:11.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:11.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:11.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:11.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:12.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:12.050 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:11 smithi032 ceph-mon[27388]: pgmap v626: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 13 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:12.050 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:11 smithi032 ceph-mon[27388]: osdmap e315: 12 total, 11 up, 12 in 2024-09-16T05:14:12.050 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/387341610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:12.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:12.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:11 smithi204 ceph-mon[30933]: pgmap v626: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 13 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:11 smithi204 ceph-mon[30933]: osdmap e315: 12 total, 11 up, 12 in 2024-09-16T05:14:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/387341610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:12.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:12.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:12.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:11 smithi196 ceph-mon[30802]: pgmap v626: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 13 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:11 smithi196 ceph-mon[30802]: osdmap e315: 12 total, 11 up, 12 in 2024-09-16T05:14:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/387341610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:12.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:12.325 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:12.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:12.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:12.455 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:12.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:12.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:12.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:12.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:12.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:12.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:12.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:12.785 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:12.785 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:12.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:14:12.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:12.858 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:12.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:12.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:12.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:12.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:13.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:13.112 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:13.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:13.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:12 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:14:13.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:12 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:14:13.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:12 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:13.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/387341610' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:13.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:12 smithi032 ceph-mon[27388]: osdmap e316: 12 total, 11 up, 12 in 2024-09-16T05:14:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:12 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:14:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:12 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:14:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:12 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/387341610' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:12 smithi204 ceph-mon[30933]: osdmap e316: 12 total, 11 up, 12 in 2024-09-16T05:14:13.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:13.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:12 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:14:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:12 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:14:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:12 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/387341610' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:12 smithi196 ceph-mon[30802]: osdmap e316: 12 total, 11 up, 12 in 2024-09-16T05:14:13.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:13.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:13.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:13.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:13.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:13.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/exp.1.8.2 2024-09-16T05:14:13.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:13.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:13.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:13.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:13.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:13.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:13.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:13.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:13.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:13.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:13.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:14:13.914 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:13.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:13.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:14.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:14.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:13 smithi032 ceph-mon[27388]: pgmap v629: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 292 KiB/s wr, 4 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:14.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:14.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:14.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:14.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:13 smithi204 ceph-mon[30933]: pgmap v629: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 292 KiB/s wr, 4 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:14.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:13 smithi196 ceph-mon[30802]: pgmap v629: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 292 KiB/s wr, 4 op/s; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:14.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:14.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:14.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:14.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:14:14.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:14.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:14.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:14.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:14.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:14.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:14.580 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:14.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:14.696 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:14.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:14.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:14.823 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:14.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:14.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:14.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:14.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:14.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:15.033 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1430857884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:15.035 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:15.035 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:15.037 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:15.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:15.078 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:15.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1430857884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:15.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:15.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:15.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1430857884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:15.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:15.334 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:15.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:15.345 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:15.345 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:15.367 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:15.367 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:14:15.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:15.388 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:15.409 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:15.410 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:15.431 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:15.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:15.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:15.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:15.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:15.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:15.505 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:15.526 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:15.548 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:15.548 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:15.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:15.572 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:15.573 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:15.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:15.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:15.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:15.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:15.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:15.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:15.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:15.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:15.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:15 smithi032 ceph-mon[27388]: pgmap v630: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:15.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:15.965 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:15.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:16.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:16.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:16.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:16.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:14:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:15 smithi204 ceph-mon[30933]: pgmap v630: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:16.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:16.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:16.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:16.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:15 smithi196 ceph-mon[30802]: pgmap v630: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:16.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:16.339 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:16.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:16.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:16.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:16.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:16.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:16.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:16.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:16.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:14:16.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:14:16.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:16.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:14:16.791 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:16.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:16.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:16.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:16.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:16.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:16.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:17.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:17.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:17.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:17.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:14:17.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:14:17.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/72581178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:17.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=2579043934,config=1549860826,health=3440148872,kv=1535347985}) 2024-09-16T05:14:17.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=829866206}) 2024-09-16T05:14:17.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3976371192}) 2024-09-16T05:14:17.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=2389958071,logm=366249512}) 2024-09-16T05:14:17.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977897098}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1041442183}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949157019}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3808629252}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733188402}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108493258}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972000966}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562769255}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33271303}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2068274323}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780846997}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960505912}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=860401591}) 2024-09-16T05:14:17.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077182821}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=924277941}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999381927}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163811123}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619214425}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2889562293}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777597581}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=307383377}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397247094}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=442997755}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3075582746}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445821191}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4056792928}) 2024-09-16T05:14:17.095 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413557732}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850632197}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894412793}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473126405}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=44,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=32} crc {logm=3283892079,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=3758174617}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=31,mon_config_key=69} crc {mgrstat=2930229986,mon_config_key=3415974734}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2294907518}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=877569741}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=4,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=39,osdmap=39} crc {mon_config_key=3752687995,monmap=2256977890,osd_metadata=2660146127,osd_pg_creating=3578286905,osd_snap=1726393976,osdmap=3813468099}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4137235789}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3507047979}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3490204922}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1809918674}) 2024-09-16T05:14:17.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=141203263}) 2024-09-16T05:14:17.097 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=4255677628}) 2024-09-16T05:14:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:14:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:14:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/72581178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=2579043934,config=1549860826,health=3440148872,kv=1535347985}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=829866206}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3976371192}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=2389958071,logm=366249512}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977897098}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1041442183}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949157019}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3808629252}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733188402}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108493258}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972000966}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562769255}) 2024-09-16T05:14:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33271303}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2068274323}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780846997}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960505912}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=860401591}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077182821}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=924277941}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999381927}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163811123}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619214425}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2889562293}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777597581}) 2024-09-16T05:14:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=307383377}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397247094}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=442997755}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3075582746}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445821191}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4056792928}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413557732}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850632197}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894412793}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473126405}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=44,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=32} crc {logm=3283892079,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=3758174617}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=31,mon_config_key=69} crc {mgrstat=2930229986,mon_config_key=3415974734}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2294907518}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=877569741}) 2024-09-16T05:14:17.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=4,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=39,osdmap=39} crc {mon_config_key=3752687995,monmap=2256977890,osd_metadata=2660146127,osd_pg_creating=3578286905,osd_snap=1726393976,osdmap=3813468099}) 2024-09-16T05:14:17.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4137235789}) 2024-09-16T05:14:17.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3507047979}) 2024-09-16T05:14:17.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3490204922}) 2024-09-16T05:14:17.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1809918674}) 2024-09-16T05:14:17.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=141203263}) 2024-09-16T05:14:17.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=4255677628}) 2024-09-16T05:14:17.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:17.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:17.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:14:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:14:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/72581178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=2579043934,config=1549860826,health=3440148872,kv=1535347985}) 2024-09-16T05:14:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=829866206}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3976371192}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=2389958071,logm=366249512}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977897098}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1041442183}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949157019}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3808629252}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733188402}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108493258}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972000966}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562769255}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33271303}) 2024-09-16T05:14:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2068274323}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2780846997}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960505912}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=860401591}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077182821}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=924277941}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999381927}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163811123}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619214425}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2889562293}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777597581}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=307383377}) 2024-09-16T05:14:17.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397247094}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=442997755}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3075582746}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445821191}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4056792928}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413557732}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850632197}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894412793}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473126405}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=44,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=32} crc {logm=3283892079,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=3758174617}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=31,mon_config_key=69} crc {mgrstat=2930229986,mon_config_key=3415974734}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2294907518}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=877569741}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=4,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=39,osdmap=39} crc {mon_config_key=3752687995,monmap=2256977890,osd_metadata=2660146127,osd_pg_creating=3578286905,osd_snap=1726393976,osdmap=3813468099}) 2024-09-16T05:14:17.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4137235789}) 2024-09-16T05:14:17.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3507047979}) 2024-09-16T05:14:17.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3490204922}) 2024-09-16T05:14:17.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1809918674}) 2024-09-16T05:14:17.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=141203263}) 2024-09-16T05:14:17.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=4255677628}) 2024-09-16T05:14:17.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:17.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:17.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:17.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:17.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:17.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:17.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:17.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:17.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:17.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:17.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:17.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:17.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:17.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:17.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:17.980 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:14:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:14:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:14:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:14:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:115 2.a e112/107 2:580e4d0e:::smithi20457400-29:head v 112'6, mlcod=112'3)", 2024-09-16T05:14:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:14:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.64427630200001, 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0311933, 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:14:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334345+0000", 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334350+0000", 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.887e-06 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365213+0000", 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030863313999999999 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365220+0000", 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7870000000000004e-06 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365291+0000", 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1397999999999996e-05 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365307+0000", 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5990000000000001e-05 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365507+0000", 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000199221 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365538+0000", 2024-09-16T05:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1167000000000002e-05 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:125 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'7, mlcod=112'4)", 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.61410617300001, 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006511045, 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365198+0000", 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068299699999999997 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369743+0000", 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045446979999999998 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369750+0000", 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1940000000000001e-06 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369826+0000", 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6160000000000003e-05 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369841+0000", 2024-09-16T05:14:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4895000000000001e-05 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.370997+0000", 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001155866 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.371026+0000", 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8564999999999998e-05 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:127 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'9, mlcod=112'5)", 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.60889526400001, 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020225074999999999, 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369726+0000", 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369728+0000", 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9410000000000001e-06 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389627+0000", 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019899751 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389632+0000", 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389704+0000", 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1418e-05 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389718+0000", 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.433e-05 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389922+0000", 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020421600000000001 2024-09-16T05:14:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389951+0000", 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8140000000000002e-05 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:133 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'10, mlcod=112'7)", 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.589361336, 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074633729999999997, 2024-09-16T05:14:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389261+0000", 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.973e-06 2024-09-16T05:14:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389613+0000", 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035121600000000001 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395726+0000", 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006113851 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395731+0000", 2024-09-16T05:14:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0359999999999997e-06 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395819+0000", 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7128999999999999e-05 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395836+0000", 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7768999999999999e-05 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396693+0000", 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085685999999999998 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396723+0000", 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9539e-05 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:135 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'12, mlcod=112'8)", 2024-09-16T05:14:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.582850361, 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069274319999999999, 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395771+0000", 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4499999999999997e-07 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395775+0000", 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6509999999999999e-06 2024-09-16T05:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402319+0000", 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065439469999999996 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402326+0000", 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5769999999999996e-06 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402424+0000", 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8171000000000006e-05 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402439+0000", 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4844999999999999e-05 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402660+0000", 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022083700000000001 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402698+0000", 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7858999999999998e-05 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:773 2.1 e159/107 2:8c1ba486:::smithi20457400-2:head v 159'34, mlcod=155'30)", 2024-09-16T05:14:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 167.18544629799999, 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060955719999999996, 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793175+0000", 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795522+0000", 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023476920000000002 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795574+0000", 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1459999999999999e-05 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795584+0000", 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0274e-05 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795649+0000", 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4901999999999996e-05 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795679+0000", 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9887e-05 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.799228+0000", 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035493679999999998 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.799270+0000", 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1813000000000001e-05 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:891 2.a e171/107 2:5089ebe1:::smithi20457400-36:head v 171'128, mlcod=171'124)", 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 155.31886800699999, 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003806771, 2024-09-16T05:14:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.659753+0000", 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1699999999999998e-07 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662145+0000", 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023911589999999999 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662170+0000", 2024-09-16T05:14:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100999999999999e-05 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662177+0000", 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5120000000000002e-06 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662246+0000", 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9264000000000006e-05 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.662262+0000", 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5128e-05 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.663526+0000", 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001264311 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:42.663559+0000", 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3679000000000001e-05 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:974 2.a e179/107 2:5827f14d:::smithi20457400-12:head v 179'156, mlcod=178'152)", 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 146.971188676, 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068838909999999996, 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007433+0000", 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.421e-06 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009914+0000", 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024808030000000002 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009966+0000", 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1456999999999999e-05 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009975+0000", 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4010000000000003e-06 2024-09-16T05:14:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010042+0000", 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010064+0000", 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2282000000000001e-05 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014259+0000", 2024-09-16T05:14:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041942020000000002 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014316+0000", 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7377999999999999e-05 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1133 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'206, mlcod=189'203)", 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:14:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 135.244394067, 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021225035, 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.85e-07 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734829+0000", 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060196800000000003 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734866+0000", 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7469000000000003e-05 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734881+0000", 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5058999999999999e-05 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734970+0000", 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9004000000000004e-05 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734998+0000", 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7606e-05 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755384+0000", 2024-09-16T05:14:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02038587 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755452+0000", 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7873999999999994e-05 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1134 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'207, mlcod=189'203)", 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 135.23960894499999, 2024-09-16T05:14:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016477325000000001, 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739014+0000", 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2819999999999999e-06 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739797+0000", 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078265599999999996 2024-09-16T05:14:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.026 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:17 smithi032 ceph-mon[27388]: pgmap v631: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:18.026 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:17 smithi032 ceph-mon[27388]: 2.4 deep-scrub starts 2024-09-16T05:14:18.026 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:17 smithi032 ceph-mon[27388]: 2.4 deep-scrub ok 2024-09-16T05:14:18.026 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:17 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:18.026 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/72581178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:18.026 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:17 smithi032 ceph-mon[27388]: osdmap e317: 12 total, 11 up, 12 in 2024-09-16T05:14:18.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739844+0000", 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7608000000000002e-05 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739854+0000", 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4239999999999999e-06 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739935+0000", 2024-09-16T05:14:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1593000000000003e-05 2024-09-16T05:14:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739963+0000", 2024-09-16T05:14:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7492e-05 2024-09-16T05:14:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755459+0000", 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015496565 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755489+0000", 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9705000000000001e-05 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1135 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'208, mlcod=189'203)", 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 135.23518610799999, 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012094011, 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:14:18.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743436+0000", 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.584e-06 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743535+0000", 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9110000000000005e-05 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743568+0000", 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2335000000000001e-05 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743576+0000", 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.191e-06 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743646+0000", 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0481000000000005e-05 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743676+0000", 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9620000000000001e-05 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755492+0000", 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011815888 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755529+0000", 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6801999999999998e-05 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1136 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'209, mlcod=189'203)", 2024-09-16T05:14:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 135.23457855000001, 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011564019, 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744043+0000", 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3600000000000003e-07 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744049+0000", 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8250000000000001e-06 2024-09-16T05:14:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744069+0000", 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0460000000000001e-05 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744074+0000", 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4730000000000002e-06 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744119+0000", 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744143+0000", 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4085999999999999e-05 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755570+0000", 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011427299 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755606+0000", 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146000000000003e-05 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1478 2.1e 2:7b642727:::smithi20457400-40:head [write 581575~705262] snapc 3d=[3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:28.693708+0000", 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 109.284912584, 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059374299999999996, 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1478 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.693708+0000", 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.693708+0000", 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.693711+0000", 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9699999999999999e-06 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695661+0000", 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019495020000000001 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695679+0000", 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8553999999999999e-05 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695688+0000", 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0070000000000003e-06 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695790+0000", 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000101944 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.695799+0000", 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4749999999999993e-06 2024-09-16T05:14:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.697065+0000", 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012665860000000001 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.697162+0000", 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7158999999999996e-05 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.697311+0000", 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000148639 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699078+0000", 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001766847 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699537+0000", 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000459328 2024-09-16T05:14:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699602+0000", 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5288000000000002e-05 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:28.699646+0000", 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3130999999999998e-05 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1589 2.e e227/209 2:75dc7feb:::smithi20457400-28:head v 227'178, mlcod=227'175)", 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:38.722921+0000", 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 99.255699234999994, 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051262199999999999, 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.722921+0000", 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.722921+0000", 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.722923+0000", 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4130000000000001e-06 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724645+0000", 2024-09-16T05:14:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001722131 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724699+0000", 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3522e-05 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724711+0000", 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2376e-05 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724779+0000", 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8391000000000003e-05 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.724805+0000", 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5177e-05 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.728005+0000", 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032002689999999999 2024-09-16T05:14:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:38.728048+0000", 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2941000000000001e-05 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1908 2.1e 2:7b642727:::smithi20457400-40:head [delete] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,3e,3a,39,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e259)", 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:10.676873+0000", 2024-09-16T05:14:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.301747833999997, 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059057830000000004, 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1908 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.676873+0000", 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.676873+0000", 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.676875+0000", 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4710000000000001e-06 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.676884+0000", 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0389999999999993e-06 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.676916+0000", 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1105000000000001e-05 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.676929+0000", 2024-09-16T05:14:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3414000000000001e-05 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.677022+0000", 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3295999999999996e-05 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.677033+0000", 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1039999999999999e-05 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.680466+0000", 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034322860000000001 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.680589+0000", 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000123793 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.680827+0000", 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00023794800000000001 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.682633+0000", 2024-09-16T05:14:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018060350000000001 2024-09-16T05:14:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:18.059 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:18.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.682652+0000", 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8471e-05 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.682720+0000", 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7780000000000005e-05 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.682779+0000", 2024-09-16T05:14:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9104999999999997e-05 2024-09-16T05:14:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:39 2.29 e259/253 2:94d33c56:::smithi20457400-8:4c v 259'246, mlcod=254'242)", 2024-09-16T05:14:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.300377014999995, 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069169729999999999, 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7399999999999999e-07 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678251+0000", 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1080000000000004e-06 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678321+0000", 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9762000000000001e-05 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678330+0000", 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0329999999999997e-06 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679306+0000", 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097581800000000004 2024-09-16T05:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.680633+0000", 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013273779999999999 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.685128+0000", 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044947850000000003 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.685161+0000", 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2814999999999997e-05 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:41 2.29 e259/253 2:96dc17ef:::smithi20457400-10:47 v 259'248, mlcod=254'242)", 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:10.679242+0000", 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.299378833999995, 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059489069999999998, 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679242+0000", 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679242+0000", 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679243+0000", 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9500000000000001e-07 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679250+0000", 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6650000000000003e-06 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679303+0000", 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2704000000000002e-05 2024-09-16T05:14:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679324+0000", 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0763000000000001e-05 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.681917+0000", 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002592717 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.681937+0000", 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0849e-05 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.685165+0000", 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003227345 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.685191+0000", 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5969000000000002e-05 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1985 2.1e 2:7b642727:::smithi20457400-40:head [write 541258~544620] snapc 54=[54,53,52,4d,4c,49,48,47,45,3e,3a,35,27] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:19.695448+0000", 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.2831726, 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050480100000000003, 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1985 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.695448+0000", 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.695448+0000", 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.695449+0000", 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8599999999999996e-07 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.698235+0000", 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027854239999999999 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.698249+0000", 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4663e-05 2024-09-16T05:14:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.698257+0000", 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7370000000000006e-06 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.698325+0000", 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7671999999999999e-05 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.698443+0000", 2024-09-16T05:14:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011790600000000001 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.698643+0000", 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020060499999999999 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.700106+0000", 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001463212 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.700418+0000", 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00031153499999999999 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.700460+0000", 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2459000000000001e-05 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.700496+0000", 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5811e-05 2024-09-16T05:14:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2057 2.e e272/209 2:75e143e9:::smithi20457400-17:head v 272'232, mlcod=272'229)", 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:23.738599+0000", 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 54.240022205999999, 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0043098240000000003, 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.738599+0000", 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.738599+0000", 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.738600+0000", 2024-09-16T05:14:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1969999999999999e-06 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.740365+0000", 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001764795 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.740423+0000", 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8579999999999998e-05 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.740436+0000", 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2599000000000001e-05 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.740521+0000", 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5134000000000002e-05 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.740551+0000", 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0318e-05 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.742856+0000", 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023046120000000002 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:23.742908+0000", 2024-09-16T05:14:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2589000000000001e-05 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2256 2.34 e290/278 2:2cffa4d6:::smithi20457400-50:head v 290'215, mlcod=281'211)", 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:41.694218+0000", 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.284402806000003, 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068076869999999998, 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:14:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.694218+0000", 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.694218+0000", 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.694218+0000", 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0599999999999999e-07 2024-09-16T05:14:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698452+0000", 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042336099999999996 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698507+0000", 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5390999999999999e-05 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:14:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698518+0000", 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0443e-05 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698586+0000", 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8015999999999994e-05 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698606+0000", 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0285999999999999e-05 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.700988+0000", 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002381958 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.701026+0000", 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7777000000000001e-05 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:14:18.086 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:14:18.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:17 smithi204 ceph-mon[30933]: pgmap v631: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:17 smithi204 ceph-mon[30933]: 2.4 deep-scrub starts 2024-09-16T05:14:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:17 smithi204 ceph-mon[30933]: 2.4 deep-scrub ok 2024-09-16T05:14:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:17 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/72581178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:17 smithi204 ceph-mon[30933]: osdmap e317: 12 total, 11 up, 12 in 2024-09-16T05:14:18.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:18.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:17 smithi196 ceph-mon[30802]: pgmap v631: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:17 smithi196 ceph-mon[30802]: 2.4 deep-scrub starts 2024-09-16T05:14:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:17 smithi196 ceph-mon[30802]: 2.4 deep-scrub ok 2024-09-16T05:14:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:17 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/72581178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:17 smithi196 ceph-mon[30802]: osdmap e317: 12 total, 11 up, 12 in 2024-09-16T05:14:18.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:18.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:18.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:18.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:18.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:18.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:18.471 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:14:18.471 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:14:18.472 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T05:14:18.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:14:18.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:18.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:18.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:18.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:18.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:14:18.697 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:18.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:18.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:18.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:18.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:14:18.912 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:18.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:18.949 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:18.949 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:18.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:18.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:19.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:19.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:19.128 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:19.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:19.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:19.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:19.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:19.325 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:19.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:19.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:19.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:19.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:19.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:19.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:19.612 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:14:19.612 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:14:19.612 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:14:19.612 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:14:19.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:19.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:19.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:19.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:14:19.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:19.808 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:14:19.809 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:14:19.831 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:19.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:19.834 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:19.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:19.858 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:14:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:19.879 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:19.899 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:19.900 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:19.920 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:19.921 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:19.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:19.941 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:19.942 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:19.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:19.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:19.964 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:19.965 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:19.987 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:19.988 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:19.988 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:19 smithi032 ceph-mon[27388]: pgmap v633: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:20.009 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:20.010 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:20.033 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:20.054 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:20.055 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:20.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:20.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:20.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:20.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:19 smithi204 ceph-mon[30933]: pgmap v633: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:20.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:20.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:20.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:20.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:19 smithi196 ceph-mon[30802]: pgmap v633: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:20.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:14:20.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:20.340 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:20.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:20.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:20.454 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":633,"stamp":"2024-09-16T05:14:18.080551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":335065978,"num_objects":149,"num_object_clones":98,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":11,"num_read":336,"num_read_kb":366536,"num_write":237,"num_write_kb":106120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13637,"ondisk_log_size":13637,"up":146,"acting":147,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":835292,"kb_used_data":427736,"kb_used_omap":18,"kb_used_meta":407277,"kb_avail":1124106532,"statfs":{"total":1151940427776,"available":1151085088768,"internally_reserved":0,"allocated":438001664,"data_stored":512241910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":417051961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.323989"},"pg_stats":[{"pgid":"2.41","version":"178'50","reported_seq":112,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980754+0000","last_change":"2024-09-16T05:13:59.750572+0000","last_active":"2024-09-16T05:14:12.980754+0000","last_peered":"2024-09-16T05:14:12.980754+0000","last_clean":"2024-09-16T05:14:12.980754+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:14:12.980754+0000","last_undegraded":"2024-09-16T05:14:12.980754+0000","last_fullsized":"2024-09-16T05:14:12.980754+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:13:48.349119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:48.349119+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:10.983476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012367630000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":1068,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980635+0000","last_change":"2024-09-16T05:13:59.754435+0000","last_active":"2024-09-16T05:14:12.980635+0000","last_peered":"2024-09-16T05:14:12.980635+0000","last_clean":"2024-09-16T05:14:12.980635+0000","last_became_active":"2024-09-16T05:13:45.682851+0000","last_became_peered":"2024-09-16T05:13:45.682851+0000","last_unstale":"2024-09-16T05:14:12.980635+0000","last_undegraded":"2024-09-16T05:14:12.980635+0000","last_fullsized":"2024-09-16T05:14:12.980635+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:13:47.461121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:47.461121+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:56.125966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003250944,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.3f","version":"310'238","reported_seq":1267,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.981009+0000","last_change":"2024-09-16T05:13:59.752025+0000","last_active":"2024-09-16T05:14:12.981009+0000","last_peered":"2024-09-16T05:14:12.981009+0000","last_clean":"2024-09-16T05:14:12.981009+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:14:12.981009+0000","last_undegraded":"2024-09-16T05:14:12.981009+0000","last_fullsized":"2024-09-16T05:14:12.981009+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"294'223","last_scrub_stamp":"2024-09-16T05:13:49.449292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:49.449292+0000","objects_scrubbed":3,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:08.493952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0031202909999999999,"stat_sum":{"num_bytes":11082440,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4770,"num_write":9,"num_write_kb":2633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.3e","version":"308'318","reported_seq":1312,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.764048+0000","last_change":"2024-09-16T05:13:59.944338+0000","last_active":"2024-09-16T05:14:11.764048+0000","last_peered":"2024-09-16T05:14:11.764048+0000","last_clean":"2024-09-16T05:14:11.764048+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:14:11.764048+0000","last_undegraded":"2024-09-16T05:14:11.764048+0000","last_fullsized":"2024-09-16T05:14:11.764048+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'304","last_scrub_stamp":"2024-09-16T05:13:48.937494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:48.937494+0000","objects_scrubbed":3,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:02.308735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.231939372,"stat_sum":{"num_bytes":6495767,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1568,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.479547+0000","last_change":"2024-09-16T05:13:59.720916+0000","last_active":"2024-09-16T05:14:12.479547+0000","last_peered":"2024-09-16T05:14:12.479547+0000","last_clean":"2024-09-16T05:14:12.479547+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:14:12.479547+0000","last_undegraded":"2024-09-16T05:14:12.479547+0000","last_fullsized":"2024-09-16T05:14:12.479547+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:13:49.521474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:49.521474+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:12.711634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086733199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.3c","version":"309'384","reported_seq":1496,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980911+0000","last_change":"2024-09-16T05:13:59.753891+0000","last_active":"2024-09-16T05:14:12.980911+0000","last_peered":"2024-09-16T05:14:12.980911+0000","last_clean":"2024-09-16T05:14:12.980911+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:14:12.980911+0000","last_undegraded":"2024-09-16T05:14:12.980911+0000","last_fullsized":"2024-09-16T05:14:12.980911+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"296'364","last_scrub_stamp":"2024-09-16T05:13:48.488582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:48.488582+0000","objects_scrubbed":8,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:54.512743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0034072130000000001,"stat_sum":{"num_bytes":19262175,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":16018,"num_write":8,"num_write_kb":3948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.3b","version":"300'290","reported_seq":1317,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.885042+0000","last_change":"2024-09-16T05:13:59.750014+0000","last_active":"2024-09-16T05:14:12.885042+0000","last_peered":"2024-09-16T05:14:12.885042+0000","last_clean":"2024-09-16T05:14:12.885042+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:14:12.885042+0000","last_undegraded":"2024-09-16T05:14:12.885042+0000","last_fullsized":"2024-09-16T05:14:12.885042+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'288","last_scrub_stamp":"2024-09-16T05:13:48.777048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:48.777048+0000","objects_scrubbed":2,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:08.948777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011205830000000001,"stat_sum":{"num_bytes":4573415,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.3a","version":"310'416","reported_seq":1394,"reported_epoch":316,"state":"active+clean+remapped","last_fresh":"2024-09-16T05:14:12.980543+0000","last_change":"2024-09-16T05:13:59.716671+0000","last_active":"2024-09-16T05:14:12.980543+0000","last_peered":"2024-09-16T05:14:12.980543+0000","last_clean":"2024-09-16T05:14:12.980543+0000","last_became_active":"2024-09-16T05:13:49.688656+0000","last_became_peered":"2024-09-16T05:13:49.688656+0000","last_unstale":"2024-09-16T05:14:12.980543+0000","last_undegraded":"2024-09-16T05:14:12.980543+0000","last_fullsized":"2024-09-16T05:14:12.980543+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":6,"last_scrub":"298'411","last_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:13:50.352903+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:59.765854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011152250000000001,"stat_sum":{"num_bytes":4201538,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9,6],"avail_no_missing":["9","6"],"object_location_counts":[{"shards":"6,9","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":116,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980445+0000","last_change":"2024-09-16T05:13:59.716170+0000","last_active":"2024-09-16T05:14:12.980445+0000","last_peered":"2024-09-16T05:14:12.980445+0000","last_clean":"2024-09-16T05:14:12.980445+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:14:12.980445+0000","last_undegraded":"2024-09-16T05:14:12.980445+0000","last_fullsized":"2024-09-16T05:14:12.980445+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:13:46.326483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:46.326483+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:08.998941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075283900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":114,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.054062+0000","last_change":"2024-09-16T05:13:59.749075+0000","last_active":"2024-09-16T05:14:12.054062+0000","last_peered":"2024-09-16T05:14:12.054062+0000","last_clean":"2024-09-16T05:14:12.054062+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:14:12.054062+0000","last_undegraded":"2024-09-16T05:14:12.054062+0000","last_fullsized":"2024-09-16T05:14:12.054062+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:13:46.576929+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:13:46.576929+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:48.020632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000544365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":200,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.981106+0000","last_change":"2024-09-16T05:13:59.751694+0000","last_active":"2024-09-16T05:14:12.981106+0000","last_peered":"2024-09-16T05:14:12.981106+0000","last_clean":"2024-09-16T05:14:12.981106+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:14:12.981106+0000","last_undegraded":"2024-09-16T05:14:12.981106+0000","last_fullsized":"2024-09-16T05:14:12.981106+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:32.155153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:13:32.155153+0000","last_clean_scrub_stamp":"2024-09-16T05:13:32.155153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:00.200602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017976629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":196,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.479875+0000","last_change":"2024-09-16T05:13:59.720577+0000","last_active":"2024-09-16T05:14:12.479875+0000","last_peered":"2024-09-16T05:14:12.479875+0000","last_clean":"2024-09-16T05:14:12.479875+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:14:12.479875+0000","last_undegraded":"2024-09-16T05:14:12.479875+0000","last_fullsized":"2024-09-16T05:14:12.479875+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:13:25.869308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.869308+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:42.298800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058460900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.35","version":"305'421","reported_seq":1470,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.479830+0000","last_change":"2024-09-16T05:13:59.721077+0000","last_active":"2024-09-16T05:14:12.479830+0000","last_peered":"2024-09-16T05:14:12.479830+0000","last_clean":"2024-09-16T05:14:12.479830+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:14:12.479830+0000","last_undegraded":"2024-09-16T05:14:12.479830+0000","last_fullsized":"2024-09-16T05:14:12.479830+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"276'397","last_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:13:28.279556+0000","objects_scrubbed":6,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:48.049872+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00084624600000000004,"stat_sum":{"num_bytes":6974938,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.34","version":"305'237","reported_seq":877,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.885198+0000","last_change":"2024-09-16T05:13:59.749741+0000","last_active":"2024-09-16T05:14:12.885198+0000","last_peered":"2024-09-16T05:14:12.885198+0000","last_clean":"2024-09-16T05:14:12.885198+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:14:12.885198+0000","last_undegraded":"2024-09-16T05:14:12.885198+0000","last_fullsized":"2024-09-16T05:14:12.885198+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"279'209","last_scrub_stamp":"2024-09-16T05:13:31.216492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:31.216492+0000","objects_scrubbed":5,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:55.270142+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0.001204608,"stat_sum":{"num_bytes":9549239,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":33268,"num_write":1,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":200,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980828+0000","last_change":"2024-09-16T05:13:59.751758+0000","last_active":"2024-09-16T05:14:12.980828+0000","last_peered":"2024-09-16T05:14:12.980828+0000","last_clean":"2024-09-16T05:14:12.980828+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:14:12.980828+0000","last_undegraded":"2024-09-16T05:14:12.980828+0000","last_fullsized":"2024-09-16T05:14:12.980828+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:13:28.549292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:28.549292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:29.130822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024372170000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.32","version":"305'97","reported_seq":251,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763743+0000","last_change":"2024-09-16T05:13:59.712907+0000","last_active":"2024-09-16T05:14:11.763743+0000","last_peered":"2024-09-16T05:14:11.763743+0000","last_clean":"2024-09-16T05:14:11.763743+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:14:11.763743+0000","last_undegraded":"2024-09-16T05:14:11.763743+0000","last_fullsized":"2024-09-16T05:14:11.763743+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:13:27.771535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:27.771535+0000","objects_scrubbed":4,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:41.961451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052160300000000004,"stat_sum":{"num_bytes":7923089,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":11,"num_write_kb":2927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.31","version":"304'130","reported_seq":778,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.979978+0000","last_change":"2024-09-16T05:13:59.713653+0000","last_active":"2024-09-16T05:14:12.979978+0000","last_peered":"2024-09-16T05:14:12.979978+0000","last_clean":"2024-09-16T05:14:12.979978+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:14:12.979978+0000","last_undegraded":"2024-09-16T05:14:12.979978+0000","last_fullsized":"2024-09-16T05:14:12.979978+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"280'123","last_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:13:31.958196+0000","objects_scrubbed":4,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:00.468750+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00048971400000000003,"stat_sum":{"num_bytes":3630312,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":164,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.981059+0000","last_change":"2024-09-16T05:13:59.750937+0000","last_active":"2024-09-16T05:14:12.981059+0000","last_peered":"2024-09-16T05:14:12.981059+0000","last_clean":"2024-09-16T05:14:12.981059+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:14:12.981059+0000","last_undegraded":"2024-09-16T05:14:12.981059+0000","last_fullsized":"2024-09-16T05:14:12.981059+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:13:27.778999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:27.778999+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:29.644149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017427250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.11","version":"272'115","reported_seq":741,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980529+0000","last_change":"2024-09-16T05:13:59.713284+0000","last_active":"2024-09-16T05:14:12.980529+0000","last_peered":"2024-09-16T05:14:12.980529+0000","last_clean":"2024-09-16T05:14:12.980529+0000","last_became_active":"2024-09-16T05:13:25.679267+0000","last_became_peered":"2024-09-16T05:13:25.679267+0000","last_unstale":"2024-09-16T05:14:12.980529+0000","last_undegraded":"2024-09-16T05:14:12.980529+0000","last_fullsized":"2024-09-16T05:14:12.980529+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:13:25.954175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.954175+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:47.795250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064440700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":652,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.054075+0000","last_change":"2024-09-16T05:13:59.749745+0000","last_active":"2024-09-16T05:14:12.054075+0000","last_peered":"2024-09-16T05:14:12.054075+0000","last_clean":"2024-09-16T05:14:12.054075+0000","last_became_active":"2024-09-16T05:13:25.680445+0000","last_became_peered":"2024-09-16T05:13:25.680445+0000","last_unstale":"2024-09-16T05:14:12.054075+0000","last_undegraded":"2024-09-16T05:14:12.054075+0000","last_fullsized":"2024-09-16T05:14:12.054075+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:13:27.647121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:27.647121+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:50.826964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001197278,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.f","version":"305'272","reported_seq":1307,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.054305+0000","last_change":"2024-09-16T05:13:59.749817+0000","last_active":"2024-09-16T05:14:12.054305+0000","last_peered":"2024-09-16T05:14:12.054305+0000","last_clean":"2024-09-16T05:14:12.054305+0000","last_became_active":"2024-09-16T05:13:25.680481+0000","last_became_peered":"2024-09-16T05:13:25.680481+0000","last_unstale":"2024-09-16T05:14:12.054305+0000","last_undegraded":"2024-09-16T05:14:12.054305+0000","last_fullsized":"2024-09-16T05:14:12.054305+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'248","last_scrub_stamp":"2024-09-16T05:13:28.609859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:28.609859+0000","objects_scrubbed":3,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:34.985795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001033808,"stat_sum":{"num_bytes":8879303,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.e","version":"272'232","reported_seq":1085,"reported_epoch":316,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:14:11.763681+0000","last_change":"2024-09-16T05:13:48.684131+0000","last_active":"2024-09-16T05:13:47.687474+0000","last_peered":"2024-09-16T05:14:11.763681+0000","last_clean":"2024-09-16T05:13:23.728963+0000","last_became_active":"2024-09-16T05:12:21.665948+0000","last_became_peered":"2024-09-16T05:13:48.684131+0000","last_unstale":"2024-09-16T05:14:11.763681+0000","last_undegraded":"2024-09-16T05:13:48.683613+0000","last_fullsized":"2024-09-16T05:13:48.683502+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:13:25.818442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.818442+0000","objects_scrubbed":2,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.621527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025018710000000001,"stat_sum":{"num_bytes":7025309,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":1843,"num_write":9,"num_write_kb":5116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"310'375","reported_seq":1399,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980936+0000","last_change":"2024-09-16T05:13:59.719107+0000","last_active":"2024-09-16T05:14:12.980936+0000","last_peered":"2024-09-16T05:14:12.980936+0000","last_clean":"2024-09-16T05:14:12.980936+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:14:12.980936+0000","last_undegraded":"2024-09-16T05:14:12.980936+0000","last_fullsized":"2024-09-16T05:14:12.980936+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'312","last_scrub_stamp":"2024-09-16T05:13:08.021858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.021858+0000","objects_scrubbed":6,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:26.587457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030601980000000001,"stat_sum":{"num_bytes":7195896,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":12165,"num_write":15,"num_write_kb":2848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.c","version":"311'213","reported_seq":1230,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763803+0000","last_change":"2024-09-16T05:13:59.712460+0000","last_active":"2024-09-16T05:14:11.763803+0000","last_peered":"2024-09-16T05:14:11.763803+0000","last_clean":"2024-09-16T05:14:11.763803+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:14:11.763803+0000","last_undegraded":"2024-09-16T05:14:11.763803+0000","last_fullsized":"2024-09-16T05:14:11.763803+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:13:07.600154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.600154+0000","objects_scrubbed":2,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:28.843559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051138799999999999,"stat_sum":{"num_bytes":7881426,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":17244,"num_write":10,"num_write_kb":5499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.8","version":"305'153","reported_seq":1173,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980201+0000","last_change":"2024-09-16T05:13:59.754191+0000","last_active":"2024-09-16T05:14:12.980201+0000","last_peered":"2024-09-16T05:14:12.980201+0000","last_clean":"2024-09-16T05:14:12.980201+0000","last_became_active":"2024-09-16T05:12:59.751630+0000","last_became_peered":"2024-09-16T05:12:59.751630+0000","last_unstale":"2024-09-16T05:14:12.980201+0000","last_undegraded":"2024-09-16T05:14:12.980201+0000","last_fullsized":"2024-09-16T05:14:12.980201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'114","last_scrub_stamp":"2024-09-16T05:13:06.516389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.516389+0000","objects_scrubbed":4,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:31.587318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041107899999999996,"stat_sum":{"num_bytes":10907705,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14139,"num_write":8,"num_write_kb":3559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.3","version":"281'58","reported_seq":1050,"reported_epoch":316,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:14:12.885277+0000","last_change":"2024-09-16T05:13:48.688781+0000","last_active":"2024-09-16T05:13:47.683286+0000","last_peered":"2024-09-16T05:14:12.885277+0000","last_clean":"2024-09-16T05:13:46.730800+0000","last_became_active":"2024-09-16T05:12:21.663580+0000","last_became_peered":"2024-09-16T05:13:48.688781+0000","last_unstale":"2024-09-16T05:14:12.885277+0000","last_undegraded":"2024-09-16T05:13:48.687893+0000","last_fullsized":"2024-09-16T05:13:48.686700+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:13:45.834929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:45.834929+0000","objects_scrubbed":3,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:12.680808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00025748200000000001,"stat_sum":{"num_bytes":6298197,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":50698,"num_write":10,"num_write_kb":2623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"35'139","reported_seq":841,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980784+0000","last_change":"2024-09-16T05:13:26.541877+0000","last_active":"2024-09-16T05:14:12.980784+0000","last_peered":"2024-09-16T05:14:12.980784+0000","last_clean":"2024-09-16T05:14:12.980784+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:14:12.980784+0000","last_undegraded":"2024-09-16T05:14:12.980784+0000","last_fullsized":"2024-09-16T05:14:12.980784+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:26.541796+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:26.541796+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:52.178618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1035,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.885141+0000","last_change":"2024-09-16T05:13:59.749041+0000","last_active":"2024-09-16T05:14:12.885141+0000","last_peered":"2024-09-16T05:14:12.885141+0000","last_clean":"2024-09-16T05:14:12.885141+0000","last_became_active":"2024-09-16T05:12:21.660508+0000","last_became_peered":"2024-09-16T05:12:21.660508+0000","last_unstale":"2024-09-16T05:14:12.885141+0000","last_undegraded":"2024-09-16T05:14:12.885141+0000","last_fullsized":"2024-09-16T05:14:12.885141+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:13:43.905153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:43.905153+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:13.012759+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051304699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":482,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.981162+0000","last_change":"2024-09-16T05:13:46.342155+0000","last_active":"2024-09-16T05:14:12.981162+0000","last_peered":"2024-09-16T05:14:12.981162+0000","last_clean":"2024-09-16T05:14:12.981162+0000","last_became_active":"2024-09-16T05:13:34.762556+0000","last_became_peered":"2024-09-16T05:13:34.762556+0000","last_unstale":"2024-09-16T05:14:12.981162+0000","last_undegraded":"2024-09-16T05:14:12.981162+0000","last_fullsized":"2024-09-16T05:14:12.981162+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:46.342074+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:46.342074+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:11.129045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":844,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980836+0000","last_change":"2024-09-16T05:14:07.583425+0000","last_active":"2024-09-16T05:14:12.980836+0000","last_peered":"2024-09-16T05:14:12.980836+0000","last_clean":"2024-09-16T05:14:12.980836+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:14:12.980836+0000","last_undegraded":"2024-09-16T05:14:12.980836+0000","last_fullsized":"2024-09-16T05:14:12.980836+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:07.583346+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:07.583346+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:25.639275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":420,"reported_epoch":316,"state":"undersized+peered","last_fresh":"2024-09-16T05:14:11.763505+0000","last_change":"2024-09-16T05:13:48.684037+0000","last_active":"2024-09-16T05:13:47.687186+0000","last_peered":"2024-09-16T05:14:11.763505+0000","last_clean":"2024-09-16T05:11:49.752659+0000","last_became_active":"2024-09-16T05:10:37.001937+0000","last_became_peered":"2024-09-16T05:13:48.684037+0000","last_unstale":"2024-09-16T05:14:11.763505+0000","last_undegraded":"2024-09-16T05:14:11.763505+0000","last_fullsized":"2024-09-16T05:13:48.683172+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:13:44.690544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:44.690544+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.621493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024364730000000002,"stat_sum":{"num_bytes":571429,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1437,"num_write":2,"num_write_kb":1770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"35'139","reported_seq":847,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.479662+0000","last_change":"2024-09-16T05:13:18.689030+0000","last_active":"2024-09-16T05:14:12.479662+0000","last_peered":"2024-09-16T05:14:12.479662+0000","last_clean":"2024-09-16T05:14:12.479662+0000","last_became_active":"2024-09-16T05:13:18.688801+0000","last_became_peered":"2024-09-16T05:13:18.688801+0000","last_unstale":"2024-09-16T05:14:12.479662+0000","last_undegraded":"2024-09-16T05:14:12.479662+0000","last_fullsized":"2024-09-16T05:14:12.479662+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:10.386111+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:10.386111+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:36.719242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1065,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980607+0000","last_change":"2024-09-16T05:13:59.754290+0000","last_active":"2024-09-16T05:14:12.980607+0000","last_peered":"2024-09-16T05:14:12.980607+0000","last_clean":"2024-09-16T05:14:12.980607+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:14:12.980607+0000","last_undegraded":"2024-09-16T05:14:12.980607+0000","last_fullsized":"2024-09-16T05:14:12.980607+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:13:44.507057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:44.507057+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:09.136832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033888220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":274,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980782+0000","last_change":"2024-09-16T05:13:59.719501+0000","last_active":"2024-09-16T05:14:12.980782+0000","last_peered":"2024-09-16T05:14:12.980782+0000","last_clean":"2024-09-16T05:14:12.980782+0000","last_became_active":"2024-09-16T05:13:15.680699+0000","last_became_peered":"2024-09-16T05:13:15.680699+0000","last_unstale":"2024-09-16T05:14:12.980782+0000","last_undegraded":"2024-09-16T05:14:12.980782+0000","last_fullsized":"2024-09-16T05:14:12.980782+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:05.895744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:05.895744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:34.479877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032244970000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":803,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763856+0000","last_change":"2024-09-16T05:13:47.941665+0000","last_active":"2024-09-16T05:14:11.763856+0000","last_peered":"2024-09-16T05:14:11.763856+0000","last_clean":"2024-09-16T05:14:11.763856+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:14:11.763856+0000","last_undegraded":"2024-09-16T05:14:11.763856+0000","last_fullsized":"2024-09-16T05:14:11.763856+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:47.941539+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:47.941539+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:10.860554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1031,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.479933+0000","last_change":"2024-09-16T05:13:59.720709+0000","last_active":"2024-09-16T05:14:12.479933+0000","last_peered":"2024-09-16T05:14:12.479933+0000","last_clean":"2024-09-16T05:14:12.479933+0000","last_became_active":"2024-09-16T05:12:59.759329+0000","last_became_peered":"2024-09-16T05:12:59.759329+0000","last_unstale":"2024-09-16T05:14:12.479933+0000","last_undegraded":"2024-09-16T05:14:12.479933+0000","last_fullsized":"2024-09-16T05:14:12.479933+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:13:07.299230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.299230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:16.217675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000685182,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":582,"reported_epoch":316,"state":"undersized+peered","last_fresh":"2024-09-16T05:14:12.981221+0000","last_change":"2024-09-16T05:13:48.694301+0000","last_active":"2024-09-16T05:13:47.687325+0000","last_peered":"2024-09-16T05:14:12.981221+0000","last_clean":"2024-09-16T05:13:46.684803+0000","last_became_active":"2024-09-16T05:13:34.764118+0000","last_became_peered":"2024-09-16T05:13:48.694301+0000","last_unstale":"2024-09-16T05:14:12.981221+0000","last_undegraded":"2024-09-16T05:14:12.981221+0000","last_fullsized":"2024-09-16T05:13:48.692651+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:44.751365+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:44.751365+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:16.360980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"305'464","reported_seq":1467,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980928+0000","last_change":"2024-09-16T05:13:59.718275+0000","last_active":"2024-09-16T05:14:12.980928+0000","last_peered":"2024-09-16T05:14:12.980928+0000","last_clean":"2024-09-16T05:14:12.980928+0000","last_became_active":"2024-09-16T05:12:59.751828+0000","last_became_peered":"2024-09-16T05:12:59.751828+0000","last_unstale":"2024-09-16T05:14:12.980928+0000","last_undegraded":"2024-09-16T05:14:12.980928+0000","last_fullsized":"2024-09-16T05:14:12.980928+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'430","last_scrub_stamp":"2024-09-16T05:13:07.051261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.051261+0000","objects_scrubbed":5,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:19.803491+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025459760000000001,"stat_sum":{"num_bytes":10050586,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":31963,"num_write":6,"num_write_kb":1322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":483,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980926+0000","last_change":"2024-09-16T05:13:47.335275+0000","last_active":"2024-09-16T05:14:12.980926+0000","last_peered":"2024-09-16T05:14:12.980926+0000","last_clean":"2024-09-16T05:14:12.980926+0000","last_became_active":"2024-09-16T05:13:34.764187+0000","last_became_peered":"2024-09-16T05:13:34.764187+0000","last_unstale":"2024-09-16T05:14:12.980926+0000","last_undegraded":"2024-09-16T05:14:12.980926+0000","last_fullsized":"2024-09-16T05:14:12.980926+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:47.335203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:47.335203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:02.228244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1105,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.479674+0000","last_change":"2024-09-16T05:13:59.721210+0000","last_active":"2024-09-16T05:14:12.479674+0000","last_peered":"2024-09-16T05:14:12.479674+0000","last_clean":"2024-09-16T05:14:12.479674+0000","last_became_active":"2024-09-16T05:12:59.749293+0000","last_became_peered":"2024-09-16T05:12:59.749293+0000","last_unstale":"2024-09-16T05:14:12.479674+0000","last_undegraded":"2024-09-16T05:14:12.479674+0000","last_fullsized":"2024-09-16T05:14:12.479674+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:13:06.313006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.313006+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:14.138580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097297099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":667,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.981031+0000","last_change":"2024-09-16T05:13:34.764429+0000","last_active":"2024-09-16T05:14:12.981031+0000","last_peered":"2024-09-16T05:14:12.981031+0000","last_clean":"2024-09-16T05:14:12.981031+0000","last_became_active":"2024-09-16T05:13:34.764109+0000","last_became_peered":"2024-09-16T05:13:34.764109+0000","last_unstale":"2024-09-16T05:14:12.981031+0000","last_undegraded":"2024-09-16T05:14:12.981031+0000","last_fullsized":"2024-09-16T05:14:12.981031+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:12:50.665816+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:12:50.665816+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:19.290266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":1207,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.054132+0000","last_change":"2024-09-16T05:13:59.749486+0000","last_active":"2024-09-16T05:14:12.054132+0000","last_peered":"2024-09-16T05:14:12.054132+0000","last_clean":"2024-09-16T05:14:12.054132+0000","last_became_active":"2024-09-16T05:12:59.748431+0000","last_became_peered":"2024-09-16T05:12:59.748431+0000","last_unstale":"2024-09-16T05:14:12.054132+0000","last_undegraded":"2024-09-16T05:14:12.054132+0000","last_fullsized":"2024-09-16T05:14:12.054132+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:13:06.465328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.465328+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:25.635572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081597799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":842,"reported_epoch":316,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:14:12.885222+0000","last_change":"2024-09-16T05:13:48.687741+0000","last_active":"2024-09-16T05:13:47.683147+0000","last_peered":"2024-09-16T05:14:12.885222+0000","last_clean":"2024-09-16T05:13:46.729894+0000","last_became_active":"2024-09-16T05:10:38.967671+0000","last_became_peered":"2024-09-16T05:13:48.687741+0000","last_unstale":"2024-09-16T05:14:12.885222+0000","last_undegraded":"2024-09-16T05:13:48.686365+0000","last_fullsized":"2024-09-16T05:13:48.686284+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:44.869889+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:44.869889+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:01.702429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"309'419","reported_seq":1447,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763704+0000","last_change":"2024-09-16T05:13:59.712824+0000","last_active":"2024-09-16T05:14:11.763704+0000","last_peered":"2024-09-16T05:14:11.763704+0000","last_clean":"2024-09-16T05:14:11.763704+0000","last_became_active":"2024-09-16T05:12:59.751132+0000","last_became_peered":"2024-09-16T05:12:59.751132+0000","last_unstale":"2024-09-16T05:14:11.763704+0000","last_undegraded":"2024-09-16T05:14:11.763704+0000","last_fullsized":"2024-09-16T05:14:11.763704+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'336","last_scrub_stamp":"2024-09-16T05:13:06.619984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.619984+0000","objects_scrubbed":8,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:33.299770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068740299999999995,"stat_sum":{"num_bytes":15605493,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":34537,"num_write":26,"num_write_kb":15598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":844,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980725+0000","last_change":"2024-09-16T05:14:11.602321+0000","last_active":"2024-09-16T05:14:12.980725+0000","last_peered":"2024-09-16T05:14:12.980725+0000","last_clean":"2024-09-16T05:14:12.980725+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:14:12.980725+0000","last_undegraded":"2024-09-16T05:14:12.980725+0000","last_fullsized":"2024-09-16T05:14:12.980725+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:11.602237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:11.602237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:17.033838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1247,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.479498+0000","last_change":"2024-09-16T05:13:59.720435+0000","last_active":"2024-09-16T05:14:12.479498+0000","last_peered":"2024-09-16T05:14:12.479498+0000","last_clean":"2024-09-16T05:14:12.479498+0000","last_became_active":"2024-09-16T05:12:59.750592+0000","last_became_peered":"2024-09-16T05:12:59.750592+0000","last_unstale":"2024-09-16T05:14:12.479498+0000","last_undegraded":"2024-09-16T05:14:12.479498+0000","last_fullsized":"2024-09-16T05:14:12.479498+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:13:08.517287+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.517287+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:18.770956+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0.00053218999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":845,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980595+0000","last_change":"2024-09-16T05:13:49.303891+0000","last_active":"2024-09-16T05:14:12.980595+0000","last_peered":"2024-09-16T05:14:12.980595+0000","last_clean":"2024-09-16T05:14:12.980595+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:14:12.980595+0000","last_undegraded":"2024-09-16T05:14:12.980595+0000","last_fullsized":"2024-09-16T05:14:12.980595+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:13:49.303815+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:13:49.303815+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:55.338081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":164,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980766+0000","last_change":"2024-09-16T05:13:59.750146+0000","last_active":"2024-09-16T05:14:12.980766+0000","last_peered":"2024-09-16T05:14:12.980766+0000","last_clean":"2024-09-16T05:14:12.980766+0000","last_became_active":"2024-09-16T05:13:31.747644+0000","last_became_peered":"2024-09-16T05:13:31.747644+0000","last_unstale":"2024-09-16T05:14:12.980766+0000","last_undegraded":"2024-09-16T05:14:12.980766+0000","last_fullsized":"2024-09-16T05:14:12.980766+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:13:26.508703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:26.508703+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:42.400619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046331,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.13","version":"181'15","reported_seq":656,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980751+0000","last_change":"2024-09-16T05:13:59.716386+0000","last_active":"2024-09-16T05:14:12.980751+0000","last_peered":"2024-09-16T05:14:12.980751+0000","last_clean":"2024-09-16T05:14:12.980751+0000","last_became_active":"2024-09-16T05:13:25.681297+0000","last_became_peered":"2024-09-16T05:13:25.681297+0000","last_unstale":"2024-09-16T05:14:12.980751+0000","last_undegraded":"2024-09-16T05:14:12.980751+0000","last_fullsized":"2024-09-16T05:14:12.980751+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:13:27.005361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:27.005361+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:55.661525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089774899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.14","version":"311'229","reported_seq":852,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980633+0000","last_change":"2024-09-16T05:13:59.716841+0000","last_active":"2024-09-16T05:14:12.980633+0000","last_peered":"2024-09-16T05:14:12.980633+0000","last_clean":"2024-09-16T05:14:12.980633+0000","last_became_active":"2024-09-16T05:13:25.684888+0000","last_became_peered":"2024-09-16T05:13:25.684888+0000","last_unstale":"2024-09-16T05:14:12.980633+0000","last_undegraded":"2024-09-16T05:14:12.980633+0000","last_fullsized":"2024-09-16T05:14:12.980633+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'197","last_scrub_stamp":"2024-09-16T05:13:28.044263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:28.044263+0000","objects_scrubbed":5,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:36.170973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001149668,"stat_sum":{"num_bytes":9680832,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.15","version":"310'409","reported_seq":1437,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980680+0000","last_change":"2024-09-16T05:13:59.720083+0000","last_active":"2024-09-16T05:14:12.980680+0000","last_peered":"2024-09-16T05:14:12.980680+0000","last_clean":"2024-09-16T05:14:12.980680+0000","last_became_active":"2024-09-16T05:13:25.680026+0000","last_became_peered":"2024-09-16T05:13:25.680026+0000","last_unstale":"2024-09-16T05:14:12.980680+0000","last_undegraded":"2024-09-16T05:14:12.980680+0000","last_fullsized":"2024-09-16T05:14:12.980680+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'387","last_scrub_stamp":"2024-09-16T05:13:25.974552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:25.974552+0000","objects_scrubbed":4,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:33.060147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041832529999999996,"stat_sum":{"num_bytes":5566629,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":19506,"num_write":12,"num_write_kb":3470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.16","version":"304'74","reported_seq":740,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.885186+0000","last_change":"2024-09-16T05:13:59.749389+0000","last_active":"2024-09-16T05:14:12.885186+0000","last_peered":"2024-09-16T05:14:12.885186+0000","last_clean":"2024-09-16T05:14:12.885186+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:14:12.885186+0000","last_undegraded":"2024-09-16T05:14:12.885186+0000","last_fullsized":"2024-09-16T05:14:12.885186+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:13:26.848310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:26.848310+0000","objects_scrubbed":3,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:34.665407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080028,"stat_sum":{"num_bytes":3142942,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":6148,"num_write":1,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.17","version":"0'0","reported_seq":834,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980545+0000","last_change":"2024-09-16T05:13:59.713728+0000","last_active":"2024-09-16T05:14:12.980545+0000","last_peered":"2024-09-16T05:14:12.980545+0000","last_clean":"2024-09-16T05:14:12.980545+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:14:12.980545+0000","last_undegraded":"2024-09-16T05:14:12.980545+0000","last_fullsized":"2024-09-16T05:14:12.980545+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:26.947197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:26.947197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:30.434279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077483000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1049,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980408+0000","last_change":"2024-09-16T05:13:59.716017+0000","last_active":"2024-09-16T05:14:12.980408+0000","last_peered":"2024-09-16T05:14:12.980408+0000","last_clean":"2024-09-16T05:14:12.980408+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:14:12.980408+0000","last_undegraded":"2024-09-16T05:14:12.980408+0000","last_fullsized":"2024-09-16T05:14:12.980408+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:13:44.314538+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:13:44.314538+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:03.802864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064059200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":654,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763355+0000","last_change":"2024-09-16T05:13:59.711722+0000","last_active":"2024-09-16T05:14:11.763355+0000","last_peered":"2024-09-16T05:14:11.763355+0000","last_clean":"2024-09-16T05:14:11.763355+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:14:11.763355+0000","last_undegraded":"2024-09-16T05:14:11.763355+0000","last_fullsized":"2024-09-16T05:14:11.763355+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:13:45.525995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:45.525995+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:12.345006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000482041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.1a","version":"311'443","reported_seq":1460,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.054230+0000","last_change":"2024-09-16T05:13:59.750502+0000","last_active":"2024-09-16T05:14:12.054230+0000","last_peered":"2024-09-16T05:14:12.054230+0000","last_clean":"2024-09-16T05:14:12.054230+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:14:12.054230+0000","last_undegraded":"2024-09-16T05:14:12.054230+0000","last_fullsized":"2024-09-16T05:14:12.054230+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'408","last_scrub_stamp":"2024-09-16T05:13:44.603313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:44.603313+0000","objects_scrubbed":7,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:04.446236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001391602,"stat_sum":{"num_bytes":13933474,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":12,"num_read_kb":16926,"num_write":9,"num_write_kb":4715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.1b","version":"304'309","reported_seq":1361,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763761+0000","last_change":"2024-09-16T05:13:59.712552+0000","last_active":"2024-09-16T05:14:11.763761+0000","last_peered":"2024-09-16T05:14:11.763761+0000","last_clean":"2024-09-16T05:14:11.763761+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:14:11.763761+0000","last_undegraded":"2024-09-16T05:14:11.763761+0000","last_fullsized":"2024-09-16T05:14:11.763761+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"288'286","last_scrub_stamp":"2024-09-16T05:13:43.896438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:43.896438+0000","objects_scrubbed":8,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:44.633239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00056189499999999997,"stat_sum":{"num_bytes":20655993,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.1c","version":"305'360","reported_seq":1452,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.981031+0000","last_change":"2024-09-16T05:13:59.750102+0000","last_active":"2024-09-16T05:14:12.981031+0000","last_peered":"2024-09-16T05:14:12.981031+0000","last_clean":"2024-09-16T05:14:12.981031+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:14:12.981031+0000","last_undegraded":"2024-09-16T05:14:12.981031+0000","last_fullsized":"2024-09-16T05:14:12.981031+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"288'352","last_scrub_stamp":"2024-09-16T05:13:44.318714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:44.318714+0000","objects_scrubbed":2,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:45.638666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093662499999999996,"stat_sum":{"num_bytes":2050487,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.1d","version":"311'525","reported_seq":1639,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.981121+0000","last_change":"2024-09-16T05:13:59.752840+0000","last_active":"2024-09-16T05:14:12.981121+0000","last_peered":"2024-09-16T05:14:12.981121+0000","last_clean":"2024-09-16T05:14:12.981121+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:14:12.981121+0000","last_undegraded":"2024-09-16T05:14:12.981121+0000","last_fullsized":"2024-09-16T05:14:12.981121+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"293'485","last_scrub_stamp":"2024-09-16T05:13:45.300504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:45.300504+0000","objects_scrubbed":14,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:45.708407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0033270169999999998,"stat_sum":{"num_bytes":25651081,"num_objects":14,"num_object_clones":10,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":11,"num_read_kb":13362,"num_write":6,"num_write_kb":3959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.1e","version":"308'305","reported_seq":1350,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763633+0000","last_change":"2024-09-16T05:13:59.713920+0000","last_active":"2024-09-16T05:14:11.763633+0000","last_peered":"2024-09-16T05:14:11.763633+0000","last_clean":"2024-09-16T05:14:11.763633+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:14:11.763633+0000","last_undegraded":"2024-09-16T05:14:11.763633+0000","last_fullsized":"2024-09-16T05:14:11.763633+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'295","last_scrub_stamp":"2024-09-16T05:13:44.507635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:44.507635+0000","objects_scrubbed":4,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:48.078809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023401020000000002,"stat_sum":{"num_bytes":9731319,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":1266,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980002+0000","last_change":"2024-09-16T05:13:59.713647+0000","last_active":"2024-09-16T05:14:12.980002+0000","last_peered":"2024-09-16T05:14:12.980002+0000","last_clean":"2024-09-16T05:14:12.980002+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:14:12.980002+0000","last_undegraded":"2024-09-16T05:14:12.980002+0000","last_fullsized":"2024-09-16T05:14:12.980002+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:13:43.947153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:43.947153+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:00.796549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043975600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.20","version":"308'74","reported_seq":1120,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:15.560858+0000","last_change":"2024-09-16T05:14:15.560894+0000","last_active":"2024-09-16T05:14:15.560858+0000","last_peered":"2024-09-16T05:14:15.560858+0000","last_clean":"2024-09-16T05:14:15.560858+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:14:15.560858+0000","last_undegraded":"2024-09-16T05:14:15.560858+0000","last_fullsized":"2024-09-16T05:14:15.560858+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"308'74","last_scrub_stamp":"2024-09-16T05:14:15.560781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:15.560781+0000","objects_scrubbed":3,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:27.924668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041985319999999996,"stat_sum":{"num_bytes":7651219,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":42972,"num_write":6,"num_write_kb":4192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":603,"reported_epoch":316,"state":"undersized+peered","last_fresh":"2024-09-16T05:14:11.763683+0000","last_change":"2024-09-16T05:13:48.685196+0000","last_active":"2024-09-16T05:12:22.800525+0000","last_peered":"2024-09-16T05:14:11.763683+0000","last_clean":"2024-09-16T05:12:21.770600+0000","last_became_active":"2024-09-16T05:12:21.667517+0000","last_became_peered":"2024-09-16T05:13:48.685196+0000","last_unstale":"2024-09-16T05:14:11.763683+0000","last_undegraded":"2024-09-16T05:14:11.763683+0000","last_fullsized":"2024-09-16T05:13:48.684421+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:13:31.784297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:31.784297+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:11.798583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019954030000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.22","version":"182'20","reported_seq":172,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980930+0000","last_change":"2024-09-16T05:14:07.323379+0000","last_active":"2024-09-16T05:14:12.980930+0000","last_peered":"2024-09-16T05:14:12.980930+0000","last_clean":"2024-09-16T05:14:12.980930+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:14:12.980930+0000","last_undegraded":"2024-09-16T05:14:12.980930+0000","last_fullsized":"2024-09-16T05:14:12.980930+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:14:07.323272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:07.323272+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:28.479124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016348300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":521,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763462+0000","last_change":"2024-09-16T05:13:59.711783+0000","last_active":"2024-09-16T05:14:11.763462+0000","last_peered":"2024-09-16T05:14:11.763462+0000","last_clean":"2024-09-16T05:14:11.763462+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:14:11.763462+0000","last_undegraded":"2024-09-16T05:14:11.763462+0000","last_fullsized":"2024-09-16T05:14:11.763462+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:13:32.510259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:32.510259+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:40.734027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046518800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":335,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980533+0000","last_change":"2024-09-16T05:13:59.717389+0000","last_active":"2024-09-16T05:14:12.980533+0000","last_peered":"2024-09-16T05:14:12.980533+0000","last_clean":"2024-09-16T05:14:12.980533+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:14:12.980533+0000","last_undegraded":"2024-09-16T05:14:12.980533+0000","last_fullsized":"2024-09-16T05:14:12.980533+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:13:06.032520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.032520+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:33.539804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001515071,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.25","version":"310'536","reported_seq":1511,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.054225+0000","last_change":"2024-09-16T05:13:59.750271+0000","last_active":"2024-09-16T05:14:12.054225+0000","last_peered":"2024-09-16T05:14:12.054225+0000","last_clean":"2024-09-16T05:14:12.054225+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:14:12.054225+0000","last_undegraded":"2024-09-16T05:14:12.054225+0000","last_fullsized":"2024-09-16T05:14:12.054225+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"254'432","last_scrub_stamp":"2024-09-16T05:13:07.464459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.464459+0000","objects_scrubbed":9,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:17.940963+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012136650000000001,"stat_sum":{"num_bytes":26487898,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":26476,"num_write":16,"num_write_kb":9276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":329,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980984+0000","last_change":"2024-09-16T05:13:59.752236+0000","last_active":"2024-09-16T05:14:12.980984+0000","last_peered":"2024-09-16T05:14:12.980984+0000","last_clean":"2024-09-16T05:14:12.980984+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:14:12.980984+0000","last_undegraded":"2024-09-16T05:14:12.980984+0000","last_fullsized":"2024-09-16T05:14:12.980984+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:13:07.538943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:07.538943+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:13.539184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027769460000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":341,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.981080+0000","last_change":"2024-09-16T05:13:59.750193+0000","last_active":"2024-09-16T05:14:12.981080+0000","last_peered":"2024-09-16T05:14:12.981080+0000","last_clean":"2024-09-16T05:14:12.981080+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:14:12.981080+0000","last_undegraded":"2024-09-16T05:14:12.981080+0000","last_fullsized":"2024-09-16T05:14:12.981080+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:13:09.483428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:09.483428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.633592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073071200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1122,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:11.763342+0000","last_change":"2024-09-16T05:13:59.712060+0000","last_active":"2024-09-16T05:14:11.763342+0000","last_peered":"2024-09-16T05:14:11.763342+0000","last_clean":"2024-09-16T05:14:11.763342+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:14:11.763342+0000","last_undegraded":"2024-09-16T05:14:11.763342+0000","last_fullsized":"2024-09-16T05:14:11.763342+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:13:06.496116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.496116+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:20.387730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081924099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.29","version":"310'362","reported_seq":1440,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980208+0000","last_change":"2024-09-16T05:13:59.714499+0000","last_active":"2024-09-16T05:14:12.980208+0000","last_peered":"2024-09-16T05:14:12.980208+0000","last_clean":"2024-09-16T05:14:12.980208+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:14:12.980208+0000","last_undegraded":"2024-09-16T05:14:12.980208+0000","last_fullsized":"2024-09-16T05:14:12.980208+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"254'244","last_scrub_stamp":"2024-09-16T05:13:06.019617+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.019617+0000","objects_scrubbed":16,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:34.921634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018815539999999999,"stat_sum":{"num_bytes":34554459,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":24,"num_read_kb":15939,"num_write":34,"num_write_kb":10364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1331,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980713+0000","last_change":"2024-09-16T05:13:59.751008+0000","last_active":"2024-09-16T05:14:12.980713+0000","last_peered":"2024-09-16T05:14:12.980713+0000","last_clean":"2024-09-16T05:14:12.980713+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:14:12.980713+0000","last_undegraded":"2024-09-16T05:14:12.980713+0000","last_fullsized":"2024-09-16T05:14:12.980713+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:13:08.516105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.516105+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:15.926590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018665190000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.2b","version":"293'194","reported_seq":1169,"reported_epoch":316,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:14:11.763913+0000","last_change":"2024-09-16T05:13:48.687621+0000","last_active":"2024-09-16T05:13:47.685503+0000","last_peered":"2024-09-16T05:14:11.763913+0000","last_clean":"2024-09-16T05:13:44.689427+0000","last_became_active":"2024-09-16T05:13:04.678973+0000","last_became_peered":"2024-09-16T05:13:48.687621+0000","last_unstale":"2024-09-16T05:14:11.763913+0000","last_undegraded":"2024-09-16T05:13:48.687140+0000","last_fullsized":"2024-09-16T05:13:48.687023+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"253'179","last_scrub_stamp":"2024-09-16T05:13:06.865580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:06.865580+0000","objects_scrubbed":4,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:33.123769+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.00033812600000000001,"stat_sum":{"num_bytes":5848902,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":7,"num_write_kb":4385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2c","version":"241'194","reported_seq":1158,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.054147+0000","last_change":"2024-09-16T05:13:59.749104+0000","last_active":"2024-09-16T05:14:12.054147+0000","last_peered":"2024-09-16T05:14:12.054147+0000","last_clean":"2024-09-16T05:14:12.054147+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:14:12.054147+0000","last_undegraded":"2024-09-16T05:14:12.054147+0000","last_fullsized":"2024-09-16T05:14:12.054147+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:13:08.484521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:08.484521+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:37.641494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047807900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.2d","version":"274'307","reported_seq":1312,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980803+0000","last_change":"2024-09-16T05:13:59.752383+0000","last_active":"2024-09-16T05:14:12.980803+0000","last_peered":"2024-09-16T05:14:12.980803+0000","last_clean":"2024-09-16T05:14:12.980803+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:14:12.980803+0000","last_undegraded":"2024-09-16T05:14:12.980803+0000","last_fullsized":"2024-09-16T05:14:12.980803+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'306","last_scrub_stamp":"2024-09-16T05:13:10.471980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:10.471980+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:27.742602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002868453,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.2e","version":"305'270","reported_seq":1284,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980976+0000","last_change":"2024-09-16T05:13:59.750241+0000","last_active":"2024-09-16T05:14:12.980976+0000","last_peered":"2024-09-16T05:14:12.980976+0000","last_clean":"2024-09-16T05:14:12.980976+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:14:12.980976+0000","last_undegraded":"2024-09-16T05:14:12.980976+0000","last_fullsized":"2024-09-16T05:14:12.980976+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"279'238","last_scrub_stamp":"2024-09-16T05:13:31.497779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:31.497779+0000","objects_scrubbed":6,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:42.961100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001068341,"stat_sum":{"num_bytes":14605283,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":7121,"num_write":8,"num_write_kb":1663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]},{"pgid":"2.2f","version":"311'266","reported_seq":1328,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-16T05:14:12.980773+0000","last_change":"2024-09-16T05:13:59.717707+0000","last_active":"2024-09-16T05:14:12.980773+0000","last_peered":"2024-09-16T05:14:12.980773+0000","last_clean":"2024-09-16T05:14:12.980773+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:14:12.980773+0000","last_undegraded":"2024-09-16T05:14:12.980773+0000","last_fullsized":"2024-09-16T05:14:12.980773+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"278'248","last_scrub_stamp":"2024-09-16T05:13:30.085727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:13:30.085727+0000","objects_scrubbed":7,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:14:44.369492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017828869999999999,"stat_sum":{"num_bytes":6806835,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"45","length":"1"},{"start":"4d","length":"1"},{"start":"57","length":"1"},{"start":"59","length":"1"},{"start":"5c","length":"1"},{"start":"5f","length":"1"},{"start":"65","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":66,"stat_sum":{"num_bytes":334475610,"num_objects":147,"num_object_clones":98,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":11,"num_read":336,"num_read_kb":366536,"num_write":237,"num_write_kb":106120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":394424320,"data_stored":485303540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12108,"ondisk_log_size":12108,"up":126,"acting":127,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":20,"acting":20,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089406,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99532,"kb_used_data":64180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93645620,"statfs":{"total":95995035648,"available":95893114880,"internally_reserved":0,"allocated":65720320,"data_stored":80662365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285635,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57260,"kb_used_data":26072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687892,"statfs":{"total":95995035648,"available":95936401408,"internally_reserved":0,"allocated":26697728,"data_stored":29530801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514569,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85076,"kb_used_data":49728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93660076,"statfs":{"total":95995035648,"available":95907917824,"internally_reserved":0,"allocated":50921472,"data_stored":65869201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841611,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62312,"kb_used_data":26960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93682840,"statfs":{"total":95995035648,"available":95931228160,"internally_reserved":0,"allocated":27607040,"data_stored":31306947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103249,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95944,"kb_used_data":60600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93649208,"statfs":{"total":95995035648,"available":95896788992,"internally_reserved":0,"allocated":62054400,"data_stored":70124815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397592,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86276,"kb_used_data":50928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93658876,"statfs":{"total":95995035648,"available":95906689024,"internally_reserved":0,"allocated":52150272,"data_stored":64397310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757339,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55356,"kb_used_data":24168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93689796,"statfs":{"total":95995035648,"available":95938351104,"internally_reserved":0,"allocated":24748032,"data_stored":26964451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920865,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49316,"kb_used_data":18124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93695836,"statfs":{"total":95995035648,"available":95944536064,"internally_reserved":0,"allocated":18558976,"data_stored":21684796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90324,"kb_used_data":54972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93654828,"statfs":{"total":95995035648,"available":95902543872,"internally_reserved":0,"allocated":56291328,"data_stored":68190767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47448,"kb_used_data":12100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93697704,"statfs":{"total":95995035648,"available":95946448896,"internally_reserved":0,"allocated":12390400,"data_stored":12214983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54640,"kb_used_data":23448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690512,"statfs":{"total":95995035648,"available":95939084288,"internally_reserved":0,"allocated":24010752,"data_stored":24965235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51808,"kb_used_data":16456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93693344,"statfs":{"total":95995035648,"available":95941984256,"internally_reserved":0,"allocated":16850944,"data_stored":16330239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13107200,"data_stored":13980569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20267008,"data_stored":22615565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10854400,"data_stored":11979591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":52547584,"data_stored":65841097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14815232,"data_stored":19335126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20799488,"data_stored":24413533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":48406528,"data_stored":62047640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":58310656,"data_stored":67775145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":23535616,"data_stored":28629597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":47177728,"data_stored":63519531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":22953984,"data_stored":27181131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61648896,"data_stored":77985015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:14:20.458 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:14:20.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 76 2024-09-16T05:14:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:20.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:20.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:20.557 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:20.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:20.592 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:20.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:20.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:20.712 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:20.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:20.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:20.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:20.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:20.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:20.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:20.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:20.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:20 smithi196 ceph-mon[30802]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:14:20.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4074310035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:21.044 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:20 smithi032 ceph-mon[27388]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:14:21.044 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4074310035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:21.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:21.078 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:14:21.079 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:14:21.091 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:21.091 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:21.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:21.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:21.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:20 smithi204 ceph-mon[30933]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:14:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4074310035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:21.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:21.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:21.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:21.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:21.313 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:21.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:21.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:21.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:21.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:21.477 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:21.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:21.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:21.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:21.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:21.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:21.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:14:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:21.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:21.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:21.792 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:14:21.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:21.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:21.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:21.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:21.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:21 smithi032 ceph-mon[27388]: pgmap v634: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:21.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3064241597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:21.956 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:21.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:21.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:21.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:21.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:21 smithi196 ceph-mon[30802]: pgmap v634: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:21.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3064241597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:22.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:22.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:22.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:21 smithi204 ceph-mon[30933]: pgmap v634: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3064241597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:22.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:22.234 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:22.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:22.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:22.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:22.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:22.447 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:22.447 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:22.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:22.486 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:22.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:22.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:22.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:22.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:22.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:22.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:22.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:22.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pg_num to 76 2024-09-16T05:14:22.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:22.863 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:22.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:22 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3064241597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:22 smithi032 ceph-mon[27388]: osdmap e318: 12 total, 11 up, 12 in 2024-09-16T05:14:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2871338687' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]: dispatch 2024-09-16T05:14:22.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:22.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:22.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:22.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:22 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3064241597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:22 smithi196 ceph-mon[30802]: osdmap e318: 12 total, 11 up, 12 in 2024-09-16T05:14:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2871338687' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]: dispatch 2024-09-16T05:14:23.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:14:23.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:14:23.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:23.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:23.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:22 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3064241597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:22 smithi204 ceph-mon[30933]: osdmap e318: 12 total, 11 up, 12 in 2024-09-16T05:14:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2871338687' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]: dispatch 2024-09-16T05:14:23.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:23.238 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:23.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:23.286 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:23.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:23.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:23.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:23.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:23.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:23.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:23.513 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:23.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:23.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:23.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:23.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:23.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:23.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:23.762 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:23.763 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:23.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:23.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:23.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:23.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:23.970 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:23.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:23.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:23 smithi196 ceph-mon[30802]: pgmap v636: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2871338687' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]': finished 2024-09-16T05:14:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:23 smithi196 ceph-mon[30802]: osdmap e319: 12 total, 11 up, 12 in 2024-09-16T05:14:23.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:23.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:23.992 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:23.992 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:23.992 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:14:24.013 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:24.013 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:24.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:23 smithi032 ceph-mon[27388]: pgmap v636: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:24.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2871338687' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]': finished 2024-09-16T05:14:24.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:23 smithi032 ceph-mon[27388]: osdmap e319: 12 total, 11 up, 12 in 2024-09-16T05:14:24.034 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:24.034 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:24.055 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:24.076 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:24.077 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:24.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:24.098 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:24.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:24.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:24.121 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:24.143 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:24.143 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:24.165 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:24.165 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:23 smithi204 ceph-mon[30933]: pgmap v636: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2871338687' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "76"}]': finished 2024-09-16T05:14:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:23 smithi204 ceph-mon[30933]: osdmap e319: 12 total, 11 up, 12 in 2024-09-16T05:14:24.187 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:24.187 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:24.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:24.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:24.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:24.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:24.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:14:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:24.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:24.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:24.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:24.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:24.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:24.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:24.597 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:24.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:14:24.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:24.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:24.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:24.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:24.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:24.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:24.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:24.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:24.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:14:24.960 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:14:24.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:24.985 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1214559843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:24.986 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]: dispatch 2024-09-16T05:14:24.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:24.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:25.094 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:14:25.095 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:14:25.095 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:14:25.095 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:14:25.095 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:14:25.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:25.121 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:25.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:25.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1214559843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:25.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]: dispatch 2024-09-16T05:14:25.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1214559843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]: dispatch 2024-09-16T05:14:25.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:25.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:25.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:25.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:25.377 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:25.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:25.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:25.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:25.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:25.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:25.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:25.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:25.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:14:25.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:25.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:25.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:25.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:25.817 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:14:25.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:25.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:25.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:25.951 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:25 smithi196 ceph-mon[30802]: pgmap v638: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]': finished 2024-09-16T05:14:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:25 smithi196 ceph-mon[30802]: osdmap e320: 12 total, 11 up, 12 in 2024-09-16T05:14:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/836707416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/836707416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:25 smithi196 ceph-mon[30802]: osdmap e321: 12 total, 11 up, 12 in 2024-09-16T05:14:25.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:26.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:26.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:26.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:26.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:26.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:25 smithi032 ceph-mon[27388]: pgmap v638: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:26.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]': finished 2024-09-16T05:14:26.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:25 smithi032 ceph-mon[27388]: osdmap e320: 12 total, 11 up, 12 in 2024-09-16T05:14:26.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/836707416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:26.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/836707416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:26.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:25 smithi032 ceph-mon[27388]: osdmap e321: 12 total, 11 up, 12 in 2024-09-16T05:14:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:25 smithi204 ceph-mon[30933]: pgmap v638: 77 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 69 active+clean; 320 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "76"}]': finished 2024-09-16T05:14:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:25 smithi204 ceph-mon[30933]: osdmap e320: 12 total, 11 up, 12 in 2024-09-16T05:14:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/836707416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/836707416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:25 smithi204 ceph-mon[30933]: osdmap e321: 12 total, 11 up, 12 in 2024-09-16T05:14:26.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:26.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:26.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:26.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:26.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:26.484 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:869 2.2 2:4e7d172c:::smithi20457400-15:head [write 437015~721812] snapc 21=[21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 166.773613506, 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053129060000000001, 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:26.485 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 869 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708617+0000", 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.708620+0000", 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0369999999999999e-06 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709599+0000", 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00097878799999999997 2024-09-16T05:14:26.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709622+0000", 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2609999999999999e-05 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709633+0000", 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1654999999999999e-05 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709764+0000", 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131023 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.709772+0000", 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6109999999999996e-06 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711031+0000", 2024-09-16T05:14:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001258753 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711097+0000", 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6699000000000006e-05 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.711246+0000", 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000148629 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.712976+0000", 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001729721 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713857+0000", 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00088113600000000003 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713898+0000", 2024-09-16T05:14:26.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0843000000000002e-05 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:39.713930+0000", 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2400999999999997e-05 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1378 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 207] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 123.892714341, 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018067625, 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:26.493 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1378 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589519+0000", 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3460000000000001e-06 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589547+0000", 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8572999999999999e-05 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589577+0000", 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9822e-05 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589592+0000", 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4630000000000001e-05 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589745+0000", 2024-09-16T05:14:26.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015316700000000001 2024-09-16T05:14:26.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589921+0000", 2024-09-16T05:14:26.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000175995 2024-09-16T05:14:26.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:14:26.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.602330+0000", 2024-09-16T05:14:26.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012408878 2024-09-16T05:14:26.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605723+0000", 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033928000000000001 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607502+0000", 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017799140000000001 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607546+0000", 2024-09-16T05:14:26.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3130999999999998e-05 2024-09-16T05:14:26.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.500 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607584+0000", 2024-09-16T05:14:26.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8368999999999998e-05 2024-09-16T05:14:26.500 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.500 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.500 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1379 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 123.892607537, 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015638332000000001, 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.501 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.502 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1379 2024-09-16T05:14:26.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.502 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:14:26.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589624+0000", 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4e-07 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589629+0000", 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.366e-06 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589642+0000", 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2114e-05 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589660+0000", 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8039000000000001e-05 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.590086+0000", 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042614600000000002 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.590119+0000", 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3395999999999997e-05 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605083+0000", 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014963611999999999 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605167+0000", 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4444999999999995e-05 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605261+0000", 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4273999999999999e-05 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1406 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 0] snapc 39=[39,38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-16T05:14:26.510 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.826720+0000", 2024-09-16T05:14:26.511 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 123.655510724, 2024-09-16T05:14:26.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051871499999999997, 2024-09-16T05:14:26.511 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.511 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.511 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.511 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.511 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.511 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1406 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826720+0000", 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826720+0000", 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826721+0000", 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3289999999999999e-06 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826727+0000", 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9490000000000001e-06 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826741+0000", 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3592e-05 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826777+0000", 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6019000000000002e-05 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826831+0000", 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4574000000000003e-05 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826895+0000", 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3492999999999997e-05 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.831907+0000", 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050121940000000002 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1407 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e211)", 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.826794+0000", 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 123.655436808, 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052833339999999998, 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1407 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.516 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826794+0000", 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826794+0000", 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826794+0000", 2024-09-16T05:14:26.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3599999999999999e-07 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826796+0000", 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8840000000000001e-06 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826802+0000", 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0159999999999997e-06 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.826806+0000", 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6569999999999999e-06 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.827022+0000", 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021612899999999999 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.827072+0000", 2024-09-16T05:14:26.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9971e-05 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.831969+0000", 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048975010000000003 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.832017+0000", 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7790000000000002e-05 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.832077+0000", 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0050000000000003e-05 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1858 2.16 2:6be67188:::smithi20457400-46:head [delete] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e249)", 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:00.671909+0000", 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 85.810321943999995, 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010133186000000001, 2024-09-16T05:14:26.521 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1858 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671909+0000", 2024-09-16T05:14:26.523 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671909+0000", 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671911+0000", 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9700000000000002e-06 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671929+0000", 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7847e-05 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671948+0000", 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9117999999999999e-05 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.525 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.671957+0000", 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4539999999999996e-06 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.672037+0000", 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9628000000000004e-05 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:14:26.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.672046+0000", 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6150000000000003e-06 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.679299+0000", 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072529539999999998 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.679400+0000", 2024-09-16T05:14:26.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100228 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682042+0000", 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002642372 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.528 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2002 2.3 2:c018a787:::smithi20457400-18:49 [read 0~3479016,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:19.884154+0000", 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 66.598077024000006, 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006940323, 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2002 2024-09-16T05:14:26.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884154+0000", 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884154+0000", 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884156+0000", 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.052e-06 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884164+0000", 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.551e-06 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884181+0000", 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7056999999999999e-05 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9709999999999993e-06 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884284+0000", 2024-09-16T05:14:26.533 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.357e-05 2024-09-16T05:14:26.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.533 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.533 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:14:26.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884300+0000", 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6135000000000001e-05 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.885492+0000", 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001191627 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.885600+0000", 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108189 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.891094+0000", 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005494171 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.535 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2003 2.3 2:c018a787:::smithi20457400-18:49 [read 0~3479016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:19.884214+0000", 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 66.598016744000006, 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080284589999999999, 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.536 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2003 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884214+0000", 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884214+0000", 2024-09-16T05:14:26.537 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884214+0000", 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7599999999999997e-07 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884218+0000", 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.647e-06 2024-09-16T05:14:26.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884228+0000", 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8199999999999992e-06 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884234+0000", 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8340000000000004e-06 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884365+0000", 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013159599999999999 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884371+0000", 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1610000000000002e-06 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.540 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.891142+0000", 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067718099999999996 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.891266+0000", 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012355600000000001 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.541 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.892242+0000", 2024-09-16T05:14:26.542 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00097645900000000005 2024-09-16T05:14:26.542 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.542 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.542 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.542 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2004 2.3 2:c018a787:::smithi20457400-18:49 [sparse-read 0~2490368] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:19.884274+0000", 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 66.597956404000001, 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083080670000000006, 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2004 2024-09-16T05:14:26.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.544 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.544 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.544 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884274+0000", 2024-09-16T05:14:26.544 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.544 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.544 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884274+0000", 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884275+0000", 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9599999999999998e-07 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884280+0000", 2024-09-16T05:14:26.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5329999999999996e-06 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884317+0000", 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7212000000000002e-05 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884326+0000", 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8040000000000007e-06 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884398+0000", 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2306999999999997e-05 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884404+0000", 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1959999999999996e-06 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.892282+0000", 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078782020000000008 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.892395+0000", 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112438 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.892582+0000", 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000187479 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(2.34 e278 from shard 3)", 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 55.615253961999997, 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34868665500000001, 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9999999999999997e-07 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866983+0000", 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.84e-06 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215470+0000", 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34848719900000003 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215480+0000", 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2129999999999997e-06 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.551 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215584+0000", 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010428400000000001 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215601+0000", 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7509000000000001e-05 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215663+0000", 2024-09-16T05:14:26.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1909999999999995e-05 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2149 2.34 2:2cffa4d6:::smithi20457400-50:49 [read 0~2201750,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e280)", 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:31.746324+0000", 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 54.735906548000003, 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049135029999999996, 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.553 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2149 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746324+0000", 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746324+0000", 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746325+0000", 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4300000000000002e-07 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746332+0000", 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7259999999999996e-06 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746343+0000", 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1175999999999999e-05 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746349+0000", 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7170000000000002e-06 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746460+0000", 2024-09-16T05:14:26.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110869 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746466+0000", 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9619999999999998e-06 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.749689+0000", 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032222879999999998 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.557 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.749807+0000", 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011849900000000001 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.751238+0000", 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001430523 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2150 2.34 2:2cffa4d6:::smithi20457400-50:49 [read 0~2201750,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e280)", 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:31.746360+0000", 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 54.735870658000003, 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056742579999999997, 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2150 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.559 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746360+0000", 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746360+0000", 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746361+0000", 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5199999999999997e-07 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746364+0000", 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8859999999999997e-06 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746372+0000", 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2949999999999996e-06 2024-09-16T05:14:26.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746377+0000", 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0379999999999999e-06 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746495+0000", 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117978 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.746500+0000", 2024-09-16T05:14:26.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3619999999999997e-06 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.751270+0000", 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047700909999999997 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.751368+0000", 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7534999999999994e-05 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.563 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.752034+0000", 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000666521 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2226 2.3 2:c018a787:::smithi20457400-18:head [read 0~2840416,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e287)", 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:38.781852+0000", 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 47.700378772999997, 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00627022, 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.564 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2226 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781852+0000", 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781852+0000", 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781859+0000", 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.781e-06 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781869+0000", 2024-09-16T05:14:26.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0744000000000001e-05 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781892+0000", 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2177000000000001e-05 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781905+0000", 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3149999999999999e-05 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.567 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781992+0000", 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6769999999999998e-05 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.782001+0000", 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7589999999999992e-06 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.784721+0000", 2024-09-16T05:14:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00271952 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.784806+0000", 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4810999999999998e-05 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.788122+0000", 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033165080000000001 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2227 2.3 2:c018a787:::smithi20457400-18:head [read 0~2840416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e287)", 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:38.781939+0000", 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 47.700291536000002, 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070388769999999998, 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2227 2024-09-16T05:14:26.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781939+0000", 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781939+0000", 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781940+0000", 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3300000000000001e-07 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781945+0000", 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2630000000000003e-06 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781954+0000", 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9740000000000008e-06 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781960+0000", 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8130000000000001e-06 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.782074+0000", 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011372899999999999 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:26.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.782078+0000", 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1799999999999998e-06 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.788161+0000", 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060830499999999996 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.788223+0000", 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1929999999999998e-05 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.788978+0000", 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000755205 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2228 2.3 2:c018a787:::smithi20457400-18:head [sparse-read 0~1928447] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e287)", 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:38.781986+0000", 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 47.700244824000002, 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071795790000000002, 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.575 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2228 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781986+0000", 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781986+0000", 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781986+0000", 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3099999999999998e-07 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781989+0000", 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0989999999999999e-06 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781995+0000", 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9660000000000001e-06 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.781999+0000", 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9559999999999999e-06 2024-09-16T05:14:26.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.782115+0000", 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011559499999999999 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.782119+0000", 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5029999999999999e-06 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.789013+0000", 2024-09-16T05:14:26.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068936620000000001 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.789073+0000", 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9941000000000001e-05 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:38.789165+0000", 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2325999999999997e-05 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.580 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2256 2.34 2:2cffa4d6:::smithi20457400-50:head [copy-from ver 319] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.800573423000003, 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019683254000000001, 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2256 2024-09-16T05:14:26.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681660+0000", 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5019999999999999e-06 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681670+0000", 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.91e-06 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681686+0000", 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6699999999999999e-05 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681695+0000", 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9590000000000001e-06 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681793+0000", 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7699000000000001e-05 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681800+0000", 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1199999999999996e-06 2024-09-16T05:14:26.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.682948+0000", 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011475470000000001 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683035+0000", 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6886999999999997e-05 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:14:26.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.693853+0000", 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010818583 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698768+0000", 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049152340000000001 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701246+0000", 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024771630000000001 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701284+0000", 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8544000000000002e-05 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701340+0000", 2024-09-16T05:14:26.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6406000000000002e-05 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2257 2.34 2:2cffa4d6:::smithi20457400-50:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.800517947000003, 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016511652000000002, 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.589 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2257 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7700000000000005e-07 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681718+0000", 2024-09-16T05:14:26.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4920000000000004e-06 2024-09-16T05:14:26.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681724+0000", 2024-09-16T05:14:26.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9900000000000002e-06 2024-09-16T05:14:26.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681727+0000", 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8179999999999997e-06 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681858+0000", 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130698 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681862+0000", 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0149999999999997e-06 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683176+0000", 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013134119999999999 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683211+0000", 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5091e-05 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698063+0000", 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014852783 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698134+0000", 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0570999999999999e-05 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698224+0000", 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0304999999999997e-05 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2317 2.34 2:2cffa4d6:::smithi20457400-50:head [sparse-read 0~2097152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.688034+0000", 2024-09-16T05:14:26.596 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 38.794196880000001, 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049249879999999999, 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2317 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.688034+0000", 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.688034+0000", 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.688034+0000", 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0999999999999998e-07 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.688038+0000", 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3330000000000001e-06 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.688048+0000", 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.91e-06 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.688055+0000", 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0040000000000002e-06 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.692819+0000", 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047643149999999999 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.692877+0000", 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7805e-05 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.692959+0000", 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2010999999999998e-05 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2376 2.20 e302/211 2:06458cf1:::smithi20457400-6:head v 302'66, mlcod=284'62)", 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:53.708536+0000", 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 32.773694972000001, 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068268479999999999, 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.708536+0000", 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.708536+0000", 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.708536+0000", 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.03e-07 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.711101+0000", 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025649330000000001 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.711168+0000", 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7261999999999994e-05 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.711180+0000", 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2179e-05 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.711259+0000", 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9134999999999994e-05 2024-09-16T05:14:26.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.711294+0000", 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4709000000000003e-05 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.715323+0000", 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040292449999999999 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.715363+0000", 2024-09-16T05:14:26.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9181999999999999e-05 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2511 2.3 2.e1e51803 (undecoded) ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e320)", 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 1.6591586270000001, 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99828526200000001, 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:14:26.606 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2511 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:26.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823073+0000", 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.04e-06 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823078+0000", 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6519999999999997e-06 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823091+0000", 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3277e-05 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823100+0000", 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4909999999999994e-06 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823141+0000", 2024-09-16T05:14:26.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0187e-05 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823147+0000", 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3929999999999996e-06 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821329+0000", 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99818146900000004 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:26.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:26.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821357+0000", 2024-09-16T05:14:26.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8753000000000001e-05 2024-09-16T05:14:26.611 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.611 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.611 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.611 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:26.611 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:26.611 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:14:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:26.612 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:26.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:26.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:26.740 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:26.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:26.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:26.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:26.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:26.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:14:26.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:26.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:26.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:27.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:27.116 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:27.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:14:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:14:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:14:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:14:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]: dispatch 2024-09-16T05:14:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:14:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:14:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:26 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]: dispatch 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:14:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:14:27.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:26 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:14:27.213 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:27.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:27.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:14:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:14:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:14:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:14:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]: dispatch 2024-09-16T05:14:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:14:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:14:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:14:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:26 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:14:27.243 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:27.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:27.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:14:27.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:27.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:27.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:27.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:27.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:27.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:27.584 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:27.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:27.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:27.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:27.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:27.740 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:27.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:27.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:27.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:27.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:27.868 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: pgmap v641: 87 pgs: 1 active+clean+remapped, 5 peering, 8 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 65 active+clean; 315 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/294 objects degraded (3.401%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 10/294 objects degraded (3.401%), 5 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]': finished 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: osdmap e322: 12 total, 11 up, 12 in 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.29 scrub ok 2024-09-16T05:14:27.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:14:27.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:14:27.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:14:27.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:14:27.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4140158717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:27.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:27.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: pgmap v641: 87 pgs: 1 active+clean+remapped, 5 peering, 8 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 65 active+clean; 315 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/294 objects degraded (3.401%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:27.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:14:27.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:14:27.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:14:27.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:14:27.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 10/294 objects degraded (3.401%), 5 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:27.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]': finished 2024-09-16T05:14:27.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: osdmap e322: 12 total, 11 up, 12 in 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.29 scrub ok 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:14:27.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4140158717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:27.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:27.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:28.069 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:14:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:14:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:14:28.069 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:14:28.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:28.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:28.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:28.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: pgmap v641: 87 pgs: 1 active+clean+remapped, 5 peering, 8 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 65 active+clean; 315 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/294 objects degraded (3.401%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 10/294 objects degraded (3.401%), 5 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "76"}]': finished 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: osdmap e322: 12 total, 11 up, 12 in 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:14:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:14:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.29 scrub ok 2024-09-16T05:14:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:14:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:14:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:14:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:14:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4140158717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:28.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:28.247 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:28.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:28.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:28.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:28.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:28.453 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T05:14:28.453 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-16T05:14:28.453 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-09-16T05:14:28.454 DEBUG:teuthology.orchestra.run.smithi032:osd.2> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.2.service 2024-09-16T05:14:28.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl start ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.2 2024-09-16T05:14:28.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:28.479 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:28.479 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:28.504 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:28.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:28.505 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:28.505 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:28.505 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:14:28.528 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:28.550 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:28.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:28.572 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:28.594 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:28.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:28.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:14:28.617 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:28.617 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:28.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:28.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:28.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:14:28.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:28.737 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:28.737 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:28 smithi032 systemd[1]: Starting Ceph osd.2 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:14:28.757 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:28.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:28.758 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:28.759 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:28.783 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:28.783 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:28.805 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:28.805 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:28.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:28.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:28.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:28.919 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:28.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:28.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:14:28.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 2.26 scrub starts 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 2.26 scrub ok 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: osdmap e323: 12 total, 11 up, 12 in 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:14:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:28 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:14:29.010 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:29.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:29.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:14:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:14:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 2.26 scrub starts 2024-09-16T05:14:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 2.26 scrub ok 2024-09-16T05:14:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: osdmap e323: 12 total, 11 up, 12 in 2024-09-16T05:14:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:14:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:14:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:14:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:14:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:14:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:28 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:14:29.171 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:28 smithi032 podman[101549]: 2024-09-16 05:14:28.650093554 +0000 UTC m=+0.016905632 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:14:29.171 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:28 smithi032 podman[101549]: 2024-09-16 05:14:28.773796911 +0000 UTC m=+0.140608957 container create a786c81670e21ef87ea1f8b5a5091b79d89aa0137768015401357fae4143c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 2.26 scrub starts 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 2.26 scrub ok 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: osdmap e323: 12 total, 11 up, 12 in 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:14:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:14:29.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:28 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:14:29.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:29.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:29.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:29.383 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:29.383 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:29.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:29.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:29.523 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:14:29.586 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:14:29.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:29.646 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:29.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:29.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:29.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:29.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:29.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:29.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:14:29.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:30.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:30.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:30.043 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:30.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:30.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:30.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:30.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: pgmap v644: 87 pgs: 1 active+clean+remapped, 5 peering, 1 remapped+peering, 6 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 66 active+clean; 318 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:30.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:14:30.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:14:30.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: 2.43 scrub starts 2024-09-16T05:14:30.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: 2.43 scrub ok 2024-09-16T05:14:30.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: osdmap e324: 12 total, 11 up, 12 in 2024-09-16T05:14:30.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:14:30.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:14:30.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:14:30.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:14:30.168 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:29 smithi032 podman[101549]: 2024-09-16 05:14:29.826554003 +0000 UTC m=+1.193366048 container init a786c81670e21ef87ea1f8b5a5091b79d89aa0137768015401357fae4143c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-09-16T05:14:30.169 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:29 smithi032 podman[101549]: 2024-09-16 05:14:29.830532366 +0000 UTC m=+1.197344420 container start a786c81670e21ef87ea1f8b5a5091b79d89aa0137768015401357fae4143c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, ceph=True) 2024-09-16T05:14:30.169 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:29 smithi032 podman[101549]: 2024-09-16 05:14:29.917314154 +0000 UTC m=+1.284126209 container attach a786c81670e21ef87ea1f8b5a5091b79d89aa0137768015401357fae4143c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-16T05:14:30.169 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:14:30.169 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:29 smithi032 bash[101549]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:14:30.169 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:29 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:14:30.169 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:29 smithi032 bash[101549]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:14:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: pgmap v644: 87 pgs: 1 active+clean+remapped, 5 peering, 1 remapped+peering, 6 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 66 active+clean; 318 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:14:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:14:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: 2.43 scrub starts 2024-09-16T05:14:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: 2.43 scrub ok 2024-09-16T05:14:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: osdmap e324: 12 total, 11 up, 12 in 2024-09-16T05:14:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:14:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:14:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:14:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:29 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:14:30.207 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: pgmap v644: 87 pgs: 1 active+clean+remapped, 5 peering, 1 remapped+peering, 6 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 66 active+clean; 318 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: 2.43 scrub starts 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: 2.43 scrub ok 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: osdmap e324: 12 total, 11 up, 12 in 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:14:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:14:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:29 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:14:30.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:30.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:30.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:30.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:30.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:30.400 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:30.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:30.502 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:30.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:30.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:30.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:30.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:30.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:30.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:30.718 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:30.718 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:30.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:30.768 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T05:14:30.768 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 bash[101549]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T05:14:30.768 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-16T05:14:30.768 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 bash[101549]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-16T05:14:30.768 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-16T05:14:30.768 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 bash[101549]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-16T05:14:30.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:30.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:30.849 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:14:30.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:30.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:30.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:30 smithi196 ceph-mon[30802]: osdmap e325: 12 total, 11 up, 12 in 2024-09-16T05:14:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:30 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:14:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:30 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:14:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:30 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:14:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:30 smithi196 ceph-mon[30802]: 2.44 scrub ok 2024-09-16T05:14:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:30 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:30.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1495042388' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:31.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:31.027 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:31.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:31.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-mon[27388]: osdmap e325: 12 total, 11 up, 12 in 2024-09-16T05:14:31.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:14:31.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:14:31.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:14:31.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-mon[27388]: 2.44 scrub ok 2024-09-16T05:14:31.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:31.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1495042388' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:31.041 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-16T05:14:31.041 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 bash[101549]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-16T05:14:31.041 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-16T05:14:31.041 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 bash[101549]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-16T05:14:31.041 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T05:14:31.042 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 bash[101549]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T05:14:31.042 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate[101633]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-16T05:14:31.042 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 bash[101549]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-16T05:14:31.042 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:30 smithi032 podman[101549]: 2024-09-16 05:14:30.79561377 +0000 UTC m=+2.162425816 container died a786c81670e21ef87ea1f8b5a5091b79d89aa0137768015401357fae4143c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-16T05:14:31.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:31.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:31.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:30 smithi204 ceph-mon[30933]: osdmap e325: 12 total, 11 up, 12 in 2024-09-16T05:14:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:30 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:14:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:30 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:14:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:30 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:14:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:30 smithi204 ceph-mon[30933]: 2.44 scrub ok 2024-09-16T05:14:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:30 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1495042388' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:31.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:31.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:31.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:31.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:31.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:14:31.405 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:31.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:31.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:31.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:31.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:31.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:14:31.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:31.636 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:31.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:31.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:31.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:31.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:31.871 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: pgmap v647: 87 pgs: 1 active+clean+remapped, 5 peering, 1 activating+remapped, 1 remapped+peering, 5 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 66 active+clean; 318 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:31.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:14:31.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:14:31.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:14:31.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:14:31.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:31.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1495042388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:31.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: osdmap e326: 12 total, 11 up, 12 in 2024-09-16T05:14:31.872 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.45 scrub starts 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.45 scrub ok 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:14:31.873 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:31 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:14:31.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:31.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:31.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:32.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:32.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: pgmap v647: 87 pgs: 1 active+clean+remapped, 5 peering, 1 activating+remapped, 1 remapped+peering, 5 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 66 active+clean; 318 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:32.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1495042388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: osdmap e326: 12 total, 11 up, 12 in 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.45 scrub starts 2024-09-16T05:14:32.033 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.45 scrub ok 2024-09-16T05:14:32.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:14:32.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:14:32.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:14:32.034 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:31 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:14:32.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:32.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:32.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:32.140 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:32.140 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:32.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: pgmap v647: 87 pgs: 1 active+clean+remapped, 5 peering, 1 activating+remapped, 1 remapped+peering, 5 unknown, 3 undersized+peered, 5 undersized+degraded+peered, 66 active+clean; 318 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 10/298 objects degraded (3.356%); 2/298 objects misplaced (0.671%) 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1495042388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: osdmap e326: 12 total, 11 up, 12 in 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:14:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.45 scrub starts 2024-09-16T05:14:32.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.45 scrub ok 2024-09-16T05:14:32.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:14:32.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:14:32.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:14:32.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:31 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:14:32.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:32.277 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:32.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:32.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:32.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:32.389 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:32.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:32.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:32.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:32 smithi032 podman[101876]: 2024-09-16 05:14:32.16270539 +0000 UTC m=+1.360208129 container remove a786c81670e21ef87ea1f8b5a5091b79d89aa0137768015401357fae4143c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-activate, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-16T05:14:32.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:32 smithi032 podman[102034]: 2024-09-16 05:14:32.281100453 +0000 UTC m=+0.018630534 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:14:32.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:32 smithi032 podman[102034]: 2024-09-16 05:14:32.380101264 +0000 UTC m=+0.117631388 container create 7b309f5042a491e074273c38642ff74261fd1db2537574e78f165e8a111104ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:14:32.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:32.540 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:32.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:32.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:32.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:32.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:32.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:32.791 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:32.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:32.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:14:32.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:32.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:14:32.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:32.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:32.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:33.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:33.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:33.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:33.056 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:33.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:32 smithi032 ceph-mon[27388]: 2.24 deep-scrub starts 2024-09-16T05:14:33.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:32 smithi032 ceph-mon[27388]: 2.24 deep-scrub ok 2024-09-16T05:14:33.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:32 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:14:33.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:32 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:14:33.105 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:32 smithi032 podman[102034]: 2024-09-16 05:14:32.793305576 +0000 UTC m=+0.530835656 container init 7b309f5042a491e074273c38642ff74261fd1db2537574e78f165e8a111104ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:14:33.105 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:32 smithi032 podman[102034]: 2024-09-16 05:14:32.797009183 +0000 UTC m=+0.534539263 container start 7b309f5042a491e074273c38642ff74261fd1db2537574e78f165e8a111104ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True) 2024-09-16T05:14:33.105 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:32 smithi032 ceph-osd[102111]: -- 172.21.15.32:0/3702814379 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564593ff5a40 con 0x564593fd3c00 2024-09-16T05:14:33.105 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:32 smithi032 bash[102034]: 7b309f5042a491e074273c38642ff74261fd1db2537574e78f165e8a111104ec 2024-09-16T05:14:33.106 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:32 smithi032 systemd[1]: Started Ceph osd.2 for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:14:33.128 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:33.129 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:33.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:33.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:14:33.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:33.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:32 smithi204 ceph-mon[30933]: 2.24 deep-scrub starts 2024-09-16T05:14:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:32 smithi204 ceph-mon[30933]: 2.24 deep-scrub ok 2024-09-16T05:14:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:32 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:14:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:32 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:14:33.173 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:33.174 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:33.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:33.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:32 smithi196 ceph-mon[30802]: 2.24 deep-scrub starts 2024-09-16T05:14:33.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:32 smithi196 ceph-mon[30802]: 2.24 deep-scrub ok 2024-09-16T05:14:33.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:32 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:14:33.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:32 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:14:33.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:33.196 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:33.218 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:33.218 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:33.241 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:33.241 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:33.262 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:33.262 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:33.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:33.284 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:33.284 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:33.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:33.305 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:33.305 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:33.327 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:33.328 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:33.349 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:33.349 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:33.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:33.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:33.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:33.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:33.546 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:14:33.546 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:14:33.546 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:14:33.546 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:14:33.546 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:14:33.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:33.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:33.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:33.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:33.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:33.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:14:33.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:33.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:33.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:33.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:33.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:34.051 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:33 smithi032 ceph-mon[27388]: pgmap v649: 87 pgs: 2 peering, 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 77 active+clean; 323 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 0 op/s; 8/300 objects degraded (2.667%); 2/300 objects misplaced (0.667%); 703 KiB/s, 1 objects/s recovering 2024-09-16T05:14:34.051 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:33 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 8/300 objects degraded (2.667%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:34.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2276553627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:34.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:14:34.080 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:14:34.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:34.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:34.160 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:33 smithi196 ceph-mon[30802]: pgmap v649: 87 pgs: 2 peering, 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 77 active+clean; 323 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 0 op/s; 8/300 objects degraded (2.667%); 2/300 objects misplaced (0.667%); 703 KiB/s, 1 objects/s recovering 2024-09-16T05:14:34.160 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:33 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 8/300 objects degraded (2.667%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:34.160 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2276553627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:33 smithi204 ceph-mon[30933]: pgmap v649: 87 pgs: 2 peering, 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 77 active+clean; 323 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 0 op/s; 8/300 objects degraded (2.667%); 2/300 objects misplaced (0.667%); 703 KiB/s, 1 objects/s recovering 2024-09-16T05:14:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:33 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 8/300 objects degraded (2.667%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2276553627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:34.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:34.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:14:34.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:34.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:no valid command found; 10 closest matches: 2024-09-16T05:14:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:0 2024-09-16T05:14:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:1 2024-09-16T05:14:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:2 2024-09-16T05:14:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:abort 2024-09-16T05:14:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:assert 2024-09-16T05:14:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:config diff 2024-09-16T05:14:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:config diff get 2024-09-16T05:14:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:config get 2024-09-16T05:14:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:config help [] 2024-09-16T05:14:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:config set ... 2024-09-16T05:14:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:admin_socket: invalid command 2024-09-16T05:14:34.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:34.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:34.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:34.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:14:34.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:34.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:34.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:34.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:34.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:34.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:34.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:34.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:34.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:34.839 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:34 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[102067]: 2024-09-16T05:14:34.683+0000 7f19c17a3640 -1 Falling back to public interface 2024-09-16T05:14:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:34.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T05:14:34.867 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-16T05:14:34.880 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:14:34.880 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:14:34.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:14:34.881 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:14:34.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.881 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:23 2.0 2:06458cf1:::smithi20457400-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:14:34.881 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:14:34.882 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 231.82540834900001, 2024-09-16T05:14:34.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103108687, 2024-09-16T05:14:34.882 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.882 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.882 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:34.883 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.883 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.883 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.883 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 23 2024-09-16T05:14:34.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.883 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:14:34.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:14:34.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:14:34.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3599999999999999e-07 2024-09-16T05:14:34.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052188+0000", 2024-09-16T05:14:34.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4169999999999999e-06 2024-09-16T05:14:34.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052214+0000", 2024-09-16T05:14:34.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5327e-05 2024-09-16T05:14:34.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052217+0000", 2024-09-16T05:14:34.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1810000000000001e-06 2024-09-16T05:14:34.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052730+0000", 2024-09-16T05:14:34.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051270700000000005 2024-09-16T05:14:34.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052818+0000", 2024-09-16T05:14:34.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8641999999999996e-05 2024-09-16T05:14:34.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:14:34.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052937+0000", 2024-09-16T05:14:34.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119009 2024-09-16T05:14:34.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:34.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053259+0000", 2024-09-16T05:14:34.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032151800000000002 2024-09-16T05:14:34.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:34.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155207+0000", 2024-09-16T05:14:34.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101948155 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155248+0000", 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1115999999999997e-05 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155294+0000", 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6179e-05 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:24 2.0 2:06458cf1:::smithi20457400-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:14:34.893 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 231.82537228800001, 2024-09-16T05:14:34.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103238884, 2024-09-16T05:14:34.894 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.894 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.894 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:34.894 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.894 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.895 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.895 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 24 2024-09-16T05:14:34.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.895 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:14:34.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:14:34.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:14:34.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8799999999999998e-07 2024-09-16T05:14:34.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052223+0000", 2024-09-16T05:14:34.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2950000000000001e-06 2024-09-16T05:14:34.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052227+0000", 2024-09-16T05:14:34.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5240000000000001e-06 2024-09-16T05:14:34.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052230+0000", 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2160000000000002e-06 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053127+0000", 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089734199999999997 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053182+0000", 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4935999999999997e-05 2024-09-16T05:14:34.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155331+0000", 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.102149077 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155389+0000", 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7691e-05 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155460+0000", 2024-09-16T05:14:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1514999999999995e-05 2024-09-16T05:14:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.903 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.904 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:391 2.8 2:1030fb5b:::smithi20457400-27:head [copy-from ver 13] snapc 8=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:14:34.904 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:14:34.904 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 219.24080696799999, 2024-09-16T05:14:34.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021838053999999999, 2024-09-16T05:14:34.904 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.905 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.905 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:34.905 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.905 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.905 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.905 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 391 2024-09-16T05:14:34.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.906 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:14:34.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:14:34.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636789+0000", 2024-09-16T05:14:34.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3829999999999999e-06 2024-09-16T05:14:34.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636801+0000", 2024-09-16T05:14:34.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1459e-05 2024-09-16T05:14:34.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636822+0000", 2024-09-16T05:14:34.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1719e-05 2024-09-16T05:14:34.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636833+0000", 2024-09-16T05:14:34.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0366e-05 2024-09-16T05:14:34.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636912+0000", 2024-09-16T05:14:34.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9060999999999994e-05 2024-09-16T05:14:34.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:14:34.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636918+0000", 2024-09-16T05:14:34.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0900000000000001e-06 2024-09-16T05:14:34.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638224+0000", 2024-09-16T05:14:34.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00130613 2024-09-16T05:14:34.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638314+0000", 2024-09-16T05:14:34.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9771000000000005e-05 2024-09-16T05:14:34.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:14:34.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.652220+0000", 2024-09-16T05:14:34.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013906112 2024-09-16T05:14:34.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:34.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.656458+0000", 2024-09-16T05:14:34.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004237673 2024-09-16T05:14:34.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:34.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658553+0000", 2024-09-16T05:14:34.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002095609 2024-09-16T05:14:34.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:34.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658592+0000", 2024-09-16T05:14:34.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8627000000000001e-05 2024-09-16T05:14:34.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658625+0000", 2024-09-16T05:14:34.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3053999999999999e-05 2024-09-16T05:14:34.919 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.919 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.919 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.919 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:392 2.8 2:1030fb5b:::smithi20457400-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:14:34.919 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:14:34.920 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 219.24073574600001, 2024-09-16T05:14:34.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019221338000000001, 2024-09-16T05:14:34.920 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.920 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.920 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 392 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636859+0000", 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0699999999999998e-07 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636861+0000", 2024-09-16T05:14:34.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5129999999999999e-06 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636867+0000", 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6030000000000004e-06 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636870+0000", 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.551e-06 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636972+0000", 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010187999999999999 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636978+0000", 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6269999999999997e-06 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638477+0000", 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014995049999999999 2024-09-16T05:14:34.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638531+0000", 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3387999999999998e-05 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.655845+0000", 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017313963000000002 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.655956+0000", 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011097799999999999 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.656079+0000", 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012392300000000001 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1675 2.1c 2:3d4bedc8:::smithi20457400-22:head [setxattr _header (57) in=7b,truncate 2151329] snapc 46=[46,45,44,42,41,40,3f,3e,3c,3a,39,37,35,33,27] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-16T05:14:34.926 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 108.017974591, 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22736652099999999, 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1675 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859620+0000", 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3099999999999997e-07 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859624+0000", 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1949999999999996e-06 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859637+0000", 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3118999999999999e-05 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859641+0000", 2024-09-16T05:14:34.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.036e-06 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860356+0000", 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00071436999999999998 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860417+0000", 2024-09-16T05:14:34.931 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:34.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1253999999999993e-05 2024-09-16T05:14:34.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 8", 2024-09-16T05:14:34.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860525+0000", 2024-09-16T05:14:34.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107742 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860988+0000", 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046300399999999999 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086882+0000", 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.225894659 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086930+0000", 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8010999999999998e-05 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086986+0000", 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5500000000000001e-05 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1676 2.1c 2:3d4bedc8:::smithi20457400-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e235)", 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 108.017931089, 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.227504338, 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1676 2024-09-16T05:14:34.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9999999999999998e-07 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859666+0000", 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7819999999999998e-06 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859699+0000", 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2950000000000001e-05 2024-09-16T05:14:34.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859707+0000", 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3459999999999994e-06 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860761+0000", 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001053315 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:14:34.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860862+0000", 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010108 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087019+0000", 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.226157467 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087091+0000", 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1536000000000001e-05 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087167+0000", 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6361999999999998e-05 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:1675 2.1c e235/232 ondisk, result = 0)", 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 108.01657544699999, 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22597392799999999, 2024-09-16T05:14:34.940 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861019+0000", 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861020+0000", 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7969999999999999e-06 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086793+0000", 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22577269899999999 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086803+0000", 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5459999999999997e-06 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086867+0000", 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4447000000000001e-05 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086876+0000", 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9760000000000001e-06 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086992+0000", 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116212 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1824 2.8 2:1030fb5b:::smithi20457400-27:head [write 521083~714336,stat] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 95.932446478000003, 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.81065307900000005, 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1824 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.945 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945150+0000", 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1260000000000001e-06 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945906+0000", 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075638699999999997 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945927+0000", 2024-09-16T05:14:34.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0849e-05 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945936+0000", 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3910000000000004e-06 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946023+0000", 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6547999999999999e-05 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946035+0000", 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2249000000000001e-05 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752337+0000", 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80630184599999999 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752493+0000", 2024-09-16T05:14:34.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000156546 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752675+0000", 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018172699999999999 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755628+0000", 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029533929999999999 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755689+0000", 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0949000000000002e-05 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755745+0000", 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5591999999999999e-05 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755800+0000", 2024-09-16T05:14:34.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5476000000000002e-05 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1825 2.8 2:1030fb5b:::smithi20457400-27:head [write 1724715~555421] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 95.926800929999999, 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80511571699999995, 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.952 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1825 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950794+0000", 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951237+0000", 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00044250000000000002 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951251+0000", 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4381e-05 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951276+0000", 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5233999999999999e-05 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951378+0000", 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010173799999999999 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951390+0000", 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2274e-05 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.753903+0000", 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80251229300000004 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754002+0000", 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9427999999999997e-05 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754132+0000", 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130207 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755633+0000", 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015010100000000001 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755844+0000", 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021044799999999999 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755880+0000", 2024-09-16T05:14:34.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6007999999999998e-05 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755909+0000", 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8941e-05 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1826 2.8 2:1030fb5b:::smithi20457400-27:head [setxattr _header (57) in=7b,truncate 2521083] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:14:34.959 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 95.926301636999995, 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80476365000000005, 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1826 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951293+0000", 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951296+0000", 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2799999999999999e-06 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951301+0000", 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6879999999999998e-06 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951309+0000", 2024-09-16T05:14:34.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7810000000000005e-06 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951421+0000", 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112333 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951426+0000", 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6650000000000002e-06 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754913+0000", 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80348719099999999 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754982+0000", 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9558e-05 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755084+0000", 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101894 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755638+0000", 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055423099999999997 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755988+0000", 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034971600000000002 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756021+0000", 2024-09-16T05:14:34.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2461e-05 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756056+0000", 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5278999999999999e-05 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1827 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:14:34.966 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 95.926279367999996, 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.804892304, 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1827 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8700000000000002e-07 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951317+0000", 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7460000000000001e-06 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951321+0000", 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3250000000000001e-06 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951325+0000", 2024-09-16T05:14:34.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1710000000000004e-06 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951464+0000", 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139111 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951475+0000", 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1045e-05 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755294+0000", 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80381846999999995 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755351+0000", 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7589e-05 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756086+0000", 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073505599999999999 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756138+0000", 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1397000000000003e-05 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756207+0000", 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9107000000000001e-05 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1828 2.1c 2:3edd0db8:::smithi20457400-24:head [copy-from ver 42] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951361+0000", 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 95.926233019999998, 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015678850000000001, 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1828 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.973 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951361+0000", 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951361+0000", 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951361+0000", 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4299999999999998e-07 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951365+0000", 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8700000000000002e-06 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951370+0000", 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2669999999999997e-06 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951375+0000", 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3579999999999996e-06 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951458+0000", 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3350999999999995e-05 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951545+0000", 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7188000000000006e-05 2024-09-16T05:14:34.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 8", 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.962074+0000", 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010528239999999999 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.965897+0000", 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038237610000000002 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:34.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.966921+0000", 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001023803 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.966986+0000", 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4380000000000004e-05 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.967040+0000", 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4188999999999999e-05 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.978 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1829 2.1c 2:3edd0db8:::smithi20457400-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951385+0000", 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 95.926209211, 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014002542999999999, 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.979 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1829 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951385+0000", 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951385+0000", 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951385+0000", 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6399999999999998e-07 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951389+0000", 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9979999999999997e-06 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951394+0000", 2024-09-16T05:14:34.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.741e-06 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951398+0000", 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951701+0000", 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030310999999999999 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951768+0000", 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6789e-05 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.965171+0000", 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013403184 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.965273+0000", 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102097 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.965387+0000", 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011381 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1834 2.8 2:12e92f6c:::smithi20457400-1:head [stat,delete] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 95.679378686000007, 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55807496000000001, 2024-09-16T05:14:34.984 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1834 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198218+0000", 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.357e-06 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198229+0000", 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1677e-05 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198254+0000", 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4312e-05 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198266+0000", 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2687000000000001e-05 2024-09-16T05:14:34.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198366+0000", 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9698000000000007e-05 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198378+0000", 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2106000000000001e-05 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755379+0000", 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55700064400000004 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755448+0000", 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8848000000000002e-05 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756290+0000", 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084263100000000004 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.9,QREQUEST,e:256,from:1,reservation_nonce:2)", 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 86.577357250999995, 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21542001499999999, 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300239+0000", 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0540000000000002e-06 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300249+0000", 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0369e-05 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300304+0000", 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5247000000000001e-05 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300312+0000", 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.089e-06 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300391+0000", 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8628999999999995e-05 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300397+0000", 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8889999999999999e-06 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515506+0000", 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21510901199999999 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515527+0000", 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0981000000000001e-05 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515657+0000", 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012974500000000001 2024-09-16T05:14:34.994 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2255 2.1d e290/282 2:ba754fcb:::smithi20457400-30:head v 290'477, mlcod=284'471)", 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 53.192783939000002, 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53570759499999998, 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684811+0000", 2024-09-16T05:14:34.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3600000000000002e-07 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684819+0000", 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5760000000000006e-06 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219817+0000", 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53499761300000004 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219843+0000", 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5908e-05 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219921+0000", 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7811000000000004e-05 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219944+0000", 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3237000000000001e-05 2024-09-16T05:14:34.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220484+0000", 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00053966300000000001 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220518+0000", 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3850999999999998e-05 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:34.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.1d,QREQUEST,e:290,from:10,reservation_nonce:2)", 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 52.657804900000002, 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.202128169, 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:35.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219790+0000", 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.452e-06 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:35.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219798+0000", 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1670000000000002e-06 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421791+0000", 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20199293700000001 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421799+0000", 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5560000000000008e-06 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421864+0000", 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5235999999999997e-05 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421881+0000", 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6742e-05 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421917+0000", 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6078999999999998e-05 2024-09-16T05:14:35.003 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.10.0:85 2.1d e300/291 2:ba754fcb:::smithi20457400-30:5f v 300'498, mlcod=300'494)", 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 43.118957295999998, 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.51609034499999995, 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:35.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:14:35.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.16e-07 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758644+0000", 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1519999999999997e-06 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273721+0000", 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.51507724799999999 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273736+0000", 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5449000000000001e-05 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273906+0000", 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016944 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273962+0000", 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6209999999999999e-05 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.274683+0000", 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072146099999999996 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.274727+0000", 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3569000000000002e-05 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2376 2.20 2:06458cf1:::smithi20457400-6:head [copy-from ver 323] snapc 65=[65,64,63,62,60,5d,5a,58,57,52,4d,49,45] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:53.697483+0000", 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 41.180110706999997, 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018188038, 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2376 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.009 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697483+0000", 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697483+0000", 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:35.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697485+0000", 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8390000000000001e-06 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697492+0000", 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7290000000000003e-06 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697507+0000", 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5071e-05 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697533+0000", 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5897999999999998e-05 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697604+0000", 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1120999999999999e-05 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697613+0000", 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0990000000000005e-06 2024-09-16T05:14:35.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.701882+0000", 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042690319999999999 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.701998+0000", 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011616 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 5", 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.708360+0000", 2024-09-16T05:14:35.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063616150000000002 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.713126+0000", 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00476648 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.715547+0000", 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024211810000000001 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.715614+0000", 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6328999999999995e-05 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.715671+0000", 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7484000000000001e-05 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.015 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2377 2.20 2:06458cf1:::smithi20457400-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e302)", 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:53.697551+0000", 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 41.180043015999999, 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015069702000000001, 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2377 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697551+0000", 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697551+0000", 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697551+0000", 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7999999999999996e-07 2024-09-16T05:14:35.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697553+0000", 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9740000000000001e-06 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697558+0000", 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.135e-06 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:14:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697562+0000", 2024-09-16T05:14:35.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4460000000000001e-06 2024-09-16T05:14:35.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:35.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697667+0000", 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105157 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.697671+0000", 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2549999999999999e-06 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.702158+0000", 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044867029999999999 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.702189+0000", 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1232999999999997e-05 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.712462+0000", 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010272695 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.712537+0000", 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5187999999999999e-05 2024-09-16T05:14:35.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:53.712621+0000", 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3436000000000005e-05 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:14:35.023 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:14:35.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:35.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:35.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:35.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:34 smithi032 ceph-mon[27388]: 2.2a scrub starts 2024-09-16T05:14:35.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:34 smithi032 ceph-mon[27388]: 2.2a scrub ok 2024-09-16T05:14:35.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:34 smithi032 ceph-mon[27388]: 2.35 scrub starts 2024-09-16T05:14:35.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:34 smithi032 ceph-mon[27388]: 2.35 scrub ok 2024-09-16T05:14:35.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:35.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:35.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:34 smithi204 ceph-mon[30933]: 2.2a scrub starts 2024-09-16T05:14:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:34 smithi204 ceph-mon[30933]: 2.2a scrub ok 2024-09-16T05:14:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:34 smithi204 ceph-mon[30933]: 2.35 scrub starts 2024-09-16T05:14:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:34 smithi204 ceph-mon[30933]: 2.35 scrub ok 2024-09-16T05:14:35.193 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:34 smithi196 ceph-mon[30802]: 2.2a scrub starts 2024-09-16T05:14:35.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:34 smithi196 ceph-mon[30802]: 2.2a scrub ok 2024-09-16T05:14:35.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:34 smithi196 ceph-mon[30802]: 2.35 scrub starts 2024-09-16T05:14:35.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:34 smithi196 ceph-mon[30802]: 2.35 scrub ok 2024-09-16T05:14:35.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:35.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:35.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:35.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:35.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:35.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:35.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:35.551 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:35.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:35.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:35.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:35.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:35.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:35.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:14:35.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:35.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:35.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:35.861 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:14:35.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:35.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:35.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:35.958 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:35.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:35 smithi196 ceph-mon[30802]: pgmap v650: 87 pgs: 2 peering, 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 77 active+clean; 323 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8/300 objects degraded (2.667%); 2/300 objects misplaced (0.667%); 498 KiB/s, 1 objects/s recovering 2024-09-16T05:14:35.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4075006611' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:36.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:36.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:36.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:36.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:35 smithi204 ceph-mon[30933]: pgmap v650: 87 pgs: 2 peering, 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 77 active+clean; 323 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8/300 objects degraded (2.667%); 2/300 objects misplaced (0.667%); 498 KiB/s, 1 objects/s recovering 2024-09-16T05:14:36.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4075006611' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:36.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:14:36.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:35 smithi032 ceph-mon[27388]: pgmap v650: 87 pgs: 2 peering, 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 77 active+clean; 323 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8/300 objects degraded (2.667%); 2/300 objects misplaced (0.667%); 498 KiB/s, 1 objects/s recovering 2024-09-16T05:14:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4075006611' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:36.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:36.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:36.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:36.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:36.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:36.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:36.414 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:14:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:14:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:14:36.414 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:14:36.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:36.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:36.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:36.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:36.543 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:36.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:36.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:36.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:36.668 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:36.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:36.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:36.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:36.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:36.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:36.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:36.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:36 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:36.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4075006611' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:36.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:36 smithi204 ceph-mon[30933]: osdmap e327: 12 total, 11 up, 12 in 2024-09-16T05:14:37.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:37.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:37.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:37.090 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:36 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:37.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4075006611' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:37.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:36 smithi032 ceph-mon[27388]: osdmap e327: 12 total, 11 up, 12 in 2024-09-16T05:14:37.108 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:37.130 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:37.130 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:37.130 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-16T05:14:37.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:37.153 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:37.153 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:37.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:37.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:37.174 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:37.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:37.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:14:37.218 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:36 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4075006611' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:36 smithi196 ceph-mon[30802]: osdmap e327: 12 total, 11 up, 12 in 2024-09-16T05:14:37.239 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:37.239 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:37.260 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:37.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:37.283 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:37.283 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:37.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:37.304 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:37.305 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:37.325 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:37.325 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:37.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:37.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:37.468 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:37.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:37.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:37.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:37.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:37.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:37.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:37.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:37.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:37.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:37.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:37.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:37.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:37.964 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:37.964 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:14:37.979 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:14:38.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:38.031 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:38.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:38.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:38.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:38.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:37 smithi032 ceph-mon[27388]: pgmap v652: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2372086973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:37 smithi032 ceph-mon[27388]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T05:14:38.170 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:37 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[102067]: 2024-09-16T05:14:37.757+0000 7f19c17a3640 -1 osd.2 295 log_to_monitors true 2024-09-16T05:14:38.170 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:14:37 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[102067]: 2024-09-16T05:14:37.901+0000 7f19b5546640 -1 osd.2 295 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:14:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:37 smithi204 ceph-mon[30933]: pgmap v652: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2372086973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:37 smithi204 ceph-mon[30933]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T05:14:38.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:37 smithi196 ceph-mon[30802]: pgmap v652: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:38.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2372086973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:38.223 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:37 smithi196 ceph-mon[30802]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T05:14:38.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:38.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:38.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:38.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:38.406 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:38.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:38.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:38.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:38.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:38.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:14:38.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:14:38.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:38.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:38.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:38.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:38.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:38.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:38.793 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:38.837 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:38.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:38.900 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:38.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:39.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:39.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:39.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:39.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:39.149 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:39.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:39.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:38 smithi204 ceph-mon[30933]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T05:14:39.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:38 smithi204 ceph-mon[30933]: osdmap e328: 12 total, 11 up, 12 in 2024-09-16T05:14:39.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:38 smithi204 ceph-mon[30933]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:14:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:38 smithi032 ceph-mon[27388]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T05:14:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:38 smithi032 ceph-mon[27388]: osdmap e328: 12 total, 11 up, 12 in 2024-09-16T05:14:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:38 smithi032 ceph-mon[27388]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:14:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:38 smithi196 ceph-mon[30802]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T05:14:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:38 smithi196 ceph-mon[30802]: osdmap e328: 12 total, 11 up, 12 in 2024-09-16T05:14:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:38 smithi196 ceph-mon[30802]: from='osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:14:39.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:39.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:39.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:39.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:39.278 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:39.278 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:39.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:39.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:39.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:39.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:39.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:39.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:39.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:39.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:39.654 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:39.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:39.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:39.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:39.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:39.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:14:39.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:39.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:14:39.903 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:14:39.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:39.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:39.913 INFO:tasks.rados.rados.2.smithi204.stdout:903: done (2 left) 2024-09-16T05:14:39.913 INFO:tasks.rados.rados.2.smithi204.stdout:975: done (1 left) 2024-09-16T05:14:39.913 INFO:tasks.rados.rados.2.smithi204.stdout:976: done (0 left) 2024-09-16T05:14:39.913 INFO:tasks.rados.rados.2.smithi204.stdout:977: rollback oid 16 current snap is 106 2024-09-16T05:14:39.913 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 16 to 91 2024-09-16T05:14:39.913 INFO:tasks.rados.rados.2.smithi204.stdout:978: snap_create 2024-09-16T05:14:39.918 INFO:tasks.rados.rados.2.smithi204.stdout:977: finishing rollback tid 0 to smithi20457400-16 2024-09-16T05:14:39.918 INFO:tasks.rados.rados.2.smithi204.stdout:977: finishing rollback tid 1 to smithi20457400-16 2024-09-16T05:14:39.918 INFO:tasks.rados.rados.2.smithi204.stdout:977: finishing rollback tid 2 to smithi20457400-16 2024-09-16T05:14:39.918 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 425 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T05:14:39.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:14:40.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:40.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:40.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:40.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:40.153 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:40.161 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:40.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:39 smithi032 ceph-mon[27388]: pgmap v654: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:39 smithi032 ceph-mon[27388]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T05:14:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:39 smithi032 ceph-mon[27388]: osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581] boot 2024-09-16T05:14:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:39 smithi032 ceph-mon[27388]: osdmap e329: 12 total, 12 up, 12 in 2024-09-16T05:14:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:39 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:14:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/826462084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/826462084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:40.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:39 smithi032 ceph-mon[27388]: osdmap e330: 12 total, 12 up, 12 in 2024-09-16T05:14:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:39 smithi204 ceph-mon[30933]: pgmap v654: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:39 smithi204 ceph-mon[30933]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T05:14:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:39 smithi204 ceph-mon[30933]: osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581] boot 2024-09-16T05:14:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:39 smithi204 ceph-mon[30933]: osdmap e329: 12 total, 12 up, 12 in 2024-09-16T05:14:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:39 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:14:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/826462084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/826462084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:39 smithi204 ceph-mon[30933]: osdmap e330: 12 total, 12 up, 12 in 2024-09-16T05:14:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:39 smithi196 ceph-mon[30802]: pgmap v654: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:39 smithi196 ceph-mon[30802]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T05:14:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:39 smithi196 ceph-mon[30802]: osd.2 [v2:172.21.15.32:6818/3973436581,v1:172.21.15.32:6819/3973436581] boot 2024-09-16T05:14:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:39 smithi196 ceph-mon[30802]: osdmap e329: 12 total, 12 up, 12 in 2024-09-16T05:14:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:39 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T05:14:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/826462084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/826462084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:39 smithi196 ceph-mon[30802]: osdmap e330: 12 total, 12 up, 12 in 2024-09-16T05:14:40.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:40.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:40.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:40.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:40.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:40.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:40.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:40.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:40.594 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:40.594 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:40.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:40.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:40.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:40.672 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:40.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:40.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:40.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:40.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:40.918 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:40.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:40.920 INFO:tasks.rados.rados.2.smithi204.stdout:977: done (1 left) 2024-09-16T05:14:40.921 INFO:tasks.rados.rados.2.smithi204.stdout:978: done (0 left) 2024-09-16T05:14:40.921 INFO:tasks.rados.rados.2.smithi204.stdout:979: copy_from oid 38 from oid 24 current snap is 107 2024-09-16T05:14:40.921 INFO:tasks.rados.rados.2.smithi204.stdout:980: write (excl) oid 27 current snap is 107 2024-09-16T05:14:40.921 INFO:tasks.rados.rados.2.smithi204.stdout:980: seq_num 364 ranges {517748=459227,1600622=406110} 2024-09-16T05:14:40.926 INFO:tasks.rados.rados.2.smithi204.stdout:980: writing smithi20457400-27 from 517748 to 976975 tid 1 2024-09-16T05:14:40.932 INFO:tasks.rados.rados.2.smithi204.stdout:980: writing smithi20457400-27 from 1600622 to 2006732 tid 2 2024-09-16T05:14:40.932 INFO:tasks.rados.rados.2.smithi204.stdout:981: write (excl) oid 32 current snap is 107 2024-09-16T05:14:40.932 INFO:tasks.rados.rados.2.smithi204.stdout:981: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-16T05:14:40.938 INFO:tasks.rados.rados.2.smithi204.stdout:981: writing smithi20457400-32 from 534555 to 1185384 tid 1 2024-09-16T05:14:40.944 INFO:tasks.rados.rados.2.smithi204.stdout:981: writing smithi20457400-32 from 1659104 to 2134555 tid 2 2024-09-16T05:14:40.944 INFO:tasks.rados.rados.2.smithi204.stdout:982: read oid 6 snap -1 2024-09-16T05:14:40.944 INFO:tasks.rados.rados.2.smithi204.stdout:982: expect (ObjNum 302 snap 83 seq_num 302) 2024-09-16T05:14:40.944 INFO:tasks.rados.rados.2.smithi204.stdout:983: snap_remove snap 102 2024-09-16T05:14:41.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:41.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:41.128 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:41.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:41.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:41.161 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:41.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:40 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 8/294 objects degraded (2.721%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:41.167 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:41.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:41.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:40 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 8/294 objects degraded (2.721%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:41.184 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:41.184 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:41.205 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:14:41.205 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-16T05:14:41.225 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:41.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:40 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 8/294 objects degraded (2.721%), 4 pgs degraded (PG_DEGRADED) 2024-09-16T05:14:41.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:14:41.246 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:41.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:41.268 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:41.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:41.292 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:41.292 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:41.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:41.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:41.313 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:41.314 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:41.355 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:41.356 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:41.378 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:41.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:41.399 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:41.399 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:41.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:41.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:41.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:41.489 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:41.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:41.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:41.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:41.621 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:14:41.622 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:14:41.622 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:14:41.622 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:14:41.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:41.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:41.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:41.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:41.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:41.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:41.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:41.921 INFO:tasks.rados.rados.2.smithi204.stdout:983: done (4 left) 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:984: copy_from oid 47 from oid 4 current snap is 107 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:980: finishing write tid 1 to smithi20457400-27 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:980: finishing write tid 2 to smithi20457400-27 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:980: finishing write tid 3 to smithi20457400-27 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:980: finishing write tid 4 to smithi20457400-27 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 287 (ObjNum 364 snap 107 seq_num 364) dirty exists 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:980: left oid 27 (ObjNum 364 snap 107 seq_num 364) 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:979: finishing copy_from racing read to smithi20457400-38 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:981: finishing write tid 1 to smithi20457400-32 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:979: finishing copy_from to smithi20457400-38 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 378 (ObjNum 346 snap 100 seq_num 346) dirty exists 2024-09-16T05:14:41.922 INFO:tasks.rados.rados.2.smithi204.stdout:981: finishing write tid 2 to smithi20457400-32 2024-09-16T05:14:41.923 INFO:tasks.rados.rados.2.smithi204.stdout:981: finishing write tid 3 to smithi20457400-32 2024-09-16T05:14:41.923 INFO:tasks.rados.rados.2.smithi204.stdout:981: finishing write tid 4 to smithi20457400-32 2024-09-16T05:14:41.923 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 447 (ObjNum 365 snap 107 seq_num 365) dirty exists 2024-09-16T05:14:41.923 INFO:tasks.rados.rados.2.smithi204.stdout:981: left oid 32 (ObjNum 365 snap 107 seq_num 365) 2024-09-16T05:14:41.923 INFO:tasks.rados.rados.2.smithi204.stdout:982: expect (ObjNum 302 snap 83 seq_num 302) 2024-09-16T05:14:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:41.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:41.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:42.035 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:14:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:14:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:14:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:14:42.035 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:14:42.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:42.064 INFO:tasks.rados.rados.2.smithi204.stdout:984: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:14:42.064 INFO:tasks.rados.rados.2.smithi204.stdout:984: finishing copy_from to smithi20457400-47 2024-09-16T05:14:42.065 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 316 (ObjNum 338 snap 96 seq_num 338) dirty exists 2024-09-16T05:14:42.065 INFO:tasks.rados.rados.2.smithi204.stdout:979: done (4 left) 2024-09-16T05:14:42.065 INFO:tasks.rados.rados.2.smithi204.stdout:980: done (3 left) 2024-09-16T05:14:42.065 INFO:tasks.rados.rados.2.smithi204.stdout:981: done (2 left) 2024-09-16T05:14:42.065 INFO:tasks.rados.rados.2.smithi204.stdout:982: done (1 left) 2024-09-16T05:14:42.065 INFO:tasks.rados.rados.2.smithi204.stdout:984: done (0 left) 2024-09-16T05:14:42.065 INFO:tasks.rados.rados.2.smithi204.stdout:985: snap_create 2024-09-16T05:14:42.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:42.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:42.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:41 smithi032 ceph-mon[27388]: pgmap v657: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:42.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:41 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:14:42.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:41 smithi032 ceph-mon[27388]: purged_snaps scrub found 1 strays 2024-09-16T05:14:42.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:41 smithi032 ceph-mon[27388]: osdmap e331: 12 total, 12 up, 12 in 2024-09-16T05:14:42.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4255873415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:42.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:42.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:41 smithi196 ceph-mon[30802]: pgmap v657: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:41 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:14:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:41 smithi196 ceph-mon[30802]: purged_snaps scrub found 1 strays 2024-09-16T05:14:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:41 smithi196 ceph-mon[30802]: osdmap e331: 12 total, 12 up, 12 in 2024-09-16T05:14:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4255873415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:42.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:42.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:42.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:42.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:41 smithi204 ceph-mon[30933]: pgmap v657: 87 pgs: 1 active+clean+remapped, 3 undersized+peered, 4 undersized+degraded+peered, 79 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 8/294 objects degraded (2.721%); 2/294 objects misplaced (0.680%) 2024-09-16T05:14:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:41 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:14:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:41 smithi204 ceph-mon[30933]: purged_snaps scrub found 1 strays 2024-09-16T05:14:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:41 smithi204 ceph-mon[30933]: osdmap e331: 12 total, 12 up, 12 in 2024-09-16T05:14:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4255873415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:42.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:42.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T05:14:42.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:42.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:42.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:42.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:14:42.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:14:42.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:42.692 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:42.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:42.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:42.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:42.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:42.876 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:42.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:42.934 INFO:tasks.rados.rados.2.smithi204.stdout:985: done (0 left) 2024-09-16T05:14:42.934 INFO:tasks.rados.rados.2.smithi204.stdout:986: rollback oid 45 current snap is 108 2024-09-16T05:14:42.934 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 98 2024-09-16T05:14:42.934 INFO:tasks.rados.rados.2.smithi204.stdout:987: write oid 2 current snap is 108 2024-09-16T05:14:42.934 INFO:tasks.rados.rados.2.smithi204.stdout:987: seq_num 366 ranges {551362=726078,1717586=409779} 2024-09-16T05:14:42.940 INFO:tasks.rados.rados.2.smithi204.stdout:987: writing smithi20457400-2 from 551362 to 1277440 tid 1 2024-09-16T05:14:42.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:42.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:42.945 INFO:tasks.rados.rados.2.smithi204.stdout:987: writing smithi20457400-2 from 1717586 to 2127365 tid 2 2024-09-16T05:14:42.945 INFO:tasks.rados.rados.2.smithi204.stdout:988: delete oid 29 current snap is 108 2024-09-16T05:14:42.946 INFO:tasks.rados.rados.2.smithi204.stdout:986: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:14:42.946 INFO:tasks.rados.rados.2.smithi204.stdout:986: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:14:42.946 INFO:tasks.rados.rados.2.smithi204.stdout:986: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:14:42.946 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 333 (ObjNum 302 snap 83 seq_num 302) dirty exists 2024-09-16T05:14:42.946 INFO:tasks.rados.rados.2.smithi204.stdout:987: finishing write tid 1 to smithi20457400-2 2024-09-16T05:14:42.949 INFO:tasks.rados.rados.2.smithi204.stdout:986: done (2 left) 2024-09-16T05:14:42.949 INFO:tasks.rados.rados.2.smithi204.stdout:988: done (1 left) 2024-09-16T05:14:42.949 INFO:tasks.rados.rados.2.smithi204.stdout:989: read oid 23 snap -1 2024-09-16T05:14:42.949 INFO:tasks.rados.rados.2.smithi204.stdout:989: expect (ObjNum 328 snap 91 seq_num 328) 2024-09-16T05:14:42.949 INFO:tasks.rados.rados.2.smithi204.stdout:990: copy_from oid 43 from oid 37 current snap is 108 2024-09-16T05:14:42.949 INFO:tasks.rados.rados.2.smithi204.stdout:987: finishing write tid 2 to smithi20457400-2 2024-09-16T05:14:42.949 INFO:tasks.rados.rados.2.smithi204.stdout:991: snap_remove snap 107 2024-09-16T05:14:42.992 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:42.995 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:43.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:43.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:43.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:43.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:43.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:42 smithi032 ceph-mon[27388]: osdmap e332: 12 total, 12 up, 12 in 2024-09-16T05:14:43.190 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:43.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:42 smithi196 ceph-mon[30802]: osdmap e332: 12 total, 12 up, 12 in 2024-09-16T05:14:43.269 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:42 smithi204 ceph-mon[30933]: osdmap e332: 12 total, 12 up, 12 in 2024-09-16T05:14:43.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:43.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:43.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:290 2.d 2:b98a79db:::smithi20457400-31:head [copy-from ver 13] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 234.027749309, 2024-09-16T05:14:43.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020074727000000001, 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 290 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:14:43.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340464+0000", 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340466+0000", 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4300000000000001e-06 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340473+0000", 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3339999999999996e-06 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340486+0000", 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.279e-05 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340495+0000", 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4890000000000001e-06 2024-09-16T05:14:43.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340559+0000", 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3962999999999997e-05 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340653+0000", 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3468999999999996e-05 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:14:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.353864+0000", 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01321114 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357823+0000", 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039590759999999997 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360478+0000", 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026554489999999998 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360511+0000", 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3052999999999997e-05 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.360539+0000", 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7534000000000001e-05 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:291 2.d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 234.02769342600001, 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016626765000000002, 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 291 2024-09-16T05:14:43.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340520+0000", 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340521+0000", 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4199999999999995e-07 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340524+0000", 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8590000000000001e-06 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340532+0000", 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4090000000000003e-06 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340538+0000", 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.835e-06 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340822+0000", 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028415500000000002 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340874+0000", 2024-09-16T05:14:43.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1558999999999999e-05 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357014+0000", 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016140379999999999 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357072+0000", 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7887999999999999e-05 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.357147+0000", 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5037999999999995e-05 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:605 2.d 2:b1061e13:::smithi20457400-33:head [copy-from ver 20] snapc 13=[13,12,11,f,e,b] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 209.565262429, 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017106367000000001, 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.382 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 605 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802951+0000", 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802952+0000", 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9999999999999998e-07 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802955+0000", 2024-09-16T05:14:43.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1379999999999999e-06 2024-09-16T05:14:43.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802961+0000", 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3820000000000001e-06 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.802966+0000", 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1370000000000001e-06 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803384+0000", 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00041787299999999997 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.803568+0000", 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018420200000000001 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.814402+0000", 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010833617 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.818577+0000", 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041754770000000004 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.819989+0000", 2024-09-16T05:14:43.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001411818 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.820026+0000", 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6613000000000003e-05 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:13.820057+0000", 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1609999999999997e-05 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.389 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1182 2.15 2:a94680a3:::smithi20457400-13:head [setxattr _header (57) in=7b,truncate 3109295] snapc 2f=[2f,2e,2d,2c,2b,2a,29,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 159.37223862299999, 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016329251999999999, 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.390 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1182 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995975+0000", 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9999999999999998e-07 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995978+0000", 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2589999999999999e-06 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995982+0000", 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3059999999999999e-06 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.995985+0000", 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1829999999999998e-06 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996680+0000", 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069462300000000001 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996787+0000", 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107268 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996940+0000", 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000152826 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997265+0000", 2024-09-16T05:14:43.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032523600000000001 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012161+0000", 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014896243999999999 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012231+0000", 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9796999999999996e-05 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012304+0000", 2024-09-16T05:14:43.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3109999999999996e-05 2024-09-16T05:14:43.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.396 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1183 2.15 2:a94680a3:::smithi20457400-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e192)", 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 159.372213367, 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016473326999999999, 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.397 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1183 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996000+0000", 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996001+0000", 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4200000000000002e-07 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996003+0000", 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1679999999999998e-06 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996009+0000", 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3890000000000002e-06 2024-09-16T05:14:43.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.996014+0000", 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7049999999999998e-06 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997177+0000", 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00116278 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:03.997224+0000", 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7601e-05 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012338+0000", 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015113685999999999 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012400+0000", 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2199999999999994e-05 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:04.012474+0000", 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3455999999999996e-05 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1329 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 141.58517663999999, 2024-09-16T05:14:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80822626200000003, 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1329 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:14:43.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783039+0000", 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.674e-06 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783047+0000", 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6619999999999994e-06 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783063+0000", 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6058000000000001e-05 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783071+0000", 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8399999999999995e-06 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783134+0000", 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2496999999999995e-05 2024-09-16T05:14:43.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783142+0000", 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1219999999999995e-06 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591169+0000", 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80802694100000005 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591263+0000", 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4468000000000004e-05 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1330 2.1d 2:b98a79db:::smithi20457400-31:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 141.585125678, 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80866329400000003, 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.408 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1330 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783092+0000", 2024-09-16T05:14:43.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5149999999999998e-06 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783104+0000", 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2412000000000001e-05 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783117+0000", 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2379e-05 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783170+0000", 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3727999999999998e-05 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783178+0000", 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3479999999999998e-06 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591633+0000", 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.808455339 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591751+0000", 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118087 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1331 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 141.58508491200001, 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80878027500000005, 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.413 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1331 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:14:43.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5400000000000001e-07 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783132+0000", 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689999999999999e-06 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783139+0000", 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7920000000000004e-06 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783144+0000", 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783192+0000", 2024-09-16T05:14:43.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8304999999999999e-05 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783196+0000", 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8480000000000003e-06 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591842+0000", 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80864587700000001 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591909+0000", 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7195999999999998e-05 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1361 2.15 2:a94680a3:::smithi20457400-13:head [copy-from ver 281] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 141.02256898499999, 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017774172000000001, 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.420 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1361 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345645+0000", 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345646+0000", 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.646e-06 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345656+0000", 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3170000000000008e-06 2024-09-16T05:14:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345672+0000", 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6821e-05 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345679+0000", 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0840000000000003e-06 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345741+0000", 2024-09-16T05:14:43.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1345000000000003e-05 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.345803+0000", 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2003999999999998e-05 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.354995+0000", 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091920379999999996 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.359012+0000", 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040172389999999997 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363338+0000", 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043261180000000003 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363376+0000", 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7613000000000001e-05 2024-09-16T05:14:43.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.363419+0000", 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2947000000000001e-05 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.9.0:220 2.1d e210/209 ondisk, result = 0)", 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 140.77520747, 2024-09-16T05:14:43.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018128111999999998, 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593006+0000", 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593007+0000", 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3480000000000001e-06 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.593019+0000", 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1963999999999999e-05 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610880+0000", 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017860157000000002 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610890+0000", 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.093e-05 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610951+0000", 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0375999999999999e-05 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610963+0000", 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1861e-05 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.611134+0000", 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000171476 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1389 2.1d 2:b98a79db:::smithi20457400-31:head [copy-from ver 20] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 140.77276365399999, 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.062531666999999999, 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1389 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:14:43.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0499999999999999e-07 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595453+0000", 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9699999999999999e-06 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595467+0000", 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3725e-05 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595472+0000", 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.028e-06 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.597983+0000", 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002510759 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.598071+0000", 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8158000000000005e-05 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.605944+0000", 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078733620000000001 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608849+0000", 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029044190000000001 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657852+0000", 2024-09-16T05:14:43.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.049003110000000002 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657923+0000", 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1388000000000002e-05 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657982+0000", 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8343e-05 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1389 2.1d e210/209 ondisk, result = 0)", 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 140.757364731, 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.047139519999999997, 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610850+0000", 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5600000000000001e-06 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610858+0000", 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2099999999999996e-06 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657761+0000", 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.046903037000000002 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657768+0000", 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4080000000000001e-06 2024-09-16T05:14:43.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657828+0000", 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0241999999999997e-05 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657843+0000", 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5043e-05 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657988+0000", 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014501999999999999 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1471 2.14 2:2ace617f:::smithi20457400-21:head [copy-from ver 158] snapc 3d=[3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 134.68313569700001, 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015385315, 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1471 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685078+0000", 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9299999999999998e-07 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685082+0000", 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3960000000000002e-06 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685089+0000", 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.323e-06 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685094+0000", 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0200000000000002e-06 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685163+0000", 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8861000000000004e-05 2024-09-16T05:14:43.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.685259+0000", 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6365000000000003e-05 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.692105+0000", 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068456330000000003 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.696821+0000", 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047159250000000002 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700376+0000", 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035550970000000001 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700421+0000", 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5158999999999999e-05 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:28.700463+0000", 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2042999999999997e-05 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1575 2.5 2:a5bc1608:::smithi20457400-11:head [setxattr _header (57) in=7b,truncate 3966452] snapc 43=[43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 124.677038891, 2024-09-16T05:14:43.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64249167500000004, 2024-09-16T05:14:43.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1575 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:14:43.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5200000000000002e-07 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691178+0000", 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.734e-06 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691184+0000", 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2940000000000002e-06 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691188+0000", 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.642e-06 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692349+0000", 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001161444 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692452+0000", 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102916 2024-09-16T05:14:43.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692596+0000", 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000143552 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692970+0000", 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037462699999999999 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333552+0000", 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64058177999999999 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333609+0000", 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6728999999999998e-05 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333666+0000", 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7504999999999999e-05 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.454 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1576 2.5 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 124.677020156, 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64272760100000004, 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1576 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691194+0000", 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.65e-07 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691195+0000", 2024-09-16T05:14:43.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-06 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691199+0000", 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0060000000000003e-06 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691202+0000", 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9239999999999999e-06 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692872+0000", 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001669467 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692954+0000", 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2361000000000006e-05 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333715+0000", 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64076073300000003 2024-09-16T05:14:43.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333817+0000", 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010184200000000001 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333921+0000", 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104249 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1596 2.15 2:ad7afe2b:::smithi20457400-38:head [copy-from ver 183] snapc 44=[44,43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:39.857462+0000", 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 123.51075120599999, 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017269851999999999, 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1596 2024-09-16T05:14:43.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857462+0000", 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857462+0000", 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857464+0000", 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9420000000000002e-06 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857474+0000", 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0135999999999999e-05 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857494+0000", 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0030999999999998e-05 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857506+0000", 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1586999999999999e-05 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857574+0000", 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8201999999999995e-05 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.857581+0000", 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2990000000000002e-06 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858652+0000", 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010710990000000001 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.858713+0000", 2024-09-16T05:14:43.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1587999999999994e-05 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.864612+0000", 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058987079999999999 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.868201+0000", 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035895290000000002 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.874670+0000", 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064688539999999996 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.874700+0000", 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9419000000000001e-05 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.874732+0000", 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2458e-05 2024-09-16T05:14:43.467 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2493 2.14 2:2ace617f:::smithi20457400-21:head [copy-from ver 319] snapc 6a=[6a,69,68,67,66,64,63,62,60,5d,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:02.736051+0000", 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 40.632162950000001, 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025553787000000001, 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.468 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2493 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736051+0000", 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736051+0000", 2024-09-16T05:14:43.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736052+0000", 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7699999999999992e-07 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736057+0000", 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2599999999999996e-06 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736069+0000", 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2045000000000001e-05 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736076+0000", 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8870000000000003e-06 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736189+0000", 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011327099999999999 2024-09-16T05:14:43.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736302+0000", 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011278 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.755011+0000", 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018709445000000002 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.760557+0000", 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055454859999999996 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.761496+0000", 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093937199999999995 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.761543+0000", 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6523999999999997e-05 2024-09-16T05:14:43.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.761604+0000", 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1740000000000002e-05 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2494 2.14 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e311)", 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:02.736096+0000", 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 40.632117860000001, 2024-09-16T05:14:43.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023797222, 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2494 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736096+0000", 2024-09-16T05:14:43.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736096+0000", 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736096+0000", 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5700000000000002e-07 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736100+0000", 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1049999999999999e-06 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736110+0000", 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0115000000000001e-05 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736115+0000", 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9960000000000001e-06 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736513+0000", 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00039875400000000002 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736570+0000", 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6767e-05 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.759662+0000", 2024-09-16T05:14:43.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023091326999999998 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.759775+0000", 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113478 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.759893+0000", 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117923 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.479 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2530 2.2f 2:f4b938a4:::smithi20457400-47:head [copy-from ver 316] snapc 6c=[6c,6b,6a,69,67,66,64,63,62,60,5d,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:41.921934+0000", 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.4462793920000001, 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021878980999999999, 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.480 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2530 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.921934+0000", 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.921934+0000", 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.921936+0000", 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9920000000000002e-06 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.921944+0000", 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6890000000000004e-06 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.921958+0000", 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4158999999999999e-05 2024-09-16T05:14:43.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.921969+0000", 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.058e-05 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922040+0000", 2024-09-16T05:14:43.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1592000000000002e-05 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922048+0000", 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0779999999999996e-06 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.924043+0000", 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019950329999999998 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.924179+0000", 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013563 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.936424+0000", 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012244802000000001 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.940810+0000", 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043859759999999998 2024-09-16T05:14:43.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.943666+0000", 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028563629999999998 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.943752+0000", 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6076999999999996e-05 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.943813+0000", 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.101e-05 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2531 2.2f 2:f4b938a4:::smithi20457400-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e332)", 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:41.922003+0000", 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.4462102030000001, 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018255825999999999, 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:14:43.487 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2531 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922003+0000", 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922003+0000", 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922004+0000", 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6100000000000001e-07 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922007+0000", 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.971e-06 2024-09-16T05:14:43.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922016+0000", 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.251e-06 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922022+0000", 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3380000000000001e-06 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922108+0000", 2024-09-16T05:14:43.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5400000000000002e-05 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.922115+0000", 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0430000000000002e-06 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.924326+0000", 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002211406 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:14:43.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.924373+0000", 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6208999999999998e-05 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.940023+0000", 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015650069999999999 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.940134+0000", 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111642 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:14:43.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:41.940259+0000", 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012493500000000001 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:14:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:14:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:43.528 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:43.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:43.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:43.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:43.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:43.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:43.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:43.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:43.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:43.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:43.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T05:14:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:14:43.941 INFO:tasks.rados.rados.2.smithi204.stdout:991: done (3 left) 2024-09-16T05:14:43.942 INFO:tasks.rados.rados.2.smithi204.stdout:992: write oid 25 current snap is 108 2024-09-16T05:14:43.942 INFO:tasks.rados.rados.2.smithi204.stdout:987: finishing write tid 3 to smithi20457400-2 2024-09-16T05:14:43.942 INFO:tasks.rados.rados.2.smithi204.stdout:992: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-16T05:14:43.947 INFO:tasks.rados.rados.2.smithi204.stdout:992: writing smithi20457400-25 from 584976 to 1061552 tid 1 2024-09-16T05:14:43.954 INFO:tasks.rados.rados.2.smithi204.stdout:992: writing smithi20457400-25 from 1834550 to 2184976 tid 2 2024-09-16T05:14:43.954 INFO:tasks.rados.rados.2.smithi204.stdout:993: copy_from oid 28 from oid 48 current snap is 108 2024-09-16T05:14:43.955 INFO:tasks.rados.rados.2.smithi204.stdout:987: finishing write tid 4 to smithi20457400-2 2024-09-16T05:14:43.955 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 162 (ObjNum 366 snap 108 seq_num 366) dirty exists 2024-09-16T05:14:43.955 INFO:tasks.rados.rados.2.smithi204.stdout:987: left oid 2 (ObjNum 366 snap 108 seq_num 366) 2024-09-16T05:14:43.955 INFO:tasks.rados.rados.2.smithi204.stdout:990: finishing copy_from to smithi20457400-43 2024-09-16T05:14:43.955 INFO:tasks.rados.rados.2.smithi204.stdout:990: got expected ENOENT (src dne) 2024-09-16T05:14:43.956 INFO:tasks.rados.rados.2.smithi204.stdout:990: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:14:43.956 INFO:tasks.rados.rados.2.smithi204.stdout:989: expect (ObjNum 328 snap 91 seq_num 328) 2024-09-16T05:14:44.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:44.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:992: finishing write tid 1 to smithi20457400-25 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:992: finishing write tid 2 to smithi20457400-25 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:992: finishing write tid 3 to smithi20457400-25 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:992: finishing write tid 4 to smithi20457400-25 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 379 (ObjNum 368 snap 108 seq_num 368) dirty exists 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:992: left oid 25 (ObjNum 368 snap 108 seq_num 368) 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:993: finishing copy_from to smithi20457400-28 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:993: got expected ENOENT (src dne) 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:993: finishing copy_from racing read to smithi20457400-28 2024-09-16T05:14:44.124 INFO:tasks.rados.rados.2.smithi204.stdout:987: done (4 left) 2024-09-16T05:14:44.125 INFO:tasks.rados.rados.2.smithi204.stdout:989: done (3 left) 2024-09-16T05:14:44.125 INFO:tasks.rados.rados.2.smithi204.stdout:990: done (2 left) 2024-09-16T05:14:44.125 INFO:tasks.rados.rados.2.smithi204.stdout:992: done (1 left) 2024-09-16T05:14:44.125 INFO:tasks.rados.rados.2.smithi204.stdout:993: done (0 left) 2024-09-16T05:14:44.125 INFO:tasks.rados.rados.2.smithi204.stdout:994: delete oid 47 current snap is 108 2024-09-16T05:14:44.125 INFO:tasks.rados.rados.2.smithi204.stdout:994: done (0 left) 2024-09-16T05:14:44.125 INFO:tasks.rados.rados.2.smithi204.stdout:995: snap_remove snap 91 2024-09-16T05:14:44.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:44.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:44.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:44.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:43 smithi032 ceph-mon[27388]: pgmap v660: 87 pgs: 1 peering, 2 undersized+peered, 2 undersized+degraded+peered, 82 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 211 KiB/s wr, 4 op/s; 6/300 objects degraded (2.000%) 2024-09-16T05:14:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:43 smithi032 ceph-mon[27388]: osdmap e333: 12 total, 12 up, 12 in 2024-09-16T05:14:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2816729957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:43 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3037961851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:14:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:43 smithi204 ceph-mon[30933]: pgmap v660: 87 pgs: 1 peering, 2 undersized+peered, 2 undersized+degraded+peered, 82 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 211 KiB/s wr, 4 op/s; 6/300 objects degraded (2.000%) 2024-09-16T05:14:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:43 smithi204 ceph-mon[30933]: osdmap e333: 12 total, 12 up, 12 in 2024-09-16T05:14:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2816729957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:43 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3037961851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:14:44.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:14:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:43 smithi196 ceph-mon[30802]: pgmap v660: 87 pgs: 1 peering, 2 undersized+peered, 2 undersized+degraded+peered, 82 active+clean; 317 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 211 KiB/s wr, 4 op/s; 6/300 objects degraded (2.000%) 2024-09-16T05:14:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:43 smithi196 ceph-mon[30802]: osdmap e333: 12 total, 12 up, 12 in 2024-09-16T05:14:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2816729957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:43 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:14:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3037961851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:14:44.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:44.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:44.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:44.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:44.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:44.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:44.491 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:44.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:44.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:44.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:44.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:44.660 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:44.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:44.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:44.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:44.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:44.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T05:14:44.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:44.910 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:14:44.911 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:14:44.911 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:14:44.911 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:14:44.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:44.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:44.951 INFO:tasks.rados.rados.2.smithi204.stdout:995: done (0 left) 2024-09-16T05:14:44.951 INFO:tasks.rados.rados.2.smithi204.stdout:996: read oid 25 snap -1 2024-09-16T05:14:44.951 INFO:tasks.rados.rados.2.smithi204.stdout:996: expect (ObjNum 368 snap 108 seq_num 368) 2024-09-16T05:14:44.951 INFO:tasks.rados.rados.2.smithi204.stdout:997: delete oid 37 current snap is 108 2024-09-16T05:14:44.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:14:44.956 INFO:tasks.rados.rados.2.smithi204.stdout:997: done (1 left) 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:998: copy_from oid 30 from oid 41 current snap is 108 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:999: read oid 34 snap -1 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:999: expect (ObjNum 357 snap 104 seq_num 357) 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:1000: read oid 19 snap -1 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:1000: expect (ObjNum 361 snap 105 seq_num 361) 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:1001: read oid 1 snap -1 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:1001: expect (ObjNum 340 snap 96 seq_num 340) 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:1002: rollback oid 44 current snap is 108 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 94 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:1003: write (excl) oid 28 current snap is 108 2024-09-16T05:14:44.957 INFO:tasks.rados.rados.2.smithi204.stdout:1003: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-16T05:14:44.964 INFO:tasks.rados.rados.2.smithi204.stdout:1003: writing smithi20457400-28 from 635397 to 1337720 tid 1 2024-09-16T05:14:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:1003: writing smithi20457400-28 from 1893643 to 2235397 tid 2 2024-09-16T05:14:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:1004: copy_from oid 36 from oid 10 current snap is 108 2024-09-16T05:14:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:1002: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:14:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 262 (ObjNum 33 snap 0 seq_num 5) dirty dne 2024-09-16T05:14:44.970 INFO:tasks.rados.rados.2.smithi204.stdout:1000: expect (ObjNum 361 snap 105 seq_num 361) 2024-09-16T05:14:45.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:45.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:45.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:45.084 INFO:tasks.rados.rados.2.smithi204.stdout:996: expect (ObjNum 368 snap 108 seq_num 368) 2024-09-16T05:14:45.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:45.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:45.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:44 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:44 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3037961851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:14:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:44 smithi032 ceph-mon[27388]: osdmap e334: 12 total, 12 up, 12 in 2024-09-16T05:14:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:44 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:14:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:44 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:14:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:44 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:44 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3037961851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:14:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:44 smithi204 ceph-mon[30933]: osdmap e334: 12 total, 12 up, 12 in 2024-09-16T05:14:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:44 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:14:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:44 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:14:45.177 INFO:tasks.rados.rados.2.smithi204.stdout:999: expect (ObjNum 357 snap 104 seq_num 357) 2024-09-16T05:14:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:44 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:44 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:14:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3037961851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:14:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:44 smithi196 ceph-mon[30802]: osdmap e334: 12 total, 12 up, 12 in 2024-09-16T05:14:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:44 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:14:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:44 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:14:45.259 INFO:tasks.rados.rados.2.smithi204.stdout:1001: expect (ObjNum 340 snap 96 seq_num 340) 2024-09-16T05:14:45.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:45.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:45.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:45.383 INFO:tasks.rados.rados.2.smithi204.stdout:1003: finishing write tid 1 to smithi20457400-28 2024-09-16T05:14:45.383 INFO:tasks.rados.rados.2.smithi204.stdout:1003: finishing write tid 2 to smithi20457400-28 2024-09-16T05:14:45.383 INFO:tasks.rados.rados.2.smithi204.stdout:1003: finishing write tid 3 to smithi20457400-28 2024-09-16T05:14:45.383 INFO:tasks.rados.rados.2.smithi204.stdout:1003: finishing write tid 4 to smithi20457400-28 2024-09-16T05:14:45.383 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 301 (ObjNum 371 snap 108 seq_num 371) dirty exists 2024-09-16T05:14:45.383 INFO:tasks.rados.rados.2.smithi204.stdout:1003: left oid 28 (ObjNum 371 snap 108 seq_num 371) 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:1004: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:1004: finishing copy_from to smithi20457400-36 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 356 (ObjNum 349 snap 101 seq_num 349) dirty exists 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:996: done (7 left) 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:999: done (6 left) 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:1000: done (5 left) 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:1001: done (4 left) 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:1002: done (3 left) 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:1003: done (2 left) 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:1004: done (1 left) 2024-09-16T05:14:45.384 INFO:tasks.rados.rados.2.smithi204.stdout:1005: snap_create 2024-09-16T05:14:45.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:45.412 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:45.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:45.445 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:45.464 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:45.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:45.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:14:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:45.536 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:45.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:45.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:45.660 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:45.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:45.693 INFO:tasks.rados.rados.2.smithi204.stdout:998: finishing copy_from racing read to smithi20457400-30 2024-09-16T05:14:45.693 INFO:tasks.rados.rados.2.smithi204.stdout:1005: done (1 left) 2024-09-16T05:14:45.694 INFO:tasks.rados.rados.2.smithi204.stdout:1006: delete oid 4 current snap is 109 2024-09-16T05:14:45.696 INFO:tasks.rados.rados.2.smithi204.stdout:998: finishing copy_from to smithi20457400-30 2024-09-16T05:14:45.696 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 324 (ObjNum 336 snap 95 seq_num 336) dirty exists 2024-09-16T05:14:45.697 INFO:tasks.rados.rados.2.smithi204.stdout:998: done (1 left) 2024-09-16T05:14:45.697 INFO:tasks.rados.rados.2.smithi204.stdout:1006: done (0 left) 2024-09-16T05:14:45.697 INFO:tasks.rados.rados.2.smithi204.stdout:1007: write (excl) oid 22 current snap is 109 2024-09-16T05:14:45.698 INFO:tasks.rados.rados.2.smithi204.stdout:1007: seq_num 373 ranges {669011=569174,2010607=258404} 2024-09-16T05:14:45.704 INFO:tasks.rados.rados.2.smithi204.stdout:1007: writing smithi20457400-22 from 669011 to 1238185 tid 1 2024-09-16T05:14:45.710 INFO:tasks.rados.rados.2.smithi204.stdout:1007: writing smithi20457400-22 from 2010607 to 2269011 tid 2 2024-09-16T05:14:45.710 INFO:tasks.rados.rados.2.smithi204.stdout:1008: copy_from oid 29 from oid 27 current snap is 109 2024-09-16T05:14:45.710 INFO:tasks.rados.rados.2.smithi204.stdout:1007: finishing write tid 1 to smithi20457400-22 2024-09-16T05:14:45.710 INFO:tasks.rados.rados.2.smithi204.stdout:1009: write (excl) oid 45 current snap is 109 2024-09-16T05:14:45.710 INFO:tasks.rados.rados.2.smithi204.stdout:1009: seq_num 374 ranges {685818=644423,2069089=216729} 2024-09-16T05:14:45.718 INFO:tasks.rados.rados.2.smithi204.stdout:1009: writing smithi20457400-45 from 685818 to 1330241 tid 1 2024-09-16T05:14:45.723 INFO:tasks.rados.rados.2.smithi204.stdout:1009: writing smithi20457400-45 from 2069089 to 2285818 tid 2 2024-09-16T05:14:45.723 INFO:tasks.rados.rados.2.smithi204.stdout:1010: write oid 32 current snap is 109 2024-09-16T05:14:45.723 INFO:tasks.rados.rados.2.smithi204.stdout:1008: finishing copy_from racing read to smithi20457400-29 2024-09-16T05:14:45.723 INFO:tasks.rados.rados.2.smithi204.stdout:1009: finishing write tid 1 to smithi20457400-45 2024-09-16T05:14:45.724 INFO:tasks.rados.rados.2.smithi204.stdout:1010: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-16T05:14:45.731 INFO:tasks.rados.rados.2.smithi204.stdout:1010: writing smithi20457400-32 from 702625 to 1422297 tid 1 2024-09-16T05:14:45.736 INFO:tasks.rados.rados.2.smithi204.stdout:1010: writing smithi20457400-32 from 2127571 to 2302625 tid 2 2024-09-16T05:14:45.736 INFO:tasks.rados.rados.2.smithi204.stdout:1011: copy_from oid 49 from oid 18 current snap is 109 2024-09-16T05:14:45.736 INFO:tasks.rados.rados.2.smithi204.stdout:1008: finishing copy_from to smithi20457400-29 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 287 (ObjNum 364 snap 107 seq_num 364) dirty exists 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:1009: finishing write tid 2 to smithi20457400-45 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:1009: finishing write tid 3 to smithi20457400-45 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:1009: finishing write tid 4 to smithi20457400-45 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 336 (ObjNum 374 snap 109 seq_num 374) dirty exists 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:1009: left oid 45 (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:1007: finishing write tid 2 to smithi20457400-22 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:1007: finishing write tid 3 to smithi20457400-22 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:1007: finishing write tid 4 to smithi20457400-22 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 392 (ObjNum 373 snap 109 seq_num 373) dirty exists 2024-09-16T05:14:45.737 INFO:tasks.rados.rados.2.smithi204.stdout:1007: left oid 22 (ObjNum 373 snap 109 seq_num 373) 2024-09-16T05:14:45.738 INFO:tasks.rados.rados.2.smithi204.stdout:1010: finishing write tid 1 to smithi20457400-32 2024-09-16T05:14:45.738 INFO:tasks.rados.rados.2.smithi204.stdout:1007: done (4 left) 2024-09-16T05:14:45.738 INFO:tasks.rados.rados.2.smithi204.stdout:1008: done (3 left) 2024-09-16T05:14:45.738 INFO:tasks.rados.rados.2.smithi204.stdout:1009: done (2 left) 2024-09-16T05:14:45.738 INFO:tasks.rados.rados.2.smithi204.stdout:1012: delete oid 30 current snap is 109 2024-09-16T05:14:45.738 INFO:tasks.rados.rados.2.smithi204.stdout:1012: done (2 left) 2024-09-16T05:14:45.738 INFO:tasks.rados.rados.2.smithi204.stdout:1013: rollback oid 9 current snap is 109 2024-09-16T05:14:45.739 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 9 to 101 2024-09-16T05:14:45.739 INFO:tasks.rados.rados.2.smithi204.stdout:1014: write oid 13 current snap is 109 2024-09-16T05:14:45.739 INFO:tasks.rados.rados.2.smithi204.stdout:1014: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-16T05:14:45.745 INFO:tasks.rados.rados.2.smithi204.stdout:1014: writing smithi20457400-13 from 736239 to 1206409 tid 1 2024-09-16T05:14:45.753 INFO:tasks.rados.rados.2.smithi204.stdout:1014: writing smithi20457400-13 from 1728182 to 2299964 tid 2 2024-09-16T05:14:45.753 INFO:tasks.rados.rados.2.smithi204.stdout:1015: snap_create 2024-09-16T05:14:45.753 INFO:tasks.rados.rados.2.smithi204.stdout:1010: finishing write tid 2 to smithi20457400-32 2024-09-16T05:14:45.753 INFO:tasks.rados.rados.2.smithi204.stdout:1013: finishing rollback tid 0 to smithi20457400-9 2024-09-16T05:14:45.753 INFO:tasks.rados.rados.2.smithi204.stdout:1013: finishing rollback tid 1 to smithi20457400-9 2024-09-16T05:14:45.754 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 369 (ObjNum 129 snap 0 seq_num 4282032128) dirty dne 2024-09-16T05:14:45.754 INFO:tasks.rados.rados.2.smithi204.stdout:1011: finishing copy_from to smithi20457400-49 2024-09-16T05:14:45.754 INFO:tasks.rados.rados.2.smithi204.stdout:1011: got expected ENOENT (src dne) 2024-09-16T05:14:45.754 INFO:tasks.rados.rados.2.smithi204.stdout:1011: finishing copy_from racing read to smithi20457400-49 2024-09-16T05:14:45.754 INFO:tasks.rados.rados.2.smithi204.stdout:1014: finishing write tid 1 to smithi20457400-13 2024-09-16T05:14:45.755 INFO:tasks.rados.rados.2.smithi204.stdout:1014: finishing write tid 2 to smithi20457400-13 2024-09-16T05:14:45.755 INFO:tasks.rados.rados.2.smithi204.stdout:1014: finishing write tid 3 to smithi20457400-13 2024-09-16T05:14:45.755 INFO:tasks.rados.rados.2.smithi204.stdout:1014: finishing write tid 4 to smithi20457400-13 2024-09-16T05:14:45.755 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 416 (ObjNum 377 snap 109 seq_num 377) dirty exists 2024-09-16T05:14:45.755 INFO:tasks.rados.rados.2.smithi204.stdout:1014: left oid 13 (ObjNum 377 snap 109 seq_num 377) 2024-09-16T05:14:45.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:45.785 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:45.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:45.785 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:45.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:45.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:45.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:45.919 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:45.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:45.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:45.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: pgmap v663: 87 pgs: 1 peering, 86 active+clean; 308 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T05:14:45.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:14:45.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:14:45.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:14:45.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:14:45.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-16T05:14:45.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/300 objects degraded (2.000%), 2 pgs degraded) 2024-09-16T05:14:45.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: osdmap e335: 12 total, 12 up, 12 in 2024-09-16T05:14:45.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:14:45.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:14:45.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:14:45.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:14:45.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:14:45.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:14:45.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: osdmap e336: 12 total, 12 up, 12 in 2024-09-16T05:14:45.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1070975922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:45.963 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:14:45.963 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:14:45.983 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:45.984 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:46.005 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:46.005 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:46.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:46.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:46.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:46.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:46.048 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:46.048 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:46.070 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:46.070 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:46.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:46.093 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:46.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:46.138 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:46.138 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:46.159 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:46.159 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:46.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:46.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: pgmap v663: 87 pgs: 1 peering, 86 active+clean; 308 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/300 objects degraded (2.000%), 2 pgs degraded) 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: osdmap e335: 12 total, 12 up, 12 in 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: osdmap e336: 12 total, 12 up, 12 in 2024-09-16T05:14:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1070975922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:46.181 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:46.181 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:46.204 INFO:tasks.rados.rados.2.smithi204.stdout:1010: finishing write tid 3 to smithi20457400-32 2024-09-16T05:14:46.204 INFO:tasks.rados.rados.2.smithi204.stdout:1010: finishing write tid 4 to smithi20457400-32 2024-09-16T05:14:46.204 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 466 (ObjNum 375 snap 109 seq_num 375) dirty exists 2024-09-16T05:14:46.204 INFO:tasks.rados.rados.2.smithi204.stdout:1010: left oid 32 (ObjNum 375 snap 109 seq_num 375) 2024-09-16T05:14:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: pgmap v663: 87 pgs: 1 peering, 86 active+clean; 308 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 0 B/s wr, 1 op/s 2024-09-16T05:14:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:14:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:14:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:14:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:14:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-16T05:14:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/300 objects degraded (2.000%), 2 pgs degraded) 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: osdmap e335: 12 total, 12 up, 12 in 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: osdmap e336: 12 total, 12 up, 12 in 2024-09-16T05:14:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1070975922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:46.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:46.283 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:46.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:46.320 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:46.321 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:46.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:46.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:46.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:46.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:46.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:46.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:46.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:46.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:1010: done (4 left) 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:1011: done (3 left) 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:1013: done (2 left) 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:1014: done (1 left) 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:1015: done (0 left) 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:1016: rollback oid 32 current snap is 110 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 109 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:1017: write (excl) oid 17 current snap is 110 2024-09-16T05:14:46.695 INFO:tasks.rados.rados.2.smithi204.stdout:1017: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-16T05:14:46.701 INFO:tasks.rados.rados.2.smithi204.stdout:1017: writing smithi20457400-17 from 753046 to 1298465 tid 1 2024-09-16T05:14:46.707 INFO:tasks.rados.rados.2.smithi204.stdout:1017: writing smithi20457400-17 from 1786664 to 2218457 tid 2 2024-09-16T05:14:46.707 INFO:tasks.rados.rados.2.smithi204.stdout:1018: read oid 16 snap 98 2024-09-16T05:14:46.707 INFO:tasks.rados.rados.2.smithi204.stdout:1018: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T05:14:46.707 INFO:tasks.rados.rados.2.smithi204.stdout:1019: read oid 21 snap -1 2024-09-16T05:14:46.708 INFO:tasks.rados.rados.2.smithi204.stdout:1019: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:14:46.708 INFO:tasks.rados.rados.2.smithi204.stdout:1020: snap_create 2024-09-16T05:14:46.708 INFO:tasks.rados.rados.2.smithi204.stdout:1016: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:14:46.708 INFO:tasks.rados.rados.2.smithi204.stdout:1016: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:14:46.709 INFO:tasks.rados.rados.2.smithi204.stdout:1016: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:14:46.709 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 470 (ObjNum 375 snap 109 seq_num 375) dirty exists 2024-09-16T05:14:46.709 INFO:tasks.rados.rados.2.smithi204.stdout:1017: finishing write tid 1 to smithi20457400-17 2024-09-16T05:14:46.710 INFO:tasks.rados.rados.2.smithi204.stdout:1017: finishing write tid 2 to smithi20457400-17 2024-09-16T05:14:46.710 INFO:tasks.rados.rados.2.smithi204.stdout:1017: finishing write tid 3 to smithi20457400-17 2024-09-16T05:14:46.710 INFO:tasks.rados.rados.2.smithi204.stdout:1017: finishing write tid 4 to smithi20457400-17 2024-09-16T05:14:46.710 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 304 (ObjNum 378 snap 110 seq_num 378) dirty exists 2024-09-16T05:14:46.710 INFO:tasks.rados.rados.2.smithi204.stdout:1017: left oid 17 (ObjNum 378 snap 110 seq_num 378) 2024-09-16T05:14:46.713 INFO:tasks.rados.rados.2.smithi204.stdout:1019: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:14:46.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:46.787 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:46.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:46.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:46.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:46.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:46.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:14:46.978 INFO:tasks.rados.rados.2.smithi204.stdout:1018: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T05:14:47.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:47.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:47.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:47.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:47.168 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:47.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:46 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:14:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:46 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:14:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:46 smithi204 ceph-mon[30933]: osdmap e337: 12 total, 12 up, 12 in 2024-09-16T05:14:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/760677336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:14:47.213 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:46 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:14:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:46 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:14:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:46 smithi196 ceph-mon[30802]: osdmap e337: 12 total, 12 up, 12 in 2024-09-16T05:14:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/760677336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:14:47.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:47.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:47.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:47.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:14:47.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:46 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:14:47.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:46 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:14:47.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:46 smithi032 ceph-mon[27388]: osdmap e337: 12 total, 12 up, 12 in 2024-09-16T05:14:47.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/760677336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:14:47.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:47.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:47.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:47.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:47.541 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:47.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:47.622 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:47.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:47.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:47.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:47.692 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T05:14:47.696 INFO:tasks.rados.rados.2.smithi204.stdout:1016: done (4 left) 2024-09-16T05:14:47.696 INFO:tasks.rados.rados.2.smithi204.stdout:1017: done (3 left) 2024-09-16T05:14:47.696 INFO:tasks.rados.rados.2.smithi204.stdout:1018: done (2 left) 2024-09-16T05:14:47.696 INFO:tasks.rados.rados.2.smithi204.stdout:1019: done (1 left) 2024-09-16T05:14:47.696 INFO:tasks.rados.rados.2.smithi204.stdout:1020: done (0 left) 2024-09-16T05:14:47.696 INFO:tasks.rados.rados.2.smithi204.stdout:1021: snap_create 2024-09-16T05:14:47.711 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:47.711 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:47.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:47.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:47.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:47.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:47.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:47.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:47.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:14:47.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: pgmap v666: 87 pgs: 87 active+clean; 318 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.7 MiB/s wr, 8 op/s; 626 KiB/s, 0 objects/s recovering 2024-09-16T05:14:47.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:14:47.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:14:47.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:14:47.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:14:47.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:14:47.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:14:47.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/760677336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:14:47.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:47 smithi032 ceph-mon[27388]: osdmap e338: 12 total, 12 up, 12 in 2024-09-16T05:14:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:14:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: pgmap v666: 87 pgs: 87 active+clean; 318 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.7 MiB/s wr, 8 op/s; 626 KiB/s, 0 objects/s recovering 2024-09-16T05:14:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:14:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:14:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:14:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:14:47.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:14:47.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:14:47.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/760677336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:14:47.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:47 smithi196 ceph-mon[30802]: osdmap e338: 12 total, 12 up, 12 in 2024-09-16T05:14:48.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:48.086 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:48.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:48.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:48.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: pgmap v666: 87 pgs: 87 active+clean; 318 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.7 MiB/s wr, 8 op/s; 626 KiB/s, 0 objects/s recovering 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/760677336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:14:48.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:47 smithi204 ceph-mon[30933]: osdmap e338: 12 total, 12 up, 12 in 2024-09-16T05:14:48.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:48.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:14:48.334 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:48.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:48.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:48.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:48.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:48.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:48.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:48.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:48.610 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:48.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:48.698 INFO:tasks.rados.rados.2.smithi204.stdout:1021: done (0 left) 2024-09-16T05:14:48.699 INFO:tasks.rados.rados.2.smithi204.stdout:1022: read oid 49 snap -1 2024-09-16T05:14:48.699 INFO:tasks.rados.rados.2.smithi204.stdout:1022: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:14:48.699 INFO:tasks.rados.rados.2.smithi204.stdout:1023: write (excl) oid 31 current snap is 112 2024-09-16T05:14:48.699 INFO:tasks.rados.rados.2.smithi204.stdout:1023: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-16T05:14:48.706 INFO:tasks.rados.rados.2.smithi204.stdout:1023: writing smithi20457400-31 from 769853 to 1390521 tid 1 2024-09-16T05:14:48.712 INFO:tasks.rados.rados.2.smithi204.stdout:1023: writing smithi20457400-31 from 1845146 to 2369853 tid 2 2024-09-16T05:14:48.712 INFO:tasks.rados.rados.2.smithi204.stdout:1024: rollback oid 9 current snap is 112 2024-09-16T05:14:48.712 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 9 to 110 2024-09-16T05:14:48.712 INFO:tasks.rados.rados.2.smithi204.stdout:1022: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:14:48.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:48.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:48.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:48.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:48.876 INFO:tasks.rados.rados.2.smithi204.stdout:1023: finishing write tid 1 to smithi20457400-31 2024-09-16T05:14:48.876 INFO:tasks.rados.rados.2.smithi204.stdout:1023: finishing write tid 2 to smithi20457400-31 2024-09-16T05:14:48.876 INFO:tasks.rados.rados.2.smithi204.stdout:1023: finishing write tid 3 to smithi20457400-31 2024-09-16T05:14:48.876 INFO:tasks.rados.rados.2.smithi204.stdout:1022: done (2 left) 2024-09-16T05:14:48.876 INFO:tasks.rados.rados.2.smithi204.stdout:1025: snap_remove snap 71 2024-09-16T05:14:48.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T05:14:48.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:49.001 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:49.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:49.019 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:48 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:14:49.020 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:48 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:14:49.020 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:48 smithi204 ceph-mon[30933]: 2.2e scrub starts 2024-09-16T05:14:49.020 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:48 smithi204 ceph-mon[30933]: 2.2e scrub ok 2024-09-16T05:14:49.020 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:48 smithi204 ceph-mon[30933]: osdmap e339: 12 total, 12 up, 12 in 2024-09-16T05:14:49.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:49.119 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:49.119 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:49.126 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:49.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:49.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:49.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:14:49.189 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:49.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:48 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:14:49.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:48 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:14:49.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:48 smithi032 ceph-mon[27388]: 2.2e scrub starts 2024-09-16T05:14:49.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:48 smithi032 ceph-mon[27388]: 2.2e scrub ok 2024-09-16T05:14:49.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:48 smithi032 ceph-mon[27388]: osdmap e339: 12 total, 12 up, 12 in 2024-09-16T05:14:49.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:48 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:14:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:48 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:14:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:48 smithi196 ceph-mon[30802]: 2.2e scrub starts 2024-09-16T05:14:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:48 smithi196 ceph-mon[30802]: 2.2e scrub ok 2024-09-16T05:14:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:48 smithi196 ceph-mon[30802]: osdmap e339: 12 total, 12 up, 12 in 2024-09-16T05:14:49.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:49.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:49.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:49.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:49.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:49.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:49.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:49.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:49.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:49.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:49.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:49.695 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:14:49.699 INFO:tasks.rados.rados.2.smithi204.stdout:1025: done (2 left) 2024-09-16T05:14:49.699 INFO:tasks.rados.rados.2.smithi204.stdout:1026: read oid 20 snap 101 2024-09-16T05:14:49.699 INFO:tasks.rados.rados.2.smithi204.stdout:1026: expect deleted 2024-09-16T05:14:49.699 INFO:tasks.rados.rados.2.smithi204.stdout:1027: snap_create 2024-09-16T05:14:49.699 INFO:tasks.rados.rados.2.smithi204.stdout:1023: finishing write tid 4 to smithi20457400-31 2024-09-16T05:14:49.699 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 538 (ObjNum 379 snap 112 seq_num 379) dirty exists 2024-09-16T05:14:49.700 INFO:tasks.rados.rados.2.smithi204.stdout:1023: left oid 31 (ObjNum 379 snap 112 seq_num 379) 2024-09-16T05:14:49.700 INFO:tasks.rados.rados.2.smithi204.stdout:1024: finishing rollback tid 1 to smithi20457400-9 2024-09-16T05:14:49.700 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 369 (ObjNum 257 snap 0 seq_num 4282423664) dirty dne 2024-09-16T05:14:49.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:49.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:14:49.755 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:49.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:49.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:49.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:49.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:49.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:49.981 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:50.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:50.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:50.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: pgmap v669: 87 pgs: 87 active+clean; 330 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 626 KiB/s, 0 objects/s recovering 2024-09-16T05:14:50.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1098729572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:50.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: 2.36 scrub starts 2024-09-16T05:14:50.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: 2.36 scrub ok 2024-09-16T05:14:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:14:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:14:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:14:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1098729572' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:50.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: osdmap e340: 12 total, 12 up, 12 in 2024-09-16T05:14:50.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4083579663' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:14:50.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:50.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:50.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:50.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: pgmap v669: 87 pgs: 87 active+clean; 330 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 626 KiB/s, 0 objects/s recovering 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1098729572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: 2.36 scrub starts 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: 2.36 scrub ok 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1098729572' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: osdmap e340: 12 total, 12 up, 12 in 2024-09-16T05:14:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4083579663' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:14:50.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:50.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:50.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:50.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: pgmap v669: 87 pgs: 87 active+clean; 330 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 626 KiB/s, 0 objects/s recovering 2024-09-16T05:14:50.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1098729572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:14:50.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: 2.36 scrub starts 2024-09-16T05:14:50.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: 2.36 scrub ok 2024-09-16T05:14:50.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:14:50.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:14:50.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:14:50.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:14:50.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1098729572' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:14:50.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: osdmap e340: 12 total, 12 up, 12 in 2024-09-16T05:14:50.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4083579663' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:14:50.378 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:50.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:50.448 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:50.475 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:50.476 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:50.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:50.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:50.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:50.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:50.626 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:50.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:50.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T05:14:50.694 INFO:tasks.rados.rados.2.smithi204.stdout:1023: done (3 left) 2024-09-16T05:14:50.694 INFO:tasks.rados.rados.2.smithi204.stdout:1024: done (2 left) 2024-09-16T05:14:50.694 INFO:tasks.rados.rados.2.smithi204.stdout:1026: done (1 left) 2024-09-16T05:14:50.694 INFO:tasks.rados.rados.2.smithi204.stdout:1027: done (0 left) 2024-09-16T05:14:50.694 INFO:tasks.rados.rados.2.smithi204.stdout:1028: snap_create 2024-09-16T05:14:50.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:50.750 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:50.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:50.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:50.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:50.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:50.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:14:50.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:51.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:51.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:50 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:14:51.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:50 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:14:51.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:50 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:14:51.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:50 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:14:51.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:50 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:14:51.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:50 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:14:51.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4083579663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:14:51.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:50 smithi204 ceph-mon[30933]: osdmap e341: 12 total, 12 up, 12 in 2024-09-16T05:14:51.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:14:51.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:51.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:51.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:51.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:51.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:50 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:14:51.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:50 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:14:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:50 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:14:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:50 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:14:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:50 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:14:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:50 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:14:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4083579663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:14:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:50 smithi196 ceph-mon[30802]: osdmap e341: 12 total, 12 up, 12 in 2024-09-16T05:14:51.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:51.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:51.331 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:14:51.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:51.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:51.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:51.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:50 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:14:51.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:50 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:14:51.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:50 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:14:51.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:50 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:14:51.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:50 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:14:51.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:50 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:14:51.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4083579663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:14:51.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:50 smithi032 ceph-mon[27388]: osdmap e341: 12 total, 12 up, 12 in 2024-09-16T05:14:51.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T05:14:51.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:51.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:51.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:51.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:51.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:51.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:51.685 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:51.686 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:51.695 INFO:tasks.rados.rados.2.smithi204.stdout:1028: done (0 left) 2024-09-16T05:14:51.695 INFO:tasks.rados.rados.2.smithi204.stdout:1029: read oid 19 snap -1 2024-09-16T05:14:51.695 INFO:tasks.rados.rados.2.smithi204.stdout:1029: expect (ObjNum 361 snap 105 seq_num 361) 2024-09-16T05:14:51.696 INFO:tasks.rados.rados.2.smithi204.stdout:1030: write (excl) oid 33 current snap is 114 2024-09-16T05:14:51.696 INFO:tasks.rados.rados.2.smithi204.stdout:1030: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-16T05:14:51.703 INFO:tasks.rados.rados.2.smithi204.stdout:1030: writing smithi20457400-33 from 786660 to 1482577 tid 1 2024-09-16T05:14:51.708 INFO:tasks.rados.rados.2.smithi204.stdout:1030: writing smithi20457400-33 from 1903628 to 2339090 tid 2 2024-09-16T05:14:51.708 INFO:tasks.rados.rados.2.smithi204.stdout:1031: read oid 31 snap 94 2024-09-16T05:14:51.708 INFO:tasks.rados.rados.2.smithi204.stdout:1031: expect deleted 2024-09-16T05:14:51.708 INFO:tasks.rados.rados.2.smithi204.stdout:1032: rollback oid 17 current snap is 114 2024-09-16T05:14:51.708 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 17 to 111 2024-09-16T05:14:51.709 INFO:tasks.rados.rados.2.smithi204.stdout:1029: expect (ObjNum 361 snap 105 seq_num 361) 2024-09-16T05:14:51.718 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:51.718 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:51.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:51.740 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:14:51.740 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:14:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:51.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:51.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:51.761 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:51.761 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:51.785 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:51.785 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:51.806 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:51.807 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:51.814 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:14:51.814 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:14:51.826 INFO:tasks.rados.rados.2.smithi204.stdout:1030: finishing write tid 1 to smithi20457400-33 2024-09-16T05:14:51.826 INFO:tasks.rados.rados.2.smithi204.stdout:1029: done (3 left) 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:1033: snap_create 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:1030: finishing write tid 2 to smithi20457400-33 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:1030: finishing write tid 3 to smithi20457400-33 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:1030: finishing write tid 4 to smithi20457400-33 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 387 (ObjNum 380 snap 114 seq_num 380) dirty exists 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:1030: left oid 33 (ObjNum 380 snap 114 seq_num 380) 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:1032: finishing rollback tid 0 to smithi20457400-17 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:1032: finishing rollback tid 1 to smithi20457400-17 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:1032: finishing rollback tid 2 to smithi20457400-17 2024-09-16T05:14:51.827 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 307 (ObjNum 378 snap 110 seq_num 378) dirty exists 2024-09-16T05:14:51.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:51.829 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:51.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:51.851 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:51.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:51.873 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:51.873 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:51.882 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:51.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:51.893 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:51.894 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:51.915 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:51.916 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:51.936 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:51.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:52.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:52.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:52.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:52.131 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:52.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:51 smithi204 ceph-mon[30933]: pgmap v672: 87 pgs: 87 active+clean; 330 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-16T05:14:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:51 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:14:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:51 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:14:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3945828039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:51 smithi204 ceph-mon[30933]: osdmap e342: 12 total, 12 up, 12 in 2024-09-16T05:14:52.232 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:52.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:51 smithi196 ceph-mon[30802]: pgmap v672: 87 pgs: 87 active+clean; 330 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-16T05:14:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:51 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:14:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:51 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:14:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3945828039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:51 smithi196 ceph-mon[30802]: osdmap e342: 12 total, 12 up, 12 in 2024-09-16T05:14:52.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:52.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:52.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:51 smithi032 ceph-mon[27388]: pgmap v672: 87 pgs: 87 active+clean; 330 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-16T05:14:52.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:51 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:14:52.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:51 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:14:52.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3945828039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:52.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:51 smithi032 ceph-mon[27388]: osdmap e342: 12 total, 12 up, 12 in 2024-09-16T05:14:52.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:52.378 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:52.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:52.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:14:52.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:14:52.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:52.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:52.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:52.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:52.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:52.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:52.700 INFO:tasks.rados.rados.2.smithi204.stdout:1030: done (3 left) 2024-09-16T05:14:52.700 INFO:tasks.rados.rados.2.smithi204.stdout:1031: done (2 left) 2024-09-16T05:14:52.700 INFO:tasks.rados.rados.2.smithi204.stdout:1032: done (1 left) 2024-09-16T05:14:52.700 INFO:tasks.rados.rados.2.smithi204.stdout:1033: done (0 left) 2024-09-16T05:14:52.700 INFO:tasks.rados.rados.2.smithi204.stdout:1034: write (excl) oid 42 current snap is 115 2024-09-16T05:14:52.700 INFO:tasks.rados.rados.2.smithi204.stdout:1034: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-16T05:14:52.704 INFO:tasks.rados.rados.2.smithi204.stdout:1034: writing smithi20457400-42 from 403467 to 890986 tid 1 2024-09-16T05:14:52.711 INFO:tasks.rados.rados.2.smithi204.stdout:1034: writing smithi20457400-42 from 1562110 to 2141230 tid 2 2024-09-16T05:14:52.711 INFO:tasks.rados.rados.2.smithi204.stdout:1035: write (excl) oid 13 current snap is 115 2024-09-16T05:14:52.711 INFO:tasks.rados.rados.2.smithi204.stdout:1035: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-16T05:14:52.716 INFO:tasks.rados.rados.2.smithi204.stdout:1035: writing smithi20457400-13 from 420274 to 983042 tid 1 2024-09-16T05:14:52.722 INFO:tasks.rados.rados.2.smithi204.stdout:1035: writing smithi20457400-13 from 1620592 to 2059723 tid 2 2024-09-16T05:14:52.723 INFO:tasks.rados.rados.2.smithi204.stdout:1036: copy_from oid 27 from oid 4 current snap is 115 2024-09-16T05:14:52.723 INFO:tasks.rados.rados.2.smithi204.stdout:1034: finishing write tid 1 to smithi20457400-42 2024-09-16T05:14:52.723 INFO:tasks.rados.rados.2.smithi204.stdout:1034: finishing write tid 2 to smithi20457400-42 2024-09-16T05:14:52.723 INFO:tasks.rados.rados.2.smithi204.stdout:1034: finishing write tid 3 to smithi20457400-42 2024-09-16T05:14:52.723 INFO:tasks.rados.rados.2.smithi204.stdout:1037: read oid 11 snap -1 2024-09-16T05:14:52.723 INFO:tasks.rados.rados.2.smithi204.stdout:1037: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:14:52.723 INFO:tasks.rados.rados.2.smithi204.stdout:1038: snap_remove snap 109 2024-09-16T05:14:52.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:52.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:52.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:52.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:52.858 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:52.874 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:52.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:52.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:52.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:53.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:53.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:53.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:53.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:53.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:53.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:52 smithi196 ceph-mon[30802]: 2.3c deep-scrub starts 2024-09-16T05:14:53.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:52 smithi196 ceph-mon[30802]: 2.3c deep-scrub ok 2024-09-16T05:14:53.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:52 smithi196 ceph-mon[30802]: osdmap e343: 12 total, 12 up, 12 in 2024-09-16T05:14:53.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:53.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:53.307 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:52 smithi032 ceph-mon[27388]: 2.3c deep-scrub starts 2024-09-16T05:14:53.307 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:52 smithi032 ceph-mon[27388]: 2.3c deep-scrub ok 2024-09-16T05:14:53.307 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:52 smithi032 ceph-mon[27388]: osdmap e343: 12 total, 12 up, 12 in 2024-09-16T05:14:53.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:53.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:52 smithi204 ceph-mon[30933]: 2.3c deep-scrub starts 2024-09-16T05:14:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:52 smithi204 ceph-mon[30933]: 2.3c deep-scrub ok 2024-09-16T05:14:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:52 smithi204 ceph-mon[30933]: osdmap e343: 12 total, 12 up, 12 in 2024-09-16T05:14:53.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:53.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:53.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:53.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:53.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:53.695 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T05:14:53.701 INFO:tasks.rados.rados.2.smithi204.stdout:1038: done (4 left) 2024-09-16T05:14:53.701 INFO:tasks.rados.rados.2.smithi204.stdout:1039: write (excl) oid 26 current snap is 115 2024-09-16T05:14:53.701 INFO:tasks.rados.rados.2.smithi204.stdout:1034: finishing write tid 4 to smithi20457400-42 2024-09-16T05:14:53.701 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 310 (ObjNum 381 snap 115 seq_num 381) dirty exists 2024-09-16T05:14:53.701 INFO:tasks.rados.rados.2.smithi204.stdout:1034: left oid 42 (ObjNum 381 snap 115 seq_num 381) 2024-09-16T05:14:53.701 INFO:tasks.rados.rados.2.smithi204.stdout:1035: finishing write tid 1 to smithi20457400-13 2024-09-16T05:14:53.701 INFO:tasks.rados.rados.2.smithi204.stdout:1039: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-16T05:14:53.707 INFO:tasks.rados.rados.2.smithi204.stdout:1039: writing smithi20457400-26 from 437081 to 1075098 tid 1 2024-09-16T05:14:53.714 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:14:53.714 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:14:53.714 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:14:53.714 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1039: writing smithi20457400-26 from 1679074 to 2261863 tid 2 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1034: done (4 left) 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1040: read oid 44 snap -1 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1040: expect deleted 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1041: rollback oid 46 current snap is 115 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 106 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1035: finishing write tid 2 to smithi20457400-13 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1035: finishing write tid 3 to smithi20457400-13 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1035: finishing write tid 4 to smithi20457400-13 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 420 (ObjNum 382 snap 115 seq_num 382) dirty exists 2024-09-16T05:14:53.715 INFO:tasks.rados.rados.2.smithi204.stdout:1035: left oid 13 (ObjNum 382 snap 115 seq_num 382) 2024-09-16T05:14:53.716 INFO:tasks.rados.rados.2.smithi204.stdout:1036: finishing copy_from to smithi20457400-27 2024-09-16T05:14:53.716 INFO:tasks.rados.rados.2.smithi204.stdout:1036: got expected ENOENT (src dne) 2024-09-16T05:14:53.716 INFO:tasks.rados.rados.2.smithi204.stdout:1035: done (5 left) 2024-09-16T05:14:53.716 INFO:tasks.rados.rados.2.smithi204.stdout:1042: snap_remove snap 80 2024-09-16T05:14:53.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:53.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:53.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:53.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:53.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:53.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:53.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:53.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:54.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:54.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:54.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:54.129 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:54.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:54.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: pgmap v675: 87 pgs: 87 active+clean; 299 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: 2.14 deep-scrub starts 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: 2.14 deep-scrub ok 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: 2.21 deep-scrub starts 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: 2.21 deep-scrub ok 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2972178733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2972178733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: osdmap e344: 12 total, 12 up, 12 in 2024-09-16T05:14:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/268579461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:54.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:54.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:54.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:54.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:54.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:54.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: pgmap v675: 87 pgs: 87 active+clean; 299 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: 2.14 deep-scrub starts 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: 2.14 deep-scrub ok 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: 2.21 deep-scrub starts 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: 2.21 deep-scrub ok 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2972178733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2972178733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: osdmap e344: 12 total, 12 up, 12 in 2024-09-16T05:14:54.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/268579461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:54.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:14:54.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: pgmap v675: 87 pgs: 87 active+clean; 299 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2024-09-16T05:14:54.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: 2.14 deep-scrub starts 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: 2.14 deep-scrub ok 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: 2.21 deep-scrub starts 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: 2.21 deep-scrub ok 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2972178733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2972178733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: osdmap e344: 12 total, 12 up, 12 in 2024-09-16T05:14:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/268579461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:14:54.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:54.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:54.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:54.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:54.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:14:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:54.638 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:54.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:54.698 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:14:54.704 INFO:tasks.rados.rados.2.smithi204.stdout:1042: done (5 left) 2024-09-16T05:14:54.704 INFO:tasks.rados.rados.2.smithi204.stdout:1043: write (excl) oid 41 current snap is 115 2024-09-16T05:14:54.704 INFO:tasks.rados.rados.2.smithi204.stdout:1036: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:14:54.704 INFO:tasks.rados.rados.2.smithi204.stdout:1043: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-16T05:14:54.711 INFO:tasks.rados.rados.2.smithi204.stdout:1043: writing smithi20457400-41 from 453888 to 1167154 tid 1 2024-09-16T05:14:54.717 INFO:tasks.rados.rados.2.smithi204.stdout:1043: writing smithi20457400-41 from 1737556 to 2180356 tid 2 2024-09-16T05:14:54.717 INFO:tasks.rados.rados.2.smithi204.stdout:1036: done (5 left) 2024-09-16T05:14:54.717 INFO:tasks.rados.rados.2.smithi204.stdout:1044: snap_remove snap 105 2024-09-16T05:14:54.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:54.762 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:54.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:54.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:54.890 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:54.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:54.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:55.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:55.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:55.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:55.146 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:55.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: 2.37 deep-scrub starts 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: 2.37 deep-scrub ok 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/268579461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:54 smithi196 ceph-mon[30802]: osdmap e345: 12 total, 12 up, 12 in 2024-09-16T05:14:55.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:55.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:55.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:55.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: 2.37 deep-scrub starts 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: 2.37 deep-scrub ok 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/268579461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:55.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:54 smithi032 ceph-mon[27388]: osdmap e345: 12 total, 12 up, 12 in 2024-09-16T05:14:55.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:55.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:55.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:14:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:14:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: 2.37 deep-scrub starts 2024-09-16T05:14:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: 2.37 deep-scrub ok 2024-09-16T05:14:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:14:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:14:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:14:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/268579461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:14:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:54 smithi204 ceph-mon[30933]: osdmap e345: 12 total, 12 up, 12 in 2024-09-16T05:14:55.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:55.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:55.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:55.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:55.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:55.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:55.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T05:14:55.697 INFO:tasks.rados.rados.2.smithi204.stdout:1044: done (5 left) 2024-09-16T05:14:55.697 INFO:tasks.rados.rados.2.smithi204.stdout:1045: rollback oid 28 current snap is 115 2024-09-16T05:14:55.697 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 28 to 114 2024-09-16T05:14:55.697 INFO:tasks.rados.rados.2.smithi204.stdout:1037: expect (ObjNum 220 snap 57 seq_num 220) 2024-09-16T05:14:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:55.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:55.780 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:55.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:55.825 INFO:tasks.rados.rados.2.smithi204.stdout:1039: finishing write tid 1 to smithi20457400-26 2024-09-16T05:14:55.825 INFO:tasks.rados.rados.2.smithi204.stdout:1039: finishing write tid 2 to smithi20457400-26 2024-09-16T05:14:55.825 INFO:tasks.rados.rados.2.smithi204.stdout:1037: done (5 left) 2024-09-16T05:14:55.825 INFO:tasks.rados.rados.2.smithi204.stdout:1040: done (4 left) 2024-09-16T05:14:55.825 INFO:tasks.rados.rados.2.smithi204.stdout:1046: write oid 6 current snap is 115 2024-09-16T05:14:55.825 INFO:tasks.rados.rados.2.smithi204.stdout:1039: finishing write tid 3 to smithi20457400-26 2024-09-16T05:14:55.825 INFO:tasks.rados.rados.2.smithi204.stdout:1046: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-16T05:14:55.831 INFO:tasks.rados.rados.2.smithi204.stdout:1046: writing smithi20457400-6 from 470695 to 1259210 tid 1 2024-09-16T05:14:55.836 INFO:tasks.rados.rados.2.smithi204.stdout:1046: writing smithi20457400-6 from 1679685 to 2266143 tid 2 2024-09-16T05:14:55.837 INFO:tasks.rados.rados.2.smithi204.stdout:1047: write (excl) oid 40 current snap is 115 2024-09-16T05:14:55.837 INFO:tasks.rados.rados.2.smithi204.stdout:1047: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-16T05:14:55.841 INFO:tasks.rados.rados.2.smithi204.stdout:1047: writing smithi20457400-40 from 487502 to 951266 tid 1 2024-09-16T05:14:55.848 INFO:tasks.rados.rados.2.smithi204.stdout:1047: writing smithi20457400-40 from 1738167 to 2184636 tid 2 2024-09-16T05:14:55.849 INFO:tasks.rados.rados.2.smithi204.stdout:1048: read oid 39 snap -1 2024-09-16T05:14:55.849 INFO:tasks.rados.rados.2.smithi204.stdout:1048: expect (ObjNum 321 snap 88 seq_num 321) 2024-09-16T05:14:55.849 INFO:tasks.rados.rados.2.smithi204.stdout:1049: read oid 4 snap -1 2024-09-16T05:14:55.849 INFO:tasks.rados.rados.2.smithi204.stdout:1049: expect deleted 2024-09-16T05:14:55.849 INFO:tasks.rados.rados.2.smithi204.stdout:1050: read oid 13 snap 101 2024-09-16T05:14:55.849 INFO:tasks.rados.rados.2.smithi204.stdout:1050: expect (ObjNum 347 snap 101 seq_num 347) 2024-09-16T05:14:55.849 INFO:tasks.rados.rados.2.smithi204.stdout:1051: snap_remove snap 94 2024-09-16T05:14:55.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:55.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:55.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:56.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:56.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:56.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:56.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:56.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:56.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:56.215 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:55 smithi032 ceph-mon[27388]: pgmap v678: 87 pgs: 87 active+clean; 274 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 535 KiB/s wr, 13 op/s 2024-09-16T05:14:56.215 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:55 smithi032 ceph-mon[27388]: osdmap e346: 12 total, 12 up, 12 in 2024-09-16T05:14:56.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:55 smithi196 ceph-mon[30802]: pgmap v678: 87 pgs: 87 active+clean; 274 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 535 KiB/s wr, 13 op/s 2024-09-16T05:14:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:55 smithi196 ceph-mon[30802]: osdmap e346: 12 total, 12 up, 12 in 2024-09-16T05:14:56.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:56.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:56.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:56.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:56.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:56.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:56.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:55 smithi204 ceph-mon[30933]: pgmap v678: 87 pgs: 87 active+clean; 274 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 535 KiB/s wr, 13 op/s 2024-09-16T05:14:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:55 smithi204 ceph-mon[30933]: osdmap e346: 12 total, 12 up, 12 in 2024-09-16T05:14:56.426 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:14:56.427 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:14:56.427 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:14:56.427 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:14:56.427 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:14:56.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:14:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:56.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:56.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:14:56.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:56.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:56.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:56.697 INFO:tasks.rados.rados.2.smithi204.stdout:1051: done (9 left) 2024-09-16T05:14:56.698 INFO:tasks.rados.rados.2.smithi204.stdout:1052: snap_remove snap 96 2024-09-16T05:14:56.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:56.785 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:56.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:56.849 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:14:56.850 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:14:56.850 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:14:56.850 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:14:56.850 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:14:56.850 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:14:56.851 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:14:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:14:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:14:56.852 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:14:56.852 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:14:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:14:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:14:56.866 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:14:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:14:56.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:56.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:56.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:57.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:57.038 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:57.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:57.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:57.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1584257870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:57.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:56 smithi032 ceph-mon[27388]: osdmap e347: 12 total, 12 up, 12 in 2024-09-16T05:14:57.163 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:57.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:57.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1584257870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:56 smithi196 ceph-mon[30802]: osdmap e347: 12 total, 12 up, 12 in 2024-09-16T05:14:57.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:57.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:57.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:57.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:57.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:57.412 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:57.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:57.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1584257870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:14:57.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:56 smithi204 ceph-mon[30933]: osdmap e347: 12 total, 12 up, 12 in 2024-09-16T05:14:57.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:14:57.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:57.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:57.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:57.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:57.593 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:57.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:57.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:57.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:57.699 INFO:tasks.rados.rados.2.smithi204.stdout:1052: done (9 left) 2024-09-16T05:14:57.699 INFO:tasks.rados.rados.2.smithi204.stdout:1053: snap_create 2024-09-16T05:14:57.699 INFO:tasks.rados.rados.2.smithi204.stdout:1039: finishing write tid 4 to smithi20457400-26 2024-09-16T05:14:57.699 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 549 (ObjNum 383 snap 115 seq_num 383) dirty exists 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1039: left oid 26 (ObjNum 383 snap 115 seq_num 383) 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1041: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1041: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1041: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 322 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1043: finishing write tid 1 to smithi20457400-41 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1043: finishing write tid 2 to smithi20457400-41 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1043: finishing write tid 3 to smithi20457400-41 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1043: finishing write tid 4 to smithi20457400-41 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 384 (ObjNum 384 snap 115 seq_num 384) dirty exists 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1043: left oid 41 (ObjNum 384 snap 115 seq_num 384) 2024-09-16T05:14:57.700 INFO:tasks.rados.rados.2.smithi204.stdout:1045: finishing rollback tid 0 to smithi20457400-28 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1045: finishing rollback tid 1 to smithi20457400-28 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1045: finishing rollback tid 2 to smithi20457400-28 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 310 (ObjNum 371 snap 108 seq_num 371) dirty exists 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1046: finishing write tid 1 to smithi20457400-6 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1046: finishing write tid 2 to smithi20457400-6 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1046: finishing write tid 3 to smithi20457400-6 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1046: finishing write tid 4 to smithi20457400-6 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 326 (ObjNum 385 snap 115 seq_num 385) dirty exists 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1046: left oid 6 (ObjNum 385 snap 115 seq_num 385) 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1047: finishing write tid 1 to smithi20457400-40 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1047: finishing write tid 2 to smithi20457400-40 2024-09-16T05:14:57.701 INFO:tasks.rados.rados.2.smithi204.stdout:1047: finishing write tid 3 to smithi20457400-40 2024-09-16T05:14:57.702 INFO:tasks.rados.rados.2.smithi204.stdout:1047: finishing write tid 4 to smithi20457400-40 2024-09-16T05:14:57.702 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 313 (ObjNum 386 snap 115 seq_num 386) dirty exists 2024-09-16T05:14:57.702 INFO:tasks.rados.rados.2.smithi204.stdout:1047: left oid 40 (ObjNum 386 snap 115 seq_num 386) 2024-09-16T05:14:57.702 INFO:tasks.rados.rados.2.smithi204.stdout:1048: expect (ObjNum 321 snap 88 seq_num 321) 2024-09-16T05:14:57.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:14:57.785 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:57.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:57.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:14:57.806 INFO:tasks.rados.rados.2.smithi204.stdout:1050: expect (ObjNum 347 snap 101 seq_num 347) 2024-09-16T05:14:57.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:57.907 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:57.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:57.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:57 smithi032 ceph-mon[27388]: pgmap v681: 87 pgs: 87 active+clean; 265 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 820 KiB/s wr, 11 op/s 2024-09-16T05:14:57.991 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:57 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:14:57.991 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:57 smithi032 ceph-mon[27388]: osdmap e348: 12 total, 12 up, 12 in 2024-09-16T05:14:58.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:58.033 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:58.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:58.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:58.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:57 smithi196 ceph-mon[30802]: pgmap v681: 87 pgs: 87 active+clean; 265 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 820 KiB/s wr, 11 op/s 2024-09-16T05:14:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:57 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:14:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:57 smithi196 ceph-mon[30802]: osdmap e348: 12 total, 12 up, 12 in 2024-09-16T05:14:58.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:14:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:58.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:58.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:14:58.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:14:58.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:58.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:14:58.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:57 smithi204 ceph-mon[30933]: pgmap v681: 87 pgs: 87 active+clean; 265 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 820 KiB/s wr, 11 op/s 2024-09-16T05:14:58.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:57 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:14:58.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:57 smithi204 ceph-mon[30933]: osdmap e348: 12 total, 12 up, 12 in 2024-09-16T05:14:58.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:14:58.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:14:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:58.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:58.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:14:58.575 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:58.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:14:58.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:58.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:14:58.697 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T05:14:58.701 INFO:tasks.rados.rados.2.smithi204.stdout:1039: done (9 left) 2024-09-16T05:14:58.701 INFO:tasks.rados.rados.2.smithi204.stdout:1041: done (8 left) 2024-09-16T05:14:58.702 INFO:tasks.rados.rados.2.smithi204.stdout:1043: done (7 left) 2024-09-16T05:14:58.702 INFO:tasks.rados.rados.2.smithi204.stdout:1045: done (6 left) 2024-09-16T05:14:58.702 INFO:tasks.rados.rados.2.smithi204.stdout:1046: done (5 left) 2024-09-16T05:14:58.702 INFO:tasks.rados.rados.2.smithi204.stdout:1047: done (4 left) 2024-09-16T05:14:58.702 INFO:tasks.rados.rados.2.smithi204.stdout:1048: done (3 left) 2024-09-16T05:14:58.702 INFO:tasks.rados.rados.2.smithi204.stdout:1049: done (2 left) 2024-09-16T05:14:58.702 INFO:tasks.rados.rados.2.smithi204.stdout:1050: done (1 left) 2024-09-16T05:14:58.703 INFO:tasks.rados.rados.2.smithi204.stdout:1053: done (0 left) 2024-09-16T05:14:58.703 INFO:tasks.rados.rados.2.smithi204.stdout:1054: rollback oid 42 current snap is 116 2024-09-16T05:14:58.703 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 115 2024-09-16T05:14:58.703 INFO:tasks.rados.rados.2.smithi204.stdout:1055: write oid 41 current snap is 116 2024-09-16T05:14:58.703 INFO:tasks.rados.rados.2.smithi204.stdout:1055: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-16T05:14:58.706 INFO:tasks.rados.rados.2.smithi204.stdout:1055: writing smithi20457400-41 from 504309 to 1043322 tid 1 2024-09-16T05:14:58.713 INFO:tasks.rados.rados.2.smithi204.stdout:1055: writing smithi20457400-41 from 1796649 to 2386776 tid 2 2024-09-16T05:14:58.713 INFO:tasks.rados.rados.2.smithi204.stdout:1056: read oid 22 snap 111 2024-09-16T05:14:58.714 INFO:tasks.rados.rados.2.smithi204.stdout:1056: expect (ObjNum 373 snap 109 seq_num 373) 2024-09-16T05:14:58.714 INFO:tasks.rados.rados.2.smithi204.stdout:1057: write (excl) oid 2 current snap is 116 2024-09-16T05:14:58.714 INFO:tasks.rados.rados.2.smithi204.stdout:1057: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-16T05:14:58.720 INFO:tasks.rados.rados.2.smithi204.stdout:1057: writing smithi20457400-2 from 521116 to 1251731 tid 1 2024-09-16T05:14:58.728 INFO:tasks.rados.rados.2.smithi204.stdout:1057: writing smithi20457400-2 from 1971484 to 2521116 tid 2 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1058: read oid 26 snap -1 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1058: expect (ObjNum 383 snap 115 seq_num 383) 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1059: delete oid 35 current snap is 116 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1054: finishing rollback tid 0 to smithi20457400-42 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1054: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1054: finishing rollback tid 2 to smithi20457400-42 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 314 (ObjNum 381 snap 115 seq_num 381) dirty exists 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1055: finishing write tid 1 to smithi20457400-41 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1055: finishing write tid 2 to smithi20457400-41 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1055: finishing write tid 3 to smithi20457400-41 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1055: finishing write tid 4 to smithi20457400-41 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 404 (ObjNum 387 snap 116 seq_num 387) dirty exists 2024-09-16T05:14:58.729 INFO:tasks.rados.rados.2.smithi204.stdout:1055: left oid 41 (ObjNum 387 snap 116 seq_num 387) 2024-09-16T05:14:58.730 INFO:tasks.rados.rados.2.smithi204.stdout:1056: expect (ObjNum 373 snap 109 seq_num 373) 2024-09-16T05:14:58.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:14:58.776 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:58.778 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:14:58.835 INFO:tasks.rados.rados.2.smithi204.stdout:1057: finishing write tid 1 to smithi20457400-2 2024-09-16T05:14:58.836 INFO:tasks.rados.rados.2.smithi204.stdout:1057: finishing write tid 2 to smithi20457400-2 2024-09-16T05:14:58.836 INFO:tasks.rados.rados.2.smithi204.stdout:1057: finishing write tid 3 to smithi20457400-2 2024-09-16T05:14:58.836 INFO:tasks.rados.rados.2.smithi204.stdout:1057: finishing write tid 4 to smithi20457400-2 2024-09-16T05:14:58.836 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 165 (ObjNum 388 snap 116 seq_num 388) dirty exists 2024-09-16T05:14:58.836 INFO:tasks.rados.rados.2.smithi204.stdout:1057: left oid 2 (ObjNum 388 snap 116 seq_num 388) 2024-09-16T05:14:58.836 INFO:tasks.rados.rados.2.smithi204.stdout:1058: expect (ObjNum 383 snap 115 seq_num 383) 2024-09-16T05:14:58.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:58.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:59.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:59.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:59.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:59.108 INFO:tasks.rados.rados.2.smithi204.stdout:1054: done (5 left) 2024-09-16T05:14:59.109 INFO:tasks.rados.rados.2.smithi204.stdout:1055: done (4 left) 2024-09-16T05:14:59.109 INFO:tasks.rados.rados.2.smithi204.stdout:1056: done (3 left) 2024-09-16T05:14:59.109 INFO:tasks.rados.rados.2.smithi204.stdout:1057: done (2 left) 2024-09-16T05:14:59.109 INFO:tasks.rados.rados.2.smithi204.stdout:1058: done (1 left) 2024-09-16T05:14:59.109 INFO:tasks.rados.rados.2.smithi204.stdout:1059: done (0 left) 2024-09-16T05:14:59.109 INFO:tasks.rados.rados.2.smithi204.stdout:1060: snap_remove snap 114 2024-09-16T05:14:59.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:14:59.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:14:59.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:14:59.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1866346435' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:14:59.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1866346435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:14:59.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:58 smithi032 ceph-mon[27388]: osdmap e349: 12 total, 12 up, 12 in 2024-09-16T05:14:59.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:58 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:14:59.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:14:58 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:14:59.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:59.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:59.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:14:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:14:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:14:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1866346435' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:14:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1866346435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:14:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:58 smithi196 ceph-mon[30802]: osdmap e349: 12 total, 12 up, 12 in 2024-09-16T05:14:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:58 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:14:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:14:58 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:14:59.251 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:14:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:59.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:14:59.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:14:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:59.401 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:59.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:14:59.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:14:59.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:14:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:14:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1866346435' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:14:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1866346435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:14:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:58 smithi204 ceph-mon[30933]: osdmap e349: 12 total, 12 up, 12 in 2024-09-16T05:14:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:58 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:14:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:14:58 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:14:59.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T05:14:59.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:14:59.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:14:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:59.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:14:59.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:14:59.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:59.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:59.701 INFO:tasks.rados.rados.2.smithi204.stdout:1060: done (0 left) 2024-09-16T05:14:59.702 INFO:tasks.rados.rados.2.smithi204.stdout:1061: snap_remove snap 103 2024-09-16T05:14:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:14:59.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:14:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:59.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:14:59.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:14:59.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:14:59.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:14:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:59.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:00.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:00.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:00.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:00.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:00 smithi032 ceph-mon[27388]: pgmap v684: 87 pgs: 87 active+clean; 251 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-16T05:15:00.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:00 smithi032 ceph-mon[27388]: osdmap e350: 12 total, 12 up, 12 in 2024-09-16T05:15:00.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1869605277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:00.174 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:00.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:00.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:00.245 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:00.251 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:00.251 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:00.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:00.323 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:00.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:00.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:00 smithi204 ceph-mon[30933]: pgmap v684: 87 pgs: 87 active+clean; 251 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-16T05:15:00.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:00 smithi204 ceph-mon[30933]: osdmap e350: 12 total, 12 up, 12 in 2024-09-16T05:15:00.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1869605277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:00.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:00.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:00.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:00.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:00 smithi196 ceph-mon[30802]: pgmap v684: 87 pgs: 87 active+clean; 251 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.9 MiB/s wr, 11 op/s 2024-09-16T05:15:00.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:00 smithi196 ceph-mon[30802]: osdmap e350: 12 total, 12 up, 12 in 2024-09-16T05:15:00.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1869605277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:00.551 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:00.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:00.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:00.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:00.694 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:15:00.697 INFO:tasks.rados.rados.2.smithi204.stdout:1061: done (0 left) 2024-09-16T05:15:00.697 INFO:tasks.rados.rados.2.smithi204.stdout:1062: read oid 30 snap -1 2024-09-16T05:15:00.697 INFO:tasks.rados.rados.2.smithi204.stdout:1062: expect deleted 2024-09-16T05:15:00.697 INFO:tasks.rados.rados.2.smithi204.stdout:1063: delete oid 50 current snap is 116 2024-09-16T05:15:00.703 INFO:tasks.rados.rados.2.smithi204.stdout:1062: done (1 left) 2024-09-16T05:15:00.703 INFO:tasks.rados.rados.2.smithi204.stdout:1063: done (0 left) 2024-09-16T05:15:00.703 INFO:tasks.rados.rados.2.smithi204.stdout:1064: write oid 9 current snap is 116 2024-09-16T05:15:00.703 INFO:tasks.rados.rados.2.smithi204.stdout:1064: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-16T05:15:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:00.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:00.709 INFO:tasks.rados.rados.2.smithi204.stdout:1064: writing smithi20457400-9 from 571537 to 1127899 tid 1 2024-09-16T05:15:00.716 INFO:tasks.rados.rados.2.smithi204.stdout:1064: writing smithi20457400-9 from 1630577 to 2228042 tid 2 2024-09-16T05:15:00.716 INFO:tasks.rados.rados.2.smithi204.stdout:1065: snap_remove snap 104 2024-09-16T05:15:00.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:00.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:00.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:00.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:00.952 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:00.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:01.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:01.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:01.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:01.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:01.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:01.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:01.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:01.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:01.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:01 smithi032 ceph-mon[27388]: pgmap v687: 87 pgs: 87 active+clean; 242 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-09-16T05:15:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:01 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1869605277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:01 smithi032 ceph-mon[27388]: osdmap e351: 12 total, 12 up, 12 in 2024-09-16T05:15:01.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/371504860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:15:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:01 smithi204 ceph-mon[30933]: pgmap v687: 87 pgs: 87 active+clean; 242 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-09-16T05:15:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:01 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1869605277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:01 smithi204 ceph-mon[30933]: osdmap e351: 12 total, 12 up, 12 in 2024-09-16T05:15:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/371504860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:15:01.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:01.456 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:01.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:01 smithi196 ceph-mon[30802]: pgmap v687: 87 pgs: 87 active+clean; 242 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-09-16T05:15:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:01 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1869605277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:01 smithi196 ceph-mon[30802]: osdmap e351: 12 total, 12 up, 12 in 2024-09-16T05:15:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/371504860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:15:01.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:01.582 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:01.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:01.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:01.695 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T05:15:01.699 INFO:tasks.rados.rados.2.smithi204.stdout:1065: done (1 left) 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:1066: read oid 25 snap -1 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:1066: expect (ObjNum 368 snap 108 seq_num 368) 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:1067: delete oid 34 current snap is 116 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:1064: finishing write tid 1 to smithi20457400-9 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:1064: finishing write tid 2 to smithi20457400-9 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:1064: finishing write tid 3 to smithi20457400-9 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:1064: finishing write tid 4 to smithi20457400-9 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 411 (ObjNum 391 snap 116 seq_num 391) dirty exists 2024-09-16T05:15:01.700 INFO:tasks.rados.rados.2.smithi204.stdout:1064: left oid 9 (ObjNum 391 snap 116 seq_num 391) 2024-09-16T05:15:01.704 INFO:tasks.rados.rados.2.smithi204.stdout:1064: done (2 left) 2024-09-16T05:15:01.704 INFO:tasks.rados.rados.2.smithi204.stdout:1067: done (1 left) 2024-09-16T05:15:01.704 INFO:tasks.rados.rados.2.smithi204.stdout:1068: snap_create 2024-09-16T05:15:01.709 INFO:tasks.rados.rados.2.smithi204.stdout:1066: expect (ObjNum 368 snap 108 seq_num 368) 2024-09-16T05:15:01.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:01.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:01.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:01.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:01.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:01.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:01.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:01.967 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:01.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:02.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:15:02.068 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:02.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:02.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:02.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:02.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:02.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:02.268 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:02.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:02.343 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:02.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:02.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:02.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:02.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:02.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:02.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:02.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:02.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:02.707 INFO:tasks.rados.rados.2.smithi204.stdout:1066: done (1 left) 2024-09-16T05:15:02.707 INFO:tasks.rados.rados.2.smithi204.stdout:1068: done (0 left) 2024-09-16T05:15:02.707 INFO:tasks.rados.rados.2.smithi204.stdout:1069: read oid 32 snap -1 2024-09-16T05:15:02.707 INFO:tasks.rados.rados.2.smithi204.stdout:1069: expect (ObjNum 375 snap 109 seq_num 375) 2024-09-16T05:15:02.708 INFO:tasks.rados.rados.2.smithi204.stdout:1070: rollback oid 22 current snap is 117 2024-09-16T05:15:02.708 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 88 2024-09-16T05:15:02.708 INFO:tasks.rados.rados.2.smithi204.stdout:1071: snap_remove snap 110 2024-09-16T05:15:02.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:02.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:02.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:02.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T05:15:02.826 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:02.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:02.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:02.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:02.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:02.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/371504860' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:15:02.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:02 smithi032 ceph-mon[27388]: osdmap e352: 12 total, 12 up, 12 in 2024-09-16T05:15:02.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:02.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:02.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/371504860' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:15:02.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:02 smithi196 ceph-mon[30802]: osdmap e352: 12 total, 12 up, 12 in 2024-09-16T05:15:03.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:03.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:03.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/371504860' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:15:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:02 smithi204 ceph-mon[30933]: osdmap e352: 12 total, 12 up, 12 in 2024-09-16T05:15:03.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:03.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:03.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:03.311 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:03.311 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:03.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:03.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:03.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:03.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:03.484 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:03.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:03.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:15:03.586 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:03.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:03.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:03.708 INFO:tasks.rados.rados.2.smithi204.stdout:1071: done (2 left) 2024-09-16T05:15:03.708 INFO:tasks.rados.rados.2.smithi204.stdout:1072: snap_create 2024-09-16T05:15:03.708 INFO:tasks.rados.rados.2.smithi204.stdout:1070: finishing rollback tid 0 to smithi20457400-22 2024-09-16T05:15:03.708 INFO:tasks.rados.rados.2.smithi204.stdout:1070: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:15:03.708 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 417 (ObjNum 1297 snap 0 seq_num 0) dirty dne 2024-09-16T05:15:03.708 INFO:tasks.rados.rados.2.smithi204.stdout:1069: expect (ObjNum 375 snap 109 seq_num 375) 2024-09-16T05:15:03.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:03.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:03.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:03.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:03 smithi032 ceph-mon[27388]: pgmap v690: 87 pgs: 87 active+clean; 242 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 313 KiB/s wr, 11 op/s 2024-09-16T05:15:03.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:03 smithi032 ceph-mon[27388]: osdmap e353: 12 total, 12 up, 12 in 2024-09-16T05:15:03.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:03 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:03.795 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:03 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:03.808 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:03.808 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:03.830 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:03.831 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:03.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:03.856 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:03.856 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:03.863 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:03.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:03.877 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:03.878 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:03.899 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:03.921 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:03.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:03.963 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:03.964 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:03.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:03.985 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:03.985 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:03.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:03 smithi196 ceph-mon[30802]: pgmap v690: 87 pgs: 87 active+clean; 242 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 313 KiB/s wr, 11 op/s 2024-09-16T05:15:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:03 smithi196 ceph-mon[30802]: osdmap e353: 12 total, 12 up, 12 in 2024-09-16T05:15:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:03 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:03.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:03 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:03.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:04.006 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:04.006 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:04.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:04.028 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:04.036 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:04.050 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:04.050 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:04.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:04.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:04.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:04.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:03 smithi204 ceph-mon[30933]: pgmap v690: 87 pgs: 87 active+clean; 242 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 313 KiB/s wr, 11 op/s 2024-09-16T05:15:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:03 smithi204 ceph-mon[30933]: osdmap e353: 12 total, 12 up, 12 in 2024-09-16T05:15:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:03 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:03 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:04.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:04.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:04.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:04.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:04.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:04.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:04.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:04.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:04.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:04.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:04.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:04.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:04.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:04.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:04.721 INFO:tasks.rados.rados.2.smithi204.stdout:1069: done (2 left) 2024-09-16T05:15:04.721 INFO:tasks.rados.rados.2.smithi204.stdout:1070: done (1 left) 2024-09-16T05:15:04.722 INFO:tasks.rados.rados.2.smithi204.stdout:1072: done (0 left) 2024-09-16T05:15:04.722 INFO:tasks.rados.rados.2.smithi204.stdout:1073: write (excl) oid 34 current snap is 118 2024-09-16T05:15:04.722 INFO:tasks.rados.rados.2.smithi204.stdout:1073: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-16T05:15:04.727 INFO:tasks.rados.rados.2.smithi204.stdout:1073: writing smithi20457400-34 from 605151 to 1312011 tid 1 2024-09-16T05:15:04.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:15:04.735 INFO:tasks.rados.rados.2.smithi204.stdout:1073: writing smithi20457400-34 from 2031188 to 2605151 tid 2 2024-09-16T05:15:04.735 INFO:tasks.rados.rados.2.smithi204.stdout:1074: snap_remove snap 108 2024-09-16T05:15:04.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:04.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:04.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:04.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:04.870 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:04.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:04.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:04.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:04 smithi196 ceph-mon[30802]: osdmap e354: 12 total, 12 up, 12 in 2024-09-16T05:15:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3619299327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:04.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:04.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:05.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:04 smithi204 ceph-mon[30933]: osdmap e354: 12 total, 12 up, 12 in 2024-09-16T05:15:05.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3619299327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:05.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:04 smithi032 ceph-mon[27388]: osdmap e354: 12 total, 12 up, 12 in 2024-09-16T05:15:05.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3619299327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:05.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:05.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:05.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:05.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:05.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:05.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:05.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:05.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:05.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:05.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:05.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:05.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:05.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:05.555 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:05.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:05.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:05.699 INFO:tasks.rados.rados.2.smithi204.stdout:1074: done (1 left) 2024-09-16T05:15:05.699 INFO:tasks.rados.rados.2.smithi204.stdout:1075: read oid 4 snap -1 2024-09-16T05:15:05.699 INFO:tasks.rados.rados.2.smithi204.stdout:1075: expect deleted 2024-09-16T05:15:05.699 INFO:tasks.rados.rados.2.smithi204.stdout:1076: snap_remove snap 100 2024-09-16T05:15:05.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:05.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:05.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:05.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:05.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:05.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:05.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:05 smithi204 ceph-mon[30933]: pgmap v693: 87 pgs: 87 active+clean; 243 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 871 KiB/s wr, 13 op/s 2024-09-16T05:15:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:05 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:05 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:05 smithi204 ceph-mon[30933]: osdmap e355: 12 total, 12 up, 12 in 2024-09-16T05:15:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:05 smithi204 ceph-mon[30933]: osdmap e356: 12 total, 12 up, 12 in 2024-09-16T05:15:05.934 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:05 smithi032 ceph-mon[27388]: pgmap v693: 87 pgs: 87 active+clean; 243 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 871 KiB/s wr, 13 op/s 2024-09-16T05:15:05.934 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:05 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:05.934 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:05 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:05.934 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:05 smithi032 ceph-mon[27388]: osdmap e355: 12 total, 12 up, 12 in 2024-09-16T05:15:05.934 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:05 smithi032 ceph-mon[27388]: osdmap e356: 12 total, 12 up, 12 in 2024-09-16T05:15:05.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:05 smithi196 ceph-mon[30802]: pgmap v693: 87 pgs: 87 active+clean; 243 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 871 KiB/s wr, 13 op/s 2024-09-16T05:15:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:05 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:05 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:05 smithi196 ceph-mon[30802]: osdmap e355: 12 total, 12 up, 12 in 2024-09-16T05:15:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:05 smithi196 ceph-mon[30802]: osdmap e356: 12 total, 12 up, 12 in 2024-09-16T05:15:06.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:06.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:06.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:06.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:06.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:06.150 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:06.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:06.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:06.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:06.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:06.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:06.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:06.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:06.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:06.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:06.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:06.639 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:15:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:15:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:15:06.640 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:15:06.640 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:15:06.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:06.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:06.697 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T05:15:06.698 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:15:06.701 INFO:tasks.rados.rados.2.smithi204.stdout:1076: done (2 left) 2024-09-16T05:15:06.701 INFO:tasks.rados.rados.2.smithi204.stdout:1077: copy_from oid 25 from oid 24 current snap is 118 2024-09-16T05:15:06.701 INFO:tasks.rados.rados.2.smithi204.stdout:1073: finishing write tid 1 to smithi20457400-34 2024-09-16T05:15:06.701 INFO:tasks.rados.rados.2.smithi204.stdout:1073: finishing write tid 2 to smithi20457400-34 2024-09-16T05:15:06.701 INFO:tasks.rados.rados.2.smithi204.stdout:1073: finishing write tid 3 to smithi20457400-34 2024-09-16T05:15:06.701 INFO:tasks.rados.rados.2.smithi204.stdout:1073: finishing write tid 4 to smithi20457400-34 2024-09-16T05:15:06.701 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 453 (ObjNum 393 snap 118 seq_num 393) dirty exists 2024-09-16T05:15:06.702 INFO:tasks.rados.rados.2.smithi204.stdout:1073: left oid 34 (ObjNum 393 snap 118 seq_num 393) 2024-09-16T05:15:06.702 INFO:tasks.rados.rados.2.smithi204.stdout:1073: done (2 left) 2024-09-16T05:15:06.702 INFO:tasks.rados.rados.2.smithi204.stdout:1075: done (1 left) 2024-09-16T05:15:06.702 INFO:tasks.rados.rados.2.smithi204.stdout:1078: delete oid 7 current snap is 118 2024-09-16T05:15:06.705 INFO:tasks.rados.rados.2.smithi204.stdout:1078: done (1 left) 2024-09-16T05:15:06.705 INFO:tasks.rados.rados.2.smithi204.stdout:1079: read oid 49 snap -1 2024-09-16T05:15:06.705 INFO:tasks.rados.rados.2.smithi204.stdout:1079: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:15:06.706 INFO:tasks.rados.rados.2.smithi204.stdout:1080: delete oid 31 current snap is 118 2024-09-16T05:15:06.711 INFO:tasks.rados.rados.2.smithi204.stdout:1080: done (2 left) 2024-09-16T05:15:06.711 INFO:tasks.rados.rados.2.smithi204.stdout:1081: copy_from oid 13 from oid 10 current snap is 118 2024-09-16T05:15:06.711 INFO:tasks.rados.rados.2.smithi204.stdout:1082: write (excl) oid 8 current snap is 118 2024-09-16T05:15:06.711 INFO:tasks.rados.rados.2.smithi204.stdout:1082: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-16T05:15:06.718 INFO:tasks.rados.rados.2.smithi204.stdout:1082: writing smithi20457400-8 from 655572 to 1304532 tid 1 2024-09-16T05:15:06.727 INFO:tasks.rados.rados.2.smithi204.stdout:1082: writing smithi20457400-8 from 1922987 to 2655572 tid 2 2024-09-16T05:15:06.728 INFO:tasks.rados.rados.2.smithi204.stdout:1083: snap_create 2024-09-16T05:15:06.728 INFO:tasks.rados.rados.2.smithi204.stdout:1079: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:15:06.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:06.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:06.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:06.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:06.888 INFO:tasks.rados.rados.2.smithi204.stdout:1077: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:15:06.888 INFO:tasks.rados.rados.2.smithi204.stdout:1077: finishing copy_from to smithi20457400-25 2024-09-16T05:15:06.888 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 378 (ObjNum 346 snap 100 seq_num 346) dirty exists 2024-09-16T05:15:06.888 INFO:tasks.rados.rados.2.smithi204.stdout:1082: finishing write tid 1 to smithi20457400-8 2024-09-16T05:15:06.888 INFO:tasks.rados.rados.2.smithi204.stdout:1081: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:15:06.889 INFO:tasks.rados.rados.2.smithi204.stdout:1081: finishing copy_from to smithi20457400-13 2024-09-16T05:15:06.889 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 356 (ObjNum 349 snap 101 seq_num 349) dirty exists 2024-09-16T05:15:06.889 INFO:tasks.rados.rados.2.smithi204.stdout:1082: finishing write tid 2 to smithi20457400-8 2024-09-16T05:15:06.889 INFO:tasks.rados.rados.2.smithi204.stdout:1082: finishing write tid 3 to smithi20457400-8 2024-09-16T05:15:06.889 INFO:tasks.rados.rados.2.smithi204.stdout:1082: finishing write tid 4 to smithi20457400-8 2024-09-16T05:15:06.889 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 429 (ObjNum 396 snap 118 seq_num 396) dirty exists 2024-09-16T05:15:06.889 INFO:tasks.rados.rados.2.smithi204.stdout:1082: left oid 8 (ObjNum 396 snap 118 seq_num 396) 2024-09-16T05:15:06.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:06.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4004472701' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:15:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1056155453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:06 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4004472701' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:15:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1056155453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:06 smithi032 ceph-mon[27388]: osdmap e357: 12 total, 12 up, 12 in 2024-09-16T05:15:06.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4004472701' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:15:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1056155453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:06 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4004472701' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:15:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1056155453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:06.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:06 smithi196 ceph-mon[30802]: osdmap e357: 12 total, 12 up, 12 in 2024-09-16T05:15:06.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:07.016 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:07.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:07.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:15:07.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:07.140 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:07.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4004472701' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:15:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1056155453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:06 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4004472701' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:15:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1056155453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:06 smithi204 ceph-mon[30933]: osdmap e357: 12 total, 12 up, 12 in 2024-09-16T05:15:07.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:07.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:07.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:07.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:07.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:07.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:07.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:15:07.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:07.511 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:07.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:07.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:07.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:07.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:07.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:07.738 INFO:tasks.rados.rados.2.smithi204.stdout:1077: done (4 left) 2024-09-16T05:15:07.738 INFO:tasks.rados.rados.2.smithi204.stdout:1079: done (3 left) 2024-09-16T05:15:07.738 INFO:tasks.rados.rados.2.smithi204.stdout:1081: done (2 left) 2024-09-16T05:15:07.738 INFO:tasks.rados.rados.2.smithi204.stdout:1082: done (1 left) 2024-09-16T05:15:07.738 INFO:tasks.rados.rados.2.smithi204.stdout:1083: done (0 left) 2024-09-16T05:15:07.738 INFO:tasks.rados.rados.2.smithi204.stdout:1084: write (excl) oid 8 current snap is 119 2024-09-16T05:15:07.738 INFO:tasks.rados.rados.2.smithi204.stdout:1084: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-16T05:15:07.744 INFO:tasks.rados.rados.2.smithi204.stdout:1084: writing smithi20457400-8 from 672379 to 1396588 tid 1 2024-09-16T05:15:07.750 INFO:tasks.rados.rados.2.smithi204.stdout:1084: writing smithi20457400-8 from 1865116 to 2473588 tid 2 2024-09-16T05:15:07.750 INFO:tasks.rados.rados.2.smithi204.stdout:1085: copy_from oid 12 from oid 32 current snap is 119 2024-09-16T05:15:07.750 INFO:tasks.rados.rados.2.smithi204.stdout:1084: finishing write tid 1 to smithi20457400-8 2024-09-16T05:15:07.750 INFO:tasks.rados.rados.2.smithi204.stdout:1086: snap_create 2024-09-16T05:15:07.753 INFO:tasks.rados.rados.2.smithi204.stdout:1084: finishing write tid 2 to smithi20457400-8 2024-09-16T05:15:07.753 INFO:tasks.rados.rados.2.smithi204.stdout:1084: finishing write tid 3 to smithi20457400-8 2024-09-16T05:15:07.753 INFO:tasks.rados.rados.2.smithi204.stdout:1084: finishing write tid 4 to smithi20457400-8 2024-09-16T05:15:07.753 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 433 (ObjNum 397 snap 119 seq_num 397) dirty exists 2024-09-16T05:15:07.753 INFO:tasks.rados.rados.2.smithi204.stdout:1084: left oid 8 (ObjNum 397 snap 119 seq_num 397) 2024-09-16T05:15:07.756 INFO:tasks.rados.rados.2.smithi204.stdout:1085: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:15:07.758 INFO:tasks.rados.rados.2.smithi204.stdout:1085: finishing copy_from to smithi20457400-12 2024-09-16T05:15:07.758 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 470 (ObjNum 375 snap 109 seq_num 375) dirty exists 2024-09-16T05:15:07.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:07.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:07.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:07.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:07.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:07.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:07.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: pgmap v696: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 240 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 871 KiB/s wr, 8 op/s 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: 2.3b scrub starts 2024-09-16T05:15:07.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:07 smithi196 ceph-mon[30802]: 2.3b scrub ok 2024-09-16T05:15:08.007 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:08.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: pgmap v696: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 240 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 871 KiB/s wr, 8 op/s 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:08.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: 2.3b scrub starts 2024-09-16T05:15:08.072 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:07 smithi032 ceph-mon[27388]: 2.3b scrub ok 2024-09-16T05:15:08.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:08.131 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:08.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: pgmap v696: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 240 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 871 KiB/s wr, 8 op/s 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: 2.3b scrub starts 2024-09-16T05:15:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:07 smithi204 ceph-mon[30933]: 2.3b scrub ok 2024-09-16T05:15:08.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:08.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:08.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:08.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:08.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:08.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:08.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:08.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:08.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:08.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:08.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:08.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:08.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T05:15:08.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:08.746 INFO:tasks.rados.rados.2.smithi204.stdout:1084: done (2 left) 2024-09-16T05:15:08.746 INFO:tasks.rados.rados.2.smithi204.stdout:1085: done (1 left) 2024-09-16T05:15:08.746 INFO:tasks.rados.rados.2.smithi204.stdout:1086: done (0 left) 2024-09-16T05:15:08.746 INFO:tasks.rados.rados.2.smithi204.stdout:1087: copy_from oid 32 from oid 28 current snap is 120 2024-09-16T05:15:08.746 INFO:tasks.rados.rados.2.smithi204.stdout:1088: read oid 30 snap -1 2024-09-16T05:15:08.746 INFO:tasks.rados.rados.2.smithi204.stdout:1088: expect deleted 2024-09-16T05:15:08.746 INFO:tasks.rados.rados.2.smithi204.stdout:1089: copy_from oid 31 from oid 43 current snap is 120 2024-09-16T05:15:08.747 INFO:tasks.rados.rados.2.smithi204.stdout:1090: copy_from oid 1 from oid 21 current snap is 120 2024-09-16T05:15:08.747 INFO:tasks.rados.rados.2.smithi204.stdout:1091: write (excl) oid 10 current snap is 120 2024-09-16T05:15:08.747 INFO:tasks.rados.rados.2.smithi204.stdout:1091: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-16T05:15:08.749 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:08.753 INFO:tasks.rados.rados.2.smithi204.stdout:1091: writing smithi20457400-10 from 689186 to 1488644 tid 1 2024-09-16T05:15:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:1091: writing smithi20457400-10 from 1923598 to 2675728 tid 2 2024-09-16T05:15:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:1092: read oid 26 snap -1 2024-09-16T05:15:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:1092: expect (ObjNum 383 snap 115 seq_num 383) 2024-09-16T05:15:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:1093: rollback oid 48 current snap is 120 2024-09-16T05:15:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 48 to 118 2024-09-16T05:15:08.760 INFO:tasks.rados.rados.2.smithi204.stdout:1089: finishing copy_from to smithi20457400-31 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1089: got expected ENOENT (src dne) 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1089: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1087: finishing copy_from racing read to smithi20457400-32 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1090: finishing copy_from racing read to smithi20457400-1 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1090: finishing copy_from to smithi20457400-1 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 319 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1088: done (6 left) 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1089: done (5 left) 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1090: done (4 left) 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1094: read oid 3 snap -1 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1094: expect (ObjNum 308 snap 86 seq_num 308) 2024-09-16T05:15:08.761 INFO:tasks.rados.rados.2.smithi204.stdout:1095: write oid 35 current snap is 120 2024-09-16T05:15:08.762 INFO:tasks.rados.rados.2.smithi204.stdout:1095: seq_num 399 ranges {705993=474707,1582080=612141} 2024-09-16T05:15:08.762 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:08.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:08.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:08.766 INFO:tasks.rados.rados.2.smithi204.stdout:1095: writing smithi20457400-35 from 705993 to 1180700 tid 1 2024-09-16T05:15:08.772 INFO:tasks.rados.rados.2.smithi204.stdout:1095: writing smithi20457400-35 from 1582080 to 2194221 tid 2 2024-09-16T05:15:08.772 INFO:tasks.rados.rados.2.smithi204.stdout:1096: read oid 23 snap -1 2024-09-16T05:15:08.772 INFO:tasks.rados.rados.2.smithi204.stdout:1096: expect (ObjNum 328 snap 91 seq_num 328) 2024-09-16T05:15:08.772 INFO:tasks.rados.rados.2.smithi204.stdout:1097: copy_from oid 50 from oid 44 current snap is 120 2024-09-16T05:15:08.773 INFO:tasks.rados.rados.2.smithi204.stdout:1093: finishing rollback tid 1 to smithi20457400-48 2024-09-16T05:15:08.773 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 341 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-09-16T05:15:08.773 INFO:tasks.rados.rados.2.smithi204.stdout:1091: finishing write tid 1 to smithi20457400-10 2024-09-16T05:15:08.773 INFO:tasks.rados.rados.2.smithi204.stdout:1091: finishing write tid 2 to smithi20457400-10 2024-09-16T05:15:08.773 INFO:tasks.rados.rados.2.smithi204.stdout:1091: finishing write tid 3 to smithi20457400-10 2024-09-16T05:15:08.773 INFO:tasks.rados.rados.2.smithi204.stdout:1094: expect (ObjNum 308 snap 86 seq_num 308) 2024-09-16T05:15:08.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:08.876 INFO:tasks.rados.rados.2.smithi204.stdout:1092: expect (ObjNum 383 snap 115 seq_num 383) 2024-09-16T05:15:08.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:08.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:08.954 INFO:tasks.rados.rados.2.smithi204.stdout:1087: finishing copy_from to smithi20457400-32 2024-09-16T05:15:08.954 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 310 (ObjNum 371 snap 108 seq_num 371) dirty exists 2024-09-16T05:15:08.955 INFO:tasks.rados.rados.2.smithi204.stdout:1095: finishing write tid 1 to smithi20457400-35 2024-09-16T05:15:08.955 INFO:tasks.rados.rados.2.smithi204.stdout:1095: finishing write tid 2 to smithi20457400-35 2024-09-16T05:15:08.955 INFO:tasks.rados.rados.2.smithi204.stdout:1095: finishing write tid 3 to smithi20457400-35 2024-09-16T05:15:08.955 INFO:tasks.rados.rados.2.smithi204.stdout:1095: finishing write tid 4 to smithi20457400-35 2024-09-16T05:15:08.955 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 438 (ObjNum 399 snap 120 seq_num 399) dirty exists 2024-09-16T05:15:08.955 INFO:tasks.rados.rados.2.smithi204.stdout:1095: left oid 35 (ObjNum 399 snap 120 seq_num 399) 2024-09-16T05:15:08.955 INFO:tasks.rados.rados.2.smithi204.stdout:1096: expect (ObjNum 328 snap 91 seq_num 328) 2024-09-16T05:15:08.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:09.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:09.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:09.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:08 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:15:09.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:08 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:15:09.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:08 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:15:09.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:08 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:15:09.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:08 smithi032 ceph-mon[27388]: osdmap e358: 12 total, 12 up, 12 in 2024-09-16T05:15:09.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:08 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:15:09.019 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:08 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:15:09.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1097: finishing copy_from to smithi20457400-50 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1097: got expected ENOENT (src dne) 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1097: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1091: finishing write tid 4 to smithi20457400-10 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 437 (ObjNum 398 snap 120 seq_num 398) dirty exists 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1091: left oid 10 (ObjNum 398 snap 120 seq_num 398) 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1087: done (7 left) 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1091: done (6 left) 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1092: done (5 left) 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1093: done (4 left) 2024-09-16T05:15:09.118 INFO:tasks.rados.rados.2.smithi204.stdout:1094: done (3 left) 2024-09-16T05:15:09.119 INFO:tasks.rados.rados.2.smithi204.stdout:1095: done (2 left) 2024-09-16T05:15:09.119 INFO:tasks.rados.rados.2.smithi204.stdout:1096: done (1 left) 2024-09-16T05:15:09.119 INFO:tasks.rados.rados.2.smithi204.stdout:1097: done (0 left) 2024-09-16T05:15:09.119 INFO:tasks.rados.rados.2.smithi204.stdout:1098: read oid 45 snap -1 2024-09-16T05:15:09.119 INFO:tasks.rados.rados.2.smithi204.stdout:1098: expect (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:15:09.119 INFO:tasks.rados.rados.2.smithi204.stdout:1099: snap_remove snap 86 2024-09-16T05:15:09.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:09.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:09.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:08 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:15:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:08 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:15:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:08 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:15:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:08 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:15:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:08 smithi204 ceph-mon[30933]: osdmap e358: 12 total, 12 up, 12 in 2024-09-16T05:15:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:08 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:15:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:08 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:15:09.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:08 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:15:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:08 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:15:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:08 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:15:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:08 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:15:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:08 smithi196 ceph-mon[30802]: osdmap e358: 12 total, 12 up, 12 in 2024-09-16T05:15:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:08 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:15:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:08 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:15:09.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:09.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:09.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:09.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:09.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:09.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:09.464 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:09.464 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:09.485 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:09.485 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:09.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:09.506 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:09.506 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:09.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:09.526 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:09.527 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:09.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:09.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:09.637 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:09.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:09.666 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:09.666 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:09.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:09.744 INFO:tasks.rados.rados.2.smithi204.stdout:1099: done (1 left) 2024-09-16T05:15:09.744 INFO:tasks.rados.rados.2.smithi204.stdout:1100: write (excl) oid 16 current snap is 120 2024-09-16T05:15:09.744 INFO:tasks.rados.rados.2.smithi204.stdout:1100: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-16T05:15:09.751 INFO:tasks.rados.rados.2.smithi204.stdout:1100: writing smithi20457400-16 from 722800 to 1272756 tid 1 2024-09-16T05:15:09.756 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:09.756 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:09.759 INFO:tasks.rados.rados.2.smithi204.stdout:1100: writing smithi20457400-16 from 2040562 to 2722800 tid 2 2024-09-16T05:15:09.759 INFO:tasks.rados.rados.2.smithi204.stdout:1101: snap_remove snap 88 2024-09-16T05:15:09.765 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:09.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:09.778 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:09.778 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:09.801 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:09.801 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:09.822 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:09.822 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:09.843 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:09.844 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:09.866 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:09.866 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:09.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:09.887 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:09.887 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:09.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:09.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:09.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:09.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:15:09.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:15:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: pgmap v699: 87 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 85 active+clean; 240 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 650 KiB/s wr, 8 op/s 2024-09-16T05:15:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: osdmap e359: 12 total, 12 up, 12 in 2024-09-16T05:15:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:15:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:15:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:15:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:15:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3236692471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:09.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:10.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:10.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:10.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:15:10.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:10.164 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:10.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: pgmap v699: 87 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 85 active+clean; 240 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 650 KiB/s wr, 8 op/s 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: osdmap e359: 12 total, 12 up, 12 in 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:15:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3236692471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:15:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:15:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: pgmap v699: 87 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 85 active+clean; 240 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 650 KiB/s wr, 8 op/s 2024-09-16T05:15:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: osdmap e359: 12 total, 12 up, 12 in 2024-09-16T05:15:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:15:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:15:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:15:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:15:10.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3236692471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:10.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:10.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:10.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:10.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:10.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:10.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:10.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:10.545 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:10.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:10.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:10.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:10.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:10.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:15:10.701 INFO:tasks.rados.rados.2.smithi204.stdout:1101: done (2 left) 2024-09-16T05:15:10.701 INFO:tasks.rados.rados.2.smithi204.stdout:1102: snap_remove snap 111 2024-09-16T05:15:10.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:10.800 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:10.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:10.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: 2.3d scrub starts 2024-09-16T05:15:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: 2.3d scrub ok 2024-09-16T05:15:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: osdmap e360: 12 total, 12 up, 12 in 2024-09-16T05:15:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: 2.3f scrub starts 2024-09-16T05:15:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: 2.3f scrub ok 2024-09-16T05:15:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:15:10.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:15:10.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:15:10.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:15:10.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:10 smithi032 ceph-mon[27388]: osdmap e361: 12 total, 12 up, 12 in 2024-09-16T05:15:10.927 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:10.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:11.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:11.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:11.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:11.123 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:11.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: 2.3d scrub starts 2024-09-16T05:15:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: 2.3d scrub ok 2024-09-16T05:15:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: osdmap e360: 12 total, 12 up, 12 in 2024-09-16T05:15:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: 2.3f scrub starts 2024-09-16T05:15:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: 2.3f scrub ok 2024-09-16T05:15:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:15:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:15:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:15:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:15:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:10 smithi204 ceph-mon[30933]: osdmap e361: 12 total, 12 up, 12 in 2024-09-16T05:15:11.181 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:11.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: 2.3d scrub starts 2024-09-16T05:15:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: 2.3d scrub ok 2024-09-16T05:15:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: osdmap e360: 12 total, 12 up, 12 in 2024-09-16T05:15:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: 2.3f scrub starts 2024-09-16T05:15:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: 2.3f scrub ok 2024-09-16T05:15:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:15:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:15:11.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:15:11.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:15:11.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:10 smithi196 ceph-mon[30802]: osdmap e361: 12 total, 12 up, 12 in 2024-09-16T05:15:11.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:11.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:11.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:11.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:11.433 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:11.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:11.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:11.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:11.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:11.624 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:11.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:11.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:11.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:11.703 INFO:tasks.rados.rados.2.smithi204.stdout:1102: done (2 left) 2024-09-16T05:15:11.703 INFO:tasks.rados.rados.2.smithi204.stdout:1103: snap_create 2024-09-16T05:15:11.703 INFO:tasks.rados.rados.2.smithi204.stdout:1098: expect (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:15:11.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:11.814 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:11.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:11.829 INFO:tasks.rados.rados.2.smithi204.stdout:1100: finishing write tid 1 to smithi20457400-16 2024-09-16T05:15:11.829 INFO:tasks.rados.rados.2.smithi204.stdout:1100: finishing write tid 2 to smithi20457400-16 2024-09-16T05:15:11.829 INFO:tasks.rados.rados.2.smithi204.stdout:1100: finishing write tid 3 to smithi20457400-16 2024-09-16T05:15:11.829 INFO:tasks.rados.rados.2.smithi204.stdout:1100: finishing write tid 4 to smithi20457400-16 2024-09-16T05:15:11.829 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 452 (ObjNum 400 snap 120 seq_num 400) dirty exists 2024-09-16T05:15:11.829 INFO:tasks.rados.rados.2.smithi204.stdout:1100: left oid 16 (ObjNum 400 snap 120 seq_num 400) 2024-09-16T05:15:11.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:11 smithi032 ceph-mon[27388]: pgmap v702: 87 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 85 active+clean; 243 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2024-09-16T05:15:11.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:11 smithi032 ceph-mon[27388]: 2.3a scrub starts 2024-09-16T05:15:11.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:11 smithi032 ceph-mon[27388]: 2.3a scrub ok 2024-09-16T05:15:11.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:11 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:15:11.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:11 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:15:11.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:11 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:11.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:11 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:11.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:11 smithi032 ceph-mon[27388]: osdmap e362: 12 total, 12 up, 12 in 2024-09-16T05:15:11.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:11.937 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:11.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:12.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:12.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:12.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:12.165 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:11 smithi204 ceph-mon[30933]: pgmap v702: 87 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 85 active+clean; 243 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2024-09-16T05:15:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:11 smithi204 ceph-mon[30933]: 2.3a scrub starts 2024-09-16T05:15:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:11 smithi204 ceph-mon[30933]: 2.3a scrub ok 2024-09-16T05:15:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:11 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:15:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:11 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:15:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:11 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:11 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:11 smithi204 ceph-mon[30933]: osdmap e362: 12 total, 12 up, 12 in 2024-09-16T05:15:12.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:12.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:11 smithi196 ceph-mon[30802]: pgmap v702: 87 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 85 active+clean; 243 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2024-09-16T05:15:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:11 smithi196 ceph-mon[30802]: 2.3a scrub starts 2024-09-16T05:15:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:11 smithi196 ceph-mon[30802]: 2.3a scrub ok 2024-09-16T05:15:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:11 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:15:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:11 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:15:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:11 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:11 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:11 smithi196 ceph-mon[30802]: osdmap e362: 12 total, 12 up, 12 in 2024-09-16T05:15:12.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:12.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:12.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:12.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:12.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:12.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:12.500 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:12.500 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:12.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:12.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:12.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:12.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:15:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T05:15:12.705 INFO:tasks.rados.rados.2.smithi204.stdout:1098: done (2 left) 2024-09-16T05:15:12.705 INFO:tasks.rados.rados.2.smithi204.stdout:1100: done (1 left) 2024-09-16T05:15:12.705 INFO:tasks.rados.rados.2.smithi204.stdout:1103: done (0 left) 2024-09-16T05:15:12.706 INFO:tasks.rados.rados.2.smithi204.stdout:1104: snap_remove snap 119 2024-09-16T05:15:12.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:12.805 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:12.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:12.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:12.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:12.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:13.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:13.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:13.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:15:13.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:15:13.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:15:13.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/326240081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:15:13.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/928922136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:13.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:13.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:15:13.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/326240081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:15:13.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/928922136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:13.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:12 smithi032 ceph-mon[27388]: osdmap e363: 12 total, 12 up, 12 in 2024-09-16T05:15:13.155 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:15:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:15:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/326240081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:15:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/928922136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:15:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/326240081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:15:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/928922136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:12 smithi204 ceph-mon[30933]: osdmap e363: 12 total, 12 up, 12 in 2024-09-16T05:15:13.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:13.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/326240081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/928922136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/326240081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/928922136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:12 smithi196 ceph-mon[30802]: osdmap e363: 12 total, 12 up, 12 in 2024-09-16T05:15:13.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:13.304 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:13.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:13.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:13.430 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:13.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:13.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:13.558 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:13.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:13.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:13.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:13.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:13.708 INFO:tasks.rados.rados.2.smithi204.stdout:1104: done (0 left) 2024-09-16T05:15:13.708 INFO:tasks.rados.rados.2.smithi204.stdout:1105: read oid 40 snap 106 2024-09-16T05:15:13.708 INFO:tasks.rados.rados.2.smithi204.stdout:1105: expect (ObjNum 298 snap 83 seq_num 298) 2024-09-16T05:15:13.708 INFO:tasks.rados.rados.2.smithi204.stdout:1106: snap_remove snap 113 2024-09-16T05:15:13.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:13.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:13.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:13.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:13.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:13.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:13.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:13.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:13 smithi032 ceph-mon[27388]: pgmap v705: 87 pgs: 1 active+clean+scrubbing, 86 active+clean; 241 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 14 op/s 2024-09-16T05:15:13.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:13 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:13.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:13 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:13.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:13 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:15:13.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:13 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:15:13.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:13 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:15:13.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:13 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:15:13.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:13 smithi032 ceph-mon[27388]: osdmap e364: 12 total, 12 up, 12 in 2024-09-16T05:15:13.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-16T05:15:14.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:14.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:14.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:14.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:13 smithi204 ceph-mon[30933]: pgmap v705: 87 pgs: 1 active+clean+scrubbing, 86 active+clean; 241 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 14 op/s 2024-09-16T05:15:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:13 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:13 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:13 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:15:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:13 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:15:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:13 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:15:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:13 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:15:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:13 smithi204 ceph-mon[30933]: osdmap e364: 12 total, 12 up, 12 in 2024-09-16T05:15:14.190 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:14.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:13 smithi196 ceph-mon[30802]: pgmap v705: 87 pgs: 1 active+clean+scrubbing, 86 active+clean; 241 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 14 op/s 2024-09-16T05:15:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:13 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:13 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:13 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:15:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:13 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:15:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:13 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:15:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:13 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:15:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:13 smithi196 ceph-mon[30802]: osdmap e364: 12 total, 12 up, 12 in 2024-09-16T05:15:14.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:14.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:14.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:14.420 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:14.421 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:14.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:15:14.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:14.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:14.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:14.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:14.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:14.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:14.699 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:14.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:14.733 INFO:tasks.rados.rados.2.smithi204.stdout:1106: done (1 left) 2024-09-16T05:15:14.734 INFO:tasks.rados.rados.2.smithi204.stdout:1107: snap_remove snap 97 2024-09-16T05:15:14.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:14.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:14.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:14.913 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:14.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:14.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:14.943 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:14.949 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:14.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:14.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:14.966 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:14.966 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:14.987 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:14.987 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:15.007 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:15.007 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:15.029 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:15.029 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:15.051 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:15.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:15.074 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:15.074 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:15.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:15.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:15.098 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:15.119 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:15.120 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:15.140 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:15.141 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:15.162 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:15.163 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:15.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:15.184 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:15.185 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:15.202 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:15.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:15.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:15.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:15.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:15.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:15.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:15:15.456 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:15.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:15.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:15.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:15.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:15.703 INFO:tasks.rados.rados.2.smithi204.stdout:1107: done (1 left) 2024-09-16T05:15:15.703 INFO:tasks.rados.rados.2.smithi204.stdout:1108: delete oid 30 current snap is 121 2024-09-16T05:15:15.704 INFO:tasks.rados.rados.2.smithi204.stdout:1105: expect (ObjNum 298 snap 83 seq_num 298) 2024-09-16T05:15:15.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:15.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:15.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:15.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:15.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:15.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: pgmap v708: 87 pgs: 87 active+clean; 216 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 16 op/s 2024-09-16T05:15:15.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:15:15.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:15:15.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: osdmap e365: 12 total, 12 up, 12 in 2024-09-16T05:15:15.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09111642597711836]}]: dispatch 2024-09-16T05:15:15.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08636587905874532]}]: dispatch 2024-09-16T05:15:15.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09166218261796641]}]: dispatch 2024-09-16T05:15:15.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08767821185471456]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0906802501331491]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045183640485402]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0856330273792523]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0847025030058694]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08808465524958604]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08484481753661685]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07942957169052113]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689702581035666]}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3678527257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:15:15.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09111642597711836]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08636587905874532]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09166218261796641]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08767821185471456]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0906802501331491]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045183640485402]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0856330273792523]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0847025030058694]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08808465524958604]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08484481753661685]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07942957169052113]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689702581035666]}]': finished 2024-09-16T05:15:15.849 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:15 smithi032 ceph-mon[27388]: osdmap e366: 12 total, 12 up, 12 in 2024-09-16T05:15:15.859 INFO:tasks.rados.rados.2.smithi204.stdout:1105: done (1 left) 2024-09-16T05:15:15.859 INFO:tasks.rados.rados.2.smithi204.stdout:1108: done (0 left) 2024-09-16T05:15:15.859 INFO:tasks.rados.rados.2.smithi204.stdout:1109: delete oid 35 current snap is 121 2024-09-16T05:15:15.860 INFO:tasks.rados.rados.2.smithi204.stdout:1109: done (0 left) 2024-09-16T05:15:15.861 INFO:tasks.rados.rados.2.smithi204.stdout:1110: copy_from oid 42 from oid 49 current snap is 121 2024-09-16T05:15:15.861 INFO:tasks.rados.rados.2.smithi204.stdout:1111: write (excl) oid 20 current snap is 121 2024-09-16T05:15:15.861 INFO:tasks.rados.rados.2.smithi204.stdout:1111: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-16T05:15:15.867 INFO:tasks.rados.rados.2.smithi204.stdout:1111: writing smithi20457400-20 from 773221 to 1265277 tid 1 2024-09-16T05:15:15.873 INFO:tasks.rados.rados.2.smithi204.stdout:1111: writing smithi20457400-20 from 1816008 to 2319134 tid 2 2024-09-16T05:15:15.873 INFO:tasks.rados.rados.2.smithi204.stdout:1112: snap_create 2024-09-16T05:15:15.873 INFO:tasks.rados.rados.2.smithi204.stdout:1111: finishing write tid 1 to smithi20457400-20 2024-09-16T05:15:15.873 INFO:tasks.rados.rados.2.smithi204.stdout:1110: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:15:15.875 INFO:tasks.rados.rados.2.smithi204.stdout:1111: finishing write tid 2 to smithi20457400-20 2024-09-16T05:15:15.876 INFO:tasks.rados.rados.2.smithi204.stdout:1111: finishing write tid 3 to smithi20457400-20 2024-09-16T05:15:15.876 INFO:tasks.rados.rados.2.smithi204.stdout:1111: finishing write tid 4 to smithi20457400-20 2024-09-16T05:15:15.876 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 568 (ObjNum 403 snap 121 seq_num 403) dirty exists 2024-09-16T05:15:15.876 INFO:tasks.rados.rados.2.smithi204.stdout:1111: left oid 20 (ObjNum 403 snap 121 seq_num 403) 2024-09-16T05:15:15.876 INFO:tasks.rados.rados.2.smithi204.stdout:1110: finishing copy_from to smithi20457400-42 2024-09-16T05:15:15.876 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 301 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:15:15.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:15.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:15.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: pgmap v708: 87 pgs: 87 active+clean; 216 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 16 op/s 2024-09-16T05:15:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:15:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:15:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: osdmap e365: 12 total, 12 up, 12 in 2024-09-16T05:15:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09111642597711836]}]: dispatch 2024-09-16T05:15:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08636587905874532]}]: dispatch 2024-09-16T05:15:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09166218261796641]}]: dispatch 2024-09-16T05:15:15.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08767821185471456]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0906802501331491]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045183640485402]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0856330273792523]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0847025030058694]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08808465524958604]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08484481753661685]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07942957169052113]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689702581035666]}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3678527257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:15:15.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09111642597711836]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08636587905874532]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09166218261796641]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08767821185471456]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0906802501331491]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045183640485402]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0856330273792523]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0847025030058694]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08808465524958604]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08484481753661685]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07942957169052113]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689702581035666]}]': finished 2024-09-16T05:15:15.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:15 smithi196 ceph-mon[30802]: osdmap e366: 12 total, 12 up, 12 in 2024-09-16T05:15:16.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:16.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:16.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:16.164 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:15:15.845+0000 7f1f8655a640 0 --2- 172.21.15.32:0/2563353211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f80102ef0 0x7f1f8010da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: pgmap v708: 87 pgs: 87 active+clean; 216 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 16 op/s 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: osdmap e365: 12 total, 12 up, 12 in 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09111642597711836]}]: dispatch 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08636587905874532]}]: dispatch 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09166218261796641]}]: dispatch 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08767821185471456]}]: dispatch 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0906802501331491]}]: dispatch 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045183640485402]}]: dispatch 2024-09-16T05:15:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0856330273792523]}]: dispatch 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0847025030058694]}]: dispatch 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08808465524958604]}]: dispatch 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08484481753661685]}]: dispatch 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07942957169052113]}]: dispatch 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689702581035666]}]: dispatch 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3678527257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09111642597711836]}]': finished 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08636587905874532]}]': finished 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09166218261796641]}]': finished 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08767821185471456]}]': finished 2024-09-16T05:15:16.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0906802501331491]}]': finished 2024-09-16T05:15:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045183640485402]}]': finished 2024-09-16T05:15:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0856330273792523]}]': finished 2024-09-16T05:15:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0847025030058694]}]': finished 2024-09-16T05:15:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08808465524958604]}]': finished 2024-09-16T05:15:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08484481753661685]}]': finished 2024-09-16T05:15:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07942957169052113]}]': finished 2024-09-16T05:15:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689702581035666]}]': finished 2024-09-16T05:15:16.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:15 smithi204 ceph-mon[30933]: osdmap e366: 12 total, 12 up, 12 in 2024-09-16T05:15:16.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:16.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:16.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:16.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:16.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:16.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:16.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:16.454 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:16.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:16.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:16.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:16.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:16.643 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:15:16.643 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:15:16.643 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:15:16.643 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:15:16.643 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:15:16.647 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:16.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:16.700 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T05:15:16.703 INFO:tasks.rados.rados.2.smithi204.stdout:1110: done (2 left) 2024-09-16T05:15:16.703 INFO:tasks.rados.rados.2.smithi204.stdout:1111: done (1 left) 2024-09-16T05:15:16.703 INFO:tasks.rados.rados.2.smithi204.stdout:1112: done (0 left) 2024-09-16T05:15:16.703 INFO:tasks.rados.rados.2.smithi204.stdout:1113: snap_remove snap 112 2024-09-16T05:15:16.708 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:16.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:16.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:16.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:16.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:16.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:16.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:16.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2161898939' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2161898939' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:15:16.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:16 smithi196 ceph-mon[30802]: osdmap e367: 12 total, 12 up, 12 in 2024-09-16T05:15:17.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:17.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:17.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:17.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:15:17.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:15:17.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:17.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:17.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2161898939' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2161898939' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:16 smithi032 ceph-mon[27388]: osdmap e367: 12 total, 12 up, 12 in 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:15:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:15:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:15:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2161898939' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:15:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2161898939' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:15:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:16 smithi204 ceph-mon[30933]: osdmap e367: 12 total, 12 up, 12 in 2024-09-16T05:15:17.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:17.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:17.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:17.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:17.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:17.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:17.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:17.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:17.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:17.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:17.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:17.592 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:17.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:17.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-16T05:15:17.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:17.708 INFO:tasks.rados.rados.2.smithi204.stdout:1113: done (0 left) 2024-09-16T05:15:17.708 INFO:tasks.rados.rados.2.smithi204.stdout:1114: write (excl) oid 27 current snap is 122 2024-09-16T05:15:17.708 INFO:tasks.rados.rados.2.smithi204.stdout:1114: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-16T05:15:17.717 INFO:tasks.rados.rados.2.smithi204.stdout:1114: writing smithi20457400-27 from 790028 to 1357333 tid 1 2024-09-16T05:15:17.724 INFO:tasks.rados.rados.2.smithi204.stdout:1114: writing smithi20457400-27 from 1874490 to 2637627 tid 2 2024-09-16T05:15:17.725 INFO:tasks.rados.rados.2.smithi204.stdout:1114: finishing write tid 1 to smithi20457400-27 2024-09-16T05:15:17.725 INFO:tasks.rados.rados.2.smithi204.stdout:1115: write oid 7 current snap is 122 2024-09-16T05:15:17.725 INFO:tasks.rados.rados.2.smithi204.stdout:1115: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-16T05:15:17.726 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:17.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:17.730 INFO:tasks.rados.rados.2.smithi204.stdout:1115: writing smithi20457400-7 from 406835 to 1049389 tid 1 2024-09-16T05:15:17.736 INFO:tasks.rados.rados.2.smithi204.stdout:1115: writing smithi20457400-7 from 1532972 to 2039767 tid 2 2024-09-16T05:15:17.741 INFO:tasks.rados.rados.2.smithi204.stdout:1115: writing smithi20457400-7 from 2772312 to 2806835 tid 3 2024-09-16T05:15:17.741 INFO:tasks.rados.rados.2.smithi204.stdout:1116: write oid 28 current snap is 122 2024-09-16T05:15:17.741 INFO:tasks.rados.rados.2.smithi204.stdout:1114: finishing write tid 2 to smithi20457400-27 2024-09-16T05:15:17.741 INFO:tasks.rados.rados.2.smithi204.stdout:1114: finishing write tid 3 to smithi20457400-27 2024-09-16T05:15:17.741 INFO:tasks.rados.rados.2.smithi204.stdout:1114: finishing write tid 4 to smithi20457400-27 2024-09-16T05:15:17.741 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 290 (ObjNum 404 snap 122 seq_num 404) dirty exists 2024-09-16T05:15:17.741 INFO:tasks.rados.rados.2.smithi204.stdout:1114: left oid 27 (ObjNum 404 snap 122 seq_num 404) 2024-09-16T05:15:17.741 INFO:tasks.rados.rados.2.smithi204.stdout:1115: finishing write tid 1 to smithi20457400-7 2024-09-16T05:15:17.742 INFO:tasks.rados.rados.2.smithi204.stdout:1115: finishing write tid 2 to smithi20457400-7 2024-09-16T05:15:17.742 INFO:tasks.rados.rados.2.smithi204.stdout:1116: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-16T05:15:17.747 INFO:tasks.rados.rados.2.smithi204.stdout:1116: writing smithi20457400-28 from 423642 to 1141445 tid 1 2024-09-16T05:15:17.755 INFO:tasks.rados.rados.2.smithi204.stdout:1116: writing smithi20457400-28 from 1875101 to 2641907 tid 2 2024-09-16T05:15:17.755 INFO:tasks.rados.rados.2.smithi204.stdout:1114: done (2 left) 2024-09-16T05:15:17.755 INFO:tasks.rados.rados.2.smithi204.stdout:1117: read oid 38 snap -1 2024-09-16T05:15:17.756 INFO:tasks.rados.rados.2.smithi204.stdout:1117: expect (ObjNum 346 snap 100 seq_num 346) 2024-09-16T05:15:17.756 INFO:tasks.rados.rados.2.smithi204.stdout:1118: delete oid 19 current snap is 122 2024-09-16T05:15:17.756 INFO:tasks.rados.rados.2.smithi204.stdout:1115: finishing write tid 3 to smithi20457400-7 2024-09-16T05:15:17.756 INFO:tasks.rados.rados.2.smithi204.stdout:1115: finishing write tid 4 to smithi20457400-7 2024-09-16T05:15:17.756 INFO:tasks.rados.rados.2.smithi204.stdout:1115: finishing write tid 5 to smithi20457400-7 2024-09-16T05:15:17.756 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 325 (ObjNum 405 snap 122 seq_num 405) dirty exists 2024-09-16T05:15:17.756 INFO:tasks.rados.rados.2.smithi204.stdout:1115: left oid 7 (ObjNum 405 snap 122 seq_num 405) 2024-09-16T05:15:17.756 INFO:tasks.rados.rados.2.smithi204.stdout:1116: finishing write tid 1 to smithi20457400-28 2024-09-16T05:15:17.757 INFO:tasks.rados.rados.2.smithi204.stdout:1115: done (3 left) 2024-09-16T05:15:17.757 INFO:tasks.rados.rados.2.smithi204.stdout:1118: done (2 left) 2024-09-16T05:15:17.757 INFO:tasks.rados.rados.2.smithi204.stdout:1119: copy_from oid 4 from oid 26 current snap is 122 2024-09-16T05:15:17.757 INFO:tasks.rados.rados.2.smithi204.stdout:1120: copy_from oid 8 from oid 45 current snap is 122 2024-09-16T05:15:17.757 INFO:tasks.rados.rados.2.smithi204.stdout:1121: snap_create 2024-09-16T05:15:17.762 INFO:tasks.rados.rados.2.smithi204.stdout:1116: finishing write tid 2 to smithi20457400-28 2024-09-16T05:15:17.763 INFO:tasks.rados.rados.2.smithi204.stdout:1119: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:15:17.766 INFO:tasks.rados.rados.2.smithi204.stdout:1119: finishing copy_from to smithi20457400-4 2024-09-16T05:15:17.766 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 549 (ObjNum 383 snap 115 seq_num 383) dirty exists 2024-09-16T05:15:17.768 INFO:tasks.rados.rados.2.smithi204.stdout:1120: finishing copy_from racing read to smithi20457400-8 2024-09-16T05:15:17.768 INFO:tasks.rados.rados.2.smithi204.stdout:1117: expect (ObjNum 346 snap 100 seq_num 346) 2024-09-16T05:15:17.820 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:17 smithi032 ceph-mon[27388]: pgmap v711: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 216 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 705 KiB/s wr, 3 op/s 2024-09-16T05:15:17.821 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:17 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:15:17.821 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:17 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:15:17.821 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:17 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:17.821 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:17 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:17.821 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:17 smithi032 ceph-mon[27388]: osdmap e368: 12 total, 12 up, 12 in 2024-09-16T05:15:17.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:17.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:17.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:17.895 INFO:tasks.rados.rados.2.smithi204.stdout:1116: finishing write tid 3 to smithi20457400-28 2024-09-16T05:15:17.895 INFO:tasks.rados.rados.2.smithi204.stdout:1116: finishing write tid 4 to smithi20457400-28 2024-09-16T05:15:17.896 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 354 (ObjNum 406 snap 122 seq_num 406) dirty exists 2024-09-16T05:15:17.896 INFO:tasks.rados.rados.2.smithi204.stdout:1116: left oid 28 (ObjNum 406 snap 122 seq_num 406) 2024-09-16T05:15:17.896 INFO:tasks.rados.rados.2.smithi204.stdout:1120: finishing copy_from to smithi20457400-8 2024-09-16T05:15:17.896 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 336 (ObjNum 374 snap 109 seq_num 374) dirty exists 2024-09-16T05:15:17.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:17.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:17.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:17.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:15:17.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:17 smithi196 ceph-mon[30802]: pgmap v711: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 216 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 705 KiB/s wr, 3 op/s 2024-09-16T05:15:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:17 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:15:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:17 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:15:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:17 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:17 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:17 smithi196 ceph-mon[30802]: osdmap e368: 12 total, 12 up, 12 in 2024-09-16T05:15:18.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:18.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:18.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:17 smithi204 ceph-mon[30933]: pgmap v711: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 216 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 705 KiB/s wr, 3 op/s 2024-09-16T05:15:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:17 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:15:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:17 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:15:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:17 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:17 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:17 smithi204 ceph-mon[30933]: osdmap e368: 12 total, 12 up, 12 in 2024-09-16T05:15:18.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:18.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:18.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:18.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:18.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:18.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:18.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:15:18.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:18.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:18.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:18.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:18.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:18.576 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:18.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:18.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:18.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:18.707 INFO:tasks.rados.rados.2.smithi204.stdout:1116: done (4 left) 2024-09-16T05:15:18.707 INFO:tasks.rados.rados.2.smithi204.stdout:1117: done (3 left) 2024-09-16T05:15:18.707 INFO:tasks.rados.rados.2.smithi204.stdout:1119: done (2 left) 2024-09-16T05:15:18.707 INFO:tasks.rados.rados.2.smithi204.stdout:1120: done (1 left) 2024-09-16T05:15:18.707 INFO:tasks.rados.rados.2.smithi204.stdout:1121: done (0 left) 2024-09-16T05:15:18.707 INFO:tasks.rados.rados.2.smithi204.stdout:1122: write (excl) oid 29 current snap is 123 2024-09-16T05:15:18.707 INFO:tasks.rados.rados.2.smithi204.stdout:1122: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-16T05:15:18.712 INFO:tasks.rados.rados.2.smithi204.stdout:1122: writing smithi20457400-29 from 457256 to 925557 tid 1 2024-09-16T05:15:18.720 INFO:tasks.rados.rados.2.smithi204.stdout:1122: writing smithi20457400-29 from 1592065 to 2362540 tid 2 2024-09-16T05:15:18.784 INFO:tasks.rados.rados.2.smithi204.stdout:1122: writing smithi20457400-29 from 2854581 to 2857256 tid 3 2024-09-16T05:15:18.785 INFO:tasks.rados.rados.2.smithi204.stdout:1123: rollback oid 30 current snap is 123 2024-09-16T05:15:18.785 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 30 to 115 2024-09-16T05:15:18.785 INFO:tasks.rados.rados.2.smithi204.stdout:1122: finishing write tid 1 to smithi20457400-29 2024-09-16T05:15:18.785 INFO:tasks.rados.rados.2.smithi204.stdout:1124: read oid 35 snap -1 2024-09-16T05:15:18.785 INFO:tasks.rados.rados.2.smithi204.stdout:1124: expect deleted 2024-09-16T05:15:18.785 INFO:tasks.rados.rados.2.smithi204.stdout:1125: copy_from oid 50 from oid 7 current snap is 123 2024-09-16T05:15:18.785 INFO:tasks.rados.rados.2.smithi204.stdout:1126: write (excl) oid 32 current snap is 123 2024-09-16T05:15:18.785 INFO:tasks.rados.rados.2.smithi204.stdout:1126: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2024-09-16T05:15:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:18.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:18.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1126: writing smithi20457400-32 from 474063 to 1017613 tid 1 2024-09-16T05:15:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1126: writing smithi20457400-32 from 1650547 to 2164680 tid 2 2024-09-16T05:15:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1126: writing smithi20457400-32 from 2765651 to 2874063 tid 3 2024-09-16T05:15:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1127: read oid 37 snap -1 2024-09-16T05:15:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1127: expect deleted 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1128: rollback oid 10 current snap is 123 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 116 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1122: finishing write tid 2 to smithi20457400-29 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1122: finishing write tid 3 to smithi20457400-29 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1122: finishing write tid 4 to smithi20457400-29 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1122: finishing write tid 5 to smithi20457400-29 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 552 (ObjNum 408 snap 123 seq_num 408) dirty exists 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1122: left oid 29 (ObjNum 408 snap 123 seq_num 408) 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1123: finishing rollback tid 1 to smithi20457400-30 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 535 (ObjNum 129 snap 0 seq_num 4282499280) dirty dne 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1126: finishing write tid 1 to smithi20457400-32 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1122: done (6 left) 2024-09-16T05:15:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1123: done (5 left) 2024-09-16T05:15:18.790 INFO:tasks.rados.rados.2.smithi204.stdout:1124: done (4 left) 2024-09-16T05:15:18.790 INFO:tasks.rados.rados.2.smithi204.stdout:1129: read oid 29 snap -1 2024-09-16T05:15:18.790 INFO:tasks.rados.rados.2.smithi204.stdout:1129: expect (ObjNum 408 snap 123 seq_num 408) 2024-09-16T05:15:18.790 INFO:tasks.rados.rados.2.smithi204.stdout:1130: snap_remove snap 115 2024-09-16T05:15:18.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:18.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:18.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:18.982 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:18 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:15:18.983 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:18 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:15:18.983 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:18 smithi032 ceph-mon[27388]: osdmap e369: 12 total, 12 up, 12 in 2024-09-16T05:15:18.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:18 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:15:18.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:18 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:15:18.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:18 smithi196 ceph-mon[30802]: osdmap e369: 12 total, 12 up, 12 in 2024-09-16T05:15:19.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:19.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:19.117 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:19.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:19.163 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:19.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:18 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:15:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:18 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:15:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:18 smithi204 ceph-mon[30933]: osdmap e369: 12 total, 12 up, 12 in 2024-09-16T05:15:19.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:19.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:19.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:19.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:19.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:19.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:19.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:19.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:19.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:19.599 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:15:19.599 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:15:19.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:15:19.599 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:15:19.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.599 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:163 2.9 2:9423b7dc:::smithi20457400-41:head [setxattr _header (56) in=7b,truncate 2672280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 276.10264664599998, 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.176412596, 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 163 2024-09-16T05:15:19.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493745+0000", 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1899999999999998e-07 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493747+0000", 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1600000000000001e-06 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493751+0000", 2024-09-16T05:15:19.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8260000000000003e-06 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493754+0000", 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8779999999999998e-06 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494256+0000", 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050281900000000005 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494314+0000", 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7592e-05 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494393+0000", 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8799000000000002e-05 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494718+0000", 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00032548400000000002 2024-09-16T05:15:19.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670063+0000", 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17534508600000001 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670109+0000", 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5427999999999999e-05 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670157+0000", 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8105000000000001e-05 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:164 2.9 2:9423b7dc:::smithi20457400-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 276.102631411, 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17655259500000001, 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 164 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7000000000000001e-07 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493762+0000", 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0660000000000002e-06 2024-09-16T05:15:19.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493766+0000", 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.732e-06 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493769+0000", 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6799999999999999e-06 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494530+0000", 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076053399999999999 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494606+0000", 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6538999999999993e-05 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670197+0000", 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17559084699999999 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670253+0000", 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5893999999999997e-05 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670312+0000", 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9033000000000002e-05 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:163 2.9 e112/107 ondisk, result = 0)", 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 276.10047441799998, 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17425236199999999, 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.14e-07 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495918+0000", 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5519999999999999e-06 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.669955+0000", 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17403649900000001 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.669969+0000", 2024-09-16T05:15:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4064e-05 2024-09-16T05:15:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670051+0000", 2024-09-16T05:15:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2313999999999998e-05 2024-09-16T05:15:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670059+0000", 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1240000000000005e-06 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670169+0000", 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000109695 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:207 2.6 2:6be67188:::smithi20457400-46:head [write 1768111~402677] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 274.595326387, 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092996950000000002, 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 207 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001066+0000", 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4900000000000004e-07 2024-09-16T05:15:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001419+0000", 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035348299999999998 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001430+0000", 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0499000000000001e-05 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001435+0000", 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.22e-06 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001488+0000", 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3588999999999998e-05 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001556+0000", 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7358000000000001e-05 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001638+0000", 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2319999999999998e-05 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.002547+0000", 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090899100000000005 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010283+0000", 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077356680000000002 2024-09-16T05:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010323+0000", 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0389000000000002e-05 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010364+0000", 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1229000000000001e-05 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:407 2.9 2:96dc17ef:::smithi20457400-10:head [write 1515641~485017] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 261.92491222400002, 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095319989999999993, 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 407 2024-09-16T05:15:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671480+0000", 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6199999999999996e-07 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671908+0000", 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042804200000000001 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671917+0000", 2024-09-16T05:15:19.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6870000000000006e-06 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671923+0000", 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0499999999999997e-06 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671969+0000", 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5461999999999999e-05 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672023+0000", 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4215000000000002e-05 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672096+0000", 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2574e-05 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672934+0000", 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083847200000000005 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680951+0000", 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080164480000000007 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680984+0000", 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3399000000000003e-05 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.681011+0000", 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6787999999999998e-05 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.628 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:409 2.9 2:96dc17ef:::smithi20457400-10:head [setxattr _header (56) in=7b,truncate 3378174] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 261.917437275, 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059654601000000002, 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 409 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7899999999999999e-07 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678957+0000", 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1109999999999999e-06 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678961+0000", 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9639999999999997e-06 2024-09-16T05:15:19.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678969+0000", 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8839999999999994e-06 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.679871+0000", 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090153499999999997 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.679945+0000", 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4518999999999996e-05 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680024+0000", 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9295999999999994e-05 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680263+0000", 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000238994 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738545+0000", 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058281865000000002 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738576+0000", 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0349999999999999e-05 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738608+0000", 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2703999999999997e-05 2024-09-16T05:15:19.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:410 2.9 2:96dc17ef:::smithi20457400-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 261.91741750599999, 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059752424999999998, 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.635 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 410 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9400000000000001e-07 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678975+0000", 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1960000000000001e-06 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678979+0000", 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7479999999999999e-06 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678982+0000", 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8820000000000001e-06 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680163+0000", 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011814880000000001 2024-09-16T05:15:19.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680245+0000", 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1411999999999999e-05 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738635+0000", 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058390240000000003 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738674+0000", 2024-09-16T05:15:19.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9119999999999998e-05 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738726+0000", 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2045000000000001e-05 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:409 2.9 e126/107 ondisk, result = 0)", 2024-09-16T05:15:19.640 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 261.91550632899998, 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057729559, 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3799999999999999e-07 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680887+0000", 2024-09-16T05:15:19.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7689999999999999e-06 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736934+0000", 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.056047685999999999 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736942+0000", 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4479999999999997e-06 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738537+0000", 2024-09-16T05:15:19.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015953720000000001 2024-09-16T05:15:19.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738542+0000", 2024-09-16T05:15:19.644 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:19.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7190000000000001e-06 2024-09-16T05:15:19.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738614+0000", 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2427000000000003e-05 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:597 2.9 2:96dc17ef:::smithi20457400-10:head [copy-from ver 0] snapc 13=[13,12,11,f,e,b] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 245.793434754, 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010909965000000001, 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 597 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802959+0000", 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4540000000000001e-06 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802992+0000", 2024-09-16T05:15:19.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3025000000000003e-05 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803026+0000", 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3903999999999999e-05 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803041+0000", 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5299e-05 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803197+0000", 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000156389 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803202+0000", 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.758e-06 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805000+0000", 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017979859999999999 2024-09-16T05:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805054+0000", 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4249999999999997e-05 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813866+0000", 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088118999999999992 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:598 2.9 2:96dc17ef:::smithi20457400-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 245.793317589, 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010979131, 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 598 2024-09-16T05:15:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803074+0000", 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.47e-07 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803078+0000", 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1030000000000004e-06 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803090+0000", 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1875999999999999e-05 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803099+0000", 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2199999999999992e-06 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803245+0000", 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014687899999999999 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:15:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803249+0000", 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0520000000000002e-06 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805228+0000", 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019794729999999998 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805254+0000", 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6016e-05 2024-09-16T05:15:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813951+0000", 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086966319999999993 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813999+0000", 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8263999999999997e-05 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.814053+0000", 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3668999999999997e-05 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:772 2.9 2:94d33c56:::smithi20457400-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e159)", 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 228.94942847199999, 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092047459999999998, 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 772 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7100000000000002e-07 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646975+0000", 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1861999999999999e-05 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646984+0000", 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2550000000000003e-06 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646991+0000", 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6340000000000003e-06 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.647057+0000", 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6353000000000007e-05 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.647061+0000", 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2720000000000002e-06 2024-09-16T05:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.648424+0000", 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013634470000000001 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.648472+0000", 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8093999999999997e-05 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.656107+0000", 2024-09-16T05:15:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076354400000000003 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.656167+0000", 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9818000000000002e-05 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:979 2.4 2:2ace617f:::smithi20457400-21:head [copy-from ver 40] snapc 28=[28,27,25,24,23,22,21,20,1f,15,f] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 208.59852691899999, 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016837338, 2024-09-16T05:15:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 979 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997865+0000", 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8899999999999999e-07 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997872+0000", 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0639999999999996e-06 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997880+0000", 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4039999999999995e-06 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997886+0000", 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4600000000000002e-06 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998030+0000", 2024-09-16T05:15:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014470800000000001 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998036+0000", 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6620000000000002e-06 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000325+0000", 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022892450000000001 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000434+0000", 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010908699999999999 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006744+0000", 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063091240000000002 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014271+0000", 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0075269869999999997 2024-09-16T05:15:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014635+0000", 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036412499999999997 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014670+0000", 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5799000000000001e-05 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014701+0000", 2024-09-16T05:15:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0984e-05 2024-09-16T05:15:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:980 2.4 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:15:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 208.59849638399999, 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013613798, 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 980 2024-09-16T05:15:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997899+0000", 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7319999999999998e-06 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997902+0000", 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.819e-06 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997913+0000", 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.083e-05 2024-09-16T05:15:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997919+0000", 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5570000000000003e-06 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998065+0000", 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000146012 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998069+0000", 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8550000000000001e-06 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000716+0000", 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026463070000000001 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000762+0000", 2024-09-16T05:15:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6443999999999998e-05 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011363+0000", 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010600559000000001 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011432+0000", 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8955000000000006e-05 2024-09-16T05:15:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011508+0000", 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6728000000000001e-05 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1004 2.4 2:2ace617f:::smithi20457400-21:head [copy-from ver 113] snapc 29=[29,28,27,25,24,23,22,21,20,1f,15] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.84427261600001, 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012670931, 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1004 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752120+0000", 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.996e-06 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752132+0000", 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1389000000000001e-05 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752150+0000", 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8533999999999999e-05 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752159+0000", 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9959999999999999e-06 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752236+0000", 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6211000000000006e-05 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752247+0000", 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1127e-05 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753383+0000", 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011362830000000001 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753447+0000", 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4009000000000003e-05 2024-09-16T05:15:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.761264+0000", 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078170510000000002 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764658+0000", 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033938449999999999 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764722+0000", 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4177999999999994e-05 2024-09-16T05:15:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764759+0000", 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7363000000000001e-05 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764789+0000", 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9949000000000001e-05 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1005 2.4 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e181)", 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 206.84420821000001, 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011596623, 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1005 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0600000000000001e-07 2024-09-16T05:15:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752185+0000", 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2419999999999999e-06 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752190+0000", 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7049999999999998e-06 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752193+0000", 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2739999999999999e-06 2024-09-16T05:15:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752304+0000", 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110301 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752307+0000", 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6559999999999998e-06 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753639+0000", 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001331201 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753690+0000", 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1826999999999997e-05 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763643+0000", 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0099530030000000002 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763709+0000", 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5469999999999995e-05 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763779+0000", 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0537999999999994e-05 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1499 2.9 2:94d33c56:::smithi20457400-8:head [copy-from ver 285] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 169.90616440700001, 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016013924999999998, 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1499 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690228+0000", 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6419999999999999e-06 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690237+0000", 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8929999999999994e-06 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690270+0000", 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2679999999999999e-05 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690281+0000", 2024-09-16T05:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1446e-05 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690342+0000", 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0681999999999999e-05 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690408+0000", 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6173000000000005e-05 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.698590+0000", 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081814639999999994 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.704358+0000", 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057679389999999997 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706113+0000", 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001755097 2024-09-16T05:15:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706183+0000", 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9991000000000001e-05 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706241+0000", 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7917999999999997e-05 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1500 2.9 2:94d33c56:::smithi20457400-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 169.90608461900001, 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012373189, 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1500 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690307+0000", 2024-09-16T05:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7599999999999997e-07 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690310+0000", 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2610000000000002e-06 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690337+0000", 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6733999999999998e-05 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690344+0000", 2024-09-16T05:15:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2779999999999996e-06 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690524+0000", 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017976900000000001 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690560+0000", 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5762000000000001e-05 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702528+0000", 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011967976 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702599+0000", 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1429999999999999e-05 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702680+0000", 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0402999999999996e-05 2024-09-16T05:15:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:611 2.15 e366/273 2:a94680a3:::smithi20457400-13:66 v 366'446, mlcod=362'442)", 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.879560176, 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119602217, 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:15:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8099999999999999e-07 2024-09-16T05:15:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716837+0000", 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4909999999999996e-06 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835914+0000", 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119077768 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:15:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835922+0000", 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.994e-06 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835962+0000", 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9230000000000002e-05 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835979+0000", 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.787e-05 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.836402+0000", 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042242199999999999 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.836433+0000", 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1260999999999998e-05 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:15:19.706 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:15:19.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:19.708 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T05:15:19.708 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:15:19.709 INFO:tasks.rados.rados.2.smithi204.stdout:1130: done (5 left) 2024-09-16T05:15:19.709 INFO:tasks.rados.rados.2.smithi204.stdout:1131: write (excl) oid 38 current snap is 123 2024-09-16T05:15:19.709 INFO:tasks.rados.rados.2.smithi204.stdout:1131: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-09-16T05:15:19.715 INFO:tasks.rados.rados.2.smithi204.stdout:1131: writing smithi20457400-38 from 490870 to 1109669 tid 1 2024-09-16T05:15:19.723 INFO:tasks.rados.rados.2.smithi204.stdout:1131: writing smithi20457400-38 from 1592676 to 2366820 tid 2 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1131: writing smithi20457400-38 from 2793074 to 2890870 tid 3 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1132: read oid 25 snap -1 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1132: expect (ObjNum 346 snap 100 seq_num 346) 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1133: rollback oid 39 current snap is 123 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 39 to 98 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1125: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1125: finishing copy_from to smithi20457400-50 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 325 (ObjNum 405 snap 122 seq_num 405) dirty exists 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1126: finishing write tid 2 to smithi20457400-32 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1125: done (7 left) 2024-09-16T05:15:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1134: delete oid 11 current snap is 123 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:1128: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:1128: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:1128: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 481 (ObjNum 349 snap 101 seq_num 349) dirty exists 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:1126: finishing write tid 3 to smithi20457400-32 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:1126: finishing write tid 4 to smithi20457400-32 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:1126: finishing write tid 5 to smithi20457400-32 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 557 (ObjNum 409 snap 123 seq_num 409) dirty exists 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:1126: left oid 32 (ObjNum 409 snap 123 seq_num 409) 2024-09-16T05:15:19.728 INFO:tasks.rados.rados.2.smithi204.stdout:1129: expect (ObjNum 408 snap 123 seq_num 408) 2024-09-16T05:15:19.741 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:19.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:19.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:19.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: pgmap v714: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 220 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 886 KiB/s wr, 2 op/s 2024-09-16T05:15:19.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1826909153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:15:19.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:15:19.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:15:19.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2579930033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:15:19.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:19.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1826909153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:15:19.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2579930033' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:15:19.860 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:19 smithi032 ceph-mon[27388]: osdmap e370: 12 total, 12 up, 12 in 2024-09-16T05:15:19.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:19.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:19.893 INFO:tasks.rados.rados.2.smithi204.stdout:1131: finishing write tid 1 to smithi20457400-38 2024-09-16T05:15:19.893 INFO:tasks.rados.rados.2.smithi204.stdout:1131: finishing write tid 2 to smithi20457400-38 2024-09-16T05:15:19.893 INFO:tasks.rados.rados.2.smithi204.stdout:1131: finishing write tid 3 to smithi20457400-38 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:1131: finishing write tid 4 to smithi20457400-38 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:1131: finishing write tid 5 to smithi20457400-38 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 471 (ObjNum 410 snap 123 seq_num 410) dirty exists 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:1131: left oid 38 (ObjNum 410 snap 123 seq_num 410) 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:1133: finishing rollback tid 0 to smithi20457400-39 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:1133: finishing rollback tid 1 to smithi20457400-39 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:1133: finishing rollback tid 2 to smithi20457400-39 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 576 (ObjNum 321 snap 88 seq_num 321) dirty exists 2024-09-16T05:15:19.894 INFO:tasks.rados.rados.2.smithi204.stdout:1132: expect (ObjNum 346 snap 100 seq_num 346) 2024-09-16T05:15:19.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: pgmap v714: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 220 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 886 KiB/s wr, 2 op/s 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1826909153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2579930033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1826909153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2579930033' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:15:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:19 smithi196 ceph-mon[30802]: osdmap e370: 12 total, 12 up, 12 in 2024-09-16T05:15:19.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:19.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:19.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:20.019 INFO:tasks.rados.rados.2.smithi204.stdout:1126: done (7 left) 2024-09-16T05:15:20.019 INFO:tasks.rados.rados.2.smithi204.stdout:1127: done (6 left) 2024-09-16T05:15:20.019 INFO:tasks.rados.rados.2.smithi204.stdout:1128: done (5 left) 2024-09-16T05:15:20.019 INFO:tasks.rados.rados.2.smithi204.stdout:1129: done (4 left) 2024-09-16T05:15:20.019 INFO:tasks.rados.rados.2.smithi204.stdout:1131: done (3 left) 2024-09-16T05:15:20.019 INFO:tasks.rados.rados.2.smithi204.stdout:1132: done (2 left) 2024-09-16T05:15:20.019 INFO:tasks.rados.rados.2.smithi204.stdout:1133: done (1 left) 2024-09-16T05:15:20.019 INFO:tasks.rados.rados.2.smithi204.stdout:1134: done (0 left) 2024-09-16T05:15:20.020 INFO:tasks.rados.rados.2.smithi204.stdout:1135: write oid 28 current snap is 123 2024-09-16T05:15:20.020 INFO:tasks.rados.rados.2.smithi204.stdout:1135: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-16T05:15:20.024 INFO:tasks.rados.rados.2.smithi204.stdout:1135: writing smithi20457400-28 from 524484 to 1010134 tid 1 2024-09-16T05:15:20.030 INFO:tasks.rados.rados.2.smithi204.stdout:1135: writing smithi20457400-28 from 1425993 to 2087453 tid 2 2024-09-16T05:15:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:1135: writing smithi20457400-28 from 2847920 to 2924484 tid 3 2024-09-16T05:15:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:1136: rollback oid 32 current snap is 123 2024-09-16T05:15:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 122 2024-09-16T05:15:20.035 INFO:tasks.rados.rados.2.smithi204.stdout:1135: finishing write tid 1 to smithi20457400-28 2024-09-16T05:15:20.035 INFO:tasks.rados.rados.2.smithi204.stdout:1135: finishing write tid 2 to smithi20457400-28 2024-09-16T05:15:20.035 INFO:tasks.rados.rados.2.smithi204.stdout:1137: snap_remove snap 117 2024-09-16T05:15:20.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:20.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:20.121 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:20.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: pgmap v714: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 220 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 886 KiB/s wr, 2 op/s 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1826909153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2579930033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1826909153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2579930033' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:15:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:19 smithi204 ceph-mon[30933]: osdmap e370: 12 total, 12 up, 12 in 2024-09-16T05:15:20.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:20.244 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:20.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:20.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:20.370 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:20.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:20.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:20.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:20.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:20.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:20.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:20.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:20.706 INFO:tasks.rados.rados.2.smithi204.stdout:1137: done (2 left) 2024-09-16T05:15:20.706 INFO:tasks.rados.rados.2.smithi204.stdout:1138: snap_remove snap 118 2024-09-16T05:15:20.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:20.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:20.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:20.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:20.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:20.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:20.995 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:20.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:21.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:21.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:21.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:21.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:21.247 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:21.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:21.350 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:21.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:21.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:21.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:21.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:21.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:21.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:21.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:21.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:21.706 INFO:tasks.rados.rados.2.smithi204.stdout:1138: done (2 left) 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1139: copy_from oid 31 from oid 8 current snap is 123 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1140: copy_from oid 5 from oid 46 current snap is 123 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1141: delete oid 15 current snap is 123 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1135: finishing write tid 3 to smithi20457400-28 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1135: finishing write tid 4 to smithi20457400-28 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1136: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1135: finishing write tid 5 to smithi20457400-28 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 361 (ObjNum 412 snap 123 seq_num 412) dirty exists 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1135: left oid 28 (ObjNum 412 snap 123 seq_num 412) 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1136: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:15:21.707 INFO:tasks.rados.rados.2.smithi204.stdout:1136: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:15:21.708 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 564 (ObjNum 371 snap 108 seq_num 371) dirty exists 2024-09-16T05:15:21.713 INFO:tasks.rados.rados.2.smithi204.stdout:1135: done (4 left) 2024-09-16T05:15:21.713 INFO:tasks.rados.rados.2.smithi204.stdout:1136: done (3 left) 2024-09-16T05:15:21.713 INFO:tasks.rados.rados.2.smithi204.stdout:1141: done (2 left) 2024-09-16T05:15:21.713 INFO:tasks.rados.rados.2.smithi204.stdout:1142: read oid 36 snap -1 2024-09-16T05:15:21.713 INFO:tasks.rados.rados.2.smithi204.stdout:1142: expect (ObjNum 349 snap 101 seq_num 349) 2024-09-16T05:15:21.713 INFO:tasks.rados.rados.2.smithi204.stdout:1143: write (excl) oid 6 current snap is 123 2024-09-16T05:15:21.713 INFO:tasks.rados.rados.2.smithi204.stdout:1143: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-16T05:15:21.721 INFO:tasks.rados.rados.2.smithi204.stdout:1143: writing smithi20457400-6 from 558098 to 1194246 tid 1 2024-09-16T05:15:21.727 INFO:tasks.rados.rados.2.smithi204.stdout:1143: writing smithi20457400-6 from 1826604 to 2491733 tid 2 2024-09-16T05:15:21.727 INFO:tasks.rados.rados.2.smithi204.stdout:1144: write oid 14 current snap is 123 2024-09-16T05:15:21.728 INFO:tasks.rados.rados.2.smithi204.stdout:1144: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-16T05:15:21.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:21.734 INFO:tasks.rados.rados.2.smithi204.stdout:1144: writing smithi20457400-14 from 574905 to 1286302 tid 1 2024-09-16T05:15:21.740 INFO:tasks.rados.rados.2.smithi204.stdout:1144: writing smithi20457400-14 from 1885086 to 2410226 tid 2 2024-09-16T05:15:21.744 INFO:tasks.rados.rados.2.smithi204.stdout:1144: writing smithi20457400-14 from 2930189 to 2974905 tid 3 2024-09-16T05:15:21.744 INFO:tasks.rados.rados.2.smithi204.stdout:1145: write oid 13 current snap is 123 2024-09-16T05:15:21.744 INFO:tasks.rados.rados.2.smithi204.stdout:1143: finishing write tid 1 to smithi20457400-6 2024-09-16T05:15:21.744 INFO:tasks.rados.rados.2.smithi204.stdout:1145: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-16T05:15:21.751 INFO:tasks.rados.rados.2.smithi204.stdout:1145: writing smithi20457400-13 from 591712 to 1378358 tid 1 2024-09-16T05:15:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:21.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:21.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:21.758 INFO:tasks.rados.rados.2.smithi204.stdout:1145: writing smithi20457400-13 from 1943568 to 2612366 tid 2 2024-09-16T05:15:21.758 INFO:tasks.rados.rados.2.smithi204.stdout:1146: snap_create 2024-09-16T05:15:21.758 INFO:tasks.rados.rados.2.smithi204.stdout:1142: expect (ObjNum 349 snap 101 seq_num 349) 2024-09-16T05:15:21.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:15:21.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:21.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:21.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:21.884 INFO:tasks.rados.rados.2.smithi204.stdout:1140: finishing copy_from racing read to smithi20457400-5 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:1139: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:1143: finishing write tid 2 to smithi20457400-6 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:1139: finishing copy_from to smithi20457400-31 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 336 (ObjNum 374 snap 109 seq_num 374) dirty exists 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:1143: finishing write tid 3 to smithi20457400-6 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:1140: finishing copy_from to smithi20457400-5 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 322 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:1143: finishing write tid 4 to smithi20457400-6 2024-09-16T05:15:21.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 329 (ObjNum 414 snap 123 seq_num 414) dirty exists 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1143: left oid 6 (ObjNum 414 snap 123 seq_num 414) 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1144: finishing write tid 1 to smithi20457400-14 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1144: finishing write tid 2 to smithi20457400-14 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1144: finishing write tid 3 to smithi20457400-14 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1144: finishing write tid 4 to smithi20457400-14 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1144: finishing write tid 5 to smithi20457400-14 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 597 (ObjNum 415 snap 123 seq_num 415) dirty exists 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1144: left oid 14 (ObjNum 415 snap 123 seq_num 415) 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1145: finishing write tid 1 to smithi20457400-13 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1145: finishing write tid 2 to smithi20457400-13 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1145: finishing write tid 3 to smithi20457400-13 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:1145: finishing write tid 4 to smithi20457400-13 2024-09-16T05:15:21.886 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 456 (ObjNum 416 snap 123 seq_num 416) dirty exists 2024-09-16T05:15:21.887 INFO:tasks.rados.rados.2.smithi204.stdout:1145: left oid 13 (ObjNum 416 snap 123 seq_num 416) 2024-09-16T05:15:21.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:21.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:21 smithi196 ceph-mon[30802]: pgmap v717: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 224 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-16T05:15:21.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:21 smithi196 ceph-mon[30802]: osdmap e371: 12 total, 12 up, 12 in 2024-09-16T05:15:22.006 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:22.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:22.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:21 smithi032 ceph-mon[27388]: pgmap v717: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 224 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-16T05:15:22.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:21 smithi032 ceph-mon[27388]: osdmap e371: 12 total, 12 up, 12 in 2024-09-16T05:15:22.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:22.129 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:22.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:21 smithi204 ceph-mon[30933]: pgmap v717: 87 pgs: 1 active+clean+snaptrim, 86 active+clean; 224 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-16T05:15:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:21 smithi204 ceph-mon[30933]: osdmap e371: 12 total, 12 up, 12 in 2024-09-16T05:15:22.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:22.252 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:22.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:22.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:22.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:22.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:22.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:22.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-16T05:15:22.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:22.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:22.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:22.554 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:22.554 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:22.554 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:22.554 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:22.554 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:22.556 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:22.556 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:22.556 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:22.556 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:22.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:22.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:22.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:22.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:22.715 INFO:tasks.rados.rados.2.smithi204.stdout:1139: done (6 left) 2024-09-16T05:15:22.715 INFO:tasks.rados.rados.2.smithi204.stdout:1140: done (5 left) 2024-09-16T05:15:22.715 INFO:tasks.rados.rados.2.smithi204.stdout:1142: done (4 left) 2024-09-16T05:15:22.715 INFO:tasks.rados.rados.2.smithi204.stdout:1143: done (3 left) 2024-09-16T05:15:22.715 INFO:tasks.rados.rados.2.smithi204.stdout:1144: done (2 left) 2024-09-16T05:15:22.716 INFO:tasks.rados.rados.2.smithi204.stdout:1145: done (1 left) 2024-09-16T05:15:22.716 INFO:tasks.rados.rados.2.smithi204.stdout:1146: done (0 left) 2024-09-16T05:15:22.716 INFO:tasks.rados.rados.2.smithi204.stdout:1147: write (excl) oid 13 current snap is 124 2024-09-16T05:15:22.716 INFO:tasks.rados.rados.2.smithi204.stdout:1147: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-16T05:15:22.721 INFO:tasks.rados.rados.2.smithi204.stdout:1147: writing smithi20457400-13 from 608519 to 1070414 tid 1 2024-09-16T05:15:22.727 INFO:tasks.rados.rados.2.smithi204.stdout:1147: writing smithi20457400-13 from 1602050 to 2130859 tid 2 2024-09-16T05:15:22.732 INFO:tasks.rados.rados.2.smithi204.stdout:1147: writing smithi20457400-13 from 2585035 to 3008519 tid 3 2024-09-16T05:15:22.732 INFO:tasks.rados.rados.2.smithi204.stdout:1148: snap_remove snap 116 2024-09-16T05:15:22.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:22.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:22.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:22.841 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:22 smithi032 ceph-mon[27388]: osdmap e372: 12 total, 12 up, 12 in 2024-09-16T05:15:22.842 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2884736912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:22.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:22.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:22.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:22.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:22 smithi196 ceph-mon[30802]: osdmap e372: 12 total, 12 up, 12 in 2024-09-16T05:15:22.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2884736912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:23.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:23.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:23.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:23.136 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:23.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:22 smithi204 ceph-mon[30933]: osdmap e372: 12 total, 12 up, 12 in 2024-09-16T05:15:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2884736912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:23.211 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:23.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:23.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:15:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:23.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:23.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:23.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:23.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:23.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:23.492 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:23.518 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:23.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:23.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:23.643 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:23.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:23.707 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:23.707 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:23.718 INFO:tasks.rados.rados.2.smithi204.stdout:1148: done (1 left) 2024-09-16T05:15:23.718 INFO:tasks.rados.rados.2.smithi204.stdout:1149: write oid 18 current snap is 124 2024-09-16T05:15:23.718 INFO:tasks.rados.rados.2.smithi204.stdout:1149: seq_num 418 ranges {625326=537144,1944179=672467} 2024-09-16T05:15:23.725 INFO:tasks.rados.rados.2.smithi204.stdout:1149: writing smithi20457400-18 from 625326 to 1162470 tid 1 2024-09-16T05:15:23.733 INFO:tasks.rados.rados.2.smithi204.stdout:1149: writing smithi20457400-18 from 1944179 to 2616646 tid 2 2024-09-16T05:15:23.733 INFO:tasks.rados.rados.2.smithi204.stdout:1150: write (excl) oid 36 current snap is 124 2024-09-16T05:15:23.733 INFO:tasks.rados.rados.2.smithi204.stdout:1150: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-16T05:15:23.740 INFO:tasks.rados.rados.2.smithi204.stdout:1150: writing smithi20457400-36 from 642133 to 1370879 tid 1 2024-09-16T05:15:23.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:23.748 INFO:tasks.rados.rados.2.smithi204.stdout:1150: writing smithi20457400-36 from 2119014 to 2651492 tid 2 2024-09-16T05:15:23.748 INFO:tasks.rados.rados.2.smithi204.stdout:1151: read oid 44 snap 123 2024-09-16T05:15:23.748 INFO:tasks.rados.rados.2.smithi204.stdout:1151: expect deleted 2024-09-16T05:15:23.748 INFO:tasks.rados.rados.2.smithi204.stdout:1152: rollback oid 11 current snap is 124 2024-09-16T05:15:23.748 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 11 to 121 2024-09-16T05:15:23.748 INFO:tasks.rados.rados.2.smithi204.stdout:1147: finishing write tid 1 to smithi20457400-13 2024-09-16T05:15:23.748 INFO:tasks.rados.rados.2.smithi204.stdout:1153: rollback oid 34 current snap is 124 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 120 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1147: finishing write tid 2 to smithi20457400-13 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1147: finishing write tid 3 to smithi20457400-13 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1147: finishing write tid 4 to smithi20457400-13 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1147: finishing write tid 5 to smithi20457400-13 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 461 (ObjNum 417 snap 124 seq_num 417) dirty exists 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1147: left oid 13 (ObjNum 417 snap 124 seq_num 417) 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1149: finishing write tid 1 to smithi20457400-18 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1149: finishing write tid 2 to smithi20457400-18 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1149: finishing write tid 3 to smithi20457400-18 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1149: finishing write tid 4 to smithi20457400-18 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 220 (ObjNum 418 snap 124 seq_num 418) dirty exists 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1149: left oid 18 (ObjNum 418 snap 124 seq_num 418) 2024-09-16T05:15:23.749 INFO:tasks.rados.rados.2.smithi204.stdout:1150: finishing write tid 1 to smithi20457400-36 2024-09-16T05:15:23.750 INFO:tasks.rados.rados.2.smithi204.stdout:1147: done (5 left) 2024-09-16T05:15:23.750 INFO:tasks.rados.rados.2.smithi204.stdout:1149: done (4 left) 2024-09-16T05:15:23.750 INFO:tasks.rados.rados.2.smithi204.stdout:1154: write oid 41 current snap is 124 2024-09-16T05:15:23.750 INFO:tasks.rados.rados.2.smithi204.stdout:1154: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-16T05:15:23.754 INFO:tasks.rados.rados.2.smithi204.stdout:1154: writing smithi20457400-41 from 658940 to 1062935 tid 1 2024-09-16T05:15:23.764 INFO:tasks.rados.rados.2.smithi204.stdout:1154: writing smithi20457400-41 from 1777496 to 2453632 tid 2 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:1154: writing smithi20457400-41 from 2950951 to 3058940 tid 3 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:1155: copy_from oid 35 from oid 50 current snap is 124 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:1150: finishing write tid 2 to smithi20457400-36 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:1153: finishing rollback tid 0 to smithi20457400-34 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:1150: finishing write tid 3 to smithi20457400-36 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:1150: finishing write tid 4 to smithi20457400-36 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 451 (ObjNum 419 snap 124 seq_num 419) dirty exists 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:1150: left oid 36 (ObjNum 419 snap 124 seq_num 419) 2024-09-16T05:15:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:1150: done (5 left) 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:1156: delete oid 18 current snap is 124 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:1153: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:1153: finishing rollback tid 2 to smithi20457400-34 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 456 (ObjNum 393 snap 118 seq_num 393) dirty exists 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:1152: finishing rollback tid 1 to smithi20457400-11 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:1152: finishing rollback tid 2 to smithi20457400-11 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 592 (ObjNum 220 snap 57 seq_num 220) dirty exists 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:1154: finishing write tid 1 to smithi20457400-41 2024-09-16T05:15:23.767 INFO:tasks.rados.rados.2.smithi204.stdout:1154: finishing write tid 2 to smithi20457400-41 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1151: done (5 left) 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1152: done (4 left) 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1153: done (3 left) 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1156: done (2 left) 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1157: read oid 6 snap -1 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1157: expect (ObjNum 414 snap 123 seq_num 414) 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1158: read oid 7 snap -1 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1158: expect (ObjNum 405 snap 122 seq_num 405) 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1159: read oid 12 snap -1 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1159: expect (ObjNum 375 snap 109 seq_num 375) 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:1160: rollback oid 27 current snap is 124 2024-09-16T05:15:23.769 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 27 to 122 2024-09-16T05:15:23.770 INFO:tasks.rados.rados.2.smithi204.stdout:1161: snap_remove snap 98 2024-09-16T05:15:23.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:23.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:23.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:23.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:23.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:23.933 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:23.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:23 smithi196 ceph-mon[30802]: pgmap v720: 87 pgs: 87 active+clean; 235 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2024-09-16T05:15:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:23 smithi196 ceph-mon[30802]: osdmap e373: 12 total, 12 up, 12 in 2024-09-16T05:15:24.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:24.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:24.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:24.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:23 smithi032 ceph-mon[27388]: pgmap v720: 87 pgs: 87 active+clean; 235 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2024-09-16T05:15:24.055 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:23 smithi032 ceph-mon[27388]: osdmap e373: 12 total, 12 up, 12 in 2024-09-16T05:15:24.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:24.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:24.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:24.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:23 smithi204 ceph-mon[30933]: pgmap v720: 87 pgs: 87 active+clean; 235 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2024-09-16T05:15:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:23 smithi204 ceph-mon[30933]: osdmap e373: 12 total, 12 up, 12 in 2024-09-16T05:15:24.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:24.276 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:24.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:24.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:24.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:24.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:24.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:24.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:24.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:24.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:24.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:24.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:24.730 INFO:tasks.rados.rados.2.smithi204.stdout:1161: done (6 left) 2024-09-16T05:15:24.730 INFO:tasks.rados.rados.2.smithi204.stdout:1162: write (excl) oid 30 current snap is 124 2024-09-16T05:15:24.730 INFO:tasks.rados.rados.2.smithi204.stdout:1162: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-09-16T05:15:24.737 INFO:tasks.rados.rados.2.smithi204.stdout:1162: writing smithi20457400-30 from 692554 to 1247047 tid 1 2024-09-16T05:15:24.744 INFO:tasks.rados.rados.2.smithi204.stdout:1162: writing smithi20457400-30 from 1778107 to 2457912 tid 2 2024-09-16T05:15:24.747 INFO:tasks.rados.rados.2.smithi204.stdout:1162: writing smithi20457400-30 from 2889444 to 3092554 tid 3 2024-09-16T05:15:24.747 INFO:tasks.rados.rados.2.smithi204.stdout:1163: snap_remove snap 106 2024-09-16T05:15:24.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:24.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:24.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:24.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:24.908 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:24.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:24.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:24 smithi196 ceph-mon[30802]: osdmap e374: 12 total, 12 up, 12 in 2024-09-16T05:15:25.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:25.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:25.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:25.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:24 smithi032 ceph-mon[27388]: osdmap e374: 12 total, 12 up, 12 in 2024-09-16T05:15:25.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:25.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:25.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:24 smithi204 ceph-mon[30933]: osdmap e374: 12 total, 12 up, 12 in 2024-09-16T05:15:25.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:15:25.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:25.284 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:25.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:25.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:15:25.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:25.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:25.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:25.463 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:25.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:25.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:25.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:25.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:25.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:25.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:25.703 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T05:15:25.703 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:15:25.706 INFO:tasks.rados.rados.2.smithi204.stdout:1163: done (7 left) 2024-09-16T05:15:25.706 INFO:tasks.rados.rados.2.smithi204.stdout:1164: write (excl) oid 15 current snap is 124 2024-09-16T05:15:25.706 INFO:tasks.rados.rados.2.smithi204.stdout:1154: finishing write tid 3 to smithi20457400-41 2024-09-16T05:15:25.707 INFO:tasks.rados.rados.2.smithi204.stdout:1154: finishing write tid 4 to smithi20457400-41 2024-09-16T05:15:25.707 INFO:tasks.rados.rados.2.smithi204.stdout:1154: finishing write tid 5 to smithi20457400-41 2024-09-16T05:15:25.707 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 504 (ObjNum 420 snap 124 seq_num 420) dirty exists 2024-09-16T05:15:25.707 INFO:tasks.rados.rados.2.smithi204.stdout:1154: left oid 41 (ObjNum 420 snap 124 seq_num 420) 2024-09-16T05:15:25.707 INFO:tasks.rados.rados.2.smithi204.stdout:1160: finishing rollback tid 0 to smithi20457400-27 2024-09-16T05:15:25.707 INFO:tasks.rados.rados.2.smithi204.stdout:1158: expect (ObjNum 405 snap 122 seq_num 405) 2024-09-16T05:15:25.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:25.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:25.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:25.807 INFO:tasks.rados.rados.2.smithi204.stdout:1159: expect (ObjNum 375 snap 109 seq_num 375) 2024-09-16T05:15:25.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:25.910 INFO:tasks.rados.rados.2.smithi204.stdout:1155: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:15:25.910 INFO:tasks.rados.rados.2.smithi204.stdout:1160: finishing rollback tid 1 to smithi20457400-27 2024-09-16T05:15:25.910 INFO:tasks.rados.rados.2.smithi204.stdout:1155: finishing copy_from to smithi20457400-35 2024-09-16T05:15:25.910 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 325 (ObjNum 405 snap 122 seq_num 405) dirty exists 2024-09-16T05:15:25.910 INFO:tasks.rados.rados.2.smithi204.stdout:1157: expect (ObjNum 414 snap 123 seq_num 414) 2024-09-16T05:15:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:25.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:25 smithi032 ceph-mon[27388]: pgmap v723: 87 pgs: 87 active+clean; 244 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2024-09-16T05:15:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:25 smithi032 ceph-mon[27388]: osdmap e375: 12 total, 12 up, 12 in 2024-09-16T05:15:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1219940591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/67489590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:15:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:25 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1219940591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/67489590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:15:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:25 smithi032 ceph-mon[27388]: osdmap e376: 12 total, 12 up, 12 in 2024-09-16T05:15:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:25 smithi196 ceph-mon[30802]: pgmap v723: 87 pgs: 87 active+clean; 244 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2024-09-16T05:15:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:25 smithi196 ceph-mon[30802]: osdmap e375: 12 total, 12 up, 12 in 2024-09-16T05:15:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1219940591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/67489590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:15:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:25 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1219940591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/67489590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:15:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:25 smithi196 ceph-mon[30802]: osdmap e376: 12 total, 12 up, 12 in 2024-09-16T05:15:26.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:26.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:26.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:26.086 INFO:tasks.rados.rados.2.smithi204.stdout:1160: finishing rollback tid 2 to smithi20457400-27 2024-09-16T05:15:26.086 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 293 (ObjNum 404 snap 122 seq_num 404) dirty exists 2024-09-16T05:15:26.086 INFO:tasks.rados.rados.2.smithi204.stdout:1162: finishing write tid 1 to smithi20457400-30 2024-09-16T05:15:26.087 INFO:tasks.rados.rados.2.smithi204.stdout:1162: finishing write tid 2 to smithi20457400-30 2024-09-16T05:15:26.087 INFO:tasks.rados.rados.2.smithi204.stdout:1162: finishing write tid 3 to smithi20457400-30 2024-09-16T05:15:26.087 INFO:tasks.rados.rados.2.smithi204.stdout:1162: finishing write tid 4 to smithi20457400-30 2024-09-16T05:15:26.087 INFO:tasks.rados.rados.2.smithi204.stdout:1162: finishing write tid 5 to smithi20457400-30 2024-09-16T05:15:26.087 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 611 (ObjNum 422 snap 124 seq_num 422) dirty exists 2024-09-16T05:15:26.087 INFO:tasks.rados.rados.2.smithi204.stdout:1162: left oid 30 (ObjNum 422 snap 124 seq_num 422) 2024-09-16T05:15:26.087 INFO:tasks.rados.rados.2.smithi204.stdout:1164: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-16T05:15:26.093 INFO:tasks.rados.rados.2.smithi204.stdout:1164: writing smithi20457400-15 from 709361 to 1339103 tid 1 2024-09-16T05:15:26.099 INFO:tasks.rados.rados.2.smithi204.stdout:1164: writing smithi20457400-15 from 1836589 to 2376405 tid 2 2024-09-16T05:15:26.103 INFO:tasks.rados.rados.2.smithi204.stdout:1164: writing smithi20457400-15 from 3033220 to 3109361 tid 3 2024-09-16T05:15:26.103 INFO:tasks.rados.rados.2.smithi204.stdout:1154: done (7 left) 2024-09-16T05:15:26.103 INFO:tasks.rados.rados.2.smithi204.stdout:1155: done (6 left) 2024-09-16T05:15:26.103 INFO:tasks.rados.rados.2.smithi204.stdout:1157: done (5 left) 2024-09-16T05:15:26.103 INFO:tasks.rados.rados.2.smithi204.stdout:1158: done (4 left) 2024-09-16T05:15:26.103 INFO:tasks.rados.rados.2.smithi204.stdout:1159: done (3 left) 2024-09-16T05:15:26.104 INFO:tasks.rados.rados.2.smithi204.stdout:1160: done (2 left) 2024-09-16T05:15:26.104 INFO:tasks.rados.rados.2.smithi204.stdout:1162: done (1 left) 2024-09-16T05:15:26.104 INFO:tasks.rados.rados.2.smithi204.stdout:1165: delete oid 39 current snap is 124 2024-09-16T05:15:26.104 INFO:tasks.rados.rados.2.smithi204.stdout:1164: finishing write tid 1 to smithi20457400-15 2024-09-16T05:15:26.104 INFO:tasks.rados.rados.2.smithi204.stdout:1164: finishing write tid 2 to smithi20457400-15 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:1164: finishing write tid 3 to smithi20457400-15 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:1164: finishing write tid 4 to smithi20457400-15 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:1165: done (1 left) 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:1166: read oid 2 snap -1 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:1166: expect (ObjNum 388 snap 116 seq_num 388) 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:1167: delete oid 25 current snap is 124 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:1164: finishing write tid 5 to smithi20457400-15 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 119 (ObjNum 423 snap 124 seq_num 423) dirty exists 2024-09-16T05:15:26.105 INFO:tasks.rados.rados.2.smithi204.stdout:1164: left oid 15 (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:15:26.106 INFO:tasks.rados.rados.2.smithi204.stdout:1164: done (2 left) 2024-09-16T05:15:26.106 INFO:tasks.rados.rados.2.smithi204.stdout:1167: done (1 left) 2024-09-16T05:15:26.107 INFO:tasks.rados.rados.2.smithi204.stdout:1168: read oid 48 snap 101 2024-09-16T05:15:26.107 INFO:tasks.rados.rados.2.smithi204.stdout:1168: expect deleted 2024-09-16T05:15:26.107 INFO:tasks.rados.rados.2.smithi204.stdout:1169: write oid 37 current snap is 124 2024-09-16T05:15:26.107 INFO:tasks.rados.rados.2.smithi204.stdout:1169: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-16T05:15:26.113 INFO:tasks.rados.rados.2.smithi204.stdout:1169: writing smithi20457400-37 from 759782 to 1331624 tid 1 2024-09-16T05:15:26.121 INFO:tasks.rados.rados.2.smithi204.stdout:1169: writing smithi20457400-37 from 2012035 to 2699178 tid 2 2024-09-16T05:15:26.121 INFO:tasks.rados.rados.2.smithi204.stdout:1170: delete oid 38 current snap is 124 2024-09-16T05:15:26.121 INFO:tasks.rados.rados.2.smithi204.stdout:1166: expect (ObjNum 388 snap 116 seq_num 388) 2024-09-16T05:15:26.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:26.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:26.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:25 smithi204 ceph-mon[30933]: pgmap v723: 87 pgs: 87 active+clean; 244 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2024-09-16T05:15:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:25 smithi204 ceph-mon[30933]: osdmap e375: 12 total, 12 up, 12 in 2024-09-16T05:15:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1219940591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/67489590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:15:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:25 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1219940591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/67489590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:15:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:25 smithi204 ceph-mon[30933]: osdmap e376: 12 total, 12 up, 12 in 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1169: finishing write tid 1 to smithi20457400-37 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1166: done (3 left) 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1168: done (2 left) 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1170: done (1 left) 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1171: write oid 16 current snap is 124 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1169: finishing write tid 2 to smithi20457400-37 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1169: finishing write tid 3 to smithi20457400-37 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1169: finishing write tid 4 to smithi20457400-37 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 320 (ObjNum 426 snap 124 seq_num 426) dirty exists 2024-09-16T05:15:26.228 INFO:tasks.rados.rados.2.smithi204.stdout:1169: left oid 37 (ObjNum 426 snap 124 seq_num 426) 2024-09-16T05:15:26.229 INFO:tasks.rados.rados.2.smithi204.stdout:1171: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-16T05:15:26.235 INFO:tasks.rados.rados.2.smithi204.stdout:1171: writing smithi20457400-16 from 793396 to 1515736 tid 1 2024-09-16T05:15:26.243 INFO:tasks.rados.rados.2.smithi204.stdout:1171: writing smithi20457400-16 from 2128999 to 2819811 tid 2 2024-09-16T05:15:26.243 INFO:tasks.rados.rados.2.smithi204.stdout:1169: done (1 left) 2024-09-16T05:15:26.243 INFO:tasks.rados.rados.2.smithi204.stdout:1172: write oid 34 current snap is 124 2024-09-16T05:15:26.243 INFO:tasks.rados.rados.2.smithi204.stdout:1172: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-16T05:15:26.249 INFO:tasks.rados.rados.2.smithi204.stdout:1172: writing smithi20457400-34 from 410203 to 1207792 tid 1 2024-09-16T05:15:26.255 INFO:tasks.rados.rados.2.smithi204.stdout:1172: writing smithi20457400-34 from 1787481 to 2221951 tid 2 2024-09-16T05:15:26.260 INFO:tasks.rados.rados.2.smithi204.stdout:1172: writing smithi20457400-34 from 2681405 to 3210203 tid 3 2024-09-16T05:15:26.260 INFO:tasks.rados.rados.2.smithi204.stdout:1173: read oid 48 snap -1 2024-09-16T05:15:26.261 INFO:tasks.rados.rados.2.smithi204.stdout:1173: expect deleted 2024-09-16T05:15:26.261 INFO:tasks.rados.rados.2.smithi204.stdout:1174: write oid 2 current snap is 124 2024-09-16T05:15:26.261 INFO:tasks.rados.rados.2.smithi204.stdout:1174: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-16T05:15:26.265 INFO:tasks.rados.rados.2.smithi204.stdout:1174: writing smithi20457400-2 from 427010 to 899848 tid 1 2024-09-16T05:15:26.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:26.271 INFO:tasks.rados.rados.2.smithi204.stdout:1174: writing smithi20457400-2 from 1329610 to 2024091 tid 2 2024-09-16T05:15:26.278 INFO:tasks.rados.rados.2.smithi204.stdout:1174: writing smithi20457400-2 from 2708828 to 3227010 tid 3 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1175: read oid 12 snap -1 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1175: expect (ObjNum 375 snap 109 seq_num 375) 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1176: copy_from oid 7 from oid 19 current snap is 124 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1171: finishing write tid 1 to smithi20457400-16 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1171: finishing write tid 2 to smithi20457400-16 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1171: finishing write tid 3 to smithi20457400-16 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1171: finishing write tid 4 to smithi20457400-16 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 493 (ObjNum 428 snap 124 seq_num 428) dirty exists 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1171: left oid 16 (ObjNum 428 snap 124 seq_num 428) 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1172: finishing write tid 1 to smithi20457400-34 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1172: finishing write tid 2 to smithi20457400-34 2024-09-16T05:15:26.279 INFO:tasks.rados.rados.2.smithi204.stdout:1171: done (5 left) 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1177: copy_from oid 30 from oid 6 current snap is 124 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1172: finishing write tid 3 to smithi20457400-34 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1172: finishing write tid 4 to smithi20457400-34 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1172: finishing write tid 5 to smithi20457400-34 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 460 (ObjNum 429 snap 124 seq_num 429) dirty exists 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1172: left oid 34 (ObjNum 429 snap 124 seq_num 429) 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1174: finishing write tid 1 to smithi20457400-2 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1174: finishing write tid 2 to smithi20457400-2 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1172: done (5 left) 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1173: done (4 left) 2024-09-16T05:15:26.280 INFO:tasks.rados.rados.2.smithi204.stdout:1178: delete oid 3 current snap is 124 2024-09-16T05:15:26.281 INFO:tasks.rados.rados.2.smithi204.stdout:1174: finishing write tid 3 to smithi20457400-2 2024-09-16T05:15:26.281 INFO:tasks.rados.rados.2.smithi204.stdout:1175: expect (ObjNum 375 snap 109 seq_num 375) 2024-09-16T05:15:26.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:26.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:26.385 INFO:tasks.rados.rados.2.smithi204.stdout:1174: finishing write tid 4 to smithi20457400-2 2024-09-16T05:15:26.385 INFO:tasks.rados.rados.2.smithi204.stdout:1174: finishing write tid 5 to smithi20457400-2 2024-09-16T05:15:26.385 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 169 (ObjNum 430 snap 124 seq_num 430) dirty exists 2024-09-16T05:15:26.385 INFO:tasks.rados.rados.2.smithi204.stdout:1174: left oid 2 (ObjNum 430 snap 124 seq_num 430) 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1176: finishing copy_from to smithi20457400-7 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1176: got expected ENOENT (src dne) 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1176: finishing copy_from racing read to smithi20457400-7 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1177: finishing copy_from racing read to smithi20457400-30 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1174: done (4 left) 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1175: done (3 left) 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1176: done (2 left) 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1178: done (1 left) 2024-09-16T05:15:26.386 INFO:tasks.rados.rados.2.smithi204.stdout:1179: snap_remove snap 120 2024-09-16T05:15:26.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:26.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:26.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:26.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:26.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:26.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:26.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-16T05:15:26.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:26.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:26.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:26.738 INFO:tasks.rados.rados.2.smithi204.stdout:1179: done (1 left) 2024-09-16T05:15:26.739 INFO:tasks.rados.rados.2.smithi204.stdout:1180: read oid 1 snap -1 2024-09-16T05:15:26.739 INFO:tasks.rados.rados.2.smithi204.stdout:1180: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:15:26.739 INFO:tasks.rados.rados.2.smithi204.stdout:1181: read oid 2 snap 123 2024-09-16T05:15:26.739 INFO:tasks.rados.rados.2.smithi204.stdout:1181: expect (ObjNum 388 snap 116 seq_num 388) 2024-09-16T05:15:26.739 INFO:tasks.rados.rados.2.smithi204.stdout:1182: write (excl) oid 25 current snap is 124 2024-09-16T05:15:26.739 INFO:tasks.rados.rados.2.smithi204.stdout:1182: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-16T05:15:26.746 INFO:tasks.rados.rados.2.smithi204.stdout:1182: writing smithi20457400-25 from 460624 to 1083960 tid 1 2024-09-16T05:15:26.755 INFO:tasks.rados.rados.2.smithi204.stdout:1182: writing smithi20457400-25 from 1846574 to 2544724 tid 2 2024-09-16T05:15:26.759 INFO:tasks.rados.rados.2.smithi204.stdout:1182: writing smithi20457400-25 from 3163674 to 3260624 tid 3 2024-09-16T05:15:26.759 INFO:tasks.rados.rados.2.smithi204.stdout:1183: snap_remove snap 101 2024-09-16T05:15:26.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:26.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:26.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:26.824 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:26.825 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:26.847 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:26.847 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:26.867 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:26.868 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:26.888 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:26.888 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:26.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:26.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:26.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:26.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:26.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:26.959 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:26.959 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:26.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:27.002 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:27.002 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:27.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:27.022 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:27.023 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:27.039 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:27.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:27.044 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:27.044 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:27.065 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:27.065 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:27.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:27.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:27.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:27.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:27.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:27.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:27.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:27.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:27.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:27.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:27.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:27.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:27.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:27.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:27.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:27.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:27.746 INFO:tasks.rados.rados.2.smithi204.stdout:1183: done (4 left) 2024-09-16T05:15:27.746 INFO:tasks.rados.rados.2.smithi204.stdout:1184: read oid 40 snap -1 2024-09-16T05:15:27.746 INFO:tasks.rados.rados.2.smithi204.stdout:1184: expect (ObjNum 386 snap 115 seq_num 386) 2024-09-16T05:15:27.746 INFO:tasks.rados.rados.2.smithi204.stdout:1185: read oid 41 snap -1 2024-09-16T05:15:27.746 INFO:tasks.rados.rados.2.smithi204.stdout:1185: expect (ObjNum 420 snap 124 seq_num 420) 2024-09-16T05:15:27.746 INFO:tasks.rados.rados.2.smithi204.stdout:1186: delete oid 34 current snap is 124 2024-09-16T05:15:27.746 INFO:tasks.rados.rados.2.smithi204.stdout:1180: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:15:27.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:27.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:27.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:27.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:27.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:27.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:27.983 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:27 smithi032 ceph-mon[27388]: pgmap v726: 87 pgs: 87 active+clean; 245 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.7 MiB/s wr, 19 op/s 2024-09-16T05:15:27.983 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:27 smithi032 ceph-mon[27388]: osdmap e377: 12 total, 12 up, 12 in 2024-09-16T05:15:27.983 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4277291616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:27.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:27 smithi196 ceph-mon[30802]: pgmap v726: 87 pgs: 87 active+clean; 245 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.7 MiB/s wr, 19 op/s 2024-09-16T05:15:27.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:27 smithi196 ceph-mon[30802]: osdmap e377: 12 total, 12 up, 12 in 2024-09-16T05:15:27.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4277291616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:28.008 INFO:tasks.rados.rados.2.smithi204.stdout:1182: finishing write tid 1 to smithi20457400-25 2024-09-16T05:15:28.008 INFO:tasks.rados.rados.2.smithi204.stdout:1181: expect (ObjNum 388 snap 116 seq_num 388) 2024-09-16T05:15:28.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:28.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:28.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:28.117 INFO:tasks.rados.rados.2.smithi204.stdout:1182: finishing write tid 2 to smithi20457400-25 2024-09-16T05:15:28.117 INFO:tasks.rados.rados.2.smithi204.stdout:1182: finishing write tid 3 to smithi20457400-25 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1182: finishing write tid 4 to smithi20457400-25 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1182: finishing write tid 5 to smithi20457400-25 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 451 (ObjNum 432 snap 124 seq_num 432) dirty exists 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1182: left oid 25 (ObjNum 432 snap 124 seq_num 432) 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1177: finishing copy_from to smithi20457400-30 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 329 (ObjNum 414 snap 123 seq_num 414) dirty exists 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1177: done (6 left) 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1180: done (5 left) 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1181: done (4 left) 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1182: done (3 left) 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1186: done (2 left) 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1187: snap_create 2024-09-16T05:15:28.118 INFO:tasks.rados.rados.2.smithi204.stdout:1184: expect (ObjNum 386 snap 115 seq_num 386) 2024-09-16T05:15:28.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:28.168 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:28.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:28.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:27 smithi204 ceph-mon[30933]: pgmap v726: 87 pgs: 87 active+clean; 245 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.7 MiB/s wr, 19 op/s 2024-09-16T05:15:28.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:27 smithi204 ceph-mon[30933]: osdmap e377: 12 total, 12 up, 12 in 2024-09-16T05:15:28.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4277291616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:28.252 INFO:tasks.rados.rados.2.smithi204.stdout:1185: expect (ObjNum 420 snap 124 seq_num 420) 2024-09-16T05:15:28.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:28.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:28.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:28.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:28.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:28.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:28.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:28.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:28.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:28.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:28.653 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:15:28.653 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:15:28.653 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:15:28.654 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:15:28.654 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:15:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:28.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:28.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:28.748 INFO:tasks.rados.rados.2.smithi204.stdout:1184: done (2 left) 2024-09-16T05:15:28.749 INFO:tasks.rados.rados.2.smithi204.stdout:1185: done (1 left) 2024-09-16T05:15:28.749 INFO:tasks.rados.rados.2.smithi204.stdout:1187: done (0 left) 2024-09-16T05:15:28.749 INFO:tasks.rados.rados.2.smithi204.stdout:1188: copy_from oid 43 from oid 47 current snap is 125 2024-09-16T05:15:28.749 INFO:tasks.rados.rados.2.smithi204.stdout:1189: write (excl) oid 49 current snap is 125 2024-09-16T05:15:28.749 INFO:tasks.rados.rados.2.smithi204.stdout:1189: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-16T05:15:28.754 INFO:tasks.rados.rados.2.smithi204.stdout:1189: writing smithi20457400-49 from 494238 to 984425 tid 1 2024-09-16T05:15:28.761 INFO:tasks.rados.rados.2.smithi204.stdout:1189: writing smithi20457400-49 from 1563538 to 2265357 tid 2 2024-09-16T05:15:28.767 INFO:tasks.rados.rados.2.smithi204.stdout:1189: writing smithi20457400-49 from 2818520 to 3294238 tid 3 2024-09-16T05:15:28.767 INFO:tasks.rados.rados.2.smithi204.stdout:1190: copy_from oid 11 from oid 18 current snap is 125 2024-09-16T05:15:28.767 INFO:tasks.rados.rados.2.smithi204.stdout:1188: finishing copy_from to smithi20457400-43 2024-09-16T05:15:28.767 INFO:tasks.rados.rados.2.smithi204.stdout:1188: got expected ENOENT (src dne) 2024-09-16T05:15:28.767 INFO:tasks.rados.rados.2.smithi204.stdout:1188: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:15:28.767 INFO:tasks.rados.rados.2.smithi204.stdout:1189: finishing write tid 1 to smithi20457400-49 2024-09-16T05:15:28.767 INFO:tasks.rados.rados.2.smithi204.stdout:1189: finishing write tid 2 to smithi20457400-49 2024-09-16T05:15:28.768 INFO:tasks.rados.rados.2.smithi204.stdout:1188: done (2 left) 2024-09-16T05:15:28.768 INFO:tasks.rados.rados.2.smithi204.stdout:1191: write oid 17 current snap is 125 2024-09-16T05:15:28.768 INFO:tasks.rados.rados.2.smithi204.stdout:1191: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-16T05:15:28.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:28.772 INFO:tasks.rados.rados.2.smithi204.stdout:1191: writing smithi20457400-17 from 511045 to 1076481 tid 1 2024-09-16T05:15:28.777 INFO:tasks.rados.rados.2.smithi204.stdout:1191: writing smithi20457400-17 from 1622020 to 2067497 tid 2 2024-09-16T05:15:28.784 INFO:tasks.rados.rados.2.smithi204.stdout:1191: writing smithi20457400-17 from 2729590 to 3286445 tid 3 2024-09-16T05:15:28.784 INFO:tasks.rados.rados.2.smithi204.stdout:1192: snap_remove snap 121 2024-09-16T05:15:28.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:28.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:28.843 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:28 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:28.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:28 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:28.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:28 smithi032 ceph-mon[27388]: osdmap e378: 12 total, 12 up, 12 in 2024-09-16T05:15:28.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:28.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:28.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:29.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:29.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:29.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:29.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:29.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:29.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:29.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:28 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:29.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:28 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:29.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:28 smithi204 ceph-mon[30933]: osdmap e378: 12 total, 12 up, 12 in 2024-09-16T05:15:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:28 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:28 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:28 smithi196 ceph-mon[30802]: osdmap e378: 12 total, 12 up, 12 in 2024-09-16T05:15:29.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:29.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:29.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:29.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:29.345 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:15:29.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:29.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:29.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:29.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:15:29.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:29.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:29.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:29.606 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:29.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:29.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:29.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1192: done (3 left) 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1193: snap_create 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1189: finishing write tid 3 to smithi20457400-49 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1190: finishing copy_from to smithi20457400-11 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1190: got expected ENOENT (src dne) 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1190: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1191: finishing write tid 1 to smithi20457400-17 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1191: finishing write tid 2 to smithi20457400-17 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1191: finishing write tid 3 to smithi20457400-17 2024-09-16T05:15:29.756 INFO:tasks.rados.rados.2.smithi204.stdout:1191: finishing write tid 4 to smithi20457400-17 2024-09-16T05:15:29.757 INFO:tasks.rados.rados.2.smithi204.stdout:1191: finishing write tid 5 to smithi20457400-17 2024-09-16T05:15:29.757 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 386 (ObjNum 435 snap 125 seq_num 435) dirty exists 2024-09-16T05:15:29.757 INFO:tasks.rados.rados.2.smithi204.stdout:1191: left oid 17 (ObjNum 435 snap 125 seq_num 435) 2024-09-16T05:15:29.757 INFO:tasks.rados.rados.2.smithi204.stdout:1189: finishing write tid 4 to smithi20457400-49 2024-09-16T05:15:29.757 INFO:tasks.rados.rados.2.smithi204.stdout:1189: finishing write tid 5 to smithi20457400-49 2024-09-16T05:15:29.757 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 328 (ObjNum 434 snap 125 seq_num 434) dirty exists 2024-09-16T05:15:29.757 INFO:tasks.rados.rados.2.smithi204.stdout:1189: left oid 49 (ObjNum 434 snap 125 seq_num 434) 2024-09-16T05:15:29.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:29.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:29.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:29.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:29.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:29.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:29.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:15:30.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:29 smithi032 ceph-mon[27388]: pgmap v729: 87 pgs: 87 active+clean; 227 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2024-09-16T05:15:30.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:29 smithi032 ceph-mon[27388]: osdmap e379: 12 total, 12 up, 12 in 2024-09-16T05:15:30.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3481834470' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:15:30.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:30.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:30.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:30.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:29 smithi204 ceph-mon[30933]: pgmap v729: 87 pgs: 87 active+clean; 227 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2024-09-16T05:15:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:29 smithi204 ceph-mon[30933]: osdmap e379: 12 total, 12 up, 12 in 2024-09-16T05:15:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3481834470' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:15:30.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:30.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:30.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:29 smithi196 ceph-mon[30802]: pgmap v729: 87 pgs: 87 active+clean; 227 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.7 MiB/s wr, 20 op/s 2024-09-16T05:15:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:29 smithi196 ceph-mon[30802]: osdmap e379: 12 total, 12 up, 12 in 2024-09-16T05:15:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3481834470' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:15:30.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:30.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:30.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:30.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:30.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:30.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:30.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:30.557 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:30.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:30.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:30.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:30.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:15:30.706 INFO:tasks.rados.rados.2.smithi204.stdout:1189: done (3 left) 2024-09-16T05:15:30.706 INFO:tasks.rados.rados.2.smithi204.stdout:1190: done (2 left) 2024-09-16T05:15:30.706 INFO:tasks.rados.rados.2.smithi204.stdout:1191: done (1 left) 2024-09-16T05:15:30.706 INFO:tasks.rados.rados.2.smithi204.stdout:1193: done (0 left) 2024-09-16T05:15:30.706 INFO:tasks.rados.rados.2.smithi204.stdout:1194: snap_remove snap 124 2024-09-16T05:15:30.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:30.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:30.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:30.845 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T05:15:30.846 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:15:30.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:30.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:30.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:31.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:31.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:31.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:31.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3481834470' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:15:31.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:30 smithi032 ceph-mon[27388]: osdmap e380: 12 total, 12 up, 12 in 2024-09-16T05:15:31.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1929431343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:31.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:30 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:31.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1929431343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:31.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:30 smithi032 ceph-mon[27388]: osdmap e381: 12 total, 12 up, 12 in 2024-09-16T05:15:31.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3481834470' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:15:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:30 smithi204 ceph-mon[30933]: osdmap e380: 12 total, 12 up, 12 in 2024-09-16T05:15:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1929431343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:30 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1929431343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:30 smithi204 ceph-mon[30933]: osdmap e381: 12 total, 12 up, 12 in 2024-09-16T05:15:31.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:31.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3481834470' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:15:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:30 smithi196 ceph-mon[30802]: osdmap e380: 12 total, 12 up, 12 in 2024-09-16T05:15:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1929431343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:30 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1929431343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:30 smithi196 ceph-mon[30802]: osdmap e381: 12 total, 12 up, 12 in 2024-09-16T05:15:31.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:31.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:31.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:31.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:31.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:31.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:31.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:31.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:31.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:31.590 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:31.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:31.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:31.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:31.710 INFO:tasks.rados.rados.2.smithi204.stdout:1194: done (0 left) 2024-09-16T05:15:31.710 INFO:tasks.rados.rados.2.smithi204.stdout:1195: rollback oid 6 current snap is 126 2024-09-16T05:15:31.710 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 6 to 123 2024-09-16T05:15:31.710 INFO:tasks.rados.rados.2.smithi204.stdout:1196: rollback oid 21 current snap is 126 2024-09-16T05:15:31.711 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 21 to 123 2024-09-16T05:15:31.711 INFO:tasks.rados.rados.2.smithi204.stdout:1197: write (excl) oid 48 current snap is 126 2024-09-16T05:15:31.711 INFO:tasks.rados.rados.2.smithi204.stdout:1197: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-09-16T05:15:31.717 INFO:tasks.rados.rados.2.smithi204.stdout:1197: writing smithi20457400-48 from 527852 to 1168537 tid 1 2024-09-16T05:15:31.723 INFO:tasks.rados.rados.2.smithi204.stdout:1197: writing smithi20457400-48 from 1680502 to 2269637 tid 2 2024-09-16T05:15:31.729 INFO:tasks.rados.rados.2.smithi204.stdout:1197: writing smithi20457400-48 from 2757013 to 3327852 tid 3 2024-09-16T05:15:31.729 INFO:tasks.rados.rados.2.smithi204.stdout:1198: snap_remove snap 125 2024-09-16T05:15:31.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:31.828 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:31.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:31.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:31 smithi032 ceph-mon[27388]: pgmap v732: 87 pgs: 87 active+clean; 214 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 903 KiB/s wr, 11 op/s 2024-09-16T05:15:31.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:31 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:15:31.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:31 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:15:31.831 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:31 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:15:31.831 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:31 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:15:31.831 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:31 smithi032 ceph-mon[27388]: osdmap e382: 12 total, 12 up, 12 in 2024-09-16T05:15:31.860 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:31.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:31.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:31.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:32.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:32.067 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:32.067 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:32.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:32.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:32.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:32.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:32.110 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:32.111 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:32.131 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:32.131 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:32.153 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:32.153 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:31 smithi204 ceph-mon[30933]: pgmap v732: 87 pgs: 87 active+clean; 214 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 903 KiB/s wr, 11 op/s 2024-09-16T05:15:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:31 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:15:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:31 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:15:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:31 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:15:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:31 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:15:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:31 smithi204 ceph-mon[30933]: osdmap e382: 12 total, 12 up, 12 in 2024-09-16T05:15:32.175 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:32.175 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:32.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:32.198 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:32.198 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:32.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:32.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:32.220 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:32.220 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:32.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:31 smithi196 ceph-mon[30802]: pgmap v732: 87 pgs: 87 active+clean; 214 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 903 KiB/s wr, 11 op/s 2024-09-16T05:15:32.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:31 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:15:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:31 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:15:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:31 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:15:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:31 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:15:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:31 smithi196 ceph-mon[30802]: osdmap e382: 12 total, 12 up, 12 in 2024-09-16T05:15:32.242 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:32.264 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:32.286 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:32.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:32.312 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:32.312 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:32.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:32.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:32.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:32.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:32.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:32.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:32.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:32.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:32.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:1198: done (3 left) 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:1199: snap_create 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:1196: finishing rollback tid 0 to smithi20457400-21 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:1196: finishing rollback tid 1 to smithi20457400-21 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:1195: finishing rollback tid 0 to smithi20457400-6 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:1196: finishing rollback tid 2 to smithi20457400-21 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 322 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:1195: finishing rollback tid 1 to smithi20457400-6 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:1195: finishing rollback tid 2 to smithi20457400-6 2024-09-16T05:15:32.711 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 332 (ObjNum 414 snap 123 seq_num 414) dirty exists 2024-09-16T05:15:32.712 INFO:tasks.rados.rados.2.smithi204.stdout:1197: finishing write tid 1 to smithi20457400-48 2024-09-16T05:15:32.712 INFO:tasks.rados.rados.2.smithi204.stdout:1197: finishing write tid 2 to smithi20457400-48 2024-09-16T05:15:32.712 INFO:tasks.rados.rados.2.smithi204.stdout:1197: finishing write tid 3 to smithi20457400-48 2024-09-16T05:15:32.712 INFO:tasks.rados.rados.2.smithi204.stdout:1197: finishing write tid 4 to smithi20457400-48 2024-09-16T05:15:32.712 INFO:tasks.rados.rados.2.smithi204.stdout:1197: finishing write tid 5 to smithi20457400-48 2024-09-16T05:15:32.712 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 373 (ObjNum 436 snap 126 seq_num 436) dirty exists 2024-09-16T05:15:32.712 INFO:tasks.rados.rados.2.smithi204.stdout:1197: left oid 48 (ObjNum 436 snap 126 seq_num 436) 2024-09-16T05:15:32.720 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:32.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:32.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:32.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:32.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:32.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:32.977 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:32.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:33.053 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:32 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:15:33.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:32 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:15:33.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:32 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2662493221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:33.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:32 smithi032 ceph-mon[27388]: osdmap e383: 12 total, 12 up, 12 in 2024-09-16T05:15:33.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:33.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:33.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:32 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:15:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:32 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:15:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:32 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2662493221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:32 smithi204 ceph-mon[30933]: osdmap e383: 12 total, 12 up, 12 in 2024-09-16T05:15:33.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:33.231 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:33.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:33.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:32 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:15:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:32 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:15:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:32 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2662493221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:32 smithi196 ceph-mon[30802]: osdmap e383: 12 total, 12 up, 12 in 2024-09-16T05:15:33.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:33.355 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:33.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:33.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:33.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:33.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:33.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:15:33.568 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:33.568 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:15:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:15:33.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:33.610 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:33.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:33.711 INFO:tasks.rados.rados.2.smithi204.stdout:1195: done (3 left) 2024-09-16T05:15:33.711 INFO:tasks.rados.rados.2.smithi204.stdout:1196: done (2 left) 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1197: done (1 left) 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1199: done (0 left) 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1200: rollback oid 47 current snap is 127 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 47 to 126 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1201: read oid 6 snap -1 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1201: expect (ObjNum 414 snap 123 seq_num 414) 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1202: rollback oid 26 current snap is 127 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 126 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1203: rollback oid 46 current snap is 127 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 126 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1204: read oid 40 snap -1 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1204: expect (ObjNum 386 snap 115 seq_num 386) 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1205: snap_create 2024-09-16T05:15:33.712 INFO:tasks.rados.rados.2.smithi204.stdout:1200: finishing rollback tid 1 to smithi20457400-47 2024-09-16T05:15:33.713 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:15:33.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:33.716 INFO:tasks.rados.rados.2.smithi204.stdout:1202: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:15:33.716 INFO:tasks.rados.rados.2.smithi204.stdout:1202: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:15:33.716 INFO:tasks.rados.rados.2.smithi204.stdout:1202: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:15:33.716 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 628 (ObjNum 383 snap 115 seq_num 383) dirty exists 2024-09-16T05:15:33.718 INFO:tasks.rados.rados.2.smithi204.stdout:1203: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:15:33.718 INFO:tasks.rados.rados.2.smithi204.stdout:1203: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:15:33.718 INFO:tasks.rados.rados.2.smithi204.stdout:1203: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:15:33.719 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 325 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:15:33.720 INFO:tasks.rados.rados.2.smithi204.stdout:1204: expect (ObjNum 386 snap 115 seq_num 386) 2024-09-16T05:15:33.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:33.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:33.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:33.857 INFO:tasks.rados.rados.2.smithi204.stdout:1201: expect (ObjNum 414 snap 123 seq_num 414) 2024-09-16T05:15:33.863 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:33.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:33.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:33.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:34.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:34.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:34.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:34.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:33 smithi032 ceph-mon[27388]: pgmap v735: 87 pgs: 87 active+clean; 173 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-16T05:15:34.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:33 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:15:34.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:33 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:15:34.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:33 smithi032 ceph-mon[27388]: osdmap e384: 12 total, 12 up, 12 in 2024-09-16T05:15:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:33 smithi204 ceph-mon[30933]: pgmap v735: 87 pgs: 87 active+clean; 173 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-16T05:15:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:33 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:15:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:33 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:15:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:33 smithi204 ceph-mon[30933]: osdmap e384: 12 total, 12 up, 12 in 2024-09-16T05:15:34.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:34.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:33 smithi196 ceph-mon[30802]: pgmap v735: 87 pgs: 87 active+clean; 173 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.9 MiB/s wr, 20 op/s 2024-09-16T05:15:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:33 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:15:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:33 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:15:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:33 smithi196 ceph-mon[30802]: osdmap e384: 12 total, 12 up, 12 in 2024-09-16T05:15:34.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:34.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:34.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:34.369 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:34.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:34.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:34.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:34.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:15:34.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:34.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:34.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:34.712 INFO:tasks.rados.rados.2.smithi204.stdout:1200: done (5 left) 2024-09-16T05:15:34.712 INFO:tasks.rados.rados.2.smithi204.stdout:1201: done (4 left) 2024-09-16T05:15:34.712 INFO:tasks.rados.rados.2.smithi204.stdout:1202: done (3 left) 2024-09-16T05:15:34.712 INFO:tasks.rados.rados.2.smithi204.stdout:1203: done (2 left) 2024-09-16T05:15:34.712 INFO:tasks.rados.rados.2.smithi204.stdout:1204: done (1 left) 2024-09-16T05:15:34.713 INFO:tasks.rados.rados.2.smithi204.stdout:1205: done (0 left) 2024-09-16T05:15:34.713 INFO:tasks.rados.rados.2.smithi204.stdout:1206: read oid 3 snap 126 2024-09-16T05:15:34.713 INFO:tasks.rados.rados.2.smithi204.stdout:1206: expect deleted 2024-09-16T05:15:34.713 INFO:tasks.rados.rados.2.smithi204.stdout:1207: write oid 24 current snap is 128 2024-09-16T05:15:34.713 INFO:tasks.rados.rados.2.smithi204.stdout:1207: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-16T05:15:34.718 INFO:tasks.rados.rados.2.smithi204.stdout:1207: writing smithi20457400-24 from 544659 to 1260593 tid 1 2024-09-16T05:15:34.723 INFO:tasks.rados.rados.2.smithi204.stdout:1207: writing smithi20457400-24 from 1738984 to 2188130 tid 2 2024-09-16T05:15:34.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:34.730 INFO:tasks.rados.rados.2.smithi204.stdout:1207: writing smithi20457400-24 from 2784436 to 3344659 tid 3 2024-09-16T05:15:34.730 INFO:tasks.rados.rados.2.smithi204.stdout:1208: copy_from oid 34 from oid 28 current snap is 128 2024-09-16T05:15:34.730 INFO:tasks.rados.rados.2.smithi204.stdout:1207: finishing write tid 1 to smithi20457400-24 2024-09-16T05:15:34.731 INFO:tasks.rados.rados.2.smithi204.stdout:1207: finishing write tid 2 to smithi20457400-24 2024-09-16T05:15:34.731 INFO:tasks.rados.rados.2.smithi204.stdout:1206: done (2 left) 2024-09-16T05:15:34.731 INFO:tasks.rados.rados.2.smithi204.stdout:1209: snap_create 2024-09-16T05:15:34.732 INFO:tasks.rados.rados.2.smithi204.stdout:1207: finishing write tid 3 to smithi20457400-24 2024-09-16T05:15:34.733 INFO:tasks.rados.rados.2.smithi204.stdout:1207: finishing write tid 4 to smithi20457400-24 2024-09-16T05:15:34.733 INFO:tasks.rados.rados.2.smithi204.stdout:1207: finishing write tid 5 to smithi20457400-24 2024-09-16T05:15:34.733 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 450 (ObjNum 437 snap 128 seq_num 437) dirty exists 2024-09-16T05:15:34.733 INFO:tasks.rados.rados.2.smithi204.stdout:1207: left oid 24 (ObjNum 437 snap 128 seq_num 437) 2024-09-16T05:15:34.740 INFO:tasks.rados.rados.2.smithi204.stdout:1208: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:15:34.742 INFO:tasks.rados.rados.2.smithi204.stdout:1208: finishing copy_from to smithi20457400-34 2024-09-16T05:15:34.742 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 361 (ObjNum 412 snap 123 seq_num 412) dirty exists 2024-09-16T05:15:34.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:34.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:34.797 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:34.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:34.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:34.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:34.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:34 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:15:34.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:34 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:15:34.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:34 smithi196 ceph-mon[30802]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:34.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:34 smithi196 ceph-mon[30802]: osdmap e385: 12 total, 12 up, 12 in 2024-09-16T05:15:34.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:15:34.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:35.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:35.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:35.015 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":736,"stamp":"2024-09-16T05:15:32.705995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161636212,"num_objects":70,"num_object_clones":25,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":4,"num_read":641,"num_read_kb":648688,"num_write":463,"num_write_kb":191010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16948,"ondisk_log_size":16948,"up":174,"acting":174,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":681444,"kb_used_data":256116,"kb_used_omap":27,"kb_used_meta":424996,"kb_avail":1124260380,"statfs":{"total":1151940427776,"available":1151242629120,"internally_reserved":0,"allocated":262262784,"data_stored":276572468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":435196071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-76040112,"num_objects":-32,"num_object_clones":-29,"num_object_copies":-64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-32,"num_whiteouts":0,"num_read":57,"num_read_kb":50617,"num_write":41,"num_write_kb":13419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.961850"},"pg_stats":[{"pgid":"2.4b","version":"244'177","reported_seq":1524,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.765132+0000","last_change":"2024-09-16T05:15:29.765448+0000","last_active":"2024-09-16T05:15:29.765132+0000","last_peered":"2024-09-16T05:15:29.765132+0000","last_clean":"2024-09-16T05:15:29.765132+0000","last_became_active":"2024-09-16T05:14:30.853998+0000","last_became_peered":"2024-09-16T05:14:30.853998+0000","last_unstale":"2024-09-16T05:15:29.765132+0000","last_undegraded":"2024-09-16T05:15:29.765132+0000","last_fullsized":"2024-09-16T05:15:29.765132+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:14:32.337052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:32.337052+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:53.303244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024360060000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.4a","version":"378'461","reported_seq":1768,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.765640+0000","last_change":"2024-09-16T05:15:29.765966+0000","last_active":"2024-09-16T05:15:29.765640+0000","last_peered":"2024-09-16T05:15:29.765640+0000","last_clean":"2024-09-16T05:15:29.765640+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:15:29.765640+0000","last_undegraded":"2024-09-16T05:15:29.765640+0000","last_fullsized":"2024-09-16T05:15:29.765640+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"309'419","last_scrub_stamp":"2024-09-16T05:14:31.471678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.471678+0000","objects_scrubbed":2,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:50.429685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022904739999999998,"stat_sum":{"num_bytes":2806835,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.49","version":"245'220","reported_seq":1563,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.769649+0000","last_change":"2024-09-16T05:15:29.769957+0000","last_active":"2024-09-16T05:15:29.769649+0000","last_peered":"2024-09-16T05:15:29.769649+0000","last_clean":"2024-09-16T05:15:29.769649+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:15:29.769649+0000","last_undegraded":"2024-09-16T05:15:29.769649+0000","last_fullsized":"2024-09-16T05:15:29.769649+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:14:30.917243+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.917243+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:40.441668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050883700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.48","version":"378'179","reported_seq":1487,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.759555+0000","last_change":"2024-09-16T05:15:29.759869+0000","last_active":"2024-09-16T05:15:29.759555+0000","last_peered":"2024-09-16T05:15:29.759555+0000","last_clean":"2024-09-16T05:15:29.759555+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:15:29.759555+0000","last_undegraded":"2024-09-16T05:15:29.759555+0000","last_fullsized":"2024-09-16T05:15:29.759555+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'153","last_scrub_stamp":"2024-09-16T05:14:28.221076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.221076+0000","objects_scrubbed":1,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:36.300029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063543499999999995,"stat_sum":{"num_bytes":3142942,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":20676,"num_write":1,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.47","version":"0'0","reported_seq":334,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.761335+0000","last_change":"2024-09-16T05:15:29.761635+0000","last_active":"2024-09-16T05:15:29.761335+0000","last_peered":"2024-09-16T05:15:29.761335+0000","last_clean":"2024-09-16T05:15:29.761335+0000","last_became_active":"2024-09-16T05:15:16.867821+0000","last_became_peered":"2024-09-16T05:15:16.867821+0000","last_unstale":"2024-09-16T05:15:29.761335+0000","last_undegraded":"2024-09-16T05:15:29.761335+0000","last_fullsized":"2024-09-16T05:15:29.761335+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:28.987418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.987418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:43.682262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053623600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.46","version":"116'9","reported_seq":1351,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763172+0000","last_change":"2024-09-16T05:15:29.763518+0000","last_active":"2024-09-16T05:15:29.763172+0000","last_peered":"2024-09-16T05:15:29.763172+0000","last_clean":"2024-09-16T05:15:29.763172+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:15:29.763172+0000","last_undegraded":"2024-09-16T05:15:29.763172+0000","last_fullsized":"2024-09-16T05:15:29.763172+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:29.313711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:29.313711+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:44.780995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091001099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.45","version":"305'464","reported_seq":1778,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763627+0000","last_change":"2024-09-16T05:15:29.764733+0000","last_active":"2024-09-16T05:15:29.763627+0000","last_peered":"2024-09-16T05:15:29.763627+0000","last_clean":"2024-09-16T05:15:29.763627+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:15:29.763627+0000","last_undegraded":"2024-09-16T05:15:29.763627+0000","last_fullsized":"2024-09-16T05:15:29.763627+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:14:31.325082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.325082+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:00.576410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019917379999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":1433,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763010+0000","last_change":"2024-09-16T05:15:29.763509+0000","last_active":"2024-09-16T05:15:29.763010+0000","last_peered":"2024-09-16T05:15:29.763010+0000","last_clean":"2024-09-16T05:15:29.763010+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:15:29.763010+0000","last_undegraded":"2024-09-16T05:15:29.763010+0000","last_fullsized":"2024-09-16T05:15:29.763010+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:30.305712+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.305712+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:48.710776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097363799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":342,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.712829+0000","last_change":"2024-09-16T05:15:31.713060+0000","last_active":"2024-09-16T05:15:31.712829+0000","last_peered":"2024-09-16T05:15:31.712829+0000","last_clean":"2024-09-16T05:15:31.712829+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:15:31.712829+0000","last_undegraded":"2024-09-16T05:15:31.712829+0000","last_fullsized":"2024-09-16T05:15:31.712829+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:14:28.659562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.659562+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:49.546066+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010566530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":295,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-16T05:15:28.768298+0000","last_change":"2024-09-16T05:15:27.752100+0000","last_active":"2024-09-16T05:15:28.768298+0000","last_peered":"2024-09-16T05:15:28.768298+0000","last_clean":"2024-09-16T05:15:28.768298+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:15:28.768298+0000","last_undegraded":"2024-09-16T05:15:28.768298+0000","last_fullsized":"2024-09-16T05:15:28.768298+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:14:28.532014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.532014+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:57.963651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0056838640000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":422,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.765322+0000","last_change":"2024-09-16T05:15:29.765561+0000","last_active":"2024-09-16T05:15:29.765322+0000","last_peered":"2024-09-16T05:15:29.765322+0000","last_clean":"2024-09-16T05:15:29.765322+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:15:29.765322+0000","last_undegraded":"2024-09-16T05:15:29.765322+0000","last_fullsized":"2024-09-16T05:15:29.765322+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:15.794439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.794439+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.674887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001891258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.40","version":"205'33","reported_seq":1388,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.717842+0000","last_change":"2024-09-16T05:15:31.718604+0000","last_active":"2024-09-16T05:15:31.717842+0000","last_peered":"2024-09-16T05:15:31.717842+0000","last_clean":"2024-09-16T05:15:31.717842+0000","last_became_active":"2024-09-16T05:13:45.682851+0000","last_became_peered":"2024-09-16T05:13:45.682851+0000","last_unstale":"2024-09-16T05:15:31.717842+0000","last_undegraded":"2024-09-16T05:15:31.717842+0000","last_fullsized":"2024-09-16T05:15:31.717842+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:07.789202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.789202+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.141078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0060912919999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3f","version":"378'282","reported_seq":1634,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.715921+0000","last_change":"2024-09-16T05:15:31.716267+0000","last_active":"2024-09-16T05:15:31.715921+0000","last_peered":"2024-09-16T05:15:31.715921+0000","last_clean":"2024-09-16T05:15:31.715921+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:15:31.715921+0000","last_undegraded":"2024-09-16T05:15:31.715921+0000","last_fullsized":"2024-09-16T05:15:31.715921+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"357'270","last_scrub_stamp":"2024-09-16T05:15:09.797411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.797411+0000","objects_scrubbed":3,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:35.904107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039964459999999999,"stat_sum":{"num_bytes":2285818,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":15220,"num_write":15,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3e","version":"378'353","reported_seq":1669,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764831+0000","last_change":"2024-09-16T05:15:31.765009+0000","last_active":"2024-09-16T05:15:31.764831+0000","last_peered":"2024-09-16T05:15:31.764831+0000","last_clean":"2024-09-16T05:15:31.764831+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:15:31.764831+0000","last_undegraded":"2024-09-16T05:15:31.764831+0000","last_fullsized":"2024-09-16T05:15:31.764831+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"360'342","last_scrub_stamp":"2024-09-16T05:15:09.920637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.920637+0000","objects_scrubbed":4,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:29.576047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081514700000000001,"stat_sum":{"num_bytes":2437081,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":2380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1880,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.764350+0000","last_change":"2024-09-16T05:15:29.764747+0000","last_active":"2024-09-16T05:15:29.764350+0000","last_peered":"2024-09-16T05:15:29.764350+0000","last_clean":"2024-09-16T05:15:29.764350+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:15:29.764350+0000","last_undegraded":"2024-09-16T05:15:29.764350+0000","last_fullsized":"2024-09-16T05:15:29.764350+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:15:09.150849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.150849+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.221984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018405509999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.3c","version":"378'445","reported_seq":1878,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.713128+0000","last_change":"2024-09-16T05:15:31.713428+0000","last_active":"2024-09-16T05:15:31.713128+0000","last_peered":"2024-09-16T05:15:31.713128+0000","last_clean":"2024-09-16T05:15:31.713128+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:15:31.713128+0000","last_undegraded":"2024-09-16T05:15:31.713128+0000","last_fullsized":"2024-09-16T05:15:31.713128+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"342'400","last_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:14:51.845721+0000","objects_scrubbed":6,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.270697+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0013605030000000001,"stat_sum":{"num_bytes":3815222,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22839,"num_write":14,"num_write_kb":4757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3b","version":"361'294","reported_seq":1637,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.770275+0000","last_change":"2024-09-16T05:15:29.770867+0000","last_active":"2024-09-16T05:15:29.770275+0000","last_peered":"2024-09-16T05:15:29.770275+0000","last_clean":"2024-09-16T05:15:29.770275+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:15:29.770275+0000","last_undegraded":"2024-09-16T05:15:29.770275+0000","last_fullsized":"2024-09-16T05:15:29.770275+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"300'290","last_scrub_stamp":"2024-09-16T05:15:06.955285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.955285+0000","objects_scrubbed":2,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:15.503831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00082238199999999997,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":31094,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.3a","version":"380'452","reported_seq":1758,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.765194+0000","last_change":"2024-09-16T05:15:29.765534+0000","last_active":"2024-09-16T05:15:29.765194+0000","last_peered":"2024-09-16T05:15:29.765194+0000","last_clean":"2024-09-16T05:15:29.765194+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:15:29.765194+0000","last_undegraded":"2024-09-16T05:15:29.765194+0000","last_fullsized":"2024-09-16T05:15:29.765194+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"356'429","last_scrub_stamp":"2024-09-16T05:15:10.141866+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.141866+0000","objects_scrubbed":2,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:27.618203+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0054042320000000001,"stat_sum":{"num_bytes":2605151,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":16,"num_read_kb":9240,"num_write":18,"num_write_kb":7078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.39","version":"182'60","reported_seq":429,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.761888+0000","last_change":"2024-09-16T05:15:29.762220+0000","last_active":"2024-09-16T05:15:29.761888+0000","last_peered":"2024-09-16T05:15:29.761888+0000","last_clean":"2024-09-16T05:15:29.761888+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:15:29.761888+0000","last_undegraded":"2024-09-16T05:15:29.761888+0000","last_fullsized":"2024-09-16T05:15:29.761888+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:15.232072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.232072+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:24.834904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027190669999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.38","version":"181'57","reported_seq":416,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-16T05:15:28.768292+0000","last_change":"2024-09-16T05:15:27.752806+0000","last_active":"2024-09-16T05:15:28.768292+0000","last_peered":"2024-09-16T05:15:28.768292+0000","last_clean":"2024-09-16T05:15:28.768292+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:15:28.768292+0000","last_undegraded":"2024-09-16T05:15:28.768292+0000","last_fullsized":"2024-09-16T05:15:28.768292+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:13.450261+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:13.450261+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:29.726423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0057651250000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":510,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.764679+0000","last_change":"2024-09-16T05:15:29.764948+0000","last_active":"2024-09-16T05:15:29.764679+0000","last_peered":"2024-09-16T05:15:29.764679+0000","last_clean":"2024-09-16T05:15:29.764679+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:15:29.764679+0000","last_undegraded":"2024-09-16T05:15:29.764679+0000","last_fullsized":"2024-09-16T05:15:29.764679+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.650972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:16.993793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015210779999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.36","version":"272'60","reported_seq":508,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.764709+0000","last_change":"2024-09-16T05:15:29.765068+0000","last_active":"2024-09-16T05:15:29.764709+0000","last_peered":"2024-09-16T05:15:29.764709+0000","last_clean":"2024-09-16T05:15:29.764709+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:15:29.764709+0000","last_undegraded":"2024-09-16T05:15:29.764709+0000","last_fullsized":"2024-09-16T05:15:29.764709+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:14:49.375386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.375386+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:59.362489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022005309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.35","version":"380'503","reported_seq":1864,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.767145+0000","last_change":"2024-09-16T05:15:29.767385+0000","last_active":"2024-09-16T05:15:29.767145+0000","last_peered":"2024-09-16T05:15:29.767145+0000","last_clean":"2024-09-16T05:15:29.767145+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:15:29.767145+0000","last_undegraded":"2024-09-16T05:15:29.767145+0000","last_fullsized":"2024-09-16T05:15:29.767145+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'421","last_scrub_stamp":"2024-09-16T05:14:34.349392+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:14:34.349392+0000","objects_scrubbed":4,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:47.147391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0042922159999999997,"stat_sum":{"num_bytes":9815498,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":23,"num_read_kb":24061,"num_write":19,"num_write_kb":7769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.34","version":"378'308","reported_seq":1252,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.770425+0000","last_change":"2024-09-16T05:15:29.770872+0000","last_active":"2024-09-16T05:15:29.770425+0000","last_peered":"2024-09-16T05:15:29.770425+0000","last_clean":"2024-09-16T05:15:29.770425+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:15:29.770425+0000","last_undegraded":"2024-09-16T05:15:29.770425+0000","last_fullsized":"2024-09-16T05:15:29.770425+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'243","last_scrub_stamp":"2024-09-16T05:14:50.913931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.913931+0000","objects_scrubbed":5,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.960895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000876755,"stat_sum":{"num_bytes":5613670,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":41071,"num_write":9,"num_write_kb":6969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":520,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.715847+0000","last_change":"2024-09-16T05:15:31.716256+0000","last_active":"2024-09-16T05:15:31.715847+0000","last_peered":"2024-09-16T05:15:31.715847+0000","last_clean":"2024-09-16T05:15:31.715847+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:15:31.715847+0000","last_undegraded":"2024-09-16T05:15:31.715847+0000","last_fullsized":"2024-09-16T05:15:31.715847+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:46.792111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.792111+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:07.000604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040551720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.32","version":"380'125","reported_seq":600,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764359+0000","last_change":"2024-09-16T05:15:31.764567+0000","last_active":"2024-09-16T05:15:31.764359+0000","last_peered":"2024-09-16T05:15:31.764359+0000","last_clean":"2024-09-16T05:15:31.764359+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:15:31.764359+0000","last_undegraded":"2024-09-16T05:15:31.764359+0000","last_fullsized":"2024-09-16T05:15:31.764359+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'101","last_scrub_stamp":"2024-09-16T05:14:49.915945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.915945+0000","objects_scrubbed":2,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.781860+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000548779,"stat_sum":{"num_bytes":6857355,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":17,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.31","version":"380'162","reported_seq":1126,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763783+0000","last_change":"2024-09-16T05:15:29.764084+0000","last_active":"2024-09-16T05:15:29.763783+0000","last_peered":"2024-09-16T05:15:29.763783+0000","last_clean":"2024-09-16T05:15:29.763783+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:15:29.763783+0000","last_undegraded":"2024-09-16T05:15:29.763783+0000","last_fullsized":"2024-09-16T05:15:29.763783+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'133","last_scrub_stamp":"2024-09-16T05:14:46.026342+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.026342+0000","objects_scrubbed":2,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:10.533387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030714620000000001,"stat_sum":{"num_bytes":4394329,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":22763,"num_write":13,"num_write_kb":4010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.30","version":"182'16","reported_seq":474,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763831+0000","last_change":"2024-09-16T05:15:29.764103+0000","last_active":"2024-09-16T05:15:29.763831+0000","last_peered":"2024-09-16T05:15:29.763831+0000","last_clean":"2024-09-16T05:15:29.763831+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:15:29.763831+0000","last_undegraded":"2024-09-16T05:15:29.763831+0000","last_fullsized":"2024-09-16T05:15:29.763831+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:06.783241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.783241+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:25.523060+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00080753300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.11","version":"272'115","reported_seq":1060,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.716215+0000","last_change":"2024-09-16T05:15:31.716651+0000","last_active":"2024-09-16T05:15:31.716215+0000","last_peered":"2024-09-16T05:15:31.716215+0000","last_clean":"2024-09-16T05:15:31.716215+0000","last_became_active":"2024-09-16T05:13:25.679267+0000","last_became_peered":"2024-09-16T05:13:25.679267+0000","last_unstale":"2024-09-16T05:15:31.716215+0000","last_undegraded":"2024-09-16T05:15:31.716215+0000","last_fullsized":"2024-09-16T05:15:31.716215+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:14:31.050468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.050468+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:39.196078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019176340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":955,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-16T05:15:28.768341+0000","last_change":"2024-09-16T05:15:27.753342+0000","last_active":"2024-09-16T05:15:28.768341+0000","last_peered":"2024-09-16T05:15:28.768341+0000","last_clean":"2024-09-16T05:15:28.768341+0000","last_became_active":"2024-09-16T05:13:25.680445+0000","last_became_peered":"2024-09-16T05:13:25.680445+0000","last_unstale":"2024-09-16T05:15:28.768341+0000","last_undegraded":"2024-09-16T05:15:28.768341+0000","last_fullsized":"2024-09-16T05:15:28.768341+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:07.531934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.531934+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:29.687436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0060651100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.f","version":"360'281","reported_seq":1619,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-16T05:15:28.768815+0000","last_change":"2024-09-16T05:15:27.748074+0000","last_active":"2024-09-16T05:15:28.768815+0000","last_peered":"2024-09-16T05:15:28.768815+0000","last_clean":"2024-09-16T05:15:28.768815+0000","last_became_active":"2024-09-16T05:13:25.680481+0000","last_became_peered":"2024-09-16T05:13:25.680481+0000","last_unstale":"2024-09-16T05:15:28.768815+0000","last_undegraded":"2024-09-16T05:15:28.768815+0000","last_fullsized":"2024-09-16T05:15:28.768815+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:50.580296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.580296+0000","objects_scrubbed":3,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:02.738936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001685275,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.e","version":"272'232","reported_seq":1471,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.751775+0000","last_change":"2024-09-16T05:15:27.752301+0000","last_active":"2024-09-16T05:15:27.751775+0000","last_peered":"2024-09-16T05:15:27.751775+0000","last_clean":"2024-09-16T05:15:27.751775+0000","last_became_active":"2024-09-16T05:14:40.261276+0000","last_became_peered":"2024-09-16T05:14:40.261276+0000","last_unstale":"2024-09-16T05:15:27.751775+0000","last_undegraded":"2024-09-16T05:15:27.751775+0000","last_fullsized":"2024-09-16T05:15:27.751775+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:14:47.777342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.777342+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:10.104341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066898700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.d","version":"380'455","reported_seq":1796,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.766453+0000","last_change":"2024-09-16T05:15:29.766752+0000","last_active":"2024-09-16T05:15:29.766453+0000","last_peered":"2024-09-16T05:15:29.766453+0000","last_clean":"2024-09-16T05:15:29.766453+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:15:29.766453+0000","last_undegraded":"2024-09-16T05:15:29.766453+0000","last_fullsized":"2024-09-16T05:15:29.766453+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'379","last_scrub_stamp":"2024-09-16T05:14:44.180613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.180613+0000","objects_scrubbed":6,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:10.127000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.006171419,"stat_sum":{"num_bytes":9462506,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":48192,"num_write":30,"num_write_kb":9875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.c","version":"380'255","reported_seq":1599,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764302+0000","last_change":"2024-09-16T05:15:31.764527+0000","last_active":"2024-09-16T05:15:31.764302+0000","last_peered":"2024-09-16T05:15:31.764302+0000","last_clean":"2024-09-16T05:15:31.764302+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:15:31.764302+0000","last_undegraded":"2024-09-16T05:15:31.764302+0000","last_fullsized":"2024-09-16T05:15:31.764302+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'213","last_scrub_stamp":"2024-09-16T05:14:26.945487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.945487+0000","objects_scrubbed":4,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:33.783300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061793999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":33266,"num_write":11,"num_write_kb":5499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.8","version":"380'209","reported_seq":1556,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.713830+0000","last_change":"2024-09-16T05:15:31.715266+0000","last_active":"2024-09-16T05:15:31.713830+0000","last_peered":"2024-09-16T05:15:31.713830+0000","last_clean":"2024-09-16T05:15:31.713830+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:15:31.713830+0000","last_undegraded":"2024-09-16T05:15:31.713830+0000","last_fullsized":"2024-09-16T05:15:31.713830+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'153","last_scrub_stamp":"2024-09-16T05:14:26.593177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.593177+0000","objects_scrubbed":3,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:53.214381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0031367830000000002,"stat_sum":{"num_bytes":2790028,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3","version":"378'98","reported_seq":1434,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.770845+0000","last_change":"2024-09-16T05:15:29.771138+0000","last_active":"2024-09-16T05:15:29.770845+0000","last_peered":"2024-09-16T05:15:29.770845+0000","last_clean":"2024-09-16T05:15:29.770845+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:15:29.770845+0000","last_undegraded":"2024-09-16T05:15:29.770845+0000","last_fullsized":"2024-09-16T05:15:29.770845+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'68","last_scrub_stamp":"2024-09-16T05:14:44.975873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.975873+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:59.104958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001020967,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":979,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.712668+0000","last_change":"2024-09-16T05:14:53.849661+0000","last_active":"2024-09-16T05:15:31.712668+0000","last_peered":"2024-09-16T05:15:31.712668+0000","last_clean":"2024-09-16T05:15:31.712668+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:15:31.712668+0000","last_undegraded":"2024-09-16T05:15:31.712668+0000","last_fullsized":"2024-09-16T05:15:31.712668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:53.849576+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.849576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:02.708781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1350,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.770089+0000","last_change":"2024-09-16T05:15:29.770365+0000","last_active":"2024-09-16T05:15:29.770089+0000","last_peered":"2024-09-16T05:15:29.770089+0000","last_clean":"2024-09-16T05:15:29.770089+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:15:29.770089+0000","last_undegraded":"2024-09-16T05:15:29.770089+0000","last_fullsized":"2024-09-16T05:15:29.770089+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:14:25.970665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.970665+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:31.267886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082117799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":616,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763196+0000","last_change":"2024-09-16T05:15:18.812427+0000","last_active":"2024-09-16T05:15:29.763196+0000","last_peered":"2024-09-16T05:15:29.763196+0000","last_clean":"2024-09-16T05:15:29.763196+0000","last_became_active":"2024-09-16T05:13:34.762556+0000","last_became_peered":"2024-09-16T05:13:34.762556+0000","last_unstale":"2024-09-16T05:15:29.763196+0000","last_undegraded":"2024-09-16T05:15:29.763196+0000","last_fullsized":"2024-09-16T05:15:29.763196+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:18.812352+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:18.812352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:21.232575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":975,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.711820+0000","last_change":"2024-09-16T05:14:07.583425+0000","last_active":"2024-09-16T05:15:31.711820+0000","last_peered":"2024-09-16T05:15:31.711820+0000","last_clean":"2024-09-16T05:15:31.711820+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:15:31.711820+0000","last_undegraded":"2024-09-16T05:15:31.711820+0000","last_fullsized":"2024-09-16T05:15:31.711820+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:07.583346+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:07.583346+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1291,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.751705+0000","last_change":"2024-09-16T05:15:27.752226+0000","last_active":"2024-09-16T05:15:27.751705+0000","last_peered":"2024-09-16T05:15:27.751705+0000","last_clean":"2024-09-16T05:15:27.751705+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:15:27.751705+0000","last_undegraded":"2024-09-16T05:15:27.751705+0000","last_fullsized":"2024-09-16T05:15:27.751705+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:07.898675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.898675+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:37.590787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069755499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":980,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.746953+0000","last_change":"2024-09-16T05:15:19.711525+0000","last_active":"2024-09-16T05:15:27.746953+0000","last_peered":"2024-09-16T05:15:27.746953+0000","last_clean":"2024-09-16T05:15:27.746953+0000","last_became_active":"2024-09-16T05:15:19.711201+0000","last_became_peered":"2024-09-16T05:15:19.711201+0000","last_unstale":"2024-09-16T05:15:27.746953+0000","last_undegraded":"2024-09-16T05:15:27.746953+0000","last_fullsized":"2024-09-16T05:15:27.746953+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:45.298203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.298203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:07.821797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1386,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.717513+0000","last_change":"2024-09-16T05:15:31.717890+0000","last_active":"2024-09-16T05:15:31.717513+0000","last_peered":"2024-09-16T05:15:31.717513+0000","last_clean":"2024-09-16T05:15:31.717513+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:15:31.717513+0000","last_undegraded":"2024-09-16T05:15:31.717513+0000","last_fullsized":"2024-09-16T05:15:31.717513+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:06.755407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.755407+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:14.021642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053046760000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":592,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.764093+0000","last_change":"2024-09-16T05:15:29.764445+0000","last_active":"2024-09-16T05:15:29.764093+0000","last_peered":"2024-09-16T05:15:29.764093+0000","last_clean":"2024-09-16T05:15:29.764093+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:15:29.764093+0000","last_undegraded":"2024-09-16T05:15:29.764093+0000","last_fullsized":"2024-09-16T05:15:29.764093+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:27.213207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.213207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:52.872911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040033899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"1.4","version":"35'139","reported_seq":942,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764325+0000","last_change":"2024-09-16T05:15:10.945891+0000","last_active":"2024-09-16T05:15:31.764325+0000","last_peered":"2024-09-16T05:15:31.764325+0000","last_clean":"2024-09-16T05:15:31.764325+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:15:31.764325+0000","last_undegraded":"2024-09-16T05:15:31.764325+0000","last_fullsized":"2024-09-16T05:15:31.764325+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:10.945793+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.945793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:24.451282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1348,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.765507+0000","last_change":"2024-09-16T05:15:29.765766+0000","last_active":"2024-09-16T05:15:29.765507+0000","last_peered":"2024-09-16T05:15:29.765507+0000","last_clean":"2024-09-16T05:15:29.765507+0000","last_became_active":"2024-09-16T05:14:25.821885+0000","last_became_peered":"2024-09-16T05:14:25.821885+0000","last_unstale":"2024-09-16T05:15:29.765507+0000","last_undegraded":"2024-09-16T05:15:29.765507+0000","last_fullsized":"2024-09-16T05:15:29.765507+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:27.379983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.379983+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:53.804013+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002639275,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":721,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763613+0000","last_change":"2024-09-16T05:14:45.491959+0000","last_active":"2024-09-16T05:15:29.763613+0000","last_peered":"2024-09-16T05:15:29.763613+0000","last_clean":"2024-09-16T05:15:29.763613+0000","last_became_active":"2024-09-16T05:14:39.908958+0000","last_became_peered":"2024-09-16T05:14:39.908958+0000","last_unstale":"2024-09-16T05:15:29.763613+0000","last_undegraded":"2024-09-16T05:15:29.763613+0000","last_fullsized":"2024-09-16T05:15:29.763613+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:45.491884+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.491884+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.370020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"380'476","reported_seq":1800,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.761249+0000","last_change":"2024-09-16T05:15:29.761407+0000","last_active":"2024-09-16T05:15:29.761249+0000","last_peered":"2024-09-16T05:15:29.761249+0000","last_clean":"2024-09-16T05:15:29.761249+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:15:29.761249+0000","last_undegraded":"2024-09-16T05:15:29.761249+0000","last_fullsized":"2024-09-16T05:15:29.761249+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:14:26.230040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.230040+0000","objects_scrubbed":3,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:39.316057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002046389,"stat_sum":{"num_bytes":3176556,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":8850,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":617,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763208+0000","last_change":"2024-09-16T05:15:11.724435+0000","last_active":"2024-09-16T05:15:29.763208+0000","last_peered":"2024-09-16T05:15:29.763208+0000","last_clean":"2024-09-16T05:15:29.763208+0000","last_became_active":"2024-09-16T05:13:34.764187+0000","last_became_peered":"2024-09-16T05:13:34.764187+0000","last_unstale":"2024-09-16T05:15:29.763208+0000","last_undegraded":"2024-09-16T05:15:29.763208+0000","last_fullsized":"2024-09-16T05:15:29.763208+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:11.724301+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:11.724301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:40.115572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1430,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.764802+0000","last_change":"2024-09-16T05:15:29.765073+0000","last_active":"2024-09-16T05:15:29.764802+0000","last_peered":"2024-09-16T05:15:29.764802+0000","last_clean":"2024-09-16T05:15:29.764802+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:15:29.764802+0000","last_undegraded":"2024-09-16T05:15:29.764802+0000","last_fullsized":"2024-09-16T05:15:29.764802+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:28.350521+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.350521+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:42.882461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021758889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":801,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763556+0000","last_change":"2024-09-16T05:14:28.552017+0000","last_active":"2024-09-16T05:15:29.763556+0000","last_peered":"2024-09-16T05:15:29.763556+0000","last_clean":"2024-09-16T05:15:29.763556+0000","last_became_active":"2024-09-16T05:13:34.764109+0000","last_became_peered":"2024-09-16T05:13:34.764109+0000","last_unstale":"2024-09-16T05:15:29.763556+0000","last_undegraded":"2024-09-16T05:15:29.763556+0000","last_fullsized":"2024-09-16T05:15:29.763556+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:28.551941+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.551941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:45.247355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":1514,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-16T05:15:28.768496+0000","last_change":"2024-09-16T05:15:27.752443+0000","last_active":"2024-09-16T05:15:28.768496+0000","last_peered":"2024-09-16T05:15:28.768496+0000","last_clean":"2024-09-16T05:15:28.768496+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:15:28.768496+0000","last_undegraded":"2024-09-16T05:15:28.768496+0000","last_fullsized":"2024-09-16T05:15:28.768496+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:14:26.495648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.495648+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:31.626399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054349979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":982,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.769287+0000","last_change":"2024-09-16T05:15:15.084154+0000","last_active":"2024-09-16T05:15:29.769287+0000","last_peered":"2024-09-16T05:15:29.769287+0000","last_clean":"2024-09-16T05:15:29.769287+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:15:29.769287+0000","last_undegraded":"2024-09-16T05:15:29.769287+0000","last_fullsized":"2024-09-16T05:15:29.769287+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:15.084022+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.084022+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:28.500567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"380'461","reported_seq":1819,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764585+0000","last_change":"2024-09-16T05:15:31.764771+0000","last_active":"2024-09-16T05:15:31.764585+0000","last_peered":"2024-09-16T05:15:31.764585+0000","last_clean":"2024-09-16T05:15:31.764585+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:15:31.764585+0000","last_undegraded":"2024-09-16T05:15:31.764585+0000","last_fullsized":"2024-09-16T05:15:31.764585+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'419","last_scrub_stamp":"2024-09-16T05:14:25.924121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.924121+0000","objects_scrubbed":4,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:31.804801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00065622399999999996,"stat_sum":{"num_bytes":5126867,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":16965,"num_write":5,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":975,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.711796+0000","last_change":"2024-09-16T05:14:11.602321+0000","last_active":"2024-09-16T05:15:31.711796+0000","last_peered":"2024-09-16T05:15:31.711796+0000","last_clean":"2024-09-16T05:15:31.711796+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:15:31.711796+0000","last_undegraded":"2024-09-16T05:15:31.711796+0000","last_fullsized":"2024-09-16T05:15:31.711796+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:11.602237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:11.602237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1567,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763690+0000","last_change":"2024-09-16T05:15:29.763990+0000","last_active":"2024-09-16T05:15:29.763690+0000","last_peered":"2024-09-16T05:15:29.763690+0000","last_clean":"2024-09-16T05:15:29.763690+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:15:29.763690+0000","last_undegraded":"2024-09-16T05:15:29.763690+0000","last_fullsized":"2024-09-16T05:15:29.763690+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:14:26.418934+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.418934+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:35.911380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012627770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":982,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.760272+0000","last_change":"2024-09-16T05:15:14.182510+0000","last_active":"2024-09-16T05:15:29.760272+0000","last_peered":"2024-09-16T05:15:29.760272+0000","last_clean":"2024-09-16T05:15:29.760272+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:15:29.760272+0000","last_undegraded":"2024-09-16T05:15:29.760272+0000","last_fullsized":"2024-09-16T05:15:29.760272+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:14.182407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:14.182407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.292535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":474,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.764606+0000","last_change":"2024-09-16T05:15:29.764855+0000","last_active":"2024-09-16T05:15:29.764606+0000","last_peered":"2024-09-16T05:15:29.764606+0000","last_clean":"2024-09-16T05:15:29.764606+0000","last_became_active":"2024-09-16T05:13:31.747644+0000","last_became_peered":"2024-09-16T05:13:31.747644+0000","last_unstale":"2024-09-16T05:15:29.764606+0000","last_undegraded":"2024-09-16T05:15:29.764606+0000","last_fullsized":"2024-09-16T05:15:29.764606+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:14:49.576522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.576522+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:03.264965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015120380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.13","version":"181'15","reported_seq":970,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.762304+0000","last_change":"2024-09-16T05:15:29.762665+0000","last_active":"2024-09-16T05:15:29.762304+0000","last_peered":"2024-09-16T05:15:29.762304+0000","last_clean":"2024-09-16T05:15:29.762304+0000","last_became_active":"2024-09-16T05:13:25.681297+0000","last_became_peered":"2024-09-16T05:13:25.681297+0000","last_unstale":"2024-09-16T05:15:29.762304+0000","last_undegraded":"2024-09-16T05:15:29.762304+0000","last_fullsized":"2024-09-16T05:15:29.762304+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:49.160833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.160833+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:19.135853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003121745,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.14","version":"378'261","reported_seq":1202,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.762757+0000","last_change":"2024-09-16T05:15:29.763322+0000","last_active":"2024-09-16T05:15:29.762757+0000","last_peered":"2024-09-16T05:15:29.762757+0000","last_clean":"2024-09-16T05:15:29.762757+0000","last_became_active":"2024-09-16T05:13:25.684888+0000","last_became_peered":"2024-09-16T05:13:25.684888+0000","last_unstale":"2024-09-16T05:15:29.762757+0000","last_undegraded":"2024-09-16T05:15:29.762757+0000","last_fullsized":"2024-09-16T05:15:29.762757+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"342'239","last_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.110587+0000","objects_scrubbed":4,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:11.102021+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0035899730000000002,"stat_sum":{"num_bytes":3142942,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":14332,"num_write":5,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.15","version":"380'471","reported_seq":1817,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.762235+0000","last_change":"2024-09-16T05:15:29.762510+0000","last_active":"2024-09-16T05:15:29.762235+0000","last_peered":"2024-09-16T05:15:29.762235+0000","last_clean":"2024-09-16T05:15:29.762235+0000","last_became_active":"2024-09-16T05:13:25.680026+0000","last_became_peered":"2024-09-16T05:13:25.680026+0000","last_unstale":"2024-09-16T05:15:29.762235+0000","last_undegraded":"2024-09-16T05:15:29.762235+0000","last_fullsized":"2024-09-16T05:15:29.762235+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"357'436","last_scrub_stamp":"2024-09-16T05:15:07.101221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.101221+0000","objects_scrubbed":5,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:11.081786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031140619999999999,"stat_sum":{"num_bytes":6735275,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":41972,"num_write":30,"num_write_kb":12050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.16","version":"378'114","reported_seq":1091,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.769742+0000","last_change":"2024-09-16T05:15:29.770067+0000","last_active":"2024-09-16T05:15:29.769742+0000","last_peered":"2024-09-16T05:15:29.769742+0000","last_clean":"2024-09-16T05:15:29.769742+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:15:29.769742+0000","last_undegraded":"2024-09-16T05:15:29.769742+0000","last_fullsized":"2024-09-16T05:15:29.769742+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"304'74","last_scrub_stamp":"2024-09-16T05:14:29.883455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:29.883455+0000","objects_scrubbed":1,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:34.361324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054708899999999995,"stat_sum":{"num_bytes":3142942,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":6148,"num_write":3,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.17","version":"0'0","reported_seq":1154,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.716288+0000","last_change":"2024-09-16T05:15:31.716714+0000","last_active":"2024-09-16T05:15:31.716288+0000","last_peered":"2024-09-16T05:15:31.716288+0000","last_clean":"2024-09-16T05:15:31.716288+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:15:31.716288+0000","last_undegraded":"2024-09-16T05:15:31.716288+0000","last_fullsized":"2024-09-16T05:15:31.716288+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:43.982972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:43.982972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:07.471726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001213754,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1363,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.761472+0000","last_change":"2024-09-16T05:15:29.761824+0000","last_active":"2024-09-16T05:15:29.761472+0000","last_peered":"2024-09-16T05:15:29.761472+0000","last_clean":"2024-09-16T05:15:29.761472+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:15:29.761472+0000","last_undegraded":"2024-09-16T05:15:29.761472+0000","last_fullsized":"2024-09-16T05:15:29.761472+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:17.188273+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:17.188273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.802882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002422497,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.19","version":"182'60","reported_seq":956,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.747196+0000","last_change":"2024-09-16T05:15:27.747468+0000","last_active":"2024-09-16T05:15:27.747196+0000","last_peered":"2024-09-16T05:15:27.747196+0000","last_clean":"2024-09-16T05:15:27.747196+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:15:27.747196+0000","last_undegraded":"2024-09-16T05:15:27.747196+0000","last_fullsized":"2024-09-16T05:15:27.747196+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:06.893298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.893298+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.346953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056415800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.1a","version":"378'596","reported_seq":1908,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-16T05:15:28.768418+0000","last_change":"2024-09-16T05:15:27.754301+0000","last_active":"2024-09-16T05:15:28.768418+0000","last_peered":"2024-09-16T05:15:28.768418+0000","last_clean":"2024-09-16T05:15:28.768418+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:15:28.768418+0000","last_undegraded":"2024-09-16T05:15:28.768418+0000","last_fullsized":"2024-09-16T05:15:28.768418+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'527","last_scrub_stamp":"2024-09-16T05:15:10.511970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.511970+0000","objects_scrubbed":9,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:27.876604+0000","scrub_duration":3,"objects_trimmed":3,"snaptrim_duration":0.0076844060000000004,"stat_sum":{"num_bytes":9446945,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":56,"num_read_kb":52510,"num_write":35,"num_write_kb":15546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.1b","version":"382'332","reported_seq":1713,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.765946+0000","last_change":"2024-09-16T05:15:31.766093+0000","last_active":"2024-09-16T05:15:31.765946+0000","last_peered":"2024-09-16T05:15:31.765946+0000","last_clean":"2024-09-16T05:15:31.765946+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:15:31.765946+0000","last_undegraded":"2024-09-16T05:15:31.765946+0000","last_fullsized":"2024-09-16T05:15:31.765946+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"347'319","last_scrub_stamp":"2024-09-16T05:15:08.957118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.957118+0000","objects_scrubbed":3,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.578187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019599259999999999,"stat_sum":{"num_bytes":5298036,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":29313,"num_write":9,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.1c","version":"361'369","reported_seq":1771,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763898+0000","last_change":"2024-09-16T05:15:29.764285+0000","last_active":"2024-09-16T05:15:29.763898+0000","last_peered":"2024-09-16T05:15:29.763898+0000","last_clean":"2024-09-16T05:15:29.763898+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:15:29.763898+0000","last_undegraded":"2024-09-16T05:15:29.763898+0000","last_fullsized":"2024-09-16T05:15:29.763898+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'364","last_scrub_stamp":"2024-09-16T05:15:08.759106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.759106+0000","objects_scrubbed":2,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.554347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093929300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.1d","version":"380'624","reported_seq":2048,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.766516+0000","last_change":"2024-09-16T05:15:29.766752+0000","last_active":"2024-09-16T05:15:29.766516+0000","last_peered":"2024-09-16T05:15:29.766516+0000","last_clean":"2024-09-16T05:15:29.766516+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:15:29.766516+0000","last_undegraded":"2024-09-16T05:15:29.766516+0000","last_fullsized":"2024-09-16T05:15:29.766516+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'584","last_scrub_stamp":"2024-09-16T05:15:16.833446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:16.833446+0000","objects_scrubbed":8,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.963147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0033001979999999998,"stat_sum":{"num_bytes":12407718,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":34800,"num_write":29,"num_write_kb":17614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.1e","version":"378'347","reported_seq":1700,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.751296+0000","last_change":"2024-09-16T05:15:27.751722+0000","last_active":"2024-09-16T05:15:27.751296+0000","last_peered":"2024-09-16T05:15:27.751296+0000","last_clean":"2024-09-16T05:15:27.751296+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:15:27.751296+0000","last_undegraded":"2024-09-16T05:15:27.751296+0000","last_fullsized":"2024-09-16T05:15:27.751296+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"362'335","last_scrub_stamp":"2024-09-16T05:15:12.926534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:12.926534+0000","objects_scrubbed":3,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:18.971383+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0044370340000000003,"stat_sum":{"num_bytes":2487502,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":14641,"num_write":5,"num_write_kb":3828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":1576,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.761617+0000","last_change":"2024-09-16T05:15:29.761950+0000","last_active":"2024-09-16T05:15:29.761617+0000","last_peered":"2024-09-16T05:15:29.761617+0000","last_clean":"2024-09-16T05:15:29.761617+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:15:29.761617+0000","last_undegraded":"2024-09-16T05:15:29.761617+0000","last_fullsized":"2024-09-16T05:15:29.761617+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:14:52.883562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.883562+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:04.500201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00082745099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.20","version":"382'132","reported_seq":1506,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.718539+0000","last_change":"2024-09-16T05:15:31.715740+0000","last_active":"2024-09-16T05:15:31.718539+0000","last_peered":"2024-09-16T05:15:31.718539+0000","last_clean":"2024-09-16T05:15:31.718539+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:15:31.718539+0000","last_undegraded":"2024-09-16T05:15:31.718539+0000","last_fullsized":"2024-09-16T05:15:31.718539+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'128","last_scrub_stamp":"2024-09-16T05:15:31.623695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.623695+0000","objects_scrubbed":2,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:33.562506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038817470000000001,"stat_sum":{"num_bytes":4238337,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":70060,"num_write":16,"num_write_kb":6808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":1290,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.752321+0000","last_change":"2024-09-16T05:15:27.752778+0000","last_active":"2024-09-16T05:15:27.752321+0000","last_peered":"2024-09-16T05:15:27.752321+0000","last_clean":"2024-09-16T05:15:27.752321+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:15:27.752321+0000","last_undegraded":"2024-09-16T05:15:27.752321+0000","last_fullsized":"2024-09-16T05:15:27.752321+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.848788+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:13.563750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001134318,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":474,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.765009+0000","last_change":"2024-09-16T05:15:29.765262+0000","last_active":"2024-09-16T05:15:29.765009+0000","last_peered":"2024-09-16T05:15:29.765009+0000","last_clean":"2024-09-16T05:15:29.765009+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:15:29.765009+0000","last_undegraded":"2024-09-16T05:15:29.765009+0000","last_fullsized":"2024-09-16T05:15:29.765009+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:14:07.323272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:07.323272+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016726309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":823,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.747562+0000","last_change":"2024-09-16T05:15:27.747914+0000","last_active":"2024-09-16T05:15:27.747562+0000","last_peered":"2024-09-16T05:15:27.747562+0000","last_clean":"2024-09-16T05:15:27.747562+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:15:27.747562+0000","last_undegraded":"2024-09-16T05:15:27.747562+0000","last_fullsized":"2024-09-16T05:15:27.747562+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:14:46.760890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.760890+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.828228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074141400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":649,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.762584+0000","last_change":"2024-09-16T05:15:29.762921+0000","last_active":"2024-09-16T05:15:29.762584+0000","last_peered":"2024-09-16T05:15:29.762584+0000","last_clean":"2024-09-16T05:15:29.762584+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:15:29.762584+0000","last_undegraded":"2024-09-16T05:15:29.762584+0000","last_fullsized":"2024-09-16T05:15:29.762584+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:31.611624+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:31.611624+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.611624+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:49.896821+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0.0032983439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]},{"pgid":"2.25","version":"378'620","reported_seq":1890,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-16T05:15:28.768712+0000","last_change":"2024-09-16T05:15:27.751650+0000","last_active":"2024-09-16T05:15:28.768712+0000","last_peered":"2024-09-16T05:15:28.768712+0000","last_clean":"2024-09-16T05:15:28.768712+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:15:28.768712+0000","last_undegraded":"2024-09-16T05:15:28.768712+0000","last_fullsized":"2024-09-16T05:15:28.768712+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"310'536","last_scrub_stamp":"2024-09-16T05:14:31.574482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.574482+0000","objects_scrubbed":8,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:55.410199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0053007979999999998,"stat_sum":{"num_bytes":13563348,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":43,"num_read_kb":57574,"num_write":25,"num_write_kb":10249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":650,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.716848+0000","last_change":"2024-09-16T05:15:31.717441+0000","last_active":"2024-09-16T05:15:31.716848+0000","last_peered":"2024-09-16T05:15:31.716848+0000","last_clean":"2024-09-16T05:15:31.716848+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:15:31.716848+0000","last_undegraded":"2024-09-16T05:15:31.716848+0000","last_fullsized":"2024-09-16T05:15:31.716848+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:27.631576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.631576+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:32.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050394630000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":662,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.716330+0000","last_change":"2024-09-16T05:15:31.716734+0000","last_active":"2024-09-16T05:15:31.716330+0000","last_peered":"2024-09-16T05:15:31.716330+0000","last_clean":"2024-09-16T05:15:31.716330+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:15:31.716330+0000","last_undegraded":"2024-09-16T05:15:31.716330+0000","last_fullsized":"2024-09-16T05:15:31.716330+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:30.713398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.713398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:46.971140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044151620000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1424,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.747122+0000","last_change":"2024-09-16T05:15:27.747496+0000","last_active":"2024-09-16T05:15:27.747122+0000","last_peered":"2024-09-16T05:15:27.747122+0000","last_clean":"2024-09-16T05:15:27.747122+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:15:27.747122+0000","last_undegraded":"2024-09-16T05:15:27.747122+0000","last_fullsized":"2024-09-16T05:15:27.747122+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:14:45.747155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.747155+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:47.938306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059571099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.29","version":"380'536","reported_seq":1915,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.714714+0000","last_change":"2024-09-16T05:15:31.715292+0000","last_active":"2024-09-16T05:15:31.714714+0000","last_peered":"2024-09-16T05:15:31.714714+0000","last_clean":"2024-09-16T05:15:31.714714+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:15:31.714714+0000","last_undegraded":"2024-09-16T05:15:31.714714+0000","last_fullsized":"2024-09-16T05:15:31.714714+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"310'362","last_scrub_stamp":"2024-09-16T05:14:27.037060+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.037060+0000","objects_scrubbed":12,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:33.419700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077023399999999996,"stat_sum":{"num_bytes":14413657,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":25964,"num_write":68,"num_write_kb":21290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1652,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.712462+0000","last_change":"2024-09-16T05:15:31.712729+0000","last_active":"2024-09-16T05:15:31.712462+0000","last_peered":"2024-09-16T05:15:31.712462+0000","last_clean":"2024-09-16T05:15:31.712462+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:15:31.712462+0000","last_undegraded":"2024-09-16T05:15:31.712462+0000","last_fullsized":"2024-09-16T05:15:31.712462+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:14:33.725061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:33.725061+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:48.150836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077903200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2b","version":"372'201","reported_seq":1448,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-16T05:15:27.752416+0000","last_change":"2024-09-16T05:15:27.752766+0000","last_active":"2024-09-16T05:15:27.752416+0000","last_peered":"2024-09-16T05:15:27.752416+0000","last_clean":"2024-09-16T05:15:27.752416+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:15:27.752416+0000","last_undegraded":"2024-09-16T05:15:27.752416+0000","last_fullsized":"2024-09-16T05:15:27.752416+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"330'198","last_scrub_stamp":"2024-09-16T05:14:46.815272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.815272+0000","objects_scrubbed":2,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:11.766501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00094418099999999999,"stat_sum":{"num_bytes":3142942,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":8,"num_write_kb":7455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":1461,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-16T05:15:28.768327+0000","last_change":"2024-09-16T05:15:27.753731+0000","last_active":"2024-09-16T05:15:28.768327+0000","last_peered":"2024-09-16T05:15:28.768327+0000","last_clean":"2024-09-16T05:15:28.768327+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:15:28.768327+0000","last_undegraded":"2024-09-16T05:15:28.768327+0000","last_fullsized":"2024-09-16T05:15:28.768327+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:14:27.492665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.492665+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:43.724590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006229648,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"1"}]},{"pgid":"2.2d","version":"377'326","reported_seq":1652,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.715343+0000","last_change":"2024-09-16T05:15:31.715729+0000","last_active":"2024-09-16T05:15:31.715343+0000","last_peered":"2024-09-16T05:15:31.715343+0000","last_clean":"2024-09-16T05:15:31.715343+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:15:31.715343+0000","last_undegraded":"2024-09-16T05:15:31.715343+0000","last_fullsized":"2024-09-16T05:15:31.715343+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'307","last_scrub_stamp":"2024-09-16T05:14:30.116429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.116429+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.944115+0000","scrub_duration":444,"objects_trimmed":0,"snaptrim_duration":0.0035826769999999998,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":4589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2e","version":"382'392","reported_seq":1709,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.718733+0000","last_change":"2024-09-16T05:15:31.718982+0000","last_active":"2024-09-16T05:15:31.718733+0000","last_peered":"2024-09-16T05:15:31.718733+0000","last_clean":"2024-09-16T05:15:31.718733+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:15:31.718733+0000","last_undegraded":"2024-09-16T05:15:31.718733+0000","last_fullsized":"2024-09-16T05:15:31.718733+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"337'278","last_scrub_stamp":"2024-09-16T05:14:47.811642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.811642+0000","objects_scrubbed":7,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.306241+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0066353510000000003,"stat_sum":{"num_bytes":8393552,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":7127,"num_write":34,"num_write_kb":7817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2f","version":"378'302","reported_seq":1678,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-16T05:15:29.763389+0000","last_change":"2024-09-16T05:15:29.764022+0000","last_active":"2024-09-16T05:15:29.763389+0000","last_peered":"2024-09-16T05:15:29.763389+0000","last_clean":"2024-09-16T05:15:29.763389+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:15:29.763389+0000","last_undegraded":"2024-09-16T05:15:29.763389+0000","last_fullsized":"2024-09-16T05:15:29.763389+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:53.110450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.110450+0000","objects_scrubbed":3,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:01.045747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0036130149999999998,"stat_sum":{"num_bytes":3159782,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":7,"num_write_kb":4826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"76","length":"3"},{"start":"7a","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":161045844,"num_objects":68,"num_object_clones":25,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":4,"num_read":641,"num_read_kb":648688,"num_write":463,"num_write_kb":191010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204787712,"data_stored":238298180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15419,"ondisk_log_size":15419,"up":152,"acting":152,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089421,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65616,"kb_used_data":30268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93679536,"statfs":{"total":95995035648,"available":95927844864,"internally_reserved":0,"allocated":30994432,"data_stored":34582195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52616,"kb_used_data":17260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93692536,"statfs":{"total":95995035648,"available":95941156864,"internally_reserved":0,"allocated":17674240,"data_stored":17889726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514585,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67448,"kb_used_data":27940,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93677704,"statfs":{"total":95995035648,"available":95925968896,"internally_reserved":0,"allocated":28610560,"data_stored":30821932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841626,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50164,"kb_used_data":14804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93694988,"statfs":{"total":95995035648,"available":95943667712,"internally_reserved":0,"allocated":15159296,"data_stored":18870321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103264,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68980,"kb_used_data":29472,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93676172,"statfs":{"total":95995035648,"available":95924400128,"internally_reserved":0,"allocated":30179328,"data_stored":33499941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397607,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67004,"kb_used_data":27492,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93678148,"statfs":{"total":95995035648,"available":95926423552,"internally_reserved":0,"allocated":28151808,"data_stored":31316440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757354,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46436,"kb_used_data":15240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93698716,"statfs":{"total":95995035648,"available":95947485184,"internally_reserved":0,"allocated":15605760,"data_stored":14421273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47516,"kb_used_data":16320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93697636,"statfs":{"total":95995035648,"available":95946379264,"internally_reserved":0,"allocated":16711680,"data_stored":14970660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117108,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75288,"kb_used_data":39932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93669864,"statfs":{"total":95995035648,"available":95917940736,"internally_reserved":0,"allocated":40890368,"data_stored":44404655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240395,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37992,"kb_used_data":9872,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93707160,"statfs":{"total":95995035648,"available":95956131840,"internally_reserved":0,"allocated":10108928,"data_stored":8447911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52212,"kb_used_data":16860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93692940,"statfs":{"total":95995035648,"available":95941570560,"internally_reserved":0,"allocated":17264640,"data_stored":17240734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50172,"kb_used_data":10656,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93694980,"statfs":{"total":95995035648,"available":95943659520,"internally_reserved":0,"allocated":10911744,"data_stored":10106680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5050368,"data_stored":5889133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12734464,"data_stored":14344947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5337088,"data_stored":5306422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33914880,"data_stored":36817057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12169216,"data_stored":12066372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10870784,"data_stored":11324238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23597056,"data_stored":28403635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":27451392,"data_stored":34220742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10326016,"data_stored":15663840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24080384,"data_stored":27926145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":13144064,"data_stored":14993939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26112000,"data_stored":31341710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:15:35.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:15:35.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:34 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:15:35.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:34 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:15:35.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:34 smithi204 ceph-mon[30933]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:35.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:34 smithi204 ceph-mon[30933]: osdmap e385: 12 total, 12 up, 12 in 2024-09-16T05:15:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:34 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:15:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:34 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:15:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:34 smithi032 ceph-mon[27388]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:34 smithi032 ceph-mon[27388]: osdmap e385: 12 total, 12 up, 12 in 2024-09-16T05:15:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:35.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:35.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:35.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:35.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:35.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:35.263 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:35.263 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:35.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:35.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:35.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:35.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:35.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:35.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:35.585 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:35.586 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:35.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:35.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:35.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:35.707 INFO:tasks.rados.rados.2.smithi204.stdout:1207: done (2 left) 2024-09-16T05:15:35.707 INFO:tasks.rados.rados.2.smithi204.stdout:1208: done (1 left) 2024-09-16T05:15:35.708 INFO:tasks.rados.rados.2.smithi204.stdout:1209: done (0 left) 2024-09-16T05:15:35.708 INFO:tasks.rados.rados.2.smithi204.stdout:1210: snap_create 2024-09-16T05:15:35.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:35.758 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:35.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:35.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:35.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:15:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:35.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:35.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:35.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:36.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:36.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:36.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:35 smithi032 ceph-mon[27388]: pgmap v738: 87 pgs: 87 active+clean; 154 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-16T05:15:36.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:35 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:15:36.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:35 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:15:36.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:35 smithi032 ceph-mon[27388]: osdmap e386: 12 total, 12 up, 12 in 2024-09-16T05:15:36.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:36.123 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:36.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:35 smithi204 ceph-mon[30933]: pgmap v738: 87 pgs: 87 active+clean; 154 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-16T05:15:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:35 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:15:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:35 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:15:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:35 smithi204 ceph-mon[30933]: osdmap e386: 12 total, 12 up, 12 in 2024-09-16T05:15:36.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:35 smithi196 ceph-mon[30802]: pgmap v738: 87 pgs: 87 active+clean; 154 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2024-09-16T05:15:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:35 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:15:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:35 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:15:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:35 smithi196 ceph-mon[30802]: osdmap e386: 12 total, 12 up, 12 in 2024-09-16T05:15:36.256 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:36.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:36.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:36.361 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:36.361 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:36.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:36.384 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:36.384 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:36.405 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:36.405 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:36.425 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:36.426 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:36.447 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:36.447 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:36.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:36.468 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:36.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:36.489 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:36.508 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:36.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:36.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:36.510 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:36.518 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:15:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:15:36.533 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:36.555 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:36.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:36.577 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:36.577 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:36.599 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:36.599 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:36.601 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:15:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:15:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:15:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:15:36.601 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:15:36.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:36.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:36.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:36.705 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:15:36.709 INFO:tasks.rados.rados.2.smithi204.stdout:1210: done (0 left) 2024-09-16T05:15:36.709 INFO:tasks.rados.rados.2.smithi204.stdout:1211: write oid 47 current snap is 130 2024-09-16T05:15:36.709 INFO:tasks.rados.rados.2.smithi204.stdout:1211: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-16T05:15:36.716 INFO:tasks.rados.rados.2.smithi204.stdout:1211: writing smithi20457400-47 from 561466 to 1352649 tid 1 2024-09-16T05:15:36.723 INFO:tasks.rados.rados.2.smithi204.stdout:1211: writing smithi20457400-47 from 2081113 to 2673917 tid 2 2024-09-16T05:15:36.728 INFO:tasks.rados.rados.2.smithi204.stdout:1211: writing smithi20457400-47 from 3095506 to 3361466 tid 3 2024-09-16T05:15:36.728 INFO:tasks.rados.rados.2.smithi204.stdout:1212: copy_from oid 32 from oid 23 current snap is 130 2024-09-16T05:15:36.728 INFO:tasks.rados.rados.2.smithi204.stdout:1211: finishing write tid 1 to smithi20457400-47 2024-09-16T05:15:36.728 INFO:tasks.rados.rados.2.smithi204.stdout:1211: finishing write tid 2 to smithi20457400-47 2024-09-16T05:15:36.728 INFO:tasks.rados.rados.2.smithi204.stdout:1213: rollback oid 12 current snap is 130 2024-09-16T05:15:36.728 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 128 2024-09-16T05:15:36.728 INFO:tasks.rados.rados.2.smithi204.stdout:1214: delete oid 20 current snap is 130 2024-09-16T05:15:36.730 INFO:tasks.rados.rados.2.smithi204.stdout:1211: finishing write tid 3 to smithi20457400-47 2024-09-16T05:15:36.730 INFO:tasks.rados.rados.2.smithi204.stdout:1214: done (3 left) 2024-09-16T05:15:36.730 INFO:tasks.rados.rados.2.smithi204.stdout:1215: read oid 13 snap -1 2024-09-16T05:15:36.730 INFO:tasks.rados.rados.2.smithi204.stdout:1215: expect (ObjNum 417 snap 124 seq_num 417) 2024-09-16T05:15:36.730 INFO:tasks.rados.rados.2.smithi204.stdout:1216: snap_create 2024-09-16T05:15:36.730 INFO:tasks.rados.rados.2.smithi204.stdout:1211: finishing write tid 4 to smithi20457400-47 2024-09-16T05:15:36.731 INFO:tasks.rados.rados.2.smithi204.stdout:1211: finishing write tid 5 to smithi20457400-47 2024-09-16T05:15:36.731 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 324 (ObjNum 438 snap 130 seq_num 438) dirty exists 2024-09-16T05:15:36.731 INFO:tasks.rados.rados.2.smithi204.stdout:1211: left oid 47 (ObjNum 438 snap 130 seq_num 438) 2024-09-16T05:15:36.731 INFO:tasks.rados.rados.2.smithi204.stdout:1213: finishing rollback tid 0 to smithi20457400-12 2024-09-16T05:15:36.731 INFO:tasks.rados.rados.2.smithi204.stdout:1213: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:15:36.731 INFO:tasks.rados.rados.2.smithi204.stdout:1213: finishing rollback tid 2 to smithi20457400-12 2024-09-16T05:15:36.732 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 604 (ObjNum 375 snap 109 seq_num 375) dirty exists 2024-09-16T05:15:36.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:36.737 INFO:tasks.rados.rados.2.smithi204.stdout:1215: expect (ObjNum 417 snap 124 seq_num 417) 2024-09-16T05:15:36.759 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:36.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:36.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:36.866 INFO:tasks.rados.rados.2.smithi204.stdout:1212: finishing copy_from racing read to smithi20457400-32 2024-09-16T05:15:36.866 INFO:tasks.rados.rados.2.smithi204.stdout:1212: finishing copy_from to smithi20457400-32 2024-09-16T05:15:36.866 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 281 (ObjNum 328 snap 91 seq_num 328) dirty exists 2024-09-16T05:15:36.882 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:36.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:36.897 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3636925079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:36.897 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/157707664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:36.897 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:36 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:36.898 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:36 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:15:36.898 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3636925079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:36.898 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:36 smithi032 ceph-mon[27388]: osdmap e387: 12 total, 12 up, 12 in 2024-09-16T05:15:36.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:37.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:37.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:37.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:37.131 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:37.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3636925079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/157707664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:36 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:36 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:15:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3636925079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:36 smithi204 ceph-mon[30933]: osdmap e387: 12 total, 12 up, 12 in 2024-09-16T05:15:37.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3636925079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:37.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/157707664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:37.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:36 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:37.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:36 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:15:37.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3636925079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:37.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:36 smithi196 ceph-mon[30802]: osdmap e387: 12 total, 12 up, 12 in 2024-09-16T05:15:37.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:15:37.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:37.263 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:37.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:37.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:37.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:37.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:37.492 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:37.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:37.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:37.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:37.644 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:37.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:37.711 INFO:tasks.rados.rados.2.smithi204.stdout:1211: done (4 left) 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1212: done (3 left) 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1213: done (2 left) 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1215: done (1 left) 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1216: done (0 left) 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1217: read oid 46 snap 123 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1217: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1218: read oid 43 snap -1 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1218: expect deleted 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1219: write oid 5 current snap is 131 2024-09-16T05:15:37.712 INFO:tasks.rados.rados.2.smithi204.stdout:1219: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-16T05:15:37.717 INFO:tasks.rados.rados.2.smithi204.stdout:1219: writing smithi20457400-5 from 595080 to 1136761 tid 1 2024-09-16T05:15:37.724 INFO:tasks.rados.rados.2.smithi204.stdout:1219: writing smithi20457400-5 from 1798077 to 2394550 tid 2 2024-09-16T05:15:37.730 INFO:tasks.rados.rados.2.smithi204.stdout:1219: writing smithi20457400-5 from 3150352 to 3395080 tid 3 2024-09-16T05:15:37.730 INFO:tasks.rados.rados.2.smithi204.stdout:1220: write oid 34 current snap is 131 2024-09-16T05:15:37.730 INFO:tasks.rados.rados.2.smithi204.stdout:1220: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-09-16T05:15:37.737 INFO:tasks.rados.rados.2.smithi204.stdout:1220: writing smithi20457400-34 from 611887 to 1345170 tid 1 2024-09-16T05:15:37.743 INFO:tasks.rados.rados.2.smithi204.stdout:1220: writing smithi20457400-34 from 1972912 to 2429396 tid 2 2024-09-16T05:15:37.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:37.749 INFO:tasks.rados.rados.2.smithi204.stdout:1220: writing smithi20457400-34 from 2894128 to 3411887 tid 3 2024-09-16T05:15:37.749 INFO:tasks.rados.rados.2.smithi204.stdout:1221: snap_create 2024-09-16T05:15:37.749 INFO:tasks.rados.rados.2.smithi204.stdout:1217: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:15:37.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:37.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:37.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:37.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:37.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:37 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:15:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:37 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:15:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:37 smithi032 ceph-mon[27388]: pgmap v741: 87 pgs: 87 active+clean; 160 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 7 op/s 2024-09-16T05:15:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:37 smithi032 ceph-mon[27388]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:37 smithi032 ceph-mon[27388]: osdmap e388: 12 total, 12 up, 12 in 2024-09-16T05:15:37.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:38.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:38.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:38.015 INFO:tasks.rados.rados.2.smithi204.stdout:1219: finishing write tid 1 to smithi20457400-5 2024-09-16T05:15:38.015 INFO:tasks.rados.rados.2.smithi204.stdout:1219: finishing write tid 2 to smithi20457400-5 2024-09-16T05:15:38.015 INFO:tasks.rados.rados.2.smithi204.stdout:1219: finishing write tid 3 to smithi20457400-5 2024-09-16T05:15:38.015 INFO:tasks.rados.rados.2.smithi204.stdout:1219: finishing write tid 4 to smithi20457400-5 2024-09-16T05:15:38.015 INFO:tasks.rados.rados.2.smithi204.stdout:1219: finishing write tid 5 to smithi20457400-5 2024-09-16T05:15:38.015 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 326 (ObjNum 440 snap 131 seq_num 440) dirty exists 2024-09-16T05:15:38.016 INFO:tasks.rados.rados.2.smithi204.stdout:1219: left oid 5 (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:15:38.016 INFO:tasks.rados.rados.2.smithi204.stdout:1220: finishing write tid 1 to smithi20457400-34 2024-09-16T05:15:38.016 INFO:tasks.rados.rados.2.smithi204.stdout:1220: finishing write tid 2 to smithi20457400-34 2024-09-16T05:15:38.016 INFO:tasks.rados.rados.2.smithi204.stdout:1220: finishing write tid 3 to smithi20457400-34 2024-09-16T05:15:38.016 INFO:tasks.rados.rados.2.smithi204.stdout:1220: finishing write tid 4 to smithi20457400-34 2024-09-16T05:15:38.016 INFO:tasks.rados.rados.2.smithi204.stdout:1220: finishing write tid 5 to smithi20457400-34 2024-09-16T05:15:38.016 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 465 (ObjNum 441 snap 131 seq_num 441) dirty exists 2024-09-16T05:15:38.016 INFO:tasks.rados.rados.2.smithi204.stdout:1220: left oid 34 (ObjNum 441 snap 131 seq_num 441) 2024-09-16T05:15:38.051 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":741,"stamp":"2024-09-16T05:15:36.097664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":167888548,"num_objects":74,"num_object_clones":28,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":3,"num_read":644,"num_read_kb":648689,"num_write":475,"num_write_kb":195626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16975,"ondisk_log_size":16975,"up":174,"acting":174,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":674740,"kb_used_data":249432,"kb_used_omap":27,"kb_used_meta":424996,"kb_avail":1124267084,"statfs":{"total":1151940427776,"available":1151249494016,"internally_reserved":0,"allocated":255418368,"data_stored":274385817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":435196071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13956078,"num_objects":-4,"num_object_clones":-4,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":-2,"num_read":13,"num_read_kb":10020,"num_write":17,"num_write_kb":6246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000734"},"pg_stats":[{"pgid":"2.4b","version":"244'177","reported_seq":1547,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724076+0000","last_change":"2024-09-16T05:15:32.707678+0000","last_active":"2024-09-16T05:15:34.724076+0000","last_peered":"2024-09-16T05:15:34.724076+0000","last_clean":"2024-09-16T05:15:34.724076+0000","last_became_active":"2024-09-16T05:14:30.853998+0000","last_became_peered":"2024-09-16T05:14:30.853998+0000","last_unstale":"2024-09-16T05:15:34.724076+0000","last_undegraded":"2024-09-16T05:15:34.724076+0000","last_fullsized":"2024-09-16T05:15:34.724076+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:14:32.337052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:32.337052+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:42.415926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064955899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.4a","version":"378'461","reported_seq":1794,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769902+0000","last_change":"2024-09-16T05:15:33.269266+0000","last_active":"2024-09-16T05:15:35.769902+0000","last_peered":"2024-09-16T05:15:35.769902+0000","last_clean":"2024-09-16T05:15:35.769902+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:15:35.769902+0000","last_undegraded":"2024-09-16T05:15:35.769902+0000","last_fullsized":"2024-09-16T05:15:35.769902+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"309'419","last_scrub_stamp":"2024-09-16T05:14:31.471678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.471678+0000","objects_scrubbed":2,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.361322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051434600000000005,"stat_sum":{"num_bytes":2806835,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.49","version":"245'220","reported_seq":1589,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.768830+0000","last_change":"2024-09-16T05:15:33.091295+0000","last_active":"2024-09-16T05:15:35.768830+0000","last_peered":"2024-09-16T05:15:35.768830+0000","last_clean":"2024-09-16T05:15:35.768830+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:15:35.768830+0000","last_undegraded":"2024-09-16T05:15:35.768830+0000","last_fullsized":"2024-09-16T05:15:35.768830+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:14:30.917243+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.917243+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:47.943132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054279999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.48","version":"378'179","reported_seq":1510,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.713745+0000","last_change":"2024-09-16T05:15:32.997315+0000","last_active":"2024-09-16T05:15:34.713745+0000","last_peered":"2024-09-16T05:15:34.713745+0000","last_clean":"2024-09-16T05:15:34.713745+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:15:34.713745+0000","last_undegraded":"2024-09-16T05:15:34.713745+0000","last_fullsized":"2024-09-16T05:15:34.713745+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'153","last_scrub_stamp":"2024-09-16T05:14:28.221076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.221076+0000","objects_scrubbed":1,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:48.495914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018136280000000001,"stat_sum":{"num_bytes":3142942,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":20676,"num_write":1,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.47","version":"0'0","reported_seq":357,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.870415+0000","last_change":"2024-09-16T05:15:32.739389+0000","last_active":"2024-09-16T05:15:34.870415+0000","last_peered":"2024-09-16T05:15:34.870415+0000","last_clean":"2024-09-16T05:15:34.870415+0000","last_became_active":"2024-09-16T05:15:16.867821+0000","last_became_peered":"2024-09-16T05:15:16.867821+0000","last_unstale":"2024-09-16T05:15:34.870415+0000","last_undegraded":"2024-09-16T05:15:34.870415+0000","last_fullsized":"2024-09-16T05:15:34.870415+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:28.987418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.987418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:50.033232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00029241899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.46","version":"116'9","reported_seq":1374,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724158+0000","last_change":"2024-09-16T05:15:32.708489+0000","last_active":"2024-09-16T05:15:34.724158+0000","last_peered":"2024-09-16T05:15:34.724158+0000","last_clean":"2024-09-16T05:15:34.724158+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:15:34.724158+0000","last_undegraded":"2024-09-16T05:15:34.724158+0000","last_fullsized":"2024-09-16T05:15:34.724158+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:29.313711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:29.313711+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:50.861068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014462800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.45","version":"305'464","reported_seq":1801,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724114+0000","last_change":"2024-09-16T05:15:32.707971+0000","last_active":"2024-09-16T05:15:34.724114+0000","last_peered":"2024-09-16T05:15:34.724114+0000","last_clean":"2024-09-16T05:15:34.724114+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:15:34.724114+0000","last_undegraded":"2024-09-16T05:15:34.724114+0000","last_fullsized":"2024-09-16T05:15:34.724114+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:14:31.325082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.325082+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:44.845614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00067654200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":1456,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724109+0000","last_change":"2024-09-16T05:15:32.707387+0000","last_active":"2024-09-16T05:15:34.724109+0000","last_peered":"2024-09-16T05:15:34.724109+0000","last_clean":"2024-09-16T05:15:34.724109+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:15:34.724109+0000","last_undegraded":"2024-09-16T05:15:34.724109+0000","last_fullsized":"2024-09-16T05:15:34.724109+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:30.305712+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.305712+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:54.920826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047164700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":342,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.712829+0000","last_change":"2024-09-16T05:15:31.713060+0000","last_active":"2024-09-16T05:15:31.712829+0000","last_peered":"2024-09-16T05:15:31.712829+0000","last_clean":"2024-09-16T05:15:31.712829+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:15:31.712829+0000","last_undegraded":"2024-09-16T05:15:31.712829+0000","last_fullsized":"2024-09-16T05:15:31.712829+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:14:28.659562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.659562+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:49.546066+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010566530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":327,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-16T05:15:33.678783+0000","last_change":"2024-09-16T05:15:33.678815+0000","last_active":"2024-09-16T05:15:33.678783+0000","last_peered":"2024-09-16T05:15:33.678783+0000","last_clean":"2024-09-16T05:15:33.678783+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:15:33.678783+0000","last_undegraded":"2024-09-16T05:15:33.678783+0000","last_fullsized":"2024-09-16T05:15:33.678783+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:33.678710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:33.678710+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:46.023519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001137971,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.41","version":"178'50","reported_seq":448,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769823+0000","last_change":"2024-09-16T05:15:33.269208+0000","last_active":"2024-09-16T05:15:35.769823+0000","last_peered":"2024-09-16T05:15:35.769823+0000","last_clean":"2024-09-16T05:15:35.769823+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:15:35.769823+0000","last_undegraded":"2024-09-16T05:15:35.769823+0000","last_fullsized":"2024-09-16T05:15:35.769823+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:15.794439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.794439+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:22.154890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050109500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.40","version":"205'33","reported_seq":1388,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.717842+0000","last_change":"2024-09-16T05:15:31.718604+0000","last_active":"2024-09-16T05:15:31.717842+0000","last_peered":"2024-09-16T05:15:31.717842+0000","last_clean":"2024-09-16T05:15:31.717842+0000","last_became_active":"2024-09-16T05:13:45.682851+0000","last_became_peered":"2024-09-16T05:13:45.682851+0000","last_unstale":"2024-09-16T05:15:31.717842+0000","last_undegraded":"2024-09-16T05:15:31.717842+0000","last_fullsized":"2024-09-16T05:15:31.717842+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:07.789202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.789202+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.141078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0060912919999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3f","version":"378'282","reported_seq":1634,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.715921+0000","last_change":"2024-09-16T05:15:31.716267+0000","last_active":"2024-09-16T05:15:31.715921+0000","last_peered":"2024-09-16T05:15:31.715921+0000","last_clean":"2024-09-16T05:15:31.715921+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:15:31.715921+0000","last_undegraded":"2024-09-16T05:15:31.715921+0000","last_fullsized":"2024-09-16T05:15:31.715921+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"357'270","last_scrub_stamp":"2024-09-16T05:15:09.797411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.797411+0000","objects_scrubbed":3,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:35.904107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039964459999999999,"stat_sum":{"num_bytes":2285818,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":15220,"num_write":15,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3e","version":"378'353","reported_seq":1669,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764831+0000","last_change":"2024-09-16T05:15:31.765009+0000","last_active":"2024-09-16T05:15:31.764831+0000","last_peered":"2024-09-16T05:15:31.764831+0000","last_clean":"2024-09-16T05:15:31.764831+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:15:31.764831+0000","last_undegraded":"2024-09-16T05:15:31.764831+0000","last_fullsized":"2024-09-16T05:15:31.764831+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"360'342","last_scrub_stamp":"2024-09-16T05:15:09.920637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.920637+0000","objects_scrubbed":4,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:29.576047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081514700000000001,"stat_sum":{"num_bytes":2437081,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":2380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1903,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724304+0000","last_change":"2024-09-16T05:15:32.708208+0000","last_active":"2024-09-16T05:15:34.724304+0000","last_peered":"2024-09-16T05:15:34.724304+0000","last_clean":"2024-09-16T05:15:34.724304+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:15:34.724304+0000","last_undegraded":"2024-09-16T05:15:34.724304+0000","last_fullsized":"2024-09-16T05:15:34.724304+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:15:09.150849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.150849+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.505002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087124800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.3c","version":"378'445","reported_seq":1878,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.713128+0000","last_change":"2024-09-16T05:15:31.713428+0000","last_active":"2024-09-16T05:15:31.713128+0000","last_peered":"2024-09-16T05:15:31.713128+0000","last_clean":"2024-09-16T05:15:31.713128+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:15:31.713128+0000","last_undegraded":"2024-09-16T05:15:31.713128+0000","last_fullsized":"2024-09-16T05:15:31.713128+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"342'400","last_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:14:51.845721+0000","objects_scrubbed":6,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.270697+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0013605030000000001,"stat_sum":{"num_bytes":3815222,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22839,"num_write":14,"num_write_kb":4757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3b","version":"361'294","reported_seq":1663,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769130+0000","last_change":"2024-09-16T05:15:33.092167+0000","last_active":"2024-09-16T05:15:35.769130+0000","last_peered":"2024-09-16T05:15:35.769130+0000","last_clean":"2024-09-16T05:15:35.769130+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:15:35.769130+0000","last_undegraded":"2024-09-16T05:15:35.769130+0000","last_fullsized":"2024-09-16T05:15:35.769130+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"300'290","last_scrub_stamp":"2024-09-16T05:15:06.955285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.955285+0000","objects_scrubbed":2,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.283149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001176238,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":31094,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.3a","version":"385'453","reported_seq":1783,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.739921+0000","last_change":"2024-09-16T05:15:32.995897+0000","last_active":"2024-09-16T05:15:34.739921+0000","last_peered":"2024-09-16T05:15:34.739921+0000","last_clean":"2024-09-16T05:15:34.739921+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:15:34.739921+0000","last_undegraded":"2024-09-16T05:15:34.739921+0000","last_fullsized":"2024-09-16T05:15:34.739921+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"356'429","last_scrub_stamp":"2024-09-16T05:15:10.141866+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.141866+0000","objects_scrubbed":2,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:24.530444+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070154600000000005,"stat_sum":{"num_bytes":5529635,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":9240,"num_write":19,"num_write_kb":9934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.39","version":"182'60","reported_seq":452,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.713658+0000","last_change":"2024-09-16T05:15:32.996589+0000","last_active":"2024-09-16T05:15:34.713658+0000","last_peered":"2024-09-16T05:15:34.713658+0000","last_clean":"2024-09-16T05:15:34.713658+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:15:34.713658+0000","last_undegraded":"2024-09-16T05:15:34.713658+0000","last_fullsized":"2024-09-16T05:15:34.713658+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:15.232072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.232072+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:31.199351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010482270000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.38","version":"181'57","reported_seq":440,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-16T05:15:32.708297+0000","last_change":"2024-09-16T05:15:32.708619+0000","last_active":"2024-09-16T05:15:32.708297+0000","last_peered":"2024-09-16T05:15:32.708297+0000","last_clean":"2024-09-16T05:15:32.708297+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:15:32.708297+0000","last_undegraded":"2024-09-16T05:15:32.708297+0000","last_fullsized":"2024-09-16T05:15:32.708297+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:13.450261+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:13.450261+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:42.398187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051423099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.37","version":"0'0","reported_seq":536,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769474+0000","last_change":"2024-09-16T05:15:33.269531+0000","last_active":"2024-09-16T05:15:35.769474+0000","last_peered":"2024-09-16T05:15:35.769474+0000","last_clean":"2024-09-16T05:15:35.769474+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:15:35.769474+0000","last_undegraded":"2024-09-16T05:15:35.769474+0000","last_fullsized":"2024-09-16T05:15:35.769474+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.650972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:03.895367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00062126800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.36","version":"272'60","reported_seq":531,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724288+0000","last_change":"2024-09-16T05:15:32.707754+0000","last_active":"2024-09-16T05:15:34.724288+0000","last_peered":"2024-09-16T05:15:34.724288+0000","last_clean":"2024-09-16T05:15:34.724288+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:15:34.724288+0000","last_undegraded":"2024-09-16T05:15:34.724288+0000","last_fullsized":"2024-09-16T05:15:34.724288+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:14:49.375386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.375386+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:59.531830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066918900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.35","version":"380'503","reported_seq":1887,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724299+0000","last_change":"2024-09-16T05:15:32.708547+0000","last_active":"2024-09-16T05:15:34.724299+0000","last_peered":"2024-09-16T05:15:34.724299+0000","last_clean":"2024-09-16T05:15:34.724299+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:15:34.724299+0000","last_undegraded":"2024-09-16T05:15:34.724299+0000","last_fullsized":"2024-09-16T05:15:34.724299+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'421","last_scrub_stamp":"2024-09-16T05:14:34.349392+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:14:34.349392+0000","objects_scrubbed":4,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:42.063010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00093675200000000001,"stat_sum":{"num_bytes":9815498,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":23,"num_read_kb":24061,"num_write":19,"num_write_kb":7769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.34","version":"378'308","reported_seq":1278,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769352+0000","last_change":"2024-09-16T05:15:33.092189+0000","last_active":"2024-09-16T05:15:35.769352+0000","last_peered":"2024-09-16T05:15:35.769352+0000","last_clean":"2024-09-16T05:15:35.769352+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:15:35.769352+0000","last_undegraded":"2024-09-16T05:15:35.769352+0000","last_fullsized":"2024-09-16T05:15:35.769352+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'243","last_scrub_stamp":"2024-09-16T05:14:50.913931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.913931+0000","objects_scrubbed":5,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:54.251328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089677500000000005,"stat_sum":{"num_bytes":5613670,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":41071,"num_write":9,"num_write_kb":6969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":520,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.715847+0000","last_change":"2024-09-16T05:15:31.716256+0000","last_active":"2024-09-16T05:15:31.715847+0000","last_peered":"2024-09-16T05:15:31.715847+0000","last_clean":"2024-09-16T05:15:31.715847+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:15:31.715847+0000","last_undegraded":"2024-09-16T05:15:31.715847+0000","last_fullsized":"2024-09-16T05:15:31.715847+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:46.792111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.792111+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:07.000604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040551720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.32","version":"380'125","reported_seq":600,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764359+0000","last_change":"2024-09-16T05:15:31.764567+0000","last_active":"2024-09-16T05:15:31.764359+0000","last_peered":"2024-09-16T05:15:31.764359+0000","last_clean":"2024-09-16T05:15:31.764359+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:15:31.764359+0000","last_undegraded":"2024-09-16T05:15:31.764359+0000","last_fullsized":"2024-09-16T05:15:31.764359+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'101","last_scrub_stamp":"2024-09-16T05:14:49.915945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.915945+0000","objects_scrubbed":2,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.781860+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000548779,"stat_sum":{"num_bytes":6857355,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":17,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.31","version":"380'162","reported_seq":1149,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.870314+0000","last_change":"2024-09-16T05:15:32.739361+0000","last_active":"2024-09-16T05:15:34.870314+0000","last_peered":"2024-09-16T05:15:34.870314+0000","last_clean":"2024-09-16T05:15:34.870314+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:15:34.870314+0000","last_undegraded":"2024-09-16T05:15:34.870314+0000","last_fullsized":"2024-09-16T05:15:34.870314+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'133","last_scrub_stamp":"2024-09-16T05:14:46.026342+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.026342+0000","objects_scrubbed":2,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.235985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000319117,"stat_sum":{"num_bytes":4394329,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":22763,"num_write":13,"num_write_kb":4010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.30","version":"182'16","reported_seq":500,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769184+0000","last_change":"2024-09-16T05:15:33.269671+0000","last_active":"2024-09-16T05:15:35.769184+0000","last_peered":"2024-09-16T05:15:35.769184+0000","last_clean":"2024-09-16T05:15:35.769184+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:15:35.769184+0000","last_undegraded":"2024-09-16T05:15:35.769184+0000","last_fullsized":"2024-09-16T05:15:35.769184+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:06.783241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.783241+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.102054+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00070626499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.11","version":"272'115","reported_seq":1060,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.716215+0000","last_change":"2024-09-16T05:15:31.716651+0000","last_active":"2024-09-16T05:15:31.716215+0000","last_peered":"2024-09-16T05:15:31.716215+0000","last_clean":"2024-09-16T05:15:31.716215+0000","last_became_active":"2024-09-16T05:13:25.679267+0000","last_became_peered":"2024-09-16T05:13:25.679267+0000","last_unstale":"2024-09-16T05:15:31.716215+0000","last_undegraded":"2024-09-16T05:15:31.716215+0000","last_fullsized":"2024-09-16T05:15:31.716215+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:14:31.050468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.050468+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:39.196078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019176340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":979,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-16T05:15:32.709038+0000","last_change":"2024-09-16T05:15:32.709473+0000","last_active":"2024-09-16T05:15:32.709038+0000","last_peered":"2024-09-16T05:15:32.709038+0000","last_clean":"2024-09-16T05:15:32.709038+0000","last_became_active":"2024-09-16T05:13:25.680445+0000","last_became_peered":"2024-09-16T05:13:25.680445+0000","last_unstale":"2024-09-16T05:15:32.709038+0000","last_undegraded":"2024-09-16T05:15:32.709038+0000","last_fullsized":"2024-09-16T05:15:32.709038+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:07.531934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.531934+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:31.821422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00088913000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.f","version":"360'281","reported_seq":1643,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-16T05:15:32.709552+0000","last_change":"2024-09-16T05:15:32.709769+0000","last_active":"2024-09-16T05:15:32.709552+0000","last_peered":"2024-09-16T05:15:32.709552+0000","last_clean":"2024-09-16T05:15:32.709552+0000","last_became_active":"2024-09-16T05:13:25.680481+0000","last_became_peered":"2024-09-16T05:13:25.680481+0000","last_unstale":"2024-09-16T05:15:32.709552+0000","last_undegraded":"2024-09-16T05:15:32.709552+0000","last_fullsized":"2024-09-16T05:15:32.709552+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:50.580296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.580296+0000","objects_scrubbed":3,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:11.893152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011776530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.e","version":"272'232","reported_seq":1491,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.705666+0000","last_change":"2024-09-16T05:15:31.705916+0000","last_active":"2024-09-16T05:15:31.705666+0000","last_peered":"2024-09-16T05:15:31.705666+0000","last_clean":"2024-09-16T05:15:31.705666+0000","last_became_active":"2024-09-16T05:14:40.261276+0000","last_became_peered":"2024-09-16T05:14:40.261276+0000","last_unstale":"2024-09-16T05:15:31.705666+0000","last_undegraded":"2024-09-16T05:15:31.705666+0000","last_fullsized":"2024-09-16T05:15:31.705666+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:14:47.777342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.777342+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:12.476383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061369799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.d","version":"380'455","reported_seq":1819,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.714104+0000","last_change":"2024-09-16T05:15:32.997594+0000","last_active":"2024-09-16T05:15:34.714104+0000","last_peered":"2024-09-16T05:15:34.714104+0000","last_clean":"2024-09-16T05:15:34.714104+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:15:34.714104+0000","last_undegraded":"2024-09-16T05:15:34.714104+0000","last_fullsized":"2024-09-16T05:15:34.714104+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'379","last_scrub_stamp":"2024-09-16T05:14:44.180613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.180613+0000","objects_scrubbed":6,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:55.496289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018794059999999999,"stat_sum":{"num_bytes":9462506,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":48192,"num_write":30,"num_write_kb":9875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.c","version":"380'255","reported_seq":1599,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764302+0000","last_change":"2024-09-16T05:15:31.764527+0000","last_active":"2024-09-16T05:15:31.764302+0000","last_peered":"2024-09-16T05:15:31.764302+0000","last_clean":"2024-09-16T05:15:31.764302+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:15:31.764302+0000","last_undegraded":"2024-09-16T05:15:31.764302+0000","last_fullsized":"2024-09-16T05:15:31.764302+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'213","last_scrub_stamp":"2024-09-16T05:14:26.945487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.945487+0000","objects_scrubbed":4,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:33.783300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061793999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":33266,"num_write":11,"num_write_kb":5499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.8","version":"380'209","reported_seq":1556,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.713830+0000","last_change":"2024-09-16T05:15:31.715266+0000","last_active":"2024-09-16T05:15:31.713830+0000","last_peered":"2024-09-16T05:15:31.713830+0000","last_clean":"2024-09-16T05:15:31.713830+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:15:31.713830+0000","last_undegraded":"2024-09-16T05:15:31.713830+0000","last_fullsized":"2024-09-16T05:15:31.713830+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'153","last_scrub_stamp":"2024-09-16T05:14:26.593177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.593177+0000","objects_scrubbed":3,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:53.214381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0031367830000000002,"stat_sum":{"num_bytes":2790028,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.3","version":"378'98","reported_seq":1460,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769353+0000","last_change":"2024-09-16T05:15:33.091688+0000","last_active":"2024-09-16T05:15:35.769353+0000","last_peered":"2024-09-16T05:15:35.769353+0000","last_clean":"2024-09-16T05:15:35.769353+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:15:35.769353+0000","last_undegraded":"2024-09-16T05:15:35.769353+0000","last_fullsized":"2024-09-16T05:15:35.769353+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'68","last_scrub_stamp":"2024-09-16T05:14:44.975873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.975873+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.716053+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00076941100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":979,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.712668+0000","last_change":"2024-09-16T05:14:53.849661+0000","last_active":"2024-09-16T05:15:31.712668+0000","last_peered":"2024-09-16T05:15:31.712668+0000","last_clean":"2024-09-16T05:15:31.712668+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:15:31.712668+0000","last_undegraded":"2024-09-16T05:15:31.712668+0000","last_fullsized":"2024-09-16T05:15:31.712668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:53.849576+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.849576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:02.708781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1376,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.768891+0000","last_change":"2024-09-16T05:15:33.091412+0000","last_active":"2024-09-16T05:15:35.768891+0000","last_peered":"2024-09-16T05:15:35.768891+0000","last_clean":"2024-09-16T05:15:35.768891+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:15:35.768891+0000","last_undegraded":"2024-09-16T05:15:35.768891+0000","last_fullsized":"2024-09-16T05:15:35.768891+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:14:25.970665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.970665+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:55.790218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046620599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":629,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769886+0000","last_change":"2024-09-16T05:15:18.812427+0000","last_active":"2024-09-16T05:15:35.769886+0000","last_peered":"2024-09-16T05:15:35.769886+0000","last_clean":"2024-09-16T05:15:35.769886+0000","last_became_active":"2024-09-16T05:13:34.762556+0000","last_became_peered":"2024-09-16T05:13:34.762556+0000","last_unstale":"2024-09-16T05:15:35.769886+0000","last_undegraded":"2024-09-16T05:15:35.769886+0000","last_fullsized":"2024-09-16T05:15:35.769886+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:18.812352+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:18.812352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:21.232575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":975,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.711820+0000","last_change":"2024-09-16T05:14:07.583425+0000","last_active":"2024-09-16T05:15:31.711820+0000","last_peered":"2024-09-16T05:15:31.711820+0000","last_clean":"2024-09-16T05:15:31.711820+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:15:31.711820+0000","last_undegraded":"2024-09-16T05:15:31.711820+0000","last_fullsized":"2024-09-16T05:15:31.711820+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:07.583346+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:07.583346+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1311,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.705468+0000","last_change":"2024-09-16T05:15:31.705609+0000","last_active":"2024-09-16T05:15:31.705468+0000","last_peered":"2024-09-16T05:15:31.705468+0000","last_clean":"2024-09-16T05:15:31.705468+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:15:31.705468+0000","last_undegraded":"2024-09-16T05:15:31.705468+0000","last_fullsized":"2024-09-16T05:15:31.705468+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:07.898675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.898675+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.496798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00032755300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":988,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.718532+0000","last_change":"2024-09-16T05:15:19.711525+0000","last_active":"2024-09-16T05:15:31.718532+0000","last_peered":"2024-09-16T05:15:31.718532+0000","last_clean":"2024-09-16T05:15:31.718532+0000","last_became_active":"2024-09-16T05:15:19.711201+0000","last_became_peered":"2024-09-16T05:15:19.711201+0000","last_unstale":"2024-09-16T05:15:31.718532+0000","last_undegraded":"2024-09-16T05:15:31.718532+0000","last_fullsized":"2024-09-16T05:15:31.718532+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:45.298203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.298203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:07.821797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1386,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.717513+0000","last_change":"2024-09-16T05:15:31.717890+0000","last_active":"2024-09-16T05:15:31.717513+0000","last_peered":"2024-09-16T05:15:31.717513+0000","last_clean":"2024-09-16T05:15:31.717513+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:15:31.717513+0000","last_undegraded":"2024-09-16T05:15:31.717513+0000","last_fullsized":"2024-09-16T05:15:31.717513+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:06.755407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.755407+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:14.021642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053046760000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":623,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.714060+0000","last_change":"2024-09-16T05:15:32.997068+0000","last_active":"2024-09-16T05:15:34.714060+0000","last_peered":"2024-09-16T05:15:34.714060+0000","last_clean":"2024-09-16T05:15:34.714060+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:15:34.714060+0000","last_undegraded":"2024-09-16T05:15:34.714060+0000","last_fullsized":"2024-09-16T05:15:34.714060+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:31.009189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.009189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:51.114384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014481120000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"1.4","version":"35'139","reported_seq":942,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764325+0000","last_change":"2024-09-16T05:15:10.945891+0000","last_active":"2024-09-16T05:15:31.764325+0000","last_peered":"2024-09-16T05:15:31.764325+0000","last_clean":"2024-09-16T05:15:31.764325+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:15:31.764325+0000","last_undegraded":"2024-09-16T05:15:31.764325+0000","last_fullsized":"2024-09-16T05:15:31.764325+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:10.945793+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.945793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:24.451282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1371,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724386+0000","last_change":"2024-09-16T05:15:32.708339+0000","last_active":"2024-09-16T05:15:34.724386+0000","last_peered":"2024-09-16T05:15:34.724386+0000","last_clean":"2024-09-16T05:15:34.724386+0000","last_became_active":"2024-09-16T05:14:25.821885+0000","last_became_peered":"2024-09-16T05:14:25.821885+0000","last_unstale":"2024-09-16T05:15:34.724386+0000","last_undegraded":"2024-09-16T05:15:34.724386+0000","last_fullsized":"2024-09-16T05:15:34.724386+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:27.379983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.379983+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:46.439627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00098575099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":734,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.770695+0000","last_change":"2024-09-16T05:14:45.491959+0000","last_active":"2024-09-16T05:15:35.770695+0000","last_peered":"2024-09-16T05:15:35.770695+0000","last_clean":"2024-09-16T05:15:35.770695+0000","last_became_active":"2024-09-16T05:14:39.908958+0000","last_became_peered":"2024-09-16T05:14:39.908958+0000","last_unstale":"2024-09-16T05:15:35.770695+0000","last_undegraded":"2024-09-16T05:15:35.770695+0000","last_fullsized":"2024-09-16T05:15:35.770695+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:45.491884+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.491884+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.370020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"380'476","reported_seq":1823,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.713909+0000","last_change":"2024-09-16T05:15:32.996501+0000","last_active":"2024-09-16T05:15:34.713909+0000","last_peered":"2024-09-16T05:15:34.713909+0000","last_clean":"2024-09-16T05:15:34.713909+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:15:34.713909+0000","last_undegraded":"2024-09-16T05:15:34.713909+0000","last_fullsized":"2024-09-16T05:15:34.713909+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:14:26.230040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.230040+0000","objects_scrubbed":3,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:51.214928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001055007,"stat_sum":{"num_bytes":3176556,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":8850,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":629,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769025+0000","last_change":"2024-09-16T05:15:11.724435+0000","last_active":"2024-09-16T05:15:35.769025+0000","last_peered":"2024-09-16T05:15:35.769025+0000","last_clean":"2024-09-16T05:15:35.769025+0000","last_became_active":"2024-09-16T05:13:34.764187+0000","last_became_peered":"2024-09-16T05:13:34.764187+0000","last_unstale":"2024-09-16T05:15:35.769025+0000","last_undegraded":"2024-09-16T05:15:35.769025+0000","last_fullsized":"2024-09-16T05:15:35.769025+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:11.724301+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:11.724301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:40.115572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1453,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.724266+0000","last_change":"2024-09-16T05:15:32.707938+0000","last_active":"2024-09-16T05:15:34.724266+0000","last_peered":"2024-09-16T05:15:34.724266+0000","last_clean":"2024-09-16T05:15:34.724266+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:15:34.724266+0000","last_undegraded":"2024-09-16T05:15:34.724266+0000","last_fullsized":"2024-09-16T05:15:34.724266+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:28.350521+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.350521+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:52.294874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071958500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":813,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769253+0000","last_change":"2024-09-16T05:14:28.552017+0000","last_active":"2024-09-16T05:15:35.769253+0000","last_peered":"2024-09-16T05:15:35.769253+0000","last_clean":"2024-09-16T05:15:35.769253+0000","last_became_active":"2024-09-16T05:13:34.764109+0000","last_became_peered":"2024-09-16T05:13:34.764109+0000","last_unstale":"2024-09-16T05:15:35.769253+0000","last_undegraded":"2024-09-16T05:15:35.769253+0000","last_fullsized":"2024-09-16T05:15:35.769253+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:28.551941+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.551941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:45.247355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":1538,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-16T05:15:32.708401+0000","last_change":"2024-09-16T05:15:32.708919+0000","last_active":"2024-09-16T05:15:32.708401+0000","last_peered":"2024-09-16T05:15:32.708401+0000","last_clean":"2024-09-16T05:15:32.708401+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:15:32.708401+0000","last_undegraded":"2024-09-16T05:15:32.708401+0000","last_fullsized":"2024-09-16T05:15:32.708401+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:14:26.495648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.495648+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:45.726401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000738107,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":994,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.768917+0000","last_change":"2024-09-16T05:15:15.084154+0000","last_active":"2024-09-16T05:15:35.768917+0000","last_peered":"2024-09-16T05:15:35.768917+0000","last_clean":"2024-09-16T05:15:35.768917+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:15:35.768917+0000","last_undegraded":"2024-09-16T05:15:35.768917+0000","last_fullsized":"2024-09-16T05:15:35.768917+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:15.084022+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.084022+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:28.500567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"380'461","reported_seq":1819,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.764585+0000","last_change":"2024-09-16T05:15:31.764771+0000","last_active":"2024-09-16T05:15:31.764585+0000","last_peered":"2024-09-16T05:15:31.764585+0000","last_clean":"2024-09-16T05:15:31.764585+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:15:31.764585+0000","last_undegraded":"2024-09-16T05:15:31.764585+0000","last_fullsized":"2024-09-16T05:15:31.764585+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'419","last_scrub_stamp":"2024-09-16T05:14:25.924121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.924121+0000","objects_scrubbed":4,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:31.804801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00065622399999999996,"stat_sum":{"num_bytes":5126867,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":16965,"num_write":5,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":975,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.711796+0000","last_change":"2024-09-16T05:14:11.602321+0000","last_active":"2024-09-16T05:15:31.711796+0000","last_peered":"2024-09-16T05:15:31.711796+0000","last_clean":"2024-09-16T05:15:31.711796+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:15:31.711796+0000","last_undegraded":"2024-09-16T05:15:31.711796+0000","last_fullsized":"2024-09-16T05:15:31.711796+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:11.602237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:11.602237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1598,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.723983+0000","last_change":"2024-09-16T05:15:32.922282+0000","last_active":"2024-09-16T05:15:34.723983+0000","last_peered":"2024-09-16T05:15:34.723983+0000","last_clean":"2024-09-16T05:15:34.723983+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:15:34.723983+0000","last_undegraded":"2024-09-16T05:15:34.723983+0000","last_fullsized":"2024-09-16T05:15:34.723983+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:15:32.922163+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:32.922163+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:57.599655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049108699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":992,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.713617+0000","last_change":"2024-09-16T05:15:14.182510+0000","last_active":"2024-09-16T05:15:34.713617+0000","last_peered":"2024-09-16T05:15:34.713617+0000","last_clean":"2024-09-16T05:15:34.713617+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:15:34.713617+0000","last_undegraded":"2024-09-16T05:15:34.713617+0000","last_fullsized":"2024-09-16T05:15:34.713617+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:14.182407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:14.182407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.292535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":500,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.770157+0000","last_change":"2024-09-16T05:15:33.269717+0000","last_active":"2024-09-16T05:15:35.770157+0000","last_peered":"2024-09-16T05:15:35.770157+0000","last_clean":"2024-09-16T05:15:35.770157+0000","last_became_active":"2024-09-16T05:13:31.747644+0000","last_became_peered":"2024-09-16T05:13:31.747644+0000","last_unstale":"2024-09-16T05:15:35.770157+0000","last_undegraded":"2024-09-16T05:15:35.770157+0000","last_fullsized":"2024-09-16T05:15:35.770157+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:14:49.576522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.576522+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:13.591497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074053300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.13","version":"181'15","reported_seq":993,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.713788+0000","last_change":"2024-09-16T05:15:32.996243+0000","last_active":"2024-09-16T05:15:34.713788+0000","last_peered":"2024-09-16T05:15:34.713788+0000","last_clean":"2024-09-16T05:15:34.713788+0000","last_became_active":"2024-09-16T05:13:25.681297+0000","last_became_peered":"2024-09-16T05:13:25.681297+0000","last_unstale":"2024-09-16T05:15:34.713788+0000","last_undegraded":"2024-09-16T05:15:34.713788+0000","last_fullsized":"2024-09-16T05:15:34.713788+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:49.160833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.160833+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.028144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084836099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.14","version":"383'267","reported_seq":1230,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.713917+0000","last_change":"2024-09-16T05:15:32.998412+0000","last_active":"2024-09-16T05:15:34.713917+0000","last_peered":"2024-09-16T05:15:34.713917+0000","last_clean":"2024-09-16T05:15:34.713917+0000","last_became_active":"2024-09-16T05:13:25.684888+0000","last_became_peered":"2024-09-16T05:13:25.684888+0000","last_unstale":"2024-09-16T05:15:34.713917+0000","last_undegraded":"2024-09-16T05:15:34.713917+0000","last_fullsized":"2024-09-16T05:15:34.713917+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"342'239","last_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.110587+0000","objects_scrubbed":4,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:52.679920+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0028409619999999998,"stat_sum":{"num_bytes":3142942,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":14332,"num_write":7,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.15","version":"380'471","reported_seq":1840,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.714021+0000","last_change":"2024-09-16T05:15:32.996373+0000","last_active":"2024-09-16T05:15:34.714021+0000","last_peered":"2024-09-16T05:15:34.714021+0000","last_clean":"2024-09-16T05:15:34.714021+0000","last_became_active":"2024-09-16T05:13:25.680026+0000","last_became_peered":"2024-09-16T05:13:25.680026+0000","last_unstale":"2024-09-16T05:15:34.714021+0000","last_undegraded":"2024-09-16T05:15:34.714021+0000","last_fullsized":"2024-09-16T05:15:34.714021+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"357'436","last_scrub_stamp":"2024-09-16T05:15:07.101221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.101221+0000","objects_scrubbed":5,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:16.483218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00091967399999999999,"stat_sum":{"num_bytes":6735275,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":41972,"num_write":30,"num_write_kb":12050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.16","version":"384'118","reported_seq":1120,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.768885+0000","last_change":"2024-09-16T05:15:33.091722+0000","last_active":"2024-09-16T05:15:35.768885+0000","last_peered":"2024-09-16T05:15:35.768885+0000","last_clean":"2024-09-16T05:15:35.768885+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:15:35.768885+0000","last_undegraded":"2024-09-16T05:15:35.768885+0000","last_fullsized":"2024-09-16T05:15:35.768885+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"304'74","last_scrub_stamp":"2024-09-16T05:14:29.883455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:29.883455+0000","objects_scrubbed":1,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:53.344093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073028399999999999,"stat_sum":{"num_bytes":3142942,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6148,"num_write":5,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.17","version":"0'0","reported_seq":1154,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.716288+0000","last_change":"2024-09-16T05:15:31.716714+0000","last_active":"2024-09-16T05:15:31.716288+0000","last_peered":"2024-09-16T05:15:31.716288+0000","last_clean":"2024-09-16T05:15:31.716288+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:15:31.716288+0000","last_undegraded":"2024-09-16T05:15:31.716288+0000","last_fullsized":"2024-09-16T05:15:31.716288+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:43.982972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:43.982972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:07.471726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001213754,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1386,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.713565+0000","last_change":"2024-09-16T05:15:32.995880+0000","last_active":"2024-09-16T05:15:34.713565+0000","last_peered":"2024-09-16T05:15:34.713565+0000","last_clean":"2024-09-16T05:15:34.713565+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:15:34.713565+0000","last_undegraded":"2024-09-16T05:15:34.713565+0000","last_fullsized":"2024-09-16T05:15:34.713565+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:17.188273+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:17.188273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:45.915759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071508599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.19","version":"182'60","reported_seq":976,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.719111+0000","last_change":"2024-09-16T05:15:31.719302+0000","last_active":"2024-09-16T05:15:31.719111+0000","last_peered":"2024-09-16T05:15:31.719111+0000","last_clean":"2024-09-16T05:15:31.719111+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:15:31.719111+0000","last_undegraded":"2024-09-16T05:15:31.719111+0000","last_fullsized":"2024-09-16T05:15:31.719111+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:06.893298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.893298+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:25.185017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064383699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.1a","version":"380'600","reported_seq":1935,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-16T05:15:32.709915+0000","last_change":"2024-09-16T05:15:32.710132+0000","last_active":"2024-09-16T05:15:32.709915+0000","last_peered":"2024-09-16T05:15:32.709915+0000","last_clean":"2024-09-16T05:15:32.709915+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:15:32.709915+0000","last_undegraded":"2024-09-16T05:15:32.709915+0000","last_fullsized":"2024-09-16T05:15:32.709915+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'527","last_scrub_stamp":"2024-09-16T05:15:10.511970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.511970+0000","objects_scrubbed":9,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:36.350164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010474919999999999,"stat_sum":{"num_bytes":9446945,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":56,"num_read_kb":52510,"num_write":35,"num_write_kb":15546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.1b","version":"382'332","reported_seq":1713,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.765946+0000","last_change":"2024-09-16T05:15:31.766093+0000","last_active":"2024-09-16T05:15:31.765946+0000","last_peered":"2024-09-16T05:15:31.765946+0000","last_clean":"2024-09-16T05:15:31.765946+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:15:31.765946+0000","last_undegraded":"2024-09-16T05:15:31.765946+0000","last_fullsized":"2024-09-16T05:15:31.765946+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"347'319","last_scrub_stamp":"2024-09-16T05:15:08.957118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.957118+0000","objects_scrubbed":3,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.578187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019599259999999999,"stat_sum":{"num_bytes":5298036,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":29313,"num_write":9,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.1c","version":"382'373","reported_seq":1802,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.769656+0000","last_change":"2024-09-16T05:15:33.269443+0000","last_active":"2024-09-16T05:15:35.769656+0000","last_peered":"2024-09-16T05:15:35.769656+0000","last_clean":"2024-09-16T05:15:35.769656+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:15:35.769656+0000","last_undegraded":"2024-09-16T05:15:35.769656+0000","last_fullsized":"2024-09-16T05:15:35.769656+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'364","last_scrub_stamp":"2024-09-16T05:15:08.759106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.759106+0000","objects_scrubbed":2,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:30.109517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060242399999999995,"stat_sum":{"num_bytes":3327852,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.1d","version":"384'628","reported_seq":2077,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.770699+0000","last_change":"2024-09-16T05:15:33.269974+0000","last_active":"2024-09-16T05:15:35.770699+0000","last_peered":"2024-09-16T05:15:35.770699+0000","last_clean":"2024-09-16T05:15:35.770699+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:15:35.770699+0000","last_undegraded":"2024-09-16T05:15:35.770699+0000","last_fullsized":"2024-09-16T05:15:35.770699+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'584","last_scrub_stamp":"2024-09-16T05:15:16.833446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:16.833446+0000","objects_scrubbed":8,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:24.483779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00085383200000000001,"stat_sum":{"num_bytes":12407718,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":34800,"num_write":31,"num_write_kb":17614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.1e","version":"378'347","reported_seq":1720,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.719157+0000","last_change":"2024-09-16T05:15:31.719365+0000","last_active":"2024-09-16T05:15:31.719157+0000","last_peered":"2024-09-16T05:15:31.719157+0000","last_clean":"2024-09-16T05:15:31.719157+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:15:31.719157+0000","last_undegraded":"2024-09-16T05:15:31.719157+0000","last_fullsized":"2024-09-16T05:15:31.719157+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"362'335","last_scrub_stamp":"2024-09-16T05:15:12.926534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:12.926534+0000","objects_scrubbed":3,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:39.638583+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00057471900000000001,"stat_sum":{"num_bytes":2487502,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":14641,"num_write":5,"num_write_kb":3828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":1599,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.870534+0000","last_change":"2024-09-16T05:15:32.739574+0000","last_active":"2024-09-16T05:15:34.870534+0000","last_peered":"2024-09-16T05:15:34.870534+0000","last_clean":"2024-09-16T05:15:34.870534+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:15:34.870534+0000","last_undegraded":"2024-09-16T05:15:34.870534+0000","last_fullsized":"2024-09-16T05:15:34.870534+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:14:52.883562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.883562+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:54.882630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040063300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.20","version":"382'132","reported_seq":1506,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.718539+0000","last_change":"2024-09-16T05:15:31.715740+0000","last_active":"2024-09-16T05:15:31.718539+0000","last_peered":"2024-09-16T05:15:31.718539+0000","last_clean":"2024-09-16T05:15:31.718539+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:15:31.718539+0000","last_undegraded":"2024-09-16T05:15:31.718539+0000","last_fullsized":"2024-09-16T05:15:31.718539+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'128","last_scrub_stamp":"2024-09-16T05:15:31.623695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.623695+0000","objects_scrubbed":2,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:33.562506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038817470000000001,"stat_sum":{"num_bytes":4238337,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":70060,"num_write":16,"num_write_kb":6808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":1310,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.705423+0000","last_change":"2024-09-16T05:15:31.705616+0000","last_active":"2024-09-16T05:15:31.705423+0000","last_peered":"2024-09-16T05:15:31.705423+0000","last_clean":"2024-09-16T05:15:31.705423+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:15:31.705423+0000","last_undegraded":"2024-09-16T05:15:31.705423+0000","last_fullsized":"2024-09-16T05:15:31.705423+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.848788+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:22.418821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000315023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":509,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-16T05:15:35.770489+0000","last_change":"2024-09-16T05:15:33.269880+0000","last_active":"2024-09-16T05:15:35.770489+0000","last_peered":"2024-09-16T05:15:35.770489+0000","last_clean":"2024-09-16T05:15:35.770489+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:15:35.770489+0000","last_undegraded":"2024-09-16T05:15:35.770489+0000","last_fullsized":"2024-09-16T05:15:35.770489+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:31.582158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.582158+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:39.214528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00083755699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":843,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.718883+0000","last_change":"2024-09-16T05:15:31.719064+0000","last_active":"2024-09-16T05:15:31.718883+0000","last_peered":"2024-09-16T05:15:31.718883+0000","last_clean":"2024-09-16T05:15:31.718883+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:15:31.718883+0000","last_undegraded":"2024-09-16T05:15:31.718883+0000","last_fullsized":"2024-09-16T05:15:31.718883+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:14:46.760890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.760890+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:14.795429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040807099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":672,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.713777+0000","last_change":"2024-09-16T05:15:32.996117+0000","last_active":"2024-09-16T05:15:34.713777+0000","last_peered":"2024-09-16T05:15:34.713777+0000","last_clean":"2024-09-16T05:15:34.713777+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:15:34.713777+0000","last_undegraded":"2024-09-16T05:15:34.713777+0000","last_fullsized":"2024-09-16T05:15:34.713777+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:31.611624+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:31.611624+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.611624+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:50.542022+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0.00076596200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.25","version":"380'624","reported_seq":1917,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-16T05:15:32.708873+0000","last_change":"2024-09-16T05:15:32.709398+0000","last_active":"2024-09-16T05:15:32.708873+0000","last_peered":"2024-09-16T05:15:32.708873+0000","last_clean":"2024-09-16T05:15:32.708873+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:15:32.708873+0000","last_undegraded":"2024-09-16T05:15:32.708873+0000","last_fullsized":"2024-09-16T05:15:32.708873+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"310'536","last_scrub_stamp":"2024-09-16T05:14:31.574482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.574482+0000","objects_scrubbed":8,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:40.943271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000878249,"stat_sum":{"num_bytes":13563348,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":43,"num_read_kb":57574,"num_write":25,"num_write_kb":10249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.26","version":"116'9","reported_seq":650,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.716848+0000","last_change":"2024-09-16T05:15:31.717441+0000","last_active":"2024-09-16T05:15:31.716848+0000","last_peered":"2024-09-16T05:15:31.716848+0000","last_clean":"2024-09-16T05:15:31.716848+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:15:31.716848+0000","last_undegraded":"2024-09-16T05:15:31.716848+0000","last_fullsized":"2024-09-16T05:15:31.716848+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:27.631576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.631576+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:32.925603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050394630000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":662,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.716330+0000","last_change":"2024-09-16T05:15:31.716734+0000","last_active":"2024-09-16T05:15:31.716330+0000","last_peered":"2024-09-16T05:15:31.716330+0000","last_clean":"2024-09-16T05:15:31.716330+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:15:31.716330+0000","last_undegraded":"2024-09-16T05:15:31.716330+0000","last_fullsized":"2024-09-16T05:15:31.716330+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:30.713398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.713398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:46.971140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044151620000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1444,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.718837+0000","last_change":"2024-09-16T05:15:31.719047+0000","last_active":"2024-09-16T05:15:31.718837+0000","last_peered":"2024-09-16T05:15:31.718837+0000","last_clean":"2024-09-16T05:15:31.718837+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:15:31.718837+0000","last_undegraded":"2024-09-16T05:15:31.718837+0000","last_fullsized":"2024-09-16T05:15:31.718837+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:14:45.747155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.747155+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:55.924614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041764200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.29","version":"380'536","reported_seq":1915,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.714714+0000","last_change":"2024-09-16T05:15:31.715292+0000","last_active":"2024-09-16T05:15:31.714714+0000","last_peered":"2024-09-16T05:15:31.714714+0000","last_clean":"2024-09-16T05:15:31.714714+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:15:31.714714+0000","last_undegraded":"2024-09-16T05:15:31.714714+0000","last_fullsized":"2024-09-16T05:15:31.714714+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"310'362","last_scrub_stamp":"2024-09-16T05:14:27.037060+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.037060+0000","objects_scrubbed":12,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:33.419700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077023399999999996,"stat_sum":{"num_bytes":14413657,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":25964,"num_write":68,"num_write_kb":21290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1652,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.712462+0000","last_change":"2024-09-16T05:15:31.712729+0000","last_active":"2024-09-16T05:15:31.712462+0000","last_peered":"2024-09-16T05:15:31.712462+0000","last_clean":"2024-09-16T05:15:31.712462+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:15:31.712462+0000","last_undegraded":"2024-09-16T05:15:31.712462+0000","last_fullsized":"2024-09-16T05:15:31.712462+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:14:33.725061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:33.725061+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:48.150836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077903200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2b","version":"372'201","reported_seq":1468,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.705721+0000","last_change":"2024-09-16T05:15:31.705916+0000","last_active":"2024-09-16T05:15:31.705721+0000","last_peered":"2024-09-16T05:15:31.705721+0000","last_clean":"2024-09-16T05:15:31.705721+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:15:31.705721+0000","last_undegraded":"2024-09-16T05:15:31.705721+0000","last_fullsized":"2024-09-16T05:15:31.705721+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"330'198","last_scrub_stamp":"2024-09-16T05:14:46.815272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.815272+0000","objects_scrubbed":2,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:51.410010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00058355600000000005,"stat_sum":{"num_bytes":3142942,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":8,"num_write_kb":7455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":1485,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-16T05:15:32.709176+0000","last_change":"2024-09-16T05:15:32.709480+0000","last_active":"2024-09-16T05:15:32.709176+0000","last_peered":"2024-09-16T05:15:32.709176+0000","last_clean":"2024-09-16T05:15:32.709176+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:15:32.709176+0000","last_undegraded":"2024-09-16T05:15:32.709176+0000","last_fullsized":"2024-09-16T05:15:32.709176+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:14:27.492665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.492665+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:33.458976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085218800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]},{"pgid":"2.2d","version":"377'326","reported_seq":1652,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.715343+0000","last_change":"2024-09-16T05:15:31.715729+0000","last_active":"2024-09-16T05:15:31.715343+0000","last_peered":"2024-09-16T05:15:31.715343+0000","last_clean":"2024-09-16T05:15:31.715343+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:15:31.715343+0000","last_undegraded":"2024-09-16T05:15:31.715343+0000","last_fullsized":"2024-09-16T05:15:31.715343+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'307","last_scrub_stamp":"2024-09-16T05:14:30.116429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.116429+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.944115+0000","scrub_duration":444,"objects_trimmed":0,"snaptrim_duration":0.0035826769999999998,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":4589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2e","version":"382'392","reported_seq":1709,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-16T05:15:31.718733+0000","last_change":"2024-09-16T05:15:31.718982+0000","last_active":"2024-09-16T05:15:31.718733+0000","last_peered":"2024-09-16T05:15:31.718733+0000","last_clean":"2024-09-16T05:15:31.718733+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:15:31.718733+0000","last_undegraded":"2024-09-16T05:15:31.718733+0000","last_fullsized":"2024-09-16T05:15:31.718733+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"337'278","last_scrub_stamp":"2024-09-16T05:14:47.811642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.811642+0000","objects_scrubbed":7,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.306241+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0066353510000000003,"stat_sum":{"num_bytes":8393552,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":7127,"num_write":34,"num_write_kb":7817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"1"}]},{"pgid":"2.2f","version":"378'302","reported_seq":1702,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-16T05:15:34.714066+0000","last_change":"2024-09-16T05:15:32.996751+0000","last_active":"2024-09-16T05:15:34.714066+0000","last_peered":"2024-09-16T05:15:34.714066+0000","last_clean":"2024-09-16T05:15:34.714066+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:15:34.714066+0000","last_undegraded":"2024-09-16T05:15:34.714066+0000","last_fullsized":"2024-09-16T05:15:34.714066+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:53.110450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.110450+0000","objects_scrubbed":3,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:11.644654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011791,"stat_sum":{"num_bytes":3159782,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":7,"num_write_kb":4826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"64","length":"1"},{"start":"67","length":"1"},{"start":"6c","length":"1"},{"start":"7a","length":"2"},{"start":"7e","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":167298180,"num_objects":72,"num_object_clones":28,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":3,"num_read":644,"num_read_kb":648689,"num_write":475,"num_write_kb":195626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201240576,"data_stored":241793689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15446,"ondisk_log_size":15446,"up":152,"acting":152,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089421,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65616,"kb_used_data":30268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93679536,"statfs":{"total":95995035648,"available":95927844864,"internally_reserved":0,"allocated":30994432,"data_stored":34582195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54444,"kb_used_data":19092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690708,"statfs":{"total":95995035648,"available":95939284992,"internally_reserved":0,"allocated":19550208,"data_stored":20953648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514586,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69860,"kb_used_data":30344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93675292,"statfs":{"total":95995035648,"available":95923499008,"internally_reserved":0,"allocated":31072256,"data_stored":35361737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841627,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50220,"kb_used_data":14864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93694932,"statfs":{"total":95995035648,"available":95943610368,"internally_reserved":0,"allocated":15220736,"data_stored":18912842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103264,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68980,"kb_used_data":29472,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93676172,"statfs":{"total":95995035648,"available":95924400128,"internally_reserved":0,"allocated":30179328,"data_stored":33499941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397607,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67004,"kb_used_data":27492,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93678148,"statfs":{"total":95995035648,"available":95926423552,"internally_reserved":0,"allocated":28151808,"data_stored":31316440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757355,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46492,"kb_used_data":15300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93698660,"statfs":{"total":95995035648,"available":95947427840,"internally_reserved":0,"allocated":15667200,"data_stored":18385312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920881,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47572,"kb_used_data":16380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93697580,"statfs":{"total":95995035648,"available":95946321920,"internally_reserved":0,"allocated":16773120,"data_stored":15013165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117109,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65536,"kb_used_data":30188,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93679616,"statfs":{"total":95995035648,"available":95927926784,"internally_reserved":0,"allocated":30912512,"data_stored":31814178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240396,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36148,"kb_used_data":8020,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93709004,"statfs":{"total":95995035648,"available":95958020096,"internally_reserved":0,"allocated":8212480,"data_stored":6553468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53968,"kb_used_data":18620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93691184,"statfs":{"total":95995035648,"available":95939772416,"internally_reserved":0,"allocated":19066880,"data_stored":19182436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738501,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48900,"kb_used_data":9392,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93696252,"statfs":{"total":95995035648,"available":95944962048,"internally_reserved":0,"allocated":9617408,"data_stored":8810455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5050368,"data_stored":5889133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14475264,"data_stored":16244192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3391488,"data_stored":3377975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26357760,"data_stored":28901373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12169216,"data_stored":12066372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10870784,"data_stored":15245820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23597056,"data_stored":28403635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":27451392,"data_stored":34220742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10326016,"data_stored":15663840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":26480640,"data_stored":32423493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":14958592,"data_stored":18015404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26112000,"data_stored":31341710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:15:38.055 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T05:15:38.055 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:15:38.056 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-16T05:15:38.056 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:15:38.056 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:15:38.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-16T05:15:38.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:37 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:15:38.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:37 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:15:38.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:37 smithi196 ceph-mon[30802]: pgmap v741: 87 pgs: 87 active+clean; 160 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 7 op/s 2024-09-16T05:15:38.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:37 smithi196 ceph-mon[30802]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:38.059 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:37 smithi196 ceph-mon[30802]: osdmap e388: 12 total, 12 up, 12 in 2024-09-16T05:15:38.071 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:15:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:15:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:15:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:15:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:53 2.d e112/107 2:ba21f1a9:::smithi20457400-14:head v 112'1, mlcod=0'0)", 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.147302+0000", 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 294.92213070600002, 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002142073, 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.147302+0000", 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.147302+0000", 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.147303+0000", 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8799999999999999e-07 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148009+0000", 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00070642199999999995 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148032+0000", 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3362000000000001e-05 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148050+0000", 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8063e-05 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148197+0000", 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014633199999999999 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148228+0000", 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1118999999999998e-05 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.149398+0000", 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011701369999999999 2024-09-16T05:15:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.149444+0000", 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6449999999999997e-05 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:290 2.d e117/107 2:b98a79db:::smithi20457400-31:head v 117'29, mlcod=116'25)", 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 288.715318282, 2024-09-16T05:15:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061542619999999998, 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:15:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354117+0000", 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2110000000000001e-06 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357718+0000", 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036011939999999998 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357775+0000", 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7274000000000001e-05 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357803+0000", 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7860000000000001e-05 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357882+0000", 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9091999999999994e-05 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357905+0000", 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2544999999999998e-05 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.360225+0000", 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023196480000000001 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.360269+0000", 2024-09-16T05:15:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4437999999999997e-05 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:395 2.d e125/107 2:ba7e58cf:::smithi20457400-26:head v 125'55, mlcod=123'52)", 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:56.650581+0000", 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 281.41885164299998, 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023262320000000001, 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.650581+0000", 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.650581+0000", 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.650582+0000", 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9299999999999998e-07 2024-09-16T05:15:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651480+0000", 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089784100000000003 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651514+0000", 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3856999999999998e-05 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651520+0000", 2024-09-16T05:15:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8220000000000001e-06 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651593+0000", 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2138999999999995e-05 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651609+0000", 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6759e-05 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.652874+0000", 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001264753 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.652908+0000", 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3568000000000001e-05 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:439 2.d e127/107 2:ba7e58cf:::smithi20457400-26:head v 127'64, mlcod=126'60)", 2024-09-16T05:15:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:58.659078+0000", 2024-09-16T05:15:38.086 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.41035473400001, 2024-09-16T05:15:38.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022239510000000001, 2024-09-16T05:15:38.086 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.086 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.087 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.087 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659078+0000", 2024-09-16T05:15:38.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659078+0000", 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659079+0000", 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0100000000000001e-07 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659874+0000", 2024-09-16T05:15:38.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079547000000000005 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659908+0000", 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3846000000000001e-05 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659918+0000", 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0285000000000001e-05 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.660081+0000", 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016303800000000001 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.660098+0000", 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6807e-05 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.661257+0000", 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001158844 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.661302+0000", 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5460000000000002e-05 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:442 2.d e127/107 2:ba7e58cf:::smithi20457400-26:head v 127'67, mlcod=127'64)", 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 279.40037927999998, 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047970019999999999, 2024-09-16T05:15:38.091 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3000000000000001e-07 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669058+0000", 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7699999999999999e-06 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669074+0000", 2024-09-16T05:15:38.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6268000000000001e-05 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669078+0000", 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7500000000000001e-06 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669137+0000", 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.944e-05 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669158+0000", 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0706000000000001e-05 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.673817+0000", 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046588840000000003 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.673851+0000", 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3754000000000002e-05 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:605 2.d e142/107 2:b1061e13:::smithi20457400-33:head v 142'81, mlcod=142'78)", 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 264.25482720100001, 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052172579999999998, 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:15:38.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814607+0000", 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.544e-06 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817665+0000", 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030580270000000001 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817694+0000", 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8353000000000001e-05 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817717+0000", 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3300000000000001e-05 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817769+0000", 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2147000000000001e-05 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817785+0000", 2024-09-16T05:15:38.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5874000000000001e-05 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.819789+0000", 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020036870000000001 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.819823+0000", 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4326000000000003e-05 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.100 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1122 2.d e190/107 2:ba754fcb:::smithi20457400-30:head v 190'205, mlcod=187'201)", 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:01.725779+0000", 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 216.343653819, 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022270990000000002, 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.725779+0000", 2024-09-16T05:15:38.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.725779+0000", 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.725779+0000", 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.91e-07 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726570+0000", 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079017499999999995 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726602+0000", 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2237000000000003e-05 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726617+0000", 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5514000000000001e-05 2024-09-16T05:15:38.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726767+0000", 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000150054 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726795+0000", 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7081999999999999e-05 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.727963+0000", 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011684950000000001 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.728006+0000", 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3350999999999999e-05 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1389 2.1d e210/209 2:b98a79db:::smithi20457400-31:head v 210'262, mlcod=0'0)", 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:15:38.105 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 195.46338679900001, 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047389830000000004, 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606048+0000", 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.781e-06 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608682+0000", 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026339990000000001 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608713+0000", 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0781999999999998e-05 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608722+0000", 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1139999999999995e-06 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608800+0000", 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8145000000000005e-05 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608820+0000", 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9692999999999999e-05 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.610714+0000", 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001894448 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.610785+0000", 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1020999999999996e-05 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1466 2.d e217/209 2:b21a57e7:::smithi20457400-25:head v 217'264, mlcod=210'260)", 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:28.673183+0000", 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 189.39625044300001, 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020333080000000002, 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.110 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.673183+0000", 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.673183+0000", 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.673183+0000", 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5699999999999999e-07 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.673905+0000", 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072266599999999996 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.673933+0000", 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.709e-05 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.673945+0000", 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2379e-05 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.674035+0000", 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9917e-05 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.674055+0000", 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9939e-05 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.675177+0000", 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011217449999999999 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.675216+0000", 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9415000000000001e-05 2024-09-16T05:15:38.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1598 2.d e228/209 2:b1061e13:::smithi20457400-33:head v 228'278, mlcod=224'274)", 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 178.20332468699999, 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029840370000000001, 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866109+0000", 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8800000000000003e-07 2024-09-16T05:15:38.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867732+0000", 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016233739999999999 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867777+0000", 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4693000000000001e-05 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867788+0000", 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1148000000000001e-05 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867850+0000", 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1811999999999997e-05 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867870+0000", 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9698e-05 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.869048+0000", 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011787259999999999 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.869092+0000", 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4097999999999997e-05 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.2.0:176 2.2b e260/252 2:d7c71ae7:::smithi20457400-5:39 v 260'182, mlcod=253'179)", 2024-09-16T05:15:38.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:11.719838+0000", 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 146.34959526599999, 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026440809999999999, 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719838+0000", 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719838+0000", 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719838+0000", 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719848+0000", 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0033e-05 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719905+0000", 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6076000000000003e-05 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719924+0000", 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9589000000000002e-05 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.720047+0000", 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000122698 2024-09-16T05:15:38.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.720075+0000", 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8232e-05 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.722413+0000", 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002338241 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.722482+0000", 2024-09-16T05:15:38.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8639000000000005e-05 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2143 2.2b e279/252 2:d7c71ae7:::smithi20457400-5:head v 279'188, mlcod=268'185)", 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 127.31199821200001, 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055854579999999997, 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.124 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:15:38.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7700000000000001e-07 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760458+0000", 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003022725 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760491+0000", 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3637000000000003e-05 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760500+0000", 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6240000000000008e-06 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760609+0000", 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108887 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760631+0000", 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2161e-05 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.762973+0000", 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023423939999999998 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.763020+0000", 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6853e-05 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2318 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:15:38.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 110.383574228, 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0025039739999999, 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2318 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685861+0000", 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6700000000000001e-06 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685867+0000", 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8179999999999998e-06 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685885+0000", 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7696000000000001e-05 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685895+0000", 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5899999999999997e-06 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685955+0000", 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0501000000000002e-05 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685966+0000", 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0462000000000001e-05 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688335+0000", 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0023690300000001 2024-09-16T05:15:38.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688363+0000", 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8206999999999999e-05 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2319 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 110.383455971, 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0024202550000001, 2024-09-16T05:15:38.133 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2319 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685978+0000", 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2500000000000005e-07 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685981+0000", 2024-09-16T05:15:38.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6150000000000001e-06 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685989+0000", 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6159999999999996e-06 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685995+0000", 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6250000000000004e-06 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686041+0000", 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6694999999999999e-05 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686049+0000", 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3170000000000004e-06 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688389+0000", 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002340623 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688397+0000", 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1389999999999995e-06 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2320 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 110.383375446, 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002368715, 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.138 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2320 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7199999999999999e-07 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686062+0000", 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5590000000000001e-06 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686069+0000", 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2710000000000003e-06 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686075+0000", 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7590000000000003e-06 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686126+0000", 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1384999999999997e-05 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686135+0000", 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1710000000000008e-06 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688420+0000", 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0022843889999999 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688426+0000", 2024-09-16T05:15:38.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6089999999999997e-06 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2455 2.d e310/247 2:b21a57e7:::smithi20457400-25:head v 310'372, mlcod=308'370)", 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:01.735254+0000", 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 96.334179362, 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00204689, 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.735254+0000", 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.735254+0000", 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.735254+0000", 2024-09-16T05:15:38.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7399999999999999e-07 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.736221+0000", 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00096720300000000003 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.736251+0000", 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9431e-05 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.736261+0000", 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0132999999999999e-05 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.736328+0000", 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7504999999999998e-05 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.736348+0000", 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9531999999999999e-05 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.737259+0000", 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091140100000000005 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.737301+0000", 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1310999999999998e-05 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.148 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2749 2.d e357/247 2:b21a57e7:::smithi20457400-25:head v 357'413, mlcod=356'409)", 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 31.352199739, 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079488969999999999, 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717234+0000", 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0210000000000001e-06 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722533+0000", 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052989389999999999 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722583+0000", 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9592000000000002e-05 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722593+0000", 2024-09-16T05:15:38.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0567999999999999e-05 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722673+0000", 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.983e-05 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722700+0000", 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6908999999999999e-05 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.725139+0000", 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024389920000000001 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.725182+0000", 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3046000000000001e-05 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2900 2.32 e372/275 2:4e7d172c:::smithi20457400-15:head v 372'109, mlcod=348'105)", 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:15:38.153 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.360611051999999, 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033081819999999998, 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7499999999999999e-07 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708827+0000", 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9969999999999998e-06 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708864+0000", 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6820000000000003e-05 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708880+0000", 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5908000000000001e-05 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708933+0000", 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2970000000000003e-05 2024-09-16T05:15:38.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708951+0000", 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8570999999999999e-05 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.712084+0000", 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003132297 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.712130+0000", 2024-09-16T05:15:38.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6443999999999998e-05 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2898 2.2b e372/329 2:d7c71ae7:::smithi20457400-5:head v 372'201, mlcod=330'198)", 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 16.344853906000001, 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085128900000000004, 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724580+0000", 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729164+0000", 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045841340000000001 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729217+0000", 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3152999999999997e-05 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729225+0000", 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9850000000000006e-06 2024-09-16T05:15:38.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729320+0000", 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5067000000000003e-05 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729381+0000", 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1297999999999995e-05 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.733031+0000", 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036494600000000002 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.733092+0000", 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1371000000000006e-05 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:3108 2.2b e388/329 2:d7c71ae7:::smithi20457400-5:head v 388'203, mlcod=360'200)", 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:37.720518+0000", 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.34891554400000002, 2024-09-16T05:15:38.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023880889999999999, 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.720518+0000", 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.720518+0000", 2024-09-16T05:15:38.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.720518+0000", 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721510+0000", 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099188000000000011 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:38.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721531+0000", 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1002999999999999e-05 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721538+0000", 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5669999999999997e-06 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721634+0000", 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6298000000000005e-05 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721660+0000", 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5588999999999998e-05 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.722873+0000", 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001213446 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.722906+0000", 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2129999999999999e-05 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.166 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.167 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:38.167 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:38.167 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:15:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:37 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:15:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:37 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:15:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:37 smithi204 ceph-mon[30933]: pgmap v741: 87 pgs: 87 active+clean; 160 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 7 op/s 2024-09-16T05:15:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:37 smithi204 ceph-mon[30933]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:37 smithi204 ceph-mon[30933]: osdmap e388: 12 total, 12 up, 12 in 2024-09-16T05:15:38.190 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:38.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:38.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:38.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:38.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:38.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:38.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:38.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:38.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:38.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:38.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:38.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:38.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:38.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:38.712 INFO:tasks.rados.rados.2.smithi204.stdout:1217: done (4 left) 2024-09-16T05:15:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:1218: done (3 left) 2024-09-16T05:15:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:1219: done (2 left) 2024-09-16T05:15:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:1220: done (1 left) 2024-09-16T05:15:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:1221: done (0 left) 2024-09-16T05:15:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:1222: read oid 14 snap 129 2024-09-16T05:15:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:1222: expect (ObjNum 415 snap 123 seq_num 415) 2024-09-16T05:15:38.713 INFO:tasks.rados.rados.2.smithi204.stdout:1223: delete oid 12 current snap is 132 2024-09-16T05:15:38.715 INFO:tasks.rados.rados.2.smithi204.stdout:1223: done (1 left) 2024-09-16T05:15:38.715 INFO:tasks.rados.rados.2.smithi204.stdout:1224: delete oid 32 current snap is 132 2024-09-16T05:15:38.717 INFO:tasks.rados.rados.2.smithi204.stdout:1224: done (1 left) 2024-09-16T05:15:38.717 INFO:tasks.rados.rados.2.smithi204.stdout:1225: read oid 24 snap 130 2024-09-16T05:15:38.717 INFO:tasks.rados.rados.2.smithi204.stdout:1225: expect (ObjNum 437 snap 128 seq_num 437) 2024-09-16T05:15:38.717 INFO:tasks.rados.rados.2.smithi204.stdout:1226: snap_remove snap 126 2024-09-16T05:15:38.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:38.812 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:38.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:38.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:38.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:15:38.930 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:38.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:38.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:39.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:39.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:38 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:15:39.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:38 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:15:39.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:38 smithi196 ceph-mon[30802]: osdmap e389: 12 total, 12 up, 12 in 2024-09-16T05:15:39.062 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:39.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:39.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:38 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:15:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:38 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:15:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:38 smithi032 ceph-mon[27388]: osdmap e389: 12 total, 12 up, 12 in 2024-09-16T05:15:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:38 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:15:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:38 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:15:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:38 smithi204 ceph-mon[30933]: osdmap e389: 12 total, 12 up, 12 in 2024-09-16T05:15:39.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:39.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:39.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:39.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:39.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:39.386 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:15:39.386 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T05:15:39.386 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:15:39.394 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:39.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:39.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:39.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:39.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:39.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:39.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:39.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:15:39.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:39.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:39.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:39.715 INFO:tasks.rados.rados.2.smithi204.stdout:1226: done (2 left) 2024-09-16T05:15:39.715 INFO:tasks.rados.rados.2.smithi204.stdout:1227: read oid 37 snap 131 2024-09-16T05:15:39.716 INFO:tasks.rados.rados.2.smithi204.stdout:1227: expect (ObjNum 426 snap 124 seq_num 426) 2024-09-16T05:15:39.716 INFO:tasks.rados.rados.2.smithi204.stdout:1228: read oid 43 snap -1 2024-09-16T05:15:39.716 INFO:tasks.rados.rados.2.smithi204.stdout:1228: expect deleted 2024-09-16T05:15:39.716 INFO:tasks.rados.rados.2.smithi204.stdout:1229: snap_create 2024-09-16T05:15:39.716 INFO:tasks.rados.rados.2.smithi204.stdout:1222: expect (ObjNum 415 snap 123 seq_num 415) 2024-09-16T05:15:39.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:39.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:39.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:39.891 INFO:tasks.rados.rados.2.smithi204.stdout:1225: expect (ObjNum 437 snap 128 seq_num 437) 2024-09-16T05:15:39.915 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:39.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:39.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:39.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:40.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:39 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:15:40.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:39 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:15:40.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:39 smithi032 ceph-mon[27388]: pgmap v744: 87 pgs: 87 active+clean; 163 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2024-09-16T05:15:40.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:39 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:15:40.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:39 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:15:40.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:39 smithi032 ceph-mon[27388]: osdmap e390: 12 total, 12 up, 12 in 2024-09-16T05:15:40.038 INFO:tasks.rados.rados.2.smithi204.stdout:1227: expect (ObjNum 426 snap 124 seq_num 426) 2024-09-16T05:15:40.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:39 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:15:40.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:39 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:15:40.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:39 smithi196 ceph-mon[30802]: pgmap v744: 87 pgs: 87 active+clean; 163 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2024-09-16T05:15:40.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:39 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:15:40.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:39 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:15:40.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:39 smithi196 ceph-mon[30802]: osdmap e390: 12 total, 12 up, 12 in 2024-09-16T05:15:40.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:40.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:40.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:39 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:15:40.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:39 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:15:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:39 smithi204 ceph-mon[30933]: pgmap v744: 87 pgs: 87 active+clean; 163 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2024-09-16T05:15:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:39 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:15:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:39 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:15:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:39 smithi204 ceph-mon[30933]: osdmap e390: 12 total, 12 up, 12 in 2024-09-16T05:15:40.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:40.208 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:40.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:40.209 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:40.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:40.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:40.400 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:40.400 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:40.421 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:40.421 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:40.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:40.441 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:40.442 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:40.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:40.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:40.462 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:40.484 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:40.484 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:40.506 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:40.506 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:40.527 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:40.528 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:40.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:15:40.550 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:40.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:40.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:40.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:40.574 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:40.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:40.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:40.585 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:40.585 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:40.596 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:40.596 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:40.617 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:40.618 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:40.639 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:40.639 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:40.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:40.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:40.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:40.708 INFO:tasks.rados.rados.2.smithi204.stdout:1222: done (4 left) 2024-09-16T05:15:40.708 INFO:tasks.rados.rados.2.smithi204.stdout:1225: done (3 left) 2024-09-16T05:15:40.708 INFO:tasks.rados.rados.2.smithi204.stdout:1227: done (2 left) 2024-09-16T05:15:40.708 INFO:tasks.rados.rados.2.smithi204.stdout:1228: done (1 left) 2024-09-16T05:15:40.708 INFO:tasks.rados.rados.2.smithi204.stdout:1229: done (0 left) 2024-09-16T05:15:40.708 INFO:tasks.rados.rados.2.smithi204.stdout:1230: snap_create 2024-09-16T05:15:40.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:40.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:40.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:40 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:15:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:40 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:15:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3794412112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:40 smithi032 ceph-mon[27388]: osdmap e391: 12 total, 12 up, 12 in 2024-09-16T05:15:40.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:40.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:40.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:41.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:41.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:41.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:41.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:40 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:15:41.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:40 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:15:41.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3794412112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:41.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:40 smithi196 ceph-mon[30802]: osdmap e391: 12 total, 12 up, 12 in 2024-09-16T05:15:41.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:41.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:40 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:15:41.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:40 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:15:41.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3794412112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:41.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:40 smithi204 ceph-mon[30933]: osdmap e391: 12 total, 12 up, 12 in 2024-09-16T05:15:41.191 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:41.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:41.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:41.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:41.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:15:41.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:41.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:41.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:41.429 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:41.446 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:41.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:41.541 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:41.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:41.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:41.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:41.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:41.696 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:41.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:41.710 INFO:tasks.rados.rados.2.smithi204.stdout:1230: done (0 left) 2024-09-16T05:15:41.710 INFO:tasks.rados.rados.2.smithi204.stdout:1231: copy_from oid 1 from oid 16 current snap is 134 2024-09-16T05:15:41.710 INFO:tasks.rados.rados.2.smithi204.stdout:1232: read oid 47 snap 127 2024-09-16T05:15:41.710 INFO:tasks.rados.rados.2.smithi204.stdout:1232: expect deleted 2024-09-16T05:15:41.710 INFO:tasks.rados.rados.2.smithi204.stdout:1233: snap_remove snap 128 2024-09-16T05:15:41.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:41.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:41.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:41.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:41.946 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:41.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:41.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:41 smithi196 ceph-mon[30802]: 2.44 deep-scrub starts 2024-09-16T05:15:41.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:41 smithi196 ceph-mon[30802]: 2.44 deep-scrub ok 2024-09-16T05:15:41.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:41 smithi196 ceph-mon[30802]: pgmap v747: 87 pgs: 87 active+clean; 166 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2024-09-16T05:15:41.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:41 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:15:41.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:41 smithi196 ceph-mon[30802]: 2.29 scrub ok 2024-09-16T05:15:41.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:41 smithi196 ceph-mon[30802]: 2.b deep-scrub starts 2024-09-16T05:15:41.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:41 smithi196 ceph-mon[30802]: 2.b deep-scrub ok 2024-09-16T05:15:41.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:41 smithi196 ceph-mon[30802]: osdmap e392: 12 total, 12 up, 12 in 2024-09-16T05:15:42.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:42.074 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:42.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:42.082 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:42.083 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:41 smithi032 ceph-mon[27388]: 2.44 deep-scrub starts 2024-09-16T05:15:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:41 smithi032 ceph-mon[27388]: 2.44 deep-scrub ok 2024-09-16T05:15:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:41 smithi032 ceph-mon[27388]: pgmap v747: 87 pgs: 87 active+clean; 166 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2024-09-16T05:15:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:41 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:15:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:41 smithi032 ceph-mon[27388]: 2.29 scrub ok 2024-09-16T05:15:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:41 smithi032 ceph-mon[27388]: 2.b deep-scrub starts 2024-09-16T05:15:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:41 smithi032 ceph-mon[27388]: 2.b deep-scrub ok 2024-09-16T05:15:42.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:41 smithi032 ceph-mon[27388]: osdmap e392: 12 total, 12 up, 12 in 2024-09-16T05:15:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:41 smithi204 ceph-mon[30933]: 2.44 deep-scrub starts 2024-09-16T05:15:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:41 smithi204 ceph-mon[30933]: 2.44 deep-scrub ok 2024-09-16T05:15:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:41 smithi204 ceph-mon[30933]: pgmap v747: 87 pgs: 87 active+clean; 166 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2024-09-16T05:15:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:41 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:15:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:41 smithi204 ceph-mon[30933]: 2.29 scrub ok 2024-09-16T05:15:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:41 smithi204 ceph-mon[30933]: 2.b deep-scrub starts 2024-09-16T05:15:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:41 smithi204 ceph-mon[30933]: 2.b deep-scrub ok 2024-09-16T05:15:42.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:41 smithi204 ceph-mon[30933]: osdmap e392: 12 total, 12 up, 12 in 2024-09-16T05:15:42.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:42.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:42.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:42.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:42.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:42.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:42.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:42.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:42.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:42.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:42.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:42.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:15:42.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:42.707 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:42.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:42.708 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:15:42.712 INFO:tasks.rados.rados.2.smithi204.stdout:1233: done (2 left) 2024-09-16T05:15:42.713 INFO:tasks.rados.rados.2.smithi204.stdout:1234: write (excl) oid 23 current snap is 134 2024-09-16T05:15:42.713 INFO:tasks.rados.rados.2.smithi204.stdout:1234: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-16T05:15:42.719 INFO:tasks.rados.rados.2.smithi204.stdout:1234: writing smithi20457400-23 from 662308 to 1221338 tid 1 2024-09-16T05:15:42.725 INFO:tasks.rados.rados.2.smithi204.stdout:1234: writing smithi20457400-23 from 1632005 to 2235816 tid 2 2024-09-16T05:15:42.732 INFO:tasks.rados.rados.2.smithi204.stdout:1234: writing smithi20457400-23 from 2860044 to 3462308 tid 3 2024-09-16T05:15:42.732 INFO:tasks.rados.rados.2.smithi204.stdout:1235: snap_create 2024-09-16T05:15:42.732 INFO:tasks.rados.rados.2.smithi204.stdout:1231: finishing copy_from racing read to smithi20457400-1 2024-09-16T05:15:42.732 INFO:tasks.rados.rados.2.smithi204.stdout:1231: finishing copy_from to smithi20457400-1 2024-09-16T05:15:42.732 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 493 (ObjNum 428 snap 124 seq_num 428) dirty exists 2024-09-16T05:15:42.732 INFO:tasks.rados.rados.2.smithi204.stdout:1234: finishing write tid 1 to smithi20457400-23 2024-09-16T05:15:42.734 INFO:tasks.rados.rados.2.smithi204.stdout:1234: finishing write tid 2 to smithi20457400-23 2024-09-16T05:15:42.734 INFO:tasks.rados.rados.2.smithi204.stdout:1234: finishing write tid 3 to smithi20457400-23 2024-09-16T05:15:42.742 INFO:tasks.rados.rados.2.smithi204.stdout:1234: finishing write tid 4 to smithi20457400-23 2024-09-16T05:15:42.742 INFO:tasks.rados.rados.2.smithi204.stdout:1234: finishing write tid 5 to smithi20457400-23 2024-09-16T05:15:42.742 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 299 (ObjNum 444 snap 134 seq_num 444) dirty exists 2024-09-16T05:15:42.742 INFO:tasks.rados.rados.2.smithi204.stdout:1234: left oid 23 (ObjNum 444 snap 134 seq_num 444) 2024-09-16T05:15:42.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:42.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:42.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:42.929 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:42.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:42.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:42.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:42.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/521559092' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:15:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:42 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:15:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:42 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:15:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:42 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:15:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:42 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:15:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:42 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/521559092' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:15:42.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:42 smithi196 ceph-mon[30802]: osdmap e393: 12 total, 12 up, 12 in 2024-09-16T05:15:43.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:43.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:43.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:43.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/521559092' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:15:43.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:42 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:15:43.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:42 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:15:43.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:42 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:43.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:42 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:43.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:42 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:43.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/521559092' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:15:43.097 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:42 smithi032 ceph-mon[27388]: osdmap e393: 12 total, 12 up, 12 in 2024-09-16T05:15:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/521559092' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:15:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:42 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:15:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:42 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:15:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:42 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:15:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:42 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:15:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:42 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/521559092' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:15:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:42 smithi204 ceph-mon[30933]: osdmap e393: 12 total, 12 up, 12 in 2024-09-16T05:15:43.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:43.208 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:43.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:43.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:43.335 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:43.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:43.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:15:43.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:43.440 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:43.440 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:43.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:43.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:43.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:43.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:43.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:43.626 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:43.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:43.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:43.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:43.716 INFO:tasks.rados.rados.2.smithi204.stdout:1231: done (3 left) 2024-09-16T05:15:43.716 INFO:tasks.rados.rados.2.smithi204.stdout:1232: done (2 left) 2024-09-16T05:15:43.716 INFO:tasks.rados.rados.2.smithi204.stdout:1234: done (1 left) 2024-09-16T05:15:43.716 INFO:tasks.rados.rados.2.smithi204.stdout:1235: done (0 left) 2024-09-16T05:15:43.716 INFO:tasks.rados.rados.2.smithi204.stdout:1236: snap_create 2024-09-16T05:15:43.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:43.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:43.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:43.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:43.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:43.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:44.033 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:43 smithi032 ceph-mon[27388]: pgmap v750: 87 pgs: 87 active+clean; 171 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-16T05:15:44.033 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:43 smithi032 ceph-mon[27388]: osdmap e394: 12 total, 12 up, 12 in 2024-09-16T05:15:44.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:44.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:44.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:44.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:43 smithi196 ceph-mon[30802]: pgmap v750: 87 pgs: 87 active+clean; 171 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-16T05:15:44.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:43 smithi196 ceph-mon[30802]: osdmap e394: 12 total, 12 up, 12 in 2024-09-16T05:15:44.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:15:44.141 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:44.141 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:44.163 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:44.163 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:43 smithi204 ceph-mon[30933]: pgmap v750: 87 pgs: 87 active+clean; 171 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2024-09-16T05:15:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:43 smithi204 ceph-mon[30933]: osdmap e394: 12 total, 12 up, 12 in 2024-09-16T05:15:44.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:44.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:44.204 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:44.205 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:44.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:44.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:44.226 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:44.226 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:44.248 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:44.249 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:44.269 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:44.269 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:44.292 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:44.292 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:44.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:44.314 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:44.314 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:44.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:44.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:44.336 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:44.337 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:44.358 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:44.359 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:44.380 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:44.381 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:44.425 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:44.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:44.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:44.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:44.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:44.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:44.579 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:44.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:44.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:44.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:44.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:44.715 INFO:tasks.rados.rados.2.smithi204.stdout:1236: done (0 left) 2024-09-16T05:15:44.715 INFO:tasks.rados.rados.2.smithi204.stdout:1237: delete oid 26 current snap is 136 2024-09-16T05:15:44.718 INFO:tasks.rados.rados.2.smithi204.stdout:1237: done (0 left) 2024-09-16T05:15:44.718 INFO:tasks.rados.rados.2.smithi204.stdout:1238: snap_create 2024-09-16T05:15:44.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:44.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:44.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:44.883 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:15:44.883 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:15:44.883 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:15:44.883 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:15:44.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:44.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:44.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:45.021 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:44 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:15:45.021 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:44 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:15:45.021 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4013132048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:45.021 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:44 smithi032 ceph-mon[27388]: osdmap e395: 12 total, 12 up, 12 in 2024-09-16T05:15:45.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:45.083 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:15:45.083 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:15:45.084 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-16T05:15:45.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:15:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:45.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:45.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:44 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:15:45.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:44 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:15:45.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4013132048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:45.161 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:44 smithi196 ceph-mon[30802]: osdmap e395: 12 total, 12 up, 12 in 2024-09-16T05:15:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:44 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:15:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:44 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:15:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4013132048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:44 smithi204 ceph-mon[30933]: osdmap e395: 12 total, 12 up, 12 in 2024-09-16T05:15:45.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:45.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:45.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:45.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:45.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:45.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:45.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:45.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:15:45.465 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:45.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:45.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:15:45.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:45.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:45.688 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:45.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:45.710 INFO:tasks.rados.rados.2.smithi204.stdout:1238: done (0 left) 2024-09-16T05:15:45.710 INFO:tasks.rados.rados.2.smithi204.stdout:1239: snap_remove snap 135 2024-09-16T05:15:45.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:45.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:45.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:45.843 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:45.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:45.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:45.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:45.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:45.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:46.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:15:46.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:45 smithi032 ceph-mon[27388]: pgmap v753: 87 pgs: 87 active+clean; 171 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 820 KiB/s wr, 12 op/s 2024-09-16T05:15:46.052 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:45 smithi032 ceph-mon[27388]: osdmap e396: 12 total, 12 up, 12 in 2024-09-16T05:15:46.055 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:15:46.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:45 smithi196 ceph-mon[30802]: pgmap v753: 87 pgs: 87 active+clean; 171 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 820 KiB/s wr, 12 op/s 2024-09-16T05:15:46.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:45 smithi196 ceph-mon[30802]: osdmap e396: 12 total, 12 up, 12 in 2024-09-16T05:15:46.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:46.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:46.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:45 smithi204 ceph-mon[30933]: pgmap v753: 87 pgs: 87 active+clean; 171 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 820 KiB/s wr, 12 op/s 2024-09-16T05:15:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:45 smithi204 ceph-mon[30933]: osdmap e396: 12 total, 12 up, 12 in 2024-09-16T05:15:46.185 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:15:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:15:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:15:46.185 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:15:46.185 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:15:46.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:46.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:46.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:46.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:46.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:46.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:46.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:46.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:46.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:46.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:46.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:46.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:46.712 INFO:tasks.rados.rados.2.smithi204.stdout:1239: done (0 left) 2024-09-16T05:15:46.712 INFO:tasks.rados.rados.2.smithi204.stdout:1240: copy_from oid 18 from oid 42 current snap is 137 2024-09-16T05:15:46.712 INFO:tasks.rados.rados.2.smithi204.stdout:1241: snap_remove snap 130 2024-09-16T05:15:46.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:46.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:46.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:46.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:15:46.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:46.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:46.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:46.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:46.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-09-16T05:15:45.706732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":182779227,"num_objects":84,"num_object_clones":37,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":6,"num_read":707,"num_read_kb":716475,"num_write":502,"num_write_kb":208495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17034,"ondisk_log_size":17034,"up":174,"acting":174,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":703392,"kb_used_data":273924,"kb_used_omap":27,"kb_used_meta":429156,"kb_avail":1124238432,"statfs":{"total":1151940427776,"available":1151220154368,"internally_reserved":0,"allocated":280498176,"data_stored":311739784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":439455911},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3193396,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10248,"num_write":1,"num_write_kb":3119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000042"},"pg_stats":[{"pgid":"2.4b","version":"244'177","reported_seq":1588,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.712639+0000","last_change":"2024-09-16T05:15:43.043597+0000","last_active":"2024-09-16T05:15:44.712639+0000","last_peered":"2024-09-16T05:15:44.712639+0000","last_clean":"2024-09-16T05:15:44.712639+0000","last_became_active":"2024-09-16T05:14:30.853998+0000","last_became_peered":"2024-09-16T05:14:30.853998+0000","last_unstale":"2024-09-16T05:15:44.712639+0000","last_undegraded":"2024-09-16T05:15:44.712639+0000","last_fullsized":"2024-09-16T05:15:44.712639+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:14:32.337052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:32.337052+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011879709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.4a","version":"378'461","reported_seq":1818,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.713884+0000","last_change":"2024-09-16T05:15:39.714064+0000","last_active":"2024-09-16T05:15:39.713884+0000","last_peered":"2024-09-16T05:15:39.713884+0000","last_clean":"2024-09-16T05:15:39.713884+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:15:39.713884+0000","last_undegraded":"2024-09-16T05:15:39.713884+0000","last_fullsized":"2024-09-16T05:15:39.713884+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"378'461","last_scrub_stamp":"2024-09-16T05:15:37.597739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:37.597739+0000","objects_scrubbed":1,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:42.564676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000925562,"stat_sum":{"num_bytes":2806835,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.49","version":"245'220","reported_seq":1605,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.894505+0000","last_change":"2024-09-16T05:15:39.894725+0000","last_active":"2024-09-16T05:15:39.894505+0000","last_peered":"2024-09-16T05:15:39.894505+0000","last_clean":"2024-09-16T05:15:39.894505+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:15:39.894505+0000","last_undegraded":"2024-09-16T05:15:39.894505+0000","last_fullsized":"2024-09-16T05:15:39.894505+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:14:30.917243+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.917243+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.995168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040488200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.48","version":"393'183","reported_seq":1557,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.818803+0000","last_change":"2024-09-16T05:15:43.170032+0000","last_active":"2024-09-16T05:15:43.818803+0000","last_peered":"2024-09-16T05:15:43.818803+0000","last_clean":"2024-09-16T05:15:43.818803+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:15:43.818803+0000","last_undegraded":"2024-09-16T05:15:43.818803+0000","last_fullsized":"2024-09-16T05:15:43.818803+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"378'179","last_scrub_stamp":"2024-09-16T05:15:39.057858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.057858+0000","objects_scrubbed":1,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:53.022845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00226304,"stat_sum":{"num_bytes":6336338,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":20676,"num_write":2,"num_write_kb":6189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":395,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.771167+0000","last_change":"2024-09-16T05:15:42.710318+0000","last_active":"2024-09-16T05:15:44.771167+0000","last_peered":"2024-09-16T05:15:44.771167+0000","last_clean":"2024-09-16T05:15:44.771167+0000","last_became_active":"2024-09-16T05:15:16.867821+0000","last_became_peered":"2024-09-16T05:15:16.867821+0000","last_unstale":"2024-09-16T05:15:44.771167+0000","last_undegraded":"2024-09-16T05:15:44.771167+0000","last_fullsized":"2024-09-16T05:15:44.771167+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:28.987418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.987418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049350999999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":1420,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.712928+0000","last_change":"2024-09-16T05:15:43.043150+0000","last_active":"2024-09-16T05:15:44.712928+0000","last_peered":"2024-09-16T05:15:44.712928+0000","last_clean":"2024-09-16T05:15:44.712928+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:15:44.712928+0000","last_undegraded":"2024-09-16T05:15:44.712928+0000","last_fullsized":"2024-09-16T05:15:44.712928+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:15:37.935128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:37.935128+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:53.008425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073424799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":1839,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.712979+0000","last_change":"2024-09-16T05:15:43.042869+0000","last_active":"2024-09-16T05:15:44.712979+0000","last_peered":"2024-09-16T05:15:44.712979+0000","last_clean":"2024-09-16T05:15:44.712979+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:15:44.712979+0000","last_undegraded":"2024-09-16T05:15:44.712979+0000","last_fullsized":"2024-09-16T05:15:44.712979+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:14:31.325082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.325082+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00066828899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":1502,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.712721+0000","last_change":"2024-09-16T05:15:43.042845+0000","last_active":"2024-09-16T05:15:44.712721+0000","last_peered":"2024-09-16T05:15:44.712721+0000","last_clean":"2024-09-16T05:15:44.712721+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:15:44.712721+0000","last_undegraded":"2024-09-16T05:15:44.712721+0000","last_fullsized":"2024-09-16T05:15:44.712721+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.970206+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:54.397074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061482599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":377,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383082+0000","last_change":"2024-09-16T05:15:39.715823+0000","last_active":"2024-09-16T05:15:41.383082+0000","last_peered":"2024-09-16T05:15:41.383082+0000","last_clean":"2024-09-16T05:15:41.383082+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:15:41.383082+0000","last_undegraded":"2024-09-16T05:15:41.383082+0000","last_fullsized":"2024-09-16T05:15:41.383082+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:14:28.659562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.659562+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:52.349654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058643599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":368,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.713946+0000","last_change":"2024-09-16T05:15:42.711540+0000","last_active":"2024-09-16T05:15:43.713946+0000","last_peered":"2024-09-16T05:15:43.713946+0000","last_clean":"2024-09-16T05:15:43.713946+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:15:43.713946+0000","last_undegraded":"2024-09-16T05:15:43.713946+0000","last_fullsized":"2024-09-16T05:15:43.713946+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:33.678710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:33.678710+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:42.981450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097023599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":464,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.713144+0000","last_change":"2024-09-16T05:15:39.713479+0000","last_active":"2024-09-16T05:15:39.713144+0000","last_peered":"2024-09-16T05:15:39.713144+0000","last_clean":"2024-09-16T05:15:39.713144+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:15:39.713144+0000","last_undegraded":"2024-09-16T05:15:39.713144+0000","last_fullsized":"2024-09-16T05:15:39.713144+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:15.794439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.794439+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:32.683731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00062625099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.40","version":"205'33","reported_seq":1423,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383428+0000","last_change":"2024-09-16T05:15:39.717846+0000","last_active":"2024-09-16T05:15:41.383428+0000","last_peered":"2024-09-16T05:15:41.383428+0000","last_clean":"2024-09-16T05:15:41.383428+0000","last_became_active":"2024-09-16T05:13:45.682851+0000","last_became_peered":"2024-09-16T05:13:45.682851+0000","last_unstale":"2024-09-16T05:15:41.383428+0000","last_undegraded":"2024-09-16T05:15:41.383428+0000","last_fullsized":"2024-09-16T05:15:41.383428+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:07.789202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.789202+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.178902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022127890000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.3f","version":"378'282","reported_seq":1669,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383595+0000","last_change":"2024-09-16T05:15:39.718167+0000","last_active":"2024-09-16T05:15:41.383595+0000","last_peered":"2024-09-16T05:15:41.383595+0000","last_clean":"2024-09-16T05:15:41.383595+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:15:41.383595+0000","last_undegraded":"2024-09-16T05:15:41.383595+0000","last_fullsized":"2024-09-16T05:15:41.383595+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"357'270","last_scrub_stamp":"2024-09-16T05:15:09.797411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.797411+0000","objects_scrubbed":3,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:33.664044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002489607,"stat_sum":{"num_bytes":2285818,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":15220,"num_write":15,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.3e","version":"378'353","reported_seq":1707,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.495744+0000","last_change":"2024-09-16T05:15:39.718647+0000","last_active":"2024-09-16T05:15:42.495744+0000","last_peered":"2024-09-16T05:15:42.495744+0000","last_clean":"2024-09-16T05:15:42.495744+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:15:42.495744+0000","last_undegraded":"2024-09-16T05:15:42.495744+0000","last_fullsized":"2024-09-16T05:15:42.495744+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"360'342","last_scrub_stamp":"2024-09-16T05:15:09.920637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.920637+0000","objects_scrubbed":4,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:34.280750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059016500000000003,"stat_sum":{"num_bytes":2437081,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":2380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1941,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.712807+0000","last_change":"2024-09-16T05:15:43.043803+0000","last_active":"2024-09-16T05:15:44.712807+0000","last_peered":"2024-09-16T05:15:44.712807+0000","last_clean":"2024-09-16T05:15:44.712807+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:15:44.712807+0000","last_undegraded":"2024-09-16T05:15:44.712807+0000","last_fullsized":"2024-09-16T05:15:44.712807+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:15:09.150849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.150849+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:14.360236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001062017,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.3c","version":"390'452","reported_seq":1923,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383082+0000","last_change":"2024-09-16T05:15:39.718347+0000","last_active":"2024-09-16T05:15:41.383082+0000","last_peered":"2024-09-16T05:15:41.383082+0000","last_clean":"2024-09-16T05:15:41.383082+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:15:41.383082+0000","last_undegraded":"2024-09-16T05:15:41.383082+0000","last_fullsized":"2024-09-16T05:15:41.383082+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"342'400","last_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:14:51.845721+0000","objects_scrubbed":6,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:02.706924+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0029740779999999998,"stat_sum":{"num_bytes":5540525,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":33088,"num_write":19,"num_write_kb":6444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.3b","version":"361'294","reported_seq":1680,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.894796+0000","last_change":"2024-09-16T05:15:39.894968+0000","last_active":"2024-09-16T05:15:39.894796+0000","last_peered":"2024-09-16T05:15:39.894796+0000","last_clean":"2024-09-16T05:15:39.894796+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:15:39.894796+0000","last_undegraded":"2024-09-16T05:15:39.894796+0000","last_fullsized":"2024-09-16T05:15:39.894796+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"300'290","last_scrub_stamp":"2024-09-16T05:15:06.955285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.955285+0000","objects_scrubbed":2,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:13.119858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059548999999999995,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":31094,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.3a","version":"393'460","reported_seq":1825,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.820075+0000","last_change":"2024-09-16T05:15:43.170929+0000","last_active":"2024-09-16T05:15:43.820075+0000","last_peered":"2024-09-16T05:15:43.820075+0000","last_clean":"2024-09-16T05:15:43.820075+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:15:43.820075+0000","last_undegraded":"2024-09-16T05:15:43.820075+0000","last_fullsized":"2024-09-16T05:15:43.820075+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"356'429","last_scrub_stamp":"2024-09-16T05:15:10.141866+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.141866+0000","objects_scrubbed":2,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:39.226652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0028339300000000001,"stat_sum":{"num_bytes":7237161,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":9241,"num_write":24,"num_write_kb":11603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":487,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.819568+0000","last_change":"2024-09-16T05:15:43.169193+0000","last_active":"2024-09-16T05:15:43.819568+0000","last_peered":"2024-09-16T05:15:43.819568+0000","last_clean":"2024-09-16T05:15:43.819568+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:15:43.819568+0000","last_undegraded":"2024-09-16T05:15:43.819568+0000","last_fullsized":"2024-09-16T05:15:43.819568+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:15.232072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.232072+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:15.950238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011642180000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":481,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.713767+0000","last_change":"2024-09-16T05:15:42.710946+0000","last_active":"2024-09-16T05:15:43.713767+0000","last_peered":"2024-09-16T05:15:43.713767+0000","last_clean":"2024-09-16T05:15:43.713767+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:15:43.713767+0000","last_undegraded":"2024-09-16T05:15:43.713767+0000","last_fullsized":"2024-09-16T05:15:43.713767+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:13.450261+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:13.450261+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:30.702308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000584083,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":552,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.714120+0000","last_change":"2024-09-16T05:15:39.714391+0000","last_active":"2024-09-16T05:15:39.714120+0000","last_peered":"2024-09-16T05:15:39.714120+0000","last_clean":"2024-09-16T05:15:39.714120+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:15:39.714120+0000","last_undegraded":"2024-09-16T05:15:39.714120+0000","last_fullsized":"2024-09-16T05:15:39.714120+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.650972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.473124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009973409999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.36","version":"272'60","reported_seq":569,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.712988+0000","last_change":"2024-09-16T05:15:43.043298+0000","last_active":"2024-09-16T05:15:44.712988+0000","last_peered":"2024-09-16T05:15:44.712988+0000","last_clean":"2024-09-16T05:15:44.712988+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:15:44.712988+0000","last_undegraded":"2024-09-16T05:15:44.712988+0000","last_fullsized":"2024-09-16T05:15:44.712988+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:14:49.375386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.375386+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.901819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084580199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.35","version":"380'503","reported_seq":1926,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.713006+0000","last_change":"2024-09-16T05:15:43.043395+0000","last_active":"2024-09-16T05:15:44.713006+0000","last_peered":"2024-09-16T05:15:44.713006+0000","last_clean":"2024-09-16T05:15:44.713006+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:15:44.713006+0000","last_undegraded":"2024-09-16T05:15:44.713006+0000","last_fullsized":"2024-09-16T05:15:44.713006+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'421","last_scrub_stamp":"2024-09-16T05:14:34.349392+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:14:34.349392+0000","objects_scrubbed":4,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:45.109323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00085642499999999996,"stat_sum":{"num_bytes":9815498,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":23,"num_read_kb":24061,"num_write":19,"num_write_kb":7769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.34","version":"378'308","reported_seq":1294,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.895306+0000","last_change":"2024-09-16T05:15:39.895487+0000","last_active":"2024-09-16T05:15:39.895306+0000","last_peered":"2024-09-16T05:15:39.895306+0000","last_clean":"2024-09-16T05:15:39.895306+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:15:39.895306+0000","last_undegraded":"2024-09-16T05:15:39.895306+0000","last_fullsized":"2024-09-16T05:15:39.895306+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'243","last_scrub_stamp":"2024-09-16T05:14:50.913931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.913931+0000","objects_scrubbed":5,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:05.438088+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011183250000000001,"stat_sum":{"num_bytes":5613670,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":41071,"num_write":9,"num_write_kb":6969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.33","version":"181'15","reported_seq":555,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383187+0000","last_change":"2024-09-16T05:15:39.716595+0000","last_active":"2024-09-16T05:15:41.383187+0000","last_peered":"2024-09-16T05:15:41.383187+0000","last_clean":"2024-09-16T05:15:41.383187+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:15:41.383187+0000","last_undegraded":"2024-09-16T05:15:41.383187+0000","last_fullsized":"2024-09-16T05:15:41.383187+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:46.792111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.792111+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:57.178885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001245768,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.32","version":"380'125","reported_seq":638,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.495793+0000","last_change":"2024-09-16T05:15:39.718544+0000","last_active":"2024-09-16T05:15:42.495793+0000","last_peered":"2024-09-16T05:15:42.495793+0000","last_clean":"2024-09-16T05:15:42.495793+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:15:42.495793+0000","last_undegraded":"2024-09-16T05:15:42.495793+0000","last_fullsized":"2024-09-16T05:15:42.495793+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'101","last_scrub_stamp":"2024-09-16T05:14:49.915945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.915945+0000","objects_scrubbed":2,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:03.335459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00087363499999999999,"stat_sum":{"num_bytes":6857355,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":17,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.31","version":"380'162","reported_seq":1187,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.771195+0000","last_change":"2024-09-16T05:15:42.710109+0000","last_active":"2024-09-16T05:15:44.771195+0000","last_peered":"2024-09-16T05:15:44.771195+0000","last_clean":"2024-09-16T05:15:44.771195+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:15:44.771195+0000","last_undegraded":"2024-09-16T05:15:44.771195+0000","last_fullsized":"2024-09-16T05:15:44.771195+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'133","last_scrub_stamp":"2024-09-16T05:14:46.026342+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.026342+0000","objects_scrubbed":2,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:48.493430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045804599999999998,"stat_sum":{"num_bytes":4394329,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":22763,"num_write":13,"num_write_kb":4010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":516,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.713079+0000","last_change":"2024-09-16T05:15:39.713556+0000","last_active":"2024-09-16T05:15:39.713079+0000","last_peered":"2024-09-16T05:15:39.713079+0000","last_clean":"2024-09-16T05:15:39.713079+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:15:39.713079+0000","last_undegraded":"2024-09-16T05:15:39.713079+0000","last_fullsized":"2024-09-16T05:15:39.713079+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:06.783241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.783241+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:32.256094+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00067158799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.11","version":"272'115","reported_seq":1098,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.495196+0000","last_change":"2024-09-16T05:15:39.718342+0000","last_active":"2024-09-16T05:15:42.495196+0000","last_peered":"2024-09-16T05:15:42.495196+0000","last_clean":"2024-09-16T05:15:42.495196+0000","last_became_active":"2024-09-16T05:13:25.679267+0000","last_became_peered":"2024-09-16T05:13:25.679267+0000","last_unstale":"2024-09-16T05:15:42.495196+0000","last_undegraded":"2024-09-16T05:15:42.495196+0000","last_fullsized":"2024-09-16T05:15:42.495196+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:14:31.050468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.050468+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:49.522951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070025800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.10","version":"182'16","reported_seq":1020,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.713788+0000","last_change":"2024-09-16T05:15:42.710936+0000","last_active":"2024-09-16T05:15:43.713788+0000","last_peered":"2024-09-16T05:15:43.713788+0000","last_clean":"2024-09-16T05:15:43.713788+0000","last_became_active":"2024-09-16T05:13:25.680445+0000","last_became_peered":"2024-09-16T05:13:25.680445+0000","last_unstale":"2024-09-16T05:15:43.713788+0000","last_undegraded":"2024-09-16T05:15:43.713788+0000","last_fullsized":"2024-09-16T05:15:43.713788+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:07.531934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.531934+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:26.084528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050319000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.f","version":"360'281","reported_seq":1684,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.713871+0000","last_change":"2024-09-16T05:15:42.712376+0000","last_active":"2024-09-16T05:15:43.713871+0000","last_peered":"2024-09-16T05:15:43.713871+0000","last_clean":"2024-09-16T05:15:43.713871+0000","last_became_active":"2024-09-16T05:13:25.680481+0000","last_became_peered":"2024-09-16T05:13:25.680481+0000","last_unstale":"2024-09-16T05:15:43.713871+0000","last_undegraded":"2024-09-16T05:15:43.713871+0000","last_fullsized":"2024-09-16T05:15:43.713871+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:50.580296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.580296+0000","objects_scrubbed":3,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:53.365851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00166446,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.e","version":"272'232","reported_seq":1529,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.494387+0000","last_change":"2024-09-16T05:15:39.712584+0000","last_active":"2024-09-16T05:15:42.494387+0000","last_peered":"2024-09-16T05:15:42.494387+0000","last_clean":"2024-09-16T05:15:42.494387+0000","last_became_active":"2024-09-16T05:14:40.261276+0000","last_became_peered":"2024-09-16T05:14:40.261276+0000","last_unstale":"2024-09-16T05:15:42.494387+0000","last_undegraded":"2024-09-16T05:15:42.494387+0000","last_fullsized":"2024-09-16T05:15:42.494387+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:14:47.777342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.777342+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.972138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076221100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.d","version":"380'455","reported_seq":1854,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.820268+0000","last_change":"2024-09-16T05:15:43.171905+0000","last_active":"2024-09-16T05:15:43.820268+0000","last_peered":"2024-09-16T05:15:43.820268+0000","last_clean":"2024-09-16T05:15:43.820268+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:15:43.820268+0000","last_undegraded":"2024-09-16T05:15:43.820268+0000","last_fullsized":"2024-09-16T05:15:43.820268+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'379","last_scrub_stamp":"2024-09-16T05:14:44.180613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.180613+0000","objects_scrubbed":6,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:05.576255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029611479999999998,"stat_sum":{"num_bytes":9462506,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":48192,"num_write":30,"num_write_kb":9875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.c","version":"380'255","reported_seq":1637,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.495627+0000","last_change":"2024-09-16T05:15:39.718318+0000","last_active":"2024-09-16T05:15:42.495627+0000","last_peered":"2024-09-16T05:15:42.495627+0000","last_clean":"2024-09-16T05:15:42.495627+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:15:42.495627+0000","last_undegraded":"2024-09-16T05:15:42.495627+0000","last_fullsized":"2024-09-16T05:15:42.495627+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'213","last_scrub_stamp":"2024-09-16T05:14:26.945487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.945487+0000","objects_scrubbed":4,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00065090199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":33266,"num_write":11,"num_write_kb":5499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.8","version":"380'209","reported_seq":1591,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.382998+0000","last_change":"2024-09-16T05:15:39.717037+0000","last_active":"2024-09-16T05:15:41.382998+0000","last_peered":"2024-09-16T05:15:41.382998+0000","last_clean":"2024-09-16T05:15:41.382998+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:15:41.382998+0000","last_undegraded":"2024-09-16T05:15:41.382998+0000","last_fullsized":"2024-09-16T05:15:41.382998+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'153","last_scrub_stamp":"2024-09-16T05:14:26.593177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.593177+0000","objects_scrubbed":3,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001692828,"stat_sum":{"num_bytes":2790028,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.3","version":"378'98","reported_seq":1476,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.895531+0000","last_change":"2024-09-16T05:15:39.895685+0000","last_active":"2024-09-16T05:15:39.895531+0000","last_peered":"2024-09-16T05:15:39.895531+0000","last_clean":"2024-09-16T05:15:39.895531+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:15:39.895531+0000","last_undegraded":"2024-09-16T05:15:39.895531+0000","last_fullsized":"2024-09-16T05:15:39.895531+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'68","last_scrub_stamp":"2024-09-16T05:14:44.975873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.975873+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:07.818034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001168145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"1.0","version":"35'139","reported_seq":997,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383589+0000","last_change":"2024-09-16T05:14:53.849661+0000","last_active":"2024-09-16T05:15:41.383589+0000","last_peered":"2024-09-16T05:15:41.383589+0000","last_clean":"2024-09-16T05:15:41.383589+0000","last_became_active":"2024-09-16T05:10:33.941952+0000","last_became_peered":"2024-09-16T05:10:33.941952+0000","last_unstale":"2024-09-16T05:15:41.383589+0000","last_undegraded":"2024-09-16T05:15:41.383589+0000","last_fullsized":"2024-09-16T05:15:41.383589+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:53.849576+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.849576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:02.708781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1392,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.895020+0000","last_change":"2024-09-16T05:15:39.895228+0000","last_active":"2024-09-16T05:15:39.895020+0000","last_peered":"2024-09-16T05:15:39.895020+0000","last_clean":"2024-09-16T05:15:39.895020+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:15:39.895020+0000","last_undegraded":"2024-09-16T05:15:39.895020+0000","last_fullsized":"2024-09-16T05:15:39.895020+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:14:25.970665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.970665+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084719000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"1.1","version":"35'139","reported_seq":637,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.712892+0000","last_change":"2024-09-16T05:15:18.812427+0000","last_active":"2024-09-16T05:15:39.712892+0000","last_peered":"2024-09-16T05:15:39.712892+0000","last_clean":"2024-09-16T05:15:39.712892+0000","last_became_active":"2024-09-16T05:13:34.762556+0000","last_became_peered":"2024-09-16T05:13:34.762556+0000","last_unstale":"2024-09-16T05:15:39.712892+0000","last_undegraded":"2024-09-16T05:15:39.712892+0000","last_fullsized":"2024-09-16T05:15:39.712892+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:18.812352+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:18.812352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:21.232575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":993,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383037+0000","last_change":"2024-09-16T05:14:07.583425+0000","last_active":"2024-09-16T05:15:41.383037+0000","last_peered":"2024-09-16T05:15:41.383037+0000","last_clean":"2024-09-16T05:15:41.383037+0000","last_became_active":"2024-09-16T05:10:35.953482+0000","last_became_peered":"2024-09-16T05:10:35.953482+0000","last_unstale":"2024-09-16T05:15:41.383037+0000","last_undegraded":"2024-09-16T05:15:41.383037+0000","last_fullsized":"2024-09-16T05:15:41.383037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:07.583346+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:07.583346+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1349,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.494417+0000","last_change":"2024-09-16T05:15:39.712439+0000","last_active":"2024-09-16T05:15:42.494417+0000","last_peered":"2024-09-16T05:15:42.494417+0000","last_clean":"2024-09-16T05:15:42.494417+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:15:42.494417+0000","last_undegraded":"2024-09-16T05:15:42.494417+0000","last_fullsized":"2024-09-16T05:15:42.494417+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:07.898675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.898675+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.911870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064574200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1009,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.767739+0000","last_change":"2024-09-16T05:15:19.711525+0000","last_active":"2024-09-16T05:15:42.767739+0000","last_peered":"2024-09-16T05:15:42.767739+0000","last_clean":"2024-09-16T05:15:42.767739+0000","last_became_active":"2024-09-16T05:15:19.711201+0000","last_became_peered":"2024-09-16T05:15:19.711201+0000","last_unstale":"2024-09-16T05:15:42.767739+0000","last_undegraded":"2024-09-16T05:15:42.767739+0000","last_fullsized":"2024-09-16T05:15:42.767739+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:45.298203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.298203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:07.821797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1421,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383394+0000","last_change":"2024-09-16T05:15:39.717307+0000","last_active":"2024-09-16T05:15:41.383394+0000","last_peered":"2024-09-16T05:15:41.383394+0000","last_clean":"2024-09-16T05:15:41.383394+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:15:41.383394+0000","last_undegraded":"2024-09-16T05:15:41.383394+0000","last_fullsized":"2024-09-16T05:15:41.383394+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:06.755407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.755407+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:10.249610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001762883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":658,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.820018+0000","last_change":"2024-09-16T05:15:43.170680+0000","last_active":"2024-09-16T05:15:43.820018+0000","last_peered":"2024-09-16T05:15:43.820018+0000","last_clean":"2024-09-16T05:15:43.820018+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:15:43.820018+0000","last_undegraded":"2024-09-16T05:15:43.820018+0000","last_fullsized":"2024-09-16T05:15:43.820018+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:31.009189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.009189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:37.421897+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002124094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":962,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.495474+0000","last_change":"2024-09-16T05:15:10.945891+0000","last_active":"2024-09-16T05:15:42.495474+0000","last_peered":"2024-09-16T05:15:42.495474+0000","last_clean":"2024-09-16T05:15:42.495474+0000","last_became_active":"2024-09-16T05:10:35.953730+0000","last_became_peered":"2024-09-16T05:10:35.953730+0000","last_unstale":"2024-09-16T05:15:42.495474+0000","last_undegraded":"2024-09-16T05:15:42.495474+0000","last_fullsized":"2024-09-16T05:15:42.495474+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:10.945793+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.945793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:24.451282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1417,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.713028+0000","last_change":"2024-09-16T05:15:43.043543+0000","last_active":"2024-09-16T05:15:44.713028+0000","last_peered":"2024-09-16T05:15:44.713028+0000","last_clean":"2024-09-16T05:15:44.713028+0000","last_became_active":"2024-09-16T05:14:25.821885+0000","last_became_peered":"2024-09-16T05:14:25.821885+0000","last_unstale":"2024-09-16T05:15:44.713028+0000","last_undegraded":"2024-09-16T05:15:44.713028+0000","last_fullsized":"2024-09-16T05:15:44.713028+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:15:35.928579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:35.928579+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:01.352424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091147000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":742,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.713144+0000","last_change":"2024-09-16T05:14:45.491959+0000","last_active":"2024-09-16T05:15:39.713144+0000","last_peered":"2024-09-16T05:15:39.713144+0000","last_clean":"2024-09-16T05:15:39.713144+0000","last_became_active":"2024-09-16T05:14:39.908958+0000","last_became_peered":"2024-09-16T05:14:39.908958+0000","last_unstale":"2024-09-16T05:15:39.713144+0000","last_undegraded":"2024-09-16T05:15:39.713144+0000","last_fullsized":"2024-09-16T05:15:39.713144+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:45.491884+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.491884+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.370020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"380'476","reported_seq":1861,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.820116+0000","last_change":"2024-09-16T05:15:43.172143+0000","last_active":"2024-09-16T05:15:43.820116+0000","last_peered":"2024-09-16T05:15:43.820116+0000","last_clean":"2024-09-16T05:15:43.820116+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:15:43.820116+0000","last_undegraded":"2024-09-16T05:15:43.820116+0000","last_fullsized":"2024-09-16T05:15:43.820116+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:14:26.230040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.230040+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037472930000000001,"stat_sum":{"num_bytes":3176556,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":8850,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":638,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.713117+0000","last_change":"2024-09-16T05:15:11.724435+0000","last_active":"2024-09-16T05:15:39.713117+0000","last_peered":"2024-09-16T05:15:39.713117+0000","last_clean":"2024-09-16T05:15:39.713117+0000","last_became_active":"2024-09-16T05:13:34.764187+0000","last_became_peered":"2024-09-16T05:13:34.764187+0000","last_unstale":"2024-09-16T05:15:39.713117+0000","last_undegraded":"2024-09-16T05:15:39.713117+0000","last_fullsized":"2024-09-16T05:15:39.713117+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:11.724301+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:11.724301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:40.115572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1491,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.712826+0000","last_change":"2024-09-16T05:15:43.043063+0000","last_active":"2024-09-16T05:15:44.712826+0000","last_peered":"2024-09-16T05:15:44.712826+0000","last_clean":"2024-09-16T05:15:44.712826+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:15:44.712826+0000","last_undegraded":"2024-09-16T05:15:44.712826+0000","last_fullsized":"2024-09-16T05:15:44.712826+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:28.350521+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.350521+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:50.085844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077829500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":822,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.713297+0000","last_change":"2024-09-16T05:14:28.552017+0000","last_active":"2024-09-16T05:15:39.713297+0000","last_peered":"2024-09-16T05:15:39.713297+0000","last_clean":"2024-09-16T05:15:39.713297+0000","last_became_active":"2024-09-16T05:13:34.764109+0000","last_became_peered":"2024-09-16T05:13:34.764109+0000","last_unstale":"2024-09-16T05:15:39.713297+0000","last_undegraded":"2024-09-16T05:15:39.713297+0000","last_fullsized":"2024-09-16T05:15:39.713297+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:28.551941+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.551941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:45.247355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":1587,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.713973+0000","last_change":"2024-09-16T05:15:42.711242+0000","last_active":"2024-09-16T05:15:43.713973+0000","last_peered":"2024-09-16T05:15:43.713973+0000","last_clean":"2024-09-16T05:15:43.713973+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:15:43.713973+0000","last_undegraded":"2024-09-16T05:15:43.713973+0000","last_fullsized":"2024-09-16T05:15:43.713973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:15:41.633260+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:11.199611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079588900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":1002,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.894300+0000","last_change":"2024-09-16T05:15:15.084154+0000","last_active":"2024-09-16T05:15:39.894300+0000","last_peered":"2024-09-16T05:15:39.894300+0000","last_clean":"2024-09-16T05:15:39.894300+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:15:39.894300+0000","last_undegraded":"2024-09-16T05:15:39.894300+0000","last_fullsized":"2024-09-16T05:15:39.894300+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:15.084022+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.084022+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:28.500567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"380'461","reported_seq":1857,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.495421+0000","last_change":"2024-09-16T05:15:39.719459+0000","last_active":"2024-09-16T05:15:42.495421+0000","last_peered":"2024-09-16T05:15:42.495421+0000","last_clean":"2024-09-16T05:15:42.495421+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:15:42.495421+0000","last_undegraded":"2024-09-16T05:15:42.495421+0000","last_fullsized":"2024-09-16T05:15:42.495421+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'419","last_scrub_stamp":"2024-09-16T05:14:25.924121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.924121+0000","objects_scrubbed":4,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012494260000000001,"stat_sum":{"num_bytes":5126867,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":16965,"num_write":5,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":993,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.382928+0000","last_change":"2024-09-16T05:14:11.602321+0000","last_active":"2024-09-16T05:15:41.382928+0000","last_peered":"2024-09-16T05:15:41.382928+0000","last_clean":"2024-09-16T05:15:41.382928+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:15:41.382928+0000","last_undegraded":"2024-09-16T05:15:41.382928+0000","last_fullsized":"2024-09-16T05:15:41.382928+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:11.602237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:11.602237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1636,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.712640+0000","last_change":"2024-09-16T05:15:43.042647+0000","last_active":"2024-09-16T05:15:44.712640+0000","last_peered":"2024-09-16T05:15:44.712640+0000","last_clean":"2024-09-16T05:15:44.712640+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:15:44.712640+0000","last_undegraded":"2024-09-16T05:15:44.712640+0000","last_fullsized":"2024-09-16T05:15:44.712640+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:15:32.922163+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:32.922163+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:46.284081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054353099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1010,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.820036+0000","last_change":"2024-09-16T05:15:14.182510+0000","last_active":"2024-09-16T05:15:43.820036+0000","last_peered":"2024-09-16T05:15:43.820036+0000","last_clean":"2024-09-16T05:15:43.820036+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:15:43.820036+0000","last_undegraded":"2024-09-16T05:15:43.820036+0000","last_fullsized":"2024-09-16T05:15:43.820036+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:14.182407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:14.182407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.292535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":516,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.713665+0000","last_change":"2024-09-16T05:15:39.713963+0000","last_active":"2024-09-16T05:15:39.713665+0000","last_peered":"2024-09-16T05:15:39.713665+0000","last_clean":"2024-09-16T05:15:39.713665+0000","last_became_active":"2024-09-16T05:13:31.747644+0000","last_became_peered":"2024-09-16T05:13:31.747644+0000","last_unstale":"2024-09-16T05:15:39.713665+0000","last_undegraded":"2024-09-16T05:15:39.713665+0000","last_fullsized":"2024-09-16T05:15:39.713665+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:14:49.576522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.576522+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:16.582498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094780899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.13","version":"181'15","reported_seq":1028,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.820021+0000","last_change":"2024-09-16T05:15:43.171435+0000","last_active":"2024-09-16T05:15:43.820021+0000","last_peered":"2024-09-16T05:15:43.820021+0000","last_clean":"2024-09-16T05:15:43.820021+0000","last_became_active":"2024-09-16T05:13:25.681297+0000","last_became_peered":"2024-09-16T05:13:25.681297+0000","last_unstale":"2024-09-16T05:15:43.820021+0000","last_undegraded":"2024-09-16T05:15:43.820021+0000","last_fullsized":"2024-09-16T05:15:43.820021+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:49.160833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.160833+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:53.123139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00267162,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.14","version":"383'267","reported_seq":1265,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.819960+0000","last_change":"2024-09-16T05:15:43.171430+0000","last_active":"2024-09-16T05:15:43.819960+0000","last_peered":"2024-09-16T05:15:43.819960+0000","last_clean":"2024-09-16T05:15:43.819960+0000","last_became_active":"2024-09-16T05:13:25.684888+0000","last_became_peered":"2024-09-16T05:13:25.684888+0000","last_unstale":"2024-09-16T05:15:43.819960+0000","last_undegraded":"2024-09-16T05:15:43.819960+0000","last_fullsized":"2024-09-16T05:15:43.819960+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"342'239","last_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.110587+0000","objects_scrubbed":4,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.311294+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0026813850000000001,"stat_sum":{"num_bytes":3142942,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":14332,"num_write":7,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.15","version":"380'471","reported_seq":1878,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.820263+0000","last_change":"2024-09-16T05:15:43.171893+0000","last_active":"2024-09-16T05:15:43.820263+0000","last_peered":"2024-09-16T05:15:43.820263+0000","last_clean":"2024-09-16T05:15:43.820263+0000","last_became_active":"2024-09-16T05:13:25.680026+0000","last_became_peered":"2024-09-16T05:13:25.680026+0000","last_unstale":"2024-09-16T05:15:43.820263+0000","last_undegraded":"2024-09-16T05:15:43.820263+0000","last_fullsized":"2024-09-16T05:15:43.820263+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"357'436","last_scrub_stamp":"2024-09-16T05:15:07.101221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.101221+0000","objects_scrubbed":5,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.360941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030843379999999998,"stat_sum":{"num_bytes":6735275,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":52138,"num_write":30,"num_write_kb":12050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.16","version":"390'120","reported_seq":1141,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.895793+0000","last_change":"2024-09-16T05:15:39.895925+0000","last_active":"2024-09-16T05:15:39.895793+0000","last_peered":"2024-09-16T05:15:39.895793+0000","last_clean":"2024-09-16T05:15:39.895793+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:15:39.895793+0000","last_undegraded":"2024-09-16T05:15:39.895793+0000","last_fullsized":"2024-09-16T05:15:39.895793+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"304'74","last_scrub_stamp":"2024-09-16T05:14:29.883455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:29.883455+0000","objects_scrubbed":1,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:57.203389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001570873,"stat_sum":{"num_bytes":3142942,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":12296,"num_write":5,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.17","version":"0'0","reported_seq":1192,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.495709+0000","last_change":"2024-09-16T05:15:39.718017+0000","last_active":"2024-09-16T05:15:42.495709+0000","last_peered":"2024-09-16T05:15:42.495709+0000","last_clean":"2024-09-16T05:15:42.495709+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:15:42.495709+0000","last_undegraded":"2024-09-16T05:15:42.495709+0000","last_fullsized":"2024-09-16T05:15:42.495709+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:43.982972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:43.982972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:50.311553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000530607,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1421,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.818883+0000","last_change":"2024-09-16T05:15:43.168589+0000","last_active":"2024-09-16T05:15:43.818883+0000","last_peered":"2024-09-16T05:15:43.818883+0000","last_clean":"2024-09-16T05:15:43.818883+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:15:43.818883+0000","last_undegraded":"2024-09-16T05:15:43.818883+0000","last_fullsized":"2024-09-16T05:15:43.818883+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:17.188273+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:17.188273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:25.050121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075274399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":1019,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.768392+0000","last_change":"2024-09-16T05:15:42.768755+0000","last_active":"2024-09-16T05:15:42.768392+0000","last_peered":"2024-09-16T05:15:42.768392+0000","last_clean":"2024-09-16T05:15:42.768392+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:15:42.768392+0000","last_undegraded":"2024-09-16T05:15:42.768392+0000","last_fullsized":"2024-09-16T05:15:42.768392+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:06.893298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.893298+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:21.174814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000865641,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.1a","version":"393'618","reported_seq":1989,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.714104+0000","last_change":"2024-09-16T05:15:42.713354+0000","last_active":"2024-09-16T05:15:43.714104+0000","last_peered":"2024-09-16T05:15:43.714104+0000","last_clean":"2024-09-16T05:15:43.714104+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:15:43.714104+0000","last_undegraded":"2024-09-16T05:15:43.714104+0000","last_fullsized":"2024-09-16T05:15:43.714104+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'527","last_scrub_stamp":"2024-09-16T05:15:10.511970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.511970+0000","objects_scrubbed":9,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:22.778373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0028499739999999999,"stat_sum":{"num_bytes":12959641,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":57,"num_read_kb":52510,"num_write":40,"num_write_kb":18977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.1b","version":"382'332","reported_seq":1751,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.495340+0000","last_change":"2024-09-16T05:15:39.719244+0000","last_active":"2024-09-16T05:15:42.495340+0000","last_peered":"2024-09-16T05:15:42.495340+0000","last_clean":"2024-09-16T05:15:42.495340+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:15:42.495340+0000","last_undegraded":"2024-09-16T05:15:42.495340+0000","last_fullsized":"2024-09-16T05:15:42.495340+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"347'319","last_scrub_stamp":"2024-09-16T05:15:08.957118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.957118+0000","objects_scrubbed":3,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:32.647702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072802900000000005,"stat_sum":{"num_bytes":5298036,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":29313,"num_write":9,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.1c","version":"382'373","reported_seq":1818,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.713558+0000","last_change":"2024-09-16T05:15:39.713833+0000","last_active":"2024-09-16T05:15:39.713558+0000","last_peered":"2024-09-16T05:15:39.713558+0000","last_clean":"2024-09-16T05:15:39.713558+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:15:39.713558+0000","last_undegraded":"2024-09-16T05:15:39.713558+0000","last_fullsized":"2024-09-16T05:15:39.713558+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'364","last_scrub_stamp":"2024-09-16T05:15:08.759106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.759106+0000","objects_scrubbed":2,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:34.642727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085855899999999995,"stat_sum":{"num_bytes":3327852,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.1d","version":"390'630","reported_seq":2098,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.715348+0000","last_change":"2024-09-16T05:15:39.715505+0000","last_active":"2024-09-16T05:15:39.715348+0000","last_peered":"2024-09-16T05:15:39.715348+0000","last_clean":"2024-09-16T05:15:39.715348+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:15:39.715348+0000","last_undegraded":"2024-09-16T05:15:39.715348+0000","last_fullsized":"2024-09-16T05:15:39.715348+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'584","last_scrub_stamp":"2024-09-16T05:15:16.833446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:16.833446+0000","objects_scrubbed":8,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:42.211138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023589119999999999,"stat_sum":{"num_bytes":12407718,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":46,"num_read_kb":41644,"num_write":31,"num_write_kb":17614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.1e","version":"378'347","reported_seq":1766,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.768447+0000","last_change":"2024-09-16T05:15:42.768727+0000","last_active":"2024-09-16T05:15:42.768447+0000","last_peered":"2024-09-16T05:15:42.768447+0000","last_clean":"2024-09-16T05:15:42.768447+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:15:42.768447+0000","last_undegraded":"2024-09-16T05:15:42.768447+0000","last_fullsized":"2024-09-16T05:15:42.768447+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"362'335","last_scrub_stamp":"2024-09-16T05:15:12.926534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:12.926534+0000","objects_scrubbed":3,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:16.981024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077580100000000001,"stat_sum":{"num_bytes":2487502,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":26093,"num_write":5,"num_write_kb":3828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":1637,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-16T05:15:44.771195+0000","last_change":"2024-09-16T05:15:42.710137+0000","last_active":"2024-09-16T05:15:44.771195+0000","last_peered":"2024-09-16T05:15:44.771195+0000","last_clean":"2024-09-16T05:15:44.771195+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:15:44.771195+0000","last_undegraded":"2024-09-16T05:15:44.771195+0000","last_fullsized":"2024-09-16T05:15:44.771195+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:14:52.883562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.883562+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:58.513312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000344526,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.20","version":"383'134","reported_seq":1546,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383120+0000","last_change":"2024-09-16T05:15:39.716177+0000","last_active":"2024-09-16T05:15:41.383120+0000","last_peered":"2024-09-16T05:15:41.383120+0000","last_clean":"2024-09-16T05:15:41.383120+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:15:41.383120+0000","last_undegraded":"2024-09-16T05:15:41.383120+0000","last_fullsized":"2024-09-16T05:15:41.383120+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'128","last_scrub_stamp":"2024-09-16T05:15:31.623695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.623695+0000","objects_scrubbed":2,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:51.120071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00091224999999999997,"stat_sum":{"num_bytes":4238337,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":79406,"num_write":16,"num_write_kb":6808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":1348,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.494591+0000","last_change":"2024-09-16T05:15:39.713110+0000","last_active":"2024-09-16T05:15:42.494591+0000","last_peered":"2024-09-16T05:15:42.494591+0000","last_clean":"2024-09-16T05:15:42.494591+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:15:42.494591+0000","last_undegraded":"2024-09-16T05:15:42.494591+0000","last_fullsized":"2024-09-16T05:15:42.494591+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.848788+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:55.563613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001107979,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":525,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-16T05:15:39.714439+0000","last_change":"2024-09-16T05:15:39.714632+0000","last_active":"2024-09-16T05:15:39.714439+0000","last_peered":"2024-09-16T05:15:39.714439+0000","last_clean":"2024-09-16T05:15:39.714439+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:15:39.714439+0000","last_undegraded":"2024-09-16T05:15:39.714439+0000","last_fullsized":"2024-09-16T05:15:39.714439+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:31.582158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.582158+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:46.770550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011460859999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7e","length":"3"}]},{"pgid":"2.23","version":"184'17","reported_seq":886,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.768088+0000","last_change":"2024-09-16T05:15:42.768356+0000","last_active":"2024-09-16T05:15:42.768088+0000","last_peered":"2024-09-16T05:15:42.768088+0000","last_clean":"2024-09-16T05:15:42.768088+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:15:42.768088+0000","last_undegraded":"2024-09-16T05:15:42.768088+0000","last_fullsized":"2024-09-16T05:15:42.768088+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:14:46.760890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.760890+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:52.925998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050101400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":707,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.819780+0000","last_change":"2024-09-16T05:15:43.170332+0000","last_active":"2024-09-16T05:15:43.819780+0000","last_peered":"2024-09-16T05:15:43.819780+0000","last_clean":"2024-09-16T05:15:43.819780+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:15:43.819780+0000","last_undegraded":"2024-09-16T05:15:43.819780+0000","last_fullsized":"2024-09-16T05:15:43.819780+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:31.611624+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:31.611624+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.611624+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:55.648239+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0.0021485380000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.25","version":"393'630","reported_seq":1971,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.714134+0000","last_change":"2024-09-16T05:15:42.715226+0000","last_active":"2024-09-16T05:15:43.714134+0000","last_peered":"2024-09-16T05:15:43.714134+0000","last_clean":"2024-09-16T05:15:43.714134+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:15:43.714134+0000","last_undegraded":"2024-09-16T05:15:43.714134+0000","last_fullsized":"2024-09-16T05:15:43.714134+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'624","last_scrub_stamp":"2024-09-16T05:15:35.625076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:35.625076+0000","objects_scrubbed":6,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:46.635889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0042941230000000004,"stat_sum":{"num_bytes":13563348,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":44,"num_read_kb":57574,"num_write":26,"num_write_kb":10249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":685,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383579+0000","last_change":"2024-09-16T05:15:39.718734+0000","last_active":"2024-09-16T05:15:41.383579+0000","last_peered":"2024-09-16T05:15:41.383579+0000","last_clean":"2024-09-16T05:15:41.383579+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:15:41.383579+0000","last_undegraded":"2024-09-16T05:15:41.383579+0000","last_fullsized":"2024-09-16T05:15:41.383579+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:27.631576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.631576+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030488619999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":697,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383903+0000","last_change":"2024-09-16T05:15:39.718542+0000","last_active":"2024-09-16T05:15:41.383903+0000","last_peered":"2024-09-16T05:15:41.383903+0000","last_clean":"2024-09-16T05:15:41.383903+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:15:41.383903+0000","last_undegraded":"2024-09-16T05:15:41.383903+0000","last_fullsized":"2024-09-16T05:15:41.383903+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:30.713398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.713398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002864255,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1487,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.768016+0000","last_change":"2024-09-16T05:15:42.768327+0000","last_active":"2024-09-16T05:15:42.768016+0000","last_peered":"2024-09-16T05:15:42.768016+0000","last_clean":"2024-09-16T05:15:42.768016+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:15:42.768016+0000","last_undegraded":"2024-09-16T05:15:42.768016+0000","last_fullsized":"2024-09-16T05:15:42.768016+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:14:45.747155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.747155+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:58.487448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051588199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.29","version":"380'536","reported_seq":1961,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.494900+0000","last_change":"2024-09-16T05:15:41.155651+0000","last_active":"2024-09-16T05:15:42.494900+0000","last_peered":"2024-09-16T05:15:42.494900+0000","last_clean":"2024-09-16T05:15:42.494900+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:15:42.494900+0000","last_undegraded":"2024-09-16T05:15:42.494900+0000","last_fullsized":"2024-09-16T05:15:42.494900+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'536","last_scrub_stamp":"2024-09-16T05:15:41.155545+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:41.155545+0000","objects_scrubbed":6,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:00.964950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00054687400000000004,"stat_sum":{"num_bytes":14413657,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":25964,"num_write":68,"num_write_kb":21290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1687,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383108+0000","last_change":"2024-09-16T05:15:39.716186+0000","last_active":"2024-09-16T05:15:41.383108+0000","last_peered":"2024-09-16T05:15:41.383108+0000","last_clean":"2024-09-16T05:15:41.383108+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:15:41.383108+0000","last_undegraded":"2024-09-16T05:15:41.383108+0000","last_fullsized":"2024-09-16T05:15:41.383108+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:14:33.725061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:33.725061+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.796596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000889645,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.2b","version":"390'208","reported_seq":1513,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-09-16T05:15:42.494372+0000","last_change":"2024-09-16T05:15:39.714535+0000","last_active":"2024-09-16T05:15:42.494372+0000","last_peered":"2024-09-16T05:15:42.494372+0000","last_clean":"2024-09-16T05:15:42.494372+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:15:42.494372+0000","last_undegraded":"2024-09-16T05:15:42.494372+0000","last_fullsized":"2024-09-16T05:15:42.494372+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"330'198","last_scrub_stamp":"2024-09-16T05:14:46.815272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.815272+0000","objects_scrubbed":2,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:51.922644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024293420000000001,"stat_sum":{"num_bytes":4533234,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":13,"num_write_kb":8806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":1534,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.713938+0000","last_change":"2024-09-16T05:15:42.711948+0000","last_active":"2024-09-16T05:15:43.713938+0000","last_peered":"2024-09-16T05:15:43.713938+0000","last_clean":"2024-09-16T05:15:43.713938+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:15:43.713938+0000","last_undegraded":"2024-09-16T05:15:43.713938+0000","last_fullsized":"2024-09-16T05:15:43.713938+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:15:39.640149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.640149+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:45.100517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012865929999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]},{"pgid":"2.2d","version":"377'326","reported_seq":1687,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383342+0000","last_change":"2024-09-16T05:15:39.716783+0000","last_active":"2024-09-16T05:15:41.383342+0000","last_peered":"2024-09-16T05:15:41.383342+0000","last_clean":"2024-09-16T05:15:41.383342+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:15:41.383342+0000","last_undegraded":"2024-09-16T05:15:41.383342+0000","last_fullsized":"2024-09-16T05:15:41.383342+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'307","last_scrub_stamp":"2024-09-16T05:14:30.116429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.116429+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:42.859864+0000","scrub_duration":444,"objects_trimmed":0,"snaptrim_duration":0.001442189,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":4589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.2e","version":"382'392","reported_seq":1745,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-09-16T05:15:41.383580+0000","last_change":"2024-09-16T05:15:39.717615+0000","last_active":"2024-09-16T05:15:41.383580+0000","last_peered":"2024-09-16T05:15:41.383580+0000","last_clean":"2024-09-16T05:15:41.383580+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:15:41.383580+0000","last_undegraded":"2024-09-16T05:15:41.383580+0000","last_fullsized":"2024-09-16T05:15:41.383580+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"337'278","last_scrub_stamp":"2024-09-16T05:14:47.811642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.811642+0000","objects_scrubbed":7,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:11.110891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001940799,"stat_sum":{"num_bytes":8393552,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":7127,"num_write":34,"num_write_kb":7817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"80","length":"1"}]},{"pgid":"2.2f","version":"387'306","reported_seq":1745,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-16T05:15:43.819856+0000","last_change":"2024-09-16T05:15:43.170996+0000","last_active":"2024-09-16T05:15:43.819856+0000","last_peered":"2024-09-16T05:15:43.819856+0000","last_clean":"2024-09-16T05:15:43.819856+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:15:43.819856+0000","last_undegraded":"2024-09-16T05:15:43.819856+0000","last_fullsized":"2024-09-16T05:15:43.819856+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:53.110450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.110450+0000","objects_scrubbed":3,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:02.883248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025298550000000001,"stat_sum":{"num_bytes":6521248,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":13580,"num_write":12,"num_write_kb":6438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"80","length":"1"},{"start":"82","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":182188859,"num_objects":82,"num_object_clones":37,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":6,"num_read":707,"num_read_kb":716475,"num_write":502,"num_write_kb":208495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":223264768,"data_stored":274669510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15505,"ondisk_log_size":15505,"up":152,"acting":152,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089423,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67344,"kb_used_data":31996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93677808,"statfs":{"total":95995035648,"available":95926075392,"internally_reserved":0,"allocated":32763904,"data_stored":36308227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54500,"kb_used_data":19152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690652,"statfs":{"total":95995035648,"available":95939227648,"internally_reserved":0,"allocated":19611648,"data_stored":20995841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514588,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75380,"kb_used_data":35864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93669772,"statfs":{"total":95995035648,"available":95917846528,"internally_reserved":0,"allocated":36724736,"data_stored":42137946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841629,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52288,"kb_used_data":16944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93692864,"statfs":{"total":95995035648,"available":95941492736,"internally_reserved":0,"allocated":17350656,"data_stored":22584423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103266,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72568,"kb_used_data":33052,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93672584,"statfs":{"total":95995035648,"available":95920726016,"internally_reserved":0,"allocated":33845248,"data_stored":40321230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397609,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69068,"kb_used_data":29560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93676084,"statfs":{"total":95995035648,"available":95924310016,"internally_reserved":0,"allocated":30269440,"data_stored":37076774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757356,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46548,"kb_used_data":15360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93698604,"statfs":{"total":95995035648,"available":95947370496,"internally_reserved":0,"allocated":15728640,"data_stored":18427505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920882,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53144,"kb_used_data":17788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93692008,"statfs":{"total":95995035648,"available":95940616192,"internally_reserved":0,"allocated":18214912,"data_stored":17388689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117111,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67772,"kb_used_data":32420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93677380,"statfs":{"total":95995035648,"available":95925637120,"internally_reserved":0,"allocated":33198080,"data_stored":34061348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41644,"kb_used_data":13524,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93703508,"statfs":{"total":95995035648,"available":95952392192,"internally_reserved":0,"allocated":13848576,"data_stored":14267625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574978,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54092,"kb_used_data":18740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93691060,"statfs":{"total":95995035648,"available":95939645440,"internally_reserved":0,"allocated":19189760,"data_stored":19266824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738503,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49044,"kb_used_data":9524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93696108,"statfs":{"total":95995035648,"available":95944814592,"internally_reserved":0,"allocated":9752576,"data_stored":8903352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5050368,"data_stored":5889133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14475264,"data_stored":16244192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8904704,"data_stored":11007650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28520448,"data_stored":31064061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13561856,"data_stored":14408252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10870784,"data_stored":15245820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25604096,"data_stored":34087884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":29192192,"data_stored":37340793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12333056,"data_stored":19250937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":32022528,"data_stored":39123727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":14958592,"data_stored":18015404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":27770880,"data_stored":32991657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:15:47.023 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:15:47.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-16T05:15:47.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:15:47.077 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:47.083 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:46 smithi032 ceph-mon[27388]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:47.083 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:46 smithi032 ceph-mon[27388]: osdmap e397: 12 total, 12 up, 12 in 2024-09-16T05:15:47.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:47.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:47.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:46 smithi204 ceph-mon[30933]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:46 smithi204 ceph-mon[30933]: osdmap e397: 12 total, 12 up, 12 in 2024-09-16T05:15:47.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:46 smithi196 ceph-mon[30802]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:47.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:46 smithi196 ceph-mon[30802]: osdmap e397: 12 total, 12 up, 12 in 2024-09-16T05:15:47.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:47.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:47.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:47.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:47.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:47.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:47.421 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:47.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:47.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:47.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:47.540 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:15:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:15:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:15:47.540 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(1.a,RELEASE,e:112,from:9,reservation_nonce:0)", 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 303.52734415100002, 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.42149128000000002, 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.541 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.542 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010445+0000", 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5999999999999999e-07 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010447+0000", 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.227e-06 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.543 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431780+0000", 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.42133351800000002 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431791+0000", 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0773e-05 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431896+0000", 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104955 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431918+0000", 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2096e-05 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431936+0000", 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7351e-05 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.1a,RELEASE,e:216,from:8,reservation_nonce:0)", 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 199.08599543700001, 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065133569000000002, 2024-09-16T05:15:47.546 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2499999999999999e-07 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451795+0000", 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.674e-06 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516800+0000", 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065004988 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516812+0000", 2024-09-16T05:15:47.549 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2265e-05 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516889+0000", 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6662999999999994e-05 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516910+0000", 2024-09-16T05:15:47.550 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0718999999999999e-05 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516927+0000", 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7034999999999999e-05 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1471 2.14 e217/185 2:2ace617f:::smithi20457400-21:head v 217'118, mlcod=208'114)", 2024-09-16T05:15:47.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:28.692382+0000", 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.84540611200001, 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077658060000000001, 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.692382+0000", 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.692382+0000", 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.692383+0000", 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3399999999999999e-07 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.553 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696469+0000", 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040866189999999997 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696505+0000", 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5453000000000001e-05 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696513+0000", 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6329999999999996e-06 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696606+0000", 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3894000000000006e-05 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.696633+0000", 2024-09-16T05:15:47.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6516000000000002e-05 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.700102+0000", 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034693390000000001 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.700148+0000", 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5818000000000001e-05 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.556 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1859 2.29 2.f7e83b69 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 166.85544411999999, 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98895853899999997, 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.557 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1859 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.558 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682346+0000", 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9640000000000002e-06 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682353+0000", 2024-09-16T05:15:47.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2330000000000003e-06 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682369+0000", 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6721000000000001e-05 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682383+0000", 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4197e-05 2024-09-16T05:15:47.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682449+0000", 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5660999999999994e-05 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682458+0000", 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8000000000000004e-06 2024-09-16T05:15:47.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:01.671285+0000", 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98882742000000001 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:01.671303+0000", 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7543000000000001e-05 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.562 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2009 2.29 2:94d33c56:::smithi20457400-8:head [sparse-read 0~2045226] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:19.884195+0000", 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 147.65359375400001, 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065356520000000003, 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.563 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2009 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884195+0000", 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884195+0000", 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884195+0000", 2024-09-16T05:15:47.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9100000000000004e-07 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884199+0000", 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8469999999999998e-06 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884205+0000", 2024-09-16T05:15:47.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2360000000000001e-06 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884210+0000", 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0540000000000002e-06 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884303+0000", 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2717e-05 2024-09-16T05:15:47.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.884308+0000", 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4650000000000002e-06 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890476+0000", 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006167802 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.568 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890576+0000", 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100146 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:19.890730+0000", 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015389399999999999 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.569 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2450 2.1a e310/291 2:5827f14d:::smithi20457400-12:head v 310'441, mlcod=305'437)", 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:01.737305+0000", 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 105.800483099, 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065060500000000002, 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.570 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.737305+0000", 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.737305+0000", 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.737306+0000", 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.34e-07 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.741580+0000", 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042739869999999999 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.741609+0000", 2024-09-16T05:15:47.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9865999999999998e-05 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.741616+0000", 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8059999999999997e-06 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.741666+0000", 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9660999999999998e-05 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.741684+0000", 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7730000000000001e-05 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.743771+0000", 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020875680000000001 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:01.743811+0000", 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0197999999999997e-05 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.6.0:125 2.29 e345/253 ondisk, result = 0)", 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.706089+0000", 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 52.831699843000003, 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0099953500000000001, 2024-09-16T05:15:47.575 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.706089+0000", 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.706089+0000", 2024-09-16T05:15:47.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.706089+0000", 2024-09-16T05:15:47.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2499999999999999e-07 2024-09-16T05:15:47.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.706090+0000", 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2359999999999999e-06 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.715925+0000", 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098348910000000001 2024-09-16T05:15:47.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.715931+0000", 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3169999999999998e-06 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.715976+0000", 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.511e-05 2024-09-16T05:15:47.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.715983+0000", 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7399999999999998e-06 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.716084+0000", 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100831 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.581 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2671 2.29 2:9423b7dc:::smithi20457400-41:head [write 453888~713266,stat] snapc 74=[74,73,72,71,70,6e,6c,6b,6a,69,67,66,64,63,62,60,5a,58] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.711627+0000", 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 52.826161143999997, 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083045270000000008, 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:47.582 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2671 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.711627+0000", 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.711627+0000", 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.711629+0000", 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9690000000000001e-06 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.712598+0000", 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00096909800000000005 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.712614+0000", 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6039999999999999e-05 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.712621+0000", 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8000000000000001e-06 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.712685+0000", 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3844999999999996e-05 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.712788+0000", 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102882 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.712998+0000", 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000210545 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:15:47.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.714829+0000", 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001830857 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719856+0000", 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050266720000000003 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:47.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719898+0000", 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1637999999999997e-05 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719932+0000", 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4180999999999997e-05 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2673 2.29 2:9423b7dc:::smithi20457400-41:head [setxattr _header (58) in=7b,truncate 2453888] snapc 74=[74,73,72,71,70,6e,6c,6b,6a,69,67,66,64,63,62,60,5a,58] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.717815+0000", 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 52.819973152000003, 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.54320599199999997, 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.590 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2673 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717815+0000", 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717815+0000", 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717816+0000", 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8599999999999998e-07 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717819+0000", 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7800000000000001e-06 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717823+0000", 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5750000000000002e-06 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717827+0000", 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.49e-06 2024-09-16T05:15:47.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.718815+0000", 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00098811699999999995 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.718896+0000", 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0698000000000006e-05 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.718996+0000", 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0001004 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719514+0000", 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051843099999999997 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:47.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260903+0000", 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.54138819699999996 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260957+0000", 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4215000000000002e-05 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261021+0000", 2024-09-16T05:15:47.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4503000000000002e-05 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2674 2.29 2:9423b7dc:::smithi20457400-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e345)", 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.717832+0000", 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 52.819955944, 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.543402791, 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.597 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2674 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717832+0000", 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717832+0000", 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717833+0000", 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4400000000000001e-07 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717834+0000", 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.42e-06 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717839+0000", 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7849999999999999e-06 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717846+0000", 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8079999999999999e-06 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719362+0000", 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001516381 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719418+0000", 2024-09-16T05:15:47.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5569000000000003e-05 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261065+0000", 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.541647726 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261145+0000", 2024-09-16T05:15:47.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9709000000000005e-05 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261235+0000", 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0148999999999994e-05 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:2673 2.29 e345/253 ondisk, result = 0)", 2024-09-16T05:15:47.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.719925+0000", 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 52.817863719999998, 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.54110561499999998, 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719925+0000", 2024-09-16T05:15:47.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719925+0000", 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719925+0000", 2024-09-16T05:15:47.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4199999999999995e-07 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719929+0000", 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6500000000000002e-06 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260799+0000", 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.54087024900000003 2024-09-16T05:15:47.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260809+0000", 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0073e-05 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260887+0000", 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7440000000000004e-05 2024-09-16T05:15:47.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260898+0000", 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0889e-05 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261030+0000", 2024-09-16T05:15:47.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013267199999999999 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.6.0:147 2.29 e354/253 ondisk, result = 0)", 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 43.788837411999999, 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92583168000000005, 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.609 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748952+0000", 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2099999999999996e-07 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748956+0000", 2024-09-16T05:15:47.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4839999999999998e-06 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674453+0000", 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92549724300000002 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674462+0000", 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8019999999999998e-06 2024-09-16T05:15:47.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674569+0000", 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107186 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674584+0000", 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.502e-05 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674783+0000", 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019832399999999999 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2818 2.2d e366/252 2:b7234079:::smithi20457400-42:head v 366'316, mlcod=349'313)", 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:15.866149+0000", 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 31.671639325000001, 2024-09-16T05:15:47.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010393319999999999, 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.866149+0000", 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.866149+0000", 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.866150+0000", 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.234e-06 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.868475+0000", 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023247369999999999 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.868537+0000", 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2139999999999998e-05 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.868567+0000", 2024-09-16T05:15:47.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9899e-05 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.868664+0000", 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7010999999999997e-05 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.868694+0000", 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9859e-05 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.876487+0000", 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077927559999999996 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:15.876542+0000", 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5683999999999997e-05 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.619 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2845 2.29 2:94d33c56:::smithi20457400-8:head [copy-from ver 336] snapc 7b=[7b,7a,78,77,76,75,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:17.757541+0000", 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.780247628000001, 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012784904, 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.620 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.621 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.621 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2845 2024-09-16T05:15:47.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.621 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757541+0000", 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757541+0000", 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757543+0000", 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6110000000000001e-06 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757555+0000", 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1097000000000001e-05 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757578+0000", 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3451999999999998e-05 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757588+0000", 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0186e-05 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757655+0000", 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7131000000000005e-05 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757738+0000", 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3065999999999994e-05 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.764864+0000", 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071259540000000003 2024-09-16T05:15:47.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.768432+0000", 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035675300000000002 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.770248+0000", 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001816462 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.770287+0000", 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9029000000000002e-05 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.770326+0000", 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8386000000000001e-05 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.627 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2846 2.29 2:94d33c56:::smithi20457400-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e368)", 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:17.757615+0000", 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 29.780173138999999, 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010347151000000001, 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.628 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2846 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757615+0000", 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.629 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757615+0000", 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757616+0000", 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7700000000000004e-07 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757619+0000", 2024-09-16T05:15:47.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1709999999999998e-06 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757628+0000", 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1120000000000002e-06 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757635+0000", 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8569999999999997e-06 2024-09-16T05:15:47.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757896+0000", 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026131499999999997 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757954+0000", 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7379000000000001e-05 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.767824+0000", 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098697090000000008 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.767891+0000", 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7633999999999997e-05 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.767962+0000", 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1297000000000005e-05 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3026 2.29 2:9423b7dc:::smithi20457400-41:head [read 0~3058940,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e378)", 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:27.745867+0000", 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.791921222999999, 2024-09-16T05:15:47.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065538590000000004, 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3026 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745867+0000", 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745867+0000", 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745870+0000", 2024-09-16T05:15:47.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2970000000000002e-06 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745881+0000", 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.148e-05 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745903+0000", 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2065e-05 2024-09-16T05:15:47.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745912+0000", 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6389999999999998e-06 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745989+0000", 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6854000000000007e-05 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745998+0000", 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3219999999999999e-06 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.747470+0000", 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014720079999999999 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.747543+0000", 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3592999999999998e-05 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.752421+0000", 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048776009999999996 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3027 2.29 2:9423b7dc:::smithi20457400-41:head [sparse-read 0~2066545] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e378)", 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:27.745934+0000", 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.791854847, 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067616999999999998, 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.641 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3027 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745934+0000", 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745934+0000", 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745934+0000", 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6000000000000004e-07 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745937+0000", 2024-09-16T05:15:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9399999999999998e-06 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745943+0000", 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.263e-06 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745948+0000", 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3069999999999996e-06 2024-09-16T05:15:47.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.746058+0000", 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010998 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.746064+0000", 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4219999999999997e-06 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.752494+0000", 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0064297240000000004 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.752563+0000", 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9074999999999998e-05 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.752695+0000", 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013242899999999999 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3028 2.29 2:9423b7dc:::smithi20457400-41:head [sparse-read 0~2066545] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e378)", 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:27.745972+0000", 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.791816688000001, 2024-09-16T05:15:47.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070320560000000001, 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3028 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745972+0000", 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745972+0000", 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745972+0000", 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5199999999999998e-07 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.745974+0000", 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0310000000000001e-06 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.746018+0000", 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3665999999999998e-05 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.746031+0000", 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2904e-05 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.746082+0000", 2024-09-16T05:15:47.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1144999999999999e-05 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.746085+0000", 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.88e-06 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.752741+0000", 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066560810000000003 2024-09-16T05:15:47.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.752845+0000", 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103841 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.753004+0000", 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015915600000000001 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.653 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.9.0:669 2.14 e378/273 2:2ace617f:::smithi20457400-21:6a v 378'261, mlcod=366'257)", 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:27.751309+0000", 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.786479458999999, 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078004779999999996, 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.751309+0000", 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.751309+0000", 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.751309+0000", 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8e-07 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.751313+0000", 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9029999999999997e-06 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.758284+0000", 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069706669999999998 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.758290+0000", 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8560000000000003e-06 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.758347+0000", 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7797999999999998e-05 2024-09-16T05:15:47.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.758372+0000", 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4950999999999999e-05 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.759072+0000", 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00069924899999999999 2024-09-16T05:15:47.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:27.759109+0000", 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7873999999999997e-05 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:3092 2.1a e387/291 2:5ae26c04:::smithi20457400-32:head v 387'606, mlcod=387'603)", 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:36.735989+0000", 2024-09-16T05:15:47.659 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 10.801799695, 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065721979999999996, 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.735989+0000", 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.735989+0000", 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.735990+0000", 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5499999999999997e-07 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.740053+0000", 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040637980000000004 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.740088+0000", 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4822e-05 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.740096+0000", 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0909999999999993e-06 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.740168+0000", 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1946000000000005e-05 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:15:47.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.740185+0000", 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.719e-05 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.742515+0000", 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023291509999999998 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:15:47.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:36.742561+0000", 2024-09-16T05:15:47.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6344999999999998e-05 2024-09-16T05:15:47.665 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.665 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.665 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.665 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:15:47.665 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:15:47.665 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:15:47.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:47.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:47.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:47.709 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:15:47.713 INFO:tasks.rados.rados.2.smithi204.stdout:1241: done (1 left) 2024-09-16T05:15:47.713 INFO:tasks.rados.rados.2.smithi204.stdout:1242: read oid 2 snap -1 2024-09-16T05:15:47.714 INFO:tasks.rados.rados.2.smithi204.stdout:1242: expect (ObjNum 430 snap 124 seq_num 430) 2024-09-16T05:15:47.714 INFO:tasks.rados.rados.2.smithi204.stdout:1243: write oid 50 current snap is 137 2024-09-16T05:15:47.714 INFO:tasks.rados.rados.2.smithi204.stdout:1243: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-16T05:15:47.721 INFO:tasks.rados.rados.2.smithi204.stdout:1243: writing smithi20457400-50 from 695922 to 1405450 tid 1 2024-09-16T05:15:47.728 INFO:tasks.rados.rados.2.smithi204.stdout:1243: writing smithi20457400-50 from 2032616 to 2640096 tid 2 2024-09-16T05:15:47.733 INFO:tasks.rados.rados.2.smithi204.stdout:1243: writing smithi20457400-50 from 3198537 to 3495922 tid 3 2024-09-16T05:15:47.733 INFO:tasks.rados.rados.2.smithi204.stdout:1244: snap_remove snap 132 2024-09-16T05:15:47.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:47.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:47.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:47.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:47.941 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:47.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:48.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:48.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:48.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:47 smithi032 ceph-mon[27388]: pgmap v756: 87 pgs: 87 active+clean; 176 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2024-09-16T05:15:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/550086389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:47 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/550086389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:47 smithi032 ceph-mon[27388]: osdmap e398: 12 total, 12 up, 12 in 2024-09-16T05:15:48.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:47 smithi204 ceph-mon[30933]: pgmap v756: 87 pgs: 87 active+clean; 176 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2024-09-16T05:15:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/550086389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:47 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/550086389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:48.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:47 smithi204 ceph-mon[30933]: osdmap e398: 12 total, 12 up, 12 in 2024-09-16T05:15:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:48.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:48.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:47 smithi196 ceph-mon[30802]: pgmap v756: 87 pgs: 87 active+clean; 176 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2024-09-16T05:15:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/550086389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:47 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/550086389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:47 smithi196 ceph-mon[30802]: osdmap e398: 12 total, 12 up, 12 in 2024-09-16T05:15:48.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:48.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:48.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:48.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:15:48.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:48.444 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:48.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:48.541 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:48.541 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:48.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:48.564 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:48.565 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:48.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:48.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:48.584 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:48.593 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:48.605 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:48.606 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:48.627 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:48.627 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:48.648 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:48.648 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:48.670 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:48.671 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:48.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:48.693 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:48.693 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:48.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:48.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:48.710 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 1 pg_num to 21 2024-09-16T05:15:48.715 INFO:tasks.rados.rados.2.smithi204.stdout:1244: done (3 left) 2024-09-16T05:15:48.715 INFO:tasks.rados.rados.2.smithi204.stdout:1245: rollback oid 3 current snap is 137 2024-09-16T05:15:48.715 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 129 2024-09-16T05:15:48.715 INFO:tasks.rados.rados.2.smithi204.stdout:1240: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:15:48.715 INFO:tasks.rados.rados.2.smithi204.stdout:1240: finishing copy_from to smithi20457400-18 2024-09-16T05:15:48.715 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 301 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:15:48.715 INFO:tasks.rados.rados.2.smithi204.stdout:1242: expect (ObjNum 430 snap 124 seq_num 430) 2024-09-16T05:15:48.716 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:48.717 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:48.746 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:48.746 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:48.769 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:48.769 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:48.791 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:48.791 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:48.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:48.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:48.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:48.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:48.873 INFO:tasks.rados.rados.2.smithi204.stdout:1243: finishing write tid 1 to smithi20457400-50 2024-09-16T05:15:48.873 INFO:tasks.rados.rados.2.smithi204.stdout:1243: finishing write tid 2 to smithi20457400-50 2024-09-16T05:15:48.873 INFO:tasks.rados.rados.2.smithi204.stdout:1243: finishing write tid 3 to smithi20457400-50 2024-09-16T05:15:48.873 INFO:tasks.rados.rados.2.smithi204.stdout:1243: finishing write tid 4 to smithi20457400-50 2024-09-16T05:15:48.873 INFO:tasks.rados.rados.2.smithi204.stdout:1243: finishing write tid 5 to smithi20457400-50 2024-09-16T05:15:48.873 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 329 (ObjNum 446 snap 137 seq_num 446) dirty exists 2024-09-16T05:15:48.874 INFO:tasks.rados.rados.2.smithi204.stdout:1243: left oid 50 (ObjNum 446 snap 137 seq_num 446) 2024-09-16T05:15:48.874 INFO:tasks.rados.rados.2.smithi204.stdout:1240: done (3 left) 2024-09-16T05:15:48.874 INFO:tasks.rados.rados.2.smithi204.stdout:1242: done (2 left) 2024-09-16T05:15:48.874 INFO:tasks.rados.rados.2.smithi204.stdout:1243: done (1 left) 2024-09-16T05:15:48.874 INFO:tasks.rados.rados.2.smithi204.stdout:1246: snap_remove snap 134 2024-09-16T05:15:48.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:48.945 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:48.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:48.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3026828523' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-16T05:15:48.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2664499257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:48.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3026828523' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T05:15:48.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:48 smithi196 ceph-mon[30802]: osdmap e399: 12 total, 12 up, 12 in 2024-09-16T05:15:49.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:49.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:49.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:49.081 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:49.082 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:49.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3026828523' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-16T05:15:49.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2664499257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:49.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3026828523' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T05:15:49.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:48 smithi032 ceph-mon[27388]: osdmap e399: 12 total, 12 up, 12 in 2024-09-16T05:15:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3026828523' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-16T05:15:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2664499257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3026828523' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-16T05:15:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:48 smithi204 ceph-mon[30933]: osdmap e399: 12 total, 12 up, 12 in 2024-09-16T05:15:49.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:49.194 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:49.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:49.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:49.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:49.319 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:49.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:49.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:49.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:49.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:49.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:49.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:49.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:49.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:15:49.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:49.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:49.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:49.818 INFO:tasks.rados.rados.2.smithi204.stdout:1246: done (1 left) 2024-09-16T05:15:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:1247: read oid 17 snap 127 2024-09-16T05:15:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:1247: expect (ObjNum 435 snap 125 seq_num 435) 2024-09-16T05:15:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:1245: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:15:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 470 (ObjNum 33 snap 0 seq_num 4282325888) dirty dne 2024-09-16T05:15:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:1248: read oid 23 snap 122 2024-09-16T05:15:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:1248: expect (ObjNum 328 snap 91 seq_num 328) 2024-09-16T05:15:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:1245: done (2 left) 2024-09-16T05:15:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:1249: snap_create 2024-09-16T05:15:49.833 INFO:tasks.rados.rados.2.smithi204.stdout:1247: expect (ObjNum 435 snap 125 seq_num 435) 2024-09-16T05:15:49.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:49.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:49.928 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:49.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:49.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:49 smithi196 ceph-mon[30802]: pgmap v759: 87 pgs: 87 active+clean; 176 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2024-09-16T05:15:49.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:49.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:50.022 INFO:tasks.rados.rados.2.smithi204.stdout:1248: expect (ObjNum 328 snap 91 seq_num 328) 2024-09-16T05:15:50.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:50.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:50.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:50.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:49 smithi032 ceph-mon[27388]: pgmap v759: 87 pgs: 87 active+clean; 176 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2024-09-16T05:15:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:49 smithi204 ceph-mon[30933]: pgmap v759: 87 pgs: 87 active+clean; 176 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2024-09-16T05:15:50.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:50.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:50.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:50.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:50.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:50.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:50.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:50.487 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:50.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:50.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:50.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:50.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:50.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:50.715 INFO:tasks.rados.rados.2.smithi204.stdout:1247: done (2 left) 2024-09-16T05:15:50.715 INFO:tasks.rados.rados.2.smithi204.stdout:1248: done (1 left) 2024-09-16T05:15:50.716 INFO:tasks.rados.rados.2.smithi204.stdout:1249: done (0 left) 2024-09-16T05:15:50.717 INFO:tasks.rados.rados.2.smithi204.stdout:1250: read oid 29 snap -1 2024-09-16T05:15:50.717 INFO:tasks.rados.rados.2.smithi204.stdout:1250: expect (ObjNum 408 snap 123 seq_num 408) 2024-09-16T05:15:50.717 INFO:tasks.rados.rados.2.smithi204.stdout:1251: snap_remove snap 123 2024-09-16T05:15:50.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:50.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:50.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:50.864 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:50.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:50 smithi032 ceph-mon[27388]: osdmap e400: 12 total, 12 up, 12 in 2024-09-16T05:15:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-16T05:15:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T05:15:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:50 smithi032 ceph-mon[27388]: osdmap e401: 12 total, 12 up, 12 in 2024-09-16T05:15:50.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:50.992 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:50.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:50 smithi196 ceph-mon[30802]: osdmap e400: 12 total, 12 up, 12 in 2024-09-16T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-16T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:50 smithi196 ceph-mon[30802]: osdmap e401: 12 total, 12 up, 12 in 2024-09-16T05:15:51.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:15:51.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:51.118 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:51.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:50 smithi204 ceph-mon[30933]: osdmap e400: 12 total, 12 up, 12 in 2024-09-16T05:15:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-16T05:15:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-16T05:15:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:50 smithi204 ceph-mon[30933]: osdmap e401: 12 total, 12 up, 12 in 2024-09-16T05:15:51.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:51.244 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:51.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:51.319 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:15:51.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:51.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:51.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:51.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:51.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:51.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:51.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:15:51.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:51.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:51.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:51.714 INFO:tasks.rados.rados.2.smithi204.stdout:1251: done (1 left) 2024-09-16T05:15:51.714 INFO:tasks.rados.rados.2.smithi204.stdout:1252: write oid 9 current snap is 138 2024-09-16T05:15:51.715 INFO:tasks.rados.rados.2.smithi204.stdout:1252: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-16T05:15:51.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:51.725 INFO:tasks.rados.rados.2.smithi204.stdout:1252: writing smithi20457400-9 from 712729 to 1497506 tid 1 2024-09-16T05:15:51.730 INFO:tasks.rados.rados.2.smithi204.stdout:1252: writing smithi20457400-9 from 2091098 to 2558589 tid 2 2024-09-16T05:15:51.736 INFO:tasks.rados.rados.2.smithi204.stdout:1252: writing smithi20457400-9 from 3342313 to 3512729 tid 3 2024-09-16T05:15:51.736 INFO:tasks.rados.rados.2.smithi204.stdout:1253: read oid 27 snap 137 2024-09-16T05:15:51.737 INFO:tasks.rados.rados.2.smithi204.stdout:1253: expect (ObjNum 404 snap 122 seq_num 404) 2024-09-16T05:15:51.737 INFO:tasks.rados.rados.2.smithi204.stdout:1254: read oid 36 snap -1 2024-09-16T05:15:51.737 INFO:tasks.rados.rados.2.smithi204.stdout:1254: expect (ObjNum 419 snap 124 seq_num 419) 2024-09-16T05:15:51.737 INFO:tasks.rados.rados.2.smithi204.stdout:1255: read oid 15 snap -1 2024-09-16T05:15:51.737 INFO:tasks.rados.rados.2.smithi204.stdout:1255: expect (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:15:51.737 INFO:tasks.rados.rados.2.smithi204.stdout:1256: rollback oid 35 current snap is 138 2024-09-16T05:15:51.737 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 35 to 136 2024-09-16T05:15:51.737 INFO:tasks.rados.rados.2.smithi204.stdout:1250: expect (ObjNum 408 snap 123 seq_num 408) 2024-09-16T05:15:51.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:51.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:51.782 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:51.797 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:51.797 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:51.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:51.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:51.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1252: finishing write tid 1 to smithi20457400-9 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1252: finishing write tid 2 to smithi20457400-9 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1252: finishing write tid 3 to smithi20457400-9 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1252: finishing write tid 4 to smithi20457400-9 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1252: finishing write tid 5 to smithi20457400-9 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 543 (ObjNum 447 snap 138 seq_num 447) dirty exists 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1252: left oid 9 (ObjNum 447 snap 138 seq_num 447) 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1256: finishing rollback tid 0 to smithi20457400-35 2024-09-16T05:15:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1256: finishing rollback tid 1 to smithi20457400-35 2024-09-16T05:15:51.907 INFO:tasks.rados.rados.2.smithi204.stdout:1250: done (5 left) 2024-09-16T05:15:51.907 INFO:tasks.rados.rados.2.smithi204.stdout:1252: done (4 left) 2024-09-16T05:15:51.907 INFO:tasks.rados.rados.2.smithi204.stdout:1257: write (excl) oid 19 current snap is 138 2024-09-16T05:15:51.907 INFO:tasks.rados.rados.2.smithi204.stdout:1256: finishing rollback tid 2 to smithi20457400-35 2024-09-16T05:15:51.907 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 465 (ObjNum 405 snap 122 seq_num 405) dirty exists 2024-09-16T05:15:51.907 INFO:tasks.rados.rados.2.smithi204.stdout:1254: expect (ObjNum 419 snap 124 seq_num 419) 2024-09-16T05:15:51.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:51.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:51 smithi196 ceph-mon[30802]: pgmap v762: 87 pgs: 87 active+clean; 176 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:15:51.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:51 smithi196 ceph-mon[30802]: osdmap e402: 12 total, 12 up, 12 in 2024-09-16T05:15:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:51.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:52.047 INFO:tasks.rados.rados.2.smithi204.stdout:1253: expect (ObjNum 404 snap 122 seq_num 404) 2024-09-16T05:15:52.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:52.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:52.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:52.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:51 smithi032 ceph-mon[27388]: pgmap v762: 87 pgs: 87 active+clean; 176 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:15:52.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:51 smithi032 ceph-mon[27388]: osdmap e402: 12 total, 12 up, 12 in 2024-09-16T05:15:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:51 smithi204 ceph-mon[30933]: pgmap v762: 87 pgs: 87 active+clean; 176 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:15:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:51 smithi204 ceph-mon[30933]: osdmap e402: 12 total, 12 up, 12 in 2024-09-16T05:15:52.179 INFO:tasks.rados.rados.2.smithi204.stdout:1255: expect (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:15:52.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:52.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:52.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:52.282 INFO:tasks.rados.rados.2.smithi204.stdout:1257: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-16T05:15:52.288 INFO:tasks.rados.rados.2.smithi204.stdout:1257: writing smithi20457400-19 from 729536 to 1189562 tid 1 2024-09-16T05:15:52.294 INFO:tasks.rados.rados.2.smithi204.stdout:1257: writing smithi20457400-19 from 1749580 to 2360729 tid 2 2024-09-16T05:15:52.300 INFO:tasks.rados.rados.2.smithi204.stdout:1257: writing smithi20457400-19 from 2853383 to 3305833 tid 3 2024-09-16T05:15:52.300 INFO:tasks.rados.rados.2.smithi204.stdout:1253: done (4 left) 2024-09-16T05:15:52.300 INFO:tasks.rados.rados.2.smithi204.stdout:1254: done (3 left) 2024-09-16T05:15:52.300 INFO:tasks.rados.rados.2.smithi204.stdout:1255: done (2 left) 2024-09-16T05:15:52.301 INFO:tasks.rados.rados.2.smithi204.stdout:1256: done (1 left) 2024-09-16T05:15:52.301 INFO:tasks.rados.rados.2.smithi204.stdout:1258: copy_from oid 13 from oid 15 current snap is 138 2024-09-16T05:15:52.301 INFO:tasks.rados.rados.2.smithi204.stdout:1257: finishing write tid 1 to smithi20457400-19 2024-09-16T05:15:52.301 INFO:tasks.rados.rados.2.smithi204.stdout:1257: finishing write tid 2 to smithi20457400-19 2024-09-16T05:15:52.301 INFO:tasks.rados.rados.2.smithi204.stdout:1259: read oid 21 snap -1 2024-09-16T05:15:52.301 INFO:tasks.rados.rados.2.smithi204.stdout:1259: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:15:52.301 INFO:tasks.rados.rados.2.smithi204.stdout:1260: delete oid 43 current snap is 138 2024-09-16T05:15:52.302 INFO:tasks.rados.rados.2.smithi204.stdout:1257: finishing write tid 3 to smithi20457400-19 2024-09-16T05:15:52.302 INFO:tasks.rados.rados.2.smithi204.stdout:1257: finishing write tid 4 to smithi20457400-19 2024-09-16T05:15:52.302 INFO:tasks.rados.rados.2.smithi204.stdout:1257: finishing write tid 5 to smithi20457400-19 2024-09-16T05:15:52.302 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 292 (ObjNum 448 snap 138 seq_num 448) dirty exists 2024-09-16T05:15:52.302 INFO:tasks.rados.rados.2.smithi204.stdout:1257: left oid 19 (ObjNum 448 snap 138 seq_num 448) 2024-09-16T05:15:52.307 INFO:tasks.rados.rados.2.smithi204.stdout:1258: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:15:52.307 INFO:tasks.rados.rados.2.smithi204.stdout:1257: done (3 left) 2024-09-16T05:15:52.307 INFO:tasks.rados.rados.2.smithi204.stdout:1260: done (2 left) 2024-09-16T05:15:52.307 INFO:tasks.rados.rados.2.smithi204.stdout:1261: snap_create 2024-09-16T05:15:52.308 INFO:tasks.rados.rados.2.smithi204.stdout:1259: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:15:52.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:52.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:52.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:52.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:15:52.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:52.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:52.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:52.574 INFO:tasks.rados.rados.2.smithi204.stdout:1258: finishing copy_from to smithi20457400-13 2024-09-16T05:15:52.574 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 119 (ObjNum 423 snap 124 seq_num 423) dirty exists 2024-09-16T05:15:52.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:52.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:52.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:52.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:15:52.655 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:15:52.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:52.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:52.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:52.829 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:15:52.831 INFO:tasks.rados.rados.2.smithi204.stdout:1258: done (2 left) 2024-09-16T05:15:52.832 INFO:tasks.rados.rados.2.smithi204.stdout:1259: done (1 left) 2024-09-16T05:15:52.832 INFO:tasks.rados.rados.2.smithi204.stdout:1261: done (0 left) 2024-09-16T05:15:52.832 INFO:tasks.rados.rados.2.smithi204.stdout:1262: delete oid 11 current snap is 139 2024-09-16T05:15:52.833 INFO:tasks.rados.rados.2.smithi204.stdout:1262: done (0 left) 2024-09-16T05:15:52.833 INFO:tasks.rados.rados.2.smithi204.stdout:1263: delete oid 43 current snap is 139 2024-09-16T05:15:52.836 INFO:tasks.rados.rados.2.smithi204.stdout:1263: done (0 left) 2024-09-16T05:15:52.836 INFO:tasks.rados.rados.2.smithi204.stdout:1264: write (excl) oid 31 current snap is 139 2024-09-16T05:15:52.836 INFO:tasks.rados.rados.2.smithi204.stdout:1264: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-16T05:15:52.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:52.843 INFO:tasks.rados.rados.2.smithi204.stdout:1264: writing smithi20457400-31 from 796764 to 1274139 tid 1 2024-09-16T05:15:52.850 INFO:tasks.rados.rados.2.smithi204.stdout:1264: writing smithi20457400-31 from 1983508 to 2601995 tid 2 2024-09-16T05:15:52.856 INFO:tasks.rados.rados.2.smithi204.stdout:1264: writing smithi20457400-31 from 3363075 to 3596764 tid 3 2024-09-16T05:15:52.857 INFO:tasks.rados.rados.2.smithi204.stdout:1265: read oid 8 snap -1 2024-09-16T05:15:52.857 INFO:tasks.rados.rados.2.smithi204.stdout:1265: expect (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:15:52.857 INFO:tasks.rados.rados.2.smithi204.stdout:1266: read oid 10 snap -1 2024-09-16T05:15:52.857 INFO:tasks.rados.rados.2.smithi204.stdout:1266: expect (ObjNum 349 snap 101 seq_num 349) 2024-09-16T05:15:52.857 INFO:tasks.rados.rados.2.smithi204.stdout:1267: copy_from oid 17 from oid 15 current snap is 139 2024-09-16T05:15:52.857 INFO:tasks.rados.rados.2.smithi204.stdout:1264: finishing write tid 1 to smithi20457400-31 2024-09-16T05:15:52.857 INFO:tasks.rados.rados.2.smithi204.stdout:1264: finishing write tid 2 to smithi20457400-31 2024-09-16T05:15:52.857 INFO:tasks.rados.rados.2.smithi204.stdout:1268: snap_create 2024-09-16T05:15:52.858 INFO:tasks.rados.rados.2.smithi204.stdout:1264: finishing write tid 3 to smithi20457400-31 2024-09-16T05:15:52.858 INFO:tasks.rados.rados.2.smithi204.stdout:1264: finishing write tid 4 to smithi20457400-31 2024-09-16T05:15:52.859 INFO:tasks.rados.rados.2.smithi204.stdout:1264: finishing write tid 5 to smithi20457400-31 2024-09-16T05:15:52.859 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 647 (ObjNum 452 snap 139 seq_num 452) dirty exists 2024-09-16T05:15:52.859 INFO:tasks.rados.rados.2.smithi204.stdout:1264: left oid 31 (ObjNum 452 snap 139 seq_num 452) 2024-09-16T05:15:52.865 INFO:tasks.rados.rados.2.smithi204.stdout:1267: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:15:52.867 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:52.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:52.868 INFO:tasks.rados.rados.2.smithi204.stdout:1265: expect (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:15:52.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:52 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-16T05:15:52.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/18671119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:52.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:52.997 INFO:tasks.rados.rados.2.smithi204.stdout:1267: finishing copy_from to smithi20457400-17 2024-09-16T05:15:52.997 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 119 (ObjNum 423 snap 124 seq_num 423) dirty exists 2024-09-16T05:15:52.997 INFO:tasks.rados.rados.2.smithi204.stdout:1266: expect (ObjNum 349 snap 101 seq_num 349) 2024-09-16T05:15:53.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:53.010 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:53.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:52 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-16T05:15:53.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/18671119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:53.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:53.131 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:53.132 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:53.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:53.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:53.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:53.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:52 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-16T05:15:53.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/18671119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:15:53.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:53.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:53.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:53.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:53.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:53.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:53.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:53.597 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:53.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:53.634 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:53.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:53.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:53.761 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:53.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:53.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:15:53.840 INFO:tasks.rados.rados.2.smithi204.stdout:1264: done (4 left) 2024-09-16T05:15:53.841 INFO:tasks.rados.rados.2.smithi204.stdout:1265: done (3 left) 2024-09-16T05:15:53.841 INFO:tasks.rados.rados.2.smithi204.stdout:1266: done (2 left) 2024-09-16T05:15:53.841 INFO:tasks.rados.rados.2.smithi204.stdout:1267: done (1 left) 2024-09-16T05:15:53.841 INFO:tasks.rados.rados.2.smithi204.stdout:1268: done (0 left) 2024-09-16T05:15:53.842 INFO:tasks.rados.rados.2.smithi204.stdout:1269: write (excl) oid 27 current snap is 140 2024-09-16T05:15:53.842 INFO:tasks.rados.rados.2.smithi204.stdout:1269: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-16T05:15:53.846 INFO:tasks.rados.rados.2.smithi204.stdout:1269: writing smithi20457400-27 from 413571 to 966195 tid 1 2024-09-16T05:15:53.855 INFO:tasks.rados.rados.2.smithi204.stdout:1269: writing smithi20457400-27 from 1641990 to 2404135 tid 2 2024-09-16T05:15:53.861 INFO:tasks.rados.rados.2.smithi204.stdout:1269: writing smithi20457400-27 from 2990498 to 3415661 tid 3 2024-09-16T05:15:53.861 INFO:tasks.rados.rados.2.smithi204.stdout:1270: read oid 2 snap -1 2024-09-16T05:15:53.861 INFO:tasks.rados.rados.2.smithi204.stdout:1270: expect (ObjNum 430 snap 124 seq_num 430) 2024-09-16T05:15:53.861 INFO:tasks.rados.rados.2.smithi204.stdout:1271: snap_create 2024-09-16T05:15:53.861 INFO:tasks.rados.rados.2.smithi204.stdout:1269: finishing write tid 1 to smithi20457400-27 2024-09-16T05:15:53.861 INFO:tasks.rados.rados.2.smithi204.stdout:1269: finishing write tid 2 to smithi20457400-27 2024-09-16T05:15:53.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:53.864 INFO:tasks.rados.rados.2.smithi204.stdout:1269: finishing write tid 3 to smithi20457400-27 2024-09-16T05:15:53.864 INFO:tasks.rados.rados.2.smithi204.stdout:1269: finishing write tid 4 to smithi20457400-27 2024-09-16T05:15:53.865 INFO:tasks.rados.rados.2.smithi204.stdout:1269: finishing write tid 5 to smithi20457400-27 2024-09-16T05:15:53.865 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 297 (ObjNum 453 snap 140 seq_num 453) dirty exists 2024-09-16T05:15:53.865 INFO:tasks.rados.rados.2.smithi204.stdout:1269: left oid 27 (ObjNum 453 snap 140 seq_num 453) 2024-09-16T05:15:53.868 INFO:tasks.rados.rados.2.smithi204.stdout:1270: expect (ObjNum 430 snap 124 seq_num 430) 2024-09-16T05:15:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:53.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:53.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:53.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:54.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:54.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:54.088 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: pgmap v765: 97 pgs: 4 peering, 8 unknown, 85 active+clean; 181 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/18671119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: osdmap e403: 12 total, 12 up, 12 in 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:15:54.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:53 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:15:54.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:54.122 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:54.122 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:54.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:54.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:54.144 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:54.144 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:54.164 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:54.164 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: pgmap v765: 97 pgs: 4 peering, 8 unknown, 85 active+clean; 181 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-16T05:15:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T05:15:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/18671119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: osdmap e403: 12 total, 12 up, 12 in 2024-09-16T05:15:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:15:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:15:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:15:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:53 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:15:54.185 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:54.185 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:54.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:54.206 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:54.228 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:54.228 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: pgmap v765: 97 pgs: 4 peering, 8 unknown, 85 active+clean; 181 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-16T05:15:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:15:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-16T05:15:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/18671119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:15:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: osdmap e403: 12 total, 12 up, 12 in 2024-09-16T05:15:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:15:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:15:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:15:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:53 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:15:54.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:54.249 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:54.249 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:54.262 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:54.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:54.271 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:54.272 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:54.296 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:54.296 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:54.314 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:54.315 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:54.337 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:54.337 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:54.358 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:54.358 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:54.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:54.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:54.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:54.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:54.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:54.572 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:15:54.573 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:15:54.573 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:15:54.574 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:15:54.574 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:15:54.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:54.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:54.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:54.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:54.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:54.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:54.840 INFO:tasks.rados.rados.2.smithi204.stdout:1269: done (2 left) 2024-09-16T05:15:54.840 INFO:tasks.rados.rados.2.smithi204.stdout:1270: done (1 left) 2024-09-16T05:15:54.841 INFO:tasks.rados.rados.2.smithi204.stdout:1271: done (0 left) 2024-09-16T05:15:54.841 INFO:tasks.rados.rados.2.smithi204.stdout:1272: read oid 18 snap -1 2024-09-16T05:15:54.841 INFO:tasks.rados.rados.2.smithi204.stdout:1272: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:15:54.841 INFO:tasks.rados.rados.2.smithi204.stdout:1273: snap_remove snap 127 2024-09-16T05:15:54.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:54.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:54.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:54.986 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:15:54.986 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:15:54.986 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-16T05:15:54.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:15:54.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:55.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: osdmap e404: 12 total, 12 up, 12 in 2024-09-16T05:15:55.010 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/577178812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:55.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:55.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:55.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:55.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:55.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: osdmap e404: 12 total, 12 up, 12 in 2024-09-16T05:15:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/577178812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:55.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:15:55.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: osdmap e404: 12 total, 12 up, 12 in 2024-09-16T05:15:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/577178812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:55.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:55.268 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:55.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:55.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:55.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:55.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:55.456 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:15:55.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:55.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:55.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:55.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:15:55.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:55.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:55.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:55.717 INFO:tasks.rados.rados.2.smithi204.stdout:1273: done (1 left) 2024-09-16T05:15:55.717 INFO:tasks.rados.rados.2.smithi204.stdout:1274: copy_from oid 9 from oid 24 current snap is 141 2024-09-16T05:15:55.717 INFO:tasks.rados.rados.2.smithi204.stdout:1275: snap_create 2024-09-16T05:15:55.717 INFO:tasks.rados.rados.2.smithi204.stdout:1272: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:15:55.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:55.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:55.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:55.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:15:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:55.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:55.889 INFO:tasks.rados.rados.2.smithi204.stdout:1274: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:15:55.889 INFO:tasks.rados.rados.2.smithi204.stdout:1274: finishing copy_from to smithi20457400-9 2024-09-16T05:15:55.889 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 450 (ObjNum 437 snap 128 seq_num 437) dirty exists 2024-09-16T05:15:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:55.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:55.964 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: pgmap v768: 97 pgs: 4 peering, 2 remapped+peering, 6 unknown, 85 active+clean; 187 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 25 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: osdmap e405: 12 total, 12 up, 12 in 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:15:55.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:15:55.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:15:55.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:15:55.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:55 smithi032 ceph-mon[27388]: osdmap e406: 12 total, 12 up, 12 in 2024-09-16T05:15:55.967 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:15:55.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:15:56.004 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:15:56.004 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:19 2.b 2:d7c71ae7:::smithi20457400-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 312.96182835899998, 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19676892500000001, 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 19 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:15:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040519+0000", 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7300000000000001e-07 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040541+0000", 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2320999999999999e-05 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040547+0000", 2024-09-16T05:15:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3060000000000003e-06 2024-09-16T05:15:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040553+0000", 2024-09-16T05:15:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2750000000000003e-06 2024-09-16T05:15:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041020+0000", 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046726200000000003 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041092+0000", 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2275999999999997e-05 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:15:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041183+0000", 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1071999999999998e-05 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041456+0000", 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027275299999999998 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237175+0000", 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19571920200000001 2024-09-16T05:15:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237230+0000", 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4855000000000002e-05 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237287+0000", 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7129999999999997e-05 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:20 2.b 2:d7c71ae7:::smithi20457400-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 312.961786625, 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19685694000000001, 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 20 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:15:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4400000000000001e-07 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040587+0000", 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6444e-05 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040593+0000", 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5320000000000003e-06 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040600+0000", 2024-09-16T05:15:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5139999999999998e-06 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041356+0000", 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075660300000000001 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041400+0000", 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3936000000000001e-05 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237316+0000", 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.195915805 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237361+0000", 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4548000000000001e-05 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237417+0000", 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6214000000000001e-05 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:274 2.b 2:d7c71ae7:::smithi20457400-5:head [write 2964119~94722] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 306.97650878500002, 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073226163999999996, 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 274 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025839+0000", 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.141e-06 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025876+0000", 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6724999999999998e-05 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025887+0000", 2024-09-16T05:15:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1413999999999999e-05 2024-09-16T05:15:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2739999999999998e-06 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025932+0000", 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.943e-05 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026014+0000", 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2540000000000006e-05 2024-09-16T05:15:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026154+0000", 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013913200000000001 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026818+0000", 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00066437899999999999 2024-09-16T05:15:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099005+0000", 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072186933999999994 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099037+0000", 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2011000000000002e-05 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099064+0000", 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7183999999999999e-05 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:275 2.b 2:d7c71ae7:::smithi20457400-5:head [setxattr _header (55) in=7b,truncate 3058841] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 306.97644338999999, 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13101358900000001, 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 275 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025904+0000", 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.34e-07 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025907+0000", 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9610000000000001e-06 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025910+0000", 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8680000000000001e-06 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025913+0000", 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9960000000000002e-06 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026440+0000", 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00052629399999999996 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026504+0000", 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3934000000000001e-05 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026588+0000", 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4745000000000003e-05 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026821+0000", 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000232301 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156835+0000", 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13001446 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156875+0000", 2024-09-16T05:15:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9360999999999998e-05 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156917+0000", 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2335e-05 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:276 2.b 2:d7c71ae7:::smithi20457400-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:15:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 306.97641832400001, 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.131149614, 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 276 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025929+0000", 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.39e-07 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025931+0000", 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5940000000000001e-06 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025939+0000", 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1829999999999997e-06 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025944+0000", 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4770000000000002e-06 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026741+0000", 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000797326 2024-09-16T05:15:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026805+0000", 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3409000000000002e-05 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156955+0000", 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13015033600000001 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.157016+0000", 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0795999999999999e-05 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.157078+0000", 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2053999999999999e-05 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:274 2.b e117/107 ondisk, result = 0)", 2024-09-16T05:15:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.026832+0000", 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 306.97551495099998, 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072250941999999999, 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026832+0000", 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026832+0000", 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026832+0000", 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0200000000000002e-07 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026836+0000", 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7570000000000002e-06 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098946+0000", 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072109698 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098953+0000", 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.041e-06 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098996+0000", 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3041999999999999e-05 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099001+0000", 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6370000000000004e-06 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099083+0000", 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1165e-05 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:286 2.b 2:da4f6049:::smithi20457400-49:head [delete] snapc 4=[4] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117)", 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 306.90421795499998, 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.242160297, 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 286 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:15:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6699999999999999e-07 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098131+0000", 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1289999999999999e-06 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098138+0000", 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5660000000000003e-06 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098143+0000", 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098194+0000", 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1368e-05 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098273+0000", 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8170999999999994e-05 2024-09-16T05:15:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098449+0000", 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176171 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099087+0000", 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00063836000000000003 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340229+0000", 2024-09-16T05:15:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.241141684 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340263+0000", 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3813000000000003e-05 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340289+0000", 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6403000000000001e-05 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:275 2.b e117/107 ondisk, result = 0)", 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098921+0000", 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 306.90342521899998, 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.058002900000000003, 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098921+0000", 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098921+0000", 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098922+0000", 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1200000000000002e-07 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098926+0000", 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6839999999999998e-06 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156740+0000", 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.057813937000000003 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156750+0000", 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7829999999999993e-06 2024-09-16T05:15:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156824+0000", 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4012999999999998e-05 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156831+0000", 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7219999999999999e-06 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156924+0000", 2024-09-16T05:15:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3048999999999996e-05 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:414 2.5 2:ad4a88eb:::smithi20457400-16:head [setxattr _header (56) in=7b,truncate 3394981] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 298.305427436, 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29453131100000002, 2024-09-16T05:15:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 414 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696920+0000", 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999996e-07 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696923+0000", 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.92e-06 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696937+0000", 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3892e-05 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696941+0000", 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3819999999999997e-06 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697487+0000", 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054579200000000002 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697548+0000", 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1167000000000006e-05 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697638+0000", 2024-09-16T05:15:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0548000000000001e-05 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697936+0000", 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029720499999999998 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991363+0000", 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29342759099999999 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991404+0000", 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1013999999999997e-05 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991451+0000", 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6303e-05 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:415 2.5 2:ad4a88eb:::smithi20457400-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 298.30539930399999, 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29466997700000003, 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 415 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696948+0000", 2024-09-16T05:15:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696952+0000", 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6500000000000002e-06 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696958+0000", 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9149999999999999e-06 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696963+0000", 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9439999999999996e-06 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697806+0000", 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084291800000000001 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697861+0000", 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4725999999999997e-05 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991493+0000", 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29363172500000001 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991551+0000", 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8622000000000002e-05 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991617+0000", 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5977999999999997e-05 2024-09-16T05:15:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:414 2.5 e126/107 ondisk, result = 0)", 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 298.30423772900002, 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29334887300000001, 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.06e-07 2024-09-16T05:15:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698112+0000", 2024-09-16T05:15:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6520000000000002e-06 2024-09-16T05:15:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991214+0000", 2024-09-16T05:15:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:56.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:56.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:15:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29310221600000003 2024-09-16T05:15:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991228+0000", 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.411e-05 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991351+0000", 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012277999999999999 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991359+0000", 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0430000000000008e-06 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991458+0000", 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8566000000000005e-05 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:884 2.b 2:d8680303:::smithi20457400-43:head [setxattr _header (57) in=7b,truncate 2470629] snapc 23=[23,22,21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 254.341671698, 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68080432499999999, 2024-09-16T05:15:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 884 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:15:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660676+0000", 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2099999999999996e-07 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660679+0000", 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1769999999999998e-06 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660685+0000", 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6529999999999997e-06 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660691+0000", 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2530000000000004e-06 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.661798+0000", 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011070520000000001 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.661889+0000", 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0785999999999994e-05 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662038+0000", 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000149777 2024-09-16T05:15:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662482+0000", 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00044356499999999999 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341385+0000", 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67890330499999996 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:15:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341430+0000", 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4613999999999997e-05 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341479+0000", 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9521999999999998e-05 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:885 2.b 2:d8680303:::smithi20457400-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", 2024-09-16T05:15:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 254.341647628, 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68095269999999997, 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 885 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999999e-07 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660702+0000", 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3099999999999999e-06 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660709+0000", 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0299999999999996e-06 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660714+0000", 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1780000000000002e-06 2024-09-16T05:15:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662345+0000", 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016309390000000001 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662461+0000", 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011565600000000001 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341517+0000", 2024-09-16T05:15:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67905638400000001 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341582+0000", 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4746999999999995e-05 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341652+0000", 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0026999999999999e-05 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:405 2.1a e216/215 ondisk, result = 0)", 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 208.33584076700001, 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26443446100000001, 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666507+0000", 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8700000000000005e-07 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666511+0000", 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1130000000000003e-06 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930764+0000", 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26425358900000001 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930771+0000", 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7329999999999997e-06 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930824+0000", 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3436000000000001e-05 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930834+0000", 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1570000000000006e-06 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930940+0000", 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106746 2024-09-16T05:15:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1814 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 177.23057236599999, 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898298185, 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1814 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771776+0000", 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5149999999999999e-06 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771783+0000", 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0180000000000004e-06 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771805+0000", 2024-09-16T05:15:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2076999999999999e-05 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771824+0000", 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8813000000000001e-05 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747524+0000", 2024-09-16T05:15:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97570031700000004 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747560+0000", 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5549000000000001e-05 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670057+0000", 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92249772500000005 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670072+0000", 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5170999999999999e-05 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1815 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 177.230470467, 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8982342750000001, 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1815 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771877+0000", 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9599999999999999e-07 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771881+0000", 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6880000000000001e-06 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771889+0000", 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6789999999999994e-06 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771896+0000", 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747584+0000", 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97568814000000004 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747590+0000", 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7440000000000004e-06 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670102+0000", 2024-09-16T05:15:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92251251499999998 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670110+0000", 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2740000000000007e-06 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1816 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 177.23038847800001, 2024-09-16T05:15:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898192844, 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1816 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:15:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771959+0000", 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1999999999999998e-07 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771964+0000", 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771971+0000", 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.576e-06 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771977+0000", 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2960000000000004e-06 2024-09-16T05:15:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747609+0000", 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97563200299999997 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747615+0000", 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2820000000000004e-06 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670144+0000", 2024-09-16T05:15:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92252909100000002 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670151+0000", 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2259999999999999e-06 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1881 2.25 2:a49174eb:::smithi20457400-39:head [delete] snapc 50=[50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e257)", 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:15:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 167.30579917700001, 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37621878800000003, 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1881 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696549+0000", 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696556+0000", 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8090000000000004e-06 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696568+0000", 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2636000000000001e-05 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696576+0000", 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3459999999999996e-06 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696625+0000", 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9830000000000002e-05 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696720+0000", 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4816999999999997e-05 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696869+0000", 2024-09-16T05:15:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014893999999999999 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.697931+0000", 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010616759999999999 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072669+0000", 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37473788800000002 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072715+0000", 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6371e-05 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072766+0000", 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1220000000000001e-05 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1881 2.25 e257/252 ondisk, result = 0)", 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 167.30427092799999, 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37469565799999999, 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:15:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698077+0000", 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5099999999999998e-07 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698082+0000", 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072556+0000", 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37447454499999999 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072564+0000", 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4120000000000004e-06 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072654+0000", 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9878999999999998e-05 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072663+0000", 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072771+0000", 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108267 2024-09-16T05:15:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:788 2.25 e406/252 ondisk, result = 0)", 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:55.723815+0000", 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.27853128199999999, 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073107094999999997, 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.723815+0000", 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.723815+0000", 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.723816+0000", 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9899999999999999e-07 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.723821+0000", 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5859999999999998e-06 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796769+0000", 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072947949999999998 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796778+0000", 2024-09-16T05:15:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9910000000000008e-06 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796828+0000", 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0112999999999999e-05 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796835+0000", 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5909999999999998e-06 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:15:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:15:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796922+0000", 2024-09-16T05:15:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7064999999999994e-05 2024-09-16T05:15:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:15:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:15:56.115 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:15:56.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: pgmap v768: 97 pgs: 4 peering, 2 remapped+peering, 6 unknown, 85 active+clean; 187 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 25 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: osdmap e405: 12 total, 12 up, 12 in 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:15:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:55 smithi204 ceph-mon[30933]: osdmap e406: 12 total, 12 up, 12 in 2024-09-16T05:15:56.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:56.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: pgmap v768: 97 pgs: 4 peering, 2 remapped+peering, 6 unknown, 85 active+clean; 187 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 2.0 MiB/s wr, 25 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: osdmap e405: 12 total, 12 up, 12 in 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:15:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:55 smithi196 ceph-mon[30802]: osdmap e406: 12 total, 12 up, 12 in 2024-09-16T05:15:56.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:56.314 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:56.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:56.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:56.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:56.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:56.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:56.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:56.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:56.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:56.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:56.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:56.715 INFO:tasks.rados.rados.2.smithi204.stdout:1272: done (2 left) 2024-09-16T05:15:56.715 INFO:tasks.rados.rados.2.smithi204.stdout:1274: done (1 left) 2024-09-16T05:15:56.715 INFO:tasks.rados.rados.2.smithi204.stdout:1275: done (0 left) 2024-09-16T05:15:56.715 INFO:tasks.rados.rados.2.smithi204.stdout:1276: read oid 45 snap -1 2024-09-16T05:15:56.715 INFO:tasks.rados.rados.2.smithi204.stdout:1276: expect (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:15:56.715 INFO:tasks.rados.rados.2.smithi204.stdout:1277: copy_from oid 6 from oid 21 current snap is 142 2024-09-16T05:15:56.715 INFO:tasks.rados.rados.2.smithi204.stdout:1278: write oid 11 current snap is 142 2024-09-16T05:15:56.715 INFO:tasks.rados.rados.2.smithi204.stdout:1278: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-16T05:15:56.721 INFO:tasks.rados.rados.2.smithi204.stdout:1278: writing smithi20457400-11 from 430378 to 1058251 tid 1 2024-09-16T05:15:56.728 INFO:tasks.rados.rados.2.smithi204.stdout:1278: writing smithi20457400-11 from 1700472 to 2322628 tid 2 2024-09-16T05:15:56.736 INFO:tasks.rados.rados.2.smithi204.stdout:1278: writing smithi20457400-11 from 3017921 to 3630378 tid 3 2024-09-16T05:15:56.736 INFO:tasks.rados.rados.2.smithi204.stdout:1279: delete oid 34 current snap is 142 2024-09-16T05:15:56.736 INFO:tasks.rados.rados.2.smithi204.stdout:1276: expect (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:15:56.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:15:56.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:56.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:56.863 INFO:tasks.rados.rados.2.smithi204.stdout:1278: finishing write tid 1 to smithi20457400-11 2024-09-16T05:15:56.863 INFO:tasks.rados.rados.2.smithi204.stdout:1277: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:15:56.863 INFO:tasks.rados.rados.2.smithi204.stdout:1277: finishing copy_from to smithi20457400-6 2024-09-16T05:15:56.863 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 322 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:15:56.863 INFO:tasks.rados.rados.2.smithi204.stdout:1276: done (3 left) 2024-09-16T05:15:56.864 INFO:tasks.rados.rados.2.smithi204.stdout:1277: done (2 left) 2024-09-16T05:15:56.864 INFO:tasks.rados.rados.2.smithi204.stdout:1279: done (1 left) 2024-09-16T05:15:56.864 INFO:tasks.rados.rados.2.smithi204.stdout:1280: write oid 1 current snap is 142 2024-09-16T05:15:56.864 INFO:tasks.rados.rados.2.smithi204.stdout:1278: finishing write tid 2 to smithi20457400-11 2024-09-16T05:15:56.864 INFO:tasks.rados.rados.2.smithi204.stdout:1278: finishing write tid 3 to smithi20457400-11 2024-09-16T05:15:56.864 INFO:tasks.rados.rados.2.smithi204.stdout:1280: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-09-16T05:15:56.869 INFO:tasks.rados.rados.2.smithi204.stdout:1280: writing smithi20457400-1 from 463992 to 1242363 tid 1 2024-09-16T05:15:56.873 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":770,"stamp":"2024-09-16T05:15:55.709348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":196690001,"num_objects":88,"num_object_clones":39,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":8,"num_read":810,"num_read_kb":825734,"num_write":547,"num_write_kb":221573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2755476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18246,"ondisk_log_size":18246,"up":190,"acting":190,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":730000,"kb_used_data":292212,"kb_used_omap":27,"kb_used_meta":437476,"kb_avail":1124211824,"statfs":{"total":1151940427776,"available":1151192907776,"internally_reserved":0,"allocated":299225088,"data_stored":328739913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":447975591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6509262,"num_objects":0,"num_object_clones":-1,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":80,"num_read_kb":74794,"num_write":33,"num_write_kb":6229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.999812"},"pg_stats":[{"pgid":"2.4b","version":"244'177","reported_seq":1638,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869575+0000","last_change":"2024-09-16T05:15:51.720193+0000","last_active":"2024-09-16T05:15:54.869575+0000","last_peered":"2024-09-16T05:15:54.869575+0000","last_clean":"2024-09-16T05:15:54.869575+0000","last_became_active":"2024-09-16T05:14:30.853998+0000","last_became_peered":"2024-09-16T05:14:30.853998+0000","last_unstale":"2024-09-16T05:15:54.869575+0000","last_undegraded":"2024-09-16T05:15:54.869575+0000","last_fullsized":"2024-09-16T05:15:54.869575+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:14:32.337052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:32.337052+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045188699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.4a","version":"378'461","reported_seq":1871,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.713617+0000","last_change":"2024-09-16T05:15:49.817251+0000","last_active":"2024-09-16T05:15:50.713617+0000","last_peered":"2024-09-16T05:15:50.713617+0000","last_clean":"2024-09-16T05:15:50.713617+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:15:50.713617+0000","last_undegraded":"2024-09-16T05:15:50.713617+0000","last_fullsized":"2024-09-16T05:15:50.713617+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"378'461","last_scrub_stamp":"2024-09-16T05:15:37.597739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:37.597739+0000","objects_scrubbed":1,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:37.643353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077508600000000003,"stat_sum":{"num_bytes":2806835,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.49","version":"245'220","reported_seq":1656,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.709420+0000","last_change":"2024-09-16T05:15:49.823405+0000","last_active":"2024-09-16T05:15:50.709420+0000","last_peered":"2024-09-16T05:15:50.709420+0000","last_clean":"2024-09-16T05:15:50.709420+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:15:50.709420+0000","last_undegraded":"2024-09-16T05:15:50.709420+0000","last_fullsized":"2024-09-16T05:15:50.709420+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:14:30.917243+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.917243+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045782199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.48","version":"402'189","reported_seq":1614,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.840629+0000","last_change":"2024-09-16T05:15:51.731874+0000","last_active":"2024-09-16T05:15:54.840629+0000","last_peered":"2024-09-16T05:15:54.840629+0000","last_clean":"2024-09-16T05:15:54.840629+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:15:54.840629+0000","last_undegraded":"2024-09-16T05:15:54.840629+0000","last_fullsized":"2024-09-16T05:15:54.840629+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"378'179","last_scrub_stamp":"2024-09-16T05:15:39.057858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.057858+0000","objects_scrubbed":1,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:02.054990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010921617999999999,"stat_sum":{"num_bytes":6336338,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":20676,"num_write":2,"num_write_kb":6189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.47","version":"0'0","reported_seq":445,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.841270+0000","last_change":"2024-09-16T05:15:51.716267+0000","last_active":"2024-09-16T05:15:54.841270+0000","last_peered":"2024-09-16T05:15:54.841270+0000","last_clean":"2024-09-16T05:15:54.841270+0000","last_became_active":"2024-09-16T05:15:16.867821+0000","last_became_peered":"2024-09-16T05:15:16.867821+0000","last_unstale":"2024-09-16T05:15:54.841270+0000","last_undegraded":"2024-09-16T05:15:54.841270+0000","last_fullsized":"2024-09-16T05:15:54.841270+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:28.987418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.987418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045769400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.46","version":"116'9","reported_seq":1470,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869639+0000","last_change":"2024-09-16T05:15:51.722366+0000","last_active":"2024-09-16T05:15:54.869639+0000","last_peered":"2024-09-16T05:15:54.869639+0000","last_clean":"2024-09-16T05:15:54.869639+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:15:54.869639+0000","last_undegraded":"2024-09-16T05:15:54.869639+0000","last_fullsized":"2024-09-16T05:15:54.869639+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:15:37.935128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:37.935128+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:07.806532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001227973,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.45","version":"305'464","reported_seq":1889,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869745+0000","last_change":"2024-09-16T05:15:51.721232+0000","last_active":"2024-09-16T05:15:54.869745+0000","last_peered":"2024-09-16T05:15:54.869745+0000","last_clean":"2024-09-16T05:15:54.869745+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:15:54.869745+0000","last_undegraded":"2024-09-16T05:15:54.869745+0000","last_fullsized":"2024-09-16T05:15:54.869745+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:14:31.325082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.325082+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00071889199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":1552,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869518+0000","last_change":"2024-09-16T05:15:51.720699+0000","last_active":"2024-09-16T05:15:54.869518+0000","last_peered":"2024-09-16T05:15:54.869518+0000","last_clean":"2024-09-16T05:15:54.869518+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:15:54.869518+0000","last_undegraded":"2024-09-16T05:15:54.869518+0000","last_fullsized":"2024-09-16T05:15:54.869518+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.970206+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:51.011214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063613199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":434,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.722801+0000","last_change":"2024-09-16T05:15:51.723003+0000","last_active":"2024-09-16T05:15:51.722801+0000","last_peered":"2024-09-16T05:15:51.722801+0000","last_clean":"2024-09-16T05:15:51.722801+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:15:51.722801+0000","last_undegraded":"2024-09-16T05:15:51.722801+0000","last_fullsized":"2024-09-16T05:15:51.722801+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:14:28.659562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.659562+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0087122020000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.42","version":"182'20","reported_seq":418,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-16T05:15:53.838447+0000","last_change":"2024-09-16T05:15:51.712784+0000","last_active":"2024-09-16T05:15:53.838447+0000","last_peered":"2024-09-16T05:15:53.838447+0000","last_clean":"2024-09-16T05:15:53.838447+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:15:53.838447+0000","last_undegraded":"2024-09-16T05:15:53.838447+0000","last_fullsized":"2024-09-16T05:15:53.838447+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:33.678710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:33.678710+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:55.991008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001142517,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.41","version":"178'50","reported_seq":518,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.713639+0000","last_change":"2024-09-16T05:15:49.816692+0000","last_active":"2024-09-16T05:15:50.713639+0000","last_peered":"2024-09-16T05:15:50.713639+0000","last_clean":"2024-09-16T05:15:50.713639+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:15:50.713639+0000","last_undegraded":"2024-09-16T05:15:50.713639+0000","last_fullsized":"2024-09-16T05:15:50.713639+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:15.794439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.794439+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.425935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050158199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.40","version":"205'33","reported_seq":1481,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.722746+0000","last_change":"2024-09-16T05:15:51.722746+0000","last_active":"2024-09-16T05:15:51.722746+0000","last_peered":"2024-09-16T05:15:51.722746+0000","last_clean":"2024-09-16T05:15:51.722746+0000","last_became_active":"2024-09-16T05:13:45.682851+0000","last_became_peered":"2024-09-16T05:13:45.682851+0000","last_unstale":"2024-09-16T05:15:51.722746+0000","last_undegraded":"2024-09-16T05:15:51.722746+0000","last_fullsized":"2024-09-16T05:15:51.722746+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:07.789202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.789202+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:18.075185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0087321529999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.3f","version":"378'282","reported_seq":1727,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.724662+0000","last_change":"2024-09-16T05:15:51.724662+0000","last_active":"2024-09-16T05:15:51.724662+0000","last_peered":"2024-09-16T05:15:51.724662+0000","last_clean":"2024-09-16T05:15:51.724662+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:15:51.724662+0000","last_undegraded":"2024-09-16T05:15:51.724662+0000","last_fullsized":"2024-09-16T05:15:51.724662+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"357'270","last_scrub_stamp":"2024-09-16T05:15:09.797411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.797411+0000","objects_scrubbed":3,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:11.693246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0092618849999999992,"stat_sum":{"num_bytes":2285818,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":15220,"num_write":15,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.3e","version":"378'353","reported_seq":1761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.714248+0000","last_change":"2024-09-16T05:15:51.714509+0000","last_active":"2024-09-16T05:15:51.714248+0000","last_peered":"2024-09-16T05:15:51.714248+0000","last_clean":"2024-09-16T05:15:51.714248+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:15:51.714248+0000","last_undegraded":"2024-09-16T05:15:51.714248+0000","last_fullsized":"2024-09-16T05:15:51.714248+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"360'342","last_scrub_stamp":"2024-09-16T05:15:09.920637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.920637+0000","objects_scrubbed":4,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:27.293473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00042716400000000001,"stat_sum":{"num_bytes":2437081,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":2380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1992,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869934+0000","last_change":"2024-09-16T05:15:51.722135+0000","last_active":"2024-09-16T05:15:54.869934+0000","last_peered":"2024-09-16T05:15:54.869934+0000","last_clean":"2024-09-16T05:15:54.869934+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:15:54.869934+0000","last_undegraded":"2024-09-16T05:15:54.869934+0000","last_fullsized":"2024-09-16T05:15:54.869934+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:15:09.150849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.150849+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:18.367400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014530960000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.3c","version":"402'454","reported_seq":1983,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.721981+0000","last_change":"2024-09-16T05:15:51.722176+0000","last_active":"2024-09-16T05:15:51.721981+0000","last_peered":"2024-09-16T05:15:51.721981+0000","last_clean":"2024-09-16T05:15:51.721981+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:15:51.721981+0000","last_undegraded":"2024-09-16T05:15:51.721981+0000","last_fullsized":"2024-09-16T05:15:51.721981+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"342'400","last_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:14:51.845721+0000","objects_scrubbed":6,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:10.096001+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0085128080000000002,"stat_sum":{"num_bytes":5540525,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":33088,"num_write":19,"num_write_kb":6444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.3b","version":"399'303","reported_seq":1743,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.709568+0000","last_change":"2024-09-16T05:15:49.830650+0000","last_active":"2024-09-16T05:15:50.709568+0000","last_peered":"2024-09-16T05:15:50.709568+0000","last_clean":"2024-09-16T05:15:50.709568+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:15:50.709568+0000","last_undegraded":"2024-09-16T05:15:50.709568+0000","last_fullsized":"2024-09-16T05:15:50.709568+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"300'290","last_scrub_stamp":"2024-09-16T05:15:06.955285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.955285+0000","objects_scrubbed":2,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:36.505207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.007534041,"stat_sum":{"num_bytes":5277801,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":46642,"num_write":5,"num_write_kb":1725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.3a","version":"402'464","reported_seq":1880,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.840737+0000","last_change":"2024-09-16T05:15:51.724270+0000","last_active":"2024-09-16T05:15:54.840737+0000","last_peered":"2024-09-16T05:15:54.840737+0000","last_clean":"2024-09-16T05:15:54.840737+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:15:54.840737+0000","last_undegraded":"2024-09-16T05:15:54.840737+0000","last_fullsized":"2024-09-16T05:15:54.840737+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"356'429","last_scrub_stamp":"2024-09-16T05:15:10.141866+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.141866+0000","objects_scrubbed":2,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:12.133286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0072600060000000003,"stat_sum":{"num_bytes":7237161,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":9241,"num_write":24,"num_write_kb":11603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.39","version":"182'60","reported_seq":538,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.840675+0000","last_change":"2024-09-16T05:15:51.729334+0000","last_active":"2024-09-16T05:15:54.840675+0000","last_peered":"2024-09-16T05:15:54.840675+0000","last_clean":"2024-09-16T05:15:54.840675+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:15:54.840675+0000","last_undegraded":"2024-09-16T05:15:54.840675+0000","last_fullsized":"2024-09-16T05:15:54.840675+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:15.232072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.232072+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.264922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0089870599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.38","version":"181'57","reported_seq":531,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-16T05:15:53.837655+0000","last_change":"2024-09-16T05:15:51.712446+0000","last_active":"2024-09-16T05:15:53.837655+0000","last_peered":"2024-09-16T05:15:53.837655+0000","last_clean":"2024-09-16T05:15:53.837655+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:15:53.837655+0000","last_undegraded":"2024-09-16T05:15:53.837655+0000","last_fullsized":"2024-09-16T05:15:53.837655+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:13.450261+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:13.450261+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:39.027970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071029999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.37","version":"0'0","reported_seq":605,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.714268+0000","last_change":"2024-09-16T05:15:49.817510+0000","last_active":"2024-09-16T05:15:50.714268+0000","last_peered":"2024-09-16T05:15:50.714268+0000","last_clean":"2024-09-16T05:15:50.714268+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:15:50.714268+0000","last_undegraded":"2024-09-16T05:15:50.714268+0000","last_fullsized":"2024-09-16T05:15:50.714268+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.650972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.835383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097576900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.36","version":"272'60","reported_seq":620,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869806+0000","last_change":"2024-09-16T05:15:51.722740+0000","last_active":"2024-09-16T05:15:54.869806+0000","last_peered":"2024-09-16T05:15:54.869806+0000","last_clean":"2024-09-16T05:15:54.869806+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:15:54.869806+0000","last_undegraded":"2024-09-16T05:15:54.869806+0000","last_fullsized":"2024-09-16T05:15:54.869806+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:14:49.375386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.375386+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:59.004324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013522619999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.35","version":"402'507","reported_seq":1980,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.870077+0000","last_change":"2024-09-16T05:15:51.727298+0000","last_active":"2024-09-16T05:15:54.870077+0000","last_peered":"2024-09-16T05:15:54.870077+0000","last_clean":"2024-09-16T05:15:54.870077+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:15:54.870077+0000","last_undegraded":"2024-09-16T05:15:54.870077+0000","last_fullsized":"2024-09-16T05:15:54.870077+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'421","last_scrub_stamp":"2024-09-16T05:14:34.349392+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:14:34.349392+0000","objects_scrubbed":4,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:03.961887+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0065986869999999998,"stat_sum":{"num_bytes":8324759,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":24061,"num_write":19,"num_write_kb":7769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.34","version":"400'317","reported_seq":1354,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.709675+0000","last_change":"2024-09-16T05:15:49.825071+0000","last_active":"2024-09-16T05:15:50.709675+0000","last_peered":"2024-09-16T05:15:50.709675+0000","last_clean":"2024-09-16T05:15:50.709675+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:15:50.709675+0000","last_undegraded":"2024-09-16T05:15:50.709675+0000","last_fullsized":"2024-09-16T05:15:50.709675+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'243","last_scrub_stamp":"2024-09-16T05:14:50.913931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.913931+0000","objects_scrubbed":5,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:12.935125+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00198525,"stat_sum":{"num_bytes":7619765,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":41072,"num_write":14,"num_write_kb":8547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":613,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.723722+0000","last_change":"2024-09-16T05:15:51.723722+0000","last_active":"2024-09-16T05:15:51.723722+0000","last_peered":"2024-09-16T05:15:51.723722+0000","last_clean":"2024-09-16T05:15:51.723722+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:15:51.723722+0000","last_undegraded":"2024-09-16T05:15:51.723722+0000","last_fullsized":"2024-09-16T05:15:51.723722+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:46.792111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.792111+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.166326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0091634339999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.32","version":"380'125","reported_seq":697,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.744289+0000","last_change":"2024-09-16T05:15:51.715561+0000","last_active":"2024-09-16T05:15:51.744289+0000","last_peered":"2024-09-16T05:15:51.744289+0000","last_clean":"2024-09-16T05:15:51.744289+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:15:51.744289+0000","last_undegraded":"2024-09-16T05:15:51.744289+0000","last_fullsized":"2024-09-16T05:15:51.744289+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'101","last_scrub_stamp":"2024-09-16T05:14:49.915945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.915945+0000","objects_scrubbed":2,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:05.462174+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011862159999999999,"stat_sum":{"num_bytes":6857355,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":8515,"num_write":17,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.31","version":"402'164","reported_seq":1254,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.841469+0000","last_change":"2024-09-16T05:15:53.749530+0000","last_active":"2024-09-16T05:15:54.841469+0000","last_peered":"2024-09-16T05:15:54.841469+0000","last_clean":"2024-09-16T05:15:54.841469+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:15:54.841469+0000","last_undegraded":"2024-09-16T05:15:54.841469+0000","last_fullsized":"2024-09-16T05:15:54.841469+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"402'164","last_scrub_stamp":"2024-09-16T05:15:53.749444+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.749444+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:55.841387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031565529999999999,"stat_sum":{"num_bytes":4394329,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":49186,"num_write":13,"num_write_kb":4010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.30","version":"182'16","reported_seq":570,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.713553+0000","last_change":"2024-09-16T05:15:49.816792+0000","last_active":"2024-09-16T05:15:50.713553+0000","last_peered":"2024-09-16T05:15:50.713553+0000","last_clean":"2024-09-16T05:15:50.713553+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:15:50.713553+0000","last_undegraded":"2024-09-16T05:15:50.713553+0000","last_fullsized":"2024-09-16T05:15:50.713553+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:06.783241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.783241+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:31.004091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00059035299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.12","version":"35'139","reported_seq":406,"reported_epoch":405,"state":"active+remapped","last_fresh":"2024-09-16T05:15:54.870477+0000","last_change":"2024-09-16T05:15:54.870179+0000","last_active":"2024-09-16T05:15:54.870477+0000","last_peered":"2024-09-16T05:15:54.870477+0000","last_clean":"2024-09-16T05:10:33.944870+0000","last_became_active":"2024-09-16T05:15:54.870179+0000","last_became_peered":"2024-09-16T05:15:54.870179+0000","last_unstale":"2024-09-16T05:15:54.870477+0000","last_undegraded":"2024-09-16T05:15:54.870477+0000","last_fullsized":"2024-09-16T05:15:54.870477+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:07.583346+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:07.583346+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[0,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1152,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.712275+0000","last_change":"2024-09-16T05:15:51.712732+0000","last_active":"2024-09-16T05:15:51.712275+0000","last_peered":"2024-09-16T05:15:51.712275+0000","last_clean":"2024-09-16T05:15:51.712275+0000","last_became_active":"2024-09-16T05:13:25.679267+0000","last_became_peered":"2024-09-16T05:13:25.679267+0000","last_unstale":"2024-09-16T05:15:51.712275+0000","last_undegraded":"2024-09-16T05:15:51.712275+0000","last_fullsized":"2024-09-16T05:15:51.712275+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:14:31.050468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.050468+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:54.584484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067983599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T05:15:50.706294+0000","last_change":"2024-09-16T05:15:50.706294+0000","last_active":"2024-09-16T05:15:50.706294+0000","last_peered":"2024-09-16T05:15:50.706294+0000","last_clean":"2024-09-16T05:15:50.706294+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T05:15:50.706294+0000","last_undegraded":"2024-09-16T05:15:50.706294+0000","last_fullsized":"2024-09-16T05:15:50.706294+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:50.706294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:15:50.706294+0000","last_clean_scrub_stamp":"2024-09-16T05:15:50.706294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":1070,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-16T05:15:53.837824+0000","last_change":"2024-09-16T05:15:51.712104+0000","last_active":"2024-09-16T05:15:53.837824+0000","last_peered":"2024-09-16T05:15:53.837824+0000","last_clean":"2024-09-16T05:15:53.837824+0000","last_became_active":"2024-09-16T05:13:25.680445+0000","last_became_peered":"2024-09-16T05:13:25.680445+0000","last_unstale":"2024-09-16T05:15:53.837824+0000","last_undegraded":"2024-09-16T05:15:53.837824+0000","last_fullsized":"2024-09-16T05:15:53.837824+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:07.531934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.531934+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.628146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061796699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.c","version":"35'139","reported_seq":351,"reported_epoch":404,"state":"remapped+peering","last_fresh":"2024-09-16T05:15:53.837449+0000","last_change":"2024-09-16T05:15:53.837449+0000","last_active":"2024-09-16T05:09:26.435455+0000","last_peered":"2024-09-16T05:09:24.527530+0000","last_clean":"2024-09-16T05:09:24.527530+0000","last_became_active":"2024-09-16T05:08:11.009336+0000","last_became_peered":"2024-09-16T05:08:11.009336+0000","last_unstale":"2024-09-16T05:15:53.837449+0000","last_undegraded":"2024-09-16T05:15:53.837449+0000","last_fullsized":"2024-09-16T05:15:53.837449+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:10.945793+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.945793+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":6,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"360'281","reported_seq":1734,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-16T05:15:53.837276+0000","last_change":"2024-09-16T05:15:51.712777+0000","last_active":"2024-09-16T05:15:53.837276+0000","last_peered":"2024-09-16T05:15:53.837276+0000","last_clean":"2024-09-16T05:15:53.837276+0000","last_became_active":"2024-09-16T05:13:25.680481+0000","last_became_peered":"2024-09-16T05:13:25.680481+0000","last_unstale":"2024-09-16T05:15:53.837276+0000","last_undegraded":"2024-09-16T05:15:53.837276+0000","last_fullsized":"2024-09-16T05:15:53.837276+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:50.580296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.580296+0000","objects_scrubbed":3,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:55.034512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094903100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.d","version":"35'139","reported_seq":764,"reported_epoch":400,"state":"unknown","last_fresh":"2024-09-16T05:15:49.816484+0000","last_change":"2024-09-16T05:15:50.986287+0000","last_active":"2024-09-16T05:15:50.986287+0000","last_peered":"2024-09-16T05:15:49.816484+0000","last_clean":"2024-09-16T05:15:49.816484+0000","last_became_active":"2024-09-16T05:14:39.908958+0000","last_became_peered":"2024-09-16T05:14:39.908958+0000","last_unstale":"2024-09-16T05:15:49.816484+0000","last_undegraded":"2024-09-16T05:15:49.816484+0000","last_fullsized":"2024-09-16T05:15:49.816484+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:45.491884+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.491884+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":1586,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.833976+0000","last_change":"2024-09-16T05:15:51.720310+0000","last_active":"2024-09-16T05:15:52.833976+0000","last_peered":"2024-09-16T05:15:52.833976+0000","last_clean":"2024-09-16T05:15:52.833976+0000","last_became_active":"2024-09-16T05:14:40.261276+0000","last_became_peered":"2024-09-16T05:14:40.261276+0000","last_unstale":"2024-09-16T05:15:52.833976+0000","last_undegraded":"2024-09-16T05:15:52.833976+0000","last_fullsized":"2024-09-16T05:15:52.833976+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:14:47.777342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.777342+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:03.322150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075732600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.e","version":"35'139","reported_seq":515,"reported_epoch":405,"state":"active+remapped","last_fresh":"2024-09-16T05:15:54.843676+0000","last_change":"2024-09-16T05:15:54.843512+0000","last_active":"2024-09-16T05:15:54.843676+0000","last_peered":"2024-09-16T05:15:54.843676+0000","last_clean":"2024-09-16T05:13:29.734717+0000","last_became_active":"2024-09-16T05:15:54.843512+0000","last_became_peered":"2024-09-16T05:15:54.843512+0000","last_unstale":"2024-09-16T05:15:54.843676+0000","last_undegraded":"2024-09-16T05:15:54.843676+0000","last_fullsized":"2024-09-16T05:15:54.843676+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:11.724301+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:11.724301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[9,4],"avail_no_missing":["9","4","7","10"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.d","version":"402'457","reported_seq":1907,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.841393+0000","last_change":"2024-09-16T05:15:51.728780+0000","last_active":"2024-09-16T05:15:54.841393+0000","last_peered":"2024-09-16T05:15:54.841393+0000","last_clean":"2024-09-16T05:15:54.841393+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:15:54.841393+0000","last_undegraded":"2024-09-16T05:15:54.841393+0000","last_fullsized":"2024-09-16T05:15:54.841393+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'379","last_scrub_stamp":"2024-09-16T05:14:44.180613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.180613+0000","objects_scrubbed":6,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011194325,"stat_sum":{"num_bytes":9462506,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":48192,"num_write":30,"num_write_kb":9875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.f","version":"35'139","reported_seq":844,"reported_epoch":400,"state":"unknown","last_fresh":"2024-09-16T05:15:49.816715+0000","last_change":"2024-09-16T05:15:50.986311+0000","last_active":"2024-09-16T05:15:50.986311+0000","last_peered":"2024-09-16T05:15:49.816715+0000","last_clean":"2024-09-16T05:15:49.816715+0000","last_became_active":"2024-09-16T05:13:34.764109+0000","last_became_peered":"2024-09-16T05:13:34.764109+0000","last_unstale":"2024-09-16T05:15:49.816715+0000","last_undegraded":"2024-09-16T05:15:49.816715+0000","last_fullsized":"2024-09-16T05:15:49.816715+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:28.551941+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.551941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.c","version":"402'259","reported_seq":1696,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.302400+0000","last_change":"2024-09-16T05:15:51.716117+0000","last_active":"2024-09-16T05:15:52.302400+0000","last_peered":"2024-09-16T05:15:52.302400+0000","last_clean":"2024-09-16T05:15:52.302400+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:15:52.302400+0000","last_undegraded":"2024-09-16T05:15:52.302400+0000","last_fullsized":"2024-09-16T05:15:52.302400+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'213","last_scrub_stamp":"2024-09-16T05:14:26.945487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.945487+0000","objects_scrubbed":4,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.841446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012217739999999999,"stat_sum":{"num_bytes":3529536,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":33267,"num_write":16,"num_write_kb":6988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T05:15:50.706294+0000","last_change":"2024-09-16T05:15:50.706294+0000","last_active":"2024-09-16T05:15:50.706294+0000","last_peered":"2024-09-16T05:15:50.706294+0000","last_clean":"2024-09-16T05:15:50.706294+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T05:15:50.706294+0000","last_undegraded":"2024-09-16T05:15:50.706294+0000","last_fullsized":"2024-09-16T05:15:50.706294+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:50.706294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:15:50.706294+0000","last_clean_scrub_stamp":"2024-09-16T05:15:50.706294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"402'211","reported_seq":1653,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.742581+0000","last_change":"2024-09-16T05:15:51.724355+0000","last_active":"2024-09-16T05:15:51.742581+0000","last_peered":"2024-09-16T05:15:51.742581+0000","last_clean":"2024-09-16T05:15:51.742581+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:15:51.742581+0000","last_undegraded":"2024-09-16T05:15:51.742581+0000","last_fullsized":"2024-09-16T05:15:51.742581+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'153","last_scrub_stamp":"2024-09-16T05:14:26.593177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.593177+0000","objects_scrubbed":3,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010564619000000001,"stat_sum":{"num_bytes":2790028,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":12879,"num_write":10,"num_write_kb":2147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.3","version":"397'99","reported_seq":1529,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.709699+0000","last_change":"2024-09-16T05:15:49.830609+0000","last_active":"2024-09-16T05:15:50.709699+0000","last_peered":"2024-09-16T05:15:50.709699+0000","last_clean":"2024-09-16T05:15:50.709699+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:15:50.709699+0000","last_undegraded":"2024-09-16T05:15:50.709699+0000","last_fullsized":"2024-09-16T05:15:50.709699+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'68","last_scrub_stamp":"2024-09-16T05:14:44.975873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.975873+0000","objects_scrubbed":4,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:59.760168+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0074023240000000001,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":1026,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.718881+0000","last_change":"2024-09-16T05:15:51.718133+0000","last_active":"2024-09-16T05:15:51.718881+0000","last_peered":"2024-09-16T05:15:51.718881+0000","last_clean":"2024-09-16T05:15:51.718881+0000","last_became_active":"2024-09-16T05:15:51.717864+0000","last_became_peered":"2024-09-16T05:15:51.717864+0000","last_unstale":"2024-09-16T05:15:51.718881+0000","last_undegraded":"2024-09-16T05:15:51.718881+0000","last_fullsized":"2024-09-16T05:15:51.718881+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:53.849576+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.849576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1443,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.709384+0000","last_change":"2024-09-16T05:15:49.823400+0000","last_active":"2024-09-16T05:15:50.709384+0000","last_peered":"2024-09-16T05:15:50.709384+0000","last_clean":"2024-09-16T05:15:50.709384+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:15:50.709384+0000","last_undegraded":"2024-09-16T05:15:50.709384+0000","last_fullsized":"2024-09-16T05:15:50.709384+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:14:25.970665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.970665+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039957300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":659,"reported_epoch":400,"state":"peering","last_fresh":"2024-09-16T05:15:49.816221+0000","last_change":"2024-09-16T05:15:50.714327+0000","last_active":"2024-09-16T05:15:50.714327+0000","last_peered":"2024-09-16T05:15:49.816221+0000","last_clean":"2024-09-16T05:15:49.816221+0000","last_became_active":"2024-09-16T05:13:34.762556+0000","last_became_peered":"2024-09-16T05:13:34.762556+0000","last_unstale":"2024-09-16T05:15:49.816221+0000","last_undegraded":"2024-09-16T05:15:49.816221+0000","last_fullsized":"2024-09-16T05:15:49.816221+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:18.812352+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:18.812352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1021,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.717203+0000","last_change":"2024-09-16T05:15:51.717203+0000","last_active":"2024-09-16T05:15:51.717203+0000","last_peered":"2024-09-16T05:15:51.717203+0000","last_clean":"2024-09-16T05:15:51.717203+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:15:51.717203+0000","last_undegraded":"2024-09-16T05:15:51.717203+0000","last_fullsized":"2024-09-16T05:15:51.717203+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:07.583346+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:07.583346+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1406,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.833161+0000","last_change":"2024-09-16T05:15:51.720110+0000","last_active":"2024-09-16T05:15:52.833161+0000","last_peered":"2024-09-16T05:15:52.833161+0000","last_clean":"2024-09-16T05:15:52.833161+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:15:52.833161+0000","last_undegraded":"2024-09-16T05:15:52.833161+0000","last_fullsized":"2024-09-16T05:15:52.833161+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:07.898675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.898675+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:14.269502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053619300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.3","version":"35'139","reported_seq":1043,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.844945+0000","last_change":"2024-09-16T05:15:52.844991+0000","last_active":"2024-09-16T05:15:52.844945+0000","last_peered":"2024-09-16T05:15:52.844945+0000","last_clean":"2024-09-16T05:15:52.844945+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:15:52.844945+0000","last_undegraded":"2024-09-16T05:15:52.844945+0000","last_fullsized":"2024-09-16T05:15:52.844945+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:52.844849+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:52.844849+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:56.124669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1478,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.722237+0000","last_change":"2024-09-16T05:15:51.722461+0000","last_active":"2024-09-16T05:15:51.722237+0000","last_peered":"2024-09-16T05:15:51.722237+0000","last_clean":"2024-09-16T05:15:51.722237+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:15:51.722237+0000","last_undegraded":"2024-09-16T05:15:51.722237+0000","last_fullsized":"2024-09-16T05:15:51.722237+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:06.755407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.755407+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:09.722029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085143739999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":709,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.841198+0000","last_change":"2024-09-16T05:15:51.730548+0000","last_active":"2024-09-16T05:15:54.841198+0000","last_peered":"2024-09-16T05:15:54.841198+0000","last_clean":"2024-09-16T05:15:54.841198+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:15:54.841198+0000","last_undegraded":"2024-09-16T05:15:54.841198+0000","last_fullsized":"2024-09-16T05:15:54.841198+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:31.009189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.009189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:46.423984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00982653,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.4","version":"35'139","reported_seq":988,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.723574+0000","last_change":"2024-09-16T05:15:51.723574+0000","last_active":"2024-09-16T05:15:51.723574+0000","last_peered":"2024-09-16T05:15:51.723574+0000","last_clean":"2024-09-16T05:15:51.723574+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:15:51.723574+0000","last_undegraded":"2024-09-16T05:15:51.723574+0000","last_fullsized":"2024-09-16T05:15:51.723574+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:10.945793+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.945793+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1467,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869837+0000","last_change":"2024-09-16T05:15:51.723041+0000","last_active":"2024-09-16T05:15:54.869837+0000","last_peered":"2024-09-16T05:15:54.869837+0000","last_clean":"2024-09-16T05:15:54.869837+0000","last_became_active":"2024-09-16T05:14:25.821885+0000","last_became_peered":"2024-09-16T05:14:25.821885+0000","last_unstale":"2024-09-16T05:15:54.869837+0000","last_undegraded":"2024-09-16T05:15:54.869837+0000","last_fullsized":"2024-09-16T05:15:54.869837+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:15:35.928579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:35.928579+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:54.845367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015242610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":764,"reported_epoch":400,"state":"peering","last_fresh":"2024-09-16T05:15:49.816484+0000","last_change":"2024-09-16T05:15:50.715161+0000","last_active":"2024-09-16T05:15:50.715161+0000","last_peered":"2024-09-16T05:15:49.816484+0000","last_clean":"2024-09-16T05:15:49.816484+0000","last_became_active":"2024-09-16T05:14:39.908958+0000","last_became_peered":"2024-09-16T05:14:39.908958+0000","last_unstale":"2024-09-16T05:15:49.816484+0000","last_undegraded":"2024-09-16T05:15:49.816484+0000","last_fullsized":"2024-09-16T05:15:49.816484+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:45.491884+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.491884+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"402'479","reported_seq":1915,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.841190+0000","last_change":"2024-09-16T05:15:51.724291+0000","last_active":"2024-09-16T05:15:54.841190+0000","last_peered":"2024-09-16T05:15:54.841190+0000","last_clean":"2024-09-16T05:15:54.841190+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:15:54.841190+0000","last_undegraded":"2024-09-16T05:15:54.841190+0000","last_fullsized":"2024-09-16T05:15:54.841190+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:14:26.230040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:26.230040+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0071947540000000003,"stat_sum":{"num_bytes":3176556,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":8850,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":660,"reported_epoch":400,"state":"peering","last_fresh":"2024-09-16T05:15:49.816412+0000","last_change":"2024-09-16T05:15:50.714375+0000","last_active":"2024-09-16T05:15:50.714375+0000","last_peered":"2024-09-16T05:15:49.816412+0000","last_clean":"2024-09-16T05:15:49.816412+0000","last_became_active":"2024-09-16T05:13:34.764187+0000","last_became_peered":"2024-09-16T05:13:34.764187+0000","last_unstale":"2024-09-16T05:15:49.816412+0000","last_undegraded":"2024-09-16T05:15:49.816412+0000","last_fullsized":"2024-09-16T05:15:49.816412+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:11.724301+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:11.724301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1541,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869753+0000","last_change":"2024-09-16T05:15:51.721181+0000","last_active":"2024-09-16T05:15:54.869753+0000","last_peered":"2024-09-16T05:15:54.869753+0000","last_clean":"2024-09-16T05:15:54.869753+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:15:54.869753+0000","last_undegraded":"2024-09-16T05:15:54.869753+0000","last_fullsized":"2024-09-16T05:15:54.869753+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:28.350521+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.350521+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:55.068764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075293500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":844,"reported_epoch":400,"state":"peering","last_fresh":"2024-09-16T05:15:49.816715+0000","last_change":"2024-09-16T05:15:50.713715+0000","last_active":"2024-09-16T05:15:50.713715+0000","last_peered":"2024-09-16T05:15:49.816715+0000","last_clean":"2024-09-16T05:15:49.816715+0000","last_became_active":"2024-09-16T05:13:34.764109+0000","last_became_peered":"2024-09-16T05:13:34.764109+0000","last_unstale":"2024-09-16T05:15:49.816715+0000","last_undegraded":"2024-09-16T05:15:49.816715+0000","last_fullsized":"2024-09-16T05:15:49.816715+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:28.551941+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.551941+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":1637,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-16T05:15:53.838380+0000","last_change":"2024-09-16T05:15:51.712062+0000","last_active":"2024-09-16T05:15:53.838380+0000","last_peered":"2024-09-16T05:15:53.838380+0000","last_clean":"2024-09-16T05:15:53.838380+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:15:53.838380+0000","last_undegraded":"2024-09-16T05:15:53.838380+0000","last_fullsized":"2024-09-16T05:15:53.838380+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:15:41.633260+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:55.255687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051473199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":1025,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.709302+0000","last_change":"2024-09-16T05:15:15.084154+0000","last_active":"2024-09-16T05:15:50.709302+0000","last_peered":"2024-09-16T05:15:50.709302+0000","last_clean":"2024-09-16T05:15:50.709302+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:15:50.709302+0000","last_undegraded":"2024-09-16T05:15:50.709302+0000","last_fullsized":"2024-09-16T05:15:50.709302+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:15.084022+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.084022+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:43.880138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"402'463","reported_seq":1916,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.741011+0000","last_change":"2024-09-16T05:15:51.718858+0000","last_active":"2024-09-16T05:15:51.741011+0000","last_peered":"2024-09-16T05:15:51.741011+0000","last_clean":"2024-09-16T05:15:51.741011+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:15:51.741011+0000","last_undegraded":"2024-09-16T05:15:51.741011+0000","last_fullsized":"2024-09-16T05:15:51.741011+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'419","last_scrub_stamp":"2024-09-16T05:14:25.924121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.924121+0000","objects_scrubbed":4,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037856729999999998,"stat_sum":{"num_bytes":5126867,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":30338,"num_write":5,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.9","version":"35'139","reported_seq":1017,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.713502+0000","last_change":"2024-09-16T05:14:11.602321+0000","last_active":"2024-09-16T05:15:51.713502+0000","last_peered":"2024-09-16T05:15:51.713502+0000","last_clean":"2024-09-16T05:15:51.713502+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:15:51.713502+0000","last_undegraded":"2024-09-16T05:15:51.713502+0000","last_fullsized":"2024-09-16T05:15:51.713502+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:11.602237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:11.602237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:56.296269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1686,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.869526+0000","last_change":"2024-09-16T05:15:51.719848+0000","last_active":"2024-09-16T05:15:54.869526+0000","last_peered":"2024-09-16T05:15:54.869526+0000","last_clean":"2024-09-16T05:15:54.869526+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:15:54.869526+0000","last_undegraded":"2024-09-16T05:15:54.869526+0000","last_fullsized":"2024-09-16T05:15:54.869526+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:15:32.922163+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:32.922163+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:46.157941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049799200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":1034,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.840699+0000","last_change":"2024-09-16T05:15:14.182510+0000","last_active":"2024-09-16T05:15:54.840699+0000","last_peered":"2024-09-16T05:15:54.840699+0000","last_clean":"2024-09-16T05:15:54.840699+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:15:54.840699+0000","last_undegraded":"2024-09-16T05:15:54.840699+0000","last_fullsized":"2024-09-16T05:15:54.840699+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:14.182407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:14.182407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:39.386863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":514,"reported_epoch":405,"state":"active+remapped","last_fresh":"2024-09-16T05:15:54.843247+0000","last_change":"2024-09-16T05:15:54.842942+0000","last_active":"2024-09-16T05:15:54.843247+0000","last_peered":"2024-09-16T05:15:54.843247+0000","last_clean":"2024-09-16T05:13:29.734159+0000","last_became_active":"2024-09-16T05:15:54.842942+0000","last_became_peered":"2024-09-16T05:15:54.842942+0000","last_unstale":"2024-09-16T05:15:54.843247+0000","last_undegraded":"2024-09-16T05:15:54.843247+0000","last_fullsized":"2024-09-16T05:15:54.843247+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:18.812352+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:18.812352+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[9,0],"avail_no_missing":["9","0","8","10"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":570,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.713925+0000","last_change":"2024-09-16T05:15:49.817859+0000","last_active":"2024-09-16T05:15:50.713925+0000","last_peered":"2024-09-16T05:15:50.713925+0000","last_clean":"2024-09-16T05:15:50.713925+0000","last_became_active":"2024-09-16T05:13:31.747644+0000","last_became_peered":"2024-09-16T05:13:31.747644+0000","last_unstale":"2024-09-16T05:15:50.713925+0000","last_undegraded":"2024-09-16T05:15:50.713925+0000","last_fullsized":"2024-09-16T05:15:50.713925+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:14:49.576522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.576522+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.963571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001270452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.10","version":"35'139","reported_seq":1025,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.718901+0000","last_change":"2024-09-16T05:15:51.718183+0000","last_active":"2024-09-16T05:15:51.718901+0000","last_peered":"2024-09-16T05:15:51.718901+0000","last_clean":"2024-09-16T05:15:51.718901+0000","last_became_active":"2024-09-16T05:15:51.717898+0000","last_became_peered":"2024-09-16T05:15:51.717898+0000","last_unstale":"2024-09-16T05:15:51.718901+0000","last_undegraded":"2024-09-16T05:15:51.718901+0000","last_fullsized":"2024-09-16T05:15:51.718901+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:53.849576+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.849576+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":1079,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.840939+0000","last_change":"2024-09-16T05:15:51.729768+0000","last_active":"2024-09-16T05:15:54.840939+0000","last_peered":"2024-09-16T05:15:54.840939+0000","last_clean":"2024-09-16T05:15:54.840939+0000","last_became_active":"2024-09-16T05:13:25.681297+0000","last_became_peered":"2024-09-16T05:13:25.681297+0000","last_unstale":"2024-09-16T05:15:54.840939+0000","last_undegraded":"2024-09-16T05:15:54.840939+0000","last_fullsized":"2024-09-16T05:15:54.840939+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:49.160833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.160833+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:00.434950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0093081610000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.14","version":"402'269","reported_seq":1322,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.840997+0000","last_change":"2024-09-16T05:15:51.730207+0000","last_active":"2024-09-16T05:15:54.840997+0000","last_peered":"2024-09-16T05:15:54.840997+0000","last_clean":"2024-09-16T05:15:54.840997+0000","last_became_active":"2024-09-16T05:13:25.684888+0000","last_became_peered":"2024-09-16T05:13:25.684888+0000","last_unstale":"2024-09-16T05:15:54.840997+0000","last_undegraded":"2024-09-16T05:15:54.840997+0000","last_fullsized":"2024-09-16T05:15:54.840997+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"342'239","last_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.110587+0000","objects_scrubbed":4,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.292019+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.012326337999999999,"stat_sum":{"num_bytes":3142942,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":26624,"num_write":7,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.15","version":"402'475","reported_seq":1933,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.841191+0000","last_change":"2024-09-16T05:15:51.728495+0000","last_active":"2024-09-16T05:15:54.841191+0000","last_peered":"2024-09-16T05:15:54.841191+0000","last_clean":"2024-09-16T05:15:54.841191+0000","last_became_active":"2024-09-16T05:13:25.680026+0000","last_became_peered":"2024-09-16T05:13:25.680026+0000","last_unstale":"2024-09-16T05:15:54.841191+0000","last_undegraded":"2024-09-16T05:15:54.841191+0000","last_fullsized":"2024-09-16T05:15:54.841191+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"357'436","last_scrub_stamp":"2024-09-16T05:15:07.101221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.101221+0000","objects_scrubbed":5,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:34.876771+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.011143002000000001,"stat_sum":{"num_bytes":9168119,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":52138,"num_write":31,"num_write_kb":15087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.16","version":"390'120","reported_seq":1200,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.709431+0000","last_change":"2024-09-16T05:15:49.823739+0000","last_active":"2024-09-16T05:15:50.709431+0000","last_peered":"2024-09-16T05:15:50.709431+0000","last_clean":"2024-09-16T05:15:50.709431+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:15:50.709431+0000","last_undegraded":"2024-09-16T05:15:50.709431+0000","last_fullsized":"2024-09-16T05:15:50.709431+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'120","last_scrub_stamp":"2024-09-16T05:15:42.209530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:42.209530+0000","objects_scrubbed":2,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:44.989655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068487400000000003,"stat_sum":{"num_bytes":3142942,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":12296,"num_write":5,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"1.14","version":"35'139","reported_seq":351,"reported_epoch":404,"state":"remapped+peering","last_fresh":"2024-09-16T05:15:53.838040+0000","last_change":"2024-09-16T05:15:53.838040+0000","last_active":"2024-09-16T05:09:26.435455+0000","last_peered":"2024-09-16T05:09:24.527530+0000","last_clean":"2024-09-16T05:09:24.527530+0000","last_became_active":"2024-09-16T05:08:11.009336+0000","last_became_peered":"2024-09-16T05:08:11.009336+0000","last_unstale":"2024-09-16T05:15:53.838040+0000","last_undegraded":"2024-09-16T05:15:53.838040+0000","last_fullsized":"2024-09-16T05:15:53.838040+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:10.945793+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.945793+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1246,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.712792+0000","last_change":"2024-09-16T05:15:51.713142+0000","last_active":"2024-09-16T05:15:51.712792+0000","last_peered":"2024-09-16T05:15:51.712792+0000","last_clean":"2024-09-16T05:15:51.712792+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:15:51.712792+0000","last_undegraded":"2024-09-16T05:15:51.712792+0000","last_fullsized":"2024-09-16T05:15:51.712792+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:43.982972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:43.982972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:10.289120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051354799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.18","version":"181'57","reported_seq":1472,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.840908+0000","last_change":"2024-09-16T05:15:51.730960+0000","last_active":"2024-09-16T05:15:54.840908+0000","last_peered":"2024-09-16T05:15:54.840908+0000","last_clean":"2024-09-16T05:15:54.840908+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:15:54.840908+0000","last_undegraded":"2024-09-16T05:15:54.840908+0000","last_fullsized":"2024-09-16T05:15:54.840908+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:17.188273+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:17.188273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:30.233132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0099769660000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.19","version":"182'60","reported_seq":1067,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.830232+0000","last_change":"2024-09-16T05:15:51.711014+0000","last_active":"2024-09-16T05:15:52.830232+0000","last_peered":"2024-09-16T05:15:52.830232+0000","last_clean":"2024-09-16T05:15:52.830232+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:15:52.830232+0000","last_undegraded":"2024-09-16T05:15:52.830232+0000","last_fullsized":"2024-09-16T05:15:52.830232+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:06.893298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.893298+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:26.500564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052728300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.1a","version":"402'626","reported_seq":2048,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-16T05:15:53.838702+0000","last_change":"2024-09-16T05:15:51.719142+0000","last_active":"2024-09-16T05:15:53.838702+0000","last_peered":"2024-09-16T05:15:53.838702+0000","last_clean":"2024-09-16T05:15:53.838702+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:15:53.838702+0000","last_undegraded":"2024-09-16T05:15:53.838702+0000","last_fullsized":"2024-09-16T05:15:53.838702+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'527","last_scrub_stamp":"2024-09-16T05:15:10.511970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.511970+0000","objects_scrubbed":9,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.999575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.007004634,"stat_sum":{"num_bytes":12959641,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":64,"num_read_kb":57716,"num_write":40,"num_write_kb":18977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.1b","version":"402'335","reported_seq":1808,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.718161+0000","last_change":"2024-09-16T05:15:51.718161+0000","last_active":"2024-09-16T05:15:51.718161+0000","last_peered":"2024-09-16T05:15:51.718161+0000","last_clean":"2024-09-16T05:15:51.718161+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:15:51.718161+0000","last_undegraded":"2024-09-16T05:15:51.718161+0000","last_fullsized":"2024-09-16T05:15:51.718161+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"347'319","last_scrub_stamp":"2024-09-16T05:15:08.957118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.957118+0000","objects_scrubbed":3,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:37.900333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0043048890000000001,"stat_sum":{"num_bytes":5298036,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":29313,"num_write":9,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.1c","version":"382'373","reported_seq":1871,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.713553+0000","last_change":"2024-09-16T05:15:49.817009+0000","last_active":"2024-09-16T05:15:50.713553+0000","last_peered":"2024-09-16T05:15:50.713553+0000","last_clean":"2024-09-16T05:15:50.713553+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:15:50.713553+0000","last_undegraded":"2024-09-16T05:15:50.713553+0000","last_fullsized":"2024-09-16T05:15:50.713553+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'364","last_scrub_stamp":"2024-09-16T05:15:08.759106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.759106+0000","objects_scrubbed":2,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:10.573553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078031499999999996,"stat_sum":{"num_bytes":3327852,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.1d","version":"400'640","reported_seq":2160,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.714055+0000","last_change":"2024-09-16T05:15:49.819670+0000","last_active":"2024-09-16T05:15:50.714055+0000","last_peered":"2024-09-16T05:15:50.714055+0000","last_clean":"2024-09-16T05:15:50.714055+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:15:50.714055+0000","last_undegraded":"2024-09-16T05:15:50.714055+0000","last_fullsized":"2024-09-16T05:15:50.714055+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'584","last_scrub_stamp":"2024-09-16T05:15:16.833446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:16.833446+0000","objects_scrubbed":8,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:42.545124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032802339999999999,"stat_sum":{"num_bytes":12407718,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":46,"num_read_kb":41644,"num_write":32,"num_write_kb":17614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.1e","version":"378'347","reported_seq":1814,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.830594+0000","last_change":"2024-09-16T05:15:51.712046+0000","last_active":"2024-09-16T05:15:52.830594+0000","last_peered":"2024-09-16T05:15:52.830594+0000","last_clean":"2024-09-16T05:15:52.830594+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:15:52.830594+0000","last_undegraded":"2024-09-16T05:15:52.830594+0000","last_fullsized":"2024-09-16T05:15:52.830594+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"362'335","last_scrub_stamp":"2024-09-16T05:15:12.926534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:12.926534+0000","objects_scrubbed":3,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.929015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053361800000000003,"stat_sum":{"num_bytes":2487502,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":26093,"num_write":5,"num_write_kb":3828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.1f","version":"284'221","reported_seq":1687,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.841469+0000","last_change":"2024-09-16T05:15:51.716468+0000","last_active":"2024-09-16T05:15:54.841469+0000","last_peered":"2024-09-16T05:15:54.841469+0000","last_clean":"2024-09-16T05:15:54.841469+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:15:54.841469+0000","last_undegraded":"2024-09-16T05:15:54.841469+0000","last_fullsized":"2024-09-16T05:15:54.841469+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:14:52.883562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.883562+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:59.388665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050306899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.20","version":"402'136","reported_seq":1605,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.718585+0000","last_change":"2024-09-16T05:15:51.718833+0000","last_active":"2024-09-16T05:15:51.718585+0000","last_peered":"2024-09-16T05:15:51.718585+0000","last_clean":"2024-09-16T05:15:51.718585+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:15:51.718585+0000","last_undegraded":"2024-09-16T05:15:51.718585+0000","last_fullsized":"2024-09-16T05:15:51.718585+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'128","last_scrub_stamp":"2024-09-16T05:15:31.623695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.623695+0000","objects_scrubbed":2,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:38.383882+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0052779159999999997,"stat_sum":{"num_bytes":4238337,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":79406,"num_write":16,"num_write_kb":6808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.21","version":"178'50","reported_seq":1405,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.833343+0000","last_change":"2024-09-16T05:15:51.720530+0000","last_active":"2024-09-16T05:15:52.833343+0000","last_peered":"2024-09-16T05:15:52.833343+0000","last_clean":"2024-09-16T05:15:52.833343+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:15:52.833343+0000","last_undegraded":"2024-09-16T05:15:52.833343+0000","last_fullsized":"2024-09-16T05:15:52.833343+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.848788+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:03.297171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094848999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.22","version":"182'20","reported_seq":578,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-09-16T05:15:50.714333+0000","last_change":"2024-09-16T05:15:49.818091+0000","last_active":"2024-09-16T05:15:50.714333+0000","last_peered":"2024-09-16T05:15:50.714333+0000","last_clean":"2024-09-16T05:15:50.714333+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:15:50.714333+0000","last_undegraded":"2024-09-16T05:15:50.714333+0000","last_fullsized":"2024-09-16T05:15:50.714333+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:31.582158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.582158+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:33.433811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00134073,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":934,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.832352+0000","last_change":"2024-09-16T05:15:51.711696+0000","last_active":"2024-09-16T05:15:52.832352+0000","last_peered":"2024-09-16T05:15:52.832352+0000","last_clean":"2024-09-16T05:15:52.832352+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:15:52.832352+0000","last_undegraded":"2024-09-16T05:15:52.832352+0000","last_fullsized":"2024-09-16T05:15:52.832352+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:14:46.760890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.760890+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046105799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.24","version":"182'89","reported_seq":758,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.840960+0000","last_change":"2024-09-16T05:15:51.729002+0000","last_active":"2024-09-16T05:15:54.840960+0000","last_peered":"2024-09-16T05:15:54.840960+0000","last_clean":"2024-09-16T05:15:54.840960+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:15:54.840960+0000","last_undegraded":"2024-09-16T05:15:54.840960+0000","last_fullsized":"2024-09-16T05:15:54.840960+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:14:31.611624+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:31.611624+0000","last_clean_scrub_stamp":"2024-09-16T05:14:31.611624+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0.011328592,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.25","version":"403'636","reported_seq":2025,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-16T05:15:53.838084+0000","last_change":"2024-09-16T05:15:51.726027+0000","last_active":"2024-09-16T05:15:53.838084+0000","last_peered":"2024-09-16T05:15:53.838084+0000","last_clean":"2024-09-16T05:15:53.838084+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:15:53.838084+0000","last_undegraded":"2024-09-16T05:15:53.838084+0000","last_fullsized":"2024-09-16T05:15:53.838084+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'624","last_scrub_stamp":"2024-09-16T05:15:35.625076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:35.625076+0000","objects_scrubbed":6,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:50.398128+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.014100005000000001,"stat_sum":{"num_bytes":13563348,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":44,"num_read_kb":57574,"num_write":27,"num_write_kb":10249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.26","version":"116'9","reported_seq":742,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.723777+0000","last_change":"2024-09-16T05:15:51.724000+0000","last_active":"2024-09-16T05:15:51.723777+0000","last_peered":"2024-09-16T05:15:51.723777+0000","last_clean":"2024-09-16T05:15:51.723777+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:15:51.723777+0000","last_undegraded":"2024-09-16T05:15:51.723777+0000","last_fullsized":"2024-09-16T05:15:51.723777+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:27.631576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.631576+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086487479999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.27","version":"0'0","reported_seq":754,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.724710+0000","last_change":"2024-09-16T05:15:51.725178+0000","last_active":"2024-09-16T05:15:51.724710+0000","last_peered":"2024-09-16T05:15:51.724710+0000","last_clean":"2024-09-16T05:15:51.724710+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:15:51.724710+0000","last_undegraded":"2024-09-16T05:15:51.724710+0000","last_fullsized":"2024-09-16T05:15:51.724710+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:30.713398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.713398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097787770000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.28","version":"245'105","reported_seq":1535,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.830187+0000","last_change":"2024-09-16T05:15:51.710955+0000","last_active":"2024-09-16T05:15:52.830187+0000","last_peered":"2024-09-16T05:15:52.830187+0000","last_clean":"2024-09-16T05:15:52.830187+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:15:52.830187+0000","last_undegraded":"2024-09-16T05:15:52.830187+0000","last_fullsized":"2024-09-16T05:15:52.830187+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:14:45.747155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:45.747155+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.972546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047130300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.29","version":"402'543","reported_seq":2022,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.738379+0000","last_change":"2024-09-16T05:15:51.714783+0000","last_active":"2024-09-16T05:15:51.738379+0000","last_peered":"2024-09-16T05:15:51.738379+0000","last_clean":"2024-09-16T05:15:51.738379+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:15:51.738379+0000","last_undegraded":"2024-09-16T05:15:51.738379+0000","last_fullsized":"2024-09-16T05:15:51.738379+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'536","last_scrub_stamp":"2024-09-16T05:15:41.155545+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:41.155545+0000","objects_scrubbed":6,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:44.272166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025379259999999998,"stat_sum":{"num_bytes":16607117,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":25965,"num_write":73,"num_write_kb":22681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1744,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.723100+0000","last_change":"2024-09-16T05:15:51.723402+0000","last_active":"2024-09-16T05:15:51.723100+0000","last_peered":"2024-09-16T05:15:51.723100+0000","last_clean":"2024-09-16T05:15:51.723100+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:15:51.723100+0000","last_undegraded":"2024-09-16T05:15:51.723100+0000","last_fullsized":"2024-09-16T05:15:51.723100+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:14:33.725061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:33.725061+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:57.004354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0093349160000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.2b","version":"402'214","reported_seq":1576,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-09-16T05:15:52.833645+0000","last_change":"2024-09-16T05:15:51.723252+0000","last_active":"2024-09-16T05:15:52.833645+0000","last_peered":"2024-09-16T05:15:52.833645+0000","last_clean":"2024-09-16T05:15:52.833645+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:15:52.833645+0000","last_undegraded":"2024-09-16T05:15:52.833645+0000","last_fullsized":"2024-09-16T05:15:52.833645+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"330'198","last_scrub_stamp":"2024-09-16T05:14:46.815272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.815272+0000","objects_scrubbed":2,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:13.067028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027695710000000002,"stat_sum":{"num_bytes":4533234,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":13,"num_write_kb":8806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.2c","version":"241'194","reported_seq":1584,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-16T05:15:53.838310+0000","last_change":"2024-09-16T05:15:51.712455+0000","last_active":"2024-09-16T05:15:53.838310+0000","last_peered":"2024-09-16T05:15:53.838310+0000","last_clean":"2024-09-16T05:15:53.838310+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:15:53.838310+0000","last_undegraded":"2024-09-16T05:15:53.838310+0000","last_fullsized":"2024-09-16T05:15:53.838310+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:15:39.640149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.640149+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:59.257291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069016399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.2d","version":"377'326","reported_seq":1745,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.714937+0000","last_change":"2024-09-16T05:15:51.716403+0000","last_active":"2024-09-16T05:15:51.714937+0000","last_peered":"2024-09-16T05:15:51.714937+0000","last_clean":"2024-09-16T05:15:51.714937+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:15:51.714937+0000","last_undegraded":"2024-09-16T05:15:51.714937+0000","last_fullsized":"2024-09-16T05:15:51.714937+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'307","last_scrub_stamp":"2024-09-16T05:14:30.116429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.116429+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":444,"objects_trimmed":0,"snaptrim_duration":0.0027503990000000002,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":4589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.2e","version":"402'394","reported_seq":1808,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-09-16T05:15:51.725297+0000","last_change":"2024-09-16T05:15:51.725544+0000","last_active":"2024-09-16T05:15:51.725297+0000","last_peered":"2024-09-16T05:15:51.725297+0000","last_clean":"2024-09-16T05:15:51.725297+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:15:51.725297+0000","last_undegraded":"2024-09-16T05:15:51.725297+0000","last_fullsized":"2024-09-16T05:15:51.725297+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"337'278","last_scrub_stamp":"2024-09-16T05:14:47.811642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.811642+0000","objects_scrubbed":7,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:15:54.277049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010144469,"stat_sum":{"num_bytes":8393552,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":22140,"num_write":34,"num_write_kb":7817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]},{"pgid":"2.2f","version":"387'306","reported_seq":1797,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-16T05:15:54.841213+0000","last_change":"2024-09-16T05:15:51.730179+0000","last_active":"2024-09-16T05:15:54.841213+0000","last_peered":"2024-09-16T05:15:54.841213+0000","last_clean":"2024-09-16T05:15:54.841213+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:15:54.841213+0000","last_undegraded":"2024-09-16T05:15:54.841213+0000","last_fullsized":"2024-09-16T05:15:54.841213+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:53.110450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.110450+0000","objects_scrubbed":3,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:16.359037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0096762540000000005,"stat_sum":{"num_bytes":6521248,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":13580,"num_write":12,"num_write_kb":6438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"84","length":"1"},{"start":"86","length":"1"},{"start":"88","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":196255472,"num_objects":86,"num_object_clones":39,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":8,"num_read":796,"num_read_kb":825722,"num_write":526,"num_write_kb":221261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":242003968,"data_stored":290306716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15605,"ondisk_log_size":15605,"up":152,"acting":152,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":434529,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":12,"num_write":21,"num_write_kb":312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":192338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1454080,"data_stored":1446976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2641,"ondisk_log_size":2641,"up":38,"acting":38,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089425,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70176,"kb_used_data":30660,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93674976,"statfs":{"total":95995035648,"available":95923175424,"internally_reserved":0,"allocated":31395840,"data_stored":33023702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54624,"kb_used_data":19272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690528,"statfs":{"total":95995035648,"available":95939100672,"internally_reserved":0,"allocated":19734528,"data_stored":21080815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514591,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77936,"kb_used_data":38424,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93667216,"statfs":{"total":95995035648,"available":95915229184,"internally_reserved":0,"allocated":39346176,"data_stored":44422736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841631,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52412,"kb_used_data":17064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93692740,"statfs":{"total":95995035648,"available":95941365760,"internally_reserved":0,"allocated":17473536,"data_stored":20847353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103268,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72696,"kb_used_data":33184,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93672456,"statfs":{"total":95995035648,"available":95920594944,"internally_reserved":0,"allocated":33980416,"data_stored":40415067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397612,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70608,"kb_used_data":31092,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93674544,"statfs":{"total":95995035648,"available":95922733056,"internally_reserved":0,"allocated":31838208,"data_stored":39195088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757358,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52816,"kb_used_data":21624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93692336,"statfs":{"total":95995035648,"available":95940952064,"internally_reserved":0,"allocated":22142976,"data_stored":26106814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54992,"kb_used_data":19644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690160,"statfs":{"total":95995035648,"available":95938723840,"internally_reserved":0,"allocated":20115456,"data_stored":19687704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117113,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75052,"kb_used_data":35544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93670100,"statfs":{"total":95995035648,"available":95918182400,"internally_reserved":0,"allocated":36397056,"data_stored":38676790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240400,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44580,"kb_used_data":16460,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93700572,"statfs":{"total":95995035648,"available":95949385728,"internally_reserved":0,"allocated":16855040,"data_stored":17236729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574980,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53452,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93691700,"statfs":{"total":95995035648,"available":95940300800,"internally_reserved":0,"allocated":18534400,"data_stored":17533646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738505,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50656,"kb_used_data":11144,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93694496,"statfs":{"total":95995035648,"available":95943163904,"internally_reserved":0,"allocated":11411456,"data_stored":10513469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6586368,"data_stored":7412758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13709312,"data_stored":14432935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11788288,"data_stored":13891234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31584256,"data_stored":35585168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15339520,"data_stored":16622293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17162240,"data_stored":22840155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27037696,"data_stored":36112281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":29192192,"data_stored":35516363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12333056,"data_stored":17428051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":34508800,"data_stored":41313234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":14958592,"data_stored":18015404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":27803648,"data_stored":31136840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:15:56.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 21 2024-09-16T05:15:56.881 INFO:tasks.rados.rados.2.smithi204.stdout:1280: writing smithi20457400-1 from 1701083 to 2326908 tid 2 2024-09-16T05:15:56.883 INFO:tasks.rados.rados.2.smithi204.stdout:1280: writing smithi20457400-1 from 2956414 to 3615393 tid 3 2024-09-16T05:15:56.883 INFO:tasks.rados.rados.2.smithi204.stdout:1281: read oid 28 snap -1 2024-09-16T05:15:56.884 INFO:tasks.rados.rados.2.smithi204.stdout:1281: expect (ObjNum 412 snap 123 seq_num 412) 2024-09-16T05:15:56.884 INFO:tasks.rados.rados.2.smithi204.stdout:1282: write oid 41 current snap is 142 2024-09-16T05:15:56.884 INFO:tasks.rados.rados.2.smithi204.stdout:1278: finishing write tid 4 to smithi20457400-11 2024-09-16T05:15:56.884 INFO:tasks.rados.rados.2.smithi204.stdout:1278: finishing write tid 5 to smithi20457400-11 2024-09-16T05:15:56.884 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 644 (ObjNum 454 snap 142 seq_num 454) dirty exists 2024-09-16T05:15:56.884 INFO:tasks.rados.rados.2.smithi204.stdout:1278: left oid 11 (ObjNum 454 snap 142 seq_num 454) 2024-09-16T05:15:56.884 INFO:tasks.rados.rados.2.smithi204.stdout:1280: finishing write tid 1 to smithi20457400-1 2024-09-16T05:15:56.884 INFO:tasks.rados.rados.2.smithi204.stdout:1282: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-16T05:15:56.889 INFO:tasks.rados.rados.2.smithi204.stdout:1282: writing smithi20457400-41 from 480799 to 1050772 tid 1 2024-09-16T05:15:56.895 INFO:tasks.rados.rados.2.smithi204.stdout:1282: writing smithi20457400-41 from 1475918 to 2245401 tid 2 2024-09-16T05:15:56.902 INFO:tasks.rados.rados.2.smithi204.stdout:1282: writing smithi20457400-41 from 2700190 to 3286794 tid 3 2024-09-16T05:15:56.902 INFO:tasks.rados.rados.2.smithi204.stdout:1278: done (3 left) 2024-09-16T05:15:56.902 INFO:tasks.rados.rados.2.smithi204.stdout:1283: write oid 39 current snap is 142 2024-09-16T05:15:56.902 INFO:tasks.rados.rados.2.smithi204.stdout:1280: finishing write tid 2 to smithi20457400-1 2024-09-16T05:15:56.902 INFO:tasks.rados.rados.2.smithi204.stdout:1283: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-16T05:15:56.907 INFO:tasks.rados.rados.2.smithi204.stdout:1283: writing smithi20457400-39 from 497606 to 1142828 tid 1 2024-09-16T05:15:56.915 INFO:tasks.rados.rados.2.smithi204.stdout:1283: writing smithi20457400-39 from 1934400 to 2563894 tid 2 2024-09-16T05:15:56.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:56.922 INFO:tasks.rados.rados.2.smithi204.stdout:1283: writing smithi20457400-39 from 3243966 to 3697606 tid 3 2024-09-16T05:15:56.922 INFO:tasks.rados.rados.2.smithi204.stdout:1284: snap_remove snap 139 2024-09-16T05:15:56.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:56.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:57.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:15:57.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:15:57.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:15:57.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:15:57.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:15:57.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:15:57.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:57.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: osdmap e407: 12 total, 12 up, 12 in 2024-09-16T05:15:57.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:15:57.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:56 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:15:57.024 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:15:57.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:57.065 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:57.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:57.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:15:57.145 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:15:57.145 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:15:57.145 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:15:57.145 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:15:57.146 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:15:57.146 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:15:57.146 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:57.146 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: osdmap e407: 12 total, 12 up, 12 in 2024-09-16T05:15:57.146 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:15:57.146 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:56 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:15:57.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:15:57.191 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:57.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:15:57.198 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:15:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:15:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:15:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:15:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:15:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:15:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:15:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: osdmap e407: 12 total, 12 up, 12 in 2024-09-16T05:15:57.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:15:57.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:56 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:15:57.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:57.319 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:57.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:57.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:57.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:57.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:57.486 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:15:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:15:57.486 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:15:57.486 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:15:57.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:57.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:57.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:15:57.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:57.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:57.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:57.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:57.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:57.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 1 pgp_num to 21 2024-09-16T05:15:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:15:57.855 INFO:tasks.rados.rados.2.smithi204.stdout:1284: done (4 left) 2024-09-16T05:15:57.856 INFO:tasks.rados.rados.2.smithi204.stdout:1285: rollback oid 3 current snap is 142 2024-09-16T05:15:57.856 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 131 2024-09-16T05:15:57.856 INFO:tasks.rados.rados.2.smithi204.stdout:1280: finishing write tid 3 to smithi20457400-1 2024-09-16T05:15:57.856 INFO:tasks.rados.rados.2.smithi204.stdout:1280: finishing write tid 4 to smithi20457400-1 2024-09-16T05:15:57.856 INFO:tasks.rados.rados.2.smithi204.stdout:1280: finishing write tid 5 to smithi20457400-1 2024-09-16T05:15:57.856 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 497 (ObjNum 456 snap 142 seq_num 456) dirty exists 2024-09-16T05:15:57.856 INFO:tasks.rados.rados.2.smithi204.stdout:1280: left oid 1 (ObjNum 456 snap 142 seq_num 456) 2024-09-16T05:15:57.856 INFO:tasks.rados.rados.2.smithi204.stdout:1281: expect (ObjNum 412 snap 123 seq_num 412) 2024-09-16T05:15:57.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: pgmap v771: 97 pgs: 4 peering, 1 active+remapped, 2 remapped+peering, 3 unknown, 87 active+clean; 190 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 27 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:15:57.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:15:57.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:15:57.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:15:57.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:15:57.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:15:57.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.b scrub ok 2024-09-16T05:15:57.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1176986771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:15:57.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/543826956' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-16T05:15:57.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:57.950 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:57.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:58.027 INFO:tasks.rados.rados.2.smithi204.stdout:1282: finishing write tid 1 to smithi20457400-41 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1282: finishing write tid 2 to smithi20457400-41 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1282: finishing write tid 3 to smithi20457400-41 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1282: finishing write tid 4 to smithi20457400-41 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1282: finishing write tid 5 to smithi20457400-41 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 552 (ObjNum 457 snap 142 seq_num 457) dirty exists 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1282: left oid 41 (ObjNum 457 snap 142 seq_num 457) 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1283: finishing write tid 1 to smithi20457400-39 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1283: finishing write tid 2 to smithi20457400-39 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1283: finishing write tid 3 to smithi20457400-39 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1280: done (4 left) 2024-09-16T05:15:58.028 INFO:tasks.rados.rados.2.smithi204.stdout:1281: done (3 left) 2024-09-16T05:15:58.029 INFO:tasks.rados.rados.2.smithi204.stdout:1282: done (2 left) 2024-09-16T05:15:58.029 INFO:tasks.rados.rados.2.smithi204.stdout:1286: write (excl) oid 43 current snap is 142 2024-09-16T05:15:58.029 INFO:tasks.rados.rados.2.smithi204.stdout:1283: finishing write tid 4 to smithi20457400-39 2024-09-16T05:15:58.029 INFO:tasks.rados.rados.2.smithi204.stdout:1286: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-16T05:15:58.034 INFO:tasks.rados.rados.2.smithi204.stdout:1286: writing smithi20457400-43 from 514413 to 1234884 tid 1 2024-09-16T05:15:58.041 INFO:tasks.rados.rados.2.smithi204.stdout:1286: writing smithi20457400-43 from 1876529 to 2649681 tid 2 2024-09-16T05:15:58.047 INFO:tasks.rados.rados.2.smithi204.stdout:1286: writing smithi20457400-43 from 3438683 to 3714413 tid 3 2024-09-16T05:15:58.048 INFO:tasks.rados.rados.2.smithi204.stdout:1287: snap_remove snap 133 2024-09-16T05:15:58.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:58.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:58.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: pgmap v771: 97 pgs: 4 peering, 1 active+remapped, 2 remapped+peering, 3 unknown, 87 active+clean; 190 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 27 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.b scrub ok 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:15:58.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1176986771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:15:58.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/543826956' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-16T05:15:58.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:15:58.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:58.201 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:58.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: pgmap v771: 97 pgs: 4 peering, 1 active+remapped, 2 remapped+peering, 3 unknown, 87 active+clean; 190 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 27 op/s; 47 KiB/s, 0 objects/s recovering 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.b scrub ok 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:15:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1176986771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:15:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/543826956' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-16T05:15:58.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:58.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:58.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:58.401 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:15:58.401 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:15:58.423 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:15:58.423 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:15:58.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:15:58.431 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:15:58.449 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:58.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:15:58.454 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:15:58.454 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:15:58.481 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:15:58.481 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:15:58.503 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:15:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:15:58.524 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:15:58.525 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:15:58.546 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:15:58.546 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:15:58.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:58.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:15:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:15:58.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:58.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:58.589 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:15:58.589 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:15:58.611 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:15:58.611 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:15:58.634 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:15:58.634 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:15:58.655 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:15:58.656 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:15:58.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:58.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:58.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:58.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:15:58.820 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:58.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:15:58.865 INFO:tasks.rados.rados.2.smithi204.stdout:1287: done (3 left) 2024-09-16T05:15:58.865 INFO:tasks.rados.rados.2.smithi204.stdout:1288: read oid 22 snap 140 2024-09-16T05:15:58.865 INFO:tasks.rados.rados.2.smithi204.stdout:1288: expect deleted 2024-09-16T05:15:58.865 INFO:tasks.rados.rados.2.smithi204.stdout:1289: snap_create 2024-09-16T05:15:58.865 INFO:tasks.rados.rados.2.smithi204.stdout:1283: finishing write tid 5 to smithi20457400-39 2024-09-16T05:15:58.865 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 648 (ObjNum 458 snap 142 seq_num 458) dirty exists 2024-09-16T05:15:58.865 INFO:tasks.rados.rados.2.smithi204.stdout:1283: left oid 39 (ObjNum 458 snap 142 seq_num 458) 2024-09-16T05:15:58.865 INFO:tasks.rados.rados.2.smithi204.stdout:1285: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:15:58.866 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 470 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-09-16T05:15:58.866 INFO:tasks.rados.rados.2.smithi204.stdout:1286: finishing write tid 1 to smithi20457400-43 2024-09-16T05:15:58.866 INFO:tasks.rados.rados.2.smithi204.stdout:1286: finishing write tid 2 to smithi20457400-43 2024-09-16T05:15:58.866 INFO:tasks.rados.rados.2.smithi204.stdout:1286: finishing write tid 3 to smithi20457400-43 2024-09-16T05:15:58.866 INFO:tasks.rados.rados.2.smithi204.stdout:1286: finishing write tid 4 to smithi20457400-43 2024-09-16T05:15:58.866 INFO:tasks.rados.rados.2.smithi204.stdout:1286: finishing write tid 5 to smithi20457400-43 2024-09-16T05:15:58.866 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 340 (ObjNum 459 snap 142 seq_num 459) dirty exists 2024-09-16T05:15:58.866 INFO:tasks.rados.rados.2.smithi204.stdout:1286: left oid 43 (ObjNum 459 snap 142 seq_num 459) 2024-09-16T05:15:58.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:15:58.931 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:15:58.931 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:15:58.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:58.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:15:59.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:15:59.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:59.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 1.d scrub starts 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 1.d scrub ok 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1176986771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:59.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/543826956' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: osdmap e408: 12 total, 12 up, 12 in 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/88567435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:15:59.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:58 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:15:59.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:15:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 1.d scrub starts 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 1.d scrub ok 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1176986771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/543826956' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: osdmap e408: 12 total, 12 up, 12 in 2024-09-16T05:15:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:15:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:15:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:15:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:15:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:15:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/88567435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:15:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:15:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:58 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:15:59.190 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:15:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:59.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:15:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:15:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:15:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 1.d scrub starts 2024-09-16T05:15:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 1.d scrub ok 2024-09-16T05:15:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:15:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:15:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:15:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:15:59.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:15:59.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1176986771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/543826956' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: osdmap e408: 12 total, 12 up, 12 in 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/88567435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:15:59.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:58 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:15:59.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:59.314 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:59.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:59.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:15:59.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:59.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:59.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:15:59.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:15:59.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:15:59.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:59.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:15:59.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:15:59.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:59.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:15:59.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:15:59.803 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:15:59.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:15:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:59.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:15:59.863 INFO:tasks.rados.rados.2.smithi204.stdout:1283: done (4 left) 2024-09-16T05:15:59.863 INFO:tasks.rados.rados.2.smithi204.stdout:1285: done (3 left) 2024-09-16T05:15:59.863 INFO:tasks.rados.rados.2.smithi204.stdout:1286: done (2 left) 2024-09-16T05:15:59.863 INFO:tasks.rados.rados.2.smithi204.stdout:1288: done (1 left) 2024-09-16T05:15:59.863 INFO:tasks.rados.rados.2.smithi204.stdout:1289: done (0 left) 2024-09-16T05:15:59.863 INFO:tasks.rados.rados.2.smithi204.stdout:1290: write (excl) oid 19 current snap is 143 2024-09-16T05:15:59.863 INFO:tasks.rados.rados.2.smithi204.stdout:1290: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-16T05:15:59.870 INFO:tasks.rados.rados.2.smithi204.stdout:1290: writing smithi20457400-19 from 531220 to 1326940 tid 1 2024-09-16T05:15:59.876 INFO:tasks.rados.rados.2.smithi204.stdout:1290: writing smithi20457400-19 from 1935011 to 2451821 tid 2 2024-09-16T05:15:59.882 INFO:tasks.rados.rados.2.smithi204.stdout:1290: writing smithi20457400-19 from 3066106 to 3486526 tid 3 2024-09-16T05:15:59.882 INFO:tasks.rados.rados.2.smithi204.stdout:1291: read oid 26 snap -1 2024-09-16T05:15:59.882 INFO:tasks.rados.rados.2.smithi204.stdout:1291: expect deleted 2024-09-16T05:15:59.883 INFO:tasks.rados.rados.2.smithi204.stdout:1292: delete oid 38 current snap is 143 2024-09-16T05:15:59.883 INFO:tasks.rados.rados.2.smithi204.stdout:1290: finishing write tid 1 to smithi20457400-19 2024-09-16T05:15:59.883 INFO:tasks.rados.rados.2.smithi204.stdout:1290: finishing write tid 2 to smithi20457400-19 2024-09-16T05:15:59.884 INFO:tasks.rados.rados.2.smithi204.stdout:1291: done (2 left) 2024-09-16T05:15:59.884 INFO:tasks.rados.rados.2.smithi204.stdout:1292: done (1 left) 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1293: read oid 20 snap -1 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1293: expect deleted 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1294: read oid 21 snap -1 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1294: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1295: delete oid 48 current snap is 143 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1290: finishing write tid 3 to smithi20457400-19 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1290: finishing write tid 4 to smithi20457400-19 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1290: finishing write tid 5 to smithi20457400-19 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 296 (ObjNum 460 snap 143 seq_num 460) dirty exists 2024-09-16T05:15:59.885 INFO:tasks.rados.rados.2.smithi204.stdout:1290: left oid 19 (ObjNum 460 snap 143 seq_num 460) 2024-09-16T05:15:59.886 INFO:tasks.rados.rados.2.smithi204.stdout:1290: done (3 left) 2024-09-16T05:15:59.887 INFO:tasks.rados.rados.2.smithi204.stdout:1293: done (2 left) 2024-09-16T05:15:59.887 INFO:tasks.rados.rados.2.smithi204.stdout:1295: done (1 left) 2024-09-16T05:15:59.887 INFO:tasks.rados.rados.2.smithi204.stdout:1296: snap_remove snap 137 2024-09-16T05:15:59.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:15:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:59 smithi032 ceph-mon[27388]: pgmap v774: 97 pgs: 4 peering, 1 active+remapped, 1 unknown, 91 active+clean; 203 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:15:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:59 smithi032 ceph-mon[27388]: osdmap e409: 12 total, 12 up, 12 in 2024-09-16T05:15:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:59 smithi032 ceph-mon[27388]: 2.24 deep-scrub starts 2024-09-16T05:15:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:59 smithi032 ceph-mon[27388]: 2.24 deep-scrub ok 2024-09-16T05:15:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:59 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:15:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:59 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:15:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:59 smithi032 ceph-mon[27388]: 1.f deep-scrub starts 2024-09-16T05:15:59.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:15:59 smithi032 ceph-mon[27388]: 1.f deep-scrub ok 2024-09-16T05:15:59.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:15:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:15:59.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:00.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:00.059 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:00.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:00.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:59 smithi204 ceph-mon[30933]: pgmap v774: 97 pgs: 4 peering, 1 active+remapped, 1 unknown, 91 active+clean; 203 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:16:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:59 smithi204 ceph-mon[30933]: osdmap e409: 12 total, 12 up, 12 in 2024-09-16T05:16:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:59 smithi204 ceph-mon[30933]: 2.24 deep-scrub starts 2024-09-16T05:16:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:59 smithi204 ceph-mon[30933]: 2.24 deep-scrub ok 2024-09-16T05:16:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:59 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:59 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:59 smithi204 ceph-mon[30933]: 1.f deep-scrub starts 2024-09-16T05:16:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:15:59 smithi204 ceph-mon[30933]: 1.f deep-scrub ok 2024-09-16T05:16:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:00.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:00.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:00.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:59 smithi196 ceph-mon[30802]: pgmap v774: 97 pgs: 4 peering, 1 active+remapped, 1 unknown, 91 active+clean; 203 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:16:00.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:59 smithi196 ceph-mon[30802]: osdmap e409: 12 total, 12 up, 12 in 2024-09-16T05:16:00.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:59 smithi196 ceph-mon[30802]: 2.24 deep-scrub starts 2024-09-16T05:16:00.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:59 smithi196 ceph-mon[30802]: 2.24 deep-scrub ok 2024-09-16T05:16:00.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:59 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:00.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:59 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:00.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:59 smithi196 ceph-mon[30802]: 1.f deep-scrub starts 2024-09-16T05:16:00.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:15:59 smithi196 ceph-mon[30802]: 1.f deep-scrub ok 2024-09-16T05:16:00.277 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:00.277 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:00.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:00.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:00.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:00.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:00.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:00.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:00.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:00.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:00.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:00.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:00.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:00.688 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:00.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:00.714 INFO:tasks.rados.rados.2.smithi204.stdout:1296: done (1 left) 2024-09-16T05:16:00.714 INFO:tasks.rados.rados.2.smithi204.stdout:1297: read oid 42 snap -1 2024-09-16T05:16:00.714 INFO:tasks.rados.rados.2.smithi204.stdout:1297: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:16:00.715 INFO:tasks.rados.rados.2.smithi204.stdout:1294: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:16:00.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:00.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:00.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:16:00.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:00.940 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:00.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:00.987 INFO:tasks.rados.rados.2.smithi204.stdout:1297: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:16:01.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:01.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:01.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:01.159 INFO:tasks.rados.rados.2.smithi204.stdout:1294: done (1 left) 2024-09-16T05:16:01.159 INFO:tasks.rados.rados.2.smithi204.stdout:1297: done (0 left) 2024-09-16T05:16:01.159 INFO:tasks.rados.rados.2.smithi204.stdout:1298: write (excl) oid 29 current snap is 143 2024-09-16T05:16:01.159 INFO:tasks.rados.rados.2.smithi204.stdout:1298: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-09-16T05:16:01.161 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:16:01.165 INFO:tasks.rados.rados.2.smithi204.stdout:1298: writing smithi20457400-29 from 581641 to 1203108 tid 1 2024-09-16T05:16:01.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: osdmap e410: 12 total, 12 up, 12 in 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: 2.45 scrub starts 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: 2.45 scrub ok 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:16:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:00 smithi032 ceph-mon[27388]: osdmap e411: 12 total, 12 up, 12 in 2024-09-16T05:16:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: osdmap e410: 12 total, 12 up, 12 in 2024-09-16T05:16:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: 2.45 scrub starts 2024-09-16T05:16:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: 2.45 scrub ok 2024-09-16T05:16:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:16:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:16:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:16:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:16:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:16:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:00 smithi204 ceph-mon[30933]: osdmap e411: 12 total, 12 up, 12 in 2024-09-16T05:16:01.174 INFO:tasks.rados.rados.2.smithi204.stdout:1298: writing smithi20457400-29 from 1994104 to 2774594 tid 2 2024-09-16T05:16:01.181 INFO:tasks.rados.rados.2.smithi204.stdout:1298: writing smithi20457400-29 from 3432022 to 3781641 tid 3 2024-09-16T05:16:01.181 INFO:tasks.rados.rados.2.smithi204.stdout:1299: read oid 41 snap -1 2024-09-16T05:16:01.181 INFO:tasks.rados.rados.2.smithi204.stdout:1299: expect (ObjNum 457 snap 142 seq_num 457) 2024-09-16T05:16:01.181 INFO:tasks.rados.rados.2.smithi204.stdout:1300: copy_from oid 22 from oid 49 current snap is 143 2024-09-16T05:16:01.181 INFO:tasks.rados.rados.2.smithi204.stdout:1298: finishing write tid 1 to smithi20457400-29 2024-09-16T05:16:01.181 INFO:tasks.rados.rados.2.smithi204.stdout:1298: finishing write tid 2 to smithi20457400-29 2024-09-16T05:16:01.181 INFO:tasks.rados.rados.2.smithi204.stdout:1301: write (excl) oid 50 current snap is 143 2024-09-16T05:16:01.181 INFO:tasks.rados.rados.2.smithi204.stdout:1301: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-16T05:16:01.186 INFO:tasks.rados.rados.2.smithi204.stdout:1301: writing smithi20457400-50 from 598448 to 1011517 tid 1 2024-09-16T05:16:01.194 INFO:tasks.rados.rados.2.smithi204.stdout:1301: writing smithi20457400-50 from 1768939 to 2293087 tid 2 2024-09-16T05:16:01.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:01.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:01.200 INFO:tasks.rados.rados.2.smithi204.stdout:1301: writing smithi20457400-50 from 2775798 to 3357659 tid 3 2024-09-16T05:16:01.203 INFO:tasks.rados.rados.2.smithi204.stdout:1301: writing smithi20457400-50 from 3781488 to 3798448 tid 4 2024-09-16T05:16:01.203 INFO:tasks.rados.rados.2.smithi204.stdout:1302: snap_create 2024-09-16T05:16:01.203 INFO:tasks.rados.rados.2.smithi204.stdout:1298: finishing write tid 3 to smithi20457400-29 2024-09-16T05:16:01.203 INFO:tasks.rados.rados.2.smithi204.stdout:1298: finishing write tid 4 to smithi20457400-29 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:1298: finishing write tid 5 to smithi20457400-29 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 635 (ObjNum 463 snap 143 seq_num 463) dirty exists 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:1298: left oid 29 (ObjNum 463 snap 143 seq_num 463) 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:1301: finishing write tid 1 to smithi20457400-50 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:1300: finishing copy_from racing read to smithi20457400-22 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:1300: finishing copy_from to smithi20457400-22 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 328 (ObjNum 434 snap 125 seq_num 434) dirty exists 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:1301: finishing write tid 2 to smithi20457400-50 2024-09-16T05:16:01.204 INFO:tasks.rados.rados.2.smithi204.stdout:1299: expect (ObjNum 457 snap 142 seq_num 457) 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: osdmap e410: 12 total, 12 up, 12 in 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: 2.45 scrub starts 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: 2.45 scrub ok 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:00 smithi196 ceph-mon[30802]: osdmap e411: 12 total, 12 up, 12 in 2024-09-16T05:16:01.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:01.323 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:01.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:01.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:01.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:01.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:01.513 INFO:tasks.rados.rados.2.smithi204.stdout:1301: finishing write tid 3 to smithi20457400-50 2024-09-16T05:16:01.513 INFO:tasks.rados.rados.2.smithi204.stdout:1301: finishing write tid 4 to smithi20457400-50 2024-09-16T05:16:01.513 INFO:tasks.rados.rados.2.smithi204.stdout:1301: finishing write tid 5 to smithi20457400-50 2024-09-16T05:16:01.513 INFO:tasks.rados.rados.2.smithi204.stdout:1301: finishing write tid 6 to smithi20457400-50 2024-09-16T05:16:01.513 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 334 (ObjNum 464 snap 143 seq_num 464) dirty exists 2024-09-16T05:16:01.513 INFO:tasks.rados.rados.2.smithi204.stdout:1301: left oid 50 (ObjNum 464 snap 143 seq_num 464) 2024-09-16T05:16:01.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:01.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:01.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:01.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:16:01.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:01.678 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:01.678 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:01.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:01.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:01.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:01.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:01.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:01.872 INFO:tasks.rados.rados.2.smithi204.stdout:1298: done (4 left) 2024-09-16T05:16:01.872 INFO:tasks.rados.rados.2.smithi204.stdout:1299: done (3 left) 2024-09-16T05:16:01.872 INFO:tasks.rados.rados.2.smithi204.stdout:1300: done (2 left) 2024-09-16T05:16:01.872 INFO:tasks.rados.rados.2.smithi204.stdout:1301: done (1 left) 2024-09-16T05:16:01.872 INFO:tasks.rados.rados.2.smithi204.stdout:1302: done (0 left) 2024-09-16T05:16:01.872 INFO:tasks.rados.rados.2.smithi204.stdout:1303: snap_remove snap 138 2024-09-16T05:16:01.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:01.926 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:01.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:01.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:02.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:01 smithi032 ceph-mon[27388]: pgmap v777: 97 pgs: 4 peering, 1 active+remapped, 1 unknown, 91 active+clean; 210 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.0 MiB/s wr, 8 op/s 2024-09-16T05:16:02.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:01 smithi032 ceph-mon[27388]: 2.35 scrub starts 2024-09-16T05:16:02.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:01 smithi032 ceph-mon[27388]: 2.35 scrub ok 2024-09-16T05:16:02.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:02.070 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:02.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:01 smithi204 ceph-mon[30933]: pgmap v777: 97 pgs: 4 peering, 1 active+remapped, 1 unknown, 91 active+clean; 210 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.0 MiB/s wr, 8 op/s 2024-09-16T05:16:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:01 smithi204 ceph-mon[30933]: 2.35 scrub starts 2024-09-16T05:16:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:01 smithi204 ceph-mon[30933]: 2.35 scrub ok 2024-09-16T05:16:02.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:02.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:02.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:02.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:01 smithi196 ceph-mon[30802]: pgmap v777: 97 pgs: 4 peering, 1 active+remapped, 1 unknown, 91 active+clean; 210 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.0 MiB/s wr, 8 op/s 2024-09-16T05:16:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:01 smithi196 ceph-mon[30802]: 2.35 scrub starts 2024-09-16T05:16:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:01 smithi196 ceph-mon[30802]: 2.35 scrub ok 2024-09-16T05:16:02.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:02.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:16:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:02.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:02.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:02.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:16:02.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:02.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:02.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:02.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:02.568 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:02.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:02.578 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:16:02.594 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:02.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:02.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:02.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:02.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:02.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:02.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:02.887 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:16:02.911 INFO:tasks.rados.rados.2.smithi204.stdout:1303: done (0 left) 2024-09-16T05:16:02.911 INFO:tasks.rados.rados.2.smithi204.stdout:1304: copy_from oid 2 from oid 49 current snap is 144 2024-09-16T05:16:02.911 INFO:tasks.rados.rados.2.smithi204.stdout:1305: rollback oid 34 current snap is 144 2024-09-16T05:16:02.911 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 122 2024-09-16T05:16:02.911 INFO:tasks.rados.rados.2.smithi204.stdout:1306: snap_create 2024-09-16T05:16:02.912 INFO:tasks.rados.rados.2.smithi204.stdout:1305: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:16:02.912 INFO:tasks.rados.rados.2.smithi204.stdout:1305: finishing rollback tid 2 to smithi20457400-34 2024-09-16T05:16:02.913 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 476 (ObjNum 393 snap 118 seq_num 393) dirty exists 2024-09-16T05:16:02.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:02.929 INFO:tasks.rados.rados.2.smithi204.stdout:1304: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:16:02.934 INFO:tasks.rados.rados.2.smithi204.stdout:1304: finishing copy_from to smithi20457400-2 2024-09-16T05:16:02.934 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 328 (ObjNum 434 snap 125 seq_num 434) dirty exists 2024-09-16T05:16:02.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:02.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:02.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:02 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:16:02.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:02 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:16:02.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:02 smithi204 ceph-mon[30933]: osdmap e412: 12 total, 12 up, 12 in 2024-09-16T05:16:02.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:02 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:16:02.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:02 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:16:02.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:02 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:16:02.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:02 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:16:02.987 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1277031506' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:02.999 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:03.000 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:03.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:03.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:02 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:16:03.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:02 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:16:03.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:02 smithi032 ceph-mon[27388]: osdmap e412: 12 total, 12 up, 12 in 2024-09-16T05:16:03.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:02 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:16:03.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:02 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:16:03.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:02 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:16:03.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:02 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:16:03.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1277031506' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:03.061 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:03.061 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:03.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:03.083 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:03.083 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:03.103 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:03.104 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:03.124 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:03.124 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:03.146 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:03.146 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:03.167 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:03.168 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:03.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:03.190 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:03.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:03.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:03.211 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:03.211 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:03.233 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:03.233 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:03.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:02 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:16:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:02 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:16:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:02 smithi196 ceph-mon[30802]: osdmap e412: 12 total, 12 up, 12 in 2024-09-16T05:16:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:02 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:16:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:02 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:16:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:02 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:16:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:02 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:16:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1277031506' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:03.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:03.254 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:03.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:03.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:03.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:03.423 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:03.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:03.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:03.477 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:03.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:03.498 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:03.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:03.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:03.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:03.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:03.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:16:03.696 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:03.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:03.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:03.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:03.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:03.898 INFO:tasks.rados.rados.2.smithi204.stdout:1304: done (2 left) 2024-09-16T05:16:03.898 INFO:tasks.rados.rados.2.smithi204.stdout:1305: done (1 left) 2024-09-16T05:16:03.898 INFO:tasks.rados.rados.2.smithi204.stdout:1306: done (0 left) 2024-09-16T05:16:03.899 INFO:tasks.rados.rados.2.smithi204.stdout:1307: snap_remove snap 140 2024-09-16T05:16:03.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: 2.d deep-scrub starts 2024-09-16T05:16:03.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: 2.d deep-scrub ok 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: pgmap v780: 97 pgs: 97 active+clean; 215 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 2.1 MiB/s wr, 7 op/s 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1277031506' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: osdmap e413: 12 total, 12 up, 12 in 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: 2.36 deep-scrub starts 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: 2.36 deep-scrub ok 2024-09-16T05:16:03.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1311668879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:03.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:03.933 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:16:03.952 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:03.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:04.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:04.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:04.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:04.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: 2.d deep-scrub starts 2024-09-16T05:16:04.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: 2.d deep-scrub ok 2024-09-16T05:16:04.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: pgmap v780: 97 pgs: 97 active+clean; 215 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 2.1 MiB/s wr, 7 op/s 2024-09-16T05:16:04.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-16T05:16:04.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:16:04.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:04.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1277031506' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:04.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: osdmap e413: 12 total, 12 up, 12 in 2024-09-16T05:16:04.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: 2.36 deep-scrub starts 2024-09-16T05:16:04.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: 2.36 deep-scrub ok 2024-09-16T05:16:04.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1311668879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:04.164 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:16:04.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:16:04.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:16:04.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:04.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:04.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:04.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: 2.d deep-scrub starts 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: 2.d deep-scrub ok 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: pgmap v780: 97 pgs: 97 active+clean; 215 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 2.1 MiB/s wr, 7 op/s 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1277031506' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: osdmap e413: 12 total, 12 up, 12 in 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: 2.36 deep-scrub starts 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: 2.36 deep-scrub ok 2024-09-16T05:16:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1311668879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:04.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:04.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:04.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:04.386 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:16:04.386 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:16:04.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:16:04.386 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:16:04.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.386 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:23 2.0 e112/107 2:06458cf1:::smithi20457400-6:head v 112'3, mlcod=0'0)", 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 321.33117006999998, 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.102016024, 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053064+0000", 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8470000000000001e-06 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053068+0000", 2024-09-16T05:16:04.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9519999999999996e-06 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154742+0000", 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.101674714 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154751+0000", 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9020000000000005e-06 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154811+0000", 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9329000000000001e-05 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154832+0000", 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1174000000000001e-05 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.155048+0000", 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021618599999999999 2024-09-16T05:16:04.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.155078+0000", 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9920000000000002e-05 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1514 2.1c 2:3edd0db8:::smithi20457400-24:head [rollback 2f] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:30.665515+0000", 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 213.718717075, 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084191119999999994, 2024-09-16T05:16:04.391 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1514 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665515+0000", 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665515+0000", 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665517+0000", 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9429999999999999e-06 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665524+0000", 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.267e-06 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665546+0000", 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1546e-05 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665555+0000", 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7359999999999996e-06 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665650+0000", 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4549000000000006e-05 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.665657+0000", 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5449999999999998e-06 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.671508+0000", 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058509449999999998 2024-09-16T05:16:04.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.671640+0000", 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013188100000000001 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.673934+0000", 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022937000000000001 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1518 2.1c 2:3edd0db8:::smithi20457400-24:head [setxattr _header (57) in=7b,truncate 3865610] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,1f] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:30.803015+0000", 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 213.581216813, 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060158909999999998, 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1518 2024-09-16T05:16:04.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803015+0000", 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803015+0000", 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803016+0000", 2024-09-16T05:16:04.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8899999999999999e-07 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803019+0000", 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.179e-06 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803025+0000", 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.28e-06 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803029+0000", 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7299999999999999e-06 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803830+0000", 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00080152399999999996 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803906+0000", 2024-09-16T05:16:04.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5210000000000007e-05 2024-09-16T05:16:04.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804014+0000", 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010878399999999999 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804346+0000", 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033128699999999999 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.808909+0000", 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004563564 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.808971+0000", 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1913000000000002e-05 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.809031+0000", 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9830999999999997e-05 2024-09-16T05:16:04.403 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1519 2.1c 2:3edd0db8:::smithi20457400-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e219)", 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:30.803051+0000", 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 213.58118100999999, 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006120154, 2024-09-16T05:16:04.404 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1519 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803051+0000", 2024-09-16T05:16:04.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803051+0000", 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803051+0000", 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4400000000000001e-07 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803054+0000", 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.261e-06 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803061+0000", 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3629999999999996e-06 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.803066+0000", 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8860000000000003e-06 2024-09-16T05:16:04.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804243+0000", 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011766470000000001 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.804327+0000", 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4711999999999998e-05 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.809067+0000", 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047402640000000001 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.809112+0000", 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4477999999999997e-05 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:30.809171+0000", 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9098999999999998e-05 2024-09-16T05:16:04.409 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2255 2.1d 2:ba754fcb:::smithi20457400-30:head [append 0~0] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 142.70255098499999, 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53925791899999997, 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2255 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7300000000000001e-07 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681684+0000", 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.582e-06 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681692+0000", 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3920000000000003e-06 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681698+0000", 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2100000000000001e-06 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681779+0000", 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1003999999999999e-05 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681782+0000", 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1190000000000001e-06 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684363+0000", 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025812399999999998 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684417+0000", 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3692e-05 2024-09-16T05:16:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684503+0000", 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6592e-05 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684818+0000", 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000314851 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220845+0000", 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53602721099999995 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220887+0000", 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1514000000000003e-05 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220939+0000", 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2039000000000001e-05 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2352 2.1d 2:b98a79db:::smithi20457400-31:head [copy-from ver 319] snapc 63=[63,62,60,5d,5a,59,58,57,52,4d,49,45] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:50.690636+0000", 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 133.693595719, 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020683322000000001, 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2352 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690636+0000", 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690636+0000", 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690638+0000", 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0190000000000001e-06 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690648+0000", 2024-09-16T05:16:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0103e-05 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690671+0000", 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3014999999999999e-05 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690683+0000", 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1537e-05 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690759+0000", 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6002999999999997e-05 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690768+0000", 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.285e-06 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.693224+0000", 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024560770000000001 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.693348+0000", 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012423100000000001 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.705292+0000", 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011943593000000001 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.708786+0000", 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034936339999999998 2024-09-16T05:16:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.711209+0000", 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00242341 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.711268+0000", 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9163000000000003e-05 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.711320+0000", 2024-09-16T05:16:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1252000000000003e-05 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2353 2.1d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e299)", 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:50.690713+0000", 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 133.69351932999999, 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017533710000000001, 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2353 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690713+0000", 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690713+0000", 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690713+0000", 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0800000000000005e-07 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690716+0000", 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2160000000000002e-06 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690733+0000", 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6659000000000001e-05 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690739+0000", 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1879999999999999e-06 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690852+0000", 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011319600000000001 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690858+0000", 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7010000000000001e-06 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.693501+0000", 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0026431879999999999 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.693546+0000", 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4580999999999999e-05 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.708030+0000", 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014484271 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.708136+0000", 2024-09-16T05:16:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010569999999999999 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.708246+0000", 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011050199999999999 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2359 2.1d 2:ba754fcb:::smithi20457400-30:5a [sparse-read 0~1979077,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e299)", 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:50.690763+0000", 2024-09-16T05:16:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 133.69346845300001, 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063185890000000003, 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2359 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690763+0000", 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690763+0000", 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690764+0000", 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9599999999999999e-07 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690768+0000", 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1280000000000001e-06 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690866+0000", 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7375999999999997e-05 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690874+0000", 2024-09-16T05:16:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8509999999999999e-06 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690934+0000", 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0569999999999998e-05 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690940+0000", 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4990000000000002e-06 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.693571+0000", 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002631169 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.693667+0000", 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6155000000000003e-05 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.697082+0000", 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034151450000000001 2024-09-16T05:16:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2360 2.1d 2:ba754fcb:::smithi20457400-30:5a [read 0~2924451,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e299)", 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:50.690893+0000", 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 133.69333844299999, 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074574749999999999, 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2360 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690893+0000", 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690893+0000", 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690894+0000", 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1900000000000001e-07 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690900+0000", 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8649999999999998e-06 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690952+0000", 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1344000000000002e-05 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690958+0000", 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5019999999999998e-06 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.691038+0000", 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0117999999999994e-05 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.691044+0000", 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4550000000000003e-06 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.697115+0000", 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060718769999999998 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.697220+0000", 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010440799999999999 2024-09-16T05:16:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.698351+0000", 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001131087 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2361 2.1d 2:ba754fcb:::smithi20457400-30:5a [read 0~2924451,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e299)", 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:50.690976+0000", 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 133.69325637200001, 2024-09-16T05:16:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0083473339999999997, 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2361 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690976+0000", 2024-09-16T05:16:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690976+0000", 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690976+0000", 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3200000000000004e-07 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690982+0000", 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3709999999999999e-06 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690990+0000", 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1349999999999992e-06 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.690995+0000", 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5520000000000004e-06 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.691067+0000", 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1512000000000002e-05 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.691072+0000", 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9559999999999996e-06 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.698419+0000", 2024-09-16T05:16:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073474630000000003 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.698494+0000", 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4799e-05 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:50.699323+0000", 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00082881399999999998 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2503 2.1d 2:ba754fcb:::smithi20457400-30:head [write 484134~708729] snapc 6a=[6a,69,68,67,66,64,63,62,60,5d,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:02.758065+0000", 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 121.626166481, 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064671980000000004, 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2503 2024-09-16T05:16:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.758065+0000", 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.758065+0000", 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.758068+0000", 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1890000000000001e-06 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.758537+0000", 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046934199999999998 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.758564+0000", 2024-09-16T05:16:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6520999999999999e-05 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.758574+0000", 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0553999999999999e-05 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.758650+0000", 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5666000000000004e-05 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:16:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.758655+0000", 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4999999999999999e-06 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.760366+0000", 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017107349999999999 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.760458+0000", 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1792000000000005e-05 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.760636+0000", 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000177978 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.762267+0000", 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016315920000000001 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.764454+0000", 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021868769999999998 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.764496+0000", 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1618e-05 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.764533+0000", 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6834e-05 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2559 2.1d 2:ba754fcb:::smithi20457400-30:head [copy-from ver 324] snapc 6d=[6c,6b,6a,69,67,66,64,63,62,60,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 79.427586812000001, 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.74015230600000004, 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2559 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956647+0000", 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2919999999999998e-06 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956655+0000", 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.729e-06 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956672+0000", 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6775000000000001e-05 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956685+0000", 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2832e-05 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956782+0000", 2024-09-16T05:16:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6862999999999997e-05 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956790+0000", 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4020000000000002e-06 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958620+0000", 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001830415 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958718+0000", 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7708999999999996e-05 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.688352+0000", 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72963416800000003 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.695488+0000", 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071356659999999997 2024-09-16T05:16:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696731+0000", 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00124332 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696767+0000", 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6171000000000003e-05 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696797+0000", 2024-09-16T05:16:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9964e-05 2024-09-16T05:16:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2560 2.1d 2:ba754fcb:::smithi20457400-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:16:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:16:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 79.427520931999993, 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73643418699999996, 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2560 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956712+0000", 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.82e-07 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956714+0000", 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6419999999999999e-06 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956737+0000", 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2668999999999999e-05 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956744+0000", 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.289e-06 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956847+0000", 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102729 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956855+0000", 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4009999999999997e-06 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958884+0000", 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020286409999999999 2024-09-16T05:16:04.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958928+0000", 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4458e-05 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693004+0000", 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73407526999999995 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693071+0000", 2024-09-16T05:16:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7176999999999997e-05 2024-09-16T05:16:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693145+0000", 2024-09-16T05:16:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4328999999999999e-05 2024-09-16T05:16:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.0.0:67 2.31 e356/279 2:8c1ba486:::smithi20457400-2:75 v 356'145, mlcod=350'141)", 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 58.678330316999997, 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.929470409, 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4499999999999998e-07 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705907+0000", 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8799999999999999e-06 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634756+0000", 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92884895999999995 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634764+0000", 2024-09-16T05:16:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9589999999999995e-06 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634827+0000", 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3280000000000004e-05 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634853+0000", 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6231999999999999e-05 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:16:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635342+0000", 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048899100000000004 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635372+0000", 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9762000000000001e-05 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2775 2.1d 2:b98a79db:::smithi20457400-31:head [copy-from ver 0] snapc 79=[79,78,77,76,75,73,72,71,6c,67,64,63,5a,58] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:08.744686+0000", 2024-09-16T05:16:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 55.639545931999997, 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011074647, 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2775 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744686+0000", 2024-09-16T05:16:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744686+0000", 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744686+0000", 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6300000000000001e-07 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744690+0000", 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3809999999999999e-06 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744771+0000", 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1378999999999995e-05 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744777+0000", 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1909999999999997e-06 2024-09-16T05:16:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744958+0000", 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018046300000000001 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744965+0000", 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7889999999999997e-06 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.747505+0000", 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025407469999999999 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.747610+0000", 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104232 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.755761+0000", 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081511020000000003 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2776 2.1d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e359)", 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:08.744787+0000", 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 55.639444943999997, 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011175291, 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2776 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744787+0000", 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744787+0000", 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744788+0000", 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1500000000000004e-07 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744791+0000", 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4659999999999999e-06 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744799+0000", 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6820000000000003e-06 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.744804+0000", 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8210000000000001e-06 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.745002+0000", 2024-09-16T05:16:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019802399999999999 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.745007+0000", 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4210000000000003e-06 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.747794+0000", 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027865339999999998 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.747853+0000", 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.914e-05 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.755836+0000", 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079829389999999997 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:08.755962+0000", 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000126549 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2816 2.1d 2:ba754fcb:::smithi20457400-30:head [stat,delete] snapc 7a=[7a,78,77,76,75,72,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:15.703908+0000", 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.680323680999997, 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072672860000000004, 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2816 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.703908+0000", 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.703908+0000", 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.703911+0000", 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3080000000000002e-06 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.703919+0000", 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9589999999999995e-06 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.703940+0000", 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1319e-05 2024-09-16T05:16:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.703950+0000", 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0063e-05 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.704063+0000", 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011323 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:16:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.704073+0000", 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4979999999999995e-06 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.708900+0000", 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048273370000000001 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.709010+0000", 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010956 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:15.711176+0000", 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0021660120000000001 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2896 2.1d 2:b98a79db:::smithi20457400-31:head [copy-from ver 336] snapc 7c=[7c,7b,7a,76,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:21.706650+0000", 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.677581855, 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.026780103999999999, 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2896 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706650+0000", 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706650+0000", 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706653+0000", 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7300000000000001e-06 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706670+0000", 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6781000000000001e-05 2024-09-16T05:16:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6733e-05 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706709+0000", 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2707999999999999e-05 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706828+0000", 2024-09-16T05:16:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119046 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706837+0000", 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6430000000000001e-06 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.713173+0000", 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063359380000000002 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.713282+0000", 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010902 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.725640+0000", 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012358159000000001 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.729235+0000", 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035948619999999999 2024-09-16T05:16:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.733348+0000", 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041137650000000001 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.733388+0000", 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9931000000000001e-05 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.733430+0000", 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1788e-05 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2897 2.1d 2:b98a79db:::smithi20457400-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:21.706758+0000", 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.677473993, 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022169442000000001, 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:16:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2897 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706758+0000", 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706758+0000", 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706759+0000", 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3699999999999999e-07 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706763+0000", 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.924e-06 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706774+0000", 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1015e-05 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706781+0000", 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1729999999999998e-06 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706915+0000", 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000133986 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706920+0000", 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0320000000000003e-06 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.713440+0000", 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0065196899999999999 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.713483+0000", 2024-09-16T05:16:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3228e-05 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.728741+0000", 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01525784 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.728834+0000", 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2905000000000006e-05 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.728927+0000", 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3812000000000002e-05 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(2.17,RELEASE,e:407,from:6,reservation_nonce:0)", 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:57.270114+0000", 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 7.114117706, 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014738950000000001, 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:16:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.270114+0000", 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.270114+0000", 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.270114+0000", 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.98e-07 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.270116+0000", 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3909999999999999e-06 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.284745+0000", 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014629401 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.284753+0000", 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0309999999999998e-06 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.284813+0000", 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9823e-05 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.284833+0000", 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0200999999999999e-05 2024-09-16T05:16:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:57.284853+0000", 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9905e-05 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:16:04.508 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:16:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:04.529 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:04.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:04.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:04.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:04.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:04.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:04.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:04.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:04.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:04.903 INFO:tasks.rados.rados.2.smithi204.stdout:1307: done (0 left) 2024-09-16T05:16:04.904 INFO:tasks.rados.rados.2.smithi204.stdout:1308: snap_create 2024-09-16T05:16:04.906 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:04.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:05.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:05.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:05.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:05.129 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:16:05.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:04 smithi032 ceph-mon[27388]: osdmap e414: 12 total, 12 up, 12 in 2024-09-16T05:16:05.132 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:16:05.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:05.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:05.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:05.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:04 smithi204 ceph-mon[30933]: osdmap e414: 12 total, 12 up, 12 in 2024-09-16T05:16:05.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:16:05.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:04 smithi196 ceph-mon[30802]: osdmap e414: 12 total, 12 up, 12 in 2024-09-16T05:16:05.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:05.281 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:05.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:05.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:05.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:05.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:05.499 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:16:05.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:05.533 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:05.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:05.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:05.659 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:05.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:05.716 INFO:tasks.rados.rados.2.smithi204.stdout:1308: done (0 left) 2024-09-16T05:16:05.716 INFO:tasks.rados.rados.2.smithi204.stdout:1309: read oid 33 snap -1 2024-09-16T05:16:05.716 INFO:tasks.rados.rados.2.smithi204.stdout:1309: expect (ObjNum 380 snap 114 seq_num 380) 2024-09-16T05:16:05.716 INFO:tasks.rados.rados.2.smithi204.stdout:1310: write oid 9 current snap is 146 2024-09-16T05:16:05.716 INFO:tasks.rados.rados.2.smithi204.stdout:1310: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-09-16T05:16:05.722 INFO:tasks.rados.rados.2.smithi204.stdout:1310: writing smithi20457400-9 from 615255 to 1103573 tid 1 2024-09-16T05:16:05.730 INFO:tasks.rados.rados.2.smithi204.stdout:1310: writing smithi20457400-9 from 1827421 to 2611580 tid 2 2024-09-16T05:16:05.737 INFO:tasks.rados.rados.2.smithi204.stdout:1310: writing smithi20457400-9 from 3203221 to 3815255 tid 3 2024-09-16T05:16:05.737 INFO:tasks.rados.rados.2.smithi204.stdout:1311: write oid 22 current snap is 146 2024-09-16T05:16:05.738 INFO:tasks.rados.rados.2.smithi204.stdout:1311: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-09-16T05:16:05.744 INFO:tasks.rados.rados.2.smithi204.stdout:1311: writing smithi20457400-22 from 632062 to 1195629 tid 1 2024-09-16T05:16:05.751 INFO:tasks.rados.rados.2.smithi204.stdout:1311: writing smithi20457400-22 from 1885903 to 2413720 tid 2 2024-09-16T05:16:05.758 INFO:tasks.rados.rados.2.smithi204.stdout:1311: writing smithi20457400-22 from 2830644 to 3551402 tid 3 2024-09-16T05:16:05.758 INFO:tasks.rados.rados.2.smithi204.stdout:1312: read oid 8 snap -1 2024-09-16T05:16:05.758 INFO:tasks.rados.rados.2.smithi204.stdout:1312: expect (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:16:05.758 INFO:tasks.rados.rados.2.smithi204.stdout:1313: snap_create 2024-09-16T05:16:05.758 INFO:tasks.rados.rados.2.smithi204.stdout:1309: expect (ObjNum 380 snap 114 seq_num 380) 2024-09-16T05:16:05.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:05.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:05.809 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":784,"stamp":"2024-09-16T05:16:04.899943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":233701195,"num_objects":102,"num_object_clones":51,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":8,"num_read":881,"num_read_kb":921449,"num_write":579,"num_write_kb":249400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18680,"ondisk_log_size":18680,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":197,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":772892,"kb_used_data":335096,"kb_used_omap":27,"kb_used_meta":437476,"kb_avail":1124168932,"statfs":{"total":1151940427776,"available":1151148986368,"internally_reserved":0,"allocated":343138304,"data_stored":379990595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":447975591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11242792,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":26,"num_read_kb":39657,"num_write":8,"num_write_kb":8220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.188919"},"pg_stats":[{"pgid":"2.4b","version":"244'177","reported_seq":1665,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876261+0000","last_change":"2024-09-16T05:15:58.873393+0000","last_active":"2024-09-16T05:15:59.876261+0000","last_peered":"2024-09-16T05:15:59.876261+0000","last_clean":"2024-09-16T05:15:59.876261+0000","last_became_active":"2024-09-16T05:14:30.853998+0000","last_became_peered":"2024-09-16T05:14:30.853998+0000","last_unstale":"2024-09-16T05:15:59.876261+0000","last_undegraded":"2024-09-16T05:15:59.876261+0000","last_fullsized":"2024-09-16T05:15:59.876261+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:14:32.337052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:32.337052+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009912059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.4a","version":"409'469","reported_seq":1921,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861223+0000","last_change":"2024-09-16T05:15:59.250961+0000","last_active":"2024-09-16T05:15:59.861223+0000","last_peered":"2024-09-16T05:15:59.861223+0000","last_clean":"2024-09-16T05:15:59.861223+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:15:59.861223+0000","last_undegraded":"2024-09-16T05:15:59.861223+0000","last_fullsized":"2024-09-16T05:15:59.861223+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"378'461","last_scrub_stamp":"2024-09-16T05:15:37.597739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:37.597739+0000","objects_scrubbed":1,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:05.648475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018772960000000001,"stat_sum":{"num_bytes":2806835,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":9,"num_write_kb":3804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":1714,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.721113+0000","last_change":"2024-09-16T05:16:00.721334+0000","last_active":"2024-09-16T05:16:00.721113+0000","last_peered":"2024-09-16T05:16:00.721113+0000","last_clean":"2024-09-16T05:16:00.721113+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:16:00.721113+0000","last_undegraded":"2024-09-16T05:16:00.721113+0000","last_fullsized":"2024-09-16T05:16:00.721113+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:16:00.136780+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.136780+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:29.332258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096012300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.48","version":"409'200","reported_seq":1652,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.885773+0000","last_change":"2024-09-16T05:15:58.869908+0000","last_active":"2024-09-16T05:15:59.885773+0000","last_peered":"2024-09-16T05:15:59.885773+0000","last_clean":"2024-09-16T05:15:59.885773+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:15:59.885773+0000","last_undegraded":"2024-09-16T05:15:59.885773+0000","last_fullsized":"2024-09-16T05:15:59.885773+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"378'179","last_scrub_stamp":"2024-09-16T05:15:39.057858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.057858+0000","objects_scrubbed":1,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:59.692829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001658961,"stat_sum":{"num_bytes":8448112,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":20677,"num_write":7,"num_write_kb":8206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":480,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.873497+0000","last_change":"2024-09-16T05:15:59.249285+0000","last_active":"2024-09-16T05:15:59.873497+0000","last_peered":"2024-09-16T05:15:59.873497+0000","last_clean":"2024-09-16T05:15:59.873497+0000","last_became_active":"2024-09-16T05:15:16.867821+0000","last_became_peered":"2024-09-16T05:15:16.867821+0000","last_unstale":"2024-09-16T05:15:59.873497+0000","last_undegraded":"2024-09-16T05:15:59.873497+0000","last_fullsized":"2024-09-16T05:15:59.873497+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:57.698141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.698141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.630739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00030955199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":1497,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876433+0000","last_change":"2024-09-16T05:15:58.872774+0000","last_active":"2024-09-16T05:15:59.876433+0000","last_peered":"2024-09-16T05:15:59.876433+0000","last_clean":"2024-09-16T05:15:59.876433+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:15:59.876433+0000","last_undegraded":"2024-09-16T05:15:59.876433+0000","last_fullsized":"2024-09-16T05:15:59.876433+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:15:37.935128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:37.935128+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:44.107519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058721900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":1924,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876978+0000","last_change":"2024-09-16T05:15:59.877024+0000","last_active":"2024-09-16T05:15:59.876978+0000","last_peered":"2024-09-16T05:15:59.876978+0000","last_clean":"2024-09-16T05:15:59.876978+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:15:59.876978+0000","last_undegraded":"2024-09-16T05:15:59.876978+0000","last_fullsized":"2024-09-16T05:15:59.876978+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:15:59.876915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.876915+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:27.690381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011135889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":1579,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876241+0000","last_change":"2024-09-16T05:15:58.872732+0000","last_active":"2024-09-16T05:15:59.876241+0000","last_peered":"2024-09-16T05:15:59.876241+0000","last_clean":"2024-09-16T05:15:59.876241+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:15:59.876241+0000","last_undegraded":"2024-09-16T05:15:59.876241+0000","last_fullsized":"2024-09-16T05:15:59.876241+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.970206+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:09.958259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000541239,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":477,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.716209+0000","last_change":"2024-09-16T05:16:00.716470+0000","last_active":"2024-09-16T05:16:00.716209+0000","last_peered":"2024-09-16T05:16:00.716209+0000","last_clean":"2024-09-16T05:16:00.716209+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:16:00.716209+0000","last_undegraded":"2024-09-16T05:16:00.716209+0000","last_fullsized":"2024-09-16T05:16:00.716209+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:14:28.659562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:28.659562+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057241499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":465,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.973740+0000","last_change":"2024-09-16T05:16:02.973931+0000","last_active":"2024-09-16T05:16:02.973740+0000","last_peered":"2024-09-16T05:16:02.973740+0000","last_clean":"2024-09-16T05:16:02.973740+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:16:02.973740+0000","last_undegraded":"2024-09-16T05:16:02.973740+0000","last_fullsized":"2024-09-16T05:16:02.973740+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:33.678710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:33.678710+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:00.994439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"3"}]},{"pgid":"2.41","version":"178'50","reported_seq":561,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861188+0000","last_change":"2024-09-16T05:15:59.249221+0000","last_active":"2024-09-16T05:15:59.861188+0000","last_peered":"2024-09-16T05:15:59.861188+0000","last_clean":"2024-09-16T05:15:59.861188+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:15:59.861188+0000","last_undegraded":"2024-09-16T05:15:59.861188+0000","last_fullsized":"2024-09-16T05:15:59.861188+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:15.794439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.794439+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:31.949874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00042154699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":1524,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.720856+0000","last_change":"2024-09-16T05:16:00.721164+0000","last_active":"2024-09-16T05:16:00.720856+0000","last_peered":"2024-09-16T05:16:00.720856+0000","last_clean":"2024-09-16T05:16:00.720856+0000","last_became_active":"2024-09-16T05:13:45.682851+0000","last_became_peered":"2024-09-16T05:13:45.682851+0000","last_unstale":"2024-09-16T05:16:00.720856+0000","last_undegraded":"2024-09-16T05:16:00.720856+0000","last_fullsized":"2024-09-16T05:16:00.720856+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:07.789202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.789202+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:14.454124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0042834129999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.3f","version":"378'282","reported_seq":1773,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.718858+0000","last_change":"2024-09-16T05:16:00.719180+0000","last_active":"2024-09-16T05:16:00.718858+0000","last_peered":"2024-09-16T05:16:00.718858+0000","last_clean":"2024-09-16T05:16:00.718858+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:16:00.718858+0000","last_undegraded":"2024-09-16T05:16:00.718858+0000","last_fullsized":"2024-09-16T05:16:00.718858+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"357'270","last_scrub_stamp":"2024-09-16T05:15:09.797411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.797411+0000","objects_scrubbed":3,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:37.495569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002764229,"stat_sum":{"num_bytes":2285818,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":22716,"num_write":15,"num_write_kb":3475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.3e","version":"378'353","reported_seq":1808,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.342523+0000","last_change":"2024-09-16T05:16:00.789634+0000","last_active":"2024-09-16T05:16:02.342523+0000","last_peered":"2024-09-16T05:16:02.342523+0000","last_clean":"2024-09-16T05:16:02.342523+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:16:02.342523+0000","last_undegraded":"2024-09-16T05:16:02.342523+0000","last_fullsized":"2024-09-16T05:16:02.342523+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"360'342","last_scrub_stamp":"2024-09-16T05:15:09.920637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.920637+0000","objects_scrubbed":4,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.810508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051316800000000004,"stat_sum":{"num_bytes":2437081,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":2380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2019,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876499+0000","last_change":"2024-09-16T05:15:58.874075+0000","last_active":"2024-09-16T05:15:59.876499+0000","last_peered":"2024-09-16T05:15:59.876499+0000","last_clean":"2024-09-16T05:15:59.876499+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:15:59.876499+0000","last_undegraded":"2024-09-16T05:15:59.876499+0000","last_fullsized":"2024-09-16T05:15:59.876499+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:15:09.150849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.150849+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:31.637012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015034860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.3c","version":"411'457","reported_seq":2031,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:01.191542+0000","last_change":"2024-09-16T05:16:00.717029+0000","last_active":"2024-09-16T05:16:01.191542+0000","last_peered":"2024-09-16T05:16:01.191542+0000","last_clean":"2024-09-16T05:16:01.191542+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:16:01.191542+0000","last_undegraded":"2024-09-16T05:16:01.191542+0000","last_fullsized":"2024-09-16T05:16:01.191542+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"342'400","last_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:14:51.845721+0000","objects_scrubbed":6,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.017965+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.00107671,"stat_sum":{"num_bytes":8834763,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":33088,"num_write":20,"num_write_kb":9662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.3b","version":"409'309","reported_seq":1799,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.720661+0000","last_change":"2024-09-16T05:16:00.721561+0000","last_active":"2024-09-16T05:16:00.720661+0000","last_peered":"2024-09-16T05:16:00.720661+0000","last_clean":"2024-09-16T05:16:00.720661+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:16:00.720661+0000","last_undegraded":"2024-09-16T05:16:00.720661+0000","last_fullsized":"2024-09-16T05:16:00.720661+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"300'290","last_scrub_stamp":"2024-09-16T05:15:06.955285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.955285+0000","objects_scrubbed":2,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:30.605026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00139357,"stat_sum":{"num_bytes":5277801,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":46642,"num_write":5,"num_write_kb":1725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.3a","version":"409'470","reported_seq":1912,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886163+0000","last_change":"2024-09-16T05:15:58.870619+0000","last_active":"2024-09-16T05:15:59.886163+0000","last_peered":"2024-09-16T05:15:59.886163+0000","last_clean":"2024-09-16T05:15:59.886163+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:15:59.886163+0000","last_undegraded":"2024-09-16T05:15:59.886163+0000","last_fullsized":"2024-09-16T05:15:59.886163+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"356'429","last_scrub_stamp":"2024-09-16T05:15:10.141866+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.141866+0000","objects_scrubbed":2,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.478426+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022497760000000002,"stat_sum":{"num_bytes":7237161,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":9241,"num_write":25,"num_write_kb":11603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":565,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.885853+0000","last_change":"2024-09-16T05:15:58.868870+0000","last_active":"2024-09-16T05:15:59.885853+0000","last_peered":"2024-09-16T05:15:59.885853+0000","last_clean":"2024-09-16T05:15:59.885853+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:15:59.885853+0000","last_undegraded":"2024-09-16T05:15:59.885853+0000","last_fullsized":"2024-09-16T05:15:59.885853+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:15.232072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.232072+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:19.294613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052819599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":578,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.973468+0000","last_change":"2024-09-16T05:16:02.973682+0000","last_active":"2024-09-16T05:16:02.973468+0000","last_peered":"2024-09-16T05:16:02.973468+0000","last_clean":"2024-09-16T05:16:02.973468+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:16:02.973468+0000","last_undegraded":"2024-09-16T05:16:02.973468+0000","last_fullsized":"2024-09-16T05:16:02.973468+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:13.450261+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:13.450261+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:25.125488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036209099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"3"}]},{"pgid":"2.37","version":"0'0","reported_seq":649,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861540+0000","last_change":"2024-09-16T05:15:59.249823+0000","last_active":"2024-09-16T05:15:59.861540+0000","last_peered":"2024-09-16T05:15:59.861540+0000","last_clean":"2024-09-16T05:15:59.861540+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:15:59.861540+0000","last_undegraded":"2024-09-16T05:15:59.861540+0000","last_fullsized":"2024-09-16T05:15:59.861540+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.650972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:08.460368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076871700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":647,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876424+0000","last_change":"2024-09-16T05:15:58.873388+0000","last_active":"2024-09-16T05:15:59.876424+0000","last_peered":"2024-09-16T05:15:59.876424+0000","last_clean":"2024-09-16T05:15:59.876424+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:15:59.876424+0000","last_undegraded":"2024-09-16T05:15:59.876424+0000","last_fullsized":"2024-09-16T05:15:59.876424+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:14:49.375386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.375386+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:18.832673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011397779999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.35","version":"410'508","reported_seq":2008,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876651+0000","last_change":"2024-09-16T05:15:58.873924+0000","last_active":"2024-09-16T05:15:59.876651+0000","last_peered":"2024-09-16T05:15:59.876651+0000","last_clean":"2024-09-16T05:15:59.876651+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:15:59.876651+0000","last_undegraded":"2024-09-16T05:15:59.876651+0000","last_fullsized":"2024-09-16T05:15:59.876651+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'421","last_scrub_stamp":"2024-09-16T05:14:34.349392+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:14:34.349392+0000","objects_scrubbed":4,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011497549999999999,"stat_sum":{"num_bytes":8324759,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":24061,"num_write":19,"num_write_kb":7769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.34","version":"409'323","reported_seq":1411,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.720747+0000","last_change":"2024-09-16T05:16:00.721019+0000","last_active":"2024-09-16T05:16:00.720747+0000","last_peered":"2024-09-16T05:16:00.720747+0000","last_clean":"2024-09-16T05:16:00.720747+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:16:00.720747+0000","last_undegraded":"2024-09-16T05:16:00.720747+0000","last_fullsized":"2024-09-16T05:16:00.720747+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'243","last_scrub_stamp":"2024-09-16T05:14:50.913931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:50.913931+0000","objects_scrubbed":5,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:19.164514+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00083457400000000004,"stat_sum":{"num_bytes":7619765,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":41072,"num_write":14,"num_write_kb":8547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":656,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.717370+0000","last_change":"2024-09-16T05:16:00.717930+0000","last_active":"2024-09-16T05:16:00.717370+0000","last_peered":"2024-09-16T05:16:00.717370+0000","last_clean":"2024-09-16T05:16:00.717370+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:16:00.717370+0000","last_undegraded":"2024-09-16T05:16:00.717370+0000","last_fullsized":"2024-09-16T05:16:00.717370+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:46.792111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.792111+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:11.060540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001905615,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.32","version":"380'125","reported_seq":744,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.342535+0000","last_change":"2024-09-16T05:16:00.789629+0000","last_active":"2024-09-16T05:16:02.342535+0000","last_peered":"2024-09-16T05:16:02.342535+0000","last_clean":"2024-09-16T05:16:02.342535+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:16:02.342535+0000","last_undegraded":"2024-09-16T05:16:02.342535+0000","last_fullsized":"2024-09-16T05:16:02.342535+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'101","last_scrub_stamp":"2024-09-16T05:14:49.915945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.915945+0000","objects_scrubbed":2,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:19.702742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00047261200000000002,"stat_sum":{"num_bytes":6857355,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":8515,"num_write":17,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.31","version":"402'164","reported_seq":1281,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.873380+0000","last_change":"2024-09-16T05:15:59.249251+0000","last_active":"2024-09-16T05:15:59.873380+0000","last_peered":"2024-09-16T05:15:59.873380+0000","last_clean":"2024-09-16T05:15:59.873380+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:15:59.873380+0000","last_undegraded":"2024-09-16T05:15:59.873380+0000","last_fullsized":"2024-09-16T05:15:59.873380+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"402'164","last_scrub_stamp":"2024-09-16T05:15:53.749444+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.749444+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:18.110989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000306526,"stat_sum":{"num_bytes":4394329,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":49186,"num_write":13,"num_write_kb":4010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":613,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861187+0000","last_change":"2024-09-16T05:15:59.249345+0000","last_active":"2024-09-16T05:15:59.861187+0000","last_peered":"2024-09-16T05:15:59.861187+0000","last_clean":"2024-09-16T05:15:59.861187+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:15:59.861187+0000","last_undegraded":"2024-09-16T05:15:59.861187+0000","last_fullsized":"2024-09-16T05:15:59.861187+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:06.783241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.783241+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:35.936971+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00051517999999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":427,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.719874+0000","last_change":"2024-09-16T05:15:57.054819+0000","last_active":"2024-09-16T05:16:00.719874+0000","last_peered":"2024-09-16T05:16:00.719874+0000","last_clean":"2024-09-16T05:16:00.719874+0000","last_became_active":"2024-09-16T05:15:56.716090+0000","last_became_peered":"2024-09-16T05:15:56.716090+0000","last_unstale":"2024-09-16T05:16:00.719874+0000","last_undegraded":"2024-09-16T05:16:00.719874+0000","last_fullsized":"2024-09-16T05:16:00.719874+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.054740+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.054740+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.994935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1203,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:01.170522+0000","last_change":"2024-09-16T05:16:01.171180+0000","last_active":"2024-09-16T05:16:01.170522+0000","last_peered":"2024-09-16T05:16:01.170522+0000","last_clean":"2024-09-16T05:16:01.170522+0000","last_became_active":"2024-09-16T05:13:25.679267+0000","last_became_peered":"2024-09-16T05:13:25.679267+0000","last_unstale":"2024-09-16T05:16:01.170522+0000","last_undegraded":"2024-09-16T05:16:01.170522+0000","last_fullsized":"2024-09-16T05:16:01.170522+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:15:53.135605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.135605+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:21.690661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088354299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1058,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876190+0000","last_change":"2024-09-16T05:15:56.852148+0000","last_active":"2024-09-16T05:15:59.876190+0000","last_peered":"2024-09-16T05:15:59.876190+0000","last_clean":"2024-09-16T05:15:59.876190+0000","last_became_active":"2024-09-16T05:15:56.714681+0000","last_became_peered":"2024-09-16T05:15:56.714681+0000","last_unstale":"2024-09-16T05:15:59.876190+0000","last_undegraded":"2024-09-16T05:15:59.876190+0000","last_fullsized":"2024-09-16T05:15:59.876190+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.852021+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.852021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:11.168034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":1117,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.974656+0000","last_change":"2024-09-16T05:16:02.974966+0000","last_active":"2024-09-16T05:16:02.974656+0000","last_peered":"2024-09-16T05:16:02.974656+0000","last_clean":"2024-09-16T05:16:02.974656+0000","last_became_active":"2024-09-16T05:13:25.680445+0000","last_became_peered":"2024-09-16T05:13:25.680445+0000","last_unstale":"2024-09-16T05:16:02.974656+0000","last_undegraded":"2024-09-16T05:16:02.974656+0000","last_fullsized":"2024-09-16T05:16:02.974656+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:07.531934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.531934+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:33.594849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001454288,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"3"}]},{"pgid":"1.c","version":"35'139","reported_seq":381,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.804062+0000","last_change":"2024-09-16T05:15:56.828705+0000","last_active":"2024-09-16T05:16:02.804062+0000","last_peered":"2024-09-16T05:16:02.804062+0000","last_clean":"2024-09-16T05:16:02.804062+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:16:02.804062+0000","last_undegraded":"2024-09-16T05:16:02.804062+0000","last_fullsized":"2024-09-16T05:16:02.804062+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.828595+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.828595+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:25.780000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"360'281","reported_seq":1789,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.973982+0000","last_change":"2024-09-16T05:16:02.974184+0000","last_active":"2024-09-16T05:16:02.973982+0000","last_peered":"2024-09-16T05:16:02.973982+0000","last_clean":"2024-09-16T05:16:02.973982+0000","last_became_active":"2024-09-16T05:13:25.680481+0000","last_became_peered":"2024-09-16T05:13:25.680481+0000","last_unstale":"2024-09-16T05:16:02.973982+0000","last_undegraded":"2024-09-16T05:16:02.973982+0000","last_fullsized":"2024-09-16T05:16:02.973982+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:16:02.438575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.438575+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.492441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074319000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"3"}]},{"pgid":"1.d","version":"35'139","reported_seq":545,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.716155+0000","last_change":"2024-09-16T05:15:57.170014+0000","last_active":"2024-09-16T05:16:00.716155+0000","last_peered":"2024-09-16T05:16:00.716155+0000","last_clean":"2024-09-16T05:16:00.716155+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:16:00.716155+0000","last_undegraded":"2024-09-16T05:16:00.716155+0000","last_fullsized":"2024-09-16T05:16:00.716155+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.169945+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.169945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.932365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":1634,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.712732+0000","last_change":"2024-09-16T05:16:00.713102+0000","last_active":"2024-09-16T05:16:00.712732+0000","last_peered":"2024-09-16T05:16:00.712732+0000","last_clean":"2024-09-16T05:16:00.712732+0000","last_became_active":"2024-09-16T05:14:40.261276+0000","last_became_peered":"2024-09-16T05:14:40.261276+0000","last_unstale":"2024-09-16T05:16:00.712732+0000","last_undegraded":"2024-09-16T05:16:00.712732+0000","last_fullsized":"2024-09-16T05:16:00.712732+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:15:57.720091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.720091+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:18.242117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010373680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":535,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.712249+0000","last_change":"2024-09-16T05:15:56.983440+0000","last_active":"2024-09-16T05:16:00.712249+0000","last_peered":"2024-09-16T05:16:00.712249+0000","last_clean":"2024-09-16T05:16:00.712249+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:16:00.712249+0000","last_undegraded":"2024-09-16T05:16:00.712249+0000","last_fullsized":"2024-09-16T05:16:00.712249+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.983358+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.983358+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:59.889126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"402'457","reported_seq":1934,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886756+0000","last_change":"2024-09-16T05:15:58.871337+0000","last_active":"2024-09-16T05:15:59.886756+0000","last_peered":"2024-09-16T05:15:59.886756+0000","last_clean":"2024-09-16T05:15:59.886756+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:15:59.886756+0000","last_undegraded":"2024-09-16T05:15:59.886756+0000","last_fullsized":"2024-09-16T05:15:59.886756+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'379","last_scrub_stamp":"2024-09-16T05:14:44.180613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.180613+0000","objects_scrubbed":6,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022404970000000001,"stat_sum":{"num_bytes":9462506,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":48192,"num_write":30,"num_write_kb":9875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":527,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.720041+0000","last_change":"2024-09-16T05:15:59.140824+0000","last_active":"2024-09-16T05:16:00.720041+0000","last_peered":"2024-09-16T05:16:00.720041+0000","last_clean":"2024-09-16T05:16:00.720041+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:16:00.720041+0000","last_undegraded":"2024-09-16T05:16:00.720041+0000","last_fullsized":"2024-09-16T05:16:00.720041+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.140738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.775574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"410'264","reported_seq":1755,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.342712+0000","last_change":"2024-09-16T05:16:00.790076+0000","last_active":"2024-09-16T05:16:02.342712+0000","last_peered":"2024-09-16T05:16:02.342712+0000","last_clean":"2024-09-16T05:16:02.342712+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:16:02.342712+0000","last_undegraded":"2024-09-16T05:16:02.342712+0000","last_fullsized":"2024-09-16T05:16:02.342712+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'259","last_scrub_stamp":"2024-09-16T05:15:55.772875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.772875+0000","objects_scrubbed":1,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:07.001731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057099600000000005,"stat_sum":{"num_bytes":5464170,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33268,"num_write":21,"num_write_kb":8682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1063,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.342474+0000","last_change":"2024-09-16T05:15:56.819273+0000","last_active":"2024-09-16T05:16:02.342474+0000","last_peered":"2024-09-16T05:16:02.342474+0000","last_clean":"2024-09-16T05:16:02.342474+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:16:02.342474+0000","last_undegraded":"2024-09-16T05:16:02.342474+0000","last_fullsized":"2024-09-16T05:16:02.342474+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.819151+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.819151+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:06.003491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"411'224","reported_seq":1717,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.721796+0000","last_change":"2024-09-16T05:16:00.722089+0000","last_active":"2024-09-16T05:16:00.721796+0000","last_peered":"2024-09-16T05:16:00.721796+0000","last_clean":"2024-09-16T05:16:00.721796+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:16:00.721796+0000","last_undegraded":"2024-09-16T05:16:00.721796+0000","last_fullsized":"2024-09-16T05:16:00.721796+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'222","last_scrub_stamp":"2024-09-16T05:16:00.063191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.063191+0000","objects_scrubbed":3,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:07.618646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0055275849999999998,"stat_sum":{"num_bytes":4928340,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":12880,"num_write":15,"num_write_kb":3848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.3","version":"397'99","reported_seq":1582,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.721418+0000","last_change":"2024-09-16T05:16:00.721812+0000","last_active":"2024-09-16T05:16:00.721418+0000","last_peered":"2024-09-16T05:16:00.721418+0000","last_clean":"2024-09-16T05:16:00.721418+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:16:00.721418+0000","last_undegraded":"2024-09-16T05:16:00.721418+0000","last_fullsized":"2024-09-16T05:16:00.721418+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'68","last_scrub_stamp":"2024-09-16T05:14:44.975873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:44.975873+0000","objects_scrubbed":4,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:01.744564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013287749999999999,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12729,"num_write":6,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":1054,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.716808+0000","last_change":"2024-09-16T05:15:53.271304+0000","last_active":"2024-09-16T05:16:00.716808+0000","last_peered":"2024-09-16T05:16:00.716808+0000","last_clean":"2024-09-16T05:16:00.716808+0000","last_became_active":"2024-09-16T05:15:51.717864+0000","last_became_peered":"2024-09-16T05:15:51.717864+0000","last_unstale":"2024-09-16T05:16:00.716808+0000","last_undegraded":"2024-09-16T05:16:00.716808+0000","last_fullsized":"2024-09-16T05:16:00.716808+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.271201+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.271201+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.039968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1501,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.720362+0000","last_change":"2024-09-16T05:16:00.720599+0000","last_active":"2024-09-16T05:16:00.720362+0000","last_peered":"2024-09-16T05:16:00.720362+0000","last_clean":"2024-09-16T05:16:00.720362+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:16:00.720362+0000","last_undegraded":"2024-09-16T05:16:00.720362+0000","last_fullsized":"2024-09-16T05:16:00.720362+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:55.047148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.047148+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:55.725580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044183599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":693,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861388+0000","last_change":"2024-09-16T05:15:54.643909+0000","last_active":"2024-09-16T05:15:59.861388+0000","last_peered":"2024-09-16T05:15:59.861388+0000","last_clean":"2024-09-16T05:15:59.861388+0000","last_became_active":"2024-09-16T05:15:51.723644+0000","last_became_peered":"2024-09-16T05:15:51.723644+0000","last_unstale":"2024-09-16T05:15:59.861388+0000","last_undegraded":"2024-09-16T05:15:59.861388+0000","last_fullsized":"2024-09-16T05:15:59.861388+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:54.643828+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:54.643828+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:59.687807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1049,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.715792+0000","last_change":"2024-09-16T05:15:54.232505+0000","last_active":"2024-09-16T05:16:00.715792+0000","last_peered":"2024-09-16T05:16:00.715792+0000","last_clean":"2024-09-16T05:16:00.715792+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:16:00.715792+0000","last_undegraded":"2024-09-16T05:16:00.715792+0000","last_fullsized":"2024-09-16T05:16:00.715792+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:54.232382+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:54.232382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:02.812437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1446,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.712394+0000","last_change":"2024-09-16T05:16:00.712672+0000","last_active":"2024-09-16T05:16:00.712394+0000","last_peered":"2024-09-16T05:16:00.712394+0000","last_clean":"2024-09-16T05:16:00.712394+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:16:00.712394+0000","last_undegraded":"2024-09-16T05:16:00.712394+0000","last_fullsized":"2024-09-16T05:16:00.712394+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:07.898675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.898675+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:29.210450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061349899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1062,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.804213+0000","last_change":"2024-09-16T05:15:52.844991+0000","last_active":"2024-09-16T05:16:02.804213+0000","last_peered":"2024-09-16T05:16:02.804213+0000","last_clean":"2024-09-16T05:16:02.804213+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:16:02.804213+0000","last_undegraded":"2024-09-16T05:16:02.804213+0000","last_fullsized":"2024-09-16T05:16:02.804213+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:52.844849+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:52.844849+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:03.165293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1521,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.720105+0000","last_change":"2024-09-16T05:16:00.720647+0000","last_active":"2024-09-16T05:16:00.720105+0000","last_peered":"2024-09-16T05:16:00.720105+0000","last_clean":"2024-09-16T05:16:00.720105+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:16:00.720105+0000","last_undegraded":"2024-09-16T05:16:00.720105+0000","last_fullsized":"2024-09-16T05:16:00.720105+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:06.755407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.755407+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:14.754156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003703638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":736,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886516+0000","last_change":"2024-09-16T05:15:58.871892+0000","last_active":"2024-09-16T05:15:59.886516+0000","last_peered":"2024-09-16T05:15:59.886516+0000","last_clean":"2024-09-16T05:15:59.886516+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:15:59.886516+0000","last_undegraded":"2024-09-16T05:15:59.886516+0000","last_fullsized":"2024-09-16T05:15:59.886516+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:31.009189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.009189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:43.092778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027044700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":1018,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.342760+0000","last_change":"2024-09-16T05:15:53.804026+0000","last_active":"2024-09-16T05:16:02.342760+0000","last_peered":"2024-09-16T05:16:02.342760+0000","last_clean":"2024-09-16T05:16:02.342760+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:16:02.342760+0000","last_undegraded":"2024-09-16T05:16:02.342760+0000","last_fullsized":"2024-09-16T05:16:02.342760+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.803903+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.803903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:57.012754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1494,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876575+0000","last_change":"2024-09-16T05:15:58.873615+0000","last_active":"2024-09-16T05:15:59.876575+0000","last_peered":"2024-09-16T05:15:59.876575+0000","last_clean":"2024-09-16T05:15:59.876575+0000","last_became_active":"2024-09-16T05:14:25.821885+0000","last_became_peered":"2024-09-16T05:14:25.821885+0000","last_unstale":"2024-09-16T05:15:59.876575+0000","last_undegraded":"2024-09-16T05:15:59.876575+0000","last_fullsized":"2024-09-16T05:15:59.876575+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:15:35.928579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:35.928579+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:55.052984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011433540000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":798,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861476+0000","last_change":"2024-09-16T05:15:56.607423+0000","last_active":"2024-09-16T05:15:59.861476+0000","last_peered":"2024-09-16T05:15:59.861476+0000","last_clean":"2024-09-16T05:15:59.861476+0000","last_became_active":"2024-09-16T05:15:51.724156+0000","last_became_peered":"2024-09-16T05:15:51.724156+0000","last_unstale":"2024-09-16T05:15:59.861476+0000","last_undegraded":"2024-09-16T05:15:59.861476+0000","last_fullsized":"2024-09-16T05:15:59.861476+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.607332+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.607332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:04.191878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"408'480","reported_seq":1951,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886644+0000","last_change":"2024-09-16T05:15:58.872388+0000","last_active":"2024-09-16T05:15:59.886644+0000","last_peered":"2024-09-16T05:15:59.886644+0000","last_clean":"2024-09-16T05:15:59.886644+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:15:59.886644+0000","last_undegraded":"2024-09-16T05:15:59.886644+0000","last_fullsized":"2024-09-16T05:15:59.886644+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'479","last_scrub_stamp":"2024-09-16T05:15:57.067519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.067519+0000","objects_scrubbed":2,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.468213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031323589999999999,"stat_sum":{"num_bytes":3176556,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":8850,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":694,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861188+0000","last_change":"2024-09-16T05:15:55.644520+0000","last_active":"2024-09-16T05:15:59.861188+0000","last_peered":"2024-09-16T05:15:59.861188+0000","last_clean":"2024-09-16T05:15:59.861188+0000","last_became_active":"2024-09-16T05:15:51.722851+0000","last_became_peered":"2024-09-16T05:15:51.722851+0000","last_unstale":"2024-09-16T05:15:59.861188+0000","last_undegraded":"2024-09-16T05:15:59.861188+0000","last_fullsized":"2024-09-16T05:15:59.861188+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:55.644436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.644436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:05.843216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1576,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876419+0000","last_change":"2024-09-16T05:15:58.873082+0000","last_active":"2024-09-16T05:15:59.876419+0000","last_peered":"2024-09-16T05:15:59.876419+0000","last_clean":"2024-09-16T05:15:59.876419+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:15:59.876419+0000","last_undegraded":"2024-09-16T05:15:59.876419+0000","last_fullsized":"2024-09-16T05:15:59.876419+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:57.869742+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.869742+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:15.009071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00084464000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":878,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861467+0000","last_change":"2024-09-16T05:15:53.643617+0000","last_active":"2024-09-16T05:15:59.861467+0000","last_peered":"2024-09-16T05:15:59.861467+0000","last_clean":"2024-09-16T05:15:59.861467+0000","last_became_active":"2024-09-16T05:15:51.718810+0000","last_became_peered":"2024-09-16T05:15:51.718810+0000","last_unstale":"2024-09-16T05:15:59.861467+0000","last_undegraded":"2024-09-16T05:15:59.861467+0000","last_fullsized":"2024-09-16T05:15:59.861467+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.643528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.643528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:08.515058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":1684,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.973501+0000","last_change":"2024-09-16T05:16:02.973669+0000","last_active":"2024-09-16T05:16:02.973501+0000","last_peered":"2024-09-16T05:16:02.973501+0000","last_clean":"2024-09-16T05:16:02.973501+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:16:02.973501+0000","last_undegraded":"2024-09-16T05:16:02.973501+0000","last_fullsized":"2024-09-16T05:16:02.973501+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:15:41.633260+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:44.276112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036784600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"3"}]},{"pgid":"1.8","version":"35'139","reported_seq":1047,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.720029+0000","last_change":"2024-09-16T05:15:15.084154+0000","last_active":"2024-09-16T05:16:00.720029+0000","last_peered":"2024-09-16T05:16:00.720029+0000","last_clean":"2024-09-16T05:16:00.720029+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:16:00.720029+0000","last_undegraded":"2024-09-16T05:16:00.720029+0000","last_fullsized":"2024-09-16T05:16:00.720029+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:15.084022+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.084022+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:18.557692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"402'463","reported_seq":1962,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.342479+0000","last_change":"2024-09-16T05:16:00.789847+0000","last_active":"2024-09-16T05:16:02.342479+0000","last_peered":"2024-09-16T05:16:02.342479+0000","last_clean":"2024-09-16T05:16:02.342479+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:16:02.342479+0000","last_undegraded":"2024-09-16T05:16:02.342479+0000","last_fullsized":"2024-09-16T05:16:02.342479+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'419","last_scrub_stamp":"2024-09-16T05:14:25.924121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.924121+0000","objects_scrubbed":4,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063694499999999998,"stat_sum":{"num_bytes":5126867,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":30338,"num_write":5,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1037,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.715798+0000","last_change":"2024-09-16T05:14:11.602321+0000","last_active":"2024-09-16T05:16:00.715798+0000","last_peered":"2024-09-16T05:16:00.715798+0000","last_clean":"2024-09-16T05:16:00.715798+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:16:00.715798+0000","last_undegraded":"2024-09-16T05:16:00.715798+0000","last_fullsized":"2024-09-16T05:16:00.715798+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:11.602237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:11.602237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1713,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.876188+0000","last_change":"2024-09-16T05:15:58.872998+0000","last_active":"2024-09-16T05:15:59.876188+0000","last_peered":"2024-09-16T05:15:59.876188+0000","last_clean":"2024-09-16T05:15:59.876188+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:15:59.876188+0000","last_undegraded":"2024-09-16T05:15:59.876188+0000","last_fullsized":"2024-09-16T05:15:59.876188+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:15:32.922163+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:32.922163+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:49.558268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079339099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1045,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886182+0000","last_change":"2024-09-16T05:15:14.182510+0000","last_active":"2024-09-16T05:15:59.886182+0000","last_peered":"2024-09-16T05:15:59.886182+0000","last_clean":"2024-09-16T05:15:59.886182+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:15:59.886182+0000","last_undegraded":"2024-09-16T05:15:59.886182+0000","last_fullsized":"2024-09-16T05:15:59.886182+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:14.182407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:14.182407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.321475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":532,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.873347+0000","last_change":"2024-09-16T05:15:56.743121+0000","last_active":"2024-09-16T05:15:59.873347+0000","last_peered":"2024-09-16T05:15:59.873347+0000","last_clean":"2024-09-16T05:15:59.873347+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:15:59.873347+0000","last_undegraded":"2024-09-16T05:15:59.873347+0000","last_fullsized":"2024-09-16T05:15:59.873347+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.743052+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.743052+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:06.703550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":613,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861346+0000","last_change":"2024-09-16T05:15:59.249616+0000","last_active":"2024-09-16T05:15:59.861346+0000","last_peered":"2024-09-16T05:15:59.861346+0000","last_clean":"2024-09-16T05:15:59.861346+0000","last_became_active":"2024-09-16T05:13:31.747644+0000","last_became_peered":"2024-09-16T05:13:31.747644+0000","last_unstale":"2024-09-16T05:15:59.861346+0000","last_undegraded":"2024-09-16T05:15:59.861346+0000","last_fullsized":"2024-09-16T05:15:59.861346+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:14:49.576522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.576522+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:10.161242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064267100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1054,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.719873+0000","last_change":"2024-09-16T05:15:58.103711+0000","last_active":"2024-09-16T05:16:00.719873+0000","last_peered":"2024-09-16T05:16:00.719873+0000","last_clean":"2024-09-16T05:16:00.719873+0000","last_became_active":"2024-09-16T05:15:56.719062+0000","last_became_peered":"2024-09-16T05:15:56.719062+0000","last_unstale":"2024-09-16T05:16:00.719873+0000","last_undegraded":"2024-09-16T05:16:00.719873+0000","last_fullsized":"2024-09-16T05:16:00.719873+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:58.103601+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:58.103601+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:27.359233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":1106,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886431+0000","last_change":"2024-09-16T05:15:58.870202+0000","last_active":"2024-09-16T05:15:59.886431+0000","last_peered":"2024-09-16T05:15:59.886431+0000","last_clean":"2024-09-16T05:15:59.886431+0000","last_became_active":"2024-09-16T05:13:25.681297+0000","last_became_peered":"2024-09-16T05:13:25.681297+0000","last_unstale":"2024-09-16T05:15:59.886431+0000","last_undegraded":"2024-09-16T05:15:59.886431+0000","last_fullsized":"2024-09-16T05:15:59.886431+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:49.160833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.160833+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.702326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001587532,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.14","version":"402'269","reported_seq":1353,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.887913+0000","last_change":"2024-09-16T05:15:58.870912+0000","last_active":"2024-09-16T05:15:59.887913+0000","last_peered":"2024-09-16T05:15:59.887913+0000","last_clean":"2024-09-16T05:15:59.887913+0000","last_became_active":"2024-09-16T05:13:25.684888+0000","last_became_peered":"2024-09-16T05:13:25.684888+0000","last_unstale":"2024-09-16T05:15:59.887913+0000","last_undegraded":"2024-09-16T05:15:59.887913+0000","last_fullsized":"2024-09-16T05:15:59.887913+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"342'239","last_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.110587+0000","objects_scrubbed":4,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:21.577677+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0021442060000000001,"stat_sum":{"num_bytes":3142942,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":40956,"num_write":7,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.15","version":"409'479","reported_seq":1964,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886563+0000","last_change":"2024-09-16T05:15:58.872869+0000","last_active":"2024-09-16T05:15:59.886563+0000","last_peered":"2024-09-16T05:15:59.886563+0000","last_clean":"2024-09-16T05:15:59.886563+0000","last_became_active":"2024-09-16T05:13:25.680026+0000","last_became_peered":"2024-09-16T05:13:25.680026+0000","last_unstale":"2024-09-16T05:15:59.886563+0000","last_undegraded":"2024-09-16T05:15:59.886563+0000","last_fullsized":"2024-09-16T05:15:59.886563+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"357'436","last_scrub_stamp":"2024-09-16T05:15:07.101221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.101221+0000","objects_scrubbed":5,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:12.905653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039874230000000004,"stat_sum":{"num_bytes":9168119,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":52138,"num_write":31,"num_write_kb":15087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.16","version":"402'122","reported_seq":1252,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.721630+0000","last_change":"2024-09-16T05:16:00.721858+0000","last_active":"2024-09-16T05:16:00.721630+0000","last_peered":"2024-09-16T05:16:00.721630+0000","last_clean":"2024-09-16T05:16:00.721630+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:16:00.721630+0000","last_undegraded":"2024-09-16T05:16:00.721630+0000","last_fullsized":"2024-09-16T05:16:00.721630+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'120","last_scrub_stamp":"2024-09-16T05:15:42.209530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:42.209530+0000","objects_scrubbed":2,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:58.075602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001332118,"stat_sum":{"num_bytes":3142942,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":12296,"num_write":5,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":378,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.711929+0000","last_change":"2024-09-16T05:15:56.762490+0000","last_active":"2024-09-16T05:16:00.711929+0000","last_peered":"2024-09-16T05:16:00.711929+0000","last_clean":"2024-09-16T05:16:00.711929+0000","last_became_active":"2024-09-16T05:15:56.718004+0000","last_became_peered":"2024-09-16T05:15:56.718004+0000","last_unstale":"2024-09-16T05:16:00.711929+0000","last_undegraded":"2024-09-16T05:16:00.711929+0000","last_fullsized":"2024-09-16T05:16:00.711929+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.762419+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.762419+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:08.231927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1297,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:01.171239+0000","last_change":"2024-09-16T05:16:01.171532+0000","last_active":"2024-09-16T05:16:01.171239+0000","last_peered":"2024-09-16T05:16:01.171239+0000","last_clean":"2024-09-16T05:16:01.171239+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:16:01.171239+0000","last_undegraded":"2024-09-16T05:16:01.171239+0000","last_fullsized":"2024-09-16T05:16:01.171239+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:57.269939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.269939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:25.998568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012329120000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1499,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.885955+0000","last_change":"2024-09-16T05:15:58.871673+0000","last_active":"2024-09-16T05:15:59.885955+0000","last_peered":"2024-09-16T05:15:59.885955+0000","last_clean":"2024-09-16T05:15:59.885955+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:15:59.885955+0000","last_undegraded":"2024-09-16T05:15:59.885955+0000","last_fullsized":"2024-09-16T05:15:59.885955+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:17.188273+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:17.188273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:37.373260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025552610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":1110,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.804084+0000","last_change":"2024-09-16T05:16:00.718249+0000","last_active":"2024-09-16T05:16:02.804084+0000","last_peered":"2024-09-16T05:16:02.804084+0000","last_clean":"2024-09-16T05:16:02.804084+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:16:02.804084+0000","last_undegraded":"2024-09-16T05:16:02.804084+0000","last_fullsized":"2024-09-16T05:16:02.804084+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:06.893298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.893298+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:35.167056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001157656,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.1a","version":"413'637","reported_seq":2105,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.975633+0000","last_change":"2024-09-16T05:16:02.975911+0000","last_active":"2024-09-16T05:16:02.975633+0000","last_peered":"2024-09-16T05:16:02.975633+0000","last_clean":"2024-09-16T05:16:02.975633+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:16:02.975633+0000","last_undegraded":"2024-09-16T05:16:02.975633+0000","last_fullsized":"2024-09-16T05:16:02.975633+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'527","last_scrub_stamp":"2024-09-16T05:15:10.511970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.511970+0000","objects_scrubbed":9,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:12.332375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025610020000000002,"stat_sum":{"num_bytes":15285983,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":66,"num_read_kb":57717,"num_write":45,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"3"}]},{"pgid":"2.1b","version":"408'340","reported_seq":1861,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.342648+0000","last_change":"2024-09-16T05:16:00.789950+0000","last_active":"2024-09-16T05:16:02.342648+0000","last_peered":"2024-09-16T05:16:02.342648+0000","last_clean":"2024-09-16T05:16:02.342648+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:16:02.342648+0000","last_undegraded":"2024-09-16T05:16:02.342648+0000","last_fullsized":"2024-09-16T05:16:02.342648+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"347'319","last_scrub_stamp":"2024-09-16T05:15:08.957118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.957118+0000","objects_scrubbed":3,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:34.283087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066995599999999998,"stat_sum":{"num_bytes":9012449,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":29314,"num_write":14,"num_write_kb":3360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.1c","version":"410'375","reported_seq":1916,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886765+0000","last_change":"2024-09-16T05:15:59.249489+0000","last_active":"2024-09-16T05:15:59.886765+0000","last_peered":"2024-09-16T05:15:59.886765+0000","last_clean":"2024-09-16T05:15:59.886765+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:15:59.886765+0000","last_undegraded":"2024-09-16T05:15:59.886765+0000","last_fullsized":"2024-09-16T05:15:59.886765+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'364","last_scrub_stamp":"2024-09-16T05:15:08.759106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.759106+0000","objects_scrubbed":2,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:36.621444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061224700000000001,"stat_sum":{"num_bytes":3327852,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.1d","version":"409'655","reported_seq":2217,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861515+0000","last_change":"2024-09-16T05:15:59.251454+0000","last_active":"2024-09-16T05:15:59.861515+0000","last_peered":"2024-09-16T05:15:59.861515+0000","last_clean":"2024-09-16T05:15:59.861515+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:15:59.861515+0000","last_undegraded":"2024-09-16T05:15:59.861515+0000","last_fullsized":"2024-09-16T05:15:59.861515+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'584","last_scrub_stamp":"2024-09-16T05:15:16.833446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:16.833446+0000","objects_scrubbed":8,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:29.188855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024565530000000002,"stat_sum":{"num_bytes":14498349,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":48,"num_read_kb":41645,"num_write":37,"num_write_kb":18914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.1e","version":"378'347","reported_seq":1857,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.804276+0000","last_change":"2024-09-16T05:16:00.718579+0000","last_active":"2024-09-16T05:16:02.804276+0000","last_peered":"2024-09-16T05:16:02.804276+0000","last_clean":"2024-09-16T05:16:02.804276+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:16:02.804276+0000","last_undegraded":"2024-09-16T05:16:02.804276+0000","last_fullsized":"2024-09-16T05:16:02.804276+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"362'335","last_scrub_stamp":"2024-09-16T05:15:12.926534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:12.926534+0000","objects_scrubbed":3,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:21.999269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063253799999999998,"stat_sum":{"num_bytes":2487502,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":26093,"num_write":5,"num_write_kb":3828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":1714,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.873517+0000","last_change":"2024-09-16T05:15:59.249429+0000","last_active":"2024-09-16T05:15:59.873517+0000","last_peered":"2024-09-16T05:15:59.873517+0000","last_clean":"2024-09-16T05:15:59.873517+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:15:59.873517+0000","last_undegraded":"2024-09-16T05:15:59.873517+0000","last_fullsized":"2024-09-16T05:15:59.873517+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:14:52.883562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.883562+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:02.165046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00028620300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.20","version":"411'144","reported_seq":1656,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.719709+0000","last_change":"2024-09-16T05:16:00.720336+0000","last_active":"2024-09-16T05:16:00.719709+0000","last_peered":"2024-09-16T05:16:00.719709+0000","last_clean":"2024-09-16T05:16:00.719709+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:16:00.719709+0000","last_undegraded":"2024-09-16T05:16:00.719709+0000","last_fullsized":"2024-09-16T05:16:00.719709+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'128","last_scrub_stamp":"2024-09-16T05:15:31.623695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.623695+0000","objects_scrubbed":2,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:33.722769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041425450000000001,"stat_sum":{"num_bytes":7381279,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":59,"num_read_kb":79406,"num_write":17,"num_write_kb":9878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":1445,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.712314+0000","last_change":"2024-09-16T05:16:00.712697+0000","last_active":"2024-09-16T05:16:00.712314+0000","last_peered":"2024-09-16T05:16:00.712314+0000","last_clean":"2024-09-16T05:16:00.712314+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:16:00.712314+0000","last_undegraded":"2024-09-16T05:16:00.712314+0000","last_fullsized":"2024-09-16T05:16:00.712314+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.848788+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:05.237703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065134199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":621,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.861397+0000","last_change":"2024-09-16T05:15:59.251173+0000","last_active":"2024-09-16T05:15:59.861397+0000","last_peered":"2024-09-16T05:15:59.861397+0000","last_clean":"2024-09-16T05:15:59.861397+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:15:59.861397+0000","last_undegraded":"2024-09-16T05:15:59.861397+0000","last_fullsized":"2024-09-16T05:15:59.861397+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:31.582158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.582158+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:01.090523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019768400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":977,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.804308+0000","last_change":"2024-09-16T05:16:00.717840+0000","last_active":"2024-09-16T05:16:02.804308+0000","last_peered":"2024-09-16T05:16:02.804308+0000","last_clean":"2024-09-16T05:16:02.804308+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:16:02.804308+0000","last_undegraded":"2024-09-16T05:16:02.804308+0000","last_fullsized":"2024-09-16T05:16:02.804308+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:14:46.760890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.760890+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094726800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":793,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886263+0000","last_change":"2024-09-16T05:15:59.030446+0000","last_active":"2024-09-16T05:15:59.886263+0000","last_peered":"2024-09-16T05:15:59.886263+0000","last_clean":"2024-09-16T05:15:59.886263+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:15:59.886263+0000","last_undegraded":"2024-09-16T05:15:59.886263+0000","last_fullsized":"2024-09-16T05:15:59.886263+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.030307+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:22.123252+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001943761,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.25","version":"413'654","reported_seq":2091,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.976266+0000","last_change":"2024-09-16T05:16:02.976416+0000","last_active":"2024-09-16T05:16:02.976266+0000","last_peered":"2024-09-16T05:16:02.976266+0000","last_clean":"2024-09-16T05:16:02.976266+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:16:02.976266+0000","last_undegraded":"2024-09-16T05:16:02.976266+0000","last_fullsized":"2024-09-16T05:16:02.976266+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'624","last_scrub_stamp":"2024-09-16T05:15:35.625076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:35.625076+0000","objects_scrubbed":6,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:47.664647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00275779,"stat_sum":{"num_bytes":20891332,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":46,"num_read_kb":57576,"num_write":37,"num_write_kb":13760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"3"}]},{"pgid":"2.26","version":"116'9","reported_seq":785,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.722137+0000","last_change":"2024-09-16T05:16:00.722402+0000","last_active":"2024-09-16T05:16:00.722137+0000","last_peered":"2024-09-16T05:16:00.722137+0000","last_clean":"2024-09-16T05:16:00.722137+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:16:00.722137+0000","last_undegraded":"2024-09-16T05:16:00.722137+0000","last_fullsized":"2024-09-16T05:16:00.722137+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:14:27.631576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:27.631576+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049844199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":797,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.720438+0000","last_change":"2024-09-16T05:16:00.720746+0000","last_active":"2024-09-16T05:16:00.720438+0000","last_peered":"2024-09-16T05:16:00.720438+0000","last_clean":"2024-09-16T05:16:00.720438+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:16:00.720438+0000","last_undegraded":"2024-09-16T05:16:00.720438+0000","last_fullsized":"2024-09-16T05:16:00.720438+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:30.713398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.713398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039960710000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1586,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.804043+0000","last_change":"2024-09-16T05:16:00.717321+0000","last_active":"2024-09-16T05:16:02.804043+0000","last_peered":"2024-09-16T05:16:02.804043+0000","last_clean":"2024-09-16T05:16:02.804043+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:16:02.804043+0000","last_undegraded":"2024-09-16T05:16:02.804043+0000","last_fullsized":"2024-09-16T05:16:02.804043+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:15:57.809122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.809122+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:15.638927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070356300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.29","version":"411'564","reported_seq":2092,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:01.183729+0000","last_change":"2024-09-16T05:16:01.172775+0000","last_active":"2024-09-16T05:16:01.183729+0000","last_peered":"2024-09-16T05:16:01.183729+0000","last_clean":"2024-09-16T05:16:01.183729+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:16:01.183729+0000","last_undegraded":"2024-09-16T05:16:01.183729+0000","last_fullsized":"2024-09-16T05:16:01.183729+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'536","last_scrub_stamp":"2024-09-16T05:15:41.155545+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:41.155545+0000","objects_scrubbed":6,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:05.291864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027924149999999999,"stat_sum":{"num_bytes":22271841,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":73,"num_read_kb":64374,"num_write":79,"num_write_kb":27830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1787,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.716118+0000","last_change":"2024-09-16T05:16:00.716430+0000","last_active":"2024-09-16T05:16:00.716118+0000","last_peered":"2024-09-16T05:16:00.716118+0000","last_clean":"2024-09-16T05:16:00.716118+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:16:00.716118+0000","last_undegraded":"2024-09-16T05:16:00.716118+0000","last_fullsized":"2024-09-16T05:16:00.716118+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:14:33.725061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:33.725061+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052174000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.2b","version":"406'216","reported_seq":1618,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.712804+0000","last_change":"2024-09-16T05:16:00.713267+0000","last_active":"2024-09-16T05:16:00.712804+0000","last_peered":"2024-09-16T05:16:00.712804+0000","last_clean":"2024-09-16T05:16:00.712804+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:16:00.712804+0000","last_undegraded":"2024-09-16T05:16:00.712804+0000","last_fullsized":"2024-09-16T05:16:00.712804+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"330'198","last_scrub_stamp":"2024-09-16T05:14:46.815272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.815272+0000","objects_scrubbed":2,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:12.098043+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001042992,"stat_sum":{"num_bytes":4533234,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":13,"num_write_kb":8806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":1631,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-09-16T05:16:02.974254+0000","last_change":"2024-09-16T05:16:02.974565+0000","last_active":"2024-09-16T05:16:02.974254+0000","last_peered":"2024-09-16T05:16:02.974254+0000","last_clean":"2024-09-16T05:16:02.974254+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:16:02.974254+0000","last_undegraded":"2024-09-16T05:16:02.974254+0000","last_fullsized":"2024-09-16T05:16:02.974254+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:15:39.640149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.640149+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:06.569144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010561979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"3"}]},{"pgid":"2.2d","version":"377'326","reported_seq":1791,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.718453+0000","last_change":"2024-09-16T05:16:00.718750+0000","last_active":"2024-09-16T05:16:00.718453+0000","last_peered":"2024-09-16T05:16:00.718453+0000","last_clean":"2024-09-16T05:16:00.718453+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:16:00.718453+0000","last_undegraded":"2024-09-16T05:16:00.718453+0000","last_fullsized":"2024-09-16T05:16:00.718453+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"274'307","last_scrub_stamp":"2024-09-16T05:14:30.116429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.116429+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":444,"objects_trimmed":0,"snaptrim_duration":0.0024054269999999999,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":12729,"num_write":7,"num_write_kb":4589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.2e","version":"411'402","reported_seq":1862,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-16T05:16:00.723174+0000","last_change":"2024-09-16T05:16:00.723347+0000","last_active":"2024-09-16T05:16:00.723174+0000","last_peered":"2024-09-16T05:16:00.723174+0000","last_clean":"2024-09-16T05:16:00.723174+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:16:00.723174+0000","last_undegraded":"2024-09-16T05:16:00.723174+0000","last_fullsized":"2024-09-16T05:16:00.723174+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"337'278","last_scrub_stamp":"2024-09-16T05:14:47.811642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.811642+0000","objects_scrubbed":7,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:03.050471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0062516660000000003,"stat_sum":{"num_bytes":11502913,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":32166,"num_write":35,"num_write_kb":10854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8b","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.2f","version":"387'306","reported_seq":1824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-16T05:15:59.886348+0000","last_change":"2024-09-16T05:15:58.872116+0000","last_active":"2024-09-16T05:15:59.886348+0000","last_peered":"2024-09-16T05:15:59.886348+0000","last_clean":"2024-09-16T05:15:59.886348+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:15:59.886348+0000","last_undegraded":"2024-09-16T05:15:59.886348+0000","last_fullsized":"2024-09-16T05:15:59.886348+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'277","last_scrub_stamp":"2024-09-16T05:14:53.110450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.110450+0000","objects_scrubbed":3,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029237880000000001,"stat_sum":{"num_bytes":6521248,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":13580,"num_write":12,"num_write_kb":6438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7d","length":"1"},{"start":"81","length":"1"},{"start":"87","length":"1"},{"start":"8d","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":233110827,"num_objects":100,"num_object_clones":51,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":8,"num_read":881,"num_read_kb":921449,"num_write":579,"num_write_kb":249400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":287682560,"data_stored":345897346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15761,"ondisk_log_size":15761,"up":152,"acting":152,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089427,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78436,"kb_used_data":38920,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93666716,"statfs":{"total":95995035648,"available":95914717184,"internally_reserved":0,"allocated":39854080,"data_stored":41789381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285656,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56060,"kb_used_data":20704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93689092,"statfs":{"total":95995035648,"available":95937630208,"internally_reserved":0,"allocated":21200896,"data_stored":24647920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514592,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81756,"kb_used_data":42248,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93663396,"statfs":{"total":95995035648,"available":95911317504,"internally_reserved":0,"allocated":43261952,"data_stored":49256400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841633,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57476,"kb_used_data":22120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93687676,"statfs":{"total":95995035648,"available":95936180224,"internally_reserved":0,"allocated":22650880,"data_stored":26734129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103270,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79264,"kb_used_data":39748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93665888,"statfs":{"total":95995035648,"available":95913869312,"internally_reserved":0,"allocated":40701952,"data_stored":46740178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397614,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76312,"kb_used_data":36804,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93668840,"statfs":{"total":95995035648,"available":95916892160,"internally_reserved":0,"allocated":37687296,"data_stored":46425953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757360,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54972,"kb_used_data":23780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93690180,"statfs":{"total":95995035648,"available":95938744320,"internally_reserved":0,"allocated":24350720,"data_stored":27823358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920886,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55116,"kb_used_data":19764,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690036,"statfs":{"total":95995035648,"available":95938596864,"internally_reserved":0,"allocated":20238336,"data_stored":19774978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117115,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84272,"kb_used_data":44756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93660880,"statfs":{"total":95995035648,"available":95908741120,"internally_reserved":0,"allocated":45830144,"data_stored":51021381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240402,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44688,"kb_used_data":16568,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93700464,"statfs":{"total":95995035648,"available":95949275136,"internally_reserved":0,"allocated":16965632,"data_stored":17315164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53508,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93691644,"statfs":{"total":95995035648,"available":95940243456,"internally_reserved":0,"allocated":18595840,"data_stored":17577125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738506,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51032,"kb_used_data":11524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93694120,"statfs":{"total":95995035648,"available":95942778880,"internally_reserved":0,"allocated":11800576,"data_stored":10884628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8331264,"data_stored":9543862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13709312,"data_stored":14432935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11788288,"data_stored":13891234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40591360,"data_stored":47532459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15339520,"data_stored":16622293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19259392,"data_stored":24477920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32776192,"data_stored":43264711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38563840,"data_stored":46348886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":17715200,"data_stored":23555957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":38363136,"data_stored":46103715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":16302080,"data_stored":21495235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":34942976,"data_stored":38628139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:16:05.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:16:05.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:05.910 INFO:tasks.rados.rados.2.smithi204.stdout:1310: finishing write tid 1 to smithi20457400-9 2024-09-16T05:16:05.910 INFO:tasks.rados.rados.2.smithi204.stdout:1310: finishing write tid 2 to smithi20457400-9 2024-09-16T05:16:05.910 INFO:tasks.rados.rados.2.smithi204.stdout:1310: finishing write tid 3 to smithi20457400-9 2024-09-16T05:16:05.910 INFO:tasks.rados.rados.2.smithi204.stdout:1310: finishing write tid 4 to smithi20457400-9 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:1310: finishing write tid 5 to smithi20457400-9 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 577 (ObjNum 465 snap 146 seq_num 465) dirty exists 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:1310: left oid 9 (ObjNum 465 snap 146 seq_num 465) 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:1311: finishing write tid 1 to smithi20457400-22 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:1311: finishing write tid 2 to smithi20457400-22 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:1311: finishing write tid 3 to smithi20457400-22 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:1311: finishing write tid 4 to smithi20457400-22 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:1311: finishing write tid 5 to smithi20457400-22 2024-09-16T05:16:05.911 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 462 (ObjNum 466 snap 146 seq_num 466) dirty exists 2024-09-16T05:16:05.912 INFO:tasks.rados.rados.2.smithi204.stdout:1311: left oid 22 (ObjNum 466 snap 146 seq_num 466) 2024-09-16T05:16:05.912 INFO:tasks.rados.rados.2.smithi204.stdout:1312: expect (ObjNum 374 snap 109 seq_num 374) 2024-09-16T05:16:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:05.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:05.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:05.980 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:16:05.981 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:16:05.981 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:16:05.981 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:16:06.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:06.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:06.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:06.103 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:06.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:06.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:06.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:06.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:05 smithi032 ceph-mon[27388]: pgmap v783: 97 pgs: 97 active+clean; 223 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2024-09-16T05:16:06.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:05 smithi032 ceph-mon[27388]: osdmap e415: 12 total, 12 up, 12 in 2024-09-16T05:16:06.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:05 smithi032 ceph-mon[27388]: osdmap e416: 12 total, 12 up, 12 in 2024-09-16T05:16:06.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:05 smithi204 ceph-mon[30933]: pgmap v783: 97 pgs: 97 active+clean; 223 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2024-09-16T05:16:06.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:05 smithi204 ceph-mon[30933]: osdmap e415: 12 total, 12 up, 12 in 2024-09-16T05:16:06.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:05 smithi204 ceph-mon[30933]: osdmap e416: 12 total, 12 up, 12 in 2024-09-16T05:16:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:05 smithi196 ceph-mon[30802]: pgmap v783: 97 pgs: 97 active+clean; 223 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2024-09-16T05:16:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:05 smithi196 ceph-mon[30802]: osdmap e415: 12 total, 12 up, 12 in 2024-09-16T05:16:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:05 smithi196 ceph-mon[30802]: osdmap e416: 12 total, 12 up, 12 in 2024-09-16T05:16:06.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:06.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:06.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:06.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:06.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:06.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:06.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:06.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:16:06.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:06.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:16:06.575 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:06.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:06.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:06.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:06.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:06.719 INFO:tasks.rados.rados.2.smithi204.stdout:1309: done (4 left) 2024-09-16T05:16:06.719 INFO:tasks.rados.rados.2.smithi204.stdout:1310: done (3 left) 2024-09-16T05:16:06.719 INFO:tasks.rados.rados.2.smithi204.stdout:1311: done (2 left) 2024-09-16T05:16:06.719 INFO:tasks.rados.rados.2.smithi204.stdout:1312: done (1 left) 2024-09-16T05:16:06.719 INFO:tasks.rados.rados.2.smithi204.stdout:1313: done (0 left) 2024-09-16T05:16:06.719 INFO:tasks.rados.rados.2.smithi204.stdout:1314: snap_remove snap 146 2024-09-16T05:16:06.765 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:06.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:06.808 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:16:06.829 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:06.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:06.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:06.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:06.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:07.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:07.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:07.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:06 smithi032 ceph-mon[27388]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:16:07.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:06 smithi032 ceph-mon[27388]: osdmap e417: 12 total, 12 up, 12 in 2024-09-16T05:16:07.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/722493873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:06 smithi204 ceph-mon[30933]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:16:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:06 smithi204 ceph-mon[30933]: osdmap e417: 12 total, 12 up, 12 in 2024-09-16T05:16:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/722493873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:07.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:07.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:07.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:07.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:06 smithi196 ceph-mon[30802]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:16:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:06 smithi196 ceph-mon[30802]: osdmap e417: 12 total, 12 up, 12 in 2024-09-16T05:16:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/722493873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:07.263 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:07.264 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:07.285 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:07.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:07.306 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:07.306 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:07.326 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:07.326 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:07.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:07.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:07.348 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:07.348 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:07.369 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:16:07.369 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:16:07.369 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:16:07.369 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:16:07.369 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:16:07.370 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:07.370 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:07.391 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:07.391 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:07.413 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:07.413 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:07.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:07.435 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:07.435 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:07.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:07.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:07.456 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:07.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:07.478 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:07.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:07.499 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:07.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:07.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:07.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:07.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:07.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:07.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:07.715 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:16:07.718 INFO:tasks.rados.rados.2.smithi204.stdout:1314: done (0 left) 2024-09-16T05:16:07.718 INFO:tasks.rados.rados.2.smithi204.stdout:1315: copy_from oid 47 from oid 4 current snap is 147 2024-09-16T05:16:07.718 INFO:tasks.rados.rados.2.smithi204.stdout:1316: snap_remove snap 122 2024-09-16T05:16:07.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:07.829 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:07.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:07.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:07.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:07.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:07.957 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:07 smithi204 ceph-mon[30933]: pgmap v786: 97 pgs: 97 active+clean; 230 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-16T05:16:07.957 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1385063897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:07.957 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/722493873' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:07.957 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:07 smithi204 ceph-mon[30933]: osdmap e418: 12 total, 12 up, 12 in 2024-09-16T05:16:07.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:16:08.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:08.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:08.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:07 smithi032 ceph-mon[27388]: pgmap v786: 97 pgs: 97 active+clean; 230 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-16T05:16:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1385063897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/722493873' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:07 smithi032 ceph-mon[27388]: osdmap e418: 12 total, 12 up, 12 in 2024-09-16T05:16:08.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:08.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:08.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:08.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:07 smithi196 ceph-mon[30802]: pgmap v786: 97 pgs: 97 active+clean; 230 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.4 MiB/s wr, 10 op/s 2024-09-16T05:16:08.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1385063897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/722493873' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:07 smithi196 ceph-mon[30802]: osdmap e418: 12 total, 12 up, 12 in 2024-09-16T05:16:08.262 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:16:08.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:08.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:08.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:08.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:08.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:08.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:08.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:08.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:08.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:08.704 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:08.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:08.721 INFO:tasks.rados.rados.2.smithi204.stdout:1316: done (1 left) 2024-09-16T05:16:08.721 INFO:tasks.rados.rados.2.smithi204.stdout:1317: copy_from oid 22 from oid 5 current snap is 147 2024-09-16T05:16:08.721 INFO:tasks.rados.rados.2.smithi204.stdout:1318: read oid 41 snap -1 2024-09-16T05:16:08.721 INFO:tasks.rados.rados.2.smithi204.stdout:1318: expect (ObjNum 457 snap 142 seq_num 457) 2024-09-16T05:16:08.721 INFO:tasks.rados.rados.2.smithi204.stdout:1319: write (excl) oid 13 current snap is 147 2024-09-16T05:16:08.721 INFO:tasks.rados.rados.2.smithi204.stdout:1319: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2024-09-16T05:16:08.729 INFO:tasks.rados.rados.2.smithi204.stdout:1319: writing smithi20457400-13 from 648869 to 1287685 tid 1 2024-09-16T05:16:08.738 INFO:tasks.rados.rados.2.smithi204.stdout:1319: writing smithi20457400-13 from 1828032 to 2615860 tid 2 2024-09-16T05:16:08.744 INFO:tasks.rados.rados.2.smithi204.stdout:1319: writing smithi20457400-13 from 3141714 to 3673744 tid 3 2024-09-16T05:16:08.744 INFO:tasks.rados.rados.2.smithi204.stdout:1320: write oid 46 current snap is 147 2024-09-16T05:16:08.744 INFO:tasks.rados.rados.2.smithi204.stdout:1315: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:16:08.744 INFO:tasks.rados.rados.2.smithi204.stdout:1315: finishing copy_from to smithi20457400-47 2024-09-16T05:16:08.744 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 549 (ObjNum 383 snap 115 seq_num 383) dirty exists 2024-09-16T05:16:08.744 INFO:tasks.rados.rados.2.smithi204.stdout:1320: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-16T05:16:08.751 INFO:tasks.rados.rados.2.smithi204.stdout:1320: writing smithi20457400-46 from 665676 to 1379741 tid 1 2024-09-16T05:16:08.757 INFO:tasks.rados.rados.2.smithi204.stdout:1320: writing smithi20457400-46 from 1886514 to 2418000 tid 2 2024-09-16T05:16:08.765 INFO:tasks.rados.rados.2.smithi204.stdout:1320: writing smithi20457400-46 from 3169137 to 3865676 tid 3 2024-09-16T05:16:08.765 INFO:tasks.rados.rados.2.smithi204.stdout:1315: done (4 left) 2024-09-16T05:16:08.765 INFO:tasks.rados.rados.2.smithi204.stdout:1321: delete oid 39 current snap is 147 2024-09-16T05:16:08.765 INFO:tasks.rados.rados.2.smithi204.stdout:1319: finishing write tid 1 to smithi20457400-13 2024-09-16T05:16:08.765 INFO:tasks.rados.rados.2.smithi204.stdout:1318: expect (ObjNum 457 snap 142 seq_num 457) 2024-09-16T05:16:08.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:08.807 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:08.807 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:08.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:08.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:08.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:08.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:08.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:09.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:1317: finishing copy_from racing read to smithi20457400-22 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:1319: finishing write tid 2 to smithi20457400-13 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:1317: finishing copy_from to smithi20457400-22 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 326 (ObjNum 440 snap 131 seq_num 440) dirty exists 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:1319: finishing write tid 3 to smithi20457400-13 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:1319: finishing write tid 4 to smithi20457400-13 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:1319: finishing write tid 5 to smithi20457400-13 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 488 (ObjNum 467 snap 147 seq_num 467) dirty exists 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:1319: left oid 13 (ObjNum 467 snap 147 seq_num 467) 2024-09-16T05:16:09.075 INFO:tasks.rados.rados.2.smithi204.stdout:1320: finishing write tid 1 to smithi20457400-46 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1317: done (4 left) 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1318: done (3 left) 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1319: done (2 left) 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1321: done (1 left) 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1322: rollback oid 38 current snap is 147 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 38 to 145 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1320: finishing write tid 2 to smithi20457400-46 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1320: finishing write tid 3 to smithi20457400-46 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1320: finishing write tid 4 to smithi20457400-46 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1323: copy_from oid 21 from oid 22 current snap is 147 2024-09-16T05:16:09.076 INFO:tasks.rados.rados.2.smithi204.stdout:1324: snap_remove snap 136 2024-09-16T05:16:09.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:09.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:09.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:09.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:09.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:09.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:09.331 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:09.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:09.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:16:09.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:09.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:09.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:09.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:09.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:09.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:09.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:09.706 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:09.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:09.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:09.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:09.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:09.908 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:09 smithi032 ceph-mon[27388]: pgmap v789: 97 pgs: 97 active+clean; 232 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-16T05:16:09.908 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:09 smithi032 ceph-mon[27388]: osdmap e419: 12 total, 12 up, 12 in 2024-09-16T05:16:09.916 INFO:tasks.rados.rados.2.smithi204.stdout:1324: done (3 left) 2024-09-16T05:16:09.916 INFO:tasks.rados.rados.2.smithi204.stdout:1325: delete oid 48 current snap is 147 2024-09-16T05:16:09.916 INFO:tasks.rados.rados.2.smithi204.stdout:1322: finishing rollback tid 1 to smithi20457400-38 2024-09-16T05:16:09.916 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 0 (ObjNum 129 snap 0 seq_num 4280833312) dirty dne 2024-09-16T05:16:09.917 INFO:tasks.rados.rados.2.smithi204.stdout:1323: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:16:09.917 INFO:tasks.rados.rados.2.smithi204.stdout:1323: finishing copy_from to smithi20457400-21 2024-09-16T05:16:09.917 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 326 (ObjNum 440 snap 131 seq_num 440) dirty exists 2024-09-16T05:16:09.917 INFO:tasks.rados.rados.2.smithi204.stdout:1320: finishing write tid 5 to smithi20457400-46 2024-09-16T05:16:09.917 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 329 (ObjNum 468 snap 147 seq_num 468) dirty exists 2024-09-16T05:16:09.917 INFO:tasks.rados.rados.2.smithi204.stdout:1320: left oid 46 (ObjNum 468 snap 147 seq_num 468) 2024-09-16T05:16:09.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:09 smithi204 ceph-mon[30933]: pgmap v789: 97 pgs: 97 active+clean; 232 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-16T05:16:09.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:09 smithi204 ceph-mon[30933]: osdmap e419: 12 total, 12 up, 12 in 2024-09-16T05:16:09.921 INFO:tasks.rados.rados.2.smithi204.stdout:1320: done (3 left) 2024-09-16T05:16:09.921 INFO:tasks.rados.rados.2.smithi204.stdout:1322: done (2 left) 2024-09-16T05:16:09.921 INFO:tasks.rados.rados.2.smithi204.stdout:1323: done (1 left) 2024-09-16T05:16:09.921 INFO:tasks.rados.rados.2.smithi204.stdout:1325: done (0 left) 2024-09-16T05:16:09.921 INFO:tasks.rados.rados.2.smithi204.stdout:1326: delete oid 30 current snap is 147 2024-09-16T05:16:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:1326: done (0 left) 2024-09-16T05:16:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:1327: copy_from oid 30 from oid 28 current snap is 147 2024-09-16T05:16:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:1328: copy_from oid 43 from oid 4 current snap is 147 2024-09-16T05:16:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:1329: rollback oid 49 current snap is 147 2024-09-16T05:16:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 144 2024-09-16T05:16:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:1330: write oid 26 current snap is 147 2024-09-16T05:16:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:1330: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-16T05:16:09.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:09.934 INFO:tasks.rados.rados.2.smithi204.stdout:1330: writing smithi20457400-26 from 732904 to 1464318 tid 1 2024-09-16T05:16:09.940 INFO:tasks.rados.rados.2.smithi204.stdout:1330: writing smithi20457400-26 from 2120442 to 2659266 tid 2 2024-09-16T05:16:09.947 INFO:tasks.rados.rados.2.smithi204.stdout:1330: writing smithi20457400-26 from 3278829 to 3783572 tid 3 2024-09-16T05:16:09.947 INFO:tasks.rados.rados.2.smithi204.stdout:1331: read oid 22 snap -1 2024-09-16T05:16:09.947 INFO:tasks.rados.rados.2.smithi204.stdout:1331: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:09.947 INFO:tasks.rados.rados.2.smithi204.stdout:1332: write oid 31 current snap is 147 2024-09-16T05:16:09.947 INFO:tasks.rados.rados.2.smithi204.stdout:1328: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:16:09.947 INFO:tasks.rados.rados.2.smithi204.stdout:1329: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:16:09.948 INFO:tasks.rados.rados.2.smithi204.stdout:1329: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:16:09.948 INFO:tasks.rados.rados.2.smithi204.stdout:1332: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-09-16T05:16:09.953 INFO:tasks.rados.rados.2.smithi204.stdout:1332: writing smithi20457400-31 from 749711 to 1156374 tid 1 2024-09-16T05:16:09.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:09.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:09.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:09.960 INFO:tasks.rados.rados.2.smithi204.stdout:1332: writing smithi20457400-31 from 1778924 to 2461406 tid 2 2024-09-16T05:16:09.969 INFO:tasks.rados.rados.2.smithi204.stdout:1332: writing smithi20457400-31 from 3189899 to 3905914 tid 3 2024-09-16T05:16:09.969 INFO:tasks.rados.rados.2.smithi204.stdout:1333: read oid 18 snap 143 2024-09-16T05:16:09.969 INFO:tasks.rados.rados.2.smithi204.stdout:1333: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:16:09.969 INFO:tasks.rados.rados.2.smithi204.stdout:1334: rollback oid 13 current snap is 147 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 13 to 142 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1329: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 346 (ObjNum 434 snap 125 seq_num 434) dirty exists 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1328: finishing copy_from to smithi20457400-43 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 549 (ObjNum 383 snap 115 seq_num 383) dirty exists 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1327: finishing copy_from racing read to smithi20457400-30 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1327: finishing copy_from to smithi20457400-30 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 361 (ObjNum 412 snap 123 seq_num 412) dirty exists 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1330: finishing write tid 1 to smithi20457400-26 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1330: finishing write tid 2 to smithi20457400-26 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1330: finishing write tid 3 to smithi20457400-26 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1330: finishing write tid 4 to smithi20457400-26 2024-09-16T05:16:09.970 INFO:tasks.rados.rados.2.smithi204.stdout:1327: done (7 left) 2024-09-16T05:16:09.971 INFO:tasks.rados.rados.2.smithi204.stdout:1328: done (6 left) 2024-09-16T05:16:09.971 INFO:tasks.rados.rados.2.smithi204.stdout:1329: done (5 left) 2024-09-16T05:16:09.971 INFO:tasks.rados.rados.2.smithi204.stdout:1335: snap_remove snap 145 2024-09-16T05:16:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:09 smithi196 ceph-mon[30802]: pgmap v789: 97 pgs: 97 active+clean; 232 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2024-09-16T05:16:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:09 smithi196 ceph-mon[30802]: osdmap e419: 12 total, 12 up, 12 in 2024-09-16T05:16:10.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:10.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:10.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:10.156 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:10.156 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:10.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:10.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:10.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:10.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:16:10.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:10.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:10.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:10.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:10.452 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:10.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:10.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:10.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:10.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:10.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:10.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:10.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:10.717 INFO:tasks.rados.rados.2.smithi204.stdout:1335: done (5 left) 2024-09-16T05:16:10.717 INFO:tasks.rados.rados.2.smithi204.stdout:1336: rollback oid 30 current snap is 147 2024-09-16T05:16:10.717 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 30 to 142 2024-09-16T05:16:10.717 INFO:tasks.rados.rados.2.smithi204.stdout:1330: finishing write tid 5 to smithi20457400-26 2024-09-16T05:16:10.718 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 672 (ObjNum 472 snap 147 seq_num 472) dirty exists 2024-09-16T05:16:10.718 INFO:tasks.rados.rados.2.smithi204.stdout:1330: left oid 26 (ObjNum 472 snap 147 seq_num 472) 2024-09-16T05:16:10.718 INFO:tasks.rados.rados.2.smithi204.stdout:1332: finishing write tid 1 to smithi20457400-31 2024-09-16T05:16:10.718 INFO:tasks.rados.rados.2.smithi204.stdout:1331: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:10.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:10.815 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:16:10.816 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:16:10.816 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-16T05:16:10.816 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:16:10.816 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:16:10.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-16T05:16:10.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:10.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:10.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:16:10.921 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:10.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:16:10.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:10.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:10.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:11.035 INFO:tasks.rados.rados.2.smithi204.stdout:1332: finishing write tid 2 to smithi20457400-31 2024-09-16T05:16:11.035 INFO:tasks.rados.rados.2.smithi204.stdout:1330: done (5 left) 2024-09-16T05:16:11.035 INFO:tasks.rados.rados.2.smithi204.stdout:1331: done (4 left) 2024-09-16T05:16:11.036 INFO:tasks.rados.rados.2.smithi204.stdout:1337: read oid 46 snap -1 2024-09-16T05:16:11.036 INFO:tasks.rados.rados.2.smithi204.stdout:1337: expect (ObjNum 468 snap 147 seq_num 468) 2024-09-16T05:16:11.036 INFO:tasks.rados.rados.2.smithi204.stdout:1338: delete oid 37 current snap is 147 2024-09-16T05:16:11.036 INFO:tasks.rados.rados.2.smithi204.stdout:1334: finishing rollback tid 0 to smithi20457400-13 2024-09-16T05:16:11.036 INFO:tasks.rados.rados.2.smithi204.stdout:1334: finishing rollback tid 1 to smithi20457400-13 2024-09-16T05:16:11.037 INFO:tasks.rados.rados.2.smithi204.stdout:1334: finishing rollback tid 2 to smithi20457400-13 2024-09-16T05:16:11.037 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 493 (ObjNum 423 snap 124 seq_num 423) dirty exists 2024-09-16T05:16:11.037 INFO:tasks.rados.rados.2.smithi204.stdout:1332: finishing write tid 3 to smithi20457400-31 2024-09-16T05:16:11.037 INFO:tasks.rados.rados.2.smithi204.stdout:1332: finishing write tid 4 to smithi20457400-31 2024-09-16T05:16:11.037 INFO:tasks.rados.rados.2.smithi204.stdout:1332: finishing write tid 5 to smithi20457400-31 2024-09-16T05:16:11.038 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 677 (ObjNum 473 snap 147 seq_num 473) dirty exists 2024-09-16T05:16:11.038 INFO:tasks.rados.rados.2.smithi204.stdout:1332: left oid 31 (ObjNum 473 snap 147 seq_num 473) 2024-09-16T05:16:11.038 INFO:tasks.rados.rados.2.smithi204.stdout:1333: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:16:11.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:11.072 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:11.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:10 smithi032 ceph-mon[27388]: osdmap e420: 12 total, 12 up, 12 in 2024-09-16T05:16:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:10 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:11.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:10 smithi032 ceph-mon[27388]: osdmap e421: 12 total, 12 up, 12 in 2024-09-16T05:16:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:10 smithi204 ceph-mon[30933]: osdmap e420: 12 total, 12 up, 12 in 2024-09-16T05:16:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:10 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:11.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:10 smithi204 ceph-mon[30933]: osdmap e421: 12 total, 12 up, 12 in 2024-09-16T05:16:11.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:11.193 INFO:tasks.rados.rados.2.smithi204.stdout:1336: finishing rollback tid 0 to smithi20457400-30 2024-09-16T05:16:11.193 INFO:tasks.rados.rados.2.smithi204.stdout:1336: finishing rollback tid 1 to smithi20457400-30 2024-09-16T05:16:11.193 INFO:tasks.rados.rados.2.smithi204.stdout:1336: finishing rollback tid 2 to smithi20457400-30 2024-09-16T05:16:11.193 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 684 (ObjNum 414 snap 123 seq_num 414) dirty exists 2024-09-16T05:16:11.193 INFO:tasks.rados.rados.2.smithi204.stdout:1332: done (5 left) 2024-09-16T05:16:11.193 INFO:tasks.rados.rados.2.smithi204.stdout:1333: done (4 left) 2024-09-16T05:16:11.193 INFO:tasks.rados.rados.2.smithi204.stdout:1334: done (3 left) 2024-09-16T05:16:11.194 INFO:tasks.rados.rados.2.smithi204.stdout:1336: done (2 left) 2024-09-16T05:16:11.194 INFO:tasks.rados.rados.2.smithi204.stdout:1338: done (1 left) 2024-09-16T05:16:11.194 INFO:tasks.rados.rados.2.smithi204.stdout:1339: snap_remove snap 143 2024-09-16T05:16:11.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:10 smithi196 ceph-mon[30802]: osdmap e420: 12 total, 12 up, 12 in 2024-09-16T05:16:11.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:10 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:11.194 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:10 smithi196 ceph-mon[30802]: osdmap e421: 12 total, 12 up, 12 in 2024-09-16T05:16:11.215 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:11.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:11.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:11.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:11.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:11.440 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:11.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:11.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:11.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:11.469 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:11.469 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:11.491 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:11.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:11.512 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:11.512 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:11.531 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:11.531 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T05:16:11.532 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:11.532 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:11.533 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:11.553 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:11.554 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:11.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:11.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:11.575 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:11.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:11.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:11.596 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:11.617 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:11.617 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:11.638 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:11.638 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:11.659 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:11.680 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:11.681 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:11.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:11.703 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:11.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:11.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:11.726 INFO:tasks.rados.rados.2.smithi204.stdout:1339: done (1 left) 2024-09-16T05:16:11.726 INFO:tasks.rados.rados.2.smithi204.stdout:1340: read oid 10 snap -1 2024-09-16T05:16:11.726 INFO:tasks.rados.rados.2.smithi204.stdout:1340: expect (ObjNum 349 snap 101 seq_num 349) 2024-09-16T05:16:11.727 INFO:tasks.rados.rados.2.smithi204.stdout:1341: copy_from oid 4 from oid 16 current snap is 147 2024-09-16T05:16:11.727 INFO:tasks.rados.rados.2.smithi204.stdout:1337: expect (ObjNum 468 snap 147 seq_num 468) 2024-09-16T05:16:11.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:11.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:11.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:11.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:11.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:11.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:12.061 INFO:tasks.rados.rados.2.smithi204.stdout:1340: expect (ObjNum 349 snap 101 seq_num 349) 2024-09-16T05:16:12.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:12.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:12.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:12.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:11 smithi032 ceph-mon[27388]: pgmap v792: 97 pgs: 97 active+clean; 223 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 903 KiB/s wr, 5 op/s 2024-09-16T05:16:12.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:11 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:12.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:11 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:12.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/9688302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:12.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:11 smithi032 ceph-mon[27388]: osdmap e422: 12 total, 12 up, 12 in 2024-09-16T05:16:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:11 smithi204 ceph-mon[30933]: pgmap v792: 97 pgs: 97 active+clean; 223 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 903 KiB/s wr, 5 op/s 2024-09-16T05:16:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:11 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:11 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/9688302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:11 smithi204 ceph-mon[30933]: osdmap e422: 12 total, 12 up, 12 in 2024-09-16T05:16:12.186 INFO:tasks.rados.rados.2.smithi204.stdout:1341: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:16:12.186 INFO:tasks.rados.rados.2.smithi204.stdout:1341: finishing copy_from to smithi20457400-4 2024-09-16T05:16:12.186 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 493 (ObjNum 428 snap 124 seq_num 428) dirty exists 2024-09-16T05:16:12.186 INFO:tasks.rados.rados.2.smithi204.stdout:1337: done (2 left) 2024-09-16T05:16:12.186 INFO:tasks.rados.rados.2.smithi204.stdout:1340: done (1 left) 2024-09-16T05:16:12.186 INFO:tasks.rados.rados.2.smithi204.stdout:1341: done (0 left) 2024-09-16T05:16:12.186 INFO:tasks.rados.rados.2.smithi204.stdout:1342: snap_create 2024-09-16T05:16:12.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:12.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:12.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:12.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:11 smithi196 ceph-mon[30802]: pgmap v792: 97 pgs: 97 active+clean; 223 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 903 KiB/s wr, 5 op/s 2024-09-16T05:16:12.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:11 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:12.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:11 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:12.213 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/9688302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:12.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:11 smithi196 ceph-mon[30802]: osdmap e422: 12 total, 12 up, 12 in 2024-09-16T05:16:12.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:12.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:12.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:12.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:12.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:12.544 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:16:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:16:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:16:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:16:12.544 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:16:12.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:12.584 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:12.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:12.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:12.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:12.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:12.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:12.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:12.922 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:16:12.925 INFO:tasks.rados.rados.2.smithi204.stdout:1342: done (0 left) 2024-09-16T05:16:12.925 INFO:tasks.rados.rados.2.smithi204.stdout:1343: rollback oid 46 current snap is 148 2024-09-16T05:16:12.925 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 144 2024-09-16T05:16:12.925 INFO:tasks.rados.rados.2.smithi204.stdout:1344: snap_create 2024-09-16T05:16:12.930 INFO:tasks.rados.rados.2.smithi204.stdout:1343: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:16:12.930 INFO:tasks.rados.rados.2.smithi204.stdout:1343: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:16:12.930 INFO:tasks.rados.rados.2.smithi204.stdout:1343: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:16:12.930 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 332 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:16:12.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:12.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:12.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:13.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:13.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2593780596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:13.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:13.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:13.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2593780596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:13.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:13.338 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:13.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:13.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:16:13.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2593780596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:13.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:13.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:13.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:13.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:13.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:13.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:13.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:13.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:13.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:13.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:13.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:13.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:13.926 INFO:tasks.rados.rados.2.smithi204.stdout:1343: done (1 left) 2024-09-16T05:16:13.926 INFO:tasks.rados.rados.2.smithi204.stdout:1344: done (0 left) 2024-09-16T05:16:13.926 INFO:tasks.rados.rados.2.smithi204.stdout:1345: copy_from oid 4 from oid 21 current snap is 149 2024-09-16T05:16:13.927 INFO:tasks.rados.rados.2.smithi204.stdout:1346: delete oid 10 current snap is 149 2024-09-16T05:16:13.931 INFO:tasks.rados.rados.2.smithi204.stdout:1346: done (1 left) 2024-09-16T05:16:13.931 INFO:tasks.rados.rados.2.smithi204.stdout:1347: snap_create 2024-09-16T05:16:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:1345: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:16:13.945 INFO:tasks.rados.rados.2.smithi204.stdout:1345: finishing copy_from to smithi20457400-4 2024-09-16T05:16:13.945 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 326 (ObjNum 440 snap 131 seq_num 440) dirty exists 2024-09-16T05:16:13.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:13.975 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:13.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:14.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:14.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:13 smithi032 ceph-mon[27388]: pgmap v795: 97 pgs: 97 active+clean; 216 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 16 op/s 2024-09-16T05:16:14.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:13 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:14.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:13 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:14.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2593780596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:14.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:13 smithi032 ceph-mon[27388]: osdmap e423: 12 total, 12 up, 12 in 2024-09-16T05:16:14.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:13 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:16:14.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:13 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:16:14.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:14.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:14.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:14.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:14.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:14.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:13 smithi196 ceph-mon[30802]: pgmap v795: 97 pgs: 97 active+clean; 216 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 16 op/s 2024-09-16T05:16:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:13 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:13 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2593780596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:13 smithi196 ceph-mon[30802]: osdmap e423: 12 total, 12 up, 12 in 2024-09-16T05:16:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:13 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:16:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:13 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:16:14.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:14.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:14.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:13 smithi204 ceph-mon[30933]: pgmap v795: 97 pgs: 97 active+clean; 216 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 16 op/s 2024-09-16T05:16:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:13 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:13 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2593780596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:13 smithi204 ceph-mon[30933]: osdmap e423: 12 total, 12 up, 12 in 2024-09-16T05:16:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:13 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:16:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:13 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:16:14.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:14.475 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:14.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:14.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:14.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:14.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:14.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:14.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:14.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:14.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:14.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:14.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:14.926 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2318 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 95.999433648999997, 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98387304600000003, 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:16:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2318 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924656+0000", 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.341e-06 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924663+0000", 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6050000000000003e-06 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924673+0000", 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8830000000000001e-06 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924680+0000", 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2230000000000001e-06 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924730+0000", 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0256999999999997e-05 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924737+0000", 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1550000000000001e-06 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.908327+0000", 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98359007499999995 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.908528+0000", 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000200507 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2319 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 95.999392929999999, 2024-09-16T05:16:14.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98447908799999995, 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2319 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:16:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5900000000000003e-07 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924698+0000", 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9309999999999998e-06 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924703+0000", 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9609999999999996e-06 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924707+0000", 2024-09-16T05:16:14.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8990000000000003e-06 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924769+0000", 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1797999999999993e-05 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924776+0000", 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.99e-06 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909060+0000", 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98428400299999996 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909175+0000", 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115147 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2320 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~0,checksum] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 95.999369540000004, 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98482718000000002, 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2320 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:16:14.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4200000000000002e-07 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924722+0000", 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4380000000000002e-06 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924726+0000", 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0709999999999996e-06 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924731+0000", 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0069999999999997e-06 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924792+0000", 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1561000000000002e-05 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924796+0000", 2024-09-16T05:16:14.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0400000000000003e-06 2024-09-16T05:16:14.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.943 INFO:tasks.rados.rados.2.smithi204.stdout:1345: done (1 left) 2024-09-16T05:16:14.943 INFO:tasks.rados.rados.2.smithi204.stdout:1347: done (0 left) 2024-09-16T05:16:14.943 INFO:tasks.rados.rados.2.smithi204.stdout:1348: write (excl) oid 45 current snap is 150 2024-09-16T05:16:14.943 INFO:tasks.rados.rados.2.smithi204.stdout:1348: seq_num 476 ranges {400132=632410,1438017=546162} 2024-09-16T05:16:14.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909278+0000", 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98448187499999995 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909546+0000", 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026784600000000002 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2511 2.3 e330/329 2:c018a787:::smithi20457400-18:head v 330'60, mlcod=0'0)", 2024-09-16T05:16:14.944 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:39.910627+0000", 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 95.013461724999999, 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001484073, 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.910627+0000", 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.910627+0000", 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.910627+0000", 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0200000000000001e-07 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.910634+0000", 2024-09-16T05:16:14.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6030000000000001e-06 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.910664+0000", 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9768999999999999e-05 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.910673+0000", 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8990000000000007e-06 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.910722+0000", 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8835999999999998e-05 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.910748+0000", 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6132e-05 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.912064+0000", 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013160279999999999 2024-09-16T05:16:14.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.912111+0000", 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7104000000000002e-05 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(2.3a 331/330 [PushOp(2:5f916a06:::smithi20457400-34:69, version: 310'413, data_included: [720896~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5f916a06:::smithi20457400-34:69@310'413, size: 3176556, copy_subset: [(0, 3176556)], clone_subset: {}, snapset: 69=[]:{69: [69, 68]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:41.831866+0000", 2024-09-16T05:16:14.949 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 93.092222561, 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020026369999999998, 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.831866+0000", 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.831866+0000", 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.831866+0000", 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2900000000000004e-07 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.832958+0000", 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001091994 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.832991+0000", 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2863e-05 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.833002+0000", 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1100999999999999e-05 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.833074+0000", 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1326000000000001e-05 2024-09-16T05:16:14.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.833083+0000", 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2529999999999993e-06 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.833869+0000", 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078557099999999999 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(2.3a 331/330 [PushOp(2:5f916a06:::smithi20457400-34:head, version: 310'416, data_included: [400099~616131,1591248~408851], data_size: 1024982, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2:5f916a06:::smithi20457400-34:head@310'416, size: 2000099, copy_subset: [(400099, 616131), (1591248, 408851)], clone_subset: {2:5f916a06:::smithi20457400-34:69: [(0, 400099), (1016230, 575018)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2000099, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:41.837977+0000", 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 93.086111661000004, 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021533070000000001, 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.837977+0000", 2024-09-16T05:16:14.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.837977+0000", 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.837977+0000", 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8400000000000004e-07 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.839358+0000", 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00138054 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.839388+0000", 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0198000000000001e-05 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.839399+0000", 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1104000000000001e-05 2024-09-16T05:16:14.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.839461+0000", 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1388000000000002e-05 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.839470+0000", 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3109999999999995e-06 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:41.840130+0000", 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066018200000000004 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2740 2.3a e355/330 2:5f916a06:::smithi20457400-34:head v 355'425, mlcod=352'422)", 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:04.730661+0000", 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 70.193427317000001, 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021166129999999998, 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.958 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.730661+0000", 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.730661+0000", 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.730661+0000", 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.73e-07 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.731388+0000", 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00072628500000000002 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.731416+0000", 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8779e-05 2024-09-16T05:16:14.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.731427+0000", 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0893000000000001e-05 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.731498+0000", 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0203000000000005e-05 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.731528+0000", 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0244999999999999e-05 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.732725+0000", 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001197486 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.732778+0000", 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2549000000000001e-05 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2741 2.3a e355/330 2:5f916a06:::smithi20457400-34:head v 355'426, mlcod=354'424)", 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:04.736022+0000", 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 70.188066946000006, 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001402592, 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.736022+0000", 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.736022+0000", 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.736023+0000", 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1289999999999999e-06 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.736555+0000", 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053184 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.736584+0000", 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9947999999999999e-05 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.736591+0000", 2024-09-16T05:16:14.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9319999999999999e-06 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.736634+0000", 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2966999999999997e-05 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.736649+0000", 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4681000000000001e-05 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.737380+0000", 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00073119600000000004 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:04.737424+0000", 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3899000000000001e-05 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.967 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:621 2.3a e370/330 2:5f916a06:::smithi20457400-34:75 v 370'437, mlcod=365'433)", 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:19.719975+0000", 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.204113980000002, 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016454639999999999, 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.719975+0000", 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.719975+0000", 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.719975+0000", 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7000000000000001e-07 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.719981+0000", 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.105e-06 2024-09-16T05:16:14.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.720015+0000", 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4564999999999999e-05 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.720026+0000", 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0404000000000001e-05 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.720162+0000", 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013595 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.720183+0000", 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1454000000000002e-05 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.721574+0000", 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001390624 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.721620+0000", 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6091999999999998e-05 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2898 2.2b 2:d7c71ae7:::smithi20457400-5:head [copy-from ver 322] snapc 7c=[7c,7b,7a,76,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 53.217392812, 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026725529000000001, 2024-09-16T05:16:14.972 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2898 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:16:14.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706697+0000", 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4789999999999999e-06 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706705+0000", 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0490000000000004e-06 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706719+0000", 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4231e-05 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706728+0000", 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9339999999999995e-06 2024-09-16T05:16:14.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706819+0000", 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0402000000000006e-05 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706891+0000", 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2605e-05 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:16:14.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.724332+0000", 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017440699 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.728315+0000", 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003982546 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733300+0000", 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049854369999999997 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733368+0000", 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8133000000000006e-05 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733421+0000", 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3013999999999998e-05 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2899 2.2b 2:d7c71ae7:::smithi20457400-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 53.217343245000002, 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020779419, 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2899 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.979 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706746+0000", 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1600000000000002e-07 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706748+0000", 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3140000000000002e-06 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706754+0000", 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2609999999999999e-06 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706768+0000", 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3788000000000001e-05 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.707033+0000", 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026479299999999998 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.707061+0000", 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8050000000000001e-05 2024-09-16T05:16:14.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727362+0000", 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020301534 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727439+0000", 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6749000000000007e-05 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727525+0000", 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5513999999999995e-05 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3108 2.2b 2:d7c71ae7:::smithi20457400-5:head [write 595080~541681] snapc 84=[84,83,82,81,80,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.717974+0000", 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.206114775000003, 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051422569999999999, 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.984 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3108 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.717974+0000", 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.717974+0000", 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.717977+0000", 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.083e-06 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718620+0000", 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00064307999999999998 2024-09-16T05:16:14.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718642+0000", 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2007000000000001e-05 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718652+0000", 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6930000000000002e-06 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718738+0000", 2024-09-16T05:16:14.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6736000000000004e-05 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718746+0000", 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1429999999999998e-06 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.719931+0000", 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001184147 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.720032+0000", 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010173199999999999 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.720228+0000", 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019584800000000001 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.721765+0000", 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015371919999999999 2024-09-16T05:16:14.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:14.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.723044+0000", 2024-09-16T05:16:14.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001278373 2024-09-16T05:16:14.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.723084+0000", 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0098999999999997e-05 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.723116+0000", 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2123999999999999e-05 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3109 2.2b 2:d7c71ae7:::smithi20457400-5:head [write 1798077~596473] snapc 84=[84,83,82,81,80,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.724839+0000", 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.199249631000001, 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023312350000000001, 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3109 2024-09-16T05:16:14.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.724839+0000", 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.724839+0000", 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:14.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.724840+0000", 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.235e-06 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.725362+0000", 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00052213499999999996 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.725376+0000", 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4015999999999999e-05 2024-09-16T05:16:14.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.725382+0000", 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9599999999999997e-06 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.725441+0000", 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8653999999999998e-05 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.725504+0000", 2024-09-16T05:16:14.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2748000000000003e-05 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.725598+0000", 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4469000000000006e-05 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.726858+0000", 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012603099999999999 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.727104+0000", 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024552500000000002 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.727141+0000", 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6674000000000002e-05 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.727170+0000", 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9509000000000002e-05 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.997 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3110 2.2b 2:d7c71ae7:::smithi20457400-5:head [write 3150352~244728] snapc 84=[84,83,82,81,80,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.730516+0000", 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.193572998, 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00174749, 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:14.998 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3110 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730516+0000", 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730516+0000", 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:14.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730518+0000", 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7199999999999998e-06 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730826+0000", 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030821199999999998 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730840+0000", 2024-09-16T05:16:15.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3339000000000001e-05 2024-09-16T05:16:15.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:15.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730846+0000", 2024-09-16T05:16:15.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:15.003 INFO:tasks.rados.rados.2.smithi204.stdout:1348: writing smithi20457400-45 from 400132 to 1032542 tid 1 2024-09-16T05:16:15.003 INFO:tasks.rados.rados.2.smithi204.stdout:1348: writing smithi20457400-45 from 1438017 to 1984179 tid 2 2024-09-16T05:16:15.003 INFO:tasks.rados.rados.2.smithi204.stdout:1349: snap_remove snap 141 2024-09-16T05:16:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9290000000000003e-06 2024-09-16T05:16:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730932+0000", 2024-09-16T05:16:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6179000000000003e-05 2024-09-16T05:16:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730988+0000", 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6042000000000003e-05 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.731078+0000", 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0241000000000006e-05 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732075+0000", 2024-09-16T05:16:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00099719600000000011 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732202+0000", 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127083 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732237+0000", 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4054000000000003e-05 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732263+0000", 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6495e-05 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3111 2.2b 2:d7c71ae7:::smithi20457400-5:head [setxattr _header (57) in=7b,truncate 3395080] snapc 84=[84,83,82,81,80,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.730859+0000", 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.193229330000001, 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015090800000000001, 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:15.006 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3111 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730859+0000", 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730859+0000", 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730860+0000", 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5299999999999999e-07 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:15.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730863+0000", 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5779999999999999e-06 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730868+0000", 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6500000000000004e-06 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730871+0000", 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3230000000000002e-06 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.731591+0000", 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071991399999999999 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.731667+0000", 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6067000000000002e-05 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.731755+0000", 2024-09-16T05:16:15.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8197999999999998e-05 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732080+0000", 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000324667 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732294+0000", 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021445500000000001 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732330+0000", 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5209000000000001e-05 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732368+0000", 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8566000000000003e-05 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3112 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e388)", 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.730876+0000", 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.193212047000003, 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001642204, 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3112 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.013 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730876+0000", 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730876+0000", 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:15.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730877+0000", 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8999999999999998e-07 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730878+0000", 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4160000000000001e-06 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730882+0000", 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8759999999999998e-06 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.730885+0000", 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7769999999999998e-06 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.731983+0000", 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001097741 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732065+0000", 2024-09-16T05:16:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2286999999999993e-05 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732409+0000", 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00034454599999999999 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732457+0000", 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7259000000000003e-05 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:15.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.732519+0000", 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2012000000000002e-05 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3113 2.3a e388/330 2:5f916a06:::smithi20457400-34:head v 388'455, mlcod=380'452)", 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.741061+0000", 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.183027590999998, 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019766710000000002, 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:15.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.741061+0000", 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.741061+0000", 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.741061+0000", 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3000000000000001e-07 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.741652+0000", 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00059044900000000005 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:15.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.741679+0000", 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7365e-05 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.741686+0000", 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8990000000000004e-06 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.741750+0000", 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3739999999999996e-05 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.741781+0000", 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1572999999999997e-05 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.742994+0000", 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001212882 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.743038+0000", 2024-09-16T05:16:15.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3232999999999998e-05 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3114 2.3a e388/330 2:5f916a06:::smithi20457400-34:head v 388'456, mlcod=385'453)", 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.744627+0000", 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.179461500999999, 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001313668, 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:15.023 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.744627+0000", 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.744627+0000", 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.744628+0000", 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3600000000000002e-07 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.744992+0000", 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036424 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.745023+0000", 2024-09-16T05:16:15.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0846000000000002e-05 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.745029+0000", 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2079999999999997e-06 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.745080+0000", 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0560999999999999e-05 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.745102+0000", 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2481e-05 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.745902+0000", 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000800046 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.745941+0000", 2024-09-16T05:16:15.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8349999999999997e-05 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3115 2.3a e388/330 2:5f916a06:::smithi20457400-34:head v 388'457, mlcod=388'455)", 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.750005+0000", 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 37.174083465000002, 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013979909999999999, 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:15.028 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.750005+0000", 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.750005+0000", 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.750006+0000", 2024-09-16T05:16:15.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7000000000000003e-07 2024-09-16T05:16:15.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:15.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.750449+0000", 2024-09-16T05:16:15.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00044340299999999998 2024-09-16T05:16:15.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.750478+0000", 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8203000000000001e-05 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.750483+0000", 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4750000000000001e-06 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.750552+0000", 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8561999999999998e-05 2024-09-16T05:16:15.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.750575+0000", 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2991000000000001e-05 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.751358+0000", 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078375899999999997 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.751403+0000", 2024-09-16T05:16:15.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4728000000000003e-05 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3152 2.3 e397/329 2:c018a787:::smithi20457400-18:head v 397'99, mlcod=375'96)", 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.201200822000001, 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066570259999999999, 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:15.035 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:15.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3300000000000002e-07 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726366+0000", 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003478259 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726395+0000", 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8347000000000002e-05 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726404+0000", 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0790000000000007e-06 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726446+0000", 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2092999999999998e-05 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726464+0000", 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8014000000000001e-05 2024-09-16T05:16:15.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.729500+0000", 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030358519999999999 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.729545+0000", 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4848999999999997e-05 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:15.039 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:15.040 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:16:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:15.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:15.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:15.190 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:15.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:14 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:16:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:14 smithi196 ceph-mon[30802]: 2.26 scrub starts 2024-09-16T05:16:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:14 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:16:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:14 smithi196 ceph-mon[30802]: 2.26 scrub ok 2024-09-16T05:16:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:14 smithi196 ceph-mon[30802]: osdmap e424: 12 total, 12 up, 12 in 2024-09-16T05:16:15.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:15.314 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:15.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:15.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:14 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:16:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:14 smithi032 ceph-mon[27388]: 2.26 scrub starts 2024-09-16T05:16:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:14 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:16:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:14 smithi032 ceph-mon[27388]: 2.26 scrub ok 2024-09-16T05:16:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:14 smithi032 ceph-mon[27388]: osdmap e424: 12 total, 12 up, 12 in 2024-09-16T05:16:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:14 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:16:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:14 smithi204 ceph-mon[30933]: 2.26 scrub starts 2024-09-16T05:16:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:14 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:16:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:14 smithi204 ceph-mon[30933]: 2.26 scrub ok 2024-09-16T05:16:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:14 smithi204 ceph-mon[30933]: osdmap e424: 12 total, 12 up, 12 in 2024-09-16T05:16:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:15.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:15.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:15.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:15.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:15.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:15.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:15.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:15.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:16:15.720 INFO:tasks.rados.rados.2.smithi204.stdout:1349: done (1 left) 2024-09-16T05:16:15.720 INFO:tasks.rados.rados.2.smithi204.stdout:1350: snap_create 2024-09-16T05:16:15.720 INFO:tasks.rados.rados.2.smithi204.stdout:1348: finishing write tid 1 to smithi20457400-45 2024-09-16T05:16:15.720 INFO:tasks.rados.rados.2.smithi204.stdout:1348: finishing write tid 2 to smithi20457400-45 2024-09-16T05:16:15.720 INFO:tasks.rados.rados.2.smithi204.stdout:1348: finishing write tid 3 to smithi20457400-45 2024-09-16T05:16:15.720 INFO:tasks.rados.rados.2.smithi204.stdout:1348: finishing write tid 4 to smithi20457400-45 2024-09-16T05:16:15.720 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 339 (ObjNum 476 snap 150 seq_num 476) dirty exists 2024-09-16T05:16:15.720 INFO:tasks.rados.rados.2.smithi204.stdout:1348: left oid 45 (ObjNum 476 snap 150 seq_num 476) 2024-09-16T05:16:15.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:15.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:15.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:15.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:16:15.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:15.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:15.991 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:16.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:16.070 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:16.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: pgmap v798: 97 pgs: 97 active+clean; 212 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 21 op/s 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: 2.43 scrub starts 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: 2.43 scrub ok 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: osdmap e425: 12 total, 12 up, 12 in 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:16.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: osdmap e426: 12 total, 12 up, 12 in 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09188981392911554]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08576718133358384]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09372722136613507]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08769063297022178]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09089944977970754]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045290316994509]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08503982714559585]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0835761784161254]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08854275236169279]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08413932319824623]}]: dispatch 2024-09-16T05:16:16.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0792675380727428]}]: dispatch 2024-09-16T05:16:16.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:15 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08655356497563813]}]: dispatch 2024-09-16T05:16:16.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:16.197 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:16.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: pgmap v798: 97 pgs: 97 active+clean; 212 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 21 op/s 2024-09-16T05:16:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:16:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:16:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: 2.43 scrub starts 2024-09-16T05:16:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: 2.43 scrub ok 2024-09-16T05:16:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: osdmap e425: 12 total, 12 up, 12 in 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: osdmap e426: 12 total, 12 up, 12 in 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09188981392911554]}]: dispatch 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08576718133358384]}]: dispatch 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09372722136613507]}]: dispatch 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08769063297022178]}]: dispatch 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09089944977970754]}]: dispatch 2024-09-16T05:16:16.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045290316994509]}]: dispatch 2024-09-16T05:16:16.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08503982714559585]}]: dispatch 2024-09-16T05:16:16.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0835761784161254]}]: dispatch 2024-09-16T05:16:16.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08854275236169279]}]: dispatch 2024-09-16T05:16:16.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08413932319824623]}]: dispatch 2024-09-16T05:16:16.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0792675380727428]}]: dispatch 2024-09-16T05:16:16.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:15 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08655356497563813]}]: dispatch 2024-09-16T05:16:16.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:16.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:16.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: pgmap v798: 97 pgs: 97 active+clean; 212 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 21 op/s 2024-09-16T05:16:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:16:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:16:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: 2.43 scrub starts 2024-09-16T05:16:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: 2.43 scrub ok 2024-09-16T05:16:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: osdmap e425: 12 total, 12 up, 12 in 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: osdmap e426: 12 total, 12 up, 12 in 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09188981392911554]}]: dispatch 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08576718133358384]}]: dispatch 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09372722136613507]}]: dispatch 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08769063297022178]}]: dispatch 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09089944977970754]}]: dispatch 2024-09-16T05:16:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045290316994509]}]: dispatch 2024-09-16T05:16:16.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08503982714559585]}]: dispatch 2024-09-16T05:16:16.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0835761784161254]}]: dispatch 2024-09-16T05:16:16.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08854275236169279]}]: dispatch 2024-09-16T05:16:16.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08413932319824623]}]: dispatch 2024-09-16T05:16:16.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0792675380727428]}]: dispatch 2024-09-16T05:16:16.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:15 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08655356497563813]}]: dispatch 2024-09-16T05:16:16.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:16.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:16:16.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:16.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:16.544 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:16.544 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:16.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:16.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:16.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:16.582 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:16.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:16.593 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:16.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:16.620 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:16.641 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:16.664 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:16.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:16.686 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:16.686 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:16.707 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:16.707 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:16.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:16.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:16.722 INFO:tasks.rados.rados.2.smithi204.stdout:1348: done (1 left) 2024-09-16T05:16:16.722 INFO:tasks.rados.rados.2.smithi204.stdout:1350: done (0 left) 2024-09-16T05:16:16.722 INFO:tasks.rados.rados.2.smithi204.stdout:1351: delete oid 24 current snap is 151 2024-09-16T05:16:16.727 INFO:tasks.rados.rados.2.smithi204.stdout:1351: done (0 left) 2024-09-16T05:16:16.727 INFO:tasks.rados.rados.2.smithi204.stdout:1352: snap_remove snap 129 2024-09-16T05:16:16.733 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:16.733 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:16.757 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:16.757 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:16.780 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:16.780 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:16.801 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:16.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:16.823 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:16.823 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:16.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:16.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:16.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:16.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:16.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:17.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:17.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:17.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:17.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:17.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:17.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1024355352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09188981392911554]}]': finished 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08576718133358384]}]': finished 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09372722136613507]}]': finished 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08769063297022178]}]': finished 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09089944977970754]}]': finished 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045290316994509]}]': finished 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08503982714559585]}]': finished 2024-09-16T05:16:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0835761784161254]}]': finished 2024-09-16T05:16:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08854275236169279]}]': finished 2024-09-16T05:16:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08413932319824623]}]': finished 2024-09-16T05:16:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0792675380727428]}]': finished 2024-09-16T05:16:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08655356497563813]}]': finished 2024-09-16T05:16:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:16 smithi196 ceph-mon[30802]: osdmap e427: 12 total, 12 up, 12 in 2024-09-16T05:16:17.277 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:16:17.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:16:17.278 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T05:16:17.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:16:17.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:16:17.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:16:17.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1024355352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:17.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09188981392911554]}]': finished 2024-09-16T05:16:17.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08576718133358384]}]': finished 2024-09-16T05:16:17.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09372722136613507]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08769063297022178]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09089944977970754]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045290316994509]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08503982714559585]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0835761784161254]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08854275236169279]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08413932319824623]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0792675380727428]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08655356497563813]}]': finished 2024-09-16T05:16:17.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:16 smithi032 ceph-mon[27388]: osdmap e427: 12 total, 12 up, 12 in 2024-09-16T05:16:17.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:17.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:17.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1024355352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09188981392911554]}]': finished 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08576718133358384]}]': finished 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09372722136613507]}]': finished 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08769063297022178]}]': finished 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09089944977970754]}]': finished 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09045290316994509]}]': finished 2024-09-16T05:16:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08503982714559585]}]': finished 2024-09-16T05:16:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0835761784161254]}]': finished 2024-09-16T05:16:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08854275236169279]}]': finished 2024-09-16T05:16:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08413932319824623]}]': finished 2024-09-16T05:16:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0792675380727428]}]': finished 2024-09-16T05:16:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08655356497563813]}]': finished 2024-09-16T05:16:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:16 smithi204 ceph-mon[30933]: osdmap e427: 12 total, 12 up, 12 in 2024-09-16T05:16:17.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:17.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:17.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:17.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:17.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:17.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:17.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:17.725 INFO:tasks.rados.rados.2.smithi204.stdout:1352: done (0 left) 2024-09-16T05:16:17.725 INFO:tasks.rados.rados.2.smithi204.stdout:1353: read oid 14 snap -1 2024-09-16T05:16:17.726 INFO:tasks.rados.rados.2.smithi204.stdout:1353: expect (ObjNum 415 snap 123 seq_num 415) 2024-09-16T05:16:17.726 INFO:tasks.rados.rados.2.smithi204.stdout:1354: write oid 38 current snap is 151 2024-09-16T05:16:17.726 INFO:tasks.rados.rados.2.smithi204.stdout:1354: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-16T05:16:17.733 INFO:tasks.rados.rados.2.smithi204.stdout:1354: writing smithi20457400-38 from 433746 to 1216654 tid 1 2024-09-16T05:16:17.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:17.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:17.738 INFO:tasks.rados.rados.2.smithi204.stdout:1354: writing smithi20457400-38 from 1954981 to 2033746 tid 2 2024-09-16T05:16:17.738 INFO:tasks.rados.rados.2.smithi204.stdout:1355: rollback oid 42 current snap is 151 2024-09-16T05:16:17.738 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 149 2024-09-16T05:16:17.738 INFO:tasks.rados.rados.2.smithi204.stdout:1353: expect (ObjNum 415 snap 123 seq_num 415) 2024-09-16T05:16:17.804 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:17.816 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:17.816 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:17.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:17.860 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:17.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:17.913 INFO:tasks.rados.rados.2.smithi204.stdout:1354: finishing write tid 1 to smithi20457400-38 2024-09-16T05:16:17.913 INFO:tasks.rados.rados.2.smithi204.stdout:1354: finishing write tid 2 to smithi20457400-38 2024-09-16T05:16:17.913 INFO:tasks.rados.rados.2.smithi204.stdout:1353: done (2 left) 2024-09-16T05:16:17.913 INFO:tasks.rados.rados.2.smithi204.stdout:1356: write (excl) oid 46 current snap is 151 2024-09-16T05:16:17.913 INFO:tasks.rados.rados.2.smithi204.stdout:1354: finishing write tid 3 to smithi20457400-38 2024-09-16T05:16:17.913 INFO:tasks.rados.rados.2.smithi204.stdout:1354: finishing write tid 4 to smithi20457400-38 2024-09-16T05:16:17.914 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 515 (ObjNum 478 snap 151 seq_num 478) dirty exists 2024-09-16T05:16:17.914 INFO:tasks.rados.rados.2.smithi204.stdout:1354: left oid 38 (ObjNum 478 snap 151 seq_num 478) 2024-09-16T05:16:17.914 INFO:tasks.rados.rados.2.smithi204.stdout:1355: finishing rollback tid 0 to smithi20457400-42 2024-09-16T05:16:17.914 INFO:tasks.rados.rados.2.smithi204.stdout:1355: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:16:17.914 INFO:tasks.rados.rados.2.smithi204.stdout:1355: finishing rollback tid 2 to smithi20457400-42 2024-09-16T05:16:17.914 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 330 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:16:17.914 INFO:tasks.rados.rados.2.smithi204.stdout:1356: seq_num 479 ranges {450553=574510,1613463=437090} 2024-09-16T05:16:17.918 INFO:tasks.rados.rados.2.smithi204.stdout:1356: writing smithi20457400-46 from 450553 to 1025063 tid 1 2024-09-16T05:16:17.923 INFO:tasks.rados.rados.2.smithi204.stdout:1356: writing smithi20457400-46 from 1613463 to 2050553 tid 2 2024-09-16T05:16:17.923 INFO:tasks.rados.rados.2.smithi204.stdout:1354: done (2 left) 2024-09-16T05:16:17.923 INFO:tasks.rados.rados.2.smithi204.stdout:1355: done (1 left) 2024-09-16T05:16:17.924 INFO:tasks.rados.rados.2.smithi204.stdout:1357: rollback oid 3 current snap is 151 2024-09-16T05:16:17.924 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 150 2024-09-16T05:16:17.924 INFO:tasks.rados.rados.2.smithi204.stdout:1356: finishing write tid 1 to smithi20457400-46 2024-09-16T05:16:17.924 INFO:tasks.rados.rados.2.smithi204.stdout:1358: snap_remove snap 144 2024-09-16T05:16:17.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:17.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:17.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:18.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:18.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:17 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:16:18.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:17 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:16:18.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:17 smithi032 ceph-mon[27388]: pgmap v801: 97 pgs: 97 active+clean; 213 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-09-16T05:16:18.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:17 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:16:18.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:17 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:16:18.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:17 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:16:18.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:17 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:16:18.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:17 smithi032 ceph-mon[27388]: osdmap e428: 12 total, 12 up, 12 in 2024-09-16T05:16:18.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:18.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:18.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:17 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:16:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:17 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:16:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:17 smithi196 ceph-mon[30802]: pgmap v801: 97 pgs: 97 active+clean; 213 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-09-16T05:16:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:17 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:16:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:17 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:16:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:17 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:16:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:17 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:16:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:17 smithi196 ceph-mon[30802]: osdmap e428: 12 total, 12 up, 12 in 2024-09-16T05:16:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:18.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:18.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:18.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:18.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:18.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:18.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:16:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:17 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:16:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:17 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:16:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:17 smithi204 ceph-mon[30933]: pgmap v801: 97 pgs: 97 active+clean; 213 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-09-16T05:16:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:17 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:16:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:17 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:16:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:17 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:16:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:17 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:16:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:17 smithi204 ceph-mon[30933]: osdmap e428: 12 total, 12 up, 12 in 2024-09-16T05:16:18.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:18.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:18.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:18.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:18.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:18.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:18.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:18.723 INFO:tasks.rados.rados.2.smithi204.stdout:1358: done (2 left) 2024-09-16T05:16:18.723 INFO:tasks.rados.rados.2.smithi204.stdout:1359: rollback oid 20 current snap is 151 2024-09-16T05:16:18.723 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 20 to 142 2024-09-16T05:16:18.723 INFO:tasks.rados.rados.2.smithi204.stdout:1357: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:16:18.723 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 593 snap 0 seq_num 4282310536) dirty dne 2024-09-16T05:16:18.723 INFO:tasks.rados.rados.2.smithi204.stdout:1356: finishing write tid 2 to smithi20457400-46 2024-09-16T05:16:18.723 INFO:tasks.rados.rados.2.smithi204.stdout:1356: finishing write tid 3 to smithi20457400-46 2024-09-16T05:16:18.724 INFO:tasks.rados.rados.2.smithi204.stdout:1356: finishing write tid 4 to smithi20457400-46 2024-09-16T05:16:18.724 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 335 (ObjNum 479 snap 151 seq_num 479) dirty exists 2024-09-16T05:16:18.724 INFO:tasks.rados.rados.2.smithi204.stdout:1356: left oid 46 (ObjNum 479 snap 151 seq_num 479) 2024-09-16T05:16:18.724 INFO:tasks.rados.rados.2.smithi204.stdout:1356: done (2 left) 2024-09-16T05:16:18.724 INFO:tasks.rados.rados.2.smithi204.stdout:1357: done (1 left) 2024-09-16T05:16:18.724 INFO:tasks.rados.rados.2.smithi204.stdout:1360: write (excl) oid 40 current snap is 151 2024-09-16T05:16:18.725 INFO:tasks.rados.rados.2.smithi204.stdout:1360: seq_num 480 ranges {467360=649759,1671945=395415} 2024-09-16T05:16:18.730 INFO:tasks.rados.rados.2.smithi204.stdout:1360: writing smithi20457400-40 from 467360 to 1117119 tid 1 2024-09-16T05:16:18.735 INFO:tasks.rados.rados.2.smithi204.stdout:1360: writing smithi20457400-40 from 1671945 to 2067360 tid 2 2024-09-16T05:16:18.735 INFO:tasks.rados.rados.2.smithi204.stdout:1361: snap_remove snap 149 2024-09-16T05:16:18.742 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:18.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:18.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:18.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:18.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:18.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:18.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:19.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:19.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:19.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:16:19.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:16:19.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:18 smithi032 ceph-mon[27388]: osdmap e429: 12 total, 12 up, 12 in 2024-09-16T05:16:19.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:19.236 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:16:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:16:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:18 smithi196 ceph-mon[30802]: osdmap e429: 12 total, 12 up, 12 in 2024-09-16T05:16:19.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:19.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:19.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:19.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:19.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:19.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:16:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:18 smithi204 ceph-mon[30933]: osdmap e429: 12 total, 12 up, 12 in 2024-09-16T05:16:19.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:19.494 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:19.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:19.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:19.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:16:19.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:19.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:19.721 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:16:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:1361: done (2 left) 2024-09-16T05:16:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:1362: read oid 22 snap -1 2024-09-16T05:16:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:1362: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:1363: rollback oid 18 current snap is 151 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 18 to 150 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:1359: finishing rollback tid 1 to smithi20457400-20 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 129 snap 0 seq_num 4281445904) dirty dne 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:1360: finishing write tid 1 to smithi20457400-40 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:1360: finishing write tid 2 to smithi20457400-40 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:1360: finishing write tid 3 to smithi20457400-40 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:1360: finishing write tid 4 to smithi20457400-40 2024-09-16T05:16:19.726 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 351 (ObjNum 480 snap 151 seq_num 480) dirty exists 2024-09-16T05:16:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1360: left oid 40 (ObjNum 480 snap 151 seq_num 480) 2024-09-16T05:16:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1359: done (3 left) 2024-09-16T05:16:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1360: done (2 left) 2024-09-16T05:16:19.727 INFO:tasks.rados.rados.2.smithi204.stdout:1364: snap_remove snap 131 2024-09-16T05:16:19.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:19.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:19.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:19.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:19.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:19.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:19.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:19.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:20.007 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:20.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:20.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:20.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:20.195 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":805,"stamp":"2024-09-16T05:16:18.719569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226842601,"num_objects":93,"num_object_clones":44,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":8,"num_read":962,"num_read_kb":1011604,"num_write":641,"num_write_kb":286880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18988,"ondisk_log_size":18988,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":195,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":792964,"kb_used_data":338528,"kb_used_omap":27,"kb_used_meta":454116,"kb_avail":1124148860,"statfs":{"total":1151940427776,"available":1151128432640,"internally_reserved":0,"allocated":346652672,"data_stored":368655880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":465014951},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4681049,"num_objects":5,"num_object_clones":5,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":3,"num_read":13,"num_read_kb":13015,"num_write":10,"num_write_kb":4468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.784800"},"pg_stats":[{"pgid":"2.4b","version":"244'177","reported_seq":1748,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.936109+0000","last_change":"2024-09-16T05:16:11.727871+0000","last_active":"2024-09-16T05:16:14.936109+0000","last_peered":"2024-09-16T05:16:14.936109+0000","last_clean":"2024-09-16T05:16:14.936109+0000","last_became_active":"2024-09-16T05:14:30.853998+0000","last_became_peered":"2024-09-16T05:14:30.853998+0000","last_unstale":"2024-09-16T05:16:14.936109+0000","last_undegraded":"2024-09-16T05:16:14.936109+0000","last_fullsized":"2024-09-16T05:16:14.936109+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:16:01.869393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:01.869393+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:22.437016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001185778,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.4a","version":"420'473","reported_seq":2007,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.738233+0000","last_change":"2024-09-16T05:16:15.738552+0000","last_active":"2024-09-16T05:16:15.738233+0000","last_peered":"2024-09-16T05:16:15.738233+0000","last_clean":"2024-09-16T05:16:15.738233+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:16:15.738233+0000","last_undegraded":"2024-09-16T05:16:15.738233+0000","last_fullsized":"2024-09-16T05:16:15.738233+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"378'461","last_scrub_stamp":"2024-09-16T05:15:37.597739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:37.597739+0000","objects_scrubbed":1,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:56.313733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033579909999999998,"stat_sum":{"num_bytes":2806835,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":9,"num_write_kb":3804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":1789,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.717840+0000","last_change":"2024-09-16T05:16:15.718105+0000","last_active":"2024-09-16T05:16:15.717840+0000","last_peered":"2024-09-16T05:16:15.717840+0000","last_clean":"2024-09-16T05:16:15.717840+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:16:15.717840+0000","last_undegraded":"2024-09-16T05:16:15.717840+0000","last_fullsized":"2024-09-16T05:16:15.717840+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:16:00.136780+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.136780+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:19.429930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065247899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.48","version":"420'210","reported_seq":1737,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084325+0000","last_change":"2024-09-16T05:16:11.721493+0000","last_active":"2024-09-16T05:16:15.084325+0000","last_peered":"2024-09-16T05:16:15.084325+0000","last_clean":"2024-09-16T05:16:15.084325+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:16:15.084325+0000","last_undegraded":"2024-09-16T05:16:15.084325+0000","last_fullsized":"2024-09-16T05:16:15.084325+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"378'179","last_scrub_stamp":"2024-09-16T05:15:39.057858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.057858+0000","objects_scrubbed":1,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:55.391343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047282599999999998,"stat_sum":{"num_bytes":8448112,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":20677,"num_write":7,"num_write_kb":8206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.47","version":"0'0","reported_seq":554,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.950180+0000","last_change":"2024-09-16T05:16:12.563857+0000","last_active":"2024-09-16T05:16:13.950180+0000","last_peered":"2024-09-16T05:16:13.950180+0000","last_clean":"2024-09-16T05:16:13.950180+0000","last_became_active":"2024-09-16T05:15:16.867821+0000","last_became_peered":"2024-09-16T05:15:16.867821+0000","last_unstale":"2024-09-16T05:16:13.950180+0000","last_undegraded":"2024-09-16T05:16:13.950180+0000","last_fullsized":"2024-09-16T05:16:13.950180+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:57.698141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.698141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:03.728992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00032338899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.46","version":"116'9","reported_seq":1572,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.936301+0000","last_change":"2024-09-16T05:16:11.727530+0000","last_active":"2024-09-16T05:16:14.936301+0000","last_peered":"2024-09-16T05:16:14.936301+0000","last_clean":"2024-09-16T05:16:14.936301+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:16:14.936301+0000","last_undegraded":"2024-09-16T05:16:14.936301+0000","last_fullsized":"2024-09-16T05:16:14.936301+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:15:37.935128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:37.935128+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:01.264717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001045001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.45","version":"305'464","reported_seq":1999,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.936829+0000","last_change":"2024-09-16T05:16:11.728198+0000","last_active":"2024-09-16T05:16:14.936829+0000","last_peered":"2024-09-16T05:16:14.936829+0000","last_clean":"2024-09-16T05:16:14.936829+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:16:14.936829+0000","last_undegraded":"2024-09-16T05:16:14.936829+0000","last_fullsized":"2024-09-16T05:16:14.936829+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:15:59.876915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.876915+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:05.370433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017299360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":1654,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.936329+0000","last_change":"2024-09-16T05:16:11.728741+0000","last_active":"2024-09-16T05:16:14.936329+0000","last_peered":"2024-09-16T05:16:14.936329+0000","last_clean":"2024-09-16T05:16:14.936329+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:16:14.936329+0000","last_undegraded":"2024-09-16T05:16:14.936329+0000","last_fullsized":"2024-09-16T05:16:14.936329+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.970206+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:50.900831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001565515,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":565,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724123+0000","last_change":"2024-09-16T05:16:15.743042+0000","last_active":"2024-09-16T05:16:16.724123+0000","last_peered":"2024-09-16T05:16:16.724123+0000","last_clean":"2024-09-16T05:16:16.724123+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:16:16.724123+0000","last_undegraded":"2024-09-16T05:16:16.724123+0000","last_fullsized":"2024-09-16T05:16:16.724123+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:16:14.198462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.198462+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.429454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040986010000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":522,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.924336+0000","last_change":"2024-09-16T05:16:11.717961+0000","last_active":"2024-09-16T05:16:13.924336+0000","last_peered":"2024-09-16T05:16:13.924336+0000","last_clean":"2024-09-16T05:16:13.924336+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:16:13.924336+0000","last_undegraded":"2024-09-16T05:16:13.924336+0000","last_fullsized":"2024-09-16T05:16:13.924336+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:33.678710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:33.678710+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:50.768879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098624900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.41","version":"178'50","reported_seq":643,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.734927+0000","last_change":"2024-09-16T05:16:15.736151+0000","last_active":"2024-09-16T05:16:15.734927+0000","last_peered":"2024-09-16T05:16:15.734927+0000","last_clean":"2024-09-16T05:16:15.734927+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:16:15.734927+0000","last_undegraded":"2024-09-16T05:16:15.734927+0000","last_fullsized":"2024-09-16T05:16:15.734927+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:15.794439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.794439+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:16.035053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015351060000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":1585,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-16T05:16:11.749086+0000","last_change":"2024-09-16T05:16:11.749324+0000","last_active":"2024-09-16T05:16:11.749086+0000","last_peered":"2024-09-16T05:16:11.749086+0000","last_clean":"2024-09-16T05:16:11.749086+0000","last_became_active":"2024-09-16T05:13:45.682851+0000","last_became_peered":"2024-09-16T05:13:45.682851+0000","last_unstale":"2024-09-16T05:16:11.749086+0000","last_undegraded":"2024-09-16T05:16:11.749086+0000","last_fullsized":"2024-09-16T05:16:11.749086+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:07.789202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.789202+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.413901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027230259999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.3f","version":"426'288","reported_seq":1859,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724039+0000","last_change":"2024-09-16T05:16:15.743487+0000","last_active":"2024-09-16T05:16:16.724039+0000","last_peered":"2024-09-16T05:16:16.724039+0000","last_clean":"2024-09-16T05:16:16.724039+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:16:16.724039+0000","last_undegraded":"2024-09-16T05:16:16.724039+0000","last_fullsized":"2024-09-16T05:16:16.724039+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"357'270","last_scrub_stamp":"2024-09-16T05:15:09.797411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.797411+0000","objects_scrubbed":3,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.503407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0048945430000000003,"stat_sum":{"num_bytes":3464390,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":22717,"num_write":19,"num_write_kb":4627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.3e","version":"426'359","reported_seq":1891,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.877017+0000","last_change":"2024-09-16T05:16:15.745201+0000","last_active":"2024-09-16T05:16:16.877017+0000","last_peered":"2024-09-16T05:16:16.877017+0000","last_clean":"2024-09-16T05:16:16.877017+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:16:16.877017+0000","last_undegraded":"2024-09-16T05:16:16.877017+0000","last_fullsized":"2024-09-16T05:16:16.877017+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"360'342","last_scrub_stamp":"2024-09-16T05:15:09.920637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.920637+0000","objects_scrubbed":4,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0087812359999999996,"stat_sum":{"num_bytes":9025557,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":6,"num_write_kb":8815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2094,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.936846+0000","last_change":"2024-09-16T05:16:11.729340+0000","last_active":"2024-09-16T05:16:14.936846+0000","last_peered":"2024-09-16T05:16:14.936846+0000","last_clean":"2024-09-16T05:16:14.936846+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:16:14.936846+0000","last_undegraded":"2024-09-16T05:16:14.936846+0000","last_fullsized":"2024-09-16T05:16:14.936846+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:15:09.150849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:09.150849+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:21.881529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015811460000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.3c","version":"427'471","reported_seq":2129,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724299+0000","last_change":"2024-09-16T05:16:15.744281+0000","last_active":"2024-09-16T05:16:16.724299+0000","last_peered":"2024-09-16T05:16:16.724299+0000","last_clean":"2024-09-16T05:16:16.724299+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:16:16.724299+0000","last_undegraded":"2024-09-16T05:16:16.724299+0000","last_fullsized":"2024-09-16T05:16:16.724299+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"342'400","last_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:14:51.845721+0000","objects_scrubbed":6,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.477184+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0050174299999999998,"stat_sum":{"num_bytes":10033977,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":44,"num_read_kb":48649,"num_write":27,"num_write_kb":14749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.3b","version":"419'311","reported_seq":1876,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.720811+0000","last_change":"2024-09-16T05:16:15.721115+0000","last_active":"2024-09-16T05:16:15.720811+0000","last_peered":"2024-09-16T05:16:15.720811+0000","last_clean":"2024-09-16T05:16:15.720811+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:16:15.720811+0000","last_undegraded":"2024-09-16T05:16:15.720811+0000","last_fullsized":"2024-09-16T05:16:15.720811+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"300'290","last_scrub_stamp":"2024-09-16T05:15:06.955285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.955285+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.884622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0034356669999999999,"stat_sum":{"num_bytes":5277801,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":46642,"num_write":5,"num_write_kb":1725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.3a","version":"420'482","reported_seq":2000,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084493+0000","last_change":"2024-09-16T05:16:11.722176+0000","last_active":"2024-09-16T05:16:15.084493+0000","last_peered":"2024-09-16T05:16:15.084493+0000","last_clean":"2024-09-16T05:16:15.084493+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:16:15.084493+0000","last_undegraded":"2024-09-16T05:16:15.084493+0000","last_fullsized":"2024-09-16T05:16:15.084493+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"356'429","last_scrub_stamp":"2024-09-16T05:15:10.141866+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.141866+0000","objects_scrubbed":2,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001045947,"stat_sum":{"num_bytes":7237161,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":9241,"num_write":26,"num_write_kb":11603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.39","version":"182'60","reported_seq":641,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084423+0000","last_change":"2024-09-16T05:16:11.721942+0000","last_active":"2024-09-16T05:16:15.084423+0000","last_peered":"2024-09-16T05:16:15.084423+0000","last_clean":"2024-09-16T05:16:15.084423+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:16:15.084423+0000","last_undegraded":"2024-09-16T05:16:15.084423+0000","last_fullsized":"2024-09-16T05:16:15.084423+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:15.232072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.232072+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:19.813175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086100200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.38","version":"181'57","reported_seq":635,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.924438+0000","last_change":"2024-09-16T05:16:11.717050+0000","last_active":"2024-09-16T05:16:13.924438+0000","last_peered":"2024-09-16T05:16:13.924438+0000","last_clean":"2024-09-16T05:16:13.924438+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:16:13.924438+0000","last_undegraded":"2024-09-16T05:16:13.924438+0000","last_fullsized":"2024-09-16T05:16:13.924438+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:13.450261+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:13.450261+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:16.843063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043911200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.37","version":"0'0","reported_seq":731,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.736297+0000","last_change":"2024-09-16T05:16:15.736581+0000","last_active":"2024-09-16T05:16:15.736297+0000","last_peered":"2024-09-16T05:16:15.736297+0000","last_clean":"2024-09-16T05:16:15.736297+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:16:15.736297+0000","last_undegraded":"2024-09-16T05:16:15.736297+0000","last_fullsized":"2024-09-16T05:16:15.736297+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:14:53.650972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001507048,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":730,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.936479+0000","last_change":"2024-09-16T05:16:11.728325+0000","last_active":"2024-09-16T05:16:14.936479+0000","last_peered":"2024-09-16T05:16:14.936479+0000","last_clean":"2024-09-16T05:16:14.936479+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:16:14.936479+0000","last_undegraded":"2024-09-16T05:16:14.936479+0000","last_fullsized":"2024-09-16T05:16:14.936479+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.904155+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.415797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001616639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.35","version":"419'512","reported_seq":2096,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.937025+0000","last_change":"2024-09-16T05:16:11.729380+0000","last_active":"2024-09-16T05:16:14.937025+0000","last_peered":"2024-09-16T05:16:14.937025+0000","last_clean":"2024-09-16T05:16:14.937025+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:16:14.937025+0000","last_undegraded":"2024-09-16T05:16:14.937025+0000","last_fullsized":"2024-09-16T05:16:14.937025+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"410'508","last_scrub_stamp":"2024-09-16T05:16:00.864640+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.864640+0000","objects_scrubbed":4,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:25.865555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014045030000000001,"stat_sum":{"num_bytes":3193396,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24061,"num_write":19,"num_write_kb":7769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.34","version":"426'337","reported_seq":1508,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.720723+0000","last_change":"2024-09-16T05:16:15.721001+0000","last_active":"2024-09-16T05:16:15.720723+0000","last_peered":"2024-09-16T05:16:15.720723+0000","last_clean":"2024-09-16T05:16:15.720723+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:16:15.720723+0000","last_undegraded":"2024-09-16T05:16:15.720723+0000","last_fullsized":"2024-09-16T05:16:15.720723+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"420'335","last_scrub_stamp":"2024-09-16T05:16:15.093220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:15.093220+0000","objects_scrubbed":4,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:42.787705+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0034372959999999998,"stat_sum":{"num_bytes":9441369,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":41073,"num_write":20,"num_write_kb":10049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":736,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724001+0000","last_change":"2024-09-16T05:16:15.741584+0000","last_active":"2024-09-16T05:16:16.724001+0000","last_peered":"2024-09-16T05:16:16.724001+0000","last_clean":"2024-09-16T05:16:16.724001+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:16:16.724001+0000","last_undegraded":"2024-09-16T05:16:16.724001+0000","last_fullsized":"2024-09-16T05:16:16.724001+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:46.792111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:46.792111+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002970473,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.32","version":"419'127","reported_seq":821,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.877281+0000","last_change":"2024-09-16T05:16:15.735638+0000","last_active":"2024-09-16T05:16:16.877281+0000","last_peered":"2024-09-16T05:16:16.877281+0000","last_clean":"2024-09-16T05:16:16.877281+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:16:16.877281+0000","last_undegraded":"2024-09-16T05:16:16.877281+0000","last_fullsized":"2024-09-16T05:16:16.877281+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"340'101","last_scrub_stamp":"2024-09-16T05:14:49.915945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.915945+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00054994400000000002,"stat_sum":{"num_bytes":3109361,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":8515,"num_write":17,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.31","version":"422'174","reported_seq":1365,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.950223+0000","last_change":"2024-09-16T05:16:12.565252+0000","last_active":"2024-09-16T05:16:13.950223+0000","last_peered":"2024-09-16T05:16:13.950223+0000","last_clean":"2024-09-16T05:16:13.950223+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:16:13.950223+0000","last_undegraded":"2024-09-16T05:16:13.950223+0000","last_fullsized":"2024-09-16T05:16:13.950223+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"402'164","last_scrub_stamp":"2024-09-16T05:15:53.749444+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.749444+0000","objects_scrubbed":2,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:02.524019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001718617,"stat_sum":{"num_bytes":6521248,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":49186,"num_write":14,"num_write_kb":7228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.30","version":"182'16","reported_seq":695,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.735441+0000","last_change":"2024-09-16T05:16:15.735728+0000","last_active":"2024-09-16T05:16:15.735441+0000","last_peered":"2024-09-16T05:16:15.735441+0000","last_clean":"2024-09-16T05:16:15.735441+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:16:15.735441+0000","last_undegraded":"2024-09-16T05:16:15.735441+0000","last_fullsized":"2024-09-16T05:16:15.735441+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:15:06.783241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.783241+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.793858+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0010954300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":456,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.717184+0000","last_change":"2024-09-16T05:15:57.054819+0000","last_active":"2024-09-16T05:16:15.717184+0000","last_peered":"2024-09-16T05:16:15.717184+0000","last_clean":"2024-09-16T05:16:15.717184+0000","last_became_active":"2024-09-16T05:15:56.716090+0000","last_became_peered":"2024-09-16T05:15:56.716090+0000","last_unstale":"2024-09-16T05:16:15.717184+0000","last_undegraded":"2024-09-16T05:16:15.717184+0000","last_fullsized":"2024-09-16T05:16:15.717184+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.054740+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.054740+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.994935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1279,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.728397+0000","last_change":"2024-09-16T05:16:15.721504+0000","last_active":"2024-09-16T05:16:16.728397+0000","last_peered":"2024-09-16T05:16:16.728397+0000","last_clean":"2024-09-16T05:16:16.728397+0000","last_became_active":"2024-09-16T05:13:25.679267+0000","last_became_peered":"2024-09-16T05:13:25.679267+0000","last_unstale":"2024-09-16T05:16:16.728397+0000","last_undegraded":"2024-09-16T05:16:16.728397+0000","last_fullsized":"2024-09-16T05:16:16.728397+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:15:53.135605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.135605+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:56.154151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011460419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1087,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.935987+0000","last_change":"2024-09-16T05:15:56.852148+0000","last_active":"2024-09-16T05:16:14.935987+0000","last_peered":"2024-09-16T05:16:14.935987+0000","last_clean":"2024-09-16T05:16:14.935987+0000","last_became_active":"2024-09-16T05:15:56.714681+0000","last_became_peered":"2024-09-16T05:15:56.714681+0000","last_unstale":"2024-09-16T05:16:14.935987+0000","last_undegraded":"2024-09-16T05:16:14.935987+0000","last_fullsized":"2024-09-16T05:16:14.935987+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.852021+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.852021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:11.168034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":1182,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.924354+0000","last_change":"2024-09-16T05:16:13.342338+0000","last_active":"2024-09-16T05:16:13.924354+0000","last_peered":"2024-09-16T05:16:13.924354+0000","last_clean":"2024-09-16T05:16:13.924354+0000","last_became_active":"2024-09-16T05:13:25.680445+0000","last_became_peered":"2024-09-16T05:13:25.680445+0000","last_unstale":"2024-09-16T05:16:13.924354+0000","last_undegraded":"2024-09-16T05:16:13.924354+0000","last_fullsized":"2024-09-16T05:16:13.924354+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:13.342246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:13.342246+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.687713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041546500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"1.c","version":"35'139","reported_seq":413,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.728568+0000","last_change":"2024-09-16T05:15:56.828705+0000","last_active":"2024-09-16T05:16:17.728568+0000","last_peered":"2024-09-16T05:16:17.728568+0000","last_clean":"2024-09-16T05:16:17.728568+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:16:17.728568+0000","last_undegraded":"2024-09-16T05:16:17.728568+0000","last_fullsized":"2024-09-16T05:16:17.728568+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.828595+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.828595+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:25.780000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"360'281","reported_seq":1846,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.924470+0000","last_change":"2024-09-16T05:16:11.717417+0000","last_active":"2024-09-16T05:16:13.924470+0000","last_peered":"2024-09-16T05:16:13.924470+0000","last_clean":"2024-09-16T05:16:13.924470+0000","last_became_active":"2024-09-16T05:13:25.680481+0000","last_became_peered":"2024-09-16T05:13:25.680481+0000","last_unstale":"2024-09-16T05:16:13.924470+0000","last_undegraded":"2024-09-16T05:16:13.924470+0000","last_fullsized":"2024-09-16T05:16:13.924470+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:16:02.438575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.438575+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:04.069671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071406599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"1.d","version":"35'139","reported_seq":577,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.723901+0000","last_change":"2024-09-16T05:15:57.170014+0000","last_active":"2024-09-16T05:16:16.723901+0000","last_peered":"2024-09-16T05:16:16.723901+0000","last_clean":"2024-09-16T05:16:16.723901+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:16:16.723901+0000","last_undegraded":"2024-09-16T05:16:16.723901+0000","last_fullsized":"2024-09-16T05:16:16.723901+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.169945+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.169945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.932365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":1719,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.722026+0000","last_change":"2024-09-16T05:16:17.722366+0000","last_active":"2024-09-16T05:16:17.722026+0000","last_peered":"2024-09-16T05:16:17.722026+0000","last_clean":"2024-09-16T05:16:17.722026+0000","last_became_active":"2024-09-16T05:14:40.261276+0000","last_became_peered":"2024-09-16T05:14:40.261276+0000","last_unstale":"2024-09-16T05:16:17.722026+0000","last_undegraded":"2024-09-16T05:16:17.722026+0000","last_fullsized":"2024-09-16T05:16:17.722026+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:15:57.720091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.720091+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:02.613781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000733886,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":565,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.910139+0000","last_change":"2024-09-16T05:15:56.983440+0000","last_active":"2024-09-16T05:16:15.910139+0000","last_peered":"2024-09-16T05:16:15.910139+0000","last_clean":"2024-09-16T05:16:15.910139+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:16:15.910139+0000","last_undegraded":"2024-09-16T05:16:15.910139+0000","last_fullsized":"2024-09-16T05:16:15.910139+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.983358+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.983358+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:59.889126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"419'461","reported_seq":2024,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084897+0000","last_change":"2024-09-16T05:16:11.726820+0000","last_active":"2024-09-16T05:16:15.084897+0000","last_peered":"2024-09-16T05:16:15.084897+0000","last_clean":"2024-09-16T05:16:15.084897+0000","last_became_active":"2024-09-16T05:12:59.752830+0000","last_became_peered":"2024-09-16T05:12:59.752830+0000","last_unstale":"2024-09-16T05:16:15.084897+0000","last_undegraded":"2024-09-16T05:16:15.084897+0000","last_fullsized":"2024-09-16T05:16:15.084897+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'459","last_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.081612+0000","objects_scrubbed":3,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:18.076800+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0050060759999999999,"stat_sum":{"num_bytes":5647284,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":59022,"num_write":30,"num_write_kb":9875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"1.f","version":"35'139","reported_seq":556,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.717191+0000","last_change":"2024-09-16T05:15:59.140824+0000","last_active":"2024-09-16T05:16:15.717191+0000","last_peered":"2024-09-16T05:16:15.717191+0000","last_clean":"2024-09-16T05:16:15.717191+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:16:15.717191+0000","last_undegraded":"2024-09-16T05:16:15.717191+0000","last_fullsized":"2024-09-16T05:16:15.717191+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.140738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.775574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"426'270","reported_seq":1836,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.876994+0000","last_change":"2024-09-16T05:16:15.737963+0000","last_active":"2024-09-16T05:16:16.876994+0000","last_peered":"2024-09-16T05:16:16.876994+0000","last_clean":"2024-09-16T05:16:16.876994+0000","last_became_active":"2024-09-16T05:12:59.749512+0000","last_became_peered":"2024-09-16T05:12:59.749512+0000","last_unstale":"2024-09-16T05:16:16.876994+0000","last_undegraded":"2024-09-16T05:16:16.876994+0000","last_fullsized":"2024-09-16T05:16:16.876994+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'259","last_scrub_stamp":"2024-09-16T05:15:55.772875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.772875+0000","objects_scrubbed":1,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:09.493395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029543460000000001,"stat_sum":{"num_bytes":5464170,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33268,"num_write":21,"num_write_kb":8682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1092,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.877060+0000","last_change":"2024-09-16T05:15:56.819273+0000","last_active":"2024-09-16T05:16:16.877060+0000","last_peered":"2024-09-16T05:16:16.877060+0000","last_clean":"2024-09-16T05:16:16.877060+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:16:16.877060+0000","last_undegraded":"2024-09-16T05:16:16.877060+0000","last_fullsized":"2024-09-16T05:16:16.877060+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.819151+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.819151+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:06.003491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"420'230","reported_seq":1803,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724213+0000","last_change":"2024-09-16T05:16:15.740763+0000","last_active":"2024-09-16T05:16:16.724213+0000","last_peered":"2024-09-16T05:16:16.724213+0000","last_clean":"2024-09-16T05:16:16.724213+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:16:16.724213+0000","last_undegraded":"2024-09-16T05:16:16.724213+0000","last_fullsized":"2024-09-16T05:16:16.724213+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'222","last_scrub_stamp":"2024-09-16T05:16:00.063191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.063191+0000","objects_scrubbed":3,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.875694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002436606,"stat_sum":{"num_bytes":4928340,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12880,"num_write":15,"num_write_kb":3848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.3","version":"397'99","reported_seq":1668,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.721069+0000","last_change":"2024-09-16T05:16:15.721316+0000","last_active":"2024-09-16T05:16:15.721069+0000","last_peered":"2024-09-16T05:16:15.721069+0000","last_clean":"2024-09-16T05:16:15.721069+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:16:15.721069+0000","last_undegraded":"2024-09-16T05:16:15.721069+0000","last_fullsized":"2024-09-16T05:16:15.721069+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'99","last_scrub_stamp":"2024-09-16T05:16:02.132568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.132568+0000","objects_scrubbed":1,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:20.957176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003359491,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":29733,"num_write":6,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":1086,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724493+0000","last_change":"2024-09-16T05:15:53.271304+0000","last_active":"2024-09-16T05:16:16.724493+0000","last_peered":"2024-09-16T05:16:16.724493+0000","last_clean":"2024-09-16T05:16:16.724493+0000","last_became_active":"2024-09-16T05:15:51.717864+0000","last_became_peered":"2024-09-16T05:15:51.717864+0000","last_unstale":"2024-09-16T05:16:16.724493+0000","last_undegraded":"2024-09-16T05:16:16.724493+0000","last_fullsized":"2024-09-16T05:16:16.724493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.271201+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.271201+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.039968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1576,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.718205+0000","last_change":"2024-09-16T05:16:15.718441+0000","last_active":"2024-09-16T05:16:15.718205+0000","last_peered":"2024-09-16T05:16:15.718205+0000","last_clean":"2024-09-16T05:16:15.718205+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:16:15.718205+0000","last_undegraded":"2024-09-16T05:16:15.718205+0000","last_fullsized":"2024-09-16T05:16:15.718205+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:55.047148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.047148+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.346062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001005901,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":724,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.734870+0000","last_change":"2024-09-16T05:15:54.643909+0000","last_active":"2024-09-16T05:16:15.734870+0000","last_peered":"2024-09-16T05:16:15.734870+0000","last_clean":"2024-09-16T05:16:15.734870+0000","last_became_active":"2024-09-16T05:15:51.723644+0000","last_became_peered":"2024-09-16T05:15:51.723644+0000","last_unstale":"2024-09-16T05:16:15.734870+0000","last_undegraded":"2024-09-16T05:16:15.734870+0000","last_fullsized":"2024-09-16T05:16:15.734870+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:54.643828+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:54.643828+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:59.687807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1081,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.723789+0000","last_change":"2024-09-16T05:15:54.232505+0000","last_active":"2024-09-16T05:16:16.723789+0000","last_peered":"2024-09-16T05:16:16.723789+0000","last_clean":"2024-09-16T05:16:16.723789+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:16:16.723789+0000","last_undegraded":"2024-09-16T05:16:16.723789+0000","last_fullsized":"2024-09-16T05:16:16.723789+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:54.232382+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:54.232382+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:02.812437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1531,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.721681+0000","last_change":"2024-09-16T05:16:17.721974+0000","last_active":"2024-09-16T05:16:17.721681+0000","last_peered":"2024-09-16T05:16:17.721681+0000","last_clean":"2024-09-16T05:16:17.721681+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:16:17.721681+0000","last_undegraded":"2024-09-16T05:16:17.721681+0000","last_fullsized":"2024-09-16T05:16:17.721681+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:15:07.898675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.898675+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:28.239152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048967399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1094,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.728413+0000","last_change":"2024-09-16T05:15:52.844991+0000","last_active":"2024-09-16T05:16:17.728413+0000","last_peered":"2024-09-16T05:16:17.728413+0000","last_clean":"2024-09-16T05:16:17.728413+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:16:17.728413+0000","last_undegraded":"2024-09-16T05:16:17.728413+0000","last_fullsized":"2024-09-16T05:16:17.728413+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:52.844849+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:52.844849+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:03.165293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1601,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724340+0000","last_change":"2024-09-16T05:16:15.744268+0000","last_active":"2024-09-16T05:16:16.724340+0000","last_peered":"2024-09-16T05:16:16.724340+0000","last_clean":"2024-09-16T05:16:16.724340+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:16:16.724340+0000","last_undegraded":"2024-09-16T05:16:16.724340+0000","last_fullsized":"2024-09-16T05:16:16.724340+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:15:06.755407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.755407+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:29.509511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046347150000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":811,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084674+0000","last_change":"2024-09-16T05:16:11.726476+0000","last_active":"2024-09-16T05:16:15.084674+0000","last_peered":"2024-09-16T05:16:15.084674+0000","last_clean":"2024-09-16T05:16:15.084674+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:16:15.084674+0000","last_undegraded":"2024-09-16T05:16:15.084674+0000","last_fullsized":"2024-09-16T05:16:15.084674+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:31.009189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.009189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:58.254749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0048589360000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"1.4","version":"35'139","reported_seq":1047,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.877266+0000","last_change":"2024-09-16T05:15:53.804026+0000","last_active":"2024-09-16T05:16:16.877266+0000","last_peered":"2024-09-16T05:16:16.877266+0000","last_clean":"2024-09-16T05:16:16.877266+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:16:16.877266+0000","last_undegraded":"2024-09-16T05:16:16.877266+0000","last_fullsized":"2024-09-16T05:16:16.877266+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.803903+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.803903+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:57.012754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":14,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.725079+0000","last_change":"2024-09-16T05:16:17.725020+0000","last_active":"2024-09-16T05:16:17.725079+0000","last_peered":"2024-09-16T05:16:17.725079+0000","last_clean":"2024-09-16T05:16:17.725079+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:16:17.725079+0000","last_undegraded":"2024-09-16T05:16:17.725079+0000","last_fullsized":"2024-09-16T05:16:17.725079+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:15:35.928579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:35.928579+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:58.316363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000394442,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":829,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.735288+0000","last_change":"2024-09-16T05:15:56.607423+0000","last_active":"2024-09-16T05:16:15.735288+0000","last_peered":"2024-09-16T05:16:15.735288+0000","last_clean":"2024-09-16T05:16:15.735288+0000","last_became_active":"2024-09-16T05:15:51.724156+0000","last_became_peered":"2024-09-16T05:15:51.724156+0000","last_unstale":"2024-09-16T05:16:15.735288+0000","last_undegraded":"2024-09-16T05:16:15.735288+0000","last_fullsized":"2024-09-16T05:16:15.735288+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.607332+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.607332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:04.191878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"419'482","reported_seq":2028,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084748+0000","last_change":"2024-09-16T05:16:11.727084+0000","last_active":"2024-09-16T05:16:15.084748+0000","last_peered":"2024-09-16T05:16:15.084748+0000","last_clean":"2024-09-16T05:16:15.084748+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:16:15.084748+0000","last_undegraded":"2024-09-16T05:16:15.084748+0000","last_fullsized":"2024-09-16T05:16:15.084748+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'479","last_scrub_stamp":"2024-09-16T05:15:57.067519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.067519+0000","objects_scrubbed":2,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.105783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0054546739999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":8850,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":725,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.734683+0000","last_change":"2024-09-16T05:15:55.644520+0000","last_active":"2024-09-16T05:16:15.734683+0000","last_peered":"2024-09-16T05:16:15.734683+0000","last_clean":"2024-09-16T05:16:15.734683+0000","last_became_active":"2024-09-16T05:15:51.722851+0000","last_became_peered":"2024-09-16T05:15:51.722851+0000","last_unstale":"2024-09-16T05:16:15.734683+0000","last_undegraded":"2024-09-16T05:16:15.734683+0000","last_fullsized":"2024-09-16T05:16:15.734683+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:55.644436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.644436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:05.843216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1651,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.936607+0000","last_change":"2024-09-16T05:16:11.728970+0000","last_active":"2024-09-16T05:16:14.936607+0000","last_peered":"2024-09-16T05:16:14.936607+0000","last_clean":"2024-09-16T05:16:14.936607+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:16:14.936607+0000","last_undegraded":"2024-09-16T05:16:14.936607+0000","last_fullsized":"2024-09-16T05:16:14.936607+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:57.869742+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.869742+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:19.502843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001692947,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":909,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.734814+0000","last_change":"2024-09-16T05:15:53.643617+0000","last_active":"2024-09-16T05:16:15.734814+0000","last_peered":"2024-09-16T05:16:15.734814+0000","last_clean":"2024-09-16T05:16:15.734814+0000","last_became_active":"2024-09-16T05:15:51.718810+0000","last_became_peered":"2024-09-16T05:15:51.718810+0000","last_unstale":"2024-09-16T05:16:15.734814+0000","last_undegraded":"2024-09-16T05:16:15.734814+0000","last_fullsized":"2024-09-16T05:16:15.734814+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.643528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.643528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:08.515058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":1741,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.924526+0000","last_change":"2024-09-16T05:16:11.717689+0000","last_active":"2024-09-16T05:16:13.924526+0000","last_peered":"2024-09-16T05:16:13.924526+0000","last_clean":"2024-09-16T05:16:13.924526+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:16:13.924526+0000","last_undegraded":"2024-09-16T05:16:13.924526+0000","last_fullsized":"2024-09-16T05:16:13.924526+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:15:41.633260+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:09.031416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086997099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":1076,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.717200+0000","last_change":"2024-09-16T05:15:15.084154+0000","last_active":"2024-09-16T05:16:15.717200+0000","last_peered":"2024-09-16T05:16:15.717200+0000","last_clean":"2024-09-16T05:16:15.717200+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:16:15.717200+0000","last_undegraded":"2024-09-16T05:16:15.717200+0000","last_fullsized":"2024-09-16T05:16:15.717200+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:15.084022+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:15.084022+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:18.557692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"419'465","reported_seq":2039,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.877386+0000","last_change":"2024-09-16T05:16:15.736107+0000","last_active":"2024-09-16T05:16:16.877386+0000","last_peered":"2024-09-16T05:16:16.877386+0000","last_clean":"2024-09-16T05:16:16.877386+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:16:16.877386+0000","last_undegraded":"2024-09-16T05:16:16.877386+0000","last_fullsized":"2024-09-16T05:16:16.877386+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'419","last_scrub_stamp":"2024-09-16T05:14:25.924121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:25.924121+0000","objects_scrubbed":4,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00094311099999999995,"stat_sum":{"num_bytes":3042133,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":30338,"num_write":5,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1069,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.723736+0000","last_change":"2024-09-16T05:14:11.602321+0000","last_active":"2024-09-16T05:16:16.723736+0000","last_peered":"2024-09-16T05:16:16.723736+0000","last_clean":"2024-09-16T05:16:16.723736+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:16:16.723736+0000","last_undegraded":"2024-09-16T05:16:16.723736+0000","last_fullsized":"2024-09-16T05:16:16.723736+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:14:11.602237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:14:11.602237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1788,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:14.935851+0000","last_change":"2024-09-16T05:16:11.726846+0000","last_active":"2024-09-16T05:16:14.935851+0000","last_peered":"2024-09-16T05:16:14.935851+0000","last_clean":"2024-09-16T05:16:14.935851+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:16:14.935851+0000","last_undegraded":"2024-09-16T05:16:14.935851+0000","last_fullsized":"2024-09-16T05:16:14.935851+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:15:32.922163+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:32.922163+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:50.778769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046106599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":1074,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084180+0000","last_change":"2024-09-16T05:15:14.182510+0000","last_active":"2024-09-16T05:16:15.084180+0000","last_peered":"2024-09-16T05:16:15.084180+0000","last_clean":"2024-09-16T05:16:15.084180+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:16:15.084180+0000","last_undegraded":"2024-09-16T05:16:15.084180+0000","last_fullsized":"2024-09-16T05:16:15.084180+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:14.182407+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:14.182407+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:17.321475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":560,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.950037+0000","last_change":"2024-09-16T05:15:56.743121+0000","last_active":"2024-09-16T05:16:13.950037+0000","last_peered":"2024-09-16T05:16:13.950037+0000","last_clean":"2024-09-16T05:16:13.950037+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:16:13.950037+0000","last_undegraded":"2024-09-16T05:16:13.950037+0000","last_fullsized":"2024-09-16T05:16:13.950037+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.743052+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.743052+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:06.703550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":703,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.738554+0000","last_change":"2024-09-16T05:16:15.739423+0000","last_active":"2024-09-16T05:16:15.738554+0000","last_peered":"2024-09-16T05:16:15.738554+0000","last_clean":"2024-09-16T05:16:15.738554+0000","last_became_active":"2024-09-16T05:13:31.747644+0000","last_became_peered":"2024-09-16T05:13:31.747644+0000","last_unstale":"2024-09-16T05:16:15.738554+0000","last_undegraded":"2024-09-16T05:16:15.738554+0000","last_fullsized":"2024-09-16T05:16:15.738554+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:16:14.771181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.771181+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.320519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0034755340000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1083,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.717038+0000","last_change":"2024-09-16T05:15:58.103711+0000","last_active":"2024-09-16T05:16:15.717038+0000","last_peered":"2024-09-16T05:16:15.717038+0000","last_clean":"2024-09-16T05:16:15.717038+0000","last_became_active":"2024-09-16T05:15:56.719062+0000","last_became_peered":"2024-09-16T05:15:56.719062+0000","last_unstale":"2024-09-16T05:16:15.717038+0000","last_undegraded":"2024-09-16T05:16:15.717038+0000","last_fullsized":"2024-09-16T05:16:15.717038+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:58.103601+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:58.103601+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:27.359233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":1181,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084662+0000","last_change":"2024-09-16T05:16:11.724460+0000","last_active":"2024-09-16T05:16:15.084662+0000","last_peered":"2024-09-16T05:16:15.084662+0000","last_clean":"2024-09-16T05:16:15.084662+0000","last_became_active":"2024-09-16T05:13:25.681297+0000","last_became_peered":"2024-09-16T05:13:25.681297+0000","last_unstale":"2024-09-16T05:16:15.084662+0000","last_undegraded":"2024-09-16T05:16:15.084662+0000","last_fullsized":"2024-09-16T05:16:15.084662+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:14:49.160833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:49.160833+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002962166,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.14","version":"422'279","reported_seq":1447,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084544+0000","last_change":"2024-09-16T05:16:13.148481+0000","last_active":"2024-09-16T05:16:15.084544+0000","last_peered":"2024-09-16T05:16:15.084544+0000","last_clean":"2024-09-16T05:16:15.084544+0000","last_became_active":"2024-09-16T05:13:25.684888+0000","last_became_peered":"2024-09-16T05:13:25.684888+0000","last_unstale":"2024-09-16T05:16:15.084544+0000","last_undegraded":"2024-09-16T05:16:15.084544+0000","last_fullsized":"2024-09-16T05:16:15.084544+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"422'279","last_scrub_stamp":"2024-09-16T05:16:13.148382+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:16:13.148382+0000","objects_scrubbed":2,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:42.679093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027430340000000001,"stat_sum":{"num_bytes":6538022,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":40956,"num_write":8,"num_write_kb":6386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.15","version":"422'497","reported_seq":2057,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084793+0000","last_change":"2024-09-16T05:16:11.726245+0000","last_active":"2024-09-16T05:16:15.084793+0000","last_peered":"2024-09-16T05:16:15.084793+0000","last_clean":"2024-09-16T05:16:15.084793+0000","last_became_active":"2024-09-16T05:13:25.680026+0000","last_became_peered":"2024-09-16T05:13:25.680026+0000","last_unstale":"2024-09-16T05:16:15.084793+0000","last_undegraded":"2024-09-16T05:16:15.084793+0000","last_fullsized":"2024-09-16T05:16:15.084793+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"357'436","last_scrub_stamp":"2024-09-16T05:15:07.101221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:07.101221+0000","objects_scrubbed":5,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:23.627691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0046758140000000004,"stat_sum":{"num_bytes":7544524,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":52139,"num_write":38,"num_write_kb":17001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.16","version":"426'141","reported_seq":1348,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.720335+0000","last_change":"2024-09-16T05:16:15.720688+0000","last_active":"2024-09-16T05:16:15.720335+0000","last_peered":"2024-09-16T05:16:15.720335+0000","last_clean":"2024-09-16T05:16:15.720335+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:16:15.720335+0000","last_undegraded":"2024-09-16T05:16:15.720335+0000","last_fullsized":"2024-09-16T05:16:15.720335+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'120","last_scrub_stamp":"2024-09-16T05:15:42.209530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:42.209530+0000","objects_scrubbed":2,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:43.224779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033492040000000002,"stat_sum":{"num_bytes":6356778,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":25311,"num_write":12,"num_write_kb":4969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":416,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.723651+0000","last_change":"2024-09-16T05:16:17.723651+0000","last_active":"2024-09-16T05:16:17.723651+0000","last_peered":"2024-09-16T05:16:17.723651+0000","last_clean":"2024-09-16T05:16:17.723651+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:16:17.723651+0000","last_undegraded":"2024-09-16T05:16:17.723651+0000","last_fullsized":"2024-09-16T05:16:17.723651+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.762419+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.762419+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:08.609426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1373,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.728576+0000","last_change":"2024-09-16T05:16:15.720928+0000","last_active":"2024-09-16T05:16:16.728576+0000","last_peered":"2024-09-16T05:16:16.728576+0000","last_clean":"2024-09-16T05:16:16.728576+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:16:16.728576+0000","last_undegraded":"2024-09-16T05:16:16.728576+0000","last_fullsized":"2024-09-16T05:16:16.728576+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:57.269939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.269939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:10.239395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074158099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1574,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084336+0000","last_change":"2024-09-16T05:16:11.721903+0000","last_active":"2024-09-16T05:16:15.084336+0000","last_peered":"2024-09-16T05:16:15.084336+0000","last_clean":"2024-09-16T05:16:15.084336+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:16:15.084336+0000","last_undegraded":"2024-09-16T05:16:15.084336+0000","last_fullsized":"2024-09-16T05:16:15.084336+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:15:17.188273+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:15:17.188273+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:20.736694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084563100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.19","version":"182'60","reported_seq":1194,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.728894+0000","last_change":"2024-09-16T05:16:17.729288+0000","last_active":"2024-09-16T05:16:17.728894+0000","last_peered":"2024-09-16T05:16:17.728894+0000","last_clean":"2024-09-16T05:16:17.728894+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:16:17.728894+0000","last_undegraded":"2024-09-16T05:16:17.728894+0000","last_fullsized":"2024-09-16T05:16:17.728894+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:15:06.893298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:06.893298+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:30.447624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000628043,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.1a","version":"420'647","reported_seq":2171,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.924653+0000","last_change":"2024-09-16T05:16:11.718638+0000","last_active":"2024-09-16T05:16:13.924653+0000","last_peered":"2024-09-16T05:16:13.924653+0000","last_clean":"2024-09-16T05:16:13.924653+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:16:13.924653+0000","last_undegraded":"2024-09-16T05:16:13.924653+0000","last_fullsized":"2024-09-16T05:16:13.924653+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'527","last_scrub_stamp":"2024-09-16T05:15:10.511970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:10.511970+0000","objects_scrubbed":9,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:16.487194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001493271,"stat_sum":{"num_bytes":13234316,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":66,"num_read_kb":57717,"num_write":45,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.1b","version":"426'358","reported_seq":1952,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.877304+0000","last_change":"2024-09-16T05:16:15.738838+0000","last_active":"2024-09-16T05:16:16.877304+0000","last_peered":"2024-09-16T05:16:16.877304+0000","last_clean":"2024-09-16T05:16:16.877304+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:16:16.877304+0000","last_undegraded":"2024-09-16T05:16:16.877304+0000","last_fullsized":"2024-09-16T05:16:16.877304+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"347'319","last_scrub_stamp":"2024-09-16T05:15:08.957118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.957118+0000","objects_scrubbed":3,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033450379999999998,"stat_sum":{"num_bytes":9445732,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":29314,"num_write":17,"num_write_kb":5740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.1c","version":"426'386","reported_seq":2009,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.737864+0000","last_change":"2024-09-16T05:16:15.738156+0000","last_active":"2024-09-16T05:16:15.737864+0000","last_peered":"2024-09-16T05:16:15.737864+0000","last_clean":"2024-09-16T05:16:15.737864+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:16:15.737864+0000","last_undegraded":"2024-09-16T05:16:15.737864+0000","last_fullsized":"2024-09-16T05:16:15.737864+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'364","last_scrub_stamp":"2024-09-16T05:15:08.759106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:08.759106+0000","objects_scrubbed":2,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:29.447954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003325841,"stat_sum":{"num_bytes":3327852,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.1d","version":"426'692","reported_seq":2333,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.740961+0000","last_change":"2024-09-16T05:16:15.741247+0000","last_active":"2024-09-16T05:16:15.740961+0000","last_peered":"2024-09-16T05:16:15.740961+0000","last_clean":"2024-09-16T05:16:15.740961+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:16:15.740961+0000","last_undegraded":"2024-09-16T05:16:15.740961+0000","last_fullsized":"2024-09-16T05:16:15.740961+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'584","last_scrub_stamp":"2024-09-16T05:15:16.833446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:16.833446+0000","objects_scrubbed":8,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:28.805498+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061525210000000002,"stat_sum":{"num_bytes":21486492,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":41647,"num_write":51,"num_write_kb":25270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.1e","version":"378'347","reported_seq":1941,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.729380+0000","last_change":"2024-09-16T05:16:17.729681+0000","last_active":"2024-09-16T05:16:17.729380+0000","last_peered":"2024-09-16T05:16:17.729380+0000","last_clean":"2024-09-16T05:16:17.729380+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:16:17.729380+0000","last_undegraded":"2024-09-16T05:16:17.729380+0000","last_fullsized":"2024-09-16T05:16:17.729380+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"362'335","last_scrub_stamp":"2024-09-16T05:15:12.926534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:12.926534+0000","objects_scrubbed":3,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:25.286974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000662486,"stat_sum":{"num_bytes":2487502,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":26093,"num_write":5,"num_write_kb":3828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":1788,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.950364+0000","last_change":"2024-09-16T05:16:12.564086+0000","last_active":"2024-09-16T05:16:13.950364+0000","last_peered":"2024-09-16T05:16:13.950364+0000","last_clean":"2024-09-16T05:16:13.950364+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:16:13.950364+0000","last_undegraded":"2024-09-16T05:16:13.950364+0000","last_fullsized":"2024-09-16T05:16:13.950364+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:14:52.883562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.883562+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:21.814270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051183700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.20","version":"426'154","reported_seq":1746,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.723940+0000","last_change":"2024-09-16T05:16:15.741866+0000","last_active":"2024-09-16T05:16:16.723940+0000","last_peered":"2024-09-16T05:16:16.723940+0000","last_clean":"2024-09-16T05:16:16.723940+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:16:16.723940+0000","last_undegraded":"2024-09-16T05:16:16.723940+0000","last_fullsized":"2024-09-16T05:16:16.723940+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'128","last_scrub_stamp":"2024-09-16T05:15:31.623695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.623695+0000","objects_scrubbed":2,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:34.273127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037446390000000001,"stat_sum":{"num_bytes":6101040,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":79406,"num_write":17,"num_write_kb":9878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":1530,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.721779+0000","last_change":"2024-09-16T05:16:17.722202+0000","last_active":"2024-09-16T05:16:17.721779+0000","last_peered":"2024-09-16T05:16:17.721779+0000","last_clean":"2024-09-16T05:16:17.721779+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:16:17.721779+0000","last_undegraded":"2024-09-16T05:16:17.721779+0000","last_fullsized":"2024-09-16T05:16:17.721779+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:14:52.848788+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066512899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":703,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.739391+0000","last_change":"2024-09-16T05:16:15.739837+0000","last_active":"2024-09-16T05:16:15.739391+0000","last_peered":"2024-09-16T05:16:15.739391+0000","last_clean":"2024-09-16T05:16:15.739391+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:16:15.739391+0000","last_undegraded":"2024-09-16T05:16:15.739391+0000","last_fullsized":"2024-09-16T05:16:15.739391+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:15:31.582158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:31.582158+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:40.211093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003928244,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":1069,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.729244+0000","last_change":"2024-09-16T05:16:17.729518+0000","last_active":"2024-09-16T05:16:17.729244+0000","last_peered":"2024-09-16T05:16:17.729244+0000","last_clean":"2024-09-16T05:16:17.729244+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:16:17.729244+0000","last_undegraded":"2024-09-16T05:16:17.729244+0000","last_fullsized":"2024-09-16T05:16:17.729244+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:16:17.214142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.214142+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:32.497148+0000","scrub_duration":429,"objects_trimmed":0,"snaptrim_duration":0.00085198800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":868,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084625+0000","last_change":"2024-09-16T05:16:11.722282+0000","last_active":"2024-09-16T05:16:15.084625+0000","last_peered":"2024-09-16T05:16:15.084625+0000","last_clean":"2024-09-16T05:16:15.084625+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:16:15.084625+0000","last_undegraded":"2024-09-16T05:16:15.084625+0000","last_fullsized":"2024-09-16T05:16:15.084625+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.030307+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:03.475061+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00095380400000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.25","version":"422'668","reported_seq":2160,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.924667+0000","last_change":"2024-09-16T05:16:11.719220+0000","last_active":"2024-09-16T05:16:13.924667+0000","last_peered":"2024-09-16T05:16:13.924667+0000","last_clean":"2024-09-16T05:16:13.924667+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:16:13.924667+0000","last_undegraded":"2024-09-16T05:16:13.924667+0000","last_fullsized":"2024-09-16T05:16:13.924667+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'624","last_scrub_stamp":"2024-09-16T05:15:35.625076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:35.625076+0000","objects_scrubbed":6,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:54.191762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024951600000000002,"stat_sum":{"num_bytes":13798745,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":46,"num_read_kb":57576,"num_write":38,"num_write_kb":13760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.26","version":"116'9","reported_seq":873,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724464+0000","last_change":"2024-09-16T05:16:15.743897+0000","last_active":"2024-09-16T05:16:16.724464+0000","last_peered":"2024-09-16T05:16:16.724464+0000","last_clean":"2024-09-16T05:16:16.724464+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:16:16.724464+0000","last_undegraded":"2024-09-16T05:16:16.724464+0000","last_fullsized":"2024-09-16T05:16:16.724464+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:16:13.148890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:13.148890+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:28.518352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0047087889999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":877,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724220+0000","last_change":"2024-09-16T05:16:15.743393+0000","last_active":"2024-09-16T05:16:16.724220+0000","last_peered":"2024-09-16T05:16:16.724220+0000","last_clean":"2024-09-16T05:16:16.724220+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:16:16.724220+0000","last_undegraded":"2024-09-16T05:16:16.724220+0000","last_fullsized":"2024-09-16T05:16:16.724220+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:14:30.713398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:30.713398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043007439999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1670,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.728812+0000","last_change":"2024-09-16T05:16:17.729175+0000","last_active":"2024-09-16T05:16:17.728812+0000","last_peered":"2024-09-16T05:16:17.728812+0000","last_clean":"2024-09-16T05:16:17.728812+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:16:17.728812+0000","last_undegraded":"2024-09-16T05:16:17.728812+0000","last_fullsized":"2024-09-16T05:16:17.728812+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:15:57.809122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.809122+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:06.548726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058015599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.29","version":"426'599","reported_seq":2206,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.728365+0000","last_change":"2024-09-16T05:16:15.723858+0000","last_active":"2024-09-16T05:16:16.728365+0000","last_peered":"2024-09-16T05:16:16.728365+0000","last_clean":"2024-09-16T05:16:16.728365+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:16:16.728365+0000","last_undegraded":"2024-09-16T05:16:16.728365+0000","last_fullsized":"2024-09-16T05:16:16.728365+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"380'536","last_scrub_stamp":"2024-09-16T05:15:41.155545+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:15:41.155545+0000","objects_scrubbed":6,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:16:54.090813+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0036780350000000001,"stat_sum":{"num_bytes":19331173,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":99,"num_read_kb":98114,"num_write":85,"num_write_kb":29671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1867,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.723934+0000","last_change":"2024-09-16T05:16:15.741178+0000","last_active":"2024-09-16T05:16:16.723934+0000","last_peered":"2024-09-16T05:16:16.723934+0000","last_clean":"2024-09-16T05:16:16.723934+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:16:16.723934+0000","last_undegraded":"2024-09-16T05:16:16.723934+0000","last_fullsized":"2024-09-16T05:16:16.723934+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:14:33.725061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:33.725061+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027205900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.2b","version":"428'218","reported_seq":1714,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-16T05:16:17.724132+0000","last_change":"2024-09-16T05:16:17.724429+0000","last_active":"2024-09-16T05:16:17.724132+0000","last_peered":"2024-09-16T05:16:17.724132+0000","last_clean":"2024-09-16T05:16:17.724132+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:16:17.724132+0000","last_undegraded":"2024-09-16T05:16:17.724132+0000","last_fullsized":"2024-09-16T05:16:17.724132+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"406'216","last_scrub_stamp":"2024-09-16T05:16:17.634887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.634887+0000","objects_scrubbed":2,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.906816+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.002783677,"stat_sum":{"num_bytes":3395080,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":13,"num_write_kb":8806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"83","length":"1"},{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":1688,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-16T05:16:13.924393+0000","last_change":"2024-09-16T05:16:11.717315+0000","last_active":"2024-09-16T05:16:13.924393+0000","last_peered":"2024-09-16T05:16:13.924393+0000","last_clean":"2024-09-16T05:16:13.924393+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:16:13.924393+0000","last_undegraded":"2024-09-16T05:16:13.924393+0000","last_fullsized":"2024-09-16T05:16:13.924393+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:15:39.640149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:39.640149+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:07.563472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067637399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]},{"pgid":"2.2d","version":"377'326","reported_seq":1879,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724054+0000","last_change":"2024-09-16T05:16:15.742681+0000","last_active":"2024-09-16T05:16:16.724054+0000","last_peered":"2024-09-16T05:16:16.724054+0000","last_clean":"2024-09-16T05:16:16.724054+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:16:16.724054+0000","last_undegraded":"2024-09-16T05:16:16.724054+0000","last_fullsized":"2024-09-16T05:16:16.724054+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"377'326","last_scrub_stamp":"2024-09-16T05:16:15.212317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:15.212317+0000","objects_scrubbed":1,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:37.715384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037531969999999998,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":12729,"num_write":7,"num_write_kb":4589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.2e","version":"420'410","reported_seq":1950,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-16T05:16:16.724396+0000","last_change":"2024-09-16T05:16:15.742187+0000","last_active":"2024-09-16T05:16:16.724396+0000","last_peered":"2024-09-16T05:16:16.724396+0000","last_clean":"2024-09-16T05:16:16.724396+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:16:16.724396+0000","last_undegraded":"2024-09-16T05:16:16.724396+0000","last_fullsized":"2024-09-16T05:16:16.724396+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"337'278","last_scrub_stamp":"2024-09-16T05:14:47.811642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:14:47.811642+0000","objects_scrubbed":7,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003418623,"stat_sum":{"num_bytes":9344890,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":32166,"num_write":35,"num_write_kb":10854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8f","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"1"}]},{"pgid":"2.2f","version":"422'318","reported_seq":1917,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-16T05:16:15.084647+0000","last_change":"2024-09-16T05:16:14.124457+0000","last_active":"2024-09-16T05:16:15.084647+0000","last_peered":"2024-09-16T05:16:15.084647+0000","last_clean":"2024-09-16T05:16:15.084647+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:16:15.084647+0000","last_undegraded":"2024-09-16T05:16:15.084647+0000","last_fullsized":"2024-09-16T05:16:15.084647+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"422'318","last_scrub_stamp":"2024-09-16T05:16:14.124319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.124319+0000","objects_scrubbed":4,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:33.227358+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.59682305300000005,"stat_sum":{"num_bytes":8958329,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":16,"num_read_kb":13580,"num_write":14,"num_write_kb":8818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"7c","length":"1"},{"start":"8a","length":"1"},{"start":"8e","length":"1"},{"start":"91","length":"1"},{"start":"93","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":76,"stat_sum":{"num_bytes":226252233,"num_objects":91,"num_object_clones":44,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":8,"num_read":962,"num_read_kb":1011604,"num_write":641,"num_write_kb":286880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282136576,"data_stored":323638998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16069,"ondisk_log_size":16069,"up":152,"acting":152,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089430,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74380,"kb_used_data":34864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93670772,"statfs":{"total":95995035648,"available":95918870528,"internally_reserved":0,"allocated":35700736,"data_stored":36767090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60396,"kb_used_data":25040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684756,"statfs":{"total":95995035648,"available":95933190144,"internally_reserved":0,"allocated":25640960,"data_stored":28548468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514595,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84500,"kb_used_data":40824,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93660652,"statfs":{"total":95995035648,"available":95908507648,"internally_reserved":0,"allocated":41803776,"data_stored":46715530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841635,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54920,"kb_used_data":19572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690232,"statfs":{"total":95995035648,"available":95938797568,"internally_reserved":0,"allocated":20041728,"data_stored":20532604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103273,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87764,"kb_used_data":44088,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93657388,"statfs":{"total":95995035648,"available":95905165312,"internally_reserved":0,"allocated":45146112,"data_stored":47604319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":44693962},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397617,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78844,"kb_used_data":39336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93666308,"statfs":{"total":95995035648,"available":95914299392,"internally_reserved":0,"allocated":40280064,"data_stored":45635813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757363,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61656,"kb_used_data":26304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93683496,"statfs":{"total":95995035648,"available":95931899904,"internally_reserved":0,"allocated":26935296,"data_stored":32023389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920889,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51452,"kb_used_data":16100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93693700,"statfs":{"total":95995035648,"available":95942348800,"internally_reserved":0,"allocated":16486400,"data_stored":15997109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117118,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82304,"kb_used_data":42796,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93662848,"statfs":{"total":95995035648,"available":95910756352,"internally_reserved":0,"allocated":43823104,"data_stored":45684164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240405,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44884,"kb_used_data":16760,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93700268,"statfs":{"total":95995035648,"available":95949074432,"internally_reserved":0,"allocated":17162240,"data_stored":17452614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56912,"kb_used_data":17400,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93688240,"statfs":{"total":95995035648,"available":95936757760,"internally_reserved":0,"allocated":17817600,"data_stored":17341527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738509,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54952,"kb_used_data":15444,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93690200,"statfs":{"total":95995035648,"available":95938764800,"internally_reserved":0,"allocated":15814656,"data_stored":14353253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10416128,"data_stored":10753958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12734464,"data_stored":14061451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9445376,"data_stored":9393886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38375424,"data_stored":42049251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11403264,"data_stored":12717033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21659648,"data_stored":28550526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":35172352,"data_stored":42337121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40050688,"data_stored":44315622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14983168,"data_stored":17269450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36732928,"data_stored":43443915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":20557824,"data_stored":25268392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":30605312,"data_stored":33478393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:16:20.210 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:16:20.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 86 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: pgmap v804: 97 pgs: 97 active+clean; 216 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3974922099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:16:20.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:16:20.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3974922099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:20.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:19 smithi032 ceph-mon[27388]: osdmap e430: 12 total, 12 up, 12 in 2024-09-16T05:16:20.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: pgmap v804: 97 pgs: 97 active+clean; 216 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3974922099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:16:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:16:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3974922099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:19 smithi196 ceph-mon[30802]: osdmap e430: 12 total, 12 up, 12 in 2024-09-16T05:16:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:20.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:20.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:20.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:20.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:20.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: pgmap v804: 97 pgs: 97 active+clean; 216 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3974922099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:16:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:16:20.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3974922099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:20.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:19 smithi204 ceph-mon[30933]: osdmap e430: 12 total, 12 up, 12 in 2024-09-16T05:16:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:20.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:20.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:20.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:20.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:20.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:20.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:20.720 INFO:tasks.rados.rados.2.smithi204.stdout:1364: done (2 left) 2024-09-16T05:16:20.720 INFO:tasks.rados.rados.2.smithi204.stdout:1365: read oid 21 snap -1 2024-09-16T05:16:20.720 INFO:tasks.rados.rados.2.smithi204.stdout:1365: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:20.720 INFO:tasks.rados.rados.2.smithi204.stdout:1366: copy_from oid 16 from oid 26 current snap is 151 2024-09-16T05:16:20.720 INFO:tasks.rados.rados.2.smithi204.stdout:1363: finishing rollback tid 0 to smithi20457400-18 2024-09-16T05:16:20.721 INFO:tasks.rados.rados.2.smithi204.stdout:1363: finishing rollback tid 1 to smithi20457400-18 2024-09-16T05:16:20.721 INFO:tasks.rados.rados.2.smithi204.stdout:1363: finishing rollback tid 2 to smithi20457400-18 2024-09-16T05:16:20.721 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 304 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:16:20.721 INFO:tasks.rados.rados.2.smithi204.stdout:1363: done (3 left) 2024-09-16T05:16:20.721 INFO:tasks.rados.rados.2.smithi204.stdout:1367: write oid 5 current snap is 151 2024-09-16T05:16:20.721 INFO:tasks.rados.rados.2.smithi204.stdout:1362: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:20.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:20.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:20.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:20.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:20.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:20.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:20.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:21.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:21.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:21.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:21.043 INFO:tasks.rados.rados.2.smithi204.stdout:1365: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:21.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:21.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:21.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:21.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:21.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: 2.2e scrub starts 2024-09-16T05:16:21.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: 2.2e scrub ok 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: pgmap v807: 97 pgs: 97 active+clean; 212 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:16:21.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:21.207 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:20 smithi032 ceph-mon[27388]: osdmap e431: 12 total, 12 up, 12 in 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: 2.2e scrub starts 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: 2.2e scrub ok 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: pgmap v807: 97 pgs: 97 active+clean; 212 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:16:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:16:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:16:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:20 smithi196 ceph-mon[30802]: osdmap e431: 12 total, 12 up, 12 in 2024-09-16T05:16:21.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:21.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:21.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:21.357 INFO:tasks.rados.rados.2.smithi204.stdout:1366: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:16:21.357 INFO:tasks.rados.rados.2.smithi204.stdout:1366: finishing copy_from to smithi20457400-16 2024-09-16T05:16:21.357 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 672 (ObjNum 472 snap 147 seq_num 472) dirty exists 2024-09-16T05:16:21.357 INFO:tasks.rados.rados.2.smithi204.stdout:1367: seq_num 481 ranges {484167=725008,1730427=353740} 2024-09-16T05:16:21.363 INFO:tasks.rados.rados.2.smithi204.stdout:1367: writing smithi20457400-5 from 484167 to 1209175 tid 1 2024-09-16T05:16:21.368 INFO:tasks.rados.rados.2.smithi204.stdout:1367: writing smithi20457400-5 from 1730427 to 2084167 tid 2 2024-09-16T05:16:21.368 INFO:tasks.rados.rados.2.smithi204.stdout:1362: done (3 left) 2024-09-16T05:16:21.368 INFO:tasks.rados.rados.2.smithi204.stdout:1365: done (2 left) 2024-09-16T05:16:21.369 INFO:tasks.rados.rados.2.smithi204.stdout:1366: done (1 left) 2024-09-16T05:16:21.369 INFO:tasks.rados.rados.2.smithi204.stdout:1368: read oid 15 snap -1 2024-09-16T05:16:21.369 INFO:tasks.rados.rados.2.smithi204.stdout:1368: expect (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:16:21.369 INFO:tasks.rados.rados.2.smithi204.stdout:1369: snap_remove snap 150 2024-09-16T05:16:21.370 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:21.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:21.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:21.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: 2.2e scrub starts 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: 2.2e scrub ok 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: pgmap v807: 97 pgs: 97 active+clean; 212 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:16:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:21.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:20 smithi204 ceph-mon[30933]: osdmap e431: 12 total, 12 up, 12 in 2024-09-16T05:16:21.435 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:21.435 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:21.436 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:21.494 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:21.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:21.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:21.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:21.639 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:21.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:21.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:21.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:21.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:21.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:21.886 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:21.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:21.958 INFO:tasks.rados.rados.2.smithi204.stdout:1369: done (2 left) 2024-09-16T05:16:21.958 INFO:tasks.rados.rados.2.smithi204.stdout:1370: read oid 14 snap -1 2024-09-16T05:16:21.959 INFO:tasks.rados.rados.2.smithi204.stdout:1370: expect (ObjNum 415 snap 123 seq_num 415) 2024-09-16T05:16:21.959 INFO:tasks.rados.rados.2.smithi204.stdout:1371: snap_remove snap 142 2024-09-16T05:16:21.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:22.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:22.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:22.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:22.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:22.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:21 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:16:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:21 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:16:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:21 smithi032 ceph-mon[27388]: 2.3e deep-scrub starts 2024-09-16T05:16:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:21 smithi032 ceph-mon[27388]: 2.3e deep-scrub ok 2024-09-16T05:16:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:21 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:21 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:21 smithi032 ceph-mon[27388]: 2.3b scrub starts 2024-09-16T05:16:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:21 smithi032 ceph-mon[27388]: 2.3b scrub ok 2024-09-16T05:16:22.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:22.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:21 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:16:22.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:21 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:16:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:21 smithi196 ceph-mon[30802]: 2.3e deep-scrub starts 2024-09-16T05:16:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:21 smithi196 ceph-mon[30802]: 2.3e deep-scrub ok 2024-09-16T05:16:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:21 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:21 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:21 smithi196 ceph-mon[30802]: 2.3b scrub starts 2024-09-16T05:16:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:21 smithi196 ceph-mon[30802]: 2.3b scrub ok 2024-09-16T05:16:22.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:22.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:22.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:22.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:22.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:21 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:16:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:21 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:16:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:21 smithi204 ceph-mon[30933]: 2.3e deep-scrub starts 2024-09-16T05:16:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:21 smithi204 ceph-mon[30933]: 2.3e deep-scrub ok 2024-09-16T05:16:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:21 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:21 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:21 smithi204 ceph-mon[30933]: 2.3b scrub starts 2024-09-16T05:16:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:21 smithi204 ceph-mon[30933]: 2.3b scrub ok 2024-09-16T05:16:22.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:22.503 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:22.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:22.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:16:22.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:22.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:22.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:22.719 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:22.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:22.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:22.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:22.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:22.876 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:22.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:22.967 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pg_num to 86 2024-09-16T05:16:22.976 INFO:tasks.rados.rados.2.smithi204.stdout:1371: done (3 left) 2024-09-16T05:16:22.976 INFO:tasks.rados.rados.2.smithi204.stdout:1372: write (excl) oid 27 current snap is 151 2024-09-16T05:16:22.976 INFO:tasks.rados.rados.2.smithi204.stdout:1372: seq_num 482 ranges {500974=400257,1388909=557169} 2024-09-16T05:16:22.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:22.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:16:22.981 INFO:tasks.rados.rados.2.smithi204.stdout:1372: writing smithi20457400-27 from 500974 to 901231 tid 1 2024-09-16T05:16:22.988 INFO:tasks.rados.rados.2.smithi204.stdout:1372: writing smithi20457400-27 from 1388909 to 1946078 tid 2 2024-09-16T05:16:22.988 INFO:tasks.rados.rados.2.smithi204.stdout:1373: copy_from oid 20 from oid 25 current snap is 151 2024-09-16T05:16:22.988 INFO:tasks.rados.rados.2.smithi204.stdout:1367: finishing write tid 1 to smithi20457400-5 2024-09-16T05:16:22.988 INFO:tasks.rados.rados.2.smithi204.stdout:1367: finishing write tid 2 to smithi20457400-5 2024-09-16T05:16:22.988 INFO:tasks.rados.rados.2.smithi204.stdout:1367: finishing write tid 3 to smithi20457400-5 2024-09-16T05:16:22.989 INFO:tasks.rados.rados.2.smithi204.stdout:1367: finishing write tid 4 to smithi20457400-5 2024-09-16T05:16:22.989 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 329 (ObjNum 481 snap 151 seq_num 481) dirty exists 2024-09-16T05:16:22.989 INFO:tasks.rados.rados.2.smithi204.stdout:1367: left oid 5 (ObjNum 481 snap 151 seq_num 481) 2024-09-16T05:16:22.989 INFO:tasks.rados.rados.2.smithi204.stdout:1368: expect (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:16:23.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:23.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:23.092 INFO:tasks.rados.rados.2.smithi204.stdout:1370: expect (ObjNum 415 snap 123 seq_num 415) 2024-09-16T05:16:23.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:23.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:23.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:22 smithi032 ceph-mon[27388]: 2.1f deep-scrub starts 2024-09-16T05:16:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:22 smithi032 ceph-mon[27388]: 2.1f deep-scrub ok 2024-09-16T05:16:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:22 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:16:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:22 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:16:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:22 smithi032 ceph-mon[27388]: osdmap e432: 12 total, 12 up, 12 in 2024-09-16T05:16:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1929701971' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]: dispatch 2024-09-16T05:16:23.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:23.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:22 smithi196 ceph-mon[30802]: 2.1f deep-scrub starts 2024-09-16T05:16:23.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:22 smithi196 ceph-mon[30802]: 2.1f deep-scrub ok 2024-09-16T05:16:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:22 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:16:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:22 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:16:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:22 smithi196 ceph-mon[30802]: osdmap e432: 12 total, 12 up, 12 in 2024-09-16T05:16:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1929701971' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]: dispatch 2024-09-16T05:16:23.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:23.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:23.267 INFO:tasks.rados.rados.2.smithi204.stdout:1372: finishing write tid 1 to smithi20457400-27 2024-09-16T05:16:23.267 INFO:tasks.rados.rados.2.smithi204.stdout:1372: finishing write tid 2 to smithi20457400-27 2024-09-16T05:16:23.267 INFO:tasks.rados.rados.2.smithi204.stdout:1372: finishing write tid 3 to smithi20457400-27 2024-09-16T05:16:23.267 INFO:tasks.rados.rados.2.smithi204.stdout:1372: finishing write tid 4 to smithi20457400-27 2024-09-16T05:16:23.267 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 300 (ObjNum 482 snap 151 seq_num 482) dirty exists 2024-09-16T05:16:23.267 INFO:tasks.rados.rados.2.smithi204.stdout:1372: left oid 27 (ObjNum 482 snap 151 seq_num 482) 2024-09-16T05:16:23.267 INFO:tasks.rados.rados.2.smithi204.stdout:1373: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:16:23.267 INFO:tasks.rados.rados.2.smithi204.stdout:1373: finishing copy_from to smithi20457400-20 2024-09-16T05:16:23.268 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 451 (ObjNum 432 snap 124 seq_num 432) dirty exists 2024-09-16T05:16:23.268 INFO:tasks.rados.rados.2.smithi204.stdout:1367: done (4 left) 2024-09-16T05:16:23.268 INFO:tasks.rados.rados.2.smithi204.stdout:1368: done (3 left) 2024-09-16T05:16:23.268 INFO:tasks.rados.rados.2.smithi204.stdout:1370: done (2 left) 2024-09-16T05:16:23.268 INFO:tasks.rados.rados.2.smithi204.stdout:1372: done (1 left) 2024-09-16T05:16:23.268 INFO:tasks.rados.rados.2.smithi204.stdout:1373: done (0 left) 2024-09-16T05:16:23.268 INFO:tasks.rados.rados.2.smithi204.stdout:1374: delete oid 41 current snap is 151 2024-09-16T05:16:23.269 INFO:tasks.rados.rados.2.smithi204.stdout:1374: done (0 left) 2024-09-16T05:16:23.269 INFO:tasks.rados.rados.2.smithi204.stdout:1375: read oid 13 snap -1 2024-09-16T05:16:23.269 INFO:tasks.rados.rados.2.smithi204.stdout:1375: expect (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:16:23.269 INFO:tasks.rados.rados.2.smithi204.stdout:1376: write oid 46 current snap is 151 2024-09-16T05:16:23.269 INFO:tasks.rados.rados.2.smithi204.stdout:1376: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-16T05:16:23.275 INFO:tasks.rados.rados.2.smithi204.stdout:1376: writing smithi20457400-46 from 534588 to 1085343 tid 1 2024-09-16T05:16:23.282 INFO:tasks.rados.rados.2.smithi204.stdout:1376: writing smithi20457400-46 from 1789520 to 2134588 tid 2 2024-09-16T05:16:23.282 INFO:tasks.rados.rados.2.smithi204.stdout:1377: read oid 24 snap -1 2024-09-16T05:16:23.282 INFO:tasks.rados.rados.2.smithi204.stdout:1377: expect deleted 2024-09-16T05:16:23.282 INFO:tasks.rados.rados.2.smithi204.stdout:1378: copy_from oid 36 from oid 18 current snap is 151 2024-09-16T05:16:23.282 INFO:tasks.rados.rados.2.smithi204.stdout:1375: expect (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:16:23.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:23.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:23.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:1376: finishing write tid 1 to smithi20457400-46 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:1375: done (3 left) 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:1379: delete oid 11 current snap is 151 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:1376: finishing write tid 2 to smithi20457400-46 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:1376: finishing write tid 3 to smithi20457400-46 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:1376: finishing write tid 4 to smithi20457400-46 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 338 (ObjNum 484 snap 151 seq_num 484) dirty exists 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:1376: left oid 46 (ObjNum 484 snap 151 seq_num 484) 2024-09-16T05:16:23.388 INFO:tasks.rados.rados.2.smithi204.stdout:1378: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:16:23.389 INFO:tasks.rados.rados.2.smithi204.stdout:1378: finishing copy_from to smithi20457400-36 2024-09-16T05:16:23.389 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 304 (ObjNum 216 snap 56 seq_num 216) dirty exists 2024-09-16T05:16:23.389 INFO:tasks.rados.rados.2.smithi204.stdout:1376: done (3 left) 2024-09-16T05:16:23.389 INFO:tasks.rados.rados.2.smithi204.stdout:1377: done (2 left) 2024-09-16T05:16:23.389 INFO:tasks.rados.rados.2.smithi204.stdout:1378: done (1 left) 2024-09-16T05:16:23.389 INFO:tasks.rados.rados.2.smithi204.stdout:1379: done (0 left) 2024-09-16T05:16:23.390 INFO:tasks.rados.rados.2.smithi204.stdout:1380: snap_remove snap 147 2024-09-16T05:16:23.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:22 smithi204 ceph-mon[30933]: 2.1f deep-scrub starts 2024-09-16T05:16:23.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:22 smithi204 ceph-mon[30933]: 2.1f deep-scrub ok 2024-09-16T05:16:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:22 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:16:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:22 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:16:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:22 smithi204 ceph-mon[30933]: osdmap e432: 12 total, 12 up, 12 in 2024-09-16T05:16:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1929701971' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]: dispatch 2024-09-16T05:16:23.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:23.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:23.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:23.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:23.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:23.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:23.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:23.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:23.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:23.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:23.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:23.978 INFO:tasks.rados.rados.2.smithi204.stdout:1380: done (0 left) 2024-09-16T05:16:23.978 INFO:tasks.rados.rados.2.smithi204.stdout:1381: write (excl) oid 26 current snap is 151 2024-09-16T05:16:23.978 INFO:tasks.rados.rados.2.smithi204.stdout:1381: seq_num 486 ranges {568202=701253,1906484=261718} 2024-09-16T05:16:23.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:23.985 INFO:tasks.rados.rados.2.smithi204.stdout:1381: writing smithi20457400-26 from 568202 to 1269455 tid 1 2024-09-16T05:16:23.990 INFO:tasks.rados.rados.2.smithi204.stdout:1381: writing smithi20457400-26 from 1906484 to 2168202 tid 2 2024-09-16T05:16:23.990 INFO:tasks.rados.rados.2.smithi204.stdout:1382: read oid 42 snap 148 2024-09-16T05:16:23.991 INFO:tasks.rados.rados.2.smithi204.stdout:1382: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:16:23.991 INFO:tasks.rados.rados.2.smithi204.stdout:1383: write oid 8 current snap is 151 2024-09-16T05:16:23.991 INFO:tasks.rados.rados.2.smithi204.stdout:1381: finishing write tid 1 to smithi20457400-26 2024-09-16T05:16:23.991 INFO:tasks.rados.rados.2.smithi204.stdout:1383: seq_num 487 ranges {585009=492855,1564966=620043} 2024-09-16T05:16:23.996 INFO:tasks.rados.rados.2.smithi204.stdout:1383: writing smithi20457400-8 from 585009 to 1077864 tid 1 2024-09-16T05:16:24.003 INFO:tasks.rados.rados.2.smithi204.stdout:1383: writing smithi20457400-8 from 1564966 to 2185009 tid 2 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1384: read oid 10 snap -1 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1384: expect deleted 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1385: copy_from oid 12 from oid 21 current snap is 151 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1381: finishing write tid 2 to smithi20457400-26 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1381: finishing write tid 3 to smithi20457400-26 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1381: finishing write tid 4 to smithi20457400-26 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 716 (ObjNum 486 snap 151 seq_num 486) dirty exists 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1381: left oid 26 (ObjNum 486 snap 151 seq_num 486) 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1381: done (4 left) 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1386: read oid 36 snap -1 2024-09-16T05:16:24.004 INFO:tasks.rados.rados.2.smithi204.stdout:1386: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:16:24.005 INFO:tasks.rados.rados.2.smithi204.stdout:1387: snap_create 2024-09-16T05:16:24.005 INFO:tasks.rados.rados.2.smithi204.stdout:1383: finishing write tid 1 to smithi20457400-8 2024-09-16T05:16:24.006 INFO:tasks.rados.rados.2.smithi204.stdout:1382: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:16:24.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:24.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:24.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:24.132 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:24.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:24.169 INFO:tasks.rados.rados.2.smithi204.stdout:1383: finishing write tid 2 to smithi20457400-8 2024-09-16T05:16:24.169 INFO:tasks.rados.rados.2.smithi204.stdout:1383: finishing write tid 3 to smithi20457400-8 2024-09-16T05:16:24.169 INFO:tasks.rados.rados.2.smithi204.stdout:1383: finishing write tid 4 to smithi20457400-8 2024-09-16T05:16:24.169 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 629 (ObjNum 487 snap 151 seq_num 487) dirty exists 2024-09-16T05:16:24.169 INFO:tasks.rados.rados.2.smithi204.stdout:1383: left oid 8 (ObjNum 487 snap 151 seq_num 487) 2024-09-16T05:16:24.170 INFO:tasks.rados.rados.2.smithi204.stdout:1385: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:16:24.170 INFO:tasks.rados.rados.2.smithi204.stdout:1385: finishing copy_from to smithi20457400-12 2024-09-16T05:16:24.170 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 326 (ObjNum 440 snap 131 seq_num 440) dirty exists 2024-09-16T05:16:24.170 INFO:tasks.rados.rados.2.smithi204.stdout:1386: expect (ObjNum 216 snap 56 seq_num 216) 2024-09-16T05:16:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:23 smithi032 ceph-mon[27388]: pgmap v810: 97 pgs: 97 active+clean; 195 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 458 KiB/s wr, 7 op/s 2024-09-16T05:16:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:23 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:23 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1929701971' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]': finished 2024-09-16T05:16:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:23 smithi032 ceph-mon[27388]: osdmap e433: 12 total, 12 up, 12 in 2024-09-16T05:16:24.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:23 smithi032 ceph-mon[27388]: 2.3d scrub starts 2024-09-16T05:16:24.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:23 smithi032 ceph-mon[27388]: 2.3d scrub ok 2024-09-16T05:16:24.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:24.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:23 smithi196 ceph-mon[30802]: pgmap v810: 97 pgs: 97 active+clean; 195 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 458 KiB/s wr, 7 op/s 2024-09-16T05:16:24.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:23 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:23 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1929701971' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]': finished 2024-09-16T05:16:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:23 smithi196 ceph-mon[30802]: osdmap e433: 12 total, 12 up, 12 in 2024-09-16T05:16:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:23 smithi196 ceph-mon[30802]: 2.3d scrub starts 2024-09-16T05:16:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:23 smithi196 ceph-mon[30802]: 2.3d scrub ok 2024-09-16T05:16:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:24.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:24.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:24.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:24.374 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:24.374 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:24.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:24.396 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:24.396 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:24.416 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:24.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:23 smithi204 ceph-mon[30933]: pgmap v810: 97 pgs: 97 active+clean; 195 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 458 KiB/s wr, 7 op/s 2024-09-16T05:16:24.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:23 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:24.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:23 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:24.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1929701971' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "86"}]': finished 2024-09-16T05:16:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:23 smithi204 ceph-mon[30933]: osdmap e433: 12 total, 12 up, 12 in 2024-09-16T05:16:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:23 smithi204 ceph-mon[30933]: 2.3d scrub starts 2024-09-16T05:16:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:23 smithi204 ceph-mon[30933]: 2.3d scrub ok 2024-09-16T05:16:24.437 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:24.437 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:24.459 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:24.459 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:24.481 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:24.481 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:24.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:24.503 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:24.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:24.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:24.524 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:24.525 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:24.545 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:24.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:24.567 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:24.588 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:24.588 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:24.610 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:24.610 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:24.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:24.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:24.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:24.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:24.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:24.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:24.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:24.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:24.872 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:24.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:24.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1382: done (5 left) 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1383: done (4 left) 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1384: done (3 left) 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1385: done (2 left) 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1386: done (1 left) 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1387: done (0 left) 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1388: copy_from oid 11 from oid 6 current snap is 152 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1389: read oid 5 snap -1 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1389: expect (ObjNum 481 snap 151 seq_num 481) 2024-09-16T05:16:24.986 INFO:tasks.rados.rados.2.smithi204.stdout:1390: write oid 26 current snap is 152 2024-09-16T05:16:24.987 INFO:tasks.rados.rados.2.smithi204.stdout:1390: seq_num 488 ranges {601816=568104,1623448=451823} 2024-09-16T05:16:24.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:24.993 INFO:tasks.rados.rados.2.smithi204.stdout:1390: writing smithi20457400-26 from 601816 to 1169920 tid 1 2024-09-16T05:16:24.999 INFO:tasks.rados.rados.2.smithi204.stdout:1390: writing smithi20457400-26 from 1623448 to 2075271 tid 2 2024-09-16T05:16:24.999 INFO:tasks.rados.rados.2.smithi204.stdout:1391: read oid 7 snap -1 2024-09-16T05:16:24.999 INFO:tasks.rados.rados.2.smithi204.stdout:1391: expect (ObjNum 405 snap 122 seq_num 405) 2024-09-16T05:16:25.000 INFO:tasks.rados.rados.2.smithi204.stdout:1392: write (excl) oid 18 current snap is 152 2024-09-16T05:16:25.000 INFO:tasks.rados.rados.2.smithi204.stdout:1392: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-16T05:16:25.006 INFO:tasks.rados.rados.2.smithi204.stdout:1392: writing smithi20457400-18 from 618623 to 1261976 tid 1 2024-09-16T05:16:25.012 INFO:tasks.rados.rados.2.smithi204.stdout:1392: writing smithi20457400-18 from 1681930 to 2218623 tid 2 2024-09-16T05:16:25.012 INFO:tasks.rados.rados.2.smithi204.stdout:1393: delete oid 29 current snap is 152 2024-09-16T05:16:25.012 INFO:tasks.rados.rados.2.smithi204.stdout:1389: expect (ObjNum 481 snap 151 seq_num 481) 2024-09-16T05:16:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:25.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:25.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:25.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:25.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1390: finishing write tid 1 to smithi20457400-26 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1388: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1390: finishing write tid 2 to smithi20457400-26 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1389: done (5 left) 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1393: done (4 left) 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1394: copy_from oid 41 from oid 37 current snap is 152 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1390: finishing write tid 3 to smithi20457400-26 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1390: finishing write tid 4 to smithi20457400-26 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 720 (ObjNum 488 snap 152 seq_num 488) dirty exists 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1390: left oid 26 (ObjNum 488 snap 152 seq_num 488) 2024-09-16T05:16:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:1388: finishing copy_from to smithi20457400-11 2024-09-16T05:16:25.190 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 322 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:16:25.190 INFO:tasks.rados.rados.2.smithi204.stdout:1391: expect (ObjNum 405 snap 122 seq_num 405) 2024-09-16T05:16:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:24 smithi196 ceph-mon[30802]: 2.2a deep-scrub starts 2024-09-16T05:16:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:24 smithi196 ceph-mon[30802]: 2.2a deep-scrub ok 2024-09-16T05:16:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:24 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:16:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:24 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:16:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:24 smithi196 ceph-mon[30802]: osdmap e434: 12 total, 12 up, 12 in 2024-09-16T05:16:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]: dispatch 2024-09-16T05:16:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3620424754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:25.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:25.260 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:25.261 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:25.282 INFO:tasks.rados.rados.2.smithi204.stdout:1392: finishing write tid 1 to smithi20457400-18 2024-09-16T05:16:25.282 INFO:tasks.rados.rados.2.smithi204.stdout:1392: finishing write tid 2 to smithi20457400-18 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1392: finishing write tid 3 to smithi20457400-18 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1392: finishing write tid 4 to smithi20457400-18 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 307 (ObjNum 489 snap 152 seq_num 489) dirty exists 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1392: left oid 18 (ObjNum 489 snap 152 seq_num 489) 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1394: finishing copy_from to smithi20457400-41 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1394: got expected ENOENT (src dne) 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1394: finishing copy_from racing read to smithi20457400-41 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1388: done (4 left) 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1390: done (3 left) 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1391: done (2 left) 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1392: done (1 left) 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1394: done (0 left) 2024-09-16T05:16:25.283 INFO:tasks.rados.rados.2.smithi204.stdout:1395: rollback oid 24 current snap is 152 2024-09-16T05:16:25.284 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 151 2024-09-16T05:16:25.284 INFO:tasks.rados.rados.2.smithi204.stdout:1396: write oid 13 current snap is 152 2024-09-16T05:16:25.284 INFO:tasks.rados.rados.2.smithi204.stdout:1396: seq_num 491 ranges {652237=793851,2198894=53343} 2024-09-16T05:16:25.290 INFO:tasks.rados.rados.2.smithi204.stdout:1396: writing smithi20457400-13 from 652237 to 1446088 tid 1 2024-09-16T05:16:25.294 INFO:tasks.rados.rados.2.smithi204.stdout:1396: writing smithi20457400-13 from 2198894 to 2252237 tid 2 2024-09-16T05:16:25.294 INFO:tasks.rados.rados.2.smithi204.stdout:1397: write oid 34 current snap is 152 2024-09-16T05:16:25.294 INFO:tasks.rados.rados.2.smithi204.stdout:1397: seq_num 492 ranges {669044=469100,1741023=459161} 2024-09-16T05:16:25.300 INFO:tasks.rados.rados.2.smithi204.stdout:1397: writing smithi20457400-34 from 669044 to 1138144 tid 1 2024-09-16T05:16:25.306 INFO:tasks.rados.rados.2.smithi204.stdout:1397: writing smithi20457400-34 from 1741023 to 2200184 tid 2 2024-09-16T05:16:25.306 INFO:tasks.rados.rados.2.smithi204.stdout:1398: write oid 21 current snap is 152 2024-09-16T05:16:25.306 INFO:tasks.rados.rados.2.smithi204.stdout:1398: seq_num 493 ranges {685851=544349,1799505=486346} 2024-09-16T05:16:25.312 INFO:tasks.rados.rados.2.smithi204.stdout:1398: writing smithi20457400-21 from 685851 to 1230200 tid 1 2024-09-16T05:16:25.318 INFO:tasks.rados.rados.2.smithi204.stdout:1398: writing smithi20457400-21 from 1799505 to 2285851 tid 2 2024-09-16T05:16:25.318 INFO:tasks.rados.rados.2.smithi204.stdout:1399: snap_remove snap 148 2024-09-16T05:16:25.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:25.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:24 smithi032 ceph-mon[27388]: 2.2a deep-scrub starts 2024-09-16T05:16:25.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:24 smithi032 ceph-mon[27388]: 2.2a deep-scrub ok 2024-09-16T05:16:25.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:24 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:16:25.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:24 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:16:25.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:24 smithi032 ceph-mon[27388]: osdmap e434: 12 total, 12 up, 12 in 2024-09-16T05:16:25.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]: dispatch 2024-09-16T05:16:25.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3620424754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:25.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:25.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:25.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:24 smithi204 ceph-mon[30933]: 2.2a deep-scrub starts 2024-09-16T05:16:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:24 smithi204 ceph-mon[30933]: 2.2a deep-scrub ok 2024-09-16T05:16:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:24 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:16:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:24 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:16:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:24 smithi204 ceph-mon[30933]: osdmap e434: 12 total, 12 up, 12 in 2024-09-16T05:16:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]: dispatch 2024-09-16T05:16:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3620424754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:25.436 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:25.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:25.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:25.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:25.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:25.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:25.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:25.725 INFO:tasks.rados.rados.2.smithi204.stdout:1399: done (4 left) 2024-09-16T05:16:25.725 INFO:tasks.rados.rados.2.smithi204.stdout:1400: copy_from oid 9 from oid 40 current snap is 152 2024-09-16T05:16:25.725 INFO:tasks.rados.rados.2.smithi204.stdout:1395: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:16:25.725 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 471 (ObjNum 129 snap 0 seq_num 4282091312) dirty dne 2024-09-16T05:16:25.726 INFO:tasks.rados.rados.2.smithi204.stdout:1397: finishing write tid 1 to smithi20457400-34 2024-09-16T05:16:25.726 INFO:tasks.rados.rados.2.smithi204.stdout:1397: finishing write tid 2 to smithi20457400-34 2024-09-16T05:16:25.726 INFO:tasks.rados.rados.2.smithi204.stdout:1397: finishing write tid 3 to smithi20457400-34 2024-09-16T05:16:25.726 INFO:tasks.rados.rados.2.smithi204.stdout:1397: finishing write tid 4 to smithi20457400-34 2024-09-16T05:16:25.726 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 492 (ObjNum 492 snap 152 seq_num 492) dirty exists 2024-09-16T05:16:25.726 INFO:tasks.rados.rados.2.smithi204.stdout:1397: left oid 34 (ObjNum 492 snap 152 seq_num 492) 2024-09-16T05:16:25.726 INFO:tasks.rados.rados.2.smithi204.stdout:1395: done (4 left) 2024-09-16T05:16:25.726 INFO:tasks.rados.rados.2.smithi204.stdout:1397: done (3 left) 2024-09-16T05:16:25.727 INFO:tasks.rados.rados.2.smithi204.stdout:1401: write (excl) oid 2 current snap is 152 2024-09-16T05:16:25.727 INFO:tasks.rados.rados.2.smithi204.stdout:1401: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-16T05:16:25.733 INFO:tasks.rados.rados.2.smithi204.stdout:1401: writing smithi20457400-2 from 702658 to 1322256 tid 1 2024-09-16T05:16:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1401: writing smithi20457400-2 from 1857987 to 2302658 tid 2 2024-09-16T05:16:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1402: write oid 50 current snap is 152 2024-09-16T05:16:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1396: finishing write tid 1 to smithi20457400-13 2024-09-16T05:16:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1401: finishing write tid 1 to smithi20457400-2 2024-09-16T05:16:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1402: seq_num 495 ranges {719465=411200,1632822=686643} 2024-09-16T05:16:25.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:25.746 INFO:tasks.rados.rados.2.smithi204.stdout:1402: writing smithi20457400-50 from 719465 to 1130665 tid 1 2024-09-16T05:16:25.753 INFO:tasks.rados.rados.2.smithi204.stdout:1402: writing smithi20457400-50 from 1632822 to 2319465 tid 2 2024-09-16T05:16:25.753 INFO:tasks.rados.rados.2.smithi204.stdout:1403: read oid 34 snap 151 2024-09-16T05:16:25.753 INFO:tasks.rados.rados.2.smithi204.stdout:1403: expect (ObjNum 393 snap 118 seq_num 393) 2024-09-16T05:16:25.754 INFO:tasks.rados.rados.2.smithi204.stdout:1404: delete oid 26 current snap is 152 2024-09-16T05:16:25.754 INFO:tasks.rados.rados.2.smithi204.stdout:1396: finishing write tid 2 to smithi20457400-13 2024-09-16T05:16:25.754 INFO:tasks.rados.rados.2.smithi204.stdout:1396: finishing write tid 3 to smithi20457400-13 2024-09-16T05:16:25.754 INFO:tasks.rados.rados.2.smithi204.stdout:1396: finishing write tid 4 to smithi20457400-13 2024-09-16T05:16:25.754 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 511 (ObjNum 491 snap 152 seq_num 491) dirty exists 2024-09-16T05:16:25.754 INFO:tasks.rados.rados.2.smithi204.stdout:1396: left oid 13 (ObjNum 491 snap 152 seq_num 491) 2024-09-16T05:16:25.754 INFO:tasks.rados.rados.2.smithi204.stdout:1398: finishing write tid 1 to smithi20457400-21 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:1398: finishing write tid 2 to smithi20457400-21 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:1398: finishing write tid 3 to smithi20457400-21 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:1398: finishing write tid 4 to smithi20457400-21 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 329 (ObjNum 493 snap 152 seq_num 493) dirty exists 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:1398: left oid 21 (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:1401: finishing write tid 2 to smithi20457400-2 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:1401: finishing write tid 3 to smithi20457400-2 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:1401: finishing write tid 4 to smithi20457400-2 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 331 (ObjNum 494 snap 152 seq_num 494) dirty exists 2024-09-16T05:16:25.755 INFO:tasks.rados.rados.2.smithi204.stdout:1401: left oid 2 (ObjNum 494 snap 152 seq_num 494) 2024-09-16T05:16:25.756 INFO:tasks.rados.rados.2.smithi204.stdout:1400: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:16:25.756 INFO:tasks.rados.rados.2.smithi204.stdout:1400: finishing copy_from to smithi20457400-9 2024-09-16T05:16:25.756 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 351 (ObjNum 480 snap 151 seq_num 480) dirty exists 2024-09-16T05:16:25.756 INFO:tasks.rados.rados.2.smithi204.stdout:1402: finishing write tid 1 to smithi20457400-50 2024-09-16T05:16:25.757 INFO:tasks.rados.rados.2.smithi204.stdout:1396: done (6 left) 2024-09-16T05:16:25.757 INFO:tasks.rados.rados.2.smithi204.stdout:1398: done (5 left) 2024-09-16T05:16:25.757 INFO:tasks.rados.rados.2.smithi204.stdout:1400: done (4 left) 2024-09-16T05:16:25.757 INFO:tasks.rados.rados.2.smithi204.stdout:1401: done (3 left) 2024-09-16T05:16:25.757 INFO:tasks.rados.rados.2.smithi204.stdout:1404: done (2 left) 2024-09-16T05:16:25.757 INFO:tasks.rados.rados.2.smithi204.stdout:1405: copy_from oid 17 from oid 30 current snap is 152 2024-09-16T05:16:25.757 INFO:tasks.rados.rados.2.smithi204.stdout:1406: write (excl) oid 37 current snap is 152 2024-09-16T05:16:25.757 INFO:tasks.rados.rados.2.smithi204.stdout:1406: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-16T05:16:25.762 INFO:tasks.rados.rados.2.smithi204.stdout:1406: writing smithi20457400-37 from 753079 to 1314777 tid 1 2024-09-16T05:16:25.766 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:25.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:25.768 INFO:tasks.rados.rados.2.smithi204.stdout:1406: writing smithi20457400-37 from 2033433 to 2353079 tid 2 2024-09-16T05:16:25.769 INFO:tasks.rados.rados.2.smithi204.stdout:1407: read oid 39 snap -1 2024-09-16T05:16:25.769 INFO:tasks.rados.rados.2.smithi204.stdout:1407: expect deleted 2024-09-16T05:16:25.769 INFO:tasks.rados.rados.2.smithi204.stdout:1408: delete oid 25 current snap is 152 2024-09-16T05:16:25.769 INFO:tasks.rados.rados.2.smithi204.stdout:1403: expect (ObjNum 393 snap 118 seq_num 393) 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1402: finishing write tid 2 to smithi20457400-50 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1402: finishing write tid 3 to smithi20457400-50 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1403: done (5 left) 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1408: done (4 left) 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1409: read oid 3 snap -1 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1409: expect deleted 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1410: delete oid 23 current snap is 152 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1402: finishing write tid 4 to smithi20457400-50 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 347 (ObjNum 495 snap 152 seq_num 495) dirty exists 2024-09-16T05:16:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:1402: left oid 50 (ObjNum 495 snap 152 seq_num 495) 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:1406: finishing write tid 1 to smithi20457400-37 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:1405: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:1405: finishing copy_from to smithi20457400-17 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 684 (ObjNum 414 snap 123 seq_num 414) dirty exists 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:1406: finishing write tid 2 to smithi20457400-37 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:1406: finishing write tid 3 to smithi20457400-37 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:1406: finishing write tid 4 to smithi20457400-37 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 553 (ObjNum 497 snap 152 seq_num 497) dirty exists 2024-09-16T05:16:25.854 INFO:tasks.rados.rados.2.smithi204.stdout:1406: left oid 37 (ObjNum 497 snap 152 seq_num 497) 2024-09-16T05:16:25.855 INFO:tasks.rados.rados.2.smithi204.stdout:1402: done (5 left) 2024-09-16T05:16:25.855 INFO:tasks.rados.rados.2.smithi204.stdout:1405: done (4 left) 2024-09-16T05:16:25.855 INFO:tasks.rados.rados.2.smithi204.stdout:1406: done (3 left) 2024-09-16T05:16:25.855 INFO:tasks.rados.rados.2.smithi204.stdout:1407: done (2 left) 2024-09-16T05:16:25.855 INFO:tasks.rados.rados.2.smithi204.stdout:1409: done (1 left) 2024-09-16T05:16:25.855 INFO:tasks.rados.rados.2.smithi204.stdout:1410: done (0 left) 2024-09-16T05:16:25.855 INFO:tasks.rados.rados.2.smithi204.stdout:1411: write oid 18 current snap is 152 2024-09-16T05:16:25.855 INFO:tasks.rados.rados.2.smithi204.stdout:1411: seq_num 500 ranges {403500=787445,1692526=473837} 2024-09-16T05:16:25.860 INFO:tasks.rados.rados.2.smithi204.stdout:1411: writing smithi20457400-18 from 403500 to 1190945 tid 1 2024-09-16T05:16:25.865 INFO:tasks.rados.rados.2.smithi204.stdout:1411: writing smithi20457400-18 from 1692526 to 2166363 tid 2 2024-09-16T05:16:25.865 INFO:tasks.rados.rados.2.smithi204.stdout:1412: read oid 15 snap -1 2024-09-16T05:16:25.866 INFO:tasks.rados.rados.2.smithi204.stdout:1412: expect (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:16:25.866 INFO:tasks.rados.rados.2.smithi204.stdout:1413: write oid 50 current snap is 152 2024-09-16T05:16:25.866 INFO:tasks.rados.rados.2.smithi204.stdout:1411: finishing write tid 1 to smithi20457400-18 2024-09-16T05:16:25.866 INFO:tasks.rados.rados.2.smithi204.stdout:1413: seq_num 501 ranges {420307=462694,1351008=617495} 2024-09-16T05:16:25.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:25.870 INFO:tasks.rados.rados.2.smithi204.stdout:1413: writing smithi20457400-50 from 420307 to 883001 tid 1 2024-09-16T05:16:25.876 INFO:tasks.rados.rados.2.smithi204.stdout:1413: writing smithi20457400-50 from 1351008 to 1968503 tid 2 2024-09-16T05:16:25.876 INFO:tasks.rados.rados.2.smithi204.stdout:1414: snap_create 2024-09-16T05:16:25.876 INFO:tasks.rados.rados.2.smithi204.stdout:1412: expect (ObjNum 423 snap 124 seq_num 423) 2024-09-16T05:16:25.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:25.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1411: finishing write tid 2 to smithi20457400-18 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1413: finishing write tid 1 to smithi20457400-50 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1411: finishing write tid 3 to smithi20457400-18 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1411: finishing write tid 4 to smithi20457400-18 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 310 (ObjNum 500 snap 152 seq_num 500) dirty exists 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1411: left oid 18 (ObjNum 500 snap 152 seq_num 500) 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1413: finishing write tid 2 to smithi20457400-50 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1413: finishing write tid 3 to smithi20457400-50 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1413: finishing write tid 4 to smithi20457400-50 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 350 (ObjNum 501 snap 152 seq_num 501) dirty exists 2024-09-16T05:16:25.980 INFO:tasks.rados.rados.2.smithi204.stdout:1413: left oid 50 (ObjNum 501 snap 152 seq_num 501) 2024-09-16T05:16:25.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:26.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:26.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:26.057 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:454 2.c e127/107 2:33100188:::smithi20457400-19:head v 127'19, mlcod=126'16)", 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 327.355715508, 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054909060000000003, 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:16:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7800000000000001e-07 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703134+0000", 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002986954 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703165+0000", 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1201000000000002e-05 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703173+0000", 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.32e-06 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703255+0000", 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1730000000000005e-05 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703282+0000", 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7389000000000002e-05 2024-09-16T05:16:26.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.705600+0000", 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023176360000000001 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.705637+0000", 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7497999999999999e-05 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.062 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:116 2.5 e141/107 2:ad4a88eb:::smithi20457400-16:a v 141'87, mlcod=141'79)", 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 313.36085397900001, 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006093261, 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695009+0000", 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6499999999999999e-07 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695014+0000", 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8350000000000003e-06 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700654+0000", 2024-09-16T05:16:26.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056407499999999999 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700660+0000", 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4879999999999998e-06 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700700+0000", 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0583000000000001e-05 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700714+0000", 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.322e-05 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701062+0000", 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00034833999999999999 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.067 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701101+0000", 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9480000000000001e-05 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:118 2.5 e141/107 2:ad7afe2b:::smithi20457400-38:d v 141'90, mlcod=141'85)", 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 313.35425965500002, 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066456359999999999, 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701603+0000", 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701606+0000", 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3890000000000001e-06 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707421+0000", 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005814601 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707428+0000", 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4839999999999998e-06 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707497+0000", 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8441000000000004e-05 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707520+0000", 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3513999999999999e-05 2024-09-16T05:16:26.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708185+0000", 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066482599999999998 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708248+0000", 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2959000000000005e-05 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.072 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:614 2.5 e144/107 2:a1d8c6d9:::smithi20457400-3:head v 144'110, mlcod=144'106)", 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 310.40529389, 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054521470000000001, 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.074 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.074 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:16:26.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650569+0000", 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7299999999999995e-07 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653581+0000", 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003012617 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653627+0000", 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5649999999999998e-05 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653639+0000", 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2126e-05 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653715+0000", 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5485e-05 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653748+0000", 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3373e-05 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.655964+0000", 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002215875 2024-09-16T05:16:26.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.656020+0000", 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6348000000000003e-05 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:172 2.c e179/107 2:3d4bedc8:::smithi20457400-22:26 v 179'101, mlcod=178'97)", 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 275.40313649199999, 2024-09-16T05:16:26.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34174038899999998, 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:16:26.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4999999999999999e-07 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652730+0000", 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8789999999999996e-06 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993974+0000", 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34124383699999999 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993980+0000", 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4139999999999999e-06 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994036+0000", 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5850999999999998e-05 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994054+0000", 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8182999999999999e-05 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994414+0000", 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036047800000000001 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994466+0000", 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1496999999999998e-05 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.083 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:972 2.c e179/107 2:33100188:::smithi20457400-19:head v 179'105, mlcod=179'101)", 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 275.05679259099998, 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.53639116399999998, 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999070+0000", 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0999999999999997e-07 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999087+0000", 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7014999999999999e-05 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534916+0000", 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.53582909000000001 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534923+0000", 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0099999999999998e-06 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534975+0000", 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1938999999999999e-05 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534994+0000", 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8504000000000001e-05 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.535427+0000", 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043298700000000002 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.535461+0000", 2024-09-16T05:16:26.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3809000000000001e-05 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:979 2.4 e179/107 2:2ace617f:::smithi20457400-21:head v 179'81, mlcod=178'78)", 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 275.04899223500001, 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076005070000000003, 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.089 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009810+0000", 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002939929 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009892+0000", 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2349999999999996e-05 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009916+0000", 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3200000000000001e-05 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009983+0000", 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7577999999999996e-05 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010005+0000", 2024-09-16T05:16:26.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2337999999999999e-05 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014425+0000", 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044196119999999998 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014470+0000", 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5297999999999999e-05 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1017 2.5 e182/107 2:a1d8c6d9:::smithi20457400-3:head v 182'254, mlcod=182'250)", 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 272.35713758700001, 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067341900000000001, 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:16:26.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:16:26.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698726+0000", 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0410000000000001e-06 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703134+0000", 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044088850000000004 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703168+0000", 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3195999999999999e-05 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703175+0000", 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703234+0000", 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8859e-05 2024-09-16T05:16:26.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703262+0000", 2024-09-16T05:16:26.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7650000000000001e-05 2024-09-16T05:16:26.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.705387+0000", 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021252670000000001 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.705459+0000", 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1598000000000001e-05 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1146 2.c e192/107 2:33100188:::smithi20457400-19:head v 192'151, mlcod=192'147)", 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 262.31290275100002, 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057288909999999998, 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742960+0000", 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3399999999999999e-07 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746015+0000", 2024-09-16T05:16:26.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030547809999999999 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746040+0000", 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5075999999999998e-05 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746048+0000", 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9030000000000004e-06 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746099+0000", 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1004999999999997e-05 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746117+0000", 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8189000000000002e-05 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.748630+0000", 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025129549999999999 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.748688+0000", 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8448e-05 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1161 2.c e192/107 2:3d4bedc8:::smithi20457400-22:head v 192'153, mlcod=192'149)", 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 262.28441720299998, 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055212639999999997, 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.106 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:16:26.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.108 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.108 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771447+0000", 2024-09-16T05:16:26.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3360000000000002e-06 2024-09-16T05:16:26.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.108 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774126+0000", 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026786790000000002 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774176+0000", 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9669999999999997e-05 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774185+0000", 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4530000000000008e-06 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774260+0000", 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5395999999999994e-05 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774291+0000", 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0561000000000001e-05 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.776739+0000", 2024-09-16T05:16:26.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002447968 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.776966+0000", 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022720100000000001 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1201 2.5 e194/183 2:a5bc1608:::smithi20457400-11:head v 194'298, mlcod=192'295)", 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 260.376919235, 2024-09-16T05:16:26.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080910559999999992, 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683786+0000", 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048424469999999997 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683833+0000", 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7540999999999998e-05 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683844+0000", 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1354e-05 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683920+0000", 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5760999999999995e-05 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683940+0000", 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9715999999999999e-05 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.686996+0000", 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003055954 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.687034+0000", 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8031999999999998e-05 2024-09-16T05:16:26.115 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.116 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.116 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.117 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1890 2.1f 2:fc733d2f:::smithi20457400-45:head [append 0~0] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:16:26.117 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:16:26.117 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 196.34517428000001, 2024-09-16T05:16:26.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080315490000000007, 2024-09-16T05:16:26.117 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1890 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710691+0000", 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7769999999999998e-06 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710704+0000", 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.38e-05 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710741+0000", 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6371999999999997e-05 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710758+0000", 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7637000000000001e-05 2024-09-16T05:16:26.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710929+0000", 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017019499999999999 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710935+0000", 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2720000000000003e-06 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716763+0000", 2024-09-16T05:16:26.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058286350000000004 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716836+0000", 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2630000000000001e-05 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716981+0000", 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014495099999999999 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:26.122 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718424+0000", 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014427769999999999 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718586+0000", 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016167699999999999 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718660+0000", 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4813000000000004e-05 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718719+0000", 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9012999999999999e-05 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1891 2.1f 2:fc733d2f:::smithi20457400-45:head [rollback 39] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 196.34506233499999, 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082173000000000003, 2024-09-16T05:16:26.124 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1891 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710801+0000", 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6700000000000002e-07 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710809+0000", 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4230000000000005e-06 2024-09-16T05:16:26.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710823+0000", 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3833e-05 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710832+0000", 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9830000000000002e-06 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710997+0000", 2024-09-16T05:16:26.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000165558 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711001+0000", 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0239999999999999e-06 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717531+0000", 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065290699999999997 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717588+0000", 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7497999999999997e-05 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717722+0000", 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134182 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718427+0000", 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070501199999999998 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718936+0000", 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050843700000000004 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718972+0000", 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6102000000000001e-05 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719017+0000", 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5311e-05 2024-09-16T05:16:26.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1892 2.1f 2:fc733d2f:::smithi20457400-45:head [append 0~0] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 196.34493594899999, 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082344800000000006, 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:26.131 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1892 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710927+0000", 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9099999999999999e-07 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710930+0000", 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9869999999999999e-06 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710937+0000", 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9940000000000003e-06 2024-09-16T05:16:26.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710941+0000", 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0099999999999997e-06 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711036+0000", 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5013999999999995e-05 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711042+0000", 2024-09-16T05:16:26.134 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.224e-06 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718122+0000", 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007080706 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718183+0000", 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0194e-05 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:26.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718285+0000", 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102118 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718538+0000", 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025345499999999998 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719083+0000", 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054496199999999996 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719116+0000", 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3052000000000001e-05 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719161+0000", 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4372999999999997e-05 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2010 2.1f 2:fc733d2f:::smithi20457400-45:head [copy-from ver 319] snapc 54=[54,53,52,4d,4c,49,48,47,45,3e,3a,35,27] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:16:26.137 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:16:26.138 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 186.171672266, 2024-09-16T05:16:26.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016119959, 2024-09-16T05:16:26.138 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:26.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.139 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.139 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2010 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884192+0000", 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7069999999999999e-06 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884200+0000", 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0679999999999997e-06 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884217+0000", 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.772e-05 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884228+0000", 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0455000000000001e-05 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884292+0000", 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4424999999999994e-05 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884357+0000", 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4498999999999994e-05 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.893200+0000", 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088430690000000003 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.896560+0000", 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033602580000000001 2024-09-16T05:16:26.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900238+0000", 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036775699999999998 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900276+0000", 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8906999999999999e-05 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900310+0000", 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3281000000000002e-05 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2011 2.1f 2:fc733d2f:::smithi20457400-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 186.17161018799999, 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011802536000000001, 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:16:26.145 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2011 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:16:26.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884253+0000", 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2200000000000004e-07 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884265+0000", 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2048999999999999e-05 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884275+0000", 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9899999999999992e-06 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884281+0000", 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6850000000000004e-06 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884503+0000", 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022159799999999999 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884535+0000", 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2288e-05 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.895921+0000", 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011386165 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.895982+0000", 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0906000000000003e-05 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.896054+0000", 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2232999999999998e-05 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(2.31 277/276 [PushOp(2:8c1ba486:::smithi20457400-2:45, version: 259'110, data_included: [480733~537415,1782380~511161], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:8c1ba486:::smithi20457400-2:45@259'110, size: 3680733, copy_subset: [(0, 3680733)], clone_subset: {2:8c1ba486:::smithi20457400-2:2c: []}, snapset: 4c=[]:{2c: [27], 45: [45, 3e], 4c: [4c, 49, 48]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2293541, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 177.35093454400001, 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45568901499999998, 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704929+0000", 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1000000000000001e-06 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.705928+0000", 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0009990349999999999 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.159934+0000", 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45400632299999999 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.159949+0000", 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4608e-05 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160041+0000", 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2523999999999997e-05 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160053+0000", 2024-09-16T05:16:26.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1525999999999999e-05 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160617+0000", 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056389899999999998 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2536 2.31 2:8c1ba486:::smithi20457400-2:head [write 551362~726078] snapc 6d=[6d,6c,6b,6a,69,67,66,64,63,62,60,5d,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 103.11467778700001, 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055342209999999998, 2024-09-16T05:16:26.154 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2536 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941188+0000", 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3469999999999999e-06 2024-09-16T05:16:26.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942118+0000", 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092990300000000004 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942150+0000", 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2888000000000001e-05 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942164+0000", 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3920000000000001e-05 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942258+0000", 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3839999999999996e-05 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942266+0000", 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4139999999999997e-06 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.943927+0000", 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016610450000000001 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.944033+0000", 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105979 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.944207+0000", 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000174105 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.945786+0000", 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001579352 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946595+0000", 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080861300000000002 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946660+0000", 2024-09-16T05:16:26.160 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4906000000000005e-05 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946719+0000", 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8909000000000001e-05 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:569 2.c e357/247 2:33100188:::smithi20457400-19:6a v 357'229, mlcod=351'225)", 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 79.350248464000003, 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14687246600000001, 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9599999999999999e-07 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705621+0000", 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7150000000000001e-06 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851933+0000", 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14631169599999999 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851940+0000", 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.221e-06 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851995+0000", 2024-09-16T05:16:26.164 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5473999999999998e-05 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852016+0000", 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0965999999999999e-05 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852452+0000", 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043602700000000002 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852486+0000", 2024-09-16T05:16:26.165 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3871000000000001e-05 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3546 2.31 2:8c1ba486:::smithi20457400-2:head [write 702658~619598,stat] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:25.733858+0000", 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.32200360300000003, 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052592689999999996, 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:16:26.166 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3546 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.733858+0000", 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.733858+0000", 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:26.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.733860+0000", 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7749999999999999e-06 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.734654+0000", 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079366099999999998 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:26.168 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.734667+0000", 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3409e-05 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.734676+0000", 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6959999999999994e-06 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.734746+0000", 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9774e-05 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.169 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.734893+0000", 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014719700000000001 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.735137+0000", 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024374999999999999 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.736595+0000", 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014584470000000001 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.738984+0000", 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023888199999999998 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.739061+0000", 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6792000000000006e-05 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.739118+0000", 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6947999999999998e-05 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.171 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:26.172 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:26.172 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:16:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:26.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:26.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: pgmap v813: 97 pgs: 97 active+clean; 186 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]': finished 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: osdmap e435: 12 total, 12 up, 12 in 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:16:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:25 smithi196 ceph-mon[30802]: osdmap e436: 12 total, 12 up, 12 in 2024-09-16T05:16:26.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:26.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:26.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:26.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:26.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:26.418 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:26.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:16:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:16:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: pgmap v813: 97 pgs: 97 active+clean; 186 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-16T05:16:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:16:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:16:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]': finished 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: osdmap e435: 12 total, 12 up, 12 in 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:25 smithi032 ceph-mon[27388]: osdmap e436: 12 total, 12 up, 12 in 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: pgmap v813: 97 pgs: 97 active+clean; 186 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:16:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "86"}]': finished 2024-09-16T05:16:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: osdmap e435: 12 total, 12 up, 12 in 2024-09-16T05:16:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:16:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:16:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:25 smithi204 ceph-mon[30933]: osdmap e436: 12 total, 12 up, 12 in 2024-09-16T05:16:26.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:26.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:26.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:26.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:26.671 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:26.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:16:26.724 INFO:tasks.rados.rados.2.smithi204.stdout:1411: done (3 left) 2024-09-16T05:16:26.725 INFO:tasks.rados.rados.2.smithi204.stdout:1412: done (2 left) 2024-09-16T05:16:26.725 INFO:tasks.rados.rados.2.smithi204.stdout:1413: done (1 left) 2024-09-16T05:16:26.725 INFO:tasks.rados.rados.2.smithi204.stdout:1414: done (0 left) 2024-09-16T05:16:26.725 INFO:tasks.rados.rados.2.smithi204.stdout:1415: delete oid 44 current snap is 153 2024-09-16T05:16:26.728 INFO:tasks.rados.rados.2.smithi204.stdout:1415: done (0 left) 2024-09-16T05:16:26.728 INFO:tasks.rados.rados.2.smithi204.stdout:1416: copy_from oid 18 from oid 10 current snap is 153 2024-09-16T05:16:26.728 INFO:tasks.rados.rados.2.smithi204.stdout:1417: delete oid 28 current snap is 153 2024-09-16T05:16:26.729 INFO:tasks.rados.rados.2.smithi204.stdout:1416: finishing copy_from to smithi20457400-18 2024-09-16T05:16:26.729 INFO:tasks.rados.rados.2.smithi204.stdout:1416: got expected ENOENT (src dne) 2024-09-16T05:16:26.730 INFO:tasks.rados.rados.2.smithi204.stdout:1416: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:16:26.730 INFO:tasks.rados.rados.2.smithi204.stdout:1416: done (1 left) 2024-09-16T05:16:26.731 INFO:tasks.rados.rados.2.smithi204.stdout:1417: done (0 left) 2024-09-16T05:16:26.731 INFO:tasks.rados.rados.2.smithi204.stdout:1418: delete oid 35 current snap is 153 2024-09-16T05:16:26.732 INFO:tasks.rados.rados.2.smithi204.stdout:1418: done (0 left) 2024-09-16T05:16:26.732 INFO:tasks.rados.rados.2.smithi204.stdout:1419: write (excl) oid 18 current snap is 153 2024-09-16T05:16:26.732 INFO:tasks.rados.rados.2.smithi204.stdout:1419: seq_num 505 ranges {487535=480043,1584936=624833} 2024-09-16T05:16:26.737 INFO:tasks.rados.rados.2.smithi204.stdout:1419: writing smithi20457400-18 from 487535 to 967578 tid 1 2024-09-16T05:16:26.744 INFO:tasks.rados.rados.2.smithi204.stdout:1419: writing smithi20457400-18 from 1584936 to 2209769 tid 2 2024-09-16T05:16:26.744 INFO:tasks.rados.rados.2.smithi204.stdout:1420: delete oid 47 current snap is 153 2024-09-16T05:16:26.744 INFO:tasks.rados.rados.2.smithi204.stdout:1419: finishing write tid 1 to smithi20457400-18 2024-09-16T05:16:26.745 INFO:tasks.rados.rados.2.smithi204.stdout:1420: done (1 left) 2024-09-16T05:16:26.746 INFO:tasks.rados.rados.2.smithi204.stdout:1421: rollback oid 1 current snap is 153 2024-09-16T05:16:26.746 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 1 to 152 2024-09-16T05:16:26.746 INFO:tasks.rados.rados.2.smithi204.stdout:1422: copy_from oid 19 from oid 48 current snap is 153 2024-09-16T05:16:26.746 INFO:tasks.rados.rados.2.smithi204.stdout:1423: snap_remove snap 151 2024-09-16T05:16:26.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:26.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:26.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:26.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:26.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:26.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:27.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:27.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:27.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:16:27.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.4e scrub starts 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.4e scrub ok 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]: dispatch 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/946437118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:27.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:16:27.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:16:27.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:27.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]': finished 2024-09-16T05:16:27.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/946437118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:27.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:26 smithi032 ceph-mon[27388]: osdmap e437: 12 total, 12 up, 12 in 2024-09-16T05:16:27.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:16:27.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:27.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:27.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:27.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:27.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:27.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:16:27.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.4e scrub starts 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.4e scrub ok 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]: dispatch 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/946437118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:16:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:16:27.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:27.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]': finished 2024-09-16T05:16:27.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/946437118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:27.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:26 smithi204 ceph-mon[30933]: osdmap e437: 12 total, 12 up, 12 in 2024-09-16T05:16:27.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:27.465 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:27.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.4e scrub starts 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.4e scrub ok 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:16:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]: dispatch 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/946437118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "86"}]': finished 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/946437118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:27.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:26 smithi196 ceph-mon[30802]: osdmap e437: 12 total, 12 up, 12 in 2024-09-16T05:16:27.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:27.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:27.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:27.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:27.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:27.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:27.726 INFO:tasks.rados.rados.2.smithi204.stdout:1423: done (3 left) 2024-09-16T05:16:27.726 INFO:tasks.rados.rados.2.smithi204.stdout:1424: rollback oid 6 current snap is 153 2024-09-16T05:16:27.726 INFO:tasks.rados.rados.2.smithi204.stdout:1419: finishing write tid 2 to smithi20457400-18 2024-09-16T05:16:27.726 INFO:tasks.rados.rados.2.smithi204.stdout:1419: finishing write tid 3 to smithi20457400-18 2024-09-16T05:16:27.726 INFO:tasks.rados.rados.2.smithi204.stdout:1419: finishing write tid 4 to smithi20457400-18 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 313 (ObjNum 505 snap 153 seq_num 505) dirty exists 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1419: left oid 18 (ObjNum 505 snap 153 seq_num 505) 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1421: finishing rollback tid 0 to smithi20457400-1 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1421: finishing rollback tid 1 to smithi20457400-1 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1421: finishing rollback tid 2 to smithi20457400-1 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 500 (ObjNum 456 snap 142 seq_num 456) dirty exists 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 6 to 152 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1419: done (3 left) 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1421: done (2 left) 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1425: read oid 43 snap -1 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1425: expect (ObjNum 383 snap 115 seq_num 383) 2024-09-16T05:16:27.727 INFO:tasks.rados.rados.2.smithi204.stdout:1426: snap_create 2024-09-16T05:16:27.734 INFO:tasks.rados.rados.2.smithi204.stdout:1424: finishing rollback tid 0 to smithi20457400-6 2024-09-16T05:16:27.734 INFO:tasks.rados.rados.2.smithi204.stdout:1424: finishing rollback tid 1 to smithi20457400-6 2024-09-16T05:16:27.735 INFO:tasks.rados.rados.2.smithi204.stdout:1424: finishing rollback tid 2 to smithi20457400-6 2024-09-16T05:16:27.735 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 335 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:16:27.738 INFO:tasks.rados.rados.2.smithi204.stdout:1425: expect (ObjNum 383 snap 115 seq_num 383) 2024-09-16T05:16:27.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:27.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:27.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:27.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:27.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:27.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:28.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:28.085 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:28.098 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:28.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:28.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:28.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:28.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:28.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:28.338 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:16:28.338 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:16:28.338 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: pgmap v816: 107 pgs: 4 peering, 9 unknown, 94 active+clean; 155 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5.0 MiB/s wr, 27 op/s 2024-09-16T05:16:28.338 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: 2.3c scrub starts 2024-09-16T05:16:28.338 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: 2.3c scrub ok 2024-09-16T05:16:28.338 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.338 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.339 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:28.339 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:28.339 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.339 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:27 smithi032 ceph-mon[27388]: osdmap e438: 12 total, 12 up, 12 in 2024-09-16T05:16:28.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:28.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: pgmap v816: 107 pgs: 4 peering, 9 unknown, 94 active+clean; 155 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5.0 MiB/s wr, 27 op/s 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: 2.3c scrub starts 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: 2.3c scrub ok 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:28.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:27 smithi204 ceph-mon[30933]: osdmap e438: 12 total, 12 up, 12 in 2024-09-16T05:16:28.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:28.476 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:28.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:16:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:16:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: pgmap v816: 107 pgs: 4 peering, 9 unknown, 94 active+clean; 155 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5.0 MiB/s wr, 27 op/s 2024-09-16T05:16:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: 2.3c scrub starts 2024-09-16T05:16:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: 2.3c scrub ok 2024-09-16T05:16:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:28.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:28.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:27 smithi196 ceph-mon[30802]: osdmap e438: 12 total, 12 up, 12 in 2024-09-16T05:16:28.573 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:28.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:28.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:28.579 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:28.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:28.602 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:28.603 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:28.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:28.623 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:28.643 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:28.644 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:28.667 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:28.688 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:28.688 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:28.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:28.710 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:28.710 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:28.714 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:16:28.714 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:16:28.714 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-16T05:16:28.714 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:16:28.715 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:16:28.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-16T05:16:28.726 INFO:tasks.rados.rados.2.smithi204.stdout:1424: done (3 left) 2024-09-16T05:16:28.726 INFO:tasks.rados.rados.2.smithi204.stdout:1425: done (2 left) 2024-09-16T05:16:28.726 INFO:tasks.rados.rados.2.smithi204.stdout:1426: done (1 left) 2024-09-16T05:16:28.726 INFO:tasks.rados.rados.2.smithi204.stdout:1427: delete oid 23 current snap is 154 2024-09-16T05:16:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:28.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:28.731 INFO:tasks.rados.rados.2.smithi204.stdout:1427: done (1 left) 2024-09-16T05:16:28.731 INFO:tasks.rados.rados.2.smithi204.stdout:1428: read oid 37 snap -1 2024-09-16T05:16:28.731 INFO:tasks.rados.rados.2.smithi204.stdout:1428: expect (ObjNum 497 snap 152 seq_num 497) 2024-09-16T05:16:28.731 INFO:tasks.rados.rados.2.smithi204.stdout:1429: snap_remove snap 153 2024-09-16T05:16:28.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:28.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:28.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:28.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:28.974 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:28.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:28.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:29.066 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:29.066 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:29.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:29.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:29.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:29.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:29.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:29.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:29.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:29.335 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2219408127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:29.335 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:28 smithi196 ceph-mon[30802]: osdmap e439: 12 total, 12 up, 12 in 2024-09-16T05:16:29.335 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:28 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:29.335 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:28 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:29.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2219408127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:29.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:28 smithi032 ceph-mon[27388]: osdmap e439: 12 total, 12 up, 12 in 2024-09-16T05:16:29.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:28 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:29.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:28 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:29.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:29.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:29.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:29.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2219408127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:28 smithi204 ceph-mon[30933]: osdmap e439: 12 total, 12 up, 12 in 2024-09-16T05:16:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:28 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:28 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:29.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:29.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:29.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:29.520 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:29.520 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "" 2024-09-16T05:16:29.520 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:29.535 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:29.536 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:29.558 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:29.558 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:29.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:29.582 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:29.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:29.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:29.605 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:29.605 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:29.626 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:29.627 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:29.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:29.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:16:29.732 INFO:tasks.rados.rados.2.smithi204.stdout:1429: done (2 left) 2024-09-16T05:16:29.732 INFO:tasks.rados.rados.2.smithi204.stdout:1430: delete oid 16 current snap is 154 2024-09-16T05:16:29.732 INFO:tasks.rados.rados.2.smithi204.stdout:1428: expect (ObjNum 497 snap 152 seq_num 497) 2024-09-16T05:16:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:29.733 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:29.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:29.814 INFO:tasks.rados.rados.2.smithi204.stdout:1422: finishing copy_from to smithi20457400-19 2024-09-16T05:16:29.814 INFO:tasks.rados.rados.2.smithi204.stdout:1422: got expected ENOENT (src dne) 2024-09-16T05:16:29.814 INFO:tasks.rados.rados.2.smithi204.stdout:1422: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:16:29.814 INFO:tasks.rados.rados.2.smithi204.stdout:1422: done (2 left) 2024-09-16T05:16:29.814 INFO:tasks.rados.rados.2.smithi204.stdout:1428: done (1 left) 2024-09-16T05:16:29.815 INFO:tasks.rados.rados.2.smithi204.stdout:1430: done (0 left) 2024-09-16T05:16:29.815 INFO:tasks.rados.rados.2.smithi204.stdout:1431: write oid 26 current snap is 154 2024-09-16T05:16:29.815 INFO:tasks.rados.rados.2.smithi204.stdout:1431: seq_num 509 ranges {554763=781039,2102511=452252} 2024-09-16T05:16:29.821 INFO:tasks.rados.rados.2.smithi204.stdout:1431: writing smithi20457400-26 from 554763 to 1335802 tid 1 2024-09-16T05:16:29.828 INFO:tasks.rados.rados.2.smithi204.stdout:1431: writing smithi20457400-26 from 2102511 to 2554763 tid 2 2024-09-16T05:16:29.828 INFO:tasks.rados.rados.2.smithi204.stdout:1432: rollback oid 34 current snap is 154 2024-09-16T05:16:29.828 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 152 2024-09-16T05:16:29.828 INFO:tasks.rados.rados.2.smithi204.stdout:1431: finishing write tid 1 to smithi20457400-26 2024-09-16T05:16:29.828 INFO:tasks.rados.rados.2.smithi204.stdout:1433: delete oid 1 current snap is 154 2024-09-16T05:16:29.830 INFO:tasks.rados.rados.2.smithi204.stdout:1433: done (2 left) 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:1434: snap_create 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:1431: finishing write tid 2 to smithi20457400-26 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:1432: finishing rollback tid 0 to smithi20457400-34 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:1431: finishing write tid 3 to smithi20457400-26 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:1432: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:1432: finishing rollback tid 2 to smithi20457400-34 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 500 (ObjNum 492 snap 152 seq_num 492) dirty exists 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:1431: finishing write tid 4 to smithi20457400-26 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 728 (ObjNum 509 snap 154 seq_num 509) dirty exists 2024-09-16T05:16:29.831 INFO:tasks.rados.rados.2.smithi204.stdout:1431: left oid 26 (ObjNum 509 snap 154 seq_num 509) 2024-09-16T05:16:29.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:29.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:29.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:29.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:29.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:29.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:30.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:16:30.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:30.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:30.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:30.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:30.229 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:30.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:30.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:30 smithi196 ceph-mon[30802]: pgmap v819: 107 pgs: 1 activating, 4 peering, 1 remapped+peering, 7 unknown, 94 active+clean; 128 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-16T05:16:30.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:30 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:16:30.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:30 smithi196 ceph-mon[30802]: osdmap e440: 12 total, 12 up, 12 in 2024-09-16T05:16:30.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:30.348 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:29 smithi032 ceph-mon[27388]: pgmap v819: 107 pgs: 1 activating, 4 peering, 1 remapped+peering, 7 unknown, 94 active+clean; 128 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-16T05:16:30.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:29 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:16:30.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:29 smithi032 ceph-mon[27388]: osdmap e440: 12 total, 12 up, 12 in 2024-09-16T05:16:30.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:30.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:29 smithi204 ceph-mon[30933]: pgmap v819: 107 pgs: 1 activating, 4 peering, 1 remapped+peering, 7 unknown, 94 active+clean; 128 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 3.4 MiB/s wr, 22 op/s 2024-09-16T05:16:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:29 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:16:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:29 smithi204 ceph-mon[30933]: osdmap e440: 12 total, 12 up, 12 in 2024-09-16T05:16:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:30.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:30.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:30.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:30.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:30.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:30.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:30.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:30.730 INFO:tasks.rados.rados.2.smithi204.stdout:1431: done (2 left) 2024-09-16T05:16:30.730 INFO:tasks.rados.rados.2.smithi204.stdout:1432: done (1 left) 2024-09-16T05:16:30.730 INFO:tasks.rados.rados.2.smithi204.stdout:1434: done (0 left) 2024-09-16T05:16:30.730 INFO:tasks.rados.rados.2.smithi204.stdout:1435: write (excl) oid 31 current snap is 155 2024-09-16T05:16:30.730 INFO:tasks.rados.rados.2.smithi204.stdout:1435: seq_num 511 ranges {588377=647890,1935828=635840} 2024-09-16T05:16:30.737 INFO:tasks.rados.rados.2.smithi204.stdout:1435: writing smithi20457400-31 from 588377 to 1236267 tid 1 2024-09-16T05:16:30.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:30.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:30.744 INFO:tasks.rados.rados.2.smithi204.stdout:1435: writing smithi20457400-31 from 1935828 to 2571668 tid 2 2024-09-16T05:16:30.745 INFO:tasks.rados.rados.2.smithi204.stdout:1436: snap_create 2024-09-16T05:16:30.745 INFO:tasks.rados.rados.2.smithi204.stdout:1435: finishing write tid 1 to smithi20457400-31 2024-09-16T05:16:30.747 INFO:tasks.rados.rados.2.smithi204.stdout:1435: finishing write tid 2 to smithi20457400-31 2024-09-16T05:16:30.747 INFO:tasks.rados.rados.2.smithi204.stdout:1435: finishing write tid 3 to smithi20457400-31 2024-09-16T05:16:30.747 INFO:tasks.rados.rados.2.smithi204.stdout:1435: finishing write tid 4 to smithi20457400-31 2024-09-16T05:16:30.747 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 732 (ObjNum 511 snap 155 seq_num 511) dirty exists 2024-09-16T05:16:30.747 INFO:tasks.rados.rados.2.smithi204.stdout:1435: left oid 31 (ObjNum 511 snap 155 seq_num 511) 2024-09-16T05:16:30.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:30.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:30.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:30.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:30.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:30.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:30.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:31.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:31.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:31.171 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:31.171 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:31.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:31.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:31.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:31.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:31.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:31.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:31.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:31.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:31.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:31.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:31.620 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:31.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:31.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:31.737 INFO:tasks.rados.rados.2.smithi204.stdout:1435: done (1 left) 2024-09-16T05:16:31.737 INFO:tasks.rados.rados.2.smithi204.stdout:1436: done (0 left) 2024-09-16T05:16:31.737 INFO:tasks.rados.rados.2.smithi204.stdout:1437: delete oid 9 current snap is 156 2024-09-16T05:16:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1437: done (0 left) 2024-09-16T05:16:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1438: write oid 47 current snap is 156 2024-09-16T05:16:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1438: seq_num 513 ranges {621991=798388,1936439=639509} 2024-09-16T05:16:31.748 INFO:tasks.rados.rados.2.smithi204.stdout:1438: writing smithi20457400-47 from 621991 to 1420379 tid 1 2024-09-16T05:16:31.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:31.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1438: writing smithi20457400-47 from 1936439 to 2575948 tid 2 2024-09-16T05:16:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1439: write oid 30 current snap is 156 2024-09-16T05:16:31.756 INFO:tasks.rados.rados.2.smithi204.stdout:1438: finishing write tid 1 to smithi20457400-47 2024-09-16T05:16:31.756 INFO:tasks.rados.rados.2.smithi204.stdout:1439: seq_num 514 ranges {638798=473637,1594921=783167} 2024-09-16T05:16:31.761 INFO:tasks.rados.rados.2.smithi204.stdout:1439: writing smithi20457400-30 from 638798 to 1112435 tid 1 2024-09-16T05:16:31.769 INFO:tasks.rados.rados.2.smithi204.stdout:1439: writing smithi20457400-30 from 1594921 to 2378088 tid 2 2024-09-16T05:16:31.769 INFO:tasks.rados.rados.2.smithi204.stdout:1438: finishing write tid 2 to smithi20457400-47 2024-09-16T05:16:31.769 INFO:tasks.rados.rados.2.smithi204.stdout:1438: finishing write tid 3 to smithi20457400-47 2024-09-16T05:16:31.769 INFO:tasks.rados.rados.2.smithi204.stdout:1440: read oid 18 snap -1 2024-09-16T05:16:31.769 INFO:tasks.rados.rados.2.smithi204.stdout:1440: expect (ObjNum 505 snap 153 seq_num 505) 2024-09-16T05:16:31.769 INFO:tasks.rados.rados.2.smithi204.stdout:1441: snap_create 2024-09-16T05:16:31.769 INFO:tasks.rados.rados.2.smithi204.stdout:1438: finishing write tid 4 to smithi20457400-47 2024-09-16T05:16:31.770 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 557 (ObjNum 513 snap 156 seq_num 513) dirty exists 2024-09-16T05:16:31.770 INFO:tasks.rados.rados.2.smithi204.stdout:1438: left oid 47 (ObjNum 513 snap 156 seq_num 513) 2024-09-16T05:16:31.770 INFO:tasks.rados.rados.2.smithi204.stdout:1439: finishing write tid 1 to smithi20457400-30 2024-09-16T05:16:31.772 INFO:tasks.rados.rados.2.smithi204.stdout:1439: finishing write tid 2 to smithi20457400-30 2024-09-16T05:16:31.772 INFO:tasks.rados.rados.2.smithi204.stdout:1439: finishing write tid 3 to smithi20457400-30 2024-09-16T05:16:31.773 INFO:tasks.rados.rados.2.smithi204.stdout:1439: finishing write tid 4 to smithi20457400-30 2024-09-16T05:16:31.773 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 736 (ObjNum 514 snap 156 seq_num 514) dirty exists 2024-09-16T05:16:31.773 INFO:tasks.rados.rados.2.smithi204.stdout:1439: left oid 30 (ObjNum 514 snap 156 seq_num 514) 2024-09-16T05:16:31.778 INFO:tasks.rados.rados.2.smithi204.stdout:1440: expect (ObjNum 505 snap 153 seq_num 505) 2024-09-16T05:16:31.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:16:31.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:31.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:31 smithi032 ceph-mon[27388]: pgmap v822: 107 pgs: 1 active+remapped, 1 activating, 4 peering, 1 remapped+peering, 6 unknown, 94 active+clean; 124 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3 op/s 2024-09-16T05:16:31.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:31 smithi032 ceph-mon[27388]: osdmap e441: 12 total, 12 up, 12 in 2024-09-16T05:16:31.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:31.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:31.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:31 smithi196 ceph-mon[30802]: pgmap v822: 107 pgs: 1 active+remapped, 1 activating, 4 peering, 1 remapped+peering, 6 unknown, 94 active+clean; 124 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3 op/s 2024-09-16T05:16:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:31 smithi196 ceph-mon[30802]: osdmap e441: 12 total, 12 up, 12 in 2024-09-16T05:16:31.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:31.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:32.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:32.111 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:32.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:32.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:32.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:31 smithi204 ceph-mon[30933]: pgmap v822: 107 pgs: 1 active+remapped, 1 activating, 4 peering, 1 remapped+peering, 6 unknown, 94 active+clean; 124 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3 op/s 2024-09-16T05:16:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:31 smithi204 ceph-mon[30933]: osdmap e441: 12 total, 12 up, 12 in 2024-09-16T05:16:32.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:32.248 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:32.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:32.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:32.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:32.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:32.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:32.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:16:32.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:32.497 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:32.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:32.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:32.622 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:32.622 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:32.622 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:32.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:32.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:32.735 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:16:32.739 INFO:tasks.rados.rados.2.smithi204.stdout:1438: done (3 left) 2024-09-16T05:16:32.740 INFO:tasks.rados.rados.2.smithi204.stdout:1439: done (2 left) 2024-09-16T05:16:32.740 INFO:tasks.rados.rados.2.smithi204.stdout:1440: done (1 left) 2024-09-16T05:16:32.740 INFO:tasks.rados.rados.2.smithi204.stdout:1441: done (0 left) 2024-09-16T05:16:32.740 INFO:tasks.rados.rados.2.smithi204.stdout:1442: copy_from oid 42 from oid 31 current snap is 157 2024-09-16T05:16:32.740 INFO:tasks.rados.rados.2.smithi204.stdout:1443: read oid 32 snap -1 2024-09-16T05:16:32.740 INFO:tasks.rados.rados.2.smithi204.stdout:1443: expect deleted 2024-09-16T05:16:32.740 INFO:tasks.rados.rados.2.smithi204.stdout:1444: write (excl) oid 2 current snap is 157 2024-09-16T05:16:32.740 INFO:tasks.rados.rados.2.smithi204.stdout:1444: seq_num 515 ranges {655605=548886,1653403=643178} 2024-09-16T05:16:32.747 INFO:tasks.rados.rados.2.smithi204.stdout:1444: writing smithi20457400-2 from 655605 to 1204491 tid 1 2024-09-16T05:16:32.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:32.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:32.754 INFO:tasks.rados.rados.2.smithi204.stdout:1444: writing smithi20457400-2 from 1653403 to 2296581 tid 2 2024-09-16T05:16:32.754 INFO:tasks.rados.rados.2.smithi204.stdout:1445: write (excl) oid 3 current snap is 157 2024-09-16T05:16:32.754 INFO:tasks.rados.rados.2.smithi204.stdout:1445: seq_num 516 ranges {672412=624135,1995532=676880} 2024-09-16T05:16:32.760 INFO:tasks.rados.rados.2.smithi204.stdout:1445: writing smithi20457400-3 from 672412 to 1296547 tid 1 2024-09-16T05:16:32.768 INFO:tasks.rados.rados.2.smithi204.stdout:1445: writing smithi20457400-3 from 1995532 to 2672412 tid 2 2024-09-16T05:16:32.768 INFO:tasks.rados.rados.2.smithi204.stdout:1446: rollback oid 6 current snap is 157 2024-09-16T05:16:32.768 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 6 to 154 2024-09-16T05:16:32.768 INFO:tasks.rados.rados.2.smithi204.stdout:1442: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1444: finishing write tid 1 to smithi20457400-2 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1442: finishing copy_from to smithi20457400-42 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 732 (ObjNum 511 snap 155 seq_num 511) dirty exists 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1444: finishing write tid 2 to smithi20457400-2 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1444: finishing write tid 3 to smithi20457400-2 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1444: finishing write tid 4 to smithi20457400-2 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 334 (ObjNum 515 snap 157 seq_num 515) dirty exists 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1444: left oid 2 (ObjNum 515 snap 157 seq_num 515) 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1442: done (4 left) 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1443: done (3 left) 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1444: done (2 left) 2024-09-16T05:16:32.769 INFO:tasks.rados.rados.2.smithi204.stdout:1447: write (excl) oid 23 current snap is 157 2024-09-16T05:16:32.770 INFO:tasks.rados.rados.2.smithi204.stdout:1445: finishing write tid 1 to smithi20457400-3 2024-09-16T05:16:32.770 INFO:tasks.rados.rados.2.smithi204.stdout:1447: seq_num 517 ranges {689219=415737,1770367=646847} 2024-09-16T05:16:32.774 INFO:tasks.rados.rados.2.smithi204.stdout:1447: writing smithi20457400-23 from 689219 to 1104956 tid 1 2024-09-16T05:16:32.781 INFO:tasks.rados.rados.2.smithi204.stdout:1447: writing smithi20457400-23 from 1770367 to 2417214 tid 2 2024-09-16T05:16:32.781 INFO:tasks.rados.rados.2.smithi204.stdout:1448: read oid 7 snap 154 2024-09-16T05:16:32.781 INFO:tasks.rados.rados.2.smithi204.stdout:1448: expect (ObjNum 405 snap 122 seq_num 405) 2024-09-16T05:16:32.782 INFO:tasks.rados.rados.2.smithi204.stdout:1449: snap_remove snap 152 2024-09-16T05:16:32.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:32.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:32.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:32.901 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:32 smithi032 ceph-mon[27388]: osdmap e442: 12 total, 12 up, 12 in 2024-09-16T05:16:32.901 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:32 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/534513789' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:32.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:32.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:32 smithi196 ceph-mon[30802]: osdmap e442: 12 total, 12 up, 12 in 2024-09-16T05:16:32.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:32 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/534513789' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:33.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:33.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:33.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:33.132 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:33.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:32 smithi204 ceph-mon[30933]: osdmap e442: 12 total, 12 up, 12 in 2024-09-16T05:16:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:32 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/534513789' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:33.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:33.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:33.261 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:33.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:16:33.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:33.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:33.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:33.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:33.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:33.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:33.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:33.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:33.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:33.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:33.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:33.741 INFO:tasks.rados.rados.2.smithi204.stdout:1449: done (4 left) 2024-09-16T05:16:33.742 INFO:tasks.rados.rados.2.smithi204.stdout:1450: write oid 5 current snap is 157 2024-09-16T05:16:33.742 INFO:tasks.rados.rados.2.smithi204.stdout:1450: seq_num 518 ranges {706026=490986,1828849=790505} 2024-09-16T05:16:33.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:33.748 INFO:tasks.rados.rados.2.smithi204.stdout:1450: writing smithi20457400-5 from 706026 to 1197012 tid 1 2024-09-16T05:16:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:1450: writing smithi20457400-5 from 1828849 to 2619354 tid 2 2024-09-16T05:16:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:1451: snap_create 2024-09-16T05:16:33.756 INFO:tasks.rados.rados.2.smithi204.stdout:1445: finishing write tid 2 to smithi20457400-3 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1446: finishing rollback tid 0 to smithi20457400-6 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1445: finishing write tid 3 to smithi20457400-3 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1446: finishing rollback tid 1 to smithi20457400-6 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1445: finishing write tid 4 to smithi20457400-3 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 485 (ObjNum 516 snap 157 seq_num 516) dirty exists 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1445: left oid 3 (ObjNum 516 snap 157 seq_num 516) 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1446: finishing rollback tid 2 to smithi20457400-6 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 338 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1447: finishing write tid 1 to smithi20457400-23 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1447: finishing write tid 2 to smithi20457400-23 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1447: finishing write tid 3 to smithi20457400-23 2024-09-16T05:16:33.757 INFO:tasks.rados.rados.2.smithi204.stdout:1448: expect (ObjNum 405 snap 122 seq_num 405) 2024-09-16T05:16:33.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:33.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:33.850 INFO:tasks.rados.rados.2.smithi204.stdout:1447: finishing write tid 4 to smithi20457400-23 2024-09-16T05:16:33.850 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 323 (ObjNum 517 snap 157 seq_num 517) dirty exists 2024-09-16T05:16:33.850 INFO:tasks.rados.rados.2.smithi204.stdout:1447: left oid 23 (ObjNum 517 snap 157 seq_num 517) 2024-09-16T05:16:33.850 INFO:tasks.rados.rados.2.smithi204.stdout:1450: finishing write tid 1 to smithi20457400-5 2024-09-16T05:16:33.850 INFO:tasks.rados.rados.2.smithi204.stdout:1450: finishing write tid 2 to smithi20457400-5 2024-09-16T05:16:33.850 INFO:tasks.rados.rados.2.smithi204.stdout:1450: finishing write tid 3 to smithi20457400-5 2024-09-16T05:16:33.850 INFO:tasks.rados.rados.2.smithi204.stdout:1450: finishing write tid 4 to smithi20457400-5 2024-09-16T05:16:33.851 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 332 (ObjNum 518 snap 157 seq_num 518) dirty exists 2024-09-16T05:16:33.851 INFO:tasks.rados.rados.2.smithi204.stdout:1450: left oid 5 (ObjNum 518 snap 157 seq_num 518) 2024-09-16T05:16:33.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:33 smithi032 ceph-mon[27388]: pgmap v825: 107 pgs: 2 peering, 1 activating, 1 unknown, 103 active+clean; 117 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 26 op/s; 520 KiB/s, 2 objects/s recovering 2024-09-16T05:16:33.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:33 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:33.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:33 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:33.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:33 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:33.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/534513789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:33.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:33 smithi032 ceph-mon[27388]: osdmap e443: 12 total, 12 up, 12 in 2024-09-16T05:16:33.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:33.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:33.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:34.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:34.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:34.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:34.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:34.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:34.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:33 smithi204 ceph-mon[30933]: pgmap v825: 107 pgs: 2 peering, 1 activating, 1 unknown, 103 active+clean; 117 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 26 op/s; 520 KiB/s, 2 objects/s recovering 2024-09-16T05:16:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:33 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:33 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:33 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/534513789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:33 smithi204 ceph-mon[30933]: osdmap e443: 12 total, 12 up, 12 in 2024-09-16T05:16:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:33 smithi196 ceph-mon[30802]: pgmap v825: 107 pgs: 2 peering, 1 activating, 1 unknown, 103 active+clean; 117 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.9 MiB/s wr, 26 op/s; 520 KiB/s, 2 objects/s recovering 2024-09-16T05:16:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:33 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:33 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:33 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/534513789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:33 smithi196 ceph-mon[30802]: osdmap e443: 12 total, 12 up, 12 in 2024-09-16T05:16:34.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:34.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:34.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:34.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:34.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:34.412 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:34.413 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:34.439 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:34.439 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:34.462 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:34.462 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:34.483 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:34.483 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:34.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:34.505 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:34.506 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:34.526 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:34.527 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:34.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:34.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:34.548 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:34.570 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:34.570 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:34.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:34.591 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:34.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:34.613 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:34.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:34.634 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:34.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:34.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:34.657 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:34.657 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:34.751 INFO:tasks.rados.rados.2.smithi204.stdout:1445: done (5 left) 2024-09-16T05:16:34.751 INFO:tasks.rados.rados.2.smithi204.stdout:1446: done (4 left) 2024-09-16T05:16:34.751 INFO:tasks.rados.rados.2.smithi204.stdout:1447: done (3 left) 2024-09-16T05:16:34.751 INFO:tasks.rados.rados.2.smithi204.stdout:1448: done (2 left) 2024-09-16T05:16:34.751 INFO:tasks.rados.rados.2.smithi204.stdout:1450: done (1 left) 2024-09-16T05:16:34.751 INFO:tasks.rados.rados.2.smithi204.stdout:1451: done (0 left) 2024-09-16T05:16:34.751 INFO:tasks.rados.rados.2.smithi204.stdout:1452: snap_create 2024-09-16T05:16:34.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:34.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:34.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:34.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:34.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:34.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:35.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:35.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:35.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:35.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:34 smithi032 ceph-mon[27388]: osdmap e444: 12 total, 12 up, 12 in 2024-09-16T05:16:35.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/858650560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:35.068 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:16:35.069 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:16:35.069 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:16:35.069 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:16:35.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:35.161 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:35.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:34 smithi204 ceph-mon[30933]: osdmap e444: 12 total, 12 up, 12 in 2024-09-16T05:16:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/858650560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:34 smithi196 ceph-mon[30802]: osdmap e444: 12 total, 12 up, 12 in 2024-09-16T05:16:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/858650560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:35.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:35.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:35.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:35.310 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:16:35.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T05:16:35.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:35.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:35.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:35.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:35.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:35.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:35.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:35.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:35.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:35.725 INFO:tasks.rados.rados.2.smithi204.stdout:1452: done (0 left) 2024-09-16T05:16:35.725 INFO:tasks.rados.rados.2.smithi204.stdout:1453: rollback oid 15 current snap is 159 2024-09-16T05:16:35.725 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 15 to 156 2024-09-16T05:16:35.725 INFO:tasks.rados.rados.2.smithi204.stdout:1454: copy_from oid 30 from oid 28 current snap is 159 2024-09-16T05:16:35.725 INFO:tasks.rados.rados.2.smithi204.stdout:1455: write (excl) oid 34 current snap is 159 2024-09-16T05:16:35.725 INFO:tasks.rados.rados.2.smithi204.stdout:1455: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-16T05:16:35.731 INFO:tasks.rados.rados.2.smithi204.stdout:1455: writing smithi20457400-34 from 722833 to 1289068 tid 1 2024-09-16T05:16:35.739 INFO:tasks.rados.rados.2.smithi204.stdout:1455: writing smithi20457400-34 from 1887331 to 2537847 tid 2 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1456: read oid 25 snap 158 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1456: expect deleted 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1457: read oid 2 snap 155 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1457: expect (ObjNum 494 snap 152 seq_num 494) 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1458: delete oid 49 current snap is 159 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1454: finishing copy_from to smithi20457400-30 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1454: got expected ENOENT (src dne) 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1454: finishing copy_from racing read to smithi20457400-30 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1453: finishing rollback tid 0 to smithi20457400-15 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1453: finishing rollback tid 1 to smithi20457400-15 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1453: finishing rollback tid 2 to smithi20457400-15 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 131 (ObjNum 423 snap 124 seq_num 423) dirty exists 2024-09-16T05:16:35.740 INFO:tasks.rados.rados.2.smithi204.stdout:1455: finishing write tid 1 to smithi20457400-34 2024-09-16T05:16:35.741 INFO:tasks.rados.rados.2.smithi204.stdout:1453: done (5 left) 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1454: done (4 left) 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1456: done (3 left) 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1458: done (2 left) 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1459: rollback oid 46 current snap is 159 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 156 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1460: rollback oid 24 current snap is 159 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 154 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1461: read oid 44 snap 157 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1461: expect deleted 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1462: read oid 49 snap -1 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1462: expect deleted 2024-09-16T05:16:35.742 INFO:tasks.rados.rados.2.smithi204.stdout:1463: snap_remove snap 158 2024-09-16T05:16:35.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:35.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:35.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:35.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:16:35.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:35.905 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:35 smithi032 ceph-mon[27388]: pgmap v828: 107 pgs: 2 peering, 105 active+clean; 110 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s; 520 KiB/s, 2 objects/s recovering 2024-09-16T05:16:35.905 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:35 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-16T05:16:35.905 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:35 smithi032 ceph-mon[27388]: osdmap e445: 12 total, 12 up, 12 in 2024-09-16T05:16:35.905 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:35 smithi032 ceph-mon[27388]: osdmap e446: 12 total, 12 up, 12 in 2024-09-16T05:16:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:35.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:36.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:36.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:36.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:36.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:16:36.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:35 smithi204 ceph-mon[30933]: pgmap v828: 107 pgs: 2 peering, 105 active+clean; 110 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s; 520 KiB/s, 2 objects/s recovering 2024-09-16T05:16:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:35 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-16T05:16:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:35 smithi204 ceph-mon[30933]: osdmap e445: 12 total, 12 up, 12 in 2024-09-16T05:16:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:35 smithi204 ceph-mon[30933]: osdmap e446: 12 total, 12 up, 12 in 2024-09-16T05:16:36.172 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:36.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:36.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:35 smithi196 ceph-mon[30802]: pgmap v828: 107 pgs: 2 peering, 105 active+clean; 110 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.2 MiB/s wr, 28 op/s; 520 KiB/s, 2 objects/s recovering 2024-09-16T05:16:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:35 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-16T05:16:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:35 smithi196 ceph-mon[30802]: osdmap e445: 12 total, 12 up, 12 in 2024-09-16T05:16:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:35 smithi196 ceph-mon[30802]: osdmap e446: 12 total, 12 up, 12 in 2024-09-16T05:16:36.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:36.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:36.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:36.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:36.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:36.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:36.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:36.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:36.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:36.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:36.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:36.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:36.741 INFO:tasks.rados.rados.2.smithi204.stdout:1463: done (6 left) 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:1464: delete oid 47 current snap is 159 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:1460: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:1455: finishing write tid 2 to smithi20457400-34 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:1455: finishing write tid 3 to smithi20457400-34 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:1455: finishing write tid 4 to smithi20457400-34 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 506 (ObjNum 519 snap 159 seq_num 519) dirty exists 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:1455: left oid 34 (ObjNum 519 snap 159 seq_num 519) 2024-09-16T05:16:36.742 INFO:tasks.rados.rados.2.smithi204.stdout:1457: expect (ObjNum 494 snap 152 seq_num 494) 2024-09-16T05:16:36.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:36.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:36.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:36.870 INFO:tasks.rados.rados.2.smithi204.stdout:1459: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:16:36.870 INFO:tasks.rados.rados.2.smithi204.stdout:1459: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:16:36.870 INFO:tasks.rados.rados.2.smithi204.stdout:1459: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:16:36.870 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 341 (ObjNum 484 snap 151 seq_num 484) dirty exists 2024-09-16T05:16:36.870 INFO:tasks.rados.rados.2.smithi204.stdout:1455: done (6 left) 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1457: done (5 left) 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1459: done (4 left) 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1460: done (3 left) 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1461: done (2 left) 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1462: done (1 left) 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1464: done (0 left) 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1465: rollback oid 26 current snap is 159 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 157 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1466: write (excl) oid 35 current snap is 159 2024-09-16T05:16:36.871 INFO:tasks.rados.rados.2.smithi204.stdout:1466: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-16T05:16:36.878 INFO:tasks.rados.rados.2.smithi204.stdout:1466: writing smithi20457400-35 from 773254 to 1565236 tid 1 2024-09-16T05:16:36.886 INFO:tasks.rados.rados.2.smithi204.stdout:1466: writing smithi20457400-35 from 2346424 to 2773254 tid 2 2024-09-16T05:16:36.886 INFO:tasks.rados.rados.2.smithi204.stdout:1467: write oid 14 current snap is 159 2024-09-16T05:16:36.886 INFO:tasks.rados.rados.2.smithi204.stdout:1465: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:16:36.886 INFO:tasks.rados.rados.2.smithi204.stdout:1465: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:16:36.886 INFO:tasks.rados.rados.2.smithi204.stdout:1465: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:16:36.886 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 744 (ObjNum 509 snap 154 seq_num 509) dirty exists 2024-09-16T05:16:36.886 INFO:tasks.rados.rados.2.smithi204.stdout:1466: finishing write tid 1 to smithi20457400-35 2024-09-16T05:16:36.886 INFO:tasks.rados.rados.2.smithi204.stdout:1467: seq_num 523 ranges {790061=467231,2004906=541501} 2024-09-16T05:16:36.892 INFO:tasks.rados.rados.2.smithi204.stdout:1467: writing smithi20457400-14 from 790061 to 1257292 tid 1 2024-09-16T05:16:36.899 INFO:tasks.rados.rados.2.smithi204.stdout:1467: writing smithi20457400-14 from 2004906 to 2546407 tid 2 2024-09-16T05:16:36.899 INFO:tasks.rados.rados.2.smithi204.stdout:1465: done (2 left) 2024-09-16T05:16:36.899 INFO:tasks.rados.rados.2.smithi204.stdout:1468: snap_remove snap 157 2024-09-16T05:16:36.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:36.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:36.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:37.037 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:37.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:37.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:37.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:37.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:37.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:37.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:37.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:37.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:37.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:37.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:37.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:37.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:37.492 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:16:37.492 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:16:37.492 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:16:37.492 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:16:37.492 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:16:37.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:16:37.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:37.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:37.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:37.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:37.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T05:16:37.752 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:16:37.755 INFO:tasks.rados.rados.2.smithi204.stdout:1468: done (2 left) 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1469: rollback oid 27 current snap is 159 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 27 to 156 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1466: finishing write tid 2 to smithi20457400-35 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1467: finishing write tid 1 to smithi20457400-14 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1467: finishing write tid 2 to smithi20457400-14 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1467: finishing write tid 3 to smithi20457400-14 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1467: finishing write tid 4 to smithi20457400-14 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 748 (ObjNum 523 snap 159 seq_num 523) dirty exists 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1467: left oid 14 (ObjNum 523 snap 159 seq_num 523) 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1466: finishing write tid 3 to smithi20457400-35 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:1466: finishing write tid 4 to smithi20457400-35 2024-09-16T05:16:37.756 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 486 (ObjNum 522 snap 159 seq_num 522) dirty exists 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1466: left oid 35 (ObjNum 522 snap 159 seq_num 522) 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1466: done (2 left) 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1467: done (1 left) 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1470: read oid 11 snap -1 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1470: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1471: read oid 50 snap -1 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1471: expect (ObjNum 501 snap 152 seq_num 501) 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1472: write oid 16 current snap is 159 2024-09-16T05:16:37.757 INFO:tasks.rados.rados.2.smithi204.stdout:1472: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-16T05:16:37.760 INFO:tasks.rados.rados.2.smithi204.stdout:1472: writing smithi20457400-16 from 406868 to 949348 tid 1 2024-09-16T05:16:37.766 INFO:tasks.rados.rados.2.smithi204.stdout:1472: writing smithi20457400-16 from 1547035 to 1948547 tid 2 2024-09-16T05:16:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:1472: writing smithi20457400-16 from 2690354 to 2806868 tid 3 2024-09-16T05:16:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:1473: delete oid 43 current snap is 159 2024-09-16T05:16:37.771 INFO:tasks.rados.rados.2.smithi204.stdout:1469: finishing rollback tid 0 to smithi20457400-27 2024-09-16T05:16:37.772 INFO:tasks.rados.rados.2.smithi204.stdout:1469: finishing rollback tid 1 to smithi20457400-27 2024-09-16T05:16:37.772 INFO:tasks.rados.rados.2.smithi204.stdout:1469: finishing rollback tid 2 to smithi20457400-27 2024-09-16T05:16:37.772 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 303 (ObjNum 482 snap 151 seq_num 482) dirty exists 2024-09-16T05:16:37.772 INFO:tasks.rados.rados.2.smithi204.stdout:1470: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:16:37.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:37.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:37.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:37 smithi032 ceph-mon[27388]: pgmap v831: 107 pgs: 107 active+clean; 105 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-16T05:16:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:37 smithi032 ceph-mon[27388]: osdmap e447: 12 total, 12 up, 12 in 2024-09-16T05:16:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:37 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:37 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1679947222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:37 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:37.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3244369111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:16:37.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:37.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:37.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:38.036 INFO:tasks.rados.rados.2.smithi204.stdout:1471: expect (ObjNum 501 snap 152 seq_num 501) 2024-09-16T05:16:38.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:37 smithi196 ceph-mon[30802]: pgmap v831: 107 pgs: 107 active+clean; 105 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-16T05:16:38.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:37 smithi196 ceph-mon[30802]: osdmap e447: 12 total, 12 up, 12 in 2024-09-16T05:16:38.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:37 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:38.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:37 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:38.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1679947222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:38.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:37 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:38.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3244369111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:16:38.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:38.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:38.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:38.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:16:38.169 INFO:tasks.rados.rados.2.smithi204.stdout:1472: finishing write tid 1 to smithi20457400-16 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1472: finishing write tid 2 to smithi20457400-16 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1472: finishing write tid 3 to smithi20457400-16 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1472: finishing write tid 4 to smithi20457400-16 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1472: finishing write tid 5 to smithi20457400-16 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 677 (ObjNum 524 snap 159 seq_num 524) dirty exists 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1472: left oid 16 (ObjNum 524 snap 159 seq_num 524) 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1469: done (4 left) 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1470: done (3 left) 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1471: done (2 left) 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1472: done (1 left) 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1473: done (0 left) 2024-09-16T05:16:38.170 INFO:tasks.rados.rados.2.smithi204.stdout:1474: snap_remove snap 154 2024-09-16T05:16:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:37 smithi204 ceph-mon[30933]: pgmap v831: 107 pgs: 107 active+clean; 105 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.5 MiB/s wr, 14 op/s 2024-09-16T05:16:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:37 smithi204 ceph-mon[30933]: osdmap e447: 12 total, 12 up, 12 in 2024-09-16T05:16:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:37 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:37 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1679947222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:37 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3244369111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:16:38.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:38.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:38.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:38.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:38.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:38.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:38.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:38.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:38.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:38.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:38.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:38.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:38.736 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:38.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:38.760 INFO:tasks.rados.rados.2.smithi204.stdout:1474: done (0 left) 2024-09-16T05:16:38.760 INFO:tasks.rados.rados.2.smithi204.stdout:1475: snap_remove snap 155 2024-09-16T05:16:38.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:38.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:38.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:38.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T05:16:38.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:38.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3244369111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: osdmap e448: 12 total, 12 up, 12 in 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: 2.53 scrub starts 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: 2.53 scrub ok 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: 2.55 scrub starts 2024-09-16T05:16:39.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:38 smithi204 ceph-mon[30933]: 2.55 scrub ok 2024-09-16T05:16:39.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:39.108 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:39.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3244369111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:16:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: osdmap e448: 12 total, 12 up, 12 in 2024-09-16T05:16:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:16:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:16:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: 2.53 scrub starts 2024-09-16T05:16:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: 2.53 scrub ok 2024-09-16T05:16:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: 2.55 scrub starts 2024-09-16T05:16:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:38 smithi032 ceph-mon[27388]: 2.55 scrub ok 2024-09-16T05:16:39.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:39.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:39.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:39.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:39.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:39.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3244369111' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:16:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: osdmap e448: 12 total, 12 up, 12 in 2024-09-16T05:16:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:16:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:16:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: 2.53 scrub starts 2024-09-16T05:16:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: 2.53 scrub ok 2024-09-16T05:16:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: 2.55 scrub starts 2024-09-16T05:16:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:38 smithi196 ceph-mon[30802]: 2.55 scrub ok 2024-09-16T05:16:39.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:39.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:39.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:39.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:39.481 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:39.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:39.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:39.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:39.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:39.681 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:39.682 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:39.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:39.723 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:39.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:39.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:39.762 INFO:tasks.rados.rados.2.smithi204.stdout:1475: done (0 left) 2024-09-16T05:16:39.762 INFO:tasks.rados.rados.2.smithi204.stdout:1476: snap_remove snap 156 2024-09-16T05:16:39.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:39.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:39.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:39.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:39.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:39.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:39.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.4d scrub starts 2024-09-16T05:16:39.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.4d scrub ok 2024-09-16T05:16:39.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.54 scrub starts 2024-09-16T05:16:39.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.54 scrub ok 2024-09-16T05:16:39.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: pgmap v834: 107 pgs: 107 active+clean; 106 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-16T05:16:39.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:16:39.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: osdmap e449: 12 total, 12 up, 12 in 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.50 scrub starts 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.50 scrub ok 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.51 scrub starts 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.51 scrub ok 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:16:39.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:39 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:16:40.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:40.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:40.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.4d scrub starts 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.4d scrub ok 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.54 scrub starts 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.54 scrub ok 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: pgmap v834: 107 pgs: 107 active+clean; 106 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:16:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: osdmap e449: 12 total, 12 up, 12 in 2024-09-16T05:16:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.50 scrub starts 2024-09-16T05:16:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.50 scrub ok 2024-09-16T05:16:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.51 scrub starts 2024-09-16T05:16:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.51 scrub ok 2024-09-16T05:16:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:16:40.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:39 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:16:40.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:40.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:40.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.4d scrub starts 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.4d scrub ok 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.54 scrub starts 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.54 scrub ok 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: pgmap v834: 107 pgs: 107 active+clean; 106 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: osdmap e449: 12 total, 12 up, 12 in 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.50 scrub starts 2024-09-16T05:16:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.50 scrub ok 2024-09-16T05:16:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.51 scrub starts 2024-09-16T05:16:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.51 scrub ok 2024-09-16T05:16:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:16:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:39 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:16:40.325 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:16:40.325 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:16:40.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:16:40.325 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:16:40.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.325 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:115 2.a e112/107 2:580e4d0e:::smithi20457400-29:head v 112'6, mlcod=112'3)", 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 356.988917818, 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0311933, 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334345+0000", 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334350+0000", 2024-09-16T05:16:40.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.887e-06 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365213+0000", 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030863313999999999 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365220+0000", 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7870000000000004e-06 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365291+0000", 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1397999999999996e-05 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365307+0000", 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5990000000000001e-05 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365507+0000", 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000199221 2024-09-16T05:16:40.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365538+0000", 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1167000000000002e-05 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:125 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'7, mlcod=112'4)", 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 356.95874768900001, 2024-09-16T05:16:40.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006511045, 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.364515+0000", 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7000000000000004e-07 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365198+0000", 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068299699999999997 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369743+0000", 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045446979999999998 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369750+0000", 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1940000000000001e-06 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369826+0000", 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6160000000000003e-05 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369841+0000", 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4895000000000001e-05 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.370997+0000", 2024-09-16T05:16:40.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001155866 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.371026+0000", 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8564999999999998e-05 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:127 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'9, mlcod=112'5)", 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 356.95353677999998, 2024-09-16T05:16:40.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020225074999999999, 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:16:40.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369726+0000", 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369728+0000", 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9410000000000001e-06 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389627+0000", 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019899751 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389632+0000", 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389704+0000", 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1418e-05 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389718+0000", 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.433e-05 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389922+0000", 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020421600000000001 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389951+0000", 2024-09-16T05:16:40.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8140000000000002e-05 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:133 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'10, mlcod=112'7)", 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 356.93400285199999, 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074633729999999997, 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389261+0000", 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.973e-06 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389613+0000", 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035121600000000001 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395726+0000", 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006113851 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395731+0000", 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0359999999999997e-06 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395819+0000", 2024-09-16T05:16:40.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7128999999999999e-05 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395836+0000", 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7768999999999999e-05 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396693+0000", 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085685999999999998 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396723+0000", 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9539e-05 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:135 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'12, mlcod=112'8)", 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 356.92749187700002, 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069274319999999999, 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395771+0000", 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4499999999999997e-07 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395775+0000", 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6509999999999999e-06 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402319+0000", 2024-09-16T05:16:40.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065439469999999996 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402326+0000", 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5769999999999996e-06 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402424+0000", 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8171000000000006e-05 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402439+0000", 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4844999999999999e-05 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402660+0000", 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022083700000000001 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402698+0000", 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7858999999999998e-05 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.350 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:773 2.1 e159/107 2:8c1ba486:::smithi20457400-2:head v 159'34, mlcod=155'30)", 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 309.53008781400001, 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060955719999999996, 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793174+0000", 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.793175+0000", 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:16:40.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795522+0000", 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023476920000000002 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795574+0000", 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1459999999999999e-05 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795584+0000", 2024-09-16T05:16:40.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0274e-05 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795649+0000", 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4901999999999996e-05 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.795679+0000", 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9887e-05 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.799228+0000", 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035493679999999998 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.799270+0000", 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1813000000000001e-05 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:974 2.a e179/107 2:5827f14d:::smithi20457400-12:head v 179'156, mlcod=178'152)", 2024-09-16T05:16:40.355 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 289.31583019200002, 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068838909999999996, 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007432+0000", 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.007433+0000", 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.421e-06 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009914+0000", 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024808030000000002 2024-09-16T05:16:40.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009966+0000", 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1456999999999999e-05 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009975+0000", 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4010000000000003e-06 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010042+0000", 2024-09-16T05:16:40.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010064+0000", 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2282000000000001e-05 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014259+0000", 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041942020000000002 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014316+0000", 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7377999999999999e-05 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1133 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'206, mlcod=189'203)", 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 277.589035583, 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021225035, 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.360 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.85e-07 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734829+0000", 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060196800000000003 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734866+0000", 2024-09-16T05:16:40.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7469000000000003e-05 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734881+0000", 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5058999999999999e-05 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734970+0000", 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9004000000000004e-05 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734998+0000", 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7606e-05 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755384+0000", 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02038587 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755452+0000", 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7873999999999994e-05 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.364 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1134 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'207, mlcod=189'203)", 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 277.58425046100001, 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016477325000000001, 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739014+0000", 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2819999999999999e-06 2024-09-16T05:16:40.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739797+0000", 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078265599999999996 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739844+0000", 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7608000000000002e-05 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739854+0000", 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4239999999999999e-06 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739935+0000", 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1593000000000003e-05 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739963+0000", 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7492e-05 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755459+0000", 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015496565 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755489+0000", 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9705000000000001e-05 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.369 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1135 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'208, mlcod=189'203)", 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 277.57982762400002, 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012094011, 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743436+0000", 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.584e-06 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743535+0000", 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9110000000000005e-05 2024-09-16T05:16:40.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743568+0000", 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2335000000000001e-05 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743576+0000", 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.191e-06 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743646+0000", 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0481000000000005e-05 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743676+0000", 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9620000000000001e-05 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755492+0000", 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011815888 2024-09-16T05:16:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755529+0000", 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6801999999999998e-05 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1136 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'209, mlcod=189'203)", 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 277.579220066, 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011564019, 2024-09-16T05:16:40.374 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744043+0000", 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3600000000000003e-07 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744049+0000", 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8250000000000001e-06 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744069+0000", 2024-09-16T05:16:40.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0460000000000001e-05 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744074+0000", 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4730000000000002e-06 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744119+0000", 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744143+0000", 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4085999999999999e-05 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755570+0000", 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011427299 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755606+0000", 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146000000000003e-05 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.378 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:39 2.29 e259/253 2:94d33c56:::smithi20457400-8:4c v 259'246, mlcod=254'242)", 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 209.64501853100001, 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069169729999999999, 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7399999999999999e-07 2024-09-16T05:16:40.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678251+0000", 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1080000000000004e-06 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678321+0000", 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9762000000000001e-05 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678330+0000", 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0329999999999997e-06 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679306+0000", 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097581800000000004 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.680633+0000", 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013273779999999999 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.685128+0000", 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044947850000000003 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.685161+0000", 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2814999999999997e-05 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2256 2.34 e290/278 2:2cffa4d6:::smithi20457400-50:head v 290'215, mlcod=281'211)", 2024-09-16T05:16:40.383 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:41.694218+0000", 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 178.629044322, 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068076869999999998, 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.694218+0000", 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.694218+0000", 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.694218+0000", 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0599999999999999e-07 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698452+0000", 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042336099999999996 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698507+0000", 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5390999999999999e-05 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698518+0000", 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0443e-05 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698586+0000", 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8015999999999994e-05 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.698606+0000", 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0285999999999999e-05 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.700988+0000", 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002381958 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:41.701026+0000", 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7777000000000001e-05 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2575 2.a e335/320 2:5089ebe1:::smithi20457400-36:head v 335'425, mlcod=332'421)", 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:44.983092+0000", 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 115.34017073699999, 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067202360000000001, 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.388 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.983092+0000", 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.983092+0000", 2024-09-16T05:16:40.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.983093+0000", 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5850000000000001e-06 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.987313+0000", 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042203299999999996 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.987383+0000", 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0026999999999999e-05 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.987398+0000", 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4146e-05 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.987468+0000", 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0029000000000003e-05 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.987491+0000", 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3689e-05 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.989764+0000", 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022729859999999998 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:44.989812+0000", 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7444000000000002e-05 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2777 2.48 e359/322 2:12e92f6c:::smithi20457400-1:head v 359'155, mlcod=0'0)", 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 91.571923902999998, 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007754843, 2024-09-16T05:16:40.393 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:16:40.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751339+0000", 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5400000000000002e-07 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755560+0000", 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004221341 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755638+0000", 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8292000000000002e-05 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755652+0000", 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3702e-05 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755795+0000", 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014327200000000001 2024-09-16T05:16:40.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755823+0000", 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7668e-05 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.759023+0000", 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003199959 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.759093+0000", 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0355e-05 2024-09-16T05:16:40.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:606 2.48 e365/322 2:12e92f6c:::smithi20457400-1:79 v 365'161, mlcod=362'157)", 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 85.252126730000001, 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64187130800000003, 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:16:40.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071139+0000", 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5609999999999999e-06 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712578+0000", 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64143852199999996 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712582+0000", 2024-09-16T05:16:40.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-06 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712651+0000", 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9102000000000003e-05 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712670+0000", 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9165999999999999e-05 2024-09-16T05:16:40.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712968+0000", 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029722899999999998 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.713007+0000", 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9125999999999998e-05 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:614 2.48 e366/322 2:12e92f6c:::smithi20457400-1:79 v 366'163, mlcod=364'159)", 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 84.6026995, 2024-09-16T05:16:40.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11945663500000001, 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:16:40.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9699999999999996e-07 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720568+0000", 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6870000000000002e-06 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839371+0000", 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11880265399999999 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839384+0000", 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3156000000000001e-05 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839531+0000", 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014728399999999999 2024-09-16T05:16:40.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839575+0000", 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4223e-05 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839981+0000", 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040524499999999999 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.840019+0000", 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8788999999999997e-05 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3235 2.29 e406/253 2:9662c29c:::smithi20457400-9:head v 406'547, mlcod=402'543)", 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 44.593011994000001, 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081876009999999992, 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.408 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730252+0000", 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.857e-06 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734729+0000", 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044764510000000002 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734790+0000", 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1465999999999997e-05 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734803+0000", 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2729e-05 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734890+0000", 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7708999999999997e-05 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734938+0000", 2024-09-16T05:16:40.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7485999999999999e-05 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.738369+0000", 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034306649999999998 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.738438+0000", 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9238000000000004e-05 2024-09-16T05:16:40.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3405 2.3e e422/296 2:7e4e65a4:::smithi20457400-4:head v 422'355, mlcod=376'351)", 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:11.734510+0000", 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 28.588752643999999, 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061884000000000002, 2024-09-16T05:16:40.413 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.734510+0000", 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.734510+0000", 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.734510+0000", 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.97e-07 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.737848+0000", 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033377110000000001 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.737884+0000", 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6309000000000001e-05 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.737904+0000", 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0165000000000001e-05 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.737965+0000", 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1315999999999994e-05 2024-09-16T05:16:40.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.737991+0000", 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5505000000000001e-05 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.740651+0000", 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002660407 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:11.740698+0000", 2024-09-16T05:16:40.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6789999999999998e-05 2024-09-16T05:16:40.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.418 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.418 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3412 2.3e e424/296 2:7e4e65a4:::smithi20457400-4:head v 424'357, mlcod=378'353)", 2024-09-16T05:16:40.418 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:16:40.419 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.385796366000001, 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007702319, 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9000000000000001e-07 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942005+0000", 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045392510000000002 2024-09-16T05:16:40.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942060+0000", 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4716000000000003e-05 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942070+0000", 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0078000000000001e-05 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942135+0000", 2024-09-16T05:16:40.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4853999999999999e-05 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942159+0000", 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3839999999999999e-05 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.945112+0000", 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029533789999999999 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:16:40.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.945168+0000", 2024-09-16T05:16:40.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6011000000000003e-05 2024-09-16T05:16:40.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:16:40.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:16:40.424 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:16:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:40.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:40.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:40.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:40.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:40.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:40.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:40.557 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:40.578 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:40.578 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:40.599 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:40.599 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:40.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:40.620 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:40.640 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:40.640 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:40.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:40.661 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:40.661 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:40.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:40.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:40.682 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:40.682 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:40.704 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:40.704 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:40.725 INFO:tasks.rados.rados.2.smithi204.stdout:1476: done (0 left) 2024-09-16T05:16:40.725 INFO:tasks.rados.rados.2.smithi204.stdout:1477: write oid 37 current snap is 159 2024-09-16T05:16:40.725 INFO:tasks.rados.rados.2.smithi204.stdout:1477: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-09-16T05:16:40.727 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:40.730 INFO:tasks.rados.rados.2.smithi204.stdout:1477: writing smithi20457400-37 from 440482 to 849813 tid 1 2024-09-16T05:16:40.736 INFO:tasks.rados.rados.2.smithi204.stdout:1477: writing smithi20457400-37 from 1380352 to 1785533 tid 2 2024-09-16T05:16:40.742 INFO:tasks.rados.rados.2.smithi204.stdout:1477: writing smithi20457400-37 from 2461553 to 2840482 tid 3 2024-09-16T05:16:40.742 INFO:tasks.rados.rados.2.smithi204.stdout:1478: write oid 36 current snap is 159 2024-09-16T05:16:40.742 INFO:tasks.rados.rados.2.smithi204.stdout:1477: finishing write tid 1 to smithi20457400-37 2024-09-16T05:16:40.742 INFO:tasks.rados.rados.2.smithi204.stdout:1477: finishing write tid 2 to smithi20457400-37 2024-09-16T05:16:40.742 INFO:tasks.rados.rados.2.smithi204.stdout:1478: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-09-16T05:16:40.747 INFO:tasks.rados.rados.2.smithi204.stdout:1478: writing smithi20457400-36 from 457289 to 941869 tid 1 2024-09-16T05:16:40.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:40.753 INFO:tasks.rados.rados.2.smithi204.stdout:1478: writing smithi20457400-36 from 1438834 to 1987673 tid 2 2024-09-16T05:16:40.758 INFO:tasks.rados.rados.2.smithi204.stdout:1478: writing smithi20457400-36 from 2772623 to 2857289 tid 3 2024-09-16T05:16:40.758 INFO:tasks.rados.rados.2.smithi204.stdout:1479: read oid 12 snap -1 2024-09-16T05:16:40.758 INFO:tasks.rados.rados.2.smithi204.stdout:1479: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:40.758 INFO:tasks.rados.rados.2.smithi204.stdout:1480: write (excl) oid 42 current snap is 159 2024-09-16T05:16:40.758 INFO:tasks.rados.rados.2.smithi204.stdout:1477: finishing write tid 3 to smithi20457400-37 2024-09-16T05:16:40.758 INFO:tasks.rados.rados.2.smithi204.stdout:1480: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-09-16T05:16:40.764 INFO:tasks.rados.rados.2.smithi204.stdout:1480: writing smithi20457400-42 from 474096 to 1033925 tid 1 2024-09-16T05:16:40.770 INFO:tasks.rados.rados.2.smithi204.stdout:1480: writing smithi20457400-42 from 1780963 to 2189813 tid 2 2024-09-16T05:16:40.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:1480: writing smithi20457400-42 from 2800046 to 2874096 tid 3 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:1481: copy_from oid 22 from oid 45 current snap is 159 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:1477: finishing write tid 4 to smithi20457400-37 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:1477: finishing write tid 5 to smithi20457400-37 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 562 (ObjNum 526 snap 159 seq_num 526) dirty exists 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:1477: left oid 37 (ObjNum 526 snap 159 seq_num 526) 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:1478: finishing write tid 1 to smithi20457400-36 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:1478: finishing write tid 2 to smithi20457400-36 2024-09-16T05:16:40.775 INFO:tasks.rados.rados.2.smithi204.stdout:1478: finishing write tid 3 to smithi20457400-36 2024-09-16T05:16:40.776 INFO:tasks.rados.rados.2.smithi204.stdout:1477: done (4 left) 2024-09-16T05:16:40.776 INFO:tasks.rados.rados.2.smithi204.stdout:1482: write oid 27 current snap is 159 2024-09-16T05:16:40.776 INFO:tasks.rados.rados.2.smithi204.stdout:1478: finishing write tid 4 to smithi20457400-36 2024-09-16T05:16:40.776 INFO:tasks.rados.rados.2.smithi204.stdout:1478: finishing write tid 5 to smithi20457400-36 2024-09-16T05:16:40.776 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 475 (ObjNum 527 snap 159 seq_num 527) dirty exists 2024-09-16T05:16:40.776 INFO:tasks.rados.rados.2.smithi204.stdout:1478: left oid 36 (ObjNum 527 snap 159 seq_num 527) 2024-09-16T05:16:40.776 INFO:tasks.rados.rados.2.smithi204.stdout:1479: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:40.777 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:40.777 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:40.797 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:40.798 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:40.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:40.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:40.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: 2.4f scrub starts 2024-09-16T05:16:40.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: 2.4f scrub ok 2024-09-16T05:16:40.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:16:40.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:16:40.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: osdmap e450: 12 total, 12 up, 12 in 2024-09-16T05:16:40.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:16:40.808 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:16:40.809 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3942780610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:40.809 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:40 smithi032 ceph-mon[27388]: osdmap e451: 12 total, 12 up, 12 in 2024-09-16T05:16:40.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:40.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:40.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:40.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:41.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:41.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:41.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:41.094 INFO:tasks.rados.rados.2.smithi204.stdout:1480: finishing write tid 1 to smithi20457400-42 2024-09-16T05:16:41.094 INFO:tasks.rados.rados.2.smithi204.stdout:1480: finishing write tid 2 to smithi20457400-42 2024-09-16T05:16:41.094 INFO:tasks.rados.rados.2.smithi204.stdout:1480: finishing write tid 3 to smithi20457400-42 2024-09-16T05:16:41.094 INFO:tasks.rados.rados.2.smithi204.stdout:1480: finishing write tid 4 to smithi20457400-42 2024-09-16T05:16:41.094 INFO:tasks.rados.rados.2.smithi204.stdout:1480: finishing write tid 5 to smithi20457400-42 2024-09-16T05:16:41.095 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 736 (ObjNum 528 snap 159 seq_num 528) dirty exists 2024-09-16T05:16:41.095 INFO:tasks.rados.rados.2.smithi204.stdout:1480: left oid 42 (ObjNum 528 snap 159 seq_num 528) 2024-09-16T05:16:41.095 INFO:tasks.rados.rados.2.smithi204.stdout:1481: finishing copy_from racing read to smithi20457400-22 2024-09-16T05:16:41.095 INFO:tasks.rados.rados.2.smithi204.stdout:1481: finishing copy_from to smithi20457400-22 2024-09-16T05:16:41.095 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 339 (ObjNum 476 snap 150 seq_num 476) dirty exists 2024-09-16T05:16:41.095 INFO:tasks.rados.rados.2.smithi204.stdout:1482: seq_num 529 ranges {490903=635078,1839445=552508} 2024-09-16T05:16:41.099 INFO:tasks.rados.rados.2.smithi204.stdout:1482: writing smithi20457400-27 from 490903 to 1125981 tid 1 2024-09-16T05:16:41.107 INFO:tasks.rados.rados.2.smithi204.stdout:1482: writing smithi20457400-27 from 1839445 to 2391953 tid 2 2024-09-16T05:16:41.107 INFO:tasks.rados.rados.2.smithi204.stdout:1478: done (4 left) 2024-09-16T05:16:41.107 INFO:tasks.rados.rados.2.smithi204.stdout:1479: done (3 left) 2024-09-16T05:16:41.107 INFO:tasks.rados.rados.2.smithi204.stdout:1480: done (2 left) 2024-09-16T05:16:41.107 INFO:tasks.rados.rados.2.smithi204.stdout:1481: done (1 left) 2024-09-16T05:16:41.107 INFO:tasks.rados.rados.2.smithi204.stdout:1483: delete oid 49 current snap is 159 2024-09-16T05:16:41.107 INFO:tasks.rados.rados.2.smithi204.stdout:1482: finishing write tid 1 to smithi20457400-27 2024-09-16T05:16:41.108 INFO:tasks.rados.rados.2.smithi204.stdout:1483: done (1 left) 2024-09-16T05:16:41.108 INFO:tasks.rados.rados.2.smithi204.stdout:1484: copy_from oid 33 from oid 31 current snap is 159 2024-09-16T05:16:41.108 INFO:tasks.rados.rados.2.smithi204.stdout:1485: write oid 8 current snap is 159 2024-09-16T05:16:41.109 INFO:tasks.rados.rados.2.smithi204.stdout:1485: seq_num 531 ranges {524517=785576,1956409=556177} 2024-09-16T05:16:41.115 INFO:tasks.rados.rados.2.smithi204.stdout:1485: writing smithi20457400-8 from 524517 to 1310093 tid 1 2024-09-16T05:16:41.122 INFO:tasks.rados.rados.2.smithi204.stdout:1485: writing smithi20457400-8 from 1956409 to 2512586 tid 2 2024-09-16T05:16:41.122 INFO:tasks.rados.rados.2.smithi204.stdout:1486: copy_from oid 5 from oid 21 current snap is 159 2024-09-16T05:16:41.122 INFO:tasks.rados.rados.2.smithi204.stdout:1482: finishing write tid 2 to smithi20457400-27 2024-09-16T05:16:41.122 INFO:tasks.rados.rados.2.smithi204.stdout:1487: delete oid 20 current snap is 159 2024-09-16T05:16:41.122 INFO:tasks.rados.rados.2.smithi204.stdout:1482: finishing write tid 3 to smithi20457400-27 2024-09-16T05:16:41.122 INFO:tasks.rados.rados.2.smithi204.stdout:1482: finishing write tid 4 to smithi20457400-27 2024-09-16T05:16:41.122 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 306 (ObjNum 529 snap 159 seq_num 529) dirty exists 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:1482: left oid 27 (ObjNum 529 snap 159 seq_num 529) 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:1484: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:1484: finishing copy_from to smithi20457400-33 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 732 (ObjNum 511 snap 155 seq_num 511) dirty exists 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:1485: finishing write tid 1 to smithi20457400-8 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:1482: done (4 left) 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:1484: done (3 left) 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:1487: done (2 left) 2024-09-16T05:16:41.123 INFO:tasks.rados.rados.2.smithi204.stdout:1488: delete oid 1 current snap is 159 2024-09-16T05:16:41.124 INFO:tasks.rados.rados.2.smithi204.stdout:1488: done (2 left) 2024-09-16T05:16:41.124 INFO:tasks.rados.rados.2.smithi204.stdout:1489: copy_from oid 31 from oid 33 current snap is 159 2024-09-16T05:16:41.124 INFO:tasks.rados.rados.2.smithi204.stdout:1490: copy_from oid 41 from oid 17 current snap is 159 2024-09-16T05:16:41.124 INFO:tasks.rados.rados.2.smithi204.stdout:1491: write oid 4 current snap is 159 2024-09-16T05:16:41.125 INFO:tasks.rados.rados.2.smithi204.stdout:1491: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-09-16T05:16:41.131 INFO:tasks.rados.rados.2.smithi204.stdout:1491: writing smithi20457400-4 from 574938 to 1302614 tid 1 2024-09-16T05:16:41.138 INFO:tasks.rados.rados.2.smithi204.stdout:1491: writing smithi20457400-4 from 1731855 to 2435359 tid 2 2024-09-16T05:16:41.141 INFO:tasks.rados.rados.2.smithi204.stdout:1491: writing smithi20457400-4 from 2848231 to 2974938 tid 3 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1492: read oid 39 snap -1 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1492: expect deleted 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1493: delete oid 23 current snap is 159 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1485: finishing write tid 2 to smithi20457400-8 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1485: finishing write tid 3 to smithi20457400-8 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1485: finishing write tid 4 to smithi20457400-8 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 650 (ObjNum 531 snap 159 seq_num 531) dirty exists 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1485: left oid 8 (ObjNum 531 snap 159 seq_num 531) 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1490: finishing copy_from racing read to smithi20457400-41 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1486: finishing copy_from racing read to smithi20457400-5 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:1486: finishing copy_from to smithi20457400-5 2024-09-16T05:16:41.142 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 329 (ObjNum 493 snap 152 seq_num 493) dirty exists 2024-09-16T05:16:41.143 INFO:tasks.rados.rados.2.smithi204.stdout:1491: finishing write tid 1 to smithi20457400-4 2024-09-16T05:16:41.143 INFO:tasks.rados.rados.2.smithi204.stdout:1490: finishing copy_from to smithi20457400-41 2024-09-16T05:16:41.143 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 684 (ObjNum 414 snap 123 seq_num 414) dirty exists 2024-09-16T05:16:41.143 INFO:tasks.rados.rados.2.smithi204.stdout:1489: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:16:41.143 INFO:tasks.rados.rados.2.smithi204.stdout:1489: finishing copy_from to smithi20457400-31 2024-09-16T05:16:41.143 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 732 (ObjNum 511 snap 155 seq_num 511) dirty exists 2024-09-16T05:16:41.144 INFO:tasks.rados.rados.2.smithi204.stdout:1491: finishing write tid 2 to smithi20457400-4 2024-09-16T05:16:41.144 INFO:tasks.rados.rados.2.smithi204.stdout:1485: done (6 left) 2024-09-16T05:16:41.144 INFO:tasks.rados.rados.2.smithi204.stdout:1486: done (5 left) 2024-09-16T05:16:41.144 INFO:tasks.rados.rados.2.smithi204.stdout:1489: done (4 left) 2024-09-16T05:16:41.144 INFO:tasks.rados.rados.2.smithi204.stdout:1490: done (3 left) 2024-09-16T05:16:41.144 INFO:tasks.rados.rados.2.smithi204.stdout:1492: done (2 left) 2024-09-16T05:16:41.144 INFO:tasks.rados.rados.2.smithi204.stdout:1493: done (1 left) 2024-09-16T05:16:41.144 INFO:tasks.rados.rados.2.smithi204.stdout:1494: snap_create 2024-09-16T05:16:41.145 INFO:tasks.rados.rados.2.smithi204.stdout:1491: finishing write tid 3 to smithi20457400-4 2024-09-16T05:16:41.145 INFO:tasks.rados.rados.2.smithi204.stdout:1491: finishing write tid 4 to smithi20457400-4 2024-09-16T05:16:41.145 INFO:tasks.rados.rados.2.smithi204.stdout:1491: finishing write tid 5 to smithi20457400-4 2024-09-16T05:16:41.145 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 553 (ObjNum 534 snap 159 seq_num 534) dirty exists 2024-09-16T05:16:41.145 INFO:tasks.rados.rados.2.smithi204.stdout:1491: left oid 4 (ObjNum 534 snap 159 seq_num 534) 2024-09-16T05:16:41.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:41.169 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:41.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: 2.4f scrub starts 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: 2.4f scrub ok 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: osdmap e450: 12 total, 12 up, 12 in 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3942780610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:40 smithi204 ceph-mon[30933]: osdmap e451: 12 total, 12 up, 12 in 2024-09-16T05:16:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: 2.4f scrub starts 2024-09-16T05:16:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: 2.4f scrub ok 2024-09-16T05:16:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:16:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:16:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: osdmap e450: 12 total, 12 up, 12 in 2024-09-16T05:16:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:16:41.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:16:41.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3942780610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:41.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:40 smithi196 ceph-mon[30802]: osdmap e451: 12 total, 12 up, 12 in 2024-09-16T05:16:41.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:41.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:41.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:41.396 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:41.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:41.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:41.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:41.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:41.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:41.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:16:41.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:41.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:16:41.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:41.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:41.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:41.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:41.773 INFO:tasks.rados.rados.2.smithi204.stdout:1491: done (1 left) 2024-09-16T05:16:41.773 INFO:tasks.rados.rados.2.smithi204.stdout:1494: done (0 left) 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1495: read oid 24 snap 159 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1495: expect deleted 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1496: read oid 30 snap -1 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1496: expect (ObjNum 514 snap 156 seq_num 514) 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1497: rollback oid 26 current snap is 160 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 159 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1498: read oid 37 snap -1 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1498: expect (ObjNum 526 snap 159 seq_num 526) 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1499: write oid 13 current snap is 160 2024-09-16T05:16:41.774 INFO:tasks.rados.rados.2.smithi204.stdout:1499: seq_num 536 ranges {608552=478174,1848819=707173} 2024-09-16T05:16:41.780 INFO:tasks.rados.rados.2.smithi204.stdout:1499: writing smithi20457400-13 from 608552 to 1086726 tid 1 2024-09-16T05:16:41.789 INFO:tasks.rados.rados.2.smithi204.stdout:1499: writing smithi20457400-13 from 1848819 to 2555992 tid 2 2024-09-16T05:16:41.789 INFO:tasks.rados.rados.2.smithi204.stdout:1500: delete oid 33 current snap is 160 2024-09-16T05:16:41.789 INFO:tasks.rados.rados.2.smithi204.stdout:1498: expect (ObjNum 526 snap 159 seq_num 526) 2024-09-16T05:16:41.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:41.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:41.855 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:41.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:41.910 INFO:tasks.rados.rados.2.smithi204.stdout:1499: finishing write tid 1 to smithi20457400-13 2024-09-16T05:16:41.910 INFO:tasks.rados.rados.2.smithi204.stdout:1496: expect (ObjNum 514 snap 156 seq_num 514) 2024-09-16T05:16:41.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:41.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:41.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:41 smithi032 ceph-mon[27388]: 2.4c scrub starts 2024-09-16T05:16:41.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:41 smithi032 ceph-mon[27388]: 2.4c scrub ok 2024-09-16T05:16:41.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:41 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:16:41.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:41 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:16:41.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:41 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:16:41.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:41 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:16:41.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:41 smithi032 ceph-mon[27388]: pgmap v837: 107 pgs: 107 active+clean; 106 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 945 KiB/s wr, 4 op/s 2024-09-16T05:16:41.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:41 smithi196 ceph-mon[30802]: 2.4c scrub starts 2024-09-16T05:16:41.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:41 smithi196 ceph-mon[30802]: 2.4c scrub ok 2024-09-16T05:16:41.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:41 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:16:41.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:41 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:16:41.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:41 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:16:41.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:41 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:16:41.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:41 smithi196 ceph-mon[30802]: pgmap v837: 107 pgs: 107 active+clean; 106 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 945 KiB/s wr, 4 op/s 2024-09-16T05:16:42.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:42.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:42.110 INFO:tasks.rados.rados.2.smithi204.stdout:1497: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:16:42.110 INFO:tasks.rados.rados.2.smithi204.stdout:1497: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1497: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 775 (ObjNum 509 snap 154 seq_num 509) dirty exists 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1499: finishing write tid 2 to smithi20457400-13 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1499: finishing write tid 3 to smithi20457400-13 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1499: finishing write tid 4 to smithi20457400-13 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 517 (ObjNum 536 snap 160 seq_num 536) dirty exists 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1499: left oid 13 (ObjNum 536 snap 160 seq_num 536) 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1495: done (5 left) 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1496: done (4 left) 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1497: done (3 left) 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1498: done (2 left) 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1499: done (1 left) 2024-09-16T05:16:42.111 INFO:tasks.rados.rados.2.smithi204.stdout:1500: done (0 left) 2024-09-16T05:16:42.112 INFO:tasks.rados.rados.2.smithi204.stdout:1501: rollback oid 49 current snap is 160 2024-09-16T05:16:42.112 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 159 2024-09-16T05:16:42.112 INFO:tasks.rados.rados.2.smithi204.stdout:1502: delete oid 24 current snap is 160 2024-09-16T05:16:42.112 INFO:tasks.rados.rados.2.smithi204.stdout:1502: done (1 left) 2024-09-16T05:16:42.112 INFO:tasks.rados.rados.2.smithi204.stdout:1503: copy_from oid 34 from oid 45 current snap is 160 2024-09-16T05:16:42.113 INFO:tasks.rados.rados.2.smithi204.stdout:1504: snap_create 2024-09-16T05:16:42.113 INFO:tasks.rados.rados.2.smithi204.stdout:1501: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:16:42.113 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 225 snap 0 seq_num 4283288000) dirty dne 2024-09-16T05:16:42.117 INFO:tasks.rados.rados.2.smithi204.stdout:1503: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:16:42.119 INFO:tasks.rados.rados.2.smithi204.stdout:1503: finishing copy_from to smithi20457400-34 2024-09-16T05:16:42.120 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 339 (ObjNum 476 snap 150 seq_num 476) dirty exists 2024-09-16T05:16:42.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:42.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:42.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:41 smithi204 ceph-mon[30933]: 2.4c scrub starts 2024-09-16T05:16:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:41 smithi204 ceph-mon[30933]: 2.4c scrub ok 2024-09-16T05:16:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:41 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:16:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:41 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:16:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:41 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:16:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:41 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:16:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:41 smithi204 ceph-mon[30933]: pgmap v837: 107 pgs: 107 active+clean; 106 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 945 KiB/s wr, 4 op/s 2024-09-16T05:16:42.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:42.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:42.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:42.343 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:42.343 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:42.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:42.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:42.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:42.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:42.541 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:42.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:42.564 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:42.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:42.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:42.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T05:16:42.781 INFO:tasks.rados.rados.2.smithi204.stdout:1501: done (2 left) 2024-09-16T05:16:42.781 INFO:tasks.rados.rados.2.smithi204.stdout:1503: done (1 left) 2024-09-16T05:16:42.781 INFO:tasks.rados.rados.2.smithi204.stdout:1504: done (0 left) 2024-09-16T05:16:42.782 INFO:tasks.rados.rados.2.smithi204.stdout:1505: snap_remove snap 159 2024-09-16T05:16:42.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:42.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:42.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:42.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:42.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:42.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:16:43.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:43.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:43.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:43.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:42 smithi032 ceph-mon[27388]: osdmap e452: 12 total, 12 up, 12 in 2024-09-16T05:16:43.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1876199079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:16:43.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:43.164 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:43.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:43.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:42 smithi204 ceph-mon[30933]: osdmap e452: 12 total, 12 up, 12 in 2024-09-16T05:16:43.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1876199079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:16:43.213 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:42 smithi196 ceph-mon[30802]: osdmap e452: 12 total, 12 up, 12 in 2024-09-16T05:16:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1876199079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:16:43.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:43.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:43.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:43.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:43.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:43.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:43.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T05:16:43.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:43.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:43.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:43.664 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:43.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:43.691 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:43.691 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:43.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:43.785 INFO:tasks.rados.rados.2.smithi204.stdout:1505: done (0 left) 2024-09-16T05:16:43.785 INFO:tasks.rados.rados.2.smithi204.stdout:1506: snap_remove snap 160 2024-09-16T05:16:43.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:43.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:43.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:43.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:43.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:43.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:43 smithi196 ceph-mon[30802]: pgmap v840: 107 pgs: 107 active+clean; 104 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:16:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1876199079' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:16:43.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:43 smithi196 ceph-mon[30802]: osdmap e453: 12 total, 12 up, 12 in 2024-09-16T05:16:44.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:44.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:44.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:44.063 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:43 smithi032 ceph-mon[27388]: pgmap v840: 107 pgs: 107 active+clean; 104 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:16:44.063 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1876199079' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:16:44.063 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:43 smithi032 ceph-mon[27388]: osdmap e453: 12 total, 12 up, 12 in 2024-09-16T05:16:44.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:16:44.142 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:44.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:44.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:43 smithi204 ceph-mon[30933]: pgmap v840: 107 pgs: 107 active+clean; 104 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:16:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1876199079' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:16:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:43 smithi204 ceph-mon[30933]: osdmap e453: 12 total, 12 up, 12 in 2024-09-16T05:16:44.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:44.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:44.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:44.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:44.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:44.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:44.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:16:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:44.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:44.540 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:44.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:44.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:44.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:44.712 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:44.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:44.788 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:16:44.789 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T05:16:44.790 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:44.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:44.793 INFO:tasks.rados.rados.2.smithi204.stdout:1506: done (0 left) 2024-09-16T05:16:44.793 INFO:tasks.rados.rados.2.smithi204.stdout:1507: read oid 16 snap -1 2024-09-16T05:16:44.793 INFO:tasks.rados.rados.2.smithi204.stdout:1507: expect (ObjNum 524 snap 159 seq_num 524) 2024-09-16T05:16:44.793 INFO:tasks.rados.rados.2.smithi204.stdout:1508: write (excl) oid 47 current snap is 161 2024-09-16T05:16:44.793 INFO:tasks.rados.rados.2.smithi204.stdout:1508: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-09-16T05:16:44.800 INFO:tasks.rados.rados.2.smithi204.stdout:1508: writing smithi20457400-47 from 658973 to 1362894 tid 1 2024-09-16T05:16:44.807 INFO:tasks.rados.rados.2.smithi204.stdout:1508: writing smithi20457400-47 from 1907912 to 2478765 tid 2 2024-09-16T05:16:44.811 INFO:tasks.rados.rados.2.smithi204.stdout:1508: writing smithi20457400-47 from 2985346 to 3058973 tid 3 2024-09-16T05:16:44.811 INFO:tasks.rados.rados.2.smithi204.stdout:1509: snap_create 2024-09-16T05:16:44.811 INFO:tasks.rados.rados.2.smithi204.stdout:1507: expect (ObjNum 524 snap 159 seq_num 524) 2024-09-16T05:16:44.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:44.907 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:44.908 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:44.908 INFO:tasks.rados.rados.2.smithi204.stdout:1508: finishing write tid 1 to smithi20457400-47 2024-09-16T05:16:44.908 INFO:tasks.rados.rados.2.smithi204.stdout:1508: finishing write tid 2 to smithi20457400-47 2024-09-16T05:16:44.908 INFO:tasks.rados.rados.2.smithi204.stdout:1508: finishing write tid 3 to smithi20457400-47 2024-09-16T05:16:44.909 INFO:tasks.rados.rados.2.smithi204.stdout:1508: finishing write tid 4 to smithi20457400-47 2024-09-16T05:16:44.909 INFO:tasks.rados.rados.2.smithi204.stdout:1508: finishing write tid 5 to smithi20457400-47 2024-09-16T05:16:44.909 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 566 (ObjNum 539 snap 161 seq_num 539) dirty exists 2024-09-16T05:16:44.909 INFO:tasks.rados.rados.2.smithi204.stdout:1508: left oid 47 (ObjNum 539 snap 161 seq_num 539) 2024-09-16T05:16:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:44.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:44.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:44.929 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:44.929 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:44.950 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:44.950 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:44.950 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: 2.3f scrub starts 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: 2.3f scrub ok 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: osdmap e454: 12 total, 12 up, 12 in 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:44.951 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/117677004' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:16:44.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/567516827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:16:44.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:44.970 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:44.992 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:44.992 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:45.013 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:45.013 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:45.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:45.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:45.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:45.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: 2.3f scrub starts 2024-09-16T05:16:45.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: 2.3f scrub ok 2024-09-16T05:16:45.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:16:45.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:16:45.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: osdmap e454: 12 total, 12 up, 12 in 2024-09-16T05:16:45.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:45.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:16:45.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:45.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:45.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:45.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:45.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/117677004' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:45.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:16:45.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:16:45.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/567516827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:16:45.051 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:45.108 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:45.108 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:45.130 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:45.130 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:45.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:45.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:45.152 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:45.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:45.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: 2.3f scrub starts 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: 2.3f scrub ok 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: osdmap e454: 12 total, 12 up, 12 in 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:16:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/117677004' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:45.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:16:45.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:16:45.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/567516827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:16:45.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:45.173 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:45.194 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:45.194 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:45.251 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:16:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:16:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:53 2.d e112/107 2:ba21f1a9:::smithi20457400-14:head v 112'1, mlcod=0'0)", 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.147302+0000", 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 362.10212480299998, 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002142073, 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.147302+0000", 2024-09-16T05:16:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.147302+0000", 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.147303+0000", 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8799999999999999e-07 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148009+0000", 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00070642199999999995 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148032+0000", 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3362000000000001e-05 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148050+0000", 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8063e-05 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148197+0000", 2024-09-16T05:16:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014633199999999999 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.148228+0000", 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1118999999999998e-05 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.149398+0000", 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011701369999999999 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.149444+0000", 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6449999999999997e-05 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:290 2.d e117/107 2:b98a79db:::smithi20457400-31:head v 117'29, mlcod=116'25)", 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 355.89531237900002, 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061542619999999998, 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354117+0000", 2024-09-16T05:16:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2110000000000001e-06 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357718+0000", 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036011939999999998 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357775+0000", 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7274000000000001e-05 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357803+0000", 2024-09-16T05:16:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7860000000000001e-05 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357882+0000", 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9091999999999994e-05 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357905+0000", 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2544999999999998e-05 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.360225+0000", 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023196480000000001 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.360269+0000", 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4437999999999997e-05 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:395 2.d e125/107 2:ba7e58cf:::smithi20457400-26:head v 125'55, mlcod=123'52)", 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:56.650581+0000", 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 348.59884574, 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023262320000000001, 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.650581+0000", 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.650581+0000", 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.650582+0000", 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9299999999999998e-07 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651480+0000", 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089784100000000003 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651514+0000", 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3856999999999998e-05 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651520+0000", 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8220000000000001e-06 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651593+0000", 2024-09-16T05:16:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2138999999999995e-05 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.651609+0000", 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6759e-05 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.652874+0000", 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001264753 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:56.652908+0000", 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3568000000000001e-05 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:439 2.d e127/107 2:ba7e58cf:::smithi20457400-26:head v 127'64, mlcod=126'60)", 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:58.659078+0000", 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 346.59034883100003, 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022239510000000001, 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659078+0000", 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659078+0000", 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659079+0000", 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0100000000000001e-07 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659874+0000", 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079547000000000005 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659908+0000", 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3846000000000001e-05 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.659918+0000", 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0285000000000001e-05 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.660081+0000", 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016303800000000001 2024-09-16T05:16:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.660098+0000", 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6807e-05 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.661257+0000", 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001158844 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.661302+0000", 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5460000000000002e-05 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:442 2.d e127/107 2:ba7e58cf:::smithi20457400-26:head v 127'67, mlcod=127'64)", 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 346.580373377, 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047970019999999999, 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3000000000000001e-07 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669058+0000", 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7699999999999999e-06 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669074+0000", 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6268000000000001e-05 2024-09-16T05:16:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669078+0000", 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7500000000000001e-06 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669137+0000", 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.944e-05 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669158+0000", 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0706000000000001e-05 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.673817+0000", 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046588840000000003 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.673851+0000", 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3754000000000002e-05 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:605 2.d e142/107 2:b1061e13:::smithi20457400-33:head v 142'81, mlcod=142'78)", 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 331.43482129799997, 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052172579999999998, 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814607+0000", 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.544e-06 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817665+0000", 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030580270000000001 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817694+0000", 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8353000000000001e-05 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817717+0000", 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3300000000000001e-05 2024-09-16T05:16:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817769+0000", 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2147000000000001e-05 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817785+0000", 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5874000000000001e-05 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.819789+0000", 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020036870000000001 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.819823+0000", 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4326000000000003e-05 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1122 2.d e190/107 2:ba754fcb:::smithi20457400-30:head v 190'205, mlcod=187'201)", 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:01.725779+0000", 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 283.52364791600002, 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022270990000000002, 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.725779+0000", 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.725779+0000", 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.725779+0000", 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.91e-07 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726570+0000", 2024-09-16T05:16:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079017499999999995 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726602+0000", 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2237000000000003e-05 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726617+0000", 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5514000000000001e-05 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726767+0000", 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000150054 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.726795+0000", 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7081999999999999e-05 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.727963+0000", 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011684950000000001 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:01.728006+0000", 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3350999999999999e-05 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1389 2.1d e210/209 2:b98a79db:::smithi20457400-31:head v 210'262, mlcod=0'0)", 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 262.643380896, 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047389830000000004, 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606048+0000", 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.781e-06 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608682+0000", 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026339990000000001 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608713+0000", 2024-09-16T05:16:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0781999999999998e-05 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608722+0000", 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1139999999999995e-06 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608800+0000", 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8145000000000005e-05 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608820+0000", 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9692999999999999e-05 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.610714+0000", 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001894448 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.610785+0000", 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1020999999999996e-05 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1598 2.d e228/209 2:b1061e13:::smithi20457400-33:head v 228'278, mlcod=224'274)", 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 245.38331878400001, 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029840370000000001, 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866109+0000", 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8800000000000003e-07 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867732+0000", 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016233739999999999 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867777+0000", 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4693000000000001e-05 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867788+0000", 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1148000000000001e-05 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867850+0000", 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1811999999999997e-05 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867870+0000", 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9698e-05 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.869048+0000", 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011787259999999999 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.869092+0000", 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4097999999999997e-05 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.2.0:176 2.2b e260/252 2:d7c71ae7:::smithi20457400-5:39 v 260'182, mlcod=253'179)", 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:11.719838+0000", 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.52958936300001, 2024-09-16T05:16:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026440809999999999, 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719838+0000", 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719838+0000", 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719838+0000", 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719848+0000", 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0033e-05 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719905+0000", 2024-09-16T05:16:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6076000000000003e-05 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.719924+0000", 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9589000000000002e-05 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.720047+0000", 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000122698 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.720075+0000", 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8232e-05 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.722413+0000", 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002338241 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:11.722482+0000", 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8639000000000005e-05 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2143 2.2b e279/252 2:d7c71ae7:::smithi20457400-5:head v 279'188, mlcod=268'185)", 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 194.49199230900001, 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055854579999999997, 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7700000000000001e-07 2024-09-16T05:16:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760458+0000", 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003022725 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760491+0000", 2024-09-16T05:16:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3637000000000003e-05 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760500+0000", 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6240000000000008e-06 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760609+0000", 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108887 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760631+0000", 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2161e-05 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.762973+0000", 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023423939999999998 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.763020+0000", 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6853e-05 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2318 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 177.56356832500001, 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0025039739999999, 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2318 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685861+0000", 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6700000000000001e-06 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685867+0000", 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8179999999999998e-06 2024-09-16T05:16:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685885+0000", 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7696000000000001e-05 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685895+0000", 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5899999999999997e-06 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685955+0000", 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0501000000000002e-05 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685966+0000", 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0462000000000001e-05 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688335+0000", 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0023690300000001 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688363+0000", 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8206999999999999e-05 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2319 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 177.56345006800001, 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0024202550000001, 2024-09-16T05:16:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2319 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685978+0000", 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2500000000000005e-07 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685981+0000", 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6150000000000001e-06 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685989+0000", 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6159999999999996e-06 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685995+0000", 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6250000000000004e-06 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686041+0000", 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6694999999999999e-05 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686049+0000", 2024-09-16T05:16:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3170000000000004e-06 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688389+0000", 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002340623 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688397+0000", 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1389999999999995e-06 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2320 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 177.56336954299999, 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002368715, 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2320 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7199999999999999e-07 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686062+0000", 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5590000000000001e-06 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686069+0000", 2024-09-16T05:16:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2710000000000003e-06 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686075+0000", 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7590000000000003e-06 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686126+0000", 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1384999999999997e-05 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686135+0000", 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1710000000000008e-06 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688420+0000", 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0022843889999999 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688426+0000", 2024-09-16T05:16:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6089999999999997e-06 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2749 2.d e357/247 2:b21a57e7:::smithi20457400-25:head v 357'413, mlcod=356'409)", 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.532193836000005, 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079488969999999999, 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717234+0000", 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0210000000000001e-06 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722533+0000", 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052989389999999999 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722583+0000", 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9592000000000002e-05 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722593+0000", 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0567999999999999e-05 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722673+0000", 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.983e-05 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722700+0000", 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6908999999999999e-05 2024-09-16T05:16:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.725139+0000", 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024389920000000001 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.725182+0000", 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3046000000000001e-05 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2900 2.32 e372/275 2:4e7d172c:::smithi20457400-15:head v 372'109, mlcod=348'105)", 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 83.540605149000001, 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033081819999999998, 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:16:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7499999999999999e-07 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708827+0000", 2024-09-16T05:16:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9969999999999998e-06 2024-09-16T05:16:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708864+0000", 2024-09-16T05:16:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6820000000000003e-05 2024-09-16T05:16:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708880+0000", 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5908000000000001e-05 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708933+0000", 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2970000000000003e-05 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708951+0000", 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8570999999999999e-05 2024-09-16T05:16:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.712084+0000", 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003132297 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.712130+0000", 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6443999999999998e-05 2024-09-16T05:16:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2898 2.2b e372/329 2:d7c71ae7:::smithi20457400-5:head v 372'201, mlcod=330'198)", 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 83.524848003000002, 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085128900000000004, 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724580+0000", 2024-09-16T05:16:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729164+0000", 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045841340000000001 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729217+0000", 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3152999999999997e-05 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729225+0000", 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9850000000000006e-06 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729320+0000", 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5067000000000003e-05 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729381+0000", 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1297999999999995e-05 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.733031+0000", 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036494600000000002 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.733092+0000", 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1371000000000006e-05 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:3108 2.2b e388/329 2:d7c71ae7:::smithi20457400-5:head v 388'203, mlcod=360'200)", 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:37.720518+0000", 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 67.528909640999998, 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023880889999999999, 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.720518+0000", 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.720518+0000", 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.720518+0000", 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721510+0000", 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099188000000000011 2024-09-16T05:16:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721531+0000", 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1002999999999999e-05 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721538+0000", 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5669999999999997e-06 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721634+0000", 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6298000000000005e-05 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.721660+0000", 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5588999999999998e-05 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.722873+0000", 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001213446 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:37.722906+0000", 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2129999999999999e-05 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.2.0:12 2.2b e406/329 2:d7c71ae7:::smithi20457400-5:84 v 406'216, mlcod=398'212)", 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:55.718743+0000", 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 49.530684143999999, 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043287330000000004, 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718743+0000", 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718743+0000", 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718743+0000", 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4900000000000001e-07 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718760+0000", 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6220000000000001e-05 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718837+0000", 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7735e-05 2024-09-16T05:16:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718846+0000", 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4940000000000003e-06 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.722330+0000", 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034842520000000002 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.722355+0000", 2024-09-16T05:16:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4890999999999999e-05 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.723021+0000", 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00066640299999999998 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.723072+0000", 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0389000000000002e-05 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:3671 2.32 e446/275 2:4e7d172c:::smithi20457400-15:head v 446'130, mlcod=380'125)", 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:35.728486+0000", 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 9.5209416149999999, 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049019800000000002, 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.728486+0000", 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.728486+0000", 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.728486+0000", 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6199999999999997e-07 2024-09-16T05:16:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.728492+0000", 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6269999999999997e-06 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.732748+0000", 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042564739999999997 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.732764+0000", 2024-09-16T05:16:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5633000000000001e-05 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.732815+0000", 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0788000000000002e-05 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.732836+0000", 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1455999999999999e-05 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.733340+0000", 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050352199999999995 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.733388+0000", 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7718e-05 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:45.352 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:45.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:45.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:45.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:45.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:45.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:45.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:45.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:45.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:45.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:45.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:45.726 INFO:tasks.rados.rados.2.smithi204.stdout:1507: done (2 left) 2024-09-16T05:16:45.727 INFO:tasks.rados.rados.2.smithi204.stdout:1508: done (1 left) 2024-09-16T05:16:45.727 INFO:tasks.rados.rados.2.smithi204.stdout:1509: done (0 left) 2024-09-16T05:16:45.727 INFO:tasks.rados.rados.2.smithi204.stdout:1510: read oid 18 snap -1 2024-09-16T05:16:45.727 INFO:tasks.rados.rados.2.smithi204.stdout:1510: expect (ObjNum 505 snap 153 seq_num 505) 2024-09-16T05:16:45.727 INFO:tasks.rados.rados.2.smithi204.stdout:1511: snap_remove snap 161 2024-09-16T05:16:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:45.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:45.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:45.851 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:45.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: pgmap v843: 107 pgs: 107 active+clean; 94 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:16:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:16:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:16:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/117677004' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/567516827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:16:45.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: osdmap e455: 12 total, 12 up, 12 in 2024-09-16T05:16:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2311546200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:16:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:16:45.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:45 smithi032 ceph-mon[27388]: osdmap e456: 12 total, 12 up, 12 in 2024-09-16T05:16:45.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:45.978 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:45.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:45.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: pgmap v843: 107 pgs: 107 active+clean; 94 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/117677004' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/567516827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: osdmap e455: 12 total, 12 up, 12 in 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2311546200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:16:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:45 smithi196 ceph-mon[30802]: osdmap e456: 12 total, 12 up, 12 in 2024-09-16T05:16:46.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:16:46.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:46.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: pgmap v843: 107 pgs: 107 active+clean; 94 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/117677004' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/567516827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: osdmap e455: 12 total, 12 up, 12 in 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2311546200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:16:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:45 smithi204 ceph-mon[30933]: osdmap e456: 12 total, 12 up, 12 in 2024-09-16T05:16:46.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:46.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:46.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:46.307 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:46.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:46.355 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:46.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:46.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:46.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:46.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:46.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:46.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:46.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:46.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:46.728 INFO:tasks.rados.rados.2.smithi204.stdout:1511: done (1 left) 2024-09-16T05:16:46.729 INFO:tasks.rados.rados.2.smithi204.stdout:1512: write (excl) oid 30 current snap is 162 2024-09-16T05:16:46.729 INFO:tasks.rados.rados.2.smithi204.stdout:1512: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-09-16T05:16:46.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:46.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:46.735 INFO:tasks.rados.rados.2.smithi204.stdout:1512: writing smithi20457400-30 from 675780 to 1171303 tid 1 2024-09-16T05:16:46.742 INFO:tasks.rados.rados.2.smithi204.stdout:1512: writing smithi20457400-30 from 1682747 to 2397258 tid 2 2024-09-16T05:16:46.746 INFO:tasks.rados.rados.2.smithi204.stdout:1512: writing smithi20457400-30 from 3012769 to 3075780 tid 3 2024-09-16T05:16:46.746 INFO:tasks.rados.rados.2.smithi204.stdout:1513: delete oid 32 current snap is 162 2024-09-16T05:16:46.746 INFO:tasks.rados.rados.2.smithi204.stdout:1510: expect (ObjNum 505 snap 153 seq_num 505) 2024-09-16T05:16:46.817 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:46.817 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:16:46.817 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:16:46.817 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:46.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:46.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:46.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:46 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:16:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:46 smithi032 ceph-mon[27388]: 2.44 scrub ok 2024-09-16T05:16:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:46 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:16:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:46 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:16:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:46 smithi032 ceph-mon[27388]: osdmap e457: 12 total, 12 up, 12 in 2024-09-16T05:16:46.921 INFO:tasks.rados.rados.2.smithi204.stdout:1512: finishing write tid 1 to smithi20457400-30 2024-09-16T05:16:46.921 INFO:tasks.rados.rados.2.smithi204.stdout:1512: finishing write tid 2 to smithi20457400-30 2024-09-16T05:16:46.921 INFO:tasks.rados.rados.2.smithi204.stdout:1512: finishing write tid 3 to smithi20457400-30 2024-09-16T05:16:46.921 INFO:tasks.rados.rados.2.smithi204.stdout:1512: finishing write tid 4 to smithi20457400-30 2024-09-16T05:16:46.921 INFO:tasks.rados.rados.2.smithi204.stdout:1512: finishing write tid 5 to smithi20457400-30 2024-09-16T05:16:46.921 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 781 (ObjNum 540 snap 162 seq_num 540) dirty exists 2024-09-16T05:16:46.922 INFO:tasks.rados.rados.2.smithi204.stdout:1512: left oid 30 (ObjNum 540 snap 162 seq_num 540) 2024-09-16T05:16:46.922 INFO:tasks.rados.rados.2.smithi204.stdout:1510: done (2 left) 2024-09-16T05:16:46.922 INFO:tasks.rados.rados.2.smithi204.stdout:1512: done (1 left) 2024-09-16T05:16:46.922 INFO:tasks.rados.rados.2.smithi204.stdout:1513: done (0 left) 2024-09-16T05:16:46.922 INFO:tasks.rados.rados.2.smithi204.stdout:1514: write (excl) oid 24 current snap is 162 2024-09-16T05:16:46.922 INFO:tasks.rados.rados.2.smithi204.stdout:1514: seq_num 542 ranges {709394=646021,2083358=718180} 2024-09-16T05:16:46.927 INFO:tasks.rados.rados.2.smithi204.stdout:1514: writing smithi20457400-24 from 709394 to 1355415 tid 1 2024-09-16T05:16:46.935 INFO:tasks.rados.rados.2.smithi204.stdout:1514: writing smithi20457400-24 from 2083358 to 2801538 tid 2 2024-09-16T05:16:46.935 INFO:tasks.rados.rados.2.smithi204.stdout:1515: read oid 21 snap -1 2024-09-16T05:16:46.935 INFO:tasks.rados.rados.2.smithi204.stdout:1515: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:16:46.935 INFO:tasks.rados.rados.2.smithi204.stdout:1516: snap_create 2024-09-16T05:16:46.936 INFO:tasks.rados.rados.2.smithi204.stdout:1514: finishing write tid 1 to smithi20457400-24 2024-09-16T05:16:46.939 INFO:tasks.rados.rados.2.smithi204.stdout:1514: finishing write tid 2 to smithi20457400-24 2024-09-16T05:16:46.939 INFO:tasks.rados.rados.2.smithi204.stdout:1514: finishing write tid 3 to smithi20457400-24 2024-09-16T05:16:46.939 INFO:tasks.rados.rados.2.smithi204.stdout:1514: finishing write tid 4 to smithi20457400-24 2024-09-16T05:16:46.939 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 495 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:16:46.939 INFO:tasks.rados.rados.2.smithi204.stdout:1514: left oid 24 (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:16:46.941 INFO:tasks.rados.rados.2.smithi204.stdout:1515: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:16:46.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:46.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:46.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:46.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:46 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:16:46.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:46 smithi196 ceph-mon[30802]: 2.44 scrub ok 2024-09-16T05:16:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:46 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:16:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:46 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:16:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:46 smithi196 ceph-mon[30802]: osdmap e457: 12 total, 12 up, 12 in 2024-09-16T05:16:47.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T05:16:47.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:47.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:47.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:46 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:16:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:46 smithi204 ceph-mon[30933]: 2.44 scrub ok 2024-09-16T05:16:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:46 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:16:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:46 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:16:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:46 smithi204 ceph-mon[30933]: osdmap e457: 12 total, 12 up, 12 in 2024-09-16T05:16:47.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:47.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:47.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:47.257 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:47.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:47.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:47.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:47.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:47.478 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:47.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:47.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:47.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:47.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:47.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:16:47.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:16:47.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:47.727 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:47.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:47.811 INFO:tasks.rados.rados.2.smithi204.stdout:1514: done (2 left) 2024-09-16T05:16:47.811 INFO:tasks.rados.rados.2.smithi204.stdout:1515: done (1 left) 2024-09-16T05:16:47.811 INFO:tasks.rados.rados.2.smithi204.stdout:1516: done (0 left) 2024-09-16T05:16:47.812 INFO:tasks.rados.rados.2.smithi204.stdout:1517: copy_from oid 35 from oid 37 current snap is 163 2024-09-16T05:16:47.812 INFO:tasks.rados.rados.2.smithi204.stdout:1518: rollback oid 23 current snap is 163 2024-09-16T05:16:47.812 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 23 to 162 2024-09-16T05:16:47.812 INFO:tasks.rados.rados.2.smithi204.stdout:1519: delete oid 31 current snap is 163 2024-09-16T05:16:47.813 INFO:tasks.rados.rados.2.smithi204.stdout:1518: finishing rollback tid 1 to smithi20457400-23 2024-09-16T05:16:47.813 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:16:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:1518: done (2 left) 2024-09-16T05:16:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:1519: done (1 left) 2024-09-16T05:16:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:1520: snap_remove snap 162 2024-09-16T05:16:47.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:47.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:47.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:47.893 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:47.914 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:47 smithi032 ceph-mon[27388]: pgmap v846: 107 pgs: 107 active+clean; 88 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.5 MiB/s wr, 12 op/s 2024-09-16T05:16:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:47 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:16:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:47 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:16:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:47 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:16:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:47 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:16:47.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:47.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:47.981 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:47.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:47 smithi196 ceph-mon[30802]: pgmap v846: 107 pgs: 107 active+clean; 88 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.5 MiB/s wr, 12 op/s 2024-09-16T05:16:47.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:47 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:16:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:47 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:16:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:47 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:16:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:47 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:16:48.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:48.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:48.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:47 smithi204 ceph-mon[30933]: pgmap v846: 107 pgs: 107 active+clean; 88 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.5 MiB/s wr, 12 op/s 2024-09-16T05:16:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:47 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:16:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:47 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:16:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:47 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:16:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:47 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:16:48.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:48.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:48.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:48.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:48.340 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:16:48.341 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:48.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:48.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:48.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:48.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:48.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:48.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:48.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:16:48.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:48.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:48.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:48.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:48.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:16:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T05:16:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:1520: done (1 left) 2024-09-16T05:16:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:1521: write (excl) oid 29 current snap is 163 2024-09-16T05:16:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:1517: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:16:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:1521: seq_num 544 ranges {743008=796519,2200322=721849} 2024-09-16T05:16:48.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:48.825 INFO:tasks.rados.rados.2.smithi204.stdout:1521: writing smithi20457400-29 from 743008 to 1539527 tid 1 2024-09-16T05:16:48.833 INFO:tasks.rados.rados.2.smithi204.stdout:1521: writing smithi20457400-29 from 2200322 to 2922171 tid 2 2024-09-16T05:16:48.833 INFO:tasks.rados.rados.2.smithi204.stdout:1522: write (excl) oid 25 current snap is 163 2024-09-16T05:16:48.833 INFO:tasks.rados.rados.2.smithi204.stdout:1522: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-09-16T05:16:48.840 INFO:tasks.rados.rados.2.smithi204.stdout:1522: writing smithi20457400-25 from 759815 to 1231583 tid 1 2024-09-16T05:16:48.845 INFO:tasks.rados.rados.2.smithi204.stdout:1522: writing smithi20457400-25 from 1742451 to 2207958 tid 2 2024-09-16T05:16:48.851 INFO:tasks.rados.rados.2.smithi204.stdout:1522: writing smithi20457400-25 from 2917178 to 3159815 tid 3 2024-09-16T05:16:48.851 INFO:tasks.rados.rados.2.smithi204.stdout:1523: write (excl) oid 2 current snap is 163 2024-09-16T05:16:48.851 INFO:tasks.rados.rados.2.smithi204.stdout:1523: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-09-16T05:16:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:48.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:48.858 INFO:tasks.rados.rados.2.smithi204.stdout:1523: writing smithi20457400-2 from 776622 to 1323639 tid 1 2024-09-16T05:16:48.864 INFO:tasks.rados.rados.2.smithi204.stdout:1523: writing smithi20457400-2 from 1800933 to 2526451 tid 2 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1523: writing smithi20457400-2 from 2944601 to 3176622 tid 3 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1524: rollback oid 18 current snap is 163 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1524: done (4 left) 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1525: read oid 6 snap -1 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1525: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1526: write (excl) oid 33 current snap is 163 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1517: finishing copy_from to smithi20457400-35 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 562 (ObjNum 526 snap 159 seq_num 526) dirty exists 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1521: finishing write tid 1 to smithi20457400-29 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1521: finishing write tid 2 to smithi20457400-29 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1521: finishing write tid 3 to smithi20457400-29 2024-09-16T05:16:48.869 INFO:tasks.rados.rados.2.smithi204.stdout:1526: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2024-09-16T05:16:48.876 INFO:tasks.rados.rados.2.smithi204.stdout:1526: writing smithi20457400-33 from 793429 to 1415695 tid 1 2024-09-16T05:16:48.882 INFO:tasks.rados.rados.2.smithi204.stdout:1526: writing smithi20457400-33 from 1859415 to 2328591 tid 2 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1526: writing smithi20457400-33 from 2972024 to 3193429 tid 3 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1517: done (5 left) 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1527: read oid 16 snap -1 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1527: expect (ObjNum 524 snap 159 seq_num 524) 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1528: rollback oid 44 current snap is 163 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1528: done (6 left) 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1529: snap_create 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1521: finishing write tid 4 to smithi20457400-29 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 674 (ObjNum 544 snap 163 seq_num 544) dirty exists 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1521: left oid 29 (ObjNum 544 snap 163 seq_num 544) 2024-09-16T05:16:48.888 INFO:tasks.rados.rados.2.smithi204.stdout:1522: finishing write tid 1 to smithi20457400-25 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1522: finishing write tid 2 to smithi20457400-25 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1522: finishing write tid 3 to smithi20457400-25 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1522: finishing write tid 4 to smithi20457400-25 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1522: finishing write tid 5 to smithi20457400-25 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 469 (ObjNum 545 snap 163 seq_num 545) dirty exists 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1522: left oid 25 (ObjNum 545 snap 163 seq_num 545) 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1523: finishing write tid 1 to smithi20457400-2 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1523: finishing write tid 2 to smithi20457400-2 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1523: finishing write tid 3 to smithi20457400-2 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1523: finishing write tid 4 to smithi20457400-2 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:1523: finishing write tid 5 to smithi20457400-2 2024-09-16T05:16:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 338 (ObjNum 546 snap 163 seq_num 546) dirty exists 2024-09-16T05:16:48.890 INFO:tasks.rados.rados.2.smithi204.stdout:1523: left oid 2 (ObjNum 546 snap 163 seq_num 546) 2024-09-16T05:16:48.890 INFO:tasks.rados.rados.2.smithi204.stdout:1525: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:16:48.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:48 smithi032 ceph-mon[27388]: osdmap e458: 12 total, 12 up, 12 in 2024-09-16T05:16:48.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/859139639' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:16:48.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/143354929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:48.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:48.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:48.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:49.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:16:49.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:48 smithi196 ceph-mon[30802]: osdmap e458: 12 total, 12 up, 12 in 2024-09-16T05:16:49.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/859139639' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:16:49.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/143354929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:49.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:49.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:49.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:49.146 INFO:tasks.rados.rados.2.smithi204.stdout:1526: finishing write tid 1 to smithi20457400-33 2024-09-16T05:16:49.146 INFO:tasks.rados.rados.2.smithi204.stdout:1526: finishing write tid 2 to smithi20457400-33 2024-09-16T05:16:49.147 INFO:tasks.rados.rados.2.smithi204.stdout:1526: finishing write tid 3 to smithi20457400-33 2024-09-16T05:16:49.147 INFO:tasks.rados.rados.2.smithi204.stdout:1526: finishing write tid 4 to smithi20457400-33 2024-09-16T05:16:49.147 INFO:tasks.rados.rados.2.smithi204.stdout:1526: finishing write tid 5 to smithi20457400-33 2024-09-16T05:16:49.147 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 737 (ObjNum 547 snap 163 seq_num 547) dirty exists 2024-09-16T05:16:49.147 INFO:tasks.rados.rados.2.smithi204.stdout:1526: left oid 33 (ObjNum 547 snap 163 seq_num 547) 2024-09-16T05:16:49.147 INFO:tasks.rados.rados.2.smithi204.stdout:1527: expect (ObjNum 524 snap 159 seq_num 524) 2024-09-16T05:16:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:48 smithi204 ceph-mon[30933]: osdmap e458: 12 total, 12 up, 12 in 2024-09-16T05:16:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/859139639' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:16:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/143354929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:16:49.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:49.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:49.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:49.292 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:49.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:49.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:49.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:49.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:49.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:49.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:49.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:49.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:49.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T05:16:49.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:49.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:49.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:49.756 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:49.756 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:16:49.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1378 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 207] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 267.16476861899997, 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018067625, 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.757 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1378 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589519+0000", 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3460000000000001e-06 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589547+0000", 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8572999999999999e-05 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589577+0000", 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9822e-05 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589592+0000", 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4630000000000001e-05 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589745+0000", 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015316700000000001 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589921+0000", 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000175995 2024-09-16T05:16:49.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.602330+0000", 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012408878 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605723+0000", 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033928000000000001 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607502+0000", 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017799140000000001 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607546+0000", 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3130999999999998e-05 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607584+0000", 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8368999999999998e-05 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1379 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:16:49.764 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 267.16466181499999, 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015638332000000001, 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1379 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589624+0000", 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4e-07 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589629+0000", 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.366e-06 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589642+0000", 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2114e-05 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589660+0000", 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8039000000000001e-05 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.590086+0000", 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042614600000000002 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.590119+0000", 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3395999999999997e-05 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605083+0000", 2024-09-16T05:16:49.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014963611999999999 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605167+0000", 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4444999999999995e-05 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605261+0000", 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4273999999999999e-05 2024-09-16T05:16:49.770 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(2.34 e278 from shard 3)", 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.88730824000001, 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34868665500000001, 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.771 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9999999999999997e-07 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866983+0000", 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.84e-06 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215470+0000", 2024-09-16T05:16:49.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34848719900000003 2024-09-16T05:16:49.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215480+0000", 2024-09-16T05:16:49.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2129999999999997e-06 2024-09-16T05:16:49.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215584+0000", 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010428400000000001 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215601+0000", 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7509000000000001e-05 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215663+0000", 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1909999999999995e-05 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2256 2.34 2:2cffa4d6:::smithi20457400-50:head [copy-from ver 319] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 188.07262770099999, 2024-09-16T05:16:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019683254000000001, 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2256 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681660+0000", 2024-09-16T05:16:49.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5019999999999999e-06 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681670+0000", 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.91e-06 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681686+0000", 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6699999999999999e-05 2024-09-16T05:16:49.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681695+0000", 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9590000000000001e-06 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681793+0000", 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7699000000000001e-05 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681800+0000", 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1199999999999996e-06 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.682948+0000", 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011475470000000001 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683035+0000", 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6886999999999997e-05 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.693853+0000", 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010818583 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698768+0000", 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049152340000000001 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701246+0000", 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024771630000000001 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701284+0000", 2024-09-16T05:16:49.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8544000000000002e-05 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701340+0000", 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6406000000000002e-05 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.785 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2257 2.34 2:2cffa4d6:::smithi20457400-50:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 188.07257222499999, 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016511652000000002, 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2257 2024-09-16T05:16:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7700000000000005e-07 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681718+0000", 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4920000000000004e-06 2024-09-16T05:16:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681724+0000", 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9900000000000002e-06 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681727+0000", 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8179999999999997e-06 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681858+0000", 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130698 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681862+0000", 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0149999999999997e-06 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683176+0000", 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013134119999999999 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683211+0000", 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5091e-05 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698063+0000", 2024-09-16T05:16:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014852783 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698134+0000", 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0570999999999999e-05 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698224+0000", 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0304999999999997e-05 2024-09-16T05:16:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2511 2.3 2.e1e51803 (undecoded) ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e320)", 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 144.931212905, 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99828526200000001, 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2511 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:16:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823073+0000", 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.04e-06 2024-09-16T05:16:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823078+0000", 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6519999999999997e-06 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823091+0000", 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3277e-05 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823100+0000", 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4909999999999994e-06 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823141+0000", 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0187e-05 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823147+0000", 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3929999999999996e-06 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821329+0000", 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99818146900000004 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821357+0000", 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8753000000000001e-05 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:136 2.34 e355/278 ondisk, result = 0)", 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 104.05251117900001, 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83858643600000005, 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.799 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701775+0000", 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6200000000000006e-07 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701786+0000", 2024-09-16T05:16:49.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0815000000000001e-05 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540112+0000", 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83832645699999997 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540147+0000", 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5416000000000001e-05 2024-09-16T05:16:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540246+0000", 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8171999999999995e-05 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540254+0000", 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540360+0000", 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010627400000000001 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2857 2.34 2:2cffa4d6:::smithi20457400-50:head [copy-from ver 325] snapc 7c=[7c,7b,7a,78,77,76,75,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:18.723212+0000", 2024-09-16T05:16:49.804 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 91.031073038000002, 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011657008999999999, 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2857 2024-09-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.723212+0000", 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.723212+0000", 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.723214+0000", 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0609999999999998e-06 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.723225+0000", 2024-09-16T05:16:49.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1161999999999999e-05 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.723248+0000", 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2799999999999999e-05 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.723258+0000", 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0526000000000001e-05 2024-09-16T05:16:49.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.723323+0000", 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4312000000000004e-05 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.723328+0000", 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6350000000000003e-06 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.724689+0000", 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001360378 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.724810+0000", 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121632 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:16:49.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.729603+0000", 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047927739999999996 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.733648+0000", 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040443930000000003 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.734786+0000", 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011386969999999999 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.734831+0000", 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4867000000000003e-05 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:18.734869+0000", 2024-09-16T05:16:49.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7771999999999997e-05 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3152 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 301] snapc 8a=[8a,88,87,86,85,84,83,81,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 63.041974725000003, 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017546488999999998, 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.813 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3152 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:16:49.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712312+0000", 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4449999999999999e-06 2024-09-16T05:16:49.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712319+0000", 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4000000000000003e-06 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712332+0000", 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2928000000000001e-05 2024-09-16T05:16:49.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712356+0000", 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4071999999999999e-05 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712421+0000", 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4709000000000007e-05 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:16:49.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712428+0000", 2024-09-16T05:16:49.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2520000000000002e-06 2024-09-16T05:16:49.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713705+0000", 2024-09-16T05:16:49.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001276672 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713800+0000", 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5266999999999994e-05 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.722741+0000", 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0089409339999999993 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.726991+0000", 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042500949999999997 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729789+0000", 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002798132 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729826+0000", 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6576999999999999e-05 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729857+0000", 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1006000000000001e-05 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3153 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e397)", 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:46.712369+0000", 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 63.041915529999997, 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01380496, 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.822 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3153 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712369+0000", 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712369+0000", 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712370+0000", 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.84e-07 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712372+0000", 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0200000000000001e-06 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712376+0000", 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5090000000000004e-06 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712379+0000", 2024-09-16T05:16:49.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.118e-06 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712473+0000", 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3710999999999998e-05 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712476+0000", 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3960000000000002e-06 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713946+0000", 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014693379999999999 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.714003+0000", 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7538999999999999e-05 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.726052+0000", 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012049169 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.726110+0000", 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7967000000000003e-05 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.726174+0000", 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3909000000000001e-05 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3573 2.3 2:c018a787:::smithi20457400-18:head [write 1692526~473837] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.888543041999998, 2024-09-16T05:16:49.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022224677000000002, 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3573 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865743+0000", 2024-09-16T05:16:49.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0100000000000001e-06 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866150+0000", 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040748299999999999 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866163+0000", 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2187999999999999e-05 2024-09-16T05:16:49.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866168+0000", 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4689999999999996e-06 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866251+0000", 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2911000000000006e-05 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866320+0000", 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8596000000000005e-05 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866403+0000", 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3796999999999997e-05 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867984+0000", 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015806399999999999 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887897+0000", 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019913157000000001 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887934+0000", 2024-09-16T05:16:49.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6482000000000001e-05 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887967+0000", 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2944000000000002e-05 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.836 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3574 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 2403500] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.888103488999999, 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039246627999999999, 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3574 2024-09-16T05:16:49.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:16:49.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866182+0000", 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866205+0000", 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2592e-05 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866222+0000", 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7422e-05 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866231+0000", 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5220000000000007e-06 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867407+0000", 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001176014 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867514+0000", 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106947 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867661+0000", 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000147396 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867987+0000", 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032622400000000001 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905316+0000", 2024-09-16T05:16:49.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037329222000000002 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905388+0000", 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1896000000000004e-05 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905428+0000", 2024-09-16T05:16:49.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9830000000000003e-05 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3575 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.888045492, 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039343647000000002, 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.844 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3575 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:16:49.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866240+0000", 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7999999999999995e-07 2024-09-16T05:16:49.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866243+0000", 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5770000000000001e-06 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866249+0000", 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866254+0000", 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-06 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867919+0000", 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016648609999999999 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867974+0000", 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4478000000000003e-05 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905464+0000", 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037490460000000003 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905516+0000", 2024-09-16T05:16:49.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1894999999999997e-05 2024-09-16T05:16:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905583+0000", 2024-09-16T05:16:49.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:16:49.851 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.851 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.851 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.851 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3579 2.34 2:2cffa4d6:::smithi20457400-50:head [write 420307~462694] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:16:49.851 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:16:49.852 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.883453313, 2024-09-16T05:16:49.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017271116999999999, 2024-09-16T05:16:49.852 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.852 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.852 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.852 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.852 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3579 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870833+0000", 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2440000000000001e-06 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871089+0000", 2024-09-16T05:16:49.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025604600000000001 2024-09-16T05:16:49.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871102+0000", 2024-09-16T05:16:49.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2955999999999999e-05 2024-09-16T05:16:49.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871110+0000", 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2700000000000004e-06 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871161+0000", 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0899999999999997e-05 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871220+0000", 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8659999999999997e-05 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871310+0000", 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0586000000000003e-05 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.872398+0000", 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010881409999999999 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888033+0000", 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015634834 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888071+0000", 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7957999999999998e-05 2024-09-16T05:16:49.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888103+0000", 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1522000000000001e-05 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3580 2.34 2:2cffa4d6:::smithi20457400-50:head [write 1351008~617495] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:16:49.860 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.877744748000001, 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030886027999999999, 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3580 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876542+0000", 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877114+0000", 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057156899999999996 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877126+0000", 2024-09-16T05:16:49.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2751999999999999e-05 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877134+0000", 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.072e-06 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0603000000000003e-05 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877240+0000", 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4561000000000002e-05 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877317+0000", 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7002999999999994e-05 2024-09-16T05:16:49.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:49.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878479+0000", 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011619130000000001 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907362+0000", 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028883039999999999 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907396+0000", 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4807000000000001e-05 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907426+0000", 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9821000000000001e-05 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3581 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2420307] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.877133954000001, 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030393973000000001, 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.870 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3581 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:16:49.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877152+0000", 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3099999999999996e-07 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877156+0000", 2024-09-16T05:16:49.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7589999999999999e-06 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877169+0000", 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3129000000000001e-05 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877175+0000", 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6259999999999997e-06 2024-09-16T05:16:49.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877997+0000", 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082123199999999997 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878081+0000", 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4914999999999996e-05 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878202+0000", 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120207 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878482+0000", 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027995500000000003 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907462+0000", 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028980381 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907500+0000", 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907545+0000", 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5120999999999997e-05 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3582 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.877099548, 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030495267999999999, 2024-09-16T05:16:49.877 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3582 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:16:49.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877186+0000", 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-07 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877189+0000", 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6800000000000002e-06 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877196+0000", 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0650000000000001e-06 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877202+0000", 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.28e-06 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878412+0000", 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001210205 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:16:49.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878470+0000", 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7759e-05 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907572+0000", 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029102086999999999 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907620+0000", 2024-09-16T05:16:49.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8384000000000003e-05 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907681+0000", 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0244000000000001e-05 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.883 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3687 2.16 2:6be67188:::smithi20457400-46:head [rollback 9e] snapc a0=[a0,9f,9e,9d,9c] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:35.741893+0000", 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.012392324, 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011433659000000001, 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3687 2024-09-16T05:16:49.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741893+0000", 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741893+0000", 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741893+0000", 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3600000000000003e-07 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741901+0000", 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3799999999999996e-06 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741909+0000", 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7940000000000003e-06 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741915+0000", 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1070000000000001e-06 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.742018+0000", 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010328900000000001 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.742024+0000", 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.711e-06 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.744775+0000", 2024-09-16T05:16:49.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002751468 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.744830+0000", 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5343000000000002e-05 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.744957+0000", 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012697699999999999 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.745554+0000", 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00059631400000000002 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.753240+0000", 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076867319999999999 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.753285+0000", 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4777000000000002e-05 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.753326+0000", 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1031000000000001e-05 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3688 2.16 2:6be67188:::smithi20457400-46:head [append 0~0] snapc a0=[a0,9f,9e,9d,9c] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:35.741946+0000", 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.012338615000001, 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011601593, 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:49.892 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3688 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741946+0000", 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741946+0000", 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741947+0000", 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7599999999999996e-07 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741962+0000", 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5164000000000001e-05 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741971+0000", 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3990000000000004e-06 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.741977+0000", 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5459999999999997e-06 2024-09-16T05:16:49.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.742066+0000", 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8843000000000003e-05 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.742070+0000", 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0729999999999998e-06 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.745257+0000", 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031867250000000001 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.745307+0000", 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0467999999999999e-05 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.745395+0000", 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7548999999999998e-05 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.745670+0000", 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027536599999999998 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:49.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.753482+0000", 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078121989999999997 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.753520+0000", 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7422000000000002e-05 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.753548+0000", 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8062999999999999e-05 2024-09-16T05:16:49.899 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:288 2.16 e446/273 ondisk, result = 0)", 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 12.870568188, 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65473014699999998, 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:49.900 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:49.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883718+0000", 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3799999999999996e-07 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883722+0000", 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.284e-06 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538184+0000", 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.654462031 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538201+0000", 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6634e-05 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538327+0000", 2024-09-16T05:16:49.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012605799999999999 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538338+0000", 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1188e-05 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:49.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538447+0000", 2024-09-16T05:16:49.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109014 2024-09-16T05:16:49.905 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.905 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.905 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.905 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:49.905 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:49.905 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:49.905 INFO:tasks.rados.rados.2.smithi204.stdout:1521: done (6 left) 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1522: done (5 left) 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1523: done (4 left) 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1525: done (3 left) 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1526: done (2 left) 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1527: done (1 left) 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1529: done (0 left) 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1530: read oid 24 snap -1 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1530: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:16:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:1531: snap_create 2024-09-16T05:16:49.907 INFO:tasks.rados.rados.2.smithi204.stdout:1530: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:16:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:49 smithi032 ceph-mon[27388]: pgmap v849: 107 pgs: 107 active+clean; 88 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-16T05:16:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:49 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/859139639' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:16:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/143354929' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:49.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:49 smithi032 ceph-mon[27388]: osdmap e459: 12 total, 12 up, 12 in 2024-09-16T05:16:49.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:49.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:49.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:49 smithi196 ceph-mon[30802]: pgmap v849: 107 pgs: 107 active+clean; 88 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-16T05:16:49.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:49 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:49.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/859139639' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:16:49.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/143354929' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:49.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:49 smithi196 ceph-mon[30802]: osdmap e459: 12 total, 12 up, 12 in 2024-09-16T05:16:50.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:50.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:50.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:50.094 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:50.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:49 smithi204 ceph-mon[30933]: pgmap v849: 107 pgs: 107 active+clean; 88 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2024-09-16T05:16:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:49 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/859139639' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:16:50.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/143354929' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:16:50.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:49 smithi204 ceph-mon[30933]: osdmap e459: 12 total, 12 up, 12 in 2024-09-16T05:16:50.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:50.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:50.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:50.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:50.296 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:50.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:50.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:50.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:50.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:50.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:50.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:50.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:50.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:16:50.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:50.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:50.728 INFO:tasks.rados.rados.2.smithi204.stdout:1530: done (1 left) 2024-09-16T05:16:50.728 INFO:tasks.rados.rados.2.smithi204.stdout:1531: done (0 left) 2024-09-16T05:16:50.728 INFO:tasks.rados.rados.2.smithi204.stdout:1532: rollback oid 44 current snap is 165 2024-09-16T05:16:50.728 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 164 2024-09-16T05:16:50.728 INFO:tasks.rados.rados.2.smithi204.stdout:1533: snap_remove snap 163 2024-09-16T05:16:50.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:50.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:50.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:50.892 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:50.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:50.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:50 smithi032 ceph-mon[27388]: osdmap e460: 12 total, 12 up, 12 in 2024-09-16T05:16:50.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:50 smithi032 ceph-mon[27388]: osdmap e461: 12 total, 12 up, 12 in 2024-09-16T05:16:50.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:50.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:50.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:50 smithi196 ceph-mon[30802]: osdmap e460: 12 total, 12 up, 12 in 2024-09-16T05:16:50.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:50 smithi196 ceph-mon[30802]: osdmap e461: 12 total, 12 up, 12 in 2024-09-16T05:16:51.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:51.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:51.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:50 smithi204 ceph-mon[30933]: osdmap e460: 12 total, 12 up, 12 in 2024-09-16T05:16:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:50 smithi204 ceph-mon[30933]: osdmap e461: 12 total, 12 up, 12 in 2024-09-16T05:16:51.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:51.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:51.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:51.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:51.323 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:51.323 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:51.323 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:51.324 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:51.324 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:51.324 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:51.336 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:51.336 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:51.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:51.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:51.357 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:51.358 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:51.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:51.380 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:51.381 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:16:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:16:51.382 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:51.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:51.402 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:51.423 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:51.423 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:51.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:51.444 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:51.467 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:51.467 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:51.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:51.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:51.488 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:51.488 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:51.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:51.510 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:51.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:51.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:51.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:51.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:51.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:51.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:51.726 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T05:16:51.730 INFO:tasks.rados.rados.2.smithi204.stdout:1533: done (1 left) 2024-09-16T05:16:51.730 INFO:tasks.rados.rados.2.smithi204.stdout:1534: rollback oid 5 current snap is 165 2024-09-16T05:16:51.730 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 5 to 164 2024-09-16T05:16:51.731 INFO:tasks.rados.rados.2.smithi204.stdout:1532: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:16:51.731 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 305 snap 0 seq_num 1) dirty dne 2024-09-16T05:16:51.731 INFO:tasks.rados.rados.2.smithi204.stdout:1532: done (1 left) 2024-09-16T05:16:51.731 INFO:tasks.rados.rados.2.smithi204.stdout:1535: read oid 14 snap 164 2024-09-16T05:16:51.731 INFO:tasks.rados.rados.2.smithi204.stdout:1535: expect (ObjNum 523 snap 159 seq_num 523) 2024-09-16T05:16:51.731 INFO:tasks.rados.rados.2.smithi204.stdout:1536: write oid 4 current snap is 165 2024-09-16T05:16:51.731 INFO:tasks.rados.rados.2.smithi204.stdout:1536: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-09-16T05:16:51.735 INFO:tasks.rados.rados.2.smithi204.stdout:1536: writing smithi20457400-4 from 410236 to 824104 tid 1 2024-09-16T05:16:51.742 INFO:tasks.rados.rados.2.smithi204.stdout:1536: writing smithi20457400-4 from 1234250 to 1963437 tid 2 2024-09-16T05:16:51.749 INFO:tasks.rados.rados.2.smithi204.stdout:1536: writing smithi20457400-4 from 2715800 to 3138863 tid 3 2024-09-16T05:16:51.749 INFO:tasks.rados.rados.2.smithi204.stdout:1537: read oid 9 snap 164 2024-09-16T05:16:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1537: expect deleted 2024-09-16T05:16:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1538: write (excl) oid 37 current snap is 165 2024-09-16T05:16:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1534: finishing rollback tid 0 to smithi20457400-5 2024-09-16T05:16:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1534: finishing rollback tid 1 to smithi20457400-5 2024-09-16T05:16:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1534: finishing rollback tid 2 to smithi20457400-5 2024-09-16T05:16:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 335 (ObjNum 493 snap 152 seq_num 493) dirty exists 2024-09-16T05:16:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1535: expect (ObjNum 523 snap 159 seq_num 523) 2024-09-16T05:16:51.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:51.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:51.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:51.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:51.935 INFO:tasks.rados.rados.2.smithi204.stdout:1536: finishing write tid 1 to smithi20457400-4 2024-09-16T05:16:51.935 INFO:tasks.rados.rados.2.smithi204.stdout:1536: finishing write tid 2 to smithi20457400-4 2024-09-16T05:16:51.936 INFO:tasks.rados.rados.2.smithi204.stdout:1536: finishing write tid 3 to smithi20457400-4 2024-09-16T05:16:51.936 INFO:tasks.rados.rados.2.smithi204.stdout:1536: finishing write tid 4 to smithi20457400-4 2024-09-16T05:16:51.936 INFO:tasks.rados.rados.2.smithi204.stdout:1536: finishing write tid 5 to smithi20457400-4 2024-09-16T05:16:51.936 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 557 (ObjNum 548 snap 165 seq_num 548) dirty exists 2024-09-16T05:16:51.936 INFO:tasks.rados.rados.2.smithi204.stdout:1536: left oid 4 (ObjNum 548 snap 165 seq_num 548) 2024-09-16T05:16:51.936 INFO:tasks.rados.rados.2.smithi204.stdout:1538: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2024-09-16T05:16:51.940 INFO:tasks.rados.rados.2.smithi204.stdout:1538: writing smithi20457400-37 from 427043 to 916160 tid 1 2024-09-16T05:16:51.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:51.946 INFO:tasks.rados.rados.2.smithi204.stdout:1538: writing smithi20457400-37 from 1576379 to 2049224 tid 2 2024-09-16T05:16:51.952 INFO:tasks.rados.rados.2.smithi204.stdout:1538: writing smithi20457400-37 from 2626870 to 3227043 tid 3 2024-09-16T05:16:51.953 INFO:tasks.rados.rados.2.smithi204.stdout:1534: done (4 left) 2024-09-16T05:16:51.953 INFO:tasks.rados.rados.2.smithi204.stdout:1535: done (3 left) 2024-09-16T05:16:51.953 INFO:tasks.rados.rados.2.smithi204.stdout:1536: done (2 left) 2024-09-16T05:16:51.953 INFO:tasks.rados.rados.2.smithi204.stdout:1537: done (1 left) 2024-09-16T05:16:51.953 INFO:tasks.rados.rados.2.smithi204.stdout:1539: delete oid 16 current snap is 165 2024-09-16T05:16:51.953 INFO:tasks.rados.rados.2.smithi204.stdout:1538: finishing write tid 1 to smithi20457400-37 2024-09-16T05:16:51.953 INFO:tasks.rados.rados.2.smithi204.stdout:1538: finishing write tid 2 to smithi20457400-37 2024-09-16T05:16:51.955 INFO:tasks.rados.rados.2.smithi204.stdout:1539: done (1 left) 2024-09-16T05:16:51.955 INFO:tasks.rados.rados.2.smithi204.stdout:1540: delete oid 22 current snap is 165 2024-09-16T05:16:51.955 INFO:tasks.rados.rados.2.smithi204.stdout:1538: finishing write tid 3 to smithi20457400-37 2024-09-16T05:16:51.955 INFO:tasks.rados.rados.2.smithi204.stdout:1538: finishing write tid 4 to smithi20457400-37 2024-09-16T05:16:51.955 INFO:tasks.rados.rados.2.smithi204.stdout:1538: finishing write tid 5 to smithi20457400-37 2024-09-16T05:16:51.955 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 570 (ObjNum 549 snap 165 seq_num 549) dirty exists 2024-09-16T05:16:51.955 INFO:tasks.rados.rados.2.smithi204.stdout:1538: left oid 37 (ObjNum 549 snap 165 seq_num 549) 2024-09-16T05:16:51.958 INFO:tasks.rados.rados.2.smithi204.stdout:1538: done (1 left) 2024-09-16T05:16:51.958 INFO:tasks.rados.rados.2.smithi204.stdout:1540: done (0 left) 2024-09-16T05:16:51.958 INFO:tasks.rados.rados.2.smithi204.stdout:1541: write oid 19 current snap is 165 2024-09-16T05:16:51.958 INFO:tasks.rados.rados.2.smithi204.stdout:1541: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-09-16T05:16:51.964 INFO:tasks.rados.rados.2.smithi204.stdout:1541: writing smithi20457400-19 from 477464 to 1192328 tid 1 2024-09-16T05:16:51.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:51.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:51.971 INFO:tasks.rados.rados.2.smithi204.stdout:1541: writing smithi20457400-19 from 1751825 to 2488350 tid 2 2024-09-16T05:16:51.976 INFO:tasks.rados.rados.2.smithi204.stdout:1541: writing smithi20457400-19 from 3225492 to 3277464 tid 3 2024-09-16T05:16:51.976 INFO:tasks.rados.rados.2.smithi204.stdout:1542: rollback oid 50 current snap is 165 2024-09-16T05:16:51.976 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 164 2024-09-16T05:16:51.976 INFO:tasks.rados.rados.2.smithi204.stdout:1541: finishing write tid 1 to smithi20457400-19 2024-09-16T05:16:51.976 INFO:tasks.rados.rados.2.smithi204.stdout:1543: snap_create 2024-09-16T05:16:51.977 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:51 smithi196 ceph-mon[30802]: pgmap v852: 107 pgs: 107 active+clean; 91 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 371 KiB/s wr, 1 op/s 2024-09-16T05:16:51.977 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3128279660' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:16:51.977 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2406026328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:51.977 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3128279660' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:16:51.977 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:51 smithi196 ceph-mon[30802]: osdmap e462: 12 total, 12 up, 12 in 2024-09-16T05:16:51.978 INFO:tasks.rados.rados.2.smithi204.stdout:1541: finishing write tid 2 to smithi20457400-19 2024-09-16T05:16:51.978 INFO:tasks.rados.rados.2.smithi204.stdout:1541: finishing write tid 3 to smithi20457400-19 2024-09-16T05:16:51.982 INFO:tasks.rados.rados.2.smithi204.stdout:1542: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:16:51.982 INFO:tasks.rados.rados.2.smithi204.stdout:1542: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:16:51.983 INFO:tasks.rados.rados.2.smithi204.stdout:1542: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:16:51.983 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 356 (ObjNum 501 snap 152 seq_num 501) dirty exists 2024-09-16T05:16:52.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:16:52.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:16:52.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:52.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:51 smithi032 ceph-mon[27388]: pgmap v852: 107 pgs: 107 active+clean; 91 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 371 KiB/s wr, 1 op/s 2024-09-16T05:16:52.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3128279660' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:16:52.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2406026328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:52.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3128279660' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:16:52.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:51 smithi032 ceph-mon[27388]: osdmap e462: 12 total, 12 up, 12 in 2024-09-16T05:16:52.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:52.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:51 smithi204 ceph-mon[30933]: pgmap v852: 107 pgs: 107 active+clean; 91 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 371 KiB/s wr, 1 op/s 2024-09-16T05:16:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3128279660' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:16:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2406026328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3128279660' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:16:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:51 smithi204 ceph-mon[30933]: osdmap e462: 12 total, 12 up, 12 in 2024-09-16T05:16:52.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:52.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:52.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:52.233 INFO:tasks.rados.rados.2.smithi204.stdout:1541: finishing write tid 4 to smithi20457400-19 2024-09-16T05:16:52.233 INFO:tasks.rados.rados.2.smithi204.stdout:1541: finishing write tid 5 to smithi20457400-19 2024-09-16T05:16:52.233 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 300 (ObjNum 552 snap 165 seq_num 552) dirty exists 2024-09-16T05:16:52.234 INFO:tasks.rados.rados.2.smithi204.stdout:1541: left oid 19 (ObjNum 552 snap 165 seq_num 552) 2024-09-16T05:16:52.297 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:52.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:52.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:52.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:52.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:52.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:52.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:52.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:52.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:52.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:52.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:52.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:52.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:52.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T05:16:52.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:52.837 INFO:tasks.rados.rados.2.smithi204.stdout:1541: done (2 left) 2024-09-16T05:16:52.837 INFO:tasks.rados.rados.2.smithi204.stdout:1542: done (1 left) 2024-09-16T05:16:52.837 INFO:tasks.rados.rados.2.smithi204.stdout:1543: done (0 left) 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1544: copy_from oid 25 from oid 1 current snap is 166 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1545: read oid 12 snap 164 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1545: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1546: read oid 36 snap -1 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1546: expect (ObjNum 527 snap 159 seq_num 527) 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1547: read oid 23 snap -1 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1547: expect deleted 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1548: write oid 29 current snap is 166 2024-09-16T05:16:52.838 INFO:tasks.rados.rados.2.smithi204.stdout:1548: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-09-16T05:16:52.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:52.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:52.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:52.844 INFO:tasks.rados.rados.2.smithi204.stdout:1548: writing smithi20457400-29 from 494271 to 1284384 tid 1 2024-09-16T05:16:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:52.848 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:52.848 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:52.850 INFO:tasks.rados.rados.2.smithi204.stdout:1548: writing smithi20457400-29 from 1693954 to 2174137 tid 2 2024-09-16T05:16:52.855 INFO:tasks.rados.rados.2.smithi204.stdout:1548: writing smithi20457400-29 from 2620209 to 3132338 tid 3 2024-09-16T05:16:52.855 INFO:tasks.rados.rados.2.smithi204.stdout:1549: delete oid 37 current snap is 166 2024-09-16T05:16:52.855 INFO:tasks.rados.rados.2.smithi204.stdout:1545: expect (ObjNum 440 snap 131 seq_num 440) 2024-09-16T05:16:52.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:52.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:52.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:53.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:53.091 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:53.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:53.168 INFO:tasks.rados.rados.2.smithi204.stdout:1546: expect (ObjNum 527 snap 159 seq_num 527) 2024-09-16T05:16:53.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:53.215 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:53.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:53.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:1544: finishing copy_from to smithi20457400-25 2024-09-16T05:16:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:1544: got expected ENOENT (src dne) 2024-09-16T05:16:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:1544: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:16:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:1548: finishing write tid 1 to smithi20457400-29 2024-09-16T05:16:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:1548: finishing write tid 2 to smithi20457400-29 2024-09-16T05:16:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:1548: finishing write tid 3 to smithi20457400-29 2024-09-16T05:16:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:1548: finishing write tid 4 to smithi20457400-29 2024-09-16T05:16:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:1548: finishing write tid 5 to smithi20457400-29 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 679 (ObjNum 553 snap 166 seq_num 553) dirty exists 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1548: left oid 29 (ObjNum 553 snap 166 seq_num 553) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1544: done (5 left) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1545: done (4 left) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1546: done (3 left) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1547: done (2 left) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1548: done (1 left) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1549: done (0 left) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1550: read oid 25 snap -1 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1550: expect (ObjNum 545 snap 163 seq_num 545) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1551: read oid 5 snap -1 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1551: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:16:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:1552: write (excl) oid 37 current snap is 166 2024-09-16T05:16:53.327 INFO:tasks.rados.rados.2.smithi204.stdout:1552: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2024-09-16T05:16:53.330 INFO:tasks.rados.rados.2.smithi204.stdout:1552: writing smithi20457400-37 from 527885 to 1184849 tid 1 2024-09-16T05:16:53.338 INFO:tasks.rados.rados.2.smithi204.stdout:1552: writing smithi20457400-37 from 1927271 to 2411123 tid 2 2024-09-16T05:16:53.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:53.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:53.345 INFO:tasks.rados.rados.2.smithi204.stdout:1552: writing smithi20457400-37 from 3191408 to 3327885 tid 3 2024-09-16T05:16:53.345 INFO:tasks.rados.rados.2.smithi204.stdout:1553: copy_from oid 20 from oid 40 current snap is 166 2024-09-16T05:16:53.345 INFO:tasks.rados.rados.2.smithi204.stdout:1552: finishing write tid 1 to smithi20457400-37 2024-09-16T05:16:53.345 INFO:tasks.rados.rados.2.smithi204.stdout:1551: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:16:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:53.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:53.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:53.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:53.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:16:53.536 INFO:tasks.rados.rados.2.smithi204.stdout:1552: finishing write tid 2 to smithi20457400-37 2024-09-16T05:16:53.536 INFO:tasks.rados.rados.2.smithi204.stdout:1550: expect (ObjNum 545 snap 163 seq_num 545) 2024-09-16T05:16:53.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:16:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:53.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:53.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:16:53.646 INFO:tasks.rados.rados.2.smithi204.stdout:1552: finishing write tid 3 to smithi20457400-37 2024-09-16T05:16:53.646 INFO:tasks.rados.rados.2.smithi204.stdout:1552: finishing write tid 4 to smithi20457400-37 2024-09-16T05:16:53.646 INFO:tasks.rados.rados.2.smithi204.stdout:1552: finishing write tid 5 to smithi20457400-37 2024-09-16T05:16:53.646 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 575 (ObjNum 555 snap 166 seq_num 555) dirty exists 2024-09-16T05:16:53.646 INFO:tasks.rados.rados.2.smithi204.stdout:1552: left oid 37 (ObjNum 555 snap 166 seq_num 555) 2024-09-16T05:16:53.646 INFO:tasks.rados.rados.2.smithi204.stdout:1550: done (3 left) 2024-09-16T05:16:53.646 INFO:tasks.rados.rados.2.smithi204.stdout:1551: done (2 left) 2024-09-16T05:16:53.646 INFO:tasks.rados.rados.2.smithi204.stdout:1552: done (1 left) 2024-09-16T05:16:53.647 INFO:tasks.rados.rados.2.smithi204.stdout:1554: write (excl) oid 5 current snap is 166 2024-09-16T05:16:53.647 INFO:tasks.rados.rados.2.smithi204.stdout:1554: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-09-16T05:16:53.652 INFO:tasks.rados.rados.2.smithi204.stdout:1554: writing smithi20457400-5 from 544692 to 1276905 tid 1 2024-09-16T05:16:53.659 INFO:tasks.rados.rados.2.smithi204.stdout:1554: writing smithi20457400-5 from 1985753 to 2729616 tid 2 2024-09-16T05:16:53.663 INFO:tasks.rados.rados.2.smithi204.stdout:1554: writing smithi20457400-5 from 3335184 to 3344692 tid 3 2024-09-16T05:16:53.663 INFO:tasks.rados.rados.2.smithi204.stdout:1555: snap_create 2024-09-16T05:16:53.663 INFO:tasks.rados.rados.2.smithi204.stdout:1553: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:16:53.663 INFO:tasks.rados.rados.2.smithi204.stdout:1553: finishing copy_from to smithi20457400-20 2024-09-16T05:16:53.663 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 351 (ObjNum 480 snap 151 seq_num 480) dirty exists 2024-09-16T05:16:53.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:53.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:53.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:53.714 INFO:tasks.rados.rados.2.smithi204.stdout:1554: finishing write tid 1 to smithi20457400-5 2024-09-16T05:16:53.714 INFO:tasks.rados.rados.2.smithi204.stdout:1554: finishing write tid 2 to smithi20457400-5 2024-09-16T05:16:53.714 INFO:tasks.rados.rados.2.smithi204.stdout:1554: finishing write tid 3 to smithi20457400-5 2024-09-16T05:16:53.714 INFO:tasks.rados.rados.2.smithi204.stdout:1554: finishing write tid 4 to smithi20457400-5 2024-09-16T05:16:53.714 INFO:tasks.rados.rados.2.smithi204.stdout:1554: finishing write tid 5 to smithi20457400-5 2024-09-16T05:16:53.714 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 339 (ObjNum 556 snap 166 seq_num 556) dirty exists 2024-09-16T05:16:53.714 INFO:tasks.rados.rados.2.smithi204.stdout:1554: left oid 5 (ObjNum 556 snap 166 seq_num 556) 2024-09-16T05:16:53.767 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:53.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:16:53.837 INFO:tasks.rados.rados.2.smithi204.stdout:1553: done (2 left) 2024-09-16T05:16:53.837 INFO:tasks.rados.rados.2.smithi204.stdout:1554: done (1 left) 2024-09-16T05:16:53.837 INFO:tasks.rados.rados.2.smithi204.stdout:1555: done (0 left) 2024-09-16T05:16:53.837 INFO:tasks.rados.rados.2.smithi204.stdout:1556: read oid 24 snap -1 2024-09-16T05:16:53.837 INFO:tasks.rados.rados.2.smithi204.stdout:1556: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:16:53.837 INFO:tasks.rados.rados.2.smithi204.stdout:1557: write oid 25 current snap is 167 2024-09-16T05:16:53.837 INFO:tasks.rados.rados.2.smithi204.stdout:1557: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-09-16T05:16:53.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:53.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:53.842 INFO:tasks.rados.rados.2.smithi204.stdout:1557: writing smithi20457400-25 from 561499 to 968961 tid 1 2024-09-16T05:16:53.848 INFO:tasks.rados.rados.2.smithi204.stdout:1557: writing smithi20457400-25 from 1527882 to 2015403 tid 2 2024-09-16T05:16:53.855 INFO:tasks.rados.rados.2.smithi204.stdout:1557: writing smithi20457400-25 from 2729901 to 3287118 tid 3 2024-09-16T05:16:53.856 INFO:tasks.rados.rados.2.smithi204.stdout:1558: snap_remove snap 165 2024-09-16T05:16:53.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:53.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:53.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:54.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:54.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:54.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:54.096 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:53 smithi196 ceph-mon[30802]: pgmap v855: 107 pgs: 107 active+clean; 96 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-16T05:16:54.096 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:53 smithi196 ceph-mon[30802]: osdmap e463: 12 total, 12 up, 12 in 2024-09-16T05:16:54.096 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2216292893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:53 smithi032 ceph-mon[27388]: pgmap v855: 107 pgs: 107 active+clean; 96 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-16T05:16:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:53 smithi032 ceph-mon[27388]: osdmap e463: 12 total, 12 up, 12 in 2024-09-16T05:16:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2216292893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:53 smithi204 ceph-mon[30933]: pgmap v855: 107 pgs: 107 active+clean; 96 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-16T05:16:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:53 smithi204 ceph-mon[30933]: osdmap e463: 12 total, 12 up, 12 in 2024-09-16T05:16:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2216292893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:16:54.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:54.204 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:54.204 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:54.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:54.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:54.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:54.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:54.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:54.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:16:54.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:54.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:54.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:54.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:54.582 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:54.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:54.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:54.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:54.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:54.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:16:54.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:54.842 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T05:16:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1558: done (2 left) 2024-09-16T05:16:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1559: snap_remove snap 164 2024-09-16T05:16:54.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:54.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:54.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:54.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:55.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:55.053 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:55.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:55.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:55.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:55.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:54 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:55.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2216292893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:55.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:54 smithi032 ceph-mon[27388]: osdmap e464: 12 total, 12 up, 12 in 2024-09-16T05:16:55.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/617407237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:16:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:54 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2216292893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:54 smithi204 ceph-mon[30933]: osdmap e464: 12 total, 12 up, 12 in 2024-09-16T05:16:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/617407237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:16:55.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:54 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:16:55.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2216292893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:16:55.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:54 smithi196 ceph-mon[30802]: osdmap e464: 12 total, 12 up, 12 in 2024-09-16T05:16:55.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/617407237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:16:55.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:55.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:55.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:55.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:55.377 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:55.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:55.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:55.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:55.509 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:16:55.509 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:16:55.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T05:16:55.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:55.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:55.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:55.731 INFO:tasks.rados.rados.2.smithi204.stdout:1559: done (2 left) 2024-09-16T05:16:55.731 INFO:tasks.rados.rados.2.smithi204.stdout:1560: snap_remove snap 166 2024-09-16T05:16:55.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:55.758 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:55.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:55.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:55.864 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:16:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:16:55.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:55.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:55.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:55.887 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:16:55.888 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:16:55.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:16:55.909 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:16:55.923 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:55.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:16:55.931 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:16:55.960 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:16:55.960 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:16:55.982 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:16:55.982 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:16:55.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:56.004 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:16:56.004 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:16:56.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:56.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:56.025 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:16:56.025 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:16:56.046 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:16:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:16:56.068 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:16:56.069 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:16:56.092 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:16:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:16:56.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:55 smithi032 ceph-mon[27388]: pgmap v858: 107 pgs: 107 active+clean; 104 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 23 op/s 2024-09-16T05:16:56.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/617407237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:16:56.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:55 smithi032 ceph-mon[27388]: osdmap e465: 12 total, 12 up, 12 in 2024-09-16T05:16:56.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:55 smithi032 ceph-mon[27388]: osdmap e466: 12 total, 12 up, 12 in 2024-09-16T05:16:56.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:55 smithi196 ceph-mon[30802]: pgmap v858: 107 pgs: 107 active+clean; 104 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 23 op/s 2024-09-16T05:16:56.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/617407237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:16:56.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:55 smithi196 ceph-mon[30802]: osdmap e465: 12 total, 12 up, 12 in 2024-09-16T05:16:56.099 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:55 smithi196 ceph-mon[30802]: osdmap e466: 12 total, 12 up, 12 in 2024-09-16T05:16:56.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:56.113 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:16:56.114 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:16:56.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:16:56.136 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:56.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:55 smithi204 ceph-mon[30933]: pgmap v858: 107 pgs: 107 active+clean; 104 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.6 MiB/s wr, 23 op/s 2024-09-16T05:16:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/617407237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:16:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:55 smithi204 ceph-mon[30933]: osdmap e465: 12 total, 12 up, 12 in 2024-09-16T05:16:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:55 smithi204 ceph-mon[30933]: osdmap e466: 12 total, 12 up, 12 in 2024-09-16T05:16:56.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:56.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:56.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:56.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:56.371 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:56.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:56.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:56.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:56.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:16:56.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:56.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:16:56.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:16:56.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:56.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:56.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:56.735 INFO:tasks.rados.rados.2.smithi204.stdout:1560: done (2 left) 2024-09-16T05:16:56.735 INFO:tasks.rados.rados.2.smithi204.stdout:1561: delete oid 43 current snap is 167 2024-09-16T05:16:56.735 INFO:tasks.rados.rados.2.smithi204.stdout:1557: finishing write tid 1 to smithi20457400-25 2024-09-16T05:16:56.735 INFO:tasks.rados.rados.2.smithi204.stdout:1556: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:16:56.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:56.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:56.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:56.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1557: finishing write tid 2 to smithi20457400-25 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1557: finishing write tid 3 to smithi20457400-25 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1557: finishing write tid 4 to smithi20457400-25 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1557: finishing write tid 5 to smithi20457400-25 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 474 (ObjNum 557 snap 167 seq_num 557) dirty exists 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1557: left oid 25 (ObjNum 557 snap 167 seq_num 557) 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1556: done (2 left) 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1557: done (1 left) 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1561: done (0 left) 2024-09-16T05:16:56.839 INFO:tasks.rados.rados.2.smithi204.stdout:1562: copy_from oid 29 from oid 43 current snap is 167 2024-09-16T05:16:56.840 INFO:tasks.rados.rados.2.smithi204.stdout:1563: read oid 42 snap -1 2024-09-16T05:16:56.840 INFO:tasks.rados.rados.2.smithi204.stdout:1563: expect (ObjNum 528 snap 159 seq_num 528) 2024-09-16T05:16:56.840 INFO:tasks.rados.rados.2.smithi204.stdout:1564: write (excl) oid 47 current snap is 167 2024-09-16T05:16:56.840 INFO:tasks.rados.rados.2.smithi204.stdout:1564: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-09-16T05:16:56.845 INFO:tasks.rados.rados.2.smithi204.stdout:1564: writing smithi20457400-47 from 595113 to 1153073 tid 1 2024-09-16T05:16:56.850 INFO:tasks.rados.rados.2.smithi204.stdout:1564: writing smithi20457400-47 from 1644846 to 2136036 tid 2 2024-09-16T05:16:56.858 INFO:tasks.rados.rados.2.smithi204.stdout:1564: writing smithi20457400-47 from 2784747 to 3395113 tid 3 2024-09-16T05:16:56.858 INFO:tasks.rados.rados.2.smithi204.stdout:1565: rollback oid 12 current snap is 167 2024-09-16T05:16:56.858 INFO:tasks.rados.rados.2.smithi204.stdout:1565: done (3 left) 2024-09-16T05:16:56.858 INFO:tasks.rados.rados.2.smithi204.stdout:1566: write (excl) oid 38 current snap is 167 2024-09-16T05:16:56.858 INFO:tasks.rados.rados.2.smithi204.stdout:1562: finishing copy_from to smithi20457400-29 2024-09-16T05:16:56.858 INFO:tasks.rados.rados.2.smithi204.stdout:1562: got expected ENOENT (src dne) 2024-09-16T05:16:56.858 INFO:tasks.rados.rados.2.smithi204.stdout:1566: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-09-16T05:16:56.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:56.864 INFO:tasks.rados.rados.2.smithi204.stdout:1566: writing smithi20457400-38 from 611920 to 1245129 tid 1 2024-09-16T05:16:56.870 INFO:tasks.rados.rados.2.smithi204.stdout:1566: writing smithi20457400-38 from 1703328 to 2338176 tid 2 2024-09-16T05:16:56.876 INFO:tasks.rados.rados.2.smithi204.stdout:1566: writing smithi20457400-38 from 2812170 to 3319556 tid 3 2024-09-16T05:16:56.877 INFO:tasks.rados.rados.2.smithi204.stdout:1562: finishing copy_from racing read to smithi20457400-29 2024-09-16T05:16:56.877 INFO:tasks.rados.rados.2.smithi204.stdout:1563: expect (ObjNum 528 snap 159 seq_num 528) 2024-09-16T05:16:56.878 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:56.878 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:16:56.878 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:16:56.878 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:16:56.878 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:56.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:56.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:56.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:16:57.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:57.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:16:57.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2113476594' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:57.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:56 smithi032 ceph-mon[27388]: osdmap e467: 12 total, 12 up, 12 in 2024-09-16T05:16:57.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:57.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:57.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:57.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:57.140 INFO:tasks.rados.rados.2.smithi204.stdout:1564: finishing write tid 1 to smithi20457400-47 2024-09-16T05:16:57.140 INFO:tasks.rados.rados.2.smithi204.stdout:1564: finishing write tid 2 to smithi20457400-47 2024-09-16T05:16:57.140 INFO:tasks.rados.rados.2.smithi204.stdout:1564: finishing write tid 3 to smithi20457400-47 2024-09-16T05:16:57.140 INFO:tasks.rados.rados.2.smithi204.stdout:1566: finishing write tid 1 to smithi20457400-38 2024-09-16T05:16:57.140 INFO:tasks.rados.rados.2.smithi204.stdout:1566: finishing write tid 2 to smithi20457400-38 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1566: finishing write tid 3 to smithi20457400-38 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1566: finishing write tid 4 to smithi20457400-38 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1566: finishing write tid 5 to smithi20457400-38 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 682 (ObjNum 560 snap 167 seq_num 560) dirty exists 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1566: left oid 38 (ObjNum 560 snap 167 seq_num 560) 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1562: done (3 left) 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1563: done (2 left) 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1566: done (1 left) 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1567: copy_from oid 7 from oid 12 current snap is 167 2024-09-16T05:16:57.141 INFO:tasks.rados.rados.2.smithi204.stdout:1568: snap_create 2024-09-16T05:16:57.148 INFO:tasks.rados.rados.2.smithi204.stdout:1567: finishing copy_from racing read to smithi20457400-7 2024-09-16T05:16:57.152 INFO:tasks.rados.rados.2.smithi204.stdout:1567: finishing copy_from to smithi20457400-7 2024-09-16T05:16:57.152 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 326 (ObjNum 440 snap 131 seq_num 440) dirty exists 2024-09-16T05:16:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2113476594' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:56 smithi204 ceph-mon[30933]: osdmap e467: 12 total, 12 up, 12 in 2024-09-16T05:16:57.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2113476594' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:57.181 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:56 smithi196 ceph-mon[30802]: osdmap e467: 12 total, 12 up, 12 in 2024-09-16T05:16:57.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:57.249 INFO:tasks.rados.rados.2.smithi204.stdout:1564: finishing write tid 4 to smithi20457400-47 2024-09-16T05:16:57.250 INFO:tasks.rados.rados.2.smithi204.stdout:1564: finishing write tid 5 to smithi20457400-47 2024-09-16T05:16:57.250 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 579 (ObjNum 559 snap 167 seq_num 559) dirty exists 2024-09-16T05:16:57.250 INFO:tasks.rados.rados.2.smithi204.stdout:1564: left oid 47 (ObjNum 559 snap 167 seq_num 559) 2024-09-16T05:16:57.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:57.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:57.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:57.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:57.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:57.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:57.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:16:57.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:57.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:57.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:57.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:57.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:57.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:57.768 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:16:57.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:57.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:57.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:57.852 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:16:57.852 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T05:16:57.862 INFO:tasks.rados.rados.2.smithi204.stdout:1564: done (2 left) 2024-09-16T05:16:57.862 INFO:tasks.rados.rados.2.smithi204.stdout:1567: done (1 left) 2024-09-16T05:16:57.862 INFO:tasks.rados.rados.2.smithi204.stdout:1568: done (0 left) 2024-09-16T05:16:57.862 INFO:tasks.rados.rados.2.smithi204.stdout:1569: write oid 37 current snap is 168 2024-09-16T05:16:57.862 INFO:tasks.rados.rados.2.smithi204.stdout:1569: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-09-16T05:16:57.869 INFO:tasks.rados.rados.2.smithi204.stdout:1569: writing smithi20457400-37 from 628727 to 1337185 tid 1 2024-09-16T05:16:57.876 INFO:tasks.rados.rados.2.smithi204.stdout:1569: writing smithi20457400-37 from 2045457 to 2540316 tid 2 2024-09-16T05:16:57.881 INFO:tasks.rados.rados.2.smithi204.stdout:1569: writing smithi20457400-37 from 3123240 to 3428727 tid 3 2024-09-16T05:16:57.881 INFO:tasks.rados.rados.2.smithi204.stdout:1570: write oid 26 current snap is 168 2024-09-16T05:16:57.882 INFO:tasks.rados.rados.2.smithi204.stdout:1569: finishing write tid 1 to smithi20457400-37 2024-09-16T05:16:57.882 INFO:tasks.rados.rados.2.smithi204.stdout:1569: finishing write tid 2 to smithi20457400-37 2024-09-16T05:16:57.882 INFO:tasks.rados.rados.2.smithi204.stdout:1570: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-09-16T05:16:57.888 INFO:tasks.rados.rados.2.smithi204.stdout:1570: writing smithi20457400-26 from 645534 to 1429241 tid 1 2024-09-16T05:16:57.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:57.895 INFO:tasks.rados.rados.2.smithi204.stdout:1570: writing smithi20457400-26 from 2103939 to 2742456 tid 2 2024-09-16T05:16:57.900 INFO:tasks.rados.rados.2.smithi204.stdout:1570: writing smithi20457400-26 from 3150663 to 3445534 tid 3 2024-09-16T05:16:57.900 INFO:tasks.rados.rados.2.smithi204.stdout:1571: read oid 42 snap 167 2024-09-16T05:16:57.900 INFO:tasks.rados.rados.2.smithi204.stdout:1571: expect (ObjNum 528 snap 159 seq_num 528) 2024-09-16T05:16:57.900 INFO:tasks.rados.rados.2.smithi204.stdout:1572: rollback oid 9 current snap is 168 2024-09-16T05:16:57.900 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 9 to 167 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1569: finishing write tid 3 to smithi20457400-37 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1573: copy_from oid 33 from oid 47 current snap is 168 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1569: finishing write tid 4 to smithi20457400-37 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1569: finishing write tid 5 to smithi20457400-37 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 583 (ObjNum 561 snap 168 seq_num 561) dirty exists 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1569: left oid 37 (ObjNum 561 snap 168 seq_num 561) 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1569: done (4 left) 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1574: copy_from oid 36 from oid 6 current snap is 168 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1575: delete oid 41 current snap is 168 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:1572: finishing rollback tid 1 to smithi20457400-9 2024-09-16T05:16:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 33 snap 0 seq_num 2969341456) dirty dne 2024-09-16T05:16:57.902 INFO:tasks.rados.rados.2.smithi204.stdout:1572: done (5 left) 2024-09-16T05:16:57.902 INFO:tasks.rados.rados.2.smithi204.stdout:1575: done (4 left) 2024-09-16T05:16:57.902 INFO:tasks.rados.rados.2.smithi204.stdout:1576: delete oid 49 current snap is 168 2024-09-16T05:16:57.902 INFO:tasks.rados.rados.2.smithi204.stdout:1576: done (4 left) 2024-09-16T05:16:57.903 INFO:tasks.rados.rados.2.smithi204.stdout:1577: write (excl) oid 49 current snap is 168 2024-09-16T05:16:57.903 INFO:tasks.rados.rados.2.smithi204.stdout:1577: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-09-16T05:16:57.911 INFO:tasks.rados.rados.2.smithi204.stdout:1577: writing smithi20457400-49 from 695955 to 1421762 tid 1 2024-09-16T05:16:57.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:57.917 INFO:tasks.rados.rados.2.smithi204.stdout:1577: writing smithi20457400-49 from 1879385 to 2381582 tid 2 2024-09-16T05:16:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:57.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:57.922 INFO:tasks.rados.rados.2.smithi204.stdout:1577: writing smithi20457400-49 from 2949285 to 3429384 tid 3 2024-09-16T05:16:57.923 INFO:tasks.rados.rados.2.smithi204.stdout:1578: delete oid 48 current snap is 168 2024-09-16T05:16:57.923 INFO:tasks.rados.rados.2.smithi204.stdout:1573: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:16:57.923 INFO:tasks.rados.rados.2.smithi204.stdout:1573: finishing copy_from to smithi20457400-33 2024-09-16T05:16:57.923 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 579 (ObjNum 559 snap 167 seq_num 559) dirty exists 2024-09-16T05:16:57.923 INFO:tasks.rados.rados.2.smithi204.stdout:1571: expect (ObjNum 528 snap 159 seq_num 528) 2024-09-16T05:16:58.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:58.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:58.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:58.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:16:58.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:58.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:58.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:16:58.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:57 smithi196 ceph-mon[30802]: pgmap v861: 107 pgs: 107 active+clean; 107 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-16T05:16:58.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2367944920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:16:58.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/834329463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:57 smithi032 ceph-mon[27388]: pgmap v861: 107 pgs: 107 active+clean; 107 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-16T05:16:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2367944920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:16:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/834329463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:57 smithi204 ceph-mon[30933]: pgmap v861: 107 pgs: 107 active+clean; 107 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2024-09-16T05:16:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2367944920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:16:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/834329463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:16:58.188 INFO:tasks.rados.rados.2.smithi204.stdout:1577: finishing write tid 1 to smithi20457400-49 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1574: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1577: finishing write tid 2 to smithi20457400-49 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1574: finishing copy_from to smithi20457400-36 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 338 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1577: finishing write tid 3 to smithi20457400-49 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1571: done (5 left) 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1573: done (4 left) 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1574: done (3 left) 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1578: done (2 left) 2024-09-16T05:16:58.189 INFO:tasks.rados.rados.2.smithi204.stdout:1579: snap_remove snap 167 2024-09-16T05:16:58.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:23 2.0 2:06458cf1:::smithi20457400-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 375.215186641, 2024-09-16T05:16:58.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103108687, 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 23 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:16:58.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052186+0000", 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3599999999999999e-07 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052188+0000", 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4169999999999999e-06 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052214+0000", 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5327e-05 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052217+0000", 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1810000000000001e-06 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052730+0000", 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051270700000000005 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052818+0000", 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8641999999999996e-05 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 10", 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052937+0000", 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119009 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053259+0000", 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032151800000000002 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155207+0000", 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.101948155 2024-09-16T05:16:58.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155248+0000", 2024-09-16T05:16:58.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1115999999999997e-05 2024-09-16T05:16:58.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155294+0000", 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6179e-05 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:24 2.0 2:06458cf1:::smithi20457400-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 375.21515058, 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103238884, 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.277 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 24 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8799999999999998e-07 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052223+0000", 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2950000000000001e-06 2024-09-16T05:16:58.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052227+0000", 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5240000000000001e-06 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052230+0000", 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2160000000000002e-06 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053127+0000", 2024-09-16T05:16:58.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089734199999999997 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053182+0000", 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4935999999999997e-05 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155331+0000", 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.102149077 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155389+0000", 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7691e-05 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155460+0000", 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1514999999999995e-05 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:391 2.8 2:1030fb5b:::smithi20457400-27:head [copy-from ver 13] snapc 8=[8,7] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 362.63058525999998, 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021838053999999999, 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 391 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.283 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636787+0000", 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636789+0000", 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3829999999999999e-06 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636801+0000", 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1459e-05 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636822+0000", 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1719e-05 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636833+0000", 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0366e-05 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636912+0000", 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9060999999999994e-05 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636918+0000", 2024-09-16T05:16:58.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0900000000000001e-06 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638224+0000", 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00130613 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638314+0000", 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9771000000000005e-05 2024-09-16T05:16:58.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.652220+0000", 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013906112 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.656458+0000", 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004237673 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658553+0000", 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002095609 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658592+0000", 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8627000000000001e-05 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.658625+0000", 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3053999999999999e-05 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:392 2.8 2:1030fb5b:::smithi20457400-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e124)", 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 362.630514038, 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019221338000000001, 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.290 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 392 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636858+0000", 2024-09-16T05:16:58.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636859+0000", 2024-09-16T05:16:58.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0699999999999998e-07 2024-09-16T05:16:58.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636861+0000", 2024-09-16T05:16:58.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5129999999999999e-06 2024-09-16T05:16:58.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636867+0000", 2024-09-16T05:16:58.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6030000000000004e-06 2024-09-16T05:16:58.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636870+0000", 2024-09-16T05:16:58.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.551e-06 2024-09-16T05:16:58.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636972+0000", 2024-09-16T05:16:58.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010187999999999999 2024-09-16T05:16:58.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:16:58.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.636978+0000", 2024-09-16T05:16:58.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6269999999999997e-06 2024-09-16T05:16:58.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638477+0000", 2024-09-16T05:16:58.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014995049999999999 2024-09-16T05:16:58.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:16:58.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.638531+0000", 2024-09-16T05:16:58.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3387999999999998e-05 2024-09-16T05:16:58.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.655845+0000", 2024-09-16T05:16:58.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017313963000000002 2024-09-16T05:16:58.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.655956+0000", 2024-09-16T05:16:58.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011097799999999999 2024-09-16T05:16:58.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:55.656079+0000", 2024-09-16T05:16:58.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012392300000000001 2024-09-16T05:16:58.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.302 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.303 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1675 2.1c 2:3d4bedc8:::smithi20457400-22:head [setxattr _header (57) in=7b,truncate 2151329] snapc 46=[46,45,44,42,41,40,3f,3e,3c,3a,39,37,35,33,27] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-16T05:16:58.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:16:58.303 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 251.407752883, 2024-09-16T05:16:58.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22736652099999999, 2024-09-16T05:16:58.303 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1675 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859620+0000", 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3099999999999997e-07 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859624+0000", 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1949999999999996e-06 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859637+0000", 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3118999999999999e-05 2024-09-16T05:16:58.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859641+0000", 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.036e-06 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860356+0000", 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00071436999999999998 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860417+0000", 2024-09-16T05:16:58.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1253999999999993e-05 2024-09-16T05:16:58.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 8", 2024-09-16T05:16:58.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860525+0000", 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107742 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860988+0000", 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046300399999999999 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086882+0000", 2024-09-16T05:16:58.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.225894659 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086930+0000", 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8010999999999998e-05 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086986+0000", 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5500000000000001e-05 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.311 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.312 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.312 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1676 2.1c 2:3d4bedc8:::smithi20457400-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e235)", 2024-09-16T05:16:58.312 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:16:58.312 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 251.40770938099999, 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.227504338, 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1676 2024-09-16T05:16:58.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9999999999999998e-07 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859666+0000", 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7819999999999998e-06 2024-09-16T05:16:58.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859699+0000", 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2950000000000001e-05 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859707+0000", 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3459999999999994e-06 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860761+0000", 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001053315 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860862+0000", 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010108 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087019+0000", 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.226157467 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087091+0000", 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1536000000000001e-05 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087167+0000", 2024-09-16T05:16:58.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6361999999999998e-05 2024-09-16T05:16:58.319 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.319 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.319 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.319 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:1675 2.1c e235/232 ondisk, result = 0)", 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 251.406353739, 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22597392799999999, 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861019+0000", 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861020+0000", 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7969999999999999e-06 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086793+0000", 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22577269899999999 2024-09-16T05:16:58.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086803+0000", 2024-09-16T05:16:58.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5459999999999997e-06 2024-09-16T05:16:58.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086867+0000", 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4447000000000001e-05 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086876+0000", 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9760000000000001e-06 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086992+0000", 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116212 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1824 2.8 2:1030fb5b:::smithi20457400-27:head [write 521083~714336,stat] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 239.32222476999999, 2024-09-16T05:16:58.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.81065307900000005, 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1824 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.326 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945150+0000", 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1260000000000001e-06 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945906+0000", 2024-09-16T05:16:58.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075638699999999997 2024-09-16T05:16:58.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945927+0000", 2024-09-16T05:16:58.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0849e-05 2024-09-16T05:16:58.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945936+0000", 2024-09-16T05:16:58.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3910000000000004e-06 2024-09-16T05:16:58.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946023+0000", 2024-09-16T05:16:58.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6547999999999999e-05 2024-09-16T05:16:58.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:58.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946035+0000", 2024-09-16T05:16:58.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2249000000000001e-05 2024-09-16T05:16:58.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752337+0000", 2024-09-16T05:16:58.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80630184599999999 2024-09-16T05:16:58.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752493+0000", 2024-09-16T05:16:58.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000156546 2024-09-16T05:16:58.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:16:58.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752675+0000", 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018172699999999999 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755628+0000", 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029533929999999999 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755689+0000", 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0949000000000002e-05 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755745+0000", 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5591999999999999e-05 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755800+0000", 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5476000000000002e-05 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1825 2.8 2:1030fb5b:::smithi20457400-27:head [write 1724715~555421] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 239.316579222, 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80511571699999995, 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1825 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.337 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950794+0000", 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951237+0000", 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00044250000000000002 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951251+0000", 2024-09-16T05:16:58.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4381e-05 2024-09-16T05:16:58.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951276+0000", 2024-09-16T05:16:58.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5233999999999999e-05 2024-09-16T05:16:58.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951378+0000", 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010173799999999999 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951390+0000", 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2274e-05 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.753903+0000", 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80251229300000004 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754002+0000", 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9427999999999997e-05 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754132+0000", 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130207 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755633+0000", 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015010100000000001 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755844+0000", 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021044799999999999 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755880+0000", 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6007999999999998e-05 2024-09-16T05:16:58.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755909+0000", 2024-09-16T05:16:58.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8941e-05 2024-09-16T05:16:58.344 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.344 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.344 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1826 2.8 2:1030fb5b:::smithi20457400-27:head [setxattr _header (57) in=7b,truncate 2521083] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 239.31607992900001, 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80476365000000005, 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1826 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951293+0000", 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951296+0000", 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2799999999999999e-06 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951301+0000", 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6879999999999998e-06 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951309+0000", 2024-09-16T05:16:58.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7810000000000005e-06 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951421+0000", 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112333 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951426+0000", 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6650000000000002e-06 2024-09-16T05:16:58.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754913+0000", 2024-09-16T05:16:58.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80348719099999999 2024-09-16T05:16:58.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754982+0000", 2024-09-16T05:16:58.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9558e-05 2024-09-16T05:16:58.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:16:58.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755084+0000", 2024-09-16T05:16:58.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101894 2024-09-16T05:16:58.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:58.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755638+0000", 2024-09-16T05:16:58.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055423099999999997 2024-09-16T05:16:58.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:58.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755988+0000", 2024-09-16T05:16:58.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034971600000000002 2024-09-16T05:16:58.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756021+0000", 2024-09-16T05:16:58.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2461e-05 2024-09-16T05:16:58.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756056+0000", 2024-09-16T05:16:58.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5278999999999999e-05 2024-09-16T05:16:58.356 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.356 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.356 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.356 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1827 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:16:58.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:16:58.357 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 239.31605766000001, 2024-09-16T05:16:58.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.804892304, 2024-09-16T05:16:58.357 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.357 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.358 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.358 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1827 2024-09-16T05:16:58.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.359 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:16:58.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:16:58.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:16:58.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8700000000000002e-07 2024-09-16T05:16:58.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951317+0000", 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7460000000000001e-06 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951321+0000", 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3250000000000001e-06 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951325+0000", 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1710000000000004e-06 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951464+0000", 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139111 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951475+0000", 2024-09-16T05:16:58.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1045e-05 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755294+0000", 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80381846999999995 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755351+0000", 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7589e-05 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756086+0000", 2024-09-16T05:16:58.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073505599999999999 2024-09-16T05:16:58.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756138+0000", 2024-09-16T05:16:58.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1397000000000003e-05 2024-09-16T05:16:58.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756207+0000", 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9107000000000001e-05 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1834 2.8 2:12e92f6c:::smithi20457400-1:head [stat,delete] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 239.069156978, 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55807496000000001, 2024-09-16T05:16:58.366 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1834 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198218+0000", 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.357e-06 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198229+0000", 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1677e-05 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198254+0000", 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4312e-05 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198266+0000", 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2687000000000001e-05 2024-09-16T05:16:58.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198366+0000", 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9698000000000007e-05 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198378+0000", 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2106000000000001e-05 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755379+0000", 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55700064400000004 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755448+0000", 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8848000000000002e-05 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756290+0000", 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084263100000000004 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.9,QREQUEST,e:256,from:1,reservation_nonce:2)", 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:16:58.372 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 229.96713554300001, 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21542001499999999, 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300239+0000", 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0540000000000002e-06 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300249+0000", 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0369e-05 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300304+0000", 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5247000000000001e-05 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300312+0000", 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.089e-06 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300391+0000", 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8628999999999995e-05 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300397+0000", 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8889999999999999e-06 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515506+0000", 2024-09-16T05:16:58.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21510901199999999 2024-09-16T05:16:58.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515527+0000", 2024-09-16T05:16:58.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0981000000000001e-05 2024-09-16T05:16:58.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515657+0000", 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012974500000000001 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2255 2.1d e290/282 2:ba754fcb:::smithi20457400-30:head v 290'477, mlcod=284'471)", 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 196.582562231, 2024-09-16T05:16:58.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53570759499999998, 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:16:58.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684811+0000", 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3600000000000002e-07 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684819+0000", 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5760000000000006e-06 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219817+0000", 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53499761300000004 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219843+0000", 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5908e-05 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219921+0000", 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7811000000000004e-05 2024-09-16T05:16:58.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219944+0000", 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3237000000000001e-05 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220484+0000", 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00053966300000000001 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220518+0000", 2024-09-16T05:16:58.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3850999999999998e-05 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.1d,QREQUEST,e:290,from:10,reservation_nonce:2)", 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 196.04758319199999, 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.202128169, 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.383 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219790+0000", 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.452e-06 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219798+0000", 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1670000000000002e-06 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421791+0000", 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20199293700000001 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421799+0000", 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5560000000000008e-06 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421864+0000", 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5235999999999997e-05 2024-09-16T05:16:58.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421881+0000", 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6742e-05 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421917+0000", 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6078999999999998e-05 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.388 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.10.0:85 2.1d e300/291 2:ba754fcb:::smithi20457400-30:5f v 300'498, mlcod=300'494)", 2024-09-16T05:16:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:58.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:58.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:58.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:16:58.390 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 186.50873558800001, 2024-09-16T05:16:58.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.51609034499999995, 2024-09-16T05:16:58.390 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.390 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.390 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.390 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.16e-07 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758644+0000", 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1519999999999997e-06 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273721+0000", 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.51507724799999999 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273736+0000", 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5449000000000001e-05 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273906+0000", 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016944 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273962+0000", 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6209999999999999e-05 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.274683+0000", 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072146099999999996 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.274727+0000", 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3569000000000002e-05 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2580 2.3c 2:3d4bedc8:::smithi20457400-22:head [write 2010607~258404] snapc 6e=[6e,6c,6b,6a,69,67,66,64,63,62,60,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:45.710412+0000", 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 132.55695974299999, 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018965652, 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:16:58.396 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2580 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710412+0000", 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710412+0000", 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710422+0000", 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9159999999999996e-06 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710611+0000", 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000188653 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710640+0000", 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8975e-05 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710664+0000", 2024-09-16T05:16:58.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4017000000000001e-05 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710726+0000", 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1932000000000003e-05 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710832+0000", 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106316 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1", 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710990+0000", 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015727199999999999 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.712061+0000", 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010717249999999999 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:16:58.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.729262+0000", 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017200803000000001 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.729328+0000", 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6449e-05 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.729378+0000", 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9594e-05 2024-09-16T05:16:58.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2582 2.3c 2:3d4bedc8:::smithi20457400-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e336)", 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:45.710716+0000", 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 132.55665603, 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019021360000000001, 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2582 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710716+0000", 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710716+0000", 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710717+0000", 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1899999999999998e-07 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710719+0000", 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0420000000000001e-06 2024-09-16T05:16:58.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710725+0000", 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0279999999999997e-06 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.710729+0000", 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1189999999999999e-06 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.711986+0000", 2024-09-16T05:16:58.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001256847 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.712048+0000", 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1928999999999996e-05 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.729552+0000", 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017504552 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.729637+0000", 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4671000000000003e-05 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:45.729737+0000", 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100753 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.408 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:382 2.2e e345/278 ondisk, result = 0)", 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.871554+0000", 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 123.39581778500001, 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.29655793200000002, 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.871554+0000", 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.871554+0000", 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.871555+0000", 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6800000000000001e-07 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.871558+0000", 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4819999999999999e-06 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.167809+0000", 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.29625046300000002 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.167824+0000", 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5244e-05 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.167945+0000", 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012105 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.167965+0000", 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0183e-05 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.168112+0000", 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014704200000000001 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.412 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:390 2.3c e346/293 ondisk, result = 0)", 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 122.55979024200001, 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62913221100000005, 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707583+0000", 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.47e-07 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707587+0000", 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7010000000000004e-06 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336491+0000", 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62890340600000005 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336497+0000", 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4640000000000003e-06 2024-09-16T05:16:58.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336590+0000", 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2163000000000005e-05 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336599+0000", 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.054e-06 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:16:58.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:16:58.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336714+0000", 2024-09-16T05:16:58.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011557600000000001 2024-09-16T05:16:58.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.417 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:16:58.417 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:16:58.417 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:16:58.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:58.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:58.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:58.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:58.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:58.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:58.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T05:16:58.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:16:58.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:58.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:58.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:16:58.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:16:58.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:16:58.871 INFO:tasks.rados.rados.2.smithi204.stdout:1579: done (2 left) 2024-09-16T05:16:58.871 INFO:tasks.rados.rados.2.smithi204.stdout:1580: rollback oid 10 current snap is 168 2024-09-16T05:16:58.871 INFO:tasks.rados.rados.2.smithi204.stdout:1580: done (2 left) 2024-09-16T05:16:58.871 INFO:tasks.rados.rados.2.smithi204.stdout:1581: rollback oid 40 current snap is 168 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1581: done (2 left) 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1582: snap_create 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1577: finishing write tid 4 to smithi20457400-49 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1577: finishing write tid 5 to smithi20457400-49 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 555 (ObjNum 565 snap 168 seq_num 565) dirty exists 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1577: left oid 49 (ObjNum 565 snap 168 seq_num 565) 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1570: finishing write tid 1 to smithi20457400-26 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1570: finishing write tid 2 to smithi20457400-26 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1570: finishing write tid 3 to smithi20457400-26 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1570: finishing write tid 4 to smithi20457400-26 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:1570: finishing write tid 5 to smithi20457400-26 2024-09-16T05:16:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 790 (ObjNum 562 snap 168 seq_num 562) dirty exists 2024-09-16T05:16:58.873 INFO:tasks.rados.rados.2.smithi204.stdout:1570: left oid 26 (ObjNum 562 snap 168 seq_num 562) 2024-09-16T05:16:58.886 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:58.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:16:58.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2367944920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:16:58.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/834329463' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:58.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:58 smithi032 ceph-mon[27388]: osdmap e468: 12 total, 12 up, 12 in 2024-09-16T05:16:58.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:58 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:16:58.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:58 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:16:58.988 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:59.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:59.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:59.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:16:59.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:59.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2367944920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:16:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/834329463' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:58 smithi204 ceph-mon[30933]: osdmap e468: 12 total, 12 up, 12 in 2024-09-16T05:16:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:58 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:16:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:58 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:16:59.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2367944920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:16:59.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/834329463' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:16:59.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:58 smithi196 ceph-mon[30802]: osdmap e468: 12 total, 12 up, 12 in 2024-09-16T05:16:59.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:58 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:16:59.184 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:58 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:16:59.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:16:59.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:16:59.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:16:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:59.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:16:59.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:16:59.382 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:59.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:16:59.461 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:16:59.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:16:59.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:16:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:59.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:16:59.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:16:59.632 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:59.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:16:59.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:16:59.757 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:59.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:16:59.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:16:59.872 INFO:tasks.rados.rados.2.smithi204.stdout:1570: done (2 left) 2024-09-16T05:16:59.872 INFO:tasks.rados.rados.2.smithi204.stdout:1577: done (1 left) 2024-09-16T05:16:59.872 INFO:tasks.rados.rados.2.smithi204.stdout:1582: done (0 left) 2024-09-16T05:16:59.872 INFO:tasks.rados.rados.2.smithi204.stdout:1583: snap_create 2024-09-16T05:16:59.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:16:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:16:59.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:16:59.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:59 smithi032 ceph-mon[27388]: pgmap v864: 107 pgs: 107 active+clean; 105 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 21 op/s 2024-09-16T05:16:59.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:59 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:16:59.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:59 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:16:59.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:59 smithi032 ceph-mon[27388]: osdmap e469: 12 total, 12 up, 12 in 2024-09-16T05:16:59.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:59 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:16:59.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:59 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:16:59.892 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:16:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/899570060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:16:59.898 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:16:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:16:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:16:59.898 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:16:59.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:00.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:00.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:00.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:59 smithi204 ceph-mon[30933]: pgmap v864: 107 pgs: 107 active+clean; 105 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 21 op/s 2024-09-16T05:17:00.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:59 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:17:00.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:59 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:17:00.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:59 smithi204 ceph-mon[30933]: osdmap e469: 12 total, 12 up, 12 in 2024-09-16T05:17:00.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:59 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:17:00.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:59 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:17:00.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:16:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/899570060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:00.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:00.131 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:00.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:00.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:00.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:00.172 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:00.172 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:00.172 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:00.172 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:00.172 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:00.172 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:00.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:00.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:00.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:00.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:00.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:00.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:00.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:00.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:00.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:00.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:59 smithi196 ceph-mon[30802]: pgmap v864: 107 pgs: 107 active+clean; 105 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 21 op/s 2024-09-16T05:17:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:59 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:17:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:59 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:17:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:59 smithi196 ceph-mon[30802]: osdmap e469: 12 total, 12 up, 12 in 2024-09-16T05:17:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:59 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:17:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:59 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:17:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:16:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/899570060' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:00.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:00.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:00.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:00.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:00.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:00.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:00.506 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:00.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:00.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:00.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T05:17:00.730 INFO:tasks.rados.rados.2.smithi204.stdout:1583: done (0 left) 2024-09-16T05:17:00.730 INFO:tasks.rados.rados.2.smithi204.stdout:1584: read oid 22 snap 169 2024-09-16T05:17:00.730 INFO:tasks.rados.rados.2.smithi204.stdout:1584: expect deleted 2024-09-16T05:17:00.730 INFO:tasks.rados.rados.2.smithi204.stdout:1585: delete oid 28 current snap is 170 2024-09-16T05:17:00.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:00.734 INFO:tasks.rados.rados.2.smithi204.stdout:1584: done (1 left) 2024-09-16T05:17:00.735 INFO:tasks.rados.rados.2.smithi204.stdout:1585: done (0 left) 2024-09-16T05:17:00.735 INFO:tasks.rados.rados.2.smithi204.stdout:1586: write oid 45 current snap is 170 2024-09-16T05:17:00.735 INFO:tasks.rados.rados.2.smithi204.stdout:1586: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-09-16T05:17:00.742 INFO:tasks.rados.rados.2.smithi204.stdout:1586: writing smithi20457400-45 from 746376 to 1297930 tid 1 2024-09-16T05:17:00.751 INFO:tasks.rados.rados.2.smithi204.stdout:1586: writing smithi20457400-45 from 2054831 to 2704355 tid 2 2024-09-16T05:17:00.757 INFO:tasks.rados.rados.2.smithi204.stdout:1586: writing smithi20457400-45 from 3315201 to 3546376 tid 3 2024-09-16T05:17:00.757 INFO:tasks.rados.rados.2.smithi204.stdout:1587: write (excl) oid 41 current snap is 170 2024-09-16T05:17:00.757 INFO:tasks.rados.rados.2.smithi204.stdout:1586: finishing write tid 1 to smithi20457400-45 2024-09-16T05:17:00.757 INFO:tasks.rados.rados.2.smithi204.stdout:1586: finishing write tid 2 to smithi20457400-45 2024-09-16T05:17:00.757 INFO:tasks.rados.rados.2.smithi204.stdout:1587: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2024-09-16T05:17:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:00.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:00.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:00.765 INFO:tasks.rados.rados.2.smithi204.stdout:1587: writing smithi20457400-41 from 763183 to 1389986 tid 1 2024-09-16T05:17:00.775 INFO:tasks.rados.rados.2.smithi204.stdout:1587: writing smithi20457400-41 from 1996960 to 2790142 tid 2 2024-09-16T05:17:00.780 INFO:tasks.rados.rados.2.smithi204.stdout:1587: writing smithi20457400-41 from 3226271 to 3563183 tid 3 2024-09-16T05:17:00.780 INFO:tasks.rados.rados.2.smithi204.stdout:1588: snap_remove snap 169 2024-09-16T05:17:00.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:17:00.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:00.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:00.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:00.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:01.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:01.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:01.042 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:01.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:17:01.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:01.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:01.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:01.163 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: osdmap e470: 12 total, 12 up, 12 in 2024-09-16T05:17:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:17:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:17:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/298507143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:17:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/298507143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:17:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: osdmap e471: 12 total, 12 up, 12 in 2024-09-16T05:17:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:17:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:00 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:17:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: osdmap e470: 12 total, 12 up, 12 in 2024-09-16T05:17:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:17:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:17:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/298507143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:17:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/298507143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:17:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: osdmap e471: 12 total, 12 up, 12 in 2024-09-16T05:17:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:17:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:00 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:17:01.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:01.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: osdmap e470: 12 total, 12 up, 12 in 2024-09-16T05:17:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:17:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:17:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/298507143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:17:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/298507143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:17:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: osdmap e471: 12 total, 12 up, 12 in 2024-09-16T05:17:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:17:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:00 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:17:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:01.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:01.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:01.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:01.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:01.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:01.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:01.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:01.508 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:01.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:01.527 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:01.527 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:01.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:01.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:01.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:01.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-16T05:17:01.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:01.746 INFO:tasks.rados.rados.2.smithi204.stdout:1588: done (2 left) 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1589: rollback oid 18 current snap is 170 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 18 to 168 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1586: finishing write tid 3 to smithi20457400-45 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1586: finishing write tid 4 to smithi20457400-45 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1586: finishing write tid 5 to smithi20457400-45 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 343 (ObjNum 568 snap 170 seq_num 568) dirty exists 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1586: left oid 45 (ObjNum 568 snap 170 seq_num 568) 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1587: finishing write tid 1 to smithi20457400-41 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1587: finishing write tid 2 to smithi20457400-41 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1587: finishing write tid 3 to smithi20457400-41 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1587: finishing write tid 4 to smithi20457400-41 2024-09-16T05:17:01.747 INFO:tasks.rados.rados.2.smithi204.stdout:1587: finishing write tid 5 to smithi20457400-41 2024-09-16T05:17:01.748 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 689 (ObjNum 569 snap 170 seq_num 569) dirty exists 2024-09-16T05:17:01.748 INFO:tasks.rados.rados.2.smithi204.stdout:1587: left oid 41 (ObjNum 569 snap 170 seq_num 569) 2024-09-16T05:17:01.748 INFO:tasks.rados.rados.2.smithi204.stdout:1586: done (2 left) 2024-09-16T05:17:01.748 INFO:tasks.rados.rados.2.smithi204.stdout:1587: done (1 left) 2024-09-16T05:17:01.748 INFO:tasks.rados.rados.2.smithi204.stdout:1590: copy_from oid 41 from oid 13 current snap is 170 2024-09-16T05:17:01.748 INFO:tasks.rados.rados.2.smithi204.stdout:1591: copy_from oid 33 from oid 12 current snap is 170 2024-09-16T05:17:01.748 INFO:tasks.rados.rados.2.smithi204.stdout:1592: delete oid 4 current snap is 170 2024-09-16T05:17:01.753 INFO:tasks.rados.rados.2.smithi204.stdout:1589: finishing rollback tid 0 to smithi20457400-18 2024-09-16T05:17:01.753 INFO:tasks.rados.rados.2.smithi204.stdout:1589: finishing rollback tid 1 to smithi20457400-18 2024-09-16T05:17:01.753 INFO:tasks.rados.rados.2.smithi204.stdout:1589: finishing rollback tid 2 to smithi20457400-18 2024-09-16T05:17:01.754 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 316 (ObjNum 505 snap 153 seq_num 505) dirty exists 2024-09-16T05:17:01.754 INFO:tasks.rados.rados.2.smithi204.stdout:1589: done (3 left) 2024-09-16T05:17:01.754 INFO:tasks.rados.rados.2.smithi204.stdout:1592: done (2 left) 2024-09-16T05:17:01.754 INFO:tasks.rados.rados.2.smithi204.stdout:1593: rollback oid 45 current snap is 170 2024-09-16T05:17:01.754 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 168 2024-09-16T05:17:01.754 INFO:tasks.rados.rados.2.smithi204.stdout:1594: write oid 6 current snap is 170 2024-09-16T05:17:01.754 INFO:tasks.rados.rados.2.smithi204.stdout:1594: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2024-09-16T05:17:01.760 INFO:tasks.rados.rados.2.smithi204.stdout:1594: writing smithi20457400-6 from 796797 to 1290451 tid 1 2024-09-16T05:17:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:01.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:01.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:01.767 INFO:tasks.rados.rados.2.smithi204.stdout:1594: writing smithi20457400-6 from 1830277 to 2627128 tid 2 2024-09-16T05:17:01.773 INFO:tasks.rados.rados.2.smithi204.stdout:1594: writing smithi20457400-6 from 3397470 to 3596797 tid 3 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1595: write (excl) oid 30 current snap is 170 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1593: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1593: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1593: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 346 (ObjNum 476 snap 150 seq_num 476) dirty exists 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1591: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1590: finishing copy_from racing read to smithi20457400-41 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1591: finishing copy_from to smithi20457400-33 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 326 (ObjNum 440 snap 131 seq_num 440) dirty exists 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1594: finishing write tid 1 to smithi20457400-6 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:1590: finishing copy_from to smithi20457400-41 2024-09-16T05:17:01.774 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 517 (ObjNum 536 snap 160 seq_num 536) dirty exists 2024-09-16T05:17:01.775 INFO:tasks.rados.rados.2.smithi204.stdout:1594: finishing write tid 2 to smithi20457400-6 2024-09-16T05:17:01.775 INFO:tasks.rados.rados.2.smithi204.stdout:1595: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-09-16T05:17:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:01.778 INFO:tasks.rados.rados.2.smithi204.stdout:1595: writing smithi20457400-30 from 413604 to 982507 tid 1 2024-09-16T05:17:01.785 INFO:tasks.rados.rados.2.smithi204.stdout:1595: writing smithi20457400-30 from 1488759 to 2145621 tid 2 2024-09-16T05:17:01.792 INFO:tasks.rados.rados.2.smithi204.stdout:1595: writing smithi20457400-30 from 2624893 to 3216602 tid 3 2024-09-16T05:17:01.792 INFO:tasks.rados.rados.2.smithi204.stdout:1590: done (4 left) 2024-09-16T05:17:01.792 INFO:tasks.rados.rados.2.smithi204.stdout:1591: done (3 left) 2024-09-16T05:17:01.792 INFO:tasks.rados.rados.2.smithi204.stdout:1593: done (2 left) 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1596: read oid 3 snap -1 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1596: expect (ObjNum 516 snap 157 seq_num 516) 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1597: read oid 41 snap 168 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1597: expect deleted 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1598: rollback oid 26 current snap is 170 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 168 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1594: finishing write tid 3 to smithi20457400-6 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1599: write oid 29 current snap is 170 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1594: finishing write tid 4 to smithi20457400-6 2024-09-16T05:17:01.793 INFO:tasks.rados.rados.2.smithi204.stdout:1594: finishing write tid 5 to smithi20457400-6 2024-09-16T05:17:01.794 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 342 (ObjNum 571 snap 170 seq_num 571) dirty exists 2024-09-16T05:17:01.794 INFO:tasks.rados.rados.2.smithi204.stdout:1594: left oid 6 (ObjNum 571 snap 170 seq_num 571) 2024-09-16T05:17:01.794 INFO:tasks.rados.rados.2.smithi204.stdout:1595: finishing write tid 1 to smithi20457400-30 2024-09-16T05:17:01.794 INFO:tasks.rados.rados.2.smithi204.stdout:1595: finishing write tid 2 to smithi20457400-30 2024-09-16T05:17:01.794 INFO:tasks.rados.rados.2.smithi204.stdout:1599: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2024-09-16T05:17:01.797 INFO:tasks.rados.rados.2.smithi204.stdout:1599: writing smithi20457400-29 from 430411 to 1074563 tid 1 2024-09-16T05:17:01.804 INFO:tasks.rados.rados.2.smithi204.stdout:1599: writing smithi20457400-29 from 1830888 to 2231408 tid 2 2024-09-16T05:17:01.810 INFO:tasks.rados.rados.2.smithi204.stdout:1599: writing smithi20457400-29 from 2935963 to 3338944 tid 3 2024-09-16T05:17:01.810 INFO:tasks.rados.rados.2.smithi204.stdout:1594: done (5 left) 2024-09-16T05:17:01.810 INFO:tasks.rados.rados.2.smithi204.stdout:1600: snap_create 2024-09-16T05:17:01.810 INFO:tasks.rados.rados.2.smithi204.stdout:1595: finishing write tid 3 to smithi20457400-30 2024-09-16T05:17:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:1595: finishing write tid 4 to smithi20457400-30 2024-09-16T05:17:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:1595: finishing write tid 5 to smithi20457400-30 2024-09-16T05:17:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 797 (ObjNum 572 snap 170 seq_num 572) dirty exists 2024-09-16T05:17:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:1595: left oid 30 (ObjNum 572 snap 170 seq_num 572) 2024-09-16T05:17:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:1598: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:17:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:1598: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:17:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:1596: expect (ObjNum 516 snap 157 seq_num 516) 2024-09-16T05:17:01.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:01.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:01.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:01.897 INFO:tasks.rados.rados.2.smithi204.stdout:1598: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:17:01.898 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 801 (ObjNum 562 snap 168 seq_num 562) dirty exists 2024-09-16T05:17:01.898 INFO:tasks.rados.rados.2.smithi204.stdout:1599: finishing write tid 1 to smithi20457400-29 2024-09-16T05:17:01.898 INFO:tasks.rados.rados.2.smithi204.stdout:1599: finishing write tid 2 to smithi20457400-29 2024-09-16T05:17:01.898 INFO:tasks.rados.rados.2.smithi204.stdout:1599: finishing write tid 3 to smithi20457400-29 2024-09-16T05:17:01.898 INFO:tasks.rados.rados.2.smithi204.stdout:1599: finishing write tid 4 to smithi20457400-29 2024-09-16T05:17:01.898 INFO:tasks.rados.rados.2.smithi204.stdout:1599: finishing write tid 5 to smithi20457400-29 2024-09-16T05:17:01.898 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 688 (ObjNum 573 snap 170 seq_num 573) dirty exists 2024-09-16T05:17:01.898 INFO:tasks.rados.rados.2.smithi204.stdout:1599: left oid 29 (ObjNum 573 snap 170 seq_num 573) 2024-09-16T05:17:01.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:02.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:02.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:02.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:02.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:02.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:02.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:02.165 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:01 smithi204 ceph-mon[30933]: pgmap v867: 107 pgs: 107 active+clean; 103 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 718 KiB/s wr, 13 op/s 2024-09-16T05:17:02.166 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:01 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:17:02.166 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:01 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:17:02.166 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:01 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:17:02.166 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:01 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:17:02.166 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:01 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:17:02.166 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:01 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:17:02.166 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:01 smithi204 ceph-mon[30933]: osdmap e472: 12 total, 12 up, 12 in 2024-09-16T05:17:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:01 smithi032 ceph-mon[27388]: pgmap v867: 107 pgs: 107 active+clean; 103 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 718 KiB/s wr, 13 op/s 2024-09-16T05:17:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:01 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:17:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:01 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:17:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:01 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:17:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:01 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:17:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:01 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:17:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:01 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:17:02.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:01 smithi032 ceph-mon[27388]: osdmap e472: 12 total, 12 up, 12 in 2024-09-16T05:17:02.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:17:02.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:01 smithi196 ceph-mon[30802]: pgmap v867: 107 pgs: 107 active+clean; 103 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 718 KiB/s wr, 13 op/s 2024-09-16T05:17:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:01 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:17:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:01 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:17:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:01 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:17:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:01 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:17:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:01 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:17:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:01 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:17:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:01 smithi196 ceph-mon[30802]: osdmap e472: 12 total, 12 up, 12 in 2024-09-16T05:17:02.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:02.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:02.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:02.335 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:02.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:02.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:02.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:02.474 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:02.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:02.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:02.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:02.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:02.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:02.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:02.733 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:17:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:1595: done (5 left) 2024-09-16T05:17:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:1596: done (4 left) 2024-09-16T05:17:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:1597: done (3 left) 2024-09-16T05:17:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:1598: done (2 left) 2024-09-16T05:17:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:1599: done (1 left) 2024-09-16T05:17:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:1600: done (0 left) 2024-09-16T05:17:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:1601: write (excl) oid 42 current snap is 171 2024-09-16T05:17:02.736 INFO:tasks.rados.rados.2.smithi204.stdout:1601: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-16T05:17:02.742 INFO:tasks.rados.rados.2.smithi204.stdout:1601: writing smithi20457400-42 from 447218 to 1166619 tid 1 2024-09-16T05:17:02.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:02.750 INFO:tasks.rados.rados.2.smithi204.stdout:1601: writing smithi20457400-42 from 1889370 to 2549901 tid 2 2024-09-16T05:17:02.756 INFO:tasks.rados.rados.2.smithi204.stdout:1601: writing smithi20457400-42 from 2963386 to 3577639 tid 3 2024-09-16T05:17:02.756 INFO:tasks.rados.rados.2.smithi204.stdout:1602: rollback oid 26 current snap is 171 2024-09-16T05:17:02.756 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 168 2024-09-16T05:17:02.756 INFO:tasks.rados.rados.2.smithi204.stdout:1601: finishing write tid 1 to smithi20457400-42 2024-09-16T05:17:02.757 INFO:tasks.rados.rados.2.smithi204.stdout:1601: finishing write tid 2 to smithi20457400-42 2024-09-16T05:17:02.757 INFO:tasks.rados.rados.2.smithi204.stdout:1603: read oid 41 snap -1 2024-09-16T05:17:02.757 INFO:tasks.rados.rados.2.smithi204.stdout:1603: expect (ObjNum 536 snap 160 seq_num 536) 2024-09-16T05:17:02.757 INFO:tasks.rados.rados.2.smithi204.stdout:1604: write (excl) oid 30 current snap is 171 2024-09-16T05:17:02.757 INFO:tasks.rados.rados.2.smithi204.stdout:1604: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-09-16T05:17:02.763 INFO:tasks.rados.rados.2.smithi204.stdout:1604: writing smithi20457400-30 from 464025 to 1258675 tid 1 2024-09-16T05:17:02.769 INFO:tasks.rados.rados.2.smithi204.stdout:1604: writing smithi20457400-30 from 1947852 to 2352041 tid 2 2024-09-16T05:17:02.774 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:02.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:02.777 INFO:tasks.rados.rados.2.smithi204.stdout:1604: writing smithi20457400-30 from 2990809 to 3416334 tid 3 2024-09-16T05:17:02.777 INFO:tasks.rados.rados.2.smithi204.stdout:1605: read oid 5 snap -1 2024-09-16T05:17:02.777 INFO:tasks.rados.rados.2.smithi204.stdout:1605: expect (ObjNum 556 snap 166 seq_num 556) 2024-09-16T05:17:02.777 INFO:tasks.rados.rados.2.smithi204.stdout:1606: write oid 14 current snap is 171 2024-09-16T05:17:02.777 INFO:tasks.rados.rados.2.smithi204.stdout:1601: finishing write tid 3 to smithi20457400-42 2024-09-16T05:17:02.777 INFO:tasks.rados.rados.2.smithi204.stdout:1601: finishing write tid 4 to smithi20457400-42 2024-09-16T05:17:02.777 INFO:tasks.rados.rados.2.smithi204.stdout:1601: finishing write tid 5 to smithi20457400-42 2024-09-16T05:17:02.778 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 740 (ObjNum 574 snap 171 seq_num 574) dirty exists 2024-09-16T05:17:02.778 INFO:tasks.rados.rados.2.smithi204.stdout:1601: left oid 42 (ObjNum 574 snap 171 seq_num 574) 2024-09-16T05:17:02.778 INFO:tasks.rados.rados.2.smithi204.stdout:1602: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:17:02.778 INFO:tasks.rados.rados.2.smithi204.stdout:1602: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:17:02.778 INFO:tasks.rados.rados.2.smithi204.stdout:1602: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:17:02.778 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 805 (ObjNum 562 snap 168 seq_num 562) dirty exists 2024-09-16T05:17:02.778 INFO:tasks.rados.rados.2.smithi204.stdout:1604: finishing write tid 1 to smithi20457400-30 2024-09-16T05:17:02.778 INFO:tasks.rados.rados.2.smithi204.stdout:1603: expect (ObjNum 536 snap 160 seq_num 536) 2024-09-16T05:17:02.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:02.900 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:02.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:02.946 INFO:tasks.rados.rados.2.smithi204.stdout:1605: expect (ObjNum 556 snap 166 seq_num 556) 2024-09-16T05:17:02.964 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:02.964 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:02.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:17:03.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: 2.3a scrub starts 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: 2.3a scrub ok 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2017805185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2017805185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:03.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:02 smithi032 ceph-mon[27388]: osdmap e473: 12 total, 12 up, 12 in 2024-09-16T05:17:03.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:03.026 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:03.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:03.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:03.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:03.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: 2.3a scrub starts 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: 2.3a scrub ok 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2017805185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:17:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2017805185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:02 smithi204 ceph-mon[30933]: osdmap e473: 12 total, 12 up, 12 in 2024-09-16T05:17:03.203 INFO:tasks.rados.rados.2.smithi204.stdout:1604: finishing write tid 2 to smithi20457400-30 2024-09-16T05:17:03.203 INFO:tasks.rados.rados.2.smithi204.stdout:1604: finishing write tid 3 to smithi20457400-30 2024-09-16T05:17:03.204 INFO:tasks.rados.rados.2.smithi204.stdout:1604: finishing write tid 4 to smithi20457400-30 2024-09-16T05:17:03.204 INFO:tasks.rados.rados.2.smithi204.stdout:1604: finishing write tid 5 to smithi20457400-30 2024-09-16T05:17:03.204 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 810 (ObjNum 575 snap 171 seq_num 575) dirty exists 2024-09-16T05:17:03.204 INFO:tasks.rados.rados.2.smithi204.stdout:1604: left oid 30 (ObjNum 575 snap 171 seq_num 575) 2024-09-16T05:17:03.204 INFO:tasks.rados.rados.2.smithi204.stdout:1606: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2024-09-16T05:17:03.208 INFO:tasks.rados.rados.2.smithi204.stdout:1606: writing smithi20457400-14 from 480832 to 950731 tid 1 2024-09-16T05:17:03.215 INFO:tasks.rados.rados.2.smithi204.stdout:1606: writing smithi20457400-14 from 1606334 to 2270534 tid 2 2024-09-16T05:17:03.223 INFO:tasks.rados.rados.2.smithi204.stdout:1606: writing smithi20457400-14 from 3018232 to 3680832 tid 3 2024-09-16T05:17:03.223 INFO:tasks.rados.rados.2.smithi204.stdout:1601: done (5 left) 2024-09-16T05:17:03.223 INFO:tasks.rados.rados.2.smithi204.stdout:1602: done (4 left) 2024-09-16T05:17:03.224 INFO:tasks.rados.rados.2.smithi204.stdout:1603: done (3 left) 2024-09-16T05:17:03.224 INFO:tasks.rados.rados.2.smithi204.stdout:1604: done (2 left) 2024-09-16T05:17:03.224 INFO:tasks.rados.rados.2.smithi204.stdout:1605: done (1 left) 2024-09-16T05:17:03.224 INFO:tasks.rados.rados.2.smithi204.stdout:1607: snap_create 2024-09-16T05:17:03.224 INFO:tasks.rados.rados.2.smithi204.stdout:1606: finishing write tid 1 to smithi20457400-14 2024-09-16T05:17:03.224 INFO:tasks.rados.rados.2.smithi204.stdout:1606: finishing write tid 2 to smithi20457400-14 2024-09-16T05:17:03.226 INFO:tasks.rados.rados.2.smithi204.stdout:1606: finishing write tid 3 to smithi20457400-14 2024-09-16T05:17:03.226 INFO:tasks.rados.rados.2.smithi204.stdout:1606: finishing write tid 4 to smithi20457400-14 2024-09-16T05:17:03.226 INFO:tasks.rados.rados.2.smithi204.stdout:1606: finishing write tid 5 to smithi20457400-14 2024-09-16T05:17:03.226 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 815 (ObjNum 576 snap 171 seq_num 576) dirty exists 2024-09-16T05:17:03.226 INFO:tasks.rados.rados.2.smithi204.stdout:1606: left oid 14 (ObjNum 576 snap 171 seq_num 576) 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: 2.3a scrub starts 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: 2.3a scrub ok 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2017805185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:17:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2017805185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:02 smithi196 ceph-mon[30802]: osdmap e473: 12 total, 12 up, 12 in 2024-09-16T05:17:03.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:03.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:03.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:03.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:03.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:03.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:03.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:03.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:03.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:03.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:03.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:03.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:03.661 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:03.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:03.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:17:03.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:03.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:03.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:03.891 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T05:17:03.893 INFO:tasks.rados.rados.2.smithi204.stdout:1606: done (1 left) 2024-09-16T05:17:03.894 INFO:tasks.rados.rados.2.smithi204.stdout:1607: done (0 left) 2024-09-16T05:17:03.894 INFO:tasks.rados.rados.2.smithi204.stdout:1608: write oid 36 current snap is 172 2024-09-16T05:17:03.894 INFO:tasks.rados.rados.2.smithi204.stdout:1608: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-16T05:17:03.899 INFO:tasks.rados.rados.2.smithi204.stdout:1608: writing smithi20457400-36 from 497639 to 1042787 tid 1 2024-09-16T05:17:03.904 INFO:tasks.rados.rados.2.smithi204.stdout:1608: writing smithi20457400-36 from 1548463 to 1956321 tid 2 2024-09-16T05:17:03.910 INFO:tasks.rados.rados.2.smithi204.stdout:1608: writing smithi20457400-36 from 2529302 to 3093724 tid 3 2024-09-16T05:17:03.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:03.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:03.914 INFO:tasks.rados.rados.2.smithi204.stdout:1608: writing smithi20457400-36 from 3528143 to 3697639 tid 4 2024-09-16T05:17:03.914 INFO:tasks.rados.rados.2.smithi204.stdout:1609: copy_from oid 4 from oid 27 current snap is 172 2024-09-16T05:17:03.914 INFO:tasks.rados.rados.2.smithi204.stdout:1608: finishing write tid 1 to smithi20457400-36 2024-09-16T05:17:03.914 INFO:tasks.rados.rados.2.smithi204.stdout:1608: finishing write tid 2 to smithi20457400-36 2024-09-16T05:17:03.914 INFO:tasks.rados.rados.2.smithi204.stdout:1608: finishing write tid 3 to smithi20457400-36 2024-09-16T05:17:03.914 INFO:tasks.rados.rados.2.smithi204.stdout:1610: read oid 9 snap -1 2024-09-16T05:17:03.915 INFO:tasks.rados.rados.2.smithi204.stdout:1610: expect deleted 2024-09-16T05:17:03.915 INFO:tasks.rados.rados.2.smithi204.stdout:1611: snap_create 2024-09-16T05:17:03.916 INFO:tasks.rados.rados.2.smithi204.stdout:1608: finishing write tid 4 to smithi20457400-36 2024-09-16T05:17:03.916 INFO:tasks.rados.rados.2.smithi204.stdout:1608: finishing write tid 5 to smithi20457400-36 2024-09-16T05:17:03.916 INFO:tasks.rados.rados.2.smithi204.stdout:1608: finishing write tid 6 to smithi20457400-36 2024-09-16T05:17:03.916 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 485 (ObjNum 577 snap 172 seq_num 577) dirty exists 2024-09-16T05:17:03.916 INFO:tasks.rados.rados.2.smithi204.stdout:1608: left oid 36 (ObjNum 577 snap 172 seq_num 577) 2024-09-16T05:17:03.926 INFO:tasks.rados.rados.2.smithi204.stdout:1609: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:17:03.929 INFO:tasks.rados.rados.2.smithi204.stdout:1609: finishing copy_from to smithi20457400-4 2024-09-16T05:17:03.929 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 306 (ObjNum 529 snap 159 seq_num 529) dirty exists 2024-09-16T05:17:04.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:04.027 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:04.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:04.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:04.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:04.147 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:03 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:17:04.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:03 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:17:04.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:03 smithi204 ceph-mon[30933]: pgmap v870: 107 pgs: 107 active+clean; 103 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.9 MiB/s wr, 15 op/s 2024-09-16T05:17:04.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/488589058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:17:04.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:03 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:17:04.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:03 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:17:04.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:03 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:17:04.148 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:03 smithi204 ceph-mon[30933]: 2.29 scrub ok 2024-09-16T05:17:04.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:04.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:03 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:17:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:03 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:17:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:03 smithi032 ceph-mon[27388]: pgmap v870: 107 pgs: 107 active+clean; 103 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.9 MiB/s wr, 15 op/s 2024-09-16T05:17:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/488589058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:17:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:03 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:17:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:03 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:17:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:03 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:17:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:03 smithi032 ceph-mon[27388]: 2.29 scrub ok 2024-09-16T05:17:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:03 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:17:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:03 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:17:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:03 smithi196 ceph-mon[30802]: pgmap v870: 107 pgs: 107 active+clean; 103 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.9 MiB/s wr, 15 op/s 2024-09-16T05:17:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/488589058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:17:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:03 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:17:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:03 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:17:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:03 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:17:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:03 smithi196 ceph-mon[30802]: 2.29 scrub ok 2024-09-16T05:17:04.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:04.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:04.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:04.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:04.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:04.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:04.434 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:04.434 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:04.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:04.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:04.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:04.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:04.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-16T05:17:04.660 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:04.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:04.762 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:04.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:04.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:04.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:04.899 INFO:tasks.rados.rados.2.smithi204.stdout:1608: done (3 left) 2024-09-16T05:17:04.900 INFO:tasks.rados.rados.2.smithi204.stdout:1609: done (2 left) 2024-09-16T05:17:04.900 INFO:tasks.rados.rados.2.smithi204.stdout:1610: done (1 left) 2024-09-16T05:17:04.900 INFO:tasks.rados.rados.2.smithi204.stdout:1611: done (0 left) 2024-09-16T05:17:04.900 INFO:tasks.rados.rados.2.smithi204.stdout:1612: snap_create 2024-09-16T05:17:04.908 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:04.908 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:04.911 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:04.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:04.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:04.954 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:04.954 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:04.961 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:04.975 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:04.975 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:05.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:05.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:05.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:05.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:17:05.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:05.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:04 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:17:05.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:04 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:17:05.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:04 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:17:05.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:04 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:17:05.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/488589058' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:17:05.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:04 smithi204 ceph-mon[30933]: osdmap e474: 12 total, 12 up, 12 in 2024-09-16T05:17:05.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2483670090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:04 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:17:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:04 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:17:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:04 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:17:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:04 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:17:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/488589058' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:17:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:04 smithi032 ceph-mon[27388]: osdmap e474: 12 total, 12 up, 12 in 2024-09-16T05:17:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2483670090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:05.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:05.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:05.199 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:05.199 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:05.221 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:05.222 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:04 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:17:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:04 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:17:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:04 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:17:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:04 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:17:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/488589058' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:17:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:04 smithi196 ceph-mon[30802]: osdmap e474: 12 total, 12 up, 12 in 2024-09-16T05:17:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2483670090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:05.252 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:05.253 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:05.274 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:05.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:05.296 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:05.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:05.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:05.317 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:05.317 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:05.338 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:05.338 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:05.358 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:05.359 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:05.359 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:05.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:05.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:05.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:05.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:05.549 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:05.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:05.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:17:05.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:05.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:05.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:05.729 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T05:17:05.733 INFO:tasks.rados.rados.2.smithi204.stdout:1612: done (0 left) 2024-09-16T05:17:05.733 INFO:tasks.rados.rados.2.smithi204.stdout:1613: delete oid 24 current snap is 174 2024-09-16T05:17:05.738 INFO:tasks.rados.rados.2.smithi204.stdout:1613: done (0 left) 2024-09-16T05:17:05.738 INFO:tasks.rados.rados.2.smithi204.stdout:1614: read oid 42 snap -1 2024-09-16T05:17:05.738 INFO:tasks.rados.rados.2.smithi204.stdout:1614: expect (ObjNum 574 snap 171 seq_num 574) 2024-09-16T05:17:05.739 INFO:tasks.rados.rados.2.smithi204.stdout:1615: delete oid 5 current snap is 174 2024-09-16T05:17:05.740 INFO:tasks.rados.rados.2.smithi204.stdout:1615: done (1 left) 2024-09-16T05:17:05.740 INFO:tasks.rados.rados.2.smithi204.stdout:1616: delete oid 8 current snap is 174 2024-09-16T05:17:05.745 INFO:tasks.rados.rados.2.smithi204.stdout:1616: done (1 left) 2024-09-16T05:17:05.745 INFO:tasks.rados.rados.2.smithi204.stdout:1617: write (excl) oid 33 current snap is 174 2024-09-16T05:17:05.745 INFO:tasks.rados.rados.2.smithi204.stdout:1617: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-09-16T05:17:05.754 INFO:tasks.rados.rados.2.smithi204.stdout:1617: writing smithi20457400-33 from 564867 to 1127364 tid 1 2024-09-16T05:17:05.762 INFO:tasks.rados.rados.2.smithi204.stdout:1617: writing smithi20457400-33 from 1898744 to 2313940 tid 2 2024-09-16T05:17:05.770 INFO:tasks.rados.rados.2.smithi204.stdout:1617: writing smithi20457400-33 from 2755347 to 3481210 tid 3 2024-09-16T05:17:05.770 INFO:tasks.rados.rados.2.smithi204.stdout:1618: snap_remove snap 170 2024-09-16T05:17:05.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:05.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:05.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:05.840 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:17:05.841 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:17:05.841 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:17:05.841 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:17:05.841 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:17:05.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:05.925 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:05.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:06.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:05 smithi032 ceph-mon[27388]: pgmap v873: 107 pgs: 107 active+clean; 109 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.5 MiB/s wr, 24 op/s 2024-09-16T05:17:06.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:05 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:17:06.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:05 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:17:06.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:05 smithi032 ceph-mon[27388]: osdmap e475: 12 total, 12 up, 12 in 2024-09-16T05:17:06.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2368027175' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:17:06.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2368027175' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:17:06.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:05 smithi032 ceph-mon[27388]: osdmap e476: 12 total, 12 up, 12 in 2024-09-16T05:17:06.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:06.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:06.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:06.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:06.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:05 smithi204 ceph-mon[30933]: pgmap v873: 107 pgs: 107 active+clean; 109 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.5 MiB/s wr, 24 op/s 2024-09-16T05:17:06.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:05 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:17:06.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:05 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:17:06.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:05 smithi204 ceph-mon[30933]: osdmap e475: 12 total, 12 up, 12 in 2024-09-16T05:17:06.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2368027175' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:17:06.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2368027175' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:17:06.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:05 smithi204 ceph-mon[30933]: osdmap e476: 12 total, 12 up, 12 in 2024-09-16T05:17:06.178 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:06.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:05 smithi196 ceph-mon[30802]: pgmap v873: 107 pgs: 107 active+clean; 109 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.5 MiB/s wr, 24 op/s 2024-09-16T05:17:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:05 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:17:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:05 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:17:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:05 smithi196 ceph-mon[30802]: osdmap e475: 12 total, 12 up, 12 in 2024-09-16T05:17:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2368027175' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:17:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2368027175' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:17:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:05 smithi196 ceph-mon[30802]: osdmap e476: 12 total, 12 up, 12 in 2024-09-16T05:17:06.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:06.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:06.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:06.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:06.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:06.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:06.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:17:06.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:06.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:06.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:06.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:06.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:06.731 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:06.736 INFO:tasks.rados.rados.2.smithi204.stdout:1618: done (2 left) 2024-09-16T05:17:06.736 INFO:tasks.rados.rados.2.smithi204.stdout:1619: snap_create 2024-09-16T05:17:06.736 INFO:tasks.rados.rados.2.smithi204.stdout:1614: expect (ObjNum 574 snap 171 seq_num 574) 2024-09-16T05:17:06.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:06.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:06.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:06.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-16T05:17:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:06.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:06.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:06.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:07.036 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:07.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:07.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:07.093 INFO:tasks.rados.rados.2.smithi204.stdout:1617: finishing write tid 1 to smithi20457400-33 2024-09-16T05:17:07.093 INFO:tasks.rados.rados.2.smithi204.stdout:1617: finishing write tid 2 to smithi20457400-33 2024-09-16T05:17:07.093 INFO:tasks.rados.rados.2.smithi204.stdout:1617: finishing write tid 3 to smithi20457400-33 2024-09-16T05:17:07.093 INFO:tasks.rados.rados.2.smithi204.stdout:1617: finishing write tid 4 to smithi20457400-33 2024-09-16T05:17:07.093 INFO:tasks.rados.rados.2.smithi204.stdout:1617: finishing write tid 5 to smithi20457400-33 2024-09-16T05:17:07.093 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 741 (ObjNum 581 snap 174 seq_num 581) dirty exists 2024-09-16T05:17:07.093 INFO:tasks.rados.rados.2.smithi204.stdout:1617: left oid 33 (ObjNum 581 snap 174 seq_num 581) 2024-09-16T05:17:07.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:06 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:17:07.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:06 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:17:07.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:06 smithi204 ceph-mon[30933]: osdmap e477: 12 total, 12 up, 12 in 2024-09-16T05:17:07.116 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:06 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:17:07.116 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:06 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:17:07.116 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:06 smithi032 ceph-mon[27388]: osdmap e477: 12 total, 12 up, 12 in 2024-09-16T05:17:07.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:07.183 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:07.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:07.210 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:17:07.210 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1329 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 285.42520374999998, 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80822626200000003, 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.211 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1329 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:17:07.212 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783039+0000", 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.674e-06 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783047+0000", 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6619999999999994e-06 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.213 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783063+0000", 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6058000000000001e-05 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783071+0000", 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8399999999999995e-06 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.214 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783134+0000", 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2496999999999995e-05 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783142+0000", 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1219999999999995e-06 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.215 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591169+0000", 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80802694100000005 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591263+0000", 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4468000000000004e-05 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.216 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1330 2.1d 2:b98a79db:::smithi20457400-31:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 285.42515278799999, 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80866329400000003, 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1330 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.217 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.218 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783092+0000", 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5149999999999998e-06 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783104+0000", 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2412000000000001e-05 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.219 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783117+0000", 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2379e-05 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783170+0000", 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3727999999999998e-05 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783178+0000", 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3479999999999998e-06 2024-09-16T05:17:07.220 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.221 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591633+0000", 2024-09-16T05:17:07.221 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.808455339 2024-09-16T05:17:07.221 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.221 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.221 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591751+0000", 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118087 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1331 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 285.42511202200001, 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80878027500000005, 2024-09-16T05:17:07.222 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1331 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.223 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5400000000000001e-07 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783132+0000", 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689999999999999e-06 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.225 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783139+0000", 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7920000000000004e-06 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783144+0000", 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783192+0000", 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8304999999999999e-05 2024-09-16T05:17:07.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783196+0000", 2024-09-16T05:17:07.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8480000000000003e-06 2024-09-16T05:17:07.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591842+0000", 2024-09-16T05:17:07.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80864587700000001 2024-09-16T05:17:07.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591909+0000", 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7195999999999998e-05 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1389 2.1d 2:b98a79db:::smithi20457400-31:head [copy-from ver 20] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 284.61279076400001, 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.062531666999999999, 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1389 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595450+0000", 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0499999999999999e-07 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595453+0000", 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9699999999999999e-06 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595467+0000", 2024-09-16T05:17:07.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3725e-05 2024-09-16T05:17:07.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.595472+0000", 2024-09-16T05:17:07.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.028e-06 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.597983+0000", 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002510759 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.598071+0000", 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8158000000000005e-05 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4", 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.605944+0000", 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078733620000000001 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.608849+0000", 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029044190000000001 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657852+0000", 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.049003110000000002 2024-09-16T05:17:07.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657923+0000", 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1388000000000002e-05 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657982+0000", 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8343e-05 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.235 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1389 2.1d e210/209 ondisk, result = 0)", 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 284.59739184099999, 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.047139519999999997, 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:17:07.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610849+0000", 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610850+0000", 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5600000000000001e-06 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.610858+0000", 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2099999999999996e-06 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657761+0000", 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.046903037000000002 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657768+0000", 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4080000000000001e-06 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657828+0000", 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0241999999999997e-05 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657843+0000", 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5043e-05 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.657988+0000", 2024-09-16T05:17:07.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014501999999999999 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1575 2.5 2:a5bc1608:::smithi20457400-11:head [setxattr _header (57) in=7b,truncate 3966452] snapc 43=[43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 268.51706600099999, 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64249167500000004, 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.240 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1575 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:17:07.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5200000000000002e-07 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691178+0000", 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.734e-06 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691184+0000", 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2940000000000002e-06 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691188+0000", 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.642e-06 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692349+0000", 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001161444 2024-09-16T05:17:07.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692452+0000", 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102916 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692596+0000", 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000143552 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692970+0000", 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037462699999999999 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333552+0000", 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64058177999999999 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333609+0000", 2024-09-16T05:17:07.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6728999999999998e-05 2024-09-16T05:17:07.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333666+0000", 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7504999999999999e-05 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1576 2.5 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 268.51704726600002, 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64272760100000004, 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.251 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1576 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691194+0000", 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.65e-07 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691195+0000", 2024-09-16T05:17:07.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-06 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691199+0000", 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0060000000000003e-06 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691202+0000", 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9239999999999999e-06 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692872+0000", 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001669467 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692954+0000", 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2361000000000006e-05 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333715+0000", 2024-09-16T05:17:07.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64076073300000003 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333817+0000", 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010184200000000001 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333921+0000", 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104249 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.259 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2493 2.14 2:2ace617f:::smithi20457400-21:head [copy-from ver 319] snapc 6a=[6a,69,68,67,66,64,63,62,60,5d,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:02.736051+0000", 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.47219006, 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025553787000000001, 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.260 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2493 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736051+0000", 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736051+0000", 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736052+0000", 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7699999999999992e-07 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736057+0000", 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2599999999999996e-06 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736069+0000", 2024-09-16T05:17:07.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2045000000000001e-05 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736076+0000", 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8870000000000003e-06 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736189+0000", 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011327099999999999 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.736302+0000", 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011278 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.755011+0000", 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018709445000000002 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.760557+0000", 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0055454859999999996 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.761496+0000", 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093937199999999995 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.761543+0000", 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6523999999999997e-05 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:02.761604+0000", 2024-09-16T05:17:07.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1740000000000002e-05 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:3036 2.1b e379/291 2:da4f6049:::smithi20457400-49:head v 379'328, mlcod=379'325)", 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.440062261999998, 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.436541917, 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.266 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768179+0000", 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2900000000000003e-07 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768185+0000", 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8309999999999997e-06 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203766+0000", 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43558113999999998 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203780+0000", 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3825999999999999e-05 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203879+0000", 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9419000000000005e-05 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203926+0000", 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6248999999999997e-05 2024-09-16T05:17:07.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204668+0000", 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00074248900000000004 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204720+0000", 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2234000000000002e-05 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.9.0:755 2.2f e422/275 ondisk, result = 0)", 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 55.484719802000001, 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.59448197899999999, 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7700000000000004e-07 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723525+0000", 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5209999999999998e-06 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317758+0000", 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.59423284399999998 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317765+0000", 2024-09-16T05:17:07.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.827e-06 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317821+0000", 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6573000000000002e-05 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317832+0000", 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0718e-05 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.318003+0000", 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017091900000000001 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3531 2.15 2:a94680a3:::smithi20457400-13:head [write 652237~793851] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:17:07.275 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:17:07.276 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.917785236999997, 2024-09-16T05:17:07.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44774744599999999, 2024-09-16T05:17:07.276 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.276 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.276 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.276 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.276 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.276 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3531 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290457+0000", 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.823e-06 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291024+0000", 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00056645899999999995 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291045+0000", 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0837999999999999e-05 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291055+0000", 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0066e-05 2024-09-16T05:17:07.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291140+0000", 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5686000000000006e-05 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291149+0000", 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7579999999999998e-06 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.733424+0000", 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.442274801 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.733559+0000", 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013477500000000001 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.734084+0000", 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00052511500000000004 2024-09-16T05:17:07.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738013+0000", 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039291179999999997 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738083+0000", 2024-09-16T05:17:07.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0208000000000003e-05 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738157+0000", 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4071000000000003e-05 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738203+0000", 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5728e-05 2024-09-16T05:17:07.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.284 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.284 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.284 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3532 2.15 2:a94680a3:::smithi20457400-13:head [write 2198894~53343] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:17:07.284 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:17:07.284 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.913454229999999, 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44479106600000001, 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3532 2024-09-16T05:17:07.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294787+0000", 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6700000000000003e-07 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294826+0000", 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8801999999999999e-05 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294837+0000", 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1079e-05 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294842+0000", 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.152e-06 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:17:07.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8766e-05 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294898+0000", 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1470000000000004e-06 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737061+0000", 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44216253500000002 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737164+0000", 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103288 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737374+0000", 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020994000000000001 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739175+0000", 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018013759999999999 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739420+0000", 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024481500000000001 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739486+0000", 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6308000000000003e-05 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739577+0000", 2024-09-16T05:17:07.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1090999999999999e-05 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3533 2.15 2:a94680a3:::smithi20457400-13:head [setxattr _header (58) in=7b,truncate 2252237] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.913386602000003, 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44535093999999997, 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.292 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3533 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294855+0000", 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0699999999999997e-07 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294858+0000", 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2200000000000001e-06 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294865+0000", 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9290000000000001e-06 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294868+0000", 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4750000000000002e-06 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294924+0000", 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6005000000000003e-05 2024-09-16T05:17:07.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294934+0000", 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0234999999999999e-05 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738506+0000", 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44357146200000003 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738606+0000", 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100338 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738777+0000", 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017038200000000001 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739632+0000", 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00085582599999999996 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740092+0000", 2024-09-16T05:17:07.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046005699999999999 2024-09-16T05:17:07.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:07.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:06 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:17:07.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:06 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:17:07.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:06 smithi196 ceph-mon[30802]: osdmap e477: 12 total, 12 up, 12 in 2024-09-16T05:17:07.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740144+0000", 2024-09-16T05:17:07.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.113e-05 2024-09-16T05:17:07.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740205+0000", 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1373999999999999e-05 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3534 2.15 2:a94680a3:::smithi20457400-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.913349316000001, 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44553962000000003, 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.301 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3534 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294892+0000", 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0299999999999999e-07 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294894+0000", 2024-09-16T05:17:07.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.198e-06 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294901+0000", 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.917e-06 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294912+0000", 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1144999999999999e-05 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294950+0000", 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7889999999999998e-05 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294955+0000", 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8949999999999997e-06 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739213+0000", 2024-09-16T05:17:07.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44425779500000001 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739349+0000", 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013630599999999999 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740238+0000", 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088882100000000003 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740329+0000", 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1366999999999994e-05 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740431+0000", 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101783 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3539 2.54 2:2ace617f:::smithi20457400-21:head [write 685851~544349] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:17:07.307 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.895047384999998, 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42847687499999998, 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3539 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313194+0000", 2024-09-16T05:17:07.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313430+0000", 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023602700000000001 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313441+0000", 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1158000000000001e-05 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313449+0000", 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3180000000000001e-06 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313509+0000", 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0010999999999999e-05 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313516+0000", 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.131e-06 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737710+0000", 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42419478900000002 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738322+0000", 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061162399999999996 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738531+0000", 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020935999999999999 2024-09-16T05:17:07.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741253+0000", 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027214650000000002 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741592+0000", 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033932199999999997 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741636+0000", 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3649000000000002e-05 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741670+0000", 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4264999999999998e-05 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3540 2.54 2:2ace617f:::smithi20457400-21:head [write 1799505~486346] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:17:07.314 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.889554707000002, 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42371521699999998, 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3540 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318687+0000", 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9400000000000004e-07 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318917+0000", 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023015399999999999 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318932+0000", 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.485e-05 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318939+0000", 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5059999999999998e-06 2024-09-16T05:17:07.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7806999999999997e-05 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319006+0000", 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1649999999999995e-06 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739899+0000", 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42089313 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739946+0000", 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6783000000000003e-05 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740114+0000", 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016756599999999999 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741289+0000", 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011754549999999999 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742318+0000", 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010284420000000001 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742363+0000", 2024-09-16T05:17:07.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5760999999999998e-05 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742401+0000", 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7803999999999999e-05 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3541 2.54 2:2ace617f:::smithi20457400-21:head [setxattr _header (58) in=7b,truncate 2285851] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.889284803999999, 2024-09-16T05:17:07.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42367250699999998, 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3541 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:17:07.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5600000000000001e-07 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318959+0000", 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7020000000000001e-06 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318970+0000", 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0991e-05 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318974+0000", 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2849999999999996e-06 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319029+0000", 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5130999999999998e-05 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319036+0000", 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9129999999999997e-06 2024-09-16T05:17:07.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740691+0000", 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42165504500000001 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740769+0000", 2024-09-16T05:17:07.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7179000000000001e-05 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740875+0000", 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106637 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741296+0000", 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042051399999999998 2024-09-16T05:17:07.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742521+0000", 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001225636 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742569+0000", 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7982000000000002e-05 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742628+0000", 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9036000000000002e-05 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3542 2.54 2:2ace617f:::smithi20457400-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.889243997000001, 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42382194699999998, 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.329 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3542 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7599999999999998e-07 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318999+0000", 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7770000000000001e-06 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319007+0000", 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4039999999999995e-06 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319011+0000", 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8260000000000003e-06 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319089+0000", 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8046000000000005e-05 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319099+0000", 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6490000000000003e-06 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741192+0000", 2024-09-16T05:17:07.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42209296299999999 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741242+0000", 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0589e-05 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742668+0000", 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001425429 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742746+0000", 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7990000000000004e-05 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742819+0000", 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2997999999999995e-05 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.335 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3951 2.2f 2:f4b938a4:::smithi20457400-47:head [setxattr _header (58) in=7b,truncate 3395113] snapc a8=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 10.350050712, 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39145527099999999, 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3951 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:17:07.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7099999999999997e-07 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858193+0000", 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3439999999999999e-06 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858197+0000", 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229999999999999e-06 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858200+0000", 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.937e-06 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859064+0000", 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00086360799999999998 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859153+0000", 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9592999999999994e-05 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859294+0000", 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014096099999999999 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859660+0000", 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036632599999999999 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249556+0000", 2024-09-16T05:17:07.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38989596300000001 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249600+0000", 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3757000000000001e-05 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249645+0000", 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5087999999999999e-05 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.341 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3952 2.2f 2:f4b938a4:::smithi20457400-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e467)", 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 10.3500189, 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39157208999999998, 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3952 2024-09-16T05:17:07.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:07.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.77e-07 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858226+0000", 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7249999999999999e-06 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858231+0000", 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.797e-06 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858234+0000", 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2320000000000001e-06 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859544+0000", 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001309892 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:07.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859613+0000", 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9261e-05 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249668+0000", 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39005463499999998 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249718+0000", 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0416e-05 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:07.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:07.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249794+0000", 2024-09-16T05:17:07.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5754999999999996e-05 2024-09-16T05:17:07.347 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.347 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.347 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.347 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:07.347 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:07.347 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:17:07.364 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:07.368 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:07.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:07.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:07.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:07.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:07.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:07.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:07.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:07.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:07.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:07.732 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:17:07.742 INFO:tasks.rados.rados.2.smithi204.stdout:1614: done (2 left) 2024-09-16T05:17:07.743 INFO:tasks.rados.rados.2.smithi204.stdout:1617: done (1 left) 2024-09-16T05:17:07.743 INFO:tasks.rados.rados.2.smithi204.stdout:1619: done (0 left) 2024-09-16T05:17:07.743 INFO:tasks.rados.rados.2.smithi204.stdout:1620: snap_remove snap 171 2024-09-16T05:17:07.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:07.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:07.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:07.873 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:07.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:07.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:07.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:08.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:08.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:08.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:17:08.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: pgmap v876: 107 pgs: 107 active+clean; 117 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.7 MiB/s wr, 18 op/s 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: 2.2 deep-scrub starts 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: 2.2 deep-scrub ok 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3805149614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3805149614' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:08.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:07 smithi032 ceph-mon[27388]: osdmap e478: 12 total, 12 up, 12 in 2024-09-16T05:17:08.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:08.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:17:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:08.131 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:08.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:08.153 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: pgmap v876: 107 pgs: 107 active+clean; 117 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.7 MiB/s wr, 18 op/s 2024-09-16T05:17:08.153 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: 2.2 deep-scrub starts 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: 2.2 deep-scrub ok 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3805149614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3805149614' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:08.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:07 smithi196 ceph-mon[30802]: osdmap e478: 12 total, 12 up, 12 in 2024-09-16T05:17:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: pgmap v876: 107 pgs: 107 active+clean; 117 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.7 MiB/s wr, 18 op/s 2024-09-16T05:17:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: 2.2 deep-scrub starts 2024-09-16T05:17:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: 2.2 deep-scrub ok 2024-09-16T05:17:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:17:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:17:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3805149614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3805149614' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:07 smithi204 ceph-mon[30933]: osdmap e478: 12 total, 12 up, 12 in 2024-09-16T05:17:08.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:08.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:08.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:08.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:08.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:08.365 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:08.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:08.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:08.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:08.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:08.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:08.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:08.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:08.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:08.636 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:08.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:08.733 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T05:17:08.737 INFO:tasks.rados.rados.2.smithi204.stdout:1620: done (0 left) 2024-09-16T05:17:08.737 INFO:tasks.rados.rados.2.smithi204.stdout:1621: write (excl) oid 49 current snap is 175 2024-09-16T05:17:08.737 INFO:tasks.rados.rados.2.smithi204.stdout:1621: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2024-09-16T05:17:08.743 INFO:tasks.rados.rados.2.smithi204.stdout:1621: writing smithi20457400-49 from 581674 to 1219420 tid 1 2024-09-16T05:17:08.751 INFO:tasks.rados.rados.2.smithi204.stdout:1621: writing smithi20457400-49 from 1840873 to 2399727 tid 2 2024-09-16T05:17:08.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:08.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:08.759 INFO:tasks.rados.rados.2.smithi204.stdout:1621: writing smithi20457400-49 from 3066417 to 3603552 tid 3 2024-09-16T05:17:08.759 INFO:tasks.rados.rados.2.smithi204.stdout:1622: write (excl) oid 43 current snap is 175 2024-09-16T05:17:08.759 INFO:tasks.rados.rados.2.smithi204.stdout:1621: finishing write tid 1 to smithi20457400-49 2024-09-16T05:17:08.759 INFO:tasks.rados.rados.2.smithi204.stdout:1621: finishing write tid 2 to smithi20457400-49 2024-09-16T05:17:08.759 INFO:tasks.rados.rados.2.smithi204.stdout:1622: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-09-16T05:17:08.765 INFO:tasks.rados.rados.2.smithi204.stdout:1622: writing smithi20457400-43 from 598481 to 1311476 tid 1 2024-09-16T05:17:08.771 INFO:tasks.rados.rados.2.smithi204.stdout:1622: writing smithi20457400-43 from 1899355 to 2318220 tid 2 2024-09-16T05:17:08.779 INFO:tasks.rados.rados.2.smithi204.stdout:1622: writing smithi20457400-43 from 3093840 to 3798481 tid 3 2024-09-16T05:17:08.779 INFO:tasks.rados.rados.2.smithi204.stdout:1623: write (excl) oid 44 current snap is 175 2024-09-16T05:17:08.780 INFO:tasks.rados.rados.2.smithi204.stdout:1621: finishing write tid 3 to smithi20457400-49 2024-09-16T05:17:08.780 INFO:tasks.rados.rados.2.smithi204.stdout:1623: seq_num 584 ranges {615288=788244,1957837=562523,3121263=559679} 2024-09-16T05:17:08.787 INFO:tasks.rados.rados.2.smithi204.stdout:1623: writing smithi20457400-44 from 615288 to 1403532 tid 1 2024-09-16T05:17:08.793 INFO:tasks.rados.rados.2.smithi204.stdout:1623: writing smithi20457400-44 from 1957837 to 2520360 tid 2 2024-09-16T05:17:08.800 INFO:tasks.rados.rados.2.smithi204.stdout:1623: writing smithi20457400-44 from 3121263 to 3680942 tid 3 2024-09-16T05:17:08.800 INFO:tasks.rados.rados.2.smithi204.stdout:1624: delete oid 38 current snap is 175 2024-09-16T05:17:08.800 INFO:tasks.rados.rados.2.smithi204.stdout:1621: finishing write tid 4 to smithi20457400-49 2024-09-16T05:17:08.800 INFO:tasks.rados.rados.2.smithi204.stdout:1621: finishing write tid 5 to smithi20457400-49 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 559 (ObjNum 582 snap 175 seq_num 582) dirty exists 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1621: left oid 49 (ObjNum 582 snap 175 seq_num 582) 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1622: finishing write tid 1 to smithi20457400-43 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1622: finishing write tid 2 to smithi20457400-43 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1622: finishing write tid 3 to smithi20457400-43 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1622: finishing write tid 4 to smithi20457400-43 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1622: finishing write tid 5 to smithi20457400-43 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 563 (ObjNum 583 snap 175 seq_num 583) dirty exists 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1622: left oid 43 (ObjNum 583 snap 175 seq_num 583) 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1623: finishing write tid 1 to smithi20457400-44 2024-09-16T05:17:08.801 INFO:tasks.rados.rados.2.smithi204.stdout:1623: finishing write tid 2 to smithi20457400-44 2024-09-16T05:17:08.802 INFO:tasks.rados.rados.2.smithi204.stdout:1623: finishing write tid 3 to smithi20457400-44 2024-09-16T05:17:08.802 INFO:tasks.rados.rados.2.smithi204.stdout:1623: finishing write tid 4 to smithi20457400-44 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1621: done (3 left) 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1622: done (2 left) 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1624: done (1 left) 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1625: rollback oid 37 current snap is 175 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 172 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1626: read oid 2 snap -1 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1626: expect (ObjNum 546 snap 163 seq_num 546) 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1627: snap_create 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1623: finishing write tid 5 to smithi20457400-44 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 287 (ObjNum 584 snap 175 seq_num 584) dirty exists 2024-09-16T05:17:08.803 INFO:tasks.rados.rados.2.smithi204.stdout:1623: left oid 44 (ObjNum 584 snap 175 seq_num 584) 2024-09-16T05:17:08.804 INFO:tasks.rados.rados.2.smithi204.stdout:1625: finishing rollback tid 0 to smithi20457400-37 2024-09-16T05:17:08.804 INFO:tasks.rados.rados.2.smithi204.stdout:1625: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:17:08.804 INFO:tasks.rados.rados.2.smithi204.stdout:1625: finishing rollback tid 2 to smithi20457400-37 2024-09-16T05:17:08.804 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 586 (ObjNum 561 snap 168 seq_num 561) dirty exists 2024-09-16T05:17:08.811 INFO:tasks.rados.rados.2.smithi204.stdout:1626: expect (ObjNum 546 snap 163 seq_num 546) 2024-09-16T05:17:08.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:08.854 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:17:08.854 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:17:08.854 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:17:08.854 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:17:08.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:08.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:08.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:09.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:09.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:09.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:09.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:09.128 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:09.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:09.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:08 smithi032 ceph-mon[27388]: 1.b deep-scrub starts 2024-09-16T05:17:09.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:08 smithi032 ceph-mon[27388]: 1.b deep-scrub ok 2024-09-16T05:17:09.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:08 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:17:09.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3584799812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:17:09.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3584799812' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:17:09.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:08 smithi032 ceph-mon[27388]: osdmap e479: 12 total, 12 up, 12 in 2024-09-16T05:17:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:08 smithi204 ceph-mon[30933]: 1.b deep-scrub starts 2024-09-16T05:17:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:08 smithi204 ceph-mon[30933]: 1.b deep-scrub ok 2024-09-16T05:17:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:08 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:17:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3584799812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:17:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3584799812' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:17:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:08 smithi204 ceph-mon[30933]: osdmap e479: 12 total, 12 up, 12 in 2024-09-16T05:17:09.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:09.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:08 smithi196 ceph-mon[30802]: 1.b deep-scrub starts 2024-09-16T05:17:09.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:08 smithi196 ceph-mon[30802]: 1.b deep-scrub ok 2024-09-16T05:17:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:08 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:17:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3584799812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:17:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3584799812' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:17:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:08 smithi196 ceph-mon[30802]: osdmap e479: 12 total, 12 up, 12 in 2024-09-16T05:17:09.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:09.253 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:09.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:09.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:09.377 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:09.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:09.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:09.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-16T05:17:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:09.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:09.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:09.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:17:09.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:09.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:09.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:09.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:09.737 INFO:tasks.rados.rados.2.smithi204.stdout:1623: done (3 left) 2024-09-16T05:17:09.737 INFO:tasks.rados.rados.2.smithi204.stdout:1625: done (2 left) 2024-09-16T05:17:09.737 INFO:tasks.rados.rados.2.smithi204.stdout:1626: done (1 left) 2024-09-16T05:17:09.737 INFO:tasks.rados.rados.2.smithi204.stdout:1627: done (0 left) 2024-09-16T05:17:09.738 INFO:tasks.rados.rados.2.smithi204.stdout:1628: snap_remove snap 172 2024-09-16T05:17:09.749 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:09.749 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:09.750 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:09.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:09.773 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:09.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:09.774 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:09.826 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:09.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:09.875 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:09.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:09.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:09.993 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:09 smithi032 ceph-mon[27388]: pgmap v879: 107 pgs: 107 active+clean; 119 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-16T05:17:09.993 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:09 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:17:09.993 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:09 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:17:09.993 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:09 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:09.993 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:09 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:09.993 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3936917257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:09.993 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:09 smithi032 ceph-mon[27388]: osdmap e480: 12 total, 12 up, 12 in 2024-09-16T05:17:09.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:10.002 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:10.002 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:10.023 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:10.023 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:10.045 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:10.045 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:10.066 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:10.066 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:10.087 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:10.087 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:10.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:10.108 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:10.109 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:10.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:10.131 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:10.152 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:10.152 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:09 smithi204 ceph-mon[30933]: pgmap v879: 107 pgs: 107 active+clean; 119 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-16T05:17:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:09 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:17:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:09 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:17:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:09 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:09 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3936917257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:09 smithi204 ceph-mon[30933]: osdmap e480: 12 total, 12 up, 12 in 2024-09-16T05:17:10.172 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:10.194 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:10.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:10.228 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:10.228 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:09 smithi196 ceph-mon[30802]: pgmap v879: 107 pgs: 107 active+clean; 119 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2024-09-16T05:17:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:09 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:17:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:09 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:17:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:09 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:09 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3936917257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:09 smithi196 ceph-mon[30802]: osdmap e480: 12 total, 12 up, 12 in 2024-09-16T05:17:10.248 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:10.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:10.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:10.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.498 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:10.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:10.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:17:10.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:10.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:10.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:10.731 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T05:17:10.737 INFO:tasks.rados.rados.2.smithi204.stdout:1628: done (0 left) 2024-09-16T05:17:10.737 INFO:tasks.rados.rados.2.smithi204.stdout:1629: copy_from oid 46 from oid 21 current snap is 176 2024-09-16T05:17:10.737 INFO:tasks.rados.rados.2.smithi204.stdout:1630: read oid 1 snap -1 2024-09-16T05:17:10.737 INFO:tasks.rados.rados.2.smithi204.stdout:1630: expect deleted 2024-09-16T05:17:10.737 INFO:tasks.rados.rados.2.smithi204.stdout:1631: delete oid 23 current snap is 176 2024-09-16T05:17:10.740 INFO:tasks.rados.rados.2.smithi204.stdout:1630: done (2 left) 2024-09-16T05:17:10.740 INFO:tasks.rados.rados.2.smithi204.stdout:1631: done (1 left) 2024-09-16T05:17:10.740 INFO:tasks.rados.rados.2.smithi204.stdout:1632: snap_remove snap 174 2024-09-16T05:17:10.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:10.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:10.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:17:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:10.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:10.974 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:10.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:10.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:11.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:11.113 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:11.120 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:11.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2156455510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:17:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2156455510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:17:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:10 smithi032 ceph-mon[27388]: osdmap e481: 12 total, 12 up, 12 in 2024-09-16T05:17:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2156455510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:17:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2156455510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:17:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:10 smithi204 ceph-mon[30933]: osdmap e481: 12 total, 12 up, 12 in 2024-09-16T05:17:11.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:11.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2156455510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:17:11.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2156455510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:17:11.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:10 smithi196 ceph-mon[30802]: osdmap e481: 12 total, 12 up, 12 in 2024-09-16T05:17:11.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:11.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:11.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:11.369 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:11.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:11.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:11.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:11.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:11.527 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:11.527 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:11.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:11.618 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:11.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:11.680 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T05:17:11.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:17:11.693 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:11.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:11.738 INFO:tasks.rados.rados.2.smithi204.stdout:1632: done (1 left) 2024-09-16T05:17:11.738 INFO:tasks.rados.rados.2.smithi204.stdout:1633: rollback oid 10 current snap is 176 2024-09-16T05:17:11.738 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 168 2024-09-16T05:17:11.738 INFO:tasks.rados.rados.2.smithi204.stdout:1629: finishing copy_from racing read to smithi20457400-46 2024-09-16T05:17:11.738 INFO:tasks.rados.rados.2.smithi204.stdout:1629: finishing copy_from to smithi20457400-46 2024-09-16T05:17:11.739 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 329 (ObjNum 493 snap 152 seq_num 493) dirty exists 2024-09-16T05:17:11.739 INFO:tasks.rados.rados.2.smithi204.stdout:1629: done (1 left) 2024-09-16T05:17:11.739 INFO:tasks.rados.rados.2.smithi204.stdout:1634: read oid 50 snap -1 2024-09-16T05:17:11.739 INFO:tasks.rados.rados.2.smithi204.stdout:1634: expect (ObjNum 501 snap 152 seq_num 501) 2024-09-16T05:17:11.739 INFO:tasks.rados.rados.2.smithi204.stdout:1635: snap_remove snap 175 2024-09-16T05:17:11.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:11.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:11.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:11.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:11.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:11.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:11.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:11.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:12.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:12.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:12.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:12.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:11 smithi032 ceph-mon[27388]: pgmap v882: 107 pgs: 107 active+clean; 128 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-16T05:17:12.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:11 smithi032 ceph-mon[27388]: osdmap e482: 12 total, 12 up, 12 in 2024-09-16T05:17:12.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:11 smithi204 ceph-mon[30933]: pgmap v882: 107 pgs: 107 active+clean; 128 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-16T05:17:12.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:11 smithi204 ceph-mon[30933]: osdmap e482: 12 total, 12 up, 12 in 2024-09-16T05:17:12.195 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:12.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:17:12.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:11 smithi196 ceph-mon[30802]: pgmap v882: 107 pgs: 107 active+clean; 128 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 2.0 MiB/s wr, 12 op/s 2024-09-16T05:17:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:11 smithi196 ceph-mon[30802]: osdmap e482: 12 total, 12 up, 12 in 2024-09-16T05:17:12.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:12.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:12.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:12.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:12.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:12.444 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:12.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:12.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:12.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:12.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:12.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:12.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:17:12.736 INFO:tasks.rados.rados.2.smithi204.stdout:1635: done (2 left) 2024-09-16T05:17:12.737 INFO:tasks.rados.rados.2.smithi204.stdout:1636: snap_create 2024-09-16T05:17:12.737 INFO:tasks.rados.rados.2.smithi204.stdout:1633: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:17:12.737 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 241 snap 0 seq_num 4282032544) dirty dne 2024-09-16T05:17:12.737 INFO:tasks.rados.rados.2.smithi204.stdout:1634: expect (ObjNum 501 snap 152 seq_num 501) 2024-09-16T05:17:12.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:12.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:12.828 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:17:12.830 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:17:12.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:12.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:12.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:12.919 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:12.919 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:12.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:13.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:13.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:13.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:17:13.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:13.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:13.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2071962764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:12 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2071962764' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:12 smithi032 ceph-mon[27388]: osdmap e483: 12 total, 12 up, 12 in 2024-09-16T05:17:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2071962764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:12 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2071962764' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:12 smithi204 ceph-mon[30933]: osdmap e483: 12 total, 12 up, 12 in 2024-09-16T05:17:13.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2071962764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:13.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:12 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:13.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2071962764' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:13.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:12 smithi196 ceph-mon[30802]: osdmap e483: 12 total, 12 up, 12 in 2024-09-16T05:17:13.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:13.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:13.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:13.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:13.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:13.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:13.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:13.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:13.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:17:13.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:13.652 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:13.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:13.737 INFO:tasks.rados.rados.2.smithi204.stdout:1633: done (2 left) 2024-09-16T05:17:13.737 INFO:tasks.rados.rados.2.smithi204.stdout:1634: done (1 left) 2024-09-16T05:17:13.737 INFO:tasks.rados.rados.2.smithi204.stdout:1636: done (0 left) 2024-09-16T05:17:13.737 INFO:tasks.rados.rados.2.smithi204.stdout:1637: snap_remove snap 176 2024-09-16T05:17:13.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:13.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:13.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:13.799 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:13.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:13.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:13.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:14.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:14.028 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:14.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:14.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:13 smithi032 ceph-mon[27388]: pgmap v885: 107 pgs: 107 active+clean; 131 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-16T05:17:14.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:13 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:14.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:13 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:14.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:13 smithi032 ceph-mon[27388]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:14.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:13 smithi032 ceph-mon[27388]: osdmap e484: 12 total, 12 up, 12 in 2024-09-16T05:17:14.065 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":886,"stamp":"2024-09-16T05:17:12.733454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":142889214,"num_objects":60,"num_object_clones":19,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":4,"num_read":1290,"num_read_kb":1301382,"num_write":935,"num_write_kb":398621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22262,"ondisk_log_size":22262,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":672088,"kb_used_data":250924,"kb_used_omap":27,"kb_used_meta":420836,"kb_avail":1124269736,"statfs":{"total":1151940427776,"available":1151252209664,"internally_reserved":0,"allocated":256946176,"data_stored":258559929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":430936231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12656914,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":18,"num_read_kb":10834,"num_write":24,"num_write_kb":9255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.999836"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":2341,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738552+0000","last_change":"2024-09-16T05:17:10.739031+0000","last_active":"2024-09-16T05:17:10.738552+0000","last_peered":"2024-09-16T05:17:10.738552+0000","last_clean":"2024-09-16T05:17:10.738552+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:17:10.738552+0000","last_undegraded":"2024-09-16T05:17:10.738552+0000","last_fullsized":"2024-09-16T05:17:10.738552+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:16:38.263205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.263205+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:40.104709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089345699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.54","version":"441'295","reported_seq":1753,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759574+0000","last_change":"2024-09-16T05:17:08.759402+0000","last_active":"2024-09-16T05:17:09.759574+0000","last_peered":"2024-09-16T05:17:09.759574+0000","last_clean":"2024-09-16T05:17:09.759574+0000","last_became_active":"2024-09-16T05:16:30.728460+0000","last_became_peered":"2024-09-16T05:16:30.728460+0000","last_unstale":"2024-09-16T05:17:09.759574+0000","last_undegraded":"2024-09-16T05:17:09.759574+0000","last_fullsized":"2024-09-16T05:17:09.759574+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"441'295","last_scrub_stamp":"2024-09-16T05:16:37.936216+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.936216+0000","objects_scrubbed":1,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:45.530193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013071129999999999,"stat_sum":{"num_bytes":2285851,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":4975,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.53","version":"181'15","reported_seq":233,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.733734+0000","last_change":"2024-09-16T05:17:10.733949+0000","last_active":"2024-09-16T05:17:10.733734+0000","last_peered":"2024-09-16T05:17:10.733734+0000","last_clean":"2024-09-16T05:17:10.733734+0000","last_became_active":"2024-09-16T05:16:27.728289+0000","last_became_peered":"2024-09-16T05:16:27.728289+0000","last_unstale":"2024-09-16T05:17:10.733734+0000","last_undegraded":"2024-09-16T05:17:10.733734+0000","last_fullsized":"2024-09-16T05:17:10.733734+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:38.004600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.004600+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:53.010518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408138,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.52","version":"272'69","reported_seq":224,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735390+0000","last_change":"2024-09-16T05:17:08.735450+0000","last_active":"2024-09-16T05:17:09.735390+0000","last_peered":"2024-09-16T05:17:09.735390+0000","last_clean":"2024-09-16T05:17:09.735390+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:17:09.735390+0000","last_undegraded":"2024-09-16T05:17:09.735390+0000","last_fullsized":"2024-09-16T05:17:09.735390+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:16:38.584934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.584934+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:05.950127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094274500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.51","version":"272'115","reported_seq":749,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935827+0000","last_change":"2024-09-16T05:17:10.936027+0000","last_active":"2024-09-16T05:17:10.935827+0000","last_peered":"2024-09-16T05:17:10.935827+0000","last_clean":"2024-09-16T05:17:10.935827+0000","last_became_active":"2024-09-16T05:16:30.731943+0000","last_became_peered":"2024-09-16T05:16:30.731943+0000","last_unstale":"2024-09-16T05:17:10.935827+0000","last_undegraded":"2024-09-16T05:17:10.935827+0000","last_fullsized":"2024-09-16T05:17:10.935827+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:16:38.834396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.834396+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:06.719775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00082524400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.50","version":"182'16","reported_seq":211,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-16T05:17:06.737779+0000","last_change":"2024-09-16T05:17:06.738122+0000","last_active":"2024-09-16T05:17:06.737779+0000","last_peered":"2024-09-16T05:17:06.737779+0000","last_clean":"2024-09-16T05:17:06.737779+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:17:06.737779+0000","last_undegraded":"2024-09-16T05:17:06.737779+0000","last_fullsized":"2024-09-16T05:17:06.737779+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:38.794214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.794214+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:01.521961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057073699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2137,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759708+0000","last_change":"2024-09-16T05:17:08.758723+0000","last_active":"2024-09-16T05:17:09.759708+0000","last_peered":"2024-09-16T05:17:09.759708+0000","last_clean":"2024-09-16T05:17:09.759708+0000","last_became_active":"2024-09-16T05:16:30.732139+0000","last_became_peered":"2024-09-16T05:16:30.732139+0000","last_unstale":"2024-09-16T05:17:09.759708+0000","last_undegraded":"2024-09-16T05:17:09.759708+0000","last_fullsized":"2024-09-16T05:17:09.759708+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:16:38.938962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.938962+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.604232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00099435999999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.4e","version":"272'232","reported_seq":1999,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738860+0000","last_change":"2024-09-16T05:17:10.739103+0000","last_active":"2024-09-16T05:17:10.738860+0000","last_peered":"2024-09-16T05:17:10.738860+0000","last_clean":"2024-09-16T05:17:10.738860+0000","last_became_active":"2024-09-16T05:16:30.732564+0000","last_became_peered":"2024-09-16T05:16:30.732564+0000","last_unstale":"2024-09-16T05:17:10.738860+0000","last_undegraded":"2024-09-16T05:17:10.738860+0000","last_fullsized":"2024-09-16T05:17:10.738860+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:16:25.855040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.855040+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:51.240034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097078599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.4d","version":"467'480","reported_seq":2325,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-16T05:17:07.733336+0000","last_change":"2024-09-16T05:17:06.732835+0000","last_active":"2024-09-16T05:17:07.733336+0000","last_peered":"2024-09-16T05:17:07.733336+0000","last_clean":"2024-09-16T05:17:07.733336+0000","last_became_active":"2024-09-16T05:16:30.731141+0000","last_became_peered":"2024-09-16T05:16:30.731141+0000","last_unstale":"2024-09-16T05:17:07.733336+0000","last_undegraded":"2024-09-16T05:17:07.733336+0000","last_fullsized":"2024-09-16T05:17:07.733336+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"441'465","last_scrub_stamp":"2024-09-16T05:16:37.930310+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.930310+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:39.967769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078215100000000003,"stat_sum":{"num_bytes":3361499,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":18522,"num_write":10,"num_write_kb":2573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"2.4c","version":"466'279","reported_seq":2109,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935319+0000","last_change":"2024-09-16T05:17:10.935522+0000","last_active":"2024-09-16T05:17:10.935319+0000","last_peered":"2024-09-16T05:17:10.935319+0000","last_clean":"2024-09-16T05:17:10.935319+0000","last_became_active":"2024-09-16T05:16:30.731529+0000","last_became_peered":"2024-09-16T05:16:30.731529+0000","last_unstale":"2024-09-16T05:17:10.935319+0000","last_undegraded":"2024-09-16T05:17:10.935319+0000","last_fullsized":"2024-09-16T05:17:10.935319+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:16:39.809684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.809684+0000","objects_scrubbed":1,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.220472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432700000000005,"stat_sum":{"num_bytes":3277464,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.4b","version":"244'177","reported_seq":2038,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760428+0000","last_change":"2024-09-16T05:17:08.758410+0000","last_active":"2024-09-16T05:17:09.760428+0000","last_peered":"2024-09-16T05:17:09.760428+0000","last_clean":"2024-09-16T05:17:09.760428+0000","last_became_active":"2024-09-16T05:14:30.853998+0000","last_became_peered":"2024-09-16T05:14:30.853998+0000","last_unstale":"2024-09-16T05:17:09.760428+0000","last_undegraded":"2024-09-16T05:17:09.760428+0000","last_fullsized":"2024-09-16T05:17:09.760428+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:17:04.875229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:04.875229+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:16.655688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079938899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.4a","version":"459'490","reported_seq":2314,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.934944+0000","last_change":"2024-09-16T05:17:10.935478+0000","last_active":"2024-09-16T05:17:10.934944+0000","last_peered":"2024-09-16T05:17:10.934944+0000","last_clean":"2024-09-16T05:17:10.934944+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:17:10.934944+0000","last_undegraded":"2024-09-16T05:17:10.934944+0000","last_fullsized":"2024-09-16T05:17:10.934944+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"459'490","last_scrub_stamp":"2024-09-16T05:17:00.898492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.898492+0000","objects_scrubbed":1,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:18.434035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00081010499999999998,"stat_sum":{"num_bytes":2840482,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":15,"num_write_kb":7769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.49","version":"245'220","reported_seq":2071,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739488+0000","last_change":"2024-09-16T05:17:10.739651+0000","last_active":"2024-09-16T05:17:10.739488+0000","last_peered":"2024-09-16T05:17:10.739488+0000","last_clean":"2024-09-16T05:17:10.739488+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:17:10.739488+0000","last_undegraded":"2024-09-16T05:17:10.739488+0000","last_fullsized":"2024-09-16T05:17:10.739488+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:16:00.136780+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.136780+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013416229999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.48","version":"451'226","reported_seq":2042,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735889+0000","last_change":"2024-09-16T05:17:08.752794+0000","last_active":"2024-09-16T05:17:09.735889+0000","last_peered":"2024-09-16T05:17:09.735889+0000","last_clean":"2024-09-16T05:17:09.735889+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:17:09.735889+0000","last_undegraded":"2024-09-16T05:17:09.735889+0000","last_fullsized":"2024-09-16T05:17:09.735889+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"451'226","last_scrub_stamp":"2024-09-16T05:17:08.272173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:08.272173+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:33.923787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022016100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20677,"num_write":10,"num_write_kb":8206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.47","version":"0'0","reported_seq":850,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735686+0000","last_change":"2024-09-16T05:17:08.735845+0000","last_active":"2024-09-16T05:17:09.735686+0000","last_peered":"2024-09-16T05:17:09.735686+0000","last_clean":"2024-09-16T05:17:09.735686+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:17:09.735686+0000","last_undegraded":"2024-09-16T05:17:09.735686+0000","last_fullsized":"2024-09-16T05:17:09.735686+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:17:00.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.838891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:13.159576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647560000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.46","version":"116'9","reported_seq":1862,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759714+0000","last_change":"2024-09-16T05:17:08.759068+0000","last_active":"2024-09-16T05:17:09.759714+0000","last_peered":"2024-09-16T05:17:09.759714+0000","last_clean":"2024-09-16T05:17:09.759714+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:17:09.759714+0000","last_undegraded":"2024-09-16T05:17:09.759714+0000","last_fullsized":"2024-09-16T05:17:09.759714+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:17:01.867464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.867464+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:26.762419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012667170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.45","version":"305'464","reported_seq":2281,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760067+0000","last_change":"2024-09-16T05:17:08.758873+0000","last_active":"2024-09-16T05:17:09.760067+0000","last_peered":"2024-09-16T05:17:09.760067+0000","last_clean":"2024-09-16T05:17:09.760067+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:17:09.760067+0000","last_undegraded":"2024-09-16T05:17:09.760067+0000","last_fullsized":"2024-09-16T05:17:09.760067+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:15:59.876915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.876915+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:20.337198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011145090000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.44","version":"182'89","reported_seq":1944,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759737+0000","last_change":"2024-09-16T05:17:08.758361+0000","last_active":"2024-09-16T05:17:09.759737+0000","last_peered":"2024-09-16T05:17:09.759737+0000","last_clean":"2024-09-16T05:17:09.759737+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:17:09.759737+0000","last_undegraded":"2024-09-16T05:17:09.759737+0000","last_fullsized":"2024-09-16T05:17:09.759737+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:16:44.877606+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:16:44.877606+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:10.789776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066299299999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.43","version":"281'58","reported_seq":844,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184522+0000","last_change":"2024-09-16T05:17:11.184765+0000","last_active":"2024-09-16T05:17:11.184522+0000","last_peered":"2024-09-16T05:17:11.184522+0000","last_clean":"2024-09-16T05:17:11.184522+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:17:11.184522+0000","last_undegraded":"2024-09-16T05:17:11.184522+0000","last_fullsized":"2024-09-16T05:17:11.184522+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:16:14.198462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.198462+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:15.625295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072659999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.42","version":"182'20","reported_seq":812,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-16T05:17:08.735900+0000","last_change":"2024-09-16T05:17:08.736177+0000","last_active":"2024-09-16T05:17:08.735900+0000","last_peered":"2024-09-16T05:17:08.735900+0000","last_clean":"2024-09-16T05:17:08.735900+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:17:08.735900+0000","last_undegraded":"2024-09-16T05:17:08.735900+0000","last_fullsized":"2024-09-16T05:17:08.735900+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:16:45.114951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:45.114951+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:46.686694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011309009999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.41","version":"178'50","reported_seq":932,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.936259+0000","last_change":"2024-09-16T05:17:10.936376+0000","last_active":"2024-09-16T05:17:10.936259+0000","last_peered":"2024-09-16T05:17:10.936259+0000","last_clean":"2024-09-16T05:17:10.936259+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:17:10.936259+0000","last_undegraded":"2024-09-16T05:17:10.936259+0000","last_fullsized":"2024-09-16T05:17:10.936259+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:59.922855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:59.922855+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:06.207085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071282699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.40","version":"205'33","reported_seq":293,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739430+0000","last_change":"2024-09-16T05:17:10.739632+0000","last_active":"2024-09-16T05:17:10.739430+0000","last_peered":"2024-09-16T05:17:10.739430+0000","last_clean":"2024-09-16T05:17:10.739430+0000","last_became_active":"2024-09-16T05:16:17.733306+0000","last_became_peered":"2024-09-16T05:16:17.733306+0000","last_unstale":"2024-09-16T05:17:10.739430+0000","last_undegraded":"2024-09-16T05:17:10.739430+0000","last_fullsized":"2024-09-16T05:17:10.739430+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:16:18.066113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.066113+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:39.698653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001321537,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3f","version":"472'312","reported_seq":2171,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185478+0000","last_change":"2024-09-16T05:17:11.185725+0000","last_active":"2024-09-16T05:17:11.185478+0000","last_peered":"2024-09-16T05:17:11.185478+0000","last_clean":"2024-09-16T05:17:11.185478+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:17:11.185478+0000","last_undegraded":"2024-09-16T05:17:11.185478+0000","last_fullsized":"2024-09-16T05:17:11.185478+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"436'302","last_scrub_stamp":"2024-09-16T05:16:43.248069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:43.248069+0000","objects_scrubbed":1,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:00.757639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001476109,"stat_sum":{"num_bytes":2551686,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":22718,"num_write":26,"num_write_kb":6027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3e","version":"474'388","reported_seq":2212,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.931801+0000","last_change":"2024-09-16T05:17:11.932214+0000","last_active":"2024-09-16T05:17:11.931801+0000","last_peered":"2024-09-16T05:17:11.931801+0000","last_clean":"2024-09-16T05:17:11.931801+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:17:11.931801+0000","last_undegraded":"2024-09-16T05:17:11.931801+0000","last_fullsized":"2024-09-16T05:17:11.931801+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"431'368","last_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.868538+0000","objects_scrubbed":3,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:38.553535+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0010406199999999999,"stat_sum":{"num_bytes":6101139,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":18,"num_write_kb":14694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2383,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760232+0000","last_change":"2024-09-16T05:17:08.760712+0000","last_active":"2024-09-16T05:17:09.760232+0000","last_peered":"2024-09-16T05:17:09.760232+0000","last_clean":"2024-09-16T05:17:09.760232+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:17:09.760232+0000","last_undegraded":"2024-09-16T05:17:09.760232+0000","last_fullsized":"2024-09-16T05:17:09.760232+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:16:23.005301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.005301+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:41.874562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025294369999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.3c","version":"481'507","reported_seq":2460,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.187013+0000","last_change":"2024-09-16T05:17:11.187175+0000","last_active":"2024-09-16T05:17:11.187013+0000","last_peered":"2024-09-16T05:17:11.187013+0000","last_clean":"2024-09-16T05:17:11.187013+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:17:11.187013+0000","last_undegraded":"2024-09-16T05:17:11.187013+0000","last_fullsized":"2024-09-16T05:17:11.187013+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"436'490","last_scrub_stamp":"2024-09-16T05:16:26.159696+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.159696+0000","objects_scrubbed":1,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:43.783124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002881116,"stat_sum":{"num_bytes":3109394,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":74,"num_read_kb":82099,"num_write":34,"num_write_kb":18036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3b","version":"481'326","reported_seq":2181,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739382+0000","last_change":"2024-09-16T05:17:10.739382+0000","last_active":"2024-09-16T05:17:10.739382+0000","last_peered":"2024-09-16T05:17:10.739382+0000","last_clean":"2024-09-16T05:17:10.739382+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:17:10.739382+0000","last_undegraded":"2024-09-16T05:17:10.739382+0000","last_fullsized":"2024-09-16T05:17:10.739382+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"431'315","last_scrub_stamp":"2024-09-16T05:16:21.147071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.147071+0000","objects_scrubbed":1,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:45.718220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010659770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":46643,"num_write":11,"num_write_kb":2763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3a","version":"454'520","reported_seq":2329,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735946+0000","last_change":"2024-09-16T05:17:08.751321+0000","last_active":"2024-09-16T05:17:09.735946+0000","last_peered":"2024-09-16T05:17:09.735946+0000","last_clean":"2024-09-16T05:17:09.735946+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:17:09.735946+0000","last_undegraded":"2024-09-16T05:17:09.735946+0000","last_fullsized":"2024-09-16T05:17:09.735946+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"454'520","last_scrub_stamp":"2024-09-16T05:17:01.101516+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.101516+0000","objects_scrubbed":1,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:20.575814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00166676,"stat_sum":{"num_bytes":2000132,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":24517,"num_write":37,"num_write_kb":15654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.39","version":"182'60","reported_seq":930,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735918+0000","last_change":"2024-09-16T05:17:08.751574+0000","last_active":"2024-09-16T05:17:09.735918+0000","last_peered":"2024-09-16T05:17:09.735918+0000","last_clean":"2024-09-16T05:17:09.735918+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:17:09.735918+0000","last_undegraded":"2024-09-16T05:17:09.735918+0000","last_fullsized":"2024-09-16T05:17:09.735918+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:17:02.077685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:02.077685+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015512049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.38","version":"181'57","reported_seq":925,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-16T05:17:08.735065+0000","last_change":"2024-09-16T05:17:08.735502+0000","last_active":"2024-09-16T05:17:08.735065+0000","last_peered":"2024-09-16T05:17:08.735065+0000","last_clean":"2024-09-16T05:17:08.735065+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:17:08.735065+0000","last_undegraded":"2024-09-16T05:17:08.735065+0000","last_fullsized":"2024-09-16T05:17:08.735065+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:16:17.338199+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.338199+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:21.565353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072274100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.37","version":"0'0","reported_seq":1020,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935580+0000","last_change":"2024-09-16T05:17:10.935763+0000","last_active":"2024-09-16T05:17:10.935580+0000","last_peered":"2024-09-16T05:17:10.935580+0000","last_clean":"2024-09-16T05:17:10.935580+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:17:10.935580+0000","last_undegraded":"2024-09-16T05:17:10.935580+0000","last_fullsized":"2024-09-16T05:17:10.935580+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:16:21.830665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.830665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.405915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074533399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.36","version":"272'60","reported_seq":1011,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759881+0000","last_change":"2024-09-16T05:17:08.759682+0000","last_active":"2024-09-16T05:17:09.759881+0000","last_peered":"2024-09-16T05:17:09.759881+0000","last_clean":"2024-09-16T05:17:09.759881+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:17:09.759881+0000","last_undegraded":"2024-09-16T05:17:09.759881+0000","last_fullsized":"2024-09-16T05:17:09.759881+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.904155+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.244733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001659743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.35","version":"479'543","reported_seq":2414,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760455+0000","last_change":"2024-09-16T05:17:08.760006+0000","last_active":"2024-09-16T05:17:09.760455+0000","last_peered":"2024-09-16T05:17:09.760455+0000","last_clean":"2024-09-16T05:17:09.760455+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:17:09.760455+0000","last_undegraded":"2024-09-16T05:17:09.760455+0000","last_fullsized":"2024-09-16T05:17:09.760455+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"410'508","last_scrub_stamp":"2024-09-16T05:16:00.864640+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.864640+0000","objects_scrubbed":4,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019585169999999999,"stat_sum":{"num_bytes":3411920,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":46,"num_read_kb":43632,"num_write":37,"num_write_kb":15224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.34","version":"467'359","reported_seq":1821,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739222+0000","last_change":"2024-09-16T05:17:10.739406+0000","last_active":"2024-09-16T05:17:10.739222+0000","last_peered":"2024-09-16T05:17:10.739222+0000","last_clean":"2024-09-16T05:17:10.739222+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:17:10.739222+0000","last_undegraded":"2024-09-16T05:17:10.739222+0000","last_fullsized":"2024-09-16T05:17:10.739222+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"420'335","last_scrub_stamp":"2024-09-16T05:16:15.093220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:15.093220+0000","objects_scrubbed":4,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.803909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011185500000000001,"stat_sum":{"num_bytes":5815387,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":76675,"num_write":31,"num_write_kb":15494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.33","version":"181'15","reported_seq":1022,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185110+0000","last_change":"2024-09-16T05:17:11.185417+0000","last_active":"2024-09-16T05:17:11.185110+0000","last_peered":"2024-09-16T05:17:11.185110+0000","last_clean":"2024-09-16T05:17:11.185110+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:17:11.185110+0000","last_undegraded":"2024-09-16T05:17:11.185110+0000","last_fullsized":"2024-09-16T05:17:11.185110+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:25.199845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.199845+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:31.585581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012291940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.32","version":"451'141","reported_seq":1131,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.930959+0000","last_change":"2024-09-16T05:17:11.931472+0000","last_active":"2024-09-16T05:17:11.930959+0000","last_peered":"2024-09-16T05:17:11.930959+0000","last_clean":"2024-09-16T05:17:11.930959+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:17:11.930959+0000","last_undegraded":"2024-09-16T05:17:11.930959+0000","last_fullsized":"2024-09-16T05:17:11.930959+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"419'127","last_scrub_stamp":"2024-09-16T05:16:18.859832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.859832+0000","objects_scrubbed":1,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:33.735551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000696399,"stat_sum":{"num_bytes":3109361,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":35253,"num_write":19,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.31","version":"459'204","reported_seq":1694,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735492+0000","last_change":"2024-09-16T05:17:08.735349+0000","last_active":"2024-09-16T05:17:09.735492+0000","last_peered":"2024-09-16T05:17:09.735492+0000","last_clean":"2024-09-16T05:17:09.735492+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:17:09.735492+0000","last_undegraded":"2024-09-16T05:17:09.735492+0000","last_fullsized":"2024-09-16T05:17:09.735492+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"459'204","last_scrub_stamp":"2024-09-16T05:17:03.863715+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.863715+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:20.160117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080613400000000002,"stat_sum":{"num_bytes":3176622,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":62,"num_read_kb":68251,"num_write":27,"num_write_kb":10906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.30","version":"182'16","reported_seq":984,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.934997+0000","last_change":"2024-09-16T05:17:10.935230+0000","last_active":"2024-09-16T05:17:10.934997+0000","last_peered":"2024-09-16T05:17:10.934997+0000","last_clean":"2024-09-16T05:17:10.934997+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:17:10.934997+0000","last_undegraded":"2024-09-16T05:17:10.934997+0000","last_fullsized":"2024-09-16T05:17:10.934997+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:24.905903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:24.905903+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.392571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053850299999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.12","version":"35'139","reported_seq":566,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738471+0000","last_change":"2024-09-16T05:15:57.054819+0000","last_active":"2024-09-16T05:17:10.738471+0000","last_peered":"2024-09-16T05:17:10.738471+0000","last_clean":"2024-09-16T05:17:10.738471+0000","last_became_active":"2024-09-16T05:15:56.716090+0000","last_became_peered":"2024-09-16T05:15:56.716090+0000","last_unstale":"2024-09-16T05:17:10.738471+0000","last_undegraded":"2024-09-16T05:17:10.738471+0000","last_fullsized":"2024-09-16T05:17:10.738471+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.054740+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.054740+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.994935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1576,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.740027+0000","last_change":"2024-09-16T05:17:11.740776+0000","last_active":"2024-09-16T05:17:11.740027+0000","last_peered":"2024-09-16T05:17:11.740027+0000","last_clean":"2024-09-16T05:17:11.740027+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:17:11.740027+0000","last_undegraded":"2024-09-16T05:17:11.740027+0000","last_fullsized":"2024-09-16T05:17:11.740027+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:16:26.623221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.623221+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:55.934505+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0.00080160599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1197,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760285+0000","last_change":"2024-09-16T05:15:56.852148+0000","last_active":"2024-09-16T05:17:09.760285+0000","last_peered":"2024-09-16T05:17:09.760285+0000","last_clean":"2024-09-16T05:17:09.760285+0000","last_became_active":"2024-09-16T05:15:56.714681+0000","last_became_peered":"2024-09-16T05:15:56.714681+0000","last_unstale":"2024-09-16T05:17:09.760285+0000","last_undegraded":"2024-09-16T05:17:09.760285+0000","last_fullsized":"2024-09-16T05:17:09.760285+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.852021+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.852021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:11.168034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":1475,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-16T05:17:08.735122+0000","last_change":"2024-09-16T05:17:08.735457+0000","last_active":"2024-09-16T05:17:08.735122+0000","last_peered":"2024-09-16T05:17:08.735122+0000","last_clean":"2024-09-16T05:17:08.735122+0000","last_became_active":"2024-09-16T05:16:25.731838+0000","last_became_peered":"2024-09-16T05:16:25.731838+0000","last_unstale":"2024-09-16T05:17:08.735122+0000","last_undegraded":"2024-09-16T05:17:08.735122+0000","last_fullsized":"2024-09-16T05:17:08.735122+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:26.399340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.399340+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:47.868621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057342700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.c","version":"35'139","reported_seq":511,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-16T05:17:06.737667+0000","last_change":"2024-09-16T05:15:56.828705+0000","last_active":"2024-09-16T05:17:06.737667+0000","last_peered":"2024-09-16T05:17:06.737667+0000","last_clean":"2024-09-16T05:17:06.737667+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:17:06.737667+0000","last_undegraded":"2024-09-16T05:17:06.737667+0000","last_fullsized":"2024-09-16T05:17:06.737667+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.828595+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.828595+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:25.780000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"479'287","reported_seq":2146,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-16T05:17:08.802084+0000","last_change":"2024-09-16T05:17:08.736369+0000","last_active":"2024-09-16T05:17:08.802084+0000","last_peered":"2024-09-16T05:17:08.802084+0000","last_clean":"2024-09-16T05:17:08.802084+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:17:08.802084+0000","last_undegraded":"2024-09-16T05:17:08.802084+0000","last_fullsized":"2024-09-16T05:17:08.802084+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"437'282","last_scrub_stamp":"2024-09-16T05:16:38.132713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.132713+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:51.683408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001329769,"stat_sum":{"num_bytes":3815288,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.d","version":"35'139","reported_seq":685,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184346+0000","last_change":"2024-09-16T05:15:57.170014+0000","last_active":"2024-09-16T05:17:11.184346+0000","last_peered":"2024-09-16T05:17:11.184346+0000","last_clean":"2024-09-16T05:17:11.184346+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:17:11.184346+0000","last_undegraded":"2024-09-16T05:17:11.184346+0000","last_fullsized":"2024-09-16T05:17:11.184346+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.169945+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.169945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.932365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":1983,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-16T05:17:07.733142+0000","last_change":"2024-09-16T05:17:06.733200+0000","last_active":"2024-09-16T05:17:07.733142+0000","last_peered":"2024-09-16T05:17:07.733142+0000","last_clean":"2024-09-16T05:17:07.733142+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:17:07.733142+0000","last_undegraded":"2024-09-16T05:17:07.733142+0000","last_fullsized":"2024-09-16T05:17:07.733142+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:16:38.980110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.980110+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:51.192793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001091194,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"1.e","version":"35'139","reported_seq":683,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.733202+0000","last_change":"2024-09-16T05:17:01.066059+0000","last_active":"2024-09-16T05:17:10.733202+0000","last_peered":"2024-09-16T05:17:10.733202+0000","last_clean":"2024-09-16T05:17:10.733202+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:17:10.733202+0000","last_undegraded":"2024-09-16T05:17:10.733202+0000","last_fullsized":"2024-09-16T05:17:10.733202+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:01.065967+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.065967+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:26.866764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"479'485","reported_seq":2344,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736543+0000","last_change":"2024-09-16T05:17:08.755601+0000","last_active":"2024-09-16T05:17:09.736543+0000","last_peered":"2024-09-16T05:17:09.736543+0000","last_clean":"2024-09-16T05:17:09.736543+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:17:09.736543+0000","last_undegraded":"2024-09-16T05:17:09.736543+0000","last_fullsized":"2024-09-16T05:17:09.736543+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'461","last_scrub_stamp":"2024-09-16T05:16:40.073052+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:16:40.073052+0000","objects_scrubbed":1,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:40.544503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040297090000000002,"stat_sum":{"num_bytes":8777406,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":14,"num_write_kb":12109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.f","version":"35'139","reported_seq":666,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738954+0000","last_change":"2024-09-16T05:15:59.140824+0000","last_active":"2024-09-16T05:17:10.738954+0000","last_peered":"2024-09-16T05:17:10.738954+0000","last_clean":"2024-09-16T05:17:10.738954+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:17:10.738954+0000","last_undegraded":"2024-09-16T05:17:10.738954+0000","last_fullsized":"2024-09-16T05:17:10.738954+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.140738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.775574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":2132,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.931519+0000","last_change":"2024-09-16T05:17:11.931745+0000","last_active":"2024-09-16T05:17:11.931519+0000","last_peered":"2024-09-16T05:17:11.931519+0000","last_clean":"2024-09-16T05:17:11.931519+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:17:11.931519+0000","last_undegraded":"2024-09-16T05:17:11.931519+0000","last_fullsized":"2024-09-16T05:17:11.931519+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:16:25.811761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.811761+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.310216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082498399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1209,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.930753+0000","last_change":"2024-09-16T05:17:07.872302+0000","last_active":"2024-09-16T05:17:11.930753+0000","last_peered":"2024-09-16T05:17:11.930753+0000","last_clean":"2024-09-16T05:17:11.930753+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:17:11.930753+0000","last_undegraded":"2024-09-16T05:17:11.930753+0000","last_fullsized":"2024-09-16T05:17:11.930753+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:17:07.872208+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:11.877781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"451'255","reported_seq":2108,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185848+0000","last_change":"2024-09-16T05:17:11.186064+0000","last_active":"2024-09-16T05:17:11.185848+0000","last_peered":"2024-09-16T05:17:11.185848+0000","last_clean":"2024-09-16T05:17:11.185848+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:17:11.185848+0000","last_undegraded":"2024-09-16T05:17:11.185848+0000","last_fullsized":"2024-09-16T05:17:11.185848+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'222","last_scrub_stamp":"2024-09-16T05:16:00.063191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.063191+0000","objects_scrubbed":3,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:28.886742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016896229999999999,"stat_sum":{"num_bytes":2890903,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":12882,"num_write":25,"num_write_kb":5945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3","version":"472'130","reported_seq":1988,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739728+0000","last_change":"2024-09-16T05:17:10.739916+0000","last_active":"2024-09-16T05:17:10.739728+0000","last_peered":"2024-09-16T05:17:10.739728+0000","last_clean":"2024-09-16T05:17:10.739728+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:17:10.739728+0000","last_undegraded":"2024-09-16T05:17:10.739728+0000","last_fullsized":"2024-09-16T05:17:10.739728+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'99","last_scrub_stamp":"2024-09-16T05:16:02.132568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.132568+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:17.257413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001307122,"stat_sum":{"num_bytes":2487535,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":49445,"num_write":22,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.0","version":"35'139","reported_seq":1194,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184683+0000","last_change":"2024-09-16T05:15:53.271304+0000","last_active":"2024-09-16T05:17:11.184683+0000","last_peered":"2024-09-16T05:17:11.184683+0000","last_clean":"2024-09-16T05:17:11.184683+0000","last_became_active":"2024-09-16T05:15:51.717864+0000","last_became_peered":"2024-09-16T05:15:51.717864+0000","last_unstale":"2024-09-16T05:17:11.184683+0000","last_undegraded":"2024-09-16T05:17:11.184683+0000","last_fullsized":"2024-09-16T05:17:11.184683+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.271201+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.271201+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.039968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1866,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739799+0000","last_change":"2024-09-16T05:17:10.739962+0000","last_active":"2024-09-16T05:17:10.739799+0000","last_peered":"2024-09-16T05:17:10.739799+0000","last_clean":"2024-09-16T05:17:10.739799+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:17:10.739799+0000","last_undegraded":"2024-09-16T05:17:10.739799+0000","last_fullsized":"2024-09-16T05:17:10.739799+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:17:06.386786+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:17.599220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011944049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.1","version":"35'139","reported_seq":842,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935450+0000","last_change":"2024-09-16T05:17:06.854768+0000","last_active":"2024-09-16T05:17:10.935450+0000","last_peered":"2024-09-16T05:17:10.935450+0000","last_clean":"2024-09-16T05:17:10.935450+0000","last_became_active":"2024-09-16T05:15:51.723644+0000","last_became_peered":"2024-09-16T05:15:51.723644+0000","last_unstale":"2024-09-16T05:17:10.935450+0000","last_undegraded":"2024-09-16T05:17:10.935450+0000","last_fullsized":"2024-09-16T05:17:10.935450+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:06.854660+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:06.854660+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:21.485402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1198,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.183662+0000","last_change":"2024-09-16T05:17:03.402949+0000","last_active":"2024-09-16T05:17:11.183662+0000","last_peered":"2024-09-16T05:17:11.183662+0000","last_clean":"2024-09-16T05:17:11.183662+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:17:11.183662+0000","last_undegraded":"2024-09-16T05:17:11.183662+0000","last_fullsized":"2024-09-16T05:17:11.183662+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:03.402860+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.402860+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:12.276154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1792,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-16T05:17:07.733051+0000","last_change":"2024-09-16T05:17:06.732063+0000","last_active":"2024-09-16T05:17:07.733051+0000","last_peered":"2024-09-16T05:17:07.733051+0000","last_clean":"2024-09-16T05:17:07.733051+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:17:07.733051+0000","last_undegraded":"2024-09-16T05:17:07.733051+0000","last_fullsized":"2024-09-16T05:17:07.733051+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:23.812825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.812825+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.802026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371209,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"1.3","version":"35'139","reported_seq":1192,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-16T05:17:06.737584+0000","last_change":"2024-09-16T05:15:52.844991+0000","last_active":"2024-09-16T05:17:06.737584+0000","last_peered":"2024-09-16T05:17:06.737584+0000","last_clean":"2024-09-16T05:17:06.737584+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:17:06.737584+0000","last_undegraded":"2024-09-16T05:17:06.737584+0000","last_fullsized":"2024-09-16T05:17:06.737584+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:52.844849+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:52.844849+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1888,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.186113+0000","last_change":"2024-09-16T05:17:11.186271+0000","last_active":"2024-09-16T05:17:11.186113+0000","last_peered":"2024-09-16T05:17:11.186113+0000","last_clean":"2024-09-16T05:17:11.186113+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:17:11.186113+0000","last_undegraded":"2024-09-16T05:17:11.186113+0000","last_fullsized":"2024-09-16T05:17:11.186113+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:16:46.325304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:46.325304+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:07.467388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018838310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.7","version":"0'0","reported_seq":1100,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736412+0000","last_change":"2024-09-16T05:17:08.753457+0000","last_active":"2024-09-16T05:17:09.736412+0000","last_peered":"2024-09-16T05:17:09.736412+0000","last_clean":"2024-09-16T05:17:09.736412+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:17:09.736412+0000","last_undegraded":"2024-09-16T05:17:09.736412+0000","last_fullsized":"2024-09-16T05:17:09.736412+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:17:05.143266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:05.143266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:14.074436+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025639149999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.4","version":"35'139","reported_seq":1165,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.930988+0000","last_change":"2024-09-16T05:16:58.840947+0000","last_active":"2024-09-16T05:17:11.930988+0000","last_peered":"2024-09-16T05:17:11.930988+0000","last_clean":"2024-09-16T05:17:11.930988+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:17:11.930988+0000","last_undegraded":"2024-09-16T05:17:11.930988+0000","last_fullsized":"2024-09-16T05:17:11.930988+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:58.840859+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:58.840859+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:20.477391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":274,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-16T05:17:07.733079+0000","last_change":"2024-09-16T05:17:06.732466+0000","last_active":"2024-09-16T05:17:07.733079+0000","last_peered":"2024-09-16T05:17:07.733079+0000","last_clean":"2024-09-16T05:17:07.733079+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:17:07.733079+0000","last_undegraded":"2024-09-16T05:17:07.733079+0000","last_fullsized":"2024-09-16T05:17:07.733079+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:17:00.760543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.760543+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:21.522619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057989999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":939,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935392+0000","last_change":"2024-09-16T05:15:56.607423+0000","last_active":"2024-09-16T05:17:10.935392+0000","last_peered":"2024-09-16T05:17:10.935392+0000","last_clean":"2024-09-16T05:17:10.935392+0000","last_became_active":"2024-09-16T05:15:51.724156+0000","last_became_peered":"2024-09-16T05:15:51.724156+0000","last_unstale":"2024-09-16T05:17:10.935392+0000","last_undegraded":"2024-09-16T05:17:10.935392+0000","last_fullsized":"2024-09-16T05:17:10.935392+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.607332+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.607332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.940520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"443'485","reported_seq":2318,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736402+0000","last_change":"2024-09-16T05:17:08.753959+0000","last_active":"2024-09-16T05:17:09.736402+0000","last_peered":"2024-09-16T05:17:09.736402+0000","last_clean":"2024-09-16T05:17:09.736402+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:17:09.736402+0000","last_undegraded":"2024-09-16T05:17:09.736402+0000","last_fullsized":"2024-09-16T05:17:09.736402+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'479","last_scrub_stamp":"2024-09-16T05:15:57.067519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.067519+0000","objects_scrubbed":2,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:12.036605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028257109999999998,"stat_sum":{"num_bytes":2672412,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20566,"num_write":5,"num_write_kb":1272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.6","version":"35'139","reported_seq":835,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.934678+0000","last_change":"2024-09-16T05:15:55.644520+0000","last_active":"2024-09-16T05:17:10.934678+0000","last_peered":"2024-09-16T05:17:10.934678+0000","last_clean":"2024-09-16T05:17:10.934678+0000","last_became_active":"2024-09-16T05:15:51.722851+0000","last_became_peered":"2024-09-16T05:15:51.722851+0000","last_unstale":"2024-09-16T05:17:10.934678+0000","last_undegraded":"2024-09-16T05:17:10.934678+0000","last_fullsized":"2024-09-16T05:17:10.934678+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:55.644436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.644436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.940555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1933,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759986+0000","last_change":"2024-09-16T05:17:08.760306+0000","last_active":"2024-09-16T05:17:09.759986+0000","last_peered":"2024-09-16T05:17:09.759986+0000","last_clean":"2024-09-16T05:17:09.759986+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:17:09.759986+0000","last_undegraded":"2024-09-16T05:17:09.759986+0000","last_fullsized":"2024-09-16T05:17:09.759986+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:57.869742+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.869742+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.876676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0021328609999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.7","version":"35'139","reported_seq":1019,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.934809+0000","last_change":"2024-09-16T05:15:53.643617+0000","last_active":"2024-09-16T05:17:10.934809+0000","last_peered":"2024-09-16T05:17:10.934809+0000","last_clean":"2024-09-16T05:17:10.934809+0000","last_became_active":"2024-09-16T05:15:51.718810+0000","last_became_peered":"2024-09-16T05:15:51.718810+0000","last_unstale":"2024-09-16T05:17:10.934809+0000","last_undegraded":"2024-09-16T05:17:10.934809+0000","last_fullsized":"2024-09-16T05:17:10.934809+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.643528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.643528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.940586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":2030,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-16T05:17:08.735551+0000","last_change":"2024-09-16T05:17:08.735820+0000","last_active":"2024-09-16T05:17:08.735551+0000","last_peered":"2024-09-16T05:17:08.735551+0000","last_clean":"2024-09-16T05:17:08.735551+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:17:08.735551+0000","last_undegraded":"2024-09-16T05:17:08.735551+0000","last_fullsized":"2024-09-16T05:17:08.735551+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:16:58.139614+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:16:58.139614+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.993303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088042299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.8","version":"35'139","reported_seq":1195,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738394+0000","last_change":"2024-09-16T05:16:20.118747+0000","last_active":"2024-09-16T05:17:10.738394+0000","last_peered":"2024-09-16T05:17:10.738394+0000","last_clean":"2024-09-16T05:17:10.738394+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:17:10.738394+0000","last_undegraded":"2024-09-16T05:17:10.738394+0000","last_fullsized":"2024-09-16T05:17:10.738394+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:20.118658+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.118658+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.459808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"479'489","reported_seq":2364,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.931021+0000","last_change":"2024-09-16T05:17:11.931329+0000","last_active":"2024-09-16T05:17:11.931021+0000","last_peered":"2024-09-16T05:17:11.931021+0000","last_clean":"2024-09-16T05:17:11.931021+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:17:11.931021+0000","last_undegraded":"2024-09-16T05:17:11.931021+0000","last_fullsized":"2024-09-16T05:17:11.931021+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'465","last_scrub_stamp":"2024-09-16T05:16:17.822233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.822233+0000","objects_scrubbed":1,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:17.927878+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00053640199999999995,"stat_sum":{"num_bytes":5215067,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":66423,"num_write":18,"num_write_kb":14367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1186,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.183599+0000","last_change":"2024-09-16T05:16:18.233415+0000","last_active":"2024-09-16T05:17:11.183599+0000","last_peered":"2024-09-16T05:17:11.183599+0000","last_clean":"2024-09-16T05:17:11.183599+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:17:11.183599+0000","last_undegraded":"2024-09-16T05:17:11.183599+0000","last_fullsized":"2024-09-16T05:17:11.183599+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:18.233282+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.233282+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:35.750725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2078,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759568+0000","last_change":"2024-09-16T05:17:08.757958+0000","last_active":"2024-09-16T05:17:09.759568+0000","last_peered":"2024-09-16T05:17:09.759568+0000","last_clean":"2024-09-16T05:17:09.759568+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:17:09.759568+0000","last_undegraded":"2024-09-16T05:17:09.759568+0000","last_fullsized":"2024-09-16T05:17:09.759568+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:16:39.912418+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.912418+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:03.966880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000608129,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.a","version":"35'139","reported_seq":1193,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736024+0000","last_change":"2024-09-16T05:16:20.111881+0000","last_active":"2024-09-16T05:17:09.736024+0000","last_peered":"2024-09-16T05:17:09.736024+0000","last_clean":"2024-09-16T05:17:09.736024+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:17:09.736024+0000","last_undegraded":"2024-09-16T05:17:09.736024+0000","last_fullsized":"2024-09-16T05:17:09.736024+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:20.111791+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.111791+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:28.416563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":675,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735303+0000","last_change":"2024-09-16T05:17:08.734411+0000","last_active":"2024-09-16T05:17:09.735303+0000","last_peered":"2024-09-16T05:17:09.735303+0000","last_clean":"2024-09-16T05:17:09.735303+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:17:09.735303+0000","last_undegraded":"2024-09-16T05:17:09.735303+0000","last_fullsized":"2024-09-16T05:17:09.735303+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.743052+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.743052+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:12.904484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":995,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935774+0000","last_change":"2024-09-16T05:17:10.935949+0000","last_active":"2024-09-16T05:17:10.935774+0000","last_peered":"2024-09-16T05:17:10.935774+0000","last_clean":"2024-09-16T05:17:10.935774+0000","last_became_active":"2024-09-16T05:16:25.724548+0000","last_became_peered":"2024-09-16T05:16:25.724548+0000","last_unstale":"2024-09-16T05:17:10.935774+0000","last_undegraded":"2024-09-16T05:17:10.935774+0000","last_fullsized":"2024-09-16T05:17:10.935774+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:16:25.926404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.926404+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:48.534554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090903499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.10","version":"35'139","reported_seq":1193,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738164+0000","last_change":"2024-09-16T05:15:58.103711+0000","last_active":"2024-09-16T05:17:10.738164+0000","last_peered":"2024-09-16T05:17:10.738164+0000","last_clean":"2024-09-16T05:17:10.738164+0000","last_became_active":"2024-09-16T05:15:56.719062+0000","last_became_peered":"2024-09-16T05:15:56.719062+0000","last_unstale":"2024-09-16T05:17:10.738164+0000","last_undegraded":"2024-09-16T05:17:10.738164+0000","last_fullsized":"2024-09-16T05:17:10.738164+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:58.103601+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:58.103601+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:27.359233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":1482,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736149+0000","last_change":"2024-09-16T05:17:08.750659+0000","last_active":"2024-09-16T05:17:09.736149+0000","last_peered":"2024-09-16T05:17:09.736149+0000","last_clean":"2024-09-16T05:17:09.736149+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:17:09.736149+0000","last_undegraded":"2024-09-16T05:17:09.736149+0000","last_fullsized":"2024-09-16T05:17:09.736149+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:26.057442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.057442+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:35.072063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010375880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.14","version":"433'289","reported_seq":1754,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736236+0000","last_change":"2024-09-16T05:17:08.752241+0000","last_active":"2024-09-16T05:17:09.736236+0000","last_peered":"2024-09-16T05:17:09.736236+0000","last_clean":"2024-09-16T05:17:09.736236+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:17:09.736236+0000","last_undegraded":"2024-09-16T05:17:09.736236+0000","last_fullsized":"2024-09-16T05:17:09.736236+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:16:37.997840+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.997840+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:04.271517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020694530000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.15","version":"454'519","reported_seq":2385,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736424+0000","last_change":"2024-09-16T05:17:08.754356+0000","last_active":"2024-09-16T05:17:09.736424+0000","last_peered":"2024-09-16T05:17:09.736424+0000","last_clean":"2024-09-16T05:17:09.736424+0000","last_became_active":"2024-09-16T05:16:25.732256+0000","last_became_peered":"2024-09-16T05:16:25.732256+0000","last_unstale":"2024-09-16T05:17:09.736424+0000","last_undegraded":"2024-09-16T05:17:09.736424+0000","last_fullsized":"2024-09-16T05:17:09.736424+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"438'513","last_scrub_stamp":"2024-09-16T05:16:39.040839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.040839+0000","objects_scrubbed":1,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:00.672366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031403799999999999,"stat_sum":{"num_bytes":3008552,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.16","version":"481'180","reported_seq":1677,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.751665+0000","last_change":"2024-09-16T05:17:10.740164+0000","last_active":"2024-09-16T05:17:10.751665+0000","last_peered":"2024-09-16T05:17:10.751665+0000","last_clean":"2024-09-16T05:17:10.751665+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:17:10.751665+0000","last_undegraded":"2024-09-16T05:17:10.751665+0000","last_fullsized":"2024-09-16T05:17:10.751665+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"451'178","last_scrub_stamp":"2024-09-16T05:16:59.339858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:59.339858+0000","objects_scrubbed":1,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:01.419261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011560940000000001,"stat_sum":{"num_bytes":4420439,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":25313,"num_write":23,"num_write_kb":9066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.14","version":"35'139","reported_seq":515,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-16T05:17:07.733066+0000","last_change":"2024-09-16T05:16:17.723651+0000","last_active":"2024-09-16T05:17:07.733066+0000","last_peered":"2024-09-16T05:17:07.733066+0000","last_clean":"2024-09-16T05:17:07.733066+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:17:07.733066+0000","last_undegraded":"2024-09-16T05:17:07.733066+0000","last_fullsized":"2024-09-16T05:17:07.733066+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.762419+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.762419+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:08.609426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1659,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.740692+0000","last_change":"2024-09-16T05:17:11.741236+0000","last_active":"2024-09-16T05:17:11.740692+0000","last_peered":"2024-09-16T05:17:11.740692+0000","last_clean":"2024-09-16T05:17:11.740692+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:17:11.740692+0000","last_undegraded":"2024-09-16T05:17:11.740692+0000","last_fullsized":"2024-09-16T05:17:11.740692+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:57.269939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.269939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099584299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1864,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735878+0000","last_change":"2024-09-16T05:17:08.753948+0000","last_active":"2024-09-16T05:17:09.735878+0000","last_peered":"2024-09-16T05:17:09.735878+0000","last_clean":"2024-09-16T05:17:09.735878+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:17:09.735878+0000","last_undegraded":"2024-09-16T05:17:09.735878+0000","last_fullsized":"2024-09-16T05:17:09.735878+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:16:44.037920+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:16:44.037920+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:52.405077+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030712309999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.19","version":"182'60","reported_seq":1454,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-16T05:17:06.737844+0000","last_change":"2024-09-16T05:17:06.738354+0000","last_active":"2024-09-16T05:17:06.737844+0000","last_peered":"2024-09-16T05:17:06.737844+0000","last_clean":"2024-09-16T05:17:06.737844+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:17:06.737844+0000","last_undegraded":"2024-09-16T05:17:06.737844+0000","last_fullsized":"2024-09-16T05:17:06.737844+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:16:19.812615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.812615+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:20.986511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071690299999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"2.1a","version":"472'688","reported_seq":2509,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-16T05:17:08.736223+0000","last_change":"2024-09-16T05:17:08.736459+0000","last_active":"2024-09-16T05:17:08.736223+0000","last_peered":"2024-09-16T05:17:08.736223+0000","last_clean":"2024-09-16T05:17:08.736223+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:17:08.736223+0000","last_undegraded":"2024-09-16T05:17:08.736223+0000","last_fullsized":"2024-09-16T05:17:08.736223+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"435'666","last_scrub_stamp":"2024-09-16T05:16:25.407253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.407253+0000","objects_scrubbed":3,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:40.984719+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013113510000000001,"stat_sum":{"num_bytes":8428471,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":80160,"num_write":61,"num_write_kb":28646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.1b","version":"482'410","reported_seq":2294,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.932756+0000","last_change":"2024-09-16T05:17:11.933100+0000","last_active":"2024-09-16T05:17:11.932756+0000","last_peered":"2024-09-16T05:17:11.932756+0000","last_clean":"2024-09-16T05:17:11.932756+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:17:11.932756+0000","last_undegraded":"2024-09-16T05:17:11.932756+0000","last_fullsized":"2024-09-16T05:17:11.932756+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"429'364","last_scrub_stamp":"2024-09-16T05:16:19.854079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.854079+0000","objects_scrubbed":4,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:46.585785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021804319999999999,"stat_sum":{"num_bytes":9206293,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":40185,"num_write":34,"num_write_kb":10899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.1c","version":"468'393","reported_seq":2305,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935529+0000","last_change":"2024-09-16T05:17:10.935739+0000","last_active":"2024-09-16T05:17:10.935529+0000","last_peered":"2024-09-16T05:17:10.935529+0000","last_clean":"2024-09-16T05:17:10.935529+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:17:10.935529+0000","last_undegraded":"2024-09-16T05:17:10.935529+0000","last_fullsized":"2024-09-16T05:17:10.935529+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'392","last_scrub_stamp":"2024-09-16T05:16:45.755860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:45.755860+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:06.827595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096189500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.1d","version":"477'821","reported_seq":2755,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.936071+0000","last_change":"2024-09-16T05:17:10.936258+0000","last_active":"2024-09-16T05:17:10.936071+0000","last_peered":"2024-09-16T05:17:10.936071+0000","last_clean":"2024-09-16T05:17:10.936071+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:17:10.936071+0000","last_undegraded":"2024-09-16T05:17:10.936071+0000","last_fullsized":"2024-09-16T05:17:10.936071+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"457'781","last_scrub_stamp":"2024-09-16T05:16:46.788437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:46.788437+0000","objects_scrubbed":4,"log_size":821,"log_dups_size":0,"ondisk_log_size":821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:57.901598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096913800000000003,"stat_sum":{"num_bytes":14083730,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":91294,"num_write":111,"num_write_kb":42451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.1e","version":"436'363","reported_seq":2220,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-16T05:17:06.738655+0000","last_change":"2024-09-16T05:17:06.739183+0000","last_active":"2024-09-16T05:17:06.738655+0000","last_peered":"2024-09-16T05:17:06.738655+0000","last_clean":"2024-09-16T05:17:06.738655+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:17:06.738655+0000","last_undegraded":"2024-09-16T05:17:06.738655+0000","last_fullsized":"2024-09-16T05:17:06.738655+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"436'363","last_scrub_stamp":"2024-09-16T05:16:39.785728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.785728+0000","objects_scrubbed":1,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:42.122225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001189708,"stat_sum":{"num_bytes":2067360,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":26094,"num_write":9,"num_write_kb":4850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"2.1f","version":"284'221","reported_seq":2081,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735753+0000","last_change":"2024-09-16T05:17:08.735935+0000","last_active":"2024-09-16T05:17:09.735753+0000","last_peered":"2024-09-16T05:17:09.735753+0000","last_clean":"2024-09-16T05:17:09.735753+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:17:09.735753+0000","last_undegraded":"2024-09-16T05:17:09.735753+0000","last_fullsized":"2024-09-16T05:17:09.735753+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.634458+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:48.717367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012194689999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.20","version":"472'179","reported_seq":2061,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184396+0000","last_change":"2024-09-16T05:17:11.185060+0000","last_active":"2024-09-16T05:17:11.184396+0000","last_peered":"2024-09-16T05:17:11.184396+0000","last_clean":"2024-09-16T05:17:11.184396+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:17:11.184396+0000","last_undegraded":"2024-09-16T05:17:11.184396+0000","last_fullsized":"2024-09-16T05:17:11.184396+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"451'174","last_scrub_stamp":"2024-09-16T05:17:01.386565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.386565+0000","objects_scrubbed":1,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:22.137223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012705920000000001,"stat_sum":{"num_bytes":4887302,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":93739,"num_write":26,"num_write_kb":11335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.21","version":"178'50","reported_seq":1791,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-16T05:17:07.733185+0000","last_change":"2024-09-16T05:17:06.732104+0000","last_active":"2024-09-16T05:17:07.733185+0000","last_peered":"2024-09-16T05:17:07.733185+0000","last_clean":"2024-09-16T05:17:07.733185+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:17:07.733185+0000","last_undegraded":"2024-09-16T05:17:07.733185+0000","last_fullsized":"2024-09-16T05:17:07.733185+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:19.705056+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.705056+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:37.392331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00041519400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"2.22","version":"182'20","reported_seq":992,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.936001+0000","last_change":"2024-09-16T05:17:10.936215+0000","last_active":"2024-09-16T05:17:10.936001+0000","last_peered":"2024-09-16T05:17:10.936001+0000","last_clean":"2024-09-16T05:17:10.936001+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:17:10.936001+0000","last_undegraded":"2024-09-16T05:17:10.936001+0000","last_fullsized":"2024-09-16T05:17:10.936001+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:16:43.744675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:43.744675+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.976692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000923097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.23","version":"184'17","reported_seq":1321,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-16T05:17:06.738402+0000","last_change":"2024-09-16T05:17:06.738783+0000","last_active":"2024-09-16T05:17:06.738402+0000","last_peered":"2024-09-16T05:17:06.738402+0000","last_clean":"2024-09-16T05:17:06.738402+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:17:06.738402+0000","last_undegraded":"2024-09-16T05:17:06.738402+0000","last_fullsized":"2024-09-16T05:17:06.738402+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:16:17.214142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.214142+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:32.437625+0000","scrub_duration":429,"objects_trimmed":0,"snaptrim_duration":0.00099567100000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"2.24","version":"182'89","reported_seq":1149,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736160+0000","last_change":"2024-09-16T05:17:08.752302+0000","last_active":"2024-09-16T05:17:09.736160+0000","last_peered":"2024-09-16T05:17:09.736160+0000","last_clean":"2024-09-16T05:17:09.736160+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:17:09.736160+0000","last_undegraded":"2024-09-16T05:17:09.736160+0000","last_fullsized":"2024-09-16T05:17:09.736160+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.030307+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.149889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00225997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.25","version":"463'688","reported_seq":2474,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-16T05:17:08.735508+0000","last_change":"2024-09-16T05:17:08.735767+0000","last_active":"2024-09-16T05:17:08.735508+0000","last_peered":"2024-09-16T05:17:08.735508+0000","last_clean":"2024-09-16T05:17:08.735508+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:17:08.735508+0000","last_undegraded":"2024-09-16T05:17:08.735508+0000","last_fullsized":"2024-09-16T05:17:08.735508+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"463'688","last_scrub_stamp":"2024-09-16T05:17:03.105650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.105650+0000","objects_scrubbed":2,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:20.350687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085683299999999999,"stat_sum":{"num_bytes":5210302,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":76000,"num_write":43,"num_write_kb":22034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.26","version":"116'9","reported_seq":1151,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185563+0000","last_change":"2024-09-16T05:17:11.185779+0000","last_active":"2024-09-16T05:17:11.185563+0000","last_peered":"2024-09-16T05:17:11.185563+0000","last_clean":"2024-09-16T05:17:11.185563+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:17:11.185563+0000","last_undegraded":"2024-09-16T05:17:11.185563+0000","last_fullsized":"2024-09-16T05:17:11.185563+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:16:13.148890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:13.148890+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:39.830710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015047509999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.27","version":"0'0","reported_seq":1163,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184845+0000","last_change":"2024-09-16T05:17:11.185130+0000","last_active":"2024-09-16T05:17:11.184845+0000","last_peered":"2024-09-16T05:17:11.184845+0000","last_clean":"2024-09-16T05:17:11.184845+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:17:11.184845+0000","last_undegraded":"2024-09-16T05:17:11.184845+0000","last_fullsized":"2024-09-16T05:17:11.184845+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:16:17.188503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.188503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:18.771471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000929556,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.28","version":"245'105","reported_seq":1921,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-16T05:17:06.738163+0000","last_change":"2024-09-16T05:17:06.738603+0000","last_active":"2024-09-16T05:17:06.738163+0000","last_peered":"2024-09-16T05:17:06.738163+0000","last_clean":"2024-09-16T05:17:06.738163+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:17:06.738163+0000","last_undegraded":"2024-09-16T05:17:06.738163+0000","last_fullsized":"2024-09-16T05:17:06.738163+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:15:57.809122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.809122+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:21.037986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00095978299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"2.29","version":"481'668","reported_seq":2567,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.740553+0000","last_change":"2024-09-16T05:17:11.741161+0000","last_active":"2024-09-16T05:17:11.740553+0000","last_peered":"2024-09-16T05:17:11.740553+0000","last_clean":"2024-09-16T05:17:11.740553+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:17:11.740553+0000","last_undegraded":"2024-09-16T05:17:11.740553+0000","last_fullsized":"2024-09-16T05:17:11.740553+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"472'660","last_scrub_stamp":"2024-09-16T05:17:03.427111+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.427111+0000","objects_scrubbed":2,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:04.591524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012141540000000001,"stat_sum":{"num_bytes":5933069,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":116,"num_read_kb":111541,"num_write":105,"num_write_kb":41636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":2153,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.183994+0000","last_change":"2024-09-16T05:17:11.184356+0000","last_active":"2024-09-16T05:17:11.183994+0000","last_peered":"2024-09-16T05:17:11.183994+0000","last_clean":"2024-09-16T05:17:11.183994+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:17:11.183994+0000","last_undegraded":"2024-09-16T05:17:11.183994+0000","last_fullsized":"2024-09-16T05:17:11.183994+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.269448+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:38.633691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061950299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.2b","version":"477'258","reported_seq":2015,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-16T05:17:07.733110+0000","last_change":"2024-09-16T05:17:06.733895+0000","last_active":"2024-09-16T05:17:07.733110+0000","last_peered":"2024-09-16T05:17:07.733110+0000","last_clean":"2024-09-16T05:17:07.733110+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:17:07.733110+0000","last_undegraded":"2024-09-16T05:17:07.733110+0000","last_fullsized":"2024-09-16T05:17:07.733110+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"406'216","last_scrub_stamp":"2024-09-16T05:16:17.634887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.634887+0000","objects_scrubbed":2,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:20.374738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021976280000000001,"stat_sum":{"num_bytes":3344692,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":41,"num_read_kb":42424,"num_write":30,"num_write_kb":14799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ab","length":"2"}]},{"pgid":"2.2c","version":"241'194","reported_seq":1977,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-16T05:17:08.735842+0000","last_change":"2024-09-16T05:17:08.736153+0000","last_active":"2024-09-16T05:17:08.735842+0000","last_peered":"2024-09-16T05:17:08.735842+0000","last_clean":"2024-09-16T05:17:08.735842+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:17:08.735842+0000","last_undegraded":"2024-09-16T05:17:08.735842+0000","last_fullsized":"2024-09-16T05:17:08.735842+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:17:02.077362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:02.077362+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:29.544187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011010830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.2d","version":"477'365","reported_seq":2211,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.186310+0000","last_change":"2024-09-16T05:17:11.186608+0000","last_active":"2024-09-16T05:17:11.186310+0000","last_peered":"2024-09-16T05:17:11.186310+0000","last_clean":"2024-09-16T05:17:11.186310+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:17:11.186310+0000","last_undegraded":"2024-09-16T05:17:11.186310+0000","last_fullsized":"2024-09-16T05:17:11.186310+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"377'326","last_scrub_stamp":"2024-09-16T05:16:15.212317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:15.212317+0000","objects_scrubbed":1,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:15.678470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020341069999999998,"stat_sum":{"num_bytes":5027150,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":63587,"num_write":20,"num_write_kb":10086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.2e","version":"471'425","reported_seq":2250,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185195+0000","last_change":"2024-09-16T05:17:11.185511+0000","last_active":"2024-09-16T05:17:11.185195+0000","last_peered":"2024-09-16T05:17:11.185195+0000","last_clean":"2024-09-16T05:17:11.185195+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:17:11.185195+0000","last_undegraded":"2024-09-16T05:17:11.185195+0000","last_fullsized":"2024-09-16T05:17:11.185195+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"428'412","last_scrub_stamp":"2024-09-16T05:16:19.216112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.216112+0000","objects_scrubbed":3,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:20.291740+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013118730000000001,"stat_sum":{"num_bytes":2958098,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":32166,"num_write":37,"num_write_kb":13743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.2f","version":"479'392","reported_seq":2278,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736424+0000","last_change":"2024-09-16T05:17:08.753349+0000","last_active":"2024-09-16T05:17:09.736424+0000","last_peered":"2024-09-16T05:17:09.736424+0000","last_clean":"2024-09-16T05:17:09.736424+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:17:09.736424+0000","last_undegraded":"2024-09-16T05:17:09.736424+0000","last_fullsized":"2024-09-16T05:17:09.736424+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"422'318","last_scrub_stamp":"2024-09-16T05:16:14.124319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.124319+0000","objects_scrubbed":4,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:29.416011+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026651800000000001,"stat_sum":{"num_bytes":6823840,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":25591,"num_write":57,"num_write_kb":19443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":142298846,"num_objects":58,"num_object_clones":19,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":4,"num_read":1290,"num_read_kb":1301382,"num_write":935,"num_write_kb":398621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190382080,"data_stored":216090578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19343,"ondisk_log_size":19343,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089441,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70496,"kb_used_data":30988,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93674656,"statfs":{"total":95995035648,"available":95922847744,"internally_reserved":0,"allocated":31731712,"data_stored":32707176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285670,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63564,"kb_used_data":24056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93681588,"statfs":{"total":95995035648,"available":95929946112,"internally_reserved":0,"allocated":24633344,"data_stored":25596681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514606,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56152,"kb_used_data":29116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93689000,"statfs":{"total":95995035648,"available":95937536000,"internally_reserved":0,"allocated":29814784,"data_stored":30828217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58400,"kb_used_data":18888,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93686752,"statfs":{"total":95995035648,"available":95935234048,"internally_reserved":0,"allocated":19341312,"data_stored":18573372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103285,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56864,"kb_used_data":29836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93688288,"statfs":{"total":95995035648,"available":95936806912,"internally_reserved":0,"allocated":30552064,"data_stored":34480631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397628,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50884,"kb_used_data":23852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93694268,"statfs":{"total":95995035648,"available":95942930432,"internally_reserved":0,"allocated":24424448,"data_stored":25794531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757374,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54308,"kb_used_data":18948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690844,"statfs":{"total":95995035648,"available":95939424256,"internally_reserved":0,"allocated":19402752,"data_stored":19867226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50520,"kb_used_data":15164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93694632,"statfs":{"total":95995035648,"available":95943303168,"internally_reserved":0,"allocated":15527936,"data_stored":14594692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117128,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64672,"kb_used_data":25156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93680480,"statfs":{"total":95995035648,"available":95928811520,"internally_reserved":0,"allocated":25759744,"data_stored":24772676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240415,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45444,"kb_used_data":13160,"kb_used_omap":10,"kb_used_meta":32245,"kb_avail":93699708,"statfs":{"total":95995035648,"available":95948500992,"internally_reserved":0,"allocated":13475840,"data_stored":13154144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":33019187},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51592,"kb_used_data":12080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93693560,"statfs":{"total":95995035648,"available":95942205440,"internally_reserved":0,"allocated":12369920,"data_stored":10313091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738520,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49192,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93695960,"statfs":{"total":95995035648,"available":95944663040,"internally_reserved":0,"allocated":9912320,"data_stored":7877492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3825664,"data_stored":3800639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6623232,"data_stored":6572365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7475200,"data_stored":9159191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19709952,"data_stored":20721128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9768960,"data_stored":10845519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15138816,"data_stored":17612832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":18640896,"data_stored":22028372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24780800,"data_stored":30722941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15527936,"data_stored":16751553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24055808,"data_stored":27079044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":18874368,"data_stored":21847508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":25960448,"data_stored":28949486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:17:14.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:17:14.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:14.149 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:14.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:14.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:13 smithi204 ceph-mon[30933]: pgmap v885: 107 pgs: 107 active+clean; 131 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-16T05:17:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:13 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:13 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:13 smithi204 ceph-mon[30933]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:13 smithi204 ceph-mon[30933]: osdmap e484: 12 total, 12 up, 12 in 2024-09-16T05:17:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:13 smithi196 ceph-mon[30802]: pgmap v885: 107 pgs: 107 active+clean; 131 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-16T05:17:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:13 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:13 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:13 smithi196 ceph-mon[30802]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:13 smithi196 ceph-mon[30802]: osdmap e484: 12 total, 12 up, 12 in 2024-09-16T05:17:14.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:14.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:14.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:14.291 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:14.291 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:14.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:14.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:14.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:14.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:14.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:14.625 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:14.625 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:14.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:14.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:14.653 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:14.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:14.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:14.674 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:14.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:14.694 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:14.694 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:14.717 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:14.739 INFO:tasks.rados.rados.2.smithi204.stdout:1637: done (0 left) 2024-09-16T05:17:14.739 INFO:tasks.rados.rados.2.smithi204.stdout:1638: read oid 19 snap -1 2024-09-16T05:17:14.739 INFO:tasks.rados.rados.2.smithi204.stdout:1638: expect (ObjNum 552 snap 165 seq_num 552) 2024-09-16T05:17:14.739 INFO:tasks.rados.rados.2.smithi204.stdout:1639: snap_create 2024-09-16T05:17:14.739 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:14.740 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:14.746 INFO:tasks.rados.rados.2.smithi204.stdout:1638: expect (ObjNum 552 snap 165 seq_num 552) 2024-09-16T05:17:14.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:14.762 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:14.762 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:14.784 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:14.784 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:14.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:14.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:14.805 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:14.805 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:14.826 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:14.848 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:14.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:17:14.869 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:14.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:14.910 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:14.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:15.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:15.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:15.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:15.112 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:17:15.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:15.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:15.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3974054395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:14 smithi032 ceph-mon[27388]: osdmap e485: 12 total, 12 up, 12 in 2024-09-16T05:17:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3974054395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:14 smithi204 ceph-mon[30933]: osdmap e485: 12 total, 12 up, 12 in 2024-09-16T05:17:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3974054395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:14 smithi196 ceph-mon[30802]: osdmap e485: 12 total, 12 up, 12 in 2024-09-16T05:17:15.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:15.287 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:15.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:15.342 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:17:15.344 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:17:15.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:15.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:15.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:15.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:17:15.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:15.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:15.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:15.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:15.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:19 2.b e112/107 2:d7c71ae7:::smithi20457400-5:head v 112'3, mlcod=0'0)", 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 392.63337878300001, 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19579059400000001, 2024-09-16T05:17:15.677 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041247+0000", 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041248+0000", 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5799999999999999e-07 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041253+0000", 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4500000000000003e-06 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236743+0000", 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19548974399999999 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236749+0000", 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6780000000000003e-06 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236795+0000", 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5164999999999999e-05 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.236811+0000", 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6453999999999999e-05 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237007+0000", 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019572799999999999 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237037+0000", 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0716999999999997e-05 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:115 2.a 2:580e4d0e:::smithi20457400-29:head [setxattr _header (56) in=7b,truncate 2470596] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 392.34143383600002, 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032560466000000003, 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.682 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 115 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333192+0000", 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.77e-07 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333194+0000", 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9580000000000001e-06 2024-09-16T05:17:15.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333200+0000", 2024-09-16T05:17:15.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3130000000000004e-06 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333204+0000", 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2710000000000001e-06 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334135+0000", 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093147499999999997 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334189+0000", 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3374000000000001e-05 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334274+0000", 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4987999999999993e-05 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334496+0000", 2024-09-16T05:17:15.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022271300000000001 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365655+0000", 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031158857000000002 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365694+0000", 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9187999999999998e-05 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365752+0000", 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:116 2.a 2:580e4d0e:::smithi20457400-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 392.34141653, 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032766964000000003, 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.689 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 116 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333209+0000", 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6e-07 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333211+0000", 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.221e-06 2024-09-16T05:17:15.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333214+0000", 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.844e-06 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.333218+0000", 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3050000000000001e-06 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334432+0000", 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012139379999999999 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.334485+0000", 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2865000000000003e-05 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365804+0000", 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031319393000000001 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365881+0000", 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6759000000000001e-05 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.365976+0000", 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5378999999999996e-05 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.694 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:128 2.a 2:5ae26c04:::smithi20457400-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 392.30559882400001, 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021253916000000001, 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 128 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369027+0000", 2024-09-16T05:17:15.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3099999999999998e-07 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369029+0000", 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9209999999999999e-06 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369036+0000", 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3309999999999998e-06 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369040+0000", 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0069999999999999e-06 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369803+0000", 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076296199999999995 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.369852+0000", 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8396000000000002e-05 2024-09-16T05:17:15.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390172+0000", 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020319918999999999 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390220+0000", 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7936000000000003e-05 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.390281+0000", 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1013e-05 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:277 2.a 2:5827f14d:::smithi20457400-12:head [copy-from ver 0] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 386.648733225, 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38105511399999997, 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.700 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 277 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025894+0000", 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.683e-06 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025900+0000", 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7740000000000001e-06 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025917+0000", 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6742999999999999e-05 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025927+0000", 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.978e-06 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025988+0000", 2024-09-16T05:17:15.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1574000000000004e-05 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026078+0000", 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0326000000000002e-05 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.406948+0000", 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38086903599999999 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:278 2.a 2:5827f14d:::smithi20457400-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 386.64867781700002, 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38122975799999997, 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 278 2024-09-16T05:17:15.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025948+0000", 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5899999999999996e-07 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025951+0000", 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7819999999999998e-06 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.708 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.708 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025959+0000", 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1019999999999997e-06 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025966+0000", 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.409e-06 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026245+0000", 2024-09-16T05:17:15.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027952800000000002 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026298+0000", 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2747999999999997e-05 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407054+0000", 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.380756119 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407114+0000", 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0350999999999999e-05 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.407178+0000", 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3159999999999998e-05 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.711 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:275 2.b e117/107 2:d7c71ae7:::smithi20457400-5:head v 117'20, mlcod=116'15)", 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 386.64798214699999, 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072246237000000005, 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026643+0000", 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026644+0000", 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6000000000000003e-07 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026651+0000", 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0210000000000002e-06 2024-09-16T05:17:15.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098551+0000", 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.071899669999999999 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098559+0000", 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.028e-06 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098612+0000", 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3485e-05 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098628+0000", 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5206e-05 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098860+0000", 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023204700000000001 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098890+0000", 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.012e-05 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:286 2.b e117/107 2:da4f6049:::smithi20457400-49:head v 117'22, mlcod=117'17)", 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 386.57610445500001, 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.24158528100000001, 2024-09-16T05:17:15.716 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098521+0000", 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098523+0000", 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.508e-06 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098530+0000", 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.075e-06 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339416+0000", 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.240886668 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339422+0000", 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0680000000000002e-06 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339502+0000", 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9698999999999997e-05 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.339527+0000", 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4780999999999999e-05 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340073+0000", 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054627699999999996 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340106+0000", 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3204999999999998e-05 2024-09-16T05:17:15.720 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:884 2.b e170/107 2:d8680303:::smithi20457400-43:head v 170'88, mlcod=165'82)", 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.662116+0000", 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 334.012509959, 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67911829000000001, 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662116+0000", 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662116+0000", 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662117+0000", 2024-09-16T05:17:15.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.111e-06 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662125+0000", 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9410000000000007e-06 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.340626+0000", 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67850096299999996 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.340633+0000", 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.08e-06 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.340718+0000", 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5235000000000006e-05 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.340766+0000", 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8031999999999997e-05 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341203+0000", 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000436759 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341234+0000", 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1168999999999999e-05 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:972 2.c 2:33100188:::smithi20457400-19:head [setxattr _header (57) in=7b,truncate 2672313] snapc 28=[28,27,25,24,23,22,21,20,1f,15,f] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:50.997840+0000", 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 324.676785505, 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53798918100000004, 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 972 2024-09-16T05:17:15.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997840+0000", 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997840+0000", 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997841+0000", 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0399999999999996e-07 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997844+0000", 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4809999999999998e-06 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997853+0000", 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2010000000000005e-06 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997859+0000", 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9920000000000004e-06 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.998655+0000", 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00079606800000000004 2024-09-16T05:17:15.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.998775+0000", 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119679 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.998945+0000", 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016966099999999999 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.000857+0000", 2024-09-16T05:17:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019118410000000001 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535713+0000", 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53485633799999999 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535763+0000", 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9926999999999998e-05 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535829+0000", 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6389000000000004e-05 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:973 2.c 2:33100188:::smithi20457400-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:50.997896+0000", 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 324.67673000399998, 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53811316099999995, 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.732 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 973 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997896+0000", 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997896+0000", 2024-09-16T05:17:15.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997896+0000", 2024-09-16T05:17:15.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8999999999999996e-07 2024-09-16T05:17:15.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997899+0000", 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.627e-06 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997906+0000", 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.622e-06 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.997911+0000", 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.989e-06 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.999224+0000", 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001312183 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:50.999334+0000", 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109947 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535872+0000", 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53653808800000002 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.535938+0000", 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6586000000000002e-05 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:51.536009+0000", 2024-09-16T05:17:15.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0529000000000002e-05 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1133 2.a 2:5ae26c04:::smithi20457400-32:head [write 591646~470441,stat] snapc 2f=[2f,2e,2d,2c,2b,2a,29,27,24,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:02.732148+0000", 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 312.94247743, 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023703246000000001, 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.738 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1133 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732148+0000", 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732148+0000", 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732151+0000", 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.554e-06 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732426+0000", 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027569299999999998 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732451+0000", 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4324e-05 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732462+0000", 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1099e-05 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732528+0000", 2024-09-16T05:17:15.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6525000000000004e-05 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.732536+0000", 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4839999999999998e-06 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.733720+0000", 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011842000000000001 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.733818+0000", 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8364999999999998e-05 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.733981+0000", 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016235800000000001 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.735390+0000", 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014088200000000001 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.755705+0000", 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020315054999999999 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.755785+0000", 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0304999999999998e-05 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:02.755851+0000", 2024-09-16T05:17:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6464000000000006e-05 2024-09-16T05:17:15.745 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.745 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.745 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.745 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:1575 2.5 e227/200 2:a5bc1608:::smithi20457400-11:head v 227'370, mlcod=227'365)", 2024-09-16T05:17:15.745 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.692673+0000", 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 276.981952707, 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64070468899999999, 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692673+0000", 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692673+0000", 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692674+0000", 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8999999999999996e-07 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692680+0000", 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1070000000000001e-06 2024-09-16T05:17:15.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.332765+0000", 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64008575199999995 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.332775+0000", 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2609999999999999e-06 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.332845+0000", 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0035000000000003e-05 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.332874+0000", 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9138000000000001e-05 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333328+0000", 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045414700000000001 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333378+0000", 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9558999999999998e-05 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:547 2.25 e278/252 2:a5bc1608:::smithi20457400-11:52 v 278'476, mlcod=277'470)", 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 225.94125798900001, 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27999907200000002, 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.750 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4900000000000005e-07 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733374+0000", 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0669999999999997e-06 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012754+0000", 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27937933999999998 2024-09-16T05:17:15.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012763+0000", 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.884e-06 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012833+0000", 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0779999999999997e-05 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012856+0000", 2024-09-16T05:17:15.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.283e-05 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.013333+0000", 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00047694000000000002 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.013367+0000", 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3482000000000002e-05 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.754 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:379 2.35 e346/275 2:ad4a88eb:::smithi20457400-16:6b v 346'436, mlcod=340'432)", 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 139.97753752200001, 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72722915899999996, 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.755 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1100000000000002e-07 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697095+0000", 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8709999999999999e-06 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423764+0000", 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72666896299999995 2024-09-16T05:17:15.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423770+0000", 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0129999999999999e-06 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423829+0000", 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8807999999999997e-05 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423855+0000", 2024-09-16T05:17:15.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5721e-05 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.424282+0000", 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042752699999999997 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.424317+0000", 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4944999999999998e-05 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.759 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:414 2.35 e378/275 2:ad4a88eb:::smithi20457400-16:6b v 378'499, mlcod=377'495)", 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 107.93275720299999, 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.883103101, 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741869+0000", 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8099999999999999e-07 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741872+0000", 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7979999999999999e-06 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624224+0000", 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.88235191599999996 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624240+0000", 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5296000000000001e-05 2024-09-16T05:17:15.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624347+0000", 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107249 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624381+0000", 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4480999999999997e-05 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624917+0000", 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053571399999999996 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624972+0000", 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4466000000000003e-05 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3036 2.1b 2:da4f6049:::smithi20457400-49:head [setxattr _header (58) in=7b,truncate 3294238] snapc 7e=[7e,7d,7c,7b] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:17:15.764 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 106.907506541, 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.437857206, 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3036 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767120+0000", 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0800000000000004e-07 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767123+0000", 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.642e-06 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767132+0000", 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2819999999999996e-06 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767138+0000", 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5200000000000003e-06 2024-09-16T05:17:15.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767869+0000", 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00073052499999999999 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767959+0000", 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0730999999999996e-05 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9", 2024-09-16T05:17:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768106+0000", 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014667900000000001 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768798+0000", 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069156399999999998 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204863+0000", 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43606497999999999 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204909+0000", 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6758000000000002e-05 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204976+0000", 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6816999999999994e-05 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.770 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3037 2.1b 2:da4f6049:::smithi20457400-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e379)", 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 106.907474794, 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43800830899999998, 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3037 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:17:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6199999999999998e-07 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767154+0000", 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6730000000000001e-06 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767161+0000", 2024-09-16T05:17:15.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.849e-06 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767166+0000", 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4709999999999998e-06 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768487+0000", 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001320543 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768584+0000", 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7547999999999995e-05 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205011+0000", 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43642700000000001 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205081+0000", 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9381000000000005e-05 2024-09-16T05:17:15.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205159+0000", 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8381999999999996e-05 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:3951 2.2f e467/275 2:f4b938a4:::smithi20457400-47:head v 467'381, mlcod=467'378)", 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.815256802, 2024-09-16T05:17:15.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39001122199999999, 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:17:15.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6400000000000002e-07 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859375+0000", 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1599999999999997e-06 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248747+0000", 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38937218699999998 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248762+0000", 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.537e-05 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248857+0000", 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4630999999999996e-05 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248883+0000", 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6448000000000002e-05 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249341+0000", 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000457486 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249380+0000", 2024-09-16T05:17:15.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9276000000000001e-05 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3977 2.a 2:5089ebe1:::smithi20457400-36:head [copy-from ver 338] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899699+0000", 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.774926691000001, 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021364978, 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:17:15.781 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3977 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899699+0000", 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899699+0000", 2024-09-16T05:17:15.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899701+0000", 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0320000000000002e-06 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899707+0000", 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0379999999999996e-06 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899730+0000", 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2694999999999999e-05 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899738+0000", 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0600000000000008e-06 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899808+0000", 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9751000000000004e-05 2024-09-16T05:17:15.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899884+0000", 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6210000000000004e-05 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.914874+0000", 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990066999999999 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.918503+0000", 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036289479999999999 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.920998+0000", 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024951679999999999 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.921031+0000", 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3461999999999999e-05 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:17:15.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.921064+0000", 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2546999999999999e-05 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:17:15.787 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:17:15.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1638: done (1 left) 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1639: done (0 left) 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1640: read oid 3 snap 173 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1640: expect (ObjNum 516 snap 157 seq_num 516) 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1641: write oid 18 current snap is 178 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1641: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1641: writing smithi20457400-18 from 665709 to 1396053 tid 1 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1641: writing smithi20457400-18 from 2133283 to 2559486 tid 2 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1641: writing smithi20457400-18 from 3203532 to 3713380 tid 3 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1642: write oid 20 current snap is 178 2024-09-16T05:17:15.790 INFO:tasks.rados.rados.2.smithi204.stdout:1642: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-09-16T05:17:15.791 INFO:tasks.rados.rados.2.smithi204.stdout:1642: writing smithi20457400-20 from 682516 to 1088109 tid 1 2024-09-16T05:17:15.791 INFO:tasks.rados.rados.2.smithi204.stdout:1642: writing smithi20457400-20 from 1791765 to 2361626 tid 2 2024-09-16T05:17:15.791 INFO:tasks.rados.rados.2.smithi204.stdout:1642: writing smithi20457400-20 from 2830955 to 3552075 tid 3 2024-09-16T05:17:15.791 INFO:tasks.rados.rados.2.smithi204.stdout:1643: snap_create 2024-09-16T05:17:15.791 INFO:tasks.rados.rados.2.smithi204.stdout:1640: expect (ObjNum 516 snap 157 seq_num 516) 2024-09-16T05:17:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:15.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:15.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:15.864 INFO:tasks.rados.rados.2.smithi204.stdout:1641: finishing write tid 1 to smithi20457400-18 2024-09-16T05:17:15.864 INFO:tasks.rados.rados.2.smithi204.stdout:1641: finishing write tid 2 to smithi20457400-18 2024-09-16T05:17:15.864 INFO:tasks.rados.rados.2.smithi204.stdout:1641: finishing write tid 3 to smithi20457400-18 2024-09-16T05:17:15.864 INFO:tasks.rados.rados.2.smithi204.stdout:1641: finishing write tid 4 to smithi20457400-18 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:1641: finishing write tid 5 to smithi20457400-18 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 320 (ObjNum 587 snap 178 seq_num 587) dirty exists 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:1641: left oid 18 (ObjNum 587 snap 178 seq_num 587) 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:1642: finishing write tid 1 to smithi20457400-20 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:1642: finishing write tid 2 to smithi20457400-20 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:1642: finishing write tid 3 to smithi20457400-20 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:1642: finishing write tid 4 to smithi20457400-20 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:1642: finishing write tid 5 to smithi20457400-20 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 693 (ObjNum 588 snap 178 seq_num 588) dirty exists 2024-09-16T05:17:15.865 INFO:tasks.rados.rados.2.smithi204.stdout:1642: left oid 20 (ObjNum 588 snap 178 seq_num 588) 2024-09-16T05:17:15.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:15.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:15.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:16.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:16.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:16.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:16.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:16.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:16.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:16.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:15 smithi196 ceph-mon[30802]: pgmap v888: 107 pgs: 107 active+clean; 136 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 8 op/s 2024-09-16T05:17:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:15 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:15 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:15 smithi196 ceph-mon[30802]: osdmap e486: 12 total, 12 up, 12 in 2024-09-16T05:17:16.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:16.315 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:16.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:16.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:15 smithi032 ceph-mon[27388]: pgmap v888: 107 pgs: 107 active+clean; 136 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 8 op/s 2024-09-16T05:17:16.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:15 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:16.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:15 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:16.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:15 smithi032 ceph-mon[27388]: osdmap e486: 12 total, 12 up, 12 in 2024-09-16T05:17:16.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:16.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:15 smithi204 ceph-mon[30933]: pgmap v888: 107 pgs: 107 active+clean; 136 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 MiB/s wr, 8 op/s 2024-09-16T05:17:16.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:15 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:15 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:15 smithi204 ceph-mon[30933]: osdmap e486: 12 total, 12 up, 12 in 2024-09-16T05:17:16.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:16.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:16.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:17:16.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:16.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:16.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:16.702 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2024-09-16T05:17:14.735895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":142889214,"num_objects":60,"num_object_clones":19,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":4,"num_read":1290,"num_read_kb":1301382,"num_write":935,"num_write_kb":398621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22266,"ondisk_log_size":22266,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":674120,"kb_used_data":252968,"kb_used_omap":27,"kb_used_meta":420836,"kb_avail":1124267704,"statfs":{"total":1151940427776,"available":1151250128896,"internally_reserved":0,"allocated":259039232,"data_stored":260588973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":430936231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6854413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":11,"num_write_kb":5723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004935"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":2341,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738552+0000","last_change":"2024-09-16T05:17:10.739031+0000","last_active":"2024-09-16T05:17:10.738552+0000","last_peered":"2024-09-16T05:17:10.738552+0000","last_clean":"2024-09-16T05:17:10.738552+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:17:10.738552+0000","last_undegraded":"2024-09-16T05:17:10.738552+0000","last_fullsized":"2024-09-16T05:17:10.738552+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:16:38.263205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.263205+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:40.104709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089345699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.54","version":"441'295","reported_seq":1753,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759574+0000","last_change":"2024-09-16T05:17:08.759402+0000","last_active":"2024-09-16T05:17:09.759574+0000","last_peered":"2024-09-16T05:17:09.759574+0000","last_clean":"2024-09-16T05:17:09.759574+0000","last_became_active":"2024-09-16T05:16:30.728460+0000","last_became_peered":"2024-09-16T05:16:30.728460+0000","last_unstale":"2024-09-16T05:17:09.759574+0000","last_undegraded":"2024-09-16T05:17:09.759574+0000","last_fullsized":"2024-09-16T05:17:09.759574+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"441'295","last_scrub_stamp":"2024-09-16T05:16:37.936216+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.936216+0000","objects_scrubbed":1,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:45.530193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013071129999999999,"stat_sum":{"num_bytes":2285851,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":4975,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.53","version":"181'15","reported_seq":233,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.733734+0000","last_change":"2024-09-16T05:17:10.733949+0000","last_active":"2024-09-16T05:17:10.733734+0000","last_peered":"2024-09-16T05:17:10.733734+0000","last_clean":"2024-09-16T05:17:10.733734+0000","last_became_active":"2024-09-16T05:16:27.728289+0000","last_became_peered":"2024-09-16T05:16:27.728289+0000","last_unstale":"2024-09-16T05:17:10.733734+0000","last_undegraded":"2024-09-16T05:17:10.733734+0000","last_fullsized":"2024-09-16T05:17:10.733734+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:38.004600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.004600+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:53.010518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000408138,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.52","version":"272'69","reported_seq":224,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735390+0000","last_change":"2024-09-16T05:17:08.735450+0000","last_active":"2024-09-16T05:17:09.735390+0000","last_peered":"2024-09-16T05:17:09.735390+0000","last_clean":"2024-09-16T05:17:09.735390+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:17:09.735390+0000","last_undegraded":"2024-09-16T05:17:09.735390+0000","last_fullsized":"2024-09-16T05:17:09.735390+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:16:38.584934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.584934+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:05.950127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094274500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.51","version":"272'115","reported_seq":749,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935827+0000","last_change":"2024-09-16T05:17:10.936027+0000","last_active":"2024-09-16T05:17:10.935827+0000","last_peered":"2024-09-16T05:17:10.935827+0000","last_clean":"2024-09-16T05:17:10.935827+0000","last_became_active":"2024-09-16T05:16:30.731943+0000","last_became_peered":"2024-09-16T05:16:30.731943+0000","last_unstale":"2024-09-16T05:17:10.935827+0000","last_undegraded":"2024-09-16T05:17:10.935827+0000","last_fullsized":"2024-09-16T05:17:10.935827+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:16:38.834396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.834396+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:06.719775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00082524400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.50","version":"182'16","reported_seq":238,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.454950+0000","last_change":"2024-09-16T05:17:12.455475+0000","last_active":"2024-09-16T05:17:12.454950+0000","last_peered":"2024-09-16T05:17:12.454950+0000","last_clean":"2024-09-16T05:17:12.454950+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:17:12.454950+0000","last_undegraded":"2024-09-16T05:17:12.454950+0000","last_fullsized":"2024-09-16T05:17:12.454950+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:38.794214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.794214+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:02.708151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073371000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2137,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759708+0000","last_change":"2024-09-16T05:17:08.758723+0000","last_active":"2024-09-16T05:17:09.759708+0000","last_peered":"2024-09-16T05:17:09.759708+0000","last_clean":"2024-09-16T05:17:09.759708+0000","last_became_active":"2024-09-16T05:16:30.732139+0000","last_became_peered":"2024-09-16T05:16:30.732139+0000","last_unstale":"2024-09-16T05:17:09.759708+0000","last_undegraded":"2024-09-16T05:17:09.759708+0000","last_fullsized":"2024-09-16T05:17:09.759708+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:16:38.938962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.938962+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.604232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00099435999999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.4e","version":"272'232","reported_seq":1999,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738860+0000","last_change":"2024-09-16T05:17:10.739103+0000","last_active":"2024-09-16T05:17:10.738860+0000","last_peered":"2024-09-16T05:17:10.738860+0000","last_clean":"2024-09-16T05:17:10.738860+0000","last_became_active":"2024-09-16T05:16:30.732564+0000","last_became_peered":"2024-09-16T05:16:30.732564+0000","last_unstale":"2024-09-16T05:17:10.738860+0000","last_undegraded":"2024-09-16T05:17:10.738860+0000","last_fullsized":"2024-09-16T05:17:10.738860+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:16:25.855040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.855040+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:51.240034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097078599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.4d","version":"467'480","reported_seq":2356,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.735201+0000","last_change":"2024-09-16T05:17:12.735406+0000","last_active":"2024-09-16T05:17:12.735201+0000","last_peered":"2024-09-16T05:17:12.735201+0000","last_clean":"2024-09-16T05:17:12.735201+0000","last_became_active":"2024-09-16T05:16:30.731141+0000","last_became_peered":"2024-09-16T05:16:30.731141+0000","last_unstale":"2024-09-16T05:17:12.735201+0000","last_undegraded":"2024-09-16T05:17:12.735201+0000","last_fullsized":"2024-09-16T05:17:12.735201+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"441'465","last_scrub_stamp":"2024-09-16T05:16:37.930310+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.930310+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:47.504386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000691517,"stat_sum":{"num_bytes":3361499,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":18522,"num_write":10,"num_write_kb":2573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"2.4c","version":"466'279","reported_seq":2109,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935319+0000","last_change":"2024-09-16T05:17:10.935522+0000","last_active":"2024-09-16T05:17:10.935319+0000","last_peered":"2024-09-16T05:17:10.935319+0000","last_clean":"2024-09-16T05:17:10.935319+0000","last_became_active":"2024-09-16T05:16:30.731529+0000","last_became_peered":"2024-09-16T05:16:30.731529+0000","last_unstale":"2024-09-16T05:17:10.935319+0000","last_undegraded":"2024-09-16T05:17:10.935319+0000","last_fullsized":"2024-09-16T05:17:10.935319+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:16:39.809684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.809684+0000","objects_scrubbed":1,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.220472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432700000000005,"stat_sum":{"num_bytes":3277464,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.4b","version":"244'177","reported_seq":2038,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760428+0000","last_change":"2024-09-16T05:17:08.758410+0000","last_active":"2024-09-16T05:17:09.760428+0000","last_peered":"2024-09-16T05:17:09.760428+0000","last_clean":"2024-09-16T05:17:09.760428+0000","last_became_active":"2024-09-16T05:14:30.853998+0000","last_became_peered":"2024-09-16T05:14:30.853998+0000","last_unstale":"2024-09-16T05:17:09.760428+0000","last_undegraded":"2024-09-16T05:17:09.760428+0000","last_fullsized":"2024-09-16T05:17:09.760428+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:17:04.875229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:04.875229+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:16.655688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079938899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.4a","version":"459'490","reported_seq":2314,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.934944+0000","last_change":"2024-09-16T05:17:10.935478+0000","last_active":"2024-09-16T05:17:10.934944+0000","last_peered":"2024-09-16T05:17:10.934944+0000","last_clean":"2024-09-16T05:17:10.934944+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:17:10.934944+0000","last_undegraded":"2024-09-16T05:17:10.934944+0000","last_fullsized":"2024-09-16T05:17:10.934944+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"459'490","last_scrub_stamp":"2024-09-16T05:17:00.898492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.898492+0000","objects_scrubbed":1,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:18.434035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00081010499999999998,"stat_sum":{"num_bytes":2840482,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":15,"num_write_kb":7769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.49","version":"245'220","reported_seq":2071,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739488+0000","last_change":"2024-09-16T05:17:10.739651+0000","last_active":"2024-09-16T05:17:10.739488+0000","last_peered":"2024-09-16T05:17:10.739488+0000","last_clean":"2024-09-16T05:17:10.739488+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:17:10.739488+0000","last_undegraded":"2024-09-16T05:17:10.739488+0000","last_fullsized":"2024-09-16T05:17:10.739488+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:16:00.136780+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.136780+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013416229999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.48","version":"451'226","reported_seq":2042,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735889+0000","last_change":"2024-09-16T05:17:08.752794+0000","last_active":"2024-09-16T05:17:09.735889+0000","last_peered":"2024-09-16T05:17:09.735889+0000","last_clean":"2024-09-16T05:17:09.735889+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:17:09.735889+0000","last_undegraded":"2024-09-16T05:17:09.735889+0000","last_fullsized":"2024-09-16T05:17:09.735889+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"451'226","last_scrub_stamp":"2024-09-16T05:17:08.272173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:08.272173+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:33.923787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022016100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20677,"num_write":10,"num_write_kb":8206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.47","version":"0'0","reported_seq":850,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735686+0000","last_change":"2024-09-16T05:17:08.735845+0000","last_active":"2024-09-16T05:17:09.735686+0000","last_peered":"2024-09-16T05:17:09.735686+0000","last_clean":"2024-09-16T05:17:09.735686+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:17:09.735686+0000","last_undegraded":"2024-09-16T05:17:09.735686+0000","last_fullsized":"2024-09-16T05:17:09.735686+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:17:00.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.838891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:13.159576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011647560000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.46","version":"116'9","reported_seq":1862,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759714+0000","last_change":"2024-09-16T05:17:08.759068+0000","last_active":"2024-09-16T05:17:09.759714+0000","last_peered":"2024-09-16T05:17:09.759714+0000","last_clean":"2024-09-16T05:17:09.759714+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:17:09.759714+0000","last_undegraded":"2024-09-16T05:17:09.759714+0000","last_fullsized":"2024-09-16T05:17:09.759714+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:17:01.867464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.867464+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:26.762419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012667170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.45","version":"305'464","reported_seq":2281,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760067+0000","last_change":"2024-09-16T05:17:08.758873+0000","last_active":"2024-09-16T05:17:09.760067+0000","last_peered":"2024-09-16T05:17:09.760067+0000","last_clean":"2024-09-16T05:17:09.760067+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:17:09.760067+0000","last_undegraded":"2024-09-16T05:17:09.760067+0000","last_fullsized":"2024-09-16T05:17:09.760067+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:15:59.876915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.876915+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:20.337198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011145090000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.44","version":"182'89","reported_seq":1944,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759737+0000","last_change":"2024-09-16T05:17:08.758361+0000","last_active":"2024-09-16T05:17:09.759737+0000","last_peered":"2024-09-16T05:17:09.759737+0000","last_clean":"2024-09-16T05:17:09.759737+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:17:09.759737+0000","last_undegraded":"2024-09-16T05:17:09.759737+0000","last_fullsized":"2024-09-16T05:17:09.759737+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:16:44.877606+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:16:44.877606+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:10.789776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066299299999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.43","version":"281'58","reported_seq":844,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184522+0000","last_change":"2024-09-16T05:17:11.184765+0000","last_active":"2024-09-16T05:17:11.184522+0000","last_peered":"2024-09-16T05:17:11.184522+0000","last_clean":"2024-09-16T05:17:11.184522+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:17:11.184522+0000","last_undegraded":"2024-09-16T05:17:11.184522+0000","last_fullsized":"2024-09-16T05:17:11.184522+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:16:14.198462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.198462+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:15.625295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072659999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.42","version":"182'20","reported_seq":836,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:13.105803+0000","last_change":"2024-09-16T05:17:13.106032+0000","last_active":"2024-09-16T05:17:13.105803+0000","last_peered":"2024-09-16T05:17:13.105803+0000","last_clean":"2024-09-16T05:17:13.105803+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:17:13.105803+0000","last_undegraded":"2024-09-16T05:17:13.105803+0000","last_fullsized":"2024-09-16T05:17:13.105803+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:16:45.114951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:45.114951+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:04.785242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093683099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"2.41","version":"178'50","reported_seq":932,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.936259+0000","last_change":"2024-09-16T05:17:10.936376+0000","last_active":"2024-09-16T05:17:10.936259+0000","last_peered":"2024-09-16T05:17:10.936259+0000","last_clean":"2024-09-16T05:17:10.936259+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:17:10.936259+0000","last_undegraded":"2024-09-16T05:17:10.936259+0000","last_fullsized":"2024-09-16T05:17:10.936259+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:59.922855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:59.922855+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:06.207085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071282699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.40","version":"205'33","reported_seq":293,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739430+0000","last_change":"2024-09-16T05:17:10.739632+0000","last_active":"2024-09-16T05:17:10.739430+0000","last_peered":"2024-09-16T05:17:10.739430+0000","last_clean":"2024-09-16T05:17:10.739430+0000","last_became_active":"2024-09-16T05:16:17.733306+0000","last_became_peered":"2024-09-16T05:16:17.733306+0000","last_unstale":"2024-09-16T05:17:10.739430+0000","last_undegraded":"2024-09-16T05:17:10.739430+0000","last_fullsized":"2024-09-16T05:17:10.739430+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:16:18.066113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.066113+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:39.698653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001321537,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3f","version":"472'312","reported_seq":2171,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185478+0000","last_change":"2024-09-16T05:17:11.185725+0000","last_active":"2024-09-16T05:17:11.185478+0000","last_peered":"2024-09-16T05:17:11.185478+0000","last_clean":"2024-09-16T05:17:11.185478+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:17:11.185478+0000","last_undegraded":"2024-09-16T05:17:11.185478+0000","last_fullsized":"2024-09-16T05:17:11.185478+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"436'302","last_scrub_stamp":"2024-09-16T05:16:43.248069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:43.248069+0000","objects_scrubbed":1,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:00.757639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001476109,"stat_sum":{"num_bytes":2551686,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":22718,"num_write":26,"num_write_kb":6027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3e","version":"474'388","reported_seq":2212,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.931801+0000","last_change":"2024-09-16T05:17:11.932214+0000","last_active":"2024-09-16T05:17:11.931801+0000","last_peered":"2024-09-16T05:17:11.931801+0000","last_clean":"2024-09-16T05:17:11.931801+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:17:11.931801+0000","last_undegraded":"2024-09-16T05:17:11.931801+0000","last_fullsized":"2024-09-16T05:17:11.931801+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"431'368","last_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.868538+0000","objects_scrubbed":3,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:38.553535+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0010406199999999999,"stat_sum":{"num_bytes":6101139,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":18,"num_write_kb":14694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2383,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760232+0000","last_change":"2024-09-16T05:17:08.760712+0000","last_active":"2024-09-16T05:17:09.760232+0000","last_peered":"2024-09-16T05:17:09.760232+0000","last_clean":"2024-09-16T05:17:09.760232+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:17:09.760232+0000","last_undegraded":"2024-09-16T05:17:09.760232+0000","last_fullsized":"2024-09-16T05:17:09.760232+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:16:23.005301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.005301+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:41.874562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025294369999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.3c","version":"481'507","reported_seq":2460,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.187013+0000","last_change":"2024-09-16T05:17:11.187175+0000","last_active":"2024-09-16T05:17:11.187013+0000","last_peered":"2024-09-16T05:17:11.187013+0000","last_clean":"2024-09-16T05:17:11.187013+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:17:11.187013+0000","last_undegraded":"2024-09-16T05:17:11.187013+0000","last_fullsized":"2024-09-16T05:17:11.187013+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"436'490","last_scrub_stamp":"2024-09-16T05:16:26.159696+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.159696+0000","objects_scrubbed":1,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:43.783124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002881116,"stat_sum":{"num_bytes":3109394,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":74,"num_read_kb":82099,"num_write":34,"num_write_kb":18036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3b","version":"481'326","reported_seq":2181,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739382+0000","last_change":"2024-09-16T05:17:10.739382+0000","last_active":"2024-09-16T05:17:10.739382+0000","last_peered":"2024-09-16T05:17:10.739382+0000","last_clean":"2024-09-16T05:17:10.739382+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:17:10.739382+0000","last_undegraded":"2024-09-16T05:17:10.739382+0000","last_fullsized":"2024-09-16T05:17:10.739382+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"431'315","last_scrub_stamp":"2024-09-16T05:16:21.147071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.147071+0000","objects_scrubbed":1,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:45.718220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010659770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":46643,"num_write":11,"num_write_kb":2763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3a","version":"454'520","reported_seq":2329,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735946+0000","last_change":"2024-09-16T05:17:08.751321+0000","last_active":"2024-09-16T05:17:09.735946+0000","last_peered":"2024-09-16T05:17:09.735946+0000","last_clean":"2024-09-16T05:17:09.735946+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:17:09.735946+0000","last_undegraded":"2024-09-16T05:17:09.735946+0000","last_fullsized":"2024-09-16T05:17:09.735946+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"454'520","last_scrub_stamp":"2024-09-16T05:17:01.101516+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.101516+0000","objects_scrubbed":1,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:20.575814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00166676,"stat_sum":{"num_bytes":2000132,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":24517,"num_write":37,"num_write_kb":15654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.39","version":"182'60","reported_seq":930,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735918+0000","last_change":"2024-09-16T05:17:08.751574+0000","last_active":"2024-09-16T05:17:09.735918+0000","last_peered":"2024-09-16T05:17:09.735918+0000","last_clean":"2024-09-16T05:17:09.735918+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:17:09.735918+0000","last_undegraded":"2024-09-16T05:17:09.735918+0000","last_fullsized":"2024-09-16T05:17:09.735918+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:17:02.077685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:02.077685+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015512049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.38","version":"181'57","reported_seq":949,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:13.105278+0000","last_change":"2024-09-16T05:17:13.105521+0000","last_active":"2024-09-16T05:17:13.105278+0000","last_peered":"2024-09-16T05:17:13.105278+0000","last_clean":"2024-09-16T05:17:13.105278+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:17:13.105278+0000","last_undegraded":"2024-09-16T05:17:13.105278+0000","last_fullsized":"2024-09-16T05:17:13.105278+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:16:17.338199+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.338199+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:35.968495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"2.37","version":"0'0","reported_seq":1020,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935580+0000","last_change":"2024-09-16T05:17:10.935763+0000","last_active":"2024-09-16T05:17:10.935580+0000","last_peered":"2024-09-16T05:17:10.935580+0000","last_clean":"2024-09-16T05:17:10.935580+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:17:10.935580+0000","last_undegraded":"2024-09-16T05:17:10.935580+0000","last_fullsized":"2024-09-16T05:17:10.935580+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:16:21.830665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.830665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.405915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074533399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.36","version":"272'60","reported_seq":1011,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759881+0000","last_change":"2024-09-16T05:17:08.759682+0000","last_active":"2024-09-16T05:17:09.759881+0000","last_peered":"2024-09-16T05:17:09.759881+0000","last_clean":"2024-09-16T05:17:09.759881+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:17:09.759881+0000","last_undegraded":"2024-09-16T05:17:09.759881+0000","last_fullsized":"2024-09-16T05:17:09.759881+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.904155+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.244733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001659743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.35","version":"479'543","reported_seq":2414,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760455+0000","last_change":"2024-09-16T05:17:08.760006+0000","last_active":"2024-09-16T05:17:09.760455+0000","last_peered":"2024-09-16T05:17:09.760455+0000","last_clean":"2024-09-16T05:17:09.760455+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:17:09.760455+0000","last_undegraded":"2024-09-16T05:17:09.760455+0000","last_fullsized":"2024-09-16T05:17:09.760455+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"410'508","last_scrub_stamp":"2024-09-16T05:16:00.864640+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.864640+0000","objects_scrubbed":4,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019585169999999999,"stat_sum":{"num_bytes":3411920,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":46,"num_read_kb":43632,"num_write":37,"num_write_kb":15224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.34","version":"467'359","reported_seq":1821,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739222+0000","last_change":"2024-09-16T05:17:10.739406+0000","last_active":"2024-09-16T05:17:10.739222+0000","last_peered":"2024-09-16T05:17:10.739222+0000","last_clean":"2024-09-16T05:17:10.739222+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:17:10.739222+0000","last_undegraded":"2024-09-16T05:17:10.739222+0000","last_fullsized":"2024-09-16T05:17:10.739222+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"420'335","last_scrub_stamp":"2024-09-16T05:16:15.093220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:15.093220+0000","objects_scrubbed":4,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.803909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011185500000000001,"stat_sum":{"num_bytes":5815387,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":76675,"num_write":31,"num_write_kb":15494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.33","version":"181'15","reported_seq":1022,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185110+0000","last_change":"2024-09-16T05:17:11.185417+0000","last_active":"2024-09-16T05:17:11.185110+0000","last_peered":"2024-09-16T05:17:11.185110+0000","last_clean":"2024-09-16T05:17:11.185110+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:17:11.185110+0000","last_undegraded":"2024-09-16T05:17:11.185110+0000","last_fullsized":"2024-09-16T05:17:11.185110+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:25.199845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.199845+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:31.585581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012291940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.32","version":"451'141","reported_seq":1131,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.930959+0000","last_change":"2024-09-16T05:17:11.931472+0000","last_active":"2024-09-16T05:17:11.930959+0000","last_peered":"2024-09-16T05:17:11.930959+0000","last_clean":"2024-09-16T05:17:11.930959+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:17:11.930959+0000","last_undegraded":"2024-09-16T05:17:11.930959+0000","last_fullsized":"2024-09-16T05:17:11.930959+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"419'127","last_scrub_stamp":"2024-09-16T05:16:18.859832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.859832+0000","objects_scrubbed":1,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:33.735551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000696399,"stat_sum":{"num_bytes":3109361,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":35253,"num_write":19,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.31","version":"459'204","reported_seq":1694,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735492+0000","last_change":"2024-09-16T05:17:08.735349+0000","last_active":"2024-09-16T05:17:09.735492+0000","last_peered":"2024-09-16T05:17:09.735492+0000","last_clean":"2024-09-16T05:17:09.735492+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:17:09.735492+0000","last_undegraded":"2024-09-16T05:17:09.735492+0000","last_fullsized":"2024-09-16T05:17:09.735492+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"459'204","last_scrub_stamp":"2024-09-16T05:17:03.863715+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.863715+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:20.160117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080613400000000002,"stat_sum":{"num_bytes":3176622,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":62,"num_read_kb":68251,"num_write":27,"num_write_kb":10906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.30","version":"182'16","reported_seq":984,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.934997+0000","last_change":"2024-09-16T05:17:10.935230+0000","last_active":"2024-09-16T05:17:10.934997+0000","last_peered":"2024-09-16T05:17:10.934997+0000","last_clean":"2024-09-16T05:17:10.934997+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:17:10.934997+0000","last_undegraded":"2024-09-16T05:17:10.934997+0000","last_fullsized":"2024-09-16T05:17:10.934997+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:24.905903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:24.905903+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.392571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053850299999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.12","version":"35'139","reported_seq":566,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738471+0000","last_change":"2024-09-16T05:15:57.054819+0000","last_active":"2024-09-16T05:17:10.738471+0000","last_peered":"2024-09-16T05:17:10.738471+0000","last_clean":"2024-09-16T05:17:10.738471+0000","last_became_active":"2024-09-16T05:15:56.716090+0000","last_became_peered":"2024-09-16T05:15:56.716090+0000","last_unstale":"2024-09-16T05:17:10.738471+0000","last_undegraded":"2024-09-16T05:17:10.738471+0000","last_fullsized":"2024-09-16T05:17:10.738471+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.054740+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.054740+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.994935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1576,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.740027+0000","last_change":"2024-09-16T05:17:11.740776+0000","last_active":"2024-09-16T05:17:11.740027+0000","last_peered":"2024-09-16T05:17:11.740027+0000","last_clean":"2024-09-16T05:17:11.740027+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:17:11.740027+0000","last_undegraded":"2024-09-16T05:17:11.740027+0000","last_fullsized":"2024-09-16T05:17:11.740027+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:16:26.623221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.623221+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:55.934505+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0.00080160599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1197,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.760285+0000","last_change":"2024-09-16T05:15:56.852148+0000","last_active":"2024-09-16T05:17:09.760285+0000","last_peered":"2024-09-16T05:17:09.760285+0000","last_clean":"2024-09-16T05:17:09.760285+0000","last_became_active":"2024-09-16T05:15:56.714681+0000","last_became_peered":"2024-09-16T05:15:56.714681+0000","last_unstale":"2024-09-16T05:17:09.760285+0000","last_undegraded":"2024-09-16T05:17:09.760285+0000","last_fullsized":"2024-09-16T05:17:09.760285+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.852021+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.852021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:11.168034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":1499,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:13.105341+0000","last_change":"2024-09-16T05:17:13.105564+0000","last_active":"2024-09-16T05:17:13.105341+0000","last_peered":"2024-09-16T05:17:13.105341+0000","last_clean":"2024-09-16T05:17:13.105341+0000","last_became_active":"2024-09-16T05:16:25.731838+0000","last_became_peered":"2024-09-16T05:16:25.731838+0000","last_unstale":"2024-09-16T05:17:13.105341+0000","last_undegraded":"2024-09-16T05:17:13.105341+0000","last_fullsized":"2024-09-16T05:17:13.105341+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:26.399340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.399340+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.857988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044780999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"1.c","version":"35'139","reported_seq":521,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.454782+0000","last_change":"2024-09-16T05:15:56.828705+0000","last_active":"2024-09-16T05:17:12.454782+0000","last_peered":"2024-09-16T05:17:12.454782+0000","last_clean":"2024-09-16T05:17:12.454782+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:17:12.454782+0000","last_undegraded":"2024-09-16T05:17:12.454782+0000","last_fullsized":"2024-09-16T05:17:12.454782+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.828595+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.828595+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:25.780000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"479'287","reported_seq":2170,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:13.105903+0000","last_change":"2024-09-16T05:17:13.106106+0000","last_active":"2024-09-16T05:17:13.105903+0000","last_peered":"2024-09-16T05:17:13.105903+0000","last_clean":"2024-09-16T05:17:13.105903+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:17:13.105903+0000","last_undegraded":"2024-09-16T05:17:13.105903+0000","last_fullsized":"2024-09-16T05:17:13.105903+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"437'282","last_scrub_stamp":"2024-09-16T05:16:38.132713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.132713+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:45.163208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083684000000000004,"stat_sum":{"num_bytes":3815288,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"1.d","version":"35'139","reported_seq":685,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184346+0000","last_change":"2024-09-16T05:15:57.170014+0000","last_active":"2024-09-16T05:17:11.184346+0000","last_peered":"2024-09-16T05:17:11.184346+0000","last_clean":"2024-09-16T05:17:11.184346+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:17:11.184346+0000","last_undegraded":"2024-09-16T05:17:11.184346+0000","last_fullsized":"2024-09-16T05:17:11.184346+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.169945+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.169945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.932365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":2014,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.735473+0000","last_change":"2024-09-16T05:17:12.735658+0000","last_active":"2024-09-16T05:17:12.735473+0000","last_peered":"2024-09-16T05:17:12.735473+0000","last_clean":"2024-09-16T05:17:12.735473+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:17:12.735473+0000","last_undegraded":"2024-09-16T05:17:12.735473+0000","last_fullsized":"2024-09-16T05:17:12.735473+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:16:38.980110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.980110+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:00.190989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086233200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"1.e","version":"35'139","reported_seq":683,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.733202+0000","last_change":"2024-09-16T05:17:01.066059+0000","last_active":"2024-09-16T05:17:10.733202+0000","last_peered":"2024-09-16T05:17:10.733202+0000","last_clean":"2024-09-16T05:17:10.733202+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:17:10.733202+0000","last_undegraded":"2024-09-16T05:17:10.733202+0000","last_fullsized":"2024-09-16T05:17:10.733202+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:01.065967+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.065967+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:26.866764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"479'485","reported_seq":2344,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736543+0000","last_change":"2024-09-16T05:17:08.755601+0000","last_active":"2024-09-16T05:17:09.736543+0000","last_peered":"2024-09-16T05:17:09.736543+0000","last_clean":"2024-09-16T05:17:09.736543+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:17:09.736543+0000","last_undegraded":"2024-09-16T05:17:09.736543+0000","last_fullsized":"2024-09-16T05:17:09.736543+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'461","last_scrub_stamp":"2024-09-16T05:16:40.073052+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:16:40.073052+0000","objects_scrubbed":1,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:40.544503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040297090000000002,"stat_sum":{"num_bytes":8777406,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":14,"num_write_kb":12109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.f","version":"35'139","reported_seq":666,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738954+0000","last_change":"2024-09-16T05:15:59.140824+0000","last_active":"2024-09-16T05:17:10.738954+0000","last_peered":"2024-09-16T05:17:10.738954+0000","last_clean":"2024-09-16T05:17:10.738954+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:17:10.738954+0000","last_undegraded":"2024-09-16T05:17:10.738954+0000","last_fullsized":"2024-09-16T05:17:10.738954+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.140738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.775574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":2132,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.931519+0000","last_change":"2024-09-16T05:17:11.931745+0000","last_active":"2024-09-16T05:17:11.931519+0000","last_peered":"2024-09-16T05:17:11.931519+0000","last_clean":"2024-09-16T05:17:11.931519+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:17:11.931519+0000","last_undegraded":"2024-09-16T05:17:11.931519+0000","last_fullsized":"2024-09-16T05:17:11.931519+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:16:25.811761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.811761+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.310216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082498399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1209,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.930753+0000","last_change":"2024-09-16T05:17:07.872302+0000","last_active":"2024-09-16T05:17:11.930753+0000","last_peered":"2024-09-16T05:17:11.930753+0000","last_clean":"2024-09-16T05:17:11.930753+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:17:11.930753+0000","last_undegraded":"2024-09-16T05:17:11.930753+0000","last_fullsized":"2024-09-16T05:17:11.930753+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:17:07.872208+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:11.877781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"451'255","reported_seq":2108,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185848+0000","last_change":"2024-09-16T05:17:11.186064+0000","last_active":"2024-09-16T05:17:11.185848+0000","last_peered":"2024-09-16T05:17:11.185848+0000","last_clean":"2024-09-16T05:17:11.185848+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:17:11.185848+0000","last_undegraded":"2024-09-16T05:17:11.185848+0000","last_fullsized":"2024-09-16T05:17:11.185848+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'222","last_scrub_stamp":"2024-09-16T05:16:00.063191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.063191+0000","objects_scrubbed":3,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:28.886742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016896229999999999,"stat_sum":{"num_bytes":2890903,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":12882,"num_write":25,"num_write_kb":5945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.3","version":"472'130","reported_seq":1988,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739728+0000","last_change":"2024-09-16T05:17:10.739916+0000","last_active":"2024-09-16T05:17:10.739728+0000","last_peered":"2024-09-16T05:17:10.739728+0000","last_clean":"2024-09-16T05:17:10.739728+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:17:10.739728+0000","last_undegraded":"2024-09-16T05:17:10.739728+0000","last_fullsized":"2024-09-16T05:17:10.739728+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'99","last_scrub_stamp":"2024-09-16T05:16:02.132568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.132568+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:17.257413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001307122,"stat_sum":{"num_bytes":2487535,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":49445,"num_write":22,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.0","version":"35'139","reported_seq":1194,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184683+0000","last_change":"2024-09-16T05:15:53.271304+0000","last_active":"2024-09-16T05:17:11.184683+0000","last_peered":"2024-09-16T05:17:11.184683+0000","last_clean":"2024-09-16T05:17:11.184683+0000","last_became_active":"2024-09-16T05:15:51.717864+0000","last_became_peered":"2024-09-16T05:15:51.717864+0000","last_unstale":"2024-09-16T05:17:11.184683+0000","last_undegraded":"2024-09-16T05:17:11.184683+0000","last_fullsized":"2024-09-16T05:17:11.184683+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.271201+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.271201+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.039968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1866,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.739799+0000","last_change":"2024-09-16T05:17:10.739962+0000","last_active":"2024-09-16T05:17:10.739799+0000","last_peered":"2024-09-16T05:17:10.739799+0000","last_clean":"2024-09-16T05:17:10.739799+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:17:10.739799+0000","last_undegraded":"2024-09-16T05:17:10.739799+0000","last_fullsized":"2024-09-16T05:17:10.739799+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:17:06.386786+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:17.599220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011944049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.1","version":"35'139","reported_seq":842,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935450+0000","last_change":"2024-09-16T05:17:06.854768+0000","last_active":"2024-09-16T05:17:10.935450+0000","last_peered":"2024-09-16T05:17:10.935450+0000","last_clean":"2024-09-16T05:17:10.935450+0000","last_became_active":"2024-09-16T05:15:51.723644+0000","last_became_peered":"2024-09-16T05:15:51.723644+0000","last_unstale":"2024-09-16T05:17:10.935450+0000","last_undegraded":"2024-09-16T05:17:10.935450+0000","last_fullsized":"2024-09-16T05:17:10.935450+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:06.854660+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:06.854660+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:21.485402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1198,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.183662+0000","last_change":"2024-09-16T05:17:03.402949+0000","last_active":"2024-09-16T05:17:11.183662+0000","last_peered":"2024-09-16T05:17:11.183662+0000","last_clean":"2024-09-16T05:17:11.183662+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:17:11.183662+0000","last_undegraded":"2024-09-16T05:17:11.183662+0000","last_fullsized":"2024-09-16T05:17:11.183662+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:03.402860+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.402860+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:12.276154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1823,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.734527+0000","last_change":"2024-09-16T05:17:12.734909+0000","last_active":"2024-09-16T05:17:12.734527+0000","last_peered":"2024-09-16T05:17:12.734527+0000","last_clean":"2024-09-16T05:17:12.734527+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:17:12.734527+0000","last_undegraded":"2024-09-16T05:17:12.734527+0000","last_fullsized":"2024-09-16T05:17:12.734527+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:23.812825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.812825+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:39.633037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045534599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"1.3","version":"35'139","reported_seq":1202,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.455009+0000","last_change":"2024-09-16T05:15:52.844991+0000","last_active":"2024-09-16T05:17:12.455009+0000","last_peered":"2024-09-16T05:17:12.455009+0000","last_clean":"2024-09-16T05:17:12.455009+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:17:12.455009+0000","last_undegraded":"2024-09-16T05:17:12.455009+0000","last_fullsized":"2024-09-16T05:17:12.455009+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:52.844849+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:52.844849+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.531754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1888,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.186113+0000","last_change":"2024-09-16T05:17:11.186271+0000","last_active":"2024-09-16T05:17:11.186113+0000","last_peered":"2024-09-16T05:17:11.186113+0000","last_clean":"2024-09-16T05:17:11.186113+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:17:11.186113+0000","last_undegraded":"2024-09-16T05:17:11.186113+0000","last_fullsized":"2024-09-16T05:17:11.186113+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:16:46.325304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:46.325304+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:07.467388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018838310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.7","version":"0'0","reported_seq":1100,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736412+0000","last_change":"2024-09-16T05:17:08.753457+0000","last_active":"2024-09-16T05:17:09.736412+0000","last_peered":"2024-09-16T05:17:09.736412+0000","last_clean":"2024-09-16T05:17:09.736412+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:17:09.736412+0000","last_undegraded":"2024-09-16T05:17:09.736412+0000","last_fullsized":"2024-09-16T05:17:09.736412+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:17:05.143266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:05.143266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:14.074436+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0025639149999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.4","version":"35'139","reported_seq":1165,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.930988+0000","last_change":"2024-09-16T05:16:58.840947+0000","last_active":"2024-09-16T05:17:11.930988+0000","last_peered":"2024-09-16T05:17:11.930988+0000","last_clean":"2024-09-16T05:17:11.930988+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:17:11.930988+0000","last_undegraded":"2024-09-16T05:17:11.930988+0000","last_fullsized":"2024-09-16T05:17:11.930988+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:58.840859+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:58.840859+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:20.477391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":305,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.734952+0000","last_change":"2024-09-16T05:17:12.735140+0000","last_active":"2024-09-16T05:17:12.734952+0000","last_peered":"2024-09-16T05:17:12.734952+0000","last_clean":"2024-09-16T05:17:12.734952+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:17:12.734952+0000","last_undegraded":"2024-09-16T05:17:12.734952+0000","last_fullsized":"2024-09-16T05:17:12.734952+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:17:00.760543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.760543+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:30.372821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053353500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":939,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935392+0000","last_change":"2024-09-16T05:15:56.607423+0000","last_active":"2024-09-16T05:17:10.935392+0000","last_peered":"2024-09-16T05:17:10.935392+0000","last_clean":"2024-09-16T05:17:10.935392+0000","last_became_active":"2024-09-16T05:15:51.724156+0000","last_became_peered":"2024-09-16T05:15:51.724156+0000","last_unstale":"2024-09-16T05:17:10.935392+0000","last_undegraded":"2024-09-16T05:17:10.935392+0000","last_fullsized":"2024-09-16T05:17:10.935392+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.607332+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.607332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.940520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"443'485","reported_seq":2318,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736402+0000","last_change":"2024-09-16T05:17:08.753959+0000","last_active":"2024-09-16T05:17:09.736402+0000","last_peered":"2024-09-16T05:17:09.736402+0000","last_clean":"2024-09-16T05:17:09.736402+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:17:09.736402+0000","last_undegraded":"2024-09-16T05:17:09.736402+0000","last_fullsized":"2024-09-16T05:17:09.736402+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'479","last_scrub_stamp":"2024-09-16T05:15:57.067519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.067519+0000","objects_scrubbed":2,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:12.036605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028257109999999998,"stat_sum":{"num_bytes":2672412,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20566,"num_write":5,"num_write_kb":1272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.6","version":"35'139","reported_seq":835,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.934678+0000","last_change":"2024-09-16T05:15:55.644520+0000","last_active":"2024-09-16T05:17:10.934678+0000","last_peered":"2024-09-16T05:17:10.934678+0000","last_clean":"2024-09-16T05:17:10.934678+0000","last_became_active":"2024-09-16T05:15:51.722851+0000","last_became_peered":"2024-09-16T05:15:51.722851+0000","last_unstale":"2024-09-16T05:17:10.934678+0000","last_undegraded":"2024-09-16T05:17:10.934678+0000","last_fullsized":"2024-09-16T05:17:10.934678+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:55.644436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.644436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.940555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1933,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759986+0000","last_change":"2024-09-16T05:17:08.760306+0000","last_active":"2024-09-16T05:17:09.759986+0000","last_peered":"2024-09-16T05:17:09.759986+0000","last_clean":"2024-09-16T05:17:09.759986+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:17:09.759986+0000","last_undegraded":"2024-09-16T05:17:09.759986+0000","last_fullsized":"2024-09-16T05:17:09.759986+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:57.869742+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.869742+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:16.876676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0021328609999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.7","version":"35'139","reported_seq":1019,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.934809+0000","last_change":"2024-09-16T05:15:53.643617+0000","last_active":"2024-09-16T05:17:10.934809+0000","last_peered":"2024-09-16T05:17:10.934809+0000","last_clean":"2024-09-16T05:17:10.934809+0000","last_became_active":"2024-09-16T05:15:51.718810+0000","last_became_peered":"2024-09-16T05:15:51.718810+0000","last_unstale":"2024-09-16T05:17:10.934809+0000","last_undegraded":"2024-09-16T05:17:10.934809+0000","last_fullsized":"2024-09-16T05:17:10.934809+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.643528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.643528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:13.940586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":2054,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:13.105579+0000","last_change":"2024-09-16T05:17:13.105764+0000","last_active":"2024-09-16T05:17:13.105579+0000","last_peered":"2024-09-16T05:17:13.105579+0000","last_clean":"2024-09-16T05:17:13.105579+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:17:13.105579+0000","last_undegraded":"2024-09-16T05:17:13.105579+0000","last_fullsized":"2024-09-16T05:17:13.105579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:16:58.139614+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:16:58.139614+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:23.481041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057372200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":1195,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738394+0000","last_change":"2024-09-16T05:16:20.118747+0000","last_active":"2024-09-16T05:17:10.738394+0000","last_peered":"2024-09-16T05:17:10.738394+0000","last_clean":"2024-09-16T05:17:10.738394+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:17:10.738394+0000","last_undegraded":"2024-09-16T05:17:10.738394+0000","last_fullsized":"2024-09-16T05:17:10.738394+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:20.118658+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.118658+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.459808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"479'489","reported_seq":2364,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.931021+0000","last_change":"2024-09-16T05:17:11.931329+0000","last_active":"2024-09-16T05:17:11.931021+0000","last_peered":"2024-09-16T05:17:11.931021+0000","last_clean":"2024-09-16T05:17:11.931021+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:17:11.931021+0000","last_undegraded":"2024-09-16T05:17:11.931021+0000","last_fullsized":"2024-09-16T05:17:11.931021+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'465","last_scrub_stamp":"2024-09-16T05:16:17.822233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.822233+0000","objects_scrubbed":1,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:17.927878+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00053640199999999995,"stat_sum":{"num_bytes":5215067,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":66423,"num_write":18,"num_write_kb":14367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1186,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.183599+0000","last_change":"2024-09-16T05:16:18.233415+0000","last_active":"2024-09-16T05:17:11.183599+0000","last_peered":"2024-09-16T05:17:11.183599+0000","last_clean":"2024-09-16T05:17:11.183599+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:17:11.183599+0000","last_undegraded":"2024-09-16T05:17:11.183599+0000","last_fullsized":"2024-09-16T05:17:11.183599+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:18.233282+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.233282+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:35.750725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2078,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.759568+0000","last_change":"2024-09-16T05:17:08.757958+0000","last_active":"2024-09-16T05:17:09.759568+0000","last_peered":"2024-09-16T05:17:09.759568+0000","last_clean":"2024-09-16T05:17:09.759568+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:17:09.759568+0000","last_undegraded":"2024-09-16T05:17:09.759568+0000","last_fullsized":"2024-09-16T05:17:09.759568+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:16:39.912418+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.912418+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:03.966880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000608129,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"1.a","version":"35'139","reported_seq":1193,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736024+0000","last_change":"2024-09-16T05:16:20.111881+0000","last_active":"2024-09-16T05:17:09.736024+0000","last_peered":"2024-09-16T05:17:09.736024+0000","last_clean":"2024-09-16T05:17:09.736024+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:17:09.736024+0000","last_undegraded":"2024-09-16T05:17:09.736024+0000","last_fullsized":"2024-09-16T05:17:09.736024+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:20.111791+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.111791+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:28.416563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":675,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735303+0000","last_change":"2024-09-16T05:17:08.734411+0000","last_active":"2024-09-16T05:17:09.735303+0000","last_peered":"2024-09-16T05:17:09.735303+0000","last_clean":"2024-09-16T05:17:09.735303+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:17:09.735303+0000","last_undegraded":"2024-09-16T05:17:09.735303+0000","last_fullsized":"2024-09-16T05:17:09.735303+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.743052+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.743052+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:12.904484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":995,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935774+0000","last_change":"2024-09-16T05:17:10.935949+0000","last_active":"2024-09-16T05:17:10.935774+0000","last_peered":"2024-09-16T05:17:10.935774+0000","last_clean":"2024-09-16T05:17:10.935774+0000","last_became_active":"2024-09-16T05:16:25.724548+0000","last_became_peered":"2024-09-16T05:16:25.724548+0000","last_unstale":"2024-09-16T05:17:10.935774+0000","last_undegraded":"2024-09-16T05:17:10.935774+0000","last_fullsized":"2024-09-16T05:17:10.935774+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:16:25.926404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.926404+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:48.534554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090903499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.10","version":"35'139","reported_seq":1193,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.738164+0000","last_change":"2024-09-16T05:15:58.103711+0000","last_active":"2024-09-16T05:17:10.738164+0000","last_peered":"2024-09-16T05:17:10.738164+0000","last_clean":"2024-09-16T05:17:10.738164+0000","last_became_active":"2024-09-16T05:15:56.719062+0000","last_became_peered":"2024-09-16T05:15:56.719062+0000","last_unstale":"2024-09-16T05:17:10.738164+0000","last_undegraded":"2024-09-16T05:17:10.738164+0000","last_fullsized":"2024-09-16T05:17:10.738164+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:58.103601+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:58.103601+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:27.359233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":1482,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736149+0000","last_change":"2024-09-16T05:17:08.750659+0000","last_active":"2024-09-16T05:17:09.736149+0000","last_peered":"2024-09-16T05:17:09.736149+0000","last_clean":"2024-09-16T05:17:09.736149+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:17:09.736149+0000","last_undegraded":"2024-09-16T05:17:09.736149+0000","last_fullsized":"2024-09-16T05:17:09.736149+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:26.057442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.057442+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:35.072063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010375880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.14","version":"433'289","reported_seq":1754,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736236+0000","last_change":"2024-09-16T05:17:08.752241+0000","last_active":"2024-09-16T05:17:09.736236+0000","last_peered":"2024-09-16T05:17:09.736236+0000","last_clean":"2024-09-16T05:17:09.736236+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:17:09.736236+0000","last_undegraded":"2024-09-16T05:17:09.736236+0000","last_fullsized":"2024-09-16T05:17:09.736236+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:16:37.997840+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.997840+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:04.271517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020694530000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.15","version":"454'519","reported_seq":2385,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736424+0000","last_change":"2024-09-16T05:17:08.754356+0000","last_active":"2024-09-16T05:17:09.736424+0000","last_peered":"2024-09-16T05:17:09.736424+0000","last_clean":"2024-09-16T05:17:09.736424+0000","last_became_active":"2024-09-16T05:16:25.732256+0000","last_became_peered":"2024-09-16T05:16:25.732256+0000","last_unstale":"2024-09-16T05:17:09.736424+0000","last_undegraded":"2024-09-16T05:17:09.736424+0000","last_fullsized":"2024-09-16T05:17:09.736424+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"438'513","last_scrub_stamp":"2024-09-16T05:16:39.040839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.040839+0000","objects_scrubbed":1,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:00.672366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031403799999999999,"stat_sum":{"num_bytes":3008552,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.16","version":"481'180","reported_seq":1677,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.751665+0000","last_change":"2024-09-16T05:17:10.740164+0000","last_active":"2024-09-16T05:17:10.751665+0000","last_peered":"2024-09-16T05:17:10.751665+0000","last_clean":"2024-09-16T05:17:10.751665+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:17:10.751665+0000","last_undegraded":"2024-09-16T05:17:10.751665+0000","last_fullsized":"2024-09-16T05:17:10.751665+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"451'178","last_scrub_stamp":"2024-09-16T05:16:59.339858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:59.339858+0000","objects_scrubbed":1,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:01.419261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011560940000000001,"stat_sum":{"num_bytes":4420439,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":25313,"num_write":23,"num_write_kb":9066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"1.14","version":"35'139","reported_seq":525,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.734387+0000","last_change":"2024-09-16T05:16:17.723651+0000","last_active":"2024-09-16T05:17:12.734387+0000","last_peered":"2024-09-16T05:17:12.734387+0000","last_clean":"2024-09-16T05:17:12.734387+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:17:12.734387+0000","last_undegraded":"2024-09-16T05:17:12.734387+0000","last_fullsized":"2024-09-16T05:17:12.734387+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.762419+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.762419+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:14.850921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1659,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.740692+0000","last_change":"2024-09-16T05:17:11.741236+0000","last_active":"2024-09-16T05:17:11.740692+0000","last_peered":"2024-09-16T05:17:11.740692+0000","last_clean":"2024-09-16T05:17:11.740692+0000","last_became_active":"2024-09-16T05:13:31.745537+0000","last_became_peered":"2024-09-16T05:13:31.745537+0000","last_unstale":"2024-09-16T05:17:11.740692+0000","last_undegraded":"2024-09-16T05:17:11.740692+0000","last_fullsized":"2024-09-16T05:17:11.740692+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:57.269939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.269939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099584299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1864,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735878+0000","last_change":"2024-09-16T05:17:08.753948+0000","last_active":"2024-09-16T05:17:09.735878+0000","last_peered":"2024-09-16T05:17:09.735878+0000","last_clean":"2024-09-16T05:17:09.735878+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:17:09.735878+0000","last_undegraded":"2024-09-16T05:17:09.735878+0000","last_fullsized":"2024-09-16T05:17:09.735878+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:16:44.037920+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:16:44.037920+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:52.405077+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030712309999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.19","version":"182'60","reported_seq":1481,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.455538+0000","last_change":"2024-09-16T05:17:12.456204+0000","last_active":"2024-09-16T05:17:12.455538+0000","last_peered":"2024-09-16T05:17:12.455538+0000","last_clean":"2024-09-16T05:17:12.455538+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:17:12.455538+0000","last_undegraded":"2024-09-16T05:17:12.455538+0000","last_fullsized":"2024-09-16T05:17:12.455538+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:16:19.812615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.812615+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:41.597148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063597600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.1a","version":"472'688","reported_seq":2533,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:13.105619+0000","last_change":"2024-09-16T05:17:13.105851+0000","last_active":"2024-09-16T05:17:13.105619+0000","last_peered":"2024-09-16T05:17:13.105619+0000","last_clean":"2024-09-16T05:17:13.105619+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:17:13.105619+0000","last_undegraded":"2024-09-16T05:17:13.105619+0000","last_fullsized":"2024-09-16T05:17:13.105619+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"435'666","last_scrub_stamp":"2024-09-16T05:16:25.407253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.407253+0000","objects_scrubbed":3,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:27.733481+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00062859999999999999,"stat_sum":{"num_bytes":8428471,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":80160,"num_write":61,"num_write_kb":28646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"2.1b","version":"482'410","reported_seq":2294,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.932756+0000","last_change":"2024-09-16T05:17:11.933100+0000","last_active":"2024-09-16T05:17:11.932756+0000","last_peered":"2024-09-16T05:17:11.932756+0000","last_clean":"2024-09-16T05:17:11.932756+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:17:11.932756+0000","last_undegraded":"2024-09-16T05:17:11.932756+0000","last_fullsized":"2024-09-16T05:17:11.932756+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"429'364","last_scrub_stamp":"2024-09-16T05:16:19.854079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.854079+0000","objects_scrubbed":4,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:46.585785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021804319999999999,"stat_sum":{"num_bytes":9206293,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":40185,"num_write":34,"num_write_kb":10899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.1c","version":"468'393","reported_seq":2305,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.935529+0000","last_change":"2024-09-16T05:17:10.935739+0000","last_active":"2024-09-16T05:17:10.935529+0000","last_peered":"2024-09-16T05:17:10.935529+0000","last_clean":"2024-09-16T05:17:10.935529+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:17:10.935529+0000","last_undegraded":"2024-09-16T05:17:10.935529+0000","last_fullsized":"2024-09-16T05:17:10.935529+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'392","last_scrub_stamp":"2024-09-16T05:16:45.755860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:45.755860+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:06.827595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096189500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.1d","version":"477'821","reported_seq":2755,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.936071+0000","last_change":"2024-09-16T05:17:10.936258+0000","last_active":"2024-09-16T05:17:10.936071+0000","last_peered":"2024-09-16T05:17:10.936071+0000","last_clean":"2024-09-16T05:17:10.936071+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:17:10.936071+0000","last_undegraded":"2024-09-16T05:17:10.936071+0000","last_fullsized":"2024-09-16T05:17:10.936071+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"457'781","last_scrub_stamp":"2024-09-16T05:16:46.788437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:46.788437+0000","objects_scrubbed":4,"log_size":821,"log_dups_size":0,"ondisk_log_size":821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:57.901598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096913800000000003,"stat_sum":{"num_bytes":14083730,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":91294,"num_write":111,"num_write_kb":42451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.1e","version":"436'363","reported_seq":2247,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.456013+0000","last_change":"2024-09-16T05:17:12.456242+0000","last_active":"2024-09-16T05:17:12.456013+0000","last_peered":"2024-09-16T05:17:12.456013+0000","last_clean":"2024-09-16T05:17:12.456013+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:17:12.456013+0000","last_undegraded":"2024-09-16T05:17:12.456013+0000","last_fullsized":"2024-09-16T05:17:12.456013+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"436'363","last_scrub_stamp":"2024-09-16T05:16:39.785728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.785728+0000","objects_scrubbed":1,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:06.426505+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053751999999999999,"stat_sum":{"num_bytes":2067360,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":26094,"num_write":9,"num_write_kb":4850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":2081,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.735753+0000","last_change":"2024-09-16T05:17:08.735935+0000","last_active":"2024-09-16T05:17:09.735753+0000","last_peered":"2024-09-16T05:17:09.735753+0000","last_clean":"2024-09-16T05:17:09.735753+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:17:09.735753+0000","last_undegraded":"2024-09-16T05:17:09.735753+0000","last_fullsized":"2024-09-16T05:17:09.735753+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.634458+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:48.717367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012194689999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.20","version":"472'179","reported_seq":2061,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184396+0000","last_change":"2024-09-16T05:17:11.185060+0000","last_active":"2024-09-16T05:17:11.184396+0000","last_peered":"2024-09-16T05:17:11.184396+0000","last_clean":"2024-09-16T05:17:11.184396+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:17:11.184396+0000","last_undegraded":"2024-09-16T05:17:11.184396+0000","last_fullsized":"2024-09-16T05:17:11.184396+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"451'174","last_scrub_stamp":"2024-09-16T05:17:01.386565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.386565+0000","objects_scrubbed":1,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:22.137223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012705920000000001,"stat_sum":{"num_bytes":4887302,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":93739,"num_write":26,"num_write_kb":11335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.21","version":"178'50","reported_seq":1822,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.734584+0000","last_change":"2024-09-16T05:17:12.734886+0000","last_active":"2024-09-16T05:17:12.734584+0000","last_peered":"2024-09-16T05:17:12.734584+0000","last_clean":"2024-09-16T05:17:12.734584+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:17:12.734584+0000","last_undegraded":"2024-09-16T05:17:12.734584+0000","last_fullsized":"2024-09-16T05:17:12.734584+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:19.705056+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.705056+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:32.132725+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044939300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"2.22","version":"182'20","reported_seq":992,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:10.936001+0000","last_change":"2024-09-16T05:17:10.936215+0000","last_active":"2024-09-16T05:17:10.936001+0000","last_peered":"2024-09-16T05:17:10.936001+0000","last_clean":"2024-09-16T05:17:10.936001+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:17:10.936001+0000","last_undegraded":"2024-09-16T05:17:10.936001+0000","last_fullsized":"2024-09-16T05:17:10.936001+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:16:43.744675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:43.744675+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.976692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000923097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.23","version":"184'17","reported_seq":1348,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.454890+0000","last_change":"2024-09-16T05:17:12.455265+0000","last_active":"2024-09-16T05:17:12.454890+0000","last_peered":"2024-09-16T05:17:12.454890+0000","last_clean":"2024-09-16T05:17:12.454890+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:17:12.454890+0000","last_undegraded":"2024-09-16T05:17:12.454890+0000","last_fullsized":"2024-09-16T05:17:12.454890+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:16:17.214142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.214142+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:36.979469+0000","scrub_duration":429,"objects_trimmed":0,"snaptrim_duration":0.00055004800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":1149,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736160+0000","last_change":"2024-09-16T05:17:08.752302+0000","last_active":"2024-09-16T05:17:09.736160+0000","last_peered":"2024-09-16T05:17:09.736160+0000","last_clean":"2024-09-16T05:17:09.736160+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:17:09.736160+0000","last_undegraded":"2024-09-16T05:17:09.736160+0000","last_fullsized":"2024-09-16T05:17:09.736160+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.030307+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.149889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00225997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]},{"pgid":"2.25","version":"463'688","reported_seq":2498,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:13.106158+0000","last_change":"2024-09-16T05:17:13.106351+0000","last_active":"2024-09-16T05:17:13.106158+0000","last_peered":"2024-09-16T05:17:13.106158+0000","last_clean":"2024-09-16T05:17:13.106158+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:17:13.106158+0000","last_undegraded":"2024-09-16T05:17:13.106158+0000","last_fullsized":"2024-09-16T05:17:13.106158+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"463'688","last_scrub_stamp":"2024-09-16T05:17:03.105650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.105650+0000","objects_scrubbed":2,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:30.625024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00099295799999999995,"stat_sum":{"num_bytes":5210302,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":76000,"num_write":43,"num_write_kb":22034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"2.26","version":"116'9","reported_seq":1151,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185563+0000","last_change":"2024-09-16T05:17:11.185779+0000","last_active":"2024-09-16T05:17:11.185563+0000","last_peered":"2024-09-16T05:17:11.185563+0000","last_clean":"2024-09-16T05:17:11.185563+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:17:11.185563+0000","last_undegraded":"2024-09-16T05:17:11.185563+0000","last_fullsized":"2024-09-16T05:17:11.185563+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:16:13.148890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:13.148890+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:39.830710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015047509999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.27","version":"0'0","reported_seq":1163,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.184845+0000","last_change":"2024-09-16T05:17:11.185130+0000","last_active":"2024-09-16T05:17:11.184845+0000","last_peered":"2024-09-16T05:17:11.184845+0000","last_clean":"2024-09-16T05:17:11.184845+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:17:11.184845+0000","last_undegraded":"2024-09-16T05:17:11.184845+0000","last_fullsized":"2024-09-16T05:17:11.184845+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:16:17.188503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.188503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:18.771471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000929556,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.28","version":"245'105","reported_seq":1948,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.455348+0000","last_change":"2024-09-16T05:17:12.456389+0000","last_active":"2024-09-16T05:17:12.455348+0000","last_peered":"2024-09-16T05:17:12.455348+0000","last_clean":"2024-09-16T05:17:12.455348+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:17:12.455348+0000","last_undegraded":"2024-09-16T05:17:12.455348+0000","last_fullsized":"2024-09-16T05:17:12.455348+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:15:57.809122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.809122+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014991189999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.29","version":"481'668","reported_seq":2567,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.740553+0000","last_change":"2024-09-16T05:17:11.741161+0000","last_active":"2024-09-16T05:17:11.740553+0000","last_peered":"2024-09-16T05:17:11.740553+0000","last_clean":"2024-09-16T05:17:11.740553+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:17:11.740553+0000","last_undegraded":"2024-09-16T05:17:11.740553+0000","last_fullsized":"2024-09-16T05:17:11.740553+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"472'660","last_scrub_stamp":"2024-09-16T05:17:03.427111+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.427111+0000","objects_scrubbed":2,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:04.591524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012141540000000001,"stat_sum":{"num_bytes":5933069,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":116,"num_read_kb":111541,"num_write":105,"num_write_kb":41636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":2153,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.183994+0000","last_change":"2024-09-16T05:17:11.184356+0000","last_active":"2024-09-16T05:17:11.183994+0000","last_peered":"2024-09-16T05:17:11.183994+0000","last_clean":"2024-09-16T05:17:11.183994+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:17:11.183994+0000","last_undegraded":"2024-09-16T05:17:11.183994+0000","last_fullsized":"2024-09-16T05:17:11.183994+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.269448+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:38.633691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061950299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.2b","version":"481'262","reported_seq":2050,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:12.734868+0000","last_change":"2024-09-16T05:17:12.735499+0000","last_active":"2024-09-16T05:17:12.734868+0000","last_peered":"2024-09-16T05:17:12.734868+0000","last_clean":"2024-09-16T05:17:12.734868+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:17:12.734868+0000","last_undegraded":"2024-09-16T05:17:12.734868+0000","last_fullsized":"2024-09-16T05:17:12.734868+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"406'216","last_scrub_stamp":"2024-09-16T05:16:17.634887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.634887+0000","objects_scrubbed":2,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:35.292904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093420600000000003,"stat_sum":{"num_bytes":3344692,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":41,"num_read_kb":42424,"num_write":30,"num_write_kb":14799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"2.2c","version":"241'194","reported_seq":2001,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-09-16T05:17:13.106092+0000","last_change":"2024-09-16T05:17:13.106328+0000","last_active":"2024-09-16T05:17:13.106092+0000","last_peered":"2024-09-16T05:17:13.106092+0000","last_clean":"2024-09-16T05:17:13.106092+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:17:13.106092+0000","last_undegraded":"2024-09-16T05:17:13.106092+0000","last_fullsized":"2024-09-16T05:17:13.106092+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:17:02.077362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:02.077362+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:19.877717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010374710000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ac","length":"3"},{"start":"b0","length":"2"}]},{"pgid":"2.2d","version":"477'365","reported_seq":2211,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.186310+0000","last_change":"2024-09-16T05:17:11.186608+0000","last_active":"2024-09-16T05:17:11.186310+0000","last_peered":"2024-09-16T05:17:11.186310+0000","last_clean":"2024-09-16T05:17:11.186310+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:17:11.186310+0000","last_undegraded":"2024-09-16T05:17:11.186310+0000","last_fullsized":"2024-09-16T05:17:11.186310+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"377'326","last_scrub_stamp":"2024-09-16T05:16:15.212317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:15.212317+0000","objects_scrubbed":1,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:15.678470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020341069999999998,"stat_sum":{"num_bytes":5027150,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":63587,"num_write":20,"num_write_kb":10086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.2e","version":"471'425","reported_seq":2250,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-09-16T05:17:11.185195+0000","last_change":"2024-09-16T05:17:11.185511+0000","last_active":"2024-09-16T05:17:11.185195+0000","last_peered":"2024-09-16T05:17:11.185195+0000","last_clean":"2024-09-16T05:17:11.185195+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:17:11.185195+0000","last_undegraded":"2024-09-16T05:17:11.185195+0000","last_fullsized":"2024-09-16T05:17:11.185195+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"428'412","last_scrub_stamp":"2024-09-16T05:16:19.216112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.216112+0000","objects_scrubbed":3,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:20.291740+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013118730000000001,"stat_sum":{"num_bytes":2958098,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":32166,"num_write":37,"num_write_kb":13743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ac","length":"3"}]},{"pgid":"2.2f","version":"479'392","reported_seq":2278,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-16T05:17:09.736424+0000","last_change":"2024-09-16T05:17:08.753349+0000","last_active":"2024-09-16T05:17:09.736424+0000","last_peered":"2024-09-16T05:17:09.736424+0000","last_clean":"2024-09-16T05:17:09.736424+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:17:09.736424+0000","last_undegraded":"2024-09-16T05:17:09.736424+0000","last_fullsized":"2024-09-16T05:17:09.736424+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"422'318","last_scrub_stamp":"2024-09-16T05:16:14.124319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.124319+0000","objects_scrubbed":4,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:29.416011+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026651800000000001,"stat_sum":{"num_bytes":6823840,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":25591,"num_write":57,"num_write_kb":19443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ab","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":142298846,"num_objects":58,"num_object_clones":19,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":4,"num_read":1290,"num_read_kb":1301382,"num_write":935,"num_write_kb":398621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190382080,"data_stored":216090578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19347,"ondisk_log_size":19347,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089441,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70496,"kb_used_data":30988,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93674656,"statfs":{"total":95995035648,"available":95922847744,"internally_reserved":0,"allocated":31731712,"data_stored":32707176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285670,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63564,"kb_used_data":24056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93681588,"statfs":{"total":95995035648,"available":95929946112,"internally_reserved":0,"allocated":24633344,"data_stored":25596681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514606,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56152,"kb_used_data":29116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93689000,"statfs":{"total":95995035648,"available":95937536000,"internally_reserved":0,"allocated":29814784,"data_stored":30828217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60324,"kb_used_data":20812,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93684828,"statfs":{"total":95995035648,"available":95933263872,"internally_reserved":0,"allocated":21311488,"data_stored":20517712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103285,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56864,"kb_used_data":29836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93688288,"statfs":{"total":95995035648,"available":95936806912,"internally_reserved":0,"allocated":30552064,"data_stored":34480631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397628,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50884,"kb_used_data":23852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93694268,"statfs":{"total":95995035648,"available":95942930432,"internally_reserved":0,"allocated":24424448,"data_stored":25794531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757374,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54308,"kb_used_data":18948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93690844,"statfs":{"total":95995035648,"available":95939424256,"internally_reserved":0,"allocated":19402752,"data_stored":19867226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50520,"kb_used_data":15164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93694632,"statfs":{"total":95995035648,"available":95943303168,"internally_reserved":0,"allocated":15527936,"data_stored":14594692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117129,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64724,"kb_used_data":25216,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93680428,"statfs":{"total":95995035648,"available":95928758272,"internally_reserved":0,"allocated":25821184,"data_stored":24814967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240416,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45500,"kb_used_data":13220,"kb_used_omap":10,"kb_used_meta":32245,"kb_avail":93699652,"statfs":{"total":95995035648,"available":95948443648,"internally_reserved":0,"allocated":13537280,"data_stored":13196557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":33019187},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51592,"kb_used_data":12080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93693560,"statfs":{"total":95995035648,"available":95942205440,"internally_reserved":0,"allocated":12369920,"data_stored":10313091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738520,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49192,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93695960,"statfs":{"total":95995035648,"available":95944663040,"internally_reserved":0,"allocated":9912320,"data_stored":7877492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3825664,"data_stored":3800639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6623232,"data_stored":6572365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7475200,"data_stored":9159191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19709952,"data_stored":20721128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9768960,"data_stored":10845519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15138816,"data_stored":17612832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":18640896,"data_stored":22028372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24780800,"data_stored":30722941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15527936,"data_stored":16751553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24055808,"data_stored":27079044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":18874368,"data_stored":21847508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":25960448,"data_stored":28949486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:17:16.707 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T05:17:16.707 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:17:16.708 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T05:17:16.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:17:16.720 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:16.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:16.737 INFO:tasks.rados.rados.2.smithi204.stdout:1640: done (3 left) 2024-09-16T05:17:16.737 INFO:tasks.rados.rados.2.smithi204.stdout:1641: done (2 left) 2024-09-16T05:17:16.737 INFO:tasks.rados.rados.2.smithi204.stdout:1642: done (1 left) 2024-09-16T05:17:16.737 INFO:tasks.rados.rados.2.smithi204.stdout:1643: done (0 left) 2024-09-16T05:17:16.737 INFO:tasks.rados.rados.2.smithi204.stdout:1644: delete oid 9 current snap is 179 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1644: done (0 left) 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1645: read oid 21 snap 173 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1645: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1646: read oid 40 snap -1 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1646: expect (ObjNum 480 snap 151 seq_num 480) 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1647: read oid 19 snap -1 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1647: expect (ObjNum 552 snap 165 seq_num 552) 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1648: write (excl) oid 26 current snap is 179 2024-09-16T05:17:16.741 INFO:tasks.rados.rados.2.smithi204.stdout:1648: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2024-09-16T05:17:16.748 INFO:tasks.rados.rados.2.smithi204.stdout:1648: writing smithi20457400-26 from 716130 to 1272221 tid 1 2024-09-16T05:17:16.755 INFO:tasks.rados.rados.2.smithi204.stdout:1648: writing smithi20457400-26 from 1792376 to 2365906 tid 2 2024-09-16T05:17:16.760 INFO:tasks.rados.rados.2.smithi204.stdout:1648: writing smithi20457400-26 from 2769448 to 3229465 tid 3 2024-09-16T05:17:16.764 INFO:tasks.rados.rados.2.smithi204.stdout:1648: writing smithi20457400-26 from 3821956 to 3916130 tid 4 2024-09-16T05:17:16.764 INFO:tasks.rados.rados.2.smithi204.stdout:1649: read oid 6 snap 177 2024-09-16T05:17:16.764 INFO:tasks.rados.rados.2.smithi204.stdout:1649: expect (ObjNum 571 snap 170 seq_num 571) 2024-09-16T05:17:16.764 INFO:tasks.rados.rados.2.smithi204.stdout:1650: read oid 20 snap -1 2024-09-16T05:17:16.764 INFO:tasks.rados.rados.2.smithi204.stdout:1650: expect (ObjNum 588 snap 178 seq_num 588) 2024-09-16T05:17:16.764 INFO:tasks.rados.rados.2.smithi204.stdout:1651: snap_create 2024-09-16T05:17:16.764 INFO:tasks.rados.rados.2.smithi204.stdout:1647: expect (ObjNum 552 snap 165 seq_num 552) 2024-09-16T05:17:16.792 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:16.802 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:16.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:16.844 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:16.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:16.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:16.962 INFO:tasks.rados.rados.2.smithi204.stdout:1646: expect (ObjNum 480 snap 151 seq_num 480) 2024-09-16T05:17:16.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:16.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:17.063 INFO:tasks.rados.rados.2.smithi204.stdout:1648: finishing write tid 1 to smithi20457400-26 2024-09-16T05:17:17.063 INFO:tasks.rados.rados.2.smithi204.stdout:1645: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:17:17.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:17.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:17.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='client.16092 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: osdmap e487: 12 total, 12 up, 12 in 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09552211654437891]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08316979992116041]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10032813749732372]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08834387676864482]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09019017869438925]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08973597265765404]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08437634142417849]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08004412249142938]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08920486825821045]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08200439243853135]}]: dispatch 2024-09-16T05:17:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07812504495077578]}]: dispatch 2024-09-16T05:17:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:16 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08650153507207352]}]: dispatch 2024-09-16T05:17:17.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:17.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:17.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='client.16092 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: osdmap e487: 12 total, 12 up, 12 in 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09552211654437891]}]: dispatch 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08316979992116041]}]: dispatch 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10032813749732372]}]: dispatch 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08834387676864482]}]: dispatch 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09019017869438925]}]: dispatch 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08973597265765404]}]: dispatch 2024-09-16T05:17:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08437634142417849]}]: dispatch 2024-09-16T05:17:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08004412249142938]}]: dispatch 2024-09-16T05:17:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08920486825821045]}]: dispatch 2024-09-16T05:17:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08200439243853135]}]: dispatch 2024-09-16T05:17:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07812504495077578]}]: dispatch 2024-09-16T05:17:17.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:16 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08650153507207352]}]: dispatch 2024-09-16T05:17:17.253 INFO:tasks.rados.rados.2.smithi204.stdout:1648: finishing write tid 2 to smithi20457400-26 2024-09-16T05:17:17.253 INFO:tasks.rados.rados.2.smithi204.stdout:1648: finishing write tid 3 to smithi20457400-26 2024-09-16T05:17:17.254 INFO:tasks.rados.rados.2.smithi204.stdout:1648: finishing write tid 4 to smithi20457400-26 2024-09-16T05:17:17.254 INFO:tasks.rados.rados.2.smithi204.stdout:1648: finishing write tid 5 to smithi20457400-26 2024-09-16T05:17:17.254 INFO:tasks.rados.rados.2.smithi204.stdout:1648: finishing write tid 6 to smithi20457400-26 2024-09-16T05:17:17.254 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 827 (ObjNum 590 snap 179 seq_num 590) dirty exists 2024-09-16T05:17:17.254 INFO:tasks.rados.rados.2.smithi204.stdout:1648: left oid 26 (ObjNum 590 snap 179 seq_num 590) 2024-09-16T05:17:17.254 INFO:tasks.rados.rados.2.smithi204.stdout:1650: expect (ObjNum 588 snap 178 seq_num 588) 2024-09-16T05:17:17.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:17.349 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:17.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='client.16092 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: osdmap e487: 12 total, 12 up, 12 in 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09552211654437891]}]: dispatch 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08316979992116041]}]: dispatch 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10032813749732372]}]: dispatch 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08834387676864482]}]: dispatch 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09019017869438925]}]: dispatch 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08973597265765404]}]: dispatch 2024-09-16T05:17:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08437634142417849]}]: dispatch 2024-09-16T05:17:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08004412249142938]}]: dispatch 2024-09-16T05:17:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08920486825821045]}]: dispatch 2024-09-16T05:17:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08200439243853135]}]: dispatch 2024-09-16T05:17:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07812504495077578]}]: dispatch 2024-09-16T05:17:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:16 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08650153507207352]}]: dispatch 2024-09-16T05:17:17.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:17.469 INFO:tasks.rados.rados.2.smithi204.stdout:1649: expect (ObjNum 571 snap 170 seq_num 571) 2024-09-16T05:17:17.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:17.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:17.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:17.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:17.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:17.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:17.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:17.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:17.737 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:17:17.791 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:17:17.791 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:17:17.791 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:17:17.792 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:17:17.795 INFO:tasks.rados.rados.2.smithi204.stdout:1645: done (6 left) 2024-09-16T05:17:17.796 INFO:tasks.rados.rados.2.smithi204.stdout:1646: done (5 left) 2024-09-16T05:17:17.796 INFO:tasks.rados.rados.2.smithi204.stdout:1647: done (4 left) 2024-09-16T05:17:17.796 INFO:tasks.rados.rados.2.smithi204.stdout:1648: done (3 left) 2024-09-16T05:17:17.796 INFO:tasks.rados.rados.2.smithi204.stdout:1649: done (2 left) 2024-09-16T05:17:17.796 INFO:tasks.rados.rados.2.smithi204.stdout:1650: done (1 left) 2024-09-16T05:17:17.796 INFO:tasks.rados.rados.2.smithi204.stdout:1651: done (0 left) 2024-09-16T05:17:17.796 INFO:tasks.rados.rados.2.smithi204.stdout:1652: snap_remove snap 179 2024-09-16T05:17:17.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:17.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:17.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:17.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:17.974 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:17.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:18.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:18.098 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:18.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:18.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:17:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: pgmap v891: 107 pgs: 107 active+clean; 139 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:17:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/40127321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:17:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09552211654437891]}]': finished 2024-09-16T05:17:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08316979992116041]}]': finished 2024-09-16T05:17:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10032813749732372]}]': finished 2024-09-16T05:17:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08834387676864482]}]': finished 2024-09-16T05:17:18.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09019017869438925]}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08973597265765404]}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08437634142417849]}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08004412249142938]}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08920486825821045]}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08200439243853135]}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07812504495077578]}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08650153507207352]}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/40127321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:17:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:17 smithi032 ceph-mon[27388]: osdmap e488: 12 total, 12 up, 12 in 2024-09-16T05:17:18.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:18.222 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:18.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: pgmap v891: 107 pgs: 107 active+clean; 139 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:17:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/40127321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:17:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09552211654437891]}]': finished 2024-09-16T05:17:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08316979992116041]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10032813749732372]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08834387676864482]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09019017869438925]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08973597265765404]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08437634142417849]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08004412249142938]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08920486825821045]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08200439243853135]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07812504495077578]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08650153507207352]}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/40127321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:17:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:17 smithi196 ceph-mon[30802]: osdmap e488: 12 total, 12 up, 12 in 2024-09-16T05:17:18.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:18.346 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:18.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: pgmap v891: 107 pgs: 107 active+clean; 139 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/40127321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09552211654437891]}]': finished 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08316979992116041]}]': finished 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10032813749732372]}]': finished 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08834387676864482]}]': finished 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09019017869438925]}]': finished 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08973597265765404]}]': finished 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08437634142417849]}]': finished 2024-09-16T05:17:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08004412249142938]}]': finished 2024-09-16T05:17:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08920486825821045]}]': finished 2024-09-16T05:17:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08200439243853135]}]': finished 2024-09-16T05:17:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07812504495077578]}]': finished 2024-09-16T05:17:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08650153507207352]}]': finished 2024-09-16T05:17:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/40127321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:17:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:17 smithi204 ceph-mon[30933]: osdmap e488: 12 total, 12 up, 12 in 2024-09-16T05:17:18.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:18.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:18.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:18.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:18.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:18.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:18.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:18.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:18.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:18.743 INFO:tasks.rados.rados.2.smithi204.stdout:1652: done (0 left) 2024-09-16T05:17:18.743 INFO:tasks.rados.rados.2.smithi204.stdout:1653: copy_from oid 11 from oid 4 current snap is 180 2024-09-16T05:17:18.743 INFO:tasks.rados.rados.2.smithi204.stdout:1654: write oid 33 current snap is 180 2024-09-16T05:17:18.743 INFO:tasks.rados.rados.2.smithi204.stdout:1654: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-09-16T05:17:18.751 INFO:tasks.rados.rados.2.smithi204.stdout:1654: writing smithi20457400-33 from 732937 to 1364277 tid 1 2024-09-16T05:17:18.756 INFO:tasks.rados.rados.2.smithi204.stdout:1654: writing smithi20457400-33 from 1850858 to 2284399 tid 2 2024-09-16T05:17:18.763 INFO:tasks.rados.rados.2.smithi204.stdout:1654: writing smithi20457400-33 from 2796871 to 3468160 tid 3 2024-09-16T05:17:18.766 INFO:tasks.rados.rados.2.smithi204.stdout:1654: writing smithi20457400-33 from 3888195 to 3932937 tid 4 2024-09-16T05:17:18.766 INFO:tasks.rados.rados.2.smithi204.stdout:1655: write oid 50 current snap is 180 2024-09-16T05:17:18.767 INFO:tasks.rados.rados.2.smithi204.stdout:1654: finishing write tid 1 to smithi20457400-33 2024-09-16T05:17:18.767 INFO:tasks.rados.rados.2.smithi204.stdout:1654: finishing write tid 2 to smithi20457400-33 2024-09-16T05:17:18.767 INFO:tasks.rados.rados.2.smithi204.stdout:1653: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:17:18.767 INFO:tasks.rados.rados.2.smithi204.stdout:1655: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2024-09-16T05:17:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:1655: writing smithi20457400-50 from 749744 to 1456333 tid 1 2024-09-16T05:17:18.779 INFO:tasks.rados.rados.2.smithi204.stdout:1655: writing smithi20457400-50 from 1909340 to 2486539 tid 2 2024-09-16T05:17:18.787 INFO:tasks.rados.rados.2.smithi204.stdout:1655: writing smithi20457400-50 from 3224294 to 3706855 tid 3 2024-09-16T05:17:18.787 INFO:tasks.rados.rados.2.smithi204.stdout:1656: read oid 2 snap -1 2024-09-16T05:17:18.787 INFO:tasks.rados.rados.2.smithi204.stdout:1656: expect (ObjNum 546 snap 163 seq_num 546) 2024-09-16T05:17:18.787 INFO:tasks.rados.rados.2.smithi204.stdout:1657: delete oid 28 current snap is 180 2024-09-16T05:17:18.787 INFO:tasks.rados.rados.2.smithi204.stdout:1653: finishing copy_from to smithi20457400-11 2024-09-16T05:17:18.787 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 306 (ObjNum 529 snap 159 seq_num 529) dirty exists 2024-09-16T05:17:18.787 INFO:tasks.rados.rados.2.smithi204.stdout:1654: finishing write tid 3 to smithi20457400-33 2024-09-16T05:17:18.787 INFO:tasks.rados.rados.2.smithi204.stdout:1654: finishing write tid 4 to smithi20457400-33 2024-09-16T05:17:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1654: finishing write tid 5 to smithi20457400-33 2024-09-16T05:17:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1654: finishing write tid 6 to smithi20457400-33 2024-09-16T05:17:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 746 (ObjNum 591 snap 180 seq_num 591) dirty exists 2024-09-16T05:17:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1654: left oid 33 (ObjNum 591 snap 180 seq_num 591) 2024-09-16T05:17:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1655: finishing write tid 1 to smithi20457400-50 2024-09-16T05:17:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1655: finishing write tid 2 to smithi20457400-50 2024-09-16T05:17:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1653: done (4 left) 2024-09-16T05:17:18.788 INFO:tasks.rados.rados.2.smithi204.stdout:1654: done (3 left) 2024-09-16T05:17:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1657: done (2 left) 2024-09-16T05:17:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1658: delete oid 20 current snap is 180 2024-09-16T05:17:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1655: finishing write tid 3 to smithi20457400-50 2024-09-16T05:17:18.789 INFO:tasks.rados.rados.2.smithi204.stdout:1655: finishing write tid 4 to smithi20457400-50 2024-09-16T05:17:18.790 INFO:tasks.rados.rados.2.smithi204.stdout:1658: done (2 left) 2024-09-16T05:17:18.790 INFO:tasks.rados.rados.2.smithi204.stdout:1659: write (excl) oid 27 current snap is 180 2024-09-16T05:17:18.791 INFO:tasks.rados.rados.2.smithi204.stdout:1659: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-16T05:17:18.796 INFO:tasks.rados.rados.2.smithi204.stdout:1659: writing smithi20457400-27 from 400165 to 1048854 tid 1 2024-09-16T05:17:18.802 INFO:tasks.rados.rados.2.smithi204.stdout:1659: writing smithi20457400-27 from 1684786 to 2000165 tid 2 2024-09-16T05:17:18.802 INFO:tasks.rados.rados.2.smithi204.stdout:1660: write oid 38 current snap is 180 2024-09-16T05:17:18.802 INFO:tasks.rados.rados.2.smithi204.stdout:1660: seq_num 596 ranges {416972=723938,1743268=273704} 2024-09-16T05:17:18.808 INFO:tasks.rados.rados.2.smithi204.stdout:1660: writing smithi20457400-38 from 416972 to 1140910 tid 1 2024-09-16T05:17:18.813 INFO:tasks.rados.rados.2.smithi204.stdout:1660: writing smithi20457400-38 from 1743268 to 2016972 tid 2 2024-09-16T05:17:18.813 INFO:tasks.rados.rados.2.smithi204.stdout:1661: read oid 13 snap 177 2024-09-16T05:17:18.813 INFO:tasks.rados.rados.2.smithi204.stdout:1661: expect (ObjNum 536 snap 160 seq_num 536) 2024-09-16T05:17:18.813 INFO:tasks.rados.rados.2.smithi204.stdout:1662: copy_from oid 34 from oid 29 current snap is 180 2024-09-16T05:17:18.813 INFO:tasks.rados.rados.2.smithi204.stdout:1656: expect (ObjNum 546 snap 163 seq_num 546) 2024-09-16T05:17:18.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:18.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:18.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:18.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:18.978 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:18.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:18.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:17:19.046 INFO:tasks.rados.rados.2.smithi204.stdout:1659: finishing write tid 1 to smithi20457400-27 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1659: finishing write tid 2 to smithi20457400-27 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1659: finishing write tid 3 to smithi20457400-27 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1659: finishing write tid 4 to smithi20457400-27 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 309 (ObjNum 595 snap 180 seq_num 595) dirty exists 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1659: left oid 27 (ObjNum 595 snap 180 seq_num 595) 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1660: finishing write tid 1 to smithi20457400-38 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1660: finishing write tid 2 to smithi20457400-38 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1660: finishing write tid 3 to smithi20457400-38 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1660: finishing write tid 4 to smithi20457400-38 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 686 (ObjNum 596 snap 180 seq_num 596) dirty exists 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1660: left oid 38 (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:19.047 INFO:tasks.rados.rados.2.smithi204.stdout:1661: expect (ObjNum 536 snap 160 seq_num 536) 2024-09-16T05:17:19.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:19.102 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:19.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:19.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:19.212 INFO:tasks.rados.rados.2.smithi204.stdout:1662: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:17:19.212 INFO:tasks.rados.rados.2.smithi204.stdout:1662: finishing copy_from to smithi20457400-34 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 688 (ObjNum 573 snap 170 seq_num 573) dirty exists 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1656: done (5 left) 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1659: done (4 left) 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1660: done (3 left) 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1661: done (2 left) 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1662: done (1 left) 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1663: copy_from oid 35 from oid 26 current snap is 180 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1664: rollback oid 28 current snap is 180 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 28 to 173 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1665: rollback oid 11 current snap is 180 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 11 to 178 2024-09-16T05:17:19.213 INFO:tasks.rados.rados.2.smithi204.stdout:1666: snap_remove snap 177 2024-09-16T05:17:19.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:19.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:19.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:19.355 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:19.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:19.457 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:19.479 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:19.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:19.581 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:19.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:19.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 489, 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:17:18.735397+0000", 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:14:38.892129+0000", 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:13:30.730953+0000", 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:17:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5804032, 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 33, 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:17:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "luminous", 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:17:19.653 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:17:19.654 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 21, 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 21, 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 21, 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 21, 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 21, 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:17:19.655 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "408", 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "401", 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:17:19.656 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:17:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:17:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:17:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:17:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:17:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:17:19.657 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:17:19.658 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:17:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.2899999618530273, 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.2899999618530273, 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:17:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.2899999618530273, 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.2899999618530273, 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:17:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 86, 2024-09-16T05:17:19.662 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 86, 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 86, 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 86, 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 86, 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "489", 2024-09-16T05:17:19.663 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "435", 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 181, 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 489, 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:17:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:17:19.665 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:17:19.666 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:17:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:17:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:17:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:17:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:17:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:17:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:17:19.667 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 1.6699999570846558, 2024-09-16T05:17:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 1.6699999570846558, 2024-09-16T05:17:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:17:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 1.6699999570846558, 2024-09-16T05:17:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 1.6699999570846558, 2024-09-16T05:17:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:17:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:17:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:17:19.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:17:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:17:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.669 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:17:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:17:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.670 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.671 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.671 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:17:19.671 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 439, 2024-09-16T05:17:19.671 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.671 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:17:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:17:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:17:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:17:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:17:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:17:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:17:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:17:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:17:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:17:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.679 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:17:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:17:19.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:17:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:17:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.681 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:17:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:17:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:17:19.682 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:17:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:17:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:17:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.684 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:17:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 488, 2024-09-16T05:17:19.685 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.686 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:17:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:17:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.687 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:17:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:17:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.688 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:17:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:17:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:17:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:17:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.691 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:17:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:17:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:17:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:17:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:17:19.695 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:17:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:17:19.696 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:17:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:17:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:17:19.697 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:17:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:17:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:17:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:17:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 21, 2024-09-16T05:17:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 295, 2024-09-16T05:17:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 329, 2024-09-16T05:17:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 488, 2024-09-16T05:17:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 296, 2024-09-16T05:17:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:17:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:17:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:17:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:17:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:17:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:17:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:17:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:17:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.706 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:17:19.707 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:17:19.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:17:19.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:17:19.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.708 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.709 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.710 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:17:19.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:17:19.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.711 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:17:19.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:17:19.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.711 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.712 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3973436581", 2024-09-16T05:17:19.712 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3973436581", 2024-09-16T05:17:19.712 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3973436581", 2024-09-16T05:17:19.713 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3973436581", 2024-09-16T05:17:19.713 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.713 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.713 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.713 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:17:19.714 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:17:19.714 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.714 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.715 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.715 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.715 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.715 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.715 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:17:19.715 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 437, 2024-09-16T05:17:19.716 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.716 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.716 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.716 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.717 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:17:19.717 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:17:19.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.717 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.718 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:17:19.718 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:17:19.718 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.718 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.718 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.718 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.720 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.722 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 439, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.723 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.724 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.726 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:17:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.728 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 488, 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.733 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 439, 2024-09-16T05:17:19.734 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.735 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.736 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.737 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.737 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.737 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:17:19.737 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:17:19.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:17:19.742 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:17:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:17:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 406, 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:17:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.747 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.749 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:17:19.750 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:17:19.751 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 439, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.754 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:17:19.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 435, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.758 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 488, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.764 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.765 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:17:19.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:17:19.770 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 435, 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:17:19.771 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.772 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.773 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.774 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:17:19.775 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:16:38.767467+0000", 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:16:28.728923+0000", 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.777 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:09.106006+0000", 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 295 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.778 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:06.186314+0000", 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:16:43.915685+0000", 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:16:36.770700+0000", 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.780 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:16:22.737223+0000", 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.781 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:14.574281+0000", 2024-09-16T05:17:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:17:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:16:21.027838+0000", 2024-09-16T05:17:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:17:19.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:12.480321+0000", 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:16:32.636104+0000", 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:17:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:16:17.783093+0000", 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [ 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: "pgid": "2.5", 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: 9, 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: 11 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:17:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [ 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pgid": "2.4b", 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: 1, 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: 7 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:17:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:17:19.789 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 174, 2024-09-16T05:17:19.790 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 176, 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 3 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 181, 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [ 2024-09-16T05:17:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 181, 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:17:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:17:19.793 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:17:19.793 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:17:19.793 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:17:19.793 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:17:19.793 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:17:19.793 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:19.793 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:17:19.795 INFO:tasks.rados.rados.2.smithi204.stdout:1666: done (4 left) 2024-09-16T05:17:19.795 INFO:tasks.rados.rados.2.smithi204.stdout:1667: write oid 16 current snap is 180 2024-09-16T05:17:19.795 INFO:tasks.rados.rados.2.smithi204.stdout:1664: finishing rollback tid 1 to smithi20457400-28 2024-09-16T05:17:19.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 5377 snap 0 seq_num 2944377632) dirty dne 2024-09-16T05:17:19.795 INFO:tasks.rados.rados.2.smithi204.stdout:1665: finishing rollback tid 0 to smithi20457400-11 2024-09-16T05:17:19.795 INFO:tasks.rados.rados.2.smithi204.stdout:1665: finishing rollback tid 1 to smithi20457400-11 2024-09-16T05:17:19.795 INFO:tasks.rados.rados.2.smithi204.stdout:1667: seq_num 597 ranges {433779=799187,1801750=232029} 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1667: writing smithi20457400-16 from 433779 to 1232966 tid 1 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1667: writing smithi20457400-16 from 1801750 to 2033779 tid 2 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1664: done (4 left) 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1668: snap_create 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1665: finishing rollback tid 2 to smithi20457400-11 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 700 (ObjNum 306 snap 85 seq_num 306) dirty exists 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1663: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1663: finishing copy_from to smithi20457400-35 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 827 (ObjNum 590 snap 179 seq_num 590) dirty exists 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1655: finishing write tid 5 to smithi20457400-50 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 364 (ObjNum 592 snap 180 seq_num 592) dirty exists 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1655: left oid 50 (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:17:19.796 INFO:tasks.rados.rados.2.smithi204.stdout:1667: finishing write tid 1 to smithi20457400-16 2024-09-16T05:17:19.797 INFO:tasks.rados.rados.2.smithi204.stdout:1667: finishing write tid 2 to smithi20457400-16 2024-09-16T05:17:19.797 INFO:tasks.rados.rados.2.smithi204.stdout:1667: finishing write tid 3 to smithi20457400-16 2024-09-16T05:17:19.797 INFO:tasks.rados.rados.2.smithi204.stdout:1667: finishing write tid 4 to smithi20457400-16 2024-09-16T05:17:19.797 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 689 (ObjNum 597 snap 180 seq_num 597) dirty exists 2024-09-16T05:17:19.797 INFO:tasks.rados.rados.2.smithi204.stdout:1667: left oid 16 (ObjNum 597 snap 180 seq_num 597) 2024-09-16T05:17:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:19.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:19.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:19 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:17:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:19 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:17:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:19 smithi032 ceph-mon[27388]: pgmap v894: 107 pgs: 1 unknown, 106 active+clean; 139 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 407 KiB/s wr, 9 op/s 2024-09-16T05:17:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:19 smithi032 ceph-mon[27388]: osdmap e489: 12 total, 12 up, 12 in 2024-09-16T05:17:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/730180815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:17:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:19.939 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:19.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:20.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:20.047 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:19 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:17:20.047 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:19 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:17:20.047 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:19 smithi204 ceph-mon[30933]: pgmap v894: 107 pgs: 1 unknown, 106 active+clean; 139 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 407 KiB/s wr, 9 op/s 2024-09-16T05:17:20.047 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:19 smithi204 ceph-mon[30933]: osdmap e489: 12 total, 12 up, 12 in 2024-09-16T05:17:20.047 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/730180815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:17:20.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:20.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:20.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:20.190 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:20.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:19 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:17:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:19 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:17:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:19 smithi196 ceph-mon[30802]: pgmap v894: 107 pgs: 1 unknown, 106 active+clean; 139 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 407 KiB/s wr, 9 op/s 2024-09-16T05:17:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:19 smithi196 ceph-mon[30802]: osdmap e489: 12 total, 12 up, 12 in 2024-09-16T05:17:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/730180815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:17:20.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:20.311 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:20.311 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:20.311 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:20.311 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:20.312 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:20.312 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:20.312 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:20.312 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:20.313 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:20.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:20.321 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:20.343 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:20.367 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:20.367 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:20.389 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:20.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:20.415 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:20.415 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:20.436 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:20.437 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:20.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:20.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:20.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:17:20.458 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:20.459 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:20.480 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:20.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:20.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:20.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:20.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:20.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:20.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:20.737 INFO:tasks.rados.rados.2.smithi204.stdout:1655: done (4 left) 2024-09-16T05:17:20.738 INFO:tasks.rados.rados.2.smithi204.stdout:1663: done (3 left) 2024-09-16T05:17:20.738 INFO:tasks.rados.rados.2.smithi204.stdout:1665: done (2 left) 2024-09-16T05:17:20.738 INFO:tasks.rados.rados.2.smithi204.stdout:1667: done (1 left) 2024-09-16T05:17:20.738 INFO:tasks.rados.rados.2.smithi204.stdout:1668: done (0 left) 2024-09-16T05:17:20.738 INFO:tasks.rados.rados.2.smithi204.stdout:1669: rollback oid 4 current snap is 181 2024-09-16T05:17:20.738 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 180 2024-09-16T05:17:20.738 INFO:tasks.rados.rados.2.smithi204.stdout:1670: write oid 9 current snap is 181 2024-09-16T05:17:20.738 INFO:tasks.rados.rados.2.smithi204.stdout:1670: seq_num 598 ranges {450586=474436,1343879=588206} 2024-09-16T05:17:20.742 INFO:tasks.rados.rados.2.smithi204.stdout:1670: writing smithi20457400-9 from 450586 to 925022 tid 1 2024-09-16T05:17:20.748 INFO:tasks.rados.rados.2.smithi204.stdout:1670: writing smithi20457400-9 from 1343879 to 1932085 tid 2 2024-09-16T05:17:20.748 INFO:tasks.rados.rados.2.smithi204.stdout:1671: copy_from oid 37 from oid 38 current snap is 181 2024-09-16T05:17:20.749 INFO:tasks.rados.rados.2.smithi204.stdout:1669: finishing rollback tid 0 to smithi20457400-4 2024-09-16T05:17:20.749 INFO:tasks.rados.rados.2.smithi204.stdout:1669: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:17:20.749 INFO:tasks.rados.rados.2.smithi204.stdout:1669: finishing rollback tid 2 to smithi20457400-4 2024-09-16T05:17:20.749 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 561 (ObjNum 529 snap 159 seq_num 529) dirty exists 2024-09-16T05:17:20.749 INFO:tasks.rados.rados.2.smithi204.stdout:1670: finishing write tid 1 to smithi20457400-9 2024-09-16T05:17:20.749 INFO:tasks.rados.rados.2.smithi204.stdout:1669: done (2 left) 2024-09-16T05:17:20.749 INFO:tasks.rados.rados.2.smithi204.stdout:1672: snap_remove snap 178 2024-09-16T05:17:20.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:20.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:20.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:20.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:20.925 INFO:tasks.thrashosds.thrasher:j is {'epoch': 489, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:17:18.735397+0000', 'last_up_change': '2024-09-16T05:14:38.892129+0000', 'last_in_change': '2024-09-16T05:13:30.730953+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 33, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '408', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '401', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 1, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 86, 'pg_placement_num': 86, 'pg_placement_num_target': 86, 'pg_num_target': 86, 'pg_num_pending': 86, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '489', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '435', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 181, 'snap_epoch': 489, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6699999570846558, 'score_stable': 1.6699999570846558, 'optimal_score': 1, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 439, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 488, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 21, 'last_clean_end': 295, 'up_from': 329, 'up_thru': 488, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3973436581}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3973436581}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3973436581}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3973436581}]}, 'public_addr': '172.21.15.32:6819/3973436581', 'cluster_addr': '172.21.15.32:6821/3973436581', 'heartbeat_back_addr': '172.21.15.32:6825/3973436581', 'heartbeat_front_addr': '172.21.15.32:6823/3973436581', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 437, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 439, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 488, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 439, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 406, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 439, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 435, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 488, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 435, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:16:38.767467+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:16:28.728923+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:09.106006+0000', 'dead_epoch': 295}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:06.186314+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:16:43.915685+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:16:36.770700+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:16:22.737223+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:14.574281+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:16:21.027838+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:12.480321+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:16:32.636104+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:16:17.783093+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [9, 11]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.4b', 'osds': [1, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 174, 'length': 1}, {'begin': 176, 'length': 3}, {'begin': 181, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 181, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:17:20.926 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:17:20.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:20.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:21.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:21.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:21.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:17:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:17:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:17:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:17:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: osdmap e490: 12 total, 12 up, 12 in 2024-09-16T05:17:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:17:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:17:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2702919939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:20 smithi032 ceph-mon[27388]: osdmap e491: 12 total, 12 up, 12 in 2024-09-16T05:17:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:17:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:17:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:17:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:17:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: osdmap e490: 12 total, 12 up, 12 in 2024-09-16T05:17:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:17:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:17:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2702919939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:20 smithi204 ceph-mon[30933]: osdmap e491: 12 total, 12 up, 12 in 2024-09-16T05:17:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:21.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:21.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: osdmap e490: 12 total, 12 up, 12 in 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2702919939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:20 smithi196 ceph-mon[30802]: osdmap e491: 12 total, 12 up, 12 in 2024-09-16T05:17:21.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:21.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:21.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:21.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:21.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:21.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:21.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:21.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:21.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:21.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:21.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:21.864 INFO:tasks.rados.rados.2.smithi204.stdout:1672: done (2 left) 2024-09-16T05:17:21.864 INFO:tasks.rados.rados.2.smithi204.stdout:1673: snap_remove snap 180 2024-09-16T05:17:21.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:21.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:21.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:22.013 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:22.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:22.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:21 smithi032 ceph-mon[27388]: pgmap v897: 107 pgs: 1 unknown, 106 active+clean; 144 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-16T05:17:22.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:21 smithi032 ceph-mon[27388]: osdmap e492: 12 total, 12 up, 12 in 2024-09-16T05:17:22.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:22.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:22.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:21 smithi204 ceph-mon[30933]: pgmap v897: 107 pgs: 1 unknown, 106 active+clean; 144 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-16T05:17:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:21 smithi204 ceph-mon[30933]: osdmap e492: 12 total, 12 up, 12 in 2024-09-16T05:17:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:21 smithi196 ceph-mon[30802]: pgmap v897: 107 pgs: 1 unknown, 106 active+clean; 144 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-16T05:17:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:21 smithi196 ceph-mon[30802]: osdmap e492: 12 total, 12 up, 12 in 2024-09-16T05:17:22.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:22.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:22.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:22.333 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:22.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:22.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:22.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:22.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:22.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:22.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:22.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:22.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:1673: done (2 left) 2024-09-16T05:17:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:1674: read oid 29 snap 173 2024-09-16T05:17:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:1674: expect (ObjNum 573 snap 170 seq_num 573) 2024-09-16T05:17:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:1675: write (excl) oid 43 current snap is 181 2024-09-16T05:17:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:1670: finishing write tid 2 to smithi20457400-9 2024-09-16T05:17:22.741 INFO:tasks.rados.rados.2.smithi204.stdout:1675: seq_num 599 ranges {467393=549685,1802361=265032} 2024-09-16T05:17:22.746 INFO:tasks.rados.rados.2.smithi204.stdout:1675: writing smithi20457400-43 from 467393 to 1017078 tid 1 2024-09-16T05:17:22.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:22.753 INFO:tasks.rados.rados.2.smithi204.stdout:1675: writing smithi20457400-43 from 1802361 to 2067393 tid 2 2024-09-16T05:17:22.753 INFO:tasks.rados.rados.2.smithi204.stdout:1676: delete oid 14 current snap is 181 2024-09-16T05:17:22.753 INFO:tasks.rados.rados.2.smithi204.stdout:1670: finishing write tid 3 to smithi20457400-9 2024-09-16T05:17:22.753 INFO:tasks.rados.rados.2.smithi204.stdout:1670: finishing write tid 4 to smithi20457400-9 2024-09-16T05:17:22.753 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 693 (ObjNum 598 snap 181 seq_num 598) dirty exists 2024-09-16T05:17:22.753 INFO:tasks.rados.rados.2.smithi204.stdout:1670: left oid 9 (ObjNum 598 snap 181 seq_num 598) 2024-09-16T05:17:22.753 INFO:tasks.rados.rados.2.smithi204.stdout:1671: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:17:22.753 INFO:tasks.rados.rados.2.smithi204.stdout:1671: finishing copy_from to smithi20457400-37 2024-09-16T05:17:22.754 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 686 (ObjNum 596 snap 180 seq_num 596) dirty exists 2024-09-16T05:17:22.754 INFO:tasks.rados.rados.2.smithi204.stdout:1674: expect (ObjNum 573 snap 170 seq_num 573) 2024-09-16T05:17:22.777 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:22.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:22.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:22.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:22.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1675: finishing write tid 1 to smithi20457400-43 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1675: finishing write tid 2 to smithi20457400-43 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1675: finishing write tid 3 to smithi20457400-43 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1675: finishing write tid 4 to smithi20457400-43 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 566 (ObjNum 599 snap 181 seq_num 599) dirty exists 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1675: left oid 43 (ObjNum 599 snap 181 seq_num 599) 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1670: done (4 left) 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1671: done (3 left) 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1674: done (2 left) 2024-09-16T05:17:22.951 INFO:tasks.rados.rados.2.smithi204.stdout:1675: done (1 left) 2024-09-16T05:17:22.952 INFO:tasks.rados.rados.2.smithi204.stdout:1676: done (0 left) 2024-09-16T05:17:22.952 INFO:tasks.rados.rados.2.smithi204.stdout:1677: delete oid 34 current snap is 181 2024-09-16T05:17:22.952 INFO:tasks.rados.rados.2.smithi204.stdout:1677: done (0 left) 2024-09-16T05:17:22.952 INFO:tasks.rados.rados.2.smithi204.stdout:1678: read oid 18 snap 173 2024-09-16T05:17:22.952 INFO:tasks.rados.rados.2.smithi204.stdout:1678: expect (ObjNum 505 snap 153 seq_num 505) 2024-09-16T05:17:22.952 INFO:tasks.rados.rados.2.smithi204.stdout:1679: delete oid 7 current snap is 181 2024-09-16T05:17:22.966 INFO:tasks.rados.rados.2.smithi204.stdout:1678: expect (ObjNum 505 snap 153 seq_num 505) 2024-09-16T05:17:23.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:23.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:23.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:23.133 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:23.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:23.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:23.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:17:23.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:17:23.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:17:23.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:17:23.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:17:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:17:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:17:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:17:23.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:22 smithi032 ceph-mon[27388]: osdmap e493: 12 total, 12 up, 12 in 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:17:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:22 smithi204 ceph-mon[30933]: osdmap e493: 12 total, 12 up, 12 in 2024-09-16T05:17:23.211 INFO:tasks.rados.rados.2.smithi204.stdout:1678: done (1 left) 2024-09-16T05:17:23.211 INFO:tasks.rados.rados.2.smithi204.stdout:1679: done (0 left) 2024-09-16T05:17:23.211 INFO:tasks.rados.rados.2.smithi204.stdout:1680: write (excl) oid 3 current snap is 181 2024-09-16T05:17:23.211 INFO:tasks.rados.rados.2.smithi204.stdout:1680: seq_num 603 ranges {534621=567034,1636289=498332} 2024-09-16T05:17:23.216 INFO:tasks.rados.rados.2.smithi204.stdout:1680: writing smithi20457400-3 from 534621 to 1101655 tid 1 2024-09-16T05:17:23.222 INFO:tasks.rados.rados.2.smithi204.stdout:1680: writing smithi20457400-3 from 1636289 to 2134621 tid 2 2024-09-16T05:17:23.223 INFO:tasks.rados.rados.2.smithi204.stdout:1681: copy_from oid 15 from oid 38 current snap is 181 2024-09-16T05:17:23.223 INFO:tasks.rados.rados.2.smithi204.stdout:1680: finishing write tid 1 to smithi20457400-3 2024-09-16T05:17:23.223 INFO:tasks.rados.rados.2.smithi204.stdout:1682: write oid 41 current snap is 181 2024-09-16T05:17:23.223 INFO:tasks.rados.rados.2.smithi204.stdout:1682: seq_num 604 ranges {551428=642283,1694771=456657} 2024-09-16T05:17:23.229 INFO:tasks.rados.rados.2.smithi204.stdout:1682: writing smithi20457400-41 from 551428 to 1193711 tid 1 2024-09-16T05:17:23.235 INFO:tasks.rados.rados.2.smithi204.stdout:1682: writing smithi20457400-41 from 1694771 to 2151428 tid 2 2024-09-16T05:17:23.235 INFO:tasks.rados.rados.2.smithi204.stdout:1683: snap_remove snap 168 2024-09-16T05:17:23.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:17:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:22 smithi196 ceph-mon[30802]: osdmap e493: 12 total, 12 up, 12 in 2024-09-16T05:17:23.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:23.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:23.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:23.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:23.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:23.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:17:23.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:23.525 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:17:23.526 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:17:23.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:23.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:23.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:23.652 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:23.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:23.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:23.768 INFO:tasks.rados.rados.2.smithi204.stdout:1683: done (3 left) 2024-09-16T05:17:23.768 INFO:tasks.rados.rados.2.smithi204.stdout:1684: write oid 25 current snap is 181 2024-09-16T05:17:23.768 INFO:tasks.rados.rados.2.smithi204.stdout:1680: finishing write tid 2 to smithi20457400-3 2024-09-16T05:17:23.768 INFO:tasks.rados.rados.2.smithi204.stdout:1684: seq_num 605 ranges {568235=717532,1753253=414982} 2024-09-16T05:17:23.776 INFO:tasks.rados.rados.2.smithi204.stdout:1684: writing smithi20457400-25 from 568235 to 1285767 tid 1 2024-09-16T05:17:23.782 INFO:tasks.rados.rados.2.smithi204.stdout:1684: writing smithi20457400-25 from 1753253 to 2168235 tid 2 2024-09-16T05:17:23.782 INFO:tasks.rados.rados.2.smithi204.stdout:1685: write oid 12 current snap is 181 2024-09-16T05:17:23.782 INFO:tasks.rados.rados.2.smithi204.stdout:1680: finishing write tid 3 to smithi20457400-3 2024-09-16T05:17:23.782 INFO:tasks.rados.rados.2.smithi204.stdout:1680: finishing write tid 4 to smithi20457400-3 2024-09-16T05:17:23.782 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 489 (ObjNum 603 snap 181 seq_num 603) dirty exists 2024-09-16T05:17:23.782 INFO:tasks.rados.rados.2.smithi204.stdout:1680: left oid 3 (ObjNum 603 snap 181 seq_num 603) 2024-09-16T05:17:23.782 INFO:tasks.rados.rados.2.smithi204.stdout:1685: seq_num 606 ranges {585042=792781,2095382=89660} 2024-09-16T05:17:23.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:23.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:23.789 INFO:tasks.rados.rados.2.smithi204.stdout:1685: writing smithi20457400-12 from 585042 to 1377823 tid 1 2024-09-16T05:17:23.794 INFO:tasks.rados.rados.2.smithi204.stdout:1685: writing smithi20457400-12 from 2095382 to 2185042 tid 2 2024-09-16T05:17:23.794 INFO:tasks.rados.rados.2.smithi204.stdout:1680: done (4 left) 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1686: write oid 17 current snap is 181 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1681: finishing copy_from racing read to smithi20457400-15 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1681: finishing copy_from to smithi20457400-15 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 686 (ObjNum 596 snap 180 seq_num 596) dirty exists 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1682: finishing write tid 1 to smithi20457400-41 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1682: finishing write tid 2 to smithi20457400-41 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1682: finishing write tid 3 to smithi20457400-41 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1682: finishing write tid 4 to smithi20457400-41 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 696 (ObjNum 604 snap 181 seq_num 604) dirty exists 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1682: left oid 41 (ObjNum 604 snap 181 seq_num 604) 2024-09-16T05:17:23.795 INFO:tasks.rados.rados.2.smithi204.stdout:1686: seq_num 607 ranges {601849=468030,1753864=447985} 2024-09-16T05:17:23.799 INFO:tasks.rados.rados.2.smithi204.stdout:1686: writing smithi20457400-17 from 601849 to 1069879 tid 1 2024-09-16T05:17:23.806 INFO:tasks.rados.rados.2.smithi204.stdout:1686: writing smithi20457400-17 from 1753864 to 2201849 tid 2 2024-09-16T05:17:23.806 INFO:tasks.rados.rados.2.smithi204.stdout:1681: done (4 left) 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1682: done (3 left) 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1687: rollback oid 37 current snap is 181 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 173 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1684: finishing write tid 1 to smithi20457400-25 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1688: rollback oid 24 current snap is 181 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 173 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1684: finishing write tid 2 to smithi20457400-25 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1684: finishing write tid 3 to smithi20457400-25 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1684: finishing write tid 4 to smithi20457400-25 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 484 (ObjNum 605 snap 181 seq_num 605) dirty exists 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1684: left oid 25 (ObjNum 605 snap 181 seq_num 605) 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1685: finishing write tid 1 to smithi20457400-12 2024-09-16T05:17:23.807 INFO:tasks.rados.rados.2.smithi204.stdout:1684: done (4 left) 2024-09-16T05:17:23.808 INFO:tasks.rados.rados.2.smithi204.stdout:1689: delete oid 40 current snap is 181 2024-09-16T05:17:23.808 INFO:tasks.rados.rados.2.smithi204.stdout:1685: finishing write tid 2 to smithi20457400-12 2024-09-16T05:17:23.808 INFO:tasks.rados.rados.2.smithi204.stdout:1685: finishing write tid 3 to smithi20457400-12 2024-09-16T05:17:23.808 INFO:tasks.rados.rados.2.smithi204.stdout:1685: finishing write tid 4 to smithi20457400-12 2024-09-16T05:17:23.808 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 694 (ObjNum 606 snap 181 seq_num 606) dirty exists 2024-09-16T05:17:23.808 INFO:tasks.rados.rados.2.smithi204.stdout:1685: left oid 12 (ObjNum 606 snap 181 seq_num 606) 2024-09-16T05:17:23.808 INFO:tasks.rados.rados.2.smithi204.stdout:1686: finishing write tid 1 to smithi20457400-17 2024-09-16T05:17:23.809 INFO:tasks.rados.rados.2.smithi204.stdout:1687: finishing rollback tid 0 to smithi20457400-37 2024-09-16T05:17:23.809 INFO:tasks.rados.rados.2.smithi204.stdout:1687: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1687: finishing rollback tid 2 to smithi20457400-37 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 689 (ObjNum 561 snap 168 seq_num 561) dirty exists 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1688: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1688: finishing rollback tid 2 to smithi20457400-24 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 511 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1686: finishing write tid 2 to smithi20457400-17 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1686: finishing write tid 3 to smithi20457400-17 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1685: done (4 left) 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1687: done (3 left) 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1688: done (2 left) 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1689: done (1 left) 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:1690: rollback oid 43 current snap is 181 2024-09-16T05:17:23.810 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 43 to 173 2024-09-16T05:17:23.811 INFO:tasks.rados.rados.2.smithi204.stdout:1691: write (excl) oid 6 current snap is 181 2024-09-16T05:17:23.811 INFO:tasks.rados.rados.2.smithi204.stdout:1691: seq_num 609 ranges {635463=734881,1987181=248282} 2024-09-16T05:17:23.816 INFO:tasks.rados.rados.2.smithi204.stdout:1691: writing smithi20457400-6 from 635463 to 1370344 tid 1 2024-09-16T05:17:23.821 INFO:tasks.rados.rados.2.smithi204.stdout:1691: writing smithi20457400-6 from 1987181 to 2235463 tid 2 2024-09-16T05:17:23.821 INFO:tasks.rados.rados.2.smithi204.stdout:1692: rollback oid 20 current snap is 181 2024-09-16T05:17:23.821 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 20 to 173 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:1686: finishing write tid 4 to smithi20457400-17 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 688 (ObjNum 607 snap 181 seq_num 607) dirty exists 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:1686: left oid 17 (ObjNum 607 snap 181 seq_num 607) 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:1690: finishing rollback tid 0 to smithi20457400-43 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:1690: finishing rollback tid 1 to smithi20457400-43 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 568 (ObjNum 129 snap 0 seq_num 4282153528) dirty dne 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:1691: finishing write tid 1 to smithi20457400-6 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:1686: done (3 left) 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:1690: done (2 left) 2024-09-16T05:17:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:1693: delete oid 37 current snap is 181 2024-09-16T05:17:23.823 INFO:tasks.rados.rados.2.smithi204.stdout:1693: done (2 left) 2024-09-16T05:17:23.823 INFO:tasks.rados.rados.2.smithi204.stdout:1694: write (excl) oid 47 current snap is 181 2024-09-16T05:17:23.823 INFO:tasks.rados.rados.2.smithi204.stdout:1694: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-16T05:17:23.828 INFO:tasks.rados.rados.2.smithi204.stdout:1694: writing smithi20457400-47 from 669077 to 1154456 tid 1 2024-09-16T05:17:23.835 INFO:tasks.rados.rados.2.smithi204.stdout:1694: writing smithi20457400-47 from 1587792 to 2269077 tid 2 2024-09-16T05:17:23.835 INFO:tasks.rados.rados.2.smithi204.stdout:1695: write (excl) oid 14 current snap is 181 2024-09-16T05:17:23.835 INFO:tasks.rados.rados.2.smithi204.stdout:1692: finishing rollback tid 1 to smithi20457400-20 2024-09-16T05:17:23.835 INFO:tasks.rados.rados.2.smithi204.stdout:1692: finishing rollback tid 2 to smithi20457400-20 2024-09-16T05:17:23.835 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 714 (ObjNum 480 snap 151 seq_num 480) dirty exists 2024-09-16T05:17:23.835 INFO:tasks.rados.rados.2.smithi204.stdout:1691: finishing write tid 2 to smithi20457400-6 2024-09-16T05:17:23.835 INFO:tasks.rados.rados.2.smithi204.stdout:1691: finishing write tid 3 to smithi20457400-6 2024-09-16T05:17:23.835 INFO:tasks.rados.rados.2.smithi204.stdout:1695: seq_num 612 ranges {685884=560628,2046274=239610} 2024-09-16T05:17:23.841 INFO:tasks.rados.rados.2.smithi204.stdout:1695: writing smithi20457400-14 from 685884 to 1246512 tid 1 2024-09-16T05:17:23.847 INFO:tasks.rados.rados.2.smithi204.stdout:1695: writing smithi20457400-14 from 2046274 to 2285884 tid 2 2024-09-16T05:17:23.847 INFO:tasks.rados.rados.2.smithi204.stdout:1692: done (3 left) 2024-09-16T05:17:23.847 INFO:tasks.rados.rados.2.smithi204.stdout:1696: snap_remove snap 173 2024-09-16T05:17:23.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:17:23.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:23.930 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:23.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:24.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:24.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:24.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:23 smithi032 ceph-mon[27388]: pgmap v900: 107 pgs: 1 active+remapped, 106 active+clean; 162 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4.8 MiB/s wr, 25 op/s 2024-09-16T05:17:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:23 smithi032 ceph-mon[27388]: 2.24 scrub starts 2024-09-16T05:17:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:23 smithi032 ceph-mon[27388]: 2.24 scrub ok 2024-09-16T05:17:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:23 smithi032 ceph-mon[27388]: 1.d scrub starts 2024-09-16T05:17:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:23 smithi032 ceph-mon[27388]: 1.d scrub ok 2024-09-16T05:17:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:23 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:17:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:23 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:17:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:23 smithi204 ceph-mon[30933]: pgmap v900: 107 pgs: 1 active+remapped, 106 active+clean; 162 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4.8 MiB/s wr, 25 op/s 2024-09-16T05:17:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:23 smithi204 ceph-mon[30933]: 2.24 scrub starts 2024-09-16T05:17:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:23 smithi204 ceph-mon[30933]: 2.24 scrub ok 2024-09-16T05:17:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:23 smithi204 ceph-mon[30933]: 1.d scrub starts 2024-09-16T05:17:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:23 smithi204 ceph-mon[30933]: 1.d scrub ok 2024-09-16T05:17:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:23 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:17:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:23 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:17:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:24.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:23 smithi196 ceph-mon[30802]: pgmap v900: 107 pgs: 1 active+remapped, 106 active+clean; 162 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4.8 MiB/s wr, 25 op/s 2024-09-16T05:17:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:23 smithi196 ceph-mon[30802]: 2.24 scrub starts 2024-09-16T05:17:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:23 smithi196 ceph-mon[30802]: 2.24 scrub ok 2024-09-16T05:17:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:23 smithi196 ceph-mon[30802]: 1.d scrub starts 2024-09-16T05:17:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:23 smithi196 ceph-mon[30802]: 1.d scrub ok 2024-09-16T05:17:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:23 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:17:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:23 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:17:24.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:24.289 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":901,"stamp":"2024-09-16T05:17:22.736612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":172062644,"num_objects":73,"num_object_clones":30,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":4,"num_read":1391,"num_read_kb":1420718,"num_write":984,"num_write_kb":421183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22352,"ondisk_log_size":22352,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":705760,"kb_used_data":284600,"kb_used_omap":27,"kb_used_meta":420836,"kb_avail":1124236064,"statfs":{"total":1151940427776,"available":1151217729536,"internally_reserved":0,"allocated":291430400,"data_stored":305262775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":430936231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26710660,"num_objects":12,"num_object_clones":10,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":74373,"num_write":44,"num_write_kb":20933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.999377"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":2388,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.974153+0000","last_change":"2024-09-16T05:17:19.974388+0000","last_active":"2024-09-16T05:17:19.974153+0000","last_peered":"2024-09-16T05:17:19.974153+0000","last_clean":"2024-09-16T05:17:19.974153+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:17:19.974153+0000","last_undegraded":"2024-09-16T05:17:19.974153+0000","last_fullsized":"2024-09-16T05:17:19.974153+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:16:38.263205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.263205+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:46.856050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002100116,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.54","version":"441'295","reported_seq":1811,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.742458+0000","last_change":"2024-09-16T05:17:19.742701+0000","last_active":"2024-09-16T05:17:19.742458+0000","last_peered":"2024-09-16T05:17:19.742458+0000","last_clean":"2024-09-16T05:17:19.742458+0000","last_became_active":"2024-09-16T05:16:30.728460+0000","last_became_peered":"2024-09-16T05:16:30.728460+0000","last_unstale":"2024-09-16T05:17:19.742458+0000","last_undegraded":"2024-09-16T05:17:19.742458+0000","last_fullsized":"2024-09-16T05:17:19.742458+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"441'295","last_scrub_stamp":"2024-09-16T05:16:37.936216+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.936216+0000","objects_scrubbed":1,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:49.927035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075488200000000001,"stat_sum":{"num_bytes":2285851,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15568,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":283,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-16T05:17:20.740249+0000","last_change":"2024-09-16T05:17:19.755459+0000","last_active":"2024-09-16T05:17:20.740249+0000","last_peered":"2024-09-16T05:17:20.740249+0000","last_clean":"2024-09-16T05:17:20.740249+0000","last_became_active":"2024-09-16T05:16:27.728289+0000","last_became_peered":"2024-09-16T05:16:27.728289+0000","last_unstale":"2024-09-16T05:17:20.740249+0000","last_undegraded":"2024-09-16T05:17:20.740249+0000","last_fullsized":"2024-09-16T05:17:20.740249+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:38.004600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.004600+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:48.179810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027702,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":278,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.739601+0000","last_change":"2024-09-16T05:17:19.740151+0000","last_active":"2024-09-16T05:17:19.739601+0000","last_peered":"2024-09-16T05:17:19.739601+0000","last_clean":"2024-09-16T05:17:19.739601+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:17:19.739601+0000","last_undegraded":"2024-09-16T05:17:19.739601+0000","last_fullsized":"2024-09-16T05:17:19.739601+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:16:38.584934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.584934+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.059315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078605299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":796,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.972350+0000","last_change":"2024-09-16T05:17:19.972566+0000","last_active":"2024-09-16T05:17:19.972350+0000","last_peered":"2024-09-16T05:17:19.972350+0000","last_clean":"2024-09-16T05:17:19.972350+0000","last_became_active":"2024-09-16T05:16:30.731943+0000","last_became_peered":"2024-09-16T05:16:30.731943+0000","last_unstale":"2024-09-16T05:17:19.972350+0000","last_undegraded":"2024-09-16T05:17:19.972350+0000","last_fullsized":"2024-09-16T05:17:19.972350+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:16:38.834396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.834396+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:52.729367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026352849999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":264,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.743137+0000","last_change":"2024-09-16T05:17:15.184249+0000","last_active":"2024-09-16T05:17:17.743137+0000","last_peered":"2024-09-16T05:17:17.743137+0000","last_clean":"2024-09-16T05:17:17.743137+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:17:17.743137+0000","last_undegraded":"2024-09-16T05:17:17.743137+0000","last_fullsized":"2024-09-16T05:17:17.743137+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:38.794214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.794214+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.199668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011554079999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2191,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.743269+0000","last_change":"2024-09-16T05:17:19.743459+0000","last_active":"2024-09-16T05:17:19.743269+0000","last_peered":"2024-09-16T05:17:19.743269+0000","last_clean":"2024-09-16T05:17:19.743269+0000","last_became_active":"2024-09-16T05:16:30.732139+0000","last_became_peered":"2024-09-16T05:16:30.732139+0000","last_unstale":"2024-09-16T05:17:19.743269+0000","last_undegraded":"2024-09-16T05:17:19.743269+0000","last_fullsized":"2024-09-16T05:17:19.743269+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:16:38.938962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.938962+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:50.343604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001259563,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.4e","version":"272'232","reported_seq":2046,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.974746+0000","last_change":"2024-09-16T05:17:19.975001+0000","last_active":"2024-09-16T05:17:19.974746+0000","last_peered":"2024-09-16T05:17:19.974746+0000","last_clean":"2024-09-16T05:17:19.974746+0000","last_became_active":"2024-09-16T05:16:30.732564+0000","last_became_peered":"2024-09-16T05:16:30.732564+0000","last_unstale":"2024-09-16T05:17:19.974746+0000","last_undegraded":"2024-09-16T05:17:19.974746+0000","last_fullsized":"2024-09-16T05:17:19.974746+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:16:25.855040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.855040+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:43.304492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026076099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.4d","version":"467'480","reported_seq":2375,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.741397+0000","last_change":"2024-09-16T05:17:15.184423+0000","last_active":"2024-09-16T05:17:17.741397+0000","last_peered":"2024-09-16T05:17:17.741397+0000","last_clean":"2024-09-16T05:17:17.741397+0000","last_became_active":"2024-09-16T05:16:30.731141+0000","last_became_peered":"2024-09-16T05:16:30.731141+0000","last_unstale":"2024-09-16T05:17:17.741397+0000","last_undegraded":"2024-09-16T05:17:17.741397+0000","last_fullsized":"2024-09-16T05:17:17.741397+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"441'465","last_scrub_stamp":"2024-09-16T05:16:37.930310+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.930310+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:01.345359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012608210000000001,"stat_sum":{"num_bytes":3361499,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":18522,"num_write":10,"num_write_kb":2573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.4c","version":"466'279","reported_seq":2162,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.971372+0000","last_change":"2024-09-16T05:17:19.971509+0000","last_active":"2024-09-16T05:17:19.971372+0000","last_peered":"2024-09-16T05:17:19.971372+0000","last_clean":"2024-09-16T05:17:19.971372+0000","last_became_active":"2024-09-16T05:16:30.731529+0000","last_became_peered":"2024-09-16T05:16:30.731529+0000","last_unstale":"2024-09-16T05:17:19.971372+0000","last_undegraded":"2024-09-16T05:17:19.971372+0000","last_fullsized":"2024-09-16T05:17:19.971372+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:16:39.809684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.809684+0000","objects_scrubbed":1,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.816771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012898219999999999,"stat_sum":{"num_bytes":3277464,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":16241,"num_write":5,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":2083,"reported_epoch":490,"state":"active+remapped","last_fresh":"2024-09-16T05:17:19.751363+0000","last_change":"2024-09-16T05:17:19.751237+0000","last_active":"2024-09-16T05:17:19.751363+0000","last_peered":"2024-09-16T05:17:19.751363+0000","last_clean":"2024-09-16T05:17:16.743445+0000","last_became_active":"2024-09-16T05:17:19.751237+0000","last_became_peered":"2024-09-16T05:17:19.751237+0000","last_unstale":"2024-09-16T05:17:19.751363+0000","last_undegraded":"2024-09-16T05:17:19.751363+0000","last_fullsized":"2024-09-16T05:17:19.751363+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:17:04.875229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:04.875229+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013917809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[1,7],"avail_no_missing":["1","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.4a","version":"490'494","reported_seq":2365,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.972093+0000","last_change":"2024-09-16T05:17:19.972300+0000","last_active":"2024-09-16T05:17:19.972093+0000","last_peered":"2024-09-16T05:17:19.972093+0000","last_clean":"2024-09-16T05:17:19.972093+0000","last_became_active":"2024-09-16T05:14:30.853656+0000","last_became_peered":"2024-09-16T05:14:30.853656+0000","last_unstale":"2024-09-16T05:17:19.972093+0000","last_undegraded":"2024-09-16T05:17:19.972093+0000","last_fullsized":"2024-09-16T05:17:19.972093+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"459'490","last_scrub_stamp":"2024-09-16T05:17:00.898492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.898492+0000","objects_scrubbed":1,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:14.144786+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0021547440000000001,"stat_sum":{"num_bytes":6756612,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":16,"num_write_kb":11594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":2118,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.974436+0000","last_change":"2024-09-16T05:17:19.974696+0000","last_active":"2024-09-16T05:17:19.974436+0000","last_peered":"2024-09-16T05:17:19.974436+0000","last_clean":"2024-09-16T05:17:19.974436+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:17:19.974436+0000","last_undegraded":"2024-09-16T05:17:19.974436+0000","last_fullsized":"2024-09-16T05:17:19.974436+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:16:00.136780+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.136780+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.778429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022585719999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.48","version":"451'226","reported_seq":2096,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.748210+0000","last_change":"2024-09-16T05:17:19.748461+0000","last_active":"2024-09-16T05:17:19.748210+0000","last_peered":"2024-09-16T05:17:19.748210+0000","last_clean":"2024-09-16T05:17:19.748210+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:17:19.748210+0000","last_undegraded":"2024-09-16T05:17:19.748210+0000","last_fullsized":"2024-09-16T05:17:19.748210+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"451'226","last_scrub_stamp":"2024-09-16T05:17:08.272173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:08.272173+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:27.675261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010936870000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20677,"num_write":10,"num_write_kb":8206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":904,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.740209+0000","last_change":"2024-09-16T05:17:19.740506+0000","last_active":"2024-09-16T05:17:19.740209+0000","last_peered":"2024-09-16T05:17:19.740209+0000","last_clean":"2024-09-16T05:17:19.740209+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:17:19.740209+0000","last_undegraded":"2024-09-16T05:17:19.740209+0000","last_fullsized":"2024-09-16T05:17:19.740209+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:17:00.838891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.838891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:13.537454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094673299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":1916,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.743007+0000","last_change":"2024-09-16T05:17:19.743215+0000","last_active":"2024-09-16T05:17:19.743007+0000","last_peered":"2024-09-16T05:17:19.743007+0000","last_clean":"2024-09-16T05:17:19.743007+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:17:19.743007+0000","last_undegraded":"2024-09-16T05:17:19.743007+0000","last_fullsized":"2024-09-16T05:17:19.743007+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:17:01.867464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.867464+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:09.822804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010991659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":2335,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.742548+0000","last_change":"2024-09-16T05:17:19.742778+0000","last_active":"2024-09-16T05:17:19.742548+0000","last_peered":"2024-09-16T05:17:19.742548+0000","last_clean":"2024-09-16T05:17:19.742548+0000","last_became_active":"2024-09-16T05:14:30.853144+0000","last_became_peered":"2024-09-16T05:14:30.853144+0000","last_unstale":"2024-09-16T05:17:19.742548+0000","last_undegraded":"2024-09-16T05:17:19.742548+0000","last_fullsized":"2024-09-16T05:17:19.742548+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:15:59.876915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.876915+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000813962,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":1998,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.743084+0000","last_change":"2024-09-16T05:17:19.743306+0000","last_active":"2024-09-16T05:17:19.743084+0000","last_peered":"2024-09-16T05:17:19.743084+0000","last_clean":"2024-09-16T05:17:19.743084+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:17:19.743084+0000","last_undegraded":"2024-09-16T05:17:19.743084+0000","last_fullsized":"2024-09-16T05:17:19.743084+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:16:44.877606+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:16:44.877606+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.598960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001151207,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":901,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.740432+0000","last_change":"2024-09-16T05:17:21.740757+0000","last_active":"2024-09-16T05:17:21.740432+0000","last_peered":"2024-09-16T05:17:21.740432+0000","last_clean":"2024-09-16T05:17:21.740432+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:17:21.740432+0000","last_undegraded":"2024-09-16T05:17:21.740432+0000","last_fullsized":"2024-09-16T05:17:21.740432+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:16:14.198462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.198462+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:22.813673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081447900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.42","version":"182'20","reported_seq":860,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-09-16T05:17:18.741691+0000","last_change":"2024-09-16T05:17:18.742015+0000","last_active":"2024-09-16T05:17:18.741691+0000","last_peered":"2024-09-16T05:17:18.741691+0000","last_clean":"2024-09-16T05:17:18.741691+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:17:18.741691+0000","last_undegraded":"2024-09-16T05:17:18.741691+0000","last_fullsized":"2024-09-16T05:17:18.741691+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:16:45.114951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:45.114951+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:06.974177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001312419,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"},{"start":"b5","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":979,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.970910+0000","last_change":"2024-09-16T05:17:19.971057+0000","last_active":"2024-09-16T05:17:19.970910+0000","last_peered":"2024-09-16T05:17:19.970910+0000","last_clean":"2024-09-16T05:17:19.970910+0000","last_became_active":"2024-09-16T05:13:45.686056+0000","last_became_peered":"2024-09-16T05:13:45.686056+0000","last_unstale":"2024-09-16T05:17:19.970910+0000","last_undegraded":"2024-09-16T05:17:19.970910+0000","last_fullsized":"2024-09-16T05:17:19.970910+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:59.922855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:59.922855+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:10.502271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000871714,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":340,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.974828+0000","last_change":"2024-09-16T05:17:19.975063+0000","last_active":"2024-09-16T05:17:19.974828+0000","last_peered":"2024-09-16T05:17:19.974828+0000","last_clean":"2024-09-16T05:17:19.974828+0000","last_became_active":"2024-09-16T05:16:17.733306+0000","last_became_peered":"2024-09-16T05:16:17.733306+0000","last_unstale":"2024-09-16T05:17:19.974828+0000","last_undegraded":"2024-09-16T05:17:19.974828+0000","last_fullsized":"2024-09-16T05:17:19.974828+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:16:18.066113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.066113+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:31.147918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023235650000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.3f","version":"472'312","reported_seq":2228,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.741499+0000","last_change":"2024-09-16T05:17:21.741737+0000","last_active":"2024-09-16T05:17:21.741499+0000","last_peered":"2024-09-16T05:17:21.741499+0000","last_clean":"2024-09-16T05:17:21.741499+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:17:21.741499+0000","last_undegraded":"2024-09-16T05:17:21.741499+0000","last_fullsized":"2024-09-16T05:17:21.741499+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"436'302","last_scrub_stamp":"2024-09-16T05:16:43.248069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:43.248069+0000","objects_scrubbed":1,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:49.053481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014697740000000001,"stat_sum":{"num_bytes":2551686,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":22718,"num_write":26,"num_write_kb":6027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.3e","version":"492'394","reported_seq":2268,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.741329+0000","last_change":"2024-09-16T05:17:21.741548+0000","last_active":"2024-09-16T05:17:21.741329+0000","last_peered":"2024-09-16T05:17:21.741329+0000","last_clean":"2024-09-16T05:17:21.741329+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:17:21.741329+0000","last_undegraded":"2024-09-16T05:17:21.741329+0000","last_fullsized":"2024-09-16T05:17:21.741329+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"431'368","last_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.868538+0000","objects_scrubbed":3,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:28.692498+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.002428347,"stat_sum":{"num_bytes":6101139,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":20,"num_write_kb":14694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2437,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.742759+0000","last_change":"2024-09-16T05:17:19.742939+0000","last_active":"2024-09-16T05:17:19.742759+0000","last_peered":"2024-09-16T05:17:19.742759+0000","last_clean":"2024-09-16T05:17:19.742759+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:17:19.742759+0000","last_undegraded":"2024-09-16T05:17:19.742759+0000","last_fullsized":"2024-09-16T05:17:19.742759+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:16:23.005301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.005301+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:41.205288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00088870900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.3c","version":"481'507","reported_seq":2517,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.741174+0000","last_change":"2024-09-16T05:17:21.741518+0000","last_active":"2024-09-16T05:17:21.741174+0000","last_peered":"2024-09-16T05:17:21.741174+0000","last_clean":"2024-09-16T05:17:21.741174+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:17:21.741174+0000","last_undegraded":"2024-09-16T05:17:21.741174+0000","last_fullsized":"2024-09-16T05:17:21.741174+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"436'490","last_scrub_stamp":"2024-09-16T05:16:26.159696+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.159696+0000","objects_scrubbed":1,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:34.223741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013038959999999999,"stat_sum":{"num_bytes":3109394,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":74,"num_read_kb":82099,"num_write":34,"num_write_kb":18036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.3b","version":"481'326","reported_seq":2228,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.975122+0000","last_change":"2024-09-16T05:17:19.975466+0000","last_active":"2024-09-16T05:17:19.975122+0000","last_peered":"2024-09-16T05:17:19.975122+0000","last_clean":"2024-09-16T05:17:19.975122+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:17:19.975122+0000","last_undegraded":"2024-09-16T05:17:19.975122+0000","last_fullsized":"2024-09-16T05:17:19.975122+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"431'315","last_scrub_stamp":"2024-09-16T05:16:21.147071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.147071+0000","objects_scrubbed":1,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:27.984454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023246030000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":46643,"num_write":11,"num_write_kb":2763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.3a","version":"490'524","reported_seq":2387,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.754965+0000","last_change":"2024-09-16T05:17:19.755289+0000","last_active":"2024-09-16T05:17:19.754965+0000","last_peered":"2024-09-16T05:17:19.754965+0000","last_clean":"2024-09-16T05:17:19.754965+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:17:19.754965+0000","last_undegraded":"2024-09-16T05:17:19.754965+0000","last_fullsized":"2024-09-16T05:17:19.754965+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"454'520","last_scrub_stamp":"2024-09-16T05:17:01.101516+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.101516+0000","objects_scrubbed":1,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:13.915261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0077775750000000001,"stat_sum":{"num_bytes":5630543,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":24517,"num_write":38,"num_write_kb":19200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":984,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.747782+0000","last_change":"2024-09-16T05:17:19.748164+0000","last_active":"2024-09-16T05:17:19.747782+0000","last_peered":"2024-09-16T05:17:19.747782+0000","last_clean":"2024-09-16T05:17:19.747782+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:17:19.747782+0000","last_undegraded":"2024-09-16T05:17:19.747782+0000","last_fullsized":"2024-09-16T05:17:19.747782+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:17:02.077685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:02.077685+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:04.949473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097018500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":973,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-09-16T05:17:18.740583+0000","last_change":"2024-09-16T05:17:18.740997+0000","last_active":"2024-09-16T05:17:18.740583+0000","last_peered":"2024-09-16T05:17:18.740583+0000","last_clean":"2024-09-16T05:17:18.740583+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:17:18.740583+0000","last_undegraded":"2024-09-16T05:17:18.740583+0000","last_fullsized":"2024-09-16T05:17:18.740583+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:16:17.338199+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.338199+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:27.145080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057389500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"},{"start":"b5","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":1067,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.971557+0000","last_change":"2024-09-16T05:17:19.971847+0000","last_active":"2024-09-16T05:17:19.971557+0000","last_peered":"2024-09-16T05:17:19.971557+0000","last_clean":"2024-09-16T05:17:19.971557+0000","last_became_active":"2024-09-16T05:13:31.746380+0000","last_became_peered":"2024-09-16T05:13:31.746380+0000","last_unstale":"2024-09-16T05:17:19.971557+0000","last_undegraded":"2024-09-16T05:17:19.971557+0000","last_fullsized":"2024-09-16T05:17:19.971557+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:16:21.830665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.830665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:51.279020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001591271,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":1065,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.743357+0000","last_change":"2024-09-16T05:17:19.743552+0000","last_active":"2024-09-16T05:17:19.743357+0000","last_peered":"2024-09-16T05:17:19.743357+0000","last_clean":"2024-09-16T05:17:19.743357+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:17:19.743357+0000","last_undegraded":"2024-09-16T05:17:19.743357+0000","last_fullsized":"2024-09-16T05:17:19.743357+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:16:02.904155+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:21.169633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00125354,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.35","version":"490'553","reported_seq":2480,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.759249+0000","last_change":"2024-09-16T05:17:19.743672+0000","last_active":"2024-09-16T05:17:19.759249+0000","last_peered":"2024-09-16T05:17:19.759249+0000","last_clean":"2024-09-16T05:17:19.759249+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:17:19.759249+0000","last_undegraded":"2024-09-16T05:17:19.759249+0000","last_fullsized":"2024-09-16T05:17:19.759249+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"410'508","last_scrub_stamp":"2024-09-16T05:16:00.864640+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:16:00.864640+0000","objects_scrubbed":4,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011072810000000001,"stat_sum":{"num_bytes":7462671,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":43634,"num_write":45,"num_write_kb":17207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.34","version":"490'366","reported_seq":1878,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.973851+0000","last_change":"2024-09-16T05:17:19.974088+0000","last_active":"2024-09-16T05:17:19.973851+0000","last_peered":"2024-09-16T05:17:19.973851+0000","last_clean":"2024-09-16T05:17:19.973851+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:17:19.973851+0000","last_undegraded":"2024-09-16T05:17:19.973851+0000","last_fullsized":"2024-09-16T05:17:19.973851+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"420'335","last_scrub_stamp":"2024-09-16T05:16:15.093220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:15.093220+0000","objects_scrubbed":4,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:31.466601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001811455,"stat_sum":{"num_bytes":8562380,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":85172,"num_write":36,"num_write_kb":17221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":1079,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.741552+0000","last_change":"2024-09-16T05:17:21.741791+0000","last_active":"2024-09-16T05:17:21.741552+0000","last_peered":"2024-09-16T05:17:21.741552+0000","last_clean":"2024-09-16T05:17:21.741552+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:17:21.741552+0000","last_undegraded":"2024-09-16T05:17:21.741552+0000","last_fullsized":"2024-09-16T05:17:21.741552+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:25.199845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.199845+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:47.992070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001499165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.32","version":"451'141","reported_seq":1181,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.739208+0000","last_change":"2024-09-16T05:17:21.739510+0000","last_active":"2024-09-16T05:17:21.739208+0000","last_peered":"2024-09-16T05:17:21.739208+0000","last_clean":"2024-09-16T05:17:21.739208+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:17:21.739208+0000","last_undegraded":"2024-09-16T05:17:21.739208+0000","last_fullsized":"2024-09-16T05:17:21.739208+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"419'127","last_scrub_stamp":"2024-09-16T05:16:18.859832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.859832+0000","objects_scrubbed":1,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:42.099297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078363100000000002,"stat_sum":{"num_bytes":3109361,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":35253,"num_write":19,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.31","version":"459'204","reported_seq":1751,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.739537+0000","last_change":"2024-09-16T05:17:19.740152+0000","last_active":"2024-09-16T05:17:19.739537+0000","last_peered":"2024-09-16T05:17:19.739537+0000","last_clean":"2024-09-16T05:17:19.739537+0000","last_became_active":"2024-09-16T05:13:31.748831+0000","last_became_peered":"2024-09-16T05:13:31.748831+0000","last_unstale":"2024-09-16T05:17:19.739537+0000","last_undegraded":"2024-09-16T05:17:19.739537+0000","last_fullsized":"2024-09-16T05:17:19.739537+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"459'204","last_scrub_stamp":"2024-09-16T05:17:03.863715+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.863715+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:03.906803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079297299999999999,"stat_sum":{"num_bytes":3176622,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":79081,"num_write":27,"num_write_kb":10906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":1031,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.970221+0000","last_change":"2024-09-16T05:17:19.970538+0000","last_active":"2024-09-16T05:17:19.970221+0000","last_peered":"2024-09-16T05:17:19.970221+0000","last_clean":"2024-09-16T05:17:19.970221+0000","last_became_active":"2024-09-16T05:13:31.747805+0000","last_became_peered":"2024-09-16T05:13:31.747805+0000","last_unstale":"2024-09-16T05:17:19.970221+0000","last_undegraded":"2024-09-16T05:17:19.970221+0000","last_fullsized":"2024-09-16T05:17:19.970221+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:24.905903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:24.905903+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:33.559045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053569100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":589,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.972090+0000","last_change":"2024-09-16T05:17:18.778720+0000","last_active":"2024-09-16T05:17:19.972090+0000","last_peered":"2024-09-16T05:17:19.972090+0000","last_clean":"2024-09-16T05:17:19.972090+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:17:19.972090+0000","last_undegraded":"2024-09-16T05:17:19.972090+0000","last_fullsized":"2024-09-16T05:17:19.972090+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.054740+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.054740+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1626,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:22.166195+0000","last_change":"2024-09-16T05:17:22.166423+0000","last_active":"2024-09-16T05:17:22.166195+0000","last_peered":"2024-09-16T05:17:22.166195+0000","last_clean":"2024-09-16T05:17:22.166195+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:17:22.166195+0000","last_undegraded":"2024-09-16T05:17:22.166195+0000","last_fullsized":"2024-09-16T05:17:22.166195+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:16:26.623221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.623221+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:34.855753+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0.00041574500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"1.13","version":"35'139","reported_seq":1217,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.741738+0000","last_change":"2024-09-16T05:15:56.852148+0000","last_active":"2024-09-16T05:17:19.741738+0000","last_peered":"2024-09-16T05:17:19.741738+0000","last_clean":"2024-09-16T05:17:19.741738+0000","last_became_active":"2024-09-16T05:15:56.714681+0000","last_became_peered":"2024-09-16T05:15:56.714681+0000","last_unstale":"2024-09-16T05:17:19.741738+0000","last_undegraded":"2024-09-16T05:17:19.741738+0000","last_fullsized":"2024-09-16T05:17:19.741738+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.852021+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.852021+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":1523,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-09-16T05:17:18.741013+0000","last_change":"2024-09-16T05:17:18.741287+0000","last_active":"2024-09-16T05:17:18.741013+0000","last_peered":"2024-09-16T05:17:18.741013+0000","last_clean":"2024-09-16T05:17:18.741013+0000","last_became_active":"2024-09-16T05:16:25.731838+0000","last_became_peered":"2024-09-16T05:16:25.731838+0000","last_unstale":"2024-09-16T05:17:18.741013+0000","last_undegraded":"2024-09-16T05:17:18.741013+0000","last_fullsized":"2024-09-16T05:17:18.741013+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:16:26.399340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.399340+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:52.848588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084360700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"},{"start":"b5","length":"1"}]},{"pgid":"1.c","version":"35'139","reported_seq":533,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.743339+0000","last_change":"2024-09-16T05:15:56.828705+0000","last_active":"2024-09-16T05:17:17.743339+0000","last_peered":"2024-09-16T05:17:17.743339+0000","last_clean":"2024-09-16T05:17:17.743339+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:17:17.743339+0000","last_undegraded":"2024-09-16T05:17:17.743339+0000","last_fullsized":"2024-09-16T05:17:17.743339+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.828595+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.828595+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:25.780000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"479'287","reported_seq":2194,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-09-16T05:17:18.741093+0000","last_change":"2024-09-16T05:17:18.741305+0000","last_active":"2024-09-16T05:17:18.741093+0000","last_peered":"2024-09-16T05:17:18.741093+0000","last_clean":"2024-09-16T05:17:18.741093+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:17:18.741093+0000","last_undegraded":"2024-09-16T05:17:18.741093+0000","last_fullsized":"2024-09-16T05:17:18.741093+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"437'282","last_scrub_stamp":"2024-09-16T05:16:38.132713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.132713+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:51.646950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086524699999999998,"stat_sum":{"num_bytes":3815288,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"},{"start":"b5","length":"1"}]},{"pgid":"1.d","version":"35'139","reported_seq":707,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.740818+0000","last_change":"2024-09-16T05:15:57.170014+0000","last_active":"2024-09-16T05:17:21.740818+0000","last_peered":"2024-09-16T05:17:21.740818+0000","last_clean":"2024-09-16T05:17:21.740818+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:17:21.740818+0000","last_undegraded":"2024-09-16T05:17:21.740818+0000","last_fullsized":"2024-09-16T05:17:21.740818+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:57.169945+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.169945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:22.397538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":2033,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.741692+0000","last_change":"2024-09-16T05:17:15.183610+0000","last_active":"2024-09-16T05:17:17.741692+0000","last_peered":"2024-09-16T05:17:17.741692+0000","last_clean":"2024-09-16T05:17:17.741692+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:17:17.741692+0000","last_undegraded":"2024-09-16T05:17:17.741692+0000","last_fullsized":"2024-09-16T05:17:17.741692+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:16:38.980110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:38.980110+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:02.537508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059088200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"1.e","version":"35'139","reported_seq":703,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-16T05:17:20.740096+0000","last_change":"2024-09-16T05:17:01.066059+0000","last_active":"2024-09-16T05:17:20.740096+0000","last_peered":"2024-09-16T05:17:20.740096+0000","last_clean":"2024-09-16T05:17:20.740096+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:17:20.740096+0000","last_undegraded":"2024-09-16T05:17:20.740096+0000","last_fullsized":"2024-09-16T05:17:20.740096+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:01.065967+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.065967+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:26.866764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"490'495","reported_seq":2408,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.755082+0000","last_change":"2024-09-16T05:17:19.755397+0000","last_active":"2024-09-16T05:17:19.755082+0000","last_peered":"2024-09-16T05:17:19.755082+0000","last_clean":"2024-09-16T05:17:19.755082+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:17:19.755082+0000","last_undegraded":"2024-09-16T05:17:19.755082+0000","last_fullsized":"2024-09-16T05:17:19.755082+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'461","last_scrub_stamp":"2024-09-16T05:16:40.073052+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:16:40.073052+0000","objects_scrubbed":1,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.221518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0076985600000000001,"stat_sum":{"num_bytes":10681646,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":20,"num_write_kb":13850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":684,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.972206+0000","last_change":"2024-09-16T05:15:59.140824+0000","last_active":"2024-09-16T05:17:19.972206+0000","last_peered":"2024-09-16T05:17:19.972206+0000","last_clean":"2024-09-16T05:17:19.972206+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:17:19.972206+0000","last_undegraded":"2024-09-16T05:17:19.972206+0000","last_fullsized":"2024-09-16T05:17:19.972206+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.140738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":2183,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.739882+0000","last_change":"2024-09-16T05:17:21.739882+0000","last_active":"2024-09-16T05:17:21.739882+0000","last_peered":"2024-09-16T05:17:21.739882+0000","last_clean":"2024-09-16T05:17:21.739882+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:17:21.739882+0000","last_undegraded":"2024-09-16T05:17:21.739882+0000","last_fullsized":"2024-09-16T05:17:21.739882+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:16:25.811761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.811761+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:47.635696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046986,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"1.b","version":"35'139","reported_seq":1229,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.738631+0000","last_change":"2024-09-16T05:17:07.872302+0000","last_active":"2024-09-16T05:17:21.738631+0000","last_peered":"2024-09-16T05:17:21.738631+0000","last_clean":"2024-09-16T05:17:21.738631+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:17:21.738631+0000","last_undegraded":"2024-09-16T05:17:21.738631+0000","last_fullsized":"2024-09-16T05:17:21.738631+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:17:07.872208+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:11.877781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"492'263","reported_seq":2181,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.743174+0000","last_change":"2024-09-16T05:17:21.743341+0000","last_active":"2024-09-16T05:17:21.743174+0000","last_peered":"2024-09-16T05:17:21.743174+0000","last_clean":"2024-09-16T05:17:21.743174+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:17:21.743174+0000","last_undegraded":"2024-09-16T05:17:21.743174+0000","last_fullsized":"2024-09-16T05:17:21.743174+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'259","last_scrub_stamp":"2024-09-16T05:17:19.452880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:19.452880+0000","objects_scrubbed":2,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:19.453757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029946550000000001,"stat_sum":{"num_bytes":3854971,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":12883,"num_write":29,"num_write_kb":6887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.3","version":"490'137","reported_seq":2050,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.974081+0000","last_change":"2024-09-16T05:17:19.974321+0000","last_active":"2024-09-16T05:17:19.974081+0000","last_peered":"2024-09-16T05:17:19.974081+0000","last_clean":"2024-09-16T05:17:19.974081+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:17:19.974081+0000","last_undegraded":"2024-09-16T05:17:19.974081+0000","last_fullsized":"2024-09-16T05:17:19.974081+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'135","last_scrub_stamp":"2024-09-16T05:17:19.355166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:19.355166+0000","objects_scrubbed":3,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:22.928356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022272799999999999,"stat_sum":{"num_bytes":4950305,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":49446,"num_write":27,"num_write_kb":9823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":1216,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.740719+0000","last_change":"2024-09-16T05:15:53.271304+0000","last_active":"2024-09-16T05:17:21.740719+0000","last_peered":"2024-09-16T05:17:21.740719+0000","last_clean":"2024-09-16T05:17:21.740719+0000","last_became_active":"2024-09-16T05:15:51.717864+0000","last_became_peered":"2024-09-16T05:15:51.717864+0000","last_unstale":"2024-09-16T05:17:21.740719+0000","last_undegraded":"2024-09-16T05:17:21.740719+0000","last_fullsized":"2024-09-16T05:17:21.740719+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.271201+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.271201+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.039968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":1913,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.974373+0000","last_change":"2024-09-16T05:17:19.974656+0000","last_active":"2024-09-16T05:17:19.974373+0000","last_peered":"2024-09-16T05:17:19.974373+0000","last_clean":"2024-09-16T05:17:19.974373+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:17:19.974373+0000","last_undegraded":"2024-09-16T05:17:19.974373+0000","last_fullsized":"2024-09-16T05:17:19.974373+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:17:06.386786+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:21.576256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022020350000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":865,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.970333+0000","last_change":"2024-09-16T05:17:18.748140+0000","last_active":"2024-09-16T05:17:19.970333+0000","last_peered":"2024-09-16T05:17:19.970333+0000","last_clean":"2024-09-16T05:17:19.970333+0000","last_became_active":"2024-09-16T05:17:18.747791+0000","last_became_peered":"2024-09-16T05:17:18.747791+0000","last_unstale":"2024-09-16T05:17:19.970333+0000","last_undegraded":"2024-09-16T05:17:19.970333+0000","last_fullsized":"2024-09-16T05:17:19.970333+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:06.854660+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:06.854660+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:09.348828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1220,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.740205+0000","last_change":"2024-09-16T05:17:03.402949+0000","last_active":"2024-09-16T05:17:21.740205+0000","last_peered":"2024-09-16T05:17:21.740205+0000","last_clean":"2024-09-16T05:17:21.740205+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:17:21.740205+0000","last_undegraded":"2024-09-16T05:17:21.740205+0000","last_fullsized":"2024-09-16T05:17:21.740205+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:03.402860+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.402860+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:12.276154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":1842,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.741053+0000","last_change":"2024-09-16T05:17:15.183839+0000","last_active":"2024-09-16T05:17:17.741053+0000","last_peered":"2024-09-16T05:17:17.741053+0000","last_clean":"2024-09-16T05:17:17.741053+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:17:17.741053+0000","last_undegraded":"2024-09-16T05:17:17.741053+0000","last_fullsized":"2024-09-16T05:17:17.741053+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:23.812825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.812825+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:36.330227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000962484,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"1.3","version":"35'139","reported_seq":1214,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.743440+0000","last_change":"2024-09-16T05:15:52.844991+0000","last_active":"2024-09-16T05:17:17.743440+0000","last_peered":"2024-09-16T05:17:17.743440+0000","last_clean":"2024-09-16T05:17:17.743440+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:17:17.743440+0000","last_undegraded":"2024-09-16T05:17:17.743440+0000","last_fullsized":"2024-09-16T05:17:17.743440+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:52.844849+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:52.844849+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:22.480579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":1945,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.742627+0000","last_change":"2024-09-16T05:17:21.742969+0000","last_active":"2024-09-16T05:17:21.742627+0000","last_peered":"2024-09-16T05:17:21.742627+0000","last_clean":"2024-09-16T05:17:21.742627+0000","last_became_active":"2024-09-16T05:13:43.688588+0000","last_became_peered":"2024-09-16T05:13:43.688588+0000","last_unstale":"2024-09-16T05:17:21.742627+0000","last_undegraded":"2024-09-16T05:17:21.742627+0000","last_fullsized":"2024-09-16T05:17:21.742627+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:16:46.325304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:46.325304+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:56.091686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024971390000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.7","version":"0'0","reported_seq":1154,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.749669+0000","last_change":"2024-09-16T05:17:19.752674+0000","last_active":"2024-09-16T05:17:19.749669+0000","last_peered":"2024-09-16T05:17:19.749669+0000","last_clean":"2024-09-16T05:17:19.749669+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:17:19.749669+0000","last_undegraded":"2024-09-16T05:17:19.749669+0000","last_fullsized":"2024-09-16T05:17:19.749669+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:17:05.143266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:05.143266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:34.607733+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0050667689999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":1185,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.738894+0000","last_change":"2024-09-16T05:16:58.840947+0000","last_active":"2024-09-16T05:17:21.738894+0000","last_peered":"2024-09-16T05:17:21.738894+0000","last_clean":"2024-09-16T05:17:21.738894+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:17:21.738894+0000","last_undegraded":"2024-09-16T05:17:21.738894+0000","last_fullsized":"2024-09-16T05:17:21.738894+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:58.840859+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:58.840859+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:20.477391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":324,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.741521+0000","last_change":"2024-09-16T05:17:15.183340+0000","last_active":"2024-09-16T05:17:17.741521+0000","last_peered":"2024-09-16T05:17:17.741521+0000","last_clean":"2024-09-16T05:17:17.741521+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:17:17.741521+0000","last_undegraded":"2024-09-16T05:17:17.741521+0000","last_fullsized":"2024-09-16T05:17:17.741521+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:17:00.760543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:00.760543+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:19.492659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040106400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"1.5","version":"35'139","reported_seq":957,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.970389+0000","last_change":"2024-09-16T05:15:56.607423+0000","last_active":"2024-09-16T05:17:19.970389+0000","last_peered":"2024-09-16T05:17:19.970389+0000","last_clean":"2024-09-16T05:17:19.970389+0000","last_became_active":"2024-09-16T05:15:51.724156+0000","last_became_peered":"2024-09-16T05:15:51.724156+0000","last_unstale":"2024-09-16T05:17:19.970389+0000","last_undegraded":"2024-09-16T05:17:19.970389+0000","last_fullsized":"2024-09-16T05:17:19.970389+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.607332+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.607332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"443'485","reported_seq":2375,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.748511+0000","last_change":"2024-09-16T05:17:19.748878+0000","last_active":"2024-09-16T05:17:19.748511+0000","last_peered":"2024-09-16T05:17:19.748511+0000","last_clean":"2024-09-16T05:17:19.748511+0000","last_became_active":"2024-09-16T05:14:25.823533+0000","last_became_peered":"2024-09-16T05:14:25.823533+0000","last_unstale":"2024-09-16T05:17:19.748511+0000","last_undegraded":"2024-09-16T05:17:19.748511+0000","last_fullsized":"2024-09-16T05:17:19.748511+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'479","last_scrub_stamp":"2024-09-16T05:15:57.067519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.067519+0000","objects_scrubbed":2,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001425017,"stat_sum":{"num_bytes":2672412,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":32281,"num_write":5,"num_write_kb":1272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":853,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.970010+0000","last_change":"2024-09-16T05:15:55.644520+0000","last_active":"2024-09-16T05:17:19.970010+0000","last_peered":"2024-09-16T05:17:19.970010+0000","last_clean":"2024-09-16T05:17:19.970010+0000","last_became_active":"2024-09-16T05:15:51.722851+0000","last_became_peered":"2024-09-16T05:15:51.722851+0000","last_unstale":"2024-09-16T05:17:19.970010+0000","last_undegraded":"2024-09-16T05:17:19.970010+0000","last_fullsized":"2024-09-16T05:17:19.970010+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:55.644436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:55.644436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1987,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.743607+0000","last_change":"2024-09-16T05:17:19.743787+0000","last_active":"2024-09-16T05:17:19.743607+0000","last_peered":"2024-09-16T05:17:19.743607+0000","last_clean":"2024-09-16T05:17:19.743607+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:17:19.743607+0000","last_undegraded":"2024-09-16T05:17:19.743607+0000","last_fullsized":"2024-09-16T05:17:19.743607+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:57.869742+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.869742+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011571190000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":1037,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.970065+0000","last_change":"2024-09-16T05:15:53.643617+0000","last_active":"2024-09-16T05:17:19.970065+0000","last_peered":"2024-09-16T05:17:19.970065+0000","last_clean":"2024-09-16T05:17:19.970065+0000","last_became_active":"2024-09-16T05:15:51.718810+0000","last_became_peered":"2024-09-16T05:15:51.718810+0000","last_unstale":"2024-09-16T05:17:19.970065+0000","last_undegraded":"2024-09-16T05:17:19.970065+0000","last_fullsized":"2024-09-16T05:17:19.970065+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:53.643528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:53.643528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":2078,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-09-16T05:17:18.740662+0000","last_change":"2024-09-16T05:17:18.741010+0000","last_active":"2024-09-16T05:17:18.740662+0000","last_peered":"2024-09-16T05:17:18.740662+0000","last_clean":"2024-09-16T05:17:18.740662+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:17:18.740662+0000","last_undegraded":"2024-09-16T05:17:18.740662+0000","last_fullsized":"2024-09-16T05:17:18.740662+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:16:58.139614+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:16:58.139614+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:13.399585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000510714,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"},{"start":"b5","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":1213,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.971977+0000","last_change":"2024-09-16T05:16:20.118747+0000","last_active":"2024-09-16T05:17:19.971977+0000","last_peered":"2024-09-16T05:17:19.971977+0000","last_clean":"2024-09-16T05:17:19.971977+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:17:19.971977+0000","last_undegraded":"2024-09-16T05:17:19.971977+0000","last_fullsized":"2024-09-16T05:17:19.971977+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:20.118658+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.118658+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.459808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"479'489","reported_seq":2422,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.739269+0000","last_change":"2024-09-16T05:17:21.739499+0000","last_active":"2024-09-16T05:17:21.739269+0000","last_peered":"2024-09-16T05:17:21.739269+0000","last_clean":"2024-09-16T05:17:21.739269+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:17:21.739269+0000","last_undegraded":"2024-09-16T05:17:21.739269+0000","last_fullsized":"2024-09-16T05:17:21.739269+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'489","last_scrub_stamp":"2024-09-16T05:17:19.936016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:19.936016+0000","objects_scrubbed":2,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:22.828356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074976800000000005,"stat_sum":{"num_bytes":5215067,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":66423,"num_write":18,"num_write_kb":14367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"1.9","version":"35'139","reported_seq":1208,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.739866+0000","last_change":"2024-09-16T05:16:18.233415+0000","last_active":"2024-09-16T05:17:21.739866+0000","last_peered":"2024-09-16T05:17:21.739866+0000","last_clean":"2024-09-16T05:17:21.739866+0000","last_became_active":"2024-09-16T05:10:35.953738+0000","last_became_peered":"2024-09-16T05:10:35.953738+0000","last_unstale":"2024-09-16T05:17:21.739866+0000","last_undegraded":"2024-09-16T05:17:21.739866+0000","last_fullsized":"2024-09-16T05:17:21.739866+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:18.233282+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:18.233282+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:35.750725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2132,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.742041+0000","last_change":"2024-09-16T05:17:19.742335+0000","last_active":"2024-09-16T05:17:19.742041+0000","last_peered":"2024-09-16T05:17:19.742041+0000","last_clean":"2024-09-16T05:17:19.742041+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:17:19.742041+0000","last_undegraded":"2024-09-16T05:17:19.742041+0000","last_fullsized":"2024-09-16T05:17:19.742041+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:16:39.912418+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.912418+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:52.676999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054256499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1213,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.747590+0000","last_change":"2024-09-16T05:16:20.111881+0000","last_active":"2024-09-16T05:17:19.747590+0000","last_peered":"2024-09-16T05:17:19.747590+0000","last_clean":"2024-09-16T05:17:19.747590+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:17:19.747590+0000","last_undegraded":"2024-09-16T05:17:19.747590+0000","last_fullsized":"2024-09-16T05:17:19.747590+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:16:20.111791+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:16:20.111791+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:28.416563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":695,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.739243+0000","last_change":"2024-09-16T05:17:08.734411+0000","last_active":"2024-09-16T05:17:19.739243+0000","last_peered":"2024-09-16T05:17:19.739243+0000","last_clean":"2024-09-16T05:17:19.739243+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:17:19.739243+0000","last_undegraded":"2024-09-16T05:17:19.739243+0000","last_fullsized":"2024-09-16T05:17:19.739243+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:56.743052+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:56.743052+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:19.924861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":1042,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.970144+0000","last_change":"2024-09-16T05:17:19.970503+0000","last_active":"2024-09-16T05:17:19.970144+0000","last_peered":"2024-09-16T05:17:19.970144+0000","last_clean":"2024-09-16T05:17:19.970144+0000","last_became_active":"2024-09-16T05:16:25.724548+0000","last_became_peered":"2024-09-16T05:16:25.724548+0000","last_unstale":"2024-09-16T05:17:19.970144+0000","last_undegraded":"2024-09-16T05:17:19.970144+0000","last_fullsized":"2024-09-16T05:17:19.970144+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:16:25.926404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.926404+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:33.154589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053968899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1216,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.972576+0000","last_change":"2024-09-16T05:17:18.743056+0000","last_active":"2024-09-16T05:17:19.972576+0000","last_peered":"2024-09-16T05:17:19.972576+0000","last_clean":"2024-09-16T05:17:19.972576+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:17:19.972576+0000","last_undegraded":"2024-09-16T05:17:19.972576+0000","last_fullsized":"2024-09-16T05:17:19.972576+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:15:58.103601+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:15:58.103601+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:24.442778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":1536,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.748270+0000","last_change":"2024-09-16T05:17:19.748499+0000","last_active":"2024-09-16T05:17:19.748270+0000","last_peered":"2024-09-16T05:17:19.748270+0000","last_clean":"2024-09-16T05:17:19.748270+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:17:19.748270+0000","last_undegraded":"2024-09-16T05:17:19.748270+0000","last_fullsized":"2024-09-16T05:17:19.748270+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:16:26.057442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:26.057442+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:49.807072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00109968,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.14","version":"433'289","reported_seq":1808,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.755347+0000","last_change":"2024-09-16T05:17:19.755647+0000","last_active":"2024-09-16T05:17:19.755347+0000","last_peered":"2024-09-16T05:17:19.755347+0000","last_clean":"2024-09-16T05:17:19.755347+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:17:19.755347+0000","last_undegraded":"2024-09-16T05:17:19.755347+0000","last_fullsized":"2024-09-16T05:17:19.755347+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:16:37.997840+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:16:37.997840+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:02.165560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0078737770000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.15","version":"454'519","reported_seq":2442,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.752851+0000","last_change":"2024-09-16T05:17:19.753090+0000","last_active":"2024-09-16T05:17:19.752851+0000","last_peered":"2024-09-16T05:17:19.752851+0000","last_clean":"2024-09-16T05:17:19.752851+0000","last_became_active":"2024-09-16T05:16:25.732256+0000","last_became_peered":"2024-09-16T05:16:25.732256+0000","last_unstale":"2024-09-16T05:17:19.752851+0000","last_undegraded":"2024-09-16T05:17:19.752851+0000","last_fullsized":"2024-09-16T05:17:19.752851+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"438'513","last_scrub_stamp":"2024-09-16T05:16:39.040839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.040839+0000","objects_scrubbed":1,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:40.251648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0054942259999999996,"stat_sum":{"num_bytes":3008552,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":13283,"num_write":4,"num_write_kb":1158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.16","version":"483'184","reported_seq":1728,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.975057+0000","last_change":"2024-09-16T05:17:19.975445+0000","last_active":"2024-09-16T05:17:19.975057+0000","last_peered":"2024-09-16T05:17:19.975057+0000","last_clean":"2024-09-16T05:17:19.975057+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:17:19.975057+0000","last_undegraded":"2024-09-16T05:17:19.975057+0000","last_fullsized":"2024-09-16T05:17:19.975057+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"451'178","last_scrub_stamp":"2024-09-16T05:16:59.339858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:59.339858+0000","objects_scrubbed":1,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:28.662201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026447939999999998,"stat_sum":{"num_bytes":4420439,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":25313,"num_write":23,"num_write_kb":9066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":543,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.822030+0000","last_change":"2024-09-16T05:17:17.822073+0000","last_active":"2024-09-16T05:17:17.822030+0000","last_peered":"2024-09-16T05:17:17.822030+0000","last_clean":"2024-09-16T05:17:17.822030+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:17:17.822030+0000","last_undegraded":"2024-09-16T05:17:17.822030+0000","last_fullsized":"2024-09-16T05:17:17.822030+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:17.821950+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:17.821950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:22.532106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":37,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.743003+0000","last_change":"2024-09-16T05:17:19.743003+0000","last_active":"2024-09-16T05:17:19.743003+0000","last_peered":"2024-09-16T05:17:19.743003+0000","last_clean":"2024-09-16T05:17:19.743003+0000","last_became_active":"2024-09-16T05:17:18.746691+0000","last_became_peered":"2024-09-16T05:17:18.746691+0000","last_unstale":"2024-09-16T05:17:19.743003+0000","last_undegraded":"2024-09-16T05:17:19.743003+0000","last_fullsized":"2024-09-16T05:17:19.743003+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:15:57.269939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.269939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095223699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":1918,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.749138+0000","last_change":"2024-09-16T05:17:19.749563+0000","last_active":"2024-09-16T05:17:19.749138+0000","last_peered":"2024-09-16T05:17:19.749138+0000","last_clean":"2024-09-16T05:17:19.749138+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:17:19.749138+0000","last_undegraded":"2024-09-16T05:17:19.749138+0000","last_fullsized":"2024-09-16T05:17:19.749138+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:16:44.037920+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:16:44.037920+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:13.726909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019770930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":1507,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.743288+0000","last_change":"2024-09-16T05:17:15.184891+0000","last_active":"2024-09-16T05:17:17.743288+0000","last_peered":"2024-09-16T05:17:17.743288+0000","last_clean":"2024-09-16T05:17:17.743288+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:17:17.743288+0000","last_undegraded":"2024-09-16T05:17:17.743288+0000","last_fullsized":"2024-09-16T05:17:17.743288+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:16:19.812615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.812615+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:22.721741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012276679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.1a","version":"472'688","reported_seq":2558,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-09-16T05:17:18.741617+0000","last_change":"2024-09-16T05:17:18.741617+0000","last_active":"2024-09-16T05:17:18.741617+0000","last_peered":"2024-09-16T05:17:18.741617+0000","last_clean":"2024-09-16T05:17:18.741617+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:17:18.741617+0000","last_undegraded":"2024-09-16T05:17:18.741617+0000","last_fullsized":"2024-09-16T05:17:18.741617+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"435'666","last_scrub_stamp":"2024-09-16T05:16:25.407253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:25.407253+0000","objects_scrubbed":3,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:26.451622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001069869,"stat_sum":{"num_bytes":8428471,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":80160,"num_write":61,"num_write_kb":28646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"},{"start":"b5","length":"1"}]},{"pgid":"2.1b","version":"482'410","reported_seq":2344,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.739624+0000","last_change":"2024-09-16T05:17:21.740046+0000","last_active":"2024-09-16T05:17:21.739624+0000","last_peered":"2024-09-16T05:17:21.739624+0000","last_clean":"2024-09-16T05:17:21.739624+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:17:21.739624+0000","last_undegraded":"2024-09-16T05:17:21.739624+0000","last_fullsized":"2024-09-16T05:17:21.739624+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"429'364","last_scrub_stamp":"2024-09-16T05:16:19.854079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.854079+0000","objects_scrubbed":4,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:39.716879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00106762,"stat_sum":{"num_bytes":9206293,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":40185,"num_write":34,"num_write_kb":10899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.1c","version":"468'393","reported_seq":2352,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.970562+0000","last_change":"2024-09-16T05:17:19.970876+0000","last_active":"2024-09-16T05:17:19.970562+0000","last_peered":"2024-09-16T05:17:19.970562+0000","last_clean":"2024-09-16T05:17:19.970562+0000","last_became_active":"2024-09-16T05:13:43.689860+0000","last_became_peered":"2024-09-16T05:13:43.689860+0000","last_unstale":"2024-09-16T05:17:19.970562+0000","last_undegraded":"2024-09-16T05:17:19.970562+0000","last_fullsized":"2024-09-16T05:17:19.970562+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'392","last_scrub_stamp":"2024-09-16T05:16:45.755860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:45.755860+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:55.869437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080964999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.1d","version":"490'829","reported_seq":2811,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.973957+0000","last_change":"2024-09-16T05:17:19.974123+0000","last_active":"2024-09-16T05:17:19.973957+0000","last_peered":"2024-09-16T05:17:19.973957+0000","last_clean":"2024-09-16T05:17:19.973957+0000","last_became_active":"2024-09-16T05:13:43.691953+0000","last_became_peered":"2024-09-16T05:13:43.691953+0000","last_unstale":"2024-09-16T05:17:19.973957+0000","last_undegraded":"2024-09-16T05:17:19.973957+0000","last_fullsized":"2024-09-16T05:17:19.973957+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"457'781","last_scrub_stamp":"2024-09-16T05:16:46.788437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:46.788437+0000","objects_scrubbed":4,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:08.899158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037966919999999999,"stat_sum":{"num_bytes":16143964,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":107,"num_read_kb":91295,"num_write":117,"num_write_kb":44098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.1e","version":"436'363","reported_seq":2276,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.744036+0000","last_change":"2024-09-16T05:17:15.184975+0000","last_active":"2024-09-16T05:17:17.744036+0000","last_peered":"2024-09-16T05:17:17.744036+0000","last_clean":"2024-09-16T05:17:17.744036+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:17:17.744036+0000","last_undegraded":"2024-09-16T05:17:17.744036+0000","last_fullsized":"2024-09-16T05:17:17.744036+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"436'363","last_scrub_stamp":"2024-09-16T05:16:39.785728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:39.785728+0000","objects_scrubbed":1,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:44.376215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00116866,"stat_sum":{"num_bytes":2067360,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":35576,"num_write":9,"num_write_kb":4850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.1f","version":"284'221","reported_seq":2135,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.740271+0000","last_change":"2024-09-16T05:17:19.740826+0000","last_active":"2024-09-16T05:17:19.740271+0000","last_peered":"2024-09-16T05:17:19.740271+0000","last_clean":"2024-09-16T05:17:19.740271+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:17:19.740271+0000","last_undegraded":"2024-09-16T05:17:19.740271+0000","last_fullsized":"2024-09-16T05:17:19.740271+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:16:21.634458+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:51.173711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011197799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.20","version":"472'179","reported_seq":2121,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.740101+0000","last_change":"2024-09-16T05:17:21.741056+0000","last_active":"2024-09-16T05:17:21.740101+0000","last_peered":"2024-09-16T05:17:21.740101+0000","last_clean":"2024-09-16T05:17:21.740101+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:17:21.740101+0000","last_undegraded":"2024-09-16T05:17:21.740101+0000","last_fullsized":"2024-09-16T05:17:21.740101+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"451'174","last_scrub_stamp":"2024-09-16T05:17:01.386565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:01.386565+0000","objects_scrubbed":1,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:22.681282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00125771,"stat_sum":{"num_bytes":4887302,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":114821,"num_write":26,"num_write_kb":11335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.21","version":"178'50","reported_seq":1841,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.741090+0000","last_change":"2024-09-16T05:17:15.184191+0000","last_active":"2024-09-16T05:17:17.741090+0000","last_peered":"2024-09-16T05:17:17.741090+0000","last_clean":"2024-09-16T05:17:17.741090+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:17:17.741090+0000","last_undegraded":"2024-09-16T05:17:17.741090+0000","last_fullsized":"2024-09-16T05:17:17.741090+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:16:19.705056+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.705056+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:45.328261+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010015989999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.22","version":"182'20","reported_seq":1039,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.971105+0000","last_change":"2024-09-16T05:17:19.971320+0000","last_active":"2024-09-16T05:17:19.971105+0000","last_peered":"2024-09-16T05:17:19.971105+0000","last_clean":"2024-09-16T05:17:19.971105+0000","last_became_active":"2024-09-16T05:13:31.746552+0000","last_became_peered":"2024-09-16T05:13:31.746552+0000","last_unstale":"2024-09-16T05:17:19.971105+0000","last_undegraded":"2024-09-16T05:17:19.971105+0000","last_fullsized":"2024-09-16T05:17:19.971105+0000","mapping_epoch":279,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:16:43.744675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:43.744675+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:02.841521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011275669999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":1374,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.743566+0000","last_change":"2024-09-16T05:17:15.184137+0000","last_active":"2024-09-16T05:17:17.743566+0000","last_peered":"2024-09-16T05:17:17.743566+0000","last_clean":"2024-09-16T05:17:17.743566+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:17:17.743566+0000","last_undegraded":"2024-09-16T05:17:17.743566+0000","last_fullsized":"2024-09-16T05:17:17.743566+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:16:17.214142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.214142+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:36.890508+0000","scrub_duration":429,"objects_trimmed":0,"snaptrim_duration":0.0010925869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.24","version":"182'89","reported_seq":1203,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.747837+0000","last_change":"2024-09-16T05:17:19.748182+0000","last_active":"2024-09-16T05:17:19.747837+0000","last_peered":"2024-09-16T05:17:19.747837+0000","last_clean":"2024-09-16T05:17:19.747837+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:17:19.747837+0000","last_undegraded":"2024-09-16T05:17:19.747837+0000","last_fullsized":"2024-09-16T05:17:19.747837+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:15:59.030307+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00092947799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]},{"pgid":"2.25","version":"489'697","reported_seq":2533,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-09-16T05:17:18.763828+0000","last_change":"2024-09-16T05:17:18.741677+0000","last_active":"2024-09-16T05:17:18.763828+0000","last_peered":"2024-09-16T05:17:18.763828+0000","last_clean":"2024-09-16T05:17:18.763828+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:17:18.763828+0000","last_undegraded":"2024-09-16T05:17:18.763828+0000","last_fullsized":"2024-09-16T05:17:18.763828+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"463'688","last_scrub_stamp":"2024-09-16T05:17:03.105650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.105650+0000","objects_scrubbed":2,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:19.227450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001021564,"stat_sum":{"num_bytes":10597549,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":71,"num_read_kb":93608,"num_write":50,"num_write_kb":26517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"},{"start":"b5","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":1209,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.743291+0000","last_change":"2024-09-16T05:17:21.743291+0000","last_active":"2024-09-16T05:17:21.743291+0000","last_peered":"2024-09-16T05:17:21.743291+0000","last_clean":"2024-09-16T05:17:21.743291+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:17:21.743291+0000","last_undegraded":"2024-09-16T05:17:21.743291+0000","last_fullsized":"2024-09-16T05:17:21.743291+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:16:13.148890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:13.148890+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:40.411448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027346639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.27","version":"0'0","reported_seq":1221,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.742549+0000","last_change":"2024-09-16T05:17:21.742549+0000","last_active":"2024-09-16T05:17:21.742549+0000","last_peered":"2024-09-16T05:17:21.742549+0000","last_clean":"2024-09-16T05:17:21.742549+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:17:21.742549+0000","last_undegraded":"2024-09-16T05:17:21.742549+0000","last_fullsized":"2024-09-16T05:17:21.742549+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:16:17.188503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.188503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:21.986947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021215460000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.28","version":"245'105","reported_seq":1974,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.743336+0000","last_change":"2024-09-16T05:17:15.184544+0000","last_active":"2024-09-16T05:17:17.743336+0000","last_peered":"2024-09-16T05:17:17.743336+0000","last_clean":"2024-09-16T05:17:17.743336+0000","last_became_active":"2024-09-16T05:13:04.680698+0000","last_became_peered":"2024-09-16T05:13:04.680698+0000","last_unstale":"2024-09-16T05:17:17.743336+0000","last_undegraded":"2024-09-16T05:17:17.743336+0000","last_fullsized":"2024-09-16T05:17:17.743336+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:15:57.809122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:15:57.809122+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:24.581534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011780060000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.29","version":"491'672","reported_seq":2622,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:22.166129+0000","last_change":"2024-09-16T05:17:22.166377+0000","last_active":"2024-09-16T05:17:22.166129+0000","last_peered":"2024-09-16T05:17:22.166129+0000","last_clean":"2024-09-16T05:17:22.166129+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:17:22.166129+0000","last_undegraded":"2024-09-16T05:17:22.166129+0000","last_fullsized":"2024-09-16T05:17:22.166129+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"472'660","last_scrub_stamp":"2024-09-16T05:17:03.427111+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:17:03.427111+0000","objects_scrubbed":2,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:18.206185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048266999999999998,"stat_sum":{"num_bytes":7983655,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":117,"num_read_kb":111542,"num_write":109,"num_write_kb":42675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.2a","version":"245'325","reported_seq":2210,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.741112+0000","last_change":"2024-09-16T05:17:21.741443+0000","last_active":"2024-09-16T05:17:21.741112+0000","last_peered":"2024-09-16T05:17:21.741112+0000","last_clean":"2024-09-16T05:17:21.741112+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:17:21.741112+0000","last_undegraded":"2024-09-16T05:17:21.741112+0000","last_fullsized":"2024-09-16T05:17:21.741112+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:16:23.269448+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:25.372858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001282341,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.2b","version":"481'262","reported_seq":2069,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-16T05:17:17.741376+0000","last_change":"2024-09-16T05:17:15.183919+0000","last_active":"2024-09-16T05:17:17.741376+0000","last_peered":"2024-09-16T05:17:17.741376+0000","last_clean":"2024-09-16T05:17:17.741376+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:17:17.741376+0000","last_undegraded":"2024-09-16T05:17:17.741376+0000","last_fullsized":"2024-09-16T05:17:17.741376+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"406'216","last_scrub_stamp":"2024-09-16T05:16:17.634887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:17.634887+0000","objects_scrubbed":2,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:17.839666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078109700000000002,"stat_sum":{"num_bytes":3344692,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":41,"num_read_kb":42424,"num_write":30,"num_write_kb":14799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"}]},{"pgid":"2.2c","version":"241'194","reported_seq":2025,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-09-16T05:17:18.741764+0000","last_change":"2024-09-16T05:17:18.742009+0000","last_active":"2024-09-16T05:17:18.741764+0000","last_peered":"2024-09-16T05:17:18.741764+0000","last_clean":"2024-09-16T05:17:18.741764+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:17:18.741764+0000","last_undegraded":"2024-09-16T05:17:18.741764+0000","last_fullsized":"2024-09-16T05:17:18.741764+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:17:02.077362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:02.077362+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:31.547041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010746600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"3"},{"start":"b5","length":"1"}]},{"pgid":"2.2d","version":"477'365","reported_seq":2268,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.741881+0000","last_change":"2024-09-16T05:17:21.742128+0000","last_active":"2024-09-16T05:17:21.741881+0000","last_peered":"2024-09-16T05:17:21.741881+0000","last_clean":"2024-09-16T05:17:21.741881+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:17:21.741881+0000","last_undegraded":"2024-09-16T05:17:21.741881+0000","last_fullsized":"2024-09-16T05:17:21.741881+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"377'326","last_scrub_stamp":"2024-09-16T05:16:15.212317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:15.212317+0000","objects_scrubbed":1,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017034179999999999,"stat_sum":{"num_bytes":5027150,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":63587,"num_write":20,"num_write_kb":10086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.2e","version":"489'426","reported_seq":2309,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-16T05:17:21.740836+0000","last_change":"2024-09-16T05:17:21.741053+0000","last_active":"2024-09-16T05:17:21.740836+0000","last_peered":"2024-09-16T05:17:21.740836+0000","last_clean":"2024-09-16T05:17:21.740836+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:17:21.740836+0000","last_undegraded":"2024-09-16T05:17:21.740836+0000","last_fullsized":"2024-09-16T05:17:21.740836+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"428'412","last_scrub_stamp":"2024-09-16T05:16:19.216112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:19.216112+0000","objects_scrubbed":3,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:23.785337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00091375899999999999,"stat_sum":{"num_bytes":2958098,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":32166,"num_write":37,"num_write_kb":13743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"6"}]},{"pgid":"2.2f","version":"482'396","reported_seq":2336,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-16T05:17:19.748544+0000","last_change":"2024-09-16T05:17:19.749083+0000","last_active":"2024-09-16T05:17:19.748544+0000","last_peered":"2024-09-16T05:17:19.748544+0000","last_clean":"2024-09-16T05:17:19.748544+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:17:19.748544+0000","last_undegraded":"2024-09-16T05:17:19.748544+0000","last_fullsized":"2024-09-16T05:17:19.748544+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"422'318","last_scrub_stamp":"2024-09-16T05:16:14.124319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:16:14.124319+0000","objects_scrubbed":4,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:17:40.313439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0015933970000000001,"stat_sum":{"num_bytes":6823840,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":25591,"num_write":57,"num_write_kb":19443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"ae","length":"1"},{"start":"b0","length":"4"},{"start":"b5","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":171472276,"num_objects":71,"num_object_clones":30,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":4,"num_read":1391,"num_read_kb":1420718,"num_write":984,"num_write_kb":421183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":222834688,"data_stored":261206904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19433,"ondisk_log_size":19433,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089443,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73716,"kb_used_data":34200,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93671436,"statfs":{"total":95995035648,"available":95919550464,"internally_reserved":0,"allocated":35020800,"data_stored":40059088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285672,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68020,"kb_used_data":28512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93677132,"statfs":{"total":95995035648,"available":95925383168,"internally_reserved":0,"allocated":29196288,"data_stored":31014381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514608,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61800,"kb_used_data":34764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93683352,"statfs":{"total":95995035648,"available":95931752448,"internally_reserved":0,"allocated":35598336,"data_stored":38260254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62064,"kb_used_data":22552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93683088,"statfs":{"total":95995035648,"available":95931482112,"internally_reserved":0,"allocated":23093248,"data_stored":23062951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103287,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59600,"kb_used_data":32568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93685552,"statfs":{"total":95995035648,"available":95934005248,"internally_reserved":0,"allocated":33349632,"data_stored":38756086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397630,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52052,"kb_used_data":25012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93693100,"statfs":{"total":95995035648,"available":95941734400,"internally_reserved":0,"allocated":25612288,"data_stored":26943522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62120,"kb_used_data":26772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93683032,"statfs":{"total":95995035648,"available":95931424768,"internally_reserved":0,"allocated":27414528,"data_stored":29560529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52388,"kb_used_data":17044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93692764,"statfs":{"total":95995035648,"available":95941390336,"internally_reserved":0,"allocated":17453056,"data_stored":17548849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117130,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64796,"kb_used_data":25276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93680356,"statfs":{"total":95995035648,"available":95928684544,"internally_reserved":0,"allocated":25882624,"data_stored":24857300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240417,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47192,"kb_used_data":14916,"kb_used_omap":10,"kb_used_meta":32245,"kb_avail":93697960,"statfs":{"total":95995035648,"available":95946711040,"internally_reserved":0,"allocated":15273984,"data_stored":15838865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":33019187},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52696,"kb_used_data":13184,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93692456,"statfs":{"total":95995035648,"available":95941074944,"internally_reserved":0,"allocated":13500416,"data_stored":11397058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738522,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49316,"kb_used_data":9800,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93695836,"statfs":{"total":95995035648,"available":95944536064,"internally_reserved":0,"allocated":10035200,"data_stored":7963892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3825664,"data_stored":3800639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8675328,"data_stored":8601223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9150464,"data_stored":11757701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19709952,"data_stored":20721128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11571200,"data_stored":13713276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21327872,"data_stored":25523708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19705856,"data_stored":23091014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":27455488,"data_stored":34912055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":19267584,"data_stored":21276091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":29716480,"data_stored":34424681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":23314432,"data_stored":27178808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":29114368,"data_stored":36206580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:17:24.294 INFO:tasks.thrashosds.thrasher:Setting 2.10 to [1, 10, 3, 4] 2024-09-16T05:17:24.294 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.10', '1', '10', '3', '4'] 2024-09-16T05:17:24.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 2.10 1 10 3 4 2024-09-16T05:17:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:24.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:24.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:24.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:24.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:24.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:24.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:24.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:24.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:24.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:24.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:24.779 INFO:tasks.rados.rados.2.smithi204.stdout:1691: finishing write tid 4 to smithi20457400-6 2024-09-16T05:17:24.779 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 345 (ObjNum 609 snap 181 seq_num 609) dirty exists 2024-09-16T05:17:24.779 INFO:tasks.rados.rados.2.smithi204.stdout:1691: left oid 6 (ObjNum 609 snap 181 seq_num 609) 2024-09-16T05:17:24.779 INFO:tasks.rados.rados.2.smithi204.stdout:1694: finishing write tid 1 to smithi20457400-47 2024-09-16T05:17:24.779 INFO:tasks.rados.rados.2.smithi204.stdout:1694: finishing write tid 2 to smithi20457400-47 2024-09-16T05:17:24.779 INFO:tasks.rados.rados.2.smithi204.stdout:1694: finishing write tid 3 to smithi20457400-47 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1694: finishing write tid 4 to smithi20457400-47 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 693 (ObjNum 611 snap 181 seq_num 611) dirty exists 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1694: left oid 47 (ObjNum 611 snap 181 seq_num 611) 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1695: finishing write tid 1 to smithi20457400-14 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1695: finishing write tid 2 to smithi20457400-14 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1695: finishing write tid 3 to smithi20457400-14 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1695: finishing write tid 4 to smithi20457400-14 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 842 (ObjNum 612 snap 181 seq_num 612) dirty exists 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1695: left oid 14 (ObjNum 612 snap 181 seq_num 612) 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1691: done (3 left) 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1694: done (2 left) 2024-09-16T05:17:24.780 INFO:tasks.rados.rados.2.smithi204.stdout:1695: done (1 left) 2024-09-16T05:17:24.781 INFO:tasks.rados.rados.2.smithi204.stdout:1696: done (0 left) 2024-09-16T05:17:24.781 INFO:tasks.rados.rados.2.smithi204.stdout:1697: read oid 11 snap -1 2024-09-16T05:17:24.781 INFO:tasks.rados.rados.2.smithi204.stdout:1697: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:17:24.781 INFO:tasks.rados.rados.2.smithi204.stdout:1698: delete oid 9 current snap is 181 2024-09-16T05:17:24.786 INFO:tasks.rados.rados.2.smithi204.stdout:1698: done (1 left) 2024-09-16T05:17:24.786 INFO:tasks.rados.rados.2.smithi204.stdout:1699: copy_from oid 7 from oid 40 current snap is 181 2024-09-16T05:17:24.786 INFO:tasks.rados.rados.2.smithi204.stdout:1700: write (excl) oid 29 current snap is 181 2024-09-16T05:17:24.786 INFO:tasks.rados.rados.2.smithi204.stdout:1700: seq_num 614 ranges {719498=711126,2046885=272613} 2024-09-16T05:17:24.795 INFO:tasks.rados.rados.2.smithi204.stdout:1700: writing smithi20457400-29 from 719498 to 1430624 tid 1 2024-09-16T05:17:24.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1700: writing smithi20457400-29 from 2046885 to 2319498 tid 2 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1701: rollback oid 33 current snap is 181 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1701: done (3 left) 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1702: rollback oid 18 current snap is 181 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1702: done (3 left) 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1703: read oid 25 snap -1 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1703: expect (ObjNum 605 snap 181 seq_num 605) 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1704: write oid 17 current snap is 181 2024-09-16T05:17:24.801 INFO:tasks.rados.rados.2.smithi204.stdout:1699: finishing copy_from to smithi20457400-7 2024-09-16T05:17:24.802 INFO:tasks.rados.rados.2.smithi204.stdout:1699: got expected ENOENT (src dne) 2024-09-16T05:17:24.802 INFO:tasks.rados.rados.2.smithi204.stdout:1697: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:17:24.819 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:24.826 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:24.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:24.858 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:24 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:17:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:24 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:17:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:24 smithi032 ceph-mon[27388]: from='client.16116 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:24 smithi032 ceph-mon[27388]: osdmap e494: 12 total, 12 up, 12 in 2024-09-16T05:17:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:24 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:17:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:24 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:17:24.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:24.949 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:24.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:25.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:25.064 INFO:tasks.rados.rados.2.smithi204.stdout:1699: finishing copy_from racing read to smithi20457400-7 2024-09-16T05:17:25.065 INFO:tasks.rados.rados.2.smithi204.stdout:1700: finishing write tid 1 to smithi20457400-29 2024-09-16T05:17:25.065 INFO:tasks.rados.rados.2.smithi204.stdout:1700: finishing write tid 2 to smithi20457400-29 2024-09-16T05:17:25.065 INFO:tasks.rados.rados.2.smithi204.stdout:1704: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-16T05:17:25.073 INFO:tasks.rados.rados.2.smithi204.stdout:1704: writing smithi20457400-17 from 736305 to 1522680 tid 1 2024-09-16T05:17:25.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:25.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:25.078 INFO:tasks.rados.rados.2.smithi204.stdout:1704: writing smithi20457400-17 from 2105367 to 2336305 tid 2 2024-09-16T05:17:25.080 INFO:tasks.rados.rados.2.smithi204.stdout:1697: done (4 left) 2024-09-16T05:17:25.080 INFO:tasks.rados.rados.2.smithi204.stdout:1699: done (3 left) 2024-09-16T05:17:25.081 INFO:tasks.rados.rados.2.smithi204.stdout:1705: write (excl) oid 19 current snap is 181 2024-09-16T05:17:25.081 INFO:tasks.rados.rados.2.smithi204.stdout:1700: finishing write tid 3 to smithi20457400-29 2024-09-16T05:17:25.081 INFO:tasks.rados.rados.2.smithi204.stdout:1700: finishing write tid 4 to smithi20457400-29 2024-09-16T05:17:25.081 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 699 (ObjNum 614 snap 181 seq_num 614) dirty exists 2024-09-16T05:17:25.081 INFO:tasks.rados.rados.2.smithi204.stdout:1700: left oid 29 (ObjNum 614 snap 181 seq_num 614) 2024-09-16T05:17:25.081 INFO:tasks.rados.rados.2.smithi204.stdout:1705: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-16T05:17:25.084 INFO:tasks.rados.rados.2.smithi204.stdout:1705: writing smithi20457400-19 from 753112 to 1331089 tid 1 2024-09-16T05:17:25.089 INFO:tasks.rados.rados.2.smithi204.stdout:1705: writing smithi20457400-19 from 1880202 to 2353112 tid 2 2024-09-16T05:17:25.090 INFO:tasks.rados.rados.2.smithi204.stdout:1700: done (3 left) 2024-09-16T05:17:25.090 INFO:tasks.rados.rados.2.smithi204.stdout:1706: write (excl) oid 49 current snap is 181 2024-09-16T05:17:25.090 INFO:tasks.rados.rados.2.smithi204.stdout:1703: expect (ObjNum 605 snap 181 seq_num 605) 2024-09-16T05:17:25.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:24 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:17:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:24 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:17:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:24 smithi204 ceph-mon[30933]: from='client.16116 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:24 smithi204 ceph-mon[30933]: osdmap e494: 12 total, 12 up, 12 in 2024-09-16T05:17:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:24 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:17:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:24 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:17:25.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:25.183 INFO:tasks.rados.rados.2.smithi204.stdout:1705: finishing write tid 1 to smithi20457400-19 2024-09-16T05:17:25.183 INFO:tasks.rados.rados.2.smithi204.stdout:1705: finishing write tid 2 to smithi20457400-19 2024-09-16T05:17:25.183 INFO:tasks.rados.rados.2.smithi204.stdout:1705: finishing write tid 3 to smithi20457400-19 2024-09-16T05:17:25.183 INFO:tasks.rados.rados.2.smithi204.stdout:1705: finishing write tid 4 to smithi20457400-19 2024-09-16T05:17:25.184 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 303 (ObjNum 616 snap 181 seq_num 616) dirty exists 2024-09-16T05:17:25.184 INFO:tasks.rados.rados.2.smithi204.stdout:1705: left oid 19 (ObjNum 616 snap 181 seq_num 616) 2024-09-16T05:17:25.184 INFO:tasks.rados.rados.2.smithi204.stdout:1706: seq_num 617 ranges {769919=653226,1938684=431235} 2024-09-16T05:17:25.190 INFO:tasks.rados.rados.2.smithi204.stdout:1706: writing smithi20457400-49 from 769919 to 1423145 tid 1 2024-09-16T05:17:25.196 INFO:tasks.rados.rados.2.smithi204.stdout:1706: writing smithi20457400-49 from 1938684 to 2369919 tid 2 2024-09-16T05:17:25.197 INFO:tasks.rados.rados.2.smithi204.stdout:1703: done (3 left) 2024-09-16T05:17:25.197 INFO:tasks.rados.rados.2.smithi204.stdout:1705: done (2 left) 2024-09-16T05:17:25.197 INFO:tasks.rados.rados.2.smithi204.stdout:1707: read oid 14 snap -1 2024-09-16T05:17:25.197 INFO:tasks.rados.rados.2.smithi204.stdout:1707: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-16T05:17:25.197 INFO:tasks.rados.rados.2.smithi204.stdout:1708: write oid 25 current snap is 181 2024-09-16T05:17:25.197 INFO:tasks.rados.rados.2.smithi204.stdout:1704: finishing write tid 1 to smithi20457400-17 2024-09-16T05:17:25.197 INFO:tasks.rados.rados.2.smithi204.stdout:1704: finishing write tid 2 to smithi20457400-17 2024-09-16T05:17:25.197 INFO:tasks.rados.rados.2.smithi204.stdout:1708: seq_num 618 ranges {786726=728475,2280813=105913} 2024-09-16T05:17:25.204 INFO:tasks.rados.rados.2.smithi204.stdout:1708: writing smithi20457400-25 from 786726 to 1515201 tid 1 2024-09-16T05:17:25.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:25.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:25.209 INFO:tasks.rados.rados.2.smithi204.stdout:1708: writing smithi20457400-25 from 2280813 to 2386726 tid 2 2024-09-16T05:17:25.209 INFO:tasks.rados.rados.2.smithi204.stdout:1709: read oid 44 snap -1 2024-09-16T05:17:25.209 INFO:tasks.rados.rados.2.smithi204.stdout:1709: expect (ObjNum 584 snap 175 seq_num 584) 2024-09-16T05:17:25.209 INFO:tasks.rados.rados.2.smithi204.stdout:1710: write (excl) oid 31 current snap is 181 2024-09-16T05:17:25.210 INFO:tasks.rados.rados.2.smithi204.stdout:1704: finishing write tid 3 to smithi20457400-17 2024-09-16T05:17:25.210 INFO:tasks.rados.rados.2.smithi204.stdout:1704: finishing write tid 4 to smithi20457400-17 2024-09-16T05:17:25.210 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 691 (ObjNum 615 snap 181 seq_num 615) dirty exists 2024-09-16T05:17:25.210 INFO:tasks.rados.rados.2.smithi204.stdout:1704: left oid 17 (ObjNum 615 snap 181 seq_num 615) 2024-09-16T05:17:25.210 INFO:tasks.rados.rados.2.smithi204.stdout:1706: finishing write tid 1 to smithi20457400-49 2024-09-16T05:17:25.210 INFO:tasks.rados.rados.2.smithi204.stdout:1707: expect (ObjNum 612 snap 181 seq_num 612) 2024-09-16T05:17:25.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:24 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:17:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:24 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:17:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:24 smithi196 ceph-mon[30802]: from='client.16116 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:17:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:24 smithi196 ceph-mon[30802]: osdmap e494: 12 total, 12 up, 12 in 2024-09-16T05:17:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:24 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:17:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:24 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:17:25.288 INFO:tasks.rados.rados.2.smithi204.stdout:1706: finishing write tid 2 to smithi20457400-49 2024-09-16T05:17:25.288 INFO:tasks.rados.rados.2.smithi204.stdout:1706: finishing write tid 3 to smithi20457400-49 2024-09-16T05:17:25.288 INFO:tasks.rados.rados.2.smithi204.stdout:1706: finishing write tid 4 to smithi20457400-49 2024-09-16T05:17:25.288 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 571 (ObjNum 617 snap 181 seq_num 617) dirty exists 2024-09-16T05:17:25.288 INFO:tasks.rados.rados.2.smithi204.stdout:1706: left oid 49 (ObjNum 617 snap 181 seq_num 617) 2024-09-16T05:17:25.288 INFO:tasks.rados.rados.2.smithi204.stdout:1708: finishing write tid 1 to smithi20457400-25 2024-09-16T05:17:25.288 INFO:tasks.rados.rados.2.smithi204.stdout:1710: seq_num 619 ranges {403533=403724,1539295=652201} 2024-09-16T05:17:25.291 INFO:tasks.rados.rados.2.smithi204.stdout:1710: writing smithi20457400-31 from 403533 to 807257 tid 1 2024-09-16T05:17:25.299 INFO:tasks.rados.rados.2.smithi204.stdout:1710: writing smithi20457400-31 from 1539295 to 2191496 tid 2 2024-09-16T05:17:25.299 INFO:tasks.rados.rados.2.smithi204.stdout:1704: done (5 left) 2024-09-16T05:17:25.299 INFO:tasks.rados.rados.2.smithi204.stdout:1706: done (4 left) 2024-09-16T05:17:25.299 INFO:tasks.rados.rados.2.smithi204.stdout:1707: done (3 left) 2024-09-16T05:17:25.299 INFO:tasks.rados.rados.2.smithi204.stdout:1711: rollback oid 36 current snap is 181 2024-09-16T05:17:25.299 INFO:tasks.rados.rados.2.smithi204.stdout:1711: done (3 left) 2024-09-16T05:17:25.299 INFO:tasks.rados.rados.2.smithi204.stdout:1712: write (excl) oid 27 current snap is 181 2024-09-16T05:17:25.300 INFO:tasks.rados.rados.2.smithi204.stdout:1708: finishing write tid 2 to smithi20457400-25 2024-09-16T05:17:25.300 INFO:tasks.rados.rados.2.smithi204.stdout:1708: finishing write tid 3 to smithi20457400-25 2024-09-16T05:17:25.300 INFO:tasks.rados.rados.2.smithi204.stdout:1712: seq_num 620 ranges {420340=478973,1597777=512212} 2024-09-16T05:17:25.304 INFO:tasks.rados.rados.2.smithi204.stdout:1712: writing smithi20457400-27 from 420340 to 899313 tid 1 2024-09-16T05:17:25.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:25.311 INFO:tasks.rados.rados.2.smithi204.stdout:1712: writing smithi20457400-27 from 1597777 to 2109989 tid 2 2024-09-16T05:17:25.311 INFO:tasks.rados.rados.2.smithi204.stdout:1713: copy_from oid 36 from oid 24 current snap is 181 2024-09-16T05:17:25.311 INFO:tasks.rados.rados.2.smithi204.stdout:1708: finishing write tid 4 to smithi20457400-25 2024-09-16T05:17:25.311 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 489 (ObjNum 618 snap 181 seq_num 618) dirty exists 2024-09-16T05:17:25.311 INFO:tasks.rados.rados.2.smithi204.stdout:1708: left oid 25 (ObjNum 618 snap 181 seq_num 618) 2024-09-16T05:17:25.311 INFO:tasks.rados.rados.2.smithi204.stdout:1709: expect (ObjNum 584 snap 175 seq_num 584) 2024-09-16T05:17:25.331 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:25.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:25.433 INFO:tasks.rados.rados.2.smithi204.stdout:1710: finishing write tid 1 to smithi20457400-31 2024-09-16T05:17:25.433 INFO:tasks.rados.rados.2.smithi204.stdout:1710: finishing write tid 2 to smithi20457400-31 2024-09-16T05:17:25.433 INFO:tasks.rados.rados.2.smithi204.stdout:1708: done (4 left) 2024-09-16T05:17:25.433 INFO:tasks.rados.rados.2.smithi204.stdout:1709: done (3 left) 2024-09-16T05:17:25.433 INFO:tasks.rados.rados.2.smithi204.stdout:1714: write oid 6 current snap is 181 2024-09-16T05:17:25.433 INFO:tasks.rados.rados.2.smithi204.stdout:1710: finishing write tid 3 to smithi20457400-31 2024-09-16T05:17:25.433 INFO:tasks.rados.rados.2.smithi204.stdout:1714: seq_num 621 ranges {437147=554222,1656259=655870} 2024-09-16T05:17:25.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:25.438 INFO:tasks.rados.rados.2.smithi204.stdout:1714: writing smithi20457400-6 from 437147 to 991369 tid 1 2024-09-16T05:17:25.445 INFO:tasks.rados.rados.2.smithi204.stdout:1714: writing smithi20457400-6 from 1656259 to 2312129 tid 2 2024-09-16T05:17:25.445 INFO:tasks.rados.rados.2.smithi204.stdout:1715: read oid 15 snap -1 2024-09-16T05:17:25.445 INFO:tasks.rados.rados.2.smithi204.stdout:1715: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:25.445 INFO:tasks.rados.rados.2.smithi204.stdout:1716: snap_create 2024-09-16T05:17:25.445 INFO:tasks.rados.rados.2.smithi204.stdout:1710: finishing write tid 4 to smithi20457400-31 2024-09-16T05:17:25.445 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 849 (ObjNum 619 snap 181 seq_num 619) dirty exists 2024-09-16T05:17:25.445 INFO:tasks.rados.rados.2.smithi204.stdout:1710: left oid 31 (ObjNum 619 snap 181 seq_num 619) 2024-09-16T05:17:25.445 INFO:tasks.rados.rados.2.smithi204.stdout:1712: finishing write tid 1 to smithi20457400-27 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:1712: finishing write tid 2 to smithi20457400-27 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:1712: finishing write tid 3 to smithi20457400-27 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:1712: finishing write tid 4 to smithi20457400-27 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 312 (ObjNum 620 snap 181 seq_num 620) dirty exists 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:1712: left oid 27 (ObjNum 620 snap 181 seq_num 620) 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:1713: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:1713: finishing copy_from to smithi20457400-36 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 511 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:25.446 INFO:tasks.rados.rados.2.smithi204.stdout:1714: finishing write tid 1 to smithi20457400-6 2024-09-16T05:17:25.448 INFO:tasks.rados.rados.2.smithi204.stdout:1714: finishing write tid 2 to smithi20457400-6 2024-09-16T05:17:25.448 INFO:tasks.rados.rados.2.smithi204.stdout:1714: finishing write tid 3 to smithi20457400-6 2024-09-16T05:17:25.448 INFO:tasks.rados.rados.2.smithi204.stdout:1715: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:25.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:25.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:25.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:25.512 INFO:tasks.rados.rados.2.smithi204.stdout:1714: finishing write tid 4 to smithi20457400-6 2024-09-16T05:17:25.512 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 348 (ObjNum 621 snap 181 seq_num 621) dirty exists 2024-09-16T05:17:25.512 INFO:tasks.rados.rados.2.smithi204.stdout:1714: left oid 6 (ObjNum 621 snap 181 seq_num 621) 2024-09-16T05:17:25.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:25.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:25.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:25.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:25.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:25.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:25.735 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:17:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1710: done (5 left) 2024-09-16T05:17:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1712: done (4 left) 2024-09-16T05:17:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1713: done (3 left) 2024-09-16T05:17:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1714: done (2 left) 2024-09-16T05:17:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1715: done (1 left) 2024-09-16T05:17:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1716: done (0 left) 2024-09-16T05:17:25.740 INFO:tasks.rados.rados.2.smithi204.stdout:1717: snap_remove snap 181 2024-09-16T05:17:25.809 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:25.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:25.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:25.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:25.960 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:25.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:26.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:26.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:26.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: pgmap v903: 107 pgs: 107 active+clean; 163 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4.2 MiB/s wr, 26 op/s 2024-09-16T05:17:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: osdmap e495: 12 total, 12 up, 12 in 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/440738045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/440738045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:25 smithi032 ceph-mon[27388]: osdmap e496: 12 total, 12 up, 12 in 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: pgmap v903: 107 pgs: 107 active+clean; 163 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4.2 MiB/s wr, 26 op/s 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: osdmap e495: 12 total, 12 up, 12 in 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/440738045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/440738045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:25 smithi204 ceph-mon[30933]: osdmap e496: 12 total, 12 up, 12 in 2024-09-16T05:17:26.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:26.214 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:26.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: pgmap v903: 107 pgs: 107 active+clean; 163 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4.2 MiB/s wr, 26 op/s 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: osdmap e495: 12 total, 12 up, 12 in 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/440738045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:17:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/440738045' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:25 smithi196 ceph-mon[30802]: osdmap e496: 12 total, 12 up, 12 in 2024-09-16T05:17:26.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:26.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:26.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:26.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:26.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:26.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:26.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:26.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:26.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:26.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:26.715 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:26.715 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:26.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:26.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:26.736 INFO:teuthology.orchestra.run.smithi032.stderr:set 2.10 pg_upmap_items mapping to [1->10,3->4] 2024-09-16T05:17:26.740 INFO:tasks.rados.rados.2.smithi204.stdout:1717: done (0 left) 2024-09-16T05:17:26.740 INFO:tasks.rados.rados.2.smithi204.stdout:1718: delete oid 14 current snap is 182 2024-09-16T05:17:26.742 INFO:tasks.rados.rados.2.smithi204.stdout:1718: done (0 left) 2024-09-16T05:17:26.742 INFO:tasks.rados.rados.2.smithi204.stdout:1719: rollback oid 45 current snap is 182 2024-09-16T05:17:26.743 INFO:tasks.rados.rados.2.smithi204.stdout:1719: done (0 left) 2024-09-16T05:17:26.743 INFO:tasks.rados.rados.2.smithi204.stdout:1720: copy_from oid 2 from oid 24 current snap is 182 2024-09-16T05:17:26.743 INFO:tasks.rados.rados.2.smithi204.stdout:1721: copy_from oid 42 from oid 20 current snap is 182 2024-09-16T05:17:26.743 INFO:tasks.rados.rados.2.smithi204.stdout:1722: delete oid 18 current snap is 182 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1722: done (2 left) 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1723: read oid 32 snap -1 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1723: expect deleted 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1724: read oid 39 snap -1 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1724: expect deleted 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1725: copy_from oid 30 from oid 46 current snap is 182 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1726: rollback oid 15 current snap is 182 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1726: done (5 left) 2024-09-16T05:17:26.750 INFO:tasks.rados.rados.2.smithi204.stdout:1727: delete oid 33 current snap is 182 2024-09-16T05:17:26.754 INFO:tasks.rados.rados.2.smithi204.stdout:1723: done (5 left) 2024-09-16T05:17:26.754 INFO:tasks.rados.rados.2.smithi204.stdout:1724: done (4 left) 2024-09-16T05:17:26.754 INFO:tasks.rados.rados.2.smithi204.stdout:1727: done (3 left) 2024-09-16T05:17:26.754 INFO:tasks.rados.rados.2.smithi204.stdout:1728: rollback oid 8 current snap is 182 2024-09-16T05:17:26.754 INFO:tasks.rados.rados.2.smithi204.stdout:1728: done (3 left) 2024-09-16T05:17:26.755 INFO:tasks.rados.rados.2.smithi204.stdout:1729: copy_from oid 25 from oid 29 current snap is 182 2024-09-16T05:17:26.755 INFO:tasks.rados.rados.2.smithi204.stdout:1730: write oid 16 current snap is 182 2024-09-16T05:17:26.755 INFO:tasks.rados.rados.2.smithi204.stdout:1730: seq_num 625 ranges {504375=571571,1490187=663208} 2024-09-16T05:17:26.759 INFO:tasks.rados.rados.2.smithi204.stdout:1730: writing smithi20457400-16 from 504375 to 1075946 tid 1 2024-09-16T05:17:26.767 INFO:tasks.rados.rados.2.smithi204.stdout:1730: writing smithi20457400-16 from 1490187 to 2153395 tid 2 2024-09-16T05:17:26.768 INFO:tasks.rados.rados.2.smithi204.stdout:1731: write oid 26 current snap is 182 2024-09-16T05:17:26.768 INFO:tasks.rados.rados.2.smithi204.stdout:1721: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:17:26.768 INFO:tasks.rados.rados.2.smithi204.stdout:1731: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-16T05:17:26.774 INFO:tasks.rados.rados.2.smithi204.stdout:1731: writing smithi20457400-26 from 521182 to 1168002 tid 1 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:1731: writing smithi20457400-26 from 1948669 to 2471888 tid 2 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:1732: rollback oid 37 current snap is 182 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:1732: done (6 left) 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:1733: write (excl) oid 44 current snap is 182 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:1721: finishing copy_from to smithi20457400-42 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 714 (ObjNum 480 snap 151 seq_num 480) dirty exists 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:1725: finishing copy_from racing read to smithi20457400-30 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:1725: finishing copy_from to smithi20457400-30 2024-09-16T05:17:26.783 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 329 (ObjNum 493 snap 152 seq_num 493) dirty exists 2024-09-16T05:17:26.784 INFO:tasks.rados.rados.2.smithi204.stdout:1720: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:17:26.784 INFO:tasks.rados.rados.2.smithi204.stdout:1730: finishing write tid 1 to smithi20457400-16 2024-09-16T05:17:26.784 INFO:tasks.rados.rados.2.smithi204.stdout:1733: seq_num 627 ranges {537989=722069,1890798=647191} 2024-09-16T05:17:26.788 INFO:tasks.rados.rados.2.smithi204.stdout:1733: writing smithi20457400-44 from 537989 to 1260058 tid 1 2024-09-16T05:17:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:1733: writing smithi20457400-44 from 1890798 to 2537989 tid 2 2024-09-16T05:17:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:1721: done (6 left) 2024-09-16T05:17:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:1725: done (5 left) 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1734: read oid 22 snap -1 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1734: expect deleted 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1735: write oid 13 current snap is 182 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1729: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1720: finishing copy_from to smithi20457400-2 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 511 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1729: finishing copy_from to smithi20457400-25 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 699 (ObjNum 614 snap 181 seq_num 614) dirty exists 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1730: finishing write tid 2 to smithi20457400-16 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1730: finishing write tid 3 to smithi20457400-16 2024-09-16T05:17:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:1735: seq_num 628 ranges {554796=797318,1949280=526888} 2024-09-16T05:17:26.802 INFO:tasks.rados.rados.2.smithi204.stdout:1735: writing smithi20457400-13 from 554796 to 1352114 tid 1 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:1735: writing smithi20457400-13 from 1949280 to 2476168 tid 2 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:1720: done (6 left) 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:1729: done (5 left) 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:1736: copy_from oid 29 from oid 43 current snap is 182 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:1730: finishing write tid 4 to smithi20457400-16 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 692 (ObjNum 625 snap 182 seq_num 625) dirty exists 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:1730: left oid 16 (ObjNum 625 snap 182 seq_num 625) 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:1731: finishing write tid 1 to smithi20457400-26 2024-09-16T05:17:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:1730: done (5 left) 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1737: read oid 46 snap -1 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1737: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1738: rollback oid 17 current snap is 182 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1738: done (6 left) 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1739: read oid 7 snap -1 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1739: expect deleted 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1740: copy_from oid 35 from oid 24 current snap is 182 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1731: finishing write tid 2 to smithi20457400-26 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1731: finishing write tid 3 to smithi20457400-26 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1731: finishing write tid 4 to smithi20457400-26 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 854 (ObjNum 626 snap 182 seq_num 626) dirty exists 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1731: left oid 26 (ObjNum 626 snap 182 seq_num 626) 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1733: finishing write tid 1 to smithi20457400-44 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1733: finishing write tid 2 to smithi20457400-44 2024-09-16T05:17:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:1735: finishing write tid 1 to smithi20457400-13 2024-09-16T05:17:26.812 INFO:tasks.rados.rados.2.smithi204.stdout:1731: done (7 left) 2024-09-16T05:17:26.812 INFO:tasks.rados.rados.2.smithi204.stdout:1734: done (6 left) 2024-09-16T05:17:26.812 INFO:tasks.rados.rados.2.smithi204.stdout:1741: write (excl) oid 3 current snap is 182 2024-09-16T05:17:26.812 INFO:tasks.rados.rados.2.smithi204.stdout:1741: seq_num 629 ranges {571603=472567,1607762=670546} 2024-09-16T05:17:26.815 INFO:tasks.rados.rados.2.smithi204.stdout:1741: writing smithi20457400-3 from 571603 to 1044170 tid 1 2024-09-16T05:17:26.822 INFO:tasks.rados.rados.2.smithi204.stdout:1741: writing smithi20457400-3 from 1607762 to 2278308 tid 2 2024-09-16T05:17:26.822 INFO:tasks.rados.rados.2.smithi204.stdout:1742: write oid 33 current snap is 182 2024-09-16T05:17:26.822 INFO:tasks.rados.rados.2.smithi204.stdout:1742: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2024-09-16T05:17:26.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:26.828 INFO:tasks.rados.rados.2.smithi204.stdout:1742: writing smithi20457400-33 from 588410 to 1136226 tid 1 2024-09-16T05:17:26.833 INFO:tasks.rados.rados.2.smithi204.stdout:1742: writing smithi20457400-33 from 1666244 to 2080448 tid 2 2024-09-16T05:17:26.837 INFO:tasks.rados.rados.2.smithi204.stdout:1742: writing smithi20457400-33 from 2484603 to 2588410 tid 3 2024-09-16T05:17:26.837 INFO:tasks.rados.rados.2.smithi204.stdout:1743: read oid 26 snap -1 2024-09-16T05:17:26.837 INFO:tasks.rados.rados.2.smithi204.stdout:1743: expect (ObjNum 626 snap 182 seq_num 626) 2024-09-16T05:17:26.837 INFO:tasks.rados.rados.2.smithi204.stdout:1744: rollback oid 5 current snap is 182 2024-09-16T05:17:26.837 INFO:tasks.rados.rados.2.smithi204.stdout:1744: done (9 left) 2024-09-16T05:17:26.837 INFO:tasks.rados.rados.2.smithi204.stdout:1745: write oid 12 current snap is 182 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:1736: finishing copy_from to smithi20457400-29 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:1736: got expected ENOENT (src dne) 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:1736: finishing copy_from racing read to smithi20457400-29 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:1735: finishing write tid 2 to smithi20457400-13 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:1735: finishing write tid 3 to smithi20457400-13 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:1735: finishing write tid 4 to smithi20457400-13 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 522 (ObjNum 628 snap 182 seq_num 628) dirty exists 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:1735: left oid 13 (ObjNum 628 snap 182 seq_num 628) 2024-09-16T05:17:26.838 INFO:tasks.rados.rados.2.smithi204.stdout:1741: finishing write tid 1 to smithi20457400-3 2024-09-16T05:17:26.839 INFO:tasks.rados.rados.2.smithi204.stdout:1737: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:17:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:26.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:26.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:26.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:26.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:27.043 INFO:tasks.rados.rados.2.smithi204.stdout:1740: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:17:27.043 INFO:tasks.rados.rados.2.smithi204.stdout:1740: finishing copy_from to smithi20457400-35 2024-09-16T05:17:27.043 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 511 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:27.043 INFO:tasks.rados.rados.2.smithi204.stdout:1741: finishing write tid 2 to smithi20457400-3 2024-09-16T05:17:27.043 INFO:tasks.rados.rados.2.smithi204.stdout:1741: finishing write tid 3 to smithi20457400-3 2024-09-16T05:17:27.043 INFO:tasks.rados.rados.2.smithi204.stdout:1745: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-16T05:17:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:1745: writing smithi20457400-12 from 605217 to 1344635 tid 1 2024-09-16T05:17:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:1745: writing smithi20457400-12 from 2124726 to 2605217 tid 2 2024-09-16T05:17:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:1735: done (9 left) 2024-09-16T05:17:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:1736: done (8 left) 2024-09-16T05:17:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:1737: done (7 left) 2024-09-16T05:17:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:1739: done (6 left) 2024-09-16T05:17:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:1740: done (5 left) 2024-09-16T05:17:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:1746: write oid 10 current snap is 182 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1741: finishing write tid 4 to smithi20457400-3 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 496 (ObjNum 629 snap 182 seq_num 629) dirty exists 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1741: left oid 3 (ObjNum 629 snap 182 seq_num 629) 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1742: finishing write tid 1 to smithi20457400-33 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1742: finishing write tid 2 to smithi20457400-33 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1742: finishing write tid 3 to smithi20457400-33 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1742: finishing write tid 4 to smithi20457400-33 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1742: finishing write tid 5 to smithi20457400-33 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 751 (ObjNum 630 snap 182 seq_num 630) dirty exists 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1742: left oid 33 (ObjNum 630 snap 182 seq_num 630) 2024-09-16T05:17:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:1743: expect (ObjNum 626 snap 182 seq_num 626) 2024-09-16T05:17:27.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:27.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:27.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:27.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:26 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:17:27.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:26 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:17:27.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:26 smithi032 ceph-mon[27388]: 2.36 scrub starts 2024-09-16T05:17:27.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:26 smithi032 ceph-mon[27388]: 2.36 scrub ok 2024-09-16T05:17:27.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2848344672' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [1, 10, 3, 4]}]: dispatch 2024-09-16T05:17:27.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2848344672' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [1, 10, 3, 4]}]': finished 2024-09-16T05:17:27.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:26 smithi032 ceph-mon[27388]: osdmap e497: 12 total, 12 up, 12 in 2024-09-16T05:17:27.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:26 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:17:27.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:26 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:17:27.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:26 smithi204 ceph-mon[30933]: 2.36 scrub starts 2024-09-16T05:17:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:26 smithi204 ceph-mon[30933]: 2.36 scrub ok 2024-09-16T05:17:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2848344672' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [1, 10, 3, 4]}]: dispatch 2024-09-16T05:17:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2848344672' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [1, 10, 3, 4]}]': finished 2024-09-16T05:17:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:26 smithi204 ceph-mon[30933]: osdmap e497: 12 total, 12 up, 12 in 2024-09-16T05:17:27.196 INFO:tasks.rados.rados.2.smithi204.stdout:1733: finishing write tid 3 to smithi20457400-44 2024-09-16T05:17:27.196 INFO:tasks.rados.rados.2.smithi204.stdout:1733: finishing write tid 4 to smithi20457400-44 2024-09-16T05:17:27.197 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 290 (ObjNum 627 snap 182 seq_num 627) dirty exists 2024-09-16T05:17:27.197 INFO:tasks.rados.rados.2.smithi204.stdout:1733: left oid 44 (ObjNum 627 snap 182 seq_num 627) 2024-09-16T05:17:27.197 INFO:tasks.rados.rados.2.smithi204.stdout:1746: seq_num 632 ranges {622024=414667,1783208=417873} 2024-09-16T05:17:27.200 INFO:tasks.rados.rados.2.smithi204.stdout:1746: writing smithi20457400-10 from 622024 to 1036691 tid 1 2024-09-16T05:17:27.206 INFO:tasks.rados.rados.2.smithi204.stdout:1746: writing smithi20457400-10 from 1783208 to 2201081 tid 2 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1733: done (5 left) 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1741: done (4 left) 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1742: done (3 left) 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1743: done (2 left) 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1747: snap_create 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1745: finishing write tid 1 to smithi20457400-12 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1745: finishing write tid 2 to smithi20457400-12 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1745: finishing write tid 3 to smithi20457400-12 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1745: finishing write tid 4 to smithi20457400-12 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 702 (ObjNum 631 snap 182 seq_num 631) dirty exists 2024-09-16T05:17:27.207 INFO:tasks.rados.rados.2.smithi204.stdout:1745: left oid 12 (ObjNum 631 snap 182 seq_num 631) 2024-09-16T05:17:27.208 INFO:tasks.rados.rados.2.smithi204.stdout:1746: finishing write tid 1 to smithi20457400-10 2024-09-16T05:17:27.209 INFO:tasks.rados.rados.2.smithi204.stdout:1746: finishing write tid 2 to smithi20457400-10 2024-09-16T05:17:27.209 INFO:tasks.rados.rados.2.smithi204.stdout:1746: finishing write tid 3 to smithi20457400-10 2024-09-16T05:17:27.209 INFO:tasks.rados.rados.2.smithi204.stdout:1746: finishing write tid 4 to smithi20457400-10 2024-09-16T05:17:27.209 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 700 (ObjNum 632 snap 182 seq_num 632) dirty exists 2024-09-16T05:17:27.209 INFO:tasks.rados.rados.2.smithi204.stdout:1746: left oid 10 (ObjNum 632 snap 182 seq_num 632) 2024-09-16T05:17:27.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:27.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:26 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:17:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:26 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:17:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:26 smithi196 ceph-mon[30802]: 2.36 scrub starts 2024-09-16T05:17:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:26 smithi196 ceph-mon[30802]: 2.36 scrub ok 2024-09-16T05:17:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2848344672' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [1, 10, 3, 4]}]: dispatch 2024-09-16T05:17:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2848344672' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [1, 10, 3, 4]}]': finished 2024-09-16T05:17:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:26 smithi196 ceph-mon[30802]: osdmap e497: 12 total, 12 up, 12 in 2024-09-16T05:17:27.239 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:27.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:27.341 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:27.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:27.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:27.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:27.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:27.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:27.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:27.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:27.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:27.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:27.740 INFO:tasks.rados.rados.2.smithi204.stdout:1745: done (2 left) 2024-09-16T05:17:27.741 INFO:tasks.rados.rados.2.smithi204.stdout:1746: done (1 left) 2024-09-16T05:17:27.741 INFO:tasks.rados.rados.2.smithi204.stdout:1747: done (0 left) 2024-09-16T05:17:27.741 INFO:tasks.rados.rados.2.smithi204.stdout:1748: snap_create 2024-09-16T05:17:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:27.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:27.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:27.751 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:27.751 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:27.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:27.796 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:27.796 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:27.816 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:27.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:27.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:27.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:27.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:17:27.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:27.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:27.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:27.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:28.021 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: pgmap v906: 107 pgs: 107 active+clean; 129 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3088693270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: osdmap e498: 12 total, 12 up, 12 in 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:17:28.022 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:27 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:17:28.040 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:28.040 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:28.062 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:28.089 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:28.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:28.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:28.111 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:28.111 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:28.117 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:28.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:28.133 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:28.133 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:28.154 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:28.155 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:28.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: pgmap v906: 107 pgs: 107 active+clean; 129 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3088693270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: osdmap e498: 12 total, 12 up, 12 in 2024-09-16T05:17:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:17:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:27 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:17:28.177 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:28.177 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:28.198 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:28.199 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:28.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: pgmap v906: 107 pgs: 107 active+clean; 129 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3088693270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:17:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:17:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:17:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: osdmap e498: 12 total, 12 up, 12 in 2024-09-16T05:17:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:17:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:27 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:17:28.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:28.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:28.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:28.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:28.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:28.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:17:28.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:28.498 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:28.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:28.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:28.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:28.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:28.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:28.743 INFO:tasks.rados.rados.2.smithi204.stdout:1748: done (0 left) 2024-09-16T05:17:28.743 INFO:tasks.rados.rados.2.smithi204.stdout:1749: read oid 14 snap -1 2024-09-16T05:17:28.743 INFO:tasks.rados.rados.2.smithi204.stdout:1749: expect deleted 2024-09-16T05:17:28.743 INFO:tasks.rados.rados.2.smithi204.stdout:1750: copy_from oid 26 from oid 22 current snap is 184 2024-09-16T05:17:28.743 INFO:tasks.rados.rados.2.smithi204.stdout:1751: copy_from oid 45 from oid 30 current snap is 184 2024-09-16T05:17:28.743 INFO:tasks.rados.rados.2.smithi204.stdout:1752: snap_create 2024-09-16T05:17:28.746 INFO:tasks.rados.rados.2.smithi204.stdout:1750: finishing copy_from to smithi20457400-26 2024-09-16T05:17:28.746 INFO:tasks.rados.rados.2.smithi204.stdout:1750: got expected ENOENT (src dne) 2024-09-16T05:17:28.747 INFO:tasks.rados.rados.2.smithi204.stdout:1750: finishing copy_from racing read to smithi20457400-26 2024-09-16T05:17:28.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:28.753 INFO:tasks.rados.rados.2.smithi204.stdout:1751: finishing copy_from racing read to smithi20457400-45 2024-09-16T05:17:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:28.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:28.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:28.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:28.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:28.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:29.003 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:29.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:29.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:28 smithi032 ceph-mon[27388]: 1.3 deep-scrub starts 2024-09-16T05:17:29.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:28 smithi032 ceph-mon[27388]: 1.3 deep-scrub ok 2024-09-16T05:17:29.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:28 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:17:29.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:28 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:17:29.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:28 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:17:29.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:28 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:17:29.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:28 smithi032 ceph-mon[27388]: osdmap e499: 12 total, 12 up, 12 in 2024-09-16T05:17:29.091 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:29.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:29.131 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:29.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:29.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:28 smithi204 ceph-mon[30933]: 1.3 deep-scrub starts 2024-09-16T05:17:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:28 smithi204 ceph-mon[30933]: 1.3 deep-scrub ok 2024-09-16T05:17:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:28 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:17:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:28 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:17:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:28 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:17:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:28 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:17:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:28 smithi204 ceph-mon[30933]: osdmap e499: 12 total, 12 up, 12 in 2024-09-16T05:17:29.232 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:29.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:28 smithi196 ceph-mon[30802]: 1.3 deep-scrub starts 2024-09-16T05:17:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:28 smithi196 ceph-mon[30802]: 1.3 deep-scrub ok 2024-09-16T05:17:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:28 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:17:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:28 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:17:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:28 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:17:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:28 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:17:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:28 smithi196 ceph-mon[30802]: osdmap e499: 12 total, 12 up, 12 in 2024-09-16T05:17:29.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:29.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:29.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:29.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:29.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:29.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:29.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:29.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:29.594 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:17:29.594 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:17:29.594 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:17:29.594 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:17:29.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:29.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:29.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:29.694 INFO:tasks.rados.rados.2.smithi204.stdout:1751: finishing copy_from to smithi20457400-45 2024-09-16T05:17:29.694 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 329 (ObjNum 493 snap 152 seq_num 493) dirty exists 2024-09-16T05:17:29.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:29.744 INFO:tasks.rados.rados.2.smithi204.stdout:1749: done (3 left) 2024-09-16T05:17:29.744 INFO:tasks.rados.rados.2.smithi204.stdout:1750: done (2 left) 2024-09-16T05:17:29.744 INFO:tasks.rados.rados.2.smithi204.stdout:1751: done (1 left) 2024-09-16T05:17:29.744 INFO:tasks.rados.rados.2.smithi204.stdout:1752: done (0 left) 2024-09-16T05:17:29.744 INFO:tasks.rados.rados.2.smithi204.stdout:1753: copy_from oid 10 from oid 30 current snap is 185 2024-09-16T05:17:29.744 INFO:tasks.rados.rados.2.smithi204.stdout:1754: snap_create 2024-09-16T05:17:29.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:29.759 INFO:tasks.rados.rados.2.smithi204.stdout:1753: finishing copy_from racing read to smithi20457400-10 2024-09-16T05:17:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:29.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:29.761 INFO:tasks.rados.rados.2.smithi204.stdout:1753: finishing copy_from to smithi20457400-10 2024-09-16T05:17:29.761 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 329 (ObjNum 493 snap 152 seq_num 493) dirty exists 2024-09-16T05:17:29.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:29.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:29.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:29.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:30.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:30.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:17:30.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:17:30.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: pgmap v909: 107 pgs: 107 active+clean; 99 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.6 MiB/s wr, 36 op/s 2024-09-16T05:17:30.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:17:30.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:17:30.042 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:17:30.042 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:17:30.042 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: 2.34 deep-scrub starts 2024-09-16T05:17:30.042 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: 2.34 deep-scrub ok 2024-09-16T05:17:30.042 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:29 smithi032 ceph-mon[27388]: osdmap e500: 12 total, 12 up, 12 in 2024-09-16T05:17:30.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:30.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:30.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: pgmap v909: 107 pgs: 107 active+clean; 99 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.6 MiB/s wr, 36 op/s 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: 2.34 deep-scrub starts 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: 2.34 deep-scrub ok 2024-09-16T05:17:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:29 smithi204 ceph-mon[30933]: osdmap e500: 12 total, 12 up, 12 in 2024-09-16T05:17:30.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:17:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:17:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: pgmap v909: 107 pgs: 107 active+clean; 99 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 6.6 MiB/s wr, 36 op/s 2024-09-16T05:17:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:17:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:17:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:17:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:17:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: 2.34 deep-scrub starts 2024-09-16T05:17:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: 2.34 deep-scrub ok 2024-09-16T05:17:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:29 smithi196 ceph-mon[30802]: osdmap e500: 12 total, 12 up, 12 in 2024-09-16T05:17:30.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:30.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:30.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:30.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:30.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:30.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:30.513 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:30.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:30.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:30.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:30.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:30.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:17:30.740 INFO:tasks.rados.rados.2.smithi204.stdout:1753: done (1 left) 2024-09-16T05:17:30.740 INFO:tasks.rados.rados.2.smithi204.stdout:1754: done (0 left) 2024-09-16T05:17:30.740 INFO:tasks.rados.rados.2.smithi204.stdout:1755: write (excl) oid 1 current snap is 186 2024-09-16T05:17:30.741 INFO:tasks.rados.rados.2.smithi204.stdout:1755: seq_num 633 ranges {638831=489916,1841690=677884} 2024-09-16T05:17:30.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:30.746 INFO:tasks.rados.rados.2.smithi204.stdout:1755: writing smithi20457400-1 from 638831 to 1128747 tid 1 2024-09-16T05:17:30.754 INFO:tasks.rados.rados.2.smithi204.stdout:1755: writing smithi20457400-1 from 1841690 to 2519574 tid 2 2024-09-16T05:17:30.754 INFO:tasks.rados.rados.2.smithi204.stdout:1755: finishing write tid 1 to smithi20457400-1 2024-09-16T05:17:30.754 INFO:tasks.rados.rados.2.smithi204.stdout:1756: read oid 13 snap -1 2024-09-16T05:17:30.755 INFO:tasks.rados.rados.2.smithi204.stdout:1756: expect (ObjNum 628 snap 182 seq_num 628) 2024-09-16T05:17:30.755 INFO:tasks.rados.rados.2.smithi204.stdout:1757: read oid 47 snap -1 2024-09-16T05:17:30.755 INFO:tasks.rados.rados.2.smithi204.stdout:1757: expect (ObjNum 611 snap 181 seq_num 611) 2024-09-16T05:17:30.755 INFO:tasks.rados.rados.2.smithi204.stdout:1758: read oid 11 snap -1 2024-09-16T05:17:30.755 INFO:tasks.rados.rados.2.smithi204.stdout:1758: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:17:30.755 INFO:tasks.rados.rados.2.smithi204.stdout:1759: snap_create 2024-09-16T05:17:30.757 INFO:tasks.rados.rados.2.smithi204.stdout:1755: finishing write tid 2 to smithi20457400-1 2024-09-16T05:17:30.765 INFO:tasks.rados.rados.2.smithi204.stdout:1755: finishing write tid 3 to smithi20457400-1 2024-09-16T05:17:30.765 INFO:tasks.rados.rados.2.smithi204.stdout:1755: finishing write tid 4 to smithi20457400-1 2024-09-16T05:17:30.765 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 504 (ObjNum 633 snap 186 seq_num 633) dirty exists 2024-09-16T05:17:30.765 INFO:tasks.rados.rados.2.smithi204.stdout:1755: left oid 1 (ObjNum 633 snap 186 seq_num 633) 2024-09-16T05:17:30.765 INFO:tasks.rados.rados.2.smithi204.stdout:1756: expect (ObjNum 628 snap 182 seq_num 628) 2024-09-16T05:17:30.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:30.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:30.831 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:17:30.869 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:30.891 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:30.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:30 smithi032 ceph-mon[27388]: 1.7 deep-scrub starts 2024-09-16T05:17:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:30 smithi032 ceph-mon[27388]: 1.7 deep-scrub ok 2024-09-16T05:17:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2635686481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:30 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2635686481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:30 smithi032 ceph-mon[27388]: osdmap e501: 12 total, 12 up, 12 in 2024-09-16T05:17:30.936 INFO:tasks.rados.rados.2.smithi204.stdout:1757: expect (ObjNum 611 snap 181 seq_num 611) 2024-09-16T05:17:30.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:31.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:31.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:31.034 INFO:tasks.rados.rados.2.smithi204.stdout:1758: expect (ObjNum 306 snap 85 seq_num 306) 2024-09-16T05:17:31.100 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:31.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:31.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:17:31.142 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:31.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:30 smithi204 ceph-mon[30933]: 1.7 deep-scrub starts 2024-09-16T05:17:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:30 smithi204 ceph-mon[30933]: 1.7 deep-scrub ok 2024-09-16T05:17:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2635686481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:30 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2635686481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:30 smithi204 ceph-mon[30933]: osdmap e501: 12 total, 12 up, 12 in 2024-09-16T05:17:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:30 smithi196 ceph-mon[30802]: 1.7 deep-scrub starts 2024-09-16T05:17:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:30 smithi196 ceph-mon[30802]: 1.7 deep-scrub ok 2024-09-16T05:17:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2635686481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:30 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2635686481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:30 smithi196 ceph-mon[30802]: osdmap e501: 12 total, 12 up, 12 in 2024-09-16T05:17:31.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:31.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:31.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:31.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:31.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:31.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:31.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:31.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:31.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:31.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:31.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:31.647 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:31.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1755: done (4 left) 2024-09-16T05:17:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1756: done (3 left) 2024-09-16T05:17:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1757: done (2 left) 2024-09-16T05:17:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1758: done (1 left) 2024-09-16T05:17:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1759: done (0 left) 2024-09-16T05:17:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1760: copy_from oid 36 from oid 17 current snap is 187 2024-09-16T05:17:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1761: write (excl) oid 14 current snap is 187 2024-09-16T05:17:31.741 INFO:tasks.rados.rados.2.smithi204.stdout:1761: seq_num 634 ranges {655638=565165,1900172=421542} 2024-09-16T05:17:31.747 INFO:tasks.rados.rados.2.smithi204.stdout:1761: writing smithi20457400-14 from 655638 to 1220803 tid 1 2024-09-16T05:17:31.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:31.753 INFO:tasks.rados.rados.2.smithi204.stdout:1761: writing smithi20457400-14 from 1900172 to 2321714 tid 2 2024-09-16T05:17:31.753 INFO:tasks.rados.rados.2.smithi204.stdout:1762: delete oid 40 current snap is 187 2024-09-16T05:17:31.753 INFO:tasks.rados.rados.2.smithi204.stdout:1761: finishing write tid 1 to smithi20457400-14 2024-09-16T05:17:31.753 INFO:tasks.rados.rados.2.smithi204.stdout:1760: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:17:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1762: done (2 left) 2024-09-16T05:17:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1763: read oid 10 snap -1 2024-09-16T05:17:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1763: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:17:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1764: read oid 49 snap 182 2024-09-16T05:17:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1764: expect (ObjNum 617 snap 181 seq_num 617) 2024-09-16T05:17:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1765: write oid 42 current snap is 187 2024-09-16T05:17:31.755 INFO:tasks.rados.rados.2.smithi204.stdout:1765: seq_num 636 ranges {689252=715663,1900783=425211} 2024-09-16T05:17:31.763 INFO:tasks.rados.rados.2.smithi204.stdout:1765: writing smithi20457400-42 from 689252 to 1404915 tid 1 2024-09-16T05:17:31.768 INFO:tasks.rados.rados.2.smithi204.stdout:1765: writing smithi20457400-42 from 1900783 to 2325994 tid 2 2024-09-16T05:17:31.768 INFO:tasks.rados.rados.2.smithi204.stdout:1766: snap_remove snap 183 2024-09-16T05:17:31.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:31.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:31.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:31.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:31.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:31.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:31 smithi032 ceph-mon[27388]: pgmap v912: 107 pgs: 107 active+clean; 90 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.7 MiB/s wr, 31 op/s 2024-09-16T05:17:31.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:31 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:17:31.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:31 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:17:31.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:31 smithi032 ceph-mon[27388]: 2.26 scrub starts 2024-09-16T05:17:31.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:31 smithi032 ceph-mon[27388]: 2.26 scrub ok 2024-09-16T05:17:31.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:31 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:17:31.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:31 smithi032 ceph-mon[27388]: osdmap e502: 12 total, 12 up, 12 in 2024-09-16T05:17:32.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:32.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:32.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:32.119 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:32.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:32.134 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:17:32.135 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:17:32.135 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:17:32.135 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:17:32.135 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:17:32.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:32.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:32.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:32.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:31 smithi204 ceph-mon[30933]: pgmap v912: 107 pgs: 107 active+clean; 90 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.7 MiB/s wr, 31 op/s 2024-09-16T05:17:32.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:31 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:17:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:31 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:17:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:31 smithi204 ceph-mon[30933]: 2.26 scrub starts 2024-09-16T05:17:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:31 smithi204 ceph-mon[30933]: 2.26 scrub ok 2024-09-16T05:17:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:31 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:17:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:31 smithi204 ceph-mon[30933]: osdmap e502: 12 total, 12 up, 12 in 2024-09-16T05:17:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:31 smithi196 ceph-mon[30802]: pgmap v912: 107 pgs: 107 active+clean; 90 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.7 MiB/s wr, 31 op/s 2024-09-16T05:17:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:31 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:17:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:31 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:17:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:31 smithi196 ceph-mon[30802]: 2.26 scrub starts 2024-09-16T05:17:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:31 smithi196 ceph-mon[30802]: 2.26 scrub ok 2024-09-16T05:17:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:31 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:17:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:31 smithi196 ceph-mon[30802]: osdmap e502: 12 total, 12 up, 12 in 2024-09-16T05:17:32.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:32.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:32.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:32.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:32.399 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:32.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:32.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:32.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:32.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:32.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:32.649 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:32.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:32.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:32.742 INFO:tasks.rados.rados.2.smithi204.stdout:1766: done (5 left) 2024-09-16T05:17:32.743 INFO:tasks.rados.rados.2.smithi204.stdout:1767: delete oid 22 current snap is 187 2024-09-16T05:17:32.743 INFO:tasks.rados.rados.2.smithi204.stdout:1761: finishing write tid 2 to smithi20457400-14 2024-09-16T05:17:32.743 INFO:tasks.rados.rados.2.smithi204.stdout:1761: finishing write tid 3 to smithi20457400-14 2024-09-16T05:17:32.743 INFO:tasks.rados.rados.2.smithi204.stdout:1761: finishing write tid 4 to smithi20457400-14 2024-09-16T05:17:32.743 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 857 (ObjNum 634 snap 187 seq_num 634) dirty exists 2024-09-16T05:17:32.743 INFO:tasks.rados.rados.2.smithi204.stdout:1761: left oid 14 (ObjNum 634 snap 187 seq_num 634) 2024-09-16T05:17:32.743 INFO:tasks.rados.rados.2.smithi204.stdout:1760: finishing copy_from to smithi20457400-36 2024-09-16T05:17:32.743 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 691 (ObjNum 615 snap 181 seq_num 615) dirty exists 2024-09-16T05:17:32.744 INFO:tasks.rados.rados.2.smithi204.stdout:1764: expect (ObjNum 617 snap 181 seq_num 617) 2024-09-16T05:17:32.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:32.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:32.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:32.849 INFO:tasks.rados.rados.2.smithi204.stdout:1763: expect (ObjNum 493 snap 152 seq_num 493) 2024-09-16T05:17:32.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:17:32.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:32.900 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:32.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:33.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:33.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:33.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1765: finishing write tid 1 to smithi20457400-42 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1765: finishing write tid 2 to smithi20457400-42 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1765: finishing write tid 3 to smithi20457400-42 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1765: finishing write tid 4 to smithi20457400-42 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 743 (ObjNum 636 snap 187 seq_num 636) dirty exists 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1765: left oid 42 (ObjNum 636 snap 187 seq_num 636) 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1760: done (5 left) 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1761: done (4 left) 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1763: done (3 left) 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1764: done (2 left) 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1765: done (1 left) 2024-09-16T05:17:33.044 INFO:tasks.rados.rados.2.smithi204.stdout:1767: done (0 left) 2024-09-16T05:17:33.045 INFO:tasks.rados.rados.2.smithi204.stdout:1768: copy_from oid 13 from oid 27 current snap is 187 2024-09-16T05:17:33.045 INFO:tasks.rados.rados.2.smithi204.stdout:1769: write (excl) oid 46 current snap is 187 2024-09-16T05:17:33.045 INFO:tasks.rados.rados.2.smithi204.stdout:1769: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2024-09-16T05:17:33.050 INFO:tasks.rados.rados.2.smithi204.stdout:1769: writing smithi20457400-46 from 722866 to 1189027 tid 1 2024-09-16T05:17:33.054 INFO:tasks.rados.rados.2.smithi204.stdout:1769: writing smithi20457400-46 from 1617747 to 2046627 tid 2 2024-09-16T05:17:33.059 INFO:tasks.rados.rados.2.smithi204.stdout:1769: writing smithi20457400-46 from 2587634 to 2722866 tid 3 2024-09-16T05:17:33.059 INFO:tasks.rados.rados.2.smithi204.stdout:1770: copy_from oid 18 from oid 2 current snap is 187 2024-09-16T05:17:33.059 INFO:tasks.rados.rados.2.smithi204.stdout:1768: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1769: finishing write tid 1 to smithi20457400-46 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1768: finishing copy_from to smithi20457400-13 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 312 (ObjNum 620 snap 181 seq_num 620) dirty exists 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1769: finishing write tid 2 to smithi20457400-46 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1768: done (2 left) 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1771: read oid 39 snap -1 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1771: expect deleted 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1772: read oid 8 snap -1 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1772: expect deleted 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1773: write (excl) oid 49 current snap is 187 2024-09-16T05:17:33.060 INFO:tasks.rados.rados.2.smithi204.stdout:1773: seq_num 639 ranges {739673=657763,2076229=663444} 2024-09-16T05:17:33.066 INFO:tasks.rados.rados.2.smithi204.stdout:1773: writing smithi20457400-49 from 739673 to 1397436 tid 1 2024-09-16T05:17:33.074 INFO:tasks.rados.rados.2.smithi204.stdout:1773: writing smithi20457400-49 from 2076229 to 2739673 tid 2 2024-09-16T05:17:33.074 INFO:tasks.rados.rados.2.smithi204.stdout:1774: read oid 23 snap 184 2024-09-16T05:17:33.074 INFO:tasks.rados.rados.2.smithi204.stdout:1774: expect deleted 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1775: rollback oid 48 current snap is 187 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 48 to 186 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1771: done (6 left) 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1772: done (5 left) 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1776: write oid 36 current snap is 187 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1769: finishing write tid 3 to smithi20457400-46 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1769: finishing write tid 4 to smithi20457400-46 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1769: finishing write tid 5 to smithi20457400-46 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 345 (ObjNum 638 snap 187 seq_num 638) dirty exists 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1769: left oid 46 (ObjNum 638 snap 187 seq_num 638) 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1773: finishing write tid 1 to smithi20457400-49 2024-09-16T05:17:33.075 INFO:tasks.rados.rados.2.smithi204.stdout:1776: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-16T05:17:33.081 INFO:tasks.rados.rados.2.smithi204.stdout:1776: writing smithi20457400-36 from 756480 to 1489492 tid 1 2024-09-16T05:17:33.088 INFO:tasks.rados.rados.2.smithi204.stdout:1776: writing smithi20457400-36 from 2134711 to 2567260 tid 2 2024-09-16T05:17:33.088 INFO:tasks.rados.rados.2.smithi204.stdout:1769: done (5 left) 2024-09-16T05:17:33.088 INFO:tasks.rados.rados.2.smithi204.stdout:1777: write oid 47 current snap is 187 2024-09-16T05:17:33.088 INFO:tasks.rados.rados.2.smithi204.stdout:1775: finishing rollback tid 1 to smithi20457400-48 2024-09-16T05:17:33.089 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 97 snap 0 seq_num 2965909488) dirty dne 2024-09-16T05:17:33.089 INFO:tasks.rados.rados.2.smithi204.stdout:1773: finishing write tid 2 to smithi20457400-49 2024-09-16T05:17:33.089 INFO:tasks.rados.rados.2.smithi204.stdout:1773: finishing write tid 3 to smithi20457400-49 2024-09-16T05:17:33.089 INFO:tasks.rados.rados.2.smithi204.stdout:1777: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-16T05:17:33.094 INFO:tasks.rados.rados.2.smithi204.stdout:1777: writing smithi20457400-47 from 773287 to 1181548 tid 1 2024-09-16T05:17:33.101 INFO:tasks.rados.rados.2.smithi204.stdout:1777: writing smithi20457400-47 from 1793193 to 2369400 tid 2 2024-09-16T05:17:33.101 INFO:tasks.rados.rados.2.smithi204.stdout:1774: done (5 left) 2024-09-16T05:17:33.101 INFO:tasks.rados.rados.2.smithi204.stdout:1775: done (4 left) 2024-09-16T05:17:33.101 INFO:tasks.rados.rados.2.smithi204.stdout:1778: write oid 25 current snap is 187 2024-09-16T05:17:33.101 INFO:tasks.rados.rados.2.smithi204.stdout:1778: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-16T05:17:33.108 INFO:tasks.rados.rados.2.smithi204.stdout:1778: writing smithi20457400-25 from 790094 to 1273604 tid 1 2024-09-16T05:17:33.114 INFO:tasks.rados.rados.2.smithi204.stdout:1778: writing smithi20457400-25 from 1851675 to 2287893 tid 2 2024-09-16T05:17:33.114 INFO:tasks.rados.rados.2.smithi204.stdout:1779: copy_from oid 26 from oid 48 current snap is 187 2024-09-16T05:17:33.114 INFO:tasks.rados.rados.2.smithi204.stdout:1773: finishing write tid 4 to smithi20457400-49 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 574 (ObjNum 639 snap 187 seq_num 639) dirty exists 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1773: left oid 49 (ObjNum 639 snap 187 seq_num 639) 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1770: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1770: finishing copy_from to smithi20457400-18 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 511 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1776: finishing write tid 1 to smithi20457400-36 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1770: done (5 left) 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1773: done (4 left) 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1780: read oid 31 snap 182 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1780: expect (ObjNum 619 snap 181 seq_num 619) 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1781: rollback oid 45 current snap is 187 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 182 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1776: finishing write tid 2 to smithi20457400-36 2024-09-16T05:17:33.115 INFO:tasks.rados.rados.2.smithi204.stdout:1776: finishing write tid 3 to smithi20457400-36 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1776: finishing write tid 4 to smithi20457400-36 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 694 (ObjNum 640 snap 187 seq_num 640) dirty exists 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1776: left oid 36 (ObjNum 640 snap 187 seq_num 640) 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1777: finishing write tid 1 to smithi20457400-47 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1777: finishing write tid 2 to smithi20457400-47 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1777: finishing write tid 3 to smithi20457400-47 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1777: finishing write tid 4 to smithi20457400-47 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 696 (ObjNum 641 snap 187 seq_num 641) dirty exists 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1777: left oid 47 (ObjNum 641 snap 187 seq_num 641) 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1778: finishing write tid 1 to smithi20457400-25 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1776: done (5 left) 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1777: done (4 left) 2024-09-16T05:17:33.116 INFO:tasks.rados.rados.2.smithi204.stdout:1782: snap_remove snap 186 2024-09-16T05:17:33.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:33.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:33.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:33.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:33.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:33.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:33.351 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:33.352 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:33.359 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:17:33.359 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:17:33.359 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-09-16T05:17:33.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd out 10 2024-09-16T05:17:33.360 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:33.375 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:33.375 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:33.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:33.397 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:33.397 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:33.399 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:33.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:33.418 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:33.440 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:33.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:33.490 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:33.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:33.511 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:33.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:33.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:33.533 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:33.555 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:33.555 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:33.577 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:33.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:17:33.599 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:33.599 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:33.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:33.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:33.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:33.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:33.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:33.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:33.803 INFO:tasks.rados.rados.2.smithi204.stdout:1782: done (4 left) 2024-09-16T05:17:33.803 INFO:tasks.rados.rados.2.smithi204.stdout:1783: read oid 42 snap -1 2024-09-16T05:17:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:1783: expect (ObjNum 636 snap 187 seq_num 636) 2024-09-16T05:17:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:1784: delete oid 30 current snap is 187 2024-09-16T05:17:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:1779: finishing copy_from to smithi20457400-26 2024-09-16T05:17:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:1779: got expected ENOENT (src dne) 2024-09-16T05:17:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:1779: finishing copy_from racing read to smithi20457400-26 2024-09-16T05:17:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:1778: finishing write tid 2 to smithi20457400-25 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:1778: finishing write tid 3 to smithi20457400-25 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:1781: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:1781: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:1778: finishing write tid 4 to smithi20457400-25 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 702 (ObjNum 642 snap 187 seq_num 642) dirty exists 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:1778: left oid 25 (ObjNum 642 snap 187 seq_num 642) 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:1781: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 349 (ObjNum 476 snap 150 seq_num 476) dirty exists 2024-09-16T05:17:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:1780: expect (ObjNum 619 snap 181 seq_num 619) 2024-09-16T05:17:33.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:33.886 INFO:tasks.rados.rados.2.smithi204.stdout:1783: expect (ObjNum 636 snap 187 seq_num 636) 2024-09-16T05:17:33.897 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:33.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:33.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:33 smithi032 ceph-mon[27388]: pgmap v915: 107 pgs: 107 active+clean; 84 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.1 MiB/s wr, 24 op/s 2024-09-16T05:17:33.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:33 smithi032 ceph-mon[27388]: osdmap e503: 12 total, 12 up, 12 in 2024-09-16T05:17:33.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2704179239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:33.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:34.010 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:34.011 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:34.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:34.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:34.029 INFO:tasks.rados.rados.2.smithi204.stdout:1778: done (5 left) 2024-09-16T05:17:34.030 INFO:tasks.rados.rados.2.smithi204.stdout:1779: done (4 left) 2024-09-16T05:17:34.030 INFO:tasks.rados.rados.2.smithi204.stdout:1780: done (3 left) 2024-09-16T05:17:34.030 INFO:tasks.rados.rados.2.smithi204.stdout:1781: done (2 left) 2024-09-16T05:17:34.030 INFO:tasks.rados.rados.2.smithi204.stdout:1783: done (1 left) 2024-09-16T05:17:34.030 INFO:tasks.rados.rados.2.smithi204.stdout:1784: done (0 left) 2024-09-16T05:17:34.030 INFO:tasks.rados.rados.2.smithi204.stdout:1785: write oid 11 current snap is 187 2024-09-16T05:17:34.030 INFO:tasks.rados.rados.2.smithi204.stdout:1785: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-09-16T05:17:34.035 INFO:tasks.rados.rados.2.smithi204.stdout:1785: writing smithi20457400-11 from 423708 to 1057716 tid 1 2024-09-16T05:17:34.042 INFO:tasks.rados.rados.2.smithi204.stdout:1785: writing smithi20457400-11 from 1852286 to 2292173 tid 2 2024-09-16T05:17:34.045 INFO:tasks.rados.rados.2.smithi204.stdout:1785: writing smithi20457400-11 from 2752172 to 2823708 tid 3 2024-09-16T05:17:34.045 INFO:tasks.rados.rados.2.smithi204.stdout:1786: snap_create 2024-09-16T05:17:34.045 INFO:tasks.rados.rados.2.smithi204.stdout:1785: finishing write tid 1 to smithi20457400-11 2024-09-16T05:17:34.045 INFO:tasks.rados.rados.2.smithi204.stdout:1785: finishing write tid 2 to smithi20457400-11 2024-09-16T05:17:34.046 INFO:tasks.rados.rados.2.smithi204.stdout:1785: finishing write tid 3 to smithi20457400-11 2024-09-16T05:17:34.047 INFO:tasks.rados.rados.2.smithi204.stdout:1785: finishing write tid 4 to smithi20457400-11 2024-09-16T05:17:34.047 INFO:tasks.rados.rados.2.smithi204.stdout:1785: finishing write tid 5 to smithi20457400-11 2024-09-16T05:17:34.047 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 723 (ObjNum 644 snap 187 seq_num 644) dirty exists 2024-09-16T05:17:34.047 INFO:tasks.rados.rados.2.smithi204.stdout:1785: left oid 11 (ObjNum 644 snap 187 seq_num 644) 2024-09-16T05:17:34.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:34.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:34.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:33 smithi204 ceph-mon[30933]: pgmap v915: 107 pgs: 107 active+clean; 84 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.1 MiB/s wr, 24 op/s 2024-09-16T05:17:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:33 smithi204 ceph-mon[30933]: osdmap e503: 12 total, 12 up, 12 in 2024-09-16T05:17:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2704179239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:33 smithi196 ceph-mon[30802]: pgmap v915: 107 pgs: 107 active+clean; 84 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.1 MiB/s wr, 24 op/s 2024-09-16T05:17:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:33 smithi196 ceph-mon[30802]: osdmap e503: 12 total, 12 up, 12 in 2024-09-16T05:17:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2704179239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:34.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:34.267 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:34.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:34.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:34.392 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:34.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:34.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:34.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:34.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:34.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:34.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:34.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:17:34.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:34.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:34.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:34.806 INFO:tasks.rados.rados.2.smithi204.stdout:1785: done (1 left) 2024-09-16T05:17:34.806 INFO:tasks.rados.rados.2.smithi204.stdout:1786: done (0 left) 2024-09-16T05:17:34.806 INFO:tasks.rados.rados.2.smithi204.stdout:1787: read oid 5 snap -1 2024-09-16T05:17:34.807 INFO:tasks.rados.rados.2.smithi204.stdout:1787: expect deleted 2024-09-16T05:17:34.807 INFO:tasks.rados.rados.2.smithi204.stdout:1788: read oid 8 snap -1 2024-09-16T05:17:34.807 INFO:tasks.rados.rados.2.smithi204.stdout:1788: expect deleted 2024-09-16T05:17:34.807 INFO:tasks.rados.rados.2.smithi204.stdout:1789: snap_create 2024-09-16T05:17:34.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:34.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:34.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:34 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:34 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:34 smithi032 ceph-mon[27388]: osdmap e504: 12 total, 12 up, 12 in 2024-09-16T05:17:34.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:35.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:35.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:35.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:35.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:35.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:35.169 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:34 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:34 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:34 smithi204 ceph-mon[30933]: osdmap e504: 12 total, 12 up, 12 in 2024-09-16T05:17:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:35.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:34 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:34 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:34 smithi196 ceph-mon[30802]: osdmap e504: 12 total, 12 up, 12 in 2024-09-16T05:17:35.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:35.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:35.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:35.406 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:35.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:35.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:35.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:35.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:35.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:35.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:35.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:35.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:35.745 INFO:tasks.rados.rados.2.smithi204.stdout:1787: done (2 left) 2024-09-16T05:17:35.745 INFO:tasks.rados.rados.2.smithi204.stdout:1788: done (1 left) 2024-09-16T05:17:35.745 INFO:tasks.rados.rados.2.smithi204.stdout:1789: done (0 left) 2024-09-16T05:17:35.745 INFO:tasks.rados.rados.2.smithi204.stdout:1790: read oid 19 snap -1 2024-09-16T05:17:35.745 INFO:tasks.rados.rados.2.smithi204.stdout:1790: expect (ObjNum 616 snap 181 seq_num 616) 2024-09-16T05:17:35.746 INFO:tasks.rados.rados.2.smithi204.stdout:1791: snap_remove snap 184 2024-09-16T05:17:35.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:35.788 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:35.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:35.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:35.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:35.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:36.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:36.033 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:35 smithi032 ceph-mon[27388]: pgmap v918: 107 pgs: 107 active+clean; 89 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.6 MiB/s wr, 25 op/s 2024-09-16T05:17:36.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:35 smithi032 ceph-mon[27388]: osdmap e505: 12 total, 12 up, 12 in 2024-09-16T05:17:36.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:35 smithi032 ceph-mon[27388]: osdmap e506: 12 total, 12 up, 12 in 2024-09-16T05:17:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:36.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:36.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:36.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:36.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:36.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:35 smithi204 ceph-mon[30933]: pgmap v918: 107 pgs: 107 active+clean; 89 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.6 MiB/s wr, 25 op/s 2024-09-16T05:17:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:35 smithi204 ceph-mon[30933]: osdmap e505: 12 total, 12 up, 12 in 2024-09-16T05:17:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:35 smithi204 ceph-mon[30933]: osdmap e506: 12 total, 12 up, 12 in 2024-09-16T05:17:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:35 smithi196 ceph-mon[30802]: pgmap v918: 107 pgs: 107 active+clean; 89 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5.6 MiB/s wr, 25 op/s 2024-09-16T05:17:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:35 smithi196 ceph-mon[30802]: osdmap e505: 12 total, 12 up, 12 in 2024-09-16T05:17:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:35 smithi196 ceph-mon[30802]: osdmap e506: 12 total, 12 up, 12 in 2024-09-16T05:17:36.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:36.287 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:36.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:36.389 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:36.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:36.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:36.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:36.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:36.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:36.563 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:17:36.329+0000 7f7c3f992640 -1 mon.a@0(leader).osd e506 definitely_dead 0 2024-09-16T05:17:36.618 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:36.618 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:36.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:36.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:36.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:36.748 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:17:36.748 INFO:teuthology.orchestra.run.smithi032.stderr:marked out osd.10. 2024-09-16T05:17:36.752 INFO:tasks.rados.rados.2.smithi204.stdout:1791: done (1 left) 2024-09-16T05:17:36.752 INFO:tasks.rados.rados.2.smithi204.stdout:1792: snap_remove snap 185 2024-09-16T05:17:36.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:36.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:36.824 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3179761114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:36.824 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3369794072' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3179761114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3369794072' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-16T05:17:36.825 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:36 smithi032 ceph-mon[27388]: osdmap e507: 12 total, 12 up, 11 in 2024-09-16T05:17:36.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:36.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:36.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:37.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:37.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:37.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:37.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:37.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:17:37.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:37.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3179761114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:17:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3369794072' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3179761114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3369794072' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-16T05:17:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:36 smithi204 ceph-mon[30933]: osdmap e507: 12 total, 12 up, 11 in 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3179761114' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3369794072' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:17:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:17:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3179761114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3369794072' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-16T05:17:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:36 smithi196 ceph-mon[30802]: osdmap e507: 12 total, 12 up, 11 in 2024-09-16T05:17:37.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:37.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:37.383 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:37.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:37.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:37.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:37.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:37.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:37.631 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:37.653 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:37.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:37.753 INFO:tasks.rados.rados.2.smithi204.stdout:1792: done (1 left) 2024-09-16T05:17:37.753 INFO:tasks.rados.rados.2.smithi204.stdout:1793: delete oid 1 current snap is 189 2024-09-16T05:17:37.754 INFO:tasks.rados.rados.2.smithi204.stdout:1790: expect (ObjNum 616 snap 181 seq_num 616) 2024-09-16T05:17:37.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:37.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:37.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:37.894 INFO:tasks.rados.rados.2.smithi204.stdout:1790: done (1 left) 2024-09-16T05:17:37.895 INFO:tasks.rados.rados.2.smithi204.stdout:1793: done (0 left) 2024-09-16T05:17:37.895 INFO:tasks.rados.rados.2.smithi204.stdout:1794: delete oid 22 current snap is 189 2024-09-16T05:17:37.896 INFO:tasks.rados.rados.2.smithi204.stdout:1794: done (0 left) 2024-09-16T05:17:37.896 INFO:tasks.rados.rados.2.smithi204.stdout:1795: copy_from oid 23 from oid 50 current snap is 189 2024-09-16T05:17:37.896 INFO:tasks.rados.rados.2.smithi204.stdout:1796: snap_remove snap 182 2024-09-16T05:17:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:37.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:37.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:38.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:38.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:38.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:38.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:38.155 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:38.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:38.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:38.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:38.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:38.403 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:38.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:38.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:38.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:38.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:38.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:38.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:38.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:38.761 INFO:tasks.rados.rados.2.smithi204.stdout:1796: done (1 left) 2024-09-16T05:17:38.761 INFO:tasks.rados.rados.2.smithi204.stdout:1797: snap_remove snap 187 2024-09-16T05:17:38.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:38.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:17:38.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:38.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:38.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:38.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:39.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:39.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:38 smithi032 ceph-mon[27388]: osdmap e508: 12 total, 12 up, 11 in 2024-09-16T05:17:39.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:39.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:39.038 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:38 smithi196 ceph-mon[30802]: osdmap e508: 12 total, 12 up, 11 in 2024-09-16T05:17:39.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:39.152 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:39.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:38 smithi204 ceph-mon[30933]: osdmap e508: 12 total, 12 up, 11 in 2024-09-16T05:17:39.234 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:39.234 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:39.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:39.257 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:39.257 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:39.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:39.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:39.278 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:39.278 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:39.279 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:39.302 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:39.302 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:39.323 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:39.324 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:39.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:39.368 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:39.368 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:39.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:39.390 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:39.390 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:39.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:39.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:39.412 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:39.413 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:39.434 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:39.434 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:39.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:39.458 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:39.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:39.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:39.525 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:39.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:39.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:39.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:39.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:39.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:17:39.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:39.762 INFO:tasks.rados.rados.2.smithi204.stdout:1797: done (1 left) 2024-09-16T05:17:39.762 INFO:tasks.rados.rados.2.smithi204.stdout:1798: delete oid 35 current snap is 189 2024-09-16T05:17:39.762 INFO:tasks.rados.rados.2.smithi204.stdout:1795: finishing copy_from racing read to smithi20457400-23 2024-09-16T05:17:39.763 INFO:tasks.rados.rados.2.smithi204.stdout:1795: finishing copy_from to smithi20457400-23 2024-09-16T05:17:39.763 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 364 (ObjNum 592 snap 180 seq_num 592) dirty exists 2024-09-16T05:17:39.767 INFO:tasks.rados.rados.2.smithi204.stdout:1795: done (1 left) 2024-09-16T05:17:39.767 INFO:tasks.rados.rados.2.smithi204.stdout:1798: done (0 left) 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1799: read oid 2 snap 188 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1799: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1800: read oid 15 snap -1 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1800: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1801: read oid 36 snap -1 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1801: expect (ObjNum 640 snap 187 seq_num 640) 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1802: read oid 4 snap -1 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1802: expect (ObjNum 529 snap 159 seq_num 529) 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:1803: rollback oid 18 current snap is 189 2024-09-16T05:17:39.768 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 18 to 188 2024-09-16T05:17:39.769 INFO:tasks.rados.rados.2.smithi204.stdout:1804: read oid 13 snap -1 2024-09-16T05:17:39.769 INFO:tasks.rados.rados.2.smithi204.stdout:1804: expect (ObjNum 620 snap 181 seq_num 620) 2024-09-16T05:17:39.769 INFO:tasks.rados.rados.2.smithi204.stdout:1805: read oid 6 snap 188 2024-09-16T05:17:39.769 INFO:tasks.rados.rados.2.smithi204.stdout:1805: expect (ObjNum 621 snap 181 seq_num 621) 2024-09-16T05:17:39.769 INFO:tasks.rados.rados.2.smithi204.stdout:1806: copy_from oid 9 from oid 19 current snap is 189 2024-09-16T05:17:39.769 INFO:tasks.rados.rados.2.smithi204.stdout:1807: write (excl) oid 38 current snap is 189 2024-09-16T05:17:39.769 INFO:tasks.rados.rados.2.smithi204.stdout:1807: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-09-16T05:17:39.776 INFO:tasks.rados.rados.2.smithi204.stdout:1807: writing smithi20457400-38 from 490936 to 1142293 tid 1 2024-09-16T05:17:39.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:39.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:39.781 INFO:tasks.rados.rados.2.smithi204.stdout:1807: writing smithi20457400-38 from 1686214 to 2133439 tid 2 2024-09-16T05:17:39.786 INFO:tasks.rados.rados.2.smithi204.stdout:1807: writing smithi20457400-38 from 2861864 to 2890936 tid 3 2024-09-16T05:17:39.786 INFO:tasks.rados.rados.2.smithi204.stdout:1808: write oid 34 current snap is 189 2024-09-16T05:17:39.786 INFO:tasks.rados.rados.2.smithi204.stdout:1808: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2024-09-16T05:17:39.792 INFO:tasks.rados.rados.2.smithi204.stdout:1808: writing smithi20457400-34 from 507743 to 1234349 tid 1 2024-09-16T05:17:39.798 INFO:tasks.rados.rados.2.smithi204.stdout:1808: writing smithi20457400-34 from 1744696 to 2335579 tid 2 2024-09-16T05:17:39.802 INFO:tasks.rados.rados.2.smithi204.stdout:1808: writing smithi20457400-34 from 2772934 to 2907743 tid 3 2024-09-16T05:17:39.802 INFO:tasks.rados.rados.2.smithi204.stdout:1809: read oid 43 snap -1 2024-09-16T05:17:39.802 INFO:tasks.rados.rados.2.smithi204.stdout:1809: expect deleted 2024-09-16T05:17:39.802 INFO:tasks.rados.rados.2.smithi204.stdout:1810: rollback oid 29 current snap is 189 2024-09-16T05:17:39.802 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 188 2024-09-16T05:17:39.802 INFO:tasks.rados.rados.2.smithi204.stdout:1803: finishing rollback tid 0 to smithi20457400-18 2024-09-16T05:17:39.802 INFO:tasks.rados.rados.2.smithi204.stdout:1803: finishing rollback tid 1 to smithi20457400-18 2024-09-16T05:17:39.802 INFO:tasks.rados.rados.2.smithi204.stdout:1803: finishing rollback tid 2 to smithi20457400-18 2024-09-16T05:17:39.803 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 514 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:39.803 INFO:tasks.rados.rados.2.smithi204.stdout:1799: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:17:39.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:39.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:39.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:39.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:1800: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:39.970 INFO:tasks.rados.rados.2.smithi204.stdout:1804: expect (ObjNum 620 snap 181 seq_num 620) 2024-09-16T05:17:39.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:39 smithi196 ceph-mon[30802]: pgmap v924: 107 pgs: 4 peering, 103 active+clean; 108 MiB data, 577 MiB used, 983 GiB / 983 GiB avail; 8.3 MiB/s rd, 3.3 MiB/s wr, 23 op/s; 679 KiB/s, 2 objects/s recovering 2024-09-16T05:17:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:39 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:17:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:39 smithi196 ceph-mon[30802]: osdmap e509: 12 total, 12 up, 11 in 2024-09-16T05:17:39.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/432621852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:40.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:40.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:40.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:39 smithi204 ceph-mon[30933]: pgmap v924: 107 pgs: 4 peering, 103 active+clean; 108 MiB data, 577 MiB used, 983 GiB / 983 GiB avail; 8.3 MiB/s rd, 3.3 MiB/s wr, 23 op/s; 679 KiB/s, 2 objects/s recovering 2024-09-16T05:17:40.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:39 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:17:40.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:39 smithi204 ceph-mon[30933]: osdmap e509: 12 total, 12 up, 11 in 2024-09-16T05:17:40.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/432621852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:40.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:39 smithi032 ceph-mon[27388]: pgmap v924: 107 pgs: 4 peering, 103 active+clean; 108 MiB data, 577 MiB used, 983 GiB / 983 GiB avail; 8.3 MiB/s rd, 3.3 MiB/s wr, 23 op/s; 679 KiB/s, 2 objects/s recovering 2024-09-16T05:17:40.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:39 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:17:40.121 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:39 smithi032 ceph-mon[27388]: osdmap e509: 12 total, 12 up, 11 in 2024-09-16T05:17:40.121 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/432621852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:40.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:40.143 INFO:tasks.rados.rados.2.smithi204.stdout:1805: expect (ObjNum 621 snap 181 seq_num 621) 2024-09-16T05:17:40.151 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:40.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:40.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:40.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:40.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:40.286 INFO:tasks.rados.rados.2.smithi204.stdout:1807: finishing write tid 1 to smithi20457400-38 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1807: finishing write tid 2 to smithi20457400-38 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1799: done (11 left) 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1800: done (10 left) 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1803: done (9 left) 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1804: done (8 left) 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1805: done (7 left) 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1811: read oid 3 snap 188 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1811: expect (ObjNum 629 snap 182 seq_num 629) 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1812: rollback oid 22 current snap is 189 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 188 2024-09-16T05:17:40.287 INFO:tasks.rados.rados.2.smithi204.stdout:1801: expect (ObjNum 640 snap 187 seq_num 640) 2024-09-16T05:17:40.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:40.391 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:17:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:17:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:17:40.392 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:17:40.392 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:17:40.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:40.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:40.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:40.509 INFO:tasks.rados.rados.2.smithi204.stdout:1802: expect (ObjNum 529 snap 159 seq_num 529) 2024-09-16T05:17:40.524 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:40.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:40.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:40.637 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:40.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:40.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:40.664 INFO:tasks.rados.rados.2.smithi204.stdout:1806: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1806: finishing copy_from to smithi20457400-9 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 303 (ObjNum 616 snap 181 seq_num 616) dirty exists 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1807: finishing write tid 3 to smithi20457400-38 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1807: finishing write tid 4 to smithi20457400-38 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1807: finishing write tid 5 to smithi20457400-38 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 696 (ObjNum 648 snap 189 seq_num 648) dirty exists 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1807: left oid 38 (ObjNum 648 snap 189 seq_num 648) 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1808: finishing write tid 1 to smithi20457400-34 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1801: done (8 left) 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1802: done (7 left) 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1806: done (6 left) 2024-09-16T05:17:40.665 INFO:tasks.rados.rados.2.smithi204.stdout:1807: done (5 left) 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1813: copy_from oid 6 from oid 48 current snap is 189 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1808: finishing write tid 2 to smithi20457400-34 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1808: finishing write tid 3 to smithi20457400-34 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1808: finishing write tid 4 to smithi20457400-34 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1808: finishing write tid 5 to smithi20457400-34 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 693 (ObjNum 649 snap 189 seq_num 649) dirty exists 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1808: left oid 34 (ObjNum 649 snap 189 seq_num 649) 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1810: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1810: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1810: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 706 (ObjNum 614 snap 181 seq_num 614) dirty exists 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:1812: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:17:40.666 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum -12833014 snap 22023 seq_num 304) dirty dne 2024-09-16T05:17:40.667 INFO:tasks.rados.rados.2.smithi204.stdout:1811: expect (ObjNum 629 snap 182 seq_num 629) 2024-09-16T05:17:40.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:40.776 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:40.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:40.803 INFO:tasks.rados.rados.2.smithi204.stdout:1813: finishing copy_from to smithi20457400-6 2024-09-16T05:17:40.803 INFO:tasks.rados.rados.2.smithi204.stdout:1813: got expected ENOENT (src dne) 2024-09-16T05:17:40.803 INFO:tasks.rados.rados.2.smithi204.stdout:1808: done (5 left) 2024-09-16T05:17:40.803 INFO:tasks.rados.rados.2.smithi204.stdout:1809: done (4 left) 2024-09-16T05:17:40.803 INFO:tasks.rados.rados.2.smithi204.stdout:1810: done (3 left) 2024-09-16T05:17:40.803 INFO:tasks.rados.rados.2.smithi204.stdout:1811: done (2 left) 2024-09-16T05:17:40.804 INFO:tasks.rados.rados.2.smithi204.stdout:1812: done (1 left) 2024-09-16T05:17:40.804 INFO:tasks.rados.rados.2.smithi204.stdout:1814: write oid 4 current snap is 189 2024-09-16T05:17:40.804 INFO:tasks.rados.rados.2.smithi204.stdout:1813: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:17:40.804 INFO:tasks.rados.rados.2.smithi204.stdout:1814: seq_num 650 ranges {524550=401855,1403178=450894,2516710=407840} 2024-09-16T05:17:40.808 INFO:tasks.rados.rados.2.smithi204.stdout:1814: writing smithi20457400-4 from 524550 to 926405 tid 1 2024-09-16T05:17:40.813 INFO:tasks.rados.rados.2.smithi204.stdout:1814: writing smithi20457400-4 from 1403178 to 1854072 tid 2 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:1814: writing smithi20457400-4 from 2516710 to 2924550 tid 3 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:1813: done (1 left) 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:1815: rollback oid 26 current snap is 189 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 188 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:1814: finishing write tid 1 to smithi20457400-4 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:1814: finishing write tid 2 to smithi20457400-4 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:1816: rollback oid 32 current snap is 189 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 188 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:1817: copy_from oid 23 from oid 6 current snap is 189 2024-09-16T05:17:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:1818: write (excl) oid 20 current snap is 189 2024-09-16T05:17:40.821 INFO:tasks.rados.rados.2.smithi204.stdout:1818: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-16T05:17:40.825 INFO:tasks.rados.rados.2.smithi204.stdout:1818: writing smithi20457400-20 from 541357 to 1018461 tid 1 2024-09-16T05:17:40.833 INFO:tasks.rados.rados.2.smithi204.stdout:1818: writing smithi20457400-20 from 1745307 to 2339859 tid 2 2024-09-16T05:17:40.833 INFO:tasks.rados.rados.2.smithi204.stdout:1819: rollback oid 34 current snap is 189 2024-09-16T05:17:40.833 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 188 2024-09-16T05:17:40.833 INFO:tasks.rados.rados.2.smithi204.stdout:1816: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:17:40.833 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 129 snap 0 seq_num 4283268752) dirty dne 2024-09-16T05:17:40.833 INFO:tasks.rados.rados.2.smithi204.stdout:1816: done (5 left) 2024-09-16T05:17:40.833 INFO:tasks.rados.rados.2.smithi204.stdout:1820: copy_from oid 47 from oid 33 current snap is 189 2024-09-16T05:17:40.833 INFO:tasks.rados.rados.2.smithi204.stdout:1814: finishing write tid 3 to smithi20457400-4 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:1814: finishing write tid 4 to smithi20457400-4 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:1814: finishing write tid 5 to smithi20457400-4 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 565 (ObjNum 650 snap 189 seq_num 650) dirty exists 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:1814: left oid 4 (ObjNum 650 snap 189 seq_num 650) 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:1815: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:1815: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:1814: done (5 left) 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:1821: write (excl) oid 22 current snap is 189 2024-09-16T05:17:40.834 INFO:tasks.rados.rados.2.smithi204.stdout:1821: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2024-09-16T05:17:40.842 INFO:tasks.rados.rados.2.smithi204.stdout:1821: writing smithi20457400-22 from 558164 to 1110517 tid 1 2024-09-16T05:17:40.845 INFO:tasks.rados.rados.2.smithi204.stdout:1821: writing smithi20457400-22 from 1803789 to 2258352 tid 2 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:1821: writing smithi20457400-22 from 2855203 to 2958164 tid 3 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:1822: rollback oid 37 current snap is 189 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 188 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:1815: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 867 (ObjNum 626 snap 182 seq_num 626) dirty exists 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:1817: finishing copy_from racing read to smithi20457400-23 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:1818: finishing write tid 1 to smithi20457400-20 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:1817: finishing copy_from to smithi20457400-23 2024-09-16T05:17:40.850 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 348 (ObjNum 621 snap 181 seq_num 621) dirty exists 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1815: done (6 left) 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1817: done (5 left) 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1823: read oid 3 snap 188 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1823: expect (ObjNum 629 snap 182 seq_num 629) 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1824: write (excl) oid 5 current snap is 189 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1818: finishing write tid 2 to smithi20457400-20 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1818: finishing write tid 3 to smithi20457400-20 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1819: finishing rollback tid 0 to smithi20457400-34 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1818: finishing write tid 4 to smithi20457400-20 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 733 (ObjNum 651 snap 189 seq_num 651) dirty exists 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1818: left oid 20 (ObjNum 651 snap 189 seq_num 651) 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1819: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 695 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:17:40.851 INFO:tasks.rados.rados.2.smithi204.stdout:1821: finishing write tid 1 to smithi20457400-22 2024-09-16T05:17:40.852 INFO:tasks.rados.rados.2.smithi204.stdout:1820: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:17:40.852 INFO:tasks.rados.rados.2.smithi204.stdout:1820: finishing copy_from to smithi20457400-47 2024-09-16T05:17:40.852 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 751 (ObjNum 630 snap 182 seq_num 630) dirty exists 2024-09-16T05:17:40.852 INFO:tasks.rados.rados.2.smithi204.stdout:1821: finishing write tid 2 to smithi20457400-22 2024-09-16T05:17:40.852 INFO:tasks.rados.rados.2.smithi204.stdout:1824: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-16T05:17:40.855 INFO:tasks.rados.rados.2.smithi204.stdout:1824: writing smithi20457400-5 from 574971 to 1202573 tid 1 2024-09-16T05:17:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:1824: writing smithi20457400-5 from 1862271 to 2460492 tid 2 2024-09-16T05:17:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:1818: done (6 left) 2024-09-16T05:17:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:1819: done (5 left) 2024-09-16T05:17:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:1820: done (4 left) 2024-09-16T05:17:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:1825: write oid 7 current snap is 189 2024-09-16T05:17:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:1822: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:17:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:17:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:1825: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-09-16T05:17:40.868 INFO:tasks.rados.rados.2.smithi204.stdout:1825: writing smithi20457400-7 from 591778 to 1010982 tid 1 2024-09-16T05:17:40.876 INFO:tasks.rados.rados.2.smithi204.stdout:1825: writing smithi20457400-7 from 1637106 to 2378985 tid 2 2024-09-16T05:17:40.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:40.879 INFO:tasks.rados.rados.2.smithi204.stdout:1825: writing smithi20457400-7 from 2910049 to 2991778 tid 3 2024-09-16T05:17:40.879 INFO:tasks.rados.rados.2.smithi204.stdout:1822: done (4 left) 2024-09-16T05:17:40.879 INFO:tasks.rados.rados.2.smithi204.stdout:1826: snap_create 2024-09-16T05:17:40.879 INFO:tasks.rados.rados.2.smithi204.stdout:1821: finishing write tid 3 to smithi20457400-22 2024-09-16T05:17:40.879 INFO:tasks.rados.rados.2.smithi204.stdout:1821: finishing write tid 4 to smithi20457400-22 2024-09-16T05:17:40.879 INFO:tasks.rados.rados.2.smithi204.stdout:1821: finishing write tid 5 to smithi20457400-22 2024-09-16T05:17:40.879 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 519 (ObjNum 652 snap 189 seq_num 652) dirty exists 2024-09-16T05:17:40.880 INFO:tasks.rados.rados.2.smithi204.stdout:1821: left oid 22 (ObjNum 652 snap 189 seq_num 652) 2024-09-16T05:17:40.880 INFO:tasks.rados.rados.2.smithi204.stdout:1823: expect (ObjNum 629 snap 182 seq_num 629) 2024-09-16T05:17:40.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:40.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:41.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:41.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:41.025 INFO:tasks.rados.rados.2.smithi204.stdout:1824: finishing write tid 1 to smithi20457400-5 2024-09-16T05:17:41.025 INFO:tasks.rados.rados.2.smithi204.stdout:1824: finishing write tid 2 to smithi20457400-5 2024-09-16T05:17:41.025 INFO:tasks.rados.rados.2.smithi204.stdout:1824: finishing write tid 3 to smithi20457400-5 2024-09-16T05:17:41.025 INFO:tasks.rados.rados.2.smithi204.stdout:1824: finishing write tid 4 to smithi20457400-5 2024-09-16T05:17:41.025 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 343 (ObjNum 653 snap 189 seq_num 653) dirty exists 2024-09-16T05:17:41.025 INFO:tasks.rados.rados.2.smithi204.stdout:1824: left oid 5 (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:17:41.025 INFO:tasks.rados.rados.2.smithi204.stdout:1825: finishing write tid 1 to smithi20457400-7 2024-09-16T05:17:41.026 INFO:tasks.rados.rados.2.smithi204.stdout:1825: finishing write tid 2 to smithi20457400-7 2024-09-16T05:17:41.026 INFO:tasks.rados.rados.2.smithi204.stdout:1825: finishing write tid 3 to smithi20457400-7 2024-09-16T05:17:41.026 INFO:tasks.rados.rados.2.smithi204.stdout:1825: finishing write tid 4 to smithi20457400-7 2024-09-16T05:17:41.026 INFO:tasks.rados.rados.2.smithi204.stdout:1825: finishing write tid 5 to smithi20457400-7 2024-09-16T05:17:41.026 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 382 (ObjNum 654 snap 189 seq_num 654) dirty exists 2024-09-16T05:17:41.026 INFO:tasks.rados.rados.2.smithi204.stdout:1825: left oid 7 (ObjNum 654 snap 189 seq_num 654) 2024-09-16T05:17:41.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:41.147 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:41.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:41.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:40 smithi032 ceph-mon[27388]: osdmap e510: 12 total, 12 up, 11 in 2024-09-16T05:17:41.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:40 smithi032 ceph-mon[27388]: osdmap e511: 12 total, 12 up, 11 in 2024-09-16T05:17:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:40 smithi204 ceph-mon[30933]: osdmap e510: 12 total, 12 up, 11 in 2024-09-16T05:17:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:40 smithi204 ceph-mon[30933]: osdmap e511: 12 total, 12 up, 11 in 2024-09-16T05:17:41.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:17:41.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:40 smithi196 ceph-mon[30802]: osdmap e510: 12 total, 12 up, 11 in 2024-09-16T05:17:41.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:40 smithi196 ceph-mon[30802]: osdmap e511: 12 total, 12 up, 11 in 2024-09-16T05:17:41.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:41.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:41.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:41.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:41.413 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:41.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:41.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:41.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:41.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:41.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:41.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:41.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:41.698 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:41.747 INFO:tasks.rados.rados.2.smithi204.stdout:1821: done (4 left) 2024-09-16T05:17:41.747 INFO:tasks.rados.rados.2.smithi204.stdout:1823: done (3 left) 2024-09-16T05:17:41.747 INFO:tasks.rados.rados.2.smithi204.stdout:1824: done (2 left) 2024-09-16T05:17:41.747 INFO:tasks.rados.rados.2.smithi204.stdout:1825: done (1 left) 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1826: done (0 left) 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1827: rollback oid 26 current snap is 190 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 189 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1828: read oid 20 snap -1 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1828: expect (ObjNum 651 snap 189 seq_num 651) 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1829: rollback oid 33 current snap is 190 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 33 to 188 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1830: rollback oid 39 current snap is 190 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 39 to 189 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1831: rollback oid 19 current snap is 190 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 19 to 189 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1832: read oid 35 snap -1 2024-09-16T05:17:41.748 INFO:tasks.rados.rados.2.smithi204.stdout:1832: expect deleted 2024-09-16T05:17:41.749 INFO:tasks.rados.rados.2.smithi204.stdout:1833: delete oid 27 current snap is 190 2024-09-16T05:17:41.752 INFO:tasks.rados.rados.2.smithi204.stdout:1832: done (6 left) 2024-09-16T05:17:41.752 INFO:tasks.rados.rados.2.smithi204.stdout:1833: done (5 left) 2024-09-16T05:17:41.752 INFO:tasks.rados.rados.2.smithi204.stdout:1834: rollback oid 4 current snap is 190 2024-09-16T05:17:41.752 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 188 2024-09-16T05:17:41.752 INFO:tasks.rados.rados.2.smithi204.stdout:1835: copy_from oid 40 from oid 2 current snap is 190 2024-09-16T05:17:41.752 INFO:tasks.rados.rados.2.smithi204.stdout:1836: copy_from oid 46 from oid 48 current snap is 190 2024-09-16T05:17:41.752 INFO:tasks.rados.rados.2.smithi204.stdout:1837: copy_from oid 23 from oid 16 current snap is 190 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:1838: read oid 14 snap -1 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:1838: expect (ObjNum 634 snap 187 seq_num 634) 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:1839: snap_create 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:1827: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:1831: finishing rollback tid 0 to smithi20457400-19 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:1831: finishing rollback tid 1 to smithi20457400-19 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:1831: finishing rollback tid 2 to smithi20457400-19 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 306 (ObjNum 616 snap 181 seq_num 616) dirty exists 2024-09-16T05:17:41.753 INFO:tasks.rados.rados.2.smithi204.stdout:1828: expect (ObjNum 651 snap 189 seq_num 651) 2024-09-16T05:17:41.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:41 smithi032 ceph-mon[27388]: pgmap v927: 107 pgs: 4 peering, 103 active+clean; 107 MiB data, 578 MiB used, 983 GiB / 983 GiB avail; 979 KiB/s rd, 1.0 MiB/s wr, 6 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-16T05:17:41.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:41 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:41.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:41 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:41.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:41 smithi032 ceph-mon[27388]: osdmap e512: 12 total, 12 up, 11 in 2024-09-16T05:17:41.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:41.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:41.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:41.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:41.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:41.923 INFO:tasks.rados.rados.2.smithi204.stdout:1830: finishing rollback tid 1 to smithi20457400-39 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 33 snap 0 seq_num 4265379192) dirty dne 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1829: finishing rollback tid 0 to smithi20457400-33 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1829: finishing rollback tid 1 to smithi20457400-33 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1829: finishing rollback tid 2 to smithi20457400-33 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 754 (ObjNum 630 snap 182 seq_num 630) dirty exists 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1836: finishing copy_from to smithi20457400-46 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1836: got expected ENOENT (src dne) 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1836: finishing copy_from racing read to smithi20457400-46 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1834: finishing rollback tid 0 to smithi20457400-4 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1834: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:1834: finishing rollback tid 2 to smithi20457400-4 2024-09-16T05:17:41.924 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 568 (ObjNum 529 snap 159 seq_num 529) dirty exists 2024-09-16T05:17:41.925 INFO:tasks.rados.rados.2.smithi204.stdout:1827: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:17:41.925 INFO:tasks.rados.rados.2.smithi204.stdout:1827: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:17:41.925 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 871 (ObjNum 626 snap 182 seq_num 626) dirty exists 2024-09-16T05:17:41.925 INFO:tasks.rados.rados.2.smithi204.stdout:1838: expect (ObjNum 634 snap 187 seq_num 634) 2024-09-16T05:17:42.012 INFO:tasks.rados.rados.2.smithi204.stdout:1837: finishing copy_from racing read to smithi20457400-23 2024-09-16T05:17:42.012 INFO:tasks.rados.rados.2.smithi204.stdout:1837: finishing copy_from to smithi20457400-23 2024-09-16T05:17:42.012 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 692 (ObjNum 625 snap 182 seq_num 625) dirty exists 2024-09-16T05:17:42.012 INFO:tasks.rados.rados.2.smithi204.stdout:1835: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:17:42.012 INFO:tasks.rados.rados.2.smithi204.stdout:1835: finishing copy_from to smithi20457400-40 2024-09-16T05:17:42.013 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 511 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:42.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:42.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:42.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:42.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:42.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:17:42.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:42.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:41 smithi204 ceph-mon[30933]: pgmap v927: 107 pgs: 4 peering, 103 active+clean; 107 MiB data, 578 MiB used, 983 GiB / 983 GiB avail; 979 KiB/s rd, 1.0 MiB/s wr, 6 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-16T05:17:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:41 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:41 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:41 smithi204 ceph-mon[30933]: osdmap e512: 12 total, 12 up, 11 in 2024-09-16T05:17:42.195 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:163 2.9 2:9423b7dc:::smithi20457400-41:head [setxattr _header (56) in=7b,truncate 2672280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 418.69962874499998, 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.176412596, 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.196 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 163 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493745+0000", 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1899999999999998e-07 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493747+0000", 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1600000000000001e-06 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493751+0000", 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8260000000000003e-06 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493754+0000", 2024-09-16T05:17:42.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8779999999999998e-06 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494256+0000", 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050281900000000005 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494314+0000", 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7592e-05 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494393+0000", 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8799000000000002e-05 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494718+0000", 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00032548400000000002 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670063+0000", 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17534508600000001 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670109+0000", 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5427999999999999e-05 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670157+0000", 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8105000000000001e-05 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.202 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:164 2.9 2:9423b7dc:::smithi20457400-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 418.69961351000001, 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17655259500000001, 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 164 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:17:42.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7000000000000001e-07 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493762+0000", 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0660000000000002e-06 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493766+0000", 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.732e-06 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493769+0000", 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6799999999999999e-06 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494530+0000", 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076053399999999999 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494606+0000", 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6538999999999993e-05 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670197+0000", 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17559084699999999 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670253+0000", 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5893999999999997e-05 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670312+0000", 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9033000000000002e-05 2024-09-16T05:17:42.207 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:163 2.9 e112/107 ondisk, result = 0)", 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 418.69745651699998, 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17425236199999999, 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.14e-07 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495918+0000", 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5519999999999999e-06 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.669955+0000", 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17403649900000001 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.669969+0000", 2024-09-16T05:17:42.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4064e-05 2024-09-16T05:17:42.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670051+0000", 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2313999999999998e-05 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670059+0000", 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1240000000000005e-06 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670169+0000", 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000109695 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:207 2.6 2:6be67188:::smithi20457400-46:head [write 1768111~402677] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 417.192308486, 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092996950000000002, 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.213 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 207 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001065+0000", 2024-09-16T05:17:42.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001066+0000", 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4900000000000004e-07 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001419+0000", 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035348299999999998 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001430+0000", 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0499000000000001e-05 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001435+0000", 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.22e-06 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001488+0000", 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3588999999999998e-05 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001556+0000", 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7358000000000001e-05 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.001638+0000", 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2319999999999998e-05 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.002547+0000", 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090899100000000005 2024-09-16T05:17:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010283+0000", 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077356680000000002 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010323+0000", 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0389000000000002e-05 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:45.010364+0000", 2024-09-16T05:17:42.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1229000000000001e-05 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:407 2.9 2:96dc17ef:::smithi20457400-10:head [write 1515641~485017] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 404.52189432300003, 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095319989999999993, 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.219 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 407 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671479+0000", 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671480+0000", 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6199999999999996e-07 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671908+0000", 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042804200000000001 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671917+0000", 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6870000000000006e-06 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671923+0000", 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0499999999999997e-06 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.671969+0000", 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5461999999999999e-05 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672023+0000", 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4215000000000002e-05 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672096+0000", 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2574e-05 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.672934+0000", 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00083847200000000005 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680951+0000", 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080164480000000007 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680984+0000", 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3399000000000003e-05 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.681011+0000", 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6787999999999998e-05 2024-09-16T05:17:42.224 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:409 2.9 2:96dc17ef:::smithi20457400-10:head [setxattr _header (56) in=7b,truncate 3378174] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 404.514419374, 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059654601000000002, 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.225 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 409 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7899999999999999e-07 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678957+0000", 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1109999999999999e-06 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678961+0000", 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9639999999999997e-06 2024-09-16T05:17:42.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678969+0000", 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8839999999999994e-06 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.679871+0000", 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090153499999999997 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.679945+0000", 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4518999999999996e-05 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680024+0000", 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9295999999999994e-05 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680263+0000", 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000238994 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738545+0000", 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058281865000000002 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738576+0000", 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0349999999999999e-05 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738608+0000", 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2703999999999997e-05 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.230 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:410 2.9 2:96dc17ef:::smithi20457400-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 404.51439960499999, 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059752424999999998, 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.231 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 410 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9400000000000001e-07 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678975+0000", 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1960000000000001e-06 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678979+0000", 2024-09-16T05:17:42.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7479999999999999e-06 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678982+0000", 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8820000000000001e-06 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680163+0000", 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011814880000000001 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:17:42.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680245+0000", 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1411999999999999e-05 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738635+0000", 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058390240000000003 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738674+0000", 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9119999999999998e-05 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738726+0000", 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2045000000000001e-05 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:409 2.9 e126/107 ondisk, result = 0)", 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 404.51248842799998, 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057729559, 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.236 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3799999999999999e-07 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680887+0000", 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7689999999999999e-06 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736934+0000", 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.056047685999999999 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736942+0000", 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4479999999999997e-06 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738537+0000", 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015953720000000001 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738542+0000", 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7190000000000001e-06 2024-09-16T05:17:42.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738614+0000", 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2427000000000003e-05 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:597 2.9 2:96dc17ef:::smithi20457400-10:head [copy-from ver 0] snapc 13=[13,12,11,f,e,b] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 388.39041685299998, 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010909965000000001, 2024-09-16T05:17:42.240 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 597 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802956+0000", 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802959+0000", 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4540000000000001e-06 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.802992+0000", 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3025000000000003e-05 2024-09-16T05:17:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803026+0000", 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3903999999999999e-05 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803041+0000", 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5299e-05 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803197+0000", 2024-09-16T05:17:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000156389 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803202+0000", 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.758e-06 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805000+0000", 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017979859999999999 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805054+0000", 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4249999999999997e-05 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813866+0000", 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088118999999999992 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:598 2.9 2:96dc17ef:::smithi20457400-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:17:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 388.39029968800003, 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010979131, 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 598 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803074+0000", 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.47e-07 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803078+0000", 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1030000000000004e-06 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803090+0000", 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1875999999999999e-05 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803099+0000", 2024-09-16T05:17:42.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2199999999999992e-06 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803245+0000", 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014687899999999999 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803249+0000", 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0520000000000002e-06 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805228+0000", 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019794729999999998 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805254+0000", 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6016e-05 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813951+0000", 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086966319999999993 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813999+0000", 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8263999999999997e-05 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.814053+0000", 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3668999999999997e-05 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:772 2.9 2:94d33c56:::smithi20457400-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e159)", 2024-09-16T05:17:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 371.54641057100002, 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092047459999999998, 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.252 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 772 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646963+0000", 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7100000000000002e-07 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646975+0000", 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1861999999999999e-05 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646984+0000", 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2550000000000003e-06 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.646991+0000", 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6340000000000003e-06 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.647057+0000", 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6353000000000007e-05 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.647061+0000", 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2720000000000002e-06 2024-09-16T05:17:42.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.648424+0000", 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013634470000000001 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.648472+0000", 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8093999999999997e-05 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.656107+0000", 2024-09-16T05:17:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076354400000000003 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:30.656167+0000", 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9818000000000002e-05 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:979 2.4 2:2ace617f:::smithi20457400-21:head [copy-from ver 40] snapc 28=[28,27,25,24,23,22,21,20,1f,15,f] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 351.195509018, 2024-09-16T05:17:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016837338, 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 979 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997865+0000", 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8899999999999999e-07 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997872+0000", 2024-09-16T05:17:42.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0639999999999996e-06 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997880+0000", 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4039999999999995e-06 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997886+0000", 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4600000000000002e-06 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998030+0000", 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014470800000000001 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998036+0000", 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6620000000000002e-06 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000325+0000", 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022892450000000001 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000434+0000", 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010908699999999999 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006744+0000", 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063091240000000002 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014271+0000", 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0075269869999999997 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014635+0000", 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036412499999999997 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014670+0000", 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5799000000000001e-05 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014701+0000", 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0984e-05 2024-09-16T05:17:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:980 2.4 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 351.19547848299999, 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013613798, 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.264 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 980 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997899+0000", 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7319999999999998e-06 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997902+0000", 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.819e-06 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997913+0000", 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.083e-05 2024-09-16T05:17:42.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997919+0000", 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5570000000000003e-06 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998065+0000", 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000146012 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:17:42.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998069+0000", 2024-09-16T05:17:42.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:42.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:41 smithi196 ceph-mon[30802]: pgmap v927: 107 pgs: 4 peering, 103 active+clean; 107 MiB data, 578 MiB used, 983 GiB / 983 GiB avail; 979 KiB/s rd, 1.0 MiB/s wr, 6 op/s; 1.0 MiB/s, 5 objects/s recovering 2024-09-16T05:17:42.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:41 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:42.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:41 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:42.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:41 smithi196 ceph-mon[30802]: osdmap e512: 12 total, 12 up, 11 in 2024-09-16T05:17:42.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8550000000000001e-06 2024-09-16T05:17:42.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000716+0000", 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026463070000000001 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000762+0000", 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6443999999999998e-05 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011363+0000", 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010600559000000001 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011432+0000", 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8955000000000006e-05 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011508+0000", 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6728000000000001e-05 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.271 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1004 2.4 2:2ace617f:::smithi20457400-21:head [copy-from ver 113] snapc 29=[29,28,27,25,24,23,22,21,20,1f,15] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 349.44125471500001, 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012670931, 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.272 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1004 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752120+0000", 2024-09-16T05:17:42.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.996e-06 2024-09-16T05:17:42.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752132+0000", 2024-09-16T05:17:42.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1389000000000001e-05 2024-09-16T05:17:42.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752150+0000", 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8533999999999999e-05 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752159+0000", 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9959999999999999e-06 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752236+0000", 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6211000000000006e-05 2024-09-16T05:17:42.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752247+0000", 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1127e-05 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753383+0000", 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011362830000000001 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753447+0000", 2024-09-16T05:17:42.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4009000000000003e-05 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.761264+0000", 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078170510000000002 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764658+0000", 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033938449999999999 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764722+0000", 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4177999999999994e-05 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764759+0000", 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7363000000000001e-05 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764789+0000", 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9949000000000001e-05 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.280 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1005 2.4 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e181)", 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 349.44119030899998, 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011596623, 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.281 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1005 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0600000000000001e-07 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752185+0000", 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2419999999999999e-06 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752190+0000", 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7049999999999998e-06 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752193+0000", 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2739999999999999e-06 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752304+0000", 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110301 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752307+0000", 2024-09-16T05:17:42.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6559999999999998e-06 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753639+0000", 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001331201 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753690+0000", 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1826999999999997e-05 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763643+0000", 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0099530030000000002 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763709+0000", 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5469999999999995e-05 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763779+0000", 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0537999999999994e-05 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.286 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1499 2.9 2:94d33c56:::smithi20457400-8:head [copy-from ver 285] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 312.50314650600001, 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016013924999999998, 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.287 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1499 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690228+0000", 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6419999999999999e-06 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690237+0000", 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8929999999999994e-06 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690270+0000", 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2679999999999999e-05 2024-09-16T05:17:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690281+0000", 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1446e-05 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690342+0000", 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0681999999999999e-05 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690408+0000", 2024-09-16T05:17:42.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6173000000000005e-05 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.698590+0000", 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081814639999999994 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.704358+0000", 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057679389999999997 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706113+0000", 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001755097 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706183+0000", 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9991000000000001e-05 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706241+0000", 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7917999999999997e-05 2024-09-16T05:17:42.292 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1500 2.9 2:94d33c56:::smithi20457400-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 312.50306671800001, 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012373189, 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.293 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1500 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690307+0000", 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7599999999999997e-07 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690310+0000", 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2610000000000002e-06 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690337+0000", 2024-09-16T05:17:42.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6733999999999998e-05 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690344+0000", 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2779999999999996e-06 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690524+0000", 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017976900000000001 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:17:42.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690560+0000", 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5762000000000001e-05 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702528+0000", 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011967976 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702599+0000", 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1429999999999999e-05 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702680+0000", 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0402999999999996e-05 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:611 2.15 e366/273 2:a94680a3:::smithi20457400-13:66 v 366'446, mlcod=362'442)", 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 146.47654227500001, 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119602217, 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8099999999999999e-07 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716837+0000", 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4909999999999996e-06 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835914+0000", 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119077768 2024-09-16T05:17:42.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835922+0000", 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.994e-06 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835962+0000", 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9230000000000002e-05 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835979+0000", 2024-09-16T05:17:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.787e-05 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.836402+0000", 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042242199999999999 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.836433+0000", 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1260999999999998e-05 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2874 2.35 2:ad7afe2b:::smithi20457400-38:head [write 490870~618799,stat] snapc 7c=[7c,7b,7a,78,77,76,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:19.716435+0000", 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 142.476937865, 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010139234, 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2874 2024-09-16T05:17:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.716435+0000", 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.716435+0000", 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.716437+0000", 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.807e-06 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.717202+0000", 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076510200000000003 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.717215+0000", 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3181e-05 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.717223+0000", 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2910000000000002e-06 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.717299+0000", 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6737999999999996e-05 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.717304+0000", 2024-09-16T05:17:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7419999999999997e-06 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.721601+0000", 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042963860000000001 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.721705+0000", 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010462 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:17:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.722193+0000", 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048818899999999998 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.724925+0000", 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027313659999999998 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.726461+0000", 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001536773 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.726510+0000", 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8687000000000003e-05 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:19.726574+0000", 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4351999999999996e-05 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4549 2.35 2:ad7afe2b:::smithi20457400-38:head [write 490936~651357,stat] snapc be=[be] ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:39.774633+0000", 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 2.4187397370000001, 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0093894029999999993, 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4549 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.774633+0000", 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.774633+0000", 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.774635+0000", 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.28e-06 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.775548+0000", 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091372700000000003 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.775560+0000", 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.185e-05 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.775567+0000", 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8870000000000003e-06 2024-09-16T05:17:42.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.775628+0000", 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0538000000000002e-05 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.775635+0000", 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4399999999999999e-06 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.777673+0000", 2024-09-16T05:17:42.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002037517 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.777762+0000", 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9845999999999993e-05 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.777877+0000", 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011493999999999999 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.779798+0000", 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001920281 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.783910+0000", 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041127189999999999 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.783971+0000", 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0754000000000001e-05 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:17:42.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:17:42.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:39.784023+0000", 2024-09-16T05:17:42.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1623999999999999e-05 2024-09-16T05:17:42.316 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.316 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.316 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.316 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:17:42.316 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:17:42.316 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:17:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:42.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:42.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:42.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:42.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:42.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:42.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:42.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:42.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:42.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:42.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:42.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:42.745 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1827: done (10 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1828: done (9 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1829: done (8 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1830: done (7 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1831: done (6 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1834: done (5 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1835: done (4 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1836: done (3 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1837: done (2 left) 2024-09-16T05:17:42.749 INFO:tasks.rados.rados.2.smithi204.stdout:1838: done (1 left) 2024-09-16T05:17:42.750 INFO:tasks.rados.rados.2.smithi204.stdout:1839: done (0 left) 2024-09-16T05:17:42.750 INFO:tasks.rados.rados.2.smithi204.stdout:1840: rollback oid 16 current snap is 191 2024-09-16T05:17:42.750 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 16 to 189 2024-09-16T05:17:42.750 INFO:tasks.rados.rados.2.smithi204.stdout:1841: write oid 49 current snap is 191 2024-09-16T05:17:42.750 INFO:tasks.rados.rados.2.smithi204.stdout:1841: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-09-16T05:17:42.754 INFO:tasks.rados.rados.2.smithi204.stdout:1841: writing smithi20457400-49 from 625392 to 1195094 tid 1 2024-09-16T05:17:42.762 INFO:tasks.rados.rados.2.smithi204.stdout:1841: writing smithi20457400-49 from 1637717 to 2383265 tid 2 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1841: writing smithi20457400-49 from 2848542 to 3025392 tid 3 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1842: rollback oid 37 current snap is 191 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 188 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1840: finishing rollback tid 0 to smithi20457400-16 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1840: finishing rollback tid 1 to smithi20457400-16 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1840: finishing rollback tid 2 to smithi20457400-16 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 699 (ObjNum 625 snap 182 seq_num 625) dirty exists 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1841: finishing write tid 1 to smithi20457400-49 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1841: finishing write tid 2 to smithi20457400-49 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1840: done (2 left) 2024-09-16T05:17:42.766 INFO:tasks.rados.rados.2.smithi204.stdout:1843: snap_remove snap 190 2024-09-16T05:17:42.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:42.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:42.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/54759058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:17:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:42 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/54759058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:17:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:42 smithi032 ceph-mon[27388]: osdmap e513: 12 total, 12 up, 11 in 2024-09-16T05:17:42.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:42.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:42.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:43.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:43.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:43.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:43.122 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:17:43.122 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:17:43.122 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T05:17:43.123 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:17:43.123 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:17:43.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-16T05:17:43.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/54759058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:17:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:42 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/54759058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:17:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:42 smithi204 ceph-mon[30933]: osdmap e513: 12 total, 12 up, 11 in 2024-09-16T05:17:43.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:43.211 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:43.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/54759058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:17:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:42 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/54759058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:17:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:42 smithi196 ceph-mon[30802]: osdmap e513: 12 total, 12 up, 11 in 2024-09-16T05:17:43.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:17:43.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:43.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:43.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:43.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:43.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:43.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:43.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:43.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:43.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:43.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:43.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:43.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:43.771 INFO:tasks.rados.rados.2.smithi204.stdout:1843: done (2 left) 2024-09-16T05:17:43.771 INFO:tasks.rados.rados.2.smithi204.stdout:1844: write oid 28 current snap is 191 2024-09-16T05:17:43.771 INFO:tasks.rados.rados.2.smithi204.stdout:1844: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-16T05:17:43.778 INFO:tasks.rados.rados.2.smithi204.stdout:1844: writing smithi20457400-28 from 642199 to 1287150 tid 1 2024-09-16T05:17:43.783 INFO:tasks.rados.rados.2.smithi204.stdout:1844: writing smithi20457400-28 from 1696199 to 2301758 tid 2 2024-09-16T05:17:43.788 INFO:tasks.rados.rados.2.smithi204.stdout:1844: writing smithi20457400-28 from 2992318 to 3042199 tid 3 2024-09-16T05:17:43.788 INFO:tasks.rados.rados.2.smithi204.stdout:1845: snap_create 2024-09-16T05:17:43.788 INFO:tasks.rados.rados.2.smithi204.stdout:1842: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:17:43.788 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 65 snap 0 seq_num 4282416864) dirty dne 2024-09-16T05:17:43.789 INFO:tasks.rados.rados.2.smithi204.stdout:1841: finishing write tid 3 to smithi20457400-49 2024-09-16T05:17:43.789 INFO:tasks.rados.rados.2.smithi204.stdout:1841: finishing write tid 4 to smithi20457400-49 2024-09-16T05:17:43.789 INFO:tasks.rados.rados.2.smithi204.stdout:1841: finishing write tid 5 to smithi20457400-49 2024-09-16T05:17:43.789 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 578 (ObjNum 656 snap 191 seq_num 656) dirty exists 2024-09-16T05:17:43.789 INFO:tasks.rados.rados.2.smithi204.stdout:1841: left oid 49 (ObjNum 656 snap 191 seq_num 656) 2024-09-16T05:17:43.789 INFO:tasks.rados.rados.2.smithi204.stdout:1844: finishing write tid 1 to smithi20457400-28 2024-09-16T05:17:43.789 INFO:tasks.rados.rados.2.smithi204.stdout:1844: finishing write tid 2 to smithi20457400-28 2024-09-16T05:17:43.790 INFO:tasks.rados.rados.2.smithi204.stdout:1844: finishing write tid 3 to smithi20457400-28 2024-09-16T05:17:43.790 INFO:tasks.rados.rados.2.smithi204.stdout:1844: finishing write tid 4 to smithi20457400-28 2024-09-16T05:17:43.790 INFO:tasks.rados.rados.2.smithi204.stdout:1844: finishing write tid 5 to smithi20457400-28 2024-09-16T05:17:43.790 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 695 (ObjNum 657 snap 191 seq_num 657) dirty exists 2024-09-16T05:17:43.790 INFO:tasks.rados.rados.2.smithi204.stdout:1844: left oid 28 (ObjNum 657 snap 191 seq_num 657) 2024-09-16T05:17:43.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:43.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:43.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:43.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:43.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:43.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:43.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:44.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:43 smithi032 ceph-mon[27388]: pgmap v930: 107 pgs: 1 peering, 106 active+clean; 110 MiB data, 583 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 20 op/s; 2.3 MiB/s, 27 objects/s recovering 2024-09-16T05:17:44.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:43 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:17:44.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:43 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:17:44.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:43 smithi032 ceph-mon[27388]: 2.51 scrub starts 2024-09-16T05:17:44.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:43 smithi032 ceph-mon[27388]: 2.51 scrub ok 2024-09-16T05:17:44.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:43 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:17:44.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:43 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:17:44.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:43 smithi032 ceph-mon[27388]: osdmap e514: 12 total, 12 up, 11 in 2024-09-16T05:17:44.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:44.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:44.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:43 smithi204 ceph-mon[30933]: pgmap v930: 107 pgs: 1 peering, 106 active+clean; 110 MiB data, 583 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 20 op/s; 2.3 MiB/s, 27 objects/s recovering 2024-09-16T05:17:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:43 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:17:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:43 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:17:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:43 smithi204 ceph-mon[30933]: 2.51 scrub starts 2024-09-16T05:17:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:43 smithi204 ceph-mon[30933]: 2.51 scrub ok 2024-09-16T05:17:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:43 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:17:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:43 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:17:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:43 smithi204 ceph-mon[30933]: osdmap e514: 12 total, 12 up, 11 in 2024-09-16T05:17:44.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:44.223 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:44.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:43 smithi196 ceph-mon[30802]: pgmap v930: 107 pgs: 1 peering, 106 active+clean; 110 MiB data, 583 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 20 op/s; 2.3 MiB/s, 27 objects/s recovering 2024-09-16T05:17:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:43 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:17:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:43 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:17:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:43 smithi196 ceph-mon[30802]: 2.51 scrub starts 2024-09-16T05:17:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:43 smithi196 ceph-mon[30802]: 2.51 scrub ok 2024-09-16T05:17:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:43 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:17:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:43 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:17:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:43 smithi196 ceph-mon[30802]: osdmap e514: 12 total, 12 up, 11 in 2024-09-16T05:17:44.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:44.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:44.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:44.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:44.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:44.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:44.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:44.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:44.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:44.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:44.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:44.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:44.782 INFO:tasks.rados.rados.2.smithi204.stdout:1841: done (3 left) 2024-09-16T05:17:44.782 INFO:tasks.rados.rados.2.smithi204.stdout:1842: done (2 left) 2024-09-16T05:17:44.783 INFO:tasks.rados.rados.2.smithi204.stdout:1844: done (1 left) 2024-09-16T05:17:44.783 INFO:tasks.rados.rados.2.smithi204.stdout:1845: done (0 left) 2024-09-16T05:17:44.783 INFO:tasks.rados.rados.2.smithi204.stdout:1846: rollback oid 24 current snap is 192 2024-09-16T05:17:44.783 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 188 2024-09-16T05:17:44.783 INFO:tasks.rados.rados.2.smithi204.stdout:1847: copy_from oid 16 from oid 15 current snap is 192 2024-09-16T05:17:44.783 INFO:tasks.rados.rados.2.smithi204.stdout:1848: copy_from oid 39 from oid 46 current snap is 192 2024-09-16T05:17:44.783 INFO:tasks.rados.rados.2.smithi204.stdout:1849: delete oid 26 current snap is 192 2024-09-16T05:17:44.790 INFO:tasks.rados.rados.2.smithi204.stdout:1846: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:17:44.790 INFO:tasks.rados.rados.2.smithi204.stdout:1846: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:17:44.790 INFO:tasks.rados.rados.2.smithi204.stdout:1846: finishing rollback tid 2 to smithi20457400-24 2024-09-16T05:17:44.791 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 523 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:44.798 INFO:tasks.rados.rados.2.smithi204.stdout:1848: finishing copy_from racing read to smithi20457400-39 2024-09-16T05:17:44.798 INFO:tasks.rados.rados.2.smithi204.stdout:1846: done (3 left) 2024-09-16T05:17:44.798 INFO:tasks.rados.rados.2.smithi204.stdout:1849: done (2 left) 2024-09-16T05:17:44.798 INFO:tasks.rados.rados.2.smithi204.stdout:1850: delete oid 3 current snap is 192 2024-09-16T05:17:44.799 INFO:tasks.rados.rados.2.smithi204.stdout:1847: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:17:44.801 INFO:tasks.rados.rados.2.smithi204.stdout:1848: finishing copy_from to smithi20457400-39 2024-09-16T05:17:44.802 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 345 (ObjNum 638 snap 187 seq_num 638) dirty exists 2024-09-16T05:17:44.802 INFO:tasks.rados.rados.2.smithi204.stdout:1847: finishing copy_from to smithi20457400-16 2024-09-16T05:17:44.802 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 686 (ObjNum 596 snap 180 seq_num 596) dirty exists 2024-09-16T05:17:44.803 INFO:tasks.rados.rados.2.smithi204.stdout:1847: done (2 left) 2024-09-16T05:17:44.803 INFO:tasks.rados.rados.2.smithi204.stdout:1848: done (1 left) 2024-09-16T05:17:44.803 INFO:tasks.rados.rados.2.smithi204.stdout:1850: done (0 left) 2024-09-16T05:17:44.804 INFO:tasks.rados.rados.2.smithi204.stdout:1851: delete oid 27 current snap is 192 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:1851: done (0 left) 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:1852: copy_from oid 33 from oid 28 current snap is 192 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:1853: read oid 7 snap 191 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:1853: expect (ObjNum 654 snap 189 seq_num 654) 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:1854: copy_from oid 23 from oid 46 current snap is 192 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:1855: read oid 47 snap -1 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:1855: expect (ObjNum 630 snap 182 seq_num 630) 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:1856: rollback oid 22 current snap is 192 2024-09-16T05:17:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 191 2024-09-16T05:17:44.807 INFO:tasks.rados.rados.2.smithi204.stdout:1857: read oid 5 snap -1 2024-09-16T05:17:44.807 INFO:tasks.rados.rados.2.smithi204.stdout:1857: expect (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:17:44.807 INFO:tasks.rados.rados.2.smithi204.stdout:1858: write (excl) oid 36 current snap is 192 2024-09-16T05:17:44.807 INFO:tasks.rados.rados.2.smithi204.stdout:1858: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-09-16T05:17:44.812 INFO:tasks.rados.rados.2.smithi204.stdout:1858: writing smithi20457400-36 from 709427 to 1255374 tid 1 2024-09-16T05:17:44.819 INFO:tasks.rados.rados.2.smithi204.stdout:1858: writing smithi20457400-36 from 1813774 to 2426671 tid 2 2024-09-16T05:17:44.823 INFO:tasks.rados.rados.2.smithi204.stdout:1858: writing smithi20457400-36 from 2985657 to 3109427 tid 3 2024-09-16T05:17:44.823 INFO:tasks.rados.rados.2.smithi204.stdout:1859: delete oid 32 current snap is 192 2024-09-16T05:17:44.823 INFO:tasks.rados.rados.2.smithi204.stdout:1856: finishing rollback tid 0 to smithi20457400-22 2024-09-16T05:17:44.824 INFO:tasks.rados.rados.2.smithi204.stdout:1856: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:17:44.824 INFO:tasks.rados.rados.2.smithi204.stdout:1856: finishing rollback tid 2 to smithi20457400-22 2024-09-16T05:17:44.824 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 527 (ObjNum 652 snap 189 seq_num 652) dirty exists 2024-09-16T05:17:44.824 INFO:tasks.rados.rados.2.smithi204.stdout:1855: expect (ObjNum 630 snap 182 seq_num 630) 2024-09-16T05:17:44.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:44.855 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:44.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:44.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:44.909 INFO:tasks.rados.rados.2.smithi204.stdout:1858: finishing write tid 1 to smithi20457400-36 2024-09-16T05:17:44.909 INFO:tasks.rados.rados.2.smithi204.stdout:1852: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:17:44.909 INFO:tasks.rados.rados.2.smithi204.stdout:1853: expect (ObjNum 654 snap 189 seq_num 654) 2024-09-16T05:17:44.934 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:44.939 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:17:44.939 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:17:44.939 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:17:44.939 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:17:44.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:44.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:44.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:45.009 INFO:tasks.rados.rados.2.smithi204.stdout:1854: finishing copy_from racing read to smithi20457400-23 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1854: finishing copy_from to smithi20457400-23 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 345 (ObjNum 638 snap 187 seq_num 638) dirty exists 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1852: finishing copy_from to smithi20457400-33 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 695 (ObjNum 657 snap 191 seq_num 657) dirty exists 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1858: finishing write tid 2 to smithi20457400-36 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1852: done (7 left) 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1853: done (6 left) 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1854: done (5 left) 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1855: done (4 left) 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1856: done (3 left) 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1859: done (2 left) 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1860: write oid 39 current snap is 192 2024-09-16T05:17:45.010 INFO:tasks.rados.rados.2.smithi204.stdout:1858: finishing write tid 3 to smithi20457400-36 2024-09-16T05:17:45.011 INFO:tasks.rados.rados.2.smithi204.stdout:1858: finishing write tid 4 to smithi20457400-36 2024-09-16T05:17:45.011 INFO:tasks.rados.rados.2.smithi204.stdout:1858: finishing write tid 5 to smithi20457400-36 2024-09-16T05:17:45.011 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 698 (ObjNum 661 snap 192 seq_num 661) dirty exists 2024-09-16T05:17:45.011 INFO:tasks.rados.rados.2.smithi204.stdout:1858: left oid 36 (ObjNum 661 snap 192 seq_num 661) 2024-09-16T05:17:45.011 INFO:tasks.rados.rados.2.smithi204.stdout:1860: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-09-16T05:17:45.015 INFO:tasks.rados.rados.2.smithi204.stdout:1860: writing smithi20457400-39 from 743041 to 1155839 tid 1 2024-09-16T05:17:45.022 INFO:tasks.rados.rados.2.smithi204.stdout:1860: writing smithi20457400-39 from 1930738 to 2547304 tid 2 2024-09-16T05:17:45.026 INFO:tasks.rados.rados.2.smithi204.stdout:1860: writing smithi20457400-39 from 3040503 to 3143041 tid 3 2024-09-16T05:17:45.026 INFO:tasks.rados.rados.2.smithi204.stdout:1858: done (2 left) 2024-09-16T05:17:45.027 INFO:tasks.rados.rados.2.smithi204.stdout:1861: read oid 26 snap 188 2024-09-16T05:17:45.027 INFO:tasks.rados.rados.2.smithi204.stdout:1861: expect (ObjNum 626 snap 182 seq_num 626) 2024-09-16T05:17:45.027 INFO:tasks.rados.rados.2.smithi204.stdout:1862: write oid 14 current snap is 192 2024-09-16T05:17:45.027 INFO:tasks.rados.rados.2.smithi204.stdout:1860: finishing write tid 1 to smithi20457400-39 2024-09-16T05:17:45.027 INFO:tasks.rados.rados.2.smithi204.stdout:1860: finishing write tid 2 to smithi20457400-39 2024-09-16T05:17:45.027 INFO:tasks.rados.rados.2.smithi204.stdout:1862: seq_num 664 ranges {759848=488047,1989220=760224} 2024-09-16T05:17:45.033 INFO:tasks.rados.rados.2.smithi204.stdout:1862: writing smithi20457400-14 from 759848 to 1247895 tid 1 2024-09-16T05:17:45.041 INFO:tasks.rados.rados.2.smithi204.stdout:1862: writing smithi20457400-14 from 1989220 to 2749444 tid 2 2024-09-16T05:17:45.042 INFO:tasks.rados.rados.2.smithi204.stdout:1863: delete oid 44 current snap is 192 2024-09-16T05:17:45.042 INFO:tasks.rados.rados.2.smithi204.stdout:1860: finishing write tid 3 to smithi20457400-39 2024-09-16T05:17:45.042 INFO:tasks.rados.rados.2.smithi204.stdout:1860: finishing write tid 4 to smithi20457400-39 2024-09-16T05:17:45.042 INFO:tasks.rados.rados.2.smithi204.stdout:1860: finishing write tid 5 to smithi20457400-39 2024-09-16T05:17:45.042 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 738 (ObjNum 663 snap 192 seq_num 663) dirty exists 2024-09-16T05:17:45.042 INFO:tasks.rados.rados.2.smithi204.stdout:1860: left oid 39 (ObjNum 663 snap 192 seq_num 663) 2024-09-16T05:17:45.042 INFO:tasks.rados.rados.2.smithi204.stdout:1861: expect (ObjNum 626 snap 182 seq_num 626) 2024-09-16T05:17:45.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:45.102 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:45.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:17:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:44 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:17:45.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:44 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:1862: finishing write tid 1 to smithi20457400-14 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:1862: finishing write tid 2 to smithi20457400-14 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:1862: finishing write tid 3 to smithi20457400-14 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:1862: finishing write tid 4 to smithi20457400-14 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 877 (ObjNum 664 snap 192 seq_num 664) dirty exists 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:1862: left oid 14 (ObjNum 664 snap 192 seq_num 664) 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:1860: done (4 left) 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:1861: done (3 left) 2024-09-16T05:17:45.189 INFO:tasks.rados.rados.2.smithi204.stdout:1862: done (2 left) 2024-09-16T05:17:45.190 INFO:tasks.rados.rados.2.smithi204.stdout:1863: done (1 left) 2024-09-16T05:17:45.190 INFO:tasks.rados.rados.2.smithi204.stdout:1864: write (excl) oid 31 current snap is 192 2024-09-16T05:17:45.190 INFO:tasks.rados.rados.2.smithi204.stdout:1864: seq_num 666 ranges {793462=638545,2106184=763893} 2024-09-16T05:17:45.195 INFO:tasks.rados.rados.2.smithi204.stdout:1864: writing smithi20457400-31 from 793462 to 1432007 tid 1 2024-09-16T05:17:45.203 INFO:tasks.rados.rados.2.smithi204.stdout:1864: writing smithi20457400-31 from 2106184 to 2870077 tid 2 2024-09-16T05:17:45.204 INFO:tasks.rados.rados.2.smithi204.stdout:1865: delete oid 47 current snap is 192 2024-09-16T05:17:45.204 INFO:tasks.rados.rados.2.smithi204.stdout:1857: expect (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:17:45.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:45.227 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:45.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:45.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:17:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:44 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:17:45.273 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:17:45.273 INFO:teuthology.orchestra.run.smithi032.stdout: "success": "" 2024-09-16T05:17:45.273 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:17:45.302 INFO:tasks.rados.rados.2.smithi204.stdout:1864: finishing write tid 1 to smithi20457400-31 2024-09-16T05:17:45.302 INFO:tasks.rados.rados.2.smithi204.stdout:1864: finishing write tid 2 to smithi20457400-31 2024-09-16T05:17:45.302 INFO:tasks.rados.rados.2.smithi204.stdout:1864: finishing write tid 3 to smithi20457400-31 2024-09-16T05:17:45.302 INFO:tasks.rados.rados.2.smithi204.stdout:1864: finishing write tid 4 to smithi20457400-31 2024-09-16T05:17:45.302 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 881 (ObjNum 666 snap 192 seq_num 666) dirty exists 2024-09-16T05:17:45.302 INFO:tasks.rados.rados.2.smithi204.stdout:1864: left oid 31 (ObjNum 666 snap 192 seq_num 666) 2024-09-16T05:17:45.302 INFO:tasks.rados.rados.2.smithi204.stdout:1857: done (2 left) 2024-09-16T05:17:45.302 INFO:tasks.rados.rados.2.smithi204.stdout:1864: done (1 left) 2024-09-16T05:17:45.303 INFO:tasks.rados.rados.2.smithi204.stdout:1865: done (0 left) 2024-09-16T05:17:45.303 INFO:tasks.rados.rados.2.smithi204.stdout:1866: snap_create 2024-09-16T05:17:45.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:45.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:45.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:45.455 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:45.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:45.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:45.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:45.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:45.603 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:45.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:45.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:45.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:45.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:45.747 INFO:tasks.rados.rados.2.smithi204.stdout:1866: done (0 left) 2024-09-16T05:17:45.747 INFO:tasks.rados.rados.2.smithi204.stdout:1867: snap_remove snap 189 2024-09-16T05:17:45.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:45.851 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:45.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:45.886 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:17:45.886 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:17:45.886 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:17:45.886 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: pgmap v933: 107 pgs: 107 active+clean; 115 MiB data, 560 MiB used, 983 GiB / 983 GiB avail; 22 MiB/s rd, 5.2 MiB/s wr, 38 op/s; 2.0 MiB/s, 24 objects/s recovering 2024-09-16T05:17:45.886 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:17:45.886 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:17:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:17:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:17:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:17:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:17:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: osdmap e515: 12 total, 12 up, 11 in 2024-09-16T05:17:45.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:45 smithi032 ceph-mon[27388]: osdmap e516: 12 total, 12 up, 11 in 2024-09-16T05:17:45.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:45.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:45.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:46.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:46.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:17:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:46.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: pgmap v933: 107 pgs: 107 active+clean; 115 MiB data, 560 MiB used, 983 GiB / 983 GiB avail; 22 MiB/s rd, 5.2 MiB/s wr, 38 op/s; 2.0 MiB/s, 24 objects/s recovering 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: osdmap e515: 12 total, 12 up, 11 in 2024-09-16T05:17:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:45 smithi204 ceph-mon[30933]: osdmap e516: 12 total, 12 up, 11 in 2024-09-16T05:17:46.204 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:46.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:46.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:46.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: pgmap v933: 107 pgs: 107 active+clean; 115 MiB data, 560 MiB used, 983 GiB / 983 GiB avail; 22 MiB/s rd, 5.2 MiB/s wr, 38 op/s; 2.0 MiB/s, 24 objects/s recovering 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: osdmap e515: 12 total, 12 up, 11 in 2024-09-16T05:17:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:45 smithi196 ceph-mon[30802]: osdmap e516: 12 total, 12 up, 11 in 2024-09-16T05:17:46.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:46.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:46.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:46.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:17:46.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:46.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:46.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:46.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:46.609 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:46.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:46.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:46.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:46.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:46.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:46.750 INFO:tasks.rados.rados.2.smithi204.stdout:1867: done (0 left) 2024-09-16T05:17:46.750 INFO:tasks.rados.rados.2.smithi204.stdout:1868: snap_create 2024-09-16T05:17:46.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:46.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:46.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:46.891 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:46.906 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:46.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:46 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:17:46.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:46 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:17:46.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:46 smithi032 ceph-mon[27388]: 2.43 scrub starts 2024-09-16T05:17:46.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:46 smithi032 ceph-mon[27388]: 2.43 scrub ok 2024-09-16T05:17:46.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:46 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:17:46.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:46 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:17:46.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:46 smithi032 ceph-mon[27388]: osdmap e517: 12 total, 12 up, 11 in 2024-09-16T05:17:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:46.929 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:46.934 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:46.934 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:46.957 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:46.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:46.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:46.977 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:46.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:46.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:47.000 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:47.000 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:47.022 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:47.045 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:47.046 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:47.067 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:47.067 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:47.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:47.089 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:47.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:47.110 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:47.111 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:47.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:47.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:47.133 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:47.133 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:47.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:47.156 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:46 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:17:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:46 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:17:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:46 smithi204 ceph-mon[30933]: 2.43 scrub starts 2024-09-16T05:17:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:46 smithi204 ceph-mon[30933]: 2.43 scrub ok 2024-09-16T05:17:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:46 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:17:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:46 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:17:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:46 smithi204 ceph-mon[30933]: osdmap e517: 12 total, 12 up, 11 in 2024-09-16T05:17:47.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:47.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:46 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:17:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:46 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:17:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:46 smithi196 ceph-mon[30802]: 2.43 scrub starts 2024-09-16T05:17:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:46 smithi196 ceph-mon[30802]: 2.43 scrub ok 2024-09-16T05:17:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:46 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:17:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:46 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:17:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:46 smithi196 ceph-mon[30802]: osdmap e517: 12 total, 12 up, 11 in 2024-09-16T05:17:47.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:47.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:47.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:47.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:47.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:47.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:47.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:47.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:47.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:47.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:47.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:47.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:47.739 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:47.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:47.750 INFO:tasks.rados.rados.2.smithi204.stdout:1868: done (0 left) 2024-09-16T05:17:47.750 INFO:tasks.rados.rados.2.smithi204.stdout:1869: write oid 12 current snap is 194 2024-09-16T05:17:47.750 INFO:tasks.rados.rados.2.smithi204.stdout:1869: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-09-16T05:17:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:17:47.756 INFO:tasks.rados.rados.2.smithi204.stdout:1869: writing smithi20457400-12 from 427076 to 1216119 tid 1 2024-09-16T05:17:47.764 INFO:tasks.rados.rados.2.smithi204.stdout:1869: writing smithi20457400-12 from 1706795 to 2474357 tid 2 2024-09-16T05:17:47.769 INFO:tasks.rados.rados.2.smithi204.stdout:1869: writing smithi20457400-12 from 2944912 to 3227076 tid 3 2024-09-16T05:17:47.769 INFO:tasks.rados.rados.2.smithi204.stdout:1870: write (excl) oid 47 current snap is 194 2024-09-16T05:17:47.769 INFO:tasks.rados.rados.2.smithi204.stdout:1869: finishing write tid 1 to smithi20457400-12 2024-09-16T05:17:47.769 INFO:tasks.rados.rados.2.smithi204.stdout:1870: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-09-16T05:17:47.773 INFO:tasks.rados.rados.2.smithi204.stdout:1870: writing smithi20457400-47 from 443883 to 908175 tid 1 2024-09-16T05:17:47.779 INFO:tasks.rados.rados.2.smithi204.stdout:1870: writing smithi20457400-47 from 1365277 to 1876497 tid 2 2024-09-16T05:17:47.786 INFO:tasks.rados.rados.2.smithi204.stdout:1870: writing smithi20457400-47 from 2572335 to 3243883 tid 3 2024-09-16T05:17:47.787 INFO:tasks.rados.rados.2.smithi204.stdout:1871: write oid 36 current snap is 194 2024-09-16T05:17:47.787 INFO:tasks.rados.rados.2.smithi204.stdout:1871: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-09-16T05:17:47.793 INFO:tasks.rados.rados.2.smithi204.stdout:1871: writing smithi20457400-36 from 460690 to 1116584 tid 1 2024-09-16T05:17:47.799 INFO:tasks.rados.rados.2.smithi204.stdout:1871: writing smithi20457400-36 from 1540112 to 2311343 tid 2 2024-09-16T05:17:47.805 INFO:tasks.rados.rados.2.smithi204.stdout:1871: writing smithi20457400-36 from 2716111 to 3255889 tid 3 2024-09-16T05:17:47.805 INFO:tasks.rados.rados.2.smithi204.stdout:1872: read oid 33 snap -1 2024-09-16T05:17:47.805 INFO:tasks.rados.rados.2.smithi204.stdout:1872: expect (ObjNum 657 snap 191 seq_num 657) 2024-09-16T05:17:47.805 INFO:tasks.rados.rados.2.smithi204.stdout:1873: write oid 42 current snap is 194 2024-09-16T05:17:47.805 INFO:tasks.rados.rados.2.smithi204.stdout:1869: finishing write tid 2 to smithi20457400-12 2024-09-16T05:17:47.805 INFO:tasks.rados.rados.2.smithi204.stdout:1873: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-09-16T05:17:47.811 INFO:tasks.rados.rados.2.smithi204.stdout:1873: writing smithi20457400-42 from 477497 to 1208640 tid 1 2024-09-16T05:17:47.820 INFO:tasks.rados.rados.2.smithi204.stdout:1873: writing smithi20457400-42 from 1998594 to 2513483 tid 2 2024-09-16T05:17:47.821 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:47.821 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:47.823 INFO:tasks.rados.rados.2.smithi204.stdout:1873: writing smithi20457400-42 from 3143534 to 3277497 tid 3 2024-09-16T05:17:47.823 INFO:tasks.rados.rados.2.smithi204.stdout:1874: copy_from oid 44 from oid 18 current snap is 194 2024-09-16T05:17:47.823 INFO:tasks.rados.rados.2.smithi204.stdout:1869: finishing write tid 3 to smithi20457400-12 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1870: finishing write tid 1 to smithi20457400-47 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1870: finishing write tid 2 to smithi20457400-47 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1870: finishing write tid 3 to smithi20457400-47 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1870: finishing write tid 4 to smithi20457400-47 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1870: finishing write tid 5 to smithi20457400-47 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 756 (ObjNum 669 snap 194 seq_num 669) dirty exists 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1870: left oid 47 (ObjNum 669 snap 194 seq_num 669) 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1871: finishing write tid 1 to smithi20457400-36 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1870: done (5 left) 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1875: write (excl) oid 21 current snap is 194 2024-09-16T05:17:47.824 INFO:tasks.rados.rados.2.smithi204.stdout:1872: expect (ObjNum 657 snap 191 seq_num 657) 2024-09-16T05:17:47.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:47.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:47.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:47.926 INFO:tasks.rados.rados.2.smithi204.stdout:1871: finishing write tid 2 to smithi20457400-36 2024-09-16T05:17:47.926 INFO:tasks.rados.rados.2.smithi204.stdout:1871: finishing write tid 3 to smithi20457400-36 2024-09-16T05:17:47.926 INFO:tasks.rados.rados.2.smithi204.stdout:1871: finishing write tid 4 to smithi20457400-36 2024-09-16T05:17:47.926 INFO:tasks.rados.rados.2.smithi204.stdout:1871: finishing write tid 5 to smithi20457400-36 2024-09-16T05:17:47.927 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 702 (ObjNum 670 snap 194 seq_num 670) dirty exists 2024-09-16T05:17:47.927 INFO:tasks.rados.rados.2.smithi204.stdout:1871: left oid 36 (ObjNum 670 snap 194 seq_num 670) 2024-09-16T05:17:47.927 INFO:tasks.rados.rados.2.smithi204.stdout:1873: finishing write tid 1 to smithi20457400-42 2024-09-16T05:17:47.927 INFO:tasks.rados.rados.2.smithi204.stdout:1873: finishing write tid 2 to smithi20457400-42 2024-09-16T05:17:47.927 INFO:tasks.rados.rados.2.smithi204.stdout:1875: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-16T05:17:47.930 INFO:tasks.rados.rados.2.smithi204.stdout:1875: writing smithi20457400-21 from 494304 to 900696 tid 1 2024-09-16T05:17:47.938 INFO:tasks.rados.rados.2.smithi204.stdout:1875: writing smithi20457400-21 from 1540723 to 2315623 tid 2 2024-09-16T05:17:47.944 INFO:tasks.rados.rados.2.smithi204.stdout:1875: writing smithi20457400-21 from 2770957 to 3294304 tid 3 2024-09-16T05:17:47.944 INFO:tasks.rados.rados.2.smithi204.stdout:1871: done (5 left) 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1872: done (4 left) 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1876: write oid 36 current snap is 194 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1873: finishing write tid 3 to smithi20457400-42 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1873: finishing write tid 4 to smithi20457400-42 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1873: finishing write tid 5 to smithi20457400-42 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 747 (ObjNum 671 snap 194 seq_num 671) dirty exists 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1873: left oid 42 (ObjNum 671 snap 194 seq_num 671) 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1874: finishing copy_from racing read to smithi20457400-44 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1874: finishing copy_from to smithi20457400-44 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 514 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1875: finishing write tid 1 to smithi20457400-21 2024-09-16T05:17:47.945 INFO:tasks.rados.rados.2.smithi204.stdout:1875: finishing write tid 2 to smithi20457400-21 2024-09-16T05:17:47.946 INFO:tasks.rados.rados.2.smithi204.stdout:1876: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-09-16T05:17:47.949 INFO:tasks.rados.rados.2.smithi204.stdout:1876: writing smithi20457400-36 from 511111 to 992752 tid 1 2024-09-16T05:17:47.955 INFO:tasks.rados.rados.2.smithi204.stdout:1876: writing smithi20457400-36 from 1599205 to 2117763 tid 2 2024-09-16T05:17:47.961 INFO:tasks.rados.rados.2.smithi204.stdout:1876: writing smithi20457400-36 from 2682027 to 3171974 tid 3 2024-09-16T05:17:47.961 INFO:tasks.rados.rados.2.smithi204.stdout:1873: done (4 left) 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1874: done (3 left) 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1877: copy_from oid 4 from oid 37 current snap is 194 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1875: finishing write tid 3 to smithi20457400-21 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1875: finishing write tid 4 to smithi20457400-21 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1875: finishing write tid 5 to smithi20457400-21 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 333 (ObjNum 672 snap 194 seq_num 672) dirty exists 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1875: left oid 21 (ObjNum 672 snap 194 seq_num 672) 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1876: finishing write tid 1 to smithi20457400-36 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1876: finishing write tid 2 to smithi20457400-36 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1875: done (3 left) 2024-09-16T05:17:47.962 INFO:tasks.rados.rados.2.smithi204.stdout:1878: snap_create 2024-09-16T05:17:47.963 INFO:tasks.rados.rados.2.smithi204.stdout:1877: finishing copy_from to smithi20457400-4 2024-09-16T05:17:47.963 INFO:tasks.rados.rados.2.smithi204.stdout:1877: got expected ENOENT (src dne) 2024-09-16T05:17:47.963 INFO:tasks.rados.rados.2.smithi204.stdout:1877: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:17:47.963 INFO:tasks.rados.rados.2.smithi204.stdout:1876: finishing write tid 3 to smithi20457400-36 2024-09-16T05:17:47.964 INFO:tasks.rados.rados.2.smithi204.stdout:1876: finishing write tid 4 to smithi20457400-36 2024-09-16T05:17:47.964 INFO:tasks.rados.rados.2.smithi204.stdout:1876: finishing write tid 5 to smithi20457400-36 2024-09-16T05:17:47.964 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 706 (ObjNum 673 snap 194 seq_num 673) dirty exists 2024-09-16T05:17:47.964 INFO:tasks.rados.rados.2.smithi204.stdout:1876: left oid 36 (ObjNum 673 snap 194 seq_num 673) 2024-09-16T05:17:47.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:47.990 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:47.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:48.030 INFO:tasks.rados.rados.2.smithi204.stdout:1869: finishing write tid 4 to smithi20457400-12 2024-09-16T05:17:48.030 INFO:tasks.rados.rados.2.smithi204.stdout:1869: finishing write tid 5 to smithi20457400-12 2024-09-16T05:17:48.030 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 712 (ObjNum 668 snap 194 seq_num 668) dirty exists 2024-09-16T05:17:48.030 INFO:tasks.rados.rados.2.smithi204.stdout:1869: left oid 12 (ObjNum 668 snap 194 seq_num 668) 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: 2.35 scrub starts 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: 2.35 scrub ok 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: pgmap v936: 107 pgs: 107 active+clean; 122 MiB data, 574 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: 2.45 scrub starts 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1364902560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1168118506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:17:48.031 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:48.032 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:47 smithi032 ceph-mon[27388]: osdmap e518: 12 total, 12 up, 11 in 2024-09-16T05:17:48.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:48.115 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:48.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: 2.35 scrub starts 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: 2.35 scrub ok 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: pgmap v936: 107 pgs: 107 active+clean; 122 MiB data, 574 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: 2.45 scrub starts 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1364902560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1168118506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:17:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:48.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:47 smithi204 ceph-mon[30933]: osdmap e518: 12 total, 12 up, 11 in 2024-09-16T05:17:48.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: 2.35 scrub starts 2024-09-16T05:17:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: 2.35 scrub ok 2024-09-16T05:17:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: pgmap v936: 107 pgs: 107 active+clean; 122 MiB data, 574 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 4.9 MiB/s wr, 24 op/s 2024-09-16T05:17:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:17:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:17:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: 2.45 scrub starts 2024-09-16T05:17:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1364902560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1168118506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:17:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:17:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:17:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:47 smithi196 ceph-mon[30802]: osdmap e518: 12 total, 12 up, 11 in 2024-09-16T05:17:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:48.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:48.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:48.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:48.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:48.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:48.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:48.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:48.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:48.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:48.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:48.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:48.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:48.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:48.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:48.802 INFO:tasks.rados.rados.2.smithi204.stdout:1869: done (3 left) 2024-09-16T05:17:48.802 INFO:tasks.rados.rados.2.smithi204.stdout:1876: done (2 left) 2024-09-16T05:17:48.802 INFO:tasks.rados.rados.2.smithi204.stdout:1877: done (1 left) 2024-09-16T05:17:48.802 INFO:tasks.rados.rados.2.smithi204.stdout:1878: done (0 left) 2024-09-16T05:17:48.802 INFO:tasks.rados.rados.2.smithi204.stdout:1879: write oid 28 current snap is 195 2024-09-16T05:17:48.802 INFO:tasks.rados.rados.2.smithi204.stdout:1879: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-09-16T05:17:48.807 INFO:tasks.rados.rados.2.smithi204.stdout:1879: writing smithi20457400-28 from 527918 to 1084808 tid 1 2024-09-16T05:17:48.814 INFO:tasks.rados.rados.2.smithi204.stdout:1879: writing smithi20457400-28 from 1657687 to 2436256 tid 2 2024-09-16T05:17:48.820 INFO:tasks.rados.rados.2.smithi204.stdout:1879: writing smithi20457400-28 from 3225803 to 3327918 tid 3 2024-09-16T05:17:48.820 INFO:tasks.rados.rados.2.smithi204.stdout:1880: read oid 49 snap -1 2024-09-16T05:17:48.820 INFO:tasks.rados.rados.2.smithi204.stdout:1880: expect (ObjNum 656 snap 191 seq_num 656) 2024-09-16T05:17:48.820 INFO:tasks.rados.rados.2.smithi204.stdout:1881: snap_create 2024-09-16T05:17:48.820 INFO:tasks.rados.rados.2.smithi204.stdout:1879: finishing write tid 1 to smithi20457400-28 2024-09-16T05:17:48.820 INFO:tasks.rados.rados.2.smithi204.stdout:1879: finishing write tid 2 to smithi20457400-28 2024-09-16T05:17:48.821 INFO:tasks.rados.rados.2.smithi204.stdout:1879: finishing write tid 3 to smithi20457400-28 2024-09-16T05:17:48.821 INFO:tasks.rados.rados.2.smithi204.stdout:1879: finishing write tid 4 to smithi20457400-28 2024-09-16T05:17:48.821 INFO:tasks.rados.rados.2.smithi204.stdout:1879: finishing write tid 5 to smithi20457400-28 2024-09-16T05:17:48.821 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 699 (ObjNum 674 snap 195 seq_num 674) dirty exists 2024-09-16T05:17:48.821 INFO:tasks.rados.rados.2.smithi204.stdout:1879: left oid 28 (ObjNum 674 snap 195 seq_num 674) 2024-09-16T05:17:48.827 INFO:tasks.rados.rados.2.smithi204.stdout:1880: expect (ObjNum 656 snap 191 seq_num 656) 2024-09-16T05:17:48.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:48.876 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:48.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:48.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:49.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:49.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:49.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:49.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:49.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.45 scrub ok 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 1.a deep-scrub starts 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 1.a deep-scrub ok 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:17:49.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:17:49.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.4c deep-scrub starts 2024-09-16T05:17:49.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:48 smithi032 ceph-mon[27388]: 2.4c deep-scrub ok 2024-09-16T05:17:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.45 scrub ok 2024-09-16T05:17:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 1.a deep-scrub starts 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 1.a deep-scrub ok 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.4c deep-scrub starts 2024-09-16T05:17:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:48 smithi204 ceph-mon[30933]: 2.4c deep-scrub ok 2024-09-16T05:17:49.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:17:49.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.45 scrub ok 2024-09-16T05:17:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 1.a deep-scrub starts 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 1.a deep-scrub ok 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.4c deep-scrub starts 2024-09-16T05:17:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:48 smithi196 ceph-mon[30802]: 2.4c deep-scrub ok 2024-09-16T05:17:49.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:49.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:49.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:49.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:49.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:49.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:49.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:49.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:49.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:49.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:49.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:49.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:49.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:49.799 INFO:tasks.rados.rados.2.smithi204.stdout:1879: done (2 left) 2024-09-16T05:17:49.799 INFO:tasks.rados.rados.2.smithi204.stdout:1880: done (1 left) 2024-09-16T05:17:49.799 INFO:tasks.rados.rados.2.smithi204.stdout:1881: done (0 left) 2024-09-16T05:17:49.799 INFO:tasks.rados.rados.2.smithi204.stdout:1882: read oid 43 snap -1 2024-09-16T05:17:49.799 INFO:tasks.rados.rados.2.smithi204.stdout:1882: expect deleted 2024-09-16T05:17:49.799 INFO:tasks.rados.rados.2.smithi204.stdout:1883: delete oid 23 current snap is 196 2024-09-16T05:17:49.805 INFO:tasks.rados.rados.2.smithi204.stdout:1883: done (1 left) 2024-09-16T05:17:49.805 INFO:tasks.rados.rados.2.smithi204.stdout:1884: delete oid 7 current snap is 196 2024-09-16T05:17:49.807 INFO:tasks.rados.rados.2.smithi204.stdout:1882: done (1 left) 2024-09-16T05:17:49.807 INFO:tasks.rados.rados.2.smithi204.stdout:1884: done (0 left) 2024-09-16T05:17:49.807 INFO:tasks.rados.rados.2.smithi204.stdout:1885: write (excl) oid 44 current snap is 196 2024-09-16T05:17:49.807 INFO:tasks.rados.rados.2.smithi204.stdout:1885: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-09-16T05:17:49.812 INFO:tasks.rados.rados.2.smithi204.stdout:1885: writing smithi20457400-44 from 578339 to 1077329 tid 1 2024-09-16T05:17:49.819 INFO:tasks.rados.rados.2.smithi204.stdout:1885: writing smithi20457400-44 from 1833133 to 2359029 tid 2 2024-09-16T05:17:49.826 INFO:tasks.rados.rados.2.smithi204.stdout:1885: writing smithi20457400-44 from 2791719 to 3378339 tid 3 2024-09-16T05:17:49.826 INFO:tasks.rados.rados.2.smithi204.stdout:1886: copy_from oid 38 from oid 12 current snap is 196 2024-09-16T05:17:49.826 INFO:tasks.rados.rados.2.smithi204.stdout:1885: finishing write tid 1 to smithi20457400-44 2024-09-16T05:17:49.826 INFO:tasks.rados.rados.2.smithi204.stdout:1885: finishing write tid 2 to smithi20457400-44 2024-09-16T05:17:49.826 INFO:tasks.rados.rados.2.smithi204.stdout:1887: write oid 30 current snap is 196 2024-09-16T05:17:49.826 INFO:tasks.rados.rados.2.smithi204.stdout:1887: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-16T05:17:49.832 INFO:tasks.rados.rados.2.smithi204.stdout:1887: writing smithi20457400-30 from 595146 to 1169385 tid 1 2024-09-16T05:17:49.840 INFO:tasks.rados.rados.2.smithi204.stdout:1887: writing smithi20457400-30 from 1891615 to 2561169 tid 2 2024-09-16T05:17:49.845 INFO:tasks.rados.rados.2.smithi204.stdout:1887: writing smithi20457400-30 from 3219142 to 3395146 tid 3 2024-09-16T05:17:49.845 INFO:tasks.rados.rados.2.smithi204.stdout:1888: delete oid 47 current snap is 196 2024-09-16T05:17:49.845 INFO:tasks.rados.rados.2.smithi204.stdout:1885: finishing write tid 3 to smithi20457400-44 2024-09-16T05:17:49.845 INFO:tasks.rados.rados.2.smithi204.stdout:1885: finishing write tid 4 to smithi20457400-44 2024-09-16T05:17:49.845 INFO:tasks.rados.rados.2.smithi204.stdout:1885: finishing write tid 5 to smithi20457400-44 2024-09-16T05:17:49.845 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 518 (ObjNum 677 snap 196 seq_num 677) dirty exists 2024-09-16T05:17:49.846 INFO:tasks.rados.rados.2.smithi204.stdout:1885: left oid 44 (ObjNum 677 snap 196 seq_num 677) 2024-09-16T05:17:49.846 INFO:tasks.rados.rados.2.smithi204.stdout:1886: finishing copy_from racing read to smithi20457400-38 2024-09-16T05:17:49.846 INFO:tasks.rados.rados.2.smithi204.stdout:1887: finishing write tid 1 to smithi20457400-30 2024-09-16T05:17:49.846 INFO:tasks.rados.rados.2.smithi204.stdout:1886: finishing copy_from to smithi20457400-38 2024-09-16T05:17:49.846 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 712 (ObjNum 668 snap 194 seq_num 668) dirty exists 2024-09-16T05:17:49.846 INFO:tasks.rados.rados.2.smithi204.stdout:1887: finishing write tid 2 to smithi20457400-30 2024-09-16T05:17:49.847 INFO:tasks.rados.rados.2.smithi204.stdout:1885: done (3 left) 2024-09-16T05:17:49.847 INFO:tasks.rados.rados.2.smithi204.stdout:1886: done (2 left) 2024-09-16T05:17:49.847 INFO:tasks.rados.rados.2.smithi204.stdout:1888: done (1 left) 2024-09-16T05:17:49.847 INFO:tasks.rados.rados.2.smithi204.stdout:1889: rollback oid 28 current snap is 196 2024-09-16T05:17:49.847 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 28 to 195 2024-09-16T05:17:49.847 INFO:tasks.rados.rados.2.smithi204.stdout:1890: copy_from oid 19 from oid 27 current snap is 196 2024-09-16T05:17:49.847 INFO:tasks.rados.rados.2.smithi204.stdout:1891: rollback oid 40 current snap is 196 2024-09-16T05:17:49.848 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 40 to 193 2024-09-16T05:17:49.848 INFO:tasks.rados.rados.2.smithi204.stdout:1892: write (excl) oid 23 current snap is 196 2024-09-16T05:17:49.848 INFO:tasks.rados.rados.2.smithi204.stdout:1892: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-09-16T05:17:49.854 INFO:tasks.rados.rados.2.smithi204.stdout:1892: writing smithi20457400-23 from 628760 to 1353497 tid 1 2024-09-16T05:17:49.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:49.861 INFO:tasks.rados.rados.2.smithi204.stdout:1892: writing smithi20457400-23 from 1892226 to 2565449 tid 2 2024-09-16T05:17:49.866 INFO:tasks.rados.rados.2.smithi204.stdout:1892: writing smithi20457400-23 from 3157635 to 3428760 tid 3 2024-09-16T05:17:49.866 INFO:tasks.rados.rados.2.smithi204.stdout:1893: read oid 21 snap -1 2024-09-16T05:17:49.866 INFO:tasks.rados.rados.2.smithi204.stdout:1893: expect (ObjNum 672 snap 194 seq_num 672) 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1894: rollback oid 42 current snap is 196 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 194 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1887: finishing write tid 3 to smithi20457400-30 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1887: finishing write tid 4 to smithi20457400-30 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1887: finishing write tid 5 to smithi20457400-30 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 891 (ObjNum 678 snap 196 seq_num 678) dirty exists 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1887: left oid 30 (ObjNum 678 snap 196 seq_num 678) 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1890: finishing copy_from to smithi20457400-19 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1890: got expected ENOENT (src dne) 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1890: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1891: finishing rollback tid 0 to smithi20457400-40 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1887: done (6 left) 2024-09-16T05:17:49.867 INFO:tasks.rados.rados.2.smithi204.stdout:1890: done (5 left) 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1895: read oid 18 snap -1 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1895: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1896: snap_create 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1891: finishing rollback tid 1 to smithi20457400-40 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1891: finishing rollback tid 2 to smithi20457400-40 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 514 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1889: finishing rollback tid 0 to smithi20457400-28 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1889: finishing rollback tid 1 to smithi20457400-28 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1889: finishing rollback tid 2 to smithi20457400-28 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 702 (ObjNum 674 snap 195 seq_num 674) dirty exists 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1892: finishing write tid 1 to smithi20457400-23 2024-09-16T05:17:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:1892: finishing write tid 2 to smithi20457400-23 2024-09-16T05:17:49.869 INFO:tasks.rados.rados.2.smithi204.stdout:1894: finishing rollback tid 0 to smithi20457400-42 2024-09-16T05:17:49.869 INFO:tasks.rados.rados.2.smithi204.stdout:1894: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:17:49.877 INFO:tasks.rados.rados.2.smithi204.stdout:1892: finishing write tid 3 to smithi20457400-23 2024-09-16T05:17:49.877 INFO:tasks.rados.rados.2.smithi204.stdout:1892: finishing write tid 4 to smithi20457400-23 2024-09-16T05:17:49.877 INFO:tasks.rados.rados.2.smithi204.stdout:1892: finishing write tid 5 to smithi20457400-23 2024-09-16T05:17:49.877 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 697 (ObjNum 680 snap 196 seq_num 680) dirty exists 2024-09-16T05:17:49.878 INFO:tasks.rados.rados.2.smithi204.stdout:1892: left oid 23 (ObjNum 680 snap 196 seq_num 680) 2024-09-16T05:17:49.881 INFO:tasks.rados.rados.2.smithi204.stdout:1893: expect (ObjNum 672 snap 194 seq_num 672) 2024-09-16T05:17:49.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:49.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:49.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:50.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:50.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:50.064 INFO:tasks.rados.rados.2.smithi204.stdout:1895: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:17:50.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:50.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: pgmap v939: 107 pgs: 107 active+clean; 129 MiB data, 587 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 20 op/s 2024-09-16T05:17:50.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: osdmap e519: 12 total, 12 up, 11 in 2024-09-16T05:17:50.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:50.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:50.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:17:50.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:17:50.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:17:50.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:17:50.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:17:50.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:49 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:17:50.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:50.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: pgmap v939: 107 pgs: 107 active+clean; 129 MiB data, 587 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 20 op/s 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: osdmap e519: 12 total, 12 up, 11 in 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:17:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:49 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:17:50.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: pgmap v939: 107 pgs: 107 active+clean; 129 MiB data, 587 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 20 op/s 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: osdmap e519: 12 total, 12 up, 11 in 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:17:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:49 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:17:50.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:50.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:50.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:50.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:50.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:50.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:50.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:50.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:50.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:50.570 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:50.570 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:50.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:50.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:50.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:50.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:50.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:17:50.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:50.750 INFO:tasks.rados.rados.2.smithi204.stdout:1889: done (6 left) 2024-09-16T05:17:50.751 INFO:tasks.rados.rados.2.smithi204.stdout:1891: done (5 left) 2024-09-16T05:17:50.751 INFO:tasks.rados.rados.2.smithi204.stdout:1892: done (4 left) 2024-09-16T05:17:50.751 INFO:tasks.rados.rados.2.smithi204.stdout:1893: done (3 left) 2024-09-16T05:17:50.751 INFO:tasks.rados.rados.2.smithi204.stdout:1895: done (2 left) 2024-09-16T05:17:50.751 INFO:tasks.rados.rados.2.smithi204.stdout:1896: done (1 left) 2024-09-16T05:17:50.751 INFO:tasks.rados.rados.2.smithi204.stdout:1897: copy_from oid 18 from oid 22 current snap is 197 2024-09-16T05:17:50.751 INFO:tasks.rados.rados.2.smithi204.stdout:1898: snap_create 2024-09-16T05:17:50.756 INFO:tasks.rados.rados.2.smithi204.stdout:1894: finishing rollback tid 2 to smithi20457400-42 2024-09-16T05:17:50.756 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 750 (ObjNum 671 snap 194 seq_num 671) dirty exists 2024-09-16T05:17:50.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:50.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:50.764 INFO:tasks.rados.rados.2.smithi204.stdout:1897: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:17:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:1897: finishing copy_from to smithi20457400-18 2024-09-16T05:17:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 527 (ObjNum 652 snap 189 seq_num 652) dirty exists 2024-09-16T05:17:50.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:50.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:50.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:50.914 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:17:50.914 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:17:50.914 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-16T05:17:50.914 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:17:50.914 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:17:50.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-16T05:17:50.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:51.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:51.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:51.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:51.131 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:51.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:51.162 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:17:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: 2.2e deep-scrub starts 2024-09-16T05:17:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: 2.2e deep-scrub ok 2024-09-16T05:17:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: osdmap e520: 12 total, 12 up, 11 in 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: 2.4d deep-scrub starts 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: 2.4d deep-scrub ok 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: 2.3b deep-scrub starts 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: 2.3b deep-scrub ok 2024-09-16T05:17:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:50 smithi032 ceph-mon[27388]: osdmap e521: 12 total, 12 up, 11 in 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: 2.2e deep-scrub starts 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: 2.2e deep-scrub ok 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: osdmap e520: 12 total, 12 up, 11 in 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: 2.4d deep-scrub starts 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: 2.4d deep-scrub ok 2024-09-16T05:17:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: 2.3b deep-scrub starts 2024-09-16T05:17:51.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: 2.3b deep-scrub ok 2024-09-16T05:17:51.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:50 smithi204 ceph-mon[30933]: osdmap e521: 12 total, 12 up, 11 in 2024-09-16T05:17:51.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:51.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: 2.2e deep-scrub starts 2024-09-16T05:17:51.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: 2.2e deep-scrub ok 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: osdmap e520: 12 total, 12 up, 11 in 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: 2.4d deep-scrub starts 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: 2.4d deep-scrub ok 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: 2.3b deep-scrub starts 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: 2.3b deep-scrub ok 2024-09-16T05:17:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:50 smithi196 ceph-mon[30802]: osdmap e521: 12 total, 12 up, 11 in 2024-09-16T05:17:51.256 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:51.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:51.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:17:51.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:51.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:51.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:51.423 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:51.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:51.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:51.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:51.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:51.644 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:17:51.644 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "" 2024-09-16T05:17:51.644 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:17:51.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:51.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1894: done (2 left) 2024-09-16T05:17:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1897: done (1 left) 2024-09-16T05:17:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1898: done (0 left) 2024-09-16T05:17:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1899: read oid 45 snap -1 2024-09-16T05:17:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1899: expect (ObjNum 476 snap 150 seq_num 476) 2024-09-16T05:17:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1900: write (excl) oid 30 current snap is 198 2024-09-16T05:17:51.750 INFO:tasks.rados.rados.2.smithi204.stdout:1900: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-09-16T05:17:51.757 INFO:tasks.rados.rados.2.smithi204.stdout:1900: writing smithi20457400-30 from 645567 to 1445553 tid 1 2024-09-16T05:17:51.763 INFO:tasks.rados.rados.2.smithi204.stdout:1900: writing smithi20457400-30 from 1950708 to 2483942 tid 2 2024-09-16T05:17:51.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:51.770 INFO:tasks.rados.rados.2.smithi204.stdout:1900: writing smithi20457400-30 from 3185058 to 3445567 tid 3 2024-09-16T05:17:51.770 INFO:tasks.rados.rados.2.smithi204.stdout:1901: delete oid 47 current snap is 198 2024-09-16T05:17:51.770 INFO:tasks.rados.rados.2.smithi204.stdout:1899: expect (ObjNum 476 snap 150 seq_num 476) 2024-09-16T05:17:51.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:51.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:51.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:1900: finishing write tid 1 to smithi20457400-30 2024-09-16T05:17:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:1900: finishing write tid 2 to smithi20457400-30 2024-09-16T05:17:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:1900: finishing write tid 3 to smithi20457400-30 2024-09-16T05:17:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:1900: finishing write tid 4 to smithi20457400-30 2024-09-16T05:17:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:1900: finishing write tid 5 to smithi20457400-30 2024-09-16T05:17:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 896 (ObjNum 681 snap 198 seq_num 681) dirty exists 2024-09-16T05:17:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:1900: left oid 30 (ObjNum 681 snap 198 seq_num 681) 2024-09-16T05:17:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:1899: done (2 left) 2024-09-16T05:17:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:1900: done (1 left) 2024-09-16T05:17:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:1901: done (0 left) 2024-09-16T05:17:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:1902: read oid 16 snap -1 2024-09-16T05:17:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:1902: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:1903: delete oid 34 current snap is 198 2024-09-16T05:17:51.902 INFO:tasks.rados.rados.2.smithi204.stdout:1903: done (1 left) 2024-09-16T05:17:51.902 INFO:tasks.rados.rados.2.smithi204.stdout:1904: copy_from oid 7 from oid 6 current snap is 198 2024-09-16T05:17:51.902 INFO:tasks.rados.rados.2.smithi204.stdout:1905: copy_from oid 2 from oid 40 current snap is 198 2024-09-16T05:17:51.902 INFO:tasks.rados.rados.2.smithi204.stdout:1906: copy_from oid 39 from oid 38 current snap is 198 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:1907: rollback oid 49 current snap is 198 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 191 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:1908: read oid 18 snap 194 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:1908: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:1909: read oid 36 snap -1 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:1909: expect (ObjNum 673 snap 194 seq_num 673) 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:1910: read oid 44 snap -1 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:1910: expect (ObjNum 677 snap 196 seq_num 677) 2024-09-16T05:17:51.903 INFO:tasks.rados.rados.2.smithi204.stdout:1911: snap_create 2024-09-16T05:17:51.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:17:51.904 INFO:tasks.rados.rados.2.smithi204.stdout:1907: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:17:51.905 INFO:tasks.rados.rados.2.smithi204.stdout:1907: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:17:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:1907: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:17:51.906 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 581 (ObjNum 656 snap 191 seq_num 656) dirty exists 2024-09-16T05:17:51.910 INFO:tasks.rados.rados.2.smithi204.stdout:1902: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:51.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:51.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:51.979 INFO:tasks.rados.rados.2.smithi204.stdout:1909: expect (ObjNum 673 snap 194 seq_num 673) 2024-09-16T05:17:51.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:52.006 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:52.006 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:52.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:52.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:52.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:52.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:52.047 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:52.048 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:52.068 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:52.069 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:52.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:51 smithi032 ceph-mon[27388]: pgmap v942: 107 pgs: 107 active+clean; 137 MiB data, 594 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 3.5 MiB/s wr, 15 op/s 2024-09-16T05:17:52.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:51 smithi032 ceph-mon[27388]: 2.50 scrub starts 2024-09-16T05:17:52.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:51 smithi032 ceph-mon[27388]: 2.50 scrub ok 2024-09-16T05:17:52.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:51 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:17:52.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:51 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:17:52.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:51 smithi032 ceph-mon[27388]: osdmap e522: 12 total, 12 up, 11 in 2024-09-16T05:17:52.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:51 smithi032 ceph-mon[27388]: 2.3d scrub starts 2024-09-16T05:17:52.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:51 smithi032 ceph-mon[27388]: 2.3d scrub ok 2024-09-16T05:17:52.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:52.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:51 smithi196 ceph-mon[30802]: pgmap v942: 107 pgs: 107 active+clean; 137 MiB data, 594 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 3.5 MiB/s wr, 15 op/s 2024-09-16T05:17:52.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:51 smithi196 ceph-mon[30802]: 2.50 scrub starts 2024-09-16T05:17:52.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:51 smithi196 ceph-mon[30802]: 2.50 scrub ok 2024-09-16T05:17:52.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:51 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:17:52.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:51 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:17:52.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:51 smithi196 ceph-mon[30802]: osdmap e522: 12 total, 12 up, 11 in 2024-09-16T05:17:52.113 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:51 smithi196 ceph-mon[30802]: 2.3d scrub starts 2024-09-16T05:17:52.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:51 smithi196 ceph-mon[30802]: 2.3d scrub ok 2024-09-16T05:17:52.114 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:52.115 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:52.135 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:52.136 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:52.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:52.159 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:52.159 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:52.167 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:52.167 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:52.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:51 smithi204 ceph-mon[30933]: pgmap v942: 107 pgs: 107 active+clean; 137 MiB data, 594 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 3.5 MiB/s wr, 15 op/s 2024-09-16T05:17:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:51 smithi204 ceph-mon[30933]: 2.50 scrub starts 2024-09-16T05:17:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:51 smithi204 ceph-mon[30933]: 2.50 scrub ok 2024-09-16T05:17:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:51 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:17:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:51 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:17:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:51 smithi204 ceph-mon[30933]: osdmap e522: 12 total, 12 up, 11 in 2024-09-16T05:17:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:51 smithi204 ceph-mon[30933]: 2.3d scrub starts 2024-09-16T05:17:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:51 smithi204 ceph-mon[30933]: 2.3d scrub ok 2024-09-16T05:17:52.180 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:52.210 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:52.210 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:52.232 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:52.232 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:52.257 INFO:tasks.rados.rados.2.smithi204.stdout:1910: expect (ObjNum 677 snap 196 seq_num 677) 2024-09-16T05:17:52.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:52.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:52.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:52.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:52.397 INFO:tasks.rados.rados.2.smithi204.stdout:1908: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:17:52.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:52.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:1904: finishing copy_from racing read to smithi20457400-7 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:1904: finishing copy_from to smithi20457400-7 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 348 (ObjNum 621 snap 181 seq_num 621) dirty exists 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:1905: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:1905: finishing copy_from to smithi20457400-2 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 514 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:1906: finishing copy_from racing read to smithi20457400-39 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:1906: finishing copy_from to smithi20457400-39 2024-09-16T05:17:52.498 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 712 (ObjNum 668 snap 194 seq_num 668) dirty exists 2024-09-16T05:17:52.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:52.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:52.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:52.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:52.670 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:17:52.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:52.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:52.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:52.801 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:52.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:52.816 INFO:tasks.rados.rados.2.smithi204.stdout:1902: done (8 left) 2024-09-16T05:17:52.816 INFO:tasks.rados.rados.2.smithi204.stdout:1904: done (7 left) 2024-09-16T05:17:52.816 INFO:tasks.rados.rados.2.smithi204.stdout:1905: done (6 left) 2024-09-16T05:17:52.816 INFO:tasks.rados.rados.2.smithi204.stdout:1906: done (5 left) 2024-09-16T05:17:52.817 INFO:tasks.rados.rados.2.smithi204.stdout:1907: done (4 left) 2024-09-16T05:17:52.817 INFO:tasks.rados.rados.2.smithi204.stdout:1908: done (3 left) 2024-09-16T05:17:52.817 INFO:tasks.rados.rados.2.smithi204.stdout:1909: done (2 left) 2024-09-16T05:17:52.817 INFO:tasks.rados.rados.2.smithi204.stdout:1910: done (1 left) 2024-09-16T05:17:52.817 INFO:tasks.rados.rados.2.smithi204.stdout:1911: done (0 left) 2024-09-16T05:17:52.817 INFO:tasks.rados.rados.2.smithi204.stdout:1912: snap_create 2024-09-16T05:17:52.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:52.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:52.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:52.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:17:52.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:17:52.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/129719588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:52.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:17:52.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:17:52.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:17:52.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:17:52.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: 2.4e scrub starts 2024-09-16T05:17:52.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:52 smithi196 ceph-mon[30802]: 2.4e scrub ok 2024-09-16T05:17:53.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:53.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:53.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:53.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:17:53.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:17:53.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/129719588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:53.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:17:53.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:17:53.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:17:53.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:17:53.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: 2.4e scrub starts 2024-09-16T05:17:53.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:52 smithi032 ceph-mon[27388]: 2.4e scrub ok 2024-09-16T05:17:53.138 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:17:53.138 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:17:53.138 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:17:53.138 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:17:53.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:17:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:17:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/129719588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:17:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:17:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:17:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:17:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: 2.4e scrub starts 2024-09-16T05:17:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:52 smithi204 ceph-mon[30933]: 2.4e scrub ok 2024-09-16T05:17:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:53.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:53.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:53.275 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:53.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:53.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:53.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:53.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:53.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:53.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:53.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:53.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:53.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:53.675 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:53.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:53.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:53.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:17:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:53.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:53.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:53.815 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:17:53.820 INFO:tasks.rados.rados.2.smithi204.stdout:1912: done (0 left) 2024-09-16T05:17:53.821 INFO:tasks.rados.rados.2.smithi204.stdout:1913: rollback oid 23 current snap is 200 2024-09-16T05:17:53.821 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 23 to 197 2024-09-16T05:17:53.821 INFO:tasks.rados.rados.2.smithi204.stdout:1914: write oid 26 current snap is 200 2024-09-16T05:17:53.821 INFO:tasks.rados.rados.2.smithi204.stdout:1914: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-09-16T05:17:53.827 INFO:tasks.rados.rados.2.smithi204.stdout:1914: writing smithi20457400-26 from 695988 to 1321721 tid 1 2024-09-16T05:17:53.835 INFO:tasks.rados.rados.2.smithi204.stdout:1914: writing smithi20457400-26 from 2009801 to 2690362 tid 2 2024-09-16T05:17:53.840 INFO:tasks.rados.rados.2.smithi204.stdout:1914: writing smithi20457400-26 from 3150974 to 3495988 tid 3 2024-09-16T05:17:53.840 INFO:tasks.rados.rados.2.smithi204.stdout:1915: snap_remove snap 196 2024-09-16T05:17:53.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:53.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:53.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:54.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:54.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:54.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:54.048 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:17:54.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:53 smithi032 ceph-mon[27388]: pgmap v945: 107 pgs: 107 active+clean; 157 MiB data, 612 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 27 op/s 2024-09-16T05:17:54.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:53 smithi032 ceph-mon[27388]: osdmap e523: 12 total, 12 up, 11 in 2024-09-16T05:17:54.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3548622153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:54.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:54.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:53 smithi196 ceph-mon[30802]: pgmap v945: 107 pgs: 107 active+clean; 157 MiB data, 612 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 27 op/s 2024-09-16T05:17:54.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:53 smithi196 ceph-mon[30802]: osdmap e523: 12 total, 12 up, 11 in 2024-09-16T05:17:54.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3548622153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:53 smithi204 ceph-mon[30933]: pgmap v945: 107 pgs: 107 active+clean; 157 MiB data, 612 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 27 op/s 2024-09-16T05:17:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:53 smithi204 ceph-mon[30933]: osdmap e523: 12 total, 12 up, 11 in 2024-09-16T05:17:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3548622153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:17:54.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:54.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:54.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:54.303 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:54.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:54.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:54.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:54.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:54.531 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:17:54.532 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:17:54.532 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:17:54.532 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:17:54.532 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:17:54.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:54.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:54.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:54.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:54.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:54.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:54.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:54.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:54.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:54.831 INFO:tasks.rados.rados.2.smithi204.stdout:1915: done (2 left) 2024-09-16T05:17:54.831 INFO:tasks.rados.rados.2.smithi204.stdout:1916: write (excl) oid 50 current snap is 200 2024-09-16T05:17:54.831 INFO:tasks.rados.rados.2.smithi204.stdout:1913: finishing rollback tid 0 to smithi20457400-23 2024-09-16T05:17:54.832 INFO:tasks.rados.rados.2.smithi204.stdout:1916: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-09-16T05:17:54.838 INFO:tasks.rados.rados.2.smithi204.stdout:1916: writing smithi20457400-50 from 712795 to 1130130 tid 1 2024-09-16T05:17:54.845 INFO:tasks.rados.rados.2.smithi204.stdout:1916: writing smithi20457400-50 from 1784636 to 2325208 tid 2 2024-09-16T05:17:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1916: writing smithi20457400-50 from 2894750 to 3352667 tid 3 2024-09-16T05:17:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1917: delete oid 6 current snap is 200 2024-09-16T05:17:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1913: finishing rollback tid 1 to smithi20457400-23 2024-09-16T05:17:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1913: finishing rollback tid 2 to smithi20457400-23 2024-09-16T05:17:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 700 (ObjNum 680 snap 196 seq_num 680) dirty exists 2024-09-16T05:17:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1914: finishing write tid 1 to smithi20457400-26 2024-09-16T05:17:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1914: finishing write tid 2 to smithi20457400-26 2024-09-16T05:17:54.851 INFO:tasks.rados.rados.2.smithi204.stdout:1914: finishing write tid 3 to smithi20457400-26 2024-09-16T05:17:54.852 INFO:tasks.rados.rados.2.smithi204.stdout:1914: finishing write tid 4 to smithi20457400-26 2024-09-16T05:17:54.852 INFO:tasks.rados.rados.2.smithi204.stdout:1914: finishing write tid 5 to smithi20457400-26 2024-09-16T05:17:54.852 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 900 (ObjNum 684 snap 200 seq_num 684) dirty exists 2024-09-16T05:17:54.852 INFO:tasks.rados.rados.2.smithi204.stdout:1914: left oid 26 (ObjNum 684 snap 200 seq_num 684) 2024-09-16T05:17:54.852 INFO:tasks.rados.rados.2.smithi204.stdout:1916: finishing write tid 1 to smithi20457400-50 2024-09-16T05:17:54.855 INFO:tasks.rados.rados.2.smithi204.stdout:1913: done (3 left) 2024-09-16T05:17:54.855 INFO:tasks.rados.rados.2.smithi204.stdout:1914: done (2 left) 2024-09-16T05:17:54.855 INFO:tasks.rados.rados.2.smithi204.stdout:1917: done (1 left) 2024-09-16T05:17:54.855 INFO:tasks.rados.rados.2.smithi204.stdout:1918: rollback oid 47 current snap is 200 2024-09-16T05:17:54.856 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 47 to 195 2024-09-16T05:17:54.856 INFO:tasks.rados.rados.2.smithi204.stdout:1919: delete oid 34 current snap is 200 2024-09-16T05:17:54.856 INFO:tasks.rados.rados.2.smithi204.stdout:1916: finishing write tid 2 to smithi20457400-50 2024-09-16T05:17:54.856 INFO:tasks.rados.rados.2.smithi204.stdout:1916: finishing write tid 3 to smithi20457400-50 2024-09-16T05:17:54.856 INFO:tasks.rados.rados.2.smithi204.stdout:1916: finishing write tid 4 to smithi20457400-50 2024-09-16T05:17:54.857 INFO:tasks.rados.rados.2.smithi204.stdout:1916: finishing write tid 5 to smithi20457400-50 2024-09-16T05:17:54.857 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 390 (ObjNum 685 snap 200 seq_num 685) dirty exists 2024-09-16T05:17:54.857 INFO:tasks.rados.rados.2.smithi204.stdout:1916: left oid 50 (ObjNum 685 snap 200 seq_num 685) 2024-09-16T05:17:54.857 INFO:tasks.rados.rados.2.smithi204.stdout:1916: done (2 left) 2024-09-16T05:17:54.857 INFO:tasks.rados.rados.2.smithi204.stdout:1919: done (1 left) 2024-09-16T05:17:54.858 INFO:tasks.rados.rados.2.smithi204.stdout:1920: snap_remove snap 188 2024-09-16T05:17:54.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:54.940 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:54.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:55.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:55.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:54 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:55.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3548622153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:55.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:54 smithi196 ceph-mon[30802]: osdmap e524: 12 total, 12 up, 11 in 2024-09-16T05:17:55.068 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:55.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:55.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:17:55.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:55.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:54 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:55.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3548622153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:55.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:54 smithi032 ceph-mon[27388]: osdmap e524: 12 total, 12 up, 11 in 2024-09-16T05:17:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:54 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:17:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3548622153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:17:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:54 smithi204 ceph-mon[30933]: osdmap e524: 12 total, 12 up, 11 in 2024-09-16T05:17:55.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:55.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:55.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:55.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:55.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:55.323 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:17:55.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:17:55.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:55.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:55.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:55.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:55.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:55.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:55.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:55.701 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:55.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:55.753 INFO:tasks.rados.rados.2.smithi204.stdout:1920: done (1 left) 2024-09-16T05:17:55.753 INFO:tasks.rados.rados.2.smithi204.stdout:1921: copy_from oid 13 from oid 39 current snap is 200 2024-09-16T05:17:55.753 INFO:tasks.rados.rados.2.smithi204.stdout:1922: write oid 25 current snap is 200 2024-09-16T05:17:55.753 INFO:tasks.rados.rados.2.smithi204.stdout:1922: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-16T05:17:55.761 INFO:tasks.rados.rados.2.smithi204.stdout:1922: writing smithi20457400-25 from 763216 to 1406298 tid 1 2024-09-16T05:17:55.768 INFO:tasks.rados.rados.2.smithi204.stdout:1922: writing smithi20457400-25 from 1843729 to 2531628 tid 2 2024-09-16T05:17:55.774 INFO:tasks.rados.rados.2.smithi204.stdout:1922: writing smithi20457400-25 from 3260666 to 3563216 tid 3 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:1923: delete oid 21 current snap is 200 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:1918: finishing rollback tid 1 to smithi20457400-47 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:1918: finishing rollback tid 2 to smithi20457400-47 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 759 (ObjNum 669 snap 194 seq_num 669) dirty exists 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:1922: finishing write tid 1 to smithi20457400-25 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:1921: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:1922: finishing write tid 2 to smithi20457400-25 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:1921: finishing copy_from to smithi20457400-13 2024-09-16T05:17:55.775 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 712 (ObjNum 668 snap 194 seq_num 668) dirty exists 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1918: done (3 left) 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1921: done (2 left) 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1923: done (1 left) 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1924: read oid 15 snap 194 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1924: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1925: read oid 22 snap -1 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1925: expect (ObjNum 652 snap 189 seq_num 652) 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1926: snap_create 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1922: finishing write tid 3 to smithi20457400-25 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1922: finishing write tid 4 to smithi20457400-25 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:1922: finishing write tid 5 to smithi20457400-25 2024-09-16T05:17:55.777 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 706 (ObjNum 688 snap 200 seq_num 688) dirty exists 2024-09-16T05:17:55.778 INFO:tasks.rados.rados.2.smithi204.stdout:1922: left oid 25 (ObjNum 688 snap 200 seq_num 688) 2024-09-16T05:17:55.778 INFO:tasks.rados.rados.2.smithi204.stdout:1924: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:17:55.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:55.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:55.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:55.846 INFO:tasks.rados.rados.2.smithi204.stdout:1925: expect (ObjNum 652 snap 189 seq_num 652) 2024-09-16T05:17:55.859 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:17:55.859 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:17:55.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:290 2.d e117/107 2:b98a79db:::smithi20457400-31:head v 117'29, mlcod=116'25)", 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 426.50338528999998, 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061542619999999998, 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354115+0000", 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.354117+0000", 2024-09-16T05:17:55.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2110000000000001e-06 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357718+0000", 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036011939999999998 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357775+0000", 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7274000000000001e-05 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357803+0000", 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7860000000000001e-05 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357882+0000", 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9091999999999994e-05 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.357905+0000", 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2544999999999998e-05 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.360225+0000", 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023196480000000001 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:49.360269+0000", 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4437999999999997e-05 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:442 2.d e127/107 2:ba7e58cf:::smithi20457400-26:head v 127'67, mlcod=127'64)", 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 417.18844628800002, 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047970019999999999, 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669054+0000", 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3000000000000001e-07 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669058+0000", 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7699999999999999e-06 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669074+0000", 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6268000000000001e-05 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669078+0000", 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7500000000000001e-06 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669137+0000", 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.944e-05 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.669158+0000", 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0706000000000001e-05 2024-09-16T05:17:55.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.673817+0000", 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0046588840000000003 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:58.673851+0000", 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3754000000000002e-05 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:605 2.d e142/107 2:b1061e13:::smithi20457400-33:head v 142'81, mlcod=142'78)", 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 402.042894209, 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052172579999999998, 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:17:55.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814606+0000", 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.814607+0000", 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.544e-06 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817665+0000", 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030580270000000001 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817694+0000", 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8353000000000001e-05 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817717+0000", 2024-09-16T05:17:55.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3300000000000001e-05 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817769+0000", 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2147000000000001e-05 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.817785+0000", 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5874000000000001e-05 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.819789+0000", 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020036870000000001 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:11:13.819823+0000", 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4326000000000003e-05 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.874 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1389 2.1d e210/209 2:b98a79db:::smithi20457400-31:head v 210'262, mlcod=0'0)", 2024-09-16T05:17:55.875 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:17:55.875 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 333.25145380700002, 2024-09-16T05:17:55.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047389830000000004, 2024-09-16T05:17:55.875 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.876 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.876 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.876 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606046+0000", 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.606048+0000", 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.781e-06 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608682+0000", 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026339990000000001 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608713+0000", 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0781999999999998e-05 2024-09-16T05:17:55.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608722+0000", 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1139999999999995e-06 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608800+0000", 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8145000000000005e-05 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.608820+0000", 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9692999999999999e-05 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.610714+0000", 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001894448 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.610785+0000", 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1020999999999996e-05 2024-09-16T05:17:55.880 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:1598 2.d e228/209 2:b1061e13:::smithi20457400-33:head v 228'278, mlcod=224'274)", 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 315.991391695, 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029840370000000001, 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.881 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866108+0000", 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.866109+0000", 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8800000000000003e-07 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867732+0000", 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016233739999999999 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867777+0000", 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4693000000000001e-05 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867788+0000", 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1148000000000001e-05 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867850+0000", 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1811999999999997e-05 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.867870+0000", 2024-09-16T05:17:55.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9698e-05 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.869048+0000", 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011787259999999999 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:39.869092+0000", 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4097999999999997e-05 2024-09-16T05:17:55.885 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2143 2.2b e279/252 2:d7c71ae7:::smithi20457400-5:head v 279'188, mlcod=268'185)", 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 265.10006521999998, 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055854579999999997, 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.757435+0000", 2024-09-16T05:17:55.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7700000000000001e-07 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760458+0000", 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003022725 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760491+0000", 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3637000000000003e-05 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760500+0000", 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6240000000000008e-06 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760609+0000", 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108887 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.760631+0000", 2024-09-16T05:17:55.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2161e-05 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.762973+0000", 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023423939999999998 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.763020+0000", 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6853e-05 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.890 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2318 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 248.171641236, 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0025039739999999, 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:55.891 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2318 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685859+0000", 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685861+0000", 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6700000000000001e-06 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685867+0000", 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8179999999999998e-06 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685885+0000", 2024-09-16T05:17:55.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7696000000000001e-05 2024-09-16T05:17:55.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685895+0000", 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5899999999999997e-06 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685955+0000", 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0501000000000002e-05 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685966+0000", 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0462000000000001e-05 2024-09-16T05:17:55.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688335+0000", 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0023690300000001 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688363+0000", 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8206999999999999e-05 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.896 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2319 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 248.171522979, 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0024202550000001, 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:17:55.897 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2319 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685977+0000", 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685978+0000", 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2500000000000005e-07 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685981+0000", 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6150000000000001e-06 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685989+0000", 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6159999999999996e-06 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.685995+0000", 2024-09-16T05:17:55.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6250000000000004e-06 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686041+0000", 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6694999999999999e-05 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686049+0000", 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3170000000000004e-06 2024-09-16T05:17:55.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688389+0000", 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002340623 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688397+0000", 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1389999999999995e-06 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.902 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2320 2.2b 2.e758e3eb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e296)", 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 248.17144245399999, 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.002368715, 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:17:55.903 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2320 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686058+0000", 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7199999999999999e-07 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686062+0000", 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5590000000000001e-06 2024-09-16T05:17:55.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686069+0000", 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2710000000000003e-06 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686075+0000", 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7590000000000003e-06 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686126+0000", 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1384999999999997e-05 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:47.686135+0000", 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1710000000000008e-06 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688420+0000", 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0022843889999999 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:48.688426+0000", 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6089999999999997e-06 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2749 2.d e357/247 2:b21a57e7:::smithi20457400-25:head v 357'413, mlcod=356'409)", 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 169.140266747, 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079488969999999999, 2024-09-16T05:17:55.908 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:17:55.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717234+0000", 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0210000000000001e-06 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722533+0000", 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052989389999999999 2024-09-16T05:17:55.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722583+0000", 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9592000000000002e-05 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722593+0000", 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0567999999999999e-05 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722673+0000", 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.983e-05 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722700+0000", 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6908999999999999e-05 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.725139+0000", 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024389920000000001 2024-09-16T05:17:55.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.725182+0000", 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3046000000000001e-05 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2900 2.32 e372/275 2:4e7d172c:::smithi20457400-15:head v 372'109, mlcod=348'105)", 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 154.14867806000001, 2024-09-16T05:17:55.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033081819999999998, 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708822+0000", 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7499999999999999e-07 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708827+0000", 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9969999999999998e-06 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708864+0000", 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6820000000000003e-05 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708880+0000", 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5908000000000001e-05 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708933+0000", 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2970000000000003e-05 2024-09-16T05:17:55.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.708951+0000", 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8570999999999999e-05 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.712084+0000", 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003132297 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.712130+0000", 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6443999999999998e-05 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2898 2.2b e372/329 2:d7c71ae7:::smithi20457400-5:head v 372'201, mlcod=330'198)", 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 154.13292091400001, 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085128900000000004, 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.918 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724580+0000", 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729164+0000", 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045841340000000001 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729217+0000", 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3152999999999997e-05 2024-09-16T05:17:55.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729225+0000", 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9850000000000006e-06 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729320+0000", 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5067000000000003e-05 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729381+0000", 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1297999999999995e-05 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.733031+0000", 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036494600000000002 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.733092+0000", 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1371000000000006e-05 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.922 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.2.0:12 2.2b e406/329 2:d7c71ae7:::smithi20457400-5:84 v 406'216, mlcod=398'212)", 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:55.718743+0000", 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 120.138757055, 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043287330000000004, 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718743+0000", 2024-09-16T05:17:55.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718743+0000", 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718743+0000", 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4900000000000001e-07 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718760+0000", 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6220000000000001e-05 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718837+0000", 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7735e-05 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718846+0000", 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4940000000000003e-06 2024-09-16T05:17:55.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.722330+0000", 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034842520000000002 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.722355+0000", 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4890999999999999e-05 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.723021+0000", 2024-09-16T05:17:55.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00066640299999999998 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.723072+0000", 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0389000000000002e-05 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:3671 2.32 e446/275 2:4e7d172c:::smithi20457400-15:head v 446'130, mlcod=380'125)", 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:35.728486+0000", 2024-09-16T05:17:55.927 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 80.129014526000006, 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049019800000000002, 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.728486+0000", 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.728486+0000", 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.728486+0000", 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6199999999999997e-07 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.728492+0000", 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6269999999999997e-06 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.732748+0000", 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042564739999999997 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.732764+0000", 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5633000000000001e-05 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.732815+0000", 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0788000000000002e-05 2024-09-16T05:17:55.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.732836+0000", 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1455999999999999e-05 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.733340+0000", 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050352199999999995 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:35.733388+0000", 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7718e-05 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:3975 2.d e468/435 2:b1061e13:::smithi20457400-33:head v 468'472, mlcod=459'469)", 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:57.903941+0000", 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 57.953559284999997, 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067494490000000002, 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.932 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.903941+0000", 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.903941+0000", 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.903941+0000", 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9000000000000001e-07 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908253+0000", 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043118399999999999 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908300+0000", 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6953999999999998e-05 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908334+0000", 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3995000000000002e-05 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908424+0000", 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0288e-05 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908443+0000", 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9184000000000001e-05 2024-09-16T05:17:55.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.910651+0000", 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022077669999999998 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.910690+0000", 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9230999999999998e-05 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4175 2.d e489/435 2:b1061e13:::smithi20457400-33:head v 489'489, mlcod=479'485)", 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:18.756234+0000", 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 37.101265712, 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027262190000000002, 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.756234+0000", 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.756234+0000", 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.756235+0000", 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.74e-07 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.757106+0000", 2024-09-16T05:17:55.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087167600000000005 2024-09-16T05:17:55.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:55.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.757143+0000", 2024-09-16T05:17:55.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6711999999999997e-05 2024-09-16T05:17:55.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.757154+0000", 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1172000000000001e-05 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.757522+0000", 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036833200000000002 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.757546+0000", 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3417e-05 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.758918+0000", 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013722910000000001 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:18.758961+0000", 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2444999999999998e-05 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4580 2.3b e511/296 2:de669526:::smithi20457400-23:head v 511'328, mlcod=481'326)", 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:40.823701+0000", 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 15.033799446, 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005463033, 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.823701+0000", 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.823701+0000", 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.823702+0000", 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.879999999999999e-07 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.824896+0000", 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011946000000000001 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.824938+0000", 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1544000000000001e-05 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.824945+0000", 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4989999999999997e-06 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.945 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.825004+0000", 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8943000000000001e-05 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.825022+0000", 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7550999999999998e-05 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.829123+0000", 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004101525 2024-09-16T05:17:55.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:40.829164+0000", 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0383000000000003e-05 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4633 2.3b e512/296 2:de669526:::smithi20457400-23:head v 512'330, mlcod=508'327)", 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:41.761868+0000", 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.095632015, 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054648379999999996, 2024-09-16T05:17:55.947 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.761868+0000", 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.761868+0000", 2024-09-16T05:17:55.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.761868+0000", 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8e-07 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.763041+0000", 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011731689999999999 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.763093+0000", 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1728999999999999e-05 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.763103+0000", 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9830000000000008e-06 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.763179+0000", 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6318999999999999e-05 2024-09-16T05:17:55.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.763206+0000", 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7002999999999999e-05 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.767290+0000", 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040834649999999997 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:41.767333+0000", 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.299e-05 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4670 2.3b e515/296 2:de669526:::smithi20457400-23:head v 515'332, mlcod=511'328)", 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:44.810292+0000", 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.047208412, 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073315560000000004, 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.952 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.810292+0000", 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.810292+0000", 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.810292+0000", 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.03e-07 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812464+0000", 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021723290000000002 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812594+0000", 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013015999999999999 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812612+0000", 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7696000000000001e-05 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812717+0000", 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010487100000000001 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812740+0000", 2024-09-16T05:17:55.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2943e-05 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.817579+0000", 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048388800000000003 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.817623+0000", 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4474000000000001e-05 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.956 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4665 2.d e515/435 2:b1061e13:::smithi20457400-33:head v 515'512, mlcod=512'509)", 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:44.814949+0000", 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.042551259, 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071594459999999999, 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.814949+0000", 2024-09-16T05:17:55.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.814949+0000", 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.814949+0000", 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.36e-07 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820286+0000", 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053367509999999998 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820312+0000", 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6325999999999998e-05 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820318+0000", 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0610000000000001e-06 2024-09-16T05:17:55.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820407+0000", 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8327999999999999e-05 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820425+0000", 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8437999999999999e-05 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.822073+0000", 2024-09-16T05:17:55.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016484220000000001 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.822108+0000", 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4884e-05 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:17:55.961 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:17:55.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:55.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:56.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:55 smithi032 ceph-mon[27388]: pgmap v948: 107 pgs: 107 active+clean; 164 MiB data, 637 MiB used, 983 GiB / 983 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 36 op/s 2024-09-16T05:17:56.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:55 smithi032 ceph-mon[27388]: osdmap e525: 12 total, 12 up, 11 in 2024-09-16T05:17:56.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:55 smithi032 ceph-mon[27388]: osdmap e526: 12 total, 12 up, 11 in 2024-09-16T05:17:56.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:56.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:56.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:56.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:55 smithi204 ceph-mon[30933]: pgmap v948: 107 pgs: 107 active+clean; 164 MiB data, 637 MiB used, 983 GiB / 983 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 36 op/s 2024-09-16T05:17:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:55 smithi204 ceph-mon[30933]: osdmap e525: 12 total, 12 up, 11 in 2024-09-16T05:17:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:55 smithi204 ceph-mon[30933]: osdmap e526: 12 total, 12 up, 11 in 2024-09-16T05:17:56.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:55 smithi196 ceph-mon[30802]: pgmap v948: 107 pgs: 107 active+clean; 164 MiB data, 637 MiB used, 983 GiB / 983 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 36 op/s 2024-09-16T05:17:56.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:55 smithi196 ceph-mon[30802]: osdmap e525: 12 total, 12 up, 11 in 2024-09-16T05:17:56.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:55 smithi196 ceph-mon[30802]: osdmap e526: 12 total, 12 up, 11 in 2024-09-16T05:17:56.185 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:17:56.186 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:17:56.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:17:56.207 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:17:56.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:56.227 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:17:56.227 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:17:56.230 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:56.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:56.247 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:17:56.248 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:17:56.268 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:17:56.269 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:17:56.292 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:17:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:17:56.313 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:17:56.314 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:17:56.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:56.335 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:17:56.335 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:17:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:56.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:56.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:56.356 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:17:56.356 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:17:56.378 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:17:56.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:17:56.400 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:17:56.400 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:17:56.422 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:17:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:17:56.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:56.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:56.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:56.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:17:56.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:56.624 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:56.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:56.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:17:56.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:56.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:56.751 INFO:tasks.rados.rados.2.smithi204.stdout:1922: done (3 left) 2024-09-16T05:17:56.751 INFO:tasks.rados.rados.2.smithi204.stdout:1924: done (2 left) 2024-09-16T05:17:56.751 INFO:tasks.rados.rados.2.smithi204.stdout:1925: done (1 left) 2024-09-16T05:17:56.752 INFO:tasks.rados.rados.2.smithi204.stdout:1926: done (0 left) 2024-09-16T05:17:56.752 INFO:tasks.rados.rados.2.smithi204.stdout:1927: delete oid 14 current snap is 201 2024-09-16T05:17:56.754 INFO:tasks.rados.rados.2.smithi204.stdout:1927: done (0 left) 2024-09-16T05:17:56.755 INFO:tasks.rados.rados.2.smithi204.stdout:1928: write oid 42 current snap is 201 2024-09-16T05:17:56.755 INFO:tasks.rados.rados.2.smithi204.stdout:1928: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-16T05:17:56.759 INFO:tasks.rados.rados.2.smithi204.stdout:1928: writing smithi20457400-42 from 413637 to 882466 tid 1 2024-09-16T05:17:56.766 INFO:tasks.rados.rados.2.smithi204.stdout:1928: writing smithi20457400-42 from 1619175 to 2054401 tid 2 2024-09-16T05:17:56.773 INFO:tasks.rados.rados.2.smithi204.stdout:1928: writing smithi20457400-42 from 2542935 to 3301190 tid 3 2024-09-16T05:17:56.773 INFO:tasks.rados.rados.2.smithi204.stdout:1929: write (excl) oid 25 current snap is 201 2024-09-16T05:17:56.773 INFO:tasks.rados.rados.2.smithi204.stdout:1929: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2024-09-16T05:17:56.778 INFO:tasks.rados.rados.2.smithi204.stdout:1929: writing smithi20457400-25 from 430444 to 1090875 tid 1 2024-09-16T05:17:56.785 INFO:tasks.rados.rados.2.smithi204.stdout:1929: writing smithi20457400-25 from 1677657 to 2372894 tid 2 2024-09-16T05:17:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:1929: writing smithi20457400-25 from 2970358 to 3539885 tid 3 2024-09-16T05:17:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:1930: write (excl) oid 15 current snap is 201 2024-09-16T05:17:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:1928: finishing write tid 1 to smithi20457400-42 2024-09-16T05:17:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:1928: finishing write tid 2 to smithi20457400-42 2024-09-16T05:17:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:1930: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-16T05:17:56.798 INFO:tasks.rados.rados.2.smithi204.stdout:1930: writing smithi20457400-15 from 447251 to 1182931 tid 1 2024-09-16T05:17:56.803 INFO:tasks.rados.rados.2.smithi204.stdout:1930: writing smithi20457400-15 from 1736139 to 2175034 tid 2 2024-09-16T05:17:56.812 INFO:tasks.rados.rados.2.smithi204.stdout:1930: writing smithi20457400-15 from 2597781 to 3378580 tid 3 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1931: rollback oid 50 current snap is 201 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 195 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1928: finishing write tid 3 to smithi20457400-42 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1928: finishing write tid 4 to smithi20457400-42 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1928: finishing write tid 5 to smithi20457400-42 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 754 (ObjNum 691 snap 201 seq_num 691) dirty exists 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1928: left oid 42 (ObjNum 691 snap 201 seq_num 691) 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1929: finishing write tid 1 to smithi20457400-25 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1928: done (3 left) 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1932: write oid 35 current snap is 201 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1929: finishing write tid 2 to smithi20457400-25 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1929: finishing write tid 3 to smithi20457400-25 2024-09-16T05:17:56.813 INFO:tasks.rados.rados.2.smithi204.stdout:1929: finishing write tid 4 to smithi20457400-25 2024-09-16T05:17:56.814 INFO:tasks.rados.rados.2.smithi204.stdout:1932: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2024-09-16T05:17:56.815 INFO:tasks.rados.rados.2.smithi204.stdout:1932: writing smithi20457400-35 from 464058 to 874987 tid 1 2024-09-16T05:17:56.823 INFO:tasks.rados.rados.2.smithi204.stdout:1932: writing smithi20457400-35 from 1394621 to 2093527 tid 2 2024-09-16T05:17:56.827 INFO:tasks.rados.rados.2.smithi204.stdout:1932: writing smithi20457400-35 from 2625204 to 3217275 tid 3 2024-09-16T05:17:56.831 INFO:tasks.rados.rados.2.smithi204.stdout:1932: writing smithi20457400-35 from 3656107 to 3664058 tid 4 2024-09-16T05:17:56.831 INFO:tasks.rados.rados.2.smithi204.stdout:1933: write oid 4 current snap is 201 2024-09-16T05:17:56.831 INFO:tasks.rados.rados.2.smithi204.stdout:1929: finishing write tid 5 to smithi20457400-25 2024-09-16T05:17:56.831 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 710 (ObjNum 692 snap 201 seq_num 692) dirty exists 2024-09-16T05:17:56.831 INFO:tasks.rados.rados.2.smithi204.stdout:1929: left oid 25 (ObjNum 692 snap 201 seq_num 692) 2024-09-16T05:17:56.831 INFO:tasks.rados.rados.2.smithi204.stdout:1930: finishing write tid 1 to smithi20457400-15 2024-09-16T05:17:56.831 INFO:tasks.rados.rados.2.smithi204.stdout:1933: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-09-16T05:17:56.836 INFO:tasks.rados.rados.2.smithi204.stdout:1933: writing smithi20457400-4 from 480865 to 967043 tid 1 2024-09-16T05:17:56.841 INFO:tasks.rados.rados.2.smithi204.stdout:1933: writing smithi20457400-4 from 1453103 to 1895667 tid 2 2024-09-16T05:17:56.848 INFO:tasks.rados.rados.2.smithi204.stdout:1933: writing smithi20457400-4 from 2652627 to 3172323 tid 3 2024-09-16T05:17:56.848 INFO:tasks.rados.rados.2.smithi204.stdout:1929: done (4 left) 2024-09-16T05:17:56.848 INFO:tasks.rados.rados.2.smithi204.stdout:1934: snap_remove snap 198 2024-09-16T05:17:56.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:56.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:17:56.889 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:17:56.973 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:56.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/135453784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:56.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:56 smithi196 ceph-mon[30802]: osdmap e527: 12 total, 12 up, 11 in 2024-09-16T05:17:56.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:56.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:57.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:57.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:57.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:57.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/135453784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:57.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:56 smithi032 ceph-mon[27388]: osdmap e527: 12 total, 12 up, 11 in 2024-09-16T05:17:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/135453784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:17:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:56 smithi204 ceph-mon[30933]: osdmap e527: 12 total, 12 up, 11 in 2024-09-16T05:17:57.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:17:57.245 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:57.248 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:57.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:17:57.341 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:57.341 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:57.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:57.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:57.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:57.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:57.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:57.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:57.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:57.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:57.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:57.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:57.700 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:17:57.700 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:17:57.700 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-16T05:17:57.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd in 10 2024-09-16T05:17:57.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:17:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:57.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:57.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:17:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:1934: done (4 left) 2024-09-16T05:17:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:1935: read oid 44 snap -1 2024-09-16T05:17:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:1935: expect (ObjNum 677 snap 196 seq_num 677) 2024-09-16T05:17:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:1936: snap_create 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1930: finishing write tid 2 to smithi20457400-15 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1930: finishing write tid 3 to smithi20457400-15 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1930: finishing write tid 4 to smithi20457400-15 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1930: finishing write tid 5 to smithi20457400-15 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 690 (ObjNum 693 snap 201 seq_num 693) dirty exists 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1930: left oid 15 (ObjNum 693 snap 201 seq_num 693) 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1931: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1931: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1931: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 396 (ObjNum 592 snap 180 seq_num 592) dirty exists 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1932: finishing write tid 1 to smithi20457400-35 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1932: finishing write tid 2 to smithi20457400-35 2024-09-16T05:17:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:1932: finishing write tid 3 to smithi20457400-35 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1932: finishing write tid 4 to smithi20457400-35 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1932: finishing write tid 5 to smithi20457400-35 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1932: finishing write tid 6 to smithi20457400-35 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 833 (ObjNum 694 snap 201 seq_num 694) dirty exists 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1932: left oid 35 (ObjNum 694 snap 201 seq_num 694) 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1933: finishing write tid 1 to smithi20457400-4 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1933: finishing write tid 2 to smithi20457400-4 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1933: finishing write tid 3 to smithi20457400-4 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1933: finishing write tid 4 to smithi20457400-4 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1933: finishing write tid 5 to smithi20457400-4 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 572 (ObjNum 695 snap 201 seq_num 695) dirty exists 2024-09-16T05:17:57.841 INFO:tasks.rados.rados.2.smithi204.stdout:1933: left oid 4 (ObjNum 695 snap 201 seq_num 695) 2024-09-16T05:17:57.843 INFO:tasks.rados.rados.2.smithi204.stdout:1935: expect (ObjNum 677 snap 196 seq_num 677) 2024-09-16T05:17:57.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:17:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:57.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:57.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:17:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:17:57.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:57.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:17:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:57.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:57.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:58.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:57 smithi032 ceph-mon[27388]: pgmap v951: 107 pgs: 107 active+clean; 170 MiB data, 627 MiB used, 983 GiB / 983 GiB avail; 16 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-16T05:17:58.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:17:58.120 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:58.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:17:58.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:57 smithi204 ceph-mon[30933]: pgmap v951: 107 pgs: 107 active+clean; 170 MiB data, 627 MiB used, 983 GiB / 983 GiB avail; 16 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-16T05:17:58.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:57 smithi196 ceph-mon[30802]: pgmap v951: 107 pgs: 107 active+clean; 170 MiB data, 627 MiB used, 983 GiB / 983 GiB avail; 16 MiB/s rd, 3.6 MiB/s wr, 21 op/s 2024-09-16T05:17:58.239 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:17:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:58.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:58.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:58.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:58.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:58.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:58.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:58.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:58.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:58.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:58.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:58.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:58.690 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:58.690 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:58.721 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:17:58.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:58.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:17:58.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:58.847 INFO:teuthology.orchestra.run.smithi032.stderr:marked in osd.10. 2024-09-16T05:17:58.847 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:17:58.858 INFO:tasks.rados.rados.2.smithi204.stdout:1930: done (5 left) 2024-09-16T05:17:58.859 INFO:tasks.rados.rados.2.smithi204.stdout:1931: done (4 left) 2024-09-16T05:17:58.859 INFO:tasks.rados.rados.2.smithi204.stdout:1932: done (3 left) 2024-09-16T05:17:58.859 INFO:tasks.rados.rados.2.smithi204.stdout:1933: done (2 left) 2024-09-16T05:17:58.859 INFO:tasks.rados.rados.2.smithi204.stdout:1935: done (1 left) 2024-09-16T05:17:58.859 INFO:tasks.rados.rados.2.smithi204.stdout:1936: done (0 left) 2024-09-16T05:17:58.859 INFO:tasks.rados.rados.2.smithi204.stdout:1937: delete oid 29 current snap is 202 2024-09-16T05:17:58.861 INFO:tasks.rados.rados.2.smithi204.stdout:1937: done (0 left) 2024-09-16T05:17:58.861 INFO:tasks.rados.rados.2.smithi204.stdout:1938: read oid 45 snap -1 2024-09-16T05:17:58.861 INFO:tasks.rados.rados.2.smithi204.stdout:1938: expect (ObjNum 476 snap 150 seq_num 476) 2024-09-16T05:17:58.861 INFO:tasks.rados.rados.2.smithi204.stdout:1939: write (excl) oid 15 current snap is 202 2024-09-16T05:17:58.861 INFO:tasks.rados.rados.2.smithi204.stdout:1939: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2024-09-16T05:17:58.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:58 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:17:58.604+0000 7f7c3f992640 -1 mon.a@0(leader).osd e528 definitely_dead 0 2024-09-16T05:17:58.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:58 smithi032 ceph-mon[27388]: osdmap e528: 12 total, 12 up, 11 in 2024-09-16T05:17:58.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/524964779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:58.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1696769685' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-09-16T05:17:58.867 INFO:tasks.rados.rados.2.smithi204.stdout:1939: writing smithi20457400-15 from 514479 to 1151155 tid 1 2024-09-16T05:17:58.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:58.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:1939: writing smithi20457400-15 from 1853714 to 2299947 tid 2 2024-09-16T05:17:58.881 INFO:tasks.rados.rados.2.smithi204.stdout:1939: writing smithi20457400-15 from 2991120 to 3533360 tid 3 2024-09-16T05:17:58.881 INFO:tasks.rados.rados.2.smithi204.stdout:1940: read oid 5 snap -1 2024-09-16T05:17:58.881 INFO:tasks.rados.rados.2.smithi204.stdout:1940: expect (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:17:58.881 INFO:tasks.rados.rados.2.smithi204.stdout:1941: write oid 44 current snap is 202 2024-09-16T05:17:58.881 INFO:tasks.rados.rados.2.smithi204.stdout:1941: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-16T05:17:58.887 INFO:tasks.rados.rados.2.smithi204.stdout:1941: writing smithi20457400-44 from 531286 to 1243211 tid 1 2024-09-16T05:17:58.895 INFO:tasks.rados.rados.2.smithi204.stdout:1941: writing smithi20457400-44 from 1912196 to 2618440 tid 2 2024-09-16T05:17:58.902 INFO:tasks.rados.rados.2.smithi204.stdout:1941: writing smithi20457400-44 from 3018543 to 3731286 tid 3 2024-09-16T05:17:58.902 INFO:tasks.rados.rados.2.smithi204.stdout:1942: delete oid 20 current snap is 202 2024-09-16T05:17:58.902 INFO:tasks.rados.rados.2.smithi204.stdout:1938: expect (ObjNum 476 snap 150 seq_num 476) 2024-09-16T05:17:58.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:58.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:58 smithi196 ceph-mon[30802]: osdmap e528: 12 total, 12 up, 11 in 2024-09-16T05:17:58.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/524964779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:58.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1696769685' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-09-16T05:17:58.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:58.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:59.032 INFO:tasks.rados.rados.2.smithi204.stdout:1939: finishing write tid 1 to smithi20457400-15 2024-09-16T05:17:59.032 INFO:tasks.rados.rados.2.smithi204.stdout:1939: finishing write tid 2 to smithi20457400-15 2024-09-16T05:17:59.032 INFO:tasks.rados.rados.2.smithi204.stdout:1939: finishing write tid 3 to smithi20457400-15 2024-09-16T05:17:59.032 INFO:tasks.rados.rados.2.smithi204.stdout:1939: finishing write tid 4 to smithi20457400-15 2024-09-16T05:17:59.032 INFO:tasks.rados.rados.2.smithi204.stdout:1939: finishing write tid 5 to smithi20457400-15 2024-09-16T05:17:59.032 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 694 (ObjNum 697 snap 202 seq_num 697) dirty exists 2024-09-16T05:17:59.032 INFO:tasks.rados.rados.2.smithi204.stdout:1939: left oid 15 (ObjNum 697 snap 202 seq_num 697) 2024-09-16T05:17:59.032 INFO:tasks.rados.rados.2.smithi204.stdout:1940: expect (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:17:59.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:17:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:59.120 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:59.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:17:59.131 INFO:tasks.rados.rados.2.smithi204.stdout:1941: finishing write tid 1 to smithi20457400-44 2024-09-16T05:17:59.131 INFO:tasks.rados.rados.2.smithi204.stdout:1941: finishing write tid 2 to smithi20457400-44 2024-09-16T05:17:59.131 INFO:tasks.rados.rados.2.smithi204.stdout:1941: finishing write tid 3 to smithi20457400-44 2024-09-16T05:17:59.131 INFO:tasks.rados.rados.2.smithi204.stdout:1941: finishing write tid 4 to smithi20457400-44 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:1941: finishing write tid 5 to smithi20457400-44 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 522 (ObjNum 698 snap 202 seq_num 698) dirty exists 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:1941: left oid 44 (ObjNum 698 snap 202 seq_num 698) 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:1938: done (4 left) 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:1939: done (3 left) 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:1940: done (2 left) 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:1941: done (1 left) 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:1942: done (0 left) 2024-09-16T05:17:59.132 INFO:tasks.rados.rados.2.smithi204.stdout:1943: delete oid 19 current snap is 202 2024-09-16T05:17:59.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:17:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:58 smithi204 ceph-mon[30933]: osdmap e528: 12 total, 12 up, 11 in 2024-09-16T05:17:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/524964779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:17:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1696769685' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-09-16T05:17:59.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:17:59.244 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:17:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:59.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:17:59.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:17:59.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:59.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:59.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:59.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:17:59.492 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:17:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:59.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:59.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:17:59.562 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-16T05:17:59.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:17:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:59.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:59.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:17:59.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:17:59.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:59.743 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:17:59.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:17:59.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:17:59.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:59.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:17:59.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:17:59.980 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:17:59.980 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:17:59.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:17:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:17:59.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:00.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:00.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:00.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:59 smithi032 ceph-mon[27388]: pgmap v954: 107 pgs: 107 active+clean; 179 MiB data, 636 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 22 op/s 2024-09-16T05:18:00.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:59 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:00.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/524964779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:00.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1696769685' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-16T05:18:00.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:17:59 smithi032 ceph-mon[27388]: osdmap e529: 12 total, 12 up, 12 in 2024-09-16T05:18:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:59 smithi204 ceph-mon[30933]: pgmap v954: 107 pgs: 107 active+clean; 179 MiB data, 636 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 22 op/s 2024-09-16T05:18:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:59 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/524964779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1696769685' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-16T05:18:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:17:59 smithi204 ceph-mon[30933]: osdmap e529: 12 total, 12 up, 12 in 2024-09-16T05:18:00.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:00.232 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:00.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:59 smithi196 ceph-mon[30802]: pgmap v954: 107 pgs: 107 active+clean; 179 MiB data, 636 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 4.1 MiB/s wr, 22 op/s 2024-09-16T05:18:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:59 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/524964779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1696769685' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-16T05:18:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:17:59 smithi196 ceph-mon[30802]: osdmap e529: 12 total, 12 up, 12 in 2024-09-16T05:18:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:00.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:00.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:00.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:00.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:00.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:00.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:00.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:18:00.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:00.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:00.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:00.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:00.743 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:00.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:00.832 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:00.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:00.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:00.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:00.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:00.984 INFO:tasks.rados.rados.2.smithi204.stdout:1943: done (0 left) 2024-09-16T05:18:00.984 INFO:tasks.rados.rados.2.smithi204.stdout:1944: write oid 10 current snap is 202 2024-09-16T05:18:00.984 INFO:tasks.rados.rados.2.smithi204.stdout:1944: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-09-16T05:18:00.991 INFO:tasks.rados.rados.2.smithi204.stdout:1944: writing smithi20457400-10 from 581707 to 1235732 tid 1 2024-09-16T05:18:00.996 INFO:tasks.rados.rados.2.smithi204.stdout:1944: writing smithi20457400-10 from 1687642 to 2141213 tid 2 2024-09-16T05:18:01.004 INFO:tasks.rados.rados.2.smithi204.stdout:1944: writing smithi20457400-10 from 2700812 to 3404493 tid 3 2024-09-16T05:18:01.004 INFO:tasks.rados.rados.2.smithi204.stdout:1945: write (excl) oid 19 current snap is 202 2024-09-16T05:18:01.004 INFO:tasks.rados.rados.2.smithi204.stdout:1945: seq_num 702 ranges {598514=729274,1746124=597229,3128235=514953} 2024-09-16T05:18:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:01.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:01.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:01.011 INFO:tasks.rados.rados.2.smithi204.stdout:1945: writing smithi20457400-19 from 598514 to 1327788 tid 1 2024-09-16T05:18:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:1945: writing smithi20457400-19 from 1746124 to 2343353 tid 2 2024-09-16T05:18:01.024 INFO:tasks.rados.rados.2.smithi204.stdout:1945: writing smithi20457400-19 from 3128235 to 3643188 tid 3 2024-09-16T05:18:01.024 INFO:tasks.rados.rados.2.smithi204.stdout:1946: read oid 20 snap -1 2024-09-16T05:18:01.024 INFO:tasks.rados.rados.2.smithi204.stdout:1946: expect deleted 2024-09-16T05:18:01.024 INFO:tasks.rados.rados.2.smithi204.stdout:1947: delete oid 47 current snap is 202 2024-09-16T05:18:01.024 INFO:tasks.rados.rados.2.smithi204.stdout:1944: finishing write tid 1 to smithi20457400-10 2024-09-16T05:18:01.025 INFO:tasks.rados.rados.2.smithi204.stdout:1944: finishing write tid 2 to smithi20457400-10 2024-09-16T05:18:01.025 INFO:tasks.rados.rados.2.smithi204.stdout:1944: finishing write tid 3 to smithi20457400-10 2024-09-16T05:18:01.025 INFO:tasks.rados.rados.2.smithi204.stdout:1944: finishing write tid 4 to smithi20457400-10 2024-09-16T05:18:01.025 INFO:tasks.rados.rados.2.smithi204.stdout:1944: finishing write tid 5 to smithi20457400-10 2024-09-16T05:18:01.025 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 704 (ObjNum 701 snap 202 seq_num 701) dirty exists 2024-09-16T05:18:01.025 INFO:tasks.rados.rados.2.smithi204.stdout:1944: left oid 10 (ObjNum 701 snap 202 seq_num 701) 2024-09-16T05:18:01.025 INFO:tasks.rados.rados.2.smithi204.stdout:1945: finishing write tid 1 to smithi20457400-19 2024-09-16T05:18:01.025 INFO:tasks.rados.rados.2.smithi204.stdout:1945: finishing write tid 2 to smithi20457400-19 2024-09-16T05:18:01.026 INFO:tasks.rados.rados.2.smithi204.stdout:1944: done (3 left) 2024-09-16T05:18:01.026 INFO:tasks.rados.rados.2.smithi204.stdout:1946: done (2 left) 2024-09-16T05:18:01.026 INFO:tasks.rados.rados.2.smithi204.stdout:1947: done (1 left) 2024-09-16T05:18:01.026 INFO:tasks.rados.rados.2.smithi204.stdout:1948: write oid 49 current snap is 202 2024-09-16T05:18:01.026 INFO:tasks.rados.rados.2.smithi204.stdout:1945: finishing write tid 3 to smithi20457400-19 2024-09-16T05:18:01.026 INFO:tasks.rados.rados.2.smithi204.stdout:1948: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-16T05:18:01.032 INFO:tasks.rados.rados.2.smithi204.stdout:1948: writing smithi20457400-49 from 632128 to 1111900 tid 1 2024-09-16T05:18:01.038 INFO:tasks.rados.rados.2.smithi204.stdout:1948: writing smithi20457400-49 from 1746735 to 2347633 tid 2 2024-09-16T05:18:01.046 INFO:tasks.rados.rados.2.smithi204.stdout:1948: writing smithi20457400-49 from 3066728 to 3720578 tid 3 2024-09-16T05:18:01.047 INFO:tasks.rados.rados.2.smithi204.stdout:1949: snap_create 2024-09-16T05:18:01.047 INFO:tasks.rados.rados.2.smithi204.stdout:1945: finishing write tid 4 to smithi20457400-19 2024-09-16T05:18:01.047 INFO:tasks.rados.rados.2.smithi204.stdout:1945: finishing write tid 5 to smithi20457400-19 2024-09-16T05:18:01.047 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 311 (ObjNum 702 snap 202 seq_num 702) dirty exists 2024-09-16T05:18:01.047 INFO:tasks.rados.rados.2.smithi204.stdout:1945: left oid 19 (ObjNum 702 snap 202 seq_num 702) 2024-09-16T05:18:01.047 INFO:tasks.rados.rados.2.smithi204.stdout:1948: finishing write tid 1 to smithi20457400-49 2024-09-16T05:18:01.047 INFO:tasks.rados.rados.2.smithi204.stdout:1948: finishing write tid 2 to smithi20457400-49 2024-09-16T05:18:01.048 INFO:tasks.rados.rados.2.smithi204.stdout:1948: finishing write tid 3 to smithi20457400-49 2024-09-16T05:18:01.049 INFO:tasks.rados.rados.2.smithi204.stdout:1948: finishing write tid 4 to smithi20457400-49 2024-09-16T05:18:01.049 INFO:tasks.rados.rados.2.smithi204.stdout:1948: finishing write tid 5 to smithi20457400-49 2024-09-16T05:18:01.049 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 585 (ObjNum 704 snap 202 seq_num 704) dirty exists 2024-09-16T05:18:01.049 INFO:tasks.rados.rados.2.smithi204.stdout:1948: left oid 49 (ObjNum 704 snap 202 seq_num 704) 2024-09-16T05:18:01.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:01.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:00 smithi196 ceph-mon[30802]: osdmap e530: 12 total, 12 up, 12 in 2024-09-16T05:18:01.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2982299718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:01.119 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:00 smithi196 ceph-mon[30802]: osdmap e531: 12 total, 12 up, 12 in 2024-09-16T05:18:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:01.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:01.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:00 smithi032 ceph-mon[27388]: osdmap e530: 12 total, 12 up, 12 in 2024-09-16T05:18:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2982299718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:00 smithi032 ceph-mon[27388]: osdmap e531: 12 total, 12 up, 12 in 2024-09-16T05:18:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:00 smithi204 ceph-mon[30933]: osdmap e530: 12 total, 12 up, 12 in 2024-09-16T05:18:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2982299718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:00 smithi204 ceph-mon[30933]: osdmap e531: 12 total, 12 up, 12 in 2024-09-16T05:18:01.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:01.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:01.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:01.346 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:01.346 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:01.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:01.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:01.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:01.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:01.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:01.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:01.569 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:01.570 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:01.592 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:01.593 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:01.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:01.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:01.614 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:01.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:01.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:01.635 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:01.635 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:01.658 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:01.679 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:01.701 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:01.701 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:01.724 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:01.724 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:01.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:01.745 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:01.745 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:01.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:01.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:01.767 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:01.767 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:01.790 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:01.810 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:01.811 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:01.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:01.877 INFO:tasks.rados.rados.2.smithi204.stdout:1945: done (2 left) 2024-09-16T05:18:01.877 INFO:tasks.rados.rados.2.smithi204.stdout:1948: done (1 left) 2024-09-16T05:18:01.877 INFO:tasks.rados.rados.2.smithi204.stdout:1949: done (0 left) 2024-09-16T05:18:01.877 INFO:tasks.rados.rados.2.smithi204.stdout:1950: copy_from oid 1 from oid 18 current snap is 203 2024-09-16T05:18:01.877 INFO:tasks.rados.rados.2.smithi204.stdout:1951: write oid 30 current snap is 203 2024-09-16T05:18:01.877 INFO:tasks.rados.rados.2.smithi204.stdout:1951: seq_num 705 ranges {648935=555021,1805217=460909,2694151=465122,3817442=31493} 2024-09-16T05:18:01.882 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:01.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:01.883 INFO:tasks.rados.rados.2.smithi204.stdout:1951: writing smithi20457400-30 from 648935 to 1203956 tid 1 2024-09-16T05:18:01.889 INFO:tasks.rados.rados.2.smithi204.stdout:1951: writing smithi20457400-30 from 1805217 to 2266126 tid 2 2024-09-16T05:18:01.894 INFO:tasks.rados.rados.2.smithi204.stdout:1951: writing smithi20457400-30 from 2694151 to 3159273 tid 3 2024-09-16T05:18:01.899 INFO:tasks.rados.rados.2.smithi204.stdout:1951: writing smithi20457400-30 from 3817442 to 3848935 tid 4 2024-09-16T05:18:01.899 INFO:tasks.rados.rados.2.smithi204.stdout:1952: read oid 42 snap 199 2024-09-16T05:18:01.899 INFO:tasks.rados.rados.2.smithi204.stdout:1952: expect (ObjNum 671 snap 194 seq_num 671) 2024-09-16T05:18:01.899 INFO:tasks.rados.rados.2.smithi204.stdout:1953: read oid 15 snap 199 2024-09-16T05:18:01.899 INFO:tasks.rados.rados.2.smithi204.stdout:1953: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:18:01.899 INFO:tasks.rados.rados.2.smithi204.stdout:1954: rollback oid 20 current snap is 203 2024-09-16T05:18:01.899 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 20 to 193 2024-09-16T05:18:01.900 INFO:tasks.rados.rados.2.smithi204.stdout:1950: finishing copy_from racing read to smithi20457400-1 2024-09-16T05:18:01.900 INFO:tasks.rados.rados.2.smithi204.stdout:1950: finishing copy_from to smithi20457400-1 2024-09-16T05:18:01.900 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 527 (ObjNum 652 snap 189 seq_num 652) dirty exists 2024-09-16T05:18:01.900 INFO:tasks.rados.rados.2.smithi204.stdout:1950: done (4 left) 2024-09-16T05:18:01.900 INFO:tasks.rados.rados.2.smithi204.stdout:1955: snap_remove snap 202 2024-09-16T05:18:01.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:02.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:02.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:02.110 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:02.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:02.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:02.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:01 smithi032 ceph-mon[27388]: pgmap v957: 107 pgs: 107 active+clean; 181 MiB data, 638 MiB used, 983 GiB / 983 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2024-09-16T05:18:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:01 smithi204 ceph-mon[30933]: pgmap v957: 107 pgs: 107 active+clean; 181 MiB data, 638 MiB used, 983 GiB / 983 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2024-09-16T05:18:02.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:18:02.200 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:01 smithi196 ceph-mon[30802]: pgmap v957: 107 pgs: 107 active+clean; 181 MiB data, 638 MiB used, 983 GiB / 983 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2024-09-16T05:18:02.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:18:02.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:02.257 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:02.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:02.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:02.382 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:02.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:02.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:02.480 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:02.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:02.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:02.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:02.561 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:02.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:02.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:02.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:02.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:02.754 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:02.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:02.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:02.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:02.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:02.885 INFO:tasks.rados.rados.2.smithi204.stdout:1955: done (4 left) 2024-09-16T05:18:02.885 INFO:tasks.rados.rados.2.smithi204.stdout:1956: write (excl) oid 21 current snap is 203 2024-09-16T05:18:02.886 INFO:tasks.rados.rados.2.smithi204.stdout:1956: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2024-09-16T05:18:02.892 INFO:tasks.rados.rados.2.smithi204.stdout:1956: writing smithi20457400-21 from 665742 to 1296012 tid 1 2024-09-16T05:18:02.898 INFO:tasks.rados.rados.2.smithi204.stdout:1956: writing smithi20457400-21 from 1863699 to 2468266 tid 2 2024-09-16T05:18:02.906 INFO:tasks.rados.rados.2.smithi204.stdout:1956: writing smithi20457400-21 from 3121574 to 3797968 tid 3 2024-09-16T05:18:02.906 INFO:tasks.rados.rados.2.smithi204.stdout:1957: delete oid 37 current snap is 203 2024-09-16T05:18:02.906 INFO:tasks.rados.rados.2.smithi204.stdout:1951: finishing write tid 1 to smithi20457400-30 2024-09-16T05:18:02.906 INFO:tasks.rados.rados.2.smithi204.stdout:1951: finishing write tid 2 to smithi20457400-30 2024-09-16T05:18:02.906 INFO:tasks.rados.rados.2.smithi204.stdout:1951: finishing write tid 3 to smithi20457400-30 2024-09-16T05:18:02.906 INFO:tasks.rados.rados.2.smithi204.stdout:1954: finishing rollback tid 1 to smithi20457400-20 2024-09-16T05:18:02.907 INFO:tasks.rados.rados.2.smithi204.stdout:1954: finishing rollback tid 2 to smithi20457400-20 2024-09-16T05:18:02.907 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 748 (ObjNum 651 snap 189 seq_num 651) dirty exists 2024-09-16T05:18:02.907 INFO:tasks.rados.rados.2.smithi204.stdout:1951: finishing write tid 4 to smithi20457400-30 2024-09-16T05:18:02.907 INFO:tasks.rados.rados.2.smithi204.stdout:1953: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:18:02.957 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:18:02.957 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:18:02.957 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:18:02.957 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:18:02.957 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:18:02.972 INFO:tasks.rados.rados.2.smithi204.stdout:1952: expect (ObjNum 671 snap 194 seq_num 671) 2024-09-16T05:18:02.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:03.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:03.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:03.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:02 smithi032 ceph-mon[27388]: osdmap e532: 12 total, 12 up, 12 in 2024-09-16T05:18:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:02 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:02 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:02 smithi204 ceph-mon[30933]: osdmap e532: 12 total, 12 up, 12 in 2024-09-16T05:18:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:02 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:02 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:03.178 INFO:tasks.rados.rados.2.smithi204.stdout:1951: finishing write tid 5 to smithi20457400-30 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1951: finishing write tid 6 to smithi20457400-30 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 918 (ObjNum 705 snap 203 seq_num 705) dirty exists 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1951: left oid 30 (ObjNum 705 snap 203 seq_num 705) 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1956: finishing write tid 1 to smithi20457400-21 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1956: finishing write tid 2 to smithi20457400-21 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1951: done (5 left) 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1952: done (4 left) 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1953: done (3 left) 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1954: done (2 left) 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1957: done (1 left) 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1958: read oid 23 snap 191 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1958: expect (ObjNum 625 snap 182 seq_num 625) 2024-09-16T05:18:03.179 INFO:tasks.rados.rados.2.smithi204.stdout:1959: write (excl) oid 3 current snap is 203 2024-09-16T05:18:03.180 INFO:tasks.rados.rados.2.smithi204.stdout:1956: finishing write tid 3 to smithi20457400-21 2024-09-16T05:18:03.180 INFO:tasks.rados.rados.2.smithi204.stdout:1956: finishing write tid 4 to smithi20457400-21 2024-09-16T05:18:03.180 INFO:tasks.rados.rados.2.smithi204.stdout:1959: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-09-16T05:18:03.185 INFO:tasks.rados.rados.2.smithi204.stdout:1959: writing smithi20457400-3 from 699356 to 1196477 tid 1 2024-09-16T05:18:03.192 INFO:tasks.rados.rados.2.smithi204.stdout:1959: writing smithi20457400-3 from 1980663 to 2588899 tid 2 2024-09-16T05:18:03.198 INFO:tasks.rados.rados.2.smithi204.stdout:1959: writing smithi20457400-3 from 3176420 to 3591711 tid 3 2024-09-16T05:18:03.198 INFO:tasks.rados.rados.2.smithi204.stdout:1960: delete oid 26 current snap is 203 2024-09-16T05:18:03.198 INFO:tasks.rados.rados.2.smithi204.stdout:1958: expect (ObjNum 625 snap 182 seq_num 625) 2024-09-16T05:18:03.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:03.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:02 smithi196 ceph-mon[30802]: osdmap e532: 12 total, 12 up, 12 in 2024-09-16T05:18:03.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:02 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:02 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:03.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:03.308 INFO:tasks.rados.rados.2.smithi204.stdout:1959: finishing write tid 1 to smithi20457400-3 2024-09-16T05:18:03.308 INFO:tasks.rados.rados.2.smithi204.stdout:1959: finishing write tid 2 to smithi20457400-3 2024-09-16T05:18:03.309 INFO:tasks.rados.rados.2.smithi204.stdout:1959: finishing write tid 3 to smithi20457400-3 2024-09-16T05:18:03.309 INFO:tasks.rados.rados.2.smithi204.stdout:1959: finishing write tid 4 to smithi20457400-3 2024-09-16T05:18:03.309 INFO:tasks.rados.rados.2.smithi204.stdout:1959: finishing write tid 5 to smithi20457400-3 2024-09-16T05:18:03.309 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 506 (ObjNum 708 snap 203 seq_num 708) dirty exists 2024-09-16T05:18:03.309 INFO:tasks.rados.rados.2.smithi204.stdout:1959: left oid 3 (ObjNum 708 snap 203 seq_num 708) 2024-09-16T05:18:03.309 INFO:tasks.rados.rados.2.smithi204.stdout:1956: finishing write tid 5 to smithi20457400-21 2024-09-16T05:18:03.309 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 338 (ObjNum 706 snap 203 seq_num 706) dirty exists 2024-09-16T05:18:03.309 INFO:tasks.rados.rados.2.smithi204.stdout:1956: left oid 21 (ObjNum 706 snap 203 seq_num 706) 2024-09-16T05:18:03.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:03.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:03.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:03.499 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:03.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:18:03.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:03.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:03.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:03.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:03.829 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:03.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:03.872 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:03.882 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:18:03.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:03.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:03.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:04.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:04.120 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:04.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:03 smithi032 ceph-mon[27388]: pgmap v960: 107 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 remapped+peering, 103 active+clean; 183 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.7 MiB/s wr, 9 op/s; 10/180 objects misplaced (5.556%); 736 KiB/s, 11 objects/s recovering 2024-09-16T05:18:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:03 smithi032 ceph-mon[27388]: osdmap e533: 12 total, 12 up, 12 in 2024-09-16T05:18:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2059682261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:03 smithi204 ceph-mon[30933]: pgmap v960: 107 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 remapped+peering, 103 active+clean; 183 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.7 MiB/s wr, 9 op/s; 10/180 objects misplaced (5.556%); 736 KiB/s, 11 objects/s recovering 2024-09-16T05:18:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:03 smithi204 ceph-mon[30933]: osdmap e533: 12 total, 12 up, 12 in 2024-09-16T05:18:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2059682261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:04.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:03 smithi196 ceph-mon[30802]: pgmap v960: 107 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 remapped+peering, 103 active+clean; 183 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.7 MiB/s wr, 9 op/s; 10/180 objects misplaced (5.556%); 736 KiB/s, 11 objects/s recovering 2024-09-16T05:18:04.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:03 smithi196 ceph-mon[30802]: osdmap e533: 12 total, 12 up, 12 in 2024-09-16T05:18:04.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2059682261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:04.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:04.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:04.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:04.312 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:18:04.312 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:18:04.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:18:04.312 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:18:04.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.312 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(1.a,RELEASE,e:112,from:9,reservation_nonce:0)", 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 440.29236308600002, 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.42149128000000002, 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010444+0000", 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010445+0000", 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5999999999999999e-07 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.010447+0000", 2024-09-16T05:18:04.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.227e-06 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431780+0000", 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.42133351800000002 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431791+0000", 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0773e-05 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431896+0000", 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104955 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431918+0000", 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2096e-05 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:44.431936+0000", 2024-09-16T05:18:04.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7351e-05 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.1a,RELEASE,e:216,from:8,reservation_nonce:0)", 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 335.85101437200001, 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065133569000000002, 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451793+0000", 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2499999999999999e-07 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.451795+0000", 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.674e-06 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516800+0000", 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065004988 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516812+0000", 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2265e-05 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516889+0000", 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6662999999999994e-05 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516910+0000", 2024-09-16T05:18:04.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0718999999999999e-05 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:28.516927+0000", 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7034999999999999e-05 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1859 2.29 2.f7e83b69 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:18:04.321 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 303.62046305500002, 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98895853899999997, 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1859 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:18:04.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682344+0000", 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682346+0000", 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9640000000000002e-06 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682353+0000", 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2330000000000003e-06 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682369+0000", 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6721000000000001e-05 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682383+0000", 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4197e-05 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682449+0000", 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5660999999999994e-05 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:00.682458+0000", 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8000000000000004e-06 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:01.671285+0000", 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98882742000000001 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:01.671303+0000", 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7543000000000001e-05 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.326 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2673 2.29 2:9423b7dc:::smithi20457400-41:head [setxattr _header (58) in=7b,truncate 2453888] snapc 74=[74,73,72,71,70,6e,6c,6b,6a,69,67,66,64,63,62,60,5a,58] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.717815+0000", 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 189.58499208699999, 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.54320599199999997, 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.327 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2673 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717815+0000", 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717815+0000", 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717816+0000", 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8599999999999998e-07 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717819+0000", 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7800000000000001e-06 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717823+0000", 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5750000000000002e-06 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717827+0000", 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.49e-06 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.718815+0000", 2024-09-16T05:18:04.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00098811699999999995 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.718896+0000", 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0698000000000006e-05 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.718996+0000", 2024-09-16T05:18:04.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0001004 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719514+0000", 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051843099999999997 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260903+0000", 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.54138819699999996 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260957+0000", 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4215000000000002e-05 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261021+0000", 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4503000000000002e-05 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2674 2.29 2:9423b7dc:::smithi20457400-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e345)", 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.717832+0000", 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 189.58497487899999, 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.543402791, 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2674 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717832+0000", 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717832+0000", 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717833+0000", 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4400000000000001e-07 2024-09-16T05:18:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717834+0000", 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.42e-06 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717839+0000", 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7849999999999999e-06 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.717846+0000", 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8079999999999999e-06 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719362+0000", 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001516381 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719418+0000", 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5569000000000003e-05 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261065+0000", 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.541647726 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261145+0000", 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9709000000000005e-05 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261235+0000", 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0148999999999994e-05 2024-09-16T05:18:04.338 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:2673 2.29 e345/253 ondisk, result = 0)", 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.719925+0000", 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 189.58288265499999, 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.54110561499999998, 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719925+0000", 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719925+0000", 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719925+0000", 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4199999999999995e-07 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.719929+0000", 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6500000000000002e-06 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260799+0000", 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.54087024900000003 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260809+0000", 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0073e-05 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260887+0000", 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7440000000000004e-05 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.260898+0000", 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0889e-05 2024-09-16T05:18:04.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.261030+0000", 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013267199999999999 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.6.0:147 2.29 e354/253 ondisk, result = 0)", 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:18:04.343 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 180.55385634699999, 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92583168000000005, 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748952+0000", 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2099999999999996e-07 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748956+0000", 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4839999999999998e-06 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674453+0000", 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92549724300000002 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674462+0000", 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8019999999999998e-06 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674569+0000", 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107186 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674584+0000", 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.502e-05 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674783+0000", 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019832399999999999 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.347 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2845 2.29 2:94d33c56:::smithi20457400-8:head [copy-from ver 336] snapc 7b=[7b,7a,78,77,76,75,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:17.757541+0000", 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 166.54526656300001, 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012784904, 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.348 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2845 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757541+0000", 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757541+0000", 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757543+0000", 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6110000000000001e-06 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757555+0000", 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1097000000000001e-05 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757578+0000", 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3451999999999998e-05 2024-09-16T05:18:04.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757588+0000", 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0186e-05 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757655+0000", 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7131000000000005e-05 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.757738+0000", 2024-09-16T05:18:04.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3065999999999994e-05 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.764864+0000", 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071259540000000003 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.768432+0000", 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035675300000000002 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.770248+0000", 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001816462 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.770287+0000", 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9029000000000002e-05 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:17.770326+0000", 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8386000000000001e-05 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.353 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3235 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 450] snapc 8e=[8e,8d,8c,8b,8a,87,85,83,7c] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:55.718646+0000", 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 128.584160897, 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020234347, 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.354 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3235 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718646+0000", 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718646+0000", 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718648+0000", 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8649999999999999e-06 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718657+0000", 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7499999999999992e-06 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718671+0000", 2024-09-16T05:18:04.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4085e-05 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718680+0000", 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1090000000000004e-06 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.719983+0000", 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001302625 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.720146+0000", 2024-09-16T05:18:04.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016294500000000001 2024-09-16T05:18:04.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.730144+0000", 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0099980529999999998 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.734998+0000", 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048545769999999997 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.738811+0000", 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038123609999999998 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.738849+0000", 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7799999999999997e-05 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.738881+0000", 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2177e-05 2024-09-16T05:18:04.361 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3236 2.29 2:9662c29c:::smithi20457400-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e406)", 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:55.719596+0000", 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 128.583211347, 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01480297, 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3236 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.719596+0000", 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.719596+0000", 2024-09-16T05:18:04.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.719597+0000", 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.71e-07 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.719786+0000", 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018862600000000001 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.719801+0000", 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5622000000000001e-05 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.719810+0000", 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8710000000000003e-06 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.720306+0000", 2024-09-16T05:18:04.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00049550400000000002 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.720355+0000", 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9011999999999998e-05 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.734242+0000", 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01388766 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.734317+0000", 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5174999999999998e-05 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.734399+0000", 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1629000000000001e-05 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3544 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 351] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:18:04.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.725348+0000", 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.577459021999999, 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021240347999999999, 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.368 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3544 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725348+0000", 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725348+0000", 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725350+0000", 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8679999999999999e-06 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725357+0000", 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0060000000000003e-06 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725371+0000", 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3981e-05 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725381+0000", 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4199999999999996e-06 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.726955+0000", 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015744610000000001 2024-09-16T05:18:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.727068+0000", 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112963 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.740216+0000", 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013147808 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:18:04.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.743799+0000", 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035831589999999998 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.746481+0000", 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026815620000000002 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.746542+0000", 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1815000000000004e-05 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.746589+0000", 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6304999999999998e-05 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3545 2.29 2:9662c29c:::smithi20457400-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.725398+0000", 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 98.577408980000001, 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018036086999999999, 2024-09-16T05:18:04.374 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3545 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725398+0000", 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725398+0000", 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725399+0000", 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3600000000000003e-07 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725402+0000", 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.19e-06 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725410+0000", 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9209999999999992e-06 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725417+0000", 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.426e-06 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.727335+0000", 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019184009999999999 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.727395+0000", 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9466000000000003e-05 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.743208+0000", 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015813105000000001 2024-09-16T05:18:04.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.743325+0000", 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117784 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.743434+0000", 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109058 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.11,GRANT,e:436,from:4,reservation_nonce:0)", 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:26.377804+0000", 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 97.925003394000001, 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.244246716, 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.377804+0000", 2024-09-16T05:18:04.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.377804+0000", 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.377805+0000", 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3710000000000001e-06 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.377810+0000", 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3370000000000001e-06 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.621781+0000", 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.24397083999999999 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.621791+0000", 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0142e-05 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.621882+0000", 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1050000000000004e-05 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.621900+0000", 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8349e-05 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.622051+0000", 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015062699999999999 2024-09-16T05:18:04.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4414 2.3f e499/293 2:fc733d2f:::smithi20457400-45:head v 499'316, mlcod=472'312)", 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 35.552820719000003, 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94419809600000004, 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:18:04.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8500000000000002e-07 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.751332+0000", 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001345001 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691571+0000", 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94023934399999998 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691580+0000", 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691653+0000", 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3137000000000001e-05 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691676+0000", 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3349e-05 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694130+0000", 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024536900000000001 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694185+0000", 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4750000000000003e-05 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4417 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 329] snapc ba=[ba,b9,b8] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 34.558693534, 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017204743000000002, 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4417 2024-09-16T05:18:04.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744116+0000", 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4099999999999998e-06 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744129+0000", 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2717e-05 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744150+0000", 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1246e-05 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744160+0000", 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.815e-06 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744237+0000", 2024-09-16T05:18:04.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6860000000000006e-05 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744246+0000", 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1220000000000001e-06 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.745711+0000", 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014654679999999999 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.745823+0000", 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011192300000000001 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.755847+0000", 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010023119 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:18:04.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.759255+0000", 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034084950000000001 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761242+0000", 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019874369999999999 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761284+0000", 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1984000000000003e-05 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761319+0000", 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4146999999999997e-05 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4418 2.29 2:96dc17ef:::smithi20457400-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e500)", 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:29.744185+0000", 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 34.558622354000001, 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014670192, 2024-09-16T05:18:04.396 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4418 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744185+0000", 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744185+0000", 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744186+0000", 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0699999999999997e-07 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744189+0000", 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3129999999999999e-06 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744199+0000", 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0147e-05 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744206+0000", 2024-09-16T05:18:04.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0129999999999996e-06 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744310+0000", 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000103507 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744317+0000", 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1409999999999999e-06 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.745968+0000", 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016512530000000001 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.746015+0000", 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6853e-05 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.758640+0000", 2024-09-16T05:18:04.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012625246 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.758729+0000", 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8437999999999996e-05 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.758855+0000", 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012667399999999999 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.7.0:741 2.2d e509/252 2:b7234079:::smithi20457400-42:bc v 509'382, mlcod=507'378)", 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.274359086, 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72794596499999997, 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028449+0000", 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5400000000000001e-07 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028455+0000", 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4339999999999997e-06 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755790+0000", 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72733470600000005 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755802+0000", 2024-09-16T05:18:04.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1771e-05 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755868+0000", 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6512000000000003e-05 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755889+0000", 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0245e-05 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.756344+0000", 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045536699999999998 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.756394+0000", 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0275999999999998e-05 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4547 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 303] snapc be=[be] ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-16T05:18:04.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.535086444000001, 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01820447, 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4547 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767723+0000", 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0549999999999998e-06 2024-09-16T05:18:04.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767732+0000", 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9930000000000001e-06 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767748+0000", 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5715000000000001e-05 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767764+0000", 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5811000000000002e-05 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767896+0000", 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013272800000000001 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767969+0000", 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3120999999999993e-05 2024-09-16T05:18:04.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.780323+0000", 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012353563 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.782979+0000", 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026563440000000002 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:04.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785858+0000", 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028785719999999998 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785889+0000", 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1569000000000002e-05 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785925+0000", 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5998999999999999e-05 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4548 2.29 2:9662c29c:::smithi20457400-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e510)", 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:39.767783+0000", 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.535023916, 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014861549999999999, 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:04.414 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4548 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767783+0000", 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767783+0000", 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767784+0000", 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7599999999999997e-07 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767787+0000", 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9380000000000001e-06 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767795+0000", 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8369999999999997e-06 2024-09-16T05:18:04.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767801+0000", 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7760000000000003e-06 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.768110+0000", 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000309077 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:18:04.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.768152+0000", 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2833000000000001e-05 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.782506+0000", 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014353714 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.782573+0000", 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6557000000000007e-05 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.782645+0000", 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2242000000000004e-05 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.8.0:932 2.1a e526/291 2:5827f14d:::smithi20457400-12:c3 v 526'716, mlcod=518'711)", 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 8.5444368419999996, 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.24009196499999999, 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:18:04.419 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3300000000000001e-07 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758373+0000", 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1909999999999999e-06 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997774+0000", 2024-09-16T05:18:04.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.23940144999999999 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997782+0000", 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7379999999999994e-06 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997836+0000", 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3557000000000002e-05 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:18:04.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997855+0000", 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9491999999999999e-05 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.998416+0000", 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056088899999999996 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.998462+0000", 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6514999999999998e-05 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.423 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:18:04.424 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:18:04.424 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:18:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:04.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:04.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:04.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:04.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:18:04.554 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:04.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:04.563 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:18:04.563 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:18:04.563 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-16T05:18:04.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-16T05:18:04.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:04.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:04.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:04.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:04.783 INFO:tasks.rados.rados.2.smithi204.stdout:1956: done (3 left) 2024-09-16T05:18:04.783 INFO:tasks.rados.rados.2.smithi204.stdout:1958: done (2 left) 2024-09-16T05:18:04.783 INFO:tasks.rados.rados.2.smithi204.stdout:1959: done (1 left) 2024-09-16T05:18:04.783 INFO:tasks.rados.rados.2.smithi204.stdout:1960: done (0 left) 2024-09-16T05:18:04.783 INFO:tasks.rados.rados.2.smithi204.stdout:1961: read oid 50 snap -1 2024-09-16T05:18:04.783 INFO:tasks.rados.rados.2.smithi204.stdout:1961: expect (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:18:04.784 INFO:tasks.rados.rados.2.smithi204.stdout:1962: snap_remove snap 197 2024-09-16T05:18:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:04.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:04.901 INFO:tasks.rados.rados.2.smithi204.stdout:1962: done (1 left) 2024-09-16T05:18:04.901 INFO:tasks.rados.rados.2.smithi204.stdout:1963: snap_create 2024-09-16T05:18:04.902 INFO:tasks.rados.rados.2.smithi204.stdout:1961: expect (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:18:04.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:04.933 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:04.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:04 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2059682261' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:04.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:04 smithi196 ceph-mon[30802]: osdmap e534: 12 total, 12 up, 12 in 2024-09-16T05:18:05.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:05.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:05.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:05.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:18:05.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:04 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2059682261' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:04 smithi032 ceph-mon[27388]: osdmap e534: 12 total, 12 up, 12 in 2024-09-16T05:18:05.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:04 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:05.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2059682261' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:05.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:04 smithi204 ceph-mon[30933]: osdmap e534: 12 total, 12 up, 12 in 2024-09-16T05:18:05.180 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:05.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:05.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:05.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:05.321 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:05.373 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:05.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:05.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:05.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:05.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:05.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:05.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:05.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:05.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:05.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:05.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:05.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:05.888 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:18:05.888 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:18:05.888 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:18:05.888 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:18:05.906 INFO:tasks.rados.rados.2.smithi204.stdout:1961: done (1 left) 2024-09-16T05:18:05.906 INFO:tasks.rados.rados.2.smithi204.stdout:1963: done (0 left) 2024-09-16T05:18:05.906 INFO:tasks.rados.rados.2.smithi204.stdout:1964: read oid 21 snap -1 2024-09-16T05:18:05.907 INFO:tasks.rados.rados.2.smithi204.stdout:1964: expect (ObjNum 706 snap 203 seq_num 706) 2024-09-16T05:18:05.907 INFO:tasks.rados.rados.2.smithi204.stdout:1965: rollback oid 6 current snap is 204 2024-09-16T05:18:05.907 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 6 to 195 2024-09-16T05:18:05.907 INFO:tasks.rados.rados.2.smithi204.stdout:1966: snap_create 2024-09-16T05:18:05.907 INFO:tasks.rados.rados.2.smithi204.stdout:1965: finishing rollback tid 1 to smithi20457400-6 2024-09-16T05:18:05.908 INFO:tasks.rados.rados.2.smithi204.stdout:1965: finishing rollback tid 2 to smithi20457400-6 2024-09-16T05:18:05.908 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 351 (ObjNum 621 snap 181 seq_num 621) dirty exists 2024-09-16T05:18:05.918 INFO:tasks.rados.rados.2.smithi204.stdout:1964: expect (ObjNum 706 snap 203 seq_num 706) 2024-09-16T05:18:05.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:05.945 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:05.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout:moved 0 / 214 (0%) 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout:avg 17.8333 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout:stddev 2.9392 -> 2.9392 (expected baseline 4.04317) 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout:min osd.0 with 13 -> 13 pgs (0.728972 -> 0.728972 * mean) 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout:max osd.7 with 24 -> 24 pgs (1.34579 -> 1.34579 * mean) 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout:oload 1000 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout:max_change 0.05 2024-09-16T05:18:05.976 INFO:teuthology.orchestra.run.smithi032.stdout:max_change_osds 4 2024-09-16T05:18:05.977 INFO:teuthology.orchestra.run.smithi032.stdout:average_utilization 0.0006 2024-09-16T05:18:05.977 INFO:teuthology.orchestra.run.smithi032.stdout:overload_utilization 0.0065 2024-09-16T05:18:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:no change 2024-09-16T05:18:06.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:06.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:06.132 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:06.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: pgmap v963: 107 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 remapped+peering, 103 active+clean; 191 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 14 op/s; 10/186 objects misplaced (5.376%); 2.0 MiB/s, 14 objects/s recovering 2024-09-16T05:18:06.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: osdmap e535: 12 total, 12 up, 12 in 2024-09-16T05:18:06.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:06.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:06.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:18:06.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:18:06.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2685525105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:06.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: 2.55 scrub starts 2024-09-16T05:18:06.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: 2.55 scrub ok 2024-09-16T05:18:06.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:18:06.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:05 smithi032 ceph-mon[27388]: 2.44 scrub ok 2024-09-16T05:18:06.154 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:06.154 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:06.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: pgmap v963: 107 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 remapped+peering, 103 active+clean; 191 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 14 op/s; 10/186 objects misplaced (5.376%); 2.0 MiB/s, 14 objects/s recovering 2024-09-16T05:18:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: osdmap e535: 12 total, 12 up, 12 in 2024-09-16T05:18:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:18:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:18:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2685525105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: 2.55 scrub starts 2024-09-16T05:18:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: 2.55 scrub ok 2024-09-16T05:18:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:18:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:05 smithi204 ceph-mon[30933]: 2.44 scrub ok 2024-09-16T05:18:06.174 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:06.175 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:06.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:06.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:06.195 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:06.196 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:06.217 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:06.217 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:06.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: pgmap v963: 107 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 remapped+peering, 103 active+clean; 191 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 14 op/s; 10/186 objects misplaced (5.376%); 2.0 MiB/s, 14 objects/s recovering 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: osdmap e535: 12 total, 12 up, 12 in 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2685525105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: 2.55 scrub starts 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: 2.55 scrub ok 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:18:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:05 smithi196 ceph-mon[30802]: 2.44 scrub ok 2024-09-16T05:18:06.239 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:06.239 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:06.260 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:06.260 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:06.281 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:06.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:06.302 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:06.302 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:06.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:06.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:06.328 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:06.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:06.350 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:06.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:06.372 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:06.372 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:06.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:06.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:06.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:06.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:06.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:18:06.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:06.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:06.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:18:06.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:06.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:06.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:06.708 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-16T05:18:06.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-16T05:18:06.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:06.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:06.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:06.869 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:06.912 INFO:tasks.rados.rados.2.smithi204.stdout:1964: done (2 left) 2024-09-16T05:18:06.912 INFO:tasks.rados.rados.2.smithi204.stdout:1965: done (1 left) 2024-09-16T05:18:06.912 INFO:tasks.rados.rados.2.smithi204.stdout:1966: done (0 left) 2024-09-16T05:18:06.912 INFO:tasks.rados.rados.2.smithi204.stdout:1967: read oid 30 snap -1 2024-09-16T05:18:06.912 INFO:tasks.rados.rados.2.smithi204.stdout:1967: expect (ObjNum 705 snap 203 seq_num 705) 2024-09-16T05:18:06.912 INFO:tasks.rados.rados.2.smithi204.stdout:1968: snap_create 2024-09-16T05:18:06.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:06.919 INFO:tasks.rados.rados.2.smithi204.stdout:1967: expect (ObjNum 705 snap 203 seq_num 705) 2024-09-16T05:18:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:06.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:06.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:07.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:07.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:07.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:07.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:06 smithi032 ceph-mon[27388]: 2.51 scrub starts 2024-09-16T05:18:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:06 smithi032 ceph-mon[27388]: 2.51 scrub ok 2024-09-16T05:18:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:06 smithi032 ceph-mon[27388]: osdmap e536: 12 total, 12 up, 12 in 2024-09-16T05:18:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:06 smithi032 ceph-mon[27388]: from='client.16236 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:06 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:18:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:06 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:18:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:06 smithi032 ceph-mon[27388]: 2.54 scrub starts 2024-09-16T05:18:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:06 smithi032 ceph-mon[27388]: 2.54 scrub ok 2024-09-16T05:18:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:06 smithi204 ceph-mon[30933]: 2.51 scrub starts 2024-09-16T05:18:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:06 smithi204 ceph-mon[30933]: 2.51 scrub ok 2024-09-16T05:18:07.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:06 smithi204 ceph-mon[30933]: osdmap e536: 12 total, 12 up, 12 in 2024-09-16T05:18:07.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:06 smithi204 ceph-mon[30933]: from='client.16236 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:07.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:06 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:18:07.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:06 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:18:07.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:06 smithi204 ceph-mon[30933]: 2.54 scrub starts 2024-09-16T05:18:07.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:06 smithi204 ceph-mon[30933]: 2.54 scrub ok 2024-09-16T05:18:07.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:07.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:06 smithi196 ceph-mon[30802]: 2.51 scrub starts 2024-09-16T05:18:07.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:06 smithi196 ceph-mon[30802]: 2.51 scrub ok 2024-09-16T05:18:07.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:06 smithi196 ceph-mon[30802]: osdmap e536: 12 total, 12 up, 12 in 2024-09-16T05:18:07.173 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:06 smithi196 ceph-mon[30802]: from='client.16236 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:07.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:06 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:18:07.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:06 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:18:07.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:06 smithi196 ceph-mon[30802]: 2.54 scrub starts 2024-09-16T05:18:07.174 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:06 smithi196 ceph-mon[30802]: 2.54 scrub ok 2024-09-16T05:18:07.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:07.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:07.274 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:07.274 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:07.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:07.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:07.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:07.399 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:07.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:07.423 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:07.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:07.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:07.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:07.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:07.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:07.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:07.688 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:07.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:07.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:07.812 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:07.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:07.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:07.918 INFO:tasks.rados.rados.2.smithi204.stdout:1967: done (1 left) 2024-09-16T05:18:07.918 INFO:tasks.rados.rados.2.smithi204.stdout:1968: done (0 left) 2024-09-16T05:18:07.918 INFO:tasks.rados.rados.2.smithi204.stdout:1969: write oid 34 current snap is 206 2024-09-16T05:18:07.919 INFO:tasks.rados.rados.2.smithi204.stdout:1969: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2024-09-16T05:18:07.921 INFO:tasks.rados.rados.2.smithi204.stdout:1969: writing smithi20457400-34 from 732970 to 1380589 tid 1 2024-09-16T05:18:07.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: pgmap v966: 107 pgs: 107 active+clean; 206 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.6 MiB/s wr, 21 op/s; 2.1 MiB/s, 4 objects/s recovering 2024-09-16T05:18:07.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:18:07.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:18:07.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: osdmap e537: 12 total, 12 up, 12 in 2024-09-16T05:18:07.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:18:07.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:18:07.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:18:07.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:18:07.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: 2.4f scrub starts 2024-09-16T05:18:07.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:07 smithi196 ceph-mon[30802]: 2.4f scrub ok 2024-09-16T05:18:07.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:18:07.929 INFO:tasks.rados.rados.2.smithi204.stdout:1969: writing smithi20457400-34 from 2097627 to 2709532 tid 2 2024-09-16T05:18:07.935 INFO:tasks.rados.rados.2.smithi204.stdout:1969: writing smithi20457400-34 from 3231266 to 3669101 tid 3 2024-09-16T05:18:07.935 INFO:tasks.rados.rados.2.smithi204.stdout:1970: copy_from oid 47 from oid 49 current snap is 206 2024-09-16T05:18:07.935 INFO:tasks.rados.rados.2.smithi204.stdout:1969: finishing write tid 1 to smithi20457400-34 2024-09-16T05:18:07.935 INFO:tasks.rados.rados.2.smithi204.stdout:1969: finishing write tid 2 to smithi20457400-34 2024-09-16T05:18:07.935 INFO:tasks.rados.rados.2.smithi204.stdout:1971: read oid 45 snap -1 2024-09-16T05:18:07.936 INFO:tasks.rados.rados.2.smithi204.stdout:1971: expect (ObjNum 476 snap 150 seq_num 476) 2024-09-16T05:18:07.936 INFO:tasks.rados.rados.2.smithi204.stdout:1972: delete oid 42 current snap is 206 2024-09-16T05:18:07.938 INFO:tasks.rados.rados.2.smithi204.stdout:1969: finishing write tid 3 to smithi20457400-34 2024-09-16T05:18:07.938 INFO:tasks.rados.rados.2.smithi204.stdout:1969: finishing write tid 4 to smithi20457400-34 2024-09-16T05:18:07.938 INFO:tasks.rados.rados.2.smithi204.stdout:1969: finishing write tid 5 to smithi20457400-34 2024-09-16T05:18:07.938 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 699 (ObjNum 710 snap 206 seq_num 710) dirty exists 2024-09-16T05:18:07.938 INFO:tasks.rados.rados.2.smithi204.stdout:1969: left oid 34 (ObjNum 710 snap 206 seq_num 710) 2024-09-16T05:18:07.941 INFO:tasks.rados.rados.2.smithi204.stdout:1971: expect (ObjNum 476 snap 150 seq_num 476) 2024-09-16T05:18:07.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:07.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:08.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:08.067 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:08.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:08.072 INFO:tasks.rados.rados.2.smithi204.stdout:1970: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:18:08.072 INFO:tasks.rados.rados.2.smithi204.stdout:1970: finishing copy_from to smithi20457400-47 2024-09-16T05:18:08.072 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 585 (ObjNum 704 snap 202 seq_num 704) dirty exists 2024-09-16T05:18:08.072 INFO:tasks.rados.rados.2.smithi204.stdout:1969: done (3 left) 2024-09-16T05:18:08.072 INFO:tasks.rados.rados.2.smithi204.stdout:1970: done (2 left) 2024-09-16T05:18:08.072 INFO:tasks.rados.rados.2.smithi204.stdout:1971: done (1 left) 2024-09-16T05:18:08.072 INFO:tasks.rados.rados.2.smithi204.stdout:1972: done (0 left) 2024-09-16T05:18:08.073 INFO:tasks.rados.rados.2.smithi204.stdout:1973: copy_from oid 10 from oid 8 current snap is 206 2024-09-16T05:18:08.073 INFO:tasks.rados.rados.2.smithi204.stdout:1974: rollback oid 45 current snap is 206 2024-09-16T05:18:08.073 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 200 2024-09-16T05:18:08.073 INFO:tasks.rados.rados.2.smithi204.stdout:1975: write oid 1 current snap is 206 2024-09-16T05:18:08.073 INFO:tasks.rados.rados.2.smithi204.stdout:1975: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-09-16T05:18:08.080 INFO:tasks.rados.rados.2.smithi204.stdout:1975: writing smithi20457400-1 from 766584 to 1564701 tid 1 2024-09-16T05:18:08.086 INFO:tasks.rados.rados.2.smithi204.stdout:1975: writing smithi20457400-1 from 2098238 to 2713812 tid 2 2024-09-16T05:18:08.091 INFO:tasks.rados.rados.2.smithi204.stdout:1975: writing smithi20457400-1 from 3169759 to 3630138 tid 3 2024-09-16T05:18:08.091 INFO:tasks.rados.rados.2.smithi204.stdout:1976: snap_remove snap 192 2024-09-16T05:18:08.166 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:08.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: pgmap v966: 107 pgs: 107 active+clean; 206 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.6 MiB/s wr, 21 op/s; 2.1 MiB/s, 4 objects/s recovering 2024-09-16T05:18:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:18:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:18:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: osdmap e537: 12 total, 12 up, 12 in 2024-09-16T05:18:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:18:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:18:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:18:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:18:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: 2.4f scrub starts 2024-09-16T05:18:08.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:07 smithi032 ceph-mon[27388]: 2.4f scrub ok 2024-09-16T05:18:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: pgmap v966: 107 pgs: 107 active+clean; 206 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.6 MiB/s wr, 21 op/s; 2.1 MiB/s, 4 objects/s recovering 2024-09-16T05:18:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:18:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:18:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: osdmap e537: 12 total, 12 up, 12 in 2024-09-16T05:18:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:18:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:18:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:18:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:18:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: 2.4f scrub starts 2024-09-16T05:18:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:07 smithi204 ceph-mon[30933]: 2.4f scrub ok 2024-09-16T05:18:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:08.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:08.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:08.291 INFO:teuthology.orchestra.run.smithi032.stdout:moved 7 / 214 (3.27103%) 2024-09-16T05:18:08.291 INFO:teuthology.orchestra.run.smithi032.stdout:avg 17.8333 2024-09-16T05:18:08.291 INFO:teuthology.orchestra.run.smithi032.stdout:stddev 2.9392 -> 3.3375 (expected baseline 4.04317) 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:min osd.0 with 13 -> 13 pgs (0.728972 -> 0.728972 * mean) 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:max osd.7 with 24 -> 25 pgs (1.34579 -> 1.40187 * mean) 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:oload 110 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:max_change 3 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:max_change_osds 4 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:average_utilization 0.0007 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:overload_utilization 0.0007 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8 weight 1.0000 -> 0.7818 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10 weight 1.0000 -> 0.8181 2024-09-16T05:18:08.292 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11 weight 1.0000 -> 0.8737 2024-09-16T05:18:08.293 INFO:teuthology.orchestra.run.smithi032.stderr:no change 2024-09-16T05:18:08.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:08.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:08.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:08.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:08.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:08.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:08.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:08.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:08.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:08.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:08.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:08.584 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:18:08.289+0000 7fe098731640 -1 mgr.server reply reply (3) No such process no change 2024-09-16T05:18:08.655 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:08.655 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:08.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:08.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:08.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:08.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:08.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:08.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:08.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:08.914 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:18:08.923 INFO:tasks.rados.rados.2.smithi204.stdout:1976: done (3 left) 2024-09-16T05:18:08.924 INFO:tasks.rados.rados.2.smithi204.stdout:1977: copy_from oid 48 from oid 50 current snap is 206 2024-09-16T05:18:08.924 INFO:tasks.rados.rados.2.smithi204.stdout:1978: read oid 33 snap -1 2024-09-16T05:18:08.924 INFO:tasks.rados.rados.2.smithi204.stdout:1978: expect (ObjNum 657 snap 191 seq_num 657) 2024-09-16T05:18:08.924 INFO:tasks.rados.rados.2.smithi204.stdout:1979: write oid 7 current snap is 206 2024-09-16T05:18:08.924 INFO:tasks.rados.rados.2.smithi204.stdout:1979: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2024-09-16T05:18:08.931 INFO:tasks.rados.rados.2.smithi204.stdout:1979: writing smithi20457400-7 from 783391 to 1256757 tid 1 2024-09-16T05:18:08.937 INFO:tasks.rados.rados.2.smithi204.stdout:1979: writing smithi20457400-7 from 1756720 to 2232305 tid 2 2024-09-16T05:18:08.947 INFO:tasks.rados.rados.2.smithi204.stdout:1979: writing smithi20457400-7 from 2797182 to 3585186 tid 3 2024-09-16T05:18:08.947 INFO:tasks.rados.rados.2.smithi204.stdout:1980: delete oid 20 current snap is 206 2024-09-16T05:18:08.947 INFO:tasks.rados.rados.2.smithi204.stdout:1974: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:18:08.947 INFO:tasks.rados.rados.2.smithi204.stdout:1973: finishing copy_from to smithi20457400-10 2024-09-16T05:18:08.947 INFO:tasks.rados.rados.2.smithi204.stdout:1973: got expected ENOENT (src dne) 2024-09-16T05:18:08.947 INFO:tasks.rados.rados.2.smithi204.stdout:1973: finishing copy_from racing read to smithi20457400-10 2024-09-16T05:18:08.947 INFO:tasks.rados.rados.2.smithi204.stdout:1974: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1974: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 352 (ObjNum 476 snap 150 seq_num 476) dirty exists 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1975: finishing write tid 1 to smithi20457400-1 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1975: finishing write tid 2 to smithi20457400-1 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1975: finishing write tid 3 to smithi20457400-1 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1975: finishing write tid 4 to smithi20457400-1 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1975: finishing write tid 5 to smithi20457400-1 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 531 (ObjNum 712 snap 206 seq_num 712) dirty exists 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1975: left oid 1 (ObjNum 712 snap 206 seq_num 712) 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1979: finishing write tid 1 to smithi20457400-7 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1977: finishing copy_from racing read to smithi20457400-48 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1979: finishing write tid 2 to smithi20457400-7 2024-09-16T05:18:08.948 INFO:tasks.rados.rados.2.smithi204.stdout:1977: finishing copy_from to smithi20457400-48 2024-09-16T05:18:08.949 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 396 (ObjNum 592 snap 180 seq_num 592) dirty exists 2024-09-16T05:18:08.949 INFO:tasks.rados.rados.2.smithi204.stdout:1973: done (6 left) 2024-09-16T05:18:08.949 INFO:tasks.rados.rados.2.smithi204.stdout:1974: done (5 left) 2024-09-16T05:18:08.949 INFO:tasks.rados.rados.2.smithi204.stdout:1975: done (4 left) 2024-09-16T05:18:08.949 INFO:tasks.rados.rados.2.smithi204.stdout:1977: done (3 left) 2024-09-16T05:18:08.949 INFO:tasks.rados.rados.2.smithi204.stdout:1980: done (2 left) 2024-09-16T05:18:08.949 INFO:tasks.rados.rados.2.smithi204.stdout:1981: snap_create 2024-09-16T05:18:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:08.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:08.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:08.951 INFO:tasks.rados.rados.2.smithi204.stdout:1979: finishing write tid 3 to smithi20457400-7 2024-09-16T05:18:08.951 INFO:tasks.rados.rados.2.smithi204.stdout:1979: finishing write tid 4 to smithi20457400-7 2024-09-16T05:18:08.952 INFO:tasks.rados.rados.2.smithi204.stdout:1979: finishing write tid 5 to smithi20457400-7 2024-09-16T05:18:08.952 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 409 (ObjNum 713 snap 206 seq_num 713) dirty exists 2024-09-16T05:18:08.952 INFO:tasks.rados.rados.2.smithi204.stdout:1979: left oid 7 (ObjNum 713 snap 206 seq_num 713) 2024-09-16T05:18:09.035 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-16T05:18:09.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-16T05:18:09.047 INFO:tasks.rados.rados.2.smithi204.stdout:1978: expect (ObjNum 657 snap 191 seq_num 657) 2024-09-16T05:18:09.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:09.074 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:09.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:18:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:18:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-mon[27388]: osdmap e538: 12 total, 12 up, 12 in 2024-09-16T05:18:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:18:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:18:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-mon[27388]: 2.53 scrub starts 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-mon[27388]: 2.53 scrub ok 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1824648317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:08 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:08 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:08 smithi204 ceph-mon[30933]: osdmap e538: 12 total, 12 up, 12 in 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:08 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:08 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:08 smithi204 ceph-mon[30933]: 2.53 scrub starts 2024-09-16T05:18:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:08 smithi204 ceph-mon[30933]: 2.53 scrub ok 2024-09-16T05:18:09.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1824648317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:09.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:09.198 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:09.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:09.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:08 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:18:09.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:08 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:18:09.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:08 smithi196 ceph-mon[30802]: osdmap e538: 12 total, 12 up, 12 in 2024-09-16T05:18:09.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:08 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:18:09.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:08 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:18:09.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:08 smithi196 ceph-mon[30802]: 2.53 scrub starts 2024-09-16T05:18:09.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:08 smithi196 ceph-mon[30802]: 2.53 scrub ok 2024-09-16T05:18:09.235 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1824648317' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:09.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:18:09.299 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:09.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:09.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:09.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:18:09.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:09.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:09.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:09.508 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:09.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:09.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:09.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:09.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:09.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:09.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:09.733 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:09.737 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:09.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:09.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:09.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: pgmap v969: 107 pgs: 107 active+clean; 208 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 790 KiB/s, 1 objects/s recovering 2024-09-16T05:18:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: from='client.16242 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.2a scrub starts 2024-09-16T05:18:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.2a scrub ok 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1824648317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: osdmap e539: 12 total, 12 up, 12 in 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:18:09.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:09 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:18:09.927 INFO:tasks.rados.rados.2.smithi204.stdout:1978: done (2 left) 2024-09-16T05:18:09.927 INFO:tasks.rados.rados.2.smithi204.stdout:1979: done (1 left) 2024-09-16T05:18:09.927 INFO:tasks.rados.rados.2.smithi204.stdout:1981: done (0 left) 2024-09-16T05:18:09.927 INFO:tasks.rados.rados.2.smithi204.stdout:1982: delete oid 4 current snap is 207 2024-09-16T05:18:09.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:09.951 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:09.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:09.997 INFO:tasks.rados.rados.2.smithi204.stdout:1982: done (0 left) 2024-09-16T05:18:09.997 INFO:tasks.rados.rados.2.smithi204.stdout:1983: copy_from oid 6 from oid 11 current snap is 207 2024-09-16T05:18:09.997 INFO:tasks.rados.rados.2.smithi204.stdout:1984: read oid 4 snap 193 2024-09-16T05:18:09.998 INFO:tasks.rados.rados.2.smithi204.stdout:1984: expect (ObjNum 529 snap 159 seq_num 529) 2024-09-16T05:18:09.998 INFO:tasks.rados.rados.2.smithi204.stdout:1985: write (excl) oid 21 current snap is 207 2024-09-16T05:18:09.998 INFO:tasks.rados.rados.2.smithi204.stdout:1985: seq_num 716 ranges {433812=415466,1648519=385293} 2024-09-16T05:18:10.002 INFO:tasks.rados.rados.2.smithi204.stdout:1985: writing smithi20457400-21 from 433812 to 849278 tid 1 2024-09-16T05:18:10.009 INFO:tasks.rados.rados.2.smithi204.stdout:1985: writing smithi20457400-21 from 1648519 to 2033812 tid 2 2024-09-16T05:18:10.009 INFO:tasks.rados.rados.2.smithi204.stdout:1986: snap_remove snap 204 2024-09-16T05:18:10.021 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:10.021 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:10.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:10.086 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:10.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: pgmap v969: 107 pgs: 107 active+clean; 208 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 790 KiB/s, 1 objects/s recovering 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: from='client.16242 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.2a scrub starts 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.2a scrub ok 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1824648317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: osdmap e539: 12 total, 12 up, 12 in 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:18:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:18:10.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:18:10.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:18:10.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:18:10.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:09 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:18:10.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:10.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: pgmap v969: 107 pgs: 107 active+clean; 208 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 790 KiB/s, 1 objects/s recovering 2024-09-16T05:18:10.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: from='client.16242 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:10.203 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.2a scrub starts 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.2a scrub ok 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1824648317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: osdmap e539: 12 total, 12 up, 12 in 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:18:10.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:18:10.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:09 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:18:10.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:10.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:10.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:10.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:10.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:10.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:10.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:10.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:10.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:10.468 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:10.469 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:10.490 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:10.490 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:10.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:10.511 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:10.531 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:10.554 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:10.554 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:10.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:10.576 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:10.576 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:10.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:10.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:10.598 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:10.598 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:10.620 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:10.621 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:10.643 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:10.643 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:10.665 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:10.666 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:10.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:10.688 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:10.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:10.709 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:10.709 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:10.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:10.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:10.754 INFO:tasks.rados.rados.2.smithi204.stdout:1986: done (3 left) 2024-09-16T05:18:10.754 INFO:tasks.rados.rados.2.smithi204.stdout:1987: copy_from oid 33 from oid 26 current snap is 207 2024-09-16T05:18:10.754 INFO:tasks.rados.rados.2.smithi204.stdout:1984: expect (ObjNum 529 snap 159 seq_num 529) 2024-09-16T05:18:10.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:10.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:18:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:10.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:10.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:moved 0 / 214 (0%) 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:avg 17.8333 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:stddev 2.9392 -> 2.9392 (expected baseline 4.04317) 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:min osd.0 with 13 -> 13 pgs (0.728972 -> 0.728972 * mean) 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:max osd.7 with 24 -> 24 pgs (1.34579 -> 1.34579 * mean) 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:oload 1000 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:max_change 3 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:max_change_osds 4 2024-09-16T05:18:10.887 INFO:teuthology.orchestra.run.smithi032.stdout:average_utilization 0.0007 2024-09-16T05:18:10.888 INFO:teuthology.orchestra.run.smithi032.stdout:overload_utilization 0.0066 2024-09-16T05:18:10.888 INFO:teuthology.orchestra.run.smithi032.stderr:no change 2024-09-16T05:18:10.924 INFO:tasks.rados.rados.2.smithi204.stdout:1985: finishing write tid 1 to smithi20457400-21 2024-09-16T05:18:10.924 INFO:tasks.rados.rados.2.smithi204.stdout:1983: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:18:10.924 INFO:tasks.rados.rados.2.smithi204.stdout:1985: finishing write tid 2 to smithi20457400-21 2024-09-16T05:18:10.924 INFO:tasks.rados.rados.2.smithi204.stdout:1984: done (3 left) 2024-09-16T05:18:10.924 INFO:tasks.rados.rados.2.smithi204.stdout:1988: snap_create 2024-09-16T05:18:10.924 INFO:tasks.rados.rados.2.smithi204.stdout:1985: finishing write tid 3 to smithi20457400-21 2024-09-16T05:18:10.924 INFO:tasks.rados.rados.2.smithi204.stdout:1985: finishing write tid 4 to smithi20457400-21 2024-09-16T05:18:10.924 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 341 (ObjNum 716 snap 207 seq_num 716) dirty exists 2024-09-16T05:18:10.925 INFO:tasks.rados.rados.2.smithi204.stdout:1985: left oid 21 (ObjNum 716 snap 207 seq_num 716) 2024-09-16T05:18:10.925 INFO:tasks.rados.rados.2.smithi204.stdout:1983: finishing copy_from to smithi20457400-6 2024-09-16T05:18:10.925 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 723 (ObjNum 644 snap 187 seq_num 644) dirty exists 2024-09-16T05:18:10.925 INFO:tasks.rados.rados.2.smithi204.stdout:1987: finishing copy_from to smithi20457400-33 2024-09-16T05:18:10.925 INFO:tasks.rados.rados.2.smithi204.stdout:1987: got expected ENOENT (src dne) 2024-09-16T05:18:10.925 INFO:tasks.rados.rados.2.smithi204.stdout:1987: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:18:10.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:10.959 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:10.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:11.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:11.083 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:11.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:11.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:11.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:10 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:18:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:10 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:18:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:10 smithi032 ceph-mon[27388]: osdmap e540: 12 total, 12 up, 12 in 2024-09-16T05:18:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:10 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:18:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:10 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:18:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1691461840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:10 smithi032 ceph-mon[27388]: osdmap e541: 12 total, 12 up, 12 in 2024-09-16T05:18:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:10 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:18:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:10 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:18:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:10 smithi204 ceph-mon[30933]: osdmap e540: 12 total, 12 up, 12 in 2024-09-16T05:18:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:10 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:18:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:10 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:18:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1691461840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:10 smithi204 ceph-mon[30933]: osdmap e541: 12 total, 12 up, 12 in 2024-09-16T05:18:11.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:11.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:11.209 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:11.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:11.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:10 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:18:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:10 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:18:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:10 smithi196 ceph-mon[30802]: osdmap e540: 12 total, 12 up, 12 in 2024-09-16T05:18:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:10 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:18:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:10 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:18:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1691461840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:10 smithi196 ceph-mon[30802]: osdmap e541: 12 total, 12 up, 12 in 2024-09-16T05:18:11.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:11.333 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:11.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:11.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:11.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:11.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:11.541 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:18:11.542 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:18:11.542 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:18:11.542 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:18:11.542 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:18:11.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:11.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:11.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:11.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:11.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:18:11.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:11.736 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:11.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:11.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:11.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:11.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:11.934 INFO:tasks.rados.rados.2.smithi204.stdout:1983: done (3 left) 2024-09-16T05:18:11.934 INFO:tasks.rados.rados.2.smithi204.stdout:1985: done (2 left) 2024-09-16T05:18:11.934 INFO:tasks.rados.rados.2.smithi204.stdout:1987: done (1 left) 2024-09-16T05:18:11.934 INFO:tasks.rados.rados.2.smithi204.stdout:1988: done (0 left) 2024-09-16T05:18:11.935 INFO:tasks.rados.rados.2.smithi204.stdout:1989: write oid 30 current snap is 208 2024-09-16T05:18:11.935 INFO:tasks.rados.rados.2.smithi204.stdout:1989: seq_num 717 ranges {450619=490715,1590648=459971} 2024-09-16T05:18:11.939 INFO:tasks.rados.rados.2.smithi204.stdout:1989: writing smithi20457400-30 from 450619 to 941334 tid 1 2024-09-16T05:18:11.945 INFO:tasks.rados.rados.2.smithi204.stdout:1989: writing smithi20457400-30 from 1590648 to 2050619 tid 2 2024-09-16T05:18:11.945 INFO:tasks.rados.rados.2.smithi204.stdout:1990: write oid 21 current snap is 208 2024-09-16T05:18:11.945 INFO:tasks.rados.rados.2.smithi204.stdout:1989: finishing write tid 1 to smithi20457400-30 2024-09-16T05:18:11.945 INFO:tasks.rados.rados.2.smithi204.stdout:1990: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-16T05:18:11.951 INFO:tasks.rados.rados.2.smithi204.stdout:1990: writing smithi20457400-21 from 467426 to 1033390 tid 1 2024-09-16T05:18:11.957 INFO:tasks.rados.rados.2.smithi204.stdout:1990: writing smithi20457400-21 from 1649130 to 2067426 tid 2 2024-09-16T05:18:11.957 INFO:tasks.rados.rados.2.smithi204.stdout:1991: delete oid 31 current snap is 208 2024-09-16T05:18:11.957 INFO:tasks.rados.rados.2.smithi204.stdout:1989: finishing write tid 2 to smithi20457400-30 2024-09-16T05:18:11.957 INFO:tasks.rados.rados.2.smithi204.stdout:1989: finishing write tid 3 to smithi20457400-30 2024-09-16T05:18:11.957 INFO:tasks.rados.rados.2.smithi204.stdout:1989: finishing write tid 4 to smithi20457400-30 2024-09-16T05:18:11.957 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 932 (ObjNum 717 snap 208 seq_num 717) dirty exists 2024-09-16T05:18:11.957 INFO:tasks.rados.rados.2.smithi204.stdout:1989: left oid 30 (ObjNum 717 snap 208 seq_num 717) 2024-09-16T05:18:11.958 INFO:tasks.rados.rados.2.smithi204.stdout:1990: finishing write tid 1 to smithi20457400-21 2024-09-16T05:18:11.958 INFO:tasks.rados.rados.2.smithi204.stdout:1989: done (2 left) 2024-09-16T05:18:11.959 INFO:tasks.rados.rados.2.smithi204.stdout:1991: done (1 left) 2024-09-16T05:18:11.959 INFO:tasks.rados.rados.2.smithi204.stdout:1992: rollback oid 4 current snap is 208 2024-09-16T05:18:11.959 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 206 2024-09-16T05:18:11.959 INFO:tasks.rados.rados.2.smithi204.stdout:1993: snap_create 2024-09-16T05:18:11.960 INFO:tasks.rados.rados.2.smithi204.stdout:1990: finishing write tid 2 to smithi20457400-21 2024-09-16T05:18:11.960 INFO:tasks.rados.rados.2.smithi204.stdout:1990: finishing write tid 3 to smithi20457400-21 2024-09-16T05:18:11.960 INFO:tasks.rados.rados.2.smithi204.stdout:1990: finishing write tid 4 to smithi20457400-21 2024-09-16T05:18:11.960 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 344 (ObjNum 718 snap 208 seq_num 718) dirty exists 2024-09-16T05:18:11.960 INFO:tasks.rados.rados.2.smithi204.stdout:1990: left oid 21 (ObjNum 718 snap 208 seq_num 718) 2024-09-16T05:18:11.962 INFO:tasks.rados.rados.2.smithi204.stdout:1992: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:18:11.962 INFO:tasks.rados.rados.2.smithi204.stdout:1992: finishing rollback tid 2 to smithi20457400-4 2024-09-16T05:18:11.962 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 575 (ObjNum 695 snap 201 seq_num 695) dirty exists 2024-09-16T05:18:11.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:11.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:11.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:12.063 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-16T05:18:12.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-16T05:18:12.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:12.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:12.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:11 smithi204 ceph-mon[30933]: pgmap v972: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 208 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T05:18:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:11 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:18:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:11 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:18:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:11 smithi204 ceph-mon[30933]: from='client.16260 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:11 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:11 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:11 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:18:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:11 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:18:12.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:12.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:18:12.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:11 smithi032 ceph-mon[27388]: pgmap v972: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 208 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T05:18:12.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:11 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:18:12.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:11 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:18:12.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:11 smithi032 ceph-mon[27388]: from='client.16260 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:12.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:11 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:12.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:11 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:12.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:11 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:18:12.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:11 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:18:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:12.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:12.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:11 smithi196 ceph-mon[30802]: pgmap v972: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 208 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 0 B/s wr, 2 op/s 2024-09-16T05:18:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:11 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:18:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:11 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:18:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:11 smithi196 ceph-mon[30802]: from='client.16260 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:11 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:11 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:11 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:18:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:11 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:18:12.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:12.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:12.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:12.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:12.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:12.476 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:12.484 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:12.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:12.586 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:12.597 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:12.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:12.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:12.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:12.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:12.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:12.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:12.860 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:12.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:12.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:18:12.940 INFO:tasks.rados.rados.2.smithi204.stdout:1990: done (2 left) 2024-09-16T05:18:12.940 INFO:tasks.rados.rados.2.smithi204.stdout:1992: done (1 left) 2024-09-16T05:18:12.940 INFO:tasks.rados.rados.2.smithi204.stdout:1993: done (0 left) 2024-09-16T05:18:12.941 INFO:tasks.rados.rados.2.smithi204.stdout:1994: write oid 14 current snap is 209 2024-09-16T05:18:12.941 INFO:tasks.rados.rados.2.smithi204.stdout:1994: seq_num 720 ranges {501040=716462,1766094=334946} 2024-09-16T05:18:12.947 INFO:tasks.rados.rados.2.smithi204.stdout:1994: writing smithi20457400-14 from 501040 to 1217502 tid 1 2024-09-16T05:18:12.952 INFO:tasks.rados.rados.2.smithi204.stdout:1994: writing smithi20457400-14 from 1766094 to 2101040 tid 2 2024-09-16T05:18:12.952 INFO:tasks.rados.rados.2.smithi204.stdout:1995: copy_from oid 44 from oid 25 current snap is 209 2024-09-16T05:18:12.952 INFO:tasks.rados.rados.2.smithi204.stdout:1996: rollback oid 45 current snap is 209 2024-09-16T05:18:12.953 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 194 2024-09-16T05:18:12.953 INFO:tasks.rados.rados.2.smithi204.stdout:1997: read oid 15 snap -1 2024-09-16T05:18:12.953 INFO:tasks.rados.rados.2.smithi204.stdout:1997: expect (ObjNum 697 snap 202 seq_num 697) 2024-09-16T05:18:12.953 INFO:tasks.rados.rados.2.smithi204.stdout:1998: copy_from oid 17 from oid 42 current snap is 209 2024-09-16T05:18:12.953 INFO:tasks.rados.rados.2.smithi204.stdout:1999: delete oid 18 current snap is 209 2024-09-16T05:18:12.953 INFO:tasks.rados.rados.2.smithi204.stdout:1998: finishing copy_from to smithi20457400-17 2024-09-16T05:18:12.954 INFO:tasks.rados.rados.2.smithi204.stdout:1998: got expected ENOENT (src dne) 2024-09-16T05:18:12.954 INFO:tasks.rados.rados.2.smithi204.stdout:1998: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:18:12.954 INFO:tasks.rados.rados.2.smithi204.stdout:1994: finishing write tid 1 to smithi20457400-14 2024-09-16T05:18:12.955 INFO:tasks.rados.rados.2.smithi204.stdout:1994: finishing write tid 2 to smithi20457400-14 2024-09-16T05:18:12.956 INFO:tasks.rados.rados.2.smithi204.stdout:1996: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:18:12.956 INFO:tasks.rados.rados.2.smithi204.stdout:1994: finishing write tid 3 to smithi20457400-14 2024-09-16T05:18:12.956 INFO:tasks.rados.rados.2.smithi204.stdout:1996: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:18:12.956 INFO:tasks.rados.rados.2.smithi204.stdout:1994: finishing write tid 4 to smithi20457400-14 2024-09-16T05:18:12.956 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 937 (ObjNum 720 snap 209 seq_num 720) dirty exists 2024-09-16T05:18:12.956 INFO:tasks.rados.rados.2.smithi204.stdout:1994: left oid 14 (ObjNum 720 snap 209 seq_num 720) 2024-09-16T05:18:12.956 INFO:tasks.rados.rados.2.smithi204.stdout:1996: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:18:12.956 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 355 (ObjNum 476 snap 150 seq_num 476) dirty exists 2024-09-16T05:18:12.957 INFO:tasks.rados.rados.2.smithi204.stdout:1994: done (5 left) 2024-09-16T05:18:12.957 INFO:tasks.rados.rados.2.smithi204.stdout:1996: done (4 left) 2024-09-16T05:18:12.957 INFO:tasks.rados.rados.2.smithi204.stdout:1998: done (3 left) 2024-09-16T05:18:12.957 INFO:tasks.rados.rados.2.smithi204.stdout:1999: done (2 left) 2024-09-16T05:18:12.957 INFO:tasks.rados.rados.2.smithi204.stdout:2000: delete oid 19 current snap is 209 2024-09-16T05:18:12.959 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:19 2.b 2:d7c71ae7:::smithi20457400-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 449.91636655899998, 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19676892500000001, 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:12.960 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 19 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040519+0000", 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7300000000000001e-07 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:12.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040541+0000", 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2320999999999999e-05 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040547+0000", 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3060000000000003e-06 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040553+0000", 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2750000000000003e-06 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041020+0000", 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046726200000000003 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041092+0000", 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2275999999999997e-05 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041183+0000", 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1071999999999998e-05 2024-09-16T05:18:12.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041456+0000", 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027275299999999998 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237175+0000", 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19571920200000001 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:12.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237230+0000", 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4855000000000002e-05 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237287+0000", 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7129999999999997e-05 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:20 2.b 2:d7c71ae7:::smithi20457400-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:18:12.966 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 449.916324825, 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19685694000000001, 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 20 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:12.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4400000000000001e-07 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040587+0000", 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6444e-05 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040593+0000", 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5320000000000003e-06 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:12.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040600+0000", 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5139999999999998e-06 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041356+0000", 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075660300000000001 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041400+0000", 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3936000000000001e-05 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237316+0000", 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.195915805 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237361+0000", 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4548000000000001e-05 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237417+0000", 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6214000000000001e-05 2024-09-16T05:18:12.971 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:274 2.b 2:d7c71ae7:::smithi20457400-5:head [write 2964119~94722] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 443.93104698500002, 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073226163999999996, 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:12.972 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:12.973 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 274 2024-09-16T05:18:12.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.973 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025838+0000", 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:12.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025839+0000", 2024-09-16T05:18:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.141e-06 2024-09-16T05:18:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:12.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025876+0000", 2024-09-16T05:18:12.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6724999999999998e-05 2024-09-16T05:18:12.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025887+0000", 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1413999999999999e-05 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025892+0000", 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2739999999999998e-06 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025932+0000", 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.943e-05 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026014+0000", 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2540000000000006e-05 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026154+0000", 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013913200000000001 2024-09-16T05:18:12.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026818+0000", 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00066437899999999999 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099005+0000", 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072186933999999994 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:12.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099037+0000", 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2011000000000002e-05 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099064+0000", 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7183999999999999e-05 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:275 2.b 2:d7c71ae7:::smithi20457400-5:head [setxattr _header (55) in=7b,truncate 3058841] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:18:12.980 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 443.93098158999999, 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13101358900000001, 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 275 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:12.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:12.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:18:12.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025903+0000", 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025904+0000", 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.34e-07 2024-09-16T05:18:12.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025907+0000", 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9610000000000001e-06 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025910+0000", 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8680000000000001e-06 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025913+0000", 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9960000000000002e-06 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026440+0000", 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00052629399999999996 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026504+0000", 2024-09-16T05:18:12.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3934000000000001e-05 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026588+0000", 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4745000000000003e-05 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026821+0000", 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000232301 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156835+0000", 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13001446 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156875+0000", 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9360999999999998e-05 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156917+0000", 2024-09-16T05:18:12.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2335e-05 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:276 2.b 2:d7c71ae7:::smithi20457400-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 443.93095652400001, 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.131149614, 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:12.988 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 276 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:18:12.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025928+0000", 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025929+0000", 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.39e-07 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025931+0000", 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5940000000000001e-06 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025939+0000", 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1829999999999997e-06 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.025944+0000", 2024-09-16T05:18:12.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4770000000000002e-06 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026741+0000", 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000797326 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.026805+0000", 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3409000000000002e-05 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:12.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.156955+0000", 2024-09-16T05:18:12.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13015033600000001 2024-09-16T05:18:12.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:12.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.157016+0000", 2024-09-16T05:18:12.994 INFO:tasks.rados.rados.2.smithi204.stdout:2000: done (2 left) 2024-09-16T05:18:12.994 INFO:tasks.rados.rados.2.smithi204.stdout:2001: write oid 45 current snap is 209 2024-09-16T05:18:12.994 INFO:tasks.rados.rados.2.smithi204.stdout:2001: seq_num 723 ranges {551461=658562,1941540=209921} 2024-09-16T05:18:12.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0795999999999999e-05 2024-09-16T05:18:12.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.157078+0000", 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2053999999999999e-05 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:286 2.b 2:da4f6049:::smithi20457400-49:head [delete] snapc 4=[4] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117)", 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 443.85875615499998, 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.242160297, 2024-09-16T05:18:12.995 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 286 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:18:12.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6699999999999999e-07 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098131+0000", 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1289999999999999e-06 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098138+0000", 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5660000000000003e-06 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098143+0000", 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:18:12.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098194+0000", 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1368e-05 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098273+0000", 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8170999999999994e-05 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098449+0000", 2024-09-16T05:18:12.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176171 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099087+0000", 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00063836000000000003 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340229+0000", 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.241141684 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340263+0000", 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3813000000000003e-05 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340289+0000", 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6403000000000001e-05 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.001 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:414 2.5 2:ad4a88eb:::smithi20457400-16:head [setxattr _header (56) in=7b,truncate 3394981] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 435.259965636, 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29453131100000002, 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 414 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696920+0000", 2024-09-16T05:18:13.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999996e-07 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696923+0000", 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.92e-06 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696937+0000", 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3892e-05 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696941+0000", 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3819999999999997e-06 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697487+0000", 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054579200000000002 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697548+0000", 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1167000000000006e-05 2024-09-16T05:18:13.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697638+0000", 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0548000000000001e-05 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697936+0000", 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029720499999999998 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:13.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991363+0000", 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29342759099999999 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991404+0000", 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1013999999999997e-05 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991451+0000", 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6303e-05 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.007 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:415 2.5 2:ad4a88eb:::smithi20457400-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 435.25993750399999, 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29466997700000003, 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:13.008 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 415 2024-09-16T05:18:13.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.009 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:18:13.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.009 INFO:tasks.rados.rados.2.smithi204.stdout:2001: writing smithi20457400-45 from 551461 to 1210023 tid 1 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696948+0000", 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696952+0000", 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6500000000000002e-06 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696958+0000", 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9149999999999999e-06 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696963+0000", 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9439999999999996e-06 2024-09-16T05:18:13.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697806+0000", 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084291800000000001 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697861+0000", 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4725999999999997e-05 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991493+0000", 2024-09-16T05:18:13.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29363172500000001 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991551+0000", 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8622000000000002e-05 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991617+0000", 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5977999999999997e-05 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.013 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:414 2.5 e126/107 ondisk, result = 0)", 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 435.25877592900002, 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29334887300000001, 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.06e-07 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698112+0000", 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6520000000000002e-06 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991214+0000", 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29310221600000003 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991228+0000", 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.411e-05 2024-09-16T05:18:13.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991351+0000", 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012277999999999999 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991359+0000", 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0430000000000008e-06 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991458+0000", 2024-09-16T05:18:13.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8566000000000005e-05 2024-09-16T05:18:13.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.018 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.018 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:884 2.b 2:d8680303:::smithi20457400-43:head [setxattr _header (57) in=7b,truncate 2470629] snapc 23=[23,22,21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-16T05:18:13.018 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 391.29620989799997, 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68080432499999999, 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 884 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660676+0000", 2024-09-16T05:18:13.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2099999999999996e-07 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660679+0000", 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1769999999999998e-06 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660685+0000", 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6529999999999997e-06 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660691+0000", 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2530000000000004e-06 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.661798+0000", 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011070520000000001 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.661889+0000", 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0785999999999994e-05 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662038+0000", 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000149777 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662482+0000", 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00044356499999999999 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341385+0000", 2024-09-16T05:18:13.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67890330499999996 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341430+0000", 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4613999999999997e-05 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341479+0000", 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9521999999999998e-05 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.025 INFO:tasks.rados.rados.2.smithi204.stdout:2001: writing smithi20457400-45 from 1941540 to 2151461 tid 2 2024-09-16T05:18:13.026 INFO:tasks.rados.rados.2.smithi204.stdout:2002: read oid 40 snap -1 2024-09-16T05:18:13.026 INFO:tasks.rados.rados.2.smithi204.stdout:2002: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:18:13.026 INFO:tasks.rados.rados.2.smithi204.stdout:2003: write oid 1 current snap is 209 2024-09-16T05:18:13.026 INFO:tasks.rados.rados.2.smithi204.stdout:2003: seq_num 724 ranges {568268=733811,2000022=168246} 2024-09-16T05:18:13.026 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:885 2.b 2:d8680303:::smithi20457400-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", 2024-09-16T05:18:13.026 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:18:13.026 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 391.29618582799998, 2024-09-16T05:18:13.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68095269999999997, 2024-09-16T05:18:13.026 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 885 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:18:13.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999999e-07 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660702+0000", 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3099999999999999e-06 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660709+0000", 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0299999999999996e-06 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660714+0000", 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1780000000000002e-06 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662345+0000", 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016309390000000001 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662461+0000", 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011565600000000001 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341517+0000", 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67905638400000001 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341582+0000", 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4746999999999995e-05 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341652+0000", 2024-09-16T05:18:13.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0026999999999999e-05 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:405 2.1a e216/215 ondisk, result = 0)", 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 345.29037896699998, 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26443446100000001, 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.032 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666507+0000", 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8700000000000005e-07 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666511+0000", 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1130000000000003e-06 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930764+0000", 2024-09-16T05:18:13.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26425358900000001 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930771+0000", 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7329999999999997e-06 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930824+0000", 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3436000000000001e-05 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930834+0000", 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1570000000000006e-06 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930940+0000", 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106746 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.036 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1814 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 314.18511056599999, 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898298185, 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1814 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771776+0000", 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5149999999999999e-06 2024-09-16T05:18:13.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771783+0000", 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0180000000000004e-06 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771805+0000", 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2076999999999999e-05 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771824+0000", 2024-09-16T05:18:13.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8813000000000001e-05 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747524+0000", 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97570031700000004 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747560+0000", 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5549000000000001e-05 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670057+0000", 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92249772500000005 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670072+0000", 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5170999999999999e-05 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.041 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1815 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:18:13.042 INFO:tasks.rados.rados.2.smithi204.stdout:2003: writing smithi20457400-1 from 568268 to 1302079 tid 1 2024-09-16T05:18:13.042 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:18:13.042 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 314.18500866699998, 2024-09-16T05:18:13.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8982342750000001, 2024-09-16T05:18:13.042 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.042 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.042 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1815 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771877+0000", 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9599999999999999e-07 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771881+0000", 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6880000000000001e-06 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771889+0000", 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6789999999999994e-06 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771896+0000", 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747584+0000", 2024-09-16T05:18:13.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97568814000000004 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747590+0000", 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7440000000000004e-06 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670102+0000", 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92251251499999998 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670110+0000", 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2740000000000007e-06 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1816 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 314.18492667800001, 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898192844, 2024-09-16T05:18:13.047 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1816 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771959+0000", 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1999999999999998e-07 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771964+0000", 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771971+0000", 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.576e-06 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771977+0000", 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2960000000000004e-06 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747609+0000", 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97563200299999997 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747615+0000", 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2820000000000004e-06 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670144+0000", 2024-09-16T05:18:13.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92252909100000002 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670151+0000", 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2259999999999999e-06 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.052 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1881 2.25 2:a49174eb:::smithi20457400-39:head [delete] snapc 50=[50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e257)", 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 304.26033737699998, 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37621878800000003, 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1881 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696549+0000", 2024-09-16T05:18:13.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696556+0000", 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8090000000000004e-06 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696568+0000", 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2636000000000001e-05 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696576+0000", 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3459999999999996e-06 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696625+0000", 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9830000000000002e-05 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696720+0000", 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4816999999999997e-05 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:18:13.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696869+0000", 2024-09-16T05:18:13.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014893999999999999 2024-09-16T05:18:13.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:13.058 INFO:tasks.rados.rados.2.smithi204.stdout:2003: writing smithi20457400-1 from 2000022 to 2168268 tid 2 2024-09-16T05:18:13.058 INFO:tasks.rados.rados.2.smithi204.stdout:2004: write (excl) oid 41 current snap is 209 2024-09-16T05:18:13.058 INFO:tasks.rados.rados.2.smithi204.stdout:1997: expect (ObjNum 697 snap 202 seq_num 697) 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.697931+0000", 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010616759999999999 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072669+0000", 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37473788800000002 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072715+0000", 2024-09-16T05:18:13.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6371e-05 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072766+0000", 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1220000000000001e-05 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1881 2.25 e257/252 ondisk, result = 0)", 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:18:13.060 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 304.258809128, 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37469565799999999, 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698077+0000", 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5099999999999998e-07 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698082+0000", 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072556+0000", 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37447454499999999 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072564+0000", 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4120000000000004e-06 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072654+0000", 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9878999999999998e-05 2024-09-16T05:18:13.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072663+0000", 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072771+0000", 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108267 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:788 2.25 e406/252 ondisk, result = 0)", 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:55.723815+0000", 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 137.23306948199999, 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073107094999999997, 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.723815+0000", 2024-09-16T05:18:13.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.723815+0000", 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.723816+0000", 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9899999999999999e-07 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.723821+0000", 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5859999999999998e-06 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796769+0000", 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072947949999999998 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796778+0000", 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9910000000000008e-06 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796828+0000", 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0112999999999999e-05 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796835+0000", 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5909999999999998e-06 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:55.796922+0000", 2024-09-16T05:18:13.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7064999999999994e-05 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:839 2.1a e438/291 ondisk, result = 0)", 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 105.22639570699999, 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51227693699999999, 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730490+0000", 2024-09-16T05:18:13.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730494+0000", 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0810000000000004e-06 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242338+0000", 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51184441999999997 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242353+0000", 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.447e-05 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242501+0000", 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014851799999999999 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242532+0000", 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0406999999999999e-05 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242766+0000", 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000234587 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGRecoveryDelete(2.4c e439,438 2:33100188:::smithi20457400-19:2f,230'190)", 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 104.1234846, 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.584288378, 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:13.073 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:13.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833401+0000", 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833406+0000", 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8940000000000001e-06 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417262+0000", 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.58385600999999998 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417279+0000", 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6880000000000001e-05 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417458+0000", 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000178751 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417489+0000", 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1192999999999998e-05 2024-09-16T05:18:13.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417689+0000", 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019989399999999999 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:13.077 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:18:13.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:13.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:13.115 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:12 smithi032 ceph-mon[27388]: osdmap e542: 12 total, 12 up, 12 in 2024-09-16T05:18:13.115 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:12 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:18:13.116 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:12 smithi032 ceph-mon[27388]: 1.b scrub ok 2024-09-16T05:18:13.116 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1131358999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:12 smithi204 ceph-mon[30933]: osdmap e542: 12 total, 12 up, 12 in 2024-09-16T05:18:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:12 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:18:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:12 smithi204 ceph-mon[30933]: 1.b scrub ok 2024-09-16T05:18:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1131358999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:13.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:13.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:13.216 INFO:tasks.rados.rados.2.smithi204.stdout:1995: finishing copy_from racing read to smithi20457400-44 2024-09-16T05:18:13.216 INFO:tasks.rados.rados.2.smithi204.stdout:2001: finishing write tid 1 to smithi20457400-45 2024-09-16T05:18:13.216 INFO:tasks.rados.rados.2.smithi204.stdout:2004: seq_num 725 ranges {585075=409060,1542151=642924} 2024-09-16T05:18:13.221 INFO:tasks.rados.rados.2.smithi204.stdout:2004: writing smithi20457400-41 from 585075 to 994135 tid 1 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:2004: writing smithi20457400-41 from 1542151 to 2185075 tid 2 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:1997: done (5 left) 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:2005: snap_create 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:1995: finishing copy_from to smithi20457400-44 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 710 (ObjNum 692 snap 201 seq_num 692) dirty exists 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:2001: finishing write tid 2 to smithi20457400-45 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:2001: finishing write tid 3 to smithi20457400-45 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:2001: finishing write tid 4 to smithi20457400-45 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 358 (ObjNum 723 snap 209 seq_num 723) dirty exists 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:2001: left oid 45 (ObjNum 723 snap 209 seq_num 723) 2024-09-16T05:18:13.229 INFO:tasks.rados.rados.2.smithi204.stdout:2003: finishing write tid 1 to smithi20457400-1 2024-09-16T05:18:13.230 INFO:tasks.rados.rados.2.smithi204.stdout:2003: finishing write tid 2 to smithi20457400-1 2024-09-16T05:18:13.230 INFO:tasks.rados.rados.2.smithi204.stdout:2003: finishing write tid 3 to smithi20457400-1 2024-09-16T05:18:13.230 INFO:tasks.rados.rados.2.smithi204.stdout:2003: finishing write tid 4 to smithi20457400-1 2024-09-16T05:18:13.230 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 534 (ObjNum 724 snap 209 seq_num 724) dirty exists 2024-09-16T05:18:13.230 INFO:tasks.rados.rados.2.smithi204.stdout:2003: left oid 1 (ObjNum 724 snap 209 seq_num 724) 2024-09-16T05:18:13.230 INFO:tasks.rados.rados.2.smithi204.stdout:2002: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:18:13.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:12 smithi196 ceph-mon[30802]: osdmap e542: 12 total, 12 up, 12 in 2024-09-16T05:18:13.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:12 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:18:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:12 smithi196 ceph-mon[30802]: 1.b scrub ok 2024-09-16T05:18:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1131358999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:13.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:moved 0 / 214 (0%) 2024-09-16T05:18:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:avg 17.8333 2024-09-16T05:18:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:stddev 2.9392 -> 2.9392 (expected baseline 4.04317) 2024-09-16T05:18:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:min osd.0 with 13 -> 13 pgs (0.728972 -> 0.728972 * mean) 2024-09-16T05:18:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:max osd.7 with 24 -> 24 pgs (1.34579 -> 1.34579 * mean) 2024-09-16T05:18:13.308 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:18:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:oload 1000 2024-09-16T05:18:13.309 INFO:teuthology.orchestra.run.smithi032.stdout:max_change 3 2024-09-16T05:18:13.309 INFO:teuthology.orchestra.run.smithi032.stdout:max_change_osds 4 2024-09-16T05:18:13.309 INFO:teuthology.orchestra.run.smithi032.stdout:average_utilization 0.0007 2024-09-16T05:18:13.309 INFO:teuthology.orchestra.run.smithi032.stdout:overload_utilization 0.0068 2024-09-16T05:18:13.309 INFO:teuthology.orchestra.run.smithi032.stderr:no change 2024-09-16T05:18:13.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:13.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:13.337 INFO:tasks.rados.rados.2.smithi204.stdout:2004: finishing write tid 1 to smithi20457400-41 2024-09-16T05:18:13.337 INFO:tasks.rados.rados.2.smithi204.stdout:2004: finishing write tid 2 to smithi20457400-41 2024-09-16T05:18:13.337 INFO:tasks.rados.rados.2.smithi204.stdout:2004: finishing write tid 3 to smithi20457400-41 2024-09-16T05:18:13.337 INFO:tasks.rados.rados.2.smithi204.stdout:2004: finishing write tid 4 to smithi20457400-41 2024-09-16T05:18:13.337 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 708 (ObjNum 725 snap 209 seq_num 725) dirty exists 2024-09-16T05:18:13.337 INFO:tasks.rados.rados.2.smithi204.stdout:2004: left oid 41 (ObjNum 725 snap 209 seq_num 725) 2024-09-16T05:18:13.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:13.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:13.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:13.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:13.576 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:13.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:13.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:18:13.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:13.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:13.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:13.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:13.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:13.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:18:13.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:13.940 INFO:tasks.rados.rados.2.smithi204.stdout:1995: done (5 left) 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2001: done (4 left) 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2002: done (3 left) 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2003: done (2 left) 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2004: done (1 left) 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2005: done (0 left) 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2006: read oid 34 snap -1 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2006: expect (ObjNum 710 snap 206 seq_num 710) 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2007: write (excl) oid 28 current snap is 210 2024-09-16T05:18:13.941 INFO:tasks.rados.rados.2.smithi204.stdout:2007: seq_num 726 ranges {601882=484309,1600633=524904} 2024-09-16T05:18:13.946 INFO:tasks.rados.rados.2.smithi204.stdout:2007: writing smithi20457400-28 from 601882 to 1086191 tid 1 2024-09-16T05:18:13.953 INFO:tasks.rados.rados.2.smithi204.stdout:2007: writing smithi20457400-28 from 1600633 to 2125537 tid 2 2024-09-16T05:18:13.953 INFO:tasks.rados.rados.2.smithi204.stdout:2008: copy_from oid 15 from oid 23 current snap is 210 2024-09-16T05:18:13.953 INFO:tasks.rados.rados.2.smithi204.stdout:2006: expect (ObjNum 710 snap 206 seq_num 710) 2024-09-16T05:18:13.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:13.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:14.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:14.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:14.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2007: finishing write tid 1 to smithi20457400-28 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2006: done (2 left) 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2009: write (excl) oid 13 current snap is 210 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2007: finishing write tid 2 to smithi20457400-28 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2007: finishing write tid 3 to smithi20457400-28 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2007: finishing write tid 4 to smithi20457400-28 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 705 (ObjNum 726 snap 210 seq_num 726) dirty exists 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2007: left oid 28 (ObjNum 726 snap 210 seq_num 726) 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2008: finishing copy_from racing read to smithi20457400-15 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2008: finishing copy_from to smithi20457400-15 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 700 (ObjNum 680 snap 196 seq_num 680) dirty exists 2024-09-16T05:18:14.088 INFO:tasks.rados.rados.2.smithi204.stdout:2009: seq_num 727 ranges {618689=559558,1659115=559574} 2024-09-16T05:18:14.093 INFO:tasks.rados.rados.2.smithi204.stdout:2009: writing smithi20457400-13 from 618689 to 1178247 tid 1 2024-09-16T05:18:14.100 INFO:tasks.rados.rados.2.smithi204.stdout:2009: writing smithi20457400-13 from 1659115 to 2218689 tid 2 2024-09-16T05:18:14.100 INFO:tasks.rados.rados.2.smithi204.stdout:2007: done (2 left) 2024-09-16T05:18:14.100 INFO:tasks.rados.rados.2.smithi204.stdout:2008: done (1 left) 2024-09-16T05:18:14.100 INFO:tasks.rados.rados.2.smithi204.stdout:2010: write oid 14 current snap is 210 2024-09-16T05:18:14.100 INFO:tasks.rados.rados.2.smithi204.stdout:2009: finishing write tid 1 to smithi20457400-13 2024-09-16T05:18:14.100 INFO:tasks.rados.rados.2.smithi204.stdout:2010: seq_num 728 ranges {635496=634807,1717597=517899} 2024-09-16T05:18:14.106 INFO:tasks.rados.rados.2.smithi204.stdout:2010: writing smithi20457400-14 from 635496 to 1270303 tid 1 2024-09-16T05:18:14.112 INFO:tasks.rados.rados.2.smithi204.stdout:2010: writing smithi20457400-14 from 1717597 to 2235496 tid 2 2024-09-16T05:18:14.113 INFO:tasks.rados.rados.2.smithi204.stdout:2011: write (excl) oid 32 current snap is 210 2024-09-16T05:18:14.113 INFO:tasks.rados.rados.2.smithi204.stdout:2009: finishing write tid 2 to smithi20457400-13 2024-09-16T05:18:14.113 INFO:tasks.rados.rados.2.smithi204.stdout:2011: seq_num 729 ranges {652303=710056,2059726=192577} 2024-09-16T05:18:14.120 INFO:tasks.rados.rados.2.smithi204.stdout:2011: writing smithi20457400-32 from 652303 to 1362359 tid 1 2024-09-16T05:18:14.125 INFO:tasks.rados.rados.2.smithi204.stdout:2011: writing smithi20457400-32 from 2059726 to 2252303 tid 2 2024-09-16T05:18:14.125 INFO:tasks.rados.rados.2.smithi204.stdout:2012: write oid 11 current snap is 210 2024-09-16T05:18:14.125 INFO:tasks.rados.rados.2.smithi204.stdout:2009: finishing write tid 3 to smithi20457400-13 2024-09-16T05:18:14.125 INFO:tasks.rados.rados.2.smithi204.stdout:2012: seq_num 730 ranges {669110=501658,1834561=434549} 2024-09-16T05:18:14.131 INFO:tasks.rados.rados.2.smithi204.stdout:2012: writing smithi20457400-11 from 669110 to 1170768 tid 1 2024-09-16T05:18:14.138 INFO:tasks.rados.rados.2.smithi204.stdout:2012: writing smithi20457400-11 from 1834561 to 2269110 tid 2 2024-09-16T05:18:14.138 INFO:tasks.rados.rados.2.smithi204.stdout:2013: rollback oid 16 current snap is 210 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 16 to 203 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2009: finishing write tid 4 to smithi20457400-13 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 715 (ObjNum 727 snap 210 seq_num 727) dirty exists 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2009: left oid 13 (ObjNum 727 snap 210 seq_num 727) 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2010: finishing write tid 1 to smithi20457400-14 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2010: finishing write tid 2 to smithi20457400-14 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2010: finishing write tid 3 to smithi20457400-14 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2010: finishing write tid 4 to smithi20457400-14 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 941 (ObjNum 728 snap 210 seq_num 728) dirty exists 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2010: left oid 14 (ObjNum 728 snap 210 seq_num 728) 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2011: finishing write tid 1 to smithi20457400-32 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2011: finishing write tid 2 to smithi20457400-32 2024-09-16T05:18:14.139 INFO:tasks.rados.rados.2.smithi204.stdout:2011: finishing write tid 3 to smithi20457400-32 2024-09-16T05:18:14.140 INFO:tasks.rados.rados.2.smithi204.stdout:2011: finishing write tid 4 to smithi20457400-32 2024-09-16T05:18:14.140 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 727 (ObjNum 729 snap 210 seq_num 729) dirty exists 2024-09-16T05:18:14.140 INFO:tasks.rados.rados.2.smithi204.stdout:2011: left oid 32 (ObjNum 729 snap 210 seq_num 729) 2024-09-16T05:18:14.140 INFO:tasks.rados.rados.2.smithi204.stdout:2012: finishing write tid 1 to smithi20457400-11 2024-09-16T05:18:14.140 INFO:tasks.rados.rados.2.smithi204.stdout:2009: done (4 left) 2024-09-16T05:18:14.140 INFO:tasks.rados.rados.2.smithi204.stdout:2010: done (3 left) 2024-09-16T05:18:14.140 INFO:tasks.rados.rados.2.smithi204.stdout:2011: done (2 left) 2024-09-16T05:18:14.140 INFO:tasks.rados.rados.2.smithi204.stdout:2014: snap_remove snap 194 2024-09-16T05:18:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:13 smithi204 ceph-mon[30933]: pgmap v975: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 227 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 17 op/s 2024-09-16T05:18:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:13 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1131358999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:13 smithi204 ceph-mon[30933]: osdmap e543: 12 total, 12 up, 12 in 2024-09-16T05:18:14.177 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:14.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:14.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:14.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:14.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:13 smithi196 ceph-mon[30802]: pgmap v975: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 227 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 17 op/s 2024-09-16T05:18:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:13 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1131358999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:13 smithi196 ceph-mon[30802]: osdmap e543: 12 total, 12 up, 12 in 2024-09-16T05:18:14.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:14.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:14.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:13 smithi032 ceph-mon[27388]: pgmap v975: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 227 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 17 op/s 2024-09-16T05:18:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:13 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1131358999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:13 smithi032 ceph-mon[27388]: osdmap e543: 12 total, 12 up, 12 in 2024-09-16T05:18:14.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:14.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:14.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:14.476 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-16T05:18:14.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-16T05:18:14.539 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:14.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:14.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:14.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:14.601 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:18:14.602 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:18:14.602 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:18:14.602 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:18:14.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:14.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:14.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:14.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:14.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:14.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:14.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:14.950 INFO:tasks.rados.rados.2.smithi204.stdout:2014: done (2 left) 2024-09-16T05:18:14.951 INFO:tasks.rados.rados.2.smithi204.stdout:2015: snap_create 2024-09-16T05:18:14.951 INFO:tasks.rados.rados.2.smithi204.stdout:2013: finishing rollback tid 0 to smithi20457400-16 2024-09-16T05:18:14.951 INFO:tasks.rados.rados.2.smithi204.stdout:2013: finishing rollback tid 1 to smithi20457400-16 2024-09-16T05:18:14.951 INFO:tasks.rados.rados.2.smithi204.stdout:2012: finishing write tid 2 to smithi20457400-11 2024-09-16T05:18:14.951 INFO:tasks.rados.rados.2.smithi204.stdout:2012: finishing write tid 3 to smithi20457400-11 2024-09-16T05:18:14.951 INFO:tasks.rados.rados.2.smithi204.stdout:2013: finishing rollback tid 2 to smithi20457400-16 2024-09-16T05:18:14.951 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 715 (ObjNum 596 snap 180 seq_num 596) dirty exists 2024-09-16T05:18:14.951 INFO:tasks.rados.rados.2.smithi204.stdout:2012: finishing write tid 4 to smithi20457400-11 2024-09-16T05:18:14.952 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 764 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:18:14.952 INFO:tasks.rados.rados.2.smithi204.stdout:2012: left oid 11 (ObjNum 730 snap 210 seq_num 730) 2024-09-16T05:18:14.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:14.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:15.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:14 smithi032 ceph-mon[27388]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:15.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:14 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:18:15.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:14 smithi032 ceph-mon[27388]: osdmap e544: 12 total, 12 up, 12 in 2024-09-16T05:18:15.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:15.068 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:15.068 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:15.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:15.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:15.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:15.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:15.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:14 smithi196 ceph-mon[30802]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:14 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:18:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:14 smithi196 ceph-mon[30802]: osdmap e544: 12 total, 12 up, 12 in 2024-09-16T05:18:15.266 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:14 smithi204 ceph-mon[30933]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:15.266 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:14 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:18:15.266 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:14 smithi204 ceph-mon[30933]: osdmap e544: 12 total, 12 up, 12 in 2024-09-16T05:18:15.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:15.291 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:15.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:18:15.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:15.312 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:15.312 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:15.331 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:15.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:15.333 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:15.333 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:15.355 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:15.356 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:15.378 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:15.378 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:15.399 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:15.400 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:15.421 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:15.421 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:15.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:15.442 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:15.442 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:15.454 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:15.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:15.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:15.465 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:15.486 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:15.486 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:15.508 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:15.508 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:15.535 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:15.556 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:15.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:15.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:15.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:15.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:15.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:15.757 INFO:tasks.rados.rados.2.smithi204.stdout:2012: done (2 left) 2024-09-16T05:18:15.757 INFO:tasks.rados.rados.2.smithi204.stdout:2013: done (1 left) 2024-09-16T05:18:15.758 INFO:tasks.rados.rados.2.smithi204.stdout:2015: done (0 left) 2024-09-16T05:18:15.758 INFO:tasks.rados.rados.2.smithi204.stdout:2016: read oid 38 snap -1 2024-09-16T05:18:15.758 INFO:tasks.rados.rados.2.smithi204.stdout:2016: expect (ObjNum 668 snap 194 seq_num 668) 2024-09-16T05:18:15.758 INFO:tasks.rados.rados.2.smithi204.stdout:2017: read oid 14 snap -1 2024-09-16T05:18:15.758 INFO:tasks.rados.rados.2.smithi204.stdout:2017: expect (ObjNum 728 snap 210 seq_num 728) 2024-09-16T05:18:15.758 INFO:tasks.rados.rados.2.smithi204.stdout:2018: snap_remove snap 209 2024-09-16T05:18:15.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:15.827 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:15.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:15.879 INFO:teuthology.orchestra.run.smithi032.stdout:moved 0 / 214 (0%) 2024-09-16T05:18:15.879 INFO:teuthology.orchestra.run.smithi032.stdout:avg 17.8333 2024-09-16T05:18:15.879 INFO:teuthology.orchestra.run.smithi032.stdout:stddev 2.9392 -> 2.9392 (expected baseline 4.04317) 2024-09-16T05:18:15.879 INFO:teuthology.orchestra.run.smithi032.stdout:min osd.0 with 13 -> 13 pgs (0.728972 -> 0.728972 * mean) 2024-09-16T05:18:15.879 INFO:teuthology.orchestra.run.smithi032.stdout:max osd.7 with 24 -> 24 pgs (1.34579 -> 1.34579 * mean) 2024-09-16T05:18:15.879 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:18:15.880 INFO:teuthology.orchestra.run.smithi032.stdout:oload 1000 2024-09-16T05:18:15.880 INFO:teuthology.orchestra.run.smithi032.stdout:max_change 3 2024-09-16T05:18:15.880 INFO:teuthology.orchestra.run.smithi032.stdout:max_change_osds 4 2024-09-16T05:18:15.880 INFO:teuthology.orchestra.run.smithi032.stdout:average_utilization 0.0007 2024-09-16T05:18:15.880 INFO:teuthology.orchestra.run.smithi032.stdout:overload_utilization 0.0070 2024-09-16T05:18:15.880 INFO:teuthology.orchestra.run.smithi032.stderr:no change 2024-09-16T05:18:15.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:15.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:18:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:15.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:15.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:16.052 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:16.052 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:16.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:16.075 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:16.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:15 smithi032 ceph-mon[27388]: pgmap v978: 107 pgs: 107 active+clean; 234 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-16T05:18:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:15 smithi032 ceph-mon[27388]: osdmap e545: 12 total, 12 up, 12 in 2024-09-16T05:18:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/697244332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:15 smithi032 ceph-mon[27388]: osdmap e546: 12 total, 12 up, 12 in 2024-09-16T05:18:16.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:16.199 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:16.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:16.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:16.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:15 smithi196 ceph-mon[30802]: pgmap v978: 107 pgs: 107 active+clean; 234 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-16T05:18:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:15 smithi196 ceph-mon[30802]: osdmap e545: 12 total, 12 up, 12 in 2024-09-16T05:18:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/697244332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:15 smithi196 ceph-mon[30802]: osdmap e546: 12 total, 12 up, 12 in 2024-09-16T05:18:16.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:16.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:16.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:16.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:15 smithi204 ceph-mon[30933]: pgmap v978: 107 pgs: 107 active+clean; 234 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2024-09-16T05:18:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:15 smithi204 ceph-mon[30933]: osdmap e545: 12 total, 12 up, 12 in 2024-09-16T05:18:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/697244332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:15 smithi204 ceph-mon[30933]: osdmap e546: 12 total, 12 up, 12 in 2024-09-16T05:18:16.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:16.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:16.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:16.556 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:16.579 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:16.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:16.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:16.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:16.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:16.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:18:16.759 INFO:tasks.rados.rados.2.smithi204.stdout:2018: done (2 left) 2024-09-16T05:18:16.759 INFO:tasks.rados.rados.2.smithi204.stdout:2019: snap_remove snap 203 2024-09-16T05:18:16.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:16.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:16.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:16.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:16.965 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:16.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:16.988 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:17.041 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:17.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:17.089 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:17.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:16 smithi204 ceph-mon[30933]: from='client.16284 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:16 smithi204 ceph-mon[30933]: osdmap e547: 12 total, 12 up, 12 in 2024-09-16T05:18:17.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:17.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:17.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:17.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:16 smithi196 ceph-mon[30802]: from='client.16284 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:17.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:16 smithi196 ceph-mon[30802]: osdmap e547: 12 total, 12 up, 12 in 2024-09-16T05:18:17.285 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:16 smithi032 ceph-mon[27388]: from='client.16284 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T05:18:17.285 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:16 smithi032 ceph-mon[27388]: osdmap e547: 12 total, 12 up, 12 in 2024-09-16T05:18:17.315 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:17.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:17.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:17.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:17.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:17.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:17.520 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:17.520 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:17.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:17.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:17.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:17.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:17.715 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:17.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:17.754 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:18:17.758 INFO:tasks.rados.rados.2.smithi204.stdout:2019: done (2 left) 2024-09-16T05:18:17.758 INFO:tasks.rados.rados.2.smithi204.stdout:2020: delete oid 33 current snap is 211 2024-09-16T05:18:17.759 INFO:tasks.rados.rados.2.smithi204.stdout:2017: expect (ObjNum 728 snap 210 seq_num 728) 2024-09-16T05:18:17.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:17.838 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:17.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:17.842 INFO:tasks.rados.rados.2.smithi204.stdout:2016: expect (ObjNum 668 snap 194 seq_num 668) 2024-09-16T05:18:17.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:17.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:17.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:18.054 INFO:tasks.rados.rados.2.smithi204.stdout:2016: done (2 left) 2024-09-16T05:18:18.054 INFO:tasks.rados.rados.2.smithi204.stdout:2017: done (1 left) 2024-09-16T05:18:18.054 INFO:tasks.rados.rados.2.smithi204.stdout:2020: done (0 left) 2024-09-16T05:18:18.054 INFO:tasks.rados.rados.2.smithi204.stdout:2021: read oid 5 snap -1 2024-09-16T05:18:18.054 INFO:tasks.rados.rados.2.smithi204.stdout:2021: expect (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:18:18.054 INFO:tasks.rados.rados.2.smithi204.stdout:2022: read oid 22 snap -1 2024-09-16T05:18:18.054 INFO:tasks.rados.rados.2.smithi204.stdout:2022: expect (ObjNum 652 snap 189 seq_num 652) 2024-09-16T05:18:18.054 INFO:tasks.rados.rados.2.smithi204.stdout:2023: snap_create 2024-09-16T05:18:18.059 INFO:tasks.rados.rados.2.smithi204.stdout:2021: expect (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:18:18.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:18.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:18.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:18.140 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: pgmap v981: 107 pgs: 107 active+clean; 241 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2207500918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2207500918' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: osdmap e548: 12 total, 12 up, 12 in 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10338500507575676]}]: dispatch 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07924851622773031]}]: dispatch 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10858728568260394]}]: dispatch 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08854761607015284]}]: dispatch 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08909727867480428]}]: dispatch 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08864500822467837]}]: dispatch 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08457069774604854]}]: dispatch 2024-09-16T05:18:18.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07416435251628216]}]: dispatch 2024-09-16T05:18:18.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08941947893351415]}]: dispatch 2024-09-16T05:18:18.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07900539350269777]}]: dispatch 2024-09-16T05:18:18.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07617893459900735]}]: dispatch 2024-09-16T05:18:18.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:17 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08669681946547353]}]: dispatch 2024-09-16T05:18:18.159 INFO:tasks.rados.rados.2.smithi204.stdout:2022: expect (ObjNum 652 snap 189 seq_num 652) 2024-09-16T05:18:18.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:18:18.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: pgmap v981: 107 pgs: 107 active+clean; 241 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-16T05:18:18.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2207500918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:18.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2207500918' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:18.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: osdmap e548: 12 total, 12 up, 12 in 2024-09-16T05:18:18.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10338500507575676]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07924851622773031]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10858728568260394]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08854761607015284]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08909727867480428]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08864500822467837]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08457069774604854]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07416435251628216]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08941947893351415]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07900539350269777]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07617893459900735]}]: dispatch 2024-09-16T05:18:18.169 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:17 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08669681946547353]}]: dispatch 2024-09-16T05:18:18.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:18.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:18.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: pgmap v981: 107 pgs: 107 active+clean; 241 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-16T05:18:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2207500918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2207500918' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: osdmap e548: 12 total, 12 up, 12 in 2024-09-16T05:18:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10338500507575676]}]: dispatch 2024-09-16T05:18:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07924851622773031]}]: dispatch 2024-09-16T05:18:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10858728568260394]}]: dispatch 2024-09-16T05:18:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08854761607015284]}]: dispatch 2024-09-16T05:18:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08909727867480428]}]: dispatch 2024-09-16T05:18:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08864500822467837]}]: dispatch 2024-09-16T05:18:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08457069774604854]}]: dispatch 2024-09-16T05:18:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07416435251628216]}]: dispatch 2024-09-16T05:18:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08941947893351415]}]: dispatch 2024-09-16T05:18:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07900539350269777]}]: dispatch 2024-09-16T05:18:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07617893459900735]}]: dispatch 2024-09-16T05:18:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:17 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08669681946547353]}]: dispatch 2024-09-16T05:18:18.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:18.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:18.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:18.409 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:18.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:18.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:18.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:18.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:18:18.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:18.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:18.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:18.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:18.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:18.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:18.760 INFO:tasks.rados.rados.2.smithi204.stdout:2021: done (2 left) 2024-09-16T05:18:18.760 INFO:tasks.rados.rados.2.smithi204.stdout:2022: done (1 left) 2024-09-16T05:18:18.760 INFO:tasks.rados.rados.2.smithi204.stdout:2023: done (0 left) 2024-09-16T05:18:18.760 INFO:tasks.rados.rados.2.smithi204.stdout:2024: read oid 15 snap -1 2024-09-16T05:18:18.761 INFO:tasks.rados.rados.2.smithi204.stdout:2024: expect (ObjNum 680 snap 196 seq_num 680) 2024-09-16T05:18:18.761 INFO:tasks.rados.rados.2.smithi204.stdout:2025: write oid 33 current snap is 212 2024-09-16T05:18:18.761 INFO:tasks.rados.rados.2.smithi204.stdout:2025: seq_num 732 ranges {702724=652156,1951525=351199} 2024-09-16T05:18:18.768 INFO:tasks.rados.rados.2.smithi204.stdout:2025: writing smithi20457400-33 from 702724 to 1354880 tid 1 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2025: writing smithi20457400-33 from 1951525 to 2302724 tid 2 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2026: read oid 28 snap -1 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2026: expect (ObjNum 726 snap 210 seq_num 726) 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2027: read oid 6 snap -1 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2027: expect (ObjNum 644 snap 187 seq_num 644) 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2028: read oid 12 snap -1 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2028: expect (ObjNum 668 snap 194 seq_num 668) 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2029: copy_from oid 11 from oid 29 current snap is 212 2024-09-16T05:18:18.774 INFO:tasks.rados.rados.2.smithi204.stdout:2024: expect (ObjNum 680 snap 196 seq_num 680) 2024-09-16T05:18:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:18.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:18.838 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:18.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:18.889 INFO:tasks.rados.rados.2.smithi204.stdout:2025: finishing write tid 1 to smithi20457400-33 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2024: done (5 left) 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2030: read oid 38 snap 210 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2030: expect (ObjNum 668 snap 194 seq_num 668) 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2031: copy_from oid 1 from oid 40 current snap is 212 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2025: finishing write tid 2 to smithi20457400-33 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2025: finishing write tid 3 to smithi20457400-33 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2025: finishing write tid 4 to smithi20457400-33 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 758 (ObjNum 732 snap 212 seq_num 732) dirty exists 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2025: left oid 33 (ObjNum 732 snap 212 seq_num 732) 2024-09-16T05:18:18.890 INFO:tasks.rados.rados.2.smithi204.stdout:2028: expect (ObjNum 668 snap 194 seq_num 668) 2024-09-16T05:18:18.910 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:18.910 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:18.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:18.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:18.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:19.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:19.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:19.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:19.100 INFO:tasks.rados.rados.2.smithi204.stdout:2029: finishing copy_from to smithi20457400-11 2024-09-16T05:18:19.100 INFO:tasks.rados.rados.2.smithi204.stdout:2029: got expected ENOENT (src dne) 2024-09-16T05:18:19.100 INFO:tasks.rados.rados.2.smithi204.stdout:2029: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:18:19.100 INFO:tasks.rados.rados.2.smithi204.stdout:2026: expect (ObjNum 726 snap 210 seq_num 726) 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10338500507575676]}]': finished 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07924851622773031]}]': finished 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10858728568260394]}]': finished 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08854761607015284]}]': finished 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08909727867480428]}]': finished 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08864500822467837]}]': finished 2024-09-16T05:18:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08457069774604854]}]': finished 2024-09-16T05:18:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07416435251628216]}]': finished 2024-09-16T05:18:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08941947893351415]}]': finished 2024-09-16T05:18:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07900539350269777]}]': finished 2024-09-16T05:18:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07617893459900735]}]': finished 2024-09-16T05:18:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08669681946547353]}]': finished 2024-09-16T05:18:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:18 smithi204 ceph-mon[30933]: osdmap e549: 12 total, 12 up, 12 in 2024-09-16T05:18:19.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:19.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:19.223 INFO:tasks.rados.rados.2.smithi204.stdout:2027: expect (ObjNum 644 snap 187 seq_num 644) 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10338500507575676]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07924851622773031]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10858728568260394]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08854761607015284]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08909727867480428]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08864500822467837]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08457069774604854]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07416435251628216]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08941947893351415]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07900539350269777]}]': finished 2024-09-16T05:18:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07617893459900735]}]': finished 2024-09-16T05:18:19.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08669681946547353]}]': finished 2024-09-16T05:18:19.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:18 smithi196 ceph-mon[30802]: osdmap e549: 12 total, 12 up, 12 in 2024-09-16T05:18:19.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:19.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:19.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10338500507575676]}]': finished 2024-09-16T05:18:19.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07924851622773031]}]': finished 2024-09-16T05:18:19.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.10858728568260394]}]': finished 2024-09-16T05:18:19.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08854761607015284]}]': finished 2024-09-16T05:18:19.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08909727867480428]}]': finished 2024-09-16T05:18:19.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08864500822467837]}]': finished 2024-09-16T05:18:19.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08457069774604854]}]': finished 2024-09-16T05:18:19.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07416435251628216]}]': finished 2024-09-16T05:18:19.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08941947893351415]}]': finished 2024-09-16T05:18:19.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07900539350269777]}]': finished 2024-09-16T05:18:19.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07617893459900735]}]': finished 2024-09-16T05:18:19.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08669681946547353]}]': finished 2024-09-16T05:18:19.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:18 smithi032 ceph-mon[27388]: osdmap e549: 12 total, 12 up, 12 in 2024-09-16T05:18:19.269 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:19.269 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:19.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:19.291 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:19.311 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:19.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:19.332 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:19.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:19.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:19.354 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:19.355 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:19.376 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:19.376 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:19.397 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:19.397 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:19.419 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:19.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:19.440 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:19.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:19.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:19.462 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:19.482 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:19.482 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:19.502 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:19.503 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:19.511 INFO:tasks.rados.rados.2.smithi204.stdout:2030: expect (ObjNum 668 snap 194 seq_num 668) 2024-09-16T05:18:19.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:19.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:18:19.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:19.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:19.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:19.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:19.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:19.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2031: finishing copy_from racing read to smithi20457400-1 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2031: finishing copy_from to smithi20457400-1 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 514 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2025: done (6 left) 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2026: done (5 left) 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2027: done (4 left) 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2028: done (3 left) 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2029: done (2 left) 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2030: done (1 left) 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2031: done (0 left) 2024-09-16T05:18:19.725 INFO:tasks.rados.rados.2.smithi204.stdout:2032: snap_remove snap 208 2024-09-16T05:18:19.761 INFO:tasks.rados.rados.2.smithi204.stdout:2032: done (0 left) 2024-09-16T05:18:19.761 INFO:tasks.rados.rados.2.smithi204.stdout:2033: snap_create 2024-09-16T05:18:19.816 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:19.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:19.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:19.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:19.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:19.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:20.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:20.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:20.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:20.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:19 smithi032 ceph-mon[27388]: pgmap v984: 107 pgs: 107 active+clean; 247 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 25 op/s 2024-09-16T05:18:20.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/226193671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:20.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:19 smithi032 ceph-mon[27388]: osdmap e550: 12 total, 12 up, 12 in 2024-09-16T05:18:20.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:20.211 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:19 smithi204 ceph-mon[30933]: pgmap v984: 107 pgs: 107 active+clean; 247 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 25 op/s 2024-09-16T05:18:20.211 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/226193671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:20.211 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:19 smithi204 ceph-mon[30933]: osdmap e550: 12 total, 12 up, 12 in 2024-09-16T05:18:20.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:20.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:19 smithi196 ceph-mon[30802]: pgmap v984: 107 pgs: 107 active+clean; 247 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 25 op/s 2024-09-16T05:18:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/226193671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:19 smithi196 ceph-mon[30802]: osdmap e550: 12 total, 12 up, 12 in 2024-09-16T05:18:20.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:20.318 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:18:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:18:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:18:20.318 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:18:20.319 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:18:20.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:20.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:20.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:20.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:20.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:20.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:18:20.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:20.593 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:20.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:20.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:20.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:20.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:20.756 INFO:tasks.rados.rados.2.smithi204.stdout:2033: done (0 left) 2024-09-16T05:18:20.756 INFO:tasks.rados.rados.2.smithi204.stdout:2034: read oid 50 snap -1 2024-09-16T05:18:20.756 INFO:tasks.rados.rados.2.smithi204.stdout:2034: expect (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:18:20.756 INFO:tasks.rados.rados.2.smithi204.stdout:2035: snap_create 2024-09-16T05:18:20.764 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:20.771 INFO:tasks.rados.rados.2.smithi204.stdout:2034: expect (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:18:20.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:20.847 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:20.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:20.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:20.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:18:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:20.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:20.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:21.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:21.095 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:21.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:21.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:20 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:21.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:20 smithi032 ceph-mon[27388]: osdmap e551: 12 total, 12 up, 12 in 2024-09-16T05:18:21.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:20 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:21.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:20 smithi204 ceph-mon[30933]: osdmap e551: 12 total, 12 up, 12 in 2024-09-16T05:18:21.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:21.214 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:21.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:21.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:20 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:20 smithi196 ceph-mon[30802]: osdmap e551: 12 total, 12 up, 12 in 2024-09-16T05:18:21.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:21.343 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:21.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:21.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:21.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:21.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:21.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:21.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:21.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:21.669 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:18:21.669 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:18:21.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:18:21.669 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:18:21.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.669 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:23 2.0 e112/107 2:06458cf1:::smithi20457400-6:head v 112'3, mlcod=0'0)", 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 458.61351344799999, 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.102016024, 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053062+0000", 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053064+0000", 2024-09-16T05:18:21.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8470000000000001e-06 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.053068+0000", 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9519999999999996e-06 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154742+0000", 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.101674714 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154751+0000", 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9020000000000005e-06 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154811+0000", 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9329000000000001e-05 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.154832+0000", 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1174000000000001e-05 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.155048+0000", 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021618599999999999 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.155078+0000", 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9920000000000002e-05 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2255 2.1d 2:ba754fcb:::smithi20457400-30:head [append 0~0] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:18:21.675 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 279.98489436300002, 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53925791899999997, 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2255 2024-09-16T05:18:21.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7300000000000001e-07 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681684+0000", 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.582e-06 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681692+0000", 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3920000000000003e-06 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681698+0000", 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2100000000000001e-06 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681779+0000", 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1003999999999999e-05 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681782+0000", 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1190000000000001e-06 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684363+0000", 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025812399999999998 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684417+0000", 2024-09-16T05:18:21.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3692e-05 2024-09-16T05:18:21.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:18:21.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684503+0000", 2024-09-16T05:18:21.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6592e-05 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684818+0000", 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000314851 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220845+0000", 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53602721099999995 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220887+0000", 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1514000000000003e-05 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220939+0000", 2024-09-16T05:18:21.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2039000000000001e-05 2024-09-16T05:18:21.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.684 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2559 2.1d 2:ba754fcb:::smithi20457400-30:head [copy-from ver 324] snapc 6d=[6c,6b,6a,69,67,66,64,63,62,60,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 216.70993018999999, 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.74015230600000004, 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2559 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956647+0000", 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2919999999999998e-06 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956655+0000", 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.729e-06 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956672+0000", 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6775000000000001e-05 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956685+0000", 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2832e-05 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956782+0000", 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6862999999999997e-05 2024-09-16T05:18:21.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956790+0000", 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4020000000000002e-06 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958620+0000", 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001830415 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958718+0000", 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7708999999999996e-05 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.688352+0000", 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72963416800000003 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.695488+0000", 2024-09-16T05:18:21.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071356659999999997 2024-09-16T05:18:21.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696731+0000", 2024-09-16T05:18:21.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00124332 2024-09-16T05:18:21.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696767+0000", 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6171000000000003e-05 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696797+0000", 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9964e-05 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.692 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2560 2.1d 2:ba754fcb:::smithi20457400-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 216.70986431, 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73643418699999996, 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:21.693 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2560 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956712+0000", 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.82e-07 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956714+0000", 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6419999999999999e-06 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956737+0000", 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2668999999999999e-05 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956744+0000", 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.289e-06 2024-09-16T05:18:21.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956847+0000", 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102729 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956855+0000", 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4009999999999997e-06 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958884+0000", 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020286409999999999 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958928+0000", 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4458e-05 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693004+0000", 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73407526999999995 2024-09-16T05:18:21.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693071+0000", 2024-09-16T05:18:21.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7176999999999997e-05 2024-09-16T05:18:21.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693145+0000", 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4328999999999999e-05 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.0.0:67 2.31 e356/279 2:8c1ba486:::smithi20457400-2:75 v 356'145, mlcod=350'141)", 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 195.960673695, 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.929470409, 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.700 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:18:21.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4499999999999998e-07 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705907+0000", 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8799999999999999e-06 2024-09-16T05:18:21.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634756+0000", 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92884895999999995 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634764+0000", 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9589999999999995e-06 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634827+0000", 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3280000000000004e-05 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634853+0000", 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6231999999999999e-05 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635342+0000", 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048899100000000004 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635372+0000", 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9762000000000001e-05 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.707 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2896 2.1d 2:b98a79db:::smithi20457400-31:head [copy-from ver 336] snapc 7c=[7c,7b,7a,76,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:21.706650+0000", 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.95992523300001, 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.026780103999999999, 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2896 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.708 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706650+0000", 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706650+0000", 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706653+0000", 2024-09-16T05:18:21.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7300000000000001e-06 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706670+0000", 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6781000000000001e-05 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6733e-05 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706709+0000", 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2707999999999999e-05 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706828+0000", 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119046 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.706837+0000", 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6430000000000001e-06 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.713173+0000", 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063359380000000002 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.713282+0000", 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010902 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.725640+0000", 2024-09-16T05:18:21.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012358159000000001 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.729235+0000", 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035948619999999999 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.733348+0000", 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041137650000000001 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.733388+0000", 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9931000000000001e-05 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:21.733430+0000", 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1788e-05 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3597 2.4c 2.118008cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:18:21.714 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 114.920457715, 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98230618599999997, 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3597 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.715 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746119+0000", 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9999999999999995e-07 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746124+0000", 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3349999999999997e-06 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746136+0000", 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2339e-05 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746143+0000", 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3940000000000001e-06 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746201+0000", 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8529000000000002e-05 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746206+0000", 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0250000000000002e-06 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728413+0000", 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98220633700000004 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728424+0000", 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1226999999999999e-05 2024-09-16T05:18:21.719 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3598 2.4c 2.118008cc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 114.920416151, 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.982592205, 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:18:21.720 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3598 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:18:21.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.722 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.722 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746160+0000", 2024-09-16T05:18:21.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6899999999999997e-07 2024-09-16T05:18:21.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746162+0000", 2024-09-16T05:18:21.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.323e-06 2024-09-16T05:18:21.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746170+0000", 2024-09-16T05:18:21.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4339999999999992e-06 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746176+0000", 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746227+0000", 2024-09-16T05:18:21.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1848999999999998e-05 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746232+0000", 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5619999999999997e-06 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728741+0000", 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98250897100000001 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728751+0000", 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3703 2.4a 2:53edbf55:::smithi20457400-35:head [setxattr _header (58) in=7b,truncate 2773254] snapc a0=[9f,9e,9d,9c] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:36.885921+0000", 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 104.780654286, 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.178956013, 2024-09-16T05:18:21.726 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3703 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885921+0000", 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885921+0000", 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885921+0000", 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8700000000000001e-07 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885924+0000", 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.977e-06 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885929+0000", 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6040000000000003e-06 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885932+0000", 2024-09-16T05:18:21.729 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0350000000000002e-06 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.886586+0000", 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00065428800000000005 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.886667+0000", 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0620999999999999e-05 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5", 2024-09-16T05:18:21.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.886799+0000", 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000132368 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.887082+0000", 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028310199999999997 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064801+0000", 2024-09-16T05:18:21.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17771809999999999 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064837+0000", 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6449000000000003e-05 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064877+0000", 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0082e-05 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.732 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3704 2.4a 2:53edbf55:::smithi20457400-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e447)", 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:36.885945+0000", 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 104.780629815, 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17907853700000001, 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.733 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3704 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885945+0000", 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885945+0000", 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885946+0000", 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.77e-07 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885947+0000", 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5099999999999999e-06 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885952+0000", 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2699999999999998e-06 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.885955+0000", 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9849999999999998e-06 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.887012+0000", 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001057051 2024-09-16T05:18:21.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.887062+0000", 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0244999999999998e-05 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064912+0000", 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17785025199999999 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064962+0000", 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0040999999999997e-05 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.065024+0000", 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1805999999999997e-05 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:3703 2.4a e447/325 ondisk, result = 0)", 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:36.887467+0000", 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 104.779107845, 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17741509599999999, 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.887467+0000", 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.887467+0000", 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.740 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.887468+0000", 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1800000000000001e-07 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:36.887470+0000", 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1349999999999999e-06 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064739+0000", 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17726906100000001 2024-09-16T05:18:21.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064746+0000", 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5370000000000003e-06 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064790+0000", 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4710999999999999e-05 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064797+0000", 2024-09-16T05:18:21.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3879999999999997e-06 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:37.064883+0000", 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5846000000000004e-05 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.10.0:319 2.4a e458/325 ondisk, result = 0)", 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:18:21.743 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.850103622999995, 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.733792216, 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816473+0000", 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.206e-06 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816480+0000", 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4789999999999999e-06 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550063+0000", 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73358232899999998 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550069+0000", 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3489999999999997e-06 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550155+0000", 2024-09-16T05:18:21.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5490999999999999e-05 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550163+0000", 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1859999999999992e-06 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550264+0000", 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010117599999999999 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.747 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.2.0:46 2.4d e466/440 2:b21a57e7:::smithi20457400-25:a8 v 466'478, mlcod=464'474)", 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.130225+0000", 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 85.536349895000001, 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13720005900000001, 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.130225+0000", 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.130225+0000", 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.130226+0000", 2024-09-16T05:18:21.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1399999999999998e-07 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.130231+0000", 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0710000000000002e-06 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.266745+0000", 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.13651454900000001 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.266772+0000", 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6783000000000001e-05 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.266833+0000", 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1022e-05 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.266854+0000", 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0471000000000001e-05 2024-09-16T05:18:21.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.267367+0000", 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000513792 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.267425+0000", 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8057000000000003e-05 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.752 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3966 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 645534~783707] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 83.777576859999996, 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.228671191, 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3966 2024-09-16T05:18:21.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889000+0000", 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.745e-06 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889398+0000", 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000397644 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889415+0000", 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7368e-05 2024-09-16T05:18:21.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889440+0000", 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4358999999999998e-05 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889496+0000", 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6168999999999997e-05 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889500+0000", 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0579999999999999e-06 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107487+0000", 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21798717200000001 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107565+0000", 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7715999999999999e-05 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107734+0000", 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016950099999999999 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111344+0000", 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036100160000000002 2024-09-16T05:18:21.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117576+0000", 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062321929999999996 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117628+0000", 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.189e-05 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117670+0000", 2024-09-16T05:18:21.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1359999999999997e-05 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3967 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 2103939~638517] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 83.770651830000006, 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.221880365, 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.760 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3967 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895924+0000", 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.85e-07 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896200+0000", 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027563199999999999 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896212+0000", 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1671999999999999e-05 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896217+0000", 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3970000000000002e-06 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896274+0000", 2024-09-16T05:18:21.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6740000000000002e-05 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896280+0000", 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109156+0000", 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.212875278 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109242+0000", 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6163999999999997e-05 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109378+0000", 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013637400000000001 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111350+0000", 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019716429999999999 2024-09-16T05:18:21.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117716+0000", 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063661830000000001 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117776+0000", 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9753999999999997e-05 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117804+0000", 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.817e-05 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3968 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 3150663~294871] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 83.766972773999996, 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218327628, 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.767 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3968 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8800000000000002e-07 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899782+0000", 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017861999999999999 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899798+0000", 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5772000000000001e-05 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899806+0000", 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.157e-06 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899867+0000", 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0711000000000002e-05 2024-09-16T05:18:21.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899873+0000", 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0850000000000002e-06 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110009+0000", 2024-09-16T05:18:21.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21013668299999999 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110084+0000", 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4176000000000003e-05 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110205+0000", 2024-09-16T05:18:21.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012158 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111356+0000", 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001150646 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117849+0000", 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064928659999999999 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117895+0000", 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6873999999999999e-05 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117930+0000", 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4669999999999998e-05 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3969 2.1d 2:ba7e58cf:::smithi20457400-26:head [setxattr _header (58) in=7b,truncate 3445534] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899824+0000", 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 83.766751540000001, 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218199477, 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3969 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.775 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899824+0000", 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899824+0000", 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899824+0000", 2024-09-16T05:18:21.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4000000000000001e-07 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899828+0000", 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8229999999999997e-06 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899836+0000", 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6480000000000003e-06 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899842+0000", 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9279999999999998e-06 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899917+0000", 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4803999999999997e-05 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899923+0000", 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4910000000000002e-06 2024-09-16T05:18:21.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110785+0000", 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.210862043 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110867+0000", 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1458000000000005e-05 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:18:21.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110995+0000", 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012817800000000001 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111361+0000", 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036600599999999999 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117961+0000", 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0065997770000000002 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117993+0000", 2024-09-16T05:18:21.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1998000000000001e-05 2024-09-16T05:18:21.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118023+0000", 2024-09-16T05:18:21.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0682999999999997e-05 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3970 2.1d 2:ba7e58cf:::smithi20457400-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 83.766724185000001, 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218352989, 2024-09-16T05:18:21.782 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.783 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.783 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:21.783 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.783 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.783 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.783 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3970 2024-09-16T05:18:21.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:18:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899852+0000", 2024-09-16T05:18:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8299999999999997e-07 2024-09-16T05:18:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899854+0000", 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.463e-06 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899861+0000", 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0079999999999996e-06 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899866+0000", 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899949+0000", 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2713000000000006e-05 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899954+0000", 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7400000000000004e-06 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111261+0000", 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.211307514 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111332+0000", 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1198999999999994e-05 2024-09-16T05:18:21.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118056+0000", 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067238530000000001 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118121+0000", 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4770000000000005e-05 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118204+0000", 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3051000000000002e-05 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3986 2.1c 2:3a3554ba:::smithi20457400-48:head [delete] snapc a9=[a9] ondisk+write+known_if_redirected+full_force+supports_pool_eio e468)", 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.922930+0000", 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 83.743645447000006, 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19188190599999999, 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.790 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:18:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3986 2024-09-16T05:18:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.922930+0000", 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.922930+0000", 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.922932+0000", 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2699999999999999e-06 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.922939+0000", 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3409999999999997e-06 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.922956+0000", 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6160000000000001e-05 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.922965+0000", 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2750000000000001e-06 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.923034+0000", 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8874999999999994e-05 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.923038+0000", 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5449999999999997e-06 2024-09-16T05:18:21.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107125+0000", 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.18408670999999999 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107190+0000", 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5017000000000004e-05 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.114812+0000", 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076217129999999996 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:4971 2.1d 2:ba7e58cf:::smithi20457400-26:head [stat,delete] snapc cc=[cb,ca,c9,c7,c5,c4,c3,c2,c1] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-09-16T05:18:21.796 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.468142746000002, 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.70425506000000004, 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4971 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.797 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:18:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198434+0000", 2024-09-16T05:18:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8300000000000001e-06 2024-09-16T05:18:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:18:21.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198443+0000", 2024-09-16T05:18:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-16T05:18:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:18:21.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198460+0000", 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7229999999999999e-05 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198467+0000", 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1409999999999999e-06 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198531+0000", 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3371e-05 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198539+0000", 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8930000000000005e-06 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891869+0000", 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69333044700000002 2024-09-16T05:18:21.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891964+0000", 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4990999999999999e-05 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892092+0000", 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012792599999999999 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:18:21.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892842+0000", 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075013 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902619+0000", 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0097773840000000001 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:18:21.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902658+0000", 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8167999999999998e-05 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902688+0000", 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9991000000000001e-05 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:18:21.805 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:18:21.806 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:18:21.807 INFO:tasks.rados.rados.2.smithi204.stdout:2034: done (1 left) 2024-09-16T05:18:21.807 INFO:tasks.rados.rados.2.smithi204.stdout:2035: done (0 left) 2024-09-16T05:18:21.807 INFO:tasks.rados.rados.2.smithi204.stdout:2036: read oid 14 snap -1 2024-09-16T05:18:21.807 INFO:tasks.rados.rados.2.smithi204.stdout:2036: expect (ObjNum 728 snap 210 seq_num 728) 2024-09-16T05:18:21.808 INFO:tasks.rados.rados.2.smithi204.stdout:2037: snap_create 2024-09-16T05:18:21.808 INFO:tasks.rados.rados.2.smithi204.stdout:2036: expect (ObjNum 728 snap 210 seq_num 728) 2024-09-16T05:18:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:21.808 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:21.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:18:21.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:21.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:21.974 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:18:21.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:18:21.974 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-16T05:18:21.974 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:18:21.974 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:18:21.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-16T05:18:22.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:22.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:22.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:22.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:22.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:21 smithi032 ceph-mon[27388]: pgmap v987: 107 pgs: 107 active+clean; 250 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-09-16T05:18:22.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:21 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:22.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:21 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:22.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/289241281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:22.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/289241281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:22.087 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:21 smithi032 ceph-mon[27388]: osdmap e552: 12 total, 12 up, 12 in 2024-09-16T05:18:22.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:22.184 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:22.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:22.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:21 smithi196 ceph-mon[30802]: pgmap v987: 107 pgs: 107 active+clean; 250 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-09-16T05:18:22.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:21 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:22.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:21 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/289241281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/289241281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:21 smithi196 ceph-mon[30802]: osdmap e552: 12 total, 12 up, 12 in 2024-09-16T05:18:22.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:22.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:18:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:22.309 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:22.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:22.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:22.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:21 smithi204 ceph-mon[30933]: pgmap v987: 107 pgs: 107 active+clean; 250 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-09-16T05:18:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:21 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:21 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/289241281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/289241281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:21 smithi204 ceph-mon[30933]: osdmap e552: 12 total, 12 up, 12 in 2024-09-16T05:18:22.432 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:22.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:22.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:22.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:22.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:22.561 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:22.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:18:22.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:22.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:22.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:22.715 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:18:22.715 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T05:18:22.715 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:18:22.758 INFO:tasks.rados.rados.2.smithi204.stdout:2036: done (1 left) 2024-09-16T05:18:22.759 INFO:tasks.rados.rados.2.smithi204.stdout:2037: done (0 left) 2024-09-16T05:18:22.759 INFO:tasks.rados.rados.2.smithi204.stdout:2038: snap_remove snap 199 2024-09-16T05:18:22.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:22.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:22.838 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:22.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:22.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:22.929 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:23.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:23.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:23.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:23.125 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:23.125 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:23.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:23.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:23.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:23.165 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:23.166 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:23.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:23.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:23.186 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:23.186 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:23.208 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:23.209 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:23.230 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:23.252 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:23.273 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:23.273 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:23.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:23.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:23.294 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:23.299 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:23.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:23.316 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:23.317 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:23.339 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:23.339 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:23.339 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:18:23.339 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:18:23.340 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:18:23.340 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:18:23.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:23.360 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:23.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:23.424 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:23.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:23.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:23.548 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:23.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:23.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:23.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:23.763 INFO:tasks.rados.rados.2.smithi204.stdout:2038: done (0 left) 2024-09-16T05:18:23.763 INFO:tasks.rados.rados.2.smithi204.stdout:2039: delete oid 2 current snap is 215 2024-09-16T05:18:23.765 INFO:tasks.rados.rados.2.smithi204.stdout:2039: done (0 left) 2024-09-16T05:18:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:2040: read oid 9 snap -1 2024-09-16T05:18:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:2040: expect (ObjNum 616 snap 181 seq_num 616) 2024-09-16T05:18:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:2041: read oid 14 snap -1 2024-09-16T05:18:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:2041: expect (ObjNum 728 snap 210 seq_num 728) 2024-09-16T05:18:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:2042: read oid 40 snap -1 2024-09-16T05:18:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:2042: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:18:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:2043: write (excl) oid 16 current snap is 215 2024-09-16T05:18:23.766 INFO:tasks.rados.rados.2.smithi204.stdout:2043: seq_num 734 ranges {736338=402654,1552136=539580} 2024-09-16T05:18:23.772 INFO:tasks.rados.rados.2.smithi204.stdout:2043: writing smithi20457400-16 from 736338 to 1138992 tid 1 2024-09-16T05:18:23.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:23.780 INFO:tasks.rados.rados.2.smithi204.stdout:2043: writing smithi20457400-16 from 1552136 to 2091716 tid 2 2024-09-16T05:18:23.780 INFO:tasks.rados.rados.2.smithi204.stdout:2041: expect (ObjNum 728 snap 210 seq_num 728) 2024-09-16T05:18:23.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:23.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:23.861 INFO:tasks.rados.rados.2.smithi204.stdout:2042: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:18:23.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:23.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:18:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:23.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:23.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:23.967 INFO:tasks.rados.rados.2.smithi204.stdout:2043: finishing write tid 1 to smithi20457400-16 2024-09-16T05:18:23.967 INFO:tasks.rados.rados.2.smithi204.stdout:2041: done (3 left) 2024-09-16T05:18:23.967 INFO:tasks.rados.rados.2.smithi204.stdout:2042: done (2 left) 2024-09-16T05:18:23.967 INFO:tasks.rados.rados.2.smithi204.stdout:2044: delete oid 15 current snap is 215 2024-09-16T05:18:23.968 INFO:tasks.rados.rados.2.smithi204.stdout:2040: expect (ObjNum 616 snap 181 seq_num 616) 2024-09-16T05:18:24.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:24.046 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:24.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:2043: finishing write tid 2 to smithi20457400-16 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:2043: finishing write tid 3 to smithi20457400-16 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:2043: finishing write tid 4 to smithi20457400-16 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 718 (ObjNum 734 snap 215 seq_num 734) dirty exists 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:2043: left oid 16 (ObjNum 734 snap 215 seq_num 734) 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:2040: done (2 left) 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:2043: done (1 left) 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:2044: done (0 left) 2024-09-16T05:18:24.098 INFO:tasks.rados.rados.2.smithi204.stdout:2045: copy_from oid 16 from oid 12 current snap is 215 2024-09-16T05:18:24.099 INFO:tasks.rados.rados.2.smithi204.stdout:2046: copy_from oid 22 from oid 48 current snap is 215 2024-09-16T05:18:24.099 INFO:tasks.rados.rados.2.smithi204.stdout:2047: read oid 10 snap -1 2024-09-16T05:18:24.099 INFO:tasks.rados.rados.2.smithi204.stdout:2047: expect (ObjNum 701 snap 202 seq_num 701) 2024-09-16T05:18:24.099 INFO:tasks.rados.rados.2.smithi204.stdout:2048: read oid 33 snap 211 2024-09-16T05:18:24.099 INFO:tasks.rados.rados.2.smithi204.stdout:2048: expect deleted 2024-09-16T05:18:24.099 INFO:tasks.rados.rados.2.smithi204.stdout:2049: delete oid 1 current snap is 215 2024-09-16T05:18:24.099 INFO:tasks.rados.rados.2.smithi204.stdout:2048: done (4 left) 2024-09-16T05:18:24.099 INFO:tasks.rados.rados.2.smithi204.stdout:2049: done (3 left) 2024-09-16T05:18:24.100 INFO:tasks.rados.rados.2.smithi204.stdout:2050: rollback oid 44 current snap is 215 2024-09-16T05:18:24.100 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 213 2024-09-16T05:18:24.100 INFO:tasks.rados.rados.2.smithi204.stdout:2051: snap_remove snap 211 2024-09-16T05:18:24.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:24.165 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: pgmap v990: 107 pgs: 107 active+clean; 254 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 1.4 MiB/s wr, 20 op/s 2024-09-16T05:18:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:18:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:18:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:18:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: osdmap e553: 12 total, 12 up, 12 in 2024-09-16T05:18:24.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:18:24.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:18:24.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2104552695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:24.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:18:24.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:23 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:18:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: pgmap v990: 107 pgs: 107 active+clean; 254 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 1.4 MiB/s wr, 20 op/s 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: osdmap e553: 12 total, 12 up, 12 in 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2104552695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:18:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:23 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:18:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:24.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:24.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: pgmap v990: 107 pgs: 107 active+clean; 254 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 1.4 MiB/s wr, 20 op/s 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: osdmap e553: 12 total, 12 up, 12 in 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2104552695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:18:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:23 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:18:24.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:24.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:24.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:24.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:24.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:24.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:24.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:18:24.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:24.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:24.610 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:24.610 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:24.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:24.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:24.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:24.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:24.776 INFO:tasks.rados.rados.2.smithi204.stdout:2051: done (4 left) 2024-09-16T05:18:24.776 INFO:tasks.rados.rados.2.smithi204.stdout:2052: write oid 2 current snap is 215 2024-09-16T05:18:24.776 INFO:tasks.rados.rados.2.smithi204.stdout:2050: finishing rollback tid 0 to smithi20457400-44 2024-09-16T05:18:24.777 INFO:tasks.rados.rados.2.smithi204.stdout:2052: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-16T05:18:24.784 INFO:tasks.rados.rados.2.smithi204.stdout:2052: writing smithi20457400-2 from 786759 to 1415160 tid 1 2024-09-16T05:18:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:24.789 INFO:tasks.rados.rados.2.smithi204.stdout:2052: writing smithi20457400-2 from 2011229 to 2386759 tid 2 2024-09-16T05:18:24.790 INFO:tasks.rados.rados.2.smithi204.stdout:2053: delete oid 27 current snap is 215 2024-09-16T05:18:24.790 INFO:tasks.rados.rados.2.smithi204.stdout:2050: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:18:24.790 INFO:tasks.rados.rados.2.smithi204.stdout:2050: finishing rollback tid 2 to smithi20457400-44 2024-09-16T05:18:24.790 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 713 (ObjNum 692 snap 201 seq_num 692) dirty exists 2024-09-16T05:18:24.790 INFO:tasks.rados.rados.2.smithi204.stdout:2047: expect (ObjNum 701 snap 202 seq_num 701) 2024-09-16T05:18:24.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:24.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:24.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: osdmap e554: 12 total, 12 up, 12 in 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:18:24.890 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:18:24.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:24.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:24.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:18:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:18:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:18:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: osdmap e554: 12 total, 12 up, 12 in 2024-09-16T05:18:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:18:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:18:24.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:18:24.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:18:24.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:18:24.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:18:25.013 INFO:tasks.rados.rados.2.smithi204.stdout:2046: finishing copy_from racing read to smithi20457400-22 2024-09-16T05:18:25.013 INFO:tasks.rados.rados.2.smithi204.stdout:2045: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2045: finishing copy_from to smithi20457400-16 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 712 (ObjNum 668 snap 194 seq_num 668) dirty exists 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2046: finishing copy_from to smithi20457400-22 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 396 (ObjNum 592 snap 180 seq_num 592) dirty exists 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2052: finishing write tid 1 to smithi20457400-2 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2052: finishing write tid 2 to smithi20457400-2 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2052: finishing write tid 3 to smithi20457400-2 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2052: finishing write tid 4 to smithi20457400-2 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 518 (ObjNum 737 snap 215 seq_num 737) dirty exists 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2052: left oid 2 (ObjNum 737 snap 215 seq_num 737) 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2045: done (5 left) 2024-09-16T05:18:25.014 INFO:tasks.rados.rados.2.smithi204.stdout:2046: done (4 left) 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2047: done (3 left) 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2050: done (2 left) 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2052: done (1 left) 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2053: done (0 left) 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2054: copy_from oid 33 from oid 35 current snap is 215 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2055: copy_from oid 4 from oid 13 current snap is 215 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2056: read oid 3 snap -1 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2056: expect (ObjNum 708 snap 203 seq_num 708) 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2057: write (excl) oid 27 current snap is 215 2024-09-16T05:18:25.015 INFO:tasks.rados.rados.2.smithi204.stdout:2057: seq_num 739 ranges {420373=495252,1444546=690576} 2024-09-16T05:18:25.018 INFO:tasks.rados.rados.2.smithi204.stdout:2057: writing smithi20457400-27 from 420373 to 915625 tid 1 2024-09-16T05:18:25.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:25.026 INFO:tasks.rados.rados.2.smithi204.stdout:2057: writing smithi20457400-27 from 1444546 to 2135122 tid 2 2024-09-16T05:18:25.026 INFO:tasks.rados.rados.2.smithi204.stdout:2058: write (excl) oid 6 current snap is 215 2024-09-16T05:18:25.026 INFO:tasks.rados.rados.2.smithi204.stdout:2058: seq_num 740 ranges {437180=570501,1503028=550587} 2024-09-16T05:18:25.031 INFO:tasks.rados.rados.2.smithi204.stdout:2058: writing smithi20457400-6 from 437180 to 1007681 tid 1 2024-09-16T05:18:25.037 INFO:tasks.rados.rados.2.smithi204.stdout:2058: writing smithi20457400-6 from 1503028 to 2053615 tid 2 2024-09-16T05:18:25.037 INFO:tasks.rados.rados.2.smithi204.stdout:2059: write oid 48 current snap is 215 2024-09-16T05:18:25.037 INFO:tasks.rados.rados.2.smithi204.stdout:2059: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-16T05:18:25.043 INFO:tasks.rados.rados.2.smithi204.stdout:2059: writing smithi20457400-48 from 453987 to 1099737 tid 1 2024-09-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:25.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:25.050 INFO:tasks.rados.rados.2.smithi204.stdout:2059: writing smithi20457400-48 from 1845157 to 2453987 tid 2 2024-09-16T05:18:25.051 INFO:tasks.rados.rados.2.smithi204.stdout:2060: rollback oid 44 current snap is 215 2024-09-16T05:18:25.051 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 206 2024-09-16T05:18:25.051 INFO:tasks.rados.rados.2.smithi204.stdout:2056: expect (ObjNum 708 snap 203 seq_num 708) 2024-09-16T05:18:25.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:18:25.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:18:25.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:18:25.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: osdmap e554: 12 total, 12 up, 12 in 2024-09-16T05:18:25.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:18:25.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:18:25.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:18:25.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:18:25.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:18:25.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:18:25.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:25.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:25.185 INFO:tasks.rados.rados.2.smithi204.stdout:2057: finishing write tid 1 to smithi20457400-27 2024-09-16T05:18:25.185 INFO:tasks.rados.rados.2.smithi204.stdout:2054: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:18:25.185 INFO:tasks.rados.rados.2.smithi204.stdout:2054: finishing copy_from to smithi20457400-33 2024-09-16T05:18:25.185 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 833 (ObjNum 694 snap 201 seq_num 694) dirty exists 2024-09-16T05:18:25.185 INFO:tasks.rados.rados.2.smithi204.stdout:2055: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2057: finishing write tid 2 to smithi20457400-27 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2054: done (6 left) 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2056: done (5 left) 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2061: read oid 50 snap 214 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2061: expect (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2062: rollback oid 25 current snap is 215 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 205 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2057: finishing write tid 3 to smithi20457400-27 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2057: finishing write tid 4 to smithi20457400-27 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 316 (ObjNum 739 snap 215 seq_num 739) dirty exists 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2057: left oid 27 (ObjNum 739 snap 215 seq_num 739) 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:2055: finishing copy_from to smithi20457400-4 2024-09-16T05:18:25.186 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 715 (ObjNum 727 snap 210 seq_num 727) dirty exists 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2058: finishing write tid 1 to smithi20457400-6 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2055: done (6 left) 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2057: done (5 left) 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2063: delete oid 38 current snap is 215 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2058: finishing write tid 2 to smithi20457400-6 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2058: finishing write tid 3 to smithi20457400-6 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2058: finishing write tid 4 to smithi20457400-6 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 726 (ObjNum 740 snap 215 seq_num 740) dirty exists 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2058: left oid 6 (ObjNum 740 snap 215 seq_num 740) 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2059: finishing write tid 1 to smithi20457400-48 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2060: finishing rollback tid 0 to smithi20457400-44 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2059: finishing write tid 2 to smithi20457400-48 2024-09-16T05:18:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:2059: finishing write tid 3 to smithi20457400-48 2024-09-16T05:18:25.188 INFO:tasks.rados.rados.2.smithi204.stdout:2060: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:18:25.188 INFO:tasks.rados.rados.2.smithi204.stdout:2059: finishing write tid 4 to smithi20457400-48 2024-09-16T05:18:25.188 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 399 (ObjNum 741 snap 215 seq_num 741) dirty exists 2024-09-16T05:18:25.188 INFO:tasks.rados.rados.2.smithi204.stdout:2059: left oid 48 (ObjNum 741 snap 215 seq_num 741) 2024-09-16T05:18:25.188 INFO:tasks.rados.rados.2.smithi204.stdout:2060: finishing rollback tid 2 to smithi20457400-44 2024-09-16T05:18:25.188 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 716 (ObjNum 698 snap 202 seq_num 698) dirty exists 2024-09-16T05:18:25.188 INFO:tasks.rados.rados.2.smithi204.stdout:2058: done (5 left) 2024-09-16T05:18:25.188 INFO:tasks.rados.rados.2.smithi204.stdout:2059: done (4 left) 2024-09-16T05:18:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:2060: done (3 left) 2024-09-16T05:18:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:2063: done (2 left) 2024-09-16T05:18:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:2064: write oid 4 current snap is 215 2024-09-16T05:18:25.189 INFO:tasks.rados.rados.2.smithi204.stdout:2064: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-16T05:18:25.193 INFO:tasks.rados.rados.2.smithi204.stdout:2064: writing smithi20457400-4 from 487601 to 1283849 tid 1 2024-09-16T05:18:25.200 INFO:tasks.rados.rados.2.smithi204.stdout:2064: writing smithi20457400-4 from 1962121 to 2487601 tid 2 2024-09-16T05:18:25.201 INFO:tasks.rados.rados.2.smithi204.stdout:2065: read oid 5 snap -1 2024-09-16T05:18:25.201 INFO:tasks.rados.rados.2.smithi204.stdout:2065: expect (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:18:25.201 INFO:tasks.rados.rados.2.smithi204.stdout:2066: snap_remove snap 207 2024-09-16T05:18:25.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:18:25.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:25.304 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:25.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:25.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:25.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:25.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:25.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:25.523 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:18:25.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:25.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:25.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:25.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:25.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:25.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:18:25.758 INFO:tasks.rados.rados.2.smithi204.stdout:2066: done (4 left) 2024-09-16T05:18:25.758 INFO:tasks.rados.rados.2.smithi204.stdout:2067: write oid 12 current snap is 215 2024-09-16T05:18:25.758 INFO:tasks.rados.rados.2.smithi204.stdout:2062: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:18:25.758 INFO:tasks.rados.rados.2.smithi204.stdout:2067: seq_num 744 ranges {504408=471497,1620603=557925} 2024-09-16T05:18:25.764 INFO:tasks.rados.rados.2.smithi204.stdout:2067: writing smithi20457400-12 from 504408 to 975905 tid 1 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:2067: writing smithi20457400-12 from 1620603 to 2178528 tid 2 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:2068: rollback oid 46 current snap is 215 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 195 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:2062: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:2062: finishing rollback tid 2 to smithi20457400-25 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 713 (ObjNum 692 snap 201 seq_num 692) dirty exists 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:2064: finishing write tid 1 to smithi20457400-4 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:2064: finishing write tid 2 to smithi20457400-4 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:2064: finishing write tid 3 to smithi20457400-4 2024-09-16T05:18:25.772 INFO:tasks.rados.rados.2.smithi204.stdout:2064: finishing write tid 4 to smithi20457400-4 2024-09-16T05:18:25.773 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 718 (ObjNum 743 snap 215 seq_num 743) dirty exists 2024-09-16T05:18:25.773 INFO:tasks.rados.rados.2.smithi204.stdout:2064: left oid 4 (ObjNum 743 snap 215 seq_num 743) 2024-09-16T05:18:25.773 INFO:tasks.rados.rados.2.smithi204.stdout:2065: expect (ObjNum 653 snap 189 seq_num 653) 2024-09-16T05:18:25.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:25.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:25.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:25.878 INFO:tasks.rados.rados.2.smithi204.stdout:2061: expect (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:18:25.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:25.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: osdmap e555: 12 total, 12 up, 12 in 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2141034973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: 2.29 scrub ok 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:18:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:18:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:18:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2141034973' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:25 smithi204 ceph-mon[30933]: osdmap e556: 12 total, 12 up, 12 in 2024-09-16T05:18:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:25.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:25.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:26.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:26.040 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:26.040 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:26.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:26.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:2067: finishing write tid 1 to smithi20457400-12 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:2068: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:2068: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:2068: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 348 (ObjNum 638 snap 187 seq_num 638) dirty exists 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:2067: finishing write tid 2 to smithi20457400-12 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:2061: done (5 left) 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:2062: done (4 left) 2024-09-16T05:18:26.071 INFO:tasks.rados.rados.2.smithi204.stdout:2064: done (3 left) 2024-09-16T05:18:26.072 INFO:tasks.rados.rados.2.smithi204.stdout:2065: done (2 left) 2024-09-16T05:18:26.072 INFO:tasks.rados.rados.2.smithi204.stdout:2068: done (1 left) 2024-09-16T05:18:26.072 INFO:tasks.rados.rados.2.smithi204.stdout:2069: snap_remove snap 195 2024-09-16T05:18:26.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: osdmap e555: 12 total, 12 up, 12 in 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2141034973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: 2.29 scrub ok 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:18:26.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:26.081 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:26.081 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:26.081 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:18:26.081 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2141034973' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:26.081 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:25 smithi032 ceph-mon[27388]: osdmap e556: 12 total, 12 up, 12 in 2024-09-16T05:18:26.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:26.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:26.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:18:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:18:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:18:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:18:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: osdmap e555: 12 total, 12 up, 12 in 2024-09-16T05:18:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2141034973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:18:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: 2.29 scrub ok 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2141034973' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:25 smithi196 ceph-mon[30802]: osdmap e556: 12 total, 12 up, 12 in 2024-09-16T05:18:26.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:18:26.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:26.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:26.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:26.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:26.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:26.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:26.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:26.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:26.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:26.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:26.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:26.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:26.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:18:26.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:26.780 INFO:tasks.rados.rados.2.smithi204.stdout:2069: done (1 left) 2024-09-16T05:18:26.780 INFO:tasks.rados.rados.2.smithi204.stdout:2070: read oid 43 snap -1 2024-09-16T05:18:26.780 INFO:tasks.rados.rados.2.smithi204.stdout:2070: expect deleted 2024-09-16T05:18:26.780 INFO:tasks.rados.rados.2.smithi204.stdout:2071: copy_from oid 17 from oid 31 current snap is 215 2024-09-16T05:18:26.780 INFO:tasks.rados.rados.2.smithi204.stdout:2072: read oid 49 snap -1 2024-09-16T05:18:26.780 INFO:tasks.rados.rados.2.smithi204.stdout:2072: expect (ObjNum 704 snap 202 seq_num 704) 2024-09-16T05:18:26.780 INFO:tasks.rados.rados.2.smithi204.stdout:2073: write (excl) oid 27 current snap is 215 2024-09-16T05:18:26.780 INFO:tasks.rados.rados.2.smithi204.stdout:2073: seq_num 745 ranges {521215=546746,1562732=701583} 2024-09-16T05:18:26.786 INFO:tasks.rados.rados.2.smithi204.stdout:2073: writing smithi20457400-27 from 521215 to 1067961 tid 1 2024-09-16T05:18:26.794 INFO:tasks.rados.rados.2.smithi204.stdout:2073: writing smithi20457400-27 from 1562732 to 2264315 tid 2 2024-09-16T05:18:26.794 INFO:tasks.rados.rados.2.smithi204.stdout:2074: write oid 9 current snap is 215 2024-09-16T05:18:26.795 INFO:tasks.rados.rados.2.smithi204.stdout:2067: finishing write tid 3 to smithi20457400-12 2024-09-16T05:18:26.795 INFO:tasks.rados.rados.2.smithi204.stdout:2067: finishing write tid 4 to smithi20457400-12 2024-09-16T05:18:26.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 735 (ObjNum 744 snap 215 seq_num 744) dirty exists 2024-09-16T05:18:26.795 INFO:tasks.rados.rados.2.smithi204.stdout:2067: left oid 12 (ObjNum 744 snap 215 seq_num 744) 2024-09-16T05:18:26.795 INFO:tasks.rados.rados.2.smithi204.stdout:2074: seq_num 746 ranges {538022=738348,1737567=561594} 2024-09-16T05:18:26.801 INFO:tasks.rados.rados.2.smithi204.stdout:2074: writing smithi20457400-9 from 538022 to 1276370 tid 1 2024-09-16T05:18:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:26.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:26.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:26.807 INFO:tasks.rados.rados.2.smithi204.stdout:2074: writing smithi20457400-9 from 1737567 to 2299161 tid 2 2024-09-16T05:18:26.807 INFO:tasks.rados.rados.2.smithi204.stdout:2067: done (5 left) 2024-09-16T05:18:26.807 INFO:tasks.rados.rados.2.smithi204.stdout:2075: snap_create 2024-09-16T05:18:26.807 INFO:tasks.rados.rados.2.smithi204.stdout:2071: finishing copy_from to smithi20457400-17 2024-09-16T05:18:26.808 INFO:tasks.rados.rados.2.smithi204.stdout:2071: got expected ENOENT (src dne) 2024-09-16T05:18:26.808 INFO:tasks.rados.rados.2.smithi204.stdout:2071: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:18:26.808 INFO:tasks.rados.rados.2.smithi204.stdout:2072: expect (ObjNum 704 snap 202 seq_num 704) 2024-09-16T05:18:26.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:26.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:26.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:27.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2073: finishing write tid 1 to smithi20457400-27 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2073: finishing write tid 2 to smithi20457400-27 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2073: finishing write tid 3 to smithi20457400-27 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2073: finishing write tid 4 to smithi20457400-27 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 319 (ObjNum 745 snap 215 seq_num 745) dirty exists 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2073: left oid 27 (ObjNum 745 snap 215 seq_num 745) 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2074: finishing write tid 1 to smithi20457400-9 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2074: finishing write tid 2 to smithi20457400-9 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2074: finishing write tid 3 to smithi20457400-9 2024-09-16T05:18:27.050 INFO:tasks.rados.rados.2.smithi204.stdout:2074: finishing write tid 4 to smithi20457400-9 2024-09-16T05:18:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 730 (ObjNum 746 snap 215 seq_num 746) dirty exists 2024-09-16T05:18:27.051 INFO:tasks.rados.rados.2.smithi204.stdout:2074: left oid 9 (ObjNum 746 snap 215 seq_num 746) 2024-09-16T05:18:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:27.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:27.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:27.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:26 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:26 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:26 smithi204 ceph-mon[30933]: 2.3c scrub starts 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:26 smithi204 ceph-mon[30933]: 2.3c scrub ok 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:26 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:26 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:26 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:26 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:18:27.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:26 smithi032 ceph-mon[27388]: 2.3c scrub starts 2024-09-16T05:18:27.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:26 smithi032 ceph-mon[27388]: 2.3c scrub ok 2024-09-16T05:18:27.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:26 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:18:27.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:26 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:18:27.175 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:27.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:27.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:18:27.176+0000 7f4158854640 0 --2- 172.21.15.32:0/1322385307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f415410c170 0x7f415419c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:18:27.205 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:27.205 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:27.227 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:27.227 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:27.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:26 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:18:27.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:26 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:18:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:26 smithi196 ceph-mon[30802]: 2.3c scrub starts 2024-09-16T05:18:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:26 smithi196 ceph-mon[30802]: 2.3c scrub ok 2024-09-16T05:18:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:26 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:18:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:26 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:18:27.247 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:27.248 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:27.268 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:27.268 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:27.277 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:27.291 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:27.291 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:27.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:27.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:27.313 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:27.313 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:27.339 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:27.360 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:27.381 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:27.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:27.403 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:27.403 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:27.424 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:27.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:27.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:27.446 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:27.446 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:27.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:27.527 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:18:27.527 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:18:27.527 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:18:27.527 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:18:27.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:27.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:27.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:27.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:27.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:27.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2070: done (5 left) 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2071: done (4 left) 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2072: done (3 left) 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2073: done (2 left) 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2074: done (1 left) 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2075: done (0 left) 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2076: read oid 19 snap -1 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2076: expect deleted 2024-09-16T05:18:27.786 INFO:tasks.rados.rados.2.smithi204.stdout:2077: snap_remove snap 212 2024-09-16T05:18:27.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:27.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:27.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:27.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:27.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:28.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:28.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:28.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: pgmap v996: 107 pgs: 107 active+clean; 264 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 21 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: osdmap e557: 12 total, 12 up, 12 in 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: 1.11 deep-scrub starts 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: 1.11 deep-scrub ok 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:18:28.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1322385307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:28.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:28.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: pgmap v996: 107 pgs: 107 active+clean; 264 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 21 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: osdmap e557: 12 total, 12 up, 12 in 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: 1.11 deep-scrub starts 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: 1.11 deep-scrub ok 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:18:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1322385307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:28.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:28.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: pgmap v996: 107 pgs: 107 active+clean; 264 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 21 op/s; 18 KiB/s, 0 objects/s recovering 2024-09-16T05:18:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:18:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:18:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: osdmap e557: 12 total, 12 up, 12 in 2024-09-16T05:18:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: 1.11 deep-scrub starts 2024-09-16T05:18:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: 1.11 deep-scrub ok 2024-09-16T05:18:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:18:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:18:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1322385307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:28.273 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:28.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:28.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:28.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:28.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:18:28.419 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:18:28.419 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-16T05:18:28.420 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T05:18:28.420 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T05:18:28.420 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-16T05:18:28.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:28.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:28.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:18:28.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:28.548 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:28.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:28.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:28.668 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:28.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:28.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:28.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:28.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:28.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:28.800 INFO:tasks.rados.rados.2.smithi204.stdout:2077: done (1 left) 2024-09-16T05:18:28.801 INFO:tasks.rados.rados.2.smithi204.stdout:2078: snap_create 2024-09-16T05:18:28.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:18:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:28.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:28.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:28.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:29.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:29.046 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:29.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:29.075 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:28 smithi196 ceph-mon[30802]: 2.6 deep-scrub starts 2024-09-16T05:18:29.076 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:28 smithi196 ceph-mon[30802]: 2.6 deep-scrub ok 2024-09-16T05:18:29.076 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:28 smithi196 ceph-mon[30802]: osdmap e558: 12 total, 12 up, 12 in 2024-09-16T05:18:29.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:28 smithi032 ceph-mon[27388]: 2.6 deep-scrub starts 2024-09-16T05:18:29.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:28 smithi032 ceph-mon[27388]: 2.6 deep-scrub ok 2024-09-16T05:18:29.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:28 smithi032 ceph-mon[27388]: osdmap e558: 12 total, 12 up, 12 in 2024-09-16T05:18:29.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:29.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:28 smithi204 ceph-mon[30933]: 2.6 deep-scrub starts 2024-09-16T05:18:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:28 smithi204 ceph-mon[30933]: 2.6 deep-scrub ok 2024-09-16T05:18:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:28 smithi204 ceph-mon[30933]: osdmap e558: 12 total, 12 up, 12 in 2024-09-16T05:18:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:29.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:29.175 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:18:29.175 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "" 2024-09-16T05:18:29.175 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:18:29.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:29.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:29.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:29.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:29.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:29.425 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:29.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:29.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:29.549 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:29.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:29.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:29.673 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:29.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:29.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:29.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:29.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:29.799 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:18:29.801 INFO:tasks.rados.rados.2.smithi204.stdout:2076: done (1 left) 2024-09-16T05:18:29.801 INFO:tasks.rados.rados.2.smithi204.stdout:2078: done (0 left) 2024-09-16T05:18:29.801 INFO:tasks.rados.rados.2.smithi204.stdout:2079: read oid 3 snap -1 2024-09-16T05:18:29.802 INFO:tasks.rados.rados.2.smithi204.stdout:2079: expect (ObjNum 708 snap 203 seq_num 708) 2024-09-16T05:18:29.802 INFO:tasks.rados.rados.2.smithi204.stdout:2080: snap_remove snap 201 2024-09-16T05:18:29.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:29.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:29.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:29.926 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:18:29.926 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:18:29.926 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:18:29.926 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:18:29.926 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:18:30.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:30.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:30.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:30.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:30.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:18:30.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:30.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:29 smithi032 ceph-mon[27388]: pgmap v999: 107 pgs: 107 active+clean; 275 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 5.3 MiB/s wr, 30 op/s 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:29 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:29 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:29 smithi032 ceph-mon[27388]: 2.3f scrub starts 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:29 smithi032 ceph-mon[27388]: 2.3f scrub ok 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:29 smithi032 ceph-mon[27388]: osdmap e559: 12 total, 12 up, 12 in 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/576627261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:29 smithi204 ceph-mon[30933]: pgmap v999: 107 pgs: 107 active+clean; 275 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 5.3 MiB/s wr, 30 op/s 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:29 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:29 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:29 smithi204 ceph-mon[30933]: 2.3f scrub starts 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:29 smithi204 ceph-mon[30933]: 2.3f scrub ok 2024-09-16T05:18:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:29 smithi204 ceph-mon[30933]: osdmap e559: 12 total, 12 up, 12 in 2024-09-16T05:18:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/576627261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:30.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:29 smithi196 ceph-mon[30802]: pgmap v999: 107 pgs: 107 active+clean; 275 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 5.3 MiB/s wr, 30 op/s 2024-09-16T05:18:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:29 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:18:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:29 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:18:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:29 smithi196 ceph-mon[30802]: 2.3f scrub starts 2024-09-16T05:18:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:29 smithi196 ceph-mon[30802]: 2.3f scrub ok 2024-09-16T05:18:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:29 smithi196 ceph-mon[30802]: osdmap e559: 12 total, 12 up, 12 in 2024-09-16T05:18:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/576627261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:30.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:30.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:30.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:30.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:30.417 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:30.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:30.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:30.540 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:30.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:30.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:30.665 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:30.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:30.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:18:30.762 INFO:tasks.rados.rados.2.smithi204.stdout:2080: done (1 left) 2024-09-16T05:18:30.762 INFO:tasks.rados.rados.2.smithi204.stdout:2081: write oid 28 current snap is 217 2024-09-16T05:18:30.763 INFO:tasks.rados.rados.2.smithi204.stdout:2081: seq_num 747 ranges {554829=413597,1396049=705252} 2024-09-16T05:18:30.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:30.768 INFO:tasks.rados.rados.2.smithi204.stdout:2081: writing smithi20457400-28 from 554829 to 968426 tid 1 2024-09-16T05:18:30.774 INFO:tasks.rados.rados.2.smithi204.stdout:2081: writing smithi20457400-28 from 1396049 to 2101301 tid 2 2024-09-16T05:18:30.775 INFO:tasks.rados.rados.2.smithi204.stdout:2082: rollback oid 25 current snap is 217 2024-09-16T05:18:30.775 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 213 2024-09-16T05:18:30.775 INFO:tasks.rados.rados.2.smithi204.stdout:2079: expect (ObjNum 708 snap 203 seq_num 708) 2024-09-16T05:18:30.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:30.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:30.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:2081: finishing write tid 1 to smithi20457400-28 2024-09-16T05:18:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:2082: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:18:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:2082: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:18:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:2082: finishing rollback tid 2 to smithi20457400-25 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 716 (ObjNum 692 snap 201 seq_num 692) dirty exists 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2081: finishing write tid 2 to smithi20457400-28 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2081: finishing write tid 3 to smithi20457400-28 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2081: finishing write tid 4 to smithi20457400-28 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 708 (ObjNum 747 snap 217 seq_num 747) dirty exists 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2081: left oid 28 (ObjNum 747 snap 217 seq_num 747) 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2079: done (2 left) 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2081: done (1 left) 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2082: done (0 left) 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2083: read oid 41 snap -1 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2083: expect (ObjNum 725 snap 209 seq_num 725) 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2084: copy_from oid 36 from oid 39 current snap is 217 2024-09-16T05:18:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:2085: delete oid 5 current snap is 217 2024-09-16T05:18:30.907 INFO:tasks.rados.rados.2.smithi204.stdout:2085: done (2 left) 2024-09-16T05:18:30.907 INFO:tasks.rados.rados.2.smithi204.stdout:2086: read oid 1 snap -1 2024-09-16T05:18:30.907 INFO:tasks.rados.rados.2.smithi204.stdout:2086: expect deleted 2024-09-16T05:18:30.907 INFO:tasks.rados.rados.2.smithi204.stdout:2087: delete oid 33 current snap is 217 2024-09-16T05:18:30.911 INFO:tasks.rados.rados.2.smithi204.stdout:2086: done (3 left) 2024-09-16T05:18:30.911 INFO:tasks.rados.rados.2.smithi204.stdout:2087: done (2 left) 2024-09-16T05:18:30.911 INFO:tasks.rados.rados.2.smithi204.stdout:2088: rollback oid 46 current snap is 217 2024-09-16T05:18:30.911 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 191 2024-09-16T05:18:30.911 INFO:tasks.rados.rados.2.smithi204.stdout:2089: snap_create 2024-09-16T05:18:30.912 INFO:tasks.rados.rados.2.smithi204.stdout:2083: expect (ObjNum 725 snap 209 seq_num 725) 2024-09-16T05:18:30.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:30.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: 1.2 deep-scrub starts 2024-09-16T05:18:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: 1.2 deep-scrub ok 2024-09-16T05:18:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:30.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/576627261' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: osdmap e560: 12 total, 12 up, 12 in 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:18:30.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:30 smithi032 ceph-mon[27388]: osdmap e561: 12 total, 12 up, 12 in 2024-09-16T05:18:31.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:31.017 INFO:tasks.rados.rados.2.smithi204.stdout:2084: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:18:31.017 INFO:tasks.rados.rados.2.smithi204.stdout:2088: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:18:31.017 INFO:tasks.rados.rados.2.smithi204.stdout:2088: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:18:31.017 INFO:tasks.rados.rados.2.smithi204.stdout:2088: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:18:31.017 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 351 (ObjNum 638 snap 187 seq_num 638) dirty exists 2024-09-16T05:18:31.017 INFO:tasks.rados.rados.2.smithi204.stdout:2084: finishing copy_from to smithi20457400-36 2024-09-16T05:18:31.017 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 712 (ObjNum 668 snap 194 seq_num 668) dirty exists 2024-09-16T05:18:31.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:31.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:31.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:31.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:31.163 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:31.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: 1.2 deep-scrub starts 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: 1.2 deep-scrub ok 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/576627261' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: osdmap e560: 12 total, 12 up, 12 in 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:18:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:30 smithi204 ceph-mon[30933]: osdmap e561: 12 total, 12 up, 12 in 2024-09-16T05:18:31.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: 1.2 deep-scrub starts 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: 1.2 deep-scrub ok 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/576627261' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: osdmap e560: 12 total, 12 up, 12 in 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:18:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:18:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:18:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:30 smithi196 ceph-mon[30802]: osdmap e561: 12 total, 12 up, 12 in 2024-09-16T05:18:31.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:31.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:31.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:31.378 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:31.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:31.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:31.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:31.512 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:31.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:31.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:31.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:31.662 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:31.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:31.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:31.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:31.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:31.809 INFO:tasks.rados.rados.2.smithi204.stdout:2083: done (3 left) 2024-09-16T05:18:31.809 INFO:tasks.rados.rados.2.smithi204.stdout:2084: done (2 left) 2024-09-16T05:18:31.809 INFO:tasks.rados.rados.2.smithi204.stdout:2088: done (1 left) 2024-09-16T05:18:31.809 INFO:tasks.rados.rados.2.smithi204.stdout:2089: done (0 left) 2024-09-16T05:18:31.809 INFO:tasks.rados.rados.2.smithi204.stdout:2090: snap_create 2024-09-16T05:18:31.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:31.911 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:18:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:18:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:18:31.911 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2318 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 232.983696595, 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98387304600000003, 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:18:31.912 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2318 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924656+0000", 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.341e-06 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924663+0000", 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6050000000000003e-06 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924673+0000", 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8830000000000001e-06 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924680+0000", 2024-09-16T05:18:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2230000000000001e-06 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924730+0000", 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0256999999999997e-05 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924737+0000", 2024-09-16T05:18:31.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1550000000000001e-06 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.908327+0000", 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98359007499999995 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.908528+0000", 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000200507 2024-09-16T05:18:31.917 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2319 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 232.983655876, 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98447908799999995, 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.918 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2319 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5900000000000003e-07 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924698+0000", 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9309999999999998e-06 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924703+0000", 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9609999999999996e-06 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924707+0000", 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8990000000000003e-06 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924769+0000", 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1797999999999993e-05 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924776+0000", 2024-09-16T05:18:31.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.99e-06 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909060+0000", 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98428400299999996 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909175+0000", 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115147 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.923 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2320 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~0,checksum] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 232.983632486, 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98482718000000002, 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:18:31.924 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2320 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4200000000000002e-07 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924722+0000", 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4380000000000002e-06 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924726+0000", 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0709999999999996e-06 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924731+0000", 2024-09-16T05:18:31.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0069999999999997e-06 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924792+0000", 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1561000000000002e-05 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924796+0000", 2024-09-16T05:18:31.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0400000000000003e-06 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909278+0000", 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98448187499999995 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909546+0000", 2024-09-16T05:18:31.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026784600000000002 2024-09-16T05:18:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2898 2.2b 2:d7c71ae7:::smithi20457400-5:head [copy-from ver 322] snapc 7c=[7c,7b,7a,76,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:18:31.930 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 190.20165575799999, 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026725529000000001, 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2898 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.931 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706697+0000", 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4789999999999999e-06 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706705+0000", 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0490000000000004e-06 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706719+0000", 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4231e-05 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706728+0000", 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9339999999999995e-06 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706819+0000", 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0402000000000006e-05 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706891+0000", 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2605e-05 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.724332+0000", 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017440699 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.728315+0000", 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003982546 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733300+0000", 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049854369999999997 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733368+0000", 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8133000000000006e-05 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733421+0000", 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3013999999999998e-05 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2899 2.2b 2:d7c71ae7:::smithi20457400-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 190.201606191, 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020779419, 2024-09-16T05:18:31.938 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2899 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706746+0000", 2024-09-16T05:18:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1600000000000002e-07 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706748+0000", 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3140000000000002e-06 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706754+0000", 2024-09-16T05:18:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2609999999999999e-06 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706768+0000", 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3788000000000001e-05 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.707033+0000", 2024-09-16T05:18:31.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026479299999999998 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.707061+0000", 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8050000000000001e-05 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727362+0000", 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020301534 2024-09-16T05:18:31.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727439+0000", 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6749000000000007e-05 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727525+0000", 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5513999999999995e-05 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3108 2.2b 2:d7c71ae7:::smithi20457400-5:head [write 595080~541681] snapc 84=[84,83,82,81,80,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:37.717974+0000", 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 174.190377721, 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051422569999999999, 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:31.945 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3108 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.717974+0000", 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.717974+0000", 2024-09-16T05:18:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:31.949 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.717977+0000", 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.083e-06 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718620+0000", 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00064307999999999998 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718642+0000", 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2007000000000001e-05 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718652+0000", 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6930000000000002e-06 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718738+0000", 2024-09-16T05:18:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6736000000000004e-05 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.718746+0000", 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1429999999999998e-06 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.719931+0000", 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001184147 2024-09-16T05:18:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.720032+0000", 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010173199999999999 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.720228+0000", 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019584800000000001 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.721765+0000", 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015371919999999999 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.723044+0000", 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001278373 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:31.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.723084+0000", 2024-09-16T05:18:31.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0098999999999997e-05 2024-09-16T05:18:31.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:37.723116+0000", 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2123999999999999e-05 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3152 2.3 e397/329 2:c018a787:::smithi20457400-18:head v 397'99, mlcod=375'96)", 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 165.18546376800001, 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066570259999999999, 2024-09-16T05:18:31.956 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:18:31.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3300000000000002e-07 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726366+0000", 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003478259 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726395+0000", 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8347000000000002e-05 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726404+0000", 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0790000000000007e-06 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726446+0000", 2024-09-16T05:18:31.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2092999999999998e-05 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726464+0000", 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8014000000000001e-05 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.729500+0000", 2024-09-16T05:18:31.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030358519999999999 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.729545+0000", 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4848999999999997e-05 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3514 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~2084167,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:18:31.962 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:24.986029+0000", 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 126.922322425, 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035461199999999998, 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3514 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986029+0000", 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986029+0000", 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986029+0000", 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.65e-07 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986032+0000", 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.773e-06 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986038+0000", 2024-09-16T05:18:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8200000000000002e-06 2024-09-16T05:18:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986041+0000", 2024-09-16T05:18:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1669999999999999e-06 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986193+0000", 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015226700000000001 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986199+0000", 2024-09-16T05:18:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9170000000000003e-06 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.988773+0000", 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025732379999999998 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.988861+0000", 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8405000000000005e-05 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.989575+0000", 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071416800000000005 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3515 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~2084167,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:18:31.969 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:24.986071+0000", 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 126.92227996299999, 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042648299999999998, 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3515 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.970 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986071+0000", 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986071+0000", 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986072+0000", 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1199999999999999e-07 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986084+0000", 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2092999999999999e-05 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986091+0000", 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6460000000000004e-06 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986094+0000", 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9149999999999998e-06 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986220+0000", 2024-09-16T05:18:31.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000125108 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.986223+0000", 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2739999999999999e-06 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.989624+0000", 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034006869999999999 2024-09-16T05:18:31.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.989693+0000", 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9447000000000001e-05 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:24.990336+0000", 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00064334799999999997 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3573 2.3 e436/329 2:c018a787:::smithi20457400-18:head v 436'117, mlcod=436'115)", 2024-09-16T05:18:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:18:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 126.041881162, 2024-09-16T05:18:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021202584, 2024-09-16T05:18:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:18:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866471+0000", 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.119e-06 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886258+0000", 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019786313 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886318+0000", 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0550000000000001e-05 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886330+0000", 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2126999999999999e-05 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886419+0000", 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8892000000000002e-05 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886447+0000", 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7959000000000001e-05 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.887619+0000", 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001171526 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.887673+0000", 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4098000000000003e-05 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:3837 2.4d 2:b21a57e7:::smithi20457400-25:head [write 759815~471768,stat] snapc a4=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:48.840408+0000", 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 103.06794319799999, 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004951362, 2024-09-16T05:18:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 3837 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.840408+0000", 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.840408+0000", 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.840412+0000", 2024-09-16T05:18:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0409999999999999e-06 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.840950+0000", 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053732900000000002 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.840979+0000", 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9002000000000001e-05 2024-09-16T05:18:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.840992+0000", 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3508e-05 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.841083+0000", 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0573000000000002e-05 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.841089+0000", 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0599999999999996e-06 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.842814+0000", 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017254010000000001 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.842924+0000", 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110182 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.843093+0000", 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016884499999999999 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.844533+0000", 2024-09-16T05:18:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001439886 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.845199+0000", 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066572199999999999 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.845303+0000", 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010383900000000001 2024-09-16T05:18:31.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:48.845360+0000", 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6974e-05 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:4202 2.3a e489/330 2:5f916a06:::smithi20457400-34:head v 489'522, mlcod=452'518)", 2024-09-16T05:18:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:18.822538+0000", 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 73.085813419000004, 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068665649999999998, 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.822538+0000", 2024-09-16T05:18:31.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.822538+0000", 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.822538+0000", 2024-09-16T05:18:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7100000000000001e-07 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826075+0000", 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035369889999999999 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826108+0000", 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2583999999999998e-05 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826118+0000", 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0288999999999999e-05 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826171+0000", 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2947e-05 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826190+0000", 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8858000000000001e-05 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.829366+0000", 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031762790000000002 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:31.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.829405+0000", 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8448000000000002e-05 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:4460 2.3 e503/329 2:c018a787:::smithi20457400-18:head v 503'143, mlcod=495'141)", 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:33.073315+0000", 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.835036826, 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053949599999999999, 2024-09-16T05:18:31.996 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.073315+0000", 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:31.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.073315+0000", 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.073315+0000", 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1799999999999999e-07 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076711+0000", 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033964170000000001 2024-09-16T05:18:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076763+0000", 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1619000000000002e-05 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076773+0000", 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.047e-05 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:31.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076848+0000", 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4480000000000005e-05 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076880+0000", 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2002000000000003e-05 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.078645+0000", 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001764821 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.078710+0000", 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4932999999999996e-05 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4484 2.4d 2:b21a57e7:::smithi20457400-25:head [write 790094~483510] snapc bc=[bc,bb,ba,b8] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:33.108316+0000", 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 58.800035485999999, 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029970370000000001, 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4484 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.108316+0000", 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.108316+0000", 2024-09-16T05:18:32.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.108317+0000", 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3909999999999999e-06 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.108881+0000", 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000563267 2024-09-16T05:18:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.108909+0000", 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8656000000000001e-05 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.108917+0000", 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8690000000000004e-06 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.108992+0000", 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5222000000000006e-05 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.109066+0000", 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3646000000000006e-05 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:18:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.109194+0000", 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000127978 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.110417+0000", 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012225249999999999 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.111212+0000", 2024-09-16T05:18:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00079523599999999995 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.111256+0000", 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3921000000000002e-05 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.111313+0000", 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7326e-05 2024-09-16T05:18:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4599 2.2b 2:d7c71ae7:::smithi20457400-5:head [write 574971~627602,stat] snapc be=[be] ondisk+write+known_if_redirected+supports_pool_eio e511)", 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:40.856233+0000", 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 51.052118346999997, 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0028269060000000001, 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4599 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.856233+0000", 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.856233+0000", 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.856235+0000", 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.317e-06 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.856944+0000", 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070899199999999998 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.856957+0000", 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2904e-05 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.856963+0000", 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7169999999999996e-06 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.857015+0000", 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1495000000000001e-05 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:32.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.857076+0000", 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1391999999999998e-05 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.857184+0000", 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107876 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.858379+0000", 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001195036 2024-09-16T05:18:32.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.858989+0000", 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061039899999999997 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.859027+0000", 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8247999999999997e-05 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:40.859060+0000", 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2530000000000002e-05 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4678 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~2974971,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 47.102201852, 2024-09-16T05:18:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26066084699999997, 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4678 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806152+0000", 2024-09-16T05:18:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0360000000000001e-06 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806162+0000", 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0380000000000001e-05 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806180+0000", 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8442999999999999e-05 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806193+0000", 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914999999999999e-05 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806258+0000", 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4830000000000001e-05 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806263+0000", 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6959999999999996e-06 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.064487+0000", 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25822445799999999 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:32.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.064549+0000", 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1278999999999994e-05 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066810+0000", 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022618099999999999 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4679 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~1225823] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 47.102133508999998, 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26074116000000003, 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4679 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:18:32.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806219+0000", 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806221+0000", 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7159999999999999e-06 2024-09-16T05:18:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806228+0000", 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4799999999999998e-06 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806234+0000", 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0939999999999996e-06 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806309+0000", 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5340999999999996e-05 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806312+0000", 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0620000000000001e-06 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.027 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066832+0000", 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26051971200000001 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066880+0000", 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7602000000000003e-05 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066959+0000", 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9597000000000004e-05 2024-09-16T05:18:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4680 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~2974971,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 47.102089786999997, 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26141080300000002, 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4680 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.99e-07 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806267+0000", 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3050000000000003e-06 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806275+0000", 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2930000000000001e-06 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806281+0000", 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2779999999999999e-06 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806346+0000", 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5061999999999995e-05 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806354+0000", 2024-09-16T05:18:32.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0669999999999992e-06 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066986+0000", 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.260631273 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.067038+0000", 2024-09-16T05:18:32.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2138999999999997e-05 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.067672+0000", 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063478700000000005 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4853 2.4d 2:b21a57e7:::smithi20457400-25:head [write 763216~643082] snapc c9=[c9,c8,c7,c5,c4,c3,c2,c1] ondisk+write+known_if_redirected+supports_pool_eio e526)", 2024-09-16T05:18:32.035 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:55.762104+0000", 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.146247168000002, 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003700643, 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4853 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.036 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.762104+0000", 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.762104+0000", 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.762106+0000", 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.578e-06 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.762891+0000", 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078503399999999999 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:32.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.762909+0000", 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7898999999999999e-05 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.762915+0000", 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1990000000000003e-06 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.762962+0000", 2024-09-16T05:18:32.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7460000000000003e-05 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.763029+0000", 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6977000000000006e-05 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.763141+0000", 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011121200000000001 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.764830+0000", 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016890379999999999 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.765731+0000", 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090176399999999999 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.765774+0000", 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2747000000000003e-05 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.765805+0000", 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0735000000000003e-05 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4854 2.4d 2:b21a57e7:::smithi20457400-25:head [write 1843729~687899] snapc c9=[c9,c8,c7,c5,c4,c3,c2,c1] ondisk+write+known_if_redirected+supports_pool_eio e526)", 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:55.769034+0000", 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 36.139317210000002, 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003448231, 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:32.043 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4854 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.769034+0000", 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.769034+0000", 2024-09-16T05:18:32.044 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.769035+0000", 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1260000000000001e-06 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.770435+0000", 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001399887 2024-09-16T05:18:32.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.770451+0000", 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5716e-05 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.770459+0000", 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5719999999999997e-06 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.770525+0000", 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6240000000000003e-05 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.770607+0000", 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2303000000000001e-05 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.770719+0000", 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011201000000000001 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.771775+0000", 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010555180000000001 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.772392+0000", 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00061770300000000005 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.772432+0000", 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9205000000000002e-05 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:55.772482+0000", 2024-09-16T05:18:32.049 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0951000000000001e-05 2024-09-16T05:18:32.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.050 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:32.050 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:32.050 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:18:32.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:32.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:32.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:32.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:32.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:32.302 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:32.303 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:32.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:32.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:32.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:32.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:32.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:32.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:32.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:32.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:32.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:32.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:32.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:32.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:32.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:32.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:32.814 INFO:tasks.rados.rados.2.smithi204.stdout:2090: done (0 left) 2024-09-16T05:18:32.814 INFO:tasks.rados.rados.2.smithi204.stdout:2091: copy_from oid 43 from oid 22 current snap is 219 2024-09-16T05:18:32.814 INFO:tasks.rados.rados.2.smithi204.stdout:2092: copy_from oid 14 from oid 34 current snap is 219 2024-09-16T05:18:32.814 INFO:tasks.rados.rados.2.smithi204.stdout:2093: copy_from oid 37 from oid 35 current snap is 219 2024-09-16T05:18:32.814 INFO:tasks.rados.rados.2.smithi204.stdout:2094: write oid 20 current snap is 219 2024-09-16T05:18:32.814 INFO:tasks.rados.rados.2.smithi204.stdout:2094: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-16T05:18:32.821 INFO:tasks.rados.rados.2.smithi204.stdout:2094: writing smithi20457400-20 from 605250 to 1244594 tid 1 2024-09-16T05:18:32.828 INFO:tasks.rados.rados.2.smithi204.stdout:2094: writing smithi20457400-20 from 1855142 to 2307721 tid 2 2024-09-16T05:18:32.828 INFO:tasks.rados.rados.2.smithi204.stdout:2095: rollback oid 23 current snap is 219 2024-09-16T05:18:32.828 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 23 to 206 2024-09-16T05:18:32.828 INFO:tasks.rados.rados.2.smithi204.stdout:2096: rollback oid 39 current snap is 219 2024-09-16T05:18:32.828 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 39 to 217 2024-09-16T05:18:32.829 INFO:tasks.rados.rados.2.smithi204.stdout:2097: copy_from oid 19 from oid 5 current snap is 219 2024-09-16T05:18:32.829 INFO:tasks.rados.rados.2.smithi204.stdout:2098: delete oid 7 current snap is 219 2024-09-16T05:18:32.829 INFO:tasks.rados.rados.2.smithi204.stdout:2094: finishing write tid 1 to smithi20457400-20 2024-09-16T05:18:32.829 INFO:tasks.rados.rados.2.smithi204.stdout:2092: finishing copy_from racing read to smithi20457400-14 2024-09-16T05:18:32.830 INFO:tasks.rados.rados.2.smithi204.stdout:2093: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:18:32.831 INFO:tasks.rados.rados.2.smithi204.stdout:2098: done (7 left) 2024-09-16T05:18:32.831 INFO:tasks.rados.rados.2.smithi204.stdout:2099: snap_remove snap 191 2024-09-16T05:18:32.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:18:32.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:32.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:33.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:33.073 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:33.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:32 smithi032 ceph-mon[27388]: osdmap e562: 12 total, 12 up, 12 in 2024-09-16T05:18:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:32 smithi204 ceph-mon[30933]: osdmap e562: 12 total, 12 up, 12 in 2024-09-16T05:18:33.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:33.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:33.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:33.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:32 smithi196 ceph-mon[30802]: osdmap e562: 12 total, 12 up, 12 in 2024-09-16T05:18:33.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:18:33.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:33.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:33.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:33.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:33.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:33.450 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:33.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:33.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:33.574 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:33.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:33.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:33.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:33.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:33.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:33.816 INFO:tasks.rados.rados.2.smithi204.stdout:2099: done (7 left) 2024-09-16T05:18:33.816 INFO:tasks.rados.rados.2.smithi204.stdout:2100: write (excl) oid 17 current snap is 219 2024-09-16T05:18:33.816 INFO:tasks.rados.rados.2.smithi204.stdout:2100: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-16T05:18:33.824 INFO:tasks.rados.rados.2.smithi204.stdout:2100: writing smithi20457400-17 from 638864 to 1428706 tid 1 2024-09-16T05:18:33.831 INFO:tasks.rados.rados.2.smithi204.stdout:2100: writing smithi20457400-17 from 1972106 to 2428354 tid 2 2024-09-16T05:18:33.831 INFO:tasks.rados.rados.2.smithi204.stdout:2101: read oid 1 snap -1 2024-09-16T05:18:33.831 INFO:tasks.rados.rados.2.smithi204.stdout:2101: expect deleted 2024-09-16T05:18:33.831 INFO:tasks.rados.rados.2.smithi204.stdout:2102: write oid 10 current snap is 219 2024-09-16T05:18:33.831 INFO:tasks.rados.rados.2.smithi204.stdout:2102: seq_num 753 ranges {655671=581444,2030588=625083} 2024-09-16T05:18:33.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:33.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:33.837 INFO:tasks.rados.rados.2.smithi204.stdout:2102: writing smithi20457400-10 from 655671 to 1237115 tid 1 2024-09-16T05:18:33.846 INFO:tasks.rados.rados.2.smithi204.stdout:2102: writing smithi20457400-10 from 2030588 to 2655671 tid 2 2024-09-16T05:18:33.846 INFO:tasks.rados.rados.2.smithi204.stdout:2103: snap_create 2024-09-16T05:18:33.846 INFO:tasks.rados.rados.2.smithi204.stdout:2095: finishing rollback tid 0 to smithi20457400-23 2024-09-16T05:18:33.846 INFO:tasks.rados.rados.2.smithi204.stdout:2095: finishing rollback tid 1 to smithi20457400-23 2024-09-16T05:18:33.846 INFO:tasks.rados.rados.2.smithi204.stdout:2095: finishing rollback tid 2 to smithi20457400-23 2024-09-16T05:18:33.846 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 703 (ObjNum 680 snap 196 seq_num 680) dirty exists 2024-09-16T05:18:33.846 INFO:tasks.rados.rados.2.smithi204.stdout:2092: finishing copy_from to smithi20457400-14 2024-09-16T05:18:33.846 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 699 (ObjNum 710 snap 206 seq_num 710) dirty exists 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2093: finishing copy_from to smithi20457400-37 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 833 (ObjNum 694 snap 201 seq_num 694) dirty exists 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2097: finishing copy_from to smithi20457400-19 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2097: got expected ENOENT (src dne) 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2091: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2097: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2094: finishing write tid 2 to smithi20457400-20 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2094: finishing write tid 3 to smithi20457400-20 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2094: finishing write tid 4 to smithi20457400-20 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 797 (ObjNum 750 snap 219 seq_num 750) dirty exists 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2094: left oid 20 (ObjNum 750 snap 219 seq_num 750) 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2096: finishing rollback tid 0 to smithi20457400-39 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2096: finishing rollback tid 1 to smithi20457400-39 2024-09-16T05:18:33.847 INFO:tasks.rados.rados.2.smithi204.stdout:2091: finishing copy_from to smithi20457400-43 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 396 (ObjNum 592 snap 180 seq_num 592) dirty exists 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:2096: finishing rollback tid 2 to smithi20457400-39 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 801 (ObjNum 668 snap 194 seq_num 668) dirty exists 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:2100: finishing write tid 1 to smithi20457400-17 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:2100: finishing write tid 2 to smithi20457400-17 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:2100: finishing write tid 3 to smithi20457400-17 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:2100: finishing write tid 4 to smithi20457400-17 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 711 (ObjNum 752 snap 219 seq_num 752) dirty exists 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:2100: left oid 17 (ObjNum 752 snap 219 seq_num 752) 2024-09-16T05:18:33.848 INFO:tasks.rados.rados.2.smithi204.stdout:2102: finishing write tid 1 to smithi20457400-10 2024-09-16T05:18:33.849 INFO:tasks.rados.rados.2.smithi204.stdout:2102: finishing write tid 2 to smithi20457400-10 2024-09-16T05:18:33.849 INFO:tasks.rados.rados.2.smithi204.stdout:2102: finishing write tid 3 to smithi20457400-10 2024-09-16T05:18:33.849 INFO:tasks.rados.rados.2.smithi204.stdout:2102: finishing write tid 4 to smithi20457400-10 2024-09-16T05:18:33.849 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 748 (ObjNum 753 snap 219 seq_num 753) dirty exists 2024-09-16T05:18:33.849 INFO:tasks.rados.rados.2.smithi204.stdout:2102: left oid 10 (ObjNum 753 snap 219 seq_num 753) 2024-09-16T05:18:33.911 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:33 smithi032 ceph-mon[27388]: pgmap v1005: 107 pgs: 107 active+clean; 274 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 525 KiB/s wr, 6 op/s 2024-09-16T05:18:33.911 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:33 smithi032 ceph-mon[27388]: osdmap e563: 12 total, 12 up, 12 in 2024-09-16T05:18:33.911 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2858603523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:33.919 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:18:33.919 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:18:33.920 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:18:33.920 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:18:33.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:33.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:33.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:34.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:34.084 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:34.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:33 smithi204 ceph-mon[30933]: pgmap v1005: 107 pgs: 107 active+clean; 274 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 525 KiB/s wr, 6 op/s 2024-09-16T05:18:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:33 smithi204 ceph-mon[30933]: osdmap e563: 12 total, 12 up, 12 in 2024-09-16T05:18:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2858603523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:34.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:34.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:34.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:33 smithi196 ceph-mon[30802]: pgmap v1005: 107 pgs: 107 active+clean; 274 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 525 KiB/s wr, 6 op/s 2024-09-16T05:18:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:33 smithi196 ceph-mon[30802]: osdmap e563: 12 total, 12 up, 12 in 2024-09-16T05:18:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2858603523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:34.256 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:34.256 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:34.278 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:34.279 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:34.299 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:34.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:34.320 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:34.320 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:34.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:34.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:34.341 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:34.342 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:34.364 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:34.364 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:34.385 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:34.385 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:34.407 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:34.407 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:34.430 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:34.430 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:34.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:34.452 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:34.452 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:34.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:34.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:34.474 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:34.474 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:34.497 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:34.497 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:34.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:34.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:34.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:34.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:34.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:34.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:34.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:18:34.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:34.824 INFO:tasks.rados.rados.2.smithi204.stdout:2091: done (10 left) 2024-09-16T05:18:34.824 INFO:tasks.rados.rados.2.smithi204.stdout:2092: done (9 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2093: done (8 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2094: done (7 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2095: done (6 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2096: done (5 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2097: done (4 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2100: done (3 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2101: done (2 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2102: done (1 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2103: done (0 left) 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2104: rollback oid 14 current snap is 220 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 216 2024-09-16T05:18:34.825 INFO:tasks.rados.rados.2.smithi204.stdout:2105: write (excl) oid 5 current snap is 220 2024-09-16T05:18:34.826 INFO:tasks.rados.rados.2.smithi204.stdout:2105: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-16T05:18:34.831 INFO:tasks.rados.rados.2.smithi204.stdout:2105: writing smithi20457400-5 from 672478 to 1329171 tid 1 2024-09-16T05:18:34.838 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:34.839 INFO:tasks.rados.rados.2.smithi204.stdout:2105: writing smithi20457400-5 from 2089070 to 2548987 tid 2 2024-09-16T05:18:34.839 INFO:tasks.rados.rados.2.smithi204.stdout:2106: write oid 35 current snap is 220 2024-09-16T05:18:34.839 INFO:tasks.rados.rados.2.smithi204.stdout:2106: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-16T05:18:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:34.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:34.845 INFO:tasks.rados.rados.2.smithi204.stdout:2106: writing smithi20457400-35 from 689285 to 1421227 tid 1 2024-09-16T05:18:34.848 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:18:34.848 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:18:34.848 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T05:18:34.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:18:34.853 INFO:tasks.rados.rados.2.smithi204.stdout:2106: writing smithi20457400-35 from 2147552 to 2689285 tid 2 2024-09-16T05:18:34.853 INFO:tasks.rados.rados.2.smithi204.stdout:2107: write (excl) oid 3 current snap is 220 2024-09-16T05:18:34.853 INFO:tasks.rados.rados.2.smithi204.stdout:2107: seq_num 756 ranges {706092=407191,1806034=463586} 2024-09-16T05:18:34.859 INFO:tasks.rados.rados.2.smithi204.stdout:2107: writing smithi20457400-3 from 706092 to 1113283 tid 1 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2107: writing smithi20457400-3 from 1806034 to 2269620 tid 2 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2108: delete oid 49 current snap is 220 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2104: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2104: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2104: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 983 (ObjNum 728 snap 210 seq_num 728) dirty exists 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2105: finishing write tid 1 to smithi20457400-5 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2105: finishing write tid 2 to smithi20457400-5 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2105: finishing write tid 3 to smithi20457400-5 2024-09-16T05:18:34.866 INFO:tasks.rados.rados.2.smithi204.stdout:2105: finishing write tid 4 to smithi20457400-5 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 347 (ObjNum 754 snap 220 seq_num 754) dirty exists 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:2105: left oid 5 (ObjNum 754 snap 220 seq_num 754) 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:2106: finishing write tid 1 to smithi20457400-35 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:2106: finishing write tid 2 to smithi20457400-35 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:2106: finishing write tid 3 to smithi20457400-35 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:2106: finishing write tid 4 to smithi20457400-35 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 836 (ObjNum 755 snap 220 seq_num 755) dirty exists 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:2106: left oid 35 (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:2107: finishing write tid 1 to smithi20457400-3 2024-09-16T05:18:34.867 INFO:tasks.rados.rados.2.smithi204.stdout:2104: done (4 left) 2024-09-16T05:18:34.868 INFO:tasks.rados.rados.2.smithi204.stdout:2105: done (3 left) 2024-09-16T05:18:34.869 INFO:tasks.rados.rados.2.smithi204.stdout:2106: done (2 left) 2024-09-16T05:18:34.869 INFO:tasks.rados.rados.2.smithi204.stdout:2108: done (1 left) 2024-09-16T05:18:34.869 INFO:tasks.rados.rados.2.smithi204.stdout:2109: snap_create 2024-09-16T05:18:34.869 INFO:tasks.rados.rados.2.smithi204.stdout:2107: finishing write tid 2 to smithi20457400-3 2024-09-16T05:18:34.870 INFO:tasks.rados.rados.2.smithi204.stdout:2107: finishing write tid 3 to smithi20457400-3 2024-09-16T05:18:34.870 INFO:tasks.rados.rados.2.smithi204.stdout:2107: finishing write tid 4 to smithi20457400-3 2024-09-16T05:18:34.870 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 512 (ObjNum 756 snap 220 seq_num 756) dirty exists 2024-09-16T05:18:34.870 INFO:tasks.rados.rados.2.smithi204.stdout:2107: left oid 3 (ObjNum 756 snap 220 seq_num 756) 2024-09-16T05:18:34.940 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:34.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:34.965 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:34.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:35.049 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:34 smithi032 ceph-mon[27388]: osdmap e564: 12 total, 12 up, 12 in 2024-09-16T05:18:35.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:35.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:35.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:35.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:34 smithi204 ceph-mon[30933]: osdmap e564: 12 total, 12 up, 12 in 2024-09-16T05:18:35.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:35.218 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:35.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:35.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:34 smithi196 ceph-mon[30802]: osdmap e564: 12 total, 12 up, 12 in 2024-09-16T05:18:35.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:35.346 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:35.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:35.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:35.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:35.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:35.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:35.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:35.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:35.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:35.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:35.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:35.762 INFO:tasks.rados.rados.2.smithi204.stdout:2107: done (1 left) 2024-09-16T05:18:35.762 INFO:tasks.rados.rados.2.smithi204.stdout:2109: done (0 left) 2024-09-16T05:18:35.762 INFO:tasks.rados.rados.2.smithi204.stdout:2110: copy_from oid 42 from oid 33 current snap is 221 2024-09-16T05:18:35.762 INFO:tasks.rados.rados.2.smithi204.stdout:2111: write (excl) oid 18 current snap is 221 2024-09-16T05:18:35.762 INFO:tasks.rados.rados.2.smithi204.stdout:2111: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-16T05:18:35.769 INFO:tasks.rados.rados.2.smithi204.stdout:2111: writing smithi20457400-18 from 739706 to 1297395 tid 1 2024-09-16T05:18:35.775 INFO:tasks.rados.rados.2.smithi204.stdout:2111: writing smithi20457400-18 from 1806645 to 2273900 tid 2 2024-09-16T05:18:35.775 INFO:tasks.rados.rados.2.smithi204.stdout:2112: write (excl) oid 36 current snap is 221 2024-09-16T05:18:35.775 INFO:tasks.rados.rados.2.smithi204.stdout:2110: finishing copy_from to smithi20457400-42 2024-09-16T05:18:35.775 INFO:tasks.rados.rados.2.smithi204.stdout:2110: got expected ENOENT (src dne) 2024-09-16T05:18:35.775 INFO:tasks.rados.rados.2.smithi204.stdout:2110: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:18:35.775 INFO:tasks.rados.rados.2.smithi204.stdout:2112: seq_num 759 ranges {756513=632938,1865127=727266} 2024-09-16T05:18:35.785 INFO:tasks.rados.rados.2.smithi204.stdout:2112: writing smithi20457400-36 from 756513 to 1389451 tid 1 2024-09-16T05:18:35.789 INFO:tasks.rados.rados.2.smithi204.stdout:2112: writing smithi20457400-36 from 1865127 to 2592393 tid 2 2024-09-16T05:18:35.789 INFO:tasks.rados.rados.2.smithi204.stdout:2110: done (2 left) 2024-09-16T05:18:35.789 INFO:tasks.rados.rados.2.smithi204.stdout:2113: delete oid 24 current snap is 221 2024-09-16T05:18:35.789 INFO:tasks.rados.rados.2.smithi204.stdout:2111: finishing write tid 1 to smithi20457400-18 2024-09-16T05:18:35.789 INFO:tasks.rados.rados.2.smithi204.stdout:2111: finishing write tid 2 to smithi20457400-18 2024-09-16T05:18:35.790 INFO:tasks.rados.rados.2.smithi204.stdout:2111: finishing write tid 3 to smithi20457400-18 2024-09-16T05:18:35.790 INFO:tasks.rados.rados.2.smithi204.stdout:2111: finishing write tid 4 to smithi20457400-18 2024-09-16T05:18:35.790 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 531 (ObjNum 758 snap 221 seq_num 758) dirty exists 2024-09-16T05:18:35.790 INFO:tasks.rados.rados.2.smithi204.stdout:2111: left oid 18 (ObjNum 758 snap 221 seq_num 758) 2024-09-16T05:18:35.790 INFO:tasks.rados.rados.2.smithi204.stdout:2112: finishing write tid 1 to smithi20457400-36 2024-09-16T05:18:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:2111: done (2 left) 2024-09-16T05:18:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:2113: done (1 left) 2024-09-16T05:18:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:2114: rollback oid 16 current snap is 221 2024-09-16T05:18:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 16 to 206 2024-09-16T05:18:35.792 INFO:tasks.rados.rados.2.smithi204.stdout:2115: rollback oid 34 current snap is 221 2024-09-16T05:18:35.792 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 205 2024-09-16T05:18:35.792 INFO:tasks.rados.rados.2.smithi204.stdout:2116: snap_remove snap 220 2024-09-16T05:18:35.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:35.852 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:35.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:35 smithi032 ceph-mon[27388]: pgmap v1008: 107 pgs: 107 active+clean; 276 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-16T05:18:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:35 smithi032 ceph-mon[27388]: osdmap e565: 12 total, 12 up, 12 in 2024-09-16T05:18:35.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:35 smithi032 ceph-mon[27388]: osdmap e566: 12 total, 12 up, 12 in 2024-09-16T05:18:35.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:35.978 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:35.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:36.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:36.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:36.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:35 smithi204 ceph-mon[30933]: pgmap v1008: 107 pgs: 107 active+clean; 276 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-16T05:18:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:35 smithi204 ceph-mon[30933]: osdmap e565: 12 total, 12 up, 12 in 2024-09-16T05:18:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:35 smithi204 ceph-mon[30933]: osdmap e566: 12 total, 12 up, 12 in 2024-09-16T05:18:36.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:36.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:36.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:35 smithi196 ceph-mon[30802]: pgmap v1008: 107 pgs: 107 active+clean; 276 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-09-16T05:18:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:35 smithi196 ceph-mon[30802]: osdmap e565: 12 total, 12 up, 12 in 2024-09-16T05:18:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:35 smithi196 ceph-mon[30802]: osdmap e566: 12 total, 12 up, 12 in 2024-09-16T05:18:36.300 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:36.301 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:36.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:36.359 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:36.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:36.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:36.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:36.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:36.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:36.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:36.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:36.736 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:36.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:36.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:18:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:2116: done (3 left) 2024-09-16T05:18:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:2117: rollback oid 46 current snap is 221 2024-09-16T05:18:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 210 2024-09-16T05:18:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:2112: finishing write tid 2 to smithi20457400-36 2024-09-16T05:18:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:2112: finishing write tid 3 to smithi20457400-36 2024-09-16T05:18:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:2112: finishing write tid 4 to smithi20457400-36 2024-09-16T05:18:36.769 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 715 (ObjNum 759 snap 221 seq_num 759) dirty exists 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2112: left oid 36 (ObjNum 759 snap 221 seq_num 759) 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2115: finishing rollback tid 0 to smithi20457400-34 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2115: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 701 (ObjNum 305 snap 0 seq_num 2671819837) dirty dne 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2114: finishing rollback tid 0 to smithi20457400-16 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2114: finishing rollback tid 1 to smithi20457400-16 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2112: done (3 left) 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2115: done (2 left) 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2118: read oid 41 snap -1 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2118: expect (ObjNum 725 snap 209 seq_num 725) 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2119: read oid 2 snap 218 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2119: expect (ObjNum 737 snap 215 seq_num 737) 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2120: copy_from oid 40 from oid 11 current snap is 221 2024-09-16T05:18:36.770 INFO:tasks.rados.rados.2.smithi204.stdout:2114: finishing rollback tid 2 to smithi20457400-16 2024-09-16T05:18:36.771 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 722 (ObjNum 596 snap 180 seq_num 596) dirty exists 2024-09-16T05:18:36.771 INFO:tasks.rados.rados.2.smithi204.stdout:2114: done (4 left) 2024-09-16T05:18:36.771 INFO:tasks.rados.rados.2.smithi204.stdout:2121: snap_create 2024-09-16T05:18:36.774 INFO:tasks.rados.rados.2.smithi204.stdout:2117: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:18:36.776 INFO:tasks.rados.rados.2.smithi204.stdout:2117: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:18:36.776 INFO:tasks.rados.rados.2.smithi204.stdout:2117: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:18:36.776 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 354 (ObjNum 638 snap 187 seq_num 638) dirty exists 2024-09-16T05:18:36.782 INFO:tasks.rados.rados.2.smithi204.stdout:2119: expect (ObjNum 737 snap 215 seq_num 737) 2024-09-16T05:18:36.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:36.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:36.863 INFO:tasks.rados.rados.2.smithi204.stdout:2118: expect (ObjNum 725 snap 209 seq_num 725) 2024-09-16T05:18:36.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1277367252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:36.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:36 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:36.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1277367252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:36.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:36 smithi032 ceph-mon[27388]: osdmap e567: 12 total, 12 up, 12 in 2024-09-16T05:18:36.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:36.968 INFO:tasks.rados.rados.2.smithi204.stdout:2120: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:18:36.968 INFO:tasks.rados.rados.2.smithi204.stdout:2120: finishing copy_from to smithi20457400-40 2024-09-16T05:18:36.968 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 764 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:18:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:36.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:36.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:37.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:37.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:37.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:37.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:18:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1277367252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:36 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1277367252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:36 smithi204 ceph-mon[30933]: osdmap e567: 12 total, 12 up, 12 in 2024-09-16T05:18:37.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:37.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:37.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:37.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1277367252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:36 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1277367252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:36 smithi196 ceph-mon[30802]: osdmap e567: 12 total, 12 up, 12 in 2024-09-16T05:18:37.278 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:37.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:18:37.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:37.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:37.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:37.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:37.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:37.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:37.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:37.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:37.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:37.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:37.737 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:37.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:37.769 INFO:tasks.rados.rados.2.smithi204.stdout:2117: done (4 left) 2024-09-16T05:18:37.769 INFO:tasks.rados.rados.2.smithi204.stdout:2118: done (3 left) 2024-09-16T05:18:37.769 INFO:tasks.rados.rados.2.smithi204.stdout:2119: done (2 left) 2024-09-16T05:18:37.769 INFO:tasks.rados.rados.2.smithi204.stdout:2120: done (1 left) 2024-09-16T05:18:37.769 INFO:tasks.rados.rados.2.smithi204.stdout:2121: done (0 left) 2024-09-16T05:18:37.769 INFO:tasks.rados.rados.2.smithi204.stdout:2122: rollback oid 1 current snap is 222 2024-09-16T05:18:37.769 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 1 to 216 2024-09-16T05:18:37.769 INFO:tasks.rados.rados.2.smithi204.stdout:2123: snap_remove snap 206 2024-09-16T05:18:37.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:37.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:37.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:37 smithi032 ceph-mon[27388]: pgmap v1011: 107 pgs: 107 active+clean; 273 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 11 op/s 2024-09-16T05:18:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:37 smithi032 ceph-mon[27388]: osdmap e568: 12 total, 12 up, 12 in 2024-09-16T05:18:37.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:37.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:37.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:38.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:38.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:38.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:37 smithi204 ceph-mon[30933]: pgmap v1011: 107 pgs: 107 active+clean; 273 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 11 op/s 2024-09-16T05:18:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:37 smithi204 ceph-mon[30933]: osdmap e568: 12 total, 12 up, 12 in 2024-09-16T05:18:38.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:37 smithi196 ceph-mon[30802]: pgmap v1011: 107 pgs: 107 active+clean; 273 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 11 op/s 2024-09-16T05:18:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:37 smithi196 ceph-mon[30802]: osdmap e568: 12 total, 12 up, 12 in 2024-09-16T05:18:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:38.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:38.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:38.303 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:38.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:38.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:38.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:38.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:18:38.483 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 568, 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:18:37.763816+0000", 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:14:38.892129+0000", 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:17:58.835972+0000", 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5806080, 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "noscrub", 2024-09-16T05:18:38.484 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 34, 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:18:38.485 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "luminous", 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:18:38.486 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 21, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 21, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 21, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 21, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 21, 2024-09-16T05:18:38.487 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "408", 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "401", 2024-09-16T05:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.2899999618530273, 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.2899999618530273, 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.2899999618530273, 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.2899999618530273, 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:18:38.493 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:18:38.494 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 86, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 86, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 86, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 86, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 86, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "568", 2024-09-16T05:18:38.495 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "435", 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 223, 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 568, 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:18:38.496 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:18:38.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:18:38.498 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 1.809999942779541, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 1.809999942779541, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 1.809999942779541, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 1.809999942779541, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:18:38.500 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 549, 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:18:38.501 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:18:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:18:38.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:18:38.506 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.507 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 529, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:18:38.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.511 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.513 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 21, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 295, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 329, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 531, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 296, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3973436581", 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3973436581", 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3973436581", 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3973436581", 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 531, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:18:38.520 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.521 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.522 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:18:38.524 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.525 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 507, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:18:38.526 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.527 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.530 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.531 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 549, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.532 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.534 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.535 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:18:38.537 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 509, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.538 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:18:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 549, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.545 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.547 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:18:38.548 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 531, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.550 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.551 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.553 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:18:38.554 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:18:38.555 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 507, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.558 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.560 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:18:38.561 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:18:38.561 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:18:38.561 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:38.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:18:38.562 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 533, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.565 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.566 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:18:38.568 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 549, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:18:38.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:18:38.573 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.574 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:49.851556+0000", 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:40.745386+0000", 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.576 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:18:11.116041+0000", 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 295 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:18:22.149353+0000", 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:18:04.924953+0000", 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:48.940007+0000", 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:18:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:27.333650+0000", 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:18:20.464356+0000", 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:32.914425+0000", 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:18:17.684982+0000", 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:18:01.890878+0000", 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:18:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:17:21.738128+0000", 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [ 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: "pgid": "2.5", 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: 9, 2024-09-16T05:18:38.584 INFO:teuthology.orchestra.run.smithi032.stdout: 11 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:18:38.585 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 193, 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 203, 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:38.587 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 222, 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:18:38.588 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:18:38.589 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:18:38.589 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:18:38.589 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:18:38.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:38.589 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:18:38.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:38.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:38.771 INFO:tasks.rados.rados.2.smithi204.stdout:2123: done (1 left) 2024-09-16T05:18:38.771 INFO:tasks.rados.rados.2.smithi204.stdout:2124: snap_remove snap 205 2024-09-16T05:18:38.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:38.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:38.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:38.914 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:38.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:38.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:39.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:39.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:39.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:38 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:38 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1771277501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:18:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:38 smithi196 ceph-mon[30802]: osdmap e569: 12 total, 12 up, 12 in 2024-09-16T05:18:39.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:38 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:38 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1771277501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:18:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:38 smithi032 ceph-mon[27388]: osdmap e569: 12 total, 12 up, 12 in 2024-09-16T05:18:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:38 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:38 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1771277501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:18:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:38 smithi204 ceph-mon[30933]: osdmap e569: 12 total, 12 up, 12 in 2024-09-16T05:18:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:39.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:39.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:39.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:39.311 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:39.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:39.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:39.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:39.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:39.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:39.564 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:39.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:39.579 INFO:tasks.thrashosds.thrasher:j is {'epoch': 568, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:18:37.763816+0000', 'last_up_change': '2024-09-16T05:14:38.892129+0000', 'last_in_change': '2024-09-16T05:17:58.835972+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 34, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '408', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '401', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 1, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 86, 'pg_placement_num': 86, 'pg_placement_num_target': 86, 'pg_num_target': 86, 'pg_num_pending': 86, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '568', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '435', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 223, 'snap_epoch': 568, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.809999942779541, 'score_stable': 1.809999942779541, 'optimal_score': 1, 'raw_score_acting': 1.809999942779541, 'raw_score_stable': 1.809999942779541, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 529, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 21, 'last_clean_end': 295, 'up_from': 329, 'up_thru': 531, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3973436581}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3973436581}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3973436581}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3973436581}]}, 'public_addr': '172.21.15.32:6819/3973436581', 'cluster_addr': '172.21.15.32:6821/3973436581', 'heartbeat_back_addr': '172.21.15.32:6825/3973436581', 'heartbeat_front_addr': '172.21.15.32:6823/3973436581', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 531, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 507, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 509, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 531, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 507, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 533, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:49.851556+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:40.745386+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:18:11.116041+0000', 'dead_epoch': 295}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:18:22.149353+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:18:04.924953+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:48.940007+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:27.333650+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:18:20.464356+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:32.914425+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:18:17.684982+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:18:01.890878+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:17:21.738128+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [9, 11]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 193, 'length': 1}, {'begin': 203, 'length': 1}, {'begin': 222, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:18:39.579 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.5 2024-09-16T05:18:39.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd rm-pg-upmap 2.5 2024-09-16T05:18:39.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:39.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:39.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:39.772 INFO:tasks.rados.rados.2.smithi204.stdout:2124: done (1 left) 2024-09-16T05:18:39.772 INFO:tasks.rados.rados.2.smithi204.stdout:2125: snap_create 2024-09-16T05:18:39.772 INFO:tasks.rados.rados.2.smithi204.stdout:2122: finishing rollback tid 1 to smithi20457400-1 2024-09-16T05:18:39.773 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 535 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:18:39.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:39.817 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:39.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:39.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:39.942 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:39.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:40.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:40.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:18:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:40.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:40.089 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:39 smithi032 ceph-mon[27388]: pgmap v1014: 107 pgs: 107 active+clean; 285 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 16 op/s 2024-09-16T05:18:40.090 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:39 smithi032 ceph-mon[27388]: osdmap e570: 12 total, 12 up, 12 in 2024-09-16T05:18:40.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:39 smithi204 ceph-mon[30933]: pgmap v1014: 107 pgs: 107 active+clean; 285 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 16 op/s 2024-09-16T05:18:40.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:39 smithi204 ceph-mon[30933]: osdmap e570: 12 total, 12 up, 12 in 2024-09-16T05:18:40.093 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:40.093 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:40.151 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:39 smithi196 ceph-mon[30802]: pgmap v1014: 107 pgs: 107 active+clean; 285 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 16 op/s 2024-09-16T05:18:40.151 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:39 smithi196 ceph-mon[30802]: osdmap e570: 12 total, 12 up, 12 in 2024-09-16T05:18:40.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:40.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:40.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:40.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:40.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:40.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:40.325 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:40.325 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:40.346 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:40.346 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:40.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:40.441 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:40.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:40.539 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:40.543 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:40.566 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:40.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:40.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:40.577 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:40.578 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:40.598 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:40.619 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:40.619 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:40.641 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:40.641 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:40.663 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:40.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:40.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:40.687 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:40.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:40.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:40.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:40.708 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:40.709 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:40.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:40.751 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:40.751 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:40.762 INFO:tasks.rados.rados.2.smithi204.stdout:2122: done (1 left) 2024-09-16T05:18:40.762 INFO:tasks.rados.rados.2.smithi204.stdout:2125: done (0 left) 2024-09-16T05:18:40.762 INFO:tasks.rados.rados.2.smithi204.stdout:2126: delete oid 23 current snap is 223 2024-09-16T05:18:40.765 INFO:tasks.rados.rados.2.smithi204.stdout:2126: done (0 left) 2024-09-16T05:18:40.766 INFO:tasks.rados.rados.2.smithi204.stdout:2127: write oid 46 current snap is 223 2024-09-16T05:18:40.766 INFO:tasks.rados.rados.2.smithi204.stdout:2127: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-16T05:18:40.771 INFO:tasks.rados.rados.2.smithi204.stdout:2127: writing smithi20457400-46 from 406934 to 981972 tid 1 2024-09-16T05:18:40.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:40.774 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:40.777 INFO:tasks.rados.rados.2.smithi204.stdout:2127: writing smithi20457400-46 from 1640573 to 2115166 tid 2 2024-09-16T05:18:40.781 INFO:tasks.rados.rados.2.smithi204.stdout:2127: writing smithi20457400-46 from 2642791 to 2806934 tid 3 2024-09-16T05:18:40.782 INFO:tasks.rados.rados.2.smithi204.stdout:2128: copy_from oid 8 from oid 45 current snap is 223 2024-09-16T05:18:40.782 INFO:tasks.rados.rados.2.smithi204.stdout:2127: finishing write tid 1 to smithi20457400-46 2024-09-16T05:18:40.782 INFO:tasks.rados.rados.2.smithi204.stdout:2127: finishing write tid 2 to smithi20457400-46 2024-09-16T05:18:40.782 INFO:tasks.rados.rados.2.smithi204.stdout:2129: write (excl) oid 48 current snap is 223 2024-09-16T05:18:40.782 INFO:tasks.rados.rados.2.smithi204.stdout:2129: seq_num 763 ranges {423741=650287,1699055=618251} 2024-09-16T05:18:40.787 INFO:tasks.rados.rados.2.smithi204.stdout:2129: writing smithi20457400-48 from 423741 to 1074028 tid 1 2024-09-16T05:18:40.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:40.794 INFO:tasks.rados.rados.2.smithi204.stdout:2129: writing smithi20457400-48 from 1699055 to 2317306 tid 2 2024-09-16T05:18:40.794 INFO:tasks.rados.rados.2.smithi204.stdout:2130: write (excl) oid 12 current snap is 223 2024-09-16T05:18:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:2127: finishing write tid 3 to smithi20457400-46 2024-09-16T05:18:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:2127: finishing write tid 4 to smithi20457400-46 2024-09-16T05:18:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:2127: finishing write tid 5 to smithi20457400-46 2024-09-16T05:18:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 358 (ObjNum 762 snap 223 seq_num 762) dirty exists 2024-09-16T05:18:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:2127: left oid 46 (ObjNum 762 snap 223 seq_num 762) 2024-09-16T05:18:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:2129: finishing write tid 1 to smithi20457400-48 2024-09-16T05:18:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:2128: finishing copy_from racing read to smithi20457400-8 2024-09-16T05:18:40.795 INFO:tasks.rados.rados.2.smithi204.stdout:2130: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2024-09-16T05:18:40.800 INFO:tasks.rados.rados.2.smithi204.stdout:2130: writing smithi20457400-12 from 440548 to 1166084 tid 1 2024-09-16T05:18:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2130: writing smithi20457400-12 from 1757537 to 2235799 tid 2 2024-09-16T05:18:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2130: writing smithi20457400-12 from 2813990 to 2840548 tid 3 2024-09-16T05:18:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2127: done (3 left) 2024-09-16T05:18:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2131: delete oid 25 current snap is 223 2024-09-16T05:18:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2128: finishing copy_from to smithi20457400-8 2024-09-16T05:18:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 358 (ObjNum 723 snap 209 seq_num 723) dirty exists 2024-09-16T05:18:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2129: finishing write tid 2 to smithi20457400-48 2024-09-16T05:18:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2130: finishing write tid 1 to smithi20457400-12 2024-09-16T05:18:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2130: finishing write tid 2 to smithi20457400-12 2024-09-16T05:18:40.812 INFO:tasks.rados.rados.2.smithi204.stdout:2130: finishing write tid 3 to smithi20457400-12 2024-09-16T05:18:40.812 INFO:tasks.rados.rados.2.smithi204.stdout:2130: finishing write tid 4 to smithi20457400-12 2024-09-16T05:18:40.812 INFO:tasks.rados.rados.2.smithi204.stdout:2130: finishing write tid 5 to smithi20457400-12 2024-09-16T05:18:40.812 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 758 (ObjNum 764 snap 223 seq_num 764) dirty exists 2024-09-16T05:18:40.812 INFO:tasks.rados.rados.2.smithi204.stdout:2130: left oid 12 (ObjNum 764 snap 223 seq_num 764) 2024-09-16T05:18:40.813 INFO:tasks.rados.rados.2.smithi204.stdout:2128: done (3 left) 2024-09-16T05:18:40.813 INFO:tasks.rados.rados.2.smithi204.stdout:2130: done (2 left) 2024-09-16T05:18:40.813 INFO:tasks.rados.rados.2.smithi204.stdout:2131: done (1 left) 2024-09-16T05:18:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2132: read oid 2 snap -1 2024-09-16T05:18:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2132: expect (ObjNum 737 snap 215 seq_num 737) 2024-09-16T05:18:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2133: copy_from oid 37 from oid 29 current snap is 223 2024-09-16T05:18:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2134: rollback oid 44 current snap is 223 2024-09-16T05:18:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 210 2024-09-16T05:18:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2135: snap_remove snap 217 2024-09-16T05:18:40.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:40.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:40.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3216268254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:40 smithi032 ceph-mon[27388]: osdmap e571: 12 total, 12 up, 12 in 2024-09-16T05:18:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:40.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:40.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:40.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3216268254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:40.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:40 smithi196 ceph-mon[30802]: osdmap e571: 12 total, 12 up, 12 in 2024-09-16T05:18:41.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:41.061 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:41.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:41.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:41.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3216268254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:41.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:40 smithi204 ceph-mon[30933]: osdmap e571: 12 total, 12 up, 12 in 2024-09-16T05:18:41.187 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:41.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:41.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:41.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:41.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:41.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:41.434 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:41.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:41.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:18:41.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:41.558 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:41.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:41.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:41.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:41.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2135: done (4 left) 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2136: read oid 25 snap -1 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2136: expect deleted 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2137: read oid 7 snap -1 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2137: expect deleted 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2138: read oid 45 snap -1 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2138: expect (ObjNum 723 snap 209 seq_num 723) 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2139: read oid 13 snap -1 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2139: expect (ObjNum 727 snap 210 seq_num 727) 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2140: write oid 39 current snap is 223 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2134: finishing rollback tid 0 to smithi20457400-44 2024-09-16T05:18:41.765 INFO:tasks.rados.rados.2.smithi204.stdout:2133: finishing copy_from to smithi20457400-37 2024-09-16T05:18:41.766 INFO:tasks.rados.rados.2.smithi204.stdout:2133: got expected ENOENT (src dne) 2024-09-16T05:18:41.766 INFO:tasks.rados.rados.2.smithi204.stdout:2133: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:18:41.766 INFO:tasks.rados.rados.2.smithi204.stdout:2134: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:18:41.766 INFO:tasks.rados.rados.2.smithi204.stdout:2140: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-09-16T05:18:41.770 INFO:tasks.rados.rados.2.smithi204.stdout:2140: writing smithi20457400-39 from 474162 to 950196 tid 1 2024-09-16T05:18:41.776 INFO:tasks.rados.rados.2.smithi204.stdout:2140: writing smithi20457400-39 from 1358148 to 1840079 tid 2 2024-09-16T05:18:41.782 INFO:tasks.rados.rados.2.smithi204.stdout:2140: writing smithi20457400-39 from 2352483 to 2817786 tid 3 2024-09-16T05:18:41.782 INFO:tasks.rados.rados.2.smithi204.stdout:2133: done (8 left) 2024-09-16T05:18:41.783 INFO:tasks.rados.rados.2.smithi204.stdout:2141: read oid 47 snap -1 2024-09-16T05:18:41.783 INFO:tasks.rados.rados.2.smithi204.stdout:2141: expect (ObjNum 704 snap 202 seq_num 704) 2024-09-16T05:18:41.783 INFO:tasks.rados.rados.2.smithi204.stdout:2142: copy_from oid 21 from oid 1 current snap is 223 2024-09-16T05:18:41.783 INFO:tasks.rados.rados.2.smithi204.stdout:2134: finishing rollback tid 2 to smithi20457400-44 2024-09-16T05:18:41.783 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 719 (ObjNum 692 snap 201 seq_num 692) dirty exists 2024-09-16T05:18:41.783 INFO:tasks.rados.rados.2.smithi204.stdout:2134: done (9 left) 2024-09-16T05:18:41.783 INFO:tasks.rados.rados.2.smithi204.stdout:2143: write oid 3 current snap is 223 2024-09-16T05:18:41.783 INFO:tasks.rados.rados.2.smithi204.stdout:2132: expect (ObjNum 737 snap 215 seq_num 737) 2024-09-16T05:18:41.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:41.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:41.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:41.862 INFO:tasks.rados.rados.2.smithi204.stdout:2129: finishing write tid 3 to smithi20457400-48 2024-09-16T05:18:41.862 INFO:tasks.rados.rados.2.smithi204.stdout:2129: finishing write tid 4 to smithi20457400-48 2024-09-16T05:18:41.863 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 408 (ObjNum 763 snap 223 seq_num 763) dirty exists 2024-09-16T05:18:41.863 INFO:tasks.rados.rados.2.smithi204.stdout:2129: left oid 48 (ObjNum 763 snap 223 seq_num 763) 2024-09-16T05:18:41.863 INFO:tasks.rados.rados.2.smithi204.stdout:2143: seq_num 767 ranges {490969=551283,1816630=625589} 2024-09-16T05:18:41.868 INFO:tasks.rados.rados.2.smithi204.stdout:2143: writing smithi20457400-3 from 490969 to 1042252 tid 1 2024-09-16T05:18:41.875 INFO:tasks.rados.rados.2.smithi204.stdout:2143: writing smithi20457400-3 from 1816630 to 2442219 tid 2 2024-09-16T05:18:41.876 INFO:tasks.rados.rados.2.smithi204.stdout:2129: done (9 left) 2024-09-16T05:18:41.876 INFO:tasks.rados.rados.2.smithi204.stdout:2132: done (8 left) 2024-09-16T05:18:41.876 INFO:tasks.rados.rados.2.smithi204.stdout:2144: delete oid 33 current snap is 223 2024-09-16T05:18:41.876 INFO:tasks.rados.rados.2.smithi204.stdout:2139: expect (ObjNum 727 snap 210 seq_num 727) 2024-09-16T05:18:41.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:41.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:41.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:42.028 INFO:tasks.rados.rados.2.smithi204.stdout:2138: expect (ObjNum 723 snap 209 seq_num 723) 2024-09-16T05:18:42.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:42.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:42.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:42.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:41 smithi032 ceph-mon[27388]: pgmap v1017: 107 pgs: 107 active+clean; 285 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2024-09-16T05:18:42.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:41 smithi032 ceph-mon[27388]: osdmap e572: 12 total, 12 up, 12 in 2024-09-16T05:18:42.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2508090855' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.5"}]: dispatch 2024-09-16T05:18:42.141 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:42.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:42.163 INFO:tasks.rados.rados.2.smithi204.stdout:2140: finishing write tid 1 to smithi20457400-39 2024-09-16T05:18:42.163 INFO:tasks.rados.rados.2.smithi204.stdout:2140: finishing write tid 2 to smithi20457400-39 2024-09-16T05:18:42.163 INFO:tasks.rados.rados.2.smithi204.stdout:2136: done (8 left) 2024-09-16T05:18:42.163 INFO:tasks.rados.rados.2.smithi204.stdout:2137: done (7 left) 2024-09-16T05:18:42.163 INFO:tasks.rados.rados.2.smithi204.stdout:2138: done (6 left) 2024-09-16T05:18:42.163 INFO:tasks.rados.rados.2.smithi204.stdout:2139: done (5 left) 2024-09-16T05:18:42.163 INFO:tasks.rados.rados.2.smithi204.stdout:2144: done (4 left) 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2145: read oid 9 snap -1 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2145: expect (ObjNum 746 snap 215 seq_num 746) 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2146: rollback oid 14 current snap is 223 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 200 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2142: finishing copy_from to smithi20457400-21 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2142: got expected ENOENT (src dne) 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2142: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2140: finishing write tid 3 to smithi20457400-39 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2140: finishing write tid 4 to smithi20457400-39 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2140: finishing write tid 5 to smithi20457400-39 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 822 (ObjNum 766 snap 223 seq_num 766) dirty exists 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2140: left oid 39 (ObjNum 766 snap 223 seq_num 766) 2024-09-16T05:18:42.164 INFO:tasks.rados.rados.2.smithi204.stdout:2141: expect (ObjNum 704 snap 202 seq_num 704) 2024-09-16T05:18:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:41 smithi204 ceph-mon[30933]: pgmap v1017: 107 pgs: 107 active+clean; 285 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2024-09-16T05:18:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:41 smithi204 ceph-mon[30933]: osdmap e572: 12 total, 12 up, 12 in 2024-09-16T05:18:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2508090855' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.5"}]: dispatch 2024-09-16T05:18:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:42.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:42.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:41 smithi196 ceph-mon[30802]: pgmap v1017: 107 pgs: 107 active+clean; 285 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2024-09-16T05:18:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:41 smithi196 ceph-mon[30802]: osdmap e572: 12 total, 12 up, 12 in 2024-09-16T05:18:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2508090855' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.5"}]: dispatch 2024-09-16T05:18:42.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:42.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:42.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:42.408 INFO:tasks.rados.rados.2.smithi204.stdout:2143: finishing write tid 1 to smithi20457400-3 2024-09-16T05:18:42.408 INFO:tasks.rados.rados.2.smithi204.stdout:2143: finishing write tid 2 to smithi20457400-3 2024-09-16T05:18:42.408 INFO:tasks.rados.rados.2.smithi204.stdout:2143: finishing write tid 3 to smithi20457400-3 2024-09-16T05:18:42.408 INFO:tasks.rados.rados.2.smithi204.stdout:2143: finishing write tid 4 to smithi20457400-3 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 522 (ObjNum 767 snap 223 seq_num 767) dirty exists 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2143: left oid 3 (ObjNum 767 snap 223 seq_num 767) 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2146: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2146: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2140: done (5 left) 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2141: done (4 left) 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2142: done (3 left) 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2143: done (2 left) 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2147: delete oid 50 current snap is 223 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2146: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 997 (ObjNum 664 snap 192 seq_num 664) dirty exists 2024-09-16T05:18:42.409 INFO:tasks.rados.rados.2.smithi204.stdout:2145: expect (ObjNum 746 snap 215 seq_num 746) 2024-09-16T05:18:42.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:42.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:42.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:42.532 INFO:tasks.rados.rados.2.smithi204.stdout:2145: done (2 left) 2024-09-16T05:18:42.532 INFO:tasks.rados.rados.2.smithi204.stdout:2146: done (1 left) 2024-09-16T05:18:42.533 INFO:tasks.rados.rados.2.smithi204.stdout:2147: done (0 left) 2024-09-16T05:18:42.533 INFO:tasks.rados.rados.2.smithi204.stdout:2148: rollback oid 49 current snap is 223 2024-09-16T05:18:42.533 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 219 2024-09-16T05:18:42.533 INFO:tasks.rados.rados.2.smithi204.stdout:2149: read oid 11 snap 200 2024-09-16T05:18:42.533 INFO:tasks.rados.rados.2.smithi204.stdout:2149: expect (ObjNum 644 snap 187 seq_num 644) 2024-09-16T05:18:42.533 INFO:tasks.rados.rados.2.smithi204.stdout:2150: write oid 26 current snap is 223 2024-09-16T05:18:42.533 INFO:tasks.rados.rados.2.smithi204.stdout:2150: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2024-09-16T05:18:42.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:42.537 INFO:tasks.rados.rados.2.smithi204.stdout:2150: writing smithi20457400-26 from 541390 to 1034773 tid 1 2024-09-16T05:18:42.543 INFO:tasks.rados.rados.2.smithi204.stdout:2150: writing smithi20457400-26 from 1592076 to 2081345 tid 2 2024-09-16T05:18:42.548 INFO:tasks.rados.rados.2.smithi204.stdout:2150: writing smithi20457400-26 from 2862175 to 2941390 tid 3 2024-09-16T05:18:42.549 INFO:tasks.rados.rados.2.smithi204.stdout:2151: snap_create 2024-09-16T05:18:42.549 INFO:tasks.rados.rados.2.smithi204.stdout:2148: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:18:42.549 INFO:tasks.rados.rados.2.smithi204.stdout:2148: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:18:42.549 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 588 (ObjNum 704 snap 202 seq_num 704) dirty exists 2024-09-16T05:18:42.549 INFO:tasks.rados.rados.2.smithi204.stdout:2149: expect (ObjNum 644 snap 187 seq_num 644) 2024-09-16T05:18:42.556 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:42.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:42.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:42.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:42.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:42.698 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:42.762 INFO:teuthology.orchestra.run.smithi032.stderr:clear 2.5 pg_upmap mapping 2024-09-16T05:18:42.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:42.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:2150: finishing write tid 1 to smithi20457400-26 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:2150: finishing write tid 2 to smithi20457400-26 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:2150: finishing write tid 3 to smithi20457400-26 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:2150: finishing write tid 4 to smithi20457400-26 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:2150: finishing write tid 5 to smithi20457400-26 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1001 (ObjNum 770 snap 223 seq_num 770) dirty exists 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:2150: left oid 26 (ObjNum 770 snap 223 seq_num 770) 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:2148: done (3 left) 2024-09-16T05:18:42.829 INFO:tasks.rados.rados.2.smithi204.stdout:2149: done (2 left) 2024-09-16T05:18:42.830 INFO:tasks.rados.rados.2.smithi204.stdout:2150: done (1 left) 2024-09-16T05:18:42.830 INFO:tasks.rados.rados.2.smithi204.stdout:2151: done (0 left) 2024-09-16T05:18:42.830 INFO:tasks.rados.rados.2.smithi204.stdout:2152: read oid 39 snap -1 2024-09-16T05:18:42.830 INFO:tasks.rados.rados.2.smithi204.stdout:2152: expect (ObjNum 766 snap 223 seq_num 766) 2024-09-16T05:18:42.830 INFO:tasks.rados.rados.2.smithi204.stdout:2153: snap_remove snap 210 2024-09-16T05:18:42.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:42.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:42.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:43.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:43.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:43.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:43.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:43.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:43.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:43.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:43.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:43.318 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:18:43.319 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:18:43.319 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:18:43.319 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:18:43.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:43.428 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:43.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:43.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:43.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:43.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:43.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:43.674 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:43.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:43.769 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:18:43.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:43.777 INFO:tasks.rados.rados.2.smithi204.stdout:2153: done (1 left) 2024-09-16T05:18:43.777 INFO:tasks.rados.rados.2.smithi204.stdout:2154: rollback oid 49 current snap is 224 2024-09-16T05:18:43.777 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 213 2024-09-16T05:18:43.777 INFO:tasks.rados.rados.2.smithi204.stdout:2152: expect (ObjNum 766 snap 223 seq_num 766) 2024-09-16T05:18:43.804 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:43.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:43.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:43.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:43.961 INFO:tasks.rados.rados.2.smithi204.stdout:2154: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:18:43.961 INFO:tasks.rados.rados.2.smithi204.stdout:2154: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:18:43.961 INFO:tasks.rados.rados.2.smithi204.stdout:2152: done (1 left) 2024-09-16T05:18:43.961 INFO:tasks.rados.rados.2.smithi204.stdout:2155: copy_from oid 4 from oid 14 current snap is 224 2024-09-16T05:18:43.961 INFO:tasks.rados.rados.2.smithi204.stdout:2154: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:18:43.961 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 591 (ObjNum 704 snap 202 seq_num 704) dirty exists 2024-09-16T05:18:43.962 INFO:tasks.rados.rados.2.smithi204.stdout:2154: done (1 left) 2024-09-16T05:18:43.962 INFO:tasks.rados.rados.2.smithi204.stdout:2156: write (excl) oid 12 current snap is 224 2024-09-16T05:18:43.962 INFO:tasks.rados.rados.2.smithi204.stdout:2156: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-09-16T05:18:43.967 INFO:tasks.rados.rados.2.smithi204.stdout:2156: writing smithi20457400-12 from 558197 to 1126829 tid 1 2024-09-16T05:18:43.973 INFO:tasks.rados.rados.2.smithi204.stdout:2156: writing smithi20457400-12 from 1650558 to 2283485 tid 2 2024-09-16T05:18:43.977 INFO:tasks.rados.rados.2.smithi204.stdout:2156: writing smithi20457400-12 from 2773245 to 2958197 tid 3 2024-09-16T05:18:43.977 INFO:tasks.rados.rados.2.smithi204.stdout:2157: copy_from oid 16 from oid 21 current snap is 224 2024-09-16T05:18:43.977 INFO:tasks.rados.rados.2.smithi204.stdout:2155: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:18:43.978 INFO:tasks.rados.rados.2.smithi204.stdout:2158: write (excl) oid 11 current snap is 224 2024-09-16T05:18:43.978 INFO:tasks.rados.rados.2.smithi204.stdout:2156: finishing write tid 1 to smithi20457400-12 2024-09-16T05:18:43.978 INFO:tasks.rados.rados.2.smithi204.stdout:2155: finishing copy_from to smithi20457400-4 2024-09-16T05:18:43.978 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 997 (ObjNum 664 snap 192 seq_num 664) dirty exists 2024-09-16T05:18:43.978 INFO:tasks.rados.rados.2.smithi204.stdout:2156: finishing write tid 2 to smithi20457400-12 2024-09-16T05:18:43.978 INFO:tasks.rados.rados.2.smithi204.stdout:2158: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-16T05:18:43.984 INFO:tasks.rados.rados.2.smithi204.stdout:2158: writing smithi20457400-11 from 575004 to 1218885 tid 1 2024-09-16T05:18:43.989 INFO:tasks.rados.rados.2.smithi204.stdout:2158: writing smithi20457400-11 from 1709040 to 2201978 tid 2 2024-09-16T05:18:43.994 INFO:tasks.rados.rados.2.smithi204.stdout:2158: writing smithi20457400-11 from 2917021 to 2975004 tid 3 2024-09-16T05:18:43.994 INFO:tasks.rados.rados.2.smithi204.stdout:2155: done (3 left) 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2159: read oid 29 snap -1 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2159: expect deleted 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2160: snap_create 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2156: finishing write tid 3 to smithi20457400-12 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2156: finishing write tid 4 to smithi20457400-12 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2156: finishing write tid 5 to smithi20457400-12 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 767 (ObjNum 771 snap 224 seq_num 771) dirty exists 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2156: left oid 12 (ObjNum 771 snap 224 seq_num 771) 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2157: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2157: finishing copy_from to smithi20457400-16 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 344 (ObjNum 718 snap 208 seq_num 718) dirty exists 2024-09-16T05:18:43.995 INFO:tasks.rados.rados.2.smithi204.stdout:2158: finishing write tid 1 to smithi20457400-11 2024-09-16T05:18:43.996 INFO:tasks.rados.rados.2.smithi204.stdout:2158: finishing write tid 2 to smithi20457400-11 2024-09-16T05:18:43.996 INFO:tasks.rados.rados.2.smithi204.stdout:2158: finishing write tid 3 to smithi20457400-11 2024-09-16T05:18:44.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:43 smithi032 ceph-mon[27388]: pgmap v1020: 107 pgs: 107 active+clean; 273 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 607 KiB/s wr, 9 op/s 2024-09-16T05:18:44.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2508090855' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.5"}]': finished 2024-09-16T05:18:44.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:43 smithi032 ceph-mon[27388]: osdmap e573: 12 total, 12 up, 12 in 2024-09-16T05:18:44.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3926449911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:44.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:43 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:44.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:43 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:44.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:44.051 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:44.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:44.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:18:44.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:44.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:18:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:43 smithi204 ceph-mon[30933]: pgmap v1020: 107 pgs: 107 active+clean; 273 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 607 KiB/s wr, 9 op/s 2024-09-16T05:18:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2508090855' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.5"}]': finished 2024-09-16T05:18:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:43 smithi204 ceph-mon[30933]: osdmap e573: 12 total, 12 up, 12 in 2024-09-16T05:18:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3926449911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:43 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:43 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:44.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:44.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:43 smithi196 ceph-mon[30802]: pgmap v1020: 107 pgs: 107 active+clean; 273 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 607 KiB/s wr, 9 op/s 2024-09-16T05:18:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2508090855' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.5"}]': finished 2024-09-16T05:18:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:43 smithi196 ceph-mon[30802]: osdmap e573: 12 total, 12 up, 12 in 2024-09-16T05:18:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3926449911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:18:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:43 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:43 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:44.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:44.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:44.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:44.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:44.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:44.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:44.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:44.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:44.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:44.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:44.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:44.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:44.686 INFO:tasks.rados.rados.2.smithi204.stdout:2158: finishing write tid 4 to smithi20457400-11 2024-09-16T05:18:44.686 INFO:tasks.rados.rados.2.smithi204.stdout:2158: finishing write tid 5 to smithi20457400-11 2024-09-16T05:18:44.686 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 829 (ObjNum 772 snap 224 seq_num 772) dirty exists 2024-09-16T05:18:44.686 INFO:tasks.rados.rados.2.smithi204.stdout:2158: left oid 11 (ObjNum 772 snap 224 seq_num 772) 2024-09-16T05:18:44.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:44.784 INFO:tasks.rados.rados.2.smithi204.stdout:2156: done (4 left) 2024-09-16T05:18:44.784 INFO:tasks.rados.rados.2.smithi204.stdout:2157: done (3 left) 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2158: done (2 left) 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2159: done (1 left) 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2160: done (0 left) 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2161: rollback oid 33 current snap is 225 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 33 to 200 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2162: read oid 9 snap 223 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2162: expect (ObjNum 746 snap 215 seq_num 746) 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2163: read oid 21 snap -1 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2163: expect (ObjNum 718 snap 208 seq_num 718) 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2164: read oid 3 snap -1 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2164: expect (ObjNum 767 snap 223 seq_num 767) 2024-09-16T05:18:44.785 INFO:tasks.rados.rados.2.smithi204.stdout:2165: write (excl) oid 49 current snap is 225 2024-09-16T05:18:44.786 INFO:tasks.rados.rados.2.smithi204.stdout:2165: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2024-09-16T05:18:44.792 INFO:tasks.rados.rados.2.smithi204.stdout:2165: writing smithi20457400-49 from 591811 to 1310941 tid 1 2024-09-16T05:18:44.798 INFO:tasks.rados.rados.2.smithi204.stdout:2165: writing smithi20457400-49 from 1767522 to 2404118 tid 2 2024-09-16T05:18:44.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:44.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:44.802 INFO:tasks.rados.rados.2.smithi204.stdout:2165: writing smithi20457400-49 from 2828091 to 2991811 tid 3 2024-09-16T05:18:44.802 INFO:tasks.rados.rados.2.smithi204.stdout:2166: write (excl) oid 14 current snap is 225 2024-09-16T05:18:44.802 INFO:tasks.rados.rados.2.smithi204.stdout:2166: seq_num 774 ranges {608618=794379,2109651=780254} 2024-09-16T05:18:44.809 INFO:tasks.rados.rados.2.smithi204.stdout:2166: writing smithi20457400-14 from 608618 to 1402997 tid 1 2024-09-16T05:18:44.818 INFO:tasks.rados.rados.2.smithi204.stdout:2166: writing smithi20457400-14 from 2109651 to 2889905 tid 2 2024-09-16T05:18:44.818 INFO:tasks.rados.rados.2.smithi204.stdout:2167: delete oid 38 current snap is 225 2024-09-16T05:18:44.818 INFO:tasks.rados.rados.2.smithi204.stdout:2161: finishing rollback tid 1 to smithi20457400-33 2024-09-16T05:18:44.818 INFO:tasks.rados.rados.2.smithi204.stdout:2161: finishing rollback tid 2 to smithi20457400-33 2024-09-16T05:18:44.818 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 836 (ObjNum 657 snap 191 seq_num 657) dirty exists 2024-09-16T05:18:44.818 INFO:tasks.rados.rados.2.smithi204.stdout:2163: expect (ObjNum 718 snap 208 seq_num 718) 2024-09-16T05:18:44.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:44.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:44.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:44.930 INFO:tasks.rados.rados.2.smithi204.stdout:2162: expect (ObjNum 746 snap 215 seq_num 746) 2024-09-16T05:18:45.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:45.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:45.055 INFO:tasks.rados.rados.2.smithi204.stdout:2165: finishing write tid 1 to smithi20457400-49 2024-09-16T05:18:45.055 INFO:tasks.rados.rados.2.smithi204.stdout:2165: finishing write tid 2 to smithi20457400-49 2024-09-16T05:18:45.055 INFO:tasks.rados.rados.2.smithi204.stdout:2165: finishing write tid 3 to smithi20457400-49 2024-09-16T05:18:45.055 INFO:tasks.rados.rados.2.smithi204.stdout:2165: finishing write tid 4 to smithi20457400-49 2024-09-16T05:18:45.055 INFO:tasks.rados.rados.2.smithi204.stdout:2165: finishing write tid 5 to smithi20457400-49 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 595 (ObjNum 773 snap 225 seq_num 773) dirty exists 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2165: left oid 49 (ObjNum 773 snap 225 seq_num 773) 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2166: finishing write tid 1 to smithi20457400-14 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2166: finishing write tid 2 to smithi20457400-14 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2166: finishing write tid 3 to smithi20457400-14 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2161: done (6 left) 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2162: done (5 left) 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2163: done (4 left) 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2165: done (3 left) 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2167: done (2 left) 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2168: rollback oid 24 current snap is 225 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 200 2024-09-16T05:18:45.056 INFO:tasks.rados.rados.2.smithi204.stdout:2166: finishing write tid 4 to smithi20457400-14 2024-09-16T05:18:45.057 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1007 (ObjNum 774 snap 225 seq_num 774) dirty exists 2024-09-16T05:18:45.057 INFO:tasks.rados.rados.2.smithi204.stdout:2166: left oid 14 (ObjNum 774 snap 225 seq_num 774) 2024-09-16T05:18:45.057 INFO:tasks.rados.rados.2.smithi204.stdout:2164: expect (ObjNum 767 snap 223 seq_num 767) 2024-09-16T05:18:45.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:45.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3926449911' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:45.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: osdmap e574: 12 total, 12 up, 12 in 2024-09-16T05:18:45.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: 2.24 scrub starts 2024-09-16T05:18:45.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: 2.24 scrub ok 2024-09-16T05:18:45.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:18:45.103 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:18:45.103 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:18:45.103 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:44 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:18:45.147 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:45.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3926449911' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: osdmap e574: 12 total, 12 up, 12 in 2024-09-16T05:18:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: 2.24 scrub starts 2024-09-16T05:18:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: 2.24 scrub ok 2024-09-16T05:18:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:18:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:18:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:18:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:44 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:18:45.172 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:45.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:45.233 INFO:tasks.rados.rados.2.smithi204.stdout:2168: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:18:45.233 INFO:tasks.rados.rados.2.smithi204.stdout:2168: finishing rollback tid 2 to smithi20457400-24 2024-09-16T05:18:45.233 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 569 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-16T05:18:45.233 INFO:tasks.rados.rados.2.smithi204.stdout:2164: done (2 left) 2024-09-16T05:18:45.233 INFO:tasks.rados.rados.2.smithi204.stdout:2166: done (1 left) 2024-09-16T05:18:45.233 INFO:tasks.rados.rados.2.smithi204.stdout:2168: done (0 left) 2024-09-16T05:18:45.233 INFO:tasks.rados.rados.2.smithi204.stdout:2169: write (excl) oid 11 current snap is 225 2024-09-16T05:18:45.233 INFO:tasks.rados.rados.2.smithi204.stdout:2169: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-16T05:18:45.236 INFO:tasks.rados.rados.2.smithi204.stdout:2169: writing smithi20457400-11 from 642232 to 1303462 tid 1 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3926449911' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: osdmap e574: 12 total, 12 up, 12 in 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: 2.24 scrub starts 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: 2.24 scrub ok 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:18:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:44 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:18:45.244 INFO:tasks.rados.rados.2.smithi204.stdout:2169: writing smithi20457400-11 from 1942968 to 2726891 tid 2 2024-09-16T05:18:45.244 INFO:tasks.rados.rados.2.smithi204.stdout:2170: rollback oid 12 current snap is 225 2024-09-16T05:18:45.244 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 219 2024-09-16T05:18:45.244 INFO:tasks.rados.rados.2.smithi204.stdout:2169: finishing write tid 1 to smithi20457400-11 2024-09-16T05:18:45.245 INFO:tasks.rados.rados.2.smithi204.stdout:2171: snap_create 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:2169: finishing write tid 2 to smithi20457400-11 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:2169: finishing write tid 3 to smithi20457400-11 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:2169: finishing write tid 4 to smithi20457400-11 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 833 (ObjNum 776 snap 225 seq_num 776) dirty exists 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:2169: left oid 11 (ObjNum 776 snap 225 seq_num 776) 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:2170: finishing rollback tid 0 to smithi20457400-12 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:2170: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:2170: finishing rollback tid 2 to smithi20457400-12 2024-09-16T05:18:45.247 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 771 (ObjNum 744 snap 215 seq_num 744) dirty exists 2024-09-16T05:18:45.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:45.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:45.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:45.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:45.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:45.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:45.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:45.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:45.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:45.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:45.650 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:45.650 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:45.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:45.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:45.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:45.677 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:45.678 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:45.709 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:45.709 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:45.730 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:45.730 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:45.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:2169: done (2 left) 2024-09-16T05:18:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:2170: done (1 left) 2024-09-16T05:18:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:2171: done (0 left) 2024-09-16T05:18:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:2172: read oid 3 snap 219 2024-09-16T05:18:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:2172: expect (ObjNum 708 snap 203 seq_num 708) 2024-09-16T05:18:45.767 INFO:tasks.rados.rados.2.smithi204.stdout:2173: delete oid 4 current snap is 226 2024-09-16T05:18:45.770 INFO:tasks.rados.rados.2.smithi204.stdout:2173: done (1 left) 2024-09-16T05:18:45.770 INFO:tasks.rados.rados.2.smithi204.stdout:2174: snap_remove snap 225 2024-09-16T05:18:45.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:45.778 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:45.799 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:45.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:45.800 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:45.800 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:45.843 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:45.843 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:45.865 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:45.886 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:45.886 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:45.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:45.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:45.909 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:45.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:45.936 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:45 smithi032 ceph-mon[27388]: pgmap v1023: 107 pgs: 107 active+clean; 279 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2024-09-16T05:18:45.936 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:45 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:18:45.936 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:45 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:18:45.936 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:45 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:18:45.936 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:45 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:18:45.936 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:45 smithi032 ceph-mon[27388]: osdmap e575: 12 total, 12 up, 12 in 2024-09-16T05:18:45.936 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3330451294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:45.936 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:45 smithi032 ceph-mon[27388]: osdmap e576: 12 total, 12 up, 12 in 2024-09-16T05:18:46.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:46.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:46.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:46.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:45 smithi204 ceph-mon[30933]: pgmap v1023: 107 pgs: 107 active+clean; 279 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2024-09-16T05:18:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:45 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:18:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:45 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:18:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:45 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:18:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:45 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:18:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:45 smithi204 ceph-mon[30933]: osdmap e575: 12 total, 12 up, 12 in 2024-09-16T05:18:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3330451294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:45 smithi204 ceph-mon[30933]: osdmap e576: 12 total, 12 up, 12 in 2024-09-16T05:18:46.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:46.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:45 smithi196 ceph-mon[30802]: pgmap v1023: 107 pgs: 107 active+clean; 279 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2024-09-16T05:18:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:45 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:18:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:45 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:18:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:45 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:18:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:45 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:18:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:45 smithi196 ceph-mon[30802]: osdmap e575: 12 total, 12 up, 12 in 2024-09-16T05:18:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3330451294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:45 smithi196 ceph-mon[30802]: osdmap e576: 12 total, 12 up, 12 in 2024-09-16T05:18:46.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:46.304 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:46.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:46.339 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:46.340 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:46.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:46.429 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:46.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:46.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:46.553 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:46.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:46.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:46.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:46.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:46.766 INFO:tasks.rados.rados.2.smithi204.stdout:2174: done (1 left) 2024-09-16T05:18:46.766 INFO:tasks.rados.rados.2.smithi204.stdout:2175: snap_remove snap 216 2024-09-16T05:18:46.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:46.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:46.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:46.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:46.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:18:47.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:47.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:47.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:47.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:46 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:18:47.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:46 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:18:47.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:46 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:18:47.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:46 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:18:47.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:46 smithi032 ceph-mon[27388]: osdmap e577: 12 total, 12 up, 12 in 2024-09-16T05:18:47.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:18:47.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:47.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:46 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:18:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:46 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:18:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:46 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:18:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:46 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:18:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:46 smithi204 ceph-mon[30933]: osdmap e577: 12 total, 12 up, 12 in 2024-09-16T05:18:47.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:47.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:47.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:46 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:18:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:46 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:18:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:46 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:18:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:46 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:18:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:46 smithi196 ceph-mon[30802]: osdmap e577: 12 total, 12 up, 12 in 2024-09-16T05:18:47.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:47.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:47.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:47.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:47.436 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:47.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:47.468 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:47.489 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:47.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:47.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:47.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:47.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:47.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:47.768 INFO:tasks.rados.rados.2.smithi204.stdout:2175: done (1 left) 2024-09-16T05:18:47.768 INFO:tasks.rados.rados.2.smithi204.stdout:2176: write (excl) oid 39 current snap is 226 2024-09-16T05:18:47.768 INFO:tasks.rados.rados.2.smithi204.stdout:2176: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-09-16T05:18:47.775 INFO:tasks.rados.rados.2.smithi204.stdout:2176: writing smithi20457400-39 from 675846 to 1087574 tid 1 2024-09-16T05:18:47.783 INFO:tasks.rados.rados.2.smithi204.stdout:2176: writing smithi20457400-39 from 1543579 to 2331171 tid 2 2024-09-16T05:18:47.787 INFO:tasks.rados.rados.2.smithi204.stdout:2176: writing smithi20457400-39 from 2848853 to 3075846 tid 3 2024-09-16T05:18:47.788 INFO:tasks.rados.rados.2.smithi204.stdout:2177: read oid 30 snap -1 2024-09-16T05:18:47.788 INFO:tasks.rados.rados.2.smithi204.stdout:2177: expect (ObjNum 717 snap 208 seq_num 717) 2024-09-16T05:18:47.788 INFO:tasks.rados.rados.2.smithi204.stdout:2178: write (excl) oid 35 current snap is 226 2024-09-16T05:18:47.788 INFO:tasks.rados.rados.2.smithi204.stdout:2172: expect (ObjNum 708 snap 203 seq_num 708) 2024-09-16T05:18:47.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:47.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:47.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:47.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:47.920 INFO:tasks.rados.rados.2.smithi204.stdout:2176: finishing write tid 1 to smithi20457400-39 2024-09-16T05:18:47.920 INFO:tasks.rados.rados.2.smithi204.stdout:2176: finishing write tid 2 to smithi20457400-39 2024-09-16T05:18:47.920 INFO:tasks.rados.rados.2.smithi204.stdout:2176: finishing write tid 3 to smithi20457400-39 2024-09-16T05:18:47.920 INFO:tasks.rados.rados.2.smithi204.stdout:2176: finishing write tid 4 to smithi20457400-39 2024-09-16T05:18:47.920 INFO:tasks.rados.rados.2.smithi204.stdout:2176: finishing write tid 5 to smithi20457400-39 2024-09-16T05:18:47.920 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 842 (ObjNum 778 snap 226 seq_num 778) dirty exists 2024-09-16T05:18:47.920 INFO:tasks.rados.rados.2.smithi204.stdout:2176: left oid 39 (ObjNum 778 snap 226 seq_num 778) 2024-09-16T05:18:47.920 INFO:tasks.rados.rados.2.smithi204.stdout:2178: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-09-16T05:18:47.925 INFO:tasks.rados.rados.2.smithi204.stdout:2178: writing smithi20457400-35 from 692653 to 1179630 tid 1 2024-09-16T05:18:47.931 INFO:tasks.rados.rados.2.smithi204.stdout:2178: writing smithi20457400-35 from 1602061 to 2249664 tid 2 2024-09-16T05:18:47.936 INFO:tasks.rados.rados.2.smithi204.stdout:2178: writing smithi20457400-35 from 2992629 to 3092653 tid 3 2024-09-16T05:18:47.937 INFO:tasks.rados.rados.2.smithi204.stdout:2172: done (3 left) 2024-09-16T05:18:47.937 INFO:tasks.rados.rados.2.smithi204.stdout:2176: done (2 left) 2024-09-16T05:18:47.937 INFO:tasks.rados.rados.2.smithi204.stdout:2179: write (excl) oid 50 current snap is 226 2024-09-16T05:18:47.937 INFO:tasks.rados.rados.2.smithi204.stdout:2177: expect (ObjNum 717 snap 208 seq_num 717) 2024-09-16T05:18:47.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:47.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:47.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:47 smithi032 ceph-mon[27388]: pgmap v1026: 107 pgs: 1 active+remapped, 106 active+clean; 290 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 3.2 MiB/s wr, 31 op/s; 871 KiB/s, 3 objects/s recovering 2024-09-16T05:18:47.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:47 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:18:47.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:47 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:18:47.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:47 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:18:47.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:47 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:18:47.996 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:47 smithi032 ceph-mon[27388]: osdmap e578: 12 total, 12 up, 12 in 2024-09-16T05:18:48.036 INFO:tasks.rados.rados.2.smithi204.stdout:2178: finishing write tid 1 to smithi20457400-35 2024-09-16T05:18:48.036 INFO:tasks.rados.rados.2.smithi204.stdout:2178: finishing write tid 2 to smithi20457400-35 2024-09-16T05:18:48.037 INFO:tasks.rados.rados.2.smithi204.stdout:2178: finishing write tid 3 to smithi20457400-35 2024-09-16T05:18:48.037 INFO:tasks.rados.rados.2.smithi204.stdout:2178: finishing write tid 4 to smithi20457400-35 2024-09-16T05:18:48.037 INFO:tasks.rados.rados.2.smithi204.stdout:2179: seq_num 780 ranges {709460=562226,2060543=791261} 2024-09-16T05:18:48.043 INFO:tasks.rados.rados.2.smithi204.stdout:2179: writing smithi20457400-50 from 709460 to 1271686 tid 1 2024-09-16T05:18:48.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:48.051 INFO:tasks.rados.rados.2.smithi204.stdout:2179: writing smithi20457400-50 from 2060543 to 2851804 tid 2 2024-09-16T05:18:48.051 INFO:tasks.rados.rados.2.smithi204.stdout:2177: done (2 left) 2024-09-16T05:18:48.052 INFO:tasks.rados.rados.2.smithi204.stdout:2180: snap_create 2024-09-16T05:18:48.052 INFO:tasks.rados.rados.2.smithi204.stdout:2179: finishing write tid 1 to smithi20457400-50 2024-09-16T05:18:48.055 INFO:tasks.rados.rados.2.smithi204.stdout:2179: finishing write tid 2 to smithi20457400-50 2024-09-16T05:18:48.055 INFO:tasks.rados.rados.2.smithi204.stdout:2179: finishing write tid 3 to smithi20457400-50 2024-09-16T05:18:48.055 INFO:tasks.rados.rados.2.smithi204.stdout:2179: finishing write tid 4 to smithi20457400-50 2024-09-16T05:18:48.055 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 452 (ObjNum 780 snap 226 seq_num 780) dirty exists 2024-09-16T05:18:48.056 INFO:tasks.rados.rados.2.smithi204.stdout:2179: left oid 50 (ObjNum 780 snap 226 seq_num 780) 2024-09-16T05:18:48.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:48.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:47 smithi204 ceph-mon[30933]: pgmap v1026: 107 pgs: 1 active+remapped, 106 active+clean; 290 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 3.2 MiB/s wr, 31 op/s; 871 KiB/s, 3 objects/s recovering 2024-09-16T05:18:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:47 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:18:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:47 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:18:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:47 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:18:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:47 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:18:48.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:47 smithi204 ceph-mon[30933]: osdmap e578: 12 total, 12 up, 12 in 2024-09-16T05:18:48.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:48.193 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:48.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:48.217 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:48.217 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:48.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:47 smithi196 ceph-mon[30802]: pgmap v1026: 107 pgs: 1 active+remapped, 106 active+clean; 290 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 3.2 MiB/s wr, 31 op/s; 871 KiB/s, 3 objects/s recovering 2024-09-16T05:18:48.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:47 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:18:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:47 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:18:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:47 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:18:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:47 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:18:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:47 smithi196 ceph-mon[30802]: osdmap e578: 12 total, 12 up, 12 in 2024-09-16T05:18:48.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:48.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:48.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:48.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:48.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:48.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:48.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:48.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:48.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:48.564 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:48.598 INFO:tasks.rados.rados.2.smithi204.stdout:2178: finishing write tid 5 to smithi20457400-35 2024-09-16T05:18:48.598 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 840 (ObjNum 779 snap 226 seq_num 779) dirty exists 2024-09-16T05:18:48.599 INFO:tasks.rados.rados.2.smithi204.stdout:2178: left oid 35 (ObjNum 779 snap 226 seq_num 779) 2024-09-16T05:18:48.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:48.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:48.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:48.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:48.792 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:18:48.796 INFO:tasks.rados.rados.2.smithi204.stdout:2178: done (2 left) 2024-09-16T05:18:48.796 INFO:tasks.rados.rados.2.smithi204.stdout:2179: done (1 left) 2024-09-16T05:18:48.797 INFO:tasks.rados.rados.2.smithi204.stdout:2180: done (0 left) 2024-09-16T05:18:48.797 INFO:tasks.rados.rados.2.smithi204.stdout:2181: read oid 37 snap -1 2024-09-16T05:18:48.797 INFO:tasks.rados.rados.2.smithi204.stdout:2181: expect (ObjNum 694 snap 201 seq_num 694) 2024-09-16T05:18:48.797 INFO:tasks.rados.rados.2.smithi204.stdout:2182: read oid 13 snap -1 2024-09-16T05:18:48.797 INFO:tasks.rados.rados.2.smithi204.stdout:2182: expect (ObjNum 727 snap 210 seq_num 727) 2024-09-16T05:18:48.797 INFO:tasks.rados.rados.2.smithi204.stdout:2183: write (excl) oid 45 current snap is 227 2024-09-16T05:18:48.797 INFO:tasks.rados.rados.2.smithi204.stdout:2183: seq_num 781 ranges {726267=637475,2119025=651272} 2024-09-16T05:18:48.804 INFO:tasks.rados.rados.2.smithi204.stdout:2183: writing smithi20457400-45 from 726267 to 1363742 tid 1 2024-09-16T05:18:48.811 INFO:tasks.rados.rados.2.smithi204.stdout:2183: writing smithi20457400-45 from 2119025 to 2770297 tid 2 2024-09-16T05:18:48.811 INFO:tasks.rados.rados.2.smithi204.stdout:2184: read oid 1 snap -1 2024-09-16T05:18:48.812 INFO:tasks.rados.rados.2.smithi204.stdout:2184: expect deleted 2024-09-16T05:18:48.812 INFO:tasks.rados.rados.2.smithi204.stdout:2185: write oid 50 current snap is 227 2024-09-16T05:18:48.812 INFO:tasks.rados.rados.2.smithi204.stdout:2185: seq_num 782 ranges {743074=712724,2061154=794930} 2024-09-16T05:18:48.814 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:48.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:48.819 INFO:tasks.rados.rados.2.smithi204.stdout:2185: writing smithi20457400-50 from 743074 to 1455798 tid 1 2024-09-16T05:18:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2185: writing smithi20457400-50 from 2061154 to 2856084 tid 2 2024-09-16T05:18:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2186: copy_from oid 47 from oid 14 current snap is 227 2024-09-16T05:18:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2182: expect (ObjNum 727 snap 210 seq_num 727) 2024-09-16T05:18:48.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:48.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:48.975 INFO:tasks.rados.rados.2.smithi204.stdout:2181: expect (ObjNum 694 snap 201 seq_num 694) 2024-09-16T05:18:49.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:49.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:49.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:49.092 INFO:tasks.rados.rados.2.smithi204.stdout:2183: finishing write tid 1 to smithi20457400-45 2024-09-16T05:18:49.092 INFO:tasks.rados.rados.2.smithi204.stdout:2183: finishing write tid 2 to smithi20457400-45 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2183: finishing write tid 3 to smithi20457400-45 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2183: finishing write tid 4 to smithi20457400-45 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 369 (ObjNum 781 snap 227 seq_num 781) dirty exists 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2183: left oid 45 (ObjNum 781 snap 227 seq_num 781) 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2185: finishing write tid 1 to smithi20457400-50 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2181: done (5 left) 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2182: done (4 left) 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2183: done (3 left) 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2184: done (2 left) 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2187: write (excl) oid 20 current snap is 227 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2185: finishing write tid 2 to smithi20457400-50 2024-09-16T05:18:49.093 INFO:tasks.rados.rados.2.smithi204.stdout:2187: seq_num 783 ranges {759881=787973,2119636=654941} 2024-09-16T05:18:49.100 INFO:tasks.rados.rados.2.smithi204.stdout:2187: writing smithi20457400-20 from 759881 to 1547854 tid 1 2024-09-16T05:18:49.107 INFO:tasks.rados.rados.2.smithi204.stdout:2187: writing smithi20457400-20 from 2119636 to 2774577 tid 2 2024-09-16T05:18:49.107 INFO:tasks.rados.rados.2.smithi204.stdout:2188: copy_from oid 42 from oid 48 current snap is 227 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2185: finishing write tid 3 to smithi20457400-50 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2185: finishing write tid 4 to smithi20457400-50 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 456 (ObjNum 782 snap 227 seq_num 782) dirty exists 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2185: left oid 50 (ObjNum 782 snap 227 seq_num 782) 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2186: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2185: done (3 left) 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2189: copy_from oid 22 from oid 38 current snap is 227 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2186: finishing copy_from to smithi20457400-47 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1007 (ObjNum 774 snap 225 seq_num 774) dirty exists 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2187: finishing write tid 1 to smithi20457400-20 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2186: done (3 left) 2024-09-16T05:18:49.108 INFO:tasks.rados.rados.2.smithi204.stdout:2190: rollback oid 43 current snap is 227 2024-09-16T05:18:49.109 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 43 to 218 2024-09-16T05:18:49.109 INFO:tasks.rados.rados.2.smithi204.stdout:2191: snap_create 2024-09-16T05:18:49.111 INFO:tasks.rados.rados.2.smithi204.stdout:2190: finishing rollback tid 0 to smithi20457400-43 2024-09-16T05:18:49.111 INFO:tasks.rados.rados.2.smithi204.stdout:2190: finishing rollback tid 1 to smithi20457400-43 2024-09-16T05:18:49.111 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 597 (ObjNum 145 snap 0 seq_num 4282368520) dirty dne 2024-09-16T05:18:49.111 INFO:tasks.rados.rados.2.smithi204.stdout:2187: finishing write tid 2 to smithi20457400-20 2024-09-16T05:18:49.111 INFO:tasks.rados.rados.2.smithi204.stdout:2187: finishing write tid 3 to smithi20457400-20 2024-09-16T05:18:49.111 INFO:tasks.rados.rados.2.smithi204.stdout:2187: finishing write tid 4 to smithi20457400-20 2024-09-16T05:18:49.111 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 846 (ObjNum 783 snap 227 seq_num 783) dirty exists 2024-09-16T05:18:49.111 INFO:tasks.rados.rados.2.smithi204.stdout:2187: left oid 20 (ObjNum 783 snap 227 seq_num 783) 2024-09-16T05:18:49.113 INFO:tasks.rados.rados.2.smithi204.stdout:2189: finishing copy_from to smithi20457400-22 2024-09-16T05:18:49.113 INFO:tasks.rados.rados.2.smithi204.stdout:2189: got expected ENOENT (src dne) 2024-09-16T05:18:49.113 INFO:tasks.rados.rados.2.smithi204.stdout:2189: finishing copy_from racing read to smithi20457400-22 2024-09-16T05:18:49.119 INFO:tasks.rados.rados.2.smithi204.stdout:2188: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:18:49.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:48 smithi032 ceph-mon[27388]: 2.36 scrub starts 2024-09-16T05:18:49.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:48 smithi032 ceph-mon[27388]: 2.36 scrub ok 2024-09-16T05:18:49.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:48 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:18:49.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:48 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:18:49.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:48 smithi032 ceph-mon[27388]: 2.3a scrub starts 2024-09-16T05:18:49.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:48 smithi032 ceph-mon[27388]: 2.3a scrub ok 2024-09-16T05:18:49.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2548682185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:49.121 INFO:tasks.rados.rados.2.smithi204.stdout:2188: finishing copy_from to smithi20457400-42 2024-09-16T05:18:49.121 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 408 (ObjNum 763 snap 223 seq_num 763) dirty exists 2024-09-16T05:18:49.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:18:49.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:48 smithi204 ceph-mon[30933]: 2.36 scrub starts 2024-09-16T05:18:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:48 smithi204 ceph-mon[30933]: 2.36 scrub ok 2024-09-16T05:18:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:48 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:18:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:48 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:18:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:48 smithi204 ceph-mon[30933]: 2.3a scrub starts 2024-09-16T05:18:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:48 smithi204 ceph-mon[30933]: 2.3a scrub ok 2024-09-16T05:18:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2548682185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:49.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:49.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:48 smithi196 ceph-mon[30802]: 2.36 scrub starts 2024-09-16T05:18:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:48 smithi196 ceph-mon[30802]: 2.36 scrub ok 2024-09-16T05:18:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:48 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:18:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:48 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:18:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:48 smithi196 ceph-mon[30802]: 2.3a scrub starts 2024-09-16T05:18:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:48 smithi196 ceph-mon[30802]: 2.3a scrub ok 2024-09-16T05:18:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2548682185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:18:49.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:49.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:49.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:49.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:49.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:18:49.440 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:49.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:49.508 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:18:49.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T05:18:49.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:49.563 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:49.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:49.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:49.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:49.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:49.797 INFO:tasks.rados.rados.2.smithi204.stdout:2187: done (4 left) 2024-09-16T05:18:49.797 INFO:tasks.rados.rados.2.smithi204.stdout:2188: done (3 left) 2024-09-16T05:18:49.797 INFO:tasks.rados.rados.2.smithi204.stdout:2189: done (2 left) 2024-09-16T05:18:49.797 INFO:tasks.rados.rados.2.smithi204.stdout:2190: done (1 left) 2024-09-16T05:18:49.797 INFO:tasks.rados.rados.2.smithi204.stdout:2191: done (0 left) 2024-09-16T05:18:49.797 INFO:tasks.rados.rados.2.smithi204.stdout:2192: copy_from oid 7 from oid 40 current snap is 228 2024-09-16T05:18:49.798 INFO:tasks.rados.rados.2.smithi204.stdout:2193: read oid 38 snap -1 2024-09-16T05:18:49.798 INFO:tasks.rados.rados.2.smithi204.stdout:2193: expect deleted 2024-09-16T05:18:49.798 INFO:tasks.rados.rados.2.smithi204.stdout:2194: read oid 31 snap 219 2024-09-16T05:18:49.798 INFO:tasks.rados.rados.2.smithi204.stdout:2194: expect deleted 2024-09-16T05:18:49.798 INFO:tasks.rados.rados.2.smithi204.stdout:2195: snap_create 2024-09-16T05:18:49.815 INFO:tasks.rados.rados.2.smithi204.stdout:2192: finishing copy_from racing read to smithi20457400-7 2024-09-16T05:18:49.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:49.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:49.825 INFO:tasks.rados.rados.2.smithi204.stdout:2192: finishing copy_from to smithi20457400-7 2024-09-16T05:18:49.825 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 764 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:18:49.844 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:49.845 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:49.849 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:49.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:49.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:49.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:50.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:18:50.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: pgmap v1029: 107 pgs: 1 active+clean+snaptrim, 1 active+remapped, 105 active+clean; 301 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.5 MiB/s wr, 22 op/s; 871 KiB/s, 3 objects/s recovering 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2548682185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: osdmap e579: 12 total, 12 up, 12 in 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:18:50.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:49 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:18:50.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:50.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:50.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: pgmap v1029: 107 pgs: 1 active+clean+snaptrim, 1 active+remapped, 105 active+clean; 301 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.5 MiB/s wr, 22 op/s; 871 KiB/s, 3 objects/s recovering 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2548682185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: osdmap e579: 12 total, 12 up, 12 in 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:18:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:49 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:18:50.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:50.198 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:50.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: pgmap v1029: 107 pgs: 1 active+clean+snaptrim, 1 active+remapped, 105 active+clean; 301 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.5 MiB/s wr, 22 op/s; 871 KiB/s, 3 objects/s recovering 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2548682185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: osdmap e579: 12 total, 12 up, 12 in 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:18:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:49 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:18:50.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:50.324 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:50.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:50.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:50.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:50.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:50.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:50.524 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:18:50.524 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:18:50.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:50.580 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:50.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:50.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:50.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:50.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:50.766 INFO:tasks.rados.rados.2.smithi204.stdout:2192: done (3 left) 2024-09-16T05:18:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:2193: done (2 left) 2024-09-16T05:18:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:2194: done (1 left) 2024-09-16T05:18:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:2195: done (0 left) 2024-09-16T05:18:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:2196: read oid 31 snap -1 2024-09-16T05:18:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:2196: expect deleted 2024-09-16T05:18:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:2197: write (excl) oid 24 current snap is 229 2024-09-16T05:18:50.767 INFO:tasks.rados.rados.2.smithi204.stdout:2197: seq_num 784 ranges {776688=579575,1894471=798599} 2024-09-16T05:18:50.776 INFO:tasks.rados.rados.2.smithi204.stdout:2197: writing smithi20457400-24 from 776688 to 1356263 tid 1 2024-09-16T05:18:50.781 INFO:tasks.rados.rados.2.smithi204.stdout:2197: writing smithi20457400-24 from 1894471 to 2693070 tid 2 2024-09-16T05:18:50.781 INFO:tasks.rados.rados.2.smithi204.stdout:2198: snap_remove snap 221 2024-09-16T05:18:50.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:50.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:50.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:18:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:18:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: osdmap e580: 12 total, 12 up, 12 in 2024-09-16T05:18:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:50.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:50.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:18:50.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:18:50.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: 1.12 deep-scrub starts 2024-09-16T05:18:50.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:50 smithi032 ceph-mon[27388]: osdmap e581: 12 total, 12 up, 12 in 2024-09-16T05:18:50.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:50.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:51.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:51.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:51.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:18:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:18:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: osdmap e580: 12 total, 12 up, 12 in 2024-09-16T05:18:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:18:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:18:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: 1.12 deep-scrub starts 2024-09-16T05:18:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:50 smithi204 ceph-mon[30933]: osdmap e581: 12 total, 12 up, 12 in 2024-09-16T05:18:51.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:51.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:51.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: osdmap e580: 12 total, 12 up, 12 in 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: 1.12 deep-scrub starts 2024-09-16T05:18:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:50 smithi196 ceph-mon[30802]: osdmap e581: 12 total, 12 up, 12 in 2024-09-16T05:18:51.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:51.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:51.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:51.387 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:51.408 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:51.409 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:51.429 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:51.430 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:51.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:51.454 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:51.454 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:51.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:51.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:51.475 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:51.476 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:51.499 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:51.500 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:51.521 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:51.543 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:51.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:51.564 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:51.564 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:51.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:51.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:51.586 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:51.586 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:51.607 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:51.628 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:51.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:51.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:51.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:51.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:51.764 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2198: done (2 left) 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2199: read oid 16 snap -1 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2199: expect (ObjNum 718 snap 208 seq_num 718) 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2200: delete oid 12 current snap is 229 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2197: finishing write tid 1 to smithi20457400-24 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2197: finishing write tid 2 to smithi20457400-24 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2197: finishing write tid 3 to smithi20457400-24 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2197: finishing write tid 4 to smithi20457400-24 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 575 (ObjNum 784 snap 229 seq_num 784) dirty exists 2024-09-16T05:18:51.768 INFO:tasks.rados.rados.2.smithi204.stdout:2197: left oid 24 (ObjNum 784 snap 229 seq_num 784) 2024-09-16T05:18:51.773 INFO:tasks.rados.rados.2.smithi204.stdout:2196: done (3 left) 2024-09-16T05:18:51.773 INFO:tasks.rados.rados.2.smithi204.stdout:2197: done (2 left) 2024-09-16T05:18:51.773 INFO:tasks.rados.rados.2.smithi204.stdout:2200: done (1 left) 2024-09-16T05:18:51.773 INFO:tasks.rados.rados.2.smithi204.stdout:2201: delete oid 35 current snap is 229 2024-09-16T05:18:51.778 INFO:tasks.rados.rados.2.smithi204.stdout:2201: done (1 left) 2024-09-16T05:18:51.778 INFO:tasks.rados.rados.2.smithi204.stdout:2202: write (excl) oid 19 current snap is 229 2024-09-16T05:18:51.778 INFO:tasks.rados.rados.2.smithi204.stdout:2202: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-09-16T05:18:51.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:18:51.783 INFO:tasks.rados.rados.2.smithi204.stdout:2202: writing smithi20457400-19 from 427109 to 832431 tid 1 2024-09-16T05:18:51.789 INFO:tasks.rados.rados.2.smithi204.stdout:2202: writing smithi20457400-19 from 1553564 to 2099490 tid 2 2024-09-16T05:18:51.795 INFO:tasks.rados.rados.2.smithi204.stdout:2202: writing smithi20457400-19 from 2579307 to 3227109 tid 3 2024-09-16T05:18:51.795 INFO:tasks.rados.rados.2.smithi204.stdout:2203: snap_remove snap 226 2024-09-16T05:18:51.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:51.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:51.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:51.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:51.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:51.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:51.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:18:52.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:52.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:52.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: pgmap v1032: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 306 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2024-09-16T05:18:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:18:52.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: 1.12 deep-scrub ok 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3435684999' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1629787708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3435684999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:51 smithi032 ceph-mon[27388]: osdmap e582: 12 total, 12 up, 12 in 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: pgmap v1032: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 306 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: 1.12 deep-scrub ok 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3435684999' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1629787708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3435684999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:18:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:51 smithi204 ceph-mon[30933]: osdmap e582: 12 total, 12 up, 12 in 2024-09-16T05:18:52.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:52.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:52.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: pgmap v1032: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 306 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: 1.12 deep-scrub ok 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3435684999' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1629787708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3435684999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:18:52.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:51 smithi196 ceph-mon[30802]: osdmap e582: 12 total, 12 up, 12 in 2024-09-16T05:18:52.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:52.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:52.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:52.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:52.454 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:52.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:52.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:52.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:52.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:52.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:52.702 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:52.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:52.769 INFO:tasks.rados.rados.2.smithi204.stdout:2203: done (2 left) 2024-09-16T05:18:52.770 INFO:tasks.rados.rados.2.smithi204.stdout:2204: write (excl) oid 17 current snap is 229 2024-09-16T05:18:52.770 INFO:tasks.rados.rados.2.smithi204.stdout:2204: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2024-09-16T05:18:52.775 INFO:tasks.rados.rados.2.smithi204.stdout:2204: writing smithi20457400-17 from 443916 to 924487 tid 1 2024-09-16T05:18:52.781 INFO:tasks.rados.rados.2.smithi204.stdout:2204: writing smithi20457400-17 from 1612046 to 2017983 tid 2 2024-09-16T05:18:52.787 INFO:tasks.rados.rados.2.smithi204.stdout:2204: writing smithi20457400-17 from 2606730 to 3101782 tid 3 2024-09-16T05:18:52.787 INFO:tasks.rados.rados.2.smithi204.stdout:2205: write (excl) oid 7 current snap is 229 2024-09-16T05:18:52.787 INFO:tasks.rados.rados.2.smithi204.stdout:2205: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2024-09-16T05:18:52.792 INFO:tasks.rados.rados.2.smithi204.stdout:2205: writing smithi20457400-7 from 460723 to 1016543 tid 1 2024-09-16T05:18:52.798 INFO:tasks.rados.rados.2.smithi204.stdout:2205: writing smithi20457400-7 from 1670528 to 2220123 tid 2 2024-09-16T05:18:52.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:52.805 INFO:tasks.rados.rados.2.smithi204.stdout:2205: writing smithi20457400-7 from 2634153 to 3260723 tid 3 2024-09-16T05:18:52.805 INFO:tasks.rados.rados.2.smithi204.stdout:2206: copy_from oid 21 from oid 11 current snap is 229 2024-09-16T05:18:52.805 INFO:tasks.rados.rados.2.smithi204.stdout:2199: expect (ObjNum 718 snap 208 seq_num 718) 2024-09-16T05:18:52.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:52.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:52.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2202: finishing write tid 1 to smithi20457400-19 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2202: finishing write tid 2 to smithi20457400-19 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2199: done (4 left) 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2207: delete oid 48 current snap is 229 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2202: finishing write tid 3 to smithi20457400-19 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2202: finishing write tid 4 to smithi20457400-19 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2202: finishing write tid 5 to smithi20457400-19 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 326 (ObjNum 787 snap 229 seq_num 787) dirty exists 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2202: left oid 19 (ObjNum 787 snap 229 seq_num 787) 2024-09-16T05:18:52.917 INFO:tasks.rados.rados.2.smithi204.stdout:2204: finishing write tid 1 to smithi20457400-17 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2204: finishing write tid 2 to smithi20457400-17 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2204: finishing write tid 3 to smithi20457400-17 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2204: finishing write tid 4 to smithi20457400-17 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2204: finishing write tid 5 to smithi20457400-17 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 715 (ObjNum 788 snap 229 seq_num 788) dirty exists 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2204: left oid 17 (ObjNum 788 snap 229 seq_num 788) 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2205: finishing write tid 1 to smithi20457400-7 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2205: finishing write tid 2 to smithi20457400-7 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2205: finishing write tid 3 to smithi20457400-7 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2206: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2206: finishing copy_from to smithi20457400-21 2024-09-16T05:18:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 833 (ObjNum 776 snap 225 seq_num 776) dirty exists 2024-09-16T05:18:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:2202: done (4 left) 2024-09-16T05:18:52.920 INFO:tasks.rados.rados.2.smithi204.stdout:2204: done (3 left) 2024-09-16T05:18:52.920 INFO:tasks.rados.rados.2.smithi204.stdout:2206: done (2 left) 2024-09-16T05:18:52.920 INFO:tasks.rados.rados.2.smithi204.stdout:2207: done (1 left) 2024-09-16T05:18:52.920 INFO:tasks.rados.rados.2.smithi204.stdout:2208: snap_create 2024-09-16T05:18:52.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:52.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:52.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:52.953 INFO:tasks.rados.rados.2.smithi204.stdout:2205: finishing write tid 4 to smithi20457400-7 2024-09-16T05:18:52.953 INFO:tasks.rados.rados.2.smithi204.stdout:2205: finishing write tid 5 to smithi20457400-7 2024-09-16T05:18:52.953 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 768 (ObjNum 789 snap 229 seq_num 789) dirty exists 2024-09-16T05:18:52.953 INFO:tasks.rados.rados.2.smithi204.stdout:2205: left oid 7 (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:18:53.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:53.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:52 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:18:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:52 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:18:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:52 smithi032 ceph-mon[27388]: 2.1a deep-scrub starts 2024-09-16T05:18:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:52 smithi032 ceph-mon[27388]: 2.1a deep-scrub ok 2024-09-16T05:18:53.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:52 smithi032 ceph-mon[27388]: osdmap e583: 12 total, 12 up, 12 in 2024-09-16T05:18:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:52 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:18:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:52 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:18:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:52 smithi204 ceph-mon[30933]: 2.1a deep-scrub starts 2024-09-16T05:18:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:52 smithi204 ceph-mon[30933]: 2.1a deep-scrub ok 2024-09-16T05:18:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:52 smithi204 ceph-mon[30933]: osdmap e583: 12 total, 12 up, 12 in 2024-09-16T05:18:53.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:53.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:53.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:52 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:18:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:52 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:18:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:52 smithi196 ceph-mon[30802]: 2.1a deep-scrub starts 2024-09-16T05:18:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:52 smithi196 ceph-mon[30802]: 2.1a deep-scrub ok 2024-09-16T05:18:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:52 smithi196 ceph-mon[30802]: osdmap e583: 12 total, 12 up, 12 in 2024-09-16T05:18:53.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:53.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:53.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:53.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:53.455 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:53.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:53.538 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:53.540 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:53.547 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:53.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:53.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:53.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:53.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:53.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:53.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:53.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:53.816 INFO:tasks.rados.rados.2.smithi204.stdout:2205: done (1 left) 2024-09-16T05:18:53.816 INFO:tasks.rados.rados.2.smithi204.stdout:2208: done (0 left) 2024-09-16T05:18:53.816 INFO:tasks.rados.rados.2.smithi204.stdout:2209: write oid 32 current snap is 230 2024-09-16T05:18:53.816 INFO:tasks.rados.rados.2.smithi204.stdout:2209: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-09-16T05:18:53.821 INFO:tasks.rados.rados.2.smithi204.stdout:2209: writing smithi20457400-32 from 494337 to 917008 tid 1 2024-09-16T05:18:53.827 INFO:tasks.rados.rados.2.smithi204.stdout:2209: writing smithi20457400-32 from 1387492 to 1940756 tid 2 2024-09-16T05:18:53.833 INFO:tasks.rados.rados.2.smithi204.stdout:2209: writing smithi20457400-32 from 2688999 to 3134220 tid 3 2024-09-16T05:18:53.834 INFO:tasks.rados.rados.2.smithi204.stdout:2210: write (excl) oid 43 current snap is 230 2024-09-16T05:18:53.834 INFO:tasks.rados.rados.2.smithi204.stdout:2209: finishing write tid 1 to smithi20457400-32 2024-09-16T05:18:53.834 INFO:tasks.rados.rados.2.smithi204.stdout:2210: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2024-09-16T05:18:53.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:53.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:53.839 INFO:tasks.rados.rados.2.smithi204.stdout:2210: writing smithi20457400-43 from 511144 to 1009064 tid 1 2024-09-16T05:18:53.844 INFO:tasks.rados.rados.2.smithi204.stdout:2210: writing smithi20457400-43 from 1445974 to 1859249 tid 2 2024-09-16T05:18:53.854 INFO:tasks.rados.rados.2.smithi204.stdout:2210: writing smithi20457400-43 from 2316422 to 2972915 tid 3 2024-09-16T05:18:53.854 INFO:tasks.rados.rados.2.smithi204.stdout:2211: copy_from oid 16 from oid 45 current snap is 230 2024-09-16T05:18:53.854 INFO:tasks.rados.rados.2.smithi204.stdout:2209: finishing write tid 2 to smithi20457400-32 2024-09-16T05:18:53.854 INFO:tasks.rados.rados.2.smithi204.stdout:2209: finishing write tid 3 to smithi20457400-32 2024-09-16T05:18:53.854 INFO:tasks.rados.rados.2.smithi204.stdout:2209: finishing write tid 4 to smithi20457400-32 2024-09-16T05:18:53.854 INFO:tasks.rados.rados.2.smithi204.stdout:2212: copy_from oid 13 from oid 26 current snap is 230 2024-09-16T05:18:53.854 INFO:tasks.rados.rados.2.smithi204.stdout:2209: finishing write tid 5 to smithi20457400-32 2024-09-16T05:18:53.854 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 789 (ObjNum 791 snap 230 seq_num 791) dirty exists 2024-09-16T05:18:53.855 INFO:tasks.rados.rados.2.smithi204.stdout:2209: left oid 32 (ObjNum 791 snap 230 seq_num 791) 2024-09-16T05:18:53.855 INFO:tasks.rados.rados.2.smithi204.stdout:2209: done (3 left) 2024-09-16T05:18:53.855 INFO:tasks.rados.rados.2.smithi204.stdout:2213: write oid 27 current snap is 230 2024-09-16T05:18:53.855 INFO:tasks.rados.rados.2.smithi204.stdout:2210: finishing write tid 1 to smithi20457400-43 2024-09-16T05:18:53.855 INFO:tasks.rados.rados.2.smithi204.stdout:2213: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2024-09-16T05:18:53.856 INFO:tasks.rados.rados.2.smithi204.stdout:2213: writing smithi20457400-27 from 527951 to 1101120 tid 1 2024-09-16T05:18:53.862 INFO:tasks.rados.rados.2.smithi204.stdout:2213: writing smithi20457400-27 from 1504456 to 2061389 tid 2 2024-09-16T05:18:53.868 INFO:tasks.rados.rados.2.smithi204.stdout:2213: writing smithi20457400-27 from 2743845 to 3211610 tid 3 2024-09-16T05:18:53.868 INFO:tasks.rados.rados.2.smithi204.stdout:2214: snap_remove snap 218 2024-09-16T05:18:53.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:53.969 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:53.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:54.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:53 smithi032 ceph-mon[27388]: pgmap v1035: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 322 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:18:54.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:53 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:18:54.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:53 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:18:54.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:53 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:18:54.017 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:53 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:18:54.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:54.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:54.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:53 smithi204 ceph-mon[30933]: pgmap v1035: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 322 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:18:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:53 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:18:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:53 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:18:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:53 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:18:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:53 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:18:54.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:54.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:54.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:53 smithi196 ceph-mon[30802]: pgmap v1035: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 322 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:18:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:53 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:18:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:53 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:18:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:53 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:18:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:53 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:18:54.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:54.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:54.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:54.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:54.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:54.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:54.531 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:18:54.565 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:18:54.565 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:18:54.566 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:18:54.566 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:18:54.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:54.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:54.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:54.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:54.733 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:54.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:54.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:54.832 INFO:tasks.rados.rados.2.smithi204.stdout:2210: finishing write tid 2 to smithi20457400-43 2024-09-16T05:18:54.832 INFO:tasks.rados.rados.2.smithi204.stdout:2210: finishing write tid 3 to smithi20457400-43 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2210: finishing write tid 4 to smithi20457400-43 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2210: finishing write tid 5 to smithi20457400-43 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 601 (ObjNum 792 snap 230 seq_num 792) dirty exists 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2210: left oid 43 (ObjNum 792 snap 230 seq_num 792) 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2211: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2212: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2213: finishing write tid 1 to smithi20457400-27 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2212: finishing copy_from to smithi20457400-13 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 1001 (ObjNum 770 snap 223 seq_num 770) dirty exists 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2213: finishing write tid 2 to smithi20457400-27 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2211: finishing copy_from to smithi20457400-16 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 369 (ObjNum 781 snap 227 seq_num 781) dirty exists 2024-09-16T05:18:54.833 INFO:tasks.rados.rados.2.smithi204.stdout:2213: finishing write tid 3 to smithi20457400-27 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2213: finishing write tid 4 to smithi20457400-27 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2213: finishing write tid 5 to smithi20457400-27 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 323 (ObjNum 793 snap 230 seq_num 793) dirty exists 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2213: left oid 27 (ObjNum 793 snap 230 seq_num 793) 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2210: done (4 left) 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2211: done (3 left) 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2212: done (2 left) 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2213: done (1 left) 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2214: done (0 left) 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2215: rollback oid 44 current snap is 230 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 224 2024-09-16T05:18:54.834 INFO:tasks.rados.rados.2.smithi204.stdout:2216: snap_create 2024-09-16T05:18:54.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:54.838 INFO:tasks.rados.rados.2.smithi204.stdout:2215: finishing rollback tid 0 to smithi20457400-44 2024-09-16T05:18:54.838 INFO:tasks.rados.rados.2.smithi204.stdout:2215: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:18:54.838 INFO:tasks.rados.rados.2.smithi204.stdout:2215: finishing rollback tid 2 to smithi20457400-44 2024-09-16T05:18:54.838 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 722 (ObjNum 692 snap 201 seq_num 692) dirty exists 2024-09-16T05:18:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:54.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:54.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:54.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:54.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:54.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:55.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:54 smithi032 ceph-mon[27388]: osdmap e584: 12 total, 12 up, 12 in 2024-09-16T05:18:55.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:54 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:18:55.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:54 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:18:55.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:54 smithi032 ceph-mon[27388]: 2.4c scrub starts 2024-09-16T05:18:55.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:54 smithi032 ceph-mon[27388]: 2.4c scrub ok 2024-09-16T05:18:55.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:55.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:55.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:54 smithi204 ceph-mon[30933]: osdmap e584: 12 total, 12 up, 12 in 2024-09-16T05:18:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:54 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:18:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:54 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:18:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:54 smithi204 ceph-mon[30933]: 2.4c scrub starts 2024-09-16T05:18:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:54 smithi204 ceph-mon[30933]: 2.4c scrub ok 2024-09-16T05:18:55.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:54 smithi196 ceph-mon[30802]: osdmap e584: 12 total, 12 up, 12 in 2024-09-16T05:18:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:54 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:18:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:54 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:18:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:54 smithi196 ceph-mon[30802]: 2.4c scrub starts 2024-09-16T05:18:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:54 smithi196 ceph-mon[30802]: 2.4c scrub ok 2024-09-16T05:18:55.244 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:55.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:55.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:55.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:55.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:55.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:55.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:18:55.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:55.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:55.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:55.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:55.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:55.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:18:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:18:55.765 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T05:18:55.769 INFO:tasks.rados.rados.2.smithi204.stdout:2215: done (1 left) 2024-09-16T05:18:55.769 INFO:tasks.rados.rados.2.smithi204.stdout:2216: done (0 left) 2024-09-16T05:18:55.769 INFO:tasks.rados.rados.2.smithi204.stdout:2217: snap_create 2024-09-16T05:18:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:55.787 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:55.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:55.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:55.914 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:55.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:56.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: pgmap v1038: 107 pgs: 107 active+clean; 326 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: osdmap e585: 12 total, 12 up, 12 in 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2928428988' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1038459297' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:18:56.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:18:56.017 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:56.017 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2928428988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:18:56.017 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1038459297' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:56.017 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:55 smithi032 ceph-mon[27388]: osdmap e586: 12 total, 12 up, 12 in 2024-09-16T05:18:56.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:56.043 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:56.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:56.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:56.168 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:56.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: pgmap v1038: 107 pgs: 107 active+clean; 326 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: osdmap e585: 12 total, 12 up, 12 in 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2928428988' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1038459297' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2928428988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:18:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1038459297' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:55 smithi204 ceph-mon[30933]: osdmap e586: 12 total, 12 up, 12 in 2024-09-16T05:18:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: pgmap v1038: 107 pgs: 107 active+clean; 326 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.2 MiB/s wr, 19 op/s 2024-09-16T05:18:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:18:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:18:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: osdmap e585: 12 total, 12 up, 12 in 2024-09-16T05:18:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2928428988' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:18:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1038459297' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:18:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:18:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:18:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:18:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:18:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:18:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2928428988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:18:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1038459297' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:18:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:55 smithi196 ceph-mon[30802]: osdmap e586: 12 total, 12 up, 12 in 2024-09-16T05:18:56.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:56.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:56.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:56.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:56.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:56.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:56.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:56.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:56.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:56.670 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:56.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:56.769 INFO:tasks.rados.rados.2.smithi204.stdout:2217: done (0 left) 2024-09-16T05:18:56.770 INFO:tasks.rados.rados.2.smithi204.stdout:2218: write (excl) oid 20 current snap is 232 2024-09-16T05:18:56.770 INFO:tasks.rados.rados.2.smithi204.stdout:2218: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-09-16T05:18:56.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:56.775 INFO:tasks.rados.rados.2.smithi204.stdout:2218: writing smithi20457400-20 from 544758 to 1193176 tid 1 2024-09-16T05:18:56.781 INFO:tasks.rados.rados.2.smithi204.stdout:2218: writing smithi20457400-20 from 1846585 to 2263529 tid 2 2024-09-16T05:18:56.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T05:18:56.788 INFO:tasks.rados.rados.2.smithi204.stdout:2218: writing smithi20457400-20 from 2771268 to 3344758 tid 3 2024-09-16T05:18:56.788 INFO:tasks.rados.rados.2.smithi204.stdout:2219: rollback oid 47 current snap is 232 2024-09-16T05:18:56.788 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 47 to 230 2024-09-16T05:18:56.788 INFO:tasks.rados.rados.2.smithi204.stdout:2218: finishing write tid 1 to smithi20457400-20 2024-09-16T05:18:56.788 INFO:tasks.rados.rados.2.smithi204.stdout:2218: finishing write tid 2 to smithi20457400-20 2024-09-16T05:18:56.789 INFO:tasks.rados.rados.2.smithi204.stdout:2220: delete oid 28 current snap is 232 2024-09-16T05:18:56.791 INFO:tasks.rados.rados.2.smithi204.stdout:2218: finishing write tid 3 to smithi20457400-20 2024-09-16T05:18:56.791 INFO:tasks.rados.rados.2.smithi204.stdout:2218: finishing write tid 4 to smithi20457400-20 2024-09-16T05:18:56.791 INFO:tasks.rados.rados.2.smithi204.stdout:2218: finishing write tid 5 to smithi20457400-20 2024-09-16T05:18:56.791 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 863 (ObjNum 794 snap 232 seq_num 794) dirty exists 2024-09-16T05:18:56.791 INFO:tasks.rados.rados.2.smithi204.stdout:2218: left oid 20 (ObjNum 794 snap 232 seq_num 794) 2024-09-16T05:18:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:2218: done (2 left) 2024-09-16T05:18:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:2220: done (1 left) 2024-09-16T05:18:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:2221: read oid 36 snap -1 2024-09-16T05:18:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:2221: expect (ObjNum 759 snap 221 seq_num 759) 2024-09-16T05:18:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:2222: rollback oid 40 current snap is 232 2024-09-16T05:18:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 40 to 222 2024-09-16T05:18:56.792 INFO:tasks.rados.rados.2.smithi204.stdout:2223: snap_remove snap 219 2024-09-16T05:18:56.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:56.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:56.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:56.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:56.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:57.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:57.046 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:57.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:57.082 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:57.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:57.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:57.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:57.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:57.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:57.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:57.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:57.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:57.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:57.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:57.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:57.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:57.561 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:115 2.a e112/107 2:580e4d0e:::smithi20457400-29:head v 112'6, mlcod=112'3)", 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 494.23971228099998, 2024-09-16T05:18:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0311933, 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:18:57.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334345+0000", 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334350+0000", 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.887e-06 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365213+0000", 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030863313999999999 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365220+0000", 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7870000000000004e-06 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365291+0000", 2024-09-16T05:18:57.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1397999999999996e-05 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365307+0000", 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5990000000000001e-05 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365507+0000", 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000199221 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.580 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365538+0000", 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1167000000000002e-05 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:127 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'9, mlcod=112'5)", 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 494.20433124300001, 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020225074999999999, 2024-09-16T05:18:57.581 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.582 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:18:57.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369726+0000", 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369728+0000", 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9410000000000001e-06 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389627+0000", 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019899751 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389632+0000", 2024-09-16T05:18:57.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-09-16T05:18:57.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389704+0000", 2024-09-16T05:18:57.586 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1418e-05 2024-09-16T05:18:57.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389718+0000", 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.433e-05 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389922+0000", 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020421600000000001 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.587 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389951+0000", 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8140000000000002e-05 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:133 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'10, mlcod=112'7)", 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:18:57.588 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 494.18479731500003, 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074633729999999997, 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:18:57.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389261+0000", 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.973e-06 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389613+0000", 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035121600000000001 2024-09-16T05:18:57.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395726+0000", 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006113851 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395731+0000", 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0359999999999997e-06 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.591 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395819+0000", 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7128999999999999e-05 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395836+0000", 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7768999999999999e-05 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396693+0000", 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085685999999999998 2024-09-16T05:18:57.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396723+0000", 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9539e-05 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:135 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'12, mlcod=112'8)", 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 494.17828634, 2024-09-16T05:18:57.593 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069274319999999999, 2024-09-16T05:18:57.594 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.594 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.594 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395771+0000", 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4499999999999997e-07 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395775+0000", 2024-09-16T05:18:57.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6509999999999999e-06 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402319+0000", 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065439469999999996 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402326+0000", 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5769999999999996e-06 2024-09-16T05:18:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402424+0000", 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8171000000000006e-05 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402439+0000", 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4844999999999999e-05 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402660+0000", 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022083700000000001 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402698+0000", 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7858999999999998e-05 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.599 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1133 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'206, mlcod=189'203)", 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 414.83983004599997, 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021225035, 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:18:57.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.85e-07 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734829+0000", 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060196800000000003 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734866+0000", 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7469000000000003e-05 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734881+0000", 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5058999999999999e-05 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734970+0000", 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9004000000000004e-05 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734998+0000", 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7606e-05 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755384+0000", 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02038587 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755452+0000", 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7873999999999994e-05 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1134 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'207, mlcod=189'203)", 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 414.83504492399999, 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016477325000000001, 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:18:57.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739014+0000", 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2819999999999999e-06 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739797+0000", 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078265599999999996 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739844+0000", 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7608000000000002e-05 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739854+0000", 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4239999999999999e-06 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739935+0000", 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1593000000000003e-05 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739963+0000", 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7492e-05 2024-09-16T05:18:57.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755459+0000", 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015496565 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755489+0000", 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9705000000000001e-05 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1135 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'208, mlcod=189'203)", 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 414.83062208699999, 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012094011, 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:18:57.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743436+0000", 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.584e-06 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743535+0000", 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9110000000000005e-05 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743568+0000", 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2335000000000001e-05 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743576+0000", 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.191e-06 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743646+0000", 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0481000000000005e-05 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743676+0000", 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9620000000000001e-05 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755492+0000", 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011815888 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755529+0000", 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6801999999999998e-05 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1136 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'209, mlcod=189'203)", 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 414.83001452899998, 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011564019, 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744043+0000", 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3600000000000003e-07 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744049+0000", 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8250000000000001e-06 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744069+0000", 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0460000000000001e-05 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744074+0000", 2024-09-16T05:18:57.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4730000000000002e-06 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744119+0000", 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744143+0000", 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4085999999999999e-05 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755570+0000", 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011427299 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755606+0000", 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146000000000003e-05 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:39 2.29 e259/253 2:94d33c56:::smithi20457400-8:4c v 259'246, mlcod=254'242)", 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 346.89581299399998, 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069169729999999999, 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678244+0000", 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7399999999999999e-07 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678251+0000", 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1080000000000004e-06 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678321+0000", 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9762000000000001e-05 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.678330+0000", 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0329999999999997e-06 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.679306+0000", 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00097581800000000004 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.680633+0000", 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013273779999999999 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.685128+0000", 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044947850000000003 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:10.685161+0000", 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2814999999999997e-05 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.624 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2777 2.48 e359/322 2:12e92f6c:::smithi20457400-1:head v 359'155, mlcod=0'0)", 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 228.822718366, 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007754843, 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751339+0000", 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5400000000000002e-07 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755560+0000", 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004221341 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755638+0000", 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8292000000000002e-05 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755652+0000", 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3702e-05 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.628 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755795+0000", 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014327200000000001 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755823+0000", 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7668e-05 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.759023+0000", 2024-09-16T05:18:57.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003199959 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.759093+0000", 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0355e-05 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:606 2.48 e365/322 2:12e92f6c:::smithi20457400-1:79 v 365'161, mlcod=362'157)", 2024-09-16T05:18:57.630 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 222.50292119299999, 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64187130800000003, 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071139+0000", 2024-09-16T05:18:57.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5609999999999999e-06 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712578+0000", 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64143852199999996 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712582+0000", 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-06 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712651+0000", 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9102000000000003e-05 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712670+0000", 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9165999999999999e-05 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712968+0000", 2024-09-16T05:18:57.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029722899999999998 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.713007+0000", 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9125999999999998e-05 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:614 2.48 e366/322 2:12e92f6c:::smithi20457400-1:79 v 366'163, mlcod=364'159)", 2024-09-16T05:18:57.635 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 221.85349396300001, 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11945663500000001, 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9699999999999996e-07 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720568+0000", 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6870000000000002e-06 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839371+0000", 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11880265399999999 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839384+0000", 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3156000000000001e-05 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839531+0000", 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014728399999999999 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839575+0000", 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4223e-05 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839981+0000", 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040524499999999999 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.840019+0000", 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8788999999999997e-05 2024-09-16T05:18:57.640 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3235 2.29 e406/253 2:9662c29c:::smithi20457400-9:head v 406'547, mlcod=402'543)", 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 181.843806457, 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081876009999999992, 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.641 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730252+0000", 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.857e-06 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734729+0000", 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044764510000000002 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734790+0000", 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1465999999999997e-05 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734803+0000", 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2729e-05 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734890+0000", 2024-09-16T05:18:57.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7708999999999997e-05 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734938+0000", 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7485999999999999e-05 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.738369+0000", 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034306649999999998 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.738438+0000", 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9238000000000004e-05 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3412 2.3e e424/296 2:7e4e65a4:::smithi20457400-4:head v 424'357, mlcod=378'353)", 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:18:57.646 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 163.636590829, 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007702319, 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9000000000000001e-07 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942005+0000", 2024-09-16T05:18:57.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045392510000000002 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942060+0000", 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4716000000000003e-05 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942070+0000", 2024-09-16T05:18:57.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0078000000000001e-05 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942135+0000", 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4853999999999999e-05 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942159+0000", 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3839999999999999e-05 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.945112+0000", 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029533789999999999 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.945168+0000", 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6011000000000003e-05 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4629 2.1e 2:7b642727:::smithi20457400-40:head [copy-from ver 511] snapc bf=[bf,be] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 75.821532445000003, 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01647931, 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:57.652 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4629 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:18:57.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752526+0000", 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.894e-06 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752536+0000", 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752555+0000", 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8907999999999999e-05 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752564+0000", 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752638+0000", 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3343999999999994e-05 2024-09-16T05:18:57.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752646+0000", 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7690000000000002e-06 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754646+0000", 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001999331 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754740+0000", 2024-09-16T05:18:57.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3778999999999998e-05 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.764207+0000", 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094672899999999997 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768754+0000", 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045470199999999997 2024-09-16T05:18:57.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768860+0000", 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010586600000000001 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768948+0000", 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8503000000000003e-05 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.769004+0000", 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5362000000000003e-05 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4630 2.1e 2:7b642727:::smithi20457400-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 75.821474695000006, 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015483190000000001, 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4630 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.34e-07 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752584+0000", 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.956e-06 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752591+0000", 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2580000000000001e-06 2024-09-16T05:18:57.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752594+0000", 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6830000000000001e-06 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752703+0000", 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108818 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:18:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752708+0000", 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.561e-06 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754913+0000", 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022055680000000002 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754965+0000", 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2222000000000002e-05 2024-09-16T05:18:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767827+0000", 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012861097 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767929+0000", 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010275799999999999 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.666 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768065+0000", 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013593500000000001 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:859 2.3e e561/296 2:7e4e65a4:::smithi20457400-4:d0 v 561'447, mlcod=557'443)", 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.809111587, 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007143066, 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:18:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2300000000000002e-07 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764950+0000", 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771606+0000", 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006656103 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771614+0000", 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.712e-06 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771652+0000", 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8497000000000001e-05 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771668+0000", 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5829e-05 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.772021+0000", 2024-09-16T05:18:57.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035285500000000001 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.772088+0000", 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7297000000000003e-05 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5275 2.a e561/320 2:5089ebe1:::smithi20457400-36:head v 561'529, mlcod=526'525)", 2024-09-16T05:18:57.672 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 26.665807493999999, 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069804639999999996, 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:18:57.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908251+0000", 2024-09-16T05:18:57.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4360000000000001e-06 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912064+0000", 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038136189999999999 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912103+0000", 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8689999999999997e-05 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912115+0000", 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.19e-05 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912182+0000", 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7565999999999997e-05 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912205+0000", 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2220000000000001e-05 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.915180+0000", 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002975536 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.915230+0000", 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9496999999999997e-05 2024-09-16T05:18:57.677 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:142 2.1e e571/291 ondisk, result = 0)", 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 15.809898540000001, 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55541789799999997, 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764159+0000", 2024-09-16T05:18:57.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764162+0000", 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599999999999998e-06 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319278+0000", 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55511595899999999 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319296+0000", 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7900000000000001e-05 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319440+0000", 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143175 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319450+0000", 2024-09-16T05:18:57.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0705e-05 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319576+0000", 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012584299999999999 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5587 2.34 e583/278 2:2d3f13f8:::smithi20457400-7:head v 583'466, mlcod=583'463)", 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:18:57.682 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 4.7669521889999995, 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14534661199999999, 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:18:57.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807109+0000", 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2030000000000002e-06 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951917+0000", 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14480800699999999 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:18:57.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951927+0000", 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0115000000000001e-05 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951990+0000", 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2689999999999998e-05 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952011+0000", 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.084e-05 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:18:57.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952413+0000", 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040190199999999999 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952451+0000", 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8302e-05 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:18:57.688 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:18:57.688 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:18:57.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:57.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:18:57.774 INFO:tasks.rados.rados.2.smithi204.stdout:2223: done (3 left) 2024-09-16T05:18:57.774 INFO:tasks.rados.rados.2.smithi204.stdout:2224: write (excl) oid 29 current snap is 232 2024-09-16T05:18:57.774 INFO:tasks.rados.rados.2.smithi204.stdout:2219: finishing rollback tid 0 to smithi20457400-47 2024-09-16T05:18:57.774 INFO:tasks.rados.rados.2.smithi204.stdout:2224: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-09-16T05:18:57.782 INFO:tasks.rados.rados.2.smithi204.stdout:2224: writing smithi20457400-29 from 578372 to 1377288 tid 1 2024-09-16T05:18:57.789 INFO:tasks.rados.rados.2.smithi204.stdout:2224: writing smithi20457400-29 from 1963549 to 2384162 tid 2 2024-09-16T05:18:57.794 INFO:tasks.rados.rados.2.smithi204.stdout:2224: writing smithi20457400-29 from 2826114 to 3244048 tid 3 2024-09-16T05:18:57.794 INFO:tasks.rados.rados.2.smithi204.stdout:2225: snap_create 2024-09-16T05:18:57.794 INFO:tasks.rados.rados.2.smithi204.stdout:2219: finishing rollback tid 1 to smithi20457400-47 2024-09-16T05:18:57.794 INFO:tasks.rados.rados.2.smithi204.stdout:2219: finishing rollback tid 2 to smithi20457400-47 2024-09-16T05:18:57.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1010 (ObjNum 774 snap 225 seq_num 774) dirty exists 2024-09-16T05:18:57.795 INFO:tasks.rados.rados.2.smithi204.stdout:2222: finishing rollback tid 0 to smithi20457400-40 2024-09-16T05:18:57.795 INFO:tasks.rados.rados.2.smithi204.stdout:2222: finishing rollback tid 1 to smithi20457400-40 2024-09-16T05:18:57.795 INFO:tasks.rados.rados.2.smithi204.stdout:2222: finishing rollback tid 2 to smithi20457400-40 2024-09-16T05:18:57.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 767 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:18:57.795 INFO:tasks.rados.rados.2.smithi204.stdout:2221: expect (ObjNum 759 snap 221 seq_num 759) 2024-09-16T05:18:57.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:57.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:57.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:57 smithi032 ceph-mon[27388]: pgmap v1041: 107 pgs: 107 active+clean; 338 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 11 op/s 2024-09-16T05:18:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:57 smithi032 ceph-mon[27388]: osdmap e587: 12 total, 12 up, 12 in 2024-09-16T05:18:57.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:18:57.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:57.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:18:57.995 INFO:tasks.rados.rados.2.smithi204.stdout:2224: finishing write tid 1 to smithi20457400-29 2024-09-16T05:18:57.995 INFO:tasks.rados.rados.2.smithi204.stdout:2224: finishing write tid 2 to smithi20457400-29 2024-09-16T05:18:57.995 INFO:tasks.rados.rados.2.smithi204.stdout:2224: finishing write tid 3 to smithi20457400-29 2024-09-16T05:18:57.995 INFO:tasks.rados.rados.2.smithi204.stdout:2224: finishing write tid 4 to smithi20457400-29 2024-09-16T05:18:57.995 INFO:tasks.rados.rados.2.smithi204.stdout:2224: finishing write tid 5 to smithi20457400-29 2024-09-16T05:18:57.995 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 801 (ObjNum 796 snap 232 seq_num 796) dirty exists 2024-09-16T05:18:57.995 INFO:tasks.rados.rados.2.smithi204.stdout:2224: left oid 29 (ObjNum 796 snap 232 seq_num 796) 2024-09-16T05:18:58.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:58.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:58.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:57 smithi204 ceph-mon[30933]: pgmap v1041: 107 pgs: 107 active+clean; 338 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 11 op/s 2024-09-16T05:18:58.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:57 smithi204 ceph-mon[30933]: osdmap e587: 12 total, 12 up, 12 in 2024-09-16T05:18:58.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:58.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:58.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:57 smithi196 ceph-mon[30802]: pgmap v1041: 107 pgs: 107 active+clean; 338 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 11 op/s 2024-09-16T05:18:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:57 smithi196 ceph-mon[30802]: osdmap e587: 12 total, 12 up, 12 in 2024-09-16T05:18:58.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:18:58.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:58.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:58.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:18:58.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:58.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:58.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:18:58.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:58.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:18:58.666 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:18:58.667 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:18:58.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:58.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:18:58.688 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:18:58.708 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:18:58.708 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:18:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:58.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:58.729 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:18:58.729 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:18:58.753 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:18:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:18:58.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:18:58.775 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:18:58.775 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T05:18:58.779 INFO:tasks.rados.rados.2.smithi204.stdout:2219: done (4 left) 2024-09-16T05:18:58.779 INFO:tasks.rados.rados.2.smithi204.stdout:2221: done (3 left) 2024-09-16T05:18:58.780 INFO:tasks.rados.rados.2.smithi204.stdout:2222: done (2 left) 2024-09-16T05:18:58.780 INFO:tasks.rados.rados.2.smithi204.stdout:2224: done (1 left) 2024-09-16T05:18:58.780 INFO:tasks.rados.rados.2.smithi204.stdout:2225: done (0 left) 2024-09-16T05:18:58.780 INFO:tasks.rados.rados.2.smithi204.stdout:2226: snap_remove snap 224 2024-09-16T05:18:58.797 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:18:58.797 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:18:58.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:58.820 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:18:58.820 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:18:58.821 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:58 smithi032 ceph-mon[27388]: osdmap e588: 12 total, 12 up, 12 in 2024-09-16T05:18:58.821 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/866586381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:58.821 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1383304856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:18:58.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:58.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:58.842 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:18:58.842 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:18:58.863 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:18:58.863 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:18:58.885 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:18:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:18:58.906 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:18:58.907 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:18:58.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:18:58.956 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:58.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:18:59.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:18:59.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:18:59.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:59.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:18:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:58 smithi204 ceph-mon[30933]: osdmap e588: 12 total, 12 up, 12 in 2024-09-16T05:18:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/866586381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1383304856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:18:59.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:59.210 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:59.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:58 smithi196 ceph-mon[30802]: osdmap e588: 12 total, 12 up, 12 in 2024-09-16T05:18:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/866586381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:18:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1383304856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:18:59.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:18:59.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:59.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:18:59.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:18:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:59.461 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:18:59.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:18:59.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:18:59.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:59.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:18:59.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:18:59.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:18:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:59.718 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:59.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:18:59.782 INFO:tasks.rados.rados.2.smithi204.stdout:2226: done (0 left) 2024-09-16T05:18:59.782 INFO:tasks.rados.rados.2.smithi204.stdout:2227: snap_create 2024-09-16T05:18:59.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:18:59.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:18:59.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:59.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:18:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:59 smithi032 ceph-mon[27388]: pgmap v1044: 107 pgs: 107 active+clean; 345 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2024-09-16T05:18:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1383304856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:18:59.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:18:59 smithi032 ceph-mon[27388]: osdmap e589: 12 total, 12 up, 12 in 2024-09-16T05:18:59.930 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:18:59.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:18:59.970 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:18:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:18:59.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:18:59.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:59 smithi196 ceph-mon[30802]: pgmap v1044: 107 pgs: 107 active+clean; 345 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2024-09-16T05:18:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1383304856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:18:59.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:18:59 smithi196 ceph-mon[30802]: osdmap e589: 12 total, 12 up, 12 in 2024-09-16T05:19:00.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:00.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:59 smithi204 ceph-mon[30933]: pgmap v1044: 107 pgs: 107 active+clean; 345 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2024-09-16T05:19:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1383304856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:19:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:18:59 smithi204 ceph-mon[30933]: osdmap e589: 12 total, 12 up, 12 in 2024-09-16T05:19:00.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:00.219 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:00.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:00.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:00.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:00.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:00.436 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:00.436 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:00.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:00.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:00.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:00.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:00.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:00.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:00.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:00.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T05:19:00.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:00.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:00.771 INFO:tasks.rados.rados.2.smithi204.stdout:2227: done (0 left) 2024-09-16T05:19:00.771 INFO:tasks.rados.rados.2.smithi204.stdout:2228: read oid 12 snap -1 2024-09-16T05:19:00.771 INFO:tasks.rados.rados.2.smithi204.stdout:2228: expect deleted 2024-09-16T05:19:00.771 INFO:tasks.rados.rados.2.smithi204.stdout:2229: read oid 26 snap 231 2024-09-16T05:19:00.771 INFO:tasks.rados.rados.2.smithi204.stdout:2229: expect (ObjNum 770 snap 223 seq_num 770) 2024-09-16T05:19:00.771 INFO:tasks.rados.rados.2.smithi204.stdout:2230: rollback oid 15 current snap is 234 2024-09-16T05:19:00.771 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 15 to 233 2024-09-16T05:19:00.772 INFO:tasks.rados.rados.2.smithi204.stdout:2231: write oid 19 current snap is 234 2024-09-16T05:19:00.772 INFO:tasks.rados.rados.2.smithi204.stdout:2231: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2024-09-16T05:19:00.776 INFO:tasks.rados.rados.2.smithi204.stdout:2231: writing smithi20457400-19 from 595179 to 1069344 tid 1 2024-09-16T05:19:00.783 INFO:tasks.rados.rados.2.smithi204.stdout:2231: writing smithi20457400-19 from 1622031 to 2186302 tid 2 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:2231: writing smithi20457400-19 from 2737184 to 3366390 tid 3 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:2232: rollback oid 35 current snap is 234 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 35 to 223 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:2230: finishing rollback tid 1 to smithi20457400-15 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 701 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:2230: done (4 left) 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:2233: rollback oid 14 current snap is 234 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 233 2024-09-16T05:19:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:2229: expect (ObjNum 770 snap 223 seq_num 770) 2024-09-16T05:19:00.794 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:00.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:00.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:00.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:00.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:00.890 INFO:tasks.rados.rados.2.smithi204.stdout:2231: finishing write tid 1 to smithi20457400-19 2024-09-16T05:19:00.890 INFO:tasks.rados.rados.2.smithi204.stdout:2231: finishing write tid 2 to smithi20457400-19 2024-09-16T05:19:00.890 INFO:tasks.rados.rados.2.smithi204.stdout:2232: finishing rollback tid 1 to smithi20457400-35 2024-09-16T05:19:00.890 INFO:tasks.rados.rados.2.smithi204.stdout:2231: finishing write tid 3 to smithi20457400-19 2024-09-16T05:19:00.890 INFO:tasks.rados.rados.2.smithi204.stdout:2231: finishing write tid 4 to smithi20457400-19 2024-09-16T05:19:00.890 INFO:tasks.rados.rados.2.smithi204.stdout:2232: finishing rollback tid 2 to smithi20457400-35 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 843 (ObjNum 755 snap 220 seq_num 755) dirty exists 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2231: finishing write tid 5 to smithi20457400-19 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 331 (ObjNum 797 snap 234 seq_num 797) dirty exists 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2231: left oid 19 (ObjNum 797 snap 234 seq_num 797) 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2233: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2228: done (4 left) 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2229: done (3 left) 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2231: done (2 left) 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2232: done (1 left) 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2234: write (excl) oid 37 current snap is 234 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2233: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:19:00.891 INFO:tasks.rados.rados.2.smithi204.stdout:2234: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2024-09-16T05:19:00.896 INFO:tasks.rados.rados.2.smithi204.stdout:2234: writing smithi20457400-37 from 611986 to 1161400 tid 1 2024-09-16T05:19:00.901 INFO:tasks.rados.rados.2.smithi204.stdout:2234: writing smithi20457400-37 from 1564160 to 1988442 tid 2 2024-09-16T05:19:00.908 INFO:tasks.rados.rados.2.smithi204.stdout:2234: writing smithi20457400-37 from 2764607 to 3205085 tid 3 2024-09-16T05:19:00.908 INFO:tasks.rados.rados.2.smithi204.stdout:2235: delete oid 44 current snap is 234 2024-09-16T05:19:00.908 INFO:tasks.rados.rados.2.smithi204.stdout:2233: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:19:00.908 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1021 (ObjNum 774 snap 225 seq_num 774) dirty exists 2024-09-16T05:19:00.908 INFO:tasks.rados.rados.2.smithi204.stdout:2234: finishing write tid 1 to smithi20457400-37 2024-09-16T05:19:00.908 INFO:tasks.rados.rados.2.smithi204.stdout:2234: finishing write tid 2 to smithi20457400-37 2024-09-16T05:19:00.909 INFO:tasks.rados.rados.2.smithi204.stdout:2234: finishing write tid 3 to smithi20457400-37 2024-09-16T05:19:00.910 INFO:tasks.rados.rados.2.smithi204.stdout:2233: done (2 left) 2024-09-16T05:19:00.910 INFO:tasks.rados.rados.2.smithi204.stdout:2235: done (1 left) 2024-09-16T05:19:00.910 INFO:tasks.rados.rados.2.smithi204.stdout:2236: snap_remove snap 213 2024-09-16T05:19:00.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:00.965 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:00.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:00.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:00 smithi196 ceph-mon[30802]: osdmap e590: 12 total, 12 up, 12 in 2024-09-16T05:19:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:00 smithi196 ceph-mon[30802]: osdmap e591: 12 total, 12 up, 12 in 2024-09-16T05:19:01.005 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:00 smithi032 ceph-mon[27388]: osdmap e590: 12 total, 12 up, 12 in 2024-09-16T05:19:01.005 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:00 smithi032 ceph-mon[27388]: osdmap e591: 12 total, 12 up, 12 in 2024-09-16T05:19:01.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:01.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:01.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:01.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:19:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:00 smithi204 ceph-mon[30933]: osdmap e590: 12 total, 12 up, 12 in 2024-09-16T05:19:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:00 smithi204 ceph-mon[30933]: osdmap e591: 12 total, 12 up, 12 in 2024-09-16T05:19:01.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:01.212 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:01.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:01.273 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:01.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:01.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:01.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:01.390 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:01.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:01.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:01.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:01.587 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:01.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:01.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:01.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:01.795 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:19:01.795 INFO:tasks.rados.rados.2.smithi204.stdout:2236: done (1 left) 2024-09-16T05:19:01.796 INFO:tasks.rados.rados.2.smithi204.stdout:2237: write (excl) oid 44 current snap is 234 2024-09-16T05:19:01.796 INFO:tasks.rados.rados.2.smithi204.stdout:2237: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2024-09-16T05:19:01.801 INFO:tasks.rados.rados.2.smithi204.stdout:2237: writing smithi20457400-44 from 645600 to 1061865 tid 1 2024-09-16T05:19:01.810 INFO:tasks.rados.rados.2.smithi204.stdout:2237: writing smithi20457400-44 from 1797477 to 2509075 tid 2 2024-09-16T05:19:01.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:01.816 INFO:tasks.rados.rados.2.smithi204.stdout:2237: writing smithi20457400-44 from 3219453 to 3445600 tid 3 2024-09-16T05:19:01.816 INFO:tasks.rados.rados.2.smithi204.stdout:2238: delete oid 21 current snap is 234 2024-09-16T05:19:01.816 INFO:tasks.rados.rados.2.smithi204.stdout:2234: finishing write tid 4 to smithi20457400-37 2024-09-16T05:19:01.816 INFO:tasks.rados.rados.2.smithi204.stdout:2234: finishing write tid 5 to smithi20457400-37 2024-09-16T05:19:01.817 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1014 (ObjNum 798 snap 234 seq_num 798) dirty exists 2024-09-16T05:19:01.817 INFO:tasks.rados.rados.2.smithi204.stdout:2234: left oid 37 (ObjNum 798 snap 234 seq_num 798) 2024-09-16T05:19:01.817 INFO:tasks.rados.rados.2.smithi204.stdout:2237: finishing write tid 1 to smithi20457400-44 2024-09-16T05:19:01.817 INFO:tasks.rados.rados.2.smithi204.stdout:2237: finishing write tid 2 to smithi20457400-44 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:2237: finishing write tid 3 to smithi20457400-44 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:2237: finishing write tid 4 to smithi20457400-44 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:2237: finishing write tid 5 to smithi20457400-44 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 727 (ObjNum 800 snap 234 seq_num 800) dirty exists 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:2237: left oid 44 (ObjNum 800 snap 234 seq_num 800) 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:2234: done (2 left) 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:2237: done (1 left) 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:2238: done (0 left) 2024-09-16T05:19:01.818 INFO:tasks.rados.rados.2.smithi204.stdout:2239: snap_create 2024-09-16T05:19:01.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:01.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:01.880 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:01.880 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:01 smithi032 ceph-mon[27388]: pgmap v1047: 107 pgs: 107 active+clean; 349 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2024-09-16T05:19:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:01 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:19:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:01 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:19:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/232591080' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:01.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:01 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:01.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:01.960 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:01.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:02.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:02.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:02.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:02.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:02.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:19:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:01 smithi204 ceph-mon[30933]: pgmap v1047: 107 pgs: 107 active+clean; 349 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2024-09-16T05:19:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:01 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:19:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:01 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:19:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/232591080' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:01 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:02.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:02.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:02.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:02.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:01 smithi196 ceph-mon[30802]: pgmap v1047: 107 pgs: 107 active+clean; 349 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2024-09-16T05:19:02.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:01 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:19:02.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:01 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:19:02.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/232591080' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:02.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:01 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:02.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:02.333 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:02.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:02.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:02.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:02.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:02.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:19:02.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:02.584 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:02.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:02.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:02.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:02.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:02.745 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:02.803 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T05:19:02.806 INFO:tasks.rados.rados.2.smithi204.stdout:2239: done (0 left) 2024-09-16T05:19:02.806 INFO:tasks.rados.rados.2.smithi204.stdout:2240: copy_from oid 17 from oid 19 current snap is 235 2024-09-16T05:19:02.806 INFO:tasks.rados.rados.2.smithi204.stdout:2241: snap_create 2024-09-16T05:19:02.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:02.821 INFO:tasks.rados.rados.2.smithi204.stdout:2240: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:19:02.825 INFO:tasks.rados.rados.2.smithi204.stdout:2240: finishing copy_from to smithi20457400-17 2024-09-16T05:19:02.825 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 331 (ObjNum 797 snap 234 seq_num 797) dirty exists 2024-09-16T05:19:02.834 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:02.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:02.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:02.961 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:02.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:03.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:02 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:03.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:02 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:03.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:02 smithi196 ceph-mon[30802]: osdmap e592: 12 total, 12 up, 12 in 2024-09-16T05:19:03.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2734094272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:19:03.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:03.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:03.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:03.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:02 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:03.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:02 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:03.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:02 smithi032 ceph-mon[27388]: osdmap e592: 12 total, 12 up, 12 in 2024-09-16T05:19:03.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2734094272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:19:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:02 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:02 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:02 smithi204 ceph-mon[30933]: osdmap e592: 12 total, 12 up, 12 in 2024-09-16T05:19:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2734094272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:19:03.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:03.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:03.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:03.296 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:03.296 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:03.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:03.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:03.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:03.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:03.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:03.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:03.556 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:03.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:03.577 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:03.577 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:03.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:03.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:03.598 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:03.599 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:03.622 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:03.622 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:03.643 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:03.643 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:03.665 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:03.665 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:03.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:03.687 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:03.688 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:03.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T05:19:03.707 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:03.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:03.709 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:03.709 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:03.730 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:03.731 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:03.752 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:03.752 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:03.774 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:03.774 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:03.796 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:03.796 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:03.809 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:03.815 INFO:tasks.rados.rados.2.smithi204.stdout:2240: done (1 left) 2024-09-16T05:19:03.815 INFO:tasks.rados.rados.2.smithi204.stdout:2241: done (0 left) 2024-09-16T05:19:03.815 INFO:tasks.rados.rados.2.smithi204.stdout:2242: snap_remove snap 214 2024-09-16T05:19:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:03.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:03.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:03.900 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:03 smithi196 ceph-mon[30802]: pgmap v1050: 107 pgs: 107 active+clean; 350 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:19:03.900 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2734094272' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:19:03.900 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:03 smithi196 ceph-mon[30802]: osdmap e593: 12 total, 12 up, 12 in 2024-09-16T05:19:03.900 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2946612637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:03.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:19:03.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:03.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:04.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:04.080 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:04.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:04.167 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:03 smithi032 ceph-mon[27388]: pgmap v1050: 107 pgs: 107 active+clean; 350 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:19:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2734094272' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:19:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:03 smithi032 ceph-mon[27388]: osdmap e593: 12 total, 12 up, 12 in 2024-09-16T05:19:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2946612637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:04.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:03 smithi204 ceph-mon[30933]: pgmap v1050: 107 pgs: 107 active+clean; 350 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:19:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2734094272' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:19:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:03 smithi204 ceph-mon[30933]: osdmap e593: 12 total, 12 up, 12 in 2024-09-16T05:19:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2946612637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:04.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:04.208 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:04.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:04.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:04.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:04.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:04.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:04.462 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:04.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:04.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:19:04.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:04.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:04.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:04.686 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:19:04.686 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:19:04.686 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:19:04.686 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:19:04.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:04.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:04.729 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:04.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:04.832 INFO:tasks.rados.rados.2.smithi204.stdout:2242: done (0 left) 2024-09-16T05:19:04.832 INFO:tasks.rados.rados.2.smithi204.stdout:2243: read oid 36 snap -1 2024-09-16T05:19:04.832 INFO:tasks.rados.rados.2.smithi204.stdout:2243: expect (ObjNum 759 snap 221 seq_num 759) 2024-09-16T05:19:04.833 INFO:tasks.rados.rados.2.smithi204.stdout:2244: copy_from oid 48 from oid 40 current snap is 236 2024-09-16T05:19:04.833 INFO:tasks.rados.rados.2.smithi204.stdout:2245: copy_from oid 37 from oid 12 current snap is 236 2024-09-16T05:19:04.833 INFO:tasks.rados.rados.2.smithi204.stdout:2246: read oid 22 snap -1 2024-09-16T05:19:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2246: expect (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:19:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2247: read oid 45 snap -1 2024-09-16T05:19:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2247: expect (ObjNum 781 snap 227 seq_num 781) 2024-09-16T05:19:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2248: copy_from oid 4 from oid 31 current snap is 236 2024-09-16T05:19:04.835 INFO:tasks.rados.rados.2.smithi204.stdout:2249: delete oid 21 current snap is 236 2024-09-16T05:19:04.835 INFO:tasks.rados.rados.2.smithi204.stdout:2249: done (6 left) 2024-09-16T05:19:04.835 INFO:tasks.rados.rados.2.smithi204.stdout:2250: copy_from oid 29 from oid 49 current snap is 236 2024-09-16T05:19:04.835 INFO:tasks.rados.rados.2.smithi204.stdout:2251: snap_create 2024-09-16T05:19:04.836 INFO:tasks.rados.rados.2.smithi204.stdout:2245: finishing copy_from to smithi20457400-37 2024-09-16T05:19:04.836 INFO:tasks.rados.rados.2.smithi204.stdout:2245: got expected ENOENT (src dne) 2024-09-16T05:19:04.837 INFO:tasks.rados.rados.2.smithi204.stdout:2245: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:19:04.837 INFO:tasks.rados.rados.2.smithi204.stdout:2243: expect (ObjNum 759 snap 221 seq_num 759) 2024-09-16T05:19:04.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:04.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:04.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:04.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:04.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:04.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:04 smithi196 ceph-mon[30802]: osdmap e594: 12 total, 12 up, 12 in 2024-09-16T05:19:05.042 INFO:tasks.rados.rados.2.smithi204.stdout:2248: finishing copy_from to smithi20457400-4 2024-09-16T05:19:05.043 INFO:tasks.rados.rados.2.smithi204.stdout:2248: got expected ENOENT (src dne) 2024-09-16T05:19:05.043 INFO:tasks.rados.rados.2.smithi204.stdout:2248: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:19:05.043 INFO:tasks.rados.rados.2.smithi204.stdout:2247: expect (ObjNum 781 snap 227 seq_num 781) 2024-09-16T05:19:05.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:05.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:05.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:05.127 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:04 smithi032 ceph-mon[27388]: osdmap e594: 12 total, 12 up, 12 in 2024-09-16T05:19:05.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:04 smithi204 ceph-mon[30933]: osdmap e594: 12 total, 12 up, 12 in 2024-09-16T05:19:05.205 INFO:tasks.rados.rados.2.smithi204.stdout:2244: finishing copy_from racing read to smithi20457400-48 2024-09-16T05:19:05.205 INFO:tasks.rados.rados.2.smithi204.stdout:2244: finishing copy_from to smithi20457400-48 2024-09-16T05:19:05.205 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 767 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:19:05.205 INFO:tasks.rados.rados.2.smithi204.stdout:2250: finishing copy_from racing read to smithi20457400-29 2024-09-16T05:19:05.205 INFO:tasks.rados.rados.2.smithi204.stdout:2246: expect (ObjNum 592 snap 180 seq_num 592) 2024-09-16T05:19:05.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:05.225 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:05.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:05.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:05.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:19:05.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:05.352 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:05.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:05.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:05.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:05.477 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:05.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:05.477 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:05.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:05.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:05.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:05.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:05.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:05.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:05.728 INFO:tasks.rados.rados.2.smithi204.stdout:2250: finishing copy_from to smithi20457400-29 2024-09-16T05:19:05.728 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 595 (ObjNum 773 snap 225 seq_num 773) dirty exists 2024-09-16T05:19:05.767 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2243: done (7 left) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2244: done (6 left) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2245: done (5 left) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2246: done (4 left) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2247: done (3 left) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2248: done (2 left) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2250: done (1 left) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2251: done (0 left) 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2252: write oid 6 current snap is 237 2024-09-16T05:19:05.773 INFO:tasks.rados.rados.2.smithi204.stdout:2252: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2024-09-16T05:19:05.780 INFO:tasks.rados.rados.2.smithi204.stdout:2252: writing smithi20457400-6 from 696021 to 1338033 tid 1 2024-09-16T05:19:05.786 INFO:tasks.rados.rados.2.smithi204.stdout:2252: writing smithi20457400-6 from 1856570 to 2431848 tid 2 2024-09-16T05:19:05.792 INFO:tasks.rados.rados.2.smithi204.stdout:2252: writing smithi20457400-6 from 3185369 to 3496021 tid 3 2024-09-16T05:19:05.792 INFO:tasks.rados.rados.2.smithi204.stdout:2253: read oid 46 snap -1 2024-09-16T05:19:05.792 INFO:tasks.rados.rados.2.smithi204.stdout:2253: expect (ObjNum 762 snap 223 seq_num 762) 2024-09-16T05:19:05.792 INFO:tasks.rados.rados.2.smithi204.stdout:2254: snap_remove snap 223 2024-09-16T05:19:05.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:05.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:05.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:05.848 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:05 smithi196 ceph-mon[30802]: pgmap v1053: 107 pgs: 107 active+clean; 344 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 738 KiB/s wr, 6 op/s 2024-09-16T05:19:05.848 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:05 smithi196 ceph-mon[30802]: osdmap e595: 12 total, 12 up, 12 in 2024-09-16T05:19:05.848 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1820738429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:19:05.848 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1820738429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:19:05.848 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:05 smithi196 ceph-mon[30802]: osdmap e596: 12 total, 12 up, 12 in 2024-09-16T05:19:05.948 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:19:05.948 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:19:05.948 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:19:05.948 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:19:05.948 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:19:05.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:05.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:05.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:06.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:06.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:05 smithi032 ceph-mon[27388]: pgmap v1053: 107 pgs: 107 active+clean; 344 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 738 KiB/s wr, 6 op/s 2024-09-16T05:19:06.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:05 smithi032 ceph-mon[27388]: osdmap e595: 12 total, 12 up, 12 in 2024-09-16T05:19:06.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1820738429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:19:06.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1820738429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:19:06.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:05 smithi032 ceph-mon[27388]: osdmap e596: 12 total, 12 up, 12 in 2024-09-16T05:19:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:06.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:06.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:06.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:05 smithi204 ceph-mon[30933]: pgmap v1053: 107 pgs: 107 active+clean; 344 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 738 KiB/s wr, 6 op/s 2024-09-16T05:19:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:05 smithi204 ceph-mon[30933]: osdmap e595: 12 total, 12 up, 12 in 2024-09-16T05:19:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1820738429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:19:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1820738429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:19:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:05 smithi204 ceph-mon[30933]: osdmap e596: 12 total, 12 up, 12 in 2024-09-16T05:19:06.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:06.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:06.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:06.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:06.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:06.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:06.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:06.467 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:06.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:06.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T05:19:06.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:06.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:19:06.591 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:06.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:06.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:06.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:19:06.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:06.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:06.768 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:19:06.773 INFO:tasks.rados.rados.2.smithi204.stdout:2252: finishing write tid 1 to smithi20457400-6 2024-09-16T05:19:06.773 INFO:tasks.rados.rados.2.smithi204.stdout:2252: finishing write tid 2 to smithi20457400-6 2024-09-16T05:19:06.773 INFO:tasks.rados.rados.2.smithi204.stdout:2252: finishing write tid 3 to smithi20457400-6 2024-09-16T05:19:06.773 INFO:tasks.rados.rados.2.smithi204.stdout:2252: finishing write tid 4 to smithi20457400-6 2024-09-16T05:19:06.773 INFO:tasks.rados.rados.2.smithi204.stdout:2252: finishing write tid 5 to smithi20457400-6 2024-09-16T05:19:06.773 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 730 (ObjNum 803 snap 237 seq_num 803) dirty exists 2024-09-16T05:19:06.773 INFO:tasks.rados.rados.2.smithi204.stdout:2252: left oid 6 (ObjNum 803 snap 237 seq_num 803) 2024-09-16T05:19:06.773 INFO:tasks.rados.rados.2.smithi204.stdout:2253: expect (ObjNum 762 snap 223 seq_num 762) 2024-09-16T05:19:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:06.821 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:06.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:06.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:06.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:06.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:06.968 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:06.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:06.981 INFO:tasks.rados.rados.2.smithi204.stdout:2252: done (2 left) 2024-09-16T05:19:06.981 INFO:tasks.rados.rados.2.smithi204.stdout:2253: done (1 left) 2024-09-16T05:19:06.981 INFO:tasks.rados.rados.2.smithi204.stdout:2254: done (0 left) 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:2255: read oid 2 snap 193 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:2255: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:2256: rollback oid 12 current snap is 237 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 235 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:2257: read oid 9 snap -1 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:2257: expect (ObjNum 746 snap 215 seq_num 746) 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:2258: copy_from oid 5 from oid 43 current snap is 237 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:2259: write (excl) oid 1 current snap is 237 2024-09-16T05:19:06.982 INFO:tasks.rados.rados.2.smithi204.stdout:2259: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2024-09-16T05:19:06.989 INFO:tasks.rados.rados.2.smithi204.stdout:2259: writing smithi20457400-1 from 712828 to 1430089 tid 1 2024-09-16T05:19:06.996 INFO:tasks.rados.rados.2.smithi204.stdout:2259: writing smithi20457400-1 from 1915052 to 2633988 tid 2 2024-09-16T05:19:07.001 INFO:tasks.rados.rados.2.smithi204.stdout:2259: writing smithi20457400-1 from 3212792 to 3512828 tid 3 2024-09-16T05:19:07.002 INFO:tasks.rados.rados.2.smithi204.stdout:2260: delete oid 48 current snap is 237 2024-09-16T05:19:07.002 INFO:tasks.rados.rados.2.smithi204.stdout:2256: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:19:07.002 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 775 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T05:19:07.002 INFO:tasks.rados.rados.2.smithi204.stdout:2255: expect (ObjNum 542 snap 162 seq_num 542) 2024-09-16T05:19:07.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1368699032' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:07.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1368699032' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:07.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:06 smithi032 ceph-mon[27388]: osdmap e597: 12 total, 12 up, 12 in 2024-09-16T05:19:07.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:07.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:07.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:07.105 INFO:tasks.rados.rados.2.smithi204.stdout:2257: expect (ObjNum 746 snap 215 seq_num 746) 2024-09-16T05:19:07.163 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1368699032' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:07.163 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1368699032' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:07.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:06 smithi196 ceph-mon[30802]: osdmap e597: 12 total, 12 up, 12 in 2024-09-16T05:19:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1368699032' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1368699032' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:06 smithi204 ceph-mon[30933]: osdmap e597: 12 total, 12 up, 12 in 2024-09-16T05:19:07.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:07.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:07.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:07.229 INFO:tasks.rados.rados.2.smithi204.stdout:2259: finishing write tid 1 to smithi20457400-1 2024-09-16T05:19:07.229 INFO:tasks.rados.rados.2.smithi204.stdout:2258: finishing copy_from racing read to smithi20457400-5 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:2259: finishing write tid 2 to smithi20457400-1 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:2258: finishing copy_from to smithi20457400-5 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 601 (ObjNum 792 snap 230 seq_num 792) dirty exists 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:2259: finishing write tid 3 to smithi20457400-1 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:2259: finishing write tid 4 to smithi20457400-1 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:2259: finishing write tid 5 to smithi20457400-1 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 539 (ObjNum 804 snap 237 seq_num 804) dirty exists 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:2259: left oid 1 (ObjNum 804 snap 237 seq_num 804) 2024-09-16T05:19:07.230 INFO:tasks.rados.rados.2.smithi204.stdout:2255: done (5 left) 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2256: done (4 left) 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2257: done (3 left) 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2258: done (2 left) 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2259: done (1 left) 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2260: done (0 left) 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2261: read oid 16 snap -1 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2261: expect (ObjNum 781 snap 227 seq_num 781) 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2262: read oid 2 snap -1 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2262: expect (ObjNum 737 snap 215 seq_num 737) 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2263: rollback oid 34 current snap is 237 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 234 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:2264: rollback oid 26 current snap is 237 2024-09-16T05:19:07.231 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 200 2024-09-16T05:19:07.232 INFO:tasks.rados.rados.2.smithi204.stdout:2265: snap_remove snap 233 2024-09-16T05:19:07.277 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:19:07.277 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1378 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 207] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 404.67441597800001, 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018067625, 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.279 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.279 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1378 2024-09-16T05:19:07.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.279 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:19:07.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589519+0000", 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3460000000000001e-06 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589547+0000", 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8572999999999999e-05 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589577+0000", 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9822e-05 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589592+0000", 2024-09-16T05:19:07.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4630000000000001e-05 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589745+0000", 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015316700000000001 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589921+0000", 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000175995 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.602330+0000", 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012408878 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605723+0000", 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033928000000000001 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607502+0000", 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017799140000000001 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607546+0000", 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3130999999999998e-05 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607584+0000", 2024-09-16T05:19:07.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8368999999999998e-05 2024-09-16T05:19:07.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.285 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.285 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1379 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:19:07.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 404.67430917399997, 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015638332000000001, 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1379 2024-09-16T05:19:07.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589623+0000", 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589624+0000", 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4e-07 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589629+0000", 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.366e-06 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589642+0000", 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2114e-05 2024-09-16T05:19:07.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589660+0000", 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8039000000000001e-05 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.590086+0000", 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042614600000000002 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:19:07.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.590119+0000", 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3395999999999997e-05 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605083+0000", 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014963611999999999 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605167+0000", 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4444999999999995e-05 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605261+0000", 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4273999999999999e-05 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(2.34 e278 from shard 3)", 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 336.39695559900002, 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34868665500000001, 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.291 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9999999999999997e-07 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866983+0000", 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.84e-06 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215470+0000", 2024-09-16T05:19:07.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34848719900000003 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215480+0000", 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2129999999999997e-06 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215584+0000", 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010428400000000001 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215601+0000", 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7509000000000001e-05 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215663+0000", 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1909999999999995e-05 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2256 2.34 2:2cffa4d6:::smithi20457400-50:head [copy-from ver 319] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 325.58227505999997, 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019683254000000001, 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2256 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681660+0000", 2024-09-16T05:19:07.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5019999999999999e-06 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681670+0000", 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.91e-06 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681686+0000", 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6699999999999999e-05 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681695+0000", 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9590000000000001e-06 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681793+0000", 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7699000000000001e-05 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681800+0000", 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1199999999999996e-06 2024-09-16T05:19:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.682948+0000", 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011475470000000001 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683035+0000", 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6886999999999997e-05 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.693853+0000", 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010818583 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698768+0000", 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049152340000000001 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701246+0000", 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024771630000000001 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701284+0000", 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8544000000000002e-05 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701340+0000", 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6406000000000002e-05 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2257 2.34 2:2cffa4d6:::smithi20457400-50:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 325.58221958399997, 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016511652000000002, 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2257 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.303 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681713+0000", 2024-09-16T05:19:07.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7700000000000005e-07 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681718+0000", 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4920000000000004e-06 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681724+0000", 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9900000000000002e-06 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681727+0000", 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8179999999999997e-06 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681858+0000", 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130698 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681862+0000", 2024-09-16T05:19:07.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0149999999999997e-06 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683176+0000", 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013134119999999999 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683211+0000", 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5091e-05 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698063+0000", 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014852783 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698134+0000", 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0570999999999999e-05 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698224+0000", 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0304999999999997e-05 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.308 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2511 2.3 2.e1e51803 (undecoded) ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e320)", 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 282.44086026399998, 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99828526200000001, 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.309 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2511 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823073+0000", 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.04e-06 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823078+0000", 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6519999999999997e-06 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823091+0000", 2024-09-16T05:19:07.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3277e-05 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823100+0000", 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4909999999999994e-06 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823141+0000", 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0187e-05 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823147+0000", 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3929999999999996e-06 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821329+0000", 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99818146900000004 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821357+0000", 2024-09-16T05:19:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8753000000000001e-05 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:136 2.34 e355/278 ondisk, result = 0)", 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 241.56215853800001, 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83858643600000005, 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701775+0000", 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6200000000000006e-07 2024-09-16T05:19:07.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701786+0000", 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0815000000000001e-05 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540112+0000", 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83832645699999997 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540147+0000", 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5416000000000001e-05 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540246+0000", 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8171999999999995e-05 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540254+0000", 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:19:07.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540360+0000", 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010627400000000001 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.318 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3152 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 301] snapc 8a=[8a,88,87,86,85,84,83,81,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 200.551622084, 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017546488999999998, 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3152 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.319 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712312+0000", 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4449999999999999e-06 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712319+0000", 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4000000000000003e-06 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712332+0000", 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2928000000000001e-05 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712356+0000", 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4071999999999999e-05 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712421+0000", 2024-09-16T05:19:07.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4709000000000007e-05 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712428+0000", 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2520000000000002e-06 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713705+0000", 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001276672 2024-09-16T05:19:07.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713800+0000", 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5266999999999994e-05 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.722741+0000", 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0089409339999999993 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.726991+0000", 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042500949999999997 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729789+0000", 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002798132 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729826+0000", 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6576999999999999e-05 2024-09-16T05:19:07.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729857+0000", 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1006000000000001e-05 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3573 2.3 2:c018a787:::smithi20457400-18:head [write 1692526~473837] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.39819040099999, 2024-09-16T05:19:07.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022224677000000002, 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3573 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:19:07.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865743+0000", 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0100000000000001e-06 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866150+0000", 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040748299999999999 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866163+0000", 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2187999999999999e-05 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866168+0000", 2024-09-16T05:19:07.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4689999999999996e-06 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866251+0000", 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2911000000000006e-05 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866320+0000", 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8596000000000005e-05 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:19:07.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866403+0000", 2024-09-16T05:19:07.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3796999999999997e-05 2024-09-16T05:19:07.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867984+0000", 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015806399999999999 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887897+0000", 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019913157000000001 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887934+0000", 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6482000000000001e-05 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887967+0000", 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2944000000000002e-05 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.334 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3574 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 2403500] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.39775084799999, 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039246627999999999, 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3574 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866182+0000", 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:19:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866205+0000", 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2592e-05 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866222+0000", 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7422e-05 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866231+0000", 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5220000000000007e-06 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867407+0000", 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001176014 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867514+0000", 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106947 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867661+0000", 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000147396 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867987+0000", 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032622400000000001 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905316+0000", 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037329222000000002 2024-09-16T05:19:07.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905388+0000", 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1896000000000004e-05 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905428+0000", 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9830000000000003e-05 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.340 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.341 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3575 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:19:07.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:19:07.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.39769285099999, 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039343647000000002, 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3575 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.343 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866240+0000", 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7999999999999995e-07 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866243+0000", 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5770000000000001e-06 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866249+0000", 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866254+0000", 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-06 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867919+0000", 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016648609999999999 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:07.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867974+0000", 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4478000000000003e-05 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905464+0000", 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037490460000000003 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905516+0000", 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1894999999999997e-05 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905583+0000", 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3579 2.34 2:2cffa4d6:::smithi20457400-50:head [write 420307~462694] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.393100672, 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017271116999999999, 2024-09-16T05:19:07.348 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3579 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870833+0000", 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2440000000000001e-06 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871089+0000", 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025604600000000001 2024-09-16T05:19:07.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871102+0000", 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2955999999999999e-05 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871110+0000", 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2700000000000004e-06 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871161+0000", 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0899999999999997e-05 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871220+0000", 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8659999999999997e-05 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871310+0000", 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0586000000000003e-05 2024-09-16T05:19:07.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.872398+0000", 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010881409999999999 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888033+0000", 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015634834 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888071+0000", 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7957999999999998e-05 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888103+0000", 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1522000000000001e-05 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3580 2.34 2:2cffa4d6:::smithi20457400-50:head [write 1351008~617495] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:19:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.38739210700001, 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030886027999999999, 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3580 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876542+0000", 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877114+0000", 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057156899999999996 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877126+0000", 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2751999999999999e-05 2024-09-16T05:19:07.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877134+0000", 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.072e-06 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:19:07.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0603000000000003e-05 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877240+0000", 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4561000000000002e-05 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877317+0000", 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7002999999999994e-05 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878479+0000", 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011619130000000001 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907362+0000", 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028883039999999999 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907396+0000", 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4807000000000001e-05 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907426+0000", 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9821000000000001e-05 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3581 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2420307] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.386781313, 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030393973000000001, 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.364 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3581 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877152+0000", 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3099999999999996e-07 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877156+0000", 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7589999999999999e-06 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877169+0000", 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3129000000000001e-05 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877175+0000", 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6259999999999997e-06 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877997+0000", 2024-09-16T05:19:07.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082123199999999997 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878081+0000", 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4914999999999996e-05 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878202+0000", 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120207 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878482+0000", 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027995500000000003 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907462+0000", 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028980381 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907500+0000", 2024-09-16T05:19:07.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907545+0000", 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5120999999999997e-05 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3582 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.386746907, 2024-09-16T05:19:07.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030495267999999999, 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3582 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877186+0000", 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-07 2024-09-16T05:19:07.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877189+0000", 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6800000000000002e-06 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877196+0000", 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0650000000000001e-06 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877202+0000", 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.28e-06 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878412+0000", 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001210205 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878470+0000", 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7759e-05 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907572+0000", 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029102086999999999 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907620+0000", 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8384000000000003e-05 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907681+0000", 2024-09-16T05:19:07.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0244000000000001e-05 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:288 2.16 e446/273 ondisk, result = 0)", 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 150.38021554700001, 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65473014699999998, 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.376 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883718+0000", 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3799999999999996e-07 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883722+0000", 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.284e-06 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538184+0000", 2024-09-16T05:19:07.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.654462031 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538201+0000", 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6634e-05 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538327+0000", 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012605799999999999 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538338+0000", 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1188e-05 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538447+0000", 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109014 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.380 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4460 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 511] snapc bc=[bc,bb,ba,b8] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 94.204631624000001, 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019695292, 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4460 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9799999999999999e-07 2024-09-16T05:19:07.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059304+0000", 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5490000000000001e-06 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059308+0000", 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3610000000000003e-06 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059311+0000", 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9179999999999998e-06 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059364+0000", 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2790000000000001e-05 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059422+0000", 2024-09-16T05:19:07.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7929000000000001e-05 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.073174+0000", 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013752234 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.077028+0000", 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038540319999999999 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078924+0000", 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001896179 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078961+0000", 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7311000000000002e-05 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078996+0000", 2024-09-16T05:19:07.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4591000000000001e-05 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4461 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 94.204610684000002, 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017250960999999999, 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.387 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4461 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4899999999999995e-07 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059326+0000", 2024-09-16T05:19:07.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9280000000000003e-06 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059333+0000", 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6529999999999997e-06 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059338+0000", 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1379999999999998e-06 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059598+0000", 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026046400000000002 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059658+0000", 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9342999999999997e-05 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076427+0000", 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016769117 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076490+0000", 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3207000000000007e-05 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076573+0000", 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2562e-05 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5587 2.34 2:2d3f13f8:::smithi20457400-7:head [setxattr _header (57) in=7b,truncate 3260723] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.458093955000001, 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.147062895, 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5587 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.393 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805839+0000", 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8299999999999997e-07 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805842+0000", 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6790000000000001e-06 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805846+0000", 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4089999999999996e-06 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805849+0000", 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.191e-06 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806887+0000", 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010375440000000001 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806953+0000", 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5833000000000005e-05 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807079+0000", 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012666500000000001 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.809689+0000", 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026092860000000002 2024-09-16T05:19:07.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952747+0000", 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14305815 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952826+0000", 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9773999999999999e-05 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952901+0000", 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4781000000000001e-05 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5588 2.34 2:2d3f13f8:::smithi20457400-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 14.458067482000001, 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14722005799999999, 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:07.399 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5588 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.65e-07 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805867+0000", 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.86e-06 2024-09-16T05:19:07.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805872+0000", 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805876+0000", 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0380000000000001e-06 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807396+0000", 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001520448 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807470+0000", 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3695000000000005e-05 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952937+0000", 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14546690300000001 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953007+0000", 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9801999999999993e-05 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953085+0000", 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8121999999999995e-05 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:07.404 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:19:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:07.406 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:07.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:07.507 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:07.528 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:07.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:07.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:07.652 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:07.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:07.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:07.778 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:07.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:07.825 INFO:tasks.rados.rados.2.smithi204.stdout:2265: done (4 left) 2024-09-16T05:19:07.825 INFO:tasks.rados.rados.2.smithi204.stdout:2266: snap_remove snap 231 2024-09-16T05:19:07.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:07.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:07.998 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:08.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:08.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:08.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:08.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:19:08.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:07 smithi196 ceph-mon[30802]: pgmap v1056: 107 pgs: 107 active+clean; 339 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:19:08.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:07 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:19:08.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:07 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:19:08.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:07 smithi032 ceph-mon[27388]: pgmap v1056: 107 pgs: 107 active+clean; 339 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:19:08.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:07 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:19:08.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:07 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:19:08.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:08.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:07 smithi204 ceph-mon[30933]: pgmap v1056: 107 pgs: 107 active+clean; 339 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2024-09-16T05:19:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:07 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:19:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:07 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:19:08.251 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:08.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:08.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:08.328 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:08.377 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:08.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:08.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:08.492 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:08.493 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:08.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:08.514 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:08.515 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:08.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:08.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:08.535 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:08.557 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:08.587 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:08.625 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:08.638 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:08.638 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:08.651 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:08.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:08.660 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:08.660 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:08.681 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:08.681 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:08.703 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:08.724 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:08.725 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:08.746 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:08.747 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:08.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:08.768 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:08.768 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:08.774 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:08.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:08.804 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:08.805 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:08.831 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T05:19:08.835 INFO:tasks.rados.rados.2.smithi204.stdout:2266: done (4 left) 2024-09-16T05:19:08.835 INFO:tasks.rados.rados.2.smithi204.stdout:2267: delete oid 5 current snap is 237 2024-09-16T05:19:08.835 INFO:tasks.rados.rados.2.smithi204.stdout:2263: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:19:08.835 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 701 (ObjNum 33 snap 0 seq_num 4282369504) dirty dne 2024-09-16T05:19:08.835 INFO:tasks.rados.rados.2.smithi204.stdout:2264: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:19:08.835 INFO:tasks.rados.rados.2.smithi204.stdout:2264: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:19:08.835 INFO:tasks.rados.rados.2.smithi204.stdout:2264: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:19:08.836 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1031 (ObjNum 684 snap 200 seq_num 684) dirty exists 2024-09-16T05:19:08.836 INFO:tasks.rados.rados.2.smithi204.stdout:2262: expect (ObjNum 737 snap 215 seq_num 737) 2024-09-16T05:19:08.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:08.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:08.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:08.915 INFO:tasks.rados.rados.2.smithi204.stdout:2261: expect (ObjNum 781 snap 227 seq_num 781) 2024-09-16T05:19:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:19:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:19:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: 2.26 scrub starts 2024-09-16T05:19:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: 2.26 scrub ok 2024-09-16T05:19:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:19:08.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:19:08.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: osdmap e598: 12 total, 12 up, 12 in 2024-09-16T05:19:08.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1979938531' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:19:08.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: 2.4d scrub starts 2024-09-16T05:19:08.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: 2.4d scrub ok 2024-09-16T05:19:08.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1304488829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:08.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: 2.26 scrub starts 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: 2.26 scrub ok 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: osdmap e598: 12 total, 12 up, 12 in 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1979938531' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: 2.4d scrub starts 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: 2.4d scrub ok 2024-09-16T05:19:08.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1304488829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:09.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:09.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:09.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:09.082 INFO:tasks.rados.rados.2.smithi204.stdout:2261: done (4 left) 2024-09-16T05:19:09.082 INFO:tasks.rados.rados.2.smithi204.stdout:2262: done (3 left) 2024-09-16T05:19:09.082 INFO:tasks.rados.rados.2.smithi204.stdout:2263: done (2 left) 2024-09-16T05:19:09.082 INFO:tasks.rados.rados.2.smithi204.stdout:2264: done (1 left) 2024-09-16T05:19:09.082 INFO:tasks.rados.rados.2.smithi204.stdout:2267: done (0 left) 2024-09-16T05:19:09.082 INFO:tasks.rados.rados.2.smithi204.stdout:2268: write oid 23 current snap is 237 2024-09-16T05:19:09.083 INFO:tasks.rados.rados.2.smithi204.stdout:2268: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2024-09-16T05:19:09.089 INFO:tasks.rados.rados.2.smithi204.stdout:2268: writing smithi20457400-23 from 763249 to 1422610 tid 1 2024-09-16T05:19:09.096 INFO:tasks.rados.rados.2.smithi204.stdout:2268: writing smithi20457400-23 from 2090498 to 2673114 tid 2 2024-09-16T05:19:09.101 INFO:tasks.rados.rados.2.smithi204.stdout:2268: writing smithi20457400-23 from 3295061 to 3563249 tid 3 2024-09-16T05:19:09.101 INFO:tasks.rados.rados.2.smithi204.stdout:2269: write oid 46 current snap is 237 2024-09-16T05:19:09.101 INFO:tasks.rados.rados.2.smithi204.stdout:2269: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2024-09-16T05:19:09.109 INFO:tasks.rados.rados.2.smithi204.stdout:2269: writing smithi20457400-46 from 780056 to 1514666 tid 1 2024-09-16T05:19:09.116 INFO:tasks.rados.rados.2.smithi204.stdout:2269: writing smithi20457400-46 from 2148980 to 2875254 tid 2 2024-09-16T05:19:09.120 INFO:tasks.rados.rados.2.smithi204.stdout:2269: writing smithi20457400-46 from 3322484 to 3580056 tid 3 2024-09-16T05:19:09.120 INFO:tasks.rados.rados.2.smithi204.stdout:2270: write (excl) oid 8 current snap is 237 2024-09-16T05:19:09.120 INFO:tasks.rados.rados.2.smithi204.stdout:2270: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2024-09-16T05:19:09.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:09.126 INFO:tasks.rados.rados.2.smithi204.stdout:2270: writing smithi20457400-8 from 796863 to 1206722 tid 1 2024-09-16T05:19:09.132 INFO:tasks.rados.rados.2.smithi204.stdout:2270: writing smithi20457400-8 from 1807462 to 2393747 tid 2 2024-09-16T05:19:09.139 INFO:tasks.rados.rados.2.smithi204.stdout:2270: writing smithi20457400-8 from 3066260 to 3596863 tid 3 2024-09-16T05:19:09.139 INFO:tasks.rados.rados.2.smithi204.stdout:2271: delete oid 18 current snap is 237 2024-09-16T05:19:09.139 INFO:tasks.rados.rados.2.smithi204.stdout:2268: finishing write tid 1 to smithi20457400-23 2024-09-16T05:19:09.139 INFO:tasks.rados.rados.2.smithi204.stdout:2268: finishing write tid 2 to smithi20457400-23 2024-09-16T05:19:09.139 INFO:tasks.rados.rados.2.smithi204.stdout:2268: finishing write tid 3 to smithi20457400-23 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:2268: finishing write tid 4 to smithi20457400-23 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:2268: finishing write tid 5 to smithi20457400-23 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 708 (ObjNum 807 snap 237 seq_num 807) dirty exists 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:2268: left oid 23 (ObjNum 807 snap 237 seq_num 807) 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:2269: finishing write tid 1 to smithi20457400-46 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:2269: finishing write tid 2 to smithi20457400-46 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:2269: finishing write tid 3 to smithi20457400-46 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:2269: finishing write tid 4 to smithi20457400-46 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:2269: finishing write tid 5 to smithi20457400-46 2024-09-16T05:19:09.140 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 362 (ObjNum 808 snap 237 seq_num 808) dirty exists 2024-09-16T05:19:09.141 INFO:tasks.rados.rados.2.smithi204.stdout:2269: left oid 46 (ObjNum 808 snap 237 seq_num 808) 2024-09-16T05:19:09.141 INFO:tasks.rados.rados.2.smithi204.stdout:2270: finishing write tid 1 to smithi20457400-8 2024-09-16T05:19:09.141 INFO:tasks.rados.rados.2.smithi204.stdout:2270: finishing write tid 2 to smithi20457400-8 2024-09-16T05:19:09.141 INFO:tasks.rados.rados.2.smithi204.stdout:2268: done (3 left) 2024-09-16T05:19:09.141 INFO:tasks.rados.rados.2.smithi204.stdout:2269: done (2 left) 2024-09-16T05:19:09.141 INFO:tasks.rados.rados.2.smithi204.stdout:2271: done (1 left) 2024-09-16T05:19:09.141 INFO:tasks.rados.rados.2.smithi204.stdout:2272: delete oid 44 current snap is 237 2024-09-16T05:19:09.142 INFO:tasks.rados.rados.2.smithi204.stdout:2270: finishing write tid 3 to smithi20457400-8 2024-09-16T05:19:09.143 INFO:tasks.rados.rados.2.smithi204.stdout:2270: finishing write tid 4 to smithi20457400-8 2024-09-16T05:19:09.143 INFO:tasks.rados.rados.2.smithi204.stdout:2270: finishing write tid 5 to smithi20457400-8 2024-09-16T05:19:09.143 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 784 (ObjNum 809 snap 237 seq_num 809) dirty exists 2024-09-16T05:19:09.143 INFO:tasks.rados.rados.2.smithi204.stdout:2270: left oid 8 (ObjNum 809 snap 237 seq_num 809) 2024-09-16T05:19:09.143 INFO:tasks.rados.rados.2.smithi204.stdout:2270: done (1 left) 2024-09-16T05:19:09.143 INFO:tasks.rados.rados.2.smithi204.stdout:2272: done (0 left) 2024-09-16T05:19:09.143 INFO:tasks.rados.rados.2.smithi204.stdout:2273: write (excl) oid 49 current snap is 237 2024-09-16T05:19:09.143 INFO:tasks.rados.rados.2.smithi204.stdout:2273: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-09-16T05:19:09.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:09.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:09.148 INFO:tasks.rados.rados.2.smithi204.stdout:2273: writing smithi20457400-49 from 447284 to 1082890 tid 1 2024-09-16T05:19:09.156 INFO:tasks.rados.rados.2.smithi204.stdout:2273: writing smithi20457400-49 from 1866555 to 2600167 tid 2 2024-09-16T05:19:09.162 INFO:tasks.rados.rados.2.smithi204.stdout:2273: writing smithi20457400-49 from 3315823 to 3647284 tid 3 2024-09-16T05:19:09.162 INFO:tasks.rados.rados.2.smithi204.stdout:2274: write (excl) oid 22 current snap is 237 2024-09-16T05:19:09.162 INFO:tasks.rados.rados.2.smithi204.stdout:2273: finishing write tid 1 to smithi20457400-49 2024-09-16T05:19:09.162 INFO:tasks.rados.rados.2.smithi204.stdout:2273: finishing write tid 2 to smithi20457400-49 2024-09-16T05:19:09.162 INFO:tasks.rados.rados.2.smithi204.stdout:2274: seq_num 813 ranges {464091=710855,1925037=477270,2943246=474970} 2024-09-16T05:19:09.168 INFO:tasks.rados.rados.2.smithi204.stdout:2274: writing smithi20457400-22 from 464091 to 1174946 tid 1 2024-09-16T05:19:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:19:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:19:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: 2.26 scrub starts 2024-09-16T05:19:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: 2.26 scrub ok 2024-09-16T05:19:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:19:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:19:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: osdmap e598: 12 total, 12 up, 12 in 2024-09-16T05:19:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1979938531' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:19:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: 2.4d scrub starts 2024-09-16T05:19:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: 2.4d scrub ok 2024-09-16T05:19:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1304488829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:09.175 INFO:tasks.rados.rados.2.smithi204.stdout:2274: writing smithi20457400-22 from 1925037 to 2402307 tid 2 2024-09-16T05:19:09.181 INFO:tasks.rados.rados.2.smithi204.stdout:2274: writing smithi20457400-22 from 2943246 to 3418216 tid 3 2024-09-16T05:19:09.181 INFO:tasks.rados.rados.2.smithi204.stdout:2275: write oid 11 current snap is 237 2024-09-16T05:19:09.181 INFO:tasks.rados.rados.2.smithi204.stdout:2273: finishing write tid 3 to smithi20457400-49 2024-09-16T05:19:09.181 INFO:tasks.rados.rados.2.smithi204.stdout:2275: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-09-16T05:19:09.186 INFO:tasks.rados.rados.2.smithi204.stdout:2275: writing smithi20457400-11 from 480898 to 983355 tid 1 2024-09-16T05:19:09.194 INFO:tasks.rados.rados.2.smithi204.stdout:2275: writing smithi20457400-11 from 1699872 to 2437153 tid 2 2024-09-16T05:19:09.202 INFO:tasks.rados.rados.2.smithi204.stdout:2275: writing smithi20457400-11 from 3087022 to 3680898 tid 3 2024-09-16T05:19:09.202 INFO:tasks.rados.rados.2.smithi204.stdout:2276: write (excl) oid 32 current snap is 237 2024-09-16T05:19:09.202 INFO:tasks.rados.rados.2.smithi204.stdout:2273: finishing write tid 4 to smithi20457400-49 2024-09-16T05:19:09.202 INFO:tasks.rados.rados.2.smithi204.stdout:2273: finishing write tid 5 to smithi20457400-49 2024-09-16T05:19:09.202 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 605 (ObjNum 812 snap 237 seq_num 812) dirty exists 2024-09-16T05:19:09.202 INFO:tasks.rados.rados.2.smithi204.stdout:2273: left oid 49 (ObjNum 812 snap 237 seq_num 812) 2024-09-16T05:19:09.202 INFO:tasks.rados.rados.2.smithi204.stdout:2274: finishing write tid 1 to smithi20457400-22 2024-09-16T05:19:09.203 INFO:tasks.rados.rados.2.smithi204.stdout:2276: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2024-09-16T05:19:09.208 INFO:tasks.rados.rados.2.smithi204.stdout:2276: writing smithi20457400-32 from 497705 to 1075411 tid 1 2024-09-16T05:19:09.213 INFO:tasks.rados.rados.2.smithi204.stdout:2276: writing smithi20457400-32 from 1642001 to 2122940 tid 2 2024-09-16T05:19:09.219 INFO:tasks.rados.rados.2.smithi204.stdout:2276: writing smithi20457400-32 from 2598092 to 3095606 tid 3 2024-09-16T05:19:09.219 INFO:tasks.rados.rados.2.smithi204.stdout:2273: done (3 left) 2024-09-16T05:19:09.219 INFO:tasks.rados.rados.2.smithi204.stdout:2277: copy_from oid 25 from oid 43 current snap is 237 2024-09-16T05:19:09.219 INFO:tasks.rados.rados.2.smithi204.stdout:2274: finishing write tid 2 to smithi20457400-22 2024-09-16T05:19:09.219 INFO:tasks.rados.rados.2.smithi204.stdout:2274: finishing write tid 3 to smithi20457400-22 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2274: finishing write tid 4 to smithi20457400-22 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2274: finishing write tid 5 to smithi20457400-22 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 594 (ObjNum 813 snap 237 seq_num 813) dirty exists 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2274: left oid 22 (ObjNum 813 snap 237 seq_num 813) 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2275: finishing write tid 1 to smithi20457400-11 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2275: finishing write tid 2 to smithi20457400-11 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2275: finishing write tid 3 to smithi20457400-11 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2275: finishing write tid 4 to smithi20457400-11 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2275: finishing write tid 5 to smithi20457400-11 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 896 (ObjNum 814 snap 237 seq_num 814) dirty exists 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2275: left oid 11 (ObjNum 814 snap 237 seq_num 814) 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2276: finishing write tid 1 to smithi20457400-32 2024-09-16T05:19:09.220 INFO:tasks.rados.rados.2.smithi204.stdout:2274: done (3 left) 2024-09-16T05:19:09.221 INFO:tasks.rados.rados.2.smithi204.stdout:2275: done (2 left) 2024-09-16T05:19:09.221 INFO:tasks.rados.rados.2.smithi204.stdout:2278: snap_create 2024-09-16T05:19:09.221 INFO:tasks.rados.rados.2.smithi204.stdout:2276: finishing write tid 2 to smithi20457400-32 2024-09-16T05:19:09.221 INFO:tasks.rados.rados.2.smithi204.stdout:2276: finishing write tid 3 to smithi20457400-32 2024-09-16T05:19:09.221 INFO:tasks.rados.rados.2.smithi204.stdout:2276: finishing write tid 4 to smithi20457400-32 2024-09-16T05:19:09.222 INFO:tasks.rados.rados.2.smithi204.stdout:2276: finishing write tid 5 to smithi20457400-32 2024-09-16T05:19:09.222 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 827 (ObjNum 815 snap 237 seq_num 815) dirty exists 2024-09-16T05:19:09.222 INFO:tasks.rados.rados.2.smithi204.stdout:2276: left oid 32 (ObjNum 815 snap 237 seq_num 815) 2024-09-16T05:19:09.226 INFO:tasks.rados.rados.2.smithi204.stdout:2277: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:19:09.232 INFO:tasks.rados.rados.2.smithi204.stdout:2277: finishing copy_from to smithi20457400-25 2024-09-16T05:19:09.232 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 601 (ObjNum 792 snap 230 seq_num 792) dirty exists 2024-09-16T05:19:09.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:09.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:09.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:09.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:09.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:09.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:09.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:19:09.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:19:09.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:09.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:09.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:09.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T05:19:09.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:09.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:09.686 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:09.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:09.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:09.843 INFO:tasks.rados.rados.2.smithi204.stdout:2276: done (2 left) 2024-09-16T05:19:09.843 INFO:tasks.rados.rados.2.smithi204.stdout:2277: done (1 left) 2024-09-16T05:19:09.843 INFO:tasks.rados.rados.2.smithi204.stdout:2278: done (0 left) 2024-09-16T05:19:09.843 INFO:tasks.rados.rados.2.smithi204.stdout:2279: delete oid 28 current snap is 238 2024-09-16T05:19:09.846 INFO:tasks.rados.rados.2.smithi204.stdout:2279: done (0 left) 2024-09-16T05:19:09.846 INFO:tasks.rados.rados.2.smithi204.stdout:2280: snap_create 2024-09-16T05:19:09.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:09 smithi032 ceph-mon[27388]: pgmap v1059: 107 pgs: 107 active+clean; 336 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2024-09-16T05:19:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:09 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:19:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:09 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:19:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1979938531' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:19:09.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:09 smithi032 ceph-mon[27388]: osdmap e599: 12 total, 12 up, 12 in 2024-09-16T05:19:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:09.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:09.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:10.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:10.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:10.052 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:09 smithi196 ceph-mon[30802]: pgmap v1059: 107 pgs: 107 active+clean; 336 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2024-09-16T05:19:10.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:09 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:19:10.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:09 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:19:10.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1979938531' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:19:10.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:09 smithi196 ceph-mon[30802]: osdmap e599: 12 total, 12 up, 12 in 2024-09-16T05:19:10.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:10.152 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:10.152 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:09 smithi204 ceph-mon[30933]: pgmap v1059: 107 pgs: 107 active+clean; 336 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.8 MiB/s wr, 17 op/s 2024-09-16T05:19:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:09 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:19:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:09 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:19:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1979938531' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:19:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:09 smithi204 ceph-mon[30933]: osdmap e599: 12 total, 12 up, 12 in 2024-09-16T05:19:10.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:10.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:10.176 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:10.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:10.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:10.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:10.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:10.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:10.523 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:10.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:10.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:10.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:10.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:10.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:19:10.775 INFO:tasks.rados.rados.2.smithi204.stdout:2280: done (0 left) 2024-09-16T05:19:10.775 INFO:tasks.rados.rados.2.smithi204.stdout:2281: delete oid 36 current snap is 239 2024-09-16T05:19:10.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:10.779 INFO:tasks.rados.rados.2.smithi204.stdout:2281: done (0 left) 2024-09-16T05:19:10.779 INFO:tasks.rados.rados.2.smithi204.stdout:2282: read oid 48 snap -1 2024-09-16T05:19:10.779 INFO:tasks.rados.rados.2.smithi204.stdout:2282: expect deleted 2024-09-16T05:19:10.779 INFO:tasks.rados.rados.2.smithi204.stdout:2283: write oid 21 current snap is 239 2024-09-16T05:19:10.779 INFO:tasks.rados.rados.2.smithi204.stdout:2283: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2024-09-16T05:19:10.784 INFO:tasks.rados.rados.2.smithi204.stdout:2283: writing smithi20457400-21 from 548126 to 951579 tid 1 2024-09-16T05:19:10.791 INFO:tasks.rados.rados.2.smithi204.stdout:2283: writing smithi20457400-21 from 1417447 to 2162066 tid 2 2024-09-16T05:19:10.796 INFO:tasks.rados.rados.2.smithi204.stdout:2283: writing smithi20457400-21 from 2680361 to 3128044 tid 3 2024-09-16T05:19:10.800 INFO:tasks.rados.rados.2.smithi204.stdout:2283: writing smithi20457400-21 from 3680450 to 3748126 tid 4 2024-09-16T05:19:10.800 INFO:tasks.rados.rados.2.smithi204.stdout:2284: read oid 1 snap -1 2024-09-16T05:19:10.800 INFO:tasks.rados.rados.2.smithi204.stdout:2284: expect (ObjNum 804 snap 237 seq_num 804) 2024-09-16T05:19:10.800 INFO:tasks.rados.rados.2.smithi204.stdout:2285: read oid 44 snap 238 2024-09-16T05:19:10.800 INFO:tasks.rados.rados.2.smithi204.stdout:2285: expect deleted 2024-09-16T05:19:10.801 INFO:tasks.rados.rados.2.smithi204.stdout:2286: delete oid 15 current snap is 239 2024-09-16T05:19:10.801 INFO:tasks.rados.rados.2.smithi204.stdout:2283: finishing write tid 1 to smithi20457400-21 2024-09-16T05:19:10.801 INFO:tasks.rados.rados.2.smithi204.stdout:2283: finishing write tid 2 to smithi20457400-21 2024-09-16T05:19:10.801 INFO:tasks.rados.rados.2.smithi204.stdout:2283: finishing write tid 3 to smithi20457400-21 2024-09-16T05:19:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:10.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:10.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:10.803 INFO:tasks.rados.rados.2.smithi204.stdout:2282: done (4 left) 2024-09-16T05:19:10.803 INFO:tasks.rados.rados.2.smithi204.stdout:2285: done (3 left) 2024-09-16T05:19:10.803 INFO:tasks.rados.rados.2.smithi204.stdout:2286: done (2 left) 2024-09-16T05:19:10.803 INFO:tasks.rados.rados.2.smithi204.stdout:2287: delete oid 26 current snap is 239 2024-09-16T05:19:10.805 INFO:tasks.rados.rados.2.smithi204.stdout:2287: done (2 left) 2024-09-16T05:19:10.805 INFO:tasks.rados.rados.2.smithi204.stdout:2288: snap_remove snap 228 2024-09-16T05:19:10.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:10.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:10.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:10.999 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:11.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:19:11.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: osdmap e600: 12 total, 12 up, 12 in 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: 2.45 scrub starts 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: 2.45 scrub ok 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:11.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:10 smithi032 ceph-mon[27388]: osdmap e601: 12 total, 12 up, 12 in 2024-09-16T05:19:11.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:11.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: osdmap e600: 12 total, 12 up, 12 in 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: 2.45 scrub starts 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: 2.45 scrub ok 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:19:11.120 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:11.121 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:10 smithi196 ceph-mon[30802]: osdmap e601: 12 total, 12 up, 12 in 2024-09-16T05:19:11.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: osdmap e600: 12 total, 12 up, 12 in 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: 2.45 scrub starts 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: 2.45 scrub ok 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:10 smithi204 ceph-mon[30933]: osdmap e601: 12 total, 12 up, 12 in 2024-09-16T05:19:11.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:11.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:11.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:11.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:11.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:11.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:11.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:11.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:11.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:11.494 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:11.494 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:11.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:11.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:11.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:11.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:11.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:11.771 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T05:19:11.771 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:19:11.774 INFO:tasks.rados.rados.2.smithi204.stdout:2288: done (2 left) 2024-09-16T05:19:11.774 INFO:tasks.rados.rados.2.smithi204.stdout:2289: rollback oid 6 current snap is 239 2024-09-16T05:19:11.775 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 6 to 237 2024-09-16T05:19:11.775 INFO:tasks.rados.rados.2.smithi204.stdout:2283: finishing write tid 4 to smithi20457400-21 2024-09-16T05:19:11.775 INFO:tasks.rados.rados.2.smithi204.stdout:2283: finishing write tid 5 to smithi20457400-21 2024-09-16T05:19:11.775 INFO:tasks.rados.rados.2.smithi204.stdout:2283: finishing write tid 6 to smithi20457400-21 2024-09-16T05:19:11.775 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 839 (ObjNum 818 snap 239 seq_num 818) dirty exists 2024-09-16T05:19:11.775 INFO:tasks.rados.rados.2.smithi204.stdout:2283: left oid 21 (ObjNum 818 snap 239 seq_num 818) 2024-09-16T05:19:11.775 INFO:tasks.rados.rados.2.smithi204.stdout:2284: expect (ObjNum 804 snap 237 seq_num 804) 2024-09-16T05:19:11.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:11.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:11.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:11.890 INFO:tasks.rados.rados.2.smithi204.stdout:2289: finishing rollback tid 0 to smithi20457400-6 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:2289: finishing rollback tid 1 to smithi20457400-6 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:2283: done (2 left) 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:2284: done (1 left) 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:2290: read oid 35 snap -1 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:2290: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:2291: write (excl) oid 29 current snap is 239 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:2289: finishing rollback tid 2 to smithi20457400-6 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 733 (ObjNum 803 snap 237 seq_num 803) dirty exists 2024-09-16T05:19:11.891 INFO:tasks.rados.rados.2.smithi204.stdout:2291: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2024-09-16T05:19:11.898 INFO:tasks.rados.rados.2.smithi204.stdout:2291: writing smithi20457400-29 from 598547 to 1227747 tid 1 2024-09-16T05:19:11.904 INFO:tasks.rados.rados.2.smithi204.stdout:2291: writing smithi20457400-29 from 1876540 to 2368486 tid 2 2024-09-16T05:19:11.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:11.912 INFO:tasks.rados.rados.2.smithi204.stdout:2291: writing smithi20457400-29 from 3046277 to 3727776 tid 3 2024-09-16T05:19:11.912 INFO:tasks.rados.rados.2.smithi204.stdout:2289: done (2 left) 2024-09-16T05:19:11.912 INFO:tasks.rados.rados.2.smithi204.stdout:2292: rollback oid 43 current snap is 239 2024-09-16T05:19:11.912 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 43 to 238 2024-09-16T05:19:11.912 INFO:tasks.rados.rados.2.smithi204.stdout:2290: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:19:11.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:11.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:12.021 INFO:tasks.rados.rados.2.smithi204.stdout:2291: finishing write tid 1 to smithi20457400-29 2024-09-16T05:19:12.021 INFO:tasks.rados.rados.2.smithi204.stdout:2291: finishing write tid 2 to smithi20457400-29 2024-09-16T05:19:12.021 INFO:tasks.rados.rados.2.smithi204.stdout:2290: done (2 left) 2024-09-16T05:19:12.021 INFO:tasks.rados.rados.2.smithi204.stdout:2293: snap_create 2024-09-16T05:19:12.021 INFO:tasks.rados.rados.2.smithi204.stdout:2291: finishing write tid 3 to smithi20457400-29 2024-09-16T05:19:12.022 INFO:tasks.rados.rados.2.smithi204.stdout:2291: finishing write tid 4 to smithi20457400-29 2024-09-16T05:19:12.022 INFO:tasks.rados.rados.2.smithi204.stdout:2291: finishing write tid 5 to smithi20457400-29 2024-09-16T05:19:12.022 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 836 (ObjNum 821 snap 239 seq_num 821) dirty exists 2024-09-16T05:19:12.022 INFO:tasks.rados.rados.2.smithi204.stdout:2291: left oid 29 (ObjNum 821 snap 239 seq_num 821) 2024-09-16T05:19:12.022 INFO:tasks.rados.rados.2.smithi204.stdout:2292: finishing rollback tid 0 to smithi20457400-43 2024-09-16T05:19:12.022 INFO:tasks.rados.rados.2.smithi204.stdout:2292: finishing rollback tid 1 to smithi20457400-43 2024-09-16T05:19:12.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:12.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:12.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:12.076 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: pgmap v1062: 107 pgs: 107 active+clean; 333 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4202140156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4054227694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:12.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:19:12.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:19:12.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:12.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:19:12.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4202140156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:19:12.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4054227694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:12.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:11 smithi196 ceph-mon[30802]: osdmap e602: 12 total, 12 up, 12 in 2024-09-16T05:19:12.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:19:12.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:19:12.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: pgmap v1062: 107 pgs: 107 active+clean; 333 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-16T05:19:12.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:19:12.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:19:12.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:19:12.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:19:12.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4202140156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:19:12.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4054227694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4202140156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4054227694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:12.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:11 smithi032 ceph-mon[27388]: osdmap e602: 12 total, 12 up, 12 in 2024-09-16T05:19:12.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: pgmap v1062: 107 pgs: 107 active+clean; 333 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 11 op/s 2024-09-16T05:19:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:19:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:19:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:19:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:19:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4202140156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:19:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:19:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4054227694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4202140156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4054227694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:12.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:11 smithi204 ceph-mon[30933]: osdmap e602: 12 total, 12 up, 12 in 2024-09-16T05:19:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:12.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:12.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:12.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:12.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:12.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:12.383 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:12.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:12.454 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:12.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:12.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:12.578 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:12.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:12.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:12.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:12.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:12.758 INFO:tasks.rados.rados.2.smithi204.stdout:2292: finishing rollback tid 2 to smithi20457400-43 2024-09-16T05:19:12.759 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 608 (ObjNum 792 snap 230 seq_num 792) dirty exists 2024-09-16T05:19:12.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T05:19:12.801 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:12.801 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:12.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:12.855 INFO:tasks.rados.rados.2.smithi204.stdout:2291: done (2 left) 2024-09-16T05:19:12.855 INFO:tasks.rados.rados.2.smithi204.stdout:2292: done (1 left) 2024-09-16T05:19:12.855 INFO:tasks.rados.rados.2.smithi204.stdout:2293: done (0 left) 2024-09-16T05:19:12.856 INFO:tasks.rados.rados.2.smithi204.stdout:2294: delete oid 21 current snap is 240 2024-09-16T05:19:12.857 INFO:tasks.rados.rados.2.smithi204.stdout:2294: done (0 left) 2024-09-16T05:19:12.857 INFO:tasks.rados.rados.2.smithi204.stdout:2295: write (excl) oid 33 current snap is 240 2024-09-16T05:19:12.857 INFO:tasks.rados.rados.2.smithi204.stdout:2295: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2024-09-16T05:19:12.859 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:12.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:12.863 INFO:tasks.rados.rados.2.smithi204.stdout:2295: writing smithi20457400-33 from 632161 to 1128212 tid 1 2024-09-16T05:19:12.869 INFO:tasks.rados.rados.2.smithi204.stdout:2295: writing smithi20457400-33 from 1593504 to 2089119 tid 2 2024-09-16T05:19:12.874 INFO:tasks.rados.rados.2.smithi204.stdout:2295: writing smithi20457400-33 from 2701123 to 3121519 tid 3 2024-09-16T05:19:12.878 INFO:tasks.rados.rados.2.smithi204.stdout:2295: writing smithi20457400-33 from 3611645 to 3832161 tid 4 2024-09-16T05:19:12.878 INFO:tasks.rados.rados.2.smithi204.stdout:2296: write (excl) oid 23 current snap is 240 2024-09-16T05:19:12.878 INFO:tasks.rados.rados.2.smithi204.stdout:2295: finishing write tid 1 to smithi20457400-33 2024-09-16T05:19:12.879 INFO:tasks.rados.rados.2.smithi204.stdout:2295: finishing write tid 2 to smithi20457400-33 2024-09-16T05:19:12.879 INFO:tasks.rados.rados.2.smithi204.stdout:2295: finishing write tid 3 to smithi20457400-33 2024-09-16T05:19:12.879 INFO:tasks.rados.rados.2.smithi204.stdout:2296: seq_num 824 ranges {648968=571300,1651986=639273,2728546=631668} 2024-09-16T05:19:12.885 INFO:tasks.rados.rados.2.smithi204.stdout:2296: writing smithi20457400-23 from 648968 to 1220268 tid 1 2024-09-16T05:19:12.891 INFO:tasks.rados.rados.2.smithi204.stdout:2296: writing smithi20457400-23 from 1651986 to 2291259 tid 2 2024-09-16T05:19:12.897 INFO:tasks.rados.rados.2.smithi204.stdout:2296: writing smithi20457400-23 from 2728546 to 3360214 tid 3 2024-09-16T05:19:12.897 INFO:tasks.rados.rados.2.smithi204.stdout:2297: read oid 15 snap -1 2024-09-16T05:19:12.897 INFO:tasks.rados.rados.2.smithi204.stdout:2297: expect deleted 2024-09-16T05:19:12.897 INFO:tasks.rados.rados.2.smithi204.stdout:2298: snap_create 2024-09-16T05:19:12.897 INFO:tasks.rados.rados.2.smithi204.stdout:2295: finishing write tid 4 to smithi20457400-33 2024-09-16T05:19:12.898 INFO:tasks.rados.rados.2.smithi204.stdout:2295: finishing write tid 5 to smithi20457400-33 2024-09-16T05:19:12.898 INFO:tasks.rados.rados.2.smithi204.stdout:2295: finishing write tid 6 to smithi20457400-33 2024-09-16T05:19:12.898 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 841 (ObjNum 823 snap 240 seq_num 823) dirty exists 2024-09-16T05:19:12.898 INFO:tasks.rados.rados.2.smithi204.stdout:2295: left oid 33 (ObjNum 823 snap 240 seq_num 823) 2024-09-16T05:19:12.898 INFO:tasks.rados.rados.2.smithi204.stdout:2296: finishing write tid 1 to smithi20457400-23 2024-09-16T05:19:12.898 INFO:tasks.rados.rados.2.smithi204.stdout:2296: finishing write tid 2 to smithi20457400-23 2024-09-16T05:19:12.899 INFO:tasks.rados.rados.2.smithi204.stdout:2296: finishing write tid 3 to smithi20457400-23 2024-09-16T05:19:12.899 INFO:tasks.rados.rados.2.smithi204.stdout:2296: finishing write tid 4 to smithi20457400-23 2024-09-16T05:19:12.899 INFO:tasks.rados.rados.2.smithi204.stdout:2296: finishing write tid 5 to smithi20457400-23 2024-09-16T05:19:12.899 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 712 (ObjNum 824 snap 240 seq_num 824) dirty exists 2024-09-16T05:19:12.899 INFO:tasks.rados.rados.2.smithi204.stdout:2296: left oid 23 (ObjNum 824 snap 240 seq_num 824) 2024-09-16T05:19:12.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:12.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:12.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:13.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:13.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:12 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:19:13.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:12 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:19:13.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:12 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:19:13.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:12 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:19:13.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:12 smithi196 ceph-mon[30802]: 2.35 scrub starts 2024-09-16T05:19:13.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:12 smithi196 ceph-mon[30802]: 2.35 scrub ok 2024-09-16T05:19:13.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:13.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:13.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:12 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:19:13.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:12 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:19:13.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:12 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:19:13.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:12 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:19:13.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:12 smithi032 ceph-mon[27388]: 2.35 scrub starts 2024-09-16T05:19:13.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:12 smithi032 ceph-mon[27388]: 2.35 scrub ok 2024-09-16T05:19:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:12 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:19:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:12 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:19:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:12 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:19:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:12 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:19:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:12 smithi204 ceph-mon[30933]: 2.35 scrub starts 2024-09-16T05:19:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:12 smithi204 ceph-mon[30933]: 2.35 scrub ok 2024-09-16T05:19:13.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:13.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:13.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:13.255 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:13.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:13.363 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:13.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:13.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:19:13.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:13.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:13.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:13.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:13.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:13.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:13.658 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:13.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:13.742 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:13.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:13.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:13.851 INFO:tasks.rados.rados.2.smithi204.stdout:2295: done (3 left) 2024-09-16T05:19:13.852 INFO:tasks.rados.rados.2.smithi204.stdout:2296: done (2 left) 2024-09-16T05:19:13.852 INFO:tasks.rados.rados.2.smithi204.stdout:2297: done (1 left) 2024-09-16T05:19:13.852 INFO:tasks.rados.rados.2.smithi204.stdout:2298: done (0 left) 2024-09-16T05:19:13.852 INFO:tasks.rados.rados.2.smithi204.stdout:2299: rollback oid 41 current snap is 241 2024-09-16T05:19:13.852 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 41 to 232 2024-09-16T05:19:13.852 INFO:tasks.rados.rados.2.smithi204.stdout:2300: snap_create 2024-09-16T05:19:13.855 INFO:tasks.rados.rados.2.smithi204.stdout:2299: finishing rollback tid 0 to smithi20457400-41 2024-09-16T05:19:13.855 INFO:tasks.rados.rados.2.smithi204.stdout:2299: finishing rollback tid 1 to smithi20457400-41 2024-09-16T05:19:13.855 INFO:tasks.rados.rados.2.smithi204.stdout:2299: finishing rollback tid 2 to smithi20457400-41 2024-09-16T05:19:13.855 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 790 (ObjNum 725 snap 209 seq_num 725) dirty exists 2024-09-16T05:19:13.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:13.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:13.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:13.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:13.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:14.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: pgmap v1065: 107 pgs: 107 active+clean; 345 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-16T05:19:14.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 1.d scrub starts 2024-09-16T05:19:14.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 1.d scrub ok 2024-09-16T05:19:14.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:19:14.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:19:14.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: osdmap e603: 12 total, 12 up, 12 in 2024-09-16T05:19:14.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:19:14.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:19:14.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:19:14.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:19:14.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:19:14.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:19:14.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 1.5 deep-scrub starts 2024-09-16T05:19:14.009 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:13 smithi032 ceph-mon[27388]: 1.5 deep-scrub ok 2024-09-16T05:19:14.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: pgmap v1065: 107 pgs: 107 active+clean; 345 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-16T05:19:14.077 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 1.d scrub starts 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 1.d scrub ok 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: osdmap e603: 12 total, 12 up, 12 in 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:19:14.078 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 1.5 deep-scrub starts 2024-09-16T05:19:14.079 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:13 smithi196 ceph-mon[30802]: 1.5 deep-scrub ok 2024-09-16T05:19:14.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:14.110 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:14.110 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:14.116 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:14.117 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:14.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:14.138 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:14.138 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:14.158 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:14.159 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: pgmap v1065: 107 pgs: 107 active+clean; 345 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2024-09-16T05:19:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 1.d scrub starts 2024-09-16T05:19:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 1.d scrub ok 2024-09-16T05:19:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:19:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:19:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: osdmap e603: 12 total, 12 up, 12 in 2024-09-16T05:19:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:19:14.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:19:14.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:19:14.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:19:14.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:19:14.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:19:14.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 1.5 deep-scrub starts 2024-09-16T05:19:14.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:13 smithi204 ceph-mon[30933]: 1.5 deep-scrub ok 2024-09-16T05:19:14.179 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:14.179 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:14.200 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:14.201 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:14.221 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:14.222 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:14.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:14.243 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:14.243 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:14.246 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:14.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:14.264 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:14.265 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:14.286 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:14.308 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:14.330 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:14.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:14.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:14.352 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:14.352 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:14.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:14.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:14.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:14.499 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:14.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:14.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:14.623 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:14.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:14.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:14.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:14.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:14.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:19:14.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:14.857 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T05:19:14.862 INFO:tasks.rados.rados.2.smithi204.stdout:2299: done (1 left) 2024-09-16T05:19:14.862 INFO:tasks.rados.rados.2.smithi204.stdout:2300: done (0 left) 2024-09-16T05:19:14.862 INFO:tasks.rados.rados.2.smithi204.stdout:2301: write (excl) oid 11 current snap is 242 2024-09-16T05:19:14.862 INFO:tasks.rados.rados.2.smithi204.stdout:2301: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2024-09-16T05:19:14.869 INFO:tasks.rados.rados.2.smithi204.stdout:2301: writing smithi20457400-11 from 665775 to 1312324 tid 1 2024-09-16T05:19:14.876 INFO:tasks.rados.rados.2.smithi204.stdout:2301: writing smithi20457400-11 from 2110468 to 2609752 tid 2 2024-09-16T05:19:14.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:14.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:14.882 INFO:tasks.rados.rados.2.smithi204.stdout:2301: writing smithi20457400-11 from 3155969 to 3598909 tid 3 2024-09-16T05:19:14.882 INFO:tasks.rados.rados.2.smithi204.stdout:2302: write oid 39 current snap is 242 2024-09-16T05:19:14.882 INFO:tasks.rados.rados.2.smithi204.stdout:2301: finishing write tid 1 to smithi20457400-11 2024-09-16T05:19:14.882 INFO:tasks.rados.rados.2.smithi204.stdout:2301: finishing write tid 2 to smithi20457400-11 2024-09-16T05:19:14.883 INFO:tasks.rados.rados.2.smithi204.stdout:2302: seq_num 826 ranges {682582=721798,2168950=642942,3583392=299190} 2024-09-16T05:19:14.890 INFO:tasks.rados.rados.2.smithi204.stdout:2302: writing smithi20457400-39 from 682582 to 1404380 tid 1 2024-09-16T05:19:14.897 INFO:tasks.rados.rados.2.smithi204.stdout:2302: writing smithi20457400-39 from 2168950 to 2811892 tid 2 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2302: writing smithi20457400-39 from 3583392 to 3882582 tid 3 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2303: rollback oid 24 current snap is 242 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 230 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2301: finishing write tid 3 to smithi20457400-11 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2301: finishing write tid 4 to smithi20457400-11 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2301: finishing write tid 5 to smithi20457400-11 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 905 (ObjNum 825 snap 242 seq_num 825) dirty exists 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2301: left oid 11 (ObjNum 825 snap 242 seq_num 825) 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2302: finishing write tid 1 to smithi20457400-39 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2301: done (2 left) 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2304: read oid 38 snap -1 2024-09-16T05:19:14.905 INFO:tasks.rados.rados.2.smithi204.stdout:2304: expect deleted 2024-09-16T05:19:14.906 INFO:tasks.rados.rados.2.smithi204.stdout:2305: write oid 30 current snap is 242 2024-09-16T05:19:14.906 INFO:tasks.rados.rados.2.smithi204.stdout:2302: finishing write tid 2 to smithi20457400-39 2024-09-16T05:19:14.906 INFO:tasks.rados.rados.2.smithi204.stdout:2305: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2024-09-16T05:19:14.912 INFO:tasks.rados.rados.2.smithi204.stdout:2305: writing smithi20457400-30 from 699389 to 1496436 tid 1 2024-09-16T05:19:14.918 INFO:tasks.rados.rados.2.smithi204.stdout:2305: writing smithi20457400-30 from 2111079 to 2614032 tid 2 2024-09-16T05:19:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:14 smithi032 ceph-mon[27388]: osdmap e604: 12 total, 12 up, 12 in 2024-09-16T05:19:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3403508011' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:19:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1460070186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2305: writing smithi20457400-30 from 3094462 to 3676299 tid 3 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2306: rollback oid 4 current snap is 242 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 240 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2302: finishing write tid 3 to smithi20457400-39 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2302: finishing write tid 4 to smithi20457400-39 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2302: finishing write tid 5 to smithi20457400-39 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 910 (ObjNum 826 snap 242 seq_num 826) dirty exists 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2302: left oid 39 (ObjNum 826 snap 242 seq_num 826) 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2303: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2303: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2302: done (4 left) 2024-09-16T05:19:14.926 INFO:tasks.rados.rados.2.smithi204.stdout:2304: done (3 left) 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:2307: rollback oid 36 current snap is 242 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 36 to 193 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:2303: finishing rollback tid 2 to smithi20457400-24 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 602 (ObjNum 784 snap 229 seq_num 784) dirty exists 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:2305: finishing write tid 1 to smithi20457400-30 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:2305: finishing write tid 2 to smithi20457400-30 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:2303: done (3 left) 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:2308: read oid 43 snap -1 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:2308: expect (ObjNum 792 snap 230 seq_num 792) 2024-09-16T05:19:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:2309: snap_remove snap 239 2024-09-16T05:19:14.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:14.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:14 smithi196 ceph-mon[30802]: osdmap e604: 12 total, 12 up, 12 in 2024-09-16T05:19:14.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3403508011' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:19:14.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1460070186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:15.000 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:19:15.004 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:15.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:15.106 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:15.131 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:15.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:15.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:14 smithi204 ceph-mon[30933]: osdmap e604: 12 total, 12 up, 12 in 2024-09-16T05:19:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3403508011' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:19:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1460070186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:15.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:19:15.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:15.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:15.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:15.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:15.387 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:15.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:15.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:15.503 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:19:15.503 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:19:15.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:19:15.503 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:19:15.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.503 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:24 2.0 2:06458cf1:::smithi20457400-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 512.44920523400003, 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103238884, 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 24 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052222+0000", 2024-09-16T05:19:15.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8799999999999998e-07 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052223+0000", 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2950000000000001e-06 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052227+0000", 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5240000000000001e-06 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.052230+0000", 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2160000000000002e-06 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053127+0000", 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089734199999999997 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.053182+0000", 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4935999999999997e-05 2024-09-16T05:19:15.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155331+0000", 2024-09-16T05:19:15.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.102149077 2024-09-16T05:19:15.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155389+0000", 2024-09-16T05:19:15.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7691e-05 2024-09-16T05:19:15.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:10:43.155460+0000", 2024-09-16T05:19:15.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1514999999999995e-05 2024-09-16T05:19:15.509 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1675 2.1c 2:3d4bedc8:::smithi20457400-22:head [setxattr _header (57) in=7b,truncate 2151329] snapc 46=[46,45,44,42,41,40,3f,3e,3c,3a,39,37,35,33,27] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 388.64180753699998, 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22736652099999999, 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.510 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1675 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859619+0000", 2024-09-16T05:19:15.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859620+0000", 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3099999999999997e-07 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859624+0000", 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1949999999999996e-06 2024-09-16T05:19:15.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859637+0000", 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3118999999999999e-05 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859641+0000", 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.036e-06 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860356+0000", 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00071436999999999998 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860417+0000", 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1253999999999993e-05 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 8", 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860525+0000", 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107742 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:15.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860988+0000", 2024-09-16T05:19:15.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046300399999999999 2024-09-16T05:19:15.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086882+0000", 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.225894659 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086930+0000", 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8010999999999998e-05 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086986+0000", 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5500000000000001e-05 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1676 2.1c 2:3d4bedc8:::smithi20457400-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e235)", 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 388.64176403499999, 2024-09-16T05:19:15.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.227504338, 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1676 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859663+0000", 2024-09-16T05:19:15.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9999999999999998e-07 2024-09-16T05:19:15.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859666+0000", 2024-09-16T05:19:15.520 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7819999999999998e-06 2024-09-16T05:19:15.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.521 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859699+0000", 2024-09-16T05:19:15.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2950000000000001e-05 2024-09-16T05:19:15.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.521 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.521 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.859707+0000", 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3459999999999994e-06 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860761+0000", 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001053315 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.860862+0000", 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010108 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087019+0000", 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.226157467 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087091+0000", 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1536000000000001e-05 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.087167+0000", 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6361999999999998e-05 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:1675 2.1c e235/232 ondisk, result = 0)", 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 388.64040839299997, 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22597392799999999, 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.524 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861018+0000", 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.525 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861019+0000", 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:46.861020+0000", 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7969999999999999e-06 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086793+0000", 2024-09-16T05:19:15.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.22577269899999999 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086803+0000", 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5459999999999997e-06 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086867+0000", 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4447000000000001e-05 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086876+0000", 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9760000000000001e-06 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:47.086992+0000", 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116212 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1824 2.8 2:1030fb5b:::smithi20457400-27:head [write 521083~714336,stat] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.55627942400002, 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.81065307900000005, 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1824 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.529 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945150+0000", 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1260000000000001e-06 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945906+0000", 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075638699999999997 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945927+0000", 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0849e-05 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945936+0000", 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3910000000000004e-06 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946023+0000", 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6547999999999999e-05 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946035+0000", 2024-09-16T05:19:15.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2249000000000001e-05 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752337+0000", 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80630184599999999 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752493+0000", 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000156546 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.533 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752675+0000", 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018172699999999999 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755628+0000", 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029533929999999999 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755689+0000", 2024-09-16T05:19:15.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0949000000000002e-05 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755745+0000", 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5591999999999999e-05 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755800+0000", 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5476000000000002e-05 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.535 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1825 2.8 2:1030fb5b:::smithi20457400-27:head [write 1724715~555421] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.55063387600001, 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80511571699999995, 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.536 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1825 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950794+0000", 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951237+0000", 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00044250000000000002 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951251+0000", 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4381e-05 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951276+0000", 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5233999999999999e-05 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951378+0000", 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010173799999999999 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.539 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951390+0000", 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2274e-05 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.753903+0000", 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80251229300000004 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754002+0000", 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9427999999999997e-05 2024-09-16T05:19:15.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754132+0000", 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130207 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755633+0000", 2024-09-16T05:19:15.541 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015010100000000001 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755844+0000", 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021044799999999999 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755880+0000", 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6007999999999998e-05 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.542 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755909+0000", 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8941e-05 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1826 2.8 2:1030fb5b:::smithi20457400-27:head [setxattr _header (57) in=7b,truncate 2521083] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.55013458299999, 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80476365000000005, 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.543 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1826 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951293+0000", 2024-09-16T05:19:15.545 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951296+0000", 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2799999999999999e-06 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951301+0000", 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6879999999999998e-06 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951309+0000", 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7810000000000005e-06 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951421+0000", 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112333 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:19:15.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951426+0000", 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6650000000000002e-06 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754913+0000", 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80348719099999999 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754982+0000", 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9558e-05 2024-09-16T05:19:15.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755084+0000", 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101894 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755638+0000", 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055423099999999997 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755988+0000", 2024-09-16T05:19:15.549 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034971600000000002 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756021+0000", 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2461e-05 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756056+0000", 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5278999999999999e-05 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1827 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.55011231399999, 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.804892304, 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1827 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.551 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8700000000000002e-07 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951317+0000", 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7460000000000001e-06 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951321+0000", 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3250000000000001e-06 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951325+0000", 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1710000000000004e-06 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951464+0000", 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139111 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951475+0000", 2024-09-16T05:19:15.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1045e-05 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755294+0000", 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80381846999999995 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755351+0000", 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7589e-05 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756086+0000", 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073505599999999999 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756138+0000", 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1397000000000003e-05 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756207+0000", 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9107000000000001e-05 2024-09-16T05:19:15.556 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1834 2.8 2:12e92f6c:::smithi20457400-1:head [stat,delete] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.303211632, 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55807496000000001, 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1834 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198218+0000", 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.357e-06 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198229+0000", 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1677e-05 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198254+0000", 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4312e-05 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198266+0000", 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2687000000000001e-05 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198366+0000", 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9698000000000007e-05 2024-09-16T05:19:15.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198378+0000", 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2106000000000001e-05 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755379+0000", 2024-09-16T05:19:15.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55700064400000004 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755448+0000", 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8848000000000002e-05 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756290+0000", 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084263100000000004 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.9,QREQUEST,e:256,from:1,reservation_nonce:2)", 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 367.20119019700002, 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21542001499999999, 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300237+0000", 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300239+0000", 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0540000000000002e-06 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300249+0000", 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0369e-05 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300304+0000", 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5247000000000001e-05 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300312+0000", 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.089e-06 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300391+0000", 2024-09-16T05:19:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8628999999999995e-05 2024-09-16T05:19:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:19:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.300397+0000", 2024-09-16T05:19:15.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8889999999999999e-06 2024-09-16T05:19:15.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.568 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.568 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515506+0000", 2024-09-16T05:19:15.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21510901199999999 2024-09-16T05:19:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:15.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:15.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:15.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515527+0000", 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0981000000000001e-05 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:08.515657+0000", 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012974500000000001 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.571 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2255 2.1d e290/282 2:ba754fcb:::smithi20457400-30:head v 290'477, mlcod=284'471)", 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 333.81661688499997, 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53570759499999998, 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684811+0000", 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3600000000000002e-07 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684819+0000", 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5760000000000006e-06 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219817+0000", 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53499761300000004 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219843+0000", 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5908e-05 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219921+0000", 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7811000000000004e-05 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219944+0000", 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3237000000000001e-05 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220484+0000", 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00053966300000000001 2024-09-16T05:19:15.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220518+0000", 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3850999999999998e-05 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.1d,QREQUEST,e:290,from:10,reservation_nonce:2)", 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:19:15.576 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 333.28163784600002, 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.202128169, 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219789+0000", 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219790+0000", 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.452e-06 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219798+0000", 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1670000000000002e-06 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421791+0000", 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20199293700000001 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421799+0000", 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5560000000000008e-06 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421864+0000", 2024-09-16T05:19:15.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5235999999999997e-05 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421881+0000", 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6742e-05 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.421917+0000", 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6078999999999998e-05 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.580 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.10.0:85 2.1d e300/291 2:ba754fcb:::smithi20457400-30:5f v 300'498, mlcod=300'494)", 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 323.74279024200001, 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.51609034499999995, 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.16e-07 2024-09-16T05:19:15.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758644+0000", 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1519999999999997e-06 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273721+0000", 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.51507724799999999 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273736+0000", 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5449000000000001e-05 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273906+0000", 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016944 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273962+0000", 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6209999999999999e-05 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.274683+0000", 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072146099999999996 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.274727+0000", 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3569000000000002e-05 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:382 2.2e e345/278 ondisk, result = 0)", 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:54.871554+0000", 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 260.629872439, 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.29655793200000002, 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.586 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.871554+0000", 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.871554+0000", 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.871555+0000", 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6800000000000001e-07 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:54.871558+0000", 2024-09-16T05:19:15.588 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4819999999999999e-06 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.167809+0000", 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.29625046300000002 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.167824+0000", 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5244e-05 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.589 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.167945+0000", 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012105 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.167965+0000", 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0183e-05 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.168112+0000", 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014704200000000001 2024-09-16T05:19:15.590 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:390 2.3c e346/293 ondisk, result = 0)", 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 259.793844896, 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62913221100000005, 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.591 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707583+0000", 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.47e-07 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707587+0000", 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7010000000000004e-06 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336491+0000", 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62890340600000005 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336497+0000", 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4640000000000003e-06 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336590+0000", 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2163000000000005e-05 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336599+0000", 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.054e-06 2024-09-16T05:19:15.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336714+0000", 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011557600000000001 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4414 2.3f 2:fc733d2f:::smithi20457400-45:head [copy-from ver 329] snapc b9=[b9,b8] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 106.758424178, 2024-09-16T05:19:15.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95144123899999999, 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4414 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:19:15.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743006+0000", 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2550000000000002e-06 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743021+0000", 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5024999999999999e-05 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743048+0000", 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6842000000000002e-05 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743063+0000", 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4969e-05 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743154+0000", 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1438000000000001e-05 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743163+0000", 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1740000000000006e-06 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.744798+0000", 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016347600000000001 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.744906+0000", 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107724 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749771+0000", 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048651129999999999 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.753713+0000", 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039421539999999998 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694348+0000", 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94063521699999997 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:19:15.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694398+0000", 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9518999999999998e-05 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694444+0000", 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6048999999999999e-05 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGRecoveryDeleteReply(2.1d e531,530 2:ba21f1a9:::smithi20457400-14:37,293'480)", 2024-09-16T05:19:15.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:01.674309+0000", 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 73.827117358999999, 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.204139719, 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.674309+0000", 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.674309+0000", 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.674316+0000", 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2500000000000003e-06 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.674326+0000", 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0115999999999999e-05 2024-09-16T05:19:15.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.877833+0000", 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20350669599999999 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.877842+0000", 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4099999999999997e-06 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.878208+0000", 2024-09-16T05:19:15.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036640300000000001 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.878449+0000", 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024084399999999999 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGRecoveryDeleteReply(2.1d e532,530 2:ba7e58cf:::smithi20457400-26:4a,340'541)", 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:01.904541+0000", 2024-09-16T05:19:15.606 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 73.596885693999994, 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32384723500000001, 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.904541+0000", 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.904541+0000", 2024-09-16T05:19:15.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.904543+0000", 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4419999999999999e-06 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:01.904547+0000", 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3370000000000001e-06 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:02.227988+0000", 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32344094400000001 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:02.228000+0000", 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1831000000000001e-05 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:02.228089+0000", 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9815999999999995e-05 2024-09-16T05:19:15.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:02.228388+0000", 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029886499999999999 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(1.9 e106 from shard 9)", 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:19:15.610 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 53.023653695, 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.63215181600000003, 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477774+0000", 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9999999999999997e-07 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477777+0000", 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5949999999999999e-06 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109802+0000", 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.63202490499999997 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109829+0000", 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6750999999999998e-05 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109891+0000", 2024-09-16T05:19:15.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1661999999999993e-05 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109900+0000", 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6569999999999992e-06 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109925+0000", 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4646000000000001e-05 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:958 2.2e e588/278 ondisk, result = 0)", 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 17.708880218000001, 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92946010700000004, 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:19:15.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.06e-07 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792556+0000", 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.208e-06 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721793+0000", 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92923791899999997 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721804+0000", 2024-09-16T05:19:15.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0936e-05 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721875+0000", 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0863000000000002e-05 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721885+0000", 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5510000000000005e-06 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:19:15.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.722007+0000", 2024-09-16T05:19:15.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012192399999999999 2024-09-16T05:19:15.619 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.619 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.619 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.619 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:19:15.619 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:19:15.619 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:19:15.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:15.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-16T05:19:15.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:15.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:15.776 INFO:tasks.rados.rados.2.smithi204.stdout:2309: done (4 left) 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2310: write (excl) oid 50 current snap is 242 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2306: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 998 (ObjNum 2353 snap 0 seq_num 4282180192) dirty dne 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2307: finishing rollback tid 1 to smithi20457400-36 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2307: finishing rollback tid 2 to smithi20457400-36 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 718 (ObjNum 661 snap 192 seq_num 661) dirty exists 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2305: finishing write tid 3 to smithi20457400-30 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2305: finishing write tid 4 to smithi20457400-30 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2305: finishing write tid 5 to smithi20457400-30 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 1050 (ObjNum 827 snap 242 seq_num 827) dirty exists 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2305: left oid 30 (ObjNum 827 snap 242 seq_num 827) 2024-09-16T05:19:15.777 INFO:tasks.rados.rados.2.smithi204.stdout:2308: expect (ObjNum 792 snap 230 seq_num 792) 2024-09-16T05:19:15.791 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:15.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:15.822 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:15.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:15.886 INFO:tasks.rados.rados.2.smithi204.stdout:2310: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2024-09-16T05:19:15.893 INFO:tasks.rados.rados.2.smithi204.stdout:2310: writing smithi20457400-50 from 716196 to 1188492 tid 1 2024-09-16T05:19:15.899 INFO:tasks.rados.rados.2.smithi204.stdout:2310: writing smithi20457400-50 from 1769561 to 2416172 tid 2 2024-09-16T05:19:15.907 INFO:tasks.rados.rados.2.smithi204.stdout:2310: writing smithi20457400-50 from 3121885 to 3914994 tid 3 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2305: done (4 left) 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2306: done (3 left) 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2307: done (2 left) 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2308: done (1 left) 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2311: read oid 27 snap -1 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2311: expect (ObjNum 793 snap 230 seq_num 793) 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2312: write (excl) oid 3 current snap is 242 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2310: finishing write tid 1 to smithi20457400-50 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2310: finishing write tid 2 to smithi20457400-50 2024-09-16T05:19:15.908 INFO:tasks.rados.rados.2.smithi204.stdout:2312: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2024-09-16T05:19:15.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: pgmap v1068: 107 pgs: 107 active+clean; 355 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 4.4 MiB/s wr, 25 op/s 2024-09-16T05:19:15.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3403508011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:19:15.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: osdmap e605: 12 total, 12 up, 12 in 2024-09-16T05:19:15.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: 2.3d deep-scrub starts 2024-09-16T05:19:15.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: 2.3d deep-scrub ok 2024-09-16T05:19:15.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:19:15.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:19:15.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:19:15.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:19:15.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:15 smithi032 ceph-mon[27388]: osdmap e606: 12 total, 12 up, 12 in 2024-09-16T05:19:15.915 INFO:tasks.rados.rados.2.smithi204.stdout:2312: writing smithi20457400-3 from 733003 to 1396901 tid 1 2024-09-16T05:19:15.921 INFO:tasks.rados.rados.2.smithi204.stdout:2312: writing smithi20457400-3 from 1944396 to 2451018 tid 2 2024-09-16T05:19:15.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:15.927 INFO:tasks.rados.rados.2.smithi204.stdout:2312: writing smithi20457400-3 from 2865661 to 3470042 tid 3 2024-09-16T05:19:15.927 INFO:tasks.rados.rados.2.smithi204.stdout:2313: snap_remove snap 215 2024-09-16T05:19:15.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:15.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:16.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:16.072 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:16.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: pgmap v1068: 107 pgs: 107 active+clean; 355 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 4.4 MiB/s wr, 25 op/s 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3403508011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: osdmap e605: 12 total, 12 up, 12 in 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: 2.3d deep-scrub starts 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: 2.3d deep-scrub ok 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:19:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:15 smithi204 ceph-mon[30933]: osdmap e606: 12 total, 12 up, 12 in 2024-09-16T05:19:16.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:16.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:16.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:16.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: pgmap v1068: 107 pgs: 107 active+clean; 355 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 4.4 MiB/s wr, 25 op/s 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3403508011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: osdmap e605: 12 total, 12 up, 12 in 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: 2.3d deep-scrub starts 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: 2.3d deep-scrub ok 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:19:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:15 smithi196 ceph-mon[30802]: osdmap e606: 12 total, 12 up, 12 in 2024-09-16T05:19:16.296 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:16.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:16.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:16.370 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:19:16.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:16.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:16.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:16.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:16.565 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:16.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:16.628 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:16.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:16.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:16.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:16.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:16.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:16.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:16.880 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:19:16.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:16.917 INFO:tasks.rados.rados.2.smithi204.stdout:2313: done (3 left) 2024-09-16T05:19:16.917 INFO:tasks.rados.rados.2.smithi204.stdout:2314: write (excl) oid 34 current snap is 242 2024-09-16T05:19:16.917 INFO:tasks.rados.rados.2.smithi204.stdout:2310: finishing write tid 3 to smithi20457400-50 2024-09-16T05:19:16.917 INFO:tasks.rados.rados.2.smithi204.stdout:2314: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2024-09-16T05:19:16.925 INFO:tasks.rados.rados.2.smithi204.stdout:2314: writing smithi20457400-34 from 749810 to 1488957 tid 1 2024-09-16T05:19:16.932 INFO:tasks.rados.rados.2.smithi204.stdout:2314: writing smithi20457400-34 from 2002878 to 2653158 tid 2 2024-09-16T05:19:16.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:16.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:16.938 INFO:tasks.rados.rados.2.smithi204.stdout:2314: writing smithi20457400-34 from 3293084 to 3708737 tid 3 2024-09-16T05:19:16.938 INFO:tasks.rados.rados.2.smithi204.stdout:2315: write oid 40 current snap is 242 2024-09-16T05:19:16.938 INFO:tasks.rados.rados.2.smithi204.stdout:2310: finishing write tid 4 to smithi20457400-50 2024-09-16T05:19:16.938 INFO:tasks.rados.rados.2.smithi204.stdout:2315: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-09-16T05:19:16.944 INFO:tasks.rados.rados.2.smithi204.stdout:2315: writing smithi20457400-40 from 766617 to 1181013 tid 1 2024-09-16T05:19:16.951 INFO:tasks.rados.rados.2.smithi204.stdout:2315: writing smithi20457400-40 from 1945007 to 2455298 tid 2 2024-09-16T05:19:16.959 INFO:tasks.rados.rados.2.smithi204.stdout:2315: writing smithi20457400-40 from 3204154 to 3831079 tid 3 2024-09-16T05:19:16.959 INFO:tasks.rados.rados.2.smithi204.stdout:2316: write (excl) oid 13 current snap is 242 2024-09-16T05:19:16.959 INFO:tasks.rados.rados.2.smithi204.stdout:2310: finishing write tid 5 to smithi20457400-50 2024-09-16T05:19:16.959 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 772 (ObjNum 828 snap 242 seq_num 828) dirty exists 2024-09-16T05:19:16.959 INFO:tasks.rados.rados.2.smithi204.stdout:2310: left oid 50 (ObjNum 828 snap 242 seq_num 828) 2024-09-16T05:19:16.959 INFO:tasks.rados.rados.2.smithi204.stdout:2312: finishing write tid 1 to smithi20457400-3 2024-09-16T05:19:16.960 INFO:tasks.rados.rados.2.smithi204.stdout:2311: expect (ObjNum 793 snap 230 seq_num 793) 2024-09-16T05:19:17.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:19:17.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:17.058 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:19:17.058 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:19:17.058 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: 2.43 scrub starts 2024-09-16T05:19:17.058 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: 2.43 scrub ok 2024-09-16T05:19:17.058 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.059 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.060 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.061 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.062 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.064 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.065 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.067 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.069 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.070 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.071 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.072 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.073 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.074 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:17.077 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:17.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:17.080 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.080 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.087 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.089 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.090 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.093 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.094 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.097 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.098 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.099 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.102 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.105 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.108 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.110 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.111 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.112 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.113 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.114 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1649826107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.115 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.116 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.119 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.120 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.121 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.125 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.129 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.135 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.136 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:16 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.158 INFO:tasks.rados.rados.2.smithi204.stdout:2312: finishing write tid 2 to smithi20457400-3 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2312: finishing write tid 3 to smithi20457400-3 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2312: finishing write tid 4 to smithi20457400-3 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2312: finishing write tid 5 to smithi20457400-3 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 541 (ObjNum 829 snap 242 seq_num 829) dirty exists 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2312: left oid 3 (ObjNum 829 snap 242 seq_num 829) 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2314: finishing write tid 1 to smithi20457400-34 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2314: finishing write tid 2 to smithi20457400-34 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2314: finishing write tid 3 to smithi20457400-34 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2314: finishing write tid 4 to smithi20457400-34 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2314: finishing write tid 5 to smithi20457400-34 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 705 (ObjNum 830 snap 242 seq_num 830) dirty exists 2024-09-16T05:19:17.159 INFO:tasks.rados.rados.2.smithi204.stdout:2314: left oid 34 (ObjNum 830 snap 242 seq_num 830) 2024-09-16T05:19:17.160 INFO:tasks.rados.rados.2.smithi204.stdout:2315: finishing write tid 1 to smithi20457400-40 2024-09-16T05:19:17.160 INFO:tasks.rados.rados.2.smithi204.stdout:2315: finishing write tid 2 to smithi20457400-40 2024-09-16T05:19:17.160 INFO:tasks.rados.rados.2.smithi204.stdout:2315: finishing write tid 3 to smithi20457400-40 2024-09-16T05:19:17.160 INFO:tasks.rados.rados.2.smithi204.stdout:2316: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-09-16T05:19:17.165 INFO:tasks.rados.rados.2.smithi204.stdout:2316: writing smithi20457400-13 from 783424 to 1273069 tid 1 2024-09-16T05:19:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:19:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:19:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: 2.43 scrub starts 2024-09-16T05:19:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: 2.43 scrub ok 2024-09-16T05:19:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.174 INFO:tasks.rados.rados.2.smithi204.stdout:2316: writing smithi20457400-13 from 2003489 to 2657438 tid 2 2024-09-16T05:19:17.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.177 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.179 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.182 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.185 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.186 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.200 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.201 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.202 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.204 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.205 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.207 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.207 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.207 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.207 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.209 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.211 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.212 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.213 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.214 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.214 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.214 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.214 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.214 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.214 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.214 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:17.216 INFO:tasks.rados.rados.2.smithi204.stdout:2316: writing smithi20457400-13 from 3231577 to 3786127 tid 3 2024-09-16T05:19:17.216 INFO:tasks.rados.rados.2.smithi204.stdout:2310: done (5 left) 2024-09-16T05:19:17.216 INFO:tasks.rados.rados.2.smithi204.stdout:2311: done (4 left) 2024-09-16T05:19:17.216 INFO:tasks.rados.rados.2.smithi204.stdout:2312: done (3 left) 2024-09-16T05:19:17.216 INFO:tasks.rados.rados.2.smithi204.stdout:2314: done (2 left) 2024-09-16T05:19:17.216 INFO:tasks.rados.rados.2.smithi204.stdout:2317: snap_remove snap 232 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.218 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.219 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.222 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.223 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.224 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.225 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.228 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1649826107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:19:17.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.232 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.237 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.238 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.240 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.243 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.248 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.249 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:16 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.254 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:19:17.254 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:19:17.254 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:19:17.254 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: 2.43 scrub starts 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: 2.43 scrub ok 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.260 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.261 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.264 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.268 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.269 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.270 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.271 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.273 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.274 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.275 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.276 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.279 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.280 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.281 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.282 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.287 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.288 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.290 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.291 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.292 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.293 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.294 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.298 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.299 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.300 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.302 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.303 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.304 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1649826107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.308 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.309 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.311 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.319 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.320 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.323 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.326 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.328 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.329 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=259467585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=3107188591,config=1549860826,health=3897446751,kv=2883009636}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=2902852476}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=592595954}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=22,logm=78} crc {kv=3867390147,logm=2095819331}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156899726}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212855762}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616061414}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=397066775}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749142081}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3210184186}) 2024-09-16T05:19:17.330 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3201905390}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551641980}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870627906}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785336304}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2214795685}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415106850}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1212654022}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3889457660}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077551285}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346535390}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3277939667}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512881990}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066678854}) 2024-09-16T05:19:17.331 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1803939497}) 2024-09-16T05:19:17.332 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334867560}) 2024-09-16T05:19:17.332 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:16 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40170865}) 2024-09-16T05:19:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:17.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:17.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:17.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:17.455 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:17.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:17.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:17.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:17.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:17.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:17.704 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:17.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:17.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:17.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:17.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:17.884 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T05:19:17.884 INFO:tasks.rados.rados.2.smithi204.stdout:2317: done (2 left) 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2318: read oid 24 snap -1 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2318: expect (ObjNum 784 snap 229 seq_num 784) 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2319: read oid 48 snap -1 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2319: expect deleted 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2320: delete oid 31 current snap is 242 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2315: finishing write tid 4 to smithi20457400-40 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2315: finishing write tid 5 to smithi20457400-40 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 771 (ObjNum 831 snap 242 seq_num 831) dirty exists 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2315: left oid 40 (ObjNum 831 snap 242 seq_num 831) 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2316: finishing write tid 1 to smithi20457400-13 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2316: finishing write tid 2 to smithi20457400-13 2024-09-16T05:19:17.885 INFO:tasks.rados.rados.2.smithi204.stdout:2316: finishing write tid 3 to smithi20457400-13 2024-09-16T05:19:17.886 INFO:tasks.rados.rados.2.smithi204.stdout:2316: finishing write tid 4 to smithi20457400-13 2024-09-16T05:19:17.886 INFO:tasks.rados.rados.2.smithi204.stdout:2316: finishing write tid 5 to smithi20457400-13 2024-09-16T05:19:17.886 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 1005 (ObjNum 832 snap 242 seq_num 832) dirty exists 2024-09-16T05:19:17.886 INFO:tasks.rados.rados.2.smithi204.stdout:2316: left oid 13 (ObjNum 832 snap 242 seq_num 832) 2024-09-16T05:19:17.890 INFO:tasks.rados.rados.2.smithi204.stdout:2315: done (4 left) 2024-09-16T05:19:17.890 INFO:tasks.rados.rados.2.smithi204.stdout:2316: done (3 left) 2024-09-16T05:19:17.891 INFO:tasks.rados.rados.2.smithi204.stdout:2319: done (2 left) 2024-09-16T05:19:17.891 INFO:tasks.rados.rados.2.smithi204.stdout:2320: done (1 left) 2024-09-16T05:19:17.891 INFO:tasks.rados.rados.2.smithi204.stdout:2321: write oid 42 current snap is 242 2024-09-16T05:19:17.891 INFO:tasks.rados.rados.2.smithi204.stdout:2321: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-16T05:19:17.897 INFO:tasks.rados.rados.2.smithi204.stdout:2321: writing smithi20457400-42 from 417038 to 1057181 tid 1 2024-09-16T05:19:17.903 INFO:tasks.rados.rados.2.smithi204.stdout:2321: writing smithi20457400-42 from 1720453 to 2017038 tid 2 2024-09-16T05:19:17.903 INFO:tasks.rados.rados.2.smithi204.stdout:2322: read oid 3 snap 227 2024-09-16T05:19:17.903 INFO:tasks.rados.rados.2.smithi204.stdout:2322: expect (ObjNum 767 snap 223 seq_num 767) 2024-09-16T05:19:17.903 INFO:tasks.rados.rados.2.smithi204.stdout:2323: delete oid 30 current snap is 242 2024-09-16T05:19:17.903 INFO:tasks.rados.rados.2.smithi204.stdout:2318: expect (ObjNum 784 snap 229 seq_num 784) 2024-09-16T05:19:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: pgmap v1071: 107 pgs: 107 active+clean; 366 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.9 MiB/s wr, 23 op/s 2024-09-16T05:19:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:19:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:19:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:17.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1649826107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: osdmap e607: 12 total, 12 up, 12 in 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:17.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4155261503}) 2024-09-16T05:19:17.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1775340792}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784489041}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=238978467}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172944129}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977431766}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2736934956}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958224044}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3241360924}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309018518}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=26,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=50} crc {logm=2282939884,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=1568660844}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3362431384}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=61,mon_config_key=39} crc {mgrstat=3593994996,mon_config_key=1791846828}) 2024-09-16T05:19:17.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2784644191}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1007188011}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=34,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=48} crc {mon_config_key=636233725,monmap=2256977890,osd_metadata=3570072447,osd_pg_creating=1704066401,osd_snap=3495385075}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=1293445309,osdmap=836565296}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3252492636}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4229719974}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=213741854}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3806617261}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1246411553}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=437137036}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1784942939}) 2024-09-16T05:19:17.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2165467888}) 2024-09-16T05:19:17.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=527106562}) 2024-09-16T05:19:17.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=71898044}) 2024-09-16T05:19:17.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3214011038}) 2024-09-16T05:19:17.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=2846426164}) 2024-09-16T05:19:17.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3797657982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:19:17.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:17 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:19:17.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:17.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:17.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:18.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:18.057 INFO:tasks.rados.rados.2.smithi204.stdout:2321: finishing write tid 1 to smithi20457400-42 2024-09-16T05:19:18.057 INFO:tasks.rados.rados.2.smithi204.stdout:2321: finishing write tid 2 to smithi20457400-42 2024-09-16T05:19:18.057 INFO:tasks.rados.rados.2.smithi204.stdout:2321: finishing write tid 3 to smithi20457400-42 2024-09-16T05:19:18.057 INFO:tasks.rados.rados.2.smithi204.stdout:2321: finishing write tid 4 to smithi20457400-42 2024-09-16T05:19:18.057 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 758 (ObjNum 834 snap 242 seq_num 834) dirty exists 2024-09-16T05:19:18.057 INFO:tasks.rados.rados.2.smithi204.stdout:2321: left oid 42 (ObjNum 834 snap 242 seq_num 834) 2024-09-16T05:19:18.057 INFO:tasks.rados.rados.2.smithi204.stdout:2322: expect (ObjNum 767 snap 223 seq_num 767) 2024-09-16T05:19:18.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:18.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: pgmap v1071: 107 pgs: 107 active+clean; 366 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.9 MiB/s wr, 23 op/s 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1649826107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: osdmap e607: 12 total, 12 up, 12 in 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:18.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:18.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4155261503}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1775340792}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784489041}) 2024-09-16T05:19:18.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=238978467}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172944129}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977431766}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2736934956}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958224044}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3241360924}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309018518}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=26,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=50} crc {logm=2282939884,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=1568660844}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3362431384}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=61,mon_config_key=39} crc {mgrstat=3593994996,mon_config_key=1791846828}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2784644191}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1007188011}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=34,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=48} crc {mon_config_key=636233725,monmap=2256977890,osd_metadata=3570072447,osd_pg_creating=1704066401,osd_snap=3495385075}) 2024-09-16T05:19:18.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=1293445309,osdmap=836565296}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3252492636}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4229719974}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=213741854}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3806617261}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1246411553}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=437137036}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1784942939}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2165467888}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=527106562}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=71898044}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3214011038}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=2846426164}) 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3797657982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:19:18.175 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:17 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:19:18.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:18.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:18.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:18.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:19:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: pgmap v1071: 107 pgs: 107 active+clean; 366 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.9 MiB/s wr, 23 op/s 2024-09-16T05:19:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:19:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:19:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719182110}) 2024-09-16T05:19:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487042920}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262418301}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1649826107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: osdmap e607: 12 total, 12 up, 12 in 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384616458}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619209185}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077960858}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474114966}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350159100}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=776970518}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988341693}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412258140}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760260284}) 2024-09-16T05:19:18.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257743653}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3936693209}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1961660664}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243893980}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869460803}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517171264}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022830221}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=326707887}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3514681097}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=786017300}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173979596}) 2024-09-16T05:19:18.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946886923}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103983446}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401655885}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4014216241}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4155261503}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1775340792}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784489041}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=238978467}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172944129}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977431766}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2736934956}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958224044}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3241360924}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309018518}) 2024-09-16T05:19:18.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=26,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=50} crc {logm=2282939884,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=1568660844}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3362431384}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=61,mon_config_key=39} crc {mgrstat=3593994996,mon_config_key=1791846828}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2784644191}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1007188011}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=34,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=48} crc {mon_config_key=636233725,monmap=2256977890,osd_metadata=3570072447,osd_pg_creating=1704066401,osd_snap=3495385075}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=1293445309,osdmap=836565296}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3252492636}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4229719974}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=213741854}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3806617261}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1246411553}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=437137036}) 2024-09-16T05:19:18.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1784942939}) 2024-09-16T05:19:18.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2165467888}) 2024-09-16T05:19:18.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=527106562}) 2024-09-16T05:19:18.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=71898044}) 2024-09-16T05:19:18.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3214011038}) 2024-09-16T05:19:18.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=2846426164}) 2024-09-16T05:19:18.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3797657982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:19:18.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:17 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:19:18.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:18.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:18.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:18.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:18.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:18.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:18.458 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:18.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:18.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:18.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:18.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:18.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:18.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:18.698 INFO:tasks.rados.rados.2.smithi204.stdout:2318: done (3 left) 2024-09-16T05:19:18.698 INFO:tasks.rados.rados.2.smithi204.stdout:2321: done (2 left) 2024-09-16T05:19:18.698 INFO:tasks.rados.rados.2.smithi204.stdout:2322: done (1 left) 2024-09-16T05:19:18.698 INFO:tasks.rados.rados.2.smithi204.stdout:2323: done (0 left) 2024-09-16T05:19:18.698 INFO:tasks.rados.rados.2.smithi204.stdout:2324: snap_create 2024-09-16T05:19:18.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-16T05:19:18.719 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:18.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:18.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:18.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:18.900 INFO:tasks.rados.rados.2.smithi204.stdout:2324: done (0 left) 2024-09-16T05:19:18.900 INFO:tasks.rados.rados.2.smithi204.stdout:2325: rollback oid 43 current snap is 243 2024-09-16T05:19:18.901 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 43 to 193 2024-09-16T05:19:18.901 INFO:tasks.rados.rados.2.smithi204.stdout:2326: read oid 33 snap 193 2024-09-16T05:19:18.901 INFO:tasks.rados.rados.2.smithi204.stdout:2326: expect (ObjNum 657 snap 191 seq_num 657) 2024-09-16T05:19:18.901 INFO:tasks.rados.rados.2.smithi204.stdout:2327: delete oid 47 current snap is 243 2024-09-16T05:19:18.904 INFO:tasks.rados.rados.2.smithi204.stdout:2325: finishing rollback tid 0 to smithi20457400-43 2024-09-16T05:19:18.904 INFO:tasks.rados.rados.2.smithi204.stdout:2325: finishing rollback tid 1 to smithi20457400-43 2024-09-16T05:19:18.905 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 610 (ObjNum 49 snap 0 seq_num 1635020658) dirty dne 2024-09-16T05:19:18.905 INFO:tasks.rados.rados.2.smithi204.stdout:2325: done (2 left) 2024-09-16T05:19:18.905 INFO:tasks.rados.rados.2.smithi204.stdout:2327: done (1 left) 2024-09-16T05:19:18.905 INFO:tasks.rados.rados.2.smithi204.stdout:2328: delete oid 9 current snap is 243 2024-09-16T05:19:18.906 INFO:tasks.rados.rados.2.smithi204.stdout:2328: done (1 left) 2024-09-16T05:19:18.906 INFO:tasks.rados.rados.2.smithi204.stdout:2329: write oid 50 current snap is 243 2024-09-16T05:19:18.907 INFO:tasks.rados.rados.2.smithi204.stdout:2329: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-16T05:19:18.912 INFO:tasks.rados.rados.2.smithi204.stdout:2329: writing smithi20457400-50 from 484266 to 1141758 tid 1 2024-09-16T05:19:18.918 INFO:tasks.rados.rados.2.smithi204.stdout:2329: writing smithi20457400-50 from 1554381 to 2084266 tid 2 2024-09-16T05:19:18.918 INFO:tasks.rados.rados.2.smithi204.stdout:2330: copy_from oid 20 from oid 3 current snap is 243 2024-09-16T05:19:18.918 INFO:tasks.rados.rados.2.smithi204.stdout:2331: snap_remove snap 236 2024-09-16T05:19:18.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:19:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: osdmap e608: 12 total, 12 up, 12 in 2024-09-16T05:19:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:19:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:19:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10551160220848944]}]: dispatch 2024-09-16T05:19:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07649252777817964]}]: dispatch 2024-09-16T05:19:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.11774996087304687]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.088933934500018]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08819765651116344]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.089025887559987]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08261769361479553]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.06944803497608189]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0885148599341356]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07717643447199501]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07540378511795306]}]: dispatch 2024-09-16T05:19:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:18 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08847400917290459]}]: dispatch 2024-09-16T05:19:18.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:18.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:18.980 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:19:18.981 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:19:18.981 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:19:18.981 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:19:18.981 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:19:19.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:19.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: osdmap e608: 12 total, 12 up, 12 in 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10551160220848944]}]: dispatch 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07649252777817964]}]: dispatch 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.11774996087304687]}]: dispatch 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.088933934500018]}]: dispatch 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08819765651116344]}]: dispatch 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.089025887559987]}]: dispatch 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08261769361479553]}]: dispatch 2024-09-16T05:19:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.06944803497608189]}]: dispatch 2024-09-16T05:19:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0885148599341356]}]: dispatch 2024-09-16T05:19:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07717643447199501]}]: dispatch 2024-09-16T05:19:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07540378511795306]}]: dispatch 2024-09-16T05:19:19.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:18 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08847400917290459]}]: dispatch 2024-09-16T05:19:19.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:19.181 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:19.181 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:19.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:19.204 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:19.204 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:19.226 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:19.227 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:19.227 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:19:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: osdmap e608: 12 total, 12 up, 12 in 2024-09-16T05:19:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:19:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:19:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10551160220848944]}]: dispatch 2024-09-16T05:19:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07649252777817964]}]: dispatch 2024-09-16T05:19:19.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.11774996087304687]}]: dispatch 2024-09-16T05:19:19.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.088933934500018]}]: dispatch 2024-09-16T05:19:19.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08819765651116344]}]: dispatch 2024-09-16T05:19:19.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.089025887559987]}]: dispatch 2024-09-16T05:19:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08261769361479553]}]: dispatch 2024-09-16T05:19:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.06944803497608189]}]: dispatch 2024-09-16T05:19:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0885148599341356]}]: dispatch 2024-09-16T05:19:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07717643447199501]}]: dispatch 2024-09-16T05:19:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07540378511795306]}]: dispatch 2024-09-16T05:19:19.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:18 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08847400917290459]}]: dispatch 2024-09-16T05:19:19.248 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:19.270 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:19.271 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:19.292 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:19.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:19.313 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:19.321 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:19.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:19.334 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:19.334 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:19.355 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:19.355 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:19.376 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:19.377 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:19.398 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:19.398 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:19.419 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:19.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:19.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:19.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:19.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:19.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:19.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:19.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:19.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:19.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:19.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:19:19.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:19.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:19.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:19.896 INFO:tasks.rados.rados.2.smithi204.stdout:2331: done (3 left) 2024-09-16T05:19:19.896 INFO:tasks.rados.rados.2.smithi204.stdout:2332: copy_from oid 36 from oid 42 current snap is 243 2024-09-16T05:19:19.896 INFO:tasks.rados.rados.2.smithi204.stdout:2333: snap_remove snap 242 2024-09-16T05:19:19.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:19.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:19:19.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:19.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:20.008 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:20.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:20.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:20.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:20.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: pgmap v1074: 107 pgs: 107 active+clean; 367 MiB data, 959 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2024-09-16T05:19:20.128 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10551160220848944]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07649252777817964]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.11774996087304687]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.088933934500018]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08819765651116344]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.089025887559987]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08261769361479553]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.06944803497608189]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0885148599341356]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07717643447199501]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07540378511795306]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08847400917290459]}]': finished 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: osdmap e609: 12 total, 12 up, 12 in 2024-09-16T05:19:20.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2625077544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:20.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:19:20.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:19 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:19:20.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: pgmap v1074: 107 pgs: 107 active+clean; 367 MiB data, 959 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2024-09-16T05:19:20.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10551160220848944]}]': finished 2024-09-16T05:19:20.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07649252777817964]}]': finished 2024-09-16T05:19:20.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.11774996087304687]}]': finished 2024-09-16T05:19:20.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.088933934500018]}]': finished 2024-09-16T05:19:20.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08819765651116344]}]': finished 2024-09-16T05:19:20.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.089025887559987]}]': finished 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08261769361479553]}]': finished 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.06944803497608189]}]': finished 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0885148599341356]}]': finished 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07717643447199501]}]': finished 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07540378511795306]}]': finished 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08847400917290459]}]': finished 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: osdmap e609: 12 total, 12 up, 12 in 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2625077544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:19:20.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:19 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:19:20.165 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:20.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:20.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:20.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: pgmap v1074: 107 pgs: 107 active+clean; 367 MiB data, 959 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2024-09-16T05:19:20.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10551160220848944]}]': finished 2024-09-16T05:19:20.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07649252777817964]}]': finished 2024-09-16T05:19:20.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.11774996087304687]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.088933934500018]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08819765651116344]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.089025887559987]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08261769361479553]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.06944803497608189]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0885148599341356]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07717643447199501]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07540378511795306]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08847400917290459]}]': finished 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: osdmap e609: 12 total, 12 up, 12 in 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2625077544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:19:20.192 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:19 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:19:20.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:20.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:20.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:20.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:20.405 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:20.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:20.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:20.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:20.457 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:20.457 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:20.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:20.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:20.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:20.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:20.683 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:20.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:20.777 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T05:19:20.781 INFO:tasks.rados.rados.2.smithi204.stdout:2333: done (4 left) 2024-09-16T05:19:20.781 INFO:tasks.rados.rados.2.smithi204.stdout:2334: delete oid 38 current snap is 243 2024-09-16T05:19:20.781 INFO:tasks.rados.rados.2.smithi204.stdout:2326: expect (ObjNum 657 snap 191 seq_num 657) 2024-09-16T05:19:20.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:20.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:20.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:20.886 INFO:tasks.rados.rados.2.smithi204.stdout:2329: finishing write tid 1 to smithi20457400-50 2024-09-16T05:19:20.886 INFO:tasks.rados.rados.2.smithi204.stdout:2329: finishing write tid 2 to smithi20457400-50 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2329: finishing write tid 3 to smithi20457400-50 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2329: finishing write tid 4 to smithi20457400-50 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 775 (ObjNum 838 snap 243 seq_num 838) dirty exists 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2329: left oid 50 (ObjNum 838 snap 243 seq_num 838) 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2330: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2330: finishing copy_from to smithi20457400-20 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 541 (ObjNum 829 snap 242 seq_num 829) dirty exists 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2332: finishing copy_from racing read to smithi20457400-36 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2332: finishing copy_from to smithi20457400-36 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 758 (ObjNum 834 snap 242 seq_num 834) dirty exists 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2326: done (4 left) 2024-09-16T05:19:20.887 INFO:tasks.rados.rados.2.smithi204.stdout:2329: done (3 left) 2024-09-16T05:19:20.888 INFO:tasks.rados.rados.2.smithi204.stdout:2330: done (2 left) 2024-09-16T05:19:20.888 INFO:tasks.rados.rados.2.smithi204.stdout:2332: done (1 left) 2024-09-16T05:19:20.888 INFO:tasks.rados.rados.2.smithi204.stdout:2334: done (0 left) 2024-09-16T05:19:20.888 INFO:tasks.rados.rados.2.smithi204.stdout:2335: delete oid 48 current snap is 243 2024-09-16T05:19:20.888 INFO:tasks.rados.rados.2.smithi204.stdout:2335: done (0 left) 2024-09-16T05:19:20.888 INFO:tasks.rados.rados.2.smithi204.stdout:2336: write (excl) oid 5 current snap is 243 2024-09-16T05:19:20.889 INFO:tasks.rados.rados.2.smithi204.stdout:2336: seq_num 841 ranges {534687=483239,1613474=412283} 2024-09-16T05:19:20.893 INFO:tasks.rados.rados.2.smithi204.stdout:2336: writing smithi20457400-5 from 534687 to 1017926 tid 1 2024-09-16T05:19:20.899 INFO:tasks.rados.rados.2.smithi204.stdout:2336: writing smithi20457400-5 from 1613474 to 2025757 tid 2 2024-09-16T05:19:20.900 INFO:tasks.rados.rados.2.smithi204.stdout:2337: rollback oid 12 current snap is 243 2024-09-16T05:19:20.900 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 193 2024-09-16T05:19:20.900 INFO:tasks.rados.rados.2.smithi204.stdout:2336: finishing write tid 1 to smithi20457400-5 2024-09-16T05:19:20.900 INFO:tasks.rados.rados.2.smithi204.stdout:2338: rollback oid 26 current snap is 243 2024-09-16T05:19:20.900 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 193 2024-09-16T05:19:20.900 INFO:tasks.rados.rados.2.smithi204.stdout:2339: write (excl) oid 1 current snap is 243 2024-09-16T05:19:20.900 INFO:tasks.rados.rados.2.smithi204.stdout:2339: seq_num 842 ranges {551494=558488,1671956=479538} 2024-09-16T05:19:20.905 INFO:tasks.rados.rados.2.smithi204.stdout:2339: writing smithi20457400-1 from 551494 to 1109982 tid 1 2024-09-16T05:19:20.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:20.912 INFO:tasks.rados.rados.2.smithi204.stdout:2339: writing smithi20457400-1 from 1671956 to 2151494 tid 2 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2340: rollback oid 10 current snap is 243 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 229 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2338: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1041 (ObjNum 481 snap 0 seq_num 2944376496) dirty dne 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2337: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2337: finishing rollback tid 2 to smithi20457400-12 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 850 (ObjNum 631 snap 182 seq_num 631) dirty exists 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2336: finishing write tid 2 to smithi20457400-5 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2336: finishing write tid 3 to smithi20457400-5 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2336: finishing write tid 4 to smithi20457400-5 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 605 (ObjNum 841 snap 243 seq_num 841) dirty exists 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2336: left oid 5 (ObjNum 841 snap 243 seq_num 841) 2024-09-16T05:19:20.913 INFO:tasks.rados.rados.2.smithi204.stdout:2339: finishing write tid 1 to smithi20457400-1 2024-09-16T05:19:20.914 INFO:tasks.rados.rados.2.smithi204.stdout:2336: done (4 left) 2024-09-16T05:19:20.914 INFO:tasks.rados.rados.2.smithi204.stdout:2337: done (3 left) 2024-09-16T05:19:20.914 INFO:tasks.rados.rados.2.smithi204.stdout:2338: done (2 left) 2024-09-16T05:19:20.914 INFO:tasks.rados.rados.2.smithi204.stdout:2341: read oid 42 snap -1 2024-09-16T05:19:20.914 INFO:tasks.rados.rados.2.smithi204.stdout:2341: expect (ObjNum 834 snap 242 seq_num 834) 2024-09-16T05:19:20.914 INFO:tasks.rados.rados.2.smithi204.stdout:2342: rollback oid 14 current snap is 243 2024-09-16T05:19:20.914 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 237 2024-09-16T05:19:20.914 INFO:tasks.rados.rados.2.smithi204.stdout:2343: snap_remove snap 234 2024-09-16T05:19:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:20.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:21.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:21.061 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:21.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:21.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:19:21.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:20 smithi032 ceph-mon[27388]: osdmap e610: 12 total, 12 up, 12 in 2024-09-16T05:19:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3566738989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:19:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3566738989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:19:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:20 smithi032 ceph-mon[27388]: osdmap e611: 12 total, 12 up, 12 in 2024-09-16T05:19:21.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:20 smithi204 ceph-mon[30933]: osdmap e610: 12 total, 12 up, 12 in 2024-09-16T05:19:21.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3566738989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:19:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3566738989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:19:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:20 smithi204 ceph-mon[30933]: osdmap e611: 12 total, 12 up, 12 in 2024-09-16T05:19:21.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:21.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:21.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:20 smithi196 ceph-mon[30802]: osdmap e610: 12 total, 12 up, 12 in 2024-09-16T05:19:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3566738989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:19:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3566738989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:19:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:20 smithi196 ceph-mon[30802]: osdmap e611: 12 total, 12 up, 12 in 2024-09-16T05:19:21.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:21.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:21.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:21.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:21.317 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:21.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:21.431 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:21.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:21.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:21.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:21.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:21.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:21.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-16T05:19:21.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:21.678 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:21.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:21.775 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:21.775 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:21.779 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:19:21.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:21.784 INFO:tasks.rados.rados.2.smithi204.stdout:2343: done (4 left) 2024-09-16T05:19:21.784 INFO:tasks.rados.rados.2.smithi204.stdout:2344: copy_from oid 32 from oid 36 current snap is 243 2024-09-16T05:19:21.784 INFO:tasks.rados.rados.2.smithi204.stdout:2345: write oid 15 current snap is 243 2024-09-16T05:19:21.784 INFO:tasks.rados.rados.2.smithi204.stdout:2345: seq_num 843 ranges {568301=633737,1614085=415952} 2024-09-16T05:19:21.792 INFO:tasks.rados.rados.2.smithi204.stdout:2345: writing smithi20457400-15 from 568301 to 1202038 tid 1 2024-09-16T05:19:21.799 INFO:tasks.rados.rados.2.smithi204.stdout:2345: writing smithi20457400-15 from 1614085 to 2030037 tid 2 2024-09-16T05:19:21.799 INFO:tasks.rados.rados.2.smithi204.stdout:2346: delete oid 21 current snap is 243 2024-09-16T05:19:21.799 INFO:tasks.rados.rados.2.smithi204.stdout:2339: finishing write tid 2 to smithi20457400-1 2024-09-16T05:19:21.799 INFO:tasks.rados.rados.2.smithi204.stdout:2339: finishing write tid 3 to smithi20457400-1 2024-09-16T05:19:21.799 INFO:tasks.rados.rados.2.smithi204.stdout:2339: finishing write tid 4 to smithi20457400-1 2024-09-16T05:19:21.799 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 542 (ObjNum 842 snap 243 seq_num 842) dirty exists 2024-09-16T05:19:21.799 INFO:tasks.rados.rados.2.smithi204.stdout:2339: left oid 1 (ObjNum 842 snap 243 seq_num 842) 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:2340: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:2340: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:2340: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 814 (ObjNum 753 snap 219 seq_num 753) dirty exists 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:2342: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:2342: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:2342: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1073 (ObjNum 774 snap 225 seq_num 774) dirty exists 2024-09-16T05:19:21.800 INFO:tasks.rados.rados.2.smithi204.stdout:2341: expect (ObjNum 834 snap 242 seq_num 834) 2024-09-16T05:19:21.810 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:21.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:21.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:21.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:21.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:21.964 INFO:tasks.rados.rados.2.smithi204.stdout:2344: finishing copy_from racing read to smithi20457400-32 2024-09-16T05:19:21.964 INFO:tasks.rados.rados.2.smithi204.stdout:2345: finishing write tid 1 to smithi20457400-15 2024-09-16T05:19:21.964 INFO:tasks.rados.rados.2.smithi204.stdout:2344: finishing copy_from to smithi20457400-32 2024-09-16T05:19:21.964 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 758 (ObjNum 834 snap 242 seq_num 834) dirty exists 2024-09-16T05:19:21.965 INFO:tasks.rados.rados.2.smithi204.stdout:2345: finishing write tid 2 to smithi20457400-15 2024-09-16T05:19:21.965 INFO:tasks.rados.rados.2.smithi204.stdout:2345: finishing write tid 3 to smithi20457400-15 2024-09-16T05:19:21.965 INFO:tasks.rados.rados.2.smithi204.stdout:2339: done (6 left) 2024-09-16T05:19:21.965 INFO:tasks.rados.rados.2.smithi204.stdout:2340: done (5 left) 2024-09-16T05:19:21.965 INFO:tasks.rados.rados.2.smithi204.stdout:2341: done (4 left) 2024-09-16T05:19:21.965 INFO:tasks.rados.rados.2.smithi204.stdout:2342: done (3 left) 2024-09-16T05:19:21.966 INFO:tasks.rados.rados.2.smithi204.stdout:2344: done (2 left) 2024-09-16T05:19:21.966 INFO:tasks.rados.rados.2.smithi204.stdout:2346: done (1 left) 2024-09-16T05:19:21.966 INFO:tasks.rados.rados.2.smithi204.stdout:2347: snap_create 2024-09-16T05:19:21.966 INFO:tasks.rados.rados.2.smithi204.stdout:2345: finishing write tid 4 to smithi20457400-15 2024-09-16T05:19:21.966 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 704 (ObjNum 843 snap 243 seq_num 843) dirty exists 2024-09-16T05:19:21.967 INFO:tasks.rados.rados.2.smithi204.stdout:2345: left oid 15 (ObjNum 843 snap 243 seq_num 843) 2024-09-16T05:19:22.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:22.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:22.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:22.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:19:22.167 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:21 smithi032 ceph-mon[27388]: pgmap v1077: 107 pgs: 107 active+clean; 375 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-16T05:19:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1654137559' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1654137559' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:21 smithi032 ceph-mon[27388]: osdmap e612: 12 total, 12 up, 12 in 2024-09-16T05:19:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:21 smithi204 ceph-mon[30933]: pgmap v1077: 107 pgs: 107 active+clean; 375 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-16T05:19:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1654137559' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1654137559' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:21 smithi204 ceph-mon[30933]: osdmap e612: 12 total, 12 up, 12 in 2024-09-16T05:19:22.192 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:22.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:21 smithi196 ceph-mon[30802]: pgmap v1077: 107 pgs: 107 active+clean; 375 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2024-09-16T05:19:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1654137559' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1654137559' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:21 smithi196 ceph-mon[30802]: osdmap e612: 12 total, 12 up, 12 in 2024-09-16T05:19:22.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:22.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:22.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:22.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:22.444 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:22.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:22.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:19:22.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:22.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:22.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:22.691 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:22.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:22.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:22.753 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:22.785 INFO:tasks.rados.rados.2.smithi204.stdout:2345: done (1 left) 2024-09-16T05:19:22.785 INFO:tasks.rados.rados.2.smithi204.stdout:2347: done (0 left) 2024-09-16T05:19:22.785 INFO:tasks.rados.rados.2.smithi204.stdout:2348: delete oid 33 current snap is 244 2024-09-16T05:19:22.788 INFO:tasks.rados.rados.2.smithi204.stdout:2348: done (0 left) 2024-09-16T05:19:22.788 INFO:tasks.rados.rados.2.smithi204.stdout:2349: snap_create 2024-09-16T05:19:22.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:22.820 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:22.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:22.921 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:22.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:22.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:23.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:23.066 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:23.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:23.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:23.185 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:23.189 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:23.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:23.258 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:23.258 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:23.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:23.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:23.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:23.419 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:23.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:23.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:23.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:23.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:23.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:23.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:23.695 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:23.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:23.734 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:23.734 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:23.755 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:23.755 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:23.775 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:23.776 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:23.782 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T05:19:23.785 INFO:tasks.rados.rados.2.smithi204.stdout:2349: done (0 left) 2024-09-16T05:19:23.785 INFO:tasks.rados.rados.2.smithi204.stdout:2350: rollback oid 42 current snap is 245 2024-09-16T05:19:23.785 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 222 2024-09-16T05:19:23.785 INFO:tasks.rados.rados.2.smithi204.stdout:2351: read oid 18 snap -1 2024-09-16T05:19:23.785 INFO:tasks.rados.rados.2.smithi204.stdout:2351: expect deleted 2024-09-16T05:19:23.785 INFO:tasks.rados.rados.2.smithi204.stdout:2352: read oid 16 snap 200 2024-09-16T05:19:23.785 INFO:tasks.rados.rados.2.smithi204.stdout:2352: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:19:23.786 INFO:tasks.rados.rados.2.smithi204.stdout:2353: read oid 35 snap -1 2024-09-16T05:19:23.786 INFO:tasks.rados.rados.2.smithi204.stdout:2353: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:19:23.786 INFO:tasks.rados.rados.2.smithi204.stdout:2354: snap_create 2024-09-16T05:19:23.789 INFO:tasks.rados.rados.2.smithi204.stdout:2353: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:19:23.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:23.805 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:23.806 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:23.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:23.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:23.828 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:23.828 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:23.850 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:23.850 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:23.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:23 smithi204 ceph-mon[30933]: pgmap v1080: 107 pgs: 1 active+clean+snaptrim, 1 remapped+peering, 105 active+clean; 360 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.8 MiB/s wr, 23 op/s 2024-09-16T05:19:23.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:23 smithi204 ceph-mon[30933]: osdmap e613: 12 total, 12 up, 12 in 2024-09-16T05:19:23.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:23 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:19:23.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2920058861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:19:23.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1428704522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:23.875 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:23.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:19:23.901 INFO:tasks.rados.rados.2.smithi204.stdout:2350: finishing rollback tid 0 to smithi20457400-42 2024-09-16T05:19:23.901 INFO:tasks.rados.rados.2.smithi204.stdout:2350: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:19:23.902 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 760 (ObjNum 97 snap 0 seq_num 4282187760) dirty dne 2024-09-16T05:19:23.902 INFO:tasks.rados.rados.2.smithi204.stdout:2352: expect (ObjNum 596 snap 180 seq_num 596) 2024-09-16T05:19:23.911 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:23.911 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:23.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:23.940 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:23.941 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:23.941 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:23.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:23.963 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:23.963 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:23.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:23.984 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:23.984 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:23 smithi032 ceph-mon[27388]: pgmap v1080: 107 pgs: 1 active+clean+snaptrim, 1 remapped+peering, 105 active+clean; 360 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.8 MiB/s wr, 23 op/s 2024-09-16T05:19:23.985 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:23 smithi032 ceph-mon[27388]: osdmap e613: 12 total, 12 up, 12 in 2024-09-16T05:19:23.985 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:23 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:19:23.985 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2920058861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:19:23.985 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1428704522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:24.008 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:24.008 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:24.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:24.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:24.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:24.146 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:24.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:24.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:24.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:24.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:23 smithi196 ceph-mon[30802]: pgmap v1080: 107 pgs: 1 active+clean+snaptrim, 1 remapped+peering, 105 active+clean; 360 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.8 MiB/s wr, 23 op/s 2024-09-16T05:19:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:23 smithi196 ceph-mon[30802]: osdmap e613: 12 total, 12 up, 12 in 2024-09-16T05:19:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:23 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:19:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2920058861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:19:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1428704522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:24.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:24.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:24.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:24.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:24.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:24.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:24.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:24.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:24.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:24.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:24.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1329 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 422.82278989600002, 2024-09-16T05:19:24.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80822626200000003, 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1329 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:19:24.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783039+0000", 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.674e-06 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783047+0000", 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6619999999999994e-06 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783063+0000", 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6058000000000001e-05 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783071+0000", 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8399999999999995e-06 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783134+0000", 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2496999999999995e-05 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783142+0000", 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1219999999999995e-06 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591169+0000", 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80802694100000005 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591263+0000", 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4468000000000004e-05 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1330 2.1d 2:b98a79db:::smithi20457400-31:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:19:24.630 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 422.82273893399997, 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80866329400000003, 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1330 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:19:24.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783092+0000", 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5149999999999998e-06 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783104+0000", 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2412000000000001e-05 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783117+0000", 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2379e-05 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783170+0000", 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3727999999999998e-05 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783178+0000", 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3479999999999998e-06 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591633+0000", 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.808455339 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591751+0000", 2024-09-16T05:19:24.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118087 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1331 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 422.82269816799999, 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80878027500000005, 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.636 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.637 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.637 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1331 2024-09-16T05:19:24.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.637 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5400000000000001e-07 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783132+0000", 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689999999999999e-06 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783139+0000", 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7920000000000004e-06 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783144+0000", 2024-09-16T05:19:24.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783192+0000", 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8304999999999999e-05 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783196+0000", 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8480000000000003e-06 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591842+0000", 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80864587700000001 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591909+0000", 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7195999999999998e-05 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1575 2.5 2:a5bc1608:::smithi20457400-11:head [setxattr _header (57) in=7b,truncate 3966452] snapc 43=[43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 405.91465214700003, 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64249167500000004, 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1575 2024-09-16T05:19:24.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5200000000000002e-07 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691178+0000", 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.734e-06 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691184+0000", 2024-09-16T05:19:24.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2940000000000002e-06 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691188+0000", 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.642e-06 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692349+0000", 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001161444 2024-09-16T05:19:24.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692452+0000", 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102916 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692596+0000", 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000143552 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:19:24.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692970+0000", 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037462699999999999 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333552+0000", 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64058177999999999 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333609+0000", 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6728999999999998e-05 2024-09-16T05:19:24.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333666+0000", 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7504999999999999e-05 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1576 2.5 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 405.914633412, 2024-09-16T05:19:24.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64272760100000004, 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1576 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:19:24.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691194+0000", 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.65e-07 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691195+0000", 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-06 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691199+0000", 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0060000000000003e-06 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691202+0000", 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9239999999999999e-06 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692872+0000", 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001669467 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692954+0000", 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2361000000000006e-05 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333715+0000", 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64076073300000003 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333817+0000", 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010184200000000001 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333921+0000", 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104249 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:3036 2.1b e379/291 2:da4f6049:::smithi20457400-49:head v 379'328, mlcod=379'325)", 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 235.83764840800001, 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.436541917, 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768179+0000", 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2900000000000003e-07 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768185+0000", 2024-09-16T05:19:24.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8309999999999997e-06 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203766+0000", 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43558113999999998 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203780+0000", 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3825999999999999e-05 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203879+0000", 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9419000000000005e-05 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203926+0000", 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6248999999999997e-05 2024-09-16T05:19:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204668+0000", 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00074248900000000004 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204720+0000", 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2234000000000002e-05 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.658 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.9.0:755 2.2f e422/275 ondisk, result = 0)", 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 192.88230594800001, 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.59448197899999999, 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7700000000000004e-07 2024-09-16T05:19:24.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723525+0000", 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5209999999999998e-06 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317758+0000", 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.59423284399999998 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317765+0000", 2024-09-16T05:19:24.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.827e-06 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317821+0000", 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6573000000000002e-05 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317832+0000", 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0718e-05 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.318003+0000", 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017091900000000001 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3531 2.15 2:a94680a3:::smithi20457400-13:head [write 652237~793851] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.31537138300001, 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44774744599999999, 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.663 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3531 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290457+0000", 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.823e-06 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291024+0000", 2024-09-16T05:19:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00056645899999999995 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291045+0000", 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0837999999999999e-05 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291055+0000", 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0066e-05 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291140+0000", 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5686000000000006e-05 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291149+0000", 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7579999999999998e-06 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.733424+0000", 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.442274801 2024-09-16T05:19:24.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.733559+0000", 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013477500000000001 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.734084+0000", 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00052511500000000004 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:19:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738013+0000", 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039291179999999997 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738083+0000", 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0208000000000003e-05 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738157+0000", 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4071000000000003e-05 2024-09-16T05:19:24.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738203+0000", 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5728e-05 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3532 2.15 2:a94680a3:::smithi20457400-13:head [write 2198894~53343] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.31104037599999, 2024-09-16T05:19:24.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44479106600000001, 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3532 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:19:24.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294787+0000", 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6700000000000003e-07 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294826+0000", 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8801999999999999e-05 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294837+0000", 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1079e-05 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294842+0000", 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.152e-06 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8766e-05 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294898+0000", 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1470000000000004e-06 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737061+0000", 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44216253500000002 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737164+0000", 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103288 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737374+0000", 2024-09-16T05:19:24.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020994000000000001 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739175+0000", 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018013759999999999 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739420+0000", 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024481500000000001 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739486+0000", 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6308000000000003e-05 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739577+0000", 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1090999999999999e-05 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.677 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3533 2.15 2:a94680a3:::smithi20457400-13:head [setxattr _header (58) in=7b,truncate 2252237] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.31097274800001, 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44535093999999997, 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3533 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294855+0000", 2024-09-16T05:19:24.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0699999999999997e-07 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294858+0000", 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2200000000000001e-06 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294865+0000", 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9290000000000001e-06 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294868+0000", 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4750000000000002e-06 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294924+0000", 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6005000000000003e-05 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294934+0000", 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0234999999999999e-05 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738506+0000", 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44357146200000003 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738606+0000", 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100338 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738777+0000", 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017038200000000001 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739632+0000", 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00085582599999999996 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740092+0000", 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046005699999999999 2024-09-16T05:19:24.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740144+0000", 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.113e-05 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740205+0000", 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1373999999999999e-05 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3534 2.15 2:a94680a3:::smithi20457400-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.310935462, 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44553962000000003, 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3534 2024-09-16T05:19:24.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:19:24.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:19:24.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:19:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294892+0000", 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0299999999999999e-07 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294894+0000", 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.198e-06 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294901+0000", 2024-09-16T05:19:24.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.917e-06 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294912+0000", 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1144999999999999e-05 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294950+0000", 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7889999999999998e-05 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294955+0000", 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8949999999999997e-06 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739213+0000", 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44425779500000001 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739349+0000", 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013630599999999999 2024-09-16T05:19:24.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740238+0000", 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088882100000000003 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740329+0000", 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1366999999999994e-05 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740431+0000", 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101783 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3539 2.54 2:2ace617f:::smithi20457400-21:head [write 685851~544349] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.29263353100001, 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42847687499999998, 2024-09-16T05:19:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3539 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:19:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313194+0000", 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313430+0000", 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023602700000000001 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313441+0000", 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1158000000000001e-05 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313449+0000", 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3180000000000001e-06 2024-09-16T05:19:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313509+0000", 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0010999999999999e-05 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313516+0000", 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.131e-06 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737710+0000", 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42419478900000002 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738322+0000", 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061162399999999996 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738531+0000", 2024-09-16T05:19:24.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020935999999999999 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741253+0000", 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027214650000000002 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741592+0000", 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033932199999999997 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741636+0000", 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3649000000000002e-05 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741670+0000", 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4264999999999998e-05 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.700 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3540 2.54 2:2ace617f:::smithi20457400-21:head [write 1799505~486346] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.28714085300001, 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42371521699999998, 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3540 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318687+0000", 2024-09-16T05:19:24.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9400000000000004e-07 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318917+0000", 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023015399999999999 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318932+0000", 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.485e-05 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318939+0000", 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5059999999999998e-06 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7806999999999997e-05 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319006+0000", 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1649999999999995e-06 2024-09-16T05:19:24.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739899+0000", 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42089313 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739946+0000", 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6783000000000003e-05 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740114+0000", 2024-09-16T05:19:24.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016756599999999999 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741289+0000", 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011754549999999999 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742318+0000", 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010284420000000001 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.706 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742363+0000", 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5760999999999998e-05 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742401+0000", 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7803999999999999e-05 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.707 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3541 2.54 2:2ace617f:::smithi20457400-21:head [setxattr _header (58) in=7b,truncate 2285851] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.28687095000001, 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42367250699999998, 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3541 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.708 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.709 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5600000000000001e-07 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318959+0000", 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7020000000000001e-06 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.710 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318970+0000", 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0991e-05 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318974+0000", 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2849999999999996e-06 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319029+0000", 2024-09-16T05:19:24.711 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5130999999999998e-05 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319036+0000", 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9129999999999997e-06 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740691+0000", 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42165504500000001 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.712 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740769+0000", 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7179000000000001e-05 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740875+0000", 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106637 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741296+0000", 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042051399999999998 2024-09-16T05:19:24.713 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742521+0000", 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001225636 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742569+0000", 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7982000000000002e-05 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742628+0000", 2024-09-16T05:19:24.714 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9036000000000002e-05 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3542 2.54 2:2ace617f:::smithi20457400-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 179.286830143, 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42382194699999998, 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:19:24.715 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3542 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.716 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7599999999999998e-07 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318999+0000", 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7770000000000001e-06 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.717 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319007+0000", 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4039999999999995e-06 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319011+0000", 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8260000000000003e-06 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319089+0000", 2024-09-16T05:19:24.718 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8046000000000005e-05 2024-09-16T05:19:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319099+0000", 2024-09-16T05:19:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6490000000000003e-06 2024-09-16T05:19:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.719 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741192+0000", 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42209296299999999 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741242+0000", 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0589e-05 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.720 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742668+0000", 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001425429 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742746+0000", 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7990000000000004e-05 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742819+0000", 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2997999999999995e-05 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.721 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3951 2.2f 2:f4b938a4:::smithi20457400-47:head [setxattr _header (58) in=7b,truncate 3395113] snapc a8=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 147.74763685799999, 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39145527099999999, 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.722 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3951 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7099999999999997e-07 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858193+0000", 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3439999999999999e-06 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.724 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858197+0000", 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229999999999999e-06 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858200+0000", 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.937e-06 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859064+0000", 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00086360799999999998 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.725 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859153+0000", 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9592999999999994e-05 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859294+0000", 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014096099999999999 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:19:24.726 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859660+0000", 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036632599999999999 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249556+0000", 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38989596300000001 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249600+0000", 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3757000000000001e-05 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249645+0000", 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5087999999999999e-05 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3952 2.2f 2:f4b938a4:::smithi20457400-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e467)", 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 147.74760504599999, 2024-09-16T05:19:24.728 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39157208999999998, 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3952 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.729 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.77e-07 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.730 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858226+0000", 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7249999999999999e-06 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858231+0000", 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.797e-06 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858234+0000", 2024-09-16T05:19:24.731 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2320000000000001e-06 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859544+0000", 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001309892 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859613+0000", 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9261e-05 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.732 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249668+0000", 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39005463499999998 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249718+0000", 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0416e-05 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249794+0000", 2024-09-16T05:19:24.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5754999999999996e-05 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5475 2.5 2:a1d8c6d9:::smithi20457400-3:head [read 0~2890969,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e575)", 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:44.784309+0000", 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 39.821518060000002, 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.072687437999999993, 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:19:24.734 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5475 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784309+0000", 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784309+0000", 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784309+0000", 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7999999999999995e-07 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784322+0000", 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2979000000000001e-05 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784329+0000", 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.165e-06 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784336+0000", 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3899999999999998e-06 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.787038+0000", 2024-09-16T05:19:24.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027023009999999998 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.787050+0000", 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2284e-05 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.855863+0000", 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.068812938000000004 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.855976+0000", 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113115 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.856996+0000", 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010196859999999999 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.739 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5476 2.5 2:a1d8c6d9:::smithi20457400-3:head [read 0~2890969,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e575)", 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:44.784344+0000", 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 39.821483092000001, 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073875179999999999, 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5476 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.740 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784344+0000", 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784344+0000", 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784344+0000", 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4299999999999998e-07 2024-09-16T05:19:24.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784347+0000", 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5220000000000002e-06 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784353+0000", 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2960000000000004e-06 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784358+0000", 2024-09-16T05:19:24.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0390000000000004e-06 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.787090+0000", 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027322980000000002 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.787098+0000", 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9670000000000001e-06 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.857056+0000", 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.069958168000000001 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.857132+0000", 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5093999999999996e-05 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.858219+0000", 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010873530000000001 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:1100 2.f e588/435 2:f08ec412:::smithi20457400-44:e0 v 588'362, mlcod=585'359)", 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.832358772999999, 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.771307407, 2024-09-16T05:19:24.745 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:19:24.746 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773469+0000", 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0100000000000004e-07 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773478+0000", 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1139999999999995e-06 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544068+0000", 2024-09-16T05:19:24.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.770589827 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544083+0000", 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5213e-05 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544231+0000", 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014854000000000001 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544252+0000", 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0568e-05 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544741+0000", 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048860799999999997 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544775+0000", 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4836000000000003e-05 2024-09-16T05:19:24.749 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:19:24.750 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:19:24.750 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:19:24.772 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:24.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:24.796 INFO:tasks.rados.rados.2.smithi204.stdout:2350: done (4 left) 2024-09-16T05:19:24.796 INFO:tasks.rados.rados.2.smithi204.stdout:2351: done (3 left) 2024-09-16T05:19:24.797 INFO:tasks.rados.rados.2.smithi204.stdout:2352: done (2 left) 2024-09-16T05:19:24.797 INFO:tasks.rados.rados.2.smithi204.stdout:2353: done (1 left) 2024-09-16T05:19:24.797 INFO:tasks.rados.rados.2.smithi204.stdout:2354: done (0 left) 2024-09-16T05:19:24.797 INFO:tasks.rados.rados.2.smithi204.stdout:2355: write (excl) oid 14 current snap is 246 2024-09-16T05:19:24.797 INFO:tasks.rados.rados.2.smithi204.stdout:2355: seq_num 846 ranges {618722=575837,1905884=312838} 2024-09-16T05:19:24.803 INFO:tasks.rados.rados.2.smithi204.stdout:2355: writing smithi20457400-14 from 618722 to 1194559 tid 1 2024-09-16T05:19:24.809 INFO:tasks.rados.rados.2.smithi204.stdout:2355: writing smithi20457400-14 from 1905884 to 2218722 tid 2 2024-09-16T05:19:24.809 INFO:tasks.rados.rados.2.smithi204.stdout:2356: read oid 42 snap -1 2024-09-16T05:19:24.809 INFO:tasks.rados.rados.2.smithi204.stdout:2356: expect deleted 2024-09-16T05:19:24.809 INFO:tasks.rados.rados.2.smithi204.stdout:2357: snap_remove snap 241 2024-09-16T05:19:24.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:24.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2920058861' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:19:24.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:24 smithi032 ceph-mon[27388]: osdmap e614: 12 total, 12 up, 12 in 2024-09-16T05:19:24.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2920058861' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:19:24.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:24 smithi204 ceph-mon[30933]: osdmap e614: 12 total, 12 up, 12 in 2024-09-16T05:19:24.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:25.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:25.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:25.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:25.149 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:25.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:25.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2920058861' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:19:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:24 smithi196 ceph-mon[30802]: osdmap e614: 12 total, 12 up, 12 in 2024-09-16T05:19:25.251 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:25.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:25.355 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:25.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:19:25.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:25.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:25.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:25.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:25.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:25.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:25.625 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:25.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:25.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:25.777 INFO:tasks.rados.rados.2.smithi204.stdout:2357: done (2 left) 2024-09-16T05:19:25.777 INFO:tasks.rados.rados.2.smithi204.stdout:2358: snap_remove snap 193 2024-09-16T05:19:25.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:25.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:25.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:25.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:25 smithi032 ceph-mon[27388]: pgmap v1083: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 368 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.1 MiB/s wr, 21 op/s 2024-09-16T05:19:25.902 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:25 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:19:25.903 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:25 smithi032 ceph-mon[27388]: osdmap e615: 12 total, 12 up, 12 in 2024-09-16T05:19:25.903 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:25 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:25.903 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:25 smithi032 ceph-mon[27388]: osdmap e616: 12 total, 12 up, 12 in 2024-09-16T05:19:25.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:25.944 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:25.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:25.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:26.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:26.073 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:26.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:26.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:25 smithi204 ceph-mon[30933]: pgmap v1083: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 368 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.1 MiB/s wr, 21 op/s 2024-09-16T05:19:26.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:25 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:19:26.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:25 smithi204 ceph-mon[30933]: osdmap e615: 12 total, 12 up, 12 in 2024-09-16T05:19:26.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:25 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:26.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:25 smithi204 ceph-mon[30933]: osdmap e616: 12 total, 12 up, 12 in 2024-09-16T05:19:26.085 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:19:26.085 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:19:26.085 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:19:26.085 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:19:26.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:26.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:26.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:26.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:25 smithi196 ceph-mon[30802]: pgmap v1083: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 368 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.1 MiB/s wr, 21 op/s 2024-09-16T05:19:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:25 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:19:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:25 smithi196 ceph-mon[30802]: osdmap e615: 12 total, 12 up, 12 in 2024-09-16T05:19:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:25 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:25 smithi196 ceph-mon[30802]: osdmap e616: 12 total, 12 up, 12 in 2024-09-16T05:19:26.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:26.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:26.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:26.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:26.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:26.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:26.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:26.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:26.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:26.573 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:26.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:26.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:26.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:26.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:26.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:19:26.775 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T05:19:26.776 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:19:26.781 INFO:tasks.rados.rados.2.smithi204.stdout:2358: done (2 left) 2024-09-16T05:19:26.781 INFO:tasks.rados.rados.2.smithi204.stdout:2359: write (excl) oid 27 current snap is 246 2024-09-16T05:19:26.781 INFO:tasks.rados.rados.2.smithi204.stdout:2359: seq_num 847 ranges {635529=651086,1848013=387516} 2024-09-16T05:19:26.789 INFO:tasks.rados.rados.2.smithi204.stdout:2359: writing smithi20457400-27 from 635529 to 1286615 tid 1 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2359: writing smithi20457400-27 from 1848013 to 2235529 tid 2 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2360: rollback oid 29 current snap is 246 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 240 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2355: finishing write tid 1 to smithi20457400-14 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2355: finishing write tid 2 to smithi20457400-14 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2355: finishing write tid 3 to smithi20457400-14 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2355: finishing write tid 4 to smithi20457400-14 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1083 (ObjNum 846 snap 246 seq_num 846) dirty exists 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2355: left oid 14 (ObjNum 846 snap 246 seq_num 846) 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2355: done (3 left) 2024-09-16T05:19:26.796 INFO:tasks.rados.rados.2.smithi204.stdout:2356: done (2 left) 2024-09-16T05:19:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:2361: copy_from oid 17 from oid 31 current snap is 246 2024-09-16T05:19:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:2362: read oid 4 snap -1 2024-09-16T05:19:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:2362: expect deleted 2024-09-16T05:19:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:2363: rollback oid 5 current snap is 246 2024-09-16T05:19:26.797 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 5 to 227 2024-09-16T05:19:26.798 INFO:tasks.rados.rados.2.smithi204.stdout:2364: read oid 21 snap -1 2024-09-16T05:19:26.798 INFO:tasks.rados.rados.2.smithi204.stdout:2364: expect deleted 2024-09-16T05:19:26.798 INFO:tasks.rados.rados.2.smithi204.stdout:2365: rollback oid 30 current snap is 246 2024-09-16T05:19:26.798 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 30 to 200 2024-09-16T05:19:26.798 INFO:tasks.rados.rados.2.smithi204.stdout:2362: done (6 left) 2024-09-16T05:19:26.798 INFO:tasks.rados.rados.2.smithi204.stdout:2366: write (excl) oid 22 current snap is 246 2024-09-16T05:19:26.798 INFO:tasks.rados.rados.2.smithi204.stdout:2366: seq_num 848 ranges {652336=726335,1906495=345841} 2024-09-16T05:19:26.803 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:26.804 INFO:tasks.rados.rados.2.smithi204.stdout:2366: writing smithi20457400-22 from 652336 to 1378671 tid 1 2024-09-16T05:19:26.809 INFO:tasks.rados.rados.2.smithi204.stdout:2366: writing smithi20457400-22 from 1906495 to 2252336 tid 2 2024-09-16T05:19:26.809 INFO:tasks.rados.rados.2.smithi204.stdout:2367: read oid 8 snap -1 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2367: expect (ObjNum 809 snap 237 seq_num 809) 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2368: read oid 15 snap -1 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2368: expect (ObjNum 843 snap 243 seq_num 843) 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2369: read oid 4 snap 245 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2369: expect deleted 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2359: finishing write tid 1 to smithi20457400-27 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2359: finishing write tid 2 to smithi20457400-27 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2360: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2365: finishing rollback tid 1 to smithi20457400-30 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2361: finishing copy_from to smithi20457400-17 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2361: got expected ENOENT (src dne) 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2365: finishing rollback tid 2 to smithi20457400-30 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 1093 (ObjNum 681 snap 198 seq_num 681) dirty exists 2024-09-16T05:19:26.810 INFO:tasks.rados.rados.2.smithi204.stdout:2360: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:2360: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 866 (ObjNum 821 snap 239 seq_num 821) dirty exists 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:2361: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:2363: finishing rollback tid 0 to smithi20457400-5 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:2363: finishing rollback tid 1 to smithi20457400-5 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:2363: finishing rollback tid 2 to smithi20457400-5 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 608 (ObjNum 754 snap 220 seq_num 754) dirty exists 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:2366: finishing write tid 1 to smithi20457400-22 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:2360: done (9 left) 2024-09-16T05:19:26.811 INFO:tasks.rados.rados.2.smithi204.stdout:2361: done (8 left) 2024-09-16T05:19:26.812 INFO:tasks.rados.rados.2.smithi204.stdout:2363: done (7 left) 2024-09-16T05:19:26.812 INFO:tasks.rados.rados.2.smithi204.stdout:2364: done (6 left) 2024-09-16T05:19:26.812 INFO:tasks.rados.rados.2.smithi204.stdout:2365: done (5 left) 2024-09-16T05:19:26.812 INFO:tasks.rados.rados.2.smithi204.stdout:2370: delete oid 3 current snap is 246 2024-09-16T05:19:26.813 INFO:tasks.rados.rados.2.smithi204.stdout:2369: done (5 left) 2024-09-16T05:19:26.813 INFO:tasks.rados.rados.2.smithi204.stdout:2370: done (4 left) 2024-09-16T05:19:26.813 INFO:tasks.rados.rados.2.smithi204.stdout:2371: snap_create 2024-09-16T05:19:26.813 INFO:tasks.rados.rados.2.smithi204.stdout:2366: finishing write tid 2 to smithi20457400-22 2024-09-16T05:19:26.813 INFO:tasks.rados.rados.2.smithi204.stdout:2366: finishing write tid 3 to smithi20457400-22 2024-09-16T05:19:26.813 INFO:tasks.rados.rados.2.smithi204.stdout:2366: finishing write tid 4 to smithi20457400-22 2024-09-16T05:19:26.813 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 630 (ObjNum 848 snap 246 seq_num 848) dirty exists 2024-09-16T05:19:26.813 INFO:tasks.rados.rados.2.smithi204.stdout:2366: left oid 22 (ObjNum 848 snap 246 seq_num 848) 2024-09-16T05:19:26.816 INFO:tasks.rados.rados.2.smithi204.stdout:2368: expect (ObjNum 843 snap 243 seq_num 843) 2024-09-16T05:19:26.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:26.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:26.886 INFO:tasks.rados.rados.2.smithi204.stdout:2367: expect (ObjNum 809 snap 237 seq_num 809) 2024-09-16T05:19:26.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:26.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:27.024 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:27.055 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:27.076 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:27.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:27.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1538038733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:19:27.144 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2465118665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:27.144 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1538038733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:19:27.144 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2465118665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:27.144 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:26 smithi204 ceph-mon[30933]: osdmap e617: 12 total, 12 up, 12 in 2024-09-16T05:19:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1538038733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:19:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2465118665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1538038733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:19:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2465118665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:26 smithi032 ceph-mon[27388]: osdmap e617: 12 total, 12 up, 12 in 2024-09-16T05:19:27.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:27.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:27.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1538038733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:19:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2465118665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1538038733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:19:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2465118665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:26 smithi196 ceph-mon[30802]: osdmap e617: 12 total, 12 up, 12 in 2024-09-16T05:19:27.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:27.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:27.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:27.345 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:19:27.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:27.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:27.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:27.518 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:19:27.518 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:19:27.518 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:19:27.519 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:19:27.519 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:19:27.553 INFO:tasks.rados.rados.2.smithi204.stdout:2359: finishing write tid 3 to smithi20457400-27 2024-09-16T05:19:27.553 INFO:tasks.rados.rados.2.smithi204.stdout:2359: finishing write tid 4 to smithi20457400-27 2024-09-16T05:19:27.553 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 326 (ObjNum 847 snap 246 seq_num 847) dirty exists 2024-09-16T05:19:27.553 INFO:tasks.rados.rados.2.smithi204.stdout:2359: left oid 27 (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:19:27.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:27.564 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T05:19:27.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:19:27.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:27.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:27.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:27.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:27.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:27.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:27.816 INFO:tasks.rados.rados.2.smithi204.stdout:2359: done (4 left) 2024-09-16T05:19:27.817 INFO:tasks.rados.rados.2.smithi204.stdout:2366: done (3 left) 2024-09-16T05:19:27.817 INFO:tasks.rados.rados.2.smithi204.stdout:2367: done (2 left) 2024-09-16T05:19:27.817 INFO:tasks.rados.rados.2.smithi204.stdout:2368: done (1 left) 2024-09-16T05:19:27.817 INFO:tasks.rados.rados.2.smithi204.stdout:2371: done (0 left) 2024-09-16T05:19:27.817 INFO:tasks.rados.rados.2.smithi204.stdout:2372: read oid 21 snap -1 2024-09-16T05:19:27.817 INFO:tasks.rados.rados.2.smithi204.stdout:2372: expect deleted 2024-09-16T05:19:27.817 INFO:tasks.rados.rados.2.smithi204.stdout:2373: snap_remove snap 222 2024-09-16T05:19:27.828 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:27.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:27.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:27.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:27.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:28.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:28.076 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:28.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:28.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:27 smithi204 ceph-mon[30933]: pgmap v1086: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 368 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2024-09-16T05:19:28.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:27 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:19:28.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:27 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:19:28.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:27 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:19:28.123 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:27 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:19:28.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:19:28.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:27 smithi032 ceph-mon[27388]: pgmap v1086: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 368 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2024-09-16T05:19:28.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:27 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:19:28.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:27 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:19:28.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:27 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:19:28.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:27 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:19:28.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:28.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:28.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:27 smithi196 ceph-mon[30802]: pgmap v1086: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 368 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2024-09-16T05:19:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:27 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:19:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:27 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:19:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:27 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:19:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:27 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:19:28.302 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:28.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:28.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:28.391 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:28.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:28.452 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:28.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:28.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:28.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:28.573 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:28.573 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:28.577 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:28.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:28.595 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:28.595 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:28.600 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:28.616 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:28.617 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:28.649 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:28.670 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:28.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:28.693 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:28.700 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:28.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:28.714 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:28.714 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:28.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:28.736 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:28.757 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:28.757 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:28.779 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:28.779 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:28.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:28.802 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:28.803 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:28.824 INFO:tasks.rados.rados.2.smithi204.stdout:2373: done (1 left) 2024-09-16T05:19:28.824 INFO:tasks.rados.rados.2.smithi204.stdout:2374: write (excl) oid 12 current snap is 247 2024-09-16T05:19:28.824 INFO:tasks.rados.rados.2.smithi204.stdout:2374: seq_num 850 ranges {685950=476833,1623459=570617} 2024-09-16T05:19:28.830 INFO:tasks.rados.rados.2.smithi204.stdout:2374: writing smithi20457400-12 from 685950 to 1162783 tid 1 2024-09-16T05:19:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:28.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:28.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:28.836 INFO:tasks.rados.rados.2.smithi204.stdout:2374: writing smithi20457400-12 from 1623459 to 2194076 tid 2 2024-09-16T05:19:28.837 INFO:tasks.rados.rados.2.smithi204.stdout:2374: finishing write tid 1 to smithi20457400-12 2024-09-16T05:19:28.837 INFO:tasks.rados.rados.2.smithi204.stdout:2372: done (1 left) 2024-09-16T05:19:28.837 INFO:tasks.rados.rados.2.smithi204.stdout:2375: delete oid 45 current snap is 247 2024-09-16T05:19:28.838 INFO:tasks.rados.rados.2.smithi204.stdout:2375: done (1 left) 2024-09-16T05:19:28.838 INFO:tasks.rados.rados.2.smithi204.stdout:2376: delete oid 24 current snap is 247 2024-09-16T05:19:28.839 INFO:tasks.rados.rados.2.smithi204.stdout:2374: finishing write tid 2 to smithi20457400-12 2024-09-16T05:19:28.840 INFO:tasks.rados.rados.2.smithi204.stdout:2374: finishing write tid 3 to smithi20457400-12 2024-09-16T05:19:28.840 INFO:tasks.rados.rados.2.smithi204.stdout:2374: finishing write tid 4 to smithi20457400-12 2024-09-16T05:19:28.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 874 (ObjNum 850 snap 247 seq_num 850) dirty exists 2024-09-16T05:19:28.840 INFO:tasks.rados.rados.2.smithi204.stdout:2374: left oid 12 (ObjNum 850 snap 247 seq_num 850) 2024-09-16T05:19:28.840 INFO:tasks.rados.rados.2.smithi204.stdout:2374: done (1 left) 2024-09-16T05:19:28.841 INFO:tasks.rados.rados.2.smithi204.stdout:2376: done (0 left) 2024-09-16T05:19:28.841 INFO:tasks.rados.rados.2.smithi204.stdout:2377: read oid 31 snap -1 2024-09-16T05:19:28.841 INFO:tasks.rados.rados.2.smithi204.stdout:2377: expect deleted 2024-09-16T05:19:28.841 INFO:tasks.rados.rados.2.smithi204.stdout:2378: copy_from oid 40 from oid 26 current snap is 247 2024-09-16T05:19:28.841 INFO:tasks.rados.rados.2.smithi204.stdout:2379: snap_remove snap 227 2024-09-16T05:19:28.895 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:28.895 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:28.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:28.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:28 smithi032 ceph-mon[27388]: osdmap e618: 12 total, 12 up, 12 in 2024-09-16T05:19:28.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:28 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:19:28.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:28 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:19:28.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:28 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:19:28.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:28 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:19:28.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3667837312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:28.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:28.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:29.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:29.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:29.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:29.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:28 smithi204 ceph-mon[30933]: osdmap e618: 12 total, 12 up, 12 in 2024-09-16T05:19:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:28 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:19:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:28 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:19:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:28 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:19:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:28 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:19:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3667837312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:29.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:29.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:29.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:28 smithi196 ceph-mon[30802]: osdmap e618: 12 total, 12 up, 12 in 2024-09-16T05:19:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:28 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:19:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:28 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:19:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:28 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:19:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:28 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:19:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3667837312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:29.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:29.335 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:29.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:29.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:29.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:29.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:29.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:29.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:19:29.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:29.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:29.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:29.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:29.807 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:29.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:29.826 INFO:tasks.rados.rados.2.smithi204.stdout:2379: done (2 left) 2024-09-16T05:19:29.826 INFO:tasks.rados.rados.2.smithi204.stdout:2380: rollback oid 39 current snap is 247 2024-09-16T05:19:29.826 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 39 to 230 2024-09-16T05:19:29.826 INFO:tasks.rados.rados.2.smithi204.stdout:2378: finishing copy_from to smithi20457400-40 2024-09-16T05:19:29.826 INFO:tasks.rados.rados.2.smithi204.stdout:2378: got expected ENOENT (src dne) 2024-09-16T05:19:29.826 INFO:tasks.rados.rados.2.smithi204.stdout:2378: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:19:29.827 INFO:tasks.rados.rados.2.smithi204.stdout:2377: done (2 left) 2024-09-16T05:19:29.827 INFO:tasks.rados.rados.2.smithi204.stdout:2378: done (1 left) 2024-09-16T05:19:29.827 INFO:tasks.rados.rados.2.smithi204.stdout:2381: rollback oid 40 current snap is 247 2024-09-16T05:19:29.827 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 40 to 238 2024-09-16T05:19:29.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:29 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:19:29.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:29 smithi204 ceph-mon[30933]: 2.44 scrub ok 2024-09-16T05:19:29.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:29 smithi204 ceph-mon[30933]: pgmap v1089: 107 pgs: 107 active+clean; 361 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 612 KiB/s wr, 13 op/s 2024-09-16T05:19:29.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:29 smithi204 ceph-mon[30933]: 2.2e scrub starts 2024-09-16T05:19:29.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:29 smithi204 ceph-mon[30933]: 2.2e scrub ok 2024-09-16T05:19:29.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:29 smithi204 ceph-mon[30933]: osdmap e619: 12 total, 12 up, 12 in 2024-09-16T05:19:29.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:29 smithi204 ceph-mon[30933]: 2.51 scrub starts 2024-09-16T05:19:29.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:29 smithi204 ceph-mon[30933]: 2.51 scrub ok 2024-09-16T05:19:29.828 INFO:tasks.rados.rados.2.smithi204.stdout:2382: snap_remove snap 200 2024-09-16T05:19:29.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:29.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:29 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:19:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:29 smithi032 ceph-mon[27388]: 2.44 scrub ok 2024-09-16T05:19:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:29 smithi032 ceph-mon[27388]: pgmap v1089: 107 pgs: 107 active+clean; 361 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 612 KiB/s wr, 13 op/s 2024-09-16T05:19:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:29 smithi032 ceph-mon[27388]: 2.2e scrub starts 2024-09-16T05:19:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:29 smithi032 ceph-mon[27388]: 2.2e scrub ok 2024-09-16T05:19:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:29 smithi032 ceph-mon[27388]: osdmap e619: 12 total, 12 up, 12 in 2024-09-16T05:19:29.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:29 smithi032 ceph-mon[27388]: 2.51 scrub starts 2024-09-16T05:19:29.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:29 smithi032 ceph-mon[27388]: 2.51 scrub ok 2024-09-16T05:19:29.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:29.963 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:19:29.965 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:19:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:29.968 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:29.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:30.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:19:30.069 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:30.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:30.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:30.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:30.220 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:30.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:29 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:19:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:29 smithi196 ceph-mon[30802]: 2.44 scrub ok 2024-09-16T05:19:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:29 smithi196 ceph-mon[30802]: pgmap v1089: 107 pgs: 107 active+clean; 361 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 612 KiB/s wr, 13 op/s 2024-09-16T05:19:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:29 smithi196 ceph-mon[30802]: 2.2e scrub starts 2024-09-16T05:19:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:29 smithi196 ceph-mon[30802]: 2.2e scrub ok 2024-09-16T05:19:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:29 smithi196 ceph-mon[30802]: osdmap e619: 12 total, 12 up, 12 in 2024-09-16T05:19:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:29 smithi196 ceph-mon[30802]: 2.51 scrub starts 2024-09-16T05:19:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:29 smithi196 ceph-mon[30802]: 2.51 scrub ok 2024-09-16T05:19:30.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:30.328 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:30.328 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:30.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:30.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:30.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:30.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:30.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:30.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:30.596 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:30.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:30.697 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1091,"stamp":"2024-09-16T05:19:29.819275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":366816922,"num_objects":177,"num_object_clones":125,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":16,"num_read":2754,"num_read_kb":2727707,"num_write":1992,"num_write_kb":817351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":26862704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25626,"ondisk_log_size":25626,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":962276,"kb_used_data":542476,"kb_used_omap":27,"kb_used_meta":419492,"kb_avail":1123979548,"statfs":{"total":1151940427776,"available":1150955057152,"internally_reserved":0,"allocated":555495424,"data_stored":674120303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":429559975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20009371,"num_objects":-10,"num_object_clones":-9,"num_object_copies":-20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":2,"num_read":30,"num_read_kb":30978,"num_write":18,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.046124"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":2995,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840653+0000","last_change":"2024-09-16T05:19:25.840876+0000","last_active":"2024-09-16T05:19:25.840653+0000","last_peered":"2024-09-16T05:19:25.840653+0000","last_clean":"2024-09-16T05:19:25.840653+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:19:25.840653+0000","last_undegraded":"2024-09-16T05:19:25.840653+0000","last_fullsized":"2024-09-16T05:19:25.840653+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:18:05.287115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:05.287115+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071237500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.54","version":"612'377","reported_seq":2498,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786475+0000","last_change":"2024-09-16T05:19:21.785431+0000","last_active":"2024-09-16T05:19:23.786475+0000","last_peered":"2024-09-16T05:19:23.786475+0000","last_clean":"2024-09-16T05:19:23.786475+0000","last_became_active":"2024-09-16T05:16:30.728460+0000","last_became_peered":"2024-09-16T05:16:30.728460+0000","last_unstale":"2024-09-16T05:19:23.786475+0000","last_undegraded":"2024-09-16T05:19:23.786475+0000","last_fullsized":"2024-09-16T05:19:23.786475+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"535'312","last_scrub_stamp":"2024-09-16T05:18:06.566530+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:18:06.566530+0000","objects_scrubbed":3,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:24.083011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014629420000000001,"stat_sum":{"num_bytes":7395509,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":54,"num_read_kb":57048,"num_write":28,"num_write_kb":9877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":886,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.787429+0000","last_change":"2024-09-16T05:19:25.787631+0000","last_active":"2024-09-16T05:19:25.787429+0000","last_peered":"2024-09-16T05:19:25.787429+0000","last_clean":"2024-09-16T05:19:25.787429+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:19:25.787429+0000","last_undegraded":"2024-09-16T05:19:25.787429+0000","last_fullsized":"2024-09-16T05:19:25.787429+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:18:08.046417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:08.046417+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041533899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.52","version":"272'69","reported_seq":884,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.388026+0000","last_change":"2024-09-16T05:19:21.780675+0000","last_active":"2024-09-16T05:19:25.388026+0000","last_peered":"2024-09-16T05:19:25.388026+0000","last_clean":"2024-09-16T05:19:25.388026+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:19:25.388026+0000","last_undegraded":"2024-09-16T05:19:25.388026+0000","last_fullsized":"2024-09-16T05:19:25.388026+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:19:08.575776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:08.575776+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:27.436801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087759299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":955,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840085+0000","last_change":"2024-09-16T05:19:25.840329+0000","last_active":"2024-09-16T05:19:25.840085+0000","last_peered":"2024-09-16T05:19:25.840085+0000","last_clean":"2024-09-16T05:19:25.840085+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:19:25.840085+0000","last_undegraded":"2024-09-16T05:19:25.840085+0000","last_fullsized":"2024-09-16T05:19:25.840085+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:18:05.726701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:05.726701+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042913599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.50","version":"182'16","reported_seq":895,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814100+0000","last_change":"2024-09-16T05:19:26.783525+0000","last_active":"2024-09-16T05:19:27.814100+0000","last_peered":"2024-09-16T05:19:27.814100+0000","last_clean":"2024-09-16T05:19:27.814100+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:19:27.814100+0000","last_undegraded":"2024-09-16T05:19:27.814100+0000","last_fullsized":"2024-09-16T05:19:27.814100+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:17:50.481238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:50.481238+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052715399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2786,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786529+0000","last_change":"2024-09-16T05:19:21.787162+0000","last_active":"2024-09-16T05:19:23.786529+0000","last_peered":"2024-09-16T05:19:23.786529+0000","last_clean":"2024-09-16T05:19:23.786529+0000","last_became_active":"2024-09-16T05:16:30.732139+0000","last_became_peered":"2024-09-16T05:16:30.732139+0000","last_unstale":"2024-09-16T05:19:23.786529+0000","last_undegraded":"2024-09-16T05:19:23.786529+0000","last_fullsized":"2024-09-16T05:19:23.786529+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:18:07.600000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:07.600000+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026077740000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.4e","version":"272'232","reported_seq":2656,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840945+0000","last_change":"2024-09-16T05:19:25.841273+0000","last_active":"2024-09-16T05:19:25.840945+0000","last_peered":"2024-09-16T05:19:25.840945+0000","last_clean":"2024-09-16T05:19:25.840945+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:19:25.840945+0000","last_undegraded":"2024-09-16T05:19:25.840945+0000","last_fullsized":"2024-09-16T05:19:25.840945+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:17:52.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:52.325783+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073032,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4d","version":"617'565","reported_seq":3155,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.803487+0000","last_change":"2024-09-16T05:19:26.803781+0000","last_active":"2024-09-16T05:19:26.803487+0000","last_peered":"2024-09-16T05:19:26.803487+0000","last_clean":"2024-09-16T05:19:26.803487+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:19:26.803487+0000","last_undegraded":"2024-09-16T05:19:26.803487+0000","last_fullsized":"2024-09-16T05:19:26.803487+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"595'560","last_scrub_stamp":"2024-09-16T05:19:08.148997+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:19:08.148997+0000","objects_scrubbed":5,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:31.171611+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0028255609999999999,"stat_sum":{"num_bytes":8866783,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":28336,"num_write":39,"num_write_kb":14373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4c","version":"608'337","reported_seq":2852,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.841025+0000","last_change":"2024-09-16T05:19:25.841163+0000","last_active":"2024-09-16T05:19:25.841025+0000","last_peered":"2024-09-16T05:19:25.841025+0000","last_clean":"2024-09-16T05:19:25.841025+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:19:25.841025+0000","last_undegraded":"2024-09-16T05:19:25.841025+0000","last_fullsized":"2024-09-16T05:19:25.841025+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"582'326","last_scrub_stamp":"2024-09-16T05:18:54.541881+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:18:54.541881+0000","objects_scrubbed":2,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:03.730115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010239279999999999,"stat_sum":{"num_bytes":7276652,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":16243,"num_write":28,"num_write_kb":7491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4b","version":"244'177","reported_seq":2698,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.803224+0000","last_change":"2024-09-16T05:19:26.803546+0000","last_active":"2024-09-16T05:19:26.803224+0000","last_peered":"2024-09-16T05:19:26.803224+0000","last_clean":"2024-09-16T05:19:26.803224+0000","last_became_active":"2024-09-16T05:17:21.741180+0000","last_became_peered":"2024-09-16T05:17:21.741180+0000","last_unstale":"2024-09-16T05:19:26.803224+0000","last_undegraded":"2024-09-16T05:19:26.803224+0000","last_fullsized":"2024-09-16T05:19:26.803224+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:18:24.156559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:24.156559+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:38.026675+0000","scrub_duration":970,"objects_trimmed":0,"snaptrim_duration":0.0022458980000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4a","version":"607'553","reported_seq":3032,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.841788+0000","last_change":"2024-09-16T05:19:25.841961+0000","last_active":"2024-09-16T05:19:25.841788+0000","last_peered":"2024-09-16T05:19:25.841788+0000","last_clean":"2024-09-16T05:19:25.841788+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:19:25.841788+0000","last_undegraded":"2024-09-16T05:19:25.841788+0000","last_fullsized":"2024-09-16T05:19:25.841788+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"527'510","last_scrub_stamp":"2024-09-16T05:18:24.643580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:24.643580+0000","objects_scrubbed":1,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:51.846050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016254170000000001,"stat_sum":{"num_bytes":6916518,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":15872,"num_write":35,"num_write_kb":18755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.49","version":"245'220","reported_seq":2733,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.841361+0000","last_change":"2024-09-16T05:19:25.841626+0000","last_active":"2024-09-16T05:19:25.841361+0000","last_peered":"2024-09-16T05:19:25.841361+0000","last_clean":"2024-09-16T05:19:25.841361+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:19:25.841361+0000","last_undegraded":"2024-09-16T05:19:25.841361+0000","last_fullsized":"2024-09-16T05:19:25.841361+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:19:07.553888+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.553888+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:17.821587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001287311,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.48","version":"611'280","reported_seq":2752,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387403+0000","last_change":"2024-09-16T05:19:21.783008+0000","last_active":"2024-09-16T05:19:25.387403+0000","last_peered":"2024-09-16T05:19:25.387403+0000","last_clean":"2024-09-16T05:19:25.387403+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:19:25.387403+0000","last_undegraded":"2024-09-16T05:19:25.387403+0000","last_fullsized":"2024-09-16T05:19:25.387403+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"550'253","last_scrub_stamp":"2024-09-16T05:18:23.942982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:23.942982+0000","objects_scrubbed":4,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:34.060146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00067080099999999995,"stat_sum":{"num_bytes":4550854,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":36106,"num_write":36,"num_write_kb":20700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":1502,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387845+0000","last_change":"2024-09-16T05:19:21.780981+0000","last_active":"2024-09-16T05:19:25.387845+0000","last_peered":"2024-09-16T05:19:25.387845+0000","last_clean":"2024-09-16T05:19:25.387845+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:19:25.387845+0000","last_undegraded":"2024-09-16T05:19:25.387845+0000","last_fullsized":"2024-09-16T05:19:25.387845+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:18:07.943110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:07.943110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057427499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":2511,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786318+0000","last_change":"2024-09-16T05:19:21.784977+0000","last_active":"2024-09-16T05:19:23.786318+0000","last_peered":"2024-09-16T05:19:23.786318+0000","last_clean":"2024-09-16T05:19:23.786318+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:19:23.786318+0000","last_undegraded":"2024-09-16T05:19:23.786318+0000","last_fullsized":"2024-09-16T05:19:23.786318+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:18:11.621883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:11.621883+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:32.755190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011496760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":2945,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786334+0000","last_change":"2024-09-16T05:19:21.786905+0000","last_active":"2024-09-16T05:19:23.786334+0000","last_peered":"2024-09-16T05:19:23.786334+0000","last_clean":"2024-09-16T05:19:23.786334+0000","last_became_active":"2024-09-16T05:17:59.867450+0000","last_became_peered":"2024-09-16T05:17:59.867450+0000","last_unstale":"2024-09-16T05:19:23.786334+0000","last_undegraded":"2024-09-16T05:19:23.786334+0000","last_fullsized":"2024-09-16T05:19:23.786334+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:19:09.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:09.927260+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:31.268876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025432089999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":2593,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786384+0000","last_change":"2024-09-16T05:19:21.785730+0000","last_active":"2024-09-16T05:19:23.786384+0000","last_peered":"2024-09-16T05:19:23.786384+0000","last_clean":"2024-09-16T05:19:23.786384+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:19:23.786384+0000","last_undegraded":"2024-09-16T05:19:23.786384+0000","last_fullsized":"2024-09-16T05:19:23.786384+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:18:05.534270+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:18:05.534270+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:25.824371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015358940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":1511,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.795430+0000","last_change":"2024-09-16T05:19:26.795638+0000","last_active":"2024-09-16T05:19:26.795430+0000","last_peered":"2024-09-16T05:19:26.795430+0000","last_clean":"2024-09-16T05:19:26.795430+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:19:26.795430+0000","last_undegraded":"2024-09-16T05:19:26.795430+0000","last_fullsized":"2024-09-16T05:19:26.795430+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:19:15.441056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.441056+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:26.455206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031787849999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.42","version":"182'20","reported_seq":1499,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.822842+0000","last_change":"2024-09-16T05:19:28.823038+0000","last_active":"2024-09-16T05:19:28.822842+0000","last_peered":"2024-09-16T05:19:28.822842+0000","last_clean":"2024-09-16T05:19:28.822842+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:19:28.822842+0000","last_undegraded":"2024-09-16T05:19:28.822842+0000","last_fullsized":"2024-09-16T05:19:28.822842+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:19:28.221037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.221037+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:36.310374+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032861869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.41","version":"178'50","reported_seq":341,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.388188+0000","last_change":"2024-09-16T05:19:21.780263+0000","last_active":"2024-09-16T05:19:25.388188+0000","last_peered":"2024-09-16T05:19:25.388188+0000","last_clean":"2024-09-16T05:19:25.388188+0000","last_became_active":"2024-09-16T05:18:19.768547+0000","last_became_peered":"2024-09-16T05:18:19.768547+0000","last_unstale":"2024-09-16T05:19:25.388188+0000","last_undegraded":"2024-09-16T05:19:25.388188+0000","last_fullsized":"2024-09-16T05:19:25.388188+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:18:46.761738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.761738+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:59.178034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048458000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":44,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.841674+0000","last_change":"2024-09-16T05:19:25.841985+0000","last_active":"2024-09-16T05:19:25.841674+0000","last_peered":"2024-09-16T05:19:25.841674+0000","last_clean":"2024-09-16T05:19:25.841674+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:19:25.841674+0000","last_undegraded":"2024-09-16T05:19:25.841674+0000","last_fullsized":"2024-09-16T05:19:25.841674+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:17:49.277366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:49.277366+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016063259999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3f","version":"617'389","reported_seq":2922,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.800877+0000","last_change":"2024-09-16T05:19:26.801258+0000","last_active":"2024-09-16T05:19:26.800877+0000","last_peered":"2024-09-16T05:19:26.800877+0000","last_clean":"2024-09-16T05:19:26.800877+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:19:26.800877+0000","last_undegraded":"2024-09-16T05:19:26.800877+0000","last_fullsized":"2024-09-16T05:19:26.800877+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'357","last_scrub_stamp":"2024-09-16T05:18:28.469781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:28.469781+0000","objects_scrubbed":3,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:56.254989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0086299340000000006,"stat_sum":{"num_bytes":4513332,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":73925,"num_write":41,"num_write_kb":10370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3e","version":"617'482","reported_seq":2971,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.785083+0000","last_change":"2024-09-16T05:19:26.785314+0000","last_active":"2024-09-16T05:19:26.785083+0000","last_peered":"2024-09-16T05:19:26.785083+0000","last_clean":"2024-09-16T05:19:26.785083+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:19:26.785083+0000","last_undegraded":"2024-09-16T05:19:26.785083+0000","last_fullsized":"2024-09-16T05:19:26.785083+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"597'477","last_scrub_stamp":"2024-09-16T05:19:11.226978+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.226978+0000","objects_scrubbed":3,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.642106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029209510000000002,"stat_sum":{"num_bytes":5378504,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":31,"num_read_kb":26573,"num_write":41,"num_write_kb":23888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3d","version":"290'477","reported_seq":3040,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786309+0000","last_change":"2024-09-16T05:19:21.787201+0000","last_active":"2024-09-16T05:19:23.786309+0000","last_peered":"2024-09-16T05:19:23.786309+0000","last_clean":"2024-09-16T05:19:23.786309+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:19:23.786309+0000","last_undegraded":"2024-09-16T05:19:23.786309+0000","last_fullsized":"2024-09-16T05:19:23.786309+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_clean_scrub_stamp":"2024-09-16T05:19:14.912091+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.126374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027246789999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.3c","version":"617'630","reported_seq":3245,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.812305+0000","last_change":"2024-09-16T05:19:26.795420+0000","last_active":"2024-09-16T05:19:26.812305+0000","last_peered":"2024-09-16T05:19:26.812305+0000","last_clean":"2024-09-16T05:19:26.812305+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:19:26.812305+0000","last_undegraded":"2024-09-16T05:19:26.812305+0000","last_fullsized":"2024-09-16T05:19:26.812305+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"555'537","last_scrub_stamp":"2024-09-16T05:18:25.440073+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:18:25.440073+0000","objects_scrubbed":5,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:51.698179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030861769999999998,"stat_sum":{"num_bytes":17307435,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":124,"num_read_kb":153859,"num_write":62,"num_write_kb":26999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3b","version":"607'401","reported_seq":2914,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.842718+0000","last_change":"2024-09-16T05:19:25.843035+0000","last_active":"2024-09-16T05:19:25.842718+0000","last_peered":"2024-09-16T05:19:25.842718+0000","last_clean":"2024-09-16T05:19:25.842718+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:19:25.842718+0000","last_undegraded":"2024-09-16T05:19:25.842718+0000","last_fullsized":"2024-09-16T05:19:25.842718+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"520'342","last_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:17:50.304398+0000","objects_scrubbed":3,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.001902787,"stat_sum":{"num_bytes":11842835,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":51,"num_read_kb":54614,"num_write":36,"num_write_kb":19014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3a","version":"607'571","reported_seq":3038,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387375+0000","last_change":"2024-09-16T05:19:21.786642+0000","last_active":"2024-09-16T05:19:25.387375+0000","last_peered":"2024-09-16T05:19:25.387375+0000","last_clean":"2024-09-16T05:19:25.387375+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:19:25.387375+0000","last_undegraded":"2024-09-16T05:19:25.387375+0000","last_fullsized":"2024-09-16T05:19:25.387375+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"578'556","last_scrub_stamp":"2024-09-16T05:18:47.796222+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:18:47.796222+0000","objects_scrubbed":2,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:06.024222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037063830000000002,"stat_sum":{"num_bytes":3949810,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":41546,"num_write":58,"num_write_kb":24043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":1582,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387371+0000","last_change":"2024-09-16T05:19:21.783366+0000","last_active":"2024-09-16T05:19:25.387371+0000","last_peered":"2024-09-16T05:19:25.387371+0000","last_clean":"2024-09-16T05:19:25.387371+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:19:25.387371+0000","last_undegraded":"2024-09-16T05:19:25.387371+0000","last_fullsized":"2024-09-16T05:19:25.387371+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:18:22.896360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:22.896360+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:31.982076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094413400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":1612,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.819994+0000","last_change":"2024-09-16T05:19:28.820273+0000","last_active":"2024-09-16T05:19:28.819994+0000","last_peered":"2024-09-16T05:19:28.819994+0000","last_clean":"2024-09-16T05:19:28.819994+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:19:28.819994+0000","last_undegraded":"2024-09-16T05:19:28.819994+0000","last_fullsized":"2024-09-16T05:19:28.819994+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:19:13.324899+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:19:13.324899+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:17.734589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066746899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.37","version":"0'0","reported_seq":444,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840430+0000","last_change":"2024-09-16T05:19:25.840674+0000","last_active":"2024-09-16T05:19:25.840430+0000","last_peered":"2024-09-16T05:19:25.840430+0000","last_clean":"2024-09-16T05:19:25.840430+0000","last_became_active":"2024-09-16T05:17:59.866037+0000","last_became_peered":"2024-09-16T05:17:59.866037+0000","last_unstale":"2024-09-16T05:19:25.840430+0000","last_undegraded":"2024-09-16T05:19:25.840430+0000","last_fullsized":"2024-09-16T05:19:25.840430+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:18:06.694045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:18:06.694045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067144899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.36","version":"272'60","reported_seq":1668,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786435+0000","last_change":"2024-09-16T05:19:21.786390+0000","last_active":"2024-09-16T05:19:23.786435+0000","last_peered":"2024-09-16T05:19:23.786435+0000","last_clean":"2024-09-16T05:19:23.786435+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:19:23.786435+0000","last_undegraded":"2024-09-16T05:19:23.786435+0000","last_fullsized":"2024-09-16T05:19:23.786435+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:18:46.886122+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.886122+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:01.087634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021573989999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.35","version":"611'678","reported_seq":3216,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.789297+0000","last_change":"2024-09-16T05:19:21.787598+0000","last_active":"2024-09-16T05:19:23.789297+0000","last_peered":"2024-09-16T05:19:23.789297+0000","last_clean":"2024-09-16T05:19:23.789297+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:19:23.789297+0000","last_undegraded":"2024-09-16T05:19:23.789297+0000","last_fullsized":"2024-09-16T05:19:23.789297+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'675","last_scrub_stamp":"2024-09-16T05:19:11.894332+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.894332+0000","objects_scrubbed":8,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:24.953143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030226509999999999,"stat_sum":{"num_bytes":15345649,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":112,"num_read_kb":114278,"num_write":70,"num_write_kb":33784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.34","version":"616'505","reported_seq":2632,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.843454+0000","last_change":"2024-09-16T05:19:25.843628+0000","last_active":"2024-09-16T05:19:25.843454+0000","last_peered":"2024-09-16T05:19:25.843454+0000","last_clean":"2024-09-16T05:19:25.843454+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:19:25.843454+0000","last_undegraded":"2024-09-16T05:19:25.843454+0000","last_fullsized":"2024-09-16T05:19:25.843454+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'482","last_scrub_stamp":"2024-09-16T05:19:12.637489+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.637489+0000","objects_scrubbed":8,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:20.405500+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.002923611,"stat_sum":{"num_bytes":20285174,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":131,"num_read_kb":141542,"num_write":81,"num_write_kb":33634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":1689,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792812+0000","last_change":"2024-09-16T05:19:26.793173+0000","last_active":"2024-09-16T05:19:26.792812+0000","last_peered":"2024-09-16T05:19:26.792812+0000","last_clean":"2024-09-16T05:19:26.792812+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:19:26.792812+0000","last_undegraded":"2024-09-16T05:19:26.792812+0000","last_fullsized":"2024-09-16T05:19:26.792812+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:16.451459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:16.451459+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:37.403447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083197700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.32","version":"617'206","reported_seq":1878,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.812125+0000","last_change":"2024-09-16T05:19:26.784504+0000","last_active":"2024-09-16T05:19:26.812125+0000","last_peered":"2024-09-16T05:19:26.812125+0000","last_clean":"2024-09-16T05:19:26.812125+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:19:26.812125+0000","last_undegraded":"2024-09-16T05:19:26.812125+0000","last_fullsized":"2024-09-16T05:19:26.812125+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"574'195","last_scrub_stamp":"2024-09-16T05:18:54.271472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:54.271472+0000","objects_scrubbed":3,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:59.339353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026757579999999999,"stat_sum":{"num_bytes":4185273,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":101969,"num_write":36,"num_write_kb":13989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.31","version":"595'244","reported_seq":2487,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.388456+0000","last_change":"2024-09-16T05:19:21.780744+0000","last_active":"2024-09-16T05:19:25.388456+0000","last_peered":"2024-09-16T05:19:25.388456+0000","last_clean":"2024-09-16T05:19:25.388456+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:19:25.388456+0000","last_undegraded":"2024-09-16T05:19:25.388456+0000","last_fullsized":"2024-09-16T05:19:25.388456+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"574'240","last_scrub_stamp":"2024-09-16T05:18:45.804025+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:18:45.804025+0000","objects_scrubbed":3,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:13.247188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00085256399999999999,"stat_sum":{"num_bytes":8605547,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":119770,"num_write":34,"num_write_kb":17961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":432,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.841219+0000","last_change":"2024-09-16T05:19:25.841474+0000","last_active":"2024-09-16T05:19:25.841219+0000","last_peered":"2024-09-16T05:19:25.841219+0000","last_clean":"2024-09-16T05:19:25.841219+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:19:25.841219+0000","last_undegraded":"2024-09-16T05:19:25.841219+0000","last_fullsized":"2024-09-16T05:19:25.841219+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:11.486148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.486148+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:39.921532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001270906,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.12","version":"35'139","reported_seq":858,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.839897+0000","last_change":"2024-09-16T05:18:50.373017+0000","last_active":"2024-09-16T05:19:25.839897+0000","last_peered":"2024-09-16T05:19:25.839897+0000","last_clean":"2024-09-16T05:19:25.839897+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:19:25.839897+0000","last_undegraded":"2024-09-16T05:19:25.839897+0000","last_fullsized":"2024-09-16T05:19:25.839897+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:18:50.372923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:07.282374+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":2238,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.781651+0000","last_change":"2024-09-16T05:19:26.781924+0000","last_active":"2024-09-16T05:19:26.781651+0000","last_peered":"2024-09-16T05:19:26.781651+0000","last_clean":"2024-09-16T05:19:26.781651+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:19:26.781651+0000","last_undegraded":"2024-09-16T05:19:26.781651+0000","last_fullsized":"2024-09-16T05:19:26.781651+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:18:55.680516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:55.680516+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:22.839031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045476099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.13","version":"35'139","reported_seq":1481,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786098+0000","last_change":"2024-09-16T05:18:47.848719+0000","last_active":"2024-09-16T05:19:23.786098+0000","last_peered":"2024-09-16T05:19:23.786098+0000","last_clean":"2024-09-16T05:19:23.786098+0000","last_became_active":"2024-09-16T05:15:56.714681+0000","last_became_peered":"2024-09-16T05:15:56.714681+0000","last_unstale":"2024-09-16T05:19:23.786098+0000","last_undegraded":"2024-09-16T05:19:23.786098+0000","last_fullsized":"2024-09-16T05:19:23.786098+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:47.848648+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:47.848648+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:12.572545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":2168,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.823315+0000","last_change":"2024-09-16T05:19:28.823573+0000","last_active":"2024-09-16T05:19:28.823315+0000","last_peered":"2024-09-16T05:19:28.823315+0000","last_clean":"2024-09-16T05:19:28.823315+0000","last_became_active":"2024-09-16T05:17:37.757474+0000","last_became_peered":"2024-09-16T05:17:37.757474+0000","last_unstale":"2024-09-16T05:19:28.823315+0000","last_undegraded":"2024-09-16T05:19:28.823315+0000","last_fullsized":"2024-09-16T05:19:28.823315+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:11.306770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.306770+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:34.041998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035779140000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.c","version":"35'139","reported_seq":810,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814383+0000","last_change":"2024-09-16T05:18:50.280781+0000","last_active":"2024-09-16T05:19:27.814383+0000","last_peered":"2024-09-16T05:19:27.814383+0000","last_clean":"2024-09-16T05:19:27.814383+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:19:27.814383+0000","last_undegraded":"2024-09-16T05:19:27.814383+0000","last_fullsized":"2024-09-16T05:19:27.814383+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:50.280689+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:50.280689+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:17.557749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"619'394","reported_seq":2946,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.822560+0000","last_change":"2024-09-16T05:19:28.822753+0000","last_active":"2024-09-16T05:19:28.822560+0000","last_peered":"2024-09-16T05:19:28.822560+0000","last_clean":"2024-09-16T05:19:28.822560+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:19:28.822560+0000","last_undegraded":"2024-09-16T05:19:28.822560+0000","last_fullsized":"2024-09-16T05:19:28.822560+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'384","last_scrub_stamp":"2024-09-16T05:19:15.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.295498+0000","objects_scrubbed":6,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:29.649684+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.003004989,"stat_sum":{"num_bytes":10454383,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":41,"num_read_kb":57662,"num_write":37,"num_write_kb":14770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.d","version":"35'139","reported_seq":973,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792200+0000","last_change":"2024-09-16T05:19:12.454525+0000","last_active":"2024-09-16T05:19:26.792200+0000","last_peered":"2024-09-16T05:19:26.792200+0000","last_clean":"2024-09-16T05:19:26.792200+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:19:26.792200+0000","last_undegraded":"2024-09-16T05:19:26.792200+0000","last_fullsized":"2024-09-16T05:19:26.792200+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:12.454446+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.454446+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.180360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":2665,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.803594+0000","last_change":"2024-09-16T05:19:26.803902+0000","last_active":"2024-09-16T05:19:26.803594+0000","last_peered":"2024-09-16T05:19:26.803594+0000","last_clean":"2024-09-16T05:19:26.803594+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:19:26.803594+0000","last_undegraded":"2024-09-16T05:19:26.803594+0000","last_fullsized":"2024-09-16T05:19:26.803594+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:11.209692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.209692+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:39.121207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026253380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.e","version":"35'139","reported_seq":960,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.786952+0000","last_change":"2024-09-16T05:18:44.055494+0000","last_active":"2024-09-16T05:19:25.786952+0000","last_peered":"2024-09-16T05:19:25.786952+0000","last_clean":"2024-09-16T05:19:25.786952+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:19:25.786952+0000","last_undegraded":"2024-09-16T05:19:25.786952+0000","last_fullsized":"2024-09-16T05:19:25.786952+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:44.055402+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:44.055402+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:45.956832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"613'572","reported_seq":3092,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.388357+0000","last_change":"2024-09-16T05:19:21.792197+0000","last_active":"2024-09-16T05:19:25.388357+0000","last_peered":"2024-09-16T05:19:25.388357+0000","last_clean":"2024-09-16T05:19:25.388357+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:19:25.388357+0000","last_undegraded":"2024-09-16T05:19:25.388357+0000","last_fullsized":"2024-09-16T05:19:25.388357+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"526'516","last_scrub_stamp":"2024-09-16T05:18:09.979691+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:18:09.979691+0000","objects_scrubbed":2,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0088928549999999999,"stat_sum":{"num_bytes":8207102,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":43,"num_read_kb":35724,"num_write":44,"num_write_kb":24018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":945,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840759+0000","last_change":"2024-09-16T05:17:27.280715+0000","last_active":"2024-09-16T05:19:25.840759+0000","last_peered":"2024-09-16T05:19:25.840759+0000","last_clean":"2024-09-16T05:19:25.840759+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:19:25.840759+0000","last_undegraded":"2024-09-16T05:19:25.840759+0000","last_fullsized":"2024-09-16T05:19:25.840759+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:27.280637+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:17:27.280637+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:26.742383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":2792,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.782275+0000","last_change":"2024-09-16T05:19:26.782490+0000","last_active":"2024-09-16T05:19:26.782275+0000","last_peered":"2024-09-16T05:19:26.782275+0000","last_clean":"2024-09-16T05:19:26.782275+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:19:26.782275+0000","last_undegraded":"2024-09-16T05:19:26.782275+0000","last_fullsized":"2024-09-16T05:19:26.782275+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:19:10.225027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.225027+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:18.177678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040657999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.b","version":"35'139","reported_seq":1487,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.781770+0000","last_change":"2024-09-16T05:18:12.154120+0000","last_active":"2024-09-16T05:19:26.781770+0000","last_peered":"2024-09-16T05:19:26.781770+0000","last_clean":"2024-09-16T05:19:26.781770+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:19:26.781770+0000","last_undegraded":"2024-09-16T05:19:26.781770+0000","last_fullsized":"2024-09-16T05:19:26.781770+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:12.154035+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:18:12.154035+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:31.168264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"617'305","reported_seq":2829,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.798642+0000","last_change":"2024-09-16T05:19:26.795358+0000","last_active":"2024-09-16T05:19:26.798642+0000","last_peered":"2024-09-16T05:19:26.798642+0000","last_clean":"2024-09-16T05:19:26.798642+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:19:26.798642+0000","last_undegraded":"2024-09-16T05:19:26.798642+0000","last_fullsized":"2024-09-16T05:19:26.798642+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'297","last_scrub_stamp":"2024-09-16T05:19:10.483443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.483443+0000","objects_scrubbed":2,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.000173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033236170000000001,"stat_sum":{"num_bytes":5496655,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":27814,"num_write":51,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3","version":"612'190","reported_seq":2720,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840592+0000","last_change":"2024-09-16T05:19:25.841355+0000","last_active":"2024-09-16T05:19:25.840592+0000","last_peered":"2024-09-16T05:19:25.840592+0000","last_clean":"2024-09-16T05:19:25.840592+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:19:25.840592+0000","last_undegraded":"2024-09-16T05:19:25.840592+0000","last_fullsized":"2024-09-16T05:19:25.840592+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"570'180","last_scrub_stamp":"2024-09-16T05:18:44.321479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:44.321479+0000","objects_scrubbed":3,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:55.584702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001270757,"stat_sum":{"num_bytes":8807264,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":77,"num_read_kb":91279,"num_write":38,"num_write_kb":16751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":498,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.781517+0000","last_change":"2024-09-16T05:18:26.212176+0000","last_active":"2024-09-16T05:19:26.781517+0000","last_peered":"2024-09-16T05:19:26.781517+0000","last_clean":"2024-09-16T05:19:26.781517+0000","last_became_active":"2024-09-16T05:18:19.768392+0000","last_became_peered":"2024-09-16T05:18:19.768392+0000","last_unstale":"2024-09-16T05:19:26.781517+0000","last_undegraded":"2024-09-16T05:19:26.781517+0000","last_fullsized":"2024-09-16T05:19:26.781517+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:26.212094+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:26.212094+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:38.132735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":2520,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.842075+0000","last_change":"2024-09-16T05:19:25.842386+0000","last_active":"2024-09-16T05:19:25.842075+0000","last_peered":"2024-09-16T05:19:25.842075+0000","last_clean":"2024-09-16T05:19:25.842075+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:19:25.842075+0000","last_undegraded":"2024-09-16T05:19:25.842075+0000","last_fullsized":"2024-09-16T05:19:25.842075+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:18:23.222127+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:18:23.222127+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001590183,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":1045,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.839943+0000","last_change":"2024-09-16T05:18:10.716507+0000","last_active":"2024-09-16T05:19:25.839943+0000","last_peered":"2024-09-16T05:19:25.839943+0000","last_clean":"2024-09-16T05:19:25.839943+0000","last_became_active":"2024-09-16T05:18:02.888512+0000","last_became_peered":"2024-09-16T05:18:02.888512+0000","last_unstale":"2024-09-16T05:19:25.839943+0000","last_undegraded":"2024-09-16T05:19:25.839943+0000","last_fullsized":"2024-09-16T05:19:25.839943+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:10.716391+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:10.716391+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:27.515923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1478,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792149+0000","last_change":"2024-09-16T05:18:29.435915+0000","last_active":"2024-09-16T05:19:26.792149+0000","last_peered":"2024-09-16T05:19:26.792149+0000","last_clean":"2024-09-16T05:19:26.792149+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:19:26.792149+0000","last_undegraded":"2024-09-16T05:19:26.792149+0000","last_fullsized":"2024-09-16T05:19:26.792149+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:29.435789+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:29.435789+0000","last_clean_scrub_stamp":"2024-09-16T05:18:29.435789+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:40.106447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":2474,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.801989+0000","last_change":"2024-09-16T05:19:26.802230+0000","last_active":"2024-09-16T05:19:26.801989+0000","last_peered":"2024-09-16T05:19:26.801989+0000","last_clean":"2024-09-16T05:19:26.801989+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:19:26.801989+0000","last_undegraded":"2024-09-16T05:19:26.801989+0000","last_fullsized":"2024-09-16T05:19:26.801989+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:18:54.327254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:54.327254+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:00.723077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012316269999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.3","version":"35'139","reported_seq":1491,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814251+0000","last_change":"2024-09-16T05:19:10.336320+0000","last_active":"2024-09-16T05:19:27.814251+0000","last_peered":"2024-09-16T05:19:27.814251+0000","last_clean":"2024-09-16T05:19:27.814251+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:19:27.814251+0000","last_undegraded":"2024-09-16T05:19:27.814251+0000","last_fullsized":"2024-09-16T05:19:27.814251+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:10.336203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.336203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.544788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":2554,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.795519+0000","last_change":"2024-09-16T05:19:26.795810+0000","last_active":"2024-09-16T05:19:26.795519+0000","last_peered":"2024-09-16T05:19:26.795519+0000","last_clean":"2024-09-16T05:19:26.795519+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:19:26.795519+0000","last_undegraded":"2024-09-16T05:19:26.795519+0000","last_fullsized":"2024-09-16T05:19:26.795519+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:18:26.463618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:26.463618+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:38.937207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033311,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":1752,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.388297+0000","last_change":"2024-09-16T05:19:21.791616+0000","last_active":"2024-09-16T05:19:25.388297+0000","last_peered":"2024-09-16T05:19:25.388297+0000","last_clean":"2024-09-16T05:19:25.388297+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:19:25.388297+0000","last_undegraded":"2024-09-16T05:19:25.388297+0000","last_fullsized":"2024-09-16T05:19:25.388297+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:18:26.961312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:26.961312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:39.504786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0088148129999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":1443,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.782151+0000","last_change":"2024-09-16T05:18:28.258901+0000","last_active":"2024-09-16T05:19:26.782151+0000","last_peered":"2024-09-16T05:19:26.782151+0000","last_clean":"2024-09-16T05:19:26.782151+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:19:26.782151+0000","last_undegraded":"2024-09-16T05:19:26.782151+0000","last_fullsized":"2024-09-16T05:19:26.782151+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:28.258800+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:28.258800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:30.389977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":948,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.802296+0000","last_change":"2024-09-16T05:19:26.802631+0000","last_active":"2024-09-16T05:19:26.802296+0000","last_peered":"2024-09-16T05:19:26.802296+0000","last_clean":"2024-09-16T05:19:26.802296+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:19:26.802296+0000","last_undegraded":"2024-09-16T05:19:26.802296+0000","last_fullsized":"2024-09-16T05:19:26.802296+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:18:27.279121+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:18:27.279121+0000","last_clean_scrub_stamp":"2024-09-16T05:18:27.279121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:50.013181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015937900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":758,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840191+0000","last_change":"2024-09-16T05:19:13.533926+0000","last_active":"2024-09-16T05:19:25.840191+0000","last_peered":"2024-09-16T05:19:25.840191+0000","last_clean":"2024-09-16T05:19:25.840191+0000","last_became_active":"2024-09-16T05:18:02.897275+0000","last_became_peered":"2024-09-16T05:18:02.897275+0000","last_unstale":"2024-09-16T05:19:25.840191+0000","last_undegraded":"2024-09-16T05:19:25.840191+0000","last_fullsized":"2024-09-16T05:19:25.840191+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:19:13.533798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:23.927688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"619'555","reported_seq":3092,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.822204+0000","last_change":"2024-09-16T05:19:28.822459+0000","last_active":"2024-09-16T05:19:28.822204+0000","last_peered":"2024-09-16T05:19:28.822204+0000","last_clean":"2024-09-16T05:19:28.822204+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:19:28.822204+0000","last_undegraded":"2024-09-16T05:19:28.822204+0000","last_fullsized":"2024-09-16T05:19:28.822204+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'526","last_scrub_stamp":"2024-09-16T05:18:49.197415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:49.197415+0000","objects_scrubbed":3,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:50.082931+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.00275699,"stat_sum":{"num_bytes":5128831,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":89,"num_read_kb":98598,"num_write":33,"num_write_kb":8652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":694,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.839764+0000","last_change":"2024-09-16T05:18:09.725282+0000","last_active":"2024-09-16T05:19:25.839764+0000","last_peered":"2024-09-16T05:19:25.839764+0000","last_clean":"2024-09-16T05:19:25.839764+0000","last_became_active":"2024-09-16T05:18:02.886933+0000","last_became_peered":"2024-09-16T05:18:02.886933+0000","last_unstale":"2024-09-16T05:19:25.839764+0000","last_undegraded":"2024-09-16T05:19:25.839764+0000","last_fullsized":"2024-09-16T05:19:25.839764+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:09.725207+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:09.725207+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":2590,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786686+0000","last_change":"2024-09-16T05:19:21.786180+0000","last_active":"2024-09-16T05:19:23.786686+0000","last_peered":"2024-09-16T05:19:23.786686+0000","last_clean":"2024-09-16T05:19:23.786686+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:19:23.786686+0000","last_undegraded":"2024-09-16T05:19:23.786686+0000","last_fullsized":"2024-09-16T05:19:23.786686+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:18:44.900790+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:18:44.900790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:51.916142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017314069999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":738,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.839778+0000","last_change":"2024-09-16T05:19:15.510805+0000","last_active":"2024-09-16T05:19:25.839778+0000","last_peered":"2024-09-16T05:19:25.839778+0000","last_clean":"2024-09-16T05:19:25.839778+0000","last_became_active":"2024-09-16T05:18:03.588064+0000","last_became_peered":"2024-09-16T05:18:03.588064+0000","last_unstale":"2024-09-16T05:19:25.839778+0000","last_undegraded":"2024-09-16T05:19:25.839778+0000","last_fullsized":"2024-09-16T05:19:25.839778+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:15.510717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.510717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:27.556487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":2709,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.822910+0000","last_change":"2024-09-16T05:19:28.823135+0000","last_active":"2024-09-16T05:19:28.822910+0000","last_peered":"2024-09-16T05:19:28.822910+0000","last_clean":"2024-09-16T05:19:28.822910+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:19:28.822910+0000","last_undegraded":"2024-09-16T05:19:28.822910+0000","last_fullsized":"2024-09-16T05:19:28.822910+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:18:09.082699+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:18:09.082699+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032531579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":1477,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840730+0000","last_change":"2024-09-16T05:17:44.296951+0000","last_active":"2024-09-16T05:19:25.840730+0000","last_peered":"2024-09-16T05:19:25.840730+0000","last_clean":"2024-09-16T05:19:25.840730+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:19:25.840730+0000","last_undegraded":"2024-09-16T05:19:25.840730+0000","last_fullsized":"2024-09-16T05:19:25.840730+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:44.296866+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:44.296866+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:26.742404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"617'575","reported_seq":3125,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.789459+0000","last_change":"2024-09-16T05:19:26.789871+0000","last_active":"2024-09-16T05:19:26.789459+0000","last_peered":"2024-09-16T05:19:26.789459+0000","last_clean":"2024-09-16T05:19:26.789459+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:19:26.789459+0000","last_undegraded":"2024-09-16T05:19:26.789459+0000","last_fullsized":"2024-09-16T05:19:26.789459+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"574'545","last_scrub_stamp":"2024-09-16T05:18:47.380196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:47.380196+0000","objects_scrubbed":4,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:07.942956+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0070007170000000001,"stat_sum":{"num_bytes":8084662,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":113160,"num_write":47,"num_write_kb":31911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.9","version":"35'139","reported_seq":1471,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792014+0000","last_change":"2024-09-16T05:19:19.900629+0000","last_active":"2024-09-16T05:19:26.792014+0000","last_peered":"2024-09-16T05:19:26.792014+0000","last_clean":"2024-09-16T05:19:26.792014+0000","last_became_active":"2024-09-16T05:19:19.900294+0000","last_became_peered":"2024-09-16T05:19:19.900294+0000","last_unstale":"2024-09-16T05:19:26.792014+0000","last_undegraded":"2024-09-16T05:19:26.792014+0000","last_fullsized":"2024-09-16T05:19:26.792014+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:23.110100+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:23.110100+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:40.719939+0000","scrub_duration":633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2727,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786168+0000","last_change":"2024-09-16T05:19:21.784964+0000","last_active":"2024-09-16T05:19:23.786168+0000","last_peered":"2024-09-16T05:19:23.786168+0000","last_clean":"2024-09-16T05:19:23.786168+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:19:23.786168+0000","last_undegraded":"2024-09-16T05:19:23.786168+0000","last_fullsized":"2024-09-16T05:19:23.786168+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:18:09.660812+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:18:09.660812+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:27.913964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016866380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1471,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387370+0000","last_change":"2024-09-16T05:17:47.986778+0000","last_active":"2024-09-16T05:19:25.387370+0000","last_peered":"2024-09-16T05:19:25.387370+0000","last_clean":"2024-09-16T05:19:25.387370+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:19:25.387370+0000","last_undegraded":"2024-09-16T05:19:25.387370+0000","last_fullsized":"2024-09-16T05:19:25.387370+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:17:47.986643+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":961,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.388292+0000","last_change":"2024-09-16T05:18:26.950493+0000","last_active":"2024-09-16T05:19:25.388292+0000","last_peered":"2024-09-16T05:19:25.388292+0000","last_clean":"2024-09-16T05:19:25.388292+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:19:25.388292+0000","last_undegraded":"2024-09-16T05:19:25.388292+0000","last_fullsized":"2024-09-16T05:19:25.388292+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:18:26.950417+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:44.447670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":432,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840029+0000","last_change":"2024-09-16T05:19:25.840411+0000","last_active":"2024-09-16T05:19:25.840029+0000","last_peered":"2024-09-16T05:19:25.840029+0000","last_clean":"2024-09-16T05:19:25.840029+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:19:25.840029+0000","last_undegraded":"2024-09-16T05:19:25.840029+0000","last_fullsized":"2024-09-16T05:19:25.840029+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:19:07.382518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.382518+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:29.392566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058557900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.10","version":"35'139","reported_seq":1477,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840549+0000","last_change":"2024-09-16T05:17:28.243643+0000","last_active":"2024-09-16T05:19:25.840549+0000","last_peered":"2024-09-16T05:19:25.840549+0000","last_clean":"2024-09-16T05:19:25.840549+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:19:25.840549+0000","last_undegraded":"2024-09-16T05:19:25.840549+0000","last_fullsized":"2024-09-16T05:19:25.840549+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:28.243528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:28.243528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:26.742450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":2134,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387670+0000","last_change":"2024-09-16T05:19:21.787895+0000","last_active":"2024-09-16T05:19:25.387670+0000","last_peered":"2024-09-16T05:19:25.387670+0000","last_clean":"2024-09-16T05:19:25.387670+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:19:25.387670+0000","last_undegraded":"2024-09-16T05:19:25.387670+0000","last_fullsized":"2024-09-16T05:19:25.387670+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:18:06.988836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:06.988836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0048017010000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.14","version":"433'289","reported_seq":2414,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387807+0000","last_change":"2024-09-16T05:19:21.787168+0000","last_active":"2024-09-16T05:19:25.387807+0000","last_peered":"2024-09-16T05:19:25.387807+0000","last_clean":"2024-09-16T05:19:25.387807+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:19:25.387807+0000","last_undegraded":"2024-09-16T05:19:25.387807+0000","last_fullsized":"2024-09-16T05:19:25.387807+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:19:12.999550+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.999550+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:14.521810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041052279999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.15","version":"612'595","reported_seq":3130,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.388249+0000","last_change":"2024-09-16T05:19:21.791146+0000","last_active":"2024-09-16T05:19:25.388249+0000","last_peered":"2024-09-16T05:19:25.388249+0000","last_clean":"2024-09-16T05:19:25.388249+0000","last_became_active":"2024-09-16T05:16:25.732256+0000","last_became_peered":"2024-09-16T05:16:25.732256+0000","last_unstale":"2024-09-16T05:19:25.388249+0000","last_undegraded":"2024-09-16T05:19:25.388249+0000","last_fullsized":"2024-09-16T05:19:25.388249+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"539'540","last_scrub_stamp":"2024-09-16T05:18:08.972490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:08.972490+0000","objects_scrubbed":2,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:30.887605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0079081169999999992,"stat_sum":{"num_bytes":11893566,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":51,"num_read_kb":47656,"num_write":20,"num_write_kb":13595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.16","version":"612'261","reported_seq":2420,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.842473+0000","last_change":"2024-09-16T05:19:25.842669+0000","last_active":"2024-09-16T05:19:25.842473+0000","last_peered":"2024-09-16T05:19:25.842473+0000","last_clean":"2024-09-16T05:19:25.842473+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:19:25.842473+0000","last_undegraded":"2024-09-16T05:19:25.842473+0000","last_fullsized":"2024-09-16T05:19:25.842473+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"509'201","last_scrub_stamp":"2024-09-16T05:18:22.258029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:22.258029+0000","objects_scrubbed":1,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:46.596945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001691344,"stat_sum":{"num_bytes":6102326,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":58,"num_read_kb":57069,"num_write":44,"num_write_kb":12941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.14","version":"35'139","reported_seq":808,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.800893+0000","last_change":"2024-09-16T05:18:46.335095+0000","last_active":"2024-09-16T05:19:26.800893+0000","last_peered":"2024-09-16T05:19:26.800893+0000","last_clean":"2024-09-16T05:19:26.800893+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:19:26.800893+0000","last_undegraded":"2024-09-16T05:19:26.800893+0000","last_fullsized":"2024-09-16T05:19:26.800893+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:46.335030+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.335030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:02.571329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":646,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-16T05:19:23.786337+0000","last_change":"2024-09-16T05:19:21.786335+0000","last_active":"2024-09-16T05:19:23.786337+0000","last_peered":"2024-09-16T05:19:23.786337+0000","last_clean":"2024-09-16T05:19:23.786337+0000","last_became_active":"2024-09-16T05:17:59.867069+0000","last_became_peered":"2024-09-16T05:17:59.867069+0000","last_unstale":"2024-09-16T05:19:23.786337+0000","last_undegraded":"2024-09-16T05:19:23.786337+0000","last_fullsized":"2024-09-16T05:19:23.786337+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:12.884960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.884960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:17.649200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002006711,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":2516,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387529+0000","last_change":"2024-09-16T05:19:21.783500+0000","last_active":"2024-09-16T05:19:25.387529+0000","last_peered":"2024-09-16T05:19:25.387529+0000","last_clean":"2024-09-16T05:19:25.387529+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:19:25.387529+0000","last_undegraded":"2024-09-16T05:19:25.387529+0000","last_fullsized":"2024-09-16T05:19:25.387529+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:18:06.018399+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:18:06.018399+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141438,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":2146,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814119+0000","last_change":"2024-09-16T05:19:26.784394+0000","last_active":"2024-09-16T05:19:27.814119+0000","last_peered":"2024-09-16T05:19:27.814119+0000","last_clean":"2024-09-16T05:19:27.814119+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:19:27.814119+0000","last_undegraded":"2024-09-16T05:19:27.814119+0000","last_fullsized":"2024-09-16T05:19:27.814119+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:19:07.269103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.269103+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:12.081147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001136387,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1a","version":"619'874","reported_seq":3376,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.839893+0000","last_change":"2024-09-16T05:19:28.827292+0000","last_active":"2024-09-16T05:19:28.839893+0000","last_peered":"2024-09-16T05:19:28.839893+0000","last_clean":"2024-09-16T05:19:28.839893+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:19:28.839893+0000","last_undegraded":"2024-09-16T05:19:28.839893+0000","last_fullsized":"2024-09-16T05:19:28.839893+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"582'782","last_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:18:52.154075+0000","objects_scrubbed":10,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:55.888467+0000","scrub_duration":29,"objects_trimmed":1,"snaptrim_duration":0.0072678259999999998,"stat_sum":{"num_bytes":27132561,"num_objects":13,"num_object_clones":10,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":130,"num_read_kb":111565,"num_write":131,"num_write_kb":50098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1b","version":"617'557","reported_seq":3109,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.786264+0000","last_change":"2024-09-16T05:19:26.786713+0000","last_active":"2024-09-16T05:19:26.786264+0000","last_peered":"2024-09-16T05:19:26.786264+0000","last_clean":"2024-09-16T05:19:26.786264+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:19:26.786264+0000","last_undegraded":"2024-09-16T05:19:26.786264+0000","last_fullsized":"2024-09-16T05:19:26.786264+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"579'497","last_scrub_stamp":"2024-09-16T05:18:49.298238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:49.298238+0000","objects_scrubbed":8,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:15.558248+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0047604559999999997,"stat_sum":{"num_bytes":15302782,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":89,"num_read_kb":89718,"num_write":86,"num_write_kb":25738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1c","version":"611'440","reported_seq":2655,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.841509+0000","last_change":"2024-09-16T05:19:25.841726+0000","last_active":"2024-09-16T05:19:25.841509+0000","last_peered":"2024-09-16T05:19:25.841509+0000","last_clean":"2024-09-16T05:19:25.841509+0000","last_became_active":"2024-09-16T05:18:02.890972+0000","last_became_peered":"2024-09-16T05:18:02.890972+0000","last_unstale":"2024-09-16T05:19:25.841509+0000","last_undegraded":"2024-09-16T05:19:25.841509+0000","last_fullsized":"2024-09-16T05:19:25.841509+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"468'393","last_scrub_stamp":"2024-09-16T05:18:07.717683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:07.717683+0000","objects_scrubbed":0,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:29.068751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015051400000000001,"stat_sum":{"num_bytes":4092279,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":8,"num_read_kb":3,"num_write":18,"num_write_kb":10300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1d","version":"616'1087","reported_seq":3848,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.843629+0000","last_change":"2024-09-16T05:19:25.843870+0000","last_active":"2024-09-16T05:19:25.843629+0000","last_peered":"2024-09-16T05:19:25.843629+0000","last_clean":"2024-09-16T05:19:25.843629+0000","last_became_active":"2024-09-16T05:18:03.891103+0000","last_became_peered":"2024-09-16T05:18:03.891103+0000","last_unstale":"2024-09-16T05:19:25.843629+0000","last_undegraded":"2024-09-16T05:19:25.843629+0000","last_fullsized":"2024-09-16T05:19:25.843629+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"535'926","last_scrub_stamp":"2024-09-16T05:18:08.738088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:08.738088+0000","objects_scrubbed":10,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:31.529095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0038717069999999998,"stat_sum":{"num_bytes":27078340,"num_objects":15,"num_object_clones":11,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":3,"num_read":226,"num_read_kb":172010,"num_write":217,"num_write_kb":71091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7155994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1e","version":"617'430","reported_seq":2993,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814359+0000","last_change":"2024-09-16T05:19:26.786772+0000","last_active":"2024-09-16T05:19:27.814359+0000","last_peered":"2024-09-16T05:19:27.814359+0000","last_clean":"2024-09-16T05:19:27.814359+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:19:27.814359+0000","last_undegraded":"2024-09-16T05:19:27.814359+0000","last_fullsized":"2024-09-16T05:19:27.814359+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"602'413","last_scrub_stamp":"2024-09-16T05:19:15.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.386065+0000","objects_scrubbed":4,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:31.860686+0000","scrub_duration":4,"objects_trimmed":1,"snaptrim_duration":0.0021839889999999999,"stat_sum":{"num_bytes":7814510,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":67,"num_read_kb":71847,"num_write":21,"num_write_kb":11620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1f","version":"284'221","reported_seq":2741,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387824+0000","last_change":"2024-09-16T05:19:21.781132+0000","last_active":"2024-09-16T05:19:25.387824+0000","last_peered":"2024-09-16T05:19:25.387824+0000","last_clean":"2024-09-16T05:19:25.387824+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:19:25.387824+0000","last_undegraded":"2024-09-16T05:19:25.387824+0000","last_fullsized":"2024-09-16T05:19:25.387824+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:19:09.544229+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:19:09.544229+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:18.954648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068830200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.20","version":"617'259","reported_seq":2811,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.804018+0000","last_change":"2024-09-16T05:19:26.804374+0000","last_active":"2024-09-16T05:19:26.804018+0000","last_peered":"2024-09-16T05:19:26.804018+0000","last_clean":"2024-09-16T05:19:26.804018+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:19:26.804018+0000","last_undegraded":"2024-09-16T05:19:26.804018+0000","last_fullsized":"2024-09-16T05:19:26.804018+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"574'228","last_scrub_stamp":"2024-09-16T05:18:44.488210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:44.488210+0000","objects_scrubbed":3,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:12.803837+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.01122269,"stat_sum":{"num_bytes":4713311,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":136173,"num_write":48,"num_write_kb":18826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.21","version":"178'50","reported_seq":2473,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.802686+0000","last_change":"2024-09-16T05:19:26.803146+0000","last_active":"2024-09-16T05:19:26.802686+0000","last_peered":"2024-09-16T05:19:26.802686+0000","last_clean":"2024-09-16T05:19:26.802686+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:19:26.802686+0000","last_undegraded":"2024-09-16T05:19:26.802686+0000","last_fullsized":"2024-09-16T05:19:26.802686+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:18:55.296457+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:18:55.296457+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:56.279635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00192373,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.22","version":"182'20","reported_seq":432,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.840734+0000","last_change":"2024-09-16T05:19:25.840991+0000","last_active":"2024-09-16T05:19:25.840734+0000","last_peered":"2024-09-16T05:19:25.840734+0000","last_clean":"2024-09-16T05:19:25.840734+0000","last_became_active":"2024-09-16T05:17:59.866047+0000","last_became_peered":"2024-09-16T05:17:59.866047+0000","last_unstale":"2024-09-16T05:19:25.840734+0000","last_undegraded":"2024-09-16T05:19:25.840734+0000","last_fullsized":"2024-09-16T05:19:25.840734+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:18:53.530513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:53.530513+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:14.576015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011578140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":2013,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814371+0000","last_change":"2024-09-16T05:19:26.783778+0000","last_active":"2024-09-16T05:19:27.814371+0000","last_peered":"2024-09-16T05:19:27.814371+0000","last_clean":"2024-09-16T05:19:27.814371+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:19:27.814371+0000","last_undegraded":"2024-09-16T05:19:27.814371+0000","last_fullsized":"2024-09-16T05:19:27.814371+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:18:48.291334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:48.291334+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:59.071253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054042200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.24","version":"182'89","reported_seq":1809,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.387384+0000","last_change":"2024-09-16T05:19:21.784075+0000","last_active":"2024-09-16T05:19:25.387384+0000","last_peered":"2024-09-16T05:19:25.387384+0000","last_clean":"2024-09-16T05:19:25.387384+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:19:25.387384+0000","last_undegraded":"2024-09-16T05:19:25.387384+0000","last_fullsized":"2024-09-16T05:19:25.387384+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:18:43.800415+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:18:43.800415+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:58.326492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014053080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]},{"pgid":"2.25","version":"619'956","reported_seq":3415,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.831419+0000","last_change":"2024-09-16T05:19:28.831635+0000","last_active":"2024-09-16T05:19:28.831419+0000","last_peered":"2024-09-16T05:19:28.831419+0000","last_clean":"2024-09-16T05:19:28.831419+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:19:28.831419+0000","last_undegraded":"2024-09-16T05:19:28.831419+0000","last_fullsized":"2024-09-16T05:19:28.831419+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"554'782","last_scrub_stamp":"2024-09-16T05:18:25.127285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:25.127285+0000","objects_scrubbed":7,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:28.930168+0000","scrub_duration":2,"objects_trimmed":3,"snaptrim_duration":0.011469935000000001,"stat_sum":{"num_bytes":27887788,"num_objects":11,"num_object_clones":8,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":138,"num_read_kb":147474,"num_write":126,"num_write_kb":54777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.26","version":"116'9","reported_seq":1818,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.801327+0000","last_change":"2024-09-16T05:19:26.801854+0000","last_active":"2024-09-16T05:19:26.801327+0000","last_peered":"2024-09-16T05:19:26.801327+0000","last_clean":"2024-09-16T05:19:26.801327+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:19:26.801327+0000","last_undegraded":"2024-09-16T05:19:26.801327+0000","last_fullsized":"2024-09-16T05:19:26.801327+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:07.483055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.483055+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:18.790615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088676199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.27","version":"0'0","reported_seq":1830,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.795703+0000","last_change":"2024-09-16T05:19:26.796052+0000","last_active":"2024-09-16T05:19:26.795703+0000","last_peered":"2024-09-16T05:19:26.795703+0000","last_clean":"2024-09-16T05:19:26.795703+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:19:26.795703+0000","last_undegraded":"2024-09-16T05:19:26.795703+0000","last_fullsized":"2024-09-16T05:19:26.795703+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:11.499561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.499561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:14.020884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035743770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.28","version":"245'105","reported_seq":841,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.303133+0000","last_change":"2024-09-16T05:19:27.303160+0000","last_active":"2024-09-16T05:19:27.303133+0000","last_peered":"2024-09-16T05:19:27.303133+0000","last_clean":"2024-09-16T05:19:27.303133+0000","last_became_active":"2024-09-16T05:19:22.783233+0000","last_became_peered":"2024-09-16T05:19:22.783233+0000","last_unstale":"2024-09-16T05:19:27.303133+0000","last_undegraded":"2024-09-16T05:19:27.303133+0000","last_fullsized":"2024-09-16T05:19:27.303133+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:19:27.303070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:27.303070+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.887894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010796600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.29","version":"617'832","reported_seq":3396,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.822515+0000","last_change":"2024-09-16T05:19:26.822756+0000","last_active":"2024-09-16T05:19:26.822515+0000","last_peered":"2024-09-16T05:19:26.822515+0000","last_clean":"2024-09-16T05:19:26.822515+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:19:26.822515+0000","last_undegraded":"2024-09-16T05:19:26.822515+0000","last_fullsized":"2024-09-16T05:19:26.822515+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"554'720","last_scrub_stamp":"2024-09-16T05:18:25.489701+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:18:25.489701+0000","objects_scrubbed":5,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:33.047953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.041415041999999999,"stat_sum":{"num_bytes":15301385,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":208,"num_read_kb":205445,"num_write":148,"num_write_kb":57937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2a","version":"245'325","reported_seq":2812,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792365+0000","last_change":"2024-09-16T05:19:26.792734+0000","last_active":"2024-09-16T05:19:26.792365+0000","last_peered":"2024-09-16T05:19:26.792365+0000","last_clean":"2024-09-16T05:19:26.792365+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:19:26.792365+0000","last_undegraded":"2024-09-16T05:19:26.792365+0000","last_fullsized":"2024-09-16T05:19:26.792365+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:18:08.448119+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:18:08.448119+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000643612,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2b","version":"617'314","reported_seq":2766,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.808927+0000","last_change":"2024-09-16T05:19:26.809319+0000","last_active":"2024-09-16T05:19:26.808927+0000","last_peered":"2024-09-16T05:19:26.808927+0000","last_clean":"2024-09-16T05:19:26.808927+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:19:26.808927+0000","last_undegraded":"2024-09-16T05:19:26.808927+0000","last_fullsized":"2024-09-16T05:19:26.808927+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"578'284","last_scrub_stamp":"2024-09-16T05:18:50.293732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:50.293732+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:11.182825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0079029979999999996,"stat_sum":{"num_bytes":10454614,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":79889,"num_write":47,"num_write_kb":21198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2c","version":"241'194","reported_seq":2656,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.823247+0000","last_change":"2024-09-16T05:19:28.823519+0000","last_active":"2024-09-16T05:19:28.823247+0000","last_peered":"2024-09-16T05:19:28.823247+0000","last_clean":"2024-09-16T05:19:28.823247+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:19:28.823247+0000","last_undegraded":"2024-09-16T05:19:28.823247+0000","last_fullsized":"2024-09-16T05:19:28.823247+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:18:23.169260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:23.169260+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:35.469119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0036232899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2d","version":"617'438","reported_seq":2960,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.806863+0000","last_change":"2024-09-16T05:19:26.809043+0000","last_active":"2024-09-16T05:19:26.806863+0000","last_peered":"2024-09-16T05:19:26.806863+0000","last_clean":"2024-09-16T05:19:26.806863+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:19:26.806863+0000","last_undegraded":"2024-09-16T05:19:26.806863+0000","last_fullsized":"2024-09-16T05:19:26.806863+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"579'423","last_scrub_stamp":"2024-09-16T05:18:51.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:51.529954+0000","objects_scrubbed":3,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:57.944044+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.015797683999999999,"stat_sum":{"num_bytes":7037966,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":86,"num_read_kb":88579,"num_write":45,"num_write_kb":19868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2e","version":"617'552","reported_seq":3032,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.803162+0000","last_change":"2024-09-16T05:19:26.803555+0000","last_active":"2024-09-16T05:19:26.803162+0000","last_peered":"2024-09-16T05:19:26.803162+0000","last_clean":"2024-09-16T05:19:26.803162+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:19:26.803162+0000","last_undegraded":"2024-09-16T05:19:26.803162+0000","last_fullsized":"2024-09-16T05:19:26.803162+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"519'444","last_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:17:49.581405+0000","objects_scrubbed":3,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":1,"snaptrim_duration":0.010798466,"stat_sum":{"num_bytes":13653493,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":63,"num_read_kb":42111,"num_write":76,"num_write_kb":26273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2f","version":"612'510","reported_seq":3065,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-09-16T05:19:25.388077+0000","last_change":"2024-09-16T05:19:21.790531+0000","last_active":"2024-09-16T05:19:25.388077+0000","last_peered":"2024-09-16T05:19:25.388077+0000","last_clean":"2024-09-16T05:19:25.388077+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:19:25.388077+0000","last_undegraded":"2024-09-16T05:19:25.388077+0000","last_fullsized":"2024-09-16T05:19:25.388077+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"579'461","last_scrub_stamp":"2024-09-16T05:18:50.885995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:50.885995+0000","objects_scrubbed":4,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:01.347505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0070217980000000001,"stat_sum":{"num_bytes":15162861,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":99,"num_read_kb":72250,"num_write":90,"num_write_kb":39298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"d9","length":"1"},{"start":"ea","length":"1"},{"start":"ec","length":"1"},{"start":"ee","length":"1"},{"start":"f1","length":"1"},{"start":"f4","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":366226554,"num_objects":175,"num_object_clones":125,"num_object_copies":350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":16,"num_read":2754,"num_read_kb":2727707,"num_write":1992,"num_write_kb":817351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":26862704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":459825152,"data_stored":605270436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22707,"ondisk_log_size":22707,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089469,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99512,"kb_used_data":68324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93645640,"statfs":{"total":95995035648,"available":95893135360,"internally_reserved":0,"allocated":69963776,"data_stored":82471833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285697,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69556,"kb_used_data":42528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93675596,"statfs":{"total":95995035648,"available":95923810304,"internally_reserved":0,"allocated":43548672,"data_stored":52649862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514634,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106764,"kb_used_data":53368,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93638388,"statfs":{"total":95995035648,"available":95885709312,"internally_reserved":0,"allocated":54648832,"data_stored":67070116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841674,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99448,"kb_used_data":68252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93645704,"statfs":{"total":95995035648,"available":95893200896,"internally_reserved":0,"allocated":69890048,"data_stored":88873890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103312,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113388,"kb_used_data":59980,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93631764,"statfs":{"total":95995035648,"available":95878926336,"internally_reserved":0,"allocated":61419520,"data_stored":78355289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397655,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81072,"kb_used_data":49880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93664080,"statfs":{"total":95995035648,"available":95912017920,"internally_reserved":0,"allocated":51077120,"data_stored":68948306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757401,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82612,"kb_used_data":43096,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93662540,"statfs":{"total":95995035648,"available":95910440960,"internally_reserved":0,"allocated":44130304,"data_stored":57761710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920927,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49512,"kb_used_data":22480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93695640,"statfs":{"total":95995035648,"available":95944335360,"internally_reserved":0,"allocated":23019520,"data_stored":24898063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117156,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88128,"kb_used_data":56940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93657024,"statfs":{"total":95995035648,"available":95904792576,"internally_reserved":0,"allocated":58306560,"data_stored":67938841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240443,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57116,"kb_used_data":20680,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93688036,"statfs":{"total":95995035648,"available":95936548864,"internally_reserved":0,"allocated":21176320,"data_stored":20428560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67816,"kb_used_data":36628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93677336,"statfs":{"total":95995035648,"available":95925592064,"internally_reserved":0,"allocated":37507072,"data_stored":41551035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738547,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47352,"kb_used_data":20320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93697800,"statfs":{"total":95995035648,"available":95946547200,"internally_reserved":0,"allocated":20807680,"data_stored":23172798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13062144,"data_stored":17945751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30101504,"data_stored":36660116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13467648,"data_stored":15249449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50536448,"data_stored":62694738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15601664,"data_stored":19998696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36507648,"data_stored":52661095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":43634688,"data_stored":64031883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":52264960,"data_stored":73985279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":58753024,"data_stored":74558327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":47243264,"data_stored":62179197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36130816,"data_stored":47750495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":62521344,"data_stored":77555410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:19:30.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:19:30.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:30.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:30.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:30.779 INFO:tasks.rados.rados.2.smithi204.stdout:2382: done (2 left) 2024-09-16T05:19:30.779 INFO:tasks.rados.rados.2.smithi204.stdout:2383: snap_create 2024-09-16T05:19:30.780 INFO:tasks.rados.rados.2.smithi204.stdout:2381: finishing rollback tid 0 to smithi20457400-40 2024-09-16T05:19:30.780 INFO:tasks.rados.rados.2.smithi204.stdout:2381: finishing rollback tid 1 to smithi20457400-40 2024-09-16T05:19:30.780 INFO:tasks.rados.rados.2.smithi204.stdout:2381: finishing rollback tid 2 to smithi20457400-40 2024-09-16T05:19:30.780 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 774 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:19:30.780 INFO:tasks.rados.rados.2.smithi204.stdout:2380: finishing rollback tid 0 to smithi20457400-39 2024-09-16T05:19:30.780 INFO:tasks.rados.rados.2.smithi204.stdout:2380: finishing rollback tid 1 to smithi20457400-39 2024-09-16T05:19:30.780 INFO:tasks.rados.rados.2.smithi204.stdout:2380: finishing rollback tid 2 to smithi20457400-39 2024-09-16T05:19:30.780 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 960 (ObjNum 778 snap 226 seq_num 778) dirty exists 2024-09-16T05:19:30.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:30.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:30.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:30.850 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:19:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:19:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 2.54 scrub starts 2024-09-16T05:19:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 2.54 scrub ok 2024-09-16T05:19:30.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 2.50 scrub starts 2024-09-16T05:19:30.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 2.50 scrub ok 2024-09-16T05:19:30.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 2.2a scrub starts 2024-09-16T05:19:30.924 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 2.2a scrub ok 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: osdmap e620: 12 total, 12 up, 12 in 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: from='client.16542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:30.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:30 smithi204 ceph-mon[30933]: osdmap e621: 12 total, 12 up, 12 in 2024-09-16T05:19:30.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:30.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 2.54 scrub starts 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 2.54 scrub ok 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 2.50 scrub starts 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 2.50 scrub ok 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 2.2a scrub starts 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 2.2a scrub ok 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:19:30.972 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:19:30.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: osdmap e620: 12 total, 12 up, 12 in 2024-09-16T05:19:30.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: from='client.16542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:19:30.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:19:30.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:19:30.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:19:30.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:30.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:30 smithi032 ceph-mon[27388]: osdmap e621: 12 total, 12 up, 12 in 2024-09-16T05:19:31.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:31.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:31.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:31.159 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:31.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:19:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:19:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 2.54 scrub starts 2024-09-16T05:19:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 2.54 scrub ok 2024-09-16T05:19:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 2.50 scrub starts 2024-09-16T05:19:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 2.50 scrub ok 2024-09-16T05:19:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 2.2a scrub starts 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 2.2a scrub ok 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: osdmap e620: 12 total, 12 up, 12 in 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: from='client.16542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:31.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:30 smithi196 ceph-mon[30802]: osdmap e621: 12 total, 12 up, 12 in 2024-09-16T05:19:31.261 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:31.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:31.377 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:31.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:31.406 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:31.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:31.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:31.532 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:31.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:31.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:19:31.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:31.660 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:31.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:31.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:31.776 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:19:31.780 INFO:tasks.rados.rados.2.smithi204.stdout:2380: done (2 left) 2024-09-16T05:19:31.780 INFO:tasks.rados.rados.2.smithi204.stdout:2381: done (1 left) 2024-09-16T05:19:31.780 INFO:tasks.rados.rados.2.smithi204.stdout:2383: done (0 left) 2024-09-16T05:19:31.780 INFO:tasks.rados.rados.2.smithi204.stdout:2384: write (excl) oid 25 current snap is 248 2024-09-16T05:19:31.780 INFO:tasks.rados.rados.2.smithi204.stdout:2384: seq_num 853 ranges {736371=418933,1798905=434297} 2024-09-16T05:19:31.787 INFO:tasks.rados.rados.2.smithi204.stdout:2384: writing smithi20457400-25 from 736371 to 1155304 tid 1 2024-09-16T05:19:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:31.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:31.794 INFO:tasks.rados.rados.2.smithi204.stdout:2384: writing smithi20457400-25 from 1798905 to 2233202 tid 2 2024-09-16T05:19:31.794 INFO:tasks.rados.rados.2.smithi204.stdout:2385: delete oid 28 current snap is 248 2024-09-16T05:19:31.794 INFO:tasks.rados.rados.2.smithi204.stdout:2384: finishing write tid 1 to smithi20457400-25 2024-09-16T05:19:31.796 INFO:tasks.rados.rados.2.smithi204.stdout:2384: finishing write tid 2 to smithi20457400-25 2024-09-16T05:19:31.796 INFO:tasks.rados.rados.2.smithi204.stdout:2384: finishing write tid 3 to smithi20457400-25 2024-09-16T05:19:31.796 INFO:tasks.rados.rados.2.smithi204.stdout:2384: finishing write tid 4 to smithi20457400-25 2024-09-16T05:19:31.796 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 720 (ObjNum 853 snap 248 seq_num 853) dirty exists 2024-09-16T05:19:31.796 INFO:tasks.rados.rados.2.smithi204.stdout:2384: left oid 25 (ObjNum 853 snap 248 seq_num 853) 2024-09-16T05:19:31.798 INFO:tasks.rados.rados.2.smithi204.stdout:2384: done (1 left) 2024-09-16T05:19:31.798 INFO:tasks.rados.rados.2.smithi204.stdout:2385: done (0 left) 2024-09-16T05:19:31.798 INFO:tasks.rados.rados.2.smithi204.stdout:2386: rollback oid 50 current snap is 248 2024-09-16T05:19:31.798 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 240 2024-09-16T05:19:31.798 INFO:tasks.rados.rados.2.smithi204.stdout:2387: rollback oid 42 current snap is 248 2024-09-16T05:19:31.798 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 245 2024-09-16T05:19:31.798 INFO:tasks.rados.rados.2.smithi204.stdout:2388: write (excl) oid 13 current snap is 248 2024-09-16T05:19:31.798 INFO:tasks.rados.rados.2.smithi204.stdout:2388: seq_num 855 ranges {769985=569431,1915869=437966} 2024-09-16T05:19:31.805 INFO:tasks.rados.rados.2.smithi204.stdout:2388: writing smithi20457400-13 from 769985 to 1339416 tid 1 2024-09-16T05:19:31.810 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:31.811 INFO:tasks.rados.rados.2.smithi204.stdout:2388: writing smithi20457400-13 from 1915869 to 2353835 tid 2 2024-09-16T05:19:31.811 INFO:tasks.rados.rados.2.smithi204.stdout:2389: snap_remove snap 246 2024-09-16T05:19:31.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:31.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:31.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:32.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:19:32.026 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: pgmap v1092: 107 pgs: 107 active+clean; 350 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 397 KiB/s wr, 8 op/s 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2714824693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:19:32.028 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:19:32.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:32.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:19:32.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2714824693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:32.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:31 smithi032 ceph-mon[27388]: osdmap e622: 12 total, 12 up, 12 in 2024-09-16T05:19:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:32.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:32.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: pgmap v1092: 107 pgs: 107 active+clean; 350 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 397 KiB/s wr, 8 op/s 2024-09-16T05:19:32.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:19:32.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:19:32.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:19:32.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:19:32.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:19:32.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:19:32.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2714824693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:32.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:19:32.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:19:32.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:32.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:19:32.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2714824693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:32.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:31 smithi204 ceph-mon[30933]: osdmap e622: 12 total, 12 up, 12 in 2024-09-16T05:19:32.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:32.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: pgmap v1092: 107 pgs: 107 active+clean; 350 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 397 KiB/s wr, 8 op/s 2024-09-16T05:19:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:19:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:19:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:19:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:19:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:19:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:19:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2714824693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:19:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:19:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:19:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2714824693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:31 smithi196 ceph-mon[30802]: osdmap e622: 12 total, 12 up, 12 in 2024-09-16T05:19:32.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:32.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:32.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:32.321 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:32.322 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:32.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:32.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:32.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:32.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:32.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:32.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:32.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:32.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:32.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:32.686 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1094,"stamp":"2024-09-16T05:19:31.776615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":313744184,"num_objects":151,"num_object_clones":101,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":13,"num_read":2754,"num_read_kb":2727707,"num_write":1993,"num_write_kb":817351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":26862704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25726,"ondisk_log_size":25726,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":926256,"kb_used_data":506460,"kb_used_omap":27,"kb_used_meta":419492,"kb_avail":1124015568,"statfs":{"total":1151940427776,"available":1150991941632,"internally_reserved":0,"allocated":518615040,"data_stored":618386322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":429559975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67341915,"num_objects":-34,"num_object_clones":-31,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":-2,"num_read":-1,"num_read_kb":0,"num_write":5,"num_write_kb":-473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.963274"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":3019,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.823053+0000","last_change":"2024-09-16T05:19:29.824211+0000","last_active":"2024-09-16T05:19:29.823053+0000","last_peered":"2024-09-16T05:19:29.823053+0000","last_clean":"2024-09-16T05:19:29.823053+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:19:29.823053+0000","last_undegraded":"2024-09-16T05:19:29.823053+0000","last_fullsized":"2024-09-16T05:19:29.823053+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:18:05.287115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:05.287115+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028309799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.54","version":"620'383","reported_seq":2546,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.835965+0000","last_change":"2024-09-16T05:19:29.836209+0000","last_active":"2024-09-16T05:19:29.835965+0000","last_peered":"2024-09-16T05:19:29.835965+0000","last_clean":"2024-09-16T05:19:29.835965+0000","last_became_active":"2024-09-16T05:16:30.728460+0000","last_became_peered":"2024-09-16T05:16:30.728460+0000","last_unstale":"2024-09-16T05:19:29.835965+0000","last_undegraded":"2024-09-16T05:19:29.835965+0000","last_fullsized":"2024-09-16T05:19:29.835965+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"619'381","last_scrub_stamp":"2024-09-16T05:19:29.293042+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.293042+0000","objects_scrubbed":3,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:46.270630+0000","scrub_duration":115,"objects_trimmed":1,"snaptrim_duration":0.0072623339999999996,"stat_sum":{"num_bytes":3042232,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":54,"num_read_kb":57048,"num_write":28,"num_write_kb":9877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.53","version":"181'15","reported_seq":917,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.776607+0000","last_change":"2024-09-16T05:19:30.776880+0000","last_active":"2024-09-16T05:19:30.776607+0000","last_peered":"2024-09-16T05:19:30.776607+0000","last_clean":"2024-09-16T05:19:30.776607+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:19:30.776607+0000","last_undegraded":"2024-09-16T05:19:30.776607+0000","last_fullsized":"2024-09-16T05:19:30.776607+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:18:08.046417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:08.046417+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048096,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":915,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.833320+0000","last_change":"2024-09-16T05:19:29.833609+0000","last_active":"2024-09-16T05:19:29.833320+0000","last_peered":"2024-09-16T05:19:29.833320+0000","last_clean":"2024-09-16T05:19:29.833320+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:19:29.833320+0000","last_undegraded":"2024-09-16T05:19:29.833320+0000","last_fullsized":"2024-09-16T05:19:29.833320+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:19:08.575776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:08.575776+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:36.730956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099386500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.51","version":"272'115","reported_seq":994,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.776929+0000","last_change":"2024-09-16T05:19:30.777121+0000","last_active":"2024-09-16T05:19:30.776929+0000","last_peered":"2024-09-16T05:19:30.776929+0000","last_clean":"2024-09-16T05:19:30.776929+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:19:30.776929+0000","last_undegraded":"2024-09-16T05:19:30.776929+0000","last_fullsized":"2024-09-16T05:19:30.776929+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:19:29.589220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.589220+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:51.655045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084575099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":895,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814100+0000","last_change":"2024-09-16T05:19:26.783525+0000","last_active":"2024-09-16T05:19:27.814100+0000","last_peered":"2024-09-16T05:19:27.814100+0000","last_clean":"2024-09-16T05:19:27.814100+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:19:27.814100+0000","last_undegraded":"2024-09-16T05:19:27.814100+0000","last_fullsized":"2024-09-16T05:19:27.814100+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:17:50.481238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:50.481238+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052715399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2820,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.836849+0000","last_change":"2024-09-16T05:19:29.837038+0000","last_active":"2024-09-16T05:19:29.836849+0000","last_peered":"2024-09-16T05:19:29.836849+0000","last_clean":"2024-09-16T05:19:29.836849+0000","last_became_active":"2024-09-16T05:16:30.732139+0000","last_became_peered":"2024-09-16T05:16:30.732139+0000","last_unstale":"2024-09-16T05:19:29.836849+0000","last_undegraded":"2024-09-16T05:19:29.836849+0000","last_fullsized":"2024-09-16T05:19:29.836849+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:18:07.600000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:07.600000+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0065091799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4e","version":"272'232","reported_seq":2680,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.824279+0000","last_change":"2024-09-16T05:19:29.824569+0000","last_active":"2024-09-16T05:19:29.824279+0000","last_peered":"2024-09-16T05:19:29.824279+0000","last_clean":"2024-09-16T05:19:29.824279+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:19:29.824279+0000","last_undegraded":"2024-09-16T05:19:29.824279+0000","last_fullsized":"2024-09-16T05:19:29.824279+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:17:52.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:52.325783+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030955800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4d","version":"617'565","reported_seq":3155,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.803487+0000","last_change":"2024-09-16T05:19:26.803781+0000","last_active":"2024-09-16T05:19:26.803487+0000","last_peered":"2024-09-16T05:19:26.803487+0000","last_clean":"2024-09-16T05:19:26.803487+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:19:26.803487+0000","last_undegraded":"2024-09-16T05:19:26.803487+0000","last_fullsized":"2024-09-16T05:19:26.803487+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"595'560","last_scrub_stamp":"2024-09-16T05:19:08.148997+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:19:08.148997+0000","objects_scrubbed":5,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:31.171611+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0028255609999999999,"stat_sum":{"num_bytes":8866783,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":28336,"num_write":39,"num_write_kb":14373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4c","version":"621'341","reported_seq":2887,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.778909+0000","last_change":"2024-09-16T05:19:30.779337+0000","last_active":"2024-09-16T05:19:30.778909+0000","last_peered":"2024-09-16T05:19:30.778909+0000","last_clean":"2024-09-16T05:19:30.778909+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:19:30.778909+0000","last_undegraded":"2024-09-16T05:19:30.778909+0000","last_fullsized":"2024-09-16T05:19:30.778909+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"582'326","last_scrub_stamp":"2024-09-16T05:18:54.541881+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:18:54.541881+0000","objects_scrubbed":2,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:57.450370+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0028552209999999998,"stat_sum":{"num_bytes":4923540,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":16243,"num_write":28,"num_write_kb":7491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":2698,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.803224+0000","last_change":"2024-09-16T05:19:26.803546+0000","last_active":"2024-09-16T05:19:26.803224+0000","last_peered":"2024-09-16T05:19:26.803224+0000","last_clean":"2024-09-16T05:19:26.803224+0000","last_became_active":"2024-09-16T05:17:21.741180+0000","last_became_peered":"2024-09-16T05:17:21.741180+0000","last_unstale":"2024-09-16T05:19:26.803224+0000","last_undegraded":"2024-09-16T05:19:26.803224+0000","last_fullsized":"2024-09-16T05:19:26.803224+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:18:24.156559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:24.156559+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:38.026675+0000","scrub_duration":970,"objects_trimmed":0,"snaptrim_duration":0.0022458980000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.4a","version":"620'557","reported_seq":3067,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.777858+0000","last_change":"2024-09-16T05:19:30.778087+0000","last_active":"2024-09-16T05:19:30.777858+0000","last_peered":"2024-09-16T05:19:30.777858+0000","last_clean":"2024-09-16T05:19:30.777858+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:19:30.777858+0000","last_undegraded":"2024-09-16T05:19:30.777858+0000","last_fullsized":"2024-09-16T05:19:30.777858+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"527'510","last_scrub_stamp":"2024-09-16T05:18:24.643580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:24.643580+0000","objects_scrubbed":1,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:45.957721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001449395,"stat_sum":{"num_bytes":2689285,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":15872,"num_write":35,"num_write_kb":18755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":2757,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.821536+0000","last_change":"2024-09-16T05:19:29.822968+0000","last_active":"2024-09-16T05:19:29.821536+0000","last_peered":"2024-09-16T05:19:29.821536+0000","last_clean":"2024-09-16T05:19:29.821536+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:19:29.821536+0000","last_undegraded":"2024-09-16T05:19:29.821536+0000","last_fullsized":"2024-09-16T05:19:29.821536+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:19:07.553888+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.553888+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:16.501002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010907969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.48","version":"616'282","reported_seq":2785,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.828954+0000","last_change":"2024-09-16T05:19:29.829300+0000","last_active":"2024-09-16T05:19:29.828954+0000","last_peered":"2024-09-16T05:19:29.828954+0000","last_clean":"2024-09-16T05:19:29.828954+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:19:29.828954+0000","last_undegraded":"2024-09-16T05:19:29.828954+0000","last_fullsized":"2024-09-16T05:19:29.828954+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"550'253","last_scrub_stamp":"2024-09-16T05:18:23.942982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:23.942982+0000","objects_scrubbed":4,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:47.679519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0033823619999999999,"stat_sum":{"num_bytes":4550854,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":36106,"num_write":36,"num_write_kb":20700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.47","version":"0'0","reported_seq":1541,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.390427+0000","last_change":"2024-09-16T05:19:30.390451+0000","last_active":"2024-09-16T05:19:30.390427+0000","last_peered":"2024-09-16T05:19:30.390427+0000","last_clean":"2024-09-16T05:19:30.390427+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:19:30.390427+0000","last_undegraded":"2024-09-16T05:19:30.390427+0000","last_fullsized":"2024-09-16T05:19:30.390427+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:30.390373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.390373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:36.769867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071750899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.46","version":"116'9","reported_seq":2545,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.832918+0000","last_change":"2024-09-16T05:19:29.833186+0000","last_active":"2024-09-16T05:19:29.832918+0000","last_peered":"2024-09-16T05:19:29.832918+0000","last_clean":"2024-09-16T05:19:29.832918+0000","last_became_active":"2024-09-16T05:14:27.834411+0000","last_became_peered":"2024-09-16T05:14:27.834411+0000","last_unstale":"2024-09-16T05:19:29.832918+0000","last_undegraded":"2024-09-16T05:19:29.832918+0000","last_fullsized":"2024-09-16T05:19:29.832918+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:18:11.621883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:11.621883+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:41.592210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032222969999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.45","version":"305'464","reported_seq":2979,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.831413+0000","last_change":"2024-09-16T05:19:29.832442+0000","last_active":"2024-09-16T05:19:29.831413+0000","last_peered":"2024-09-16T05:19:29.831413+0000","last_clean":"2024-09-16T05:19:29.831413+0000","last_became_active":"2024-09-16T05:17:59.867450+0000","last_became_peered":"2024-09-16T05:17:59.867450+0000","last_unstale":"2024-09-16T05:19:29.831413+0000","last_undegraded":"2024-09-16T05:19:29.831413+0000","last_fullsized":"2024-09-16T05:19:29.831413+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:19:09.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:09.927260+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:18.759281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002642026,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":2635,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.829262+0000","last_change":"2024-09-16T05:19:29.831032+0000","last_active":"2024-09-16T05:19:29.829262+0000","last_peered":"2024-09-16T05:19:29.829262+0000","last_clean":"2024-09-16T05:19:29.829262+0000","last_became_active":"2024-09-16T05:14:27.835785+0000","last_became_peered":"2024-09-16T05:14:27.835785+0000","last_unstale":"2024-09-16T05:19:29.829262+0000","last_undegraded":"2024-09-16T05:19:29.829262+0000","last_fullsized":"2024-09-16T05:19:29.829262+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:28.142163+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.142163+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:57.697846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020456659999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":1511,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.795430+0000","last_change":"2024-09-16T05:19:26.795638+0000","last_active":"2024-09-16T05:19:26.795430+0000","last_peered":"2024-09-16T05:19:26.795430+0000","last_clean":"2024-09-16T05:19:26.795430+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:19:26.795430+0000","last_undegraded":"2024-09-16T05:19:26.795430+0000","last_fullsized":"2024-09-16T05:19:26.795430+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:19:15.441056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.441056+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:26.455206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031787849999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.42","version":"182'20","reported_seq":1499,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.822842+0000","last_change":"2024-09-16T05:19:28.823038+0000","last_active":"2024-09-16T05:19:28.822842+0000","last_peered":"2024-09-16T05:19:28.822842+0000","last_clean":"2024-09-16T05:19:28.822842+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:19:28.822842+0000","last_undegraded":"2024-09-16T05:19:28.822842+0000","last_fullsized":"2024-09-16T05:19:28.822842+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:19:28.221037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.221037+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:36.310374+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032861869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.41","version":"178'50","reported_seq":372,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.833699+0000","last_change":"2024-09-16T05:19:29.834023+0000","last_active":"2024-09-16T05:19:29.833699+0000","last_peered":"2024-09-16T05:19:29.833699+0000","last_clean":"2024-09-16T05:19:29.833699+0000","last_became_active":"2024-09-16T05:18:19.768547+0000","last_became_peered":"2024-09-16T05:18:19.768547+0000","last_unstale":"2024-09-16T05:19:29.833699+0000","last_undegraded":"2024-09-16T05:19:29.833699+0000","last_fullsized":"2024-09-16T05:19:29.833699+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:18:46.761738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.761738+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:03.167510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000738672,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.40","version":"205'33","reported_seq":68,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.824472+0000","last_change":"2024-09-16T05:19:29.824780+0000","last_active":"2024-09-16T05:19:29.824472+0000","last_peered":"2024-09-16T05:19:29.824472+0000","last_clean":"2024-09-16T05:19:29.824472+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:19:29.824472+0000","last_undegraded":"2024-09-16T05:19:29.824472+0000","last_fullsized":"2024-09-16T05:19:29.824472+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:17:49.277366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:17:49.277366+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031764979999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3f","version":"617'389","reported_seq":2922,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.800877+0000","last_change":"2024-09-16T05:19:26.801258+0000","last_active":"2024-09-16T05:19:26.800877+0000","last_peered":"2024-09-16T05:19:26.800877+0000","last_clean":"2024-09-16T05:19:26.800877+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:19:26.800877+0000","last_undegraded":"2024-09-16T05:19:26.800877+0000","last_fullsized":"2024-09-16T05:19:26.800877+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"557'357","last_scrub_stamp":"2024-09-16T05:18:28.469781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:28.469781+0000","objects_scrubbed":3,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:56.254989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0086299340000000006,"stat_sum":{"num_bytes":4513332,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":73925,"num_write":41,"num_write_kb":10370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3e","version":"617'482","reported_seq":2971,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.785083+0000","last_change":"2024-09-16T05:19:26.785314+0000","last_active":"2024-09-16T05:19:26.785083+0000","last_peered":"2024-09-16T05:19:26.785083+0000","last_clean":"2024-09-16T05:19:26.785083+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:19:26.785083+0000","last_undegraded":"2024-09-16T05:19:26.785083+0000","last_fullsized":"2024-09-16T05:19:26.785083+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"597'477","last_scrub_stamp":"2024-09-16T05:19:11.226978+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.226978+0000","objects_scrubbed":3,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.642106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029209510000000002,"stat_sum":{"num_bytes":5378504,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":31,"num_read_kb":26573,"num_write":41,"num_write_kb":23888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3d","version":"290'477","reported_seq":3074,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.832501+0000","last_change":"2024-09-16T05:19:29.832842+0000","last_active":"2024-09-16T05:19:29.832501+0000","last_peered":"2024-09-16T05:19:29.832501+0000","last_clean":"2024-09-16T05:19:29.832501+0000","last_became_active":"2024-09-16T05:13:48.691228+0000","last_became_peered":"2024-09-16T05:13:48.691228+0000","last_unstale":"2024-09-16T05:19:29.832501+0000","last_undegraded":"2024-09-16T05:19:29.832501+0000","last_fullsized":"2024-09-16T05:19:29.832501+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_clean_scrub_stamp":"2024-09-16T05:19:14.912091+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:20.533553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030437620000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3c","version":"617'630","reported_seq":3245,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.812305+0000","last_change":"2024-09-16T05:19:26.795420+0000","last_active":"2024-09-16T05:19:26.812305+0000","last_peered":"2024-09-16T05:19:26.812305+0000","last_clean":"2024-09-16T05:19:26.812305+0000","last_became_active":"2024-09-16T05:13:45.685387+0000","last_became_peered":"2024-09-16T05:13:45.685387+0000","last_unstale":"2024-09-16T05:19:26.812305+0000","last_undegraded":"2024-09-16T05:19:26.812305+0000","last_fullsized":"2024-09-16T05:19:26.812305+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"555'537","last_scrub_stamp":"2024-09-16T05:18:25.440073+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:18:25.440073+0000","objects_scrubbed":5,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:51.698179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030861769999999998,"stat_sum":{"num_bytes":17307435,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":124,"num_read_kb":153859,"num_write":62,"num_write_kb":26999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3b","version":"619'405","reported_seq":2942,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.821376+0000","last_change":"2024-09-16T05:19:29.822809+0000","last_active":"2024-09-16T05:19:29.821376+0000","last_peered":"2024-09-16T05:19:29.821376+0000","last_clean":"2024-09-16T05:19:29.821376+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:19:29.821376+0000","last_undegraded":"2024-09-16T05:19:29.821376+0000","last_fullsized":"2024-09-16T05:19:29.821376+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"520'342","last_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:17:50.304398+0000","objects_scrubbed":3,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00080321600000000004,"stat_sum":{"num_bytes":9119969,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":54614,"num_write":36,"num_write_kb":19014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3a","version":"607'571","reported_seq":3069,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.826746+0000","last_change":"2024-09-16T05:19:29.827265+0000","last_active":"2024-09-16T05:19:29.826746+0000","last_peered":"2024-09-16T05:19:29.826746+0000","last_clean":"2024-09-16T05:19:29.826746+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:19:29.826746+0000","last_undegraded":"2024-09-16T05:19:29.826746+0000","last_fullsized":"2024-09-16T05:19:29.826746+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"578'556","last_scrub_stamp":"2024-09-16T05:18:47.796222+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:18:47.796222+0000","objects_scrubbed":2,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:07.973138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001500012,"stat_sum":{"num_bytes":3949810,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":41546,"num_write":58,"num_write_kb":24043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.39","version":"182'60","reported_seq":1613,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.826069+0000","last_change":"2024-09-16T05:19:29.826639+0000","last_active":"2024-09-16T05:19:29.826069+0000","last_peered":"2024-09-16T05:19:29.826069+0000","last_clean":"2024-09-16T05:19:29.826069+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:19:29.826069+0000","last_undegraded":"2024-09-16T05:19:29.826069+0000","last_fullsized":"2024-09-16T05:19:29.826069+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:18:22.896360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:22.896360+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:32.155537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001056303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.38","version":"181'57","reported_seq":1612,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.819994+0000","last_change":"2024-09-16T05:19:28.820273+0000","last_active":"2024-09-16T05:19:28.819994+0000","last_peered":"2024-09-16T05:19:28.819994+0000","last_clean":"2024-09-16T05:19:28.819994+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:19:28.819994+0000","last_undegraded":"2024-09-16T05:19:28.819994+0000","last_fullsized":"2024-09-16T05:19:28.819994+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:19:13.324899+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:19:13.324899+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:17.734589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066746899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.37","version":"0'0","reported_seq":475,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.777573+0000","last_change":"2024-09-16T05:19:30.777795+0000","last_active":"2024-09-16T05:19:30.777573+0000","last_peered":"2024-09-16T05:19:30.777573+0000","last_clean":"2024-09-16T05:19:30.777573+0000","last_became_active":"2024-09-16T05:17:59.866037+0000","last_became_peered":"2024-09-16T05:17:59.866037+0000","last_unstale":"2024-09-16T05:19:30.777573+0000","last_undegraded":"2024-09-16T05:19:30.777573+0000","last_fullsized":"2024-09-16T05:19:30.777573+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:18:06.694045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:18:06.694045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001168809,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":1702,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.836553+0000","last_change":"2024-09-16T05:19:29.836844+0000","last_active":"2024-09-16T05:19:29.836553+0000","last_peered":"2024-09-16T05:19:29.836553+0000","last_clean":"2024-09-16T05:19:29.836553+0000","last_became_active":"2024-09-16T05:13:27.687465+0000","last_became_peered":"2024-09-16T05:13:27.687465+0000","last_unstale":"2024-09-16T05:19:29.836553+0000","last_undegraded":"2024-09-16T05:19:29.836553+0000","last_fullsized":"2024-09-16T05:19:29.836553+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:18:46.886122+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.886122+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:05.349129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.006523544,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.35","version":"620'686","reported_seq":3257,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.836524+0000","last_change":"2024-09-16T05:19:29.836787+0000","last_active":"2024-09-16T05:19:29.836524+0000","last_peered":"2024-09-16T05:19:29.836524+0000","last_clean":"2024-09-16T05:19:29.836524+0000","last_became_active":"2024-09-16T05:13:27.687820+0000","last_became_peered":"2024-09-16T05:13:27.687820+0000","last_unstale":"2024-09-16T05:19:29.836524+0000","last_undegraded":"2024-09-16T05:19:29.836524+0000","last_fullsized":"2024-09-16T05:19:29.836524+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'675","last_scrub_stamp":"2024-09-16T05:19:11.894332+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.894332+0000","objects_scrubbed":8,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:39.887553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0067131619999999999,"stat_sum":{"num_bytes":10437741,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":112,"num_read_kb":114278,"num_write":70,"num_write_kb":33784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.34","version":"620'513","reported_seq":2663,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.827762+0000","last_change":"2024-09-16T05:19:29.828107+0000","last_active":"2024-09-16T05:19:29.827762+0000","last_peered":"2024-09-16T05:19:29.827762+0000","last_clean":"2024-09-16T05:19:29.827762+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:19:29.827762+0000","last_undegraded":"2024-09-16T05:19:29.827762+0000","last_fullsized":"2024-09-16T05:19:29.827762+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'482","last_scrub_stamp":"2024-09-16T05:19:12.637489+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.637489+0000","objects_scrubbed":8,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:18.359756+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0060798889999999998,"stat_sum":{"num_bytes":13587850,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":131,"num_read_kb":141542,"num_write":81,"num_write_kb":33634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":1689,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792812+0000","last_change":"2024-09-16T05:19:26.793173+0000","last_active":"2024-09-16T05:19:26.792812+0000","last_peered":"2024-09-16T05:19:26.792812+0000","last_clean":"2024-09-16T05:19:26.792812+0000","last_became_active":"2024-09-16T05:13:27.682656+0000","last_became_peered":"2024-09-16T05:13:27.682656+0000","last_unstale":"2024-09-16T05:19:26.792812+0000","last_undegraded":"2024-09-16T05:19:26.792812+0000","last_fullsized":"2024-09-16T05:19:26.792812+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:16.451459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:16.451459+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:37.403447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083197700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.32","version":"617'206","reported_seq":1878,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.812125+0000","last_change":"2024-09-16T05:19:26.784504+0000","last_active":"2024-09-16T05:19:26.812125+0000","last_peered":"2024-09-16T05:19:26.812125+0000","last_clean":"2024-09-16T05:19:26.812125+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:19:26.812125+0000","last_undegraded":"2024-09-16T05:19:26.812125+0000","last_fullsized":"2024-09-16T05:19:26.812125+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"574'195","last_scrub_stamp":"2024-09-16T05:18:54.271472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:54.271472+0000","objects_scrubbed":3,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:59.339353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026757579999999999,"stat_sum":{"num_bytes":4185273,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":101969,"num_write":36,"num_write_kb":13989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.31","version":"617'246","reported_seq":2520,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.833933+0000","last_change":"2024-09-16T05:19:29.834224+0000","last_active":"2024-09-16T05:19:29.833933+0000","last_peered":"2024-09-16T05:19:29.833933+0000","last_clean":"2024-09-16T05:19:29.833933+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:19:29.833933+0000","last_undegraded":"2024-09-16T05:19:29.833933+0000","last_fullsized":"2024-09-16T05:19:29.833933+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"574'240","last_scrub_stamp":"2024-09-16T05:18:45.804025+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:18:45.804025+0000","objects_scrubbed":3,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:46.075851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00066297600000000002,"stat_sum":{"num_bytes":5496153,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":119770,"num_write":34,"num_write_kb":17961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.30","version":"182'16","reported_seq":463,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.776856+0000","last_change":"2024-09-16T05:19:30.777108+0000","last_active":"2024-09-16T05:19:30.776856+0000","last_peered":"2024-09-16T05:19:30.776856+0000","last_clean":"2024-09-16T05:19:30.776856+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:19:30.776856+0000","last_undegraded":"2024-09-16T05:19:30.776856+0000","last_fullsized":"2024-09-16T05:19:30.776856+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:11.486148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.486148+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:30.896727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00083539300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":866,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.820930+0000","last_change":"2024-09-16T05:18:50.373017+0000","last_active":"2024-09-16T05:19:29.820930+0000","last_peered":"2024-09-16T05:19:29.820930+0000","last_clean":"2024-09-16T05:19:29.820930+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:19:29.820930+0000","last_undegraded":"2024-09-16T05:19:29.820930+0000","last_fullsized":"2024-09-16T05:19:29.820930+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:18:50.372923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:07.282374+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":2238,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.781651+0000","last_change":"2024-09-16T05:19:26.781924+0000","last_active":"2024-09-16T05:19:26.781651+0000","last_peered":"2024-09-16T05:19:26.781651+0000","last_clean":"2024-09-16T05:19:26.781651+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:19:26.781651+0000","last_undegraded":"2024-09-16T05:19:26.781651+0000","last_fullsized":"2024-09-16T05:19:26.781651+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:18:55.680516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:55.680516+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:22.839031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045476099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.13","version":"35'139","reported_seq":1493,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.828442+0000","last_change":"2024-09-16T05:18:47.848719+0000","last_active":"2024-09-16T05:19:29.828442+0000","last_peered":"2024-09-16T05:19:29.828442+0000","last_clean":"2024-09-16T05:19:29.828442+0000","last_became_active":"2024-09-16T05:15:56.714681+0000","last_became_peered":"2024-09-16T05:15:56.714681+0000","last_unstale":"2024-09-16T05:19:29.828442+0000","last_undegraded":"2024-09-16T05:19:29.828442+0000","last_fullsized":"2024-09-16T05:19:29.828442+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:47.848648+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:47.848648+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:12.572545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":2168,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.823315+0000","last_change":"2024-09-16T05:19:28.823573+0000","last_active":"2024-09-16T05:19:28.823315+0000","last_peered":"2024-09-16T05:19:28.823315+0000","last_clean":"2024-09-16T05:19:28.823315+0000","last_became_active":"2024-09-16T05:17:37.757474+0000","last_became_peered":"2024-09-16T05:17:37.757474+0000","last_unstale":"2024-09-16T05:19:28.823315+0000","last_undegraded":"2024-09-16T05:19:28.823315+0000","last_fullsized":"2024-09-16T05:19:28.823315+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:11.306770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.306770+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:34.041998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035779140000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.c","version":"35'139","reported_seq":810,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814383+0000","last_change":"2024-09-16T05:18:50.280781+0000","last_active":"2024-09-16T05:19:27.814383+0000","last_peered":"2024-09-16T05:19:27.814383+0000","last_clean":"2024-09-16T05:19:27.814383+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:19:27.814383+0000","last_undegraded":"2024-09-16T05:19:27.814383+0000","last_fullsized":"2024-09-16T05:19:27.814383+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:50.280689+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:50.280689+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:17.557749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"619'394","reported_seq":2946,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.822560+0000","last_change":"2024-09-16T05:19:28.822753+0000","last_active":"2024-09-16T05:19:28.822560+0000","last_peered":"2024-09-16T05:19:28.822560+0000","last_clean":"2024-09-16T05:19:28.822560+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:19:28.822560+0000","last_undegraded":"2024-09-16T05:19:28.822560+0000","last_fullsized":"2024-09-16T05:19:28.822560+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'384","last_scrub_stamp":"2024-09-16T05:19:15.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.295498+0000","objects_scrubbed":6,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:29.649684+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.003004989,"stat_sum":{"num_bytes":10454383,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":41,"num_read_kb":57662,"num_write":37,"num_write_kb":14770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.d","version":"35'139","reported_seq":973,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792200+0000","last_change":"2024-09-16T05:19:12.454525+0000","last_active":"2024-09-16T05:19:26.792200+0000","last_peered":"2024-09-16T05:19:26.792200+0000","last_clean":"2024-09-16T05:19:26.792200+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:19:26.792200+0000","last_undegraded":"2024-09-16T05:19:26.792200+0000","last_fullsized":"2024-09-16T05:19:26.792200+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:12.454446+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.454446+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.180360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":2665,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.803594+0000","last_change":"2024-09-16T05:19:26.803902+0000","last_active":"2024-09-16T05:19:26.803594+0000","last_peered":"2024-09-16T05:19:26.803594+0000","last_clean":"2024-09-16T05:19:26.803594+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:19:26.803594+0000","last_undegraded":"2024-09-16T05:19:26.803594+0000","last_fullsized":"2024-09-16T05:19:26.803594+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:11.209692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.209692+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:39.121207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026253380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.e","version":"35'139","reported_seq":970,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.776229+0000","last_change":"2024-09-16T05:18:44.055494+0000","last_active":"2024-09-16T05:19:30.776229+0000","last_peered":"2024-09-16T05:19:30.776229+0000","last_clean":"2024-09-16T05:19:30.776229+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:19:30.776229+0000","last_undegraded":"2024-09-16T05:19:30.776229+0000","last_fullsized":"2024-09-16T05:19:30.776229+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:44.055402+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:44.055402+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:45.956832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"617'576","reported_seq":3134,"reported_epoch":620,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T05:19:29.830239+0000","last_change":"2024-09-16T05:19:29.830282+0000","last_active":"2024-09-16T05:19:29.830239+0000","last_peered":"2024-09-16T05:19:29.830239+0000","last_clean":"2024-09-16T05:19:29.830239+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:19:29.830239+0000","last_undegraded":"2024-09-16T05:19:29.830239+0000","last_fullsized":"2024-09-16T05:19:29.830239+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"617'576","last_scrub_stamp":"2024-09-16T05:19:26.973559+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:19:26.973559+0000","objects_scrubbed":4,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.752469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.007026554,"stat_sum":{"num_bytes":8207102,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":43,"num_read_kb":35724,"num_write":44,"num_write_kb":24018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.f","version":"35'139","reported_seq":961,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.821123+0000","last_change":"2024-09-16T05:19:29.650332+0000","last_active":"2024-09-16T05:19:29.821123+0000","last_peered":"2024-09-16T05:19:29.821123+0000","last_clean":"2024-09-16T05:19:29.821123+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:19:29.821123+0000","last_undegraded":"2024-09-16T05:19:29.821123+0000","last_fullsized":"2024-09-16T05:19:29.821123+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:29.650209+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.650209+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:34.196552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":2792,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.782275+0000","last_change":"2024-09-16T05:19:26.782490+0000","last_active":"2024-09-16T05:19:26.782275+0000","last_peered":"2024-09-16T05:19:26.782275+0000","last_clean":"2024-09-16T05:19:26.782275+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:19:26.782275+0000","last_undegraded":"2024-09-16T05:19:26.782275+0000","last_fullsized":"2024-09-16T05:19:26.782275+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:19:10.225027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.225027+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:18.177678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040657999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.b","version":"35'139","reported_seq":1487,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.781770+0000","last_change":"2024-09-16T05:18:12.154120+0000","last_active":"2024-09-16T05:19:26.781770+0000","last_peered":"2024-09-16T05:19:26.781770+0000","last_clean":"2024-09-16T05:19:26.781770+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:19:26.781770+0000","last_undegraded":"2024-09-16T05:19:26.781770+0000","last_fullsized":"2024-09-16T05:19:26.781770+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:12.154035+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:18:12.154035+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:31.168264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"617'305","reported_seq":2829,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.798642+0000","last_change":"2024-09-16T05:19:26.795358+0000","last_active":"2024-09-16T05:19:26.798642+0000","last_peered":"2024-09-16T05:19:26.798642+0000","last_clean":"2024-09-16T05:19:26.798642+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:19:26.798642+0000","last_undegraded":"2024-09-16T05:19:26.798642+0000","last_fullsized":"2024-09-16T05:19:26.798642+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'297","last_scrub_stamp":"2024-09-16T05:19:10.483443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.483443+0000","objects_scrubbed":2,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.000173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033236170000000001,"stat_sum":{"num_bytes":5496655,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":27814,"num_write":51,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.3","version":"620'196","reported_seq":2750,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.824181+0000","last_change":"2024-09-16T05:19:29.824411+0000","last_active":"2024-09-16T05:19:29.824181+0000","last_peered":"2024-09-16T05:19:29.824181+0000","last_clean":"2024-09-16T05:19:29.824181+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:19:29.824181+0000","last_undegraded":"2024-09-16T05:19:29.824181+0000","last_fullsized":"2024-09-16T05:19:29.824181+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"570'180","last_scrub_stamp":"2024-09-16T05:18:44.321479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:44.321479+0000","objects_scrubbed":3,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:11.015982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0031107259999999999,"stat_sum":{"num_bytes":5697870,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":77,"num_read_kb":91279,"num_write":38,"num_write_kb":16751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":498,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.781517+0000","last_change":"2024-09-16T05:18:26.212176+0000","last_active":"2024-09-16T05:19:26.781517+0000","last_peered":"2024-09-16T05:19:26.781517+0000","last_clean":"2024-09-16T05:19:26.781517+0000","last_became_active":"2024-09-16T05:18:19.768392+0000","last_became_peered":"2024-09-16T05:18:19.768392+0000","last_unstale":"2024-09-16T05:19:26.781517+0000","last_undegraded":"2024-09-16T05:19:26.781517+0000","last_fullsized":"2024-09-16T05:19:26.781517+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:26.212094+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:26.212094+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:38.132735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":2544,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.824897+0000","last_change":"2024-09-16T05:19:29.825289+0000","last_active":"2024-09-16T05:19:29.824897+0000","last_peered":"2024-09-16T05:19:29.824897+0000","last_clean":"2024-09-16T05:19:29.824897+0000","last_became_active":"2024-09-16T05:14:25.821988+0000","last_became_peered":"2024-09-16T05:14:25.821988+0000","last_unstale":"2024-09-16T05:19:29.824897+0000","last_undegraded":"2024-09-16T05:19:29.824897+0000","last_fullsized":"2024-09-16T05:19:29.824897+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:18:23.222127+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:18:23.222127+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:46.357250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00336477,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":1063,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.776218+0000","last_change":"2024-09-16T05:19:30.558195+0000","last_active":"2024-09-16T05:19:30.776218+0000","last_peered":"2024-09-16T05:19:30.776218+0000","last_clean":"2024-09-16T05:19:30.776218+0000","last_became_active":"2024-09-16T05:18:02.888512+0000","last_became_peered":"2024-09-16T05:18:02.888512+0000","last_unstale":"2024-09-16T05:19:30.776218+0000","last_undegraded":"2024-09-16T05:19:30.776218+0000","last_fullsized":"2024-09-16T05:19:30.776218+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.558069+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.558069+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.688000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1478,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792149+0000","last_change":"2024-09-16T05:18:29.435915+0000","last_active":"2024-09-16T05:19:26.792149+0000","last_peered":"2024-09-16T05:19:26.792149+0000","last_clean":"2024-09-16T05:19:26.792149+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:19:26.792149+0000","last_undegraded":"2024-09-16T05:19:26.792149+0000","last_fullsized":"2024-09-16T05:19:26.792149+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:29.435789+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:29.435789+0000","last_clean_scrub_stamp":"2024-09-16T05:18:29.435789+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:40.106447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":2474,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.801989+0000","last_change":"2024-09-16T05:19:26.802230+0000","last_active":"2024-09-16T05:19:26.801989+0000","last_peered":"2024-09-16T05:19:26.801989+0000","last_clean":"2024-09-16T05:19:26.801989+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:19:26.801989+0000","last_undegraded":"2024-09-16T05:19:26.801989+0000","last_fullsized":"2024-09-16T05:19:26.801989+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:18:54.327254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:54.327254+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:00.723077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012316269999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.3","version":"35'139","reported_seq":1491,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814251+0000","last_change":"2024-09-16T05:19:10.336320+0000","last_active":"2024-09-16T05:19:27.814251+0000","last_peered":"2024-09-16T05:19:27.814251+0000","last_clean":"2024-09-16T05:19:27.814251+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:19:27.814251+0000","last_undegraded":"2024-09-16T05:19:27.814251+0000","last_fullsized":"2024-09-16T05:19:27.814251+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:10.336203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.336203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.544788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":2554,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.795519+0000","last_change":"2024-09-16T05:19:26.795810+0000","last_active":"2024-09-16T05:19:26.795519+0000","last_peered":"2024-09-16T05:19:26.795519+0000","last_clean":"2024-09-16T05:19:26.795519+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:19:26.795519+0000","last_undegraded":"2024-09-16T05:19:26.795519+0000","last_fullsized":"2024-09-16T05:19:26.795519+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:18:26.463618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:26.463618+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:38.937207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033311,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":1783,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.839146+0000","last_change":"2024-09-16T05:19:29.839449+0000","last_active":"2024-09-16T05:19:29.839146+0000","last_peered":"2024-09-16T05:19:29.839146+0000","last_clean":"2024-09-16T05:19:29.839146+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:19:29.839146+0000","last_undegraded":"2024-09-16T05:19:29.839146+0000","last_fullsized":"2024-09-16T05:19:29.839146+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:18:26.961312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:26.961312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:40.138009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.01280092,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.4","version":"35'139","reported_seq":1443,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.782151+0000","last_change":"2024-09-16T05:18:28.258901+0000","last_active":"2024-09-16T05:19:26.782151+0000","last_peered":"2024-09-16T05:19:26.782151+0000","last_clean":"2024-09-16T05:19:26.782151+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:19:26.782151+0000","last_undegraded":"2024-09-16T05:19:26.782151+0000","last_fullsized":"2024-09-16T05:19:26.782151+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:28.258800+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:28.258800+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:30.389977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":948,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.802296+0000","last_change":"2024-09-16T05:19:26.802631+0000","last_active":"2024-09-16T05:19:26.802296+0000","last_peered":"2024-09-16T05:19:26.802296+0000","last_clean":"2024-09-16T05:19:26.802296+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:19:26.802296+0000","last_undegraded":"2024-09-16T05:19:26.802296+0000","last_fullsized":"2024-09-16T05:19:26.802296+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:18:27.279121+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:18:27.279121+0000","last_clean_scrub_stamp":"2024-09-16T05:18:27.279121+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:50.013181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015937900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":768,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.776424+0000","last_change":"2024-09-16T05:19:13.533926+0000","last_active":"2024-09-16T05:19:30.776424+0000","last_peered":"2024-09-16T05:19:30.776424+0000","last_clean":"2024-09-16T05:19:30.776424+0000","last_became_active":"2024-09-16T05:18:02.897275+0000","last_became_peered":"2024-09-16T05:18:02.897275+0000","last_unstale":"2024-09-16T05:19:30.776424+0000","last_undegraded":"2024-09-16T05:19:30.776424+0000","last_fullsized":"2024-09-16T05:19:30.776424+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:19:13.533798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:23.927688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"619'555","reported_seq":3092,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.822204+0000","last_change":"2024-09-16T05:19:28.822459+0000","last_active":"2024-09-16T05:19:28.822204+0000","last_peered":"2024-09-16T05:19:28.822204+0000","last_clean":"2024-09-16T05:19:28.822204+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:19:28.822204+0000","last_undegraded":"2024-09-16T05:19:28.822204+0000","last_fullsized":"2024-09-16T05:19:28.822204+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'526","last_scrub_stamp":"2024-09-16T05:18:49.197415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:49.197415+0000","objects_scrubbed":3,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:50.082931+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.00275699,"stat_sum":{"num_bytes":5128831,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":89,"num_read_kb":98598,"num_write":33,"num_write_kb":8652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":704,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.776346+0000","last_change":"2024-09-16T05:18:09.725282+0000","last_active":"2024-09-16T05:19:30.776346+0000","last_peered":"2024-09-16T05:19:30.776346+0000","last_clean":"2024-09-16T05:19:30.776346+0000","last_became_active":"2024-09-16T05:18:02.886933+0000","last_became_peered":"2024-09-16T05:18:02.886933+0000","last_unstale":"2024-09-16T05:19:30.776346+0000","last_undegraded":"2024-09-16T05:19:30.776346+0000","last_fullsized":"2024-09-16T05:19:30.776346+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:09.725207+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:09.725207+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:31.520385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":2624,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.831146+0000","last_change":"2024-09-16T05:19:29.831494+0000","last_active":"2024-09-16T05:19:29.831146+0000","last_peered":"2024-09-16T05:19:29.831146+0000","last_clean":"2024-09-16T05:19:29.831146+0000","last_became_active":"2024-09-16T05:14:25.821186+0000","last_became_peered":"2024-09-16T05:14:25.821186+0000","last_unstale":"2024-09-16T05:19:29.831146+0000","last_undegraded":"2024-09-16T05:19:29.831146+0000","last_fullsized":"2024-09-16T05:19:29.831146+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:18:44.900790+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:18:44.900790+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:53.623656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020455109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":748,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.776452+0000","last_change":"2024-09-16T05:19:15.510805+0000","last_active":"2024-09-16T05:19:30.776452+0000","last_peered":"2024-09-16T05:19:30.776452+0000","last_clean":"2024-09-16T05:19:30.776452+0000","last_became_active":"2024-09-16T05:18:03.588064+0000","last_became_peered":"2024-09-16T05:18:03.588064+0000","last_unstale":"2024-09-16T05:19:30.776452+0000","last_undegraded":"2024-09-16T05:19:30.776452+0000","last_fullsized":"2024-09-16T05:19:30.776452+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:15.510717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.510717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:27.556487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":2709,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.822910+0000","last_change":"2024-09-16T05:19:28.823135+0000","last_active":"2024-09-16T05:19:28.822910+0000","last_peered":"2024-09-16T05:19:28.822910+0000","last_clean":"2024-09-16T05:19:28.822910+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:19:28.822910+0000","last_undegraded":"2024-09-16T05:19:28.822910+0000","last_fullsized":"2024-09-16T05:19:28.822910+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:18:09.082699+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:18:09.082699+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032531579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":1493,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.613080+0000","last_change":"2024-09-16T05:19:30.613103+0000","last_active":"2024-09-16T05:19:30.613080+0000","last_peered":"2024-09-16T05:19:30.613080+0000","last_clean":"2024-09-16T05:19:30.613080+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:19:30.613080+0000","last_undegraded":"2024-09-16T05:19:30.613080+0000","last_fullsized":"2024-09-16T05:19:30.613080+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.613024+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.613024+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.741797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"617'575","reported_seq":3125,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.789459+0000","last_change":"2024-09-16T05:19:26.789871+0000","last_active":"2024-09-16T05:19:26.789459+0000","last_peered":"2024-09-16T05:19:26.789459+0000","last_clean":"2024-09-16T05:19:26.789459+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:19:26.789459+0000","last_undegraded":"2024-09-16T05:19:26.789459+0000","last_fullsized":"2024-09-16T05:19:26.789459+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"574'545","last_scrub_stamp":"2024-09-16T05:18:47.380196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:47.380196+0000","objects_scrubbed":4,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:07.942956+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0070007170000000001,"stat_sum":{"num_bytes":8084662,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":113160,"num_write":47,"num_write_kb":31911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.9","version":"35'139","reported_seq":1471,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792014+0000","last_change":"2024-09-16T05:19:19.900629+0000","last_active":"2024-09-16T05:19:26.792014+0000","last_peered":"2024-09-16T05:19:26.792014+0000","last_clean":"2024-09-16T05:19:26.792014+0000","last_became_active":"2024-09-16T05:19:19.900294+0000","last_became_peered":"2024-09-16T05:19:19.900294+0000","last_unstale":"2024-09-16T05:19:26.792014+0000","last_undegraded":"2024-09-16T05:19:26.792014+0000","last_fullsized":"2024-09-16T05:19:26.792014+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:23.110100+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:23.110100+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:40.719939+0000","scrub_duration":633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2761,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.836283+0000","last_change":"2024-09-16T05:19:29.836493+0000","last_active":"2024-09-16T05:19:29.836283+0000","last_peered":"2024-09-16T05:19:29.836283+0000","last_clean":"2024-09-16T05:19:29.836283+0000","last_became_active":"2024-09-16T05:15:16.707051+0000","last_became_peered":"2024-09-16T05:15:16.707051+0000","last_unstale":"2024-09-16T05:19:29.836283+0000","last_undegraded":"2024-09-16T05:19:29.836283+0000","last_fullsized":"2024-09-16T05:19:29.836283+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:18:09.660812+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:18:09.660812+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082003930000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":1481,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.825574+0000","last_change":"2024-09-16T05:17:47.986778+0000","last_active":"2024-09-16T05:19:29.825574+0000","last_peered":"2024-09-16T05:19:29.825574+0000","last_clean":"2024-09-16T05:19:29.825574+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:19:29.825574+0000","last_undegraded":"2024-09-16T05:19:29.825574+0000","last_fullsized":"2024-09-16T05:19:29.825574+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:17:47.986643+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:31.004288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":971,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.833425+0000","last_change":"2024-09-16T05:18:26.950493+0000","last_active":"2024-09-16T05:19:29.833425+0000","last_peered":"2024-09-16T05:19:29.833425+0000","last_clean":"2024-09-16T05:19:29.833425+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:19:29.833425+0000","last_undegraded":"2024-09-16T05:19:29.833425+0000","last_fullsized":"2024-09-16T05:19:29.833425+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:18:26.950417+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:44.447670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":463,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.777176+0000","last_change":"2024-09-16T05:19:30.777398+0000","last_active":"2024-09-16T05:19:30.777176+0000","last_peered":"2024-09-16T05:19:30.777176+0000","last_clean":"2024-09-16T05:19:30.777176+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:19:30.777176+0000","last_undegraded":"2024-09-16T05:19:30.777176+0000","last_fullsized":"2024-09-16T05:19:30.777176+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:19:07.382518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.382518+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:20.655870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011069980000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1485,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.820859+0000","last_change":"2024-09-16T05:17:28.243643+0000","last_active":"2024-09-16T05:19:29.820859+0000","last_peered":"2024-09-16T05:19:29.820859+0000","last_clean":"2024-09-16T05:19:29.820859+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:19:29.820859+0000","last_undegraded":"2024-09-16T05:19:29.820859+0000","last_fullsized":"2024-09-16T05:19:29.820859+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:17:28.243528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:17:28.243528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":2165,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.828495+0000","last_change":"2024-09-16T05:19:29.828887+0000","last_active":"2024-09-16T05:19:29.828495+0000","last_peered":"2024-09-16T05:19:29.828495+0000","last_clean":"2024-09-16T05:19:29.828495+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:19:29.828495+0000","last_undegraded":"2024-09-16T05:19:29.828495+0000","last_fullsized":"2024-09-16T05:19:29.828495+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:18:06.988836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:06.988836+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030295909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.14","version":"433'289","reported_seq":2445,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.829371+0000","last_change":"2024-09-16T05:19:29.829694+0000","last_active":"2024-09-16T05:19:29.829371+0000","last_peered":"2024-09-16T05:19:29.829371+0000","last_clean":"2024-09-16T05:19:29.829371+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:19:29.829371+0000","last_undegraded":"2024-09-16T05:19:29.829371+0000","last_fullsized":"2024-09-16T05:19:29.829371+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:19:12.999550+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.999550+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:13.255718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036790870000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.15","version":"620'603","reported_seq":3177,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.834199+0000","last_change":"2024-09-16T05:19:29.834586+0000","last_active":"2024-09-16T05:19:29.834199+0000","last_peered":"2024-09-16T05:19:29.834199+0000","last_clean":"2024-09-16T05:19:29.834199+0000","last_became_active":"2024-09-16T05:16:25.732256+0000","last_became_peered":"2024-09-16T05:16:25.732256+0000","last_unstale":"2024-09-16T05:19:29.834199+0000","last_undegraded":"2024-09-16T05:19:29.834199+0000","last_fullsized":"2024-09-16T05:19:29.834199+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"617'599","last_scrub_stamp":"2024-09-16T05:19:28.020270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.020270+0000","objects_scrubbed":4,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:28.047283+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0088440069999999992,"stat_sum":{"num_bytes":9473226,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":51,"num_read_kb":47656,"num_write":20,"num_write_kb":13595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.16","version":"620'267","reported_seq":2450,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.827667+0000","last_change":"2024-09-16T05:19:29.828027+0000","last_active":"2024-09-16T05:19:29.827667+0000","last_peered":"2024-09-16T05:19:29.827667+0000","last_clean":"2024-09-16T05:19:29.827667+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:19:29.827667+0000","last_undegraded":"2024-09-16T05:19:29.827667+0000","last_fullsized":"2024-09-16T05:19:29.827667+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"509'201","last_scrub_stamp":"2024-09-16T05:18:22.258029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:22.258029+0000","objects_scrubbed":1,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:43.001660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0062901770000000001,"stat_sum":{"num_bytes":6102326,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":57069,"num_write":44,"num_write_kb":12941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"1.14","version":"35'139","reported_seq":808,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.800893+0000","last_change":"2024-09-16T05:18:46.335095+0000","last_active":"2024-09-16T05:19:26.800893+0000","last_peered":"2024-09-16T05:19:26.800893+0000","last_clean":"2024-09-16T05:19:26.800893+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:19:26.800893+0000","last_undegraded":"2024-09-16T05:19:26.800893+0000","last_fullsized":"2024-09-16T05:19:26.800893+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:46.335030+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.335030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:02.571329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":680,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.836916+0000","last_change":"2024-09-16T05:19:29.837137+0000","last_active":"2024-09-16T05:19:29.836916+0000","last_peered":"2024-09-16T05:19:29.836916+0000","last_clean":"2024-09-16T05:19:29.836916+0000","last_became_active":"2024-09-16T05:17:59.867069+0000","last_became_peered":"2024-09-16T05:17:59.867069+0000","last_unstale":"2024-09-16T05:19:29.836916+0000","last_undegraded":"2024-09-16T05:19:29.836916+0000","last_fullsized":"2024-09-16T05:19:29.836916+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:12.884960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.884960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:19.935786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065710109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.18","version":"181'57","reported_seq":2547,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.829801+0000","last_change":"2024-09-16T05:19:29.830175+0000","last_active":"2024-09-16T05:19:29.829801+0000","last_peered":"2024-09-16T05:19:29.829801+0000","last_clean":"2024-09-16T05:19:29.829801+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:19:29.829801+0000","last_undegraded":"2024-09-16T05:19:29.829801+0000","last_fullsized":"2024-09-16T05:19:29.829801+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:18:06.018399+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:18:06.018399+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040116630000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.19","version":"182'60","reported_seq":2146,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814119+0000","last_change":"2024-09-16T05:19:26.784394+0000","last_active":"2024-09-16T05:19:27.814119+0000","last_peered":"2024-09-16T05:19:27.814119+0000","last_clean":"2024-09-16T05:19:27.814119+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:19:27.814119+0000","last_undegraded":"2024-09-16T05:19:27.814119+0000","last_fullsized":"2024-09-16T05:19:27.814119+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:19:07.269103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.269103+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:12.081147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001136387,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1a","version":"619'874","reported_seq":3376,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.839893+0000","last_change":"2024-09-16T05:19:28.827292+0000","last_active":"2024-09-16T05:19:28.839893+0000","last_peered":"2024-09-16T05:19:28.839893+0000","last_clean":"2024-09-16T05:19:28.839893+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:19:28.839893+0000","last_undegraded":"2024-09-16T05:19:28.839893+0000","last_fullsized":"2024-09-16T05:19:28.839893+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"582'782","last_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:18:52.154075+0000","objects_scrubbed":10,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:55.888467+0000","scrub_duration":29,"objects_trimmed":1,"snaptrim_duration":0.0072678259999999998,"stat_sum":{"num_bytes":27132561,"num_objects":13,"num_object_clones":10,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":130,"num_read_kb":111565,"num_write":131,"num_write_kb":50098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1b","version":"617'557","reported_seq":3109,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.786264+0000","last_change":"2024-09-16T05:19:26.786713+0000","last_active":"2024-09-16T05:19:26.786264+0000","last_peered":"2024-09-16T05:19:26.786264+0000","last_clean":"2024-09-16T05:19:26.786264+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:19:26.786264+0000","last_undegraded":"2024-09-16T05:19:26.786264+0000","last_fullsized":"2024-09-16T05:19:26.786264+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"579'497","last_scrub_stamp":"2024-09-16T05:18:49.298238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:49.298238+0000","objects_scrubbed":8,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:15.558248+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0047604559999999997,"stat_sum":{"num_bytes":15302782,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":89,"num_read_kb":89718,"num_write":86,"num_write_kb":25738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1c","version":"620'444","reported_seq":2690,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.777244+0000","last_change":"2024-09-16T05:19:30.777594+0000","last_active":"2024-09-16T05:19:30.777244+0000","last_peered":"2024-09-16T05:19:30.777244+0000","last_clean":"2024-09-16T05:19:30.777244+0000","last_became_active":"2024-09-16T05:18:02.890972+0000","last_became_peered":"2024-09-16T05:18:02.890972+0000","last_unstale":"2024-09-16T05:19:30.777244+0000","last_undegraded":"2024-09-16T05:19:30.777244+0000","last_fullsized":"2024-09-16T05:19:30.777244+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"468'393","last_scrub_stamp":"2024-09-16T05:18:07.717683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:07.717683+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011691869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":18,"num_write_kb":10300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"2.1d","version":"621'1111","reported_seq":3898,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.786219+0000","last_change":"2024-09-16T05:19:30.786867+0000","last_active":"2024-09-16T05:19:30.786219+0000","last_peered":"2024-09-16T05:19:30.786219+0000","last_clean":"2024-09-16T05:19:30.786219+0000","last_became_active":"2024-09-16T05:18:03.891103+0000","last_became_peered":"2024-09-16T05:18:03.891103+0000","last_unstale":"2024-09-16T05:19:30.786219+0000","last_undegraded":"2024-09-16T05:19:30.786219+0000","last_fullsized":"2024-09-16T05:19:30.786219+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"535'926","last_scrub_stamp":"2024-09-16T05:18:08.738088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:08.738088+0000","objects_scrubbed":10,"log_size":1111,"log_dups_size":0,"ondisk_log_size":1111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:36.447969+0000","scrub_duration":3,"objects_trimmed":5,"snaptrim_duration":0.0099115959999999999,"stat_sum":{"num_bytes":15830857,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":226,"num_read_kb":172010,"num_write":218,"num_write_kb":71091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7155994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"2.1e","version":"617'430","reported_seq":2993,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814359+0000","last_change":"2024-09-16T05:19:26.786772+0000","last_active":"2024-09-16T05:19:27.814359+0000","last_peered":"2024-09-16T05:19:27.814359+0000","last_clean":"2024-09-16T05:19:27.814359+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:19:27.814359+0000","last_undegraded":"2024-09-16T05:19:27.814359+0000","last_fullsized":"2024-09-16T05:19:27.814359+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"602'413","last_scrub_stamp":"2024-09-16T05:19:15.386065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.386065+0000","objects_scrubbed":4,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:31.860686+0000","scrub_duration":4,"objects_trimmed":1,"snaptrim_duration":0.0021839889999999999,"stat_sum":{"num_bytes":7814510,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":67,"num_read_kb":71847,"num_write":21,"num_write_kb":11620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.1f","version":"284'221","reported_seq":2772,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.831872+0000","last_change":"2024-09-16T05:19:29.833578+0000","last_active":"2024-09-16T05:19:29.831872+0000","last_peered":"2024-09-16T05:19:29.831872+0000","last_clean":"2024-09-16T05:19:29.831872+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:19:29.831872+0000","last_undegraded":"2024-09-16T05:19:29.831872+0000","last_fullsized":"2024-09-16T05:19:29.831872+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:19:09.544229+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:19:09.544229+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:22.679183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019421589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.20","version":"617'259","reported_seq":2811,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.804018+0000","last_change":"2024-09-16T05:19:26.804374+0000","last_active":"2024-09-16T05:19:26.804018+0000","last_peered":"2024-09-16T05:19:26.804018+0000","last_clean":"2024-09-16T05:19:26.804018+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:19:26.804018+0000","last_undegraded":"2024-09-16T05:19:26.804018+0000","last_fullsized":"2024-09-16T05:19:26.804018+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"574'228","last_scrub_stamp":"2024-09-16T05:18:44.488210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:44.488210+0000","objects_scrubbed":3,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:12.803837+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.01122269,"stat_sum":{"num_bytes":4713311,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":136173,"num_write":48,"num_write_kb":18826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.21","version":"178'50","reported_seq":2473,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.802686+0000","last_change":"2024-09-16T05:19:26.803146+0000","last_active":"2024-09-16T05:19:26.802686+0000","last_peered":"2024-09-16T05:19:26.802686+0000","last_clean":"2024-09-16T05:19:26.802686+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:19:26.802686+0000","last_undegraded":"2024-09-16T05:19:26.802686+0000","last_fullsized":"2024-09-16T05:19:26.802686+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:18:55.296457+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:18:55.296457+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:56.279635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00192373,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.22","version":"182'20","reported_seq":463,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-16T05:19:30.778162+0000","last_change":"2024-09-16T05:19:30.778449+0000","last_active":"2024-09-16T05:19:30.778162+0000","last_peered":"2024-09-16T05:19:30.778162+0000","last_clean":"2024-09-16T05:19:30.778162+0000","last_became_active":"2024-09-16T05:17:59.866047+0000","last_became_peered":"2024-09-16T05:17:59.866047+0000","last_unstale":"2024-09-16T05:19:30.778162+0000","last_undegraded":"2024-09-16T05:19:30.778162+0000","last_fullsized":"2024-09-16T05:19:30.778162+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:18:53.530513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:53.530513+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:15.558278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001783714,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ca","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":2013,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.814371+0000","last_change":"2024-09-16T05:19:26.783778+0000","last_active":"2024-09-16T05:19:27.814371+0000","last_peered":"2024-09-16T05:19:27.814371+0000","last_clean":"2024-09-16T05:19:27.814371+0000","last_became_active":"2024-09-16T05:12:23.819282+0000","last_became_peered":"2024-09-16T05:12:23.819282+0000","last_unstale":"2024-09-16T05:19:27.814371+0000","last_undegraded":"2024-09-16T05:19:27.814371+0000","last_fullsized":"2024-09-16T05:19:27.814371+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:18:48.291334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:48.291334+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:59.071253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054042200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.24","version":"182'89","reported_seq":1840,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.827324+0000","last_change":"2024-09-16T05:19:29.828419+0000","last_active":"2024-09-16T05:19:29.827324+0000","last_peered":"2024-09-16T05:19:29.827324+0000","last_clean":"2024-09-16T05:19:29.827324+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:19:29.827324+0000","last_undegraded":"2024-09-16T05:19:29.827324+0000","last_fullsized":"2024-09-16T05:19:29.827324+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:18:43.800415+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:18:43.800415+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:07.699680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025813640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.25","version":"619'956","reported_seq":3415,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.831419+0000","last_change":"2024-09-16T05:19:28.831635+0000","last_active":"2024-09-16T05:19:28.831419+0000","last_peered":"2024-09-16T05:19:28.831419+0000","last_clean":"2024-09-16T05:19:28.831419+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:19:28.831419+0000","last_undegraded":"2024-09-16T05:19:28.831419+0000","last_fullsized":"2024-09-16T05:19:28.831419+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"554'782","last_scrub_stamp":"2024-09-16T05:18:25.127285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:25.127285+0000","objects_scrubbed":7,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:28.930168+0000","scrub_duration":2,"objects_trimmed":3,"snaptrim_duration":0.011469935000000001,"stat_sum":{"num_bytes":27887788,"num_objects":11,"num_object_clones":8,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":138,"num_read_kb":147474,"num_write":126,"num_write_kb":54777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.26","version":"116'9","reported_seq":1818,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.801327+0000","last_change":"2024-09-16T05:19:26.801854+0000","last_active":"2024-09-16T05:19:26.801327+0000","last_peered":"2024-09-16T05:19:26.801327+0000","last_clean":"2024-09-16T05:19:26.801327+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:19:26.801327+0000","last_undegraded":"2024-09-16T05:19:26.801327+0000","last_fullsized":"2024-09-16T05:19:26.801327+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:07.483055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.483055+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:18.790615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088676199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.27","version":"0'0","reported_seq":1830,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.795703+0000","last_change":"2024-09-16T05:19:26.796052+0000","last_active":"2024-09-16T05:19:26.795703+0000","last_peered":"2024-09-16T05:19:26.795703+0000","last_clean":"2024-09-16T05:19:26.795703+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:19:26.795703+0000","last_undegraded":"2024-09-16T05:19:26.795703+0000","last_fullsized":"2024-09-16T05:19:26.795703+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:11.499561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.499561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:14.020884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035743770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.28","version":"245'105","reported_seq":841,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:27.303133+0000","last_change":"2024-09-16T05:19:27.303160+0000","last_active":"2024-09-16T05:19:27.303133+0000","last_peered":"2024-09-16T05:19:27.303133+0000","last_clean":"2024-09-16T05:19:27.303133+0000","last_became_active":"2024-09-16T05:19:22.783233+0000","last_became_peered":"2024-09-16T05:19:22.783233+0000","last_unstale":"2024-09-16T05:19:27.303133+0000","last_undegraded":"2024-09-16T05:19:27.303133+0000","last_fullsized":"2024-09-16T05:19:27.303133+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:19:27.303070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:27.303070+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.887894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010796600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.29","version":"617'832","reported_seq":3396,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.822515+0000","last_change":"2024-09-16T05:19:26.822756+0000","last_active":"2024-09-16T05:19:26.822515+0000","last_peered":"2024-09-16T05:19:26.822515+0000","last_clean":"2024-09-16T05:19:26.822515+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:19:26.822515+0000","last_undegraded":"2024-09-16T05:19:26.822515+0000","last_fullsized":"2024-09-16T05:19:26.822515+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"554'720","last_scrub_stamp":"2024-09-16T05:18:25.489701+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:18:25.489701+0000","objects_scrubbed":5,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:33.047953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.041415041999999999,"stat_sum":{"num_bytes":15301385,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":208,"num_read_kb":205445,"num_write":148,"num_write_kb":57937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2a","version":"245'325","reported_seq":2812,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.792365+0000","last_change":"2024-09-16T05:19:26.792734+0000","last_active":"2024-09-16T05:19:26.792365+0000","last_peered":"2024-09-16T05:19:26.792365+0000","last_clean":"2024-09-16T05:19:26.792365+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:19:26.792365+0000","last_undegraded":"2024-09-16T05:19:26.792365+0000","last_fullsized":"2024-09-16T05:19:26.792365+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:18:08.448119+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:18:08.448119+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000643612,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2b","version":"617'314","reported_seq":2766,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.808927+0000","last_change":"2024-09-16T05:19:26.809319+0000","last_active":"2024-09-16T05:19:26.808927+0000","last_peered":"2024-09-16T05:19:26.808927+0000","last_clean":"2024-09-16T05:19:26.808927+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:19:26.808927+0000","last_undegraded":"2024-09-16T05:19:26.808927+0000","last_fullsized":"2024-09-16T05:19:26.808927+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"578'284","last_scrub_stamp":"2024-09-16T05:18:50.293732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:50.293732+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:11.182825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0079029979999999996,"stat_sum":{"num_bytes":10454614,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":79889,"num_write":47,"num_write_kb":21198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2c","version":"241'194","reported_seq":2656,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-16T05:19:28.823247+0000","last_change":"2024-09-16T05:19:28.823519+0000","last_active":"2024-09-16T05:19:28.823247+0000","last_peered":"2024-09-16T05:19:28.823247+0000","last_clean":"2024-09-16T05:19:28.823247+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:19:28.823247+0000","last_undegraded":"2024-09-16T05:19:28.823247+0000","last_fullsized":"2024-09-16T05:19:28.823247+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:18:23.169260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:23.169260+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:35.469119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0036232899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2d","version":"617'438","reported_seq":2960,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.806863+0000","last_change":"2024-09-16T05:19:26.809043+0000","last_active":"2024-09-16T05:19:26.806863+0000","last_peered":"2024-09-16T05:19:26.806863+0000","last_clean":"2024-09-16T05:19:26.806863+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:19:26.806863+0000","last_undegraded":"2024-09-16T05:19:26.806863+0000","last_fullsized":"2024-09-16T05:19:26.806863+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"579'423","last_scrub_stamp":"2024-09-16T05:18:51.529954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:51.529954+0000","objects_scrubbed":3,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:19:57.944044+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.015797683999999999,"stat_sum":{"num_bytes":7037966,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":86,"num_read_kb":88579,"num_write":45,"num_write_kb":19868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2e","version":"617'552","reported_seq":3032,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-16T05:19:26.803162+0000","last_change":"2024-09-16T05:19:26.803555+0000","last_active":"2024-09-16T05:19:26.803162+0000","last_peered":"2024-09-16T05:19:26.803162+0000","last_clean":"2024-09-16T05:19:26.803162+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:19:26.803162+0000","last_undegraded":"2024-09-16T05:19:26.803162+0000","last_fullsized":"2024-09-16T05:19:26.803162+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"519'444","last_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:17:49.581405+0000","objects_scrubbed":3,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":1,"snaptrim_duration":0.010798466,"stat_sum":{"num_bytes":13653493,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":63,"num_read_kb":42111,"num_write":76,"num_write_kb":26273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c3","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]},{"pgid":"2.2f","version":"620'520","reported_seq":3104,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-09-16T05:19:29.838502+0000","last_change":"2024-09-16T05:19:29.839086+0000","last_active":"2024-09-16T05:19:29.838502+0000","last_peered":"2024-09-16T05:19:29.838502+0000","last_clean":"2024-09-16T05:19:29.838502+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:19:29.838502+0000","last_undegraded":"2024-09-16T05:19:29.838502+0000","last_fullsized":"2024-09-16T05:19:29.838502+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"579'461","last_scrub_stamp":"2024-09-16T05:18:50.885995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:50.885995+0000","objects_scrubbed":4,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:17.456758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012580183,"stat_sum":{"num_bytes":11330733,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":99,"num_read_kb":72250,"num_write":90,"num_write_kb":39298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c3","length":"1"},{"start":"e0","length":"1"},{"start":"e5","length":"1"},{"start":"ec","length":"1"},{"start":"f3","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":313153816,"num_objects":149,"num_object_clones":101,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":13,"num_read":2754,"num_read_kb":2727707,"num_write":1993,"num_write_kb":817351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":26862704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":414867456,"data_stored":536433676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22807,"ondisk_log_size":22807,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089469,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99512,"kb_used_data":68324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93645640,"statfs":{"total":95995035648,"available":95893135360,"internally_reserved":0,"allocated":69963776,"data_stored":82471833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285698,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59400,"kb_used_data":32364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93685752,"statfs":{"total":95995035648,"available":95934210048,"internally_reserved":0,"allocated":33140736,"data_stored":38533783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514635,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98528,"kb_used_data":45132,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93646624,"statfs":{"total":95995035648,"available":95894142976,"internally_reserved":0,"allocated":46215168,"data_stored":56509237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841674,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99448,"kb_used_data":68252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93645704,"statfs":{"total":95995035648,"available":95893200896,"internally_reserved":0,"allocated":69890048,"data_stored":88873890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103312,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113388,"kb_used_data":59980,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93631764,"statfs":{"total":95995035648,"available":95878926336,"internally_reserved":0,"allocated":61419520,"data_stored":78355289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397655,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81072,"kb_used_data":49880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93664080,"statfs":{"total":95995035648,"available":95912017920,"internally_reserved":0,"allocated":51077120,"data_stored":68948306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757402,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73924,"kb_used_data":34412,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93671228,"statfs":{"total":95995035648,"available":95919337472,"internally_reserved":0,"allocated":35237888,"data_stored":38226029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920928,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47676,"kb_used_data":20648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93697476,"statfs":{"total":95995035648,"available":95946215424,"internally_reserved":0,"allocated":21143552,"data_stored":22451570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117156,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88128,"kb_used_data":56940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93657024,"statfs":{"total":95995035648,"available":95904792576,"internally_reserved":0,"allocated":58306560,"data_stored":67938841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240443,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57116,"kb_used_data":20680,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93688036,"statfs":{"total":95995035648,"available":95936548864,"internally_reserved":0,"allocated":21176320,"data_stored":20428560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63080,"kb_used_data":31892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682072,"statfs":{"total":95995035648,"available":95930441728,"internally_reserved":0,"allocated":32657408,"data_stored":35049126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738548,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44984,"kb_used_data":17956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93700168,"statfs":{"total":95995035648,"available":95948972032,"internally_reserved":0,"allocated":18386944,"data_stored":20599858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10579968,"data_stored":15330395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25178112,"data_stored":30107343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13467648,"data_stored":15249449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50536448,"data_stored":62694738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12562432,"data_stored":14875448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27553792,"data_stored":33082998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":43634688,"data_stored":64031883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":52264960,"data_stored":73985279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":58753024,"data_stored":74558327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":38735872,"data_stored":51567454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":23394952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":62521344,"data_stored":77555410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:19:32.691 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T05:19:32.691 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:19:32.692 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-09-16T05:19:32.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-16T05:19:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:32.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:32.782 INFO:tasks.rados.rados.2.smithi204.stdout:2389: done (3 left) 2024-09-16T05:19:32.783 INFO:tasks.rados.rados.2.smithi204.stdout:2390: snap_remove snap 243 2024-09-16T05:19:32.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:32.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:32.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:32.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: from='client.16554 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:19:32.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:19:32.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.29 scrub ok 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.53 scrub starts 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.53 scrub ok 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.25 deep-scrub starts 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.25 deep-scrub ok 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:32.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:19:32.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:19:32.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:32 smithi032 ceph-mon[27388]: osdmap e623: 12 total, 12 up, 12 in 2024-09-16T05:19:32.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:32.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:32.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:19:33.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:33.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:33.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:33.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: from='client.16554 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.29 scrub ok 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.53 scrub starts 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.53 scrub ok 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.25 deep-scrub starts 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.25 deep-scrub ok 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:19:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:32 smithi204 ceph-mon[30933]: osdmap e623: 12 total, 12 up, 12 in 2024-09-16T05:19:33.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:33.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:33.188 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:33.207 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: from='client.16554 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.29 scrub ok 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.53 scrub starts 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.53 scrub ok 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.25 deep-scrub starts 2024-09-16T05:19:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.25 deep-scrub ok 2024-09-16T05:19:33.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:33.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:33.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:19:33.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:19:33.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:32 smithi196 ceph-mon[30802]: osdmap e623: 12 total, 12 up, 12 in 2024-09-16T05:19:33.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:33.301 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:33.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:33.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:33.424 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:33.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:33.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:33.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:33.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:33.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:33.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:33.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:33.695 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:33.695 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:33.782 INFO:tasks.rados.rados.2.smithi204.stdout:2390: done (3 left) 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2391: delete oid 38 current snap is 248 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2387: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 760 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2386: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2386: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2386: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 778 (ObjNum 782 snap 227 seq_num 782) dirty exists 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2388: finishing write tid 1 to smithi20457400-13 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2388: finishing write tid 2 to smithi20457400-13 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2388: finishing write tid 3 to smithi20457400-13 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:2388: finishing write tid 4 to smithi20457400-13 2024-09-16T05:19:33.783 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 1008 (ObjNum 855 snap 248 seq_num 855) dirty exists 2024-09-16T05:19:33.784 INFO:tasks.rados.rados.2.smithi204.stdout:2388: left oid 13 (ObjNum 855 snap 248 seq_num 855) 2024-09-16T05:19:33.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:33.784 INFO:tasks.rados.rados.2.smithi204.stdout:2386: done (3 left) 2024-09-16T05:19:33.785 INFO:tasks.rados.rados.2.smithi204.stdout:2387: done (2 left) 2024-09-16T05:19:33.785 INFO:tasks.rados.rados.2.smithi204.stdout:2388: done (1 left) 2024-09-16T05:19:33.785 INFO:tasks.rados.rados.2.smithi204.stdout:2391: done (0 left) 2024-09-16T05:19:33.785 INFO:tasks.rados.rados.2.smithi204.stdout:2392: delete oid 18 current snap is 248 2024-09-16T05:19:33.785 INFO:tasks.rados.rados.2.smithi204.stdout:2392: done (0 left) 2024-09-16T05:19:33.785 INFO:tasks.rados.rados.2.smithi204.stdout:2393: write (excl) oid 20 current snap is 248 2024-09-16T05:19:33.785 INFO:tasks.rados.rados.2.smithi204.stdout:2393: seq_num 858 ranges {420406=795178,1974962=445444} 2024-09-16T05:19:33.793 INFO:tasks.rados.rados.2.smithi204.stdout:2393: writing smithi20457400-20 from 420406 to 1215584 tid 1 2024-09-16T05:19:33.801 INFO:tasks.rados.rados.2.smithi204.stdout:2393: writing smithi20457400-20 from 1974962 to 2420406 tid 2 2024-09-16T05:19:33.801 INFO:tasks.rados.rados.2.smithi204.stdout:2394: copy_from oid 49 from oid 46 current snap is 248 2024-09-16T05:19:33.801 INFO:tasks.rados.rados.2.smithi204.stdout:2395: delete oid 39 current snap is 248 2024-09-16T05:19:33.803 INFO:tasks.rados.rados.2.smithi204.stdout:2393: finishing write tid 1 to smithi20457400-20 2024-09-16T05:19:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:2393: finishing write tid 2 to smithi20457400-20 2024-09-16T05:19:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:2393: finishing write tid 3 to smithi20457400-20 2024-09-16T05:19:33.804 INFO:tasks.rados.rados.2.smithi204.stdout:2393: finishing write tid 4 to smithi20457400-20 2024-09-16T05:19:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 985 (ObjNum 858 snap 248 seq_num 858) dirty exists 2024-09-16T05:19:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:2393: left oid 20 (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:19:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:2393: done (2 left) 2024-09-16T05:19:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:2395: done (1 left) 2024-09-16T05:19:33.805 INFO:tasks.rados.rados.2.smithi204.stdout:2396: delete oid 23 current snap is 248 2024-09-16T05:19:33.808 INFO:tasks.rados.rados.2.smithi204.stdout:2396: done (1 left) 2024-09-16T05:19:33.808 INFO:tasks.rados.rados.2.smithi204.stdout:2397: read oid 36 snap -1 2024-09-16T05:19:33.808 INFO:tasks.rados.rados.2.smithi204.stdout:2397: expect (ObjNum 834 snap 242 seq_num 834) 2024-09-16T05:19:33.808 INFO:tasks.rados.rados.2.smithi204.stdout:2398: snap_create 2024-09-16T05:19:33.810 INFO:tasks.rados.rados.2.smithi204.stdout:2397: expect (ObjNum 834 snap 242 seq_num 834) 2024-09-16T05:19:33.815 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:33.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:33.831 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:33.852 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:33.872 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:33.874 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:33.893 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:33.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:33.914 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:33.915 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:33.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: pgmap v1095: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 283 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 256 KiB/s wr, 3 op/s 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: 1.b scrub ok 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: osdmap e624: 12 total, 12 up, 12 in 2024-09-16T05:19:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4015221059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:33.936 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:33.936 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:33.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:33.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:33.971 INFO:tasks.rados.rados.2.smithi204.stdout:2394: finishing copy_from racing read to smithi20457400-49 2024-09-16T05:19:33.972 INFO:tasks.rados.rados.2.smithi204.stdout:2394: finishing copy_from to smithi20457400-49 2024-09-16T05:19:33.972 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 362 (ObjNum 808 snap 237 seq_num 808) dirty exists 2024-09-16T05:19:33.974 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:33.974 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: pgmap v1095: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 283 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 256 KiB/s wr, 3 op/s 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: 1.b scrub ok 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: osdmap e624: 12 total, 12 up, 12 in 2024-09-16T05:19:33.975 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4015221059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:34.011 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:34.012 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:34.034 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:34.034 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:34.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:34.055 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:34.055 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:34.067 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:34.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:34.076 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:34.076 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:34.097 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:34.097 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:34.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:34.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:34.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: pgmap v1095: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 283 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 256 KiB/s wr, 3 op/s 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: 1.b scrub ok 2024-09-16T05:19:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: osdmap e624: 12 total, 12 up, 12 in 2024-09-16T05:19:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4015221059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:34.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:34.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:19:34.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:34.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:34.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:34.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:34.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:34.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:34.574 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:34.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:34.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:34 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:19:34.224+0000 7f7c3f992640 -1 mon.a@0(leader).osd e624 definitely_dead 0 2024-09-16T05:19:34.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:34.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:34.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:34.781 INFO:teuthology.orchestra.run.smithi032.stderr:marked out osd.1. 2024-09-16T05:19:34.788 INFO:tasks.rados.rados.2.smithi204.stdout:2394: done (2 left) 2024-09-16T05:19:34.788 INFO:tasks.rados.rados.2.smithi204.stdout:2397: done (1 left) 2024-09-16T05:19:34.788 INFO:tasks.rados.rados.2.smithi204.stdout:2398: done (0 left) 2024-09-16T05:19:34.788 INFO:tasks.rados.rados.2.smithi204.stdout:2399: write oid 11 current snap is 249 2024-09-16T05:19:34.789 INFO:tasks.rados.rados.2.smithi204.stdout:2399: seq_num 861 ranges {470827=737278,1750408=720419} 2024-09-16T05:19:34.795 INFO:tasks.rados.rados.2.smithi204.stdout:2399: writing smithi20457400-11 from 470827 to 1208105 tid 1 2024-09-16T05:19:34.802 INFO:tasks.rados.rados.2.smithi204.stdout:2399: writing smithi20457400-11 from 1750408 to 2470827 tid 2 2024-09-16T05:19:34.802 INFO:tasks.rados.rados.2.smithi204.stdout:2400: delete oid 4 current snap is 249 2024-09-16T05:19:34.802 INFO:tasks.rados.rados.2.smithi204.stdout:2399: finishing write tid 1 to smithi20457400-11 2024-09-16T05:19:34.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:34.803 INFO:tasks.rados.rados.2.smithi204.stdout:2400: done (1 left) 2024-09-16T05:19:34.803 INFO:tasks.rados.rados.2.smithi204.stdout:2401: read oid 25 snap -1 2024-09-16T05:19:34.804 INFO:tasks.rados.rados.2.smithi204.stdout:2401: expect (ObjNum 853 snap 248 seq_num 853) 2024-09-16T05:19:34.804 INFO:tasks.rados.rados.2.smithi204.stdout:2402: delete oid 29 current snap is 249 2024-09-16T05:19:34.809 INFO:tasks.rados.rados.2.smithi204.stdout:2399: finishing write tid 2 to smithi20457400-11 2024-09-16T05:19:34.809 INFO:tasks.rados.rados.2.smithi204.stdout:2399: finishing write tid 3 to smithi20457400-11 2024-09-16T05:19:34.809 INFO:tasks.rados.rados.2.smithi204.stdout:2399: finishing write tid 4 to smithi20457400-11 2024-09-16T05:19:34.809 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 991 (ObjNum 861 snap 249 seq_num 861) dirty exists 2024-09-16T05:19:34.809 INFO:tasks.rados.rados.2.smithi204.stdout:2399: left oid 11 (ObjNum 861 snap 249 seq_num 861) 2024-09-16T05:19:34.809 INFO:tasks.rados.rados.2.smithi204.stdout:2399: done (2 left) 2024-09-16T05:19:34.809 INFO:tasks.rados.rados.2.smithi204.stdout:2402: done (1 left) 2024-09-16T05:19:34.809 INFO:tasks.rados.rados.2.smithi204.stdout:2403: copy_from oid 37 from oid 33 current snap is 249 2024-09-16T05:19:34.810 INFO:tasks.rados.rados.2.smithi204.stdout:2404: read oid 35 snap 244 2024-09-16T05:19:34.810 INFO:tasks.rados.rados.2.smithi204.stdout:2404: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:19:34.810 INFO:tasks.rados.rados.2.smithi204.stdout:2405: write oid 29 current snap is 249 2024-09-16T05:19:34.810 INFO:tasks.rados.rados.2.smithi204.stdout:2405: seq_num 864 ranges {521248=563025,1809501=596300} 2024-09-16T05:19:34.812 INFO:tasks.rados.rados.2.smithi204.stdout:2405: writing smithi20457400-29 from 521248 to 1084273 tid 1 2024-09-16T05:19:34.819 INFO:tasks.rados.rados.2.smithi204.stdout:2405: writing smithi20457400-29 from 1809501 to 2405801 tid 2 2024-09-16T05:19:34.819 INFO:tasks.rados.rados.2.smithi204.stdout:2406: copy_from oid 50 from oid 30 current snap is 249 2024-09-16T05:19:34.819 INFO:tasks.rados.rados.2.smithi204.stdout:2403: finishing copy_from to smithi20457400-37 2024-09-16T05:19:34.819 INFO:tasks.rados.rados.2.smithi204.stdout:2403: got expected ENOENT (src dne) 2024-09-16T05:19:34.819 INFO:tasks.rados.rados.2.smithi204.stdout:2403: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:19:34.820 INFO:tasks.rados.rados.2.smithi204.stdout:2404: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:19:34.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:34.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:34.929 INFO:tasks.rados.rados.2.smithi204.stdout:2401: expect (ObjNum 853 snap 248 seq_num 853) 2024-09-16T05:19:34.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:34.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:34.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:35.035 INFO:tasks.rados.rados.2.smithi204.stdout:2405: finishing write tid 1 to smithi20457400-29 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2405: finishing write tid 2 to smithi20457400-29 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2405: finishing write tid 3 to smithi20457400-29 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2405: finishing write tid 4 to smithi20457400-29 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 893 (ObjNum 864 snap 249 seq_num 864) dirty exists 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2405: left oid 29 (ObjNum 864 snap 249 seq_num 864) 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2406: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2406: finishing copy_from to smithi20457400-50 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1093 (ObjNum 681 snap 198 seq_num 681) dirty exists 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2401: done (4 left) 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2403: done (3 left) 2024-09-16T05:19:35.036 INFO:tasks.rados.rados.2.smithi204.stdout:2404: done (2 left) 2024-09-16T05:19:35.037 INFO:tasks.rados.rados.2.smithi204.stdout:2405: done (1 left) 2024-09-16T05:19:35.037 INFO:tasks.rados.rados.2.smithi204.stdout:2406: done (0 left) 2024-09-16T05:19:35.037 INFO:tasks.rados.rados.2.smithi204.stdout:2407: snap_remove snap 235 2024-09-16T05:19:35.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:35.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:35.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:35.083 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:34 smithi032 ceph-mon[27388]: 2.4f scrub starts 2024-09-16T05:19:35.083 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:34 smithi032 ceph-mon[27388]: 2.4f scrub ok 2024-09-16T05:19:35.083 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/101436684' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-16T05:19:35.083 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:34 smithi032 ceph-mon[27388]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-16T05:19:35.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/101436684' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-16T05:19:35.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:34 smithi032 ceph-mon[27388]: osdmap e625: 12 total, 12 up, 11 in 2024-09-16T05:19:35.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:19:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:34 smithi204 ceph-mon[30933]: 2.4f scrub starts 2024-09-16T05:19:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:34 smithi204 ceph-mon[30933]: 2.4f scrub ok 2024-09-16T05:19:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/101436684' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-16T05:19:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:34 smithi204 ceph-mon[30933]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-16T05:19:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/101436684' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-16T05:19:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:34 smithi204 ceph-mon[30933]: osdmap e625: 12 total, 12 up, 11 in 2024-09-16T05:19:35.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:35.201 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:35.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:34 smithi196 ceph-mon[30802]: 2.4f scrub starts 2024-09-16T05:19:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:34 smithi196 ceph-mon[30802]: 2.4f scrub ok 2024-09-16T05:19:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/101436684' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-16T05:19:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:34 smithi196 ceph-mon[30802]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-16T05:19:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/101436684' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-16T05:19:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:34 smithi196 ceph-mon[30802]: osdmap e625: 12 total, 12 up, 11 in 2024-09-16T05:19:35.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:35.325 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:35.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:35.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:35.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:35.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:35.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:35.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:35.580 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:35.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:35.581 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:35.602 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:35.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:35.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:35.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:35.784 INFO:tasks.rados.rados.2.smithi204.stdout:2407: done (0 left) 2024-09-16T05:19:35.784 INFO:tasks.rados.rados.2.smithi204.stdout:2408: rollback oid 1 current snap is 249 2024-09-16T05:19:35.784 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 1 to 244 2024-09-16T05:19:35.784 INFO:tasks.rados.rados.2.smithi204.stdout:2409: rollback oid 25 current snap is 249 2024-09-16T05:19:35.784 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 238 2024-09-16T05:19:35.785 INFO:tasks.rados.rados.2.smithi204.stdout:2410: rollback oid 15 current snap is 249 2024-09-16T05:19:35.785 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 15 to 237 2024-09-16T05:19:35.785 INFO:tasks.rados.rados.2.smithi204.stdout:2411: snap_create 2024-09-16T05:19:35.790 INFO:tasks.rados.rados.2.smithi204.stdout:2409: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:19:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:2409: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:19:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:2409: finishing rollback tid 2 to smithi20457400-25 2024-09-16T05:19:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 723 (ObjNum 792 snap 230 seq_num 792) dirty exists 2024-09-16T05:19:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:2410: finishing rollback tid 0 to smithi20457400-15 2024-09-16T05:19:35.791 INFO:tasks.rados.rados.2.smithi204.stdout:2410: finishing rollback tid 1 to smithi20457400-15 2024-09-16T05:19:35.792 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 706 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T05:19:35.796 INFO:tasks.rados.rados.2.smithi204.stdout:2408: finishing rollback tid 0 to smithi20457400-1 2024-09-16T05:19:35.796 INFO:tasks.rados.rados.2.smithi204.stdout:2408: finishing rollback tid 1 to smithi20457400-1 2024-09-16T05:19:35.796 INFO:tasks.rados.rados.2.smithi204.stdout:2408: finishing rollback tid 2 to smithi20457400-1 2024-09-16T05:19:35.796 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 545 (ObjNum 842 snap 243 seq_num 842) dirty exists 2024-09-16T05:19:35.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:35.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:35.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:35.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:35.955 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:35.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:36.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:36.063 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:35 smithi032 ceph-mon[27388]: pgmap v1098: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 239 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 512 KiB/s wr, 5 op/s 2024-09-16T05:19:36.063 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:35 smithi032 ceph-mon[27388]: 2.3c scrub starts 2024-09-16T05:19:36.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:35 smithi032 ceph-mon[27388]: 2.3c scrub ok 2024-09-16T05:19:36.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:35 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:19:36.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:35 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:19:36.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:35 smithi032 ceph-mon[27388]: osdmap e626: 12 total, 12 up, 11 in 2024-09-16T05:19:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:36.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:36.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:36.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:35 smithi204 ceph-mon[30933]: pgmap v1098: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 239 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 512 KiB/s wr, 5 op/s 2024-09-16T05:19:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:35 smithi204 ceph-mon[30933]: 2.3c scrub starts 2024-09-16T05:19:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:35 smithi204 ceph-mon[30933]: 2.3c scrub ok 2024-09-16T05:19:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:35 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:19:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:35 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:19:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:35 smithi204 ceph-mon[30933]: osdmap e626: 12 total, 12 up, 11 in 2024-09-16T05:19:36.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:36.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:36.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:35 smithi196 ceph-mon[30802]: pgmap v1098: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 239 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 512 KiB/s wr, 5 op/s 2024-09-16T05:19:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:35 smithi196 ceph-mon[30802]: 2.3c scrub starts 2024-09-16T05:19:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:35 smithi196 ceph-mon[30802]: 2.3c scrub ok 2024-09-16T05:19:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:35 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:19:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:35 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:19:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:35 smithi196 ceph-mon[30802]: osdmap e626: 12 total, 12 up, 11 in 2024-09-16T05:19:36.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:36.328 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:36.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:36.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:36.454 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:36.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:36.517 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:19:36.517 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:19:36.517 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:19:36.517 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:19:36.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:36.581 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:36.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:36.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:36.706 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:36.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:19:36.782 INFO:tasks.rados.rados.2.smithi204.stdout:2408: done (3 left) 2024-09-16T05:19:36.782 INFO:tasks.rados.rados.2.smithi204.stdout:2409: done (2 left) 2024-09-16T05:19:36.782 INFO:tasks.rados.rados.2.smithi204.stdout:2410: done (1 left) 2024-09-16T05:19:36.782 INFO:tasks.rados.rados.2.smithi204.stdout:2411: done (0 left) 2024-09-16T05:19:36.783 INFO:tasks.rados.rados.2.smithi204.stdout:2412: read oid 23 snap -1 2024-09-16T05:19:36.783 INFO:tasks.rados.rados.2.smithi204.stdout:2412: expect deleted 2024-09-16T05:19:36.783 INFO:tasks.rados.rados.2.smithi204.stdout:2413: read oid 29 snap -1 2024-09-16T05:19:36.783 INFO:tasks.rados.rados.2.smithi204.stdout:2413: expect (ObjNum 864 snap 249 seq_num 864) 2024-09-16T05:19:36.783 INFO:tasks.rados.rados.2.smithi204.stdout:2414: snap_create 2024-09-16T05:19:36.786 INFO:tasks.rados.rados.2.smithi204.stdout:2413: expect (ObjNum 864 snap 249 seq_num 864) 2024-09-16T05:19:36.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:36.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:36.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:36.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:19:36.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:36.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:36.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:37.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:37.086 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:37.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:19:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:19:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: 2.3d scrub starts 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: 2.3d scrub ok 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1763212381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1763212381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:19:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:36 smithi032 ceph-mon[27388]: osdmap e627: 12 total, 12 up, 11 in 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: 2.3d scrub starts 2024-09-16T05:19:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: 2.3d scrub ok 2024-09-16T05:19:37.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1763212381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:19:37.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:37.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1763212381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:19:37.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:36 smithi204 ceph-mon[30933]: osdmap e627: 12 total, 12 up, 11 in 2024-09-16T05:19:37.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:37.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:37.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:19:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:19:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:19:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:19:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:19:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:19:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:19:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:19:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: 2.3d scrub starts 2024-09-16T05:19:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: 2.3d scrub ok 2024-09-16T05:19:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1763212381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:19:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1763212381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:19:37.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:36 smithi196 ceph-mon[30802]: osdmap e627: 12 total, 12 up, 11 in 2024-09-16T05:19:37.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:37.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:19:37.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:37.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:37.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:37.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:37.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:37.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:37.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:37.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:37.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:37.716 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:37.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:37.783 INFO:tasks.rados.rados.2.smithi204.stdout:2412: done (2 left) 2024-09-16T05:19:37.783 INFO:tasks.rados.rados.2.smithi204.stdout:2413: done (1 left) 2024-09-16T05:19:37.784 INFO:tasks.rados.rados.2.smithi204.stdout:2414: done (0 left) 2024-09-16T05:19:37.784 INFO:tasks.rados.rados.2.smithi204.stdout:2415: snap_create 2024-09-16T05:19:37.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:37.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:37.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:37.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:37.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:37.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:38.009 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:38.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:38.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:38.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:38.131 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:37 smithi032 ceph-mon[27388]: pgmap v1101: 107 pgs: 2 peering, 105 active+clean; 224 MiB data, 725 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-16T05:19:38.131 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:37 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:19:38.131 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:37 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:19:38.131 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:37 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:19:38.132 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:37 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:19:38.132 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:37 smithi032 ceph-mon[27388]: osdmap e628: 12 total, 12 up, 11 in 2024-09-16T05:19:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:37 smithi204 ceph-mon[30933]: pgmap v1101: 107 pgs: 2 peering, 105 active+clean; 224 MiB data, 725 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-16T05:19:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:37 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:19:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:37 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:19:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:37 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:19:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:37 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:19:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:37 smithi204 ceph-mon[30933]: osdmap e628: 12 total, 12 up, 11 in 2024-09-16T05:19:38.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:38.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:38.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:37 smithi196 ceph-mon[30802]: pgmap v1101: 107 pgs: 2 peering, 105 active+clean; 224 MiB data, 725 MiB used, 983 GiB / 983 GiB avail; 4.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s 2024-09-16T05:19:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:37 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:19:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:37 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:19:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:37 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:19:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:37 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:19:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:37 smithi196 ceph-mon[30802]: osdmap e628: 12 total, 12 up, 11 in 2024-09-16T05:19:38.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:38.362 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:38.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:38.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:38.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:38.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:38.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:38.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:38.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:38.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:38.785 INFO:tasks.rados.rados.2.smithi204.stdout:2415: done (0 left) 2024-09-16T05:19:38.785 INFO:tasks.rados.rados.2.smithi204.stdout:2416: snap_create 2024-09-16T05:19:38.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:38.867 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:38.867 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:38.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:38.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:38.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:38.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:38 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:19:38.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:38 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:19:38.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:38 smithi032 ceph-mon[27388]: 2.48 deep-scrub starts 2024-09-16T05:19:38.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:38 smithi032 ceph-mon[27388]: 2.48 deep-scrub ok 2024-09-16T05:19:38.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2111519089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:38.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:38 smithi032 ceph-mon[27388]: osdmap e629: 12 total, 12 up, 11 in 2024-09-16T05:19:38.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:38.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:38.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:39.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:39.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:39.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:38 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:19:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:38 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:19:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:38 smithi196 ceph-mon[30802]: 2.48 deep-scrub starts 2024-09-16T05:19:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:38 smithi196 ceph-mon[30802]: 2.48 deep-scrub ok 2024-09-16T05:19:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2111519089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:39.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:38 smithi196 ceph-mon[30802]: osdmap e629: 12 total, 12 up, 11 in 2024-09-16T05:19:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:38 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:19:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:38 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:19:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:38 smithi204 ceph-mon[30933]: 2.48 deep-scrub starts 2024-09-16T05:19:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:38 smithi204 ceph-mon[30933]: 2.48 deep-scrub ok 2024-09-16T05:19:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2111519089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:38 smithi204 ceph-mon[30933]: osdmap e629: 12 total, 12 up, 11 in 2024-09-16T05:19:39.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:39.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:39.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:39.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:39.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:39.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:39.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:39.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:39.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:39.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:19:39.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:39.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:39.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:39.694 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:39.694 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:39.694 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:39.694 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:39.695 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:39.695 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:39.696 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:39.697 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:39.697 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:39.697 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:39.697 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:39.731 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:39.755 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:39.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:39.787 INFO:tasks.rados.rados.2.smithi204.stdout:2416: done (0 left) 2024-09-16T05:19:39.787 INFO:tasks.rados.rados.2.smithi204.stdout:2417: read oid 34 snap -1 2024-09-16T05:19:39.787 INFO:tasks.rados.rados.2.smithi204.stdout:2417: expect (ObjNum 830 snap 242 seq_num 830) 2024-09-16T05:19:39.787 INFO:tasks.rados.rados.2.smithi204.stdout:2418: copy_from oid 17 from oid 28 current snap is 253 2024-09-16T05:19:39.788 INFO:tasks.rados.rados.2.smithi204.stdout:2419: delete oid 23 current snap is 253 2024-09-16T05:19:39.790 INFO:tasks.rados.rados.2.smithi204.stdout:2418: finishing copy_from to smithi20457400-17 2024-09-16T05:19:39.790 INFO:tasks.rados.rados.2.smithi204.stdout:2418: got expected ENOENT (src dne) 2024-09-16T05:19:39.790 INFO:tasks.rados.rados.2.smithi204.stdout:2419: done (2 left) 2024-09-16T05:19:39.790 INFO:tasks.rados.rados.2.smithi204.stdout:2420: rollback oid 33 current snap is 253 2024-09-16T05:19:39.790 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 33 to 240 2024-09-16T05:19:39.790 INFO:tasks.rados.rados.2.smithi204.stdout:2421: snap_create 2024-09-16T05:19:39.790 INFO:tasks.rados.rados.2.smithi204.stdout:2418: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:19:39.792 INFO:tasks.rados.rados.2.smithi204.stdout:2417: expect (ObjNum 830 snap 242 seq_num 830) 2024-09-16T05:19:39.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:39.879 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:39.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:39.922 INFO:tasks.rados.rados.2.smithi204.stdout:2420: finishing rollback tid 1 to smithi20457400-33 2024-09-16T05:19:39.922 INFO:tasks.rados.rados.2.smithi204.stdout:2420: finishing rollback tid 2 to smithi20457400-33 2024-09-16T05:19:39.922 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 844 (ObjNum 823 snap 240 seq_num 823) dirty exists 2024-09-16T05:19:39.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:19:39.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:40.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:40.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:40.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:39 smithi032 ceph-mon[27388]: pgmap v1104: 107 pgs: 2 peering, 1 active+recovering, 104 active+clean; 224 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 8.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s; 2/246 objects misplaced (0.813%); 1.9 MiB/s, 16 objects/s recovering 2024-09-16T05:19:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:39 smithi032 ceph-mon[27388]: osdmap e630: 12 total, 12 up, 11 in 2024-09-16T05:19:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:39 smithi204 ceph-mon[30933]: pgmap v1104: 107 pgs: 2 peering, 1 active+recovering, 104 active+clean; 224 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 8.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s; 2/246 objects misplaced (0.813%); 1.9 MiB/s, 16 objects/s recovering 2024-09-16T05:19:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:39 smithi204 ceph-mon[30933]: osdmap e630: 12 total, 12 up, 11 in 2024-09-16T05:19:40.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:39 smithi196 ceph-mon[30802]: pgmap v1104: 107 pgs: 2 peering, 1 active+recovering, 104 active+clean; 224 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 8.8 MiB/s rd, 2.5 MiB/s wr, 14 op/s; 2/246 objects misplaced (0.813%); 1.9 MiB/s, 16 objects/s recovering 2024-09-16T05:19:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:39 smithi196 ceph-mon[30802]: osdmap e630: 12 total, 12 up, 11 in 2024-09-16T05:19:40.251 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:40.323 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:40.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:40.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:40.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:40.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:40.504 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:19:40.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T05:19:40.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:40.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:40.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:40.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:40.748 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:40.782 INFO:tasks.rados.rados.2.smithi204.stdout:2417: done (3 left) 2024-09-16T05:19:40.782 INFO:tasks.rados.rados.2.smithi204.stdout:2418: done (2 left) 2024-09-16T05:19:40.782 INFO:tasks.rados.rados.2.smithi204.stdout:2420: done (1 left) 2024-09-16T05:19:40.783 INFO:tasks.rados.rados.2.smithi204.stdout:2421: done (0 left) 2024-09-16T05:19:40.783 INFO:tasks.rados.rados.2.smithi204.stdout:2422: write oid 9 current snap is 254 2024-09-16T05:19:40.783 INFO:tasks.rados.rados.2.smithi204.stdout:2422: seq_num 866 ranges {554862=713523,1926465=599969} 2024-09-16T05:19:40.788 INFO:tasks.rados.rados.2.smithi204.stdout:2422: writing smithi20457400-9 from 554862 to 1268385 tid 1 2024-09-16T05:19:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:2422: writing smithi20457400-9 from 1926465 to 2526434 tid 2 2024-09-16T05:19:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:2423: read oid 38 snap -1 2024-09-16T05:19:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:2423: expect deleted 2024-09-16T05:19:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:2424: copy_from oid 35 from oid 39 current snap is 254 2024-09-16T05:19:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:2422: finishing write tid 1 to smithi20457400-9 2024-09-16T05:19:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:2425: write oid 49 current snap is 254 2024-09-16T05:19:40.796 INFO:tasks.rados.rados.2.smithi204.stdout:2425: seq_num 867 ranges {571669=505125,1701300=743627} 2024-09-16T05:19:40.802 INFO:tasks.rados.rados.2.smithi204.stdout:2425: writing smithi20457400-49 from 571669 to 1076794 tid 1 2024-09-16T05:19:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2425: writing smithi20457400-49 from 1701300 to 2444927 tid 2 2024-09-16T05:19:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2426: read oid 41 snap -1 2024-09-16T05:19:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2426: expect (ObjNum 725 snap 209 seq_num 725) 2024-09-16T05:19:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2427: read oid 24 snap -1 2024-09-16T05:19:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2427: expect deleted 2024-09-16T05:19:40.810 INFO:tasks.rados.rados.2.smithi204.stdout:2428: read oid 36 snap -1 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2428: expect (ObjNum 834 snap 242 seq_num 834) 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2429: copy_from oid 6 from oid 47 current snap is 254 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2422: finishing write tid 2 to smithi20457400-9 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2422: finishing write tid 3 to smithi20457400-9 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2422: finishing write tid 4 to smithi20457400-9 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 863 (ObjNum 866 snap 254 seq_num 866) dirty exists 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2422: left oid 9 (ObjNum 866 snap 254 seq_num 866) 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2425: finishing write tid 1 to smithi20457400-49 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2424: finishing copy_from to smithi20457400-35 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2424: got expected ENOENT (src dne) 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2424: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2422: done (7 left) 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2423: done (6 left) 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2424: done (5 left) 2024-09-16T05:19:40.811 INFO:tasks.rados.rados.2.smithi204.stdout:2430: snap_create 2024-09-16T05:19:40.812 INFO:tasks.rados.rados.2.smithi204.stdout:2428: expect (ObjNum 834 snap 242 seq_num 834) 2024-09-16T05:19:40.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:40.866 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:40.874 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:40.974 INFO:tasks.rados.rados.2.smithi204.stdout:2429: finishing copy_from to smithi20457400-6 2024-09-16T05:19:40.975 INFO:tasks.rados.rados.2.smithi204.stdout:2429: got expected ENOENT (src dne) 2024-09-16T05:19:40.975 INFO:tasks.rados.rados.2.smithi204.stdout:2425: finishing write tid 2 to smithi20457400-49 2024-09-16T05:19:40.975 INFO:tasks.rados.rados.2.smithi204.stdout:2429: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:19:40.975 INFO:tasks.rados.rados.2.smithi204.stdout:2425: finishing write tid 3 to smithi20457400-49 2024-09-16T05:19:40.975 INFO:tasks.rados.rados.2.smithi204.stdout:2425: finishing write tid 4 to smithi20457400-49 2024-09-16T05:19:40.975 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 613 (ObjNum 867 snap 254 seq_num 867) dirty exists 2024-09-16T05:19:40.975 INFO:tasks.rados.rados.2.smithi204.stdout:2425: left oid 49 (ObjNum 867 snap 254 seq_num 867) 2024-09-16T05:19:40.975 INFO:tasks.rados.rados.2.smithi204.stdout:2426: expect (ObjNum 725 snap 209 seq_num 725) 2024-09-16T05:19:40.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:40.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:40.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:41.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:41.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:41.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:41.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:41.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:41.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:41.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:41.351 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:41.351 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:41.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:41.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:41.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:41.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:41.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:41.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:41.619 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:41.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:41.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:41.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:41.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:41.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:19:41.790 INFO:tasks.rados.rados.2.smithi204.stdout:2425: done (5 left) 2024-09-16T05:19:41.790 INFO:tasks.rados.rados.2.smithi204.stdout:2426: done (4 left) 2024-09-16T05:19:41.790 INFO:tasks.rados.rados.2.smithi204.stdout:2427: done (3 left) 2024-09-16T05:19:41.790 INFO:tasks.rados.rados.2.smithi204.stdout:2428: done (2 left) 2024-09-16T05:19:41.790 INFO:tasks.rados.rados.2.smithi204.stdout:2429: done (1 left) 2024-09-16T05:19:41.790 INFO:tasks.rados.rados.2.smithi204.stdout:2430: done (0 left) 2024-09-16T05:19:41.790 INFO:tasks.rados.rados.2.smithi204.stdout:2431: read oid 26 snap 240 2024-09-16T05:19:41.791 INFO:tasks.rados.rados.2.smithi204.stdout:2431: expect deleted 2024-09-16T05:19:41.791 INFO:tasks.rados.rados.2.smithi204.stdout:2432: copy_from oid 45 from oid 13 current snap is 255 2024-09-16T05:19:41.791 INFO:tasks.rados.rados.2.smithi204.stdout:2433: read oid 35 snap -1 2024-09-16T05:19:41.791 INFO:tasks.rados.rados.2.smithi204.stdout:2433: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:19:41.791 INFO:tasks.rados.rados.2.smithi204.stdout:2434: snap_remove snap 230 2024-09-16T05:19:41.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:41.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:41.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:41.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:41.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:41.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:42.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:42.118 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:42.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:42.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:41 smithi032 ceph-mon[27388]: pgmap v1107: 107 pgs: 2 peering, 1 active+recovering, 104 active+clean; 221 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 6.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 2/246 objects misplaced (0.813%); 2.4 MiB/s, 21 objects/s recovering 2024-09-16T05:19:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:41 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:19:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:41 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:19:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:41 smithi032 ceph-mon[27388]: osdmap e631: 12 total, 12 up, 11 in 2024-09-16T05:19:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3332056445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:41 smithi204 ceph-mon[30933]: pgmap v1107: 107 pgs: 2 peering, 1 active+recovering, 104 active+clean; 221 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 6.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 2/246 objects misplaced (0.813%); 2.4 MiB/s, 21 objects/s recovering 2024-09-16T05:19:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:41 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:19:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:41 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:19:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:41 smithi204 ceph-mon[30933]: osdmap e631: 12 total, 12 up, 11 in 2024-09-16T05:19:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3332056445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:42.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:41 smithi196 ceph-mon[30802]: pgmap v1107: 107 pgs: 2 peering, 1 active+recovering, 104 active+clean; 221 MiB data, 732 MiB used, 983 GiB / 983 GiB avail; 6.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 2/246 objects misplaced (0.813%); 2.4 MiB/s, 21 objects/s recovering 2024-09-16T05:19:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:41 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:19:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:41 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:19:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:41 smithi196 ceph-mon[30802]: osdmap e631: 12 total, 12 up, 11 in 2024-09-16T05:19:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3332056445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:42.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:42.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:42.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:19:42.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:42.371 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:42.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:42.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:42.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:42.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:42.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:42.623 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:42.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:42.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:42.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:42.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:42.786 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T05:19:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:2434: done (3 left) 2024-09-16T05:19:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:2435: rollback oid 12 current snap is 255 2024-09-16T05:19:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 237 2024-09-16T05:19:42.791 INFO:tasks.rados.rados.2.smithi204.stdout:2433: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:19:42.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:42.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:42.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:42.887 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:42 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:42.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3332056445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:42.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:42 smithi032 ceph-mon[27388]: osdmap e632: 12 total, 12 up, 11 in 2024-09-16T05:19:42.888 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/352289143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:19:42.903 INFO:tasks.rados.rados.2.smithi204.stdout:2432: finishing copy_from racing read to smithi20457400-45 2024-09-16T05:19:42.903 INFO:tasks.rados.rados.2.smithi204.stdout:2432: finishing copy_from to smithi20457400-45 2024-09-16T05:19:42.903 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1008 (ObjNum 855 snap 248 seq_num 855) dirty exists 2024-09-16T05:19:42.903 INFO:tasks.rados.rados.2.smithi204.stdout:2431: done (3 left) 2024-09-16T05:19:42.903 INFO:tasks.rados.rados.2.smithi204.stdout:2432: done (2 left) 2024-09-16T05:19:42.903 INFO:tasks.rados.rados.2.smithi204.stdout:2433: done (1 left) 2024-09-16T05:19:42.903 INFO:tasks.rados.rados.2.smithi204.stdout:2436: write (excl) oid 49 current snap is 255 2024-09-16T05:19:42.904 INFO:tasks.rados.rados.2.smithi204.stdout:2435: finishing rollback tid 0 to smithi20457400-12 2024-09-16T05:19:42.904 INFO:tasks.rados.rados.2.smithi204.stdout:2435: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:19:42.904 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 900 (ObjNum 129 snap 0 seq_num 1618998234) dirty dne 2024-09-16T05:19:42.904 INFO:tasks.rados.rados.2.smithi204.stdout:2436: seq_num 868 ranges {588476=580374,1643429=603638} 2024-09-16T05:19:42.909 INFO:tasks.rados.rados.2.smithi204.stdout:2436: writing smithi20457400-49 from 588476 to 1168850 tid 1 2024-09-16T05:19:42.915 INFO:tasks.rados.rados.2.smithi204.stdout:2436: writing smithi20457400-49 from 1643429 to 2247067 tid 2 2024-09-16T05:19:42.915 INFO:tasks.rados.rados.2.smithi204.stdout:2435: done (1 left) 2024-09-16T05:19:42.915 INFO:tasks.rados.rados.2.smithi204.stdout:2437: write oid 25 current snap is 255 2024-09-16T05:19:42.915 INFO:tasks.rados.rados.2.smithi204.stdout:2436: finishing write tid 1 to smithi20457400-49 2024-09-16T05:19:42.915 INFO:tasks.rados.rados.2.smithi204.stdout:2437: seq_num 869 ranges {605283=655623,1701911=747296} 2024-09-16T05:19:42.921 INFO:tasks.rados.rados.2.smithi204.stdout:2437: writing smithi20457400-25 from 605283 to 1260906 tid 1 2024-09-16T05:19:42.929 INFO:tasks.rados.rados.2.smithi204.stdout:2437: writing smithi20457400-25 from 1701911 to 2449207 tid 2 2024-09-16T05:19:42.929 INFO:tasks.rados.rados.2.smithi204.stdout:2438: write (excl) oid 45 current snap is 255 2024-09-16T05:19:42.930 INFO:tasks.rados.rados.2.smithi204.stdout:2436: finishing write tid 2 to smithi20457400-49 2024-09-16T05:19:42.930 INFO:tasks.rados.rados.2.smithi204.stdout:2436: finishing write tid 3 to smithi20457400-49 2024-09-16T05:19:42.930 INFO:tasks.rados.rados.2.smithi204.stdout:2436: finishing write tid 4 to smithi20457400-49 2024-09-16T05:19:42.930 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 616 (ObjNum 868 snap 255 seq_num 868) dirty exists 2024-09-16T05:19:42.930 INFO:tasks.rados.rados.2.smithi204.stdout:2436: left oid 49 (ObjNum 868 snap 255 seq_num 868) 2024-09-16T05:19:42.930 INFO:tasks.rados.rados.2.smithi204.stdout:2437: finishing write tid 1 to smithi20457400-25 2024-09-16T05:19:42.930 INFO:tasks.rados.rados.2.smithi204.stdout:2438: seq_num 870 ranges {622090=730872,1760393=607307} 2024-09-16T05:19:42.936 INFO:tasks.rados.rados.2.smithi204.stdout:2438: writing smithi20457400-45 from 622090 to 1352962 tid 1 2024-09-16T05:19:42.942 INFO:tasks.rados.rados.2.smithi204.stdout:2438: writing smithi20457400-45 from 1760393 to 2367700 tid 2 2024-09-16T05:19:42.942 INFO:tasks.rados.rados.2.smithi204.stdout:2436: done (2 left) 2024-09-16T05:19:42.942 INFO:tasks.rados.rados.2.smithi204.stdout:2439: delete oid 43 current snap is 255 2024-09-16T05:19:42.942 INFO:tasks.rados.rados.2.smithi204.stdout:2437: finishing write tid 2 to smithi20457400-25 2024-09-16T05:19:42.942 INFO:tasks.rados.rados.2.smithi204.stdout:2437: finishing write tid 3 to smithi20457400-25 2024-09-16T05:19:42.942 INFO:tasks.rados.rados.2.smithi204.stdout:2437: finishing write tid 4 to smithi20457400-25 2024-09-16T05:19:42.943 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 726 (ObjNum 869 snap 255 seq_num 869) dirty exists 2024-09-16T05:19:42.943 INFO:tasks.rados.rados.2.smithi204.stdout:2437: left oid 25 (ObjNum 869 snap 255 seq_num 869) 2024-09-16T05:19:42.943 INFO:tasks.rados.rados.2.smithi204.stdout:2438: finishing write tid 1 to smithi20457400-45 2024-09-16T05:19:42.944 INFO:tasks.rados.rados.2.smithi204.stdout:2437: done (2 left) 2024-09-16T05:19:42.944 INFO:tasks.rados.rados.2.smithi204.stdout:2439: done (1 left) 2024-09-16T05:19:42.944 INFO:tasks.rados.rados.2.smithi204.stdout:2440: read oid 4 snap -1 2024-09-16T05:19:42.944 INFO:tasks.rados.rados.2.smithi204.stdout:2440: expect deleted 2024-09-16T05:19:42.944 INFO:tasks.rados.rados.2.smithi204.stdout:2441: write oid 28 current snap is 255 2024-09-16T05:19:42.944 INFO:tasks.rados.rados.2.smithi204.stdout:2441: seq_num 872 ranges {655704=481370,1761004=494623} 2024-09-16T05:19:42.949 INFO:tasks.rados.rados.2.smithi204.stdout:2441: writing smithi20457400-28 from 655704 to 1137074 tid 1 2024-09-16T05:19:42.955 INFO:tasks.rados.rados.2.smithi204.stdout:2441: writing smithi20457400-28 from 1761004 to 2255627 tid 2 2024-09-16T05:19:42.956 INFO:tasks.rados.rados.2.smithi204.stdout:2442: delete oid 37 current snap is 255 2024-09-16T05:19:42.956 INFO:tasks.rados.rados.2.smithi204.stdout:2438: finishing write tid 2 to smithi20457400-45 2024-09-16T05:19:42.956 INFO:tasks.rados.rados.2.smithi204.stdout:2438: finishing write tid 3 to smithi20457400-45 2024-09-16T05:19:42.956 INFO:tasks.rados.rados.2.smithi204.stdout:2438: finishing write tid 4 to smithi20457400-45 2024-09-16T05:19:42.956 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1011 (ObjNum 870 snap 255 seq_num 870) dirty exists 2024-09-16T05:19:42.956 INFO:tasks.rados.rados.2.smithi204.stdout:2438: left oid 45 (ObjNum 870 snap 255 seq_num 870) 2024-09-16T05:19:42.956 INFO:tasks.rados.rados.2.smithi204.stdout:2441: finishing write tid 1 to smithi20457400-28 2024-09-16T05:19:42.958 INFO:tasks.rados.rados.2.smithi204.stdout:2438: done (3 left) 2024-09-16T05:19:42.958 INFO:tasks.rados.rados.2.smithi204.stdout:2440: done (2 left) 2024-09-16T05:19:42.958 INFO:tasks.rados.rados.2.smithi204.stdout:2442: done (1 left) 2024-09-16T05:19:42.958 INFO:tasks.rados.rados.2.smithi204.stdout:2443: read oid 45 snap -1 2024-09-16T05:19:42.958 INFO:tasks.rados.rados.2.smithi204.stdout:2443: expect (ObjNum 870 snap 255 seq_num 870) 2024-09-16T05:19:42.958 INFO:tasks.rados.rados.2.smithi204.stdout:2444: snap_remove snap 253 2024-09-16T05:19:42.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:43.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:43.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:43.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:19:43.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:43.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:43.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:43.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:42 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:43.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3332056445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:42 smithi204 ceph-mon[30933]: osdmap e632: 12 total, 12 up, 11 in 2024-09-16T05:19:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/352289143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:19:43.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:42 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3332056445' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:42 smithi196 ceph-mon[30802]: osdmap e632: 12 total, 12 up, 11 in 2024-09-16T05:19:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/352289143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:19:43.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:43.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:43.323 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:43.325 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:43.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:43.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:43.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:43.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:43.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:43.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:43.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:43.636 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:43.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:43.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:43.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:43.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:43.796 INFO:tasks.rados.rados.2.smithi204.stdout:2444: done (2 left) 2024-09-16T05:19:43.797 INFO:tasks.rados.rados.2.smithi204.stdout:2445: delete oid 12 current snap is 255 2024-09-16T05:19:43.797 INFO:tasks.rados.rados.2.smithi204.stdout:2441: finishing write tid 2 to smithi20457400-28 2024-09-16T05:19:43.797 INFO:tasks.rados.rados.2.smithi204.stdout:2443: expect (ObjNum 870 snap 255 seq_num 870) 2024-09-16T05:19:43.866 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:43.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:43.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:43.984 INFO:tasks.rados.rados.2.smithi204.stdout:2441: finishing write tid 3 to smithi20457400-28 2024-09-16T05:19:43.984 INFO:tasks.rados.rados.2.smithi204.stdout:2441: finishing write tid 4 to smithi20457400-28 2024-09-16T05:19:43.984 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 719 (ObjNum 872 snap 255 seq_num 872) dirty exists 2024-09-16T05:19:43.985 INFO:tasks.rados.rados.2.smithi204.stdout:2441: left oid 28 (ObjNum 872 snap 255 seq_num 872) 2024-09-16T05:19:43.985 INFO:tasks.rados.rados.2.smithi204.stdout:2441: done (2 left) 2024-09-16T05:19:43.985 INFO:tasks.rados.rados.2.smithi204.stdout:2443: done (1 left) 2024-09-16T05:19:43.985 INFO:tasks.rados.rados.2.smithi204.stdout:2445: done (0 left) 2024-09-16T05:19:43.985 INFO:tasks.rados.rados.2.smithi204.stdout:2446: snap_remove snap 237 2024-09-16T05:19:43.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:44.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:44.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:44.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:43 smithi032 ceph-mon[27388]: pgmap v1110: 107 pgs: 107 active+clean; 225 MiB data, 749 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 5 op/s; 2.5 MiB/s, 11 objects/s recovering 2024-09-16T05:19:44.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:43 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:19:44.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/352289143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:19:44.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:43 smithi032 ceph-mon[27388]: osdmap e633: 12 total, 12 up, 11 in 2024-09-16T05:19:44.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T05:19:44.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:44.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:44.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:43 smithi204 ceph-mon[30933]: pgmap v1110: 107 pgs: 107 active+clean; 225 MiB data, 749 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 5 op/s; 2.5 MiB/s, 11 objects/s recovering 2024-09-16T05:19:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:43 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:19:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/352289143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:19:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:43 smithi204 ceph-mon[30933]: osdmap e633: 12 total, 12 up, 11 in 2024-09-16T05:19:44.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:44.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:43 smithi196 ceph-mon[30802]: pgmap v1110: 107 pgs: 107 active+clean; 225 MiB data, 749 MiB used, 983 GiB / 983 GiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 5 op/s; 2.5 MiB/s, 11 objects/s recovering 2024-09-16T05:19:44.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:43 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:19:44.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/352289143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:19:44.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:43 smithi196 ceph-mon[30802]: osdmap e633: 12 total, 12 up, 11 in 2024-09-16T05:19:44.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:44.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:44.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:44.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:44.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:44.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:44.519 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:44.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:44.614 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:19:44.615 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:19:44.615 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:19:44.615 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:19:44.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:44.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:19:44.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:44.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:44.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:44.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:44.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:44.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:2446: done (0 left) 2024-09-16T05:19:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:2447: write (excl) oid 32 current snap is 255 2024-09-16T05:19:44.806 INFO:tasks.rados.rados.2.smithi204.stdout:2447: seq_num 875 ranges {706125=423470,1652803=758303} 2024-09-16T05:19:44.813 INFO:tasks.rados.rados.2.smithi204.stdout:2447: writing smithi20457400-32 from 706125 to 1129595 tid 1 2024-09-16T05:19:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2447: writing smithi20457400-32 from 1652803 to 2411106 tid 2 2024-09-16T05:19:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2448: write oid 10 current snap is 255 2024-09-16T05:19:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2448: seq_num 876 ranges {722932=498719,1994932=501961} 2024-09-16T05:19:44.827 INFO:tasks.rados.rados.2.smithi204.stdout:2448: writing smithi20457400-10 from 722932 to 1221651 tid 1 2024-09-16T05:19:44.835 INFO:tasks.rados.rados.2.smithi204.stdout:2448: writing smithi20457400-10 from 1994932 to 2496893 tid 2 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2449: rollback oid 41 current snap is 255 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 41 to 244 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2447: finishing write tid 1 to smithi20457400-32 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2450: read oid 19 snap -1 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2450: expect (ObjNum 797 snap 234 seq_num 797) 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2451: copy_from oid 2 from oid 4 current snap is 255 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2447: finishing write tid 2 to smithi20457400-32 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2447: finishing write tid 3 to smithi20457400-32 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2447: finishing write tid 4 to smithi20457400-32 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 913 (ObjNum 875 snap 255 seq_num 875) dirty exists 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2447: left oid 32 (ObjNum 875 snap 255 seq_num 875) 2024-09-16T05:19:44.836 INFO:tasks.rados.rados.2.smithi204.stdout:2448: finishing write tid 1 to smithi20457400-10 2024-09-16T05:19:44.837 INFO:tasks.rados.rados.2.smithi204.stdout:2447: done (4 left) 2024-09-16T05:19:44.837 INFO:tasks.rados.rados.2.smithi204.stdout:2452: read oid 22 snap -1 2024-09-16T05:19:44.837 INFO:tasks.rados.rados.2.smithi204.stdout:2452: expect (ObjNum 848 snap 246 seq_num 848) 2024-09-16T05:19:44.837 INFO:tasks.rados.rados.2.smithi204.stdout:2453: snap_create 2024-09-16T05:19:44.837 INFO:tasks.rados.rados.2.smithi204.stdout:2451: finishing copy_from to smithi20457400-2 2024-09-16T05:19:44.837 INFO:tasks.rados.rados.2.smithi204.stdout:2451: got expected ENOENT (src dne) 2024-09-16T05:19:44.837 INFO:tasks.rados.rados.2.smithi204.stdout:2451: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:19:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2448: finishing write tid 2 to smithi20457400-10 2024-09-16T05:19:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2448: finishing write tid 3 to smithi20457400-10 2024-09-16T05:19:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2448: finishing write tid 4 to smithi20457400-10 2024-09-16T05:19:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 881 (ObjNum 876 snap 255 seq_num 876) dirty exists 2024-09-16T05:19:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2448: left oid 10 (ObjNum 876 snap 255 seq_num 876) 2024-09-16T05:19:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2449: finishing rollback tid 0 to smithi20457400-41 2024-09-16T05:19:44.841 INFO:tasks.rados.rados.2.smithi204.stdout:2449: finishing rollback tid 1 to smithi20457400-41 2024-09-16T05:19:44.841 INFO:tasks.rados.rados.2.smithi204.stdout:2449: finishing rollback tid 2 to smithi20457400-41 2024-09-16T05:19:44.841 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 885 (ObjNum 725 snap 209 seq_num 725) dirty exists 2024-09-16T05:19:44.842 INFO:tasks.rados.rados.2.smithi204.stdout:2450: expect (ObjNum 797 snap 234 seq_num 797) 2024-09-16T05:19:44.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:44.901 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:44.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:44.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:44 smithi032 ceph-mon[27388]: osdmap e634: 12 total, 12 up, 11 in 2024-09-16T05:19:44.963 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2121196773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:45.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:45.003 INFO:tasks.rados.rados.2.smithi204.stdout:2452: expect (ObjNum 848 snap 246 seq_num 848) 2024-09-16T05:19:45.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:45.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:45.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:45.150 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:45.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:44 smithi204 ceph-mon[30933]: osdmap e634: 12 total, 12 up, 11 in 2024-09-16T05:19:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2121196773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:45.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:44 smithi196 ceph-mon[30802]: osdmap e634: 12 total, 12 up, 11 in 2024-09-16T05:19:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2121196773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:45.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:45.274 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:45.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:45.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:19:45.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:45.398 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:45.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:45.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:45.522 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:45.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:45.547 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:45.548 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:45.548 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:45.548 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:45.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:45.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:45.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:45.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:45.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:45.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:45.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:45.550 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:45.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:45.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:45.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:45.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:45.769 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:45.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:45.782 INFO:tasks.rados.rados.2.smithi204.stdout:2448: done (5 left) 2024-09-16T05:19:45.782 INFO:tasks.rados.rados.2.smithi204.stdout:2449: done (4 left) 2024-09-16T05:19:45.782 INFO:tasks.rados.rados.2.smithi204.stdout:2450: done (3 left) 2024-09-16T05:19:45.782 INFO:tasks.rados.rados.2.smithi204.stdout:2451: done (2 left) 2024-09-16T05:19:45.783 INFO:tasks.rados.rados.2.smithi204.stdout:2452: done (1 left) 2024-09-16T05:19:45.783 INFO:tasks.rados.rados.2.smithi204.stdout:2453: done (0 left) 2024-09-16T05:19:45.783 INFO:tasks.rados.rados.2.smithi204.stdout:2454: write oid 12 current snap is 256 2024-09-16T05:19:45.783 INFO:tasks.rados.rados.2.smithi204.stdout:2454: seq_num 877 ranges {739739=573968,2053414=686325} 2024-09-16T05:19:45.789 INFO:tasks.rados.rados.2.smithi204.stdout:2454: writing smithi20457400-12 from 739739 to 1313707 tid 1 2024-09-16T05:19:45.797 INFO:tasks.rados.rados.2.smithi204.stdout:2454: writing smithi20457400-12 from 2053414 to 2739739 tid 2 2024-09-16T05:19:45.797 INFO:tasks.rados.rados.2.smithi204.stdout:2455: write oid 2 current snap is 256 2024-09-16T05:19:45.797 INFO:tasks.rados.rados.2.smithi204.stdout:2455: seq_num 878 ranges {756546=649217,2111896=505630} 2024-09-16T05:19:45.805 INFO:tasks.rados.rados.2.smithi204.stdout:2455: writing smithi20457400-2 from 756546 to 1405763 tid 1 2024-09-16T05:19:45.812 INFO:tasks.rados.rados.2.smithi204.stdout:2455: writing smithi20457400-2 from 2111896 to 2617526 tid 2 2024-09-16T05:19:45.812 INFO:tasks.rados.rados.2.smithi204.stdout:2456: write (excl) oid 50 current snap is 256 2024-09-16T05:19:45.812 INFO:tasks.rados.rados.2.smithi204.stdout:2454: finishing write tid 1 to smithi20457400-12 2024-09-16T05:19:45.812 INFO:tasks.rados.rados.2.smithi204.stdout:2454: finishing write tid 2 to smithi20457400-12 2024-09-16T05:19:45.812 INFO:tasks.rados.rados.2.smithi204.stdout:2454: finishing write tid 3 to smithi20457400-12 2024-09-16T05:19:45.812 INFO:tasks.rados.rados.2.smithi204.stdout:2454: finishing write tid 4 to smithi20457400-12 2024-09-16T05:19:45.813 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 916 (ObjNum 877 snap 256 seq_num 877) dirty exists 2024-09-16T05:19:45.813 INFO:tasks.rados.rados.2.smithi204.stdout:2454: left oid 12 (ObjNum 877 snap 256 seq_num 877) 2024-09-16T05:19:45.813 INFO:tasks.rados.rados.2.smithi204.stdout:2455: finishing write tid 1 to smithi20457400-2 2024-09-16T05:19:45.813 INFO:tasks.rados.rados.2.smithi204.stdout:2456: seq_num 879 ranges {773353=724466,2170378=602975} 2024-09-16T05:19:45.820 INFO:tasks.rados.rados.2.smithi204.stdout:2456: writing smithi20457400-50 from 773353 to 1497819 tid 1 2024-09-16T05:19:45.827 INFO:tasks.rados.rados.2.smithi204.stdout:2456: writing smithi20457400-50 from 2170378 to 2773353 tid 2 2024-09-16T05:19:45.827 INFO:tasks.rados.rados.2.smithi204.stdout:2454: done (2 left) 2024-09-16T05:19:45.827 INFO:tasks.rados.rados.2.smithi204.stdout:2457: write (excl) oid 24 current snap is 256 2024-09-16T05:19:45.827 INFO:tasks.rados.rados.2.smithi204.stdout:2457: seq_num 880 ranges {790160=799715,2112507=509299} 2024-09-16T05:19:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:2457: writing smithi20457400-24 from 790160 to 1589875 tid 1 2024-09-16T05:19:45.841 INFO:tasks.rados.rados.2.smithi204.stdout:2457: writing smithi20457400-24 from 2112507 to 2621806 tid 2 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2458: rollback oid 4 current snap is 256 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 247 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2455: finishing write tid 2 to smithi20457400-2 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2455: finishing write tid 3 to smithi20457400-2 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2459: rollback oid 34 current snap is 256 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 229 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2455: finishing write tid 4 to smithi20457400-2 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 521 (ObjNum 878 snap 256 seq_num 878) dirty exists 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2455: left oid 2 (ObjNum 878 snap 256 seq_num 878) 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2456: finishing write tid 1 to smithi20457400-50 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2456: finishing write tid 2 to smithi20457400-50 2024-09-16T05:19:45.842 INFO:tasks.rados.rados.2.smithi204.stdout:2456: finishing write tid 3 to smithi20457400-50 2024-09-16T05:19:45.843 INFO:tasks.rados.rados.2.smithi204.stdout:2457: finishing write tid 1 to smithi20457400-24 2024-09-16T05:19:45.843 INFO:tasks.rados.rados.2.smithi204.stdout:2455: done (4 left) 2024-09-16T05:19:45.843 INFO:tasks.rados.rados.2.smithi204.stdout:2460: snap_create 2024-09-16T05:19:45.843 INFO:tasks.rados.rados.2.smithi204.stdout:2458: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:19:45.843 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 544235886 snap 171062577 seq_num 304) dirty dne 2024-09-16T05:19:45.844 INFO:tasks.rados.rados.2.smithi204.stdout:2459: finishing rollback tid 0 to smithi20457400-34 2024-09-16T05:19:45.845 INFO:tasks.rados.rados.2.smithi204.stdout:2459: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:19:45.845 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 707 (ObjNum 49 snap 0 seq_num 2671988744) dirty dne 2024-09-16T05:19:45.845 INFO:tasks.rados.rados.2.smithi204.stdout:2457: finishing write tid 2 to smithi20457400-24 2024-09-16T05:19:45.845 INFO:tasks.rados.rados.2.smithi204.stdout:2457: finishing write tid 3 to smithi20457400-24 2024-09-16T05:19:45.846 INFO:tasks.rados.rados.2.smithi204.stdout:2457: finishing write tid 4 to smithi20457400-24 2024-09-16T05:19:45.846 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 663 (ObjNum 880 snap 256 seq_num 880) dirty exists 2024-09-16T05:19:45.846 INFO:tasks.rados.rados.2.smithi204.stdout:2457: left oid 24 (ObjNum 880 snap 256 seq_num 880) 2024-09-16T05:19:45.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:45.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:45.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:45.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:46.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:46.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:46.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:46.142 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:46.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:46.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:45 smithi032 ceph-mon[27388]: pgmap v1113: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 228 MiB data, 754 MiB used, 983 GiB / 983 GiB avail; 4.9 MiB/s rd, 647 KiB/s wr, 8 op/s; 2.0 MiB/s, 6 objects/s recovering 2024-09-16T05:19:46.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:45 smithi032 ceph-mon[27388]: osdmap e635: 12 total, 12 up, 11 in 2024-09-16T05:19:46.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:45 smithi032 ceph-mon[27388]: osdmap e636: 12 total, 12 up, 11 in 2024-09-16T05:19:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:45 smithi204 ceph-mon[30933]: pgmap v1113: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 228 MiB data, 754 MiB used, 983 GiB / 983 GiB avail; 4.9 MiB/s rd, 647 KiB/s wr, 8 op/s; 2.0 MiB/s, 6 objects/s recovering 2024-09-16T05:19:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:45 smithi204 ceph-mon[30933]: osdmap e635: 12 total, 12 up, 11 in 2024-09-16T05:19:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:45 smithi204 ceph-mon[30933]: osdmap e636: 12 total, 12 up, 11 in 2024-09-16T05:19:46.198 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:46.207 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:45 smithi196 ceph-mon[30802]: pgmap v1113: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 228 MiB data, 754 MiB used, 983 GiB / 983 GiB avail; 4.9 MiB/s rd, 647 KiB/s wr, 8 op/s; 2.0 MiB/s, 6 objects/s recovering 2024-09-16T05:19:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:45 smithi196 ceph-mon[30802]: osdmap e635: 12 total, 12 up, 11 in 2024-09-16T05:19:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:45 smithi196 ceph-mon[30802]: osdmap e636: 12 total, 12 up, 11 in 2024-09-16T05:19:46.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:46.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:46.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:46.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:46.395 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:46.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:46.397 INFO:tasks.rados.rados.2.smithi204.stdout:2456: finishing write tid 4 to smithi20457400-50 2024-09-16T05:19:46.397 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1096 (ObjNum 879 snap 256 seq_num 879) dirty exists 2024-09-16T05:19:46.397 INFO:tasks.rados.rados.2.smithi204.stdout:2456: left oid 50 (ObjNum 879 snap 256 seq_num 879) 2024-09-16T05:19:46.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:46.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:46.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:46.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:46.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:46.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:46.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:46.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:46.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:46.807 INFO:tasks.rados.rados.2.smithi204.stdout:2456: done (4 left) 2024-09-16T05:19:46.807 INFO:tasks.rados.rados.2.smithi204.stdout:2457: done (3 left) 2024-09-16T05:19:46.807 INFO:tasks.rados.rados.2.smithi204.stdout:2458: done (2 left) 2024-09-16T05:19:46.807 INFO:tasks.rados.rados.2.smithi204.stdout:2459: done (1 left) 2024-09-16T05:19:46.808 INFO:tasks.rados.rados.2.smithi204.stdout:2460: done (0 left) 2024-09-16T05:19:46.808 INFO:tasks.rados.rados.2.smithi204.stdout:2461: write oid 6 current snap is 257 2024-09-16T05:19:46.808 INFO:tasks.rados.rados.2.smithi204.stdout:2461: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-09-16T05:19:46.811 INFO:tasks.rados.rados.2.smithi204.stdout:2461: writing smithi20457400-6 from 406967 to 881931 tid 1 2024-09-16T05:19:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:2461: writing smithi20457400-6 from 1370989 to 2140299 tid 2 2024-09-16T05:19:46.822 INFO:tasks.rados.rados.2.smithi204.stdout:2461: writing smithi20457400-6 from 2677186 to 2806967 tid 3 2024-09-16T05:19:46.822 INFO:tasks.rados.rados.2.smithi204.stdout:2462: read oid 4 snap -1 2024-09-16T05:19:46.822 INFO:tasks.rados.rados.2.smithi204.stdout:2462: expect deleted 2024-09-16T05:19:46.822 INFO:tasks.rados.rados.2.smithi204.stdout:2463: delete oid 10 current snap is 257 2024-09-16T05:19:46.822 INFO:tasks.rados.rados.2.smithi204.stdout:2461: finishing write tid 1 to smithi20457400-6 2024-09-16T05:19:46.822 INFO:tasks.rados.rados.2.smithi204.stdout:2461: finishing write tid 2 to smithi20457400-6 2024-09-16T05:19:46.824 INFO:tasks.rados.rados.2.smithi204.stdout:2461: finishing write tid 3 to smithi20457400-6 2024-09-16T05:19:46.824 INFO:tasks.rados.rados.2.smithi204.stdout:2461: finishing write tid 4 to smithi20457400-6 2024-09-16T05:19:46.825 INFO:tasks.rados.rados.2.smithi204.stdout:2461: finishing write tid 5 to smithi20457400-6 2024-09-16T05:19:46.825 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 737 (ObjNum 881 snap 257 seq_num 881) dirty exists 2024-09-16T05:19:46.825 INFO:tasks.rados.rados.2.smithi204.stdout:2461: left oid 6 (ObjNum 881 snap 257 seq_num 881) 2024-09-16T05:19:46.826 INFO:tasks.rados.rados.2.smithi204.stdout:2461: done (2 left) 2024-09-16T05:19:46.826 INFO:tasks.rados.rados.2.smithi204.stdout:2462: done (1 left) 2024-09-16T05:19:46.826 INFO:tasks.rados.rados.2.smithi204.stdout:2463: done (0 left) 2024-09-16T05:19:46.826 INFO:tasks.rados.rados.2.smithi204.stdout:2464: snap_create 2024-09-16T05:19:46.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:46.879 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:46.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:46.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:46.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:47.021 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:47.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:47.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:47.145 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:47.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:47.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:47.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:47.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:47.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:47.312 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:19:47.312 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:19:47.312 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:19:47.312 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:19:47.312 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:19:47.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:47.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:47.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:47.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:47.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:47.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:47.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:47.650 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:47.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:47.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:47.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:47.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T05:19:47.809 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2464: done (0 left) 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2465: read oid 48 snap -1 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2465: expect deleted 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2466: copy_from oid 20 from oid 44 current snap is 258 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2467: read oid 31 snap -1 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2467: expect deleted 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2468: read oid 27 snap -1 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2468: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:19:47.814 INFO:tasks.rados.rados.2.smithi204.stdout:2469: copy_from oid 18 from oid 42 current snap is 258 2024-09-16T05:19:47.815 INFO:tasks.rados.rados.2.smithi204.stdout:2470: read oid 24 snap 238 2024-09-16T05:19:47.815 INFO:tasks.rados.rados.2.smithi204.stdout:2470: expect (ObjNum 784 snap 229 seq_num 784) 2024-09-16T05:19:47.815 INFO:tasks.rados.rados.2.smithi204.stdout:2471: read oid 6 snap -1 2024-09-16T05:19:47.815 INFO:tasks.rados.rados.2.smithi204.stdout:2471: expect (ObjNum 881 snap 257 seq_num 881) 2024-09-16T05:19:47.815 INFO:tasks.rados.rados.2.smithi204.stdout:2472: copy_from oid 3 from oid 32 current snap is 258 2024-09-16T05:19:47.815 INFO:tasks.rados.rados.2.smithi204.stdout:2473: read oid 1 snap -1 2024-09-16T05:19:47.815 INFO:tasks.rados.rados.2.smithi204.stdout:2473: expect (ObjNum 842 snap 243 seq_num 842) 2024-09-16T05:19:47.815 INFO:tasks.rados.rados.2.smithi204.stdout:2474: snap_remove snap 252 2024-09-16T05:19:47.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:47.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:47.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:48.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:48.026 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:48.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:48.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:48.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:48.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:48.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:19:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:47 smithi032 ceph-mon[27388]: pgmap v1116: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 229 MiB data, 758 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 950 KiB/s wr, 13 op/s 2024-09-16T05:19:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:47 smithi032 ceph-mon[27388]: osdmap e637: 12 total, 12 up, 11 in 2024-09-16T05:19:48.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2472295522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:48.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1850265989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:19:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:47 smithi204 ceph-mon[30933]: pgmap v1116: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 229 MiB data, 758 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 950 KiB/s wr, 13 op/s 2024-09-16T05:19:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:47 smithi204 ceph-mon[30933]: osdmap e637: 12 total, 12 up, 11 in 2024-09-16T05:19:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2472295522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1850265989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:19:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:47 smithi196 ceph-mon[30802]: pgmap v1116: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 229 MiB data, 758 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 950 KiB/s wr, 13 op/s 2024-09-16T05:19:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:47 smithi196 ceph-mon[30802]: osdmap e637: 12 total, 12 up, 11 in 2024-09-16T05:19:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2472295522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:19:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1850265989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:19:48.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:48.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:48.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:48.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:19:48.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:48.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:48.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:48.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:48.534 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:48.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:48.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:48.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:48.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:48.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:48.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:48.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:48.816 INFO:tasks.rados.rados.2.smithi204.stdout:2474: done (9 left) 2024-09-16T05:19:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:2475: copy_from oid 17 from oid 16 current snap is 258 2024-09-16T05:19:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:2469: finishing copy_from to smithi20457400-18 2024-09-16T05:19:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:2469: got expected ENOENT (src dne) 2024-09-16T05:19:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:2469: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:19:48.817 INFO:tasks.rados.rados.2.smithi204.stdout:2473: expect (ObjNum 842 snap 243 seq_num 842) 2024-09-16T05:19:48.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:48.909 INFO:tasks.rados.rados.2.smithi204.stdout:2466: finishing copy_from to smithi20457400-20 2024-09-16T05:19:48.909 INFO:tasks.rados.rados.2.smithi204.stdout:2466: got expected ENOENT (src dne) 2024-09-16T05:19:48.909 INFO:tasks.rados.rados.2.smithi204.stdout:2466: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:19:48.909 INFO:tasks.rados.rados.2.smithi204.stdout:2468: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:19:48.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:48.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:49.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:48 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:49.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2472295522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:49.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1850265989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:19:49.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:48 smithi032 ceph-mon[27388]: osdmap e638: 12 total, 12 up, 11 in 2024-09-16T05:19:49.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:48 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:19:49.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:48 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:19:49.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:48 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:19:49.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:48 smithi032 ceph-mon[27388]: 2.44 scrub ok 2024-09-16T05:19:49.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:49.035 INFO:tasks.rados.rados.2.smithi204.stdout:2471: expect (ObjNum 881 snap 257 seq_num 881) 2024-09-16T05:19:49.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:49.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:49.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:49.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:49.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:48 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2472295522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1850265989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:19:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:48 smithi204 ceph-mon[30933]: osdmap e638: 12 total, 12 up, 11 in 2024-09-16T05:19:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:48 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:19:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:48 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:19:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:48 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:19:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:48 smithi204 ceph-mon[30933]: 2.44 scrub ok 2024-09-16T05:19:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:48 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:19:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2472295522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:19:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1850265989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:19:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:48 smithi196 ceph-mon[30802]: osdmap e638: 12 total, 12 up, 11 in 2024-09-16T05:19:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:48 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:19:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:48 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:19:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:48 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:19:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:48 smithi196 ceph-mon[30802]: 2.44 scrub ok 2024-09-16T05:19:49.250 INFO:tasks.rados.rados.2.smithi204.stdout:2470: expect (ObjNum 784 snap 229 seq_num 784) 2024-09-16T05:19:49.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:49.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:49.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:49.400 INFO:tasks.rados.rados.2.smithi204.stdout:2472: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:19:49.401 INFO:tasks.rados.rados.2.smithi204.stdout:2472: finishing copy_from to smithi20457400-3 2024-09-16T05:19:49.401 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 913 (ObjNum 875 snap 255 seq_num 875) dirty exists 2024-09-16T05:19:49.401 INFO:tasks.rados.rados.2.smithi204.stdout:2475: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:19:49.401 INFO:tasks.rados.rados.2.smithi204.stdout:2475: finishing copy_from to smithi20457400-17 2024-09-16T05:19:49.401 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 369 (ObjNum 781 snap 227 seq_num 781) dirty exists 2024-09-16T05:19:49.402 INFO:tasks.rados.rados.2.smithi204.stdout:2465: done (9 left) 2024-09-16T05:19:49.402 INFO:tasks.rados.rados.2.smithi204.stdout:2466: done (8 left) 2024-09-16T05:19:49.402 INFO:tasks.rados.rados.2.smithi204.stdout:2467: done (7 left) 2024-09-16T05:19:49.402 INFO:tasks.rados.rados.2.smithi204.stdout:2468: done (6 left) 2024-09-16T05:19:49.402 INFO:tasks.rados.rados.2.smithi204.stdout:2469: done (5 left) 2024-09-16T05:19:49.403 INFO:tasks.rados.rados.2.smithi204.stdout:2470: done (4 left) 2024-09-16T05:19:49.403 INFO:tasks.rados.rados.2.smithi204.stdout:2471: done (3 left) 2024-09-16T05:19:49.403 INFO:tasks.rados.rados.2.smithi204.stdout:2472: done (2 left) 2024-09-16T05:19:49.403 INFO:tasks.rados.rados.2.smithi204.stdout:2473: done (1 left) 2024-09-16T05:19:49.403 INFO:tasks.rados.rados.2.smithi204.stdout:2475: done (0 left) 2024-09-16T05:19:49.404 INFO:tasks.rados.rados.2.smithi204.stdout:2476: copy_from oid 16 from oid 29 current snap is 258 2024-09-16T05:19:49.404 INFO:tasks.rados.rados.2.smithi204.stdout:2477: write oid 25 current snap is 258 2024-09-16T05:19:49.404 INFO:tasks.rados.rados.2.smithi204.stdout:2477: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-09-16T05:19:49.406 INFO:tasks.rados.rados.2.smithi204.stdout:2477: writing smithi20457400-25 from 440581 to 1182396 tid 1 2024-09-16T05:19:49.412 INFO:tasks.rados.rados.2.smithi204.stdout:2477: writing smithi20457400-25 from 1604306 to 2260932 tid 2 2024-09-16T05:19:49.416 INFO:tasks.rados.rados.2.smithi204.stdout:2477: writing smithi20457400-25 from 2732032 to 2840581 tid 3 2024-09-16T05:19:49.416 INFO:tasks.rados.rados.2.smithi204.stdout:2478: snap_create 2024-09-16T05:19:49.416 INFO:tasks.rados.rados.2.smithi204.stdout:2476: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:19:49.417 INFO:tasks.rados.rados.2.smithi204.stdout:2476: finishing copy_from to smithi20457400-16 2024-09-16T05:19:49.417 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 893 (ObjNum 864 snap 249 seq_num 864) dirty exists 2024-09-16T05:19:49.417 INFO:tasks.rados.rados.2.smithi204.stdout:2477: finishing write tid 1 to smithi20457400-25 2024-09-16T05:19:49.417 INFO:tasks.rados.rados.2.smithi204.stdout:2477: finishing write tid 2 to smithi20457400-25 2024-09-16T05:19:49.418 INFO:tasks.rados.rados.2.smithi204.stdout:2477: finishing write tid 3 to smithi20457400-25 2024-09-16T05:19:49.419 INFO:tasks.rados.rados.2.smithi204.stdout:2477: finishing write tid 4 to smithi20457400-25 2024-09-16T05:19:49.419 INFO:tasks.rados.rados.2.smithi204.stdout:2477: finishing write tid 5 to smithi20457400-25 2024-09-16T05:19:49.419 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 730 (ObjNum 883 snap 258 seq_num 883) dirty exists 2024-09-16T05:19:49.419 INFO:tasks.rados.rados.2.smithi204.stdout:2477: left oid 25 (ObjNum 883 snap 258 seq_num 883) 2024-09-16T05:19:49.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:49.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:49.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:49.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:49.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:49.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:49.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:49.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:49.707 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:49.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T05:19:49.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:49.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:49.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:49.821 INFO:tasks.rados.rados.2.smithi204.stdout:2476: done (2 left) 2024-09-16T05:19:49.821 INFO:tasks.rados.rados.2.smithi204.stdout:2477: done (1 left) 2024-09-16T05:19:49.821 INFO:tasks.rados.rados.2.smithi204.stdout:2478: done (0 left) 2024-09-16T05:19:49.821 INFO:tasks.rados.rados.2.smithi204.stdout:2479: write oid 38 current snap is 259 2024-09-16T05:19:49.821 INFO:tasks.rados.rados.2.smithi204.stdout:2479: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2024-09-16T05:19:49.825 INFO:tasks.rados.rados.2.smithi204.stdout:2479: writing smithi20457400-38 from 457388 to 874452 tid 1 2024-09-16T05:19:49.832 INFO:tasks.rados.rados.2.smithi204.stdout:2479: writing smithi20457400-38 from 1546435 to 2063072 tid 2 2024-09-16T05:19:49.837 INFO:tasks.rados.rados.2.smithi204.stdout:2479: writing smithi20457400-38 from 2643102 to 2857388 tid 3 2024-09-16T05:19:49.837 INFO:tasks.rados.rados.2.smithi204.stdout:2480: snap_remove snap 238 2024-09-16T05:19:49.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:49.924 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:49.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:49.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:49 smithi032 ceph-mon[27388]: pgmap v1119: 107 pgs: 107 active+clean; 239 MiB data, 770 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 21 op/s 2024-09-16T05:19:49.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:49 smithi032 ceph-mon[27388]: 2.4f scrub starts 2024-09-16T05:19:49.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:49 smithi032 ceph-mon[27388]: 2.4f scrub ok 2024-09-16T05:19:49.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:49 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:19:49.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:49 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:19:49.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:49 smithi032 ceph-mon[27388]: osdmap e639: 12 total, 12 up, 11 in 2024-09-16T05:19:50.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:50.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:50.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:50.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:50.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:49 smithi204 ceph-mon[30933]: pgmap v1119: 107 pgs: 107 active+clean; 239 MiB data, 770 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 21 op/s 2024-09-16T05:19:50.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:49 smithi204 ceph-mon[30933]: 2.4f scrub starts 2024-09-16T05:19:50.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:49 smithi204 ceph-mon[30933]: 2.4f scrub ok 2024-09-16T05:19:50.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:49 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:19:50.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:49 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:19:50.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:49 smithi204 ceph-mon[30933]: osdmap e639: 12 total, 12 up, 11 in 2024-09-16T05:19:50.175 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:454 2.c e127/107 2:33100188:::smithi20457400-19:head v 127'19, mlcod=126'16)", 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 531.47386052299998, 2024-09-16T05:19:50.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054909060000000003, 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.700147+0000", 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7800000000000001e-07 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703134+0000", 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002986954 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703165+0000", 2024-09-16T05:19:50.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1201000000000002e-05 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703173+0000", 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.32e-06 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703255+0000", 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1730000000000005e-05 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.703282+0000", 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7389000000000002e-05 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.705600+0000", 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023176360000000001 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:58.705637+0000", 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7497999999999999e-05 2024-09-16T05:19:50.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:116 2.5 e141/107 2:ad4a88eb:::smithi20457400-16:a v 141'87, mlcod=141'79)", 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 517.47899899399999, 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006093261, 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695009+0000", 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6499999999999999e-07 2024-09-16T05:19:50.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695014+0000", 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8350000000000003e-06 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700654+0000", 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056407499999999999 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700660+0000", 2024-09-16T05:19:50.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4879999999999998e-06 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700700+0000", 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0583000000000001e-05 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700714+0000", 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.322e-05 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701062+0000", 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00034833999999999999 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701101+0000", 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9480000000000001e-05 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:118 2.5 e141/107 2:ad7afe2b:::smithi20457400-38:d v 141'90, mlcod=141'85)", 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:19:50.186 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 517.47240466999995, 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066456359999999999, 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:19:50.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701603+0000", 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701606+0000", 2024-09-16T05:19:50.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3890000000000001e-06 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707421+0000", 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005814601 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.189 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707428+0000", 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4839999999999998e-06 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707497+0000", 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8441000000000004e-05 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707520+0000", 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3513999999999999e-05 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708185+0000", 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066482599999999998 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708248+0000", 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2959000000000005e-05 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.192 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:614 2.5 e144/107 2:a1d8c6d9:::smithi20457400-3:head v 144'110, mlcod=144'106)", 2024-09-16T05:19:50.193 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:19:50.193 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 514.52343890500003, 2024-09-16T05:19:50.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054521470000000001, 2024-09-16T05:19:50.193 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.193 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.194 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:50.194 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.194 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650569+0000", 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7299999999999995e-07 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653581+0000", 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003012617 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653627+0000", 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5649999999999998e-05 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653639+0000", 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2126e-05 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653715+0000", 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5485e-05 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653748+0000", 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3373e-05 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.655964+0000", 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002215875 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.656020+0000", 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6348000000000003e-05 2024-09-16T05:19:50.198 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:172 2.c e179/107 2:3d4bedc8:::smithi20457400-22:26 v 179'101, mlcod=178'97)", 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 479.52128150700003, 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34174038899999998, 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4999999999999999e-07 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652730+0000", 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8789999999999996e-06 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993974+0000", 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34124383699999999 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993980+0000", 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4139999999999999e-06 2024-09-16T05:19:50.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994036+0000", 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5850999999999998e-05 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994054+0000", 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8182999999999999e-05 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994414+0000", 2024-09-16T05:19:50.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036047800000000001 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994466+0000", 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1496999999999998e-05 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:972 2.c e179/107 2:33100188:::smithi20457400-19:head v 179'105, mlcod=179'101)", 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 479.17493760600001, 2024-09-16T05:19:50.203 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.53639116399999998, 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999070+0000", 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0999999999999997e-07 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999087+0000", 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7014999999999999e-05 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534916+0000", 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.53582909000000001 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534923+0000", 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0099999999999998e-06 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534975+0000", 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1938999999999999e-05 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534994+0000", 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8504000000000001e-05 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.535427+0000", 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043298700000000002 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.535461+0000", 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3809000000000001e-05 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:979 2.4 e179/107 2:2ace617f:::smithi20457400-21:head v 179'81, mlcod=178'78)", 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:19:50.208 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 479.16713725, 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076005070000000003, 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:19:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:50.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:50.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009810+0000", 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002939929 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009892+0000", 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2349999999999996e-05 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009916+0000", 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3200000000000001e-05 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009983+0000", 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7577999999999996e-05 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010005+0000", 2024-09-16T05:19:50.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2337999999999999e-05 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014425+0000", 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044196119999999998 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014470+0000", 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5297999999999999e-05 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1017 2.5 e182/107 2:a1d8c6d9:::smithi20457400-3:head v 182'254, mlcod=182'250)", 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 476.47528260199999, 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067341900000000001, 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698726+0000", 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0410000000000001e-06 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703134+0000", 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044088850000000004 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703168+0000", 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3195999999999999e-05 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703175+0000", 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703234+0000", 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8859e-05 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703262+0000", 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7650000000000001e-05 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.705387+0000", 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021252670000000001 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.705459+0000", 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1598000000000001e-05 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1146 2.c e192/107 2:33100188:::smithi20457400-19:head v 192'151, mlcod=192'147)", 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 466.43104776600001, 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057288909999999998, 2024-09-16T05:19:50.219 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742960+0000", 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3399999999999999e-07 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746015+0000", 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030547809999999999 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746040+0000", 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5075999999999998e-05 2024-09-16T05:19:50.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746048+0000", 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9030000000000004e-06 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746099+0000", 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1004999999999997e-05 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746117+0000", 2024-09-16T05:19:50.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8189000000000002e-05 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.748630+0000", 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025129549999999999 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.748688+0000", 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8448e-05 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1161 2.c e192/107 2:3d4bedc8:::smithi20457400-22:head v 192'153, mlcod=192'149)", 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 466.40256221800001, 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055212639999999997, 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771447+0000", 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3360000000000002e-06 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774126+0000", 2024-09-16T05:19:50.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026786790000000002 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774176+0000", 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9669999999999997e-05 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774185+0000", 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4530000000000008e-06 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774260+0000", 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5395999999999994e-05 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774291+0000", 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0561000000000001e-05 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.776739+0000", 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002447968 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.776966+0000", 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022720100000000001 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1201 2.5 e194/183 2:a5bc1608:::smithi20457400-11:head v 194'298, mlcod=192'295)", 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 464.49506424999998, 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080910559999999992, 2024-09-16T05:19:50.228 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.229 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.229 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:19:50.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:19:50.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683786+0000", 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048424469999999997 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683833+0000", 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7540999999999998e-05 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683844+0000", 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1354e-05 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683920+0000", 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5760999999999995e-05 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683940+0000", 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9715999999999999e-05 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.686996+0000", 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003055954 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.687034+0000", 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8031999999999998e-05 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.233 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1890 2.1f 2:fc733d2f:::smithi20457400-45:head [append 0~0] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 400.46331929500002, 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080315490000000007, 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1890 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710691+0000", 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7769999999999998e-06 2024-09-16T05:19:50.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710704+0000", 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.38e-05 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710741+0000", 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6371999999999997e-05 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710758+0000", 2024-09-16T05:19:50.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7637000000000001e-05 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710929+0000", 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017019499999999999 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710935+0000", 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2720000000000003e-06 2024-09-16T05:19:50.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716763+0000", 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058286350000000004 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716836+0000", 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2630000000000001e-05 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716981+0000", 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014495099999999999 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718424+0000", 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014427769999999999 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718586+0000", 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016167699999999999 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718660+0000", 2024-09-16T05:19:50.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4813000000000004e-05 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718719+0000", 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9012999999999999e-05 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1891 2.1f 2:fc733d2f:::smithi20457400-45:head [rollback 39] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 400.46320735, 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082173000000000003, 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1891 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:19:50.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710801+0000", 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6700000000000002e-07 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710809+0000", 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4230000000000005e-06 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710823+0000", 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3833e-05 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710832+0000", 2024-09-16T05:19:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9830000000000002e-06 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710997+0000", 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000165558 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711001+0000", 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0239999999999999e-06 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717531+0000", 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065290699999999997 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717588+0000", 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7497999999999997e-05 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717722+0000", 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134182 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718427+0000", 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070501199999999998 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718936+0000", 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050843700000000004 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718972+0000", 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6102000000000001e-05 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719017+0000", 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5311e-05 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1892 2.1f 2:fc733d2f:::smithi20457400-45:head [append 0~0] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 400.46308096400003, 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082344800000000006, 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1892 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:19:50.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710927+0000", 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9099999999999999e-07 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710930+0000", 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9869999999999999e-06 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710937+0000", 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9940000000000003e-06 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710941+0000", 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0099999999999997e-06 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711036+0000", 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5013999999999995e-05 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711042+0000", 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.224e-06 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718122+0000", 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007080706 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718183+0000", 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0194e-05 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718285+0000", 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102118 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718538+0000", 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025345499999999998 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719083+0000", 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054496199999999996 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719116+0000", 2024-09-16T05:19:50.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3052000000000001e-05 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719161+0000", 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4372999999999997e-05 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2010 2.1f 2:fc733d2f:::smithi20457400-45:head [copy-from ver 319] snapc 54=[54,53,52,4d,4c,49,48,47,45,3e,3a,35,27] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:19:50.256 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 390.28981728100001, 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016119959, 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2010 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884192+0000", 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7069999999999999e-06 2024-09-16T05:19:50.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884200+0000", 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0679999999999997e-06 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884217+0000", 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.772e-05 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884228+0000", 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0455000000000001e-05 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884292+0000", 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4424999999999994e-05 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884357+0000", 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4498999999999994e-05 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.893200+0000", 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088430690000000003 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.896560+0000", 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033602580000000001 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900238+0000", 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036775699999999998 2024-09-16T05:19:50.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900276+0000", 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8906999999999999e-05 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900310+0000", 2024-09-16T05:19:50.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3281000000000002e-05 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2011 2.1f 2:fc733d2f:::smithi20457400-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 390.28975520300003, 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011802536000000001, 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:50.263 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2011 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884253+0000", 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2200000000000004e-07 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884265+0000", 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2048999999999999e-05 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884275+0000", 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9899999999999992e-06 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884281+0000", 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6850000000000004e-06 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884503+0000", 2024-09-16T05:19:50.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022159799999999999 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884535+0000", 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2288e-05 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.895921+0000", 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011386165 2024-09-16T05:19:50.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.895982+0000", 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0906000000000003e-05 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.896054+0000", 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2232999999999998e-05 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(2.31 277/276 [PushOp(2:8c1ba486:::smithi20457400-2:45, version: 259'110, data_included: [480733~537415,1782380~511161], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:8c1ba486:::smithi20457400-2:45@259'110, size: 3680733, copy_subset: [(0, 3680733)], clone_subset: {2:8c1ba486:::smithi20457400-2:2c: []}, snapset: 4c=[]:{2c: [27], 45: [45, 3e], 4c: [4c, 49, 48]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2293541, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 381.46907955900002, 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45568901499999998, 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:19:50.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704929+0000", 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1000000000000001e-06 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.705928+0000", 2024-09-16T05:19:50.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0009990349999999999 2024-09-16T05:19:50.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:49 smithi196 ceph-mon[30802]: pgmap v1119: 107 pgs: 107 active+clean; 239 MiB data, 770 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 21 op/s 2024-09-16T05:19:50.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:49 smithi196 ceph-mon[30802]: 2.4f scrub starts 2024-09-16T05:19:50.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:49 smithi196 ceph-mon[30802]: 2.4f scrub ok 2024-09-16T05:19:50.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:49 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:19:50.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:49 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:19:50.272 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:49 smithi196 ceph-mon[30802]: osdmap e639: 12 total, 12 up, 11 in 2024-09-16T05:19:50.272 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:50.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.159934+0000", 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45400632299999999 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.159949+0000", 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4608e-05 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160041+0000", 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2523999999999997e-05 2024-09-16T05:19:50.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160053+0000", 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1525999999999999e-05 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160617+0000", 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056389899999999998 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2536 2.31 2:8c1ba486:::smithi20457400-2:head [write 551362~726078] snapc 6d=[6d,6c,6b,6a,69,67,66,64,63,62,60,5d,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 307.23282280199999, 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055342209999999998, 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2536 2024-09-16T05:19:50.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941188+0000", 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3469999999999999e-06 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942118+0000", 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092990300000000004 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942150+0000", 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2888000000000001e-05 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942164+0000", 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3920000000000001e-05 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942258+0000", 2024-09-16T05:19:50.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3839999999999996e-05 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942266+0000", 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4139999999999997e-06 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.943927+0000", 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016610450000000001 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.944033+0000", 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105979 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.944207+0000", 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000174105 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:50.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.945786+0000", 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001579352 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946595+0000", 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080861300000000002 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946660+0000", 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4906000000000005e-05 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946719+0000", 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8909000000000001e-05 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:569 2.c e357/247 2:33100188:::smithi20457400-19:6a v 357'229, mlcod=351'225)", 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 283.46839347899999, 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14687246600000001, 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9599999999999999e-07 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705621+0000", 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7150000000000001e-06 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851933+0000", 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14631169599999999 2024-09-16T05:19:50.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851940+0000", 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.221e-06 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851995+0000", 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5473999999999998e-05 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852016+0000", 2024-09-16T05:19:50.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0965999999999999e-05 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852452+0000", 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043602700000000002 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852486+0000", 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3871000000000001e-05 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.286 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.31,GRANT,e:542,from:10,reservation_nonce:0)", 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 97.157764255999993, 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.69991632400000003, 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:50.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016244+0000", 2024-09-16T05:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7400000000000002e-07 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016249+0000", 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0259999999999998e-06 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.715937+0000", 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.69968885199999997 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:50.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.715950+0000", 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914999999999999e-05 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716018+0000", 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7972000000000006e-05 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716036+0000", 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7302000000000001e-05 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:50.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716159+0000", 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000123483 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:50.291 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:19:50.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:50.335 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:50.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:50.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:50.459 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:50.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:50.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:50.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:50.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:50.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:50.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:50.785 INFO:tasks.rados.rados.2.smithi204.stdout:2480: done (1 left) 2024-09-16T05:19:50.785 INFO:tasks.rados.rados.2.smithi204.stdout:2481: snap_remove snap 254 2024-09-16T05:19:50.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:50.837 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:50.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:50.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:50.963 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:50.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:50.983 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:50.983 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:51.004 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:51.005 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:51.030 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:51.030 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:51.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:19:51.051 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:51.051 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:51.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:51.075 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:51.075 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: osdmap e640: 12 total, 12 up, 11 in 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:19:51.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:50 smithi032 ceph-mon[27388]: osdmap e641: 12 total, 12 up, 11 in 2024-09-16T05:19:51.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:51.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:51.097 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:51.097 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:51.119 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:51.140 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:19:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:19:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:19:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:19:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: osdmap e640: 12 total, 12 up, 11 in 2024-09-16T05:19:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:19:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:19:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:19:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:19:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:50 smithi204 ceph-mon[30933]: osdmap e641: 12 total, 12 up, 11 in 2024-09-16T05:19:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:51.184 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:51.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:51.207 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:51.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:51.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:51.235 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:19:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:19:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:19:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:19:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: osdmap e640: 12 total, 12 up, 11 in 2024-09-16T05:19:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:19:51.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:19:51.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:19:51.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:19:51.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:50 smithi196 ceph-mon[30802]: osdmap e641: 12 total, 12 up, 11 in 2024-09-16T05:19:51.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:51.341 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:51.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:51.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:51.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:51.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:51.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:19:51.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:51.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:51.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:51.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:51.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:51.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:51.790 INFO:tasks.rados.rados.2.smithi204.stdout:2481: done (1 left) 2024-09-16T05:19:51.790 INFO:tasks.rados.rados.2.smithi204.stdout:2482: copy_from oid 8 from oid 21 current snap is 259 2024-09-16T05:19:51.790 INFO:tasks.rados.rados.2.smithi204.stdout:2479: finishing write tid 1 to smithi20457400-38 2024-09-16T05:19:51.790 INFO:tasks.rados.rados.2.smithi204.stdout:2479: finishing write tid 2 to smithi20457400-38 2024-09-16T05:19:51.790 INFO:tasks.rados.rados.2.smithi204.stdout:2479: finishing write tid 3 to smithi20457400-38 2024-09-16T05:19:51.790 INFO:tasks.rados.rados.2.smithi204.stdout:2479: finishing write tid 4 to smithi20457400-38 2024-09-16T05:19:51.791 INFO:tasks.rados.rados.2.smithi204.stdout:2479: finishing write tid 5 to smithi20457400-38 2024-09-16T05:19:51.791 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 897 (ObjNum 884 snap 259 seq_num 884) dirty exists 2024-09-16T05:19:51.791 INFO:tasks.rados.rados.2.smithi204.stdout:2479: left oid 38 (ObjNum 884 snap 259 seq_num 884) 2024-09-16T05:19:51.791 INFO:tasks.rados.rados.2.smithi204.stdout:2479: done (1 left) 2024-09-16T05:19:51.791 INFO:tasks.rados.rados.2.smithi204.stdout:2483: snap_create 2024-09-16T05:19:51.792 INFO:tasks.rados.rados.2.smithi204.stdout:2482: finishing copy_from to smithi20457400-8 2024-09-16T05:19:51.792 INFO:tasks.rados.rados.2.smithi204.stdout:2482: got expected ENOENT (src dne) 2024-09-16T05:19:51.792 INFO:tasks.rados.rados.2.smithi204.stdout:2482: finishing copy_from racing read to smithi20457400-8 2024-09-16T05:19:51.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:51.849 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:51.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:51.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:51.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:51.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:52.056 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: pgmap v1122: 107 pgs: 107 active+clean; 246 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 18 op/s 2024-09-16T05:19:52.056 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: 2.4e scrub starts 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: 2.4e scrub ok 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2307272178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:19:52.057 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:51 smithi032 ceph-mon[27388]: osdmap e642: 12 total, 12 up, 11 in 2024-09-16T05:19:52.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:52.101 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:52.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:52.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:52.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: pgmap v1122: 107 pgs: 107 active+clean; 246 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 18 op/s 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: 2.4e scrub starts 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: 2.4e scrub ok 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2307272178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:19:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:19:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:19:52.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:51 smithi204 ceph-mon[30933]: osdmap e642: 12 total, 12 up, 11 in 2024-09-16T05:19:52.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:52.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:52.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: pgmap v1122: 107 pgs: 107 active+clean; 246 MiB data, 772 MiB used, 983 GiB / 983 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 18 op/s 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: 2.4e scrub starts 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: 2.4e scrub ok 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2307272178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:19:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:19:52.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:51 smithi196 ceph-mon[30802]: osdmap e642: 12 total, 12 up, 11 in 2024-09-16T05:19:52.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:52.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:52.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:52.382 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:52.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:52.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:52.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:52.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:52.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:52.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:52.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:52.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:52.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2482: done (1 left) 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2483: done (0 left) 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2484: read oid 7 snap -1 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2484: expect (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2485: read oid 20 snap -1 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2485: expect (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2486: read oid 30 snap 248 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2486: expect (ObjNum 681 snap 198 seq_num 681) 2024-09-16T05:19:52.788 INFO:tasks.rados.rados.2.smithi204.stdout:2487: snap_create 2024-09-16T05:19:52.794 INFO:tasks.rados.rados.2.smithi204.stdout:2485: expect (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:19:52.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:19:52.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:52.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:19:52.958 INFO:tasks.rados.rados.2.smithi204.stdout:2486: expect (ObjNum 681 snap 198 seq_num 681) 2024-09-16T05:19:52.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:52.988 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:52.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:53.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:53.106 INFO:tasks.rados.rados.2.smithi204.stdout:2484: expect (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:19:53.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:53.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:53.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:52 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:19:53.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:52 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:19:53.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3643365859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:19:53.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3643365859' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:19:53.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:52 smithi032 ceph-mon[27388]: osdmap e643: 12 total, 12 up, 11 in 2024-09-16T05:19:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:52 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:19:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:52 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:19:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3643365859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:19:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3643365859' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:19:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:52 smithi204 ceph-mon[30933]: osdmap e643: 12 total, 12 up, 11 in 2024-09-16T05:19:53.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:53.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:52 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:19:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:52 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:19:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3643365859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:19:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3643365859' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:19:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:52 smithi196 ceph-mon[30802]: osdmap e643: 12 total, 12 up, 11 in 2024-09-16T05:19:53.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:53.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:53.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:53.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:53.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:53.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:53.490 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:53.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:53.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:53.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:53.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:53.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:19:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:53.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:19:53.788 INFO:tasks.rados.rados.2.smithi204.stdout:2484: done (3 left) 2024-09-16T05:19:53.789 INFO:tasks.rados.rados.2.smithi204.stdout:2485: done (2 left) 2024-09-16T05:19:53.789 INFO:tasks.rados.rados.2.smithi204.stdout:2486: done (1 left) 2024-09-16T05:19:53.789 INFO:tasks.rados.rados.2.smithi204.stdout:2487: done (0 left) 2024-09-16T05:19:53.789 INFO:tasks.rados.rados.2.smithi204.stdout:2488: rollback oid 25 current snap is 261 2024-09-16T05:19:53.789 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 229 2024-09-16T05:19:53.790 INFO:tasks.rados.rados.2.smithi204.stdout:2489: snap_remove snap 244 2024-09-16T05:19:53.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:53.863 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:53.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:53.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:53 smithi032 ceph-mon[27388]: pgmap v1125: 107 pgs: 107 active+clean; 245 MiB data, 786 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-16T05:19:53.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:53 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:19:53.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:53 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:19:53.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:53 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:19:53.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:53 smithi032 ceph-mon[27388]: osdmap e644: 12 total, 12 up, 11 in 2024-09-16T05:19:53.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:53.988 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:53.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:54.013 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:19:54.013 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:19:54.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:54.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:54.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:53 smithi204 ceph-mon[30933]: pgmap v1125: 107 pgs: 107 active+clean; 245 MiB data, 786 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-16T05:19:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:53 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:19:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:53 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:19:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:53 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:19:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:53 smithi204 ceph-mon[30933]: osdmap e644: 12 total, 12 up, 11 in 2024-09-16T05:19:54.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:53 smithi196 ceph-mon[30802]: pgmap v1125: 107 pgs: 107 active+clean; 245 MiB data, 786 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2024-09-16T05:19:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:53 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:19:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:53 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:19:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:53 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:19:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:53 smithi196 ceph-mon[30802]: osdmap e644: 12 total, 12 up, 11 in 2024-09-16T05:19:54.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:54.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:54.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:54.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:54.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:54.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:54.492 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:54.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:54.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:54.615 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:54.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:54.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:19:54.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:54.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:54.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:54.793 INFO:tasks.rados.rados.2.smithi204.stdout:2489: done (1 left) 2024-09-16T05:19:54.794 INFO:tasks.rados.rados.2.smithi204.stdout:2490: rollback oid 37 current snap is 261 2024-09-16T05:19:54.794 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 257 2024-09-16T05:19:54.794 INFO:tasks.rados.rados.2.smithi204.stdout:2491: snap_create 2024-09-16T05:19:54.794 INFO:tasks.rados.rados.2.smithi204.stdout:2488: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:19:54.794 INFO:tasks.rados.rados.2.smithi204.stdout:2488: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:19:54.794 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 732 (ObjNum 49 snap 0 seq_num 2672633136) dirty dne 2024-09-16T05:19:54.798 INFO:tasks.rados.rados.2.smithi204.stdout:2490: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:19:54.798 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1016 (ObjNum 145 snap 0 seq_num 4283302480) dirty dne 2024-09-16T05:19:54.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T05:19:54.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:19:54.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:54.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:54.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:54.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:54 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:19:54.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:54 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:19:54.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:54 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:19:54.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:54 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:19:54.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:54 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:19:54.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:54 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:19:54.934 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:54 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:19:54.934 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:54 smithi032 ceph-mon[27388]: osdmap e645: 12 total, 12 up, 11 in 2024-09-16T05:19:54.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:55.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:55.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:55.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:55.153 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:55.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:54 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:19:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:54 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:19:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:54 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:19:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:54 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:19:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:54 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:19:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:54 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:19:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:54 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:19:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:54 smithi204 ceph-mon[30933]: osdmap e645: 12 total, 12 up, 11 in 2024-09-16T05:19:55.213 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:55.213 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:54 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:19:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:54 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:19:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:54 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:19:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:54 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:19:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:54 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:19:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:54 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:19:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:54 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:19:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:54 smithi196 ceph-mon[30802]: osdmap e645: 12 total, 12 up, 11 in 2024-09-16T05:19:55.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:55.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:55.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:55.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:55.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:55.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:55.491 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:55.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:55.530 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:55.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:55.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:55.657 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:55.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:55.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:55.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:55.799 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:19:55.803 INFO:tasks.rados.rados.2.smithi204.stdout:2488: done (2 left) 2024-09-16T05:19:55.803 INFO:tasks.rados.rados.2.smithi204.stdout:2490: done (1 left) 2024-09-16T05:19:55.803 INFO:tasks.rados.rados.2.smithi204.stdout:2491: done (0 left) 2024-09-16T05:19:55.803 INFO:tasks.rados.rados.2.smithi204.stdout:2492: copy_from oid 42 from oid 22 current snap is 262 2024-09-16T05:19:55.803 INFO:tasks.rados.rados.2.smithi204.stdout:2493: snap_remove snap 259 2024-09-16T05:19:55.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:55.906 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:55.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:55 smithi032 ceph-mon[27388]: pgmap v1128: 107 pgs: 107 active+clean; 250 MiB data, 795 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-16T05:19:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/886023279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:55 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:55 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:19:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/886023279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:55 smithi032 ceph-mon[27388]: osdmap e646: 12 total, 12 up, 11 in 2024-09-16T05:19:56.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:56.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:56.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:56.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:56.161 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:56.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:56.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:55 smithi204 ceph-mon[30933]: pgmap v1128: 107 pgs: 107 active+clean; 250 MiB data, 795 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-16T05:19:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/886023279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:55 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:55 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:19:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/886023279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:55 smithi204 ceph-mon[30933]: osdmap e646: 12 total, 12 up, 11 in 2024-09-16T05:19:56.176 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:19:56.176 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:19:56.197 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:19:56.198 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:19:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:55 smithi196 ceph-mon[30802]: pgmap v1128: 107 pgs: 107 active+clean; 250 MiB data, 795 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 17 op/s 2024-09-16T05:19:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/886023279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:19:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:55 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:19:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:55 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:19:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/886023279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:19:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:55 smithi196 ceph-mon[30802]: osdmap e646: 12 total, 12 up, 11 in 2024-09-16T05:19:56.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:56.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:56.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:56.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:56.418 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:56.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:56.420 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:19:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:19:56.424 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:19:56.425 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:19:56.425 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:19:56.425 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:19:56.425 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:19:56.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:19:56.451 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:19:56.473 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:19:56.473 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:19:56.495 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:19:56.495 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:19:56.516 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:19:56.517 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:19:56.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:56.538 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:19:56.539 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:19:56.541 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:56.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:56.560 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:19:56.560 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:19:56.582 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:19:56.583 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:19:56.604 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:19:56.605 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:19:56.626 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:19:56.626 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:19:56.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:56.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:56.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:56.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:56.797 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:56.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:2493: done (1 left) 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:2494: read oid 9 snap -1 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:2494: expect (ObjNum 866 snap 254 seq_num 866) 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:2492: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:2492: finishing copy_from to smithi20457400-42 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 630 (ObjNum 848 snap 246 seq_num 848) dirty exists 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:2495: read oid 20 snap -1 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:2495: expect (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:19:56.805 INFO:tasks.rados.rados.2.smithi204.stdout:2492: done (2 left) 2024-09-16T05:19:56.806 INFO:tasks.rados.rados.2.smithi204.stdout:2496: snap_remove snap 257 2024-09-16T05:19:56.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:56.922 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:56.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:57.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:57.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:57.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:19:57.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:57.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:56 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:19:57.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:56 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:19:57.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:56 smithi032 ceph-mon[27388]: 2.3b scrub starts 2024-09-16T05:19:57.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:56 smithi032 ceph-mon[27388]: 2.3b scrub ok 2024-09-16T05:19:57.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2800750343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:57.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:56 smithi032 ceph-mon[27388]: osdmap e647: 12 total, 12 up, 11 in 2024-09-16T05:19:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:56 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:19:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:56 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:19:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:56 smithi204 ceph-mon[30933]: 2.3b scrub starts 2024-09-16T05:19:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:56 smithi204 ceph-mon[30933]: 2.3b scrub ok 2024-09-16T05:19:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2800750343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:56 smithi204 ceph-mon[30933]: osdmap e647: 12 total, 12 up, 11 in 2024-09-16T05:19:57.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:57.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:56 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:19:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:56 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:19:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:56 smithi196 ceph-mon[30802]: 2.3b scrub starts 2024-09-16T05:19:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:56 smithi196 ceph-mon[30802]: 2.3b scrub ok 2024-09-16T05:19:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2800750343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:19:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:56 smithi196 ceph-mon[30802]: osdmap e647: 12 total, 12 up, 11 in 2024-09-16T05:19:57.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:57.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:57.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:57.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:57.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:57.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:57.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:57.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:57.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:57.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:57.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:57.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:57.792 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:57.806 INFO:tasks.rados.rados.2.smithi204.stdout:2496: done (2 left) 2024-09-16T05:19:57.806 INFO:tasks.rados.rados.2.smithi204.stdout:2497: write (excl) oid 23 current snap is 262 2024-09-16T05:19:57.806 INFO:tasks.rados.rados.2.smithi204.stdout:2497: seq_num 885 ranges {474195=492313,1604917=660295,2670525=203670} 2024-09-16T05:19:57.811 INFO:tasks.rados.rados.2.smithi204.stdout:2497: writing smithi20457400-23 from 474195 to 966508 tid 1 2024-09-16T05:19:57.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:19:57.818 INFO:tasks.rados.rados.2.smithi204.stdout:2497: writing smithi20457400-23 from 1604917 to 2265212 tid 2 2024-09-16T05:19:57.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:57.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:57.822 INFO:tasks.rados.rados.2.smithi204.stdout:2497: writing smithi20457400-23 from 2670525 to 2874195 tid 3 2024-09-16T05:19:57.822 INFO:tasks.rados.rados.2.smithi204.stdout:2498: rollback oid 1 current snap is 262 2024-09-16T05:19:57.822 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 1 to 258 2024-09-16T05:19:57.822 INFO:tasks.rados.rados.2.smithi204.stdout:2495: expect (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:19:57.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:57.943 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:57.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:57.986 INFO:tasks.rados.rados.2.smithi204.stdout:2494: expect (ObjNum 866 snap 254 seq_num 866) 2024-09-16T05:19:58.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:19:58.068 INFO:tasks.rados.rados.2.smithi204.stdout:2497: finishing write tid 1 to smithi20457400-23 2024-09-16T05:19:58.068 INFO:tasks.rados.rados.2.smithi204.stdout:2497: finishing write tid 2 to smithi20457400-23 2024-09-16T05:19:58.068 INFO:tasks.rados.rados.2.smithi204.stdout:2497: finishing write tid 3 to smithi20457400-23 2024-09-16T05:19:58.068 INFO:tasks.rados.rados.2.smithi204.stdout:2497: finishing write tid 4 to smithi20457400-23 2024-09-16T05:19:58.068 INFO:tasks.rados.rados.2.smithi204.stdout:2497: finishing write tid 5 to smithi20457400-23 2024-09-16T05:19:58.068 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 717 (ObjNum 885 snap 262 seq_num 885) dirty exists 2024-09-16T05:19:58.068 INFO:tasks.rados.rados.2.smithi204.stdout:2497: left oid 23 (ObjNum 885 snap 262 seq_num 885) 2024-09-16T05:19:58.068 INFO:tasks.rados.rados.2.smithi204.stdout:2498: finishing rollback tid 0 to smithi20457400-1 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2498: finishing rollback tid 1 to smithi20457400-1 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2498: finishing rollback tid 2 to smithi20457400-1 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 548 (ObjNum 842 snap 243 seq_num 842) dirty exists 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2494: done (3 left) 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2495: done (2 left) 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2497: done (1 left) 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2498: done (0 left) 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2499: copy_from oid 15 from oid 22 current snap is 262 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2500: write (excl) oid 20 current snap is 262 2024-09-16T05:19:58.069 INFO:tasks.rados.rados.2.smithi204.stdout:2500: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-09-16T05:19:58.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:58.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:19:58.074 INFO:tasks.rados.rados.2.smithi204.stdout:2500: writing smithi20457400-20 from 491002 to 1058564 tid 1 2024-09-16T05:19:58.080 INFO:tasks.rados.rados.2.smithi204.stdout:2500: writing smithi20457400-20 from 1663399 to 2183705 tid 2 2024-09-16T05:19:58.084 INFO:tasks.rados.rados.2.smithi204.stdout:2500: writing smithi20457400-20 from 2814301 to 2891002 tid 3 2024-09-16T05:19:58.085 INFO:tasks.rados.rados.2.smithi204.stdout:2501: read oid 4 snap -1 2024-09-16T05:19:58.085 INFO:tasks.rados.rados.2.smithi204.stdout:2501: expect deleted 2024-09-16T05:19:58.085 INFO:tasks.rados.rados.2.smithi204.stdout:2502: write oid 11 current snap is 262 2024-09-16T05:19:58.085 INFO:tasks.rados.rados.2.smithi204.stdout:2499: finishing copy_from racing read to smithi20457400-15 2024-09-16T05:19:58.085 INFO:tasks.rados.rados.2.smithi204.stdout:2502: seq_num 887 ranges {507809=642811,1721881=663964} 2024-09-16T05:19:58.090 INFO:tasks.rados.rados.2.smithi204.stdout:2502: writing smithi20457400-11 from 507809 to 1150620 tid 1 2024-09-16T05:19:58.098 INFO:tasks.rados.rados.2.smithi204.stdout:2502: writing smithi20457400-11 from 1721881 to 2385845 tid 2 2024-09-16T05:19:58.098 INFO:tasks.rados.rados.2.smithi204.stdout:2503: rollback oid 50 current snap is 262 2024-09-16T05:19:58.098 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 247 2024-09-16T05:19:58.098 INFO:tasks.rados.rados.2.smithi204.stdout:2500: finishing write tid 1 to smithi20457400-20 2024-09-16T05:19:58.098 INFO:tasks.rados.rados.2.smithi204.stdout:2499: finishing copy_from to smithi20457400-15 2024-09-16T05:19:58.098 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 630 (ObjNum 848 snap 246 seq_num 848) dirty exists 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2500: finishing write tid 2 to smithi20457400-20 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2500: finishing write tid 3 to smithi20457400-20 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2500: finishing write tid 4 to smithi20457400-20 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2500: finishing write tid 5 to smithi20457400-20 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1026 (ObjNum 886 snap 262 seq_num 886) dirty exists 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2500: left oid 20 (ObjNum 886 snap 262 seq_num 886) 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2502: finishing write tid 1 to smithi20457400-11 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2499: done (4 left) 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2500: done (3 left) 2024-09-16T05:19:58.099 INFO:tasks.rados.rados.2.smithi204.stdout:2504: rollback oid 36 current snap is 262 2024-09-16T05:19:58.100 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 36 to 229 2024-09-16T05:19:58.100 INFO:tasks.rados.rados.2.smithi204.stdout:2505: read oid 18 snap -1 2024-09-16T05:19:58.100 INFO:tasks.rados.rados.2.smithi204.stdout:2505: expect deleted 2024-09-16T05:19:58.100 INFO:tasks.rados.rados.2.smithi204.stdout:2506: write (excl) oid 39 current snap is 262 2024-09-16T05:19:58.100 INFO:tasks.rados.rados.2.smithi204.stdout:2506: seq_num 888 ranges {524616=718060,1664010=523975,2752794=171822} 2024-09-16T05:19:58.104 INFO:tasks.rados.rados.2.smithi204.stdout:2506: writing smithi20457400-39 from 524616 to 1242676 tid 1 2024-09-16T05:19:58.109 INFO:tasks.rados.rados.2.smithi204.stdout:2506: writing smithi20457400-39 from 1664010 to 2187985 tid 2 2024-09-16T05:19:58.113 INFO:tasks.rados.rados.2.smithi204.stdout:2506: writing smithi20457400-39 from 2752794 to 2924616 tid 3 2024-09-16T05:19:58.113 INFO:tasks.rados.rados.2.smithi204.stdout:2507: write oid 33 current snap is 262 2024-09-16T05:19:58.113 INFO:tasks.rados.rados.2.smithi204.stdout:2507: seq_num 889 ranges {541423=793309,2122492=667633} 2024-09-16T05:19:58.120 INFO:tasks.rados.rados.2.smithi204.stdout:2507: writing smithi20457400-33 from 541423 to 1334732 tid 1 2024-09-16T05:19:58.130 INFO:tasks.rados.rados.2.smithi204.stdout:2507: writing smithi20457400-33 from 2122492 to 2790125 tid 2 2024-09-16T05:19:58.130 INFO:tasks.rados.rados.2.smithi204.stdout:2508: copy_from oid 17 from oid 14 current snap is 262 2024-09-16T05:19:58.130 INFO:tasks.rados.rados.2.smithi204.stdout:2502: finishing write tid 2 to smithi20457400-11 2024-09-16T05:19:58.130 INFO:tasks.rados.rados.2.smithi204.stdout:2503: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:19:58.130 INFO:tasks.rados.rados.2.smithi204.stdout:2502: finishing write tid 3 to smithi20457400-11 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2504: finishing rollback tid 0 to smithi20457400-36 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2503: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2502: finishing write tid 4 to smithi20457400-11 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1030 (ObjNum 887 snap 262 seq_num 887) dirty exists 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2502: left oid 11 (ObjNum 887 snap 262 seq_num 887) 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2503: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1099 (ObjNum 838 snap 243 seq_num 838) dirty exists 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2504: finishing rollback tid 1 to smithi20457400-36 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2501: done (7 left) 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2502: done (6 left) 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2503: done (5 left) 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2505: done (4 left) 2024-09-16T05:19:58.131 INFO:tasks.rados.rados.2.smithi204.stdout:2509: read oid 3 snap -1 2024-09-16T05:19:58.132 INFO:tasks.rados.rados.2.smithi204.stdout:2509: expect (ObjNum 875 snap 255 seq_num 875) 2024-09-16T05:19:58.132 INFO:tasks.rados.rados.2.smithi204.stdout:2510: write oid 26 current snap is 262 2024-09-16T05:19:58.132 INFO:tasks.rados.rados.2.smithi204.stdout:2506: finishing write tid 1 to smithi20457400-39 2024-09-16T05:19:58.132 INFO:tasks.rados.rados.2.smithi204.stdout:2506: finishing write tid 2 to smithi20457400-39 2024-09-16T05:19:58.132 INFO:tasks.rados.rados.2.smithi204.stdout:2510: seq_num 890 ranges {558230=584911,1897327=527644,2923993=34237} 2024-09-16T05:19:58.136 INFO:tasks.rados.rados.2.smithi204.stdout:2510: writing smithi20457400-26 from 558230 to 1143141 tid 1 2024-09-16T05:19:58.143 INFO:tasks.rados.rados.2.smithi204.stdout:2510: writing smithi20457400-26 from 1897327 to 2424971 tid 2 2024-09-16T05:19:58.146 INFO:tasks.rados.rados.2.smithi204.stdout:2510: writing smithi20457400-26 from 2923993 to 2958230 tid 3 2024-09-16T05:19:58.146 INFO:tasks.rados.rados.2.smithi204.stdout:2511: write (excl) oid 15 current snap is 262 2024-09-16T05:19:58.146 INFO:tasks.rados.rados.2.smithi204.stdout:2506: finishing write tid 3 to smithi20457400-39 2024-09-16T05:19:58.146 INFO:tasks.rados.rados.2.smithi204.stdout:2506: finishing write tid 4 to smithi20457400-39 2024-09-16T05:19:58.146 INFO:tasks.rados.rados.2.smithi204.stdout:2506: finishing write tid 5 to smithi20457400-39 2024-09-16T05:19:58.147 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1034 (ObjNum 888 snap 262 seq_num 888) dirty exists 2024-09-16T05:19:58.147 INFO:tasks.rados.rados.2.smithi204.stdout:2506: left oid 39 (ObjNum 888 snap 262 seq_num 888) 2024-09-16T05:19:58.147 INFO:tasks.rados.rados.2.smithi204.stdout:2507: finishing write tid 1 to smithi20457400-33 2024-09-16T05:19:58.147 INFO:tasks.rados.rados.2.smithi204.stdout:2511: seq_num 891 ranges {575037=660160,1955809=671302} 2024-09-16T05:19:58.152 INFO:tasks.rados.rados.2.smithi204.stdout:2511: writing smithi20457400-15 from 575037 to 1235197 tid 1 2024-09-16T05:19:58.160 INFO:tasks.rados.rados.2.smithi204.stdout:2511: writing smithi20457400-15 from 1955809 to 2627111 tid 2 2024-09-16T05:19:58.160 INFO:tasks.rados.rados.2.smithi204.stdout:2506: done (6 left) 2024-09-16T05:19:58.161 INFO:tasks.rados.rados.2.smithi204.stdout:2512: delete oid 42 current snap is 262 2024-09-16T05:19:58.161 INFO:tasks.rados.rados.2.smithi204.stdout:2509: expect (ObjNum 875 snap 255 seq_num 875) 2024-09-16T05:19:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:57 smithi032 ceph-mon[27388]: pgmap v1131: 107 pgs: 107 active+clean; 250 MiB data, 796 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-16T05:19:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:57 smithi032 ceph-mon[27388]: 1.2 deep-scrub starts 2024-09-16T05:19:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:57 smithi032 ceph-mon[27388]: 1.2 deep-scrub ok 2024-09-16T05:19:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:57 smithi032 ceph-mon[27388]: osdmap e648: 12 total, 12 up, 11 in 2024-09-16T05:19:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:57 smithi204 ceph-mon[30933]: pgmap v1131: 107 pgs: 107 active+clean; 250 MiB data, 796 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-16T05:19:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:57 smithi204 ceph-mon[30933]: 1.2 deep-scrub starts 2024-09-16T05:19:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:57 smithi204 ceph-mon[30933]: 1.2 deep-scrub ok 2024-09-16T05:19:58.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:57 smithi204 ceph-mon[30933]: osdmap e648: 12 total, 12 up, 11 in 2024-09-16T05:19:58.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:58.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:58.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:57 smithi196 ceph-mon[30802]: pgmap v1131: 107 pgs: 107 active+clean; 250 MiB data, 796 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-16T05:19:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:57 smithi196 ceph-mon[30802]: 1.2 deep-scrub starts 2024-09-16T05:19:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:57 smithi196 ceph-mon[30802]: 1.2 deep-scrub ok 2024-09-16T05:19:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:57 smithi196 ceph-mon[30802]: osdmap e648: 12 total, 12 up, 11 in 2024-09-16T05:19:58.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:19:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:58.315 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:58.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:19:58.335 INFO:tasks.rados.rados.2.smithi204.stdout:2507: finishing write tid 2 to smithi20457400-33 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2507: finishing write tid 3 to smithi20457400-33 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2509: done (6 left) 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2512: done (5 left) 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2513: delete oid 30 current snap is 262 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2507: finishing write tid 4 to smithi20457400-33 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 847 (ObjNum 889 snap 262 seq_num 889) dirty exists 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2507: left oid 33 (ObjNum 889 snap 262 seq_num 889) 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2510: finishing write tid 1 to smithi20457400-26 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2508: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2508: finishing copy_from to smithi20457400-17 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1083 (ObjNum 846 snap 246 seq_num 846) dirty exists 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2510: finishing write tid 2 to smithi20457400-26 2024-09-16T05:19:58.336 INFO:tasks.rados.rados.2.smithi204.stdout:2510: finishing write tid 3 to smithi20457400-26 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:2510: finishing write tid 4 to smithi20457400-26 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:2510: finishing write tid 5 to smithi20457400-26 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1143 (ObjNum 890 snap 262 seq_num 890) dirty exists 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:2510: left oid 26 (ObjNum 890 snap 262 seq_num 890) 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:2504: finishing rollback tid 2 to smithi20457400-36 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 761 (ObjNum 759 snap 221 seq_num 759) dirty exists 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:2511: finishing write tid 1 to smithi20457400-15 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:2511: finishing write tid 2 to smithi20457400-15 2024-09-16T05:19:58.337 INFO:tasks.rados.rados.2.smithi204.stdout:2511: finishing write tid 3 to smithi20457400-15 2024-09-16T05:19:58.338 INFO:tasks.rados.rados.2.smithi204.stdout:2504: done (5 left) 2024-09-16T05:19:58.338 INFO:tasks.rados.rados.2.smithi204.stdout:2507: done (4 left) 2024-09-16T05:19:58.338 INFO:tasks.rados.rados.2.smithi204.stdout:2508: done (3 left) 2024-09-16T05:19:58.338 INFO:tasks.rados.rados.2.smithi204.stdout:2510: done (2 left) 2024-09-16T05:19:58.338 INFO:tasks.rados.rados.2.smithi204.stdout:2513: done (1 left) 2024-09-16T05:19:58.338 INFO:tasks.rados.rados.2.smithi204.stdout:2514: read oid 42 snap -1 2024-09-16T05:19:58.338 INFO:tasks.rados.rados.2.smithi204.stdout:2514: expect deleted 2024-09-16T05:19:58.338 INFO:tasks.rados.rados.2.smithi204.stdout:2515: snap_remove snap 250 2024-09-16T05:19:58.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:58.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:58.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:58.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:58.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:58.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:58.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:19:58.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:58.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:19:58.713 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:19:58.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:58.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:58.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:58.851 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T05:19:58.855 INFO:tasks.rados.rados.2.smithi204.stdout:2515: done (2 left) 2024-09-16T05:19:58.855 INFO:tasks.rados.rados.2.smithi204.stdout:2516: write (excl) oid 10 current snap is 262 2024-09-16T05:19:58.855 INFO:tasks.rados.rados.2.smithi204.stdout:2516: seq_num 894 ranges {625458=485907,1614902=534982,2917332=108126} 2024-09-16T05:19:58.862 INFO:tasks.rados.rados.2.smithi204.stdout:2516: writing smithi20457400-10 from 625458 to 1111365 tid 1 2024-09-16T05:19:58.868 INFO:tasks.rados.rados.2.smithi204.stdout:2516: writing smithi20457400-10 from 1614902 to 2149884 tid 2 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:2516: writing smithi20457400-10 from 2917332 to 3025458 tid 3 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:2517: copy_from oid 48 from oid 44 current snap is 262 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:2511: finishing write tid 4 to smithi20457400-15 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 709 (ObjNum 891 snap 262 seq_num 891) dirty exists 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:2511: left oid 15 (ObjNum 891 snap 262 seq_num 891) 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:2511: done (3 left) 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:2514: done (2 left) 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:2518: write (excl) oid 46 current snap is 262 2024-09-16T05:19:58.874 INFO:tasks.rados.rados.2.smithi204.stdout:2518: seq_num 895 ranges {642265=561156,1673384=678640,2828402=213863} 2024-09-16T05:19:58.880 INFO:tasks.rados.rados.2.smithi204.stdout:2518: writing smithi20457400-46 from 642265 to 1203421 tid 1 2024-09-16T05:19:58.886 INFO:tasks.rados.rados.2.smithi204.stdout:2518: writing smithi20457400-46 from 1673384 to 2352024 tid 2 2024-09-16T05:19:58.890 INFO:tasks.rados.rados.2.smithi204.stdout:2518: writing smithi20457400-46 from 2828402 to 3042265 tid 3 2024-09-16T05:19:58.890 INFO:tasks.rados.rados.2.smithi204.stdout:2519: rollback oid 37 current snap is 262 2024-09-16T05:19:58.890 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 229 2024-09-16T05:19:58.890 INFO:tasks.rados.rados.2.smithi204.stdout:2517: finishing copy_from to smithi20457400-48 2024-09-16T05:19:58.890 INFO:tasks.rados.rados.2.smithi204.stdout:2517: got expected ENOENT (src dne) 2024-09-16T05:19:58.890 INFO:tasks.rados.rados.2.smithi204.stdout:2517: finishing copy_from racing read to smithi20457400-48 2024-09-16T05:19:58.891 INFO:tasks.rados.rados.2.smithi204.stdout:2516: finishing write tid 1 to smithi20457400-10 2024-09-16T05:19:58.891 INFO:tasks.rados.rados.2.smithi204.stdout:2517: done (3 left) 2024-09-16T05:19:58.891 INFO:tasks.rados.rados.2.smithi204.stdout:2520: read oid 25 snap -1 2024-09-16T05:19:58.891 INFO:tasks.rados.rados.2.smithi204.stdout:2520: expect deleted 2024-09-16T05:19:58.891 INFO:tasks.rados.rados.2.smithi204.stdout:2521: read oid 22 snap -1 2024-09-16T05:19:58.891 INFO:tasks.rados.rados.2.smithi204.stdout:2521: expect (ObjNum 848 snap 246 seq_num 848) 2024-09-16T05:19:58.891 INFO:tasks.rados.rados.2.smithi204.stdout:2522: snap_remove snap 251 2024-09-16T05:19:58.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:58 smithi032 ceph-mon[27388]: 2.24 scrub starts 2024-09-16T05:19:58.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:58 smithi032 ceph-mon[27388]: 2.24 scrub ok 2024-09-16T05:19:58.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2411232372' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:19:58.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:19:58.943 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:58.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:19:59.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:59.068 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:59.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:59.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:19:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:58 smithi204 ceph-mon[30933]: 2.24 scrub starts 2024-09-16T05:19:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:58 smithi204 ceph-mon[30933]: 2.24 scrub ok 2024-09-16T05:19:59.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2411232372' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:19:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:59.193 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:59.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:19:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:58 smithi196 ceph-mon[30802]: 2.24 scrub starts 2024-09-16T05:19:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:58 smithi196 ceph-mon[30802]: 2.24 scrub ok 2024-09-16T05:19:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2411232372' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:19:59.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:59.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:59.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:19:59.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:19:59.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:59.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:19:59.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:19:59.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T05:19:59.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:19:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:59.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:19:59.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:19:59.716 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:19:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:59.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:19:59.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:19:59.840 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:19:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:19:59.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:19:59.868 INFO:tasks.rados.rados.2.smithi204.stdout:2522: done (5 left) 2024-09-16T05:19:59.868 INFO:tasks.rados.rados.2.smithi204.stdout:2523: write oid 14 current snap is 262 2024-09-16T05:19:59.868 INFO:tasks.rados.rados.2.smithi204.stdout:2523: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2024-09-16T05:19:59.876 INFO:tasks.rados.rados.2.smithi204.stdout:2523: writing smithi20457400-14 from 659072 to 1295477 tid 1 2024-09-16T05:19:59.881 INFO:tasks.rados.rados.2.smithi204.stdout:2523: writing smithi20457400-14 from 1731866 to 2154164 tid 2 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2523: writing smithi20457400-14 from 2855825 to 3059072 tid 3 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2524: read oid 9 snap -1 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2524: expect (ObjNum 866 snap 254 seq_num 866) 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2525: rollback oid 50 current snap is 262 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 260 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2516: finishing write tid 2 to smithi20457400-10 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2516: finishing write tid 3 to smithi20457400-10 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2516: finishing write tid 4 to smithi20457400-10 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2516: finishing write tid 5 to smithi20457400-10 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 913 (ObjNum 894 snap 262 seq_num 894) dirty exists 2024-09-16T05:19:59.888 INFO:tasks.rados.rados.2.smithi204.stdout:2516: left oid 10 (ObjNum 894 snap 262 seq_num 894) 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2518: finishing write tid 1 to smithi20457400-46 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2518: finishing write tid 2 to smithi20457400-46 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2516: done (7 left) 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2526: snap_create 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2519: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2519: finishing rollback tid 2 to smithi20457400-37 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1018 (ObjNum 694 snap 201 seq_num 694) dirty exists 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2518: finishing write tid 3 to smithi20457400-46 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2518: finishing write tid 4 to smithi20457400-46 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2518: finishing write tid 5 to smithi20457400-46 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 366 (ObjNum 895 snap 262 seq_num 895) dirty exists 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2518: left oid 46 (ObjNum 895 snap 262 seq_num 895) 2024-09-16T05:19:59.889 INFO:tasks.rados.rados.2.smithi204.stdout:2521: expect (ObjNum 848 snap 246 seq_num 848) 2024-09-16T05:19:59.920 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:163 2.9 2:9423b7dc:::smithi20457400-41:head [setxattr _header (56) in=7b,truncate 2672280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 556.42456790699998, 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.176412596, 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 163 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493744+0000", 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493745+0000", 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1899999999999998e-07 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493747+0000", 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1600000000000001e-06 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493751+0000", 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8260000000000003e-06 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493754+0000", 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8779999999999998e-06 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494256+0000", 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050281900000000005 2024-09-16T05:19:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494314+0000", 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7592e-05 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494393+0000", 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8799000000000002e-05 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494718+0000", 2024-09-16T05:19:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00032548400000000002 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670063+0000", 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17534508600000001 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670109+0000", 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5427999999999999e-05 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670157+0000", 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8105000000000001e-05 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:164 2.9 2:9423b7dc:::smithi20457400-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 556.424552672, 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17655259500000001, 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 164 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493760+0000", 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7000000000000001e-07 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493762+0000", 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0660000000000002e-06 2024-09-16T05:19:59.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493766+0000", 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.732e-06 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.493769+0000", 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6799999999999999e-06 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494530+0000", 2024-09-16T05:19:59.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076053399999999999 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.494606+0000", 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6538999999999993e-05 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670197+0000", 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17559084699999999 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670253+0000", 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5893999999999997e-05 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670312+0000", 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9033000000000002e-05 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.932 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:163 2.9 e112/107 ondisk, result = 0)", 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 556.42239567900003, 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17425236199999999, 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495917+0000", 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.14e-07 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.495918+0000", 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5519999999999999e-06 2024-09-16T05:19:59.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.669955+0000", 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17403649900000001 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.669969+0000", 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4064e-05 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670051+0000", 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2313999999999998e-05 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670059+0000", 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1240000000000005e-06 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.670169+0000", 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000109695 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.936 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:409 2.9 2:96dc17ef:::smithi20457400-10:head [setxattr _header (56) in=7b,truncate 3378174] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 542.23935853600005, 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059654601000000002, 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.937 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 409 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678954+0000", 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7899999999999999e-07 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678957+0000", 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1109999999999999e-06 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678961+0000", 2024-09-16T05:19:59.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9639999999999997e-06 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678969+0000", 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8839999999999994e-06 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.679871+0000", 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090153499999999997 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.679945+0000", 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4518999999999996e-05 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680024+0000", 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9295999999999994e-05 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680263+0000", 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000238994 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738545+0000", 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058281865000000002 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738576+0000", 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0349999999999999e-05 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738608+0000", 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2703999999999997e-05 2024-09-16T05:19:59.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:410 2.9 2:96dc17ef:::smithi20457400-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 542.23933876700005, 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.059752424999999998, 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 410 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:19:59.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678974+0000", 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9400000000000001e-07 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678975+0000", 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1960000000000001e-06 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678979+0000", 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7479999999999999e-06 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.678982+0000", 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8820000000000001e-06 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680163+0000", 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011814880000000001 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680245+0000", 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1411999999999999e-05 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738635+0000", 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.058390240000000003 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738674+0000", 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9119999999999998e-05 2024-09-16T05:19:59.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738726+0000", 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2045000000000001e-05 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:409 2.9 e126/107 ondisk, result = 0)", 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 542.23742759000004, 2024-09-16T05:19:59.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.057729559, 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.950 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:19:59.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680885+0000", 2024-09-16T05:19:59.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3799999999999999e-07 2024-09-16T05:19:59.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.680887+0000", 2024-09-16T05:19:59.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7689999999999999e-06 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736934+0000", 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.056047685999999999 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.736942+0000", 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4479999999999997e-06 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738537+0000", 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015953720000000001 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738542+0000", 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7190000000000001e-06 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:57.738614+0000", 2024-09-16T05:19:59.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2427000000000003e-05 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:598 2.9 2:96dc17ef:::smithi20457400-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e142)", 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 526.11523884999997, 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010979131, 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:59.954 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 598 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803073+0000", 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803074+0000", 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.47e-07 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803078+0000", 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1030000000000004e-06 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803090+0000", 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1875999999999999e-05 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803099+0000", 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2199999999999992e-06 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803245+0000", 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014687899999999999 2024-09-16T05:19:59.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.803249+0000", 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0520000000000002e-06 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805228+0000", 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019794729999999998 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:19:59.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.805254+0000", 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6016e-05 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813951+0000", 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086966319999999993 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.813999+0000", 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8263999999999997e-05 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:13.814053+0000", 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3668999999999997e-05 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:979 2.4 2:2ace617f:::smithi20457400-21:head [copy-from ver 40] snapc 28=[28,27,25,24,23,22,21,20,1f,15,f] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 488.92044817999999, 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016837338, 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 979 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997864+0000", 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997865+0000", 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8899999999999999e-07 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997872+0000", 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0639999999999996e-06 2024-09-16T05:19:59.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997880+0000", 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4039999999999995e-06 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997886+0000", 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4600000000000002e-06 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998030+0000", 2024-09-16T05:19:59.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014470800000000001 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998036+0000", 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6620000000000002e-06 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000325+0000", 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022892450000000001 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000434+0000", 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010908699999999999 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006744+0000", 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063091240000000002 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014271+0000", 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0075269869999999997 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014635+0000", 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036412499999999997 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014670+0000", 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5799000000000001e-05 2024-09-16T05:19:59.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014701+0000", 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0984e-05 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:980 2.4 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e179)", 2024-09-16T05:19:59.967 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 488.92041764499999, 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013613798, 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 980 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997895+0000", 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997899+0000", 2024-09-16T05:19:59.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7319999999999998e-06 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997902+0000", 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.819e-06 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997913+0000", 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.083e-05 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.997919+0000", 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5570000000000003e-06 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998065+0000", 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000146012 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.998069+0000", 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8550000000000001e-06 2024-09-16T05:19:59.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000716+0000", 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026463070000000001 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.000762+0000", 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6443999999999998e-05 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011363+0000", 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010600559000000001 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011432+0000", 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8955000000000006e-05 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.011508+0000", 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6728000000000001e-05 2024-09-16T05:19:59.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1004 2.4 2:2ace617f:::smithi20457400-21:head [copy-from ver 113] snapc 29=[29,28,27,25,24,23,22,21,20,1f,15] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 487.16619387700001, 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012670931, 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.974 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1004 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752118+0000", 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752120+0000", 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.996e-06 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752132+0000", 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1389000000000001e-05 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752150+0000", 2024-09-16T05:19:59.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8533999999999999e-05 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752159+0000", 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9959999999999999e-06 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752236+0000", 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6211000000000006e-05 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752247+0000", 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1127e-05 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753383+0000", 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011362830000000001 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753447+0000", 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4009000000000003e-05 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.761264+0000", 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078170510000000002 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764658+0000", 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033938449999999999 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764722+0000", 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4177999999999994e-05 2024-09-16T05:19:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764759+0000", 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7363000000000001e-05 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.764789+0000", 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9949000000000001e-05 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1005 2.4 2:2ace617f:::smithi20457400-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e181)", 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 487.16612947099998, 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011596623, 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.981 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1005 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752183+0000", 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0600000000000001e-07 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752185+0000", 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2419999999999999e-06 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752190+0000", 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7049999999999998e-06 2024-09-16T05:19:59.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752193+0000", 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2739999999999999e-06 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752304+0000", 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000110301 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:19:59.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.752307+0000", 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6559999999999998e-06 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753639+0000", 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001331201 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.753690+0000", 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1826999999999997e-05 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763643+0000", 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0099530030000000002 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763709+0000", 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5469999999999995e-05 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:52.763779+0000", 2024-09-16T05:19:59.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0537999999999994e-05 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1499 2.9 2:94d33c56:::smithi20457400-8:head [copy-from ver 285] snapc 3e=[3e,3d,3c,3b,3a,39,38,37,35,33,2f,2d,27,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 450.22808566800001, 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016013924999999998, 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.987 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1499 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690227+0000", 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690228+0000", 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6419999999999999e-06 2024-09-16T05:19:59.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690237+0000", 2024-09-16T05:19:59.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8929999999999994e-06 2024-09-16T05:19:59.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690270+0000", 2024-09-16T05:19:59.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2679999999999999e-05 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690281+0000", 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1446e-05 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690342+0000", 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0681999999999999e-05 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:19:59.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690408+0000", 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6173000000000005e-05 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.698590+0000", 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081814639999999994 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.704358+0000", 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057679389999999997 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706113+0000", 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001755097 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706183+0000", 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9991000000000001e-05 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.706241+0000", 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7917999999999997e-05 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1500 2.9 2:94d33c56:::smithi20457400-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e218)", 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 450.22800588000001, 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012373189, 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:19:59.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1500 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:19:59.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:19:59.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690306+0000", 2024-09-16T05:19:59.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:19:59.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:19:59.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690307+0000", 2024-09-16T05:19:59.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7599999999999997e-07 2024-09-16T05:19:59.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690310+0000", 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2610000000000002e-06 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690337+0000", 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6733999999999998e-05 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690344+0000", 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2779999999999996e-06 2024-09-16T05:19:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690524+0000", 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017976900000000001 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.690560+0000", 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5762000000000001e-05 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702528+0000", 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011967976 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702599+0000", 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1429999999999999e-05 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:29.702680+0000", 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0402999999999996e-05 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.001 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:611 2.15 e366/273 2:a94680a3:::smithi20457400-13:66 v 366'446, mlcod=362'442)", 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 284.20148143699998, 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119602217, 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:00.002 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8099999999999999e-07 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.003 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716837+0000", 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4909999999999996e-06 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835914+0000", 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119077768 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835922+0000", 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.994e-06 2024-09-16T05:20:00.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835962+0000", 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9230000000000002e-05 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835979+0000", 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.787e-05 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.836402+0000", 2024-09-16T05:20:00.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042242199999999999 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.836433+0000", 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1260999999999998e-05 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4658 2.35 2:ad4a88eb:::smithi20457400-16:head [copy-from ver 686] snapc c1=[c1,bf,be] ondisk+write+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.782375+0000", 2024-09-16T05:20:00.006 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 135.13593745099999, 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019484680000000001, 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4658 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782375+0000", 2024-09-16T05:20:00.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782375+0000", 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782377+0000", 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2579999999999998e-06 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782399+0000", 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1585000000000001e-05 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782417+0000", 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8658e-05 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782426+0000", 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2029999999999992e-06 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782493+0000", 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7094000000000005e-05 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782498+0000", 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2349999999999999e-06 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.784105+0000", 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016068079999999999 2024-09-16T05:20:00.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.784171+0000", 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6023000000000001e-05 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.796572+0000", 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012401392000000001 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.799172+0000", 2024-09-16T05:20:00.011 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025995240000000002 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.801780+0000", 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026087430000000002 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.801815+0000", 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4950000000000002e-05 2024-09-16T05:20:00.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.013 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:00.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.013 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:00.013 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.801860+0000", 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4206999999999999e-05 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4659 2.35 2:ad4a88eb:::smithi20457400-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.782448+0000", 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 135.13586476399999, 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016349405000000001, 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:00.014 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4659 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782448+0000", 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.015 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782448+0000", 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782448+0000", 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4300000000000003e-07 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782451+0000", 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6570000000000001e-06 2024-09-16T05:20:00.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782459+0000", 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1610000000000003e-06 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782465+0000", 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7790000000000001e-06 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782548+0000", 2024-09-16T05:20:00.017 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3323000000000002e-05 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782552+0000", 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6710000000000001e-06 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.784265+0000", 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017134850000000001 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.784292+0000", 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7180000000000001e-05 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.798667+0000", 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014374408 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.798731+0000", 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4429000000000003e-05 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.798797+0000", 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5769e-05 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5589 2.54 2:2ace617f:::smithi20457400-21:head [copy-from ver 833] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.804999+0000", 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 67.113313206000001, 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013042259, 2024-09-16T05:20:00.020 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5589 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.804999+0000", 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.804999+0000", 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805000+0000", 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3090000000000001e-06 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:00.022 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805006+0000", 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5550000000000002e-06 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805018+0000", 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1995e-05 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805024+0000", 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8279999999999999e-06 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805088+0000", 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4295999999999996e-05 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805198+0000", 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011024299999999999 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.811590+0000", 2024-09-16T05:20:00.024 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063921100000000003 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.815801+0000", 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042101969999999997 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.817919+0000", 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002118071 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.025 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.817988+0000", 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9666999999999996e-05 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.818041+0000", 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2988000000000002e-05 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.026 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5603 2.35 2:ad4a88eb:::smithi20457400-16:head [copy-from ver 369] snapc e7=[e7,e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e584)", 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:53.850711+0000", 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 66.067600834999993, 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015532102000000001, 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5603 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:00.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850711+0000", 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850711+0000", 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:00.028 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850713+0000", 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3289999999999999e-06 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850718+0000", 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7339999999999999e-06 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850730+0000", 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2695e-05 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850737+0000", 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2010000000000002e-06 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850792+0000", 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4817e-05 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850868+0000", 2024-09-16T05:20:00.030 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5765000000000004e-05 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.858838+0000", 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079695340000000003 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.863812+0000", 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049744610000000003 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.031 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.866156+0000", 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023442889999999998 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.866206+0000", 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0120000000000001e-05 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.866244+0000", 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7156999999999997e-05 2024-09-16T05:20:00.032 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5604 2.35 2:ad4a88eb:::smithi20457400-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e584)", 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:53.850750+0000", 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 66.067561917999996, 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011376676, 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:00.033 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5604 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850750+0000", 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850750+0000", 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850751+0000", 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850756+0000", 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6630000000000001e-06 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:00.035 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850763+0000", 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4680000000000003e-06 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850769+0000", 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7699999999999998e-06 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850987+0000", 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021844799999999999 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.036 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.851020+0000", 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3238999999999998e-05 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.861934+0000", 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010913983 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.862047+0000", 2024-09-16T05:20:00.037 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000112487 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.862127+0000", 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0131999999999998e-05 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.7.0:974 2.3c e595/293 2:3d4bedc8:::smithi20457400-22:d8 v 595'589, mlcod=592'583)", 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:04.831453+0000", 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 55.086859511999997, 2024-09-16T05:20:00.038 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.549161342, 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:04.831453+0000", 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:04.831453+0000", 2024-09-16T05:20:00.039 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:04.831454+0000", 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8899999999999998e-07 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:04.831458+0000", 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5879999999999999e-06 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.379926+0000", 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.548467912 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.379935+0000", 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6459999999999999e-06 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.380009+0000", 2024-09-16T05:20:00.041 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3879000000000002e-05 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.380036+0000", 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7574999999999999e-05 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.380573+0000", 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053675900000000004 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:00.042 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.380614+0000", 2024-09-16T05:20:00.043 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1094000000000003e-05 2024-09-16T05:20:00.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.043 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:00.043 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:00.043 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:20:00.044 INFO:tasks.rados.rados.2.smithi204.stdout:2523: finishing write tid 1 to smithi20457400-14 2024-09-16T05:20:00.044 INFO:tasks.rados.rados.2.smithi204.stdout:2525: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:20:00.044 INFO:tasks.rados.rados.2.smithi204.stdout:2523: finishing write tid 2 to smithi20457400-14 2024-09-16T05:20:00.044 INFO:tasks.rados.rados.2.smithi204.stdout:2525: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:20:00.044 INFO:tasks.rados.rados.2.smithi204.stdout:2523: finishing write tid 3 to smithi20457400-14 2024-09-16T05:20:00.044 INFO:tasks.rados.rados.2.smithi204.stdout:2525: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:20:00.044 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1102 (ObjNum 879 snap 256 seq_num 879) dirty exists 2024-09-16T05:20:00.044 INFO:tasks.rados.rados.2.smithi204.stdout:2524: expect (ObjNum 866 snap 254 seq_num 866) 2024-09-16T05:20:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:00.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:00.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:00.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:00.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:00.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: pgmap v1134: 107 pgs: 107 active+clean; 251 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 550 KiB/s wr, 8 op/s 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: 2.3f scrub starts 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: 2.3f scrub ok 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2411232372' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: osdmap e649: 12 total, 12 up, 11 in 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:20:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:19:59 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: pgmap v1134: 107 pgs: 107 active+clean; 251 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 550 KiB/s wr, 8 op/s 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: 2.3f scrub starts 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: 2.3f scrub ok 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2411232372' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: osdmap e649: 12 total, 12 up, 11 in 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:20:00.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:19:59 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:20:00.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: pgmap v1134: 107 pgs: 107 active+clean; 251 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 550 KiB/s wr, 8 op/s 2024-09-16T05:20:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: 2.3f scrub starts 2024-09-16T05:20:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: 2.3f scrub ok 2024-09-16T05:20:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2411232372' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:20:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: osdmap e649: 12 total, 12 up, 11 in 2024-09-16T05:20:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:20:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:19:59 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:20:00.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:00.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:00.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:00.351 INFO:tasks.rados.rados.2.smithi204.stdout:2523: finishing write tid 4 to smithi20457400-14 2024-09-16T05:20:00.351 INFO:tasks.rados.rados.2.smithi204.stdout:2523: finishing write tid 5 to smithi20457400-14 2024-09-16T05:20:00.351 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1154 (ObjNum 896 snap 262 seq_num 896) dirty exists 2024-09-16T05:20:00.351 INFO:tasks.rados.rados.2.smithi204.stdout:2523: left oid 14 (ObjNum 896 snap 262 seq_num 896) 2024-09-16T05:20:00.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:00.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:00.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:00.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:00.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:00.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:00.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:00.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:20:00.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:00.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:00.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:00.789 INFO:tasks.rados.rados.2.smithi204.stdout:2518: done (7 left) 2024-09-16T05:20:00.789 INFO:tasks.rados.rados.2.smithi204.stdout:2519: done (6 left) 2024-09-16T05:20:00.789 INFO:tasks.rados.rados.2.smithi204.stdout:2520: done (5 left) 2024-09-16T05:20:00.789 INFO:tasks.rados.rados.2.smithi204.stdout:2521: done (4 left) 2024-09-16T05:20:00.789 INFO:tasks.rados.rados.2.smithi204.stdout:2523: done (3 left) 2024-09-16T05:20:00.789 INFO:tasks.rados.rados.2.smithi204.stdout:2524: done (2 left) 2024-09-16T05:20:00.789 INFO:tasks.rados.rados.2.smithi204.stdout:2525: done (1 left) 2024-09-16T05:20:00.789 INFO:tasks.rados.rados.2.smithi204.stdout:2526: done (0 left) 2024-09-16T05:20:00.790 INFO:tasks.rados.rados.2.smithi204.stdout:2527: snap_remove snap 247 2024-09-16T05:20:00.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:00.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:00.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:00.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:00.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:01.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:01.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:01.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:01.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:00 smithi032 ceph-mon[27388]: osdmap e650: 12 total, 12 up, 11 in 2024-09-16T05:20:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:00 smithi032 ceph-mon[27388]: overall HEALTH_OK 2024-09-16T05:20:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:00 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:20:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:00 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:20:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:00 smithi032 ceph-mon[27388]: osdmap e651: 12 total, 12 up, 11 in 2024-09-16T05:20:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:00 smithi204 ceph-mon[30933]: osdmap e650: 12 total, 12 up, 11 in 2024-09-16T05:20:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:00 smithi204 ceph-mon[30933]: overall HEALTH_OK 2024-09-16T05:20:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:00 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:20:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:00 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:20:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:00 smithi204 ceph-mon[30933]: osdmap e651: 12 total, 12 up, 11 in 2024-09-16T05:20:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:01.172 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:01.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:00 smithi196 ceph-mon[30802]: osdmap e650: 12 total, 12 up, 11 in 2024-09-16T05:20:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:00 smithi196 ceph-mon[30802]: overall HEALTH_OK 2024-09-16T05:20:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:00 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:20:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:00 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:20:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:00 smithi196 ceph-mon[30802]: osdmap e651: 12 total, 12 up, 11 in 2024-09-16T05:20:01.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:01.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:01.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:01.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:20:01.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:01.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:01.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:01.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:01.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:01.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:01.589 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:01.617 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:01.617 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:01.630 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:01.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:01.661 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:01.661 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:01.669 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:01.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:01.682 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:01.682 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:01.704 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:01.704 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:01.725 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:01.726 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:01.747 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:01.747 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:01.769 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:01.769 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:01.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:01.790 INFO:tasks.rados.rados.2.smithi204.stdout:2527: done (0 left) 2024-09-16T05:20:01.791 INFO:tasks.rados.rados.2.smithi204.stdout:2528: read oid 50 snap -1 2024-09-16T05:20:01.791 INFO:tasks.rados.rados.2.smithi204.stdout:2528: expect (ObjNum 879 snap 256 seq_num 879) 2024-09-16T05:20:01.791 INFO:tasks.rados.rados.2.smithi204.stdout:2529: rollback oid 22 current snap is 263 2024-09-16T05:20:01.791 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 258 2024-09-16T05:20:01.791 INFO:tasks.rados.rados.2.smithi204.stdout:2530: read oid 12 snap -1 2024-09-16T05:20:01.791 INFO:tasks.rados.rados.2.smithi204.stdout:2530: expect (ObjNum 877 snap 256 seq_num 877) 2024-09-16T05:20:01.791 INFO:tasks.rados.rados.2.smithi204.stdout:2531: delete oid 11 current snap is 263 2024-09-16T05:20:01.794 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:01.794 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:01.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:01.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:01.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:01.800 INFO:tasks.rados.rados.2.smithi204.stdout:2529: finishing rollback tid 0 to smithi20457400-22 2024-09-16T05:20:01.801 INFO:tasks.rados.rados.2.smithi204.stdout:2529: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:20:01.802 INFO:tasks.rados.rados.2.smithi204.stdout:2529: finishing rollback tid 2 to smithi20457400-22 2024-09-16T05:20:01.802 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 675 (ObjNum 848 snap 246 seq_num 848) dirty exists 2024-09-16T05:20:01.804 INFO:tasks.rados.rados.2.smithi204.stdout:2528: expect (ObjNum 879 snap 256 seq_num 879) 2024-09-16T05:20:01.820 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:01.820 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:01.841 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:01.841 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:01.862 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:01.863 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:01.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:01.919 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:01.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:01.956 INFO:tasks.rados.rados.2.smithi204.stdout:2530: expect (ObjNum 877 snap 256 seq_num 877) 2024-09-16T05:20:02.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:02.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:02.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:02.045 INFO:tasks.rados.rados.2.smithi204.stdout:2528: done (3 left) 2024-09-16T05:20:02.045 INFO:tasks.rados.rados.2.smithi204.stdout:2529: done (2 left) 2024-09-16T05:20:02.045 INFO:tasks.rados.rados.2.smithi204.stdout:2530: done (1 left) 2024-09-16T05:20:02.045 INFO:tasks.rados.rados.2.smithi204.stdout:2531: done (0 left) 2024-09-16T05:20:02.045 INFO:tasks.rados.rados.2.smithi204.stdout:2532: copy_from oid 46 from oid 13 current snap is 263 2024-09-16T05:20:02.045 INFO:tasks.rados.rados.2.smithi204.stdout:2533: snap_remove snap 249 2024-09-16T05:20:02.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:01 smithi032 ceph-mon[27388]: pgmap v1137: 107 pgs: 107 active+clean; 257 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 8.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-16T05:20:02.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:01 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:20:02.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:01 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:20:02.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:01 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:20:02.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:01 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:20:02.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1562709539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:02.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:01 smithi032 ceph-mon[27388]: osdmap e652: 12 total, 12 up, 11 in 2024-09-16T05:20:02.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:01 smithi204 ceph-mon[30933]: pgmap v1137: 107 pgs: 107 active+clean; 257 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 8.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-16T05:20:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:01 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:20:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:01 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:20:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:01 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:20:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:01 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:20:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1562709539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:01 smithi204 ceph-mon[30933]: osdmap e652: 12 total, 12 up, 11 in 2024-09-16T05:20:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:02.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:02.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:01 smithi196 ceph-mon[30802]: pgmap v1137: 107 pgs: 107 active+clean; 257 MiB data, 802 MiB used, 983 GiB / 983 GiB avail; 8.6 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-16T05:20:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:01 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:20:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:01 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:20:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:01 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:20:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:01 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:20:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1562709539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:01 smithi196 ceph-mon[30802]: osdmap e652: 12 total, 12 up, 11 in 2024-09-16T05:20:02.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:02.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:02.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:02.352 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:02.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:02.420 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:02.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:02.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:02.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:02.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:02.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:02.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:02.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:02.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:02.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:02.800 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:02.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:02.885 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T05:20:02.889 INFO:tasks.rados.rados.2.smithi204.stdout:2533: done (1 left) 2024-09-16T05:20:02.889 INFO:tasks.rados.rados.2.smithi204.stdout:2534: copy_from oid 17 from oid 11 current snap is 263 2024-09-16T05:20:02.890 INFO:tasks.rados.rados.2.smithi204.stdout:2535: read oid 37 snap -1 2024-09-16T05:20:02.890 INFO:tasks.rados.rados.2.smithi204.stdout:2535: expect (ObjNum 694 snap 201 seq_num 694) 2024-09-16T05:20:02.890 INFO:tasks.rados.rados.2.smithi204.stdout:2536: read oid 34 snap 229 2024-09-16T05:20:02.890 INFO:tasks.rados.rados.2.smithi204.stdout:2536: expect deleted 2024-09-16T05:20:02.890 INFO:tasks.rados.rados.2.smithi204.stdout:2537: read oid 31 snap -1 2024-09-16T05:20:02.890 INFO:tasks.rados.rados.2.smithi204.stdout:2537: expect deleted 2024-09-16T05:20:02.890 INFO:tasks.rados.rados.2.smithi204.stdout:2538: write (excl) oid 29 current snap is 263 2024-09-16T05:20:02.890 INFO:tasks.rados.rados.2.smithi204.stdout:2538: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2024-09-16T05:20:02.896 INFO:tasks.rados.rados.2.smithi204.stdout:2538: writing smithi20457400-29 from 692686 to 1195942 tid 1 2024-09-16T05:20:02.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:02.903 INFO:tasks.rados.rados.2.smithi204.stdout:2538: writing smithi20457400-29 from 1848830 to 2274797 tid 2 2024-09-16T05:20:02.908 INFO:tasks.rados.rados.2.smithi204.stdout:2538: writing smithi20457400-29 from 2910671 to 3092686 tid 3 2024-09-16T05:20:02.908 INFO:tasks.rados.rados.2.smithi204.stdout:2539: read oid 43 snap 256 2024-09-16T05:20:02.908 INFO:tasks.rados.rados.2.smithi204.stdout:2539: expect deleted 2024-09-16T05:20:02.908 INFO:tasks.rados.rados.2.smithi204.stdout:2540: write oid 23 current snap is 263 2024-09-16T05:20:02.908 INFO:tasks.rados.rados.2.smithi204.stdout:2532: finishing copy_from racing read to smithi20457400-46 2024-09-16T05:20:02.908 INFO:tasks.rados.rados.2.smithi204.stdout:2532: finishing copy_from to smithi20457400-46 2024-09-16T05:20:02.908 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1008 (ObjNum 855 snap 248 seq_num 855) dirty exists 2024-09-16T05:20:02.908 INFO:tasks.rados.rados.2.smithi204.stdout:2540: seq_num 899 ranges {709493=578505,1907312=685978} 2024-09-16T05:20:02.914 INFO:tasks.rados.rados.2.smithi204.stdout:2540: writing smithi20457400-23 from 709493 to 1287998 tid 1 2024-09-16T05:20:02.922 INFO:tasks.rados.rados.2.smithi204.stdout:2540: writing smithi20457400-23 from 1907312 to 2593290 tid 2 2024-09-16T05:20:02.922 INFO:tasks.rados.rados.2.smithi204.stdout:2532: done (7 left) 2024-09-16T05:20:02.922 INFO:tasks.rados.rados.2.smithi204.stdout:2541: write oid 39 current snap is 263 2024-09-16T05:20:02.922 INFO:tasks.rados.rados.2.smithi204.stdout:2541: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2024-09-16T05:20:02.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:02.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:02.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:02.929 INFO:tasks.rados.rados.2.smithi204.stdout:2541: writing smithi20457400-39 from 726300 to 1380054 tid 1 2024-09-16T05:20:02.935 INFO:tasks.rados.rados.2.smithi204.stdout:2541: writing smithi20457400-39 from 1965794 to 2395430 tid 2 2024-09-16T05:20:02.939 INFO:tasks.rados.rados.2.smithi204.stdout:2541: writing smithi20457400-39 from 2965517 to 3126300 tid 3 2024-09-16T05:20:02.940 INFO:tasks.rados.rados.2.smithi204.stdout:2542: rollback oid 24 current snap is 263 2024-09-16T05:20:02.940 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 258 2024-09-16T05:20:02.940 INFO:tasks.rados.rados.2.smithi204.stdout:2535: expect (ObjNum 694 snap 201 seq_num 694) 2024-09-16T05:20:03.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:03.051 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:03.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:03.058 INFO:tasks.rados.rados.2.smithi204.stdout:2534: finishing copy_from to smithi20457400-17 2024-09-16T05:20:03.058 INFO:tasks.rados.rados.2.smithi204.stdout:2534: got expected ENOENT (src dne) 2024-09-16T05:20:03.058 INFO:tasks.rados.rados.2.smithi204.stdout:2534: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:20:03.058 INFO:tasks.rados.rados.2.smithi204.stdout:2538: finishing write tid 1 to smithi20457400-29 2024-09-16T05:20:03.058 INFO:tasks.rados.rados.2.smithi204.stdout:2538: finishing write tid 2 to smithi20457400-29 2024-09-16T05:20:03.058 INFO:tasks.rados.rados.2.smithi204.stdout:2534: done (8 left) 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2535: done (7 left) 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2536: done (6 left) 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2537: done (5 left) 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2543: rollback oid 50 current snap is 263 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 240 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2538: finishing write tid 3 to smithi20457400-29 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2538: finishing write tid 4 to smithi20457400-29 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2538: finishing write tid 5 to smithi20457400-29 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 957 (ObjNum 898 snap 263 seq_num 898) dirty exists 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2538: left oid 29 (ObjNum 898 snap 263 seq_num 898) 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2540: finishing write tid 1 to smithi20457400-23 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2540: finishing write tid 2 to smithi20457400-23 2024-09-16T05:20:03.059 INFO:tasks.rados.rados.2.smithi204.stdout:2540: finishing write tid 3 to smithi20457400-23 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:2540: finishing write tid 4 to smithi20457400-23 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 720 (ObjNum 899 snap 263 seq_num 899) dirty exists 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:2540: left oid 23 (ObjNum 899 snap 263 seq_num 899) 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:2541: finishing write tid 1 to smithi20457400-39 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:2541: finishing write tid 2 to smithi20457400-39 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:2541: finishing write tid 3 to smithi20457400-39 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:2541: finishing write tid 4 to smithi20457400-39 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:2538: done (5 left) 2024-09-16T05:20:03.060 INFO:tasks.rados.rados.2.smithi204.stdout:2539: done (4 left) 2024-09-16T05:20:03.061 INFO:tasks.rados.rados.2.smithi204.stdout:2540: done (3 left) 2024-09-16T05:20:03.061 INFO:tasks.rados.rados.2.smithi204.stdout:2544: write (excl) oid 28 current snap is 263 2024-09-16T05:20:03.061 INFO:tasks.rados.rados.2.smithi204.stdout:2542: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:20:03.061 INFO:tasks.rados.rados.2.smithi204.stdout:2541: finishing write tid 5 to smithi20457400-39 2024-09-16T05:20:03.061 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1059 (ObjNum 900 snap 263 seq_num 900) dirty exists 2024-09-16T05:20:03.061 INFO:tasks.rados.rados.2.smithi204.stdout:2541: left oid 39 (ObjNum 900 snap 263 seq_num 900) 2024-09-16T05:20:03.061 INFO:tasks.rados.rados.2.smithi204.stdout:2542: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:20:03.061 INFO:tasks.rados.rados.2.smithi204.stdout:2544: seq_num 901 ranges {743107=729003,1907923=689647} 2024-09-16T05:20:03.065 INFO:tasks.rados.rados.2.smithi204.stdout:2544: writing smithi20457400-28 from 743107 to 1472110 tid 1 2024-09-16T05:20:03.071 INFO:tasks.rados.rados.2.smithi204.stdout:2544: writing smithi20457400-28 from 1907923 to 2597570 tid 2 2024-09-16T05:20:03.071 INFO:tasks.rados.rados.2.smithi204.stdout:2541: done (3 left) 2024-09-16T05:20:03.071 INFO:tasks.rados.rados.2.smithi204.stdout:2545: snap_remove snap 245 2024-09-16T05:20:03.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:02 smithi032 ceph-mon[27388]: 2.55 scrub starts 2024-09-16T05:20:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:02 smithi032 ceph-mon[27388]: 2.55 scrub ok 2024-09-16T05:20:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3602250982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:20:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:02 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:20:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:02 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:20:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:02 smithi204 ceph-mon[30933]: 2.55 scrub starts 2024-09-16T05:20:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:02 smithi204 ceph-mon[30933]: 2.55 scrub ok 2024-09-16T05:20:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3602250982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:20:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:02 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:20:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:02 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:20:03.175 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:03.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:02 smithi196 ceph-mon[30802]: 2.55 scrub starts 2024-09-16T05:20:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:02 smithi196 ceph-mon[30802]: 2.55 scrub ok 2024-09-16T05:20:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3602250982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:20:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:02 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:20:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:02 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:20:03.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:03.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:03.299 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:03.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:03.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:03.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:03.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:03.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:03.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T05:20:03.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:03.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:03.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:03.677 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:03.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:03.766 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:03.766 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:03.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:03.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:03.890 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:20:03.895 INFO:tasks.rados.rados.2.smithi204.stdout:2545: done (3 left) 2024-09-16T05:20:03.895 INFO:tasks.rados.rados.2.smithi204.stdout:2546: read oid 25 snap -1 2024-09-16T05:20:03.895 INFO:tasks.rados.rados.2.smithi204.stdout:2546: expect deleted 2024-09-16T05:20:03.895 INFO:tasks.rados.rados.2.smithi204.stdout:2547: read oid 48 snap 258 2024-09-16T05:20:03.895 INFO:tasks.rados.rados.2.smithi204.stdout:2547: expect deleted 2024-09-16T05:20:03.896 INFO:tasks.rados.rados.2.smithi204.stdout:2548: read oid 1 snap -1 2024-09-16T05:20:03.896 INFO:tasks.rados.rados.2.smithi204.stdout:2548: expect (ObjNum 842 snap 243 seq_num 842) 2024-09-16T05:20:03.896 INFO:tasks.rados.rados.2.smithi204.stdout:2549: read oid 33 snap 260 2024-09-16T05:20:03.896 INFO:tasks.rados.rados.2.smithi204.stdout:2549: expect (ObjNum 823 snap 240 seq_num 823) 2024-09-16T05:20:03.896 INFO:tasks.rados.rados.2.smithi204.stdout:2550: snap_create 2024-09-16T05:20:03.896 INFO:tasks.rados.rados.2.smithi204.stdout:2542: finishing rollback tid 2 to smithi20457400-24 2024-09-16T05:20:03.897 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 681 (ObjNum 880 snap 256 seq_num 880) dirty exists 2024-09-16T05:20:03.897 INFO:tasks.rados.rados.2.smithi204.stdout:2543: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:20:03.897 INFO:tasks.rados.rados.2.smithi204.stdout:2543: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:20:03.897 INFO:tasks.rados.rados.2.smithi204.stdout:2543: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:20:03.897 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1105 (ObjNum 782 snap 227 seq_num 782) dirty exists 2024-09-16T05:20:03.897 INFO:tasks.rados.rados.2.smithi204.stdout:2544: finishing write tid 1 to smithi20457400-28 2024-09-16T05:20:03.898 INFO:tasks.rados.rados.2.smithi204.stdout:2544: finishing write tid 2 to smithi20457400-28 2024-09-16T05:20:03.898 INFO:tasks.rados.rados.2.smithi204.stdout:2544: finishing write tid 3 to smithi20457400-28 2024-09-16T05:20:03.898 INFO:tasks.rados.rados.2.smithi204.stdout:2544: finishing write tid 4 to smithi20457400-28 2024-09-16T05:20:03.898 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1086 (ObjNum 901 snap 263 seq_num 901) dirty exists 2024-09-16T05:20:03.898 INFO:tasks.rados.rados.2.smithi204.stdout:2544: left oid 28 (ObjNum 901 snap 263 seq_num 901) 2024-09-16T05:20:03.901 INFO:tasks.rados.rados.2.smithi204.stdout:2548: expect (ObjNum 842 snap 243 seq_num 842) 2024-09-16T05:20:03.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:03.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:03.991 INFO:tasks.rados.rados.2.smithi204.stdout:2549: expect (ObjNum 823 snap 240 seq_num 823) 2024-09-16T05:20:04.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:04.053 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:04.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:04.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: pgmap v1140: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 273 MiB data, 823 MiB used, 983 GiB / 983 GiB avail; 7.6 MiB/s rd, 3.0 MiB/s wr, 22 op/s 2024-09-16T05:20:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 2.36 scrub starts 2024-09-16T05:20:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 2.36 scrub ok 2024-09-16T05:20:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3602250982' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: osdmap e653: 12 total, 12 up, 11 in 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3296933193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:20:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:03 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: pgmap v1140: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 273 MiB data, 823 MiB used, 983 GiB / 983 GiB avail; 7.6 MiB/s rd, 3.0 MiB/s wr, 22 op/s 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 2.36 scrub starts 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 2.36 scrub ok 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3602250982' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: osdmap e653: 12 total, 12 up, 11 in 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:20:04.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:20:04.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3296933193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:04.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:20:04.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:03 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:20:04.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:04.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: pgmap v1140: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 273 MiB data, 823 MiB used, 983 GiB / 983 GiB avail; 7.6 MiB/s rd, 3.0 MiB/s wr, 22 op/s 2024-09-16T05:20:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 2.36 scrub starts 2024-09-16T05:20:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 2.36 scrub ok 2024-09-16T05:20:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:20:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:20:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:20:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:20:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3602250982' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:20:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: osdmap e653: 12 total, 12 up, 11 in 2024-09-16T05:20:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:20:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:20:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3296933193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:20:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:03 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:20:04.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:04.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:04.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:04.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:04.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:04.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:04.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:04.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:04.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:04.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:04.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:04.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:04.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:04.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:04.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:04.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:04.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:04.903 INFO:tasks.rados.rados.2.smithi204.stdout:2542: done (7 left) 2024-09-16T05:20:04.903 INFO:tasks.rados.rados.2.smithi204.stdout:2543: done (6 left) 2024-09-16T05:20:04.903 INFO:tasks.rados.rados.2.smithi204.stdout:2544: done (5 left) 2024-09-16T05:20:04.903 INFO:tasks.rados.rados.2.smithi204.stdout:2546: done (4 left) 2024-09-16T05:20:04.904 INFO:tasks.rados.rados.2.smithi204.stdout:2547: done (3 left) 2024-09-16T05:20:04.904 INFO:tasks.rados.rados.2.smithi204.stdout:2548: done (2 left) 2024-09-16T05:20:04.904 INFO:tasks.rados.rados.2.smithi204.stdout:2549: done (1 left) 2024-09-16T05:20:04.904 INFO:tasks.rados.rados.2.smithi204.stdout:2550: done (0 left) 2024-09-16T05:20:04.904 INFO:tasks.rados.rados.2.smithi204.stdout:2551: delete oid 8 current snap is 264 2024-09-16T05:20:04.908 INFO:tasks.rados.rados.2.smithi204.stdout:2551: done (0 left) 2024-09-16T05:20:04.908 INFO:tasks.rados.rados.2.smithi204.stdout:2552: snap_create 2024-09-16T05:20:04.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:04.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:04.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:05.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:05.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:20:05.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:05.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:05.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:05.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:20:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:20:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:20:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:20:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:20:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:20:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3296933193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:05.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:04 smithi032 ceph-mon[27388]: osdmap e654: 12 total, 12 up, 11 in 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3296933193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:05.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:04 smithi204 ceph-mon[30933]: osdmap e654: 12 total, 12 up, 11 in 2024-09-16T05:20:05.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:05.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3296933193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:04 smithi196 ceph-mon[30802]: osdmap e654: 12 total, 12 up, 11 in 2024-09-16T05:20:05.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:05.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:05.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:05.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:05.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:05.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:05.490 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:05.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:05.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:05.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:05.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:05.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:05.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:05.790 INFO:tasks.rados.rados.2.smithi204.stdout:2552: done (0 left) 2024-09-16T05:20:05.790 INFO:tasks.rados.rados.2.smithi204.stdout:2553: read oid 47 snap -1 2024-09-16T05:20:05.790 INFO:tasks.rados.rados.2.smithi204.stdout:2553: expect deleted 2024-09-16T05:20:05.790 INFO:tasks.rados.rados.2.smithi204.stdout:2554: snap_remove snap 248 2024-09-16T05:20:05.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:05.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:05.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:05.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:05.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:05.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:05 smithi032 ceph-mon[27388]: pgmap v1143: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 273 MiB data, 836 MiB used, 983 GiB / 983 GiB avail; 7.4 MiB/s rd, 3.8 MiB/s wr, 26 op/s 2024-09-16T05:20:05.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:05 smithi032 ceph-mon[27388]: osdmap e655: 12 total, 12 up, 11 in 2024-09-16T05:20:05.997 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:05 smithi032 ceph-mon[27388]: osdmap e656: 12 total, 12 up, 11 in 2024-09-16T05:20:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:06.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:06.082 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:06.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:06.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:05 smithi204 ceph-mon[30933]: pgmap v1143: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 273 MiB data, 836 MiB used, 983 GiB / 983 GiB avail; 7.4 MiB/s rd, 3.8 MiB/s wr, 26 op/s 2024-09-16T05:20:06.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:05 smithi204 ceph-mon[30933]: osdmap e655: 12 total, 12 up, 11 in 2024-09-16T05:20:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:05 smithi204 ceph-mon[30933]: osdmap e656: 12 total, 12 up, 11 in 2024-09-16T05:20:06.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:06.210 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:06.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:05 smithi196 ceph-mon[30802]: pgmap v1143: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 273 MiB data, 836 MiB used, 983 GiB / 983 GiB avail; 7.4 MiB/s rd, 3.8 MiB/s wr, 26 op/s 2024-09-16T05:20:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:05 smithi196 ceph-mon[30802]: osdmap e655: 12 total, 12 up, 11 in 2024-09-16T05:20:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:05 smithi196 ceph-mon[30802]: osdmap e656: 12 total, 12 up, 11 in 2024-09-16T05:20:06.312 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:06.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:06.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:06.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:06.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:06.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:06.533 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:06.533 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:06.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:06.589 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:06.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:06.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:20:06.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:06.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:06.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T05:20:06.790 INFO:tasks.rados.rados.2.smithi204.stdout:2554: done (1 left) 2024-09-16T05:20:06.791 INFO:tasks.rados.rados.2.smithi204.stdout:2555: snap_create 2024-09-16T05:20:06.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:06.845 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:06.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:06.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:06.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:06.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:07.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:07.098 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:07.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1194363263' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:20:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1194363263' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:20:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:06 smithi032 ceph-mon[27388]: osdmap e657: 12 total, 12 up, 11 in 2024-09-16T05:20:07.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1194363263' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:20:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1194363263' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:20:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:06 smithi204 ceph-mon[30933]: osdmap e657: 12 total, 12 up, 11 in 2024-09-16T05:20:07.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:07.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:07.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:07.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1194363263' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:20:07.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1194363263' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:20:07.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:06 smithi196 ceph-mon[30802]: osdmap e657: 12 total, 12 up, 11 in 2024-09-16T05:20:07.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:20:07.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:07.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:07.356 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:07.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:07.480 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:07.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:07.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:20:07.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:07.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:07.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:07.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:07.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:07.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:07.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:07.791 INFO:tasks.rados.rados.2.smithi204.stdout:2553: done (1 left) 2024-09-16T05:20:07.791 INFO:tasks.rados.rados.2.smithi204.stdout:2555: done (0 left) 2024-09-16T05:20:07.791 INFO:tasks.rados.rados.2.smithi204.stdout:2556: snap_create 2024-09-16T05:20:07.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:07.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:07.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:07.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:07.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:07.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:08.059 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:08.094 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:08.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T05:20:08.120 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:08.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:07 smithi032 ceph-mon[27388]: pgmap v1146: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 276 MiB data, 832 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 23 op/s 2024-09-16T05:20:08.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:07 smithi032 ceph-mon[27388]: osdmap e658: 12 total, 12 up, 11 in 2024-09-16T05:20:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:07 smithi204 ceph-mon[30933]: pgmap v1146: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 276 MiB data, 832 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 23 op/s 2024-09-16T05:20:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:07 smithi204 ceph-mon[30933]: osdmap e658: 12 total, 12 up, 11 in 2024-09-16T05:20:08.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:08.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:07 smithi196 ceph-mon[30802]: pgmap v1146: 107 pgs: 1 active+clean+snaptrim, 106 active+clean; 276 MiB data, 832 MiB used, 983 GiB / 983 GiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 23 op/s 2024-09-16T05:20:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:07 smithi196 ceph-mon[30802]: osdmap e658: 12 total, 12 up, 11 in 2024-09-16T05:20:08.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:08.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:08.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:08.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:08.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:08.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:08.506 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:08.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:08.543 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:08.562 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:08.562 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:08.588 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:08.608 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:08.609 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:08.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:08.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:08.630 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:08.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:08.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:08.652 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:08.652 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:08.673 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:08.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:08.695 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:08.718 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:08.719 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:08.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:08.738 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:08.739 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:08.757 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:08.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:08.761 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:08.761 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:08.783 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:08.793 INFO:tasks.rados.rados.2.smithi204.stdout:2556: done (0 left) 2024-09-16T05:20:08.793 INFO:tasks.rados.rados.2.smithi204.stdout:2557: write oid 41 current snap is 267 2024-09-16T05:20:08.793 INFO:tasks.rados.rados.2.smithi204.stdout:2557: seq_num 903 ranges {776721=479501,2024887=693316} 2024-09-16T05:20:08.799 INFO:tasks.rados.rados.2.smithi204.stdout:2557: writing smithi20457400-41 from 776721 to 1256222 tid 1 2024-09-16T05:20:08.807 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:08.808 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:08.808 INFO:tasks.rados.rados.2.smithi204.stdout:2557: writing smithi20457400-41 from 2024887 to 2718203 tid 2 2024-09-16T05:20:08.809 INFO:tasks.rados.rados.2.smithi204.stdout:2558: read oid 4 snap -1 2024-09-16T05:20:08.809 INFO:tasks.rados.rados.2.smithi204.stdout:2558: expect deleted 2024-09-16T05:20:08.809 INFO:tasks.rados.rados.2.smithi204.stdout:2559: rollback oid 24 current snap is 267 2024-09-16T05:20:08.809 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 255 2024-09-16T05:20:08.809 INFO:tasks.rados.rados.2.smithi204.stdout:2557: finishing write tid 1 to smithi20457400-41 2024-09-16T05:20:08.809 INFO:tasks.rados.rados.2.smithi204.stdout:2560: delete oid 2 current snap is 267 2024-09-16T05:20:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2558: done (3 left) 2024-09-16T05:20:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2560: done (2 left) 2024-09-16T05:20:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2561: rollback oid 10 current snap is 267 2024-09-16T05:20:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 240 2024-09-16T05:20:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2562: rollback oid 11 current snap is 267 2024-09-16T05:20:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 11 to 263 2024-09-16T05:20:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2563: snap_remove snap 256 2024-09-16T05:20:08.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:08.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:08.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:08.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:09.006 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:09.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:09.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:09.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:09.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:09.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2870029331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:09.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:08 smithi032 ceph-mon[27388]: osdmap e659: 12 total, 12 up, 11 in 2024-09-16T05:20:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2870029331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:08 smithi204 ceph-mon[30933]: osdmap e659: 12 total, 12 up, 11 in 2024-09-16T05:20:09.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:09.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2870029331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:09.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:08 smithi196 ceph-mon[30802]: osdmap e659: 12 total, 12 up, 11 in 2024-09-16T05:20:09.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:09.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:09.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:09.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:09.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:09.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:09.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:09.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:09.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:09.620 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:09.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:09.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:09.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:09.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:09.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:20:09.798 INFO:tasks.rados.rados.2.smithi204.stdout:2563: done (4 left) 2024-09-16T05:20:09.798 INFO:tasks.rados.rados.2.smithi204.stdout:2564: delete oid 44 current snap is 267 2024-09-16T05:20:09.798 INFO:tasks.rados.rados.2.smithi204.stdout:2559: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:20:09.798 INFO:tasks.rados.rados.2.smithi204.stdout:2559: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:20:09.798 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 690 (ObjNum 65 snap 0 seq_num 4282177120) dirty dne 2024-09-16T05:20:09.798 INFO:tasks.rados.rados.2.smithi204.stdout:2562: finishing rollback tid 1 to smithi20457400-11 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1044 (ObjNum 593 snap 0 seq_num 4283011848) dirty dne 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:2557: finishing write tid 2 to smithi20457400-41 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:2557: finishing write tid 3 to smithi20457400-41 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:2557: finishing write tid 4 to smithi20457400-41 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 941 (ObjNum 903 snap 267 seq_num 903) dirty exists 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:2557: left oid 41 (ObjNum 903 snap 267 seq_num 903) 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:2561: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:2561: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:2561: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:20:09.799 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 945 (ObjNum 753 snap 219 seq_num 753) dirty exists 2024-09-16T05:20:09.802 INFO:tasks.rados.rados.2.smithi204.stdout:2557: done (4 left) 2024-09-16T05:20:09.802 INFO:tasks.rados.rados.2.smithi204.stdout:2559: done (3 left) 2024-09-16T05:20:09.802 INFO:tasks.rados.rados.2.smithi204.stdout:2561: done (2 left) 2024-09-16T05:20:09.802 INFO:tasks.rados.rados.2.smithi204.stdout:2562: done (1 left) 2024-09-16T05:20:09.802 INFO:tasks.rados.rados.2.smithi204.stdout:2564: done (0 left) 2024-09-16T05:20:09.802 INFO:tasks.rados.rados.2.smithi204.stdout:2565: write (excl) oid 6 current snap is 267 2024-09-16T05:20:09.802 INFO:tasks.rados.rados.2.smithi204.stdout:2565: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2024-09-16T05:20:09.806 INFO:tasks.rados.rados.2.smithi204.stdout:2565: writing smithi20457400-6 from 427142 to 848743 tid 1 2024-09-16T05:20:09.812 INFO:tasks.rados.rados.2.smithi204.stdout:2565: writing smithi20457400-6 from 1400333 to 1840976 tid 2 2024-09-16T05:20:09.819 INFO:tasks.rados.rados.2.smithi204.stdout:2565: writing smithi20457400-6 from 2613702 to 3064028 tid 3 2024-09-16T05:20:09.819 INFO:tasks.rados.rados.2.smithi204.stdout:2566: snap_create 2024-09-16T05:20:09.819 INFO:tasks.rados.rados.2.smithi204.stdout:2565: finishing write tid 1 to smithi20457400-6 2024-09-16T05:20:09.819 INFO:tasks.rados.rados.2.smithi204.stdout:2565: finishing write tid 2 to smithi20457400-6 2024-09-16T05:20:09.822 INFO:tasks.rados.rados.2.smithi204.stdout:2565: finishing write tid 3 to smithi20457400-6 2024-09-16T05:20:09.822 INFO:tasks.rados.rados.2.smithi204.stdout:2565: finishing write tid 4 to smithi20457400-6 2024-09-16T05:20:09.822 INFO:tasks.rados.rados.2.smithi204.stdout:2565: finishing write tid 5 to smithi20457400-6 2024-09-16T05:20:09.822 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 741 (ObjNum 906 snap 267 seq_num 906) dirty exists 2024-09-16T05:20:09.822 INFO:tasks.rados.rados.2.smithi204.stdout:2565: left oid 6 (ObjNum 906 snap 267 seq_num 906) 2024-09-16T05:20:09.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:09.896 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:09.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:09.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:09 smithi032 ceph-mon[27388]: pgmap v1149: 107 pgs: 107 active+clean; 269 MiB data, 821 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-16T05:20:09.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3586574193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:09.973 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:09 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:09.974 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3586574193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:09.974 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:09 smithi032 ceph-mon[27388]: osdmap e660: 12 total, 12 up, 11 in 2024-09-16T05:20:09.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:10.056 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:10.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:10.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:09 smithi204 ceph-mon[30933]: pgmap v1149: 107 pgs: 107 active+clean; 269 MiB data, 821 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-16T05:20:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3586574193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:09 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3586574193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:09 smithi204 ceph-mon[30933]: osdmap e660: 12 total, 12 up, 11 in 2024-09-16T05:20:10.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:10.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:09 smithi196 ceph-mon[30802]: pgmap v1149: 107 pgs: 107 active+clean; 269 MiB data, 821 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-16T05:20:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3586574193' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:09 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3586574193' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:09 smithi196 ceph-mon[30802]: osdmap e660: 12 total, 12 up, 11 in 2024-09-16T05:20:10.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:10.307 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:10.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:10.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:10.433 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:10.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:10.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:10.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:10.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:10.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:10.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:10.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:10.775 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:10.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:10.790 INFO:tasks.rados.rados.2.smithi204.stdout:2565: done (1 left) 2024-09-16T05:20:10.790 INFO:tasks.rados.rados.2.smithi204.stdout:2566: done (0 left) 2024-09-16T05:20:10.790 INFO:tasks.rados.rados.2.smithi204.stdout:2567: write oid 21 current snap is 268 2024-09-16T05:20:10.790 INFO:tasks.rados.rados.2.smithi204.stdout:2567: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-09-16T05:20:10.794 INFO:tasks.rados.rados.2.smithi204.stdout:2567: writing smithi20457400-21 from 443949 to 940799 tid 1 2024-09-16T05:20:10.801 INFO:tasks.rados.rados.2.smithi204.stdout:2567: writing smithi20457400-21 from 1458815 to 2043116 tid 2 2024-09-16T05:20:10.807 INFO:tasks.rados.rados.2.smithi204.stdout:2567: writing smithi20457400-21 from 2641125 to 3243949 tid 3 2024-09-16T05:20:10.807 INFO:tasks.rados.rados.2.smithi204.stdout:2568: snap_create 2024-09-16T05:20:10.807 INFO:tasks.rados.rados.2.smithi204.stdout:2567: finishing write tid 1 to smithi20457400-21 2024-09-16T05:20:10.808 INFO:tasks.rados.rados.2.smithi204.stdout:2567: finishing write tid 2 to smithi20457400-21 2024-09-16T05:20:10.810 INFO:tasks.rados.rados.2.smithi204.stdout:2567: finishing write tid 3 to smithi20457400-21 2024-09-16T05:20:10.810 INFO:tasks.rados.rados.2.smithi204.stdout:2567: finishing write tid 4 to smithi20457400-21 2024-09-16T05:20:10.810 INFO:tasks.rados.rados.2.smithi204.stdout:2567: finishing write tid 5 to smithi20457400-21 2024-09-16T05:20:10.810 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 844 (ObjNum 907 snap 268 seq_num 907) dirty exists 2024-09-16T05:20:10.810 INFO:tasks.rados.rados.2.smithi204.stdout:2567: left oid 21 (ObjNum 907 snap 268 seq_num 907) 2024-09-16T05:20:10.811 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:10.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:10.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:10.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:10.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:11.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:11.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:11.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:10 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:11.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:10 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:11.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:10 smithi032 ceph-mon[27388]: osdmap e661: 12 total, 12 up, 11 in 2024-09-16T05:20:11.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:10 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:10 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:10 smithi204 ceph-mon[30933]: osdmap e661: 12 total, 12 up, 11 in 2024-09-16T05:20:11.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:11.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:11.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:10 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:10 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:10 smithi196 ceph-mon[30802]: osdmap e661: 12 total, 12 up, 11 in 2024-09-16T05:20:11.258 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:20:11.258 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:20:11.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:20:11.258 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:20:11.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.258 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:115 2.a e112/107 2:580e4d0e:::smithi20457400-29:head v 112'6, mlcod=112'3)", 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 567.92194958599998, 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0311933, 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334344+0000", 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334345+0000", 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.334350+0000", 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.887e-06 2024-09-16T05:20:11.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365213+0000", 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.030863313999999999 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365220+0000", 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7870000000000004e-06 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365291+0000", 2024-09-16T05:20:11.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1397999999999996e-05 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365307+0000", 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5990000000000001e-05 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365507+0000", 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000199221 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.365538+0000", 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1167000000000002e-05 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:127 2.a e112/107 2:5ae26c04:::smithi20457400-32:head v 112'9, mlcod=112'5)", 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 567.88656854800001, 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020225074999999999, 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369725+0000", 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369726+0000", 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9200000000000002e-07 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.369728+0000", 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9410000000000001e-06 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389627+0000", 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019899751 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389632+0000", 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9869999999999999e-06 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389704+0000", 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1418e-05 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389718+0000", 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.433e-05 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389922+0000", 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020421600000000001 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389951+0000", 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8140000000000002e-05 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.267 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:133 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'10, mlcod=112'7)", 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 567.86703462000003, 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074633729999999997, 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389259+0000", 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389261+0000", 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.973e-06 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.389613+0000", 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035121600000000001 2024-09-16T05:20:11.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395726+0000", 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006113851 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395731+0000", 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0359999999999997e-06 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395819+0000", 2024-09-16T05:20:11.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7128999999999999e-05 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395836+0000", 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7768999999999999e-05 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396693+0000", 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00085685999999999998 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.396723+0000", 2024-09-16T05:20:11.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9539e-05 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:135 2.a e112/107 2:5f916a06:::smithi20457400-34:head v 112'12, mlcod=112'8)", 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 567.86052364499994, 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069274319999999999, 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.272 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.273 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395770+0000", 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395771+0000", 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4499999999999997e-07 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.395775+0000", 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6509999999999999e-06 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402319+0000", 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065439469999999996 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402326+0000", 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5769999999999996e-06 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402424+0000", 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8171000000000006e-05 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402439+0000", 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4844999999999999e-05 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402660+0000", 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022083700000000001 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:10:43.402698+0000", 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7858999999999998e-05 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.277 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1133 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'206, mlcod=189'203)", 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 488.52206735099998, 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021225035, 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.85e-07 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734829+0000", 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060196800000000003 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734866+0000", 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7469000000000003e-05 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734881+0000", 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5058999999999999e-05 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734970+0000", 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9004000000000004e-05 2024-09-16T05:20:11.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734998+0000", 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7606e-05 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755384+0000", 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02038587 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755452+0000", 2024-09-16T05:20:11.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7873999999999994e-05 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1134 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'207, mlcod=189'203)", 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 488.51728222899999, 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016477325000000001, 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739014+0000", 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2819999999999999e-06 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739797+0000", 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078265599999999996 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739844+0000", 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7608000000000002e-05 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739854+0000", 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4239999999999999e-06 2024-09-16T05:20:11.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739935+0000", 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1593000000000003e-05 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739963+0000", 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7492e-05 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755459+0000", 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015496565 2024-09-16T05:20:11.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755489+0000", 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9705000000000001e-05 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1135 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'208, mlcod=189'203)", 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 488.512859392, 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012094011, 2024-09-16T05:20:11.286 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743436+0000", 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.584e-06 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743535+0000", 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9110000000000005e-05 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743568+0000", 2024-09-16T05:20:11.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2335000000000001e-05 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743576+0000", 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.191e-06 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743646+0000", 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0481000000000005e-05 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743676+0000", 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9620000000000001e-05 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755492+0000", 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011815888 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755529+0000", 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6801999999999998e-05 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1136 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'209, mlcod=189'203)", 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 488.51225183399998, 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011564019, 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:20:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744043+0000", 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3600000000000003e-07 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744049+0000", 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8250000000000001e-06 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744069+0000", 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0460000000000001e-05 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744074+0000", 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4730000000000002e-06 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744119+0000", 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744143+0000", 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4085999999999999e-05 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755570+0000", 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011427299 2024-09-16T05:20:11.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755606+0000", 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146000000000003e-05 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2777 2.48 e359/322 2:12e92f6c:::smithi20457400-1:head v 359'155, mlcod=0'0)", 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 302.504955671, 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007754843, 2024-09-16T05:20:11.295 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751339+0000", 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5400000000000002e-07 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755560+0000", 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004221341 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755638+0000", 2024-09-16T05:20:11.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8292000000000002e-05 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755652+0000", 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3702e-05 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755795+0000", 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014327200000000001 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755823+0000", 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7668e-05 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.759023+0000", 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003199959 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.759093+0000", 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0355e-05 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:606 2.48 e365/322 2:12e92f6c:::smithi20457400-1:79 v 365'161, mlcod=362'157)", 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 296.18515849800002, 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64187130800000003, 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:20:11.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071139+0000", 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5609999999999999e-06 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712578+0000", 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64143852199999996 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712582+0000", 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-06 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712651+0000", 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9102000000000003e-05 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712670+0000", 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9165999999999999e-05 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712968+0000", 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029722899999999998 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.713007+0000", 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9125999999999998e-05 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:614 2.48 e366/322 2:12e92f6c:::smithi20457400-1:79 v 366'163, mlcod=364'159)", 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 295.53573126800001, 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11945663500000001, 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.304 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9699999999999996e-07 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720568+0000", 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6870000000000002e-06 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839371+0000", 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11880265399999999 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839384+0000", 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3156000000000001e-05 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839531+0000", 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014728399999999999 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839575+0000", 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4223e-05 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839981+0000", 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040524499999999999 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.840019+0000", 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8788999999999997e-05 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.308 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3235 2.29 e406/253 2:9662c29c:::smithi20457400-9:head v 406'547, mlcod=402'543)", 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 255.526043762, 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081876009999999992, 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730252+0000", 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.857e-06 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734729+0000", 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044764510000000002 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734790+0000", 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1465999999999997e-05 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734803+0000", 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2729e-05 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734890+0000", 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7708999999999997e-05 2024-09-16T05:20:11.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734938+0000", 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7485999999999999e-05 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.738369+0000", 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034306649999999998 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.738438+0000", 2024-09-16T05:20:11.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9238000000000004e-05 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3412 2.3e e424/296 2:7e4e65a4:::smithi20457400-4:head v 424'357, mlcod=378'353)", 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 237.318828134, 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007702319, 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9000000000000001e-07 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942005+0000", 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045392510000000002 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942060+0000", 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4716000000000003e-05 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942070+0000", 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0078000000000001e-05 2024-09-16T05:20:11.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942135+0000", 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4853999999999999e-05 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942159+0000", 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3839999999999999e-05 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.945112+0000", 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029533789999999999 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.945168+0000", 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6011000000000003e-05 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4629 2.1e 2:7b642727:::smithi20457400-40:head [copy-from ver 511] snapc bf=[bf,be] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:20:11.317 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 149.50376975, 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01647931, 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4629 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:20:11.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752526+0000", 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.894e-06 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752536+0000", 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752555+0000", 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8907999999999999e-05 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752564+0000", 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752638+0000", 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3343999999999994e-05 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752646+0000", 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7690000000000002e-06 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754646+0000", 2024-09-16T05:20:11.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001999331 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754740+0000", 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3778999999999998e-05 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.764207+0000", 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094672899999999997 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768754+0000", 2024-09-16T05:20:11.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045470199999999997 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768860+0000", 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010586600000000001 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768948+0000", 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8503000000000003e-05 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.769004+0000", 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5362000000000003e-05 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4630 2.1e 2:7b642727:::smithi20457400-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 149.50371200000001, 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015483190000000001, 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.324 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4630 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:20:11.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.34e-07 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752584+0000", 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.956e-06 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752591+0000", 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2580000000000001e-06 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752594+0000", 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6830000000000001e-06 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752703+0000", 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108818 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752708+0000", 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.561e-06 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754913+0000", 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022055680000000002 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754965+0000", 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2222000000000002e-05 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767827+0000", 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012861097 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767929+0000", 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010275799999999999 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768065+0000", 2024-09-16T05:20:11.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:11.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013593500000000001 2024-09-16T05:20:11.331 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.331 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.331 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.331 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:859 2.3e e561/296 2:7e4e65a4:::smithi20457400-4:d0 v 561'447, mlcod=557'443)", 2024-09-16T05:20:11.331 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 100.491348892, 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007143066, 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2300000000000002e-07 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764950+0000", 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771606+0000", 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006656103 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771614+0000", 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.712e-06 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771652+0000", 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8497000000000001e-05 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771668+0000", 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5829e-05 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.772021+0000", 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035285500000000001 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.772088+0000", 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7297000000000003e-05 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.336 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5275 2.a e561/320 2:5089ebe1:::smithi20457400-36:head v 561'529, mlcod=526'525)", 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 100.34804479899999, 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069804639999999996, 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.337 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:20:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908251+0000", 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4360000000000001e-06 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912064+0000", 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038136189999999999 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912103+0000", 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8689999999999997e-05 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912115+0000", 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.19e-05 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912182+0000", 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7565999999999997e-05 2024-09-16T05:20:11.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912205+0000", 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2220000000000001e-05 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.915180+0000", 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002975536 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.915230+0000", 2024-09-16T05:20:11.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9496999999999997e-05 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:142 2.1e e571/291 ondisk, result = 0)", 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 89.492135845000007, 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55541789799999997, 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:11.342 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764159+0000", 2024-09-16T05:20:11.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764162+0000", 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599999999999998e-06 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319278+0000", 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55511595899999999 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319296+0000", 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7900000000000001e-05 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319440+0000", 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143175 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319450+0000", 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0705e-05 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319576+0000", 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012584299999999999 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5587 2.34 e583/278 2:2d3f13f8:::smithi20457400-7:head v 583'466, mlcod=583'463)", 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 78.449189493999995, 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14534661199999999, 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.346 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807109+0000", 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2030000000000002e-06 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951917+0000", 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14480800699999999 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951927+0000", 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0115000000000001e-05 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951990+0000", 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2689999999999998e-05 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952011+0000", 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.084e-05 2024-09-16T05:20:11.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952413+0000", 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040190199999999999 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952451+0000", 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8302e-05 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.350 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:383 2.29 e617/253 2:9662c29c:::smithi20457400-9:d8 v 617'830, mlcod=616'824)", 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 44.473501313, 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038031031999999999, 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782794+0000", 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3500000000000005e-07 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782801+0000", 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1210000000000001e-06 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819928+0000", 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037126740999999998 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819938+0000", 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0845e-05 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820001+0000", 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2786999999999994e-05 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820023+0000", 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.213e-05 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820771+0000", 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074818699999999998 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820824+0000", 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2286e-05 2024-09-16T05:20:11.354 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.355 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.355 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.355 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:11.355 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:11.355 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:20:11.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:11.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:11.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:11.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:11.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:11.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:11.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:11.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:11.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:11.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:11.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:11.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:20:11.788 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T05:20:11.793 INFO:tasks.rados.rados.2.smithi204.stdout:2567: done (1 left) 2024-09-16T05:20:11.793 INFO:tasks.rados.rados.2.smithi204.stdout:2568: done (0 left) 2024-09-16T05:20:11.793 INFO:tasks.rados.rados.2.smithi204.stdout:2569: read oid 20 snap 261 2024-09-16T05:20:11.793 INFO:tasks.rados.rados.2.smithi204.stdout:2569: expect (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:20:11.793 INFO:tasks.rados.rados.2.smithi204.stdout:2570: snap_create 2024-09-16T05:20:11.798 INFO:tasks.rados.rados.2.smithi204.stdout:2569: expect (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:20:11.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:11.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:11.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:11.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:12.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:12.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:11 smithi032 ceph-mon[27388]: pgmap v1152: 107 pgs: 107 active+clean; 264 MiB data, 811 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 346 KiB/s wr, 2 op/s 2024-09-16T05:20:12.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3275913798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:20:12.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3275913798' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:20:12.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:11 smithi032 ceph-mon[27388]: osdmap e662: 12 total, 12 up, 11 in 2024-09-16T05:20:12.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:20:12.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:12.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:12.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:12.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:11 smithi204 ceph-mon[30933]: pgmap v1152: 107 pgs: 107 active+clean; 264 MiB data, 811 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 346 KiB/s wr, 2 op/s 2024-09-16T05:20:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3275913798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:20:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3275913798' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:20:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:11 smithi204 ceph-mon[30933]: osdmap e662: 12 total, 12 up, 11 in 2024-09-16T05:20:12.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:11 smithi196 ceph-mon[30802]: pgmap v1152: 107 pgs: 107 active+clean; 264 MiB data, 811 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 346 KiB/s wr, 2 op/s 2024-09-16T05:20:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3275913798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:20:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3275913798' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:20:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:11 smithi196 ceph-mon[30802]: osdmap e662: 12 total, 12 up, 11 in 2024-09-16T05:20:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:12.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:12.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:12.326 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:12.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:12.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:12.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:12.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:12.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:12.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:12.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:20:12.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:12.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:12.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:12.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:12.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:12.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:12.794 INFO:tasks.rados.rados.2.smithi204.stdout:2569: done (1 left) 2024-09-16T05:20:12.795 INFO:tasks.rados.rados.2.smithi204.stdout:2570: done (0 left) 2024-09-16T05:20:12.795 INFO:tasks.rados.rados.2.smithi204.stdout:2571: snap_create 2024-09-16T05:20:12.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-16T05:20:12.823 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:20:12.823 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:20:12.823 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:20:12.823 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:20:12.849 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:12.851 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:12.861 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:12.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:12.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:12.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:13.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:13.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:13.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:13.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:13.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:13.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:13.253 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:13.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:13.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:13.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:13.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:13.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:20:13.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:13.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:13.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:13.516 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:13.517 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:13.538 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:13.538 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:13.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:13.559 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:13.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:13.579 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:13.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:13.600 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:13.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:13.623 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:13.623 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:13.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:13.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:13.645 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:13.645 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:13.668 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:13.668 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:13.690 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:13.690 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:13.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:13.713 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:13.719 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:13.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:13.735 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:13.735 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:13.752 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:13.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:13.757 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:13.758 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:13.764 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:13.798 INFO:tasks.rados.rados.2.smithi204.stdout:2571: done (0 left) 2024-09-16T05:20:13.798 INFO:tasks.rados.rados.2.smithi204.stdout:2572: rollback oid 37 current snap is 271 2024-09-16T05:20:13.798 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 270 2024-09-16T05:20:13.798 INFO:tasks.rados.rados.2.smithi204.stdout:2573: read oid 16 snap -1 2024-09-16T05:20:13.798 INFO:tasks.rados.rados.2.smithi204.stdout:2573: expect (ObjNum 864 snap 249 seq_num 864) 2024-09-16T05:20:13.798 INFO:tasks.rados.rados.2.smithi204.stdout:2574: write oid 48 current snap is 271 2024-09-16T05:20:13.798 INFO:tasks.rados.rados.2.smithi204.stdout:2574: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-09-16T05:20:13.803 INFO:tasks.rados.rados.2.smithi204.stdout:2574: writing smithi20457400-48 from 460756 to 1032855 tid 1 2024-09-16T05:20:13.809 INFO:tasks.rados.rados.2.smithi204.stdout:2574: writing smithi20457400-48 from 1517297 to 1961609 tid 2 2024-09-16T05:20:13.816 INFO:tasks.rados.rados.2.smithi204.stdout:2574: writing smithi20457400-48 from 2668548 to 3141418 tid 3 2024-09-16T05:20:13.816 INFO:tasks.rados.rados.2.smithi204.stdout:2575: delete oid 21 current snap is 271 2024-09-16T05:20:13.816 INFO:tasks.rados.rados.2.smithi204.stdout:2572: finishing rollback tid 0 to smithi20457400-37 2024-09-16T05:20:13.816 INFO:tasks.rados.rados.2.smithi204.stdout:2572: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:20:13.816 INFO:tasks.rados.rados.2.smithi204.stdout:2573: expect (ObjNum 864 snap 249 seq_num 864) 2024-09-16T05:20:13.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:13.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:13.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:13.900 INFO:tasks.rados.rados.2.smithi204.stdout:2572: finishing rollback tid 2 to smithi20457400-37 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1021 (ObjNum 694 snap 201 seq_num 694) dirty exists 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2574: finishing write tid 1 to smithi20457400-48 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2574: finishing write tid 2 to smithi20457400-48 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2572: done (3 left) 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2573: done (2 left) 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2575: done (1 left) 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2576: rollback oid 25 current snap is 271 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 262 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2574: finishing write tid 3 to smithi20457400-48 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2574: finishing write tid 4 to smithi20457400-48 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2574: finishing write tid 5 to smithi20457400-48 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 772 (ObjNum 908 snap 271 seq_num 908) dirty exists 2024-09-16T05:20:13.901 INFO:tasks.rados.rados.2.smithi204.stdout:2574: left oid 48 (ObjNum 908 snap 271 seq_num 908) 2024-09-16T05:20:13.902 INFO:tasks.rados.rados.2.smithi204.stdout:2574: done (1 left) 2024-09-16T05:20:13.902 INFO:tasks.rados.rados.2.smithi204.stdout:2577: read oid 20 snap -1 2024-09-16T05:20:13.902 INFO:tasks.rados.rados.2.smithi204.stdout:2577: expect (ObjNum 886 snap 262 seq_num 886) 2024-09-16T05:20:13.902 INFO:tasks.rados.rados.2.smithi204.stdout:2578: read oid 14 snap -1 2024-09-16T05:20:13.902 INFO:tasks.rados.rados.2.smithi204.stdout:2578: expect (ObjNum 896 snap 262 seq_num 896) 2024-09-16T05:20:13.902 INFO:tasks.rados.rados.2.smithi204.stdout:2579: write (excl) oid 48 current snap is 271 2024-09-16T05:20:13.902 INFO:tasks.rados.rados.2.smithi204.stdout:2579: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2024-09-16T05:20:13.908 INFO:tasks.rados.rados.2.smithi204.stdout:2579: writing smithi20457400-48 from 494370 to 1216967 tid 1 2024-09-16T05:20:13.913 INFO:tasks.rados.rados.2.smithi204.stdout:2579: writing smithi20457400-48 from 1917908 to 2365889 tid 2 2024-09-16T05:20:13.918 INFO:tasks.rados.rados.2.smithi204.stdout:2579: writing smithi20457400-48 from 3007041 to 3294370 tid 3 2024-09-16T05:20:13.919 INFO:tasks.rados.rados.2.smithi204.stdout:2580: write oid 8 current snap is 271 2024-09-16T05:20:13.919 INFO:tasks.rados.rados.2.smithi204.stdout:2580: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2024-09-16T05:20:13.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:13 smithi032 ceph-mon[27388]: pgmap v1155: 107 pgs: 107 active+clean; 263 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 321 KiB/s wr, 2 op/s 2024-09-16T05:20:13.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:13 smithi032 ceph-mon[27388]: osdmap e663: 12 total, 12 up, 11 in 2024-09-16T05:20:13.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:13 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:13.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:13 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:13.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/563258314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:13.925 INFO:tasks.rados.rados.2.smithi204.stdout:2580: writing smithi20457400-8 from 511177 to 1309023 tid 1 2024-09-16T05:20:13.932 INFO:tasks.rados.rados.2.smithi204.stdout:2580: writing smithi20457400-8 from 1976390 to 2568029 tid 2 2024-09-16T05:20:13.937 INFO:tasks.rados.rados.2.smithi204.stdout:2580: writing smithi20457400-8 from 3034464 to 3311177 tid 3 2024-09-16T05:20:13.937 INFO:tasks.rados.rados.2.smithi204.stdout:2581: write oid 50 current snap is 271 2024-09-16T05:20:13.937 INFO:tasks.rados.rados.2.smithi204.stdout:2581: seq_num 912 ranges {527984=473095,1634872=451650,2661887=634311} 2024-09-16T05:20:13.942 INFO:tasks.rados.rados.2.smithi204.stdout:2581: writing smithi20457400-50 from 527984 to 1001079 tid 1 2024-09-16T05:20:13.947 INFO:tasks.rados.rados.2.smithi204.stdout:2581: writing smithi20457400-50 from 1634872 to 2086522 tid 2 2024-09-16T05:20:13.952 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:13 smithi196 ceph-mon[30802]: pgmap v1155: 107 pgs: 107 active+clean; 263 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 321 KiB/s wr, 2 op/s 2024-09-16T05:20:13.952 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:13 smithi196 ceph-mon[30802]: osdmap e663: 12 total, 12 up, 11 in 2024-09-16T05:20:13.952 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:13 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:13.952 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:13 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:13.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/563258314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:13.953 INFO:tasks.rados.rados.2.smithi204.stdout:2581: writing smithi20457400-50 from 2661887 to 3296198 tid 3 2024-09-16T05:20:13.954 INFO:tasks.rados.rados.2.smithi204.stdout:2582: snap_remove snap 258 2024-09-16T05:20:13.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:14.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:14.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:14.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:13 smithi204 ceph-mon[30933]: pgmap v1155: 107 pgs: 107 active+clean; 263 MiB data, 794 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 321 KiB/s wr, 2 op/s 2024-09-16T05:20:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:13 smithi204 ceph-mon[30933]: osdmap e663: 12 total, 12 up, 11 in 2024-09-16T05:20:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:13 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:13 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/563258314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:14.195 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:14.195 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:14.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:14.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:14.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:14.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.371 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:14.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:14.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:14.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:14.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:14.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:14.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:20:14.800 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:20:14.805 INFO:tasks.rados.rados.2.smithi204.stdout:2582: done (6 left) 2024-09-16T05:20:14.805 INFO:tasks.rados.rados.2.smithi204.stdout:2583: copy_from oid 12 from oid 3 current snap is 271 2024-09-16T05:20:14.805 INFO:tasks.rados.rados.2.smithi204.stdout:2576: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:20:14.805 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 732 (ObjNum 593 snap 0 seq_num 4282190744) dirty dne 2024-09-16T05:20:14.805 INFO:tasks.rados.rados.2.smithi204.stdout:2578: expect (ObjNum 896 snap 262 seq_num 896) 2024-09-16T05:20:14.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:14.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:14 smithi032 ceph-mon[27388]: osdmap e664: 12 total, 12 up, 11 in 2024-09-16T05:20:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/867036535' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:14.961 INFO:tasks.rados.rados.2.smithi204.stdout:2577: expect (ObjNum 886 snap 262 seq_num 886) 2024-09-16T05:20:14.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:14.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:14 smithi196 ceph-mon[30802]: osdmap e664: 12 total, 12 up, 11 in 2024-09-16T05:20:14.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/867036535' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:14.993 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:14.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:15.012 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:15.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2579: finishing write tid 1 to smithi20457400-48 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2579: finishing write tid 2 to smithi20457400-48 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2579: finishing write tid 3 to smithi20457400-48 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2580: finishing write tid 1 to smithi20457400-8 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2580: finishing write tid 2 to smithi20457400-8 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2580: finishing write tid 3 to smithi20457400-8 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2580: finishing write tid 4 to smithi20457400-8 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2580: finishing write tid 5 to smithi20457400-8 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 955 (ObjNum 911 snap 271 seq_num 911) dirty exists 2024-09-16T05:20:15.108 INFO:tasks.rados.rados.2.smithi204.stdout:2580: left oid 8 (ObjNum 911 snap 271 seq_num 911) 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2581: finishing write tid 1 to smithi20457400-50 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2576: done (6 left) 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2577: done (5 left) 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2578: done (4 left) 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2580: done (3 left) 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2584: rollback oid 37 current snap is 271 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 265 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2581: finishing write tid 2 to smithi20457400-50 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2581: finishing write tid 3 to smithi20457400-50 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2581: finishing write tid 4 to smithi20457400-50 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2581: finishing write tid 5 to smithi20457400-50 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1109 (ObjNum 912 snap 271 seq_num 912) dirty exists 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2581: left oid 50 (ObjNum 912 snap 271 seq_num 912) 2024-09-16T05:20:15.109 INFO:tasks.rados.rados.2.smithi204.stdout:2579: finishing write tid 4 to smithi20457400-48 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:2579: finishing write tid 5 to smithi20457400-48 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 776 (ObjNum 910 snap 271 seq_num 910) dirty exists 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:2579: left oid 48 (ObjNum 910 snap 271 seq_num 910) 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:2583: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:2583: finishing copy_from to smithi20457400-12 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 913 (ObjNum 875 snap 255 seq_num 875) dirty exists 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:2579: done (3 left) 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:2581: done (2 left) 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:2583: done (1 left) 2024-09-16T05:20:15.110 INFO:tasks.rados.rados.2.smithi204.stdout:2585: snap_remove snap 229 2024-09-16T05:20:15.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:15.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:15.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:14 smithi204 ceph-mon[30933]: osdmap e664: 12 total, 12 up, 11 in 2024-09-16T05:20:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/867036535' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:15.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:15.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:15.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:15.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:15.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:15.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:15.445 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:15.445 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:15.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:15.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:15.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:15.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:15.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:15.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:15.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:15.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:15.788 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T05:20:15.792 INFO:tasks.rados.rados.2.smithi204.stdout:2585: done (1 left) 2024-09-16T05:20:15.792 INFO:tasks.rados.rados.2.smithi204.stdout:2586: snap_remove snap 270 2024-09-16T05:20:15.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:15.863 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:15.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:15.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:15.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:15.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:16.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:16.112 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:16.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:16.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: pgmap v1158: 107 pgs: 107 active+clean; 271 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 6.8 MiB/s rd, 1019 KiB/s wr, 10 op/s 2024-09-16T05:20:16.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/867036535' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: osdmap e665: 12 total, 12 up, 11 in 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1811630765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1811630765' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:20:16.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:15 smithi032 ceph-mon[27388]: osdmap e666: 12 total, 12 up, 11 in 2024-09-16T05:20:16.125 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: pgmap v1158: 107 pgs: 107 active+clean; 271 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 6.8 MiB/s rd, 1019 KiB/s wr, 10 op/s 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/867036535' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: osdmap e665: 12 total, 12 up, 11 in 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1811630765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1811630765' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:20:16.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:15 smithi196 ceph-mon[30802]: osdmap e666: 12 total, 12 up, 11 in 2024-09-16T05:20:16.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:20:16.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: pgmap v1158: 107 pgs: 107 active+clean; 271 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 6.8 MiB/s rd, 1019 KiB/s wr, 10 op/s 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/867036535' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: osdmap e665: 12 total, 12 up, 11 in 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1811630765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1811630765' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:20:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:15 smithi204 ceph-mon[30933]: osdmap e666: 12 total, 12 up, 11 in 2024-09-16T05:20:16.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:16.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:16.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:16.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:16.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:16.395 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:16.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:16.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:16.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:16.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:16.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-16T05:20:16.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:16.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:16.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:16.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:20:16.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:16.742 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:16.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:16.794 INFO:tasks.rados.rados.2.smithi204.stdout:2586: done (1 left) 2024-09-16T05:20:16.794 INFO:tasks.rados.rados.2.smithi204.stdout:2587: rollback oid 40 current snap is 271 2024-09-16T05:20:16.794 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 40 to 264 2024-09-16T05:20:16.794 INFO:tasks.rados.rados.2.smithi204.stdout:2588: write oid 12 current snap is 271 2024-09-16T05:20:16.794 INFO:tasks.rados.rados.2.smithi204.stdout:2588: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-09-16T05:20:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:16.800 INFO:tasks.rados.rados.2.smithi204.stdout:2588: writing smithi20457400-12 from 544791 to 1209488 tid 1 2024-09-16T05:20:16.806 INFO:tasks.rados.rados.2.smithi204.stdout:2588: writing smithi20457400-12 from 1693354 to 2288662 tid 2 2024-09-16T05:20:16.811 INFO:tasks.rados.rados.2.smithi204.stdout:2588: writing smithi20457400-12 from 2689310 to 3134893 tid 3 2024-09-16T05:20:16.811 INFO:tasks.rados.rados.2.smithi204.stdout:2589: rollback oid 34 current snap is 271 2024-09-16T05:20:16.811 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 260 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2584: finishing rollback tid 0 to smithi20457400-37 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2584: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2584: finishing rollback tid 2 to smithi20457400-37 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1024 (ObjNum 694 snap 201 seq_num 694) dirty exists 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2587: finishing rollback tid 0 to smithi20457400-40 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2587: finishing rollback tid 1 to smithi20457400-40 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2587: finishing rollback tid 2 to smithi20457400-40 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 777 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2588: finishing write tid 1 to smithi20457400-12 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2588: finishing write tid 2 to smithi20457400-12 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2584: done (3 left) 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2587: done (2 left) 2024-09-16T05:20:16.812 INFO:tasks.rados.rados.2.smithi204.stdout:2590: write (excl) oid 16 current snap is 271 2024-09-16T05:20:16.813 INFO:tasks.rados.rados.2.smithi204.stdout:2590: seq_num 914 ranges {561598=739946,1751836=455319,2716733=644865} 2024-09-16T05:20:16.818 INFO:tasks.rados.rados.2.smithi204.stdout:2590: writing smithi20457400-16 from 561598 to 1301544 tid 1 2024-09-16T05:20:16.823 INFO:tasks.rados.rados.2.smithi204.stdout:2590: writing smithi20457400-16 from 1751836 to 2207155 tid 2 2024-09-16T05:20:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:2590: writing smithi20457400-16 from 2716733 to 3361598 tid 3 2024-09-16T05:20:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:2591: snap_create 2024-09-16T05:20:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:2589: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:20:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 707 (ObjNum 353 snap 0 seq_num 2944376368) dirty dne 2024-09-16T05:20:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:2588: finishing write tid 3 to smithi20457400-12 2024-09-16T05:20:16.831 INFO:tasks.rados.rados.2.smithi204.stdout:2588: finishing write tid 4 to smithi20457400-12 2024-09-16T05:20:16.831 INFO:tasks.rados.rados.2.smithi204.stdout:2588: finishing write tid 5 to smithi20457400-12 2024-09-16T05:20:16.831 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 983 (ObjNum 913 snap 271 seq_num 913) dirty exists 2024-09-16T05:20:16.831 INFO:tasks.rados.rados.2.smithi204.stdout:2588: left oid 12 (ObjNum 913 snap 271 seq_num 913) 2024-09-16T05:20:16.831 INFO:tasks.rados.rados.2.smithi204.stdout:2590: finishing write tid 1 to smithi20457400-16 2024-09-16T05:20:16.832 INFO:tasks.rados.rados.2.smithi204.stdout:2590: finishing write tid 2 to smithi20457400-16 2024-09-16T05:20:16.832 INFO:tasks.rados.rados.2.smithi204.stdout:2590: finishing write tid 3 to smithi20457400-16 2024-09-16T05:20:16.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:16.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:16.867 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:16.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:16.889 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:16.890 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:16.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:16.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:16.991 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:16.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:17.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:16 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:20:17.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:16 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:20:17.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:16 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:20:17.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:16 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:20:17.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:16 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:20:17.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:16 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:20:17.062 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:16 smithi032 ceph-mon[27388]: osdmap e667: 12 total, 12 up, 11 in 2024-09-16T05:20:17.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:17.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:17.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:17.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:16 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:20:17.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:16 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:20:17.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:16 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:20:17.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:16 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:20:17.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:16 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:20:17.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:16 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:20:17.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:16 smithi196 ceph-mon[30802]: osdmap e667: 12 total, 12 up, 11 in 2024-09-16T05:20:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:16 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:20:17.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:16 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:20:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:16 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:20:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:16 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:20:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:16 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:20:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:16 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:20:17.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:16 smithi204 ceph-mon[30933]: osdmap e667: 12 total, 12 up, 11 in 2024-09-16T05:20:17.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:17.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:17.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:17.289 INFO:tasks.rados.rados.2.smithi204.stdout:2590: finishing write tid 4 to smithi20457400-16 2024-09-16T05:20:17.289 INFO:tasks.rados.rados.2.smithi204.stdout:2590: finishing write tid 5 to smithi20457400-16 2024-09-16T05:20:17.290 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 901 (ObjNum 914 snap 271 seq_num 914) dirty exists 2024-09-16T05:20:17.290 INFO:tasks.rados.rados.2.smithi204.stdout:2590: left oid 16 (ObjNum 914 snap 271 seq_num 914) 2024-09-16T05:20:17.322 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:17.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:17.362 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:17.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:17.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:20:17.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:17.487 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:17.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:17.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:20:17.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:17.610 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:17.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:17.703 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:17.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:17.737 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:17.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:17.767 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:17.768 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:17.789 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:17.789 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:17.810 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T05:20:17.814 INFO:tasks.rados.rados.2.smithi204.stdout:2588: done (3 left) 2024-09-16T05:20:17.814 INFO:tasks.rados.rados.2.smithi204.stdout:2589: done (2 left) 2024-09-16T05:20:17.814 INFO:tasks.rados.rados.2.smithi204.stdout:2590: done (1 left) 2024-09-16T05:20:17.814 INFO:tasks.rados.rados.2.smithi204.stdout:2591: done (0 left) 2024-09-16T05:20:17.814 INFO:tasks.rados.rados.2.smithi204.stdout:2592: copy_from oid 16 from oid 42 current snap is 272 2024-09-16T05:20:17.814 INFO:tasks.rados.rados.2.smithi204.stdout:2593: delete oid 8 current snap is 272 2024-09-16T05:20:17.815 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:17.815 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:17.818 INFO:tasks.rados.rados.2.smithi204.stdout:2592: finishing copy_from to smithi20457400-16 2024-09-16T05:20:17.818 INFO:tasks.rados.rados.2.smithi204.stdout:2592: got expected ENOENT (src dne) 2024-09-16T05:20:17.818 INFO:tasks.rados.rados.2.smithi204.stdout:2592: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:20:17.821 INFO:tasks.rados.rados.2.smithi204.stdout:2592: done (1 left) 2024-09-16T05:20:17.821 INFO:tasks.rados.rados.2.smithi204.stdout:2593: done (0 left) 2024-09-16T05:20:17.821 INFO:tasks.rados.rados.2.smithi204.stdout:2594: copy_from oid 43 from oid 11 current snap is 272 2024-09-16T05:20:17.821 INFO:tasks.rados.rados.2.smithi204.stdout:2595: read oid 19 snap 255 2024-09-16T05:20:17.822 INFO:tasks.rados.rados.2.smithi204.stdout:2595: expect (ObjNum 797 snap 234 seq_num 797) 2024-09-16T05:20:17.822 INFO:tasks.rados.rados.2.smithi204.stdout:2596: snap_create 2024-09-16T05:20:17.824 INFO:tasks.rados.rados.2.smithi204.stdout:2595: expect (ObjNum 797 snap 234 seq_num 797) 2024-09-16T05:20:17.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:17.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:17.860 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:17.861 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:17.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:17.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:17.882 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:17.882 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:17.903 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:17.904 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:17.925 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:17.926 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:17.947 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:17.947 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:17.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:17.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:17.968 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:17.980 INFO:tasks.rados.rados.2.smithi204.stdout:2594: finishing copy_from to smithi20457400-43 2024-09-16T05:20:17.980 INFO:tasks.rados.rados.2.smithi204.stdout:2594: got expected ENOENT (src dne) 2024-09-16T05:20:17.980 INFO:tasks.rados.rados.2.smithi204.stdout:2594: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:20:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:17.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:17.990 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:18.010 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:18.011 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:18.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:17 smithi032 ceph-mon[27388]: pgmap v1161: 107 pgs: 107 active+clean; 267 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 9.0 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2024-09-16T05:20:18.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:17 smithi032 ceph-mon[27388]: 2.4d scrub starts 2024-09-16T05:20:18.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:17 smithi032 ceph-mon[27388]: 2.4d scrub ok 2024-09-16T05:20:18.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:17 smithi032 ceph-mon[27388]: 2.4c scrub starts 2024-09-16T05:20:18.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:17 smithi032 ceph-mon[27388]: 2.4c scrub ok 2024-09-16T05:20:18.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2176063897' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:20:18.011 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1650833172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:18.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:17 smithi196 ceph-mon[30802]: pgmap v1161: 107 pgs: 107 active+clean; 267 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 9.0 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2024-09-16T05:20:18.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:17 smithi196 ceph-mon[30802]: 2.4d scrub starts 2024-09-16T05:20:18.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:17 smithi196 ceph-mon[30802]: 2.4d scrub ok 2024-09-16T05:20:18.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:17 smithi196 ceph-mon[30802]: 2.4c scrub starts 2024-09-16T05:20:18.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:17 smithi196 ceph-mon[30802]: 2.4c scrub ok 2024-09-16T05:20:18.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2176063897' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:20:18.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1650833172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:18.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:18.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:18.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:17 smithi204 ceph-mon[30933]: pgmap v1161: 107 pgs: 107 active+clean; 267 MiB data, 804 MiB used, 983 GiB / 983 GiB avail; 9.0 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2024-09-16T05:20:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:17 smithi204 ceph-mon[30933]: 2.4d scrub starts 2024-09-16T05:20:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:17 smithi204 ceph-mon[30933]: 2.4d scrub ok 2024-09-16T05:20:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:17 smithi204 ceph-mon[30933]: 2.4c scrub starts 2024-09-16T05:20:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:17 smithi204 ceph-mon[30933]: 2.4c scrub ok 2024-09-16T05:20:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2176063897' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:20:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1650833172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:18.188 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:20:18.188 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:20:18.188 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:20:18.188 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:20:18.189 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:20:18.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:18.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:18.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:18.361 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:18.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:18.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:18.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:18.586 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:18.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:18.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:18.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:18.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:18.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:18.829 INFO:tasks.rados.rados.2.smithi204.stdout:2594: done (2 left) 2024-09-16T05:20:18.829 INFO:tasks.rados.rados.2.smithi204.stdout:2595: done (1 left) 2024-09-16T05:20:18.829 INFO:tasks.rados.rados.2.smithi204.stdout:2596: done (0 left) 2024-09-16T05:20:18.829 INFO:tasks.rados.rados.2.smithi204.stdout:2597: delete oid 12 current snap is 273 2024-09-16T05:20:18.832 INFO:tasks.rados.rados.2.smithi204.stdout:2597: done (0 left) 2024-09-16T05:20:18.833 INFO:tasks.rados.rados.2.smithi204.stdout:2598: rollback oid 49 current snap is 273 2024-09-16T05:20:18.833 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 263 2024-09-16T05:20:18.833 INFO:tasks.rados.rados.2.smithi204.stdout:2599: read oid 4 snap -1 2024-09-16T05:20:18.833 INFO:tasks.rados.rados.2.smithi204.stdout:2599: expect deleted 2024-09-16T05:20:18.833 INFO:tasks.rados.rados.2.smithi204.stdout:2600: snap_remove snap 271 2024-09-16T05:20:18.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:18.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:18.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:20:18.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:20:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:20:18.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:20:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:20:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2176063897' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:20:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: osdmap e668: 12 total, 12 up, 11 in 2024-09-16T05:20:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:20:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:20:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:20:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:18 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:20:18.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:18.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:18.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:19.084 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:19.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:19.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:19.142 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2176063897' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: osdmap e668: 12 total, 12 up, 11 in 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:20:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:18 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:20:19.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-16T05:20:19.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:19.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:19.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:19.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:19.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2176063897' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: osdmap e668: 12 total, 12 up, 11 in 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:20:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:18 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:20:19.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:19.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:19.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:19.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:19.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:19.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:19.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:19.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:19.672 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:20:19.672 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:20:19.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1378 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 207] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 477.07908133699999, 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018067625, 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.673 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1378 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589519+0000", 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3460000000000001e-06 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589547+0000", 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8572999999999999e-05 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589577+0000", 2024-09-16T05:20:19.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9822e-05 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589592+0000", 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4630000000000001e-05 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589745+0000", 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015316700000000001 2024-09-16T05:20:19.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589921+0000", 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000175995 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.602330+0000", 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012408878 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.677 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605723+0000", 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033928000000000001 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607502+0000", 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017799140000000001 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607546+0000", 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3130999999999998e-05 2024-09-16T05:20:19.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607584+0000", 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8368999999999998e-05 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(2.34 e278 from shard 3)", 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 408.801620958, 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34868665500000001, 2024-09-16T05:20:19.679 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.680 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.680 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:20:19.680 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.680 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.680 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:20:19.680 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9999999999999997e-07 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866983+0000", 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.84e-06 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215470+0000", 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34848719900000003 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215480+0000", 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2129999999999997e-06 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.682 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215584+0000", 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010428400000000001 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215601+0000", 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7509000000000001e-05 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215663+0000", 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1909999999999995e-05 2024-09-16T05:20:19.683 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2256 2.34 2:2cffa4d6:::smithi20457400-50:head [copy-from ver 319] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 397.98694041900001, 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019683254000000001, 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.684 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2256 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.685 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681660+0000", 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5019999999999999e-06 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681670+0000", 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.91e-06 2024-09-16T05:20:19.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681686+0000", 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6699999999999999e-05 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681695+0000", 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9590000000000001e-06 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681793+0000", 2024-09-16T05:20:19.687 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7699000000000001e-05 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681800+0000", 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1199999999999996e-06 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.682948+0000", 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011475470000000001 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.688 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683035+0000", 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6886999999999997e-05 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.693853+0000", 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010818583 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.689 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698768+0000", 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049152340000000001 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701246+0000", 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024771630000000001 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701284+0000", 2024-09-16T05:20:19.690 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8544000000000002e-05 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701340+0000", 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6406000000000002e-05 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2511 2.3 2.e1e51803 (undecoded) ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e320)", 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 354.84552562300001, 2024-09-16T05:20:19.691 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99828526200000001, 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2511 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823073+0000", 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.04e-06 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.693 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823078+0000", 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6519999999999997e-06 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823091+0000", 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3277e-05 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823100+0000", 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4909999999999994e-06 2024-09-16T05:20:19.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823141+0000", 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0187e-05 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823147+0000", 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3929999999999996e-06 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.695 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821329+0000", 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99818146900000004 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821357+0000", 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8753000000000001e-05 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:136 2.34 e355/278 ondisk, result = 0)", 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 313.96682389699998, 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83858643600000005, 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701775+0000", 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6200000000000006e-07 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701786+0000", 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0815000000000001e-05 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540112+0000", 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83832645699999997 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540147+0000", 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5416000000000001e-05 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540246+0000", 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8171999999999995e-05 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540254+0000", 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540360+0000", 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010627400000000001 2024-09-16T05:20:19.700 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3152 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 301] snapc 8a=[8a,88,87,86,85,84,83,81,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 272.95628744300001, 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017546488999999998, 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.701 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3152 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712312+0000", 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4449999999999999e-06 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712319+0000", 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4000000000000003e-06 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712332+0000", 2024-09-16T05:20:19.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2928000000000001e-05 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712356+0000", 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4071999999999999e-05 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712421+0000", 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4709000000000007e-05 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712428+0000", 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2520000000000002e-06 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713705+0000", 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001276672 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713800+0000", 2024-09-16T05:20:19.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5266999999999994e-05 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.722741+0000", 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0089409339999999993 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.726991+0000", 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042500949999999997 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729789+0000", 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002798132 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729826+0000", 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6576999999999999e-05 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729857+0000", 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1006000000000001e-05 2024-09-16T05:20:19.707 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3573 2.3 2:c018a787:::smithi20457400-18:head [write 1692526~473837] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 233.80285576, 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022224677000000002, 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.708 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3573 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865743+0000", 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0100000000000001e-06 2024-09-16T05:20:19.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866150+0000", 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040748299999999999 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866163+0000", 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2187999999999999e-05 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866168+0000", 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4689999999999996e-06 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866251+0000", 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2911000000000006e-05 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866320+0000", 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8596000000000005e-05 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866403+0000", 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3796999999999997e-05 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867984+0000", 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015806399999999999 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887897+0000", 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019913157000000001 2024-09-16T05:20:19.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887934+0000", 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6482000000000001e-05 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887967+0000", 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2944000000000002e-05 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3574 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 2403500] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 233.80241620699999, 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039246627999999999, 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3574 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.715 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866182+0000", 2024-09-16T05:20:19.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866205+0000", 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2592e-05 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866222+0000", 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7422e-05 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866231+0000", 2024-09-16T05:20:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5220000000000007e-06 2024-09-16T05:20:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867407+0000", 2024-09-16T05:20:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001176014 2024-09-16T05:20:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867514+0000", 2024-09-16T05:20:19.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106947 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867661+0000", 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000147396 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867987+0000", 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032622400000000001 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905316+0000", 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037329222000000002 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905388+0000", 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1896000000000004e-05 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905428+0000", 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9830000000000003e-05 2024-09-16T05:20:19.722 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3575 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 233.80235820999999, 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039343647000000002, 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:20:19.723 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3575 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866240+0000", 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7999999999999995e-07 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866243+0000", 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5770000000000001e-06 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866249+0000", 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866254+0000", 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-06 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867919+0000", 2024-09-16T05:20:19.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016648609999999999 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867974+0000", 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4478000000000003e-05 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905464+0000", 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037490460000000003 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905516+0000", 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1894999999999997e-05 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905583+0000", 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3579 2.34 2:2cffa4d6:::smithi20457400-50:head [write 420307~462694] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 233.79776603100001, 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017271116999999999, 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3579 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.729 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870833+0000", 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2440000000000001e-06 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871089+0000", 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025604600000000001 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871102+0000", 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2955999999999999e-05 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871110+0000", 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2700000000000004e-06 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871161+0000", 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0899999999999997e-05 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871220+0000", 2024-09-16T05:20:19.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8659999999999997e-05 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871310+0000", 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0586000000000003e-05 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.872398+0000", 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010881409999999999 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888033+0000", 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015634834 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888071+0000", 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7957999999999998e-05 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888103+0000", 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1522000000000001e-05 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3580 2.34 2:2cffa4d6:::smithi20457400-50:head [write 1351008~617495] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 233.79205746599999, 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030886027999999999, 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.735 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3580 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876542+0000", 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877114+0000", 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057156899999999996 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877126+0000", 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2751999999999999e-05 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877134+0000", 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.072e-06 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0603000000000003e-05 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877240+0000", 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4561000000000002e-05 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877317+0000", 2024-09-16T05:20:19.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7002999999999994e-05 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878479+0000", 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011619130000000001 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907362+0000", 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028883039999999999 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907396+0000", 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4807000000000001e-05 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907426+0000", 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9821000000000001e-05 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3581 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2420307] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 233.79144667200001, 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030393973000000001, 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3581 2024-09-16T05:20:19.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877152+0000", 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3099999999999996e-07 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877156+0000", 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7589999999999999e-06 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877169+0000", 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3129000000000001e-05 2024-09-16T05:20:19.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877175+0000", 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6259999999999997e-06 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877997+0000", 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082123199999999997 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878081+0000", 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4914999999999996e-05 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878202+0000", 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120207 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878482+0000", 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027995500000000003 2024-09-16T05:20:19.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907462+0000", 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028980381 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907500+0000", 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907545+0000", 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5120999999999997e-05 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3582 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 233.79141226600001, 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030495267999999999, 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.748 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3582 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877186+0000", 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-07 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877189+0000", 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6800000000000002e-06 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877196+0000", 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0650000000000001e-06 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877202+0000", 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.28e-06 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878412+0000", 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001210205 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878470+0000", 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7759e-05 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907572+0000", 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029102086999999999 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907620+0000", 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8384000000000003e-05 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907681+0000", 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0244000000000001e-05 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.753 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:288 2.16 e446/273 ondisk, result = 0)", 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 222.78488090600001, 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65473014699999998, 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883718+0000", 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3799999999999996e-07 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883722+0000", 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.284e-06 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538184+0000", 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.654462031 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538201+0000", 2024-09-16T05:20:19.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6634e-05 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538327+0000", 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012605799999999999 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538338+0000", 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1188e-05 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538447+0000", 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109014 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4460 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 511] snapc bc=[bc,bb,ba,b8] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 166.60929698300001, 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019695292, 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.758 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4460 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9799999999999999e-07 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059304+0000", 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5490000000000001e-06 2024-09-16T05:20:19.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059308+0000", 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3610000000000003e-06 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059311+0000", 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9179999999999998e-06 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059364+0000", 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2790000000000001e-05 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059422+0000", 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7929000000000001e-05 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.073174+0000", 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013752234 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.077028+0000", 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038540319999999999 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078924+0000", 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001896179 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078961+0000", 2024-09-16T05:20:19.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7311000000000002e-05 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078996+0000", 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4591000000000001e-05 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4461 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:20:19.764 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 166.60927604299999, 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017250960999999999, 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4461 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4899999999999995e-07 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059326+0000", 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9280000000000003e-06 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059333+0000", 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6529999999999997e-06 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059338+0000", 2024-09-16T05:20:19.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1379999999999998e-06 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059598+0000", 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026046400000000002 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059658+0000", 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9342999999999997e-05 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076427+0000", 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016769117 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076490+0000", 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3207000000000007e-05 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076573+0000", 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2562e-05 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5587 2.34 2:2d3f13f8:::smithi20457400-7:head [setxattr _header (57) in=7b,truncate 3260723] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 86.862759314000002, 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.147062895, 2024-09-16T05:20:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5587 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805839+0000", 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8299999999999997e-07 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805842+0000", 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6790000000000001e-06 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805846+0000", 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4089999999999996e-06 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805849+0000", 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.191e-06 2024-09-16T05:20:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806887+0000", 2024-09-16T05:20:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010375440000000001 2024-09-16T05:20:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806953+0000", 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5833000000000005e-05 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807079+0000", 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012666500000000001 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.809689+0000", 2024-09-16T05:20:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026092860000000002 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952747+0000", 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14305815 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952826+0000", 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9773999999999999e-05 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952901+0000", 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4781000000000001e-05 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5588 2.34 2:2d3f13f8:::smithi20457400-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 86.862732840999996, 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14722005799999999, 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5588 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.65e-07 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805867+0000", 2024-09-16T05:20:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.86e-06 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805872+0000", 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805876+0000", 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0380000000000001e-06 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807396+0000", 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001520448 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807470+0000", 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3695000000000005e-05 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952937+0000", 2024-09-16T05:20:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14546690300000001 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953007+0000", 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9801999999999993e-05 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953085+0000", 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8121999999999995e-05 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:534 2.34 e617/278 ondisk, result = 0)", 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 52.856442704999999, 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93956844500000003, 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812156+0000", 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0599999999999999e-07 2024-09-16T05:20:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812160+0000", 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.437e-06 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751436+0000", 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93927549399999999 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751483+0000", 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7117999999999998e-05 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751598+0000", 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115448 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751609+0000", 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1156e-05 2024-09-16T05:20:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751723+0000", 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113986 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6080 2.34 2:2cffa4d6:::smithi20457400-50:head [copy-from ver 1093] snapc fa=[fa,f9,f7,f6,f2,f0,ef,ed,e8,e7] ondisk+write+known_if_redirected+supports_pool_eio e625)", 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:34.819560+0000", 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.849037727999999, 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018127142999999998, 2024-09-16T05:20:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6080 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819560+0000", 2024-09-16T05:20:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819560+0000", 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819562+0000", 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.764e-06 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:20:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819571+0000", 2024-09-16T05:20:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7119999999999995e-06 2024-09-16T05:20:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:20:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819591+0000", 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9340000000000001e-05 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819599+0000", 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.994e-06 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819693+0000", 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4585000000000003e-05 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819790+0000", 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6237999999999996e-05 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.830983+0000", 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011193613999999999 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.836190+0000", 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052071920000000002 2024-09-16T05:20:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.837611+0000", 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014208809999999999 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.837652+0000", 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1242000000000003e-05 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:20:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.837687+0000", 2024-09-16T05:20:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4581e-05 2024-09-16T05:20:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:20:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:20:19.794 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:20:19.795 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:19.796 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:19.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:19.832 INFO:tasks.rados.rados.2.smithi204.stdout:2600: done (2 left) 2024-09-16T05:20:19.832 INFO:tasks.rados.rados.2.smithi204.stdout:2601: copy_from oid 15 from oid 6 current snap is 273 2024-09-16T05:20:19.832 INFO:tasks.rados.rados.2.smithi204.stdout:2598: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:20:19.832 INFO:tasks.rados.rados.2.smithi204.stdout:2598: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:20:19.832 INFO:tasks.rados.rados.2.smithi204.stdout:2598: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:20:19.832 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 619 (ObjNum 868 snap 255 seq_num 868) dirty exists 2024-09-16T05:20:19.833 INFO:tasks.rados.rados.2.smithi204.stdout:2598: done (2 left) 2024-09-16T05:20:19.833 INFO:tasks.rados.rados.2.smithi204.stdout:2599: done (1 left) 2024-09-16T05:20:19.833 INFO:tasks.rados.rados.2.smithi204.stdout:2602: write oid 38 current snap is 273 2024-09-16T05:20:19.833 INFO:tasks.rados.rados.2.smithi204.stdout:2602: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-09-16T05:20:19.838 INFO:tasks.rados.rados.2.smithi204.stdout:2602: writing smithi20457400-38 from 612019 to 1177712 tid 1 2024-09-16T05:20:19.846 INFO:tasks.rados.rados.2.smithi204.stdout:2602: writing smithi20457400-38 from 1810929 to 2413575 tid 2 2024-09-16T05:20:19.852 INFO:tasks.rados.rados.2.smithi204.stdout:2602: writing smithi20457400-38 from 3082649 to 3412019 tid 3 2024-09-16T05:20:19.852 INFO:tasks.rados.rados.2.smithi204.stdout:2603: read oid 10 snap -1 2024-09-16T05:20:19.852 INFO:tasks.rados.rados.2.smithi204.stdout:2603: expect (ObjNum 753 snap 219 seq_num 753) 2024-09-16T05:20:19.852 INFO:tasks.rados.rados.2.smithi204.stdout:2604: write oid 41 current snap is 273 2024-09-16T05:20:19.852 INFO:tasks.rados.rados.2.smithi204.stdout:2602: finishing write tid 1 to smithi20457400-38 2024-09-16T05:20:19.852 INFO:tasks.rados.rados.2.smithi204.stdout:2604: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2024-09-16T05:20:19.859 INFO:tasks.rados.rados.2.smithi204.stdout:2604: writing smithi20457400-41 from 628826 to 1269768 tid 1 2024-09-16T05:20:19.864 INFO:tasks.rados.rados.2.smithi204.stdout:2604: writing smithi20457400-41 from 1869411 to 2332068 tid 2 2024-09-16T05:20:19.870 INFO:tasks.rados.rados.2.smithi204.stdout:2604: writing smithi20457400-41 from 2826425 to 3361074 tid 3 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2605: rollback oid 39 current snap is 273 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 39 to 261 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2602: finishing write tid 2 to smithi20457400-38 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2601: finishing copy_from racing read to smithi20457400-15 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2602: finishing write tid 3 to smithi20457400-38 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2602: finishing write tid 4 to smithi20457400-38 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2602: finishing write tid 5 to smithi20457400-38 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 905 (ObjNum 917 snap 273 seq_num 917) dirty exists 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2602: left oid 38 (ObjNum 917 snap 273 seq_num 917) 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2601: finishing copy_from to smithi20457400-15 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 741 (ObjNum 906 snap 267 seq_num 906) dirty exists 2024-09-16T05:20:19.871 INFO:tasks.rados.rados.2.smithi204.stdout:2603: expect (ObjNum 753 snap 219 seq_num 753) 2024-09-16T05:20:19.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:20:19.872+0000 7f30f77fe640 0 --2- 172.21.15.32:0/1304025409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30f8102790 0x7f30f81983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:20:19.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:19.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:19.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:20.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:20.033 INFO:tasks.rados.rados.2.smithi204.stdout:2604: finishing write tid 1 to smithi20457400-41 2024-09-16T05:20:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:2604: finishing write tid 2 to smithi20457400-41 2024-09-16T05:20:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:2604: finishing write tid 3 to smithi20457400-41 2024-09-16T05:20:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:2604: finishing write tid 4 to smithi20457400-41 2024-09-16T05:20:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:2601: done (4 left) 2024-09-16T05:20:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:2602: done (3 left) 2024-09-16T05:20:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:2603: done (2 left) 2024-09-16T05:20:20.034 INFO:tasks.rados.rados.2.smithi204.stdout:2606: snap_remove snap 268 2024-09-16T05:20:20.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:20.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:20.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: pgmap v1164: 107 pgs: 107 active+clean; 253 MiB data, 818 MiB used, 983 GiB / 983 GiB avail; 3.1 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: osdmap e669: 12 total, 12 up, 11 in 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.11647081760502673]}]: dispatch 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0764923095703125]}]: dispatch 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12884947091363508]}]: dispatch 2024-09-16T05:20:20.146 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09588230003650035]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09448587090141426]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09598101208183649]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08798866726725987]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07288257251261883]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09426196345587498]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08319780221080685]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07987483648038546]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09767107325339094]}]: dispatch 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:20:20.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:19 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:20:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:20.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:20.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: pgmap v1164: 107 pgs: 107 active+clean; 253 MiB data, 818 MiB used, 983 GiB / 983 GiB avail; 3.1 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: osdmap e669: 12 total, 12 up, 11 in 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.11647081760502673]}]: dispatch 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0764923095703125]}]: dispatch 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12884947091363508]}]: dispatch 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09588230003650035]}]: dispatch 2024-09-16T05:20:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09448587090141426]}]: dispatch 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09598101208183649]}]: dispatch 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08798866726725987]}]: dispatch 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07288257251261883]}]: dispatch 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09426196345587498]}]: dispatch 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08319780221080685]}]: dispatch 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07987483648038546]}]: dispatch 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09767107325339094]}]: dispatch 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:20:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:19 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:20:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: pgmap v1164: 107 pgs: 107 active+clean; 253 MiB data, 818 MiB used, 983 GiB / 983 GiB avail; 3.1 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-16T05:20:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:20:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:20:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:20:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:20:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: osdmap e669: 12 total, 12 up, 11 in 2024-09-16T05:20:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.11647081760502673]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0764923095703125]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12884947091363508]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09588230003650035]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09448587090141426]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09598101208183649]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08798866726725987]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07288257251261883]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09426196345587498]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08319780221080685]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07987483648038546]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09767107325339094]}]: dispatch 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:20:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:19 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:20:20.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:20.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:20.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:20.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:20.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:20.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:20.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:20:20.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:20.538 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:20.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:20.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:20.666 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:20.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:20.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:20:20.733 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:20.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:20.790 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:20:20.790 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T05:20:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:20.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:2606: done (2 left) 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:2607: write oid 3 current snap is 273 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:2605: finishing rollback tid 0 to smithi20457400-39 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:2604: finishing write tid 5 to smithi20457400-41 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 976 (ObjNum 918 snap 273 seq_num 918) dirty exists 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:2604: left oid 41 (ObjNum 918 snap 273 seq_num 918) 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:2605: finishing rollback tid 1 to smithi20457400-39 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1087 (ObjNum 177 snap 0 seq_num 4281529088) dirty dne 2024-09-16T05:20:20.795 INFO:tasks.rados.rados.2.smithi204.stdout:2607: seq_num 919 ranges {645633=716191,1927893=606315,3137495=308138} 2024-09-16T05:20:20.803 INFO:tasks.rados.rados.2.smithi204.stdout:2607: writing smithi20457400-3 from 645633 to 1361824 tid 1 2024-09-16T05:20:20.810 INFO:tasks.rados.rados.2.smithi204.stdout:2607: writing smithi20457400-3 from 1927893 to 2534208 tid 2 2024-09-16T05:20:20.815 INFO:tasks.rados.rados.2.smithi204.stdout:2607: writing smithi20457400-3 from 3137495 to 3445633 tid 3 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2604: done (2 left) 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2605: done (1 left) 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2608: copy_from oid 39 from oid 13 current snap is 273 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2607: finishing write tid 1 to smithi20457400-3 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2607: finishing write tid 2 to smithi20457400-3 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2609: read oid 31 snap -1 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2609: expect deleted 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2610: write (excl) oid 18 current snap is 273 2024-09-16T05:20:20.816 INFO:tasks.rados.rados.2.smithi204.stdout:2610: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2024-09-16T05:20:20.822 INFO:tasks.rados.rados.2.smithi204.stdout:2610: writing smithi20457400-18 from 662440 to 1170233 tid 1 2024-09-16T05:20:20.829 INFO:tasks.rados.rados.2.smithi204.stdout:2610: writing smithi20457400-18 from 1702728 to 2452701 tid 2 2024-09-16T05:20:20.834 INFO:tasks.rados.rados.2.smithi204.stdout:2610: writing smithi20457400-18 from 2881271 to 3438464 tid 3 2024-09-16T05:20:20.835 INFO:tasks.rados.rados.2.smithi204.stdout:2611: write oid 49 current snap is 273 2024-09-16T05:20:20.835 INFO:tasks.rados.rados.2.smithi204.stdout:2611: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-09-16T05:20:20.841 INFO:tasks.rados.rados.2.smithi204.stdout:2611: writing smithi20457400-49 from 679247 to 1262289 tid 1 2024-09-16T05:20:20.849 INFO:tasks.rados.rados.2.smithi204.stdout:2611: writing smithi20457400-49 from 2044857 to 2654841 tid 2 2024-09-16T05:20:20.854 INFO:tasks.rados.rados.2.smithi204.stdout:2611: writing smithi20457400-49 from 3192341 to 3479247 tid 3 2024-09-16T05:20:20.854 INFO:tasks.rados.rados.2.smithi204.stdout:2612: snap_create 2024-09-16T05:20:20.854 INFO:tasks.rados.rados.2.smithi204.stdout:2607: finishing write tid 3 to smithi20457400-3 2024-09-16T05:20:20.854 INFO:tasks.rados.rados.2.smithi204.stdout:2607: finishing write tid 4 to smithi20457400-3 2024-09-16T05:20:20.854 INFO:tasks.rados.rados.2.smithi204.stdout:2607: finishing write tid 5 to smithi20457400-3 2024-09-16T05:20:20.854 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 917 (ObjNum 919 snap 273 seq_num 919) dirty exists 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2607: left oid 3 (ObjNum 919 snap 273 seq_num 919) 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2608: finishing copy_from racing read to smithi20457400-39 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2608: finishing copy_from to smithi20457400-39 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1008 (ObjNum 855 snap 248 seq_num 855) dirty exists 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2610: finishing write tid 1 to smithi20457400-18 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2610: finishing write tid 2 to smithi20457400-18 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2610: finishing write tid 3 to smithi20457400-18 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2610: finishing write tid 4 to smithi20457400-18 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2610: finishing write tid 5 to smithi20457400-18 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 536 (ObjNum 920 snap 273 seq_num 920) dirty exists 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2610: left oid 18 (ObjNum 920 snap 273 seq_num 920) 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2611: finishing write tid 1 to smithi20457400-49 2024-09-16T05:20:20.855 INFO:tasks.rados.rados.2.smithi204.stdout:2611: finishing write tid 2 to smithi20457400-49 2024-09-16T05:20:20.856 INFO:tasks.rados.rados.2.smithi204.stdout:2611: finishing write tid 3 to smithi20457400-49 2024-09-16T05:20:20.856 INFO:tasks.rados.rados.2.smithi204.stdout:2611: finishing write tid 4 to smithi20457400-49 2024-09-16T05:20:20.856 INFO:tasks.rados.rados.2.smithi204.stdout:2611: finishing write tid 5 to smithi20457400-49 2024-09-16T05:20:20.856 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 624 (ObjNum 921 snap 273 seq_num 921) dirty exists 2024-09-16T05:20:20.857 INFO:tasks.rados.rados.2.smithi204.stdout:2611: left oid 49 (ObjNum 921 snap 273 seq_num 921) 2024-09-16T05:20:20.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:20.916 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:20.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:21.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:21.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:21.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.11647081760502673]}]': finished 2024-09-16T05:20:21.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0764923095703125]}]': finished 2024-09-16T05:20:21.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12884947091363508]}]': finished 2024-09-16T05:20:21.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09588230003650035]}]': finished 2024-09-16T05:20:21.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09448587090141426]}]': finished 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09598101208183649]}]': finished 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08798866726725987]}]': finished 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07288257251261883]}]': finished 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09426196345587498]}]': finished 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08319780221080685]}]': finished 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07987483648038546]}]': finished 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09767107325339094]}]': finished 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: osdmap e670: 12 total, 12 up, 11 in 2024-09-16T05:20:21.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1304025409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2150132102' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1304025409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2150132102' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:20:21.133 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:20 smithi196 ceph-mon[30802]: osdmap e671: 12 total, 12 up, 11 in 2024-09-16T05:20:21.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:21.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.11647081760502673]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0764923095703125]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12884947091363508]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09588230003650035]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09448587090141426]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09598101208183649]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08798866726725987]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07288257251261883]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09426196345587498]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08319780221080685]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07987483648038546]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09767107325339094]}]': finished 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: osdmap e670: 12 total, 12 up, 11 in 2024-09-16T05:20:21.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1304025409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2150132102' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:20:21.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:20:21.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:20:21.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1304025409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:21.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2150132102' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:20:21.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:20 smithi032 ceph-mon[27388]: osdmap e671: 12 total, 12 up, 11 in 2024-09-16T05:20:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.11647081760502673]}]': finished 2024-09-16T05:20:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0764923095703125]}]': finished 2024-09-16T05:20:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12884947091363508]}]': finished 2024-09-16T05:20:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09588230003650035]}]': finished 2024-09-16T05:20:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09448587090141426]}]': finished 2024-09-16T05:20:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.09598101208183649]}]': finished 2024-09-16T05:20:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08798866726725987]}]': finished 2024-09-16T05:20:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07288257251261883]}]': finished 2024-09-16T05:20:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09426196345587498]}]': finished 2024-09-16T05:20:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08319780221080685]}]': finished 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07987483648038546]}]': finished 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09767107325339094]}]': finished 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: osdmap e670: 12 total, 12 up, 11 in 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1304025409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2150132102' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:20:21.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:20:21.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:20:21.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1304025409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:21.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2150132102' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:20:21.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:20 smithi204 ceph-mon[30933]: osdmap e671: 12 total, 12 up, 11 in 2024-09-16T05:20:21.230 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:20:21.230 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:20:21.230 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:20:21.230 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:20:21.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:21.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:21.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:21.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:21.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:21.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:21.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:21.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:21.785 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:20:21.786 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:21.794 INFO:tasks.rados.rados.2.smithi204.stdout:2607: done (5 left) 2024-09-16T05:20:21.794 INFO:tasks.rados.rados.2.smithi204.stdout:2608: done (4 left) 2024-09-16T05:20:21.794 INFO:tasks.rados.rados.2.smithi204.stdout:2609: done (3 left) 2024-09-16T05:20:21.794 INFO:tasks.rados.rados.2.smithi204.stdout:2610: done (2 left) 2024-09-16T05:20:21.794 INFO:tasks.rados.rados.2.smithi204.stdout:2611: done (1 left) 2024-09-16T05:20:21.794 INFO:tasks.rados.rados.2.smithi204.stdout:2612: done (0 left) 2024-09-16T05:20:21.795 INFO:tasks.rados.rados.2.smithi204.stdout:2613: copy_from oid 44 from oid 4 current snap is 274 2024-09-16T05:20:21.795 INFO:tasks.rados.rados.2.smithi204.stdout:2614: write oid 41 current snap is 274 2024-09-16T05:20:21.795 INFO:tasks.rados.rados.2.smithi204.stdout:2614: seq_num 922 ranges {696054=658291,2103339=753642} 2024-09-16T05:20:21.801 INFO:tasks.rados.rados.2.smithi204.stdout:2614: writing smithi20457400-41 from 696054 to 1354345 tid 1 2024-09-16T05:20:21.810 INFO:tasks.rados.rados.2.smithi204.stdout:2614: writing smithi20457400-41 from 2103339 to 2856981 tid 2 2024-09-16T05:20:21.810 INFO:tasks.rados.rados.2.smithi204.stdout:2615: delete oid 13 current snap is 274 2024-09-16T05:20:21.810 INFO:tasks.rados.rados.2.smithi204.stdout:2613: finishing copy_from to smithi20457400-44 2024-09-16T05:20:21.810 INFO:tasks.rados.rados.2.smithi204.stdout:2613: got expected ENOENT (src dne) 2024-09-16T05:20:21.810 INFO:tasks.rados.rados.2.smithi204.stdout:2613: finishing copy_from racing read to smithi20457400-44 2024-09-16T05:20:21.813 INFO:tasks.rados.rados.2.smithi204.stdout:2614: finishing write tid 1 to smithi20457400-41 2024-09-16T05:20:21.813 INFO:tasks.rados.rados.2.smithi204.stdout:2614: finishing write tid 2 to smithi20457400-41 2024-09-16T05:20:21.813 INFO:tasks.rados.rados.2.smithi204.stdout:2614: finishing write tid 3 to smithi20457400-41 2024-09-16T05:20:21.814 INFO:tasks.rados.rados.2.smithi204.stdout:2613: done (2 left) 2024-09-16T05:20:21.814 INFO:tasks.rados.rados.2.smithi204.stdout:2615: done (1 left) 2024-09-16T05:20:21.814 INFO:tasks.rados.rados.2.smithi204.stdout:2616: snap_create 2024-09-16T05:20:21.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:21.911 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:22.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:22.030 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:22.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:22.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:22.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:22.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:21 smithi032 ceph-mon[27388]: pgmap v1167: 107 pgs: 107 active+clean; 246 MiB data, 813 MiB used, 983 GiB / 983 GiB avail; 962 KiB/s rd, 1.9 MiB/s wr, 7 op/s 2024-09-16T05:20:22.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:21 smithi032 ceph-mon[27388]: 2.3a scrub starts 2024-09-16T05:20:22.081 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:21 smithi032 ceph-mon[27388]: 2.3a scrub ok 2024-09-16T05:20:22.081 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:21 smithi032 ceph-mon[27388]: osdmap e672: 12 total, 12 up, 11 in 2024-09-16T05:20:22.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-16T05:20:22.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:21 smithi196 ceph-mon[30802]: pgmap v1167: 107 pgs: 107 active+clean; 246 MiB data, 813 MiB used, 983 GiB / 983 GiB avail; 962 KiB/s rd, 1.9 MiB/s wr, 7 op/s 2024-09-16T05:20:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:21 smithi196 ceph-mon[30802]: 2.3a scrub starts 2024-09-16T05:20:22.143 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:21 smithi196 ceph-mon[30802]: 2.3a scrub ok 2024-09-16T05:20:22.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:21 smithi196 ceph-mon[30802]: osdmap e672: 12 total, 12 up, 11 in 2024-09-16T05:20:22.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:22.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:21 smithi204 ceph-mon[30933]: pgmap v1167: 107 pgs: 107 active+clean; 246 MiB data, 813 MiB used, 983 GiB / 983 GiB avail; 962 KiB/s rd, 1.9 MiB/s wr, 7 op/s 2024-09-16T05:20:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:21 smithi204 ceph-mon[30933]: 2.3a scrub starts 2024-09-16T05:20:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:21 smithi204 ceph-mon[30933]: 2.3a scrub ok 2024-09-16T05:20:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:21 smithi204 ceph-mon[30933]: osdmap e672: 12 total, 12 up, 11 in 2024-09-16T05:20:22.230 INFO:tasks.rados.rados.2.smithi204.stdout:2614: finishing write tid 4 to smithi20457400-41 2024-09-16T05:20:22.231 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 982 (ObjNum 922 snap 274 seq_num 922) dirty exists 2024-09-16T05:20:22.231 INFO:tasks.rados.rados.2.smithi204.stdout:2614: left oid 41 (ObjNum 922 snap 274 seq_num 922) 2024-09-16T05:20:22.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:22.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:22.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:22.345 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:22.372 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:22.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:22.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:22.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:22.479 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:20:22.479 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:20:22.479 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:20:22.479 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:20:22.479 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:20:22.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:22.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:22.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:22.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:22.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:22.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:22.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:22.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:22.787 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:22.795 INFO:tasks.rados.rados.2.smithi204.stdout:2614: done (1 left) 2024-09-16T05:20:22.795 INFO:tasks.rados.rados.2.smithi204.stdout:2616: done (0 left) 2024-09-16T05:20:22.796 INFO:tasks.rados.rados.2.smithi204.stdout:2617: write oid 39 current snap is 275 2024-09-16T05:20:22.796 INFO:tasks.rados.rados.2.smithi204.stdout:2617: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-09-16T05:20:22.801 INFO:tasks.rados.rados.2.smithi204.stdout:2617: writing smithi20457400-39 from 729668 to 1138457 tid 1 2024-09-16T05:20:22.809 INFO:tasks.rados.rados.2.smithi204.stdout:2617: writing smithi20457400-39 from 1820303 to 2577614 tid 2 2024-09-16T05:20:22.815 INFO:tasks.rados.rados.2.smithi204.stdout:2617: writing smithi20457400-39 from 3274610 to 3529668 tid 3 2024-09-16T05:20:22.815 INFO:tasks.rados.rados.2.smithi204.stdout:2618: write oid 32 current snap is 275 2024-09-16T05:20:22.815 INFO:tasks.rados.rados.2.smithi204.stdout:2617: finishing write tid 1 to smithi20457400-39 2024-09-16T05:20:22.815 INFO:tasks.rados.rados.2.smithi204.stdout:2617: finishing write tid 2 to smithi20457400-39 2024-09-16T05:20:22.815 INFO:tasks.rados.rados.2.smithi204.stdout:2618: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2024-09-16T05:20:22.821 INFO:tasks.rados.rados.2.smithi204.stdout:2618: writing smithi20457400-32 from 746475 to 1230513 tid 1 2024-09-16T05:20:22.829 INFO:tasks.rados.rados.2.smithi204.stdout:2618: writing smithi20457400-32 from 1762432 to 2379754 tid 2 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:2618: writing smithi20457400-32 from 2785680 to 3315586 tid 3 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:2619: rollback oid 7 current snap is 275 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 7 to 260 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:2617: finishing write tid 3 to smithi20457400-39 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:2617: finishing write tid 4 to smithi20457400-39 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:2617: finishing write tid 5 to smithi20457400-39 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1095 (ObjNum 924 snap 275 seq_num 924) dirty exists 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:2617: left oid 39 (ObjNum 924 snap 275 seq_num 924) 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:2617: done (2 left) 2024-09-16T05:20:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:2620: snap_remove snap 269 2024-09-16T05:20:22.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:22.906 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:22.912 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:22.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:22.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:22.927 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:22.948 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:22.948 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:22.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:22.968 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:22.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:22 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:20:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:22 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:20:22.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:22 smithi196 ceph-mon[30802]: osdmap e673: 12 total, 12 up, 11 in 2024-09-16T05:20:22.989 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:22.989 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:23.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:23.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:23.015 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:23.039 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:23.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:23.042 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:23.043 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:23.043 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:22 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:20:23.043 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:22 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:20:23.043 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:22 smithi032 ceph-mon[27388]: osdmap e673: 12 total, 12 up, 11 in 2024-09-16T05:20:23.064 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:23.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:20:23.085 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:23.086 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:23.107 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:23.107 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:23.129 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:23.129 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:23.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:23.150 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:23.150 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:23.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:23.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:23.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:22 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:20:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:22 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:20:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:22 smithi204 ceph-mon[30933]: osdmap e673: 12 total, 12 up, 11 in 2024-09-16T05:20:23.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:23.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:23.316 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:23.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:23.413 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:23.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:23.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:23.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:23.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:23.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:23.666 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:23.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:23.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:23.787 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:23.787 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:23.794 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T05:20:23.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:23.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:23.798 INFO:tasks.rados.rados.2.smithi204.stdout:2620: done (2 left) 2024-09-16T05:20:23.798 INFO:tasks.rados.rados.2.smithi204.stdout:2621: write oid 22 current snap is 275 2024-09-16T05:20:23.798 INFO:tasks.rados.rados.2.smithi204.stdout:2621: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2024-09-16T05:20:23.806 INFO:tasks.rados.rados.2.smithi204.stdout:2621: writing smithi20457400-22 from 763282 to 1322569 tid 1 2024-09-16T05:20:23.814 INFO:tasks.rados.rados.2.smithi204.stdout:2621: writing smithi20457400-22 from 1820914 to 2581894 tid 2 2024-09-16T05:20:23.820 INFO:tasks.rados.rados.2.smithi204.stdout:2621: writing smithi20457400-22 from 3213103 to 3563282 tid 3 2024-09-16T05:20:23.820 INFO:tasks.rados.rados.2.smithi204.stdout:2622: delete oid 43 current snap is 275 2024-09-16T05:20:23.821 INFO:tasks.rados.rados.2.smithi204.stdout:2618: finishing write tid 1 to smithi20457400-32 2024-09-16T05:20:23.821 INFO:tasks.rados.rados.2.smithi204.stdout:2618: finishing write tid 2 to smithi20457400-32 2024-09-16T05:20:23.821 INFO:tasks.rados.rados.2.smithi204.stdout:2618: finishing write tid 3 to smithi20457400-32 2024-09-16T05:20:23.821 INFO:tasks.rados.rados.2.smithi204.stdout:2618: finishing write tid 4 to smithi20457400-32 2024-09-16T05:20:23.821 INFO:tasks.rados.rados.2.smithi204.stdout:2618: finishing write tid 5 to smithi20457400-32 2024-09-16T05:20:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 994 (ObjNum 925 snap 275 seq_num 925) dirty exists 2024-09-16T05:20:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:2618: left oid 32 (ObjNum 925 snap 275 seq_num 925) 2024-09-16T05:20:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:2619: finishing rollback tid 0 to smithi20457400-7 2024-09-16T05:20:23.822 INFO:tasks.rados.rados.2.smithi204.stdout:2619: finishing rollback tid 1 to smithi20457400-7 2024-09-16T05:20:23.823 INFO:tasks.rados.rados.2.smithi204.stdout:2619: finishing rollback tid 2 to smithi20457400-7 2024-09-16T05:20:23.823 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1112 (ObjNum 789 snap 229 seq_num 789) dirty exists 2024-09-16T05:20:23.823 INFO:tasks.rados.rados.2.smithi204.stdout:2621: finishing write tid 1 to smithi20457400-22 2024-09-16T05:20:23.823 INFO:tasks.rados.rados.2.smithi204.stdout:2621: finishing write tid 2 to smithi20457400-22 2024-09-16T05:20:23.826 INFO:tasks.rados.rados.2.smithi204.stdout:2618: done (3 left) 2024-09-16T05:20:23.826 INFO:tasks.rados.rados.2.smithi204.stdout:2619: done (2 left) 2024-09-16T05:20:23.826 INFO:tasks.rados.rados.2.smithi204.stdout:2622: done (1 left) 2024-09-16T05:20:23.827 INFO:tasks.rados.rados.2.smithi204.stdout:2623: read oid 38 snap 264 2024-09-16T05:20:23.827 INFO:tasks.rados.rados.2.smithi204.stdout:2623: expect (ObjNum 884 snap 259 seq_num 884) 2024-09-16T05:20:23.827 INFO:tasks.rados.rados.2.smithi204.stdout:2624: rollback oid 43 current snap is 275 2024-09-16T05:20:23.827 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 43 to 264 2024-09-16T05:20:23.827 INFO:tasks.rados.rados.2.smithi204.stdout:2625: read oid 48 snap -1 2024-09-16T05:20:23.828 INFO:tasks.rados.rados.2.smithi204.stdout:2625: expect (ObjNum 910 snap 271 seq_num 910) 2024-09-16T05:20:23.828 INFO:tasks.rados.rados.2.smithi204.stdout:2626: read oid 44 snap -1 2024-09-16T05:20:23.828 INFO:tasks.rados.rados.2.smithi204.stdout:2626: expect deleted 2024-09-16T05:20:23.828 INFO:tasks.rados.rados.2.smithi204.stdout:2627: delete oid 37 current snap is 275 2024-09-16T05:20:23.828 INFO:tasks.rados.rados.2.smithi204.stdout:2621: finishing write tid 3 to smithi20457400-22 2024-09-16T05:20:23.828 INFO:tasks.rados.rados.2.smithi204.stdout:2624: finishing rollback tid 1 to smithi20457400-43 2024-09-16T05:20:23.829 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 610 (ObjNum 81 snap 0 seq_num 0) dirty dne 2024-09-16T05:20:23.829 INFO:tasks.rados.rados.2.smithi204.stdout:2624: done (5 left) 2024-09-16T05:20:23.829 INFO:tasks.rados.rados.2.smithi204.stdout:2626: done (4 left) 2024-09-16T05:20:23.829 INFO:tasks.rados.rados.2.smithi204.stdout:2627: done (3 left) 2024-09-16T05:20:23.829 INFO:tasks.rados.rados.2.smithi204.stdout:2628: rollback oid 39 current snap is 275 2024-09-16T05:20:23.830 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 39 to 264 2024-09-16T05:20:23.830 INFO:tasks.rados.rados.2.smithi204.stdout:2629: write (excl) oid 19 current snap is 275 2024-09-16T05:20:23.830 INFO:tasks.rados.rados.2.smithi204.stdout:2629: seq_num 929 ranges {413703=501387,1596360=624660,2895372=691347} 2024-09-16T05:20:23.830 INFO:tasks.rados.rados.2.smithi204.stdout:2629: writing smithi20457400-19 from 413703 to 915090 tid 1 2024-09-16T05:20:23.836 INFO:tasks.rados.rados.2.smithi204.stdout:2629: writing smithi20457400-19 from 1596360 to 2221020 tid 2 2024-09-16T05:20:23.844 INFO:tasks.rados.rados.2.smithi204.stdout:2629: writing smithi20457400-19 from 2895372 to 3586719 tid 3 2024-09-16T05:20:23.844 INFO:tasks.rados.rados.2.smithi204.stdout:2630: write (excl) oid 1 current snap is 275 2024-09-16T05:20:23.844 INFO:tasks.rados.rados.2.smithi204.stdout:2630: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-09-16T05:20:23.849 INFO:tasks.rados.rados.2.smithi204.stdout:2630: writing smithi20457400-1 from 430510 to 1007146 tid 1 2024-09-16T05:20:23.857 INFO:tasks.rados.rados.2.smithi204.stdout:2630: writing smithi20457400-1 from 1654842 to 2423160 tid 2 2024-09-16T05:20:23.863 INFO:tasks.rados.rados.2.smithi204.stdout:2630: writing smithi20457400-1 from 2922795 to 3425414 tid 3 2024-09-16T05:20:23.863 INFO:tasks.rados.rados.2.smithi204.stdout:2631: read oid 18 snap -1 2024-09-16T05:20:23.863 INFO:tasks.rados.rados.2.smithi204.stdout:2631: expect (ObjNum 920 snap 273 seq_num 920) 2024-09-16T05:20:23.863 INFO:tasks.rados.rados.2.smithi204.stdout:2632: write oid 36 current snap is 275 2024-09-16T05:20:23.863 INFO:tasks.rados.rados.2.smithi204.stdout:2632: seq_num 931 ranges {447317=651885,1713324=628329,3066571=580746} 2024-09-16T05:20:23.869 INFO:tasks.rados.rados.2.smithi204.stdout:2632: writing smithi20457400-36 from 447317 to 1099202 tid 1 2024-09-16T05:20:23.876 INFO:tasks.rados.rados.2.smithi204.stdout:2632: writing smithi20457400-36 from 1713324 to 2341653 tid 2 2024-09-16T05:20:23.883 INFO:tasks.rados.rados.2.smithi204.stdout:2632: writing smithi20457400-36 from 3066571 to 3647317 tid 3 2024-09-16T05:20:23.883 INFO:tasks.rados.rados.2.smithi204.stdout:2633: rollback oid 32 current snap is 275 2024-09-16T05:20:23.884 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 240 2024-09-16T05:20:23.884 INFO:tasks.rados.rados.2.smithi204.stdout:2621: finishing write tid 4 to smithi20457400-22 2024-09-16T05:20:23.884 INFO:tasks.rados.rados.2.smithi204.stdout:2621: finishing write tid 5 to smithi20457400-22 2024-09-16T05:20:23.884 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 707 (ObjNum 926 snap 275 seq_num 926) dirty exists 2024-09-16T05:20:23.884 INFO:tasks.rados.rados.2.smithi204.stdout:2621: left oid 22 (ObjNum 926 snap 275 seq_num 926) 2024-09-16T05:20:23.884 INFO:tasks.rados.rados.2.smithi204.stdout:2628: finishing rollback tid 0 to smithi20457400-39 2024-09-16T05:20:23.884 INFO:tasks.rados.rados.2.smithi204.stdout:2628: finishing rollback tid 1 to smithi20457400-39 2024-09-16T05:20:23.884 INFO:tasks.rados.rados.2.smithi204.stdout:2623: expect (ObjNum 884 snap 259 seq_num 884) 2024-09-16T05:20:23.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:23.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:23.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:23.979 INFO:tasks.rados.rados.2.smithi204.stdout:2625: expect (ObjNum 910 snap 271 seq_num 910) 2024-09-16T05:20:24.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:24.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:24.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:23 smithi196 ceph-mon[30802]: pgmap v1170: 107 pgs: 1 remapped+peering, 106 active+clean; 245 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2024-09-16T05:20:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:23 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:20:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:23 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:20:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/847782758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/459095341' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:20:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/459095341' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:20:24.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:23 smithi196 ceph-mon[30802]: osdmap e674: 12 total, 12 up, 11 in 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:2629: finishing write tid 1 to smithi20457400-19 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:2629: finishing write tid 2 to smithi20457400-19 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:2629: finishing write tid 3 to smithi20457400-19 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:2629: finishing write tid 4 to smithi20457400-19 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:2629: finishing write tid 5 to smithi20457400-19 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 350 (ObjNum 929 snap 275 seq_num 929) dirty exists 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:2629: left oid 19 (ObjNum 929 snap 275 seq_num 929) 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:2630: finishing write tid 1 to smithi20457400-1 2024-09-16T05:20:24.123 INFO:tasks.rados.rados.2.smithi204.stdout:2621: done (8 left) 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2623: done (7 left) 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2625: done (6 left) 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2629: done (5 left) 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2634: read oid 17 snap -1 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2634: expect (ObjNum 846 snap 246 seq_num 846) 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2635: snap_create 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2630: finishing write tid 2 to smithi20457400-1 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2630: finishing write tid 3 to smithi20457400-1 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2630: finishing write tid 4 to smithi20457400-1 2024-09-16T05:20:24.124 INFO:tasks.rados.rados.2.smithi204.stdout:2630: finishing write tid 5 to smithi20457400-1 2024-09-16T05:20:24.125 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 552 (ObjNum 930 snap 275 seq_num 930) dirty exists 2024-09-16T05:20:24.125 INFO:tasks.rados.rados.2.smithi204.stdout:2630: left oid 1 (ObjNum 930 snap 275 seq_num 930) 2024-09-16T05:20:24.125 INFO:tasks.rados.rados.2.smithi204.stdout:2632: finishing write tid 1 to smithi20457400-36 2024-09-16T05:20:24.125 INFO:tasks.rados.rados.2.smithi204.stdout:2631: expect (ObjNum 920 snap 273 seq_num 920) 2024-09-16T05:20:24.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:24.168 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:24.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:23 smithi032 ceph-mon[27388]: pgmap v1170: 107 pgs: 1 remapped+peering, 106 active+clean; 245 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2024-09-16T05:20:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:23 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:20:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:23 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:20:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/847782758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/459095341' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:20:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/459095341' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:20:24.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:23 smithi032 ceph-mon[27388]: osdmap e674: 12 total, 12 up, 11 in 2024-09-16T05:20:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:23 smithi204 ceph-mon[30933]: pgmap v1170: 107 pgs: 1 remapped+peering, 106 active+clean; 245 MiB data, 798 MiB used, 983 GiB / 983 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2024-09-16T05:20:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:23 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:20:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:23 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:20:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/847782758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/459095341' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:20:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/459095341' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:20:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:23 smithi204 ceph-mon[30933]: osdmap e674: 12 total, 12 up, 11 in 2024-09-16T05:20:24.231 INFO:tasks.rados.rados.2.smithi204.stdout:2632: finishing write tid 2 to smithi20457400-36 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:2633: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:2633: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:2633: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1001 (ObjNum 815 snap 237 seq_num 815) dirty exists 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:2632: finishing write tid 3 to smithi20457400-36 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:2632: finishing write tid 4 to smithi20457400-36 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:2632: finishing write tid 5 to smithi20457400-36 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 765 (ObjNum 931 snap 275 seq_num 931) dirty exists 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:2632: left oid 36 (ObjNum 931 snap 275 seq_num 931) 2024-09-16T05:20:24.232 INFO:tasks.rados.rados.2.smithi204.stdout:2634: expect (ObjNum 846 snap 246 seq_num 846) 2024-09-16T05:20:24.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:24.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:24.292 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:24.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:20:24.357 INFO:tasks.rados.rados.2.smithi204.stdout:2628: finishing rollback tid 2 to smithi20457400-39 2024-09-16T05:20:24.357 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1100 (ObjNum 900 snap 263 seq_num 900) dirty exists 2024-09-16T05:20:24.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:20:24.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:24.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:24.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:24.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:24.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:24.623 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T05:20:24.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:20:24.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:24.644 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:24.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:24.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:24.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:24.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:24.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:24.859 INFO:tasks.rados.rados.2.smithi204.stdout:2628: done (6 left) 2024-09-16T05:20:24.859 INFO:tasks.rados.rados.2.smithi204.stdout:2630: done (5 left) 2024-09-16T05:20:24.859 INFO:tasks.rados.rados.2.smithi204.stdout:2631: done (4 left) 2024-09-16T05:20:24.859 INFO:tasks.rados.rados.2.smithi204.stdout:2632: done (3 left) 2024-09-16T05:20:24.859 INFO:tasks.rados.rados.2.smithi204.stdout:2633: done (2 left) 2024-09-16T05:20:24.859 INFO:tasks.rados.rados.2.smithi204.stdout:2634: done (1 left) 2024-09-16T05:20:24.859 INFO:tasks.rados.rados.2.smithi204.stdout:2635: done (0 left) 2024-09-16T05:20:24.859 INFO:tasks.rados.rados.2.smithi204.stdout:2636: snap_create 2024-09-16T05:20:24.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:24.911 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:24.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:25.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:25.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:25.036 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:25.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:24 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:20:25.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:24 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:20:25.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:24 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:20:25.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:24 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:20:25.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:24 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:20:25.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:24 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:20:25.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:25.143 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:25.143 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:25.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:24 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:20:25.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:24 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:20:25.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:24 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:20:25.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:24 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:20:25.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:24 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:20:25.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:24 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:20:25.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:25.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:25.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:24 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:20:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:24 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:20:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:24 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:20:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:24 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:20:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:24 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:20:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:24 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:20:25.263 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:25.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:25.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:25.301 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:25.390 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:25.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:25.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:25.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:25.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:25.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:25.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:25.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:25.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:25.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:25.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:25.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:25.790 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:20:25.797 INFO:tasks.rados.rados.2.smithi204.stdout:2636: done (0 left) 2024-09-16T05:20:25.797 INFO:tasks.rados.rados.2.smithi204.stdout:2637: delete oid 38 current snap is 277 2024-09-16T05:20:25.802 INFO:tasks.rados.rados.2.smithi204.stdout:2637: done (0 left) 2024-09-16T05:20:25.802 INFO:tasks.rados.rados.2.smithi204.stdout:2638: delete oid 24 current snap is 277 2024-09-16T05:20:25.805 INFO:tasks.rados.rados.2.smithi204.stdout:2638: done (0 left) 2024-09-16T05:20:25.805 INFO:tasks.rados.rados.2.smithi204.stdout:2639: read oid 1 snap -1 2024-09-16T05:20:25.805 INFO:tasks.rados.rados.2.smithi204.stdout:2639: expect (ObjNum 930 snap 275 seq_num 930) 2024-09-16T05:20:25.805 INFO:tasks.rados.rados.2.smithi204.stdout:2640: read oid 4 snap -1 2024-09-16T05:20:25.806 INFO:tasks.rados.rados.2.smithi204.stdout:2640: expect deleted 2024-09-16T05:20:25.806 INFO:tasks.rados.rados.2.smithi204.stdout:2641: read oid 13 snap -1 2024-09-16T05:20:25.806 INFO:tasks.rados.rados.2.smithi204.stdout:2641: expect deleted 2024-09-16T05:20:25.806 INFO:tasks.rados.rados.2.smithi204.stdout:2642: snap_remove snap 261 2024-09-16T05:20:25.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:20:25.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:25.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:25.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:20:25.952 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:20:25.953 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:20:26.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:26.035 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:26.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:26.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:26.115 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:26.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:26.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:25 smithi032 ceph-mon[27388]: pgmap v1173: 107 pgs: 1 peering, 106 active+clean; 262 MiB data, 819 MiB used, 983 GiB / 983 GiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 19 op/s; 411 KiB/s, 5 objects/s recovering 2024-09-16T05:20:26.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:25 smithi032 ceph-mon[27388]: osdmap e675: 12 total, 12 up, 11 in 2024-09-16T05:20:26.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1106653227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:26.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:25 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:20:26.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:25 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:20:26.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:25 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:26.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1106653227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:26.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:25 smithi032 ceph-mon[27388]: osdmap e676: 12 total, 12 up, 11 in 2024-09-16T05:20:26.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:26.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:25 smithi204 ceph-mon[30933]: pgmap v1173: 107 pgs: 1 peering, 106 active+clean; 262 MiB data, 819 MiB used, 983 GiB / 983 GiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 19 op/s; 411 KiB/s, 5 objects/s recovering 2024-09-16T05:20:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:25 smithi204 ceph-mon[30933]: osdmap e675: 12 total, 12 up, 11 in 2024-09-16T05:20:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1106653227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:25 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:20:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:25 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:20:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:25 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1106653227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:25 smithi204 ceph-mon[30933]: osdmap e676: 12 total, 12 up, 11 in 2024-09-16T05:20:26.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:25 smithi196 ceph-mon[30802]: pgmap v1173: 107 pgs: 1 peering, 106 active+clean; 262 MiB data, 819 MiB used, 983 GiB / 983 GiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 19 op/s; 411 KiB/s, 5 objects/s recovering 2024-09-16T05:20:26.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:25 smithi196 ceph-mon[30802]: osdmap e675: 12 total, 12 up, 11 in 2024-09-16T05:20:26.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1106653227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:26.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:25 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:20:26.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:25 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:20:26.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:25 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:26.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1106653227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:26.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:25 smithi196 ceph-mon[30802]: osdmap e676: 12 total, 12 up, 11 in 2024-09-16T05:20:26.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:26.283 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:26.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:26.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:26.408 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:26.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:26.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:26.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:26.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:26.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:26.592 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:26.592 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:26.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:26.659 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:26.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:26.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:26.893 INFO:tasks.rados.rados.2.smithi204.stdout:2642: done (3 left) 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2643: write (excl) oid 14 current snap is 277 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2643: seq_num 934 ranges {497738=477632,1772417=775656,3316134=381604} 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2643: writing smithi20457400-14 from 497738 to 975370 tid 1 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2643: writing smithi20457400-14 from 1772417 to 2548073 tid 2 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2643: writing smithi20457400-14 from 3316134 to 3697738 tid 3 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2644: read oid 32 snap -1 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2644: expect (ObjNum 815 snap 237 seq_num 815) 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2645: read oid 25 snap -1 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2645: expect deleted 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2646: write oid 29 current snap is 277 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2646: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2024-09-16T05:20:26.894 INFO:tasks.rados.rados.2.smithi204.stdout:2646: writing smithi20457400-29 from 514545 to 1067426 tid 1 2024-09-16T05:20:26.895 INFO:tasks.rados.rados.2.smithi204.stdout:2646: writing smithi20457400-29 from 1830899 to 2350213 tid 2 2024-09-16T05:20:26.895 INFO:tasks.rados.rados.2.smithi204.stdout:2646: writing smithi20457400-29 from 2943557 to 3418889 tid 3 2024-09-16T05:20:26.895 INFO:tasks.rados.rados.2.smithi204.stdout:2647: snap_remove snap 274 2024-09-16T05:20:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:26.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:26.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:27.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:27.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:27.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:27.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:26 smithi032 ceph-mon[27388]: from='client.16746 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:27.126 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:26 smithi032 ceph-mon[27388]: osdmap e677: 12 total, 12 up, 11 in 2024-09-16T05:20:27.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:27.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:27.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:26 smithi196 ceph-mon[30802]: from='client.16746 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:27.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:26 smithi196 ceph-mon[30802]: osdmap e677: 12 total, 12 up, 11 in 2024-09-16T05:20:27.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:26 smithi204 ceph-mon[30933]: from='client.16746 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:27.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:26 smithi204 ceph-mon[30933]: osdmap e677: 12 total, 12 up, 11 in 2024-09-16T05:20:27.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:20:27.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:27.291 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:27.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:27.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:27.419 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:27.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:27.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:27.474 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:20:27.505 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1175,"stamp":"2024-09-16T05:20:25.790737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":277276206,"num_objects":129,"num_object_clones":80,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":14,"num_read":3158,"num_read_kb":3106178,"num_write":2313,"num_write_kb":920856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":47028772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27162,"ondisk_log_size":27162,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":216,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":843524,"kb_used_data":424396,"kb_used_omap":26,"kb_used_meta":418853,"kb_avail":1030353148,"statfs":{"total":1055945392128,"available":1055081623552,"internally_reserved":0,"allocated":434581504,"data_stored":484167026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26910,"internal_metadata":428906210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20649836,"num_objects":12,"num_object_clones":12,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":21,"num_read_kb":9236,"num_write":52,"num_write_kb":17742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1683975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.999732"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":3280,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.824643+0000","last_change":"2024-09-16T05:20:20.824839+0000","last_active":"2024-09-16T05:20:20.824643+0000","last_peered":"2024-09-16T05:20:20.824643+0000","last_clean":"2024-09-16T05:20:20.824643+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:20:20.824643+0000","last_undegraded":"2024-09-16T05:20:20.824643+0000","last_fullsized":"2024-09-16T05:20:20.824643+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:20:01.826054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:01.826054+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.481846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036188000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.54","version":"667'395","reported_seq":2815,"reported_epoch":673,"state":"peering","last_fresh":"2024-09-16T05:20:21.868588+0000","last_change":"2024-09-16T05:20:22.925300+0000","last_active":"2024-09-16T05:20:22.925300+0000","last_peered":"2024-09-16T05:20:21.868588+0000","last_clean":"2024-09-16T05:20:16.791436+0000","last_became_active":"2024-09-16T05:20:21.841248+0000","last_became_peered":"2024-09-16T05:20:21.841248+0000","last_unstale":"2024-09-16T05:20:21.868588+0000","last_undegraded":"2024-09-16T05:20:21.868588+0000","last_fullsized":"2024-09-16T05:20:21.868588+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"619'381","last_scrub_stamp":"2024-09-16T05:19:29.293042+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.293042+0000","objects_scrubbed":3,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.0024586579999999999,"stat_sum":{"num_bytes":3243949,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":56,"num_read_kb":57049,"num_write":34,"num_write_kb":11523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"110","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":1171,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.796376+0000","last_change":"2024-09-16T05:20:20.797171+0000","last_active":"2024-09-16T05:20:20.796376+0000","last_peered":"2024-09-16T05:20:20.796376+0000","last_clean":"2024-09-16T05:20:20.796376+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:20:20.796376+0000","last_undegraded":"2024-09-16T05:20:20.796376+0000","last_fullsized":"2024-09-16T05:20:20.796376+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:32.054132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.054132+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:59.964999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011494560000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.52","version":"272'69","reported_seq":1190,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155269+0000","last_change":"2024-09-16T05:20:23.799550+0000","last_active":"2024-09-16T05:20:25.155269+0000","last_peered":"2024-09-16T05:20:25.155269+0000","last_clean":"2024-09-16T05:20:25.155269+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:20:25.155269+0000","last_undegraded":"2024-09-16T05:20:25.155269+0000","last_fullsized":"2024-09-16T05:20:25.155269+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:18.588272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.588272+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:33.267855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040587400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.51","version":"272'115","reported_seq":1238,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.802483+0000","last_change":"2024-09-16T05:20:20.802673+0000","last_active":"2024-09-16T05:20:20.802483+0000","last_peered":"2024-09-16T05:20:20.802483+0000","last_clean":"2024-09-16T05:20:20.802483+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:20:20.802483+0000","last_undegraded":"2024-09-16T05:20:20.802483+0000","last_fullsized":"2024-09-16T05:20:20.802483+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:19:29.589220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.589220+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.961250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049270599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.50","version":"182'16","reported_seq":1176,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.836443+0000","last_change":"2024-09-16T05:20:20.795938+0000","last_active":"2024-09-16T05:20:22.836443+0000","last_peered":"2024-09-16T05:20:22.836443+0000","last_clean":"2024-09-16T05:20:22.836443+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:20:22.836443+0000","last_undegraded":"2024-09-16T05:20:22.836443+0000","last_fullsized":"2024-09-16T05:20:22.836443+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:29.302740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.302740+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:38.170625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047692500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2164,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837252+0000","last_change":"2024-09-16T05:20:20.797423+0000","last_active":"2024-09-16T05:20:21.837252+0000","last_peered":"2024-09-16T05:20:21.837252+0000","last_clean":"2024-09-16T05:20:21.837252+0000","last_became_active":"2024-09-16T05:19:35.789805+0000","last_became_peered":"2024-09-16T05:19:35.789805+0000","last_unstale":"2024-09-16T05:20:21.837252+0000","last_undegraded":"2024-09-16T05:20:21.837252+0000","last_fullsized":"2024-09-16T05:20:21.837252+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:19:48.302807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.302807+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.600781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001869798,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.4e","version":"272'232","reported_seq":2941,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.824731+0000","last_change":"2024-09-16T05:20:20.824934+0000","last_active":"2024-09-16T05:20:20.824731+0000","last_peered":"2024-09-16T05:20:20.824731+0000","last_clean":"2024-09-16T05:20:20.824731+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:20:20.824731+0000","last_undegraded":"2024-09-16T05:20:20.824731+0000","last_fullsized":"2024-09-16T05:20:20.824731+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:50.792260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.792260+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:00.315321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00038801200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.4d","version":"665'626","reported_seq":3499,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.796095+0000","last_change":"2024-09-16T05:20:20.795950+0000","last_active":"2024-09-16T05:20:22.796095+0000","last_peered":"2024-09-16T05:20:22.796095+0000","last_clean":"2024-09-16T05:20:22.796095+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:20:22.796095+0000","last_undegraded":"2024-09-16T05:20:22.796095+0000","last_fullsized":"2024-09-16T05:20:22.796095+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"665'626","last_scrub_stamp":"2024-09-16T05:20:16.492166+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.492166+0000","objects_scrubbed":4,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.858661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00094616299999999997,"stat_sum":{"num_bytes":6503878,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":45,"num_read_kb":38966,"num_write":56,"num_write_kb":18053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.4c","version":"666'345","reported_seq":3149,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.804224+0000","last_change":"2024-09-16T05:20:20.804441+0000","last_active":"2024-09-16T05:20:20.804224+0000","last_peered":"2024-09-16T05:20:20.804224+0000","last_clean":"2024-09-16T05:20:20.804224+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:20:20.804224+0000","last_undegraded":"2024-09-16T05:20:20.804224+0000","last_fullsized":"2024-09-16T05:20:20.804224+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"666'345","last_scrub_stamp":"2024-09-16T05:20:17.450506+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.450506+0000","objects_scrubbed":1,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:24.217211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014005630000000001,"stat_sum":{"num_bytes":3395179,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":37571,"num_write":28,"num_write_kb":7491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.4b","version":"244'177","reported_seq":2981,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.796271+0000","last_change":"2024-09-16T05:20:20.795657+0000","last_active":"2024-09-16T05:20:22.796271+0000","last_peered":"2024-09-16T05:20:22.796271+0000","last_clean":"2024-09-16T05:20:22.796271+0000","last_became_active":"2024-09-16T05:17:21.741180+0000","last_became_peered":"2024-09-16T05:17:21.741180+0000","last_unstale":"2024-09-16T05:20:22.796271+0000","last_undegraded":"2024-09-16T05:20:22.796271+0000","last_fullsized":"2024-09-16T05:20:22.796271+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:19:32.417815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.417815+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:41.352507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065165000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.4a","version":"620'557","reported_seq":3327,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.804348+0000","last_change":"2024-09-16T05:20:20.804348+0000","last_active":"2024-09-16T05:20:20.804348+0000","last_peered":"2024-09-16T05:20:20.804348+0000","last_clean":"2024-09-16T05:20:20.804348+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:20:20.804348+0000","last_undegraded":"2024-09-16T05:20:20.804348+0000","last_fullsized":"2024-09-16T05:20:20.804348+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"620'557","last_scrub_stamp":"2024-09-16T05:19:54.555606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.555606+0000","objects_scrubbed":1,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:05.184058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001324862,"stat_sum":{"num_bytes":2689285,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":47,"num_read_kb":41138,"num_write":35,"num_write_kb":18755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.49","version":"245'220","reported_seq":3018,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.825874+0000","last_change":"2024-09-16T05:20:20.826047+0000","last_active":"2024-09-16T05:20:20.825874+0000","last_peered":"2024-09-16T05:20:20.825874+0000","last_clean":"2024-09-16T05:20:20.825874+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:20:20.825874+0000","last_undegraded":"2024-09-16T05:20:20.825874+0000","last_fullsized":"2024-09-16T05:20:20.825874+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:20.094577+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.094577+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:33.754472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001580304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.48","version":"674'318","reported_seq":3100,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857853+0000","last_change":"2024-09-16T05:20:23.807986+0000","last_active":"2024-09-16T05:20:24.857853+0000","last_peered":"2024-09-16T05:20:24.857853+0000","last_clean":"2024-09-16T05:20:24.857853+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:20:24.857853+0000","last_undegraded":"2024-09-16T05:20:24.857853+0000","last_fullsized":"2024-09-16T05:20:24.857853+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"628'287","last_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:19:38.000359+0000","objects_scrubbed":3,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:39.573694+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0038763560000000001,"stat_sum":{"num_bytes":7103158,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":55585,"num_write":45,"num_write_kb":22506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.47","version":"0'0","reported_seq":1808,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155348+0000","last_change":"2024-09-16T05:20:23.800484+0000","last_active":"2024-09-16T05:20:25.155348+0000","last_peered":"2024-09-16T05:20:25.155348+0000","last_clean":"2024-09-16T05:20:25.155348+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:20:25.155348+0000","last_undegraded":"2024-09-16T05:20:25.155348+0000","last_fullsized":"2024-09-16T05:20:25.155348+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:30.390373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.390373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.511493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085393000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.46","version":"116'9","reported_seq":24,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.796081+0000","last_change":"2024-09-16T05:20:20.801427+0000","last_active":"2024-09-16T05:20:22.796081+0000","last_peered":"2024-09-16T05:20:22.796081+0000","last_clean":"2024-09-16T05:20:22.796081+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:20:22.796081+0000","last_undegraded":"2024-09-16T05:20:22.796081+0000","last_fullsized":"2024-09-16T05:20:22.796081+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:49.313870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.313870+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:12.792365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033601579999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.45","version":"305'464","reported_seq":2756,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.804458+0000","last_change":"2024-09-16T05:20:20.804734+0000","last_active":"2024-09-16T05:20:20.804458+0000","last_peered":"2024-09-16T05:20:20.804458+0000","last_clean":"2024-09-16T05:20:20.804458+0000","last_became_active":"2024-09-16T05:19:35.786934+0000","last_became_peered":"2024-09-16T05:19:35.786934+0000","last_unstale":"2024-09-16T05:20:20.804458+0000","last_undegraded":"2024-09-16T05:20:20.804458+0000","last_fullsized":"2024-09-16T05:20:20.804458+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:19:09.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:09.927260+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:22.047808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012528319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":1377,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.795986+0000","last_change":"2024-09-16T05:20:20.796947+0000","last_active":"2024-09-16T05:20:21.795986+0000","last_peered":"2024-09-16T05:20:21.795986+0000","last_clean":"2024-09-16T05:20:21.795986+0000","last_became_active":"2024-09-16T05:19:35.788828+0000","last_became_peered":"2024-09-16T05:19:35.788828+0000","last_unstale":"2024-09-16T05:20:21.795986+0000","last_undegraded":"2024-09-16T05:20:21.795986+0000","last_fullsized":"2024-09-16T05:20:21.795986+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:48.163046+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.163046+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:06.181028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00037501599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":1782,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.836773+0000","last_change":"2024-09-16T05:20:20.795097+0000","last_active":"2024-09-16T05:20:21.836773+0000","last_peered":"2024-09-16T05:20:21.836773+0000","last_clean":"2024-09-16T05:20:21.836773+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:20:21.836773+0000","last_undegraded":"2024-09-16T05:20:21.836773+0000","last_fullsized":"2024-09-16T05:20:21.836773+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:19:15.441056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.441056+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.013411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043540000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.42","version":"182'20","reported_seq":1772,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796581+0000","last_change":"2024-09-16T05:20:23.797334+0000","last_active":"2024-09-16T05:20:23.796581+0000","last_peered":"2024-09-16T05:20:23.796581+0000","last_clean":"2024-09-16T05:20:23.796581+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:20:23.796581+0000","last_undegraded":"2024-09-16T05:20:23.796581+0000","last_fullsized":"2024-09-16T05:20:23.796581+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:19:28.221037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.221037+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:49.020047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014226099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.41","version":"178'50","reported_seq":648,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155222+0000","last_change":"2024-09-16T05:20:23.800056+0000","last_active":"2024-09-16T05:20:25.155222+0000","last_peered":"2024-09-16T05:20:25.155222+0000","last_clean":"2024-09-16T05:20:25.155222+0000","last_became_active":"2024-09-16T05:18:19.768547+0000","last_became_peered":"2024-09-16T05:18:19.768547+0000","last_unstale":"2024-09-16T05:20:25.155222+0000","last_undegraded":"2024-09-16T05:20:25.155222+0000","last_fullsized":"2024-09-16T05:20:25.155222+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:19:55.515448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.515448+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.732888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00042290300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.40","version":"205'33","reported_seq":329,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.825265+0000","last_change":"2024-09-16T05:20:20.825511+0000","last_active":"2024-09-16T05:20:20.825265+0000","last_peered":"2024-09-16T05:20:20.825265+0000","last_clean":"2024-09-16T05:20:20.825265+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:20:20.825265+0000","last_undegraded":"2024-09-16T05:20:20.825265+0000","last_fullsized":"2024-09-16T05:20:20.825265+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:53.588594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.588594+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:08.741674+0000","scrub_duration":798,"objects_trimmed":0,"snaptrim_duration":0.00081381599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.3f","version":"666'419","reported_seq":3236,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837440+0000","last_change":"2024-09-16T05:20:20.797768+0000","last_active":"2024-09-16T05:20:21.837440+0000","last_peered":"2024-09-16T05:20:21.837440+0000","last_clean":"2024-09-16T05:20:21.837440+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:20:21.837440+0000","last_undegraded":"2024-09-16T05:20:21.837440+0000","last_fullsized":"2024-09-16T05:20:21.837440+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'415","last_scrub_stamp":"2024-09-16T05:19:58.393481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:58.393481+0000","objects_scrubbed":2,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.980989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002381691,"stat_sum":{"num_bytes":5748357,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":82274,"num_write":47,"num_write_kb":13993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.3e","version":"625'487","reported_seq":3259,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.404425+0000","last_change":"2024-09-16T05:20:22.404478+0000","last_active":"2024-09-16T05:20:22.404425+0000","last_peered":"2024-09-16T05:20:22.404425+0000","last_clean":"2024-09-16T05:20:22.404425+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:20:22.404425+0000","last_undegraded":"2024-09-16T05:20:22.404425+0000","last_fullsized":"2024-09-16T05:20:22.404425+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"625'487","last_scrub_stamp":"2024-09-16T05:20:22.404334+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:20:22.404334+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:50.563545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023799630000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":26573,"num_write":41,"num_write_kb":23888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1718,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.797408+0000","last_change":"2024-09-16T05:20:23.797774+0000","last_active":"2024-09-16T05:20:23.797408+0000","last_peered":"2024-09-16T05:20:23.797408+0000","last_clean":"2024-09-16T05:20:23.797408+0000","last_became_active":"2024-09-16T05:19:35.784967+0000","last_became_peered":"2024-09-16T05:19:35.784967+0000","last_unstale":"2024-09-16T05:20:23.797408+0000","last_undegraded":"2024-09-16T05:20:23.797408+0000","last_fullsized":"2024-09-16T05:20:23.797408+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:19:36.333160+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.333160+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.555665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001630732,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.3c","version":"666'702","reported_seq":3670,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837037+0000","last_change":"2024-09-16T05:20:20.797399+0000","last_active":"2024-09-16T05:20:21.837037+0000","last_peered":"2024-09-16T05:20:21.837037+0000","last_clean":"2024-09-16T05:20:21.837037+0000","last_became_active":"2024-09-16T05:19:35.800730+0000","last_became_peered":"2024-09-16T05:19:35.800730+0000","last_unstale":"2024-09-16T05:20:21.837037+0000","last_undegraded":"2024-09-16T05:20:21.837037+0000","last_fullsized":"2024-09-16T05:20:21.837037+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"624'648","last_scrub_stamp":"2024-09-16T05:19:34.258477+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:19:34.258477+0000","objects_scrubbed":6,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:51.325979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014961250000000001,"stat_sum":{"num_bytes":10703115,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":160,"num_read_kb":208718,"num_write":73,"num_write_kb":28278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":6910647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.3b","version":"654'428","reported_seq":3226,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.826224+0000","last_change":"2024-09-16T05:20:20.826378+0000","last_active":"2024-09-16T05:20:20.826224+0000","last_peered":"2024-09-16T05:20:20.826224+0000","last_clean":"2024-09-16T05:20:20.826224+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:20:20.826224+0000","last_undegraded":"2024-09-16T05:20:20.826224+0000","last_fullsized":"2024-09-16T05:20:20.826224+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'416","last_scrub_stamp":"2024-09-16T05:19:55.767476+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.767476+0000","objects_scrubbed":2,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.623637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016358449999999999,"stat_sum":{"num_bytes":8222944,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54616,"num_write":46,"num_write_kb":21574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.3a","version":"667'593","reported_seq":3368,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857904+0000","last_change":"2024-09-16T05:20:23.803488+0000","last_active":"2024-09-16T05:20:24.857904+0000","last_peered":"2024-09-16T05:20:24.857904+0000","last_clean":"2024-09-16T05:20:24.857904+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:20:24.857904+0000","last_undegraded":"2024-09-16T05:20:24.857904+0000","last_fullsized":"2024-09-16T05:20:24.857904+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"667'593","last_scrub_stamp":"2024-09-16T05:20:20.620494+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.620494+0000","objects_scrubbed":2,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.955507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070838100000000001,"stat_sum":{"num_bytes":3949810,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":53,"num_read_kb":46839,"num_write":60,"num_write_kb":24043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.39","version":"182'60","reported_seq":1888,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857923+0000","last_change":"2024-09-16T05:20:23.804556+0000","last_active":"2024-09-16T05:20:24.857923+0000","last_peered":"2024-09-16T05:20:24.857923+0000","last_clean":"2024-09-16T05:20:24.857923+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:20:24.857923+0000","last_undegraded":"2024-09-16T05:20:24.857923+0000","last_fullsized":"2024-09-16T05:20:24.857923+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:19:49.888753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.888753+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:14.187423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001803121,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.38","version":"181'57","reported_seq":1893,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.795860+0000","last_change":"2024-09-16T05:20:23.796180+0000","last_active":"2024-09-16T05:20:23.795860+0000","last_peered":"2024-09-16T05:20:23.795860+0000","last_clean":"2024-09-16T05:20:23.795860+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:20:23.795860+0000","last_undegraded":"2024-09-16T05:20:23.795860+0000","last_fullsized":"2024-09-16T05:20:23.795860+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:20:20.396645+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.396645+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:29.475306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057224100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.37","version":"0'0","reported_seq":727,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.805176+0000","last_change":"2024-09-16T05:20:20.805420+0000","last_active":"2024-09-16T05:20:20.805176+0000","last_peered":"2024-09-16T05:20:20.805176+0000","last_clean":"2024-09-16T05:20:20.805176+0000","last_became_active":"2024-09-16T05:17:59.866037+0000","last_became_peered":"2024-09-16T05:17:59.866037+0000","last_unstale":"2024-09-16T05:20:20.805176+0000","last_undegraded":"2024-09-16T05:20:20.805176+0000","last_fullsized":"2024-09-16T05:20:20.805176+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:31.567784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:19:31.567784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:38.537428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00140766,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.36","version":"272'60","reported_seq":246,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.796017+0000","last_change":"2024-09-16T05:20:20.799662+0000","last_active":"2024-09-16T05:20:21.796017+0000","last_peered":"2024-09-16T05:20:21.796017+0000","last_clean":"2024-09-16T05:20:21.796017+0000","last_became_active":"2024-09-16T05:19:35.795592+0000","last_became_peered":"2024-09-16T05:19:35.795592+0000","last_unstale":"2024-09-16T05:20:21.796017+0000","last_undegraded":"2024-09-16T05:20:21.796017+0000","last_fullsized":"2024-09-16T05:20:21.796017+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:20:02.294024+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.294024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:17.688500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024858649999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.35","version":"671'735","reported_seq":3616,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.796003+0000","last_change":"2024-09-16T05:20:20.802943+0000","last_active":"2024-09-16T05:20:21.796003+0000","last_peered":"2024-09-16T05:20:21.796003+0000","last_clean":"2024-09-16T05:20:21.796003+0000","last_became_active":"2024-09-16T05:19:35.802234+0000","last_became_peered":"2024-09-16T05:19:35.802234+0000","last_unstale":"2024-09-16T05:20:21.796003+0000","last_undegraded":"2024-09-16T05:20:21.796003+0000","last_fullsized":"2024-09-16T05:20:21.796003+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'675","last_scrub_stamp":"2024-09-16T05:19:11.894332+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.894332+0000","objects_scrubbed":8,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:26.276165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0058298380000000004,"stat_sum":{"num_bytes":12263488,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":125,"num_read_kb":118125,"num_write":86,"num_write_kb":40632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3771362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.34","version":"671'583","reported_seq":2988,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.826296+0000","last_change":"2024-09-16T05:20:20.826469+0000","last_active":"2024-09-16T05:20:20.826296+0000","last_peered":"2024-09-16T05:20:20.826296+0000","last_clean":"2024-09-16T05:20:20.826296+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:20:20.826296+0000","last_undegraded":"2024-09-16T05:20:20.826296+0000","last_fullsized":"2024-09-16T05:20:20.826296+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'482","last_scrub_stamp":"2024-09-16T05:19:12.637489+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.637489+0000","objects_scrubbed":8,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0019178140000000001,"stat_sum":{"num_bytes":18684074,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":154,"num_read_kb":173311,"num_write":99,"num_write_kb":39821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":1963,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.836925+0000","last_change":"2024-09-16T05:20:20.797685+0000","last_active":"2024-09-16T05:20:21.836925+0000","last_peered":"2024-09-16T05:20:21.836925+0000","last_clean":"2024-09-16T05:20:21.836925+0000","last_became_active":"2024-09-16T05:19:35.789794+0000","last_became_peered":"2024-09-16T05:19:35.789794+0000","last_unstale":"2024-09-16T05:20:21.836925+0000","last_undegraded":"2024-09-16T05:20:21.836925+0000","last_fullsized":"2024-09-16T05:20:21.836925+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:16.451459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:16.451459+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001911077,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.32","version":"671'227","reported_seq":2173,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.795776+0000","last_change":"2024-09-16T05:20:20.798644+0000","last_active":"2024-09-16T05:20:21.795776+0000","last_peered":"2024-09-16T05:20:21.795776+0000","last_clean":"2024-09-16T05:20:21.795776+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:20:21.795776+0000","last_undegraded":"2024-09-16T05:20:21.795776+0000","last_fullsized":"2024-09-16T05:20:21.795776+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"574'195","last_scrub_stamp":"2024-09-16T05:18:54.271472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:54.271472+0000","objects_scrubbed":3,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022049999999999999,"stat_sum":{"num_bytes":6202179,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":101970,"num_write":44,"num_write_kb":20642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.31","version":"666'280","reported_seq":2829,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155452+0000","last_change":"2024-09-16T05:20:23.800301+0000","last_active":"2024-09-16T05:20:25.155452+0000","last_peered":"2024-09-16T05:20:25.155452+0000","last_clean":"2024-09-16T05:20:25.155452+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:20:25.155452+0000","last_undegraded":"2024-09-16T05:20:25.155452+0000","last_fullsized":"2024-09-16T05:20:25.155452+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'268","last_scrub_stamp":"2024-09-16T05:20:03.547514+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.547514+0000","objects_scrubbed":2,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:13.452601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094748100000000002,"stat_sum":{"num_bytes":3680626,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":116,"num_read_kb":119771,"num_write":39,"num_write_kb":19090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.30","version":"182'16","reported_seq":716,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.803117+0000","last_change":"2024-09-16T05:20:20.803117+0000","last_active":"2024-09-16T05:20:20.803117+0000","last_peered":"2024-09-16T05:20:20.803117+0000","last_clean":"2024-09-16T05:20:20.803117+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:20:20.803117+0000","last_undegraded":"2024-09-16T05:20:20.803117+0000","last_fullsized":"2024-09-16T05:20:20.803117+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:20:16.443442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.443442+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.288658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091142199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.12","version":"35'139","reported_seq":976,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.824306+0000","last_change":"2024-09-16T05:20:18.091859+0000","last_active":"2024-09-16T05:20:20.824306+0000","last_peered":"2024-09-16T05:20:20.824306+0000","last_clean":"2024-09-16T05:20:20.824306+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:20:20.824306+0000","last_undegraded":"2024-09-16T05:20:20.824306+0000","last_fullsized":"2024-09-16T05:20:20.824306+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.091747+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.091747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.641366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":2517,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.792377+0000","last_change":"2024-09-16T05:20:20.794679+0000","last_active":"2024-09-16T05:20:21.792377+0000","last_peered":"2024-09-16T05:20:21.792377+0000","last_clean":"2024-09-16T05:20:21.792377+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:20:21.792377+0000","last_undegraded":"2024-09-16T05:20:21.792377+0000","last_fullsized":"2024-09-16T05:20:21.792377+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:20:03.076376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.076376+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:10.343332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053290200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.13","version":"35'139","reported_seq":1143,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.795707+0000","last_change":"2024-09-16T05:19:36.252852+0000","last_active":"2024-09-16T05:20:21.795707+0000","last_peered":"2024-09-16T05:20:21.795707+0000","last_clean":"2024-09-16T05:20:21.795707+0000","last_became_active":"2024-09-16T05:19:35.788516+0000","last_became_peered":"2024-09-16T05:19:35.788516+0000","last_unstale":"2024-09-16T05:20:21.795707+0000","last_undegraded":"2024-09-16T05:20:21.795707+0000","last_fullsized":"2024-09-16T05:20:21.795707+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:36.252730+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.252730+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.441323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":2444,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796195+0000","last_change":"2024-09-16T05:20:23.796445+0000","last_active":"2024-09-16T05:20:23.796195+0000","last_peered":"2024-09-16T05:20:23.796195+0000","last_clean":"2024-09-16T05:20:23.796195+0000","last_became_active":"2024-09-16T05:19:35.791894+0000","last_became_peered":"2024-09-16T05:19:35.791894+0000","last_unstale":"2024-09-16T05:20:23.796195+0000","last_undegraded":"2024-09-16T05:20:23.796195+0000","last_fullsized":"2024-09-16T05:20:23.796195+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:11.306770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.306770+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072383600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.c","version":"35'139","reported_seq":929,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.836640+0000","last_change":"2024-09-16T05:20:18.363196+0000","last_active":"2024-09-16T05:20:22.836640+0000","last_peered":"2024-09-16T05:20:22.836640+0000","last_clean":"2024-09-16T05:20:22.836640+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:20:22.836640+0000","last_undegraded":"2024-09-16T05:20:22.836640+0000","last_fullsized":"2024-09-16T05:20:22.836640+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.363111+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.363111+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:36.359138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"666'405","reported_seq":3231,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796879+0000","last_change":"2024-09-16T05:20:23.797106+0000","last_active":"2024-09-16T05:20:23.796879+0000","last_peered":"2024-09-16T05:20:23.796879+0000","last_clean":"2024-09-16T05:20:23.796879+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:20:23.796879+0000","last_undegraded":"2024-09-16T05:20:23.796879+0000","last_fullsized":"2024-09-16T05:20:23.796879+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'384","last_scrub_stamp":"2024-09-16T05:19:15.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.295498+0000","objects_scrubbed":6,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011026530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":57662,"num_write":37,"num_write_kb":14770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.d","version":"35'139","reported_seq":1082,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.836835+0000","last_change":"2024-09-16T05:19:12.454525+0000","last_active":"2024-09-16T05:20:21.836835+0000","last_peered":"2024-09-16T05:20:21.836835+0000","last_clean":"2024-09-16T05:20:21.836835+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:20:21.836835+0000","last_undegraded":"2024-09-16T05:20:21.836835+0000","last_fullsized":"2024-09-16T05:20:21.836835+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:12.454446+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.454446+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.180360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":2939,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.796103+0000","last_change":"2024-09-16T05:20:20.795312+0000","last_active":"2024-09-16T05:20:22.796103+0000","last_peered":"2024-09-16T05:20:22.796103+0000","last_clean":"2024-09-16T05:20:22.796103+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:20:22.796103+0000","last_undegraded":"2024-09-16T05:20:22.796103+0000","last_fullsized":"2024-09-16T05:20:22.796103+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:11.209692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.209692+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079975399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.e","version":"35'139","reported_seq":1078,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.796881+0000","last_change":"2024-09-16T05:19:59.197776+0000","last_active":"2024-09-16T05:20:20.796881+0000","last_peered":"2024-09-16T05:20:20.796881+0000","last_clean":"2024-09-16T05:20:20.796881+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:20:20.796881+0000","last_undegraded":"2024-09-16T05:20:20.796881+0000","last_fullsized":"2024-09-16T05:20:20.796881+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:59.197664+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:59.197664+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.493240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"666'602","reported_seq":3432,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858239+0000","last_change":"2024-09-16T05:20:23.805426+0000","last_active":"2024-09-16T05:20:24.858239+0000","last_peered":"2024-09-16T05:20:24.858239+0000","last_clean":"2024-09-16T05:20:24.858239+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:20:24.858239+0000","last_undegraded":"2024-09-16T05:20:24.858239+0000","last_fullsized":"2024-09-16T05:20:24.858239+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"617'576","last_scrub_stamp":"2024-09-16T05:19:26.973559+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:19:26.973559+0000","objects_scrubbed":4,"log_size":602,"log_dups_size":0,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:48.652195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019755530000000001,"stat_sum":{"num_bytes":9125264,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":52779,"num_write":49,"num_write_kb":25445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.f","version":"35'139","reported_seq":1063,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.824431+0000","last_change":"2024-09-16T05:19:29.650332+0000","last_active":"2024-09-16T05:20:20.824431+0000","last_peered":"2024-09-16T05:20:20.824431+0000","last_clean":"2024-09-16T05:20:20.824431+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:20:20.824431+0000","last_undegraded":"2024-09-16T05:20:20.824431+0000","last_fullsized":"2024-09-16T05:20:20.824431+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:29.650209+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.650209+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:34.196552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3066,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.795863+0000","last_change":"2024-09-16T05:20:20.797226+0000","last_active":"2024-09-16T05:20:21.795863+0000","last_peered":"2024-09-16T05:20:21.795863+0000","last_clean":"2024-09-16T05:20:21.795863+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:20:21.795863+0000","last_undegraded":"2024-09-16T05:20:21.795863+0000","last_fullsized":"2024-09-16T05:20:21.795863+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:19:10.225027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.225027+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:26.253727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060176200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.b","version":"35'139","reported_seq":1605,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.795976+0000","last_change":"2024-09-16T05:19:33.235762+0000","last_active":"2024-09-16T05:20:21.795976+0000","last_peered":"2024-09-16T05:20:21.795976+0000","last_clean":"2024-09-16T05:20:21.795976+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:20:21.795976+0000","last_undegraded":"2024-09-16T05:20:21.795976+0000","last_fullsized":"2024-09-16T05:20:21.795976+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:33.235681+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:19:33.235681+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.500872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"666'325","reported_seq":3125,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837509+0000","last_change":"2024-09-16T05:20:20.796419+0000","last_active":"2024-09-16T05:20:21.837509+0000","last_peered":"2024-09-16T05:20:21.837509+0000","last_clean":"2024-09-16T05:20:21.837509+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:20:21.837509+0000","last_undegraded":"2024-09-16T05:20:21.837509+0000","last_fullsized":"2024-09-16T05:20:21.837509+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'297","last_scrub_stamp":"2024-09-16T05:19:10.483443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.483443+0000","objects_scrubbed":2,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:39.975281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014317430000000001,"stat_sum":{"num_bytes":4366553,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":35425,"num_write":51,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.3","version":"671'209","reported_seq":3025,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.838308+0000","last_change":"2024-09-16T05:20:20.826605+0000","last_active":"2024-09-16T05:20:20.838308+0000","last_peered":"2024-09-16T05:20:20.838308+0000","last_clean":"2024-09-16T05:20:20.838308+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:20:20.838308+0000","last_undegraded":"2024-09-16T05:20:20.838308+0000","last_fullsized":"2024-09-16T05:20:20.838308+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'205","last_scrub_stamp":"2024-09-16T05:20:19.137710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:19.137710+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:35.296982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017520859999999999,"stat_sum":{"num_bytes":3462440,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":91280,"num_write":43,"num_write_kb":18525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":617,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.795709+0000","last_change":"2024-09-16T05:20:00.246338+0000","last_active":"2024-09-16T05:20:21.795709+0000","last_peered":"2024-09-16T05:20:21.795709+0000","last_clean":"2024-09-16T05:20:21.795709+0000","last_became_active":"2024-09-16T05:18:19.768392+0000","last_became_peered":"2024-09-16T05:18:19.768392+0000","last_unstale":"2024-09-16T05:20:21.795709+0000","last_undegraded":"2024-09-16T05:20:21.795709+0000","last_fullsized":"2024-09-16T05:20:21.795709+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:00.246242+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.246242+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.738053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":2808,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.825018+0000","last_change":"2024-09-16T05:20:20.825182+0000","last_active":"2024-09-16T05:20:20.825018+0000","last_peered":"2024-09-16T05:20:20.825018+0000","last_clean":"2024-09-16T05:20:20.825018+0000","last_became_active":"2024-09-16T05:19:35.791085+0000","last_became_peered":"2024-09-16T05:19:35.791085+0000","last_unstale":"2024-09-16T05:20:20.825018+0000","last_undegraded":"2024-09-16T05:20:20.825018+0000","last_fullsized":"2024-09-16T05:20:20.825018+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:02.847979+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.847979+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:14.840142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061612099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":1163,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.802338+0000","last_change":"2024-09-16T05:19:30.558195+0000","last_active":"2024-09-16T05:20:20.802338+0000","last_peered":"2024-09-16T05:20:20.802338+0000","last_clean":"2024-09-16T05:20:20.802338+0000","last_became_active":"2024-09-16T05:18:02.888512+0000","last_became_peered":"2024-09-16T05:18:02.888512+0000","last_unstale":"2024-09-16T05:20:20.802338+0000","last_undegraded":"2024-09-16T05:20:20.802338+0000","last_fullsized":"2024-09-16T05:20:20.802338+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.558069+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.558069+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.688000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1595,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.836703+0000","last_change":"2024-09-16T05:19:56.353335+0000","last_active":"2024-09-16T05:20:21.836703+0000","last_peered":"2024-09-16T05:20:21.836703+0000","last_clean":"2024-09-16T05:20:21.836703+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:20:21.836703+0000","last_undegraded":"2024-09-16T05:20:21.836703+0000","last_fullsized":"2024-09-16T05:20:21.836703+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:19:56.353243+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.079495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":2748,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.795680+0000","last_change":"2024-09-16T05:20:20.795345+0000","last_active":"2024-09-16T05:20:22.795680+0000","last_peered":"2024-09-16T05:20:22.795680+0000","last_clean":"2024-09-16T05:20:22.795680+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:20:22.795680+0000","last_undegraded":"2024-09-16T05:20:22.795680+0000","last_fullsized":"2024-09-16T05:20:22.795680+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:18:54.327254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:18:54.327254+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098306099999999992,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.3","version":"35'139","reported_seq":1602,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.836578+0000","last_change":"2024-09-16T05:19:10.336320+0000","last_active":"2024-09-16T05:20:22.836578+0000","last_peered":"2024-09-16T05:20:22.836578+0000","last_clean":"2024-09-16T05:20:22.836578+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:20:22.836578+0000","last_undegraded":"2024-09-16T05:20:22.836578+0000","last_fullsized":"2024-09-16T05:20:22.836578+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:10.336203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.336203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.544788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":2833,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837675+0000","last_change":"2024-09-16T05:20:20.798006+0000","last_active":"2024-09-16T05:20:21.837675+0000","last_peered":"2024-09-16T05:20:21.837675+0000","last_clean":"2024-09-16T05:20:21.837675+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:20:21.837675+0000","last_undegraded":"2024-09-16T05:20:21.837675+0000","last_fullsized":"2024-09-16T05:20:21.837675+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:51.365804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.365804+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:10.845345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025202089999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":2059,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858300+0000","last_change":"2024-09-16T05:20:23.808288+0000","last_active":"2024-09-16T05:20:24.858300+0000","last_peered":"2024-09-16T05:20:24.858300+0000","last_clean":"2024-09-16T05:20:24.858300+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:20:24.858300+0000","last_undegraded":"2024-09-16T05:20:24.858300+0000","last_fullsized":"2024-09-16T05:20:24.858300+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:50.852123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.852123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:00.103629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038802469999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.4","version":"35'139","reported_seq":1561,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.796105+0000","last_change":"2024-09-16T05:19:53.322398+0000","last_active":"2024-09-16T05:20:21.796105+0000","last_peered":"2024-09-16T05:20:21.796105+0000","last_clean":"2024-09-16T05:20:21.796105+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:20:21.796105+0000","last_undegraded":"2024-09-16T05:20:21.796105+0000","last_fullsized":"2024-09-16T05:20:21.796105+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:53.322303+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.322303+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:11.892229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1230,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.795895+0000","last_change":"2024-09-16T05:20:20.795012+0000","last_active":"2024-09-16T05:20:22.795895+0000","last_peered":"2024-09-16T05:20:22.795895+0000","last_clean":"2024-09-16T05:20:22.795895+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:20:22.795895+0000","last_undegraded":"2024-09-16T05:20:22.795895+0000","last_fullsized":"2024-09-16T05:20:22.795895+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:53.382868+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:18:27.279121+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.382868+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:02.541295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072936799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":868,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.803776+0000","last_change":"2024-09-16T05:19:13.533926+0000","last_active":"2024-09-16T05:20:20.803776+0000","last_peered":"2024-09-16T05:20:20.803776+0000","last_clean":"2024-09-16T05:20:20.803776+0000","last_became_active":"2024-09-16T05:18:02.897275+0000","last_became_peered":"2024-09-16T05:18:02.897275+0000","last_unstale":"2024-09-16T05:20:20.803776+0000","last_undegraded":"2024-09-16T05:20:20.803776+0000","last_fullsized":"2024-09-16T05:20:20.803776+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:19:13.533798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:23.927688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"674'581","reported_seq":3404,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.799103+0000","last_change":"2024-09-16T05:20:23.799472+0000","last_active":"2024-09-16T05:20:23.799103+0000","last_peered":"2024-09-16T05:20:23.799103+0000","last_clean":"2024-09-16T05:20:23.799103+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:20:23.799103+0000","last_undegraded":"2024-09-16T05:20:23.799103+0000","last_fullsized":"2024-09-16T05:20:23.799103+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'568","last_scrub_stamp":"2024-09-16T05:19:54.247232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.247232+0000","objects_scrubbed":3,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:05.980654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032423719999999999,"stat_sum":{"num_bytes":7659108,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":107857,"num_write":39,"num_write_kb":12889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.6","version":"35'139","reported_seq":812,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.802356+0000","last_change":"2024-09-16T05:19:37.639763+0000","last_active":"2024-09-16T05:20:20.802356+0000","last_peered":"2024-09-16T05:20:20.802356+0000","last_clean":"2024-09-16T05:20:20.802356+0000","last_became_active":"2024-09-16T05:18:02.886933+0000","last_became_peered":"2024-09-16T05:18:02.886933+0000","last_unstale":"2024-09-16T05:20:20.802356+0000","last_undegraded":"2024-09-16T05:20:20.802356+0000","last_fullsized":"2024-09-16T05:20:20.802356+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:37.639669+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:37.639669+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:42.374370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":696,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155832+0000","last_change":"2024-09-16T05:20:23.800424+0000","last_active":"2024-09-16T05:20:25.155832+0000","last_peered":"2024-09-16T05:20:25.155832+0000","last_clean":"2024-09-16T05:20:25.155832+0000","last_became_active":"2024-09-16T05:19:35.793136+0000","last_became_peered":"2024-09-16T05:19:35.793136+0000","last_unstale":"2024-09-16T05:20:25.155832+0000","last_undegraded":"2024-09-16T05:20:25.155832+0000","last_fullsized":"2024-09-16T05:20:25.155832+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:36.435156+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.435156+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:42.430535+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00064612000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.7","version":"35'139","reported_seq":848,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.803839+0000","last_change":"2024-09-16T05:19:15.510805+0000","last_active":"2024-09-16T05:20:20.803839+0000","last_peered":"2024-09-16T05:20:20.803839+0000","last_clean":"2024-09-16T05:20:20.803839+0000","last_became_active":"2024-09-16T05:18:03.588064+0000","last_became_peered":"2024-09-16T05:18:03.588064+0000","last_unstale":"2024-09-16T05:20:20.803839+0000","last_undegraded":"2024-09-16T05:20:20.803839+0000","last_fullsized":"2024-09-16T05:20:20.803839+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:15.510717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.510717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:27.556487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":2991,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796521+0000","last_change":"2024-09-16T05:20:23.796781+0000","last_active":"2024-09-16T05:20:23.796521+0000","last_peered":"2024-09-16T05:20:23.796521+0000","last_clean":"2024-09-16T05:20:23.796521+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:20:23.796521+0000","last_undegraded":"2024-09-16T05:20:23.796521+0000","last_fullsized":"2024-09-16T05:20:23.796521+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:19:29.221207+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.221207+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:42.084933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089577700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.8","version":"35'139","reported_seq":1595,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.824525+0000","last_change":"2024-09-16T05:19:30.613103+0000","last_active":"2024-09-16T05:20:20.824525+0000","last_peered":"2024-09-16T05:20:20.824525+0000","last_clean":"2024-09-16T05:20:20.824525+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:20:20.824525+0000","last_undegraded":"2024-09-16T05:20:20.824525+0000","last_fullsized":"2024-09-16T05:20:20.824525+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.613024+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.613024+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.741797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"666'611","reported_seq":3447,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.795798+0000","last_change":"2024-09-16T05:20:20.798984+0000","last_active":"2024-09-16T05:20:21.795798+0000","last_peered":"2024-09-16T05:20:21.795798+0000","last_clean":"2024-09-16T05:20:21.795798+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:20:21.795798+0000","last_undegraded":"2024-09-16T05:20:21.795798+0000","last_fullsized":"2024-09-16T05:20:21.795798+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"653'601","last_scrub_stamp":"2024-09-16T05:20:03.311152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.311152+0000","objects_scrubbed":3,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:12.359872+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0021177420000000001,"stat_sum":{"num_bytes":4773551,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":127392,"num_write":49,"num_write_kb":31911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.9","version":"35'139","reported_seq":1593,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.836672+0000","last_change":"2024-09-16T05:20:20.802104+0000","last_active":"2024-09-16T05:20:21.836672+0000","last_peered":"2024-09-16T05:20:21.836672+0000","last_clean":"2024-09-16T05:20:21.836672+0000","last_became_active":"2024-09-16T05:20:20.801737+0000","last_became_peered":"2024-09-16T05:20:20.801737+0000","last_unstale":"2024-09-16T05:20:21.836672+0000","last_undegraded":"2024-09-16T05:20:21.836672+0000","last_fullsized":"2024-09-16T05:20:21.836672+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:02.384248+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.384248+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.409130+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1710,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.796437+0000","last_change":"2024-09-16T05:20:20.797159+0000","last_active":"2024-09-16T05:20:20.796437+0000","last_peered":"2024-09-16T05:20:20.796437+0000","last_clean":"2024-09-16T05:20:20.796437+0000","last_became_active":"2024-09-16T05:19:35.786308+0000","last_became_peered":"2024-09-16T05:19:35.786308+0000","last_unstale":"2024-09-16T05:20:20.796437+0000","last_undegraded":"2024-09-16T05:20:20.796437+0000","last_fullsized":"2024-09-16T05:20:20.796437+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:19:31.185579+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:31.185579+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:52.244549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001017207,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":1598,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857860+0000","last_change":"2024-09-16T05:19:32.947634+0000","last_active":"2024-09-16T05:20:24.857860+0000","last_peered":"2024-09-16T05:20:24.857860+0000","last_clean":"2024-09-16T05:20:24.857860+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:20:24.857860+0000","last_undegraded":"2024-09-16T05:20:24.857860+0000","last_fullsized":"2024-09-16T05:20:24.857860+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:32.947528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.947528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.139142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1088,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155283+0000","last_change":"2024-09-16T05:19:49.535280+0000","last_active":"2024-09-16T05:20:25.155283+0000","last_peered":"2024-09-16T05:20:25.155283+0000","last_clean":"2024-09-16T05:20:25.155283+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:20:25.155283+0000","last_undegraded":"2024-09-16T05:20:25.155283+0000","last_fullsized":"2024-09-16T05:20:25.155283+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:49.535206+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.535206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:17.718180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":708,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.803294+0000","last_change":"2024-09-16T05:20:20.803294+0000","last_active":"2024-09-16T05:20:20.803294+0000","last_peered":"2024-09-16T05:20:20.803294+0000","last_clean":"2024-09-16T05:20:20.803294+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:20:20.803294+0000","last_undegraded":"2024-09-16T05:20:20.803294+0000","last_fullsized":"2024-09-16T05:20:20.803294+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:19:07.382518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.382518+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.932593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070910600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.10","version":"35'139","reported_seq":1595,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.824394+0000","last_change":"2024-09-16T05:19:48.737475+0000","last_active":"2024-09-16T05:20:20.824394+0000","last_peered":"2024-09-16T05:20:20.824394+0000","last_clean":"2024-09-16T05:20:20.824394+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:20:20.824394+0000","last_undegraded":"2024-09-16T05:20:20.824394+0000","last_fullsized":"2024-09-16T05:20:20.824394+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:48.737348+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.737348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:57.195160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":2440,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858123+0000","last_change":"2024-09-16T05:20:23.806171+0000","last_active":"2024-09-16T05:20:24.858123+0000","last_peered":"2024-09-16T05:20:24.858123+0000","last_clean":"2024-09-16T05:20:24.858123+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:20:24.858123+0000","last_undegraded":"2024-09-16T05:20:24.858123+0000","last_fullsized":"2024-09-16T05:20:24.858123+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:36.009233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.009233+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:38.540686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026423110000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.14","version":"433'289","reported_seq":2712,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858147+0000","last_change":"2024-09-16T05:20:23.804360+0000","last_active":"2024-09-16T05:20:24.858147+0000","last_peered":"2024-09-16T05:20:24.858147+0000","last_clean":"2024-09-16T05:20:24.858147+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:20:24.858147+0000","last_undegraded":"2024-09-16T05:20:24.858147+0000","last_fullsized":"2024-09-16T05:20:24.858147+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:19:12.999550+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.999550+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.277329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013382859999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.15","version":"674'633","reported_seq":3529,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858188+0000","last_change":"2024-09-16T05:20:23.807266+0000","last_active":"2024-09-16T05:20:24.858188+0000","last_peered":"2024-09-16T05:20:24.858188+0000","last_clean":"2024-09-16T05:20:24.858188+0000","last_became_active":"2024-09-16T05:19:35.801137+0000","last_became_peered":"2024-09-16T05:19:35.801137+0000","last_unstale":"2024-09-16T05:20:24.858188+0000","last_undegraded":"2024-09-16T05:20:24.858188+0000","last_fullsized":"2024-09-16T05:20:24.858188+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"617'599","last_scrub_stamp":"2024-09-16T05:19:28.020270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.020270+0000","objects_scrubbed":4,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.489114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038891009999999998,"stat_sum":{"num_bytes":3964937,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":5506324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.16","version":"666'296","reported_seq":2741,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.825494+0000","last_change":"2024-09-16T05:20:20.825817+0000","last_active":"2024-09-16T05:20:20.825494+0000","last_peered":"2024-09-16T05:20:20.825494+0000","last_clean":"2024-09-16T05:20:20.825494+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:20:20.825494+0000","last_undegraded":"2024-09-16T05:20:20.825494+0000","last_fullsized":"2024-09-16T05:20:20.825494+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"665'294","last_scrub_stamp":"2024-09-16T05:20:15.010850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.010850+0000","objects_scrubbed":4,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.161084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001174942,"stat_sum":{"num_bytes":7403700,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":57070,"num_write":50,"num_write_kb":16677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"1.14","version":"35'139","reported_seq":919,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.796363+0000","last_change":"2024-09-16T05:18:46.335095+0000","last_active":"2024-09-16T05:20:22.796363+0000","last_peered":"2024-09-16T05:20:22.796363+0000","last_clean":"2024-09-16T05:20:22.796363+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:20:22.796363+0000","last_undegraded":"2024-09-16T05:20:22.796363+0000","last_fullsized":"2024-09-16T05:20:22.796363+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:46.335030+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.335030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":466,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.804710+0000","last_change":"2024-09-16T05:20:20.805116+0000","last_active":"2024-09-16T05:20:20.804710+0000","last_peered":"2024-09-16T05:20:20.804710+0000","last_clean":"2024-09-16T05:20:20.804710+0000","last_became_active":"2024-09-16T05:19:35.784084+0000","last_became_peered":"2024-09-16T05:19:35.784084+0000","last_unstale":"2024-09-16T05:20:20.804710+0000","last_undegraded":"2024-09-16T05:20:20.804710+0000","last_fullsized":"2024-09-16T05:20:20.804710+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:20:20.393358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.393358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:29.190706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013189619999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.18","version":"181'57","reported_seq":2822,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857937+0000","last_change":"2024-09-16T05:20:23.807570+0000","last_active":"2024-09-16T05:20:24.857937+0000","last_peered":"2024-09-16T05:20:24.857937+0000","last_clean":"2024-09-16T05:20:24.857937+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:20:24.857937+0000","last_undegraded":"2024-09-16T05:20:24.857937+0000","last_fullsized":"2024-09-16T05:20:24.857937+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:19:47.951135+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:19:47.951135+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:52.083499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003790168,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.19","version":"182'60","reported_seq":2427,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.836651+0000","last_change":"2024-09-16T05:20:20.797488+0000","last_active":"2024-09-16T05:20:22.836651+0000","last_peered":"2024-09-16T05:20:22.836651+0000","last_clean":"2024-09-16T05:20:22.836651+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:20:22.836651+0000","last_undegraded":"2024-09-16T05:20:22.836651+0000","last_fullsized":"2024-09-16T05:20:22.836651+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:20:17.403045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.403045+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:31.300467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016535600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.1a","version":"674'1001","reported_seq":3778,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.885939+0000","last_change":"2024-09-16T05:20:23.804002+0000","last_active":"2024-09-16T05:20:23.885939+0000","last_peered":"2024-09-16T05:20:23.885939+0000","last_clean":"2024-09-16T05:20:23.885939+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:20:23.885939+0000","last_undegraded":"2024-09-16T05:20:23.885939+0000","last_fullsized":"2024-09-16T05:20:23.885939+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"653'957","last_scrub_stamp":"2024-09-16T05:20:03.383437+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.383437+0000","objects_scrubbed":11,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.864097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0074082899999999997,"stat_sum":{"num_bytes":24006521,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":157,"num_read_kb":131937,"num_write":165,"num_write_kb":60609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.1b","version":"671'624","reported_seq":3454,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.795925+0000","last_change":"2024-09-16T05:20:20.799381+0000","last_active":"2024-09-16T05:20:21.795925+0000","last_peered":"2024-09-16T05:20:21.795925+0000","last_clean":"2024-09-16T05:20:21.795925+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:20:21.795925+0000","last_undegraded":"2024-09-16T05:20:21.795925+0000","last_fullsized":"2024-09-16T05:20:21.795925+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"666'612","last_scrub_stamp":"2024-09-16T05:20:16.258329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.258329+0000","objects_scrubbed":4,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:37.875166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027188680000000002,"stat_sum":{"num_bytes":11564336,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":94,"num_read_kb":89721,"num_write":102,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.1c","version":"664'452","reported_seq":2952,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.803359+0000","last_change":"2024-09-16T05:20:20.803588+0000","last_active":"2024-09-16T05:20:20.803359+0000","last_peered":"2024-09-16T05:20:20.803359+0000","last_clean":"2024-09-16T05:20:20.803359+0000","last_became_active":"2024-09-16T05:18:02.890972+0000","last_became_peered":"2024-09-16T05:18:02.890972+0000","last_unstale":"2024-09-16T05:20:20.803359+0000","last_undegraded":"2024-09-16T05:20:20.803359+0000","last_fullsized":"2024-09-16T05:20:20.803359+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"620'444","last_scrub_stamp":"2024-09-16T05:19:51.567043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.567043+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.900013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075473499999999998,"stat_sum":{"num_bytes":3294370,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":28,"num_write_kb":13180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.1d","version":"666'1182","reported_seq":4219,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.804527+0000","last_change":"2024-09-16T05:20:20.805111+0000","last_active":"2024-09-16T05:20:20.804527+0000","last_peered":"2024-09-16T05:20:20.804527+0000","last_clean":"2024-09-16T05:20:20.804527+0000","last_became_active":"2024-09-16T05:18:03.891103+0000","last_became_peered":"2024-09-16T05:18:03.891103+0000","last_unstale":"2024-09-16T05:20:20.804527+0000","last_undegraded":"2024-09-16T05:20:20.804527+0000","last_fullsized":"2024-09-16T05:20:20.804527+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"624'1113","last_scrub_stamp":"2024-09-16T05:19:35.593327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:35.593327+0000","objects_scrubbed":9,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:00.071753+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0013267649999999999,"stat_sum":{"num_bytes":14250762,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":246,"num_read_kb":196095,"num_write":229,"num_write_kb":73447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7155994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.1e","version":"671'464","reported_seq":3308,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.836982+0000","last_change":"2024-09-16T05:20:21.358794+0000","last_active":"2024-09-16T05:20:22.836982+0000","last_peered":"2024-09-16T05:20:22.836982+0000","last_clean":"2024-09-16T05:20:22.836982+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:20:22.836982+0000","last_undegraded":"2024-09-16T05:20:22.836982+0000","last_fullsized":"2024-09-16T05:20:22.836982+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"671'464","last_scrub_stamp":"2024-09-16T05:20:21.358700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:21.358700+0000","objects_scrubbed":3,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.772243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0047046620000000001,"stat_sum":{"num_bytes":3007609,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":71847,"num_write":25,"num_write_kb":11620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.1f","version":"284'221","reported_seq":3047,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155527+0000","last_change":"2024-09-16T05:20:23.800774+0000","last_active":"2024-09-16T05:20:25.155527+0000","last_peered":"2024-09-16T05:20:25.155527+0000","last_clean":"2024-09-16T05:20:25.155527+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:20:25.155527+0000","last_undegraded":"2024-09-16T05:20:25.155527+0000","last_fullsized":"2024-09-16T05:20:25.155527+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:20:17.544083+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.544083+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:36.068162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009795540000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.20","version":"666'305","reported_seq":3142,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837851+0000","last_change":"2024-09-16T05:20:20.796146+0000","last_active":"2024-09-16T05:20:21.837851+0000","last_peered":"2024-09-16T05:20:21.837851+0000","last_clean":"2024-09-16T05:20:21.837851+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:20:21.837851+0000","last_undegraded":"2024-09-16T05:20:21.837851+0000","last_fullsized":"2024-09-16T05:20:21.837851+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'290","last_scrub_stamp":"2024-09-16T05:20:03.351330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.351330+0000","objects_scrubbed":3,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:04.107696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013866110000000001,"stat_sum":{"num_bytes":6345760,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":144671,"num_write":58,"num_write_kb":21452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.21","version":"178'50","reported_seq":2755,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.795689+0000","last_change":"2024-09-16T05:20:20.795573+0000","last_active":"2024-09-16T05:20:22.795689+0000","last_peered":"2024-09-16T05:20:22.795689+0000","last_clean":"2024-09-16T05:20:22.795689+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:20:22.795689+0000","last_undegraded":"2024-09-16T05:20:22.795689+0000","last_fullsized":"2024-09-16T05:20:22.795689+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:02.565969+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.565969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:07.063824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098974200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.22","version":"182'20","reported_seq":716,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-16T05:20:20.803158+0000","last_change":"2024-09-16T05:20:20.803558+0000","last_active":"2024-09-16T05:20:20.803158+0000","last_peered":"2024-09-16T05:20:20.803158+0000","last_clean":"2024-09-16T05:20:20.803158+0000","last_became_active":"2024-09-16T05:17:59.866047+0000","last_became_peered":"2024-09-16T05:17:59.866047+0000","last_unstale":"2024-09-16T05:20:20.803158+0000","last_undegraded":"2024-09-16T05:20:20.803158+0000","last_fullsized":"2024-09-16T05:20:20.803158+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:15.452705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.452705+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.222526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074852199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":2300,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.836785+0000","last_change":"2024-09-16T05:20:20.802550+0000","last_active":"2024-09-16T05:20:22.836785+0000","last_peered":"2024-09-16T05:20:22.836785+0000","last_clean":"2024-09-16T05:20:22.836785+0000","last_became_active":"2024-09-16T05:20:20.801064+0000","last_became_peered":"2024-09-16T05:20:20.801064+0000","last_unstale":"2024-09-16T05:20:22.836785+0000","last_undegraded":"2024-09-16T05:20:22.836785+0000","last_fullsized":"2024-09-16T05:20:22.836785+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:20:00.415257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.415257+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.760486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00099487100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.24","version":"182'89","reported_seq":2116,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858054+0000","last_change":"2024-09-16T05:20:23.803973+0000","last_active":"2024-09-16T05:20:24.858054+0000","last_peered":"2024-09-16T05:20:24.858054+0000","last_clean":"2024-09-16T05:20:24.858054+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:20:24.858054+0000","last_undegraded":"2024-09-16T05:20:24.858054+0000","last_fullsized":"2024-09-16T05:20:24.858054+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:57.859756+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:19:57.859756+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.776621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097636700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.25","version":"674'1100","reported_seq":3835,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.829577+0000","last_change":"2024-09-16T05:20:23.800887+0000","last_active":"2024-09-16T05:20:23.829577+0000","last_peered":"2024-09-16T05:20:23.829577+0000","last_clean":"2024-09-16T05:20:23.829577+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:20:23.829577+0000","last_undegraded":"2024-09-16T05:20:23.829577+0000","last_fullsized":"2024-09-16T05:20:23.829577+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"622'977","last_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.332491+0000","objects_scrubbed":9,"log_size":1100,"log_dups_size":0,"ondisk_log_size":1100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:50.406942+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0.0045244719999999999,"stat_sum":{"num_bytes":27092020,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":188,"num_read_kb":197711,"num_write":167,"num_write_kb":66134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.26","version":"116'9","reported_seq":2090,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837301+0000","last_change":"2024-09-16T05:20:20.798451+0000","last_active":"2024-09-16T05:20:21.837301+0000","last_peered":"2024-09-16T05:20:21.837301+0000","last_clean":"2024-09-16T05:20:21.837301+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:20:21.837301+0000","last_undegraded":"2024-09-16T05:20:21.837301+0000","last_fullsized":"2024-09-16T05:20:21.837301+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:07.483055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.483055+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024903289999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.27","version":"0'0","reported_seq":2102,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837618+0000","last_change":"2024-09-16T05:20:20.798013+0000","last_active":"2024-09-16T05:20:21.837618+0000","last_peered":"2024-09-16T05:20:21.837618+0000","last_clean":"2024-09-16T05:20:21.837618+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:20:21.837618+0000","last_undegraded":"2024-09-16T05:20:21.837618+0000","last_fullsized":"2024-09-16T05:20:21.837618+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:11.499561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.499561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:38.263413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023941380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.28","version":"245'105","reported_seq":1118,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.795618+0000","last_change":"2024-09-16T05:20:20.794206+0000","last_active":"2024-09-16T05:20:22.795618+0000","last_peered":"2024-09-16T05:20:22.795618+0000","last_clean":"2024-09-16T05:20:22.795618+0000","last_became_active":"2024-09-16T05:19:35.785220+0000","last_became_peered":"2024-09-16T05:19:35.785220+0000","last_unstale":"2024-09-16T05:20:22.795618+0000","last_undegraded":"2024-09-16T05:20:22.795618+0000","last_fullsized":"2024-09-16T05:20:22.795618+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:19:27.303070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:27.303070+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:44.402060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048973199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.29","version":"672'982","reported_seq":3816,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.813525+0000","last_change":"2024-09-16T05:20:20.798341+0000","last_active":"2024-09-16T05:20:21.813525+0000","last_peered":"2024-09-16T05:20:21.813525+0000","last_clean":"2024-09-16T05:20:21.813525+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:20:21.813525+0000","last_undegraded":"2024-09-16T05:20:21.813525+0000","last_fullsized":"2024-09-16T05:20:21.813525+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"621'852","last_scrub_stamp":"2024-09-16T05:19:32.048085+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.048085+0000","objects_scrubbed":8,"log_size":982,"log_dups_size":0,"ondisk_log_size":982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.837233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0043544380000000004,"stat_sum":{"num_bytes":25096059,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":255,"num_read_kb":246175,"num_write":186,"num_write_kb":67059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.2a","version":"245'325","reported_seq":3092,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.836780+0000","last_change":"2024-09-16T05:20:20.798237+0000","last_active":"2024-09-16T05:20:21.836780+0000","last_peered":"2024-09-16T05:20:21.836780+0000","last_clean":"2024-09-16T05:20:21.836780+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:20:21.836780+0000","last_undegraded":"2024-09-16T05:20:21.836780+0000","last_fullsized":"2024-09-16T05:20:21.836780+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:19:29.329416+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.329416+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:36.194641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025722700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.2b","version":"666'332","reported_seq":3066,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-09-16T05:20:22.795954+0000","last_change":"2024-09-16T05:20:20.796194+0000","last_active":"2024-09-16T05:20:22.795954+0000","last_peered":"2024-09-16T05:20:22.795954+0000","last_clean":"2024-09-16T05:20:22.795954+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:20:22.795954+0000","last_undegraded":"2024-09-16T05:20:22.795954+0000","last_fullsized":"2024-09-16T05:20:22.795954+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"666'332","last_scrub_stamp":"2024-09-16T05:20:18.508468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.508468+0000","objects_scrubbed":1,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:24.646519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011705800000000001,"stat_sum":{"num_bytes":2672478,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":79889,"num_write":47,"num_write_kb":21198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.2c","version":"241'194","reported_seq":2938,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796263+0000","last_change":"2024-09-16T05:20:23.796528+0000","last_active":"2024-09-16T05:20:23.796263+0000","last_peered":"2024-09-16T05:20:23.796263+0000","last_clean":"2024-09-16T05:20:23.796263+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:20:23.796263+0000","last_undegraded":"2024-09-16T05:20:23.796263+0000","last_fullsized":"2024-09-16T05:20:23.796263+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:19:51.201665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.201665+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:59.115742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068113899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.2d","version":"666'459","reported_seq":3262,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.836916+0000","last_change":"2024-09-16T05:20:20.796879+0000","last_active":"2024-09-16T05:20:21.836916+0000","last_peered":"2024-09-16T05:20:21.836916+0000","last_clean":"2024-09-16T05:20:21.836916+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:20:21.836916+0000","last_undegraded":"2024-09-16T05:20:21.836916+0000","last_fullsized":"2024-09-16T05:20:21.836916+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"648'457","last_scrub_stamp":"2024-09-16T05:20:15.223777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.223777+0000","objects_scrubbed":2,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.126285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001509947,"stat_sum":{"num_bytes":2823741,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":87,"num_read_kb":88579,"num_write":47,"num_write_kb":22068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.2e","version":"666'612","reported_seq":3367,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-09-16T05:20:21.837129+0000","last_change":"2024-09-16T05:20:20.798364+0000","last_active":"2024-09-16T05:20:21.837129+0000","last_peered":"2024-09-16T05:20:21.837129+0000","last_clean":"2024-09-16T05:20:21.837129+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:20:21.837129+0000","last_undegraded":"2024-09-16T05:20:21.837129+0000","last_fullsized":"2024-09-16T05:20:21.837129+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"617'552","last_scrub_stamp":"2024-09-16T05:19:28.348887+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.348887+0000","objects_scrubbed":7,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:39.253290+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0.002512542,"stat_sum":{"num_bytes":13301925,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":42113,"num_write":86,"num_write_kb":33834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"1"},{"start":"110","length":"2"}]},{"pgid":"2.2f","version":"674'580","reported_seq":3436,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858256+0000","last_change":"2024-09-16T05:20:23.809617+0000","last_active":"2024-09-16T05:20:24.858256+0000","last_peered":"2024-09-16T05:20:24.858256+0000","last_clean":"2024-09-16T05:20:24.858256+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:20:24.858256+0000","last_undegraded":"2024-09-16T05:20:24.858256+0000","last_fullsized":"2024-09-16T05:20:24.858256+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"650'553","last_scrub_stamp":"2024-09-16T05:20:00.804910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.804910+0000","objects_scrubbed":6,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.252445+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0057028620000000004,"stat_sum":{"num_bytes":10084662,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":108,"num_read_kb":77807,"num_write":97,"num_write_kb":39298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":276685838,"num_objects":127,"num_object_clones":80,"num_object_copies":254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":14,"num_read":3158,"num_read_kb":3106178,"num_write":2313,"num_write_kb":920856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":47028772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":361631744,"data_stored":451527333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24243,"ondisk_log_size":24243,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089480,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108280,"kb_used_data":54880,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93636872,"statfs":{"total":95995035648,"available":95884156928,"internally_reserved":0,"allocated":56197120,"data_stored":59986025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285708,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74828,"kb_used_data":43636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93670324,"statfs":{"total":95995035648,"available":95918411776,"internally_reserved":0,"allocated":44683264,"data_stored":48574872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514646,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89200,"kb_used_data":35808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93655952,"statfs":{"total":95995035648,"available":95903694848,"internally_reserved":0,"allocated":36667392,"data_stored":43437466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841685,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81608,"kb_used_data":50420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93663544,"statfs":{"total":95995035648,"available":95911469056,"internally_reserved":0,"allocated":51630080,"data_stored":53570372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103323,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104272,"kb_used_data":46716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93640880,"statfs":{"total":95995035648,"available":95888261120,"internally_reserved":0,"allocated":47837184,"data_stored":52287011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397666,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83104,"kb_used_data":47748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93662048,"statfs":{"total":95995035648,"available":95909937152,"internally_reserved":0,"allocated":48893952,"data_stored":55368735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757412,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63772,"kb_used_data":36744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93681380,"statfs":{"total":95995035648,"available":95929733120,"internally_reserved":0,"allocated":37625856,"data_stored":44002904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920938,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51136,"kb_used_data":24100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93694016,"statfs":{"total":95995035648,"available":95942672384,"internally_reserved":0,"allocated":24678400,"data_stored":25411407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117167,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83504,"kb_used_data":48152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93661648,"statfs":{"total":95995035648,"available":95909527552,"internally_reserved":0,"allocated":49307648,"data_stored":64627970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240454,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55520,"kb_used_data":19080,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93689632,"statfs":{"total":95995035648,"available":95938183168,"internally_reserved":0,"allocated":19537920,"data_stored":17895321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48300,"kb_used_data":17112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93696852,"statfs":{"total":95995035648,"available":95945576448,"internally_reserved":0,"allocated":17522688,"data_stored":19004943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9052160,"data_stored":13275819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18554880,"data_stored":23395830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11153408,"data_stored":12248106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40861696,"data_stored":61221619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16584704,"data_stored":20043945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31154176,"data_stored":40249153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40775680,"data_stored":49984208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":39731200,"data_stored":46910937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40570880,"data_stored":48352786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28524544,"data_stored":38036022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36589568,"data_stored":43207410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":48078848,"data_stored":54601498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:20:27.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:20:27.512 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:27.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:27.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:27.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:27.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:27.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:27.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:27.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:27.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:27.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:27.804 INFO:tasks.rados.rados.2.smithi204.stdout:2647: done (7 left) 2024-09-16T05:20:27.804 INFO:tasks.rados.rados.2.smithi204.stdout:2648: snap_remove snap 267 2024-09-16T05:20:27.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:27.918 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:27.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:28.008 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:28.008 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:28.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:28.044 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:28.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:28.065 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:28.067 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:28.067 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: pgmap v1176: 107 pgs: 1 peering, 106 active+clean; 266 MiB data, 827 MiB used, 983 GiB / 983 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 24 op/s; 411 KiB/s, 5 objects/s recovering 2024-09-16T05:20:28.067 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: osdmap e678: 12 total, 12 up, 11 in 2024-09-16T05:20:28.067 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:20:28.067 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:20:28.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:20:28.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:20:28.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:20:28.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:20:28.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:27 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:20:28.071 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:28.087 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:28.087 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:28.108 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:28.109 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:28.129 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:28.130 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:28.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:28.150 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:28.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:28.169 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:28.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:28.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: pgmap v1176: 107 pgs: 1 peering, 106 active+clean; 266 MiB data, 827 MiB used, 983 GiB / 983 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 24 op/s; 411 KiB/s, 5 objects/s recovering 2024-09-16T05:20:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: osdmap e678: 12 total, 12 up, 11 in 2024-09-16T05:20:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:20:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:20:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:20:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:20:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:20:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:20:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:27 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:20:28.172 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:28.172 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:28.194 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:28.194 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:28.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:28.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: pgmap v1176: 107 pgs: 1 peering, 106 active+clean; 266 MiB data, 827 MiB used, 983 GiB / 983 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 24 op/s; 411 KiB/s, 5 objects/s recovering 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: osdmap e678: 12 total, 12 up, 11 in 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:20:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:27 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:20:28.239 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:28.239 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:28.261 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:28.261 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:28.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:28.284 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:28.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:28.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:28.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:28.305 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:28.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:28.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:28.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:28.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:28.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:28.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:28.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:28.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:20:28.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:28.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:28.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:28.791 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:28.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:28.808 INFO:tasks.rados.rados.2.smithi204.stdout:2648: done (7 left) 2024-09-16T05:20:28.808 INFO:tasks.rados.rados.2.smithi204.stdout:2649: read oid 26 snap -1 2024-09-16T05:20:28.808 INFO:tasks.rados.rados.2.smithi204.stdout:2649: expect (ObjNum 890 snap 262 seq_num 890) 2024-09-16T05:20:28.808 INFO:tasks.rados.rados.2.smithi204.stdout:2650: copy_from oid 24 from oid 40 current snap is 277 2024-09-16T05:20:28.808 INFO:tasks.rados.rados.2.smithi204.stdout:2639: expect (ObjNum 930 snap 275 seq_num 930) 2024-09-16T05:20:28.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:28.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:28.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:28.929 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:28.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:20:28.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/782655132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:28.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:28 smithi196 ceph-mon[30802]: osdmap e679: 12 total, 12 up, 11 in 2024-09-16T05:20:29.008 INFO:tasks.rados.rados.2.smithi204.stdout:2643: finishing write tid 1 to smithi20457400-14 2024-09-16T05:20:29.008 INFO:tasks.rados.rados.2.smithi204.stdout:2643: finishing write tid 2 to smithi20457400-14 2024-09-16T05:20:29.009 INFO:tasks.rados.rados.2.smithi204.stdout:2643: finishing write tid 3 to smithi20457400-14 2024-09-16T05:20:29.009 INFO:tasks.rados.rados.2.smithi204.stdout:2643: finishing write tid 4 to smithi20457400-14 2024-09-16T05:20:29.009 INFO:tasks.rados.rados.2.smithi204.stdout:2643: finishing write tid 5 to smithi20457400-14 2024-09-16T05:20:29.009 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1191 (ObjNum 934 snap 277 seq_num 934) dirty exists 2024-09-16T05:20:29.009 INFO:tasks.rados.rados.2.smithi204.stdout:2643: left oid 14 (ObjNum 934 snap 277 seq_num 934) 2024-09-16T05:20:29.009 INFO:tasks.rados.rados.2.smithi204.stdout:2644: expect (ObjNum 815 snap 237 seq_num 815) 2024-09-16T05:20:29.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:29.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:29.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:29.049 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/782655132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:29.049 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:28 smithi204 ceph-mon[30933]: osdmap e679: 12 total, 12 up, 11 in 2024-09-16T05:20:29.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:29.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:29.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/782655132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:28 smithi032 ceph-mon[27388]: osdmap e679: 12 total, 12 up, 11 in 2024-09-16T05:20:29.228 INFO:tasks.rados.rados.2.smithi204.stdout:2646: finishing write tid 1 to smithi20457400-29 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2646: finishing write tid 2 to smithi20457400-29 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2646: finishing write tid 3 to smithi20457400-29 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2639: done (8 left) 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2640: done (7 left) 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2641: done (6 left) 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2643: done (5 left) 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2644: done (4 left) 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2645: done (3 left) 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2651: read oid 20 snap 260 2024-09-16T05:20:29.229 INFO:tasks.rados.rados.2.smithi204.stdout:2651: expect (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:20:29.235 INFO:tasks.rados.rados.2.smithi204.stdout:2652: rollback oid 14 current snap is 277 2024-09-16T05:20:29.235 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 263 2024-09-16T05:20:29.235 INFO:tasks.rados.rados.2.smithi204.stdout:2646: finishing write tid 4 to smithi20457400-29 2024-09-16T05:20:29.235 INFO:tasks.rados.rados.2.smithi204.stdout:2646: finishing write tid 5 to smithi20457400-29 2024-09-16T05:20:29.236 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1012 (ObjNum 935 snap 277 seq_num 935) dirty exists 2024-09-16T05:20:29.236 INFO:tasks.rados.rados.2.smithi204.stdout:2646: left oid 29 (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:20:29.236 INFO:tasks.rados.rados.2.smithi204.stdout:2649: expect (ObjNum 890 snap 262 seq_num 890) 2024-09-16T05:20:29.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:29.290 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:29.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:29.325 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:29.333 INFO:tasks.rados.rados.2.smithi204.stdout:2650: finishing copy_from racing read to smithi20457400-24 2024-09-16T05:20:29.333 INFO:tasks.rados.rados.2.smithi204.stdout:2646: done (4 left) 2024-09-16T05:20:29.333 INFO:tasks.rados.rados.2.smithi204.stdout:2649: done (3 left) 2024-09-16T05:20:29.334 INFO:tasks.rados.rados.2.smithi204.stdout:2653: write (excl) oid 1 current snap is 277 2024-09-16T05:20:29.334 INFO:tasks.rados.rados.2.smithi204.stdout:2650: finishing copy_from to smithi20457400-24 2024-09-16T05:20:29.334 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 777 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:20:29.334 INFO:tasks.rados.rados.2.smithi204.stdout:2653: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2024-09-16T05:20:29.340 INFO:tasks.rados.rados.2.smithi204.stdout:2653: writing smithi20457400-1 from 531352 to 1275835 tid 1 2024-09-16T05:20:29.348 INFO:tasks.rados.rados.2.smithi204.stdout:2653: writing smithi20457400-1 from 2005734 to 2785059 tid 2 2024-09-16T05:20:29.354 INFO:tasks.rados.rados.2.smithi204.stdout:2653: writing smithi20457400-1 from 3487333 to 3731352 tid 3 2024-09-16T05:20:29.355 INFO:tasks.rados.rados.2.smithi204.stdout:2650: done (3 left) 2024-09-16T05:20:29.355 INFO:tasks.rados.rados.2.smithi204.stdout:2654: copy_from oid 9 from oid 49 current snap is 277 2024-09-16T05:20:29.355 INFO:tasks.rados.rados.2.smithi204.stdout:2651: expect (ObjNum 858 snap 248 seq_num 858) 2024-09-16T05:20:29.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:29.405 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:20:29.405 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:20:29.405 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:20:29.405 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:20:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:29.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:29.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:20:29.473 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:20:29.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:29.524 INFO:tasks.rados.rados.2.smithi204.stdout:2652: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:20:29.524 INFO:tasks.rados.rados.2.smithi204.stdout:2652: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:20:29.524 INFO:tasks.rados.rados.2.smithi204.stdout:2651: done (3 left) 2024-09-16T05:20:29.524 INFO:tasks.rados.rados.2.smithi204.stdout:2655: read oid 7 snap -1 2024-09-16T05:20:29.524 INFO:tasks.rados.rados.2.smithi204.stdout:2655: expect (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:20:29.524 INFO:tasks.rados.rados.2.smithi204.stdout:2656: snap_create 2024-09-16T05:20:29.524 INFO:tasks.rados.rados.2.smithi204.stdout:2652: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:20:29.524 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1198 (ObjNum 896 snap 262 seq_num 896) dirty exists 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:2653: finishing write tid 1 to smithi20457400-1 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:2653: finishing write tid 2 to smithi20457400-1 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:2653: finishing write tid 3 to smithi20457400-1 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:2653: finishing write tid 4 to smithi20457400-1 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:2653: finishing write tid 5 to smithi20457400-1 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 556 (ObjNum 936 snap 277 seq_num 936) dirty exists 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:2653: left oid 1 (ObjNum 936 snap 277 seq_num 936) 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:2654: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:2654: finishing copy_from to smithi20457400-9 2024-09-16T05:20:29.525 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 624 (ObjNum 921 snap 273 seq_num 921) dirty exists 2024-09-16T05:20:29.534 INFO:tasks.rados.rados.2.smithi204.stdout:2655: expect (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:20:29.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:29.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:29.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:29.662 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:29.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:29.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:29.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:29.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:29.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:2652: done (4 left) 2024-09-16T05:20:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:2653: done (3 left) 2024-09-16T05:20:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:2654: done (2 left) 2024-09-16T05:20:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:2655: done (1 left) 2024-09-16T05:20:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:2656: done (0 left) 2024-09-16T05:20:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:2657: write (excl) oid 45 current snap is 278 2024-09-16T05:20:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:2657: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-09-16T05:20:29.875 INFO:tasks.rados.rados.2.smithi204.stdout:2657: writing smithi20457400-45 from 548159 to 967891 tid 1 2024-09-16T05:20:29.882 INFO:tasks.rados.rados.2.smithi204.stdout:2657: writing smithi20457400-45 from 1664216 to 2187199 tid 2 2024-09-16T05:20:29.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:29.889 INFO:tasks.rados.rados.2.smithi204.stdout:2657: writing smithi20457400-45 from 2714756 to 3328985 tid 3 2024-09-16T05:20:29.889 INFO:tasks.rados.rados.2.smithi204.stdout:2658: delete oid 28 current snap is 278 2024-09-16T05:20:29.889 INFO:tasks.rados.rados.2.smithi204.stdout:2657: finishing write tid 1 to smithi20457400-45 2024-09-16T05:20:29.889 INFO:tasks.rados.rados.2.smithi204.stdout:2657: finishing write tid 2 to smithi20457400-45 2024-09-16T05:20:29.892 INFO:tasks.rados.rados.2.smithi204.stdout:2657: finishing write tid 3 to smithi20457400-45 2024-09-16T05:20:29.892 INFO:tasks.rados.rados.2.smithi204.stdout:2657: finishing write tid 4 to smithi20457400-45 2024-09-16T05:20:29.892 INFO:tasks.rados.rados.2.smithi204.stdout:2657: finishing write tid 5 to smithi20457400-45 2024-09-16T05:20:29.892 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1015 (ObjNum 937 snap 278 seq_num 937) dirty exists 2024-09-16T05:20:29.892 INFO:tasks.rados.rados.2.smithi204.stdout:2657: left oid 45 (ObjNum 937 snap 278 seq_num 937) 2024-09-16T05:20:29.892 INFO:tasks.rados.rados.2.smithi204.stdout:2657: done (1 left) 2024-09-16T05:20:29.892 INFO:tasks.rados.rados.2.smithi204.stdout:2658: done (0 left) 2024-09-16T05:20:29.892 INFO:tasks.rados.rados.2.smithi204.stdout:2659: snap_remove snap 275 2024-09-16T05:20:29.911 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:29.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:30.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:30.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:30.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:30.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:20:30.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:30.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:30.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:29 smithi032 ceph-mon[27388]: pgmap v1179: 107 pgs: 107 active+clean; 267 MiB data, 834 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-16T05:20:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:29 smithi204 ceph-mon[30933]: pgmap v1179: 107 pgs: 107 active+clean; 267 MiB data, 834 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-16T05:20:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:29 smithi196 ceph-mon[30802]: pgmap v1179: 107 pgs: 107 active+clean; 267 MiB data, 834 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-16T05:20:30.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:30.277 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1180,"stamp":"2024-09-16T05:20:28.801806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280349308,"num_objects":132,"num_object_clones":84,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":14,"num_read":3199,"num_read_kb":3162571,"num_write":2331,"num_write_kb":926085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":47028772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27228,"ondisk_log_size":27228,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":215,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":854524,"kb_used_data":435416,"kb_used_omap":26,"kb_used_meta":418853,"kb_avail":1030342148,"statfs":{"total":1055945392128,"available":1055070359552,"internally_reserved":0,"allocated":445865984,"data_stored":508442077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26910,"internal_metadata":428906210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5625406,"num_objects":6,"num_object_clones":7,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":43,"num_read_kb":56394,"num_write":25,"num_write_kb":7035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.947435"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":3296,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156230+0000","last_change":"2024-09-16T05:20:23.808412+0000","last_active":"2024-09-16T05:20:25.156230+0000","last_peered":"2024-09-16T05:20:25.156230+0000","last_clean":"2024-09-16T05:20:25.156230+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:20:25.156230+0000","last_undegraded":"2024-09-16T05:20:25.156230+0000","last_fullsized":"2024-09-16T05:20:25.156230+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:20:01.826054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:01.826054+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.488219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012806409999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.54","version":"674'397","reported_seq":2850,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828533+0000","last_change":"2024-09-16T05:20:26.828782+0000","last_active":"2024-09-16T05:20:26.828533+0000","last_peered":"2024-09-16T05:20:26.828533+0000","last_clean":"2024-09-16T05:20:26.828533+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:20:26.828533+0000","last_undegraded":"2024-09-16T05:20:26.828533+0000","last_fullsized":"2024-09-16T05:20:26.828533+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"619'381","last_scrub_stamp":"2024-09-16T05:19:29.293042+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.293042+0000","objects_scrubbed":3,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:43.684504+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.001110469,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":57049,"num_write":34,"num_write_kb":11523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":1190,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.794330+0000","last_change":"2024-09-16T05:20:23.795697+0000","last_active":"2024-09-16T05:20:25.794330+0000","last_peered":"2024-09-16T05:20:25.794330+0000","last_clean":"2024-09-16T05:20:25.794330+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:20:25.794330+0000","last_undegraded":"2024-09-16T05:20:25.794330+0000","last_fullsized":"2024-09-16T05:20:25.794330+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:32.054132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.054132+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.553292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040824999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":1190,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155269+0000","last_change":"2024-09-16T05:20:23.799550+0000","last_active":"2024-09-16T05:20:25.155269+0000","last_peered":"2024-09-16T05:20:25.155269+0000","last_clean":"2024-09-16T05:20:25.155269+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:20:25.155269+0000","last_undegraded":"2024-09-16T05:20:25.155269+0000","last_fullsized":"2024-09-16T05:20:25.155269+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:18.588272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.588272+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:33.267855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040587400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.51","version":"272'115","reported_seq":1257,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801486+0000","last_change":"2024-09-16T05:20:23.808960+0000","last_active":"2024-09-16T05:20:25.801486+0000","last_peered":"2024-09-16T05:20:25.801486+0000","last_clean":"2024-09-16T05:20:25.801486+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:20:25.801486+0000","last_undegraded":"2024-09-16T05:20:25.801486+0000","last_fullsized":"2024-09-16T05:20:25.801486+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:19:29.589220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.589220+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:40.708156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001479932,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":1203,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.808945+0000","last_change":"2024-09-16T05:20:27.809239+0000","last_active":"2024-09-16T05:20:27.808945+0000","last_peered":"2024-09-16T05:20:27.808945+0000","last_clean":"2024-09-16T05:20:27.808945+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:20:27.808945+0000","last_undegraded":"2024-09-16T05:20:27.808945+0000","last_fullsized":"2024-09-16T05:20:27.808945+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:29.302740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.302740+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.020231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000430252,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2187,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.809688+0000","last_change":"2024-09-16T05:20:26.810092+0000","last_active":"2024-09-16T05:20:26.809688+0000","last_peered":"2024-09-16T05:20:26.809688+0000","last_clean":"2024-09-16T05:20:26.809688+0000","last_became_active":"2024-09-16T05:19:35.789805+0000","last_became_peered":"2024-09-16T05:19:35.789805+0000","last_unstale":"2024-09-16T05:20:26.809688+0000","last_undegraded":"2024-09-16T05:20:26.809688+0000","last_fullsized":"2024-09-16T05:20:26.809688+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:19:48.302807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.302807+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:06.121391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0063601899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.4e","version":"272'232","reported_seq":2957,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156424+0000","last_change":"2024-09-16T05:20:23.808015+0000","last_active":"2024-09-16T05:20:25.156424+0000","last_peered":"2024-09-16T05:20:25.156424+0000","last_clean":"2024-09-16T05:20:25.156424+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:20:25.156424+0000","last_undegraded":"2024-09-16T05:20:25.156424+0000","last_fullsized":"2024-09-16T05:20:25.156424+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:50.792260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.792260+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:15.909321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00088863899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.4d","version":"665'626","reported_seq":3519,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828730+0000","last_change":"2024-09-16T05:20:26.828984+0000","last_active":"2024-09-16T05:20:26.828730+0000","last_peered":"2024-09-16T05:20:26.828730+0000","last_clean":"2024-09-16T05:20:26.828730+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:20:26.828730+0000","last_undegraded":"2024-09-16T05:20:26.828730+0000","last_fullsized":"2024-09-16T05:20:26.828730+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"665'626","last_scrub_stamp":"2024-09-16T05:20:16.492166+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.492166+0000","objects_scrubbed":4,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.198501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013191909999999999,"stat_sum":{"num_bytes":6503878,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":45,"num_read_kb":38966,"num_write":56,"num_write_kb":18053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.4c","version":"674'350","reported_seq":3174,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.802040+0000","last_change":"2024-09-16T05:20:23.808587+0000","last_active":"2024-09-16T05:20:25.802040+0000","last_peered":"2024-09-16T05:20:25.802040+0000","last_clean":"2024-09-16T05:20:25.802040+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:20:25.802040+0000","last_undegraded":"2024-09-16T05:20:25.802040+0000","last_fullsized":"2024-09-16T05:20:25.802040+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"666'345","last_scrub_stamp":"2024-09-16T05:20:17.450506+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.450506+0000","objects_scrubbed":1,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.178106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001716029,"stat_sum":{"num_bytes":5239557,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":37572,"num_write":33,"num_write_kb":9268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":3001,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828242+0000","last_change":"2024-09-16T05:20:26.828501+0000","last_active":"2024-09-16T05:20:26.828242+0000","last_peered":"2024-09-16T05:20:26.828242+0000","last_clean":"2024-09-16T05:20:26.828242+0000","last_became_active":"2024-09-16T05:17:21.741180+0000","last_became_peered":"2024-09-16T05:17:21.741180+0000","last_unstale":"2024-09-16T05:20:26.828242+0000","last_undegraded":"2024-09-16T05:20:26.828242+0000","last_fullsized":"2024-09-16T05:20:26.828242+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:19:32.417815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.417815+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:40.936009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096174599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.4a","version":"620'557","reported_seq":3346,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801566+0000","last_change":"2024-09-16T05:20:23.809821+0000","last_active":"2024-09-16T05:20:25.801566+0000","last_peered":"2024-09-16T05:20:25.801566+0000","last_clean":"2024-09-16T05:20:25.801566+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:20:25.801566+0000","last_undegraded":"2024-09-16T05:20:25.801566+0000","last_fullsized":"2024-09-16T05:20:25.801566+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"620'557","last_scrub_stamp":"2024-09-16T05:19:54.555606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.555606+0000","objects_scrubbed":1,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.900460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099282899999999998,"stat_sum":{"num_bytes":2689285,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":47,"num_read_kb":41138,"num_write":35,"num_write_kb":18755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":3034,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155846+0000","last_change":"2024-09-16T05:20:23.807313+0000","last_active":"2024-09-16T05:20:25.155846+0000","last_peered":"2024-09-16T05:20:25.155846+0000","last_clean":"2024-09-16T05:20:25.155846+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:20:25.155846+0000","last_undegraded":"2024-09-16T05:20:25.155846+0000","last_fullsized":"2024-09-16T05:20:25.155846+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:20.094577+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.094577+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.217200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043004700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.48","version":"674'318","reported_seq":3100,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857853+0000","last_change":"2024-09-16T05:20:23.807986+0000","last_active":"2024-09-16T05:20:24.857853+0000","last_peered":"2024-09-16T05:20:24.857853+0000","last_clean":"2024-09-16T05:20:24.857853+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:20:24.857853+0000","last_undegraded":"2024-09-16T05:20:24.857853+0000","last_fullsized":"2024-09-16T05:20:24.857853+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"628'287","last_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:19:38.000359+0000","objects_scrubbed":3,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:39.573694+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0038763560000000001,"stat_sum":{"num_bytes":7103158,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":55585,"num_write":45,"num_write_kb":22506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.47","version":"0'0","reported_seq":1808,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155348+0000","last_change":"2024-09-16T05:20:23.800484+0000","last_active":"2024-09-16T05:20:25.155348+0000","last_peered":"2024-09-16T05:20:25.155348+0000","last_clean":"2024-09-16T05:20:25.155348+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:20:25.155348+0000","last_undegraded":"2024-09-16T05:20:25.155348+0000","last_fullsized":"2024-09-16T05:20:25.155348+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:30.390373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.390373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.511493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085393000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.46","version":"116'9","reported_seq":44,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828474+0000","last_change":"2024-09-16T05:20:26.828650+0000","last_active":"2024-09-16T05:20:26.828474+0000","last_peered":"2024-09-16T05:20:26.828474+0000","last_clean":"2024-09-16T05:20:26.828474+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:20:26.828474+0000","last_undegraded":"2024-09-16T05:20:26.828474+0000","last_fullsized":"2024-09-16T05:20:26.828474+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:49.313870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.313870+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:56.333549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001011571,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":2775,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801482+0000","last_change":"2024-09-16T05:20:23.808800+0000","last_active":"2024-09-16T05:20:25.801482+0000","last_peered":"2024-09-16T05:20:25.801482+0000","last_clean":"2024-09-16T05:20:25.801482+0000","last_became_active":"2024-09-16T05:19:35.786934+0000","last_became_peered":"2024-09-16T05:19:35.786934+0000","last_unstale":"2024-09-16T05:20:25.801482+0000","last_undegraded":"2024-09-16T05:20:25.801482+0000","last_fullsized":"2024-09-16T05:20:25.801482+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:19:09.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:09.927260+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:29.166375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001907401,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":1400,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.817134+0000","last_change":"2024-09-16T05:20:26.817356+0000","last_active":"2024-09-16T05:20:26.817134+0000","last_peered":"2024-09-16T05:20:26.817134+0000","last_clean":"2024-09-16T05:20:26.817134+0000","last_became_active":"2024-09-16T05:19:35.788828+0000","last_became_peered":"2024-09-16T05:19:35.788828+0000","last_unstale":"2024-09-16T05:20:26.817134+0000","last_undegraded":"2024-09-16T05:20:26.817134+0000","last_fullsized":"2024-09-16T05:20:26.817134+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:48.163046+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.163046+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:13.176337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0054578780000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":1805,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.808080+0000","last_change":"2024-09-16T05:20:26.808512+0000","last_active":"2024-09-16T05:20:26.808080+0000","last_peered":"2024-09-16T05:20:26.808080+0000","last_clean":"2024-09-16T05:20:26.808080+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:20:26.808080+0000","last_undegraded":"2024-09-16T05:20:26.808080+0000","last_fullsized":"2024-09-16T05:20:26.808080+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:19:15.441056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.441056+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:27.117522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053896129999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":1772,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796581+0000","last_change":"2024-09-16T05:20:23.797334+0000","last_active":"2024-09-16T05:20:23.796581+0000","last_peered":"2024-09-16T05:20:23.796581+0000","last_clean":"2024-09-16T05:20:23.796581+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:20:23.796581+0000","last_undegraded":"2024-09-16T05:20:23.796581+0000","last_fullsized":"2024-09-16T05:20:23.796581+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:19:28.221037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.221037+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:49.020047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014226099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.41","version":"178'50","reported_seq":648,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155222+0000","last_change":"2024-09-16T05:20:23.800056+0000","last_active":"2024-09-16T05:20:25.155222+0000","last_peered":"2024-09-16T05:20:25.155222+0000","last_clean":"2024-09-16T05:20:25.155222+0000","last_became_active":"2024-09-16T05:18:19.768547+0000","last_became_peered":"2024-09-16T05:18:19.768547+0000","last_unstale":"2024-09-16T05:20:25.155222+0000","last_undegraded":"2024-09-16T05:20:25.155222+0000","last_fullsized":"2024-09-16T05:20:25.155222+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:19:55.515448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.515448+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.732888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00042290300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.40","version":"205'33","reported_seq":345,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156704+0000","last_change":"2024-09-16T05:20:23.807332+0000","last_active":"2024-09-16T05:20:25.156704+0000","last_peered":"2024-09-16T05:20:25.156704+0000","last_clean":"2024-09-16T05:20:25.156704+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:20:25.156704+0000","last_undegraded":"2024-09-16T05:20:25.156704+0000","last_fullsized":"2024-09-16T05:20:25.156704+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:53.588594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.588594+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:12.149127+0000","scrub_duration":798,"objects_trimmed":0,"snaptrim_duration":0.00054597499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.3f","version":"666'419","reported_seq":3259,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.808630+0000","last_change":"2024-09-16T05:20:26.809168+0000","last_active":"2024-09-16T05:20:26.808630+0000","last_peered":"2024-09-16T05:20:26.808630+0000","last_clean":"2024-09-16T05:20:26.808630+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:20:26.808630+0000","last_undegraded":"2024-09-16T05:20:26.808630+0000","last_fullsized":"2024-09-16T05:20:26.808630+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'415","last_scrub_stamp":"2024-09-16T05:19:58.393481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:58.393481+0000","objects_scrubbed":2,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.473419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0057641130000000004,"stat_sum":{"num_bytes":5748357,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":82274,"num_write":47,"num_write_kb":13993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.3e","version":"625'487","reported_seq":3282,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.812160+0000","last_change":"2024-09-16T05:20:26.812437+0000","last_active":"2024-09-16T05:20:26.812160+0000","last_peered":"2024-09-16T05:20:26.812160+0000","last_clean":"2024-09-16T05:20:26.812160+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:20:26.812160+0000","last_undegraded":"2024-09-16T05:20:26.812160+0000","last_fullsized":"2024-09-16T05:20:26.812160+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"625'487","last_scrub_stamp":"2024-09-16T05:20:22.404334+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:20:22.404334+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.501331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001331675,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":26573,"num_write":41,"num_write_kb":23888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1718,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.797408+0000","last_change":"2024-09-16T05:20:23.797774+0000","last_active":"2024-09-16T05:20:23.797408+0000","last_peered":"2024-09-16T05:20:23.797408+0000","last_clean":"2024-09-16T05:20:23.797408+0000","last_became_active":"2024-09-16T05:19:35.784967+0000","last_became_peered":"2024-09-16T05:19:35.784967+0000","last_unstale":"2024-09-16T05:20:23.797408+0000","last_undegraded":"2024-09-16T05:20:23.797408+0000","last_fullsized":"2024-09-16T05:20:23.797408+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:19:36.333160+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.333160+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.555665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001630732,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.3c","version":"677'712","reported_seq":3702,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.807101+0000","last_change":"2024-09-16T05:20:26.807705+0000","last_active":"2024-09-16T05:20:26.807101+0000","last_peered":"2024-09-16T05:20:26.807101+0000","last_clean":"2024-09-16T05:20:26.807101+0000","last_became_active":"2024-09-16T05:19:35.800730+0000","last_became_peered":"2024-09-16T05:19:35.800730+0000","last_unstale":"2024-09-16T05:20:26.807101+0000","last_undegraded":"2024-09-16T05:20:26.807101+0000","last_fullsized":"2024-09-16T05:20:26.807101+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"624'648","last_scrub_stamp":"2024-09-16T05:19:34.258477+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:19:34.258477+0000","objects_scrubbed":6,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:52.675399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0045688839999999996,"stat_sum":{"num_bytes":13004770,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":161,"num_read_kb":208719,"num_write":78,"num_write_kb":29911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":6910647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.3b","version":"654'428","reported_seq":3242,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156907+0000","last_change":"2024-09-16T05:20:23.807772+0000","last_active":"2024-09-16T05:20:25.156907+0000","last_peered":"2024-09-16T05:20:25.156907+0000","last_clean":"2024-09-16T05:20:25.156907+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:20:25.156907+0000","last_undegraded":"2024-09-16T05:20:25.156907+0000","last_fullsized":"2024-09-16T05:20:25.156907+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'416","last_scrub_stamp":"2024-09-16T05:19:55.767476+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.767476+0000","objects_scrubbed":2,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.864203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075645699999999998,"stat_sum":{"num_bytes":8222944,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54616,"num_write":46,"num_write_kb":21574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.3a","version":"667'593","reported_seq":3368,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857904+0000","last_change":"2024-09-16T05:20:23.803488+0000","last_active":"2024-09-16T05:20:24.857904+0000","last_peered":"2024-09-16T05:20:24.857904+0000","last_clean":"2024-09-16T05:20:24.857904+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:20:24.857904+0000","last_undegraded":"2024-09-16T05:20:24.857904+0000","last_fullsized":"2024-09-16T05:20:24.857904+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"667'593","last_scrub_stamp":"2024-09-16T05:20:20.620494+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.620494+0000","objects_scrubbed":2,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.955507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070838100000000001,"stat_sum":{"num_bytes":3949810,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":53,"num_read_kb":46839,"num_write":60,"num_write_kb":24043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.39","version":"182'60","reported_seq":1888,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857923+0000","last_change":"2024-09-16T05:20:23.804556+0000","last_active":"2024-09-16T05:20:24.857923+0000","last_peered":"2024-09-16T05:20:24.857923+0000","last_clean":"2024-09-16T05:20:24.857923+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:20:24.857923+0000","last_undegraded":"2024-09-16T05:20:24.857923+0000","last_fullsized":"2024-09-16T05:20:24.857923+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:19:49.888753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.888753+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:14.187423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001803121,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.38","version":"181'57","reported_seq":1893,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.795860+0000","last_change":"2024-09-16T05:20:23.796180+0000","last_active":"2024-09-16T05:20:23.795860+0000","last_peered":"2024-09-16T05:20:23.795860+0000","last_clean":"2024-09-16T05:20:23.795860+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:20:23.795860+0000","last_undegraded":"2024-09-16T05:20:23.795860+0000","last_fullsized":"2024-09-16T05:20:23.795860+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:20:20.396645+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.396645+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:29.475306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057224100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.37","version":"0'0","reported_seq":747,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801788+0000","last_change":"2024-09-16T05:20:23.809915+0000","last_active":"2024-09-16T05:20:25.801788+0000","last_peered":"2024-09-16T05:20:25.801788+0000","last_clean":"2024-09-16T05:20:25.801788+0000","last_became_active":"2024-09-16T05:17:59.866037+0000","last_became_peered":"2024-09-16T05:17:59.866037+0000","last_unstale":"2024-09-16T05:20:25.801788+0000","last_undegraded":"2024-09-16T05:20:25.801788+0000","last_fullsized":"2024-09-16T05:20:25.801788+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:31.567784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:19:31.567784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:55.722805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017300499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":269,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.812501+0000","last_change":"2024-09-16T05:20:26.812792+0000","last_active":"2024-09-16T05:20:26.812501+0000","last_peered":"2024-09-16T05:20:26.812501+0000","last_clean":"2024-09-16T05:20:26.812501+0000","last_became_active":"2024-09-16T05:19:35.795592+0000","last_became_peered":"2024-09-16T05:19:35.795592+0000","last_unstale":"2024-09-16T05:20:26.812501+0000","last_undegraded":"2024-09-16T05:20:26.812501+0000","last_fullsized":"2024-09-16T05:20:26.812501+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:20:02.294024+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.294024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.400789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00140982,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.35","version":"677'745","reported_seq":3649,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.817784+0000","last_change":"2024-09-16T05:20:26.818129+0000","last_active":"2024-09-16T05:20:26.817784+0000","last_peered":"2024-09-16T05:20:26.817784+0000","last_clean":"2024-09-16T05:20:26.817784+0000","last_became_active":"2024-09-16T05:19:35.802234+0000","last_became_peered":"2024-09-16T05:19:35.802234+0000","last_unstale":"2024-09-16T05:20:26.817784+0000","last_undegraded":"2024-09-16T05:20:26.817784+0000","last_fullsized":"2024-09-16T05:20:26.817784+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'675","last_scrub_stamp":"2024-09-16T05:19:11.894332+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.894332+0000","objects_scrubbed":8,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0067402750000000004,"stat_sum":{"num_bytes":12263488,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":134,"num_read_kb":130415,"num_write":87,"num_write_kb":40632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3771362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.34","version":"674'591","reported_seq":3010,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.157922+0000","last_change":"2024-09-16T05:20:23.809853+0000","last_active":"2024-09-16T05:20:25.157922+0000","last_peered":"2024-09-16T05:20:25.157922+0000","last_clean":"2024-09-16T05:20:25.157922+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:20:25.157922+0000","last_undegraded":"2024-09-16T05:20:25.157922+0000","last_fullsized":"2024-09-16T05:20:25.157922+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'482","last_scrub_stamp":"2024-09-16T05:19:12.637489+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.637489+0000","objects_scrubbed":8,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:26.960724+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0028600679999999999,"stat_sum":{"num_bytes":18684074,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":154,"num_read_kb":173311,"num_write":101,"num_write_kb":39821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.33","version":"181'15","reported_seq":1986,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.808557+0000","last_change":"2024-09-16T05:20:26.808867+0000","last_active":"2024-09-16T05:20:26.808557+0000","last_peered":"2024-09-16T05:20:26.808557+0000","last_clean":"2024-09-16T05:20:26.808557+0000","last_became_active":"2024-09-16T05:19:35.789794+0000","last_became_peered":"2024-09-16T05:19:35.789794+0000","last_unstale":"2024-09-16T05:20:26.808557+0000","last_undegraded":"2024-09-16T05:20:26.808557+0000","last_fullsized":"2024-09-16T05:20:26.808557+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:16.451459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:16.451459+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.511342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056268500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.32","version":"674'229","reported_seq":2207,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.817057+0000","last_change":"2024-09-16T05:20:26.817057+0000","last_active":"2024-09-16T05:20:26.817057+0000","last_peered":"2024-09-16T05:20:26.817057+0000","last_clean":"2024-09-16T05:20:26.817057+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:20:26.817057+0000","last_undegraded":"2024-09-16T05:20:26.817057+0000","last_fullsized":"2024-09-16T05:20:26.817057+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"671'227","last_scrub_stamp":"2024-09-16T05:20:23.405472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.405472+0000","objects_scrubbed":2,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.355290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053593729999999997,"stat_sum":{"num_bytes":6202179,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":101970,"num_write":44,"num_write_kb":20642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.31","version":"666'280","reported_seq":2829,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155452+0000","last_change":"2024-09-16T05:20:23.800301+0000","last_active":"2024-09-16T05:20:25.155452+0000","last_peered":"2024-09-16T05:20:25.155452+0000","last_clean":"2024-09-16T05:20:25.155452+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:20:25.155452+0000","last_undegraded":"2024-09-16T05:20:25.155452+0000","last_fullsized":"2024-09-16T05:20:25.155452+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'268","last_scrub_stamp":"2024-09-16T05:20:03.547514+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.547514+0000","objects_scrubbed":2,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:13.452601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094748100000000002,"stat_sum":{"num_bytes":3680626,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":116,"num_read_kb":119771,"num_write":39,"num_write_kb":19090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.30","version":"182'16","reported_seq":735,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801198+0000","last_change":"2024-09-16T05:20:23.807538+0000","last_active":"2024-09-16T05:20:25.801198+0000","last_peered":"2024-09-16T05:20:25.801198+0000","last_clean":"2024-09-16T05:20:25.801198+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:20:25.801198+0000","last_undegraded":"2024-09-16T05:20:25.801198+0000","last_fullsized":"2024-09-16T05:20:25.801198+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:20:16.443442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.443442+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.543953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023386,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":984,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.157057+0000","last_change":"2024-09-16T05:20:18.091859+0000","last_active":"2024-09-16T05:20:25.157057+0000","last_peered":"2024-09-16T05:20:25.157057+0000","last_clean":"2024-09-16T05:20:25.157057+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:20:25.157057+0000","last_undegraded":"2024-09-16T05:20:25.157057+0000","last_fullsized":"2024-09-16T05:20:25.157057+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.091747+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.091747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.641366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":2540,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.809441+0000","last_change":"2024-09-16T05:20:26.810224+0000","last_active":"2024-09-16T05:20:26.809441+0000","last_peered":"2024-09-16T05:20:26.809441+0000","last_clean":"2024-09-16T05:20:26.809441+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:20:26.809441+0000","last_undegraded":"2024-09-16T05:20:26.809441+0000","last_fullsized":"2024-09-16T05:20:26.809441+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:20:03.076376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.076376+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:14.349215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095940499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1153,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.810665+0000","last_change":"2024-09-16T05:19:36.252852+0000","last_active":"2024-09-16T05:20:26.810665+0000","last_peered":"2024-09-16T05:20:26.810665+0000","last_clean":"2024-09-16T05:20:26.810665+0000","last_became_active":"2024-09-16T05:19:35.788516+0000","last_became_peered":"2024-09-16T05:19:35.788516+0000","last_unstale":"2024-09-16T05:20:26.810665+0000","last_undegraded":"2024-09-16T05:20:26.810665+0000","last_fullsized":"2024-09-16T05:20:26.810665+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:36.252730+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.252730+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.441323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":2444,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796195+0000","last_change":"2024-09-16T05:20:23.796445+0000","last_active":"2024-09-16T05:20:23.796195+0000","last_peered":"2024-09-16T05:20:23.796195+0000","last_clean":"2024-09-16T05:20:23.796195+0000","last_became_active":"2024-09-16T05:19:35.791894+0000","last_became_peered":"2024-09-16T05:19:35.791894+0000","last_unstale":"2024-09-16T05:20:23.796195+0000","last_undegraded":"2024-09-16T05:20:23.796195+0000","last_fullsized":"2024-09-16T05:20:23.796195+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:11.306770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.306770+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072383600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.c","version":"35'139","reported_seq":939,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.808845+0000","last_change":"2024-09-16T05:20:18.363196+0000","last_active":"2024-09-16T05:20:27.808845+0000","last_peered":"2024-09-16T05:20:27.808845+0000","last_clean":"2024-09-16T05:20:27.808845+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:20:27.808845+0000","last_undegraded":"2024-09-16T05:20:27.808845+0000","last_fullsized":"2024-09-16T05:20:27.808845+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.363111+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.363111+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:36.359138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"666'405","reported_seq":3231,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796879+0000","last_change":"2024-09-16T05:20:23.797106+0000","last_active":"2024-09-16T05:20:23.796879+0000","last_peered":"2024-09-16T05:20:23.796879+0000","last_clean":"2024-09-16T05:20:23.796879+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:20:23.796879+0000","last_undegraded":"2024-09-16T05:20:23.796879+0000","last_fullsized":"2024-09-16T05:20:23.796879+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'384","last_scrub_stamp":"2024-09-16T05:19:15.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.295498+0000","objects_scrubbed":6,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011026530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":57662,"num_write":37,"num_write_kb":14770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.d","version":"35'139","reported_seq":1093,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.803450+0000","last_change":"2024-09-16T05:19:12.454525+0000","last_active":"2024-09-16T05:20:26.803450+0000","last_peered":"2024-09-16T05:20:26.803450+0000","last_clean":"2024-09-16T05:20:26.803450+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:20:26.803450+0000","last_undegraded":"2024-09-16T05:20:26.803450+0000","last_fullsized":"2024-09-16T05:20:26.803450+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:12.454446+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.454446+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.180360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":2959,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.829120+0000","last_change":"2024-09-16T05:20:26.829358+0000","last_active":"2024-09-16T05:20:26.829120+0000","last_peered":"2024-09-16T05:20:26.829120+0000","last_clean":"2024-09-16T05:20:26.829120+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:20:26.829120+0000","last_undegraded":"2024-09-16T05:20:26.829120+0000","last_fullsized":"2024-09-16T05:20:26.829120+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:11.209692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.209692+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00146159,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":1089,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.794230+0000","last_change":"2024-09-16T05:19:59.197776+0000","last_active":"2024-09-16T05:20:25.794230+0000","last_peered":"2024-09-16T05:20:25.794230+0000","last_clean":"2024-09-16T05:20:25.794230+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:20:25.794230+0000","last_undegraded":"2024-09-16T05:20:25.794230+0000","last_fullsized":"2024-09-16T05:20:25.794230+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:59.197664+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:59.197664+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.493240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"666'602","reported_seq":3432,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858239+0000","last_change":"2024-09-16T05:20:23.805426+0000","last_active":"2024-09-16T05:20:24.858239+0000","last_peered":"2024-09-16T05:20:24.858239+0000","last_clean":"2024-09-16T05:20:24.858239+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:20:24.858239+0000","last_undegraded":"2024-09-16T05:20:24.858239+0000","last_fullsized":"2024-09-16T05:20:24.858239+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"617'576","last_scrub_stamp":"2024-09-16T05:19:26.973559+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:19:26.973559+0000","objects_scrubbed":4,"log_size":602,"log_dups_size":0,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:48.652195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019755530000000001,"stat_sum":{"num_bytes":9125264,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":52779,"num_write":49,"num_write_kb":25445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.f","version":"35'139","reported_seq":1072,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155737+0000","last_change":"2024-09-16T05:19:29.650332+0000","last_active":"2024-09-16T05:20:25.155737+0000","last_peered":"2024-09-16T05:20:25.155737+0000","last_clean":"2024-09-16T05:20:25.155737+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:20:25.155737+0000","last_undegraded":"2024-09-16T05:20:25.155737+0000","last_fullsized":"2024-09-16T05:20:25.155737+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:29.650209+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.650209+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:34.196552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3089,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.816430+0000","last_change":"2024-09-16T05:20:26.816642+0000","last_active":"2024-09-16T05:20:26.816430+0000","last_peered":"2024-09-16T05:20:26.816430+0000","last_clean":"2024-09-16T05:20:26.816430+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:20:26.816430+0000","last_undegraded":"2024-09-16T05:20:26.816430+0000","last_fullsized":"2024-09-16T05:20:26.816430+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:19:10.225027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.225027+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:36.770636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0049958149999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1616,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.810872+0000","last_change":"2024-09-16T05:19:33.235762+0000","last_active":"2024-09-16T05:20:26.810872+0000","last_peered":"2024-09-16T05:20:26.810872+0000","last_clean":"2024-09-16T05:20:26.810872+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:20:26.810872+0000","last_undegraded":"2024-09-16T05:20:26.810872+0000","last_fullsized":"2024-09-16T05:20:26.810872+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:33.235681+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:19:33.235681+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.500872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"666'325","reported_seq":3156,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.802947+0000","last_change":"2024-09-16T05:20:26.803219+0000","last_active":"2024-09-16T05:20:26.802947+0000","last_peered":"2024-09-16T05:20:26.802947+0000","last_clean":"2024-09-16T05:20:26.802947+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:20:26.802947+0000","last_undegraded":"2024-09-16T05:20:26.802947+0000","last_fullsized":"2024-09-16T05:20:26.802947+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'325","last_scrub_stamp":"2024-09-16T05:20:23.462954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.462954+0000","objects_scrubbed":2,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.623140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052462199999999998,"stat_sum":{"num_bytes":4366553,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":35425,"num_write":51,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.3","version":"671'209","reported_seq":3044,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.157073+0000","last_change":"2024-09-16T05:20:23.808993+0000","last_active":"2024-09-16T05:20:25.157073+0000","last_peered":"2024-09-16T05:20:25.157073+0000","last_clean":"2024-09-16T05:20:25.157073+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:20:25.157073+0000","last_undegraded":"2024-09-16T05:20:25.157073+0000","last_fullsized":"2024-09-16T05:20:25.157073+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'205","last_scrub_stamp":"2024-09-16T05:20:19.137710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:19.137710+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.573760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014350680000000001,"stat_sum":{"num_bytes":3462440,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":111576,"num_write":43,"num_write_kb":18525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.0","version":"35'139","reported_seq":627,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.810695+0000","last_change":"2024-09-16T05:20:00.246338+0000","last_active":"2024-09-16T05:20:26.810695+0000","last_peered":"2024-09-16T05:20:26.810695+0000","last_clean":"2024-09-16T05:20:26.810695+0000","last_became_active":"2024-09-16T05:18:19.768392+0000","last_became_peered":"2024-09-16T05:18:19.768392+0000","last_unstale":"2024-09-16T05:20:26.810695+0000","last_undegraded":"2024-09-16T05:20:26.810695+0000","last_fullsized":"2024-09-16T05:20:26.810695+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:00.246242+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.246242+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.738053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":2824,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.157299+0000","last_change":"2024-09-16T05:20:23.807602+0000","last_active":"2024-09-16T05:20:25.157299+0000","last_peered":"2024-09-16T05:20:25.157299+0000","last_clean":"2024-09-16T05:20:25.157299+0000","last_became_active":"2024-09-16T05:19:35.791085+0000","last_became_peered":"2024-09-16T05:19:35.791085+0000","last_unstale":"2024-09-16T05:20:25.157299+0000","last_undegraded":"2024-09-16T05:20:25.157299+0000","last_fullsized":"2024-09-16T05:20:25.157299+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:02.847979+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.847979+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.318132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066067600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.1","version":"35'139","reported_seq":1173,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801101+0000","last_change":"2024-09-16T05:19:30.558195+0000","last_active":"2024-09-16T05:20:25.801101+0000","last_peered":"2024-09-16T05:20:25.801101+0000","last_clean":"2024-09-16T05:20:25.801101+0000","last_became_active":"2024-09-16T05:18:02.888512+0000","last_became_peered":"2024-09-16T05:18:02.888512+0000","last_unstale":"2024-09-16T05:20:25.801101+0000","last_undegraded":"2024-09-16T05:20:25.801101+0000","last_fullsized":"2024-09-16T05:20:25.801101+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.558069+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.558069+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.688000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1605,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.802963+0000","last_change":"2024-09-16T05:19:56.353335+0000","last_active":"2024-09-16T05:20:26.802963+0000","last_peered":"2024-09-16T05:20:26.802963+0000","last_clean":"2024-09-16T05:20:26.802963+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:20:26.802963+0000","last_undegraded":"2024-09-16T05:20:26.802963+0000","last_fullsized":"2024-09-16T05:20:26.802963+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:19:56.353243+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.079495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":2776,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.827723+0000","last_change":"2024-09-16T05:20:26.828079+0000","last_active":"2024-09-16T05:20:26.827723+0000","last_peered":"2024-09-16T05:20:26.827723+0000","last_clean":"2024-09-16T05:20:26.827723+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:20:26.827723+0000","last_undegraded":"2024-09-16T05:20:26.827723+0000","last_fullsized":"2024-09-16T05:20:26.827723+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:25.536951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:25.536951+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.986576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065442300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1612,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.808881+0000","last_change":"2024-09-16T05:19:10.336320+0000","last_active":"2024-09-16T05:20:27.808881+0000","last_peered":"2024-09-16T05:20:27.808881+0000","last_clean":"2024-09-16T05:20:27.808881+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:20:27.808881+0000","last_undegraded":"2024-09-16T05:20:27.808881+0000","last_fullsized":"2024-09-16T05:20:27.808881+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:10.336203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.336203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.544788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":2856,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.807790+0000","last_change":"2024-09-16T05:20:26.808333+0000","last_active":"2024-09-16T05:20:26.807790+0000","last_peered":"2024-09-16T05:20:26.807790+0000","last_clean":"2024-09-16T05:20:26.807790+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:20:26.807790+0000","last_undegraded":"2024-09-16T05:20:26.807790+0000","last_fullsized":"2024-09-16T05:20:26.807790+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:51.365804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.365804+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:59.310751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053471650000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":2059,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858300+0000","last_change":"2024-09-16T05:20:23.808288+0000","last_active":"2024-09-16T05:20:24.858300+0000","last_peered":"2024-09-16T05:20:24.858300+0000","last_clean":"2024-09-16T05:20:24.858300+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:20:24.858300+0000","last_undegraded":"2024-09-16T05:20:24.858300+0000","last_fullsized":"2024-09-16T05:20:24.858300+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:50.852123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.852123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:00.103629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038802469999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.4","version":"35'139","reported_seq":1571,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.812032+0000","last_change":"2024-09-16T05:19:53.322398+0000","last_active":"2024-09-16T05:20:26.812032+0000","last_peered":"2024-09-16T05:20:26.812032+0000","last_clean":"2024-09-16T05:20:26.812032+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:20:26.812032+0000","last_undegraded":"2024-09-16T05:20:26.812032+0000","last_fullsized":"2024-09-16T05:20:26.812032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:53.322303+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.322303+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:11.892229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1250,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.829056+0000","last_change":"2024-09-16T05:20:26.829291+0000","last_active":"2024-09-16T05:20:26.829056+0000","last_peered":"2024-09-16T05:20:26.829056+0000","last_clean":"2024-09-16T05:20:26.829056+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:20:26.829056+0000","last_undegraded":"2024-09-16T05:20:26.829056+0000","last_fullsized":"2024-09-16T05:20:26.829056+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:53.382868+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:18:27.279121+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.382868+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:13.528470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014349199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":878,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801846+0000","last_change":"2024-09-16T05:19:13.533926+0000","last_active":"2024-09-16T05:20:25.801846+0000","last_peered":"2024-09-16T05:20:25.801846+0000","last_clean":"2024-09-16T05:20:25.801846+0000","last_became_active":"2024-09-16T05:18:02.897275+0000","last_became_peered":"2024-09-16T05:18:02.897275+0000","last_unstale":"2024-09-16T05:20:25.801846+0000","last_undegraded":"2024-09-16T05:20:25.801846+0000","last_fullsized":"2024-09-16T05:20:25.801846+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:19:13.533798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"674'581","reported_seq":3404,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.799103+0000","last_change":"2024-09-16T05:20:23.799472+0000","last_active":"2024-09-16T05:20:23.799103+0000","last_peered":"2024-09-16T05:20:23.799103+0000","last_clean":"2024-09-16T05:20:23.799103+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:20:23.799103+0000","last_undegraded":"2024-09-16T05:20:23.799103+0000","last_fullsized":"2024-09-16T05:20:23.799103+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'568","last_scrub_stamp":"2024-09-16T05:19:54.247232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.247232+0000","objects_scrubbed":3,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:05.980654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032423719999999999,"stat_sum":{"num_bytes":7659108,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":107857,"num_write":39,"num_write_kb":12889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.6","version":"35'139","reported_seq":822,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801359+0000","last_change":"2024-09-16T05:19:37.639763+0000","last_active":"2024-09-16T05:20:25.801359+0000","last_peered":"2024-09-16T05:20:25.801359+0000","last_clean":"2024-09-16T05:20:25.801359+0000","last_became_active":"2024-09-16T05:18:02.886933+0000","last_became_peered":"2024-09-16T05:18:02.886933+0000","last_unstale":"2024-09-16T05:20:25.801359+0000","last_undegraded":"2024-09-16T05:20:25.801359+0000","last_fullsized":"2024-09-16T05:20:25.801359+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:37.639669+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:37.639669+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:42.374370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":696,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155832+0000","last_change":"2024-09-16T05:20:23.800424+0000","last_active":"2024-09-16T05:20:25.155832+0000","last_peered":"2024-09-16T05:20:25.155832+0000","last_clean":"2024-09-16T05:20:25.155832+0000","last_became_active":"2024-09-16T05:19:35.793136+0000","last_became_peered":"2024-09-16T05:19:35.793136+0000","last_unstale":"2024-09-16T05:20:25.155832+0000","last_undegraded":"2024-09-16T05:20:25.155832+0000","last_fullsized":"2024-09-16T05:20:25.155832+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:36.435156+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.435156+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:42.430535+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00064612000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.7","version":"35'139","reported_seq":858,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801431+0000","last_change":"2024-09-16T05:19:15.510805+0000","last_active":"2024-09-16T05:20:25.801431+0000","last_peered":"2024-09-16T05:20:25.801431+0000","last_clean":"2024-09-16T05:20:25.801431+0000","last_became_active":"2024-09-16T05:18:03.588064+0000","last_became_peered":"2024-09-16T05:18:03.588064+0000","last_unstale":"2024-09-16T05:20:25.801431+0000","last_undegraded":"2024-09-16T05:20:25.801431+0000","last_fullsized":"2024-09-16T05:20:25.801431+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:15.510717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.510717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:27.556487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":2991,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796521+0000","last_change":"2024-09-16T05:20:23.796781+0000","last_active":"2024-09-16T05:20:23.796521+0000","last_peered":"2024-09-16T05:20:23.796521+0000","last_clean":"2024-09-16T05:20:23.796521+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:20:23.796521+0000","last_undegraded":"2024-09-16T05:20:23.796521+0000","last_fullsized":"2024-09-16T05:20:23.796521+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:19:29.221207+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.221207+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:42.084933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089577700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.8","version":"35'139","reported_seq":1603,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156926+0000","last_change":"2024-09-16T05:19:30.613103+0000","last_active":"2024-09-16T05:20:25.156926+0000","last_peered":"2024-09-16T05:20:25.156926+0000","last_clean":"2024-09-16T05:20:25.156926+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:20:25.156926+0000","last_undegraded":"2024-09-16T05:20:25.156926+0000","last_fullsized":"2024-09-16T05:20:25.156926+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.613024+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.613024+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.741797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"677'618","reported_seq":3477,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.819050+0000","last_change":"2024-09-16T05:20:26.819264+0000","last_active":"2024-09-16T05:20:26.819050+0000","last_peered":"2024-09-16T05:20:26.819050+0000","last_clean":"2024-09-16T05:20:26.819050+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:20:26.819050+0000","last_undegraded":"2024-09-16T05:20:26.819050+0000","last_fullsized":"2024-09-16T05:20:26.819050+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"653'601","last_scrub_stamp":"2024-09-16T05:20:03.311152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.311152+0000","objects_scrubbed":3,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:07.029849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0073471539999999998,"stat_sum":{"num_bytes":6944569,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":127393,"num_write":54,"num_write_kb":33730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1603,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.802598+0000","last_change":"2024-09-16T05:20:20.802104+0000","last_active":"2024-09-16T05:20:26.802598+0000","last_peered":"2024-09-16T05:20:26.802598+0000","last_clean":"2024-09-16T05:20:26.802598+0000","last_became_active":"2024-09-16T05:20:20.801737+0000","last_became_peered":"2024-09-16T05:20:20.801737+0000","last_unstale":"2024-09-16T05:20:26.802598+0000","last_undegraded":"2024-09-16T05:20:26.802598+0000","last_fullsized":"2024-09-16T05:20:26.802598+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:02.384248+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.384248+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.409130+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1729,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.794098+0000","last_change":"2024-09-16T05:20:23.795538+0000","last_active":"2024-09-16T05:20:25.794098+0000","last_peered":"2024-09-16T05:20:25.794098+0000","last_clean":"2024-09-16T05:20:25.794098+0000","last_became_active":"2024-09-16T05:19:35.786308+0000","last_became_peered":"2024-09-16T05:19:35.786308+0000","last_unstale":"2024-09-16T05:20:25.794098+0000","last_undegraded":"2024-09-16T05:20:25.794098+0000","last_fullsized":"2024-09-16T05:20:25.794098+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:19:31.185579+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:31.185579+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:52.530015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000428066,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1598,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857860+0000","last_change":"2024-09-16T05:19:32.947634+0000","last_active":"2024-09-16T05:20:24.857860+0000","last_peered":"2024-09-16T05:20:24.857860+0000","last_clean":"2024-09-16T05:20:24.857860+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:20:24.857860+0000","last_undegraded":"2024-09-16T05:20:24.857860+0000","last_fullsized":"2024-09-16T05:20:24.857860+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:32.947528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.947528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.139142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1088,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155283+0000","last_change":"2024-09-16T05:19:49.535280+0000","last_active":"2024-09-16T05:20:25.155283+0000","last_peered":"2024-09-16T05:20:25.155283+0000","last_clean":"2024-09-16T05:20:25.155283+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:20:25.155283+0000","last_undegraded":"2024-09-16T05:20:25.155283+0000","last_fullsized":"2024-09-16T05:20:25.155283+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:49.535206+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.535206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:17.718180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":735,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801201+0000","last_change":"2024-09-16T05:20:24.333496+0000","last_active":"2024-09-16T05:20:25.801201+0000","last_peered":"2024-09-16T05:20:25.801201+0000","last_clean":"2024-09-16T05:20:25.801201+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:20:25.801201+0000","last_undegraded":"2024-09-16T05:20:25.801201+0000","last_fullsized":"2024-09-16T05:20:25.801201+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:24.333369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:24.333369+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:32.117168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001134688,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1603,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156679+0000","last_change":"2024-09-16T05:19:48.737475+0000","last_active":"2024-09-16T05:20:25.156679+0000","last_peered":"2024-09-16T05:20:25.156679+0000","last_clean":"2024-09-16T05:20:25.156679+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:20:25.156679+0000","last_undegraded":"2024-09-16T05:20:25.156679+0000","last_fullsized":"2024-09-16T05:20:25.156679+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:48.737348+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.737348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:57.195160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":2440,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858123+0000","last_change":"2024-09-16T05:20:23.806171+0000","last_active":"2024-09-16T05:20:24.858123+0000","last_peered":"2024-09-16T05:20:24.858123+0000","last_clean":"2024-09-16T05:20:24.858123+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:20:24.858123+0000","last_undegraded":"2024-09-16T05:20:24.858123+0000","last_fullsized":"2024-09-16T05:20:24.858123+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:36.009233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.009233+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:38.540686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026423110000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.14","version":"433'289","reported_seq":2712,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858147+0000","last_change":"2024-09-16T05:20:23.804360+0000","last_active":"2024-09-16T05:20:24.858147+0000","last_peered":"2024-09-16T05:20:24.858147+0000","last_clean":"2024-09-16T05:20:24.858147+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:20:24.858147+0000","last_undegraded":"2024-09-16T05:20:24.858147+0000","last_fullsized":"2024-09-16T05:20:24.858147+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:19:12.999550+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.999550+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.277329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013382859999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.15","version":"674'633","reported_seq":3529,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858188+0000","last_change":"2024-09-16T05:20:23.807266+0000","last_active":"2024-09-16T05:20:24.858188+0000","last_peered":"2024-09-16T05:20:24.858188+0000","last_clean":"2024-09-16T05:20:24.858188+0000","last_became_active":"2024-09-16T05:19:35.801137+0000","last_became_peered":"2024-09-16T05:19:35.801137+0000","last_unstale":"2024-09-16T05:20:24.858188+0000","last_undegraded":"2024-09-16T05:20:24.858188+0000","last_fullsized":"2024-09-16T05:20:24.858188+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"617'599","last_scrub_stamp":"2024-09-16T05:19:28.020270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.020270+0000","objects_scrubbed":4,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.489114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038891009999999998,"stat_sum":{"num_bytes":3964937,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":5506324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.16","version":"666'296","reported_seq":2757,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156576+0000","last_change":"2024-09-16T05:20:23.808837+0000","last_active":"2024-09-16T05:20:25.156576+0000","last_peered":"2024-09-16T05:20:25.156576+0000","last_clean":"2024-09-16T05:20:25.156576+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:20:25.156576+0000","last_undegraded":"2024-09-16T05:20:25.156576+0000","last_fullsized":"2024-09-16T05:20:25.156576+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"665'294","last_scrub_stamp":"2024-09-16T05:20:15.010850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.010850+0000","objects_scrubbed":4,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.414078+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0015138899999999999,"stat_sum":{"num_bytes":7403700,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":57070,"num_write":50,"num_write_kb":16677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.14","version":"35'139","reported_seq":927,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.827479+0000","last_change":"2024-09-16T05:18:46.335095+0000","last_active":"2024-09-16T05:20:26.827479+0000","last_peered":"2024-09-16T05:20:26.827479+0000","last_clean":"2024-09-16T05:20:26.827479+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:20:26.827479+0000","last_undegraded":"2024-09-16T05:20:26.827479+0000","last_fullsized":"2024-09-16T05:20:26.827479+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:46.335030+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.335030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:31.492721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":485,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801461+0000","last_change":"2024-09-16T05:20:23.809199+0000","last_active":"2024-09-16T05:20:25.801461+0000","last_peered":"2024-09-16T05:20:25.801461+0000","last_clean":"2024-09-16T05:20:25.801461+0000","last_became_active":"2024-09-16T05:19:35.784084+0000","last_became_peered":"2024-09-16T05:19:35.784084+0000","last_unstale":"2024-09-16T05:20:25.801461+0000","last_undegraded":"2024-09-16T05:20:25.801461+0000","last_fullsized":"2024-09-16T05:20:25.801461+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:20:20.393358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.393358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:24.632271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00229188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":2822,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.857937+0000","last_change":"2024-09-16T05:20:23.807570+0000","last_active":"2024-09-16T05:20:24.857937+0000","last_peered":"2024-09-16T05:20:24.857937+0000","last_clean":"2024-09-16T05:20:24.857937+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:20:24.857937+0000","last_undegraded":"2024-09-16T05:20:24.857937+0000","last_fullsized":"2024-09-16T05:20:24.857937+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:19:47.951135+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:19:47.951135+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:52.083499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003790168,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.19","version":"182'60","reported_seq":2454,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.809214+0000","last_change":"2024-09-16T05:20:27.809411+0000","last_active":"2024-09-16T05:20:27.809214+0000","last_peered":"2024-09-16T05:20:27.809214+0000","last_clean":"2024-09-16T05:20:27.809214+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:20:27.809214+0000","last_undegraded":"2024-09-16T05:20:27.809214+0000","last_fullsized":"2024-09-16T05:20:27.809214+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:20:17.403045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.403045+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.218144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037962800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.1a","version":"674'1001","reported_seq":3778,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.885939+0000","last_change":"2024-09-16T05:20:23.804002+0000","last_active":"2024-09-16T05:20:23.885939+0000","last_peered":"2024-09-16T05:20:23.885939+0000","last_clean":"2024-09-16T05:20:23.885939+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:20:23.885939+0000","last_undegraded":"2024-09-16T05:20:23.885939+0000","last_fullsized":"2024-09-16T05:20:23.885939+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"653'957","last_scrub_stamp":"2024-09-16T05:20:03.383437+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.383437+0000","objects_scrubbed":11,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.864097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0074082899999999997,"stat_sum":{"num_bytes":24006521,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":157,"num_read_kb":131937,"num_write":165,"num_write_kb":60609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.1b","version":"677'630","reported_seq":3483,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.816086+0000","last_change":"2024-09-16T05:20:26.816353+0000","last_active":"2024-09-16T05:20:26.816086+0000","last_peered":"2024-09-16T05:20:26.816086+0000","last_clean":"2024-09-16T05:20:26.816086+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:20:26.816086+0000","last_undegraded":"2024-09-16T05:20:26.816086+0000","last_fullsized":"2024-09-16T05:20:26.816086+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"666'612","last_scrub_stamp":"2024-09-16T05:20:16.258329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.258329+0000","objects_scrubbed":4,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.997193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053052799999999999,"stat_sum":{"num_bytes":11564336,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":94,"num_read_kb":89721,"num_write":102,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.1c","version":"664'452","reported_seq":2974,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.802009+0000","last_change":"2024-09-16T05:20:23.808221+0000","last_active":"2024-09-16T05:20:25.802009+0000","last_peered":"2024-09-16T05:20:25.802009+0000","last_clean":"2024-09-16T05:20:25.802009+0000","last_became_active":"2024-09-16T05:18:02.890972+0000","last_became_peered":"2024-09-16T05:18:02.890972+0000","last_unstale":"2024-09-16T05:20:25.802009+0000","last_undegraded":"2024-09-16T05:20:25.802009+0000","last_fullsized":"2024-09-16T05:20:25.802009+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"620'444","last_scrub_stamp":"2024-09-16T05:19:51.567043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.567043+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:20.840931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014105910000000001,"stat_sum":{"num_bytes":3294370,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":10803,"num_write":28,"num_write_kb":13180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.1d","version":"666'1182","reported_seq":4238,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801937+0000","last_change":"2024-09-16T05:20:23.809322+0000","last_active":"2024-09-16T05:20:25.801937+0000","last_peered":"2024-09-16T05:20:25.801937+0000","last_clean":"2024-09-16T05:20:25.801937+0000","last_became_active":"2024-09-16T05:18:03.891103+0000","last_became_peered":"2024-09-16T05:18:03.891103+0000","last_unstale":"2024-09-16T05:20:25.801937+0000","last_undegraded":"2024-09-16T05:20:25.801937+0000","last_fullsized":"2024-09-16T05:20:25.801937+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"624'1113","last_scrub_stamp":"2024-09-16T05:19:35.593327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:35.593327+0000","objects_scrubbed":9,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:53.246158+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0023436300000000002,"stat_sum":{"num_bytes":14250762,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":246,"num_read_kb":196095,"num_write":229,"num_write_kb":73447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7155994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.1e","version":"677'468","reported_seq":3339,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.809536+0000","last_change":"2024-09-16T05:20:27.809720+0000","last_active":"2024-09-16T05:20:27.809536+0000","last_peered":"2024-09-16T05:20:27.809536+0000","last_clean":"2024-09-16T05:20:27.809536+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:20:27.809536+0000","last_undegraded":"2024-09-16T05:20:27.809536+0000","last_fullsized":"2024-09-16T05:20:27.809536+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"671'464","last_scrub_stamp":"2024-09-16T05:20:21.358700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:21.358700+0000","objects_scrubbed":3,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.219910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044219300000000001,"stat_sum":{"num_bytes":3007609,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":71847,"num_write":25,"num_write_kb":11620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":3047,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155527+0000","last_change":"2024-09-16T05:20:23.800774+0000","last_active":"2024-09-16T05:20:25.155527+0000","last_peered":"2024-09-16T05:20:25.155527+0000","last_clean":"2024-09-16T05:20:25.155527+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:20:25.155527+0000","last_undegraded":"2024-09-16T05:20:25.155527+0000","last_fullsized":"2024-09-16T05:20:25.155527+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:20:17.544083+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.544083+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:36.068162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009795540000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.20","version":"677'307","reported_seq":3167,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.806422+0000","last_change":"2024-09-16T05:20:26.807410+0000","last_active":"2024-09-16T05:20:26.806422+0000","last_peered":"2024-09-16T05:20:26.806422+0000","last_clean":"2024-09-16T05:20:26.806422+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:20:26.806422+0000","last_undegraded":"2024-09-16T05:20:26.806422+0000","last_fullsized":"2024-09-16T05:20:26.806422+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'290","last_scrub_stamp":"2024-09-16T05:20:03.351330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.351330+0000","objects_scrubbed":3,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.928969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0046326850000000001,"stat_sum":{"num_bytes":6345760,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":144671,"num_write":58,"num_write_kb":21452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":2775,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828162+0000","last_change":"2024-09-16T05:20:26.828400+0000","last_active":"2024-09-16T05:20:26.828162+0000","last_peered":"2024-09-16T05:20:26.828162+0000","last_clean":"2024-09-16T05:20:26.828162+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:20:26.828162+0000","last_undegraded":"2024-09-16T05:20:26.828162+0000","last_fullsized":"2024-09-16T05:20:26.828162+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:02.565969+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.565969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:10.707965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093526500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":735,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801231+0000","last_change":"2024-09-16T05:20:23.809600+0000","last_active":"2024-09-16T05:20:25.801231+0000","last_peered":"2024-09-16T05:20:25.801231+0000","last_clean":"2024-09-16T05:20:25.801231+0000","last_became_active":"2024-09-16T05:17:59.866047+0000","last_became_peered":"2024-09-16T05:17:59.866047+0000","last_unstale":"2024-09-16T05:20:25.801231+0000","last_undegraded":"2024-09-16T05:20:25.801231+0000","last_fullsized":"2024-09-16T05:20:25.801231+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:15.452705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.452705+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.285695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010043820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":2327,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.809309+0000","last_change":"2024-09-16T05:20:27.809613+0000","last_active":"2024-09-16T05:20:27.809309+0000","last_peered":"2024-09-16T05:20:27.809309+0000","last_clean":"2024-09-16T05:20:27.809309+0000","last_became_active":"2024-09-16T05:20:20.801064+0000","last_became_peered":"2024-09-16T05:20:20.801064+0000","last_unstale":"2024-09-16T05:20:27.809309+0000","last_undegraded":"2024-09-16T05:20:27.809309+0000","last_fullsized":"2024-09-16T05:20:27.809309+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:20:00.415257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.415257+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.527133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056295900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":2116,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858054+0000","last_change":"2024-09-16T05:20:23.803973+0000","last_active":"2024-09-16T05:20:24.858054+0000","last_peered":"2024-09-16T05:20:24.858054+0000","last_clean":"2024-09-16T05:20:24.858054+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:20:24.858054+0000","last_undegraded":"2024-09-16T05:20:24.858054+0000","last_fullsized":"2024-09-16T05:20:24.858054+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:57.859756+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:19:57.859756+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.776621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097636700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.25","version":"674'1100","reported_seq":3835,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.829577+0000","last_change":"2024-09-16T05:20:23.800887+0000","last_active":"2024-09-16T05:20:23.829577+0000","last_peered":"2024-09-16T05:20:23.829577+0000","last_clean":"2024-09-16T05:20:23.829577+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:20:23.829577+0000","last_undegraded":"2024-09-16T05:20:23.829577+0000","last_fullsized":"2024-09-16T05:20:23.829577+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"622'977","last_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.332491+0000","objects_scrubbed":9,"log_size":1100,"log_dups_size":0,"ondisk_log_size":1100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:50.406942+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0.0045244719999999999,"stat_sum":{"num_bytes":27092020,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":188,"num_read_kb":197711,"num_write":167,"num_write_kb":66134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.26","version":"116'9","reported_seq":2113,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.810073+0000","last_change":"2024-09-16T05:20:26.810328+0000","last_active":"2024-09-16T05:20:26.810073+0000","last_peered":"2024-09-16T05:20:26.810073+0000","last_clean":"2024-09-16T05:20:26.810073+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:20:26.810073+0000","last_undegraded":"2024-09-16T05:20:26.810073+0000","last_fullsized":"2024-09-16T05:20:26.810073+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:07.483055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.483055+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:28.339992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062933349999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":2125,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.809250+0000","last_change":"2024-09-16T05:20:26.809590+0000","last_active":"2024-09-16T05:20:26.809250+0000","last_peered":"2024-09-16T05:20:26.809250+0000","last_clean":"2024-09-16T05:20:26.809250+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:20:26.809250+0000","last_undegraded":"2024-09-16T05:20:26.809250+0000","last_fullsized":"2024-09-16T05:20:26.809250+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:11.499561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.499561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.405687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058943429999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1138,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.827660+0000","last_change":"2024-09-16T05:20:26.828082+0000","last_active":"2024-09-16T05:20:26.827660+0000","last_peered":"2024-09-16T05:20:26.827660+0000","last_clean":"2024-09-16T05:20:26.827660+0000","last_became_active":"2024-09-16T05:19:35.785220+0000","last_became_peered":"2024-09-16T05:19:35.785220+0000","last_unstale":"2024-09-16T05:20:26.827660+0000","last_undegraded":"2024-09-16T05:20:26.827660+0000","last_fullsized":"2024-09-16T05:20:26.827660+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:19:27.303070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:27.303070+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.735002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068899800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.29","version":"677'988","reported_seq":3844,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.811791+0000","last_change":"2024-09-16T05:20:26.812158+0000","last_active":"2024-09-16T05:20:26.811791+0000","last_peered":"2024-09-16T05:20:26.811791+0000","last_clean":"2024-09-16T05:20:26.811791+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:20:26.811791+0000","last_undegraded":"2024-09-16T05:20:26.811791+0000","last_fullsized":"2024-09-16T05:20:26.811791+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"621'852","last_scrub_stamp":"2024-09-16T05:19:32.048085+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.048085+0000","objects_scrubbed":8,"log_size":988,"log_dups_size":0,"ondisk_log_size":988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:34.417652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0042748109999999999,"stat_sum":{"num_bytes":25096059,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":255,"num_read_kb":246175,"num_write":186,"num_write_kb":67059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":3115,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.807544+0000","last_change":"2024-09-16T05:20:26.807846+0000","last_active":"2024-09-16T05:20:26.807544+0000","last_peered":"2024-09-16T05:20:26.807544+0000","last_clean":"2024-09-16T05:20:26.807544+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:20:26.807544+0000","last_undegraded":"2024-09-16T05:20:26.807544+0000","last_fullsized":"2024-09-16T05:20:26.807544+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:19:29.329416+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.329416+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:59.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050244959999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2b","version":"666'332","reported_seq":3086,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828844+0000","last_change":"2024-09-16T05:20:26.829026+0000","last_active":"2024-09-16T05:20:26.828844+0000","last_peered":"2024-09-16T05:20:26.828844+0000","last_clean":"2024-09-16T05:20:26.828844+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:20:26.828844+0000","last_undegraded":"2024-09-16T05:20:26.828844+0000","last_fullsized":"2024-09-16T05:20:26.828844+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"666'332","last_scrub_stamp":"2024-09-16T05:20:18.508468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.508468+0000","objects_scrubbed":1,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.883681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013043390000000001,"stat_sum":{"num_bytes":2672478,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":79889,"num_write":47,"num_write_kb":21198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":2938,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-09-16T05:20:23.796263+0000","last_change":"2024-09-16T05:20:23.796528+0000","last_active":"2024-09-16T05:20:23.796263+0000","last_peered":"2024-09-16T05:20:23.796263+0000","last_clean":"2024-09-16T05:20:23.796263+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:20:23.796263+0000","last_undegraded":"2024-09-16T05:20:23.796263+0000","last_fullsized":"2024-09-16T05:20:23.796263+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:19:51.201665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.201665+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:59.115742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068113899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.2d","version":"666'459","reported_seq":3285,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.808932+0000","last_change":"2024-09-16T05:20:26.809908+0000","last_active":"2024-09-16T05:20:26.808932+0000","last_peered":"2024-09-16T05:20:26.808932+0000","last_clean":"2024-09-16T05:20:26.808932+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:20:26.808932+0000","last_undegraded":"2024-09-16T05:20:26.808932+0000","last_fullsized":"2024-09-16T05:20:26.808932+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"648'457","last_scrub_stamp":"2024-09-16T05:20:15.223777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.223777+0000","objects_scrubbed":2,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:15.497568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063157839999999996,"stat_sum":{"num_bytes":2823741,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":87,"num_read_kb":88579,"num_write":47,"num_write_kb":22068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2e","version":"677'616","reported_seq":3397,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.812309+0000","last_change":"2024-09-16T05:20:26.812309+0000","last_active":"2024-09-16T05:20:26.812309+0000","last_peered":"2024-09-16T05:20:26.812309+0000","last_clean":"2024-09-16T05:20:26.812309+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:20:26.812309+0000","last_undegraded":"2024-09-16T05:20:26.812309+0000","last_fullsized":"2024-09-16T05:20:26.812309+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"617'552","last_scrub_stamp":"2024-09-16T05:19:28.348887+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.348887+0000","objects_scrubbed":7,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:46.054638+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0.0082899770000000005,"stat_sum":{"num_bytes":13301925,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":80,"num_read_kb":55119,"num_write":86,"num_write_kb":33834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2f","version":"674'580","reported_seq":3436,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:24.858256+0000","last_change":"2024-09-16T05:20:23.809617+0000","last_active":"2024-09-16T05:20:24.858256+0000","last_peered":"2024-09-16T05:20:24.858256+0000","last_clean":"2024-09-16T05:20:24.858256+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:20:24.858256+0000","last_undegraded":"2024-09-16T05:20:24.858256+0000","last_fullsized":"2024-09-16T05:20:24.858256+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"650'553","last_scrub_stamp":"2024-09-16T05:20:00.804910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.804910+0000","objects_scrubbed":6,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.252445+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0057028620000000004,"stat_sum":{"num_bytes":10084662,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":108,"num_read_kb":77807,"num_write":97,"num_write_kb":39298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":279758940,"num_objects":130,"num_object_clones":84,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":14,"num_read":3199,"num_read_kb":3162571,"num_write":2331,"num_write_kb":926085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":47028772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372248576,"data_stored":473649358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24309,"ondisk_log_size":24309,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089481,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110164,"kb_used_data":56768,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93634988,"statfs":{"total":95995035648,"available":95882227712,"internally_reserved":0,"allocated":58130432,"data_stored":64560127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285709,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78312,"kb_used_data":47124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93666840,"statfs":{"total":95995035648,"available":95914844160,"internally_reserved":0,"allocated":48254976,"data_stored":53486620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514646,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89200,"kb_used_data":35808,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93655952,"statfs":{"total":95995035648,"available":95903694848,"internally_reserved":0,"allocated":36667392,"data_stored":43437466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841685,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81608,"kb_used_data":50420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93663544,"statfs":{"total":95995035648,"available":95911469056,"internally_reserved":0,"allocated":51630080,"data_stored":53570372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103324,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105968,"kb_used_data":48416,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93639184,"statfs":{"total":95995035648,"available":95886524416,"internally_reserved":0,"allocated":49577984,"data_stored":54868142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397667,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84684,"kb_used_data":49332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93660468,"statfs":{"total":95995035648,"available":95908319232,"internally_reserved":0,"allocated":50515968,"data_stored":59465888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757413,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65624,"kb_used_data":38588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93679528,"statfs":{"total":95995035648,"available":95927836672,"internally_reserved":0,"allocated":39514112,"data_stored":48166101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920939,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49540,"kb_used_data":22508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93695612,"statfs":{"total":95995035648,"available":95944306688,"internally_reserved":0,"allocated":23048192,"data_stored":23769814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117168,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87200,"kb_used_data":51852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93657952,"statfs":{"total":95995035648,"available":95905742848,"internally_reserved":0,"allocated":53096448,"data_stored":71858981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240455,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53924,"kb_used_data":17488,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93691228,"statfs":{"total":95995035648,"available":95939817472,"internally_reserved":0,"allocated":17907712,"data_stored":16253623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48300,"kb_used_data":17112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93696852,"statfs":{"total":95995035648,"available":95945576448,"internally_reserved":0,"allocated":17522688,"data_stored":19004943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9052160,"data_stored":13275819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18554880,"data_stored":23395830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9461760,"data_stored":10564131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44589056,"data_stored":66104497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14893056,"data_stored":18359970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31154176,"data_stored":42555009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42336256,"data_stored":54039084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":43061248,"data_stored":51756111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40570880,"data_stored":48352786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28524544,"data_stored":38036022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":40099840,"data_stored":48076776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":49950720,"data_stored":59133323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:20:30.338 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T05:20:30.338 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:20:30.338 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-16T05:20:30.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:20:30.339 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:30.340 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:30.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:30.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:30.465 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:30.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:30.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:30.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:30.642 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:30.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:30.717 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:30.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:30.788 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:20:30.789 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:20:30.789 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:20:30.789 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:20:30.789 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:20:30.792 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:20:30.796 INFO:tasks.rados.rados.2.smithi204.stdout:2659: done (0 left) 2024-09-16T05:20:30.796 INFO:tasks.rados.rados.2.smithi204.stdout:2660: write oid 39 current snap is 278 2024-09-16T05:20:30.796 INFO:tasks.rados.rados.2.smithi204.stdout:2660: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2024-09-16T05:20:30.803 INFO:tasks.rados.rados.2.smithi204.stdout:2660: writing smithi20457400-39 from 581773 to 1152003 tid 1 2024-09-16T05:20:30.811 INFO:tasks.rados.rados.2.smithi204.stdout:2660: writing smithi20457400-39 from 1664827 to 2191479 tid 2 2024-09-16T05:20:30.817 INFO:tasks.rados.rados.2.smithi204.stdout:2660: writing smithi20457400-39 from 2653249 to 3290022 tid 3 2024-09-16T05:20:30.818 INFO:tasks.rados.rados.2.smithi204.stdout:2661: write oid 42 current snap is 278 2024-09-16T05:20:30.818 INFO:tasks.rados.rados.2.smithi204.stdout:2660: finishing write tid 1 to smithi20457400-39 2024-09-16T05:20:30.818 INFO:tasks.rados.rados.2.smithi204.stdout:2660: finishing write tid 2 to smithi20457400-39 2024-09-16T05:20:30.818 INFO:tasks.rados.rados.2.smithi204.stdout:2661: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2024-09-16T05:20:30.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:30.824 INFO:tasks.rados.rados.2.smithi204.stdout:2661: writing smithi20457400-42 from 598580 to 1244059 tid 1 2024-09-16T05:20:30.831 INFO:tasks.rados.rados.2.smithi204.stdout:2661: writing smithi20457400-42 from 1723309 to 2509972 tid 2 2024-09-16T05:20:30.837 INFO:tasks.rados.rados.2.smithi204.stdout:2661: writing smithi20457400-42 from 3080672 to 3528717 tid 3 2024-09-16T05:20:30.837 INFO:tasks.rados.rados.2.smithi204.stdout:2662: write (excl) oid 9 current snap is 278 2024-09-16T05:20:30.837 INFO:tasks.rados.rados.2.smithi204.stdout:2660: finishing write tid 3 to smithi20457400-39 2024-09-16T05:20:30.837 INFO:tasks.rados.rados.2.smithi204.stdout:2660: finishing write tid 4 to smithi20457400-39 2024-09-16T05:20:30.837 INFO:tasks.rados.rados.2.smithi204.stdout:2662: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2024-09-16T05:20:30.840 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:30.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:30.844 INFO:tasks.rados.rados.2.smithi204.stdout:2662: writing smithi20457400-9 from 615387 to 1336115 tid 1 2024-09-16T05:20:30.850 INFO:tasks.rados.rados.2.smithi204.stdout:2662: writing smithi20457400-9 from 1781791 to 2312112 tid 2 2024-09-16T05:20:30.858 INFO:tasks.rados.rados.2.smithi204.stdout:2662: writing smithi20457400-9 from 3108095 to 3815387 tid 3 2024-09-16T05:20:30.858 INFO:tasks.rados.rados.2.smithi204.stdout:2663: write (excl) oid 28 current snap is 278 2024-09-16T05:20:30.858 INFO:tasks.rados.rados.2.smithi204.stdout:2660: finishing write tid 5 to smithi20457400-39 2024-09-16T05:20:30.858 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1113 (ObjNum 939 snap 278 seq_num 939) dirty exists 2024-09-16T05:20:30.859 INFO:tasks.rados.rados.2.smithi204.stdout:2660: left oid 39 (ObjNum 939 snap 278 seq_num 939) 2024-09-16T05:20:30.859 INFO:tasks.rados.rados.2.smithi204.stdout:2663: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-16T05:20:30.865 INFO:tasks.rados.rados.2.smithi204.stdout:2663: writing smithi20457400-28 from 632194 to 1428171 tid 1 2024-09-16T05:20:30.873 INFO:tasks.rados.rados.2.smithi204.stdout:2663: writing smithi20457400-28 from 2123920 to 2914252 tid 2 2024-09-16T05:20:30.879 INFO:tasks.rados.rados.2.smithi204.stdout:2663: writing smithi20457400-28 from 3419165 to 3832194 tid 3 2024-09-16T05:20:30.879 INFO:tasks.rados.rados.2.smithi204.stdout:2660: done (3 left) 2024-09-16T05:20:30.879 INFO:tasks.rados.rados.2.smithi204.stdout:2664: delete oid 44 current snap is 278 2024-09-16T05:20:30.879 INFO:tasks.rados.rados.2.smithi204.stdout:2661: finishing write tid 1 to smithi20457400-42 2024-09-16T05:20:30.879 INFO:tasks.rados.rados.2.smithi204.stdout:2661: finishing write tid 2 to smithi20457400-42 2024-09-16T05:20:30.879 INFO:tasks.rados.rados.2.smithi204.stdout:2661: finishing write tid 3 to smithi20457400-42 2024-09-16T05:20:30.879 INFO:tasks.rados.rados.2.smithi204.stdout:2661: finishing write tid 4 to smithi20457400-42 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2661: finishing write tid 5 to smithi20457400-42 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 765 (ObjNum 940 snap 278 seq_num 940) dirty exists 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2661: left oid 42 (ObjNum 940 snap 278 seq_num 940) 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2662: finishing write tid 1 to smithi20457400-9 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2662: finishing write tid 2 to smithi20457400-9 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2662: finishing write tid 3 to smithi20457400-9 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2662: finishing write tid 4 to smithi20457400-9 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2662: finishing write tid 5 to smithi20457400-9 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 999 (ObjNum 941 snap 278 seq_num 941) dirty exists 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2662: left oid 9 (ObjNum 941 snap 278 seq_num 941) 2024-09-16T05:20:30.880 INFO:tasks.rados.rados.2.smithi204.stdout:2663: finishing write tid 1 to smithi20457400-28 2024-09-16T05:20:30.881 INFO:tasks.rados.rados.2.smithi204.stdout:2663: finishing write tid 2 to smithi20457400-28 2024-09-16T05:20:30.881 INFO:tasks.rados.rados.2.smithi204.stdout:2661: done (3 left) 2024-09-16T05:20:30.881 INFO:tasks.rados.rados.2.smithi204.stdout:2662: done (2 left) 2024-09-16T05:20:30.882 INFO:tasks.rados.rados.2.smithi204.stdout:2664: done (1 left) 2024-09-16T05:20:30.882 INFO:tasks.rados.rados.2.smithi204.stdout:2665: read oid 2 snap 273 2024-09-16T05:20:30.882 INFO:tasks.rados.rados.2.smithi204.stdout:2665: expect deleted 2024-09-16T05:20:30.882 INFO:tasks.rados.rados.2.smithi204.stdout:2666: write oid 42 current snap is 278 2024-09-16T05:20:30.882 INFO:tasks.rados.rados.2.smithi204.stdout:2666: seq_num 944 ranges {665808=662828,1957237=677648,3074011=609486} 2024-09-16T05:20:30.887 INFO:tasks.rados.rados.2.smithi204.stdout:2666: writing smithi20457400-42 from 665808 to 1328636 tid 1 2024-09-16T05:20:30.894 INFO:tasks.rados.rados.2.smithi204.stdout:2666: writing smithi20457400-42 from 1957237 to 2634885 tid 2 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:2666: writing smithi20457400-42 from 3074011 to 3683497 tid 3 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:2667: snap_create 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:2663: finishing write tid 3 to smithi20457400-28 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:2663: finishing write tid 4 to smithi20457400-28 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:2663: finishing write tid 5 to smithi20457400-28 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1091 (ObjNum 942 snap 278 seq_num 942) dirty exists 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:2663: left oid 28 (ObjNum 942 snap 278 seq_num 942) 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:2666: finishing write tid 1 to smithi20457400-42 2024-09-16T05:20:30.900 INFO:tasks.rados.rados.2.smithi204.stdout:2666: finishing write tid 2 to smithi20457400-42 2024-09-16T05:20:30.902 INFO:tasks.rados.rados.2.smithi204.stdout:2666: finishing write tid 3 to smithi20457400-42 2024-09-16T05:20:30.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:30.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:31.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:20:31.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:31.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:31.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:31.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:30 smithi204 ceph-mon[30933]: from='client.16758 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:31.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:30 smithi204 ceph-mon[30933]: osdmap e680: 12 total, 12 up, 11 in 2024-09-16T05:20:31.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4032329826' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:31.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:30 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:31.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4032329826' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:31.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:30 smithi204 ceph-mon[30933]: osdmap e681: 12 total, 12 up, 11 in 2024-09-16T05:20:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:30 smithi032 ceph-mon[27388]: from='client.16758 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:30 smithi032 ceph-mon[27388]: osdmap e680: 12 total, 12 up, 11 in 2024-09-16T05:20:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4032329826' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:30 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4032329826' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:30 smithi032 ceph-mon[27388]: osdmap e681: 12 total, 12 up, 11 in 2024-09-16T05:20:31.204 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:31.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:31.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:31.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:30 smithi196 ceph-mon[30802]: from='client.16758 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:31.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:30 smithi196 ceph-mon[30802]: osdmap e680: 12 total, 12 up, 11 in 2024-09-16T05:20:31.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4032329826' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:30 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4032329826' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:30 smithi196 ceph-mon[30802]: osdmap e681: 12 total, 12 up, 11 in 2024-09-16T05:20:31.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:31.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:31.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:31.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:20:31.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:31.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:20:31.473 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:20:31.476 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:31.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:31.562 INFO:tasks.rados.rados.2.smithi204.stdout:2666: finishing write tid 4 to smithi20457400-42 2024-09-16T05:20:31.562 INFO:tasks.rados.rados.2.smithi204.stdout:2666: finishing write tid 5 to smithi20457400-42 2024-09-16T05:20:31.562 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 769 (ObjNum 944 snap 278 seq_num 944) dirty exists 2024-09-16T05:20:31.562 INFO:tasks.rados.rados.2.smithi204.stdout:2666: left oid 42 (ObjNum 944 snap 278 seq_num 944) 2024-09-16T05:20:31.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:31.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:31.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:31.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:31.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:31.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:31.732 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:31.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:31.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:31.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:31.883 INFO:tasks.rados.rados.2.smithi204.stdout:2663: done (3 left) 2024-09-16T05:20:31.883 INFO:tasks.rados.rados.2.smithi204.stdout:2665: done (2 left) 2024-09-16T05:20:31.883 INFO:tasks.rados.rados.2.smithi204.stdout:2666: done (1 left) 2024-09-16T05:20:31.883 INFO:tasks.rados.rados.2.smithi204.stdout:2667: done (0 left) 2024-09-16T05:20:31.883 INFO:tasks.rados.rados.2.smithi204.stdout:2668: write (excl) oid 42 current snap is 279 2024-09-16T05:20:31.883 INFO:tasks.rados.rados.2.smithi204.stdout:2668: seq_num 945 ranges {682615=738077,2015719=537659,3217787=420758} 2024-09-16T05:20:31.891 INFO:tasks.rados.rados.2.smithi204.stdout:2668: writing smithi20457400-42 from 682615 to 1420692 tid 1 2024-09-16T05:20:31.897 INFO:tasks.rados.rados.2.smithi204.stdout:2668: writing smithi20457400-42 from 2015719 to 2553378 tid 2 2024-09-16T05:20:31.908 INFO:tasks.rados.rados.2.smithi204.stdout:2668: writing smithi20457400-42 from 3217787 to 3638545 tid 3 2024-09-16T05:20:31.908 INFO:tasks.rados.rados.2.smithi204.stdout:2669: snap_remove snap 272 2024-09-16T05:20:31.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:31.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:31.977 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:31.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:32.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:32.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:32.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:32.118 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:32.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:31 smithi204 ceph-mon[30933]: pgmap v1182: 107 pgs: 107 active+clean; 269 MiB data, 833 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-16T05:20:32.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:31 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:32.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:31 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:31 smithi032 ceph-mon[27388]: pgmap v1182: 107 pgs: 107 active+clean; 269 MiB data, 833 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-16T05:20:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:31 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:31 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:32.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:32.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:32.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:32.231 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:20:32.231 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:19 2.b 2:d7c71ae7:::smithi20457400-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 589.18935420599996, 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19676892500000001, 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.232 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 19 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040518+0000", 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040519+0000", 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7300000000000001e-07 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040541+0000", 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2320999999999999e-05 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040547+0000", 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3060000000000003e-06 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040553+0000", 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2750000000000003e-06 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041020+0000", 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046726200000000003 2024-09-16T05:20:32.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041092+0000", 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2275999999999997e-05 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041183+0000", 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1071999999999998e-05 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:32.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041456+0000", 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027275299999999998 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237175+0000", 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19571920200000001 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237230+0000", 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4855000000000002e-05 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237287+0000", 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7129999999999997e-05 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:20 2.b 2:d7c71ae7:::smithi20457400-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 589.18931247199998, 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19685694000000001, 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.238 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 20 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040560+0000", 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4400000000000001e-07 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040587+0000", 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6444e-05 2024-09-16T05:20:32.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040593+0000", 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5320000000000003e-06 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.040600+0000", 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5139999999999998e-06 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041356+0000", 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075660300000000001 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.041400+0000", 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3936000000000001e-05 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237316+0000", 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.195915805 2024-09-16T05:20:32.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237361+0000", 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4548000000000001e-05 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:43.237417+0000", 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6214000000000001e-05 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.243 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:286 2.b 2:da4f6049:::smithi20457400-49:head [delete] snapc 4=[4] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117)", 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 583.13174380199996, 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.242160297, 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.244 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 286 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098129+0000", 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6699999999999999e-07 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098131+0000", 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1289999999999999e-06 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:31 smithi196 ceph-mon[30802]: pgmap v1182: 107 pgs: 107 active+clean; 269 MiB data, 833 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-09-16T05:20:32.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:31 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:32.247 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:31 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:32.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098138+0000", 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5660000000000003e-06 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098143+0000", 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098194+0000", 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1368e-05 2024-09-16T05:20:32.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098273+0000", 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8170999999999994e-05 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.098449+0000", 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176171 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:32.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.099087+0000", 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00063836000000000003 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340229+0000", 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.241141684 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340263+0000", 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3813000000000003e-05 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:49.340289+0000", 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6403000000000001e-05 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:414 2.5 2:ad4a88eb:::smithi20457400-16:head [setxattr _header (56) in=7b,truncate 3394981] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 574.53295328299998, 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29453131100000002, 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.251 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 414 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696919+0000", 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696920+0000", 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999996e-07 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696923+0000", 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.92e-06 2024-09-16T05:20:32.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696937+0000", 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3892e-05 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696941+0000", 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3819999999999997e-06 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697487+0000", 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00054579200000000002 2024-09-16T05:20:32.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697548+0000", 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1167000000000006e-05 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697638+0000", 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0548000000000001e-05 2024-09-16T05:20:32.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697936+0000", 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029720499999999998 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991363+0000", 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29342759099999999 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:32.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991404+0000", 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1013999999999997e-05 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991451+0000", 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6303e-05 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:415 2.5 2:ad4a88eb:::smithi20457400-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:20:32.257 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 574.53292515099997, 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29466997700000003, 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 415 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696947+0000", 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696948+0000", 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696952+0000", 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6500000000000002e-06 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696958+0000", 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9149999999999999e-06 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.696963+0000", 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9439999999999996e-06 2024-09-16T05:20:32.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697806+0000", 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084291800000000001 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.697861+0000", 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4725999999999997e-05 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991493+0000", 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29363172500000001 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991551+0000", 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8622000000000002e-05 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991617+0000", 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5977999999999997e-05 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.262 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:414 2.5 e126/107 ondisk, result = 0)", 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 574.531763576, 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29334887300000001, 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698109+0000", 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.06e-07 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.698112+0000", 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6520000000000002e-06 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991214+0000", 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.29310221600000003 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991228+0000", 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.411e-05 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991351+0000", 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012277999999999999 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991359+0000", 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0430000000000008e-06 2024-09-16T05:20:32.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:10:57.991458+0000", 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8566000000000005e-05 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:884 2.b 2:d8680303:::smithi20457400-43:head [setxattr _header (57) in=7b,truncate 2470629] snapc 23=[23,22,21,20,1f,1b,19,15,12,f] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 530.56919754499995, 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68080432499999999, 2024-09-16T05:20:32.267 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 884 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660675+0000", 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660676+0000", 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2099999999999996e-07 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660679+0000", 2024-09-16T05:20:32.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1769999999999998e-06 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660685+0000", 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6529999999999997e-06 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660691+0000", 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2530000000000004e-06 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.661798+0000", 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011070520000000001 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.661889+0000", 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0785999999999994e-05 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662038+0000", 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000149777 2024-09-16T05:20:32.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662482+0000", 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00044356499999999999 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341385+0000", 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67890330499999996 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:32.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341430+0000", 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4613999999999997e-05 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341479+0000", 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9521999999999998e-05 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:885 2.b 2:d8680303:::smithi20457400-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", 2024-09-16T05:20:32.273 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 530.56917347499996, 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68095269999999997, 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 885 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660699+0000", 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999999e-07 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660702+0000", 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3099999999999999e-06 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660709+0000", 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0299999999999996e-06 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.660714+0000", 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1780000000000002e-06 2024-09-16T05:20:32.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662345+0000", 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016309390000000001 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:41.662461+0000", 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011565600000000001 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341517+0000", 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67905638400000001 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341582+0000", 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4746999999999995e-05 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:11:42.341652+0000", 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0026999999999999e-05 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.278 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:405 2.1a e216/215 ondisk, result = 0)", 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 484.56336661400002, 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26443446100000001, 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666506+0000", 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666507+0000", 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8700000000000005e-07 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.666511+0000", 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1130000000000003e-06 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930764+0000", 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26425358900000001 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930771+0000", 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7329999999999997e-06 2024-09-16T05:20:32.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930824+0000", 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3436000000000001e-05 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930834+0000", 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1570000000000006e-06 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:27.930940+0000", 2024-09-16T05:20:32.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106746 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1814 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 453.45809821300003, 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898298185, 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:20:32.283 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1814 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771776+0000", 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5149999999999999e-06 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771783+0000", 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0180000000000004e-06 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771805+0000", 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2076999999999999e-05 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771824+0000", 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8813000000000001e-05 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747524+0000", 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97570031700000004 2024-09-16T05:20:32.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747560+0000", 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5549000000000001e-05 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670057+0000", 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92249772500000005 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670072+0000", 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5170999999999999e-05 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1815 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 453.45799631400001, 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8982342750000001, 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:20:32.288 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1815 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771877+0000", 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9599999999999999e-07 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771881+0000", 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6880000000000001e-06 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771889+0000", 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6789999999999994e-06 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771896+0000", 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747584+0000", 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97568814000000004 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747590+0000", 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7440000000000004e-06 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670102+0000", 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92251251499999998 2024-09-16T05:20:32.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670110+0000", 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2740000000000007e-06 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1816 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:20:32.294 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 453.45791432499999, 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898192844, 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1816 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771959+0000", 2024-09-16T05:20:32.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1999999999999998e-07 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771964+0000", 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771971+0000", 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.576e-06 2024-09-16T05:20:32.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771977+0000", 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2960000000000004e-06 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747609+0000", 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97563200299999997 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747615+0000", 2024-09-16T05:20:32.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2820000000000004e-06 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670144+0000", 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92252909100000002 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670151+0000", 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2259999999999999e-06 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.299 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1881 2.25 2:a49174eb:::smithi20457400-39:head [delete] snapc 50=[50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e257)", 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 443.53332502400002, 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37621878800000003, 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.300 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1881 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696549+0000", 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696556+0000", 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8090000000000004e-06 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696568+0000", 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2636000000000001e-05 2024-09-16T05:20:32.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696576+0000", 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3459999999999996e-06 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696625+0000", 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9830000000000002e-05 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696720+0000", 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4816999999999997e-05 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696869+0000", 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014893999999999999 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.697931+0000", 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010616759999999999 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072669+0000", 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37473788800000002 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072715+0000", 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6371e-05 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072766+0000", 2024-09-16T05:20:32.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1220000000000001e-05 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1881 2.25 e257/252 ondisk, result = 0)", 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 443.53179677499998, 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37469565799999999, 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.306 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698077+0000", 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5099999999999998e-07 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698082+0000", 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072556+0000", 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37447454499999999 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072564+0000", 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4120000000000004e-06 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072654+0000", 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9878999999999998e-05 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072663+0000", 2024-09-16T05:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072771+0000", 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108267 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:839 2.1a e438/291 ondisk, result = 0)", 2024-09-16T05:20:32.310 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 244.499383354, 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51227693699999999, 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730490+0000", 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730494+0000", 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0810000000000004e-06 2024-09-16T05:20:32.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242338+0000", 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51184441999999997 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242353+0000", 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.447e-05 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242501+0000", 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014851799999999999 2024-09-16T05:20:32.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242532+0000", 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0406999999999999e-05 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242766+0000", 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000234587 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGRecoveryDelete(2.4c e439,438 2:33100188:::smithi20457400-19:2f,230'190)", 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 243.39647224699999, 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.584288378, 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833401+0000", 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833406+0000", 2024-09-16T05:20:32.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8940000000000001e-06 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417262+0000", 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.58385600999999998 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417279+0000", 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6880000000000001e-05 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417458+0000", 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000178751 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417489+0000", 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1192999999999998e-05 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417689+0000", 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019989399999999999 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5460 2.25 2:a5bc1608:::smithi20457400-11:head [setxattr _header (58) in=7b,truncate 2975004] snapc e1=[e1,e0,df,dd,dc,da,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 108.23544592899999, 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69151579299999999, 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.319 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5460 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994430+0000", 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2729999999999998e-06 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994437+0000", 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9670000000000004e-06 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994443+0000", 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7780000000000004e-06 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995089+0000", 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064609900000000002 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995143+0000", 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4042000000000002e-05 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995231+0000", 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7798000000000002e-05 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995551+0000", 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000319847 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685849+0000", 2024-09-16T05:20:32.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.690298254 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685893+0000", 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3368000000000002e-05 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685942+0000", 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9811000000000001e-05 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5461 2.25 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 108.235418784, 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69165347799999999, 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5461 2024-09-16T05:20:32.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994457+0000", 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.429e-06 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994463+0000", 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994469+0000", 2024-09-16T05:20:32.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4380000000000003e-06 2024-09-16T05:20:32.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995446+0000", 2024-09-16T05:20:32.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00097678900000000004 2024-09-16T05:20:32.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995491+0000", 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5153e-05 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685982+0000", 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69049136499999997 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686042+0000", 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9896999999999999e-05 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686107+0000", 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5270000000000004e-05 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6242 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 913] snapc 103=[103,102,101,100,fe,fd,fc,fb,fa,f9,f7,f6,f2,f0,e7] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.416545177000003, 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47407041599999999, 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6242 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813328+0000", 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9399999999999993e-07 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813334+0000", 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1619999999999996e-06 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813347+0000", 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2145e-05 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813354+0000", 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1230000000000002e-06 2024-09-16T05:20:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813414+0000", 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0489999999999999e-05 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813421+0000", 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1010000000000003e-06 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816226+0000", 2024-09-16T05:20:32.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002804647 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816349+0000", 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012330699999999999 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.281344+0000", 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.464994937 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284684+0000", 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033402309999999999 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287216+0000", 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002531298 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287319+0000", 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102865 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287398+0000", 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9116000000000006e-05 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6243 2.5 2:a1d8c6d9:::smithi20457400-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.416499799999997, 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47072968100000001, 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:32.342 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6243 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1800000000000005e-07 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813377+0000", 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3699999999999999e-06 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813385+0000", 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5280000000000003e-06 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813392+0000", 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0600000000000002e-06 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813515+0000", 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012306 2024-09-16T05:20:32.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813521+0000", 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.203e-06 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816658+0000", 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003137365 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816689+0000", 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0592000000000001e-05 2024-09-16T05:20:32.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.283968+0000", 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46727931 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284030+0000", 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2316000000000004e-05 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284102+0000", 2024-09-16T05:20:32.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2158999999999998e-05 2024-09-16T05:20:32.348 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.348 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.348 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.348 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:32.348 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:32.348 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:20:32.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:32.371 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:32.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:32.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:32.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:32.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:32.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:32.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:32.689 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1183,"stamp":"2024-09-16T05:20:30.792660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":283960318,"num_objects":134,"num_object_clones":86,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":14,"num_read":3219,"num_read_kb":3190773,"num_write":2341,"num_write_kb":929327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":47028772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27284,"ondisk_log_size":27284,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":215,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":859288,"kb_used_data":435996,"kb_used_omap":26,"kb_used_meta":423013,"kb_avail":1030337384,"statfs":{"total":1055945392128,"available":1055065481216,"internally_reserved":0,"allocated":446459904,"data_stored":513406724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26910,"internal_metadata":433166050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4839734,"num_objects":3,"num_object_clones":4,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":53500,"num_write":21,"num_write_kb":6694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.993684"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":3296,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156230+0000","last_change":"2024-09-16T05:20:23.808412+0000","last_active":"2024-09-16T05:20:25.156230+0000","last_peered":"2024-09-16T05:20:25.156230+0000","last_clean":"2024-09-16T05:20:25.156230+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:20:25.156230+0000","last_undegraded":"2024-09-16T05:20:25.156230+0000","last_fullsized":"2024-09-16T05:20:25.156230+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:20:01.826054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:01.826054+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.488219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012806409999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.54","version":"674'397","reported_seq":2850,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828533+0000","last_change":"2024-09-16T05:20:26.828782+0000","last_active":"2024-09-16T05:20:26.828533+0000","last_peered":"2024-09-16T05:20:26.828533+0000","last_clean":"2024-09-16T05:20:26.828533+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:20:26.828533+0000","last_undegraded":"2024-09-16T05:20:26.828533+0000","last_fullsized":"2024-09-16T05:20:26.828533+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"619'381","last_scrub_stamp":"2024-09-16T05:19:29.293042+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.293042+0000","objects_scrubbed":3,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:43.684504+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.001110469,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":57049,"num_write":34,"num_write_kb":11523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":1190,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.794330+0000","last_change":"2024-09-16T05:20:23.795697+0000","last_active":"2024-09-16T05:20:25.794330+0000","last_peered":"2024-09-16T05:20:25.794330+0000","last_clean":"2024-09-16T05:20:25.794330+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:20:25.794330+0000","last_undegraded":"2024-09-16T05:20:25.794330+0000","last_fullsized":"2024-09-16T05:20:25.794330+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:32.054132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.054132+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.553292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040824999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":1217,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:30.037548+0000","last_change":"2024-09-16T05:20:28.809081+0000","last_active":"2024-09-16T05:20:30.037548+0000","last_peered":"2024-09-16T05:20:30.037548+0000","last_clean":"2024-09-16T05:20:30.037548+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:20:30.037548+0000","last_undegraded":"2024-09-16T05:20:30.037548+0000","last_fullsized":"2024-09-16T05:20:30.037548+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:18.588272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.588272+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:43.807082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071203200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":1257,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801486+0000","last_change":"2024-09-16T05:20:23.808960+0000","last_active":"2024-09-16T05:20:25.801486+0000","last_peered":"2024-09-16T05:20:25.801486+0000","last_clean":"2024-09-16T05:20:25.801486+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:20:25.801486+0000","last_undegraded":"2024-09-16T05:20:25.801486+0000","last_fullsized":"2024-09-16T05:20:25.801486+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:19:29.589220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.589220+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:40.708156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001479932,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":1203,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.808945+0000","last_change":"2024-09-16T05:20:27.809239+0000","last_active":"2024-09-16T05:20:27.808945+0000","last_peered":"2024-09-16T05:20:27.808945+0000","last_clean":"2024-09-16T05:20:27.808945+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:20:27.808945+0000","last_undegraded":"2024-09-16T05:20:27.808945+0000","last_fullsized":"2024-09-16T05:20:27.808945+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:29.302740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.302740+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.020231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000430252,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2187,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.809688+0000","last_change":"2024-09-16T05:20:26.810092+0000","last_active":"2024-09-16T05:20:26.809688+0000","last_peered":"2024-09-16T05:20:26.809688+0000","last_clean":"2024-09-16T05:20:26.809688+0000","last_became_active":"2024-09-16T05:19:35.789805+0000","last_became_peered":"2024-09-16T05:19:35.789805+0000","last_unstale":"2024-09-16T05:20:26.809688+0000","last_undegraded":"2024-09-16T05:20:26.809688+0000","last_fullsized":"2024-09-16T05:20:26.809688+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:19:48.302807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.302807+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:06.121391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0063601899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.4e","version":"272'232","reported_seq":2957,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156424+0000","last_change":"2024-09-16T05:20:23.808015+0000","last_active":"2024-09-16T05:20:25.156424+0000","last_peered":"2024-09-16T05:20:25.156424+0000","last_clean":"2024-09-16T05:20:25.156424+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:20:25.156424+0000","last_undegraded":"2024-09-16T05:20:25.156424+0000","last_fullsized":"2024-09-16T05:20:25.156424+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:50.792260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.792260+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:15.909321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00088863899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.4d","version":"665'626","reported_seq":3519,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828730+0000","last_change":"2024-09-16T05:20:26.828984+0000","last_active":"2024-09-16T05:20:26.828730+0000","last_peered":"2024-09-16T05:20:26.828730+0000","last_clean":"2024-09-16T05:20:26.828730+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:20:26.828730+0000","last_undegraded":"2024-09-16T05:20:26.828730+0000","last_fullsized":"2024-09-16T05:20:26.828730+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"665'626","last_scrub_stamp":"2024-09-16T05:20:16.492166+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.492166+0000","objects_scrubbed":4,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.198501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013191909999999999,"stat_sum":{"num_bytes":6503878,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":45,"num_read_kb":38966,"num_write":56,"num_write_kb":18053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.4c","version":"674'350","reported_seq":3174,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.802040+0000","last_change":"2024-09-16T05:20:23.808587+0000","last_active":"2024-09-16T05:20:25.802040+0000","last_peered":"2024-09-16T05:20:25.802040+0000","last_clean":"2024-09-16T05:20:25.802040+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:20:25.802040+0000","last_undegraded":"2024-09-16T05:20:25.802040+0000","last_fullsized":"2024-09-16T05:20:25.802040+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"666'345","last_scrub_stamp":"2024-09-16T05:20:17.450506+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.450506+0000","objects_scrubbed":1,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.178106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001716029,"stat_sum":{"num_bytes":5239557,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":37572,"num_write":33,"num_write_kb":9268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":3001,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828242+0000","last_change":"2024-09-16T05:20:26.828501+0000","last_active":"2024-09-16T05:20:26.828242+0000","last_peered":"2024-09-16T05:20:26.828242+0000","last_clean":"2024-09-16T05:20:26.828242+0000","last_became_active":"2024-09-16T05:17:21.741180+0000","last_became_peered":"2024-09-16T05:17:21.741180+0000","last_unstale":"2024-09-16T05:20:26.828242+0000","last_undegraded":"2024-09-16T05:20:26.828242+0000","last_fullsized":"2024-09-16T05:20:26.828242+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:19:32.417815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.417815+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:40.936009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096174599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.4a","version":"620'557","reported_seq":3346,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801566+0000","last_change":"2024-09-16T05:20:23.809821+0000","last_active":"2024-09-16T05:20:25.801566+0000","last_peered":"2024-09-16T05:20:25.801566+0000","last_clean":"2024-09-16T05:20:25.801566+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:20:25.801566+0000","last_undegraded":"2024-09-16T05:20:25.801566+0000","last_fullsized":"2024-09-16T05:20:25.801566+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"620'557","last_scrub_stamp":"2024-09-16T05:19:54.555606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.555606+0000","objects_scrubbed":1,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.900460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099282899999999998,"stat_sum":{"num_bytes":2689285,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":47,"num_read_kb":41138,"num_write":35,"num_write_kb":18755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":3034,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155846+0000","last_change":"2024-09-16T05:20:23.807313+0000","last_active":"2024-09-16T05:20:25.155846+0000","last_peered":"2024-09-16T05:20:25.155846+0000","last_clean":"2024-09-16T05:20:25.155846+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:20:25.155846+0000","last_undegraded":"2024-09-16T05:20:25.155846+0000","last_fullsized":"2024-09-16T05:20:25.155846+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:20.094577+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.094577+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.217200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043004700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.48","version":"679'329","reported_seq":3141,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870656+0000","last_change":"2024-09-16T05:20:28.822954+0000","last_active":"2024-09-16T05:20:29.870656+0000","last_peered":"2024-09-16T05:20:29.870656+0000","last_clean":"2024-09-16T05:20:29.870656+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:20:29.870656+0000","last_undegraded":"2024-09-16T05:20:29.870656+0000","last_fullsized":"2024-09-16T05:20:29.870656+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"628'287","last_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:19:38.000359+0000","objects_scrubbed":3,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.036620+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0081425730000000002,"stat_sum":{"num_bytes":8870985,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67118,"num_write":50,"num_write_kb":24235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":1835,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:30.037775+0000","last_change":"2024-09-16T05:20:28.809528+0000","last_active":"2024-09-16T05:20:30.037775+0000","last_peered":"2024-09-16T05:20:30.037775+0000","last_clean":"2024-09-16T05:20:30.037775+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:20:30.037775+0000","last_undegraded":"2024-09-16T05:20:30.037775+0000","last_fullsized":"2024-09-16T05:20:30.037775+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:30.390373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.390373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:36.420619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065691600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":44,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828474+0000","last_change":"2024-09-16T05:20:26.828650+0000","last_active":"2024-09-16T05:20:26.828474+0000","last_peered":"2024-09-16T05:20:26.828474+0000","last_clean":"2024-09-16T05:20:26.828474+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:20:26.828474+0000","last_undegraded":"2024-09-16T05:20:26.828474+0000","last_fullsized":"2024-09-16T05:20:26.828474+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:49.313870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.313870+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:56.333549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001011571,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":2775,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801482+0000","last_change":"2024-09-16T05:20:23.808800+0000","last_active":"2024-09-16T05:20:25.801482+0000","last_peered":"2024-09-16T05:20:25.801482+0000","last_clean":"2024-09-16T05:20:25.801482+0000","last_became_active":"2024-09-16T05:19:35.786934+0000","last_became_peered":"2024-09-16T05:19:35.786934+0000","last_unstale":"2024-09-16T05:20:25.801482+0000","last_undegraded":"2024-09-16T05:20:25.801482+0000","last_fullsized":"2024-09-16T05:20:25.801482+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:19:09.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:09.927260+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:29.166375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001907401,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":1400,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.817134+0000","last_change":"2024-09-16T05:20:26.817356+0000","last_active":"2024-09-16T05:20:26.817134+0000","last_peered":"2024-09-16T05:20:26.817134+0000","last_clean":"2024-09-16T05:20:26.817134+0000","last_became_active":"2024-09-16T05:19:35.788828+0000","last_became_peered":"2024-09-16T05:19:35.788828+0000","last_unstale":"2024-09-16T05:20:26.817134+0000","last_undegraded":"2024-09-16T05:20:26.817134+0000","last_fullsized":"2024-09-16T05:20:26.817134+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:48.163046+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.163046+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:13.176337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0054578780000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":1805,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.808080+0000","last_change":"2024-09-16T05:20:26.808512+0000","last_active":"2024-09-16T05:20:26.808080+0000","last_peered":"2024-09-16T05:20:26.808080+0000","last_clean":"2024-09-16T05:20:26.808080+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:20:26.808080+0000","last_undegraded":"2024-09-16T05:20:26.808080+0000","last_fullsized":"2024-09-16T05:20:26.808080+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:19:15.441056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.441056+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:27.117522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053896129999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":1799,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.806181+0000","last_change":"2024-09-16T05:20:28.806480+0000","last_active":"2024-09-16T05:20:28.806181+0000","last_peered":"2024-09-16T05:20:28.806181+0000","last_clean":"2024-09-16T05:20:28.806181+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:20:28.806181+0000","last_undegraded":"2024-09-16T05:20:28.806181+0000","last_fullsized":"2024-09-16T05:20:28.806181+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:19:28.221037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.221037+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.070956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000895402,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":675,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:30.037523+0000","last_change":"2024-09-16T05:20:28.809123+0000","last_active":"2024-09-16T05:20:30.037523+0000","last_peered":"2024-09-16T05:20:30.037523+0000","last_clean":"2024-09-16T05:20:30.037523+0000","last_became_active":"2024-09-16T05:18:19.768547+0000","last_became_peered":"2024-09-16T05:18:19.768547+0000","last_unstale":"2024-09-16T05:20:30.037523+0000","last_undegraded":"2024-09-16T05:20:30.037523+0000","last_fullsized":"2024-09-16T05:20:30.037523+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:19:55.515448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.515448+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.175440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075038000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":345,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156704+0000","last_change":"2024-09-16T05:20:23.807332+0000","last_active":"2024-09-16T05:20:25.156704+0000","last_peered":"2024-09-16T05:20:25.156704+0000","last_clean":"2024-09-16T05:20:25.156704+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:20:25.156704+0000","last_undegraded":"2024-09-16T05:20:25.156704+0000","last_fullsized":"2024-09-16T05:20:25.156704+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:53.588594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.588594+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:12.149127+0000","scrub_duration":798,"objects_trimmed":0,"snaptrim_duration":0.00054597499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.3f","version":"666'419","reported_seq":3259,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.808630+0000","last_change":"2024-09-16T05:20:26.809168+0000","last_active":"2024-09-16T05:20:26.808630+0000","last_peered":"2024-09-16T05:20:26.808630+0000","last_clean":"2024-09-16T05:20:26.808630+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:20:26.808630+0000","last_undegraded":"2024-09-16T05:20:26.808630+0000","last_fullsized":"2024-09-16T05:20:26.808630+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'415","last_scrub_stamp":"2024-09-16T05:19:58.393481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:58.393481+0000","objects_scrubbed":2,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.473419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0057641130000000004,"stat_sum":{"num_bytes":5748357,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":82274,"num_write":47,"num_write_kb":13993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.3e","version":"625'487","reported_seq":3282,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.812160+0000","last_change":"2024-09-16T05:20:26.812437+0000","last_active":"2024-09-16T05:20:26.812160+0000","last_peered":"2024-09-16T05:20:26.812160+0000","last_clean":"2024-09-16T05:20:26.812160+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:20:26.812160+0000","last_undegraded":"2024-09-16T05:20:26.812160+0000","last_fullsized":"2024-09-16T05:20:26.812160+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"625'487","last_scrub_stamp":"2024-09-16T05:20:22.404334+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:20:22.404334+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.501331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001331675,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":26573,"num_write":41,"num_write_kb":23888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1745,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.808815+0000","last_change":"2024-09-16T05:20:28.809140+0000","last_active":"2024-09-16T05:20:28.808815+0000","last_peered":"2024-09-16T05:20:28.808815+0000","last_clean":"2024-09-16T05:20:28.808815+0000","last_became_active":"2024-09-16T05:19:35.784967+0000","last_became_peered":"2024-09-16T05:19:35.784967+0000","last_unstale":"2024-09-16T05:20:28.808815+0000","last_undegraded":"2024-09-16T05:20:28.808815+0000","last_fullsized":"2024-09-16T05:20:28.808815+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:19:36.333160+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.333160+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:53.684049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029877720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.3c","version":"677'712","reported_seq":3702,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.807101+0000","last_change":"2024-09-16T05:20:26.807705+0000","last_active":"2024-09-16T05:20:26.807101+0000","last_peered":"2024-09-16T05:20:26.807101+0000","last_clean":"2024-09-16T05:20:26.807101+0000","last_became_active":"2024-09-16T05:19:35.800730+0000","last_became_peered":"2024-09-16T05:19:35.800730+0000","last_unstale":"2024-09-16T05:20:26.807101+0000","last_undegraded":"2024-09-16T05:20:26.807101+0000","last_fullsized":"2024-09-16T05:20:26.807101+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"624'648","last_scrub_stamp":"2024-09-16T05:19:34.258477+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:19:34.258477+0000","objects_scrubbed":6,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:52.675399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0045688839999999996,"stat_sum":{"num_bytes":13004770,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":161,"num_read_kb":208719,"num_write":78,"num_write_kb":29911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":6910647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.3b","version":"654'428","reported_seq":3242,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156907+0000","last_change":"2024-09-16T05:20:23.807772+0000","last_active":"2024-09-16T05:20:25.156907+0000","last_peered":"2024-09-16T05:20:25.156907+0000","last_clean":"2024-09-16T05:20:25.156907+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:20:25.156907+0000","last_undegraded":"2024-09-16T05:20:25.156907+0000","last_fullsized":"2024-09-16T05:20:25.156907+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'416","last_scrub_stamp":"2024-09-16T05:19:55.767476+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.767476+0000","objects_scrubbed":2,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.864203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075645699999999998,"stat_sum":{"num_bytes":8222944,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54616,"num_write":46,"num_write_kb":21574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.3a","version":"667'593","reported_seq":3395,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870837+0000","last_change":"2024-09-16T05:20:28.817228+0000","last_active":"2024-09-16T05:20:29.870837+0000","last_peered":"2024-09-16T05:20:29.870837+0000","last_clean":"2024-09-16T05:20:29.870837+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:20:29.870837+0000","last_undegraded":"2024-09-16T05:20:29.870837+0000","last_fullsized":"2024-09-16T05:20:29.870837+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"667'593","last_scrub_stamp":"2024-09-16T05:20:20.620494+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.620494+0000","objects_scrubbed":2,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.942886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020560999999999999,"stat_sum":{"num_bytes":3949810,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":53,"num_read_kb":46839,"num_write":60,"num_write_kb":24043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":1915,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870683+0000","last_change":"2024-09-16T05:20:28.815834+0000","last_active":"2024-09-16T05:20:29.870683+0000","last_peered":"2024-09-16T05:20:29.870683+0000","last_clean":"2024-09-16T05:20:29.870683+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:20:29.870683+0000","last_undegraded":"2024-09-16T05:20:29.870683+0000","last_fullsized":"2024-09-16T05:20:29.870683+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:19:49.888753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.888753+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:08.014706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010177789999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":1920,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.805738+0000","last_change":"2024-09-16T05:20:28.806037+0000","last_active":"2024-09-16T05:20:28.805738+0000","last_peered":"2024-09-16T05:20:28.805738+0000","last_clean":"2024-09-16T05:20:28.805738+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:20:28.805738+0000","last_undegraded":"2024-09-16T05:20:28.805738+0000","last_fullsized":"2024-09-16T05:20:28.805738+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:20:20.396645+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.396645+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.420034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050226299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":747,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801788+0000","last_change":"2024-09-16T05:20:23.809915+0000","last_active":"2024-09-16T05:20:25.801788+0000","last_peered":"2024-09-16T05:20:25.801788+0000","last_clean":"2024-09-16T05:20:25.801788+0000","last_became_active":"2024-09-16T05:17:59.866037+0000","last_became_peered":"2024-09-16T05:17:59.866037+0000","last_unstale":"2024-09-16T05:20:25.801788+0000","last_undegraded":"2024-09-16T05:20:25.801788+0000","last_fullsized":"2024-09-16T05:20:25.801788+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:31.567784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:19:31.567784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:55.722805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017300499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":269,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.812501+0000","last_change":"2024-09-16T05:20:26.812792+0000","last_active":"2024-09-16T05:20:26.812501+0000","last_peered":"2024-09-16T05:20:26.812501+0000","last_clean":"2024-09-16T05:20:26.812501+0000","last_became_active":"2024-09-16T05:19:35.795592+0000","last_became_peered":"2024-09-16T05:19:35.795592+0000","last_unstale":"2024-09-16T05:20:26.812501+0000","last_undegraded":"2024-09-16T05:20:26.812501+0000","last_fullsized":"2024-09-16T05:20:26.812501+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:20:02.294024+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.294024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.400789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00140982,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.35","version":"677'745","reported_seq":3649,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.817784+0000","last_change":"2024-09-16T05:20:26.818129+0000","last_active":"2024-09-16T05:20:26.817784+0000","last_peered":"2024-09-16T05:20:26.817784+0000","last_clean":"2024-09-16T05:20:26.817784+0000","last_became_active":"2024-09-16T05:19:35.802234+0000","last_became_peered":"2024-09-16T05:19:35.802234+0000","last_unstale":"2024-09-16T05:20:26.817784+0000","last_undegraded":"2024-09-16T05:20:26.817784+0000","last_fullsized":"2024-09-16T05:20:26.817784+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'675","last_scrub_stamp":"2024-09-16T05:19:11.894332+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.894332+0000","objects_scrubbed":8,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0067402750000000004,"stat_sum":{"num_bytes":12263488,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":134,"num_read_kb":130415,"num_write":87,"num_write_kb":40632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3771362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.34","version":"674'591","reported_seq":3010,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.157922+0000","last_change":"2024-09-16T05:20:23.809853+0000","last_active":"2024-09-16T05:20:25.157922+0000","last_peered":"2024-09-16T05:20:25.157922+0000","last_clean":"2024-09-16T05:20:25.157922+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:20:25.157922+0000","last_undegraded":"2024-09-16T05:20:25.157922+0000","last_fullsized":"2024-09-16T05:20:25.157922+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"602'482","last_scrub_stamp":"2024-09-16T05:19:12.637489+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.637489+0000","objects_scrubbed":8,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:26.960724+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0028600679999999999,"stat_sum":{"num_bytes":18684074,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":154,"num_read_kb":173311,"num_write":101,"num_write_kb":39821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"2.33","version":"181'15","reported_seq":1986,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.808557+0000","last_change":"2024-09-16T05:20:26.808867+0000","last_active":"2024-09-16T05:20:26.808557+0000","last_peered":"2024-09-16T05:20:26.808557+0000","last_clean":"2024-09-16T05:20:26.808557+0000","last_became_active":"2024-09-16T05:19:35.789794+0000","last_became_peered":"2024-09-16T05:19:35.789794+0000","last_unstale":"2024-09-16T05:20:26.808557+0000","last_undegraded":"2024-09-16T05:20:26.808557+0000","last_fullsized":"2024-09-16T05:20:26.808557+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:16.451459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:16.451459+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.511342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056268500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.32","version":"674'229","reported_seq":2207,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.817057+0000","last_change":"2024-09-16T05:20:26.817057+0000","last_active":"2024-09-16T05:20:26.817057+0000","last_peered":"2024-09-16T05:20:26.817057+0000","last_clean":"2024-09-16T05:20:26.817057+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:20:26.817057+0000","last_undegraded":"2024-09-16T05:20:26.817057+0000","last_fullsized":"2024-09-16T05:20:26.817057+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"671'227","last_scrub_stamp":"2024-09-16T05:20:23.405472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.405472+0000","objects_scrubbed":2,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.355290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053593729999999997,"stat_sum":{"num_bytes":6202179,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":101970,"num_write":44,"num_write_kb":20642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.31","version":"677'282","reported_seq":2859,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:30.037773+0000","last_change":"2024-09-16T05:20:28.809813+0000","last_active":"2024-09-16T05:20:30.037773+0000","last_peered":"2024-09-16T05:20:30.037773+0000","last_clean":"2024-09-16T05:20:30.037773+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:20:30.037773+0000","last_undegraded":"2024-09-16T05:20:30.037773+0000","last_fullsized":"2024-09-16T05:20:30.037773+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'268","last_scrub_stamp":"2024-09-16T05:20:03.547514+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.547514+0000","objects_scrubbed":2,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:09.995613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001074146,"stat_sum":{"num_bytes":3680626,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":116,"num_read_kb":119771,"num_write":39,"num_write_kb":19090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":735,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801198+0000","last_change":"2024-09-16T05:20:23.807538+0000","last_active":"2024-09-16T05:20:25.801198+0000","last_peered":"2024-09-16T05:20:25.801198+0000","last_clean":"2024-09-16T05:20:25.801198+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:20:25.801198+0000","last_undegraded":"2024-09-16T05:20:25.801198+0000","last_fullsized":"2024-09-16T05:20:25.801198+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:20:16.443442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.443442+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.543953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023386,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":984,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.157057+0000","last_change":"2024-09-16T05:20:18.091859+0000","last_active":"2024-09-16T05:20:25.157057+0000","last_peered":"2024-09-16T05:20:25.157057+0000","last_clean":"2024-09-16T05:20:25.157057+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:20:25.157057+0000","last_undegraded":"2024-09-16T05:20:25.157057+0000","last_fullsized":"2024-09-16T05:20:25.157057+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.091747+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.091747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.641366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":2540,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.809441+0000","last_change":"2024-09-16T05:20:26.810224+0000","last_active":"2024-09-16T05:20:26.809441+0000","last_peered":"2024-09-16T05:20:26.809441+0000","last_clean":"2024-09-16T05:20:26.809441+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:20:26.809441+0000","last_undegraded":"2024-09-16T05:20:26.809441+0000","last_fullsized":"2024-09-16T05:20:26.809441+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:20:03.076376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.076376+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:14.349215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095940499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1153,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.810665+0000","last_change":"2024-09-16T05:19:36.252852+0000","last_active":"2024-09-16T05:20:26.810665+0000","last_peered":"2024-09-16T05:20:26.810665+0000","last_clean":"2024-09-16T05:20:26.810665+0000","last_became_active":"2024-09-16T05:19:35.788516+0000","last_became_peered":"2024-09-16T05:19:35.788516+0000","last_unstale":"2024-09-16T05:20:26.810665+0000","last_undegraded":"2024-09-16T05:20:26.810665+0000","last_fullsized":"2024-09-16T05:20:26.810665+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:36.252730+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.252730+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.441323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":2471,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.805789+0000","last_change":"2024-09-16T05:20:28.806171+0000","last_active":"2024-09-16T05:20:28.805789+0000","last_peered":"2024-09-16T05:20:28.805789+0000","last_clean":"2024-09-16T05:20:28.805789+0000","last_became_active":"2024-09-16T05:19:35.791894+0000","last_became_peered":"2024-09-16T05:19:35.791894+0000","last_unstale":"2024-09-16T05:20:28.805789+0000","last_undegraded":"2024-09-16T05:20:28.805789+0000","last_fullsized":"2024-09-16T05:20:28.805789+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:11.306770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.306770+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058850299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"1.c","version":"35'139","reported_seq":939,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.808845+0000","last_change":"2024-09-16T05:20:18.363196+0000","last_active":"2024-09-16T05:20:27.808845+0000","last_peered":"2024-09-16T05:20:27.808845+0000","last_clean":"2024-09-16T05:20:27.808845+0000","last_became_active":"2024-09-16T05:15:56.716776+0000","last_became_peered":"2024-09-16T05:15:56.716776+0000","last_unstale":"2024-09-16T05:20:27.808845+0000","last_undegraded":"2024-09-16T05:20:27.808845+0000","last_fullsized":"2024-09-16T05:20:27.808845+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.363111+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.363111+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:36.359138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"666'405","reported_seq":3258,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.806525+0000","last_change":"2024-09-16T05:20:28.806736+0000","last_active":"2024-09-16T05:20:28.806525+0000","last_peered":"2024-09-16T05:20:28.806525+0000","last_clean":"2024-09-16T05:20:28.806525+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:20:28.806525+0000","last_undegraded":"2024-09-16T05:20:28.806525+0000","last_fullsized":"2024-09-16T05:20:28.806525+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'384","last_scrub_stamp":"2024-09-16T05:19:15.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.295498+0000","objects_scrubbed":6,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:44.750370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090170400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":57662,"num_write":37,"num_write_kb":14770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"1.d","version":"35'139","reported_seq":1093,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.803450+0000","last_change":"2024-09-16T05:19:12.454525+0000","last_active":"2024-09-16T05:20:26.803450+0000","last_peered":"2024-09-16T05:20:26.803450+0000","last_clean":"2024-09-16T05:20:26.803450+0000","last_became_active":"2024-09-16T05:15:56.717962+0000","last_became_peered":"2024-09-16T05:15:56.717962+0000","last_unstale":"2024-09-16T05:20:26.803450+0000","last_undegraded":"2024-09-16T05:20:26.803450+0000","last_fullsized":"2024-09-16T05:20:26.803450+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:12.454446+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.454446+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:32.180360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":2959,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.829120+0000","last_change":"2024-09-16T05:20:26.829358+0000","last_active":"2024-09-16T05:20:26.829120+0000","last_peered":"2024-09-16T05:20:26.829120+0000","last_clean":"2024-09-16T05:20:26.829120+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:20:26.829120+0000","last_undegraded":"2024-09-16T05:20:26.829120+0000","last_fullsized":"2024-09-16T05:20:26.829120+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:11.209692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.209692+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00146159,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":1089,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.794230+0000","last_change":"2024-09-16T05:19:59.197776+0000","last_active":"2024-09-16T05:20:25.794230+0000","last_peered":"2024-09-16T05:20:25.794230+0000","last_clean":"2024-09-16T05:20:25.794230+0000","last_became_active":"2024-09-16T05:15:56.718549+0000","last_became_peered":"2024-09-16T05:15:56.718549+0000","last_unstale":"2024-09-16T05:20:25.794230+0000","last_undegraded":"2024-09-16T05:20:25.794230+0000","last_fullsized":"2024-09-16T05:20:25.794230+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:59.197664+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:59.197664+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.493240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"677'604","reported_seq":3461,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870924+0000","last_change":"2024-09-16T05:20:28.820774+0000","last_active":"2024-09-16T05:20:29.870924+0000","last_peered":"2024-09-16T05:20:29.870924+0000","last_clean":"2024-09-16T05:20:29.870924+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:20:29.870924+0000","last_undegraded":"2024-09-16T05:20:29.870924+0000","last_fullsized":"2024-09-16T05:20:29.870924+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"617'576","last_scrub_stamp":"2024-09-16T05:19:26.973559+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:19:26.973559+0000","objects_scrubbed":4,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:38.374366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0050531229999999996,"stat_sum":{"num_bytes":9125264,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":52779,"num_write":49,"num_write_kb":25445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":1072,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.155737+0000","last_change":"2024-09-16T05:19:29.650332+0000","last_active":"2024-09-16T05:20:25.155737+0000","last_peered":"2024-09-16T05:20:25.155737+0000","last_clean":"2024-09-16T05:20:25.155737+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:20:25.155737+0000","last_undegraded":"2024-09-16T05:20:25.155737+0000","last_fullsized":"2024-09-16T05:20:25.155737+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:29.650209+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.650209+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:34.196552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3089,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.816430+0000","last_change":"2024-09-16T05:20:26.816642+0000","last_active":"2024-09-16T05:20:26.816430+0000","last_peered":"2024-09-16T05:20:26.816430+0000","last_clean":"2024-09-16T05:20:26.816430+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:20:26.816430+0000","last_undegraded":"2024-09-16T05:20:26.816430+0000","last_fullsized":"2024-09-16T05:20:26.816430+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:19:10.225027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.225027+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:36.770636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0049958149999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1616,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.810872+0000","last_change":"2024-09-16T05:19:33.235762+0000","last_active":"2024-09-16T05:20:26.810872+0000","last_peered":"2024-09-16T05:20:26.810872+0000","last_clean":"2024-09-16T05:20:26.810872+0000","last_became_active":"2024-09-16T05:15:56.714987+0000","last_became_peered":"2024-09-16T05:15:56.714987+0000","last_unstale":"2024-09-16T05:20:26.810872+0000","last_undegraded":"2024-09-16T05:20:26.810872+0000","last_fullsized":"2024-09-16T05:20:26.810872+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:33.235681+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:19:33.235681+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.500872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"666'325","reported_seq":3156,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.802947+0000","last_change":"2024-09-16T05:20:26.803219+0000","last_active":"2024-09-16T05:20:26.802947+0000","last_peered":"2024-09-16T05:20:26.802947+0000","last_clean":"2024-09-16T05:20:26.802947+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:20:26.802947+0000","last_undegraded":"2024-09-16T05:20:26.802947+0000","last_fullsized":"2024-09-16T05:20:26.802947+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'325","last_scrub_stamp":"2024-09-16T05:20:23.462954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.462954+0000","objects_scrubbed":2,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.623140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052462199999999998,"stat_sum":{"num_bytes":4366553,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":35425,"num_write":51,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.3","version":"671'209","reported_seq":3044,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.157073+0000","last_change":"2024-09-16T05:20:23.808993+0000","last_active":"2024-09-16T05:20:25.157073+0000","last_peered":"2024-09-16T05:20:25.157073+0000","last_clean":"2024-09-16T05:20:25.157073+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:20:25.157073+0000","last_undegraded":"2024-09-16T05:20:25.157073+0000","last_fullsized":"2024-09-16T05:20:25.157073+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'205","last_scrub_stamp":"2024-09-16T05:20:19.137710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:19.137710+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.573760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014350680000000001,"stat_sum":{"num_bytes":3462440,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":111576,"num_write":43,"num_write_kb":18525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.0","version":"35'139","reported_seq":627,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.810695+0000","last_change":"2024-09-16T05:20:00.246338+0000","last_active":"2024-09-16T05:20:26.810695+0000","last_peered":"2024-09-16T05:20:26.810695+0000","last_clean":"2024-09-16T05:20:26.810695+0000","last_became_active":"2024-09-16T05:18:19.768392+0000","last_became_peered":"2024-09-16T05:18:19.768392+0000","last_unstale":"2024-09-16T05:20:26.810695+0000","last_undegraded":"2024-09-16T05:20:26.810695+0000","last_fullsized":"2024-09-16T05:20:26.810695+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:00.246242+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.246242+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.738053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":2824,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.157299+0000","last_change":"2024-09-16T05:20:23.807602+0000","last_active":"2024-09-16T05:20:25.157299+0000","last_peered":"2024-09-16T05:20:25.157299+0000","last_clean":"2024-09-16T05:20:25.157299+0000","last_became_active":"2024-09-16T05:19:35.791085+0000","last_became_peered":"2024-09-16T05:19:35.791085+0000","last_unstale":"2024-09-16T05:20:25.157299+0000","last_undegraded":"2024-09-16T05:20:25.157299+0000","last_fullsized":"2024-09-16T05:20:25.157299+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:02.847979+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.847979+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.318132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066067600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.1","version":"35'139","reported_seq":1173,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801101+0000","last_change":"2024-09-16T05:19:30.558195+0000","last_active":"2024-09-16T05:20:25.801101+0000","last_peered":"2024-09-16T05:20:25.801101+0000","last_clean":"2024-09-16T05:20:25.801101+0000","last_became_active":"2024-09-16T05:18:02.888512+0000","last_became_peered":"2024-09-16T05:18:02.888512+0000","last_unstale":"2024-09-16T05:20:25.801101+0000","last_undegraded":"2024-09-16T05:20:25.801101+0000","last_fullsized":"2024-09-16T05:20:25.801101+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.558069+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.558069+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.688000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1605,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.802963+0000","last_change":"2024-09-16T05:19:56.353335+0000","last_active":"2024-09-16T05:20:26.802963+0000","last_peered":"2024-09-16T05:20:26.802963+0000","last_clean":"2024-09-16T05:20:26.802963+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:20:26.802963+0000","last_undegraded":"2024-09-16T05:20:26.802963+0000","last_fullsized":"2024-09-16T05:20:26.802963+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:19:56.353243+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.079495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":2776,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.827723+0000","last_change":"2024-09-16T05:20:26.828079+0000","last_active":"2024-09-16T05:20:26.827723+0000","last_peered":"2024-09-16T05:20:26.827723+0000","last_clean":"2024-09-16T05:20:26.827723+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:20:26.827723+0000","last_undegraded":"2024-09-16T05:20:26.827723+0000","last_fullsized":"2024-09-16T05:20:26.827723+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:25.536951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:25.536951+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.986576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065442300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1612,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.808881+0000","last_change":"2024-09-16T05:19:10.336320+0000","last_active":"2024-09-16T05:20:27.808881+0000","last_peered":"2024-09-16T05:20:27.808881+0000","last_clean":"2024-09-16T05:20:27.808881+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:20:27.808881+0000","last_undegraded":"2024-09-16T05:20:27.808881+0000","last_fullsized":"2024-09-16T05:20:27.808881+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:10.336203+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:19:10.336203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:35.544788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":2856,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.807790+0000","last_change":"2024-09-16T05:20:26.808333+0000","last_active":"2024-09-16T05:20:26.807790+0000","last_peered":"2024-09-16T05:20:26.807790+0000","last_clean":"2024-09-16T05:20:26.807790+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:20:26.807790+0000","last_undegraded":"2024-09-16T05:20:26.807790+0000","last_fullsized":"2024-09-16T05:20:26.807790+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:51.365804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.365804+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:59.310751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053471650000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":2086,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.871032+0000","last_change":"2024-09-16T05:20:28.821104+0000","last_active":"2024-09-16T05:20:29.871032+0000","last_peered":"2024-09-16T05:20:29.871032+0000","last_clean":"2024-09-16T05:20:29.871032+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:20:29.871032+0000","last_undegraded":"2024-09-16T05:20:29.871032+0000","last_fullsized":"2024-09-16T05:20:29.871032+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:50.852123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.852123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.185918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.005375192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":1571,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.812032+0000","last_change":"2024-09-16T05:19:53.322398+0000","last_active":"2024-09-16T05:20:26.812032+0000","last_peered":"2024-09-16T05:20:26.812032+0000","last_clean":"2024-09-16T05:20:26.812032+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:20:26.812032+0000","last_undegraded":"2024-09-16T05:20:26.812032+0000","last_fullsized":"2024-09-16T05:20:26.812032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:53.322303+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.322303+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:11.892229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1250,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.829056+0000","last_change":"2024-09-16T05:20:26.829291+0000","last_active":"2024-09-16T05:20:26.829056+0000","last_peered":"2024-09-16T05:20:26.829056+0000","last_clean":"2024-09-16T05:20:26.829056+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:20:26.829056+0000","last_undegraded":"2024-09-16T05:20:26.829056+0000","last_fullsized":"2024-09-16T05:20:26.829056+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:53.382868+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:18:27.279121+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.382868+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:13.528470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014349199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":878,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801846+0000","last_change":"2024-09-16T05:19:13.533926+0000","last_active":"2024-09-16T05:20:25.801846+0000","last_peered":"2024-09-16T05:20:25.801846+0000","last_clean":"2024-09-16T05:20:25.801846+0000","last_became_active":"2024-09-16T05:18:02.897275+0000","last_became_peered":"2024-09-16T05:18:02.897275+0000","last_unstale":"2024-09-16T05:20:25.801846+0000","last_undegraded":"2024-09-16T05:20:25.801846+0000","last_fullsized":"2024-09-16T05:20:25.801846+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:19:13.533798+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"679'585","reported_seq":3435,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.808510+0000","last_change":"2024-09-16T05:20:28.808753+0000","last_active":"2024-09-16T05:20:28.808510+0000","last_peered":"2024-09-16T05:20:28.808510+0000","last_clean":"2024-09-16T05:20:28.808510+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:20:28.808510+0000","last_undegraded":"2024-09-16T05:20:28.808510+0000","last_fullsized":"2024-09-16T05:20:28.808510+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'568","last_scrub_stamp":"2024-09-16T05:19:54.247232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.247232+0000","objects_scrubbed":3,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:07.363223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025914340000000001,"stat_sum":{"num_bytes":7659108,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":107857,"num_write":39,"num_write_kb":12889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":822,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801359+0000","last_change":"2024-09-16T05:19:37.639763+0000","last_active":"2024-09-16T05:20:25.801359+0000","last_peered":"2024-09-16T05:20:25.801359+0000","last_clean":"2024-09-16T05:20:25.801359+0000","last_became_active":"2024-09-16T05:18:02.886933+0000","last_became_peered":"2024-09-16T05:18:02.886933+0000","last_unstale":"2024-09-16T05:20:25.801359+0000","last_undegraded":"2024-09-16T05:20:25.801359+0000","last_fullsized":"2024-09-16T05:20:25.801359+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:37.639669+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:37.639669+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:42.374370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":723,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:30.037837+0000","last_change":"2024-09-16T05:20:28.809626+0000","last_active":"2024-09-16T05:20:30.037837+0000","last_peered":"2024-09-16T05:20:30.037837+0000","last_clean":"2024-09-16T05:20:30.037837+0000","last_became_active":"2024-09-16T05:19:35.793136+0000","last_became_peered":"2024-09-16T05:19:35.793136+0000","last_unstale":"2024-09-16T05:20:30.037837+0000","last_undegraded":"2024-09-16T05:20:30.037837+0000","last_fullsized":"2024-09-16T05:20:30.037837+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:36.435156+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.435156+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:42.618989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00082952899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":858,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801431+0000","last_change":"2024-09-16T05:19:15.510805+0000","last_active":"2024-09-16T05:20:25.801431+0000","last_peered":"2024-09-16T05:20:25.801431+0000","last_clean":"2024-09-16T05:20:25.801431+0000","last_became_active":"2024-09-16T05:18:03.588064+0000","last_became_peered":"2024-09-16T05:18:03.588064+0000","last_unstale":"2024-09-16T05:20:25.801431+0000","last_undegraded":"2024-09-16T05:20:25.801431+0000","last_fullsized":"2024-09-16T05:20:25.801431+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:15.510717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.510717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:27.556487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":3018,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.806243+0000","last_change":"2024-09-16T05:20:28.806422+0000","last_active":"2024-09-16T05:20:28.806243+0000","last_peered":"2024-09-16T05:20:28.806243+0000","last_clean":"2024-09-16T05:20:28.806243+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:20:28.806243+0000","last_undegraded":"2024-09-16T05:20:28.806243+0000","last_fullsized":"2024-09-16T05:20:28.806243+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:19:29.221207+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.221207+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:53.352302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072244599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":1603,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156926+0000","last_change":"2024-09-16T05:19:30.613103+0000","last_active":"2024-09-16T05:20:25.156926+0000","last_peered":"2024-09-16T05:20:25.156926+0000","last_clean":"2024-09-16T05:20:25.156926+0000","last_became_active":"2024-09-16T05:14:39.908976+0000","last_became_peered":"2024-09-16T05:14:39.908976+0000","last_unstale":"2024-09-16T05:20:25.156926+0000","last_undegraded":"2024-09-16T05:20:25.156926+0000","last_fullsized":"2024-09-16T05:20:25.156926+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.613024+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.613024+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.741797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"677'618","reported_seq":3477,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.819050+0000","last_change":"2024-09-16T05:20:26.819264+0000","last_active":"2024-09-16T05:20:26.819050+0000","last_peered":"2024-09-16T05:20:26.819050+0000","last_clean":"2024-09-16T05:20:26.819050+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:20:26.819050+0000","last_undegraded":"2024-09-16T05:20:26.819050+0000","last_fullsized":"2024-09-16T05:20:26.819050+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"653'601","last_scrub_stamp":"2024-09-16T05:20:03.311152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.311152+0000","objects_scrubbed":3,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:07.029849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0073471539999999998,"stat_sum":{"num_bytes":6944569,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":127393,"num_write":54,"num_write_kb":33730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1603,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.802598+0000","last_change":"2024-09-16T05:20:20.802104+0000","last_active":"2024-09-16T05:20:26.802598+0000","last_peered":"2024-09-16T05:20:26.802598+0000","last_clean":"2024-09-16T05:20:26.802598+0000","last_became_active":"2024-09-16T05:20:20.801737+0000","last_became_peered":"2024-09-16T05:20:20.801737+0000","last_unstale":"2024-09-16T05:20:26.802598+0000","last_undegraded":"2024-09-16T05:20:26.802598+0000","last_fullsized":"2024-09-16T05:20:26.802598+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:02.384248+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.384248+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.409130+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1729,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.794098+0000","last_change":"2024-09-16T05:20:23.795538+0000","last_active":"2024-09-16T05:20:25.794098+0000","last_peered":"2024-09-16T05:20:25.794098+0000","last_clean":"2024-09-16T05:20:25.794098+0000","last_became_active":"2024-09-16T05:19:35.786308+0000","last_became_peered":"2024-09-16T05:19:35.786308+0000","last_unstale":"2024-09-16T05:20:25.794098+0000","last_undegraded":"2024-09-16T05:20:25.794098+0000","last_fullsized":"2024-09-16T05:20:25.794098+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:19:31.185579+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:31.185579+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:52.530015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000428066,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1608,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870626+0000","last_change":"2024-09-16T05:19:32.947634+0000","last_active":"2024-09-16T05:20:29.870626+0000","last_peered":"2024-09-16T05:20:29.870626+0000","last_clean":"2024-09-16T05:20:29.870626+0000","last_became_active":"2024-09-16T05:10:38.969552+0000","last_became_peered":"2024-09-16T05:10:38.969552+0000","last_unstale":"2024-09-16T05:20:29.870626+0000","last_undegraded":"2024-09-16T05:20:29.870626+0000","last_fullsized":"2024-09-16T05:20:29.870626+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:32.947528+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.947528+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.139142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1098,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:30.037437+0000","last_change":"2024-09-16T05:19:49.535280+0000","last_active":"2024-09-16T05:20:30.037437+0000","last_peered":"2024-09-16T05:20:30.037437+0000","last_clean":"2024-09-16T05:20:30.037437+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:20:30.037437+0000","last_undegraded":"2024-09-16T05:20:30.037437+0000","last_fullsized":"2024-09-16T05:20:30.037437+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:49.535206+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.535206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:17.718180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":735,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801201+0000","last_change":"2024-09-16T05:20:24.333496+0000","last_active":"2024-09-16T05:20:25.801201+0000","last_peered":"2024-09-16T05:20:25.801201+0000","last_clean":"2024-09-16T05:20:25.801201+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:20:25.801201+0000","last_undegraded":"2024-09-16T05:20:25.801201+0000","last_fullsized":"2024-09-16T05:20:25.801201+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:24.333369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:24.333369+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:32.117168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001134688,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1603,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156679+0000","last_change":"2024-09-16T05:19:48.737475+0000","last_active":"2024-09-16T05:20:25.156679+0000","last_peered":"2024-09-16T05:20:25.156679+0000","last_clean":"2024-09-16T05:20:25.156679+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:20:25.156679+0000","last_undegraded":"2024-09-16T05:20:25.156679+0000","last_fullsized":"2024-09-16T05:20:25.156679+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:48.737348+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.737348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:57.195160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":2467,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.871038+0000","last_change":"2024-09-16T05:20:28.816303+0000","last_active":"2024-09-16T05:20:29.871038+0000","last_peered":"2024-09-16T05:20:29.871038+0000","last_clean":"2024-09-16T05:20:29.871038+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:20:29.871038+0000","last_undegraded":"2024-09-16T05:20:29.871038+0000","last_fullsized":"2024-09-16T05:20:29.871038+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:36.009233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.009233+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:45.071682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001039309,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.14","version":"433'289","reported_seq":2739,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.871084+0000","last_change":"2024-09-16T05:20:28.819602+0000","last_active":"2024-09-16T05:20:29.871084+0000","last_peered":"2024-09-16T05:20:29.871084+0000","last_clean":"2024-09-16T05:20:29.871084+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:20:29.871084+0000","last_undegraded":"2024-09-16T05:20:29.871084+0000","last_fullsized":"2024-09-16T05:20:29.871084+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:19:12.999550+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:12.999550+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040787530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.15","version":"679'637","reported_seq":3560,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870884+0000","last_change":"2024-09-16T05:20:28.819224+0000","last_active":"2024-09-16T05:20:29.870884+0000","last_peered":"2024-09-16T05:20:29.870884+0000","last_clean":"2024-09-16T05:20:29.870884+0000","last_became_active":"2024-09-16T05:19:35.801137+0000","last_became_peered":"2024-09-16T05:19:35.801137+0000","last_unstale":"2024-09-16T05:20:29.870884+0000","last_undegraded":"2024-09-16T05:20:29.870884+0000","last_fullsized":"2024-09-16T05:20:29.870884+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"617'599","last_scrub_stamp":"2024-09-16T05:19:28.020270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.020270+0000","objects_scrubbed":4,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:41.409407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037725699999999998,"stat_sum":{"num_bytes":3964937,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":5506324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.16","version":"666'296","reported_seq":2757,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.156576+0000","last_change":"2024-09-16T05:20:23.808837+0000","last_active":"2024-09-16T05:20:25.156576+0000","last_peered":"2024-09-16T05:20:25.156576+0000","last_clean":"2024-09-16T05:20:25.156576+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:20:25.156576+0000","last_undegraded":"2024-09-16T05:20:25.156576+0000","last_fullsized":"2024-09-16T05:20:25.156576+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"665'294","last_scrub_stamp":"2024-09-16T05:20:15.010850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.010850+0000","objects_scrubbed":4,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.414078+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0015138899999999999,"stat_sum":{"num_bytes":7403700,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":57070,"num_write":50,"num_write_kb":16677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e7","length":"1"},{"start":"104","length":"1"},{"start":"10e","length":"4"}]},{"pgid":"1.14","version":"35'139","reported_seq":927,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.827479+0000","last_change":"2024-09-16T05:18:46.335095+0000","last_active":"2024-09-16T05:20:26.827479+0000","last_peered":"2024-09-16T05:20:26.827479+0000","last_clean":"2024-09-16T05:20:26.827479+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:20:26.827479+0000","last_undegraded":"2024-09-16T05:20:26.827479+0000","last_fullsized":"2024-09-16T05:20:26.827479+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:18:46.335030+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:18:46.335030+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:31.492721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":485,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801461+0000","last_change":"2024-09-16T05:20:23.809199+0000","last_active":"2024-09-16T05:20:25.801461+0000","last_peered":"2024-09-16T05:20:25.801461+0000","last_clean":"2024-09-16T05:20:25.801461+0000","last_became_active":"2024-09-16T05:19:35.784084+0000","last_became_peered":"2024-09-16T05:19:35.784084+0000","last_unstale":"2024-09-16T05:20:25.801461+0000","last_undegraded":"2024-09-16T05:20:25.801461+0000","last_fullsized":"2024-09-16T05:20:25.801461+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:20:20.393358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.393358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:24.632271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00229188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.18","version":"181'57","reported_seq":2849,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870853+0000","last_change":"2024-09-16T05:20:28.816424+0000","last_active":"2024-09-16T05:20:29.870853+0000","last_peered":"2024-09-16T05:20:29.870853+0000","last_clean":"2024-09-16T05:20:29.870853+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:20:29.870853+0000","last_undegraded":"2024-09-16T05:20:29.870853+0000","last_fullsized":"2024-09-16T05:20:29.870853+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:19:47.951135+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:19:47.951135+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:04.227221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001377135,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.19","version":"182'60","reported_seq":2454,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.809214+0000","last_change":"2024-09-16T05:20:27.809411+0000","last_active":"2024-09-16T05:20:27.809214+0000","last_peered":"2024-09-16T05:20:27.809214+0000","last_clean":"2024-09-16T05:20:27.809214+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:20:27.809214+0000","last_undegraded":"2024-09-16T05:20:27.809214+0000","last_fullsized":"2024-09-16T05:20:27.809214+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:20:17.403045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.403045+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.218144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037962800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.1a","version":"679'1022","reported_seq":3826,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.811164+0000","last_change":"2024-09-16T05:20:28.811399+0000","last_active":"2024-09-16T05:20:28.811164+0000","last_peered":"2024-09-16T05:20:28.811164+0000","last_clean":"2024-09-16T05:20:28.811164+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:20:28.811164+0000","last_undegraded":"2024-09-16T05:20:28.811164+0000","last_fullsized":"2024-09-16T05:20:28.811164+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"653'957","last_scrub_stamp":"2024-09-16T05:20:03.383437+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.383437+0000","objects_scrubbed":11,"log_size":1022,"log_dups_size":0,"ondisk_log_size":1022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:12.778527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0055545020000000002,"stat_sum":{"num_bytes":25849704,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":167,"num_read_kb":148606,"num_write":170,"num_write_kb":62122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.1b","version":"677'630","reported_seq":3483,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.816086+0000","last_change":"2024-09-16T05:20:26.816353+0000","last_active":"2024-09-16T05:20:26.816086+0000","last_peered":"2024-09-16T05:20:26.816086+0000","last_clean":"2024-09-16T05:20:26.816086+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:20:26.816086+0000","last_undegraded":"2024-09-16T05:20:26.816086+0000","last_fullsized":"2024-09-16T05:20:26.816086+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"666'612","last_scrub_stamp":"2024-09-16T05:20:16.258329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.258329+0000","objects_scrubbed":4,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.997193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053052799999999999,"stat_sum":{"num_bytes":11564336,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":94,"num_read_kb":89721,"num_write":102,"num_write_kb":33060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.1c","version":"664'452","reported_seq":2974,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.802009+0000","last_change":"2024-09-16T05:20:23.808221+0000","last_active":"2024-09-16T05:20:25.802009+0000","last_peered":"2024-09-16T05:20:25.802009+0000","last_clean":"2024-09-16T05:20:25.802009+0000","last_became_active":"2024-09-16T05:18:02.890972+0000","last_became_peered":"2024-09-16T05:18:02.890972+0000","last_unstale":"2024-09-16T05:20:25.802009+0000","last_undegraded":"2024-09-16T05:20:25.802009+0000","last_fullsized":"2024-09-16T05:20:25.802009+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"620'444","last_scrub_stamp":"2024-09-16T05:19:51.567043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.567043+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:20.840931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014105910000000001,"stat_sum":{"num_bytes":3294370,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":10803,"num_write":28,"num_write_kb":13180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.1d","version":"666'1182","reported_seq":4238,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801937+0000","last_change":"2024-09-16T05:20:23.809322+0000","last_active":"2024-09-16T05:20:25.801937+0000","last_peered":"2024-09-16T05:20:25.801937+0000","last_clean":"2024-09-16T05:20:25.801937+0000","last_became_active":"2024-09-16T05:18:03.891103+0000","last_became_peered":"2024-09-16T05:18:03.891103+0000","last_unstale":"2024-09-16T05:20:25.801937+0000","last_undegraded":"2024-09-16T05:20:25.801937+0000","last_fullsized":"2024-09-16T05:20:25.801937+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"624'1113","last_scrub_stamp":"2024-09-16T05:19:35.593327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:35.593327+0000","objects_scrubbed":9,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:53.246158+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0023436300000000002,"stat_sum":{"num_bytes":14250762,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":246,"num_read_kb":196095,"num_write":229,"num_write_kb":73447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7155994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.1e","version":"677'468","reported_seq":3339,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.809536+0000","last_change":"2024-09-16T05:20:27.809720+0000","last_active":"2024-09-16T05:20:27.809536+0000","last_peered":"2024-09-16T05:20:27.809536+0000","last_clean":"2024-09-16T05:20:27.809536+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:20:27.809536+0000","last_undegraded":"2024-09-16T05:20:27.809536+0000","last_fullsized":"2024-09-16T05:20:27.809536+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"671'464","last_scrub_stamp":"2024-09-16T05:20:21.358700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:21.358700+0000","objects_scrubbed":3,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.219910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044219300000000001,"stat_sum":{"num_bytes":3007609,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":71847,"num_write":25,"num_write_kb":11620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.1f","version":"284'221","reported_seq":3074,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:30.037830+0000","last_change":"2024-09-16T05:20:28.809949+0000","last_active":"2024-09-16T05:20:30.037830+0000","last_peered":"2024-09-16T05:20:30.037830+0000","last_clean":"2024-09-16T05:20:30.037830+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:20:30.037830+0000","last_undegraded":"2024-09-16T05:20:30.037830+0000","last_fullsized":"2024-09-16T05:20:30.037830+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:20:17.544083+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.544083+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:26.801428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001175733,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.20","version":"677'307","reported_seq":3167,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.806422+0000","last_change":"2024-09-16T05:20:26.807410+0000","last_active":"2024-09-16T05:20:26.806422+0000","last_peered":"2024-09-16T05:20:26.806422+0000","last_clean":"2024-09-16T05:20:26.806422+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:20:26.806422+0000","last_undegraded":"2024-09-16T05:20:26.806422+0000","last_fullsized":"2024-09-16T05:20:26.806422+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'290","last_scrub_stamp":"2024-09-16T05:20:03.351330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.351330+0000","objects_scrubbed":3,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.928969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0046326850000000001,"stat_sum":{"num_bytes":6345760,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":144671,"num_write":58,"num_write_kb":21452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":2775,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828162+0000","last_change":"2024-09-16T05:20:26.828400+0000","last_active":"2024-09-16T05:20:26.828162+0000","last_peered":"2024-09-16T05:20:26.828162+0000","last_clean":"2024-09-16T05:20:26.828162+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:20:26.828162+0000","last_undegraded":"2024-09-16T05:20:26.828162+0000","last_fullsized":"2024-09-16T05:20:26.828162+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:02.565969+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.565969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:10.707965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093526500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":735,"reported_epoch":676,"state":"active+clean","last_fresh":"2024-09-16T05:20:25.801231+0000","last_change":"2024-09-16T05:20:23.809600+0000","last_active":"2024-09-16T05:20:25.801231+0000","last_peered":"2024-09-16T05:20:25.801231+0000","last_clean":"2024-09-16T05:20:25.801231+0000","last_became_active":"2024-09-16T05:17:59.866047+0000","last_became_peered":"2024-09-16T05:17:59.866047+0000","last_unstale":"2024-09-16T05:20:25.801231+0000","last_undegraded":"2024-09-16T05:20:25.801231+0000","last_fullsized":"2024-09-16T05:20:25.801231+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:15.452705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.452705+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.285695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010043820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":2327,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-09-16T05:20:27.809309+0000","last_change":"2024-09-16T05:20:27.809613+0000","last_active":"2024-09-16T05:20:27.809309+0000","last_peered":"2024-09-16T05:20:27.809309+0000","last_clean":"2024-09-16T05:20:27.809309+0000","last_became_active":"2024-09-16T05:20:20.801064+0000","last_became_peered":"2024-09-16T05:20:20.801064+0000","last_unstale":"2024-09-16T05:20:27.809309+0000","last_undegraded":"2024-09-16T05:20:27.809309+0000","last_fullsized":"2024-09-16T05:20:27.809309+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:20:00.415257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.415257+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.527133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056295900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":2143,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870966+0000","last_change":"2024-09-16T05:20:28.816788+0000","last_active":"2024-09-16T05:20:29.870966+0000","last_peered":"2024-09-16T05:20:29.870966+0000","last_clean":"2024-09-16T05:20:29.870966+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:20:29.870966+0000","last_undegraded":"2024-09-16T05:20:29.870966+0000","last_fullsized":"2024-09-16T05:20:29.870966+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:57.859756+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:19:57.859756+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:09.447103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001501304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.25","version":"679'1108","reported_seq":3870,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.814557+0000","last_change":"2024-09-16T05:20:28.814828+0000","last_active":"2024-09-16T05:20:28.814557+0000","last_peered":"2024-09-16T05:20:28.814557+0000","last_clean":"2024-09-16T05:20:28.814557+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:20:28.814557+0000","last_undegraded":"2024-09-16T05:20:28.814557+0000","last_fullsized":"2024-09-16T05:20:28.814557+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"622'977","last_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.332491+0000","objects_scrubbed":9,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:37.947396+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0.008487596,"stat_sum":{"num_bytes":27092020,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":188,"num_read_kb":197711,"num_write":167,"num_write_kb":66134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":2113,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.810073+0000","last_change":"2024-09-16T05:20:26.810328+0000","last_active":"2024-09-16T05:20:26.810073+0000","last_peered":"2024-09-16T05:20:26.810073+0000","last_clean":"2024-09-16T05:20:26.810073+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:20:26.810073+0000","last_undegraded":"2024-09-16T05:20:26.810073+0000","last_fullsized":"2024-09-16T05:20:26.810073+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:07.483055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.483055+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:28.339992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062933349999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":2125,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.809250+0000","last_change":"2024-09-16T05:20:26.809590+0000","last_active":"2024-09-16T05:20:26.809250+0000","last_peered":"2024-09-16T05:20:26.809250+0000","last_clean":"2024-09-16T05:20:26.809250+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:20:26.809250+0000","last_undegraded":"2024-09-16T05:20:26.809250+0000","last_fullsized":"2024-09-16T05:20:26.809250+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:11.499561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.499561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.405687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058943429999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1138,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.827660+0000","last_change":"2024-09-16T05:20:26.828082+0000","last_active":"2024-09-16T05:20:26.827660+0000","last_peered":"2024-09-16T05:20:26.827660+0000","last_clean":"2024-09-16T05:20:26.827660+0000","last_became_active":"2024-09-16T05:19:35.785220+0000","last_became_peered":"2024-09-16T05:19:35.785220+0000","last_unstale":"2024-09-16T05:20:26.827660+0000","last_undegraded":"2024-09-16T05:20:26.827660+0000","last_fullsized":"2024-09-16T05:20:26.827660+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:19:27.303070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:27.303070+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:33.735002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068899800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.29","version":"677'988","reported_seq":3844,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.811791+0000","last_change":"2024-09-16T05:20:26.812158+0000","last_active":"2024-09-16T05:20:26.811791+0000","last_peered":"2024-09-16T05:20:26.811791+0000","last_clean":"2024-09-16T05:20:26.811791+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:20:26.811791+0000","last_undegraded":"2024-09-16T05:20:26.811791+0000","last_fullsized":"2024-09-16T05:20:26.811791+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"621'852","last_scrub_stamp":"2024-09-16T05:19:32.048085+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.048085+0000","objects_scrubbed":8,"log_size":988,"log_dups_size":0,"ondisk_log_size":988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:34.417652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0042748109999999999,"stat_sum":{"num_bytes":25096059,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":255,"num_read_kb":246175,"num_write":186,"num_write_kb":67059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":3115,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.807544+0000","last_change":"2024-09-16T05:20:26.807846+0000","last_active":"2024-09-16T05:20:26.807544+0000","last_peered":"2024-09-16T05:20:26.807544+0000","last_clean":"2024-09-16T05:20:26.807544+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:20:26.807544+0000","last_undegraded":"2024-09-16T05:20:26.807544+0000","last_fullsized":"2024-09-16T05:20:26.807544+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:19:29.329416+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.329416+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:59.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050244959999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2b","version":"666'332","reported_seq":3086,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.828844+0000","last_change":"2024-09-16T05:20:26.829026+0000","last_active":"2024-09-16T05:20:26.828844+0000","last_peered":"2024-09-16T05:20:26.828844+0000","last_clean":"2024-09-16T05:20:26.828844+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:20:26.828844+0000","last_undegraded":"2024-09-16T05:20:26.828844+0000","last_fullsized":"2024-09-16T05:20:26.828844+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"666'332","last_scrub_stamp":"2024-09-16T05:20:18.508468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.508468+0000","objects_scrubbed":1,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.883681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013043390000000001,"stat_sum":{"num_bytes":2672478,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":79889,"num_write":47,"num_write_kb":21198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":2965,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-09-16T05:20:28.806800+0000","last_change":"2024-09-16T05:20:28.807074+0000","last_active":"2024-09-16T05:20:28.806800+0000","last_peered":"2024-09-16T05:20:28.806800+0000","last_clean":"2024-09-16T05:20:28.806800+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:20:28.806800+0000","last_undegraded":"2024-09-16T05:20:28.806800+0000","last_fullsized":"2024-09-16T05:20:28.806800+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:19:51.201665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.201665+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.577066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001195654,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]},{"pgid":"2.2d","version":"666'459","reported_seq":3285,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.808932+0000","last_change":"2024-09-16T05:20:26.809908+0000","last_active":"2024-09-16T05:20:26.808932+0000","last_peered":"2024-09-16T05:20:26.808932+0000","last_clean":"2024-09-16T05:20:26.808932+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:20:26.808932+0000","last_undegraded":"2024-09-16T05:20:26.808932+0000","last_fullsized":"2024-09-16T05:20:26.808932+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"648'457","last_scrub_stamp":"2024-09-16T05:20:15.223777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.223777+0000","objects_scrubbed":2,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:15.497568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063157839999999996,"stat_sum":{"num_bytes":2823741,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":87,"num_read_kb":88579,"num_write":47,"num_write_kb":22068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2e","version":"677'616","reported_seq":3397,"reported_epoch":677,"state":"active+clean","last_fresh":"2024-09-16T05:20:26.812309+0000","last_change":"2024-09-16T05:20:26.812309+0000","last_active":"2024-09-16T05:20:26.812309+0000","last_peered":"2024-09-16T05:20:26.812309+0000","last_clean":"2024-09-16T05:20:26.812309+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:20:26.812309+0000","last_undegraded":"2024-09-16T05:20:26.812309+0000","last_fullsized":"2024-09-16T05:20:26.812309+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"617'552","last_scrub_stamp":"2024-09-16T05:19:28.348887+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.348887+0000","objects_scrubbed":7,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:46.054638+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0.0082899770000000005,"stat_sum":{"num_bytes":13301925,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":80,"num_read_kb":55119,"num_write":86,"num_write_kb":33834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"107","length":"1"},{"start":"10e","length":"2"},{"start":"111","length":"1"}]},{"pgid":"2.2f","version":"679'584","reported_seq":3468,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-09-16T05:20:29.870986+0000","last_change":"2024-09-16T05:20:28.823118+0000","last_active":"2024-09-16T05:20:29.870986+0000","last_peered":"2024-09-16T05:20:29.870986+0000","last_clean":"2024-09-16T05:20:29.870986+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:20:29.870986+0000","last_undegraded":"2024-09-16T05:20:29.870986+0000","last_fullsized":"2024-09-16T05:20:29.870986+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"650'553","last_scrub_stamp":"2024-09-16T05:20:00.804910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.804910+0000","objects_scrubbed":6,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.331579+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0076035950000000003,"stat_sum":{"num_bytes":10084662,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":108,"num_read_kb":77807,"num_write":97,"num_write_kb":39298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"107","length":"1"},{"start":"10d","length":"3"},{"start":"111","length":"1"},{"start":"114","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":283369950,"num_objects":132,"num_object_clones":86,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":14,"num_read":3219,"num_read_kb":3190773,"num_write":2341,"num_write_kb":929327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":47028772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":375586816,"data_stored":478311707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24365,"ondisk_log_size":24365,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089481,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110164,"kb_used_data":56768,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93634988,"statfs":{"total":95995035648,"available":95882227712,"internally_reserved":0,"allocated":58130432,"data_stored":64560127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285709,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78312,"kb_used_data":47124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93666840,"statfs":{"total":95995035648,"available":95914844160,"internally_reserved":0,"allocated":48254976,"data_stored":53486620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514647,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95164,"kb_used_data":37604,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93649988,"statfs":{"total":95995035648,"available":95897587712,"internally_reserved":0,"allocated":38506496,"data_stored":46356204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841686,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80340,"kb_used_data":49144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93664812,"statfs":{"total":95995035648,"available":95912767488,"internally_reserved":0,"allocated":50323456,"data_stored":55573977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103324,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105968,"kb_used_data":48416,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93639184,"statfs":{"total":95995035648,"available":95886524416,"internally_reserved":0,"allocated":49577984,"data_stored":54868142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397667,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84684,"kb_used_data":49332,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93660468,"statfs":{"total":95995035648,"available":95908319232,"internally_reserved":0,"allocated":50515968,"data_stored":59465888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757413,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65624,"kb_used_data":38588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93679528,"statfs":{"total":95995035648,"available":95927836672,"internally_reserved":0,"allocated":39514112,"data_stored":48166101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920939,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49540,"kb_used_data":22508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93695612,"statfs":{"total":95995035648,"available":95944306688,"internally_reserved":0,"allocated":23048192,"data_stored":23769814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117168,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87200,"kb_used_data":51852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93657952,"statfs":{"total":95995035648,"available":95905742848,"internally_reserved":0,"allocated":53096448,"data_stored":71858981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240455,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53924,"kb_used_data":17488,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93691228,"statfs":{"total":95995035648,"available":95939817472,"internally_reserved":0,"allocated":17907712,"data_stored":16253623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738560,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48368,"kb_used_data":17172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93696784,"statfs":{"total":95995035648,"available":95945506816,"internally_reserved":0,"allocated":17584128,"data_stored":19047247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9052160,"data_stored":13275819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18554880,"data_stored":23395830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9461760,"data_stored":10564131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":44589056,"data_stored":66104497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14893056,"data_stored":18359970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31154176,"data_stored":42555009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42336256,"data_stored":54039084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":43061248,"data_stored":51756111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":42131456,"data_stored":50138701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":30302208,"data_stored":40912456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":40099840,"data_stored":48076776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":49950720,"data_stored":59133323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:20:32.694 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:20:32.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-09-16T05:20:32.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:32.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:32.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:32.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:32.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:32.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:32.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:32.893 INFO:tasks.rados.rados.2.smithi204.stdout:2669: done (1 left) 2024-09-16T05:20:32.893 INFO:tasks.rados.rados.2.smithi204.stdout:2670: write (excl) oid 10 current snap is 279 2024-09-16T05:20:32.893 INFO:tasks.rados.rados.2.smithi204.stdout:2668: finishing write tid 1 to smithi20457400-42 2024-09-16T05:20:32.893 INFO:tasks.rados.rados.2.smithi204.stdout:2670: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2024-09-16T05:20:32.900 INFO:tasks.rados.rados.2.smithi204.stdout:2670: writing smithi20457400-10 from 699422 to 1112748 tid 1 2024-09-16T05:20:32.907 INFO:tasks.rados.rados.2.smithi204.stdout:2670: writing smithi20457400-10 from 1557848 to 2239165 tid 2 2024-09-16T05:20:32.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:32 smithi032 ceph-mon[27388]: from='client.16770 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:32.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:32 smithi032 ceph-mon[27388]: osdmap e682: 12 total, 12 up, 11 in 2024-09-16T05:20:32.915 INFO:tasks.rados.rados.2.smithi204.stdout:2670: writing smithi20457400-10 from 3012504 to 3760887 tid 3 2024-09-16T05:20:32.916 INFO:tasks.rados.rados.2.smithi204.stdout:2671: read oid 2 snap 278 2024-09-16T05:20:32.916 INFO:tasks.rados.rados.2.smithi204.stdout:2671: expect deleted 2024-09-16T05:20:32.916 INFO:tasks.rados.rados.2.smithi204.stdout:2672: read oid 14 snap 263 2024-09-16T05:20:32.916 INFO:tasks.rados.rados.2.smithi204.stdout:2672: expect (ObjNum 896 snap 262 seq_num 896) 2024-09-16T05:20:32.916 INFO:tasks.rados.rados.2.smithi204.stdout:2673: snap_remove snap 264 2024-09-16T05:20:32.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:33.000 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:33.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:33.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:32 smithi204 ceph-mon[30933]: from='client.16770 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:33.088 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:32 smithi204 ceph-mon[30933]: osdmap e682: 12 total, 12 up, 11 in 2024-09-16T05:20:33.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:33.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:20:33.115 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:33.123 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:33.123 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:33.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:33.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:33.156 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:33.174 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:33.175 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:33.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:33.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:32 smithi196 ceph-mon[30802]: from='client.16770 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:33.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:32 smithi196 ceph-mon[30802]: osdmap e682: 12 total, 12 up, 11 in 2024-09-16T05:20:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:33.252 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:33.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:33.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:33.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:33.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:33.403 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:33.405 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:33.405 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:33.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:33.448 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:33.470 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:33.470 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:33.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:33.491 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:33.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:33.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:33.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:33.514 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:33.535 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:33.536 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:33.560 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:33.582 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:33.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:33.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:33.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:33.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:33.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:33.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:33.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:33.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:20:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:33.879 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:33.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 1 pg_num to 31 2024-09-16T05:20:33.900 INFO:tasks.rados.rados.2.smithi204.stdout:2673: done (4 left) 2024-09-16T05:20:33.900 INFO:tasks.rados.rados.2.smithi204.stdout:2674: write (excl) oid 37 current snap is 279 2024-09-16T05:20:33.900 INFO:tasks.rados.rados.2.smithi204.stdout:2668: finishing write tid 2 to smithi20457400-42 2024-09-16T05:20:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:2668: finishing write tid 3 to smithi20457400-42 2024-09-16T05:20:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:2668: finishing write tid 4 to smithi20457400-42 2024-09-16T05:20:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:2668: finishing write tid 5 to smithi20457400-42 2024-09-16T05:20:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 773 (ObjNum 945 snap 279 seq_num 945) dirty exists 2024-09-16T05:20:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:2668: left oid 42 (ObjNum 945 snap 279 seq_num 945) 2024-09-16T05:20:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:2670: finishing write tid 1 to smithi20457400-10 2024-09-16T05:20:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:2670: finishing write tid 2 to smithi20457400-10 2024-09-16T05:20:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:2674: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-09-16T05:20:33.908 INFO:tasks.rados.rados.2.smithi204.stdout:2674: writing smithi20457400-37 from 716229 to 1204804 tid 1 2024-09-16T05:20:33.915 INFO:tasks.rados.rados.2.smithi204.stdout:2674: writing smithi20457400-37 from 1616330 to 2157658 tid 2 2024-09-16T05:20:33.915 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:33.915 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:33.923 INFO:tasks.rados.rados.2.smithi204.stdout:2674: writing smithi20457400-37 from 2756280 to 3315935 tid 3 2024-09-16T05:20:33.923 INFO:tasks.rados.rados.2.smithi204.stdout:2668: done (4 left) 2024-09-16T05:20:33.923 INFO:tasks.rados.rados.2.smithi204.stdout:2675: read oid 22 snap -1 2024-09-16T05:20:33.923 INFO:tasks.rados.rados.2.smithi204.stdout:2675: expect (ObjNum 926 snap 275 seq_num 926) 2024-09-16T05:20:33.923 INFO:tasks.rados.rados.2.smithi204.stdout:2676: snap_create 2024-09-16T05:20:33.924 INFO:tasks.rados.rados.2.smithi204.stdout:2670: finishing write tid 3 to smithi20457400-10 2024-09-16T05:20:33.924 INFO:tasks.rados.rados.2.smithi204.stdout:2670: finishing write tid 4 to smithi20457400-10 2024-09-16T05:20:33.924 INFO:tasks.rados.rados.2.smithi204.stdout:2670: finishing write tid 5 to smithi20457400-10 2024-09-16T05:20:33.924 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1008 (ObjNum 946 snap 279 seq_num 946) dirty exists 2024-09-16T05:20:33.924 INFO:tasks.rados.rados.2.smithi204.stdout:2670: left oid 10 (ObjNum 946 snap 279 seq_num 946) 2024-09-16T05:20:33.924 INFO:tasks.rados.rados.2.smithi204.stdout:2672: expect (ObjNum 896 snap 262 seq_num 896) 2024-09-16T05:20:33.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:34.003 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:34.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:34.079 INFO:tasks.rados.rados.2.smithi204.stdout:2674: finishing write tid 1 to smithi20457400-37 2024-09-16T05:20:34.079 INFO:tasks.rados.rados.2.smithi204.stdout:2674: finishing write tid 2 to smithi20457400-37 2024-09-16T05:20:34.080 INFO:tasks.rados.rados.2.smithi204.stdout:2674: finishing write tid 3 to smithi20457400-37 2024-09-16T05:20:34.080 INFO:tasks.rados.rados.2.smithi204.stdout:2674: finishing write tid 4 to smithi20457400-37 2024-09-16T05:20:34.080 INFO:tasks.rados.rados.2.smithi204.stdout:2674: finishing write tid 5 to smithi20457400-37 2024-09-16T05:20:34.080 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1029 (ObjNum 947 snap 279 seq_num 947) dirty exists 2024-09-16T05:20:34.080 INFO:tasks.rados.rados.2.smithi204.stdout:2674: left oid 37 (ObjNum 947 snap 279 seq_num 947) 2024-09-16T05:20:34.080 INFO:tasks.rados.rados.2.smithi204.stdout:2675: expect (ObjNum 926 snap 275 seq_num 926) 2024-09-16T05:20:34.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:34.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:34.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:34.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:33 smithi032 ceph-mon[27388]: pgmap v1185: 107 pgs: 107 active+clean; 283 MiB data, 854 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2024-09-16T05:20:34.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:33 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:34.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:33 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:34.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:33 smithi032 ceph-mon[27388]: osdmap e683: 12 total, 12 up, 11 in 2024-09-16T05:20:34.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1173466378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:34.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/847860507' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-09-16T05:20:34.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:33 smithi204 ceph-mon[30933]: pgmap v1185: 107 pgs: 107 active+clean; 283 MiB data, 854 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2024-09-16T05:20:34.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:33 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:34.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:33 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:34.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:33 smithi204 ceph-mon[30933]: osdmap e683: 12 total, 12 up, 11 in 2024-09-16T05:20:34.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1173466378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:34.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/847860507' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-09-16T05:20:34.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:33 smithi196 ceph-mon[30802]: pgmap v1185: 107 pgs: 107 active+clean; 283 MiB data, 854 MiB used, 983 GiB / 983 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 21 op/s 2024-09-16T05:20:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:33 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:33 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:33 smithi196 ceph-mon[30802]: osdmap e683: 12 total, 12 up, 11 in 2024-09-16T05:20:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1173466378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/847860507' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-09-16T05:20:34.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:34.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:34.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:34.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:34.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:34.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:34.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:34.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:34.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:20:34.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:34.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:34.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:34.663 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:34.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:34.752 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:34.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:34.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:34.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:34.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:34.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:34.904 INFO:tasks.rados.rados.2.smithi204.stdout:2670: done (5 left) 2024-09-16T05:20:34.904 INFO:tasks.rados.rados.2.smithi204.stdout:2671: done (4 left) 2024-09-16T05:20:34.904 INFO:tasks.rados.rados.2.smithi204.stdout:2672: done (3 left) 2024-09-16T05:20:34.904 INFO:tasks.rados.rados.2.smithi204.stdout:2674: done (2 left) 2024-09-16T05:20:34.904 INFO:tasks.rados.rados.2.smithi204.stdout:2675: done (1 left) 2024-09-16T05:20:34.904 INFO:tasks.rados.rados.2.smithi204.stdout:2676: done (0 left) 2024-09-16T05:20:34.904 INFO:tasks.rados.rados.2.smithi204.stdout:2677: write (excl) oid 20 current snap is 280 2024-09-16T05:20:34.904 INFO:tasks.rados.rados.2.smithi204.stdout:2677: seq_num 948 ranges {733036=563824,2074812=684986,3183703=749333} 2024-09-16T05:20:34.911 INFO:tasks.rados.rados.2.smithi204.stdout:2677: writing smithi20457400-20 from 733036 to 1296860 tid 1 2024-09-16T05:20:34.919 INFO:tasks.rados.rados.2.smithi204.stdout:2677: writing smithi20457400-20 from 2074812 to 2759798 tid 2 2024-09-16T05:20:34.926 INFO:tasks.rados.rados.2.smithi204.stdout:2677: writing smithi20457400-20 from 3183703 to 3933036 tid 3 2024-09-16T05:20:34.926 INFO:tasks.rados.rados.2.smithi204.stdout:2678: write oid 16 current snap is 280 2024-09-16T05:20:34.926 INFO:tasks.rados.rados.2.smithi204.stdout:2677: finishing write tid 1 to smithi20457400-20 2024-09-16T05:20:34.926 INFO:tasks.rados.rados.2.smithi204.stdout:2678: seq_num 949 ranges {749843=639073,2133294=544997,3211126=582199} 2024-09-16T05:20:34.933 INFO:tasks.rados.rados.2.smithi204.stdout:2678: writing smithi20457400-16 from 749843 to 1388916 tid 1 2024-09-16T05:20:34.940 INFO:tasks.rados.rados.2.smithi204.stdout:2678: writing smithi20457400-16 from 2133294 to 2678291 tid 2 2024-09-16T05:20:34.946 INFO:tasks.rados.rados.2.smithi204.stdout:2678: writing smithi20457400-16 from 3211126 to 3793325 tid 3 2024-09-16T05:20:34.946 INFO:tasks.rados.rados.2.smithi204.stdout:2679: snap_remove snap 260 2024-09-16T05:20:34.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:35.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:35.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/847860507' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-16T05:20:35.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:34 smithi032 ceph-mon[27388]: osdmap e684: 12 total, 12 up, 11 in 2024-09-16T05:20:35.016 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:34 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-16T05:20:35.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:35.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/847860507' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-16T05:20:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:34 smithi204 ceph-mon[30933]: osdmap e684: 12 total, 12 up, 11 in 2024-09-16T05:20:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:34 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-16T05:20:35.224 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:35.225 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:35.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/847860507' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-16T05:20:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:34 smithi196 ceph-mon[30802]: osdmap e684: 12 total, 12 up, 11 in 2024-09-16T05:20:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:34 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-16T05:20:35.252 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:35.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:35.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:35.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:35.500 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:35.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.624 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:35.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:35.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:35.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:35.794 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2679: done (2 left) 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2680: write (excl) oid 43 current snap is 280 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2677: finishing write tid 2 to smithi20457400-20 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2677: finishing write tid 3 to smithi20457400-20 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2677: finishing write tid 4 to smithi20457400-20 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2677: finishing write tid 5 to smithi20457400-20 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1122 (ObjNum 948 snap 280 seq_num 948) dirty exists 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2677: left oid 20 (ObjNum 948 snap 280 seq_num 948) 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2678: finishing write tid 1 to smithi20457400-16 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2678: finishing write tid 2 to smithi20457400-16 2024-09-16T05:20:35.805 INFO:tasks.rados.rados.2.smithi204.stdout:2678: finishing write tid 3 to smithi20457400-16 2024-09-16T05:20:35.806 INFO:tasks.rados.rados.2.smithi204.stdout:2678: finishing write tid 4 to smithi20457400-16 2024-09-16T05:20:35.806 INFO:tasks.rados.rados.2.smithi204.stdout:2678: finishing write tid 5 to smithi20457400-16 2024-09-16T05:20:35.806 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 910 (ObjNum 949 snap 280 seq_num 949) dirty exists 2024-09-16T05:20:35.806 INFO:tasks.rados.rados.2.smithi204.stdout:2678: left oid 16 (ObjNum 949 snap 280 seq_num 949) 2024-09-16T05:20:35.806 INFO:tasks.rados.rados.2.smithi204.stdout:2680: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2024-09-16T05:20:35.814 INFO:tasks.rados.rados.2.smithi204.stdout:2680: writing smithi20457400-43 from 766650 to 1480972 tid 1 2024-09-16T05:20:35.821 INFO:tasks.rados.rados.2.smithi204.stdout:2680: writing smithi20457400-43 from 2075423 to 2764078 tid 2 2024-09-16T05:20:35.828 INFO:tasks.rados.rados.2.smithi204.stdout:2680: writing smithi20457400-43 from 3522196 to 3966650 tid 3 2024-09-16T05:20:35.828 INFO:tasks.rados.rados.2.smithi204.stdout:2677: done (2 left) 2024-09-16T05:20:35.828 INFO:tasks.rados.rados.2.smithi204.stdout:2678: done (1 left) 2024-09-16T05:20:35.828 INFO:tasks.rados.rados.2.smithi204.stdout:2681: read oid 21 snap -1 2024-09-16T05:20:35.829 INFO:tasks.rados.rados.2.smithi204.stdout:2681: expect deleted 2024-09-16T05:20:35.829 INFO:tasks.rados.rados.2.smithi204.stdout:2682: read oid 25 snap 265 2024-09-16T05:20:35.829 INFO:tasks.rados.rados.2.smithi204.stdout:2682: expect deleted 2024-09-16T05:20:35.829 INFO:tasks.rados.rados.2.smithi204.stdout:2683: write oid 42 current snap is 280 2024-09-16T05:20:35.829 INFO:tasks.rados.rados.2.smithi204.stdout:2680: finishing write tid 1 to smithi20457400-43 2024-09-16T05:20:35.829 INFO:tasks.rados.rados.2.smithi204.stdout:2683: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2024-09-16T05:20:35.835 INFO:tasks.rados.rados.2.smithi204.stdout:2683: writing smithi20457400-42 from 783457 to 1289381 tid 1 2024-09-16T05:20:35.841 INFO:tasks.rados.rados.2.smithi204.stdout:2683: writing smithi20457400-42 from 1850258 to 2398924 tid 2 2024-09-16T05:20:35.848 INFO:tasks.rados.rados.2.smithi204.stdout:2683: writing smithi20457400-42 from 2865972 to 3587068 tid 3 2024-09-16T05:20:35.848 INFO:tasks.rados.rados.2.smithi204.stdout:2684: rollback oid 24 current snap is 280 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 276 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2680: finishing write tid 2 to smithi20457400-43 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2680: finishing write tid 3 to smithi20457400-43 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2680: finishing write tid 4 to smithi20457400-43 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2681: done (4 left) 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2682: done (3 left) 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2685: copy_from oid 25 from oid 22 current snap is 280 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2680: finishing write tid 5 to smithi20457400-43 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 642 (ObjNum 950 snap 280 seq_num 950) dirty exists 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2680: left oid 43 (ObjNum 950 snap 280 seq_num 950) 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2683: finishing write tid 1 to smithi20457400-42 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2683: finishing write tid 2 to smithi20457400-42 2024-09-16T05:20:35.849 INFO:tasks.rados.rados.2.smithi204.stdout:2680: done (3 left) 2024-09-16T05:20:35.850 INFO:tasks.rados.rados.2.smithi204.stdout:2686: read oid 14 snap -1 2024-09-16T05:20:35.850 INFO:tasks.rados.rados.2.smithi204.stdout:2686: expect (ObjNum 896 snap 262 seq_num 896) 2024-09-16T05:20:35.850 INFO:tasks.rados.rados.2.smithi204.stdout:2687: delete oid 21 current snap is 280 2024-09-16T05:20:35.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:35.851 INFO:tasks.rados.rados.2.smithi204.stdout:2687: done (4 left) 2024-09-16T05:20:35.851 INFO:tasks.rados.rados.2.smithi204.stdout:2688: write (excl) oid 26 current snap is 280 2024-09-16T05:20:35.851 INFO:tasks.rados.rados.2.smithi204.stdout:2688: seq_num 953 ranges {417071=656422,1567222=449849} 2024-09-16T05:20:35.855 INFO:tasks.rados.rados.2.smithi204.stdout:2688: writing smithi20457400-26 from 417071 to 1073493 tid 1 2024-09-16T05:20:35.861 INFO:tasks.rados.rados.2.smithi204.stdout:2688: writing smithi20457400-26 from 1567222 to 2017071 tid 2 2024-09-16T05:20:35.861 INFO:tasks.rados.rados.2.smithi204.stdout:2689: delete oid 30 current snap is 280 2024-09-16T05:20:35.861 INFO:tasks.rados.rados.2.smithi204.stdout:2686: expect (ObjNum 896 snap 262 seq_num 896) 2024-09-16T05:20:35.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:35.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:20:35.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:35.997 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:35.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:35.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:36.023 INFO:tasks.rados.rados.2.smithi204.stdout:2684: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2684: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 779 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2683: finishing write tid 3 to smithi20457400-42 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2683: finishing write tid 4 to smithi20457400-42 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2684: done (5 left) 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2686: done (4 left) 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2689: done (3 left) 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2690: snap_create 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2683: finishing write tid 5 to smithi20457400-42 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 777 (ObjNum 951 snap 280 seq_num 951) dirty exists 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2683: left oid 42 (ObjNum 951 snap 280 seq_num 951) 2024-09-16T05:20:36.024 INFO:tasks.rados.rados.2.smithi204.stdout:2688: finishing write tid 1 to smithi20457400-26 2024-09-16T05:20:36.025 INFO:tasks.rados.rados.2.smithi204.stdout:2685: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:20:36.025 INFO:tasks.rados.rados.2.smithi204.stdout:2688: finishing write tid 2 to smithi20457400-26 2024-09-16T05:20:36.025 INFO:tasks.rados.rados.2.smithi204.stdout:2688: finishing write tid 3 to smithi20457400-26 2024-09-16T05:20:36.025 INFO:tasks.rados.rados.2.smithi204.stdout:2688: finishing write tid 4 to smithi20457400-26 2024-09-16T05:20:36.025 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1212 (ObjNum 953 snap 280 seq_num 953) dirty exists 2024-09-16T05:20:36.025 INFO:tasks.rados.rados.2.smithi204.stdout:2688: left oid 26 (ObjNum 953 snap 280 seq_num 953) 2024-09-16T05:20:36.025 INFO:tasks.rados.rados.2.smithi204.stdout:2685: finishing copy_from to smithi20457400-25 2024-09-16T05:20:36.025 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 707 (ObjNum 926 snap 275 seq_num 926) dirty exists 2024-09-16T05:20:36.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:36.121 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:36.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:35 smithi032 ceph-mon[27388]: pgmap v1188: 107 pgs: 107 active+clean; 288 MiB data, 890 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2024-09-16T05:20:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:35 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-16T05:20:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:35 smithi032 ceph-mon[27388]: osdmap e685: 12 total, 12 up, 11 in 2024-09-16T05:20:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1621053048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:35 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1621053048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:35 smithi032 ceph-mon[27388]: osdmap e686: 12 total, 12 up, 11 in 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:35 smithi204 ceph-mon[30933]: pgmap v1188: 107 pgs: 107 active+clean; 288 MiB data, 890 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:35 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:35 smithi204 ceph-mon[30933]: osdmap e685: 12 total, 12 up, 11 in 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1621053048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:35 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1621053048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:36.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:35 smithi204 ceph-mon[30933]: osdmap e686: 12 total, 12 up, 11 in 2024-09-16T05:20:36.174 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:36.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:35 smithi196 ceph-mon[30802]: pgmap v1188: 107 pgs: 107 active+clean; 288 MiB data, 890 MiB used, 983 GiB / 983 GiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2024-09-16T05:20:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:35 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-16T05:20:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:35 smithi196 ceph-mon[30802]: osdmap e685: 12 total, 12 up, 11 in 2024-09-16T05:20:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1621053048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:35 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1621053048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:35 smithi196 ceph-mon[30802]: osdmap e686: 12 total, 12 up, 11 in 2024-09-16T05:20:36.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:36.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:36.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:36.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:36.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:36.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:20:36.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:36.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:36.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:36.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:36.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:36.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:36.649 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:36.649 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:36.662 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:36.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:36.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:36.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:36.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:36.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:36.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:2683: done (3 left) 2024-09-16T05:20:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:2685: done (2 left) 2024-09-16T05:20:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:2688: done (1 left) 2024-09-16T05:20:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:2690: done (0 left) 2024-09-16T05:20:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:2691: rollback oid 45 current snap is 281 2024-09-16T05:20:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 279 2024-09-16T05:20:36.920 INFO:tasks.rados.rados.2.smithi204.stdout:2692: copy_from oid 28 from oid 39 current snap is 281 2024-09-16T05:20:36.921 INFO:tasks.rados.rados.2.smithi204.stdout:2693: write oid 18 current snap is 281 2024-09-16T05:20:36.921 INFO:tasks.rados.rados.2.smithi204.stdout:2693: seq_num 955 ranges {450685=406920,1567833=482852} 2024-09-16T05:20:36.925 INFO:tasks.rados.rados.2.smithi204.stdout:2693: writing smithi20457400-18 from 450685 to 857605 tid 1 2024-09-16T05:20:36.932 INFO:tasks.rados.rados.2.smithi204.stdout:2693: writing smithi20457400-18 from 1567833 to 2050685 tid 2 2024-09-16T05:20:36.932 INFO:tasks.rados.rados.2.smithi204.stdout:2694: snap_create 2024-09-16T05:20:36.932 INFO:tasks.rados.rados.2.smithi204.stdout:2691: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:20:36.932 INFO:tasks.rados.rados.2.smithi204.stdout:2691: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:20:36.932 INFO:tasks.rados.rados.2.smithi204.stdout:2691: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:20:36.932 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1018 (ObjNum 937 snap 278 seq_num 937) dirty exists 2024-09-16T05:20:36.932 INFO:tasks.rados.rados.2.smithi204.stdout:2693: finishing write tid 1 to smithi20457400-18 2024-09-16T05:20:36.935 INFO:tasks.rados.rados.2.smithi204.stdout:2692: finishing copy_from racing read to smithi20457400-28 2024-09-16T05:20:36.935 INFO:tasks.rados.rados.2.smithi204.stdout:2693: finishing write tid 2 to smithi20457400-18 2024-09-16T05:20:36.935 INFO:tasks.rados.rados.2.smithi204.stdout:2693: finishing write tid 3 to smithi20457400-18 2024-09-16T05:20:36.936 INFO:tasks.rados.rados.2.smithi204.stdout:2693: finishing write tid 4 to smithi20457400-18 2024-09-16T05:20:36.936 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 539 (ObjNum 955 snap 281 seq_num 955) dirty exists 2024-09-16T05:20:36.936 INFO:tasks.rados.rados.2.smithi204.stdout:2693: left oid 18 (ObjNum 955 snap 281 seq_num 955) 2024-09-16T05:20:36.939 INFO:tasks.rados.rados.2.smithi204.stdout:2692: finishing copy_from to smithi20457400-28 2024-09-16T05:20:36.939 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1113 (ObjNum 939 snap 278 seq_num 939) dirty exists 2024-09-16T05:20:36.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:37.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:37.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:37.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:37.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:37.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:37.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:36 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:20:37.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:36 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:20:37.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:36 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:20:37.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:36 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:20:37.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:36 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-16T05:20:37.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:36 smithi032 ceph-mon[27388]: 1.17 scrub starts 2024-09-16T05:20:37.137 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:36 smithi032 ceph-mon[27388]: 1.17 scrub ok 2024-09-16T05:20:37.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:36 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:20:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:36 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:20:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:36 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:20:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:36 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:20:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:36 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-16T05:20:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:36 smithi204 ceph-mon[30933]: 1.17 scrub starts 2024-09-16T05:20:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:36 smithi204 ceph-mon[30933]: 1.17 scrub ok 2024-09-16T05:20:37.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:37.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:36 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:20:37.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:36 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:20:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:36 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:20:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:36 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:20:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:36 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-16T05:20:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:36 smithi196 ceph-mon[30802]: 1.17 scrub starts 2024-09-16T05:20:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:36 smithi196 ceph-mon[30802]: 1.17 scrub ok 2024-09-16T05:20:37.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:37.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:37.269 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:37.269 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:37.269 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:37.269 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:37.288 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:37.309 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:37.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:20:37.330 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:37.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:37.351 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:37.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:37.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:37.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:37.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:37.395 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:37.395 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:37.416 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:37.416 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:37.437 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:37.437 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:37.458 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:37.459 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:37.479 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:37.480 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:37.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:37.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:37.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:37.573 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:37.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:37.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:37.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:37.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:37.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:37.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:37.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:37.884 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:37.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:37.920 INFO:tasks.rados.rados.2.smithi204.stdout:2691: done (3 left) 2024-09-16T05:20:37.921 INFO:tasks.rados.rados.2.smithi204.stdout:2692: done (2 left) 2024-09-16T05:20:37.921 INFO:tasks.rados.rados.2.smithi204.stdout:2693: done (1 left) 2024-09-16T05:20:37.921 INFO:tasks.rados.rados.2.smithi204.stdout:2694: done (0 left) 2024-09-16T05:20:37.921 INFO:tasks.rados.rados.2.smithi204.stdout:2695: rollback oid 21 current snap is 282 2024-09-16T05:20:37.921 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 21 to 280 2024-09-16T05:20:37.921 INFO:tasks.rados.rados.2.smithi204.stdout:2696: write oid 4 current snap is 282 2024-09-16T05:20:37.922 INFO:tasks.rados.rados.2.smithi204.stdout:2696: seq_num 956 ranges {467492=482169,1626315=441177} 2024-09-16T05:20:37.925 INFO:tasks.rados.rados.2.smithi204.stdout:2696: writing smithi20457400-4 from 467492 to 949661 tid 1 2024-09-16T05:20:37.932 INFO:tasks.rados.rados.2.smithi204.stdout:2696: writing smithi20457400-4 from 1626315 to 2067492 tid 2 2024-09-16T05:20:37.932 INFO:tasks.rados.rados.2.smithi204.stdout:2697: read oid 7 snap -1 2024-09-16T05:20:37.932 INFO:tasks.rados.rados.2.smithi204.stdout:2697: expect (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:20:37.932 INFO:tasks.rados.rados.2.smithi204.stdout:2698: read oid 17 snap -1 2024-09-16T05:20:37.933 INFO:tasks.rados.rados.2.smithi204.stdout:2698: expect (ObjNum 846 snap 246 seq_num 846) 2024-09-16T05:20:37.933 INFO:tasks.rados.rados.2.smithi204.stdout:2699: copy_from oid 39 from oid 5 current snap is 282 2024-09-16T05:20:37.933 INFO:tasks.rados.rados.2.smithi204.stdout:2695: finishing rollback tid 1 to smithi20457400-21 2024-09-16T05:20:37.933 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:20:37.933 INFO:tasks.rados.rados.2.smithi204.stdout:2695: done (4 left) 2024-09-16T05:20:37.933 INFO:tasks.rados.rados.2.smithi204.stdout:2700: snap_remove snap 263 2024-09-16T05:20:37.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: pgmap v1191: 117 pgs: 1 peering, 5 unknown, 111 active+clean; 291 MiB data, 894 MiB used, 983 GiB / 983 GiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2024-09-16T05:20:37.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:20:37.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:20:37.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:20:37.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:20:37.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.b scrub ok 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.1a scrub starts 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.1a scrub ok 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: osdmap e687: 12 total, 12 up, 11 in 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.18 scrub starts 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.18 scrub ok 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1302257798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:20:37.984 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:37 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:20:37.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:38.008 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:38.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:38.083 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:20:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:20:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:20:38.084 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:20:38.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:38.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:38.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: pgmap v1191: 117 pgs: 1 peering, 5 unknown, 111 active+clean; 291 MiB data, 894 MiB used, 983 GiB / 983 GiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2024-09-16T05:20:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:20:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:20:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:20:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.b scrub ok 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.1a scrub starts 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.1a scrub ok 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: osdmap e687: 12 total, 12 up, 11 in 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.18 scrub starts 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.18 scrub ok 2024-09-16T05:20:38.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1302257798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:38.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:20:38.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:20:38.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:20:38.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:37 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: pgmap v1191: 117 pgs: 1 peering, 5 unknown, 111 active+clean; 291 MiB data, 894 MiB used, 983 GiB / 983 GiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 12 op/s 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.b scrub ok 2024-09-16T05:20:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.1a scrub starts 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.1a scrub ok 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: osdmap e687: 12 total, 12 up, 11 in 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.18 scrub starts 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.18 scrub ok 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1302257798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:20:38.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:37 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:20:38.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:38.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:38.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:38.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:38.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:38.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:38.513 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:38.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:38.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:20:38.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:38.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:38.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:38.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:20:38.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:38.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:38.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:38.853 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:38.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:38.899 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:38.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:38.921 INFO:tasks.rados.rados.2.smithi204.stdout:2700: done (4 left) 2024-09-16T05:20:38.921 INFO:tasks.rados.rados.2.smithi204.stdout:2701: write (excl) oid 46 current snap is 282 2024-09-16T05:20:38.921 INFO:tasks.rados.rados.2.smithi204.stdout:2701: seq_num 957 ranges {484299=557418,1684797=399502} 2024-09-16T05:20:38.930 INFO:tasks.rados.rados.2.smithi204.stdout:2701: writing smithi20457400-46 from 484299 to 1041717 tid 1 2024-09-16T05:20:38.936 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 1.d scrub starts 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 1.d scrub ok 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: osdmap e688: 12 total, 12 up, 11 in 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:20:38.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:20:38.938 INFO:tasks.rados.rados.2.smithi204.stdout:2701: writing smithi20457400-46 from 1684797 to 2084299 tid 2 2024-09-16T05:20:38.938 INFO:tasks.rados.rados.2.smithi204.stdout:2702: read oid 6 snap -1 2024-09-16T05:20:38.938 INFO:tasks.rados.rados.2.smithi204.stdout:2702: expect (ObjNum 906 snap 267 seq_num 906) 2024-09-16T05:20:38.938 INFO:tasks.rados.rados.2.smithi204.stdout:2703: write oid 33 current snap is 282 2024-09-16T05:20:38.938 INFO:tasks.rados.rados.2.smithi204.stdout:2703: seq_num 958 ranges {501106=749020,1743279=357827} 2024-09-16T05:20:38.945 INFO:tasks.rados.rados.2.smithi204.stdout:2703: writing smithi20457400-33 from 501106 to 1250126 tid 1 2024-09-16T05:20:38.951 INFO:tasks.rados.rados.2.smithi204.stdout:2703: writing smithi20457400-33 from 1743279 to 2101106 tid 2 2024-09-16T05:20:38.952 INFO:tasks.rados.rados.2.smithi204.stdout:2704: snap_create 2024-09-16T05:20:38.952 INFO:tasks.rados.rados.2.smithi204.stdout:2696: finishing write tid 1 to smithi20457400-4 2024-09-16T05:20:38.952 INFO:tasks.rados.rados.2.smithi204.stdout:2696: finishing write tid 2 to smithi20457400-4 2024-09-16T05:20:38.952 INFO:tasks.rados.rados.2.smithi204.stdout:2696: finishing write tid 3 to smithi20457400-4 2024-09-16T05:20:38.952 INFO:tasks.rados.rados.2.smithi204.stdout:2696: finishing write tid 4 to smithi20457400-4 2024-09-16T05:20:38.952 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1001 (ObjNum 956 snap 282 seq_num 956) dirty exists 2024-09-16T05:20:38.952 INFO:tasks.rados.rados.2.smithi204.stdout:2696: left oid 4 (ObjNum 956 snap 282 seq_num 956) 2024-09-16T05:20:38.952 INFO:tasks.rados.rados.2.smithi204.stdout:2698: expect (ObjNum 846 snap 246 seq_num 846) 2024-09-16T05:20:38.973 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:39.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:39.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:39.080 INFO:tasks.rados.rados.2.smithi204.stdout:2697: expect (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:20:39.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:39.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:39.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:39.202 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:20:39.202 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:38 smithi204 ceph-mon[30933]: osdmap e689: 12 total, 12 up, 11 in 2024-09-16T05:20:39.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 1.d scrub starts 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 1.d scrub ok 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: osdmap e688: 12 total, 12 up, 11 in 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:20:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:38 smithi196 ceph-mon[30802]: osdmap e689: 12 total, 12 up, 11 in 2024-09-16T05:20:39.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 1.d scrub starts 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 1.d scrub ok 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: osdmap e688: 12 total, 12 up, 11 in 2024-09-16T05:20:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:20:39.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:20:39.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:20:39.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:20:39.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:38 smithi032 ceph-mon[27388]: osdmap e689: 12 total, 12 up, 11 in 2024-09-16T05:20:39.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:39.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:39.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:2699: finishing copy_from racing read to smithi20457400-39 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:2699: finishing copy_from to smithi20457400-39 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 608 (ObjNum 754 snap 220 seq_num 754) dirty exists 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:2701: finishing write tid 1 to smithi20457400-46 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:2701: finishing write tid 2 to smithi20457400-46 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:2701: finishing write tid 3 to smithi20457400-46 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:2701: finishing write tid 4 to smithi20457400-46 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1011 (ObjNum 957 snap 282 seq_num 957) dirty exists 2024-09-16T05:20:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:2701: left oid 46 (ObjNum 957 snap 282 seq_num 957) 2024-09-16T05:20:39.286 INFO:tasks.rados.rados.2.smithi204.stdout:2702: expect (ObjNum 906 snap 267 seq_num 906) 2024-09-16T05:20:39.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:39.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:39.442 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:20:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:20:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:20:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:20:39.442 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:20:39.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:39.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:39.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:39.567 INFO:tasks.rados.rados.2.smithi204.stdout:2703: finishing write tid 1 to smithi20457400-33 2024-09-16T05:20:39.567 INFO:tasks.rados.rados.2.smithi204.stdout:2703: finishing write tid 2 to smithi20457400-33 2024-09-16T05:20:39.567 INFO:tasks.rados.rados.2.smithi204.stdout:2703: finishing write tid 3 to smithi20457400-33 2024-09-16T05:20:39.567 INFO:tasks.rados.rados.2.smithi204.stdout:2703: finishing write tid 4 to smithi20457400-33 2024-09-16T05:20:39.567 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 850 (ObjNum 958 snap 282 seq_num 958) dirty exists 2024-09-16T05:20:39.567 INFO:tasks.rados.rados.2.smithi204.stdout:2703: left oid 33 (ObjNum 958 snap 282 seq_num 958) 2024-09-16T05:20:39.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:39.645 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:39.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:39.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:39.769 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:39.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:39.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:39.893 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:39.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:39.926 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:20:39.932 INFO:tasks.rados.rados.2.smithi204.stdout:2696: done (7 left) 2024-09-16T05:20:39.932 INFO:tasks.rados.rados.2.smithi204.stdout:2697: done (6 left) 2024-09-16T05:20:39.932 INFO:tasks.rados.rados.2.smithi204.stdout:2698: done (5 left) 2024-09-16T05:20:39.932 INFO:tasks.rados.rados.2.smithi204.stdout:2699: done (4 left) 2024-09-16T05:20:39.932 INFO:tasks.rados.rados.2.smithi204.stdout:2701: done (3 left) 2024-09-16T05:20:39.932 INFO:tasks.rados.rados.2.smithi204.stdout:2702: done (2 left) 2024-09-16T05:20:39.933 INFO:tasks.rados.rados.2.smithi204.stdout:2703: done (1 left) 2024-09-16T05:20:39.933 INFO:tasks.rados.rados.2.smithi204.stdout:2704: done (0 left) 2024-09-16T05:20:39.933 INFO:tasks.rados.rados.2.smithi204.stdout:2705: read oid 43 snap -1 2024-09-16T05:20:39.933 INFO:tasks.rados.rados.2.smithi204.stdout:2705: expect (ObjNum 950 snap 280 seq_num 950) 2024-09-16T05:20:39.933 INFO:tasks.rados.rados.2.smithi204.stdout:2706: read oid 4 snap -1 2024-09-16T05:20:39.933 INFO:tasks.rados.rados.2.smithi204.stdout:2706: expect (ObjNum 956 snap 282 seq_num 956) 2024-09-16T05:20:39.933 INFO:tasks.rados.rados.2.smithi204.stdout:2707: delete oid 12 current snap is 283 2024-09-16T05:20:39.935 INFO:tasks.rados.rados.2.smithi204.stdout:2707: done (2 left) 2024-09-16T05:20:39.935 INFO:tasks.rados.rados.2.smithi204.stdout:2708: read oid 40 snap -1 2024-09-16T05:20:39.935 INFO:tasks.rados.rados.2.smithi204.stdout:2708: expect (ObjNum 730 snap 210 seq_num 730) 2024-09-16T05:20:39.935 INFO:tasks.rados.rados.2.smithi204.stdout:2709: write oid 15 current snap is 283 2024-09-16T05:20:39.935 INFO:tasks.rados.rados.2.smithi204.stdout:2709: seq_num 960 ranges {534720=499518,1460243=674477} 2024-09-16T05:20:39.941 INFO:tasks.rados.rados.2.smithi204.stdout:2709: writing smithi20457400-15 from 534720 to 1034238 tid 1 2024-09-16T05:20:39.948 INFO:tasks.rados.rados.2.smithi204.stdout:2709: writing smithi20457400-15 from 1460243 to 2134720 tid 2 2024-09-16T05:20:39.948 INFO:tasks.rados.rados.2.smithi204.stdout:2710: snap_create 2024-09-16T05:20:39.948 INFO:tasks.rados.rados.2.smithi204.stdout:2706: expect (ObjNum 956 snap 282 seq_num 956) 2024-09-16T05:20:39.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:40.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:40.018 INFO:tasks.rados.rados.2.smithi204.stdout:2705: expect (ObjNum 950 snap 280 seq_num 950) 2024-09-16T05:20:40.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:20:40.043 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:20:40.044 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:20:40.044 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T05:20:40.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:20:40.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: pgmap v1194: 117 pgs: 1 active+clean+scrubbing, 1 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 300 MiB data, 905 MiB used, 983 GiB / 983 GiB avail; 7.2 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-16T05:20:40.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:20:40.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:20:40.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.35 scrub starts 2024-09-16T05:20:40.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.35 scrub ok 2024-09-16T05:20:40.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.43 scrub starts 2024-09-16T05:20:40.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.43 scrub ok 2024-09-16T05:20:40.095 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2313054693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2313054693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:40.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:39 smithi204 ceph-mon[30933]: osdmap e690: 12 total, 12 up, 11 in 2024-09-16T05:20:40.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:40.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:20:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:40.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:40.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:40.150 INFO:tasks.rados.rados.2.smithi204.stdout:2708: expect (ObjNum 730 snap 210 seq_num 730) 2024-09-16T05:20:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: pgmap v1194: 117 pgs: 1 active+clean+scrubbing, 1 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 300 MiB data, 905 MiB used, 983 GiB / 983 GiB avail; 7.2 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-16T05:20:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:20:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:20:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.35 scrub starts 2024-09-16T05:20:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.35 scrub ok 2024-09-16T05:20:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.43 scrub starts 2024-09-16T05:20:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.43 scrub ok 2024-09-16T05:20:40.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:20:40.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:20:40.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:20:40.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:20:40.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:20:40.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2313054693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:40.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:20:40.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:20:40.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:20:40.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2313054693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:40.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:39 smithi196 ceph-mon[30802]: osdmap e690: 12 total, 12 up, 11 in 2024-09-16T05:20:40.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:40.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: pgmap v1194: 117 pgs: 1 active+clean+scrubbing, 1 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 300 MiB data, 905 MiB used, 983 GiB / 983 GiB avail; 7.2 MiB/s rd, 3.6 MiB/s wr, 18 op/s 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.35 scrub starts 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.35 scrub ok 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.43 scrub starts 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.43 scrub ok 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:20:40.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:20:40.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2313054693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:40.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:20:40.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:20:40.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:20:40.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2313054693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:40.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:39 smithi032 ceph-mon[27388]: osdmap e690: 12 total, 12 up, 11 in 2024-09-16T05:20:40.263 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:40.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:40.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:40.369 INFO:tasks.rados.rados.2.smithi204.stdout:2709: finishing write tid 1 to smithi20457400-15 2024-09-16T05:20:40.369 INFO:tasks.rados.rados.2.smithi204.stdout:2709: finishing write tid 2 to smithi20457400-15 2024-09-16T05:20:40.369 INFO:tasks.rados.rados.2.smithi204.stdout:2709: finishing write tid 3 to smithi20457400-15 2024-09-16T05:20:40.369 INFO:tasks.rados.rados.2.smithi204.stdout:2709: finishing write tid 4 to smithi20457400-15 2024-09-16T05:20:40.369 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 744 (ObjNum 960 snap 283 seq_num 960) dirty exists 2024-09-16T05:20:40.369 INFO:tasks.rados.rados.2.smithi204.stdout:2709: left oid 15 (ObjNum 960 snap 283 seq_num 960) 2024-09-16T05:20:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:40.390 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:40.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:40.405 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:40.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:40.513 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:40.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:40.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:40.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:40.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:40.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:40.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:40.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2705: done (4 left) 2024-09-16T05:20:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2706: done (3 left) 2024-09-16T05:20:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2708: done (2 left) 2024-09-16T05:20:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2709: done (1 left) 2024-09-16T05:20:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2710: done (0 left) 2024-09-16T05:20:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2711: read oid 44 snap -1 2024-09-16T05:20:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2711: expect deleted 2024-09-16T05:20:40.806 INFO:tasks.rados.rados.2.smithi204.stdout:2712: write (excl) oid 35 current snap is 284 2024-09-16T05:20:40.807 INFO:tasks.rados.rados.2.smithi204.stdout:2712: seq_num 961 ranges {551527=574767,1918725=232802} 2024-09-16T05:20:40.812 INFO:tasks.rados.rados.2.smithi204.stdout:2712: writing smithi20457400-35 from 551527 to 1126294 tid 1 2024-09-16T05:20:40.818 INFO:tasks.rados.rados.2.smithi204.stdout:2712: writing smithi20457400-35 from 1918725 to 2151527 tid 2 2024-09-16T05:20:40.818 INFO:tasks.rados.rados.2.smithi204.stdout:2713: snap_create 2024-09-16T05:20:40.818 INFO:tasks.rados.rados.2.smithi204.stdout:2712: finishing write tid 1 to smithi20457400-35 2024-09-16T05:20:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:2712: finishing write tid 2 to smithi20457400-35 2024-09-16T05:20:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:2712: finishing write tid 3 to smithi20457400-35 2024-09-16T05:20:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:2712: finishing write tid 4 to smithi20457400-35 2024-09-16T05:20:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 846 (ObjNum 961 snap 284 seq_num 961) dirty exists 2024-09-16T05:20:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:2712: left oid 35 (ObjNum 961 snap 284 seq_num 961) 2024-09-16T05:20:40.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:40.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:40.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:40.889 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:20:40.889 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:20:40.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:20:40.889 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2255 2.1d 2:ba754fcb:::smithi20457400-30:head [append 0~0] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 419.20619749299999, 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53925791899999997, 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2255 2024-09-16T05:20:40.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7300000000000001e-07 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681684+0000", 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.582e-06 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681692+0000", 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3920000000000003e-06 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681698+0000", 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2100000000000001e-06 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681779+0000", 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1003999999999999e-05 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:40.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681782+0000", 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1190000000000001e-06 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684363+0000", 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025812399999999998 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684417+0000", 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3692e-05 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684503+0000", 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6592e-05 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684818+0000", 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000314851 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:40.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220845+0000", 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53602721099999995 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220887+0000", 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1514000000000003e-05 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220939+0000", 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2039000000000001e-05 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2559 2.1d 2:ba754fcb:::smithi20457400-30:head [copy-from ver 324] snapc 6d=[6c,6b,6a,69,67,66,64,63,62,60,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 355.93123331999999, 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.74015230600000004, 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.897 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2559 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956647+0000", 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2919999999999998e-06 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956655+0000", 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.729e-06 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956672+0000", 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6775000000000001e-05 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956685+0000", 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2832e-05 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956782+0000", 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6862999999999997e-05 2024-09-16T05:20:40.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956790+0000", 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4020000000000002e-06 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958620+0000", 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001830415 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958718+0000", 2024-09-16T05:20:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7708999999999996e-05 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.688352+0000", 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72963416800000003 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.695488+0000", 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071356659999999997 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:40.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696731+0000", 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00124332 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696767+0000", 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6171000000000003e-05 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696797+0000", 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9964e-05 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.903 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.904 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2560 2.1d 2:ba754fcb:::smithi20457400-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 355.93116744000002, 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73643418699999996, 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2560 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956712+0000", 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.82e-07 2024-09-16T05:20:40.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956714+0000", 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6419999999999999e-06 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956737+0000", 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2668999999999999e-05 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956744+0000", 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.289e-06 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956847+0000", 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102729 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956855+0000", 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4009999999999997e-06 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958884+0000", 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020286409999999999 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958928+0000", 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4458e-05 2024-09-16T05:20:40.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693004+0000", 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73407526999999995 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693071+0000", 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7176999999999997e-05 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693145+0000", 2024-09-16T05:20:40.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4328999999999999e-05 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.0.0:67 2.31 e356/279 2:8c1ba486:::smithi20457400-2:75 v 356'145, mlcod=350'141)", 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 335.18197682499999, 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.929470409, 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.911 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4499999999999998e-07 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705907+0000", 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8799999999999999e-06 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634756+0000", 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92884895999999995 2024-09-16T05:20:40.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634764+0000", 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9589999999999995e-06 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634827+0000", 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3280000000000004e-05 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634853+0000", 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6231999999999999e-05 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635342+0000", 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048899100000000004 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635372+0000", 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9762000000000001e-05 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.915 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3597 2.4c 2.118008cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 254.14176084499999, 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98230618599999997, 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.916 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3597 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746119+0000", 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9999999999999995e-07 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746124+0000", 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3349999999999997e-06 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746136+0000", 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2339e-05 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746143+0000", 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3940000000000001e-06 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746201+0000", 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8529000000000002e-05 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746206+0000", 2024-09-16T05:20:40.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0250000000000002e-06 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728413+0000", 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98220633700000004 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728424+0000", 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1226999999999999e-05 2024-09-16T05:20:40.920 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3598 2.4c 2.118008cc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 254.14171928100001, 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.982592205, 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.921 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3598 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746160+0000", 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6899999999999997e-07 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746162+0000", 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.323e-06 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746170+0000", 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4339999999999992e-06 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746176+0000", 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746227+0000", 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1848999999999998e-05 2024-09-16T05:20:40.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746232+0000", 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5619999999999997e-06 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728741+0000", 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98250897100000001 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728751+0000", 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.10.0:319 2.4a e458/325 ondisk, result = 0)", 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 232.07140675299999, 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.733792216, 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:40.926 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816473+0000", 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.206e-06 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816480+0000", 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4789999999999999e-06 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550063+0000", 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73358232899999998 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550069+0000", 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3489999999999997e-06 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550155+0000", 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5490999999999999e-05 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550163+0000", 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1859999999999992e-06 2024-09-16T05:20:40.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550264+0000", 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010117599999999999 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3966 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 645534~783707] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 222.99887999000001, 2024-09-16T05:20:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.228671191, 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3966 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889000+0000", 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.745e-06 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889398+0000", 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000397644 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889415+0000", 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7368e-05 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889440+0000", 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4358999999999998e-05 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889496+0000", 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6168999999999997e-05 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889500+0000", 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0579999999999999e-06 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107487+0000", 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21798717200000001 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107565+0000", 2024-09-16T05:20:40.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7715999999999999e-05 2024-09-16T05:20:40.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107734+0000", 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016950099999999999 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111344+0000", 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036100160000000002 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117576+0000", 2024-09-16T05:20:40.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062321929999999996 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117628+0000", 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.189e-05 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117670+0000", 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1359999999999997e-05 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.937 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3967 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 2103939~638517] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 222.99195495999999, 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.221880365, 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3967 2024-09-16T05:20:40.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895924+0000", 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.85e-07 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896200+0000", 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027563199999999999 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896212+0000", 2024-09-16T05:20:40.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1671999999999999e-05 2024-09-16T05:20:40.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896217+0000", 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3970000000000002e-06 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896274+0000", 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6740000000000002e-05 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:40.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896280+0000", 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109156+0000", 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.212875278 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109242+0000", 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6163999999999997e-05 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109378+0000", 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013637400000000001 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111350+0000", 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019716429999999999 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117716+0000", 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063661830000000001 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117776+0000", 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9753999999999997e-05 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117804+0000", 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.817e-05 2024-09-16T05:20:40.948 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3968 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 3150663~294871] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 222.98827590400001, 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218327628, 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.949 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3968 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:20:40.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8800000000000002e-07 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899782+0000", 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017861999999999999 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899798+0000", 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5772000000000001e-05 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899806+0000", 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.157e-06 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899867+0000", 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0711000000000002e-05 2024-09-16T05:20:40.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899873+0000", 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0850000000000002e-06 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110009+0000", 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21013668299999999 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110084+0000", 2024-09-16T05:20:40.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4176000000000003e-05 2024-09-16T05:20:40.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:40.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110205+0000", 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012158 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111356+0000", 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001150646 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117849+0000", 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064928659999999999 2024-09-16T05:20:40.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117895+0000", 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6873999999999999e-05 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117930+0000", 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4669999999999998e-05 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3970 2.1d 2:ba7e58cf:::smithi20457400-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 222.98802731500001, 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218352989, 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3970 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.958 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899852+0000", 2024-09-16T05:20:40.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8299999999999997e-07 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899854+0000", 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.463e-06 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899861+0000", 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0079999999999996e-06 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899866+0000", 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899949+0000", 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2713000000000006e-05 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899954+0000", 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7400000000000004e-06 2024-09-16T05:20:40.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111261+0000", 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.211307514 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111332+0000", 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1198999999999994e-05 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118056+0000", 2024-09-16T05:20:40.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067238530000000001 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118121+0000", 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4770000000000005e-05 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118204+0000", 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3051000000000002e-05 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:4971 2.1d 2:ba7e58cf:::smithi20457400-26:head [stat,delete] snapc cc=[cb,ca,c9,c7,c5,c4,c3,c2,c1] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 157.68944587600001, 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.70425506000000004, 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.964 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4971 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198434+0000", 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8300000000000001e-06 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198443+0000", 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198460+0000", 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7229999999999999e-05 2024-09-16T05:20:40.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198467+0000", 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1409999999999999e-06 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198531+0000", 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3371e-05 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:20:40.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198539+0000", 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8930000000000005e-06 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891869+0000", 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69333044700000002 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891964+0000", 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4990999999999999e-05 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892092+0000", 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012792599999999999 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892842+0000", 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075013 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902619+0000", 2024-09-16T05:20:40.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0097773840000000001 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902658+0000", 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8167999999999998e-05 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902688+0000", 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9991000000000001e-05 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5379 2.1c 2:3a3554ba:::smithi20457400-48:head [setxattr _header (58) in=7b,truncate 2823741] snapc e0=[e0,df,dd,dc,db,da,d9,d8,d7,d4,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 120.09311171100001, 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46119776499999998, 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.971 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5379 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9500000000000002e-07 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794770+0000", 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9890000000000001e-06 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794783+0000", 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2281e-05 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794788+0000", 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.011e-06 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795397+0000", 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060981000000000002 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795462+0000", 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4869000000000005e-05 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795576+0000", 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001141 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795836+0000", 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026010999999999997 2024-09-16T05:20:40.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255870+0000", 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46003327399999999 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255915+0000", 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5064000000000001e-05 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255964+0000", 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9662e-05 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5380 2.1c 2:3a3554ba:::smithi20457400-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 120.093077157, 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46133110599999999, 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:40.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5380 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794802+0000", 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7599999999999998e-07 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794804+0000", 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0590000000000001e-06 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794810+0000", 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794815+0000", 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6700000000000002e-06 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795747+0000", 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093155799999999997 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795821+0000", 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3975999999999998e-05 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256004+0000", 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46018367199999999 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256065+0000", 2024-09-16T05:20:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0299e-05 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256132+0000", 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7776999999999998e-05 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:5379 2.1c e571/532 ondisk, result = 0)", 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 120.09104347100001, 2024-09-16T05:20:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45914334200000001, 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0800000000000001e-07 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796837+0000", 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9850000000000001e-06 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255760+0000", 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45892265399999999 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255769+0000", 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3449999999999995e-06 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255857+0000", 2024-09-16T05:20:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7547999999999996e-05 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255865+0000", 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6810000000000004e-06 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255978+0000", 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112821 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5901 2.1d 2:ba754fcb:::smithi20457400-30:head [stat,delete] snapc f3=[f3,f2,f0,ef,ee,ed,ec,e8,e7,e5,e0,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 82.984571235999994, 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79482897200000002, 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5901 2024-09-16T05:20:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903309+0000", 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4699999999999999e-06 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903314+0000", 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903330+0000", 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5770999999999999e-05 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903340+0000", 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0498e-05 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903402+0000", 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1385999999999998e-05 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903513+0000", 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111113 2024-09-16T05:20:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903707+0000", 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019455500000000001 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.904477+0000", 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076932900000000002 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698051+0000", 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79357449800000002 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698095+0000", 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3452999999999999e-05 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698136+0000", 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1482e-05 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.7.0:1084 2.3c e626/625 2:3d4bedc8:::smithi20457400-22:ee v 626'652, mlcod=624'646)", 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 63.413617727999998, 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45487590700000002, 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8800000000000002e-07 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474267+0000", 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928275+0000", 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45400790800000002 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928291+0000", 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5852999999999999e-05 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928455+0000", 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164026 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928479+0000", 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4258000000000001e-05 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929099+0000", 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000620117 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929137+0000", 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7639999999999999e-05 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6377 2.1d 2:ba21f1a9:::smithi20457400-14:head [setxattr _header (58) in=7b,truncate 3059072] snapc 107=[107,106,104,102,101,fb,fa,f9,f7,f2,e7] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.000842402000004, 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46418828400000001, 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6377 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887037+0000", 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:20:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887040+0000", 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5039999999999998e-06 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887047+0000", 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0330000000000003e-06 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887053+0000", 2024-09-16T05:20:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1430000000000003e-06 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888098+0000", 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001044866 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888199+0000", 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101143 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:20:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888364+0000", 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164403 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888873+0000", 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050974699999999996 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351123+0000", 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46224975499999998 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351169+0000", 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6316000000000002e-05 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351224+0000", 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4837999999999999e-05 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6378 2.1d 2:ba21f1a9:::smithi20457400-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.000815473999999, 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46436558900000002, 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6378 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887064+0000", 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3799999999999997e-07 2024-09-16T05:20:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887071+0000", 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9670000000000001e-06 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887077+0000", 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9739999999999999e-06 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887081+0000", 2024-09-16T05:20:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.405e-06 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888760+0000", 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016788319999999999 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888855+0000", 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5670999999999999e-05 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351265+0000", 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46241011799999998 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351353+0000", 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7416000000000005e-05 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351429+0000", 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5667999999999994e-05 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(1.7,GRANT,e:689,from:6,reservation_nonce:0)", 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.4305666370000001, 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388634586, 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:20:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:20:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457313+0000", 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0020000000000001e-06 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457317+0000", 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.279e-06 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845738+0000", 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388420879 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845745+0000", 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1849999999999998e-06 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845793+0000", 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8300000000000002e-05 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845814+0000", 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0353999999999999e-05 2024-09-16T05:20:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845946+0000", 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013258700000000001 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:20:41.013 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:20:41.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:41.014 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:41.037 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:41.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:41.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:20:41.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:20:41.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:20:41.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:20:41.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:20:41.038 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:20:41.038 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:20:41.038 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:20:41.038 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:40 smithi204 ceph-mon[30933]: osdmap e691: 12 total, 12 up, 11 in 2024-09-16T05:20:41.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:41.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:41.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:41.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:20:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:20:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:20:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:20:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:20:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:20:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:20:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:20:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:40 smithi196 ceph-mon[30802]: osdmap e691: 12 total, 12 up, 11 in 2024-09-16T05:20:41.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:20:41.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:20:41.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:20:41.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:20:41.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:20:41.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:20:41.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:20:41.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:20:41.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:40 smithi032 ceph-mon[27388]: osdmap e691: 12 total, 12 up, 11 in 2024-09-16T05:20:41.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:41.265 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:41.267 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:41.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:41.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:41.288 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:41.289 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:41.309 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:41.309 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:41.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:41.352 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:41.352 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:41.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:41.373 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:41.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:41.394 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:41.395 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:41.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:41.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:41.416 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:41.416 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:41.437 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:41.437 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:41.460 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:41.460 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:41.481 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:41.481 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:41.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:41.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:41.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:41.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:41.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:41.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:41.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:20:41.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:41.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:41.802 INFO:tasks.rados.rados.2.smithi204.stdout:2711: done (2 left) 2024-09-16T05:20:41.803 INFO:tasks.rados.rados.2.smithi204.stdout:2712: done (1 left) 2024-09-16T05:20:41.803 INFO:tasks.rados.rados.2.smithi204.stdout:2713: done (0 left) 2024-09-16T05:20:41.803 INFO:tasks.rados.rados.2.smithi204.stdout:2714: read oid 18 snap -1 2024-09-16T05:20:41.803 INFO:tasks.rados.rados.2.smithi204.stdout:2714: expect (ObjNum 955 snap 281 seq_num 955) 2024-09-16T05:20:41.803 INFO:tasks.rados.rados.2.smithi204.stdout:2715: write oid 20 current snap is 285 2024-09-16T05:20:41.803 INFO:tasks.rados.rados.2.smithi204.stdout:2715: seq_num 962 ranges {568334=650016,1860854=307480} 2024-09-16T05:20:41.810 INFO:tasks.rados.rados.2.smithi204.stdout:2715: writing smithi20457400-20 from 568334 to 1218350 tid 1 2024-09-16T05:20:41.816 INFO:tasks.rados.rados.2.smithi204.stdout:2715: writing smithi20457400-20 from 1860854 to 2168334 tid 2 2024-09-16T05:20:41.817 INFO:tasks.rados.rados.2.smithi204.stdout:2716: read oid 30 snap 282 2024-09-16T05:20:41.817 INFO:tasks.rados.rados.2.smithi204.stdout:2716: expect deleted 2024-09-16T05:20:41.817 INFO:tasks.rados.rados.2.smithi204.stdout:2717: write (excl) oid 38 current snap is 285 2024-09-16T05:20:41.817 INFO:tasks.rados.rados.2.smithi204.stdout:2717: seq_num 963 ranges {585141=725265,1919336=265805} 2024-09-16T05:20:41.825 INFO:tasks.rados.rados.2.smithi204.stdout:2717: writing smithi20457400-38 from 585141 to 1310406 tid 1 2024-09-16T05:20:41.831 INFO:tasks.rados.rados.2.smithi204.stdout:2717: writing smithi20457400-38 from 1919336 to 2185141 tid 2 2024-09-16T05:20:41.831 INFO:tasks.rados.rados.2.smithi204.stdout:2718: write (excl) oid 5 current snap is 285 2024-09-16T05:20:41.831 INFO:tasks.rados.rados.2.smithi204.stdout:2714: expect (ObjNum 955 snap 281 seq_num 955) 2024-09-16T05:20:41.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:41.907 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:41.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:41.920 INFO:tasks.rados.rados.2.smithi204.stdout:2715: finishing write tid 1 to smithi20457400-20 2024-09-16T05:20:41.920 INFO:tasks.rados.rados.2.smithi204.stdout:2715: finishing write tid 2 to smithi20457400-20 2024-09-16T05:20:41.920 INFO:tasks.rados.rados.2.smithi204.stdout:2717: finishing write tid 1 to smithi20457400-38 2024-09-16T05:20:41.920 INFO:tasks.rados.rados.2.smithi204.stdout:2717: finishing write tid 2 to smithi20457400-38 2024-09-16T05:20:41.920 INFO:tasks.rados.rados.2.smithi204.stdout:2717: finishing write tid 3 to smithi20457400-38 2024-09-16T05:20:41.921 INFO:tasks.rados.rados.2.smithi204.stdout:2717: finishing write tid 4 to smithi20457400-38 2024-09-16T05:20:41.921 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 913 (ObjNum 963 snap 285 seq_num 963) dirty exists 2024-09-16T05:20:41.921 INFO:tasks.rados.rados.2.smithi204.stdout:2717: left oid 38 (ObjNum 963 snap 285 seq_num 963) 2024-09-16T05:20:41.921 INFO:tasks.rados.rados.2.smithi204.stdout:2718: seq_num 964 ranges {601948=400514,1577818=624130} 2024-09-16T05:20:41.925 INFO:tasks.rados.rados.2.smithi204.stdout:2718: writing smithi20457400-5 from 601948 to 1002462 tid 1 2024-09-16T05:20:41.938 INFO:tasks.rados.rados.2.smithi204.stdout:2718: writing smithi20457400-5 from 1577818 to 2201948 tid 2 2024-09-16T05:20:41.938 INFO:tasks.rados.rados.2.smithi204.stdout:2714: done (4 left) 2024-09-16T05:20:41.938 INFO:tasks.rados.rados.2.smithi204.stdout:2716: done (3 left) 2024-09-16T05:20:41.938 INFO:tasks.rados.rados.2.smithi204.stdout:2717: done (2 left) 2024-09-16T05:20:41.938 INFO:tasks.rados.rados.2.smithi204.stdout:2719: delete oid 47 current snap is 285 2024-09-16T05:20:41.938 INFO:tasks.rados.rados.2.smithi204.stdout:2715: finishing write tid 3 to smithi20457400-20 2024-09-16T05:20:41.939 INFO:tasks.rados.rados.2.smithi204.stdout:2715: finishing write tid 4 to smithi20457400-20 2024-09-16T05:20:41.939 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1136 (ObjNum 962 snap 285 seq_num 962) dirty exists 2024-09-16T05:20:41.939 INFO:tasks.rados.rados.2.smithi204.stdout:2715: left oid 20 (ObjNum 962 snap 285 seq_num 962) 2024-09-16T05:20:41.939 INFO:tasks.rados.rados.2.smithi204.stdout:2718: finishing write tid 1 to smithi20457400-5 2024-09-16T05:20:41.939 INFO:tasks.rados.rados.2.smithi204.stdout:2715: done (2 left) 2024-09-16T05:20:41.939 INFO:tasks.rados.rados.2.smithi204.stdout:2719: done (1 left) 2024-09-16T05:20:41.939 INFO:tasks.rados.rados.2.smithi204.stdout:2720: write oid 48 current snap is 285 2024-09-16T05:20:41.939 INFO:tasks.rados.rados.2.smithi204.stdout:2720: seq_num 966 ranges {635562=667365,2094782=140780} 2024-09-16T05:20:41.942 INFO:tasks.rados.rados.2.smithi204.stdout:2720: writing smithi20457400-48 from 635562 to 1302927 tid 1 2024-09-16T05:20:41.944 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: pgmap v1197: 117 pgs: 1 active+clean+scrubbing, 1 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 305 MiB data, 908 MiB used, 983 GiB / 983 GiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 13 op/s 2024-09-16T05:20:41.944 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 2.53 scrub starts 2024-09-16T05:20:41.944 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 2.53 scrub ok 2024-09-16T05:20:41.945 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 1.15 scrub starts 2024-09-16T05:20:41.948 INFO:tasks.rados.rados.2.smithi204.stdout:2720: writing smithi20457400-48 from 2094782 to 2235562 tid 2 2024-09-16T05:20:41.948 INFO:tasks.rados.rados.2.smithi204.stdout:2721: delete oid 8 current snap is 285 2024-09-16T05:20:41.949 INFO:tasks.rados.rados.2.smithi204.stdout:2718: finishing write tid 2 to smithi20457400-5 2024-09-16T05:20:41.949 INFO:tasks.rados.rados.2.smithi204.stdout:2718: finishing write tid 3 to smithi20457400-5 2024-09-16T05:20:41.949 INFO:tasks.rados.rados.2.smithi204.stdout:2718: finishing write tid 4 to smithi20457400-5 2024-09-16T05:20:41.949 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 611 (ObjNum 964 snap 285 seq_num 964) dirty exists 2024-09-16T05:20:41.949 INFO:tasks.rados.rados.2.smithi204.stdout:2718: left oid 5 (ObjNum 964 snap 285 seq_num 964) 2024-09-16T05:20:41.949 INFO:tasks.rados.rados.2.smithi204.stdout:2720: finishing write tid 1 to smithi20457400-48 2024-09-16T05:20:41.950 INFO:tasks.rados.rados.2.smithi204.stdout:2718: done (2 left) 2024-09-16T05:20:41.951 INFO:tasks.rados.rados.2.smithi204.stdout:2721: done (1 left) 2024-09-16T05:20:41.951 INFO:tasks.rados.rados.2.smithi204.stdout:2722: write oid 47 current snap is 285 2024-09-16T05:20:41.951 INFO:tasks.rados.rados.2.smithi204.stdout:2720: finishing write tid 2 to smithi20457400-48 2024-09-16T05:20:41.951 INFO:tasks.rados.rados.2.smithi204.stdout:2722: seq_num 968 ranges {669176=417863,1811746=457430} 2024-09-16T05:20:41.955 INFO:tasks.rados.rados.2.smithi204.stdout:2722: writing smithi20457400-47 from 669176 to 1087039 tid 1 2024-09-16T05:20:41.962 INFO:tasks.rados.rados.2.smithi204.stdout:2722: writing smithi20457400-47 from 1811746 to 2269176 tid 2 2024-09-16T05:20:41.962 INFO:tasks.rados.rados.2.smithi204.stdout:2723: snap_create 2024-09-16T05:20:41.962 INFO:tasks.rados.rados.2.smithi204.stdout:2720: finishing write tid 3 to smithi20457400-48 2024-09-16T05:20:41.963 INFO:tasks.rados.rados.2.smithi204.stdout:2720: finishing write tid 4 to smithi20457400-48 2024-09-16T05:20:41.963 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 779 (ObjNum 966 snap 285 seq_num 966) dirty exists 2024-09-16T05:20:41.963 INFO:tasks.rados.rados.2.smithi204.stdout:2720: left oid 48 (ObjNum 966 snap 285 seq_num 966) 2024-09-16T05:20:41.963 INFO:tasks.rados.rados.2.smithi204.stdout:2722: finishing write tid 1 to smithi20457400-47 2024-09-16T05:20:41.965 INFO:tasks.rados.rados.2.smithi204.stdout:2722: finishing write tid 2 to smithi20457400-47 2024-09-16T05:20:41.965 INFO:tasks.rados.rados.2.smithi204.stdout:2722: finishing write tid 3 to smithi20457400-47 2024-09-16T05:20:41.965 INFO:tasks.rados.rados.2.smithi204.stdout:2722: finishing write tid 4 to smithi20457400-47 2024-09-16T05:20:41.966 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1032 (ObjNum 968 snap 285 seq_num 968) dirty exists 2024-09-16T05:20:41.966 INFO:tasks.rados.rados.2.smithi204.stdout:2722: left oid 47 (ObjNum 968 snap 285 seq_num 968) 2024-09-16T05:20:41.973 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:42.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:42.029 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:42.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:42.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:42.154 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:42.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: pgmap v1197: 117 pgs: 1 active+clean+scrubbing, 1 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 305 MiB data, 908 MiB used, 983 GiB / 983 GiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 13 op/s 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 2.53 scrub starts 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 2.53 scrub ok 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 1.15 scrub starts 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 1.15 scrub ok 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 1.1b deep-scrub starts 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 1.1b deep-scrub ok 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/806310160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 1.19 scrub starts 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 1.19 scrub ok 2024-09-16T05:20:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 1.16 deep-scrub starts 2024-09-16T05:20:42.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: 1.16 deep-scrub ok 2024-09-16T05:20:42.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:41 smithi196 ceph-mon[30802]: osdmap e692: 12 total, 12 up, 11 in 2024-09-16T05:20:42.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 1.15 scrub ok 2024-09-16T05:20:42.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 1.1b deep-scrub starts 2024-09-16T05:20:42.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 1.1b deep-scrub ok 2024-09-16T05:20:42.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/806310160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:42.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 1.19 scrub starts 2024-09-16T05:20:42.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 1.19 scrub ok 2024-09-16T05:20:42.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 1.16 deep-scrub starts 2024-09-16T05:20:42.254 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: 1.16 deep-scrub ok 2024-09-16T05:20:42.254 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:41 smithi204 ceph-mon[30933]: osdmap e692: 12 total, 12 up, 11 in 2024-09-16T05:20:42.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:42.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:42.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:42.292 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: pgmap v1197: 117 pgs: 1 active+clean+scrubbing, 1 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 305 MiB data, 908 MiB used, 983 GiB / 983 GiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 13 op/s 2024-09-16T05:20:42.292 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 2.53 scrub starts 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 2.53 scrub ok 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 1.15 scrub starts 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 1.15 scrub ok 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 1.1b deep-scrub starts 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 1.1b deep-scrub ok 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/806310160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 1.19 scrub starts 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 1.19 scrub ok 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 1.16 deep-scrub starts 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: 1.16 deep-scrub ok 2024-09-16T05:20:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:41 smithi032 ceph-mon[27388]: osdmap e692: 12 total, 12 up, 11 in 2024-09-16T05:20:42.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:42.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:42.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:42.460 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:42.460 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:42.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:42.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:42.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:42.565 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:20:42.565 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:20:42.565 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 692, 2024-09-16T05:20:42.565 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:20:41.794845+0000", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:14:38.892129+0000", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:19:34.779035+0000", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 36, 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.567 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:20:42.568 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 31, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 31, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 31, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 31, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 31, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:20:42.569 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "687", 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "685", 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:20:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:20:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:20:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:20:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.3199999332427979, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.3199999332427979, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.92000001668930054, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.130000114440918, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.130000114440918, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:20:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:20:42.575 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 86, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 86, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 86, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 86, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 86, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "692", 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "435", 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 286, 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 692, 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:20:42.577 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:20:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 1.7999999523162842, 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 1.7999999523162842, 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.92000001668930054, 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 1.6599999666213989, 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 1.6599999666213989, 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 689, 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.583 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.583 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:20:42.583 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:42.583 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.583 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.584 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.584 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.584 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.584 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:20:42.584 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:42.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:20:42.585 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.586 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:20:42.587 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:42.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:20:42.588 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 0, 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 0, 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.589 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 611, 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.590 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:20:42.594 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.595 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.596 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 21, 2024-09-16T05:20:42.597 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 295, 2024-09-16T05:20:42.597 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 329, 2024-09-16T05:20:42.597 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:42.597 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 296, 2024-09-16T05:20:42.597 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.598 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.598 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.598 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3973436581", 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3973436581", 2024-09-16T05:20:42.602 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3973436581", 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3973436581", 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.603 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 685, 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.608 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.609 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.612 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.613 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:20:42.614 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.615 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.616 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:20:42.619 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.620 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 689, 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:20:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:42.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.622 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:20:42.622 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:42.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.623 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:20:42.624 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:42.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:20:42.625 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:20:42.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 689, 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:42.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:42.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:42.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:20:42.631 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.632 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.633 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.635 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:20:42.638 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 689, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.639 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:20:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:20:42.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:42.645 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.645 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.645 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.645 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.645 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:20:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:42.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:42.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:42.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:20:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:09.291721+0000", 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 65536, 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:17.125178+0000", 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:18.686998+0000", 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 295 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:35.844565+0000", 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:32.848448+0000", 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:20.219477+0000", 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.661 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:13.190588+0000", 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.662 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:50.053405+0000", 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:40.278582+0000", 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:35.665165+0000", 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:20:42.664 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:30.643238+0000", 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:42.665 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:58.888327+0000", 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:42.666 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:20:42.667 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 262, 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 265, 2024-09-16T05:20:42.668 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:20:42.669 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:20:42.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:42.670 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:20:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:42.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:42.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:42.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:20:42.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:42.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:42.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:42.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:42.918 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:42.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:42.954 INFO:tasks.rados.rados.2.smithi204.stdout:2720: done (2 left) 2024-09-16T05:20:42.955 INFO:tasks.rados.rados.2.smithi204.stdout:2722: done (1 left) 2024-09-16T05:20:42.955 INFO:tasks.rados.rados.2.smithi204.stdout:2723: done (0 left) 2024-09-16T05:20:42.955 INFO:tasks.rados.rados.2.smithi204.stdout:2724: snap_remove snap 280 2024-09-16T05:20:43.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:43.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:43.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:43.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:20:43.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:43.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:43.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: 1.1a scrub starts 2024-09-16T05:20:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: 1.1a scrub ok 2024-09-16T05:20:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: 1.1d scrub starts 2024-09-16T05:20:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: 1.1d scrub ok 2024-09-16T05:20:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: 1.1c scrub starts 2024-09-16T05:20:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: 1.1c scrub ok 2024-09-16T05:20:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: 2.45 scrub starts 2024-09-16T05:20:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: 2.45 scrub ok 2024-09-16T05:20:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3247841647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:20:43.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:43.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:43.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:43.347 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:43.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:43.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:43.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:43.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: 1.1a scrub starts 2024-09-16T05:20:43.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: 1.1a scrub ok 2024-09-16T05:20:43.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: 1.1d scrub starts 2024-09-16T05:20:43.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: 1.1d scrub ok 2024-09-16T05:20:43.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: 1.1c scrub starts 2024-09-16T05:20:43.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: 1.1c scrub ok 2024-09-16T05:20:43.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: 2.45 scrub starts 2024-09-16T05:20:43.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: 2.45 scrub ok 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3247841647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: 1.1a scrub starts 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: 1.1a scrub ok 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: 1.1d scrub starts 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: 1.1d scrub ok 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: 1.1c scrub starts 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: 1.1c scrub ok 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: 2.45 scrub starts 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: 2.45 scrub ok 2024-09-16T05:20:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3247841647' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:20:43.451 INFO:tasks.thrashosds.thrasher:j is {'epoch': 692, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:20:41.794845+0000', 'last_up_change': '2024-09-16T05:14:38.892129+0000', 'last_in_change': '2024-09-16T05:19:34.779035+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 36, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '687', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '685', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 86, 'pg_placement_num': 86, 'pg_placement_num_target': 86, 'pg_num_target': 86, 'pg_num_pending': 86, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '692', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '435', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 286, 'snap_epoch': 692, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7999999523162842, 'score_stable': 1.7999999523162842, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.659999966621399, 'raw_score_stable': 1.659999966621399, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 21, 'last_clean_end': 295, 'up_from': 329, 'up_thru': 690, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3973436581}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3973436581}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3973436581}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3973436581}]}, 'public_addr': '172.21.15.32:6819/3973436581', 'cluster_addr': '172.21.15.32:6821/3973436581', 'heartbeat_back_addr': '172.21.15.32:6825/3973436581', 'heartbeat_front_addr': '172.21.15.32:6823/3973436581', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 685, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:09.291721+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T05:20:17.125178+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:18.686998+0000', 'dead_epoch': 295}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:35.844565+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:32.848448+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:20.219477+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:13.190588+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:50.053405+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:40.278582+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:35.665165+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:30.643238+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:58.888327+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 262, 'length': 1}, {'begin': 265, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:20:43.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:20:43.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:43.525 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:43.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:43.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:43.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:43.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:43.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:43.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:43.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:43.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:43.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:43.798 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:43.799 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:43.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:43.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:43.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:43.957 INFO:tasks.rados.rados.2.smithi204.stdout:2724: done (0 left) 2024-09-16T05:20:43.957 INFO:tasks.rados.rados.2.smithi204.stdout:2725: snap_remove snap 255 2024-09-16T05:20:44.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:44.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: pgmap v1200: 117 pgs: 4 unknown, 113 active+clean; 312 MiB data, 920 MiB used, 983 GiB / 983 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 18 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:20:44.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: 1.1e scrub starts 2024-09-16T05:20:44.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: 1.1e scrub ok 2024-09-16T05:20:44.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:20:44.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:20:44.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: osdmap e693: 12 total, 12 up, 11 in 2024-09-16T05:20:44.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: 1.17 scrub starts 2024-09-16T05:20:44.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: 1.17 scrub ok 2024-09-16T05:20:44.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: 2.54 scrub starts 2024-09-16T05:20:44.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:43 smithi032 ceph-mon[27388]: 2.54 scrub ok 2024-09-16T05:20:44.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:44.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:44.074 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:44.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: pgmap v1200: 117 pgs: 4 unknown, 113 active+clean; 312 MiB data, 920 MiB used, 983 GiB / 983 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 18 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:20:44.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: 1.1e scrub starts 2024-09-16T05:20:44.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: 1.1e scrub ok 2024-09-16T05:20:44.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:20:44.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:20:44.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: osdmap e693: 12 total, 12 up, 11 in 2024-09-16T05:20:44.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: 1.17 scrub starts 2024-09-16T05:20:44.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: 1.17 scrub ok 2024-09-16T05:20:44.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: 2.54 scrub starts 2024-09-16T05:20:44.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:43 smithi204 ceph-mon[30933]: 2.54 scrub ok 2024-09-16T05:20:44.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:44.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:20:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:44.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:44.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: pgmap v1200: 117 pgs: 4 unknown, 113 active+clean; 312 MiB data, 920 MiB used, 983 GiB / 983 GiB avail; 7.8 MiB/s rd, 2.9 MiB/s wr, 18 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: 1.1e scrub starts 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: 1.1e scrub ok 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: osdmap e693: 12 total, 12 up, 11 in 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: 1.17 scrub starts 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: 1.17 scrub ok 2024-09-16T05:20:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: 2.54 scrub starts 2024-09-16T05:20:44.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:43 smithi196 ceph-mon[30802]: 2.54 scrub ok 2024-09-16T05:20:44.275 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:44.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:44.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:44.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:44.414 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:20:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:44.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:44.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:44.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:44.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:44.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:44.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:44.653 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:20:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:44.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:44.741 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:20:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:20:44.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:44.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:44.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:44.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:44.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:44.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:44.960 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:20:44.966 INFO:tasks.rados.rados.2.smithi204.stdout:2725: done (0 left) 2024-09-16T05:20:44.966 INFO:tasks.rados.rados.2.smithi204.stdout:2726: read oid 21 snap 282 2024-09-16T05:20:44.966 INFO:tasks.rados.rados.2.smithi204.stdout:2726: expect deleted 2024-09-16T05:20:44.966 INFO:tasks.rados.rados.2.smithi204.stdout:2727: copy_from oid 37 from oid 40 current snap is 286 2024-09-16T05:20:44.966 INFO:tasks.rados.rados.2.smithi204.stdout:2728: snap_remove snap 277 2024-09-16T05:20:45.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:45.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:45.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:45.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:44 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:20:45.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:44 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:20:45.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:44 smithi204 ceph-mon[30933]: osdmap e694: 12 total, 12 up, 11 in 2024-09-16T05:20:45.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/546234615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:45.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:44 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:45.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:44 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:45.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:44 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:20:45.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:44 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:20:45.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:44 smithi032 ceph-mon[27388]: osdmap e694: 12 total, 12 up, 11 in 2024-09-16T05:20:45.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/546234615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:45.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:44 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:45.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:44 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:45.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:45.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:45.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:45.182 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:45.182 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:44 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:20:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:44 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:20:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:44 smithi196 ceph-mon[30802]: osdmap e694: 12 total, 12 up, 11 in 2024-09-16T05:20:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/546234615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:20:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:44 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:44 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:45.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:45.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:45.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:45.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:45.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:45.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:45.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:45.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:45.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:45.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:45.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:45.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:45.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:45.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:45.800 INFO:tasks.rados.rados.2.smithi204.stdout:2728: done (2 left) 2024-09-16T05:20:45.800 INFO:tasks.rados.rados.2.smithi204.stdout:2729: snap_create 2024-09-16T05:20:45.800 INFO:tasks.rados.rados.2.smithi204.stdout:2727: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:20:45.800 INFO:tasks.rados.rados.2.smithi204.stdout:2727: finishing copy_from to smithi20457400-37 2024-09-16T05:20:45.800 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 777 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:20:45.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:20:45.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:45.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:46.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:46.044 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:46.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:46.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:46.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:45 smithi032 ceph-mon[27388]: pgmap v1203: 117 pgs: 117 active+clean; 319 MiB data, 938 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 3.0 MiB/s wr, 29 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:20:46.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:45 smithi032 ceph-mon[27388]: from='client.16824 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:46.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:45 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:46.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/546234615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:46.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:45 smithi032 ceph-mon[27388]: osdmap e695: 12 total, 12 up, 11 in 2024-09-16T05:20:46.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1162849252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:46.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:45 smithi032 ceph-mon[27388]: osdmap e696: 12 total, 12 up, 11 in 2024-09-16T05:20:46.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:45 smithi204 ceph-mon[30933]: pgmap v1203: 117 pgs: 117 active+clean; 319 MiB data, 938 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 3.0 MiB/s wr, 29 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:20:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:45 smithi204 ceph-mon[30933]: from='client.16824 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:45 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/546234615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:45 smithi204 ceph-mon[30933]: osdmap e695: 12 total, 12 up, 11 in 2024-09-16T05:20:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1162849252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:45 smithi204 ceph-mon[30933]: osdmap e696: 12 total, 12 up, 11 in 2024-09-16T05:20:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:46.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:46.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:46.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:45 smithi196 ceph-mon[30802]: pgmap v1203: 117 pgs: 117 active+clean; 319 MiB data, 938 MiB used, 983 GiB / 983 GiB avail; 17 MiB/s rd, 3.0 MiB/s wr, 29 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:20:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:45 smithi196 ceph-mon[30802]: from='client.16824 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:20:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:45 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/546234615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:20:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:45 smithi196 ceph-mon[30802]: osdmap e695: 12 total, 12 up, 11 in 2024-09-16T05:20:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1162849252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:45 smithi196 ceph-mon[30802]: osdmap e696: 12 total, 12 up, 11 in 2024-09-16T05:20:46.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:46.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:46.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:46.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:46.427 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:46.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:46.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:46.555 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:46.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:46.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:46.685 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:46.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:46.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:46.802 INFO:tasks.rados.rados.2.smithi204.stdout:2726: done (2 left) 2024-09-16T05:20:46.802 INFO:tasks.rados.rados.2.smithi204.stdout:2727: done (1 left) 2024-09-16T05:20:46.802 INFO:tasks.rados.rados.2.smithi204.stdout:2729: done (0 left) 2024-09-16T05:20:46.802 INFO:tasks.rados.rados.2.smithi204.stdout:2730: snap_remove snap 282 2024-09-16T05:20:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:46.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:46.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:46.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:46.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:46.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:47.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:47.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:47.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:47.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:47.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:47.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:47.321 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1203,"stamp":"2024-09-16T05:20:44.184957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334688528,"num_objects":153,"num_object_clones":104,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":9,"num_read":3371,"num_read_kb":3325392,"num_write":2471,"num_write_kb":979233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":47124941,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29084,"ondisk_log_size":29084,"up":234,"acting":234,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":249,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":960892,"kb_used_data":507084,"kb_used_omap":26,"kb_used_meta":453541,"kb_avail":1030235780,"statfs":{"total":1055945392128,"available":1054961438720,"internally_reserved":0,"allocated":519254016,"data_stored":605490921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26910,"internal_metadata":464426722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14978484,"num_objects":10,"num_object_clones":9,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":-1,"num_read":72,"num_read_kb":68750,"num_write":47,"num_write_kb":12462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000895"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":3376,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800610+0000","last_change":"2024-09-16T05:20:38.930858+0000","last_active":"2024-09-16T05:20:40.800610+0000","last_peered":"2024-09-16T05:20:40.800610+0000","last_clean":"2024-09-16T05:20:40.800610+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:20:40.800610+0000","last_undegraded":"2024-09-16T05:20:40.800610+0000","last_fullsized":"2024-09-16T05:20:40.800610+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:20:01.826054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:01.826054+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:20.448061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066679800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.54","version":"686'398","reported_seq":2925,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.927874+0000","last_change":"2024-09-16T05:20:38.922310+0000","last_active":"2024-09-16T05:20:41.927874+0000","last_peered":"2024-09-16T05:20:41.927874+0000","last_clean":"2024-09-16T05:20:41.927874+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:20:41.927874+0000","last_undegraded":"2024-09-16T05:20:41.927874+0000","last_fullsized":"2024-09-16T05:20:41.927874+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"619'381","last_scrub_stamp":"2024-09-16T05:19:29.293042+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.293042+0000","objects_scrubbed":3,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.00087471200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":57049,"num_write":34,"num_write_kb":11523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":1275,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.803499+0000","last_change":"2024-09-16T05:20:40.312096+0000","last_active":"2024-09-16T05:20:40.803499+0000","last_peered":"2024-09-16T05:20:40.803499+0000","last_clean":"2024-09-16T05:20:40.803499+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:20:40.803499+0000","last_undegraded":"2024-09-16T05:20:40.803499+0000","last_fullsized":"2024-09-16T05:20:40.803499+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:20:40.312020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.312020+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.503619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098722799999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":1267,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.930175+0000","last_change":"2024-09-16T05:20:38.920262+0000","last_active":"2024-09-16T05:20:39.930175+0000","last_peered":"2024-09-16T05:20:39.930175+0000","last_clean":"2024-09-16T05:20:39.930175+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:20:39.930175+0000","last_undegraded":"2024-09-16T05:20:39.930175+0000","last_fullsized":"2024-09-16T05:20:39.930175+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:18.588272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.588272+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.673859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090963000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":1335,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805071+0000","last_change":"2024-09-16T05:20:38.922742+0000","last_active":"2024-09-16T05:20:40.805071+0000","last_peered":"2024-09-16T05:20:40.805071+0000","last_clean":"2024-09-16T05:20:40.805071+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:20:40.805071+0000","last_undegraded":"2024-09-16T05:20:40.805071+0000","last_fullsized":"2024-09-16T05:20:40.805071+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:19:29.589220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.589220+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:51.078821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074234500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":1269,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.949755+0000","last_change":"2024-09-16T05:20:38.918640+0000","last_active":"2024-09-16T05:20:41.949755+0000","last_peered":"2024-09-16T05:20:41.949755+0000","last_clean":"2024-09-16T05:20:41.949755+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:20:41.949755+0000","last_undegraded":"2024-09-16T05:20:41.949755+0000","last_fullsized":"2024-09-16T05:20:41.949755+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:29.302740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.302740+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:57.270217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073153300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2257,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805472+0000","last_change":"2024-09-16T05:20:38.932823+0000","last_active":"2024-09-16T05:20:40.805472+0000","last_peered":"2024-09-16T05:20:40.805472+0000","last_clean":"2024-09-16T05:20:40.805472+0000","last_became_active":"2024-09-16T05:19:35.789805+0000","last_became_peered":"2024-09-16T05:19:35.789805+0000","last_unstale":"2024-09-16T05:20:40.805472+0000","last_undegraded":"2024-09-16T05:20:40.805472+0000","last_fullsized":"2024-09-16T05:20:40.805472+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:19:48.302807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.302807+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:17.962275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0064454150000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.4e","version":"272'232","reported_seq":3037,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800765+0000","last_change":"2024-09-16T05:20:38.930439+0000","last_active":"2024-09-16T05:20:40.800765+0000","last_peered":"2024-09-16T05:20:40.800765+0000","last_clean":"2024-09-16T05:20:40.800765+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:20:40.800765+0000","last_undegraded":"2024-09-16T05:20:40.800765+0000","last_fullsized":"2024-09-16T05:20:40.800765+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:19:50.792260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.792260+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:11.524151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015069840000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.4d","version":"686'629","reported_seq":3596,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.928120+0000","last_change":"2024-09-16T05:20:38.923373+0000","last_active":"2024-09-16T05:20:41.928120+0000","last_peered":"2024-09-16T05:20:41.928120+0000","last_clean":"2024-09-16T05:20:41.928120+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:20:41.928120+0000","last_undegraded":"2024-09-16T05:20:41.928120+0000","last_fullsized":"2024-09-16T05:20:41.928120+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"665'626","last_scrub_stamp":"2024-09-16T05:20:16.492166+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.492166+0000","objects_scrubbed":4,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:38.487470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014774300000000001,"stat_sum":{"num_bytes":8433421,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":38966,"num_write":57,"num_write_kb":21533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.4c","version":"689'364","reported_seq":3265,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.806866+0000","last_change":"2024-09-16T05:20:38.930206+0000","last_active":"2024-09-16T05:20:40.806866+0000","last_peered":"2024-09-16T05:20:40.806866+0000","last_clean":"2024-09-16T05:20:40.806866+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:20:40.806866+0000","last_undegraded":"2024-09-16T05:20:40.806866+0000","last_fullsized":"2024-09-16T05:20:40.806866+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"666'345","last_scrub_stamp":"2024-09-16T05:20:17.450506+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.450506+0000","objects_scrubbed":1,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.534733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005614946,"stat_sum":{"num_bytes":5239557,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":37572,"num_write":33,"num_write_kb":9268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":3082,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.927789+0000","last_change":"2024-09-16T05:20:40.593201+0000","last_active":"2024-09-16T05:20:41.927789+0000","last_peered":"2024-09-16T05:20:41.927789+0000","last_clean":"2024-09-16T05:20:41.927789+0000","last_became_active":"2024-09-16T05:17:21.741180+0000","last_became_peered":"2024-09-16T05:17:21.741180+0000","last_unstale":"2024-09-16T05:20:41.927789+0000","last_undegraded":"2024-09-16T05:20:41.927789+0000","last_fullsized":"2024-09-16T05:20:41.927789+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:20:40.593078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.593078+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:43.730957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069667100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.4a","version":"691'561","reported_seq":3427,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.820095+0000","last_change":"2024-09-16T05:20:38.923372+0000","last_active":"2024-09-16T05:20:40.820095+0000","last_peered":"2024-09-16T05:20:40.820095+0000","last_clean":"2024-09-16T05:20:40.820095+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:20:40.820095+0000","last_undegraded":"2024-09-16T05:20:40.820095+0000","last_fullsized":"2024-09-16T05:20:40.820095+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"620'557","last_scrub_stamp":"2024-09-16T05:19:54.555606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.555606+0000","objects_scrubbed":1,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.378822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001013108,"stat_sum":{"num_bytes":3496854,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":41139,"num_write":39,"num_write_kb":19545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":3114,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800080+0000","last_change":"2024-09-16T05:20:38.929623+0000","last_active":"2024-09-16T05:20:40.800080+0000","last_peered":"2024-09-16T05:20:40.800080+0000","last_clean":"2024-09-16T05:20:40.800080+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:20:40.800080+0000","last_undegraded":"2024-09-16T05:20:40.800080+0000","last_fullsized":"2024-09-16T05:20:40.800080+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:20.094577+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.094577+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:48.097122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065918599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.48","version":"689'339","reported_seq":3201,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.932901+0000","last_change":"2024-09-16T05:20:38.933162+0000","last_active":"2024-09-16T05:20:39.932901+0000","last_peered":"2024-09-16T05:20:39.932901+0000","last_clean":"2024-09-16T05:20:39.932901+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:20:39.932901+0000","last_undegraded":"2024-09-16T05:20:39.932901+0000","last_fullsized":"2024-09-16T05:20:39.932901+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"628'287","last_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:19:38.000359+0000","objects_scrubbed":3,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.128554+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0070076959999999999,"stat_sum":{"num_bytes":8870985,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67118,"num_write":50,"num_write_kb":24235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":1885,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.930370+0000","last_change":"2024-09-16T05:20:38.922372+0000","last_active":"2024-09-16T05:20:39.930370+0000","last_peered":"2024-09-16T05:20:39.930370+0000","last_clean":"2024-09-16T05:20:39.930370+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:20:39.930370+0000","last_undegraded":"2024-09-16T05:20:39.930370+0000","last_fullsized":"2024-09-16T05:20:39.930370+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:30.390373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.390373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:44.975793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021741669999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":117,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.928003+0000","last_change":"2024-09-16T05:20:38.922681+0000","last_active":"2024-09-16T05:20:41.928003+0000","last_peered":"2024-09-16T05:20:41.928003+0000","last_clean":"2024-09-16T05:20:41.928003+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:20:41.928003+0000","last_undegraded":"2024-09-16T05:20:41.928003+0000","last_fullsized":"2024-09-16T05:20:41.928003+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:49.313870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.313870+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:11.361208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001215575,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":2852,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.806067+0000","last_change":"2024-09-16T05:20:38.926792+0000","last_active":"2024-09-16T05:20:40.806067+0000","last_peered":"2024-09-16T05:20:40.806067+0000","last_clean":"2024-09-16T05:20:40.806067+0000","last_became_active":"2024-09-16T05:19:35.786934+0000","last_became_peered":"2024-09-16T05:19:35.786934+0000","last_unstale":"2024-09-16T05:20:40.806067+0000","last_undegraded":"2024-09-16T05:20:40.806067+0000","last_fullsized":"2024-09-16T05:20:40.806067+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:19:09.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:09.927260+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023999830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":1473,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813694+0000","last_change":"2024-09-16T05:20:38.932185+0000","last_active":"2024-09-16T05:20:41.813694+0000","last_peered":"2024-09-16T05:20:41.813694+0000","last_clean":"2024-09-16T05:20:41.813694+0000","last_became_active":"2024-09-16T05:19:35.788828+0000","last_became_peered":"2024-09-16T05:19:35.788828+0000","last_unstale":"2024-09-16T05:20:41.813694+0000","last_undegraded":"2024-09-16T05:20:41.813694+0000","last_fullsized":"2024-09-16T05:20:41.813694+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:48.163046+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:39.970206+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.163046+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:05.886451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0043591159999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":1883,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805059+0000","last_change":"2024-09-16T05:20:38.931892+0000","last_active":"2024-09-16T05:20:40.805059+0000","last_peered":"2024-09-16T05:20:40.805059+0000","last_clean":"2024-09-16T05:20:40.805059+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:20:40.805059+0000","last_undegraded":"2024-09-16T05:20:40.805059+0000","last_fullsized":"2024-09-16T05:20:40.805059+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:20:38.464664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.464664+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:59.136732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005930785,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":1868,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.955973+0000","last_change":"2024-09-16T05:20:43.956385+0000","last_active":"2024-09-16T05:20:43.955973+0000","last_peered":"2024-09-16T05:20:43.955973+0000","last_clean":"2024-09-16T05:20:43.955973+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:20:43.955973+0000","last_undegraded":"2024-09-16T05:20:43.955973+0000","last_fullsized":"2024-09-16T05:20:43.955973+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:19:28.221037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.221037+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059119800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":725,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.930407+0000","last_change":"2024-09-16T05:20:38.919621+0000","last_active":"2024-09-16T05:20:39.930407+0000","last_peered":"2024-09-16T05:20:39.930407+0000","last_clean":"2024-09-16T05:20:39.930407+0000","last_became_active":"2024-09-16T05:18:19.768547+0000","last_became_peered":"2024-09-16T05:18:19.768547+0000","last_unstale":"2024-09-16T05:20:39.930407+0000","last_undegraded":"2024-09-16T05:20:39.930407+0000","last_fullsized":"2024-09-16T05:20:39.930407+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:19:55.515448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.515448+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:55.827944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00039832399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":425,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800030+0000","last_change":"2024-09-16T05:20:38.933661+0000","last_active":"2024-09-16T05:20:40.800030+0000","last_peered":"2024-09-16T05:20:40.800030+0000","last_clean":"2024-09-16T05:20:40.800030+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:20:40.800030+0000","last_undegraded":"2024-09-16T05:20:40.800030+0000","last_fullsized":"2024-09-16T05:20:40.800030+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:53.588594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.588594+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:59.070852+0000","scrub_duration":798,"objects_trimmed":0,"snaptrim_duration":0.001620699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.3f","version":"689'438","reported_seq":3347,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805720+0000","last_change":"2024-09-16T05:20:38.936386+0000","last_active":"2024-09-16T05:20:40.805720+0000","last_peered":"2024-09-16T05:20:40.805720+0000","last_clean":"2024-09-16T05:20:40.805720+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:20:40.805720+0000","last_undegraded":"2024-09-16T05:20:40.805720+0000","last_fullsized":"2024-09-16T05:20:40.805720+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'415","last_scrub_stamp":"2024-09-16T05:19:58.393481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:58.393481+0000","objects_scrubbed":2,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.256718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0097876869999999998,"stat_sum":{"num_bytes":7817141,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":94,"num_read_kb":82275,"num_write":54,"num_write_kb":15514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.3e","version":"688'490","reported_seq":3362,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813764+0000","last_change":"2024-09-16T05:20:38.932610+0000","last_active":"2024-09-16T05:20:41.813764+0000","last_peered":"2024-09-16T05:20:41.813764+0000","last_clean":"2024-09-16T05:20:41.813764+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:20:41.813764+0000","last_undegraded":"2024-09-16T05:20:41.813764+0000","last_fullsized":"2024-09-16T05:20:41.813764+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"625'487","last_scrub_stamp":"2024-09-16T05:20:22.404334+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:20:22.404334+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.453806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0034187810000000001,"stat_sum":{"num_bytes":2067492,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":41,"num_read_kb":35560,"num_write":45,"num_write_kb":24790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1814,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.958126+0000","last_change":"2024-09-16T05:20:43.958424+0000","last_active":"2024-09-16T05:20:43.958126+0000","last_peered":"2024-09-16T05:20:43.958126+0000","last_clean":"2024-09-16T05:20:43.958126+0000","last_became_active":"2024-09-16T05:19:35.784967+0000","last_became_peered":"2024-09-16T05:19:35.784967+0000","last_unstale":"2024-09-16T05:20:43.958126+0000","last_undegraded":"2024-09-16T05:20:43.958126+0000","last_fullsized":"2024-09-16T05:20:43.958126+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:19:36.333160+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.333160+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:00.416881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020867220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"2.3c","version":"689'734","reported_seq":3795,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805292+0000","last_change":"2024-09-16T05:20:38.932312+0000","last_active":"2024-09-16T05:20:40.805292+0000","last_peered":"2024-09-16T05:20:40.805292+0000","last_clean":"2024-09-16T05:20:40.805292+0000","last_became_active":"2024-09-16T05:19:35.800730+0000","last_became_peered":"2024-09-16T05:19:35.800730+0000","last_unstale":"2024-09-16T05:20:40.805292+0000","last_undegraded":"2024-09-16T05:20:40.805292+0000","last_fullsized":"2024-09-16T05:20:40.805292+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"624'648","last_scrub_stamp":"2024-09-16T05:19:34.258477+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:19:34.258477+0000","objects_scrubbed":6,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:44.090914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0056641110000000003,"stat_sum":{"num_bytes":15273880,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":171,"num_read_kb":225123,"num_write":81,"num_write_kb":32127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":6910647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.3b","version":"654'428","reported_seq":3322,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800722+0000","last_change":"2024-09-16T05:20:38.932714+0000","last_active":"2024-09-16T05:20:40.800722+0000","last_peered":"2024-09-16T05:20:40.800722+0000","last_clean":"2024-09-16T05:20:40.800722+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:20:40.800722+0000","last_undegraded":"2024-09-16T05:20:40.800722+0000","last_fullsized":"2024-09-16T05:20:40.800722+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'416","last_scrub_stamp":"2024-09-16T05:19:55.767476+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.767476+0000","objects_scrubbed":2,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:10.618334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092534099999999999,"stat_sum":{"num_bytes":8222944,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54616,"num_write":46,"num_write_kb":21574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.3a","version":"667'593","reported_seq":3445,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933317+0000","last_change":"2024-09-16T05:20:38.930513+0000","last_active":"2024-09-16T05:20:39.933317+0000","last_peered":"2024-09-16T05:20:39.933317+0000","last_clean":"2024-09-16T05:20:39.933317+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:20:39.933317+0000","last_undegraded":"2024-09-16T05:20:39.933317+0000","last_fullsized":"2024-09-16T05:20:39.933317+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"667'593","last_scrub_stamp":"2024-09-16T05:20:20.620494+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.620494+0000","objects_scrubbed":2,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:20.670468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001793717,"stat_sum":{"num_bytes":3949810,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":53,"num_read_kb":46839,"num_write":60,"num_write_kb":24043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":1965,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.932519+0000","last_change":"2024-09-16T05:20:38.929982+0000","last_active":"2024-09-16T05:20:39.932519+0000","last_peered":"2024-09-16T05:20:39.932519+0000","last_clean":"2024-09-16T05:20:39.932519+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:20:39.932519+0000","last_undegraded":"2024-09-16T05:20:39.932519+0000","last_fullsized":"2024-09-16T05:20:39.932519+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:19:49.888753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.888753+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:52.091396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015679579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":1989,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.957580+0000","last_change":"2024-09-16T05:20:43.958080+0000","last_active":"2024-09-16T05:20:43.957580+0000","last_peered":"2024-09-16T05:20:43.957580+0000","last_clean":"2024-09-16T05:20:43.957580+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:20:43.957580+0000","last_undegraded":"2024-09-16T05:20:43.957580+0000","last_fullsized":"2024-09-16T05:20:43.957580+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:20:20.396645+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.396645+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:30.406017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022992709999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":824,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.807535+0000","last_change":"2024-09-16T05:20:38.925587+0000","last_active":"2024-09-16T05:20:40.807535+0000","last_peered":"2024-09-16T05:20:40.807535+0000","last_clean":"2024-09-16T05:20:40.807535+0000","last_became_active":"2024-09-16T05:17:59.866037+0000","last_became_peered":"2024-09-16T05:17:59.866037+0000","last_unstale":"2024-09-16T05:20:40.807535+0000","last_undegraded":"2024-09-16T05:20:40.807535+0000","last_fullsized":"2024-09-16T05:20:40.807535+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:31.567784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:19:31.567784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001409682,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":342,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813919+0000","last_change":"2024-09-16T05:20:38.932927+0000","last_active":"2024-09-16T05:20:41.813919+0000","last_peered":"2024-09-16T05:20:41.813919+0000","last_clean":"2024-09-16T05:20:41.813919+0000","last_became_active":"2024-09-16T05:19:35.795592+0000","last_became_peered":"2024-09-16T05:19:35.795592+0000","last_unstale":"2024-09-16T05:20:41.813919+0000","last_undegraded":"2024-09-16T05:20:41.813919+0000","last_fullsized":"2024-09-16T05:20:41.813919+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:20:02.294024+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.294024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.155361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034987379999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.35","version":"692'775","reported_seq":3757,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.833272+0000","last_change":"2024-09-16T05:20:38.940408+0000","last_active":"2024-09-16T05:20:41.833272+0000","last_peered":"2024-09-16T05:20:41.833272+0000","last_clean":"2024-09-16T05:20:41.833272+0000","last_became_active":"2024-09-16T05:19:35.802234+0000","last_became_peered":"2024-09-16T05:19:35.802234+0000","last_unstale":"2024-09-16T05:20:41.833272+0000","last_undegraded":"2024-09-16T05:20:41.833272+0000","last_fullsized":"2024-09-16T05:20:41.833272+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"686'768","last_scrub_stamp":"2024-09-16T05:20:38.441947+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.441947+0000","objects_scrubbed":7,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:51.194593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0081007219999999994,"stat_sum":{"num_bytes":16371416,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":137,"num_read_kb":130417,"num_write":96,"num_write_kb":43328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3771362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.34","version":"689'615","reported_seq":3123,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800268+0000","last_change":"2024-09-16T05:20:38.934995+0000","last_active":"2024-09-16T05:20:40.800268+0000","last_peered":"2024-09-16T05:20:40.800268+0000","last_clean":"2024-09-16T05:20:40.800268+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:20:40.800268+0000","last_undegraded":"2024-09-16T05:20:40.800268+0000","last_fullsized":"2024-09-16T05:20:40.800268+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"686'611","last_scrub_stamp":"2024-09-16T05:20:38.002711+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.002711+0000","objects_scrubbed":7,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:57.758891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039275559999999996,"stat_sum":{"num_bytes":17356633,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":170,"num_read_kb":195067,"num_write":101,"num_write_kb":39821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":2057,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.804871+0000","last_change":"2024-09-16T05:20:38.932178+0000","last_active":"2024-09-16T05:20:40.804871+0000","last_peered":"2024-09-16T05:20:40.804871+0000","last_clean":"2024-09-16T05:20:40.804871+0000","last_became_active":"2024-09-16T05:19:35.789794+0000","last_became_peered":"2024-09-16T05:19:35.789794+0000","last_unstale":"2024-09-16T05:20:40.804871+0000","last_undegraded":"2024-09-16T05:20:40.804871+0000","last_fullsized":"2024-09-16T05:20:40.804871+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:16.451459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:16.451459+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062518549999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.32","version":"690'241","reported_seq":2292,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813999+0000","last_change":"2024-09-16T05:20:38.935776+0000","last_active":"2024-09-16T05:20:41.813999+0000","last_peered":"2024-09-16T05:20:41.813999+0000","last_clean":"2024-09-16T05:20:41.813999+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:20:41.813999+0000","last_undegraded":"2024-09-16T05:20:41.813999+0000","last_fullsized":"2024-09-16T05:20:41.813999+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"671'227","last_scrub_stamp":"2024-09-16T05:20:23.405472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.405472+0000","objects_scrubbed":2,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:51.741564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0060990269999999999,"stat_sum":{"num_bytes":7376174,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":101971,"num_write":48,"num_write_kb":21789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.31","version":"689'288","reported_seq":2915,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.930594+0000","last_change":"2024-09-16T05:20:38.922915+0000","last_active":"2024-09-16T05:20:39.930594+0000","last_peered":"2024-09-16T05:20:39.930594+0000","last_clean":"2024-09-16T05:20:39.930594+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:20:39.930594+0000","last_undegraded":"2024-09-16T05:20:39.930594+0000","last_fullsized":"2024-09-16T05:20:39.930594+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'268","last_scrub_stamp":"2024-09-16T05:20:03.547514+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.547514+0000","objects_scrubbed":2,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:07.584142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0035168339999999999,"stat_sum":{"num_bytes":3680626,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":116,"num_read_kb":119771,"num_write":39,"num_write_kb":19090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":812,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805054+0000","last_change":"2024-09-16T05:20:38.922312+0000","last_active":"2024-09-16T05:20:40.805054+0000","last_peered":"2024-09-16T05:20:40.805054+0000","last_clean":"2024-09-16T05:20:40.805054+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:20:40.805054+0000","last_undegraded":"2024-09-16T05:20:40.805054+0000","last_fullsized":"2024-09-16T05:20:40.805054+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:20:16.443442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.443442+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:31.584426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052954100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":1020,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800305+0000","last_change":"2024-09-16T05:20:18.091859+0000","last_active":"2024-09-16T05:20:40.800305+0000","last_peered":"2024-09-16T05:20:40.800305+0000","last_clean":"2024-09-16T05:20:40.800305+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:20:40.800305+0000","last_undegraded":"2024-09-16T05:20:40.800305+0000","last_fullsized":"2024-09-16T05:20:40.800305+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.091747+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.091747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.362844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":2613,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.800727+0000","last_change":"2024-09-16T05:20:38.923665+0000","last_active":"2024-09-16T05:20:41.800727+0000","last_peered":"2024-09-16T05:20:41.800727+0000","last_clean":"2024-09-16T05:20:41.800727+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:20:41.800727+0000","last_undegraded":"2024-09-16T05:20:41.800727+0000","last_fullsized":"2024-09-16T05:20:41.800727+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:20:03.076376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.076376+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:09.974173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084059,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1186,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813625+0000","last_change":"2024-09-16T05:19:36.252852+0000","last_active":"2024-09-16T05:20:41.813625+0000","last_peered":"2024-09-16T05:20:41.813625+0000","last_clean":"2024-09-16T05:20:41.813625+0000","last_became_active":"2024-09-16T05:19:35.788516+0000","last_became_peered":"2024-09-16T05:19:35.788516+0000","last_unstale":"2024-09-16T05:20:41.813625+0000","last_undegraded":"2024-09-16T05:20:41.813625+0000","last_fullsized":"2024-09-16T05:20:41.813625+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:36.252730+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.252730+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":2540,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.956535+0000","last_change":"2024-09-16T05:20:43.956732+0000","last_active":"2024-09-16T05:20:43.956535+0000","last_peered":"2024-09-16T05:20:43.956535+0000","last_clean":"2024-09-16T05:20:43.956535+0000","last_became_active":"2024-09-16T05:19:35.791894+0000","last_became_peered":"2024-09-16T05:19:35.791894+0000","last_unstale":"2024-09-16T05:20:43.956535+0000","last_undegraded":"2024-09-16T05:20:43.956535+0000","last_fullsized":"2024-09-16T05:20:43.956535+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:19:11.306770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.306770+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074652799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"1.c","version":"35'139","reported_seq":983,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.949923+0000","last_change":"2024-09-16T05:20:36.348317+0000","last_active":"2024-09-16T05:20:41.949923+0000","last_peered":"2024-09-16T05:20:41.949923+0000","last_clean":"2024-09-16T05:20:41.949923+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:20:41.949923+0000","last_undegraded":"2024-09-16T05:20:41.949923+0000","last_fullsized":"2024-09-16T05:20:41.949923+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.348227+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.348227+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.541309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"681'406","reported_seq":3328,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.956813+0000","last_change":"2024-09-16T05:20:43.957033+0000","last_active":"2024-09-16T05:20:43.956813+0000","last_peered":"2024-09-16T05:20:43.956813+0000","last_clean":"2024-09-16T05:20:43.956813+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:20:43.956813+0000","last_undegraded":"2024-09-16T05:20:43.956813+0000","last_fullsized":"2024-09-16T05:20:43.956813+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'384","last_scrub_stamp":"2024-09-16T05:19:15.295498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:15.295498+0000","objects_scrubbed":6,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098476800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":57662,"num_write":37,"num_write_kb":14770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"1.d","version":"35'139","reported_seq":1136,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805464+0000","last_change":"2024-09-16T05:20:37.488421+0000","last_active":"2024-09-16T05:20:40.805464+0000","last_peered":"2024-09-16T05:20:40.805464+0000","last_clean":"2024-09-16T05:20:40.805464+0000","last_became_active":"2024-09-16T05:20:35.804924+0000","last_became_peered":"2024-09-16T05:20:35.804924+0000","last_unstale":"2024-09-16T05:20:40.805464+0000","last_undegraded":"2024-09-16T05:20:40.805464+0000","last_fullsized":"2024-09-16T05:20:40.805464+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.488293+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.488293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.991686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":3040,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.928206+0000","last_change":"2024-09-16T05:20:39.564021+0000","last_active":"2024-09-16T05:20:41.928206+0000","last_peered":"2024-09-16T05:20:41.928206+0000","last_clean":"2024-09-16T05:20:41.928206+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:20:41.928206+0000","last_undegraded":"2024-09-16T05:20:41.928206+0000","last_fullsized":"2024-09-16T05:20:41.928206+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:20:39.563939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.563939+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:04.513735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001551537,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":1134,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.803219+0000","last_change":"2024-09-16T05:20:36.368708+0000","last_active":"2024-09-16T05:20:40.803219+0000","last_peered":"2024-09-16T05:20:40.803219+0000","last_clean":"2024-09-16T05:20:40.803219+0000","last_became_active":"2024-09-16T05:20:35.802192+0000","last_became_peered":"2024-09-16T05:20:35.802192+0000","last_unstale":"2024-09-16T05:20:40.803219+0000","last_undegraded":"2024-09-16T05:20:40.803219+0000","last_fullsized":"2024-09-16T05:20:40.803219+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.368583+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.368583+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.464774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"689'610","reported_seq":3517,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.934056+0000","last_change":"2024-09-16T05:20:38.934041+0000","last_active":"2024-09-16T05:20:39.934056+0000","last_peered":"2024-09-16T05:20:39.934056+0000","last_clean":"2024-09-16T05:20:39.934056+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:20:39.934056+0000","last_undegraded":"2024-09-16T05:20:39.934056+0000","last_fullsized":"2024-09-16T05:20:39.934056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"617'576","last_scrub_stamp":"2024-09-16T05:19:26.973559+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:19:26.973559+0000","objects_scrubbed":4,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003973898,"stat_sum":{"num_bytes":10232111,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":54,"num_read_kb":52780,"num_write":53,"num_write_kb":26527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":1115,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800196+0000","last_change":"2024-09-16T05:20:39.039439+0000","last_active":"2024-09-16T05:20:40.800196+0000","last_peered":"2024-09-16T05:20:40.800196+0000","last_clean":"2024-09-16T05:20:40.800196+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:20:40.800196+0000","last_undegraded":"2024-09-16T05:20:40.800196+0000","last_fullsized":"2024-09-16T05:20:40.800196+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:39.039365+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.039365+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.718627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3170,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813876+0000","last_change":"2024-09-16T05:20:39.456916+0000","last_active":"2024-09-16T05:20:41.813876+0000","last_peered":"2024-09-16T05:20:41.813876+0000","last_clean":"2024-09-16T05:20:41.813876+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:20:41.813876+0000","last_undegraded":"2024-09-16T05:20:41.813876+0000","last_fullsized":"2024-09-16T05:20:41.813876+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:20:39.456780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.456780+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.209766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00458918,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1661,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813784+0000","last_change":"2024-09-16T05:20:36.468183+0000","last_active":"2024-09-16T05:20:41.813784+0000","last_peered":"2024-09-16T05:20:41.813784+0000","last_clean":"2024-09-16T05:20:41.813784+0000","last_became_active":"2024-09-16T05:20:35.805506+0000","last_became_peered":"2024-09-16T05:20:35.805506+0000","last_unstale":"2024-09-16T05:20:41.813784+0000","last_undegraded":"2024-09-16T05:20:41.813784+0000","last_fullsized":"2024-09-16T05:20:41.813784+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.468079+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.468079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.101000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"666'325","reported_seq":3226,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805149+0000","last_change":"2024-09-16T05:20:38.925321+0000","last_active":"2024-09-16T05:20:40.805149+0000","last_peered":"2024-09-16T05:20:40.805149+0000","last_clean":"2024-09-16T05:20:40.805149+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:20:40.805149+0000","last_undegraded":"2024-09-16T05:20:40.805149+0000","last_fullsized":"2024-09-16T05:20:40.805149+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'325","last_scrub_stamp":"2024-09-16T05:20:23.462954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.462954+0000","objects_scrubbed":2,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.096089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00064173799999999999,"stat_sum":{"num_bytes":4366553,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":35425,"num_write":51,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.3","version":"687'213","reported_seq":3128,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800873+0000","last_change":"2024-09-16T05:20:38.934423+0000","last_active":"2024-09-16T05:20:40.800873+0000","last_peered":"2024-09-16T05:20:40.800873+0000","last_clean":"2024-09-16T05:20:40.800873+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:20:40.800873+0000","last_undegraded":"2024-09-16T05:20:40.800873+0000","last_fullsized":"2024-09-16T05:20:40.800873+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'205","last_scrub_stamp":"2024-09-16T05:20:19.137710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:19.137710+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:35.418031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001894125,"stat_sum":{"num_bytes":4352212,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":111577,"num_write":47,"num_write_kb":19395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":660,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813381+0000","last_change":"2024-09-16T05:20:00.246338+0000","last_active":"2024-09-16T05:20:41.813381+0000","last_peered":"2024-09-16T05:20:41.813381+0000","last_clean":"2024-09-16T05:20:41.813381+0000","last_became_active":"2024-09-16T05:18:19.768392+0000","last_became_peered":"2024-09-16T05:18:19.768392+0000","last_unstale":"2024-09-16T05:20:41.813381+0000","last_undegraded":"2024-09-16T05:20:41.813381+0000","last_fullsized":"2024-09-16T05:20:41.813381+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:00.246242+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.246242+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:13.060953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":2904,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800621+0000","last_change":"2024-09-16T05:20:38.929932+0000","last_active":"2024-09-16T05:20:40.800621+0000","last_peered":"2024-09-16T05:20:40.800621+0000","last_clean":"2024-09-16T05:20:40.800621+0000","last_became_active":"2024-09-16T05:19:35.791085+0000","last_became_peered":"2024-09-16T05:19:35.791085+0000","last_unstale":"2024-09-16T05:20:40.800621+0000","last_undegraded":"2024-09-16T05:20:40.800621+0000","last_fullsized":"2024-09-16T05:20:40.800621+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:02.847979+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.847979+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.643554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068034699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":1206,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805208+0000","last_change":"2024-09-16T05:19:30.558195+0000","last_active":"2024-09-16T05:20:40.805208+0000","last_peered":"2024-09-16T05:20:40.805208+0000","last_clean":"2024-09-16T05:20:40.805208+0000","last_became_active":"2024-09-16T05:18:02.888512+0000","last_became_peered":"2024-09-16T05:18:02.888512+0000","last_unstale":"2024-09-16T05:20:40.805208+0000","last_undegraded":"2024-09-16T05:20:40.805208+0000","last_fullsized":"2024-09-16T05:20:40.805208+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:30.558069+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:30.558069+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:54.576886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1636,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805000+0000","last_change":"2024-09-16T05:19:56.353335+0000","last_active":"2024-09-16T05:20:40.805000+0000","last_peered":"2024-09-16T05:20:40.805000+0000","last_clean":"2024-09-16T05:20:40.805000+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:20:40.805000+0000","last_undegraded":"2024-09-16T05:20:40.805000+0000","last_fullsized":"2024-09-16T05:20:40.805000+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:19:56.353243+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.757169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":2849,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.927781+0000","last_change":"2024-09-16T05:20:38.923604+0000","last_active":"2024-09-16T05:20:41.927781+0000","last_peered":"2024-09-16T05:20:41.927781+0000","last_clean":"2024-09-16T05:20:41.927781+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:20:41.927781+0000","last_undegraded":"2024-09-16T05:20:41.927781+0000","last_fullsized":"2024-09-16T05:20:41.927781+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:25.536951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:25.536951+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:26.639867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017025339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1651,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.949836+0000","last_change":"2024-09-16T05:20:37.311421+0000","last_active":"2024-09-16T05:20:41.949836+0000","last_peered":"2024-09-16T05:20:41.949836+0000","last_clean":"2024-09-16T05:20:41.949836+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:20:41.949836+0000","last_undegraded":"2024-09-16T05:20:41.949836+0000","last_fullsized":"2024-09-16T05:20:41.949836+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.311338+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.311338+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:37.446300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":2927,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805348+0000","last_change":"2024-09-16T05:20:38.926735+0000","last_active":"2024-09-16T05:20:40.805348+0000","last_peered":"2024-09-16T05:20:40.805348+0000","last_clean":"2024-09-16T05:20:40.805348+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:20:40.805348+0000","last_undegraded":"2024-09-16T05:20:40.805348+0000","last_fullsized":"2024-09-16T05:20:40.805348+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:51.365804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.365804+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:08.054690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017803249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":2136,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933961+0000","last_change":"2024-09-16T05:20:38.936836+0000","last_active":"2024-09-16T05:20:39.933961+0000","last_peered":"2024-09-16T05:20:39.933961+0000","last_clean":"2024-09-16T05:20:39.933961+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:20:39.933961+0000","last_undegraded":"2024-09-16T05:20:39.933961+0000","last_fullsized":"2024-09-16T05:20:39.933961+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:50.852123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.852123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:57.023237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058767239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":1604,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813812+0000","last_change":"2024-09-16T05:19:53.322398+0000","last_active":"2024-09-16T05:20:41.813812+0000","last_peered":"2024-09-16T05:20:41.813812+0000","last_clean":"2024-09-16T05:20:41.813812+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:20:41.813812+0000","last_undegraded":"2024-09-16T05:20:41.813812+0000","last_fullsized":"2024-09-16T05:20:41.813812+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:53.322303+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.322303+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:12.744427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1324,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.928116+0000","last_change":"2024-09-16T05:20:38.922754+0000","last_active":"2024-09-16T05:20:41.928116+0000","last_peered":"2024-09-16T05:20:41.928116+0000","last_clean":"2024-09-16T05:20:41.928116+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:20:41.928116+0000","last_undegraded":"2024-09-16T05:20:41.928116+0000","last_fullsized":"2024-09-16T05:20:41.928116+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:53.382868+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:18:27.279121+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.382868+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.212427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012745879999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":923,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.807413+0000","last_change":"2024-09-16T05:20:38.411870+0000","last_active":"2024-09-16T05:20:40.807413+0000","last_peered":"2024-09-16T05:20:40.807413+0000","last_clean":"2024-09-16T05:20:40.807413+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:20:40.807413+0000","last_undegraded":"2024-09-16T05:20:40.807413+0000","last_fullsized":"2024-09-16T05:20:40.807413+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:38.411767+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.411767+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:52.750017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"689'593","reported_seq":3512,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.957829+0000","last_change":"2024-09-16T05:20:43.958352+0000","last_active":"2024-09-16T05:20:43.957829+0000","last_peered":"2024-09-16T05:20:43.957829+0000","last_clean":"2024-09-16T05:20:43.957829+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:20:43.957829+0000","last_undegraded":"2024-09-16T05:20:43.957829+0000","last_fullsized":"2024-09-16T05:20:43.957829+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'568","last_scrub_stamp":"2024-09-16T05:19:54.247232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.247232+0000","objects_scrubbed":3,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:03.591222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019756790000000002,"stat_sum":{"num_bytes":7659108,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":107857,"num_write":39,"num_write_kb":12889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":867,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805006+0000","last_change":"2024-09-16T05:20:37.414043+0000","last_active":"2024-09-16T05:20:40.805006+0000","last_peered":"2024-09-16T05:20:40.805006+0000","last_clean":"2024-09-16T05:20:40.805006+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:20:40.805006+0000","last_undegraded":"2024-09-16T05:20:40.805006+0000","last_fullsized":"2024-09-16T05:20:40.805006+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.413968+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.413968+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.433963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":781,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.930822+0000","last_change":"2024-09-16T05:20:38.920720+0000","last_active":"2024-09-16T05:20:39.930822+0000","last_peered":"2024-09-16T05:20:39.930822+0000","last_clean":"2024-09-16T05:20:39.930822+0000","last_became_active":"2024-09-16T05:19:35.793136+0000","last_became_peered":"2024-09-16T05:19:35.793136+0000","last_unstale":"2024-09-16T05:20:39.930822+0000","last_undegraded":"2024-09-16T05:20:39.930822+0000","last_fullsized":"2024-09-16T05:20:39.930822+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:20:38.827535+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.827535+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:49.149337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086850999999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":903,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.807362+0000","last_change":"2024-09-16T05:20:39.847071+0000","last_active":"2024-09-16T05:20:40.807362+0000","last_peered":"2024-09-16T05:20:40.807362+0000","last_clean":"2024-09-16T05:20:40.807362+0000","last_became_active":"2024-09-16T05:20:35.811624+0000","last_became_peered":"2024-09-16T05:20:35.811624+0000","last_unstale":"2024-09-16T05:20:40.807362+0000","last_undegraded":"2024-09-16T05:20:40.807362+0000","last_fullsized":"2024-09-16T05:20:40.807362+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:39.846968+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.846968+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:46.110945+0000","scrub_duration":389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":3087,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.956480+0000","last_change":"2024-09-16T05:20:43.957740+0000","last_active":"2024-09-16T05:20:43.956480+0000","last_peered":"2024-09-16T05:20:43.956480+0000","last_clean":"2024-09-16T05:20:43.956480+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:20:43.956480+0000","last_undegraded":"2024-09-16T05:20:43.956480+0000","last_fullsized":"2024-09-16T05:20:43.956480+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:19:29.221207+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.221207+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:51.978391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001767663,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":1650,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800191+0000","last_change":"2024-09-16T05:20:36.004657+0000","last_active":"2024-09-16T05:20:40.800191+0000","last_peered":"2024-09-16T05:20:40.800191+0000","last_clean":"2024-09-16T05:20:40.800191+0000","last_became_active":"2024-09-16T05:20:35.807550+0000","last_became_peered":"2024-09-16T05:20:35.807550+0000","last_unstale":"2024-09-16T05:20:40.800191+0000","last_undegraded":"2024-09-16T05:20:40.800191+0000","last_fullsized":"2024-09-16T05:20:40.800191+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.004558+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.004558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.268598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"689'630","reported_seq":3562,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813714+0000","last_change":"2024-09-16T05:20:38.937697+0000","last_active":"2024-09-16T05:20:41.813714+0000","last_peered":"2024-09-16T05:20:41.813714+0000","last_clean":"2024-09-16T05:20:41.813714+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:20:41.813714+0000","last_undegraded":"2024-09-16T05:20:41.813714+0000","last_fullsized":"2024-09-16T05:20:41.813714+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"653'601","last_scrub_stamp":"2024-09-16T05:20:03.311152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.311152+0000","objects_scrubbed":3,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.529508+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0079923340000000002,"stat_sum":{"num_bytes":6944569,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":127393,"num_write":54,"num_write_kb":33730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1647,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.804946+0000","last_change":"2024-09-16T05:20:37.260334+0000","last_active":"2024-09-16T05:20:40.804946+0000","last_peered":"2024-09-16T05:20:40.804946+0000","last_clean":"2024-09-16T05:20:40.804946+0000","last_became_active":"2024-09-16T05:20:35.804221+0000","last_became_peered":"2024-09-16T05:20:35.804221+0000","last_unstale":"2024-09-16T05:20:40.804946+0000","last_undegraded":"2024-09-16T05:20:40.804946+0000","last_fullsized":"2024-09-16T05:20:40.804946+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.260191+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.260191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:55.461837+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1814,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.803381+0000","last_change":"2024-09-16T05:20:38.921768+0000","last_active":"2024-09-16T05:20:40.803381+0000","last_peered":"2024-09-16T05:20:40.803381+0000","last_clean":"2024-09-16T05:20:40.803381+0000","last_became_active":"2024-09-16T05:19:35.786308+0000","last_became_peered":"2024-09-16T05:19:35.786308+0000","last_unstale":"2024-09-16T05:20:40.803381+0000","last_undegraded":"2024-09-16T05:20:40.803381+0000","last_fullsized":"2024-09-16T05:20:40.803381+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:38.316049+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.316049+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:48.490104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049697800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1643,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933275+0000","last_change":"2024-09-16T05:20:35.832422+0000","last_active":"2024-09-16T05:20:39.933275+0000","last_peered":"2024-09-16T05:20:39.933275+0000","last_clean":"2024-09-16T05:20:39.933275+0000","last_became_active":"2024-09-16T05:20:35.812754+0000","last_became_peered":"2024-09-16T05:20:35.812754+0000","last_unstale":"2024-09-16T05:20:39.933275+0000","last_undegraded":"2024-09-16T05:20:39.933275+0000","last_fullsized":"2024-09-16T05:20:39.933275+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:35.832353+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:20:35.832353+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:43.196095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1122,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.930545+0000","last_change":"2024-09-16T05:19:49.535280+0000","last_active":"2024-09-16T05:20:39.930545+0000","last_peered":"2024-09-16T05:20:39.930545+0000","last_clean":"2024-09-16T05:20:39.930545+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:20:39.930545+0000","last_undegraded":"2024-09-16T05:20:39.930545+0000","last_fullsized":"2024-09-16T05:20:39.930545+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:49.535206+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.535206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:53.471649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":812,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805220+0000","last_change":"2024-09-16T05:20:38.922795+0000","last_active":"2024-09-16T05:20:40.805220+0000","last_peered":"2024-09-16T05:20:40.805220+0000","last_clean":"2024-09-16T05:20:40.805220+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:20:40.805220+0000","last_undegraded":"2024-09-16T05:20:40.805220+0000","last_fullsized":"2024-09-16T05:20:40.805220+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:24.333369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:24.333369+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.919563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068025900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1639,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800011+0000","last_change":"2024-09-16T05:19:48.737475+0000","last_active":"2024-09-16T05:20:40.800011+0000","last_peered":"2024-09-16T05:20:40.800011+0000","last_clean":"2024-09-16T05:20:40.800011+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:20:40.800011+0000","last_undegraded":"2024-09-16T05:20:40.800011+0000","last_fullsized":"2024-09-16T05:20:40.800011+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:48.737348+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.737348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:06.914437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":2517,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933850+0000","last_change":"2024-09-16T05:20:38.929411+0000","last_active":"2024-09-16T05:20:39.933850+0000","last_peered":"2024-09-16T05:20:39.933850+0000","last_clean":"2024-09-16T05:20:39.933850+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:20:39.933850+0000","last_undegraded":"2024-09-16T05:20:39.933850+0000","last_fullsized":"2024-09-16T05:20:39.933850+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:19:36.009233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:36.009233+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016299070000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"1.17","version":"35'139","reported_seq":534,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.955913+0000","last_change":"2024-09-16T05:20:43.198542+0000","last_active":"2024-09-16T05:20:43.955913+0000","last_peered":"2024-09-16T05:20:43.955913+0000","last_clean":"2024-09-16T05:20:43.955913+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:20:43.955913+0000","last_undegraded":"2024-09-16T05:20:43.955913+0000","last_fullsized":"2024-09-16T05:20:43.955913+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:43.198473+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:20:43.198473+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.563368+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":2797,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933620+0000","last_change":"2024-09-16T05:20:38.931289+0000","last_active":"2024-09-16T05:20:39.933620+0000","last_peered":"2024-09-16T05:20:39.933620+0000","last_clean":"2024-09-16T05:20:39.933620+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:20:39.933620+0000","last_undegraded":"2024-09-16T05:20:39.933620+0000","last_fullsized":"2024-09-16T05:20:39.933620+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:20:37.810519+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.810519+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.344271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001931704,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"1.16","version":"35'139","reported_seq":587,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.927972+0000","last_change":"2024-09-16T05:20:41.564653+0000","last_active":"2024-09-16T05:20:41.927972+0000","last_peered":"2024-09-16T05:20:41.927972+0000","last_clean":"2024-09-16T05:20:41.927972+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:20:41.927972+0000","last_undegraded":"2024-09-16T05:20:41.927972+0000","last_fullsized":"2024-09-16T05:20:41.927972+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.564525+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:54.885944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"689'645","reported_seq":3626,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933640+0000","last_change":"2024-09-16T05:20:39.846517+0000","last_active":"2024-09-16T05:20:39.933640+0000","last_peered":"2024-09-16T05:20:39.933640+0000","last_clean":"2024-09-16T05:20:39.933640+0000","last_became_active":"2024-09-16T05:19:35.801137+0000","last_became_peered":"2024-09-16T05:19:35.801137+0000","last_unstale":"2024-09-16T05:20:39.933640+0000","last_undegraded":"2024-09-16T05:20:39.933640+0000","last_fullsized":"2024-09-16T05:20:39.933640+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"689'645","last_scrub_stamp":"2024-09-16T05:20:39.846432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.846432+0000","objects_scrubbed":3,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:57.844463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0067966779999999996,"stat_sum":{"num_bytes":3964937,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":5506324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"1.15","version":"35'139","reported_seq":603,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.949219+0000","last_change":"2024-09-16T05:20:40.949219+0000","last_active":"2024-09-16T05:20:40.949219+0000","last_peered":"2024-09-16T05:20:40.949219+0000","last_clean":"2024-09-16T05:20:40.949219+0000","last_became_active":"2024-09-16T05:20:40.802090+0000","last_became_peered":"2024-09-16T05:20:40.802090+0000","last_unstale":"2024-09-16T05:20:40.949219+0000","last_undegraded":"2024-09-16T05:20:40.949219+0000","last_fullsized":"2024-09-16T05:20:40.949219+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:40.949147+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.949147+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.611229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"689'304","reported_seq":2845,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.800701+0000","last_change":"2024-09-16T05:20:38.930873+0000","last_active":"2024-09-16T05:20:40.800701+0000","last_peered":"2024-09-16T05:20:40.800701+0000","last_clean":"2024-09-16T05:20:40.800701+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:20:40.800701+0000","last_undegraded":"2024-09-16T05:20:40.800701+0000","last_fullsized":"2024-09-16T05:20:40.800701+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"665'294","last_scrub_stamp":"2024-09-16T05:20:15.010850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.010850+0000","objects_scrubbed":4,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.012948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000771504,"stat_sum":{"num_bytes":8360620,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":63,"num_read_kb":57071,"num_write":54,"num_write_kb":17613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":969,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.927992+0000","last_change":"2024-09-16T05:20:37.550357+0000","last_active":"2024-09-16T05:20:41.927992+0000","last_peered":"2024-09-16T05:20:41.927992+0000","last_clean":"2024-09-16T05:20:41.927992+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:20:41.927992+0000","last_undegraded":"2024-09-16T05:20:41.927992+0000","last_fullsized":"2024-09-16T05:20:41.927992+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.550279+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.550279+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.895726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":562,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.807241+0000","last_change":"2024-09-16T05:20:38.924391+0000","last_active":"2024-09-16T05:20:40.807241+0000","last_peered":"2024-09-16T05:20:40.807241+0000","last_clean":"2024-09-16T05:20:40.807241+0000","last_became_active":"2024-09-16T05:19:35.784084+0000","last_became_peered":"2024-09-16T05:19:35.784084+0000","last_unstale":"2024-09-16T05:20:40.807241+0000","last_undegraded":"2024-09-16T05:20:40.807241+0000","last_fullsized":"2024-09-16T05:20:40.807241+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:20:20.393358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.393358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.521997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049122200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.1b","version":"35'139","reported_seq":1059,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.955845+0000","last_change":"2024-09-16T05:20:41.173883+0000","last_active":"2024-09-16T05:20:43.955845+0000","last_peered":"2024-09-16T05:20:43.955845+0000","last_clean":"2024-09-16T05:20:43.955845+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:20:43.955845+0000","last_undegraded":"2024-09-16T05:20:43.955845+0000","last_fullsized":"2024-09-16T05:20:43.955845+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.173808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:00.160931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":2899,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933104+0000","last_change":"2024-09-16T05:20:38.930267+0000","last_active":"2024-09-16T05:20:39.933104+0000","last_peered":"2024-09-16T05:20:39.933104+0000","last_clean":"2024-09-16T05:20:39.933104+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:20:39.933104+0000","last_undegraded":"2024-09-16T05:20:39.933104+0000","last_fullsized":"2024-09-16T05:20:39.933104+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:19:47.951135+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:19:47.951135+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:58.195442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023034290000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"1.1a","version":"35'139","reported_seq":410,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.807137+0000","last_change":"2024-09-16T05:20:40.806574+0000","last_active":"2024-09-16T05:20:40.807137+0000","last_peered":"2024-09-16T05:20:40.807137+0000","last_clean":"2024-09-16T05:20:40.807137+0000","last_became_active":"2024-09-16T05:20:40.806260+0000","last_became_peered":"2024-09-16T05:20:40.806260+0000","last_unstale":"2024-09-16T05:20:40.807137+0000","last_undegraded":"2024-09-16T05:20:40.807137+0000","last_fullsized":"2024-09-16T05:20:40.807137+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.790944+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.790944+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":2520,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.949909+0000","last_change":"2024-09-16T05:20:38.919004+0000","last_active":"2024-09-16T05:20:41.949909+0000","last_peered":"2024-09-16T05:20:41.949909+0000","last_clean":"2024-09-16T05:20:41.949909+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:20:41.949909+0000","last_undegraded":"2024-09-16T05:20:41.949909+0000","last_fullsized":"2024-09-16T05:20:41.949909+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:20:17.403045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.403045+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:29.106063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00082444599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.19","version":"35'139","reported_seq":1640,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.808286+0000","last_change":"2024-09-16T05:20:40.805422+0000","last_active":"2024-09-16T05:20:40.808286+0000","last_peered":"2024-09-16T05:20:40.808286+0000","last_clean":"2024-09-16T05:20:40.808286+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:20:40.808286+0000","last_undegraded":"2024-09-16T05:20:40.808286+0000","last_fullsized":"2024-09-16T05:20:40.808286+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:02.384248+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.384248+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"690'1049","reported_seq":3918,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.956051+0000","last_change":"2024-09-16T05:20:43.956412+0000","last_active":"2024-09-16T05:20:43.956051+0000","last_peered":"2024-09-16T05:20:43.956051+0000","last_clean":"2024-09-16T05:20:43.956051+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:20:43.956051+0000","last_undegraded":"2024-09-16T05:20:43.956051+0000","last_fullsized":"2024-09-16T05:20:43.956051+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"653'957","last_scrub_stamp":"2024-09-16T05:20:03.383437+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.383437+0000","objects_scrubbed":11,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:32.040248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051559100000000003,"stat_sum":{"num_bytes":22504913,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":167,"num_read_kb":148606,"num_write":170,"num_write_kb":62122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"1.18","version":"35'139","reported_seq":1651,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.799900+0000","last_change":"2024-09-16T05:20:37.024610+0000","last_active":"2024-09-16T05:20:40.799900+0000","last_peered":"2024-09-16T05:20:40.799900+0000","last_clean":"2024-09-16T05:20:40.799900+0000","last_became_active":"2024-09-16T05:20:35.809442+0000","last_became_peered":"2024-09-16T05:20:35.809442+0000","last_unstale":"2024-09-16T05:20:40.799900+0000","last_undegraded":"2024-09-16T05:20:40.799900+0000","last_fullsized":"2024-09-16T05:20:40.799900+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.024521+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.024521+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.640487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"689'644","reported_seq":3575,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.814090+0000","last_change":"2024-09-16T05:20:38.937457+0000","last_active":"2024-09-16T05:20:41.814090+0000","last_peered":"2024-09-16T05:20:41.814090+0000","last_clean":"2024-09-16T05:20:41.814090+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:20:41.814090+0000","last_undegraded":"2024-09-16T05:20:41.814090+0000","last_fullsized":"2024-09-16T05:20:41.814090+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"666'612","last_scrub_stamp":"2024-09-16T05:20:16.258329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.258329+0000","objects_scrubbed":4,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:32.457007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0089940539999999996,"stat_sum":{"num_bytes":15530986,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":105,"num_read_kb":107027,"num_write":107,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.1c","version":"664'452","reported_seq":3052,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.806633+0000","last_change":"2024-09-16T05:20:38.929421+0000","last_active":"2024-09-16T05:20:40.806633+0000","last_peered":"2024-09-16T05:20:40.806633+0000","last_clean":"2024-09-16T05:20:40.806633+0000","last_became_active":"2024-09-16T05:18:02.890972+0000","last_became_peered":"2024-09-16T05:18:02.890972+0000","last_unstale":"2024-09-16T05:20:40.806633+0000","last_undegraded":"2024-09-16T05:20:40.806633+0000","last_fullsized":"2024-09-16T05:20:40.806633+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"620'444","last_scrub_stamp":"2024-09-16T05:19:51.567043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.567043+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:09.289631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.004404776,"stat_sum":{"num_bytes":3294370,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":10803,"num_write":28,"num_write_kb":13180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1137,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:42.448875+0000","last_change":"2024-09-16T05:20:42.448898+0000","last_active":"2024-09-16T05:20:42.448875+0000","last_peered":"2024-09-16T05:20:42.448875+0000","last_clean":"2024-09-16T05:20:42.448875+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:20:42.448875+0000","last_undegraded":"2024-09-16T05:20:42.448875+0000","last_fullsized":"2024-09-16T05:20:42.448875+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:42.448822+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:42.448822+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.950314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"689'1217","reported_seq":4357,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.807288+0000","last_change":"2024-09-16T05:20:38.936061+0000","last_active":"2024-09-16T05:20:40.807288+0000","last_peered":"2024-09-16T05:20:40.807288+0000","last_clean":"2024-09-16T05:20:40.807288+0000","last_became_active":"2024-09-16T05:18:03.891103+0000","last_became_peered":"2024-09-16T05:18:03.891103+0000","last_unstale":"2024-09-16T05:20:40.807288+0000","last_undegraded":"2024-09-16T05:20:40.807288+0000","last_fullsized":"2024-09-16T05:20:40.807288+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"624'1113","last_scrub_stamp":"2024-09-16T05:19:35.593327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:35.593327+0000","objects_scrubbed":9,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:05.061752+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0095046660000000002,"stat_sum":{"num_bytes":16610321,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":275,"num_read_kb":226669,"num_write":240,"num_write_kb":76127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7155994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.1d","version":"35'139","reported_seq":539,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.813605+0000","last_change":"2024-09-16T05:20:41.421678+0000","last_active":"2024-09-16T05:20:41.813605+0000","last_peered":"2024-09-16T05:20:41.813605+0000","last_clean":"2024-09-16T05:20:41.813605+0000","last_became_active":"2024-09-16T05:20:40.803676+0000","last_became_peered":"2024-09-16T05:20:40.803676+0000","last_unstale":"2024-09-16T05:20:41.813605+0000","last_undegraded":"2024-09-16T05:20:41.813605+0000","last_fullsized":"2024-09-16T05:20:41.813605+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.421551+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.421551+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:01.574352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1e","version":"689'476","reported_seq":3418,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.950100+0000","last_change":"2024-09-16T05:20:38.923122+0000","last_active":"2024-09-16T05:20:41.950100+0000","last_peered":"2024-09-16T05:20:41.950100+0000","last_clean":"2024-09-16T05:20:41.950100+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:20:41.950100+0000","last_undegraded":"2024-09-16T05:20:41.950100+0000","last_fullsized":"2024-09-16T05:20:41.950100+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"671'464","last_scrub_stamp":"2024-09-16T05:20:21.358700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:21.358700+0000","objects_scrubbed":3,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:31.461609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0045745839999999996,"stat_sum":{"num_bytes":3007609,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":82355,"num_write":25,"num_write_kb":11620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.1c","version":"35'139","reported_seq":379,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.956044+0000","last_change":"2024-09-16T05:20:42.167372+0000","last_active":"2024-09-16T05:20:43.956044+0000","last_peered":"2024-09-16T05:20:43.956044+0000","last_clean":"2024-09-16T05:20:43.956044+0000","last_became_active":"2024-09-16T05:20:40.805879+0000","last_became_peered":"2024-09-16T05:20:40.805879+0000","last_unstale":"2024-09-16T05:20:43.956044+0000","last_undegraded":"2024-09-16T05:20:43.956044+0000","last_fullsized":"2024-09-16T05:20:43.956044+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:42.167269+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:42.167269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:51.343885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":3124,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.930519+0000","last_change":"2024-09-16T05:20:38.922118+0000","last_active":"2024-09-16T05:20:39.930519+0000","last_peered":"2024-09-16T05:20:39.930519+0000","last_clean":"2024-09-16T05:20:39.930519+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:20:39.930519+0000","last_undegraded":"2024-09-16T05:20:39.930519+0000","last_fullsized":"2024-09-16T05:20:39.930519+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:20:17.544083+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.544083+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.663675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021754169999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.20","version":"689'313","reported_seq":3247,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805184+0000","last_change":"2024-09-16T05:20:38.934783+0000","last_active":"2024-09-16T05:20:40.805184+0000","last_peered":"2024-09-16T05:20:40.805184+0000","last_clean":"2024-09-16T05:20:40.805184+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:20:40.805184+0000","last_undegraded":"2024-09-16T05:20:40.805184+0000","last_fullsized":"2024-09-16T05:20:40.805184+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'290","last_scrub_stamp":"2024-09-16T05:20:03.351330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.351330+0000","objects_scrubbed":3,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:29.176091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010086431,"stat_sum":{"num_bytes":6345760,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":155851,"num_write":58,"num_write_kb":21452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":2848,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.928006+0000","last_change":"2024-09-16T05:20:38.924093+0000","last_active":"2024-09-16T05:20:41.928006+0000","last_peered":"2024-09-16T05:20:41.928006+0000","last_clean":"2024-09-16T05:20:41.928006+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:20:41.928006+0000","last_undegraded":"2024-09-16T05:20:41.928006+0000","last_fullsized":"2024-09-16T05:20:41.928006+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:02.565969+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.565969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.309026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001840092,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":812,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805476+0000","last_change":"2024-09-16T05:20:38.923169+0000","last_active":"2024-09-16T05:20:40.805476+0000","last_peered":"2024-09-16T05:20:40.805476+0000","last_clean":"2024-09-16T05:20:40.805476+0000","last_became_active":"2024-09-16T05:17:59.866047+0000","last_became_peered":"2024-09-16T05:17:59.866047+0000","last_unstale":"2024-09-16T05:20:40.805476+0000","last_undegraded":"2024-09-16T05:20:40.805476+0000","last_fullsized":"2024-09-16T05:20:40.805476+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:15.452705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.452705+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.511456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010285979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":2394,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.950022+0000","last_change":"2024-09-16T05:20:38.918741+0000","last_active":"2024-09-16T05:20:41.950022+0000","last_peered":"2024-09-16T05:20:41.950022+0000","last_clean":"2024-09-16T05:20:41.950022+0000","last_became_active":"2024-09-16T05:20:20.801064+0000","last_became_peered":"2024-09-16T05:20:20.801064+0000","last_unstale":"2024-09-16T05:20:41.950022+0000","last_undegraded":"2024-09-16T05:20:41.950022+0000","last_fullsized":"2024-09-16T05:20:41.950022+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:20:00.415257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.415257+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.891305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059567400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":2193,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933609+0000","last_change":"2024-09-16T05:20:38.930957+0000","last_active":"2024-09-16T05:20:39.933609+0000","last_peered":"2024-09-16T05:20:39.933609+0000","last_clean":"2024-09-16T05:20:39.933609+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:20:39.933609+0000","last_undegraded":"2024-09-16T05:20:39.933609+0000","last_fullsized":"2024-09-16T05:20:39.933609+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:57.859756+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:19:57.859756+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:20.082181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018431439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]},{"pgid":"2.25","version":"694'1140","reported_seq":3972,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.961901+0000","last_change":"2024-09-16T05:20:43.962123+0000","last_active":"2024-09-16T05:20:43.961901+0000","last_peered":"2024-09-16T05:20:43.961901+0000","last_clean":"2024-09-16T05:20:43.961901+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:20:43.961901+0000","last_undegraded":"2024-09-16T05:20:43.961901+0000","last_fullsized":"2024-09-16T05:20:43.961901+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"622'977","last_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.332491+0000","objects_scrubbed":9,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:46.725476+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0.0056309680000000001,"stat_sum":{"num_bytes":35238244,"num_objects":16,"num_object_clones":13,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":201,"num_read_kb":205714,"num_write":182,"num_write_kb":73326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":2183,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805409+0000","last_change":"2024-09-16T05:20:38.937161+0000","last_active":"2024-09-16T05:20:40.805409+0000","last_peered":"2024-09-16T05:20:40.805409+0000","last_clean":"2024-09-16T05:20:40.805409+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:20:40.805409+0000","last_undegraded":"2024-09-16T05:20:40.805409+0000","last_fullsized":"2024-09-16T05:20:40.805409+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:19:07.483055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:07.483055+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010248709,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":2195,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.806128+0000","last_change":"2024-09-16T05:20:38.936655+0000","last_active":"2024-09-16T05:20:40.806128+0000","last_peered":"2024-09-16T05:20:40.806128+0000","last_clean":"2024-09-16T05:20:40.806128+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:20:40.806128+0000","last_undegraded":"2024-09-16T05:20:40.806128+0000","last_fullsized":"2024-09-16T05:20:40.806128+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:11.499561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:11.499561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010093492000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1211,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.927757+0000","last_change":"2024-09-16T05:20:38.921852+0000","last_active":"2024-09-16T05:20:41.927757+0000","last_peered":"2024-09-16T05:20:41.927757+0000","last_clean":"2024-09-16T05:20:41.927757+0000","last_became_active":"2024-09-16T05:19:35.785220+0000","last_became_peered":"2024-09-16T05:19:35.785220+0000","last_unstale":"2024-09-16T05:20:41.927757+0000","last_undegraded":"2024-09-16T05:20:41.927757+0000","last_fullsized":"2024-09-16T05:20:41.927757+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:19:27.303070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:27.303070+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:50.758057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058673099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.29","version":"692'1029","reported_seq":3953,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.800625+0000","last_change":"2024-09-16T05:20:38.934055+0000","last_active":"2024-09-16T05:20:41.800625+0000","last_peered":"2024-09-16T05:20:41.800625+0000","last_clean":"2024-09-16T05:20:41.800625+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:20:41.800625+0000","last_undegraded":"2024-09-16T05:20:41.800625+0000","last_fullsized":"2024-09-16T05:20:41.800625+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"621'852","last_scrub_stamp":"2024-09-16T05:19:32.048085+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:19:32.048085+0000","objects_scrubbed":8,"log_size":1029,"log_dups_size":0,"ondisk_log_size":1029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:44.806100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011490279000000001,"stat_sum":{"num_bytes":31735063,"num_objects":15,"num_object_clones":11,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":260,"num_read_kb":246177,"num_write":197,"num_write_kb":74171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":3185,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.804867+0000","last_change":"2024-09-16T05:20:38.925236+0000","last_active":"2024-09-16T05:20:40.804867+0000","last_peered":"2024-09-16T05:20:40.804867+0000","last_clean":"2024-09-16T05:20:40.804867+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:20:40.804867+0000","last_undegraded":"2024-09-16T05:20:40.804867+0000","last_fullsized":"2024-09-16T05:20:40.804867+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:19:29.329416+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:16:23.269448+0000","last_clean_scrub_stamp":"2024-09-16T05:19:29.329416+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:20:47.050609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062899899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.2b","version":"692'336","reported_seq":3164,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T05:20:41.937047+0000","last_change":"2024-09-16T05:20:38.923233+0000","last_active":"2024-09-16T05:20:41.937047+0000","last_peered":"2024-09-16T05:20:41.937047+0000","last_clean":"2024-09-16T05:20:41.937047+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:20:41.937047+0000","last_undegraded":"2024-09-16T05:20:41.937047+0000","last_fullsized":"2024-09-16T05:20:41.937047+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"666'332","last_scrub_stamp":"2024-09-16T05:20:18.508468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.508468+0000","objects_scrubbed":1,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:31.378982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001630444,"stat_sum":{"num_bytes":3697122,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":79890,"num_write":51,"num_write_kb":22200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":3034,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T05:20:43.957128+0000","last_change":"2024-09-16T05:20:43.957493+0000","last_active":"2024-09-16T05:20:43.957128+0000","last_peered":"2024-09-16T05:20:43.957128+0000","last_clean":"2024-09-16T05:20:43.957128+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:20:43.957128+0000","last_undegraded":"2024-09-16T05:20:43.957128+0000","last_fullsized":"2024-09-16T05:20:43.957128+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:19:51.201665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.201665+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:01.284720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001379091,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"2.2d","version":"686'477","reported_seq":3375,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805647+0000","last_change":"2024-09-16T05:20:38.937426+0000","last_active":"2024-09-16T05:20:40.805647+0000","last_peered":"2024-09-16T05:20:40.805647+0000","last_clean":"2024-09-16T05:20:40.805647+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:20:40.805647+0000","last_undegraded":"2024-09-16T05:20:40.805647+0000","last_fullsized":"2024-09-16T05:20:40.805647+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"648'457","last_scrub_stamp":"2024-09-16T05:20:15.223777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.223777+0000","objects_scrubbed":2,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.099881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010557351,"stat_sum":{"num_bytes":10279378,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":92,"num_read_kb":88583,"num_write":67,"num_write_kb":29206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.2e","version":"689'636","reported_seq":3489,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-09-16T05:20:40.805155+0000","last_change":"2024-09-16T05:20:38.938066+0000","last_active":"2024-09-16T05:20:40.805155+0000","last_peered":"2024-09-16T05:20:40.805155+0000","last_clean":"2024-09-16T05:20:40.805155+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:20:40.805155+0000","last_undegraded":"2024-09-16T05:20:40.805155+0000","last_fullsized":"2024-09-16T05:20:40.805155+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"617'552","last_scrub_stamp":"2024-09-16T05:19:28.348887+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:19:28.348887+0000","objects_scrubbed":7,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0.011564313999999999,"stat_sum":{"num_bytes":17789625,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":64999,"num_write":93,"num_write_kb":39482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"2.2f","version":"689'594","reported_seq":3529,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-09-16T05:20:39.933863+0000","last_change":"2024-09-16T05:20:38.938566+0000","last_active":"2024-09-16T05:20:39.933863+0000","last_peered":"2024-09-16T05:20:39.933863+0000","last_clean":"2024-09-16T05:20:39.933863+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:20:39.933863+0000","last_undegraded":"2024-09-16T05:20:39.933863+0000","last_fullsized":"2024-09-16T05:20:39.933863+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"650'553","last_scrub_stamp":"2024-09-16T05:20:00.804910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.804910+0000","objects_scrubbed":6,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.917155+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0078242139999999995,"stat_sum":{"num_bytes":14000891,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":110,"num_read_kb":77808,"num_write":102,"num_write_kb":40852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"106","length":"1"},{"start":"109","length":"2"},{"start":"112","length":"1"},{"start":"115","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":334071325,"num_objects":151,"num_object_clones":104,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":9,"num_read":3365,"num_read_kb":3325387,"num_write":2463,"num_write_kb":979099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":47028772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445628416,"data_stored":568430840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24775,"ondisk_log_size":24775,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089484,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":125196,"kb_used_data":71800,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93619956,"statfs":{"total":95995035648,"available":95866834944,"internally_reserved":0,"allocated":73523200,"data_stored":83529829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285712,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84004,"kb_used_data":52816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93661148,"statfs":{"total":95995035648,"available":95909015552,"internally_reserved":0,"allocated":54083584,"data_stored":61376753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514649,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99752,"kb_used_data":42196,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93645400,"statfs":{"total":95995035648,"available":95892889600,"internally_reserved":0,"allocated":43208704,"data_stored":51221222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841689,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88088,"kb_used_data":52732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93657064,"statfs":{"total":95995035648,"available":95904833536,"internally_reserved":0,"allocated":53997568,"data_stored":61164979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118012,"kb_used_data":60448,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93627140,"statfs":{"total":95995035648,"available":95874191360,"internally_reserved":0,"allocated":61898752,"data_stored":73008714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397670,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114112,"kb_used_data":60716,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93631040,"statfs":{"total":95995035648,"available":95878184960,"internally_reserved":0,"allocated":62173184,"data_stored":76584717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757416,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68420,"kb_used_data":41392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93676732,"statfs":{"total":95995035648,"available":95924973568,"internally_reserved":0,"allocated":42385408,"data_stored":50741423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920942,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56124,"kb_used_data":24932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93689028,"statfs":{"total":95995035648,"available":95937564672,"internally_reserved":0,"allocated":25530368,"data_stored":26981904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117171,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97516,"kb_used_data":62168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93647636,"statfs":{"total":95995035648,"available":95895179264,"internally_reserved":0,"allocated":63660032,"data_stored":82322745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240458,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61188,"kb_used_data":20592,"kb_used_omap":10,"kb_used_meta":40565,"kb_avail":93683964,"statfs":{"total":95995035648,"available":95932379136,"internally_reserved":0,"allocated":21086208,"data_stored":19424587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":41538867},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738562,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48480,"kb_used_data":17292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93696672,"statfs":{"total":95995035648,"available":95945392128,"internally_reserved":0,"allocated":17707008,"data_stored":19134048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9052160,"data_stored":13275819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18554880,"data_stored":23395830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12455936,"data_stored":13605761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54968320,"data_stored":76438927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17190912,"data_stored":21442702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33828864,"data_stored":44992421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53809152,"data_stored":71028579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53547008,"data_stored":67460960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45621248,"data_stored":55600457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":34881536,"data_stored":45690673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":57525020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":65159168,"data_stored":77973691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:20:47.326 INFO:tasks.thrashosds.thrasher:Setting 1.1 to [2, 3] 2024-09-16T05:20:47.326 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.1', '2', '3'] 2024-09-16T05:20:47.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 1.1 2 3 2024-09-16T05:20:47.327 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:47.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:47.367 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:47.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:47.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:47.495 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:47.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:47.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:47.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:47.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:47.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:47.749 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:47.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:47.777 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:20:47.777 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:20:47.777 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:20:47.777 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:20:47.809 INFO:tasks.rados.rados.2.smithi204.stdout:2730: done (0 left) 2024-09-16T05:20:47.809 INFO:tasks.rados.rados.2.smithi204.stdout:2731: read oid 43 snap -1 2024-09-16T05:20:47.809 INFO:tasks.rados.rados.2.smithi204.stdout:2731: expect (ObjNum 950 snap 280 seq_num 950) 2024-09-16T05:20:47.809 INFO:tasks.rados.rados.2.smithi204.stdout:2732: write (excl) oid 46 current snap is 287 2024-09-16T05:20:47.810 INFO:tasks.rados.rados.2.smithi204.stdout:2732: seq_num 969 ranges {685983=493112,1870228=415755} 2024-09-16T05:20:47.816 INFO:tasks.rados.rados.2.smithi204.stdout:2732: writing smithi20457400-46 from 685983 to 1179095 tid 1 2024-09-16T05:20:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:2732: writing smithi20457400-46 from 1870228 to 2285983 tid 2 2024-09-16T05:20:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:2733: copy_from oid 9 from oid 5 current snap is 287 2024-09-16T05:20:47.822 INFO:tasks.rados.rados.2.smithi204.stdout:2731: expect (ObjNum 950 snap 280 seq_num 950) 2024-09-16T05:20:47.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:47.874 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:47.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:47.951 INFO:tasks.rados.rados.2.smithi204.stdout:2732: finishing write tid 1 to smithi20457400-46 2024-09-16T05:20:47.951 INFO:tasks.rados.rados.2.smithi204.stdout:2732: finishing write tid 2 to smithi20457400-46 2024-09-16T05:20:47.951 INFO:tasks.rados.rados.2.smithi204.stdout:2732: finishing write tid 3 to smithi20457400-46 2024-09-16T05:20:47.951 INFO:tasks.rados.rados.2.smithi204.stdout:2732: finishing write tid 4 to smithi20457400-46 2024-09-16T05:20:47.951 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1014 (ObjNum 969 snap 287 seq_num 969) dirty exists 2024-09-16T05:20:47.951 INFO:tasks.rados.rados.2.smithi204.stdout:2732: left oid 46 (ObjNum 969 snap 287 seq_num 969) 2024-09-16T05:20:47.951 INFO:tasks.rados.rados.2.smithi204.stdout:2733: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:20:47.951 INFO:tasks.rados.rados.2.smithi204.stdout:2733: finishing copy_from to smithi20457400-9 2024-09-16T05:20:47.952 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 611 (ObjNum 964 snap 285 seq_num 964) dirty exists 2024-09-16T05:20:47.952 INFO:tasks.rados.rados.2.smithi204.stdout:2731: done (2 left) 2024-09-16T05:20:47.952 INFO:tasks.rados.rados.2.smithi204.stdout:2732: done (1 left) 2024-09-16T05:20:47.952 INFO:tasks.rados.rados.2.smithi204.stdout:2733: done (0 left) 2024-09-16T05:20:47.952 INFO:tasks.rados.rados.2.smithi204.stdout:2734: write oid 21 current snap is 287 2024-09-16T05:20:47.952 INFO:tasks.rados.rados.2.smithi204.stdout:2734: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-16T05:20:47.957 INFO:tasks.rados.rados.2.smithi204.stdout:2734: writing smithi20457400-21 from 702790 to 1271151 tid 1 2024-09-16T05:20:47.963 INFO:tasks.rados.rados.2.smithi204.stdout:2734: writing smithi20457400-21 from 1812357 to 2302790 tid 2 2024-09-16T05:20:47.963 INFO:tasks.rados.rados.2.smithi204.stdout:2735: read oid 30 snap -1 2024-09-16T05:20:47.963 INFO:tasks.rados.rados.2.smithi204.stdout:2735: expect deleted 2024-09-16T05:20:47.963 INFO:tasks.rados.rados.2.smithi204.stdout:2736: copy_from oid 42 from oid 36 current snap is 287 2024-09-16T05:20:47.964 INFO:tasks.rados.rados.2.smithi204.stdout:2734: finishing write tid 1 to smithi20457400-21 2024-09-16T05:20:47.964 INFO:tasks.rados.rados.2.smithi204.stdout:2737: write (excl) oid 41 current snap is 287 2024-09-16T05:20:47.964 INFO:tasks.rados.rados.2.smithi204.stdout:2737: seq_num 971 ranges {719597=643610,1870839=448758} 2024-09-16T05:20:47.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:20:47.971 INFO:tasks.rados.rados.2.smithi204.stdout:2737: writing smithi20457400-41 from 719597 to 1363207 tid 1 2024-09-16T05:20:47.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:47.977 INFO:tasks.rados.rados.2.smithi204.stdout:2737: writing smithi20457400-41 from 1870839 to 2319597 tid 2 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2738: copy_from oid 16 from oid 45 current snap is 287 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2734: finishing write tid 2 to smithi20457400-21 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2734: finishing write tid 3 to smithi20457400-21 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2734: finishing write tid 4 to smithi20457400-21 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 848 (ObjNum 970 snap 287 seq_num 970) dirty exists 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2734: left oid 21 (ObjNum 970 snap 287 seq_num 970) 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2734: done (4 left) 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2735: done (3 left) 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2739: rollback oid 31 current snap is 287 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 31 to 276 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2740: read oid 35 snap -1 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2740: expect (ObjNum 961 snap 284 seq_num 961) 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2741: write (excl) oid 11 current snap is 287 2024-09-16T05:20:47.978 INFO:tasks.rados.rados.2.smithi204.stdout:2741: seq_num 972 ranges {736404=718859,1929321=407083} 2024-09-16T05:20:47.984 INFO:tasks.rados.rados.2.smithi204.stdout:2741: writing smithi20457400-11 from 736404 to 1455263 tid 1 2024-09-16T05:20:47.990 INFO:tasks.rados.rados.2.smithi204.stdout:2741: writing smithi20457400-11 from 1929321 to 2336404 tid 2 2024-09-16T05:20:47.990 INFO:tasks.rados.rados.2.smithi204.stdout:2742: copy_from oid 33 from oid 28 current snap is 287 2024-09-16T05:20:47.990 INFO:tasks.rados.rados.2.smithi204.stdout:2739: finishing rollback tid 1 to smithi20457400-31 2024-09-16T05:20:47.990 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:20:47.990 INFO:tasks.rados.rados.2.smithi204.stdout:2740: expect (ObjNum 961 snap 284 seq_num 961) 2024-09-16T05:20:48.035 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:48.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:48.043 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:48.104 INFO:tasks.rados.rados.2.smithi204.stdout:2737: finishing write tid 1 to smithi20457400-41 2024-09-16T05:20:48.104 INFO:tasks.rados.rados.2.smithi204.stdout:2737: finishing write tid 2 to smithi20457400-41 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2737: finishing write tid 3 to smithi20457400-41 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2737: finishing write tid 4 to smithi20457400-41 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1047 (ObjNum 971 snap 287 seq_num 971) dirty exists 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2737: left oid 41 (ObjNum 971 snap 287 seq_num 971) 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2736: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2736: finishing copy_from to smithi20457400-42 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 765 (ObjNum 931 snap 275 seq_num 931) dirty exists 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2741: finishing write tid 1 to smithi20457400-11 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2738: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2738: finishing copy_from to smithi20457400-16 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1018 (ObjNum 937 snap 278 seq_num 937) dirty exists 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2741: finishing write tid 2 to smithi20457400-11 2024-09-16T05:20:48.105 INFO:tasks.rados.rados.2.smithi204.stdout:2736: done (6 left) 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2737: done (5 left) 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2738: done (4 left) 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2739: done (3 left) 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2740: done (2 left) 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2743: rollback oid 44 current snap is 287 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 284 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2741: finishing write tid 3 to smithi20457400-11 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2741: finishing write tid 4 to smithi20457400-11 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1153 (ObjNum 972 snap 287 seq_num 972) dirty exists 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2741: left oid 11 (ObjNum 972 snap 287 seq_num 972) 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2742: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2742: finishing copy_from to smithi20457400-33 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 1113 (ObjNum 939 snap 278 seq_num 939) dirty exists 2024-09-16T05:20:48.106 INFO:tasks.rados.rados.2.smithi204.stdout:2741: done (2 left) 2024-09-16T05:20:48.107 INFO:tasks.rados.rados.2.smithi204.stdout:2742: done (1 left) 2024-09-16T05:20:48.107 INFO:tasks.rados.rados.2.smithi204.stdout:2744: delete oid 28 current snap is 287 2024-09-16T05:20:48.107 INFO:tasks.rados.rados.2.smithi204.stdout:2743: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:20:48.107 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 33 snap 0 seq_num 4282019792) dirty dne 2024-09-16T05:20:48.109 INFO:tasks.rados.rados.2.smithi204.stdout:2743: done (1 left) 2024-09-16T05:20:48.109 INFO:tasks.rados.rados.2.smithi204.stdout:2744: done (0 left) 2024-09-16T05:20:48.109 INFO:tasks.rados.rados.2.smithi204.stdout:2745: read oid 6 snap 266 2024-09-16T05:20:48.109 INFO:tasks.rados.rados.2.smithi204.stdout:2745: expect (ObjNum 881 snap 257 seq_num 881) 2024-09-16T05:20:48.109 INFO:tasks.rados.rados.2.smithi204.stdout:2746: read oid 41 snap -1 2024-09-16T05:20:48.109 INFO:tasks.rados.rados.2.smithi204.stdout:2746: expect (ObjNum 971 snap 287 seq_num 971) 2024-09-16T05:20:48.110 INFO:tasks.rados.rados.2.smithi204.stdout:2747: read oid 21 snap -1 2024-09-16T05:20:48.110 INFO:tasks.rados.rados.2.smithi204.stdout:2747: expect (ObjNum 970 snap 287 seq_num 970) 2024-09-16T05:20:48.110 INFO:tasks.rados.rados.2.smithi204.stdout:2748: snap_remove snap 278 2024-09-16T05:20:48.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:48.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:48.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:48.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:47 smithi032 ceph-mon[27388]: pgmap v1206: 117 pgs: 117 active+clean; 311 MiB data, 938 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-16T05:20:48.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:47 smithi032 ceph-mon[27388]: osdmap e697: 12 total, 12 up, 11 in 2024-09-16T05:20:48.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:47 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:48.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:47 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:47 smithi204 ceph-mon[30933]: pgmap v1206: 117 pgs: 117 active+clean; 311 MiB data, 938 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-16T05:20:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:47 smithi204 ceph-mon[30933]: osdmap e697: 12 total, 12 up, 11 in 2024-09-16T05:20:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:47 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:47 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:47 smithi196 ceph-mon[30802]: pgmap v1206: 117 pgs: 117 active+clean; 311 MiB data, 938 MiB used, 983 GiB / 983 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 14 op/s 2024-09-16T05:20:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:47 smithi196 ceph-mon[30802]: osdmap e697: 12 total, 12 up, 11 in 2024-09-16T05:20:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:47 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:47 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:48.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:48.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:48.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:48.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:48.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:48.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:48.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:48.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:20:48.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:48.558 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:48.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:48.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:48.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:48.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:48.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:48.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:48.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:20:48.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:48.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2748: done (3 left) 2024-09-16T05:20:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2749: delete oid 29 current snap is 287 2024-09-16T05:20:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2747: expect (ObjNum 970 snap 287 seq_num 970) 2024-09-16T05:20:48.908 INFO:tasks.rados.rados.2.smithi204.stdout:2745: expect (ObjNum 881 snap 257 seq_num 881) 2024-09-16T05:20:48.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:48.938 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:48.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:49.024 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:48 smithi032 ceph-mon[27388]: osdmap e698: 12 total, 12 up, 11 in 2024-09-16T05:20:49.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:49.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:49.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:49.134 INFO:tasks.rados.rados.2.smithi204.stdout:2746: expect (ObjNum 971 snap 287 seq_num 971) 2024-09-16T05:20:49.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:48 smithi204 ceph-mon[30933]: osdmap e698: 12 total, 12 up, 11 in 2024-09-16T05:20:49.185 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:49.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:48 smithi196 ceph-mon[30802]: osdmap e698: 12 total, 12 up, 11 in 2024-09-16T05:20:49.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:49.308 INFO:tasks.rados.rados.2.smithi204.stdout:2745: done (3 left) 2024-09-16T05:20:49.308 INFO:tasks.rados.rados.2.smithi204.stdout:2746: done (2 left) 2024-09-16T05:20:49.308 INFO:tasks.rados.rados.2.smithi204.stdout:2747: done (1 left) 2024-09-16T05:20:49.308 INFO:tasks.rados.rados.2.smithi204.stdout:2749: done (0 left) 2024-09-16T05:20:49.308 INFO:tasks.rados.rados.2.smithi204.stdout:2750: write oid 12 current snap is 287 2024-09-16T05:20:49.308 INFO:tasks.rados.rados.2.smithi204.stdout:2750: seq_num 975 ranges {786825=660959,2104767=282058} 2024-09-16T05:20:49.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:49.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:49.315 INFO:tasks.rados.rados.2.smithi204.stdout:2750: writing smithi20457400-12 from 786825 to 1447784 tid 1 2024-09-16T05:20:49.321 INFO:tasks.rados.rados.2.smithi204.stdout:2750: writing smithi20457400-12 from 2104767 to 2386825 tid 2 2024-09-16T05:20:49.321 INFO:tasks.rados.rados.2.smithi204.stdout:2751: rollback oid 26 current snap is 287 2024-09-16T05:20:49.322 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 284 2024-09-16T05:20:49.322 INFO:tasks.rados.rados.2.smithi204.stdout:2750: finishing write tid 1 to smithi20457400-12 2024-09-16T05:20:49.322 INFO:tasks.rados.rados.2.smithi204.stdout:2752: write oid 14 current snap is 287 2024-09-16T05:20:49.322 INFO:tasks.rados.rados.2.smithi204.stdout:2752: seq_num 976 ranges {403632=736208,1763249=619999} 2024-09-16T05:20:49.327 INFO:tasks.rados.rados.2.smithi204.stdout:2752: writing smithi20457400-14 from 403632 to 1139840 tid 1 2024-09-16T05:20:49.334 INFO:tasks.rados.rados.2.smithi204.stdout:2752: writing smithi20457400-14 from 1763249 to 2383248 tid 2 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2753: read oid 23 snap -1 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2753: expect (ObjNum 899 snap 263 seq_num 899) 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2754: snap_create 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2750: finishing write tid 2 to smithi20457400-12 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2750: finishing write tid 3 to smithi20457400-12 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2751: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2750: finishing write tid 4 to smithi20457400-12 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1058 (ObjNum 975 snap 287 seq_num 975) dirty exists 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2750: left oid 12 (ObjNum 975 snap 287 seq_num 975) 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2751: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:2751: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:20:49.335 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1229 (ObjNum 953 snap 280 seq_num 953) dirty exists 2024-09-16T05:20:49.336 INFO:tasks.rados.rados.2.smithi204.stdout:2752: finishing write tid 1 to smithi20457400-14 2024-09-16T05:20:49.337 INFO:tasks.rados.rados.2.smithi204.stdout:2752: finishing write tid 2 to smithi20457400-14 2024-09-16T05:20:49.337 INFO:tasks.rados.rados.2.smithi204.stdout:2752: finishing write tid 3 to smithi20457400-14 2024-09-16T05:20:49.337 INFO:tasks.rados.rados.2.smithi204.stdout:2752: finishing write tid 4 to smithi20457400-14 2024-09-16T05:20:49.337 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1233 (ObjNum 976 snap 287 seq_num 976) dirty exists 2024-09-16T05:20:49.338 INFO:tasks.rados.rados.2.smithi204.stdout:2752: left oid 14 (ObjNum 976 snap 287 seq_num 976) 2024-09-16T05:20:49.344 INFO:tasks.rados.rados.2.smithi204.stdout:2753: expect (ObjNum 899 snap 263 seq_num 899) 2024-09-16T05:20:49.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:49.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:49.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:49.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:49.549 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:49.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:49.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:49.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:49.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:49.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:49.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:49.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:49.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:49.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:50.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:50.042 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:20:50.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:20:50.042 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:20:50.042 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:20:50.042 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:20:50.053 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: osdmap e699: 12 total, 12 up, 11 in 2024-09-16T05:20:50.053 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:20:50.053 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:20:50.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:20:50.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:20:50.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:20:50.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:20:50.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:20:50.054 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2983271442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:50.055 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3262296777' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 3]}]: dispatch 2024-09-16T05:20:50.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:50.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:50.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: osdmap e699: 12 total, 12 up, 11 in 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2983271442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3262296777' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 3]}]: dispatch 2024-09-16T05:20:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:50.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:50.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: osdmap e699: 12 total, 12 up, 11 in 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2983271442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:20:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3262296777' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 3]}]: dispatch 2024-09-16T05:20:50.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:50.291 INFO:teuthology.orchestra.run.smithi032.stderr:set 1.1 pg_upmap mapping to [2,3] 2024-09-16T05:20:50.291 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:20:50.296 INFO:tasks.rados.rados.2.smithi204.stdout:2750: done (4 left) 2024-09-16T05:20:50.296 INFO:tasks.rados.rados.2.smithi204.stdout:2751: done (3 left) 2024-09-16T05:20:50.296 INFO:tasks.rados.rados.2.smithi204.stdout:2752: done (2 left) 2024-09-16T05:20:50.297 INFO:tasks.rados.rados.2.smithi204.stdout:2753: done (1 left) 2024-09-16T05:20:50.297 INFO:tasks.rados.rados.2.smithi204.stdout:2754: done (0 left) 2024-09-16T05:20:50.297 INFO:tasks.rados.rados.2.smithi204.stdout:2755: write oid 1 current snap is 288 2024-09-16T05:20:50.297 INFO:tasks.rados.rados.2.smithi204.stdout:2755: seq_num 977 ranges {420439=411457,1421731=480010} 2024-09-16T05:20:50.301 INFO:tasks.rados.rados.2.smithi204.stdout:2755: writing smithi20457400-1 from 420439 to 831896 tid 1 2024-09-16T05:20:50.307 INFO:tasks.rados.rados.2.smithi204.stdout:2755: writing smithi20457400-1 from 1421731 to 1901741 tid 2 2024-09-16T05:20:50.307 INFO:tasks.rados.rados.2.smithi204.stdout:2756: write (excl) oid 30 current snap is 288 2024-09-16T05:20:50.307 INFO:tasks.rados.rados.2.smithi204.stdout:2756: seq_num 978 ranges {437246=486706,1363860=623668} 2024-09-16T05:20:50.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:50.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:50.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:50.312 INFO:tasks.rados.rados.2.smithi204.stdout:2756: writing smithi20457400-30 from 437246 to 923952 tid 1 2024-09-16T05:20:50.318 INFO:tasks.rados.rados.2.smithi204.stdout:2756: writing smithi20457400-30 from 1363860 to 1987528 tid 2 2024-09-16T05:20:50.318 INFO:tasks.rados.rados.2.smithi204.stdout:2757: snap_remove snap 285 2024-09-16T05:20:50.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:50.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:50.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:50.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:50.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:50.661 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:50.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:50.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:50.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2757: done (2 left) 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2758: copy_from oid 17 from oid 35 current snap is 288 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2755: finishing write tid 1 to smithi20457400-1 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2755: finishing write tid 2 to smithi20457400-1 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2755: finishing write tid 3 to smithi20457400-1 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2755: finishing write tid 4 to smithi20457400-1 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 559 (ObjNum 977 snap 288 seq_num 977) dirty exists 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2755: left oid 1 (ObjNum 977 snap 288 seq_num 977) 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2756: finishing write tid 1 to smithi20457400-30 2024-09-16T05:20:50.804 INFO:tasks.rados.rados.2.smithi204.stdout:2756: finishing write tid 2 to smithi20457400-30 2024-09-16T05:20:50.805 INFO:tasks.rados.rados.2.smithi204.stdout:2756: finishing write tid 3 to smithi20457400-30 2024-09-16T05:20:50.805 INFO:tasks.rados.rados.2.smithi204.stdout:2756: finishing write tid 4 to smithi20457400-30 2024-09-16T05:20:50.805 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 1236 (ObjNum 978 snap 288 seq_num 978) dirty exists 2024-09-16T05:20:50.805 INFO:tasks.rados.rados.2.smithi204.stdout:2756: left oid 30 (ObjNum 978 snap 288 seq_num 978) 2024-09-16T05:20:50.805 INFO:tasks.rados.rados.2.smithi204.stdout:2755: done (2 left) 2024-09-16T05:20:50.805 INFO:tasks.rados.rados.2.smithi204.stdout:2756: done (1 left) 2024-09-16T05:20:50.805 INFO:tasks.rados.rados.2.smithi204.stdout:2759: write oid 4 current snap is 288 2024-09-16T05:20:50.805 INFO:tasks.rados.rados.2.smithi204.stdout:2759: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-09-16T05:20:50.809 INFO:tasks.rados.rados.2.smithi204.stdout:2759: writing smithi20457400-4 from 454053 to 1016008 tid 1 2024-09-16T05:20:50.815 INFO:tasks.rados.rados.2.smithi204.stdout:2759: writing smithi20457400-4 from 1422342 to 1906021 tid 2 2024-09-16T05:20:50.818 INFO:tasks.rados.rados.2.smithi204.stdout:2759: writing smithi20457400-4 from 2368404 to 2454053 tid 3 2024-09-16T05:20:50.818 INFO:tasks.rados.rados.2.smithi204.stdout:2760: snap_remove snap 265 2024-09-16T05:20:50.819 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:50.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:50.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:50.944 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:50.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:51.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:51.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:51.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:51.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:50 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2983271442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3262296777' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 3]}]': finished 2024-09-16T05:20:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:50 smithi204 ceph-mon[30933]: osdmap e700: 12 total, 12 up, 11 in 2024-09-16T05:20:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:50 smithi204 ceph-mon[30933]: osdmap e701: 12 total, 12 up, 11 in 2024-09-16T05:20:51.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:50 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:51.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2983271442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:51.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3262296777' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 3]}]': finished 2024-09-16T05:20:51.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:50 smithi032 ceph-mon[27388]: osdmap e700: 12 total, 12 up, 11 in 2024-09-16T05:20:51.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:50 smithi032 ceph-mon[27388]: osdmap e701: 12 total, 12 up, 11 in 2024-09-16T05:20:51.198 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:51.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:51.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:50 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2983271442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:20:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3262296777' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 3]}]': finished 2024-09-16T05:20:51.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:50 smithi196 ceph-mon[30802]: osdmap e700: 12 total, 12 up, 11 in 2024-09-16T05:20:51.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:50 smithi196 ceph-mon[30802]: osdmap e701: 12 total, 12 up, 11 in 2024-09-16T05:20:51.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:51.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:51.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:51.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:51.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:51.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:51.549 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:51.576 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:51.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:51.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:20:51.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:51.702 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:51.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:51.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:51.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:51.803 INFO:tasks.rados.rados.2.smithi204.stdout:2760: done (2 left) 2024-09-16T05:20:51.803 INFO:tasks.rados.rados.2.smithi204.stdout:2761: write (excl) oid 5 current snap is 288 2024-09-16T05:20:51.803 INFO:tasks.rados.rados.2.smithi204.stdout:2758: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:20:51.804 INFO:tasks.rados.rados.2.smithi204.stdout:2761: seq_num 980 ranges {470860=637204,1880824=590036} 2024-09-16T05:20:51.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:51.811 INFO:tasks.rados.rados.2.smithi204.stdout:2761: writing smithi20457400-5 from 470860 to 1108064 tid 1 2024-09-16T05:20:51.818 INFO:tasks.rados.rados.2.smithi204.stdout:2761: writing smithi20457400-5 from 1880824 to 2470860 tid 2 2024-09-16T05:20:51.819 INFO:tasks.rados.rados.2.smithi204.stdout:2759: finishing write tid 1 to smithi20457400-4 2024-09-16T05:20:51.819 INFO:tasks.rados.rados.2.smithi204.stdout:2759: finishing write tid 2 to smithi20457400-4 2024-09-16T05:20:51.819 INFO:tasks.rados.rados.2.smithi204.stdout:2759: finishing write tid 3 to smithi20457400-4 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2759: finishing write tid 4 to smithi20457400-4 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2759: finishing write tid 5 to smithi20457400-4 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1005 (ObjNum 979 snap 288 seq_num 979) dirty exists 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2759: left oid 4 (ObjNum 979 snap 288 seq_num 979) 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2758: finishing copy_from to smithi20457400-17 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 846 (ObjNum 961 snap 284 seq_num 961) dirty exists 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2761: finishing write tid 1 to smithi20457400-5 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2758: done (2 left) 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2759: done (1 left) 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2762: write (excl) oid 29 current snap is 288 2024-09-16T05:20:51.820 INFO:tasks.rados.rados.2.smithi204.stdout:2762: seq_num 981 ranges {487667=428806,1655659=770995} 2024-09-16T05:20:51.824 INFO:tasks.rados.rados.2.smithi204.stdout:2762: writing smithi20457400-29 from 487667 to 916473 tid 1 2024-09-16T05:20:51.833 INFO:tasks.rados.rados.2.smithi204.stdout:2762: writing smithi20457400-29 from 1655659 to 2426654 tid 2 2024-09-16T05:20:51.833 INFO:tasks.rados.rados.2.smithi204.stdout:2763: snap_create 2024-09-16T05:20:51.833 INFO:tasks.rados.rados.2.smithi204.stdout:2761: finishing write tid 2 to smithi20457400-5 2024-09-16T05:20:51.833 INFO:tasks.rados.rados.2.smithi204.stdout:2761: finishing write tid 3 to smithi20457400-5 2024-09-16T05:20:51.833 INFO:tasks.rados.rados.2.smithi204.stdout:2761: finishing write tid 4 to smithi20457400-5 2024-09-16T05:20:51.833 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 614 (ObjNum 980 snap 288 seq_num 980) dirty exists 2024-09-16T05:20:51.833 INFO:tasks.rados.rados.2.smithi204.stdout:2761: left oid 5 (ObjNum 980 snap 288 seq_num 980) 2024-09-16T05:20:51.834 INFO:tasks.rados.rados.2.smithi204.stdout:2762: finishing write tid 1 to smithi20457400-29 2024-09-16T05:20:51.834 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:51.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:51.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:2762: finishing write tid 2 to smithi20457400-29 2024-09-16T05:20:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:2762: finishing write tid 3 to smithi20457400-29 2024-09-16T05:20:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:2762: finishing write tid 4 to smithi20457400-29 2024-09-16T05:20:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1069 (ObjNum 981 snap 288 seq_num 981) dirty exists 2024-09-16T05:20:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:2762: left oid 29 (ObjNum 981 snap 288 seq_num 981) 2024-09-16T05:20:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:51 smithi032 ceph-mon[27388]: pgmap v1211: 117 pgs: 117 active+clean; 305 MiB data, 924 MiB used, 983 GiB / 983 GiB avail; 1.4 MiB/s rd, 400 KiB/s wr, 1 op/s 2024-09-16T05:20:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:51 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:20:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:51 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:20:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:51 smithi032 ceph-mon[27388]: osdmap e702: 12 total, 12 up, 11 in 2024-09-16T05:20:51.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:51.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:51.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:52.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:52.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:52.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:51 smithi204 ceph-mon[30933]: pgmap v1211: 117 pgs: 117 active+clean; 305 MiB data, 924 MiB used, 983 GiB / 983 GiB avail; 1.4 MiB/s rd, 400 KiB/s wr, 1 op/s 2024-09-16T05:20:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:51 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:20:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:51 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:20:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:51 smithi204 ceph-mon[30933]: osdmap e702: 12 total, 12 up, 11 in 2024-09-16T05:20:52.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:52.210 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:52.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:51 smithi196 ceph-mon[30802]: pgmap v1211: 117 pgs: 117 active+clean; 305 MiB data, 924 MiB used, 983 GiB / 983 GiB avail; 1.4 MiB/s rd, 400 KiB/s wr, 1 op/s 2024-09-16T05:20:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:51 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:20:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:51 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:20:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:51 smithi196 ceph-mon[30802]: osdmap e702: 12 total, 12 up, 11 in 2024-09-16T05:20:52.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:52.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:52.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:52.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:52.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:52.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:52.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:52.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:52.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:52.680 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:52.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:52.705 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:52.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:52.804 INFO:tasks.rados.rados.2.smithi204.stdout:2761: done (2 left) 2024-09-16T05:20:52.804 INFO:tasks.rados.rados.2.smithi204.stdout:2762: done (1 left) 2024-09-16T05:20:52.804 INFO:tasks.rados.rados.2.smithi204.stdout:2763: done (0 left) 2024-09-16T05:20:52.804 INFO:tasks.rados.rados.2.smithi204.stdout:2764: snap_remove snap 273 2024-09-16T05:20:52.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:52.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:52.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:52.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:52.956 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:52.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:53.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:53.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2219902136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:53.058 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:52 smithi032 ceph-mon[27388]: osdmap e703: 12 total, 12 up, 11 in 2024-09-16T05:20:53.081 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:53.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2318 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 374.23408551400001, 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98387304600000003, 2024-09-16T05:20:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2318 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924656+0000", 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.341e-06 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924663+0000", 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6050000000000003e-06 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924673+0000", 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8830000000000001e-06 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924680+0000", 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2230000000000001e-06 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924730+0000", 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0256999999999997e-05 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924737+0000", 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1550000000000001e-06 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.908327+0000", 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98359007499999995 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.908528+0000", 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000200507 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2319 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:20:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 374.23404479499999, 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98447908799999995, 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2319 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5900000000000003e-07 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924698+0000", 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9309999999999998e-06 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924703+0000", 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9609999999999996e-06 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924707+0000", 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8990000000000003e-06 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924769+0000", 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1797999999999993e-05 2024-09-16T05:20:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924776+0000", 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.99e-06 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909060+0000", 2024-09-16T05:20:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98428400299999996 2024-09-16T05:20:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909175+0000", 2024-09-16T05:20:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115147 2024-09-16T05:20:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2320 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~0,checksum] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:20:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:20:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 374.23402140500002, 2024-09-16T05:20:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98482718000000002, 2024-09-16T05:20:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:20:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2320 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4200000000000002e-07 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924722+0000", 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4380000000000002e-06 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924726+0000", 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0709999999999996e-06 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924731+0000", 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0069999999999997e-06 2024-09-16T05:20:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924792+0000", 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1561000000000002e-05 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924796+0000", 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0400000000000003e-06 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909278+0000", 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98448187499999995 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909546+0000", 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026784600000000002 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2898 2.2b 2:d7c71ae7:::smithi20457400-5:head [copy-from ver 322] snapc 7c=[7c,7b,7a,76,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:20:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:20:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 331.452044677, 2024-09-16T05:20:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026725529000000001, 2024-09-16T05:20:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2898 2024-09-16T05:20:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:20:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:20:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706697+0000", 2024-09-16T05:20:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4789999999999999e-06 2024-09-16T05:20:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706705+0000", 2024-09-16T05:20:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0490000000000004e-06 2024-09-16T05:20:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706719+0000", 2024-09-16T05:20:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4231e-05 2024-09-16T05:20:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706728+0000", 2024-09-16T05:20:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9339999999999995e-06 2024-09-16T05:20:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706819+0000", 2024-09-16T05:20:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0402000000000006e-05 2024-09-16T05:20:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706891+0000", 2024-09-16T05:20:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2605e-05 2024-09-16T05:20:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:20:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.724332+0000", 2024-09-16T05:20:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017440699 2024-09-16T05:20:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.728315+0000", 2024-09-16T05:20:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003982546 2024-09-16T05:20:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733300+0000", 2024-09-16T05:20:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049854369999999997 2024-09-16T05:20:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733368+0000", 2024-09-16T05:20:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8133000000000006e-05 2024-09-16T05:20:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733421+0000", 2024-09-16T05:20:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3013999999999998e-05 2024-09-16T05:20:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2899 2.2b 2:d7c71ae7:::smithi20457400-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:20:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:20:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 331.45199510999998, 2024-09-16T05:20:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020779419, 2024-09-16T05:20:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2899 2024-09-16T05:20:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:20:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:20:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706746+0000", 2024-09-16T05:20:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1600000000000002e-07 2024-09-16T05:20:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706748+0000", 2024-09-16T05:20:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3140000000000002e-06 2024-09-16T05:20:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706754+0000", 2024-09-16T05:20:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2609999999999999e-06 2024-09-16T05:20:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706768+0000", 2024-09-16T05:20:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3788000000000001e-05 2024-09-16T05:20:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.707033+0000", 2024-09-16T05:20:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026479299999999998 2024-09-16T05:20:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.707061+0000", 2024-09-16T05:20:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8050000000000001e-05 2024-09-16T05:20:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727362+0000", 2024-09-16T05:20:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020301534 2024-09-16T05:20:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727439+0000", 2024-09-16T05:20:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6749000000000007e-05 2024-09-16T05:20:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727525+0000", 2024-09-16T05:20:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5513999999999995e-05 2024-09-16T05:20:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3152 2.3 e397/329 2:c018a787:::smithi20457400-18:head v 397'99, mlcod=375'96)", 2024-09-16T05:20:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:20:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 306.43585268700002, 2024-09-16T05:20:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066570259999999999, 2024-09-16T05:20:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:20:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.722888+0000", 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3300000000000002e-07 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726366+0000", 2024-09-16T05:20:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003478259 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726395+0000", 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8347000000000002e-05 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726404+0000", 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0790000000000007e-06 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726446+0000", 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2092999999999998e-05 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.726464+0000", 2024-09-16T05:20:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8014000000000001e-05 2024-09-16T05:20:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.729500+0000", 2024-09-16T05:20:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030358519999999999 2024-09-16T05:20:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:46.729545+0000", 2024-09-16T05:20:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4848999999999997e-05 2024-09-16T05:20:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3573 2.3 e436/329 2:c018a787:::smithi20457400-18:head v 436'117, mlcod=436'115)", 2024-09-16T05:20:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:20:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 267.29227008100003, 2024-09-16T05:20:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021202584, 2024-09-16T05:20:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:20:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:20:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866471+0000", 2024-09-16T05:20:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.119e-06 2024-09-16T05:20:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886258+0000", 2024-09-16T05:20:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019786313 2024-09-16T05:20:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886318+0000", 2024-09-16T05:20:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0550000000000001e-05 2024-09-16T05:20:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886330+0000", 2024-09-16T05:20:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2126999999999999e-05 2024-09-16T05:20:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886419+0000", 2024-09-16T05:20:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8892000000000002e-05 2024-09-16T05:20:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886447+0000", 2024-09-16T05:20:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7959000000000001e-05 2024-09-16T05:20:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.887619+0000", 2024-09-16T05:20:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001171526 2024-09-16T05:20:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.887673+0000", 2024-09-16T05:20:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4098000000000003e-05 2024-09-16T05:20:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:4202 2.3a e489/330 2:5f916a06:::smithi20457400-34:head v 489'522, mlcod=452'518)", 2024-09-16T05:20:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:18.822538+0000", 2024-09-16T05:20:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 214.33620233799999, 2024-09-16T05:20:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0068665649999999998, 2024-09-16T05:20:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.822538+0000", 2024-09-16T05:20:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.822538+0000", 2024-09-16T05:20:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.822538+0000", 2024-09-16T05:20:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7100000000000001e-07 2024-09-16T05:20:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826075+0000", 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035369889999999999 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826108+0000", 2024-09-16T05:20:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2583999999999998e-05 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826118+0000", 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0288999999999999e-05 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826171+0000", 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2947e-05 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.826190+0000", 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8858000000000001e-05 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.829366+0000", 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031762790000000002 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:18.829405+0000", 2024-09-16T05:20:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8448000000000002e-05 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:4460 2.3 e503/329 2:c018a787:::smithi20457400-18:head v 503'143, mlcod=495'141)", 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:33.073315+0000", 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 200.08542574500001, 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0053949599999999999, 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.073315+0000", 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.073315+0000", 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.073315+0000", 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1799999999999999e-07 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076711+0000", 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033964170000000001 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076763+0000", 2024-09-16T05:20:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1619000000000002e-05 2024-09-16T05:20:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.242 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:53.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2219902136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:53.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:52 smithi196 ceph-mon[30802]: osdmap e703: 12 total, 12 up, 11 in 2024-09-16T05:20:53.244 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2219902136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:20:53.244 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:52 smithi204 ceph-mon[30933]: osdmap e703: 12 total, 12 up, 11 in 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076773+0000", 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.047e-05 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076848+0000", 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4480000000000005e-05 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.076880+0000", 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2002000000000003e-05 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.078645+0000", 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001764821 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:33.078710+0000", 2024-09-16T05:20:53.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4932999999999996e-05 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4678 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~2974971,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 188.352590771, 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26066084699999997, 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.247 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4678 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:20:53.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806152+0000", 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0360000000000001e-06 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806162+0000", 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0380000000000001e-05 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806180+0000", 2024-09-16T05:20:53.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8442999999999999e-05 2024-09-16T05:20:53.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806193+0000", 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914999999999999e-05 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806258+0000", 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4830000000000001e-05 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806263+0000", 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6959999999999996e-06 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.064487+0000", 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25822445799999999 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.064549+0000", 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1278999999999994e-05 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066810+0000", 2024-09-16T05:20:53.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022618099999999999 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4679 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~1225823] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 188.35252242799999, 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26074116000000003, 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:53.256 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4679 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806219+0000", 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806221+0000", 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7159999999999999e-06 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806228+0000", 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4799999999999998e-06 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806234+0000", 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0939999999999996e-06 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806309+0000", 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5340999999999996e-05 2024-09-16T05:20:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806312+0000", 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0620000000000001e-06 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066832+0000", 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26051971200000001 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066880+0000", 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7602000000000003e-05 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066959+0000", 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9597000000000004e-05 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4680 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~2974971,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 188.352478706, 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26141080300000002, 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4680 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.99e-07 2024-09-16T05:20:53.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806267+0000", 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3050000000000003e-06 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806275+0000", 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2930000000000001e-06 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806281+0000", 2024-09-16T05:20:53.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2779999999999999e-06 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806346+0000", 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5061999999999995e-05 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806354+0000", 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0669999999999992e-06 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066986+0000", 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.260631273 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.067038+0000", 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2138999999999997e-05 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.067672+0000", 2024-09-16T05:20:53.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063478700000000005 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5712 2.2b 2:d7c71ae7:::smithi20457400-5:head [copy-from ver 601] snapc ee=[ee,ed,ec,eb,ea,e9,e8,e7,e6,e5,e0,d9,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e597)", 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:06.981347+0000", 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 106.177393165, 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019455719, 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.267 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5712 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981347+0000", 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.268 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981347+0000", 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981349+0000", 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5e-06 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981358+0000", 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1200000000000008e-06 2024-09-16T05:20:53.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981379+0000", 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1044999999999999e-05 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981388+0000", 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.577e-06 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981441+0000", 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3003000000000001e-05 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981508+0000", 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6804999999999995e-05 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.995338+0000", 2024-09-16T05:20:53.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013829641 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.999508+0000", 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041704999999999997 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:07.000738+0000", 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00122931 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:07.000774+0000", 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146999999999998e-05 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:07.000803+0000", 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9071e-05 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5713 2.2b 2:d7c71ae7:::smithi20457400-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e597)", 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:06.981408+0000", 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 106.17733243399999, 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017268882999999999, 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5713 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.274 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981408+0000", 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981408+0000", 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981408+0000", 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8299999999999998e-07 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981411+0000", 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3470000000000001e-06 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981417+0000", 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0780000000000001e-06 2024-09-16T05:20:53.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981421+0000", 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8720000000000004e-06 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981637+0000", 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021596500000000001 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981686+0000", 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9379000000000003e-05 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.998521+0000", 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016834644999999999 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.998601+0000", 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0031000000000007e-05 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.998677+0000", 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6182999999999999e-05 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5986 2.2b 2:d7c71ae7:::smithi20457400-5:head [append 0~0] snapc f7=[f7,f6,f5,f2,f0,ef,ed,e8,e7,e5,e0,ca] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.795670+0000", 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 86.363070132999994, 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092971640000000001, 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.280 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5986 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795670+0000", 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795670+0000", 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795672+0000", 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0150000000000002e-06 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795679+0000", 2024-09-16T05:20:53.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3509999999999998e-06 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795700+0000", 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1118999999999999e-05 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795711+0000", 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0893000000000001e-05 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795791+0000", 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0489999999999997e-05 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795799+0000", 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7069999999999992e-06 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.801548+0000", 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057488770000000003 2024-09-16T05:20:53.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.801651+0000", 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103482 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.801870+0000", 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021886200000000001 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.804237+0000", 2024-09-16T05:20:53.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023670620000000001 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.804838+0000", 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060096300000000004 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.804930+0000", 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1878000000000004e-05 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.804967+0000", 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7465000000000001e-05 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5987 2.2b 2:d7c71ae7:::smithi20457400-5:head [rollback e5] snapc f7=[f7,f6,f5,f2,f0,ef,ed,e8,e7,e5,e0,ca] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.795737+0000", 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 86.363003559999996, 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094084870000000001, 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5987 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795737+0000", 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795737+0000", 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795737+0000", 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9299999999999998e-07 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795750+0000", 2024-09-16T05:20:53.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2248e-05 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795758+0000", 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7830000000000004e-06 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795765+0000", 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.007e-06 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795878+0000", 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000112965 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795885+0000", 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1809999999999998e-06 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.802621+0000", 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067368749999999998 2024-09-16T05:20:53.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.802716+0000", 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4724999999999999e-05 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.802885+0000", 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016901399999999999 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:53.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.804974+0000", 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020885449999999998 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.805033+0000", 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9718e-05 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.805096+0000", 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2169999999999996e-05 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.805145+0000", 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9663000000000002e-05 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5988 2.2b 2:d7c71ae7:::smithi20457400-5:head [append 0~0] snapc f7=[f7,f6,f5,f2,f0,ef,ed,e8,e7,e5,e0,ca] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.795776+0000", 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 86.362964527000003, 2024-09-16T05:20:53.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094804280000000008, 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5988 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795776+0000", 2024-09-16T05:20:53.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795776+0000", 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795776+0000", 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3499999999999996e-07 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795779+0000", 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.452e-06 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795786+0000", 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4909999999999999e-06 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795792+0000", 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7470000000000002e-06 2024-09-16T05:20:53.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795917+0000", 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012472199999999999 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.795922+0000", 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9919999999999998e-06 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.803300+0000", 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0073783260000000002 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.803371+0000", 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0523000000000002e-05 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.803520+0000", 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014963800000000001 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.805186+0000", 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001665438 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.805193+0000", 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3339999999999996e-06 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.805231+0000", 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7645999999999998e-05 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.805256+0000", 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5584000000000001e-05 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:6086 2.4d 2:b21a57e7:::smithi20457400-25:head [append 0~0] snapc fa=[fa,f9,f7,f6,f2,f0,ef,e8,e7] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:35.784567+0000", 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.374173721000005, 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0059319120000000001, 2024-09-16T05:20:53.304 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 6086 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784567+0000", 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784567+0000", 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784568+0000", 2024-09-16T05:20:53.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6759999999999999e-06 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784573+0000", 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9370000000000003e-06 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784587+0000", 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3816e-05 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.307 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784601+0000", 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3705e-05 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784659+0000", 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8517999999999998e-05 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784767+0000", 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010724400000000001 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784973+0000", 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020639599999999999 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.786742+0000", 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017694309999999999 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.790368+0000", 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036257099999999999 2024-09-16T05:20:53.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.790447+0000", 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9078000000000004e-05 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.790499+0000", 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1400999999999998e-05 2024-09-16T05:20:53.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.311 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.311 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.311 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:6087 2.4d 2:b21a57e7:::smithi20457400-25:head [rollback f0] snapc fa=[fa,f9,f7,f6,f2,f0,ef,e8,e7] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-09-16T05:20:53.311 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:35.784622+0000", 2024-09-16T05:20:53.311 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.374118499999994, 2024-09-16T05:20:53.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0060226660000000003, 2024-09-16T05:20:53.312 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.312 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.312 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.312 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.312 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 6087 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784622+0000", 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784622+0000", 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784622+0000", 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6899999999999997e-07 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784625+0000", 2024-09-16T05:20:53.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0350000000000002e-06 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784633+0000", 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8469999999999996e-06 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784639+0000", 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8810000000000001e-06 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.785591+0000", 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00095169799999999995 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.785675+0000", 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4578000000000002e-05 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.785870+0000", 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019424000000000001 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.786746+0000", 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00087579200000000002 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.790547+0000", 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003801357 2024-09-16T05:20:53.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.790601+0000", 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4029000000000001e-05 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.790645+0000", 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3640000000000002e-05 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:6088 2.4d 2:b21a57e7:::smithi20457400-25:head [append 0~0] snapc fa=[fa,f9,f7,f6,f2,f0,ef,e8,e7] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:35.784647+0000", 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.374093508000001, 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063777950000000003, 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 6088 2024-09-16T05:20:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784647+0000", 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784647+0000", 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784647+0000", 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8099999999999998e-07 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784649+0000", 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0710000000000002e-06 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784656+0000", 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9299999999999997e-06 2024-09-16T05:20:53.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.784662+0000", 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4750000000000001e-06 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.786310+0000", 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016481009999999999 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.786384+0000", 2024-09-16T05:20:53.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4543999999999996e-05 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.786552+0000", 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016740400000000001 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.786859+0000", 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00030682300000000001 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.323 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.790925+0000", 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0040660000000000002 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.790983+0000", 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8038999999999998e-05 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:35.791025+0000", 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2027000000000003e-05 2024-09-16T05:20:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.325 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:53.325 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:53.325 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:20:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:53.327 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:53.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:53.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:53.450 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:53.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:53.484 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:20:53.485 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:20:53.506 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:20:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:20:53.526 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:20:53.527 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:20:53.547 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:20:53.547 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:20:53.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:53.568 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:20:53.569 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:20:53.574 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:53.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:53.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:20:53.591 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:20:53.613 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:20:53.613 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:20:53.634 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:20:53.634 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:20:53.655 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:20:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:20:53.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:53.679 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:20:53.679 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:20:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:53.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:53.700 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:20:53.700 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:20:53.721 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:20:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:20:53.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:53.806 INFO:tasks.rados.rados.2.smithi204.stdout:2764: done (0 left) 2024-09-16T05:20:53.807 INFO:tasks.rados.rados.2.smithi204.stdout:2765: write (excl) oid 21 current snap is 289 2024-09-16T05:20:53.807 INFO:tasks.rados.rados.2.smithi204.stdout:2765: seq_num 982 ranges {504474=504055,1714141=631006} 2024-09-16T05:20:53.813 INFO:tasks.rados.rados.2.smithi204.stdout:2765: writing smithi20457400-21 from 504474 to 1008529 tid 1 2024-09-16T05:20:53.822 INFO:tasks.rados.rados.2.smithi204.stdout:2765: writing smithi20457400-21 from 1714141 to 2345147 tid 2 2024-09-16T05:20:53.822 INFO:tasks.rados.rados.2.smithi204.stdout:2766: write (excl) oid 18 current snap is 289 2024-09-16T05:20:53.822 INFO:tasks.rados.rados.2.smithi204.stdout:2765: finishing write tid 1 to smithi20457400-21 2024-09-16T05:20:53.822 INFO:tasks.rados.rados.2.smithi204.stdout:2766: seq_num 983 ranges {521281=579304,1656270=774664} 2024-09-16T05:20:53.828 INFO:tasks.rados.rados.2.smithi204.stdout:2766: writing smithi20457400-18 from 521281 to 1100585 tid 1 2024-09-16T05:20:53.831 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:53.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:53.835 INFO:tasks.rados.rados.2.smithi204.stdout:2766: writing smithi20457400-18 from 1656270 to 2430934 tid 2 2024-09-16T05:20:53.835 INFO:tasks.rados.rados.2.smithi204.stdout:2767: write oid 4 current snap is 289 2024-09-16T05:20:53.835 INFO:tasks.rados.rados.2.smithi204.stdout:2767: seq_num 984 ranges {538088=654553,1714752=634675} 2024-09-16T05:20:53.842 INFO:tasks.rados.rados.2.smithi204.stdout:2767: writing smithi20457400-4 from 538088 to 1192641 tid 1 2024-09-16T05:20:53.849 INFO:tasks.rados.rados.2.smithi204.stdout:2767: writing smithi20457400-4 from 1714752 to 2349427 tid 2 2024-09-16T05:20:53.849 INFO:tasks.rados.rados.2.smithi204.stdout:2768: write (excl) oid 26 current snap is 289 2024-09-16T05:20:53.849 INFO:tasks.rados.rados.2.smithi204.stdout:2765: finishing write tid 2 to smithi20457400-21 2024-09-16T05:20:53.849 INFO:tasks.rados.rados.2.smithi204.stdout:2765: finishing write tid 3 to smithi20457400-21 2024-09-16T05:20:53.849 INFO:tasks.rados.rados.2.smithi204.stdout:2765: finishing write tid 4 to smithi20457400-21 2024-09-16T05:20:53.849 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 851 (ObjNum 982 snap 289 seq_num 982) dirty exists 2024-09-16T05:20:53.849 INFO:tasks.rados.rados.2.smithi204.stdout:2765: left oid 21 (ObjNum 982 snap 289 seq_num 982) 2024-09-16T05:20:53.850 INFO:tasks.rados.rados.2.smithi204.stdout:2766: finishing write tid 1 to smithi20457400-18 2024-09-16T05:20:53.850 INFO:tasks.rados.rados.2.smithi204.stdout:2766: finishing write tid 2 to smithi20457400-18 2024-09-16T05:20:53.850 INFO:tasks.rados.rados.2.smithi204.stdout:2766: finishing write tid 3 to smithi20457400-18 2024-09-16T05:20:53.850 INFO:tasks.rados.rados.2.smithi204.stdout:2766: finishing write tid 4 to smithi20457400-18 2024-09-16T05:20:53.850 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 542 (ObjNum 983 snap 289 seq_num 983) dirty exists 2024-09-16T05:20:53.850 INFO:tasks.rados.rados.2.smithi204.stdout:2766: left oid 18 (ObjNum 983 snap 289 seq_num 983) 2024-09-16T05:20:53.850 INFO:tasks.rados.rados.2.smithi204.stdout:2767: finishing write tid 1 to smithi20457400-4 2024-09-16T05:20:53.850 INFO:tasks.rados.rados.2.smithi204.stdout:2768: seq_num 985 ranges {554895=729802,1773234=778333} 2024-09-16T05:20:53.855 INFO:tasks.rados.rados.2.smithi204.stdout:2768: writing smithi20457400-26 from 554895 to 1284697 tid 1 2024-09-16T05:20:53.862 INFO:tasks.rados.rados.2.smithi204.stdout:2768: writing smithi20457400-26 from 1773234 to 2551567 tid 2 2024-09-16T05:20:53.862 INFO:tasks.rados.rados.2.smithi204.stdout:2765: done (3 left) 2024-09-16T05:20:53.862 INFO:tasks.rados.rados.2.smithi204.stdout:2766: done (2 left) 2024-09-16T05:20:53.864 INFO:tasks.rados.rados.2.smithi204.stdout:2769: snap_remove snap 240 2024-09-16T05:20:53.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:53.956 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:53.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:54.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:54.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:54.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:54.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:20:54.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:53 smithi032 ceph-mon[27388]: pgmap v1215: 117 pgs: 117 active+clean; 318 MiB data, 943 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 15 op/s 2024-09-16T05:20:54.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:53 smithi032 ceph-mon[27388]: osdmap e704: 12 total, 12 up, 11 in 2024-09-16T05:20:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:53 smithi204 ceph-mon[30933]: pgmap v1215: 117 pgs: 117 active+clean; 318 MiB data, 943 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 15 op/s 2024-09-16T05:20:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:53 smithi204 ceph-mon[30933]: osdmap e704: 12 total, 12 up, 11 in 2024-09-16T05:20:54.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:54.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:54.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:54.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:53 smithi196 ceph-mon[30802]: pgmap v1215: 117 pgs: 117 active+clean; 318 MiB data, 943 MiB used, 983 GiB / 983 GiB avail; 5.8 MiB/s rd, 3.6 MiB/s wr, 15 op/s 2024-09-16T05:20:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:53 smithi196 ceph-mon[30802]: osdmap e704: 12 total, 12 up, 11 in 2024-09-16T05:20:54.251 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:20:54.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:54.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:54.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:54.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:54.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:54.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:54.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:54.587 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:54.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:54.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:54.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:54.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:54.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:54.843 INFO:tasks.rados.rados.2.smithi204.stdout:2769: done (2 left) 2024-09-16T05:20:54.843 INFO:tasks.rados.rados.2.smithi204.stdout:2770: read oid 41 snap -1 2024-09-16T05:20:54.843 INFO:tasks.rados.rados.2.smithi204.stdout:2770: expect (ObjNum 971 snap 287 seq_num 971) 2024-09-16T05:20:54.843 INFO:tasks.rados.rados.2.smithi204.stdout:2771: snap_create 2024-09-16T05:20:54.843 INFO:tasks.rados.rados.2.smithi204.stdout:2767: finishing write tid 2 to smithi20457400-4 2024-09-16T05:20:54.843 INFO:tasks.rados.rados.2.smithi204.stdout:2767: finishing write tid 3 to smithi20457400-4 2024-09-16T05:20:54.843 INFO:tasks.rados.rados.2.smithi204.stdout:2767: finishing write tid 4 to smithi20457400-4 2024-09-16T05:20:54.844 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1008 (ObjNum 984 snap 289 seq_num 984) dirty exists 2024-09-16T05:20:54.844 INFO:tasks.rados.rados.2.smithi204.stdout:2767: left oid 4 (ObjNum 984 snap 289 seq_num 984) 2024-09-16T05:20:54.844 INFO:tasks.rados.rados.2.smithi204.stdout:2768: finishing write tid 1 to smithi20457400-26 2024-09-16T05:20:54.844 INFO:tasks.rados.rados.2.smithi204.stdout:2768: finishing write tid 2 to smithi20457400-26 2024-09-16T05:20:54.844 INFO:tasks.rados.rados.2.smithi204.stdout:2768: finishing write tid 3 to smithi20457400-26 2024-09-16T05:20:54.844 INFO:tasks.rados.rados.2.smithi204.stdout:2768: finishing write tid 4 to smithi20457400-26 2024-09-16T05:20:54.844 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1252 (ObjNum 985 snap 289 seq_num 985) dirty exists 2024-09-16T05:20:54.844 INFO:tasks.rados.rados.2.smithi204.stdout:2768: left oid 26 (ObjNum 985 snap 289 seq_num 985) 2024-09-16T05:20:54.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:54.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:54.854 INFO:tasks.rados.rados.2.smithi204.stdout:2770: expect (ObjNum 971 snap 287 seq_num 971) 2024-09-16T05:20:54.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:54.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:54.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:55.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:55.087 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:55.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:55.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:54 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:20:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:54 smithi032 ceph-mon[27388]: osdmap e705: 12 total, 12 up, 11 in 2024-09-16T05:20:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:54 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:20:55.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:54 smithi204 ceph-mon[30933]: osdmap e705: 12 total, 12 up, 11 in 2024-09-16T05:20:55.196 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:55.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:55.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:55.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:54 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:20:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:54 smithi196 ceph-mon[30802]: osdmap e705: 12 total, 12 up, 11 in 2024-09-16T05:20:55.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:55.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:55.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:55.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:55.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:55.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:55.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:55.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:55.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:55.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:55.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:55.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:55.804 INFO:tasks.rados.rados.2.smithi204.stdout:2767: done (3 left) 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:2768: done (2 left) 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:2770: done (1 left) 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:2771: done (0 left) 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:2772: rollback oid 10 current snap is 290 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 262 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:2773: copy_from oid 12 from oid 5 current snap is 290 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:2774: read oid 13 snap -1 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:2774: expect deleted 2024-09-16T05:20:55.805 INFO:tasks.rados.rados.2.smithi204.stdout:2775: snap_create 2024-09-16T05:20:55.810 INFO:tasks.rados.rados.2.smithi204.stdout:2772: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:20:55.810 INFO:tasks.rados.rados.2.smithi204.stdout:2772: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:20:55.811 INFO:tasks.rados.rados.2.smithi204.stdout:2772: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:20:55.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1081 (ObjNum 894 snap 262 seq_num 894) dirty exists 2024-09-16T05:20:55.819 INFO:tasks.rados.rados.2.smithi204.stdout:2773: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:20:55.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:55.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:55.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:55.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:55.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:55.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:56.069 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:56.090 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:56.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:56.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:55 smithi032 ceph-mon[27388]: pgmap v1218: 117 pgs: 1 peering, 116 active+clean; 332 MiB data, 965 MiB used, 982 GiB / 983 GiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 29 op/s 2024-09-16T05:20:56.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:55 smithi032 ceph-mon[27388]: osdmap e706: 12 total, 12 up, 11 in 2024-09-16T05:20:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:55 smithi204 ceph-mon[30933]: pgmap v1218: 117 pgs: 1 peering, 116 active+clean; 332 MiB data, 965 MiB used, 982 GiB / 983 GiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 29 op/s 2024-09-16T05:20:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:55 smithi204 ceph-mon[30933]: osdmap e706: 12 total, 12 up, 11 in 2024-09-16T05:20:56.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:56.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:56.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:56.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:55 smithi196 ceph-mon[30802]: pgmap v1218: 117 pgs: 1 peering, 116 active+clean; 332 MiB data, 965 MiB used, 982 GiB / 983 GiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 29 op/s 2024-09-16T05:20:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:55 smithi196 ceph-mon[30802]: osdmap e706: 12 total, 12 up, 11 in 2024-09-16T05:20:56.239 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:20:56.239 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:20:56.241 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:20:56.242 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:20:56.242 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T05:20:56.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:20:56.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:56.339 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:56.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:56.371 INFO:tasks.rados.rados.2.smithi204.stdout:2773: finishing copy_from to smithi20457400-12 2024-09-16T05:20:56.371 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 614 (ObjNum 980 snap 288 seq_num 980) dirty exists 2024-09-16T05:20:56.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:56.461 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:56.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:56.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:56.587 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:56.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:56.602 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:56.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:56.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:56.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:56.806 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:20:56.812 INFO:tasks.rados.rados.2.smithi204.stdout:2772: done (3 left) 2024-09-16T05:20:56.812 INFO:tasks.rados.rados.2.smithi204.stdout:2773: done (2 left) 2024-09-16T05:20:56.812 INFO:tasks.rados.rados.2.smithi204.stdout:2774: done (1 left) 2024-09-16T05:20:56.812 INFO:tasks.rados.rados.2.smithi204.stdout:2775: done (0 left) 2024-09-16T05:20:56.812 INFO:tasks.rados.rados.2.smithi204.stdout:2776: read oid 35 snap 281 2024-09-16T05:20:56.812 INFO:tasks.rados.rados.2.smithi204.stdout:2776: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:20:56.812 INFO:tasks.rados.rados.2.smithi204.stdout:2777: snap_create 2024-09-16T05:20:56.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:56.817 INFO:tasks.rados.rados.2.smithi204.stdout:2776: expect (ObjNum 755 snap 220 seq_num 755) 2024-09-16T05:20:56.835 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:56.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:56.937 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:56.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:56.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:56.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:20:56.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2300620664' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:56.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:56 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:56.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2300620664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:56.998 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:56 smithi032 ceph-mon[27388]: osdmap e707: 12 total, 12 up, 11 in 2024-09-16T05:20:57.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:57.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:57.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:57.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:20:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2300620664' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:56 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2300620664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:56 smithi204 ceph-mon[30933]: osdmap e707: 12 total, 12 up, 11 in 2024-09-16T05:20:57.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:57.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:57.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:57.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2300620664' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:20:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:56 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:20:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2300620664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:20:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:56 smithi196 ceph-mon[30802]: osdmap e707: 12 total, 12 up, 11 in 2024-09-16T05:20:57.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:57.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:57.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:57.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:20:57.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:57.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:57.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:57.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:57.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:57.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:20:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:57.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:57.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:20:57.811 INFO:tasks.rados.rados.2.smithi204.stdout:2776: done (1 left) 2024-09-16T05:20:57.812 INFO:tasks.rados.rados.2.smithi204.stdout:2777: done (0 left) 2024-09-16T05:20:57.812 INFO:tasks.rados.rados.2.smithi204.stdout:2778: rollback oid 10 current snap is 292 2024-09-16T05:20:57.812 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 262 2024-09-16T05:20:57.812 INFO:tasks.rados.rados.2.smithi204.stdout:2779: write oid 42 current snap is 292 2024-09-16T05:20:57.812 INFO:tasks.rados.rados.2.smithi204.stdout:2779: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-09-16T05:20:57.816 INFO:tasks.rados.rados.2.smithi204.stdout:2779: writing smithi20457400-42 from 571702 to 976753 tid 1 2024-09-16T05:20:57.822 INFO:tasks.rados.rados.2.smithi204.stdout:2779: writing smithi20457400-42 from 1431716 to 2070060 tid 2 2024-09-16T05:20:57.825 INFO:tasks.rados.rados.2.smithi204.stdout:2779: writing smithi20457400-42 from 2560365 to 2571702 tid 3 2024-09-16T05:20:57.825 INFO:tasks.rados.rados.2.smithi204.stdout:2780: write oid 39 current snap is 292 2024-09-16T05:20:57.825 INFO:tasks.rados.rados.2.smithi204.stdout:2780: seq_num 987 ranges {588509=480300,1773845=782002} 2024-09-16T05:20:57.830 INFO:tasks.rados.rados.2.smithi204.stdout:2780: writing smithi20457400-39 from 588509 to 1068809 tid 1 2024-09-16T05:20:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:2780: writing smithi20457400-39 from 1773845 to 2555847 tid 2 2024-09-16T05:20:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:2781: delete oid 23 current snap is 292 2024-09-16T05:20:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:2778: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:20:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:2778: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:2778: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1085 (ObjNum 894 snap 262 seq_num 894) dirty exists 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:2779: finishing write tid 1 to smithi20457400-42 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:2779: finishing write tid 2 to smithi20457400-42 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:2779: finishing write tid 3 to smithi20457400-42 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:2779: finishing write tid 4 to smithi20457400-42 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:2779: finishing write tid 5 to smithi20457400-42 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 781 (ObjNum 986 snap 292 seq_num 986) dirty exists 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:2779: left oid 42 (ObjNum 986 snap 292 seq_num 986) 2024-09-16T05:20:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:2780: finishing write tid 1 to smithi20457400-39 2024-09-16T05:20:57.842 INFO:tasks.rados.rados.2.smithi204.stdout:2780: finishing write tid 2 to smithi20457400-39 2024-09-16T05:20:57.842 INFO:tasks.rados.rados.2.smithi204.stdout:2780: finishing write tid 3 to smithi20457400-39 2024-09-16T05:20:57.842 INFO:tasks.rados.rados.2.smithi204.stdout:2780: finishing write tid 4 to smithi20457400-39 2024-09-16T05:20:57.843 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1175 (ObjNum 987 snap 292 seq_num 987) dirty exists 2024-09-16T05:20:57.843 INFO:tasks.rados.rados.2.smithi204.stdout:2780: left oid 39 (ObjNum 987 snap 292 seq_num 987) 2024-09-16T05:20:57.843 INFO:tasks.rados.rados.2.smithi204.stdout:2778: done (3 left) 2024-09-16T05:20:57.843 INFO:tasks.rados.rados.2.smithi204.stdout:2779: done (2 left) 2024-09-16T05:20:57.844 INFO:tasks.rados.rados.2.smithi204.stdout:2780: done (1 left) 2024-09-16T05:20:57.844 INFO:tasks.rados.rados.2.smithi204.stdout:2781: done (0 left) 2024-09-16T05:20:57.844 INFO:tasks.rados.rados.2.smithi204.stdout:2782: copy_from oid 28 from oid 41 current snap is 292 2024-09-16T05:20:57.844 INFO:tasks.rados.rados.2.smithi204.stdout:2783: copy_from oid 21 from oid 31 current snap is 292 2024-09-16T05:20:57.844 INFO:tasks.rados.rados.2.smithi204.stdout:2784: write oid 20 current snap is 292 2024-09-16T05:20:57.844 INFO:tasks.rados.rados.2.smithi204.stdout:2784: seq_num 989 ranges {622123=747151,2007162=614961} 2024-09-16T05:20:57.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:57.851 INFO:tasks.rados.rados.2.smithi204.stdout:2784: writing smithi20457400-20 from 622123 to 1369274 tid 1 2024-09-16T05:20:57.858 INFO:tasks.rados.rados.2.smithi204.stdout:2784: writing smithi20457400-20 from 2007162 to 2622123 tid 2 2024-09-16T05:20:57.858 INFO:tasks.rados.rados.2.smithi204.stdout:2785: write oid 40 current snap is 292 2024-09-16T05:20:57.858 INFO:tasks.rados.rados.2.smithi204.stdout:2783: finishing copy_from to smithi20457400-21 2024-09-16T05:20:57.858 INFO:tasks.rados.rados.2.smithi204.stdout:2783: got expected ENOENT (src dne) 2024-09-16T05:20:57.858 INFO:tasks.rados.rados.2.smithi204.stdout:2783: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:20:57.858 INFO:tasks.rados.rados.2.smithi204.stdout:2784: finishing write tid 1 to smithi20457400-20 2024-09-16T05:20:57.858 INFO:tasks.rados.rados.2.smithi204.stdout:2782: finishing copy_from racing read to smithi20457400-28 2024-09-16T05:20:57.859 INFO:tasks.rados.rados.2.smithi204.stdout:2782: finishing copy_from to smithi20457400-28 2024-09-16T05:20:57.859 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1047 (ObjNum 971 snap 287 seq_num 971) dirty exists 2024-09-16T05:20:57.859 INFO:tasks.rados.rados.2.smithi204.stdout:2785: seq_num 990 ranges {638930=422400,1665644=645682} 2024-09-16T05:20:57.863 INFO:tasks.rados.rados.2.smithi204.stdout:2785: writing smithi20457400-40 from 638930 to 1061330 tid 1 2024-09-16T05:20:57.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:57.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:57.871 INFO:tasks.rados.rados.2.smithi204.stdout:2785: writing smithi20457400-40 from 1665644 to 2311326 tid 2 2024-09-16T05:20:57.871 INFO:tasks.rados.rados.2.smithi204.stdout:2782: done (3 left) 2024-09-16T05:20:57.871 INFO:tasks.rados.rados.2.smithi204.stdout:2783: done (2 left) 2024-09-16T05:20:57.871 INFO:tasks.rados.rados.2.smithi204.stdout:2786: copy_from oid 9 from oid 26 current snap is 292 2024-09-16T05:20:57.871 INFO:tasks.rados.rados.2.smithi204.stdout:2784: finishing write tid 2 to smithi20457400-20 2024-09-16T05:20:57.871 INFO:tasks.rados.rados.2.smithi204.stdout:2784: finishing write tid 3 to smithi20457400-20 2024-09-16T05:20:57.872 INFO:tasks.rados.rados.2.smithi204.stdout:2784: finishing write tid 4 to smithi20457400-20 2024-09-16T05:20:57.872 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1179 (ObjNum 989 snap 292 seq_num 989) dirty exists 2024-09-16T05:20:57.872 INFO:tasks.rados.rados.2.smithi204.stdout:2784: left oid 20 (ObjNum 989 snap 292 seq_num 989) 2024-09-16T05:20:57.872 INFO:tasks.rados.rados.2.smithi204.stdout:2785: finishing write tid 1 to smithi20457400-40 2024-09-16T05:20:57.872 INFO:tasks.rados.rados.2.smithi204.stdout:2784: done (2 left) 2024-09-16T05:20:57.872 INFO:tasks.rados.rados.2.smithi204.stdout:2787: delete oid 2 current snap is 292 2024-09-16T05:20:57.872 INFO:tasks.rados.rados.2.smithi204.stdout:2787: done (2 left) 2024-09-16T05:20:57.872 INFO:tasks.rados.rados.2.smithi204.stdout:2788: delete oid 47 current snap is 292 2024-09-16T05:20:57.873 INFO:tasks.rados.rados.2.smithi204.stdout:2785: finishing write tid 2 to smithi20457400-40 2024-09-16T05:20:57.873 INFO:tasks.rados.rados.2.smithi204.stdout:2785: finishing write tid 3 to smithi20457400-40 2024-09-16T05:20:57.873 INFO:tasks.rados.rados.2.smithi204.stdout:2785: finishing write tid 4 to smithi20457400-40 2024-09-16T05:20:57.873 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 780 (ObjNum 990 snap 292 seq_num 990) dirty exists 2024-09-16T05:20:57.873 INFO:tasks.rados.rados.2.smithi204.stdout:2785: left oid 40 (ObjNum 990 snap 292 seq_num 990) 2024-09-16T05:20:57.876 INFO:tasks.rados.rados.2.smithi204.stdout:2785: done (2 left) 2024-09-16T05:20:57.876 INFO:tasks.rados.rados.2.smithi204.stdout:2788: done (1 left) 2024-09-16T05:20:57.876 INFO:tasks.rados.rados.2.smithi204.stdout:2789: rollback oid 16 current snap is 292 2024-09-16T05:20:57.877 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 16 to 262 2024-09-16T05:20:57.877 INFO:tasks.rados.rados.2.smithi204.stdout:2790: copy_from oid 40 from oid 41 current snap is 292 2024-09-16T05:20:57.877 INFO:tasks.rados.rados.2.smithi204.stdout:2791: read oid 3 snap -1 2024-09-16T05:20:57.877 INFO:tasks.rados.rados.2.smithi204.stdout:2791: expect (ObjNum 919 snap 273 seq_num 919) 2024-09-16T05:20:57.877 INFO:tasks.rados.rados.2.smithi204.stdout:2792: copy_from oid 34 from oid 42 current snap is 292 2024-09-16T05:20:57.877 INFO:tasks.rados.rados.2.smithi204.stdout:2793: write oid 36 current snap is 292 2024-09-16T05:20:57.877 INFO:tasks.rados.rados.2.smithi204.stdout:2793: seq_num 993 ranges {689351=648147,2124737=564614} 2024-09-16T05:20:57.884 INFO:tasks.rados.rados.2.smithi204.stdout:2793: writing smithi20457400-36 from 689351 to 1337498 tid 1 2024-09-16T05:20:57.892 INFO:tasks.rados.rados.2.smithi204.stdout:2793: writing smithi20457400-36 from 2124737 to 2689351 tid 2 2024-09-16T05:20:57.892 INFO:tasks.rados.rados.2.smithi204.stdout:2794: read oid 6 snap -1 2024-09-16T05:20:57.892 INFO:tasks.rados.rados.2.smithi204.stdout:2794: expect (ObjNum 906 snap 267 seq_num 906) 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:2795: read oid 37 snap 286 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:2795: expect (ObjNum 730 snap 210 seq_num 730) 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:2796: write (excl) oid 2 current snap is 292 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:2789: finishing rollback tid 0 to smithi20457400-16 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:2789: finishing rollback tid 1 to smithi20457400-16 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:2789: finishing rollback tid 2 to smithi20457400-16 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1021 (ObjNum 864 snap 249 seq_num 864) dirty exists 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:2786: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:20:57.893 INFO:tasks.rados.rados.2.smithi204.stdout:2796: seq_num 994 ranges {706158=723396,2183219=522939} 2024-09-16T05:20:57.901 INFO:tasks.rados.rados.2.smithi204.stdout:2796: writing smithi20457400-2 from 706158 to 1429554 tid 1 2024-09-16T05:20:57.908 INFO:tasks.rados.rados.2.smithi204.stdout:2796: writing smithi20457400-2 from 2183219 to 2706158 tid 2 2024-09-16T05:20:57.908 INFO:tasks.rados.rados.2.smithi204.stdout:2789: done (8 left) 2024-09-16T05:20:57.908 INFO:tasks.rados.rados.2.smithi204.stdout:2797: snap_remove snap 266 2024-09-16T05:20:57.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:20:57.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:57.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:20:58.063 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:57 smithi032 ceph-mon[27388]: pgmap v1221: 117 pgs: 1 peering, 116 active+clean; 308 MiB data, 934 MiB used, 983 GiB / 983 GiB avail; 6.3 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-16T05:20:58.063 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:57 smithi032 ceph-mon[27388]: osdmap e708: 12 total, 12 up, 11 in 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 708, 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:20:57.806279+0000", 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:14:38.892129+0000", 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:19:34.779035+0000", 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5804032, 2024-09-16T05:20:58.064 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 36, 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:20:58.065 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "luminous", 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:20:58.066 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 31, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 31, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 31, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 31, 2024-09-16T05:20:58.067 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 31, 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "687", 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "685", 2024-09-16T05:20:58.068 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:20:58.069 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:20:58.069 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:20:58.069 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:20:58.070 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:20:58.071 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:20:58.072 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.3199999332427979, 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.3199999332427979, 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.92000001668930054, 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.130000114440918, 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.130000114440918, 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:20:58.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:20:58.074 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 86, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 86, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 86, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 86, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 86, 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:20:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "708", 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "435", 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:20:58.076 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 293, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 708, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:20:58.077 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:20:58.078 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:20:58.078 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:20:58.078 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:20:58.078 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:20:58.078 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:20:58.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:20:58.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:20:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 1.7999999523162842, 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 1.7999999523162842, 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.92000001668930054, 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 1.6599999666213989, 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 1.6599999666213989, 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:20:58.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.081 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 689, 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:20:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:20:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.087 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:20:58.087 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:58.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.087 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.087 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.087 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.088 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:20:58.089 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 0, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 0, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 611, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.090 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.092 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:58.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.094 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.095 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 21, 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 295, 2024-09-16T05:20:58.096 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 329, 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 703, 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 296, 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:20:58.097 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.099 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.100 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3973436581", 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3973436581", 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3973436581", 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3973436581", 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 685, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.102 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.104 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.105 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:20:58.106 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:20:58.107 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.108 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.109 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:20:58.112 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:20:58.113 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:20:58.114 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.114 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.114 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.114 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.114 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.114 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.114 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:20:58.114 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:58.115 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.115 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.115 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.115 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.115 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:20:58.119 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.120 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 689, 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:58.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:20:58.123 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:20:58.125 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.126 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 689, 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:58.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:20:58.128 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:20:58.131 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.132 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 702, 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.134 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.137 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:20:58.138 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 689, 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:58.139 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:20:58.140 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.143 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:20:58.143 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:20:58.143 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.143 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.143 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:20:58.143 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:20:58.143 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:20:58.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:20:58.144 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.145 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:58.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.150 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:20:58.150 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:20:58.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.150 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.150 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:20:58.152 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 690, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.153 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.154 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:20:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:20:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.158 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:09.291721+0000", 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.159 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 65536, 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:17.125178+0000", 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:44.703412+0000", 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 295 2024-09-16T05:20:58.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:50.830187+0000", 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.161 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:32.848448+0000", 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.162 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:20.219477+0000", 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:13.190588+0000", 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:50.053405+0000", 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.164 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:40.278582+0000", 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:46.829682+0000", 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:30.643238+0000", 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.166 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:19:58.888327+0000", 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [ 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "pgid": "1.1", 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:20:58.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2, 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: 3 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:58.168 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:20:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 242, 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 267, 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 275, 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 287, 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.171 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:20:58.172 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:20:58.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:57 smithi204 ceph-mon[30933]: pgmap v1221: 117 pgs: 1 peering, 116 active+clean; 308 MiB data, 934 MiB used, 983 GiB / 983 GiB avail; 6.3 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-16T05:20:58.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:57 smithi204 ceph-mon[30933]: osdmap e708: 12 total, 12 up, 11 in 2024-09-16T05:20:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:58.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:58.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:20:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:57 smithi196 ceph-mon[30802]: pgmap v1221: 117 pgs: 1 peering, 116 active+clean; 308 MiB data, 934 MiB used, 983 GiB / 983 GiB avail; 6.3 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-16T05:20:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:57 smithi196 ceph-mon[30802]: osdmap e708: 12 total, 12 up, 11 in 2024-09-16T05:20:58.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:20:58.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:58.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:20:58.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:58.442 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:58.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:58.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:20:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:58.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:58.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:20:58.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:58.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:58.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:58.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:58.825 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:58.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:58.906 INFO:tasks.rados.rados.2.smithi204.stdout:2797: done (8 left) 2024-09-16T05:20:58.906 INFO:tasks.rados.rados.2.smithi204.stdout:2798: write oid 44 current snap is 292 2024-09-16T05:20:58.906 INFO:tasks.rados.rados.2.smithi204.stdout:2798: seq_num 995 ranges {722965=798645,2125348=597617} 2024-09-16T05:20:58.915 INFO:tasks.rados.rados.2.smithi204.stdout:2798: writing smithi20457400-44 from 722965 to 1521610 tid 1 2024-09-16T05:20:58.922 INFO:tasks.rados.rados.2.smithi204.stdout:2798: writing smithi20457400-44 from 2125348 to 2722965 tid 2 2024-09-16T05:20:58.922 INFO:tasks.rados.rados.2.smithi204.stdout:2799: read oid 1 snap -1 2024-09-16T05:20:58.922 INFO:tasks.rados.rados.2.smithi204.stdout:2799: expect (ObjNum 977 snap 288 seq_num 977) 2024-09-16T05:20:58.922 INFO:tasks.rados.rados.2.smithi204.stdout:2800: delete oid 21 current snap is 292 2024-09-16T05:20:58.922 INFO:tasks.rados.rados.2.smithi204.stdout:2791: expect (ObjNum 919 snap 273 seq_num 919) 2024-09-16T05:20:58.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:20:58.948 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:58.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:58.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:20:59.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:20:59.052 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:20:59.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:59.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:20:59.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:58 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:20:59.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:58 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:20:59.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:58 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/338014283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:20:59.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:58 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:20:59.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:58 smithi032 ceph-mon[27388]: 2.29 scrub ok 2024-09-16T05:20:59.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:58 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:20:59.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:58 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:20:59.162 INFO:tasks.rados.rados.2.smithi204.stdout:2786: finishing copy_from to smithi20457400-9 2024-09-16T05:20:59.162 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1252 (ObjNum 985 snap 289 seq_num 985) dirty exists 2024-09-16T05:20:59.162 INFO:tasks.rados.rados.2.smithi204.stdout:2792: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:20:59.162 INFO:tasks.rados.rados.2.smithi204.stdout:2790: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:20:59.162 INFO:tasks.rados.rados.2.smithi204.stdout:2786: done (10 left) 2024-09-16T05:20:59.162 INFO:tasks.rados.rados.2.smithi204.stdout:2791: done (9 left) 2024-09-16T05:20:59.163 INFO:tasks.rados.rados.2.smithi204.stdout:2800: done (8 left) 2024-09-16T05:20:59.163 INFO:tasks.rados.rados.2.smithi204.stdout:2801: snap_create 2024-09-16T05:20:59.163 INFO:tasks.rados.rados.2.smithi204.stdout:2793: finishing write tid 1 to smithi20457400-36 2024-09-16T05:20:59.163 INFO:tasks.rados.rados.2.smithi204.stdout:2792: finishing copy_from to smithi20457400-34 2024-09-16T05:20:59.163 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 781 (ObjNum 986 snap 292 seq_num 986) dirty exists 2024-09-16T05:20:59.163 INFO:tasks.rados.rados.2.smithi204.stdout:2790: finishing copy_from to smithi20457400-40 2024-09-16T05:20:59.164 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1047 (ObjNum 971 snap 287 seq_num 971) dirty exists 2024-09-16T05:20:59.164 INFO:tasks.rados.rados.2.smithi204.stdout:2793: finishing write tid 2 to smithi20457400-36 2024-09-16T05:20:59.164 INFO:tasks.rados.rados.2.smithi204.stdout:2793: finishing write tid 3 to smithi20457400-36 2024-09-16T05:20:59.164 INFO:tasks.rados.rados.2.smithi204.stdout:2793: finishing write tid 4 to smithi20457400-36 2024-09-16T05:20:59.164 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 768 (ObjNum 993 snap 292 seq_num 993) dirty exists 2024-09-16T05:20:59.164 INFO:tasks.rados.rados.2.smithi204.stdout:2793: left oid 36 (ObjNum 993 snap 292 seq_num 993) 2024-09-16T05:20:59.165 INFO:tasks.rados.rados.2.smithi204.stdout:2795: expect (ObjNum 730 snap 210 seq_num 730) 2024-09-16T05:20:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:58 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:20:59.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:58 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:20:59.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:58 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/338014283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:20:59.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:58 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:20:59.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:58 smithi204 ceph-mon[30933]: 2.29 scrub ok 2024-09-16T05:20:59.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:58 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:20:59.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:58 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:20:59.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:59.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:59.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:58 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:20:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:58 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:20:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:58 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/338014283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:20:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:58 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:20:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:58 smithi196 ceph-mon[30802]: 2.29 scrub ok 2024-09-16T05:20:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:58 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:20:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:58 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:20:59.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:59.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:59.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:59.365 INFO:tasks.rados.rados.2.smithi204.stdout:2794: expect (ObjNum 906 snap 267 seq_num 906) 2024-09-16T05:20:59.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:20:59.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:59.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:20:59.524 INFO:tasks.thrashosds.thrasher:j is {'epoch': 708, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:20:57.806279+0000', 'last_up_change': '2024-09-16T05:14:38.892129+0000', 'last_in_change': '2024-09-16T05:19:34.779035+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 36, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '687', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '685', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 86, 'pg_placement_num': 86, 'pg_placement_num_target': 86, 'pg_num_target': 86, 'pg_num_pending': 86, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '708', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '435', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 293, 'snap_epoch': 708, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7999999523162842, 'score_stable': 1.7999999523162842, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.659999966621399, 'raw_score_stable': 1.659999966621399, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 611, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 21, 'last_clean_end': 295, 'up_from': 329, 'up_thru': 703, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3973436581}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3973436581}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3973436581}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3973436581}]}, 'public_addr': '172.21.15.32:6819/3973436581', 'cluster_addr': '172.21.15.32:6821/3973436581', 'heartbeat_back_addr': '172.21.15.32:6825/3973436581', 'heartbeat_front_addr': '172.21.15.32:6823/3973436581', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 685, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 702, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 689, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 690, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:09.291721+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T05:20:17.125178+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:44.703412+0000', 'dead_epoch': 295}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:50.830187+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:32.848448+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:20.219477+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:13.190588+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:50.053405+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:40.278582+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:46.829682+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:30.643238+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:19:58.888327+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [2, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 242, 'length': 1}, {'begin': 267, 'length': 1}, {'begin': 275, 'length': 1}, {'begin': 287, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:20:59.524 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T05:20:59.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:20:59.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:59.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:20:59.636 INFO:tasks.rados.rados.2.smithi204.stdout:2796: finishing write tid 1 to smithi20457400-2 2024-09-16T05:20:59.636 INFO:tasks.rados.rados.2.smithi204.stdout:2796: finishing write tid 2 to smithi20457400-2 2024-09-16T05:20:59.636 INFO:tasks.rados.rados.2.smithi204.stdout:2796: finishing write tid 3 to smithi20457400-2 2024-09-16T05:20:59.637 INFO:tasks.rados.rados.2.smithi204.stdout:2796: finishing write tid 4 to smithi20457400-2 2024-09-16T05:20:59.637 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 525 (ObjNum 994 snap 292 seq_num 994) dirty exists 2024-09-16T05:20:59.637 INFO:tasks.rados.rados.2.smithi204.stdout:2796: left oid 2 (ObjNum 994 snap 292 seq_num 994) 2024-09-16T05:20:59.637 INFO:tasks.rados.rados.2.smithi204.stdout:2799: expect (ObjNum 977 snap 288 seq_num 977) 2024-09-16T05:20:59.689 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:20:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:59.712 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:20:59.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:20:59.796 INFO:tasks.rados.rados.2.smithi204.stdout:2798: finishing write tid 1 to smithi20457400-44 2024-09-16T05:20:59.796 INFO:tasks.rados.rados.2.smithi204.stdout:2798: finishing write tid 2 to smithi20457400-44 2024-09-16T05:20:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:2798: finishing write tid 3 to smithi20457400-44 2024-09-16T05:20:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:2798: finishing write tid 4 to smithi20457400-44 2024-09-16T05:20:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 731 (ObjNum 995 snap 292 seq_num 995) dirty exists 2024-09-16T05:20:59.797 INFO:tasks.rados.rados.2.smithi204.stdout:2798: left oid 44 (ObjNum 995 snap 292 seq_num 995) 2024-09-16T05:20:59.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:20:59.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:20:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:59.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2790: done (8 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2792: done (7 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2793: done (6 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2794: done (5 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2795: done (4 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2796: done (3 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2798: done (2 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2799: done (1 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2801: done (0 left) 2024-09-16T05:20:59.905 INFO:tasks.rados.rados.2.smithi204.stdout:2802: rollback oid 1 current snap is 293 2024-09-16T05:20:59.906 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 1 to 284 2024-09-16T05:20:59.906 INFO:tasks.rados.rados.2.smithi204.stdout:2803: write oid 8 current snap is 293 2024-09-16T05:20:59.906 INFO:tasks.rados.rados.2.smithi204.stdout:2803: seq_num 997 ranges {756579=665496,1958665=400347} 2024-09-16T05:20:59.912 INFO:tasks.rados.rados.2.smithi204.stdout:2803: writing smithi20457400-8 from 756579 to 1422075 tid 1 2024-09-16T05:20:59.918 INFO:tasks.rados.rados.2.smithi204.stdout:2803: writing smithi20457400-8 from 1958665 to 2359012 tid 2 2024-09-16T05:20:59.918 INFO:tasks.rados.rados.2.smithi204.stdout:2804: read oid 3 snap -1 2024-09-16T05:20:59.918 INFO:tasks.rados.rados.2.smithi204.stdout:2804: expect (ObjNum 919 snap 273 seq_num 919) 2024-09-16T05:20:59.918 INFO:tasks.rados.rados.2.smithi204.stdout:2805: copy_from oid 45 from oid 26 current snap is 293 2024-09-16T05:20:59.918 INFO:tasks.rados.rados.2.smithi204.stdout:2802: finishing rollback tid 0 to smithi20457400-1 2024-09-16T05:20:59.918 INFO:tasks.rados.rados.2.smithi204.stdout:2802: finishing rollback tid 1 to smithi20457400-1 2024-09-16T05:20:59.918 INFO:tasks.rados.rados.2.smithi204.stdout:2802: finishing rollback tid 2 to smithi20457400-1 2024-09-16T05:20:59.918 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 562 (ObjNum 936 snap 277 seq_num 936) dirty exists 2024-09-16T05:20:59.919 INFO:tasks.rados.rados.2.smithi204.stdout:2802: done (3 left) 2024-09-16T05:20:59.919 INFO:tasks.rados.rados.2.smithi204.stdout:2806: snap_create 2024-09-16T05:20:59.919 INFO:tasks.rados.rados.2.smithi204.stdout:2803: finishing write tid 1 to smithi20457400-8 2024-09-16T05:20:59.919 INFO:tasks.rados.rados.2.smithi204.stdout:2804: expect (ObjNum 919 snap 273 seq_num 919) 2024-09-16T05:20:59.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:20:59.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:20:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:59.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:00.005 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:00.006 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:00.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:59 smithi032 ceph-mon[27388]: pgmap v1224: 117 pgs: 117 active+clean; 276 MiB data, 889 MiB used, 983 GiB / 983 GiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2024-09-16T05:21:00.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:59 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:21:00.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:59 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:21:00.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:59 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:21:00.006 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:20:59 smithi032 ceph-mon[27388]: osdmap e709: 12 total, 12 up, 11 in 2024-09-16T05:21:00.027 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:00.028 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:00.048 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:00.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:00.068 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:00.069 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:00.085 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:00.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:00.090 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:00.117 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:00.118 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:00.139 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:00.140 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:00.161 INFO:tasks.rados.rados.2.smithi204.stdout:2803: finishing write tid 2 to smithi20457400-8 2024-09-16T05:21:00.161 INFO:tasks.rados.rados.2.smithi204.stdout:2803: finishing write tid 3 to smithi20457400-8 2024-09-16T05:21:00.161 INFO:tasks.rados.rados.2.smithi204.stdout:2803: finishing write tid 4 to smithi20457400-8 2024-09-16T05:21:00.161 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1255 (ObjNum 997 snap 293 seq_num 997) dirty exists 2024-09-16T05:21:00.161 INFO:tasks.rados.rados.2.smithi204.stdout:2803: left oid 8 (ObjNum 997 snap 293 seq_num 997) 2024-09-16T05:21:00.161 INFO:tasks.rados.rados.2.smithi204.stdout:2805: finishing copy_from racing read to smithi20457400-45 2024-09-16T05:21:00.161 INFO:tasks.rados.rados.2.smithi204.stdout:2805: finishing copy_from to smithi20457400-45 2024-09-16T05:21:00.161 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1252 (ObjNum 985 snap 289 seq_num 985) dirty exists 2024-09-16T05:21:00.162 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:00.162 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:59 smithi204 ceph-mon[30933]: pgmap v1224: 117 pgs: 117 active+clean; 276 MiB data, 889 MiB used, 983 GiB / 983 GiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2024-09-16T05:21:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:59 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:21:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:59 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:21:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:59 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:21:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:20:59 smithi204 ceph-mon[30933]: osdmap e709: 12 total, 12 up, 11 in 2024-09-16T05:21:00.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:00.183 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:00.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:00.204 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:00.204 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:00.207 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:00.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:00.225 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:00.226 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:00.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:59 smithi196 ceph-mon[30802]: pgmap v1224: 117 pgs: 117 active+clean; 276 MiB data, 889 MiB used, 983 GiB / 983 GiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2024-09-16T05:21:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:59 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:21:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:59 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:21:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:59 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:21:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:20:59 smithi196 ceph-mon[30802]: osdmap e709: 12 total, 12 up, 11 in 2024-09-16T05:21:00.247 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:00.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:00.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:00.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:00.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:00.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:00.457 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:00.458 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:00.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:00.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:00.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:00.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:00.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:00.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:00.727 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:21:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:21:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:21:00.727 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:21:00.727 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:21:00.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:21:00.809 INFO:tasks.rados.rados.2.smithi204.stdout:2803: done (3 left) 2024-09-16T05:21:00.809 INFO:tasks.rados.rados.2.smithi204.stdout:2804: done (2 left) 2024-09-16T05:21:00.809 INFO:tasks.rados.rados.2.smithi204.stdout:2805: done (1 left) 2024-09-16T05:21:00.809 INFO:tasks.rados.rados.2.smithi204.stdout:2806: done (0 left) 2024-09-16T05:21:00.809 INFO:tasks.rados.rados.2.smithi204.stdout:2807: snap_create 2024-09-16T05:21:00.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:00.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:00.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:00.975 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:00.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:01.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:01.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:01.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.50 scrub starts 2024-09-16T05:21:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.50 scrub ok 2024-09-16T05:21:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.51 scrub starts 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.51 scrub ok 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: osdmap e710: 12 total, 12 up, 11 in 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1688197923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:21:01.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:00 smithi032 ceph-mon[27388]: osdmap e711: 12 total, 12 up, 11 in 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.50 scrub starts 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.50 scrub ok 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.51 scrub starts 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.51 scrub ok 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: osdmap e710: 12 total, 12 up, 11 in 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1688197923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:21:01.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:21:01.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:21:01.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:00 smithi204 ceph-mon[30933]: osdmap e711: 12 total, 12 up, 11 in 2024-09-16T05:21:01.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:01.224 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:01.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:01.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.50 scrub starts 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.50 scrub ok 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.51 scrub starts 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.51 scrub ok 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: osdmap e710: 12 total, 12 up, 11 in 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1688197923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:21:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:00 smithi196 ceph-mon[30802]: osdmap e711: 12 total, 12 up, 11 in 2024-09-16T05:21:01.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:01.347 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:01.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:01.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:01.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:01.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:01.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:01.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:01.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:01.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:01.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:01.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:2807: done (0 left) 2024-09-16T05:21:01.811 INFO:tasks.rados.rados.2.smithi204.stdout:2808: snap_create 2024-09-16T05:21:01.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:21:01.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:01.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:01.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:01.866 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:01.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:01.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:01.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:01 smithi032 ceph-mon[27388]: pgmap v1227: 117 pgs: 117 active+clean; 264 MiB data, 882 MiB used, 983 GiB / 983 GiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-16T05:21:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:01 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:21:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:01 smithi032 ceph-mon[27388]: 2.26 scrub starts 2024-09-16T05:21:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:01 smithi032 ceph-mon[27388]: 2.26 scrub ok 2024-09-16T05:21:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:01 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:21:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:01 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:21:02.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:01 smithi032 ceph-mon[27388]: osdmap e712: 12 total, 12 up, 11 in 2024-09-16T05:21:02.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:02.099 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:02.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:01 smithi204 ceph-mon[30933]: pgmap v1227: 117 pgs: 117 active+clean; 264 MiB data, 882 MiB used, 983 GiB / 983 GiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-16T05:21:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:01 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:21:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:01 smithi204 ceph-mon[30933]: 2.26 scrub starts 2024-09-16T05:21:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:01 smithi204 ceph-mon[30933]: 2.26 scrub ok 2024-09-16T05:21:02.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:01 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:21:02.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:01 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:21:02.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:01 smithi204 ceph-mon[30933]: osdmap e712: 12 total, 12 up, 11 in 2024-09-16T05:21:02.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:02.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:02.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:02.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:01 smithi196 ceph-mon[30802]: pgmap v1227: 117 pgs: 117 active+clean; 264 MiB data, 882 MiB used, 983 GiB / 983 GiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-16T05:21:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:01 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:21:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:01 smithi196 ceph-mon[30802]: 2.26 scrub starts 2024-09-16T05:21:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:01 smithi196 ceph-mon[30802]: 2.26 scrub ok 2024-09-16T05:21:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:01 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:21:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:01 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:21:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:01 smithi196 ceph-mon[30802]: osdmap e712: 12 total, 12 up, 11 in 2024-09-16T05:21:02.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:02.351 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:02.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:02.387 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:02.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:02.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:02.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:02.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:02.601 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:02.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:02.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:02.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:02.805 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:21:02.809 INFO:tasks.rados.rados.2.smithi204.stdout:2808: done (0 left) 2024-09-16T05:21:02.809 INFO:tasks.rados.rados.2.smithi204.stdout:2809: write (excl) oid 24 current snap is 296 2024-09-16T05:21:02.809 INFO:tasks.rados.rados.2.smithi204.stdout:2809: seq_num 998 ranges {773386=740745,2017147=544005} 2024-09-16T05:21:02.817 INFO:tasks.rados.rados.2.smithi204.stdout:2809: writing smithi20457400-24 from 773386 to 1514131 tid 1 2024-09-16T05:21:02.823 INFO:tasks.rados.rados.2.smithi204.stdout:2809: writing smithi20457400-24 from 2017147 to 2561152 tid 2 2024-09-16T05:21:02.823 INFO:tasks.rados.rados.2.smithi204.stdout:2810: rollback oid 32 current snap is 296 2024-09-16T05:21:02.823 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 292 2024-09-16T05:21:02.823 INFO:tasks.rados.rados.2.smithi204.stdout:2811: snap_remove snap 288 2024-09-16T05:21:02.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:02.849 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:02.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:02.853 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:21:02.853 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:21:02.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:116 2.5 e141/107 2:ad4a88eb:::smithi20457400-16:a v 141'87, mlcod=141'79)", 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 590.15656246599997, 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006093261, 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695008+0000", 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695009+0000", 2024-09-16T05:21:02.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6499999999999999e-07 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.695014+0000", 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8350000000000003e-06 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700654+0000", 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056407499999999999 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700660+0000", 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4879999999999998e-06 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700700+0000", 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0583000000000001e-05 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.700714+0000", 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.322e-05 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701062+0000", 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00034833999999999999 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701101+0000", 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9480000000000001e-05 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.858 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.8.0:118 2.5 e141/107 2:ad7afe2b:::smithi20457400-38:d v 141'90, mlcod=141'85)", 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 590.14996814200003, 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0066456359999999999, 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701602+0000", 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701603+0000", 2024-09-16T05:21:02.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.701606+0000", 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3890000000000001e-06 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707421+0000", 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.005814601 2024-09-16T05:21:02.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707428+0000", 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4839999999999998e-06 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707497+0000", 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8441000000000004e-05 2024-09-16T05:21:02.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.707520+0000", 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3513999999999999e-05 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708185+0000", 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066482599999999998 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:12.708248+0000", 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2959000000000005e-05 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:614 2.5 e144/107 2:a1d8c6d9:::smithi20457400-3:head v 144'110, mlcod=144'106)", 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 587.20100237700001, 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054521470000000001, 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650568+0000", 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.650569+0000", 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7299999999999995e-07 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653581+0000", 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003012617 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653627+0000", 2024-09-16T05:21:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5649999999999998e-05 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653639+0000", 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2126e-05 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653715+0000", 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5485e-05 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.653748+0000", 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3373e-05 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.655964+0000", 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002215875 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:15.656020+0000", 2024-09-16T05:21:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6348000000000003e-05 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:172 2.c e179/107 2:3d4bedc8:::smithi20457400-22:26 v 179'101, mlcod=178'97)", 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 552.198844979, 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34174038899999998, 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.870 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652726+0000", 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4999999999999999e-07 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.652730+0000", 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8789999999999996e-06 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993974+0000", 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34124383699999999 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.993980+0000", 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4139999999999999e-06 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994036+0000", 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5850999999999998e-05 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994054+0000", 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8182999999999999e-05 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994414+0000", 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036047800000000001 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.994466+0000", 2024-09-16T05:21:02.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1496999999999998e-05 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:972 2.c e179/107 2:33100188:::smithi20457400-19:head v 179'105, mlcod=179'101)", 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 551.85250107800005, 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.53639116399999998, 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.875 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999069+0000", 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999070+0000", 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0999999999999997e-07 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:50.999087+0000", 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7014999999999999e-05 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534916+0000", 2024-09-16T05:21:02.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.53582909000000001 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534923+0000", 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0099999999999998e-06 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534975+0000", 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1938999999999999e-05 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.534994+0000", 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8504000000000001e-05 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.535427+0000", 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043298700000000002 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.535461+0000", 2024-09-16T05:21:02.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3809000000000001e-05 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:979 2.4 e179/107 2:2ace617f:::smithi20457400-21:head v 179'81, mlcod=178'78)", 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 551.84470072199997, 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076005070000000003, 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.880 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.881 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.006870+0000", 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009810+0000", 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002939929 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009892+0000", 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2349999999999996e-05 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009916+0000", 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3200000000000001e-05 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.009983+0000", 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7577999999999996e-05 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.010005+0000", 2024-09-16T05:21:02.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2337999999999999e-05 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014425+0000", 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044196119999999998 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:51.014470+0000", 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5297999999999999e-05 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1017 2.5 e182/107 2:a1d8c6d9:::smithi20457400-3:head v 182'254, mlcod=182'250)", 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 549.15284607399997, 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0067341900000000001, 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:21:02.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698724+0000", 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.698726+0000", 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0410000000000001e-06 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703134+0000", 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044088850000000004 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703168+0000", 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3195999999999999e-05 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703175+0000", 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6939999999999995e-06 2024-09-16T05:21:02.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703234+0000", 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8859e-05 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.703262+0000", 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7650000000000001e-05 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.705387+0000", 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021252670000000001 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:11:53.705459+0000", 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1598000000000001e-05 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1146 2.c e192/107 2:33100188:::smithi20457400-19:head v 192'151, mlcod=192'147)", 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 539.10861123799998, 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057288909999999998, 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742959+0000", 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.742960+0000", 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3399999999999999e-07 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746015+0000", 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030547809999999999 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746040+0000", 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5075999999999998e-05 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746048+0000", 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9030000000000004e-06 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746099+0000", 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.1004999999999997e-05 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.746117+0000", 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8189000000000002e-05 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.748630+0000", 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025129549999999999 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.748688+0000", 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8448e-05 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.894 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1161 2.c e192/107 2:3d4bedc8:::smithi20457400-22:head v 192'153, mlcod=192'149)", 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 539.08012569000005, 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055212639999999997, 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771445+0000", 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.771447+0000", 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3360000000000002e-06 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774126+0000", 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026786790000000002 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774176+0000", 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9669999999999997e-05 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774185+0000", 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4530000000000008e-06 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774260+0000", 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5395999999999994e-05 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.774291+0000", 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0561000000000001e-05 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.776739+0000", 2024-09-16T05:21:02.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002447968 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:03.776966+0000", 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022720100000000001 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1201 2.5 e194/183 2:a5bc1608:::smithi20457400-11:head v 194'298, mlcod=192'295)", 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 537.17262772200002, 2024-09-16T05:21:02.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080910559999999992, 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.678943+0000", 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683786+0000", 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048424469999999997 2024-09-16T05:21:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683833+0000", 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7540999999999998e-05 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683844+0000", 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1354e-05 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683920+0000", 2024-09-16T05:21:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5760999999999995e-05 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.683940+0000", 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9715999999999999e-05 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.686996+0000", 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003055954 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:05.687034+0000", 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8031999999999998e-05 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1890 2.1f 2:fc733d2f:::smithi20457400-45:head [append 0~0] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 473.14088276699999, 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0080315490000000007, 2024-09-16T05:21:02.904 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1890 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710688+0000", 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710691+0000", 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7769999999999998e-06 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710704+0000", 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.38e-05 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710741+0000", 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6371999999999997e-05 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710758+0000", 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7637000000000001e-05 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710929+0000", 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017019499999999999 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710935+0000", 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2720000000000003e-06 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716763+0000", 2024-09-16T05:21:02.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058286350000000004 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716836+0000", 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2630000000000001e-05 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.716981+0000", 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014495099999999999 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718424+0000", 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014427769999999999 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718586+0000", 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016167699999999999 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718660+0000", 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4813000000000004e-05 2024-09-16T05:21:02.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718719+0000", 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9012999999999999e-05 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1891 2.1f 2:fc733d2f:::smithi20457400-45:head [rollback 39] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 473.14077082199998, 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082173000000000003, 2024-09-16T05:21:02.911 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1891 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710800+0000", 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710801+0000", 2024-09-16T05:21:02.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6700000000000002e-07 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710809+0000", 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4230000000000005e-06 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710823+0000", 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3833e-05 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710832+0000", 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9830000000000002e-06 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710997+0000", 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000165558 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711001+0000", 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0239999999999999e-06 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717531+0000", 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0065290699999999997 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717588+0000", 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7497999999999997e-05 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.717722+0000", 2024-09-16T05:21:02.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134182 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718427+0000", 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070501199999999998 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718936+0000", 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050843700000000004 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.917 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718972+0000", 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6102000000000001e-05 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719017+0000", 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5311e-05 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.918 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:1892 2.1f 2:fc733d2f:::smithi20457400-45:head [append 0~0] snapc 51=[51,50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 473.140644436, 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0082344800000000006, 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:02.919 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 1892 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710926+0000", 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710927+0000", 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9099999999999999e-07 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710930+0000", 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9869999999999999e-06 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710937+0000", 2024-09-16T05:21:02.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9940000000000003e-06 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.710941+0000", 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0099999999999997e-06 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711036+0000", 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5013999999999995e-05 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.711042+0000", 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.224e-06 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718122+0000", 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007080706 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718183+0000", 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0194e-05 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718285+0000", 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000102118 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.718538+0000", 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00025345499999999998 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719083+0000", 2024-09-16T05:21:02.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00054496199999999996 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719116+0000", 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3052000000000001e-05 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:09.719161+0000", 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4372999999999997e-05 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2010 2.1f 2:fc733d2f:::smithi20457400-45:head [copy-from ver 319] snapc 54=[54,53,52,4d,4c,49,48,47,45,3e,3a,35,27] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 462.96738075299999, 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016119959, 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2010 2024-09-16T05:21:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884190+0000", 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.927 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884192+0000", 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7069999999999999e-06 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884200+0000", 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0679999999999997e-06 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884217+0000", 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.772e-05 2024-09-16T05:21:02.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884228+0000", 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0455000000000001e-05 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884292+0000", 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4424999999999994e-05 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884357+0000", 2024-09-16T05:21:02.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4498999999999994e-05 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 7", 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.893200+0000", 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088430690000000003 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.896560+0000", 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0033602580000000001 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:02.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900238+0000", 2024-09-16T05:21:02.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036775699999999998 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900276+0000", 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8906999999999999e-05 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.900310+0000", 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3281000000000002e-05 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2011 2.1f 2:fc733d2f:::smithi20457400-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e268)", 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 462.967318675, 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011802536000000001, 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2011 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884252+0000", 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884253+0000", 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2200000000000004e-07 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884265+0000", 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2048999999999999e-05 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884275+0000", 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9899999999999992e-06 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884281+0000", 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6850000000000004e-06 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884503+0000", 2024-09-16T05:21:02.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022159799999999999 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.884535+0000", 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2288e-05 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.895921+0000", 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011386165 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.937 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.895982+0000", 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0906000000000003e-05 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:19.896054+0000", 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2232999999999998e-05 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDPGPush(2.31 277/276 [PushOp(2:8c1ba486:::smithi20457400-2:45, version: 259'110, data_included: [480733~537415,1782380~511161], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:8c1ba486:::smithi20457400-2:45@259'110, size: 3680733, copy_subset: [(0, 3680733)], clone_subset: {2:8c1ba486:::smithi20457400-2:2c: []}, snapset: 4c=[]:{2c: [27], 45: [45, 3e], 4c: [4c, 49, 48]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2293541, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T05:21:02.938 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 454.146643031, 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45568901499999998, 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704928+0000", 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.704929+0000", 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1000000000000001e-06 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:28.705928+0000", 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0009990349999999999 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.159934+0000", 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.45400632299999999 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.159949+0000", 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4608e-05 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160041+0000", 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2523999999999997e-05 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160053+0000", 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1525999999999999e-05 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:13:29.160617+0000", 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00056389899999999998 2024-09-16T05:21:02.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2536 2.31 2:8c1ba486:::smithi20457400-2:head [write 551362~726078] snapc 6d=[6d,6c,6b,6a,69,67,66,64,63,62,60,5d,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 379.91038627400002, 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055342209999999998, 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:21:02.943 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2536 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:21:02.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941188+0000", 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3469999999999999e-06 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942118+0000", 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092990300000000004 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942150+0000", 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2888000000000001e-05 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942164+0000", 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3920000000000001e-05 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942258+0000", 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3839999999999996e-05 2024-09-16T05:21:02.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942266+0000", 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4139999999999997e-06 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.943927+0000", 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016610450000000001 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.944033+0000", 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105979 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.944207+0000", 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000174105 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.945786+0000", 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001579352 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946595+0000", 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080861300000000002 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946660+0000", 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4906000000000005e-05 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946719+0000", 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8909000000000001e-05 2024-09-16T05:21:02.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:569 2.c e357/247 2:33100188:::smithi20457400-19:6a v 357'229, mlcod=351'225)", 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 356.14595695100002, 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14687246600000001, 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:02.950 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:21:02.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9599999999999999e-07 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705621+0000", 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7150000000000001e-06 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851933+0000", 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14631169599999999 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851940+0000", 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.221e-06 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851995+0000", 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5473999999999998e-05 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852016+0000", 2024-09-16T05:21:02.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0965999999999999e-05 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852452+0000", 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043602700000000002 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852486+0000", 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3871000000000001e-05 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.31,GRANT,e:542,from:10,reservation_nonce:0)", 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 169.83532772800001, 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.69991632400000003, 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:21:02.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016244+0000", 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7400000000000002e-07 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016249+0000", 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0259999999999998e-06 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.715937+0000", 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.69968885199999997 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.715950+0000", 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914999999999999e-05 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716018+0000", 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7972000000000006e-05 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716036+0000", 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7302000000000001e-05 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716159+0000", 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000123483 2024-09-16T05:21:02.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.47,GRANT,e:712,from:9,reservation_nonce:0)", 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:01.975022+0000", 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.87654904600000005, 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00022020899999999999, 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:02.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975022+0000", 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975022+0000", 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975023+0000", 2024-09-16T05:21:02.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1599999999999999e-06 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975027+0000", 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5900000000000001e-06 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975054+0000", 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7175999999999999e-05 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:02.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975064+0000", 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2949999999999999e-06 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975113+0000", 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8860000000000003e-05 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975128+0000", 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5557999999999999e-05 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:02.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:02.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:01.975242+0000", 2024-09-16T05:21:02.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011357 2024-09-16T05:21:02.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.963 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:02.963 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:02.963 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:21:02.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:02.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:02.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:03.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:03.114 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:03.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:21:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 2.44 scrub ok 2024-09-16T05:21:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2024603412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2024603412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:02 smithi032 ceph-mon[27388]: osdmap e713: 12 total, 12 up, 11 in 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 2.44 scrub ok 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:21:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:21:03.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:21:03.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:21:03.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2024603412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:03.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:03.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:21:03.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2024603412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:03.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:02 smithi204 ceph-mon[30933]: osdmap e713: 12 total, 12 up, 11 in 2024-09-16T05:21:03.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:21:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 2.44 scrub ok 2024-09-16T05:21:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2024603412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2024603412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:03.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:02 smithi196 ceph-mon[30802]: osdmap e713: 12 total, 12 up, 11 in 2024-09-16T05:21:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:03.241 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:03.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:03.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:03.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:03.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:03.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:21:03.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:03.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:03.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:03.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:03.617 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:03.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:03.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:03.744 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:03.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:03.810 INFO:tasks.rados.rados.2.smithi204.stdout:2811: done (2 left) 2024-09-16T05:21:03.810 INFO:tasks.rados.rados.2.smithi204.stdout:2812: rollback oid 50 current snap is 296 2024-09-16T05:21:03.810 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 294 2024-09-16T05:21:03.810 INFO:tasks.rados.rados.2.smithi204.stdout:2813: delete oid 42 current snap is 296 2024-09-16T05:21:03.810 INFO:tasks.rados.rados.2.smithi204.stdout:2809: finishing write tid 1 to smithi20457400-24 2024-09-16T05:21:03.810 INFO:tasks.rados.rados.2.smithi204.stdout:2809: finishing write tid 2 to smithi20457400-24 2024-09-16T05:21:03.810 INFO:tasks.rados.rados.2.smithi204.stdout:2809: finishing write tid 3 to smithi20457400-24 2024-09-16T05:21:03.811 INFO:tasks.rados.rados.2.smithi204.stdout:2810: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:21:03.811 INFO:tasks.rados.rados.2.smithi204.stdout:2809: finishing write tid 4 to smithi20457400-24 2024-09-16T05:21:03.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 782 (ObjNum 998 snap 296 seq_num 998) dirty exists 2024-09-16T05:21:03.811 INFO:tasks.rados.rados.2.smithi204.stdout:2809: left oid 24 (ObjNum 998 snap 296 seq_num 998) 2024-09-16T05:21:03.811 INFO:tasks.rados.rados.2.smithi204.stdout:2810: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:21:03.811 INFO:tasks.rados.rados.2.smithi204.stdout:2810: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:21:03.811 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1089 (ObjNum 815 snap 237 seq_num 815) dirty exists 2024-09-16T05:21:03.816 INFO:tasks.rados.rados.2.smithi204.stdout:2812: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:21:03.816 INFO:tasks.rados.rados.2.smithi204.stdout:2812: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:21:03.816 INFO:tasks.rados.rados.2.smithi204.stdout:2812: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:21:03.817 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1115 (ObjNum 912 snap 271 seq_num 912) dirty exists 2024-09-16T05:21:03.819 INFO:tasks.rados.rados.2.smithi204.stdout:2809: done (3 left) 2024-09-16T05:21:03.819 INFO:tasks.rados.rados.2.smithi204.stdout:2810: done (2 left) 2024-09-16T05:21:03.819 INFO:tasks.rados.rados.2.smithi204.stdout:2812: done (1 left) 2024-09-16T05:21:03.819 INFO:tasks.rados.rados.2.smithi204.stdout:2813: done (0 left) 2024-09-16T05:21:03.819 INFO:tasks.rados.rados.2.smithi204.stdout:2814: snap_remove snap 281 2024-09-16T05:21:03.845 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:03.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:03.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:03.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:03.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:03.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:03.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:04.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:04.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:04.118 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:04.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:03 smithi032 ceph-mon[27388]: pgmap v1230: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 275 MiB data, 911 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-16T05:21:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:03 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:21:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:03 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:21:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:03 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:21:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:03 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:21:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:03 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:21:04.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:03 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:21:04.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:03 smithi032 ceph-mon[27388]: osdmap e714: 12 total, 12 up, 11 in 2024-09-16T05:21:04.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:04.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:03 smithi196 ceph-mon[30802]: pgmap v1230: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 275 MiB data, 911 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-16T05:21:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:03 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:21:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:03 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:21:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:03 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:21:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:03 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:21:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:03 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:21:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:03 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:21:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:03 smithi196 ceph-mon[30802]: osdmap e714: 12 total, 12 up, 11 in 2024-09-16T05:21:04.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:04.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:04.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:04.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:04.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:03 smithi204 ceph-mon[30933]: pgmap v1230: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 275 MiB data, 911 MiB used, 983 GiB / 983 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-16T05:21:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:03 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:21:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:03 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:21:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:03 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:21:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:03 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:21:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:03 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:21:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:03 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:21:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:03 smithi204 ceph-mon[30933]: osdmap e714: 12 total, 12 up, 11 in 2024-09-16T05:21:04.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:04.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:21:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:04.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:04.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:04.522 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:04.525 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:21:04.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:21:04.526 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-16T05:21:04.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-16T05:21:04.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:04.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:04.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:04.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:04.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:04.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:04.812 INFO:tasks.rados.rados.2.smithi204.stdout:2814: done (0 left) 2024-09-16T05:21:04.813 INFO:tasks.rados.rados.2.smithi204.stdout:2815: delete oid 40 current snap is 296 2024-09-16T05:21:04.817 INFO:tasks.rados.rados.2.smithi204.stdout:2815: done (0 left) 2024-09-16T05:21:04.817 INFO:tasks.rados.rados.2.smithi204.stdout:2816: write (excl) oid 16 current snap is 296 2024-09-16T05:21:04.817 INFO:tasks.rados.rados.2.smithi204.stdout:2816: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-09-16T05:21:04.823 INFO:tasks.rados.rados.2.smithi204.stdout:2816: writing smithi20457400-16 from 423807 to 990299 tid 1 2024-09-16T05:21:04.829 INFO:tasks.rados.rados.2.smithi204.stdout:2816: writing smithi20457400-16 from 1676240 to 2083925 tid 2 2024-09-16T05:21:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2816: writing smithi20457400-16 from 2739004 to 2823807 tid 3 2024-09-16T05:21:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2817: read oid 3 snap -1 2024-09-16T05:21:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2817: expect (ObjNum 919 snap 273 seq_num 919) 2024-09-16T05:21:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2818: snap_create 2024-09-16T05:21:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2816: finishing write tid 1 to smithi20457400-16 2024-09-16T05:21:04.834 INFO:tasks.rados.rados.2.smithi204.stdout:2816: finishing write tid 2 to smithi20457400-16 2024-09-16T05:21:04.836 INFO:tasks.rados.rados.2.smithi204.stdout:2816: finishing write tid 3 to smithi20457400-16 2024-09-16T05:21:04.836 INFO:tasks.rados.rados.2.smithi204.stdout:2816: finishing write tid 4 to smithi20457400-16 2024-09-16T05:21:04.836 INFO:tasks.rados.rados.2.smithi204.stdout:2816: finishing write tid 5 to smithi20457400-16 2024-09-16T05:21:04.836 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1025 (ObjNum 1001 snap 296 seq_num 1001) dirty exists 2024-09-16T05:21:04.836 INFO:tasks.rados.rados.2.smithi204.stdout:2816: left oid 16 (ObjNum 1001 snap 296 seq_num 1001) 2024-09-16T05:21:04.838 INFO:tasks.rados.rados.2.smithi204.stdout:2817: expect (ObjNum 919 snap 273 seq_num 919) 2024-09-16T05:21:04.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:04.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:04.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:04.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:04.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:05.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:05.083 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:05.083 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:04 smithi032 ceph-mon[27388]: 2.2a deep-scrub starts 2024-09-16T05:21:05.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:04 smithi032 ceph-mon[27388]: 2.2a deep-scrub ok 2024-09-16T05:21:05.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:04 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:21:05.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:04 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:21:05.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:04 smithi032 ceph-mon[27388]: osdmap e715: 12 total, 12 up, 11 in 2024-09-16T05:21:05.094 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:05.095 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:05.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:05.106 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:05.107 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:05.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:05.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:05.134 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:05.135 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:05.155 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:05.177 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:05.177 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:05.199 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:05.199 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:05.221 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:05.221 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:05.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:05.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:04 smithi196 ceph-mon[30802]: 2.2a deep-scrub starts 2024-09-16T05:21:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:04 smithi196 ceph-mon[30802]: 2.2a deep-scrub ok 2024-09-16T05:21:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:04 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:21:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:04 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:21:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:04 smithi196 ceph-mon[30802]: osdmap e715: 12 total, 12 up, 11 in 2024-09-16T05:21:05.243 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:05.243 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:05.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:05.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:05.265 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:05.265 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:05.286 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:05.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:05.307 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:05.328 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:05.328 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:05.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:05.380 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:05.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:05.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:04 smithi204 ceph-mon[30933]: 2.2a deep-scrub starts 2024-09-16T05:21:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:04 smithi204 ceph-mon[30933]: 2.2a deep-scrub ok 2024-09-16T05:21:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:04 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:21:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:04 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:21:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:04 smithi204 ceph-mon[30933]: osdmap e715: 12 total, 12 up, 11 in 2024-09-16T05:21:05.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:05.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:05.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:05.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:05.629 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:05.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:05.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:05.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:05.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:05.807 INFO:tasks.rados.rados.2.smithi204.stdout:2816: done (2 left) 2024-09-16T05:21:05.807 INFO:tasks.rados.rados.2.smithi204.stdout:2817: done (1 left) 2024-09-16T05:21:05.807 INFO:tasks.rados.rados.2.smithi204.stdout:2818: done (0 left) 2024-09-16T05:21:05.807 INFO:tasks.rados.rados.2.smithi204.stdout:2819: snap_create 2024-09-16T05:21:05.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:05.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:05.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:05.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:05.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:21:05.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:05 smithi032 ceph-mon[27388]: pgmap v1233: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 288 MiB data, 949 MiB used, 982 GiB / 983 GiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 21 op/s 2024-09-16T05:21:05.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3110001845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:05.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:05 smithi032 ceph-mon[27388]: 2.3d scrub starts 2024-09-16T05:21:05.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:05 smithi032 ceph-mon[27388]: 2.3d scrub ok 2024-09-16T05:21:05.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:05 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:21:05.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:05 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:21:05.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:05 smithi032 ceph-mon[27388]: osdmap e716: 12 total, 12 up, 11 in 2024-09-16T05:21:05.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:06.002 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:06.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:06.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:06.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:06.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:06.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:06.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:05 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:21:05.977+0000 7f7c3f992640 -1 mon.a@0(leader).osd e716 definitely_dead 0 2024-09-16T05:21:06.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:06.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:05 smithi196 ceph-mon[30802]: pgmap v1233: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 288 MiB data, 949 MiB used, 982 GiB / 983 GiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 21 op/s 2024-09-16T05:21:06.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3110001845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:05 smithi196 ceph-mon[30802]: 2.3d scrub starts 2024-09-16T05:21:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:05 smithi196 ceph-mon[30802]: 2.3d scrub ok 2024-09-16T05:21:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:05 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:21:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:05 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:21:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:05 smithi196 ceph-mon[30802]: osdmap e716: 12 total, 12 up, 11 in 2024-09-16T05:21:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:06.254 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:06.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:06.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:06.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:06.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:06.408 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:06.408 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:06.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:05 smithi204 ceph-mon[30933]: pgmap v1233: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 288 MiB data, 949 MiB used, 982 GiB / 983 GiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 21 op/s 2024-09-16T05:21:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3110001845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:05 smithi204 ceph-mon[30933]: 2.3d scrub starts 2024-09-16T05:21:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:05 smithi204 ceph-mon[30933]: 2.3d scrub ok 2024-09-16T05:21:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:05 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:21:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:05 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:21:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:05 smithi204 ceph-mon[30933]: osdmap e716: 12 total, 12 up, 11 in 2024-09-16T05:21:06.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:06.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:06.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:06.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:06.633 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:06.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:06.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:06.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:06.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:06.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:06.805 INFO:teuthology.orchestra.run.smithi032.stderr:marked in osd.1. 2024-09-16T05:21:06.810 INFO:tasks.rados.rados.2.smithi204.stdout:2819: done (0 left) 2024-09-16T05:21:06.810 INFO:tasks.rados.rados.2.smithi204.stdout:2820: delete oid 30 current snap is 298 2024-09-16T05:21:06.814 INFO:tasks.rados.rados.2.smithi204.stdout:2820: done (0 left) 2024-09-16T05:21:06.814 INFO:tasks.rados.rados.2.smithi204.stdout:2821: snap_remove snap 287 2024-09-16T05:21:06.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:06.886 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:06.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:06.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:07.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:07.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:07.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:07.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:07.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:06 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:21:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:06 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:21:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3885884920' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-16T05:21:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3885884920' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-16T05:21:07.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:06 smithi032 ceph-mon[27388]: osdmap e717: 12 total, 12 up, 12 in 2024-09-16T05:21:07.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:06 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:21:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:06 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:21:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3885884920' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-16T05:21:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3885884920' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-16T05:21:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:06 smithi196 ceph-mon[30802]: osdmap e717: 12 total, 12 up, 12 in 2024-09-16T05:21:07.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:07.266 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:07.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:07.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:07.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:07.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:07.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:06 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:21:07.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:06 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:21:07.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3885884920' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-16T05:21:07.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3885884920' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-16T05:21:07.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:06 smithi204 ceph-mon[30933]: osdmap e717: 12 total, 12 up, 12 in 2024-09-16T05:21:07.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:07.523 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:07.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:07.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:07.650 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:07.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:07.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:07.778 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:07.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:07.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:21:07.812 INFO:tasks.rados.rados.2.smithi204.stdout:2821: done (0 left) 2024-09-16T05:21:07.812 INFO:tasks.rados.rados.2.smithi204.stdout:2822: snap_remove snap 283 2024-09-16T05:21:07.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:07.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:07.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:08.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:08.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:08.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:08.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:08.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:21:08.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:21:08.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: pgmap v1236: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 288 MiB data, 952 MiB used, 982 GiB / 983 GiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-16T05:21:08.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: 2.2e scrub starts 2024-09-16T05:21:08.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: 2.2e scrub ok 2024-09-16T05:21:08.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: 2.4e scrub starts 2024-09-16T05:21:08.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: 2.4e scrub ok 2024-09-16T05:21:08.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:08.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:08.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:07 smithi032 ceph-mon[27388]: osdmap e718: 12 total, 12 up, 12 in 2024-09-16T05:21:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:08.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:08.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:08.169 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-16T05:21:08.181 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:08.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: pgmap v1236: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 288 MiB data, 952 MiB used, 982 GiB / 983 GiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: 2.2e scrub starts 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: 2.2e scrub ok 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: 2.4e scrub starts 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: 2.4e scrub ok 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:07 smithi196 ceph-mon[30802]: osdmap e718: 12 total, 12 up, 12 in 2024-09-16T05:21:08.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:08.281 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:08.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:08.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:08.407 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:08.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:08.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: pgmap v1236: 117 pgs: 1 active+clean+scrubbing, 116 active+clean; 288 MiB data, 952 MiB used, 982 GiB / 983 GiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: 2.2e scrub starts 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: 2.2e scrub ok 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: 2.4e scrub starts 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: 2.4e scrub ok 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:07 smithi204 ceph-mon[30933]: osdmap e718: 12 total, 12 up, 12 in 2024-09-16T05:21:08.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:08.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:08.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:08.616 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:08.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:08.651 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:08.651 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:08.662 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:08.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:08.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:08.789 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:08.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2822: done (0 left) 2024-09-16T05:21:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2823: read oid 7 snap -1 2024-09-16T05:21:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2823: expect (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:21:08.812 INFO:tasks.rados.rados.2.smithi204.stdout:2824: snap_create 2024-09-16T05:21:08.830 INFO:tasks.rados.rados.2.smithi204.stdout:2823: expect (ObjNum 789 snap 229 seq_num 789) 2024-09-16T05:21:08.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:08.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:08.913 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:08.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:09.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:09.039 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:09.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:09.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:09.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:09.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:21:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:21:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:21:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:21:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: 2.44 deep-scrub starts 2024-09-16T05:21:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: 2.44 deep-scrub ok 2024-09-16T05:21:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: 2.6 deep-scrub starts 2024-09-16T05:21:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: 2.6 deep-scrub ok 2024-09-16T05:21:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:08 smithi032 ceph-mon[27388]: osdmap e719: 12 total, 12 up, 12 in 2024-09-16T05:21:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:21:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:21:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:21:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:21:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: 2.44 deep-scrub starts 2024-09-16T05:21:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: 2.44 deep-scrub ok 2024-09-16T05:21:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: 2.6 deep-scrub starts 2024-09-16T05:21:09.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: 2.6 deep-scrub ok 2024-09-16T05:21:09.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:08 smithi196 ceph-mon[30802]: osdmap e719: 12 total, 12 up, 12 in 2024-09-16T05:21:09.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:09.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:09.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:09.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:09.415 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:09.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:09.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:09.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:21:09.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:21:09.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:21:09.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:21:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: 2.44 deep-scrub starts 2024-09-16T05:21:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: 2.44 deep-scrub ok 2024-09-16T05:21:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: 2.6 deep-scrub starts 2024-09-16T05:21:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: 2.6 deep-scrub ok 2024-09-16T05:21:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:08 smithi204 ceph-mon[30933]: osdmap e719: 12 total, 12 up, 12 in 2024-09-16T05:21:09.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:21:09.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:09.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:09.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:09.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:09.663 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:09.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:09.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:09.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:09.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:09.809 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:21:09.812 INFO:tasks.rados.rados.2.smithi204.stdout:2823: done (1 left) 2024-09-16T05:21:09.813 INFO:tasks.rados.rados.2.smithi204.stdout:2824: done (0 left) 2024-09-16T05:21:09.813 INFO:tasks.rados.rados.2.smithi204.stdout:2825: snap_create 2024-09-16T05:21:09.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:09.909 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:09.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:09.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:21:10.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:10.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:10.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:10.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:10.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:10.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: pgmap v1239: 117 pgs: 4 remapped+peering, 2 peering, 111 active+clean; 288 MiB data, 957 MiB used, 982 GiB / 983 GiB avail; 255 B/s rd, 314 KiB/s wr, 3 op/s 2024-09-16T05:21:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-16T05:21:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:21:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:21:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2969339496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:21:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:21:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2969339496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:09 smithi032 ceph-mon[27388]: osdmap e720: 12 total, 12 up, 12 in 2024-09-16T05:21:10.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: pgmap v1239: 117 pgs: 4 remapped+peering, 2 peering, 111 active+clean; 288 MiB data, 957 MiB used, 982 GiB / 983 GiB avail; 255 B/s rd, 314 KiB/s wr, 3 op/s 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2969339496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2969339496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:09 smithi196 ceph-mon[30802]: osdmap e720: 12 total, 12 up, 12 in 2024-09-16T05:21:10.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:10.284 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:10.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: pgmap v1239: 117 pgs: 4 remapped+peering, 2 peering, 111 active+clean; 288 MiB data, 957 MiB used, 982 GiB / 983 GiB avail; 255 B/s rd, 314 KiB/s wr, 3 op/s 2024-09-16T05:21:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-16T05:21:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:21:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:21:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2969339496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:21:10.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:21:10.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:10.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2969339496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:10.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:09 smithi204 ceph-mon[30933]: osdmap e720: 12 total, 12 up, 12 in 2024-09-16T05:21:10.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:10.409 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:10.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:10.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:10.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:10.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:10.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:10.664 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:10.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:10.737 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:10.752 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:10.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:10.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:10.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:2825: done (0 left) 2024-09-16T05:21:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:2826: write (excl) oid 50 current snap is 300 2024-09-16T05:21:10.814 INFO:tasks.rados.rados.2.smithi204.stdout:2826: seq_num 1003 ranges {457421=716990,1676851=411354,2677497=179924} 2024-09-16T05:21:10.821 INFO:tasks.rados.rados.2.smithi204.stdout:2826: writing smithi20457400-50 from 457421 to 1174411 tid 1 2024-09-16T05:21:10.826 INFO:tasks.rados.rados.2.smithi204.stdout:2826: writing smithi20457400-50 from 1676851 to 2088205 tid 2 2024-09-16T05:21:10.833 INFO:tasks.rados.rados.2.smithi204.stdout:2826: writing smithi20457400-50 from 2677497 to 2857421 tid 3 2024-09-16T05:21:10.833 INFO:tasks.rados.rados.2.smithi204.stdout:2827: snap_remove snap 293 2024-09-16T05:21:10.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:10.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:10.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:11.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:11.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:11.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:11.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:11.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:11.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:11.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:11.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:11.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:11.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:11.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:11.423 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:11.423 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:11.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:11.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:11.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:11.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:11.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:11.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:11.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:11.795 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:11.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:11.829 INFO:tasks.rados.rados.2.smithi204.stdout:2827: done (1 left) 2024-09-16T05:21:11.829 INFO:tasks.rados.rados.2.smithi204.stdout:2828: snap_remove snap 284 2024-09-16T05:21:11.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:11.920 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:11.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:11.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:11 smithi204 ceph-mon[30933]: pgmap v1242: 117 pgs: 9 peering, 108 active+clean; 303 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 7.8 MiB/s, 16 objects/s recovering 2024-09-16T05:21:11.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:11 smithi204 ceph-mon[30933]: osdmap e721: 12 total, 12 up, 12 in 2024-09-16T05:21:11.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:11 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-16T05:21:12.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:12.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:12.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:12.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:11 smithi032 ceph-mon[27388]: pgmap v1242: 117 pgs: 9 peering, 108 active+clean; 303 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 7.8 MiB/s, 16 objects/s recovering 2024-09-16T05:21:12.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:11 smithi032 ceph-mon[27388]: osdmap e721: 12 total, 12 up, 12 in 2024-09-16T05:21:12.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:11 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-16T05:21:12.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:21:12.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:12.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:12.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:11 smithi196 ceph-mon[30802]: pgmap v1242: 117 pgs: 9 peering, 108 active+clean; 303 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 7.8 MiB/s, 16 objects/s recovering 2024-09-16T05:21:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:11 smithi196 ceph-mon[30802]: osdmap e721: 12 total, 12 up, 12 in 2024-09-16T05:21:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:11 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-09-16T05:21:12.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:12.300 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:12.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:12.380 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:12.380 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:12.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:12.402 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:12.403 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:12.425 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:12.426 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:12.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:12.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:12.453 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:12.453 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:12.460 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:12.475 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:12.498 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:12.520 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:12.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:12.542 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:12.543 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:12.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:12.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:12.564 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:12.588 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:12.588 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:12.609 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:12.609 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:12.630 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:12.631 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:12.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:12.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:12.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:21:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:12.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:12.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:12.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:12.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:12.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:12.836 INFO:tasks.rados.rados.2.smithi204.stdout:2828: done (1 left) 2024-09-16T05:21:12.836 INFO:tasks.rados.rados.2.smithi204.stdout:2829: read oid 19 snap 292 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2829: expect (ObjNum 929 snap 275 seq_num 929) 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2830: read oid 28 snap -1 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2830: expect (ObjNum 971 snap 287 seq_num 971) 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2831: copy_from oid 38 from oid 42 current snap is 300 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2832: snap_create 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2826: finishing write tid 1 to smithi20457400-50 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2826: finishing write tid 2 to smithi20457400-50 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2826: finishing write tid 3 to smithi20457400-50 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2826: finishing write tid 4 to smithi20457400-50 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2826: finishing write tid 5 to smithi20457400-50 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1119 (ObjNum 1003 snap 300 seq_num 1003) dirty exists 2024-09-16T05:21:12.837 INFO:tasks.rados.rados.2.smithi204.stdout:2826: left oid 50 (ObjNum 1003 snap 300 seq_num 1003) 2024-09-16T05:21:12.839 INFO:tasks.rados.rados.2.smithi204.stdout:2831: finishing copy_from to smithi20457400-38 2024-09-16T05:21:12.839 INFO:tasks.rados.rados.2.smithi204.stdout:2831: got expected ENOENT (src dne) 2024-09-16T05:21:12.839 INFO:tasks.rados.rados.2.smithi204.stdout:2831: finishing copy_from racing read to smithi20457400-38 2024-09-16T05:21:12.840 INFO:tasks.rados.rados.2.smithi204.stdout:2829: expect (ObjNum 929 snap 275 seq_num 929) 2024-09-16T05:21:12.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:13.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:12 smithi196 ceph-mon[30802]: osdmap e722: 12 total, 12 up, 12 in 2024-09-16T05:21:13.283 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/343889694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:13.284 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:21:13.284 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:21:13.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:21:13.285 INFO:tasks.rados.rados.2.smithi204.stdout:2830: expect (ObjNum 971 snap 287 seq_num 971) 2024-09-16T05:21:13.285 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:12 smithi032 ceph-mon[27388]: osdmap e722: 12 total, 12 up, 12 in 2024-09-16T05:21:13.285 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/343889694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:13.286 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:12 smithi204 ceph-mon[30933]: osdmap e722: 12 total, 12 up, 12 in 2024-09-16T05:21:13.286 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/343889694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:13.286 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:21:13.286 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:21:13.286 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:21:13.286 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:21:13.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:13.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:13.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:13.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:13.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:13.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:13.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:13.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:13.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:13.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:13.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:13.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:13.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:21:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:13.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:13.834 INFO:tasks.rados.rados.2.smithi204.stdout:2826: done (4 left) 2024-09-16T05:21:13.834 INFO:tasks.rados.rados.2.smithi204.stdout:2829: done (3 left) 2024-09-16T05:21:13.834 INFO:tasks.rados.rados.2.smithi204.stdout:2830: done (2 left) 2024-09-16T05:21:13.834 INFO:tasks.rados.rados.2.smithi204.stdout:2831: done (1 left) 2024-09-16T05:21:13.834 INFO:tasks.rados.rados.2.smithi204.stdout:2832: done (0 left) 2024-09-16T05:21:13.834 INFO:tasks.rados.rados.2.smithi204.stdout:2833: copy_from oid 9 from oid 21 current snap is 301 2024-09-16T05:21:13.834 INFO:tasks.rados.rados.2.smithi204.stdout:2834: snap_create 2024-09-16T05:21:13.837 INFO:tasks.rados.rados.2.smithi204.stdout:2833: finishing copy_from to smithi20457400-9 2024-09-16T05:21:13.837 INFO:tasks.rados.rados.2.smithi204.stdout:2833: got expected ENOENT (src dne) 2024-09-16T05:21:13.837 INFO:tasks.rados.rados.2.smithi204.stdout:2833: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:21:13.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:13.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:13.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:14.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:14.064 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:14.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:14.165 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:14.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:13 smithi032 ceph-mon[27388]: pgmap v1245: 117 pgs: 7 peering, 110 active+clean; 305 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 10 MiB/s, 33 objects/s recovering 2024-09-16T05:21:14.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:13 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:14.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:13 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:14.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:13 smithi032 ceph-mon[27388]: osdmap e723: 12 total, 12 up, 12 in 2024-09-16T05:21:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:13 smithi204 ceph-mon[30933]: pgmap v1245: 117 pgs: 7 peering, 110 active+clean; 305 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 10 MiB/s, 33 objects/s recovering 2024-09-16T05:21:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:13 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:13 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:13 smithi204 ceph-mon[30933]: osdmap e723: 12 total, 12 up, 12 in 2024-09-16T05:21:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:14.188 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:14.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:13 smithi196 ceph-mon[30802]: pgmap v1245: 117 pgs: 7 peering, 110 active+clean; 305 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 10 MiB/s, 33 objects/s recovering 2024-09-16T05:21:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:13 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:13 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:13 smithi196 ceph-mon[30802]: osdmap e723: 12 total, 12 up, 12 in 2024-09-16T05:21:14.252 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:14.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:14.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:14.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:14.413 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:14.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:14.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:14.528 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:14.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:14.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:14.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:14.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:14.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:14.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:14.777 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:21:14.777 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:21:14.777 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:21:14.777 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:21:14.777 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:21:14.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:14.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:14.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:14.842 INFO:tasks.rados.rados.2.smithi204.stdout:2833: done (1 left) 2024-09-16T05:21:14.843 INFO:tasks.rados.rados.2.smithi204.stdout:2834: done (0 left) 2024-09-16T05:21:14.843 INFO:tasks.rados.rados.2.smithi204.stdout:2835: snap_create 2024-09-16T05:21:14.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:14.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:14.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:15.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:15.059 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:14 smithi032 ceph-mon[27388]: osdmap e724: 12 total, 12 up, 12 in 2024-09-16T05:21:15.065 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:15.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:15.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:15.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:14 smithi204 ceph-mon[30933]: osdmap e724: 12 total, 12 up, 12 in 2024-09-16T05:21:15.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:15.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:14 smithi196 ceph-mon[30802]: osdmap e724: 12 total, 12 up, 12 in 2024-09-16T05:21:15.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:15.317 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:15.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:15.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:15.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:15.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:15.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:15.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:15.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:15.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:21:15.675 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:15.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:15.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:15.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2835: done (0 left) 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2836: read oid 30 snap -1 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2836: expect deleted 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2837: read oid 12 snap 297 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2837: expect (ObjNum 980 snap 288 seq_num 980) 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2838: rollback oid 25 current snap is 303 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 262 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2839: read oid 10 snap 286 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2839: expect (ObjNum 946 snap 279 seq_num 946) 2024-09-16T05:21:15.810 INFO:tasks.rados.rados.2.smithi204.stdout:2840: snap_remove snap 299 2024-09-16T05:21:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:15.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:15 smithi032 ceph-mon[27388]: pgmap v1248: 117 pgs: 7 peering, 110 active+clean; 302 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 320 KiB/s wr, 4 op/s; 2.2 MiB/s, 16 objects/s recovering 2024-09-16T05:21:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:15 smithi032 ceph-mon[27388]: osdmap e725: 12 total, 12 up, 12 in 2024-09-16T05:21:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:15 smithi032 ceph-mon[27388]: osdmap e726: 12 total, 12 up, 12 in 2024-09-16T05:21:15.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:15.953 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:15.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:16.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:16.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:16.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:16.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:15 smithi204 ceph-mon[30933]: pgmap v1248: 117 pgs: 7 peering, 110 active+clean; 302 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 320 KiB/s wr, 4 op/s; 2.2 MiB/s, 16 objects/s recovering 2024-09-16T05:21:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:15 smithi204 ceph-mon[30933]: osdmap e725: 12 total, 12 up, 12 in 2024-09-16T05:21:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:15 smithi204 ceph-mon[30933]: osdmap e726: 12 total, 12 up, 12 in 2024-09-16T05:21:16.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:16.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:21:16.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:16.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:16.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:15 smithi196 ceph-mon[30802]: pgmap v1248: 117 pgs: 7 peering, 110 active+clean; 302 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 320 KiB/s wr, 4 op/s; 2.2 MiB/s, 16 objects/s recovering 2024-09-16T05:21:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:15 smithi196 ceph-mon[30802]: osdmap e725: 12 total, 12 up, 12 in 2024-09-16T05:21:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:15 smithi196 ceph-mon[30802]: osdmap e726: 12 total, 12 up, 12 in 2024-09-16T05:21:16.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:16.329 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:21:16.331 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:21:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:16.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:16.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:16.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:16.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:16.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:16.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:16.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:16.596 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:16.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:16.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:16.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:21:16.811 INFO:tasks.rados.rados.2.smithi204.stdout:2840: done (4 left) 2024-09-16T05:21:16.811 INFO:tasks.rados.rados.2.smithi204.stdout:2841: write (excl) oid 35 current snap is 303 2024-09-16T05:21:16.811 INFO:tasks.rados.rados.2.smithi204.stdout:2841: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-16T05:21:16.816 INFO:tasks.rados.rados.2.smithi204.stdout:2841: writing smithi20457400-35 from 474228 to 982820 tid 1 2024-09-16T05:21:16.821 INFO:tasks.rados.rados.2.smithi204.stdout:2841: writing smithi20457400-35 from 1451686 to 2006698 tid 2 2024-09-16T05:21:16.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:16.826 INFO:tasks.rados.rados.2.smithi204.stdout:2841: writing smithi20457400-35 from 2704920 to 2874228 tid 3 2024-09-16T05:21:16.826 INFO:tasks.rados.rados.2.smithi204.stdout:2842: snap_remove snap 302 2024-09-16T05:21:16.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:16.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:16.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1423270926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:16.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:16 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:16.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1423270926' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:16.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:16 smithi032 ceph-mon[27388]: osdmap e727: 12 total, 12 up, 12 in 2024-09-16T05:21:16.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:16.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:16.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:17.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:17.105 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:17.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:17.108 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1251,"stamp":"2024-09-16T05:21:16.192615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317079109,"num_objects":149,"num_object_clones":96,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":8,"num_read":3579,"num_read_kb":3507184,"num_write":2639,"num_write_kb":1050196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":89317650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29935,"ondisk_log_size":29935,"up":234,"acting":234,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":238,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1020940,"kb_used_data":491504,"kb_used_omap":27,"kb_used_meta":529124,"kb_avail":1123920884,"statfs":{"total":1151940427776,"available":1150894985216,"internally_reserved":0,"allocated":503300096,"data_stored":557135242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":541823143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2212334,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":9,"num_read_kb":11616,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000828"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":3541,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971681+0000","last_change":"2024-09-16T05:21:12.948493+0000","last_active":"2024-09-16T05:21:15.971681+0000","last_peered":"2024-09-16T05:21:15.971681+0000","last_clean":"2024-09-16T05:21:15.971681+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:21:15.971681+0000","last_undegraded":"2024-09-16T05:21:15.971681+0000","last_fullsized":"2024-09-16T05:21:15.971681+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:20:01.826054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:01.826054+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.334273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025008830000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.54","version":"718'411","reported_seq":3100,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.194892+0000","last_change":"2024-09-16T05:21:12.195071+0000","last_active":"2024-09-16T05:21:12.194892+0000","last_peered":"2024-09-16T05:21:12.194892+0000","last_clean":"2024-09-16T05:21:12.194892+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:21:12.194892+0000","last_undegraded":"2024-09-16T05:21:12.194892+0000","last_fullsized":"2024-09-16T05:21:12.194892+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"686'398","last_scrub_stamp":"2024-09-16T05:20:43.585524+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:20:43.585524+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:47.907899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097164699999999996,"stat_sum":{"num_bytes":2504474,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":68,"num_read_kb":63621,"num_write":43,"num_write_kb":13668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":1442,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808211+0000","last_change":"2024-09-16T05:21:12.831725+0000","last_active":"2024-09-16T05:21:15.808211+0000","last_peered":"2024-09-16T05:21:15.808211+0000","last_clean":"2024-09-16T05:21:15.808211+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:21:15.808211+0000","last_undegraded":"2024-09-16T05:21:15.808211+0000","last_fullsized":"2024-09-16T05:21:15.808211+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:20:40.312020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.312020+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:42.781776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000450913,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":1436,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.840868+0000","last_change":"2024-09-16T05:21:12.830973+0000","last_active":"2024-09-16T05:21:14.840868+0000","last_peered":"2024-09-16T05:21:14.840868+0000","last_clean":"2024-09-16T05:21:14.840868+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:21:14.840868+0000","last_undegraded":"2024-09-16T05:21:14.840868+0000","last_fullsized":"2024-09-16T05:21:14.840868+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:18.588272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.588272+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:34.725768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079791200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":1510,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.807667+0000","last_change":"2024-09-16T05:21:12.834033+0000","last_active":"2024-09-16T05:21:15.807667+0000","last_peered":"2024-09-16T05:21:15.807667+0000","last_clean":"2024-09-16T05:21:15.807667+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:21:15.807667+0000","last_undegraded":"2024-09-16T05:21:15.807667+0000","last_fullsized":"2024-09-16T05:21:15.807667+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:20:59.668844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:59.668844+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:17.171950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000696647,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":1433,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.840830+0000","last_change":"2024-09-16T05:21:12.841491+0000","last_active":"2024-09-16T05:21:12.840830+0000","last_peered":"2024-09-16T05:21:12.840830+0000","last_clean":"2024-09-16T05:21:12.840830+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:21:12.840830+0000","last_undegraded":"2024-09-16T05:21:12.840830+0000","last_fullsized":"2024-09-16T05:21:12.840830+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:20:59.312163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:59.312163+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:16.005588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085962300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":1963,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320518+0000","last_change":"2024-09-16T05:21:12.837991+0000","last_active":"2024-09-16T05:21:14.320518+0000","last_peered":"2024-09-16T05:21:14.320518+0000","last_clean":"2024-09-16T05:21:14.320518+0000","last_became_active":"2024-09-16T05:21:10.817392+0000","last_became_peered":"2024-09-16T05:21:10.817392+0000","last_unstale":"2024-09-16T05:21:14.320518+0000","last_undegraded":"2024-09-16T05:21:14.320518+0000","last_fullsized":"2024-09-16T05:21:14.320518+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:19:48.302807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.302807+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093809700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.4e","version":"272'232","reported_seq":3211,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971926+0000","last_change":"2024-09-16T05:21:12.946490+0000","last_active":"2024-09-16T05:21:15.971926+0000","last_peered":"2024-09-16T05:21:15.971926+0000","last_clean":"2024-09-16T05:21:15.971926+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:21:15.971926+0000","last_undegraded":"2024-09-16T05:21:15.971926+0000","last_fullsized":"2024-09-16T05:21:15.971926+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:21:07.065851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:07.065851+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:21.398618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061225899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.4d","version":"705'633","reported_seq":3751,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.195340+0000","last_change":"2024-09-16T05:21:12.195340+0000","last_active":"2024-09-16T05:21:12.195340+0000","last_peered":"2024-09-16T05:21:12.195340+0000","last_clean":"2024-09-16T05:21:12.195340+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:21:12.195340+0000","last_undegraded":"2024-09-16T05:21:12.195340+0000","last_fullsized":"2024-09-16T05:21:12.195340+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"665'626","last_scrub_stamp":"2024-09-16T05:20:16.492166+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.492166+0000","objects_scrubbed":4,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:34.630676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001232709,"stat_sum":{"num_bytes":3563282,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":38966,"num_write":57,"num_write_kb":21533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.4c","version":"709'374","reported_seq":3445,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808535+0000","last_change":"2024-09-16T05:21:12.834527+0000","last_active":"2024-09-16T05:21:15.808535+0000","last_peered":"2024-09-16T05:21:15.808535+0000","last_clean":"2024-09-16T05:21:15.808535+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:21:15.808535+0000","last_undegraded":"2024-09-16T05:21:15.808535+0000","last_fullsized":"2024-09-16T05:21:15.808535+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"666'345","last_scrub_stamp":"2024-09-16T05:20:17.450506+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.450506+0000","objects_scrubbed":1,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.268265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091859400000000001,"stat_sum":{"num_bytes":5239557,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":49188,"num_write":33,"num_write_kb":9268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":3232,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.194643+0000","last_change":"2024-09-16T05:21:12.195063+0000","last_active":"2024-09-16T05:21:12.194643+0000","last_peered":"2024-09-16T05:21:12.194643+0000","last_clean":"2024-09-16T05:21:12.194643+0000","last_became_active":"2024-09-16T05:17:21.741180+0000","last_became_peered":"2024-09-16T05:17:21.741180+0000","last_unstale":"2024-09-16T05:21:12.194643+0000","last_undegraded":"2024-09-16T05:21:12.194643+0000","last_fullsized":"2024-09-16T05:21:12.194643+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:20:40.593078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.593078+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:04.950579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001012539,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.4a","version":"719'583","reported_seq":3623,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.807731+0000","last_change":"2024-09-16T05:21:12.834275+0000","last_active":"2024-09-16T05:21:15.807731+0000","last_peered":"2024-09-16T05:21:15.807731+0000","last_clean":"2024-09-16T05:21:15.807731+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:21:15.807731+0000","last_undegraded":"2024-09-16T05:21:15.807731+0000","last_fullsized":"2024-09-16T05:21:15.807731+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"620'557","last_scrub_stamp":"2024-09-16T05:19:54.555606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.555606+0000","objects_scrubbed":1,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.921408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078888000000000005,"stat_sum":{"num_bytes":3496854,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":55010,"num_write":39,"num_write_kb":19545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":3280,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971551+0000","last_change":"2024-09-16T05:21:12.948219+0000","last_active":"2024-09-16T05:21:15.971551+0000","last_peered":"2024-09-16T05:21:15.971551+0000","last_clean":"2024-09-16T05:21:15.971551+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:21:15.971551+0000","last_undegraded":"2024-09-16T05:21:15.971551+0000","last_fullsized":"2024-09-16T05:21:15.971551+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:20.094577+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.094577+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:36.439529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022968419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.48","version":"723'369","reported_seq":3406,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.108985+0000","last_change":"2024-09-16T05:21:13.147952+0000","last_active":"2024-09-16T05:21:14.108985+0000","last_peered":"2024-09-16T05:21:14.108985+0000","last_clean":"2024-09-16T05:21:14.108985+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:21:14.108985+0000","last_undegraded":"2024-09-16T05:21:14.108985+0000","last_fullsized":"2024-09-16T05:21:14.108985+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"705'353","last_scrub_stamp":"2024-09-16T05:20:58.009284+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:20:58.009284+0000","objects_scrubbed":4,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:03.209612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001965849,"stat_sum":{"num_bytes":9565472,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":79,"num_read_kb":75759,"num_write":56,"num_write_kb":25106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":2063,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.841204+0000","last_change":"2024-09-16T05:21:12.831250+0000","last_active":"2024-09-16T05:21:14.841204+0000","last_peered":"2024-09-16T05:21:14.841204+0000","last_clean":"2024-09-16T05:21:14.841204+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:21:14.841204+0000","last_undegraded":"2024-09-16T05:21:14.841204+0000","last_fullsized":"2024-09-16T05:21:14.841204+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:01.976068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:01.976068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:29.785667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00067305499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":275,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.195179+0000","last_change":"2024-09-16T05:21:12.195328+0000","last_active":"2024-09-16T05:21:12.195179+0000","last_peered":"2024-09-16T05:21:12.195179+0000","last_clean":"2024-09-16T05:21:12.195179+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:21:12.195179+0000","last_undegraded":"2024-09-16T05:21:12.195179+0000","last_fullsized":"2024-09-16T05:21:12.195179+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:21:05.606142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:05.606142+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:20.167443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001158163,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":2566,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320611+0000","last_change":"2024-09-16T05:21:12.838218+0000","last_active":"2024-09-16T05:21:14.320611+0000","last_peered":"2024-09-16T05:21:14.320611+0000","last_clean":"2024-09-16T05:21:14.320611+0000","last_became_active":"2024-09-16T05:21:10.818112+0000","last_became_peered":"2024-09-16T05:21:10.818112+0000","last_unstale":"2024-09-16T05:21:14.320611+0000","last_undegraded":"2024-09-16T05:21:14.320611+0000","last_fullsized":"2024-09-16T05:21:14.320611+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:20:42.442340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:42.442340+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:02.009045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011407209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":52,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320339+0000","last_change":"2024-09-16T05:21:12.839109+0000","last_active":"2024-09-16T05:21:14.320339+0000","last_peered":"2024-09-16T05:21:14.320339+0000","last_clean":"2024-09-16T05:21:14.320339+0000","last_became_active":"2024-09-16T05:21:07.820306+0000","last_became_peered":"2024-09-16T05:21:07.820306+0000","last_unstale":"2024-09-16T05:21:14.320339+0000","last_undegraded":"2024-09-16T05:21:14.320339+0000","last_fullsized":"2024-09-16T05:21:14.320339+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_clean_scrub_stamp":"2024-09-16T05:21:08.436254+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:23.664161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001746926,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":2028,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816083+0000","last_change":"2024-09-16T05:21:08.818186+0000","last_active":"2024-09-16T05:21:10.816083+0000","last_peered":"2024-09-16T05:21:10.816083+0000","last_clean":"2024-09-16T05:21:10.816083+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:21:10.816083+0000","last_undegraded":"2024-09-16T05:21:10.816083+0000","last_fullsized":"2024-09-16T05:21:10.816083+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:20:38.464664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.464664+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:08.176848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918919999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.42","version":"182'20","reported_seq":2025,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832491+0000","last_change":"2024-09-16T05:21:12.946421+0000","last_active":"2024-09-16T05:21:13.832491+0000","last_peered":"2024-09-16T05:21:13.832491+0000","last_clean":"2024-09-16T05:21:13.832491+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:21:13.832491+0000","last_undegraded":"2024-09-16T05:21:13.832491+0000","last_fullsized":"2024-09-16T05:21:13.832491+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:21:09.465647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:09.465647+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:29.683353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078495099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":902,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.840842+0000","last_change":"2024-09-16T05:21:12.830916+0000","last_active":"2024-09-16T05:21:14.840842+0000","last_peered":"2024-09-16T05:21:14.840842+0000","last_clean":"2024-09-16T05:21:14.840842+0000","last_became_active":"2024-09-16T05:18:19.768547+0000","last_became_peered":"2024-09-16T05:18:19.768547+0000","last_unstale":"2024-09-16T05:21:14.840842+0000","last_undegraded":"2024-09-16T05:21:14.840842+0000","last_fullsized":"2024-09-16T05:21:14.840842+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:21:05.881934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:05.881934+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:19.559384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078067200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":598,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971356+0000","last_change":"2024-09-16T05:21:12.948651+0000","last_active":"2024-09-16T05:21:15.971356+0000","last_peered":"2024-09-16T05:21:15.971356+0000","last_clean":"2024-09-16T05:21:15.971356+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:21:15.971356+0000","last_undegraded":"2024-09-16T05:21:15.971356+0000","last_fullsized":"2024-09-16T05:21:15.971356+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:20:59.988307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:59.988307+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:22.223286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002743283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.3f","version":"719'464","reported_seq":3518,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816797+0000","last_change":"2024-09-16T05:21:08.825788+0000","last_active":"2024-09-16T05:21:10.816797+0000","last_peered":"2024-09-16T05:21:10.816797+0000","last_clean":"2024-09-16T05:21:10.816797+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:21:10.816797+0000","last_undegraded":"2024-09-16T05:21:10.816797+0000","last_fullsized":"2024-09-16T05:21:10.816797+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'415","last_scrub_stamp":"2024-09-16T05:19:58.393481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:58.393481+0000","objects_scrubbed":2,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.648325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0091848629999999997,"stat_sum":{"num_bytes":7245769,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":95,"num_read_kb":82275,"num_write":55,"num_write_kb":18010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.3e","version":"719'505","reported_seq":3525,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.047489+0000","last_change":"2024-09-16T05:21:12.047654+0000","last_active":"2024-09-16T05:21:12.047489+0000","last_peered":"2024-09-16T05:21:12.047489+0000","last_clean":"2024-09-16T05:21:12.047489+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:21:12.047489+0000","last_undegraded":"2024-09-16T05:21:12.047489+0000","last_fullsized":"2024-09-16T05:21:12.047489+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"625'487","last_scrub_stamp":"2024-09-16T05:20:22.404334+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:20:22.404334+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:30.010959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071506800000000002,"stat_sum":{"num_bytes":3921826,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":35562,"num_write":54,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1504,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320783+0000","last_change":"2024-09-16T05:21:12.838874+0000","last_active":"2024-09-16T05:21:14.320783+0000","last_peered":"2024-09-16T05:21:14.320783+0000","last_clean":"2024-09-16T05:21:14.320783+0000","last_became_active":"2024-09-16T05:21:10.817387+0000","last_became_peered":"2024-09-16T05:21:10.817387+0000","last_unstale":"2024-09-16T05:21:14.320783+0000","last_undegraded":"2024-09-16T05:21:14.320783+0000","last_fullsized":"2024-09-16T05:21:14.320783+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:21:05.402898+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_clean_scrub_stamp":"2024-09-16T05:21:05.402898+0000","objects_scrubbed":9,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001524068,"stat_sum":{"num_bytes":14300236,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":17,"num_read_kb":11647,"num_write":24,"num_write_kb":7654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":19591132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.3c","version":"713'757","reported_seq":4045,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816435+0000","last_change":"2024-09-16T05:21:08.824893+0000","last_active":"2024-09-16T05:21:10.816435+0000","last_peered":"2024-09-16T05:21:10.816435+0000","last_clean":"2024-09-16T05:21:10.816435+0000","last_became_active":"2024-09-16T05:21:07.828778+0000","last_became_peered":"2024-09-16T05:21:07.828778+0000","last_unstale":"2024-09-16T05:21:10.816435+0000","last_undegraded":"2024-09-16T05:21:10.816435+0000","last_fullsized":"2024-09-16T05:21:10.816435+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"624'648","last_scrub_stamp":"2024-09-16T05:19:34.258477+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:19:34.258477+0000","objects_scrubbed":6,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0082302399999999998,"stat_sum":{"num_bytes":12386647,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":173,"num_read_kb":225124,"num_write":85,"num_write_kb":33383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":14713263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.3b","version":"723'444","reported_seq":3505,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971564+0000","last_change":"2024-09-16T05:21:12.950331+0000","last_active":"2024-09-16T05:21:15.971564+0000","last_peered":"2024-09-16T05:21:15.971564+0000","last_clean":"2024-09-16T05:21:15.971564+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:21:15.971564+0000","last_undegraded":"2024-09-16T05:21:15.971564+0000","last_fullsized":"2024-09-16T05:21:15.971564+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'416","last_scrub_stamp":"2024-09-16T05:19:55.767476+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.767476+0000","objects_scrubbed":2,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0042210980000000004,"stat_sum":{"num_bytes":4373976,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":63,"num_read_kb":64612,"num_write":47,"num_write_kb":21574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.3a","version":"708'596","reported_seq":3614,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.108972+0000","last_change":"2024-09-16T05:21:13.146662+0000","last_active":"2024-09-16T05:21:14.108972+0000","last_peered":"2024-09-16T05:21:14.108972+0000","last_clean":"2024-09-16T05:21:14.108972+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:21:14.108972+0000","last_undegraded":"2024-09-16T05:21:14.108972+0000","last_fullsized":"2024-09-16T05:21:14.108972+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"667'593","last_scrub_stamp":"2024-09-16T05:20:20.620494+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.620494+0000","objects_scrubbed":2,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.329022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065468299999999998,"stat_sum":{"num_bytes":2571702,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":54,"num_read_kb":46839,"num_write":61,"num_write_kb":26555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":2130,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109039+0000","last_change":"2024-09-16T05:21:13.147453+0000","last_active":"2024-09-16T05:21:14.109039+0000","last_peered":"2024-09-16T05:21:14.109039+0000","last_clean":"2024-09-16T05:21:14.109039+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:21:14.109039+0000","last_undegraded":"2024-09-16T05:21:14.109039+0000","last_fullsized":"2024-09-16T05:21:14.109039+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:19:49.888753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.888753+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014636759999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":2137,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832466+0000","last_change":"2024-09-16T05:21:12.949592+0000","last_active":"2024-09-16T05:21:13.832466+0000","last_peered":"2024-09-16T05:21:13.832466+0000","last_clean":"2024-09-16T05:21:13.832466+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:21:13.832466+0000","last_undegraded":"2024-09-16T05:21:13.832466+0000","last_fullsized":"2024-09-16T05:21:13.832466+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:20:20.396645+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.396645+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:37.421126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035334160000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":999,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808631+0000","last_change":"2024-09-16T05:21:12.834785+0000","last_active":"2024-09-16T05:21:15.808631+0000","last_peered":"2024-09-16T05:21:15.808631+0000","last_clean":"2024-09-16T05:21:15.808631+0000","last_became_active":"2024-09-16T05:17:59.866037+0000","last_became_peered":"2024-09-16T05:17:59.866037+0000","last_unstale":"2024-09-16T05:21:15.808631+0000","last_undegraded":"2024-09-16T05:21:15.808631+0000","last_fullsized":"2024-09-16T05:21:15.808631+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:00.633624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:21:00.633624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:12.722463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001135928,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":44,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320169+0000","last_change":"2024-09-16T05:21:12.837693+0000","last_active":"2024-09-16T05:21:14.320169+0000","last_peered":"2024-09-16T05:21:14.320169+0000","last_clean":"2024-09-16T05:21:14.320169+0000","last_became_active":"2024-09-16T05:21:07.819657+0000","last_became_peered":"2024-09-16T05:21:07.819657+0000","last_unstale":"2024-09-16T05:21:14.320169+0000","last_undegraded":"2024-09-16T05:21:14.320169+0000","last_fullsized":"2024-09-16T05:21:14.320169+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:20:02.294024+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.294024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072806100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.35","version":"723'820","reported_seq":3949,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320596+0000","last_change":"2024-09-16T05:21:12.841004+0000","last_active":"2024-09-16T05:21:14.320596+0000","last_peered":"2024-09-16T05:21:14.320596+0000","last_clean":"2024-09-16T05:21:14.320596+0000","last_became_active":"2024-09-16T05:21:10.815197+0000","last_became_peered":"2024-09-16T05:21:10.815197+0000","last_unstale":"2024-09-16T05:21:14.320596+0000","last_undegraded":"2024-09-16T05:21:14.320596+0000","last_fullsized":"2024-09-16T05:21:14.320596+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"686'768","last_scrub_stamp":"2024-09-16T05:20:38.441947+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.441947+0000","objects_scrubbed":7,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.424234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003490525,"stat_sum":{"num_bytes":20791292,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":140,"num_read_kb":130417,"num_write":104,"num_write_kb":48025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":17054243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.34","version":"723'652","reported_seq":3324,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971365+0000","last_change":"2024-09-16T05:21:12.947919+0000","last_active":"2024-09-16T05:21:15.971365+0000","last_peered":"2024-09-16T05:21:15.971365+0000","last_clean":"2024-09-16T05:21:15.971365+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:21:15.971365+0000","last_undegraded":"2024-09-16T05:21:15.971365+0000","last_fullsized":"2024-09-16T05:21:15.971365+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"686'611","last_scrub_stamp":"2024-09-16T05:20:38.002711+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.002711+0000","objects_scrubbed":7,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.665288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022040089999999998,"stat_sum":{"num_bytes":10670328,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":181,"num_read_kb":210064,"num_write":108,"num_write_kb":41100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":2212,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816498+0000","last_change":"2024-09-16T05:21:08.818206+0000","last_active":"2024-09-16T05:21:10.816498+0000","last_peered":"2024-09-16T05:21:10.816498+0000","last_clean":"2024-09-16T05:21:10.816498+0000","last_became_active":"2024-09-16T05:21:07.818018+0000","last_became_peered":"2024-09-16T05:21:07.818018+0000","last_unstale":"2024-09-16T05:21:10.816498+0000","last_undegraded":"2024-09-16T05:21:10.816498+0000","last_fullsized":"2024-09-16T05:21:10.816498+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:20:59.791825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:59.791825+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:09.255412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019353899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.32","version":"715'257","reported_seq":2456,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.046981+0000","last_change":"2024-09-16T05:21:12.047212+0000","last_active":"2024-09-16T05:21:12.046981+0000","last_peered":"2024-09-16T05:21:12.046981+0000","last_clean":"2024-09-16T05:21:12.046981+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:21:12.046981+0000","last_undegraded":"2024-09-16T05:21:12.046981+0000","last_fullsized":"2024-09-16T05:21:12.046981+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"671'227","last_scrub_stamp":"2024-09-16T05:20:23.405472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.405472+0000","objects_scrubbed":2,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.646369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051191599999999996,"stat_sum":{"num_bytes":7376174,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":101971,"num_write":48,"num_write_kb":21789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.31","version":"709'300","reported_seq":3097,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.841014+0000","last_change":"2024-09-16T05:21:12.831246+0000","last_active":"2024-09-16T05:21:14.841014+0000","last_peered":"2024-09-16T05:21:14.841014+0000","last_clean":"2024-09-16T05:21:14.841014+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:21:14.841014+0000","last_undegraded":"2024-09-16T05:21:14.841014+0000","last_fullsized":"2024-09-16T05:21:14.841014+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'268","last_scrub_stamp":"2024-09-16T05:20:03.547514+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.547514+0000","objects_scrubbed":2,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.446096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094785900000000001,"stat_sum":{"num_bytes":5462704,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":119772,"num_write":43,"num_write_kb":20308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":979,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808738+0000","last_change":"2024-09-16T05:21:12.833842+0000","last_active":"2024-09-16T05:21:15.808738+0000","last_peered":"2024-09-16T05:21:15.808738+0000","last_clean":"2024-09-16T05:21:15.808738+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:21:15.808738+0000","last_undegraded":"2024-09-16T05:21:15.808738+0000","last_fullsized":"2024-09-16T05:21:15.808738+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:20:16.443442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.443442+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:24.348096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064080599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":1088,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971121+0000","last_change":"2024-09-16T05:20:18.091859+0000","last_active":"2024-09-16T05:21:15.971121+0000","last_peered":"2024-09-16T05:21:15.971121+0000","last_clean":"2024-09-16T05:21:15.971121+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:21:15.971121+0000","last_undegraded":"2024-09-16T05:21:15.971121+0000","last_fullsized":"2024-09-16T05:21:15.971121+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.091747+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.091747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.362844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":2761,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:11.830933+0000","last_change":"2024-09-16T05:21:11.831246+0000","last_active":"2024-09-16T05:21:11.830933+0000","last_peered":"2024-09-16T05:21:11.830933+0000","last_clean":"2024-09-16T05:21:11.830933+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:21:11.830933+0000","last_undegraded":"2024-09-16T05:21:11.830933+0000","last_fullsized":"2024-09-16T05:21:11.830933+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:20:03.076376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.076376+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:30.075245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051118300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1054,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.319991+0000","last_change":"2024-09-16T05:21:10.815050+0000","last_active":"2024-09-16T05:21:14.319991+0000","last_peered":"2024-09-16T05:21:14.319991+0000","last_clean":"2024-09-16T05:21:14.319991+0000","last_became_active":"2024-09-16T05:21:10.814832+0000","last_became_peered":"2024-09-16T05:21:10.814832+0000","last_unstale":"2024-09-16T05:21:14.319991+0000","last_undegraded":"2024-09-16T05:21:14.319991+0000","last_fullsized":"2024-09-16T05:21:14.319991+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:43.443433+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:43.443433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.444270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":2699,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832552+0000","last_change":"2024-09-16T05:21:12.949645+0000","last_active":"2024-09-16T05:21:13.832552+0000","last_peered":"2024-09-16T05:21:13.832552+0000","last_clean":"2024-09-16T05:21:13.832552+0000","last_became_active":"2024-09-16T05:21:07.815584+0000","last_became_peered":"2024-09-16T05:21:07.815584+0000","last_unstale":"2024-09-16T05:21:13.832552+0000","last_undegraded":"2024-09-16T05:21:13.832552+0000","last_fullsized":"2024-09-16T05:21:13.832552+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:21:02.441586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.441586+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:29.593077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035637749999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.c","version":"35'139","reported_seq":1044,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.840722+0000","last_change":"2024-09-16T05:20:36.348317+0000","last_active":"2024-09-16T05:21:12.840722+0000","last_peered":"2024-09-16T05:21:12.840722+0000","last_clean":"2024-09-16T05:21:12.840722+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:21:12.840722+0000","last_undegraded":"2024-09-16T05:21:12.840722+0000","last_fullsized":"2024-09-16T05:21:12.840722+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.348227+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.348227+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.541309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"709'409","reported_seq":3489,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.833019+0000","last_change":"2024-09-16T05:21:12.949358+0000","last_active":"2024-09-16T05:21:13.833019+0000","last_peered":"2024-09-16T05:21:13.833019+0000","last_clean":"2024-09-16T05:21:13.833019+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:21:13.833019+0000","last_undegraded":"2024-09-16T05:21:13.833019+0000","last_fullsized":"2024-09-16T05:21:13.833019+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"709'409","last_scrub_stamp":"2024-09-16T05:21:03.413122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:03.413122+0000","objects_scrubbed":1,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:13.631340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033826749999999999,"stat_sum":{"num_bytes":2722965,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57663,"num_write":41,"num_write_kb":16134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.d","version":"35'139","reported_seq":1195,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816678+0000","last_change":"2024-09-16T05:20:37.488421+0000","last_active":"2024-09-16T05:21:10.816678+0000","last_peered":"2024-09-16T05:21:10.816678+0000","last_clean":"2024-09-16T05:21:10.816678+0000","last_became_active":"2024-09-16T05:20:35.804924+0000","last_became_peered":"2024-09-16T05:20:35.804924+0000","last_unstale":"2024-09-16T05:21:10.816678+0000","last_undegraded":"2024-09-16T05:21:10.816678+0000","last_fullsized":"2024-09-16T05:21:10.816678+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.488293+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.488293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.991686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":3190,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.195464+0000","last_change":"2024-09-16T05:21:12.195633+0000","last_active":"2024-09-16T05:21:12.195464+0000","last_peered":"2024-09-16T05:21:12.195464+0000","last_clean":"2024-09-16T05:21:12.195464+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:21:12.195464+0000","last_undegraded":"2024-09-16T05:21:12.195464+0000","last_fullsized":"2024-09-16T05:21:12.195464+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:20:39.563939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.563939+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:09.056514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012348039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":1203,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.807750+0000","last_change":"2024-09-16T05:20:36.368708+0000","last_active":"2024-09-16T05:21:15.807750+0000","last_peered":"2024-09-16T05:21:15.807750+0000","last_clean":"2024-09-16T05:21:15.807750+0000","last_became_active":"2024-09-16T05:20:35.802192+0000","last_became_peered":"2024-09-16T05:20:35.802192+0000","last_unstale":"2024-09-16T05:21:15.807750+0000","last_undegraded":"2024-09-16T05:21:15.807750+0000","last_fullsized":"2024-09-16T05:21:15.807750+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.368583+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.368583+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.464774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"723'636","reported_seq":3715,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109264+0000","last_change":"2024-09-16T05:21:13.149756+0000","last_active":"2024-09-16T05:21:14.109264+0000","last_peered":"2024-09-16T05:21:14.109264+0000","last_clean":"2024-09-16T05:21:14.109264+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:21:14.109264+0000","last_undegraded":"2024-09-16T05:21:14.109264+0000","last_fullsized":"2024-09-16T05:21:14.109264+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"709'630","last_scrub_stamp":"2024-09-16T05:21:00.044713+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:21:00.044713+0000","objects_scrubbed":3,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:25.680432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033003759999999998,"stat_sum":{"num_bytes":7830043,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":52780,"num_write":54,"num_write_kb":30221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":1183,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971417+0000","last_change":"2024-09-16T05:20:39.039439+0000","last_active":"2024-09-16T05:21:15.971417+0000","last_peered":"2024-09-16T05:21:15.971417+0000","last_clean":"2024-09-16T05:21:15.971417+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:21:15.971417+0000","last_undegraded":"2024-09-16T05:21:15.971417+0000","last_fullsized":"2024-09-16T05:21:15.971417+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:39.039365+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.039365+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.718627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3318,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.047265+0000","last_change":"2024-09-16T05:21:12.047433+0000","last_active":"2024-09-16T05:21:12.047265+0000","last_peered":"2024-09-16T05:21:12.047265+0000","last_clean":"2024-09-16T05:21:12.047265+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:21:12.047265+0000","last_undegraded":"2024-09-16T05:21:12.047265+0000","last_fullsized":"2024-09-16T05:21:12.047265+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:20:39.456780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.456780+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:08.323976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064851999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1720,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.046634+0000","last_change":"2024-09-16T05:20:36.468183+0000","last_active":"2024-09-16T05:21:12.046634+0000","last_peered":"2024-09-16T05:21:12.046634+0000","last_clean":"2024-09-16T05:21:12.046634+0000","last_became_active":"2024-09-16T05:20:35.805506+0000","last_became_peered":"2024-09-16T05:20:35.805506+0000","last_unstale":"2024-09-16T05:21:12.046634+0000","last_undegraded":"2024-09-16T05:21:12.046634+0000","last_fullsized":"2024-09-16T05:21:12.046634+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.468079+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.468079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.101000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"705'327","reported_seq":3372,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.815878+0000","last_change":"2024-09-16T05:21:08.816827+0000","last_active":"2024-09-16T05:21:10.815878+0000","last_peered":"2024-09-16T05:21:10.815878+0000","last_clean":"2024-09-16T05:21:10.815878+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:21:10.815878+0000","last_undegraded":"2024-09-16T05:21:10.815878+0000","last_fullsized":"2024-09-16T05:21:10.815878+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'325","last_scrub_stamp":"2024-09-16T05:20:23.462954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.462954+0000","objects_scrubbed":2,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:46.503842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096999499999999997,"stat_sum":{"num_bytes":2235529,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":47,"num_read_kb":35425,"num_write":51,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.3","version":"723'235","reported_seq":3319,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971968+0000","last_change":"2024-09-16T05:21:12.948187+0000","last_active":"2024-09-16T05:21:15.971968+0000","last_peered":"2024-09-16T05:21:15.971968+0000","last_clean":"2024-09-16T05:21:15.971968+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:21:15.971968+0000","last_undegraded":"2024-09-16T05:21:15.971968+0000","last_fullsized":"2024-09-16T05:21:15.971968+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'205","last_scrub_stamp":"2024-09-16T05:20:19.137710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:19.137710+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.270962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022847969999999999,"stat_sum":{"num_bytes":5796527,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":117938,"num_write":51,"num_write_kb":20718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":719,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.046563+0000","last_change":"2024-09-16T05:20:00.246338+0000","last_active":"2024-09-16T05:21:12.046563+0000","last_peered":"2024-09-16T05:21:12.046563+0000","last_clean":"2024-09-16T05:21:12.046563+0000","last_became_active":"2024-09-16T05:18:19.768392+0000","last_became_peered":"2024-09-16T05:18:19.768392+0000","last_unstale":"2024-09-16T05:21:12.046563+0000","last_undegraded":"2024-09-16T05:21:12.046563+0000","last_fullsized":"2024-09-16T05:21:12.046563+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:00.246242+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.246242+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:13.060953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":3073,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971094+0000","last_change":"2024-09-16T05:21:12.946286+0000","last_active":"2024-09-16T05:21:15.971094+0000","last_peered":"2024-09-16T05:21:15.971094+0000","last_clean":"2024-09-16T05:21:15.971094+0000","last_became_active":"2024-09-16T05:21:07.817459+0000","last_became_peered":"2024-09-16T05:21:07.817459+0000","last_unstale":"2024-09-16T05:21:15.971094+0000","last_undegraded":"2024-09-16T05:21:15.971094+0000","last_fullsized":"2024-09-16T05:21:15.971094+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:02.847979+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.847979+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:26.303661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055376500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":962,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.193752+0000","last_change":"2024-09-16T05:21:02.510553+0000","last_active":"2024-09-16T05:21:12.193752+0000","last_peered":"2024-09-16T05:21:12.193752+0000","last_clean":"2024-09-16T05:21:12.193752+0000","last_became_active":"2024-09-16T05:20:53.807695+0000","last_became_peered":"2024-09-16T05:20:53.807695+0000","last_unstale":"2024-09-16T05:21:12.193752+0000","last_undegraded":"2024-09-16T05:21:12.193752+0000","last_fullsized":"2024-09-16T05:21:12.193752+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:02.510463+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.510463+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:24.081875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1696,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816134+0000","last_change":"2024-09-16T05:19:56.353335+0000","last_active":"2024-09-16T05:21:10.816134+0000","last_peered":"2024-09-16T05:21:10.816134+0000","last_clean":"2024-09-16T05:21:10.816134+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:21:10.816134+0000","last_undegraded":"2024-09-16T05:21:10.816134+0000","last_fullsized":"2024-09-16T05:21:10.816134+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:19:56.353243+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.757169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":3000,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.194372+0000","last_change":"2024-09-16T05:21:12.194580+0000","last_active":"2024-09-16T05:21:12.194372+0000","last_peered":"2024-09-16T05:21:12.194372+0000","last_clean":"2024-09-16T05:21:12.194372+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:21:12.194372+0000","last_undegraded":"2024-09-16T05:21:12.194372+0000","last_fullsized":"2024-09-16T05:21:12.194372+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:25.536951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:25.536951+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:32.390131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074242799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1712,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.840704+0000","last_change":"2024-09-16T05:20:37.311421+0000","last_active":"2024-09-16T05:21:12.840704+0000","last_peered":"2024-09-16T05:21:12.840704+0000","last_clean":"2024-09-16T05:21:12.840704+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:21:12.840704+0000","last_undegraded":"2024-09-16T05:21:12.840704+0000","last_fullsized":"2024-09-16T05:21:12.840704+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.311338+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.311338+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:37.446300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":3071,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816103+0000","last_change":"2024-09-16T05:21:08.817812+0000","last_active":"2024-09-16T05:21:10.816103+0000","last_peered":"2024-09-16T05:21:10.816103+0000","last_clean":"2024-09-16T05:21:10.816103+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:21:10.816103+0000","last_undegraded":"2024-09-16T05:21:10.816103+0000","last_fullsized":"2024-09-16T05:21:10.816103+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:51.365804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.365804+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001926222,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":2300,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.108910+0000","last_change":"2024-09-16T05:21:13.150017+0000","last_active":"2024-09-16T05:21:14.108910+0000","last_peered":"2024-09-16T05:21:14.108910+0000","last_clean":"2024-09-16T05:21:14.108910+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:21:14.108910+0000","last_undegraded":"2024-09-16T05:21:14.108910+0000","last_fullsized":"2024-09-16T05:21:14.108910+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:50.852123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.852123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003342243,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":1664,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.046748+0000","last_change":"2024-09-16T05:19:53.322398+0000","last_active":"2024-09-16T05:21:12.046748+0000","last_peered":"2024-09-16T05:21:12.046748+0000","last_clean":"2024-09-16T05:21:12.046748+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:21:12.046748+0000","last_undegraded":"2024-09-16T05:21:12.046748+0000","last_fullsized":"2024-09-16T05:21:12.046748+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:53.322303+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.322303+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:12.744427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1482,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.195393+0000","last_change":"2024-09-16T05:21:12.195561+0000","last_active":"2024-09-16T05:21:12.195393+0000","last_peered":"2024-09-16T05:21:12.195393+0000","last_clean":"2024-09-16T05:21:12.195393+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:21:12.195393+0000","last_undegraded":"2024-09-16T05:21:12.195393+0000","last_fullsized":"2024-09-16T05:21:12.195393+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_clean_scrub_stamp":"2024-09-16T05:21:08.532148+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:20.019769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001232808,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":992,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808918+0000","last_change":"2024-09-16T05:20:38.411870+0000","last_active":"2024-09-16T05:21:15.808918+0000","last_peered":"2024-09-16T05:21:15.808918+0000","last_clean":"2024-09-16T05:21:15.808918+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:21:15.808918+0000","last_undegraded":"2024-09-16T05:21:15.808918+0000","last_fullsized":"2024-09-16T05:21:15.808918+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:38.411767+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.411767+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:52.750017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"709'599","reported_seq":3676,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832841+0000","last_change":"2024-09-16T05:21:12.946790+0000","last_active":"2024-09-16T05:21:13.832841+0000","last_peered":"2024-09-16T05:21:13.832841+0000","last_clean":"2024-09-16T05:21:13.832841+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:21:13.832841+0000","last_undegraded":"2024-09-16T05:21:13.832841+0000","last_fullsized":"2024-09-16T05:21:13.832841+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'568","last_scrub_stamp":"2024-09-16T05:19:54.247232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.247232+0000","objects_scrubbed":3,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087716900000000002,"stat_sum":{"num_bytes":4768139,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":146749,"num_write":39,"num_write_kb":12889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":936,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808019+0000","last_change":"2024-09-16T05:20:37.414043+0000","last_active":"2024-09-16T05:21:15.808019+0000","last_peered":"2024-09-16T05:21:15.808019+0000","last_clean":"2024-09-16T05:21:15.808019+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:21:15.808019+0000","last_undegraded":"2024-09-16T05:21:15.808019+0000","last_fullsized":"2024-09-16T05:21:15.808019+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.413968+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.413968+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.433963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":44,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320165+0000","last_change":"2024-09-16T05:21:12.838450+0000","last_active":"2024-09-16T05:21:14.320165+0000","last_peered":"2024-09-16T05:21:14.320165+0000","last_clean":"2024-09-16T05:21:14.320165+0000","last_became_active":"2024-09-16T05:21:07.816472+0000","last_became_peered":"2024-09-16T05:21:07.816472+0000","last_unstale":"2024-09-16T05:21:14.320165+0000","last_undegraded":"2024-09-16T05:21:14.320165+0000","last_fullsized":"2024-09-16T05:21:14.320165+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:20:38.827535+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.827535+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:04.792150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013341519999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":972,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808308+0000","last_change":"2024-09-16T05:20:39.847071+0000","last_active":"2024-09-16T05:21:15.808308+0000","last_peered":"2024-09-16T05:21:15.808308+0000","last_clean":"2024-09-16T05:21:15.808308+0000","last_became_active":"2024-09-16T05:20:35.811624+0000","last_became_peered":"2024-09-16T05:20:35.811624+0000","last_unstale":"2024-09-16T05:21:15.808308+0000","last_undegraded":"2024-09-16T05:21:15.808308+0000","last_fullsized":"2024-09-16T05:21:15.808308+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:39.846968+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.846968+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:46.110945+0000","scrub_duration":389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":3243,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832579+0000","last_change":"2024-09-16T05:21:12.946473+0000","last_active":"2024-09-16T05:21:13.832579+0000","last_peered":"2024-09-16T05:21:13.832579+0000","last_clean":"2024-09-16T05:21:13.832579+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:21:13.832579+0000","last_undegraded":"2024-09-16T05:21:13.832579+0000","last_fullsized":"2024-09-16T05:21:13.832579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:21:04.422977+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:21:04.422977+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:16.393334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075590900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":1718,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971687+0000","last_change":"2024-09-16T05:20:36.004657+0000","last_active":"2024-09-16T05:21:15.971687+0000","last_peered":"2024-09-16T05:21:15.971687+0000","last_clean":"2024-09-16T05:21:15.971687+0000","last_became_active":"2024-09-16T05:20:35.807550+0000","last_became_peered":"2024-09-16T05:20:35.807550+0000","last_unstale":"2024-09-16T05:21:15.971687+0000","last_undegraded":"2024-09-16T05:21:15.971687+0000","last_fullsized":"2024-09-16T05:21:15.971687+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.004558+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.004558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.268598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"719'650","reported_seq":3731,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.047061+0000","last_change":"2024-09-16T05:21:12.047246+0000","last_active":"2024-09-16T05:21:12.047061+0000","last_peered":"2024-09-16T05:21:12.047061+0000","last_clean":"2024-09-16T05:21:12.047061+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:21:12.047061+0000","last_undegraded":"2024-09-16T05:21:12.047061+0000","last_fullsized":"2024-09-16T05:21:12.047061+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"653'601","last_scrub_stamp":"2024-09-16T05:20:03.311152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.311152+0000","objects_scrubbed":3,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:12.596750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055488100000000002,"stat_sum":{"num_bytes":6140292,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":127394,"num_write":58,"num_write_kb":34915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1706,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.815829+0000","last_change":"2024-09-16T05:20:37.260334+0000","last_active":"2024-09-16T05:21:10.815829+0000","last_peered":"2024-09-16T05:21:10.815829+0000","last_clean":"2024-09-16T05:21:10.815829+0000","last_became_active":"2024-09-16T05:20:35.804221+0000","last_became_peered":"2024-09-16T05:20:35.804221+0000","last_unstale":"2024-09-16T05:21:10.815829+0000","last_undegraded":"2024-09-16T05:21:10.815829+0000","last_fullsized":"2024-09-16T05:21:10.815829+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.260191+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.260191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:55.461837+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1518,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.319751+0000","last_change":"2024-09-16T05:21:12.837319+0000","last_active":"2024-09-16T05:21:14.319751+0000","last_peered":"2024-09-16T05:21:14.319751+0000","last_clean":"2024-09-16T05:21:14.319751+0000","last_became_active":"2024-09-16T05:21:10.815913+0000","last_became_peered":"2024-09-16T05:21:10.815913+0000","last_unstale":"2024-09-16T05:21:14.319751+0000","last_undegraded":"2024-09-16T05:21:14.319751+0000","last_fullsized":"2024-09-16T05:21:14.319751+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:38.316049+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.316049+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:38.674121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051039100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1710,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109051+0000","last_change":"2024-09-16T05:20:35.832422+0000","last_active":"2024-09-16T05:21:14.109051+0000","last_peered":"2024-09-16T05:21:14.109051+0000","last_clean":"2024-09-16T05:21:14.109051+0000","last_became_active":"2024-09-16T05:20:35.812754+0000","last_became_peered":"2024-09-16T05:20:35.812754+0000","last_unstale":"2024-09-16T05:21:14.109051+0000","last_undegraded":"2024-09-16T05:21:14.109051+0000","last_fullsized":"2024-09-16T05:21:14.109051+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:35.832353+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:20:35.832353+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:43.196095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1200,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.840778+0000","last_change":"2024-09-16T05:21:02.949505+0000","last_active":"2024-09-16T05:21:14.840778+0000","last_peered":"2024-09-16T05:21:14.840778+0000","last_clean":"2024-09-16T05:21:14.840778+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:21:14.840778+0000","last_undegraded":"2024-09-16T05:21:14.840778+0000","last_fullsized":"2024-09-16T05:21:14.840778+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:02.949433+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.949433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:17.475261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":979,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808045+0000","last_change":"2024-09-16T05:21:12.833770+0000","last_active":"2024-09-16T05:21:15.808045+0000","last_peered":"2024-09-16T05:21:15.808045+0000","last_clean":"2024-09-16T05:21:15.808045+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:21:15.808045+0000","last_undegraded":"2024-09-16T05:21:15.808045+0000","last_fullsized":"2024-09-16T05:21:15.808045+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:24.333369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:24.333369+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:49.706782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053709299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1707,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971217+0000","last_change":"2024-09-16T05:19:48.737475+0000","last_active":"2024-09-16T05:21:15.971217+0000","last_peered":"2024-09-16T05:21:15.971217+0000","last_clean":"2024-09-16T05:21:15.971217+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:21:15.971217+0000","last_undegraded":"2024-09-16T05:21:15.971217+0000","last_fullsized":"2024-09-16T05:21:15.971217+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:48.737348+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.737348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.039442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":2689,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109223+0000","last_change":"2024-09-16T05:21:13.148130+0000","last_active":"2024-09-16T05:21:14.109223+0000","last_peered":"2024-09-16T05:21:14.109223+0000","last_clean":"2024-09-16T05:21:14.109223+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:21:14.109223+0000","last_undegraded":"2024-09-16T05:21:14.109223+0000","last_fullsized":"2024-09-16T05:21:14.109223+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:21:01.077265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:01.077265+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:21.109522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001849769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.17","version":"35'139","reported_seq":593,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832842+0000","last_change":"2024-09-16T05:20:43.198542+0000","last_active":"2024-09-16T05:21:13.832842+0000","last_peered":"2024-09-16T05:21:13.832842+0000","last_clean":"2024-09-16T05:21:13.832842+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:21:13.832842+0000","last_undegraded":"2024-09-16T05:21:13.832842+0000","last_fullsized":"2024-09-16T05:21:13.832842+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:43.198473+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:20:43.198473+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.563368+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":2961,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109189+0000","last_change":"2024-09-16T05:21:13.148279+0000","last_active":"2024-09-16T05:21:14.109189+0000","last_peered":"2024-09-16T05:21:14.109189+0000","last_clean":"2024-09-16T05:21:14.109189+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:21:14.109189+0000","last_undegraded":"2024-09-16T05:21:14.109189+0000","last_fullsized":"2024-09-16T05:21:14.109189+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:20:37.810519+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.810519+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:47.135956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001970627,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.16","version":"35'139","reported_seq":647,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.193983+0000","last_change":"2024-09-16T05:20:41.564653+0000","last_active":"2024-09-16T05:21:12.193983+0000","last_peered":"2024-09-16T05:21:12.193983+0000","last_clean":"2024-09-16T05:21:12.193983+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:21:12.193983+0000","last_undegraded":"2024-09-16T05:21:12.193983+0000","last_fullsized":"2024-09-16T05:21:12.193983+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.564525+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:54.885944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"709'655","reported_seq":3855,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109239+0000","last_change":"2024-09-16T05:21:13.148468+0000","last_active":"2024-09-16T05:21:14.109239+0000","last_peered":"2024-09-16T05:21:14.109239+0000","last_clean":"2024-09-16T05:21:14.109239+0000","last_became_active":"2024-09-16T05:21:07.826042+0000","last_became_peered":"2024-09-16T05:21:07.826042+0000","last_unstale":"2024-09-16T05:21:14.109239+0000","last_undegraded":"2024-09-16T05:21:14.109239+0000","last_fullsized":"2024-09-16T05:21:14.109239+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"689'645","last_scrub_stamp":"2024-09-16T05:20:39.846432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.846432+0000","objects_scrubbed":3,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:02.581487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020627330000000002,"stat_sum":{"num_bytes":2369985,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":7118573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.15","version":"35'139","reported_seq":671,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971230+0000","last_change":"2024-09-16T05:20:40.949219+0000","last_active":"2024-09-16T05:21:15.971230+0000","last_peered":"2024-09-16T05:21:15.971230+0000","last_clean":"2024-09-16T05:21:15.971230+0000","last_became_active":"2024-09-16T05:20:40.802090+0000","last_became_peered":"2024-09-16T05:20:40.802090+0000","last_unstale":"2024-09-16T05:21:15.971230+0000","last_undegraded":"2024-09-16T05:21:15.971230+0000","last_fullsized":"2024-09-16T05:21:15.971230+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:40.949147+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.949147+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.611229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"723'332","reported_seq":3038,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971680+0000","last_change":"2024-09-16T05:21:12.950485+0000","last_active":"2024-09-16T05:21:15.971680+0000","last_peered":"2024-09-16T05:21:15.971680+0000","last_clean":"2024-09-16T05:21:15.971680+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:21:15.971680+0000","last_undegraded":"2024-09-16T05:21:15.971680+0000","last_fullsized":"2024-09-16T05:21:15.971680+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"665'294","last_scrub_stamp":"2024-09-16T05:20:15.010850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.010850+0000","objects_scrubbed":4,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:31.269418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0042705909999999998,"stat_sum":{"num_bytes":7278037,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":65,"num_read_kb":57072,"num_write":58,"num_write_kb":18502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":1029,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.194232+0000","last_change":"2024-09-16T05:20:37.550357+0000","last_active":"2024-09-16T05:21:12.194232+0000","last_peered":"2024-09-16T05:21:12.194232+0000","last_clean":"2024-09-16T05:21:12.194232+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:21:12.194232+0000","last_undegraded":"2024-09-16T05:21:12.194232+0000","last_fullsized":"2024-09-16T05:21:12.194232+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.550279+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.550279+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.895726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":52,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320358+0000","last_change":"2024-09-16T05:21:12.837477+0000","last_active":"2024-09-16T05:21:14.320358+0000","last_peered":"2024-09-16T05:21:14.320358+0000","last_clean":"2024-09-16T05:21:14.320358+0000","last_became_active":"2024-09-16T05:21:07.822615+0000","last_became_peered":"2024-09-16T05:21:07.822615+0000","last_unstale":"2024-09-16T05:21:14.320358+0000","last_undegraded":"2024-09-16T05:21:14.320358+0000","last_fullsized":"2024-09-16T05:21:14.320358+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:20:20.393358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.393358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:33.111667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052486700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.1b","version":"35'139","reported_seq":1118,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832459+0000","last_change":"2024-09-16T05:20:41.173883+0000","last_active":"2024-09-16T05:21:13.832459+0000","last_peered":"2024-09-16T05:21:13.832459+0000","last_clean":"2024-09-16T05:21:13.832459+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:21:13.832459+0000","last_undegraded":"2024-09-16T05:21:13.832459+0000","last_fullsized":"2024-09-16T05:21:13.832459+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.173808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:00.160931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":3071,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109031+0000","last_change":"2024-09-16T05:21:13.147155+0000","last_active":"2024-09-16T05:21:14.109031+0000","last_peered":"2024-09-16T05:21:14.109031+0000","last_clean":"2024-09-16T05:21:14.109031+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:21:14.109031+0000","last_undegraded":"2024-09-16T05:21:14.109031+0000","last_fullsized":"2024-09-16T05:21:14.109031+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:21:06.114038+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:21:06.114038+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:26.710146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011519239999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.1a","version":"35'139","reported_seq":487,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.807481+0000","last_change":"2024-09-16T05:20:41.339619+0000","last_active":"2024-09-16T05:21:15.807481+0000","last_peered":"2024-09-16T05:21:15.807481+0000","last_clean":"2024-09-16T05:21:15.807481+0000","last_became_active":"2024-09-16T05:20:40.806260+0000","last_became_peered":"2024-09-16T05:20:40.806260+0000","last_unstale":"2024-09-16T05:21:15.807481+0000","last_undegraded":"2024-09-16T05:21:15.807481+0000","last_fullsized":"2024-09-16T05:21:15.807481+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.339515+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.339515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.872026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":2675,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.840943+0000","last_change":"2024-09-16T05:21:12.841815+0000","last_active":"2024-09-16T05:21:12.840943+0000","last_peered":"2024-09-16T05:21:12.840943+0000","last_clean":"2024-09-16T05:21:12.840943+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:21:12.840943+0000","last_undegraded":"2024-09-16T05:21:12.840943+0000","last_fullsized":"2024-09-16T05:21:12.840943+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:20:17.403045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.403045+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:43.673752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001039346,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.19","version":"35'139","reported_seq":1717,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808284+0000","last_change":"2024-09-16T05:20:41.437268+0000","last_active":"2024-09-16T05:21:15.808284+0000","last_peered":"2024-09-16T05:21:15.808284+0000","last_clean":"2024-09-16T05:21:15.808284+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:21:15.808284+0000","last_undegraded":"2024-09-16T05:21:15.808284+0000","last_fullsized":"2024-09-16T05:21:15.808284+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.437156+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.437156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.770457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"723'1111","reported_seq":4118,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832604+0000","last_change":"2024-09-16T05:21:12.949370+0000","last_active":"2024-09-16T05:21:13.832604+0000","last_peered":"2024-09-16T05:21:13.832604+0000","last_clean":"2024-09-16T05:21:13.832604+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:21:13.832604+0000","last_undegraded":"2024-09-16T05:21:13.832604+0000","last_fullsized":"2024-09-16T05:21:13.832604+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"653'957","last_scrub_stamp":"2024-09-16T05:20:03.383437+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.383437+0000","objects_scrubbed":11,"log_size":1111,"log_dups_size":0,"ondisk_log_size":1111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0034691050000000001,"stat_sum":{"num_bytes":22354013,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":171,"num_read_kb":148608,"num_write":182,"num_write_kb":66629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.18","version":"35'139","reported_seq":1719,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971038+0000","last_change":"2024-09-16T05:20:37.024610+0000","last_active":"2024-09-16T05:21:15.971038+0000","last_peered":"2024-09-16T05:21:15.971038+0000","last_clean":"2024-09-16T05:21:15.971038+0000","last_became_active":"2024-09-16T05:20:35.809442+0000","last_became_peered":"2024-09-16T05:20:35.809442+0000","last_unstale":"2024-09-16T05:21:15.971038+0000","last_undegraded":"2024-09-16T05:21:15.971038+0000","last_fullsized":"2024-09-16T05:21:15.971038+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.024521+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.024521+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.640487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"709'654","reported_seq":3735,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.047347+0000","last_change":"2024-09-16T05:21:12.047499+0000","last_active":"2024-09-16T05:21:12.047347+0000","last_peered":"2024-09-16T05:21:12.047347+0000","last_clean":"2024-09-16T05:21:12.047347+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:21:12.047347+0000","last_undegraded":"2024-09-16T05:21:12.047347+0000","last_fullsized":"2024-09-16T05:21:12.047347+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"666'612","last_scrub_stamp":"2024-09-16T05:20:16.258329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.258329+0000","objects_scrubbed":4,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:26.987717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063432699999999996,"stat_sum":{"num_bytes":8572558,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":118389,"num_write":107,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.1c","version":"723'472","reported_seq":3247,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808139+0000","last_change":"2024-09-16T05:21:12.835691+0000","last_active":"2024-09-16T05:21:15.808139+0000","last_peered":"2024-09-16T05:21:15.808139+0000","last_clean":"2024-09-16T05:21:15.808139+0000","last_became_active":"2024-09-16T05:18:02.890972+0000","last_became_peered":"2024-09-16T05:18:02.890972+0000","last_unstale":"2024-09-16T05:21:15.808139+0000","last_undegraded":"2024-09-16T05:21:15.808139+0000","last_fullsized":"2024-09-16T05:21:15.808139+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"704'466","last_scrub_stamp":"2024-09-16T05:21:02.653288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.653288+0000","objects_scrubbed":2,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:09.998774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021458340000000001,"stat_sum":{"num_bytes":4102515,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":10804,"num_write":32,"num_write_kb":13970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1196,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.046874+0000","last_change":"2024-09-16T05:20:42.448898+0000","last_active":"2024-09-16T05:21:12.046874+0000","last_peered":"2024-09-16T05:21:12.046874+0000","last_clean":"2024-09-16T05:21:12.046874+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:21:12.046874+0000","last_undegraded":"2024-09-16T05:21:12.046874+0000","last_fullsized":"2024-09-16T05:21:12.046874+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:42.448822+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:42.448822+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.950314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"723'1280","reported_seq":4588,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808730+0000","last_change":"2024-09-16T05:21:12.837807+0000","last_active":"2024-09-16T05:21:15.808730+0000","last_peered":"2024-09-16T05:21:15.808730+0000","last_clean":"2024-09-16T05:21:15.808730+0000","last_became_active":"2024-09-16T05:18:03.891103+0000","last_became_peered":"2024-09-16T05:18:03.891103+0000","last_unstale":"2024-09-16T05:21:15.808730+0000","last_undegraded":"2024-09-16T05:21:15.808730+0000","last_fullsized":"2024-09-16T05:21:15.808730+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"705'1256","last_scrub_stamp":"2024-09-16T05:20:58.648411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:58.648411+0000","objects_scrubbed":8,"log_size":1280,"log_dups_size":0,"ondisk_log_size":1280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:19.358820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038714259999999999,"stat_sum":{"num_bytes":13681777,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":281,"num_read_kb":226672,"num_write":255,"num_write_kb":80012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7155994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"1.1d","version":"35'139","reported_seq":539,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.320668+0000","last_change":"2024-09-16T05:21:10.815207+0000","last_active":"2024-09-16T05:21:14.320668+0000","last_peered":"2024-09-16T05:21:14.320668+0000","last_clean":"2024-09-16T05:21:14.320668+0000","last_became_active":"2024-09-16T05:21:10.815037+0000","last_became_peered":"2024-09-16T05:21:10.815037+0000","last_unstale":"2024-09-16T05:21:14.320668+0000","last_undegraded":"2024-09-16T05:21:14.320668+0000","last_fullsized":"2024-09-16T05:21:14.320668+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.421551+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.421551+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.444309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"723'503","reported_seq":3601,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.843480+0000","last_change":"2024-09-16T05:21:12.843703+0000","last_active":"2024-09-16T05:21:12.843480+0000","last_peered":"2024-09-16T05:21:12.843480+0000","last_clean":"2024-09-16T05:21:12.843480+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:21:12.843480+0000","last_undegraded":"2024-09-16T05:21:12.843480+0000","last_fullsized":"2024-09-16T05:21:12.843480+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"671'464","last_scrub_stamp":"2024-09-16T05:20:21.358700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:21.358700+0000","objects_scrubbed":3,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:27.156151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023589560000000002,"stat_sum":{"num_bytes":4588707,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":80,"num_read_kb":82356,"num_write":31,"num_write_kb":14930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"1.1c","version":"35'139","reported_seq":438,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832499+0000","last_change":"2024-09-16T05:20:42.167372+0000","last_active":"2024-09-16T05:21:13.832499+0000","last_peered":"2024-09-16T05:21:13.832499+0000","last_clean":"2024-09-16T05:21:13.832499+0000","last_became_active":"2024-09-16T05:20:40.805879+0000","last_became_peered":"2024-09-16T05:20:40.805879+0000","last_unstale":"2024-09-16T05:21:13.832499+0000","last_undegraded":"2024-09-16T05:21:13.832499+0000","last_fullsized":"2024-09-16T05:21:13.832499+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:42.167269+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:42.167269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:51.343885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":3294,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.841226+0000","last_change":"2024-09-16T05:21:12.831566+0000","last_active":"2024-09-16T05:21:14.841226+0000","last_peered":"2024-09-16T05:21:14.841226+0000","last_clean":"2024-09-16T05:21:14.841226+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:21:14.841226+0000","last_undegraded":"2024-09-16T05:21:14.841226+0000","last_fullsized":"2024-09-16T05:21:14.841226+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:20:17.544083+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.544083+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:29.769594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077637699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.20","version":"709'321","reported_seq":3405,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.815868+0000","last_change":"2024-09-16T05:21:08.817646+0000","last_active":"2024-09-16T05:21:10.815868+0000","last_peered":"2024-09-16T05:21:10.815868+0000","last_clean":"2024-09-16T05:21:10.815868+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:21:10.815868+0000","last_undegraded":"2024-09-16T05:21:10.815868+0000","last_fullsized":"2024-09-16T05:21:10.815868+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'290","last_scrub_stamp":"2024-09-16T05:20:03.351330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.351330+0000","objects_scrubbed":3,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:18.579677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001240035,"stat_sum":{"num_bytes":4282651,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":177272,"num_write":58,"num_write_kb":21452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.21","version":"178'50","reported_seq":3007,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.194579+0000","last_change":"2024-09-16T05:21:12.194770+0000","last_active":"2024-09-16T05:21:12.194579+0000","last_peered":"2024-09-16T05:21:12.194579+0000","last_clean":"2024-09-16T05:21:12.194579+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:21:12.194579+0000","last_undegraded":"2024-09-16T05:21:12.194579+0000","last_fullsized":"2024-09-16T05:21:12.194579+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:21:09.544178+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:21:09.544178+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:24.151312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080499100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":979,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.808600+0000","last_change":"2024-09-16T05:21:12.834091+0000","last_active":"2024-09-16T05:21:15.808600+0000","last_peered":"2024-09-16T05:21:15.808600+0000","last_clean":"2024-09-16T05:21:15.808600+0000","last_became_active":"2024-09-16T05:17:59.866047+0000","last_became_peered":"2024-09-16T05:17:59.866047+0000","last_unstale":"2024-09-16T05:21:15.808600+0000","last_undegraded":"2024-09-16T05:21:15.808600+0000","last_fullsized":"2024-09-16T05:21:15.808600+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:15.452705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.452705+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.430235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069019999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":2557,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.841558+0000","last_change":"2024-09-16T05:21:12.841818+0000","last_active":"2024-09-16T05:21:12.841558+0000","last_peered":"2024-09-16T05:21:12.841558+0000","last_clean":"2024-09-16T05:21:12.841558+0000","last_became_active":"2024-09-16T05:20:20.801064+0000","last_became_peered":"2024-09-16T05:20:20.801064+0000","last_unstale":"2024-09-16T05:21:12.841558+0000","last_undegraded":"2024-09-16T05:21:12.841558+0000","last_fullsized":"2024-09-16T05:21:12.841558+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:21:07.244384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:07.244384+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:26.314692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086552600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":2357,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109274+0000","last_change":"2024-09-16T05:21:13.147799+0000","last_active":"2024-09-16T05:21:14.109274+0000","last_peered":"2024-09-16T05:21:14.109274+0000","last_clean":"2024-09-16T05:21:14.109274+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:21:14.109274+0000","last_undegraded":"2024-09-16T05:21:14.109274+0000","last_fullsized":"2024-09-16T05:21:14.109274+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:57.859756+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:19:57.859756+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.685127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015900790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.25","version":"723'1203","reported_seq":4180,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832961+0000","last_change":"2024-09-16T05:21:12.948892+0000","last_active":"2024-09-16T05:21:13.832961+0000","last_peered":"2024-09-16T05:21:13.832961+0000","last_clean":"2024-09-16T05:21:13.832961+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:21:13.832961+0000","last_undegraded":"2024-09-16T05:21:13.832961+0000","last_fullsized":"2024-09-16T05:21:13.832961+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"718'1195","last_scrub_stamp":"2024-09-16T05:21:08.434367+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:21:08.434367+0000","objects_scrubbed":11,"log_size":1203,"log_dups_size":0,"ondisk_log_size":1203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:17.916660+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0031206699999999999,"stat_sum":{"num_bytes":22744435,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":205,"num_read_kb":205717,"num_write":194,"num_write_kb":76992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":2335,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816639+0000","last_change":"2024-09-16T05:21:08.818971+0000","last_active":"2024-09-16T05:21:10.816639+0000","last_peered":"2024-09-16T05:21:10.816639+0000","last_clean":"2024-09-16T05:21:10.816639+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:21:10.816639+0000","last_undegraded":"2024-09-16T05:21:10.816639+0000","last_fullsized":"2024-09-16T05:21:10.816639+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:21:00.818968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:00.818968+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:21.211383+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018730929999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.27","version":"0'0","reported_seq":2347,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816822+0000","last_change":"2024-09-16T05:21:08.824544+0000","last_active":"2024-09-16T05:21:10.816822+0000","last_peered":"2024-09-16T05:21:10.816822+0000","last_clean":"2024-09-16T05:21:10.816822+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:21:10.816822+0000","last_undegraded":"2024-09-16T05:21:10.816822+0000","last_fullsized":"2024-09-16T05:21:10.816822+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:02.023438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.023438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:16.901151+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0.0078822600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.28","version":"245'105","reported_seq":1372,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.194301+0000","last_change":"2024-09-16T05:21:12.194526+0000","last_active":"2024-09-16T05:21:12.194301+0000","last_peered":"2024-09-16T05:21:12.194301+0000","last_clean":"2024-09-16T05:21:12.194301+0000","last_became_active":"2024-09-16T05:21:07.818030+0000","last_became_peered":"2024-09-16T05:21:07.818030+0000","last_unstale":"2024-09-16T05:21:12.194301+0000","last_undegraded":"2024-09-16T05:21:12.194301+0000","last_fullsized":"2024-09-16T05:21:12.194301+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:20:58.501508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:58.501508+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.311800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072631499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.29","version":"719'1116","reported_seq":4189,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:11.830898+0000","last_change":"2024-09-16T05:21:11.831349+0000","last_active":"2024-09-16T05:21:11.830898+0000","last_peered":"2024-09-16T05:21:11.830898+0000","last_clean":"2024-09-16T05:21:11.830898+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:21:11.830898+0000","last_undegraded":"2024-09-16T05:21:11.830898+0000","last_fullsized":"2024-09-16T05:21:11.830898+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"708'1087","last_scrub_stamp":"2024-09-16T05:20:58.246006+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:58.246006+0000","objects_scrubbed":14,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:08.638798+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00065084599999999995,"stat_sum":{"num_bytes":37180578,"num_objects":14,"num_object_clones":10,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":282,"num_read_kb":264862,"num_write":211,"num_write_kb":80927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":3337,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816357+0000","last_change":"2024-09-16T05:21:08.817728+0000","last_active":"2024-09-16T05:21:10.816357+0000","last_peered":"2024-09-16T05:21:10.816357+0000","last_clean":"2024-09-16T05:21:10.816357+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:21:10.816357+0000","last_undegraded":"2024-09-16T05:21:10.816357+0000","last_fullsized":"2024-09-16T05:21:10.816357+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_clean_scrub_stamp":"2024-09-16T05:21:03.783094+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:21.054110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013279800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.2b","version":"719'362","reported_seq":3342,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-09-16T05:21:12.195613+0000","last_change":"2024-09-16T05:21:12.195866+0000","last_active":"2024-09-16T05:21:12.195613+0000","last_peered":"2024-09-16T05:21:12.195613+0000","last_clean":"2024-09-16T05:21:12.195613+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:21:12.195613+0000","last_undegraded":"2024-09-16T05:21:12.195613+0000","last_fullsized":"2024-09-16T05:21:12.195613+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"666'332","last_scrub_stamp":"2024-09-16T05:20:18.508468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.508468+0000","objects_scrubbed":1,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:21.205925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014428889999999999,"stat_sum":{"num_bytes":4924362,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":79891,"num_write":55,"num_write_kb":23400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":3182,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:13.832851+0000","last_change":"2024-09-16T05:21:12.949175+0000","last_active":"2024-09-16T05:21:13.832851+0000","last_peered":"2024-09-16T05:21:13.832851+0000","last_clean":"2024-09-16T05:21:13.832851+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:21:13.832851+0000","last_undegraded":"2024-09-16T05:21:13.832851+0000","last_fullsized":"2024-09-16T05:21:13.832851+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:19:51.201665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.201665+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032721640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]},{"pgid":"2.2d","version":"719'500","reported_seq":3542,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816707+0000","last_change":"2024-09-16T05:21:08.824197+0000","last_active":"2024-09-16T05:21:10.816707+0000","last_peered":"2024-09-16T05:21:10.816707+0000","last_clean":"2024-09-16T05:21:10.816707+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:21:10.816707+0000","last_undegraded":"2024-09-16T05:21:10.816707+0000","last_fullsized":"2024-09-16T05:21:10.816707+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"648'457","last_scrub_stamp":"2024-09-16T05:20:15.223777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.223777+0000","objects_scrubbed":2,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.063441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076881969999999999,"stat_sum":{"num_bytes":10461192,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":94,"num_read_kb":88584,"num_write":74,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.2e","version":"719'677","reported_seq":3677,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-09-16T05:21:10.816360+0000","last_change":"2024-09-16T05:21:08.822901+0000","last_active":"2024-09-16T05:21:10.816360+0000","last_peered":"2024-09-16T05:21:10.816360+0000","last_clean":"2024-09-16T05:21:10.816360+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:21:10.816360+0000","last_undegraded":"2024-09-16T05:21:10.816360+0000","last_fullsized":"2024-09-16T05:21:10.816360+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"715'671","last_scrub_stamp":"2024-09-16T05:21:06.794552+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:21:06.794552+0000","objects_scrubbed":7,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:25.081535+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0050083219999999999,"stat_sum":{"num_bytes":18865570,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":95,"num_read_kb":64999,"num_write":96,"num_write_kb":43850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"1"},{"start":"121","length":"2"}]},{"pgid":"2.2f","version":"723'626","reported_seq":3727,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-09-16T05:21:14.109334+0000","last_change":"2024-09-16T05:21:13.150082+0000","last_active":"2024-09-16T05:21:14.109334+0000","last_peered":"2024-09-16T05:21:14.109334+0000","last_clean":"2024-09-16T05:21:14.109334+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:21:14.109334+0000","last_undegraded":"2024-09-16T05:21:14.109334+0000","last_fullsized":"2024-09-16T05:21:14.109334+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"650'553","last_scrub_stamp":"2024-09-16T05:20:00.804910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.804910+0000","objects_scrubbed":6,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:16.395450+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.003576891,"stat_sum":{"num_bytes":12118573,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":121,"num_read_kb":85525,"num_write":108,"num_write_kb":43924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11b","length":"1"},{"start":"11d","length":"2"},{"start":"121","length":"2"},{"start":"127","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":316488741,"num_objects":147,"num_object_clones":96,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":8,"num_read":3579,"num_read_kb":3507184,"num_write":2639,"num_write_kb":1050196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":89317650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397197312,"data_stored":486143517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25626,"ondisk_log_size":25626,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089490,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114424,"kb_used_data":61028,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93630728,"statfs":{"total":95995035648,"available":95877865472,"internally_reserved":0,"allocated":62492672,"data_stored":69992940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285719,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84324,"kb_used_data":30928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93660828,"statfs":{"total":95995035648,"available":95908687872,"internally_reserved":0,"allocated":31670272,"data_stored":32884593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514656,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95528,"kb_used_data":37968,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93649624,"statfs":{"total":95995035648,"available":95897214976,"internally_reserved":0,"allocated":38879232,"data_stored":42575244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841695,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81124,"kb_used_data":45768,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93664028,"statfs":{"total":95995035648,"available":95911964672,"internally_reserved":0,"allocated":46866432,"data_stored":50400973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103333,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113520,"kb_used_data":55956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93631632,"statfs":{"total":95995035648,"available":95878791168,"internally_reserved":0,"allocated":57298944,"data_stored":64147457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397676,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118812,"kb_used_data":61256,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93626340,"statfs":{"total":95995035648,"available":95873372160,"internally_reserved":0,"allocated":62726144,"data_stored":72806827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757423,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62656,"kb_used_data":35624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93682496,"statfs":{"total":95995035648,"available":95930875904,"internally_reserved":0,"allocated":36478976,"data_stored":43597902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920949,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57108,"kb_used_data":25912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688044,"statfs":{"total":95995035648,"available":95936557056,"internally_reserved":0,"allocated":26533888,"data_stored":25914646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117177,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119192,"kb_used_data":65796,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93625960,"statfs":{"total":95995035648,"available":95872983040,"internally_reserved":0,"allocated":67375104,"data_stored":83480690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240464,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62708,"kb_used_data":22112,"kb_used_omap":10,"kb_used_meta":40565,"kb_avail":93682444,"statfs":{"total":95995035648,"available":95930822656,"internally_reserved":0,"allocated":22642688,"data_stored":21050506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":41538867},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62016,"kb_used_data":30820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683136,"statfs":{"total":95995035648,"available":95931531264,"internally_reserved":0,"allocated":31559680,"data_stored":31980832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738569,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49528,"kb_used_data":18336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93695624,"statfs":{"total":95995035648,"available":95944318976,"internally_reserved":0,"allocated":18776064,"data_stored":18302632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9691136,"data_stored":12145873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22802432,"data_stored":26151753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13643776,"data_stored":14975735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58314752,"data_stored":77340927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17776640,"data_stored":20085567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27504640,"data_stored":37559130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53993472,"data_stored":66994744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48578560,"data_stored":58343875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":38121472,"data_stored":44580339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":30121984,"data_stored":36746165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":22888448,"data_stored":27038552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53760000,"data_stored":64180857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:21:17.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1423270926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:16 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1423270926' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:16 smithi204 ceph-mon[30933]: osdmap e727: 12 total, 12 up, 12 in 2024-09-16T05:21:17.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:17.230 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:17.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:17.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1423270926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:16 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1423270926' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:16 smithi196 ceph-mon[30802]: osdmap e727: 12 total, 12 up, 12 in 2024-09-16T05:21:17.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:17.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:17.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:17.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:17.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:17.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:17.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:17.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:17.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:17.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:17.737 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:17.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:17.813 INFO:tasks.rados.rados.2.smithi204.stdout:2838: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:21:17.813 INFO:tasks.rados.rados.2.smithi204.stdout:2838: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:21:17.813 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 734 (ObjNum 129 snap 0 seq_num 4283038280) dirty dne 2024-09-16T05:21:17.813 INFO:tasks.rados.rados.2.smithi204.stdout:2837: expect (ObjNum 980 snap 288 seq_num 980) 2024-09-16T05:21:17.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:17.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:17.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:17.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:17.921 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:17.922 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:17.936 INFO:tasks.rados.rados.2.smithi204.stdout:2839: expect (ObjNum 946 snap 279 seq_num 946) 2024-09-16T05:21:17.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:17.987 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:17.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:18.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:17 smithi032 ceph-mon[27388]: pgmap v1251: 117 pgs: 117 active+clean; 302 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2 op/s 2024-09-16T05:21:18.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:17 smithi032 ceph-mon[27388]: from='client.16902 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:21:18.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:17 smithi032 ceph-mon[27388]: osdmap e728: 12 total, 12 up, 12 in 2024-09-16T05:21:18.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:18.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:18.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2841: finishing write tid 1 to smithi20457400-35 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2841: finishing write tid 2 to smithi20457400-35 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2836: done (5 left) 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2837: done (4 left) 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2838: done (3 left) 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2839: done (2 left) 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2842: done (1 left) 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2843: copy_from oid 9 from oid 39 current snap is 303 2024-09-16T05:21:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:2841: finishing write tid 3 to smithi20457400-35 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2841: finishing write tid 4 to smithi20457400-35 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2841: finishing write tid 5 to smithi20457400-35 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 850 (ObjNum 1004 snap 303 seq_num 1004) dirty exists 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2841: left oid 35 (ObjNum 1004 snap 303 seq_num 1004) 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2841: done (1 left) 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2844: read oid 29 snap -1 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2844: expect (ObjNum 981 snap 288 seq_num 981) 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2845: read oid 40 snap 262 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2845: expect (ObjNum 730 snap 210 seq_num 730) 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2846: write (excl) oid 4 current snap is 303 2024-09-16T05:21:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:2846: seq_num 1005 ranges {491035=583841,1510168=698670,2732343=158692} 2024-09-16T05:21:18.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:17 smithi204 ceph-mon[30933]: pgmap v1251: 117 pgs: 117 active+clean; 302 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2 op/s 2024-09-16T05:21:18.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:17 smithi204 ceph-mon[30933]: from='client.16902 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:21:18.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:17 smithi204 ceph-mon[30933]: osdmap e728: 12 total, 12 up, 12 in 2024-09-16T05:21:18.177 INFO:tasks.rados.rados.2.smithi204.stdout:2846: writing smithi20457400-4 from 491035 to 1074876 tid 1 2024-09-16T05:21:18.183 INFO:tasks.rados.rados.2.smithi204.stdout:2846: writing smithi20457400-4 from 1510168 to 2208838 tid 2 2024-09-16T05:21:18.187 INFO:tasks.rados.rados.2.smithi204.stdout:2846: writing smithi20457400-4 from 2732343 to 2891035 tid 3 2024-09-16T05:21:18.187 INFO:tasks.rados.rados.2.smithi204.stdout:2847: read oid 25 snap -1 2024-09-16T05:21:18.187 INFO:tasks.rados.rados.2.smithi204.stdout:2847: expect deleted 2024-09-16T05:21:18.187 INFO:tasks.rados.rados.2.smithi204.stdout:2848: rollback oid 31 current snap is 303 2024-09-16T05:21:18.188 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 31 to 294 2024-09-16T05:21:18.188 INFO:tasks.rados.rados.2.smithi204.stdout:2844: expect (ObjNum 981 snap 288 seq_num 981) 2024-09-16T05:21:18.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:18.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:17 smithi196 ceph-mon[30802]: pgmap v1251: 117 pgs: 117 active+clean; 302 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2 op/s 2024-09-16T05:21:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:17 smithi196 ceph-mon[30802]: from='client.16902 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:21:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:17 smithi196 ceph-mon[30802]: osdmap e728: 12 total, 12 up, 12 in 2024-09-16T05:21:18.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:18.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:18.267 INFO:tasks.rados.rados.2.smithi204.stdout:2845: expect (ObjNum 730 snap 210 seq_num 730) 2024-09-16T05:21:18.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:18.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:18.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:18.469 INFO:tasks.rados.rados.2.smithi204.stdout:2846: finishing write tid 1 to smithi20457400-4 2024-09-16T05:21:18.469 INFO:tasks.rados.rados.2.smithi204.stdout:2846: finishing write tid 2 to smithi20457400-4 2024-09-16T05:21:18.469 INFO:tasks.rados.rados.2.smithi204.stdout:2846: finishing write tid 3 to smithi20457400-4 2024-09-16T05:21:18.470 INFO:tasks.rados.rados.2.smithi204.stdout:2846: finishing write tid 4 to smithi20457400-4 2024-09-16T05:21:18.470 INFO:tasks.rados.rados.2.smithi204.stdout:2846: finishing write tid 5 to smithi20457400-4 2024-09-16T05:21:18.470 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1012 (ObjNum 1005 snap 303 seq_num 1005) dirty exists 2024-09-16T05:21:18.470 INFO:tasks.rados.rados.2.smithi204.stdout:2846: left oid 4 (ObjNum 1005 snap 303 seq_num 1005) 2024-09-16T05:21:18.470 INFO:tasks.rados.rados.2.smithi204.stdout:2848: finishing rollback tid 1 to smithi20457400-31 2024-09-16T05:21:18.471 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 33 snap 0 seq_num 4294967291) dirty dne 2024-09-16T05:21:18.471 INFO:tasks.rados.rados.2.smithi204.stdout:2843: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:21:18.471 INFO:tasks.rados.rados.2.smithi204.stdout:2843: finishing copy_from to smithi20457400-9 2024-09-16T05:21:18.471 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1175 (ObjNum 987 snap 292 seq_num 987) dirty exists 2024-09-16T05:21:18.471 INFO:tasks.rados.rados.2.smithi204.stdout:2843: done (5 left) 2024-09-16T05:21:18.471 INFO:tasks.rados.rados.2.smithi204.stdout:2844: done (4 left) 2024-09-16T05:21:18.471 INFO:tasks.rados.rados.2.smithi204.stdout:2845: done (3 left) 2024-09-16T05:21:18.472 INFO:tasks.rados.rados.2.smithi204.stdout:2846: done (2 left) 2024-09-16T05:21:18.472 INFO:tasks.rados.rados.2.smithi204.stdout:2847: done (1 left) 2024-09-16T05:21:18.472 INFO:tasks.rados.rados.2.smithi204.stdout:2848: done (0 left) 2024-09-16T05:21:18.472 INFO:tasks.rados.rados.2.smithi204.stdout:2849: write (excl) oid 23 current snap is 303 2024-09-16T05:21:18.472 INFO:tasks.rados.rados.2.smithi204.stdout:2849: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2024-09-16T05:21:18.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:18.475 INFO:tasks.rados.rados.2.smithi204.stdout:2849: writing smithi20457400-23 from 507842 to 1166932 tid 1 2024-09-16T05:21:18.479 INFO:tasks.rados.rados.2.smithi204.stdout:2849: writing smithi20457400-23 from 1568650 to 2127331 tid 2 2024-09-16T05:21:18.484 INFO:tasks.rados.rados.2.smithi204.stdout:2849: writing smithi20457400-23 from 2759766 to 2907842 tid 3 2024-09-16T05:21:18.485 INFO:tasks.rados.rados.2.smithi204.stdout:2850: delete oid 3 current snap is 303 2024-09-16T05:21:18.485 INFO:tasks.rados.rados.2.smithi204.stdout:2849: finishing write tid 1 to smithi20457400-23 2024-09-16T05:21:18.485 INFO:tasks.rados.rados.2.smithi204.stdout:2849: finishing write tid 2 to smithi20457400-23 2024-09-16T05:21:18.486 INFO:tasks.rados.rados.2.smithi204.stdout:2850: done (1 left) 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:2851: copy_from oid 34 from oid 8 current snap is 303 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:2852: rollback oid 9 current snap is 303 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 9 to 297 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:2853: snap_create 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:2849: finishing write tid 3 to smithi20457400-23 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:2849: finishing write tid 4 to smithi20457400-23 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:2849: finishing write tid 5 to smithi20457400-23 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 725 (ObjNum 1006 snap 303 seq_num 1006) dirty exists 2024-09-16T05:21:18.487 INFO:tasks.rados.rados.2.smithi204.stdout:2849: left oid 23 (ObjNum 1006 snap 303 seq_num 1006) 2024-09-16T05:21:18.488 INFO:tasks.rados.rados.2.smithi204.stdout:2852: finishing rollback tid 0 to smithi20457400-9 2024-09-16T05:21:18.488 INFO:tasks.rados.rados.2.smithi204.stdout:2852: finishing rollback tid 1 to smithi20457400-9 2024-09-16T05:21:18.492 INFO:tasks.rados.rados.2.smithi204.stdout:2852: finishing rollback tid 2 to smithi20457400-9 2024-09-16T05:21:18.492 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1258 (ObjNum 985 snap 289 seq_num 985) dirty exists 2024-09-16T05:21:18.497 INFO:tasks.rados.rados.2.smithi204.stdout:2851: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:21:18.500 INFO:tasks.rados.rados.2.smithi204.stdout:2851: finishing copy_from to smithi20457400-34 2024-09-16T05:21:18.500 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1255 (ObjNum 997 snap 293 seq_num 997) dirty exists 2024-09-16T05:21:18.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:18.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:18.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:18.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:18.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:18.729 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:21:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:21:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:21:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1133 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'206, mlcod=189'203)", 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 555.99245138000003, 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021225035, 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734227+0000", 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.85e-07 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734829+0000", 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060196800000000003 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734866+0000", 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7469000000000003e-05 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734881+0000", 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5058999999999999e-05 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734970+0000", 2024-09-16T05:21:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9004000000000004e-05 2024-09-16T05:21:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.734998+0000", 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7606e-05 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755384+0000", 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.02038587 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755452+0000", 2024-09-16T05:21:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7873999999999994e-05 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1134 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'207, mlcod=189'203)", 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 555.98766625799999, 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016477325000000001, 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739012+0000", 2024-09-16T05:21:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739014+0000", 2024-09-16T05:21:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2819999999999999e-06 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739797+0000", 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00078265599999999996 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739844+0000", 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7608000000000002e-05 2024-09-16T05:21:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739854+0000", 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4239999999999999e-06 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739935+0000", 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1593000000000003e-05 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.739963+0000", 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7492e-05 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755459+0000", 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015496565 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755489+0000", 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9705000000000001e-05 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1135 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'208, mlcod=189'203)", 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 555.983243421, 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012094011, 2024-09-16T05:21:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743435+0000", 2024-09-16T05:21:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743436+0000", 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.584e-06 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743535+0000", 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9110000000000005e-05 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743568+0000", 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2335000000000001e-05 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743576+0000", 2024-09-16T05:21:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.191e-06 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743646+0000", 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0481000000000005e-05 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.743676+0000", 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9620000000000001e-05 2024-09-16T05:21:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755492+0000", 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011815888 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755529+0000", 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6801999999999998e-05 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:1136 2.a e191/107 2:5ae26c04:::smithi20457400-32:head v 191'209, mlcod=189'203)", 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 555.98263586300004, 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011564019, 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:18.750 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744042+0000", 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744043+0000", 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3600000000000003e-07 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744049+0000", 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8250000000000001e-06 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744069+0000", 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0460000000000001e-05 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744074+0000", 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4730000000000002e-06 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744119+0000", 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5093999999999999e-05 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.744143+0000", 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4085999999999999e-05 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755570+0000", 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011427299 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:12:02.755606+0000", 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146000000000003e-05 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:2777 2.48 e359/322 2:12e92f6c:::smithi20457400-1:head v 359'155, mlcod=0'0)", 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 369.97533970000001, 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007754843, 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:21:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751338+0000", 2024-09-16T05:21:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.751339+0000", 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5400000000000002e-07 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755560+0000", 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004221341 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755638+0000", 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8292000000000002e-05 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755652+0000", 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3702e-05 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755795+0000", 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014327200000000001 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.755823+0000", 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7668e-05 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.759023+0000", 2024-09-16T05:21:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003199959 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:08.759093+0000", 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0355e-05 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:606 2.48 e365/322 2:12e92f6c:::smithi20457400-1:79 v 365'161, mlcod=362'157)", 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 363.65554252700002, 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64187130800000003, 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071139+0000", 2024-09-16T05:21:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5609999999999999e-06 2024-09-16T05:21:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712578+0000", 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64143852199999996 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712582+0000", 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-06 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712651+0000", 2024-09-16T05:21:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9102000000000003e-05 2024-09-16T05:21:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712670+0000", 2024-09-16T05:21:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9165999999999999e-05 2024-09-16T05:21:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712968+0000", 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029722899999999998 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.713007+0000", 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9125999999999998e-05 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:614 2.48 e366/322 2:12e92f6c:::smithi20457400-1:79 v 366'163, mlcod=364'159)", 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 363.00611529700001, 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11945663500000001, 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:21:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9699999999999996e-07 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720568+0000", 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6870000000000002e-06 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839371+0000", 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11880265399999999 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839384+0000", 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3156000000000001e-05 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839531+0000", 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014728399999999999 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839575+0000", 2024-09-16T05:21:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4223e-05 2024-09-16T05:21:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839981+0000", 2024-09-16T05:21:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040524499999999999 2024-09-16T05:21:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.840019+0000", 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8788999999999997e-05 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3235 2.29 e406/253 2:9662c29c:::smithi20457400-9:head v 406'547, mlcod=402'543)", 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 322.99642779099997, 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081876009999999992, 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730250+0000", 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.730252+0000", 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.857e-06 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734729+0000", 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044764510000000002 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734790+0000", 2024-09-16T05:21:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1465999999999997e-05 2024-09-16T05:21:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734803+0000", 2024-09-16T05:21:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2729e-05 2024-09-16T05:21:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734890+0000", 2024-09-16T05:21:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7708999999999997e-05 2024-09-16T05:21:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.780 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.734938+0000", 2024-09-16T05:21:18.780 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7485999999999999e-05 2024-09-16T05:21:18.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.738369+0000", 2024-09-16T05:21:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034306649999999998 2024-09-16T05:21:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:55.738438+0000", 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9238000000000004e-05 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3412 2.3e e424/296 2:7e4e65a4:::smithi20457400-4:head v 424'357, mlcod=378'353)", 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 304.789212163, 2024-09-16T05:21:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007702319, 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.937466+0000", 2024-09-16T05:21:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9000000000000001e-07 2024-09-16T05:21:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942005+0000", 2024-09-16T05:21:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045392510000000002 2024-09-16T05:21:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942060+0000", 2024-09-16T05:21:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4716000000000003e-05 2024-09-16T05:21:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942070+0000", 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0078000000000001e-05 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942135+0000", 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4853999999999999e-05 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.942159+0000", 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3839999999999999e-05 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.945112+0000", 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0029533789999999999 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:13.945168+0000", 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6011000000000003e-05 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4629 2.1e 2:7b642727:::smithi20457400-40:head [copy-from ver 511] snapc bf=[bf,be] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.97415377900001, 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01647931, 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4629 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752526+0000", 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.894e-06 2024-09-16T05:21:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752536+0000", 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752555+0000", 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8907999999999999e-05 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752564+0000", 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752638+0000", 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3343999999999994e-05 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752646+0000", 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7690000000000002e-06 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754646+0000", 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001999331 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754740+0000", 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3778999999999998e-05 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.764207+0000", 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094672899999999997 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:21:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768754+0000", 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045470199999999997 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768860+0000", 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010586600000000001 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768948+0000", 2024-09-16T05:21:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8503000000000003e-05 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.769004+0000", 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5362000000000003e-05 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4630 2.1e 2:7b642727:::smithi20457400-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:21:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 216.97409602900001, 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015483190000000001, 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4630 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.34e-07 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752584+0000", 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.956e-06 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752591+0000", 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2580000000000001e-06 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752594+0000", 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6830000000000001e-06 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752703+0000", 2024-09-16T05:21:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108818 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752708+0000", 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.561e-06 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754913+0000", 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022055680000000002 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754965+0000", 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2222000000000002e-05 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767827+0000", 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012861097 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767929+0000", 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010275799999999999 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768065+0000", 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013593500000000001 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:859 2.3e e561/296 2:7e4e65a4:::smithi20457400-4:d0 v 561'447, mlcod=557'443)", 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 167.96173292099999, 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007143066, 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:21:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:21:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764945+0000", 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2300000000000002e-07 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.764950+0000", 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771606+0000", 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006656103 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771614+0000", 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.712e-06 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771652+0000", 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8497000000000001e-05 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.771668+0000", 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5829e-05 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.772021+0000", 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00035285500000000001 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.772088+0000", 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7297000000000003e-05 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5275 2.a e561/320 2:5089ebe1:::smithi20457400-36:head v 561'529, mlcod=526'525)", 2024-09-16T05:21:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:21:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 167.81842882800001, 2024-09-16T05:21:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0069804639999999996, 2024-09-16T05:21:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908249+0000", 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.908251+0000", 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4360000000000001e-06 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912064+0000", 2024-09-16T05:21:18.814 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038136189999999999 2024-09-16T05:21:18.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.816 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.816 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912103+0000", 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8689999999999997e-05 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912115+0000", 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.19e-05 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912182+0000", 2024-09-16T05:21:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7565999999999997e-05 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.912205+0000", 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2220000000000001e-05 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.915180+0000", 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002975536 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:30.915230+0000", 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9496999999999997e-05 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:142 2.1e e571/291 ondisk, result = 0)", 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 156.96251987400001, 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55541789799999997, 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764159+0000", 2024-09-16T05:21:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:21:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764162+0000", 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599999999999998e-06 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319278+0000", 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55511595899999999 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319296+0000", 2024-09-16T05:21:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7900000000000001e-05 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319440+0000", 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143175 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319450+0000", 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0705e-05 2024-09-16T05:21:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319576+0000", 2024-09-16T05:21:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012584299999999999 2024-09-16T05:21:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5587 2.34 e583/278 2:2d3f13f8:::smithi20457400-7:head v 583'466, mlcod=583'463)", 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 145.919573523, 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14534661199999999, 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807109+0000", 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2030000000000002e-06 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951917+0000", 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14480800699999999 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951927+0000", 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0115000000000001e-05 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951990+0000", 2024-09-16T05:21:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2689999999999998e-05 2024-09-16T05:21:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952011+0000", 2024-09-16T05:21:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.084e-05 2024-09-16T05:21:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952413+0000", 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040190199999999999 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952451+0000", 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8302e-05 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:383 2.29 e617/253 2:9662c29c:::smithi20457400-9:d8 v 617'830, mlcod=616'824)", 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 111.943885342, 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038031031999999999, 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782794+0000", 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3500000000000005e-07 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782801+0000", 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1210000000000001e-06 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819928+0000", 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037126740999999998 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819938+0000", 2024-09-16T05:21:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0845e-05 2024-09-16T05:21:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820001+0000", 2024-09-16T05:21:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2786999999999994e-05 2024-09-16T05:21:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820023+0000", 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.213e-05 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820771+0000", 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074818699999999998 2024-09-16T05:21:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820824+0000", 2024-09-16T05:21:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2286e-05 2024-09-16T05:21:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:6936 2.29 e698/253 2:9662c29c:::smithi20457400-9:head v 698'1043, mlcod=696'1039)", 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:20:47.833283+0000", 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.893394849, 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0051589330000000001, 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.833283+0000", 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.833283+0000", 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.833284+0000", 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3000000000000001e-07 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.836401+0000", 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031171060000000001 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.836424+0000", 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3329999999999999e-05 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.836431+0000", 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6880000000000002e-06 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.836472+0000", 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0698000000000002e-05 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.836489+0000", 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7693000000000002e-05 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.838408+0000", 2024-09-16T05:21:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019186240000000001 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:47.838442+0000", 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4264000000000003e-05 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:7000 2.2e e701/278 2:75e143e9:::smithi20457400-17:head v 701'652, mlcod=699'648)", 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:20:50.820810+0000", 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.905868362, 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0054030179999999999, 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.820810+0000", 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.820810+0000", 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.820810+0000", 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7999999999999995e-07 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.823932+0000", 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031213959999999998 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.823948+0000", 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6395000000000001e-05 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.823954+0000", 2024-09-16T05:21:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7529999999999998e-06 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.823996+0000", 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2027999999999998e-05 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.824015+0000", 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8967e-05 2024-09-16T05:21:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.826152+0000", 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0021370429999999999 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:20:50.826213+0000", 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0755999999999999e-05 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:199 2.1e e714/291 ondisk, result = 0)", 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 14.914469227, 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222874399, 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:21:18.850 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2099999999999997e-07 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812213+0000", 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3859999999999999e-06 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034914+0000", 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222700804 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034920+0000", 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6579999999999996e-06 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034968+0000", 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7833000000000001e-05 2024-09-16T05:21:18.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034975+0000", 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0400000000000004e-06 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.035083+0000", 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108157 2024-09-16T05:21:18.854 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:7200 2.29 e728/253 2:9662c29c:::smithi20457400-9:head v 728'1124, mlcod=723'1120)", 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:18.186932+0000", 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 0.53974647200000003, 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057431619999999996, 2024-09-16T05:21:18.855 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:21:18.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:21:18.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:18.856 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:21:18.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:21:18.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.186932+0000", 2024-09-16T05:21:18.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:21:18.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.186932+0000", 2024-09-16T05:21:18.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:21:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:21:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.186932+0000", 2024-09-16T05:21:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1099999999999995e-07 2024-09-16T05:21:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:21:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.190681+0000", 2024-09-16T05:21:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0037491260000000002 2024-09-16T05:21:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:21:18.860 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.190706+0000", 2024-09-16T05:21:18.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4604e-05 2024-09-16T05:21:18.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.860 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:21:18.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.190711+0000", 2024-09-16T05:21:18.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0370000000000002e-06 2024-09-16T05:21:18.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.861 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:21:18.861 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.190757+0000", 2024-09-16T05:21:18.861 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6229000000000001e-05 2024-09-16T05:21:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:21:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.190774+0000", 2024-09-16T05:21:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7031e-05 2024-09-16T05:21:18.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.192637+0000", 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0018627170000000001 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:18.192675+0000", 2024-09-16T05:21:18.863 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7707000000000003e-05 2024-09-16T05:21:18.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.864 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:18.864 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:18.864 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:21:18.866 INFO:tasks.rados.rados.2.smithi204.stdout:2849: done (3 left) 2024-09-16T05:21:18.867 INFO:tasks.rados.rados.2.smithi204.stdout:2851: done (2 left) 2024-09-16T05:21:18.867 INFO:tasks.rados.rados.2.smithi204.stdout:2852: done (1 left) 2024-09-16T05:21:18.867 INFO:tasks.rados.rados.2.smithi204.stdout:2853: done (0 left) 2024-09-16T05:21:18.867 INFO:tasks.rados.rados.2.smithi204.stdout:2854: copy_from oid 34 from oid 11 current snap is 304 2024-09-16T05:21:18.867 INFO:tasks.rados.rados.2.smithi204.stdout:2855: delete oid 17 current snap is 304 2024-09-16T05:21:18.867 INFO:tasks.rados.rados.2.smithi204.stdout:2855: done (1 left) 2024-09-16T05:21:18.868 INFO:tasks.rados.rados.2.smithi204.stdout:2856: read oid 19 snap -1 2024-09-16T05:21:18.868 INFO:tasks.rados.rados.2.smithi204.stdout:2856: expect (ObjNum 929 snap 275 seq_num 929) 2024-09-16T05:21:18.868 INFO:tasks.rados.rados.2.smithi204.stdout:2857: snap_remove snap 300 2024-09-16T05:21:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:18.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:18 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:18.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:18 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:18.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:19.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:19.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:19.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:19.146 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:19.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:18 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:18 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:18 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:18 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:19.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:19.273 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:19.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:19.374 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:19.397 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:19.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:19.498 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:19.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:19.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:19.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:19.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:21:19.647 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:19.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:19.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:19.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:19.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:19.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:21:19.857 INFO:tasks.rados.rados.2.smithi204.stdout:2857: done (2 left) 2024-09-16T05:21:19.857 INFO:tasks.rados.rados.2.smithi204.stdout:2858: snap_remove snap 297 2024-09-16T05:21:19.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:19.875 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:19.895 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:19.896 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:19.896 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:19.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:19.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:19.933 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:19.933 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:19.954 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:19.954 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:19.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:19 smithi196 ceph-mon[30802]: pgmap v1254: 117 pgs: 117 active+clean; 302 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T05:21:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:19 smithi196 ceph-mon[30802]: osdmap e729: 12 total, 12 up, 12 in 2024-09-16T05:21:19.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4110367339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:19.992 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:19.992 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:19.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:20.022 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:20.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:20.030 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:20.030 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:20.069 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:20.069 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:20.100 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:20.100 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:20.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:20.130 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:20.130 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:20.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:19 smithi032 ceph-mon[27388]: pgmap v1254: 117 pgs: 117 active+clean; 302 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T05:21:20.130 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:19 smithi032 ceph-mon[27388]: osdmap e729: 12 total, 12 up, 12 in 2024-09-16T05:21:20.131 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4110367339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:20.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:20.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:20.151 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:20.151 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:20.166 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:19 smithi204 ceph-mon[30933]: pgmap v1254: 117 pgs: 117 active+clean; 302 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 0 B/s wr, 6 op/s 2024-09-16T05:21:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:19 smithi204 ceph-mon[30933]: osdmap e729: 12 total, 12 up, 12 in 2024-09-16T05:21:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4110367339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:20.173 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:20.174 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:20.203 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:20.203 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:20.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:20.270 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:20.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:20.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:20.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:20.396 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:20.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:20.482 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:20.482 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:20.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:20.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:20.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:20.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:20.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:20.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:20.769 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:20.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:20.812 INFO:tasks.rados.rados.2.smithi204.stdout:2858: done (2 left) 2024-09-16T05:21:20.812 INFO:tasks.rados.rados.2.smithi204.stdout:2859: rollback oid 7 current snap is 304 2024-09-16T05:21:20.812 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 7 to 292 2024-09-16T05:21:20.812 INFO:tasks.rados.rados.2.smithi204.stdout:2854: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:21:20.812 INFO:tasks.rados.rados.2.smithi204.stdout:2854: finishing copy_from to smithi20457400-34 2024-09-16T05:21:20.812 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1153 (ObjNum 972 snap 287 seq_num 972) dirty exists 2024-09-16T05:21:20.812 INFO:tasks.rados.rados.2.smithi204.stdout:2856: expect (ObjNum 929 snap 275 seq_num 929) 2024-09-16T05:21:20.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:20.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:20.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:20.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:21.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:21.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:21.021 INFO:tasks.rados.rados.2.smithi204.stdout:2859: finishing rollback tid 0 to smithi20457400-7 2024-09-16T05:21:21.021 INFO:tasks.rados.rados.2.smithi204.stdout:2859: finishing rollback tid 1 to smithi20457400-7 2024-09-16T05:21:21.021 INFO:tasks.rados.rados.2.smithi204.stdout:2859: finishing rollback tid 2 to smithi20457400-7 2024-09-16T05:21:21.021 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1122 (ObjNum 789 snap 229 seq_num 789) dirty exists 2024-09-16T05:21:21.021 INFO:tasks.rados.rados.2.smithi204.stdout:2854: done (2 left) 2024-09-16T05:21:21.022 INFO:tasks.rados.rados.2.smithi204.stdout:2856: done (1 left) 2024-09-16T05:21:21.022 INFO:tasks.rados.rados.2.smithi204.stdout:2859: done (0 left) 2024-09-16T05:21:21.022 INFO:tasks.rados.rados.2.smithi204.stdout:2860: write (excl) oid 5 current snap is 304 2024-09-16T05:21:21.022 INFO:tasks.rados.rados.2.smithi204.stdout:2860: seq_num 1009 ranges {558263=484837,1627743=706008} 2024-09-16T05:21:21.026 INFO:tasks.rados.rados.2.smithi204.stdout:2860: writing smithi20457400-5 from 558263 to 1043100 tid 1 2024-09-16T05:21:21.034 INFO:tasks.rados.rados.2.smithi204.stdout:2860: writing smithi20457400-5 from 1627743 to 2333751 tid 2 2024-09-16T05:21:21.034 INFO:tasks.rados.rados.2.smithi204.stdout:2861: read oid 31 snap 291 2024-09-16T05:21:21.034 INFO:tasks.rados.rados.2.smithi204.stdout:2861: expect deleted 2024-09-16T05:21:21.034 INFO:tasks.rados.rados.2.smithi204.stdout:2862: read oid 22 snap -1 2024-09-16T05:21:21.034 INFO:tasks.rados.rados.2.smithi204.stdout:2862: expect (ObjNum 926 snap 275 seq_num 926) 2024-09-16T05:21:21.034 INFO:tasks.rados.rados.2.smithi204.stdout:2863: snap_remove snap 290 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]: dispatch 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]: dispatch 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]: dispatch 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]: dispatch 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]: dispatch 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]: dispatch 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]: dispatch 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]: dispatch 2024-09-16T05:21:21.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: osdmap e730: 12 total, 12 up, 12 in 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]: dispatch 2024-09-16T05:21:21.076 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]: dispatch 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]: dispatch 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]: dispatch 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]: dispatch 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]: dispatch 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]: dispatch 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]': finished 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]': finished 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]': finished 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]': finished 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]': finished 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]': finished 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]': finished 2024-09-16T05:21:21.077 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]': finished 2024-09-16T05:21:21.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]': finished 2024-09-16T05:21:21.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]': finished 2024-09-16T05:21:21.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]': finished 2024-09-16T05:21:21.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]': finished 2024-09-16T05:21:21.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:20 smithi032 ceph-mon[27388]: osdmap e731: 12 total, 12 up, 12 in 2024-09-16T05:21:21.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]: dispatch 2024-09-16T05:21:21.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: osdmap e730: 12 total, 12 up, 12 in 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]: dispatch 2024-09-16T05:21:21.141 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]: dispatch 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]: dispatch 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]': finished 2024-09-16T05:21:21.142 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:20 smithi196 ceph-mon[30802]: osdmap e731: 12 total, 12 up, 12 in 2024-09-16T05:21:21.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:21.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]: dispatch 2024-09-16T05:21:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]: dispatch 2024-09-16T05:21:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]: dispatch 2024-09-16T05:21:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]: dispatch 2024-09-16T05:21:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]: dispatch 2024-09-16T05:21:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]: dispatch 2024-09-16T05:21:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]: dispatch 2024-09-16T05:21:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: osdmap e730: 12 total, 12 up, 12 in 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]: dispatch 2024-09-16T05:21:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]: dispatch 2024-09-16T05:21:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]: dispatch 2024-09-16T05:21:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]: dispatch 2024-09-16T05:21:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]: dispatch 2024-09-16T05:21:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10940182872964282]}]': finished 2024-09-16T05:21:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07613140388007014]}]': finished 2024-09-16T05:21:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12102568345252246]}]': finished 2024-09-16T05:21:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09258657376476608]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08874834580033386]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08786986032681823]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08264259720360548]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.0612300373040673]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08241859521079685]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07616319213065391]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07501757782928738]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09431069108618541]}]': finished 2024-09-16T05:21:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:20 smithi204 ceph-mon[30933]: osdmap e731: 12 total, 12 up, 12 in 2024-09-16T05:21:21.245 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:21.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:21.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:21.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:21:21.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:21.390 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:21.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:21.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:21.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:21.609 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:21.616 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:21.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:21.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:21.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:21.761 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:21.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2863: done (3 left) 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2864: copy_from oid 3 from oid 10 current snap is 304 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2865: delete oid 18 current snap is 304 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2860: finishing write tid 1 to smithi20457400-5 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2860: finishing write tid 2 to smithi20457400-5 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2860: finishing write tid 3 to smithi20457400-5 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2860: finishing write tid 4 to smithi20457400-5 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 617 (ObjNum 1009 snap 304 seq_num 1009) dirty exists 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2860: left oid 5 (ObjNum 1009 snap 304 seq_num 1009) 2024-09-16T05:21:21.815 INFO:tasks.rados.rados.2.smithi204.stdout:2862: expect (ObjNum 926 snap 275 seq_num 926) 2024-09-16T05:21:21.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:21.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:21.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:21 smithi204 ceph-mon[30933]: pgmap v1257: 117 pgs: 117 active+clean; 302 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T05:21:21.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2551523558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:21:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:21 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2551523558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:21:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:21 smithi204 ceph-mon[30933]: osdmap e732: 12 total, 12 up, 12 in 2024-09-16T05:21:21.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:21 smithi196 ceph-mon[30802]: pgmap v1257: 117 pgs: 117 active+clean; 302 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T05:21:21.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2551523558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:21:21.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:21 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:21.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2551523558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:21:21.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:21 smithi196 ceph-mon[30802]: osdmap e732: 12 total, 12 up, 12 in 2024-09-16T05:21:21.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:22.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:22.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:22.032 INFO:tasks.rados.rados.2.smithi204.stdout:2864: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:21:22.032 INFO:tasks.rados.rados.2.smithi204.stdout:2864: finishing copy_from to smithi20457400-3 2024-09-16T05:21:22.032 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1085 (ObjNum 894 snap 262 seq_num 894) dirty exists 2024-09-16T05:21:22.032 INFO:tasks.rados.rados.2.smithi204.stdout:2860: done (4 left) 2024-09-16T05:21:22.032 INFO:tasks.rados.rados.2.smithi204.stdout:2861: done (3 left) 2024-09-16T05:21:22.033 INFO:tasks.rados.rados.2.smithi204.stdout:2862: done (2 left) 2024-09-16T05:21:22.033 INFO:tasks.rados.rados.2.smithi204.stdout:2864: done (1 left) 2024-09-16T05:21:22.033 INFO:tasks.rados.rados.2.smithi204.stdout:2865: done (0 left) 2024-09-16T05:21:22.033 INFO:tasks.rados.rados.2.smithi204.stdout:2866: snap_remove snap 262 2024-09-16T05:21:22.081 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:22.081 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:22.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:22.115 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:21:22.115 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:21:22.115 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T05:21:22.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:21:22.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:22.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:22.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:21 smithi032 ceph-mon[27388]: pgmap v1257: 117 pgs: 117 active+clean; 302 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 0 B/s wr, 9 op/s 2024-09-16T05:21:22.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2551523558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:21:22.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:21 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:22.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2551523558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:21:22.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:21 smithi032 ceph-mon[27388]: osdmap e732: 12 total, 12 up, 12 in 2024-09-16T05:21:22.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:22.258 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:22.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:22.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:22.381 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:22.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:22.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:22.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:22.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:22.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:22.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:22.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:22.705 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:22.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:22.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:22.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:22.813 INFO:tasks.rados.rados.2.smithi204.stdout:2866: done (0 left) 2024-09-16T05:21:22.813 INFO:tasks.rados.rados.2.smithi204.stdout:2867: snap_remove snap 294 2024-09-16T05:21:22.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:22.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:22.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:22.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:21:22.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:22.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:23.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:23.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:22 smithi032 ceph-mon[27388]: 2.3c scrub starts 2024-09-16T05:21:23.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:22 smithi032 ceph-mon[27388]: 2.3c scrub ok 2024-09-16T05:21:23.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:22 smithi032 ceph-mon[27388]: 2.4f scrub starts 2024-09-16T05:21:23.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:22 smithi032 ceph-mon[27388]: 2.4f scrub ok 2024-09-16T05:21:23.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:22 smithi032 ceph-mon[27388]: osdmap e733: 12 total, 12 up, 12 in 2024-09-16T05:21:23.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:22 smithi204 ceph-mon[30933]: 2.3c scrub starts 2024-09-16T05:21:23.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:22 smithi204 ceph-mon[30933]: 2.3c scrub ok 2024-09-16T05:21:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:22 smithi204 ceph-mon[30933]: 2.4f scrub starts 2024-09-16T05:21:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:22 smithi204 ceph-mon[30933]: 2.4f scrub ok 2024-09-16T05:21:23.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:22 smithi204 ceph-mon[30933]: osdmap e733: 12 total, 12 up, 12 in 2024-09-16T05:21:23.194 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:23.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:23.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:22 smithi196 ceph-mon[30802]: 2.3c scrub starts 2024-09-16T05:21:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:22 smithi196 ceph-mon[30802]: 2.3c scrub ok 2024-09-16T05:21:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:22 smithi196 ceph-mon[30802]: 2.4f scrub starts 2024-09-16T05:21:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:22 smithi196 ceph-mon[30802]: 2.4f scrub ok 2024-09-16T05:21:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:22 smithi196 ceph-mon[30802]: osdmap e733: 12 total, 12 up, 12 in 2024-09-16T05:21:23.246 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:23.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:23.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:23.397 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:21:23.399 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:21:23.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:23.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:21:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:23.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:23.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:23.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:23.653 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:23.653 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:23.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:23.741 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:23.815 INFO:tasks.rados.rados.2.smithi204.stdout:2867: done (0 left) 2024-09-16T05:21:23.815 INFO:tasks.rados.rados.2.smithi204.stdout:2868: snap_create 2024-09-16T05:21:23.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:23.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:23.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:23.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:23 smithi196 ceph-mon[30802]: pgmap v1260: 117 pgs: 1 remapped+peering, 1 active+clean+snaptrim, 115 active+clean; 312 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-16T05:21:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:23 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:21:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:23 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:21:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:23 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:21:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:23 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:21:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:23 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:21:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:23 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:21:23.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:23 smithi196 ceph-mon[30802]: osdmap e734: 12 total, 12 up, 12 in 2024-09-16T05:21:23.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:23.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:24.038 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:23 smithi032 ceph-mon[27388]: pgmap v1260: 117 pgs: 1 remapped+peering, 1 active+clean+snaptrim, 115 active+clean; 312 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-16T05:21:24.038 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:23 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:21:24.038 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:23 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:21:24.038 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:23 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:21:24.038 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:23 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:21:24.038 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:23 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:21:24.038 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:23 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:21:24.038 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:23 smithi032 ceph-mon[27388]: osdmap e734: 12 total, 12 up, 12 in 2024-09-16T05:21:24.065 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1261,"stamp":"2024-09-16T05:21:22.810332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331056592,"num_objects":158,"num_object_clones":105,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":10,"num_read":3643,"num_read_kb":3588116,"num_write":2665,"num_write_kb":1061648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":89317650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30036,"ondisk_log_size":30036,"up":234,"acting":234,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":234,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1033024,"kb_used_data":499408,"kb_used_omap":27,"kb_used_meta":533284,"kb_avail":1123908800,"statfs":{"total":1151940427776,"available":1150882611200,"internally_reserved":0,"allocated":511393792,"data_stored":567553429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":546082983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13977483,"num_objects":8,"num_object_clones":8,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":47,"num_read_kb":59612,"num_write":24,"num_write_kb":11452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.958370"},"pg_stats":[{"pgid":"2.55","version":"433'507","reported_seq":3541,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-09-16T05:21:15.971681+0000","last_change":"2024-09-16T05:21:12.948493+0000","last_active":"2024-09-16T05:21:15.971681+0000","last_peered":"2024-09-16T05:21:15.971681+0000","last_clean":"2024-09-16T05:21:15.971681+0000","last_became_active":"2024-09-16T05:16:30.734471+0000","last_became_peered":"2024-09-16T05:16:30.734471+0000","last_unstale":"2024-09-16T05:21:15.971681+0000","last_undegraded":"2024-09-16T05:21:15.971681+0000","last_fullsized":"2024-09-16T05:21:15.971681+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:20:01.826054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:01.826054+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.334273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025008830000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.54","version":"718'411","reported_seq":3130,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.810931+0000","last_change":"2024-09-16T05:21:17.811208+0000","last_active":"2024-09-16T05:21:17.810931+0000","last_peered":"2024-09-16T05:21:17.810931+0000","last_clean":"2024-09-16T05:21:17.810931+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:21:17.810931+0000","last_undegraded":"2024-09-16T05:21:17.810931+0000","last_fullsized":"2024-09-16T05:21:17.810931+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"686'398","last_scrub_stamp":"2024-09-16T05:20:43.585524+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:20:43.585524+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.820658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014127440000000001,"stat_sum":{"num_bytes":2504474,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":68,"num_read_kb":63621,"num_write":43,"num_write_kb":13668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":1474,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.820487+0000","last_change":"2024-09-16T05:21:20.820487+0000","last_active":"2024-09-16T05:21:20.820487+0000","last_peered":"2024-09-16T05:21:20.820487+0000","last_clean":"2024-09-16T05:21:20.820487+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:21:20.820487+0000","last_undegraded":"2024-09-16T05:21:20.820487+0000","last_fullsized":"2024-09-16T05:21:20.820487+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:20:40.312020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.312020+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:04.170440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00038077400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":1464,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:19.858408+0000","last_change":"2024-09-16T05:21:19.858408+0000","last_active":"2024-09-16T05:21:19.858408+0000","last_peered":"2024-09-16T05:21:19.858408+0000","last_clean":"2024-09-16T05:21:19.858408+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:21:19.858408+0000","last_undegraded":"2024-09-16T05:21:19.858408+0000","last_fullsized":"2024-09-16T05:21:19.858408+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:18.588272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.588272+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:38.900708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081176399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":1541,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815397+0000","last_change":"2024-09-16T05:21:20.815646+0000","last_active":"2024-09-16T05:21:20.815397+0000","last_peered":"2024-09-16T05:21:20.815397+0000","last_clean":"2024-09-16T05:21:20.815397+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:21:20.815397+0000","last_undegraded":"2024-09-16T05:21:20.815397+0000","last_fullsized":"2024-09-16T05:21:20.815397+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:20:59.668844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:59.668844+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:10.348512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077903200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":1447,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-09-16T05:21:16.823648+0000","last_change":"2024-09-16T05:21:16.823881+0000","last_active":"2024-09-16T05:21:16.823648+0000","last_peered":"2024-09-16T05:21:16.823648+0000","last_clean":"2024-09-16T05:21:16.823648+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:21:16.823648+0000","last_undegraded":"2024-09-16T05:21:16.823648+0000","last_fullsized":"2024-09-16T05:21:16.823648+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:20:59.312163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:59.312163+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:15.036969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044243199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":1984,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852797+0000","last_change":"2024-09-16T05:21:18.007443+0000","last_active":"2024-09-16T05:21:18.852797+0000","last_peered":"2024-09-16T05:21:18.852797+0000","last_clean":"2024-09-16T05:21:18.852797+0000","last_became_active":"2024-09-16T05:21:10.817392+0000","last_became_peered":"2024-09-16T05:21:10.817392+0000","last_unstale":"2024-09-16T05:21:18.852797+0000","last_undegraded":"2024-09-16T05:21:18.852797+0000","last_fullsized":"2024-09-16T05:21:18.852797+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:19:48.302807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.302807+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098885800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.4e","version":"272'232","reported_seq":3242,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.817757+0000","last_change":"2024-09-16T05:21:20.818468+0000","last_active":"2024-09-16T05:21:20.817757+0000","last_peered":"2024-09-16T05:21:20.817757+0000","last_clean":"2024-09-16T05:21:20.817757+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:21:20.817757+0000","last_undegraded":"2024-09-16T05:21:20.817757+0000","last_fullsized":"2024-09-16T05:21:20.817757+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:21:07.065851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:07.065851+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:25.232572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020030529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.4d","version":"728'640","reported_seq":3787,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.814517+0000","last_change":"2024-09-16T05:21:17.814689+0000","last_active":"2024-09-16T05:21:17.814517+0000","last_peered":"2024-09-16T05:21:17.814517+0000","last_clean":"2024-09-16T05:21:17.814517+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:21:17.814517+0000","last_undegraded":"2024-09-16T05:21:17.814517+0000","last_fullsized":"2024-09-16T05:21:17.814517+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"665'626","last_scrub_stamp":"2024-09-16T05:20:16.492166+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.492166+0000","objects_scrubbed":4,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0049834090000000003,"stat_sum":{"num_bytes":3563282,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":46,"num_read_kb":38966,"num_write":59,"num_write_kb":21533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.4c","version":"709'374","reported_seq":3479,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.820256+0000","last_change":"2024-09-16T05:21:20.820408+0000","last_active":"2024-09-16T05:21:20.820256+0000","last_peered":"2024-09-16T05:21:20.820256+0000","last_clean":"2024-09-16T05:21:20.820256+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:21:20.820256+0000","last_undegraded":"2024-09-16T05:21:20.820256+0000","last_fullsized":"2024-09-16T05:21:20.820256+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"666'345","last_scrub_stamp":"2024-09-16T05:20:17.450506+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.450506+0000","objects_scrubbed":1,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:26.549033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041704510000000004,"stat_sum":{"num_bytes":5239557,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":65588,"num_write":33,"num_write_kb":9268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":3262,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.811266+0000","last_change":"2024-09-16T05:21:17.811431+0000","last_active":"2024-09-16T05:21:17.811266+0000","last_peered":"2024-09-16T05:21:17.811266+0000","last_clean":"2024-09-16T05:21:17.811266+0000","last_became_active":"2024-09-16T05:17:21.741180+0000","last_became_peered":"2024-09-16T05:17:21.741180+0000","last_unstale":"2024-09-16T05:21:17.811266+0000","last_undegraded":"2024-09-16T05:21:17.811266+0000","last_fullsized":"2024-09-16T05:21:17.811266+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:20:40.593078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.593078+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:46.387072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015801389999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.4a","version":"731'594","reported_seq":3665,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.820032+0000","last_change":"2024-09-16T05:21:20.820198+0000","last_active":"2024-09-16T05:21:20.820032+0000","last_peered":"2024-09-16T05:21:20.820032+0000","last_clean":"2024-09-16T05:21:20.820032+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:21:20.820032+0000","last_undegraded":"2024-09-16T05:21:20.820032+0000","last_fullsized":"2024-09-16T05:21:20.820032+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"620'557","last_scrub_stamp":"2024-09-16T05:19:54.555606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.555606+0000","objects_scrubbed":1,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052647830000000003,"stat_sum":{"num_bytes":5283159,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55011,"num_write":44,"num_write_kb":20751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":3311,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.816577+0000","last_change":"2024-09-16T05:21:20.816827+0000","last_active":"2024-09-16T05:21:20.816577+0000","last_peered":"2024-09-16T05:21:20.816577+0000","last_clean":"2024-09-16T05:21:20.816577+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:21:20.816577+0000","last_undegraded":"2024-09-16T05:21:20.816577+0000","last_fullsized":"2024-09-16T05:21:20.816577+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:20.094577+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.094577+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:36.467478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055020999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.48","version":"723'369","reported_seq":3434,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.105235+0000","last_change":"2024-09-16T05:21:20.106025+0000","last_active":"2024-09-16T05:21:20.105235+0000","last_peered":"2024-09-16T05:21:20.105235+0000","last_clean":"2024-09-16T05:21:20.105235+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:21:20.105235+0000","last_undegraded":"2024-09-16T05:21:20.105235+0000","last_fullsized":"2024-09-16T05:21:20.105235+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"705'353","last_scrub_stamp":"2024-09-16T05:20:58.009284+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:20:58.009284+0000","objects_scrubbed":4,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:11.293044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010142370000000001,"stat_sum":{"num_bytes":9565472,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":79,"num_read_kb":75759,"num_write":56,"num_write_kb":25106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":2090,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:19.857914+0000","last_change":"2024-09-16T05:21:19.858125+0000","last_active":"2024-09-16T05:21:19.857914+0000","last_peered":"2024-09-16T05:21:19.857914+0000","last_clean":"2024-09-16T05:21:19.857914+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:21:19.857914+0000","last_undegraded":"2024-09-16T05:21:19.857914+0000","last_fullsized":"2024-09-16T05:21:19.857914+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:01.976068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:01.976068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:10.001387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00045007599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":305,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.810570+0000","last_change":"2024-09-16T05:21:17.810861+0000","last_active":"2024-09-16T05:21:17.810570+0000","last_peered":"2024-09-16T05:21:17.810570+0000","last_clean":"2024-09-16T05:21:17.810570+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:21:17.810570+0000","last_undegraded":"2024-09-16T05:21:17.810570+0000","last_fullsized":"2024-09-16T05:21:17.810570+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:21:05.606142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:05.606142+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:33.430801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001085241,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":2587,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852854+0000","last_change":"2024-09-16T05:21:18.006911+0000","last_active":"2024-09-16T05:21:18.852854+0000","last_peered":"2024-09-16T05:21:18.852854+0000","last_clean":"2024-09-16T05:21:18.852854+0000","last_became_active":"2024-09-16T05:21:10.818112+0000","last_became_peered":"2024-09-16T05:21:10.818112+0000","last_unstale":"2024-09-16T05:21:18.852854+0000","last_undegraded":"2024-09-16T05:21:18.852854+0000","last_fullsized":"2024-09-16T05:21:18.852854+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:20:42.442340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:42.442340+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.697298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086325699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":73,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852640+0000","last_change":"2024-09-16T05:21:18.007380+0000","last_active":"2024-09-16T05:21:18.852640+0000","last_peered":"2024-09-16T05:21:18.852640+0000","last_clean":"2024-09-16T05:21:18.852640+0000","last_became_active":"2024-09-16T05:21:07.820306+0000","last_became_peered":"2024-09-16T05:21:07.820306+0000","last_unstale":"2024-09-16T05:21:18.852640+0000","last_undegraded":"2024-09-16T05:21:18.852640+0000","last_fullsized":"2024-09-16T05:21:18.852640+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_clean_scrub_stamp":"2024-09-16T05:21:08.436254+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:20.248204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001074006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":2087,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.816658+0000","last_change":"2024-09-16T05:21:21.816866+0000","last_active":"2024-09-16T05:21:21.816658+0000","last_peered":"2024-09-16T05:21:21.816658+0000","last_clean":"2024-09-16T05:21:21.816658+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:21:21.816658+0000","last_undegraded":"2024-09-16T05:21:21.816658+0000","last_fullsized":"2024-09-16T05:21:21.816658+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:20:38.464664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.464664+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:02.351285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355932,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":2048,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853233+0000","last_change":"2024-09-16T05:21:17.812795+0000","last_active":"2024-09-16T05:21:18.853233+0000","last_peered":"2024-09-16T05:21:18.853233+0000","last_clean":"2024-09-16T05:21:18.853233+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:21:18.853233+0000","last_undegraded":"2024-09-16T05:21:18.853233+0000","last_fullsized":"2024-09-16T05:21:18.853233+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:21:09.465647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:09.465647+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:17.299035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001843892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":929,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:19.858265+0000","last_change":"2024-09-16T05:21:19.858493+0000","last_active":"2024-09-16T05:21:19.858265+0000","last_peered":"2024-09-16T05:21:19.858265+0000","last_clean":"2024-09-16T05:21:19.858265+0000","last_became_active":"2024-09-16T05:18:19.768547+0000","last_became_peered":"2024-09-16T05:18:19.768547+0000","last_unstale":"2024-09-16T05:21:19.858265+0000","last_undegraded":"2024-09-16T05:21:19.858265+0000","last_fullsized":"2024-09-16T05:21:19.858265+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:21:05.881934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:05.881934+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:27.268061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048040000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":629,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815915+0000","last_change":"2024-09-16T05:21:20.816227+0000","last_active":"2024-09-16T05:21:20.815915+0000","last_peered":"2024-09-16T05:21:20.815915+0000","last_clean":"2024-09-16T05:21:20.815915+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:21:20.815915+0000","last_undegraded":"2024-09-16T05:21:20.815915+0000","last_fullsized":"2024-09-16T05:21:20.815915+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:20:59.988307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:59.988307+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:16.338037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063972699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.3f","version":"732'468","reported_seq":3581,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.819275+0000","last_change":"2024-09-16T05:21:21.819465+0000","last_active":"2024-09-16T05:21:21.819275+0000","last_peered":"2024-09-16T05:21:21.819275+0000","last_clean":"2024-09-16T05:21:21.819275+0000","last_became_active":"2024-09-16T05:13:45.685060+0000","last_became_peered":"2024-09-16T05:13:45.685060+0000","last_unstale":"2024-09-16T05:21:21.819275+0000","last_undegraded":"2024-09-16T05:21:21.819275+0000","last_fullsized":"2024-09-16T05:21:21.819275+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'415","last_scrub_stamp":"2024-09-16T05:19:58.393481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:58.393481+0000","objects_scrubbed":2,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0036523570000000002,"stat_sum":{"num_bytes":7245769,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":95,"num_read_kb":82275,"num_write":55,"num_write_kb":18010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.3e","version":"732'518","reported_seq":3592,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.817360+0000","last_change":"2024-09-16T05:21:21.817560+0000","last_active":"2024-09-16T05:21:21.817360+0000","last_peered":"2024-09-16T05:21:21.817360+0000","last_clean":"2024-09-16T05:21:21.817360+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:21:21.817360+0000","last_undegraded":"2024-09-16T05:21:21.817360+0000","last_fullsized":"2024-09-16T05:21:21.817360+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"625'487","last_scrub_stamp":"2024-09-16T05:20:22.404334+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:20:22.404334+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.163192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0052854490000000002,"stat_sum":{"num_bytes":5557284,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":35563,"num_write":59,"num_write_kb":28565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":1525,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852916+0000","last_change":"2024-09-16T05:21:18.007245+0000","last_active":"2024-09-16T05:21:18.852916+0000","last_peered":"2024-09-16T05:21:18.852916+0000","last_clean":"2024-09-16T05:21:18.852916+0000","last_became_active":"2024-09-16T05:21:10.817387+0000","last_became_peered":"2024-09-16T05:21:10.817387+0000","last_unstale":"2024-09-16T05:21:18.852916+0000","last_undegraded":"2024-09-16T05:21:18.852916+0000","last_fullsized":"2024-09-16T05:21:18.852916+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:21:05.402898+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:19:14.912091+0000","last_clean_scrub_stamp":"2024-09-16T05:21:05.402898+0000","objects_scrubbed":9,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095047700000000003,"stat_sum":{"num_bytes":14300236,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":17,"num_read_kb":11647,"num_write":24,"num_write_kb":7654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":19591132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.3c","version":"713'757","reported_seq":4115,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.825079+0000","last_change":"2024-09-16T05:21:21.825105+0000","last_active":"2024-09-16T05:21:21.825079+0000","last_peered":"2024-09-16T05:21:21.825079+0000","last_clean":"2024-09-16T05:21:21.825079+0000","last_became_active":"2024-09-16T05:21:07.828778+0000","last_became_peered":"2024-09-16T05:21:07.828778+0000","last_unstale":"2024-09-16T05:21:21.825079+0000","last_undegraded":"2024-09-16T05:21:21.825079+0000","last_fullsized":"2024-09-16T05:21:21.825079+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"713'757","last_scrub_stamp":"2024-09-16T05:21:21.825033+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:21:21.825033+0000","objects_scrubbed":5,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:49.840663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001297056,"stat_sum":{"num_bytes":12386647,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":183,"num_read_kb":246008,"num_write":85,"num_write_kb":33383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":14713263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.3b","version":"728'448","reported_seq":3542,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.816999+0000","last_change":"2024-09-16T05:21:20.816999+0000","last_active":"2024-09-16T05:21:20.816999+0000","last_peered":"2024-09-16T05:21:20.816999+0000","last_clean":"2024-09-16T05:21:20.816999+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:21:20.816999+0000","last_undegraded":"2024-09-16T05:21:20.816999+0000","last_fullsized":"2024-09-16T05:21:20.816999+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"645'416","last_scrub_stamp":"2024-09-16T05:19:55.767476+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:19:55.767476+0000","objects_scrubbed":2,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069382099999999996,"stat_sum":{"num_bytes":7281818,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":65,"num_read_kb":64613,"num_write":52,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.3a","version":"730'602","reported_seq":3648,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.107352+0000","last_change":"2024-09-16T05:21:20.107619+0000","last_active":"2024-09-16T05:21:20.107352+0000","last_peered":"2024-09-16T05:21:20.107352+0000","last_clean":"2024-09-16T05:21:20.107352+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:21:20.107352+0000","last_undegraded":"2024-09-16T05:21:20.107352+0000","last_fullsized":"2024-09-16T05:21:20.107352+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"667'593","last_scrub_stamp":"2024-09-16T05:20:20.620494+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.620494+0000","objects_scrubbed":2,"log_size":602,"log_dups_size":0,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:49.430446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025576380000000001,"stat_sum":{"num_bytes":7664685,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":56,"num_read_kb":46839,"num_write":63,"num_write_kb":31529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":2158,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.106084+0000","last_change":"2024-09-16T05:21:20.106515+0000","last_active":"2024-09-16T05:21:20.106084+0000","last_peered":"2024-09-16T05:21:20.106084+0000","last_clean":"2024-09-16T05:21:20.106084+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:21:20.106084+0000","last_undegraded":"2024-09-16T05:21:20.106084+0000","last_fullsized":"2024-09-16T05:21:20.106084+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:19:49.888753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:49.888753+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014210830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":2161,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853167+0000","last_change":"2024-09-16T05:21:17.811020+0000","last_active":"2024-09-16T05:21:18.853167+0000","last_peered":"2024-09-16T05:21:18.853167+0000","last_clean":"2024-09-16T05:21:18.853167+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:21:18.853167+0000","last_undegraded":"2024-09-16T05:21:18.853167+0000","last_fullsized":"2024-09-16T05:21:18.853167+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:20:20.396645+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.396645+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.645599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077296099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":1030,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.816402+0000","last_change":"2024-09-16T05:21:20.817008+0000","last_active":"2024-09-16T05:21:20.816402+0000","last_peered":"2024-09-16T05:21:20.816402+0000","last_clean":"2024-09-16T05:21:20.816402+0000","last_became_active":"2024-09-16T05:17:59.866037+0000","last_became_peered":"2024-09-16T05:17:59.866037+0000","last_unstale":"2024-09-16T05:21:20.816402+0000","last_undegraded":"2024-09-16T05:21:20.816402+0000","last_fullsized":"2024-09-16T05:21:20.816402+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:00.633624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:21:00.633624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:13.620989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001247026,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":65,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852607+0000","last_change":"2024-09-16T05:21:18.006674+0000","last_active":"2024-09-16T05:21:18.852607+0000","last_peered":"2024-09-16T05:21:18.852607+0000","last_clean":"2024-09-16T05:21:18.852607+0000","last_became_active":"2024-09-16T05:21:07.819657+0000","last_became_peered":"2024-09-16T05:21:07.819657+0000","last_unstale":"2024-09-16T05:21:18.852607+0000","last_undegraded":"2024-09-16T05:21:18.852607+0000","last_fullsized":"2024-09-16T05:21:18.852607+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:20:02.294024+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.294024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067128999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.35","version":"723'820","reported_seq":3970,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852611+0000","last_change":"2024-09-16T05:21:18.006735+0000","last_active":"2024-09-16T05:21:18.852611+0000","last_peered":"2024-09-16T05:21:18.852611+0000","last_clean":"2024-09-16T05:21:18.852611+0000","last_became_active":"2024-09-16T05:21:10.815197+0000","last_became_peered":"2024-09-16T05:21:10.815197+0000","last_unstale":"2024-09-16T05:21:18.852611+0000","last_undegraded":"2024-09-16T05:21:18.852611+0000","last_fullsized":"2024-09-16T05:21:18.852611+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"686'768","last_scrub_stamp":"2024-09-16T05:20:38.441947+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.441947+0000","objects_scrubbed":7,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:03.874016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070509800000000003,"stat_sum":{"num_bytes":20791292,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":140,"num_read_kb":130417,"num_write":104,"num_write_kb":48025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":17054243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.34","version":"731'660","reported_seq":3359,"reported_epoch":731,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T05:21:20.822424+0000","last_change":"2024-09-16T05:21:20.820821+0000","last_active":"2024-09-16T05:21:20.822424+0000","last_peered":"2024-09-16T05:21:20.822424+0000","last_clean":"2024-09-16T05:21:20.822424+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:21:20.822424+0000","last_undegraded":"2024-09-16T05:21:20.822424+0000","last_fullsized":"2024-09-16T05:21:20.822424+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"686'611","last_scrub_stamp":"2024-09-16T05:20:38.002711+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.002711+0000","objects_scrubbed":7,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:59.386164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00066555,"stat_sum":{"num_bytes":10670328,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":181,"num_read_kb":210064,"num_write":110,"num_write_kb":41100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":2271,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.820124+0000","last_change":"2024-09-16T05:21:21.820413+0000","last_active":"2024-09-16T05:21:21.820124+0000","last_peered":"2024-09-16T05:21:21.820124+0000","last_clean":"2024-09-16T05:21:21.820124+0000","last_became_active":"2024-09-16T05:21:07.818018+0000","last_became_peered":"2024-09-16T05:21:07.818018+0000","last_unstale":"2024-09-16T05:21:21.820124+0000","last_undegraded":"2024-09-16T05:21:21.820124+0000","last_fullsized":"2024-09-16T05:21:21.820124+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:20:59.791825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:59.791825+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:21.587605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040062350000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.32","version":"715'257","reported_seq":2510,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.813438+0000","last_change":"2024-09-16T05:21:21.813697+0000","last_active":"2024-09-16T05:21:21.813438+0000","last_peered":"2024-09-16T05:21:21.813438+0000","last_clean":"2024-09-16T05:21:21.813438+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:21:21.813438+0000","last_undegraded":"2024-09-16T05:21:21.813438+0000","last_fullsized":"2024-09-16T05:21:21.813438+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"671'227","last_scrub_stamp":"2024-09-16T05:20:23.405472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.405472+0000","objects_scrubbed":2,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:42.296890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001359994,"stat_sum":{"num_bytes":7376174,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":101971,"num_write":48,"num_write_kb":21789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.31","version":"709'300","reported_seq":3124,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:19.858471+0000","last_change":"2024-09-16T05:21:19.858662+0000","last_active":"2024-09-16T05:21:19.858471+0000","last_peered":"2024-09-16T05:21:19.858471+0000","last_clean":"2024-09-16T05:21:19.858471+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:21:19.858471+0000","last_undegraded":"2024-09-16T05:21:19.858471+0000","last_fullsized":"2024-09-16T05:21:19.858471+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'268","last_scrub_stamp":"2024-09-16T05:20:03.547514+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.547514+0000","objects_scrubbed":2,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:24.013685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000494353,"stat_sum":{"num_bytes":5462704,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":119772,"num_write":43,"num_write_kb":20308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":1010,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815006+0000","last_change":"2024-09-16T05:21:20.815249+0000","last_active":"2024-09-16T05:21:20.815006+0000","last_peered":"2024-09-16T05:21:20.815006+0000","last_clean":"2024-09-16T05:21:20.815006+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:21:20.815006+0000","last_undegraded":"2024-09-16T05:21:20.815006+0000","last_fullsized":"2024-09-16T05:21:20.815006+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:20:16.443442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.443442+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:46.072319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061525999999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":1098,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815234+0000","last_change":"2024-09-16T05:20:18.091859+0000","last_active":"2024-09-16T05:21:20.815234+0000","last_peered":"2024-09-16T05:21:20.815234+0000","last_clean":"2024-09-16T05:21:20.815234+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:21:20.815234+0000","last_undegraded":"2024-09-16T05:21:20.815234+0000","last_fullsized":"2024-09-16T05:21:20.815234+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:18.091747+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.091747+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.362844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":2814,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.818648+0000","last_change":"2024-09-16T05:21:21.818912+0000","last_active":"2024-09-16T05:21:21.818648+0000","last_peered":"2024-09-16T05:21:21.818648+0000","last_clean":"2024-09-16T05:21:21.818648+0000","last_became_active":"2024-09-16T05:16:25.724362+0000","last_became_peered":"2024-09-16T05:16:25.724362+0000","last_unstale":"2024-09-16T05:21:21.818648+0000","last_undegraded":"2024-09-16T05:21:21.818648+0000","last_fullsized":"2024-09-16T05:21:21.818648+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:20:03.076376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.076376+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.456257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039795299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1063,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852529+0000","last_change":"2024-09-16T05:21:10.815050+0000","last_active":"2024-09-16T05:21:18.852529+0000","last_peered":"2024-09-16T05:21:18.852529+0000","last_clean":"2024-09-16T05:21:18.852529+0000","last_became_active":"2024-09-16T05:21:10.814832+0000","last_became_peered":"2024-09-16T05:21:10.814832+0000","last_unstale":"2024-09-16T05:21:18.852529+0000","last_undegraded":"2024-09-16T05:21:18.852529+0000","last_fullsized":"2024-09-16T05:21:18.852529+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:43.443433+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:43.443433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.358661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":2723,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853088+0000","last_change":"2024-09-16T05:21:17.811356+0000","last_active":"2024-09-16T05:21:18.853088+0000","last_peered":"2024-09-16T05:21:18.853088+0000","last_clean":"2024-09-16T05:21:18.853088+0000","last_became_active":"2024-09-16T05:21:07.815584+0000","last_became_peered":"2024-09-16T05:21:07.815584+0000","last_unstale":"2024-09-16T05:21:18.853088+0000","last_undegraded":"2024-09-16T05:21:18.853088+0000","last_fullsized":"2024-09-16T05:21:18.853088+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:21:02.441586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.441586+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:32.435816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093701400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.c","version":"35'139","reported_seq":1051,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-09-16T05:21:16.823597+0000","last_change":"2024-09-16T05:20:36.348317+0000","last_active":"2024-09-16T05:21:16.823597+0000","last_peered":"2024-09-16T05:21:16.823597+0000","last_clean":"2024-09-16T05:21:16.823597+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:21:16.823597+0000","last_undegraded":"2024-09-16T05:21:16.823597+0000","last_fullsized":"2024-09-16T05:21:16.823597+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.348227+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.348227+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.541309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"709'409","reported_seq":3512,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853190+0000","last_change":"2024-09-16T05:21:17.811329+0000","last_active":"2024-09-16T05:21:18.853190+0000","last_peered":"2024-09-16T05:21:18.853190+0000","last_clean":"2024-09-16T05:21:18.853190+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:21:18.853190+0000","last_undegraded":"2024-09-16T05:21:18.853190+0000","last_fullsized":"2024-09-16T05:21:18.853190+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"709'409","last_scrub_stamp":"2024-09-16T05:21:03.413122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:03.413122+0000","objects_scrubbed":1,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:19.657178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090584499999999996,"stat_sum":{"num_bytes":2722965,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57663,"num_write":41,"num_write_kb":16134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.d","version":"35'139","reported_seq":1216,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.815674+0000","last_change":"2024-09-16T05:20:37.488421+0000","last_active":"2024-09-16T05:21:21.815674+0000","last_peered":"2024-09-16T05:21:21.815674+0000","last_clean":"2024-09-16T05:21:21.815674+0000","last_became_active":"2024-09-16T05:20:35.804924+0000","last_became_peered":"2024-09-16T05:20:35.804924+0000","last_unstale":"2024-09-16T05:21:21.815674+0000","last_undegraded":"2024-09-16T05:21:21.815674+0000","last_fullsized":"2024-09-16T05:21:21.815674+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.488293+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.488293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.991686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":3220,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.811993+0000","last_change":"2024-09-16T05:21:17.812151+0000","last_active":"2024-09-16T05:21:17.811993+0000","last_peered":"2024-09-16T05:21:17.811993+0000","last_clean":"2024-09-16T05:21:17.811993+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:21:17.811993+0000","last_undegraded":"2024-09-16T05:21:17.811993+0000","last_fullsized":"2024-09-16T05:21:17.811993+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:20:39.563939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.563939+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:55.608439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019956679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":1213,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.819915+0000","last_change":"2024-09-16T05:20:36.368708+0000","last_active":"2024-09-16T05:21:20.819915+0000","last_peered":"2024-09-16T05:21:20.819915+0000","last_clean":"2024-09-16T05:21:20.819915+0000","last_became_active":"2024-09-16T05:20:35.802192+0000","last_became_peered":"2024-09-16T05:20:35.802192+0000","last_unstale":"2024-09-16T05:21:20.819915+0000","last_undegraded":"2024-09-16T05:21:20.819915+0000","last_fullsized":"2024-09-16T05:21:20.819915+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.368583+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.368583+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.464774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"723'636","reported_seq":3743,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.107220+0000","last_change":"2024-09-16T05:21:20.107458+0000","last_active":"2024-09-16T05:21:20.107220+0000","last_peered":"2024-09-16T05:21:20.107220+0000","last_clean":"2024-09-16T05:21:20.107220+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:21:20.107220+0000","last_undegraded":"2024-09-16T05:21:20.107220+0000","last_fullsized":"2024-09-16T05:21:20.107220+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"709'630","last_scrub_stamp":"2024-09-16T05:21:00.044713+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:21:00.044713+0000","objects_scrubbed":3,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:17.004162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002043722,"stat_sum":{"num_bytes":7830043,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":52780,"num_write":54,"num_write_kb":30221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":1193,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.816055+0000","last_change":"2024-09-16T05:20:39.039439+0000","last_active":"2024-09-16T05:21:20.816055+0000","last_peered":"2024-09-16T05:21:20.816055+0000","last_clean":"2024-09-16T05:21:20.816055+0000","last_became_active":"2024-09-16T05:15:56.716457+0000","last_became_peered":"2024-09-16T05:15:56.716457+0000","last_unstale":"2024-09-16T05:21:20.816055+0000","last_undegraded":"2024-09-16T05:21:20.816055+0000","last_fullsized":"2024-09-16T05:21:20.816055+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:39.039365+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.039365+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.718627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3372,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.812584+0000","last_change":"2024-09-16T05:21:21.813293+0000","last_active":"2024-09-16T05:21:21.812584+0000","last_peered":"2024-09-16T05:21:21.812584+0000","last_clean":"2024-09-16T05:21:21.812584+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:21:21.812584+0000","last_undegraded":"2024-09-16T05:21:21.812584+0000","last_fullsized":"2024-09-16T05:21:21.812584+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:20:39.456780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.456780+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:47.069121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010570829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1740,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.812631+0000","last_change":"2024-09-16T05:20:36.468183+0000","last_active":"2024-09-16T05:21:21.812631+0000","last_peered":"2024-09-16T05:21:21.812631+0000","last_clean":"2024-09-16T05:21:21.812631+0000","last_became_active":"2024-09-16T05:20:35.805506+0000","last_became_peered":"2024-09-16T05:20:35.805506+0000","last_unstale":"2024-09-16T05:21:21.812631+0000","last_undegraded":"2024-09-16T05:21:21.812631+0000","last_fullsized":"2024-09-16T05:21:21.812631+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.468079+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.468079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:44.101000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.8","version":"705'327","reported_seq":3431,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.816074+0000","last_change":"2024-09-16T05:21:21.816471+0000","last_active":"2024-09-16T05:21:21.816074+0000","last_peered":"2024-09-16T05:21:21.816074+0000","last_clean":"2024-09-16T05:21:21.816074+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:21:21.816074+0000","last_undegraded":"2024-09-16T05:21:21.816074+0000","last_fullsized":"2024-09-16T05:21:21.816074+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'325","last_scrub_stamp":"2024-09-16T05:20:23.462954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:23.462954+0000","objects_scrubbed":2,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.416140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012034350000000001,"stat_sum":{"num_bytes":2235529,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":47,"num_read_kb":35425,"num_write":51,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.3","version":"723'235","reported_seq":3350,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.821158+0000","last_change":"2024-09-16T05:21:20.821327+0000","last_active":"2024-09-16T05:21:20.821158+0000","last_peered":"2024-09-16T05:21:20.821158+0000","last_clean":"2024-09-16T05:21:20.821158+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:21:20.821158+0000","last_undegraded":"2024-09-16T05:21:20.821158+0000","last_fullsized":"2024-09-16T05:21:20.821158+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'205","last_scrub_stamp":"2024-09-16T05:20:19.137710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:19.137710+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:34.000370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039933269999999996,"stat_sum":{"num_bytes":5796527,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":117938,"num_write":51,"num_write_kb":20718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":739,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.811953+0000","last_change":"2024-09-16T05:20:00.246338+0000","last_active":"2024-09-16T05:21:21.811953+0000","last_peered":"2024-09-16T05:21:21.811953+0000","last_clean":"2024-09-16T05:21:21.811953+0000","last_became_active":"2024-09-16T05:18:19.768392+0000","last_became_peered":"2024-09-16T05:18:19.768392+0000","last_unstale":"2024-09-16T05:21:21.811953+0000","last_undegraded":"2024-09-16T05:21:21.811953+0000","last_fullsized":"2024-09-16T05:21:21.811953+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:00.246242+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.246242+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.597183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":3104,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.816913+0000","last_change":"2024-09-16T05:21:20.817656+0000","last_active":"2024-09-16T05:21:20.816913+0000","last_peered":"2024-09-16T05:21:20.816913+0000","last_clean":"2024-09-16T05:21:20.816913+0000","last_became_active":"2024-09-16T05:21:07.817459+0000","last_became_peered":"2024-09-16T05:21:07.817459+0000","last_unstale":"2024-09-16T05:21:20.816913+0000","last_undegraded":"2024-09-16T05:21:20.816913+0000","last_fullsized":"2024-09-16T05:21:20.816913+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:02.847979+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:20:02.847979+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076572700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":974,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.809444+0000","last_change":"2024-09-16T05:21:02.510553+0000","last_active":"2024-09-16T05:21:17.809444+0000","last_peered":"2024-09-16T05:21:17.809444+0000","last_clean":"2024-09-16T05:21:17.809444+0000","last_became_active":"2024-09-16T05:20:53.807695+0000","last_became_peered":"2024-09-16T05:20:53.807695+0000","last_unstale":"2024-09-16T05:21:17.809444+0000","last_undegraded":"2024-09-16T05:21:17.809444+0000","last_fullsized":"2024-09-16T05:21:17.809444+0000","mapping_epoch":703,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":704,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:02.510463+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.510463+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:24.081875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":1717,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.816178+0000","last_change":"2024-09-16T05:19:56.353335+0000","last_active":"2024-09-16T05:21:21.816178+0000","last_peered":"2024-09-16T05:21:21.816178+0000","last_clean":"2024-09-16T05:21:21.816178+0000","last_became_active":"2024-09-16T05:15:51.716771+0000","last_became_peered":"2024-09-16T05:15:51.716771+0000","last_unstale":"2024-09-16T05:21:21.816178+0000","last_undegraded":"2024-09-16T05:21:21.816178+0000","last_fullsized":"2024-09-16T05:21:21.816178+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:19:56.353243+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.757169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":3030,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.809848+0000","last_change":"2024-09-16T05:21:17.810163+0000","last_active":"2024-09-16T05:21:17.809848+0000","last_peered":"2024-09-16T05:21:17.809848+0000","last_clean":"2024-09-16T05:21:17.809848+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:21:17.809848+0000","last_undegraded":"2024-09-16T05:21:17.809848+0000","last_fullsized":"2024-09-16T05:21:17.809848+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:20:25.536951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:25.536951+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:51.058818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060550299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1719,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-09-16T05:21:16.823604+0000","last_change":"2024-09-16T05:20:37.311421+0000","last_active":"2024-09-16T05:21:16.823604+0000","last_peered":"2024-09-16T05:21:16.823604+0000","last_clean":"2024-09-16T05:21:16.823604+0000","last_became_active":"2024-09-16T05:15:51.713608+0000","last_became_peered":"2024-09-16T05:15:51.713608+0000","last_unstale":"2024-09-16T05:21:16.823604+0000","last_undegraded":"2024-09-16T05:21:16.823604+0000","last_fullsized":"2024-09-16T05:21:16.823604+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.311338+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.311338+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:37.446300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":3130,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.819218+0000","last_change":"2024-09-16T05:21:21.819419+0000","last_active":"2024-09-16T05:21:21.819218+0000","last_peered":"2024-09-16T05:21:21.819218+0000","last_clean":"2024-09-16T05:21:21.819218+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:21:21.819218+0000","last_undegraded":"2024-09-16T05:21:21.819218+0000","last_fullsized":"2024-09-16T05:21:21.819218+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:19:51.365804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.365804+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032518960000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":2328,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.108091+0000","last_change":"2024-09-16T05:21:20.108270+0000","last_active":"2024-09-16T05:21:20.108091+0000","last_peered":"2024-09-16T05:21:20.108091+0000","last_clean":"2024-09-16T05:21:20.108091+0000","last_became_active":"2024-09-16T05:14:25.831603+0000","last_became_peered":"2024-09-16T05:14:25.831603+0000","last_unstale":"2024-09-16T05:21:20.108091+0000","last_undegraded":"2024-09-16T05:21:20.108091+0000","last_fullsized":"2024-09-16T05:21:20.108091+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:19:50.852123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:50.852123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025985880000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":1684,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.812348+0000","last_change":"2024-09-16T05:19:53.322398+0000","last_active":"2024-09-16T05:21:21.812348+0000","last_peered":"2024-09-16T05:21:21.812348+0000","last_clean":"2024-09-16T05:21:21.812348+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:21:21.812348+0000","last_undegraded":"2024-09-16T05:21:21.812348+0000","last_fullsized":"2024-09-16T05:21:21.812348+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:53.322303+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:53.322303+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.597163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1512,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.811726+0000","last_change":"2024-09-16T05:21:17.811931+0000","last_active":"2024-09-16T05:21:17.811726+0000","last_peered":"2024-09-16T05:21:17.811726+0000","last_clean":"2024-09-16T05:21:17.811726+0000","last_became_active":"2024-09-16T05:16:17.724163+0000","last_became_peered":"2024-09-16T05:16:17.724163+0000","last_unstale":"2024-09-16T05:21:17.811726+0000","last_undegraded":"2024-09-16T05:21:17.811726+0000","last_fullsized":"2024-09-16T05:21:17.811726+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_clean_scrub_stamp":"2024-09-16T05:21:08.532148+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:12.958790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001889588,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":1002,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.816349+0000","last_change":"2024-09-16T05:20:38.411870+0000","last_active":"2024-09-16T05:21:20.816349+0000","last_peered":"2024-09-16T05:21:20.816349+0000","last_clean":"2024-09-16T05:21:20.816349+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:21:20.816349+0000","last_undegraded":"2024-09-16T05:21:20.816349+0000","last_fullsized":"2024-09-16T05:21:20.816349+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:38.411767+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.411767+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:52.750017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"728'601","reported_seq":3700,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853372+0000","last_change":"2024-09-16T05:21:17.812244+0000","last_active":"2024-09-16T05:21:18.853372+0000","last_peered":"2024-09-16T05:21:18.853372+0000","last_clean":"2024-09-16T05:21:18.853372+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:21:18.853372+0000","last_undegraded":"2024-09-16T05:21:18.853372+0000","last_fullsized":"2024-09-16T05:21:18.853372+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'568","last_scrub_stamp":"2024-09-16T05:19:54.247232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:54.247232+0000","objects_scrubbed":3,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:23.269476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001547969,"stat_sum":{"num_bytes":4768139,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":124,"num_read_kb":146749,"num_write":40,"num_write_kb":12889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":946,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815417+0000","last_change":"2024-09-16T05:20:37.414043+0000","last_active":"2024-09-16T05:21:20.815417+0000","last_peered":"2024-09-16T05:21:20.815417+0000","last_clean":"2024-09-16T05:21:20.815417+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:21:20.815417+0000","last_undegraded":"2024-09-16T05:21:20.815417+0000","last_fullsized":"2024-09-16T05:21:20.815417+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.413968+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.413968+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:40.433963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":65,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852631+0000","last_change":"2024-09-16T05:21:18.006970+0000","last_active":"2024-09-16T05:21:18.852631+0000","last_peered":"2024-09-16T05:21:18.852631+0000","last_clean":"2024-09-16T05:21:18.852631+0000","last_became_active":"2024-09-16T05:21:07.816472+0000","last_became_peered":"2024-09-16T05:21:07.816472+0000","last_unstale":"2024-09-16T05:21:18.852631+0000","last_undegraded":"2024-09-16T05:21:18.852631+0000","last_fullsized":"2024-09-16T05:21:18.852631+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:20:38.827535+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.827535+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:58.320433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088640599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":982,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815547+0000","last_change":"2024-09-16T05:20:39.847071+0000","last_active":"2024-09-16T05:21:20.815547+0000","last_peered":"2024-09-16T05:21:20.815547+0000","last_clean":"2024-09-16T05:21:20.815547+0000","last_became_active":"2024-09-16T05:20:35.811624+0000","last_became_peered":"2024-09-16T05:20:35.811624+0000","last_unstale":"2024-09-16T05:21:20.815547+0000","last_undegraded":"2024-09-16T05:21:20.815547+0000","last_fullsized":"2024-09-16T05:21:20.815547+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:39.846968+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.846968+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:46.110945+0000","scrub_duration":389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":3266,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853018+0000","last_change":"2024-09-16T05:21:17.811098+0000","last_active":"2024-09-16T05:21:18.853018+0000","last_peered":"2024-09-16T05:21:18.853018+0000","last_clean":"2024-09-16T05:21:18.853018+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:21:18.853018+0000","last_undegraded":"2024-09-16T05:21:18.853018+0000","last_fullsized":"2024-09-16T05:21:18.853018+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:21:04.422977+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:21:04.422977+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:24.413935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079622600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":1728,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.816066+0000","last_change":"2024-09-16T05:20:36.004657+0000","last_active":"2024-09-16T05:21:20.816066+0000","last_peered":"2024-09-16T05:21:20.816066+0000","last_clean":"2024-09-16T05:21:20.816066+0000","last_became_active":"2024-09-16T05:20:35.807550+0000","last_became_peered":"2024-09-16T05:20:35.807550+0000","last_unstale":"2024-09-16T05:21:20.816066+0000","last_undegraded":"2024-09-16T05:21:20.816066+0000","last_fullsized":"2024-09-16T05:21:20.816066+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:36.004558+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:36.004558+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.268598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"732'654","reported_seq":3789,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.817153+0000","last_change":"2024-09-16T05:21:21.817327+0000","last_active":"2024-09-16T05:21:21.817153+0000","last_peered":"2024-09-16T05:21:21.817153+0000","last_clean":"2024-09-16T05:21:21.817153+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:21:21.817153+0000","last_undegraded":"2024-09-16T05:21:21.817153+0000","last_fullsized":"2024-09-16T05:21:21.817153+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"653'601","last_scrub_stamp":"2024-09-16T05:20:03.311152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.311152+0000","objects_scrubbed":3,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:25.004097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0048388010000000002,"stat_sum":{"num_bytes":6140292,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":127394,"num_write":58,"num_write_kb":34915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1727,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.815086+0000","last_change":"2024-09-16T05:20:37.260334+0000","last_active":"2024-09-16T05:21:21.815086+0000","last_peered":"2024-09-16T05:21:21.815086+0000","last_clean":"2024-09-16T05:21:21.815086+0000","last_became_active":"2024-09-16T05:20:35.804221+0000","last_became_peered":"2024-09-16T05:20:35.804221+0000","last_unstale":"2024-09-16T05:21:21.815086+0000","last_undegraded":"2024-09-16T05:21:21.815086+0000","last_fullsized":"2024-09-16T05:21:21.815086+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.260191+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.260191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:55.461837+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":1539,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852739+0000","last_change":"2024-09-16T05:21:18.005266+0000","last_active":"2024-09-16T05:21:18.852739+0000","last_peered":"2024-09-16T05:21:18.852739+0000","last_clean":"2024-09-16T05:21:18.852739+0000","last_became_active":"2024-09-16T05:21:10.815913+0000","last_became_peered":"2024-09-16T05:21:10.815913+0000","last_unstale":"2024-09-16T05:21:18.852739+0000","last_undegraded":"2024-09-16T05:21:18.852739+0000","last_fullsized":"2024-09-16T05:21:18.852739+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:20:38.316049+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:38.316049+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.095043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043175199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":1721,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.105049+0000","last_change":"2024-09-16T05:20:35.832422+0000","last_active":"2024-09-16T05:21:20.105049+0000","last_peered":"2024-09-16T05:21:20.105049+0000","last_clean":"2024-09-16T05:21:20.105049+0000","last_became_active":"2024-09-16T05:20:35.812754+0000","last_became_peered":"2024-09-16T05:20:35.812754+0000","last_unstale":"2024-09-16T05:21:20.105049+0000","last_undegraded":"2024-09-16T05:21:20.105049+0000","last_fullsized":"2024-09-16T05:21:20.105049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:35.832353+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:20:35.832353+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:43.196095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1210,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:19.857965+0000","last_change":"2024-09-16T05:21:02.949505+0000","last_active":"2024-09-16T05:21:19.857965+0000","last_peered":"2024-09-16T05:21:19.857965+0000","last_clean":"2024-09-16T05:21:19.857965+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:21:19.857965+0000","last_undegraded":"2024-09-16T05:21:19.857965+0000","last_fullsized":"2024-09-16T05:21:19.857965+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:02.949433+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.949433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:17.475261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":1010,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815084+0000","last_change":"2024-09-16T05:21:20.815274+0000","last_active":"2024-09-16T05:21:20.815084+0000","last_peered":"2024-09-16T05:21:20.815084+0000","last_clean":"2024-09-16T05:21:20.815084+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:21:20.815084+0000","last_undegraded":"2024-09-16T05:21:20.815084+0000","last_fullsized":"2024-09-16T05:21:20.815084+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:20:24.333369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:24.333369+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:24.908312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063111999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1717,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815904+0000","last_change":"2024-09-16T05:19:48.737475+0000","last_active":"2024-09-16T05:21:20.815904+0000","last_peered":"2024-09-16T05:21:20.815904+0000","last_clean":"2024-09-16T05:21:20.815904+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:21:20.815904+0000","last_undegraded":"2024-09-16T05:21:20.815904+0000","last_fullsized":"2024-09-16T05:21:20.815904+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:19:48.737348+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:19:48.737348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:24.999619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":2718,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.106818+0000","last_change":"2024-09-16T05:21:20.106818+0000","last_active":"2024-09-16T05:21:20.106818+0000","last_peered":"2024-09-16T05:21:20.106818+0000","last_clean":"2024-09-16T05:21:20.106818+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:21:20.106818+0000","last_undegraded":"2024-09-16T05:21:20.106818+0000","last_fullsized":"2024-09-16T05:21:20.106818+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:21:01.077265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:01.077265+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:14.631723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00147325,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.17","version":"35'139","reported_seq":603,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853044+0000","last_change":"2024-09-16T05:20:43.198542+0000","last_active":"2024-09-16T05:21:18.853044+0000","last_peered":"2024-09-16T05:21:18.853044+0000","last_clean":"2024-09-16T05:21:18.853044+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:21:18.853044+0000","last_undegraded":"2024-09-16T05:21:18.853044+0000","last_fullsized":"2024-09-16T05:21:18.853044+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:43.198473+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:20:43.198473+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.563368+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":2990,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.107775+0000","last_change":"2024-09-16T05:21:20.107775+0000","last_active":"2024-09-16T05:21:20.107775+0000","last_peered":"2024-09-16T05:21:20.107775+0000","last_clean":"2024-09-16T05:21:20.107775+0000","last_became_active":"2024-09-16T05:16:25.734249+0000","last_became_peered":"2024-09-16T05:16:25.734249+0000","last_unstale":"2024-09-16T05:21:20.107775+0000","last_undegraded":"2024-09-16T05:21:20.107775+0000","last_fullsized":"2024-09-16T05:21:20.107775+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:20:37.810519+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.810519+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:38.505683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023386050000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.16","version":"35'139","reported_seq":659,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.809578+0000","last_change":"2024-09-16T05:20:41.564653+0000","last_active":"2024-09-16T05:21:17.809578+0000","last_peered":"2024-09-16T05:21:17.809578+0000","last_clean":"2024-09-16T05:21:17.809578+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:21:17.809578+0000","last_undegraded":"2024-09-16T05:21:17.809578+0000","last_fullsized":"2024-09-16T05:21:17.809578+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.564525+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:54.885944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"709'655","reported_seq":3883,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.106863+0000","last_change":"2024-09-16T05:21:20.107149+0000","last_active":"2024-09-16T05:21:20.106863+0000","last_peered":"2024-09-16T05:21:20.106863+0000","last_clean":"2024-09-16T05:21:20.106863+0000","last_became_active":"2024-09-16T05:21:07.826042+0000","last_became_peered":"2024-09-16T05:21:07.826042+0000","last_unstale":"2024-09-16T05:21:20.106863+0000","last_undegraded":"2024-09-16T05:21:20.106863+0000","last_fullsized":"2024-09-16T05:21:20.106863+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"689'645","last_scrub_stamp":"2024-09-16T05:20:39.846432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:39.846432+0000","objects_scrubbed":3,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:59.026949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001760307,"stat_sum":{"num_bytes":2369985,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":7118573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.15","version":"35'139","reported_seq":681,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.816032+0000","last_change":"2024-09-16T05:20:40.949219+0000","last_active":"2024-09-16T05:21:20.816032+0000","last_peered":"2024-09-16T05:21:20.816032+0000","last_clean":"2024-09-16T05:21:20.816032+0000","last_became_active":"2024-09-16T05:20:40.802090+0000","last_became_peered":"2024-09-16T05:20:40.802090+0000","last_unstale":"2024-09-16T05:21:20.816032+0000","last_undegraded":"2024-09-16T05:21:20.816032+0000","last_fullsized":"2024-09-16T05:21:20.816032+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:40.949147+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:20:40.949147+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:41.611229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"723'332","reported_seq":3069,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.820868+0000","last_change":"2024-09-16T05:21:20.821113+0000","last_active":"2024-09-16T05:21:20.820868+0000","last_peered":"2024-09-16T05:21:20.820868+0000","last_clean":"2024-09-16T05:21:20.820868+0000","last_became_active":"2024-09-16T05:13:25.683915+0000","last_became_peered":"2024-09-16T05:13:25.683915+0000","last_unstale":"2024-09-16T05:21:20.820868+0000","last_undegraded":"2024-09-16T05:21:20.820868+0000","last_fullsized":"2024-09-16T05:21:20.820868+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"665'294","last_scrub_stamp":"2024-09-16T05:20:15.010850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.010850+0000","objects_scrubbed":4,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:36.376324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0043344020000000002,"stat_sum":{"num_bytes":7278037,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":65,"num_read_kb":57072,"num_write":58,"num_write_kb":18502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":1041,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.809952+0000","last_change":"2024-09-16T05:20:37.550357+0000","last_active":"2024-09-16T05:21:17.809952+0000","last_peered":"2024-09-16T05:21:17.809952+0000","last_clean":"2024-09-16T05:21:17.809952+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:21:17.809952+0000","last_undegraded":"2024-09-16T05:21:17.809952+0000","last_fullsized":"2024-09-16T05:21:17.809952+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.550279+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.550279+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:39.895726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":73,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852612+0000","last_change":"2024-09-16T05:21:18.007154+0000","last_active":"2024-09-16T05:21:18.852612+0000","last_peered":"2024-09-16T05:21:18.852612+0000","last_clean":"2024-09-16T05:21:18.852612+0000","last_became_active":"2024-09-16T05:21:07.822615+0000","last_became_peered":"2024-09-16T05:21:07.822615+0000","last_unstale":"2024-09-16T05:21:18.852612+0000","last_undegraded":"2024-09-16T05:21:18.852612+0000","last_fullsized":"2024-09-16T05:21:18.852612+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:20:20.393358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:20.393358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:26.379240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00091381299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.1b","version":"35'139","reported_seq":1128,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852904+0000","last_change":"2024-09-16T05:20:41.173883+0000","last_active":"2024-09-16T05:21:18.852904+0000","last_peered":"2024-09-16T05:21:18.852904+0000","last_clean":"2024-09-16T05:21:18.852904+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:21:18.852904+0000","last_undegraded":"2024-09-16T05:21:18.852904+0000","last_fullsized":"2024-09-16T05:21:18.852904+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.173808+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:00.160931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":3099,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.105174+0000","last_change":"2024-09-16T05:21:20.105759+0000","last_active":"2024-09-16T05:21:20.105174+0000","last_peered":"2024-09-16T05:21:20.105174+0000","last_clean":"2024-09-16T05:21:20.105174+0000","last_became_active":"2024-09-16T05:13:43.692784+0000","last_became_peered":"2024-09-16T05:13:43.692784+0000","last_unstale":"2024-09-16T05:21:20.105174+0000","last_undegraded":"2024-09-16T05:21:20.105174+0000","last_fullsized":"2024-09-16T05:21:20.105174+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:21:06.114038+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:21:06.114038+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:08.736966+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077531900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.1a","version":"35'139","reported_seq":497,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.819665+0000","last_change":"2024-09-16T05:20:41.339619+0000","last_active":"2024-09-16T05:21:20.819665+0000","last_peered":"2024-09-16T05:21:20.819665+0000","last_clean":"2024-09-16T05:21:20.819665+0000","last_became_active":"2024-09-16T05:20:40.806260+0000","last_became_peered":"2024-09-16T05:20:40.806260+0000","last_unstale":"2024-09-16T05:21:20.819665+0000","last_undegraded":"2024-09-16T05:21:20.819665+0000","last_fullsized":"2024-09-16T05:21:20.819665+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.339515+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.339515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:53.872026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":2689,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-09-16T05:21:16.823778+0000","last_change":"2024-09-16T05:21:16.824069+0000","last_active":"2024-09-16T05:21:16.823778+0000","last_peered":"2024-09-16T05:21:16.823778+0000","last_clean":"2024-09-16T05:21:16.823778+0000","last_became_active":"2024-09-16T05:13:43.685435+0000","last_became_peered":"2024-09-16T05:13:43.685435+0000","last_unstale":"2024-09-16T05:21:16.823778+0000","last_undegraded":"2024-09-16T05:21:16.823778+0000","last_fullsized":"2024-09-16T05:21:16.823778+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:20:17.403045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.403045+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:34.312649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045355100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"}]},{"pgid":"1.19","version":"35'139","reported_seq":1727,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.814513+0000","last_change":"2024-09-16T05:20:41.437268+0000","last_active":"2024-09-16T05:21:20.814513+0000","last_peered":"2024-09-16T05:21:20.814513+0000","last_clean":"2024-09-16T05:21:20.814513+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:21:20.814513+0000","last_undegraded":"2024-09-16T05:21:20.814513+0000","last_fullsized":"2024-09-16T05:21:20.814513+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.437156+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.437156+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.770457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"723'1111","reported_seq":4147,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853155+0000","last_change":"2024-09-16T05:21:17.812623+0000","last_active":"2024-09-16T05:21:18.853155+0000","last_peered":"2024-09-16T05:21:18.853155+0000","last_clean":"2024-09-16T05:21:18.853155+0000","last_became_active":"2024-09-16T05:13:43.686758+0000","last_became_peered":"2024-09-16T05:13:43.686758+0000","last_unstale":"2024-09-16T05:21:18.853155+0000","last_undegraded":"2024-09-16T05:21:18.853155+0000","last_fullsized":"2024-09-16T05:21:18.853155+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"653'957","last_scrub_stamp":"2024-09-16T05:20:03.383437+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.383437+0000","objects_scrubbed":11,"log_size":1111,"log_dups_size":0,"ondisk_log_size":1111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017923850000000001,"stat_sum":{"num_bytes":22354013,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":189,"num_read_kb":170933,"num_write":182,"num_write_kb":66629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"1.18","version":"35'139","reported_seq":1729,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815368+0000","last_change":"2024-09-16T05:20:37.024610+0000","last_active":"2024-09-16T05:21:20.815368+0000","last_peered":"2024-09-16T05:21:20.815368+0000","last_clean":"2024-09-16T05:21:20.815368+0000","last_became_active":"2024-09-16T05:20:35.809442+0000","last_became_peered":"2024-09-16T05:20:35.809442+0000","last_unstale":"2024-09-16T05:21:20.815368+0000","last_undegraded":"2024-09-16T05:21:20.815368+0000","last_fullsized":"2024-09-16T05:21:20.815368+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:37.024521+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:37.024521+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:06.640487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"709'654","reported_seq":3789,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.817430+0000","last_change":"2024-09-16T05:21:21.817856+0000","last_active":"2024-09-16T05:21:21.817430+0000","last_peered":"2024-09-16T05:21:21.817430+0000","last_clean":"2024-09-16T05:21:21.817430+0000","last_became_active":"2024-09-16T05:13:43.692266+0000","last_became_peered":"2024-09-16T05:13:43.692266+0000","last_unstale":"2024-09-16T05:21:21.817430+0000","last_undegraded":"2024-09-16T05:21:21.817430+0000","last_fullsized":"2024-09-16T05:21:21.817430+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"666'612","last_scrub_stamp":"2024-09-16T05:20:16.258329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:16.258329+0000","objects_scrubbed":4,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0051037410000000002,"stat_sum":{"num_bytes":8572558,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":118389,"num_write":107,"num_write_kb":34866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.1c","version":"723'472","reported_seq":3278,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.817102+0000","last_change":"2024-09-16T05:21:20.817374+0000","last_active":"2024-09-16T05:21:20.817102+0000","last_peered":"2024-09-16T05:21:20.817102+0000","last_clean":"2024-09-16T05:21:20.817102+0000","last_became_active":"2024-09-16T05:18:02.890972+0000","last_became_peered":"2024-09-16T05:18:02.890972+0000","last_unstale":"2024-09-16T05:21:20.817102+0000","last_undegraded":"2024-09-16T05:21:20.817102+0000","last_fullsized":"2024-09-16T05:21:20.817102+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"704'466","last_scrub_stamp":"2024-09-16T05:21:02.653288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.653288+0000","objects_scrubbed":2,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:30.434355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014109470000000001,"stat_sum":{"num_bytes":4102515,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":10804,"num_write":32,"num_write_kb":13970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1216,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.812077+0000","last_change":"2024-09-16T05:20:42.448898+0000","last_active":"2024-09-16T05:21:21.812077+0000","last_peered":"2024-09-16T05:21:21.812077+0000","last_clean":"2024-09-16T05:21:21.812077+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:21:21.812077+0000","last_undegraded":"2024-09-16T05:21:21.812077+0000","last_fullsized":"2024-09-16T05:21:21.812077+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:42.448822+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:42.448822+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:45.950314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"731'1283","reported_seq":4622,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.820104+0000","last_change":"2024-09-16T05:21:20.820291+0000","last_active":"2024-09-16T05:21:20.820104+0000","last_peered":"2024-09-16T05:21:20.820104+0000","last_clean":"2024-09-16T05:21:20.820104+0000","last_became_active":"2024-09-16T05:18:03.891103+0000","last_became_peered":"2024-09-16T05:18:03.891103+0000","last_unstale":"2024-09-16T05:21:20.820104+0000","last_undegraded":"2024-09-16T05:21:20.820104+0000","last_fullsized":"2024-09-16T05:21:20.820104+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":5,"last_scrub":"705'1256","last_scrub_stamp":"2024-09-16T05:20:58.648411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:58.648411+0000","objects_scrubbed":8,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:16.750414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0041069670000000004,"stat_sum":{"num_bytes":13681777,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":281,"num_read_kb":226672,"num_write":255,"num_write_kb":80012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":7155994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"1.1d","version":"35'139","reported_seq":548,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852813+0000","last_change":"2024-09-16T05:21:10.815207+0000","last_active":"2024-09-16T05:21:18.852813+0000","last_peered":"2024-09-16T05:21:18.852813+0000","last_clean":"2024-09-16T05:21:18.852813+0000","last_became_active":"2024-09-16T05:21:10.815037+0000","last_became_peered":"2024-09-16T05:21:10.815037+0000","last_unstale":"2024-09-16T05:21:18.852813+0000","last_undegraded":"2024-09-16T05:21:18.852813+0000","last_fullsized":"2024-09-16T05:21:18.852813+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:41.421551+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:41.421551+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:22.358679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"723'503","reported_seq":3615,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-09-16T05:21:16.824216+0000","last_change":"2024-09-16T05:21:16.824363+0000","last_active":"2024-09-16T05:21:16.824216+0000","last_peered":"2024-09-16T05:21:16.824216+0000","last_clean":"2024-09-16T05:21:16.824216+0000","last_became_active":"2024-09-16T05:13:43.684889+0000","last_became_peered":"2024-09-16T05:13:43.684889+0000","last_unstale":"2024-09-16T05:21:16.824216+0000","last_undegraded":"2024-09-16T05:21:16.824216+0000","last_fullsized":"2024-09-16T05:21:16.824216+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"671'464","last_scrub_stamp":"2024-09-16T05:20:21.358700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:21.358700+0000","objects_scrubbed":3,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:31.947693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00033486399999999998,"stat_sum":{"num_bytes":4588707,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":80,"num_read_kb":82356,"num_write":31,"num_write_kb":14930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"}]},{"pgid":"1.1c","version":"35'139","reported_seq":448,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.852948+0000","last_change":"2024-09-16T05:20:42.167372+0000","last_active":"2024-09-16T05:21:18.852948+0000","last_peered":"2024-09-16T05:21:18.852948+0000","last_clean":"2024-09-16T05:21:18.852948+0000","last_became_active":"2024-09-16T05:20:40.805879+0000","last_became_peered":"2024-09-16T05:20:40.805879+0000","last_unstale":"2024-09-16T05:21:18.852948+0000","last_undegraded":"2024-09-16T05:21:18.852948+0000","last_fullsized":"2024-09-16T05:21:18.852948+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:20:42.167269+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:20:42.167269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:51.343885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":3321,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:19.857831+0000","last_change":"2024-09-16T05:21:19.858090+0000","last_active":"2024-09-16T05:21:19.857831+0000","last_peered":"2024-09-16T05:21:19.857831+0000","last_clean":"2024-09-16T05:21:19.857831+0000","last_became_active":"2024-09-16T05:13:43.686665+0000","last_became_peered":"2024-09-16T05:13:43.686665+0000","last_unstale":"2024-09-16T05:21:19.857831+0000","last_undegraded":"2024-09-16T05:21:19.857831+0000","last_fullsized":"2024-09-16T05:21:19.857831+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:20:17.544083+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:20:17.544083+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:31.432711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000464747,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.20","version":"709'321","reported_seq":3464,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.816597+0000","last_change":"2024-09-16T05:21:21.816855+0000","last_active":"2024-09-16T05:21:21.816597+0000","last_peered":"2024-09-16T05:21:21.816597+0000","last_clean":"2024-09-16T05:21:21.816597+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:21:21.816597+0000","last_undegraded":"2024-09-16T05:21:21.816597+0000","last_fullsized":"2024-09-16T05:21:21.816597+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"653'290","last_scrub_stamp":"2024-09-16T05:20:03.351330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:03.351330+0000","objects_scrubbed":3,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:31.055715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00144885,"stat_sum":{"num_bytes":4282651,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":177272,"num_write":58,"num_write_kb":21452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":3037,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.810214+0000","last_change":"2024-09-16T05:21:17.810494+0000","last_active":"2024-09-16T05:21:17.810214+0000","last_peered":"2024-09-16T05:21:17.810214+0000","last_clean":"2024-09-16T05:21:17.810214+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:21:17.810214+0000","last_undegraded":"2024-09-16T05:21:17.810214+0000","last_fullsized":"2024-09-16T05:21:17.810214+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:21:09.544178+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:21:09.544178+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:37.535400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00083711300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":1010,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.815337+0000","last_change":"2024-09-16T05:21:20.815648+0000","last_active":"2024-09-16T05:21:20.815337+0000","last_peered":"2024-09-16T05:21:20.815337+0000","last_clean":"2024-09-16T05:21:20.815337+0000","last_became_active":"2024-09-16T05:17:59.866047+0000","last_became_peered":"2024-09-16T05:17:59.866047+0000","last_unstale":"2024-09-16T05:21:20.815337+0000","last_undegraded":"2024-09-16T05:21:20.815337+0000","last_fullsized":"2024-09-16T05:21:20.815337+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:20:15.452705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.452705+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:28.475480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083684299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":2571,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-09-16T05:21:16.824025+0000","last_change":"2024-09-16T05:21:16.824206+0000","last_active":"2024-09-16T05:21:16.824025+0000","last_peered":"2024-09-16T05:21:16.824025+0000","last_clean":"2024-09-16T05:21:16.824025+0000","last_became_active":"2024-09-16T05:20:20.801064+0000","last_became_peered":"2024-09-16T05:20:20.801064+0000","last_unstale":"2024-09-16T05:21:16.824025+0000","last_undegraded":"2024-09-16T05:21:16.824025+0000","last_fullsized":"2024-09-16T05:21:16.824025+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:21:07.244384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:07.244384+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:29.157264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000363366,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":2385,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.107829+0000","last_change":"2024-09-16T05:21:20.108065+0000","last_active":"2024-09-16T05:21:20.107829+0000","last_peered":"2024-09-16T05:21:20.107829+0000","last_clean":"2024-09-16T05:21:20.107829+0000","last_became_active":"2024-09-16T05:13:01.674994+0000","last_became_peered":"2024-09-16T05:13:01.674994+0000","last_unstale":"2024-09-16T05:21:20.107829+0000","last_undegraded":"2024-09-16T05:21:20.107829+0000","last_fullsized":"2024-09-16T05:21:20.107829+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:19:57.859756+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:19:57.859756+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025724950000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.25","version":"723'1203","reported_seq":4206,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853504+0000","last_change":"2024-09-16T05:21:17.812577+0000","last_active":"2024-09-16T05:21:18.853504+0000","last_peered":"2024-09-16T05:21:18.853504+0000","last_clean":"2024-09-16T05:21:18.853504+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:21:18.853504+0000","last_undegraded":"2024-09-16T05:21:18.853504+0000","last_fullsized":"2024-09-16T05:21:18.853504+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"718'1195","last_scrub_stamp":"2024-09-16T05:21:08.434367+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:21:08.434367+0000","objects_scrubbed":11,"log_size":1203,"log_dups_size":0,"ondisk_log_size":1203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:31.827495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001576296,"stat_sum":{"num_bytes":22744435,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":205,"num_read_kb":205717,"num_write":194,"num_write_kb":76992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":2394,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.819524+0000","last_change":"2024-09-16T05:21:21.820058+0000","last_active":"2024-09-16T05:21:21.819524+0000","last_peered":"2024-09-16T05:21:21.819524+0000","last_clean":"2024-09-16T05:21:21.819524+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:21:21.819524+0000","last_undegraded":"2024-09-16T05:21:21.819524+0000","last_fullsized":"2024-09-16T05:21:21.819524+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:21:00.818968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:00.818968+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:01.669200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038780139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":2406,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.820467+0000","last_change":"2024-09-16T05:21:21.820682+0000","last_active":"2024-09-16T05:21:21.820467+0000","last_peered":"2024-09-16T05:21:21.820467+0000","last_clean":"2024-09-16T05:21:21.820467+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:21:21.820467+0000","last_undegraded":"2024-09-16T05:21:21.820467+0000","last_fullsized":"2024-09-16T05:21:21.820467+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:02.023438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:02.023438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:13.278033+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0.0041923619999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1402,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.809737+0000","last_change":"2024-09-16T05:21:17.810330+0000","last_active":"2024-09-16T05:21:17.809737+0000","last_peered":"2024-09-16T05:21:17.809737+0000","last_clean":"2024-09-16T05:21:17.809737+0000","last_became_active":"2024-09-16T05:21:07.818030+0000","last_became_peered":"2024-09-16T05:21:07.818030+0000","last_unstale":"2024-09-16T05:21:17.809737+0000","last_undegraded":"2024-09-16T05:21:17.809737+0000","last_fullsized":"2024-09-16T05:21:17.809737+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:20:58.501508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:58.501508+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:59.901946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00082646599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.29","version":"732'1135","reported_seq":4264,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.821625+0000","last_change":"2024-09-16T05:21:21.821625+0000","last_active":"2024-09-16T05:21:21.821625+0000","last_peered":"2024-09-16T05:21:21.821625+0000","last_clean":"2024-09-16T05:21:21.821625+0000","last_became_active":"2024-09-16T05:13:05.684068+0000","last_became_peered":"2024-09-16T05:13:05.684068+0000","last_unstale":"2024-09-16T05:21:21.821625+0000","last_undegraded":"2024-09-16T05:21:21.821625+0000","last_fullsized":"2024-09-16T05:21:21.821625+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":6,"last_scrub":"708'1087","last_scrub_stamp":"2024-09-16T05:20:58.246006+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:20:58.246006+0000","objects_scrubbed":14,"log_size":1135,"log_dups_size":0,"ondisk_log_size":1135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:27.558468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0033024479999999999,"stat_sum":{"num_bytes":39735473,"num_objects":15,"num_object_clones":11,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":292,"num_read_kb":278138,"num_write":214,"num_write_kb":83455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":3383,"reported_epoch":730,"state":"remapped+peering","last_fresh":"2024-09-16T05:21:20.112696+0000","last_change":"2024-09-16T05:21:21.815935+0000","last_active":"2024-09-16T05:21:21.815935+0000","last_peered":"2024-09-16T05:21:20.112696+0000","last_clean":"2024-09-16T05:21:20.112696+0000","last_became_active":"2024-09-16T05:13:04.681515+0000","last_became_peered":"2024-09-16T05:13:04.681515+0000","last_unstale":"2024-09-16T05:21:20.112696+0000","last_undegraded":"2024-09-16T05:21:20.112696+0000","last_fullsized":"2024-09-16T05:21:20.112696+0000","mapping_epoch":732,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_clean_scrub_stamp":"2024-09-16T05:21:03.783094+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084979100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[8,9],"up_primary":9,"acting_primary":7,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.2b","version":"723'364","reported_seq":3374,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-09-16T05:21:17.811477+0000","last_change":"2024-09-16T05:21:17.811688+0000","last_active":"2024-09-16T05:21:17.811477+0000","last_peered":"2024-09-16T05:21:17.811477+0000","last_clean":"2024-09-16T05:21:17.811477+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:21:17.811477+0000","last_undegraded":"2024-09-16T05:21:17.811477+0000","last_fullsized":"2024-09-16T05:21:17.811477+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"666'332","last_scrub_stamp":"2024-09-16T05:20:18.508468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:18.508468+0000","objects_scrubbed":1,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:19.324689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017823719999999999,"stat_sum":{"num_bytes":4924362,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":79891,"num_write":55,"num_write_kb":23400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":3205,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-16T05:21:18.853316+0000","last_change":"2024-09-16T05:21:17.811884+0000","last_active":"2024-09-16T05:21:18.853316+0000","last_peered":"2024-09-16T05:21:18.853316+0000","last_clean":"2024-09-16T05:21:18.853316+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:21:18.853316+0000","last_undegraded":"2024-09-16T05:21:18.853316+0000","last_fullsized":"2024-09-16T05:21:18.853316+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:19:51.201665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:19:51.201665+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012100399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"1"},{"start":"130","length":"1"}]},{"pgid":"2.2d","version":"732'506","reported_seq":3607,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.818944+0000","last_change":"2024-09-16T05:21:21.819154+0000","last_active":"2024-09-16T05:21:21.818944+0000","last_peered":"2024-09-16T05:21:21.818944+0000","last_clean":"2024-09-16T05:21:21.818944+0000","last_became_active":"2024-09-16T05:13:04.681355+0000","last_became_peered":"2024-09-16T05:13:04.681355+0000","last_unstale":"2024-09-16T05:21:21.818944+0000","last_undegraded":"2024-09-16T05:21:21.818944+0000","last_fullsized":"2024-09-16T05:21:21.818944+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"648'457","last_scrub_stamp":"2024-09-16T05:20:15.223777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:15.223777+0000","objects_scrubbed":2,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:21:38.482270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031596800000000002,"stat_sum":{"num_bytes":10461192,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":94,"num_read_kb":88584,"num_write":74,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.2e","version":"732'689","reported_seq":3749,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-16T05:21:21.822018+0000","last_change":"2024-09-16T05:21:21.822168+0000","last_active":"2024-09-16T05:21:21.822018+0000","last_peered":"2024-09-16T05:21:21.822018+0000","last_clean":"2024-09-16T05:21:21.822018+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:21:21.822018+0000","last_undegraded":"2024-09-16T05:21:21.822018+0000","last_fullsized":"2024-09-16T05:21:21.822018+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"715'671","last_scrub_stamp":"2024-09-16T05:21:06.794552+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:21:06.794552+0000","objects_scrubbed":7,"log_size":689,"log_dups_size":0,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:36.713746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0059914800000000004,"stat_sum":{"num_bytes":18865570,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":104,"num_read_kb":73043,"num_write":97,"num_write_kb":43850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"124","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]},{"pgid":"2.2f","version":"723'626","reported_seq":3755,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-16T05:21:20.107676+0000","last_change":"2024-09-16T05:21:20.108068+0000","last_active":"2024-09-16T05:21:20.107676+0000","last_peered":"2024-09-16T05:21:20.107676+0000","last_clean":"2024-09-16T05:21:20.107676+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:21:20.107676+0000","last_undegraded":"2024-09-16T05:21:20.107676+0000","last_fullsized":"2024-09-16T05:21:20.107676+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"650'553","last_scrub_stamp":"2024-09-16T05:20:00.804910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:20:00.804910+0000","objects_scrubbed":6,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00258443,"stat_sum":{"num_bytes":12118573,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":121,"num_read_kb":85525,"num_write":108,"num_write_kb":43924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"11d","length":"2"},{"start":"121","length":"1"},{"start":"127","length":"1"},{"start":"12d","length":"2"},{"start":"130","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":86,"stat_sum":{"num_bytes":330466224,"num_objects":156,"num_object_clones":105,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":10,"num_read":3643,"num_read_kb":3588116,"num_write":2665,"num_write_kb":1061648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":89317650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":404369408,"data_stored":498224740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25727,"ondisk_log_size":25727,"up":172,"acting":172,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089492,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114088,"kb_used_data":60684,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93631064,"statfs":{"total":95995035648,"available":95878209536,"internally_reserved":0,"allocated":62140416,"data_stored":69045914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285720,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85588,"kb_used_data":32188,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93659564,"statfs":{"total":95995035648,"available":95907393536,"internally_reserved":0,"allocated":32960512,"data_stored":34775416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514657,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97948,"kb_used_data":40388,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93647204,"statfs":{"total":95995035648,"available":95894736896,"internally_reserved":0,"allocated":41357312,"data_stored":45028192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841696,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81180,"kb_used_data":45828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93663972,"statfs":{"total":95995035648,"available":95911907328,"internally_reserved":0,"allocated":46927872,"data_stored":50443448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103335,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117808,"kb_used_data":56088,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93627344,"statfs":{"total":95995035648,"available":95874400256,"internally_reserved":0,"allocated":57434112,"data_stored":64242737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397678,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118940,"kb_used_data":61376,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93626212,"statfs":{"total":95995035648,"available":95873241088,"internally_reserved":0,"allocated":62849024,"data_stored":74855590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757424,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65268,"kb_used_data":38232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93679884,"statfs":{"total":95995035648,"available":95928201216,"internally_reserved":0,"allocated":39149568,"data_stored":46854572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920950,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58528,"kb_used_data":27332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93686624,"statfs":{"total":95995035648,"available":95935102976,"internally_reserved":0,"allocated":27987968,"data_stored":27331501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117178,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119256,"kb_used_data":65856,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93625896,"statfs":{"total":95995035648,"available":95872917504,"internally_reserved":0,"allocated":67436544,"data_stored":83523149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240465,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62760,"kb_used_data":22160,"kb_used_omap":10,"kb_used_meta":40565,"kb_avail":93682392,"statfs":{"total":95995035648,"available":95930769408,"internally_reserved":0,"allocated":22691840,"data_stored":21084464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":41538867},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62072,"kb_used_data":30880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683080,"statfs":{"total":95995035648,"available":95931473920,"internally_reserved":0,"allocated":31621120,"data_stored":32023323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738570,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49588,"kb_used_data":18396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93695564,"statfs":{"total":95995035648,"available":95944257536,"internally_reserved":0,"allocated":18837504,"data_stored":18345123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9691136,"data_stored":12145873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22802432,"data_stored":26151753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13643776,"data_stored":14975735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58314752,"data_stored":77340927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19156992,"data_stored":21451414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":30126080,"data_stored":43087610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53993472,"data_stored":68956728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48578560,"data_stored":58343875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":38121472,"data_stored":44580339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":32526336,"data_stored":39148105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24129536,"data_stored":28895329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":53284864,"data_stored":63147052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:21:24.070 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:21:24.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 96 2024-09-16T05:21:24.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:24.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:24.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:23 smithi204 ceph-mon[30933]: pgmap v1260: 117 pgs: 1 remapped+peering, 1 active+clean+snaptrim, 115 active+clean; 312 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-09-16T05:21:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:23 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:21:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:23 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:21:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:23 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:21:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:23 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:21:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:23 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:21:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:23 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:21:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:23 smithi204 ceph-mon[30933]: osdmap e734: 12 total, 12 up, 12 in 2024-09-16T05:21:24.180 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:24.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:24.238 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:24.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:24.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:24.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:24.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:24.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:21:24.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:24.500 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:24.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:24.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:21:24.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:24.624 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:24.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:24.660 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:24.661 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:24.666 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:24.682 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:24.701 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:24.704 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:24.705 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:24.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:24.732 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:24.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:24.754 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:24.754 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:24.775 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:24.776 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:24.797 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:24.797 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:24.818 INFO:tasks.rados.rados.2.smithi204.stdout:2868: done (0 left) 2024-09-16T05:21:24.818 INFO:tasks.rados.rados.2.smithi204.stdout:2869: snap_create 2024-09-16T05:21:24.821 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:24.821 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:24.842 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:24.843 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:24.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:24.864 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:24.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:24.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:24.887 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:24.888 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:24.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:24.910 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:24.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:25.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:25.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: from='client.16926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: 2.55 scrub starts 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: 2.55 scrub ok 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: 1.1d scrub starts 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: 1.1d scrub ok 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/189560400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:25.103 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:24 smithi196 ceph-mon[30802]: osdmap e735: 12 total, 12 up, 12 in 2024-09-16T05:21:25.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:25.128 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:21:25.128 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:21:25.128 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:21:25.128 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:21:25.139 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:25.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:25.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: from='client.16926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:21:25.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: 2.55 scrub starts 2024-09-16T05:21:25.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: 2.55 scrub ok 2024-09-16T05:21:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: 1.1d scrub starts 2024-09-16T05:21:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: 1.1d scrub ok 2024-09-16T05:21:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:21:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:21:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/189560400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:24 smithi032 ceph-mon[27388]: osdmap e735: 12 total, 12 up, 12 in 2024-09-16T05:21:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: from='client.16926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:21:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: 2.55 scrub starts 2024-09-16T05:21:25.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: 2.55 scrub ok 2024-09-16T05:21:25.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: 1.1d scrub starts 2024-09-16T05:21:25.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: 1.1d scrub ok 2024-09-16T05:21:25.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:21:25.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:21:25.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/189560400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:25.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:24 smithi204 ceph-mon[30933]: osdmap e735: 12 total, 12 up, 12 in 2024-09-16T05:21:25.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:25.264 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:25.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:25.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:25.393 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:25.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:25.465 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:25.494 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:25.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:25.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:25.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:25.647 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:25.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:25.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:25.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:25.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:25.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:21:25.814 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pg_num to 96 2024-09-16T05:21:25.818 INFO:tasks.rados.rados.2.smithi204.stdout:2869: done (0 left) 2024-09-16T05:21:25.818 INFO:tasks.rados.rados.2.smithi204.stdout:2870: write oid 40 current snap is 306 2024-09-16T05:21:25.818 INFO:tasks.rados.rados.2.smithi204.stdout:2870: seq_num 1011 ranges {591877=635335,1628354=709677} 2024-09-16T05:21:25.824 INFO:tasks.rados.rados.2.smithi204.stdout:2870: writing smithi20457400-40 from 591877 to 1227212 tid 1 2024-09-16T05:21:25.830 INFO:tasks.rados.rados.2.smithi204.stdout:2870: writing smithi20457400-40 from 1628354 to 2338031 tid 2 2024-09-16T05:21:25.830 INFO:tasks.rados.rados.2.smithi204.stdout:2871: delete oid 32 current snap is 306 2024-09-16T05:21:25.830 INFO:tasks.rados.rados.2.smithi204.stdout:2870: finishing write tid 1 to smithi20457400-40 2024-09-16T05:21:25.835 INFO:tasks.rados.rados.2.smithi204.stdout:2870: finishing write tid 2 to smithi20457400-40 2024-09-16T05:21:25.835 INFO:tasks.rados.rados.2.smithi204.stdout:2870: finishing write tid 3 to smithi20457400-40 2024-09-16T05:21:25.835 INFO:tasks.rados.rados.2.smithi204.stdout:2870: finishing write tid 4 to smithi20457400-40 2024-09-16T05:21:25.835 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1051 (ObjNum 1011 snap 306 seq_num 1011) dirty exists 2024-09-16T05:21:25.835 INFO:tasks.rados.rados.2.smithi204.stdout:2870: left oid 40 (ObjNum 1011 snap 306 seq_num 1011) 2024-09-16T05:21:25.835 INFO:tasks.rados.rados.2.smithi204.stdout:2870: done (1 left) 2024-09-16T05:21:25.835 INFO:tasks.rados.rados.2.smithi204.stdout:2871: done (0 left) 2024-09-16T05:21:25.836 INFO:tasks.rados.rados.2.smithi204.stdout:2872: write (excl) oid 12 current snap is 306 2024-09-16T05:21:25.836 INFO:tasks.rados.rados.2.smithi204.stdout:2872: seq_num 1013 ranges {625491=502186,1861671=713346} 2024-09-16T05:21:25.840 INFO:tasks.rados.rados.2.smithi204.stdout:2872: writing smithi20457400-12 from 625491 to 1127677 tid 1 2024-09-16T05:21:25.848 INFO:tasks.rados.rados.2.smithi204.stdout:2872: writing smithi20457400-12 from 1861671 to 2575017 tid 2 2024-09-16T05:21:25.849 INFO:tasks.rados.rados.2.smithi204.stdout:2873: copy_from oid 21 from oid 32 current snap is 306 2024-09-16T05:21:25.849 INFO:tasks.rados.rados.2.smithi204.stdout:2872: finishing write tid 1 to smithi20457400-12 2024-09-16T05:21:25.849 INFO:tasks.rados.rados.2.smithi204.stdout:2874: snap_remove snap 286 2024-09-16T05:21:25.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:25.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:25.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:25.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:26.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:26.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:26.076 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:26.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:26.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:26.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: pgmap v1263: 117 pgs: 1 remapped+peering, 1 active+clean+snaptrim, 115 active+clean; 307 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 16 op/s 2024-09-16T05:21:26.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:21:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:21:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:21:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:21:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: 2.3d scrub starts 2024-09-16T05:21:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: 2.3d scrub ok 2024-09-16T05:21:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1707716759' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]: dispatch 2024-09-16T05:21:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1707716759' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]': finished 2024-09-16T05:21:26.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:25 smithi032 ceph-mon[27388]: osdmap e736: 12 total, 12 up, 12 in 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: pgmap v1263: 117 pgs: 1 remapped+peering, 1 active+clean+snaptrim, 115 active+clean; 307 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 16 op/s 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: 2.3d scrub starts 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: 2.3d scrub ok 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1707716759' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]: dispatch 2024-09-16T05:21:26.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1707716759' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]': finished 2024-09-16T05:21:26.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:25 smithi204 ceph-mon[30933]: osdmap e736: 12 total, 12 up, 12 in 2024-09-16T05:21:26.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: pgmap v1263: 117 pgs: 1 remapped+peering, 1 active+clean+snaptrim, 115 active+clean; 307 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 16 op/s 2024-09-16T05:21:26.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:21:26.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:21:26.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:21:26.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:21:26.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: 2.3d scrub starts 2024-09-16T05:21:26.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: 2.3d scrub ok 2024-09-16T05:21:26.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1707716759' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]: dispatch 2024-09-16T05:21:26.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1707716759' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "96"}]': finished 2024-09-16T05:21:26.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:25 smithi196 ceph-mon[30802]: osdmap e736: 12 total, 12 up, 12 in 2024-09-16T05:21:26.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:26.270 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:26.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:26.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:26.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:26.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:26.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:26.522 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:26.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:26.593 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:21:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:21:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:21:26.593 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:21:26.593 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:21:26.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:26.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:26.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:26.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:26.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:26.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:26.824 INFO:tasks.rados.rados.2.smithi204.stdout:2873: finishing copy_from to smithi20457400-21 2024-09-16T05:21:26.824 INFO:tasks.rados.rados.2.smithi204.stdout:2873: got expected ENOENT (src dne) 2024-09-16T05:21:26.824 INFO:tasks.rados.rados.2.smithi204.stdout:2872: finishing write tid 2 to smithi20457400-12 2024-09-16T05:21:26.824 INFO:tasks.rados.rados.2.smithi204.stdout:2873: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:21:26.824 INFO:tasks.rados.rados.2.smithi204.stdout:2872: finishing write tid 3 to smithi20457400-12 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2872: finishing write tid 4 to smithi20457400-12 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1127 (ObjNum 1013 snap 306 seq_num 1013) dirty exists 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2872: left oid 12 (ObjNum 1013 snap 306 seq_num 1013) 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2872: done (2 left) 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2873: done (1 left) 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2874: done (0 left) 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2875: read oid 12 snap -1 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2875: expect (ObjNum 1013 snap 306 seq_num 1013) 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2876: rollback oid 16 current snap is 306 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 16 to 276 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2877: copy_from oid 41 from oid 21 current snap is 306 2024-09-16T05:21:26.825 INFO:tasks.rados.rados.2.smithi204.stdout:2878: snap_remove snap 295 2024-09-16T05:21:26.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:26.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:26.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:26.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:27.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:27.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:27.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: 2.55 scrub starts 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: 2.55 scrub ok 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]': finished 2024-09-16T05:21:27.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:26 smithi032 ceph-mon[27388]: osdmap e737: 12 total, 12 up, 12 in 2024-09-16T05:21:27.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:27.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:27.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:27.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: 2.55 scrub starts 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: 2.55 scrub ok 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]': finished 2024-09-16T05:21:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:26 smithi204 ceph-mon[30933]: osdmap e737: 12 total, 12 up, 12 in 2024-09-16T05:21:27.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: 2.55 scrub starts 2024-09-16T05:21:27.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: 2.55 scrub ok 2024-09-16T05:21:27.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:21:27.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:21:27.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:21:27.198 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:21:27.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:27.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:27.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "96"}]': finished 2024-09-16T05:21:27.199 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:26 smithi196 ceph-mon[30802]: osdmap e737: 12 total, 12 up, 12 in 2024-09-16T05:21:27.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:21:27.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:27.273 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:27.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:27.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:27.398 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:27.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:27.467 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:27.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:27.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:27.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:27.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:27.646 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:27.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:27.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:27.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:27.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:2878: done (3 left) 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:2879: delete oid 2 current snap is 306 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:2877: finishing copy_from to smithi20457400-41 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:2877: got expected ENOENT (src dne) 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:2877: finishing copy_from racing read to smithi20457400-41 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:2876: finishing rollback tid 0 to smithi20457400-16 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:2876: finishing rollback tid 1 to smithi20457400-16 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:2876: finishing rollback tid 2 to smithi20457400-16 2024-09-16T05:21:27.825 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1028 (ObjNum 914 snap 271 seq_num 914) dirty exists 2024-09-16T05:21:27.828 INFO:tasks.rados.rados.2.smithi204.stdout:2876: done (3 left) 2024-09-16T05:21:27.828 INFO:tasks.rados.rados.2.smithi204.stdout:2877: done (2 left) 2024-09-16T05:21:27.828 INFO:tasks.rados.rados.2.smithi204.stdout:2879: done (1 left) 2024-09-16T05:21:27.828 INFO:tasks.rados.rados.2.smithi204.stdout:2880: snap_create 2024-09-16T05:21:27.832 INFO:tasks.rados.rados.2.smithi204.stdout:2875: expect (ObjNum 1013 snap 306 seq_num 1013) 2024-09-16T05:21:27.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:27.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:27.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:27.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:21:27.979 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:21:27.979 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:21:27.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:21:27.979 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:21:27.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.979 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1378 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 207] snapc 38=[38,37,35,33,31,2f,2d,27,22,21,20,1f] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:21:27.979 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 545.38785733099996, 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018067625, 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1378 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:27.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589516+0000", 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:27.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589519+0000", 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3460000000000001e-06 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589547+0000", 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8572999999999999e-05 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589577+0000", 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9822e-05 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589592+0000", 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4630000000000001e-05 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589745+0000", 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015316700000000001 2024-09-16T05:21:27.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.589921+0000", 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000175995 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.602330+0000", 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012408878 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.605723+0000", 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0033928000000000001 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607502+0000", 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017799140000000001 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607546+0000", 2024-09-16T05:21:27.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3130999999999998e-05 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:22.607584+0000", 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8368999999999998e-05 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.986 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(2.34 e278 from shard 3)", 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 477.11039695199997, 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34868665500000001, 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:27.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:21:27.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:27.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9999999999999997e-07 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866983+0000", 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.84e-06 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215470+0000", 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34848719900000003 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215480+0000", 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2129999999999997e-06 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215584+0000", 2024-09-16T05:21:27.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010428400000000001 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215601+0000", 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7509000000000001e-05 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215663+0000", 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1909999999999995e-05 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:27.991 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2256 2.34 2:2cffa4d6:::smithi20457400-50:head [copy-from ver 319] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 466.29571641299998, 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019683254000000001, 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:27.992 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2256 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:27.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681657+0000", 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681660+0000", 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5019999999999999e-06 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681670+0000", 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.91e-06 2024-09-16T05:21:27.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681686+0000", 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6699999999999999e-05 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681695+0000", 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9590000000000001e-06 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681793+0000", 2024-09-16T05:21:27.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7699000000000001e-05 2024-09-16T05:21:27.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.996 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.996 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:21:27.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.681800+0000", 2024-09-16T05:21:27.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1199999999999996e-06 2024-09-16T05:21:27.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:27.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.682948+0000", 2024-09-16T05:21:27.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011475470000000001 2024-09-16T05:21:27.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:27.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.683035+0000", 2024-09-16T05:21:27.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6886999999999997e-05 2024-09-16T05:21:27.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:27.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:21:27.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.693853+0000", 2024-09-16T05:21:27.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010818583 2024-09-16T05:21:27.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:27.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.698768+0000", 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049152340000000001 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701246+0000", 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024771630000000001 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701284+0000", 2024-09-16T05:21:28.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8544000000000002e-05 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.701340+0000", 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6406000000000002e-05 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2511 2.3 2.e1e51803 (undecoded) ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e320)", 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:21:28.001 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 423.15430161699999, 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99828526200000001, 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2511 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823073+0000", 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.04e-06 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823078+0000", 2024-09-16T05:21:28.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6519999999999997e-06 2024-09-16T05:21:28.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823091+0000", 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3277e-05 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823100+0000", 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4909999999999994e-06 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823141+0000", 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0187e-05 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823147+0000", 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3929999999999996e-06 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821329+0000", 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99818146900000004 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821357+0000", 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8753000000000001e-05 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.007 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:136 2.34 e355/278 ondisk, result = 0)", 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 382.27559989100001, 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83858643600000005, 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701775+0000", 2024-09-16T05:21:28.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6200000000000006e-07 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701786+0000", 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0815000000000001e-05 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540112+0000", 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83832645699999997 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540147+0000", 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5416000000000001e-05 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540246+0000", 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8171999999999995e-05 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540254+0000", 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:21:28.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540360+0000", 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010627400000000001 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3152 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 301] snapc 8a=[8a,88,87,86,85,84,83,81,7d,7c] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.26506343699998, 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017546488999999998, 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3152 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.013 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712310+0000", 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712312+0000", 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4449999999999999e-06 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712319+0000", 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4000000000000003e-06 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712332+0000", 2024-09-16T05:21:28.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2928000000000001e-05 2024-09-16T05:21:28.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712356+0000", 2024-09-16T05:21:28.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4071999999999999e-05 2024-09-16T05:21:28.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712421+0000", 2024-09-16T05:21:28.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4709000000000007e-05 2024-09-16T05:21:28.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.712428+0000", 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2520000000000002e-06 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713705+0000", 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001276672 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.713800+0000", 2024-09-16T05:21:28.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5266999999999994e-05 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.722741+0000", 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0089409339999999993 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.726991+0000", 2024-09-16T05:21:28.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042500949999999997 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729789+0000", 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002798132 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729826+0000", 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6576999999999999e-05 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:46.729857+0000", 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1006000000000001e-05 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3573 2.3 2:c018a787:::smithi20457400-18:head [write 1692526~473837] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 302.11163175399997, 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022224677000000002, 2024-09-16T05:21:28.021 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3573 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:21:28.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865743+0000", 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0100000000000001e-06 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866150+0000", 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040748299999999999 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866163+0000", 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2187999999999999e-05 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866168+0000", 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4689999999999996e-06 2024-09-16T05:21:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866251+0000", 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2911000000000006e-05 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866320+0000", 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8596000000000005e-05 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866403+0000", 2024-09-16T05:21:28.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3796999999999997e-05 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867984+0000", 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015806399999999999 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887897+0000", 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019913157000000001 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887934+0000", 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6482000000000001e-05 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887967+0000", 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2944000000000002e-05 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3574 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 2403500] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:21:28.027 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 302.11119220099999, 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039246627999999999, 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.028 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3574 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866182+0000", 2024-09-16T05:21:28.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:21:28.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866205+0000", 2024-09-16T05:21:28.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2592e-05 2024-09-16T05:21:28.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866222+0000", 2024-09-16T05:21:28.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7422e-05 2024-09-16T05:21:28.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866231+0000", 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5220000000000007e-06 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867407+0000", 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001176014 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867514+0000", 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106947 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867661+0000", 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000147396 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867987+0000", 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032622400000000001 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905316+0000", 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037329222000000002 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.035 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905388+0000", 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1896000000000004e-05 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905428+0000", 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9830000000000003e-05 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3575 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 302.111134204, 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039343647000000002, 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3575 2024-09-16T05:21:28.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866240+0000", 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7999999999999995e-07 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866243+0000", 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5770000000000001e-06 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866249+0000", 2024-09-16T05:21:28.040 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:21:28.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866254+0000", 2024-09-16T05:21:28.040 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-06 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867919+0000", 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016648609999999999 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:28.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867974+0000", 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4478000000000003e-05 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905464+0000", 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037490460000000003 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905516+0000", 2024-09-16T05:21:28.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1894999999999997e-05 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905583+0000", 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3579 2.34 2:2cffa4d6:::smithi20457400-50:head [write 420307~462694] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:21:28.043 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 302.10654202500001, 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017271116999999999, 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3579 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870832+0000", 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.870833+0000", 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2440000000000001e-06 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871089+0000", 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025604600000000001 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871102+0000", 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2955999999999999e-05 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871110+0000", 2024-09-16T05:21:28.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2700000000000004e-06 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871161+0000", 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0899999999999997e-05 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871220+0000", 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8659999999999997e-05 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:21:28.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.871310+0000", 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0586000000000003e-05 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.872398+0000", 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010881409999999999 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888033+0000", 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015634834 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888071+0000", 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7957999999999998e-05 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.888103+0000", 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1522000000000001e-05 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3580 2.34 2:2cffa4d6:::smithi20457400-50:head [write 1351008~617495] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 302.10083345999999, 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030886027999999999, 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3580 2024-09-16T05:21:28.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876542+0000", 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877114+0000", 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057156899999999996 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877126+0000", 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2751999999999999e-05 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877134+0000", 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.072e-06 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0603000000000003e-05 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877240+0000", 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4561000000000002e-05 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877317+0000", 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7002999999999994e-05 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878479+0000", 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011619130000000001 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907362+0000", 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028883039999999999 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907396+0000", 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4807000000000001e-05 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907426+0000", 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9821000000000001e-05 2024-09-16T05:21:28.055 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3581 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2420307] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 302.10022266599998, 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030393973000000001, 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.056 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3581 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877152+0000", 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3099999999999996e-07 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877156+0000", 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7589999999999999e-06 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877169+0000", 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3129000000000001e-05 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877175+0000", 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6259999999999997e-06 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877997+0000", 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082123199999999997 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878081+0000", 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4914999999999996e-05 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878202+0000", 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120207 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878482+0000", 2024-09-16T05:21:28.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027995500000000003 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907462+0000", 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028980381 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907500+0000", 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907545+0000", 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5120999999999997e-05 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3582 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 302.10018825999998, 2024-09-16T05:21:28.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030495267999999999, 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3582 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877186+0000", 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-07 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877189+0000", 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6800000000000002e-06 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877196+0000", 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0650000000000001e-06 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877202+0000", 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.28e-06 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878412+0000", 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001210205 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878470+0000", 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7759e-05 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907572+0000", 2024-09-16T05:21:28.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029102086999999999 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907620+0000", 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8384000000000003e-05 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907681+0000", 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0244000000000001e-05 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:288 2.16 e446/273 ondisk, result = 0)", 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 291.09365689999998, 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65473014699999998, 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:21:28.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883718+0000", 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3799999999999996e-07 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883722+0000", 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.284e-06 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538184+0000", 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.654462031 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538201+0000", 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6634e-05 2024-09-16T05:21:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538327+0000", 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012605799999999999 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538338+0000", 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1188e-05 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538447+0000", 2024-09-16T05:21:28.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109014 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4460 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 511] snapc bc=[bc,bb,ba,b8] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 234.91807297700001, 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019695292, 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.072 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4460 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9799999999999999e-07 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059304+0000", 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5490000000000001e-06 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059308+0000", 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3610000000000003e-06 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059311+0000", 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9179999999999998e-06 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059364+0000", 2024-09-16T05:21:28.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2790000000000001e-05 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059422+0000", 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7929000000000001e-05 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.073174+0000", 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013752234 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.077028+0000", 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038540319999999999 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078924+0000", 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001896179 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078961+0000", 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7311000000000002e-05 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078996+0000", 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4591000000000001e-05 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4461 2.3 2:c018a787:::smithi20457400-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 234.918052037, 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017250960999999999, 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.078 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.079 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:28.079 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.079 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.079 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.079 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4461 2024-09-16T05:21:28.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.079 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059322+0000", 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4899999999999995e-07 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059326+0000", 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9280000000000003e-06 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059333+0000", 2024-09-16T05:21:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6529999999999997e-06 2024-09-16T05:21:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059338+0000", 2024-09-16T05:21:28.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1379999999999998e-06 2024-09-16T05:21:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059598+0000", 2024-09-16T05:21:28.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026046400000000002 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059658+0000", 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9342999999999997e-05 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076427+0000", 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016769117 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076490+0000", 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3207000000000007e-05 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.076573+0000", 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2562e-05 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.086 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5587 2.34 2:2d3f13f8:::smithi20457400-7:head [setxattr _header (57) in=7b,truncate 3260723] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 155.17153530799999, 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.147062895, 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.087 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5587 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805839+0000", 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8299999999999997e-07 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805842+0000", 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6790000000000001e-06 2024-09-16T05:21:28.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805846+0000", 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4089999999999996e-06 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805849+0000", 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.191e-06 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806887+0000", 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010375440000000001 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806953+0000", 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5833000000000005e-05 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807079+0000", 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012666500000000001 2024-09-16T05:21:28.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.809689+0000", 2024-09-16T05:21:28.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026092860000000002 2024-09-16T05:21:28.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952747+0000", 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14305815 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952826+0000", 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9773999999999999e-05 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952901+0000", 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4781000000000001e-05 2024-09-16T05:21:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5588 2.34 2:2d3f13f8:::smithi20457400-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 155.171508835, 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14722005799999999, 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5588 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.65e-07 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805867+0000", 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.86e-06 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805872+0000", 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805876+0000", 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0380000000000001e-06 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807396+0000", 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001520448 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807470+0000", 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3695000000000005e-05 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952937+0000", 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14546690300000001 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953007+0000", 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9801999999999993e-05 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953085+0000", 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8121999999999995e-05 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.099 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:534 2.34 e617/278 ondisk, result = 0)", 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 121.16521869899999, 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93956844500000003, 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.100 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812156+0000", 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0599999999999999e-07 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812160+0000", 2024-09-16T05:21:28.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.437e-06 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751436+0000", 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93927549399999999 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751483+0000", 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7117999999999998e-05 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751598+0000", 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115448 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751609+0000", 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1156e-05 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751723+0000", 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113986 2024-09-16T05:21:28.103 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6080 2.34 2:2cffa4d6:::smithi20457400-50:head [copy-from ver 1093] snapc fa=[fa,f9,f7,f6,f2,f0,ef,ed,e8,e7] ondisk+write+known_if_redirected+supports_pool_eio e625)", 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:34.819560+0000", 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 113.157813722, 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018127142999999998, 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:28.104 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6080 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819560+0000", 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819560+0000", 2024-09-16T05:21:28.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819562+0000", 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.764e-06 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819571+0000", 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7119999999999995e-06 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:28.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819591+0000", 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9340000000000001e-05 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819599+0000", 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.994e-06 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819693+0000", 2024-09-16T05:21:28.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4585000000000003e-05 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.819790+0000", 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6237999999999996e-05 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.830983+0000", 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011193613999999999 2024-09-16T05:21:28.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.836190+0000", 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052071920000000002 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.837611+0000", 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014208809999999999 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.837652+0000", 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1242000000000003e-05 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:34.837687+0000", 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4581e-05 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:28.110 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:21:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:28.112 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:28.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: pgmap v1266: 117 pgs: 117 active+clean; 287 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:21:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:21:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:21:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:21:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:21:28.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4191703405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: 2.4d scrub starts 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: 2.4d scrub ok 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4191703405' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:27 smithi032 ceph-mon[27388]: osdmap e738: 12 total, 12 up, 12 in 2024-09-16T05:21:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: pgmap v1266: 117 pgs: 117 active+clean; 287 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4191703405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: 2.4d scrub starts 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: 2.4d scrub ok 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4191703405' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:27 smithi204 ceph-mon[30933]: osdmap e738: 12 total, 12 up, 12 in 2024-09-16T05:21:28.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:28.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:28.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:28.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:28.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: pgmap v1266: 117 pgs: 117 active+clean; 287 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4191703405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: 2.4d scrub starts 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: 2.4d scrub ok 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:21:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4191703405' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:27 smithi196 ceph-mon[30802]: osdmap e738: 12 total, 12 up, 12 in 2024-09-16T05:21:28.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:28.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:28.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:28.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:28.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:28.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:28.606 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:28.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:28.690 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:28.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:28.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:28.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:28.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:21:28.824 INFO:tasks.rados.rados.2.smithi204.stdout:2875: done (1 left) 2024-09-16T05:21:28.824 INFO:tasks.rados.rados.2.smithi204.stdout:2880: done (0 left) 2024-09-16T05:21:28.825 INFO:tasks.rados.rados.2.smithi204.stdout:2881: write oid 32 current snap is 307 2024-09-16T05:21:28.825 INFO:tasks.rados.rados.2.smithi204.stdout:2881: seq_num 1015 ranges {659105=652684,1862282=717015} 2024-09-16T05:21:28.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:28.832 INFO:tasks.rados.rados.2.smithi204.stdout:2881: writing smithi20457400-32 from 659105 to 1311789 tid 1 2024-09-16T05:21:28.838 INFO:tasks.rados.rados.2.smithi204.stdout:2881: writing smithi20457400-32 from 1862282 to 2579297 tid 2 2024-09-16T05:21:28.838 INFO:tasks.rados.rados.2.smithi204.stdout:2882: snap_create 2024-09-16T05:21:28.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:28.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:28.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:28.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:28.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:29.033 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:29.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:29.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:29.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:29.153 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:21:29.154 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]': finished 2024-09-16T05:21:29.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:28 smithi196 ceph-mon[30802]: osdmap e739: 12 total, 12 up, 12 in 2024-09-16T05:21:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:21:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:21:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:21:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:21:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:21:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:21:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]': finished 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:28 smithi032 ceph-mon[27388]: osdmap e739: 12 total, 12 up, 12 in 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:21:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]: dispatch 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "96"}]': finished 2024-09-16T05:21:29.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:28 smithi204 ceph-mon[30933]: osdmap e739: 12 total, 12 up, 12 in 2024-09-16T05:21:29.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:29.213 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:29.213 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:29.224 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:29.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:29.234 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:29.235 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:29.256 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:29.256 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:29.276 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:29.277 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:29.298 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:29.298 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:29.319 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:29.319 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:29.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:29.340 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:29.341 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:29.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:29.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:29.362 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:29.362 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:29.383 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:29.384 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:29.405 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:29.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:29.427 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:29.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:29.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:29.450 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:29.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:29.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:29.559 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:29.559 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:29.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:29.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:29.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:29.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:29.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:29.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:29.825 INFO:tasks.rados.rados.2.smithi204.stdout:2882: done (1 left) 2024-09-16T05:21:29.826 INFO:tasks.rados.rados.2.smithi204.stdout:2883: snap_create 2024-09-16T05:21:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:29.847 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:29.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:29.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:29.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:29.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:30.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:30.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:30.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:21:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:21:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: pgmap v1269: 127 pgs: 1 activating, 1 peering, 8 unknown, 117 active+clean; 264 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 329 KiB/s wr, 0 op/s 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.3f scrub starts 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.3f scrub ok 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.3b scrub starts 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.3b scrub ok 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3129570099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.36 scrub starts 2024-09-16T05:21:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: 2.36 scrub ok 2024-09-16T05:21:30.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:29 smithi032 ceph-mon[27388]: osdmap e740: 12 total, 12 up, 12 in 2024-09-16T05:21:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:21:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:21:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: pgmap v1269: 127 pgs: 1 activating, 1 peering, 8 unknown, 117 active+clean; 264 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 329 KiB/s wr, 0 op/s 2024-09-16T05:21:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:21:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:21:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:21:30.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.3f scrub starts 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.3f scrub ok 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.3b scrub starts 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.3b scrub ok 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3129570099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.36 scrub starts 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: 2.36 scrub ok 2024-09-16T05:21:30.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:29 smithi204 ceph-mon[30933]: osdmap e740: 12 total, 12 up, 12 in 2024-09-16T05:21:30.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: pgmap v1269: 127 pgs: 1 activating, 1 peering, 8 unknown, 117 active+clean; 264 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 329 KiB/s wr, 0 op/s 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.3f scrub starts 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.3f scrub ok 2024-09-16T05:21:30.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.3b scrub starts 2024-09-16T05:21:30.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.3b scrub ok 2024-09-16T05:21:30.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:21:30.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:21:30.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3129570099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:30.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.36 scrub starts 2024-09-16T05:21:30.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: 2.36 scrub ok 2024-09-16T05:21:30.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:29 smithi196 ceph-mon[30802]: osdmap e740: 12 total, 12 up, 12 in 2024-09-16T05:21:30.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:30.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:30.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:21:30.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:30.341 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:30.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:30.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:30.466 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:30.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:30.515 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:30.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:30.589 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:30.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:30.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:21:30.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:30.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:30.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:30.823 INFO:tasks.rados.rados.2.smithi204.stdout:2883: done (1 left) 2024-09-16T05:21:30.823 INFO:tasks.rados.rados.2.smithi204.stdout:2884: write (excl) oid 50 current snap is 309 2024-09-16T05:21:30.824 INFO:tasks.rados.rados.2.smithi204.stdout:2884: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2024-09-16T05:21:30.832 INFO:tasks.rados.rados.2.smithi204.stdout:2884: writing smithi20457400-50 from 675912 to 1403845 tid 1 2024-09-16T05:21:30.837 INFO:tasks.rados.rados.2.smithi204.stdout:2884: writing smithi20457400-50 from 1920764 to 2381437 tid 2 2024-09-16T05:21:30.841 INFO:tasks.rados.rados.2.smithi204.stdout:2884: writing smithi20457400-50 from 2801290 to 3075912 tid 3 2024-09-16T05:21:30.841 INFO:tasks.rados.rados.2.smithi204.stdout:2885: delete oid 18 current snap is 309 2024-09-16T05:21:30.841 INFO:tasks.rados.rados.2.smithi204.stdout:2884: finishing write tid 1 to smithi20457400-50 2024-09-16T05:21:30.842 INFO:tasks.rados.rados.2.smithi204.stdout:2884: finishing write tid 2 to smithi20457400-50 2024-09-16T05:21:30.845 INFO:tasks.rados.rados.2.smithi204.stdout:2884: finishing write tid 3 to smithi20457400-50 2024-09-16T05:21:30.845 INFO:tasks.rados.rados.2.smithi204.stdout:2881: finishing write tid 1 to smithi20457400-32 2024-09-16T05:21:30.845 INFO:tasks.rados.rados.2.smithi204.stdout:2884: finishing write tid 4 to smithi20457400-50 2024-09-16T05:21:30.845 INFO:tasks.rados.rados.2.smithi204.stdout:2884: finishing write tid 5 to smithi20457400-50 2024-09-16T05:21:30.845 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1126 (ObjNum 1016 snap 309 seq_num 1016) dirty exists 2024-09-16T05:21:30.846 INFO:tasks.rados.rados.2.smithi204.stdout:2884: left oid 50 (ObjNum 1016 snap 309 seq_num 1016) 2024-09-16T05:21:30.848 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:30.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:30.850 INFO:tasks.rados.rados.2.smithi204.stdout:2881: finishing write tid 2 to smithi20457400-32 2024-09-16T05:21:30.850 INFO:tasks.rados.rados.2.smithi204.stdout:2881: finishing write tid 3 to smithi20457400-32 2024-09-16T05:21:30.851 INFO:tasks.rados.rados.2.smithi204.stdout:2881: finishing write tid 4 to smithi20457400-32 2024-09-16T05:21:30.851 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1132 (ObjNum 1015 snap 307 seq_num 1015) dirty exists 2024-09-16T05:21:30.851 INFO:tasks.rados.rados.2.smithi204.stdout:2881: left oid 32 (ObjNum 1015 snap 307 seq_num 1015) 2024-09-16T05:21:30.852 INFO:tasks.rados.rados.2.smithi204.stdout:2881: done (2 left) 2024-09-16T05:21:30.852 INFO:tasks.rados.rados.2.smithi204.stdout:2884: done (1 left) 2024-09-16T05:21:30.852 INFO:tasks.rados.rados.2.smithi204.stdout:2885: done (0 left) 2024-09-16T05:21:30.852 INFO:tasks.rados.rados.2.smithi204.stdout:2886: rollback oid 29 current snap is 309 2024-09-16T05:21:30.852 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 279 2024-09-16T05:21:30.852 INFO:tasks.rados.rados.2.smithi204.stdout:2887: write oid 34 current snap is 309 2024-09-16T05:21:30.853 INFO:tasks.rados.rados.2.smithi204.stdout:2887: seq_num 1018 ranges {709526=478431,1637728=464342,2856136=253390} 2024-09-16T05:21:30.858 INFO:tasks.rados.rados.2.smithi204.stdout:2887: writing smithi20457400-34 from 709526 to 1187957 tid 1 2024-09-16T05:21:30.863 INFO:tasks.rados.rados.2.smithi204.stdout:2887: writing smithi20457400-34 from 1637728 to 2102070 tid 2 2024-09-16T05:21:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:30.870 INFO:tasks.rados.rados.2.smithi204.stdout:2887: writing smithi20457400-34 from 2856136 to 3109526 tid 3 2024-09-16T05:21:30.870 INFO:tasks.rados.rados.2.smithi204.stdout:2888: rollback oid 2 current snap is 309 2024-09-16T05:21:30.870 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 2 to 308 2024-09-16T05:21:30.870 INFO:tasks.rados.rados.2.smithi204.stdout:2886: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:21:30.870 INFO:tasks.rados.rados.2.smithi204.stdout:2889: read oid 45 snap -1 2024-09-16T05:21:30.870 INFO:tasks.rados.rados.2.smithi204.stdout:2889: expect (ObjNum 985 snap 289 seq_num 985) 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:2890: rollback oid 42 current snap is 309 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 305 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:2886: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:2886: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1135 (ObjNum 935 snap 277 seq_num 935) dirty exists 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:2887: finishing write tid 1 to smithi20457400-34 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:2887: finishing write tid 2 to smithi20457400-34 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:2886: done (4 left) 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:2891: write (excl) oid 1 current snap is 309 2024-09-16T05:21:30.871 INFO:tasks.rados.rados.2.smithi204.stdout:2891: seq_num 1019 ranges {726333=670033,2096210=724353} 2024-09-16T05:21:30.878 INFO:tasks.rados.rados.2.smithi204.stdout:2891: writing smithi20457400-1 from 726333 to 1396366 tid 1 2024-09-16T05:21:30.886 INFO:tasks.rados.rados.2.smithi204.stdout:2891: writing smithi20457400-1 from 2096210 to 2820563 tid 2 2024-09-16T05:21:30.886 INFO:tasks.rados.rados.2.smithi204.stdout:2892: read oid 9 snap -1 2024-09-16T05:21:30.886 INFO:tasks.rados.rados.2.smithi204.stdout:2892: expect (ObjNum 985 snap 289 seq_num 985) 2024-09-16T05:21:30.886 INFO:tasks.rados.rados.2.smithi204.stdout:2893: read oid 40 snap -1 2024-09-16T05:21:30.886 INFO:tasks.rados.rados.2.smithi204.stdout:2893: expect (ObjNum 1011 snap 306 seq_num 1011) 2024-09-16T05:21:30.886 INFO:tasks.rados.rados.2.smithi204.stdout:2888: finishing rollback tid 1 to smithi20457400-2 2024-09-16T05:21:30.887 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 526 (ObjNum 129 snap 0 seq_num 4282409120) dirty dne 2024-09-16T05:21:30.887 INFO:tasks.rados.rados.2.smithi204.stdout:2887: finishing write tid 3 to smithi20457400-34 2024-09-16T05:21:30.887 INFO:tasks.rados.rados.2.smithi204.stdout:2887: finishing write tid 4 to smithi20457400-34 2024-09-16T05:21:30.887 INFO:tasks.rados.rados.2.smithi204.stdout:2888: done (6 left) 2024-09-16T05:21:30.887 INFO:tasks.rados.rados.2.smithi204.stdout:2894: snap_remove snap 289 2024-09-16T05:21:30.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:30.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:30.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:31.005 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:31.006 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:31.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:31.098 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:31.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:31.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 2.19 deep-scrub starts 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 2.19 deep-scrub ok 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 2.57 scrub starts 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 2.57 scrub ok 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 2.58 scrub starts 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 2.58 scrub ok 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:21:31.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:30 smithi032 ceph-mon[27388]: osdmap e741: 12 total, 12 up, 12 in 2024-09-16T05:21:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 2.19 deep-scrub starts 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 2.19 deep-scrub ok 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 2.57 scrub starts 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 2.57 scrub ok 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 2.58 scrub starts 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 2.58 scrub ok 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:21:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:30 smithi204 ceph-mon[30933]: osdmap e741: 12 total, 12 up, 12 in 2024-09-16T05:21:31.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:31.206 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 2.19 deep-scrub starts 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 2.19 deep-scrub ok 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 2.57 scrub starts 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 2.57 scrub ok 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 2.58 scrub starts 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 2.58 scrub ok 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:21:31.207 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:30 smithi196 ceph-mon[30802]: osdmap e741: 12 total, 12 up, 12 in 2024-09-16T05:21:31.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:31.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:31.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:31.347 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:31.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:31.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:31.470 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:31.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:31.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:31.595 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:31.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:31.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:21:31.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:31.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:31.821 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:21:31.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:31.826 INFO:tasks.rados.rados.2.smithi204.stdout:2894: done (6 left) 2024-09-16T05:21:31.826 INFO:tasks.rados.rados.2.smithi204.stdout:2895: copy_from oid 19 from oid 35 current snap is 309 2024-09-16T05:21:31.826 INFO:tasks.rados.rados.2.smithi204.stdout:2890: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:21:31.826 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 782 (ObjNum 49 snap 0 seq_num 4265378928) dirty dne 2024-09-16T05:21:31.827 INFO:tasks.rados.rados.2.smithi204.stdout:2887: finishing write tid 5 to smithi20457400-34 2024-09-16T05:21:31.827 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1259 (ObjNum 1018 snap 309 seq_num 1018) dirty exists 2024-09-16T05:21:31.827 INFO:tasks.rados.rados.2.smithi204.stdout:2887: left oid 34 (ObjNum 1018 snap 309 seq_num 1018) 2024-09-16T05:21:31.827 INFO:tasks.rados.rados.2.smithi204.stdout:2889: expect (ObjNum 985 snap 289 seq_num 985) 2024-09-16T05:21:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:31.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:31.860 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:21:31.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:21:31.860 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10, 1] 2024-09-16T05:21:31.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd out 6 2024-09-16T05:21:31.878 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:31.945 INFO:tasks.rados.rados.2.smithi204.stdout:2891: finishing write tid 1 to smithi20457400-1 2024-09-16T05:21:31.945 INFO:tasks.rados.rados.2.smithi204.stdout:2891: finishing write tid 2 to smithi20457400-1 2024-09-16T05:21:31.945 INFO:tasks.rados.rados.2.smithi204.stdout:2891: finishing write tid 3 to smithi20457400-1 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:2891: finishing write tid 4 to smithi20457400-1 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 565 (ObjNum 1019 snap 309 seq_num 1019) dirty exists 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:2891: left oid 1 (ObjNum 1019 snap 309 seq_num 1019) 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:2887: done (6 left) 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:2889: done (5 left) 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:2890: done (4 left) 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:2891: done (3 left) 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:2896: snap_create 2024-09-16T05:21:31.946 INFO:tasks.rados.rados.2.smithi204.stdout:2893: expect (ObjNum 1011 snap 306 seq_num 1011) 2024-09-16T05:21:31.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:31.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:31.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:32.042 INFO:tasks.rados.rados.2.smithi204.stdout:2892: expect (ObjNum 985 snap 289 seq_num 985) 2024-09-16T05:21:32.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:32.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:32.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:32.158 INFO:tasks.rados.rados.2.smithi204.stdout:2895: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:21:32.158 INFO:tasks.rados.rados.2.smithi204.stdout:2895: finishing copy_from to smithi20457400-19 2024-09-16T05:21:32.159 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 850 (ObjNum 1004 snap 303 seq_num 1004) dirty exists 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: pgmap v1272: 127 pgs: 1 activating, 1 peering, 8 unknown, 117 active+clean; 250 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: 2.59 scrub starts 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: 2.59 scrub ok 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: 2.24 scrub starts 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: 2.24 scrub ok 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3956375860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:21:32.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:21:32.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:32.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3956375860' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:32.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:31 smithi032 ceph-mon[27388]: osdmap e742: 12 total, 12 up, 12 in 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: pgmap v1272: 127 pgs: 1 activating, 1 peering, 8 unknown, 117 active+clean; 250 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: 2.59 scrub starts 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: 2.59 scrub ok 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: 2.24 scrub starts 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: 2.24 scrub ok 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3956375860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:21:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:21:32.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:32.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3956375860' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:32.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:31 smithi204 ceph-mon[30933]: osdmap e742: 12 total, 12 up, 12 in 2024-09-16T05:21:32.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:32.222 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:32.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:32.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: pgmap v1272: 127 pgs: 1 activating, 1 peering, 8 unknown, 117 active+clean; 250 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: 2.59 scrub starts 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: 2.59 scrub ok 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: 2.24 scrub starts 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: 2.24 scrub ok 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3956375860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3956375860' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:31 smithi196 ceph-mon[30802]: osdmap e742: 12 total, 12 up, 12 in 2024-09-16T05:21:32.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:32.341 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:32.341 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:32.345 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:32.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:32.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:32.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:32.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:32.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:32.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:32.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:32.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:32.612 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:32.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:32.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:32.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:32.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:32.826 INFO:tasks.rados.rados.2.smithi204.stdout:2892: done (3 left) 2024-09-16T05:21:32.826 INFO:tasks.rados.rados.2.smithi204.stdout:2893: done (2 left) 2024-09-16T05:21:32.826 INFO:tasks.rados.rados.2.smithi204.stdout:2895: done (1 left) 2024-09-16T05:21:32.826 INFO:tasks.rados.rados.2.smithi204.stdout:2896: done (0 left) 2024-09-16T05:21:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:2897: read oid 2 snap 307 2024-09-16T05:21:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:2897: expect deleted 2024-09-16T05:21:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:2898: write (excl) oid 19 current snap is 310 2024-09-16T05:21:32.827 INFO:tasks.rados.rados.2.smithi204.stdout:2898: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-09-16T05:21:32.834 INFO:tasks.rados.rados.2.smithi204.stdout:2898: writing smithi20457400-19 from 743140 to 1488422 tid 1 2024-09-16T05:21:32.841 INFO:tasks.rados.rados.2.smithi204.stdout:2898: writing smithi20457400-19 from 2154692 to 2622703 tid 2 2024-09-16T05:21:32.841 INFO:tasks.rados.rados.2.smithi204.stdout:2899: read oid 25 snap -1 2024-09-16T05:21:32.841 INFO:tasks.rados.rados.2.smithi204.stdout:2899: expect deleted 2024-09-16T05:21:32.841 INFO:tasks.rados.rados.2.smithi204.stdout:2900: snap_remove snap 303 2024-09-16T05:21:32.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:32.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:32.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:21:32.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:32.969 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:32.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:33.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:33.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:33.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:33.112 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:33.135 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:33.135 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:33.157 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:33.177 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:33.177 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:33.183 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:33.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:33.197 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:33.198 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:33.215 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:33.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:33.219 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:33.219 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:33.240 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:33.240 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:33.262 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:33.262 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:33.283 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:33.283 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:33.304 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:33.305 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:33.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:33.325 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:33.326 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:33.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:33.351 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:33.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:33.373 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:33.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:33.463 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:33.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:33.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:33.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:33.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:33.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:33.673 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:21:33.673 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:21:33.674 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:21:33.674 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:21:33.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:33.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:33.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:33.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:33.833 INFO:tasks.rados.rados.2.smithi204.stdout:2900: done (3 left) 2024-09-16T05:21:33.833 INFO:tasks.rados.rados.2.smithi204.stdout:2901: rollback oid 26 current snap is 310 2024-09-16T05:21:33.833 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 305 2024-09-16T05:21:33.833 INFO:tasks.rados.rados.2.smithi204.stdout:2898: finishing write tid 1 to smithi20457400-19 2024-09-16T05:21:33.833 INFO:tasks.rados.rados.2.smithi204.stdout:2898: finishing write tid 2 to smithi20457400-19 2024-09-16T05:21:33.833 INFO:tasks.rados.rados.2.smithi204.stdout:2898: finishing write tid 3 to smithi20457400-19 2024-09-16T05:21:33.833 INFO:tasks.rados.rados.2.smithi204.stdout:2898: finishing write tid 4 to smithi20457400-19 2024-09-16T05:21:33.834 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 853 (ObjNum 1020 snap 310 seq_num 1020) dirty exists 2024-09-16T05:21:33.834 INFO:tasks.rados.rados.2.smithi204.stdout:2898: left oid 19 (ObjNum 1020 snap 310 seq_num 1020) 2024-09-16T05:21:33.834 INFO:tasks.rados.rados.2.smithi204.stdout:2897: done (3 left) 2024-09-16T05:21:33.834 INFO:tasks.rados.rados.2.smithi204.stdout:2898: done (2 left) 2024-09-16T05:21:33.834 INFO:tasks.rados.rados.2.smithi204.stdout:2899: done (1 left) 2024-09-16T05:21:33.834 INFO:tasks.rados.rados.2.smithi204.stdout:2902: read oid 48 snap -1 2024-09-16T05:21:33.834 INFO:tasks.rados.rados.2.smithi204.stdout:2902: expect (ObjNum 966 snap 285 seq_num 966) 2024-09-16T05:21:33.834 INFO:tasks.rados.rados.2.smithi204.stdout:2903: delete oid 41 current snap is 310 2024-09-16T05:21:33.836 INFO:tasks.rados.rados.2.smithi204.stdout:2903: done (2 left) 2024-09-16T05:21:33.836 INFO:tasks.rados.rados.2.smithi204.stdout:2904: read oid 11 snap -1 2024-09-16T05:21:33.836 INFO:tasks.rados.rados.2.smithi204.stdout:2904: expect (ObjNum 972 snap 287 seq_num 972) 2024-09-16T05:21:33.836 INFO:tasks.rados.rados.2.smithi204.stdout:2905: read oid 14 snap -1 2024-09-16T05:21:33.836 INFO:tasks.rados.rados.2.smithi204.stdout:2905: expect (ObjNum 976 snap 287 seq_num 976) 2024-09-16T05:21:33.836 INFO:tasks.rados.rados.2.smithi204.stdout:2906: read oid 38 snap -1 2024-09-16T05:21:33.836 INFO:tasks.rados.rados.2.smithi204.stdout:2906: expect (ObjNum 963 snap 285 seq_num 963) 2024-09-16T05:21:33.836 INFO:tasks.rados.rados.2.smithi204.stdout:2907: snap_create 2024-09-16T05:21:33.839 INFO:tasks.rados.rados.2.smithi204.stdout:2904: expect (ObjNum 972 snap 287 seq_num 972) 2024-09-16T05:21:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:33.845 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:33.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:33.916 INFO:tasks.rados.rados.2.smithi204.stdout:2901: finishing rollback tid 0 to smithi20457400-26 2024-09-16T05:21:33.916 INFO:tasks.rados.rados.2.smithi204.stdout:2901: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:21:33.916 INFO:tasks.rados.rados.2.smithi204.stdout:2901: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:21:33.916 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1303 (ObjNum 985 snap 289 seq_num 985) dirty exists 2024-09-16T05:21:33.916 INFO:tasks.rados.rados.2.smithi204.stdout:2905: expect (ObjNum 976 snap 287 seq_num 976) 2024-09-16T05:21:33.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:33.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:34.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:33 smithi032 ceph-mon[27388]: pgmap v1275: 127 pgs: 1 active+remapped, 1 peering, 3 remapped+peering, 3 unknown, 119 active+clean; 230 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:34.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:33 smithi032 ceph-mon[27388]: osdmap e743: 12 total, 12 up, 12 in 2024-09-16T05:21:34.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:33 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:34.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:33 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:34.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3059650408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:34.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:34.073 INFO:tasks.rados.rados.2.smithi204.stdout:2906: expect (ObjNum 963 snap 285 seq_num 963) 2024-09-16T05:21:34.092 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:34.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:33 smithi204 ceph-mon[30933]: pgmap v1275: 127 pgs: 1 active+remapped, 1 peering, 3 remapped+peering, 3 unknown, 119 active+clean; 230 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:33 smithi204 ceph-mon[30933]: osdmap e743: 12 total, 12 up, 12 in 2024-09-16T05:21:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:33 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:33 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3059650408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:34.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:34.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:34.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:34.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:33 smithi196 ceph-mon[30802]: pgmap v1275: 127 pgs: 1 active+remapped, 1 peering, 3 remapped+peering, 3 unknown, 119 active+clean; 230 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:33 smithi196 ceph-mon[30802]: osdmap e743: 12 total, 12 up, 12 in 2024-09-16T05:21:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:33 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:33 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3059650408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:34.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:34.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:34 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:21:34.032+0000 7f7c3f992640 -1 mon.a@0(leader).osd e744 definitely_dead 0 2024-09-16T05:21:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:34.340 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:34.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:34.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:21:34.382 INFO:tasks.rados.rados.2.smithi204.stdout:2902: expect (ObjNum 966 snap 285 seq_num 966) 2024-09-16T05:21:34.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:34.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:34.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:34.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:34.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:21:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:34.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:34.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:34.599 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:34.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:34.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:34.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:34.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:34.834 INFO:teuthology.orchestra.run.smithi032.stderr:marked out osd.6. 2024-09-16T05:21:34.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:34.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:34.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:34.843 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:34 smithi032 ceph-mon[27388]: osdmap e744: 12 total, 12 up, 12 in 2024-09-16T05:21:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/435390194' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-16T05:21:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:34 smithi032 ceph-mon[27388]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-16T05:21:34.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:34.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:34.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:34.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:34.969 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:34 smithi196 ceph-mon[30802]: osdmap e744: 12 total, 12 up, 12 in 2024-09-16T05:21:34.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/435390194' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-16T05:21:34.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:34 smithi196 ceph-mon[30802]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-16T05:21:35.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:35.065 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:21:35.065 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:21:35.065 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:21:35.066 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:21:35.066 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:21:35.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:35.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:34 smithi204 ceph-mon[30933]: osdmap e744: 12 total, 12 up, 12 in 2024-09-16T05:21:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/435390194' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-16T05:21:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:34 smithi204 ceph-mon[30933]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-16T05:21:35.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:35.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:35.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:35.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:35.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:35.338 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:35.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:35.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:35.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:35.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:35.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:35.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:35.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:21:35.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:35.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:35.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:35.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:35.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:35.836 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:35.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:35.842 INFO:tasks.rados.rados.2.smithi204.stdout:2901: done (5 left) 2024-09-16T05:21:35.842 INFO:tasks.rados.rados.2.smithi204.stdout:2902: done (4 left) 2024-09-16T05:21:35.842 INFO:tasks.rados.rados.2.smithi204.stdout:2904: done (3 left) 2024-09-16T05:21:35.842 INFO:tasks.rados.rados.2.smithi204.stdout:2905: done (2 left) 2024-09-16T05:21:35.842 INFO:tasks.rados.rados.2.smithi204.stdout:2906: done (1 left) 2024-09-16T05:21:35.842 INFO:tasks.rados.rados.2.smithi204.stdout:2907: done (0 left) 2024-09-16T05:21:35.842 INFO:tasks.rados.rados.2.smithi204.stdout:2908: delete oid 26 current snap is 311 2024-09-16T05:21:35.852 INFO:tasks.rados.rados.2.smithi204.stdout:2908: done (0 left) 2024-09-16T05:21:35.852 INFO:tasks.rados.rados.2.smithi204.stdout:2909: delete oid 7 current snap is 311 2024-09-16T05:21:35.854 INFO:tasks.rados.rados.2.smithi204.stdout:2909: done (0 left) 2024-09-16T05:21:35.854 INFO:tasks.rados.rados.2.smithi204.stdout:2910: snap_remove snap 310 2024-09-16T05:21:35.879 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:21:35.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:35.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:35.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:35.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:35 smithi196 ceph-mon[30802]: pgmap v1278: 127 pgs: 1 active+remapped, 3 remapped+peering, 2 unknown, 121 active+clean; 240 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:35.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/435390194' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-16T05:21:35.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:35 smithi196 ceph-mon[30802]: osdmap e745: 12 total, 12 up, 11 in 2024-09-16T05:21:36.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:35 smithi032 ceph-mon[27388]: pgmap v1278: 127 pgs: 1 active+remapped, 3 remapped+peering, 2 unknown, 121 active+clean; 240 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/435390194' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-16T05:21:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:35 smithi032 ceph-mon[27388]: osdmap e745: 12 total, 12 up, 11 in 2024-09-16T05:21:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:35 smithi204 ceph-mon[30933]: pgmap v1278: 127 pgs: 1 active+remapped, 3 remapped+peering, 2 unknown, 121 active+clean; 240 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:21:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/435390194' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-16T05:21:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:35 smithi204 ceph-mon[30933]: osdmap e745: 12 total, 12 up, 11 in 2024-09-16T05:21:36.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:36.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:36.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:36.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:36.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:21:36.365 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:21:36.365 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:21:36.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:21:36.365 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:21:36.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1824 2.8 2:1030fb5b:::smithi20457400-27:head [write 521083~714336,stat] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 517.41157893100001, 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.81065307900000005, 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1824 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.366 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945150+0000", 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1260000000000001e-06 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945906+0000", 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075638699999999997 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945927+0000", 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0849e-05 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945936+0000", 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3910000000000004e-06 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946023+0000", 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6547999999999999e-05 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946035+0000", 2024-09-16T05:21:36.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2249000000000001e-05 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752337+0000", 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80630184599999999 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752493+0000", 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000156546 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752675+0000", 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018172699999999999 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755628+0000", 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029533929999999999 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755689+0000", 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0949000000000002e-05 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755745+0000", 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5591999999999999e-05 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755800+0000", 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5476000000000002e-05 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1825 2.8 2:1030fb5b:::smithi20457400-27:head [write 1724715~555421] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 517.40593338300005, 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80511571699999995, 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1825 2024-09-16T05:21:36.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950794+0000", 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951237+0000", 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00044250000000000002 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951251+0000", 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4381e-05 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951276+0000", 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5233999999999999e-05 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951378+0000", 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010173799999999999 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951390+0000", 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2274e-05 2024-09-16T05:21:36.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.753903+0000", 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80251229300000004 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754002+0000", 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9427999999999997e-05 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754132+0000", 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130207 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755633+0000", 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015010100000000001 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755844+0000", 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021044799999999999 2024-09-16T05:21:36.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755880+0000", 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6007999999999998e-05 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755909+0000", 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8941e-05 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1826 2.8 2:1030fb5b:::smithi20457400-27:head [setxattr _header (57) in=7b,truncate 2521083] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 517.40543408999997, 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80476365000000005, 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.380 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1826 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951293+0000", 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951296+0000", 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2799999999999999e-06 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951301+0000", 2024-09-16T05:21:36.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6879999999999998e-06 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951309+0000", 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7810000000000005e-06 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951421+0000", 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112333 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951426+0000", 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6650000000000002e-06 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754913+0000", 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80348719099999999 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754982+0000", 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9558e-05 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755084+0000", 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101894 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755638+0000", 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055423099999999997 2024-09-16T05:21:36.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755988+0000", 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034971600000000002 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756021+0000", 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2461e-05 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756056+0000", 2024-09-16T05:21:36.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5278999999999999e-05 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1827 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 517.40541182100003, 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.804892304, 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.387 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1827 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8700000000000002e-07 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951317+0000", 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7460000000000001e-06 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951321+0000", 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3250000000000001e-06 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951325+0000", 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1710000000000004e-06 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951464+0000", 2024-09-16T05:21:36.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139111 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951475+0000", 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1045e-05 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755294+0000", 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80381846999999995 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755351+0000", 2024-09-16T05:21:36.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7589e-05 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756086+0000", 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073505599999999999 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756138+0000", 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1397000000000003e-05 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756207+0000", 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9107000000000001e-05 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1834 2.8 2:12e92f6c:::smithi20457400-1:head [stat,delete] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 517.15851113899998, 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55807496000000001, 2024-09-16T05:21:36.393 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1834 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198218+0000", 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.357e-06 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198229+0000", 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1677e-05 2024-09-16T05:21:36.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198254+0000", 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4312e-05 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198266+0000", 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2687000000000001e-05 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198366+0000", 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9698000000000007e-05 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198378+0000", 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2106000000000001e-05 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755379+0000", 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55700064400000004 2024-09-16T05:21:36.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755448+0000", 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8848000000000002e-05 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756290+0000", 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084263100000000004 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2255 2.1d e290/282 2:ba754fcb:::smithi20457400-30:head v 290'477, mlcod=284'471)", 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 474.67191639200001, 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53570759499999998, 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684811+0000", 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3600000000000002e-07 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684819+0000", 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5760000000000006e-06 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219817+0000", 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53499761300000004 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219843+0000", 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5908e-05 2024-09-16T05:21:36.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219921+0000", 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7811000000000004e-05 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219944+0000", 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3237000000000001e-05 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220484+0000", 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00053966300000000001 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220518+0000", 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3850999999999998e-05 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.10.0:85 2.1d e300/291 2:ba754fcb:::smithi20457400-30:5f v 300'498, mlcod=300'494)", 2024-09-16T05:21:36.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 464.598089749, 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.51609034499999995, 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758637+0000", 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.16e-07 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:51.758644+0000", 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1519999999999997e-06 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273721+0000", 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.51507724799999999 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273736+0000", 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5449000000000001e-05 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273906+0000", 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016944 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.273962+0000", 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6209999999999999e-05 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.274683+0000", 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00072146099999999996 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:52.274727+0000", 2024-09-16T05:21:36.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3569000000000002e-05 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:390 2.3c e346/293 ondisk, result = 0)", 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 400.64914440299998, 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62913221100000005, 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707583+0000", 2024-09-16T05:21:36.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.47e-07 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707587+0000", 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7010000000000004e-06 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336491+0000", 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62890340600000005 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336497+0000", 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4640000000000003e-06 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336590+0000", 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2163000000000005e-05 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336599+0000", 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.054e-06 2024-09-16T05:21:36.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336714+0000", 2024-09-16T05:21:36.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011557600000000001 2024-09-16T05:21:36.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.413 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.414 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4414 2.3f 2:fc733d2f:::smithi20457400-45:head [copy-from ver 329] snapc b9=[b9,b8] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-16T05:21:36.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:21:36.414 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 247.613723685, 2024-09-16T05:21:36.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95144123899999999, 2024-09-16T05:21:36.414 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.414 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4414 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743006+0000", 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2550000000000002e-06 2024-09-16T05:21:36.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743021+0000", 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5024999999999999e-05 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743048+0000", 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6842000000000002e-05 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743063+0000", 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4969e-05 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743154+0000", 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1438000000000001e-05 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743163+0000", 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1740000000000006e-06 2024-09-16T05:21:36.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.744798+0000", 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016347600000000001 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.744906+0000", 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107724 2024-09-16T05:21:36.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749771+0000", 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048651129999999999 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.753713+0000", 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039421539999999998 2024-09-16T05:21:36.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694348+0000", 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94063521699999997 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694398+0000", 2024-09-16T05:21:36.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9518999999999998e-05 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694444+0000", 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6048999999999999e-05 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(1.9 e106 from shard 9)", 2024-09-16T05:21:36.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 193.87895320199999, 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.63215181600000003, 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477774+0000", 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9999999999999997e-07 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477777+0000", 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5949999999999999e-06 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109802+0000", 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.63202490499999997 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109829+0000", 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6750999999999998e-05 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109891+0000", 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1661999999999993e-05 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109900+0000", 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6569999999999992e-06 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109925+0000", 2024-09-16T05:21:36.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4646000000000001e-05 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:958 2.2e e588/278 ondisk, result = 0)", 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 158.564179725, 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92946010700000004, 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.06e-07 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792556+0000", 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.208e-06 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721793+0000", 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92923791899999997 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721804+0000", 2024-09-16T05:21:36.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0936e-05 2024-09-16T05:21:36.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721875+0000", 2024-09-16T05:21:36.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0863000000000002e-05 2024-09-16T05:21:36.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721885+0000", 2024-09-16T05:21:36.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5510000000000005e-06 2024-09-16T05:21:36.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.722007+0000", 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012192399999999999 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:5901 2.1d e608/533 2:ba754fcb:::smithi20457400-30:head v 608'1064, mlcod=608'1061)", 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:17.903833+0000", 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 138.45289349800001, 2024-09-16T05:21:36.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.79383940900000005, 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:17.903833+0000", 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:17.903833+0000", 2024-09-16T05:21:36.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:17.903833+0000", 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:17.903838+0000", 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9269999999999996e-06 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.696799+0000", 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.79296047700000005 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.696807+0000", 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5210000000000002e-06 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.696876+0000", 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8779e-05 2024-09-16T05:21:36.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.696897+0000", 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1141999999999998e-05 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.697637+0000", 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073942099999999998 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.697672+0000", 2024-09-16T05:21:36.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5589000000000001e-05 2024-09-16T05:21:36.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.438 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.438 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5976 2.8 2:1030fb5b:::smithi20457400-27:head [setxattr _header (58) in=7b,truncate 2235529] snapc f7=[f7,f6,f5,f2,f0,ef,ed,e8,e7,e5,e0,ca] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-09-16T05:21:36.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.795849+0000", 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 129.56087735200001, 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75718527599999996, 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5976 2024-09-16T05:21:36.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795849+0000", 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795849+0000", 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795850+0000", 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1500000000000003e-07 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795854+0000", 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7479999999999999e-06 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795862+0000", 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1429999999999998e-06 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795869+0000", 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1770000000000001e-06 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.796994+0000", 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011255149999999999 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.797063+0000", 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8616999999999997e-05 2024-09-16T05:21:36.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.797164+0000", 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101081 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.800936+0000", 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003771445 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.552882+0000", 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75194649300000005 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.552958+0000", 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5736999999999997e-05 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.553034+0000", 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6404999999999998e-05 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.445 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.446 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.446 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5977 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e617)", 2024-09-16T05:21:36.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.795882+0000", 2024-09-16T05:21:36.446 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 129.56084471200001, 2024-09-16T05:21:36.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75735637700000003, 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5977 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795882+0000", 2024-09-16T05:21:36.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795882+0000", 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795882+0000", 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7599999999999997e-07 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795885+0000", 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7199999999999998e-06 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795892+0000", 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4399999999999999e-06 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795898+0000", 2024-09-16T05:21:36.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7490000000000004e-06 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.797481+0000", 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015833010000000001 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.797561+0000", 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9832000000000004e-05 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.553077+0000", 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75551588999999997 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.553153+0000", 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5705999999999996e-05 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.553238+0000", 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5162999999999998e-05 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGPushReply(2.3c 626/625 [PushReplyOp(2:3d4bedc8:::smithi20457400-22:f7)])", 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:35.873422+0000", 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 120.483304416, 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.76198344399999995, 2024-09-16T05:21:36.452 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:35.873422+0000", 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:35.873422+0000", 2024-09-16T05:21:36.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:35.873422+0000", 2024-09-16T05:21:36.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7400000000000003e-07 2024-09-16T05:21:36.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:35.873426+0000", 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9489999999999998e-06 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.634292+0000", 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.76086524600000005 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.634298+0000", 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.251e-06 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.634368+0000", 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0129000000000006e-05 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.635405+0000", 2024-09-16T05:21:36.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010372949999999999 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGPushReply(2.3c 626/625 [PushReplyOp(2:3d4bedc8:::smithi20457400-22:f7)])", 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:36.637827+0000", 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 119.718899345, 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.82730668500000004, 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.457 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:21:36.458 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.637827+0000", 2024-09-16T05:21:36.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.637827+0000", 2024-09-16T05:21:36.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.637828+0000", 2024-09-16T05:21:36.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5199999999999995e-07 2024-09-16T05:21:36.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.637833+0000", 2024-09-16T05:21:36.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2089999999999996e-06 2024-09-16T05:21:36.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:37.464824+0000", 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.82699144300000005 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:37.464837+0000", 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2396e-05 2024-09-16T05:21:36.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:37.464925+0000", 2024-09-16T05:21:36.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7723e-05 2024-09-16T05:21:36.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:37.465134+0000", 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020906200000000001 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6760 2.2d 2:b7234079:::smithi20457400-42:head [setxattr _header (58) in=7b,truncate 3865808] snapc 117=[117,116,113,112,10c,10b,10a,109,108,106,101,f2] ondisk+write+known_if_redirected+supports_pool_eio e681)", 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:30.900604+0000", 2024-09-16T05:21:36.464 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 65.456121831000004, 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.66116237099999997, 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6760 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900604+0000", 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900604+0000", 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900605+0000", 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1900000000000002e-07 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900607+0000", 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.261e-06 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900612+0000", 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.216e-06 2024-09-16T05:21:36.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900615+0000", 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1949999999999999e-06 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901352+0000", 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073710400000000004 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901409+0000", 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6903000000000001e-05 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901493+0000", 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3992000000000005e-05 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901911+0000", 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041862900000000001 2024-09-16T05:21:36.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561576+0000", 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65966484700000005 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561678+0000", 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102012 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561767+0000", 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8592999999999997e-05 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6761 2.2d 2:b7234079:::smithi20457400-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e681)", 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:30.900627+0000", 2024-09-16T05:21:36.471 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 65.456099730000005, 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.66137536900000005, 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6761 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.472 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900627+0000", 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900627+0000", 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900627+0000", 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9799999999999999e-07 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900628+0000", 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.488e-06 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900637+0000", 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0710000000000001e-06 2024-09-16T05:21:36.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900642+0000", 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5410000000000002e-06 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901676+0000", 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001033647 2024-09-16T05:21:36.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901720+0000", 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4282000000000001e-05 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561816+0000", 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.66009643299999998 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561898+0000", 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1323999999999996e-05 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.562002+0000", 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104285 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.477 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.478 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:6760 2.2d e681/252 ondisk, result = 0)", 2024-09-16T05:21:36.478 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:30.902124+0000", 2024-09-16T05:21:36.478 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 65.454602116999993, 2024-09-16T05:21:36.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65965149099999998, 2024-09-16T05:21:36.478 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.479 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.479 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.479 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.902124+0000", 2024-09-16T05:21:36.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.902124+0000", 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.902125+0000", 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.902129+0000", 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7809999999999999e-06 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561389+0000", 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65926033500000003 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561410+0000", 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1041999999999999e-05 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561555+0000", 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014517299999999999 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561569+0000", 2024-09-16T05:21:36.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.414e-05 2024-09-16T05:21:36.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561776+0000", 2024-09-16T05:21:36.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020644700000000001 2024-09-16T05:21:36.483 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:1258 2.3c e696/625 ondisk, result = 0)", 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:45.814351+0000", 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 50.542375644000003, 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96819708100000001, 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:21:36.484 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:45.814351+0000", 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:45.814351+0000", 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:45.814351+0000", 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8599999999999998e-07 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:45.814361+0000", 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8870000000000004e-06 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782270+0000", 2024-09-16T05:21:36.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96790891700000004 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782285+0000", 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5265999999999999e-05 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782425+0000", 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139245 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782435+0000", 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0769e-05 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782548+0000", 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112311 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.488 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.489 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:21:36.489 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:21:36.489 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:21:36.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:36.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:36.598 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:36.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:36.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:36.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:36.759 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:36.848 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:21:36.858 INFO:tasks.rados.rados.2.smithi204.stdout:2910: done (0 left) 2024-09-16T05:21:36.858 INFO:tasks.rados.rados.2.smithi204.stdout:2911: write oid 8 current snap is 311 2024-09-16T05:21:36.858 INFO:tasks.rados.rados.2.smithi204.stdout:2911: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2024-09-16T05:21:36.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:36.864 INFO:tasks.rados.rados.2.smithi204.stdout:2911: writing smithi20457400-8 from 410368 to 1056646 tid 1 2024-09-16T05:21:36.868 INFO:tasks.rados.rados.2.smithi204.stdout:2911: writing smithi20457400-8 from 1472267 to 1947616 tid 2 2024-09-16T05:21:36.874 INFO:tasks.rados.rados.2.smithi204.stdout:2911: writing smithi20457400-8 from 2504321 to 2909921 tid 3 2024-09-16T05:21:36.874 INFO:tasks.rados.rados.2.smithi204.stdout:2912: write (excl) oid 2 current snap is 311 2024-09-16T05:21:36.874 INFO:tasks.rados.rados.2.smithi204.stdout:2911: finishing write tid 1 to smithi20457400-8 2024-09-16T05:21:36.874 INFO:tasks.rados.rados.2.smithi204.stdout:2912: seq_num 1025 ranges {427175=721527,1930749=735360} 2024-09-16T05:21:36.880 INFO:tasks.rados.rados.2.smithi204.stdout:2912: writing smithi20457400-2 from 427175 to 1148702 tid 1 2024-09-16T05:21:36.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:36.889 INFO:tasks.rados.rados.2.smithi204.stdout:2912: writing smithi20457400-2 from 1930749 to 2666109 tid 2 2024-09-16T05:21:36.890 INFO:tasks.rados.rados.2.smithi204.stdout:2913: write (excl) oid 18 current snap is 311 2024-09-16T05:21:36.890 INFO:tasks.rados.rados.2.smithi204.stdout:2911: finishing write tid 2 to smithi20457400-8 2024-09-16T05:21:36.890 INFO:tasks.rados.rados.2.smithi204.stdout:2913: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2024-09-16T05:21:36.896 INFO:tasks.rados.rados.2.smithi204.stdout:2913: writing smithi20457400-18 from 443982 to 1240758 tid 1 2024-09-16T05:21:36.903 INFO:tasks.rados.rados.2.smithi204.stdout:2913: writing smithi20457400-18 from 1989231 to 2468249 tid 2 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2913: writing smithi20457400-18 from 2959167 to 3243982 tid 3 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2914: snap_create 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2911: finishing write tid 3 to smithi20457400-8 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2911: finishing write tid 4 to smithi20457400-8 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2911: finishing write tid 5 to smithi20457400-8 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1263 (ObjNum 1024 snap 311 seq_num 1024) dirty exists 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2911: left oid 8 (ObjNum 1024 snap 311 seq_num 1024) 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2912: finishing write tid 1 to smithi20457400-2 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2912: finishing write tid 2 to smithi20457400-2 2024-09-16T05:21:36.908 INFO:tasks.rados.rados.2.smithi204.stdout:2912: finishing write tid 3 to smithi20457400-2 2024-09-16T05:21:36.909 INFO:tasks.rados.rados.2.smithi204.stdout:2912: finishing write tid 4 to smithi20457400-2 2024-09-16T05:21:36.909 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 529 (ObjNum 1025 snap 311 seq_num 1025) dirty exists 2024-09-16T05:21:36.909 INFO:tasks.rados.rados.2.smithi204.stdout:2912: left oid 2 (ObjNum 1025 snap 311 seq_num 1025) 2024-09-16T05:21:36.944 INFO:tasks.rados.rados.2.smithi204.stdout:2913: finishing write tid 1 to smithi20457400-18 2024-09-16T05:21:36.945 INFO:tasks.rados.rados.2.smithi204.stdout:2913: finishing write tid 2 to smithi20457400-18 2024-09-16T05:21:36.945 INFO:tasks.rados.rados.2.smithi204.stdout:2913: finishing write tid 3 to smithi20457400-18 2024-09-16T05:21:36.945 INFO:tasks.rados.rados.2.smithi204.stdout:2913: finishing write tid 4 to smithi20457400-18 2024-09-16T05:21:36.946 INFO:tasks.rados.rados.2.smithi204.stdout:2913: finishing write tid 5 to smithi20457400-18 2024-09-16T05:21:36.946 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 547 (ObjNum 1026 snap 311 seq_num 1026) dirty exists 2024-09-16T05:21:36.946 INFO:tasks.rados.rados.2.smithi204.stdout:2913: left oid 18 (ObjNum 1026 snap 311 seq_num 1026) 2024-09-16T05:21:36.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:36.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:36 smithi196 ceph-mon[30802]: osdmap e746: 12 total, 12 up, 11 in 2024-09-16T05:21:36.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2209851181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:37.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:37.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:37.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:37.131 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:37.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:37.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:21:37.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:36 smithi032 ceph-mon[27388]: osdmap e746: 12 total, 12 up, 11 in 2024-09-16T05:21:37.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2209851181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:37.156 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:36 smithi204 ceph-mon[30933]: osdmap e746: 12 total, 12 up, 11 in 2024-09-16T05:21:37.157 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2209851181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:37.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:37.255 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:37.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:37.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:37.373 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:37.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:37.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:37.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:37.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:37.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:37.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:37.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:37.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:37.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:37.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:37.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:37.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:2911: done (3 left) 2024-09-16T05:21:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:2912: done (2 left) 2024-09-16T05:21:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:2913: done (1 left) 2024-09-16T05:21:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:2914: done (0 left) 2024-09-16T05:21:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:2915: copy_from oid 38 from oid 26 current snap is 312 2024-09-16T05:21:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:2916: snap_remove snap 307 2024-09-16T05:21:37.861 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:37.861 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:37.876 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:37.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:37.883 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:37.883 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:37.903 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:37.903 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:37.909 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:21:37.909 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:21:37.909 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:21:37.909 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:21:37.924 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:37.924 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:37.945 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:37.945 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:37.966 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:37.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:37.988 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:38.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:38.010 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:38.010 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:38.030 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:38.052 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:38.053 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:38.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:37 smithi204 ceph-mon[30933]: pgmap v1281: 127 pgs: 5 peering, 122 active+clean; 254 MiB data, 880 MiB used, 983 GiB / 983 GiB avail; 43 MiB/s rd, 16 MiB/s wr, 101 op/s; 4.3 MiB/s, 36 objects/s recovering 2024-09-16T05:21:38.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:37 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:38.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2209851181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:38.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:37 smithi204 ceph-mon[30933]: osdmap e747: 12 total, 12 up, 11 in 2024-09-16T05:21:38.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:37 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:38.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:37 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:38.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3515482042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:38.074 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:38.074 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:38.095 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:38.096 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:38.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:37 smithi032 ceph-mon[27388]: pgmap v1281: 127 pgs: 5 peering, 122 active+clean; 254 MiB data, 880 MiB used, 983 GiB / 983 GiB avail; 43 MiB/s rd, 16 MiB/s wr, 101 op/s; 4.3 MiB/s, 36 objects/s recovering 2024-09-16T05:21:38.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:37 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:38.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2209851181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:38.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:37 smithi032 ceph-mon[27388]: osdmap e747: 12 total, 12 up, 11 in 2024-09-16T05:21:38.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:37 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:38.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:37 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:38.097 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3515482042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:38.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:38.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:37 smithi196 ceph-mon[30802]: pgmap v1281: 127 pgs: 5 peering, 122 active+clean; 254 MiB data, 880 MiB used, 983 GiB / 983 GiB avail; 43 MiB/s rd, 16 MiB/s wr, 101 op/s; 4.3 MiB/s, 36 objects/s recovering 2024-09-16T05:21:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:37 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2209851181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:37 smithi196 ceph-mon[30802]: osdmap e747: 12 total, 12 up, 11 in 2024-09-16T05:21:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:37 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:37 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3515482042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:38.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:38.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:38.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:38.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:38.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:38.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:38.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:21:38.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:38.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:38.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:38.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:38.863 INFO:tasks.rados.rados.2.smithi204.stdout:2916: done (1 left) 2024-09-16T05:21:38.863 INFO:tasks.rados.rados.2.smithi204.stdout:2917: write oid 37 current snap is 312 2024-09-16T05:21:38.863 INFO:tasks.rados.rados.2.smithi204.stdout:2917: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2024-09-16T05:21:38.871 INFO:tasks.rados.rados.2.smithi204.stdout:2917: writing smithi20457400-37 from 460789 to 1049167 tid 1 2024-09-16T05:21:38.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:38.878 INFO:tasks.rados.rados.2.smithi204.stdout:2917: writing smithi20457400-37 from 1764066 to 2503095 tid 2 2024-09-16T05:21:38.882 INFO:tasks.rados.rados.2.smithi204.stdout:2917: writing smithi20457400-37 from 3219296 to 3260789 tid 3 2024-09-16T05:21:38.882 INFO:tasks.rados.rados.2.smithi204.stdout:2918: snap_create 2024-09-16T05:21:38.882 INFO:tasks.rados.rados.2.smithi204.stdout:2915: finishing copy_from to smithi20457400-38 2024-09-16T05:21:38.882 INFO:tasks.rados.rados.2.smithi204.stdout:2915: got expected ENOENT (src dne) 2024-09-16T05:21:38.883 INFO:tasks.rados.rados.2.smithi204.stdout:2915: finishing copy_from racing read to smithi20457400-38 2024-09-16T05:21:38.883 INFO:tasks.rados.rados.2.smithi204.stdout:2917: finishing write tid 1 to smithi20457400-37 2024-09-16T05:21:38.883 INFO:tasks.rados.rados.2.smithi204.stdout:2917: finishing write tid 2 to smithi20457400-37 2024-09-16T05:21:38.884 INFO:tasks.rados.rados.2.smithi204.stdout:2917: finishing write tid 3 to smithi20457400-37 2024-09-16T05:21:38.884 INFO:tasks.rados.rados.2.smithi204.stdout:2917: finishing write tid 4 to smithi20457400-37 2024-09-16T05:21:38.884 INFO:tasks.rados.rados.2.smithi204.stdout:2917: finishing write tid 5 to smithi20457400-37 2024-09-16T05:21:38.884 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1037 (ObjNum 1027 snap 312 seq_num 1027) dirty exists 2024-09-16T05:21:38.885 INFO:tasks.rados.rados.2.smithi204.stdout:2917: left oid 37 (ObjNum 1027 snap 312 seq_num 1027) 2024-09-16T05:21:38.916 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:38.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:38.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:38.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:39.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:39.122 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:39.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:39.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:38 smithi196 ceph-mon[30802]: osdmap e748: 12 total, 12 up, 11 in 2024-09-16T05:21:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:38 smithi032 ceph-mon[27388]: osdmap e748: 12 total, 12 up, 11 in 2024-09-16T05:21:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:38 smithi204 ceph-mon[30933]: osdmap e748: 12 total, 12 up, 11 in 2024-09-16T05:21:39.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:39.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:39.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:39.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:39.375 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:39.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:39.381 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:39.381 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:39.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:39.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:39.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:21:39.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:39.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:39.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:39.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:39.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:39.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:39.832 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:39.859 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:39.866 INFO:tasks.rados.rados.2.smithi204.stdout:2915: done (2 left) 2024-09-16T05:21:39.867 INFO:tasks.rados.rados.2.smithi204.stdout:2917: done (1 left) 2024-09-16T05:21:39.867 INFO:tasks.rados.rados.2.smithi204.stdout:2918: done (0 left) 2024-09-16T05:21:39.867 INFO:tasks.rados.rados.2.smithi204.stdout:2919: write oid 35 current snap is 313 2024-09-16T05:21:39.867 INFO:tasks.rados.rados.2.smithi204.stdout:2919: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-09-16T05:21:39.873 INFO:tasks.rados.rados.2.smithi204.stdout:2919: writing smithi20457400-35 from 477596 to 1141223 tid 1 2024-09-16T05:21:39.879 INFO:tasks.rados.rados.2.smithi204.stdout:2919: writing smithi20457400-35 from 1706195 to 2188882 tid 2 2024-09-16T05:21:39.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:39.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:39.885 INFO:tasks.rados.rados.2.smithi204.stdout:2919: writing smithi20457400-35 from 2614013 to 3181054 tid 3 2024-09-16T05:21:39.885 INFO:tasks.rados.rados.2.smithi204.stdout:2920: write oid 12 current snap is 313 2024-09-16T05:21:39.885 INFO:tasks.rados.rados.2.smithi204.stdout:2919: finishing write tid 1 to smithi20457400-35 2024-09-16T05:21:39.885 INFO:tasks.rados.rados.2.smithi204.stdout:2920: seq_num 1029 ranges {494403=738876,1764677=626345,3041436=252967} 2024-09-16T05:21:39.891 INFO:tasks.rados.rados.2.smithi204.stdout:2920: writing smithi20457400-12 from 494403 to 1233279 tid 1 2024-09-16T05:21:39.898 INFO:tasks.rados.rados.2.smithi204.stdout:2920: writing smithi20457400-12 from 1764677 to 2391022 tid 2 2024-09-16T05:21:39.903 INFO:tasks.rados.rados.2.smithi204.stdout:2920: writing smithi20457400-12 from 3041436 to 3294403 tid 3 2024-09-16T05:21:39.903 INFO:tasks.rados.rados.2.smithi204.stdout:2921: rollback oid 25 current snap is 313 2024-09-16T05:21:39.903 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 305 2024-09-16T05:21:39.903 INFO:tasks.rados.rados.2.smithi204.stdout:2919: finishing write tid 2 to smithi20457400-35 2024-09-16T05:21:39.903 INFO:tasks.rados.rados.2.smithi204.stdout:2919: finishing write tid 3 to smithi20457400-35 2024-09-16T05:21:39.903 INFO:tasks.rados.rados.2.smithi204.stdout:2919: finishing write tid 4 to smithi20457400-35 2024-09-16T05:21:39.903 INFO:tasks.rados.rados.2.smithi204.stdout:2919: finishing write tid 5 to smithi20457400-35 2024-09-16T05:21:39.904 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 854 (ObjNum 1028 snap 313 seq_num 1028) dirty exists 2024-09-16T05:21:39.904 INFO:tasks.rados.rados.2.smithi204.stdout:2919: left oid 35 (ObjNum 1028 snap 313 seq_num 1028) 2024-09-16T05:21:39.904 INFO:tasks.rados.rados.2.smithi204.stdout:2919: done (2 left) 2024-09-16T05:21:39.904 INFO:tasks.rados.rados.2.smithi204.stdout:2922: snap_create 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:2921: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 734 (ObjNum 129 snap 0 seq_num 4283043352) dirty dne 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:2920: finishing write tid 1 to smithi20457400-12 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:2920: finishing write tid 2 to smithi20457400-12 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:2920: finishing write tid 3 to smithi20457400-12 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:2920: finishing write tid 4 to smithi20457400-12 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:2920: finishing write tid 5 to smithi20457400-12 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1150 (ObjNum 1029 snap 313 seq_num 1029) dirty exists 2024-09-16T05:21:39.905 INFO:tasks.rados.rados.2.smithi204.stdout:2920: left oid 12 (ObjNum 1029 snap 313 seq_num 1029) 2024-09-16T05:21:39.955 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:39 smithi204 ceph-mon[30933]: pgmap v1284: 127 pgs: 5 peering, 1 active+recovering, 121 active+clean; 256 MiB data, 879 MiB used, 983 GiB / 983 GiB avail; 43 MiB/s rd, 14 MiB/s wr, 92 op/s; 4/244 objects misplaced (1.639%); 10 MiB/s, 62 objects/s recovering 2024-09-16T05:21:39.955 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:39 smithi204 ceph-mon[30933]: osdmap e749: 12 total, 12 up, 11 in 2024-09-16T05:21:39.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:21:39.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:40.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:40.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:40.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:40.132 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:40.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:39 smithi032 ceph-mon[27388]: pgmap v1284: 127 pgs: 5 peering, 1 active+recovering, 121 active+clean; 256 MiB data, 879 MiB used, 983 GiB / 983 GiB avail; 43 MiB/s rd, 14 MiB/s wr, 92 op/s; 4/244 objects misplaced (1.639%); 10 MiB/s, 62 objects/s recovering 2024-09-16T05:21:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:39 smithi032 ceph-mon[27388]: osdmap e749: 12 total, 12 up, 11 in 2024-09-16T05:21:40.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:39 smithi196 ceph-mon[30802]: pgmap v1284: 127 pgs: 5 peering, 1 active+recovering, 121 active+clean; 256 MiB data, 879 MiB used, 983 GiB / 983 GiB avail; 43 MiB/s rd, 14 MiB/s wr, 92 op/s; 4/244 objects misplaced (1.639%); 10 MiB/s, 62 objects/s recovering 2024-09-16T05:21:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:39 smithi196 ceph-mon[30802]: osdmap e749: 12 total, 12 up, 11 in 2024-09-16T05:21:40.256 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:40.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:40.261 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:40.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:40.380 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:40.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:40.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:40.505 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:40.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:40.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:40.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:40.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:21:40.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:40.767 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:40.767 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:40.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:40.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:40.810 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:21:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2920: done (2 left) 2024-09-16T05:21:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2921: done (1 left) 2024-09-16T05:21:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2922: done (0 left) 2024-09-16T05:21:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2923: rollback oid 15 current snap is 314 2024-09-16T05:21:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 15 to 279 2024-09-16T05:21:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2924: read oid 46 snap -1 2024-09-16T05:21:40.814 INFO:tasks.rados.rados.2.smithi204.stdout:2924: expect (ObjNum 969 snap 287 seq_num 969) 2024-09-16T05:21:40.815 INFO:tasks.rados.rados.2.smithi204.stdout:2925: rollback oid 43 current snap is 314 2024-09-16T05:21:40.815 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 43 to 296 2024-09-16T05:21:40.815 INFO:tasks.rados.rados.2.smithi204.stdout:2926: snap_create 2024-09-16T05:21:40.819 INFO:tasks.rados.rados.2.smithi204.stdout:2923: finishing rollback tid 0 to smithi20457400-15 2024-09-16T05:21:40.819 INFO:tasks.rados.rados.2.smithi204.stdout:2923: finishing rollback tid 1 to smithi20457400-15 2024-09-16T05:21:40.819 INFO:tasks.rados.rados.2.smithi204.stdout:2923: finishing rollback tid 2 to smithi20457400-15 2024-09-16T05:21:40.819 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 747 (ObjNum 906 snap 267 seq_num 906) dirty exists 2024-09-16T05:21:40.819 INFO:tasks.rados.rados.2.smithi204.stdout:2925: finishing rollback tid 0 to smithi20457400-43 2024-09-16T05:21:40.819 INFO:tasks.rados.rados.2.smithi204.stdout:2925: finishing rollback tid 1 to smithi20457400-43 2024-09-16T05:21:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:2925: finishing rollback tid 2 to smithi20457400-43 2024-09-16T05:21:40.820 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 660 (ObjNum 950 snap 280 seq_num 950) dirty exists 2024-09-16T05:21:40.823 INFO:tasks.rados.rados.2.smithi204.stdout:2924: expect (ObjNum 969 snap 287 seq_num 969) 2024-09-16T05:21:40.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:40.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:40.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:40.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:40 smithi204 ceph-mon[30933]: osdmap e750: 12 total, 12 up, 11 in 2024-09-16T05:21:40.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:40 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:40.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:40 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:40.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/618881969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:21:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/618881969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:21:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:40 smithi204 ceph-mon[30933]: osdmap e751: 12 total, 12 up, 11 in 2024-09-16T05:21:40.949 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1] out_osds: [6] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:21:40.949 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:21:40.949 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T05:21:40.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:21:41.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:41.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:41.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:41.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:41.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:40 smithi032 ceph-mon[27388]: osdmap e750: 12 total, 12 up, 11 in 2024-09-16T05:21:41.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:40 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:41.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:40 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:41.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/618881969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:21:41.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/618881969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:21:41.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:40 smithi032 ceph-mon[27388]: osdmap e751: 12 total, 12 up, 11 in 2024-09-16T05:21:41.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:41.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:40 smithi196 ceph-mon[30802]: osdmap e750: 12 total, 12 up, 11 in 2024-09-16T05:21:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:40 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:40 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/618881969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:21:41.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/618881969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:21:41.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:40 smithi196 ceph-mon[30802]: osdmap e751: 12 total, 12 up, 11 in 2024-09-16T05:21:41.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:41.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:41.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:41.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:41.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:41.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:41.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:21:41.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:41.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:41.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:41.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:41.651 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:41.668 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:41.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:41.710 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:41.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:41.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:41.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:41.816 INFO:tasks.rados.rados.2.smithi204.stdout:2923: done (3 left) 2024-09-16T05:21:41.816 INFO:tasks.rados.rados.2.smithi204.stdout:2924: done (2 left) 2024-09-16T05:21:41.816 INFO:tasks.rados.rados.2.smithi204.stdout:2925: done (1 left) 2024-09-16T05:21:41.816 INFO:tasks.rados.rados.2.smithi204.stdout:2926: done (0 left) 2024-09-16T05:21:41.817 INFO:tasks.rados.rados.2.smithi204.stdout:2927: read oid 7 snap -1 2024-09-16T05:21:41.817 INFO:tasks.rados.rados.2.smithi204.stdout:2927: expect deleted 2024-09-16T05:21:41.817 INFO:tasks.rados.rados.2.smithi204.stdout:2928: snap_remove snap 309 2024-09-16T05:21:41.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:41.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:41.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:42.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:42.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:42.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:42.151 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:41 smithi204 ceph-mon[30933]: pgmap v1287: 127 pgs: 5 peering, 1 active+recovering, 121 active+clean; 256 MiB data, 900 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 373 KiB/s wr, 2 op/s; 4/244 objects misplaced (1.639%); 8.3 MiB/s, 37 objects/s recovering 2024-09-16T05:21:42.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:41 smithi204 ceph-mon[30933]: 2.5c scrub starts 2024-09-16T05:21:42.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:41 smithi204 ceph-mon[30933]: 2.5c scrub ok 2024-09-16T05:21:42.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:41 smithi204 ceph-mon[30933]: 2.5e scrub starts 2024-09-16T05:21:42.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:41 smithi204 ceph-mon[30933]: 2.5e scrub ok 2024-09-16T05:21:42.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:41 smithi204 ceph-mon[30933]: 2.5b scrub starts 2024-09-16T05:21:42.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:41 smithi204 ceph-mon[30933]: 2.5b scrub ok 2024-09-16T05:21:42.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:41 smithi204 ceph-mon[30933]: osdmap e752: 12 total, 12 up, 11 in 2024-09-16T05:21:42.168 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:42.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:42.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:42.169 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:42.169 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:42.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:41 smithi032 ceph-mon[27388]: pgmap v1287: 127 pgs: 5 peering, 1 active+recovering, 121 active+clean; 256 MiB data, 900 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 373 KiB/s wr, 2 op/s; 4/244 objects misplaced (1.639%); 8.3 MiB/s, 37 objects/s recovering 2024-09-16T05:21:42.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:41 smithi032 ceph-mon[27388]: 2.5c scrub starts 2024-09-16T05:21:42.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:41 smithi032 ceph-mon[27388]: 2.5c scrub ok 2024-09-16T05:21:42.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:41 smithi032 ceph-mon[27388]: 2.5e scrub starts 2024-09-16T05:21:42.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:41 smithi032 ceph-mon[27388]: 2.5e scrub ok 2024-09-16T05:21:42.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:41 smithi032 ceph-mon[27388]: 2.5b scrub starts 2024-09-16T05:21:42.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:41 smithi032 ceph-mon[27388]: 2.5b scrub ok 2024-09-16T05:21:42.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:41 smithi032 ceph-mon[27388]: osdmap e752: 12 total, 12 up, 11 in 2024-09-16T05:21:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:41 smithi196 ceph-mon[30802]: pgmap v1287: 127 pgs: 5 peering, 1 active+recovering, 121 active+clean; 256 MiB data, 900 MiB used, 983 GiB / 983 GiB avail; 255 B/s rd, 373 KiB/s wr, 2 op/s; 4/244 objects misplaced (1.639%); 8.3 MiB/s, 37 objects/s recovering 2024-09-16T05:21:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:41 smithi196 ceph-mon[30802]: 2.5c scrub starts 2024-09-16T05:21:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:41 smithi196 ceph-mon[30802]: 2.5c scrub ok 2024-09-16T05:21:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:41 smithi196 ceph-mon[30802]: 2.5e scrub starts 2024-09-16T05:21:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:41 smithi196 ceph-mon[30802]: 2.5e scrub ok 2024-09-16T05:21:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:41 smithi196 ceph-mon[30802]: 2.5b scrub starts 2024-09-16T05:21:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:41 smithi196 ceph-mon[30802]: 2.5b scrub ok 2024-09-16T05:21:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:41 smithi196 ceph-mon[30802]: osdmap e752: 12 total, 12 up, 11 in 2024-09-16T05:21:42.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:42.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:42.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:42.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:42.421 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:42.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:42.423 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:42.424 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:42.443 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:42.464 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:42.464 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:42.484 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:42.506 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:42.506 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:42.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:42.529 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:42.529 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:42.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:42.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:42.550 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:42.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:42.572 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:42.572 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:42.593 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:42.594 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:42.614 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:42.614 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:42.637 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:42.637 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:42.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:42.658 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:42.658 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:42.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:42.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:42.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:42.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:42.818 INFO:tasks.rados.rados.2.smithi204.stdout:2928: done (1 left) 2024-09-16T05:21:42.818 INFO:tasks.rados.rados.2.smithi204.stdout:2929: read oid 6 snap -1 2024-09-16T05:21:42.818 INFO:tasks.rados.rados.2.smithi204.stdout:2929: expect (ObjNum 906 snap 267 seq_num 906) 2024-09-16T05:21:42.818 INFO:tasks.rados.rados.2.smithi204.stdout:2930: copy_from oid 17 from oid 19 current snap is 315 2024-09-16T05:21:42.818 INFO:tasks.rados.rados.2.smithi204.stdout:2927: done (2 left) 2024-09-16T05:21:42.818 INFO:tasks.rados.rados.2.smithi204.stdout:2931: copy_from oid 43 from oid 37 current snap is 315 2024-09-16T05:21:42.818 INFO:tasks.rados.rados.2.smithi204.stdout:2932: snap_remove snap 311 2024-09-16T05:21:42.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:21:42.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:21:42.905 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 753, 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:21:42.811873+0000", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:14:38.892129+0000", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:21:34.827763+0000", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5804032, 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:21:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 37, 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "luminous", 2024-09-16T05:21:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:21:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 31, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 31, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 31, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 31, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 31, 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:21:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "687", 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "685", 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:21:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:21:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:21:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:21:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.3199999332427979, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.3199999332427979, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.92000001668930054, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.130000114440918, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.130000114440918, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:21:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:21:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 96, 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 96, 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 96, 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 96, 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 96, 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:21:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "753", 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "737", 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 317, 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 753, 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:21:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:21:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:21:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 1.6200000047683716, 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 1.6200000047683716, 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.92000001668930054, 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 1.4900000095367432, 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 1.4900000095367432, 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:21:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:21:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:21:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:21:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 742, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 21, 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 295, 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 329, 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 731, 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 296, 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:21:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:21:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:21:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:21:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3973436581", 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3973436581", 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3973436581", 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3973436581", 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:21:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:21:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:21:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:21:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:21:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:21:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:21:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 0, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 0, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 741, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:21:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:21:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:21:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:21:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:21:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:21:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:21:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:21:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 745, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:21:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:21:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:21:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:21:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:21:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:21:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:21:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:21:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:12.809159+0000", 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:36.931580+0000", 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:44.703412+0000", 2024-09-16T05:21:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 295 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:50.830187+0000", 2024-09-16T05:21:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:32.848448+0000", 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:21:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:32.858075+0000", 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 65536, 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:27.364615+0000", 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:17.698437+0000", 2024-09-16T05:21:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:07.217379+0000", 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:21:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:20:46.829682+0000", 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:40.328121+0000", 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:01.580040+0000", 2024-09-16T05:21:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [ 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: "pgid": "1.1", 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: 2, 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: 3 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:21:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:21:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 309, 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 311, 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:21:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 313, 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [ 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 311, 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:21:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:21:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:21:43.011 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:21:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:43.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:43.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:43.099 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:43.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:43.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:43.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:21:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:21:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:21:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:21:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.3a scrub starts 2024-09-16T05:21:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.3a scrub ok 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/615544536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.5d scrub starts 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.5d scrub ok 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:42 smithi032 ceph-mon[27388]: osdmap e753: 12 total, 12 up, 11 in 2024-09-16T05:21:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.3a scrub starts 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.3a scrub ok 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/615544536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.5d scrub starts 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.5d scrub ok 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:21:43.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:42 smithi204 ceph-mon[30933]: osdmap e753: 12 total, 12 up, 11 in 2024-09-16T05:21:43.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.3a scrub starts 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.3a scrub ok 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/615544536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.5d scrub starts 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.5d scrub ok 2024-09-16T05:21:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:21:43.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:21:43.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:42 smithi196 ceph-mon[30802]: osdmap e753: 12 total, 12 up, 11 in 2024-09-16T05:21:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:43.258 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:43.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:43.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:43.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:43.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:43.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:43.509 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:43.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:43.577 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:21:43.578 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:21:43.578 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:21:43.578 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:21:43.578 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:21:43.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:21:43.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:43.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:43.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:43.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:43.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:43.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:43.841 INFO:tasks.rados.rados.2.smithi204.stdout:2932: done (3 left) 2024-09-16T05:21:43.841 INFO:tasks.rados.rados.2.smithi204.stdout:2933: read oid 29 snap -1 2024-09-16T05:21:43.841 INFO:tasks.rados.rados.2.smithi204.stdout:2933: expect (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:21:43.841 INFO:tasks.rados.rados.2.smithi204.stdout:2934: snap_create 2024-09-16T05:21:43.841 INFO:tasks.rados.rados.2.smithi204.stdout:2929: expect (ObjNum 906 snap 267 seq_num 906) 2024-09-16T05:21:43.843 INFO:tasks.thrashosds.thrasher:j is {'epoch': 753, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:21:42.811873+0000', 'last_up_change': '2024-09-16T05:14:38.892129+0000', 'last_in_change': '2024-09-16T05:21:34.827763+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 37, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '687', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '685', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 96, 'pg_placement_num': 96, 'pg_placement_num_target': 96, 'pg_num_target': 96, 'pg_num_pending': 96, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '753', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '737', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 317, 'snap_epoch': 753, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6200000047683716, 'score_stable': 1.6200000047683716, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.4900000095367432, 'raw_score_stable': 1.4900000095367432, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 742, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 21, 'last_clean_end': 295, 'up_from': 329, 'up_thru': 731, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3973436581}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3973436581}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3973436581}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3973436581}]}, 'public_addr': '172.21.15.32:6819/3973436581', 'cluster_addr': '172.21.15.32:6821/3973436581', 'heartbeat_back_addr': '172.21.15.32:6825/3973436581', 'heartbeat_front_addr': '172.21.15.32:6823/3973436581', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 741, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:21:12.809159+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:21:36.931580+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:44.703412+0000', 'dead_epoch': 295}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:50.830187+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:32.848448+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:21:32.858075+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T05:21:27.364615+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:21:17.698437+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:21:07.217379+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:20:46.829682+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:21:40.328121+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:21:01.580040+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [2, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 309, 'length': 1}, {'begin': 311, 'length': 1}, {'begin': 313, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 311, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:21:43.843 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 1.1 2024-09-16T05:21:43.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd rm-pg-upmap 1.1 2024-09-16T05:21:43.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:43.945 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:43.966 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:43.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:44.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:44.090 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:44.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:44.101 INFO:tasks.rados.rados.2.smithi204.stdout:2930: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:21:44.102 INFO:tasks.rados.rados.2.smithi204.stdout:2931: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:21:44.102 INFO:tasks.rados.rados.2.smithi204.stdout:2931: finishing copy_from to smithi20457400-43 2024-09-16T05:21:44.102 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1037 (ObjNum 1027 snap 312 seq_num 1027) dirty exists 2024-09-16T05:21:44.102 INFO:tasks.rados.rados.2.smithi204.stdout:2930: finishing copy_from to smithi20457400-17 2024-09-16T05:21:44.102 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 853 (ObjNum 1020 snap 310 seq_num 1020) dirty exists 2024-09-16T05:21:44.102 INFO:tasks.rados.rados.2.smithi204.stdout:2933: expect (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:21:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: pgmap v1290: 127 pgs: 127 active+clean; 269 MiB data, 924 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-16T05:21:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.5a scrub starts 2024-09-16T05:21:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.5a scrub ok 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.5f scrub starts 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.5f scrub ok 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.56 scrub starts 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.56 scrub ok 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4044224613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:21:44.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:43 smithi032 ceph-mon[27388]: osdmap e754: 12 total, 12 up, 11 in 2024-09-16T05:21:44.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: pgmap v1290: 127 pgs: 127 active+clean; 269 MiB data, 924 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-16T05:21:44.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.5a scrub starts 2024-09-16T05:21:44.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.5a scrub ok 2024-09-16T05:21:44.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.5f scrub starts 2024-09-16T05:21:44.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.5f scrub ok 2024-09-16T05:21:44.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.56 scrub starts 2024-09-16T05:21:44.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.56 scrub ok 2024-09-16T05:21:44.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4044224613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:21:44.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:43 smithi204 ceph-mon[30933]: osdmap e754: 12 total, 12 up, 11 in 2024-09-16T05:21:44.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:44.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:21:44.213 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:44.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:44.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: pgmap v1290: 127 pgs: 127 active+clean; 269 MiB data, 924 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.5a scrub starts 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.5a scrub ok 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.5f scrub starts 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.5f scrub ok 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.56 scrub starts 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.56 scrub ok 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4044224613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:21:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:21:44.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:21:44.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:21:44.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:21:44.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:43 smithi196 ceph-mon[30802]: osdmap e754: 12 total, 12 up, 11 in 2024-09-16T05:21:44.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:44.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:44.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:44.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:44.453 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:44.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:44.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:44.467 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:44.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:44.589 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:44.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:44.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:44.715 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:44.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:44.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2929: done (4 left) 2024-09-16T05:21:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2930: done (3 left) 2024-09-16T05:21:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2931: done (2 left) 2024-09-16T05:21:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2933: done (1 left) 2024-09-16T05:21:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2934: done (0 left) 2024-09-16T05:21:44.821 INFO:tasks.rados.rados.2.smithi204.stdout:2935: write oid 32 current snap is 316 2024-09-16T05:21:44.822 INFO:tasks.rados.rados.2.smithi204.stdout:2935: seq_num 1030 ranges {511210=414125,1423159=486356,2668859=589585} 2024-09-16T05:21:44.826 INFO:tasks.rados.rados.2.smithi204.stdout:2935: writing smithi20457400-32 from 511210 to 925335 tid 1 2024-09-16T05:21:44.831 INFO:tasks.rados.rados.2.smithi204.stdout:2935: writing smithi20457400-32 from 1423159 to 1909515 tid 2 2024-09-16T05:21:44.839 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2935: writing smithi20457400-32 from 2668859 to 3258444 tid 3 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2936: read oid 29 snap -1 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2936: expect (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2937: copy_from oid 30 from oid 27 current snap is 316 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2935: finishing write tid 1 to smithi20457400-32 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2938: rollback oid 31 current snap is 316 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 31 to 298 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2939: write (excl) oid 43 current snap is 316 2024-09-16T05:21:44.840 INFO:tasks.rados.rados.2.smithi204.stdout:2939: seq_num 1031 ranges {528017=489374,1481641=630014,2696282=400857} 2024-09-16T05:21:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:44.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:44.845 INFO:tasks.rados.rados.2.smithi204.stdout:2939: writing smithi20457400-43 from 528017 to 1017391 tid 1 2024-09-16T05:21:44.852 INFO:tasks.rados.rados.2.smithi204.stdout:2939: writing smithi20457400-43 from 1481641 to 2111655 tid 2 2024-09-16T05:21:44.858 INFO:tasks.rados.rados.2.smithi204.stdout:2939: writing smithi20457400-43 from 2696282 to 3097139 tid 3 2024-09-16T05:21:44.858 INFO:tasks.rados.rados.2.smithi204.stdout:2940: delete oid 21 current snap is 316 2024-09-16T05:21:44.858 INFO:tasks.rados.rados.2.smithi204.stdout:2935: finishing write tid 2 to smithi20457400-32 2024-09-16T05:21:44.858 INFO:tasks.rados.rados.2.smithi204.stdout:2935: finishing write tid 3 to smithi20457400-32 2024-09-16T05:21:44.858 INFO:tasks.rados.rados.2.smithi204.stdout:2938: finishing rollback tid 1 to smithi20457400-31 2024-09-16T05:21:44.858 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:21:44.858 INFO:tasks.rados.rados.2.smithi204.stdout:2936: expect (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:21:44.936 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:21:44.936 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:21:44.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:21:44.936 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1329 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 563.15049124699999, 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80822626200000003, 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.937 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1329 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783037+0000", 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783039+0000", 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.674e-06 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783047+0000", 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6619999999999994e-06 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783063+0000", 2024-09-16T05:21:44.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6058000000000001e-05 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783071+0000", 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8399999999999995e-06 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783134+0000", 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2496999999999995e-05 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783142+0000", 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1219999999999995e-06 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591169+0000", 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80802694100000005 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591263+0000", 2024-09-16T05:21:44.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4468000000000004e-05 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1330 2.1d 2:b98a79db:::smithi20457400-31:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 563.15044028499995, 2024-09-16T05:21:44.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80866329400000003, 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1330 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:21:44.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783088+0000", 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783092+0000", 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5149999999999998e-06 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783104+0000", 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2412000000000001e-05 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783117+0000", 2024-09-16T05:21:44.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2379e-05 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783170+0000", 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3727999999999998e-05 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783178+0000", 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3479999999999998e-06 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591633+0000", 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.808455339 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591751+0000", 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000118087 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.947 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1331 2.1d 2:b98a79db:::smithi20457400-31:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e210)", 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 563.15039951899996, 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80878027500000005, 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1331 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783129+0000", 2024-09-16T05:21:44.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5400000000000001e-07 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783132+0000", 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689999999999999e-06 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783139+0000", 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7920000000000004e-06 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783144+0000", 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783192+0000", 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8304999999999999e-05 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:21.783196+0000", 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8480000000000003e-06 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591842+0000", 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.80864587700000001 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:22.591909+0000", 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7195999999999998e-05 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1575 2.5 2:a5bc1608:::smithi20457400-11:head [setxattr _header (57) in=7b,truncate 3966452] snapc 43=[43,42,41,40,3f,3e,3d,3c,3a,39,38,37,35,33,2d,27,21] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 546.24235349800006, 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64249167500000004, 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1575 2024-09-16T05:21:44.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691175+0000", 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5200000000000002e-07 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691178+0000", 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.734e-06 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691184+0000", 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2940000000000002e-06 2024-09-16T05:21:44.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691188+0000", 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.642e-06 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692349+0000", 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001161444 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:44.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692452+0000", 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102916 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692596+0000", 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000143552 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692970+0000", 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037462699999999999 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333552+0000", 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64058177999999999 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333609+0000", 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6728999999999998e-05 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333666+0000", 2024-09-16T05:21:44.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7504999999999999e-05 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1576 2.5 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e227)", 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 546.24233476300003, 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64272760100000004, 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:44.959 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1576 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691193+0000", 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691194+0000", 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.65e-07 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691195+0000", 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-06 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691199+0000", 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0060000000000003e-06 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.691202+0000", 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9239999999999999e-06 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692872+0000", 2024-09-16T05:21:44.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001669467 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692954+0000", 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2361000000000006e-05 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333715+0000", 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64076073300000003 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333817+0000", 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010184200000000001 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333921+0000", 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104249 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:3036 2.1b e379/291 2:da4f6049:::smithi20457400-49:head v 379'328, mlcod=379'325)", 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 376.16534975899998, 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.436541917, 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768178+0000", 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768179+0000", 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2900000000000003e-07 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768185+0000", 2024-09-16T05:21:44.967 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:44.968 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8309999999999997e-06 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203766+0000", 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43558113999999998 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203780+0000", 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3825999999999999e-05 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203879+0000", 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9419000000000005e-05 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.203926+0000", 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6248999999999997e-05 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204668+0000", 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00074248900000000004 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204720+0000", 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2234000000000002e-05 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.9.0:755 2.2f e422/275 ondisk, result = 0)", 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 333.21000729899998, 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.59448197899999999, 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:21:44.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723521+0000", 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7700000000000004e-07 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:11.723525+0000", 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5209999999999998e-06 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317758+0000", 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.59423284399999998 2024-09-16T05:21:44.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317765+0000", 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.827e-06 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317821+0000", 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6573000000000002e-05 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:44.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.317832+0000", 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0718e-05 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:12.318003+0000", 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017091900000000001 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3531 2.15 2:a94680a3:::smithi20457400-13:head [write 652237~793851] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:21:44.976 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.64307273399999, 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44774744599999999, 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3531 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290455+0000", 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.290457+0000", 2024-09-16T05:21:44.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.823e-06 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291024+0000", 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00056645899999999995 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291045+0000", 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0837999999999999e-05 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291055+0000", 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0066e-05 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291140+0000", 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5686000000000006e-05 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.291149+0000", 2024-09-16T05:21:44.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7579999999999998e-06 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.733424+0000", 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.442274801 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.733559+0000", 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013477500000000001 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.734084+0000", 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00052511500000000004 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738013+0000", 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039291179999999997 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738083+0000", 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0208000000000003e-05 2024-09-16T05:21:44.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738157+0000", 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4071000000000003e-05 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738203+0000", 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5728e-05 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3532 2.15 2:a94680a3:::smithi20457400-13:head [write 2198894~53343] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.63874172700002, 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44479106600000001, 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3532 2024-09-16T05:21:44.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294786+0000", 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294787+0000", 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6700000000000003e-07 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294826+0000", 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8801999999999999e-05 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294837+0000", 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1079e-05 2024-09-16T05:21:44.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294842+0000", 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.152e-06 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8766e-05 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294898+0000", 2024-09-16T05:21:44.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1470000000000004e-06 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737061+0000", 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44216253500000002 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737164+0000", 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103288 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737374+0000", 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020994000000000001 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739175+0000", 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018013759999999999 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739420+0000", 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024481500000000001 2024-09-16T05:21:44.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739486+0000", 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6308000000000003e-05 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739577+0000", 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1090999999999999e-05 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3533 2.15 2:a94680a3:::smithi20457400-13:head [setxattr _header (58) in=7b,truncate 2252237] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.63867409900001, 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44535093999999997, 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3533 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.991 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294854+0000", 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294855+0000", 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0699999999999997e-07 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294858+0000", 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2200000000000001e-06 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294865+0000", 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9290000000000001e-06 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294868+0000", 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4750000000000002e-06 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294924+0000", 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6005000000000003e-05 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294934+0000", 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0234999999999999e-05 2024-09-16T05:21:44.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738506+0000", 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44357146200000003 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738606+0000", 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100338 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 1", 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738777+0000", 2024-09-16T05:21:44.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017038200000000001 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739632+0000", 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00085582599999999996 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740092+0000", 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00046005699999999999 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740144+0000", 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.113e-05 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740205+0000", 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1373999999999999e-05 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.997 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3534 2.15 2:a94680a3:::smithi20457400-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.63863681300001, 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44553962000000003, 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3534 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294891+0000", 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294892+0000", 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0299999999999999e-07 2024-09-16T05:21:44.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294894+0000", 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.198e-06 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294901+0000", 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.917e-06 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294912+0000", 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1144999999999999e-05 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294950+0000", 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7889999999999998e-05 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.294955+0000", 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8949999999999997e-06 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739213+0000", 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.44425779500000001 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739349+0000", 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013630599999999999 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740238+0000", 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088882100000000003 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740329+0000", 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1366999999999994e-05 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740431+0000", 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101783 2024-09-16T05:21:45.003 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3539 2.54 2:2ace617f:::smithi20457400-21:head [write 685851~544349] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.62033488200001, 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42847687499999998, 2024-09-16T05:21:45.004 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3539 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313193+0000", 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313194+0000", 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313430+0000", 2024-09-16T05:21:45.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023602700000000001 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313441+0000", 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1158000000000001e-05 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313449+0000", 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3180000000000001e-06 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313509+0000", 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0010999999999999e-05 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.313516+0000", 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.131e-06 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.737710+0000", 2024-09-16T05:21:45.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42419478900000002 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738322+0000", 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061162399999999996 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.738531+0000", 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020935999999999999 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741253+0000", 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027214650000000002 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741592+0000", 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033932199999999997 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741636+0000", 2024-09-16T05:21:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3649000000000002e-05 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741670+0000", 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4264999999999998e-05 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3540 2.54 2:2ace617f:::smithi20457400-21:head [write 1799505~486346] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:21:45.011 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.61484220400001, 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42371521699999998, 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3540 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318686+0000", 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318687+0000", 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9400000000000004e-07 2024-09-16T05:21:45.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318917+0000", 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023015399999999999 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318932+0000", 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.485e-05 2024-09-16T05:21:45.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318939+0000", 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5059999999999998e-06 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7806999999999997e-05 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319006+0000", 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1649999999999995e-06 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739899+0000", 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42089313 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.739946+0000", 2024-09-16T05:21:45.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6783000000000003e-05 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740114+0000", 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016756599999999999 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741289+0000", 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0011754549999999999 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742318+0000", 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010284420000000001 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742363+0000", 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5760999999999998e-05 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742401+0000", 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7803999999999999e-05 2024-09-16T05:21:45.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3541 2.54 2:2ace617f:::smithi20457400-21:head [setxattr _header (58) in=7b,truncate 2285851] snapc 99=[99,96] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.61457230100001, 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42367250699999998, 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:45.019 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3541 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318956+0000", 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5600000000000001e-07 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318959+0000", 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7020000000000001e-06 2024-09-16T05:21:45.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318970+0000", 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0991e-05 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318974+0000", 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2849999999999996e-06 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319029+0000", 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5130999999999998e-05 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319036+0000", 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9129999999999997e-06 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740691+0000", 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42165504500000001 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740769+0000", 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7179000000000001e-05 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.740875+0000", 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106637 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741296+0000", 2024-09-16T05:21:45.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042051399999999998 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742521+0000", 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001225636 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742569+0000", 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7982000000000002e-05 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742628+0000", 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9036000000000002e-05 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3542 2.54 2:2ace617f:::smithi20457400-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 319.614531494, 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42382194699999998, 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.026 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3542 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318997+0000", 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7599999999999998e-07 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.318999+0000", 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7770000000000001e-06 2024-09-16T05:21:45.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319007+0000", 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4039999999999995e-06 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319011+0000", 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8260000000000003e-06 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319089+0000", 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8046000000000005e-05 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.319099+0000", 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6490000000000003e-06 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741192+0000", 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.42209296299999999 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.741242+0000", 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0589e-05 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742668+0000", 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001425429 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742746+0000", 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7990000000000004e-05 2024-09-16T05:21:45.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:25.742819+0000", 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2997999999999995e-05 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3951 2.2f 2:f4b938a4:::smithi20457400-47:head [setxattr _header (58) in=7b,truncate 3395113] snapc a8=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 288.07533820899999, 2024-09-16T05:21:45.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39145527099999999, 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3951 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:21:45.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858190+0000", 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7099999999999997e-07 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858193+0000", 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3439999999999999e-06 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858197+0000", 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229999999999999e-06 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858200+0000", 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.937e-06 2024-09-16T05:21:45.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859064+0000", 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00086360799999999998 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859153+0000", 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9592999999999994e-05 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:21:45.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859294+0000", 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014096099999999999 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859660+0000", 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036632599999999999 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249556+0000", 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38989596300000001 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249600+0000", 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3757000000000001e-05 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249645+0000", 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5087999999999999e-05 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3952 2.2f 2:f4b938a4:::smithi20457400-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e467)", 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 288.07530639700002, 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39157208999999998, 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3952 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858222+0000", 2024-09-16T05:21:45.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.77e-07 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858226+0000", 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7249999999999999e-06 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858231+0000", 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.797e-06 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.858234+0000", 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2320000000000001e-06 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859544+0000", 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001309892 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859613+0000", 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9261e-05 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249668+0000", 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39005463499999998 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249718+0000", 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0416e-05 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249794+0000", 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5754999999999996e-05 2024-09-16T05:21:45.045 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.046 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.047 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.047 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5476 2.5 2:a1d8c6d9:::smithi20457400-3:head [read 0~2890969,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e575)", 2024-09-16T05:21:45.047 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:44.784344+0000", 2024-09-16T05:21:45.047 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 180.149184443, 2024-09-16T05:21:45.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.073875179999999999, 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5476 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784344+0000", 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784344+0000", 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784344+0000", 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4299999999999998e-07 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784347+0000", 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5220000000000002e-06 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784353+0000", 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2960000000000004e-06 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.784358+0000", 2024-09-16T05:21:45.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0390000000000004e-06 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.787090+0000", 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027322980000000002 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.787098+0000", 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9670000000000001e-06 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.857056+0000", 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.069958168000000001 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.857132+0000", 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5093999999999996e-05 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.858219+0000", 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010873530000000001 2024-09-16T05:21:45.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:1100 2.f e588/435 2:f08ec412:::smithi20457400-44:e0 v 588'362, mlcod=585'359)", 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 167.16006012400001, 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.771307407, 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773469+0000", 2024-09-16T05:21:45.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0100000000000004e-07 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773478+0000", 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1139999999999995e-06 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544068+0000", 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.770589827 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544083+0000", 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5213e-05 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544231+0000", 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014854000000000001 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544252+0000", 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0568e-05 2024-09-16T05:21:45.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544741+0000", 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048860799999999997 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544775+0000", 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4836000000000003e-05 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:5976 2.8 e617/320 2:1030fb5b:::smithi20457400-27:head v 617'305, mlcod=602'299)", 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:26.797538+0000", 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 138.135989822, 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.75484651300000005, 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:26.797538+0000", 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:26.797538+0000", 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:26.797539+0000", 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0580000000000001e-06 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:45.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:26.797545+0000", 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9309999999999996e-06 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.551777+0000", 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.75423199100000005 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.551785+0000", 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2200000000000003e-06 2024-09-16T05:21:45.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.551874+0000", 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8968000000000006e-05 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.551893+0000", 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.95e-05 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.552352+0000", 2024-09-16T05:21:45.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045916300000000001 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.552385+0000", 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2682000000000003e-05 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:45.062 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:21:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:45.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:21:45.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:44 smithi032 ceph-mon[27388]: osdmap e755: 12 total, 12 up, 11 in 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:2937: finishing copy_from racing read to smithi20457400-30 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:2939: finishing write tid 1 to smithi20457400-43 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:2937: finishing copy_from to smithi20457400-30 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 326 (ObjNum 847 snap 246 seq_num 847) dirty exists 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:2939: finishing write tid 2 to smithi20457400-43 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:2936: done (5 left) 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:2937: done (4 left) 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:2938: done (3 left) 2024-09-16T05:21:45.077 INFO:tasks.rados.rados.2.smithi204.stdout:2940: done (2 left) 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:2941: snap_create 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:2939: finishing write tid 3 to smithi20457400-43 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:2939: finishing write tid 4 to smithi20457400-43 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:2939: finishing write tid 5 to smithi20457400-43 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1041 (ObjNum 1031 snap 316 seq_num 1031) dirty exists 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:2939: left oid 43 (ObjNum 1031 snap 316 seq_num 1031) 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:2935: finishing write tid 4 to smithi20457400-32 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:2935: finishing write tid 5 to smithi20457400-32 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1141 (ObjNum 1030 snap 316 seq_num 1030) dirty exists 2024-09-16T05:21:45.078 INFO:tasks.rados.rados.2.smithi204.stdout:2935: left oid 32 (ObjNum 1030 snap 316 seq_num 1030) 2024-09-16T05:21:45.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:21:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:21:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:21:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:21:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:21:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:21:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:21:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:21:45.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:44 smithi204 ceph-mon[30933]: osdmap e755: 12 total, 12 up, 11 in 2024-09-16T05:21:45.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:45.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:45.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:21:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:21:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:21:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:21:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:21:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:21:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:21:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:21:45.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:44 smithi196 ceph-mon[30802]: osdmap e755: 12 total, 12 up, 11 in 2024-09-16T05:21:45.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:45.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:45.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:45.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:45.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:45.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:45.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:45.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:45.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:45.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:45.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:45.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:45.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:45.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:45.812 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:21:45.812 INFO:teuthology.orchestra.run.smithi032.stderr:clear 1.1 pg_upmap mapping 2024-09-16T05:21:45.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:21:45.819 INFO:tasks.rados.rados.2.smithi204.stdout:2935: done (2 left) 2024-09-16T05:21:45.819 INFO:tasks.rados.rados.2.smithi204.stdout:2939: done (1 left) 2024-09-16T05:21:45.819 INFO:tasks.rados.rados.2.smithi204.stdout:2941: done (0 left) 2024-09-16T05:21:45.819 INFO:tasks.rados.rados.2.smithi204.stdout:2942: read oid 27 snap -1 2024-09-16T05:21:45.819 INFO:tasks.rados.rados.2.smithi204.stdout:2942: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:21:45.819 INFO:tasks.rados.rados.2.smithi204.stdout:2943: read oid 30 snap -1 2024-09-16T05:21:45.819 INFO:tasks.rados.rados.2.smithi204.stdout:2943: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:21:45.819 INFO:tasks.rados.rados.2.smithi204.stdout:2944: rollback oid 19 current snap is 317 2024-09-16T05:21:45.820 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 19 to 313 2024-09-16T05:21:45.820 INFO:tasks.rados.rados.2.smithi204.stdout:2945: snap_remove snap 315 2024-09-16T05:21:45.911 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:45.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:45.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:46.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:46.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:46.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:46.065 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:46.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: pgmap v1293: 127 pgs: 127 active+clean; 268 MiB data, 928 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:46.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:21:46.133 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:21:46.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/374769558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:46.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3170076046' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.1"}]: dispatch 2024-09-16T05:21:46.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:46.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:46.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:21:46.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/374769558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:46.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3170076046' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.1"}]': finished 2024-09-16T05:21:46.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:45 smithi032 ceph-mon[27388]: osdmap e756: 12 total, 12 up, 11 in 2024-09-16T05:21:46.165 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: pgmap v1293: 127 pgs: 127 active+clean; 268 MiB data, 928 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:21:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:21:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/374769558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3170076046' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.1"}]: dispatch 2024-09-16T05:21:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:21:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/374769558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3170076046' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.1"}]': finished 2024-09-16T05:21:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:45 smithi204 ceph-mon[30933]: osdmap e756: 12 total, 12 up, 11 in 2024-09-16T05:21:46.187 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:46.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: pgmap v1293: 127 pgs: 127 active+clean; 268 MiB data, 928 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/374769558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3170076046' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.1"}]: dispatch 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:21:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/374769558' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3170076046' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.1"}]': finished 2024-09-16T05:21:46.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:45 smithi196 ceph-mon[30802]: osdmap e756: 12 total, 12 up, 11 in 2024-09-16T05:21:46.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:46.313 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:46.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:46.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:46.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:21:46.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:46.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:46.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:46.561 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:21:46.561 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:21:46.561 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:21:46.562 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:21:46.565 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:46.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:46.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:46.689 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:46.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:46.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:46.817 INFO:tasks.rados.rados.2.smithi204.stdout:2945: done (3 left) 2024-09-16T05:21:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:2946: read oid 25 snap -1 2024-09-16T05:21:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:2946: expect deleted 2024-09-16T05:21:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:2947: write (excl) oid 3 current snap is 317 2024-09-16T05:21:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:2944: finishing rollback tid 0 to smithi20457400-19 2024-09-16T05:21:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:2944: finishing rollback tid 1 to smithi20457400-19 2024-09-16T05:21:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:2944: finishing rollback tid 2 to smithi20457400-19 2024-09-16T05:21:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 856 (ObjNum 1020 snap 310 seq_num 1020) dirty exists 2024-09-16T05:21:46.818 INFO:tasks.rados.rados.2.smithi204.stdout:2942: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:21:46.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:46.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:46.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:46.943 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:46.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:46.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:46.946 INFO:tasks.rados.rados.2.smithi204.stdout:2943: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:21:47.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:47.071 INFO:tasks.rados.rados.2.smithi204.stdout:2947: seq_num 1033 ranges {561631=639872,1882252=633683,3034775=326856} 2024-09-16T05:21:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:47.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:47.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:47.077 INFO:tasks.rados.rados.2.smithi204.stdout:2947: writing smithi20457400-3 from 561631 to 1201503 tid 1 2024-09-16T05:21:47.084 INFO:tasks.rados.rados.2.smithi204.stdout:2947: writing smithi20457400-3 from 1882252 to 2515935 tid 2 2024-09-16T05:21:47.089 INFO:tasks.rados.rados.2.smithi204.stdout:2947: writing smithi20457400-3 from 3034775 to 3361631 tid 3 2024-09-16T05:21:47.090 INFO:tasks.rados.rados.2.smithi204.stdout:2942: done (4 left) 2024-09-16T05:21:47.090 INFO:tasks.rados.rados.2.smithi204.stdout:2943: done (3 left) 2024-09-16T05:21:47.090 INFO:tasks.rados.rados.2.smithi204.stdout:2944: done (2 left) 2024-09-16T05:21:47.090 INFO:tasks.rados.rados.2.smithi204.stdout:2946: done (1 left) 2024-09-16T05:21:47.090 INFO:tasks.rados.rados.2.smithi204.stdout:2948: snap_remove snap 304 2024-09-16T05:21:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:21:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:21:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:21:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:21:47.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:46 smithi032 ceph-mon[27388]: osdmap e757: 12 total, 12 up, 11 in 2024-09-16T05:21:47.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:21:47.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:46 smithi204 ceph-mon[30933]: osdmap e757: 12 total, 12 up, 11 in 2024-09-16T05:21:47.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:47.197 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:47.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:47.202 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:21:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:21:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:21:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:21:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:21:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:21:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:21:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:21:47.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:21:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:21:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:21:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:21:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:21:47.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:46 smithi196 ceph-mon[30802]: osdmap e757: 12 total, 12 up, 11 in 2024-09-16T05:21:47.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:47.319 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:47.320 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:47.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:47.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:47.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:47.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:47.448 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:47.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:47.569 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:47.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:47.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:47.692 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:47.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:47.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:47.818 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:47.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:47.819 INFO:tasks.rados.rados.2.smithi204.stdout:2948: done (1 left) 2024-09-16T05:21:47.819 INFO:tasks.rados.rados.2.smithi204.stdout:2949: write (excl) oid 5 current snap is 317 2024-09-16T05:21:47.819 INFO:tasks.rados.rados.2.smithi204.stdout:2949: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2024-09-16T05:21:47.825 INFO:tasks.rados.rados.2.smithi204.stdout:2949: writing smithi20457400-5 from 578438 to 1009912 tid 1 2024-09-16T05:21:47.832 INFO:tasks.rados.rados.2.smithi204.stdout:2949: writing smithi20457400-5 from 1657087 to 2150781 tid 2 2024-09-16T05:21:47.839 INFO:tasks.rados.rados.2.smithi204.stdout:2949: writing smithi20457400-5 from 2778551 to 3378438 tid 3 2024-09-16T05:21:47.839 INFO:tasks.rados.rados.2.smithi204.stdout:2950: write oid 35 current snap is 317 2024-09-16T05:21:47.839 INFO:tasks.rados.rados.2.smithi204.stdout:2947: finishing write tid 1 to smithi20457400-3 2024-09-16T05:21:47.839 INFO:tasks.rados.rados.2.smithi204.stdout:2950: seq_num 1035 ranges {595245=506723,1715569=637352,2805974=562298} 2024-09-16T05:21:47.845 INFO:tasks.rados.rados.2.smithi204.stdout:2950: writing smithi20457400-35 from 595245 to 1101968 tid 1 2024-09-16T05:21:47.851 INFO:tasks.rados.rados.2.smithi204.stdout:2950: writing smithi20457400-35 from 1715569 to 2352921 tid 2 2024-09-16T05:21:47.857 INFO:tasks.rados.rados.2.smithi204.stdout:2950: writing smithi20457400-35 from 2805974 to 3368272 tid 3 2024-09-16T05:21:47.857 INFO:tasks.rados.rados.2.smithi204.stdout:2951: snap_remove snap 313 2024-09-16T05:21:47.891 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:47.891 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:47.891 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:47.891 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:47.905 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:47.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:47.926 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:47.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:47.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:47.948 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:47.948 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:47.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:47.971 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:47.992 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:47.992 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:48.013 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:48.014 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:48.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:47 smithi032 ceph-mon[27388]: pgmap v1296: 127 pgs: 1 unknown, 126 active+clean; 277 MiB data, 932 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:48.014 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:47 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:21:48.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:47 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:21:48.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:47 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:21:48.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:47 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:21:48.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:47 smithi032 ceph-mon[27388]: osdmap e758: 12 total, 12 up, 11 in 2024-09-16T05:21:48.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2299478991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:48.035 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:48.035 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:48.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:48.054 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:21:48.054 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:21:48.054 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:21:48.054 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:21:48.054 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:21:48.056 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:48.057 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:48.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:48.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:48.078 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:48.078 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:48.099 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:48.099 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:48.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:47 smithi204 ceph-mon[30933]: pgmap v1296: 127 pgs: 1 unknown, 126 active+clean; 277 MiB data, 932 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:47 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:21:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:47 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:21:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:47 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:21:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:47 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:21:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:47 smithi204 ceph-mon[30933]: osdmap e758: 12 total, 12 up, 11 in 2024-09-16T05:21:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2299478991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:48.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:48.196 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:48.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:47 smithi196 ceph-mon[30802]: pgmap v1296: 127 pgs: 1 unknown, 126 active+clean; 277 MiB data, 932 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:47 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:21:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:47 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:21:48.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:47 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:21:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:47 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:21:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:47 smithi196 ceph-mon[30802]: osdmap e758: 12 total, 12 up, 11 in 2024-09-16T05:21:48.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2299478991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:48.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:48.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:48.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:48.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:48.445 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:48.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:48.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:48.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:21:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:48.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:48.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:48.694 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:48.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:48.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:21:48.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:48.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:48.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2951: done (3 left) 2024-09-16T05:21:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2952: write oid 41 current snap is 317 2024-09-16T05:21:48.828 INFO:tasks.rados.rados.2.smithi204.stdout:2952: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-16T05:21:48.834 INFO:tasks.rados.rados.2.smithi204.stdout:2952: writing smithi20457400-41 from 612052 to 1194024 tid 1 2024-09-16T05:21:48.839 INFO:tasks.rados.rados.2.smithi204.stdout:2952: writing smithi20457400-41 from 1657698 to 2155061 tid 2 2024-09-16T05:21:48.846 INFO:tasks.rados.rados.2.smithi204.stdout:2952: writing smithi20457400-41 from 2717044 to 3412052 tid 3 2024-09-16T05:21:48.847 INFO:tasks.rados.rados.2.smithi204.stdout:2953: snap_remove snap 291 2024-09-16T05:21:48.849 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:48.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:48.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:48.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:48.980 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:49.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:49.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:49.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:49.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:48 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:21:49.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:48 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:21:49.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:48 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:21:49.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:48 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:21:49.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:48 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:21:49.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:48 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:21:49.101 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:48 smithi204 ceph-mon[30933]: osdmap e759: 12 total, 12 up, 11 in 2024-09-16T05:21:49.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:48 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:21:49.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:48 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:21:49.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:48 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:21:49.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:48 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:21:49.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:48 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:21:49.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:48 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:21:49.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:48 smithi032 ceph-mon[27388]: osdmap e759: 12 total, 12 up, 11 in 2024-09-16T05:21:49.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:49.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:48 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:21:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:48 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:21:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:48 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:21:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:48 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:21:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:48 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:21:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:48 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:21:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:48 smithi196 ceph-mon[30802]: osdmap e759: 12 total, 12 up, 11 in 2024-09-16T05:21:49.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:49.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:49.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:49.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:49.448 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:49.448 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:49.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:49.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:49.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:49.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:49.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:49.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:49.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:49.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:49.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:49.825 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:21:49.832 INFO:tasks.rados.rados.2.smithi204.stdout:2953: done (4 left) 2024-09-16T05:21:49.833 INFO:tasks.rados.rados.2.smithi204.stdout:2954: write (excl) oid 8 current snap is 317 2024-09-16T05:21:49.833 INFO:tasks.rados.rados.2.smithi204.stdout:2954: seq_num 1037 ranges {628859=657221,1716180=641021,3144467=284392} 2024-09-16T05:21:49.840 INFO:tasks.rados.rados.2.smithi204.stdout:2954: writing smithi20457400-8 from 628859 to 1286080 tid 1 2024-09-16T05:21:49.841 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:49.842 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:49.846 INFO:tasks.rados.rados.2.smithi204.stdout:2954: writing smithi20457400-8 from 1716180 to 2357201 tid 2 2024-09-16T05:21:49.853 INFO:tasks.rados.rados.2.smithi204.stdout:2954: writing smithi20457400-8 from 3144467 to 3428859 tid 3 2024-09-16T05:21:49.854 INFO:tasks.rados.rados.2.smithi204.stdout:2955: read oid 30 snap -1 2024-09-16T05:21:49.854 INFO:tasks.rados.rados.2.smithi204.stdout:2955: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:21:49.854 INFO:tasks.rados.rados.2.smithi204.stdout:2956: read oid 23 snap -1 2024-09-16T05:21:49.854 INFO:tasks.rados.rados.2.smithi204.stdout:2956: expect (ObjNum 1006 snap 303 seq_num 1006) 2024-09-16T05:21:49.854 INFO:tasks.rados.rados.2.smithi204.stdout:2957: write oid 45 current snap is 317 2024-09-16T05:21:49.855 INFO:tasks.rados.rados.2.smithi204.stdout:2947: finishing write tid 2 to smithi20457400-3 2024-09-16T05:21:49.855 INFO:tasks.rados.rados.2.smithi204.stdout:2947: finishing write tid 3 to smithi20457400-3 2024-09-16T05:21:49.855 INFO:tasks.rados.rados.2.smithi204.stdout:2957: seq_num 1038 ranges {645666=732470,2174662=501032,3288243=157423} 2024-09-16T05:21:49.861 INFO:tasks.rados.rados.2.smithi204.stdout:2957: writing smithi20457400-45 from 645666 to 1378136 tid 1 2024-09-16T05:21:49.868 INFO:tasks.rados.rados.2.smithi204.stdout:2957: writing smithi20457400-45 from 2174662 to 2675694 tid 2 2024-09-16T05:21:49.873 INFO:tasks.rados.rados.2.smithi204.stdout:2957: writing smithi20457400-45 from 3288243 to 3445666 tid 3 2024-09-16T05:21:49.873 INFO:tasks.rados.rados.2.smithi204.stdout:2958: delete oid 13 current snap is 317 2024-09-16T05:21:49.873 INFO:tasks.rados.rados.2.smithi204.stdout:2947: finishing write tid 4 to smithi20457400-3 2024-09-16T05:21:49.873 INFO:tasks.rados.rados.2.smithi204.stdout:2947: finishing write tid 5 to smithi20457400-3 2024-09-16T05:21:49.873 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1089 (ObjNum 1033 snap 317 seq_num 1033) dirty exists 2024-09-16T05:21:49.873 INFO:tasks.rados.rados.2.smithi204.stdout:2947: left oid 3 (ObjNum 1033 snap 317 seq_num 1033) 2024-09-16T05:21:49.873 INFO:tasks.rados.rados.2.smithi204.stdout:2949: finishing write tid 1 to smithi20457400-5 2024-09-16T05:21:49.873 INFO:tasks.rados.rados.2.smithi204.stdout:2949: finishing write tid 2 to smithi20457400-5 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2949: finishing write tid 3 to smithi20457400-5 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2949: finishing write tid 4 to smithi20457400-5 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2949: finishing write tid 5 to smithi20457400-5 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 621 (ObjNum 1034 snap 317 seq_num 1034) dirty exists 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2949: left oid 5 (ObjNum 1034 snap 317 seq_num 1034) 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2950: finishing write tid 1 to smithi20457400-35 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2950: finishing write tid 2 to smithi20457400-35 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2950: finishing write tid 3 to smithi20457400-35 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2950: finishing write tid 4 to smithi20457400-35 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2950: finishing write tid 5 to smithi20457400-35 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 858 (ObjNum 1035 snap 317 seq_num 1035) dirty exists 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2950: left oid 35 (ObjNum 1035 snap 317 seq_num 1035) 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2952: finishing write tid 1 to smithi20457400-41 2024-09-16T05:21:49.874 INFO:tasks.rados.rados.2.smithi204.stdout:2952: finishing write tid 2 to smithi20457400-41 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2952: finishing write tid 3 to smithi20457400-41 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2952: finishing write tid 4 to smithi20457400-41 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2952: finishing write tid 5 to smithi20457400-41 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1267 (ObjNum 1036 snap 317 seq_num 1036) dirty exists 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2952: left oid 41 (ObjNum 1036 snap 317 seq_num 1036) 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2954: finishing write tid 1 to smithi20457400-8 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2954: finishing write tid 2 to smithi20457400-8 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2954: finishing write tid 3 to smithi20457400-8 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2954: finishing write tid 4 to smithi20457400-8 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2954: finishing write tid 5 to smithi20457400-8 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1271 (ObjNum 1037 snap 317 seq_num 1037) dirty exists 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2954: left oid 8 (ObjNum 1037 snap 317 seq_num 1037) 2024-09-16T05:21:49.875 INFO:tasks.rados.rados.2.smithi204.stdout:2956: expect (ObjNum 1006 snap 303 seq_num 1006) 2024-09-16T05:21:49.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:49.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:49.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:49.970 INFO:tasks.rados.rados.2.smithi204.stdout:2955: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:21:50.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:50.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: pgmap v1299: 127 pgs: 1 active+remapped, 126 active+clean; 286 MiB data, 941 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: 2.30 deep-scrub starts 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: 2.30 deep-scrub ok 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: 2.45 scrub starts 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: 2.45 scrub ok 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2269873361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2269873361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:50.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: osdmap e760: 12 total, 12 up, 11 in 2024-09-16T05:21:50.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:49 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:21:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:50.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:2957: finishing write tid 1 to smithi20457400-45 2024-09-16T05:21:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:2957: finishing write tid 2 to smithi20457400-45 2024-09-16T05:21:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:2957: finishing write tid 3 to smithi20457400-45 2024-09-16T05:21:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:2957: finishing write tid 4 to smithi20457400-45 2024-09-16T05:21:50.098 INFO:tasks.rados.rados.2.smithi204.stdout:2947: done (8 left) 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2949: done (7 left) 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2950: done (6 left) 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2952: done (5 left) 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2954: done (4 left) 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2955: done (3 left) 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2956: done (2 left) 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2958: done (1 left) 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2959: copy_from oid 34 from oid 14 current snap is 317 2024-09-16T05:21:50.099 INFO:tasks.rados.rados.2.smithi204.stdout:2957: finishing write tid 5 to smithi20457400-45 2024-09-16T05:21:50.100 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1256 (ObjNum 1038 snap 317 seq_num 1038) dirty exists 2024-09-16T05:21:50.100 INFO:tasks.rados.rados.2.smithi204.stdout:2957: left oid 45 (ObjNum 1038 snap 317 seq_num 1038) 2024-09-16T05:21:50.100 INFO:tasks.rados.rados.2.smithi204.stdout:2957: done (1 left) 2024-09-16T05:21:50.101 INFO:tasks.rados.rados.2.smithi204.stdout:2960: read oid 9 snap -1 2024-09-16T05:21:50.101 INFO:tasks.rados.rados.2.smithi204.stdout:2960: expect (ObjNum 985 snap 289 seq_num 985) 2024-09-16T05:21:50.101 INFO:tasks.rados.rados.2.smithi204.stdout:2961: delete oid 22 current snap is 317 2024-09-16T05:21:50.102 INFO:tasks.rados.rados.2.smithi204.stdout:2961: done (2 left) 2024-09-16T05:21:50.102 INFO:tasks.rados.rados.2.smithi204.stdout:2962: delete oid 1 current snap is 317 2024-09-16T05:21:50.102 INFO:tasks.rados.rados.2.smithi204.stdout:2962: done (2 left) 2024-09-16T05:21:50.102 INFO:tasks.rados.rados.2.smithi204.stdout:2963: snap_remove snap 301 2024-09-16T05:21:50.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: pgmap v1299: 127 pgs: 1 active+remapped, 126 active+clean; 286 MiB data, 941 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: 2.30 deep-scrub starts 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: 2.30 deep-scrub ok 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: 2.45 scrub starts 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: 2.45 scrub ok 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2269873361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:21:50.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:21:50.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:50.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2269873361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:50.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: osdmap e760: 12 total, 12 up, 11 in 2024-09-16T05:21:50.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:49 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:21:50.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:50.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:50.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: pgmap v1299: 127 pgs: 1 active+remapped, 126 active+clean; 286 MiB data, 941 MiB used, 983 GiB / 983 GiB avail 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: 2.30 deep-scrub starts 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: 2.30 deep-scrub ok 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: 2.45 scrub starts 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: 2.45 scrub ok 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2269873361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2269873361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: osdmap e760: 12 total, 12 up, 11 in 2024-09-16T05:21:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:49 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:21:50.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:50.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:50.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:50.392 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:50.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:50.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:50.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:50.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:50.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:50.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:50.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:50.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:50.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:50.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:21:50.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:2963: done (2 left) 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:2964: rollback oid 47 current snap is 317 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 47 to 279 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:2965: rollback oid 37 current snap is 317 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 37 to 305 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:2966: read oid 32 snap 298 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:2966: expect (ObjNum 815 snap 237 seq_num 815) 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:2967: read oid 24 snap -1 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:2967: expect (ObjNum 998 snap 296 seq_num 998) 2024-09-16T05:21:50.826 INFO:tasks.rados.rados.2.smithi204.stdout:2968: read oid 50 snap -1 2024-09-16T05:21:50.827 INFO:tasks.rados.rados.2.smithi204.stdout:2968: expect (ObjNum 1016 snap 309 seq_num 1016) 2024-09-16T05:21:50.827 INFO:tasks.rados.rados.2.smithi204.stdout:2969: read oid 5 snap 305 2024-09-16T05:21:50.827 INFO:tasks.rados.rados.2.smithi204.stdout:2969: expect (ObjNum 1009 snap 304 seq_num 1009) 2024-09-16T05:21:50.827 INFO:tasks.rados.rados.2.smithi204.stdout:2970: write (excl) oid 38 current snap is 317 2024-09-16T05:21:50.827 INFO:tasks.rados.rados.2.smithi204.stdout:2970: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-09-16T05:21:50.831 INFO:tasks.rados.rados.2.smithi204.stdout:2970: writing smithi20457400-38 from 712894 to 1462713 tid 1 2024-09-16T05:21:50.840 INFO:tasks.rados.rados.2.smithi204.stdout:2970: writing smithi20457400-38 from 2008590 to 2800607 tid 2 2024-09-16T05:21:50.845 INFO:tasks.rados.rados.2.smithi204.stdout:2970: writing smithi20457400-38 from 3281582 to 3512894 tid 3 2024-09-16T05:21:50.845 INFO:tasks.rados.rados.2.smithi204.stdout:2971: read oid 40 snap -1 2024-09-16T05:21:50.845 INFO:tasks.rados.rados.2.smithi204.stdout:2971: expect (ObjNum 1011 snap 306 seq_num 1011) 2024-09-16T05:21:50.846 INFO:tasks.rados.rados.2.smithi204.stdout:2972: copy_from oid 6 from oid 22 current snap is 317 2024-09-16T05:21:50.846 INFO:tasks.rados.rados.2.smithi204.stdout:2959: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:21:50.846 INFO:tasks.rados.rados.2.smithi204.stdout:2959: finishing copy_from to smithi20457400-34 2024-09-16T05:21:50.846 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1233 (ObjNum 976 snap 287 seq_num 976) dirty exists 2024-09-16T05:21:50.846 INFO:tasks.rados.rados.2.smithi204.stdout:2960: expect (ObjNum 985 snap 289 seq_num 985) 2024-09-16T05:21:50.863 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:50.863 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:50.873 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:50.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:50.968 INFO:tasks.rados.rados.2.smithi204.stdout:2964: finishing rollback tid 1 to smithi20457400-47 2024-09-16T05:21:50.968 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:21:50.968 INFO:tasks.rados.rados.2.smithi204.stdout:2965: finishing rollback tid 0 to smithi20457400-37 2024-09-16T05:21:50.968 INFO:tasks.rados.rados.2.smithi204.stdout:2969: expect (ObjNum 1009 snap 304 seq_num 1009) 2024-09-16T05:21:50.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:50.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:50.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:51.078 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:51.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:50 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:21:51.096 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:50 smithi204 ceph-mon[30933]: osdmap e761: 12 total, 12 up, 11 in 2024-09-16T05:21:51.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:51.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:51.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:51.148 INFO:tasks.rados.rados.2.smithi204.stdout:2965: finishing rollback tid 1 to smithi20457400-37 2024-09-16T05:21:51.148 INFO:tasks.rados.rados.2.smithi204.stdout:2965: finishing rollback tid 2 to smithi20457400-37 2024-09-16T05:21:51.148 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1040 (ObjNum 730 snap 210 seq_num 730) dirty exists 2024-09-16T05:21:51.148 INFO:tasks.rados.rados.2.smithi204.stdout:2970: finishing write tid 1 to smithi20457400-38 2024-09-16T05:21:51.148 INFO:tasks.rados.rados.2.smithi204.stdout:2970: finishing write tid 2 to smithi20457400-38 2024-09-16T05:21:51.148 INFO:tasks.rados.rados.2.smithi204.stdout:2970: finishing write tid 3 to smithi20457400-38 2024-09-16T05:21:51.149 INFO:tasks.rados.rados.2.smithi204.stdout:2971: expect (ObjNum 1011 snap 306 seq_num 1011) 2024-09-16T05:21:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:50 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:21:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:50 smithi032 ceph-mon[27388]: osdmap e761: 12 total, 12 up, 11 in 2024-09-16T05:21:51.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:51.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:50 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:21:51.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:50 smithi196 ceph-mon[30802]: osdmap e761: 12 total, 12 up, 11 in 2024-09-16T05:21:51.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:51.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:51.245 INFO:tasks.rados.rados.2.smithi204.stdout:2967: expect (ObjNum 998 snap 296 seq_num 998) 2024-09-16T05:21:51.335 INFO:tasks.rados.rados.2.smithi204.stdout:2972: finishing copy_from to smithi20457400-6 2024-09-16T05:21:51.336 INFO:tasks.rados.rados.2.smithi204.stdout:2972: got expected ENOENT (src dne) 2024-09-16T05:21:51.336 INFO:tasks.rados.rados.2.smithi204.stdout:2972: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:21:51.336 INFO:tasks.rados.rados.2.smithi204.stdout:2968: expect (ObjNum 1016 snap 309 seq_num 1016) 2024-09-16T05:21:51.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:51.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:51.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:51.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:51.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:51.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:21:51.556 INFO:tasks.rados.rados.2.smithi204.stdout:2966: expect (ObjNum 815 snap 237 seq_num 815) 2024-09-16T05:21:51.570 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:51.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:51.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:51.611 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:51.612 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:51.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:51.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:51.632 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:51.654 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:51.654 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:51.675 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:51.697 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:51.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:51.718 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:51.718 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:51.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:51.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:51.742 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:51.742 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:51.747 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:51.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:51.763 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:51.781 INFO:tasks.rados.rados.2.smithi204.stdout:2970: finishing write tid 4 to smithi20457400-38 2024-09-16T05:21:51.781 INFO:tasks.rados.rados.2.smithi204.stdout:2970: finishing write tid 5 to smithi20457400-38 2024-09-16T05:21:51.781 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 1032 (ObjNum 1042 snap 317 seq_num 1042) dirty exists 2024-09-16T05:21:51.781 INFO:tasks.rados.rados.2.smithi204.stdout:2970: left oid 38 (ObjNum 1042 snap 317 seq_num 1042) 2024-09-16T05:21:51.781 INFO:tasks.rados.rados.2.smithi204.stdout:2959: done (10 left) 2024-09-16T05:21:51.781 INFO:tasks.rados.rados.2.smithi204.stdout:2960: done (9 left) 2024-09-16T05:21:51.781 INFO:tasks.rados.rados.2.smithi204.stdout:2964: done (8 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2965: done (7 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2966: done (6 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2967: done (5 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2968: done (4 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2969: done (3 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2970: done (2 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2971: done (1 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2972: done (0 left) 2024-09-16T05:21:51.782 INFO:tasks.rados.rados.2.smithi204.stdout:2973: snap_remove snap 308 2024-09-16T05:21:51.784 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:51.784 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:51.805 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:51.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:51.861 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:51.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:51.913 INFO:tasks.rados.rados.2.smithi204.stdout:2973: done (0 left) 2024-09-16T05:21:51.913 INFO:tasks.rados.rados.2.smithi204.stdout:2974: write (excl) oid 50 current snap is 317 2024-09-16T05:21:51.913 INFO:tasks.rados.rados.2.smithi204.stdout:2974: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-09-16T05:21:51.920 INFO:tasks.rados.rados.2.smithi204.stdout:2974: writing smithi20457400-50 from 729701 to 1154769 tid 1 2024-09-16T05:21:51.926 INFO:tasks.rados.rados.2.smithi204.stdout:2974: writing smithi20457400-50 from 1667072 to 2319100 tid 2 2024-09-16T05:21:51.932 INFO:tasks.rados.rados.2.smithi204.stdout:2974: writing smithi20457400-50 from 2909005 to 3394185 tid 3 2024-09-16T05:21:51.933 INFO:tasks.rados.rados.2.smithi204.stdout:2975: rollback oid 32 current snap is 317 2024-09-16T05:21:51.933 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 314 2024-09-16T05:21:51.933 INFO:tasks.rados.rados.2.smithi204.stdout:2974: finishing write tid 1 to smithi20457400-50 2024-09-16T05:21:51.933 INFO:tasks.rados.rados.2.smithi204.stdout:2974: finishing write tid 2 to smithi20457400-50 2024-09-16T05:21:51.933 INFO:tasks.rados.rados.2.smithi204.stdout:2976: delete oid 42 current snap is 317 2024-09-16T05:21:51.934 INFO:tasks.rados.rados.2.smithi204.stdout:2975: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:21:51.935 INFO:tasks.rados.rados.2.smithi204.stdout:2975: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:21:51.935 INFO:tasks.rados.rados.2.smithi204.stdout:2975: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:21:51.935 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1157 (ObjNum 1015 snap 307 seq_num 1015) dirty exists 2024-09-16T05:21:51.936 INFO:tasks.rados.rados.2.smithi204.stdout:2974: finishing write tid 3 to smithi20457400-50 2024-09-16T05:21:51.937 INFO:tasks.rados.rados.2.smithi204.stdout:2974: finishing write tid 4 to smithi20457400-50 2024-09-16T05:21:51.937 INFO:tasks.rados.rados.2.smithi204.stdout:2974: finishing write tid 5 to smithi20457400-50 2024-09-16T05:21:51.937 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1131 (ObjNum 1043 snap 317 seq_num 1043) dirty exists 2024-09-16T05:21:51.937 INFO:tasks.rados.rados.2.smithi204.stdout:2974: left oid 50 (ObjNum 1043 snap 317 seq_num 1043) 2024-09-16T05:21:51.937 INFO:tasks.rados.rados.2.smithi204.stdout:2974: done (2 left) 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:2975: done (1 left) 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:2976: done (0 left) 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:2977: copy_from oid 14 from oid 40 current snap is 317 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:2978: read oid 34 snap -1 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:2978: expect (ObjNum 976 snap 287 seq_num 976) 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:2979: read oid 23 snap -1 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:2979: expect (ObjNum 1006 snap 303 seq_num 1006) 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:2980: rollback oid 29 current snap is 317 2024-09-16T05:21:51.938 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 314 2024-09-16T05:21:51.939 INFO:tasks.rados.rados.2.smithi204.stdout:2981: delete oid 6 current snap is 317 2024-09-16T05:21:51.939 INFO:tasks.rados.rados.2.smithi204.stdout:2981: done (4 left) 2024-09-16T05:21:51.939 INFO:tasks.rados.rados.2.smithi204.stdout:2982: read oid 1 snap -1 2024-09-16T05:21:51.939 INFO:tasks.rados.rados.2.smithi204.stdout:2982: expect deleted 2024-09-16T05:21:51.939 INFO:tasks.rados.rados.2.smithi204.stdout:2983: write oid 28 current snap is 317 2024-09-16T05:21:51.939 INFO:tasks.rados.rados.2.smithi204.stdout:2983: seq_num 1046 ranges {780122=650815,2126165=799355} 2024-09-16T05:21:51.948 INFO:tasks.rados.rados.2.smithi204.stdout:2983: writing smithi20457400-28 from 780122 to 1430937 tid 1 2024-09-16T05:21:51.956 INFO:tasks.rados.rados.2.smithi204.stdout:2983: writing smithi20457400-28 from 2126165 to 2925520 tid 2 2024-09-16T05:21:51.957 INFO:tasks.rados.rados.2.smithi204.stdout:2984: copy_from oid 16 from oid 4 current snap is 317 2024-09-16T05:21:51.957 INFO:tasks.rados.rados.2.smithi204.stdout:2980: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:21:51.957 INFO:tasks.rados.rados.2.smithi204.stdout:2980: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:21:51.957 INFO:tasks.rados.rados.2.smithi204.stdout:2980: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:21:51.957 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1174 (ObjNum 935 snap 277 seq_num 935) dirty exists 2024-09-16T05:21:51.957 INFO:tasks.rados.rados.2.smithi204.stdout:2978: expect (ObjNum 976 snap 287 seq_num 976) 2024-09-16T05:21:51.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:51.986 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:51.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:52.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:52.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:51 smithi032 ceph-mon[27388]: pgmap v1302: 127 pgs: 1 active+remapped, 126 active+clean; 284 MiB data, 944 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 17 op/s 2024-09-16T05:21:52.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2123123345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:52.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:52.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:52.121 INFO:tasks.rados.rados.2.smithi204.stdout:2977: finishing copy_from racing read to smithi20457400-14 2024-09-16T05:21:52.121 INFO:tasks.rados.rados.2.smithi204.stdout:2979: expect (ObjNum 1006 snap 303 seq_num 1006) 2024-09-16T05:21:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:51 smithi204 ceph-mon[30933]: pgmap v1302: 127 pgs: 1 active+remapped, 126 active+clean; 284 MiB data, 944 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 17 op/s 2024-09-16T05:21:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2123123345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:52.197 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:21:52.197 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:21:52.197 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:21:52.197 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:21:52.198 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:21:52.201 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1] out_osds: [6] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:21:52.201 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:21:52.204 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-16T05:21:52.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd in 6 2024-09-16T05:21:52.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:52.220 INFO:tasks.rados.rados.2.smithi204.stdout:2977: finishing copy_from to smithi20457400-14 2024-09-16T05:21:52.220 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1051 (ObjNum 1011 snap 306 seq_num 1011) dirty exists 2024-09-16T05:21:52.220 INFO:tasks.rados.rados.2.smithi204.stdout:2983: finishing write tid 1 to smithi20457400-28 2024-09-16T05:21:52.220 INFO:tasks.rados.rados.2.smithi204.stdout:2977: done (6 left) 2024-09-16T05:21:52.220 INFO:tasks.rados.rados.2.smithi204.stdout:2978: done (5 left) 2024-09-16T05:21:52.220 INFO:tasks.rados.rados.2.smithi204.stdout:2979: done (4 left) 2024-09-16T05:21:52.220 INFO:tasks.rados.rados.2.smithi204.stdout:2980: done (3 left) 2024-09-16T05:21:52.220 INFO:tasks.rados.rados.2.smithi204.stdout:2982: done (2 left) 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2985: copy_from oid 6 from oid 34 current snap is 317 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2983: finishing write tid 2 to smithi20457400-28 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2983: finishing write tid 3 to smithi20457400-28 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2983: finishing write tid 4 to smithi20457400-28 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1118 (ObjNum 1046 snap 317 seq_num 1046) dirty exists 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2983: left oid 28 (ObjNum 1046 snap 317 seq_num 1046) 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2984: finishing copy_from racing read to smithi20457400-16 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2984: finishing copy_from to smithi20457400-16 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1012 (ObjNum 1005 snap 303 seq_num 1005) dirty exists 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2983: done (2 left) 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2984: done (1 left) 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2986: read oid 22 snap -1 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2986: expect deleted 2024-09-16T05:21:52.221 INFO:tasks.rados.rados.2.smithi204.stdout:2987: copy_from oid 3 from oid 36 current snap is 317 2024-09-16T05:21:52.222 INFO:tasks.rados.rados.2.smithi204.stdout:2988: write oid 23 current snap is 317 2024-09-16T05:21:52.222 INFO:tasks.rados.rados.2.smithi204.stdout:2988: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-16T05:21:52.228 INFO:tasks.rados.rados.2.smithi204.stdout:2988: writing smithi20457400-23 from 796929 to 1522993 tid 1 2024-09-16T05:21:52.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:52.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:52.235 INFO:tasks.rados.rados.2.smithi204.stdout:2988: writing smithi20457400-23 from 2184647 to 2844013 tid 2 2024-09-16T05:21:52.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:51 smithi196 ceph-mon[30802]: pgmap v1302: 127 pgs: 1 active+remapped, 126 active+clean; 284 MiB data, 944 MiB used, 983 GiB / 983 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 17 op/s 2024-09-16T05:21:52.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2123123345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:52.239 INFO:tasks.rados.rados.2.smithi204.stdout:2988: writing smithi20457400-23 from 3302344 to 3596929 tid 3 2024-09-16T05:21:52.239 INFO:tasks.rados.rados.2.smithi204.stdout:2989: write oid 25 current snap is 317 2024-09-16T05:21:52.239 INFO:tasks.rados.rados.2.smithi204.stdout:2989: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2024-09-16T05:21:52.244 INFO:tasks.rados.rados.2.smithi204.stdout:2989: writing smithi20457400-25 from 413736 to 815049 tid 1 2024-09-16T05:21:52.249 INFO:tasks.rados.rados.2.smithi204.stdout:2989: writing smithi20457400-25 from 1326776 to 1729800 tid 2 2024-09-16T05:21:52.255 INFO:tasks.rados.rados.2.smithi204.stdout:2989: writing smithi20457400-25 from 2413414 to 2871307 tid 3 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:2989: writing smithi20457400-25 from 3438716 to 3613736 tid 4 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:2990: delete oid 7 current snap is 317 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:2985: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:2987: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:2985: finishing copy_from to smithi20457400-6 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 1233 (ObjNum 976 snap 287 seq_num 976) dirty exists 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:2987: finishing copy_from to smithi20457400-3 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 768 (ObjNum 993 snap 292 seq_num 993) dirty exists 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:2988: finishing write tid 1 to smithi20457400-23 2024-09-16T05:21:52.260 INFO:tasks.rados.rados.2.smithi204.stdout:2988: finishing write tid 2 to smithi20457400-23 2024-09-16T05:21:52.261 INFO:tasks.rados.rados.2.smithi204.stdout:2988: finishing write tid 3 to smithi20457400-23 2024-09-16T05:21:52.261 INFO:tasks.rados.rados.2.smithi204.stdout:2988: finishing write tid 4 to smithi20457400-23 2024-09-16T05:21:52.261 INFO:tasks.rados.rados.2.smithi204.stdout:2988: finishing write tid 5 to smithi20457400-23 2024-09-16T05:21:52.261 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 729 (ObjNum 1047 snap 317 seq_num 1047) dirty exists 2024-09-16T05:21:52.261 INFO:tasks.rados.rados.2.smithi204.stdout:2988: left oid 23 (ObjNum 1047 snap 317 seq_num 1047) 2024-09-16T05:21:52.261 INFO:tasks.rados.rados.2.smithi204.stdout:2989: finishing write tid 1 to smithi20457400-25 2024-09-16T05:21:52.261 INFO:tasks.rados.rados.2.smithi204.stdout:2989: finishing write tid 2 to smithi20457400-25 2024-09-16T05:21:52.261 INFO:tasks.rados.rados.2.smithi204.stdout:2989: finishing write tid 3 to smithi20457400-25 2024-09-16T05:21:52.262 INFO:tasks.rados.rados.2.smithi204.stdout:2985: done (5 left) 2024-09-16T05:21:52.262 INFO:tasks.rados.rados.2.smithi204.stdout:2986: done (4 left) 2024-09-16T05:21:52.262 INFO:tasks.rados.rados.2.smithi204.stdout:2987: done (3 left) 2024-09-16T05:21:52.262 INFO:tasks.rados.rados.2.smithi204.stdout:2988: done (2 left) 2024-09-16T05:21:52.262 INFO:tasks.rados.rados.2.smithi204.stdout:2990: done (1 left) 2024-09-16T05:21:52.262 INFO:tasks.rados.rados.2.smithi204.stdout:2991: copy_from oid 4 from oid 43 current snap is 317 2024-09-16T05:21:52.262 INFO:tasks.rados.rados.2.smithi204.stdout:2992: write (excl) oid 37 current snap is 317 2024-09-16T05:21:52.262 INFO:tasks.rados.rados.2.smithi204.stdout:2992: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2024-09-16T05:21:52.268 INFO:tasks.rados.rados.2.smithi204.stdout:2992: writing smithi20457400-37 from 447350 to 1115514 tid 1 2024-09-16T05:21:52.272 INFO:tasks.rados.rados.2.smithi204.stdout:2992: writing smithi20457400-37 from 1560093 to 1966786 tid 2 2024-09-16T05:21:52.279 INFO:tasks.rados.rados.2.smithi204.stdout:2992: writing smithi20457400-37 from 2584613 to 3065050 tid 3 2024-09-16T05:21:52.279 INFO:tasks.rados.rados.2.smithi204.stdout:2993: snap_create 2024-09-16T05:21:52.279 INFO:tasks.rados.rados.2.smithi204.stdout:2989: finishing write tid 4 to smithi20457400-25 2024-09-16T05:21:52.279 INFO:tasks.rados.rados.2.smithi204.stdout:2989: finishing write tid 5 to smithi20457400-25 2024-09-16T05:21:52.279 INFO:tasks.rados.rados.2.smithi204.stdout:2989: finishing write tid 6 to smithi20457400-25 2024-09-16T05:21:52.279 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 739 (ObjNum 1048 snap 317 seq_num 1048) dirty exists 2024-09-16T05:21:52.279 INFO:tasks.rados.rados.2.smithi204.stdout:2989: left oid 25 (ObjNum 1048 snap 317 seq_num 1048) 2024-09-16T05:21:52.279 INFO:tasks.rados.rados.2.smithi204.stdout:2991: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:21:52.280 INFO:tasks.rados.rados.2.smithi204.stdout:2991: finishing copy_from to smithi20457400-4 2024-09-16T05:21:52.280 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1041 (ObjNum 1031 snap 316 seq_num 1031) dirty exists 2024-09-16T05:21:52.280 INFO:tasks.rados.rados.2.smithi204.stdout:2992: finishing write tid 1 to smithi20457400-37 2024-09-16T05:21:52.280 INFO:tasks.rados.rados.2.smithi204.stdout:2992: finishing write tid 2 to smithi20457400-37 2024-09-16T05:21:52.280 INFO:tasks.rados.rados.2.smithi204.stdout:2992: finishing write tid 3 to smithi20457400-37 2024-09-16T05:21:52.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:52.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:52.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:52.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:52.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:52.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:52.500 INFO:tasks.rados.rados.2.smithi204.stdout:2992: finishing write tid 4 to smithi20457400-37 2024-09-16T05:21:52.500 INFO:tasks.rados.rados.2.smithi204.stdout:2992: finishing write tid 5 to smithi20457400-37 2024-09-16T05:21:52.500 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1044 (ObjNum 1050 snap 317 seq_num 1050) dirty exists 2024-09-16T05:21:52.500 INFO:tasks.rados.rados.2.smithi204.stdout:2992: left oid 37 (ObjNum 1050 snap 317 seq_num 1050) 2024-09-16T05:21:52.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:52.606 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:52.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:52.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:52.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:21:52.640 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:52.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:52.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:52.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:52.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:52.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:52.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:21:52.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2989: done (3 left) 2024-09-16T05:21:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2991: done (2 left) 2024-09-16T05:21:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:2992: done (1 left) 2024-09-16T05:21:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:2993: done (0 left) 2024-09-16T05:21:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:2994: write (excl) oid 30 current snap is 318 2024-09-16T05:21:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:2994: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2024-09-16T05:21:52.924 INFO:tasks.rados.rados.2.smithi204.stdout:2994: writing smithi20457400-30 from 464157 to 1207570 tid 1 2024-09-16T05:21:52.931 INFO:tasks.rados.rados.2.smithi204.stdout:2994: writing smithi20457400-30 from 1618575 to 2285279 tid 2 2024-09-16T05:21:52.937 INFO:tasks.rados.rados.2.smithi204.stdout:2994: writing smithi20457400-30 from 3012036 to 3420098 tid 3 2024-09-16T05:21:52.937 INFO:tasks.rados.rados.2.smithi204.stdout:2995: write oid 23 current snap is 318 2024-09-16T05:21:52.937 INFO:tasks.rados.rados.2.smithi204.stdout:2994: finishing write tid 1 to smithi20457400-30 2024-09-16T05:21:52.937 INFO:tasks.rados.rados.2.smithi204.stdout:2994: finishing write tid 2 to smithi20457400-30 2024-09-16T05:21:52.937 INFO:tasks.rados.rados.2.smithi204.stdout:2995: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2024-09-16T05:21:52.941 INFO:tasks.rados.rados.2.smithi204.stdout:2995: writing smithi20457400-23 from 480964 to 899626 tid 1 2024-09-16T05:21:52.947 INFO:tasks.rados.rados.2.smithi204.stdout:2995: writing smithi20457400-23 from 1560704 to 1971066 tid 2 2024-09-16T05:21:52.954 INFO:tasks.rados.rados.2.smithi204.stdout:2995: writing smithi20457400-23 from 2523106 to 3142440 tid 3 2024-09-16T05:21:52.954 INFO:tasks.rados.rados.2.smithi204.stdout:2996: write (excl) oid 8 current snap is 318 2024-09-16T05:21:52.954 INFO:tasks.rados.rados.2.smithi204.stdout:2994: finishing write tid 3 to smithi20457400-30 2024-09-16T05:21:52.954 INFO:tasks.rados.rados.2.smithi204.stdout:2994: finishing write tid 4 to smithi20457400-30 2024-09-16T05:21:52.954 INFO:tasks.rados.rados.2.smithi204.stdout:2994: finishing write tid 5 to smithi20457400-30 2024-09-16T05:21:52.955 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 1329 (ObjNum 1051 snap 318 seq_num 1051) dirty exists 2024-09-16T05:21:52.955 INFO:tasks.rados.rados.2.smithi204.stdout:2994: left oid 30 (ObjNum 1051 snap 318 seq_num 1051) 2024-09-16T05:21:52.955 INFO:tasks.rados.rados.2.smithi204.stdout:2995: finishing write tid 1 to smithi20457400-23 2024-09-16T05:21:52.955 INFO:tasks.rados.rados.2.smithi204.stdout:2995: finishing write tid 2 to smithi20457400-23 2024-09-16T05:21:52.955 INFO:tasks.rados.rados.2.smithi204.stdout:2996: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-09-16T05:21:52.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:52.959 INFO:tasks.rados.rados.2.smithi204.stdout:2996: writing smithi20457400-8 from 497771 to 991682 tid 1 2024-09-16T05:21:52.966 INFO:tasks.rados.rados.2.smithi204.stdout:2996: writing smithi20457400-8 from 1619186 to 2173206 tid 2 2024-09-16T05:21:52.973 INFO:tasks.rados.rados.2.smithi204.stdout:2996: writing smithi20457400-8 from 2950529 to 3381135 tid 3 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:2994: done (2 left) 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:2997: snap_create 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:2995: finishing write tid 3 to smithi20457400-23 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:2995: finishing write tid 4 to smithi20457400-23 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:2995: finishing write tid 5 to smithi20457400-23 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 733 (ObjNum 1052 snap 318 seq_num 1052) dirty exists 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:2995: left oid 23 (ObjNum 1052 snap 318 seq_num 1052) 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:2996: finishing write tid 1 to smithi20457400-8 2024-09-16T05:21:52.974 INFO:tasks.rados.rados.2.smithi204.stdout:2996: finishing write tid 2 to smithi20457400-8 2024-09-16T05:21:52.976 INFO:tasks.rados.rados.2.smithi204.stdout:2996: finishing write tid 3 to smithi20457400-8 2024-09-16T05:21:52.976 INFO:tasks.rados.rados.2.smithi204.stdout:2996: finishing write tid 4 to smithi20457400-8 2024-09-16T05:21:52.976 INFO:tasks.rados.rados.2.smithi204.stdout:2996: finishing write tid 5 to smithi20457400-8 2024-09-16T05:21:52.977 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1275 (ObjNum 1053 snap 318 seq_num 1053) dirty exists 2024-09-16T05:21:52.977 INFO:tasks.rados.rados.2.smithi204.stdout:2996: left oid 8 (ObjNum 1053 snap 318 seq_num 1053) 2024-09-16T05:21:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:52.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:53.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:53.100 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:21:53.103 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:53.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:53.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:52 smithi032 ceph-mon[27388]: osdmap e762: 12 total, 12 up, 11 in 2024-09-16T05:21:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:52 smithi204 ceph-mon[30933]: osdmap e762: 12 total, 12 up, 11 in 2024-09-16T05:21:53.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:53.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:53.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:53.235 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:53.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:52 smithi196 ceph-mon[30802]: osdmap e762: 12 total, 12 up, 11 in 2024-09-16T05:21:53.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:53.355 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:53.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:53.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:21:53.362+0000 7fdafd331640 0 --2- 172.21.15.32:0/4146133814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdaf8102790 0x7fdaf8198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:21:53.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:53.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:53.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:53.557 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:21:53.557 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:21:53.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:1575 2.5 e227/200 2:a5bc1608:::smithi20457400-11:head v 227'370, mlcod=227'365)", 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:38.692673+0000", 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 554.86199569899998, 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64070468899999999, 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692673+0000", 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692673+0000", 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692674+0000", 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8999999999999996e-07 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:38.692680+0000", 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1070000000000001e-06 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.332765+0000", 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.64008575199999995 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.332775+0000", 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2609999999999999e-06 2024-09-16T05:21:53.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.332845+0000", 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0035000000000003e-05 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.332874+0000", 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9138000000000001e-05 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333328+0000", 2024-09-16T05:21:53.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045414700000000001 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:39.333378+0000", 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9558999999999998e-05 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:547 2.25 e278/252 2:a5bc1608:::smithi20457400-11:52 v 278'476, mlcod=277'470)", 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:21:53.562 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 503.82130098099998, 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27999907200000002, 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:21:53.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4900000000000005e-07 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733374+0000", 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0669999999999997e-06 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012754+0000", 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27937933999999998 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012763+0000", 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.884e-06 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012833+0000", 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0779999999999997e-05 2024-09-16T05:21:53.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012856+0000", 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.283e-05 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.013333+0000", 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00047694000000000002 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.013367+0000", 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3482000000000002e-05 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:379 2.35 e346/275 2:ad4a88eb:::smithi20457400-16:6b v 346'436, mlcod=340'432)", 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 417.85758051400001, 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72722915899999996, 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.567 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1100000000000002e-07 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697095+0000", 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8709999999999999e-06 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423764+0000", 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72666896299999995 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423770+0000", 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0129999999999999e-06 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423829+0000", 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8807999999999997e-05 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423855+0000", 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5721e-05 2024-09-16T05:21:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.424282+0000", 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042752699999999997 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.424317+0000", 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4944999999999998e-05 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:414 2.35 e378/275 2:ad4a88eb:::smithi20457400-16:6b v 378'499, mlcod=377'495)", 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 385.81280019500002, 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.883103101, 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741869+0000", 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8099999999999999e-07 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741872+0000", 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7979999999999999e-06 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624224+0000", 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.88235191599999996 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624240+0000", 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5296000000000001e-05 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624347+0000", 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107249 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624381+0000", 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4480999999999997e-05 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624917+0000", 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053571399999999996 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624972+0000", 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4466000000000003e-05 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3036 2.1b 2:da4f6049:::smithi20457400-49:head [setxattr _header (58) in=7b,truncate 3294238] snapc 7e=[7e,7d,7c,7b] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 384.787549533, 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.437857206, 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3036 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.577 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767120+0000", 2024-09-16T05:21:53.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0800000000000004e-07 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767123+0000", 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.642e-06 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767132+0000", 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2819999999999996e-06 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767138+0000", 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5200000000000003e-06 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767869+0000", 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00073052499999999999 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767959+0000", 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0730999999999996e-05 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9", 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768106+0000", 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014667900000000001 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768798+0000", 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069156399999999998 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204863+0000", 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43606497999999999 2024-09-16T05:21:53.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204909+0000", 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6758000000000002e-05 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204976+0000", 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6816999999999994e-05 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3037 2.1b 2:da4f6049:::smithi20457400-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e379)", 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 384.78751778600002, 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43800830899999998, 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3037 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6199999999999998e-07 2024-09-16T05:21:53.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767154+0000", 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6730000000000001e-06 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767161+0000", 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.849e-06 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767166+0000", 2024-09-16T05:21:53.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4709999999999998e-06 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768487+0000", 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001320543 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768584+0000", 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7547999999999995e-05 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:53.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205011+0000", 2024-09-16T05:21:53.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43642700000000001 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205081+0000", 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9381000000000005e-05 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205159+0000", 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8381999999999996e-05 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.589 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:3951 2.2f e467/275 2:f4b938a4:::smithi20457400-47:head v 467'381, mlcod=467'378)", 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 296.69529979399999, 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39001122199999999, 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6400000000000002e-07 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859375+0000", 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1599999999999997e-06 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248747+0000", 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38937218699999998 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248762+0000", 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.537e-05 2024-09-16T05:21:53.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248857+0000", 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4630999999999996e-05 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248883+0000", 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6448000000000002e-05 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249341+0000", 2024-09-16T05:21:53.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000457486 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249380+0000", 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9276000000000001e-05 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:500 2.35 e550/275 2:ad4a88eb:::smithi20457400-16:d3 v 550'597, mlcod=547'593)", 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:19.772586+0000", 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 213.78208307200001, 2024-09-16T05:21:53.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.068775681000000005, 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.772586+0000", 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.772586+0000", 2024-09-16T05:21:53.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.772586+0000", 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0100000000000001e-07 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.772593+0000", 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7959999999999998e-06 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.840636+0000", 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.068043812999999995 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.840645+0000", 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0750000000000004e-06 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.840712+0000", 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6260999999999995e-05 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.840745+0000", 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3658999999999997e-05 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.841326+0000", 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00058029200000000005 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.841361+0000", 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5583999999999997e-05 2024-09-16T05:21:53.598 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:521 2.35 e564/275 2:ad7afe2b:::smithi20457400-38:c5 v 564'628, mlcod=561'622)", 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:33.829005+0000", 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 199.725663481, 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19962600699999999, 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:33.829005+0000", 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:33.829005+0000", 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:33.829006+0000", 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5300000000000002e-07 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:33.829013+0000", 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5339999999999996e-06 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.027927+0000", 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19891482599999999 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.027941+0000", 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3913e-05 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.028031+0000", 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9858999999999995e-05 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.028051+0000", 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9726000000000001e-05 2024-09-16T05:21:53.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.028589+0000", 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053826500000000001 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.028631+0000", 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1931000000000002e-05 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:1041 2.25 e574/252 2:a49174eb:::smithi20457400-39:dc v 574'824, mlcod=572'821)", 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.783282+0000", 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 189.771387011, 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.203064307, 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.783282+0000", 2024-09-16T05:21:53.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.783282+0000", 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.783282+0000", 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8900000000000001e-07 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.783286+0000", 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9829999999999998e-06 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.985535+0000", 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.202249346 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.985540+0000", 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8810000000000004e-06 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.985588+0000", 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8241000000000001e-05 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.985606+0000", 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7227e-05 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.986312+0000", 2024-09-16T05:21:53.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00070611199999999995 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.986346+0000", 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4227999999999998e-05 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:5460 2.25 e574/252 2:a5bc1608:::smithi20457400-11:head v 574'829, mlcod=574'824)", 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.995283+0000", 2024-09-16T05:21:53.608 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 189.55938572599999, 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69040396400000004, 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995283+0000", 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995283+0000", 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995284+0000", 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9599999999999999e-07 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995289+0000", 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5709999999999997e-06 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.684770+0000", 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68948120400000001 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.684792+0000", 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1500000000000001e-05 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.684936+0000", 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014458400000000001 2024-09-16T05:21:53.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.684983+0000", 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6122999999999999e-05 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685633+0000", 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000650615 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685687+0000", 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3671000000000002e-05 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6951 2.35 2:ad4a88eb:::smithi20457400-16:head [copy-from ver 1018] snapc 120=[120,11f,11e,11d,11b,119,118,116,113,10c,10b,108,f2] ondisk+write+known_if_redirected+supports_pool_eio e698)", 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:47.976824+0000", 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 65.577844948999996, 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016999719999999999, 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.613 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6951 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976824+0000", 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976824+0000", 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976825+0000", 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5960000000000001e-06 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976834+0000", 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1970000000000002e-06 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976852+0000", 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8026999999999999e-05 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976860+0000", 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5689999999999999e-06 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976939+0000", 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8893999999999994e-05 2024-09-16T05:21:53.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.977047+0000", 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010775799999999999 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 10", 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.985484+0000", 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084370569999999995 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.993343+0000", 2024-09-16T05:21:53.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078589039999999999 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.993708+0000", 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000365752 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.993768+0000", 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9867999999999997e-05 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.993823+0000", 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5098e-05 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6952 2.35 2:ad4a88eb:::smithi20457400-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e698)", 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:47.976880+0000", 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 65.577788600999995, 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013156342, 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.619 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6952 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976880+0000", 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976880+0000", 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976881+0000", 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976883+0000", 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5679999999999998e-06 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976897+0000", 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3559e-05 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976903+0000", 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9440000000000002e-06 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.977185+0000", 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028197399999999999 2024-09-16T05:21:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.977224+0000", 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9551000000000001e-05 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.989863+0000", 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01263923 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.989942+0000", 2024-09-16T05:21:53.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8119000000000002e-05 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.990036+0000", 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4833999999999994e-05 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:591 2.35 e737/720 2:ad4a88eb:::smithi20457400-16:120 v 737'830, mlcod=733'826)", 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.822768+0000", 2024-09-16T05:21:53.624 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.731900256999999, 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075932930000000001, 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.822768+0000", 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.822768+0000", 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.822769+0000", 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.67e-07 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.822786+0000", 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6957e-05 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.824422+0000", 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016366110000000001 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.824431+0000", 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0140000000000004e-06 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.824546+0000", 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011441400000000001 2024-09-16T05:21:53.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.824573+0000", 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7613000000000001e-05 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.830314+0000", 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005741192 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.830362+0000", 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7225000000000003e-05 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7406 2.1b 2:d8680303:::smithi20457400-43:head [copy-from ver 1037] snapc 13d=[13d,13c,13b,13a,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e753)", 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:42.818053+0000", 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 10.736615268, 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020181930000000001, 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.629 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7406 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818053+0000", 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818053+0000", 2024-09-16T05:21:53.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818055+0000", 2024-09-16T05:21:53.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.858e-06 2024-09-16T05:21:53.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818063+0000", 2024-09-16T05:21:53.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1820000000000006e-06 2024-09-16T05:21:53.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818077+0000", 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3909e-05 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818085+0000", 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9109999999999993e-06 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818161+0000", 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5327999999999994e-05 2024-09-16T05:21:53.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818167+0000", 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.195e-06 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.820800+0000", 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002633196 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.820900+0000", 2024-09-16T05:21:53.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010043 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9", 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.830921+0000", 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010020443 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.835343+0000", 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044221850000000004 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.838150+0000", 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028066440000000001 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.838195+0000", 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5076e-05 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.838235+0000", 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0572999999999999e-05 2024-09-16T05:21:53.636 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7407 2.1b 2:d8680303:::smithi20457400-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e753)", 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:42.818106+0000", 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 10.736562171999999, 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016685466999999999, 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:53.637 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7407 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818106+0000", 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818106+0000", 2024-09-16T05:21:53.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818107+0000", 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2900000000000003e-07 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818110+0000", 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9119999999999998e-06 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818118+0000", 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0579999999999998e-06 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818124+0000", 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4150000000000004e-06 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818226+0000", 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010162600000000001 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818232+0000", 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5369999999999997e-06 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.821131+0000", 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028996080000000001 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.821179+0000", 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7923999999999997e-05 2024-09-16T05:21:53.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.834591+0000", 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013412076 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.834701+0000", 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109648 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.834792+0000", 2024-09-16T05:21:53.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1033999999999996e-05 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.9.0:1394 2.2f e762/275 2:f6b419a1:::smithi20457400-37:13a v 762'658, mlcod=761'655)", 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:51.918605+0000", 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.6360637840000001, 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.35210398199999998, 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.643 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:51.918605+0000", 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:51.918605+0000", 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:51.918606+0000", 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9999999999999996e-07 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:51.918614+0000", 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2689999999999999e-06 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.269687+0000", 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.351073262 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.269697+0000", 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5310000000000007e-06 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.269766+0000", 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8917000000000004e-05 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.269792+0000", 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5976e-05 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.270666+0000", 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087391499999999996 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.270709+0000", 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3211999999999999e-05 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7568 2.3e 2:7e4e65a4:::smithi20457400-4:head [copy-from ver 1041] snapc 13f=[13f,13d,13b,134,133,12c,12a,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.261590+0000", 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.293078924, 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012384367, 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7568 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261590+0000", 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261590+0000", 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261592+0000", 2024-09-16T05:21:53.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0480000000000001e-06 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261601+0000", 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5279999999999992e-06 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261621+0000", 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9890000000000001e-05 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261631+0000", 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3500000000000003e-06 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261691+0000", 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0608e-05 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261780+0000", 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8609999999999994e-05 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.267171+0000", 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053912819999999998 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.271918+0000", 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004746686 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.273857+0000", 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001939084 2024-09-16T05:21:53.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.273922+0000", 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4955000000000004e-05 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.273974+0000", 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2326e-05 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7569 2.3e 2:7e4e65a4:::smithi20457400-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.261649+0000", 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.293019548, 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095769540000000004, 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7569 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261649+0000", 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261649+0000", 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261650+0000", 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4799999999999999e-07 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261652+0000", 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5799999999999999e-06 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261661+0000", 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3049999999999998e-06 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261667+0000", 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1959999999999999e-06 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261893+0000", 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000226047 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261932+0000", 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9419000000000003e-05 2024-09-16T05:21:53.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.271017+0000", 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0090846869999999993 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.271120+0000", 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010286 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.271226+0000", 2024-09-16T05:21:53.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106412 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:7573 2.2f e762/275 2:f6b419a1:::smithi20457400-37:head v 762'662, mlcod=762'658)", 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.280222+0000", 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.274447111, 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21922077500000001, 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:21:53.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280222+0000", 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280222+0000", 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280222+0000", 2024-09-16T05:21:53.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9299999999999998e-07 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280227+0000", 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5399999999999997e-06 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.498773+0000", 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218545884 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:21:53.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.498781+0000", 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1780000000000003e-06 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.498869+0000", 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8764000000000006e-05 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.498891+0000", 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1588999999999999e-05 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.499404+0000", 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00051307000000000002 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.499442+0000", 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8157000000000001e-05 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:21:53.663 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:21:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:53.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:53.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:53.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:53.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:53.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:53.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:marked in osd.6. 2024-09-16T05:21:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:21:53.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:53 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:21:53.559+0000 7f7c3f992640 -1 mon.a@0(leader).osd e763 definitely_dead 0 2024-09-16T05:21:53.928 INFO:tasks.rados.rados.2.smithi204.stdout:2995: done (2 left) 2024-09-16T05:21:53.928 INFO:tasks.rados.rados.2.smithi204.stdout:2996: done (1 left) 2024-09-16T05:21:53.928 INFO:tasks.rados.rados.2.smithi204.stdout:2997: done (0 left) 2024-09-16T05:21:53.928 INFO:tasks.rados.rados.2.smithi204.stdout:2998: copy_from oid 23 from oid 15 current snap is 319 2024-09-16T05:21:53.928 INFO:tasks.rados.rados.2.smithi204.stdout:2999: rollback oid 25 current snap is 319 2024-09-16T05:21:53.928 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 276 2024-09-16T05:21:53.928 INFO:tasks.rados.rados.2.smithi204.stdout:3000: snap_create 2024-09-16T05:21:53.932 INFO:tasks.rados.rados.2.smithi204.stdout:2999: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:21:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:53.942 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:53.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:53.942 INFO:tasks.rados.rados.2.smithi204.stdout:2998: finishing copy_from racing read to smithi20457400-23 2024-09-16T05:21:53.945 INFO:tasks.rados.rados.2.smithi204.stdout:2998: finishing copy_from to smithi20457400-23 2024-09-16T05:21:53.945 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 747 (ObjNum 906 snap 267 seq_num 906) dirty exists 2024-09-16T05:21:54.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:54.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:54.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:54.100 INFO:tasks.rados.rados.2.smithi204.stdout:2999: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:21:54.101 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 741 (ObjNum -12854528 snap 22023 seq_num 304) dirty dne 2024-09-16T05:21:54.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:53 smithi032 ceph-mon[27388]: pgmap v1305: 127 pgs: 127 active+clean; 286 MiB data, 950 MiB used, 982 GiB / 983 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-09-16T05:21:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:53 smithi032 ceph-mon[27388]: osdmap e763: 12 total, 12 up, 11 in 2024-09-16T05:21:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4146133814' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-16T05:21:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1753592686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:54.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:54.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:53 smithi196 ceph-mon[30802]: pgmap v1305: 127 pgs: 127 active+clean; 286 MiB data, 950 MiB used, 982 GiB / 983 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-09-16T05:21:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:53 smithi196 ceph-mon[30802]: osdmap e763: 12 total, 12 up, 11 in 2024-09-16T05:21:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4146133814' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-16T05:21:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1753592686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:54.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:54.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:54.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:54.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:53 smithi204 ceph-mon[30933]: pgmap v1305: 127 pgs: 127 active+clean; 286 MiB data, 950 MiB used, 982 GiB / 983 GiB avail; 17 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-09-16T05:21:54.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:53 smithi204 ceph-mon[30933]: osdmap e763: 12 total, 12 up, 11 in 2024-09-16T05:21:54.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4146133814' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-16T05:21:54.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1753592686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:21:54.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:21:54.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:54.481 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:54.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:54.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:21:54.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:54.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:54.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:54.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:54.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:54.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:54.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:54.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:54.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:54.924 INFO:tasks.rados.rados.2.smithi204.stdout:2998: done (2 left) 2024-09-16T05:21:54.924 INFO:tasks.rados.rados.2.smithi204.stdout:2999: done (1 left) 2024-09-16T05:21:54.924 INFO:tasks.rados.rados.2.smithi204.stdout:3000: done (0 left) 2024-09-16T05:21:54.924 INFO:tasks.rados.rados.2.smithi204.stdout:3001: read oid 48 snap 312 2024-09-16T05:21:54.924 INFO:tasks.rados.rados.2.smithi204.stdout:3001: expect (ObjNum 966 snap 285 seq_num 966) 2024-09-16T05:21:54.925 INFO:tasks.rados.rados.2.smithi204.stdout:3002: write (excl) oid 21 current snap is 320 2024-09-16T05:21:54.925 INFO:tasks.rados.rados.2.smithi204.stdout:3002: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2024-09-16T05:21:54.929 INFO:tasks.rados.rados.2.smithi204.stdout:3002: writing smithi20457400-21 from 514578 to 1083738 tid 1 2024-09-16T05:21:54.935 INFO:tasks.rados.rados.2.smithi204.stdout:3002: writing smithi20457400-21 from 1677668 to 2091699 tid 2 2024-09-16T05:21:54.942 INFO:tasks.rados.rados.2.smithi204.stdout:3002: writing smithi20457400-21 from 2577952 to 3219830 tid 3 2024-09-16T05:21:54.942 INFO:tasks.rados.rados.2.smithi204.stdout:3003: copy_from oid 6 from oid 36 current snap is 320 2024-09-16T05:21:54.942 INFO:tasks.rados.rados.2.smithi204.stdout:3001: expect (ObjNum 966 snap 285 seq_num 966) 2024-09-16T05:21:54.958 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-16T05:21:54.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:54.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:54.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:55.077 INFO:tasks.rados.rados.2.smithi204.stdout:3002: finishing write tid 1 to smithi20457400-21 2024-09-16T05:21:55.077 INFO:tasks.rados.rados.2.smithi204.stdout:3002: finishing write tid 2 to smithi20457400-21 2024-09-16T05:21:55.077 INFO:tasks.rados.rados.2.smithi204.stdout:3002: finishing write tid 3 to smithi20457400-21 2024-09-16T05:21:55.077 INFO:tasks.rados.rados.2.smithi204.stdout:3002: finishing write tid 4 to smithi20457400-21 2024-09-16T05:21:55.077 INFO:tasks.rados.rados.2.smithi204.stdout:3002: finishing write tid 5 to smithi20457400-21 2024-09-16T05:21:55.077 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 856 (ObjNum 1054 snap 320 seq_num 1054) dirty exists 2024-09-16T05:21:55.078 INFO:tasks.rados.rados.2.smithi204.stdout:3002: left oid 21 (ObjNum 1054 snap 320 seq_num 1054) 2024-09-16T05:21:55.078 INFO:tasks.rados.rados.2.smithi204.stdout:3003: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:21:55.078 INFO:tasks.rados.rados.2.smithi204.stdout:3003: finishing copy_from to smithi20457400-6 2024-09-16T05:21:55.078 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 768 (ObjNum 993 snap 292 seq_num 993) dirty exists 2024-09-16T05:21:55.078 INFO:tasks.rados.rados.2.smithi204.stdout:3001: done (2 left) 2024-09-16T05:21:55.078 INFO:tasks.rados.rados.2.smithi204.stdout:3002: done (1 left) 2024-09-16T05:21:55.078 INFO:tasks.rados.rados.2.smithi204.stdout:3003: done (0 left) 2024-09-16T05:21:55.078 INFO:tasks.rados.rados.2.smithi204.stdout:3004: snap_remove snap 306 2024-09-16T05:21:55.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:55.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:55.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:55.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:55.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:55.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:55.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4146133814' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-16T05:21:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1753592686' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:54 smithi196 ceph-mon[30802]: osdmap e764: 12 total, 12 up, 12 in 2024-09-16T05:21:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:54 smithi196 ceph-mon[30802]: osdmap e765: 12 total, 12 up, 12 in 2024-09-16T05:21:55.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4146133814' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-16T05:21:55.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1753592686' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:55.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:54 smithi032 ceph-mon[27388]: osdmap e764: 12 total, 12 up, 12 in 2024-09-16T05:21:55.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:54 smithi032 ceph-mon[27388]: osdmap e765: 12 total, 12 up, 12 in 2024-09-16T05:21:55.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:55.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:55.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:55.360 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:55.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4146133814' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-16T05:21:55.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1753592686' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:21:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:54 smithi204 ceph-mon[30933]: osdmap e764: 12 total, 12 up, 12 in 2024-09-16T05:21:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:54 smithi204 ceph-mon[30933]: osdmap e765: 12 total, 12 up, 12 in 2024-09-16T05:21:55.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:21:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:55.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:55.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:21:55.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:55.586 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:55.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:55.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:55.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:55.735 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:55.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:55.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:55.823 INFO:tasks.rados.rados.2.smithi204.stdout:3004: done (0 left) 2024-09-16T05:21:55.824 INFO:tasks.rados.rados.2.smithi204.stdout:3005: snap_create 2024-09-16T05:21:55.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:55.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:55.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:55.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:55.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:55.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:55.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:56.086 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:21:56.086 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:21:56.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:21:56.108 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:21:56.108 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:21:56.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:56.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:56.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:21:56.128 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:21:56.129 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:21:56.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:55 smithi032 ceph-mon[27388]: pgmap v1308: 127 pgs: 2 peering, 125 active+clean; 290 MiB data, 960 MiB used, 982 GiB / 983 GiB avail; 23 MiB/s rd, 3.9 MiB/s wr, 28 op/s 2024-09-16T05:21:56.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:55 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:56.129 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:55 smithi032 ceph-mon[27388]: osdmap e766: 12 total, 12 up, 12 in 2024-09-16T05:21:56.149 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:21:56.149 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:21:56.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:21:56.171 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:21:56.193 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:21:56.193 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:21:56.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:56.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:21:56.215 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:21:56.237 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:21:56.237 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:21:56.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:55 smithi196 ceph-mon[30802]: pgmap v1308: 127 pgs: 2 peering, 125 active+clean; 290 MiB data, 960 MiB used, 982 GiB / 983 GiB avail; 23 MiB/s rd, 3.9 MiB/s wr, 28 op/s 2024-09-16T05:21:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:55 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:55 smithi196 ceph-mon[30802]: osdmap e766: 12 total, 12 up, 12 in 2024-09-16T05:21:56.238 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:56.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:56.286 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:56.286 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:56.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:56.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:56.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:55 smithi204 ceph-mon[30933]: pgmap v1308: 127 pgs: 2 peering, 125 active+clean; 290 MiB data, 960 MiB used, 982 GiB / 983 GiB avail; 23 MiB/s rd, 3.9 MiB/s wr, 28 op/s 2024-09-16T05:21:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:55 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:21:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:55 smithi204 ceph-mon[30933]: osdmap e766: 12 total, 12 up, 12 in 2024-09-16T05:21:56.460 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:21:56.461 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:21:56.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:56.486 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:21:56.487 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:21:56.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:56.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:56.511 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:21:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:21:56.533 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:21:56.533 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:21:56.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:56.616 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:56.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:56.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:56.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:56.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:56.819 INFO:tasks.rados.rados.2.smithi204.stdout:3005: done (0 left) 2024-09-16T05:21:56.820 INFO:tasks.rados.rados.2.smithi204.stdout:3006: copy_from oid 34 from oid 46 current snap is 321 2024-09-16T05:21:56.820 INFO:tasks.rados.rados.2.smithi204.stdout:3007: delete oid 8 current snap is 321 2024-09-16T05:21:56.833 INFO:tasks.rados.rados.2.smithi204.stdout:3006: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:21:56.836 INFO:tasks.rados.rados.2.smithi204.stdout:3006: finishing copy_from to smithi20457400-34 2024-09-16T05:21:56.836 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1014 (ObjNum 969 snap 287 seq_num 969) dirty exists 2024-09-16T05:21:56.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:56.867 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:56.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:56.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:21:56.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:56.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:56.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:21:56.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:21:57.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:57.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:57.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:57.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:57.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:56 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:21:57.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:56 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:21:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1077437520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:56 smithi196 ceph-mon[30802]: osdmap e767: 12 total, 12 up, 12 in 2024-09-16T05:21:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:57.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:57.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:57.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:21:57.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:56 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:21:57.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:56 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:21:57.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1077437520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:57.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:56 smithi032 ceph-mon[27388]: osdmap e767: 12 total, 12 up, 12 in 2024-09-16T05:21:57.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:57.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:57.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:56 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:21:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:56 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:21:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1077437520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:21:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:56 smithi204 ceph-mon[30933]: osdmap e767: 12 total, 12 up, 12 in 2024-09-16T05:21:57.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:57.491 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:57.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:57.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:57.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:57.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:57.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:57.741 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:57.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:57.795 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:57.825 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:21:57.826 INFO:tasks.rados.rados.2.smithi204.stdout:3006: done (1 left) 2024-09-16T05:21:57.826 INFO:tasks.rados.rados.2.smithi204.stdout:3007: done (0 left) 2024-09-16T05:21:57.826 INFO:tasks.rados.rados.2.smithi204.stdout:3008: write oid 16 current snap is 321 2024-09-16T05:21:57.826 INFO:tasks.rados.rados.2.smithi204.stdout:3008: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2024-09-16T05:21:57.833 INFO:tasks.rados.rados.2.smithi204.stdout:3008: writing smithi20457400-16 from 548192 to 1267850 tid 1 2024-09-16T05:21:57.838 INFO:tasks.rados.rados.2.smithi204.stdout:3008: writing smithi20457400-16 from 1678279 to 2095979 tid 2 2024-09-16T05:21:57.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:57.847 INFO:tasks.rados.rados.2.smithi204.stdout:3008: writing smithi20457400-16 from 2516445 to 3297220 tid 3 2024-09-16T05:21:57.847 INFO:tasks.rados.rados.2.smithi204.stdout:3009: snap_remove snap 317 2024-09-16T05:21:57.867 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:57.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:57.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:57.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:57.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:58.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:57 smithi032 ceph-mon[27388]: pgmap v1311: 127 pgs: 1 remapped+peering, 2 peering, 124 active+clean; 313 MiB data, 984 MiB used, 982 GiB / 983 GiB avail; 22 MiB/s rd, 7.4 MiB/s wr, 35 op/s; 1.1 MiB/s, 5 objects/s recovering 2024-09-16T05:21:58.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:57 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:21:58.068 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:57 smithi032 ceph-mon[27388]: osdmap e768: 12 total, 12 up, 12 in 2024-09-16T05:21:58.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:58.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:58.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:58.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:57 smithi196 ceph-mon[30802]: pgmap v1311: 127 pgs: 1 remapped+peering, 2 peering, 124 active+clean; 313 MiB data, 984 MiB used, 982 GiB / 983 GiB avail; 22 MiB/s rd, 7.4 MiB/s wr, 35 op/s; 1.1 MiB/s, 5 objects/s recovering 2024-09-16T05:21:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:57 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:21:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:57 smithi196 ceph-mon[30802]: osdmap e768: 12 total, 12 up, 12 in 2024-09-16T05:21:58.252 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:58.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:58.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:21:58.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:58.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:58.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:21:58.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:57 smithi204 ceph-mon[30933]: pgmap v1311: 127 pgs: 1 remapped+peering, 2 peering, 124 active+clean; 313 MiB data, 984 MiB used, 982 GiB / 983 GiB avail; 22 MiB/s rd, 7.4 MiB/s wr, 35 op/s; 1.1 MiB/s, 5 objects/s recovering 2024-09-16T05:21:58.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:57 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:21:58.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:57 smithi204 ceph-mon[30933]: osdmap e768: 12 total, 12 up, 12 in 2024-09-16T05:21:58.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:58.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:58.605 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:21:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:58.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:58.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:21:58.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:58.756 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:58.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:58.823 INFO:tasks.rados.rados.2.smithi204.stdout:3009: done (1 left) 2024-09-16T05:21:58.823 INFO:tasks.rados.rados.2.smithi204.stdout:3010: snap_create 2024-09-16T05:21:58.824 INFO:tasks.rados.rados.2.smithi204.stdout:3008: finishing write tid 1 to smithi20457400-16 2024-09-16T05:21:58.824 INFO:tasks.rados.rados.2.smithi204.stdout:3008: finishing write tid 2 to smithi20457400-16 2024-09-16T05:21:58.824 INFO:tasks.rados.rados.2.smithi204.stdout:3008: finishing write tid 3 to smithi20457400-16 2024-09-16T05:21:58.824 INFO:tasks.rados.rados.2.smithi204.stdout:3008: finishing write tid 4 to smithi20457400-16 2024-09-16T05:21:58.824 INFO:tasks.rados.rados.2.smithi204.stdout:3008: finishing write tid 5 to smithi20457400-16 2024-09-16T05:21:58.824 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1036 (ObjNum 1056 snap 321 seq_num 1056) dirty exists 2024-09-16T05:21:58.824 INFO:tasks.rados.rados.2.smithi204.stdout:3008: left oid 16 (ObjNum 1056 snap 321 seq_num 1056) 2024-09-16T05:21:58.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:21:58.881 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:58.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:21:58.968 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:21:58.968 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:21:58.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:21:59.010 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:59.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:21:59.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:59.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:59.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:59.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:21:59.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:59.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:21:59.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:21:59.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:59.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:21:59.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:21:59.516 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:21:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:59.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:21:59.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:59.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:59.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:59.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:21:59.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:21:59.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:21:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:59.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:21:59.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:21:59.821 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:21:59.825 INFO:tasks.rados.rados.2.smithi204.stdout:3008: done (1 left) 2024-09-16T05:21:59.825 INFO:tasks.rados.rados.2.smithi204.stdout:3010: done (0 left) 2024-09-16T05:21:59.825 INFO:tasks.rados.rados.2.smithi204.stdout:3011: snap_create 2024-09-16T05:21:59.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:21:59.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:21:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:59.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:21:59.959 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:21:59.960 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:21:59.960 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-16T05:21:59.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-16T05:21:59.984 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:59 smithi032 ceph-mon[27388]: pgmap v1314: 127 pgs: 4 peering, 123 active+clean; 326 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.1 MiB/s wr, 26 op/s; 9.6 MiB/s, 35 objects/s recovering 2024-09-16T05:21:59.984 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:59 smithi032 ceph-mon[27388]: osdmap e769: 12 total, 12 up, 12 in 2024-09-16T05:21:59.985 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3038238781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:21:59.985 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:21:59 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:21:59.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:00.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:00.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:00.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:00.148 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:00.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:00.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:59 smithi204 ceph-mon[30933]: pgmap v1314: 127 pgs: 4 peering, 123 active+clean; 326 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.1 MiB/s wr, 26 op/s; 9.6 MiB/s, 35 objects/s recovering 2024-09-16T05:22:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:59 smithi204 ceph-mon[30933]: osdmap e769: 12 total, 12 up, 12 in 2024-09-16T05:22:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3038238781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:22:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:21:59 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:22:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:59 smithi196 ceph-mon[30802]: pgmap v1314: 127 pgs: 4 peering, 123 active+clean; 326 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 7.1 MiB/s wr, 26 op/s; 9.6 MiB/s, 35 objects/s recovering 2024-09-16T05:22:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:59 smithi196 ceph-mon[30802]: osdmap e769: 12 total, 12 up, 12 in 2024-09-16T05:22:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3038238781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:22:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:21:59 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-16T05:22:00.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:00.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:00.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:00.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:00.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:00.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:00.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:00.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:00.531 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:00.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:00.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:00.656 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:00.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:00.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:00.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:00.787 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:00.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:00.822 INFO:tasks.rados.rados.2.smithi204.stdout:3011: done (0 left) 2024-09-16T05:22:00.822 INFO:tasks.rados.rados.2.smithi204.stdout:3012: read oid 43 snap -1 2024-09-16T05:22:00.822 INFO:tasks.rados.rados.2.smithi204.stdout:3012: expect (ObjNum 1031 snap 316 seq_num 1031) 2024-09-16T05:22:00.822 INFO:tasks.rados.rados.2.smithi204.stdout:3013: delete oid 33 current snap is 323 2024-09-16T05:22:00.824 INFO:tasks.rados.rados.2.smithi204.stdout:3013: done (1 left) 2024-09-16T05:22:00.824 INFO:tasks.rados.rados.2.smithi204.stdout:3014: delete oid 36 current snap is 323 2024-09-16T05:22:00.829 INFO:tasks.rados.rados.2.smithi204.stdout:3012: expect (ObjNum 1031 snap 316 seq_num 1031) 2024-09-16T05:22:00.836 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3038238781' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:00.836 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:00 smithi032 ceph-mon[27388]: osdmap e770: 12 total, 12 up, 12 in 2024-09-16T05:22:00.836 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:00 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:22:00.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:00.912 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:00.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:01.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:01.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:01.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:01.096 INFO:tasks.rados.rados.2.smithi204.stdout:3012: done (1 left) 2024-09-16T05:22:01.097 INFO:tasks.rados.rados.2.smithi204.stdout:3014: done (0 left) 2024-09-16T05:22:01.097 INFO:tasks.rados.rados.2.smithi204.stdout:3015: read oid 4 snap -1 2024-09-16T05:22:01.097 INFO:tasks.rados.rados.2.smithi204.stdout:3015: expect (ObjNum 1031 snap 316 seq_num 1031) 2024-09-16T05:22:01.097 INFO:tasks.rados.rados.2.smithi204.stdout:3016: copy_from oid 11 from oid 48 current snap is 323 2024-09-16T05:22:01.097 INFO:tasks.rados.rados.2.smithi204.stdout:3017: delete oid 8 current snap is 323 2024-09-16T05:22:01.098 INFO:tasks.rados.rados.2.smithi204.stdout:3017: done (2 left) 2024-09-16T05:22:01.098 INFO:tasks.rados.rados.2.smithi204.stdout:3018: write oid 36 current snap is 323 2024-09-16T05:22:01.099 INFO:tasks.rados.rados.2.smithi204.stdout:3018: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-09-16T05:22:01.106 INFO:tasks.rados.rados.2.smithi204.stdout:3018: writing smithi20457400-36 from 615420 to 1352427 tid 1 2024-09-16T05:22:01.111 INFO:tasks.rados.rados.2.smithi204.stdout:3018: writing smithi20457400-36 from 1912207 to 2337245 tid 2 2024-09-16T05:22:01.118 INFO:tasks.rados.rados.2.smithi204.stdout:3018: writing smithi20457400-36 from 3026137 to 3568353 tid 3 2024-09-16T05:22:01.119 INFO:tasks.rados.rados.2.smithi204.stdout:3019: write (excl) oid 25 current snap is 323 2024-09-16T05:22:01.119 INFO:tasks.rados.rados.2.smithi204.stdout:3019: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2024-09-16T05:22:01.124 INFO:tasks.rados.rados.2.smithi204.stdout:3019: writing smithi20457400-25 from 632227 to 1044483 tid 1 2024-09-16T05:22:01.130 INFO:tasks.rados.rados.2.smithi204.stdout:3019: writing smithi20457400-25 from 1570689 to 2139385 tid 2 2024-09-16T05:22:01.137 INFO:tasks.rados.rados.2.smithi204.stdout:3019: writing smithi20457400-25 from 2653560 to 3407048 tid 3 2024-09-16T05:22:01.137 INFO:tasks.rados.rados.2.smithi204.stdout:3020: delete oid 45 current snap is 323 2024-09-16T05:22:01.137 INFO:tasks.rados.rados.2.smithi204.stdout:3015: expect (ObjNum 1031 snap 316 seq_num 1031) 2024-09-16T05:22:01.140 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:01.161 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:01.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:00 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:22:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:00 smithi032 ceph-mon[27388]: 1.17 scrub starts 2024-09-16T05:22:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:00 smithi032 ceph-mon[27388]: 1.17 scrub ok 2024-09-16T05:22:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:00 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:01.165 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:00 smithi032 ceph-mon[27388]: osdmap e771: 12 total, 12 up, 12 in 2024-09-16T05:22:01.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3038238781' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:01.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:00 smithi204 ceph-mon[30933]: osdmap e770: 12 total, 12 up, 12 in 2024-09-16T05:22:01.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:00 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:22:01.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:00 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:22:01.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:00 smithi204 ceph-mon[30933]: 1.17 scrub starts 2024-09-16T05:22:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:00 smithi204 ceph-mon[30933]: 1.17 scrub ok 2024-09-16T05:22:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:00 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:00 smithi204 ceph-mon[30933]: osdmap e771: 12 total, 12 up, 12 in 2024-09-16T05:22:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3038238781' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:00 smithi196 ceph-mon[30802]: osdmap e770: 12 total, 12 up, 12 in 2024-09-16T05:22:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:00 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:22:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:00 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:22:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:00 smithi196 ceph-mon[30802]: 1.17 scrub starts 2024-09-16T05:22:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:00 smithi196 ceph-mon[30802]: 1.17 scrub ok 2024-09-16T05:22:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:00 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:00 smithi196 ceph-mon[30802]: osdmap e771: 12 total, 12 up, 12 in 2024-09-16T05:22:01.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:01.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:01.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:01.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:01.405 INFO:tasks.rados.rados.2.smithi204.stdout:3016: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:22:01.405 INFO:tasks.rados.rados.2.smithi204.stdout:3018: finishing write tid 1 to smithi20457400-36 2024-09-16T05:22:01.405 INFO:tasks.rados.rados.2.smithi204.stdout:3016: finishing copy_from to smithi20457400-11 2024-09-16T05:22:01.405 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 779 (ObjNum 966 snap 285 seq_num 966) dirty exists 2024-09-16T05:22:01.405 INFO:tasks.rados.rados.2.smithi204.stdout:3018: finishing write tid 2 to smithi20457400-36 2024-09-16T05:22:01.405 INFO:tasks.rados.rados.2.smithi204.stdout:3018: finishing write tid 3 to smithi20457400-36 2024-09-16T05:22:01.406 INFO:tasks.rados.rados.2.smithi204.stdout:3015: done (4 left) 2024-09-16T05:22:01.406 INFO:tasks.rados.rados.2.smithi204.stdout:3016: done (3 left) 2024-09-16T05:22:01.406 INFO:tasks.rados.rados.2.smithi204.stdout:3020: done (2 left) 2024-09-16T05:22:01.406 INFO:tasks.rados.rados.2.smithi204.stdout:3021: copy_from oid 41 from oid 26 current snap is 323 2024-09-16T05:22:01.406 INFO:tasks.rados.rados.2.smithi204.stdout:3018: finishing write tid 4 to smithi20457400-36 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:3018: finishing write tid 5 to smithi20457400-36 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 773 (ObjNum 1060 snap 323 seq_num 1060) dirty exists 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:3018: left oid 36 (ObjNum 1060 snap 323 seq_num 1060) 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:3019: finishing write tid 1 to smithi20457400-25 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:3019: finishing write tid 2 to smithi20457400-25 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:3019: finishing write tid 3 to smithi20457400-25 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:3019: finishing write tid 4 to smithi20457400-25 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:3019: finishing write tid 5 to smithi20457400-25 2024-09-16T05:22:01.407 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 745 (ObjNum 1061 snap 323 seq_num 1061) dirty exists 2024-09-16T05:22:01.408 INFO:tasks.rados.rados.2.smithi204.stdout:3019: left oid 25 (ObjNum 1061 snap 323 seq_num 1061) 2024-09-16T05:22:01.408 INFO:tasks.rados.rados.2.smithi204.stdout:3018: done (2 left) 2024-09-16T05:22:01.408 INFO:tasks.rados.rados.2.smithi204.stdout:3019: done (1 left) 2024-09-16T05:22:01.408 INFO:tasks.rados.rados.2.smithi204.stdout:3022: copy_from oid 9 from oid 1 current snap is 323 2024-09-16T05:22:01.408 INFO:tasks.rados.rados.2.smithi204.stdout:3023: snap_create 2024-09-16T05:22:01.408 INFO:tasks.rados.rados.2.smithi204.stdout:3022: finishing copy_from to smithi20457400-9 2024-09-16T05:22:01.408 INFO:tasks.rados.rados.2.smithi204.stdout:3022: got expected ENOENT (src dne) 2024-09-16T05:22:01.409 INFO:tasks.rados.rados.2.smithi204.stdout:3022: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:22:01.410 INFO:tasks.rados.rados.2.smithi204.stdout:3021: finishing copy_from to smithi20457400-41 2024-09-16T05:22:01.410 INFO:tasks.rados.rados.2.smithi204.stdout:3021: got expected ENOENT (src dne) 2024-09-16T05:22:01.411 INFO:tasks.rados.rados.2.smithi204.stdout:3021: finishing copy_from racing read to smithi20457400-41 2024-09-16T05:22:01.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:01.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:01.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:01.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:01.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:01.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:01.670 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:01.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:01.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:01.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:01.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:01.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:01.846 INFO:tasks.rados.rados.2.smithi204.stdout:3021: done (2 left) 2024-09-16T05:22:01.846 INFO:tasks.rados.rados.2.smithi204.stdout:3022: done (1 left) 2024-09-16T05:22:01.846 INFO:tasks.rados.rados.2.smithi204.stdout:3023: done (0 left) 2024-09-16T05:22:01.846 INFO:tasks.rados.rados.2.smithi204.stdout:3024: read oid 33 snap -1 2024-09-16T05:22:01.847 INFO:tasks.rados.rados.2.smithi204.stdout:3024: expect deleted 2024-09-16T05:22:01.847 INFO:tasks.rados.rados.2.smithi204.stdout:3025: copy_from oid 45 from oid 44 current snap is 324 2024-09-16T05:22:01.847 INFO:tasks.rados.rados.2.smithi204.stdout:3026: snap_remove snap 316 2024-09-16T05:22:01.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:01.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:01.922 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:01.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:02.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:02.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-16T05:22:02.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:02.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:02.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:01 smithi032 ceph-mon[27388]: pgmap v1317: 127 pgs: 2 peering, 125 active+clean; 328 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.8 MiB/s wr, 7 op/s; 11 MiB/s, 40 objects/s recovering 2024-09-16T05:22:02.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:01 smithi032 ceph-mon[27388]: 2.35 scrub starts 2024-09-16T05:22:02.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:01 smithi032 ceph-mon[27388]: 2.53 scrub starts 2024-09-16T05:22:02.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:01 smithi032 ceph-mon[27388]: 2.53 scrub ok 2024-09-16T05:22:02.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:01 smithi032 ceph-mon[27388]: 2.54 scrub starts 2024-09-16T05:22:02.061 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:01 smithi032 ceph-mon[27388]: 2.54 scrub ok 2024-09-16T05:22:02.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:01 smithi204 ceph-mon[30933]: pgmap v1317: 127 pgs: 2 peering, 125 active+clean; 328 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.8 MiB/s wr, 7 op/s; 11 MiB/s, 40 objects/s recovering 2024-09-16T05:22:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:01 smithi204 ceph-mon[30933]: 2.35 scrub starts 2024-09-16T05:22:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:01 smithi204 ceph-mon[30933]: 2.53 scrub starts 2024-09-16T05:22:02.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:01 smithi204 ceph-mon[30933]: 2.53 scrub ok 2024-09-16T05:22:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:01 smithi204 ceph-mon[30933]: 2.54 scrub starts 2024-09-16T05:22:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:01 smithi204 ceph-mon[30933]: 2.54 scrub ok 2024-09-16T05:22:02.174 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:02.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:02.234 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:02.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:01 smithi196 ceph-mon[30802]: pgmap v1317: 127 pgs: 2 peering, 125 active+clean; 328 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.8 MiB/s wr, 7 op/s; 11 MiB/s, 40 objects/s recovering 2024-09-16T05:22:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:01 smithi196 ceph-mon[30802]: 2.35 scrub starts 2024-09-16T05:22:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:01 smithi196 ceph-mon[30802]: 2.53 scrub starts 2024-09-16T05:22:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:01 smithi196 ceph-mon[30802]: 2.53 scrub ok 2024-09-16T05:22:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:01 smithi196 ceph-mon[30802]: 2.54 scrub starts 2024-09-16T05:22:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:01 smithi196 ceph-mon[30802]: 2.54 scrub ok 2024-09-16T05:22:02.260 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:02.260 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:02.275 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:02.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:02.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:02.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:02.423 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:02.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:02.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:02.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:02.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:02.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:22:02.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:02.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:02.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:02.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:02.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:02.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:02.857 INFO:tasks.rados.rados.2.smithi204.stdout:3026: done (2 left) 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:3027: rollback oid 17 current snap is 324 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 17 to 296 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:3025: finishing copy_from racing read to smithi20457400-45 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:3025: finishing copy_from to smithi20457400-45 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 731 (ObjNum 995 snap 292 seq_num 995) dirty exists 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:3024: done (2 left) 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:3025: done (1 left) 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:3028: read oid 42 snap -1 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:3028: expect deleted 2024-09-16T05:22:02.858 INFO:tasks.rados.rados.2.smithi204.stdout:3029: delete oid 41 current snap is 324 2024-09-16T05:22:02.860 INFO:tasks.rados.rados.2.smithi204.stdout:3029: done (2 left) 2024-09-16T05:22:02.860 INFO:tasks.rados.rados.2.smithi204.stdout:3030: write oid 2 current snap is 324 2024-09-16T05:22:02.860 INFO:tasks.rados.rados.2.smithi204.stdout:3030: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2024-09-16T05:22:02.868 INFO:tasks.rados.rados.2.smithi204.stdout:3030: writing smithi20457400-2 from 682648 to 1320651 tid 1 2024-09-16T05:22:02.874 INFO:tasks.rados.rados.2.smithi204.stdout:3030: writing smithi20457400-2 from 2029782 to 2462158 tid 2 2024-09-16T05:22:02.881 INFO:tasks.rados.rados.2.smithi204.stdout:3030: writing smithi20457400-2 from 3019476 to 3606780 tid 3 2024-09-16T05:22:02.881 INFO:tasks.rados.rados.2.smithi204.stdout:3031: write (excl) oid 7 current snap is 324 2024-09-16T05:22:02.881 INFO:tasks.rados.rados.2.smithi204.stdout:3031: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2024-09-16T05:22:02.887 INFO:tasks.rados.rados.2.smithi204.stdout:3031: writing smithi20457400-7 from 699455 to 1129060 tid 1 2024-09-16T05:22:02.893 INFO:tasks.rados.rados.2.smithi204.stdout:3031: writing smithi20457400-7 from 1804617 to 2380651 tid 2 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:3031: writing smithi20457400-7 from 3163252 to 3678181 tid 3 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:3032: snap_create 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:3027: finishing rollback tid 0 to smithi20457400-17 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:3027: finishing rollback tid 1 to smithi20457400-17 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:3027: finishing rollback tid 2 to smithi20457400-17 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1121 (ObjNum 961 snap 284 seq_num 961) dirty exists 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:3030: finishing write tid 1 to smithi20457400-2 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:3030: finishing write tid 2 to smithi20457400-2 2024-09-16T05:22:02.901 INFO:tasks.rados.rados.2.smithi204.stdout:3030: finishing write tid 3 to smithi20457400-2 2024-09-16T05:22:02.902 INFO:tasks.rados.rados.2.smithi204.stdout:3030: finishing write tid 4 to smithi20457400-2 2024-09-16T05:22:02.902 INFO:tasks.rados.rados.2.smithi204.stdout:3030: finishing write tid 5 to smithi20457400-2 2024-09-16T05:22:02.902 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 533 (ObjNum 1064 snap 324 seq_num 1064) dirty exists 2024-09-16T05:22:02.902 INFO:tasks.rados.rados.2.smithi204.stdout:3030: left oid 2 (ObjNum 1064 snap 324 seq_num 1064) 2024-09-16T05:22:02.902 INFO:tasks.rados.rados.2.smithi204.stdout:3031: finishing write tid 1 to smithi20457400-7 2024-09-16T05:22:02.902 INFO:tasks.rados.rados.2.smithi204.stdout:3031: finishing write tid 2 to smithi20457400-7 2024-09-16T05:22:02.903 INFO:tasks.rados.rados.2.smithi204.stdout:3031: finishing write tid 3 to smithi20457400-7 2024-09-16T05:22:02.903 INFO:tasks.rados.rados.2.smithi204.stdout:3031: finishing write tid 4 to smithi20457400-7 2024-09-16T05:22:02.903 INFO:tasks.rados.rados.2.smithi204.stdout:3031: finishing write tid 5 to smithi20457400-7 2024-09-16T05:22:02.903 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1135 (ObjNum 1065 snap 324 seq_num 1065) dirty exists 2024-09-16T05:22:02.903 INFO:tasks.rados.rados.2.smithi204.stdout:3031: left oid 7 (ObjNum 1065 snap 324 seq_num 1065) 2024-09-16T05:22:02.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:02.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:02 smithi032 ceph-mon[27388]: 2.35 scrub ok 2024-09-16T05:22:02.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:02 smithi032 ceph-mon[27388]: osdmap e772: 12 total, 12 up, 12 in 2024-09-16T05:22:02.932 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:02.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:03.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:03.057 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:03.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:03.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:03.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:02 smithi204 ceph-mon[30933]: 2.35 scrub ok 2024-09-16T05:22:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:02 smithi204 ceph-mon[30933]: osdmap e772: 12 total, 12 up, 12 in 2024-09-16T05:22:03.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:03.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:03.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:22:03.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:02 smithi196 ceph-mon[30802]: 2.35 scrub ok 2024-09-16T05:22:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:02 smithi196 ceph-mon[30802]: osdmap e772: 12 total, 12 up, 12 in 2024-09-16T05:22:03.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:03.306 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:03.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:03.328 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:03.328 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:03.328 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:03.328 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:03.328 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:03.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:03.330 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:03.330 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:03.330 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:03.330 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:03.330 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:03.331 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:03.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:03.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:03.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:03.527 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:03.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:03.562 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:03.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:03.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:03.686 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:03.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:03.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:03.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:03.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:03.857 INFO:tasks.rados.rados.2.smithi204.stdout:3027: done (4 left) 2024-09-16T05:22:03.857 INFO:tasks.rados.rados.2.smithi204.stdout:3028: done (3 left) 2024-09-16T05:22:03.857 INFO:tasks.rados.rados.2.smithi204.stdout:3030: done (2 left) 2024-09-16T05:22:03.857 INFO:tasks.rados.rados.2.smithi204.stdout:3031: done (1 left) 2024-09-16T05:22:03.857 INFO:tasks.rados.rados.2.smithi204.stdout:3032: done (0 left) 2024-09-16T05:22:03.857 INFO:tasks.rados.rados.2.smithi204.stdout:3033: write oid 32 current snap is 325 2024-09-16T05:22:03.857 INFO:tasks.rados.rados.2.smithi204.stdout:3033: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2024-09-16T05:22:03.863 INFO:tasks.rados.rados.2.smithi204.stdout:3033: writing smithi20457400-32 from 716262 to 1221116 tid 1 2024-09-16T05:22:03.873 INFO:tasks.rados.rados.2.smithi204.stdout:3033: writing smithi20457400-32 from 1863099 to 2582791 tid 2 2024-09-16T05:22:03.878 INFO:tasks.rados.rados.2.smithi204.stdout:3033: writing smithi20457400-32 from 3074322 to 3800523 tid 3 2024-09-16T05:22:03.878 INFO:tasks.rados.rados.2.smithi204.stdout:3034: rollback oid 43 current snap is 325 2024-09-16T05:22:03.878 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 43 to 296 2024-09-16T05:22:03.878 INFO:tasks.rados.rados.2.smithi204.stdout:3033: finishing write tid 1 to smithi20457400-32 2024-09-16T05:22:03.878 INFO:tasks.rados.rados.2.smithi204.stdout:3033: finishing write tid 2 to smithi20457400-32 2024-09-16T05:22:03.878 INFO:tasks.rados.rados.2.smithi204.stdout:3035: rollback oid 16 current snap is 325 2024-09-16T05:22:03.878 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 16 to 319 2024-09-16T05:22:03.879 INFO:tasks.rados.rados.2.smithi204.stdout:3036: snap_remove snap 314 2024-09-16T05:22:03.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: pgmap v1320: 127 pgs: 2 peering, 125 active+clean; 332 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.7 MiB/s wr, 2 op/s; 2.5 MiB/s, 10 objects/s recovering 2024-09-16T05:22:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:22:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:22:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:22:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:22:03.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: osdmap e773: 12 total, 12 up, 12 in 2024-09-16T05:22:03.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1045205145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:03.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: 2.4c scrub starts 2024-09-16T05:22:03.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:03 smithi032 ceph-mon[27388]: 2.4c scrub ok 2024-09-16T05:22:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:03.941 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:03.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:04.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:04.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:04.123 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:04.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: pgmap v1320: 127 pgs: 2 peering, 125 active+clean; 332 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.7 MiB/s wr, 2 op/s; 2.5 MiB/s, 10 objects/s recovering 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: osdmap e773: 12 total, 12 up, 12 in 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1045205145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: 2.4c scrub starts 2024-09-16T05:22:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:03 smithi204 ceph-mon[30933]: 2.4c scrub ok 2024-09-16T05:22:04.190 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:04.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: pgmap v1320: 127 pgs: 2 peering, 125 active+clean; 332 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.7 MiB/s wr, 2 op/s; 2.5 MiB/s, 10 objects/s recovering 2024-09-16T05:22:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:22:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:22:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:22:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:22:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: osdmap e773: 12 total, 12 up, 12 in 2024-09-16T05:22:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1045205145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: 2.4c scrub starts 2024-09-16T05:22:04.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:03 smithi196 ceph-mon[30802]: 2.4c scrub ok 2024-09-16T05:22:04.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:04.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:04.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:04.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:04.449 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:04.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:04.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:04.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:04.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:04.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:04.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:04.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:04.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:04.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:04.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:04.863 INFO:tasks.rados.rados.2.smithi204.stdout:3036: done (3 left) 2024-09-16T05:22:04.863 INFO:tasks.rados.rados.2.smithi204.stdout:3037: delete oid 25 current snap is 325 2024-09-16T05:22:04.863 INFO:tasks.rados.rados.2.smithi204.stdout:3033: finishing write tid 3 to smithi20457400-32 2024-09-16T05:22:04.863 INFO:tasks.rados.rados.2.smithi204.stdout:3033: finishing write tid 4 to smithi20457400-32 2024-09-16T05:22:04.863 INFO:tasks.rados.rados.2.smithi204.stdout:3033: finishing write tid 5 to smithi20457400-32 2024-09-16T05:22:04.863 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1164 (ObjNum 1066 snap 325 seq_num 1066) dirty exists 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:3033: left oid 32 (ObjNum 1066 snap 325 seq_num 1066) 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:3034: finishing rollback tid 0 to smithi20457400-43 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:3034: finishing rollback tid 1 to smithi20457400-43 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:3035: finishing rollback tid 0 to smithi20457400-16 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:3035: finishing rollback tid 1 to smithi20457400-16 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:3034: finishing rollback tid 2 to smithi20457400-43 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1044 (ObjNum 950 snap 280 seq_num 950) dirty exists 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:3035: finishing rollback tid 2 to smithi20457400-16 2024-09-16T05:22:04.864 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1039 (ObjNum 1005 snap 303 seq_num 1005) dirty exists 2024-09-16T05:22:04.869 INFO:tasks.rados.rados.2.smithi204.stdout:3033: done (3 left) 2024-09-16T05:22:04.869 INFO:tasks.rados.rados.2.smithi204.stdout:3034: done (2 left) 2024-09-16T05:22:04.869 INFO:tasks.rados.rados.2.smithi204.stdout:3035: done (1 left) 2024-09-16T05:22:04.869 INFO:tasks.rados.rados.2.smithi204.stdout:3037: done (0 left) 2024-09-16T05:22:04.869 INFO:tasks.rados.rados.2.smithi204.stdout:3038: rollback oid 27 current snap is 325 2024-09-16T05:22:04.869 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 27 to 321 2024-09-16T05:22:04.869 INFO:tasks.rados.rados.2.smithi204.stdout:3039: write (excl) oid 16 current snap is 325 2024-09-16T05:22:04.869 INFO:tasks.rados.rados.2.smithi204.stdout:3039: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2024-09-16T05:22:04.877 INFO:tasks.rados.rados.2.smithi204.stdout:3039: writing smithi20457400-16 from 749876 to 1405228 tid 1 2024-09-16T05:22:04.883 INFO:tasks.rados.rados.2.smithi204.stdout:3039: writing smithi20457400-16 from 1863710 to 2587071 tid 2 2024-09-16T05:22:04.890 INFO:tasks.rados.rados.2.smithi204.stdout:3039: writing smithi20457400-16 from 3012815 to 3761560 tid 3 2024-09-16T05:22:04.890 INFO:tasks.rados.rados.2.smithi204.stdout:3040: read oid 39 snap -1 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:3040: expect (ObjNum 987 snap 292 seq_num 987) 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:3041: rollback oid 24 current snap is 325 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 276 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:3038: finishing rollback tid 0 to smithi20457400-27 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:3038: finishing rollback tid 1 to smithi20457400-27 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:3038: finishing rollback tid 2 to smithi20457400-27 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 331 (ObjNum 847 snap 246 seq_num 847) dirty exists 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:3038: done (3 left) 2024-09-16T05:22:04.891 INFO:tasks.rados.rados.2.smithi204.stdout:3042: snap_remove snap 296 2024-09-16T05:22:04.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:04 smithi032 ceph-mon[27388]: 2.43 scrub starts 2024-09-16T05:22:04.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:04 smithi032 ceph-mon[27388]: 2.43 scrub ok 2024-09-16T05:22:04.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:04 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:22:04.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:04 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:22:04.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:04 smithi032 ceph-mon[27388]: osdmap e774: 12 total, 12 up, 12 in 2024-09-16T05:22:04.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:04.955 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:04.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:05.008 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:22:05.008 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:22:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:22:05.009 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:22:05.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:05.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:05.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:05.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:04 smithi204 ceph-mon[30933]: 2.43 scrub starts 2024-09-16T05:22:05.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:04 smithi204 ceph-mon[30933]: 2.43 scrub ok 2024-09-16T05:22:05.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:04 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:22:05.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:04 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:22:05.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:04 smithi204 ceph-mon[30933]: osdmap e774: 12 total, 12 up, 12 in 2024-09-16T05:22:05.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:05.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:05.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:04 smithi196 ceph-mon[30802]: 2.43 scrub starts 2024-09-16T05:22:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:04 smithi196 ceph-mon[30802]: 2.43 scrub ok 2024-09-16T05:22:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:04 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:22:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:04 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:22:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:04 smithi196 ceph-mon[30802]: osdmap e774: 12 total, 12 up, 12 in 2024-09-16T05:22:05.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:05.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:05.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:05.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:05.461 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:05.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:05.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:05.589 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:05.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:05.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:05.717 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:05.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:05.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:22:05.819 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:22:05.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:05.823 INFO:tasks.rados.rados.2.smithi204.stdout:3042: done (3 left) 2024-09-16T05:22:05.823 INFO:tasks.rados.rados.2.smithi204.stdout:3043: rollback oid 26 current snap is 325 2024-09-16T05:22:05.824 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 26 to 298 2024-09-16T05:22:05.824 INFO:tasks.rados.rados.2.smithi204.stdout:3039: finishing write tid 1 to smithi20457400-16 2024-09-16T05:22:05.824 INFO:tasks.rados.rados.2.smithi204.stdout:3039: finishing write tid 2 to smithi20457400-16 2024-09-16T05:22:05.824 INFO:tasks.rados.rados.2.smithi204.stdout:3040: expect (ObjNum 987 snap 292 seq_num 987) 2024-09-16T05:22:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:05.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:05.929 INFO:tasks.rados.rados.2.smithi204.stdout:3041: finishing rollback tid 0 to smithi20457400-24 2024-09-16T05:22:05.930 INFO:tasks.rados.rados.2.smithi204.stdout:3041: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:22:05.930 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 785 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-16T05:22:05.930 INFO:tasks.rados.rados.2.smithi204.stdout:3039: finishing write tid 3 to smithi20457400-16 2024-09-16T05:22:05.930 INFO:tasks.rados.rados.2.smithi204.stdout:3039: finishing write tid 4 to smithi20457400-16 2024-09-16T05:22:05.930 INFO:tasks.rados.rados.2.smithi204.stdout:3039: finishing write tid 5 to smithi20457400-16 2024-09-16T05:22:05.931 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1043 (ObjNum 1068 snap 325 seq_num 1068) dirty exists 2024-09-16T05:22:05.931 INFO:tasks.rados.rados.2.smithi204.stdout:3039: left oid 16 (ObjNum 1068 snap 325 seq_num 1068) 2024-09-16T05:22:05.931 INFO:tasks.rados.rados.2.smithi204.stdout:3039: done (3 left) 2024-09-16T05:22:05.931 INFO:tasks.rados.rados.2.smithi204.stdout:3040: done (2 left) 2024-09-16T05:22:05.931 INFO:tasks.rados.rados.2.smithi204.stdout:3041: done (1 left) 2024-09-16T05:22:05.931 INFO:tasks.rados.rados.2.smithi204.stdout:3044: snap_create 2024-09-16T05:22:05.932 INFO:tasks.rados.rados.2.smithi204.stdout:3043: finishing rollback tid 1 to smithi20457400-26 2024-09-16T05:22:05.932 INFO:tasks.rados.rados.2.smithi204.stdout:3043: finishing rollback tid 2 to smithi20457400-26 2024-09-16T05:22:05.932 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1343 (ObjNum 985 snap 289 seq_num 985) dirty exists 2024-09-16T05:22:05.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:05.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:05.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: pgmap v1323: 127 pgs: 127 active+clean; 337 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: osdmap e775: 12 total, 12 up, 12 in 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/140110875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: 1.16 scrub starts 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: 1.16 scrub ok 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:06.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:22:06.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/140110875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:06.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:05 smithi032 ceph-mon[27388]: osdmap e776: 12 total, 12 up, 12 in 2024-09-16T05:22:06.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:22:06.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:06.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:06.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: pgmap v1323: 127 pgs: 127 active+clean; 337 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-16T05:22:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:22:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:22:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: osdmap e775: 12 total, 12 up, 12 in 2024-09-16T05:22:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/140110875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: 1.16 scrub starts 2024-09-16T05:22:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: 1.16 scrub ok 2024-09-16T05:22:06.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:22:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/140110875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:06.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:05 smithi204 ceph-mon[30933]: osdmap e776: 12 total, 12 up, 12 in 2024-09-16T05:22:06.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:06.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:06.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:06.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: pgmap v1323: 127 pgs: 127 active+clean; 337 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: osdmap e775: 12 total, 12 up, 12 in 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/140110875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: 1.16 scrub starts 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: 1.16 scrub ok 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/140110875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:05 smithi196 ceph-mon[30802]: osdmap e776: 12 total, 12 up, 12 in 2024-09-16T05:22:06.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:06.361 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:06.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:06.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:06.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:06.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:06.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:06.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:06.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:06.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:06.739 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:06.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:06.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:06.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:06.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:06.878 INFO:tasks.rados.rados.2.smithi204.stdout:3043: done (1 left) 2024-09-16T05:22:06.878 INFO:tasks.rados.rados.2.smithi204.stdout:3044: done (0 left) 2024-09-16T05:22:06.878 INFO:tasks.rados.rados.2.smithi204.stdout:3045: copy_from oid 9 from oid 8 current snap is 326 2024-09-16T05:22:06.878 INFO:tasks.rados.rados.2.smithi204.stdout:3046: snap_create 2024-09-16T05:22:06.880 INFO:tasks.rados.rados.2.smithi204.stdout:3045: finishing copy_from to smithi20457400-9 2024-09-16T05:22:06.880 INFO:tasks.rados.rados.2.smithi204.stdout:3045: got expected ENOENT (src dne) 2024-09-16T05:22:06.880 INFO:tasks.rados.rados.2.smithi204.stdout:3045: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:22:06.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:06.987 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:06.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:07.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:07.110 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:07.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:06 smithi032 ceph-mon[27388]: 2.15 deep-scrub starts 2024-09-16T05:22:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:06 smithi032 ceph-mon[27388]: 2.15 deep-scrub ok 2024-09-16T05:22:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:06 smithi032 ceph-mon[27388]: 1.1c scrub starts 2024-09-16T05:22:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:06 smithi032 ceph-mon[27388]: 1.1c scrub ok 2024-09-16T05:22:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:06 smithi204 ceph-mon[30933]: 2.15 deep-scrub starts 2024-09-16T05:22:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:06 smithi204 ceph-mon[30933]: 2.15 deep-scrub ok 2024-09-16T05:22:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:06 smithi204 ceph-mon[30933]: 1.1c scrub starts 2024-09-16T05:22:07.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:06 smithi204 ceph-mon[30933]: 1.1c scrub ok 2024-09-16T05:22:07.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:07.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:06 smithi196 ceph-mon[30802]: 2.15 deep-scrub starts 2024-09-16T05:22:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:06 smithi196 ceph-mon[30802]: 2.15 deep-scrub ok 2024-09-16T05:22:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:06 smithi196 ceph-mon[30802]: 1.1c scrub starts 2024-09-16T05:22:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:06 smithi196 ceph-mon[30802]: 1.1c scrub ok 2024-09-16T05:22:07.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:07.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:07.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:07.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-16T05:22:07.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:07.364 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:07.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:07.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:07.489 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:07.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:07.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:07.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:07.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:07.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:07.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:07.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:07.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:07.865 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:07.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:07.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:07.884 INFO:tasks.rados.rados.2.smithi204.stdout:3045: done (1 left) 2024-09-16T05:22:07.884 INFO:tasks.rados.rados.2.smithi204.stdout:3046: done (0 left) 2024-09-16T05:22:07.884 INFO:tasks.rados.rados.2.smithi204.stdout:3047: read oid 19 snap -1 2024-09-16T05:22:07.885 INFO:tasks.rados.rados.2.smithi204.stdout:3047: expect (ObjNum 1020 snap 310 seq_num 1020) 2024-09-16T05:22:07.885 INFO:tasks.rados.rados.2.smithi204.stdout:3048: delete oid 36 current snap is 327 2024-09-16T05:22:07.889 INFO:tasks.rados.rados.2.smithi204.stdout:3048: done (1 left) 2024-09-16T05:22:07.889 INFO:tasks.rados.rados.2.smithi204.stdout:3049: copy_from oid 31 from oid 20 current snap is 327 2024-09-16T05:22:07.889 INFO:tasks.rados.rados.2.smithi204.stdout:3050: write oid 41 current snap is 327 2024-09-16T05:22:07.889 INFO:tasks.rados.rados.2.smithi204.stdout:3050: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2024-09-16T05:22:07.896 INFO:tasks.rados.rados.2.smithi204.stdout:3050: writing smithi20457400-41 from 783490 to 1189340 tid 1 2024-09-16T05:22:07.905 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:07.905 INFO:tasks.rados.rados.2.smithi204.stdout:3050: writing smithi20457400-41 from 1980674 to 2707704 tid 2 2024-09-16T05:22:07.910 INFO:tasks.rados.rados.2.smithi204.stdout:3050: writing smithi20457400-41 from 3184014 to 3671656 tid 3 2024-09-16T05:22:07.910 INFO:tasks.rados.rados.2.smithi204.stdout:3051: rollback oid 4 current snap is 327 2024-09-16T05:22:07.911 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 279 2024-09-16T05:22:07.911 INFO:tasks.rados.rados.2.smithi204.stdout:3047: expect (ObjNum 1020 snap 310 seq_num 1020) 2024-09-16T05:22:07.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:07.990 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:07.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:08.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:08.095 INFO:tasks.rados.rados.2.smithi204.stdout:3049: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:22:08.095 INFO:tasks.rados.rados.2.smithi204.stdout:3049: finishing copy_from to smithi20457400-31 2024-09-16T05:22:08.095 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 1179 (ObjNum 989 snap 292 seq_num 989) dirty exists 2024-09-16T05:22:08.095 INFO:tasks.rados.rados.2.smithi204.stdout:3050: finishing write tid 1 to smithi20457400-41 2024-09-16T05:22:08.095 INFO:tasks.rados.rados.2.smithi204.stdout:3047: done (3 left) 2024-09-16T05:22:08.095 INFO:tasks.rados.rados.2.smithi204.stdout:3049: done (2 left) 2024-09-16T05:22:08.095 INFO:tasks.rados.rados.2.smithi204.stdout:3052: copy_from oid 44 from oid 17 current snap is 327 2024-09-16T05:22:08.095 INFO:tasks.rados.rados.2.smithi204.stdout:3050: finishing write tid 2 to smithi20457400-41 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:3051: finishing rollback tid 0 to smithi20457400-4 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:3051: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1043 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:3050: finishing write tid 3 to smithi20457400-41 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:3050: finishing write tid 4 to smithi20457400-41 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:3050: finishing write tid 5 to smithi20457400-41 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1281 (ObjNum 1070 snap 327 seq_num 1070) dirty exists 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:3050: left oid 41 (ObjNum 1070 snap 327 seq_num 1070) 2024-09-16T05:22:08.096 INFO:tasks.rados.rados.2.smithi204.stdout:3050: done (2 left) 2024-09-16T05:22:08.097 INFO:tasks.rados.rados.2.smithi204.stdout:3051: done (1 left) 2024-09-16T05:22:08.097 INFO:tasks.rados.rados.2.smithi204.stdout:3053: rollback oid 45 current snap is 327 2024-09-16T05:22:08.097 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 326 2024-09-16T05:22:08.097 INFO:tasks.rados.rados.2.smithi204.stdout:3054: read oid 20 snap 320 2024-09-16T05:22:08.097 INFO:tasks.rados.rados.2.smithi204.stdout:3054: expect (ObjNum 989 snap 292 seq_num 989) 2024-09-16T05:22:08.097 INFO:tasks.rados.rados.2.smithi204.stdout:3055: read oid 46 snap -1 2024-09-16T05:22:08.097 INFO:tasks.rados.rados.2.smithi204.stdout:3055: expect (ObjNum 969 snap 287 seq_num 969) 2024-09-16T05:22:08.097 INFO:tasks.rados.rados.2.smithi204.stdout:3056: snap_remove snap 319 2024-09-16T05:22:08.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: pgmap v1326: 127 pgs: 127 active+clean; 342 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-16T05:22:08.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 2.26 deep-scrub starts 2024-09-16T05:22:08.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 2.26 deep-scrub ok 2024-09-16T05:22:08.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 1.15 scrub starts 2024-09-16T05:22:08.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 1.15 scrub ok 2024-09-16T05:22:08.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: osdmap e777: 12 total, 12 up, 12 in 2024-09-16T05:22:08.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:22:08.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:22:08.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:22:08.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:22:08.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 1.1b scrub starts 2024-09-16T05:22:08.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:07 smithi032 ceph-mon[27388]: 1.1b scrub ok 2024-09-16T05:22:08.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:08.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: pgmap v1326: 127 pgs: 127 active+clean; 342 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-16T05:22:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 2.26 deep-scrub starts 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 2.26 deep-scrub ok 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 1.15 scrub starts 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 1.15 scrub ok 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: osdmap e777: 12 total, 12 up, 12 in 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 1.1b scrub starts 2024-09-16T05:22:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:07 smithi204 ceph-mon[30933]: 1.1b scrub ok 2024-09-16T05:22:08.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:08.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: pgmap v1326: 127 pgs: 127 active+clean; 342 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.3 MiB/s wr, 13 op/s 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 2.26 deep-scrub starts 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 2.26 deep-scrub ok 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 1.15 scrub starts 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 1.15 scrub ok 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: osdmap e777: 12 total, 12 up, 12 in 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 1.1b scrub starts 2024-09-16T05:22:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:07 smithi196 ceph-mon[30802]: 1.1b scrub ok 2024-09-16T05:22:08.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:08.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:08.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:08.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:08.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:08.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:08.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:08.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:08.537 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:22:08.537 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:22:08.537 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:22:08.538 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:22:08.538 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:22:08.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:08.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:08.619 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:08.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:08.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:08.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:08.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:08.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:08.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:08.893 INFO:tasks.rados.rados.2.smithi204.stdout:3056: done (4 left) 2024-09-16T05:22:08.893 INFO:tasks.rados.rados.2.smithi204.stdout:3057: write oid 50 current snap is 327 2024-09-16T05:22:08.893 INFO:tasks.rados.rados.2.smithi204.stdout:3057: seq_num 1071 ranges {400297=481099,1639156=361141} 2024-09-16T05:22:08.898 INFO:tasks.rados.rados.2.smithi204.stdout:3057: writing smithi20457400-50 from 400297 to 881396 tid 1 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:3057: writing smithi20457400-50 from 1639156 to 2000297 tid 2 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:3058: read oid 38 snap 322 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:3058: expect (ObjNum 1042 snap 317 seq_num 1042) 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:3059: rollback oid 22 current snap is 327 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 326 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:3053: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:3060: read oid 13 snap 323 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:3060: expect deleted 2024-09-16T05:22:08.905 INFO:tasks.rados.rados.2.smithi204.stdout:3061: delete oid 40 current snap is 327 2024-09-16T05:22:08.906 INFO:tasks.rados.rados.2.smithi204.stdout:3053: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:22:08.906 INFO:tasks.rados.rados.2.smithi204.stdout:3054: expect (ObjNum 989 snap 292 seq_num 989) 2024-09-16T05:22:08.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:08.994 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:08.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:3053: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:22:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1260 (ObjNum 995 snap 292 seq_num 995) dirty exists 2024-09-16T05:22:09.073 INFO:tasks.rados.rados.2.smithi204.stdout:3055: expect (ObjNum 969 snap 287 seq_num 969) 2024-09-16T05:22:09.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:09.118 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:09.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:09.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:08 smithi032 ceph-mon[27388]: 1.1a scrub starts 2024-09-16T05:22:09.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:08 smithi032 ceph-mon[27388]: 1.1a scrub ok 2024-09-16T05:22:09.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:08 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:22:09.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:08 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:22:09.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:08 smithi032 ceph-mon[27388]: osdmap e778: 12 total, 12 up, 12 in 2024-09-16T05:22:09.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:08 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:22:09.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:08 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:22:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:08 smithi204 ceph-mon[30933]: 1.1a scrub starts 2024-09-16T05:22:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:08 smithi204 ceph-mon[30933]: 1.1a scrub ok 2024-09-16T05:22:09.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:08 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:22:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:08 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:22:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:08 smithi204 ceph-mon[30933]: osdmap e778: 12 total, 12 up, 12 in 2024-09-16T05:22:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:08 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:22:09.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:08 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:3052: finishing copy_from racing read to smithi20457400-44 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:3052: finishing copy_from to smithi20457400-44 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1121 (ObjNum 961 snap 284 seq_num 961) dirty exists 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:3052: done (8 left) 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:3053: done (7 left) 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:3054: done (6 left) 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:3055: done (5 left) 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:3061: done (4 left) 2024-09-16T05:22:09.216 INFO:tasks.rados.rados.2.smithi204.stdout:3062: rollback oid 1 current snap is 327 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 1 to 322 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:3057: finishing write tid 1 to smithi20457400-50 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:3059: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 783 (ObjNum 33 snap 0 seq_num 4265379192) dirty dne 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:3057: finishing write tid 2 to smithi20457400-50 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:3057: finishing write tid 3 to smithi20457400-50 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:3057: finishing write tid 4 to smithi20457400-50 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1138 (ObjNum 1071 snap 327 seq_num 1071) dirty exists 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:3057: left oid 50 (ObjNum 1071 snap 327 seq_num 1071) 2024-09-16T05:22:09.217 INFO:tasks.rados.rados.2.smithi204.stdout:3058: expect (ObjNum 1042 snap 317 seq_num 1042) 2024-09-16T05:22:09.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:09.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:08 smithi196 ceph-mon[30802]: 1.1a scrub starts 2024-09-16T05:22:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:08 smithi196 ceph-mon[30802]: 1.1a scrub ok 2024-09-16T05:22:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:08 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:22:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:08 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:22:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:08 smithi196 ceph-mon[30802]: osdmap e778: 12 total, 12 up, 12 in 2024-09-16T05:22:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:08 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:22:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:08 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:22:09.241 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:09.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:09.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:09.365 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:09.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:09.369 INFO:tasks.rados.rados.2.smithi204.stdout:3062: finishing rollback tid 1 to smithi20457400-1 2024-09-16T05:22:09.369 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 566 (ObjNum 257 snap 0 seq_num 4282987680) dirty dne 2024-09-16T05:22:09.369 INFO:tasks.rados.rados.2.smithi204.stdout:3057: done (4 left) 2024-09-16T05:22:09.369 INFO:tasks.rados.rados.2.smithi204.stdout:3058: done (3 left) 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3059: done (2 left) 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3060: done (1 left) 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3062: done (0 left) 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3063: read oid 31 snap 318 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3063: expect deleted 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3064: read oid 43 snap 279 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3064: expect deleted 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3065: read oid 13 snap -1 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3065: expect deleted 2024-09-16T05:22:09.370 INFO:tasks.rados.rados.2.smithi204.stdout:3066: snap_remove snap 325 2024-09-16T05:22:09.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:22:09.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:09.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:09.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:09.516 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:09.516 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:09.516 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:09.516 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:09.517 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:09.518 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:09.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:09.618 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:09.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:09.700 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:09.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:09.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:09.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:09.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:22:09.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:09.896 INFO:tasks.rados.rados.2.smithi204.stdout:3066: done (3 left) 2024-09-16T05:22:09.896 INFO:tasks.rados.rados.2.smithi204.stdout:3067: read oid 47 snap -1 2024-09-16T05:22:09.896 INFO:tasks.rados.rados.2.smithi204.stdout:3067: expect deleted 2024-09-16T05:22:09.896 INFO:tasks.rados.rados.2.smithi204.stdout:3068: write (excl) oid 39 current snap is 327 2024-09-16T05:22:09.897 INFO:tasks.rados.rados.2.smithi204.stdout:3068: seq_num 1073 ranges {433911=747950,1756120=277791} 2024-09-16T05:22:09.902 INFO:tasks.rados.rados.2.smithi204.stdout:3068: writing smithi20457400-39 from 433911 to 1181861 tid 1 2024-09-16T05:22:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:09.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:09.908 INFO:tasks.rados.rados.2.smithi204.stdout:3068: writing smithi20457400-39 from 1756120 to 2033911 tid 2 2024-09-16T05:22:09.908 INFO:tasks.rados.rados.2.smithi204.stdout:3069: read oid 42 snap -1 2024-09-16T05:22:09.908 INFO:tasks.rados.rados.2.smithi204.stdout:3069: expect deleted 2024-09-16T05:22:09.908 INFO:tasks.rados.rados.2.smithi204.stdout:3070: write (excl) oid 46 current snap is 327 2024-09-16T05:22:09.908 INFO:tasks.rados.rados.2.smithi204.stdout:3070: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-09-16T05:22:09.912 INFO:tasks.rados.rados.2.smithi204.stdout:3070: writing smithi20457400-46 from 450718 to 873917 tid 1 2024-09-16T05:22:09.918 INFO:tasks.rados.rados.2.smithi204.stdout:3070: writing smithi20457400-46 from 1414602 to 2050718 tid 2 2024-09-16T05:22:09.918 INFO:tasks.rados.rados.2.smithi204.stdout:3071: delete oid 18 current snap is 327 2024-09-16T05:22:09.918 INFO:tasks.rados.rados.2.smithi204.stdout:3068: finishing write tid 1 to smithi20457400-39 2024-09-16T05:22:09.918 INFO:tasks.rados.rados.2.smithi204.stdout:3068: finishing write tid 2 to smithi20457400-39 2024-09-16T05:22:09.918 INFO:tasks.rados.rados.2.smithi204.stdout:3068: finishing write tid 3 to smithi20457400-39 2024-09-16T05:22:09.919 INFO:tasks.rados.rados.2.smithi204.stdout:3068: finishing write tid 4 to smithi20457400-39 2024-09-16T05:22:09.919 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1239 (ObjNum 1073 snap 327 seq_num 1073) dirty exists 2024-09-16T05:22:09.919 INFO:tasks.rados.rados.2.smithi204.stdout:3068: left oid 39 (ObjNum 1073 snap 327 seq_num 1073) 2024-09-16T05:22:09.919 INFO:tasks.rados.rados.2.smithi204.stdout:3070: finishing write tid 1 to smithi20457400-46 2024-09-16T05:22:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:3063: done (7 left) 2024-09-16T05:22:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:3064: done (6 left) 2024-09-16T05:22:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:3065: done (5 left) 2024-09-16T05:22:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:3067: done (4 left) 2024-09-16T05:22:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:3068: done (3 left) 2024-09-16T05:22:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:3069: done (2 left) 2024-09-16T05:22:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:3071: done (1 left) 2024-09-16T05:22:09.925 INFO:tasks.rados.rados.2.smithi204.stdout:3072: read oid 27 snap -1 2024-09-16T05:22:09.926 INFO:tasks.rados.rados.2.smithi204.stdout:3072: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:22:09.926 INFO:tasks.rados.rados.2.smithi204.stdout:3073: write oid 25 current snap is 327 2024-09-16T05:22:09.926 INFO:tasks.rados.rados.2.smithi204.stdout:3070: finishing write tid 2 to smithi20457400-46 2024-09-16T05:22:09.926 INFO:tasks.rados.rados.2.smithi204.stdout:3070: finishing write tid 3 to smithi20457400-46 2024-09-16T05:22:09.926 INFO:tasks.rados.rados.2.smithi204.stdout:3073: seq_num 1076 ranges {484332=573697,1531566=552766} 2024-09-16T05:22:09.927 INFO:tasks.rados.rados.2.smithi204.stdout:3073: writing smithi20457400-25 from 484332 to 1058029 tid 1 2024-09-16T05:22:09.934 INFO:tasks.rados.rados.2.smithi204.stdout:3073: writing smithi20457400-25 from 1531566 to 2084332 tid 2 2024-09-16T05:22:09.935 INFO:tasks.rados.rados.2.smithi204.stdout:3074: rollback oid 8 current snap is 327 2024-09-16T05:22:09.935 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 8 to 324 2024-09-16T05:22:09.935 INFO:tasks.rados.rados.2.smithi204.stdout:3070: finishing write tid 4 to smithi20457400-46 2024-09-16T05:22:09.935 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1017 (ObjNum 1074 snap 327 seq_num 1074) dirty exists 2024-09-16T05:22:09.935 INFO:tasks.rados.rados.2.smithi204.stdout:3070: left oid 46 (ObjNum 1074 snap 327 seq_num 1074) 2024-09-16T05:22:09.935 INFO:tasks.rados.rados.2.smithi204.stdout:3070: done (3 left) 2024-09-16T05:22:09.935 INFO:tasks.rados.rados.2.smithi204.stdout:3075: snap_remove snap 318 2024-09-16T05:22:10.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:10.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:10.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:10.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:09 smithi032 ceph-mon[27388]: pgmap v1329: 127 pgs: 127 active+clean; 350 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-16T05:22:10.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:09 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:22:10.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:09 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:22:10.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:09 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:22:10.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:09 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:22:10.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:09 smithi032 ceph-mon[27388]: osdmap e779: 12 total, 12 up, 12 in 2024-09-16T05:22:10.120 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1200523409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:10.134 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:10.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:09 smithi204 ceph-mon[30933]: pgmap v1329: 127 pgs: 127 active+clean; 350 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-16T05:22:10.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:09 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:22:10.149 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:09 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:22:10.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:09 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:22:10.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:09 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:22:10.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:09 smithi204 ceph-mon[30933]: osdmap e779: 12 total, 12 up, 12 in 2024-09-16T05:22:10.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1200523409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:10.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:10.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:10.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:10.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:09 smithi196 ceph-mon[30802]: pgmap v1329: 127 pgs: 127 active+clean; 350 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-16T05:22:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:09 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:22:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:09 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:22:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:09 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:22:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:09 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:22:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:09 smithi196 ceph-mon[30802]: osdmap e779: 12 total, 12 up, 12 in 2024-09-16T05:22:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1200523409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:10.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:10.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:10.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:10.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:10.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:10.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:10.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:10.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:10.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:10.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:10.631 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:10.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:10.660 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:10.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:10.762 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:10.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:10.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:10.825 INFO:tasks.rados.rados.2.smithi204.stdout:3075: done (3 left) 2024-09-16T05:22:10.825 INFO:tasks.rados.rados.2.smithi204.stdout:3076: delete oid 39 current snap is 327 2024-09-16T05:22:10.825 INFO:tasks.rados.rados.2.smithi204.stdout:3073: finishing write tid 1 to smithi20457400-25 2024-09-16T05:22:10.826 INFO:tasks.rados.rados.2.smithi204.stdout:3072: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:22:10.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:10.896 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:10.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:10.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3074: finishing rollback tid 1 to smithi20457400-8 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1276 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3073: finishing write tid 2 to smithi20457400-25 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3073: finishing write tid 3 to smithi20457400-25 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3073: finishing write tid 4 to smithi20457400-25 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 749 (ObjNum 1076 snap 327 seq_num 1076) dirty exists 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3073: left oid 25 (ObjNum 1076 snap 327 seq_num 1076) 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3072: done (3 left) 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3073: done (2 left) 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3074: done (1 left) 2024-09-16T05:22:10.957 INFO:tasks.rados.rados.2.smithi204.stdout:3076: done (0 left) 2024-09-16T05:22:10.958 INFO:tasks.rados.rados.2.smithi204.stdout:3077: rollback oid 45 current snap is 327 2024-09-16T05:22:10.958 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 321 2024-09-16T05:22:10.958 INFO:tasks.rados.rados.2.smithi204.stdout:3078: snap_create 2024-09-16T05:22:10.962 INFO:tasks.rados.rados.2.smithi204.stdout:3077: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:22:10.962 INFO:tasks.rados.rados.2.smithi204.stdout:3077: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:22:10.963 INFO:tasks.rados.rados.2.smithi204.stdout:3077: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:22:10.963 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1263 (ObjNum 1038 snap 317 seq_num 1038) dirty exists 2024-09-16T05:22:11.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:11.037 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:11.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:11.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:11.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:10 smithi032 ceph-mon[27388]: osdmap e780: 12 total, 12 up, 12 in 2024-09-16T05:22:11.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:10 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:22:11.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:10 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:22:11.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:10 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:22:11.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:10 smithi032 ceph-mon[27388]: 2.44 scrub ok 2024-09-16T05:22:11.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:10 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:22:11.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:10 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:22:11.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:10 smithi032 ceph-mon[27388]: osdmap e781: 12 total, 12 up, 12 in 2024-09-16T05:22:11.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:11.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:10 smithi204 ceph-mon[30933]: osdmap e780: 12 total, 12 up, 12 in 2024-09-16T05:22:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:10 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:22:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:10 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:22:11.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:10 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:22:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:10 smithi204 ceph-mon[30933]: 2.44 scrub ok 2024-09-16T05:22:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:10 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:22:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:10 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:22:11.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:10 smithi204 ceph-mon[30933]: osdmap e781: 12 total, 12 up, 12 in 2024-09-16T05:22:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:10 smithi196 ceph-mon[30802]: osdmap e780: 12 total, 12 up, 12 in 2024-09-16T05:22:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:10 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:22:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:10 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:22:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:10 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:22:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:10 smithi196 ceph-mon[30802]: 2.44 scrub ok 2024-09-16T05:22:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:10 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:22:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:10 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:22:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:10 smithi196 ceph-mon[30802]: osdmap e781: 12 total, 12 up, 12 in 2024-09-16T05:22:11.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:11.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:11.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:11.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:11.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:11.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:11.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:11.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:11.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:11.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-16T05:22:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:11.668 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:11.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:11.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:11.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:11.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:11.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:11.909 INFO:tasks.rados.rados.2.smithi204.stdout:3077: done (1 left) 2024-09-16T05:22:11.910 INFO:tasks.rados.rados.2.smithi204.stdout:3078: done (0 left) 2024-09-16T05:22:11.910 INFO:tasks.rados.rados.2.smithi204.stdout:3079: delete oid 8 current snap is 328 2024-09-16T05:22:11.914 INFO:tasks.rados.rados.2.smithi204.stdout:3079: done (0 left) 2024-09-16T05:22:11.914 INFO:tasks.rados.rados.2.smithi204.stdout:3080: read oid 38 snap -1 2024-09-16T05:22:11.914 INFO:tasks.rados.rados.2.smithi204.stdout:3080: expect (ObjNum 1042 snap 317 seq_num 1042) 2024-09-16T05:22:11.914 INFO:tasks.rados.rados.2.smithi204.stdout:3081: copy_from oid 12 from oid 35 current snap is 328 2024-09-16T05:22:11.914 INFO:tasks.rados.rados.2.smithi204.stdout:3082: copy_from oid 1 from oid 14 current snap is 328 2024-09-16T05:22:11.914 INFO:tasks.rados.rados.2.smithi204.stdout:3083: write oid 10 current snap is 328 2024-09-16T05:22:11.914 INFO:tasks.rados.rados.2.smithi204.stdout:3083: seq_num 1079 ranges {534753=799444,1990659=144094} 2024-09-16T05:22:11.921 INFO:tasks.rados.rados.2.smithi204.stdout:3083: writing smithi20457400-10 from 534753 to 1334197 tid 1 2024-09-16T05:22:11.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:11.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:11.927 INFO:tasks.rados.rados.2.smithi204.stdout:3083: writing smithi20457400-10 from 1990659 to 2134753 tid 2 2024-09-16T05:22:11.927 INFO:tasks.rados.rados.2.smithi204.stdout:3084: snap_remove snap 322 2024-09-16T05:22:12.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:12.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:12.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:12.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:22:12.151 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:12.166 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: pgmap v1332: 127 pgs: 127 active+clean; 355 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1002 KiB/s wr, 11 op/s 2024-09-16T05:22:12.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:22:12.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:22:12.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: 2.47 deep-scrub starts 2024-09-16T05:22:12.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: 2.47 deep-scrub ok 2024-09-16T05:22:12.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: 2.51 deep-scrub starts 2024-09-16T05:22:12.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: 2.51 deep-scrub ok 2024-09-16T05:22:12.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:22:12.167 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:11 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:22:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: pgmap v1332: 127 pgs: 127 active+clean; 355 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1002 KiB/s wr, 11 op/s 2024-09-16T05:22:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:22:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:22:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: 2.47 deep-scrub starts 2024-09-16T05:22:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: 2.47 deep-scrub ok 2024-09-16T05:22:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: 2.51 deep-scrub starts 2024-09-16T05:22:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: 2.51 deep-scrub ok 2024-09-16T05:22:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:22:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:11 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:22:12.173 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:12.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:12.202 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:12.202 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: pgmap v1332: 127 pgs: 127 active+clean; 355 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1002 KiB/s wr, 11 op/s 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: 2.47 deep-scrub starts 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: 2.47 deep-scrub ok 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: 2.51 deep-scrub starts 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: 2.51 deep-scrub ok 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:22:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:11 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:22:12.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:12.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:12.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:12.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:12.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:12.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:12.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:12.555 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:12.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:12.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:12.681 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:12.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:12.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:12.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:12.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:12.912 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:22:12.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:12.917 INFO:tasks.rados.rados.2.smithi204.stdout:3084: done (4 left) 2024-09-16T05:22:12.917 INFO:tasks.rados.rados.2.smithi204.stdout:3085: snap_create 2024-09-16T05:22:12.917 INFO:tasks.rados.rados.2.smithi204.stdout:3080: expect (ObjNum 1042 snap 317 seq_num 1042) 2024-09-16T05:22:12.946 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:12.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:12.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:22:13.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:13.070 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:13.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3081: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3081: finishing copy_from to smithi20457400-12 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 858 (ObjNum 1035 snap 317 seq_num 1035) dirty exists 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3083: finishing write tid 1 to smithi20457400-10 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3083: finishing write tid 2 to smithi20457400-10 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3083: finishing write tid 3 to smithi20457400-10 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3083: finishing write tid 4 to smithi20457400-10 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1284 (ObjNum 1079 snap 328 seq_num 1079) dirty exists 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3083: left oid 10 (ObjNum 1079 snap 328 seq_num 1079) 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3082: finishing copy_from racing read to smithi20457400-1 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:3082: finishing copy_from to smithi20457400-1 2024-09-16T05:22:13.072 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 1051 (ObjNum 1011 snap 306 seq_num 1011) dirty exists 2024-09-16T05:22:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:12 smithi032 ceph-mon[27388]: osdmap e782: 12 total, 12 up, 12 in 2024-09-16T05:22:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2916053411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:12 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:22:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:12 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:22:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:12 smithi204 ceph-mon[30933]: osdmap e782: 12 total, 12 up, 12 in 2024-09-16T05:22:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2916053411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:12 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:22:13.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:12 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:22:13.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:13.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:13.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:12 smithi196 ceph-mon[30802]: osdmap e782: 12 total, 12 up, 12 in 2024-09-16T05:22:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2916053411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:12 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:22:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:12 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:22:13.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:13.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:13.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:13.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:13.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:13.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:13.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:13.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:13.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:13.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:13.704 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:13.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:13.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:13.829 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:13.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:13.838 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:13.920 INFO:tasks.rados.rados.2.smithi204.stdout:3080: done (4 left) 2024-09-16T05:22:13.920 INFO:tasks.rados.rados.2.smithi204.stdout:3081: done (3 left) 2024-09-16T05:22:13.920 INFO:tasks.rados.rados.2.smithi204.stdout:3082: done (2 left) 2024-09-16T05:22:13.920 INFO:tasks.rados.rados.2.smithi204.stdout:3083: done (1 left) 2024-09-16T05:22:13.920 INFO:tasks.rados.rados.2.smithi204.stdout:3085: done (0 left) 2024-09-16T05:22:13.921 INFO:tasks.rados.rados.2.smithi204.stdout:3086: read oid 24 snap -1 2024-09-16T05:22:13.921 INFO:tasks.rados.rados.2.smithi204.stdout:3086: expect deleted 2024-09-16T05:22:13.921 INFO:tasks.rados.rados.2.smithi204.stdout:3087: write oid 14 current snap is 329 2024-09-16T05:22:13.921 INFO:tasks.rados.rados.2.smithi204.stdout:3087: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-09-16T05:22:13.926 INFO:tasks.rados.rados.2.smithi204.stdout:3087: writing smithi20457400-14 from 551560 to 1142606 tid 1 2024-09-16T05:22:13.931 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:13.932 INFO:tasks.rados.rados.2.smithi204.stdout:3087: writing smithi20457400-14 from 1765494 to 2151560 tid 2 2024-09-16T05:22:13.932 INFO:tasks.rados.rados.2.smithi204.stdout:3088: write (excl) oid 26 current snap is 329 2024-09-16T05:22:13.932 INFO:tasks.rados.rados.2.smithi204.stdout:3088: seq_num 1081 ranges {568367=666295,1707623=460744} 2024-09-16T05:22:13.938 INFO:tasks.rados.rados.2.smithi204.stdout:3088: writing smithi20457400-26 from 568367 to 1234662 tid 1 2024-09-16T05:22:13.944 INFO:tasks.rados.rados.2.smithi204.stdout:3088: writing smithi20457400-26 from 1707623 to 2168367 tid 2 2024-09-16T05:22:13.945 INFO:tasks.rados.rados.2.smithi204.stdout:3089: snap_remove snap 324 2024-09-16T05:22:13.955 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:13.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:13.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:14.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:14.080 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:14.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:14.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:13 smithi032 ceph-mon[27388]: pgmap v1335: 127 pgs: 127 active+clean; 357 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2024-09-16T05:22:14.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:13 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:22:14.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:13 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:22:14.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:13 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:14.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2916053411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:14.085 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:13 smithi032 ceph-mon[27388]: osdmap e783: 12 total, 12 up, 12 in 2024-09-16T05:22:14.111 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:13 smithi204 ceph-mon[30933]: pgmap v1335: 127 pgs: 127 active+clean; 357 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2024-09-16T05:22:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:13 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:22:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:13 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:22:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:13 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2916053411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:13 smithi204 ceph-mon[30933]: osdmap e783: 12 total, 12 up, 12 in 2024-09-16T05:22:14.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:14.208 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:14.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:13 smithi196 ceph-mon[30802]: pgmap v1335: 127 pgs: 127 active+clean; 357 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2024-09-16T05:22:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:13 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:22:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:13 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:22:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:13 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2916053411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:13 smithi196 ceph-mon[30802]: osdmap e783: 12 total, 12 up, 12 in 2024-09-16T05:22:14.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:14.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:14.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:14.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:14.463 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:14.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:14.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:14.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:14.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:14.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:14.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:14.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:3089: done (3 left) 2024-09-16T05:22:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:3090: write oid 43 current snap is 329 2024-09-16T05:22:14.928 INFO:tasks.rados.rados.2.smithi204.stdout:3090: seq_num 1082 ranges {585174=741544,1766105=419069} 2024-09-16T05:22:14.935 INFO:tasks.rados.rados.2.smithi204.stdout:3090: writing smithi20457400-43 from 585174 to 1326718 tid 1 2024-09-16T05:22:14.940 INFO:tasks.rados.rados.2.smithi204.stdout:3090: writing smithi20457400-43 from 1766105 to 2185174 tid 2 2024-09-16T05:22:14.940 INFO:tasks.rados.rados.2.smithi204.stdout:3091: snap_remove snap 298 2024-09-16T05:22:15.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:15.020 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:14 smithi032 ceph-mon[27388]: osdmap e784: 12 total, 12 up, 12 in 2024-09-16T05:22:15.020 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:14 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:15.020 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:14 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:15.029 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:15.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:15.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:15.157 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:15.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:15.221 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:15.221 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:14 smithi196 ceph-mon[30802]: osdmap e784: 12 total, 12 up, 12 in 2024-09-16T05:22:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:14 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:14 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:15.242 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:15.242 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:15.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:15.263 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:15.263 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:15.283 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:15.284 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:15.285 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:15.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:15.305 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:15.305 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:15.327 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:15.327 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:15.350 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:15.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:15.372 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:15.373 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:15.386 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:15.394 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:15.394 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:15.412 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:15.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:15.416 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:14 smithi204 ceph-mon[30933]: osdmap e784: 12 total, 12 up, 12 in 2024-09-16T05:22:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:14 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:14 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:15.438 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:15.439 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:15.440 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:22:15.440 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:22:15.440 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:22:15.440 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:22:15.461 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:15.461 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:15.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:15.538 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:15.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:15.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:15.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:15.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:15.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:15.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:22:15.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:15.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:15.826 INFO:tasks.rados.rados.2.smithi204.stdout:3091: done (4 left) 2024-09-16T05:22:15.826 INFO:tasks.rados.rados.2.smithi204.stdout:3092: write oid 16 current snap is 329 2024-09-16T05:22:15.827 INFO:tasks.rados.rados.2.smithi204.stdout:3092: seq_num 1083 ranges {601981=416793,1424587=492702} 2024-09-16T05:22:15.832 INFO:tasks.rados.rados.2.smithi204.stdout:3092: writing smithi20457400-16 from 601981 to 1018774 tid 1 2024-09-16T05:22:15.839 INFO:tasks.rados.rados.2.smithi204.stdout:3092: writing smithi20457400-16 from 1424587 to 1917289 tid 2 2024-09-16T05:22:15.839 INFO:tasks.rados.rados.2.smithi204.stdout:3093: rollback oid 34 current snap is 329 2024-09-16T05:22:15.839 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 321 2024-09-16T05:22:15.839 INFO:tasks.rados.rados.2.smithi204.stdout:3087: finishing write tid 1 to smithi20457400-14 2024-09-16T05:22:15.839 INFO:tasks.rados.rados.2.smithi204.stdout:3087: finishing write tid 2 to smithi20457400-14 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3087: finishing write tid 3 to smithi20457400-14 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3087: finishing write tid 4 to smithi20457400-14 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1353 (ObjNum 1080 snap 329 seq_num 1080) dirty exists 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3087: left oid 14 (ObjNum 1080 snap 329 seq_num 1080) 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3088: finishing write tid 1 to smithi20457400-26 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3088: finishing write tid 2 to smithi20457400-26 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3088: finishing write tid 3 to smithi20457400-26 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3088: finishing write tid 4 to smithi20457400-26 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1357 (ObjNum 1081 snap 329 seq_num 1081) dirty exists 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3088: left oid 26 (ObjNum 1081 snap 329 seq_num 1081) 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3090: finishing write tid 1 to smithi20457400-43 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3090: finishing write tid 2 to smithi20457400-43 2024-09-16T05:22:15.840 INFO:tasks.rados.rados.2.smithi204.stdout:3090: finishing write tid 3 to smithi20457400-43 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3090: finishing write tid 4 to smithi20457400-43 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1047 (ObjNum 1082 snap 329 seq_num 1082) dirty exists 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3090: left oid 43 (ObjNum 1082 snap 329 seq_num 1082) 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3092: finishing write tid 1 to smithi20457400-16 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3086: done (5 left) 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3087: done (4 left) 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3088: done (3 left) 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3090: done (2 left) 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3094: write (excl) oid 6 current snap is 329 2024-09-16T05:22:15.841 INFO:tasks.rados.rados.2.smithi204.stdout:3094: seq_num 1084 ranges {618788=492042,1883069=335719} 2024-09-16T05:22:15.845 INFO:tasks.rados.rados.2.smithi204.stdout:3094: writing smithi20457400-6 from 618788 to 1110830 tid 1 2024-09-16T05:22:15.852 INFO:tasks.rados.rados.2.smithi204.stdout:3094: writing smithi20457400-6 from 1883069 to 2218788 tid 2 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3095: rollback oid 40 current snap is 329 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 40 to 312 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3092: finishing write tid 2 to smithi20457400-16 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3096: snap_create 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3092: finishing write tid 3 to smithi20457400-16 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3092: finishing write tid 4 to smithi20457400-16 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 16 v 1046 (ObjNum 1083 snap 329 seq_num 1083) dirty exists 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3092: left oid 16 (ObjNum 1083 snap 329 seq_num 1083) 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3093: finishing rollback tid 0 to smithi20457400-34 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3093: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3093: finishing rollback tid 2 to smithi20457400-34 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1262 (ObjNum 969 snap 287 seq_num 969) dirty exists 2024-09-16T05:22:15.853 INFO:tasks.rados.rados.2.smithi204.stdout:3094: finishing write tid 1 to smithi20457400-6 2024-09-16T05:22:15.854 INFO:tasks.rados.rados.2.smithi204.stdout:3095: finishing rollback tid 1 to smithi20457400-40 2024-09-16T05:22:15.854 INFO:tasks.rados.rados.2.smithi204.stdout:3095: finishing rollback tid 2 to smithi20457400-40 2024-09-16T05:22:15.855 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1054 (ObjNum 1011 snap 306 seq_num 1011) dirty exists 2024-09-16T05:22:15.855 INFO:tasks.rados.rados.2.smithi204.stdout:3094: finishing write tid 2 to smithi20457400-6 2024-09-16T05:22:15.855 INFO:tasks.rados.rados.2.smithi204.stdout:3094: finishing write tid 3 to smithi20457400-6 2024-09-16T05:22:15.855 INFO:tasks.rados.rados.2.smithi204.stdout:3094: finishing write tid 4 to smithi20457400-6 2024-09-16T05:22:15.855 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 1236 (ObjNum 1084 snap 329 seq_num 1084) dirty exists 2024-09-16T05:22:15.855 INFO:tasks.rados.rados.2.smithi204.stdout:3094: left oid 6 (ObjNum 1084 snap 329 seq_num 1084) 2024-09-16T05:22:15.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:15.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:15.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:16.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:16.046 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:16.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:16.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:16.175 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:16.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:15 smithi196 ceph-mon[30802]: pgmap v1338: 127 pgs: 127 active+clean; 364 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 20 op/s 2024-09-16T05:22:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:15 smithi196 ceph-mon[30802]: osdmap e785: 12 total, 12 up, 12 in 2024-09-16T05:22:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2689959678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:15 smithi196 ceph-mon[30802]: osdmap e786: 12 total, 12 up, 12 in 2024-09-16T05:22:16.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:16.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:16.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:16.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:16.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:16.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:15 smithi032 ceph-mon[27388]: pgmap v1338: 127 pgs: 127 active+clean; 364 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 20 op/s 2024-09-16T05:22:16.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:15 smithi032 ceph-mon[27388]: osdmap e785: 12 total, 12 up, 12 in 2024-09-16T05:22:16.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2689959678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:16.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:15 smithi032 ceph-mon[27388]: osdmap e786: 12 total, 12 up, 12 in 2024-09-16T05:22:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:15 smithi204 ceph-mon[30933]: pgmap v1338: 127 pgs: 127 active+clean; 364 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 20 op/s 2024-09-16T05:22:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:15 smithi204 ceph-mon[30933]: osdmap e785: 12 total, 12 up, 12 in 2024-09-16T05:22:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2689959678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:15 smithi204 ceph-mon[30933]: osdmap e786: 12 total, 12 up, 12 in 2024-09-16T05:22:16.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:16.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:16.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:16.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:16.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:16.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:16.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:16.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:16.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:16.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:16.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:16.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:16.829 INFO:tasks.rados.rados.2.smithi204.stdout:3092: done (4 left) 2024-09-16T05:22:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:3093: done (3 left) 2024-09-16T05:22:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:3094: done (2 left) 2024-09-16T05:22:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:3095: done (1 left) 2024-09-16T05:22:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:3096: done (0 left) 2024-09-16T05:22:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:3097: read oid 44 snap -1 2024-09-16T05:22:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:3097: expect (ObjNum 961 snap 284 seq_num 961) 2024-09-16T05:22:16.830 INFO:tasks.rados.rados.2.smithi204.stdout:3098: snap_remove snap 329 2024-09-16T05:22:16.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:22:16.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:16.966 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:16.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:17.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:17.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:17.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:17.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:17.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:17.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:17.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:17.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:17.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-16T05:22:17.468 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:17.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:17.502 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:17.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:17.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:17.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:17.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:17.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:17.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:17.839 INFO:tasks.rados.rados.2.smithi204.stdout:3098: done (1 left) 2024-09-16T05:22:17.839 INFO:tasks.rados.rados.2.smithi204.stdout:3099: write (excl) oid 11 current snap is 330 2024-09-16T05:22:17.840 INFO:tasks.rados.rados.2.smithi204.stdout:3099: seq_num 1085 ranges {635595=567291,1825198=410397} 2024-09-16T05:22:17.846 INFO:tasks.rados.rados.2.smithi204.stdout:3099: writing smithi20457400-11 from 635595 to 1202886 tid 1 2024-09-16T05:22:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:17.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:17.853 INFO:tasks.rados.rados.2.smithi204.stdout:3099: writing smithi20457400-11 from 1825198 to 2235595 tid 2 2024-09-16T05:22:17.853 INFO:tasks.rados.rados.2.smithi204.stdout:3100: write oid 40 current snap is 330 2024-09-16T05:22:17.853 INFO:tasks.rados.rados.2.smithi204.stdout:3100: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-16T05:22:17.860 INFO:tasks.rados.rados.2.smithi204.stdout:3100: writing smithi20457400-40 from 652402 to 1294942 tid 1 2024-09-16T05:22:17.866 INFO:tasks.rados.rados.2.smithi204.stdout:3100: writing smithi20457400-40 from 1883680 to 2252402 tid 2 2024-09-16T05:22:17.866 INFO:tasks.rados.rados.2.smithi204.stdout:3101: write oid 15 current snap is 330 2024-09-16T05:22:17.866 INFO:tasks.rados.rados.2.smithi204.stdout:3101: seq_num 1087 ranges {669209=717789,1942162=327047} 2024-09-16T05:22:17.874 INFO:tasks.rados.rados.2.smithi204.stdout:3101: writing smithi20457400-15 from 669209 to 1386998 tid 1 2024-09-16T05:22:17.878 INFO:tasks.rados.rados.2.smithi204.stdout:3101: writing smithi20457400-15 from 1942162 to 2269209 tid 2 2024-09-16T05:22:17.879 INFO:tasks.rados.rados.2.smithi204.stdout:3102: delete oid 6 current snap is 330 2024-09-16T05:22:17.879 INFO:tasks.rados.rados.2.smithi204.stdout:3097: expect (ObjNum 961 snap 284 seq_num 961) 2024-09-16T05:22:17.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:17.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:17.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:17.993 INFO:tasks.rados.rados.2.smithi204.stdout:3099: finishing write tid 1 to smithi20457400-11 2024-09-16T05:22:17.993 INFO:tasks.rados.rados.2.smithi204.stdout:3099: finishing write tid 2 to smithi20457400-11 2024-09-16T05:22:17.993 INFO:tasks.rados.rados.2.smithi204.stdout:3099: finishing write tid 3 to smithi20457400-11 2024-09-16T05:22:17.994 INFO:tasks.rados.rados.2.smithi204.stdout:3099: finishing write tid 4 to smithi20457400-11 2024-09-16T05:22:17.994 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1258 (ObjNum 1085 snap 330 seq_num 1085) dirty exists 2024-09-16T05:22:17.994 INFO:tasks.rados.rados.2.smithi204.stdout:3099: left oid 11 (ObjNum 1085 snap 330 seq_num 1085) 2024-09-16T05:22:17.994 INFO:tasks.rados.rados.2.smithi204.stdout:3100: finishing write tid 1 to smithi20457400-40 2024-09-16T05:22:17.994 INFO:tasks.rados.rados.2.smithi204.stdout:3100: finishing write tid 2 to smithi20457400-40 2024-09-16T05:22:17.995 INFO:tasks.rados.rados.2.smithi204.stdout:3097: done (4 left) 2024-09-16T05:22:17.995 INFO:tasks.rados.rados.2.smithi204.stdout:3099: done (3 left) 2024-09-16T05:22:17.995 INFO:tasks.rados.rados.2.smithi204.stdout:3102: done (2 left) 2024-09-16T05:22:17.995 INFO:tasks.rados.rados.2.smithi204.stdout:3103: write oid 23 current snap is 330 2024-09-16T05:22:17.995 INFO:tasks.rados.rados.2.smithi204.stdout:3100: finishing write tid 3 to smithi20457400-40 2024-09-16T05:22:17.995 INFO:tasks.rados.rados.2.smithi204.stdout:3100: finishing write tid 4 to smithi20457400-40 2024-09-16T05:22:17.996 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1057 (ObjNum 1086 snap 330 seq_num 1086) dirty exists 2024-09-16T05:22:17.996 INFO:tasks.rados.rados.2.smithi204.stdout:3100: left oid 40 (ObjNum 1086 snap 330 seq_num 1086) 2024-09-16T05:22:17.996 INFO:tasks.rados.rados.2.smithi204.stdout:3101: finishing write tid 1 to smithi20457400-15 2024-09-16T05:22:17.996 INFO:tasks.rados.rados.2.smithi204.stdout:3103: seq_num 1089 ranges {702823=584640,2059126=243697} 2024-09-16T05:22:17.999 INFO:tasks.rados.rados.2.smithi204.stdout:3103: writing smithi20457400-23 from 702823 to 1287463 tid 1 2024-09-16T05:22:18.006 INFO:tasks.rados.rados.2.smithi204.stdout:3103: writing smithi20457400-23 from 2059126 to 2302823 tid 2 2024-09-16T05:22:18.006 INFO:tasks.rados.rados.2.smithi204.stdout:3100: done (2 left) 2024-09-16T05:22:18.006 INFO:tasks.rados.rados.2.smithi204.stdout:3104: copy_from oid 21 from oid 37 current snap is 330 2024-09-16T05:22:18.006 INFO:tasks.rados.rados.2.smithi204.stdout:3101: finishing write tid 2 to smithi20457400-15 2024-09-16T05:22:18.006 INFO:tasks.rados.rados.2.smithi204.stdout:3101: finishing write tid 3 to smithi20457400-15 2024-09-16T05:22:18.006 INFO:tasks.rados.rados.2.smithi204.stdout:3101: finishing write tid 4 to smithi20457400-15 2024-09-16T05:22:18.007 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 750 (ObjNum 1087 snap 330 seq_num 1087) dirty exists 2024-09-16T05:22:18.007 INFO:tasks.rados.rados.2.smithi204.stdout:3101: left oid 15 (ObjNum 1087 snap 330 seq_num 1087) 2024-09-16T05:22:18.007 INFO:tasks.rados.rados.2.smithi204.stdout:3103: finishing write tid 1 to smithi20457400-23 2024-09-16T05:22:18.007 INFO:tasks.rados.rados.2.smithi204.stdout:3101: done (2 left) 2024-09-16T05:22:18.007 INFO:tasks.rados.rados.2.smithi204.stdout:3105: delete oid 9 current snap is 330 2024-09-16T05:22:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3103: finishing write tid 2 to smithi20457400-23 2024-09-16T05:22:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3103: finishing write tid 3 to smithi20457400-23 2024-09-16T05:22:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3103: finishing write tid 4 to smithi20457400-23 2024-09-16T05:22:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 750 (ObjNum 1089 snap 330 seq_num 1089) dirty exists 2024-09-16T05:22:18.010 INFO:tasks.rados.rados.2.smithi204.stdout:3103: left oid 23 (ObjNum 1089 snap 330 seq_num 1089) 2024-09-16T05:22:18.010 INFO:tasks.rados.rados.2.smithi204.stdout:3103: done (2 left) 2024-09-16T05:22:18.010 INFO:tasks.rados.rados.2.smithi204.stdout:3105: done (1 left) 2024-09-16T05:22:18.010 INFO:tasks.rados.rados.2.smithi204.stdout:3106: snap_create 2024-09-16T05:22:18.018 INFO:tasks.rados.rados.2.smithi204.stdout:3104: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:22:18.020 INFO:tasks.rados.rados.2.smithi204.stdout:3104: finishing copy_from to smithi20457400-21 2024-09-16T05:22:18.021 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 1044 (ObjNum 1050 snap 317 seq_num 1050) dirty exists 2024-09-16T05:22:18.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:17 smithi032 ceph-mon[27388]: pgmap v1341: 127 pgs: 127 active+clean; 371 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-16T05:22:18.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:17 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:18.042 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:17 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:18.042 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:17 smithi032 ceph-mon[27388]: osdmap e787: 12 total, 12 up, 12 in 2024-09-16T05:22:18.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:18.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:18.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:17 smithi204 ceph-mon[30933]: pgmap v1341: 127 pgs: 127 active+clean; 371 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-16T05:22:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:17 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:17 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:17 smithi204 ceph-mon[30933]: osdmap e787: 12 total, 12 up, 12 in 2024-09-16T05:22:18.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:18.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:18.228 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:18.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:17 smithi196 ceph-mon[30802]: pgmap v1341: 127 pgs: 127 active+clean; 371 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-16T05:22:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:17 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:17 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:17 smithi196 ceph-mon[30802]: osdmap e787: 12 total, 12 up, 12 in 2024-09-16T05:22:18.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:18.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:18.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:18.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:18.477 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:18.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:18.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:22:18.547 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:22:18.547 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:22:18.547 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:22:18.547 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:22:18.547 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:22:18.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:18.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:18.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:18.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:18.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:18.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:22:18.849 INFO:tasks.rados.rados.2.smithi204.stdout:3104: done (1 left) 2024-09-16T05:22:18.849 INFO:tasks.rados.rados.2.smithi204.stdout:3106: done (0 left) 2024-09-16T05:22:18.849 INFO:tasks.rados.rados.2.smithi204.stdout:3107: snap_remove snap 320 2024-09-16T05:22:18.855 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:18.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:18.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:18.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:18.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:19.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:19.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:19.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:18 smithi032 ceph-mon[27388]: osdmap e788: 12 total, 12 up, 12 in 2024-09-16T05:22:19.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1458790812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:18 smithi204 ceph-mon[30933]: osdmap e788: 12 total, 12 up, 12 in 2024-09-16T05:22:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1458790812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:19.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:19.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:19.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:19.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:18 smithi196 ceph-mon[30802]: osdmap e788: 12 total, 12 up, 12 in 2024-09-16T05:22:19.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1458790812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:19.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:22:19.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:19.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:19.458 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:19.458 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:19.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:19.484 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:19.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:19.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:19.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:19.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:19.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:19.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:19.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:19.857 INFO:tasks.rados.rados.2.smithi204.stdout:3107: done (0 left) 2024-09-16T05:22:19.858 INFO:tasks.rados.rados.2.smithi204.stdout:3108: copy_from oid 26 from oid 33 current snap is 331 2024-09-16T05:22:19.858 INFO:tasks.rados.rados.2.smithi204.stdout:3109: snap_create 2024-09-16T05:22:19.858 INFO:tasks.rados.rados.2.smithi204.stdout:3108: finishing copy_from to smithi20457400-26 2024-09-16T05:22:19.858 INFO:tasks.rados.rados.2.smithi204.stdout:3108: got expected ENOENT (src dne) 2024-09-16T05:22:19.858 INFO:tasks.rados.rados.2.smithi204.stdout:3108: finishing copy_from racing read to smithi20457400-26 2024-09-16T05:22:19.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:19.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:19.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:19.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:19.996 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:19.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:20.090 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:19 smithi032 ceph-mon[27388]: pgmap v1344: 127 pgs: 127 active+clean; 373 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-16T05:22:20.090 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:19 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:20.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:19 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:20.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:19 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:20.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1458790812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:20.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:19 smithi032 ceph-mon[27388]: osdmap e789: 12 total, 12 up, 12 in 2024-09-16T05:22:20.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:20.121 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:20.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:20.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:19 smithi204 ceph-mon[30933]: pgmap v1344: 127 pgs: 127 active+clean; 373 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-16T05:22:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:19 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:19 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:19 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1458790812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:19 smithi204 ceph-mon[30933]: osdmap e789: 12 total, 12 up, 12 in 2024-09-16T05:22:20.199 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:20.200 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:20.222 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:20.223 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:20.223 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:19 smithi196 ceph-mon[30802]: pgmap v1344: 127 pgs: 127 active+clean; 373 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2024-09-16T05:22:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:19 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:19 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:19 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1458790812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:19 smithi196 ceph-mon[30802]: osdmap e789: 12 total, 12 up, 12 in 2024-09-16T05:22:20.243 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:20.243 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:20.245 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:20.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:20.265 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:20.265 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:20.285 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:20.285 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:20.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:20.307 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:20.329 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:20.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:20.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:20.351 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:20.372 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:20.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:20.374 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:20.374 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:20.396 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:20.396 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:20.418 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:20.418 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:20.440 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:20.473 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:20.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:20.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:20.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:20.626 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:20.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:20.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:20.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:20.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:20.831 INFO:tasks.rados.rados.2.smithi204.stdout:3108: done (1 left) 2024-09-16T05:22:20.831 INFO:tasks.rados.rados.2.smithi204.stdout:3109: done (0 left) 2024-09-16T05:22:20.831 INFO:tasks.rados.rados.2.smithi204.stdout:3110: rollback oid 21 current snap is 332 2024-09-16T05:22:20.831 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 21 to 305 2024-09-16T05:22:20.831 INFO:tasks.rados.rados.2.smithi204.stdout:3111: rollback oid 3 current snap is 332 2024-09-16T05:22:20.832 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 327 2024-09-16T05:22:20.832 INFO:tasks.rados.rados.2.smithi204.stdout:3112: write oid 10 current snap is 332 2024-09-16T05:22:20.832 INFO:tasks.rados.rados.2.smithi204.stdout:3112: seq_num 1091 ranges {736437=735138,2176090=160347} 2024-09-16T05:22:20.839 INFO:tasks.rados.rados.2.smithi204.stdout:3112: writing smithi20457400-10 from 736437 to 1471575 tid 1 2024-09-16T05:22:20.844 INFO:tasks.rados.rados.2.smithi204.stdout:3112: writing smithi20457400-10 from 2176090 to 2336437 tid 2 2024-09-16T05:22:20.844 INFO:tasks.rados.rados.2.smithi204.stdout:3113: snap_remove snap 321 2024-09-16T05:22:20.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:20.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:20.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:20.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: osdmap e790: 12 total, 12 up, 12 in 2024-09-16T05:22:20.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1366783419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:20.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: osdmap e791: 12 total, 12 up, 12 in 2024-09-16T05:22:20.964 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]: dispatch 2024-09-16T05:22:20.965 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]: dispatch 2024-09-16T05:22:20.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:20 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]: dispatch 2024-09-16T05:22:20.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:21.001 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:21.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:21.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:21.128 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:21.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: osdmap e790: 12 total, 12 up, 12 in 2024-09-16T05:22:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1366783419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: osdmap e791: 12 total, 12 up, 12 in 2024-09-16T05:22:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]: dispatch 2024-09-16T05:22:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]: dispatch 2024-09-16T05:22:21.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]: dispatch 2024-09-16T05:22:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]: dispatch 2024-09-16T05:22:21.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:20 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]: dispatch 2024-09-16T05:22:21.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: osdmap e790: 12 total, 12 up, 12 in 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1366783419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: osdmap e791: 12 total, 12 up, 12 in 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]: dispatch 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]: dispatch 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]: dispatch 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]: dispatch 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]: dispatch 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]: dispatch 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]: dispatch 2024-09-16T05:22:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]: dispatch 2024-09-16T05:22:21.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]: dispatch 2024-09-16T05:22:21.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]: dispatch 2024-09-16T05:22:21.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]: dispatch 2024-09-16T05:22:21.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]: dispatch 2024-09-16T05:22:21.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:20 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]: dispatch 2024-09-16T05:22:21.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:21.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:21.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:21.380 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:21.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:21.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:21.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:21.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:21.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:21.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:21.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:21.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-16T05:22:21.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:21.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:21.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:21.833 INFO:tasks.rados.rados.2.smithi204.stdout:3113: done (3 left) 2024-09-16T05:22:21.833 INFO:tasks.rados.rados.2.smithi204.stdout:3114: copy_from oid 8 from oid 9 current snap is 332 2024-09-16T05:22:21.833 INFO:tasks.rados.rados.2.smithi204.stdout:3115: write oid 44 current snap is 332 2024-09-16T05:22:21.833 INFO:tasks.rados.rados.2.smithi204.stdout:3115: seq_num 1092 ranges {753244=410387,1834572=518672} 2024-09-16T05:22:21.840 INFO:tasks.rados.rados.2.smithi204.stdout:3115: writing smithi20457400-44 from 753244 to 1163631 tid 1 2024-09-16T05:22:21.847 INFO:tasks.rados.rados.2.smithi204.stdout:3115: writing smithi20457400-44 from 1834572 to 2353244 tid 2 2024-09-16T05:22:21.847 INFO:tasks.rados.rados.2.smithi204.stdout:3116: copy_from oid 1 from oid 50 current snap is 332 2024-09-16T05:22:21.847 INFO:tasks.rados.rados.2.smithi204.stdout:3111: finishing rollback tid 0 to smithi20457400-3 2024-09-16T05:22:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:3111: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:22:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:3111: finishing rollback tid 2 to smithi20457400-3 2024-09-16T05:22:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1092 (ObjNum 993 snap 292 seq_num 993) dirty exists 2024-09-16T05:22:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:3110: finishing rollback tid 0 to smithi20457400-21 2024-09-16T05:22:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:3110: finishing rollback tid 1 to smithi20457400-21 2024-09-16T05:22:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 1046 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:22:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:3112: finishing write tid 1 to smithi20457400-10 2024-09-16T05:22:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:3112: finishing write tid 2 to smithi20457400-10 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3112: finishing write tid 3 to smithi20457400-10 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3112: finishing write tid 4 to smithi20457400-10 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1288 (ObjNum 1091 snap 332 seq_num 1091) dirty exists 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3112: left oid 10 (ObjNum 1091 snap 332 seq_num 1091) 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3114: finishing copy_from to smithi20457400-8 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3114: got expected ENOENT (src dne) 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3114: finishing copy_from racing read to smithi20457400-8 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3115: finishing write tid 1 to smithi20457400-44 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3110: done (5 left) 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3111: done (4 left) 2024-09-16T05:22:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:3112: done (3 left) 2024-09-16T05:22:21.850 INFO:tasks.rados.rados.2.smithi204.stdout:3114: done (2 left) 2024-09-16T05:22:21.850 INFO:tasks.rados.rados.2.smithi204.stdout:3117: read oid 26 snap -1 2024-09-16T05:22:21.850 INFO:tasks.rados.rados.2.smithi204.stdout:3117: expect (ObjNum 1081 snap 329 seq_num 1081) 2024-09-16T05:22:21.850 INFO:tasks.rados.rados.2.smithi204.stdout:3118: read oid 23 snap -1 2024-09-16T05:22:21.850 INFO:tasks.rados.rados.2.smithi204.stdout:3118: expect (ObjNum 1089 snap 330 seq_num 1089) 2024-09-16T05:22:21.850 INFO:tasks.rados.rados.2.smithi204.stdout:3119: delete oid 43 current snap is 332 2024-09-16T05:22:21.851 INFO:tasks.rados.rados.2.smithi204.stdout:3119: done (4 left) 2024-09-16T05:22:21.851 INFO:tasks.rados.rados.2.smithi204.stdout:3120: rollback oid 12 current snap is 332 2024-09-16T05:22:21.851 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 312 2024-09-16T05:22:21.851 INFO:tasks.rados.rados.2.smithi204.stdout:3121: write oid 32 current snap is 332 2024-09-16T05:22:21.851 INFO:tasks.rados.rados.2.smithi204.stdout:3121: seq_num 1094 ranges {786858=560885,1835183=551675} 2024-09-16T05:22:21.856 INFO:tasks.rados.rados.2.smithi204.stdout:3121: writing smithi20457400-32 from 786858 to 1347743 tid 1 2024-09-16T05:22:21.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:21.863 INFO:tasks.rados.rados.2.smithi204.stdout:3121: writing smithi20457400-32 from 1835183 to 2386858 tid 2 2024-09-16T05:22:21.863 INFO:tasks.rados.rados.2.smithi204.stdout:3122: copy_from oid 43 from oid 9 current snap is 332 2024-09-16T05:22:21.863 INFO:tasks.rados.rados.2.smithi204.stdout:3115: finishing write tid 2 to smithi20457400-44 2024-09-16T05:22:21.863 INFO:tasks.rados.rados.2.smithi204.stdout:3115: finishing write tid 3 to smithi20457400-44 2024-09-16T05:22:21.863 INFO:tasks.rados.rados.2.smithi204.stdout:3123: read oid 49 snap -1 2024-09-16T05:22:21.863 INFO:tasks.rados.rados.2.smithi204.stdout:3123: expect (ObjNum 921 snap 273 seq_num 921) 2024-09-16T05:22:21.864 INFO:tasks.rados.rados.2.smithi204.stdout:3124: snap_remove snap 305 2024-09-16T05:22:21.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:21.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:21.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:21.962 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:21.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:22.005 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:22.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:22.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:22.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: pgmap v1347: 127 pgs: 127 active+clean; 374 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 947 KiB/s wr, 7 op/s 2024-09-16T05:22:22.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]': finished 2024-09-16T05:22:22.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]': finished 2024-09-16T05:22:22.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]': finished 2024-09-16T05:22:22.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]': finished 2024-09-16T05:22:22.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]': finished 2024-09-16T05:22:22.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]': finished 2024-09-16T05:22:22.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]': finished 2024-09-16T05:22:22.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]': finished 2024-09-16T05:22:22.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]': finished 2024-09-16T05:22:22.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]': finished 2024-09-16T05:22:22.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]': finished 2024-09-16T05:22:22.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]': finished 2024-09-16T05:22:22.114 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:21 smithi032 ceph-mon[27388]: osdmap e792: 12 total, 12 up, 12 in 2024-09-16T05:22:22.130 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:22.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: pgmap v1347: 127 pgs: 127 active+clean; 374 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 947 KiB/s wr, 7 op/s 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]': finished 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]': finished 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]': finished 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]': finished 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]': finished 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]': finished 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]': finished 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]': finished 2024-09-16T05:22:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]': finished 2024-09-16T05:22:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]': finished 2024-09-16T05:22:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]': finished 2024-09-16T05:22:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]': finished 2024-09-16T05:22:22.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:21 smithi204 ceph-mon[30933]: osdmap e792: 12 total, 12 up, 12 in 2024-09-16T05:22:22.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:22.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: pgmap v1347: 127 pgs: 127 active+clean; 374 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 947 KiB/s wr, 7 op/s 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10802296587260853]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07805689189343366]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12408683295386297]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09363785770755097]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08976393282134769]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08886876292732462]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08160864599889076]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05981482852879262]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08052340201495736]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07520471790628946]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07257565248916453]}]': finished 2024-09-16T05:22:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09538189560452695]}]': finished 2024-09-16T05:22:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:21 smithi196 ceph-mon[30802]: osdmap e792: 12 total, 12 up, 12 in 2024-09-16T05:22:22.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:22.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:22.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:22.381 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:22.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:22.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:22.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:22.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:22.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:22.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:22.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:22.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:22:22.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:22.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:22.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:22.833 INFO:tasks.rados.rados.2.smithi204.stdout:3124: done (8 left) 2024-09-16T05:22:22.833 INFO:tasks.rados.rados.2.smithi204.stdout:3125: rollback oid 39 current snap is 332 2024-09-16T05:22:22.833 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 39 to 292 2024-09-16T05:22:22.833 INFO:tasks.rados.rados.2.smithi204.stdout:3126: write oid 36 current snap is 332 2024-09-16T05:22:22.834 INFO:tasks.rados.rados.2.smithi204.stdout:3126: seq_num 1095 ranges {403665=752487,1610018=514716} 2024-09-16T05:22:22.838 INFO:tasks.rados.rados.2.smithi204.stdout:3126: writing smithi20457400-36 from 403665 to 1156152 tid 1 2024-09-16T05:22:22.845 INFO:tasks.rados.rados.2.smithi204.stdout:3126: writing smithi20457400-36 from 1610018 to 2124734 tid 2 2024-09-16T05:22:22.846 INFO:tasks.rados.rados.2.smithi204.stdout:3127: read oid 45 snap 328 2024-09-16T05:22:22.846 INFO:tasks.rados.rados.2.smithi204.stdout:3127: expect (ObjNum 1038 snap 317 seq_num 1038) 2024-09-16T05:22:22.847 INFO:tasks.rados.rados.2.smithi204.stdout:3128: write oid 37 current snap is 332 2024-09-16T05:22:22.847 INFO:tasks.rados.rados.2.smithi204.stdout:3115: finishing write tid 4 to smithi20457400-44 2024-09-16T05:22:22.847 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1124 (ObjNum 1092 snap 332 seq_num 1092) dirty exists 2024-09-16T05:22:22.847 INFO:tasks.rados.rados.2.smithi204.stdout:3115: left oid 44 (ObjNum 1092 snap 332 seq_num 1092) 2024-09-16T05:22:22.847 INFO:tasks.rados.rados.2.smithi204.stdout:3120: finishing rollback tid 0 to smithi20457400-12 2024-09-16T05:22:22.847 INFO:tasks.rados.rados.2.smithi204.stdout:3120: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:22:22.848 INFO:tasks.rados.rados.2.smithi204.stdout:3120: finishing rollback tid 2 to smithi20457400-12 2024-09-16T05:22:22.848 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1204 (ObjNum 1013 snap 306 seq_num 1013) dirty exists 2024-09-16T05:22:22.848 INFO:tasks.rados.rados.2.smithi204.stdout:3117: expect (ObjNum 1081 snap 329 seq_num 1081) 2024-09-16T05:22:22.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:22.887 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:22.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:22.949 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:22.950 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:22.960 INFO:tasks.rados.rados.2.smithi204.stdout:3118: expect (ObjNum 1089 snap 330 seq_num 1089) 2024-09-16T05:22:22.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:23.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:23.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:23.126 INFO:tasks.rados.rados.2.smithi204.stdout:3121: finishing write tid 1 to smithi20457400-32 2024-09-16T05:22:23.126 INFO:tasks.rados.rados.2.smithi204.stdout:3116: finishing copy_from racing read to smithi20457400-1 2024-09-16T05:22:23.126 INFO:tasks.rados.rados.2.smithi204.stdout:3122: finishing copy_from to smithi20457400-43 2024-09-16T05:22:23.126 INFO:tasks.rados.rados.2.smithi204.stdout:3122: got expected ENOENT (src dne) 2024-09-16T05:22:23.126 INFO:tasks.rados.rados.2.smithi204.stdout:3116: finishing copy_from to smithi20457400-1 2024-09-16T05:22:23.126 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 1138 (ObjNum 1071 snap 327 seq_num 1071) dirty exists 2024-09-16T05:22:23.126 INFO:tasks.rados.rados.2.smithi204.stdout:3122: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:22:23.127 INFO:tasks.rados.rados.2.smithi204.stdout:3121: finishing write tid 2 to smithi20457400-32 2024-09-16T05:22:23.127 INFO:tasks.rados.rados.2.smithi204.stdout:3121: finishing write tid 3 to smithi20457400-32 2024-09-16T05:22:23.127 INFO:tasks.rados.rados.2.smithi204.stdout:3121: finishing write tid 4 to smithi20457400-32 2024-09-16T05:22:23.127 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1186 (ObjNum 1094 snap 332 seq_num 1094) dirty exists 2024-09-16T05:22:23.127 INFO:tasks.rados.rados.2.smithi204.stdout:3121: left oid 32 (ObjNum 1094 snap 332 seq_num 1094) 2024-09-16T05:22:23.127 INFO:tasks.rados.rados.2.smithi204.stdout:3123: expect (ObjNum 921 snap 273 seq_num 921) 2024-09-16T05:22:23.135 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:23.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:23.170 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:22:22 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:22:22.850+0000 7fecfac0d640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 133 after trimming was started 2024-09-16T05:22:23.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:23.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:23.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:23.353 INFO:tasks.rados.rados.2.smithi204.stdout:3125: finishing rollback tid 1 to smithi20457400-39 2024-09-16T05:22:23.354 INFO:tasks.rados.rados.2.smithi204.stdout:3125: finishing rollback tid 2 to smithi20457400-39 2024-09-16T05:22:23.354 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1268 (ObjNum 987 snap 292 seq_num 987) dirty exists 2024-09-16T05:22:23.354 INFO:tasks.rados.rados.2.smithi204.stdout:3126: finishing write tid 1 to smithi20457400-36 2024-09-16T05:22:23.354 INFO:tasks.rados.rados.2.smithi204.stdout:3126: finishing write tid 2 to smithi20457400-36 2024-09-16T05:22:23.354 INFO:tasks.rados.rados.2.smithi204.stdout:3126: finishing write tid 3 to smithi20457400-36 2024-09-16T05:22:23.354 INFO:tasks.rados.rados.2.smithi204.stdout:3128: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2024-09-16T05:22:23.357 INFO:tasks.rados.rados.2.smithi204.stdout:3128: writing smithi20457400-37 from 420472 to 848208 tid 1 2024-09-16T05:22:23.363 INFO:tasks.rados.rados.2.smithi204.stdout:3128: writing smithi20457400-37 from 1268500 to 1926874 tid 2 2024-09-16T05:22:23.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:23.368 INFO:tasks.rados.rados.2.smithi204.stdout:3128: writing smithi20457400-37 from 2347953 to 2420472 tid 3 2024-09-16T05:22:23.368 INFO:tasks.rados.rados.2.smithi204.stdout:3115: done (11 left) 2024-09-16T05:22:23.368 INFO:tasks.rados.rados.2.smithi204.stdout:3116: done (10 left) 2024-09-16T05:22:23.368 INFO:tasks.rados.rados.2.smithi204.stdout:3117: done (9 left) 2024-09-16T05:22:23.368 INFO:tasks.rados.rados.2.smithi204.stdout:3118: done (8 left) 2024-09-16T05:22:23.368 INFO:tasks.rados.rados.2.smithi204.stdout:3120: done (7 left) 2024-09-16T05:22:23.369 INFO:tasks.rados.rados.2.smithi204.stdout:3121: done (6 left) 2024-09-16T05:22:23.369 INFO:tasks.rados.rados.2.smithi204.stdout:3122: done (5 left) 2024-09-16T05:22:23.369 INFO:tasks.rados.rados.2.smithi204.stdout:3123: done (4 left) 2024-09-16T05:22:23.369 INFO:tasks.rados.rados.2.smithi204.stdout:3125: done (3 left) 2024-09-16T05:22:23.369 INFO:tasks.rados.rados.2.smithi204.stdout:3129: snap_remove snap 323 2024-09-16T05:22:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:23.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:23.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:23.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:23.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:23.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:23.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:23.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:22:23.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:23.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:23.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:23.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:23.652 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:23.660 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:23.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:23.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:23.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:23.832 INFO:tasks.rados.rados.2.smithi204.stdout:3129: done (3 left) 2024-09-16T05:22:23.832 INFO:tasks.rados.rados.2.smithi204.stdout:3130: rollback oid 38 current snap is 332 2024-09-16T05:22:23.832 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 38 to 279 2024-09-16T05:22:23.832 INFO:tasks.rados.rados.2.smithi204.stdout:3126: finishing write tid 4 to smithi20457400-36 2024-09-16T05:22:23.832 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 777 (ObjNum 1095 snap 332 seq_num 1095) dirty exists 2024-09-16T05:22:23.832 INFO:tasks.rados.rados.2.smithi204.stdout:3126: left oid 36 (ObjNum 1095 snap 332 seq_num 1095) 2024-09-16T05:22:23.832 INFO:tasks.rados.rados.2.smithi204.stdout:3127: expect (ObjNum 1038 snap 317 seq_num 1038) 2024-09-16T05:22:23.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:23.895 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:23.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:23.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:24.012 INFO:tasks.rados.rados.2.smithi204.stdout:3128: finishing write tid 1 to smithi20457400-37 2024-09-16T05:22:24.012 INFO:tasks.rados.rados.2.smithi204.stdout:3128: finishing write tid 2 to smithi20457400-37 2024-09-16T05:22:24.012 INFO:tasks.rados.rados.2.smithi204.stdout:3126: done (3 left) 2024-09-16T05:22:24.012 INFO:tasks.rados.rados.2.smithi204.stdout:3127: done (2 left) 2024-09-16T05:22:24.012 INFO:tasks.rados.rados.2.smithi204.stdout:3131: snap_remove snap 327 2024-09-16T05:22:24.018 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:24.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:24.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:24.132 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:23 smithi032 ceph-mon[27388]: pgmap v1350: 127 pgs: 127 active+clean; 369 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 911 KiB/s wr, 8 op/s 2024-09-16T05:22:24.132 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:23 smithi032 ceph-mon[27388]: osdmap e793: 12 total, 12 up, 12 in 2024-09-16T05:22:24.141 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:24.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:23 smithi204 ceph-mon[30933]: pgmap v1350: 127 pgs: 127 active+clean; 369 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 911 KiB/s wr, 8 op/s 2024-09-16T05:22:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:23 smithi204 ceph-mon[30933]: osdmap e793: 12 total, 12 up, 12 in 2024-09-16T05:22:24.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:23 smithi196 ceph-mon[30802]: pgmap v1350: 127 pgs: 127 active+clean; 369 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 911 KiB/s wr, 8 op/s 2024-09-16T05:22:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:23 smithi196 ceph-mon[30802]: osdmap e793: 12 total, 12 up, 12 in 2024-09-16T05:22:24.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:24.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:24.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:24.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:24.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:24.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:24.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:24.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:22:24.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:24.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:24.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:24.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:24.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:24.707 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:24.749 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:24.772 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:24.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:24.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:24.839 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:22:24.843 INFO:tasks.rados.rados.2.smithi204.stdout:3131: done (2 left) 2024-09-16T05:22:24.843 INFO:tasks.rados.rados.2.smithi204.stdout:3132: write oid 6 current snap is 332 2024-09-16T05:22:24.843 INFO:tasks.rados.rados.2.smithi204.stdout:3132: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-16T05:22:24.850 INFO:tasks.rados.rados.2.smithi204.stdout:3132: writing smithi20457400-6 from 437279 to 940264 tid 1 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3132: writing smithi20457400-6 from 1610629 to 2129014 tid 2 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3133: delete oid 12 current snap is 332 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3128: finishing write tid 3 to smithi20457400-37 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3128: finishing write tid 4 to smithi20457400-37 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3128: finishing write tid 5 to smithi20457400-37 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1048 (ObjNum 1096 snap 332 seq_num 1096) dirty exists 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3128: left oid 37 (ObjNum 1096 snap 332 seq_num 1096) 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3130: finishing rollback tid 0 to smithi20457400-38 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3130: finishing rollback tid 1 to smithi20457400-38 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 38 v 1048 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3132: finishing write tid 1 to smithi20457400-6 2024-09-16T05:22:24.863 INFO:tasks.rados.rados.2.smithi204.stdout:3132: finishing write tid 2 to smithi20457400-6 2024-09-16T05:22:24.864 INFO:tasks.rados.rados.2.smithi204.stdout:3132: finishing write tid 3 to smithi20457400-6 2024-09-16T05:22:24.864 INFO:tasks.rados.rados.2.smithi204.stdout:3132: finishing write tid 4 to smithi20457400-6 2024-09-16T05:22:24.864 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 1240 (ObjNum 1097 snap 332 seq_num 1097) dirty exists 2024-09-16T05:22:24.864 INFO:tasks.rados.rados.2.smithi204.stdout:3132: left oid 6 (ObjNum 1097 snap 332 seq_num 1097) 2024-09-16T05:22:24.880 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:24.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:24.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:25.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:25.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:25.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:25.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:25.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:24 smithi032 ceph-mon[27388]: osdmap e794: 12 total, 12 up, 12 in 2024-09-16T05:22:25.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/78074854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:22:25.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:25.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:25.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:24 smithi204 ceph-mon[30933]: osdmap e794: 12 total, 12 up, 12 in 2024-09-16T05:22:25.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/78074854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:22:25.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:24 smithi196 ceph-mon[30802]: osdmap e794: 12 total, 12 up, 12 in 2024-09-16T05:22:25.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/78074854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:22:25.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:25.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:25.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:25.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:25.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:25.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:25.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-09-16T05:22:25.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:25.535 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:25.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:25.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:25.659 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:25.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:25.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3128: done (3 left) 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3130: done (2 left) 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3132: done (1 left) 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3133: done (0 left) 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3134: read oid 16 snap 276 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3134: expect (ObjNum 914 snap 271 seq_num 914) 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3135: copy_from oid 4 from oid 14 current snap is 332 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3136: write oid 50 current snap is 332 2024-09-16T05:22:25.763 INFO:tasks.rados.rados.2.smithi204.stdout:3136: seq_num 1099 ranges {470893=653483,1727593=522054} 2024-09-16T05:22:25.768 INFO:tasks.rados.rados.2.smithi204.stdout:3136: writing smithi20457400-50 from 470893 to 1124376 tid 1 2024-09-16T05:22:25.775 INFO:tasks.rados.rados.2.smithi204.stdout:3136: writing smithi20457400-50 from 1727593 to 2249647 tid 2 2024-09-16T05:22:25.775 INFO:tasks.rados.rados.2.smithi204.stdout:3137: write oid 1 current snap is 332 2024-09-16T05:22:25.775 INFO:tasks.rados.rados.2.smithi204.stdout:3137: seq_num 1100 ranges {487700=728732,1786075=665712} 2024-09-16T05:22:25.781 INFO:tasks.rados.rados.2.smithi204.stdout:3137: writing smithi20457400-1 from 487700 to 1216432 tid 1 2024-09-16T05:22:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:25.789 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:25.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:25.789 INFO:tasks.rados.rados.2.smithi204.stdout:3137: writing smithi20457400-1 from 1786075 to 2451787 tid 2 2024-09-16T05:22:25.790 INFO:tasks.rados.rados.2.smithi204.stdout:3138: snap_create 2024-09-16T05:22:25.790 INFO:tasks.rados.rados.2.smithi204.stdout:3134: expect (ObjNum 914 snap 271 seq_num 914) 2024-09-16T05:22:25.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:25.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:25.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3135: finishing copy_from racing read to smithi20457400-4 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3136: finishing write tid 1 to smithi20457400-50 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3136: finishing write tid 2 to smithi20457400-50 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3136: finishing write tid 3 to smithi20457400-50 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3136: finishing write tid 4 to smithi20457400-50 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1141 (ObjNum 1099 snap 332 seq_num 1099) dirty exists 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3136: left oid 50 (ObjNum 1099 snap 332 seq_num 1099) 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3135: finishing copy_from to smithi20457400-4 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1353 (ObjNum 1080 snap 329 seq_num 1080) dirty exists 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3137: finishing write tid 1 to smithi20457400-1 2024-09-16T05:22:25.935 INFO:tasks.rados.rados.2.smithi204.stdout:3137: finishing write tid 2 to smithi20457400-1 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:3134: done (4 left) 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:3135: done (3 left) 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:3136: done (2 left) 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:3138: done (1 left) 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:3139: snap_create 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:3137: finishing write tid 3 to smithi20457400-1 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:3137: finishing write tid 4 to smithi20457400-1 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 1141 (ObjNum 1100 snap 332 seq_num 1100) dirty exists 2024-09-16T05:22:25.936 INFO:tasks.rados.rados.2.smithi204.stdout:3137: left oid 1 (ObjNum 1100 snap 332 seq_num 1100) 2024-09-16T05:22:25.968 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:25.969 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:25.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: pgmap v1353: 127 pgs: 1 active+clean+remapped, 126 active+clean; 387 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2024-09-16T05:22:25.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:25.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/78074854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:25.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: osdmap e795: 12 total, 12 up, 12 in 2024-09-16T05:22:25.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:22:25.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:22:25.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:22:25.969 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:22:25.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:22:25.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:22:25.970 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:25 smithi032 ceph-mon[27388]: osdmap e796: 12 total, 12 up, 12 in 2024-09-16T05:22:26.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:26.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:26.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:26.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:26.163 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:26.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: pgmap v1353: 127 pgs: 1 active+clean+remapped, 126 active+clean; 387 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2024-09-16T05:22:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/78074854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: osdmap e795: 12 total, 12 up, 12 in 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:22:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:25 smithi204 ceph-mon[30933]: osdmap e796: 12 total, 12 up, 12 in 2024-09-16T05:22:26.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: pgmap v1353: 127 pgs: 1 active+clean+remapped, 126 active+clean; 387 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/78074854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: osdmap e795: 12 total, 12 up, 12 in 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:22:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:25 smithi196 ceph-mon[30802]: osdmap e796: 12 total, 12 up, 12 in 2024-09-16T05:22:26.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:26.286 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:26.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:26.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:26.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:26.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:26.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:26.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:26.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:26.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:26.668 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:26.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:26.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:26.792 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:26.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:26.849 INFO:tasks.rados.rados.2.smithi204.stdout:3137: done (1 left) 2024-09-16T05:22:26.849 INFO:tasks.rados.rados.2.smithi204.stdout:3139: done (0 left) 2024-09-16T05:22:26.849 INFO:tasks.rados.rados.2.smithi204.stdout:3140: read oid 22 snap -1 2024-09-16T05:22:26.849 INFO:tasks.rados.rados.2.smithi204.stdout:3140: expect deleted 2024-09-16T05:22:26.849 INFO:tasks.rados.rados.2.smithi204.stdout:3141: delete oid 40 current snap is 334 2024-09-16T05:22:26.852 INFO:tasks.rados.rados.2.smithi204.stdout:3140: done (1 left) 2024-09-16T05:22:26.853 INFO:tasks.rados.rados.2.smithi204.stdout:3141: done (0 left) 2024-09-16T05:22:26.853 INFO:tasks.rados.rados.2.smithi204.stdout:3142: snap_create 2024-09-16T05:22:26.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:26.915 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:26.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:27.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:27.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:27.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:26 smithi032 ceph-mon[27388]: 2.50 scrub starts 2024-09-16T05:22:27.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:26 smithi032 ceph-mon[27388]: 2.50 scrub ok 2024-09-16T05:22:27.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:26 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:22:27.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:26 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:22:27.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:26 smithi032 ceph-mon[27388]: 1.1e scrub starts 2024-09-16T05:22:27.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:26 smithi032 ceph-mon[27388]: 1.1e scrub ok 2024-09-16T05:22:27.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:26 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:22:27.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:26 smithi032 ceph-mon[27388]: 2.29 scrub ok 2024-09-16T05:22:27.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:27.165 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:27.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:26 smithi204 ceph-mon[30933]: 2.50 scrub starts 2024-09-16T05:22:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:26 smithi204 ceph-mon[30933]: 2.50 scrub ok 2024-09-16T05:22:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:26 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:22:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:26 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:22:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:26 smithi204 ceph-mon[30933]: 1.1e scrub starts 2024-09-16T05:22:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:26 smithi204 ceph-mon[30933]: 1.1e scrub ok 2024-09-16T05:22:27.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:26 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:22:27.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:26 smithi204 ceph-mon[30933]: 2.29 scrub ok 2024-09-16T05:22:27.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:26 smithi196 ceph-mon[30802]: 2.50 scrub starts 2024-09-16T05:22:27.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:26 smithi196 ceph-mon[30802]: 2.50 scrub ok 2024-09-16T05:22:27.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:26 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:22:27.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:26 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:22:27.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:26 smithi196 ceph-mon[30802]: 1.1e scrub starts 2024-09-16T05:22:27.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:26 smithi196 ceph-mon[30802]: 1.1e scrub ok 2024-09-16T05:22:27.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:26 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:22:27.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:26 smithi196 ceph-mon[30802]: 2.29 scrub ok 2024-09-16T05:22:27.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:27.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:27.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:27.413 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:27.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:27.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:27.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:27.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:27.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:27.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:27.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:27.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:27.796 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:27.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:27.854 INFO:tasks.rados.rados.2.smithi204.stdout:3142: done (0 left) 2024-09-16T05:22:27.855 INFO:tasks.rados.rados.2.smithi204.stdout:3143: rollback oid 25 current snap is 335 2024-09-16T05:22:27.855 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 330 2024-09-16T05:22:27.855 INFO:tasks.rados.rados.2.smithi204.stdout:3144: delete oid 39 current snap is 335 2024-09-16T05:22:27.857 INFO:tasks.rados.rados.2.smithi204.stdout:3144: done (1 left) 2024-09-16T05:22:27.857 INFO:tasks.rados.rados.2.smithi204.stdout:3145: write (excl) oid 10 current snap is 335 2024-09-16T05:22:27.857 INFO:tasks.rados.rados.2.smithi204.stdout:3145: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-09-16T05:22:27.863 INFO:tasks.rados.rados.2.smithi204.stdout:3145: writing smithi20457400-10 from 538121 to 1208953 tid 1 2024-09-16T05:22:27.870 INFO:tasks.rados.rados.2.smithi204.stdout:3145: writing smithi20457400-10 from 1961521 to 2374560 tid 2 2024-09-16T05:22:27.870 INFO:tasks.rados.rados.2.smithi204.stdout:3146: read oid 15 snap 292 2024-09-16T05:22:27.870 INFO:tasks.rados.rados.2.smithi204.stdout:3146: expect (ObjNum 960 snap 283 seq_num 960) 2024-09-16T05:22:27.870 INFO:tasks.rados.rados.2.smithi204.stdout:3147: snap_remove snap 312 2024-09-16T05:22:27.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:27.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:28.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:28.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:28.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:28.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:28.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:22:28.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:22:28.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: pgmap v1356: 127 pgs: 1 peering, 126 active+clean; 385 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.5 MiB/s wr, 22 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:22:28.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:22:28.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:22:28.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 1.d scrub starts 2024-09-16T05:22:28.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 1.d scrub ok 2024-09-16T05:22:28.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 1.b scrub ok 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.2a scrub starts 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.2a scrub ok 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: osdmap e797: 12 total, 12 up, 12 in 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:22:28.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:22:28.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:22:28.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:22:28.150 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:27 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: pgmap v1356: 127 pgs: 1 peering, 126 active+clean; 385 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.5 MiB/s wr, 22 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 1.d scrub starts 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 1.d scrub ok 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:22:28.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:22:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:22:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 1.b scrub ok 2024-09-16T05:22:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:28.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.2a scrub starts 2024-09-16T05:22:28.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.2a scrub ok 2024-09-16T05:22:28.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:22:28.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: osdmap e797: 12 total, 12 up, 12 in 2024-09-16T05:22:28.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:22:28.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:22:28.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:22:28.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:22:28.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:22:28.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:27 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:22:28.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:22:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:28.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:28.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:22:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:22:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: pgmap v1356: 127 pgs: 1 peering, 126 active+clean; 385 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.5 MiB/s wr, 22 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:22:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:22:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:22:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 1.d scrub starts 2024-09-16T05:22:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 1.d scrub ok 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 1.b scrub ok 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.2a scrub starts 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.2a scrub ok 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: osdmap e797: 12 total, 12 up, 12 in 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:22:28.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:22:28.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:22:28.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:22:28.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:22:28.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:27 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:22:28.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:28.305 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:28.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:28.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:28.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:28.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:28.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:28.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:28.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:28.565 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:28.565 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:28.565 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:28.566 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:28.566 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:28.566 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:28.566 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:28.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:28.576 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:28.576 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:28.597 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:28.598 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:28.621 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:28.644 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:28.645 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:28.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:28.667 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:28.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:28.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:28.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:28.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:28.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:28.866 INFO:tasks.rados.rados.2.smithi204.stdout:3147: done (3 left) 2024-09-16T05:22:28.866 INFO:tasks.rados.rados.2.smithi204.stdout:3148: delete oid 31 current snap is 335 2024-09-16T05:22:28.866 INFO:tasks.rados.rados.2.smithi204.stdout:3143: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:22:28.866 INFO:tasks.rados.rados.2.smithi204.stdout:3143: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:22:28.866 INFO:tasks.rados.rados.2.smithi204.stdout:3143: finishing rollback tid 2 to smithi20457400-25 2024-09-16T05:22:28.866 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 752 (ObjNum 1076 snap 327 seq_num 1076) dirty exists 2024-09-16T05:22:28.866 INFO:tasks.rados.rados.2.smithi204.stdout:3145: finishing write tid 1 to smithi20457400-10 2024-09-16T05:22:28.866 INFO:tasks.rados.rados.2.smithi204.stdout:3145: finishing write tid 2 to smithi20457400-10 2024-09-16T05:22:28.867 INFO:tasks.rados.rados.2.smithi204.stdout:3146: expect (ObjNum 960 snap 283 seq_num 960) 2024-09-16T05:22:28.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:28.939 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:28.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:28.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:22:29.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:28 smithi032 ceph-mon[27388]: osdmap e798: 12 total, 12 up, 12 in 2024-09-16T05:22:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:28 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:22:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:28 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:22:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:28 smithi032 ceph-mon[27388]: 1.19 scrub starts 2024-09-16T05:22:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:28 smithi032 ceph-mon[27388]: 1.19 scrub ok 2024-09-16T05:22:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2445342640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:29.014 INFO:tasks.rados.rados.2.smithi204.stdout:3145: finishing write tid 3 to smithi20457400-10 2024-09-16T05:22:29.014 INFO:tasks.rados.rados.2.smithi204.stdout:3145: finishing write tid 4 to smithi20457400-10 2024-09-16T05:22:29.014 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1301 (ObjNum 1103 snap 335 seq_num 1103) dirty exists 2024-09-16T05:22:29.014 INFO:tasks.rados.rados.2.smithi204.stdout:3145: left oid 10 (ObjNum 1103 snap 335 seq_num 1103) 2024-09-16T05:22:29.014 INFO:tasks.rados.rados.2.smithi204.stdout:3143: done (3 left) 2024-09-16T05:22:29.014 INFO:tasks.rados.rados.2.smithi204.stdout:3145: done (2 left) 2024-09-16T05:22:29.014 INFO:tasks.rados.rados.2.smithi204.stdout:3146: done (1 left) 2024-09-16T05:22:29.015 INFO:tasks.rados.rados.2.smithi204.stdout:3148: done (0 left) 2024-09-16T05:22:29.015 INFO:tasks.rados.rados.2.smithi204.stdout:3149: write oid 32 current snap is 335 2024-09-16T05:22:29.015 INFO:tasks.rados.rados.2.smithi204.stdout:3149: seq_num 1105 ranges {571735=421330,1562132=416708} 2024-09-16T05:22:29.019 INFO:tasks.rados.rados.2.smithi204.stdout:3149: writing smithi20457400-32 from 571735 to 993065 tid 1 2024-09-16T05:22:29.025 INFO:tasks.rados.rados.2.smithi204.stdout:3149: writing smithi20457400-32 from 1562132 to 1978840 tid 2 2024-09-16T05:22:29.025 INFO:tasks.rados.rados.2.smithi204.stdout:3150: write (excl) oid 36 current snap is 335 2024-09-16T05:22:29.025 INFO:tasks.rados.rados.2.smithi204.stdout:3149: finishing write tid 1 to smithi20457400-32 2024-09-16T05:22:29.025 INFO:tasks.rados.rados.2.smithi204.stdout:3150: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-09-16T05:22:29.030 INFO:tasks.rados.rados.2.smithi204.stdout:3150: writing smithi20457400-36 from 588542 to 1085121 tid 1 2024-09-16T05:22:29.037 INFO:tasks.rados.rados.2.smithi204.stdout:3150: writing smithi20457400-36 from 1620614 to 2297333 tid 2 2024-09-16T05:22:29.037 INFO:tasks.rados.rados.2.smithi204.stdout:3151: snap_remove snap 332 2024-09-16T05:22:29.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:29.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:29.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:29.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:28 smithi204 ceph-mon[30933]: osdmap e798: 12 total, 12 up, 12 in 2024-09-16T05:22:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:28 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:22:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:28 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:22:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:28 smithi204 ceph-mon[30933]: 1.19 scrub starts 2024-09-16T05:22:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:28 smithi204 ceph-mon[30933]: 1.19 scrub ok 2024-09-16T05:22:29.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2445342640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:29.191 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:29.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:28 smithi196 ceph-mon[30802]: osdmap e798: 12 total, 12 up, 12 in 2024-09-16T05:22:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:28 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:22:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:28 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:22:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:28 smithi196 ceph-mon[30802]: 1.19 scrub starts 2024-09-16T05:22:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:28 smithi196 ceph-mon[30802]: 1.19 scrub ok 2024-09-16T05:22:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2445342640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:29.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:29.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:29.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:29.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:29.447 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:29.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:29.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:29.570 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:29.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:29.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:29.696 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:29.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:29.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:29.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:29.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:29.868 INFO:tasks.rados.rados.2.smithi204.stdout:3151: done (2 left) 2024-09-16T05:22:29.868 INFO:tasks.rados.rados.2.smithi204.stdout:3152: read oid 7 snap 331 2024-09-16T05:22:29.868 INFO:tasks.rados.rados.2.smithi204.stdout:3152: expect (ObjNum 1065 snap 324 seq_num 1065) 2024-09-16T05:22:29.868 INFO:tasks.rados.rados.2.smithi204.stdout:3153: read oid 27 snap 328 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3153: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3154: rollback oid 9 current snap is 335 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 9 to 328 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3149: finishing write tid 2 to smithi20457400-32 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3149: finishing write tid 3 to smithi20457400-32 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3149: finishing write tid 4 to smithi20457400-32 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1198 (ObjNum 1105 snap 335 seq_num 1105) dirty exists 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3149: left oid 32 (ObjNum 1105 snap 335 seq_num 1105) 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3150: finishing write tid 1 to smithi20457400-36 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3150: finishing write tid 2 to smithi20457400-36 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3150: finishing write tid 3 to smithi20457400-36 2024-09-16T05:22:29.869 INFO:tasks.rados.rados.2.smithi204.stdout:3150: finishing write tid 4 to smithi20457400-36 2024-09-16T05:22:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 780 (ObjNum 1106 snap 335 seq_num 1106) dirty exists 2024-09-16T05:22:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:3150: left oid 36 (ObjNum 1106 snap 335 seq_num 1106) 2024-09-16T05:22:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:3149: done (4 left) 2024-09-16T05:22:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:3150: done (3 left) 2024-09-16T05:22:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:3155: read oid 26 snap 333 2024-09-16T05:22:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:3155: expect (ObjNum 1081 snap 329 seq_num 1081) 2024-09-16T05:22:29.870 INFO:tasks.rados.rados.2.smithi204.stdout:3156: delete oid 18 current snap is 335 2024-09-16T05:22:29.871 INFO:tasks.rados.rados.2.smithi204.stdout:3156: done (4 left) 2024-09-16T05:22:29.871 INFO:tasks.rados.rados.2.smithi204.stdout:3157: snap_create 2024-09-16T05:22:29.878 INFO:tasks.rados.rados.2.smithi204.stdout:3154: finishing rollback tid 1 to smithi20457400-9 2024-09-16T05:22:29.879 INFO:tasks.rados.rados.2.smithi204.stdout:3154: finishing rollback tid 2 to smithi20457400-9 2024-09-16T05:22:29.880 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1309 (ObjNum 985 snap 289 seq_num 985) dirty exists 2024-09-16T05:22:29.881 INFO:tasks.rados.rados.2.smithi204.stdout:3155: expect (ObjNum 1081 snap 329 seq_num 1081) 2024-09-16T05:22:29.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:29.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:29.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:29.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:29.993 INFO:tasks.rados.rados.2.smithi204.stdout:3152: expect (ObjNum 1065 snap 324 seq_num 1065) 2024-09-16T05:22:30.025 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:29 smithi032 ceph-mon[27388]: pgmap v1359: 127 pgs: 1 peering, 126 active+clean; 377 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.1 MiB/s wr, 18 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:22:30.025 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:29 smithi032 ceph-mon[27388]: 2.2e scrub starts 2024-09-16T05:22:30.025 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:29 smithi032 ceph-mon[27388]: 2.2e scrub ok 2024-09-16T05:22:30.025 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:29 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:22:30.025 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:29 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:22:30.025 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:29 smithi032 ceph-mon[27388]: osdmap e799: 12 total, 12 up, 12 in 2024-09-16T05:22:30.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-09-16T05:22:30.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:30.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:30.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:30.123 INFO:tasks.rados.rados.2.smithi204.stdout:3153: expect (ObjNum 847 snap 246 seq_num 847) 2024-09-16T05:22:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:29 smithi204 ceph-mon[30933]: pgmap v1359: 127 pgs: 1 peering, 126 active+clean; 377 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.1 MiB/s wr, 18 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:22:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:29 smithi204 ceph-mon[30933]: 2.2e scrub starts 2024-09-16T05:22:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:29 smithi204 ceph-mon[30933]: 2.2e scrub ok 2024-09-16T05:22:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:29 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:22:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:29 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:22:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:29 smithi204 ceph-mon[30933]: osdmap e799: 12 total, 12 up, 12 in 2024-09-16T05:22:30.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:30.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:30.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:30.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:29 smithi196 ceph-mon[30802]: pgmap v1359: 127 pgs: 1 peering, 126 active+clean; 377 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.1 MiB/s wr, 18 op/s; 23 KiB/s, 0 objects/s recovering 2024-09-16T05:22:30.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:29 smithi196 ceph-mon[30802]: 2.2e scrub starts 2024-09-16T05:22:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:29 smithi196 ceph-mon[30802]: 2.2e scrub ok 2024-09-16T05:22:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:29 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:22:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:29 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:22:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:29 smithi196 ceph-mon[30802]: osdmap e799: 12 total, 12 up, 12 in 2024-09-16T05:22:30.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:30.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:30.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:30.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:30.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:30.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:30.497 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:22:30.497 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:22:30.497 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:22:30.497 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:22:30.497 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:22:30.499 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:30.545 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:30.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:30.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:30.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:30.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:30.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:30.795 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:30.817 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:30.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:30.830 INFO:tasks.rados.rados.2.smithi204.stdout:3152: done (4 left) 2024-09-16T05:22:30.830 INFO:tasks.rados.rados.2.smithi204.stdout:3153: done (3 left) 2024-09-16T05:22:30.830 INFO:tasks.rados.rados.2.smithi204.stdout:3154: done (2 left) 2024-09-16T05:22:30.830 INFO:tasks.rados.rados.2.smithi204.stdout:3155: done (1 left) 2024-09-16T05:22:30.830 INFO:tasks.rados.rados.2.smithi204.stdout:3157: done (0 left) 2024-09-16T05:22:30.830 INFO:tasks.rados.rados.2.smithi204.stdout:3158: write oid 26 current snap is 336 2024-09-16T05:22:30.830 INFO:tasks.rados.rados.2.smithi204.stdout:3158: seq_num 1108 ranges {622156=647077,1737578=680388} 2024-09-16T05:22:30.836 INFO:tasks.rados.rados.2.smithi204.stdout:3158: writing smithi20457400-26 from 622156 to 1269233 tid 1 2024-09-16T05:22:30.843 INFO:tasks.rados.rados.2.smithi204.stdout:3158: writing smithi20457400-26 from 1737578 to 2417966 tid 2 2024-09-16T05:22:30.843 INFO:tasks.rados.rados.2.smithi204.stdout:3159: snap_remove snap 330 2024-09-16T05:22:30.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:30.943 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:30.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:31.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:31.068 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:31.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:31.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:30 smithi032 ceph-mon[27388]: osdmap e800: 12 total, 12 up, 12 in 2024-09-16T05:22:31.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:30 smithi032 ceph-mon[27388]: osdmap e801: 12 total, 12 up, 12 in 2024-09-16T05:22:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:30 smithi204 ceph-mon[30933]: osdmap e800: 12 total, 12 up, 12 in 2024-09-16T05:22:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:30 smithi204 ceph-mon[30933]: osdmap e801: 12 total, 12 up, 12 in 2024-09-16T05:22:31.193 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:31.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:31.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:30 smithi196 ceph-mon[30802]: osdmap e800: 12 total, 12 up, 12 in 2024-09-16T05:22:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:30 smithi196 ceph-mon[30802]: osdmap e801: 12 total, 12 up, 12 in 2024-09-16T05:22:31.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:31.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:31.319 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:31.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:31.444 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:31.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:31.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:22:31.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:22:31.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:31.597 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:31.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:31.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:31.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:31.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:31.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:31.833 INFO:tasks.rados.rados.2.smithi204.stdout:3159: done (1 left) 2024-09-16T05:22:31.833 INFO:tasks.rados.rados.2.smithi204.stdout:3160: copy_from oid 45 from oid 6 current snap is 336 2024-09-16T05:22:31.833 INFO:tasks.rados.rados.2.smithi204.stdout:3161: write oid 32 current snap is 336 2024-09-16T05:22:31.833 INFO:tasks.rados.rados.2.smithi204.stdout:3161: seq_num 1109 ranges {638963=722326,2079707=424046} 2024-09-16T05:22:31.840 INFO:tasks.rados.rados.2.smithi204.stdout:3161: writing smithi20457400-32 from 638963 to 1361289 tid 1 2024-09-16T05:22:31.847 INFO:tasks.rados.rados.2.smithi204.stdout:3161: writing smithi20457400-32 from 2079707 to 2503753 tid 2 2024-09-16T05:22:31.847 INFO:tasks.rados.rados.2.smithi204.stdout:3162: write oid 43 current snap is 336 2024-09-16T05:22:31.848 INFO:tasks.rados.rados.2.smithi204.stdout:3162: seq_num 1110 ranges {655770=513928,1854542=684057} 2024-09-16T05:22:31.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:31.853 INFO:tasks.rados.rados.2.smithi204.stdout:3162: writing smithi20457400-43 from 655770 to 1169698 tid 1 2024-09-16T05:22:31.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:31.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:31.861 INFO:tasks.rados.rados.2.smithi204.stdout:3162: writing smithi20457400-43 from 1854542 to 2538599 tid 2 2024-09-16T05:22:31.861 INFO:tasks.rados.rados.2.smithi204.stdout:3163: read oid 30 snap -1 2024-09-16T05:22:31.861 INFO:tasks.rados.rados.2.smithi204.stdout:3163: expect (ObjNum 1051 snap 318 seq_num 1051) 2024-09-16T05:22:31.861 INFO:tasks.rados.rados.2.smithi204.stdout:3164: write oid 20 current snap is 336 2024-09-16T05:22:31.861 INFO:tasks.rados.rados.2.smithi204.stdout:3164: seq_num 1111 ranges {672577=589177,1913024=427715} 2024-09-16T05:22:31.868 INFO:tasks.rados.rados.2.smithi204.stdout:3164: writing smithi20457400-20 from 672577 to 1261754 tid 1 2024-09-16T05:22:31.876 INFO:tasks.rados.rados.2.smithi204.stdout:3164: writing smithi20457400-20 from 1913024 to 2340739 tid 2 2024-09-16T05:22:31.876 INFO:tasks.rados.rados.2.smithi204.stdout:3165: snap_create 2024-09-16T05:22:31.876 INFO:tasks.rados.rados.2.smithi204.stdout:3158: finishing write tid 1 to smithi20457400-26 2024-09-16T05:22:31.876 INFO:tasks.rados.rados.2.smithi204.stdout:3158: finishing write tid 2 to smithi20457400-26 2024-09-16T05:22:31.876 INFO:tasks.rados.rados.2.smithi204.stdout:3158: finishing write tid 3 to smithi20457400-26 2024-09-16T05:22:31.876 INFO:tasks.rados.rados.2.smithi204.stdout:3158: finishing write tid 4 to smithi20457400-26 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1383 (ObjNum 1108 snap 336 seq_num 1108) dirty exists 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3158: left oid 26 (ObjNum 1108 snap 336 seq_num 1108) 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3161: finishing write tid 1 to smithi20457400-32 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3160: finishing copy_from racing read to smithi20457400-45 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3160: finishing copy_from to smithi20457400-45 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1240 (ObjNum 1097 snap 332 seq_num 1097) dirty exists 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3161: finishing write tid 2 to smithi20457400-32 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3161: finishing write tid 3 to smithi20457400-32 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3161: finishing write tid 4 to smithi20457400-32 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1206 (ObjNum 1109 snap 336 seq_num 1109) dirty exists 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3161: left oid 32 (ObjNum 1109 snap 336 seq_num 1109) 2024-09-16T05:22:31.877 INFO:tasks.rados.rados.2.smithi204.stdout:3162: finishing write tid 1 to smithi20457400-43 2024-09-16T05:22:31.878 INFO:tasks.rados.rados.2.smithi204.stdout:3162: finishing write tid 2 to smithi20457400-43 2024-09-16T05:22:31.878 INFO:tasks.rados.rados.2.smithi204.stdout:3162: finishing write tid 3 to smithi20457400-43 2024-09-16T05:22:31.878 INFO:tasks.rados.rados.2.smithi204.stdout:3162: finishing write tid 4 to smithi20457400-43 2024-09-16T05:22:31.878 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1051 (ObjNum 1110 snap 336 seq_num 1110) dirty exists 2024-09-16T05:22:31.878 INFO:tasks.rados.rados.2.smithi204.stdout:3162: left oid 43 (ObjNum 1110 snap 336 seq_num 1110) 2024-09-16T05:22:31.878 INFO:tasks.rados.rados.2.smithi204.stdout:3164: finishing write tid 1 to smithi20457400-20 2024-09-16T05:22:31.878 INFO:tasks.rados.rados.2.smithi204.stdout:3163: expect (ObjNum 1051 snap 318 seq_num 1051) 2024-09-16T05:22:31.898 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:31.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:31.977 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:31.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:32.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:32.089 INFO:tasks.rados.rados.2.smithi204.stdout:3164: finishing write tid 2 to smithi20457400-20 2024-09-16T05:22:32.089 INFO:tasks.rados.rados.2.smithi204.stdout:3164: finishing write tid 3 to smithi20457400-20 2024-09-16T05:22:32.090 INFO:tasks.rados.rados.2.smithi204.stdout:3164: finishing write tid 4 to smithi20457400-20 2024-09-16T05:22:32.090 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1290 (ObjNum 1111 snap 336 seq_num 1111) dirty exists 2024-09-16T05:22:32.090 INFO:tasks.rados.rados.2.smithi204.stdout:3164: left oid 20 (ObjNum 1111 snap 336 seq_num 1111) 2024-09-16T05:22:32.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:32.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:31 smithi032 ceph-mon[27388]: pgmap v1362: 127 pgs: 2 active+clean+snaptrim, 125 active+clean; 367 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-16T05:22:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:31 smithi032 ceph-mon[27388]: 1.18 scrub starts 2024-09-16T05:22:32.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:31 smithi032 ceph-mon[27388]: 1.18 scrub ok 2024-09-16T05:22:32.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:31 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:22:32.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:31 smithi032 ceph-mon[27388]: osdmap e802: 12 total, 12 up, 12 in 2024-09-16T05:22:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:31 smithi204 ceph-mon[30933]: pgmap v1362: 127 pgs: 2 active+clean+snaptrim, 125 active+clean; 367 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-16T05:22:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:31 smithi204 ceph-mon[30933]: 1.18 scrub starts 2024-09-16T05:22:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:31 smithi204 ceph-mon[30933]: 1.18 scrub ok 2024-09-16T05:22:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:31 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:22:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:31 smithi204 ceph-mon[30933]: osdmap e802: 12 total, 12 up, 12 in 2024-09-16T05:22:32.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:32.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:32.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:32.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:31 smithi196 ceph-mon[30802]: pgmap v1362: 127 pgs: 2 active+clean+snaptrim, 125 active+clean; 367 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-09-16T05:22:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:31 smithi196 ceph-mon[30802]: 1.18 scrub starts 2024-09-16T05:22:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:31 smithi196 ceph-mon[30802]: 1.18 scrub ok 2024-09-16T05:22:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:31 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-16T05:22:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:31 smithi196 ceph-mon[30802]: osdmap e802: 12 total, 12 up, 12 in 2024-09-16T05:22:32.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:32.348 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:32.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:32.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:32.479 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:32.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:32.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:32.604 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:32.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:32.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:32.612 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:32.651 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:32.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:32.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:32.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:32.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:32.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:32.874 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:22:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:3158: done (6 left) 2024-09-16T05:22:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:3160: done (5 left) 2024-09-16T05:22:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:3161: done (4 left) 2024-09-16T05:22:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:3162: done (3 left) 2024-09-16T05:22:32.879 INFO:tasks.rados.rados.2.smithi204.stdout:3163: done (2 left) 2024-09-16T05:22:32.880 INFO:tasks.rados.rados.2.smithi204.stdout:3164: done (1 left) 2024-09-16T05:22:32.880 INFO:tasks.rados.rados.2.smithi204.stdout:3165: done (0 left) 2024-09-16T05:22:32.880 INFO:tasks.rados.rados.2.smithi204.stdout:3166: read oid 18 snap 331 2024-09-16T05:22:32.880 INFO:tasks.rados.rados.2.smithi204.stdout:3166: expect deleted 2024-09-16T05:22:32.880 INFO:tasks.rados.rados.2.smithi204.stdout:3167: write (excl) oid 46 current snap is 337 2024-09-16T05:22:32.880 INFO:tasks.rados.rados.2.smithi204.stdout:3167: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-09-16T05:22:32.886 INFO:tasks.rados.rados.2.smithi204.stdout:3167: writing smithi20457400-46 from 689384 to 1353810 tid 1 2024-09-16T05:22:32.893 INFO:tasks.rados.rados.2.smithi204.stdout:3167: writing smithi20457400-46 from 1971506 to 2659232 tid 2 2024-09-16T05:22:32.893 INFO:tasks.rados.rados.2.smithi204.stdout:3168: write oid 25 current snap is 337 2024-09-16T05:22:32.893 INFO:tasks.rados.rados.2.smithi204.stdout:3168: seq_num 1113 ranges {706191=739675,1913635=431384} 2024-09-16T05:22:32.901 INFO:tasks.rados.rados.2.smithi204.stdout:3168: writing smithi20457400-25 from 706191 to 1445866 tid 1 2024-09-16T05:22:32.906 INFO:tasks.rados.rados.2.smithi204.stdout:3168: writing smithi20457400-25 from 1913635 to 2345019 tid 2 2024-09-16T05:22:32.906 INFO:tasks.rados.rados.2.smithi204.stdout:3169: delete oid 30 current snap is 337 2024-09-16T05:22:32.906 INFO:tasks.rados.rados.2.smithi204.stdout:3167: finishing write tid 1 to smithi20457400-46 2024-09-16T05:22:32.907 INFO:tasks.rados.rados.2.smithi204.stdout:3167: finishing write tid 2 to smithi20457400-46 2024-09-16T05:22:32.907 INFO:tasks.rados.rados.2.smithi204.stdout:3167: finishing write tid 3 to smithi20457400-46 2024-09-16T05:22:32.907 INFO:tasks.rados.rados.2.smithi204.stdout:3167: finishing write tid 4 to smithi20457400-46 2024-09-16T05:22:32.907 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1020 (ObjNum 1112 snap 337 seq_num 1112) dirty exists 2024-09-16T05:22:32.907 INFO:tasks.rados.rados.2.smithi204.stdout:3167: left oid 46 (ObjNum 1112 snap 337 seq_num 1112) 2024-09-16T05:22:32.910 INFO:tasks.rados.rados.2.smithi204.stdout:3167: done (3 left) 2024-09-16T05:22:32.910 INFO:tasks.rados.rados.2.smithi204.stdout:3169: done (2 left) 2024-09-16T05:22:32.910 INFO:tasks.rados.rados.2.smithi204.stdout:3170: write oid 22 current snap is 337 2024-09-16T05:22:32.910 INFO:tasks.rados.rados.2.smithi204.stdout:3170: seq_num 1115 ranges {739805=490173,1630599=435053,2636284=103521} 2024-09-16T05:22:32.917 INFO:tasks.rados.rados.2.smithi204.stdout:3170: writing smithi20457400-22 from 739805 to 1229978 tid 1 2024-09-16T05:22:32.921 INFO:tasks.rados.rados.2.smithi204.stdout:3170: writing smithi20457400-22 from 1630599 to 2065652 tid 2 2024-09-16T05:22:32.925 INFO:tasks.rados.rados.2.smithi204.stdout:3170: writing smithi20457400-22 from 2636284 to 2739805 tid 3 2024-09-16T05:22:32.926 INFO:tasks.rados.rados.2.smithi204.stdout:3171: write (excl) oid 46 current snap is 337 2024-09-16T05:22:32.926 INFO:tasks.rados.rados.2.smithi204.stdout:3170: finishing write tid 1 to smithi20457400-22 2024-09-16T05:22:32.926 INFO:tasks.rados.rados.2.smithi204.stdout:3171: seq_num 1116 ranges {756612=565422,2089081=578711} 2024-09-16T05:22:32.932 INFO:tasks.rados.rados.2.smithi204.stdout:3171: writing smithi20457400-46 from 756612 to 1322034 tid 1 2024-09-16T05:22:32.940 INFO:tasks.rados.rados.2.smithi204.stdout:3171: writing smithi20457400-46 from 2089081 to 2667792 tid 2 2024-09-16T05:22:32.940 INFO:tasks.rados.rados.2.smithi204.stdout:3172: rollback oid 10 current snap is 337 2024-09-16T05:22:32.940 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 331 2024-09-16T05:22:32.940 INFO:tasks.rados.rados.2.smithi204.stdout:3170: finishing write tid 2 to smithi20457400-22 2024-09-16T05:22:32.940 INFO:tasks.rados.rados.2.smithi204.stdout:3170: finishing write tid 3 to smithi20457400-22 2024-09-16T05:22:32.940 INFO:tasks.rados.rados.2.smithi204.stdout:3170: finishing write tid 4 to smithi20457400-22 2024-09-16T05:22:32.940 INFO:tasks.rados.rados.2.smithi204.stdout:3171: finishing write tid 1 to smithi20457400-46 2024-09-16T05:22:32.941 INFO:tasks.rados.rados.2.smithi204.stdout:3173: write (excl) oid 41 current snap is 337 2024-09-16T05:22:32.941 INFO:tasks.rados.rados.2.smithi204.stdout:3173: seq_num 1117 ranges {773419=640671,2147563=438722} 2024-09-16T05:22:32.947 INFO:tasks.rados.rados.2.smithi204.stdout:3173: writing smithi20457400-41 from 773419 to 1414090 tid 1 2024-09-16T05:22:32.954 INFO:tasks.rados.rados.2.smithi204.stdout:3173: writing smithi20457400-41 from 2147563 to 2586285 tid 2 2024-09-16T05:22:32.954 INFO:tasks.rados.rados.2.smithi204.stdout:3174: read oid 47 snap -1 2024-09-16T05:22:32.954 INFO:tasks.rados.rados.2.smithi204.stdout:3174: expect deleted 2024-09-16T05:22:32.954 INFO:tasks.rados.rados.2.smithi204.stdout:3175: snap_remove snap 333 2024-09-16T05:22:32.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:32.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:32.980 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:33.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:33.104 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:33.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:32 smithi032 ceph-mon[27388]: 2.3c scrub starts 2024-09-16T05:22:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:32 smithi032 ceph-mon[27388]: 2.3c scrub ok 2024-09-16T05:22:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:32 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:22:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:32 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:22:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:32 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:22:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:32 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:22:33.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:32 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4192431711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:32 smithi204 ceph-mon[30933]: 2.3c scrub starts 2024-09-16T05:22:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:32 smithi204 ceph-mon[30933]: 2.3c scrub ok 2024-09-16T05:22:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:32 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:22:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:32 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:22:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:32 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:22:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:32 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:22:33.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:32 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4192431711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:33.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:33.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:33.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:33.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:32 smithi196 ceph-mon[30802]: 2.3c scrub starts 2024-09-16T05:22:33.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:32 smithi196 ceph-mon[30802]: 2.3c scrub ok 2024-09-16T05:22:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:32 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:22:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:32 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:22:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:32 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:22:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:32 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:22:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:32 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4192431711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:33.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:33.356 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:33.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:33.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:33.482 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:33.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:33.525 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:22:33.525 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:22:33.525 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:22:33.525 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:606 2.48 e365/322 2:12e92f6c:::smithi20457400-1:79 v 365'161, mlcod=362'157)", 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 438.45186855399999, 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64187130800000003, 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:22:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071139+0000", 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5609999999999999e-06 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712578+0000", 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64143852199999996 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712582+0000", 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-06 2024-09-16T05:22:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712651+0000", 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9102000000000003e-05 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712670+0000", 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9165999999999999e-05 2024-09-16T05:22:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712968+0000", 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029722899999999998 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.713007+0000", 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9125999999999998e-05 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.531 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:614 2.48 e366/322 2:12e92f6c:::smithi20457400-1:79 v 366'163, mlcod=364'159)", 2024-09-16T05:22:33.531 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:22:33.531 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 437.80244132399997, 2024-09-16T05:22:33.531 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11945663500000001, 2024-09-16T05:22:33.531 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.531 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.532 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.532 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.532 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.532 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:22:33.532 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9699999999999996e-07 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720568+0000", 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6870000000000002e-06 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839371+0000", 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11880265399999999 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839384+0000", 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3156000000000001e-05 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839531+0000", 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014728399999999999 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839575+0000", 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4223e-05 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839981+0000", 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040524499999999999 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.840019+0000", 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8788999999999997e-05 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4629 2.1e 2:7b642727:::smithi20457400-40:head [copy-from ver 511] snapc bf=[bf,be] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:22:33.536 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 291.77047980600003, 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01647931, 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4629 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752526+0000", 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.894e-06 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752536+0000", 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:22:33.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752555+0000", 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8907999999999999e-05 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752564+0000", 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752638+0000", 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3343999999999994e-05 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752646+0000", 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7690000000000002e-06 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754646+0000", 2024-09-16T05:22:33.541 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001999331 2024-09-16T05:22:33.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.542 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754740+0000", 2024-09-16T05:22:33.542 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3778999999999998e-05 2024-09-16T05:22:33.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.542 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:22:33.543 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.764207+0000", 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094672899999999997 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768754+0000", 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045470199999999997 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768860+0000", 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010586600000000001 2024-09-16T05:22:33.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768948+0000", 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8503000000000003e-05 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.769004+0000", 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5362000000000003e-05 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4630 2.1e 2:7b642727:::smithi20457400-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 291.77042205599997, 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015483190000000001, 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:22:33.546 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4630 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:22:33.547 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.34e-07 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752584+0000", 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.956e-06 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752591+0000", 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2580000000000001e-06 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752594+0000", 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6830000000000001e-06 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.549 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752703+0000", 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108818 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752708+0000", 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.561e-06 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754913+0000", 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022055680000000002 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754965+0000", 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2222000000000002e-05 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767827+0000", 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012861097 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767929+0000", 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010275799999999999 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768065+0000", 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013593500000000001 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:142 2.1e e571/291 ondisk, result = 0)", 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 231.758845901, 2024-09-16T05:22:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55541789799999997, 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:22:33.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.555 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.555 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764159+0000", 2024-09-16T05:22:33.555 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764162+0000", 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599999999999998e-06 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319278+0000", 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55511595899999999 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319296+0000", 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7900000000000001e-05 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319440+0000", 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143175 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319450+0000", 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0705e-05 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319576+0000", 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012584299999999999 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.558 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5587 2.34 e583/278 2:2d3f13f8:::smithi20457400-7:head v 583'466, mlcod=583'463)", 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 220.71589954999999, 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14534661199999999, 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:22:33.559 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.560 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:22:33.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:22:33.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.561 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.561 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807109+0000", 2024-09-16T05:22:33.561 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2030000000000002e-06 2024-09-16T05:22:33.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951917+0000", 2024-09-16T05:22:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14480800699999999 2024-09-16T05:22:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951927+0000", 2024-09-16T05:22:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0115000000000001e-05 2024-09-16T05:22:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.563 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951990+0000", 2024-09-16T05:22:33.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2689999999999998e-05 2024-09-16T05:22:33.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.564 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.564 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952011+0000", 2024-09-16T05:22:33.564 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.084e-05 2024-09-16T05:22:33.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.565 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952413+0000", 2024-09-16T05:22:33.565 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040190199999999999 2024-09-16T05:22:33.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.565 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.566 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952451+0000", 2024-09-16T05:22:33.566 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8302e-05 2024-09-16T05:22:33.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.566 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.566 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.567 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:383 2.29 e617/253 2:9662c29c:::smithi20457400-9:d8 v 617'830, mlcod=616'824)", 2024-09-16T05:22:33.567 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:22:33.567 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 186.74021136900001, 2024-09-16T05:22:33.567 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038031031999999999, 2024-09-16T05:22:33.567 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.568 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.568 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.568 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.568 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:22:33.568 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.569 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.569 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:22:33.569 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782794+0000", 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3500000000000005e-07 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782801+0000", 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1210000000000001e-06 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.570 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819928+0000", 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037126740999999998 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819938+0000", 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0845e-05 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820001+0000", 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2786999999999994e-05 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820023+0000", 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.213e-05 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820771+0000", 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074818699999999998 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820824+0000", 2024-09-16T05:22:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2286e-05 2024-09-16T05:22:33.573 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.573 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.573 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.573 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:199 2.1e e714/291 ondisk, result = 0)", 2024-09-16T05:22:33.574 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:22:33.574 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 89.710795254000004, 2024-09-16T05:22:33.574 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222874399, 2024-09-16T05:22:33.574 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.574 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.574 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:22:33.575 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.575 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.575 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:22:33.575 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:22:33.576 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2099999999999997e-07 2024-09-16T05:22:33.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.577 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.577 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812213+0000", 2024-09-16T05:22:33.577 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3859999999999999e-06 2024-09-16T05:22:33.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034914+0000", 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222700804 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034920+0000", 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6579999999999996e-06 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034968+0000", 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7833000000000001e-05 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034975+0000", 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0400000000000004e-06 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.035083+0000", 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108157 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.579 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7351 2.29 2:94d33c56:::smithi20457400-8:head [write 410368~646278] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:36.863973+0000", 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.659030629, 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084857950000000008, 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:22:33.580 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7351 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.863973+0000", 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.863973+0000", 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.863975+0000", 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9630000000000001e-06 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.864689+0000", 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071403299999999999 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.864704+0000", 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4307999999999999e-05 2024-09-16T05:22:33.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.864712+0000", 2024-09-16T05:22:33.583 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.67e-06 2024-09-16T05:22:33.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.583 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.583 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.865089+0000", 2024-09-16T05:22:33.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037642600000000002 2024-09-16T05:22:33.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.584 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.584 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.865172+0000", 2024-09-16T05:22:33.584 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3288000000000006e-05 2024-09-16T05:22:33.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.585 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:22:33.585 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.865331+0000", 2024-09-16T05:22:33.585 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000158765 2024-09-16T05:22:33.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.586 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:22:33.586 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.867114+0000", 2024-09-16T05:22:33.587 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017832880000000001 2024-09-16T05:22:33.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.588 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:33.588 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.872161+0000", 2024-09-16T05:22:33.588 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050464459999999996 2024-09-16T05:22:33.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.589 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.589 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.872261+0000", 2024-09-16T05:22:33.589 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010050799999999999 2024-09-16T05:22:33.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.590 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.590 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.872459+0000", 2024-09-16T05:22:33.590 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019809999999999999 2024-09-16T05:22:33.590 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.590 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.591 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7352 2.29 2:94d33c56:::smithi20457400-8:head [write 1472267~475349] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:22:33.591 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:36.868856+0000", 2024-09-16T05:22:33.591 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 56.654148018999997, 2024-09-16T05:22:33.592 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0087654470000000009, 2024-09-16T05:22:33.592 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.592 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.592 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:22:33.593 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:22:33.593 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:33.593 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7352 2024-09-16T05:22:33.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.594 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.594 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.594 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.868856+0000", 2024-09-16T05:22:33.594 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.868856+0000", 2024-09-16T05:22:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.868858+0000", 2024-09-16T05:22:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.221e-06 2024-09-16T05:22:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869362+0000", 2024-09-16T05:22:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050372899999999996 2024-09-16T05:22:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869384+0000", 2024-09-16T05:22:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1503000000000001e-05 2024-09-16T05:22:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869396+0000", 2024-09-16T05:22:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2016e-05 2024-09-16T05:22:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869477+0000", 2024-09-16T05:22:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1207999999999999e-05 2024-09-16T05:22:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869610+0000", 2024-09-16T05:22:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013292600000000001 2024-09-16T05:22:33.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.601 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:22:33.601 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869778+0000", 2024-09-16T05:22:33.601 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016856900000000001 2024-09-16T05:22:33.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.602 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:22:33.602 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.871275+0000", 2024-09-16T05:22:33.602 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001496961 2024-09-16T05:22:33.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.877452+0000", 2024-09-16T05:22:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061771819999999998 2024-09-16T05:22:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.877510+0000", 2024-09-16T05:22:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7181e-05 2024-09-16T05:22:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.877622+0000", 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111951 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7414 2.5a 2:5ae26c04:::smithi20457400-32:head [write 1423159~486356] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.690823082000001, 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010834233, 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7414 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.606 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832183+0000", 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6589999999999999e-06 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832584+0000", 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040119599999999999 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832598+0000", 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3736e-05 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832607+0000", 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1870000000000003e-06 2024-09-16T05:22:33.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832655+0000", 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8043000000000001e-05 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832725+0000", 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0136999999999996e-05 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832832+0000", 2024-09-16T05:22:33.609 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010700399999999999 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834013+0000", 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011807479999999999 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842930+0000", 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0089172439999999995 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.610 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842977+0000", 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7547999999999999e-05 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.843015+0000", 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7731000000000002e-05 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:7414 2.5a e755/742 ondisk, result = 0)", 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:22:33.611 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.688793441999998, 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088119329999999992, 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.612 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834212+0000", 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2699999999999998e-07 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834216+0000", 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2139999999999998e-06 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.613 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842854+0000", 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086381679999999999 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842859+0000", 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842913+0000", 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3235000000000001e-05 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842922+0000", 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.431e-06 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.843023+0000", 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100485 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7416 2.5a 2:5ae26c04:::smithi20457400-32:head [setxattr _header (58) in=7b,truncate 3311210] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.682383137999999, 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.075538587000000004, 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7416 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840622+0000", 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6400000000000001e-07 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840626+0000", 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8589999999999998e-06 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840639+0000", 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2982999999999999e-05 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840644+0000", 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.57e-06 2024-09-16T05:22:33.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.841987+0000", 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013431599999999999 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842059+0000", 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1450000000000002e-05 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842166+0000", 2024-09-16T05:22:33.619 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107421 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842788+0000", 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062151399999999999 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916025+0000", 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073237675000000002 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.620 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916092+0000", 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6631999999999995e-05 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916160+0000", 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7459000000000006e-05 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7417 2.5a 2:5ae26c04:::smithi20457400-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:22:33.621 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.682326857, 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.075671839000000005, 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7417 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.622 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840678+0000", 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9800000000000003e-07 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.623 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840682+0000", 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9310000000000001e-06 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840695+0000", 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2672e-05 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840705+0000", 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0498e-05 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842425+0000", 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001719497 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842530+0000", 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105824 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916202+0000", 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073671367000000001 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916273+0000", 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1316000000000006e-05 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916349+0000", 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5935999999999999e-05 2024-09-16T05:22:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:7416 2.5a e755/742 ondisk, result = 0)", 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 48.680051814999999, 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073216748999999998, 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842953+0000", 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4000000000000002e-07 2024-09-16T05:22:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842956+0000", 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.915928+0000", 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.072972180999999997 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.915937+0000", 2024-09-16T05:22:33.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3689999999999996e-06 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916004+0000", 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6836999999999997e-05 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916017+0000", 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3200000000000001e-05 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916169+0000", 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000151512 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:251 2.29 e762/745 ondisk, result = 0)", 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 41.600529594000001, 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.502823558, 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.633 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922475+0000", 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3000000000000001e-07 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922478+0000", 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.18e-06 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425018+0000", 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.50254011200000004 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425039+0000", 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0832999999999999e-05 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425166+0000", 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012726 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425177+0000", 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0217999999999999e-05 2024-09-16T05:22:33.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425298+0000", 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121425 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:580 2.29 e776/767 2:94d33c56:::smithi20457400-8:139 v 776'1225, mlcod=773'1219)", 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:05.823510+0000", 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 27.699493908000001, 2024-09-16T05:22:33.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084587550000000001, 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.823510+0000", 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.823510+0000", 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.823511+0000", 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2399999999999996e-07 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.823517+0000", 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1639999999999998e-06 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.823550+0000", 2024-09-16T05:22:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2960999999999998e-05 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.823558+0000", 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7780000000000007e-06 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.825471+0000", 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0019130810000000001 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.825493+0000", 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2410999999999999e-05 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.831913+0000", 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064194079999999997 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.831969+0000", 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6028e-05 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.7.0:1425 2.2e e786/278 2:75dc7feb:::smithi20457400-28:13f v 786'738, mlcod=781'732)", 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 17.672517267, 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72979714900000003, 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:22:33.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850493+0000", 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1120000000000001e-06 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579433+0000", 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72893962899999998 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579449+0000", 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6059999999999999e-05 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579568+0000", 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118911 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579618+0000", 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0436000000000003e-05 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.580252+0000", 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063313400000000004 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.580284+0000", 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2381e-05 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.2e,QREQUEST,e:797,from:7,reservation_nonce:8)", 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 5.1806024490000002, 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27875738500000002, 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342404+0000", 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9300000000000002e-06 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342412+0000", 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6319999999999997e-06 2024-09-16T05:22:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.620932+0000", 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27851996800000001 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.620943+0000", 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0892e-05 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621083+0000", 2024-09-16T05:22:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013946999999999999 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621110+0000", 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7934000000000001e-05 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621159+0000", 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8559e-05 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7978 2.5a 2:5ae26c04:::smithi20457400-32:head [write 638963~722326] snapc 152=[152,151,150,14e,14b,149,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e802)", 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:31.841110+0000", 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.6818940100000002, 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035719179999999999, 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:33.650 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7978 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.841110+0000", 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.841110+0000", 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.651 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.841112+0000", 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.592e-06 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.842051+0000", 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00093977299999999998 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.842069+0000", 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7558e-05 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.842079+0000", 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5959999999999993e-06 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.842132+0000", 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3752999999999998e-05 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.842235+0000", 2024-09-16T05:22:33.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103052 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.842426+0000", 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000190865 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.844260+0000", 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001834177 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.844584+0000", 2024-09-16T05:22:33.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00032393299999999999 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.844635+0000", 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0510000000000003e-05 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:31.844682+0000", 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7108999999999999e-05 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.655 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:22:33.656 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:22:33.656 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:22:33.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:33.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:33.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:33.806 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:33.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:33.883 INFO:tasks.rados.rados.2.smithi204.stdout:3171: finishing write tid 2 to smithi20457400-46 2024-09-16T05:22:33.883 INFO:tasks.rados.rados.2.smithi204.stdout:3171: finishing write tid 3 to smithi20457400-46 2024-09-16T05:22:33.883 INFO:tasks.rados.rados.2.smithi204.stdout:3171: finishing write tid 4 to smithi20457400-46 2024-09-16T05:22:33.883 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1023 (ObjNum 1116 snap 337 seq_num 1116) dirty exists 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3171: left oid 46 (ObjNum 1116 snap 337 seq_num 1116) 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3172: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3172: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3172: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1317 (ObjNum 1079 snap 328 seq_num 1079) dirty exists 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3173: finishing write tid 1 to smithi20457400-41 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3168: finishing write tid 1 to smithi20457400-25 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3168: finishing write tid 2 to smithi20457400-25 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3168: finishing write tid 3 to smithi20457400-25 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3168: finishing write tid 4 to smithi20457400-25 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 755 (ObjNum 1113 snap 337 seq_num 1113) dirty exists 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3168: left oid 25 (ObjNum 1113 snap 337 seq_num 1113) 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3173: finishing write tid 2 to smithi20457400-41 2024-09-16T05:22:33.884 INFO:tasks.rados.rados.2.smithi204.stdout:3173: finishing write tid 3 to smithi20457400-41 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3173: finishing write tid 4 to smithi20457400-41 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1321 (ObjNum 1117 snap 337 seq_num 1117) dirty exists 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3173: left oid 41 (ObjNum 1117 snap 337 seq_num 1117) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3170: finishing write tid 5 to smithi20457400-22 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 811 (ObjNum 1115 snap 337 seq_num 1115) dirty exists 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3170: left oid 22 (ObjNum 1115 snap 337 seq_num 1115) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3166: done (7 left) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3168: done (6 left) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3170: done (5 left) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3171: done (4 left) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3172: done (3 left) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3173: done (2 left) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3174: done (1 left) 2024-09-16T05:22:33.885 INFO:tasks.rados.rados.2.smithi204.stdout:3175: done (0 left) 2024-09-16T05:22:33.886 INFO:tasks.rados.rados.2.smithi204.stdout:3176: write (excl) oid 34 current snap is 337 2024-09-16T05:22:33.886 INFO:tasks.rados.rados.2.smithi204.stdout:3176: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-09-16T05:22:33.889 INFO:tasks.rados.rados.2.smithi204.stdout:3176: writing smithi20457400-34 from 790226 to 1222499 tid 1 2024-09-16T05:22:33.896 INFO:tasks.rados.rados.2.smithi204.stdout:3176: writing smithi20457400-34 from 1806045 to 2388425 tid 2 2024-09-16T05:22:33.896 INFO:tasks.rados.rados.2.smithi204.stdout:3177: write (excl) oid 20 current snap is 337 2024-09-16T05:22:33.896 INFO:tasks.rados.rados.2.smithi204.stdout:3177: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2024-09-16T05:22:33.901 INFO:tasks.rados.rados.2.smithi204.stdout:3177: writing smithi20457400-20 from 407033 to 914555 tid 1 2024-09-16T05:22:33.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:33.907 INFO:tasks.rados.rados.2.smithi204.stdout:3177: writing smithi20457400-20 from 1464527 to 1906918 tid 2 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3177: writing smithi20457400-20 from 2345976 to 2807033 tid 3 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3178: write oid 37 current snap is 337 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3176: finishing write tid 1 to smithi20457400-34 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3176: finishing write tid 2 to smithi20457400-34 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3176: finishing write tid 3 to smithi20457400-34 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3176: finishing write tid 4 to smithi20457400-34 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1265 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3176: left oid 34 (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3177: finishing write tid 1 to smithi20457400-20 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3177: finishing write tid 2 to smithi20457400-20 2024-09-16T05:22:33.913 INFO:tasks.rados.rados.2.smithi204.stdout:3178: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2024-09-16T05:22:33.918 INFO:tasks.rados.rados.2.smithi204.stdout:3178: writing smithi20457400-37 from 423840 to 1006611 tid 1 2024-09-16T05:22:33.924 INFO:tasks.rados.rados.2.smithi204.stdout:3178: writing smithi20457400-37 from 1523009 to 2109058 tid 2 2024-09-16T05:22:33.929 INFO:tasks.rados.rados.2.smithi204.stdout:3178: writing smithi20457400-37 from 2657046 to 2823840 tid 3 2024-09-16T05:22:33.929 INFO:tasks.rados.rados.2.smithi204.stdout:3176: done (2 left) 2024-09-16T05:22:33.929 INFO:tasks.rados.rados.2.smithi204.stdout:3179: read oid 49 snap -1 2024-09-16T05:22:33.929 INFO:tasks.rados.rados.2.smithi204.stdout:3179: expect (ObjNum 921 snap 273 seq_num 921) 2024-09-16T05:22:33.929 INFO:tasks.rados.rados.2.smithi204.stdout:3180: rollback oid 30 current snap is 337 2024-09-16T05:22:33.929 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 30 to 279 2024-09-16T05:22:33.929 INFO:tasks.rados.rados.2.smithi204.stdout:3177: finishing write tid 3 to smithi20457400-20 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:3177: finishing write tid 4 to smithi20457400-20 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:3181: rollback oid 36 current snap is 337 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 36 to 276 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:3177: finishing write tid 5 to smithi20457400-20 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1299 (ObjNum 1119 snap 337 seq_num 1119) dirty exists 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:3177: left oid 20 (ObjNum 1119 snap 337 seq_num 1119) 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:3178: finishing write tid 1 to smithi20457400-37 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:3178: finishing write tid 2 to smithi20457400-37 2024-09-16T05:22:33.930 INFO:tasks.rados.rados.2.smithi204.stdout:3177: done (4 left) 2024-09-16T05:22:33.931 INFO:tasks.rados.rados.2.smithi204.stdout:3182: write (excl) oid 28 current snap is 337 2024-09-16T05:22:33.931 INFO:tasks.rados.rados.2.smithi204.stdout:3182: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-09-16T05:22:33.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:33.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:33.935 INFO:tasks.rados.rados.2.smithi204.stdout:3182: writing smithi20457400-28 from 440647 to 1098667 tid 1 2024-09-16T05:22:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:3182: writing smithi20457400-28 from 1581491 to 2027551 tid 2 2024-09-16T05:22:33.945 INFO:tasks.rados.rados.2.smithi204.stdout:3182: writing smithi20457400-28 from 2800822 to 2840647 tid 3 2024-09-16T05:22:33.945 INFO:tasks.rados.rados.2.smithi204.stdout:3183: read oid 50 snap -1 2024-09-16T05:22:33.945 INFO:tasks.rados.rados.2.smithi204.stdout:3183: expect (ObjNum 1099 snap 332 seq_num 1099) 2024-09-16T05:22:33.945 INFO:tasks.rados.rados.2.smithi204.stdout:3184: write oid 5 current snap is 337 2024-09-16T05:22:33.945 INFO:tasks.rados.rados.2.smithi204.stdout:3178: finishing write tid 3 to smithi20457400-37 2024-09-16T05:22:33.945 INFO:tasks.rados.rados.2.smithi204.stdout:3184: seq_num 1122 ranges {457454=733269,1923620=589718} 2024-09-16T05:22:33.951 INFO:tasks.rados.rados.2.smithi204.stdout:3184: writing smithi20457400-5 from 457454 to 1190723 tid 1 2024-09-16T05:22:33.959 INFO:tasks.rados.rados.2.smithi204.stdout:3184: writing smithi20457400-5 from 1923620 to 2513338 tid 2 2024-09-16T05:22:33.959 INFO:tasks.rados.rados.2.smithi204.stdout:3185: snap_remove snap 334 2024-09-16T05:22:34.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:34.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:34.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: pgmap v1365: 127 pgs: 2 active+clean+snaptrim, 125 active+clean; 343 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6 op/s 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4192431711' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: osdmap e803: 12 total, 12 up, 12 in 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:22:34.092 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:22:34.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: 2.4f scrub starts 2024-09-16T05:22:34.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: 2.4f scrub ok 2024-09-16T05:22:34.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2129646869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:34.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: pgmap v1365: 127 pgs: 2 active+clean+snaptrim, 125 active+clean; 343 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6 op/s 2024-09-16T05:22:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:22:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:22:34.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:22:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4192431711' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: osdmap e803: 12 total, 12 up, 12 in 2024-09-16T05:22:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:22:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:22:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: 2.4f scrub starts 2024-09-16T05:22:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: 2.4f scrub ok 2024-09-16T05:22:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2129646869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:34.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:34.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: pgmap v1365: 127 pgs: 2 active+clean+snaptrim, 125 active+clean; 343 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6 op/s 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4192431711' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: osdmap e803: 12 total, 12 up, 12 in 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:22:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:22:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: 2.4f scrub starts 2024-09-16T05:22:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: 2.4f scrub ok 2024-09-16T05:22:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2129646869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:34.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:34.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:34.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:34.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:34.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:34.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:34.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:34.562 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:34.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:34.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:34.684 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:34.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:34.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:34.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:34.707 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:34.707 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:34.728 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:34.728 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:34.749 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:34.749 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:34.771 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:34.771 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:34.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:34.793 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:34.793 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:34.814 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:34.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:34.816 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:34.836 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:34.837 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:34.858 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:34.859 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:34.880 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:34.881 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:34.893 INFO:tasks.rados.rados.2.smithi204.stdout:3185: done (7 left) 2024-09-16T05:22:34.893 INFO:tasks.rados.rados.2.smithi204.stdout:3186: delete oid 46 current snap is 337 2024-09-16T05:22:34.893 INFO:tasks.rados.rados.2.smithi204.stdout:3178: finishing write tid 4 to smithi20457400-37 2024-09-16T05:22:34.893 INFO:tasks.rados.rados.2.smithi204.stdout:3178: finishing write tid 5 to smithi20457400-37 2024-09-16T05:22:34.893 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1052 (ObjNum 1120 snap 337 seq_num 1120) dirty exists 2024-09-16T05:22:34.893 INFO:tasks.rados.rados.2.smithi204.stdout:3178: left oid 37 (ObjNum 1120 snap 337 seq_num 1120) 2024-09-16T05:22:34.893 INFO:tasks.rados.rados.2.smithi204.stdout:3181: finishing rollback tid 0 to smithi20457400-36 2024-09-16T05:22:34.894 INFO:tasks.rados.rados.2.smithi204.stdout:3181: finishing rollback tid 1 to smithi20457400-36 2024-09-16T05:22:34.894 INFO:tasks.rados.rados.2.smithi204.stdout:3181: finishing rollback tid 2 to smithi20457400-36 2024-09-16T05:22:34.894 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 783 (ObjNum 931 snap 275 seq_num 931) dirty exists 2024-09-16T05:22:34.894 INFO:tasks.rados.rados.2.smithi204.stdout:3182: finishing write tid 1 to smithi20457400-28 2024-09-16T05:22:34.894 INFO:tasks.rados.rados.2.smithi204.stdout:3179: expect (ObjNum 921 snap 273 seq_num 921) 2024-09-16T05:22:34.906 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:34.907 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:34.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:34.928 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:34.929 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:34.938 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:34.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:35.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:35.063 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:35.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:35.126 INFO:tasks.rados.rados.2.smithi204.stdout:3180: finishing rollback tid 1 to smithi20457400-30 2024-09-16T05:22:35.126 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 1387 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3182: finishing write tid 2 to smithi20457400-28 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3178: done (7 left) 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3179: done (6 left) 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3180: done (5 left) 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3181: done (4 left) 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3186: done (3 left) 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3187: delete oid 17 current snap is 337 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3182: finishing write tid 3 to smithi20457400-28 2024-09-16T05:22:35.127 INFO:tasks.rados.rados.2.smithi204.stdout:3183: expect (ObjNum 1099 snap 332 seq_num 1099) 2024-09-16T05:22:35.163 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:34 smithi032 ceph-mon[27388]: osdmap e804: 12 total, 12 up, 12 in 2024-09-16T05:22:35.164 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:34 smithi032 ceph-mon[27388]: 2.3d deep-scrub starts 2024-09-16T05:22:35.164 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:34 smithi032 ceph-mon[27388]: 2.3d deep-scrub ok 2024-09-16T05:22:35.164 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:34 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:22:35.164 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:34 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:22:35.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:34 smithi204 ceph-mon[30933]: osdmap e804: 12 total, 12 up, 12 in 2024-09-16T05:22:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:34 smithi204 ceph-mon[30933]: 2.3d deep-scrub starts 2024-09-16T05:22:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:34 smithi204 ceph-mon[30933]: 2.3d deep-scrub ok 2024-09-16T05:22:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:34 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:22:35.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:34 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:22:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:35.186 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:35.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:35.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:22:35.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:34 smithi196 ceph-mon[30802]: osdmap e804: 12 total, 12 up, 12 in 2024-09-16T05:22:35.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:34 smithi196 ceph-mon[30802]: 2.3d deep-scrub starts 2024-09-16T05:22:35.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:34 smithi196 ceph-mon[30802]: 2.3d deep-scrub ok 2024-09-16T05:22:35.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:34 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:22:35.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:34 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:22:35.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:35.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:35.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:35.324 INFO:tasks.rados.rados.2.smithi204.stdout:3184: finishing write tid 1 to smithi20457400-5 2024-09-16T05:22:35.324 INFO:tasks.rados.rados.2.smithi204.stdout:3184: finishing write tid 2 to smithi20457400-5 2024-09-16T05:22:35.324 INFO:tasks.rados.rados.2.smithi204.stdout:3184: finishing write tid 3 to smithi20457400-5 2024-09-16T05:22:35.324 INFO:tasks.rados.rados.2.smithi204.stdout:3184: finishing write tid 4 to smithi20457400-5 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 624 (ObjNum 1122 snap 337 seq_num 1122) dirty exists 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3184: left oid 5 (ObjNum 1122 snap 337 seq_num 1122) 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3183: done (3 left) 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3184: done (2 left) 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3187: done (1 left) 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3188: write (excl) oid 41 current snap is 337 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3182: finishing write tid 4 to smithi20457400-28 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3182: finishing write tid 5 to smithi20457400-28 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1125 (ObjNum 1121 snap 337 seq_num 1121) dirty exists 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3182: left oid 28 (ObjNum 1121 snap 337 seq_num 1121) 2024-09-16T05:22:35.325 INFO:tasks.rados.rados.2.smithi204.stdout:3188: seq_num 1125 ranges {507875=559016,1699066=737045} 2024-09-16T05:22:35.329 INFO:tasks.rados.rados.2.smithi204.stdout:3188: writing smithi20457400-41 from 507875 to 1066891 tid 1 2024-09-16T05:22:35.336 INFO:tasks.rados.rados.2.smithi204.stdout:3188: writing smithi20457400-41 from 1699066 to 2436111 tid 2 2024-09-16T05:22:35.337 INFO:tasks.rados.rados.2.smithi204.stdout:3182: done (1 left) 2024-09-16T05:22:35.337 INFO:tasks.rados.rados.2.smithi204.stdout:3189: snap_remove snap 328 2024-09-16T05:22:35.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:35.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:35.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:35.450 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:35.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:35.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:35.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:35.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:35.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:35.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:35.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-09-16T05:22:35.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:35.832 INFO:tasks.rados.rados.2.smithi204.stdout:3189: done (1 left) 2024-09-16T05:22:35.832 INFO:tasks.rados.rados.2.smithi204.stdout:3190: write (excl) oid 36 current snap is 337 2024-09-16T05:22:35.832 INFO:tasks.rados.rados.2.smithi204.stdout:3190: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-09-16T05:22:35.839 INFO:tasks.rados.rados.2.smithi204.stdout:3190: writing smithi20457400-36 from 524682 to 1275300 tid 1 2024-09-16T05:22:35.845 INFO:tasks.rados.rados.2.smithi204.stdout:3190: writing smithi20457400-36 from 1757548 to 2354604 tid 2 2024-09-16T05:22:35.846 INFO:tasks.rados.rados.2.smithi204.stdout:3191: snap_remove snap 335 2024-09-16T05:22:35.852 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:35.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:35.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:35.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:35.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:35.981 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:35.981 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:36.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:36.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:36.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:36.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: pgmap v1368: 127 pgs: 127 active+clean; 347 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-16T05:22:36.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:22:36.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:22:36.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: osdmap e805: 12 total, 12 up, 12 in 2024-09-16T05:22:36.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: 2.59 scrub starts 2024-09-16T05:22:36.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: 2.59 scrub ok 2024-09-16T05:22:36.126 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:22:36.127 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:22:36.127 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:35 smithi196 ceph-mon[30802]: osdmap e806: 12 total, 12 up, 12 in 2024-09-16T05:22:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: pgmap v1368: 127 pgs: 127 active+clean; 347 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-16T05:22:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:22:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:22:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: osdmap e805: 12 total, 12 up, 12 in 2024-09-16T05:22:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: 2.59 scrub starts 2024-09-16T05:22:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: 2.59 scrub ok 2024-09-16T05:22:36.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:35 smithi032 ceph-mon[27388]: osdmap e806: 12 total, 12 up, 12 in 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: pgmap v1368: 127 pgs: 127 active+clean; 347 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: osdmap e805: 12 total, 12 up, 12 in 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: 2.59 scrub starts 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: 2.59 scrub ok 2024-09-16T05:22:36.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:22:36.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:22:36.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:35 smithi204 ceph-mon[30933]: osdmap e806: 12 total, 12 up, 12 in 2024-09-16T05:22:36.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:36.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:36.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:36.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:36.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:36.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:36.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:36.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:36.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:36.476 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:36.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:22:36.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:36.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:36.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:22:36.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:36.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:36.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:36.837 INFO:tasks.rados.rados.2.smithi204.stdout:3191: done (2 left) 2024-09-16T05:22:36.837 INFO:tasks.rados.rados.2.smithi204.stdout:3192: copy_from oid 26 from oid 49 current snap is 337 2024-09-16T05:22:36.838 INFO:tasks.rados.rados.2.smithi204.stdout:3193: snap_remove snap 292 2024-09-16T05:22:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:36.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:36.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:36.911 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:36.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:36.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:36.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:37.033 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:37.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:37.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:37.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:22:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:22:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:22:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:22:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: 2.17 deep-scrub starts 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: 2.17 deep-scrub ok 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:36 smithi032 ceph-mon[27388]: osdmap e807: 12 total, 12 up, 12 in 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:22:37.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:22:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:22:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: 2.17 deep-scrub starts 2024-09-16T05:22:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: 2.17 deep-scrub ok 2024-09-16T05:22:37.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:36 smithi204 ceph-mon[30933]: osdmap e807: 12 total, 12 up, 12 in 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: 2.17 deep-scrub starts 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: 2.17 deep-scrub ok 2024-09-16T05:22:37.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:36 smithi196 ceph-mon[30802]: osdmap e807: 12 total, 12 up, 12 in 2024-09-16T05:22:37.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:37.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:37.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:37.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:37.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:37.398 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:37.399 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:37.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:37.475 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:37.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:37.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:37.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:37.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:37.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:37.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:37.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:37.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3193: done (3 left) 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3194: copy_from oid 18 from oid 15 current snap is 337 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3195: snap_create 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3188: finishing write tid 1 to smithi20457400-41 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3188: finishing write tid 2 to smithi20457400-41 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3188: finishing write tid 3 to smithi20457400-41 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3188: finishing write tid 4 to smithi20457400-41 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1332 (ObjNum 1125 snap 337 seq_num 1125) dirty exists 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3188: left oid 41 (ObjNum 1125 snap 337 seq_num 1125) 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3190: finishing write tid 1 to smithi20457400-36 2024-09-16T05:22:37.840 INFO:tasks.rados.rados.2.smithi204.stdout:3190: finishing write tid 2 to smithi20457400-36 2024-09-16T05:22:37.841 INFO:tasks.rados.rados.2.smithi204.stdout:3190: finishing write tid 3 to smithi20457400-36 2024-09-16T05:22:37.841 INFO:tasks.rados.rados.2.smithi204.stdout:3190: finishing write tid 4 to smithi20457400-36 2024-09-16T05:22:37.841 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 786 (ObjNum 1126 snap 337 seq_num 1126) dirty exists 2024-09-16T05:22:37.841 INFO:tasks.rados.rados.2.smithi204.stdout:3190: left oid 36 (ObjNum 1126 snap 337 seq_num 1126) 2024-09-16T05:22:37.841 INFO:tasks.rados.rados.2.smithi204.stdout:3192: finishing copy_from racing read to smithi20457400-26 2024-09-16T05:22:37.841 INFO:tasks.rados.rados.2.smithi204.stdout:3192: finishing copy_from to smithi20457400-26 2024-09-16T05:22:37.841 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 624 (ObjNum 921 snap 273 seq_num 921) dirty exists 2024-09-16T05:22:37.855 INFO:tasks.rados.rados.2.smithi204.stdout:3194: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:22:37.858 INFO:tasks.rados.rados.2.smithi204.stdout:3194: finishing copy_from to smithi20457400-18 2024-09-16T05:22:37.859 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 750 (ObjNum 1087 snap 330 seq_num 1087) dirty exists 2024-09-16T05:22:37.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:37.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:37.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:37.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: pgmap v1371: 127 pgs: 127 active+clean; 352 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 26 op/s 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: 1.1d scrub starts 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: 1.1d scrub ok 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: 2.4d scrub starts 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: 2.4d scrub ok 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: osdmap e808: 12 total, 12 up, 12 in 2024-09-16T05:22:37.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2473314089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:37.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:37.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:37.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:22:38.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-09-16T05:22:38.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:38.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:38.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: pgmap v1371: 127 pgs: 127 active+clean; 352 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 26 op/s 2024-09-16T05:22:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:22:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:22:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: 1.1d scrub starts 2024-09-16T05:22:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: 1.1d scrub ok 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: 2.4d scrub starts 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: 2.4d scrub ok 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: osdmap e808: 12 total, 12 up, 12 in 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2473314089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: pgmap v1371: 127 pgs: 127 active+clean; 352 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 26 op/s 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:38.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: 1.1d scrub starts 2024-09-16T05:22:38.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: 1.1d scrub ok 2024-09-16T05:22:38.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: 2.4d scrub starts 2024-09-16T05:22:38.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: 2.4d scrub ok 2024-09-16T05:22:38.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: osdmap e808: 12 total, 12 up, 12 in 2024-09-16T05:22:38.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2473314089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:38.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:38.236 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:38.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:38.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:38.282 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:38.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:38.364 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:38.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:38.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:22:38.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:38.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:38.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:38.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:38.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:38.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:38.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:38.720 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:38.720 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:38.738 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:38.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:38.835 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:22:38.839 INFO:tasks.rados.rados.2.smithi204.stdout:3188: done (4 left) 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3190: done (3 left) 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3192: done (2 left) 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3194: done (1 left) 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3195: done (0 left) 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3196: copy_from oid 3 from oid 30 current snap is 338 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3197: read oid 20 snap -1 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3197: expect (ObjNum 1119 snap 337 seq_num 1119) 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3198: read oid 18 snap -1 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3198: expect (ObjNum 1087 snap 330 seq_num 1087) 2024-09-16T05:22:38.840 INFO:tasks.rados.rados.2.smithi204.stdout:3199: write oid 35 current snap is 338 2024-09-16T05:22:38.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:38.841 INFO:tasks.rados.rados.2.smithi204.stdout:3199: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-09-16T05:22:38.845 INFO:tasks.rados.rados.2.smithi204.stdout:3199: writing smithi20457400-35 from 541489 to 967356 tid 1 2024-09-16T05:22:38.852 INFO:tasks.rados.rados.2.smithi204.stdout:3199: writing smithi20457400-35 from 1416030 to 2156744 tid 2 2024-09-16T05:22:38.857 INFO:tasks.rados.rados.2.smithi204.stdout:3199: writing smithi20457400-35 from 2732654 to 2941489 tid 3 2024-09-16T05:22:38.857 INFO:tasks.rados.rados.2.smithi204.stdout:3200: read oid 43 snap -1 2024-09-16T05:22:38.857 INFO:tasks.rados.rados.2.smithi204.stdout:3200: expect (ObjNum 1110 snap 336 seq_num 1110) 2024-09-16T05:22:38.857 INFO:tasks.rados.rados.2.smithi204.stdout:3201: delete oid 9 current snap is 338 2024-09-16T05:22:38.857 INFO:tasks.rados.rados.2.smithi204.stdout:3196: finishing copy_from to smithi20457400-3 2024-09-16T05:22:38.857 INFO:tasks.rados.rados.2.smithi204.stdout:3196: got expected ENOENT (src dne) 2024-09-16T05:22:38.858 INFO:tasks.rados.rados.2.smithi204.stdout:3196: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:22:38.858 INFO:tasks.rados.rados.2.smithi204.stdout:3197: expect (ObjNum 1119 snap 337 seq_num 1119) 2024-09-16T05:22:38.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:38.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:38.952 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:38 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:22:38.953 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:38 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:22:38.953 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:38 smithi032 ceph-mon[27388]: 1.1 deep-scrub starts 2024-09-16T05:22:38.953 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:38 smithi032 ceph-mon[27388]: 1.1 deep-scrub ok 2024-09-16T05:22:38.953 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:38 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:38.953 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2473314089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:38.953 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:38 smithi032 ceph-mon[27388]: osdmap e809: 12 total, 12 up, 12 in 2024-09-16T05:22:38.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:38.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:38.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:39.059 INFO:tasks.rados.rados.2.smithi204.stdout:3198: expect (ObjNum 1087 snap 330 seq_num 1087) 2024-09-16T05:22:39.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:39.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:38 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:22:39.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:38 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:22:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:38 smithi204 ceph-mon[30933]: 1.1 deep-scrub starts 2024-09-16T05:22:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:38 smithi204 ceph-mon[30933]: 1.1 deep-scrub ok 2024-09-16T05:22:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:38 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2473314089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:39.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:38 smithi204 ceph-mon[30933]: osdmap e809: 12 total, 12 up, 12 in 2024-09-16T05:22:39.181 INFO:tasks.rados.rados.2.smithi204.stdout:3199: finishing write tid 1 to smithi20457400-35 2024-09-16T05:22:39.181 INFO:tasks.rados.rados.2.smithi204.stdout:3199: finishing write tid 2 to smithi20457400-35 2024-09-16T05:22:39.181 INFO:tasks.rados.rados.2.smithi204.stdout:3199: finishing write tid 3 to smithi20457400-35 2024-09-16T05:22:39.181 INFO:tasks.rados.rados.2.smithi204.stdout:3199: finishing write tid 4 to smithi20457400-35 2024-09-16T05:22:39.181 INFO:tasks.rados.rados.2.smithi204.stdout:3199: finishing write tid 5 to smithi20457400-35 2024-09-16T05:22:39.181 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 862 (ObjNum 1127 snap 338 seq_num 1127) dirty exists 2024-09-16T05:22:39.181 INFO:tasks.rados.rados.2.smithi204.stdout:3199: left oid 35 (ObjNum 1127 snap 338 seq_num 1127) 2024-09-16T05:22:39.181 INFO:tasks.rados.rados.2.smithi204.stdout:3200: expect (ObjNum 1110 snap 336 seq_num 1110) 2024-09-16T05:22:39.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:38 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:22:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:38 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:22:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:38 smithi196 ceph-mon[30802]: 1.1 deep-scrub starts 2024-09-16T05:22:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:38 smithi196 ceph-mon[30802]: 1.1 deep-scrub ok 2024-09-16T05:22:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:38 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2473314089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:38 smithi196 ceph-mon[30802]: osdmap e809: 12 total, 12 up, 12 in 2024-09-16T05:22:39.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:39.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3196: done (5 left) 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3197: done (4 left) 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3198: done (3 left) 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3199: done (2 left) 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3200: done (1 left) 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3201: done (0 left) 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3202: read oid 16 snap -1 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3202: expect (ObjNum 1083 snap 329 seq_num 1083) 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3203: read oid 41 snap -1 2024-09-16T05:22:39.269 INFO:tasks.rados.rados.2.smithi204.stdout:3203: expect (ObjNum 1125 snap 337 seq_num 1125) 2024-09-16T05:22:39.270 INFO:tasks.rados.rados.2.smithi204.stdout:3204: delete oid 7 current snap is 338 2024-09-16T05:22:39.271 INFO:tasks.rados.rados.2.smithi204.stdout:3204: done (2 left) 2024-09-16T05:22:39.271 INFO:tasks.rados.rados.2.smithi204.stdout:3205: read oid 13 snap -1 2024-09-16T05:22:39.271 INFO:tasks.rados.rados.2.smithi204.stdout:3205: expect deleted 2024-09-16T05:22:39.271 INFO:tasks.rados.rados.2.smithi204.stdout:3206: rollback oid 7 current snap is 338 2024-09-16T05:22:39.271 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 7 to 336 2024-09-16T05:22:39.271 INFO:tasks.rados.rados.2.smithi204.stdout:3207: copy_from oid 28 from oid 15 current snap is 338 2024-09-16T05:22:39.271 INFO:tasks.rados.rados.2.smithi204.stdout:3208: write oid 37 current snap is 338 2024-09-16T05:22:39.271 INFO:tasks.rados.rados.2.smithi204.stdout:3208: seq_num 1130 ranges {591910=651614,1875123=604394} 2024-09-16T05:22:39.277 INFO:tasks.rados.rados.2.smithi204.stdout:3208: writing smithi20457400-37 from 591910 to 1243524 tid 1 2024-09-16T05:22:39.284 INFO:tasks.rados.rados.2.smithi204.stdout:3208: writing smithi20457400-37 from 1875123 to 2479517 tid 2 2024-09-16T05:22:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:3209: read oid 18 snap 336 2024-09-16T05:22:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:3209: expect deleted 2024-09-16T05:22:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:3210: read oid 29 snap 279 2024-09-16T05:22:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:3210: expect (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:22:39.285 INFO:tasks.rados.rados.2.smithi204.stdout:3211: snap_remove snap 337 2024-09-16T05:22:39.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:22:39.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:39.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:39.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:39.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:39.491 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:39.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:39.527 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:39.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:39.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:39.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:39.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:39.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:39.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:39.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:39.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:39.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:39.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:39.908 INFO:tasks.rados.rados.2.smithi204.stdout:3206: finishing rollback tid 1 to smithi20457400-7 2024-09-16T05:22:39.909 INFO:tasks.rados.rados.2.smithi204.stdout:3206: finishing rollback tid 2 to smithi20457400-7 2024-09-16T05:22:39.909 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1144 (ObjNum 1065 snap 324 seq_num 1065) dirty exists 2024-09-16T05:22:39.909 INFO:tasks.rados.rados.2.smithi204.stdout:3202: expect (ObjNum 1083 snap 329 seq_num 1083) 2024-09-16T05:22:39.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:39.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:40.041 INFO:tasks.rados.rados.2.smithi204.stdout:3207: finishing copy_from racing read to smithi20457400-28 2024-09-16T05:22:40.041 INFO:tasks.rados.rados.2.smithi204.stdout:3208: finishing write tid 1 to smithi20457400-37 2024-09-16T05:22:40.042 INFO:tasks.rados.rados.2.smithi204.stdout:3203: expect (ObjNum 1125 snap 337 seq_num 1125) 2024-09-16T05:22:40.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:39 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:40.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:39 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:40.075 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:39 smithi032 ceph-mon[27388]: pgmap v1374: 127 pgs: 127 active+clean; 356 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-16T05:22:40.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:40.089 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:40.095 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:40.095 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:40.112 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:40.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:40.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-16T05:22:40.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:39 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:39 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:40.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:39 smithi204 ceph-mon[30933]: pgmap v1374: 127 pgs: 127 active+clean; 356 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-16T05:22:40.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:40.215 INFO:tasks.rados.rados.2.smithi204.stdout:3207: finishing copy_from to smithi20457400-28 2024-09-16T05:22:40.215 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 750 (ObjNum 1087 snap 330 seq_num 1087) dirty exists 2024-09-16T05:22:40.216 INFO:tasks.rados.rados.2.smithi204.stdout:3208: finishing write tid 2 to smithi20457400-37 2024-09-16T05:22:40.216 INFO:tasks.rados.rados.2.smithi204.stdout:3208: finishing write tid 3 to smithi20457400-37 2024-09-16T05:22:40.216 INFO:tasks.rados.rados.2.smithi204.stdout:3208: finishing write tid 4 to smithi20457400-37 2024-09-16T05:22:40.216 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1055 (ObjNum 1130 snap 338 seq_num 1130) dirty exists 2024-09-16T05:22:40.216 INFO:tasks.rados.rados.2.smithi204.stdout:3208: left oid 37 (ObjNum 1130 snap 338 seq_num 1130) 2024-09-16T05:22:40.216 INFO:tasks.rados.rados.2.smithi204.stdout:3210: expect (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:22:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:39 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:39 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:39 smithi196 ceph-mon[30802]: pgmap v1374: 127 pgs: 127 active+clean; 356 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.5 MiB/s wr, 26 op/s 2024-09-16T05:22:40.238 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:40.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:40.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:40.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:40.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3202: done (8 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3203: done (7 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3205: done (6 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3206: done (5 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3207: done (4 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3208: done (3 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3209: done (2 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3210: done (1 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3211: done (0 left) 2024-09-16T05:22:40.428 INFO:tasks.rados.rados.2.smithi204.stdout:3212: write oid 7 current snap is 338 2024-09-16T05:22:40.429 INFO:tasks.rados.rados.2.smithi204.stdout:3212: seq_num 1131 ranges {608717=726863,1933605=748052} 2024-09-16T05:22:40.434 INFO:tasks.rados.rados.2.smithi204.stdout:3212: writing smithi20457400-7 from 608717 to 1335580 tid 1 2024-09-16T05:22:40.441 INFO:tasks.rados.rados.2.smithi204.stdout:3212: writing smithi20457400-7 from 1933605 to 2681657 tid 2 2024-09-16T05:22:40.442 INFO:tasks.rados.rados.2.smithi204.stdout:3213: write (excl) oid 44 current snap is 338 2024-09-16T05:22:40.442 INFO:tasks.rados.rados.2.smithi204.stdout:3213: seq_num 1132 ranges {625524=402112,1592087=608063,2869769=155755} 2024-09-16T05:22:40.447 INFO:tasks.rados.rados.2.smithi204.stdout:3213: writing smithi20457400-44 from 625524 to 1027636 tid 1 2024-09-16T05:22:40.453 INFO:tasks.rados.rados.2.smithi204.stdout:3213: writing smithi20457400-44 from 1592087 to 2200150 tid 2 2024-09-16T05:22:40.458 INFO:tasks.rados.rados.2.smithi204.stdout:3213: writing smithi20457400-44 from 2869769 to 3025524 tid 3 2024-09-16T05:22:40.458 INFO:tasks.rados.rados.2.smithi204.stdout:3214: read oid 6 snap -1 2024-09-16T05:22:40.458 INFO:tasks.rados.rados.2.smithi204.stdout:3214: expect (ObjNum 1097 snap 332 seq_num 1097) 2024-09-16T05:22:40.458 INFO:tasks.rados.rados.2.smithi204.stdout:3215: delete oid 48 current snap is 338 2024-09-16T05:22:40.458 INFO:tasks.rados.rados.2.smithi204.stdout:3212: finishing write tid 1 to smithi20457400-7 2024-09-16T05:22:40.459 INFO:tasks.rados.rados.2.smithi204.stdout:3212: finishing write tid 2 to smithi20457400-7 2024-09-16T05:22:40.459 INFO:tasks.rados.rados.2.smithi204.stdout:3212: finishing write tid 3 to smithi20457400-7 2024-09-16T05:22:40.459 INFO:tasks.rados.rados.2.smithi204.stdout:3212: finishing write tid 4 to smithi20457400-7 2024-09-16T05:22:40.459 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1147 (ObjNum 1131 snap 338 seq_num 1131) dirty exists 2024-09-16T05:22:40.459 INFO:tasks.rados.rados.2.smithi204.stdout:3212: left oid 7 (ObjNum 1131 snap 338 seq_num 1131) 2024-09-16T05:22:40.459 INFO:tasks.rados.rados.2.smithi204.stdout:3213: finishing write tid 1 to smithi20457400-44 2024-09-16T05:22:40.460 INFO:tasks.rados.rados.2.smithi204.stdout:3213: finishing write tid 2 to smithi20457400-44 2024-09-16T05:22:40.461 INFO:tasks.rados.rados.2.smithi204.stdout:3212: done (3 left) 2024-09-16T05:22:40.461 INFO:tasks.rados.rados.2.smithi204.stdout:3215: done (2 left) 2024-09-16T05:22:40.461 INFO:tasks.rados.rados.2.smithi204.stdout:3216: snap_create 2024-09-16T05:22:40.461 INFO:tasks.rados.rados.2.smithi204.stdout:3213: finishing write tid 3 to smithi20457400-44 2024-09-16T05:22:40.462 INFO:tasks.rados.rados.2.smithi204.stdout:3213: finishing write tid 4 to smithi20457400-44 2024-09-16T05:22:40.462 INFO:tasks.rados.rados.2.smithi204.stdout:3213: finishing write tid 5 to smithi20457400-44 2024-09-16T05:22:40.462 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1128 (ObjNum 1132 snap 338 seq_num 1132) dirty exists 2024-09-16T05:22:40.462 INFO:tasks.rados.rados.2.smithi204.stdout:3213: left oid 44 (ObjNum 1132 snap 338 seq_num 1132) 2024-09-16T05:22:40.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:40.465 INFO:tasks.rados.rados.2.smithi204.stdout:3214: expect (ObjNum 1097 snap 332 seq_num 1097) 2024-09-16T05:22:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:40.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:40.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:40.582 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:40.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:40.603 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:40.603 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:40.609 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:40.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:40.624 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:40.624 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:40.646 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:40.659 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:40.668 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:40.668 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:40.694 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:40.694 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:40.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:40.716 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:40.716 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:40.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:40.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:40.737 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:40.738 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:40.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:22:40.759 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:40.759 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:40.780 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:40.781 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:40.802 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:40.802 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:40.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:40.831 INFO:tasks.rados.rados.2.smithi204.stdout:3213: done (2 left) 2024-09-16T05:22:40.831 INFO:tasks.rados.rados.2.smithi204.stdout:3214: done (1 left) 2024-09-16T05:22:40.832 INFO:tasks.rados.rados.2.smithi204.stdout:3216: done (0 left) 2024-09-16T05:22:40.832 INFO:tasks.rados.rados.2.smithi204.stdout:3217: snap_create 2024-09-16T05:22:40.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:40.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:40.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:40.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:40.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:40.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:40.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:40 smithi196 ceph-mon[30802]: osdmap e810: 12 total, 12 up, 12 in 2024-09-16T05:22:40.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4220927532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:40.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:40 smithi196 ceph-mon[30802]: osdmap e811: 12 total, 12 up, 12 in 2024-09-16T05:22:41.034 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:41.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:41.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:41.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:41.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:40 smithi032 ceph-mon[27388]: osdmap e810: 12 total, 12 up, 12 in 2024-09-16T05:22:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4220927532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:40 smithi032 ceph-mon[27388]: osdmap e811: 12 total, 12 up, 12 in 2024-09-16T05:22:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:40 smithi204 ceph-mon[30933]: osdmap e810: 12 total, 12 up, 12 in 2024-09-16T05:22:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4220927532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:40 smithi204 ceph-mon[30933]: osdmap e811: 12 total, 12 up, 12 in 2024-09-16T05:22:41.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:41.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:41.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:41.329 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:41.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:41.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:41.454 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:41.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:41.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:41.537 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:22:41.537 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:22:41.537 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:22:41.537 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:22:41.538 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:22:41.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:41.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:41.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:41.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:41.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:41.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:22:41.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:41.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:41.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:41.835 INFO:tasks.rados.rados.2.smithi204.stdout:3217: done (0 left) 2024-09-16T05:22:41.835 INFO:tasks.rados.rados.2.smithi204.stdout:3218: delete oid 41 current snap is 340 2024-09-16T05:22:41.837 INFO:tasks.rados.rados.2.smithi204.stdout:3218: done (0 left) 2024-09-16T05:22:41.837 INFO:tasks.rados.rados.2.smithi204.stdout:3219: rollback oid 23 current snap is 340 2024-09-16T05:22:41.838 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 23 to 276 2024-09-16T05:22:41.838 INFO:tasks.rados.rados.2.smithi204.stdout:3220: read oid 12 snap -1 2024-09-16T05:22:41.838 INFO:tasks.rados.rados.2.smithi204.stdout:3220: expect deleted 2024-09-16T05:22:41.838 INFO:tasks.rados.rados.2.smithi204.stdout:3221: delete oid 5 current snap is 340 2024-09-16T05:22:41.839 INFO:tasks.rados.rados.2.smithi204.stdout:3220: done (2 left) 2024-09-16T05:22:41.839 INFO:tasks.rados.rados.2.smithi204.stdout:3221: done (1 left) 2024-09-16T05:22:41.839 INFO:tasks.rados.rados.2.smithi204.stdout:3222: snap_remove snap 339 2024-09-16T05:22:41.850 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:41.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:41.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:41.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:41.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:41.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:42.076 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:42.097 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:42.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:42.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:42.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:41 smithi196 ceph-mon[30802]: pgmap v1377: 127 pgs: 127 active+clean; 342 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-16T05:22:42.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:41 smithi196 ceph-mon[30802]: osdmap e812: 12 total, 12 up, 12 in 2024-09-16T05:22:42.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:22:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:41 smithi032 ceph-mon[27388]: pgmap v1377: 127 pgs: 127 active+clean; 342 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-16T05:22:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:41 smithi032 ceph-mon[27388]: osdmap e812: 12 total, 12 up, 12 in 2024-09-16T05:22:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:41 smithi204 ceph-mon[30933]: pgmap v1377: 127 pgs: 127 active+clean; 342 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-16T05:22:42.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:41 smithi204 ceph-mon[30933]: osdmap e812: 12 total, 12 up, 12 in 2024-09-16T05:22:42.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:42.222 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:42.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:42.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:42.346 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:42.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:42.384 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:42.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:42.471 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:42.471 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:42.557 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:42.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:42.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-16T05:22:42.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:42.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:42.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:42.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:42.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:42.814 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(2.34 e278 from shard 3)", 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 551.94569831199999, 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34868665500000001, 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.815 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866977+0000", 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9999999999999997e-07 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:30.866983+0000", 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.84e-06 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215470+0000", 2024-09-16T05:22:42.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34848719900000003 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215480+0000", 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2129999999999997e-06 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215584+0000", 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010428400000000001 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215601+0000", 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7509000000000001e-05 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:31.215663+0000", 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1909999999999995e-05 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2511 2.3 2.e1e51803 (undecoded) ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e320)", 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 497.989602977, 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99828526200000001, 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2511 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.820 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823073+0000", 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.04e-06 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823078+0000", 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6519999999999997e-06 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823091+0000", 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3277e-05 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823100+0000", 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4909999999999994e-06 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823141+0000", 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0187e-05 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823147+0000", 2024-09-16T05:22:42.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3929999999999996e-06 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821329+0000", 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99818146900000004 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821357+0000", 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8753000000000001e-05 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.824 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:136 2.34 e355/278 ondisk, result = 0)", 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 457.11090125099997, 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83858643600000005, 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:22:42.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701775+0000", 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6200000000000006e-07 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701786+0000", 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0815000000000001e-05 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540112+0000", 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83832645699999997 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540147+0000", 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5416000000000001e-05 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540246+0000", 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8171999999999995e-05 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540254+0000", 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:22:42.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540360+0000", 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010627400000000001 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3573 2.3 2:c018a787:::smithi20457400-18:head [write 1692526~473837] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:22:42.829 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.94693311399999, 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022224677000000002, 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3573 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:42.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:22:42.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865743+0000", 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0100000000000001e-06 2024-09-16T05:22:42.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866150+0000", 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040748299999999999 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866163+0000", 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2187999999999999e-05 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866168+0000", 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4689999999999996e-06 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866251+0000", 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2911000000000006e-05 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866320+0000", 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8596000000000005e-05 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866403+0000", 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3796999999999997e-05 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867984+0000", 2024-09-16T05:22:42.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015806399999999999 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887897+0000", 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019913157000000001 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887934+0000", 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6482000000000001e-05 2024-09-16T05:22:42.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887967+0000", 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2944000000000002e-05 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3574 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 2403500] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.94649356100001, 2024-09-16T05:22:42.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039246627999999999, 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3574 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:22:42.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866182+0000", 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866205+0000", 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2592e-05 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866222+0000", 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7422e-05 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866231+0000", 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5220000000000007e-06 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867407+0000", 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001176014 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867514+0000", 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106947 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867661+0000", 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000147396 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867987+0000", 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032622400000000001 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905316+0000", 2024-09-16T05:22:42.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037329222000000002 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905388+0000", 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1896000000000004e-05 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905428+0000", 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9830000000000003e-05 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.844 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3575 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.94643556400001, 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039343647000000002, 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.845 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3575 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866240+0000", 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7999999999999995e-07 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866243+0000", 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5770000000000001e-06 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866249+0000", 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:22:42.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866254+0000", 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-06 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867919+0000", 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016648609999999999 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867974+0000", 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4478000000000003e-05 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905464+0000", 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037490460000000003 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905516+0000", 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1894999999999997e-05 2024-09-16T05:22:42.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905583+0000", 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3580 2.34 2:2cffa4d6:::smithi20457400-50:head [write 1351008~617495] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.93613482000001, 2024-09-16T05:22:42.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030886027999999999, 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3580 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:22:42.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876542+0000", 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877114+0000", 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057156899999999996 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877126+0000", 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2751999999999999e-05 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877134+0000", 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.072e-06 2024-09-16T05:22:42.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0603000000000003e-05 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877240+0000", 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4561000000000002e-05 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:22:42.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877317+0000", 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7002999999999994e-05 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878479+0000", 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011619130000000001 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907362+0000", 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028883039999999999 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907396+0000", 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4807000000000001e-05 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907426+0000", 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9821000000000001e-05 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3581 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2420307] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.935524026, 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030393973000000001, 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3581 2024-09-16T05:22:42.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877152+0000", 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3099999999999996e-07 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877156+0000", 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7589999999999999e-06 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877169+0000", 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3129000000000001e-05 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877175+0000", 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6259999999999997e-06 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877997+0000", 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082123199999999997 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878081+0000", 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4914999999999996e-05 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878202+0000", 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120207 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878482+0000", 2024-09-16T05:22:42.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027995500000000003 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907462+0000", 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028980381 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907500+0000", 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907545+0000", 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5120999999999997e-05 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3582 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.93548962, 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030495267999999999, 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.863 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3582 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877186+0000", 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-07 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877189+0000", 2024-09-16T05:22:42.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6800000000000002e-06 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877196+0000", 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0650000000000001e-06 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877202+0000", 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.28e-06 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878412+0000", 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001210205 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878470+0000", 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7759e-05 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907572+0000", 2024-09-16T05:22:42.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029102086999999999 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907620+0000", 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8384000000000003e-05 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907681+0000", 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0244000000000001e-05 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.868 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:288 2.16 e446/273 ondisk, result = 0)", 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 365.92895826, 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65473014699999998, 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883718+0000", 2024-09-16T05:22:42.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3799999999999996e-07 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883722+0000", 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.284e-06 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538184+0000", 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.654462031 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538201+0000", 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6634e-05 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538327+0000", 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012605799999999999 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538338+0000", 2024-09-16T05:22:42.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1188e-05 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538447+0000", 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109014 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4460 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 511] snapc bc=[bc,bb,ba,b8] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:22:42.873 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 309.75337433700003, 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019695292, 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4460 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9799999999999999e-07 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059304+0000", 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5490000000000001e-06 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059308+0000", 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3610000000000003e-06 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059311+0000", 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9179999999999998e-06 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059364+0000", 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2790000000000001e-05 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059422+0000", 2024-09-16T05:22:42.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7929000000000001e-05 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.073174+0000", 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013752234 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.077028+0000", 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038540319999999999 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078924+0000", 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001896179 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078961+0000", 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7311000000000002e-05 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078996+0000", 2024-09-16T05:22:42.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4591000000000001e-05 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5587 2.34 2:2d3f13f8:::smithi20457400-7:head [setxattr _header (57) in=7b,truncate 3260723] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 230.00683666800001, 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.147062895, 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.880 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5587 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805839+0000", 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8299999999999997e-07 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805842+0000", 2024-09-16T05:22:42.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6790000000000001e-06 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805846+0000", 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4089999999999996e-06 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805849+0000", 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.191e-06 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806887+0000", 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010375440000000001 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806953+0000", 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5833000000000005e-05 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807079+0000", 2024-09-16T05:22:42.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012666500000000001 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.809689+0000", 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026092860000000002 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952747+0000", 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14305815 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952826+0000", 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9773999999999999e-05 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952901+0000", 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4781000000000001e-05 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5588 2.34 2:2d3f13f8:::smithi20457400-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 230.00681019500001, 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14722005799999999, 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5588 2024-09-16T05:22:42.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.65e-07 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805867+0000", 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.86e-06 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805872+0000", 2024-09-16T05:22:42.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805876+0000", 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0380000000000001e-06 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807396+0000", 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001520448 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807470+0000", 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3695000000000005e-05 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952937+0000", 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14546690300000001 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953007+0000", 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9801999999999993e-05 2024-09-16T05:22:42.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953085+0000", 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8121999999999995e-05 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:534 2.34 e617/278 ondisk, result = 0)", 2024-09-16T05:22:42.892 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 196.000520059, 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93956844500000003, 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:22:42.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812156+0000", 2024-09-16T05:22:42.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0599999999999999e-07 2024-09-16T05:22:42.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812160+0000", 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.437e-06 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751436+0000", 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93927549399999999 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751483+0000", 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7117999999999998e-05 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751598+0000", 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115448 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751609+0000", 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1156e-05 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751723+0000", 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113986 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7360 2.3 2:c018a787:::smithi20457400-18:head [write 443982~796776,stat] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:22:42.899 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 65.915721399999995, 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.046574619999999997, 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7360 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896958+0000", 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3869999999999997e-06 2024-09-16T05:22:42.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897803+0000", 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084549400000000002 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897817+0000", 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3172e-05 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897822+0000", 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7400000000000001e-06 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897880+0000", 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7275999999999998e-05 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897887+0000", 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0450000000000003e-06 2024-09-16T05:22:42.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899506+0000", 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016191300000000001 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899612+0000", 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010592200000000001 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899807+0000", 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019501699999999999 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.900940+0000", 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011327760000000001 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943442+0000", 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.042502861000000003 2024-09-16T05:22:42.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943488+0000", 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5266000000000003e-05 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943528+0000", 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0534000000000002e-05 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7361 2.3 2:c018a787:::smithi20457400-18:head [write 1989231~479018] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 65.909056520000007, 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040030532000000001, 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7361 2024-09-16T05:22:42.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903620+0000", 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.885e-06 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904044+0000", 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042330699999999999 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904056+0000", 2024-09-16T05:22:42.909 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2088e-05 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904064+0000", 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3820000000000004e-06 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904117+0000", 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2352000000000003e-05 2024-09-16T05:22:42.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904226+0000", 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109895 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904403+0000", 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017676899999999999 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.905359+0000", 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00095604999999999996 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943572+0000", 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038213241000000002 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943611+0000", 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8668999999999999e-05 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943649+0000", 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7894e-05 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7362 2.3 2:c018a787:::smithi20457400-18:head [write 2959167~284815] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:22:42.913 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 65.904806149999999, 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035905712999999999, 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7362 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907870+0000", 2024-09-16T05:22:42.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4200000000000005e-07 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908124+0000", 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025446300000000002 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908135+0000", 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0964999999999999e-05 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908143+0000", 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5190000000000003e-06 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9714e-05 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908239+0000", 2024-09-16T05:22:42.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6212999999999999e-05 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908346+0000", 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010734999999999999 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909154+0000", 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080830699999999997 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943689+0000", 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034534978000000001 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943732+0000", 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2558999999999997e-05 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943775+0000", 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2803000000000003e-05 2024-09-16T05:22:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7363 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 3243982] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 65.904517850999994, 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035723752999999997, 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:42.920 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7363 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908158+0000", 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908161+0000", 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1410000000000001e-06 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908167+0000", 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.674e-06 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908173+0000", 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4149999999999998e-06 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908684+0000", 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051148000000000003 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908742+0000", 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7358999999999997e-05 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908839+0000", 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6930999999999997e-05 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909158+0000", 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031886399999999997 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943805+0000", 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034647491000000002 2024-09-16T05:22:42.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943842+0000", 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7004e-05 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943881+0000", 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8868999999999997e-05 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.926 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7364 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 65.904492990999998, 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035867990000000002, 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.927 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7364 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5999999999999999e-07 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908185+0000", 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1600000000000001e-06 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908191+0000", 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6869999999999997e-06 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908196+0000", 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9989999999999999e-06 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909088+0000", 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089121600000000001 2024-09-16T05:22:42.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909143+0000", 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5887000000000002e-05 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943913+0000", 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034769683000000003 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943981+0000", 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.792e-05 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.944050+0000", 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8978000000000003e-05 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.932 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:830 2.3 e799/329 ondisk, result = 0)", 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 13.942410257000001, 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.330002407, 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870266+0000", 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.223e-06 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:42.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870277+0000", 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0560999999999999e-05 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.199989+0000", 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32971226399999998 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200004+0000", 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5366000000000002e-05 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200139+0000", 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013504000000000001 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200150+0000", 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0305999999999999e-05 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:42.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200267+0000", 2024-09-16T05:22:42.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011764699999999999 2024-09-16T05:22:42.937 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.937 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.937 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.937 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:42.937 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:42.937 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:22:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:3222: done (1 left) 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:3223: read oid 34 snap -1 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:3223: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:3224: snap_create 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:3219: finishing rollback tid 0 to smithi20457400-23 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:3219: finishing rollback tid 1 to smithi20457400-23 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:3219: finishing rollback tid 2 to smithi20457400-23 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 753 (ObjNum 899 snap 263 seq_num 899) dirty exists 2024-09-16T05:22:42.938 INFO:tasks.rados.rados.2.smithi204.stdout:3223: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:22:42.939 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:42.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:43.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:43.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:43.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:43.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:43.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:43.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:43.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:43.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:43.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:43.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:43.433 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:43.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:43.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:43.559 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:43.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:43.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:22:43.660 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:43.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:43.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:43.785 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:43.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:43.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:43.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:22:43.833 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:22:43.836 INFO:tasks.rados.rados.2.smithi204.stdout:3219: done (2 left) 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3223: done (1 left) 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3224: done (0 left) 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3225: read oid 30 snap 331 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3225: expect (ObjNum 1051 snap 318 seq_num 1051) 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3226: rollback oid 22 current snap is 341 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 22 to 336 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3227: read oid 46 snap 276 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3227: expect (ObjNum 855 snap 248 seq_num 855) 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3228: read oid 36 snap -1 2024-09-16T05:22:43.837 INFO:tasks.rados.rados.2.smithi204.stdout:3228: expect (ObjNum 1126 snap 337 seq_num 1126) 2024-09-16T05:22:43.838 INFO:tasks.rados.rados.2.smithi204.stdout:3229: copy_from oid 25 from oid 34 current snap is 341 2024-09-16T05:22:43.838 INFO:tasks.rados.rados.2.smithi204.stdout:3230: delete oid 47 current snap is 341 2024-09-16T05:22:43.840 INFO:tasks.rados.rados.2.smithi204.stdout:3226: finishing rollback tid 0 to smithi20457400-22 2024-09-16T05:22:43.840 INFO:tasks.rados.rados.2.smithi204.stdout:3226: finishing rollback tid 1 to smithi20457400-22 2024-09-16T05:22:43.841 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 816 (ObjNum 33 snap 0 seq_num 0) dirty dne 2024-09-16T05:22:43.841 INFO:tasks.rados.rados.2.smithi204.stdout:3228: expect (ObjNum 1126 snap 337 seq_num 1126) 2024-09-16T05:22:43.856 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:43.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:43.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:43.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:43.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:44.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:44.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:44.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:44.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:43 smithi196 ceph-mon[30802]: pgmap v1380: 127 pgs: 127 active+clean; 330 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-16T05:22:44.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:43 smithi196 ceph-mon[30802]: osdmap e813: 12 total, 12 up, 12 in 2024-09-16T05:22:44.112 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1407397511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:44.117 INFO:tasks.rados.rados.2.smithi204.stdout:3229: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:22:44.117 INFO:tasks.rados.rados.2.smithi204.stdout:3227: expect (ObjNum 855 snap 248 seq_num 855) 2024-09-16T05:22:44.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:43 smithi032 ceph-mon[27388]: pgmap v1380: 127 pgs: 127 active+clean; 330 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-16T05:22:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:43 smithi032 ceph-mon[27388]: osdmap e813: 12 total, 12 up, 12 in 2024-09-16T05:22:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1407397511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:43 smithi204 ceph-mon[30933]: pgmap v1380: 127 pgs: 127 active+clean; 330 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2024-09-16T05:22:44.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:43 smithi204 ceph-mon[30933]: osdmap e813: 12 total, 12 up, 12 in 2024-09-16T05:22:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1407397511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:44.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:44.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:44.256 INFO:tasks.rados.rados.2.smithi204.stdout:3225: expect (ObjNum 1051 snap 318 seq_num 1051) 2024-09-16T05:22:44.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:44.304 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:44.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:44.313 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:22:44.313 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:22:44.313 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:22:44.313 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:22:44.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:44.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:44.430 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:44.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:3229: finishing copy_from to smithi20457400-25 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1265 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:3225: done (5 left) 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:3226: done (4 left) 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:3227: done (3 left) 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:3228: done (2 left) 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:3229: done (1 left) 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:3230: done (0 left) 2024-09-16T05:22:44.471 INFO:tasks.rados.rados.2.smithi204.stdout:3231: rollback oid 2 current snap is 341 2024-09-16T05:22:44.472 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 2 to 331 2024-09-16T05:22:44.472 INFO:tasks.rados.rados.2.smithi204.stdout:3232: copy_from oid 15 from oid 5 current snap is 341 2024-09-16T05:22:44.472 INFO:tasks.rados.rados.2.smithi204.stdout:3233: copy_from oid 11 from oid 45 current snap is 341 2024-09-16T05:22:44.472 INFO:tasks.rados.rados.2.smithi204.stdout:3234: write (excl) oid 4 current snap is 341 2024-09-16T05:22:44.472 INFO:tasks.rados.rados.2.smithi204.stdout:3234: seq_num 1137 ranges {709559=494710,1884497=759059} 2024-09-16T05:22:44.477 INFO:tasks.rados.rados.2.smithi204.stdout:3234: writing smithi20457400-4 from 709559 to 1204269 tid 1 2024-09-16T05:22:44.485 INFO:tasks.rados.rados.2.smithi204.stdout:3234: writing smithi20457400-4 from 1884497 to 2643556 tid 2 2024-09-16T05:22:44.486 INFO:tasks.rados.rados.2.smithi204.stdout:3235: snap_remove snap 276 2024-09-16T05:22:44.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:44.553 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:44.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:44.657 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:44.680 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:44.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:44.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:44.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:44.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:44.850 INFO:tasks.rados.rados.2.smithi204.stdout:3235: done (4 left) 2024-09-16T05:22:44.850 INFO:tasks.rados.rados.2.smithi204.stdout:3236: snap_remove snap 340 2024-09-16T05:22:44.882 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1407397511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:44.883 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:44 smithi032 ceph-mon[27388]: osdmap e814: 12 total, 12 up, 12 in 2024-09-16T05:22:44.883 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:44 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:44.883 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:44 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:44.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:44.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:44.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:44.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1407397511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:44.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:44 smithi196 ceph-mon[30802]: osdmap e814: 12 total, 12 up, 12 in 2024-09-16T05:22:44.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:44 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:44.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:44 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:44.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:22:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:45.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:45.054 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:45.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:45.155 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1407397511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:44 smithi204 ceph-mon[30933]: osdmap e814: 12 total, 12 up, 12 in 2024-09-16T05:22:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:44 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:45.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:44 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:45.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:45.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:45.260 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:45.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:45.302 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:45.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:45.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:45.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:45.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:45.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:45.537 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:45.537 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:45.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:45.559 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:45.559 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:45.579 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:45.580 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:45.600 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:45.600 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:45.621 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:45.621 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:45.642 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:45.643 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:45.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:45.663 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:45.664 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:45.677 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:45.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:45.684 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:45.706 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:45.707 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:45.728 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:45.749 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:45.764 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:45.764 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:45.770 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:45.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:45.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:45.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:45.834 INFO:tasks.rados.rados.2.smithi204.stdout:3236: done (4 left) 2024-09-16T05:22:45.834 INFO:tasks.rados.rados.2.smithi204.stdout:3237: copy_from oid 21 from oid 29 current snap is 341 2024-09-16T05:22:45.834 INFO:tasks.rados.rados.2.smithi204.stdout:3238: copy_from oid 34 from oid 32 current snap is 341 2024-09-16T05:22:45.834 INFO:tasks.rados.rados.2.smithi204.stdout:3239: delete oid 10 current snap is 341 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3231: finishing rollback tid 0 to smithi20457400-2 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3231: finishing rollback tid 1 to smithi20457400-2 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3231: finishing rollback tid 2 to smithi20457400-2 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 536 (ObjNum 1064 snap 324 seq_num 1064) dirty exists 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3232: finishing copy_from to smithi20457400-15 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3232: got expected ENOENT (src dne) 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3232: finishing copy_from racing read to smithi20457400-15 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3233: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3233: finishing copy_from to smithi20457400-11 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1240 (ObjNum 1097 snap 332 seq_num 1097) dirty exists 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3234: finishing write tid 1 to smithi20457400-4 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3234: finishing write tid 2 to smithi20457400-4 2024-09-16T05:22:45.835 INFO:tasks.rados.rados.2.smithi204.stdout:3234: finishing write tid 3 to smithi20457400-4 2024-09-16T05:22:45.836 INFO:tasks.rados.rados.2.smithi204.stdout:3234: finishing write tid 4 to smithi20457400-4 2024-09-16T05:22:45.836 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1356 (ObjNum 1137 snap 341 seq_num 1137) dirty exists 2024-09-16T05:22:45.836 INFO:tasks.rados.rados.2.smithi204.stdout:3234: left oid 4 (ObjNum 1137 snap 341 seq_num 1137) 2024-09-16T05:22:45.838 INFO:tasks.rados.rados.2.smithi204.stdout:3231: done (6 left) 2024-09-16T05:22:45.838 INFO:tasks.rados.rados.2.smithi204.stdout:3232: done (5 left) 2024-09-16T05:22:45.838 INFO:tasks.rados.rados.2.smithi204.stdout:3233: done (4 left) 2024-09-16T05:22:45.838 INFO:tasks.rados.rados.2.smithi204.stdout:3234: done (3 left) 2024-09-16T05:22:45.839 INFO:tasks.rados.rados.2.smithi204.stdout:3239: done (2 left) 2024-09-16T05:22:45.839 INFO:tasks.rados.rados.2.smithi204.stdout:3240: copy_from oid 25 from oid 42 current snap is 341 2024-09-16T05:22:45.839 INFO:tasks.rados.rados.2.smithi204.stdout:3241: read oid 19 snap -1 2024-09-16T05:22:45.839 INFO:tasks.rados.rados.2.smithi204.stdout:3241: expect (ObjNum 1020 snap 310 seq_num 1020) 2024-09-16T05:22:45.840 INFO:tasks.rados.rados.2.smithi204.stdout:3242: write oid 45 current snap is 341 2024-09-16T05:22:45.840 INFO:tasks.rados.rados.2.smithi204.stdout:3242: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-09-16T05:22:45.848 INFO:tasks.rados.rados.2.smithi204.stdout:3242: writing smithi20457400-45 from 743173 to 1388381 tid 1 2024-09-16T05:22:45.856 INFO:tasks.rados.rados.2.smithi204.stdout:3242: writing smithi20457400-45 from 1885108 to 2647836 tid 2 2024-09-16T05:22:45.856 INFO:tasks.rados.rados.2.smithi204.stdout:3243: snap_remove snap 336 2024-09-16T05:22:45.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:45.925 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:45.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:45 smithi196 ceph-mon[30802]: pgmap v1383: 127 pgs: 127 active+clean; 319 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-16T05:22:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:45 smithi196 ceph-mon[30802]: osdmap e815: 12 total, 12 up, 12 in 2024-09-16T05:22:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3895050734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:45 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:45.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:45 smithi196 ceph-mon[30802]: osdmap e816: 12 total, 12 up, 12 in 2024-09-16T05:22:46.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:46.048 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:46.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:46.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:45 smithi032 ceph-mon[27388]: pgmap v1383: 127 pgs: 127 active+clean; 319 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-16T05:22:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:45 smithi032 ceph-mon[27388]: osdmap e815: 12 total, 12 up, 12 in 2024-09-16T05:22:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3895050734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:45 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:45 smithi032 ceph-mon[27388]: osdmap e816: 12 total, 12 up, 12 in 2024-09-16T05:22:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:45 smithi204 ceph-mon[30933]: pgmap v1383: 127 pgs: 127 active+clean; 319 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-16T05:22:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:45 smithi204 ceph-mon[30933]: osdmap e815: 12 total, 12 up, 12 in 2024-09-16T05:22:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3895050734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:45 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:45 smithi204 ceph-mon[30933]: osdmap e816: 12 total, 12 up, 12 in 2024-09-16T05:22:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:46.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:46.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:46.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:46.297 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:46.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:46.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:22:46.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:46.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:46.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:46.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:46.546 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:46.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:46.601 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:46.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:46.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:46.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:46.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:46.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:46.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:46.887 INFO:tasks.rados.rados.2.smithi204.stdout:3243: done (5 left) 2024-09-16T05:22:46.887 INFO:tasks.rados.rados.2.smithi204.stdout:3244: snap_remove snap 338 2024-09-16T05:22:46.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:46.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:22:46.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:46.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:47.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:47.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:47.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:47.062 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:47.062 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:47.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:47.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:47.167 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:47.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:47.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:47.294 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:47.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:47.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:47.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:47.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:47.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:47.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:47.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:47.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:47.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:22:47.668 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:47.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:47.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:47.796 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:47.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:47.885 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:22:47.892 INFO:tasks.rados.rados.2.smithi204.stdout:3244: done (5 left) 2024-09-16T05:22:47.892 INFO:tasks.rados.rados.2.smithi204.stdout:3245: read oid 17 snap 331 2024-09-16T05:22:47.892 INFO:tasks.rados.rados.2.smithi204.stdout:3245: expect (ObjNum 961 snap 284 seq_num 961) 2024-09-16T05:22:47.892 INFO:tasks.rados.rados.2.smithi204.stdout:3246: write (excl) oid 3 current snap is 341 2024-09-16T05:22:47.892 INFO:tasks.rados.rados.2.smithi204.stdout:3240: finishing copy_from to smithi20457400-25 2024-09-16T05:22:47.892 INFO:tasks.rados.rados.2.smithi204.stdout:3240: got expected ENOENT (src dne) 2024-09-16T05:22:47.892 INFO:tasks.rados.rados.2.smithi204.stdout:3240: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:22:47.892 INFO:tasks.rados.rados.2.smithi204.stdout:3241: expect (ObjNum 1020 snap 310 seq_num 1020) 2024-09-16T05:22:47.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:47.906 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:47.925 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:47.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:47.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:47 smithi196 ceph-mon[30802]: pgmap v1386: 127 pgs: 127 active+clean; 321 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 285 KiB/s wr, 18 op/s 2024-09-16T05:22:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:47 smithi196 ceph-mon[30802]: osdmap e817: 12 total, 12 up, 12 in 2024-09-16T05:22:47.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2968673514' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:22:48.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:48.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:48.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:48.079 INFO:tasks.rados.rados.2.smithi204.stdout:3238: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:22:48.079 INFO:tasks.rados.rados.2.smithi204.stdout:3242: finishing write tid 1 to smithi20457400-45 2024-09-16T05:22:48.079 INFO:tasks.rados.rados.2.smithi204.stdout:3238: finishing copy_from to smithi20457400-34 2024-09-16T05:22:48.079 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1206 (ObjNum 1109 snap 336 seq_num 1109) dirty exists 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:3237: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:3237: finishing copy_from to smithi20457400-21 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 1174 (ObjNum 935 snap 277 seq_num 935) dirty exists 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:3242: finishing write tid 2 to smithi20457400-45 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:3242: finishing write tid 3 to smithi20457400-45 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:3242: finishing write tid 4 to smithi20457400-45 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1266 (ObjNum 1139 snap 341 seq_num 1139) dirty exists 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:3242: left oid 45 (ObjNum 1139 snap 341 seq_num 1139) 2024-09-16T05:22:48.080 INFO:tasks.rados.rados.2.smithi204.stdout:3246: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2024-09-16T05:22:48.087 INFO:tasks.rados.rados.2.smithi204.stdout:3246: writing smithi20457400-3 from 759980 to 1480437 tid 1 2024-09-16T05:22:48.092 INFO:tasks.rados.rados.2.smithi204.stdout:3246: writing smithi20457400-3 from 1943590 to 2449976 tid 2 2024-09-16T05:22:48.097 INFO:tasks.rados.rados.2.smithi204.stdout:3246: writing smithi20457400-3 from 2856447 to 3159980 tid 3 2024-09-16T05:22:48.097 INFO:tasks.rados.rados.2.smithi204.stdout:3237: done (6 left) 2024-09-16T05:22:48.097 INFO:tasks.rados.rados.2.smithi204.stdout:3238: done (5 left) 2024-09-16T05:22:48.097 INFO:tasks.rados.rados.2.smithi204.stdout:3240: done (4 left) 2024-09-16T05:22:48.097 INFO:tasks.rados.rados.2.smithi204.stdout:3241: done (3 left) 2024-09-16T05:22:48.098 INFO:tasks.rados.rados.2.smithi204.stdout:3242: done (2 left) 2024-09-16T05:22:48.098 INFO:tasks.rados.rados.2.smithi204.stdout:3247: write oid 37 current snap is 341 2024-09-16T05:22:48.098 INFO:tasks.rados.rados.2.smithi204.stdout:3247: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-09-16T05:22:48.104 INFO:tasks.rados.rados.2.smithi204.stdout:3247: writing smithi20457400-37 from 776787 to 1288846 tid 1 2024-09-16T05:22:48.111 INFO:tasks.rados.rados.2.smithi204.stdout:3247: writing smithi20457400-37 from 1718425 to 2484822 tid 2 2024-09-16T05:22:48.115 INFO:tasks.rados.rados.2.smithi204.stdout:3247: writing smithi20457400-37 from 3000223 to 3176787 tid 3 2024-09-16T05:22:48.115 INFO:tasks.rados.rados.2.smithi204.stdout:3248: copy_from oid 27 from oid 25 current snap is 341 2024-09-16T05:22:48.115 INFO:tasks.rados.rados.2.smithi204.stdout:3245: expect (ObjNum 961 snap 284 seq_num 961) 2024-09-16T05:22:48.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:47 smithi032 ceph-mon[27388]: pgmap v1386: 127 pgs: 127 active+clean; 321 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 285 KiB/s wr, 18 op/s 2024-09-16T05:22:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:47 smithi032 ceph-mon[27388]: osdmap e817: 12 total, 12 up, 12 in 2024-09-16T05:22:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2968673514' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:22:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:47 smithi204 ceph-mon[30933]: pgmap v1386: 127 pgs: 127 active+clean; 321 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 285 KiB/s wr, 18 op/s 2024-09-16T05:22:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:47 smithi204 ceph-mon[30933]: osdmap e817: 12 total, 12 up, 12 in 2024-09-16T05:22:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2968673514' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:22:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:48.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:48.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:48.231 INFO:tasks.rados.rados.2.smithi204.stdout:3246: finishing write tid 1 to smithi20457400-3 2024-09-16T05:22:48.232 INFO:tasks.rados.rados.2.smithi204.stdout:3246: finishing write tid 2 to smithi20457400-3 2024-09-16T05:22:48.232 INFO:tasks.rados.rados.2.smithi204.stdout:3246: finishing write tid 3 to smithi20457400-3 2024-09-16T05:22:48.232 INFO:tasks.rados.rados.2.smithi204.stdout:3246: finishing write tid 4 to smithi20457400-3 2024-09-16T05:22:48.232 INFO:tasks.rados.rados.2.smithi204.stdout:3246: finishing write tid 5 to smithi20457400-3 2024-09-16T05:22:48.232 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1096 (ObjNum 1140 snap 341 seq_num 1140) dirty exists 2024-09-16T05:22:48.232 INFO:tasks.rados.rados.2.smithi204.stdout:3246: left oid 3 (ObjNum 1140 snap 341 seq_num 1140) 2024-09-16T05:22:48.232 INFO:tasks.rados.rados.2.smithi204.stdout:3247: finishing write tid 1 to smithi20457400-37 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3245: done (3 left) 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3246: done (2 left) 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3249: copy_from oid 44 from oid 1 current snap is 341 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3247: finishing write tid 2 to smithi20457400-37 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3247: finishing write tid 3 to smithi20457400-37 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3247: finishing write tid 4 to smithi20457400-37 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3247: finishing write tid 5 to smithi20457400-37 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1059 (ObjNum 1141 snap 341 seq_num 1141) dirty exists 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3247: left oid 37 (ObjNum 1141 snap 341 seq_num 1141) 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3248: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3248: finishing copy_from to smithi20457400-27 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 1265 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-09-16T05:22:48.233 INFO:tasks.rados.rados.2.smithi204.stdout:3247: done (2 left) 2024-09-16T05:22:48.234 INFO:tasks.rados.rados.2.smithi204.stdout:3248: done (1 left) 2024-09-16T05:22:48.234 INFO:tasks.rados.rados.2.smithi204.stdout:3250: write (excl) oid 47 current snap is 341 2024-09-16T05:22:48.234 INFO:tasks.rados.rados.2.smithi204.stdout:3250: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-09-16T05:22:48.239 INFO:tasks.rados.rados.2.smithi204.stdout:3250: writing smithi20457400-47 from 793594 to 1380902 tid 1 2024-09-16T05:22:48.246 INFO:tasks.rados.rados.2.smithi204.stdout:3250: writing smithi20457400-47 from 2060554 to 2570609 tid 2 2024-09-16T05:22:48.246 INFO:tasks.rados.rados.2.smithi204.stdout:3251: delete oid 40 current snap is 341 2024-09-16T05:22:48.246 INFO:tasks.rados.rados.2.smithi204.stdout:3249: finishing copy_from racing read to smithi20457400-44 2024-09-16T05:22:48.246 INFO:tasks.rados.rados.2.smithi204.stdout:3249: finishing copy_from to smithi20457400-44 2024-09-16T05:22:48.247 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1141 (ObjNum 1100 snap 332 seq_num 1100) dirty exists 2024-09-16T05:22:48.247 INFO:tasks.rados.rados.2.smithi204.stdout:3250: finishing write tid 1 to smithi20457400-47 2024-09-16T05:22:48.248 INFO:tasks.rados.rados.2.smithi204.stdout:3249: done (2 left) 2024-09-16T05:22:48.248 INFO:tasks.rados.rados.2.smithi204.stdout:3251: done (1 left) 2024-09-16T05:22:48.248 INFO:tasks.rados.rados.2.smithi204.stdout:3252: write (excl) oid 37 current snap is 341 2024-09-16T05:22:48.248 INFO:tasks.rados.rados.2.smithi204.stdout:3252: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2024-09-16T05:22:48.253 INFO:tasks.rados.rados.2.smithi204.stdout:3252: writing smithi20457400-37 from 427208 to 1165014 tid 1 2024-09-16T05:22:48.259 INFO:tasks.rados.rados.2.smithi204.stdout:3252: writing smithi20457400-37 from 1777518 to 2291242 tid 2 2024-09-16T05:22:48.265 INFO:tasks.rados.rados.2.smithi204.stdout:3252: writing smithi20457400-37 from 2966139 to 3227208 tid 3 2024-09-16T05:22:48.265 INFO:tasks.rados.rados.2.smithi204.stdout:3253: snap_remove snap 279 2024-09-16T05:22:48.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:48.296 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:48.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:48.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:48.384 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:48.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:48.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:48.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:48.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:22:48.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:48.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:48.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:48.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:48.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:48.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:48.723 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:48.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:48.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:48.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:48.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:3253: done (2 left) 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:3254: read oid 48 snap -1 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:3254: expect deleted 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:3255: copy_from oid 34 from oid 45 current snap is 341 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:3256: rollback oid 14 current snap is 341 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 331 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:3250: finishing write tid 2 to smithi20457400-47 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:3250: finishing write tid 3 to smithi20457400-47 2024-09-16T05:22:48.902 INFO:tasks.rados.rados.2.smithi204.stdout:3250: finishing write tid 4 to smithi20457400-47 2024-09-16T05:22:48.903 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1062 (ObjNum 1142 snap 341 seq_num 1142) dirty exists 2024-09-16T05:22:48.903 INFO:tasks.rados.rados.2.smithi204.stdout:3250: left oid 47 (ObjNum 1142 snap 341 seq_num 1142) 2024-09-16T05:22:48.903 INFO:tasks.rados.rados.2.smithi204.stdout:3252: finishing write tid 1 to smithi20457400-37 2024-09-16T05:22:48.903 INFO:tasks.rados.rados.2.smithi204.stdout:3250: done (4 left) 2024-09-16T05:22:48.903 INFO:tasks.rados.rados.2.smithi204.stdout:3257: copy_from oid 17 from oid 2 current snap is 341 2024-09-16T05:22:48.903 INFO:tasks.rados.rados.2.smithi204.stdout:3258: snap_remove snap 326 2024-09-16T05:22:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:48.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:48.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:48.977 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2968673514' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:48.977 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:48 smithi196 ceph-mon[30802]: osdmap e818: 12 total, 12 up, 12 in 2024-09-16T05:22:48.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:48 smithi196 ceph-mon[30802]: 2.57 scrub starts 2024-09-16T05:22:48.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:48 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:22:48.978 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:48 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:22:49.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:22:49.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:49.048 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:49.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:49.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2968673514' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:49.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:48 smithi032 ceph-mon[27388]: osdmap e818: 12 total, 12 up, 12 in 2024-09-16T05:22:49.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:48 smithi032 ceph-mon[27388]: 2.57 scrub starts 2024-09-16T05:22:49.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:48 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:22:49.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:48 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:22:49.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2968673514' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:22:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:48 smithi204 ceph-mon[30933]: osdmap e818: 12 total, 12 up, 12 in 2024-09-16T05:22:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:48 smithi204 ceph-mon[30933]: 2.57 scrub starts 2024-09-16T05:22:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:48 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:22:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:48 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:22:49.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:49.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:49.244 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:49.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:49.290 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:22:49.290 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:22:49.290 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-16T05:22:49.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:22:49.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:49.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:49.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:49.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:49.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:49.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:49.542 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:49.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:49.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:49.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:49.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:49.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:49.764 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:49.764 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:49.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:49.791 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:49.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:49.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:49.909 INFO:tasks.rados.rados.2.smithi204.stdout:3258: done (5 left) 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3259: read oid 24 snap 331 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3259: expect deleted 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3260: rollback oid 49 current snap is 341 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 331 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3252: finishing write tid 2 to smithi20457400-37 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3252: finishing write tid 3 to smithi20457400-37 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3252: finishing write tid 4 to smithi20457400-37 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3252: finishing write tid 5 to smithi20457400-37 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1066 (ObjNum 1144 snap 341 seq_num 1144) dirty exists 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3252: left oid 37 (ObjNum 1144 snap 341 seq_num 1144) 2024-09-16T05:22:49.910 INFO:tasks.rados.rados.2.smithi204.stdout:3256: finishing rollback tid 0 to smithi20457400-14 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3256: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3256: finishing rollback tid 2 to smithi20457400-14 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1430 (ObjNum 1080 snap 329 seq_num 1080) dirty exists 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3255: finishing copy_from racing read to smithi20457400-34 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3257: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3255: finishing copy_from to smithi20457400-34 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1266 (ObjNum 1139 snap 341 seq_num 1139) dirty exists 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3257: finishing copy_from to smithi20457400-17 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 536 (ObjNum 1064 snap 324 seq_num 1064) dirty exists 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3252: done (6 left) 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3254: done (5 left) 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3255: done (4 left) 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3256: done (3 left) 2024-09-16T05:22:49.911 INFO:tasks.rados.rados.2.smithi204.stdout:3257: done (2 left) 2024-09-16T05:22:49.912 INFO:tasks.rados.rados.2.smithi204.stdout:3261: snap_create 2024-09-16T05:22:49.914 INFO:tasks.rados.rados.2.smithi204.stdout:3260: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:22:49.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 2.3f scrub starts 2024-09-16T05:22:49.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 2.3f scrub ok 2024-09-16T05:22:49.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: pgmap v1389: 127 pgs: 127 active+clean; 317 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2024-09-16T05:22:49.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: osdmap e819: 12 total, 12 up, 12 in 2024-09-16T05:22:49.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 2.57 scrub ok 2024-09-16T05:22:49.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:22:49.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:22:49.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1466458670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:49.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 2.36 scrub starts 2024-09-16T05:22:49.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 2.36 scrub ok 2024-09-16T05:22:49.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:22:49.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:49 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:22:49.919 INFO:tasks.rados.rados.2.smithi204.stdout:3260: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:22:49.919 INFO:tasks.rados.rados.2.smithi204.stdout:3260: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:22:49.920 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 660 (ObjNum 921 snap 273 seq_num 921) dirty exists 2024-09-16T05:22:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:49.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:49.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 2.3f scrub starts 2024-09-16T05:22:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 2.3f scrub ok 2024-09-16T05:22:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: pgmap v1389: 127 pgs: 127 active+clean; 317 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2024-09-16T05:22:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: osdmap e819: 12 total, 12 up, 12 in 2024-09-16T05:22:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 2.57 scrub ok 2024-09-16T05:22:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:22:49.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:22:49.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1466458670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:49.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 2.36 scrub starts 2024-09-16T05:22:49.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 2.36 scrub ok 2024-09-16T05:22:49.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:22:49.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:49 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:22:50.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:50.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:50.071 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:50.071 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:50.094 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:50.094 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:50.115 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:50.115 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:50.137 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:50.137 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:50.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:50.159 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:50.159 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:50.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 2.3f scrub starts 2024-09-16T05:22:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 2.3f scrub ok 2024-09-16T05:22:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: pgmap v1389: 127 pgs: 127 active+clean; 317 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2024-09-16T05:22:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: osdmap e819: 12 total, 12 up, 12 in 2024-09-16T05:22:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 2.57 scrub ok 2024-09-16T05:22:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:22:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:22:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1466458670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 2.36 scrub starts 2024-09-16T05:22:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 2.36 scrub ok 2024-09-16T05:22:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:22:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:49 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:22:50.180 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:50.181 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:50.202 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:50.203 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:50.225 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:50.225 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:50.246 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:50.246 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:50.267 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:50.268 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:50.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:50.290 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:50.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:50.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:50.312 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:50.348 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:22:50.351 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:22:50.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:50.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:50.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:50.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:22:50.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:50.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:50.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:50.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:22:50.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:50.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:50.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:50.722 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:22:50.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:50.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:50.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:50.835 INFO:tasks.rados.rados.2.smithi204.stdout:3259: done (2 left) 2024-09-16T05:22:50.835 INFO:tasks.rados.rados.2.smithi204.stdout:3260: done (1 left) 2024-09-16T05:22:50.835 INFO:tasks.rados.rados.2.smithi204.stdout:3261: done (0 left) 2024-09-16T05:22:50.835 INFO:tasks.rados.rados.2.smithi204.stdout:3262: snap_remove snap 341 2024-09-16T05:22:50.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:50.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:50.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:51.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:51.047 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:51.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:51.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: 2.4e scrub starts 2024-09-16T05:22:51.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: 2.4e scrub ok 2024-09-16T05:22:51.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:51.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:51.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: osdmap e820: 12 total, 12 up, 12 in 2024-09-16T05:22:51.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:22:51.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:22:51.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:22:51.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:22:51.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:51.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:50 smithi196 ceph-mon[30802]: osdmap e821: 12 total, 12 up, 12 in 2024-09-16T05:22:51.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: 2.4e scrub starts 2024-09-16T05:22:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: 2.4e scrub ok 2024-09-16T05:22:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: osdmap e820: 12 total, 12 up, 12 in 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:50 smithi032 ceph-mon[27388]: osdmap e821: 12 total, 12 up, 12 in 2024-09-16T05:22:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: 2.4e scrub starts 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: 2.4e scrub ok 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: osdmap e820: 12 total, 12 up, 12 in 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:51.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:50 smithi204 ceph-mon[30933]: osdmap e821: 12 total, 12 up, 12 in 2024-09-16T05:22:51.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:51.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:51.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:51.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:51.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:51.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:51.348 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1824 2.8 2:1030fb5b:::smithi20457400-27:head [write 521083~714336,stat] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 592.40123730100004, 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.81065307900000005, 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.349 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1824 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945147+0000", 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945150+0000", 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1260000000000001e-06 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945906+0000", 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075638699999999997 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945927+0000", 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0849e-05 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.945936+0000", 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3910000000000004e-06 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946023+0000", 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6547999999999999e-05 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.946035+0000", 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2249000000000001e-05 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752337+0000", 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80630184599999999 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752493+0000", 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000156546 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.752675+0000", 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018172699999999999 2024-09-16T05:22:51.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755628+0000", 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029533929999999999 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755689+0000", 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0949000000000002e-05 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:51.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755745+0000", 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5591999999999999e-05 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755800+0000", 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5476000000000002e-05 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1825 2.8 2:1030fb5b:::smithi20457400-27:head [write 1724715~555421] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:22:51.356 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 592.39559175299996, 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80511571699999995, 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1825 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950793+0000", 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.950794+0000", 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951237+0000", 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00044250000000000002 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951251+0000", 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4381e-05 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951276+0000", 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5233999999999999e-05 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951378+0000", 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010173799999999999 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951390+0000", 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2274e-05 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.753903+0000", 2024-09-16T05:22:51.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80251229300000004 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754002+0000", 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9427999999999997e-05 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754132+0000", 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130207 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:51.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755633+0000", 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015010100000000001 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755844+0000", 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021044799999999999 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755880+0000", 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6007999999999998e-05 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755909+0000", 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8941e-05 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1826 2.8 2:1030fb5b:::smithi20457400-27:head [setxattr _header (57) in=7b,truncate 2521083] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 592.39509246, 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80476365000000005, 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.363 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1826 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951292+0000", 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951293+0000", 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951296+0000", 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2799999999999999e-06 2024-09-16T05:22:51.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951301+0000", 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6879999999999998e-06 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951309+0000", 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7810000000000005e-06 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951421+0000", 2024-09-16T05:22:51.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112333 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951426+0000", 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6650000000000002e-06 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754913+0000", 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80348719099999999 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.754982+0000", 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9558e-05 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755084+0000", 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101894 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755638+0000", 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055423099999999997 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755988+0000", 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034971600000000002 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756021+0000", 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2461e-05 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756056+0000", 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5278999999999999e-05 2024-09-16T05:22:51.369 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1827 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 592.39507019099995, 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.804892304, 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.370 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1827 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951315+0000", 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8700000000000002e-07 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951317+0000", 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7460000000000001e-06 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951321+0000", 2024-09-16T05:22:51.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3250000000000001e-06 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951325+0000", 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1710000000000004e-06 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951464+0000", 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139111 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:58.951475+0000", 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1045e-05 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755294+0000", 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80381846999999995 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755351+0000", 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7589e-05 2024-09-16T05:22:51.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756086+0000", 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073505599999999999 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756138+0000", 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1397000000000003e-05 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756207+0000", 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9107000000000001e-05 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:1834 2.8 2:12e92f6c:::smithi20457400-1:head [stat,delete] snapc 4c=[4c,4b,4a,49,48,47,45,44,42,3e,3a,39,37,35,27] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 592.14816950900001, 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55807496000000001, 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.376 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1834 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:22:51.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198215+0000", 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198218+0000", 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.357e-06 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198229+0000", 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1677e-05 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198254+0000", 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4312e-05 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198266+0000", 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2687000000000001e-05 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198366+0000", 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9698000000000007e-05 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.198378+0000", 2024-09-16T05:22:51.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2106000000000001e-05 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755379+0000", 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.55700064400000004 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.755448+0000", 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8848000000000002e-05 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:12:59.756290+0000", 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084263100000000004 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2255 2.1d e290/282 2:ba754fcb:::smithi20457400-30:head v 290'477, mlcod=284'471)", 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 549.66157476199999, 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53570759499999998, 2024-09-16T05:22:51.383 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684810+0000", 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684811+0000", 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3600000000000002e-07 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:41.684819+0000", 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5760000000000006e-06 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219817+0000", 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.53499761300000004 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219843+0000", 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5908e-05 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219921+0000", 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7811000000000004e-05 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.219944+0000", 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3237000000000001e-05 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220484+0000", 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00053966300000000001 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:13:42.220518+0000", 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3850999999999998e-05 2024-09-16T05:22:51.387 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:390 2.3c e346/293 ondisk, result = 0)", 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 475.63880277300001, 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62913221100000005, 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707582+0000", 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707583+0000", 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.47e-07 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:55.707587+0000", 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7010000000000004e-06 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336491+0000", 2024-09-16T05:22:51.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62890340600000005 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336497+0000", 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4640000000000003e-06 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336590+0000", 2024-09-16T05:22:51.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2163000000000005e-05 2024-09-16T05:22:51.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336599+0000", 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.054e-06 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:56.336714+0000", 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011557600000000001 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.393 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4414 2.3f 2:fc733d2f:::smithi20457400-45:head [copy-from ver 329] snapc b9=[b9,b8] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 322.603382055, 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.95144123899999999, 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4414 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743003+0000", 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743006+0000", 2024-09-16T05:22:51.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2550000000000002e-06 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743021+0000", 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5024999999999999e-05 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743048+0000", 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6842000000000002e-05 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743063+0000", 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4969e-05 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743154+0000", 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1438000000000001e-05 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.743163+0000", 2024-09-16T05:22:51.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1740000000000006e-06 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.744798+0000", 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016347600000000001 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.744906+0000", 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107724 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749771+0000", 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048651129999999999 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.753713+0000", 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039421539999999998 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694348+0000", 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94063521699999997 2024-09-16T05:22:51.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694398+0000", 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9518999999999998e-05 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694444+0000", 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6048999999999999e-05 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(1.9 e106 from shard 9)", 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 268.86861157200002, 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.63215181600000003, 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:22:51.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477773+0000", 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477774+0000", 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9999999999999997e-07 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:22.477777+0000", 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5949999999999999e-06 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109802+0000", 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.63202490499999997 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109829+0000", 2024-09-16T05:22:51.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6750999999999998e-05 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109891+0000", 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1661999999999993e-05 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109900+0000", 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6569999999999992e-06 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:23.109925+0000", 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4646000000000001e-05 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:958 2.2e e588/278 ondisk, result = 0)", 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 233.553838095, 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92946010700000004, 2024-09-16T05:22:51.405 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792547+0000", 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.06e-07 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:57.792556+0000", 2024-09-16T05:22:51.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.208e-06 2024-09-16T05:22:51.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721793+0000", 2024-09-16T05:22:51.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92923791899999997 2024-09-16T05:22:51.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721804+0000", 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0936e-05 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721875+0000", 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0863000000000002e-05 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.721885+0000", 2024-09-16T05:22:51.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5510000000000005e-06 2024-09-16T05:22:51.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:58.722007+0000", 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012192399999999999 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:5901 2.1d e608/533 2:ba754fcb:::smithi20457400-30:head v 608'1064, mlcod=608'1061)", 2024-09-16T05:22:51.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:17.903833+0000", 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.44255186800001, 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.79383940900000005, 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:17.903833+0000", 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:17.903833+0000", 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:17.903833+0000", 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:17.903838+0000", 2024-09-16T05:22:51.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9269999999999996e-06 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.696799+0000", 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.79296047700000005 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.696807+0000", 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5210000000000002e-06 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.696876+0000", 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8779e-05 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.696897+0000", 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1141999999999998e-05 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.697637+0000", 2024-09-16T05:22:51.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073942099999999998 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:18.697672+0000", 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5589000000000001e-05 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5976 2.8 2:1030fb5b:::smithi20457400-27:head [setxattr _header (58) in=7b,truncate 2235529] snapc f7=[f7,f6,f5,f2,f0,ef,ed,e8,e7,e5,e0,ca] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-09-16T05:22:51.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.795849+0000", 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 204.55053572200001, 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75718527599999996, 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5976 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795849+0000", 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795849+0000", 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795850+0000", 2024-09-16T05:22:51.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1500000000000003e-07 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795854+0000", 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7479999999999999e-06 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795862+0000", 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1429999999999998e-06 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795869+0000", 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1770000000000001e-06 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.796994+0000", 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011255149999999999 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.797063+0000", 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8616999999999997e-05 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 9", 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.797164+0000", 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101081 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:51.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.800936+0000", 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003771445 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.552882+0000", 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75194649300000005 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.552958+0000", 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5736999999999997e-05 2024-09-16T05:22:51.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.553034+0000", 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6404999999999998e-05 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5977 2.8 2:1030fb5b:::smithi20457400-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e617)", 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.795882+0000", 2024-09-16T05:22:51.424 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 204.55050308200001, 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75735637700000003, 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5977 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.425 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795882+0000", 2024-09-16T05:22:51.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795882+0000", 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795882+0000", 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7599999999999997e-07 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795885+0000", 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7199999999999998e-06 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795892+0000", 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4399999999999999e-06 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.795898+0000", 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7490000000000004e-06 2024-09-16T05:22:51.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.797481+0000", 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015833010000000001 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.797561+0000", 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9832000000000004e-05 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.553077+0000", 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75551588999999997 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.553153+0000", 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5705999999999996e-05 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.553238+0000", 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5162999999999998e-05 2024-09-16T05:22:51.430 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGPushReply(2.3c 626/625 [PushReplyOp(2:3d4bedc8:::smithi20457400-22:f7)])", 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:35.873422+0000", 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 195.47296278600001, 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.76198344399999995, 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:35.873422+0000", 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:35.873422+0000", 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:35.873422+0000", 2024-09-16T05:22:51.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7400000000000003e-07 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:35.873426+0000", 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9489999999999998e-06 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.634292+0000", 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.76086524600000005 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.634298+0000", 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.251e-06 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.634368+0000", 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0129000000000006e-05 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.635405+0000", 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010372949999999999 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.434 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGPushReply(2.3c 626/625 [PushReplyOp(2:3d4bedc8:::smithi20457400-22:f7)])", 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:36.637827+0000", 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 194.70855771500001, 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.82730668500000004, 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.637827+0000", 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.637827+0000", 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.637828+0000", 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5199999999999995e-07 2024-09-16T05:22:51.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:36.637833+0000", 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2089999999999996e-06 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:37.464824+0000", 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.82699144300000005 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:37.464837+0000", 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2396e-05 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:37.464925+0000", 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7723e-05 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:37.465134+0000", 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020906200000000001 2024-09-16T05:22:51.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6760 2.2d 2:b7234079:::smithi20457400-42:head [setxattr _header (58) in=7b,truncate 3865808] snapc 117=[117,116,113,112,10c,10b,10a,109,108,106,101,f2] ondisk+write+known_if_redirected+supports_pool_eio e681)", 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:30.900604+0000", 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 140.44578020099999, 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.66116237099999997, 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6760 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900604+0000", 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900604+0000", 2024-09-16T05:22:51.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900605+0000", 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1900000000000002e-07 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900607+0000", 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.261e-06 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900612+0000", 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.216e-06 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900615+0000", 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1949999999999999e-06 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901352+0000", 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00073710400000000004 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901409+0000", 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6903000000000001e-05 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 6", 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901493+0000", 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3992000000000005e-05 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901911+0000", 2024-09-16T05:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00041862900000000001 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561576+0000", 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65966484700000005 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561678+0000", 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102012 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561767+0000", 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8592999999999997e-05 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6761 2.2d 2:b7234079:::smithi20457400-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e681)", 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:30.900627+0000", 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 140.44575810000001, 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.66137536900000005, 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.445 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6761 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900627+0000", 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900627+0000", 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900627+0000", 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9799999999999999e-07 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900628+0000", 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.488e-06 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900637+0000", 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0710000000000001e-06 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.900642+0000", 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5410000000000002e-06 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901676+0000", 2024-09-16T05:22:51.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001033647 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.901720+0000", 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4282000000000001e-05 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561816+0000", 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.66009643299999998 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561898+0000", 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1323999999999996e-05 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.562002+0000", 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104285 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.450 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.24571.0:6760 2.2d e681/252 ondisk, result = 0)", 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:30.902124+0000", 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 140.44426048700001, 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65965149099999998, 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.902124+0000", 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.902124+0000", 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.902125+0000", 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7299999999999996e-07 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:30.902129+0000", 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7809999999999999e-06 2024-09-16T05:22:51.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561389+0000", 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65926033500000003 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561410+0000", 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1041999999999999e-05 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561555+0000", 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014517299999999999 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561569+0000", 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.414e-05 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:31.561776+0000", 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020644700000000001 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.7.0:1258 2.3c e696/625 ondisk, result = 0)", 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:45.814351+0000", 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.532034014, 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96819708100000001, 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:45.814351+0000", 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:45.814351+0000", 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:45.814351+0000", 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8599999999999998e-07 2024-09-16T05:22:51.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:45.814361+0000", 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8870000000000004e-06 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782270+0000", 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96790891700000004 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782285+0000", 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5265999999999999e-05 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782425+0000", 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000139245 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782435+0000", 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0769e-05 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:46.782548+0000", 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112311 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGRecoveryDeleteReply(2.2d e746,745 2:b7234079:::smithi20457400-42:9,141'73)", 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:35.904530+0000", 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 75.441854574999994, 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93167596100000005, 2024-09-16T05:22:51.459 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:35.904530+0000", 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:35.904530+0000", 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:22:51.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:35.904531+0000", 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8100000000000003e-07 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:35.904535+0000", 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.968e-06 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:22:51.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.835815+0000", 2024-09-16T05:22:51.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93128024099999995 2024-09-16T05:22:51.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:22:51.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.835823+0000", 2024-09-16T05:22:51.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2970000000000004e-06 2024-09-16T05:22:51.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.835887+0000", 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4249000000000001e-05 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.836206+0000", 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031872500000000001 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.463 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.464 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.464 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:22:51.464 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:22:51.464 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:22:51.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:51.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:51.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:51.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:51.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:51.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:51.702 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1392,"stamp":"2024-09-16T05:22:50.214538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298501850,"num_objects":135,"num_object_clones":86,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":17,"num_read":3537,"num_read_kb":3531702,"num_write":2484,"num_write_kb":1018877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37227,"ondisk_log_size":37227,"up":254,"acting":254,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":254,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1086508,"kb_used_data":523720,"kb_used_omap":27,"kb_used_meta":562468,"kb_avail":1123855316,"statfs":{"total":1151940427776,"available":1150827843584,"internally_reserved":0,"allocated":536289280,"data_stored":557872424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":575967399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-37671041,"num_objects":-17,"num_object_clones":-17,"num_object_copies":-34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-17,"num_whiteouts":1,"num_read":19,"num_read_kb":9689,"num_write":24,"num_write_kb":15262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000862"},"pg_stats":[{"pgid":"2.5f","version":"284'221","reported_seq":3780,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.832400+0000","last_change":"2024-09-16T05:22:45.832619+0000","last_active":"2024-09-16T05:22:45.832400+0000","last_peered":"2024-09-16T05:22:45.832400+0000","last_clean":"2024-09-16T05:22:45.832400+0000","last_became_active":"2024-09-16T05:21:54.939189+0000","last_became_peered":"2024-09-16T05:21:54.939189+0000","last_unstale":"2024-09-16T05:22:45.832400+0000","last_undegraded":"2024-09-16T05:22:45.832400+0000","last_fullsized":"2024-09-16T05:22:45.832400+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:21:42.460245+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:21:42.460245+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044288800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.5e","version":"815'555","reported_seq":4250,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.835034+0000","last_change":"2024-09-16T05:22:45.835261+0000","last_active":"2024-09-16T05:22:45.835034+0000","last_peered":"2024-09-16T05:22:45.835034+0000","last_clean":"2024-09-16T05:22:45.835034+0000","last_became_active":"2024-09-16T05:21:54.937455+0000","last_became_peered":"2024-09-16T05:21:54.937455+0000","last_unstale":"2024-09-16T05:22:45.835034+0000","last_undegraded":"2024-09-16T05:22:45.835034+0000","last_fullsized":"2024-09-16T05:22:45.835034+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"743'518","last_scrub_stamp":"2024-09-16T05:21:40.979621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:40.979621+0000","objects_scrubbed":3,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:52.991447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013813440000000001,"stat_sum":{"num_bytes":7513389,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":12,"num_read_kb":13007,"num_write":7,"num_write_kb":989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4819607,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.5d","version":"820'1436","reported_seq":5252,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.936171+0000","last_change":"2024-09-16T05:22:49.936373+0000","last_active":"2024-09-16T05:22:49.936171+0000","last_peered":"2024-09-16T05:22:49.936171+0000","last_clean":"2024-09-16T05:22:49.936171+0000","last_became_active":"2024-09-16T05:21:32.830157+0000","last_became_peered":"2024-09-16T05:21:32.830157+0000","last_unstale":"2024-09-16T05:22:49.936171+0000","last_undegraded":"2024-09-16T05:22:49.936171+0000","last_fullsized":"2024-09-16T05:22:49.936171+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"749'1309","last_scrub_stamp":"2024-09-16T05:21:42.405784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:42.405784+0000","objects_scrubbed":8,"log_size":1436,"log_dups_size":0,"ondisk_log_size":1436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":2,"snaptrim_duration":0.025076676999999999,"stat_sum":{"num_bytes":11463331,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":59,"num_read_kb":52456,"num_write":24,"num_write_kb":13635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.5c","version":"815'476","reported_seq":3642,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.850174+0000","last_change":"2024-09-16T05:22:44.850665+0000","last_active":"2024-09-16T05:22:44.850174+0000","last_peered":"2024-09-16T05:22:44.850174+0000","last_clean":"2024-09-16T05:22:44.850174+0000","last_became_active":"2024-09-16T05:21:32.828660+0000","last_became_peered":"2024-09-16T05:21:32.828660+0000","last_unstale":"2024-09-16T05:22:44.850174+0000","last_undegraded":"2024-09-16T05:22:44.850174+0000","last_fullsized":"2024-09-16T05:22:44.850174+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"723'472","last_scrub_stamp":"2024-09-16T05:21:40.889101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:40.889101+0000","objects_scrubbed":2,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:08.321058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033435269999999998,"stat_sum":{"num_bytes":4102515,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":7420,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.5b","version":"820'660","reported_seq":4236,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.918081+0000","last_change":"2024-09-16T05:22:49.914176+0000","last_active":"2024-09-16T05:22:49.918081+0000","last_peered":"2024-09-16T05:22:49.918081+0000","last_clean":"2024-09-16T05:22:49.918081+0000","last_became_active":"2024-09-16T05:21:32.829681+0000","last_became_peered":"2024-09-16T05:21:32.829681+0000","last_unstale":"2024-09-16T05:22:49.918081+0000","last_undegraded":"2024-09-16T05:22:49.918081+0000","last_fullsized":"2024-09-16T05:22:49.918081+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"733'656","last_scrub_stamp":"2024-09-16T05:21:41.447936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:41.447936+0000","objects_scrubbed":1,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:02.236542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030888439999999999,"stat_sum":{"num_bytes":3479247,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":32170,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.5a","version":"815'1218","reported_seq":4726,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-16T05:22:47.886684+0000","last_change":"2024-09-16T05:22:47.886884+0000","last_active":"2024-09-16T05:22:47.886684+0000","last_peered":"2024-09-16T05:22:47.886684+0000","last_clean":"2024-09-16T05:22:47.886684+0000","last_became_active":"2024-09-16T05:21:32.826884+0000","last_became_peered":"2024-09-16T05:21:32.826884+0000","last_unstale":"2024-09-16T05:22:47.886684+0000","last_undegraded":"2024-09-16T05:22:47.886684+0000","last_fullsized":"2024-09-16T05:22:47.886684+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"744'1136","last_scrub_stamp":"2024-09-16T05:21:42.339564+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:21:42.339564+0000","objects_scrubbed":3,"log_size":1218,"log_dups_size":0,"ondisk_log_size":1218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:55.084337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000805389,"stat_sum":{"num_bytes":9480352,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":21681,"num_write":24,"num_write_kb":6390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.59","version":"182'60","reported_seq":430,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.890764+0000","last_change":"2024-09-16T05:22:46.890967+0000","last_active":"2024-09-16T05:22:46.890764+0000","last_peered":"2024-09-16T05:22:46.890764+0000","last_clean":"2024-09-16T05:22:46.890764+0000","last_became_active":"2024-09-16T05:21:54.931981+0000","last_became_peered":"2024-09-16T05:21:54.931981+0000","last_unstale":"2024-09-16T05:22:46.890764+0000","last_undegraded":"2024-09-16T05:22:46.890764+0000","last_fullsized":"2024-09-16T05:22:46.890764+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:35.068053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.068053+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:40.149395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066923199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.58","version":"181'57","reported_seq":3551,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.854787+0000","last_change":"2024-09-16T05:22:44.855260+0000","last_active":"2024-09-16T05:22:44.854787+0000","last_peered":"2024-09-16T05:22:44.854787+0000","last_clean":"2024-09-16T05:22:44.854787+0000","last_became_active":"2024-09-16T05:21:29.826747+0000","last_became_peered":"2024-09-16T05:21:29.826747+0000","last_unstale":"2024-09-16T05:22:44.854787+0000","last_undegraded":"2024-09-16T05:22:44.854787+0000","last_fullsized":"2024-09-16T05:22:44.854787+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:21:30.068545+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:21:30.068545+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00095609899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.57","version":"0'0","reported_seq":1094,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.835711+0000","last_change":"2024-09-16T05:22:45.836033+0000","last_active":"2024-09-16T05:22:45.835711+0000","last_peered":"2024-09-16T05:22:45.835711+0000","last_clean":"2024-09-16T05:22:45.835711+0000","last_became_active":"2024-09-16T05:21:29.827377+0000","last_became_peered":"2024-09-16T05:21:29.827377+0000","last_unstale":"2024-09-16T05:22:45.835711+0000","last_undegraded":"2024-09-16T05:22:45.835711+0000","last_fullsized":"2024-09-16T05:22:45.835711+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:29.974178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:29.974178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001707599,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.56","version":"815'378","reported_seq":3565,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.849606+0000","last_change":"2024-09-16T05:22:44.849813+0000","last_active":"2024-09-16T05:22:44.849606+0000","last_peered":"2024-09-16T05:22:44.849606+0000","last_clean":"2024-09-16T05:22:44.849606+0000","last_became_active":"2024-09-16T05:21:32.829699+0000","last_became_peered":"2024-09-16T05:21:32.829699+0000","last_unstale":"2024-09-16T05:22:44.849606+0000","last_undegraded":"2024-09-16T05:22:44.849606+0000","last_fullsized":"2024-09-16T05:22:44.849606+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"738'336","last_scrub_stamp":"2024-09-16T05:21:42.738509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:42.738509+0000","objects_scrubbed":2,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:03.111267+0000","scrub_duration":828,"objects_trimmed":0,"snaptrim_duration":0.0023072679999999999,"stat_sum":{"num_bytes":4725282,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":24,"num_read_kb":18484,"num_write":13,"num_write_kb":3476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.55","version":"433'507","reported_seq":2566,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-16T05:22:47.887058+0000","last_change":"2024-09-16T05:22:47.887346+0000","last_active":"2024-09-16T05:22:47.887058+0000","last_peered":"2024-09-16T05:22:47.887058+0000","last_clean":"2024-09-16T05:22:47.887058+0000","last_became_active":"2024-09-16T05:21:24.815957+0000","last_became_peered":"2024-09-16T05:21:24.815957+0000","last_unstale":"2024-09-16T05:22:47.887058+0000","last_undegraded":"2024-09-16T05:22:47.887058+0000","last_fullsized":"2024-09-16T05:22:47.887058+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:21:25.388344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:25.388344+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077867500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.54","version":"816'440","reported_seq":3631,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.892219+0000","last_change":"2024-09-16T05:22:46.892219+0000","last_active":"2024-09-16T05:22:46.892219+0000","last_peered":"2024-09-16T05:22:46.892219+0000","last_clean":"2024-09-16T05:22:46.892219+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:22:46.892219+0000","last_undegraded":"2024-09-16T05:22:46.892219+0000","last_fullsized":"2024-09-16T05:22:46.892219+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"765'422","last_scrub_stamp":"2024-09-16T05:22:01.692016+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:22:01.692016+0000","objects_scrubbed":1,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:13.680143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010708639999999999,"stat_sum":{"num_bytes":11076473,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":63622,"num_write":52,"num_write_kb":22446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.53","version":"181'15","reported_seq":1920,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.832477+0000","last_change":"2024-09-16T05:22:45.832672+0000","last_active":"2024-09-16T05:22:45.832477+0000","last_peered":"2024-09-16T05:22:45.832477+0000","last_clean":"2024-09-16T05:22:45.832477+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:22:45.832477+0000","last_undegraded":"2024-09-16T05:22:45.832477+0000","last_fullsized":"2024-09-16T05:22:45.832477+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:22:00.537384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.537384+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:14.062854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047957799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.52","version":"272'69","reported_seq":1910,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.847344+0000","last_change":"2024-09-16T05:22:44.847644+0000","last_active":"2024-09-16T05:22:44.847344+0000","last_peered":"2024-09-16T05:22:44.847344+0000","last_clean":"2024-09-16T05:22:44.847344+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:22:44.847344+0000","last_undegraded":"2024-09-16T05:22:44.847344+0000","last_fullsized":"2024-09-16T05:22:44.847344+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:21:44.952108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:44.952108+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:57.905689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050783200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":1987,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.835651+0000","last_change":"2024-09-16T05:22:45.835942+0000","last_active":"2024-09-16T05:22:45.835651+0000","last_peered":"2024-09-16T05:22:45.835651+0000","last_clean":"2024-09-16T05:22:45.835651+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:22:45.835651+0000","last_undegraded":"2024-09-16T05:22:45.835651+0000","last_fullsized":"2024-09-16T05:22:45.835651+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_deep_scrub":"272'115","last_deep_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_clean_scrub_stamp":"2024-09-16T05:22:11.104571+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:36.365426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001845064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.50","version":"182'16","reported_seq":1927,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-16T05:22:47.886205+0000","last_change":"2024-09-16T05:22:47.886582+0000","last_active":"2024-09-16T05:22:47.886205+0000","last_peered":"2024-09-16T05:22:47.886205+0000","last_clean":"2024-09-16T05:22:47.886205+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:22:47.886205+0000","last_undegraded":"2024-09-16T05:22:47.886205+0000","last_fullsized":"2024-09-16T05:22:47.886205+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:22:25.344602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.344602+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:29.062093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058783900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2480,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.913963+0000","last_change":"2024-09-16T05:22:49.914319+0000","last_active":"2024-09-16T05:22:49.913963+0000","last_peered":"2024-09-16T05:22:49.913963+0000","last_clean":"2024-09-16T05:22:49.913963+0000","last_became_active":"2024-09-16T05:21:10.817392+0000","last_became_peered":"2024-09-16T05:21:10.817392+0000","last_unstale":"2024-09-16T05:22:49.913963+0000","last_undegraded":"2024-09-16T05:22:49.913963+0000","last_fullsized":"2024-09-16T05:22:49.913963+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:22:33.493608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:33.493608+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:53.111959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031318700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.4e","version":"272'232","reported_seq":3680,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.961676+0000","last_change":"2024-09-16T05:22:45.961832+0000","last_active":"2024-09-16T05:22:45.961676+0000","last_peered":"2024-09-16T05:22:45.961676+0000","last_clean":"2024-09-16T05:22:45.961676+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:22:45.961676+0000","last_undegraded":"2024-09-16T05:22:45.961676+0000","last_fullsized":"2024-09-16T05:22:45.961676+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:21:07.065851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:07.065851+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001050919,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.4d","version":"817'716","reported_seq":4342,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.894582+0000","last_change":"2024-09-16T05:22:46.894794+0000","last_active":"2024-09-16T05:22:46.894582+0000","last_peered":"2024-09-16T05:22:46.894582+0000","last_clean":"2024-09-16T05:22:46.894582+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:22:46.894582+0000","last_undegraded":"2024-09-16T05:22:46.894582+0000","last_fullsized":"2024-09-16T05:22:46.894582+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"807'708","last_scrub_stamp":"2024-09-16T05:22:37.651060+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.651060+0000","objects_scrubbed":4,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:46.166477+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0034537589999999998,"stat_sum":{"num_bytes":6406789,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":38970,"num_write":84,"num_write_kb":29604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.4c","version":"815'422","reported_seq":3983,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.842595+0000","last_change":"2024-09-16T05:22:45.835601+0000","last_active":"2024-09-16T05:22:45.842595+0000","last_peered":"2024-09-16T05:22:45.842595+0000","last_clean":"2024-09-16T05:22:45.842595+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:22:45.842595+0000","last_undegraded":"2024-09-16T05:22:45.842595+0000","last_fullsized":"2024-09-16T05:22:45.842595+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"773'408","last_scrub_stamp":"2024-09-16T05:22:03.012954+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:22:03.012954+0000","objects_scrubbed":3,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:12.084496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015172289999999999,"stat_sum":{"num_bytes":6756843,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":109332,"num_write":40,"num_write_kb":13261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.4b","version":"244'177","reported_seq":3736,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.894815+0000","last_change":"2024-09-16T05:22:46.895034+0000","last_active":"2024-09-16T05:22:46.894815+0000","last_peered":"2024-09-16T05:22:46.894815+0000","last_clean":"2024-09-16T05:22:46.894815+0000","last_became_active":"2024-09-16T05:22:22.832492+0000","last_became_peered":"2024-09-16T05:22:22.832492+0000","last_unstale":"2024-09-16T05:22:46.894815+0000","last_undegraded":"2024-09-16T05:22:46.894815+0000","last_fullsized":"2024-09-16T05:22:46.894815+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:22:10.688513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.688513+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:26.326897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003461885,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.4a","version":"815'655","reported_seq":4175,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.836418+0000","last_change":"2024-09-16T05:22:45.837066+0000","last_active":"2024-09-16T05:22:45.836418+0000","last_peered":"2024-09-16T05:22:45.836418+0000","last_clean":"2024-09-16T05:22:45.836418+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:22:45.836418+0000","last_undegraded":"2024-09-16T05:22:45.836418+0000","last_fullsized":"2024-09-16T05:22:45.836418+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"733'598","last_scrub_stamp":"2024-09-16T05:21:22.862824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:22.862824+0000","objects_scrubbed":3,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022407099999999999,"stat_sum":{"num_bytes":6675255,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":55014,"num_write":59,"num_write_kb":25438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.49","version":"245'220","reported_seq":3757,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.960865+0000","last_change":"2024-09-16T05:22:45.961050+0000","last_active":"2024-09-16T05:22:45.960865+0000","last_peered":"2024-09-16T05:22:45.960865+0000","last_clean":"2024-09-16T05:22:45.960865+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:22:45.960865+0000","last_undegraded":"2024-09-16T05:22:45.960865+0000","last_fullsized":"2024-09-16T05:22:45.960865+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:21:46.823721+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:21:46.823721+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:49.819030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041794199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.48","version":"815'424","reported_seq":3936,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.856292+0000","last_change":"2024-09-16T05:22:44.856564+0000","last_active":"2024-09-16T05:22:44.856292+0000","last_peered":"2024-09-16T05:22:44.856292+0000","last_clean":"2024-09-16T05:22:44.856292+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:22:44.856292+0000","last_undegraded":"2024-09-16T05:22:44.856292+0000","last_fullsized":"2024-09-16T05:22:44.856292+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"776'403","last_scrub_stamp":"2024-09-16T05:22:06.889233+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.889233+0000","objects_scrubbed":6,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:11.989109+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0021865949999999999,"stat_sum":{"num_bytes":9210929,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":75761,"num_write":67,"num_write_kb":32708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":2536,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.850552+0000","last_change":"2024-09-16T05:22:44.850807+0000","last_active":"2024-09-16T05:22:44.850552+0000","last_peered":"2024-09-16T05:22:44.850552+0000","last_clean":"2024-09-16T05:22:44.850552+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:22:44.850552+0000","last_undegraded":"2024-09-16T05:22:44.850552+0000","last_fullsized":"2024-09-16T05:22:44.850552+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.838189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:24.910823+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0031264660000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":776,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.894878+0000","last_change":"2024-09-16T05:22:46.895096+0000","last_active":"2024-09-16T05:22:46.894878+0000","last_peered":"2024-09-16T05:22:46.894878+0000","last_clean":"2024-09-16T05:22:46.894878+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:22:46.894878+0000","last_undegraded":"2024-09-16T05:22:46.894878+0000","last_fullsized":"2024-09-16T05:22:46.894878+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:32.588419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.588419+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:45.300333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0034907340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.45","version":"305'464","reported_seq":3075,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.915042+0000","last_change":"2024-09-16T05:22:49.915620+0000","last_active":"2024-09-16T05:22:49.915042+0000","last_peered":"2024-09-16T05:22:49.915042+0000","last_clean":"2024-09-16T05:22:49.915042+0000","last_became_active":"2024-09-16T05:21:10.818112+0000","last_became_peered":"2024-09-16T05:21:10.818112+0000","last_unstale":"2024-09-16T05:22:49.915042+0000","last_undegraded":"2024-09-16T05:22:49.915042+0000","last_fullsized":"2024-09-16T05:22:49.915042+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:21:49.394285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:49.394285+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:12.439336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038390690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":561,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.912503+0000","last_change":"2024-09-16T05:22:49.912806+0000","last_active":"2024-09-16T05:22:49.912503+0000","last_peered":"2024-09-16T05:22:49.912503+0000","last_clean":"2024-09-16T05:22:49.912503+0000","last_became_active":"2024-09-16T05:21:07.820306+0000","last_became_peered":"2024-09-16T05:21:07.820306+0000","last_unstale":"2024-09-16T05:22:49.912503+0000","last_undegraded":"2024-09-16T05:22:49.912503+0000","last_fullsized":"2024-09-16T05:22:49.912503+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:22:10.381270+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.381270+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:27.670727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001840235,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":2526,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.843855+0000","last_change":"2024-09-16T05:22:45.843974+0000","last_active":"2024-09-16T05:22:45.843855+0000","last_peered":"2024-09-16T05:22:45.843855+0000","last_clean":"2024-09-16T05:22:45.843855+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:22:45.843855+0000","last_undegraded":"2024-09-16T05:22:45.843855+0000","last_fullsized":"2024-09-16T05:22:45.843855+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:22:03.589783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:03.589783+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:08.863450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001859716,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.42","version":"182'20","reported_seq":2529,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.903409+0000","last_change":"2024-09-16T05:22:48.903588+0000","last_active":"2024-09-16T05:22:48.903409+0000","last_peered":"2024-09-16T05:22:48.903409+0000","last_clean":"2024-09-16T05:22:48.903409+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:22:48.903409+0000","last_undegraded":"2024-09-16T05:22:48.903409+0000","last_fullsized":"2024-09-16T05:22:48.903409+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:22:25.741168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.741168+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:48.161454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0043356280000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.41","version":"178'50","reported_seq":1382,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.847279+0000","last_change":"2024-09-16T05:22:44.847549+0000","last_active":"2024-09-16T05:22:44.847279+0000","last_peered":"2024-09-16T05:22:44.847279+0000","last_clean":"2024-09-16T05:22:44.847279+0000","last_became_active":"2024-09-16T05:21:54.930175+0000","last_became_peered":"2024-09-16T05:21:54.930175+0000","last_unstale":"2024-09-16T05:22:44.847279+0000","last_undegraded":"2024-09-16T05:22:44.847279+0000","last_fullsized":"2024-09-16T05:22:44.847279+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:22:28.678308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.678308+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:55.532993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048351999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":1076,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.961964+0000","last_change":"2024-09-16T05:22:45.962073+0000","last_active":"2024-09-16T05:22:45.961964+0000","last_peered":"2024-09-16T05:22:45.961964+0000","last_clean":"2024-09-16T05:22:45.961964+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:22:45.961964+0000","last_undegraded":"2024-09-16T05:22:45.961964+0000","last_fullsized":"2024-09-16T05:22:45.961964+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:22:35.757429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.757429+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:01.739675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011019319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.3f","version":"816'541","reported_seq":4200,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.859691+0000","last_change":"2024-09-16T05:22:45.843530+0000","last_active":"2024-09-16T05:22:45.859691+0000","last_peered":"2024-09-16T05:22:45.859691+0000","last_clean":"2024-09-16T05:22:45.859691+0000","last_became_active":"2024-09-16T05:21:54.939233+0000","last_became_peered":"2024-09-16T05:21:54.939233+0000","last_unstale":"2024-09-16T05:22:45.859691+0000","last_undegraded":"2024-09-16T05:22:45.859691+0000","last_fullsized":"2024-09-16T05:22:45.859691+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"737'472","last_scrub_stamp":"2024-09-16T05:21:28.802674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:28.802674+0000","objects_scrubbed":4,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015540759999999999,"stat_sum":{"num_bytes":14257342,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":116,"num_read_kb":104032,"num_write":71,"num_write_kb":25787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":11726959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.3e","version":"817'576","reported_seq":4101,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.893351+0000","last_change":"2024-09-16T05:22:46.893579+0000","last_active":"2024-09-16T05:22:46.893351+0000","last_peered":"2024-09-16T05:22:46.893351+0000","last_clean":"2024-09-16T05:22:46.893351+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:22:46.893351+0000","last_undegraded":"2024-09-16T05:22:46.893351+0000","last_fullsized":"2024-09-16T05:22:46.893351+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"742'526","last_scrub_stamp":"2024-09-16T05:21:43.625432+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.625432+0000","objects_scrubbed":2,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:12.583915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002908042,"stat_sum":{"num_bytes":7199349,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":47318,"num_write":67,"num_write_kb":35144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2021,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.922154+0000","last_change":"2024-09-16T05:22:49.922790+0000","last_active":"2024-09-16T05:22:49.922154+0000","last_peered":"2024-09-16T05:22:49.922154+0000","last_clean":"2024-09-16T05:22:49.922154+0000","last_became_active":"2024-09-16T05:21:10.817387+0000","last_became_peered":"2024-09-16T05:21:10.817387+0000","last_unstale":"2024-09-16T05:22:49.922154+0000","last_undegraded":"2024-09-16T05:22:49.922154+0000","last_fullsized":"2024-09-16T05:22:49.922154+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.497749+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:44.846991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010715544,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.3c","version":"816'820","reported_seq":4614,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.844163+0000","last_change":"2024-09-16T05:22:45.844319+0000","last_active":"2024-09-16T05:22:45.844163+0000","last_peered":"2024-09-16T05:22:45.844163+0000","last_clean":"2024-09-16T05:22:45.844163+0000","last_became_active":"2024-09-16T05:21:07.828778+0000","last_became_peered":"2024-09-16T05:21:07.828778+0000","last_unstale":"2024-09-16T05:22:45.844163+0000","last_undegraded":"2024-09-16T05:22:45.844163+0000","last_fullsized":"2024-09-16T05:22:45.844163+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'807","last_scrub_stamp":"2024-09-16T05:22:31.305689+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:22:31.305689+0000","objects_scrubbed":4,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:50.806775+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024336359999999999,"stat_sum":{"num_bytes":8572197,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":194,"num_read_kb":258104,"num_write":95,"num_write_kb":34389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":14713263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.3b","version":"815'506","reported_seq":4054,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.962141+0000","last_change":"2024-09-16T05:22:45.962410+0000","last_active":"2024-09-16T05:22:45.962141+0000","last_peered":"2024-09-16T05:22:45.962141+0000","last_clean":"2024-09-16T05:22:45.962141+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:22:45.962141+0000","last_undegraded":"2024-09-16T05:22:45.962141+0000","last_fullsized":"2024-09-16T05:22:45.962141+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"737'454","last_scrub_stamp":"2024-09-16T05:21:28.918225+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:21:28.918225+0000","objects_scrubbed":2,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001068591,"stat_sum":{"num_bytes":10274465,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":94,"num_read_kb":93898,"num_write":69,"num_write_kb":29927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.3a","version":"808'671","reported_seq":4170,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.855313+0000","last_change":"2024-09-16T05:22:44.855670+0000","last_active":"2024-09-16T05:22:44.855313+0000","last_peered":"2024-09-16T05:22:44.855313+0000","last_clean":"2024-09-16T05:22:44.855313+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:22:44.855313+0000","last_undegraded":"2024-09-16T05:22:44.855313+0000","last_fullsized":"2024-09-16T05:22:44.855313+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"749'617","last_scrub_stamp":"2024-09-16T05:21:41.857414+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:21:41.857414+0000","objects_scrubbed":3,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:11.840290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012739159999999999,"stat_sum":{"num_bytes":3702437,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":66016,"num_write":76,"num_write_kb":38272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":2604,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.855758+0000","last_change":"2024-09-16T05:22:44.856198+0000","last_active":"2024-09-16T05:22:44.855758+0000","last_peered":"2024-09-16T05:22:44.855758+0000","last_clean":"2024-09-16T05:22:44.855758+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:22:44.855758+0000","last_undegraded":"2024-09-16T05:22:44.855758+0000","last_fullsized":"2024-09-16T05:22:44.855758+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:21:25.069163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:25.069163+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:48.972236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017766800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":2642,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.898567+0000","last_change":"2024-09-16T05:22:48.898915+0000","last_active":"2024-09-16T05:22:48.898567+0000","last_peered":"2024-09-16T05:22:48.898567+0000","last_clean":"2024-09-16T05:22:48.898567+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:22:48.898567+0000","last_undegraded":"2024-09-16T05:22:48.898567+0000","last_fullsized":"2024-09-16T05:22:48.898567+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:21:46.699624+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:21:46.699624+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:00.771067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056674099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.37","version":"0'0","reported_seq":1482,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.836475+0000","last_change":"2024-09-16T05:22:45.836784+0000","last_active":"2024-09-16T05:22:45.836475+0000","last_peered":"2024-09-16T05:22:45.836475+0000","last_clean":"2024-09-16T05:22:45.836475+0000","last_became_active":"2024-09-16T05:21:54.938965+0000","last_became_peered":"2024-09-16T05:21:54.938965+0000","last_unstale":"2024-09-16T05:22:45.836475+0000","last_undegraded":"2024-09-16T05:22:45.836475+0000","last_fullsized":"2024-09-16T05:22:45.836475+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:25.115048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.115048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:41.718014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001946417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.36","version":"272'60","reported_seq":561,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.914421+0000","last_change":"2024-09-16T05:22:49.914941+0000","last_active":"2024-09-16T05:22:49.914421+0000","last_peered":"2024-09-16T05:22:49.914421+0000","last_clean":"2024-09-16T05:22:49.914421+0000","last_became_active":"2024-09-16T05:21:07.819657+0000","last_became_peered":"2024-09-16T05:21:07.819657+0000","last_unstale":"2024-09-16T05:22:49.914421+0000","last_undegraded":"2024-09-16T05:22:49.914421+0000","last_fullsized":"2024-09-16T05:22:49.914421+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:22:49.686720+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.686720+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:17.747865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035358450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.35","version":"820'943","reported_seq":4582,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.921763+0000","last_change":"2024-09-16T05:22:49.922100+0000","last_active":"2024-09-16T05:22:49.921763+0000","last_peered":"2024-09-16T05:22:49.921763+0000","last_clean":"2024-09-16T05:22:49.921763+0000","last_became_active":"2024-09-16T05:21:10.815197+0000","last_became_peered":"2024-09-16T05:21:10.815197+0000","last_unstale":"2024-09-16T05:22:49.921763+0000","last_undegraded":"2024-09-16T05:22:49.921763+0000","last_fullsized":"2024-09-16T05:22:49.921763+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"769'866","last_scrub_stamp":"2024-09-16T05:22:00.928689+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.928689+0000","objects_scrubbed":9,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:30.462221+0000","scrub_duration":601,"objects_trimmed":1,"snaptrim_duration":0.010182605000000001,"stat_sum":{"num_bytes":5714875,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":191,"num_read_kb":188287,"num_write":130,"num_write_kb":57425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":17054243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.34","version":"815'794","reported_seq":3934,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.961894+0000","last_change":"2024-09-16T05:22:45.962012+0000","last_active":"2024-09-16T05:22:45.961894+0000","last_peered":"2024-09-16T05:22:45.961894+0000","last_clean":"2024-09-16T05:22:45.961894+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:22:45.961894+0000","last_undegraded":"2024-09-16T05:22:45.961894+0000","last_fullsized":"2024-09-16T05:22:45.961894+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"781'749","last_scrub_stamp":"2024-09-16T05:22:12.568382+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:22:12.568382+0000","objects_scrubbed":9,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:20.768070+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001224036,"stat_sum":{"num_bytes":18248895,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":217,"num_read_kb":245667,"num_write":140,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":2710,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.844202+0000","last_change":"2024-09-16T05:22:45.844353+0000","last_active":"2024-09-16T05:22:45.844202+0000","last_peered":"2024-09-16T05:22:45.844202+0000","last_clean":"2024-09-16T05:22:45.844202+0000","last_became_active":"2024-09-16T05:21:07.818018+0000","last_became_peered":"2024-09-16T05:21:07.818018+0000","last_unstale":"2024-09-16T05:22:45.844202+0000","last_undegraded":"2024-09-16T05:22:45.844202+0000","last_fullsized":"2024-09-16T05:22:45.844202+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:22:08.520148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:08.520148+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:34.089079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020279410000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.32","version":"815'305","reported_seq":3011,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.893919+0000","last_change":"2024-09-16T05:22:46.893919+0000","last_active":"2024-09-16T05:22:46.893919+0000","last_peered":"2024-09-16T05:22:46.893919+0000","last_clean":"2024-09-16T05:22:46.893919+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:22:46.893919+0000","last_undegraded":"2024-09-16T05:22:46.893919+0000","last_fullsized":"2024-09-16T05:22:46.893919+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'267","last_scrub_stamp":"2024-09-16T05:21:45.593289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:45.593289+0000","objects_scrubbed":3,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:53.058717+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032186789999999999,"stat_sum":{"num_bytes":6538395,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":112017,"num_write":54,"num_write_kb":22810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.31","version":"814'347","reported_seq":3616,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.847701+0000","last_change":"2024-09-16T05:22:44.847991+0000","last_active":"2024-09-16T05:22:44.847701+0000","last_peered":"2024-09-16T05:22:44.847701+0000","last_clean":"2024-09-16T05:22:44.847701+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:22:44.847701+0000","last_undegraded":"2024-09-16T05:22:44.847701+0000","last_fullsized":"2024-09-16T05:22:44.847701+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"747'310","last_scrub_stamp":"2024-09-16T05:21:43.946354+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.946354+0000","objects_scrubbed":2,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:07.708850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00076469400000000001,"stat_sum":{"num_bytes":3882648,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":119774,"num_write":55,"num_write_kb":23353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":1457,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.834958+0000","last_change":"2024-09-16T05:22:45.835297+0000","last_active":"2024-09-16T05:22:45.834958+0000","last_peered":"2024-09-16T05:22:45.834958+0000","last_clean":"2024-09-16T05:22:45.834958+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:22:45.834958+0000","last_undegraded":"2024-09-16T05:22:45.834958+0000","last_fullsized":"2024-09-16T05:22:45.834958+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:21:49.039822+0000","last_deep_scrub":"182'16","last_deep_scrub_stamp":"2024-09-16T05:21:49.039822+0000","last_clean_scrub_stamp":"2024-09-16T05:21:49.039822+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:52.239196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001569118,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"1.12","version":"35'139","reported_seq":1276,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.960550+0000","last_change":"2024-09-16T05:22:07.661072+0000","last_active":"2024-09-16T05:22:45.960550+0000","last_peered":"2024-09-16T05:22:45.960550+0000","last_clean":"2024-09-16T05:22:45.960550+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:22:45.960550+0000","last_undegraded":"2024-09-16T05:22:45.960550+0000","last_fullsized":"2024-09-16T05:22:45.960550+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.660982+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.660982+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:10.170857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":813,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.890024+0000","last_change":"2024-09-16T05:22:46.890343+0000","last_active":"2024-09-16T05:22:46.890024+0000","last_peered":"2024-09-16T05:22:46.890024+0000","last_clean":"2024-09-16T05:22:46.890024+0000","last_became_active":"2024-09-16T05:21:57.821735+0000","last_became_peered":"2024-09-16T05:21:57.821735+0000","last_unstale":"2024-09-16T05:22:46.890024+0000","last_undegraded":"2024-09-16T05:22:46.890024+0000","last_fullsized":"2024-09-16T05:22:46.890024+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:22:00.473439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.473439+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:01.762428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051453200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"1.13","version":"35'139","reported_seq":1253,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.911436+0000","last_change":"2024-09-16T05:21:23.189136+0000","last_active":"2024-09-16T05:22:49.911436+0000","last_peered":"2024-09-16T05:22:49.911436+0000","last_clean":"2024-09-16T05:22:49.911436+0000","last_became_active":"2024-09-16T05:21:10.814832+0000","last_became_peered":"2024-09-16T05:21:10.814832+0000","last_unstale":"2024-09-16T05:22:49.911436+0000","last_undegraded":"2024-09-16T05:22:49.911436+0000","last_fullsized":"2024-09-16T05:22:49.911436+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:23.189059+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:23.189059+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":3204,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.898991+0000","last_change":"2024-09-16T05:22:48.899174+0000","last_active":"2024-09-16T05:22:48.898991+0000","last_peered":"2024-09-16T05:22:48.898991+0000","last_clean":"2024-09-16T05:22:48.898991+0000","last_became_active":"2024-09-16T05:21:07.815584+0000","last_became_peered":"2024-09-16T05:21:07.815584+0000","last_unstale":"2024-09-16T05:22:48.898991+0000","last_undegraded":"2024-09-16T05:22:48.898991+0000","last_fullsized":"2024-09-16T05:22:48.898991+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:22:11.576311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:11.576311+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:28.939313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077527599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.c","version":"35'139","reported_seq":1240,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-16T05:22:47.886115+0000","last_change":"2024-09-16T05:21:43.300385+0000","last_active":"2024-09-16T05:22:47.886115+0000","last_peered":"2024-09-16T05:22:47.886115+0000","last_clean":"2024-09-16T05:22:47.886115+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:22:47.886115+0000","last_undegraded":"2024-09-16T05:22:47.886115+0000","last_fullsized":"2024-09-16T05:22:47.886115+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:43.300299+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.300299+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:55.115544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"818'453","reported_seq":4043,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.902899+0000","last_change":"2024-09-16T05:22:48.903333+0000","last_active":"2024-09-16T05:22:48.902899+0000","last_peered":"2024-09-16T05:22:48.902899+0000","last_clean":"2024-09-16T05:22:48.902899+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:22:48.902899+0000","last_undegraded":"2024-09-16T05:22:48.902899+0000","last_fullsized":"2024-09-16T05:22:48.902899+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'437","last_scrub_stamp":"2024-09-16T05:22:27.742914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.742914+0000","objects_scrubbed":3,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:27.797500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0042459619999999998,"stat_sum":{"num_bytes":7362192,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":56,"num_read_kb":67731,"num_write":52,"num_write_kb":22714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.d","version":"35'139","reported_seq":1393,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.841614+0000","last_change":"2024-09-16T05:22:26.361667+0000","last_active":"2024-09-16T05:22:45.841614+0000","last_peered":"2024-09-16T05:22:45.841614+0000","last_clean":"2024-09-16T05:22:45.841614+0000","last_became_active":"2024-09-16T05:20:35.804924+0000","last_became_peered":"2024-09-16T05:20:35.804924+0000","last_unstale":"2024-09-16T05:22:45.841614+0000","last_undegraded":"2024-09-16T05:22:45.841614+0000","last_fullsized":"2024-09-16T05:22:45.841614+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.361570+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.361570+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.129133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":3691,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.895095+0000","last_change":"2024-09-16T05:22:46.895463+0000","last_active":"2024-09-16T05:22:46.895095+0000","last_peered":"2024-09-16T05:22:46.895095+0000","last_clean":"2024-09-16T05:22:46.895095+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:22:46.895095+0000","last_undegraded":"2024-09-16T05:22:46.895095+0000","last_fullsized":"2024-09-16T05:22:46.895095+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:22:25.628406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.628406+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:38.259631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0035662530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"1.e","version":"35'139","reported_seq":1392,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.832211+0000","last_change":"2024-09-16T05:21:45.495056+0000","last_active":"2024-09-16T05:22:45.832211+0000","last_peered":"2024-09-16T05:22:45.832211+0000","last_clean":"2024-09-16T05:22:45.832211+0000","last_became_active":"2024-09-16T05:20:35.802192+0000","last_became_peered":"2024-09-16T05:20:35.802192+0000","last_unstale":"2024-09-16T05:22:45.832211+0000","last_undegraded":"2024-09-16T05:22:45.832211+0000","last_fullsized":"2024-09-16T05:22:45.832211+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:45.494969+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:45.494969+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:50.034617+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"815'668","reported_seq":4221,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.858974+0000","last_change":"2024-09-16T05:22:44.859220+0000","last_active":"2024-09-16T05:22:44.858974+0000","last_peered":"2024-09-16T05:22:44.858974+0000","last_clean":"2024-09-16T05:22:44.858974+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:22:44.858974+0000","last_undegraded":"2024-09-16T05:22:44.858974+0000","last_fullsized":"2024-09-16T05:22:44.858974+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"793'662","last_scrub_stamp":"2024-09-16T05:22:26.926701+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.926701+0000","objects_scrubbed":3,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:38.441379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0042352700000000002,"stat_sum":{"num_bytes":2941423,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":56,"num_read_kb":52780,"num_write":55,"num_write_kb":30221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":1382,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.960781+0000","last_change":"2024-09-16T05:22:08.676849+0000","last_active":"2024-09-16T05:22:45.960781+0000","last_peered":"2024-09-16T05:22:45.960781+0000","last_clean":"2024-09-16T05:22:45.960781+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:22:45.960781+0000","last_undegraded":"2024-09-16T05:22:45.960781+0000","last_fullsized":"2024-09-16T05:22:45.960781+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:08.676744+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:22:08.676744+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:24.347150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3819,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.890459+0000","last_change":"2024-09-16T05:22:46.890695+0000","last_active":"2024-09-16T05:22:46.890459+0000","last_peered":"2024-09-16T05:22:46.890459+0000","last_clean":"2024-09-16T05:22:46.890459+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:22:46.890459+0000","last_undegraded":"2024-09-16T05:22:46.890459+0000","last_fullsized":"2024-09-16T05:22:46.890459+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:21:47.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:47.558894+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:15.208799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000748078,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"1.b","version":"35'139","reported_seq":1101,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.911246+0000","last_change":"2024-09-16T05:22:26.488008+0000","last_active":"2024-09-16T05:22:49.911246+0000","last_peered":"2024-09-16T05:22:49.911246+0000","last_clean":"2024-09-16T05:22:49.911246+0000","last_became_active":"2024-09-16T05:22:25.828286+0000","last_became_peered":"2024-09-16T05:22:25.828286+0000","last_unstale":"2024-09-16T05:22:49.911246+0000","last_undegraded":"2024-09-16T05:22:49.911246+0000","last_fullsized":"2024-09-16T05:22:49.911246+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.487937+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.487937+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:28.854050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"815'357","reported_seq":3910,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.844026+0000","last_change":"2024-09-16T05:22:45.844163+0000","last_active":"2024-09-16T05:22:45.844026+0000","last_peered":"2024-09-16T05:22:45.844026+0000","last_clean":"2024-09-16T05:22:45.844026+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:22:45.844026+0000","last_undegraded":"2024-09-16T05:22:45.844026+0000","last_fullsized":"2024-09-16T05:22:45.844026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'327","last_scrub_stamp":"2024-09-16T05:22:02.625744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:02.625744+0000","objects_scrubbed":1,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.542493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024662830000000001,"stat_sum":{"num_bytes":2235529,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":66502,"num_write":53,"num_write_kb":12811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.3","version":"815'282","reported_seq":3846,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.962390+0000","last_change":"2024-09-16T05:22:45.962556+0000","last_active":"2024-09-16T05:22:45.962390+0000","last_peered":"2024-09-16T05:22:45.962390+0000","last_clean":"2024-09-16T05:22:45.962390+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:22:45.962390+0000","last_undegraded":"2024-09-16T05:22:45.962390+0000","last_fullsized":"2024-09-16T05:22:45.962390+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"761'256","last_scrub_stamp":"2024-09-16T05:22:03.687180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:03.687180+0000","objects_scrubbed":3,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:25.362315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00090251100000000005,"stat_sum":{"num_bytes":8975631,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":128644,"num_write":59,"num_write_kb":24461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":928,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.889835+0000","last_change":"2024-09-16T05:21:54.932391+0000","last_active":"2024-09-16T05:22:46.889835+0000","last_peered":"2024-09-16T05:22:46.889835+0000","last_clean":"2024-09-16T05:22:46.889835+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:22:46.889835+0000","last_undegraded":"2024-09-16T05:22:46.889835+0000","last_fullsized":"2024-09-16T05:22:46.889835+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:41.614652+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:41.614652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:52.262172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":3550,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.962246+0000","last_change":"2024-09-16T05:22:45.962415+0000","last_active":"2024-09-16T05:22:45.962246+0000","last_peered":"2024-09-16T05:22:45.962246+0000","last_clean":"2024-09-16T05:22:45.962246+0000","last_became_active":"2024-09-16T05:21:07.817459+0000","last_became_peered":"2024-09-16T05:21:07.817459+0000","last_unstale":"2024-09-16T05:22:45.962246+0000","last_undegraded":"2024-09-16T05:22:45.962246+0000","last_fullsized":"2024-09-16T05:22:45.962246+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:21:30.962662+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:21:30.962662+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011154299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":1160,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.833758+0000","last_change":"2024-09-16T05:22:38.207552+0000","last_active":"2024-09-16T05:22:45.833758+0000","last_peered":"2024-09-16T05:22:45.833758+0000","last_clean":"2024-09-16T05:22:45.833758+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:22:45.833758+0000","last_undegraded":"2024-09-16T05:22:45.833758+0000","last_fullsized":"2024-09-16T05:22:45.833758+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:22:38.207426+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:00.581775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":457,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.832284+0000","last_change":"2024-09-16T05:22:26.467188+0000","last_active":"2024-09-16T05:22:45.832284+0000","last_peered":"2024-09-16T05:22:45.832284+0000","last_clean":"2024-09-16T05:22:45.832284+0000","last_became_active":"2024-09-16T05:22:25.830349+0000","last_became_peered":"2024-09-16T05:22:25.830349+0000","last_unstale":"2024-09-16T05:22:45.832284+0000","last_undegraded":"2024-09-16T05:22:45.832284+0000","last_fullsized":"2024-09-16T05:22:45.832284+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.467087+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.467087+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:32.527602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":3500,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.891475+0000","last_change":"2024-09-16T05:22:46.891779+0000","last_active":"2024-09-16T05:22:46.891475+0000","last_peered":"2024-09-16T05:22:46.891475+0000","last_clean":"2024-09-16T05:22:46.891475+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:22:46.891475+0000","last_undegraded":"2024-09-16T05:22:46.891475+0000","last_fullsized":"2024-09-16T05:22:46.891475+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:21:43.740165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.740165+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:48.122456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070521500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"1.3","version":"35'139","reported_seq":1918,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-16T05:22:47.886124+0000","last_change":"2024-09-16T05:22:02.352598+0000","last_active":"2024-09-16T05:22:47.886124+0000","last_peered":"2024-09-16T05:22:47.886124+0000","last_clean":"2024-09-16T05:22:47.886124+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:22:47.886124+0000","last_undegraded":"2024-09-16T05:22:47.886124+0000","last_fullsized":"2024-09-16T05:22:47.886124+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:02.352511+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:22:02.352511+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:28.908704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":3578,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.843591+0000","last_change":"2024-09-16T05:22:45.843783+0000","last_active":"2024-09-16T05:22:45.843591+0000","last_peered":"2024-09-16T05:22:45.843591+0000","last_clean":"2024-09-16T05:22:45.843591+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:22:45.843591+0000","last_undegraded":"2024-09-16T05:22:45.843591+0000","last_fullsized":"2024-09-16T05:22:45.843591+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:22:36.317360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.317360+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:39.551996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017482439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":2781,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.857425+0000","last_change":"2024-09-16T05:22:44.857800+0000","last_active":"2024-09-16T05:22:44.857425+0000","last_peered":"2024-09-16T05:22:44.857425+0000","last_clean":"2024-09-16T05:22:44.857425+0000","last_became_active":"2024-09-16T05:21:54.939117+0000","last_became_peered":"2024-09-16T05:21:54.939117+0000","last_unstale":"2024-09-16T05:22:44.857425+0000","last_undegraded":"2024-09-16T05:22:44.857425+0000","last_fullsized":"2024-09-16T05:22:44.857425+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:26.067761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:26.067761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:46.618345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002873385,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":1862,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.890084+0000","last_change":"2024-09-16T05:21:26.463103+0000","last_active":"2024-09-16T05:22:46.890084+0000","last_peered":"2024-09-16T05:22:46.890084+0000","last_clean":"2024-09-16T05:22:46.890084+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:22:46.890084+0000","last_undegraded":"2024-09-16T05:22:46.890084+0000","last_fullsized":"2024-09-16T05:22:46.890084+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:26.463004+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:26.463004+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":1986,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.895411+0000","last_change":"2024-09-16T05:22:46.895411+0000","last_active":"2024-09-16T05:22:46.895411+0000","last_peered":"2024-09-16T05:22:46.895411+0000","last_clean":"2024-09-16T05:22:46.895411+0000","last_became_active":"2024-09-16T05:21:21.816054+0000","last_became_peered":"2024-09-16T05:21:21.816054+0000","last_unstale":"2024-09-16T05:22:46.895411+0000","last_undegraded":"2024-09-16T05:22:46.895411+0000","last_fullsized":"2024-09-16T05:22:46.895411+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:36.384222+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.384222+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:38.246649+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0.0035045440000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":1181,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.835690+0000","last_change":"2024-09-16T05:22:10.087414+0000","last_active":"2024-09-16T05:22:45.835690+0000","last_peered":"2024-09-16T05:22:45.835690+0000","last_clean":"2024-09-16T05:22:45.835690+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:22:45.835690+0000","last_undegraded":"2024-09-16T05:22:45.835690+0000","last_fullsized":"2024-09-16T05:22:45.835690+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:10.087343+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.087343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:22.116328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"819'672","reported_seq":4255,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.901294+0000","last_change":"2024-09-16T05:22:48.901482+0000","last_active":"2024-09-16T05:22:48.901294+0000","last_peered":"2024-09-16T05:22:48.901294+0000","last_clean":"2024-09-16T05:22:48.901294+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:22:48.901294+0000","last_undegraded":"2024-09-16T05:22:48.901294+0000","last_fullsized":"2024-09-16T05:22:48.901294+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'618","last_scrub_stamp":"2024-09-16T05:21:42.686550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:42.686550+0000","objects_scrubbed":2,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:02.909279+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0028836370000000001,"stat_sum":{"num_bytes":4386823,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":146751,"num_write":54,"num_write_kb":21531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":1125,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.834053+0000","last_change":"2024-09-16T05:22:07.007557+0000","last_active":"2024-09-16T05:22:45.834053+0000","last_peered":"2024-09-16T05:22:45.834053+0000","last_clean":"2024-09-16T05:22:45.834053+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:22:45.834053+0000","last_undegraded":"2024-09-16T05:22:45.834053+0000","last_fullsized":"2024-09-16T05:22:45.834053+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.007487+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.007487+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.416611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":553,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.912448+0000","last_change":"2024-09-16T05:22:49.912843+0000","last_active":"2024-09-16T05:22:49.912448+0000","last_peered":"2024-09-16T05:22:49.912448+0000","last_clean":"2024-09-16T05:22:49.912448+0000","last_became_active":"2024-09-16T05:21:07.816472+0000","last_became_peered":"2024-09-16T05:21:07.816472+0000","last_unstale":"2024-09-16T05:22:49.912448+0000","last_undegraded":"2024-09-16T05:22:49.912448+0000","last_fullsized":"2024-09-16T05:22:49.912448+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:21:48.443201+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:21:48.443201+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:58.309370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020002430000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":1171,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.834621+0000","last_change":"2024-09-16T05:22:08.048856+0000","last_active":"2024-09-16T05:22:45.834621+0000","last_peered":"2024-09-16T05:22:45.834621+0000","last_clean":"2024-09-16T05:22:45.834621+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:22:45.834621+0000","last_undegraded":"2024-09-16T05:22:45.834621+0000","last_fullsized":"2024-09-16T05:22:45.834621+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:08.048756+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:08.048756+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.883180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":3748,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.899382+0000","last_change":"2024-09-16T05:22:48.899574+0000","last_active":"2024-09-16T05:22:48.899382+0000","last_peered":"2024-09-16T05:22:48.899382+0000","last_clean":"2024-09-16T05:22:48.899382+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:22:48.899382+0000","last_undegraded":"2024-09-16T05:22:48.899382+0000","last_fullsized":"2024-09-16T05:22:48.899382+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:22:35.723844+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.723844+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:47.090836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094875500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":1907,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.960888+0000","last_change":"2024-09-16T05:22:10.608245+0000","last_active":"2024-09-16T05:22:45.960888+0000","last_peered":"2024-09-16T05:22:45.960888+0000","last_clean":"2024-09-16T05:22:45.960888+0000","last_became_active":"2024-09-16T05:20:35.807550+0000","last_became_peered":"2024-09-16T05:20:35.807550+0000","last_unstale":"2024-09-16T05:22:45.960888+0000","last_undegraded":"2024-09-16T05:22:45.960888+0000","last_fullsized":"2024-09-16T05:22:45.960888+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:10.608132+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.608132+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.753900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"817'726","reported_seq":4322,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.893957+0000","last_change":"2024-09-16T05:22:46.894099+0000","last_active":"2024-09-16T05:22:46.893957+0000","last_peered":"2024-09-16T05:22:46.893957+0000","last_clean":"2024-09-16T05:22:46.893957+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:22:46.893957+0000","last_undegraded":"2024-09-16T05:22:46.893957+0000","last_fullsized":"2024-09-16T05:22:46.893957+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'707","last_scrub_stamp":"2024-09-16T05:22:31.956529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:31.956529+0000","objects_scrubbed":5,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:40.228023+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.003093316,"stat_sum":{"num_bytes":10387419,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":137084,"num_write":79,"num_write_kb":40283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"1.9","version":"35'139","reported_seq":1904,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.841603+0000","last_change":"2024-09-16T05:22:04.552683+0000","last_active":"2024-09-16T05:22:45.841603+0000","last_peered":"2024-09-16T05:22:45.841603+0000","last_clean":"2024-09-16T05:22:45.841603+0000","last_became_active":"2024-09-16T05:20:35.804221+0000","last_became_peered":"2024-09-16T05:20:35.804221+0000","last_unstale":"2024-09-16T05:22:45.841603+0000","last_undegraded":"2024-09-16T05:22:45.841603+0000","last_fullsized":"2024-09-16T05:22:45.841603+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:04.552582+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:04.552582+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:17.369449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2027,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.911998+0000","last_change":"2024-09-16T05:22:49.912403+0000","last_active":"2024-09-16T05:22:49.911998+0000","last_peered":"2024-09-16T05:22:49.911998+0000","last_clean":"2024-09-16T05:22:49.911998+0000","last_became_active":"2024-09-16T05:21:10.815913+0000","last_became_peered":"2024-09-16T05:21:10.815913+0000","last_unstale":"2024-09-16T05:22:49.911998+0000","last_undegraded":"2024-09-16T05:22:49.911998+0000","last_fullsized":"2024-09-16T05:22:49.911998+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:21:43.417102+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.417102+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:56.162880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001888371,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":1910,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.854364+0000","last_change":"2024-09-16T05:21:54.943368+0000","last_active":"2024-09-16T05:22:44.854364+0000","last_peered":"2024-09-16T05:22:44.854364+0000","last_clean":"2024-09-16T05:22:44.854364+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:22:44.854364+0000","last_undegraded":"2024-09-16T05:22:44.854364+0000","last_fullsized":"2024-09-16T05:22:44.854364+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:45.767900+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:21:45.767900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:09.643423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1388,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.847004+0000","last_change":"2024-09-16T05:22:32.678108+0000","last_active":"2024-09-16T05:22:44.847004+0000","last_peered":"2024-09-16T05:22:44.847004+0000","last_clean":"2024-09-16T05:22:44.847004+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:22:44.847004+0000","last_undegraded":"2024-09-16T05:22:44.847004+0000","last_fullsized":"2024-09-16T05:22:44.847004+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:32.678034+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.678034+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:39.197820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":1457,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.836872+0000","last_change":"2024-09-16T05:22:45.837271+0000","last_active":"2024-09-16T05:22:45.836872+0000","last_peered":"2024-09-16T05:22:45.836872+0000","last_clean":"2024-09-16T05:22:45.836872+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:22:45.836872+0000","last_undegraded":"2024-09-16T05:22:45.836872+0000","last_fullsized":"2024-09-16T05:22:45.836872+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:21:41.968085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:41.968085+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:07.823019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019982720000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"1.10","version":"35'139","reported_seq":1895,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.960768+0000","last_change":"2024-09-16T05:21:29.967318+0000","last_active":"2024-09-16T05:22:45.960768+0000","last_peered":"2024-09-16T05:22:45.960768+0000","last_clean":"2024-09-16T05:22:45.960768+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:22:45.960768+0000","last_undegraded":"2024-09-16T05:22:45.960768+0000","last_fullsized":"2024-09-16T05:22:45.960768+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:29.967221+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:29.967221+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":3164,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.857467+0000","last_change":"2024-09-16T05:22:44.857733+0000","last_active":"2024-09-16T05:22:44.857467+0000","last_peered":"2024-09-16T05:22:44.857467+0000","last_clean":"2024-09-16T05:22:44.857467+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:22:44.857467+0000","last_undegraded":"2024-09-16T05:22:44.857467+0000","last_fullsized":"2024-09-16T05:22:44.857467+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:22:26.133349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.133349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:26.224034+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0.0027479869999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"1.17","version":"35'139","reported_seq":791,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.898415+0000","last_change":"2024-09-16T05:22:00.654377+0000","last_active":"2024-09-16T05:22:48.898415+0000","last_peered":"2024-09-16T05:22:48.898415+0000","last_clean":"2024-09-16T05:22:48.898415+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:22:48.898415+0000","last_undegraded":"2024-09-16T05:22:48.898415+0000","last_fullsized":"2024-09-16T05:22:48.898415+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:00.654245+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.654245+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:13.780881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":3442,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.856754+0000","last_change":"2024-09-16T05:22:44.857001+0000","last_active":"2024-09-16T05:22:44.856754+0000","last_peered":"2024-09-16T05:22:44.856754+0000","last_clean":"2024-09-16T05:22:44.856754+0000","last_became_active":"2024-09-16T05:21:54.931954+0000","last_became_peered":"2024-09-16T05:21:54.931954+0000","last_unstale":"2024-09-16T05:22:44.856754+0000","last_undegraded":"2024-09-16T05:22:44.856754+0000","last_fullsized":"2024-09-16T05:22:44.856754+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:21:46.719776+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:21:46.719776+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:53.253941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002361845,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"1.16","version":"35'139","reported_seq":845,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.891339+0000","last_change":"2024-09-16T05:22:05.738733+0000","last_active":"2024-09-16T05:22:46.891339+0000","last_peered":"2024-09-16T05:22:46.891339+0000","last_clean":"2024-09-16T05:22:46.891339+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:22:46.891339+0000","last_undegraded":"2024-09-16T05:22:46.891339+0000","last_fullsized":"2024-09-16T05:22:46.891339+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:05.738655+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:22:05.738655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:07.277398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"760'658","reported_seq":4333,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.857085+0000","last_change":"2024-09-16T05:22:44.857384+0000","last_active":"2024-09-16T05:22:44.857085+0000","last_peered":"2024-09-16T05:22:44.857085+0000","last_clean":"2024-09-16T05:22:44.857085+0000","last_became_active":"2024-09-16T05:21:07.826042+0000","last_became_peered":"2024-09-16T05:21:07.826042+0000","last_unstale":"2024-09-16T05:22:44.857085+0000","last_undegraded":"2024-09-16T05:22:44.857085+0000","last_fullsized":"2024-09-16T05:22:44.857085+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"760'658","last_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_deep_scrub":"760'658","last_deep_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_clean_scrub_stamp":"2024-09-16T05:22:05.847540+0000","objects_scrubbed":0,"log_size":658,"log_dups_size":0,"ondisk_log_size":658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:06.356678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025380770000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":7118573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"1.15","version":"35'139","reported_seq":859,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.960611+0000","last_change":"2024-09-16T05:22:06.689465+0000","last_active":"2024-09-16T05:22:45.960611+0000","last_peered":"2024-09-16T05:22:45.960611+0000","last_clean":"2024-09-16T05:22:45.960611+0000","last_became_active":"2024-09-16T05:20:40.802090+0000","last_became_peered":"2024-09-16T05:20:40.802090+0000","last_unstale":"2024-09-16T05:22:45.960611+0000","last_undegraded":"2024-09-16T05:22:45.960611+0000","last_fullsized":"2024-09-16T05:22:45.960611+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:06.689376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.689376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:29.936194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"733'334","reported_seq":3527,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.962054+0000","last_change":"2024-09-16T05:22:45.962190+0000","last_active":"2024-09-16T05:22:45.962054+0000","last_peered":"2024-09-16T05:22:45.962054+0000","last_clean":"2024-09-16T05:22:45.962054+0000","last_became_active":"2024-09-16T05:21:54.937285+0000","last_became_peered":"2024-09-16T05:21:54.937285+0000","last_unstale":"2024-09-16T05:22:45.962054+0000","last_undegraded":"2024-09-16T05:22:45.962054+0000","last_fullsized":"2024-09-16T05:22:45.962054+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'334","last_scrub_stamp":"2024-09-16T05:21:27.946398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:27.946398+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00108406,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"1.14","version":"35'139","reported_seq":1227,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.891239+0000","last_change":"2024-09-16T05:21:48.712051+0000","last_active":"2024-09-16T05:22:46.891239+0000","last_peered":"2024-09-16T05:22:46.891239+0000","last_clean":"2024-09-16T05:22:46.891239+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:22:46.891239+0000","last_undegraded":"2024-09-16T05:22:46.891239+0000","last_fullsized":"2024-09-16T05:22:46.891239+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:48.711960+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:48.711960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:02.377018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":572,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.912873+0000","last_change":"2024-09-16T05:22:49.913395+0000","last_active":"2024-09-16T05:22:49.912873+0000","last_peered":"2024-09-16T05:22:49.912873+0000","last_clean":"2024-09-16T05:22:49.912873+0000","last_became_active":"2024-09-16T05:21:27.826090+0000","last_became_peered":"2024-09-16T05:21:27.826090+0000","last_unstale":"2024-09-16T05:22:49.912873+0000","last_undegraded":"2024-09-16T05:22:49.912873+0000","last_fullsized":"2024-09-16T05:22:49.912873+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.831827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:41.961148+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0.0023829519999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.1b","version":"35'139","reported_seq":1316,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.898330+0000","last_change":"2024-09-16T05:22:07.577401+0000","last_active":"2024-09-16T05:22:48.898330+0000","last_peered":"2024-09-16T05:22:48.898330+0000","last_clean":"2024-09-16T05:22:48.898330+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:22:48.898330+0000","last_undegraded":"2024-09-16T05:22:48.898330+0000","last_fullsized":"2024-09-16T05:22:48.898330+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.577328+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.577328+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.106050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":3557,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.856620+0000","last_change":"2024-09-16T05:22:44.856966+0000","last_active":"2024-09-16T05:22:44.856620+0000","last_peered":"2024-09-16T05:22:44.856620+0000","last_clean":"2024-09-16T05:22:44.856620+0000","last_became_active":"2024-09-16T05:21:27.832312+0000","last_became_peered":"2024-09-16T05:21:27.832312+0000","last_unstale":"2024-09-16T05:22:44.856620+0000","last_undegraded":"2024-09-16T05:22:44.856620+0000","last_fullsized":"2024-09-16T05:22:44.856620+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:22:32.994229+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.994229+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:58.068005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002338316,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"1.1a","version":"35'139","reported_seq":675,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.832053+0000","last_change":"2024-09-16T05:22:07.494046+0000","last_active":"2024-09-16T05:22:45.832053+0000","last_peered":"2024-09-16T05:22:45.832053+0000","last_clean":"2024-09-16T05:22:45.832053+0000","last_became_active":"2024-09-16T05:20:40.806260+0000","last_became_peered":"2024-09-16T05:20:40.806260+0000","last_unstale":"2024-09-16T05:22:45.832053+0000","last_undegraded":"2024-09-16T05:22:45.832053+0000","last_fullsized":"2024-09-16T05:22:45.832053+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.493964+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.493964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.726913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":3195,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-16T05:22:47.886279+0000","last_change":"2024-09-16T05:22:47.886950+0000","last_active":"2024-09-16T05:22:47.886279+0000","last_peered":"2024-09-16T05:22:47.886279+0000","last_clean":"2024-09-16T05:22:47.886279+0000","last_became_active":"2024-09-16T05:21:54.930126+0000","last_became_peered":"2024-09-16T05:21:54.930126+0000","last_unstale":"2024-09-16T05:22:47.886279+0000","last_undegraded":"2024-09-16T05:22:47.886279+0000","last_fullsized":"2024-09-16T05:22:47.886279+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:35.348819+0000","last_deep_scrub":"182'60","last_deep_scrub_stamp":"2024-09-16T05:21:29.356695+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.348819+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:02.493647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091228800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.19","version":"35'139","reported_seq":1905,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.833865+0000","last_change":"2024-09-16T05:22:28.141713+0000","last_active":"2024-09-16T05:22:45.833865+0000","last_peered":"2024-09-16T05:22:45.833865+0000","last_clean":"2024-09-16T05:22:45.833865+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:22:45.833865+0000","last_undegraded":"2024-09-16T05:22:45.833865+0000","last_fullsized":"2024-09-16T05:22:45.833865+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:28.141626+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.141626+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:45.344742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"819'1229","reported_seq":4929,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.902538+0000","last_change":"2024-09-16T05:22:48.902743+0000","last_active":"2024-09-16T05:22:48.902538+0000","last_peered":"2024-09-16T05:22:48.902538+0000","last_clean":"2024-09-16T05:22:48.902538+0000","last_became_active":"2024-09-16T05:21:54.943140+0000","last_became_peered":"2024-09-16T05:21:54.943140+0000","last_unstale":"2024-09-16T05:22:48.902538+0000","last_undegraded":"2024-09-16T05:22:48.902538+0000","last_fullsized":"2024-09-16T05:22:48.902538+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"738'1131","last_scrub_stamp":"2024-09-16T05:21:28.663922+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:21:28.663922+0000","objects_scrubbed":6,"log_size":1229,"log_dups_size":0,"ondisk_log_size":1229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0040633709999999996,"stat_sum":{"num_bytes":10404193,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":27,"num_read_kb":37762,"num_write":13,"num_write_kb":4898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":19652756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.18","version":"35'139","reported_seq":1907,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.960487+0000","last_change":"2024-09-16T05:22:30.658597+0000","last_active":"2024-09-16T05:22:45.960487+0000","last_peered":"2024-09-16T05:22:45.960487+0000","last_clean":"2024-09-16T05:22:45.960487+0000","last_became_active":"2024-09-16T05:20:35.809442+0000","last_became_peered":"2024-09-16T05:20:35.809442+0000","last_unstale":"2024-09-16T05:22:45.960487+0000","last_undegraded":"2024-09-16T05:22:45.960487+0000","last_fullsized":"2024-09-16T05:22:45.960487+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:30.658505+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:30.658505+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.751732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"808'730","reported_seq":4321,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.890536+0000","last_change":"2024-09-16T05:22:46.890768+0000","last_active":"2024-09-16T05:22:46.890536+0000","last_peered":"2024-09-16T05:22:46.890536+0000","last_clean":"2024-09-16T05:22:46.890536+0000","last_became_active":"2024-09-16T05:21:27.832003+0000","last_became_peered":"2024-09-16T05:21:27.832003+0000","last_unstale":"2024-09-16T05:22:46.890536+0000","last_undegraded":"2024-09-16T05:22:46.890536+0000","last_fullsized":"2024-09-16T05:22:46.890536+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'656","last_scrub_stamp":"2024-09-16T05:21:28.523219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:28.523219+0000","objects_scrubbed":1,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:55.403540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067710000000000003,"stat_sum":{"num_bytes":7783033,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":27074,"num_write":19,"num_write_kb":6977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.1c","version":"723'472","reported_seq":3727,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.836098+0000","last_change":"2024-09-16T05:22:45.836303+0000","last_active":"2024-09-16T05:22:45.836098+0000","last_peered":"2024-09-16T05:22:45.836098+0000","last_clean":"2024-09-16T05:22:45.836098+0000","last_became_active":"2024-09-16T05:21:27.831591+0000","last_became_peered":"2024-09-16T05:21:27.831591+0000","last_unstale":"2024-09-16T05:22:45.836098+0000","last_undegraded":"2024-09-16T05:22:45.836098+0000","last_fullsized":"2024-09-16T05:22:45.836098+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"723'472","last_scrub_stamp":"2024-09-16T05:21:27.904287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:27.904287+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:46.102233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023504379999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1395,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.890452+0000","last_change":"2024-09-16T05:22:25.948680+0000","last_active":"2024-09-16T05:22:46.890452+0000","last_peered":"2024-09-16T05:22:46.890452+0000","last_clean":"2024-09-16T05:22:46.890452+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:22:46.890452+0000","last_undegraded":"2024-09-16T05:22:46.890452+0000","last_fullsized":"2024-09-16T05:22:46.890452+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:25.948598+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.948598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:47.051847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"806'1305","reported_seq":5093,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.837046+0000","last_change":"2024-09-16T05:22:45.837225+0000","last_active":"2024-09-16T05:22:45.837046+0000","last_peered":"2024-09-16T05:22:45.837046+0000","last_clean":"2024-09-16T05:22:45.837046+0000","last_became_active":"2024-09-16T05:21:27.833962+0000","last_became_peered":"2024-09-16T05:21:27.833962+0000","last_unstale":"2024-09-16T05:22:45.837046+0000","last_undegraded":"2024-09-16T05:22:45.837046+0000","last_fullsized":"2024-09-16T05:22:45.837046+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"734'1291","last_scrub_stamp":"2024-09-16T05:21:28.946046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:28.946046+0000","objects_scrubbed":0,"log_size":1305,"log_dups_size":0,"ondisk_log_size":1305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001816404,"stat_sum":{"num_bytes":2622123,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":2561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"1.1d","version":"35'139","reported_seq":746,"reported_epoch":820,"state":"active+clean","last_fresh":"2024-09-16T05:22:49.911916+0000","last_change":"2024-09-16T05:22:37.530745+0000","last_active":"2024-09-16T05:22:49.911916+0000","last_peered":"2024-09-16T05:22:49.911916+0000","last_clean":"2024-09-16T05:22:49.911916+0000","last_became_active":"2024-09-16T05:21:10.815037+0000","last_became_peered":"2024-09-16T05:21:10.815037+0000","last_unstale":"2024-09-16T05:22:49.911916+0000","last_undegraded":"2024-09-16T05:22:49.911916+0000","last_fullsized":"2024-09-16T05:22:49.911916+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:37.530632+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.530632+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:37.641748+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"736'512","reported_seq":4120,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-16T05:22:47.887454+0000","last_change":"2024-09-16T05:22:47.887651+0000","last_active":"2024-09-16T05:22:47.887454+0000","last_peered":"2024-09-16T05:22:47.887454+0000","last_clean":"2024-09-16T05:22:47.887454+0000","last_became_active":"2024-09-16T05:21:27.826212+0000","last_became_peered":"2024-09-16T05:21:27.826212+0000","last_unstale":"2024-09-16T05:22:47.887454+0000","last_undegraded":"2024-09-16T05:22:47.887454+0000","last_fullsized":"2024-09-16T05:22:47.887454+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'512","last_scrub_stamp":"2024-09-16T05:22:34.394958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.394958+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:38.050370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080308099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"1.1c","version":"35'139","reported_seq":646,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.898447+0000","last_change":"2024-09-16T05:22:06.589785+0000","last_active":"2024-09-16T05:22:48.898447+0000","last_peered":"2024-09-16T05:22:48.898447+0000","last_clean":"2024-09-16T05:22:48.898447+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:22:48.898447+0000","last_undegraded":"2024-09-16T05:22:48.898447+0000","last_fullsized":"2024-09-16T05:22:48.898447+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:06.589701+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.589701+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:21.427227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":3778,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.850835+0000","last_change":"2024-09-16T05:22:44.851066+0000","last_active":"2024-09-16T05:22:44.850835+0000","last_peered":"2024-09-16T05:22:44.850835+0000","last_clean":"2024-09-16T05:22:44.850835+0000","last_became_active":"2024-09-16T05:21:27.831337+0000","last_became_peered":"2024-09-16T05:21:27.831337+0000","last_unstale":"2024-09-16T05:22:44.850835+0000","last_undegraded":"2024-09-16T05:22:44.850835+0000","last_fullsized":"2024-09-16T05:22:44.850835+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:22:37.720977+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.720977+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:03.901338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032291569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.20","version":"815'370","reported_seq":3962,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.842461+0000","last_change":"2024-09-16T05:22:45.842461+0000","last_active":"2024-09-16T05:22:45.842461+0000","last_peered":"2024-09-16T05:22:45.842461+0000","last_clean":"2024-09-16T05:22:45.842461+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:22:45.842461+0000","last_undegraded":"2024-09-16T05:22:45.842461+0000","last_fullsized":"2024-09-16T05:22:45.842461+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"733'323","last_scrub_stamp":"2024-09-16T05:21:43.765502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.765502+0000","objects_scrubbed":1,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:11.481452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00083367200000000004,"stat_sum":{"num_bytes":8353772,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":165,"num_read_kb":195112,"num_write":70,"num_write_kb":28235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.21","version":"178'50","reported_seq":3507,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.892348+0000","last_change":"2024-09-16T05:22:46.892677+0000","last_active":"2024-09-16T05:22:46.892348+0000","last_peered":"2024-09-16T05:22:46.892348+0000","last_clean":"2024-09-16T05:22:46.892348+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:22:46.892348+0000","last_undegraded":"2024-09-16T05:22:46.892348+0000","last_fullsized":"2024-09-16T05:22:46.892348+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:22:27.632823+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.632823+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:36.572682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014012289999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.22","version":"182'20","reported_seq":1463,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.836036+0000","last_change":"2024-09-16T05:22:45.836403+0000","last_active":"2024-09-16T05:22:45.836036+0000","last_peered":"2024-09-16T05:22:45.836036+0000","last_clean":"2024-09-16T05:22:45.836036+0000","last_became_active":"2024-09-16T05:21:54.942985+0000","last_became_peered":"2024-09-16T05:21:54.942985+0000","last_unstale":"2024-09-16T05:22:45.836036+0000","last_undegraded":"2024-09-16T05:22:45.836036+0000","last_fullsized":"2024-09-16T05:22:45.836036+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:21:46.010447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:46.010447+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:53.525582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019655689999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":3055,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-16T05:22:47.887341+0000","last_change":"2024-09-16T05:22:47.887341+0000","last_active":"2024-09-16T05:22:47.887341+0000","last_peered":"2024-09-16T05:22:47.887341+0000","last_clean":"2024-09-16T05:22:47.887341+0000","last_became_active":"2024-09-16T05:22:22.838582+0000","last_became_peered":"2024-09-16T05:22:22.838582+0000","last_unstale":"2024-09-16T05:22:47.887341+0000","last_undegraded":"2024-09-16T05:22:47.887341+0000","last_fullsized":"2024-09-16T05:22:47.887341+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:22:26.319237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.319237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:40.910516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098851299999999989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.24","version":"182'89","reported_seq":2838,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.856259+0000","last_change":"2024-09-16T05:22:44.856692+0000","last_active":"2024-09-16T05:22:44.856259+0000","last_peered":"2024-09-16T05:22:44.856259+0000","last_clean":"2024-09-16T05:22:44.856259+0000","last_became_active":"2024-09-16T05:21:54.932630+0000","last_became_peered":"2024-09-16T05:21:54.932630+0000","last_unstale":"2024-09-16T05:22:44.856259+0000","last_undegraded":"2024-09-16T05:22:44.856259+0000","last_fullsized":"2024-09-16T05:22:44.856259+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:21:31.054480+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:21:31.054480+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021356190000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]},{"pgid":"2.25","version":"819'1333","reported_seq":4810,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.909682+0000","last_change":"2024-09-16T05:22:48.909923+0000","last_active":"2024-09-16T05:22:48.909682+0000","last_peered":"2024-09-16T05:22:48.909682+0000","last_clean":"2024-09-16T05:22:48.909682+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:22:48.909682+0000","last_undegraded":"2024-09-16T05:22:48.909682+0000","last_fullsized":"2024-09-16T05:22:48.909682+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"781'1244","last_scrub_stamp":"2024-09-16T05:22:12.573275+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:22:12.573275+0000","objects_scrubbed":8,"log_size":1333,"log_dups_size":0,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:27.441893+0000","scrub_duration":4,"objects_trimmed":2,"snaptrim_duration":0.011027734000000001,"stat_sum":{"num_bytes":13047045,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":250,"num_read_kb":248921,"num_write":216,"num_write_kb":85889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.26","version":"116'9","reported_seq":2834,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.842890+0000","last_change":"2024-09-16T05:22:45.843119+0000","last_active":"2024-09-16T05:22:45.842890+0000","last_peered":"2024-09-16T05:22:45.842890+0000","last_clean":"2024-09-16T05:22:45.842890+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:22:45.842890+0000","last_undegraded":"2024-09-16T05:22:45.842890+0000","last_fullsized":"2024-09-16T05:22:45.842890+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.590423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:24.533785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011858210000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.27","version":"0'0","reported_seq":2846,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.844348+0000","last_change":"2024-09-16T05:22:45.844501+0000","last_active":"2024-09-16T05:22:45.844348+0000","last_peered":"2024-09-16T05:22:45.844348+0000","last_clean":"2024-09-16T05:22:45.844348+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:22:45.844348+0000","last_undegraded":"2024-09-16T05:22:45.844348+0000","last_fullsized":"2024-09-16T05:22:45.844348+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:25.368861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.368861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:25.387616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020029499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.28","version":"245'105","reported_seq":1873,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.891400+0000","last_change":"2024-09-16T05:22:46.891810+0000","last_active":"2024-09-16T05:22:46.891400+0000","last_peered":"2024-09-16T05:22:46.891400+0000","last_clean":"2024-09-16T05:22:46.891400+0000","last_became_active":"2024-09-16T05:21:07.818030+0000","last_became_peered":"2024-09-16T05:21:07.818030+0000","last_unstale":"2024-09-16T05:22:46.891400+0000","last_undegraded":"2024-09-16T05:22:46.891400+0000","last_fullsized":"2024-09-16T05:22:46.891400+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:22:34.641066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.641066+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:56.599256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077724999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.29","version":"817'1370","reported_seq":5235,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.894793+0000","last_change":"2024-09-16T05:22:46.895101+0000","last_active":"2024-09-16T05:22:46.894793+0000","last_peered":"2024-09-16T05:22:46.894793+0000","last_clean":"2024-09-16T05:22:46.894793+0000","last_became_active":"2024-09-16T05:21:57.823393+0000","last_became_peered":"2024-09-16T05:21:57.823393+0000","last_unstale":"2024-09-16T05:22:46.894793+0000","last_undegraded":"2024-09-16T05:22:46.894793+0000","last_fullsized":"2024-09-16T05:22:46.894793+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":6,"last_scrub":"795'1297","last_scrub_stamp":"2024-09-16T05:22:26.617228+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.617228+0000","objects_scrubbed":15,"log_size":1370,"log_dups_size":0,"ondisk_log_size":1370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:42.968389+0000","scrub_duration":4,"objects_trimmed":2,"snaptrim_duration":0.005123308,"stat_sum":{"num_bytes":24752820,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":346,"num_read_kb":316397,"num_write":269,"num_write_kb":96437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":45326140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1142,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.961609+0000","last_change":"2024-09-16T05:22:45.961857+0000","last_active":"2024-09-16T05:22:45.961609+0000","last_peered":"2024-09-16T05:22:45.961609+0000","last_clean":"2024-09-16T05:22:45.961609+0000","last_became_active":"2024-09-16T05:22:25.829990+0000","last_became_peered":"2024-09-16T05:22:25.829990+0000","last_unstale":"2024-09-16T05:22:45.961609+0000","last_undegraded":"2024-09-16T05:22:45.961609+0000","last_fullsized":"2024-09-16T05:22:45.961609+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:22:26.604651+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.604651+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:29.158691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202158,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.2b","version":"817'429","reported_seq":3912,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-16T05:22:46.894542+0000","last_change":"2024-09-16T05:22:46.894746+0000","last_active":"2024-09-16T05:22:46.894542+0000","last_peered":"2024-09-16T05:22:46.894542+0000","last_clean":"2024-09-16T05:22:46.894542+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:22:46.894542+0000","last_undegraded":"2024-09-16T05:22:46.894542+0000","last_fullsized":"2024-09-16T05:22:46.894542+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"738'378","last_scrub_stamp":"2024-09-16T05:21:31.585673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:31.585673+0000","objects_scrubbed":3,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0036508780000000002,"stat_sum":{"num_bytes":6433289,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":99,"num_read_kb":89098,"num_write":69,"num_write_kb":27348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"116","length":"1"},{"start":"153","length":"2"},{"start":"157","length":"2"}]},{"pgid":"2.2c","version":"241'194","reported_seq":3695,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-16T05:22:48.901547+0000","last_change":"2024-09-16T05:22:48.902636+0000","last_active":"2024-09-16T05:22:48.901547+0000","last_peered":"2024-09-16T05:22:48.901547+0000","last_clean":"2024-09-16T05:22:48.901547+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:22:48.901547+0000","last_undegraded":"2024-09-16T05:22:48.901547+0000","last_fullsized":"2024-09-16T05:22:48.901547+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:22:48.737646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.737646+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:03.507035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038482939999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"153","length":"3"},{"start":"157","length":"2"}]},{"pgid":"2.2d","version":"808'520","reported_seq":4147,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.842704+0000","last_change":"2024-09-16T05:22:45.842838+0000","last_active":"2024-09-16T05:22:45.842704+0000","last_peered":"2024-09-16T05:22:45.842704+0000","last_clean":"2024-09-16T05:22:45.842704+0000","last_became_active":"2024-09-16T05:21:54.933266+0000","last_became_peered":"2024-09-16T05:21:54.933266+0000","last_unstale":"2024-09-16T05:22:45.842704+0000","last_undegraded":"2024-09-16T05:22:45.842704+0000","last_fullsized":"2024-09-16T05:22:45.842704+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"742'515","last_scrub_stamp":"2024-09-16T05:21:45.737196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:45.737196+0000","objects_scrubbed":4,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:47.651697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098398600000000006,"stat_sum":{"num_bytes":3882615,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":94,"num_read_kb":88584,"num_write":74,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":10583843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.2e","version":"815'777","reported_seq":4269,"reported_epoch":816,"state":"active+clean","last_fresh":"2024-09-16T05:22:45.843345+0000","last_change":"2024-09-16T05:22:45.843345+0000","last_active":"2024-09-16T05:22:45.843345+0000","last_peered":"2024-09-16T05:22:45.843345+0000","last_clean":"2024-09-16T05:22:45.843345+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:22:45.843345+0000","last_undegraded":"2024-09-16T05:22:45.843345+0000","last_fullsized":"2024-09-16T05:22:45.843345+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"794'746","last_scrub_stamp":"2024-09-16T05:22:28.624283+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.624283+0000","objects_scrubbed":7,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:51.578268+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0.0013935849999999999,"stat_sum":{"num_bytes":14051774,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":110,"num_read_kb":73045,"num_write":111,"num_write_kb":51672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"116","length":"1"},{"start":"154","length":"1"},{"start":"157","length":"2"}]},{"pgid":"2.2f","version":"814'707","reported_seq":4291,"reported_epoch":815,"state":"active+clean","last_fresh":"2024-09-16T05:22:44.857017+0000","last_change":"2024-09-16T05:22:44.857408+0000","last_active":"2024-09-16T05:22:44.857017+0000","last_peered":"2024-09-16T05:22:44.857017+0000","last_clean":"2024-09-16T05:22:44.857017+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:22:44.857017+0000","last_undegraded":"2024-09-16T05:22:44.857017+0000","last_fullsized":"2024-09-16T05:22:44.857017+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"795'683","last_scrub_stamp":"2024-09-16T05:22:27.948411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.948411+0000","objects_scrubbed":5,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:50.792837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027374550000000002,"stat_sum":{"num_bytes":11010793,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":127,"num_read_kb":85530,"num_write":134,"num_write_kb":50449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"126","length":"1"},{"start":"14b","length":"1"},{"start":"151","length":"2"},{"start":"154","length":"1"},{"start":"157","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":96,"stat_sum":{"num_bytes":297911482,"num_objects":133,"num_object_clones":86,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":17,"num_read":3537,"num_read_kb":3531702,"num_write":2484,"num_write_kb":1018877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397639680,"data_stored":453689116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32918,"ondisk_log_size":32918,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089509,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112620,"kb_used_data":55056,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93632532,"statfs":{"total":95995035648,"available":95879712768,"internally_reserved":0,"allocated":56377344,"data_stored":59763540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":58915268},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285737,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106148,"kb_used_data":48584,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93639004,"statfs":{"total":95995035648,"available":95886340096,"internally_reserved":0,"allocated":49750016,"data_stored":52663162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514674,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83336,"kb_used_data":38252,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93661816,"statfs":{"total":95995035648,"available":95909699584,"internally_reserved":0,"allocated":39170048,"data_stored":42792456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841715,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111224,"kb_used_data":53668,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93633928,"statfs":{"total":95995035648,"available":95881142272,"internally_reserved":0,"allocated":54956032,"data_stored":56055785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103352,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93060,"kb_used_data":43820,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93652092,"statfs":{"total":95995035648,"available":95899742208,"internally_reserved":0,"allocated":44871680,"data_stored":43675240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397695,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94944,"kb_used_data":49868,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93650208,"statfs":{"total":95995035648,"available":95897812992,"internally_reserved":0,"allocated":51064832,"data_stored":52687472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757441,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75000,"kb_used_data":43808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93670152,"statfs":{"total":95995035648,"available":95918235648,"internally_reserved":0,"allocated":44859392,"data_stored":47117808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920967,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60572,"kb_used_data":25224,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684580,"statfs":{"total":95995035648,"available":95933009920,"internally_reserved":0,"allocated":25829376,"data_stored":29764881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117196,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114988,"kb_used_data":69908,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93630164,"statfs":{"total":95995035648,"available":95877287936,"internally_reserved":0,"allocated":71585792,"data_stored":72498949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240483,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65972,"kb_used_data":37852,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93679180,"statfs":{"total":95995035648,"available":95927480320,"internally_reserved":0,"allocated":38760448,"data_stored":39890899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85700,"kb_used_data":28140,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93659452,"statfs":{"total":95995035648,"available":95907278848,"internally_reserved":0,"allocated":28815360,"data_stored":28239627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738587,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82944,"kb_used_data":29540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93662208,"statfs":{"total":95995035648,"available":95910100992,"internally_reserved":0,"allocated":30248960,"data_stored":32722605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20045824,"data_stored":25786874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13987840,"data_stored":17705725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28594176,"data_stored":33003128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58793984,"data_stored":62331741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15953920,"data_stored":23156830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34779136,"data_stored":40308509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":39817216,"data_stored":42412381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":34983936,"data_stored":37058613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":35037184,"data_stored":36547342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":29294592,"data_stored":36184405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":39874560,"data_stored":46055111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":46477312,"data_stored":53138457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:22:51.709 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:22:51.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-09-16T05:22:51.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:51.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:51.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:51.800 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:3262: done (0 left) 2024-09-16T05:22:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:3263: read oid 40 snap -1 2024-09-16T05:22:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:3263: expect deleted 2024-09-16T05:22:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:3264: copy_from oid 9 from oid 27 current snap is 342 2024-09-16T05:22:51.837 INFO:tasks.rados.rados.2.smithi204.stdout:3265: snap_remove snap 331 2024-09-16T05:22:51.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:51.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:51.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:51.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: pgmap v1392: 127 pgs: 127 active+clean; 285 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 10 op/s 2024-09-16T05:22:51.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: from='client.17232 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:22:51.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:22:51.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:22:51.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:22:51.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:22:51.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:22:51.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:22:51.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: 2.5d scrub starts 2024-09-16T05:22:51.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: 2.5d scrub ok 2024-09-16T05:22:51.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:51 smithi032 ceph-mon[27388]: osdmap e822: 12 total, 12 up, 12 in 2024-09-16T05:22:51.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:51.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:51.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:52.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:52.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:52.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: pgmap v1392: 127 pgs: 127 active+clean; 285 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 10 op/s 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: from='client.17232 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: 2.5d scrub starts 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: 2.5d scrub ok 2024-09-16T05:22:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:51 smithi204 ceph-mon[30933]: osdmap e822: 12 total, 12 up, 12 in 2024-09-16T05:22:52.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: pgmap v1392: 127 pgs: 127 active+clean; 285 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 10 op/s 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: from='client.17232 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: 2.5d scrub starts 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: 2.5d scrub ok 2024-09-16T05:22:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:51 smithi196 ceph-mon[30802]: osdmap e822: 12 total, 12 up, 12 in 2024-09-16T05:22:52.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:52.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:52.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:22:52.337 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:52.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:52.362 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:52.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:52.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:52.487 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:52.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:52.571 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:22:52.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:52.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:52.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:52.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:52.738 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:52.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:52.834 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:22:52.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:52.841 INFO:tasks.rados.rados.2.smithi204.stdout:3265: done (2 left) 2024-09-16T05:22:52.841 INFO:tasks.rados.rados.2.smithi204.stdout:3266: write (excl) oid 44 current snap is 342 2024-09-16T05:22:52.841 INFO:tasks.rados.rados.2.smithi204.stdout:3266: seq_num 1145 ranges {444015=413055,1436000=773735,2709915=534100} 2024-09-16T05:22:52.846 INFO:tasks.rados.rados.2.smithi204.stdout:3266: writing smithi20457400-44 from 444015 to 857070 tid 1 2024-09-16T05:22:52.855 INFO:tasks.rados.rados.2.smithi204.stdout:3266: writing smithi20457400-44 from 1436000 to 2209735 tid 2 2024-09-16T05:22:52.862 INFO:tasks.rados.rados.2.smithi204.stdout:3266: writing smithi20457400-44 from 2709915 to 3244015 tid 3 2024-09-16T05:22:52.862 INFO:tasks.rados.rados.2.smithi204.stdout:3267: delete oid 28 current snap is 342 2024-09-16T05:22:52.862 INFO:tasks.rados.rados.2.smithi204.stdout:3264: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:22:52.862 INFO:tasks.rados.rados.2.smithi204.stdout:3264: finishing copy_from to smithi20457400-9 2024-09-16T05:22:52.862 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1265 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-09-16T05:22:52.862 INFO:tasks.rados.rados.2.smithi204.stdout:3266: finishing write tid 1 to smithi20457400-44 2024-09-16T05:22:52.863 INFO:tasks.rados.rados.2.smithi204.stdout:3263: done (3 left) 2024-09-16T05:22:52.863 INFO:tasks.rados.rados.2.smithi204.stdout:3264: done (2 left) 2024-09-16T05:22:52.863 INFO:tasks.rados.rados.2.smithi204.stdout:3267: done (1 left) 2024-09-16T05:22:52.863 INFO:tasks.rados.rados.2.smithi204.stdout:3268: read oid 8 snap -1 2024-09-16T05:22:52.863 INFO:tasks.rados.rados.2.smithi204.stdout:3268: expect deleted 2024-09-16T05:22:52.864 INFO:tasks.rados.rados.2.smithi204.stdout:3269: write oid 36 current snap is 342 2024-09-16T05:22:52.864 INFO:tasks.rados.rados.2.smithi204.stdout:3269: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-09-16T05:22:52.869 INFO:tasks.rados.rados.2.smithi204.stdout:3269: writing smithi20457400-36 from 477629 to 1041182 tid 1 2024-09-16T05:22:52.872 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:52.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:52.877 INFO:tasks.rados.rados.2.smithi204.stdout:3269: writing smithi20457400-36 from 1836611 to 2614015 tid 2 2024-09-16T05:22:52.882 INFO:tasks.rados.rados.2.smithi204.stdout:3269: writing smithi20457400-36 from 3048408 to 3277629 tid 3 2024-09-16T05:22:52.882 INFO:tasks.rados.rados.2.smithi204.stdout:3270: copy_from oid 42 from oid 13 current snap is 342 2024-09-16T05:22:52.882 INFO:tasks.rados.rados.2.smithi204.stdout:3266: finishing write tid 2 to smithi20457400-44 2024-09-16T05:22:52.882 INFO:tasks.rados.rados.2.smithi204.stdout:3266: finishing write tid 3 to smithi20457400-44 2024-09-16T05:22:52.882 INFO:tasks.rados.rados.2.smithi204.stdout:3266: finishing write tid 4 to smithi20457400-44 2024-09-16T05:22:52.882 INFO:tasks.rados.rados.2.smithi204.stdout:3266: finishing write tid 5 to smithi20457400-44 2024-09-16T05:22:52.882 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1145 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:22:52.882 INFO:tasks.rados.rados.2.smithi204.stdout:3266: left oid 44 (ObjNum 1145 snap 342 seq_num 1145) 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3269: finishing write tid 1 to smithi20457400-36 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3266: done (3 left) 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3268: done (2 left) 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3271: read oid 15 snap -1 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3271: expect (ObjNum 1087 snap 330 seq_num 1087) 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3272: rollback oid 32 current snap is 342 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3272: done (3 left) 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3273: write oid 6 current snap is 342 2024-09-16T05:22:52.883 INFO:tasks.rados.rados.2.smithi204.stdout:3273: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2024-09-16T05:22:52.888 INFO:tasks.rados.rados.2.smithi204.stdout:3273: writing smithi20457400-6 from 494436 to 1249591 tid 1 2024-09-16T05:22:52.896 INFO:tasks.rados.rados.2.smithi204.stdout:3273: writing smithi20457400-6 from 2011446 to 2532508 tid 2 2024-09-16T05:22:52.900 INFO:tasks.rados.rados.2.smithi204.stdout:3273: writing smithi20457400-6 from 3075831 to 3294436 tid 3 2024-09-16T05:22:52.900 INFO:tasks.rados.rados.2.smithi204.stdout:3274: write oid 11 current snap is 342 2024-09-16T05:22:52.900 INFO:tasks.rados.rados.2.smithi204.stdout:3274: seq_num 1149 ranges {511243=430404,1669928=664720,3103254=207989} 2024-09-16T05:22:52.906 INFO:tasks.rados.rados.2.smithi204.stdout:3274: writing smithi20457400-11 from 511243 to 941647 tid 1 2024-09-16T05:22:52.913 INFO:tasks.rados.rados.2.smithi204.stdout:3274: writing smithi20457400-11 from 1669928 to 2334648 tid 2 2024-09-16T05:22:52.918 INFO:tasks.rados.rados.2.smithi204.stdout:3274: writing smithi20457400-11 from 3103254 to 3311243 tid 3 2024-09-16T05:22:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:3275: read oid 14 snap -1 2024-09-16T05:22:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:3275: expect (ObjNum 1080 snap 329 seq_num 1080) 2024-09-16T05:22:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:3276: write oid 19 current snap is 342 2024-09-16T05:22:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:3270: finishing copy_from to smithi20457400-42 2024-09-16T05:22:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:3270: got expected ENOENT (src dne) 2024-09-16T05:22:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:3270: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:22:52.919 INFO:tasks.rados.rados.2.smithi204.stdout:3271: expect (ObjNum 1087 snap 330 seq_num 1087) 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3648169433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 2.5b scrub starts 2024-09-16T05:22:52.953 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: 2.5b scrub ok 2024-09-16T05:22:52.954 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:52.954 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:22:52.954 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3648169433' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:52.954 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:52 smithi204 ceph-mon[30933]: osdmap e823: 12 total, 12 up, 12 in 2024-09-16T05:22:52.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:52.999 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:52.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:53.003 INFO:teuthology.orchestra.run.smithi032.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-09-16T05:22:53.046 INFO:tasks.rados.rados.2.smithi204.stdout:3269: finishing write tid 2 to smithi20457400-36 2024-09-16T05:22:53.046 INFO:tasks.rados.rados.2.smithi204.stdout:3269: finishing write tid 3 to smithi20457400-36 2024-09-16T05:22:53.046 INFO:tasks.rados.rados.2.smithi204.stdout:3276: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-09-16T05:22:53.051 INFO:tasks.rados.rados.2.smithi204.stdout:3276: writing smithi20457400-19 from 528050 to 1033703 tid 1 2024-09-16T05:22:53.061 INFO:tasks.rados.rados.2.smithi204.stdout:3276: writing smithi20457400-19 from 1612057 to 2136788 tid 2 2024-09-16T05:22:53.064 INFO:tasks.rados.rados.2.smithi204.stdout:3276: writing smithi20457400-19 from 2614324 to 3181727 tid 3 2024-09-16T05:22:53.064 INFO:tasks.rados.rados.2.smithi204.stdout:3270: done (6 left) 2024-09-16T05:22:53.064 INFO:tasks.rados.rados.2.smithi204.stdout:3271: done (5 left) 2024-09-16T05:22:53.064 INFO:tasks.rados.rados.2.smithi204.stdout:3277: snap_create 2024-09-16T05:22:53.064 INFO:tasks.rados.rados.2.smithi204.stdout:3269: finishing write tid 4 to smithi20457400-36 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3269: finishing write tid 5 to smithi20457400-36 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 790 (ObjNum 1147 snap 342 seq_num 1147) dirty exists 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3269: left oid 36 (ObjNum 1147 snap 342 seq_num 1147) 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3273: finishing write tid 1 to smithi20457400-6 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3273: finishing write tid 2 to smithi20457400-6 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3273: finishing write tid 3 to smithi20457400-6 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3273: finishing write tid 4 to smithi20457400-6 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3274: finishing write tid 1 to smithi20457400-11 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3274: finishing write tid 2 to smithi20457400-11 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3274: finishing write tid 3 to smithi20457400-11 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3274: finishing write tid 4 to smithi20457400-11 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3274: finishing write tid 5 to smithi20457400-11 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1347 (ObjNum 1149 snap 342 seq_num 1149) dirty exists 2024-09-16T05:22:53.065 INFO:tasks.rados.rados.2.smithi204.stdout:3274: left oid 11 (ObjNum 1149 snap 342 seq_num 1149) 2024-09-16T05:22:53.066 INFO:tasks.rados.rados.2.smithi204.stdout:3275: expect (ObjNum 1080 snap 329 seq_num 1080) 2024-09-16T05:22:53.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:22:53.089 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:22:53.089 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:22:53.089 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:22:53.089 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:22:53.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:53.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3648169433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 2.5b scrub starts 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: 2.5b scrub ok 2024-09-16T05:22:53.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:53.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:22:53.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3648169433' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:53.113 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:52 smithi032 ceph-mon[27388]: osdmap e823: 12 total, 12 up, 12 in 2024-09-16T05:22:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:53.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:53.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:53.163 INFO:tasks.rados.rados.2.smithi204.stdout:3276: finishing write tid 1 to smithi20457400-19 2024-09-16T05:22:53.163 INFO:tasks.rados.rados.2.smithi204.stdout:3276: finishing write tid 2 to smithi20457400-19 2024-09-16T05:22:53.164 INFO:tasks.rados.rados.2.smithi204.stdout:3276: finishing write tid 3 to smithi20457400-19 2024-09-16T05:22:53.164 INFO:tasks.rados.rados.2.smithi204.stdout:3276: finishing write tid 4 to smithi20457400-19 2024-09-16T05:22:53.164 INFO:tasks.rados.rados.2.smithi204.stdout:3276: finishing write tid 5 to smithi20457400-19 2024-09-16T05:22:53.164 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 860 (ObjNum 1150 snap 342 seq_num 1150) dirty exists 2024-09-16T05:22:53.164 INFO:tasks.rados.rados.2.smithi204.stdout:3276: left oid 19 (ObjNum 1150 snap 342 seq_num 1150) 2024-09-16T05:22:53.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3648169433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 2.5b scrub starts 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: 2.5b scrub ok 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:22:53.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3648169433' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:22:53.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:52 smithi196 ceph-mon[30802]: osdmap e823: 12 total, 12 up, 12 in 2024-09-16T05:22:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:53.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:53.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:53.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:53.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:53.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:53.402 INFO:tasks.rados.rados.2.smithi204.stdout:3273: finishing write tid 5 to smithi20457400-6 2024-09-16T05:22:53.402 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 1244 (ObjNum 1148 snap 342 seq_num 1148) dirty exists 2024-09-16T05:22:53.402 INFO:tasks.rados.rados.2.smithi204.stdout:3273: left oid 6 (ObjNum 1148 snap 342 seq_num 1148) 2024-09-16T05:22:53.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:53.512 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:53.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:53.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:53.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:53.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:53.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:53.767 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:53.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:53.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:22:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T05:22:53.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:53.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:53.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3269: done (5 left) 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3273: done (4 left) 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3274: done (3 left) 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3275: done (2 left) 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3276: done (1 left) 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3277: done (0 left) 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3278: rollback oid 25 current snap is 343 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 342 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3279: read oid 19 snap -1 2024-09-16T05:22:53.918 INFO:tasks.rados.rados.2.smithi204.stdout:3279: expect (ObjNum 1150 snap 342 seq_num 1150) 2024-09-16T05:22:53.919 INFO:tasks.rados.rados.2.smithi204.stdout:3280: copy_from oid 20 from oid 18 current snap is 343 2024-09-16T05:22:53.919 INFO:tasks.rados.rados.2.smithi204.stdout:3281: write oid 5 current snap is 343 2024-09-16T05:22:53.919 INFO:tasks.rados.rados.2.smithi204.stdout:3281: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2024-09-16T05:22:53.925 INFO:tasks.rados.rados.2.smithi204.stdout:3281: writing smithi20457400-5 from 544857 to 1125759 tid 1 2024-09-16T05:22:53.932 INFO:tasks.rados.rados.2.smithi204.stdout:3281: writing smithi20457400-5 from 1670539 to 2338928 tid 2 2024-09-16T05:22:53.938 INFO:tasks.rados.rados.2.smithi204.stdout:3281: writing smithi20457400-5 from 3041747 to 3344857 tid 3 2024-09-16T05:22:53.938 INFO:tasks.rados.rados.2.smithi204.stdout:3282: write (excl) oid 2 current snap is 343 2024-09-16T05:22:53.938 INFO:tasks.rados.rados.2.smithi204.stdout:3278: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:22:53.938 INFO:tasks.rados.rados.2.smithi204.stdout:3278: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:22:53.938 INFO:tasks.rados.rados.2.smithi204.stdout:3278: finishing rollback tid 2 to smithi20457400-25 2024-09-16T05:22:53.938 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1268 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-09-16T05:22:53.938 INFO:tasks.rados.rados.2.smithi204.stdout:3279: expect (ObjNum 1150 snap 342 seq_num 1150) 2024-09-16T05:22:53.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:54.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:54.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:54.077 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:22:54.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:54.144 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:54.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:54.158 INFO:tasks.rados.rados.2.smithi204.stdout:3281: finishing write tid 1 to smithi20457400-5 2024-09-16T05:22:54.158 INFO:tasks.rados.rados.2.smithi204.stdout:3280: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:22:54.158 INFO:tasks.rados.rados.2.smithi204.stdout:3280: finishing copy_from to smithi20457400-20 2024-09-16T05:22:54.158 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 750 (ObjNum 1087 snap 330 seq_num 1087) dirty exists 2024-09-16T05:22:54.158 INFO:tasks.rados.rados.2.smithi204.stdout:3281: finishing write tid 2 to smithi20457400-5 2024-09-16T05:22:54.159 INFO:tasks.rados.rados.2.smithi204.stdout:3282: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2024-09-16T05:22:54.164 INFO:tasks.rados.rados.2.smithi204.stdout:3282: writing smithi20457400-2 from 561664 to 1217815 tid 1 2024-09-16T05:22:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:53 smithi032 ceph-mon[27388]: pgmap v1395: 127 pgs: 127 active+clean; 221 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 12 op/s 2024-09-16T05:22:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:53 smithi032 ceph-mon[27388]: 2.5c scrub starts 2024-09-16T05:22:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:53 smithi032 ceph-mon[27388]: 2.5c scrub ok 2024-09-16T05:22:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3400853126' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-09-16T05:22:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:53 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:22:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:53 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:22:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:53 smithi204 ceph-mon[30933]: pgmap v1395: 127 pgs: 127 active+clean; 221 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 12 op/s 2024-09-16T05:22:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:53 smithi204 ceph-mon[30933]: 2.5c scrub starts 2024-09-16T05:22:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:53 smithi204 ceph-mon[30933]: 2.5c scrub ok 2024-09-16T05:22:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3400853126' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-09-16T05:22:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:53 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:22:54.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:53 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:22:54.172 INFO:tasks.rados.rados.2.smithi204.stdout:3282: writing smithi20457400-2 from 1729021 to 2257421 tid 2 2024-09-16T05:22:54.177 INFO:tasks.rados.rados.2.smithi204.stdout:3282: writing smithi20457400-2 from 2669170 to 3361664 tid 3 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3278: done (4 left) 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3279: done (3 left) 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3280: done (2 left) 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3283: snap_create 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3281: finishing write tid 3 to smithi20457400-5 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3281: finishing write tid 4 to smithi20457400-5 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3281: finishing write tid 5 to smithi20457400-5 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 629 (ObjNum 1151 snap 343 seq_num 1151) dirty exists 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3281: left oid 5 (ObjNum 1151 snap 343 seq_num 1151) 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3282: finishing write tid 1 to smithi20457400-2 2024-09-16T05:22:54.178 INFO:tasks.rados.rados.2.smithi204.stdout:3282: finishing write tid 2 to smithi20457400-2 2024-09-16T05:22:54.180 INFO:tasks.rados.rados.2.smithi204.stdout:3282: finishing write tid 3 to smithi20457400-2 2024-09-16T05:22:54.187 INFO:tasks.rados.rados.2.smithi204.stdout:3282: finishing write tid 4 to smithi20457400-2 2024-09-16T05:22:54.188 INFO:tasks.rados.rados.2.smithi204.stdout:3282: finishing write tid 5 to smithi20457400-2 2024-09-16T05:22:54.188 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 540 (ObjNum 1152 snap 343 seq_num 1152) dirty exists 2024-09-16T05:22:54.188 INFO:tasks.rados.rados.2.smithi204.stdout:3282: left oid 2 (ObjNum 1152 snap 343 seq_num 1152) 2024-09-16T05:22:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:53 smithi196 ceph-mon[30802]: pgmap v1395: 127 pgs: 127 active+clean; 221 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 12 op/s 2024-09-16T05:22:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:53 smithi196 ceph-mon[30802]: 2.5c scrub starts 2024-09-16T05:22:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:53 smithi196 ceph-mon[30802]: 2.5c scrub ok 2024-09-16T05:22:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3400853126' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-09-16T05:22:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:53 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:22:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:53 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:22:54.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:54.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:54.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:54.320 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:54.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:54.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:54.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:54.492 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:54.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:54.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:54.581 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:22:54.581 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:22:54.581 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:22:54.581 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:22:54.581 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:22:54.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:54.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:54.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:54.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:54.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:54.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:54.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:54.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:54.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:54.932 INFO:tasks.rados.rados.2.smithi204.stdout:3281: done (2 left) 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3282: done (1 left) 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3283: done (0 left) 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3284: read oid 27 snap 343 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3284: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3285: read oid 23 snap -1 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3285: expect (ObjNum 899 snap 263 seq_num 899) 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3286: read oid 11 snap 342 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3286: expect (ObjNum 1149 snap 342 seq_num 1149) 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3287: rollback oid 25 current snap is 344 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 342 2024-09-16T05:22:54.933 INFO:tasks.rados.rados.2.smithi204.stdout:3288: delete oid 5 current snap is 344 2024-09-16T05:22:54.935 INFO:tasks.rados.rados.2.smithi204.stdout:3287: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:22:54.936 INFO:tasks.rados.rados.2.smithi204.stdout:3287: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:22:54.937 INFO:tasks.rados.rados.2.smithi204.stdout:3288: done (4 left) 2024-09-16T05:22:54.937 INFO:tasks.rados.rados.2.smithi204.stdout:3289: rollback oid 31 current snap is 344 2024-09-16T05:22:54.937 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 31 to 343 2024-09-16T05:22:54.937 INFO:tasks.rados.rados.2.smithi204.stdout:3290: snap_create 2024-09-16T05:22:54.938 INFO:tasks.rados.rados.2.smithi204.stdout:3289: finishing rollback tid 1 to smithi20457400-31 2024-09-16T05:22:54.938 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 2385 snap 0 seq_num 4283161424) dirty dne 2024-09-16T05:22:54.939 INFO:tasks.rados.rados.2.smithi204.stdout:3286: expect (ObjNum 1149 snap 342 seq_num 1149) 2024-09-16T05:22:54.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:55.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:55.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:55.090 INFO:tasks.rados.rados.2.smithi204.stdout:3284: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:22:55.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:55.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:55.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: 2.55 scrub starts 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: 2.55 scrub ok 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: osdmap e824: 12 total, 12 up, 12 in 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:22:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/20170222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:55.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:22:55.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: 2.55 scrub starts 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: 2.55 scrub ok 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: osdmap e824: 12 total, 12 up, 12 in 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:22:55.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/20170222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:55.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:55.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:55.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:55.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:55.295 INFO:tasks.rados.rados.2.smithi204.stdout:3285: expect (ObjNum 899 snap 263 seq_num 899) 2024-09-16T05:22:55.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:55.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:55.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:55.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:55.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: 2.55 scrub starts 2024-09-16T05:22:55.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: 2.55 scrub ok 2024-09-16T05:22:55.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: osdmap e824: 12 total, 12 up, 12 in 2024-09-16T05:22:55.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:22:55.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:22:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:22:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:22:55.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/20170222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:22:55.456 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:22:55.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:55.503 INFO:tasks.rados.rados.2.smithi204.stdout:3287: finishing rollback tid 2 to smithi20457400-25 2024-09-16T05:22:55.503 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1271 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-09-16T05:22:55.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:55.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:55.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:55.636 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:22:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:22:55.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:55.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:55.659 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:22:55.659 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:22:55.680 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:22:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:22:55.701 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:22:55.702 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:22:55.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:55.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:55.836 INFO:tasks.rados.rados.2.smithi204.stdout:3284: done (5 left) 2024-09-16T05:22:55.836 INFO:tasks.rados.rados.2.smithi204.stdout:3285: done (4 left) 2024-09-16T05:22:55.836 INFO:tasks.rados.rados.2.smithi204.stdout:3286: done (3 left) 2024-09-16T05:22:55.836 INFO:tasks.rados.rados.2.smithi204.stdout:3287: done (2 left) 2024-09-16T05:22:55.836 INFO:tasks.rados.rados.2.smithi204.stdout:3289: done (1 left) 2024-09-16T05:22:55.836 INFO:tasks.rados.rados.2.smithi204.stdout:3290: done (0 left) 2024-09-16T05:22:55.836 INFO:tasks.rados.rados.2.smithi204.stdout:3291: snap_remove snap 344 2024-09-16T05:22:55.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:55.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:55.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:55.924 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:22:55.924 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:22:55.947 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:22:55.948 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:22:55.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:22:55.968 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:22:55.990 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:22:55.990 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:22:56.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:56.004 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:22:56.004 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:22:56.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:22:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:22:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1814 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:22:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:22:56.005 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 597.17018172400003, 2024-09-16T05:22:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898298185, 2024-09-16T05:22:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:22:56.006 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1814 2024-09-16T05:22:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.007 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:22:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771774+0000", 2024-09-16T05:22:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771776+0000", 2024-09-16T05:22:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5149999999999999e-06 2024-09-16T05:22:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771783+0000", 2024-09-16T05:22:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0180000000000004e-06 2024-09-16T05:22:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771805+0000", 2024-09-16T05:22:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2076999999999999e-05 2024-09-16T05:22:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771824+0000", 2024-09-16T05:22:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8813000000000001e-05 2024-09-16T05:22:56.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747524+0000", 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97570031700000004 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747560+0000", 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5549000000000001e-05 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670057+0000", 2024-09-16T05:22:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92249772500000005 2024-09-16T05:22:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670072+0000", 2024-09-16T05:22:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5170999999999999e-05 2024-09-16T05:22:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1815 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 597.17007982500002, 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8982342750000001, 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1815 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771876+0000", 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771877+0000", 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9599999999999999e-07 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771881+0000", 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6880000000000001e-06 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771889+0000", 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6789999999999994e-06 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771896+0000", 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747584+0000", 2024-09-16T05:22:56.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97568814000000004 2024-09-16T05:22:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:22:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747590+0000", 2024-09-16T05:22:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7440000000000004e-06 2024-09-16T05:22:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670102+0000", 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92251251499999998 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670110+0000", 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2740000000000007e-06 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1816 2.25 2.cc9e77e5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e247)", 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 597.16999783599999, 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.898192844, 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1816 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.023 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771958+0000", 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771959+0000", 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1999999999999998e-07 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771964+0000", 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4499999999999997e-06 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771971+0000", 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.576e-06 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:58.771977+0000", 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2960000000000004e-06 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747609+0000", 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.97563200299999997 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:12:59.747615+0000", 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2820000000000004e-06 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670144+0000", 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92252909100000002 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:00.670151+0000", 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2259999999999999e-06 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:1881 2.25 2:a49174eb:::smithi20457400-39:head [delete] snapc 50=[50,4f,4e,4d,4c,4a,49,48,47,45,44,3e,3a,39,35,27] ondisk+write+known_if_redirected+full_force+supports_pool_eio e257)", 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 587.24540853500002, 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37621878800000003, 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.028 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1881 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696547+0000", 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696549+0000", 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2550000000000001e-06 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696556+0000", 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8090000000000004e-06 2024-09-16T05:22:56.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696568+0000", 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2636000000000001e-05 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696576+0000", 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3459999999999996e-06 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696625+0000", 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9830000000000002e-05 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696720+0000", 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4816999999999997e-05 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.696869+0000", 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014893999999999999 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.697931+0000", 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010616759999999999 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072669+0000", 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37473788800000002 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:22:56.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072715+0000", 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6371e-05 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072766+0000", 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1220000000000001e-05 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:1881 2.25 e257/252 ondisk, result = 0)", 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 587.24388028600004, 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37469565799999999, 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:22:56.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698076+0000", 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698077+0000", 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5099999999999998e-07 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:08.698082+0000", 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1340000000000003e-06 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072556+0000", 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.37447454499999999 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072564+0000", 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4120000000000004e-06 2024-09-16T05:22:56.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072654+0000", 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9878999999999998e-05 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072663+0000", 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:09.072771+0000", 2024-09-16T05:22:56.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108267 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:839 2.1a e438/291 ondisk, result = 0)", 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 388.21146686499998, 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51227693699999999, 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730490+0000", 2024-09-16T05:22:56.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730494+0000", 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0810000000000004e-06 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242338+0000", 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51184441999999997 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242353+0000", 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.447e-05 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242501+0000", 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014851799999999999 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242532+0000", 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0406999999999999e-05 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242766+0000", 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000234587 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGRecoveryDelete(2.4c e439,438 2:33100188:::smithi20457400-19:2f,230'190)", 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 387.10855575800002, 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.584288378, 2024-09-16T05:22:56.043 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:22:56.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833401+0000", 2024-09-16T05:22:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:22:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833406+0000", 2024-09-16T05:22:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8940000000000001e-06 2024-09-16T05:22:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417262+0000", 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.58385600999999998 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417279+0000", 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6880000000000001e-05 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417458+0000", 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000178751 2024-09-16T05:22:56.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417489+0000", 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1192999999999998e-05 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417689+0000", 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019989399999999999 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5460 2.25 2:a5bc1608:::smithi20457400-11:head [setxattr _header (58) in=7b,truncate 2975004] snapc e1=[e1,e0,df,dd,dc,da,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 251.94752944000001, 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69151579299999999, 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5460 2024-09-16T05:22:56.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994430+0000", 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2729999999999998e-06 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994437+0000", 2024-09-16T05:22:56.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9670000000000004e-06 2024-09-16T05:22:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994443+0000", 2024-09-16T05:22:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7780000000000004e-06 2024-09-16T05:22:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995089+0000", 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064609900000000002 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995143+0000", 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4042000000000002e-05 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995231+0000", 2024-09-16T05:22:56.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7798000000000002e-05 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995551+0000", 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000319847 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685849+0000", 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.690298254 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685893+0000", 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3368000000000002e-05 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685942+0000", 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9811000000000001e-05 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5461 2.25 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:22:56.055 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 251.94750229499999, 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69165347799999999, 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5461 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994457+0000", 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.429e-06 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994463+0000", 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994469+0000", 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4380000000000003e-06 2024-09-16T05:22:56.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995446+0000", 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00097678900000000004 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995491+0000", 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5153e-05 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685982+0000", 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69049136499999997 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686042+0000", 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9896999999999999e-05 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686107+0000", 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5270000000000004e-05 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.060 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6242 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 913] snapc 103=[103,102,101,100,fe,fd,fc,fb,fa,f9,f7,f6,f2,f0,e7] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 188.12862868799999, 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47407041599999999, 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.061 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6242 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:22:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813328+0000", 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9399999999999993e-07 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813334+0000", 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1619999999999996e-06 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813347+0000", 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2145e-05 2024-09-16T05:22:56.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813354+0000", 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1230000000000002e-06 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813414+0000", 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0489999999999999e-05 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813421+0000", 2024-09-16T05:22:56.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1010000000000003e-06 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816226+0000", 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002804647 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816349+0000", 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012330699999999999 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.281344+0000", 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.464994937 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284684+0000", 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033402309999999999 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287216+0000", 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002531298 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287319+0000", 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102865 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287398+0000", 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9116000000000006e-05 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6243 2.5 2:a1d8c6d9:::smithi20457400-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 188.128583311, 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47072968100000001, 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6243 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1800000000000005e-07 2024-09-16T05:22:56.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813377+0000", 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3699999999999999e-06 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813385+0000", 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5280000000000003e-06 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813392+0000", 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0600000000000002e-06 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813515+0000", 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012306 2024-09-16T05:22:56.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813521+0000", 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.203e-06 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816658+0000", 2024-09-16T05:22:56.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003137365 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816689+0000", 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0592000000000001e-05 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.283968+0000", 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46727931 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284030+0000", 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2316000000000004e-05 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284102+0000", 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2158999999999998e-05 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.075 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7270 2.1a 2:5827f14d:::smithi20457400-12:head [sparse-read 0~1919657,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 89.118452658999999, 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.003199682, 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7270 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.076 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823506+0000", 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1099999999999999e-06 2024-09-16T05:22:56.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823515+0000", 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6219999999999998e-06 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823531+0000", 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.632e-05 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823544+0000", 2024-09-16T05:22:56.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2508e-05 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832369+0000", 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088252939999999992 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832385+0000", 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5534999999999999e-05 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.822672+0000", 2024-09-16T05:22:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99028730499999995 2024-09-16T05:22:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.822833+0000", 2024-09-16T05:22:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016117400000000001 2024-09-16T05:22:56.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.826703+0000", 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038698140000000001 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7271 2.1a 2:5827f14d:::smithi20457400-12:head [sparse-read 0~1919657] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:22:56.081 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 89.118396249, 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0044004120000001, 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7271 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.082 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:22:56.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6699999999999999e-07 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823563+0000", 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.824e-06 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823569+0000", 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.319e-06 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823574+0000", 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0180000000000001e-06 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832415+0000", 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088402089999999999 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832423+0000", 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8089999999999999e-06 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827702+0000", 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.995278938 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827810+0000", 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107433 2024-09-16T05:22:56.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827960+0000", 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150395 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7272 2.1a 2:5827f14d:::smithi20457400-12:head [read 0~3025491,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 89.118375639999996, 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0059436770000001, 2024-09-16T05:22:56.087 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7272 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823581+0000", 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1800000000000002e-07 2024-09-16T05:22:56.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823600+0000", 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9296999999999998e-05 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823607+0000", 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.063e-06 2024-09-16T05:22:56.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823612+0000", 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0699999999999997e-06 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832476+0000", 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088636289999999996 2024-09-16T05:22:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832485+0000", 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5359999999999998e-06 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827992+0000", 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99550709199999998 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.828085+0000", 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3086999999999998e-05 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.829524+0000", 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001438585 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.093 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPush(2.29 746/745 [PushOp(2:9662c29c:::smithi20457400-9:120, version: 737'1148, data_included: [2490368~196608,3108095~707292], data_size: 903900, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9662c29c:::smithi20457400-9:120@737'1148, size: 3815387, copy_subset: [(0, 3815387)], clone_subset: {2:9662c29c:::smithi20457400-9:116: []}, snapset: 130=[]:{116: [116], 120: [119], 125: [125], 130: [12f, 12c, 12a, 126]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3815387, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2490368, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 79.952281001000003, 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86202506199999995, 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.094 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989676+0000", 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8999999999999997e-07 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.990550+0000", 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087471100000000002 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850633+0000", 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86008260599999997 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850658+0000", 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5537999999999998e-05 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850763+0000", 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104546 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850778+0000", 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4603999999999999e-05 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.851700+0000", 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00092256700000000001 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7554 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 768] snapc 13f=[13f,13d,13b,134,133,12c,12a,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 63.722174854999999, 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010270486000000001, 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.098 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7554 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219783+0000", 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1430000000000001e-06 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219795+0000", 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2174999999999999e-05 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219821+0000", 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5364e-05 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219831+0000", 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0069e-05 2024-09-16T05:22:56.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219911+0000", 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0131999999999998e-05 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.220032+0000", 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000121305 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.224076+0000", 2024-09-16T05:22:56.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040437390000000002 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.227596+0000", 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035194359999999999 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.229981+0000", 2024-09-16T05:22:56.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023851279999999998 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.230018+0000", 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7030999999999998e-05 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.230052+0000", 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3964000000000002e-05 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7717 2.f 2:f08ec412:::smithi20457400-44:head [copy-from ver 1121] snapc 149=[149,148,147,146,145,144,143,142,141,13b,133,12c,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e778)", 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.847807744999997, 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010866597, 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.105 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7717 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094150+0000", 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4899999999999999e-06 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094158+0000", 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9980000000000003e-06 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094174+0000", 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6291e-05 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094184+0000", 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6369999999999994e-06 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094245+0000", 2024-09-16T05:22:56.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1261999999999997e-05 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094313+0000", 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7793999999999995e-05 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9", 2024-09-16T05:22:56.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.100828+0000", 2024-09-16T05:22:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0065157000000000001 2024-09-16T05:22:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:22:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104619+0000", 2024-09-16T05:22:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037903139999999999 2024-09-16T05:22:56.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104944+0000", 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032508400000000001 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104979+0000", 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5567e-05 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.105015+0000", 2024-09-16T05:22:56.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5460000000000003e-05 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:1707 2.1a e799/764 ondisk, result = 0)", 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 27.080085638, 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69806231100000005, 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861871+0000", 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7800000000000001e-07 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861876+0000", 2024-09-16T05:22:56.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4889999999999997e-06 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559739+0000", 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69786377399999999 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559746+0000", 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3409999999999999e-06 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559833+0000", 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7454000000000007e-05 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559840+0000", 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559933+0000", 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2936000000000006e-05 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8288 2.25 2:a7ee7933:::smithi20457400-20:head [copy-from ver 750] snapc 15a=[15a] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.0238900750000002, 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01619603, 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.117 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8288 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918068+0000", 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4499999999999998e-06 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918095+0000", 2024-09-16T05:22:56.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6695000000000001e-05 2024-09-16T05:22:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918117+0000", 2024-09-16T05:22:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2120000000000002e-05 2024-09-16T05:22:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918127+0000", 2024-09-16T05:22:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0074e-05 2024-09-16T05:22:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918253+0000", 2024-09-16T05:22:56.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125339 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918260+0000", 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9530000000000002e-06 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.921653+0000", 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033933230000000002 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.921758+0000", 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104949 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.927615+0000", 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058567039999999999 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:22:56.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.933643+0000", 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006028233 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934133+0000", 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00049045800000000004 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934202+0000", 2024-09-16T05:22:56.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8817000000000002e-05 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934262+0000", 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9914999999999998e-05 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.125 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8289 2.25 2:a7ee7933:::smithi20457400-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.0237911909999999, 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012771173, 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8289 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2500000000000005e-07 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918187+0000", 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1101999999999999e-05 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918198+0000", 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1139e-05 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918203+0000", 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5589999999999996e-06 2024-09-16T05:22:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918361+0000", 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015787099999999999 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918367+0000", 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2389999999999999e-06 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.922539+0000", 2024-09-16T05:22:56.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041721240000000001 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.922594+0000", 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4969000000000002e-05 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930701+0000", 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081063300000000001 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:22:56.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930817+0000", 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116116 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930936+0000", 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011909900000000001 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:22:56.131 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:22:56.133 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:22:56.133 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:22:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:56.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:56.134 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:22:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:22:56.135 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:22:56.135 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:22:56.136 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:22:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:22:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:55 smithi204 ceph-mon[30933]: 2.2d deep-scrub starts 2024-09-16T05:22:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:55 smithi204 ceph-mon[30933]: 2.2d deep-scrub ok 2024-09-16T05:22:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:55 smithi204 ceph-mon[30933]: pgmap v1398: 127 pgs: 1 active+clean+snaptrim, 126 active+clean; 152 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.9 MiB/s wr, 12 op/s 2024-09-16T05:22:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:55 smithi204 ceph-mon[30933]: 2.5a scrub starts 2024-09-16T05:22:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:55 smithi204 ceph-mon[30933]: 2.5a scrub ok 2024-09-16T05:22:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:55 smithi204 ceph-mon[30933]: osdmap e825: 12 total, 12 up, 12 in 2024-09-16T05:22:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:55 smithi204 ceph-mon[30933]: osdmap e826: 12 total, 12 up, 12 in 2024-09-16T05:22:56.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:22:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:55 smithi196 ceph-mon[30802]: 2.2d deep-scrub starts 2024-09-16T05:22:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:55 smithi196 ceph-mon[30802]: 2.2d deep-scrub ok 2024-09-16T05:22:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:55 smithi196 ceph-mon[30802]: pgmap v1398: 127 pgs: 1 active+clean+snaptrim, 126 active+clean; 152 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.9 MiB/s wr, 12 op/s 2024-09-16T05:22:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:55 smithi196 ceph-mon[30802]: 2.5a scrub starts 2024-09-16T05:22:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:55 smithi196 ceph-mon[30802]: 2.5a scrub ok 2024-09-16T05:22:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:55 smithi196 ceph-mon[30802]: osdmap e825: 12 total, 12 up, 12 in 2024-09-16T05:22:56.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:55 smithi196 ceph-mon[30802]: osdmap e826: 12 total, 12 up, 12 in 2024-09-16T05:22:56.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:56.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:22:56.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:22:56.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:55 smithi032 ceph-mon[27388]: 2.2d deep-scrub starts 2024-09-16T05:22:56.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:55 smithi032 ceph-mon[27388]: 2.2d deep-scrub ok 2024-09-16T05:22:56.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:55 smithi032 ceph-mon[27388]: pgmap v1398: 127 pgs: 1 active+clean+snaptrim, 126 active+clean; 152 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.9 MiB/s wr, 12 op/s 2024-09-16T05:22:56.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:55 smithi032 ceph-mon[27388]: 2.5a scrub starts 2024-09-16T05:22:56.302 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:55 smithi032 ceph-mon[27388]: 2.5a scrub ok 2024-09-16T05:22:56.303 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:55 smithi032 ceph-mon[27388]: osdmap e825: 12 total, 12 up, 12 in 2024-09-16T05:22:56.303 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:55 smithi032 ceph-mon[27388]: osdmap e826: 12 total, 12 up, 12 in 2024-09-16T05:22:56.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:56.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:56.386 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:56.386 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:56.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:56.510 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:56.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:56.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:56.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:56.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:56.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:56.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:56.759 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:56.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:56.840 INFO:tasks.rados.rados.2.smithi204.stdout:3291: done (0 left) 2024-09-16T05:22:56.840 INFO:tasks.rados.rados.2.smithi204.stdout:3292: snap_remove snap 342 2024-09-16T05:22:56.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:56.889 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:56.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:56.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:22:56.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:57.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:57.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:22:57.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:22:57.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:22:57.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:22:57.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: 2.58 scrub starts 2024-09-16T05:22:57.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: 2.58 scrub ok 2024-09-16T05:22:57.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:22:57.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:22:57.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:56 smithi032 ceph-mon[27388]: osdmap e827: 12 total, 12 up, 12 in 2024-09-16T05:22:57.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:57.137 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:22:57.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:57.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:22:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:22:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:22:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:22:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: 2.58 scrub starts 2024-09-16T05:22:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: 2.58 scrub ok 2024-09-16T05:22:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:22:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:22:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:56 smithi204 ceph-mon[30933]: osdmap e827: 12 total, 12 up, 12 in 2024-09-16T05:22:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:22:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:22:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:22:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:22:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: 2.58 scrub starts 2024-09-16T05:22:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: 2.58 scrub ok 2024-09-16T05:22:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:22:57.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:22:57.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:56 smithi196 ceph-mon[30802]: osdmap e827: 12 total, 12 up, 12 in 2024-09-16T05:22:57.244 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:22:57.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:57.266 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:57.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:22:57.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:22:57.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:57.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:22:57.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:57.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:57.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:57.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:57.637 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:57.637 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:57.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:57.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:57.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:57.769 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:57.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:57.836 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:22:57.839 INFO:tasks.rados.rados.2.smithi204.stdout:3292: done (0 left) 2024-09-16T05:22:57.840 INFO:tasks.rados.rados.2.smithi204.stdout:3293: snap_remove snap 343 2024-09-16T05:22:57.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:57.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:57.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:57.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:57.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:22:58.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:58.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:58.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:22:58.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:58.146 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:58.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: pgmap v1401: 127 pgs: 1 active+clean+snaptrim, 126 active+clean; 120 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.8 MiB/s wr, 13 op/s 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: 2.3b scrub starts 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: 2.3b scrub ok 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/857785668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:58.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/857785668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:58.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:57 smithi032 ceph-mon[27388]: osdmap e828: 12 total, 12 up, 12 in 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: pgmap v1401: 127 pgs: 1 active+clean+snaptrim, 126 active+clean; 120 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.8 MiB/s wr, 13 op/s 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: 2.3b scrub starts 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: 2.3b scrub ok 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/857785668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:22:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/857785668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:58.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:57 smithi196 ceph-mon[30802]: osdmap e828: 12 total, 12 up, 12 in 2024-09-16T05:22:58.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: pgmap v1401: 127 pgs: 1 active+clean+snaptrim, 126 active+clean; 120 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.8 MiB/s wr, 13 op/s 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: 2.3b scrub starts 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: 2.3b scrub ok 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/857785668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:22:58.251 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:22:58.252 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/857785668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:22:58.252 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:57 smithi204 ceph-mon[30933]: osdmap e828: 12 total, 12 up, 12 in 2024-09-16T05:22:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:58.271 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:58.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:22:58.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:22:58.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:58.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:58.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:58.520 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:58.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:58.564 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:22:58.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:58.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:58.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:58.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:22:58.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:58.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:22:58.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:22:58.834 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:22:58.834 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:22:58.834 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-16T05:22:58.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:22:58.841 INFO:tasks.rados.rados.2.smithi204.stdout:3293: done (0 left) 2024-09-16T05:22:58.841 INFO:tasks.rados.rados.2.smithi204.stdout:3294: rollback oid 34 current snap is 345 2024-09-16T05:22:58.841 INFO:tasks.rados.rados.2.smithi204.stdout:3294: done (0 left) 2024-09-16T05:22:58.841 INFO:tasks.rados.rados.2.smithi204.stdout:3295: copy_from oid 31 from oid 14 current snap is 345 2024-09-16T05:22:58.841 INFO:tasks.rados.rados.2.smithi204.stdout:3296: read oid 46 snap -1 2024-09-16T05:22:58.841 INFO:tasks.rados.rados.2.smithi204.stdout:3296: expect deleted 2024-09-16T05:22:58.841 INFO:tasks.rados.rados.2.smithi204.stdout:3297: delete oid 16 current snap is 345 2024-09-16T05:22:58.850 INFO:tasks.rados.rados.2.smithi204.stdout:3296: done (2 left) 2024-09-16T05:22:58.850 INFO:tasks.rados.rados.2.smithi204.stdout:3297: done (1 left) 2024-09-16T05:22:58.850 INFO:tasks.rados.rados.2.smithi204.stdout:3298: write oid 17 current snap is 345 2024-09-16T05:22:58.850 INFO:tasks.rados.rados.2.smithi204.stdout:3298: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-09-16T05:22:58.856 INFO:tasks.rados.rados.2.smithi204.stdout:3298: writing smithi20457400-17 from 612085 to 1093983 tid 1 2024-09-16T05:22:58.864 INFO:tasks.rados.rados.2.smithi204.stdout:3298: writing smithi20457400-17 from 1788114 to 2463841 tid 2 2024-09-16T05:22:58.869 INFO:tasks.rados.rados.2.smithi204.stdout:3298: writing smithi20457400-17 from 3035086 to 3412085 tid 3 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3299: copy_from oid 10 from oid 29 current snap is 345 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3295: finishing copy_from racing read to smithi20457400-31 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3298: finishing write tid 1 to smithi20457400-17 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3298: finishing write tid 2 to smithi20457400-17 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3300: copy_from oid 50 from oid 44 current snap is 345 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3301: read oid 33 snap -1 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3301: expect deleted 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3302: read oid 4 snap -1 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3302: expect (ObjNum 1137 snap 341 seq_num 1137) 2024-09-16T05:22:58.870 INFO:tasks.rados.rados.2.smithi204.stdout:3303: snap_create 2024-09-16T05:22:58.872 INFO:tasks.rados.rados.2.smithi204.stdout:3298: finishing write tid 3 to smithi20457400-17 2024-09-16T05:22:58.873 INFO:tasks.rados.rados.2.smithi204.stdout:3298: finishing write tid 4 to smithi20457400-17 2024-09-16T05:22:58.873 INFO:tasks.rados.rados.2.smithi204.stdout:3298: finishing write tid 5 to smithi20457400-17 2024-09-16T05:22:58.873 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1131 (ObjNum 1155 snap 345 seq_num 1155) dirty exists 2024-09-16T05:22:58.873 INFO:tasks.rados.rados.2.smithi204.stdout:3298: left oid 17 (ObjNum 1155 snap 345 seq_num 1155) 2024-09-16T05:22:58.876 INFO:tasks.rados.rados.2.smithi204.stdout:3302: expect (ObjNum 1137 snap 341 seq_num 1137) 2024-09-16T05:22:58.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:22:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:58.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:22:59.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:22:59.028 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:59.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:22:59.047 INFO:tasks.rados.rados.2.smithi204.stdout:3300: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:22:59.048 INFO:tasks.rados.rados.2.smithi204.stdout:3300: finishing copy_from to smithi20457400-50 2024-09-16T05:22:59.048 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1145 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:22:59.048 INFO:tasks.rados.rados.2.smithi204.stdout:3295: finishing copy_from to smithi20457400-31 2024-09-16T05:22:59.048 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 1430 (ObjNum 1080 snap 329 seq_num 1080) dirty exists 2024-09-16T05:22:59.048 INFO:tasks.rados.rados.2.smithi204.stdout:3299: finishing copy_from racing read to smithi20457400-10 2024-09-16T05:22:59.048 INFO:tasks.rados.rados.2.smithi204.stdout:3299: finishing copy_from to smithi20457400-10 2024-09-16T05:22:59.048 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1174 (ObjNum 935 snap 277 seq_num 935) dirty exists 2024-09-16T05:22:59.067 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:22:59.067 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:22:59.112 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:59.113 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:22:59.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:22:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:59.153 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:59.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:22:59.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:58 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:22:59.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:58 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:22:59.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:58 smithi032 ceph-mon[27388]: 2.7 deep-scrub starts 2024-09-16T05:22:59.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:58 smithi032 ceph-mon[27388]: 2.7 deep-scrub ok 2024-09-16T05:22:59.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:58 smithi032 ceph-mon[27388]: osdmap e829: 12 total, 12 up, 12 in 2024-09-16T05:22:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:58 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:22:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:58 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:22:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:58 smithi196 ceph-mon[30802]: 2.7 deep-scrub starts 2024-09-16T05:22:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:58 smithi196 ceph-mon[30802]: 2.7 deep-scrub ok 2024-09-16T05:22:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:58 smithi196 ceph-mon[30802]: osdmap e829: 12 total, 12 up, 12 in 2024-09-16T05:22:59.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:59.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:59.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:59.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:22:59.384 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:58 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:22:59.384 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:58 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:22:59.384 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:58 smithi204 ceph-mon[30933]: 2.7 deep-scrub starts 2024-09-16T05:22:59.384 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:58 smithi204 ceph-mon[30933]: 2.7 deep-scrub ok 2024-09-16T05:22:59.384 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:58 smithi204 ceph-mon[30933]: osdmap e829: 12 total, 12 up, 12 in 2024-09-16T05:22:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:59.401 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:22:59.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:22:59.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:59.530 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:59.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:59.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:59.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:59.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:59.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:22:59.760 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:22:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:59.784 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:22:59.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:22:59.841 INFO:tasks.rados.rados.2.smithi204.stdout:3295: done (6 left) 2024-09-16T05:22:59.842 INFO:tasks.rados.rados.2.smithi204.stdout:3298: done (5 left) 2024-09-16T05:22:59.842 INFO:tasks.rados.rados.2.smithi204.stdout:3299: done (4 left) 2024-09-16T05:22:59.842 INFO:tasks.rados.rados.2.smithi204.stdout:3300: done (3 left) 2024-09-16T05:22:59.842 INFO:tasks.rados.rados.2.smithi204.stdout:3301: done (2 left) 2024-09-16T05:22:59.842 INFO:tasks.rados.rados.2.smithi204.stdout:3302: done (1 left) 2024-09-16T05:22:59.842 INFO:tasks.rados.rados.2.smithi204.stdout:3303: done (0 left) 2024-09-16T05:22:59.842 INFO:tasks.rados.rados.2.smithi204.stdout:3304: snap_create 2024-09-16T05:22:59.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:22:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:22:59.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:22:59.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:22:59.934 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:00.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:00.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:00.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:00.100 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:23:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:23:00.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:00.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:00.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:00.199 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:59 smithi204 ceph-mon[30933]: pgmap v1404: 127 pgs: 127 active+clean; 100 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 19 op/s 2024-09-16T05:23:00.199 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:59 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:23:00.199 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:59 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:23:00.199 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:22:59 smithi204 ceph-mon[30933]: osdmap e830: 12 total, 12 up, 12 in 2024-09-16T05:23:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:59 smithi196 ceph-mon[30802]: pgmap v1404: 127 pgs: 127 active+clean; 100 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 19 op/s 2024-09-16T05:23:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:59 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:23:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:59 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:23:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:22:59 smithi196 ceph-mon[30802]: osdmap e830: 12 total, 12 up, 12 in 2024-09-16T05:23:00.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:59 smithi032 ceph-mon[27388]: pgmap v1404: 127 pgs: 127 active+clean; 100 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 19 op/s 2024-09-16T05:23:00.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:59 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:23:00.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:59 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:23:00.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:22:59 smithi032 ceph-mon[27388]: osdmap e830: 12 total, 12 up, 12 in 2024-09-16T05:23:00.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:00.286 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:00.286 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:00.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:00.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:00.308 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:23:00.308 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:23:00.329 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:00.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:00.349 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:00.349 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:00.371 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:00.371 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:00.393 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:00.393 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:00.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:00.415 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:00.415 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:00.420 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:00.420 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:00.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:00.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:00.436 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:00.436 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:00.457 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:00.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:00.500 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:00.521 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:00.522 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:00.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:00.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:00.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:00.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:23:00.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:00.669 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:00.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:00.772 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:00.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:00.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:00.837 INFO:tasks.rados.rados.2.smithi204.stdout:3304: done (0 left) 2024-09-16T05:23:00.837 INFO:tasks.rados.rados.2.smithi204.stdout:3305: write (excl) oid 2 current snap is 347 2024-09-16T05:23:00.837 INFO:tasks.rados.rados.2.smithi204.stdout:3305: seq_num 1156 ranges {628892=673500,1962949=535738,3178862=250030} 2024-09-16T05:23:00.844 INFO:tasks.rados.rados.2.smithi204.stdout:3305: writing smithi20457400-2 from 628892 to 1302392 tid 1 2024-09-16T05:23:00.851 INFO:tasks.rados.rados.2.smithi204.stdout:3305: writing smithi20457400-2 from 1962949 to 2498687 tid 2 2024-09-16T05:23:00.856 INFO:tasks.rados.rados.2.smithi204.stdout:3305: writing smithi20457400-2 from 3178862 to 3428892 tid 3 2024-09-16T05:23:00.857 INFO:tasks.rados.rados.2.smithi204.stdout:3306: read oid 8 snap 346 2024-09-16T05:23:00.857 INFO:tasks.rados.rados.2.smithi204.stdout:3306: expect deleted 2024-09-16T05:23:00.857 INFO:tasks.rados.rados.2.smithi204.stdout:3307: delete oid 4 current snap is 347 2024-09-16T05:23:00.861 INFO:tasks.rados.rados.2.smithi204.stdout:3305: finishing write tid 1 to smithi20457400-2 2024-09-16T05:23:00.861 INFO:tasks.rados.rados.2.smithi204.stdout:3305: finishing write tid 2 to smithi20457400-2 2024-09-16T05:23:00.862 INFO:tasks.rados.rados.2.smithi204.stdout:3305: finishing write tid 3 to smithi20457400-2 2024-09-16T05:23:00.862 INFO:tasks.rados.rados.2.smithi204.stdout:3305: finishing write tid 4 to smithi20457400-2 2024-09-16T05:23:00.863 INFO:tasks.rados.rados.2.smithi204.stdout:3305: finishing write tid 5 to smithi20457400-2 2024-09-16T05:23:00.863 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 544 (ObjNum 1156 snap 347 seq_num 1156) dirty exists 2024-09-16T05:23:00.863 INFO:tasks.rados.rados.2.smithi204.stdout:3305: left oid 2 (ObjNum 1156 snap 347 seq_num 1156) 2024-09-16T05:23:00.863 INFO:tasks.rados.rados.2.smithi204.stdout:3305: done (2 left) 2024-09-16T05:23:00.863 INFO:tasks.rados.rados.2.smithi204.stdout:3306: done (1 left) 2024-09-16T05:23:00.863 INFO:tasks.rados.rados.2.smithi204.stdout:3307: done (0 left) 2024-09-16T05:23:00.863 INFO:tasks.rados.rados.2.smithi204.stdout:3308: snap_create 2024-09-16T05:23:00.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:00.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:00.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:01.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:01.040 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:01.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:01.051 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:00 smithi204 ceph-mon[30933]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:23:01.051 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3136404402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:01.051 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:00 smithi204 ceph-mon[30933]: osdmap e831: 12 total, 12 up, 12 in 2024-09-16T05:23:01.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:23:01.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:01.167 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:01.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:00 smithi032 ceph-mon[27388]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:23:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3136404402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:00 smithi032 ceph-mon[27388]: osdmap e831: 12 total, 12 up, 12 in 2024-09-16T05:23:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:00 smithi196 ceph-mon[30802]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:23:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3136404402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:00 smithi196 ceph-mon[30802]: osdmap e831: 12 total, 12 up, 12 in 2024-09-16T05:23:01.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:01.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:01.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:01.359 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:01.394 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:01.416 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:01.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:01.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:01.539 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:01.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:01.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:01.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:01.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:01.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:01.795 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:01.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:01.839 INFO:tasks.rados.rados.2.smithi204.stdout:3308: done (0 left) 2024-09-16T05:23:01.839 INFO:tasks.rados.rados.2.smithi204.stdout:3309: read oid 10 snap -1 2024-09-16T05:23:01.839 INFO:tasks.rados.rados.2.smithi204.stdout:3309: expect (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:23:01.839 INFO:tasks.rados.rados.2.smithi204.stdout:3310: snap_remove snap 346 2024-09-16T05:23:01.847 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:23:01.847 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:23:01.847 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:23:01.847 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:23:01.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:01.899 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1406,"stamp":"2024-09-16T05:22:59.839179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102242081,"num_objects":38,"num_object_clones":3,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":1,"num_read":3616,"num_read_kb":3599840,"num_write":2545,"num_write_kb":1050826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37518,"ondisk_log_size":37518,"up":254,"acting":254,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":254,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":815216,"kb_used_data":251896,"kb_used_omap":27,"kb_used_meta":562468,"kb_avail":1124126608,"statfs":{"total":1151940427776,"available":1151105646592,"internally_reserved":0,"allocated":257941504,"data_stored":225664233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":575967399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-37263232,"num_objects":-18,"num_object_clones":-13,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-18,"num_whiteouts":-4,"num_read":46,"num_read_kb":50961,"num_write":26,"num_write_kb":6076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005609"},"pg_stats":[{"pgid":"2.5f","version":"284'221","reported_seq":3834,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:55.836947+0000","last_change":"2024-09-16T05:22:52.836198+0000","last_active":"2024-09-16T05:22:55.836947+0000","last_peered":"2024-09-16T05:22:55.836947+0000","last_clean":"2024-09-16T05:22:55.836947+0000","last_became_active":"2024-09-16T05:21:54.939189+0000","last_became_peered":"2024-09-16T05:21:54.939189+0000","last_unstale":"2024-09-16T05:22:55.836947+0000","last_undegraded":"2024-09-16T05:22:55.836947+0000","last_fullsized":"2024-09-16T05:22:55.836947+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:21:42.460245+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:21:42.460245+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:57.893015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048676200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.5e","version":"823'562","reported_seq":4308,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935692+0000","last_change":"2024-09-16T05:22:52.847417+0000","last_active":"2024-09-16T05:22:54.935692+0000","last_peered":"2024-09-16T05:22:54.935692+0000","last_clean":"2024-09-16T05:22:54.935692+0000","last_became_active":"2024-09-16T05:21:54.937455+0000","last_became_peered":"2024-09-16T05:21:54.937455+0000","last_unstale":"2024-09-16T05:22:54.935692+0000","last_undegraded":"2024-09-16T05:22:54.935692+0000","last_fullsized":"2024-09-16T05:22:54.935692+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"743'518","last_scrub_stamp":"2024-09-16T05:21:40.979621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:40.979621+0000","objects_scrubbed":3,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":2,"snaptrim_duration":0.0086934649999999992,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":13007,"num_write":7,"num_write_kb":989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4819607,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.5d","version":"823'1442","reported_seq":5291,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957581+0000","last_change":"2024-09-16T05:22:52.928975+0000","last_active":"2024-09-16T05:22:54.957581+0000","last_peered":"2024-09-16T05:22:54.957581+0000","last_clean":"2024-09-16T05:22:54.957581+0000","last_became_active":"2024-09-16T05:21:32.830157+0000","last_became_peered":"2024-09-16T05:21:32.830157+0000","last_unstale":"2024-09-16T05:22:54.957581+0000","last_undegraded":"2024-09-16T05:22:54.957581+0000","last_fullsized":"2024-09-16T05:22:54.957581+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'1436","last_scrub_stamp":"2024-09-16T05:22:51.640147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.640147+0000","objects_scrubbed":6,"log_size":1442,"log_dups_size":0,"ondisk_log_size":1442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:10.896447+0000","scrub_duration":3,"objects_trimmed":4,"snaptrim_duration":0.0091076170000000001,"stat_sum":{"num_bytes":5630807,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":62111,"num_write":24,"num_write_kb":13635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.5c","version":"823'484","reported_seq":3716,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.925230+0000","last_change":"2024-09-16T05:22:52.862254+0000","last_active":"2024-09-16T05:22:54.925230+0000","last_peered":"2024-09-16T05:22:54.925230+0000","last_clean":"2024-09-16T05:22:54.925230+0000","last_became_active":"2024-09-16T05:21:32.828660+0000","last_became_peered":"2024-09-16T05:21:32.828660+0000","last_unstale":"2024-09-16T05:22:54.925230+0000","last_undegraded":"2024-09-16T05:22:54.925230+0000","last_fullsized":"2024-09-16T05:22:54.925230+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'482","last_scrub_stamp":"2024-09-16T05:22:52.859116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:52.859116+0000","objects_scrubbed":2,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:00.454838+0000","scrub_duration":39,"objects_trimmed":2,"snaptrim_duration":0.026194578,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7420,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.5b","version":"823'662","reported_seq":4269,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957375+0000","last_change":"2024-09-16T05:22:52.922100+0000","last_active":"2024-09-16T05:22:54.957375+0000","last_peered":"2024-09-16T05:22:54.957375+0000","last_clean":"2024-09-16T05:22:54.957375+0000","last_became_active":"2024-09-16T05:21:32.829681+0000","last_became_peered":"2024-09-16T05:21:32.829681+0000","last_unstale":"2024-09-16T05:22:54.957375+0000","last_undegraded":"2024-09-16T05:22:54.957375+0000","last_fullsized":"2024-09-16T05:22:54.957375+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'660","last_scrub_stamp":"2024-09-16T05:22:52.654103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:52.654103+0000","objects_scrubbed":2,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:55.142123+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.002237228,"stat_sum":{"num_bytes":3479247,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":32170,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.5a","version":"823'1224","reported_seq":4792,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.845901+0000","last_change":"2024-09-16T05:22:57.847860+0000","last_active":"2024-09-16T05:22:57.845901+0000","last_peered":"2024-09-16T05:22:57.845901+0000","last_clean":"2024-09-16T05:22:57.845901+0000","last_became_active":"2024-09-16T05:21:32.826884+0000","last_became_peered":"2024-09-16T05:21:32.826884+0000","last_unstale":"2024-09-16T05:22:57.845901+0000","last_undegraded":"2024-09-16T05:22:57.845901+0000","last_fullsized":"2024-09-16T05:22:57.845901+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"823'1224","last_scrub_stamp":"2024-09-16T05:22:54.387969+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.387969+0000","objects_scrubbed":1,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:17.929065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0054399629999999999,"stat_sum":{"num_bytes":2638963,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":21681,"num_write":24,"num_write_kb":6390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.59","version":"182'60","reported_seq":484,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.012456+0000","last_change":"2024-09-16T05:22:57.013020+0000","last_active":"2024-09-16T05:22:57.012456+0000","last_peered":"2024-09-16T05:22:57.012456+0000","last_clean":"2024-09-16T05:22:57.012456+0000","last_became_active":"2024-09-16T05:21:54.931981+0000","last_became_peered":"2024-09-16T05:21:54.931981+0000","last_unstale":"2024-09-16T05:22:57.012456+0000","last_undegraded":"2024-09-16T05:22:57.012456+0000","last_fullsized":"2024-09-16T05:22:57.012456+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:35.068053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.068053+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:46.178909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085211900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"2.58","version":"181'57","reported_seq":3609,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957740+0000","last_change":"2024-09-16T05:22:52.849151+0000","last_active":"2024-09-16T05:22:54.957740+0000","last_peered":"2024-09-16T05:22:54.957740+0000","last_clean":"2024-09-16T05:22:54.957740+0000","last_became_active":"2024-09-16T05:21:29.826747+0000","last_became_peered":"2024-09-16T05:21:29.826747+0000","last_unstale":"2024-09-16T05:22:54.957740+0000","last_undegraded":"2024-09-16T05:22:54.957740+0000","last_fullsized":"2024-09-16T05:22:54.957740+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:21:30.068545+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:21:30.068545+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0064446219999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.57","version":"0'0","reported_seq":1153,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935903+0000","last_change":"2024-09-16T05:22:52.840165+0000","last_active":"2024-09-16T05:22:54.935903+0000","last_peered":"2024-09-16T05:22:54.935903+0000","last_clean":"2024-09-16T05:22:54.935903+0000","last_became_active":"2024-09-16T05:21:29.827377+0000","last_became_peered":"2024-09-16T05:21:29.827377+0000","last_unstale":"2024-09-16T05:22:54.935903+0000","last_undegraded":"2024-09-16T05:22:54.935903+0000","last_fullsized":"2024-09-16T05:22:54.935903+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:48.915248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.915248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:56.674699+0000","scrub_duration":557,"objects_trimmed":0,"snaptrim_duration":0.0019775420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.56","version":"823'386","reported_seq":3631,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.925249+0000","last_change":"2024-09-16T05:22:52.839923+0000","last_active":"2024-09-16T05:22:54.925249+0000","last_peered":"2024-09-16T05:22:54.925249+0000","last_clean":"2024-09-16T05:22:54.925249+0000","last_became_active":"2024-09-16T05:21:32.829699+0000","last_became_peered":"2024-09-16T05:21:32.829699+0000","last_unstale":"2024-09-16T05:22:54.925249+0000","last_undegraded":"2024-09-16T05:22:54.925249+0000","last_fullsized":"2024-09-16T05:22:54.925249+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"738'336","last_scrub_stamp":"2024-09-16T05:21:42.738509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:42.738509+0000","objects_scrubbed":2,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:08.066467+0000","scrub_duration":828,"objects_trimmed":2,"snaptrim_duration":0.0028576930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":18484,"num_write":13,"num_write_kb":3476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.55","version":"433'507","reported_seq":2626,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.842874+0000","last_change":"2024-09-16T05:22:57.845298+0000","last_active":"2024-09-16T05:22:57.842874+0000","last_peered":"2024-09-16T05:22:57.842874+0000","last_clean":"2024-09-16T05:22:57.842874+0000","last_became_active":"2024-09-16T05:21:24.815957+0000","last_became_peered":"2024-09-16T05:21:24.815957+0000","last_unstale":"2024-09-16T05:22:57.842874+0000","last_undegraded":"2024-09-16T05:22:57.842874+0000","last_fullsized":"2024-09-16T05:22:57.842874+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:22:53.432495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:53.432495+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:58.558381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0042283190000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.54","version":"823'444","reported_seq":3696,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.842249+0000","last_change":"2024-09-16T05:22:57.844077+0000","last_active":"2024-09-16T05:22:57.842249+0000","last_peered":"2024-09-16T05:22:57.842249+0000","last_clean":"2024-09-16T05:22:57.842249+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:22:57.842249+0000","last_undegraded":"2024-09-16T05:22:57.842249+0000","last_fullsized":"2024-09-16T05:22:57.842249+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"765'422","last_scrub_stamp":"2024-09-16T05:22:01.692016+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:22:01.692016+0000","objects_scrubbed":1,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:17.515300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0044623099999999997,"stat_sum":{"num_bytes":3714545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":72,"num_read_kb":63622,"num_write":52,"num_write_kb":22446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":1974,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:55.837140+0000","last_change":"2024-09-16T05:22:52.836269+0000","last_active":"2024-09-16T05:22:55.837140+0000","last_peered":"2024-09-16T05:22:55.837140+0000","last_clean":"2024-09-16T05:22:55.837140+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:22:55.837140+0000","last_undegraded":"2024-09-16T05:22:55.837140+0000","last_fullsized":"2024-09-16T05:22:55.837140+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:22:00.537384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.537384+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:09.291871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000383166,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":1968,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.925116+0000","last_change":"2024-09-16T05:22:52.836570+0000","last_active":"2024-09-16T05:22:54.925116+0000","last_peered":"2024-09-16T05:22:54.925116+0000","last_clean":"2024-09-16T05:22:54.925116+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:22:54.925116+0000","last_undegraded":"2024-09-16T05:22:54.925116+0000","last_fullsized":"2024-09-16T05:22:54.925116+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:21:44.952108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:44.952108+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089626599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.51","version":"272'115","reported_seq":2039,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935511+0000","last_change":"2024-09-16T05:22:52.839455+0000","last_active":"2024-09-16T05:22:54.935511+0000","last_peered":"2024-09-16T05:22:54.935511+0000","last_clean":"2024-09-16T05:22:54.935511+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:22:54.935511+0000","last_undegraded":"2024-09-16T05:22:54.935511+0000","last_fullsized":"2024-09-16T05:22:54.935511+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_deep_scrub":"272'115","last_deep_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_clean_scrub_stamp":"2024-09-16T05:22:11.104571+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:32.266306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001503788,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.50","version":"182'16","reported_seq":1979,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.840810+0000","last_change":"2024-09-16T05:22:57.843757+0000","last_active":"2024-09-16T05:22:57.840810+0000","last_peered":"2024-09-16T05:22:57.840810+0000","last_clean":"2024-09-16T05:22:57.840810+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:22:57.840810+0000","last_undegraded":"2024-09-16T05:22:57.840810+0000","last_fullsized":"2024-09-16T05:22:57.840810+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:22:25.344602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.344602+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:54.475909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003151549,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2503,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957713+0000","last_change":"2024-09-16T05:22:52.927448+0000","last_active":"2024-09-16T05:22:54.957713+0000","last_peered":"2024-09-16T05:22:54.957713+0000","last_clean":"2024-09-16T05:22:54.957713+0000","last_became_active":"2024-09-16T05:21:10.817392+0000","last_became_peered":"2024-09-16T05:21:10.817392+0000","last_unstale":"2024-09-16T05:22:54.957713+0000","last_undegraded":"2024-09-16T05:22:54.957713+0000","last_fullsized":"2024-09-16T05:22:54.957713+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:22:33.493608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:33.493608+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:59.491434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.007272784,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.4e","version":"272'232","reported_seq":3739,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.930902+0000","last_change":"2024-09-16T05:22:52.847846+0000","last_active":"2024-09-16T05:22:54.930902+0000","last_peered":"2024-09-16T05:22:54.930902+0000","last_clean":"2024-09-16T05:22:54.930902+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:22:54.930902+0000","last_undegraded":"2024-09-16T05:22:54.930902+0000","last_fullsized":"2024-09-16T05:22:54.930902+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:22:49.748809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.748809+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:58.092994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0048675539999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.4d","version":"828'730","reported_seq":4416,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.846808+0000","last_change":"2024-09-16T05:22:57.847180+0000","last_active":"2024-09-16T05:22:57.846808+0000","last_peered":"2024-09-16T05:22:57.846808+0000","last_clean":"2024-09-16T05:22:57.846808+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:22:57.846808+0000","last_undegraded":"2024-09-16T05:22:57.846808+0000","last_fullsized":"2024-09-16T05:22:57.846808+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"807'708","last_scrub_stamp":"2024-09-16T05:22:37.651060+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.651060+0000","objects_scrubbed":4,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:05.358526+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0035753,"stat_sum":{"num_bytes":2790226,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":38970,"num_write":88,"num_write_kb":29604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.4c","version":"823'428","reported_seq":4044,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935684+0000","last_change":"2024-09-16T05:22:52.845867+0000","last_active":"2024-09-16T05:22:54.935684+0000","last_peered":"2024-09-16T05:22:54.935684+0000","last_clean":"2024-09-16T05:22:54.935684+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:22:54.935684+0000","last_undegraded":"2024-09-16T05:22:54.935684+0000","last_fullsized":"2024-09-16T05:22:54.935684+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"773'408","last_scrub_stamp":"2024-09-16T05:22:03.012954+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:22:03.012954+0000","objects_scrubbed":3,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:16.822531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0073481049999999997,"stat_sum":{"num_bytes":3328050,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":98,"num_read_kb":121159,"num_write":45,"num_write_kb":14823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.4b","version":"244'177","reported_seq":3797,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.844180+0000","last_change":"2024-09-16T05:22:57.844744+0000","last_active":"2024-09-16T05:22:57.844180+0000","last_peered":"2024-09-16T05:22:57.844180+0000","last_clean":"2024-09-16T05:22:57.844180+0000","last_became_active":"2024-09-16T05:22:22.832492+0000","last_became_peered":"2024-09-16T05:22:22.832492+0000","last_unstale":"2024-09-16T05:22:57.844180+0000","last_undegraded":"2024-09-16T05:22:57.844180+0000","last_fullsized":"2024-09-16T05:22:57.844180+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:22:10.688513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.688513+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:36.639446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.004599639,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.4a","version":"823'663","reported_seq":4242,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935570+0000","last_change":"2024-09-16T05:22:52.843666+0000","last_active":"2024-09-16T05:22:54.935570+0000","last_peered":"2024-09-16T05:22:54.935570+0000","last_clean":"2024-09-16T05:22:54.935570+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:22:54.935570+0000","last_undegraded":"2024-09-16T05:22:54.935570+0000","last_fullsized":"2024-09-16T05:22:54.935570+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'661","last_scrub_stamp":"2024-09-16T05:22:52.300466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:52.300466+0000","objects_scrubbed":2,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:53.681618+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0048882229999999997,"stat_sum":{"num_bytes":2941489,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":55014,"num_write":59,"num_write_kb":25438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.49","version":"245'220","reported_seq":3808,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.930742+0000","last_change":"2024-09-16T05:22:52.843704+0000","last_active":"2024-09-16T05:22:54.930742+0000","last_peered":"2024-09-16T05:22:54.930742+0000","last_clean":"2024-09-16T05:22:54.930742+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:22:54.930742+0000","last_undegraded":"2024-09-16T05:22:54.930742+0000","last_fullsized":"2024-09-16T05:22:54.930742+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:21:46.823721+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:21:46.823721+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:06.983016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001645683,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.48","version":"823'428","reported_seq":3999,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957756+0000","last_change":"2024-09-16T05:22:52.848437+0000","last_active":"2024-09-16T05:22:54.957756+0000","last_peered":"2024-09-16T05:22:54.957756+0000","last_clean":"2024-09-16T05:22:54.957756+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:22:54.957756+0000","last_undegraded":"2024-09-16T05:22:54.957756+0000","last_fullsized":"2024-09-16T05:22:54.957756+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"776'403","last_scrub_stamp":"2024-09-16T05:22:06.889233+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.889233+0000","objects_scrubbed":6,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:30.655347+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0058703820000000004,"stat_sum":{"num_bytes":2487700,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":75761,"num_write":67,"num_write_kb":32708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.47","version":"0'0","reported_seq":2594,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.925350+0000","last_change":"2024-09-16T05:22:52.836152+0000","last_active":"2024-09-16T05:22:54.925350+0000","last_peered":"2024-09-16T05:22:54.925350+0000","last_clean":"2024-09-16T05:22:54.925350+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:22:54.925350+0000","last_undegraded":"2024-09-16T05:22:54.925350+0000","last_fullsized":"2024-09-16T05:22:54.925350+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.838189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:40.796137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00059727399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.46","version":"116'9","reported_seq":837,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.845049+0000","last_change":"2024-09-16T05:22:57.845533+0000","last_active":"2024-09-16T05:22:57.845049+0000","last_peered":"2024-09-16T05:22:57.845049+0000","last_clean":"2024-09-16T05:22:57.845049+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:22:57.845049+0000","last_undegraded":"2024-09-16T05:22:57.845049+0000","last_fullsized":"2024-09-16T05:22:57.845049+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:32.588419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.588419+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:55.817232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001347654,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":3098,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957810+0000","last_change":"2024-09-16T05:22:52.927089+0000","last_active":"2024-09-16T05:22:54.957810+0000","last_peered":"2024-09-16T05:22:54.957810+0000","last_clean":"2024-09-16T05:22:54.957810+0000","last_became_active":"2024-09-16T05:21:10.818112+0000","last_became_peered":"2024-09-16T05:21:10.818112+0000","last_unstale":"2024-09-16T05:22:54.957810+0000","last_undegraded":"2024-09-16T05:22:54.957810+0000","last_fullsized":"2024-09-16T05:22:54.957810+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:21:49.394285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:49.394285+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:06.023108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.006947951,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":584,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957575+0000","last_change":"2024-09-16T05:22:52.928307+0000","last_active":"2024-09-16T05:22:54.957575+0000","last_peered":"2024-09-16T05:22:54.957575+0000","last_clean":"2024-09-16T05:22:54.957575+0000","last_became_active":"2024-09-16T05:21:07.820306+0000","last_became_peered":"2024-09-16T05:21:07.820306+0000","last_unstale":"2024-09-16T05:22:54.957575+0000","last_undegraded":"2024-09-16T05:22:54.957575+0000","last_fullsized":"2024-09-16T05:22:54.957575+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:22:10.381270+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.381270+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:26.665482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0080220559999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":2580,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.135025+0000","last_change":"2024-09-16T05:22:52.848022+0000","last_active":"2024-09-16T05:22:56.135025+0000","last_peered":"2024-09-16T05:22:56.135025+0000","last_clean":"2024-09-16T05:22:56.135025+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:22:56.135025+0000","last_undegraded":"2024-09-16T05:22:56.135025+0000","last_fullsized":"2024-09-16T05:22:56.135025+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:22:03.589783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:03.589783+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:11.354924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040708660000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":2583,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.843907+0000","last_change":"2024-09-16T05:22:58.845048+0000","last_active":"2024-09-16T05:22:58.843907+0000","last_peered":"2024-09-16T05:22:58.843907+0000","last_clean":"2024-09-16T05:22:58.843907+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:22:58.843907+0000","last_undegraded":"2024-09-16T05:22:58.843907+0000","last_fullsized":"2024-09-16T05:22:58.843907+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:22:25.741168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.741168+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:55.368668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0055297870000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"2.41","version":"178'50","reported_seq":1440,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.925049+0000","last_change":"2024-09-16T05:22:52.835773+0000","last_active":"2024-09-16T05:22:54.925049+0000","last_peered":"2024-09-16T05:22:54.925049+0000","last_clean":"2024-09-16T05:22:54.925049+0000","last_became_active":"2024-09-16T05:21:54.930175+0000","last_became_peered":"2024-09-16T05:21:54.930175+0000","last_unstale":"2024-09-16T05:22:54.925049+0000","last_undegraded":"2024-09-16T05:22:54.925049+0000","last_fullsized":"2024-09-16T05:22:54.925049+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:22:28.678308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.678308+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:33.671732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047470800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.40","version":"205'33","reported_seq":1127,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.930664+0000","last_change":"2024-09-16T05:22:52.843197+0000","last_active":"2024-09-16T05:22:54.930664+0000","last_peered":"2024-09-16T05:22:54.930664+0000","last_clean":"2024-09-16T05:22:54.930664+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:22:54.930664+0000","last_undegraded":"2024-09-16T05:22:54.930664+0000","last_fullsized":"2024-09-16T05:22:54.930664+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:22:35.757429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.757429+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:03.600057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001249354,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.3f","version":"823'551","reported_seq":4273,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.136685+0000","last_change":"2024-09-16T05:22:52.851496+0000","last_active":"2024-09-16T05:22:56.136685+0000","last_peered":"2024-09-16T05:22:56.136685+0000","last_clean":"2024-09-16T05:22:56.136685+0000","last_became_active":"2024-09-16T05:21:54.939233+0000","last_became_peered":"2024-09-16T05:21:54.939233+0000","last_unstale":"2024-09-16T05:22:56.136685+0000","last_undegraded":"2024-09-16T05:22:56.136685+0000","last_fullsized":"2024-09-16T05:22:56.136685+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"818'545","last_scrub_stamp":"2024-09-16T05:22:48.213000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.213000+0000","objects_scrubbed":4,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:18.092444+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0071027060000000003,"stat_sum":{"num_bytes":3143173,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":116,"num_read_kb":104032,"num_write":71,"num_write_kb":25787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":11726959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.3e","version":"820'580","reported_seq":4167,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.013606+0000","last_change":"2024-09-16T05:22:57.013871+0000","last_active":"2024-09-16T05:22:57.013606+0000","last_peered":"2024-09-16T05:22:57.013606+0000","last_clean":"2024-09-16T05:22:57.013606+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:22:57.013606+0000","last_undegraded":"2024-09-16T05:22:57.013606+0000","last_fullsized":"2024-09-16T05:22:57.013606+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"820'580","last_scrub_stamp":"2024-09-16T05:22:51.944900+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.944900+0000","objects_scrubbed":1,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:58.485604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011461430000000001,"stat_sum":{"num_bytes":3109559,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":57,"num_read_kb":47318,"num_write":67,"num_write_kb":35144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2044,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957891+0000","last_change":"2024-09-16T05:22:52.928747+0000","last_active":"2024-09-16T05:22:54.957891+0000","last_peered":"2024-09-16T05:22:54.957891+0000","last_clean":"2024-09-16T05:22:54.957891+0000","last_became_active":"2024-09-16T05:21:10.817387+0000","last_became_peered":"2024-09-16T05:21:10.817387+0000","last_unstale":"2024-09-16T05:22:54.957891+0000","last_undegraded":"2024-09-16T05:22:54.957891+0000","last_fullsized":"2024-09-16T05:22:54.957891+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.497749+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:48.937821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0083076139999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.3c","version":"819'826","reported_seq":4673,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.135379+0000","last_change":"2024-09-16T05:22:52.844697+0000","last_active":"2024-09-16T05:22:56.135379+0000","last_peered":"2024-09-16T05:22:56.135379+0000","last_clean":"2024-09-16T05:22:56.135379+0000","last_became_active":"2024-09-16T05:21:07.828778+0000","last_became_peered":"2024-09-16T05:21:07.828778+0000","last_unstale":"2024-09-16T05:22:56.135379+0000","last_undegraded":"2024-09-16T05:22:56.135379+0000","last_fullsized":"2024-09-16T05:22:56.135379+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'807","last_scrub_stamp":"2024-09-16T05:22:31.305689+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:22:31.305689+0000","objects_scrubbed":4,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:45.238989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001022759,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":194,"num_read_kb":258104,"num_write":95,"num_write_kb":34389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":14713263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.3b","version":"823'516","reported_seq":4118,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935182+0000","last_change":"2024-09-16T05:22:52.847393+0000","last_active":"2024-09-16T05:22:54.935182+0000","last_peered":"2024-09-16T05:22:54.935182+0000","last_clean":"2024-09-16T05:22:54.935182+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:22:54.935182+0000","last_undegraded":"2024-09-16T05:22:54.935182+0000","last_fullsized":"2024-09-16T05:22:54.935182+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"737'454","last_scrub_stamp":"2024-09-16T05:21:28.918225+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:21:28.918225+0000","objects_scrubbed":2,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0052813779999999998,"stat_sum":{"num_bytes":3109493,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":102,"num_read_kb":103894,"num_write":69,"num_write_kb":29927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.3a","version":"823'682","reported_seq":4240,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957817+0000","last_change":"2024-09-16T05:22:52.846875+0000","last_active":"2024-09-16T05:22:54.957817+0000","last_peered":"2024-09-16T05:22:54.957817+0000","last_clean":"2024-09-16T05:22:54.957817+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:22:54.957817+0000","last_undegraded":"2024-09-16T05:22:54.957817+0000","last_fullsized":"2024-09-16T05:22:54.957817+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"749'617","last_scrub_stamp":"2024-09-16T05:21:41.857414+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:21:41.857414+0000","objects_scrubbed":3,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0044780080000000003,"stat_sum":{"num_bytes":3143173,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":66016,"num_write":78,"num_write_kb":43920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.39","version":"182'60","reported_seq":2670,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957800+0000","last_change":"2024-09-16T05:22:52.850294+0000","last_active":"2024-09-16T05:22:54.957800+0000","last_peered":"2024-09-16T05:22:54.957800+0000","last_clean":"2024-09-16T05:22:54.957800+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:22:54.957800+0000","last_undegraded":"2024-09-16T05:22:54.957800+0000","last_fullsized":"2024-09-16T05:22:54.957800+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:50.962024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:50.962024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:03.827889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0073587929999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.38","version":"181'57","reported_seq":2704,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.838935+0000","last_change":"2024-09-16T05:22:58.840115+0000","last_active":"2024-09-16T05:22:58.838935+0000","last_peered":"2024-09-16T05:22:58.838935+0000","last_clean":"2024-09-16T05:22:58.838935+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:22:58.838935+0000","last_undegraded":"2024-09-16T05:22:58.838935+0000","last_fullsized":"2024-09-16T05:22:58.838935+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:22:54.645961+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.645961+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:17.132243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014084189999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"2.37","version":"0'0","reported_seq":1534,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.936162+0000","last_change":"2024-09-16T05:22:52.846486+0000","last_active":"2024-09-16T05:22:54.936162+0000","last_peered":"2024-09-16T05:22:54.936162+0000","last_clean":"2024-09-16T05:22:54.936162+0000","last_became_active":"2024-09-16T05:21:54.938965+0000","last_became_peered":"2024-09-16T05:21:54.938965+0000","last_unstale":"2024-09-16T05:22:54.936162+0000","last_undegraded":"2024-09-16T05:22:54.936162+0000","last_fullsized":"2024-09-16T05:22:54.936162+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:25.115048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.115048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:38.269938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0070602219999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.36","version":"272'60","reported_seq":584,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957528+0000","last_change":"2024-09-16T05:22:52.922784+0000","last_active":"2024-09-16T05:22:54.957528+0000","last_peered":"2024-09-16T05:22:54.957528+0000","last_clean":"2024-09-16T05:22:54.957528+0000","last_became_active":"2024-09-16T05:21:07.819657+0000","last_became_peered":"2024-09-16T05:21:07.819657+0000","last_unstale":"2024-09-16T05:22:54.957528+0000","last_undegraded":"2024-09-16T05:22:54.957528+0000","last_fullsized":"2024-09-16T05:22:54.957528+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:22:49.686720+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.686720+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:03.136045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029385510000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.35","version":"823'945","reported_seq":4607,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957741+0000","last_change":"2024-09-16T05:22:52.926189+0000","last_active":"2024-09-16T05:22:54.957741+0000","last_peered":"2024-09-16T05:22:54.957741+0000","last_clean":"2024-09-16T05:22:54.957741+0000","last_became_active":"2024-09-16T05:21:10.815197+0000","last_became_peered":"2024-09-16T05:21:10.815197+0000","last_unstale":"2024-09-16T05:22:54.957741+0000","last_undegraded":"2024-09-16T05:22:54.957741+0000","last_fullsized":"2024-09-16T05:22:54.957741+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"769'866","last_scrub_stamp":"2024-09-16T05:22:00.928689+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.928689+0000","objects_scrubbed":9,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:09.119265+0000","scrub_duration":601,"objects_trimmed":2,"snaptrim_duration":0.0061735890000000002,"stat_sum":{"num_bytes":2201981,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":191,"num_read_kb":188287,"num_write":130,"num_write_kb":57425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":17054243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.34","version":"823'808","reported_seq":3996,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.931121+0000","last_change":"2024-09-16T05:22:52.850796+0000","last_active":"2024-09-16T05:22:54.931121+0000","last_peered":"2024-09-16T05:22:54.931121+0000","last_clean":"2024-09-16T05:22:54.931121+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:22:54.931121+0000","last_undegraded":"2024-09-16T05:22:54.931121+0000","last_fullsized":"2024-09-16T05:22:54.931121+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"781'749","last_scrub_stamp":"2024-09-16T05:22:12.568382+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:22:12.568382+0000","objects_scrubbed":9,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:17.174953+0000","scrub_duration":4,"objects_trimmed":2,"snaptrim_duration":0.0081037550000000007,"stat_sum":{"num_bytes":5479610,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":245667,"num_write":140,"num_write_kb":48956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":2764,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.135367+0000","last_change":"2024-09-16T05:22:52.849258+0000","last_active":"2024-09-16T05:22:56.135367+0000","last_peered":"2024-09-16T05:22:56.135367+0000","last_clean":"2024-09-16T05:22:56.135367+0000","last_became_active":"2024-09-16T05:21:07.818018+0000","last_became_peered":"2024-09-16T05:21:07.818018+0000","last_unstale":"2024-09-16T05:22:56.135367+0000","last_undegraded":"2024-09-16T05:22:56.135367+0000","last_fullsized":"2024-09-16T05:22:56.135367+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:22:08.520148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:08.520148+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:27.368313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0051773770000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.32","version":"820'309","reported_seq":3072,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.012514+0000","last_change":"2024-09-16T05:22:57.013032+0000","last_active":"2024-09-16T05:22:57.012514+0000","last_peered":"2024-09-16T05:22:57.012514+0000","last_clean":"2024-09-16T05:22:57.012514+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:22:57.012514+0000","last_undegraded":"2024-09-16T05:22:57.012514+0000","last_fullsized":"2024-09-16T05:22:57.012514+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'267","last_scrub_stamp":"2024-09-16T05:21:45.593289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:45.593289+0000","objects_scrubbed":3,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000746549,"stat_sum":{"num_bytes":2269209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":123,"num_read_kb":122722,"num_write":54,"num_write_kb":22810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"2.31","version":"824'362","reported_seq":3691,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.925506+0000","last_change":"2024-09-16T05:22:52.840533+0000","last_active":"2024-09-16T05:22:54.925506+0000","last_peered":"2024-09-16T05:22:54.925506+0000","last_clean":"2024-09-16T05:22:54.925506+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:22:54.925506+0000","last_undegraded":"2024-09-16T05:22:54.925506+0000","last_fullsized":"2024-09-16T05:22:54.925506+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"747'310","last_scrub_stamp":"2024-09-16T05:21:43.946354+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.946354+0000","objects_scrubbed":2,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:01.859965+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0050703969999999999,"stat_sum":{"num_bytes":5759693,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":119775,"num_write":60,"num_write_kb":25188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.30","version":"182'16","reported_seq":1508,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935303+0000","last_change":"2024-09-16T05:22:52.839009+0000","last_active":"2024-09-16T05:22:54.935303+0000","last_peered":"2024-09-16T05:22:54.935303+0000","last_clean":"2024-09-16T05:22:54.935303+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:22:54.935303+0000","last_undegraded":"2024-09-16T05:22:54.935303+0000","last_fullsized":"2024-09-16T05:22:54.935303+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:21:49.039822+0000","last_deep_scrub":"182'16","last_deep_scrub_stamp":"2024-09-16T05:21:49.039822+0000","last_clean_scrub_stamp":"2024-09-16T05:21:49.039822+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013286039999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.12","version":"35'139","reported_seq":1295,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.931220+0000","last_change":"2024-09-16T05:22:07.661072+0000","last_active":"2024-09-16T05:22:54.931220+0000","last_peered":"2024-09-16T05:22:54.931220+0000","last_clean":"2024-09-16T05:22:54.931220+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:22:54.931220+0000","last_undegraded":"2024-09-16T05:22:54.931220+0000","last_fullsized":"2024-09-16T05:22:54.931220+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.660982+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.660982+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:10.170857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":867,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.014246+0000","last_change":"2024-09-16T05:22:57.015358+0000","last_active":"2024-09-16T05:22:57.014246+0000","last_peered":"2024-09-16T05:22:57.014246+0000","last_clean":"2024-09-16T05:22:57.014246+0000","last_became_active":"2024-09-16T05:21:57.821735+0000","last_became_peered":"2024-09-16T05:21:57.821735+0000","last_unstale":"2024-09-16T05:22:57.014246+0000","last_undegraded":"2024-09-16T05:22:57.014246+0000","last_fullsized":"2024-09-16T05:22:57.014246+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:22:00.473439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.473439+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:27.705634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001311153,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1263,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957527+0000","last_change":"2024-09-16T05:21:23.189136+0000","last_active":"2024-09-16T05:22:54.957527+0000","last_peered":"2024-09-16T05:22:54.957527+0000","last_clean":"2024-09-16T05:22:54.957527+0000","last_became_active":"2024-09-16T05:21:10.814832+0000","last_became_peered":"2024-09-16T05:21:10.814832+0000","last_unstale":"2024-09-16T05:22:54.957527+0000","last_undegraded":"2024-09-16T05:22:54.957527+0000","last_fullsized":"2024-09-16T05:22:54.957527+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:23.189059+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:23.189059+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":3258,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.838990+0000","last_change":"2024-09-16T05:22:58.840130+0000","last_active":"2024-09-16T05:22:58.838990+0000","last_peered":"2024-09-16T05:22:58.838990+0000","last_clean":"2024-09-16T05:22:58.838990+0000","last_became_active":"2024-09-16T05:21:07.815584+0000","last_became_peered":"2024-09-16T05:21:07.815584+0000","last_unstale":"2024-09-16T05:22:58.838990+0000","last_undegraded":"2024-09-16T05:22:58.838990+0000","last_fullsized":"2024-09-16T05:22:58.838990+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:22:11.576311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:11.576311+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:30.877321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013432629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"1.c","version":"35'139","reported_seq":1259,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.840712+0000","last_change":"2024-09-16T05:21:43.300385+0000","last_active":"2024-09-16T05:22:57.840712+0000","last_peered":"2024-09-16T05:22:57.840712+0000","last_clean":"2024-09-16T05:22:57.840712+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:22:57.840712+0000","last_undegraded":"2024-09-16T05:22:57.840712+0000","last_fullsized":"2024-09-16T05:22:57.840712+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:43.300299+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.300299+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"823'461","reported_seq":4107,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.842519+0000","last_change":"2024-09-16T05:22:58.842519+0000","last_active":"2024-09-16T05:22:58.842519+0000","last_peered":"2024-09-16T05:22:58.842519+0000","last_clean":"2024-09-16T05:22:58.842519+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:22:58.842519+0000","last_undegraded":"2024-09-16T05:22:58.842519+0000","last_fullsized":"2024-09-16T05:22:58.842519+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'437","last_scrub_stamp":"2024-09-16T05:22:27.742914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.742914+0000","objects_scrubbed":3,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:32.150573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003454628,"stat_sum":{"num_bytes":3244015,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":58,"num_read_kb":67732,"num_write":57,"num_write_kb":24396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"1.d","version":"35'139","reported_seq":1413,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.135214+0000","last_change":"2024-09-16T05:22:26.361667+0000","last_active":"2024-09-16T05:22:56.135214+0000","last_peered":"2024-09-16T05:22:56.135214+0000","last_clean":"2024-09-16T05:22:56.135214+0000","last_became_active":"2024-09-16T05:20:35.804924+0000","last_became_peered":"2024-09-16T05:20:35.804924+0000","last_unstale":"2024-09-16T05:22:56.135214+0000","last_undegraded":"2024-09-16T05:22:56.135214+0000","last_fullsized":"2024-09-16T05:22:56.135214+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.361570+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.361570+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.129133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":3752,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.846946+0000","last_change":"2024-09-16T05:22:57.847618+0000","last_active":"2024-09-16T05:22:57.846946+0000","last_peered":"2024-09-16T05:22:57.846946+0000","last_clean":"2024-09-16T05:22:57.846946+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:22:57.846946+0000","last_undegraded":"2024-09-16T05:22:57.846946+0000","last_fullsized":"2024-09-16T05:22:57.846946+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:22:25.628406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.628406+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:26.353392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029674829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":1420,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:55.836805+0000","last_change":"2024-09-16T05:22:51.495227+0000","last_active":"2024-09-16T05:22:55.836805+0000","last_peered":"2024-09-16T05:22:55.836805+0000","last_clean":"2024-09-16T05:22:55.836805+0000","last_became_active":"2024-09-16T05:20:35.802192+0000","last_became_peered":"2024-09-16T05:20:35.802192+0000","last_unstale":"2024-09-16T05:22:55.836805+0000","last_undegraded":"2024-09-16T05:22:55.836805+0000","last_fullsized":"2024-09-16T05:22:55.836805+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:51.495148+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.495148+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:05.939482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"819'670","reported_seq":4281,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.958390+0000","last_change":"2024-09-16T05:22:52.850527+0000","last_active":"2024-09-16T05:22:54.958390+0000","last_peered":"2024-09-16T05:22:54.958390+0000","last_clean":"2024-09-16T05:22:54.958390+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:22:54.958390+0000","last_undegraded":"2024-09-16T05:22:54.958390+0000","last_fullsized":"2024-09-16T05:22:54.958390+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"793'662","last_scrub_stamp":"2024-09-16T05:22:26.926701+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.926701+0000","objects_scrubbed":3,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:56.251137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0075600240000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":52780,"num_write":55,"num_write_kb":30221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.f","version":"35'139","reported_seq":1400,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.931039+0000","last_change":"2024-09-16T05:22:08.676849+0000","last_active":"2024-09-16T05:22:54.931039+0000","last_peered":"2024-09-16T05:22:54.931039+0000","last_clean":"2024-09-16T05:22:54.931039+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:22:54.931039+0000","last_undegraded":"2024-09-16T05:22:54.931039+0000","last_fullsized":"2024-09-16T05:22:54.931039+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:08.676744+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:22:08.676744+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:24.347150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3873,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.013078+0000","last_change":"2024-09-16T05:22:57.013374+0000","last_active":"2024-09-16T05:22:57.013078+0000","last_peered":"2024-09-16T05:22:57.013078+0000","last_clean":"2024-09-16T05:22:57.013078+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:22:57.013078+0000","last_undegraded":"2024-09-16T05:22:57.013078+0000","last_fullsized":"2024-09-16T05:22:57.013078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:21:47.558894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:47.558894+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:15.226574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010798030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1111,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957408+0000","last_change":"2024-09-16T05:22:26.488008+0000","last_active":"2024-09-16T05:22:54.957408+0000","last_peered":"2024-09-16T05:22:54.957408+0000","last_clean":"2024-09-16T05:22:54.957408+0000","last_became_active":"2024-09-16T05:22:25.828286+0000","last_became_peered":"2024-09-16T05:22:25.828286+0000","last_unstale":"2024-09-16T05:22:54.957408+0000","last_undegraded":"2024-09-16T05:22:54.957408+0000","last_fullsized":"2024-09-16T05:22:54.957408+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.487937+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.487937+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:28.854050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"823'365","reported_seq":3976,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.134932+0000","last_change":"2024-09-16T05:22:52.847231+0000","last_active":"2024-09-16T05:22:56.134932+0000","last_peered":"2024-09-16T05:22:56.134932+0000","last_clean":"2024-09-16T05:22:56.134932+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:22:56.134932+0000","last_undegraded":"2024-09-16T05:22:56.134932+0000","last_fullsized":"2024-09-16T05:22:56.134932+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'327","last_scrub_stamp":"2024-09-16T05:22:02.625744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:02.625744+0000","objects_scrubbed":1,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:08.845511+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0033713430000000002,"stat_sum":{"num_bytes":2790226,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":75412,"num_write":54,"num_write_kb":15536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.3","version":"820'286","reported_seq":3902,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.931297+0000","last_change":"2024-09-16T05:22:52.845679+0000","last_active":"2024-09-16T05:22:54.931297+0000","last_peered":"2024-09-16T05:22:54.931297+0000","last_clean":"2024-09-16T05:22:54.931297+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:22:54.931297+0000","last_undegraded":"2024-09-16T05:22:54.931297+0000","last_fullsized":"2024-09-16T05:22:54.931297+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"761'256","last_scrub_stamp":"2024-09-16T05:22:03.687180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:03.687180+0000","objects_scrubbed":3,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:05.953891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030436600000000001,"stat_sum":{"num_bytes":2269209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":113,"num_read_kb":128644,"num_write":59,"num_write_kb":24461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":956,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.013455+0000","last_change":"2024-09-16T05:22:57.013492+0000","last_active":"2024-09-16T05:22:57.013455+0000","last_peered":"2024-09-16T05:22:57.013455+0000","last_clean":"2024-09-16T05:22:57.013455+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:22:57.013455+0000","last_undegraded":"2024-09-16T05:22:57.013455+0000","last_fullsized":"2024-09-16T05:22:57.013455+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:57.013376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:57.013376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:25.634739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":3609,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:55.657737+0000","last_change":"2024-09-16T05:22:55.657775+0000","last_active":"2024-09-16T05:22:55.657737+0000","last_peered":"2024-09-16T05:22:55.657737+0000","last_clean":"2024-09-16T05:22:55.657737+0000","last_became_active":"2024-09-16T05:21:07.817459+0000","last_became_peered":"2024-09-16T05:21:07.817459+0000","last_unstale":"2024-09-16T05:22:55.657737+0000","last_undegraded":"2024-09-16T05:22:55.657737+0000","last_fullsized":"2024-09-16T05:22:55.657737+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:22:55.657664+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:22:55.657664+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:21.034132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025611929999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":1178,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935194+0000","last_change":"2024-09-16T05:22:38.207552+0000","last_active":"2024-09-16T05:22:54.935194+0000","last_peered":"2024-09-16T05:22:54.935194+0000","last_clean":"2024-09-16T05:22:54.935194+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:22:54.935194+0000","last_undegraded":"2024-09-16T05:22:54.935194+0000","last_fullsized":"2024-09-16T05:22:54.935194+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:22:38.207426+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:00.581775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":477,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:55.836739+0000","last_change":"2024-09-16T05:22:26.467188+0000","last_active":"2024-09-16T05:22:55.836739+0000","last_peered":"2024-09-16T05:22:55.836739+0000","last_clean":"2024-09-16T05:22:55.836739+0000","last_became_active":"2024-09-16T05:22:25.830349+0000","last_became_peered":"2024-09-16T05:22:25.830349+0000","last_unstale":"2024-09-16T05:22:55.836739+0000","last_undegraded":"2024-09-16T05:22:55.836739+0000","last_fullsized":"2024-09-16T05:22:55.836739+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.467087+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.467087+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:32.527602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":3562,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.840772+0000","last_change":"2024-09-16T05:22:57.840772+0000","last_active":"2024-09-16T05:22:57.840772+0000","last_peered":"2024-09-16T05:22:57.840772+0000","last_clean":"2024-09-16T05:22:57.840772+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:22:57.840772+0000","last_undegraded":"2024-09-16T05:22:57.840772+0000","last_fullsized":"2024-09-16T05:22:57.840772+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:21:43.740165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.740165+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:22:58.871886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001701875,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1937,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.841256+0000","last_change":"2024-09-16T05:22:02.352598+0000","last_active":"2024-09-16T05:22:57.841256+0000","last_peered":"2024-09-16T05:22:57.841256+0000","last_clean":"2024-09-16T05:22:57.841256+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:22:57.841256+0000","last_undegraded":"2024-09-16T05:22:57.841256+0000","last_fullsized":"2024-09-16T05:22:57.841256+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:02.352511+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:22:02.352511+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:28.908704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":3632,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.135975+0000","last_change":"2024-09-16T05:22:52.846972+0000","last_active":"2024-09-16T05:22:56.135975+0000","last_peered":"2024-09-16T05:22:56.135975+0000","last_clean":"2024-09-16T05:22:56.135975+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:22:56.135975+0000","last_undegraded":"2024-09-16T05:22:56.135975+0000","last_fullsized":"2024-09-16T05:22:56.135975+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:22:36.317360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.317360+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:02.733275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030049489999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":2839,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.958224+0000","last_change":"2024-09-16T05:22:52.849009+0000","last_active":"2024-09-16T05:22:54.958224+0000","last_peered":"2024-09-16T05:22:54.958224+0000","last_clean":"2024-09-16T05:22:54.958224+0000","last_became_active":"2024-09-16T05:21:54.939117+0000","last_became_peered":"2024-09-16T05:21:54.939117+0000","last_unstale":"2024-09-16T05:22:54.958224+0000","last_undegraded":"2024-09-16T05:22:54.958224+0000","last_fullsized":"2024-09-16T05:22:54.958224+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:21:26.067761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:26.067761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062249130000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.4","version":"35'139","reported_seq":1890,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.012568+0000","last_change":"2024-09-16T05:22:48.936002+0000","last_active":"2024-09-16T05:22:57.012568+0000","last_peered":"2024-09-16T05:22:57.012568+0000","last_clean":"2024-09-16T05:22:57.012568+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:22:57.012568+0000","last_undegraded":"2024-09-16T05:22:57.012568+0000","last_fullsized":"2024-09-16T05:22:57.012568+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:48.935924+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.935924+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:07.656055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":2047,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.845993+0000","last_change":"2024-09-16T05:22:57.846892+0000","last_active":"2024-09-16T05:22:57.845993+0000","last_peered":"2024-09-16T05:22:57.845993+0000","last_clean":"2024-09-16T05:22:57.845993+0000","last_became_active":"2024-09-16T05:21:21.816054+0000","last_became_peered":"2024-09-16T05:21:21.816054+0000","last_unstale":"2024-09-16T05:22:57.845993+0000","last_undegraded":"2024-09-16T05:22:57.845993+0000","last_fullsized":"2024-09-16T05:22:57.845993+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:36.384222+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.384222+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:51.036736+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0.002455895,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":1199,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.936236+0000","last_change":"2024-09-16T05:22:10.087414+0000","last_active":"2024-09-16T05:22:54.936236+0000","last_peered":"2024-09-16T05:22:54.936236+0000","last_clean":"2024-09-16T05:22:54.936236+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:22:54.936236+0000","last_undegraded":"2024-09-16T05:22:54.936236+0000","last_fullsized":"2024-09-16T05:22:54.936236+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:10.087343+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.087343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:22.116328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"823'676","reported_seq":4314,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.842561+0000","last_change":"2024-09-16T05:22:58.842561+0000","last_active":"2024-09-16T05:22:58.842561+0000","last_peered":"2024-09-16T05:22:58.842561+0000","last_clean":"2024-09-16T05:22:58.842561+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:22:58.842561+0000","last_undegraded":"2024-09-16T05:22:58.842561+0000","last_fullsized":"2024-09-16T05:22:58.842561+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'618","last_scrub_stamp":"2024-09-16T05:21:42.686550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:42.686550+0000","objects_scrubbed":2,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:02.061382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032845629999999999,"stat_sum":{"num_bytes":3159980,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":131,"num_read_kb":146751,"num_write":54,"num_write_kb":21531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"1.6","version":"35'139","reported_seq":1143,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935810+0000","last_change":"2024-09-16T05:22:07.007557+0000","last_active":"2024-09-16T05:22:54.935810+0000","last_peered":"2024-09-16T05:22:54.935810+0000","last_clean":"2024-09-16T05:22:54.935810+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:22:54.935810+0000","last_undegraded":"2024-09-16T05:22:54.935810+0000","last_fullsized":"2024-09-16T05:22:54.935810+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.007487+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.007487+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.416611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":576,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957502+0000","last_change":"2024-09-16T05:22:52.923176+0000","last_active":"2024-09-16T05:22:54.957502+0000","last_peered":"2024-09-16T05:22:54.957502+0000","last_clean":"2024-09-16T05:22:54.957502+0000","last_became_active":"2024-09-16T05:21:07.816472+0000","last_became_peered":"2024-09-16T05:21:07.816472+0000","last_unstale":"2024-09-16T05:22:54.957502+0000","last_undegraded":"2024-09-16T05:22:54.957502+0000","last_fullsized":"2024-09-16T05:22:54.957502+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:21:48.443201+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:21:48.443201+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:14.786907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003177207,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":1189,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.936016+0000","last_change":"2024-09-16T05:22:08.048856+0000","last_active":"2024-09-16T05:22:54.936016+0000","last_peered":"2024-09-16T05:22:54.936016+0000","last_clean":"2024-09-16T05:22:54.936016+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:22:54.936016+0000","last_undegraded":"2024-09-16T05:22:54.936016+0000","last_fullsized":"2024-09-16T05:22:54.936016+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:08.048756+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:08.048756+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.883180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":3802,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.840196+0000","last_change":"2024-09-16T05:22:58.841508+0000","last_active":"2024-09-16T05:22:58.840196+0000","last_peered":"2024-09-16T05:22:58.840196+0000","last_clean":"2024-09-16T05:22:58.840196+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:22:58.840196+0000","last_undegraded":"2024-09-16T05:22:58.840196+0000","last_fullsized":"2024-09-16T05:22:58.840196+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:22:35.723844+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.723844+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:04.781033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024275239999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"1.8","version":"35'139","reported_seq":1925,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.931046+0000","last_change":"2024-09-16T05:22:10.608245+0000","last_active":"2024-09-16T05:22:54.931046+0000","last_peered":"2024-09-16T05:22:54.931046+0000","last_clean":"2024-09-16T05:22:54.931046+0000","last_became_active":"2024-09-16T05:20:35.807550+0000","last_became_peered":"2024-09-16T05:20:35.807550+0000","last_unstale":"2024-09-16T05:22:54.931046+0000","last_undegraded":"2024-09-16T05:22:54.931046+0000","last_fullsized":"2024-09-16T05:22:54.931046+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:10.608132+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.608132+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.753900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"823'734","reported_seq":4385,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.013444+0000","last_change":"2024-09-16T05:22:57.013816+0000","last_active":"2024-09-16T05:22:57.013444+0000","last_peered":"2024-09-16T05:22:57.013444+0000","last_clean":"2024-09-16T05:22:57.013444+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:22:57.013444+0000","last_undegraded":"2024-09-16T05:22:57.013444+0000","last_fullsized":"2024-09-16T05:22:57.013444+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'707","last_scrub_stamp":"2024-09-16T05:22:31.956529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:31.956529+0000","objects_scrubbed":5,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:50.810701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013323059999999999,"stat_sum":{"num_bytes":3277629,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":125,"num_read_kb":137085,"num_write":84,"num_write_kb":41818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1924,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.134924+0000","last_change":"2024-09-16T05:22:04.552683+0000","last_active":"2024-09-16T05:22:56.134924+0000","last_peered":"2024-09-16T05:22:56.134924+0000","last_clean":"2024-09-16T05:22:56.134924+0000","last_became_active":"2024-09-16T05:20:35.804221+0000","last_became_peered":"2024-09-16T05:20:35.804221+0000","last_unstale":"2024-09-16T05:22:56.134924+0000","last_undegraded":"2024-09-16T05:22:56.134924+0000","last_fullsized":"2024-09-16T05:22:56.134924+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:04.552582+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:04.552582+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:17.369449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2050,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957368+0000","last_change":"2024-09-16T05:22:52.920470+0000","last_active":"2024-09-16T05:22:54.957368+0000","last_peered":"2024-09-16T05:22:54.957368+0000","last_clean":"2024-09-16T05:22:54.957368+0000","last_became_active":"2024-09-16T05:21:10.815913+0000","last_became_peered":"2024-09-16T05:21:10.815913+0000","last_unstale":"2024-09-16T05:22:54.957368+0000","last_undegraded":"2024-09-16T05:22:54.957368+0000","last_fullsized":"2024-09-16T05:22:54.957368+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:21:43.417102+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.417102+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:07.655160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071426400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":1930,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957929+0000","last_change":"2024-09-16T05:21:54.943368+0000","last_active":"2024-09-16T05:22:54.957929+0000","last_peered":"2024-09-16T05:22:54.957929+0000","last_clean":"2024-09-16T05:22:54.957929+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:22:54.957929+0000","last_undegraded":"2024-09-16T05:22:54.957929+0000","last_fullsized":"2024-09-16T05:22:54.957929+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:45.767900+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:21:45.767900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:09.643423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1408,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.925003+0000","last_change":"2024-09-16T05:22:32.678108+0000","last_active":"2024-09-16T05:22:54.925003+0000","last_peered":"2024-09-16T05:22:54.925003+0000","last_clean":"2024-09-16T05:22:54.925003+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:22:54.925003+0000","last_undegraded":"2024-09-16T05:22:54.925003+0000","last_fullsized":"2024-09-16T05:22:54.925003+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:32.678034+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.678034+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:39.197820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":1508,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935877+0000","last_change":"2024-09-16T05:22:52.839868+0000","last_active":"2024-09-16T05:22:54.935877+0000","last_peered":"2024-09-16T05:22:54.935877+0000","last_clean":"2024-09-16T05:22:54.935877+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:22:54.935877+0000","last_undegraded":"2024-09-16T05:22:54.935877+0000","last_fullsized":"2024-09-16T05:22:54.935877+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:21:41.968085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:41.968085+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018822400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.10","version":"35'139","reported_seq":1922,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.930791+0000","last_change":"2024-09-16T05:22:50.729358+0000","last_active":"2024-09-16T05:22:54.930791+0000","last_peered":"2024-09-16T05:22:54.930791+0000","last_clean":"2024-09-16T05:22:54.930791+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:22:54.930791+0000","last_undegraded":"2024-09-16T05:22:54.930791+0000","last_fullsized":"2024-09-16T05:22:54.930791+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:50.729283+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:50.729283+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:18.895673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":3222,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.958142+0000","last_change":"2024-09-16T05:22:52.847530+0000","last_active":"2024-09-16T05:22:54.958142+0000","last_peered":"2024-09-16T05:22:54.958142+0000","last_clean":"2024-09-16T05:22:54.958142+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:22:54.958142+0000","last_undegraded":"2024-09-16T05:22:54.958142+0000","last_fullsized":"2024-09-16T05:22:54.958142+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:22:26.133349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.133349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:34.455620+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0.0048759550000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.17","version":"35'139","reported_seq":811,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.838902+0000","last_change":"2024-09-16T05:22:00.654377+0000","last_active":"2024-09-16T05:22:58.838902+0000","last_peered":"2024-09-16T05:22:58.838902+0000","last_clean":"2024-09-16T05:22:58.838902+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:22:58.838902+0000","last_undegraded":"2024-09-16T05:22:58.838902+0000","last_fullsized":"2024-09-16T05:22:58.838902+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:00.654245+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.654245+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:13.780881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":3500,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.958079+0000","last_change":"2024-09-16T05:22:52.848244+0000","last_active":"2024-09-16T05:22:54.958079+0000","last_peered":"2024-09-16T05:22:54.958079+0000","last_clean":"2024-09-16T05:22:54.958079+0000","last_became_active":"2024-09-16T05:21:54.931954+0000","last_became_peered":"2024-09-16T05:21:54.931954+0000","last_unstale":"2024-09-16T05:22:54.958079+0000","last_undegraded":"2024-09-16T05:22:54.958079+0000","last_fullsized":"2024-09-16T05:22:54.958079+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:21:46.719776+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:21:46.719776+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:15.052611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055381270000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.16","version":"35'139","reported_seq":867,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.843992+0000","last_change":"2024-09-16T05:22:05.738733+0000","last_active":"2024-09-16T05:22:57.843992+0000","last_peered":"2024-09-16T05:22:57.843992+0000","last_clean":"2024-09-16T05:22:57.843992+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:22:57.843992+0000","last_undegraded":"2024-09-16T05:22:57.843992+0000","last_fullsized":"2024-09-16T05:22:57.843992+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:05.738655+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:22:05.738655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:07.277398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"760'658","reported_seq":4391,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.958157+0000","last_change":"2024-09-16T05:22:52.849610+0000","last_active":"2024-09-16T05:22:54.958157+0000","last_peered":"2024-09-16T05:22:54.958157+0000","last_clean":"2024-09-16T05:22:54.958157+0000","last_became_active":"2024-09-16T05:21:07.826042+0000","last_became_peered":"2024-09-16T05:21:07.826042+0000","last_unstale":"2024-09-16T05:22:54.958157+0000","last_undegraded":"2024-09-16T05:22:54.958157+0000","last_fullsized":"2024-09-16T05:22:54.958157+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"760'658","last_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_deep_scrub":"760'658","last_deep_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_clean_scrub_stamp":"2024-09-16T05:22:05.847540+0000","objects_scrubbed":0,"log_size":658,"log_dups_size":0,"ondisk_log_size":658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:06.876836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.006756583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":7118573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.15","version":"35'139","reported_seq":877,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.930861+0000","last_change":"2024-09-16T05:22:06.689465+0000","last_active":"2024-09-16T05:22:54.930861+0000","last_peered":"2024-09-16T05:22:54.930861+0000","last_clean":"2024-09-16T05:22:54.930861+0000","last_became_active":"2024-09-16T05:20:40.802090+0000","last_became_peered":"2024-09-16T05:20:40.802090+0000","last_unstale":"2024-09-16T05:22:54.930861+0000","last_undegraded":"2024-09-16T05:22:54.930861+0000","last_fullsized":"2024-09-16T05:22:54.930861+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:06.689376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.689376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:29.936194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"733'334","reported_seq":3586,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.931163+0000","last_change":"2024-09-16T05:22:52.849023+0000","last_active":"2024-09-16T05:22:54.931163+0000","last_peered":"2024-09-16T05:22:54.931163+0000","last_clean":"2024-09-16T05:22:54.931163+0000","last_became_active":"2024-09-16T05:21:54.937285+0000","last_became_peered":"2024-09-16T05:21:54.937285+0000","last_unstale":"2024-09-16T05:22:54.931163+0000","last_undegraded":"2024-09-16T05:22:54.931163+0000","last_fullsized":"2024-09-16T05:22:54.931163+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'334","last_scrub_stamp":"2024-09-16T05:22:51.700579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.700579+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:09.993152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058042450000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.14","version":"35'139","reported_seq":1250,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.839766+0000","last_change":"2024-09-16T05:21:48.712051+0000","last_active":"2024-09-16T05:22:57.839766+0000","last_peered":"2024-09-16T05:22:57.839766+0000","last_clean":"2024-09-16T05:22:57.839766+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:22:57.839766+0000","last_undegraded":"2024-09-16T05:22:57.839766+0000","last_fullsized":"2024-09-16T05:22:57.839766+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:48.711960+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:48.711960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:02.377018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":595,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957515+0000","last_change":"2024-09-16T05:22:52.926611+0000","last_active":"2024-09-16T05:22:54.957515+0000","last_peered":"2024-09-16T05:22:54.957515+0000","last_clean":"2024-09-16T05:22:54.957515+0000","last_became_active":"2024-09-16T05:21:27.826090+0000","last_became_peered":"2024-09-16T05:21:27.826090+0000","last_unstale":"2024-09-16T05:22:54.957515+0000","last_undegraded":"2024-09-16T05:22:54.957515+0000","last_fullsized":"2024-09-16T05:22:54.957515+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.831827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:05.013983+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0.0065985100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.1b","version":"35'139","reported_seq":1336,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.838805+0000","last_change":"2024-09-16T05:22:07.577401+0000","last_active":"2024-09-16T05:22:58.838805+0000","last_peered":"2024-09-16T05:22:58.838805+0000","last_clean":"2024-09-16T05:22:58.838805+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:22:58.838805+0000","last_undegraded":"2024-09-16T05:22:58.838805+0000","last_fullsized":"2024-09-16T05:22:58.838805+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.577328+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.577328+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.106050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":3615,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957871+0000","last_change":"2024-09-16T05:22:52.851051+0000","last_active":"2024-09-16T05:22:54.957871+0000","last_peered":"2024-09-16T05:22:54.957871+0000","last_clean":"2024-09-16T05:22:54.957871+0000","last_became_active":"2024-09-16T05:21:27.832312+0000","last_became_peered":"2024-09-16T05:21:27.832312+0000","last_unstale":"2024-09-16T05:22:54.957871+0000","last_undegraded":"2024-09-16T05:22:54.957871+0000","last_fullsized":"2024-09-16T05:22:54.957871+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:22:32.994229+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.994229+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:38.216614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0079174510000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.1a","version":"35'139","reported_seq":695,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:55.836981+0000","last_change":"2024-09-16T05:22:07.494046+0000","last_active":"2024-09-16T05:22:55.836981+0000","last_peered":"2024-09-16T05:22:55.836981+0000","last_clean":"2024-09-16T05:22:55.836981+0000","last_became_active":"2024-09-16T05:20:40.806260+0000","last_became_peered":"2024-09-16T05:20:40.806260+0000","last_unstale":"2024-09-16T05:22:55.836981+0000","last_undegraded":"2024-09-16T05:22:55.836981+0000","last_fullsized":"2024-09-16T05:22:55.836981+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.493964+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.493964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.726913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":3247,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.841130+0000","last_change":"2024-09-16T05:22:57.842905+0000","last_active":"2024-09-16T05:22:57.841130+0000","last_peered":"2024-09-16T05:22:57.841130+0000","last_clean":"2024-09-16T05:22:57.841130+0000","last_became_active":"2024-09-16T05:21:54.930126+0000","last_became_peered":"2024-09-16T05:21:54.930126+0000","last_unstale":"2024-09-16T05:22:57.841130+0000","last_undegraded":"2024-09-16T05:22:57.841130+0000","last_fullsized":"2024-09-16T05:22:57.841130+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:35.348819+0000","last_deep_scrub":"182'60","last_deep_scrub_stamp":"2024-09-16T05:21:29.356695+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.348819+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:42.457160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020577170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"1.19","version":"35'139","reported_seq":1923,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935212+0000","last_change":"2024-09-16T05:22:28.141713+0000","last_active":"2024-09-16T05:22:54.935212+0000","last_peered":"2024-09-16T05:22:54.935212+0000","last_clean":"2024-09-16T05:22:54.935212+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:22:54.935212+0000","last_undegraded":"2024-09-16T05:22:54.935212+0000","last_fullsized":"2024-09-16T05:22:54.935212+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:28.141626+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.141626+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:45.344742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"823'1233","reported_seq":4996,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.841435+0000","last_change":"2024-09-16T05:22:58.841435+0000","last_active":"2024-09-16T05:22:58.841435+0000","last_peered":"2024-09-16T05:22:58.841435+0000","last_clean":"2024-09-16T05:22:58.841435+0000","last_became_active":"2024-09-16T05:21:54.943140+0000","last_became_peered":"2024-09-16T05:21:54.943140+0000","last_unstale":"2024-09-16T05:22:58.841435+0000","last_undegraded":"2024-09-16T05:22:58.841435+0000","last_fullsized":"2024-09-16T05:22:58.841435+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"819'1229","last_scrub_stamp":"2024-09-16T05:22:49.730316+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.730316+0000","objects_scrubbed":4,"log_size":1233,"log_dups_size":0,"ondisk_log_size":1233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:19.152619+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002474763,"stat_sum":{"num_bytes":3714545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":37762,"num_write":13,"num_write_kb":4898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":19652756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"1.18","version":"35'139","reported_seq":1925,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.930544+0000","last_change":"2024-09-16T05:22:30.658597+0000","last_active":"2024-09-16T05:22:54.930544+0000","last_peered":"2024-09-16T05:22:54.930544+0000","last_clean":"2024-09-16T05:22:54.930544+0000","last_became_active":"2024-09-16T05:20:35.809442+0000","last_became_peered":"2024-09-16T05:20:35.809442+0000","last_unstale":"2024-09-16T05:22:54.930544+0000","last_undegraded":"2024-09-16T05:22:54.930544+0000","last_fullsized":"2024-09-16T05:22:54.930544+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:30.658505+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:30.658505+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.751732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"823'734","reported_seq":4387,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.013140+0000","last_change":"2024-09-16T05:22:57.013533+0000","last_active":"2024-09-16T05:22:57.013140+0000","last_peered":"2024-09-16T05:22:57.013140+0000","last_clean":"2024-09-16T05:22:57.013140+0000","last_became_active":"2024-09-16T05:21:27.832003+0000","last_became_peered":"2024-09-16T05:21:27.832003+0000","last_unstale":"2024-09-16T05:22:57.013140+0000","last_undegraded":"2024-09-16T05:22:57.013140+0000","last_fullsized":"2024-09-16T05:22:57.013140+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"823'734","last_scrub_stamp":"2024-09-16T05:22:54.026063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.026063+0000","objects_scrubbed":1,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:23.472391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0012151009999999999,"stat_sum":{"num_bytes":2655770,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":27074,"num_write":19,"num_write_kb":6977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"2.1c","version":"723'472","reported_seq":3786,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935671+0000","last_change":"2024-09-16T05:22:52.841040+0000","last_active":"2024-09-16T05:22:54.935671+0000","last_peered":"2024-09-16T05:22:54.935671+0000","last_clean":"2024-09-16T05:22:54.935671+0000","last_became_active":"2024-09-16T05:21:27.831591+0000","last_became_peered":"2024-09-16T05:21:27.831591+0000","last_unstale":"2024-09-16T05:22:54.935671+0000","last_undegraded":"2024-09-16T05:22:54.935671+0000","last_fullsized":"2024-09-16T05:22:54.935671+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"723'472","last_scrub_stamp":"2024-09-16T05:22:51.347894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.347894+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:19.932119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026611270000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1415,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.012573+0000","last_change":"2024-09-16T05:22:25.948680+0000","last_active":"2024-09-16T05:22:57.012573+0000","last_peered":"2024-09-16T05:22:57.012573+0000","last_clean":"2024-09-16T05:22:57.012573+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:22:57.012573+0000","last_undegraded":"2024-09-16T05:22:57.012573+0000","last_fullsized":"2024-09-16T05:22:57.012573+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:25.948598+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.948598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:47.051847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"823'1307","reported_seq":5155,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.936226+0000","last_change":"2024-09-16T05:22:53.289317+0000","last_active":"2024-09-16T05:22:54.936226+0000","last_peered":"2024-09-16T05:22:54.936226+0000","last_clean":"2024-09-16T05:22:54.936226+0000","last_became_active":"2024-09-16T05:21:27.833962+0000","last_became_peered":"2024-09-16T05:21:27.833962+0000","last_unstale":"2024-09-16T05:22:54.936226+0000","last_undegraded":"2024-09-16T05:22:54.936226+0000","last_fullsized":"2024-09-16T05:22:54.936226+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"823'1307","last_scrub_stamp":"2024-09-16T05:22:53.289164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:53.289164+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:08.401373+0000","scrub_duration":1,"objects_trimmed":2,"snaptrim_duration":0.0067716959999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":2561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"1.1d","version":"35'139","reported_seq":756,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.957804+0000","last_change":"2024-09-16T05:22:37.530745+0000","last_active":"2024-09-16T05:22:54.957804+0000","last_peered":"2024-09-16T05:22:54.957804+0000","last_clean":"2024-09-16T05:22:54.957804+0000","last_became_active":"2024-09-16T05:21:10.815037+0000","last_became_peered":"2024-09-16T05:21:10.815037+0000","last_unstale":"2024-09-16T05:22:54.957804+0000","last_undegraded":"2024-09-16T05:22:54.957804+0000","last_fullsized":"2024-09-16T05:22:54.957804+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:37.530632+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.530632+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:37.641748+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"736'512","reported_seq":4172,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.845404+0000","last_change":"2024-09-16T05:22:57.847150+0000","last_active":"2024-09-16T05:22:57.845404+0000","last_peered":"2024-09-16T05:22:57.845404+0000","last_clean":"2024-09-16T05:22:57.845404+0000","last_became_active":"2024-09-16T05:21:27.826212+0000","last_became_peered":"2024-09-16T05:21:27.826212+0000","last_unstale":"2024-09-16T05:22:57.845404+0000","last_undegraded":"2024-09-16T05:22:57.845404+0000","last_fullsized":"2024-09-16T05:22:57.845404+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'512","last_scrub_stamp":"2024-09-16T05:22:34.394958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.394958+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:51.927806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0050909199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"1.1c","version":"35'139","reported_seq":666,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.839024+0000","last_change":"2024-09-16T05:22:06.589785+0000","last_active":"2024-09-16T05:22:58.839024+0000","last_peered":"2024-09-16T05:22:58.839024+0000","last_clean":"2024-09-16T05:22:58.839024+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:22:58.839024+0000","last_undegraded":"2024-09-16T05:22:58.839024+0000","last_fullsized":"2024-09-16T05:22:58.839024+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:06.589701+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.589701+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:21.427227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":3837,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.925418+0000","last_change":"2024-09-16T05:22:52.836910+0000","last_active":"2024-09-16T05:22:54.925418+0000","last_peered":"2024-09-16T05:22:54.925418+0000","last_clean":"2024-09-16T05:22:54.925418+0000","last_became_active":"2024-09-16T05:21:27.831337+0000","last_became_peered":"2024-09-16T05:21:27.831337+0000","last_unstale":"2024-09-16T05:22:54.925418+0000","last_undegraded":"2024-09-16T05:22:54.925418+0000","last_fullsized":"2024-09-16T05:22:54.925418+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:22:37.720977+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.720977+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:45.563196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010985019999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.20","version":"823'378","reported_seq":4025,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.134937+0000","last_change":"2024-09-16T05:22:52.844688+0000","last_active":"2024-09-16T05:22:56.134937+0000","last_peered":"2024-09-16T05:22:56.134937+0000","last_clean":"2024-09-16T05:22:56.134937+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:22:56.134937+0000","last_undegraded":"2024-09-16T05:22:56.134937+0000","last_fullsized":"2024-09-16T05:22:56.134937+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"733'323","last_scrub_stamp":"2024-09-16T05:21:43.765502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.765502+0000","objects_scrubbed":1,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:10.949020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001136193,"stat_sum":{"num_bytes":3294436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":166,"num_read_kb":195113,"num_write":75,"num_write_kb":29696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":3569,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.843288+0000","last_change":"2024-09-16T05:22:57.843288+0000","last_active":"2024-09-16T05:22:57.843288+0000","last_peered":"2024-09-16T05:22:57.843288+0000","last_clean":"2024-09-16T05:22:57.843288+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:22:57.843288+0000","last_undegraded":"2024-09-16T05:22:57.843288+0000","last_fullsized":"2024-09-16T05:22:57.843288+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:22:27.632823+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.632823+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:37.612079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0035155630000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":1514,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.935353+0000","last_change":"2024-09-16T05:22:52.840509+0000","last_active":"2024-09-16T05:22:54.935353+0000","last_peered":"2024-09-16T05:22:54.935353+0000","last_clean":"2024-09-16T05:22:54.935353+0000","last_became_active":"2024-09-16T05:21:54.942985+0000","last_became_peered":"2024-09-16T05:21:54.942985+0000","last_unstale":"2024-09-16T05:22:54.935353+0000","last_undegraded":"2024-09-16T05:22:54.935353+0000","last_fullsized":"2024-09-16T05:22:54.935353+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:21:46.010447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:46.010447+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:04.861462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023403730000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":3107,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.843864+0000","last_change":"2024-09-16T05:22:57.845831+0000","last_active":"2024-09-16T05:22:57.843864+0000","last_peered":"2024-09-16T05:22:57.843864+0000","last_clean":"2024-09-16T05:22:57.843864+0000","last_became_active":"2024-09-16T05:22:22.838582+0000","last_became_peered":"2024-09-16T05:22:22.838582+0000","last_unstale":"2024-09-16T05:22:57.843864+0000","last_undegraded":"2024-09-16T05:22:57.843864+0000","last_fullsized":"2024-09-16T05:22:57.843864+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:22:26.319237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.319237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:26.319503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0043434880000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":2896,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.958079+0000","last_change":"2024-09-16T05:22:52.844576+0000","last_active":"2024-09-16T05:22:54.958079+0000","last_peered":"2024-09-16T05:22:54.958079+0000","last_clean":"2024-09-16T05:22:54.958079+0000","last_became_active":"2024-09-16T05:21:54.932630+0000","last_became_peered":"2024-09-16T05:21:54.932630+0000","last_unstale":"2024-09-16T05:22:54.958079+0000","last_undegraded":"2024-09-16T05:22:54.958079+0000","last_fullsized":"2024-09-16T05:22:54.958079+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:21:31.054480+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:21:31.054480+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021439380000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.25","version":"828'1351","reported_seq":4884,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.842576+0000","last_change":"2024-09-16T05:22:58.843844+0000","last_active":"2024-09-16T05:22:58.842576+0000","last_peered":"2024-09-16T05:22:58.842576+0000","last_clean":"2024-09-16T05:22:58.842576+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:22:58.842576+0000","last_undegraded":"2024-09-16T05:22:58.842576+0000","last_fullsized":"2024-09-16T05:22:58.842576+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"781'1244","last_scrub_stamp":"2024-09-16T05:22:12.573275+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:22:12.573275+0000","objects_scrubbed":8,"log_size":1351,"log_dups_size":0,"ondisk_log_size":1351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:29.662368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0044722690000000001,"stat_sum":{"num_bytes":5580452,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":260,"num_read_kb":258442,"num_write":222,"num_write_kb":89381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"2.26","version":"116'9","reported_seq":2888,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.135845+0000","last_change":"2024-09-16T05:22:52.852864+0000","last_active":"2024-09-16T05:22:56.135845+0000","last_peered":"2024-09-16T05:22:56.135845+0000","last_clean":"2024-09-16T05:22:56.135845+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:22:56.135845+0000","last_undegraded":"2024-09-16T05:22:56.135845+0000","last_fullsized":"2024-09-16T05:22:56.135845+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.590423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:12.593663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081555900000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":2900,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.137104+0000","last_change":"2024-09-16T05:22:52.850771+0000","last_active":"2024-09-16T05:22:56.137104+0000","last_peered":"2024-09-16T05:22:56.137104+0000","last_clean":"2024-09-16T05:22:56.137104+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:22:56.137104+0000","last_undegraded":"2024-09-16T05:22:56.137104+0000","last_fullsized":"2024-09-16T05:22:56.137104+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:25.368861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.368861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:46.345423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0062868079999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1934,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.840144+0000","last_change":"2024-09-16T05:22:57.841162+0000","last_active":"2024-09-16T05:22:57.840144+0000","last_peered":"2024-09-16T05:22:57.840144+0000","last_clean":"2024-09-16T05:22:57.840144+0000","last_became_active":"2024-09-16T05:21:07.818030+0000","last_became_peered":"2024-09-16T05:21:07.818030+0000","last_unstale":"2024-09-16T05:22:57.840144+0000","last_undegraded":"2024-09-16T05:22:57.840144+0000","last_fullsized":"2024-09-16T05:22:57.840144+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:22:34.641066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.641066+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.805873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001801836,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.29","version":"823'1389","reported_seq":5305,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.014297+0000","last_change":"2024-09-16T05:22:57.015360+0000","last_active":"2024-09-16T05:22:57.014297+0000","last_peered":"2024-09-16T05:22:57.014297+0000","last_clean":"2024-09-16T05:22:57.014297+0000","last_became_active":"2024-09-16T05:21:57.823393+0000","last_became_peered":"2024-09-16T05:21:57.823393+0000","last_unstale":"2024-09-16T05:22:57.014297+0000","last_undegraded":"2024-09-16T05:22:57.014297+0000","last_fullsized":"2024-09-16T05:22:57.014297+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":6,"last_scrub":"795'1297","last_scrub_stamp":"2024-09-16T05:22:26.617228+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.617228+0000","objects_scrubbed":15,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:54.217460+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013056400000000001,"stat_sum":{"num_bytes":2790226,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":347,"num_read_kb":316397,"num_write":270,"num_write_kb":99162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":45326140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"},{"start":"15c","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1193,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.930808+0000","last_change":"2024-09-16T05:22:52.843073+0000","last_active":"2024-09-16T05:22:54.930808+0000","last_peered":"2024-09-16T05:22:54.930808+0000","last_clean":"2024-09-16T05:22:54.930808+0000","last_became_active":"2024-09-16T05:22:25.829990+0000","last_became_peered":"2024-09-16T05:22:25.829990+0000","last_unstale":"2024-09-16T05:22:54.930808+0000","last_undegraded":"2024-09-16T05:22:54.930808+0000","last_fullsized":"2024-09-16T05:22:54.930808+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:22:26.604651+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.604651+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:34.491343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001095887,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]},{"pgid":"2.2b","version":"825'443","reported_seq":3995,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-09-16T05:22:57.843474+0000","last_change":"2024-09-16T05:22:57.844913+0000","last_active":"2024-09-16T05:22:57.843474+0000","last_peered":"2024-09-16T05:22:57.843474+0000","last_clean":"2024-09-16T05:22:57.843474+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:22:57.843474+0000","last_undegraded":"2024-09-16T05:22:57.843474+0000","last_fullsized":"2024-09-16T05:22:57.843474+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"825'443","last_scrub_stamp":"2024-09-16T05:22:57.584272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:57.584272+0000","objects_scrubbed":2,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:14.572363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0051047779999999999,"stat_sum":{"num_bytes":3344857,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":100,"num_read_kb":89099,"num_write":75,"num_write_kb":28866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"2"},{"start":"15c","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":3749,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-09-16T05:22:58.842637+0000","last_change":"2024-09-16T05:22:58.843845+0000","last_active":"2024-09-16T05:22:58.842637+0000","last_peered":"2024-09-16T05:22:58.842637+0000","last_clean":"2024-09-16T05:22:58.842637+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:22:58.842637+0000","last_undegraded":"2024-09-16T05:22:58.842637+0000","last_fullsized":"2024-09-16T05:22:58.842637+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:22:48.737646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.737646+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:10.723051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044739469999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"4"}]},{"pgid":"2.2d","version":"819'522","reported_seq":4211,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.135469+0000","last_change":"2024-09-16T05:22:54.200199+0000","last_active":"2024-09-16T05:22:56.135469+0000","last_peered":"2024-09-16T05:22:56.135469+0000","last_clean":"2024-09-16T05:22:56.135469+0000","last_became_active":"2024-09-16T05:21:54.933266+0000","last_became_peered":"2024-09-16T05:21:54.933266+0000","last_unstale":"2024-09-16T05:22:56.135469+0000","last_undegraded":"2024-09-16T05:22:56.135469+0000","last_fullsized":"2024-09-16T05:22:56.135469+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"819'522","last_scrub_stamp":"2024-09-16T05:22:54.200053+0000","last_deep_scrub":"819'522","last_deep_scrub_stamp":"2024-09-16T05:22:54.200053+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.200053+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:58.898283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00167932,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":88584,"num_write":74,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":10583843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.2e","version":"823'795","reported_seq":4341,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-09-16T05:22:56.135841+0000","last_change":"2024-09-16T05:22:52.855025+0000","last_active":"2024-09-16T05:22:56.135841+0000","last_peered":"2024-09-16T05:22:56.135841+0000","last_clean":"2024-09-16T05:22:56.135841+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:22:56.135841+0000","last_undegraded":"2024-09-16T05:22:56.135841+0000","last_fullsized":"2024-09-16T05:22:56.135841+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"794'746","last_scrub_stamp":"2024-09-16T05:22:28.624283+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.624283+0000","objects_scrubbed":7,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:42.179663+0000","scrub_duration":281,"objects_trimmed":2,"snaptrim_duration":0.010562722,"stat_sum":{"num_bytes":3882648,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":80561,"num_write":113,"num_write_kb":55464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14e","length":"1"},{"start":"159","length":"1"}]},{"pgid":"2.2f","version":"823'726","reported_seq":4372,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-09-16T05:22:54.958210+0000","last_change":"2024-09-16T05:22:52.854508+0000","last_active":"2024-09-16T05:22:54.958210+0000","last_peered":"2024-09-16T05:22:54.958210+0000","last_clean":"2024-09-16T05:22:54.958210+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:22:54.958210+0000","last_undegraded":"2024-09-16T05:22:54.958210+0000","last_fullsized":"2024-09-16T05:22:54.958210+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"795'683","last_scrub_stamp":"2024-09-16T05:22:27.948411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.948411+0000","objects_scrubbed":5,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:44.387156+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.011185725000000001,"stat_sum":{"num_bytes":6420802,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":132,"num_read_kb":85533,"num_write":148,"num_write_kb":54423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"116","length":"1"},{"start":"119","length":"1"},{"start":"149","length":"1"},{"start":"14e","length":"1"},{"start":"153","length":"1"},{"start":"155","length":"1"},{"start":"158","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":96,"stat_sum":{"num_bytes":101651713,"num_objects":36,"num_object_clones":3,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":1,"num_read":3616,"num_read_kb":3599840,"num_write":2545,"num_write_kb":1050826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":136736768,"data_stored":142389925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33209,"ondisk_log_size":33209,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089511,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85448,"kb_used_data":27884,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93659704,"statfs":{"total":95995035648,"available":95907536896,"internally_reserved":0,"allocated":28553216,"data_stored":25066542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":58915268},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285739,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84292,"kb_used_data":26736,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93660860,"statfs":{"total":95995035648,"available":95908720640,"internally_reserved":0,"allocated":27377664,"data_stored":28128145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514676,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67944,"kb_used_data":22868,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93677208,"statfs":{"total":95995035648,"available":95925460992,"internally_reserved":0,"allocated":23416832,"data_stored":20060255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841717,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82140,"kb_used_data":24580,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93663012,"statfs":{"total":95995035648,"available":95910924288,"internally_reserved":0,"allocated":25169920,"data_stored":21763556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103354,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67072,"kb_used_data":17828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93678080,"statfs":{"total":95995035648,"available":95926353920,"internally_reserved":0,"allocated":18255872,"data_stored":14918924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397697,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60372,"kb_used_data":15292,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93684780,"statfs":{"total":95995035648,"available":95933214720,"internally_reserved":0,"allocated":15659008,"data_stored":12327211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757443,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":52900,"kb_used_data":21160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93692252,"statfs":{"total":95995035648,"available":95940866048,"internally_reserved":0,"allocated":21667840,"data_stored":18219488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920969,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50476,"kb_used_data":15124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93694676,"statfs":{"total":95995035648,"available":95943348224,"internally_reserved":0,"allocated":15486976,"data_stored":12127327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117198,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70184,"kb_used_data":25112,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93674968,"statfs":{"total":95995035648,"available":95923167232,"internally_reserved":0,"allocated":25714688,"data_stored":22282941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240485,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47352,"kb_used_data":19236,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93697800,"statfs":{"total":95995035648,"available":95946547200,"internally_reserved":0,"allocated":19697664,"data_stored":19903945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75416,"kb_used_data":17860,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93669736,"statfs":{"total":95995035648,"available":95917809664,"internally_reserved":0,"allocated":18288640,"data_stored":14941518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738589,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71620,"kb_used_data":18216,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93673532,"statfs":{"total":95995035648,"available":95921696768,"internally_reserved":0,"allocated":18653184,"data_stored":15924381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8339456,"data_stored":8912078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8302592,"data_stored":8256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9408512,"data_stored":11161642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15364096,"data_stored":15245118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5488640,"data_stored":5434240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11464704,"data_stored":11325153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5648384,"data_stored":5625524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8245248,"data_stored":8217237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15134720,"data_stored":15044957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13418496,"data_stored":13367168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":17379328,"data_stored":21435058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18542592,"data_stored":18364855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:23:01.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 96 2024-09-16T05:23:01.915 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:01.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:01.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:01.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:02.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:02.049 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:02.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:02.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:02.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:01 smithi032 ceph-mon[27388]: pgmap v1407: 127 pgs: 127 active+clean; 95 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-16T05:23:02.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:01 smithi032 ceph-mon[27388]: osdmap e832: 12 total, 12 up, 12 in 2024-09-16T05:23:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:01 smithi204 ceph-mon[30933]: pgmap v1407: 127 pgs: 127 active+clean; 95 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-16T05:23:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:01 smithi204 ceph-mon[30933]: osdmap e832: 12 total, 12 up, 12 in 2024-09-16T05:23:02.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:02.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:02.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:02.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:01 smithi196 ceph-mon[30802]: pgmap v1407: 127 pgs: 127 active+clean; 95 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-09-16T05:23:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:01 smithi196 ceph-mon[30802]: osdmap e832: 12 total, 12 up, 12 in 2024-09-16T05:23:02.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:02.305 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:02.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:02.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:02.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:02.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:02.433 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:02.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:23:02.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:02.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:02.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:02.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:02.683 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:02.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:02.741 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:02.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:02.808 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:02.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:02.837 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:23:02.841 INFO:tasks.rados.rados.2.smithi204.stdout:3310: done (1 left) 2024-09-16T05:23:02.841 INFO:tasks.rados.rados.2.smithi204.stdout:3311: read oid 37 snap -1 2024-09-16T05:23:02.841 INFO:tasks.rados.rados.2.smithi204.stdout:3311: expect (ObjNum 1144 snap 341 seq_num 1144) 2024-09-16T05:23:02.841 INFO:tasks.rados.rados.2.smithi204.stdout:3312: read oid 11 snap 345 2024-09-16T05:23:02.841 INFO:tasks.rados.rados.2.smithi204.stdout:3312: expect (ObjNum 1149 snap 342 seq_num 1149) 2024-09-16T05:23:02.841 INFO:tasks.rados.rados.2.smithi204.stdout:3313: write (excl) oid 36 current snap is 348 2024-09-16T05:23:02.841 INFO:tasks.rados.rados.2.smithi204.stdout:3309: expect (ObjNum 935 snap 277 seq_num 935) 2024-09-16T05:23:02.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:02.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:02.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:03.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:03.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:03.062 INFO:tasks.rados.rados.2.smithi204.stdout:3312: expect (ObjNum 1149 snap 342 seq_num 1149) 2024-09-16T05:23:03.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2307557228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:02 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2307557228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:02 smithi032 ceph-mon[27388]: osdmap e833: 12 total, 12 up, 12 in 2024-09-16T05:23:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2307557228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:02 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:03.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2307557228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:03.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:02 smithi204 ceph-mon[30933]: osdmap e833: 12 total, 12 up, 12 in 2024-09-16T05:23:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:03.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:03.213 INFO:tasks.rados.rados.2.smithi204.stdout:3311: expect (ObjNum 1144 snap 341 seq_num 1144) 2024-09-16T05:23:03.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2307557228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:02 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2307557228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:02 smithi196 ceph-mon[30802]: osdmap e833: 12 total, 12 up, 12 in 2024-09-16T05:23:03.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:23:03.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:03.303 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:03.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:03.309 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:23:03.309 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:23:03.309 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:23:03.309 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:23:03.309 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:23:03.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:03.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:03.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:03.434 INFO:tasks.rados.rados.2.smithi204.stdout:3313: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2024-09-16T05:23:03.440 INFO:tasks.rados.rados.2.smithi204.stdout:3313: writing smithi20457400-36 from 662506 to 1086504 tid 1 2024-09-16T05:23:03.446 INFO:tasks.rados.rados.2.smithi204.stdout:3313: writing smithi20457400-36 from 1563560 to 2102967 tid 2 2024-09-16T05:23:03.452 INFO:tasks.rados.rados.2.smithi204.stdout:3313: writing smithi20457400-36 from 2717355 to 3207640 tid 3 2024-09-16T05:23:03.452 INFO:tasks.rados.rados.2.smithi204.stdout:3309: done (3 left) 2024-09-16T05:23:03.453 INFO:tasks.rados.rados.2.smithi204.stdout:3311: done (2 left) 2024-09-16T05:23:03.453 INFO:tasks.rados.rados.2.smithi204.stdout:3312: done (1 left) 2024-09-16T05:23:03.453 INFO:tasks.rados.rados.2.smithi204.stdout:3314: write oid 26 current snap is 348 2024-09-16T05:23:03.453 INFO:tasks.rados.rados.2.smithi204.stdout:3313: finishing write tid 1 to smithi20457400-36 2024-09-16T05:23:03.453 INFO:tasks.rados.rados.2.smithi204.stdout:3313: finishing write tid 2 to smithi20457400-36 2024-09-16T05:23:03.453 INFO:tasks.rados.rados.2.smithi204.stdout:3314: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2024-09-16T05:23:03.458 INFO:tasks.rados.rados.2.smithi204.stdout:3314: writing smithi20457400-26 from 679313 to 1178560 tid 1 2024-09-16T05:23:03.465 INFO:tasks.rados.rados.2.smithi204.stdout:3314: writing smithi20457400-26 from 1622042 to 2305107 tid 2 2024-09-16T05:23:03.471 INFO:tasks.rados.rados.2.smithi204.stdout:3314: writing smithi20457400-26 from 2744778 to 3446335 tid 3 2024-09-16T05:23:03.471 INFO:tasks.rados.rados.2.smithi204.stdout:3315: snap_remove snap 345 2024-09-16T05:23:03.527 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:03.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:03.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:03.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:03.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:03.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:03.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:03.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:03.806 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:03.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:03.831 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:03.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:03.928 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:03.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 2 pgp_num to 96 2024-09-16T05:23:03.959 INFO:tasks.rados.rados.2.smithi204.stdout:3315: done (2 left) 2024-09-16T05:23:03.959 INFO:tasks.rados.rados.2.smithi204.stdout:3316: read oid 43 snap -1 2024-09-16T05:23:03.959 INFO:tasks.rados.rados.2.smithi204.stdout:3316: expect (ObjNum 1110 snap 336 seq_num 1110) 2024-09-16T05:23:03.959 INFO:tasks.rados.rados.2.smithi204.stdout:3317: rollback oid 33 current snap is 348 2024-09-16T05:23:03.959 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 33 to 347 2024-09-16T05:23:03.959 INFO:tasks.rados.rados.2.smithi204.stdout:3313: finishing write tid 3 to smithi20457400-36 2024-09-16T05:23:03.959 INFO:tasks.rados.rados.2.smithi204.stdout:3313: finishing write tid 4 to smithi20457400-36 2024-09-16T05:23:03.959 INFO:tasks.rados.rados.2.smithi204.stdout:3313: finishing write tid 5 to smithi20457400-36 2024-09-16T05:23:03.960 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 794 (ObjNum 1158 snap 348 seq_num 1158) dirty exists 2024-09-16T05:23:03.960 INFO:tasks.rados.rados.2.smithi204.stdout:3313: left oid 36 (ObjNum 1158 snap 348 seq_num 1158) 2024-09-16T05:23:03.960 INFO:tasks.rados.rados.2.smithi204.stdout:3313: done (3 left) 2024-09-16T05:23:03.960 INFO:tasks.rados.rados.2.smithi204.stdout:3318: write (excl) oid 46 current snap is 348 2024-09-16T05:23:03.960 INFO:tasks.rados.rados.2.smithi204.stdout:3318: seq_num 1160 ranges {696120=574496,1680524=543076,2888554=512829} 2024-09-16T05:23:03.967 INFO:tasks.rados.rados.2.smithi204.stdout:3318: writing smithi20457400-46 from 696120 to 1270616 tid 1 2024-09-16T05:23:03.972 INFO:tasks.rados.rados.2.smithi204.stdout:3318: writing smithi20457400-46 from 1680524 to 2223600 tid 2 2024-09-16T05:23:03.980 INFO:tasks.rados.rados.2.smithi204.stdout:3318: writing smithi20457400-46 from 2888554 to 3401383 tid 3 2024-09-16T05:23:03.980 INFO:tasks.rados.rados.2.smithi204.stdout:3319: snap_create 2024-09-16T05:23:03.980 INFO:tasks.rados.rados.2.smithi204.stdout:3314: finishing write tid 1 to smithi20457400-26 2024-09-16T05:23:03.980 INFO:tasks.rados.rados.2.smithi204.stdout:3314: finishing write tid 2 to smithi20457400-26 2024-09-16T05:23:03.980 INFO:tasks.rados.rados.2.smithi204.stdout:3314: finishing write tid 3 to smithi20457400-26 2024-09-16T05:23:03.980 INFO:tasks.rados.rados.2.smithi204.stdout:3314: finishing write tid 4 to smithi20457400-26 2024-09-16T05:23:03.981 INFO:tasks.rados.rados.2.smithi204.stdout:3314: finishing write tid 5 to smithi20457400-26 2024-09-16T05:23:03.981 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1447 (ObjNum 1159 snap 348 seq_num 1159) dirty exists 2024-09-16T05:23:03.981 INFO:tasks.rados.rados.2.smithi204.stdout:3314: left oid 26 (ObjNum 1159 snap 348 seq_num 1159) 2024-09-16T05:23:03.981 INFO:tasks.rados.rados.2.smithi204.stdout:3317: finishing rollback tid 1 to smithi20457400-33 2024-09-16T05:23:03.981 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 129 snap 0 seq_num 4283112016) dirty dne 2024-09-16T05:23:03.981 INFO:tasks.rados.rados.2.smithi204.stdout:3316: expect (ObjNum 1110 snap 336 seq_num 1110) 2024-09-16T05:23:04.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:04.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:23:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:04.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:04.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:04.070 INFO:tasks.rados.rados.2.smithi204.stdout:3318: finishing write tid 1 to smithi20457400-46 2024-09-16T05:23:04.070 INFO:tasks.rados.rados.2.smithi204.stdout:3318: finishing write tid 2 to smithi20457400-46 2024-09-16T05:23:04.070 INFO:tasks.rados.rados.2.smithi204.stdout:3318: finishing write tid 3 to smithi20457400-46 2024-09-16T05:23:04.070 INFO:tasks.rados.rados.2.smithi204.stdout:3318: finishing write tid 4 to smithi20457400-46 2024-09-16T05:23:04.070 INFO:tasks.rados.rados.2.smithi204.stdout:3318: finishing write tid 5 to smithi20457400-46 2024-09-16T05:23:04.071 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1028 (ObjNum 1160 snap 348 seq_num 1160) dirty exists 2024-09-16T05:23:04.071 INFO:tasks.rados.rados.2.smithi204.stdout:3318: left oid 46 (ObjNum 1160 snap 348 seq_num 1160) 2024-09-16T05:23:04.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:04.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:04.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:03 smithi196 ceph-mon[30802]: pgmap v1410: 127 pgs: 127 active+clean; 95 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-16T05:23:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1072425096' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "96"}]: dispatch 2024-09-16T05:23:04.252 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:03 smithi204 ceph-mon[30933]: pgmap v1410: 127 pgs: 127 active+clean; 95 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-16T05:23:04.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1072425096' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "96"}]: dispatch 2024-09-16T05:23:04.281 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:04.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:04.305 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:04.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:04.316 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:04.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:03 smithi032 ceph-mon[27388]: pgmap v1410: 127 pgs: 127 active+clean; 95 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.7 MiB/s wr, 4 op/s 2024-09-16T05:23:04.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1072425096' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "96"}]: dispatch 2024-09-16T05:23:04.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:04.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:04.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:04.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:04.552 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:04.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:04.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:04.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:04.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:04.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:04.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:04.805 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:23:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:23:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:23:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:23:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2255 2.1d 2:ba754fcb:::smithi20457400-30:head [append 0~0] snapc 5f=[5f,5e,5d,5c,5a,59,58,57,52,4d,49,45,35] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-16T05:23:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:23:04.805 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 563.12157957199997, 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53925791899999997, 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2255 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:23:04.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681681+0000", 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7300000000000001e-07 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681684+0000", 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.582e-06 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681692+0000", 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3920000000000003e-06 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681698+0000", 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2100000000000001e-06 2024-09-16T05:23:04.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681779+0000", 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1003999999999999e-05 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.681782+0000", 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1190000000000001e-06 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684363+0000", 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025812399999999998 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684417+0000", 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3692e-05 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684503+0000", 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6592e-05 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:41.684818+0000", 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000314851 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220845+0000", 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.53602721099999995 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220887+0000", 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1514000000000003e-05 2024-09-16T05:23:04.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:42.220939+0000", 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2039000000000001e-05 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2559 2.1d 2:ba754fcb:::smithi20457400-30:head [copy-from ver 324] snapc 6d=[6c,6b,6a,69,67,66,64,63,62,60,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:23:04.812 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 499.84661539899997, 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.74015230600000004, 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2559 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956647+0000", 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2919999999999998e-06 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956655+0000", 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.729e-06 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956672+0000", 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6775000000000001e-05 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956685+0000", 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2832e-05 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956782+0000", 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6862999999999997e-05 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956790+0000", 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4020000000000002e-06 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958620+0000", 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001830415 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958718+0000", 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7708999999999996e-05 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.688352+0000", 2024-09-16T05:23:04.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72963416800000003 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.695488+0000", 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071356659999999997 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696731+0000", 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00124332 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696767+0000", 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6171000000000003e-05 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696797+0000", 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9964e-05 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.819 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2560 2.1d 2:ba754fcb:::smithi20457400-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:23:04.820 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 499.84654951900001, 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73643418699999996, 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2560 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956712+0000", 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.82e-07 2024-09-16T05:23:04.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956714+0000", 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6419999999999999e-06 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956737+0000", 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2668999999999999e-05 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956744+0000", 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.289e-06 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956847+0000", 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102729 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956855+0000", 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4009999999999997e-06 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958884+0000", 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020286409999999999 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958928+0000", 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4458e-05 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693004+0000", 2024-09-16T05:23:04.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73407526999999995 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693071+0000", 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7176999999999997e-05 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693145+0000", 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4328999999999999e-05 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.0.0:67 2.31 e356/279 2:8c1ba486:::smithi20457400-2:75 v 356'145, mlcod=350'141)", 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 479.09735890399998, 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.929470409, 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:23:04.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4499999999999998e-07 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705907+0000", 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8799999999999999e-06 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634756+0000", 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92884895999999995 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634764+0000", 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9589999999999995e-06 2024-09-16T05:23:04.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634827+0000", 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3280000000000004e-05 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634853+0000", 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6231999999999999e-05 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635342+0000", 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048899100000000004 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635372+0000", 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9762000000000001e-05 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3597 2.4c 2.118008cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:23:04.831 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 398.057142924, 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98230618599999997, 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3597 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746119+0000", 2024-09-16T05:23:04.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9999999999999995e-07 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746124+0000", 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3349999999999997e-06 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746136+0000", 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2339e-05 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746143+0000", 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3940000000000001e-06 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746201+0000", 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8529000000000002e-05 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746206+0000", 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0250000000000002e-06 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728413+0000", 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98220633700000004 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728424+0000", 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1226999999999999e-05 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3598 2.4c 2.118008cc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 398.05710135999999, 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.982592205, 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.837 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3598 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746160+0000", 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6899999999999997e-07 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746162+0000", 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.323e-06 2024-09-16T05:23:04.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746170+0000", 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4339999999999992e-06 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746176+0000", 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746227+0000", 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1848999999999998e-05 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746232+0000", 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5619999999999997e-06 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728741+0000", 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98250897100000001 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728751+0000", 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.10.0:319 2.4a e458/325 ondisk, result = 0)", 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 375.986788832, 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.733792216, 2024-09-16T05:23:04.842 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816473+0000", 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.206e-06 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816480+0000", 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4789999999999999e-06 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550063+0000", 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73358232899999998 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550069+0000", 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3489999999999997e-06 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550155+0000", 2024-09-16T05:23:04.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5490999999999999e-05 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550163+0000", 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1859999999999992e-06 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550264+0000", 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010117599999999999 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.846 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3966 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 645534~783707] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 366.91426206900002, 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.228671191, 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.847 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3966 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889000+0000", 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.745e-06 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889398+0000", 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000397644 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889415+0000", 2024-09-16T05:23:04.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7368e-05 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889440+0000", 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4358999999999998e-05 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889496+0000", 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6168999999999997e-05 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889500+0000", 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0579999999999999e-06 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107487+0000", 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21798717200000001 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107565+0000", 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7715999999999999e-05 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107734+0000", 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016950099999999999 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111344+0000", 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036100160000000002 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117576+0000", 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062321929999999996 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117628+0000", 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.189e-05 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117670+0000", 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1359999999999997e-05 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3967 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 2103939~638517] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 366.90733703900003, 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.221880365, 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.854 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3967 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895924+0000", 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.85e-07 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896200+0000", 2024-09-16T05:23:04.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027563199999999999 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896212+0000", 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1671999999999999e-05 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896217+0000", 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3970000000000002e-06 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896274+0000", 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6740000000000002e-05 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896280+0000", 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109156+0000", 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.212875278 2024-09-16T05:23:04.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109242+0000", 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6163999999999997e-05 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109378+0000", 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013637400000000001 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111350+0000", 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019716429999999999 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117716+0000", 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063661830000000001 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117776+0000", 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9753999999999997e-05 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117804+0000", 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.817e-05 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3968 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 3150663~294871] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 366.90365798300002, 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218327628, 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.861 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3968 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899603+0000", 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8800000000000002e-07 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899782+0000", 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017861999999999999 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899798+0000", 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5772000000000001e-05 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899806+0000", 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.157e-06 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899867+0000", 2024-09-16T05:23:04.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0711000000000002e-05 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899873+0000", 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0850000000000002e-06 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110009+0000", 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21013668299999999 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110084+0000", 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4176000000000003e-05 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.110205+0000", 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012158 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111356+0000", 2024-09-16T05:23:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001150646 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117849+0000", 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064928659999999999 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117895+0000", 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6873999999999999e-05 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117930+0000", 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4669999999999998e-05 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3970 2.1d 2:ba7e58cf:::smithi20457400-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 366.90340939399999, 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218352989, 2024-09-16T05:23:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3970 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899852+0000", 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8299999999999997e-07 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899854+0000", 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.463e-06 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899861+0000", 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0079999999999996e-06 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899866+0000", 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-16T05:23:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899949+0000", 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2713000000000006e-05 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899954+0000", 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7400000000000004e-06 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111261+0000", 2024-09-16T05:23:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.211307514 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111332+0000", 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1198999999999994e-05 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118056+0000", 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067238530000000001 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118121+0000", 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4770000000000005e-05 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118204+0000", 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3051000000000002e-05 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:4971 2.1d 2:ba7e58cf:::smithi20457400-26:head [stat,delete] snapc cc=[cb,ca,c9,c7,c5,c4,c3,c2,c1] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 301.60482795500002, 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.70425506000000004, 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4971 2024-09-16T05:23:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198434+0000", 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8300000000000001e-06 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198443+0000", 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198460+0000", 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7229999999999999e-05 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198467+0000", 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1409999999999999e-06 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198531+0000", 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3371e-05 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198539+0000", 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8930000000000005e-06 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891869+0000", 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69333044700000002 2024-09-16T05:23:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891964+0000", 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4990999999999999e-05 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892092+0000", 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012792599999999999 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892842+0000", 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075013 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902619+0000", 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0097773840000000001 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902658+0000", 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8167999999999998e-05 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902688+0000", 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9991000000000001e-05 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5379 2.1c 2:3a3554ba:::smithi20457400-48:head [setxattr _header (58) in=7b,truncate 2823741] snapc e0=[e0,df,dd,dc,db,da,d9,d8,d7,d4,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 264.00849378999999, 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46119776499999998, 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5379 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9500000000000002e-07 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794770+0000", 2024-09-16T05:23:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9890000000000001e-06 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794783+0000", 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2281e-05 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794788+0000", 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.011e-06 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795397+0000", 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060981000000000002 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795462+0000", 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4869000000000005e-05 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795576+0000", 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001141 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795836+0000", 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026010999999999997 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255870+0000", 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46003327399999999 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255915+0000", 2024-09-16T05:23:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5064000000000001e-05 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255964+0000", 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9662e-05 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5380 2.1c 2:3a3554ba:::smithi20457400-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 264.00845923600002, 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46133110599999999, 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5380 2024-09-16T05:23:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794802+0000", 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7599999999999998e-07 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794804+0000", 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0590000000000001e-06 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794810+0000", 2024-09-16T05:23:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794815+0000", 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6700000000000002e-06 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795747+0000", 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093155799999999997 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795821+0000", 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3975999999999998e-05 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256004+0000", 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46018367199999999 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256065+0000", 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0299e-05 2024-09-16T05:23:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256132+0000", 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7776999999999998e-05 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:5379 2.1c e571/532 ondisk, result = 0)", 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 264.00642555000002, 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45914334200000001, 2024-09-16T05:23:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0800000000000001e-07 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796837+0000", 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9850000000000001e-06 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255760+0000", 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45892265399999999 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255769+0000", 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3449999999999995e-06 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255857+0000", 2024-09-16T05:23:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7547999999999996e-05 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255865+0000", 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6810000000000004e-06 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255978+0000", 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112821 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5901 2.1d 2:ba754fcb:::smithi20457400-30:head [stat,delete] snapc f3=[f3,f2,f0,ef,ee,ed,ec,e8,e7,e5,e0,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 226.899953315, 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79482897200000002, 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5901 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903309+0000", 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4699999999999999e-06 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903314+0000", 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903330+0000", 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5770999999999999e-05 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903340+0000", 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0498e-05 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903402+0000", 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1385999999999998e-05 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903513+0000", 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111113 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903707+0000", 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019455500000000001 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.904477+0000", 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076932900000000002 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698051+0000", 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79357449800000002 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698095+0000", 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3452999999999999e-05 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698136+0000", 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1482e-05 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.7.0:1084 2.3c e626/625 2:3d4bedc8:::smithi20457400-22:ee v 626'652, mlcod=624'646)", 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 207.328999807, 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45487590700000002, 2024-09-16T05:23:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8800000000000002e-07 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474267+0000", 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928275+0000", 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45400790800000002 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928291+0000", 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5852999999999999e-05 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928455+0000", 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164026 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928479+0000", 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4258000000000001e-05 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929099+0000", 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000620117 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929137+0000", 2024-09-16T05:23:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7639999999999999e-05 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6377 2.1d 2:ba21f1a9:::smithi20457400-14:head [setxattr _header (58) in=7b,truncate 3059072] snapc 107=[107,106,104,102,101,fb,fa,f9,f7,f2,e7] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.916224481, 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46418828400000001, 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6377 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:23:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887037+0000", 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887040+0000", 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5039999999999998e-06 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887047+0000", 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0330000000000003e-06 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887053+0000", 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1430000000000003e-06 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888098+0000", 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001044866 2024-09-16T05:23:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888199+0000", 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101143 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888364+0000", 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164403 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888873+0000", 2024-09-16T05:23:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050974699999999996 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351123+0000", 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46224975499999998 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351169+0000", 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6316000000000002e-05 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351224+0000", 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4837999999999999e-05 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6378 2.1d 2:ba21f1a9:::smithi20457400-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 184.91619755299999, 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46436558900000002, 2024-09-16T05:23:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6378 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887064+0000", 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3799999999999997e-07 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887071+0000", 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9670000000000001e-06 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887077+0000", 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9739999999999999e-06 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887081+0000", 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.405e-06 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888760+0000", 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016788319999999999 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888855+0000", 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5670999999999999e-05 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351265+0000", 2024-09-16T05:23:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46241011799999998 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351353+0000", 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7416000000000005e-05 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351429+0000", 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5667999999999994e-05 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(1.7,GRANT,e:689,from:6,reservation_nonce:0)", 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 145.34594871600001, 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388634586, 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457313+0000", 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0020000000000001e-06 2024-09-16T05:23:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457317+0000", 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.279e-06 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845738+0000", 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388420879 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845745+0000", 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1849999999999998e-06 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845793+0000", 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8300000000000002e-05 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845814+0000", 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0353999999999999e-05 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845946+0000", 2024-09-16T05:23:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013258700000000001 2024-09-16T05:23:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:04.927 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:23:04.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:04.932 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:04.950 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:04.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:04.956 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:23:04.956 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:23:04.963 INFO:tasks.rados.rados.2.smithi204.stdout:3314: done (4 left) 2024-09-16T05:23:04.963 INFO:tasks.rados.rados.2.smithi204.stdout:3316: done (3 left) 2024-09-16T05:23:04.963 INFO:tasks.rados.rados.2.smithi204.stdout:3317: done (2 left) 2024-09-16T05:23:04.963 INFO:tasks.rados.rados.2.smithi204.stdout:3318: done (1 left) 2024-09-16T05:23:04.963 INFO:tasks.rados.rados.2.smithi204.stdout:3319: done (0 left) 2024-09-16T05:23:04.963 INFO:tasks.rados.rados.2.smithi204.stdout:3320: rollback oid 48 current snap is 349 2024-09-16T05:23:04.964 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 48 to 347 2024-09-16T05:23:04.964 INFO:tasks.rados.rados.2.smithi204.stdout:3321: read oid 13 snap 348 2024-09-16T05:23:04.964 INFO:tasks.rados.rados.2.smithi204.stdout:3321: expect deleted 2024-09-16T05:23:04.964 INFO:tasks.rados.rados.2.smithi204.stdout:3322: copy_from oid 23 from oid 35 current snap is 349 2024-09-16T05:23:04.964 INFO:tasks.rados.rados.2.smithi204.stdout:3323: delete oid 21 current snap is 349 2024-09-16T05:23:04.964 INFO:tasks.rados.rados.2.smithi204.stdout:3320: finishing rollback tid 1 to smithi20457400-48 2024-09-16T05:23:04.964 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-16T05:23:04.966 INFO:tasks.rados.rados.2.smithi204.stdout:3320: done (3 left) 2024-09-16T05:23:04.966 INFO:tasks.rados.rados.2.smithi204.stdout:3321: done (2 left) 2024-09-16T05:23:04.966 INFO:tasks.rados.rados.2.smithi204.stdout:3323: done (1 left) 2024-09-16T05:23:04.966 INFO:tasks.rados.rados.2.smithi204.stdout:3324: rollback oid 7 current snap is 349 2024-09-16T05:23:04.966 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 7 to 347 2024-09-16T05:23:04.967 INFO:tasks.rados.rados.2.smithi204.stdout:3325: rollback oid 49 current snap is 349 2024-09-16T05:23:04.967 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 348 2024-09-16T05:23:04.967 INFO:tasks.rados.rados.2.smithi204.stdout:3326: snap_create 2024-09-16T05:23:04.969 INFO:tasks.rados.rados.2.smithi204.stdout:3325: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:23:04.969 INFO:tasks.rados.rados.2.smithi204.stdout:3325: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:23:04.969 INFO:tasks.rados.rados.2.smithi204.stdout:3324: finishing rollback tid 0 to smithi20457400-7 2024-09-16T05:23:04.969 INFO:tasks.rados.rados.2.smithi204.stdout:3325: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:23:04.969 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 666 (ObjNum 921 snap 273 seq_num 921) dirty exists 2024-09-16T05:23:04.969 INFO:tasks.rados.rados.2.smithi204.stdout:3324: finishing rollback tid 1 to smithi20457400-7 2024-09-16T05:23:04.970 INFO:tasks.rados.rados.2.smithi204.stdout:3324: finishing rollback tid 2 to smithi20457400-7 2024-09-16T05:23:04.970 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1150 (ObjNum 1131 snap 338 seq_num 1131) dirty exists 2024-09-16T05:23:04.977 INFO:tasks.rados.rados.2.smithi204.stdout:3322: finishing copy_from racing read to smithi20457400-23 2024-09-16T05:23:04.978 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:04.978 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:04.981 INFO:tasks.rados.rados.2.smithi204.stdout:3322: finishing copy_from to smithi20457400-23 2024-09-16T05:23:04.981 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 862 (ObjNum 1127 snap 338 seq_num 1127) dirty exists 2024-09-16T05:23:04.998 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:04.998 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:05.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:05.019 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:05.041 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:05.042 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:05.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:05.063 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:05.063 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:05.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:05.085 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:05.107 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:05.107 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:05.128 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:05.129 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:05.150 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:05.150 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:05.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1072425096' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "96"}]': finished 2024-09-16T05:23:05.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:04 smithi204 ceph-mon[30933]: osdmap e834: 12 total, 12 up, 12 in 2024-09-16T05:23:05.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3290339814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:05.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:05.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:05.199 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:05.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1072425096' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "96"}]': finished 2024-09-16T05:23:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:04 smithi196 ceph-mon[30802]: osdmap e834: 12 total, 12 up, 12 in 2024-09-16T05:23:05.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3290339814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:05.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:05.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:05.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:05.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1072425096' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "96"}]': finished 2024-09-16T05:23:05.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:04 smithi032 ceph-mon[27388]: osdmap e834: 12 total, 12 up, 12 in 2024-09-16T05:23:05.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3290339814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:05.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:05.453 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:05.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:05.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:05.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:05.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:05.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:23:05.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:05.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:05.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:05.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:23:05.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:05.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:05.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:05.838 INFO:tasks.rados.rados.2.smithi204.stdout:3322: done (3 left) 2024-09-16T05:23:05.839 INFO:tasks.rados.rados.2.smithi204.stdout:3324: done (2 left) 2024-09-16T05:23:05.839 INFO:tasks.rados.rados.2.smithi204.stdout:3325: done (1 left) 2024-09-16T05:23:05.839 INFO:tasks.rados.rados.2.smithi204.stdout:3326: done (0 left) 2024-09-16T05:23:05.839 INFO:tasks.rados.rados.2.smithi204.stdout:3327: snap_remove snap 347 2024-09-16T05:23:05.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:05.936 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:05.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:05.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:06.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:06.082 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:06.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:06.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:05 smithi032 ceph-mon[27388]: pgmap v1413: 127 pgs: 127 active+clean; 96 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-16T05:23:06.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:05 smithi032 ceph-mon[27388]: osdmap e835: 12 total, 12 up, 12 in 2024-09-16T05:23:06.096 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:05 smithi032 ceph-mon[27388]: osdmap e836: 12 total, 12 up, 12 in 2024-09-16T05:23:06.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:06.206 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:06.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:05 smithi196 ceph-mon[30802]: pgmap v1413: 127 pgs: 127 active+clean; 96 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-16T05:23:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:05 smithi196 ceph-mon[30802]: osdmap e835: 12 total, 12 up, 12 in 2024-09-16T05:23:06.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:05 smithi196 ceph-mon[30802]: osdmap e836: 12 total, 12 up, 12 in 2024-09-16T05:23:06.307 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:05 smithi204 ceph-mon[30933]: pgmap v1413: 127 pgs: 127 active+clean; 96 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 9 op/s 2024-09-16T05:23:06.307 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:05 smithi204 ceph-mon[30933]: osdmap e835: 12 total, 12 up, 12 in 2024-09-16T05:23:06.307 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:05 smithi204 ceph-mon[30933]: osdmap e836: 12 total, 12 up, 12 in 2024-09-16T05:23:06.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:06.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:06.407 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:06.407 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:06.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:06.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:06.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:06.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:06.589 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:06.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:06.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:06.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:06.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:06.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:06.837 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:23:06.841 INFO:tasks.rados.rados.2.smithi204.stdout:3327: done (0 left) 2024-09-16T05:23:06.841 INFO:tasks.rados.rados.2.smithi204.stdout:3328: read oid 39 snap -1 2024-09-16T05:23:06.841 INFO:tasks.rados.rados.2.smithi204.stdout:3328: expect deleted 2024-09-16T05:23:06.841 INFO:tasks.rados.rados.2.smithi204.stdout:3329: delete oid 14 current snap is 350 2024-09-16T05:23:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:06.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:06.845 INFO:tasks.rados.rados.2.smithi204.stdout:3328: done (1 left) 2024-09-16T05:23:06.845 INFO:tasks.rados.rados.2.smithi204.stdout:3329: done (0 left) 2024-09-16T05:23:06.845 INFO:tasks.rados.rados.2.smithi204.stdout:3330: snap_create 2024-09-16T05:23:06.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:06.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:06.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:07.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1486106641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:07.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1486106641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:07.005 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:06 smithi204 ceph-mon[30933]: osdmap e837: 12 total, 12 up, 12 in 2024-09-16T05:23:07.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:23:07.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:07.099 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:07.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1486106641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1486106641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:06 smithi032 ceph-mon[27388]: osdmap e837: 12 total, 12 up, 12 in 2024-09-16T05:23:07.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:07.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:07.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:07.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1486106641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1486106641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:06 smithi196 ceph-mon[30802]: osdmap e837: 12 total, 12 up, 12 in 2024-09-16T05:23:07.275 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:07.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:07.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:07.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:07.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:07.475 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:07.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:07.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:07.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:07.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:07.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:07.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:07.725 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:07.725 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:07.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:07.842 INFO:tasks.rados.rados.2.smithi204.stdout:3330: done (0 left) 2024-09-16T05:23:07.842 INFO:tasks.rados.rados.2.smithi204.stdout:3331: snap_remove snap 349 2024-09-16T05:23:07.849 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:07.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:07.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:23:07.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:07.975 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:07.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:08.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:08.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:08.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:08.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:08.227 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:08.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:07 smithi196 ceph-mon[30802]: pgmap v1416: 127 pgs: 127 active+clean; 104 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-16T05:23:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:07 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:23:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:07 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:23:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:07 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:23:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:07 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:23:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:07 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:23:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:07 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:23:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:07 smithi196 ceph-mon[30802]: osdmap e838: 12 total, 12 up, 12 in 2024-09-16T05:23:08.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:07 smithi032 ceph-mon[27388]: pgmap v1416: 127 pgs: 127 active+clean; 104 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-16T05:23:08.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:07 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:23:08.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:07 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:23:08.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:07 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:23:08.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:07 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:23:08.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:07 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:23:08.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:07 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:23:08.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:07 smithi032 ceph-mon[27388]: osdmap e838: 12 total, 12 up, 12 in 2024-09-16T05:23:08.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:08.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:08.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:08.369 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:07 smithi204 ceph-mon[30933]: pgmap v1416: 127 pgs: 127 active+clean; 104 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2024-09-16T05:23:08.369 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:07 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:23:08.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:07 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:23:08.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:07 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:23:08.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:07 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:23:08.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:07 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:23:08.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:07 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:23:08.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:07 smithi204 ceph-mon[30933]: osdmap e838: 12 total, 12 up, 12 in 2024-09-16T05:23:08.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:23:08.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:08.477 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:08.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:08.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:08.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:08.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:08.639 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:08.682 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:08.682 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:08.703 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:23:08.704 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:23:08.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:08.724 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:08.724 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:08.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:08.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:08.744 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:08.744 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:08.766 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:08.766 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:08.787 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:08.809 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:08.809 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:08.830 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:08.831 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:08.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:08.843 INFO:tasks.rados.rados.2.smithi204.stdout:3331: done (0 left) 2024-09-16T05:23:08.844 INFO:tasks.rados.rados.2.smithi204.stdout:3332: copy_from oid 12 from oid 44 current snap is 351 2024-09-16T05:23:08.844 INFO:tasks.rados.rados.2.smithi204.stdout:3333: snap_create 2024-09-16T05:23:08.850 INFO:tasks.rados.rados.2.smithi204.stdout:3332: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:23:08.853 INFO:tasks.rados.rados.2.smithi204.stdout:3332: finishing copy_from to smithi20457400-12 2024-09-16T05:23:08.853 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1145 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:23:08.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:08.857 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:08.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:08.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:08.878 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:08.879 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:08.900 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:08.900 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:08.922 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:08.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:08.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:08.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:09.013 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: 2.56 scrub starts 2024-09-16T05:23:09.013 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: 2.56 scrub ok 2024-09-16T05:23:09.013 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:23:09.014 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:23:09.014 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:23:09.014 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:23:09.014 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: 2.24 scrub starts 2024-09-16T05:23:09.014 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: 2.24 scrub ok 2024-09-16T05:23:09.014 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3825100305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:09.014 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:08 smithi204 ceph-mon[30933]: osdmap e839: 12 total, 12 up, 12 in 2024-09-16T05:23:09.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:09.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:09.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: 2.56 scrub starts 2024-09-16T05:23:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: 2.56 scrub ok 2024-09-16T05:23:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:23:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:23:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:23:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:23:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: 2.24 scrub starts 2024-09-16T05:23:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: 2.24 scrub ok 2024-09-16T05:23:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3825100305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:09.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:08 smithi032 ceph-mon[27388]: osdmap e839: 12 total, 12 up, 12 in 2024-09-16T05:23:09.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:09.227 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:547 2.25 e278/252 2:a5bc1608:::smithi20457400-11:52 v 278'476, mlcod=277'470)", 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 579.49242760100003, 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27999907200000002, 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.228 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733368+0000", 2024-09-16T05:23:09.229 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4900000000000005e-07 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:29.733374+0000", 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0669999999999997e-06 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012754+0000", 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.27937933999999998 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012763+0000", 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.884e-06 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012833+0000", 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0779999999999997e-05 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.012856+0000", 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.283e-05 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.013333+0000", 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00047694000000000002 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:13:30.013367+0000", 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3482000000000002e-05 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.232 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:379 2.35 e346/275 2:ad4a88eb:::smithi20457400-16:6b v 346'436, mlcod=340'432)", 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 493.528707134, 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72722915899999996, 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697088+0000", 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1100000000000002e-07 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:55.697095+0000", 2024-09-16T05:23:09.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8709999999999999e-06 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423764+0000", 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72666896299999995 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423770+0000", 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0129999999999999e-06 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423829+0000", 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8807999999999997e-05 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.423855+0000", 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5721e-05 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.424282+0000", 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042752699999999997 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:56.424317+0000", 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4944999999999998e-05 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:414 2.35 e378/275 2:ad4a88eb:::smithi20457400-16:6b v 378'499, mlcod=377'495)", 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 461.48392681500002, 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.883103101, 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.237 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741868+0000", 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741869+0000", 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8099999999999999e-07 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:27.741872+0000", 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7979999999999999e-06 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624224+0000", 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.88235191599999996 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624240+0000", 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5296000000000001e-05 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624347+0000", 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107249 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624381+0000", 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4480999999999997e-05 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624917+0000", 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053571399999999996 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.624972+0000", 2024-09-16T05:23:09.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4466000000000003e-05 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3036 2.1b 2:da4f6049:::smithi20457400-49:head [setxattr _header (58) in=7b,truncate 3294238] snapc 7e=[7e,7d,7c,7b] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 460.458676153, 2024-09-16T05:23:09.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.437857206, 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: 2.56 scrub starts 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: 2.56 scrub ok 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: 2.24 scrub starts 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: 2.24 scrub ok 2024-09-16T05:23:09.243 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3825100305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:09.244 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:08 smithi196 ceph-mon[30802]: osdmap e839: 12 total, 12 up, 12 in 2024-09-16T05:23:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:09.244 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3036 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767119+0000", 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767120+0000", 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0800000000000004e-07 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767123+0000", 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.642e-06 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767132+0000", 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2819999999999996e-06 2024-09-16T05:23:09.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767138+0000", 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5200000000000003e-06 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767869+0000", 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00073052499999999999 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767959+0000", 2024-09-16T05:23:09.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0730999999999996e-05 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9", 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768106+0000", 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014667900000000001 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768798+0000", 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069156399999999998 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:09.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204863+0000", 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43606497999999999 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204909+0000", 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6758000000000002e-05 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.204976+0000", 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6816999999999994e-05 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.249 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3037 2.1b 2:da4f6049:::smithi20457400-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e379)", 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 460.45864440600002, 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43800830899999998, 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:09.250 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3037 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767151+0000", 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6199999999999998e-07 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767154+0000", 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6730000000000001e-06 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767161+0000", 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.849e-06 2024-09-16T05:23:09.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.767166+0000", 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4709999999999998e-06 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768487+0000", 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001320543 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:23:09.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:28.768584+0000", 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7547999999999995e-05 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205011+0000", 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43642700000000001 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205081+0000", 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9381000000000005e-05 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:29.205159+0000", 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8381999999999996e-05 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:3951 2.2f e467/275 2:f4b938a4:::smithi20457400-47:head v 467'381, mlcod=467'378)", 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 372.36642641399999, 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39001122199999999, 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.255 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859369+0000", 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6400000000000002e-07 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:56.859375+0000", 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1599999999999997e-06 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248747+0000", 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.38937218699999998 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248762+0000", 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.537e-05 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248857+0000", 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4630999999999996e-05 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.248883+0000", 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6448000000000002e-05 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249341+0000", 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000457486 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.249380+0000", 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9276000000000001e-05 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.262 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:500 2.35 e550/275 2:ad4a88eb:::smithi20457400-16:d3 v 550'597, mlcod=547'593)", 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:19.772586+0000", 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 289.45320969199997, 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.068775681000000005, 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.772586+0000", 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.772586+0000", 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.772586+0000", 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0100000000000001e-07 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.772593+0000", 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7959999999999998e-06 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.840636+0000", 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.068043812999999995 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.840645+0000", 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0750000000000004e-06 2024-09-16T05:23:09.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.840712+0000", 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6260999999999995e-05 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.840745+0000", 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3658999999999997e-05 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.841326+0000", 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00058029200000000005 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:19.841361+0000", 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5583999999999997e-05 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:521 2.35 e564/275 2:ad7afe2b:::smithi20457400-38:c5 v 564'628, mlcod=561'622)", 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:33.829005+0000", 2024-09-16T05:23:09.267 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 275.39679010100002, 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19962600699999999, 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:33.829005+0000", 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:33.829005+0000", 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:33.829006+0000", 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5300000000000002e-07 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:33.829013+0000", 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5339999999999996e-06 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.027927+0000", 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.19891482599999999 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.027941+0000", 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3913e-05 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.028031+0000", 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9858999999999995e-05 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.028051+0000", 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9726000000000001e-05 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.028589+0000", 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053826500000000001 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:34.028631+0000", 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1931000000000002e-05 2024-09-16T05:23:09.271 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:1041 2.25 e574/252 2:a49174eb:::smithi20457400-39:dc v 574'824, mlcod=572'821)", 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.783282+0000", 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 265.442513631, 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.203064307, 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.783282+0000", 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.783282+0000", 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.783282+0000", 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8900000000000001e-07 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.783286+0000", 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9829999999999998e-06 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.985535+0000", 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.202249346 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.985540+0000", 2024-09-16T05:23:09.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8810000000000004e-06 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.985588+0000", 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8241000000000001e-05 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.985606+0000", 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7227e-05 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.986312+0000", 2024-09-16T05:23:09.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00070611199999999995 2024-09-16T05:23:09.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:09.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:09.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:09.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.986346+0000", 2024-09-16T05:23:09.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4227999999999998e-05 2024-09-16T05:23:09.278 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.278 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.278 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:5460 2.25 e574/252 2:a5bc1608:::smithi20457400-11:head v 574'829, mlcod=574'824)", 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.995283+0000", 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 265.23051234600001, 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69040396400000004, 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995283+0000", 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995283+0000", 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995284+0000", 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9599999999999999e-07 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995289+0000", 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5709999999999997e-06 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.684770+0000", 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.68948120400000001 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.684792+0000", 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1500000000000001e-05 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.684936+0000", 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014458400000000001 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.684983+0000", 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6122999999999999e-05 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685633+0000", 2024-09-16T05:23:09.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000650615 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685687+0000", 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3671000000000002e-05 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6951 2.35 2:ad4a88eb:::smithi20457400-16:head [copy-from ver 1018] snapc 120=[120,11f,11e,11d,11b,119,118,116,113,10c,10b,108,f2] ondisk+write+known_if_redirected+supports_pool_eio e698)", 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:47.976824+0000", 2024-09-16T05:23:09.283 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 141.24897156899999, 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016999719999999999, 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6951 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976824+0000", 2024-09-16T05:23:09.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976824+0000", 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976825+0000", 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5960000000000001e-06 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976834+0000", 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1970000000000002e-06 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976852+0000", 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8026999999999999e-05 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976860+0000", 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5689999999999999e-06 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976939+0000", 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8893999999999994e-05 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.977047+0000", 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010775799999999999 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 10", 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.985484+0000", 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084370569999999995 2024-09-16T05:23:09.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.993343+0000", 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078589039999999999 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.993708+0000", 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000365752 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.993768+0000", 2024-09-16T05:23:09.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9867999999999997e-05 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.993823+0000", 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5098e-05 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6952 2.35 2:ad4a88eb:::smithi20457400-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e698)", 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:47.976880+0000", 2024-09-16T05:23:09.289 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 141.248915221, 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013156342, 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6952 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976880+0000", 2024-09-16T05:23:09.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976880+0000", 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976881+0000", 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976883+0000", 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5679999999999998e-06 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976897+0000", 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3559e-05 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.976903+0000", 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9440000000000002e-06 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.977185+0000", 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00028197399999999999 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.977224+0000", 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9551000000000001e-05 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.989863+0000", 2024-09-16T05:23:09.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01263923 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.989942+0000", 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8119000000000002e-05 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:47.990036+0000", 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4833999999999994e-05 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7406 2.1b 2:d8680303:::smithi20457400-43:head [copy-from ver 1037] snapc 13d=[13d,13c,13b,13a,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e753)", 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:42.818053+0000", 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 86.407741888000004, 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020181930000000001, 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7406 2024-09-16T05:23:09.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818053+0000", 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818053+0000", 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818055+0000", 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.858e-06 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818063+0000", 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1820000000000006e-06 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818077+0000", 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3909e-05 2024-09-16T05:23:09.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818085+0000", 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9109999999999993e-06 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818161+0000", 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5327999999999994e-05 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818167+0000", 2024-09-16T05:23:09.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.195e-06 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.820800+0000", 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002633196 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.820900+0000", 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010043 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9", 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.830921+0000", 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010020443 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.835343+0000", 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044221850000000004 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.838150+0000", 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028066440000000001 2024-09-16T05:23:09.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.838195+0000", 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5076e-05 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.838235+0000", 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0572999999999999e-05 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7407 2.1b 2:d8680303:::smithi20457400-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e753)", 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:42.818106+0000", 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 86.407688792000002, 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016685466999999999, 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7407 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818106+0000", 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818106+0000", 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818107+0000", 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2900000000000003e-07 2024-09-16T05:23:09.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818110+0000", 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9119999999999998e-06 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818118+0000", 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0579999999999998e-06 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818124+0000", 2024-09-16T05:23:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4150000000000004e-06 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818226+0000", 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010162600000000001 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.818232+0000", 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5369999999999997e-06 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.821131+0000", 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028996080000000001 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.821179+0000", 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7923999999999997e-05 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.834591+0000", 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013412076 2024-09-16T05:23:09.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.834701+0000", 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000109648 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:42.834792+0000", 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1033999999999996e-05 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.9.0:1394 2.2f e762/275 2:f6b419a1:::smithi20457400-37:13a v 762'658, mlcod=761'655)", 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:51.918605+0000", 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 77.307190403999996, 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.35210398199999998, 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:51.918605+0000", 2024-09-16T05:23:09.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:51.918605+0000", 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:51.918606+0000", 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9999999999999996e-07 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:51.918614+0000", 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2689999999999999e-06 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.269687+0000", 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.351073262 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.269697+0000", 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5310000000000007e-06 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.269766+0000", 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8917000000000004e-05 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.269792+0000", 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5976e-05 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.270666+0000", 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087391499999999996 2024-09-16T05:23:09.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.270709+0000", 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3211999999999999e-05 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7568 2.3e 2:7e4e65a4:::smithi20457400-4:head [copy-from ver 1041] snapc 13f=[13f,13d,13b,134,133,12c,12a,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.261590+0000", 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 76.964205543999995, 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012384367, 2024-09-16T05:23:09.312 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7568 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261590+0000", 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261590+0000", 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261592+0000", 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0480000000000001e-06 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261601+0000", 2024-09-16T05:23:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5279999999999992e-06 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261621+0000", 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9890000000000001e-05 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261631+0000", 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3500000000000003e-06 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261691+0000", 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0608e-05 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.261780+0000", 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8609999999999994e-05 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.267171+0000", 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053912819999999998 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.271918+0000", 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004746686 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.273857+0000", 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001939084 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.273922+0000", 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4955000000000004e-05 2024-09-16T05:23:09.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.273974+0000", 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2326e-05 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:7573 2.2f e762/275 2:f6b419a1:::smithi20457400-37:head v 762'662, mlcod=762'658)", 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.280222+0000", 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 76.945573730999996, 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21922077500000001, 2024-09-16T05:23:09.318 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280222+0000", 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280222+0000", 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280222+0000", 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9299999999999998e-07 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280227+0000", 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5399999999999997e-06 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.498773+0000", 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218545884 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.498781+0000", 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1780000000000003e-06 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.498869+0000", 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8764000000000006e-05 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.498891+0000", 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1588999999999999e-05 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.499404+0000", 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00051307000000000002 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.499442+0000", 2024-09-16T05:23:09.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8157000000000001e-05 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:1709 2.25 e799/252 2:a49174eb:::smithi20457400-39:149 v 799'1282, mlcod=795'1278)", 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:28.871715+0000", 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 40.354079900999999, 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26391335700000002, 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.323 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.871715+0000", 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.871715+0000", 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.871716+0000", 2024-09-16T05:23:09.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8400000000000005e-07 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.871731+0000", 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5429000000000001e-05 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.134747+0000", 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.26301555199999999 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.134760+0000", 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.329e-05 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.134882+0000", 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012176 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.134901+0000", 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.895e-05 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.135596+0000", 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069477199999999997 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.135629+0000", 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3120000000000001e-05 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:1710 2.25 e799/252 2:a5bc1608:::smithi20457400-11:145 v 799'1284, mlcod=795'1278)", 2024-09-16T05:23:09.327 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:28.872021+0000", 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 40.353773832000002, 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67472381400000003, 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.872021+0000", 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.872021+0000", 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.872022+0000", 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2900000000000002e-07 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.872030+0000", 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6420000000000007e-06 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.545740+0000", 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.67371039799999999 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.545769+0000", 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8237000000000001e-05 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.545860+0000", 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1558999999999995e-05 2024-09-16T05:23:09.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.545880+0000", 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0298000000000001e-05 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.546703+0000", 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00082228100000000001 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.546745+0000", 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2469999999999998e-05 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8254 2.a 2:5089ebe1:::smithi20457400-36:head [write 1836611~777404] snapc 159=[] ondisk+write+known_if_redirected+supports_pool_eio e823)", 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:52.878050+0000", 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 16.347745331999999, 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011970587, 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8254 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878050+0000", 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878050+0000", 2024-09-16T05:23:09.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878051+0000", 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000001e-07 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878392+0000", 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00034073699999999997 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878402+0000", 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0685e-05 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878408+0000", 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5600000000000001e-06 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878449+0000", 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1119999999999999e-05 2024-09-16T05:23:09.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878515+0000", 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5919000000000004e-05 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 3", 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.878616+0000", 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101389 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:23:09.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.880047+0000", 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001431007 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.889923+0000", 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098758679999999995 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.889976+0000", 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2516999999999999e-05 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:23:09.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:52.890020+0000", 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4920999999999999e-05 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:23:09.338 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:23:09.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:09.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:09.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:09.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:09.527 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:09.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:09.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:23:09.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:09.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:09.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:09.693 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:23:09.693 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:23:09.693 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T05:23:09.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:23:09.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:09.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:09.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:09.844 INFO:tasks.rados.rados.2.smithi204.stdout:3332: done (1 left) 2024-09-16T05:23:09.844 INFO:tasks.rados.rados.2.smithi204.stdout:3333: done (0 left) 2024-09-16T05:23:09.844 INFO:tasks.rados.rados.2.smithi204.stdout:3334: snap_create 2024-09-16T05:23:09.846 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:09.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:23:09.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:09.903 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:09.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:10.004 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:10.028 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:10.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:10.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:10.153 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:10.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: pgmap v1419: 127 pgs: 127 active+clean; 102 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: 2.5f scrub starts 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: 2.5f scrub ok 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: 2.45 scrub starts 2024-09-16T05:23:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: 2.45 scrub ok 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:09 smithi032 ceph-mon[27388]: osdmap e840: 12 total, 12 up, 12 in 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: pgmap v1419: 127 pgs: 127 active+clean; 102 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: 2.5f scrub starts 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: 2.5f scrub ok 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:23:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: 2.45 scrub starts 2024-09-16T05:23:10.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: 2.45 scrub ok 2024-09-16T05:23:10.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:09 smithi204 ceph-mon[30933]: osdmap e840: 12 total, 12 up, 12 in 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: pgmap v1419: 127 pgs: 127 active+clean; 102 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: 2.5f scrub starts 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: 2.5f scrub ok 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: 2.45 scrub starts 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: 2.45 scrub ok 2024-09-16T05:23:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:09 smithi196 ceph-mon[30802]: osdmap e840: 12 total, 12 up, 12 in 2024-09-16T05:23:10.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:10.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:10.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:10.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:10.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:10.409 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:10.413 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:10.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:10.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:10.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:10.631 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:10.653 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:10.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:10.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:10.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:10.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:10.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:23:10.841 INFO:tasks.rados.rados.2.smithi204.stdout:3334: done (0 left) 2024-09-16T05:23:10.841 INFO:tasks.rados.rados.2.smithi204.stdout:3335: write (excl) oid 18 current snap is 353 2024-09-16T05:23:10.841 INFO:tasks.rados.rados.2.smithi204.stdout:3335: seq_num 1163 ranges {746541=400243,1739617=690403,3138117=408424} 2024-09-16T05:23:10.846 INFO:tasks.rados.rados.2.smithi204.stdout:3335: writing smithi20457400-18 from 746541 to 1146784 tid 1 2024-09-16T05:23:10.854 INFO:tasks.rados.rados.2.smithi204.stdout:3335: writing smithi20457400-18 from 1739617 to 2430020 tid 2 2024-09-16T05:23:10.860 INFO:tasks.rados.rados.2.smithi204.stdout:3335: writing smithi20457400-18 from 3138117 to 3546541 tid 3 2024-09-16T05:23:10.860 INFO:tasks.rados.rados.2.smithi204.stdout:3336: snap_create 2024-09-16T05:23:10.860 INFO:tasks.rados.rados.2.smithi204.stdout:3335: finishing write tid 1 to smithi20457400-18 2024-09-16T05:23:10.860 INFO:tasks.rados.rados.2.smithi204.stdout:3335: finishing write tid 2 to smithi20457400-18 2024-09-16T05:23:10.862 INFO:tasks.rados.rados.2.smithi204.stdout:3335: finishing write tid 3 to smithi20457400-18 2024-09-16T05:23:10.862 INFO:tasks.rados.rados.2.smithi204.stdout:3335: finishing write tid 4 to smithi20457400-18 2024-09-16T05:23:10.862 INFO:tasks.rados.rados.2.smithi204.stdout:3335: finishing write tid 5 to smithi20457400-18 2024-09-16T05:23:10.862 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 754 (ObjNum 1163 snap 353 seq_num 1163) dirty exists 2024-09-16T05:23:10.863 INFO:tasks.rados.rados.2.smithi204.stdout:3335: left oid 18 (ObjNum 1163 snap 353 seq_num 1163) 2024-09-16T05:23:10.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:10.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:10.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:11.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:11.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:11.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:11.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:11.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:11.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2472250459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2472250459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:11.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:10 smithi032 ceph-mon[27388]: osdmap e841: 12 total, 12 up, 12 in 2024-09-16T05:23:11.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2472250459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2472250459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:10 smithi196 ceph-mon[30802]: osdmap e841: 12 total, 12 up, 12 in 2024-09-16T05:23:11.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:11.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:11.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:11.338 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:23:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:23:11.338 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:23:11.338 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:23:11.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:11.403 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:11.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:11.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:23:11.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:23:11.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:23:11.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:23:11.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2472250459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:11.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:11.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:23:11.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:23:11.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2472250459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:11.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:10 smithi204 ceph-mon[30933]: osdmap e841: 12 total, 12 up, 12 in 2024-09-16T05:23:11.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:11.531 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:11.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:11.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:11.656 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:11.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:11.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:23:11.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:11.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:11.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:11.843 INFO:tasks.rados.rados.2.smithi204.stdout:3335: done (1 left) 2024-09-16T05:23:11.843 INFO:tasks.rados.rados.2.smithi204.stdout:3336: done (0 left) 2024-09-16T05:23:11.843 INFO:tasks.rados.rados.2.smithi204.stdout:3337: copy_from oid 33 from oid 30 current snap is 354 2024-09-16T05:23:11.843 INFO:tasks.rados.rados.2.smithi204.stdout:3338: read oid 9 snap -1 2024-09-16T05:23:11.843 INFO:tasks.rados.rados.2.smithi204.stdout:3338: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:23:11.843 INFO:tasks.rados.rados.2.smithi204.stdout:3339: write (excl) oid 41 current snap is 354 2024-09-16T05:23:11.843 INFO:tasks.rados.rados.2.smithi204.stdout:3339: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-09-16T05:23:11.849 INFO:tasks.rados.rados.2.smithi204.stdout:3339: writing smithi20457400-41 from 763348 to 1355193 tid 1 2024-09-16T05:23:11.855 INFO:tasks.rados.rados.2.smithi204.stdout:3339: writing smithi20457400-41 from 1914452 to 2348513 tid 2 2024-09-16T05:23:11.862 INFO:tasks.rados.rados.2.smithi204.stdout:3339: writing smithi20457400-41 from 2881893 to 3556163 tid 3 2024-09-16T05:23:11.862 INFO:tasks.rados.rados.2.smithi204.stdout:3340: delete oid 10 current snap is 354 2024-09-16T05:23:11.862 INFO:tasks.rados.rados.2.smithi204.stdout:3337: finishing copy_from to smithi20457400-33 2024-09-16T05:23:11.862 INFO:tasks.rados.rados.2.smithi204.stdout:3337: got expected ENOENT (src dne) 2024-09-16T05:23:11.862 INFO:tasks.rados.rados.2.smithi204.stdout:3337: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:23:11.862 INFO:tasks.rados.rados.2.smithi204.stdout:3338: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:23:11.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:11.911 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:11.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:12.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:12.033 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:12.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:12.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:23:12.063 INFO:tasks.rados.rados.2.smithi204.stdout:3339: finishing write tid 1 to smithi20457400-41 2024-09-16T05:23:12.063 INFO:tasks.rados.rados.2.smithi204.stdout:3339: finishing write tid 2 to smithi20457400-41 2024-09-16T05:23:12.063 INFO:tasks.rados.rados.2.smithi204.stdout:3339: finishing write tid 3 to smithi20457400-41 2024-09-16T05:23:12.063 INFO:tasks.rados.rados.2.smithi204.stdout:3339: finishing write tid 4 to smithi20457400-41 2024-09-16T05:23:12.063 INFO:tasks.rados.rados.2.smithi204.stdout:3339: finishing write tid 5 to smithi20457400-41 2024-09-16T05:23:12.063 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1394 (ObjNum 1164 snap 354 seq_num 1164) dirty exists 2024-09-16T05:23:12.063 INFO:tasks.rados.rados.2.smithi204.stdout:3339: left oid 41 (ObjNum 1164 snap 354 seq_num 1164) 2024-09-16T05:23:12.063 INFO:tasks.rados.rados.2.smithi204.stdout:3337: done (3 left) 2024-09-16T05:23:12.064 INFO:tasks.rados.rados.2.smithi204.stdout:3338: done (2 left) 2024-09-16T05:23:12.064 INFO:tasks.rados.rados.2.smithi204.stdout:3339: done (1 left) 2024-09-16T05:23:12.064 INFO:tasks.rados.rados.2.smithi204.stdout:3340: done (0 left) 2024-09-16T05:23:12.064 INFO:tasks.rados.rados.2.smithi204.stdout:3341: delete oid 17 current snap is 354 2024-09-16T05:23:12.065 INFO:tasks.rados.rados.2.smithi204.stdout:3341: done (0 left) 2024-09-16T05:23:12.065 INFO:tasks.rados.rados.2.smithi204.stdout:3342: write (excl) oid 30 current snap is 354 2024-09-16T05:23:12.065 INFO:tasks.rados.rados.2.smithi204.stdout:3342: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2024-09-16T05:23:12.070 INFO:tasks.rados.rados.2.smithi204.stdout:3342: writing smithi20457400-30 from 413769 to 831361 tid 1 2024-09-16T05:23:12.077 INFO:tasks.rados.rados.2.smithi204.stdout:3342: writing smithi20457400-30 from 1573545 to 2271286 tid 2 2024-09-16T05:23:12.084 INFO:tasks.rados.rados.2.smithi204.stdout:3342: writing smithi20457400-30 from 2847809 to 3472248 tid 3 2024-09-16T05:23:12.084 INFO:tasks.rados.rados.2.smithi204.stdout:3343: read oid 48 snap -1 2024-09-16T05:23:12.084 INFO:tasks.rados.rados.2.smithi204.stdout:3343: expect deleted 2024-09-16T05:23:12.085 INFO:tasks.rados.rados.2.smithi204.stdout:3344: write oid 2 current snap is 354 2024-09-16T05:23:12.085 INFO:tasks.rados.rados.2.smithi204.stdout:3344: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-09-16T05:23:12.089 INFO:tasks.rados.rados.2.smithi204.stdout:3344: writing smithi20457400-2 from 430576 to 923417 tid 1 2024-09-16T05:23:12.095 INFO:tasks.rados.rados.2.smithi204.stdout:3344: writing smithi20457400-2 from 1632027 to 2073426 tid 2 2024-09-16T05:23:12.100 INFO:tasks.rados.rados.2.smithi204.stdout:3344: writing smithi20457400-2 from 2475232 to 2910943 tid 3 2024-09-16T05:23:12.104 INFO:tasks.rados.rados.2.smithi204.stdout:3344: writing smithi20457400-2 from 3365397 to 3630576 tid 4 2024-09-16T05:23:12.104 INFO:tasks.rados.rados.2.smithi204.stdout:3345: read oid 22 snap 353 2024-09-16T05:23:12.104 INFO:tasks.rados.rados.2.smithi204.stdout:3345: expect deleted 2024-09-16T05:23:12.104 INFO:tasks.rados.rados.2.smithi204.stdout:3346: snap_remove snap 350 2024-09-16T05:23:12.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:12.156 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:12.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:12.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:11 smithi032 ceph-mon[27388]: 2.3a scrub starts 2024-09-16T05:23:12.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:11 smithi032 ceph-mon[27388]: 2.3a scrub ok 2024-09-16T05:23:12.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:11 smithi032 ceph-mon[27388]: pgmap v1422: 127 pgs: 127 active+clean; 104 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-16T05:23:12.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:11 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:23:12.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:11 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:23:12.236 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:11 smithi032 ceph-mon[27388]: osdmap e842: 12 total, 12 up, 12 in 2024-09-16T05:23:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:11 smithi196 ceph-mon[30802]: 2.3a scrub starts 2024-09-16T05:23:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:11 smithi196 ceph-mon[30802]: 2.3a scrub ok 2024-09-16T05:23:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:11 smithi196 ceph-mon[30802]: pgmap v1422: 127 pgs: 127 active+clean; 104 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-16T05:23:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:11 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:23:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:11 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:23:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:11 smithi196 ceph-mon[30802]: osdmap e842: 12 total, 12 up, 12 in 2024-09-16T05:23:12.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:12.282 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:12.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:12.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:12.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:12.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:12.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:11 smithi204 ceph-mon[30933]: 2.3a scrub starts 2024-09-16T05:23:12.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:11 smithi204 ceph-mon[30933]: 2.3a scrub ok 2024-09-16T05:23:12.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:11 smithi204 ceph-mon[30933]: pgmap v1422: 127 pgs: 127 active+clean; 104 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.1 MiB/s wr, 4 op/s 2024-09-16T05:23:12.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:11 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:23:12.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:11 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:23:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:11 smithi204 ceph-mon[30933]: osdmap e842: 12 total, 12 up, 12 in 2024-09-16T05:23:12.423 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:23:12.423 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 842, 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:23:11.836536+0000", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:14:38.892129+0000", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:21:53.914532+0000", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:23:12.424 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:23:12.425 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:12.425 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 38, 2024-09-16T05:23:12.425 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:23:12.425 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:23:12.425 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:23:12.425 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:23:12.425 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:23:12.425 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:23:12.426 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 31, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 31, 2024-09-16T05:23:12.427 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 31, 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 31, 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 31, 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.428 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "687", 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "685", 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:23:12.429 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:23:12.430 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:23:12.431 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:23:12.432 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.3199999332427979, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.3199999332427979, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.3199999332427979, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.3199999332427979, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:23:12.433 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:23:12.434 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 96, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 96, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 96, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 96, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 96, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:23:12.435 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "842", 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "737", 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 357, 2024-09-16T05:23:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 842, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:23:12.437 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:23:12.438 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 1.5, 2024-09-16T05:23:12.439 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 1.5, 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 1.5, 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 1.5, 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:23:12.440 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 794, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.443 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.443 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:23:12.443 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:12.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.444 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.446 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:23:12.447 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 795, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.450 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:23:12.451 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.452 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:23:12.453 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 21, 2024-09-16T05:23:12.454 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 295, 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 329, 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 792, 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 296, 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:23:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:23:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3973436581", 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3973436581", 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3973436581", 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3973436581", 2024-09-16T05:23:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 792, 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:23:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:23:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 795, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:23:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 795, 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:23:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:23:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 767, 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:23:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:23:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 764, 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 794, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:23:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:23:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 764, 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:23:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 764, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:23:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 764, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:38.130007+0000", 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:53.256817+0000", 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:21:55.957222+0000", 2024-09-16T05:23:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 295 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:18.313133+0000", 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:23:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:57.887383+0000", 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:49.885060+0000", 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:44.205006+0000", 2024-09-16T05:23:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:36.577250+0000", 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:23:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:14.707362+0000", 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:16.519346+0000", 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:59.519632+0000", 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:26.588661+0000", 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:23:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:23:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 351, 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 353, 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2024-09-16T05:23:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:12.527 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:23:12.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:12.550 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:12.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:12.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:12.675 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:12.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:12.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:12.801 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:12.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:12.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:12.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:12.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:12.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:12.987 INFO:tasks.rados.rados.2.smithi204.stdout:3346: done (4 left) 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3347: delete oid 16 current snap is 354 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3342: finishing write tid 1 to smithi20457400-30 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3342: finishing write tid 2 to smithi20457400-30 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3342: finishing write tid 3 to smithi20457400-30 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3342: finishing write tid 4 to smithi20457400-30 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3342: finishing write tid 5 to smithi20457400-30 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 1459 (ObjNum 1167 snap 354 seq_num 1167) dirty exists 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3342: left oid 30 (ObjNum 1167 snap 354 seq_num 1167) 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3344: finishing write tid 1 to smithi20457400-2 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3344: finishing write tid 2 to smithi20457400-2 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3344: finishing write tid 3 to smithi20457400-2 2024-09-16T05:23:12.988 INFO:tasks.rados.rados.2.smithi204.stdout:3344: finishing write tid 4 to smithi20457400-2 2024-09-16T05:23:12.989 INFO:tasks.rados.rados.2.smithi204.stdout:3344: finishing write tid 5 to smithi20457400-2 2024-09-16T05:23:12.989 INFO:tasks.rados.rados.2.smithi204.stdout:3344: finishing write tid 6 to smithi20457400-2 2024-09-16T05:23:12.989 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 549 (ObjNum 1168 snap 354 seq_num 1168) dirty exists 2024-09-16T05:23:12.989 INFO:tasks.rados.rados.2.smithi204.stdout:3344: left oid 2 (ObjNum 1168 snap 354 seq_num 1168) 2024-09-16T05:23:12.991 INFO:tasks.rados.rados.2.smithi204.stdout:3342: done (4 left) 2024-09-16T05:23:12.992 INFO:tasks.rados.rados.2.smithi204.stdout:3343: done (3 left) 2024-09-16T05:23:12.992 INFO:tasks.rados.rados.2.smithi204.stdout:3344: done (2 left) 2024-09-16T05:23:12.992 INFO:tasks.rados.rados.2.smithi204.stdout:3345: done (1 left) 2024-09-16T05:23:12.992 INFO:tasks.rados.rados.2.smithi204.stdout:3347: done (0 left) 2024-09-16T05:23:12.992 INFO:tasks.rados.rados.2.smithi204.stdout:3348: snap_remove snap 353 2024-09-16T05:23:13.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:13.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:13.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/652440240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:23:13.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:12 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:23:13.134 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:12 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:23:13.154 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:13.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:13.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:13.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/652440240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:23:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:12 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:23:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:12 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:23:13.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:13.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:13.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:13.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:13.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/652440240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:23:13.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:12 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:23:13.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:12 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:23:13.437 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:13.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:13.533 INFO:tasks.thrashosds.thrasher:j is {'epoch': 842, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:23:11.836536+0000', 'last_up_change': '2024-09-16T05:14:38.892129+0000', 'last_in_change': '2024-09-16T05:21:53.914532+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 38, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '687', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '685', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 1, 'raw_score_acting': 2.319999933242798, 'raw_score_stable': 2.319999933242798, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 96, 'pg_placement_num': 96, 'pg_placement_num_target': 96, 'pg_num_target': 96, 'pg_num_pending': 96, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '842', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '737', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 357, 'snap_epoch': 842, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 794, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 795, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 21, 'last_clean_end': 295, 'up_from': 329, 'up_thru': 792, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3973436581}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3973436581}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3973436581}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3973436581}]}, 'public_addr': '172.21.15.32:6819/3973436581', 'cluster_addr': '172.21.15.32:6821/3973436581', 'heartbeat_back_addr': '172.21.15.32:6825/3973436581', 'heartbeat_front_addr': '172.21.15.32:6823/3973436581', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 795, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 795, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 767, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 794, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:38.130007+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:53.256817+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:21:55.957222+0000', 'dead_epoch': 295}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:18.313133+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:57.887383+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:49.885060+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:44.205006+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:36.577250+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:14.707362+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:16.519346+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:59.519632+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:26.588661+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 351, 'length': 1}, {'begin': 353, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:23:13.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:23:13.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:13.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:13.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:13.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:13.604 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:13.633 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:13.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:13.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:13.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:13.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:13.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:13.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:13.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:13.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:13.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:13.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:13.994 INFO:tasks.rados.rados.2.smithi204.stdout:3348: done (0 left) 2024-09-16T05:23:13.994 INFO:tasks.rados.rados.2.smithi204.stdout:3349: read oid 39 snap 348 2024-09-16T05:23:13.994 INFO:tasks.rados.rados.2.smithi204.stdout:3349: expect deleted 2024-09-16T05:23:13.994 INFO:tasks.rados.rados.2.smithi204.stdout:3350: snap_remove snap 352 2024-09-16T05:23:14.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:14.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:13 smithi032 ceph-mon[27388]: pgmap v1425: 127 pgs: 127 active+clean; 106 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:23:14.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:13 smithi032 ceph-mon[27388]: 2.53 scrub starts 2024-09-16T05:23:14.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:13 smithi032 ceph-mon[27388]: 2.53 scrub ok 2024-09-16T05:23:14.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:13 smithi032 ceph-mon[27388]: osdmap e843: 12 total, 12 up, 12 in 2024-09-16T05:23:14.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:13 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:23:14.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:13 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:23:14.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:13 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:23:14.112 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:13 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:23:14.112 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:14.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:14.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:14.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:14.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:13 smithi196 ceph-mon[30802]: pgmap v1425: 127 pgs: 127 active+clean; 106 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:23:14.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:13 smithi196 ceph-mon[30802]: 2.53 scrub starts 2024-09-16T05:23:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:13 smithi196 ceph-mon[30802]: 2.53 scrub ok 2024-09-16T05:23:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:13 smithi196 ceph-mon[30802]: osdmap e843: 12 total, 12 up, 12 in 2024-09-16T05:23:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:13 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:23:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:13 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:23:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:13 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:23:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:13 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:23:14.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:14.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:14.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:14.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:14.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:14.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:13 smithi204 ceph-mon[30933]: pgmap v1425: 127 pgs: 127 active+clean; 106 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-16T05:23:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:13 smithi204 ceph-mon[30933]: 2.53 scrub starts 2024-09-16T05:23:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:13 smithi204 ceph-mon[30933]: 2.53 scrub ok 2024-09-16T05:23:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:13 smithi204 ceph-mon[30933]: osdmap e843: 12 total, 12 up, 12 in 2024-09-16T05:23:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:13 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:23:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:13 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:23:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:13 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:23:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:13 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:23:14.440 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:14.441 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:14.462 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:23:14.462 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:23:14.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:14.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:14.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:14.575 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:23:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:23:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:23:14.575 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:23:14.575 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:23:14.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:14.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:14.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:14.684 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:14.685 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:14.705 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:14.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:14.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:14.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:14.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:23:14.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:14.888 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:14.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:14.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:14.931 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:14.969 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:14.969 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:14.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:14.991 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:14.992 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:15.000 INFO:tasks.rados.rados.2.smithi204.stdout:3350: done (1 left) 2024-09-16T05:23:15.000 INFO:tasks.rados.rados.2.smithi204.stdout:3351: read oid 30 snap -1 2024-09-16T05:23:15.000 INFO:tasks.rados.rados.2.smithi204.stdout:3351: expect (ObjNum 1167 snap 354 seq_num 1167) 2024-09-16T05:23:15.000 INFO:tasks.rados.rados.2.smithi204.stdout:3352: snap_create 2024-09-16T05:23:15.014 INFO:tasks.rados.rados.2.smithi204.stdout:3351: expect (ObjNum 1167 snap 354 seq_num 1167) 2024-09-16T05:23:15.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:15.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:15.040 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:15.040 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:15.061 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:15.083 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:15.084 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:15.105 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:15.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:15.141 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:15.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:15.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:15.267 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:15.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:15.296 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:14 smithi032 ceph-mon[27388]: osdmap e844: 12 total, 12 up, 12 in 2024-09-16T05:23:15.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/767811622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:15.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:23:15.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:15.390 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:15.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:14 smithi204 ceph-mon[30933]: osdmap e844: 12 total, 12 up, 12 in 2024-09-16T05:23:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/767811622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: osdmap e844: 12 total, 12 up, 12 in 2024-09-16T05:23:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/767811622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:15.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:15.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:15.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:15.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:15.638 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:15.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:15.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:15.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:15.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:15.842 INFO:tasks.rados.rados.2.smithi204.stdout:3349: done (2 left) 2024-09-16T05:23:15.842 INFO:tasks.rados.rados.2.smithi204.stdout:3351: done (1 left) 2024-09-16T05:23:15.842 INFO:tasks.rados.rados.2.smithi204.stdout:3352: done (0 left) 2024-09-16T05:23:15.842 INFO:tasks.rados.rados.2.smithi204.stdout:3353: read oid 41 snap -1 2024-09-16T05:23:15.842 INFO:tasks.rados.rados.2.smithi204.stdout:3353: expect (ObjNum 1164 snap 354 seq_num 1164) 2024-09-16T05:23:15.842 INFO:tasks.rados.rados.2.smithi204.stdout:3354: snap_create 2024-09-16T05:23:15.854 INFO:tasks.rados.rados.2.smithi204.stdout:3353: expect (ObjNum 1164 snap 354 seq_num 1164) 2024-09-16T05:23:15.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:15.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:15.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:16.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:16.064 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:16.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:16.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:16.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:16.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:16.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: pgmap v1428: 127 pgs: 127 active+clean; 108 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 781 KiB/s wr, 6 op/s 2024-09-16T05:23:16.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:23:16.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:23:16.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:23:16.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:23:16.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: osdmap e845: 12 total, 12 up, 12 in 2024-09-16T05:23:16.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: 1.17 scrub starts 2024-09-16T05:23:16.234 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: 1.17 scrub ok 2024-09-16T05:23:16.235 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:15 smithi032 ceph-mon[27388]: osdmap e846: 12 total, 12 up, 12 in 2024-09-16T05:23:16.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:16.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:16.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:16.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: pgmap v1428: 127 pgs: 127 active+clean; 108 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 781 KiB/s wr, 6 op/s 2024-09-16T05:23:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:23:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:23:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:23:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:23:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: osdmap e845: 12 total, 12 up, 12 in 2024-09-16T05:23:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: 1.17 scrub starts 2024-09-16T05:23:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: 1.17 scrub ok 2024-09-16T05:23:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:15 smithi204 ceph-mon[30933]: osdmap e846: 12 total, 12 up, 12 in 2024-09-16T05:23:16.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:16.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:16.477 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:16.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: pgmap v1428: 127 pgs: 127 active+clean; 108 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 781 KiB/s wr, 6 op/s 2024-09-16T05:23:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:23:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:23:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:23:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:23:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: osdmap e845: 12 total, 12 up, 12 in 2024-09-16T05:23:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: 1.17 scrub starts 2024-09-16T05:23:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: 1.17 scrub ok 2024-09-16T05:23:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:15 smithi196 ceph-mon[30802]: osdmap e846: 12 total, 12 up, 12 in 2024-09-16T05:23:16.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:16.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:16.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:16.661 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:23:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:23:16.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:16.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:16.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:16.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:16.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:16.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:16.844 INFO:tasks.rados.rados.2.smithi204.stdout:3353: done (1 left) 2024-09-16T05:23:16.844 INFO:tasks.rados.rados.2.smithi204.stdout:3354: done (0 left) 2024-09-16T05:23:16.844 INFO:tasks.rados.rados.2.smithi204.stdout:3355: copy_from oid 21 from oid 44 current snap is 356 2024-09-16T05:23:16.844 INFO:tasks.rados.rados.2.smithi204.stdout:3356: read oid 49 snap -1 2024-09-16T05:23:16.845 INFO:tasks.rados.rados.2.smithi204.stdout:3356: expect (ObjNum 921 snap 273 seq_num 921) 2024-09-16T05:23:16.845 INFO:tasks.rados.rados.2.smithi204.stdout:3357: copy_from oid 41 from oid 47 current snap is 356 2024-09-16T05:23:16.845 INFO:tasks.rados.rados.2.smithi204.stdout:3358: rollback oid 50 current snap is 356 2024-09-16T05:23:16.845 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 351 2024-09-16T05:23:16.845 INFO:tasks.rados.rados.2.smithi204.stdout:3359: read oid 36 snap -1 2024-09-16T05:23:16.845 INFO:tasks.rados.rados.2.smithi204.stdout:3359: expect (ObjNum 1158 snap 348 seq_num 1158) 2024-09-16T05:23:16.845 INFO:tasks.rados.rados.2.smithi204.stdout:3360: delete oid 15 current snap is 356 2024-09-16T05:23:16.849 INFO:tasks.rados.rados.2.smithi204.stdout:3358: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:23:16.849 INFO:tasks.rados.rados.2.smithi204.stdout:3358: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:23:16.850 INFO:tasks.rados.rados.2.smithi204.stdout:3358: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:23:16.850 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1153 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:23:16.854 INFO:tasks.rados.rados.2.smithi204.stdout:3359: expect (ObjNum 1158 snap 348 seq_num 1158) 2024-09-16T05:23:16.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:16.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:16.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:17.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:17.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:17.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:17.126 INFO:tasks.rados.rados.2.smithi204.stdout:3355: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:23:17.126 INFO:tasks.rados.rados.2.smithi204.stdout:3357: finishing copy_from racing read to smithi20457400-41 2024-09-16T05:23:17.126 INFO:tasks.rados.rados.2.smithi204.stdout:3356: expect (ObjNum 921 snap 273 seq_num 921) 2024-09-16T05:23:17.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:17.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:17.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:17.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:17.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:17.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:17.353 INFO:tasks.rados.rados.2.smithi204.stdout:3355: finishing copy_from to smithi20457400-21 2024-09-16T05:23:17.353 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 1145 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:23:17.354 INFO:tasks.rados.rados.2.smithi204.stdout:3357: finishing copy_from to smithi20457400-41 2024-09-16T05:23:17.354 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1062 (ObjNum 1142 snap 341 seq_num 1142) dirty exists 2024-09-16T05:23:17.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:23:17.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:23:17.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:23:17.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:23:17.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: 2.43 scrub starts 2024-09-16T05:23:17.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: 2.43 scrub ok 2024-09-16T05:23:17.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: 2.5e scrub starts 2024-09-16T05:23:17.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: 2.5e scrub ok 2024-09-16T05:23:17.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:17 smithi032 ceph-mon[27388]: osdmap e847: 12 total, 12 up, 12 in 2024-09-16T05:23:17.415 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1431,"stamp":"2024-09-16T05:23:16.221031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118309528,"num_objects":46,"num_object_clones":8,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":3,"num_read":3700,"num_read_kb":3695807,"num_write":2605,"num_write_kb":1079974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37626,"ondisk_log_size":37626,"up":254,"acting":254,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":254,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":845612,"kb_used_data":274492,"kb_used_omap":27,"kb_used_meta":570788,"kb_avail":1124096212,"statfs":{"total":1151940427776,"available":1151074521088,"internally_reserved":0,"allocated":281079808,"data_stored":252329582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":584487079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7358557,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":25,"num_read_kb":32180,"num_write":17,"num_write_kb":4960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000767"},"pg_stats":[{"pgid":"2.5f","version":"284'221","reported_seq":3937,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.004734+0000","last_change":"2024-09-16T05:23:15.004973+0000","last_active":"2024-09-16T05:23:15.004734+0000","last_peered":"2024-09-16T05:23:15.004734+0000","last_clean":"2024-09-16T05:23:15.004734+0000","last_became_active":"2024-09-16T05:21:54.939189+0000","last_became_peered":"2024-09-16T05:21:54.939189+0000","last_unstale":"2024-09-16T05:23:15.004734+0000","last_undegraded":"2024-09-16T05:23:15.004734+0000","last_fullsized":"2024-09-16T05:23:15.004734+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:23:08.674041+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:23:08.674041+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:11.629531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000450772,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.5e","version":"823'562","reported_seq":4408,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.001614+0000","last_change":"2024-09-16T05:23:15.001851+0000","last_active":"2024-09-16T05:23:15.001614+0000","last_peered":"2024-09-16T05:23:15.001614+0000","last_clean":"2024-09-16T05:23:15.001614+0000","last_became_active":"2024-09-16T05:21:54.937455+0000","last_became_peered":"2024-09-16T05:21:54.937455+0000","last_unstale":"2024-09-16T05:23:15.001614+0000","last_undegraded":"2024-09-16T05:23:15.001614+0000","last_fullsized":"2024-09-16T05:23:15.001614+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"743'518","last_scrub_stamp":"2024-09-16T05:21:40.979621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:40.979621+0000","objects_scrubbed":3,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013167999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":13007,"num_write":7,"num_write_kb":989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4819607,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.5d","version":"842'1459","reported_seq":5410,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.007873+0000","last_change":"2024-09-16T05:23:15.008161+0000","last_active":"2024-09-16T05:23:15.007873+0000","last_peered":"2024-09-16T05:23:15.007873+0000","last_clean":"2024-09-16T05:23:15.007873+0000","last_became_active":"2024-09-16T05:21:32.830157+0000","last_became_peered":"2024-09-16T05:21:32.830157+0000","last_unstale":"2024-09-16T05:23:15.007873+0000","last_undegraded":"2024-09-16T05:23:15.007873+0000","last_fullsized":"2024-09-16T05:23:15.007873+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'1436","last_scrub_stamp":"2024-09-16T05:22:51.640147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.640147+0000","objects_scrubbed":6,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:57.407913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0052656040000000001,"stat_sum":{"num_bytes":9244642,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":80,"num_read_kb":77936,"num_write":35,"num_write_kb":17177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.5c","version":"823'484","reported_seq":3815,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:14.997651+0000","last_change":"2024-09-16T05:23:14.997827+0000","last_active":"2024-09-16T05:23:14.997651+0000","last_peered":"2024-09-16T05:23:14.997651+0000","last_clean":"2024-09-16T05:23:14.997651+0000","last_became_active":"2024-09-16T05:21:32.828660+0000","last_became_peered":"2024-09-16T05:21:32.828660+0000","last_unstale":"2024-09-16T05:23:14.997651+0000","last_undegraded":"2024-09-16T05:23:14.997651+0000","last_fullsized":"2024-09-16T05:23:14.997651+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'482","last_scrub_stamp":"2024-09-16T05:22:52.859116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:52.859116+0000","objects_scrubbed":2,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:19.754555+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0.00073972199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7420,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.5b","version":"837'668","reported_seq":4372,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.002274+0000","last_change":"2024-09-16T05:23:15.002727+0000","last_active":"2024-09-16T05:23:15.002274+0000","last_peered":"2024-09-16T05:23:15.002274+0000","last_clean":"2024-09-16T05:23:15.002274+0000","last_became_active":"2024-09-16T05:21:32.829681+0000","last_became_peered":"2024-09-16T05:21:32.829681+0000","last_unstale":"2024-09-16T05:23:15.002274+0000","last_undegraded":"2024-09-16T05:23:15.002274+0000","last_fullsized":"2024-09-16T05:23:15.002274+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'660","last_scrub_stamp":"2024-09-16T05:22:52.654103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:52.654103+0000","objects_scrubbed":2,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:16.304659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061513900000000001,"stat_sum":{"num_bytes":3479247,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":32170,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.5a","version":"823'1224","reported_seq":4854,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.853482+0000","last_change":"2024-09-16T05:23:09.037121+0000","last_active":"2024-09-16T05:23:11.853482+0000","last_peered":"2024-09-16T05:23:11.853482+0000","last_clean":"2024-09-16T05:23:11.853482+0000","last_became_active":"2024-09-16T05:21:32.826884+0000","last_became_peered":"2024-09-16T05:21:32.826884+0000","last_unstale":"2024-09-16T05:23:11.853482+0000","last_undegraded":"2024-09-16T05:23:11.853482+0000","last_fullsized":"2024-09-16T05:23:11.853482+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"823'1224","last_scrub_stamp":"2024-09-16T05:22:54.387969+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.387969+0000","objects_scrubbed":1,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:13.938043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013227359999999999,"stat_sum":{"num_bytes":2638963,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":21681,"num_write":24,"num_write_kb":6390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.59","version":"182'60","reported_seq":551,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.866591+0000","last_change":"2024-09-16T05:23:09.039170+0000","last_active":"2024-09-16T05:23:11.866591+0000","last_peered":"2024-09-16T05:23:11.866591+0000","last_clean":"2024-09-16T05:23:11.866591+0000","last_became_active":"2024-09-16T05:21:54.931981+0000","last_became_peered":"2024-09-16T05:21:54.931981+0000","last_unstale":"2024-09-16T05:23:11.866591+0000","last_undegraded":"2024-09-16T05:23:11.866591+0000","last_fullsized":"2024-09-16T05:23:11.866591+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:35.068053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.068053+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:53.421358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033405739999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.58","version":"181'57","reported_seq":3715,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.185511+0000","last_change":"2024-09-16T05:23:15.186503+0000","last_active":"2024-09-16T05:23:15.185511+0000","last_peered":"2024-09-16T05:23:15.185511+0000","last_clean":"2024-09-16T05:23:15.185511+0000","last_became_active":"2024-09-16T05:21:29.826747+0000","last_became_peered":"2024-09-16T05:21:29.826747+0000","last_unstale":"2024-09-16T05:23:15.185511+0000","last_undegraded":"2024-09-16T05:23:15.185511+0000","last_fullsized":"2024-09-16T05:23:15.185511+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:22:56.040114+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:22:56.040114+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:08.404445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0037104030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.57","version":"0'0","reported_seq":1253,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.001897+0000","last_change":"2024-09-16T05:23:15.002114+0000","last_active":"2024-09-16T05:23:15.001897+0000","last_peered":"2024-09-16T05:23:15.001897+0000","last_clean":"2024-09-16T05:23:15.001897+0000","last_became_active":"2024-09-16T05:21:29.827377+0000","last_became_peered":"2024-09-16T05:21:29.827377+0000","last_unstale":"2024-09-16T05:23:15.001897+0000","last_undegraded":"2024-09-16T05:23:15.001897+0000","last_fullsized":"2024-09-16T05:23:15.001897+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:48.915248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.915248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:05.028858+0000","scrub_duration":557,"objects_trimmed":0,"snaptrim_duration":0.0011977789999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.56","version":"834'390","reported_seq":3742,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:14.997880+0000","last_change":"2024-09-16T05:23:14.998055+0000","last_active":"2024-09-16T05:23:14.997880+0000","last_peered":"2024-09-16T05:23:14.997880+0000","last_clean":"2024-09-16T05:23:14.997880+0000","last_became_active":"2024-09-16T05:21:32.829699+0000","last_became_peered":"2024-09-16T05:21:32.829699+0000","last_unstale":"2024-09-16T05:23:14.997880+0000","last_undegraded":"2024-09-16T05:23:14.997880+0000","last_fullsized":"2024-09-16T05:23:14.997880+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"834'390","last_scrub_stamp":"2024-09-16T05:23:07.704997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.704997+0000","objects_scrubbed":1,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:09.382167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000884963,"stat_sum":{"num_bytes":3496120,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":18485,"num_write":18,"num_write_kb":5070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.55","version":"433'507","reported_seq":2688,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.853646+0000","last_change":"2024-09-16T05:23:09.037999+0000","last_active":"2024-09-16T05:23:11.853646+0000","last_peered":"2024-09-16T05:23:11.853646+0000","last_clean":"2024-09-16T05:23:11.853646+0000","last_became_active":"2024-09-16T05:21:24.815957+0000","last_became_peered":"2024-09-16T05:21:24.815957+0000","last_unstale":"2024-09-16T05:23:11.853646+0000","last_undegraded":"2024-09-16T05:23:11.853646+0000","last_fullsized":"2024-09-16T05:23:11.853646+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:22:53.432495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:53.432495+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:07.992138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019198729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.54","version":"837'448","reported_seq":3757,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839694+0000","last_change":"2024-09-16T05:23:09.081198+0000","last_active":"2024-09-16T05:23:11.839694+0000","last_peered":"2024-09-16T05:23:11.839694+0000","last_clean":"2024-09-16T05:23:11.839694+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:23:11.839694+0000","last_undegraded":"2024-09-16T05:23:11.839694+0000","last_fullsized":"2024-09-16T05:23:11.839694+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"765'422","last_scrub_stamp":"2024-09-16T05:22:01.692016+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:22:01.692016+0000","objects_scrubbed":1,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:29.786455+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00082727400000000002,"stat_sum":{"num_bytes":3714545,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":73,"num_read_kb":63622,"num_write":53,"num_write_kb":22446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":2077,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.004673+0000","last_change":"2024-09-16T05:23:15.004908+0000","last_active":"2024-09-16T05:23:15.004673+0000","last_peered":"2024-09-16T05:23:15.004673+0000","last_clean":"2024-09-16T05:23:15.004673+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:23:15.004673+0000","last_undegraded":"2024-09-16T05:23:15.004673+0000","last_fullsized":"2024-09-16T05:23:15.004673+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:23:12.688624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:12.688624+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:41.761292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041085900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.52","version":"272'69","reported_seq":2074,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:14.997256+0000","last_change":"2024-09-16T05:23:14.997493+0000","last_active":"2024-09-16T05:23:14.997256+0000","last_peered":"2024-09-16T05:23:14.997256+0000","last_clean":"2024-09-16T05:23:14.997256+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:23:14.997256+0000","last_undegraded":"2024-09-16T05:23:14.997256+0000","last_fullsized":"2024-09-16T05:23:14.997256+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:22:55.815233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:55.815233+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:19.907565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060989499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.51","version":"272'115","reported_seq":2139,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.001162+0000","last_change":"2024-09-16T05:23:15.001439+0000","last_active":"2024-09-16T05:23:15.001162+0000","last_peered":"2024-09-16T05:23:15.001162+0000","last_clean":"2024-09-16T05:23:15.001162+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:23:15.001162+0000","last_undegraded":"2024-09-16T05:23:15.001162+0000","last_fullsized":"2024-09-16T05:23:15.001162+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_deep_scrub":"272'115","last_deep_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_clean_scrub_stamp":"2024-09-16T05:22:11.104571+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:17.910019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010584220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.50","version":"182'16","reported_seq":2041,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.853541+0000","last_change":"2024-09-16T05:23:09.037716+0000","last_active":"2024-09-16T05:23:11.853541+0000","last_peered":"2024-09-16T05:23:11.853541+0000","last_clean":"2024-09-16T05:23:11.853541+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:23:11.853541+0000","last_undegraded":"2024-09-16T05:23:11.853541+0000","last_fullsized":"2024-09-16T05:23:11.853541+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:22:25.344602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.344602+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:36.542038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001717554,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2601,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.003467+0000","last_change":"2024-09-16T05:23:15.003760+0000","last_active":"2024-09-16T05:23:15.003467+0000","last_peered":"2024-09-16T05:23:15.003467+0000","last_clean":"2024-09-16T05:23:15.003467+0000","last_became_active":"2024-09-16T05:21:10.817392+0000","last_became_peered":"2024-09-16T05:21:10.817392+0000","last_unstale":"2024-09-16T05:23:15.003467+0000","last_undegraded":"2024-09-16T05:23:15.003467+0000","last_fullsized":"2024-09-16T05:23:15.003467+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:22:33.493608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:33.493608+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:56.966508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001032135,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.4e","version":"272'232","reported_seq":3835,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.183949+0000","last_change":"2024-09-16T05:23:15.184258+0000","last_active":"2024-09-16T05:23:15.183949+0000","last_peered":"2024-09-16T05:23:15.183949+0000","last_clean":"2024-09-16T05:23:15.183949+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:23:15.183949+0000","last_undegraded":"2024-09-16T05:23:15.183949+0000","last_fullsized":"2024-09-16T05:23:15.183949+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:22:49.748809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.748809+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:09.443205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098934000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.4d","version":"829'732","reported_seq":4476,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839362+0000","last_change":"2024-09-16T05:23:09.081140+0000","last_active":"2024-09-16T05:23:11.839362+0000","last_peered":"2024-09-16T05:23:11.839362+0000","last_clean":"2024-09-16T05:23:11.839362+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:23:11.839362+0000","last_undegraded":"2024-09-16T05:23:11.839362+0000","last_fullsized":"2024-09-16T05:23:11.839362+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"807'708","last_scrub_stamp":"2024-09-16T05:22:37.651060+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.651060+0000","objects_scrubbed":4,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:53.541816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077219400000000003,"stat_sum":{"num_bytes":2790226,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":38970,"num_write":88,"num_write_kb":29604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.4c","version":"823'428","reported_seq":4144,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.001305+0000","last_change":"2024-09-16T05:23:15.001558+0000","last_active":"2024-09-16T05:23:15.001305+0000","last_peered":"2024-09-16T05:23:15.001305+0000","last_clean":"2024-09-16T05:23:15.001305+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:23:15.001305+0000","last_undegraded":"2024-09-16T05:23:15.001305+0000","last_fullsized":"2024-09-16T05:23:15.001305+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"773'408","last_scrub_stamp":"2024-09-16T05:22:03.012954+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:22:03.012954+0000","objects_scrubbed":3,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:15.034971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011454600000000001,"stat_sum":{"num_bytes":3328050,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":98,"num_read_kb":121159,"num_write":45,"num_write_kb":14823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.4b","version":"244'177","reported_seq":3855,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839427+0000","last_change":"2024-09-16T05:23:09.081393+0000","last_active":"2024-09-16T05:23:11.839427+0000","last_peered":"2024-09-16T05:23:11.839427+0000","last_clean":"2024-09-16T05:23:11.839427+0000","last_became_active":"2024-09-16T05:22:22.832492+0000","last_became_peered":"2024-09-16T05:22:22.832492+0000","last_unstale":"2024-09-16T05:23:11.839427+0000","last_undegraded":"2024-09-16T05:23:11.839427+0000","last_fullsized":"2024-09-16T05:23:11.839427+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:22:10.688513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.688513+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:16.655166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001043499,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.4a","version":"823'663","reported_seq":4343,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.002195+0000","last_change":"2024-09-16T05:23:15.002405+0000","last_active":"2024-09-16T05:23:15.002195+0000","last_peered":"2024-09-16T05:23:15.002195+0000","last_clean":"2024-09-16T05:23:15.002195+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:23:15.002195+0000","last_undegraded":"2024-09-16T05:23:15.002195+0000","last_fullsized":"2024-09-16T05:23:15.002195+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'661","last_scrub_stamp":"2024-09-16T05:22:52.300466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:52.300466+0000","objects_scrubbed":2,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:17.172204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00162685,"stat_sum":{"num_bytes":2941489,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":55014,"num_write":59,"num_write_kb":25438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.49","version":"245'220","reported_seq":3912,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184716+0000","last_change":"2024-09-16T05:23:15.185134+0000","last_active":"2024-09-16T05:23:15.184716+0000","last_peered":"2024-09-16T05:23:15.184716+0000","last_clean":"2024-09-16T05:23:15.184716+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:23:15.184716+0000","last_undegraded":"2024-09-16T05:23:15.184716+0000","last_fullsized":"2024-09-16T05:23:15.184716+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:23:07.713041+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.713041+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:15.371380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001382676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.48","version":"823'428","reported_seq":4097,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.182623+0000","last_change":"2024-09-16T05:23:15.184435+0000","last_active":"2024-09-16T05:23:15.182623+0000","last_peered":"2024-09-16T05:23:15.182623+0000","last_clean":"2024-09-16T05:23:15.182623+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:23:15.182623+0000","last_undegraded":"2024-09-16T05:23:15.182623+0000","last_fullsized":"2024-09-16T05:23:15.182623+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"776'403","last_scrub_stamp":"2024-09-16T05:22:06.889233+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.889233+0000","objects_scrubbed":6,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019808040000000001,"stat_sum":{"num_bytes":2487700,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":75761,"num_write":67,"num_write_kb":32708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.47","version":"0'0","reported_seq":2692,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:14.997581+0000","last_change":"2024-09-16T05:23:14.997750+0000","last_active":"2024-09-16T05:23:14.997581+0000","last_peered":"2024-09-16T05:23:14.997581+0000","last_clean":"2024-09-16T05:23:14.997581+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:23:14.997581+0000","last_undegraded":"2024-09-16T05:23:14.997581+0000","last_fullsized":"2024-09-16T05:23:14.997581+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.838189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:37.678283+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00067516799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.46","version":"116'9","reported_seq":895,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839868+0000","last_change":"2024-09-16T05:23:09.081647+0000","last_active":"2024-09-16T05:23:11.839868+0000","last_peered":"2024-09-16T05:23:11.839868+0000","last_clean":"2024-09-16T05:23:11.839868+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:23:11.839868+0000","last_undegraded":"2024-09-16T05:23:11.839868+0000","last_fullsized":"2024-09-16T05:23:11.839868+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:32.588419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.588419+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:58.462792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00097810900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":3204,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.008230+0000","last_change":"2024-09-16T05:23:15.008497+0000","last_active":"2024-09-16T05:23:15.008230+0000","last_peered":"2024-09-16T05:23:15.008230+0000","last_clean":"2024-09-16T05:23:15.008230+0000","last_became_active":"2024-09-16T05:21:10.818112+0000","last_became_peered":"2024-09-16T05:21:10.818112+0000","last_unstale":"2024-09-16T05:23:15.008230+0000","last_undegraded":"2024-09-16T05:23:15.008230+0000","last_fullsized":"2024-09-16T05:23:15.008230+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:23:09.592714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.592714+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:20.862751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053497060000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.44","version":"182'89","reported_seq":682,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.003401+0000","last_change":"2024-09-16T05:23:15.003792+0000","last_active":"2024-09-16T05:23:15.003401+0000","last_peered":"2024-09-16T05:23:15.003401+0000","last_clean":"2024-09-16T05:23:15.003401+0000","last_became_active":"2024-09-16T05:21:07.820306+0000","last_became_peered":"2024-09-16T05:21:07.820306+0000","last_unstale":"2024-09-16T05:23:15.003401+0000","last_undegraded":"2024-09-16T05:23:15.003401+0000","last_fullsized":"2024-09-16T05:23:15.003401+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:22:10.381270+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.381270+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:39.452660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011496169999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.43","version":"281'58","reported_seq":2654,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921348+0000","last_change":"2024-09-16T05:23:08.856241+0000","last_active":"2024-09-16T05:23:10.921348+0000","last_peered":"2024-09-16T05:23:10.921348+0000","last_clean":"2024-09-16T05:23:10.921348+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:23:10.921348+0000","last_undegraded":"2024-09-16T05:23:10.921348+0000","last_fullsized":"2024-09-16T05:23:10.921348+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:22:03.589783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:03.589783+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023794969999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":2653,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.993258+0000","last_change":"2024-09-16T05:23:13.993801+0000","last_active":"2024-09-16T05:23:13.993258+0000","last_peered":"2024-09-16T05:23:13.993258+0000","last_clean":"2024-09-16T05:23:13.993258+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:23:13.993258+0000","last_undegraded":"2024-09-16T05:23:13.993258+0000","last_fullsized":"2024-09-16T05:23:13.993258+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:22:25.741168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.741168+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.293780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049613499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":1538,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:14.997105+0000","last_change":"2024-09-16T05:23:14.997555+0000","last_active":"2024-09-16T05:23:14.997105+0000","last_peered":"2024-09-16T05:23:14.997105+0000","last_clean":"2024-09-16T05:23:14.997105+0000","last_became_active":"2024-09-16T05:21:54.930175+0000","last_became_peered":"2024-09-16T05:21:54.930175+0000","last_unstale":"2024-09-16T05:23:14.997105+0000","last_undegraded":"2024-09-16T05:23:14.997105+0000","last_fullsized":"2024-09-16T05:23:14.997105+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:22:28.678308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.678308+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:41.096575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072521100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.40","version":"205'33","reported_seq":1223,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.185091+0000","last_change":"2024-09-16T05:23:15.185399+0000","last_active":"2024-09-16T05:23:15.185091+0000","last_peered":"2024-09-16T05:23:15.185091+0000","last_clean":"2024-09-16T05:23:15.185091+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:23:15.185091+0000","last_undegraded":"2024-09-16T05:23:15.185091+0000","last_fullsized":"2024-09-16T05:23:15.185091+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:22:35.757429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.757429+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:02.219157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011788479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.3f","version":"823'551","reported_seq":4346,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.923283+0000","last_change":"2024-09-16T05:23:08.861771+0000","last_active":"2024-09-16T05:23:10.923283+0000","last_peered":"2024-09-16T05:23:10.923283+0000","last_clean":"2024-09-16T05:23:10.923283+0000","last_became_active":"2024-09-16T05:21:54.939233+0000","last_became_peered":"2024-09-16T05:21:54.939233+0000","last_unstale":"2024-09-16T05:23:10.923283+0000","last_undegraded":"2024-09-16T05:23:10.923283+0000","last_fullsized":"2024-09-16T05:23:10.923283+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"818'545","last_scrub_stamp":"2024-09-16T05:22:48.213000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.213000+0000","objects_scrubbed":4,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:08.199786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0062963979999999999,"stat_sum":{"num_bytes":3143173,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":116,"num_read_kb":104032,"num_write":71,"num_write_kb":25787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":11726959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.3e","version":"834'586","reported_seq":4242,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.867542+0000","last_change":"2024-09-16T05:23:09.040560+0000","last_active":"2024-09-16T05:23:11.867542+0000","last_peered":"2024-09-16T05:23:11.867542+0000","last_clean":"2024-09-16T05:23:11.867542+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:23:11.867542+0000","last_undegraded":"2024-09-16T05:23:11.867542+0000","last_fullsized":"2024-09-16T05:23:11.867542+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"820'580","last_scrub_stamp":"2024-09-16T05:22:51.944900+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.944900+0000","objects_scrubbed":1,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:04.166758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038616359999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":61220,"num_write":68,"num_write_kb":35144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2143,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.008920+0000","last_change":"2024-09-16T05:23:15.008920+0000","last_active":"2024-09-16T05:23:15.008920+0000","last_peered":"2024-09-16T05:23:15.008920+0000","last_clean":"2024-09-16T05:23:15.008920+0000","last_became_active":"2024-09-16T05:21:10.817387+0000","last_became_peered":"2024-09-16T05:21:10.817387+0000","last_unstale":"2024-09-16T05:23:15.008920+0000","last_undegraded":"2024-09-16T05:23:15.008920+0000","last_fullsized":"2024-09-16T05:23:15.008920+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.497749+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:47.295288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0056747289999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.3c","version":"819'826","reported_seq":4746,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921945+0000","last_change":"2024-09-16T05:23:08.859319+0000","last_active":"2024-09-16T05:23:10.921945+0000","last_peered":"2024-09-16T05:23:10.921945+0000","last_clean":"2024-09-16T05:23:10.921945+0000","last_became_active":"2024-09-16T05:21:07.828778+0000","last_became_peered":"2024-09-16T05:21:07.828778+0000","last_unstale":"2024-09-16T05:23:10.921945+0000","last_undegraded":"2024-09-16T05:23:10.921945+0000","last_fullsized":"2024-09-16T05:23:10.921945+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'807","last_scrub_stamp":"2024-09-16T05:22:31.305689+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:22:31.305689+0000","objects_scrubbed":4,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:33.864020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0051768949999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":194,"num_read_kb":258104,"num_write":95,"num_write_kb":34389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":14713263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.3b","version":"837'520","reported_seq":4226,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.187167+0000","last_change":"2024-09-16T05:23:15.187510+0000","last_active":"2024-09-16T05:23:15.187167+0000","last_peered":"2024-09-16T05:23:15.187167+0000","last_clean":"2024-09-16T05:23:15.187167+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:23:15.187167+0000","last_undegraded":"2024-09-16T05:23:15.187167+0000","last_fullsized":"2024-09-16T05:23:15.187167+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"823'516","last_scrub_stamp":"2024-09-16T05:22:56.618547+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:22:56.618547+0000","objects_scrubbed":1,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:21.463415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015863979999999999,"stat_sum":{"num_bytes":6050982,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":103894,"num_write":70,"num_write_kb":32800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.3a","version":"823'682","reported_seq":4346,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184494+0000","last_change":"2024-09-16T05:23:15.185257+0000","last_active":"2024-09-16T05:23:15.184494+0000","last_peered":"2024-09-16T05:23:15.184494+0000","last_clean":"2024-09-16T05:23:15.184494+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:23:15.184494+0000","last_undegraded":"2024-09-16T05:23:15.184494+0000","last_fullsized":"2024-09-16T05:23:15.184494+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"823'682","last_scrub_stamp":"2024-09-16T05:23:10.127124+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:23:10.127124+0000","objects_scrubbed":1,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:34.028575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026753929999999999,"stat_sum":{"num_bytes":3143173,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":66016,"num_write":78,"num_write_kb":43920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.39","version":"182'60","reported_seq":2768,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.183009+0000","last_change":"2024-09-16T05:23:15.184401+0000","last_active":"2024-09-16T05:23:15.183009+0000","last_peered":"2024-09-16T05:23:15.183009+0000","last_clean":"2024-09-16T05:23:15.183009+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:23:15.183009+0000","last_undegraded":"2024-09-16T05:23:15.183009+0000","last_fullsized":"2024-09-16T05:23:15.183009+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:50.962024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:50.962024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:05.016104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018899240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.38","version":"181'57","reported_seq":2773,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.992801+0000","last_change":"2024-09-16T05:23:13.993047+0000","last_active":"2024-09-16T05:23:13.992801+0000","last_peered":"2024-09-16T05:23:13.992801+0000","last_clean":"2024-09-16T05:23:13.992801+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:23:13.992801+0000","last_undegraded":"2024-09-16T05:23:13.992801+0000","last_fullsized":"2024-09-16T05:23:13.992801+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:22:54.645961+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.645961+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:06.681826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000456959,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":1634,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.002300+0000","last_change":"2024-09-16T05:23:15.002434+0000","last_active":"2024-09-16T05:23:15.002300+0000","last_peered":"2024-09-16T05:23:15.002300+0000","last_clean":"2024-09-16T05:23:15.002300+0000","last_became_active":"2024-09-16T05:21:54.938965+0000","last_became_peered":"2024-09-16T05:21:54.938965+0000","last_unstale":"2024-09-16T05:23:15.002300+0000","last_undegraded":"2024-09-16T05:23:15.002300+0000","last_fullsized":"2024-09-16T05:23:15.002300+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:25.115048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.115048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:43.946329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00103396,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.36","version":"272'60","reported_seq":682,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.003012+0000","last_change":"2024-09-16T05:23:15.003340+0000","last_active":"2024-09-16T05:23:15.003012+0000","last_peered":"2024-09-16T05:23:15.003012+0000","last_clean":"2024-09-16T05:23:15.003012+0000","last_became_active":"2024-09-16T05:21:07.819657+0000","last_became_peered":"2024-09-16T05:21:07.819657+0000","last_unstale":"2024-09-16T05:23:15.003012+0000","last_undegraded":"2024-09-16T05:23:15.003012+0000","last_fullsized":"2024-09-16T05:23:15.003012+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:22:49.686720+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.686720+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:08.669151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087992699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.35","version":"843'947","reported_seq":4707,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.003829+0000","last_change":"2024-09-16T05:23:15.008083+0000","last_active":"2024-09-16T05:23:15.003829+0000","last_peered":"2024-09-16T05:23:15.003829+0000","last_clean":"2024-09-16T05:23:15.003829+0000","last_became_active":"2024-09-16T05:21:10.815197+0000","last_became_peered":"2024-09-16T05:21:10.815197+0000","last_unstale":"2024-09-16T05:23:15.003829+0000","last_undegraded":"2024-09-16T05:23:15.003829+0000","last_fullsized":"2024-09-16T05:23:15.003829+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"769'866","last_scrub_stamp":"2024-09-16T05:22:00.928689+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.928689+0000","objects_scrubbed":9,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:23.688456+0000","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0.0051585399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":191,"num_read_kb":188287,"num_write":131,"num_write_kb":57425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":17054243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.34","version":"837'815","reported_seq":4100,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184022+0000","last_change":"2024-09-16T05:23:15.184273+0000","last_active":"2024-09-16T05:23:15.184022+0000","last_peered":"2024-09-16T05:23:15.184022+0000","last_clean":"2024-09-16T05:23:15.184022+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:23:15.184022+0000","last_undegraded":"2024-09-16T05:23:15.184022+0000","last_fullsized":"2024-09-16T05:23:15.184022+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"781'749","last_scrub_stamp":"2024-09-16T05:22:12.568382+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:22:12.568382+0000","objects_scrubbed":9,"log_size":815,"log_dups_size":0,"ondisk_log_size":815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:33.544877+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001007855,"stat_sum":{"num_bytes":6252732,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":245667,"num_write":143,"num_write_kb":52124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.33","version":"181'15","reported_seq":2846,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921492+0000","last_change":"2024-09-16T05:23:10.097672+0000","last_active":"2024-09-16T05:23:10.921492+0000","last_peered":"2024-09-16T05:23:10.921492+0000","last_clean":"2024-09-16T05:23:10.921492+0000","last_became_active":"2024-09-16T05:21:07.818018+0000","last_became_peered":"2024-09-16T05:21:07.818018+0000","last_unstale":"2024-09-16T05:23:10.921492+0000","last_undegraded":"2024-09-16T05:23:10.921492+0000","last_fullsized":"2024-09-16T05:23:10.921492+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:23:10.097585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:10.097585+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:14.756239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0038436820000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.32","version":"820'309","reported_seq":3139,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.866801+0000","last_change":"2024-09-16T05:23:09.041343+0000","last_active":"2024-09-16T05:23:11.866801+0000","last_peered":"2024-09-16T05:23:11.866801+0000","last_clean":"2024-09-16T05:23:11.866801+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:23:11.866801+0000","last_undegraded":"2024-09-16T05:23:11.866801+0000","last_fullsized":"2024-09-16T05:23:11.866801+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"754'267","last_scrub_stamp":"2024-09-16T05:21:45.593289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:45.593289+0000","objects_scrubbed":3,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0044472629999999999,"stat_sum":{"num_bytes":2269209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":123,"num_read_kb":122722,"num_write":54,"num_write_kb":22810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.31","version":"845'385","reported_seq":3820,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.000116+0000","last_change":"2024-09-16T05:23:15.000326+0000","last_active":"2024-09-16T05:23:15.000116+0000","last_peered":"2024-09-16T05:23:15.000116+0000","last_clean":"2024-09-16T05:23:15.000116+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:23:15.000116+0000","last_undegraded":"2024-09-16T05:23:15.000116+0000","last_fullsized":"2024-09-16T05:23:15.000116+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"834'373","last_scrub_stamp":"2024-09-16T05:23:09.677247+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.677247+0000","objects_scrubbed":1,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:18.910795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029598929999999999,"stat_sum":{"num_bytes":5064022,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":119777,"num_write":71,"num_write_kb":28214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.30","version":"182'16","reported_seq":1616,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.000389+0000","last_change":"2024-09-16T05:23:15.001088+0000","last_active":"2024-09-16T05:23:15.000389+0000","last_peered":"2024-09-16T05:23:15.000389+0000","last_clean":"2024-09-16T05:23:15.000389+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:23:15.000389+0000","last_undegraded":"2024-09-16T05:23:15.000389+0000","last_fullsized":"2024-09-16T05:23:15.000389+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:23:09.435437+0000","last_deep_scrub":"182'16","last_deep_scrub_stamp":"2024-09-16T05:21:49.039822+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.435437+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:31.608762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000920074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.12","version":"35'139","reported_seq":1341,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.783265+0000","last_change":"2024-09-16T05:23:15.783311+0000","last_active":"2024-09-16T05:23:15.783265+0000","last_peered":"2024-09-16T05:23:15.783265+0000","last_clean":"2024-09-16T05:23:15.783265+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:23:15.783265+0000","last_undegraded":"2024-09-16T05:23:15.783265+0000","last_fullsized":"2024-09-16T05:23:15.783265+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:15.783191+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:23:15.783191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:38.927996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":944,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.844254+0000","last_change":"2024-09-16T05:23:08.849852+0000","last_active":"2024-09-16T05:23:11.844254+0000","last_peered":"2024-09-16T05:23:11.844254+0000","last_clean":"2024-09-16T05:23:11.844254+0000","last_became_active":"2024-09-16T05:21:57.821735+0000","last_became_peered":"2024-09-16T05:21:57.821735+0000","last_unstale":"2024-09-16T05:23:11.844254+0000","last_undegraded":"2024-09-16T05:23:11.844254+0000","last_fullsized":"2024-09-16T05:23:11.844254+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:23:06.972379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:06.972379+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:20.164344+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0012498030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1310,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.002322+0000","last_change":"2024-09-16T05:23:13.594927+0000","last_active":"2024-09-16T05:23:15.002322+0000","last_peered":"2024-09-16T05:23:15.002322+0000","last_clean":"2024-09-16T05:23:15.002322+0000","last_became_active":"2024-09-16T05:21:10.814832+0000","last_became_peered":"2024-09-16T05:21:10.814832+0000","last_unstale":"2024-09-16T05:23:15.002322+0000","last_undegraded":"2024-09-16T05:23:15.002322+0000","last_fullsized":"2024-09-16T05:23:15.002322+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:13.594789+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:13.594789+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:36.070226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":3327,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.992870+0000","last_change":"2024-09-16T05:23:13.993092+0000","last_active":"2024-09-16T05:23:13.992870+0000","last_peered":"2024-09-16T05:23:13.992870+0000","last_clean":"2024-09-16T05:23:13.992870+0000","last_became_active":"2024-09-16T05:21:07.815584+0000","last_became_peered":"2024-09-16T05:21:07.815584+0000","last_unstale":"2024-09-16T05:23:13.992870+0000","last_undegraded":"2024-09-16T05:23:13.992870+0000","last_fullsized":"2024-09-16T05:23:13.992870+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:22:11.576311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:11.576311+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:20.785397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040563600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.c","version":"35'139","reported_seq":1287,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.853755+0000","last_change":"2024-09-16T05:21:43.300385+0000","last_active":"2024-09-16T05:23:11.853755+0000","last_peered":"2024-09-16T05:23:11.853755+0000","last_clean":"2024-09-16T05:23:11.853755+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:23:11.853755+0000","last_undegraded":"2024-09-16T05:23:11.853755+0000","last_fullsized":"2024-09-16T05:23:11.853755+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:43.300299+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:43.300299+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"823'461","reported_seq":4177,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.993980+0000","last_change":"2024-09-16T05:23:13.994190+0000","last_active":"2024-09-16T05:23:13.993980+0000","last_peered":"2024-09-16T05:23:13.993980+0000","last_clean":"2024-09-16T05:23:13.993980+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:23:13.993980+0000","last_undegraded":"2024-09-16T05:23:13.993980+0000","last_fullsized":"2024-09-16T05:23:13.993980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'437","last_scrub_stamp":"2024-09-16T05:22:27.742914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.742914+0000","objects_scrubbed":3,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:29.659607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00040550500000000001,"stat_sum":{"num_bytes":3244015,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":58,"num_read_kb":67732,"num_write":57,"num_write_kb":24396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.d","version":"35'139","reported_seq":1443,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921413+0000","last_change":"2024-09-16T05:22:26.361667+0000","last_active":"2024-09-16T05:23:10.921413+0000","last_peered":"2024-09-16T05:23:10.921413+0000","last_clean":"2024-09-16T05:23:10.921413+0000","last_became_active":"2024-09-16T05:20:35.804924+0000","last_became_peered":"2024-09-16T05:20:35.804924+0000","last_unstale":"2024-09-16T05:23:10.921413+0000","last_undegraded":"2024-09-16T05:23:10.921413+0000","last_fullsized":"2024-09-16T05:23:10.921413+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.361570+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.361570+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.129133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":3810,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.840060+0000","last_change":"2024-09-16T05:23:09.081966+0000","last_active":"2024-09-16T05:23:11.840060+0000","last_peered":"2024-09-16T05:23:11.840060+0000","last_clean":"2024-09-16T05:23:11.840060+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:23:11.840060+0000","last_undegraded":"2024-09-16T05:23:11.840060+0000","last_fullsized":"2024-09-16T05:23:11.840060+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:22:25.628406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.628406+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:53.027573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010623659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":1457,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.004438+0000","last_change":"2024-09-16T05:22:51.495227+0000","last_active":"2024-09-16T05:23:15.004438+0000","last_peered":"2024-09-16T05:23:15.004438+0000","last_clean":"2024-09-16T05:23:15.004438+0000","last_became_active":"2024-09-16T05:20:35.802192+0000","last_became_peered":"2024-09-16T05:20:35.802192+0000","last_unstale":"2024-09-16T05:23:15.004438+0000","last_undegraded":"2024-09-16T05:23:15.004438+0000","last_fullsized":"2024-09-16T05:23:15.004438+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:51.495148+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.495148+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:05.939482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"819'670","reported_seq":4380,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.186543+0000","last_change":"2024-09-16T05:23:15.187857+0000","last_active":"2024-09-16T05:23:15.186543+0000","last_peered":"2024-09-16T05:23:15.186543+0000","last_clean":"2024-09-16T05:23:15.186543+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:23:15.186543+0000","last_undegraded":"2024-09-16T05:23:15.186543+0000","last_fullsized":"2024-09-16T05:23:15.186543+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"793'662","last_scrub_stamp":"2024-09-16T05:22:26.926701+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.926701+0000","objects_scrubbed":3,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:41.573554+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0049867610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":52780,"num_write":55,"num_write_kb":30221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.f","version":"35'139","reported_seq":1438,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.183545+0000","last_change":"2024-09-16T05:22:08.676849+0000","last_active":"2024-09-16T05:23:15.183545+0000","last_peered":"2024-09-16T05:23:15.183545+0000","last_clean":"2024-09-16T05:23:15.183545+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:23:15.183545+0000","last_undegraded":"2024-09-16T05:23:15.183545+0000","last_fullsized":"2024-09-16T05:23:15.183545+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:08.676744+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:22:08.676744+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:24.347150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":3948,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.867161+0000","last_change":"2024-09-16T05:23:09.040416+0000","last_active":"2024-09-16T05:23:11.867161+0000","last_peered":"2024-09-16T05:23:11.867161+0000","last_clean":"2024-09-16T05:23:11.867161+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:23:11.867161+0000","last_undegraded":"2024-09-16T05:23:11.867161+0000","last_fullsized":"2024-09-16T05:23:11.867161+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:23:07.822545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.822545+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:16.854086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042471779999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1150,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.002176+0000","last_change":"2024-09-16T05:22:26.488008+0000","last_active":"2024-09-16T05:23:15.002176+0000","last_peered":"2024-09-16T05:23:15.002176+0000","last_clean":"2024-09-16T05:23:15.002176+0000","last_became_active":"2024-09-16T05:22:25.828286+0000","last_became_peered":"2024-09-16T05:22:25.828286+0000","last_unstale":"2024-09-16T05:23:15.002176+0000","last_undegraded":"2024-09-16T05:23:15.002176+0000","last_fullsized":"2024-09-16T05:23:15.002176+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.487937+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.487937+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:28.854050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"823'365","reported_seq":4049,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921537+0000","last_change":"2024-09-16T05:23:08.857782+0000","last_active":"2024-09-16T05:23:10.921537+0000","last_peered":"2024-09-16T05:23:10.921537+0000","last_clean":"2024-09-16T05:23:10.921537+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:23:10.921537+0000","last_undegraded":"2024-09-16T05:23:10.921537+0000","last_fullsized":"2024-09-16T05:23:10.921537+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'327","last_scrub_stamp":"2024-09-16T05:22:02.625744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:02.625744+0000","objects_scrubbed":1,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:14.669312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037610260000000002,"stat_sum":{"num_bytes":2790226,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":75412,"num_write":54,"num_write_kb":15536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.3","version":"845'295","reported_seq":4016,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.703943+0000","last_change":"2024-09-16T05:23:15.704161+0000","last_active":"2024-09-16T05:23:15.703943+0000","last_peered":"2024-09-16T05:23:15.703943+0000","last_clean":"2024-09-16T05:23:15.703943+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:23:15.703943+0000","last_undegraded":"2024-09-16T05:23:15.703943+0000","last_fullsized":"2024-09-16T05:23:15.703943+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'293","last_scrub_stamp":"2024-09-16T05:23:14.786482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:14.786482+0000","objects_scrubbed":2,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:38.847456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.51783820999999997,"stat_sum":{"num_bytes":4476376,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":128645,"num_write":64,"num_write_kb":25926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.0","version":"35'139","reported_seq":985,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.866499+0000","last_change":"2024-09-16T05:22:57.013492+0000","last_active":"2024-09-16T05:23:11.866499+0000","last_peered":"2024-09-16T05:23:11.866499+0000","last_clean":"2024-09-16T05:23:11.866499+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:23:11.866499+0000","last_undegraded":"2024-09-16T05:23:11.866499+0000","last_fullsized":"2024-09-16T05:23:11.866499+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:57.013376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:57.013376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:25.634739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":3706,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184425+0000","last_change":"2024-09-16T05:23:15.185015+0000","last_active":"2024-09-16T05:23:15.184425+0000","last_peered":"2024-09-16T05:23:15.184425+0000","last_clean":"2024-09-16T05:23:15.184425+0000","last_became_active":"2024-09-16T05:21:07.817459+0000","last_became_peered":"2024-09-16T05:21:07.817459+0000","last_unstale":"2024-09-16T05:23:15.184425+0000","last_undegraded":"2024-09-16T05:23:15.184425+0000","last_fullsized":"2024-09-16T05:23:15.184425+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:22:55.657664+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:22:55.657664+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:22.598658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013090720000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.1","version":"35'139","reported_seq":1218,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.000047+0000","last_change":"2024-09-16T05:22:38.207552+0000","last_active":"2024-09-16T05:23:15.000047+0000","last_peered":"2024-09-16T05:23:15.000047+0000","last_clean":"2024-09-16T05:23:15.000047+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:23:15.000047+0000","last_undegraded":"2024-09-16T05:23:15.000047+0000","last_fullsized":"2024-09-16T05:23:15.000047+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:22:38.207426+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:00.581775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":514,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.004161+0000","last_change":"2024-09-16T05:22:26.467188+0000","last_active":"2024-09-16T05:23:15.004161+0000","last_peered":"2024-09-16T05:23:15.004161+0000","last_clean":"2024-09-16T05:23:15.004161+0000","last_became_active":"2024-09-16T05:22:25.830349+0000","last_became_peered":"2024-09-16T05:22:25.830349+0000","last_unstale":"2024-09-16T05:23:15.004161+0000","last_undegraded":"2024-09-16T05:23:15.004161+0000","last_fullsized":"2024-09-16T05:23:15.004161+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.467087+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.467087+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:32.527602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":3628,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:12.564065+0000","last_change":"2024-09-16T05:23:12.564114+0000","last_active":"2024-09-16T05:23:12.564065+0000","last_peered":"2024-09-16T05:23:12.564065+0000","last_clean":"2024-09-16T05:23:12.564065+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:23:12.564065+0000","last_undegraded":"2024-09-16T05:23:12.564065+0000","last_fullsized":"2024-09-16T05:23:12.564065+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:23:12.563996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:12.563996+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:35.492360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053842699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":1965,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.853730+0000","last_change":"2024-09-16T05:22:02.352598+0000","last_active":"2024-09-16T05:23:11.853730+0000","last_peered":"2024-09-16T05:23:11.853730+0000","last_clean":"2024-09-16T05:23:11.853730+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:23:11.853730+0000","last_undegraded":"2024-09-16T05:23:11.853730+0000","last_fullsized":"2024-09-16T05:23:11.853730+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:02.352511+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:22:02.352511+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:28.908704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":3705,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921796+0000","last_change":"2024-09-16T05:23:08.859250+0000","last_active":"2024-09-16T05:23:10.921796+0000","last_peered":"2024-09-16T05:23:10.921796+0000","last_clean":"2024-09-16T05:23:10.921796+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:23:10.921796+0000","last_undegraded":"2024-09-16T05:23:10.921796+0000","last_fullsized":"2024-09-16T05:23:10.921796+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:22:36.317360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.317360+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:49.630220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050119880000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":2945,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.189062+0000","last_change":"2024-09-16T05:23:15.189841+0000","last_active":"2024-09-16T05:23:15.189062+0000","last_peered":"2024-09-16T05:23:15.189062+0000","last_clean":"2024-09-16T05:23:15.189062+0000","last_became_active":"2024-09-16T05:21:54.939117+0000","last_became_peered":"2024-09-16T05:21:54.939117+0000","last_unstale":"2024-09-16T05:23:15.189062+0000","last_undegraded":"2024-09-16T05:23:15.189062+0000","last_fullsized":"2024-09-16T05:23:15.189062+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:58.088776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:58.088776+0000","last_clean_scrub_stamp":"2024-09-16T05:22:58.088776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:19.869988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0069429590000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.4","version":"35'139","reported_seq":1920,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.867201+0000","last_change":"2024-09-16T05:22:48.936002+0000","last_active":"2024-09-16T05:23:11.867201+0000","last_peered":"2024-09-16T05:23:11.867201+0000","last_clean":"2024-09-16T05:23:11.867201+0000","last_became_active":"2024-09-16T05:15:51.723270+0000","last_became_peered":"2024-09-16T05:15:51.723270+0000","last_unstale":"2024-09-16T05:23:11.867201+0000","last_undegraded":"2024-09-16T05:23:11.867201+0000","last_fullsized":"2024-09-16T05:23:11.867201+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:48.935924+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.935924+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:07.656055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":2105,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839947+0000","last_change":"2024-09-16T05:23:09.081892+0000","last_active":"2024-09-16T05:23:11.839947+0000","last_peered":"2024-09-16T05:23:11.839947+0000","last_clean":"2024-09-16T05:23:11.839947+0000","last_became_active":"2024-09-16T05:21:21.816054+0000","last_became_peered":"2024-09-16T05:21:21.816054+0000","last_unstale":"2024-09-16T05:23:11.839947+0000","last_undegraded":"2024-09-16T05:23:11.839947+0000","last_fullsized":"2024-09-16T05:23:11.839947+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:36.384222+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.384222+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:53.435852+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0.0011465729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":1239,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.000857+0000","last_change":"2024-09-16T05:22:10.087414+0000","last_active":"2024-09-16T05:23:15.000857+0000","last_peered":"2024-09-16T05:23:15.000857+0000","last_clean":"2024-09-16T05:23:15.000857+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:23:15.000857+0000","last_undegraded":"2024-09-16T05:23:15.000857+0000","last_fullsized":"2024-09-16T05:23:15.000857+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:10.087343+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.087343+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:22.116328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"823'676","reported_seq":4391,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.994289+0000","last_change":"2024-09-16T05:23:13.994527+0000","last_active":"2024-09-16T05:23:13.994289+0000","last_peered":"2024-09-16T05:23:13.994289+0000","last_clean":"2024-09-16T05:23:13.994289+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:23:13.994289+0000","last_undegraded":"2024-09-16T05:23:13.994289+0000","last_fullsized":"2024-09-16T05:23:13.994289+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"823'676","last_scrub_stamp":"2024-09-16T05:23:07.681237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.681237+0000","objects_scrubbed":1,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:30.553362+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001045563,"stat_sum":{"num_bytes":3159980,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":131,"num_read_kb":146751,"num_write":54,"num_write_kb":21531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":1183,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.000691+0000","last_change":"2024-09-16T05:22:07.007557+0000","last_active":"2024-09-16T05:23:15.000691+0000","last_peered":"2024-09-16T05:23:15.000691+0000","last_clean":"2024-09-16T05:23:15.000691+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:23:15.000691+0000","last_undegraded":"2024-09-16T05:23:15.000691+0000","last_fullsized":"2024-09-16T05:23:15.000691+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.007487+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.007487+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.416611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":675,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.008564+0000","last_change":"2024-09-16T05:23:15.008564+0000","last_active":"2024-09-16T05:23:15.008564+0000","last_peered":"2024-09-16T05:23:15.008564+0000","last_clean":"2024-09-16T05:23:15.008564+0000","last_became_active":"2024-09-16T05:21:07.816472+0000","last_became_peered":"2024-09-16T05:21:07.816472+0000","last_unstale":"2024-09-16T05:23:15.008564+0000","last_undegraded":"2024-09-16T05:23:15.008564+0000","last_fullsized":"2024-09-16T05:23:15.008564+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:21:48.443201+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:21:48.443201+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056238659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.7","version":"35'139","reported_seq":1229,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.001228+0000","last_change":"2024-09-16T05:22:08.048856+0000","last_active":"2024-09-16T05:23:15.001228+0000","last_peered":"2024-09-16T05:23:15.001228+0000","last_clean":"2024-09-16T05:23:15.001228+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:23:15.001228+0000","last_undegraded":"2024-09-16T05:23:15.001228+0000","last_fullsized":"2024-09-16T05:23:15.001228+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:08.048756+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:08.048756+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.883180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":3871,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.993451+0000","last_change":"2024-09-16T05:23:13.993672+0000","last_active":"2024-09-16T05:23:13.993451+0000","last_peered":"2024-09-16T05:23:13.993451+0000","last_clean":"2024-09-16T05:23:13.993451+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:23:13.993451+0000","last_undegraded":"2024-09-16T05:23:13.993451+0000","last_fullsized":"2024-09-16T05:23:13.993451+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:22:35.723844+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.723844+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:52.243276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067779600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":1963,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184619+0000","last_change":"2024-09-16T05:22:10.608245+0000","last_active":"2024-09-16T05:23:15.184619+0000","last_peered":"2024-09-16T05:23:15.184619+0000","last_clean":"2024-09-16T05:23:15.184619+0000","last_became_active":"2024-09-16T05:20:35.807550+0000","last_became_peered":"2024-09-16T05:20:35.807550+0000","last_unstale":"2024-09-16T05:23:15.184619+0000","last_undegraded":"2024-09-16T05:23:15.184619+0000","last_fullsized":"2024-09-16T05:23:15.184619+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:10.608132+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:10.608132+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.753900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"837'743","reported_seq":4461,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.866974+0000","last_change":"2024-09-16T05:23:09.041437+0000","last_active":"2024-09-16T05:23:11.866974+0000","last_peered":"2024-09-16T05:23:11.866974+0000","last_clean":"2024-09-16T05:23:11.866974+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:23:11.866974+0000","last_undegraded":"2024-09-16T05:23:11.866974+0000","last_fullsized":"2024-09-16T05:23:11.866974+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'707","last_scrub_stamp":"2024-09-16T05:22:31.956529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:31.956529+0000","objects_scrubbed":5,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:54.641541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0043994139999999999,"stat_sum":{"num_bytes":3462506,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":127,"num_read_kb":137086,"num_write":89,"num_write_kb":43239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1954,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921153+0000","last_change":"2024-09-16T05:22:04.552683+0000","last_active":"2024-09-16T05:23:10.921153+0000","last_peered":"2024-09-16T05:23:10.921153+0000","last_clean":"2024-09-16T05:23:10.921153+0000","last_became_active":"2024-09-16T05:20:35.804221+0000","last_became_peered":"2024-09-16T05:20:35.804221+0000","last_unstale":"2024-09-16T05:23:10.921153+0000","last_undegraded":"2024-09-16T05:23:10.921153+0000","last_fullsized":"2024-09-16T05:23:10.921153+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:04.552582+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:04.552582+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:17.369449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2156,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.002152+0000","last_change":"2024-09-16T05:23:15.002502+0000","last_active":"2024-09-16T05:23:15.002152+0000","last_peered":"2024-09-16T05:23:15.002152+0000","last_clean":"2024-09-16T05:23:15.002152+0000","last_became_active":"2024-09-16T05:21:10.815913+0000","last_became_peered":"2024-09-16T05:21:10.815913+0000","last_unstale":"2024-09-16T05:23:15.002152+0000","last_undegraded":"2024-09-16T05:23:15.002152+0000","last_fullsized":"2024-09-16T05:23:15.002152+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:23:07.627317+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.627317+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:26.786596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050101200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.a","version":"35'139","reported_seq":1969,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.182696+0000","last_change":"2024-09-16T05:21:54.943368+0000","last_active":"2024-09-16T05:23:15.182696+0000","last_peered":"2024-09-16T05:23:15.182696+0000","last_clean":"2024-09-16T05:23:15.182696+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:23:15.182696+0000","last_undegraded":"2024-09-16T05:23:15.182696+0000","last_fullsized":"2024-09-16T05:23:15.182696+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:45.767900+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:21:45.767900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1448,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:14.997145+0000","last_change":"2024-09-16T05:22:32.678108+0000","last_active":"2024-09-16T05:23:14.997145+0000","last_peered":"2024-09-16T05:23:14.997145+0000","last_clean":"2024-09-16T05:23:14.997145+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:23:14.997145+0000","last_undegraded":"2024-09-16T05:23:14.997145+0000","last_fullsized":"2024-09-16T05:23:14.997145+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:32.678034+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.678034+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:39.197820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":1616,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.001496+0000","last_change":"2024-09-16T05:23:15.001658+0000","last_active":"2024-09-16T05:23:15.001496+0000","last_peered":"2024-09-16T05:23:15.001496+0000","last_clean":"2024-09-16T05:23:15.001496+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:23:15.001496+0000","last_undegraded":"2024-09-16T05:23:15.001496+0000","last_fullsized":"2024-09-16T05:23:15.001496+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:22:56.409531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:56.409531+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:16.827778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001259294,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.10","version":"35'139","reported_seq":1960,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184360+0000","last_change":"2024-09-16T05:22:50.729358+0000","last_active":"2024-09-16T05:23:15.184360+0000","last_peered":"2024-09-16T05:23:15.184360+0000","last_clean":"2024-09-16T05:23:15.184360+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:23:15.184360+0000","last_undegraded":"2024-09-16T05:23:15.184360+0000","last_fullsized":"2024-09-16T05:23:15.184360+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:50.729283+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:50.729283+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:18.895673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":3320,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.186632+0000","last_change":"2024-09-16T05:23:15.187652+0000","last_active":"2024-09-16T05:23:15.186632+0000","last_peered":"2024-09-16T05:23:15.186632+0000","last_clean":"2024-09-16T05:23:15.186632+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:23:15.186632+0000","last_undegraded":"2024-09-16T05:23:15.186632+0000","last_fullsized":"2024-09-16T05:23:15.186632+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:22:26.133349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.133349+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:55.618661+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0.0047752109999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.17","version":"35'139","reported_seq":842,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.993816+0000","last_change":"2024-09-16T05:22:00.654377+0000","last_active":"2024-09-16T05:23:13.993816+0000","last_peered":"2024-09-16T05:23:13.993816+0000","last_clean":"2024-09-16T05:23:13.993816+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:23:13.993816+0000","last_undegraded":"2024-09-16T05:23:13.993816+0000","last_fullsized":"2024-09-16T05:23:13.993816+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:00.654245+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:00.654245+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":3606,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.187696+0000","last_change":"2024-09-16T05:23:15.189002+0000","last_active":"2024-09-16T05:23:15.187696+0000","last_peered":"2024-09-16T05:23:15.187696+0000","last_clean":"2024-09-16T05:23:15.187696+0000","last_became_active":"2024-09-16T05:21:54.931954+0000","last_became_peered":"2024-09-16T05:21:54.931954+0000","last_unstale":"2024-09-16T05:23:15.187696+0000","last_undegraded":"2024-09-16T05:23:15.187696+0000","last_fullsized":"2024-09-16T05:23:15.187696+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:23:11.097094+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:23:11.097094+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:22.750488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0061094779999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.16","version":"35'139","reported_seq":894,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839772+0000","last_change":"2024-09-16T05:22:05.738733+0000","last_active":"2024-09-16T05:23:11.839772+0000","last_peered":"2024-09-16T05:23:11.839772+0000","last_clean":"2024-09-16T05:23:11.839772+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:23:11.839772+0000","last_undegraded":"2024-09-16T05:23:11.839772+0000","last_fullsized":"2024-09-16T05:23:11.839772+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:05.738655+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:22:05.738655+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"760'658","reported_seq":4489,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184560+0000","last_change":"2024-09-16T05:23:15.185410+0000","last_active":"2024-09-16T05:23:15.184560+0000","last_peered":"2024-09-16T05:23:15.184560+0000","last_clean":"2024-09-16T05:23:15.184560+0000","last_became_active":"2024-09-16T05:21:07.826042+0000","last_became_peered":"2024-09-16T05:21:07.826042+0000","last_unstale":"2024-09-16T05:23:15.184560+0000","last_undegraded":"2024-09-16T05:23:15.184560+0000","last_fullsized":"2024-09-16T05:23:15.184560+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"760'658","last_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_deep_scrub":"760'658","last_deep_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_clean_scrub_stamp":"2024-09-16T05:22:05.847540+0000","objects_scrubbed":0,"log_size":658,"log_dups_size":0,"ondisk_log_size":658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:26.576434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027688669999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":47657,"num_write":25,"num_write_kb":14580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":7118573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.15","version":"35'139","reported_seq":916,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.183327+0000","last_change":"2024-09-16T05:22:06.689465+0000","last_active":"2024-09-16T05:23:15.183327+0000","last_peered":"2024-09-16T05:23:15.183327+0000","last_clean":"2024-09-16T05:23:15.183327+0000","last_became_active":"2024-09-16T05:20:40.802090+0000","last_became_peered":"2024-09-16T05:20:40.802090+0000","last_unstale":"2024-09-16T05:23:15.183327+0000","last_undegraded":"2024-09-16T05:23:15.183327+0000","last_fullsized":"2024-09-16T05:23:15.183327+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:06.689376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.689376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:29.936194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"733'334","reported_seq":3682,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184366+0000","last_change":"2024-09-16T05:23:15.184660+0000","last_active":"2024-09-16T05:23:15.184366+0000","last_peered":"2024-09-16T05:23:15.184366+0000","last_clean":"2024-09-16T05:23:15.184366+0000","last_became_active":"2024-09-16T05:21:54.937285+0000","last_became_peered":"2024-09-16T05:21:54.937285+0000","last_unstale":"2024-09-16T05:23:15.184366+0000","last_undegraded":"2024-09-16T05:23:15.184366+0000","last_fullsized":"2024-09-16T05:23:15.184366+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'334","last_scrub_stamp":"2024-09-16T05:22:51.700579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.700579+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:52.259916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001025668,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.14","version":"35'139","reported_seq":1276,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839681+0000","last_change":"2024-09-16T05:21:48.712051+0000","last_active":"2024-09-16T05:23:11.839681+0000","last_peered":"2024-09-16T05:23:11.839681+0000","last_clean":"2024-09-16T05:23:11.839681+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:23:11.839681+0000","last_undegraded":"2024-09-16T05:23:11.839681+0000","last_fullsized":"2024-09-16T05:23:11.839681+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:21:48.711960+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:21:48.711960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":693,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.003080+0000","last_change":"2024-09-16T05:23:15.003399+0000","last_active":"2024-09-16T05:23:15.003080+0000","last_peered":"2024-09-16T05:23:15.003080+0000","last_clean":"2024-09-16T05:23:15.003080+0000","last_became_active":"2024-09-16T05:21:27.826090+0000","last_became_peered":"2024-09-16T05:21:27.826090+0000","last_unstale":"2024-09-16T05:23:15.003080+0000","last_undegraded":"2024-09-16T05:23:15.003080+0000","last_fullsized":"2024-09-16T05:23:15.003080+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.831827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:53.227564+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0.00089919399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.1b","version":"35'139","reported_seq":1367,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.992827+0000","last_change":"2024-09-16T05:22:07.577401+0000","last_active":"2024-09-16T05:23:13.992827+0000","last_peered":"2024-09-16T05:23:13.992827+0000","last_clean":"2024-09-16T05:23:13.992827+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:23:13.992827+0000","last_undegraded":"2024-09-16T05:23:13.992827+0000","last_fullsized":"2024-09-16T05:23:13.992827+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.577328+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.577328+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:35.106050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":3713,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.187933+0000","last_change":"2024-09-16T05:23:15.189263+0000","last_active":"2024-09-16T05:23:15.187933+0000","last_peered":"2024-09-16T05:23:15.187933+0000","last_clean":"2024-09-16T05:23:15.187933+0000","last_became_active":"2024-09-16T05:21:27.832312+0000","last_became_peered":"2024-09-16T05:21:27.832312+0000","last_unstale":"2024-09-16T05:23:15.187933+0000","last_undegraded":"2024-09-16T05:23:15.187933+0000","last_fullsized":"2024-09-16T05:23:15.187933+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:22:32.994229+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:22:32.994229+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:36.882942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0063603119999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.1a","version":"35'139","reported_seq":733,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.004280+0000","last_change":"2024-09-16T05:22:07.494046+0000","last_active":"2024-09-16T05:23:15.004280+0000","last_peered":"2024-09-16T05:23:15.004280+0000","last_clean":"2024-09-16T05:23:15.004280+0000","last_became_active":"2024-09-16T05:20:40.806260+0000","last_became_peered":"2024-09-16T05:20:40.806260+0000","last_unstale":"2024-09-16T05:23:15.004280+0000","last_undegraded":"2024-09-16T05:23:15.004280+0000","last_fullsized":"2024-09-16T05:23:15.004280+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:07.493964+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:22:07.493964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.726913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":3309,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.853538+0000","last_change":"2024-09-16T05:23:09.037704+0000","last_active":"2024-09-16T05:23:11.853538+0000","last_peered":"2024-09-16T05:23:11.853538+0000","last_clean":"2024-09-16T05:23:11.853538+0000","last_became_active":"2024-09-16T05:21:54.930126+0000","last_became_peered":"2024-09-16T05:21:54.930126+0000","last_unstale":"2024-09-16T05:23:11.853538+0000","last_undegraded":"2024-09-16T05:23:11.853538+0000","last_fullsized":"2024-09-16T05:23:11.853538+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:35.348819+0000","last_deep_scrub":"182'60","last_deep_scrub_stamp":"2024-09-16T05:21:29.356695+0000","last_clean_scrub_stamp":"2024-09-16T05:22:35.348819+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:48.038572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016532980000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"1.19","version":"35'139","reported_seq":1964,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.000131+0000","last_change":"2024-09-16T05:22:28.141713+0000","last_active":"2024-09-16T05:23:15.000131+0000","last_peered":"2024-09-16T05:23:15.000131+0000","last_clean":"2024-09-16T05:23:15.000131+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:23:15.000131+0000","last_undegraded":"2024-09-16T05:23:15.000131+0000","last_fullsized":"2024-09-16T05:23:15.000131+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:28.141626+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.141626+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:45.344742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"839'1234","reported_seq":5067,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.993148+0000","last_change":"2024-09-16T05:23:13.993573+0000","last_active":"2024-09-16T05:23:13.993148+0000","last_peered":"2024-09-16T05:23:13.993148+0000","last_clean":"2024-09-16T05:23:13.993148+0000","last_became_active":"2024-09-16T05:21:54.943140+0000","last_became_peered":"2024-09-16T05:21:54.943140+0000","last_unstale":"2024-09-16T05:23:13.993148+0000","last_undegraded":"2024-09-16T05:23:13.993148+0000","last_fullsized":"2024-09-16T05:23:13.993148+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"819'1229","last_scrub_stamp":"2024-09-16T05:22:49.730316+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.730316+0000","objects_scrubbed":4,"log_size":1234,"log_dups_size":0,"ondisk_log_size":1234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:04.610680+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000428572,"stat_sum":{"num_bytes":6958560,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":37762,"num_write":14,"num_write_kb":8066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":19652756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"1.18","version":"35'139","reported_seq":1964,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.184039+0000","last_change":"2024-09-16T05:22:30.658597+0000","last_active":"2024-09-16T05:23:15.184039+0000","last_peered":"2024-09-16T05:23:15.184039+0000","last_clean":"2024-09-16T05:23:15.184039+0000","last_became_active":"2024-09-16T05:20:35.809442+0000","last_became_peered":"2024-09-16T05:20:35.809442+0000","last_unstale":"2024-09-16T05:23:15.184039+0000","last_undegraded":"2024-09-16T05:23:15.184039+0000","last_fullsized":"2024-09-16T05:23:15.184039+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:30.658505+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:30.658505+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.751732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"823'734","reported_seq":4457,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.866884+0000","last_change":"2024-09-16T05:23:09.039098+0000","last_active":"2024-09-16T05:23:11.866884+0000","last_peered":"2024-09-16T05:23:11.866884+0000","last_clean":"2024-09-16T05:23:11.866884+0000","last_became_active":"2024-09-16T05:21:27.832003+0000","last_became_peered":"2024-09-16T05:21:27.832003+0000","last_unstale":"2024-09-16T05:23:11.866884+0000","last_undegraded":"2024-09-16T05:23:11.866884+0000","last_fullsized":"2024-09-16T05:23:11.866884+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"823'734","last_scrub_stamp":"2024-09-16T05:22:54.026063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.026063+0000","objects_scrubbed":1,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:04.683135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.003132686,"stat_sum":{"num_bytes":2655770,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30588,"num_write":19,"num_write_kb":6977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.1c","version":"723'472","reported_seq":3887,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.000820+0000","last_change":"2024-09-16T05:23:15.001148+0000","last_active":"2024-09-16T05:23:15.000820+0000","last_peered":"2024-09-16T05:23:15.000820+0000","last_clean":"2024-09-16T05:23:15.000820+0000","last_became_active":"2024-09-16T05:21:27.831591+0000","last_became_peered":"2024-09-16T05:21:27.831591+0000","last_unstale":"2024-09-16T05:23:15.000820+0000","last_undegraded":"2024-09-16T05:23:15.000820+0000","last_fullsized":"2024-09-16T05:23:15.000820+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"723'472","last_scrub_stamp":"2024-09-16T05:22:51.347894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.347894+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:59.614893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089062600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1444,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.867435+0000","last_change":"2024-09-16T05:22:25.948680+0000","last_active":"2024-09-16T05:23:11.867435+0000","last_peered":"2024-09-16T05:23:11.867435+0000","last_clean":"2024-09-16T05:23:11.867435+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:23:11.867435+0000","last_undegraded":"2024-09-16T05:23:11.867435+0000","last_fullsized":"2024-09-16T05:23:11.867435+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:25.948598+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.948598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:47.051847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"829'1308","reported_seq":5258,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.002003+0000","last_change":"2024-09-16T05:23:15.002250+0000","last_active":"2024-09-16T05:23:15.002003+0000","last_peered":"2024-09-16T05:23:15.002003+0000","last_clean":"2024-09-16T05:23:15.002003+0000","last_became_active":"2024-09-16T05:21:27.833962+0000","last_became_peered":"2024-09-16T05:21:27.833962+0000","last_unstale":"2024-09-16T05:23:15.002003+0000","last_undegraded":"2024-09-16T05:23:15.002003+0000","last_fullsized":"2024-09-16T05:23:15.002003+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"823'1307","last_scrub_stamp":"2024-09-16T05:22:53.289164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:53.289164+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:06.902643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011993679999999999,"stat_sum":{"num_bytes":2151560,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":4663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"1.1d","version":"35'139","reported_seq":795,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.003134+0000","last_change":"2024-09-16T05:22:37.530745+0000","last_active":"2024-09-16T05:23:15.003134+0000","last_peered":"2024-09-16T05:23:15.003134+0000","last_clean":"2024-09-16T05:23:15.003134+0000","last_became_active":"2024-09-16T05:21:10.815037+0000","last_became_peered":"2024-09-16T05:21:10.815037+0000","last_unstale":"2024-09-16T05:23:15.003134+0000","last_undegraded":"2024-09-16T05:23:15.003134+0000","last_fullsized":"2024-09-16T05:23:15.003134+0000","mapping_epoch":720,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:37.530632+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.530632+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:37.641748+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"736'512","reported_seq":4234,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.854102+0000","last_change":"2024-09-16T05:23:09.038059+0000","last_active":"2024-09-16T05:23:11.854102+0000","last_peered":"2024-09-16T05:23:11.854102+0000","last_clean":"2024-09-16T05:23:11.854102+0000","last_became_active":"2024-09-16T05:21:27.826212+0000","last_became_peered":"2024-09-16T05:21:27.826212+0000","last_unstale":"2024-09-16T05:23:11.854102+0000","last_undegraded":"2024-09-16T05:23:11.854102+0000","last_fullsized":"2024-09-16T05:23:11.854102+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'512","last_scrub_stamp":"2024-09-16T05:22:34.394958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.394958+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:54.619256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018286520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"1.1c","version":"35'139","reported_seq":697,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.993284+0000","last_change":"2024-09-16T05:22:06.589785+0000","last_active":"2024-09-16T05:23:13.993284+0000","last_peered":"2024-09-16T05:23:13.993284+0000","last_clean":"2024-09-16T05:23:13.993284+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:23:13.993284+0000","last_undegraded":"2024-09-16T05:23:13.993284+0000","last_fullsized":"2024-09-16T05:23:13.993284+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:06.589701+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.589701+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:21.427227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":3935,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:14.997819+0000","last_change":"2024-09-16T05:23:14.997993+0000","last_active":"2024-09-16T05:23:14.997819+0000","last_peered":"2024-09-16T05:23:14.997819+0000","last_clean":"2024-09-16T05:23:14.997819+0000","last_became_active":"2024-09-16T05:21:27.831337+0000","last_became_peered":"2024-09-16T05:21:27.831337+0000","last_unstale":"2024-09-16T05:23:14.997819+0000","last_undegraded":"2024-09-16T05:23:14.997819+0000","last_fullsized":"2024-09-16T05:23:14.997819+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:22:37.720977+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.720977+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:01.381213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00085916900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.20","version":"823'378","reported_seq":4106,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921314+0000","last_change":"2024-09-16T05:23:09.047432+0000","last_active":"2024-09-16T05:23:10.921314+0000","last_peered":"2024-09-16T05:23:10.921314+0000","last_clean":"2024-09-16T05:23:10.921314+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:23:10.921314+0000","last_undegraded":"2024-09-16T05:23:10.921314+0000","last_fullsized":"2024-09-16T05:23:10.921314+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"823'378","last_scrub_stamp":"2024-09-16T05:23:09.047313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.047313+0000","objects_scrubbed":1,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:13.346171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002478175,"stat_sum":{"num_bytes":3294436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":166,"num_read_kb":195113,"num_write":75,"num_write_kb":29696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":3627,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839438+0000","last_change":"2024-09-16T05:23:09.081471+0000","last_active":"2024-09-16T05:23:11.839438+0000","last_peered":"2024-09-16T05:23:11.839438+0000","last_clean":"2024-09-16T05:23:11.839438+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:23:11.839438+0000","last_undegraded":"2024-09-16T05:23:11.839438+0000","last_fullsized":"2024-09-16T05:23:11.839438+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:22:27.632823+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.632823+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:52.192520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094877899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":1614,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.001717+0000","last_change":"2024-09-16T05:23:15.001947+0000","last_active":"2024-09-16T05:23:15.001717+0000","last_peered":"2024-09-16T05:23:15.001717+0000","last_clean":"2024-09-16T05:23:15.001717+0000","last_became_active":"2024-09-16T05:21:54.942985+0000","last_became_peered":"2024-09-16T05:21:54.942985+0000","last_unstale":"2024-09-16T05:23:15.001717+0000","last_undegraded":"2024-09-16T05:23:15.001717+0000","last_fullsized":"2024-09-16T05:23:15.001717+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:21:46.010447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:21:46.010447+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011889960000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.23","version":"184'17","reported_seq":3169,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.853876+0000","last_change":"2024-09-16T05:23:09.038420+0000","last_active":"2024-09-16T05:23:11.853876+0000","last_peered":"2024-09-16T05:23:11.853876+0000","last_clean":"2024-09-16T05:23:11.853876+0000","last_became_active":"2024-09-16T05:22:22.838582+0000","last_became_peered":"2024-09-16T05:22:22.838582+0000","last_unstale":"2024-09-16T05:23:11.853876+0000","last_undegraded":"2024-09-16T05:23:11.853876+0000","last_fullsized":"2024-09-16T05:23:11.853876+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:22:26.319237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.319237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:27.391944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002044073,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":3002,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.185302+0000","last_change":"2024-09-16T05:23:15.186476+0000","last_active":"2024-09-16T05:23:15.185302+0000","last_peered":"2024-09-16T05:23:15.185302+0000","last_clean":"2024-09-16T05:23:15.185302+0000","last_became_active":"2024-09-16T05:21:54.932630+0000","last_became_peered":"2024-09-16T05:21:54.932630+0000","last_unstale":"2024-09-16T05:23:15.185302+0000","last_undegraded":"2024-09-16T05:23:15.185302+0000","last_fullsized":"2024-09-16T05:23:15.185302+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:23:08.095941+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:23:08.095941+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:21.842940+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003821723,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.25","version":"828'1351","reported_seq":4956,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.994342+0000","last_change":"2024-09-16T05:23:13.994564+0000","last_active":"2024-09-16T05:23:13.994342+0000","last_peered":"2024-09-16T05:23:13.994342+0000","last_clean":"2024-09-16T05:23:13.994342+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:23:13.994342+0000","last_undegraded":"2024-09-16T05:23:13.994342+0000","last_fullsized":"2024-09-16T05:23:13.994342+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"781'1244","last_scrub_stamp":"2024-09-16T05:22:12.573275+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:22:12.573275+0000","objects_scrubbed":8,"log_size":1351,"log_dups_size":0,"ondisk_log_size":1351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:31.729844+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00064954699999999999,"stat_sum":{"num_bytes":5580452,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":268,"num_read_kb":267962,"num_write":222,"num_write_kb":89381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":2961,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.923111+0000","last_change":"2024-09-16T05:23:08.861764+0000","last_active":"2024-09-16T05:23:10.923111+0000","last_peered":"2024-09-16T05:23:10.923111+0000","last_clean":"2024-09-16T05:23:10.923111+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:23:10.923111+0000","last_undegraded":"2024-09-16T05:23:10.923111+0000","last_fullsized":"2024-09-16T05:23:10.923111+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_clean_scrub_stamp":"2024-09-16T05:22:06.590423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:23.184674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070650160000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":2973,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.923047+0000","last_change":"2024-09-16T05:23:08.862936+0000","last_active":"2024-09-16T05:23:10.923047+0000","last_peered":"2024-09-16T05:23:10.923047+0000","last_clean":"2024-09-16T05:23:10.923047+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:23:10.923047+0000","last_undegraded":"2024-09-16T05:23:10.923047+0000","last_fullsized":"2024-09-16T05:23:10.923047+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:25.368861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:25.368861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:52.643487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0073817190000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":1992,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839265+0000","last_change":"2024-09-16T05:23:09.080842+0000","last_active":"2024-09-16T05:23:11.839265+0000","last_peered":"2024-09-16T05:23:11.839265+0000","last_clean":"2024-09-16T05:23:11.839265+0000","last_became_active":"2024-09-16T05:21:07.818030+0000","last_became_peered":"2024-09-16T05:21:07.818030+0000","last_unstale":"2024-09-16T05:23:11.839265+0000","last_undegraded":"2024-09-16T05:23:11.839265+0000","last_fullsized":"2024-09-16T05:23:11.839265+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:22:34.641066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.641066+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:51.013066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058576799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.29","version":"842'1396","reported_seq":5388,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.866997+0000","last_change":"2024-09-16T05:23:08.849851+0000","last_active":"2024-09-16T05:23:11.866997+0000","last_peered":"2024-09-16T05:23:11.866997+0000","last_clean":"2024-09-16T05:23:11.866997+0000","last_became_active":"2024-09-16T05:21:57.823393+0000","last_became_peered":"2024-09-16T05:21:57.823393+0000","last_unstale":"2024-09-16T05:23:11.866997+0000","last_undegraded":"2024-09-16T05:23:11.866997+0000","last_fullsized":"2024-09-16T05:23:11.866997+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":6,"last_scrub":"795'1297","last_scrub_stamp":"2024-09-16T05:22:26.617228+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.617228+0000","objects_scrubbed":15,"log_size":1396,"log_dups_size":0,"ondisk_log_size":1396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:32.449955+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0016573060000000001,"stat_sum":{"num_bytes":10068119,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":371,"num_read_kb":354524,"num_write":277,"num_write_kb":104451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":45326140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1289,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.185319+0000","last_change":"2024-09-16T05:23:15.185577+0000","last_active":"2024-09-16T05:23:15.185319+0000","last_peered":"2024-09-16T05:23:15.185319+0000","last_clean":"2024-09-16T05:23:15.185319+0000","last_became_active":"2024-09-16T05:22:25.829990+0000","last_became_peered":"2024-09-16T05:22:25.829990+0000","last_unstale":"2024-09-16T05:23:15.185319+0000","last_undegraded":"2024-09-16T05:23:15.185319+0000","last_fullsized":"2024-09-16T05:23:15.185319+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:22:26.604651+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.604651+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:37.617204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042923800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]},{"pgid":"2.2b","version":"829'445","reported_seq":4055,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-09-16T05:23:11.839530+0000","last_change":"2024-09-16T05:23:09.081753+0000","last_active":"2024-09-16T05:23:11.839530+0000","last_peered":"2024-09-16T05:23:11.839530+0000","last_clean":"2024-09-16T05:23:11.839530+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:23:11.839530+0000","last_undegraded":"2024-09-16T05:23:11.839530+0000","last_fullsized":"2024-09-16T05:23:11.839530+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"825'443","last_scrub_stamp":"2024-09-16T05:22:57.584272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:57.584272+0000","objects_scrubbed":2,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:04.133415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010706769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":89099,"num_write":75,"num_write_kb":28866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":3818,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-09-16T05:23:13.994120+0000","last_change":"2024-09-16T05:23:13.994300+0000","last_active":"2024-09-16T05:23:13.994120+0000","last_peered":"2024-09-16T05:23:13.994120+0000","last_clean":"2024-09-16T05:23:13.994120+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:23:13.994120+0000","last_undegraded":"2024-09-16T05:23:13.994120+0000","last_fullsized":"2024-09-16T05:23:13.994120+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:22:48.737646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.737646+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:53.274325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075550699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"165","length":"1"}]},{"pgid":"2.2d","version":"819'522","reported_seq":4284,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.921604+0000","last_change":"2024-09-16T05:23:08.860251+0000","last_active":"2024-09-16T05:23:10.921604+0000","last_peered":"2024-09-16T05:23:10.921604+0000","last_clean":"2024-09-16T05:23:10.921604+0000","last_became_active":"2024-09-16T05:21:54.933266+0000","last_became_peered":"2024-09-16T05:21:54.933266+0000","last_unstale":"2024-09-16T05:23:10.921604+0000","last_undegraded":"2024-09-16T05:23:10.921604+0000","last_fullsized":"2024-09-16T05:23:10.921604+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"819'522","last_scrub_stamp":"2024-09-16T05:22:54.200053+0000","last_deep_scrub":"819'522","last_deep_scrub_stamp":"2024-09-16T05:22:54.200053+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.200053+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:14.700044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058303779999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":88584,"num_write":74,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":10583843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.2e","version":"829'799","reported_seq":4419,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-09-16T05:23:10.922849+0000","last_change":"2024-09-16T05:23:08.860247+0000","last_active":"2024-09-16T05:23:10.922849+0000","last_peered":"2024-09-16T05:23:10.922849+0000","last_clean":"2024-09-16T05:23:10.922849+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:23:10.922849+0000","last_undegraded":"2024-09-16T05:23:10.922849+0000","last_fullsized":"2024-09-16T05:23:10.922849+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"794'746","last_scrub_stamp":"2024-09-16T05:22:28.624283+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:22:28.624283+0000","objects_scrubbed":7,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:49.009375+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0.0056064610000000001,"stat_sum":{"num_bytes":3412085,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":121,"num_read_kb":80562,"num_write":118,"num_write_kb":56964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"1"}]},{"pgid":"2.2f","version":"823'726","reported_seq":4473,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-09-16T05:23:15.189299+0000","last_change":"2024-09-16T05:23:15.189991+0000","last_active":"2024-09-16T05:23:15.189299+0000","last_peered":"2024-09-16T05:23:15.189299+0000","last_clean":"2024-09-16T05:23:15.189299+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:23:15.189299+0000","last_undegraded":"2024-09-16T05:23:15.189299+0000","last_fullsized":"2024-09-16T05:23:15.189299+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"795'683","last_scrub_stamp":"2024-09-16T05:22:27.948411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.948411+0000","objects_scrubbed":5,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:28.079674+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0069810419999999998,"stat_sum":{"num_bytes":6420802,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":100606,"num_write":148,"num_write_kb":54423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"15f","length":"1"},{"start":"161","length":"2"},{"start":"164","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":96,"stat_sum":{"num_bytes":117719160,"num_objects":44,"num_object_clones":8,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":3,"num_read":3700,"num_read_kb":3695807,"num_write":2605,"num_write_kb":1079974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":157294592,"data_stored":169054554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33317,"ondisk_log_size":33317,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089514,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80128,"kb_used_data":22564,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93665024,"statfs":{"total":95995035648,"available":95912984576,"internally_reserved":0,"allocated":23105536,"data_stored":19621189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":58915268},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285743,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88076,"kb_used_data":30512,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93657076,"statfs":{"total":95995035648,"available":95904845824,"internally_reserved":0,"allocated":31244288,"data_stored":31489867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514681,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72336,"kb_used_data":23096,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93672816,"statfs":{"total":95995035648,"available":95920963584,"internally_reserved":0,"allocated":23650304,"data_stored":20220507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841720,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84912,"kb_used_data":27352,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93660240,"statfs":{"total":95995035648,"available":95908085760,"internally_reserved":0,"allocated":28008448,"data_stored":24544031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103357,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68388,"kb_used_data":19144,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93676764,"statfs":{"total":95995035648,"available":95925006336,"internally_reserved":0,"allocated":19603456,"data_stored":16199387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397700,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66816,"kb_used_data":21732,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93678336,"statfs":{"total":95995035648,"available":95926616064,"internally_reserved":0,"allocated":22253568,"data_stored":18847950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757448,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61388,"kb_used_data":26036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93683764,"statfs":{"total":95995035648,"available":95932174336,"internally_reserved":0,"allocated":26660864,"data_stored":25707371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920973,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51420,"kb_used_data":16072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93693732,"statfs":{"total":95995035648,"available":95942381568,"internally_reserved":0,"allocated":16457728,"data_stored":13029763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117201,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73468,"kb_used_data":28384,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93671684,"statfs":{"total":95995035648,"available":95919804416,"internally_reserved":0,"allocated":29065216,"data_stored":27197655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240488,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49084,"kb_used_data":20964,"kb_used_omap":10,"kb_used_meta":28085,"kb_avail":93696068,"statfs":{"total":95995035648,"available":95944773632,"internally_reserved":0,"allocated":21467136,"data_stored":19014529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":28759347},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575068,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75988,"kb_used_data":18428,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93669164,"statfs":{"total":95995035648,"available":95917223936,"internally_reserved":0,"allocated":18870272,"data_stored":17990722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738593,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73608,"kb_used_data":20208,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93671544,"statfs":{"total":95995035648,"available":95919661056,"internally_reserved":0,"allocated":20692992,"data_stored":18466611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10133504,"data_stored":11285592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8638464,"data_stored":11137383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10993664,"data_stored":11915398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18530304,"data_stored":20033532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6213632,"data_stored":6167968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16211968,"data_stored":18644328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12046336,"data_stored":12011507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9408512,"data_stored":9371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":17788928,"data_stored":17699148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13418496,"data_stored":13367168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":21000192,"data_stored":24628072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12910592,"data_stored":12793130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:23:17.421 INFO:tasks.thrashosds.thrasher:Setting 1.11 to [6, 1, 2, 5] 2024-09-16T05:23:17.421 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.11', '6', '1', '2', '5'] 2024-09-16T05:23:17.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 1.11 6 1 2 5 2024-09-16T05:23:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:23:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:23:17.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:23:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:23:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: 2.43 scrub starts 2024-09-16T05:23:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: 2.43 scrub ok 2024-09-16T05:23:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: 2.5e scrub starts 2024-09-16T05:23:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: 2.5e scrub ok 2024-09-16T05:23:17.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:17 smithi204 ceph-mon[30933]: osdmap e847: 12 total, 12 up, 12 in 2024-09-16T05:23:17.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:23:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:23:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:23:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:23:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: 2.43 scrub starts 2024-09-16T05:23:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: 2.43 scrub ok 2024-09-16T05:23:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: 2.5e scrub starts 2024-09-16T05:23:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: 2.5e scrub ok 2024-09-16T05:23:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:17 smithi196 ceph-mon[30802]: osdmap e847: 12 total, 12 up, 12 in 2024-09-16T05:23:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:17.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:17.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:17.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:17.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:17.700 INFO:tasks.rados.rados.2.smithi204.stdout:3355: done (5 left) 2024-09-16T05:23:17.700 INFO:tasks.rados.rados.2.smithi204.stdout:3356: done (4 left) 2024-09-16T05:23:17.700 INFO:tasks.rados.rados.2.smithi204.stdout:3357: done (3 left) 2024-09-16T05:23:17.700 INFO:tasks.rados.rados.2.smithi204.stdout:3358: done (2 left) 2024-09-16T05:23:17.700 INFO:tasks.rados.rados.2.smithi204.stdout:3359: done (1 left) 2024-09-16T05:23:17.700 INFO:tasks.rados.rados.2.smithi204.stdout:3360: done (0 left) 2024-09-16T05:23:17.700 INFO:tasks.rados.rados.2.smithi204.stdout:3361: copy_from oid 18 from oid 2 current snap is 356 2024-09-16T05:23:17.700 INFO:tasks.rados.rados.2.smithi204.stdout:3362: rollback oid 47 current snap is 356 2024-09-16T05:23:17.701 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 47 to 348 2024-09-16T05:23:17.701 INFO:tasks.rados.rados.2.smithi204.stdout:3363: read oid 8 snap 351 2024-09-16T05:23:17.701 INFO:tasks.rados.rados.2.smithi204.stdout:3363: expect deleted 2024-09-16T05:23:17.701 INFO:tasks.rados.rados.2.smithi204.stdout:3364: snap_create 2024-09-16T05:23:17.701 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:17.702 INFO:tasks.rados.rados.2.smithi204.stdout:3362: finishing rollback tid 0 to smithi20457400-47 2024-09-16T05:23:17.702 INFO:tasks.rados.rados.2.smithi204.stdout:3362: finishing rollback tid 1 to smithi20457400-47 2024-09-16T05:23:17.702 INFO:tasks.rados.rados.2.smithi204.stdout:3362: finishing rollback tid 2 to smithi20457400-47 2024-09-16T05:23:17.702 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1069 (ObjNum 1142 snap 341 seq_num 1142) dirty exists 2024-09-16T05:23:17.712 INFO:tasks.rados.rados.2.smithi204.stdout:3361: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:23:17.715 INFO:tasks.rados.rados.2.smithi204.stdout:3361: finishing copy_from to smithi20457400-18 2024-09-16T05:23:17.715 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 549 (ObjNum 1168 snap 354 seq_num 1168) dirty exists 2024-09-16T05:23:17.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:17.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:17.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:17.874 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:17.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:17.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:18.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:18.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:18.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:23:18.003 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3361: done (3 left) 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3362: done (2 left) 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3363: done (1 left) 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3364: done (0 left) 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3365: rollback oid 30 current snap is 357 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 30 to 348 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3366: copy_from oid 27 from oid 2 current snap is 357 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3367: copy_from oid 13 from oid 7 current snap is 357 2024-09-16T05:23:18.009 INFO:tasks.rados.rados.2.smithi204.stdout:3368: snap_remove snap 356 2024-09-16T05:23:18.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:18.125 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:18.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:18.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:18.251 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:18.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:18.353 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:18 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:23:18.353 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:18 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:23:18.353 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:18 smithi032 ceph-mon[27388]: pgmap v1431: 127 pgs: 127 active+clean; 113 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-16T05:23:18.353 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:18 smithi032 ceph-mon[27388]: from='client.17322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:23:18.354 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:18 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:23:18.354 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:18 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:23:18.354 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3659422864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:23:18.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:18.378 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:18.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:18.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:18 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:23:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:18 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:23:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:18 smithi204 ceph-mon[30933]: pgmap v1431: 127 pgs: 127 active+clean; 113 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-16T05:23:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:18 smithi204 ceph-mon[30933]: from='client.17322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:23:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:18 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:23:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:18 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:23:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3659422864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:23:18.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:18.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:18 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:23:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:18 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:23:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:18 smithi196 ceph-mon[30802]: pgmap v1431: 127 pgs: 127 active+clean; 113 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-16T05:23:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:18 smithi196 ceph-mon[30802]: from='client.17322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:23:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:18 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:23:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:18 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:23:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3659422864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:23:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:18.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:18.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:18.630 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:18.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:18.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:18.757 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:18.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:18.778 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:18.778 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:18.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:18.882 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:18.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:18.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:18.985 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:19.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:19.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:19.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:19.019 INFO:tasks.rados.rados.2.smithi204.stdout:3368: done (3 left) 2024-09-16T05:23:19.019 INFO:tasks.rados.rados.2.smithi204.stdout:3369: read oid 14 snap -1 2024-09-16T05:23:19.019 INFO:tasks.rados.rados.2.smithi204.stdout:3369: expect deleted 2024-09-16T05:23:19.019 INFO:tasks.rados.rados.2.smithi204.stdout:3370: read oid 38 snap -1 2024-09-16T05:23:19.019 INFO:tasks.rados.rados.2.smithi204.stdout:3370: expect deleted 2024-09-16T05:23:19.019 INFO:tasks.rados.rados.2.smithi204.stdout:3371: copy_from oid 25 from oid 8 current snap is 357 2024-09-16T05:23:19.019 INFO:tasks.rados.rados.2.smithi204.stdout:3372: snap_remove snap 355 2024-09-16T05:23:19.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:19.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:19.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:19.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:19.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:19.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:19.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:19.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:19 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:23:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:19 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:23:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:19 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3659422864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:23:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:19 smithi032 ceph-mon[27388]: osdmap e848: 12 total, 12 up, 12 in 2024-09-16T05:23:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:19 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:23:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:19 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:23:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:19 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3659422864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:23:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:19 smithi204 ceph-mon[30933]: osdmap e848: 12 total, 12 up, 12 in 2024-09-16T05:23:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:19 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:23:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:19 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:23:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:19 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3659422864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:23:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:19 smithi196 ceph-mon[30802]: osdmap e848: 12 total, 12 up, 12 in 2024-09-16T05:23:19.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:19.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:23:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:19.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:19.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:19.545 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:19.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:19.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:19.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:19.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:19.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:19.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:19.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:19.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:19.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:19.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:20.024 INFO:tasks.rados.rados.2.smithi204.stdout:3372: done (6 left) 2024-09-16T05:23:20.024 INFO:tasks.rados.rados.2.smithi204.stdout:3373: read oid 6 snap 348 2024-09-16T05:23:20.024 INFO:tasks.rados.rados.2.smithi204.stdout:3373: expect (ObjNum 1148 snap 342 seq_num 1148) 2024-09-16T05:23:20.024 INFO:tasks.rados.rados.2.smithi204.stdout:3374: copy_from oid 19 from oid 32 current snap is 357 2024-09-16T05:23:20.024 INFO:tasks.rados.rados.2.smithi204.stdout:3365: finishing rollback tid 0 to smithi20457400-30 2024-09-16T05:23:20.024 INFO:tasks.rados.rados.2.smithi204.stdout:3365: finishing rollback tid 1 to smithi20457400-30 2024-09-16T05:23:20.024 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 1462 (ObjNum -1350591248 snap 32571 seq_num 304) dirty dne 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:3367: finishing copy_from racing read to smithi20457400-13 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:3367: finishing copy_from to smithi20457400-13 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 1150 (ObjNum 1131 snap 338 seq_num 1131) dirty exists 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:3366: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:3366: finishing copy_from to smithi20457400-27 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 549 (ObjNum 1168 snap 354 seq_num 1168) dirty exists 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:3371: finishing copy_from to smithi20457400-25 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:3371: got expected ENOENT (src dne) 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:3371: finishing copy_from racing read to smithi20457400-25 2024-09-16T05:23:20.025 INFO:tasks.rados.rados.2.smithi204.stdout:3365: done (7 left) 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3366: done (6 left) 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3367: done (5 left) 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3369: done (4 left) 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3370: done (3 left) 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3371: done (2 left) 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3375: read oid 42 snap -1 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3375: expect deleted 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3376: read oid 27 snap -1 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3376: expect (ObjNum 1168 snap 354 seq_num 1168) 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3377: copy_from oid 35 from oid 25 current snap is 357 2024-09-16T05:23:20.026 INFO:tasks.rados.rados.2.smithi204.stdout:3378: delete oid 33 current snap is 357 2024-09-16T05:23:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:20.027 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:20.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:20.029 INFO:tasks.rados.rados.2.smithi204.stdout:3375: done (5 left) 2024-09-16T05:23:20.029 INFO:tasks.rados.rados.2.smithi204.stdout:3378: done (4 left) 2024-09-16T05:23:20.029 INFO:tasks.rados.rados.2.smithi204.stdout:3379: delete oid 47 current snap is 357 2024-09-16T05:23:20.033 INFO:tasks.rados.rados.2.smithi204.stdout:3379: done (4 left) 2024-09-16T05:23:20.033 INFO:tasks.rados.rados.2.smithi204.stdout:3380: snap_create 2024-09-16T05:23:20.040 INFO:tasks.rados.rados.2.smithi204.stdout:3374: finishing copy_from racing read to smithi20457400-19 2024-09-16T05:23:20.044 INFO:tasks.rados.rados.2.smithi204.stdout:3376: expect (ObjNum 1168 snap 354 seq_num 1168) 2024-09-16T05:23:20.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:20.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:20 smithi032 ceph-mon[27388]: pgmap v1434: 127 pgs: 127 active+clean; 119 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 11 op/s 2024-09-16T05:23:20.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:20 smithi032 ceph-mon[27388]: osdmap e849: 12 total, 12 up, 12 in 2024-09-16T05:23:20.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:20.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:20.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:20.271 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:20.273 INFO:tasks.rados.rados.2.smithi204.stdout:3377: finishing copy_from racing read to smithi20457400-35 2024-09-16T05:23:20.273 INFO:tasks.rados.rados.2.smithi204.stdout:3374: finishing copy_from to smithi20457400-19 2024-09-16T05:23:20.273 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 1206 (ObjNum 1109 snap 336 seq_num 1109) dirty exists 2024-09-16T05:23:20.273 INFO:tasks.rados.rados.2.smithi204.stdout:3377: finishing copy_from to smithi20457400-35 2024-09-16T05:23:20.273 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 1271 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-09-16T05:23:20.273 INFO:tasks.rados.rados.2.smithi204.stdout:3373: expect (ObjNum 1148 snap 342 seq_num 1148) 2024-09-16T05:23:20.279 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:20.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:20.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:20.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:20.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:20.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:20 smithi204 ceph-mon[30933]: pgmap v1434: 127 pgs: 127 active+clean; 119 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 11 op/s 2024-09-16T05:23:20.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:20 smithi204 ceph-mon[30933]: osdmap e849: 12 total, 12 up, 12 in 2024-09-16T05:23:20.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:20 smithi196 ceph-mon[30802]: pgmap v1434: 127 pgs: 127 active+clean; 119 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.3 MiB/s wr, 11 op/s 2024-09-16T05:23:20.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:20 smithi196 ceph-mon[30802]: osdmap e849: 12 total, 12 up, 12 in 2024-09-16T05:23:20.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:20.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:20.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:20.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:20.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:20.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:23:20.725 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:20.725 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:23:20.725 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:20.725 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:20.726 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:20.733 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:20.733 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:20.754 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:20.754 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:20.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:20.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:20.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:set 1.11 pg_upmap_items mapping to [6->1,2->5] 2024-09-16T05:23:20.843 INFO:tasks.rados.rados.2.smithi204.stdout:3373: done (4 left) 2024-09-16T05:23:20.843 INFO:tasks.rados.rados.2.smithi204.stdout:3374: done (3 left) 2024-09-16T05:23:20.843 INFO:tasks.rados.rados.2.smithi204.stdout:3376: done (2 left) 2024-09-16T05:23:20.843 INFO:tasks.rados.rados.2.smithi204.stdout:3377: done (1 left) 2024-09-16T05:23:20.843 INFO:tasks.rados.rados.2.smithi204.stdout:3380: done (0 left) 2024-09-16T05:23:20.843 INFO:tasks.rados.rados.2.smithi204.stdout:3381: snap_remove snap 348 2024-09-16T05:23:20.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:20.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:21.010 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:23:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:23:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:23:21.010 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:23:21.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:21.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:21.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:21.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:21.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:21.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:21.287 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:21.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:21.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:21.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:21.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:21.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:21 smithi032 ceph-mon[27388]: osdmap e850: 12 total, 12 up, 12 in 2024-09-16T05:23:21.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2198535764' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.11", "id": [6, 1, 2, 5]}]: dispatch 2024-09-16T05:23:21.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/656747333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:21.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2198535764' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.11", "id": [6, 1, 2, 5]}]': finished 2024-09-16T05:23:21.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:21 smithi032 ceph-mon[27388]: osdmap e851: 12 total, 12 up, 12 in 2024-09-16T05:23:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:21 smithi204 ceph-mon[30933]: osdmap e850: 12 total, 12 up, 12 in 2024-09-16T05:23:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2198535764' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.11", "id": [6, 1, 2, 5]}]: dispatch 2024-09-16T05:23:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/656747333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2198535764' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.11", "id": [6, 1, 2, 5]}]': finished 2024-09-16T05:23:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:21 smithi204 ceph-mon[30933]: osdmap e851: 12 total, 12 up, 12 in 2024-09-16T05:23:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:21 smithi196 ceph-mon[30802]: osdmap e850: 12 total, 12 up, 12 in 2024-09-16T05:23:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2198535764' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.11", "id": [6, 1, 2, 5]}]: dispatch 2024-09-16T05:23:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/656747333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2198535764' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.11", "id": [6, 1, 2, 5]}]': finished 2024-09-16T05:23:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:21 smithi196 ceph-mon[30802]: osdmap e851: 12 total, 12 up, 12 in 2024-09-16T05:23:21.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:21.540 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:21.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:21.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:21.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:21.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:21.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:21.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:21.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:21.846 INFO:tasks.rados.rados.2.smithi204.stdout:3381: done (0 left) 2024-09-16T05:23:21.846 INFO:tasks.rados.rados.2.smithi204.stdout:3382: read oid 39 snap -1 2024-09-16T05:23:21.846 INFO:tasks.rados.rados.2.smithi204.stdout:3382: expect deleted 2024-09-16T05:23:21.846 INFO:tasks.rados.rados.2.smithi204.stdout:3383: rollback oid 42 current snap is 358 2024-09-16T05:23:21.846 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 351 2024-09-16T05:23:21.846 INFO:tasks.rados.rados.2.smithi204.stdout:3384: rollback oid 32 current snap is 358 2024-09-16T05:23:21.846 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 357 2024-09-16T05:23:21.846 INFO:tasks.rados.rados.2.smithi204.stdout:3385: read oid 37 snap -1 2024-09-16T05:23:21.847 INFO:tasks.rados.rados.2.smithi204.stdout:3385: expect (ObjNum 1144 snap 341 seq_num 1144) 2024-09-16T05:23:21.847 INFO:tasks.rados.rados.2.smithi204.stdout:3386: delete oid 9 current snap is 358 2024-09-16T05:23:21.848 INFO:tasks.rados.rados.2.smithi204.stdout:3383: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:23:21.849 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 145 snap 0 seq_num 2944376160) dirty dne 2024-09-16T05:23:21.852 INFO:tasks.rados.rados.2.smithi204.stdout:3384: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:23:21.854 INFO:tasks.rados.rados.2.smithi204.stdout:3384: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:23:21.854 INFO:tasks.rados.rados.2.smithi204.stdout:3384: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:23:21.854 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1228 (ObjNum 1109 snap 336 seq_num 1109) dirty exists 2024-09-16T05:23:21.855 INFO:tasks.rados.rados.2.smithi204.stdout:3382: done (4 left) 2024-09-16T05:23:21.855 INFO:tasks.rados.rados.2.smithi204.stdout:3383: done (3 left) 2024-09-16T05:23:21.855 INFO:tasks.rados.rados.2.smithi204.stdout:3384: done (2 left) 2024-09-16T05:23:21.855 INFO:tasks.rados.rados.2.smithi204.stdout:3386: done (1 left) 2024-09-16T05:23:21.855 INFO:tasks.rados.rados.2.smithi204.stdout:3387: copy_from oid 29 from oid 24 current snap is 358 2024-09-16T05:23:21.855 INFO:tasks.rados.rados.2.smithi204.stdout:3388: snap_create 2024-09-16T05:23:21.856 INFO:tasks.rados.rados.2.smithi204.stdout:3385: expect (ObjNum 1144 snap 341 seq_num 1144) 2024-09-16T05:23:21.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:21.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:22.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:22.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:22.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:22.072 INFO:tasks.rados.rados.2.smithi204.stdout:3387: finishing copy_from to smithi20457400-29 2024-09-16T05:23:22.072 INFO:tasks.rados.rados.2.smithi204.stdout:3387: got expected ENOENT (src dne) 2024-09-16T05:23:22.072 INFO:tasks.rados.rados.2.smithi204.stdout:3387: finishing copy_from racing read to smithi20457400-29 2024-09-16T05:23:22.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:22.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:22.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:22.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: pgmap v1437: 127 pgs: 127 active+clean; 119 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: osdmap e852: 12 total, 12 up, 12 in 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]: dispatch 2024-09-16T05:23:22.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]: dispatch 2024-09-16T05:23:22.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:22 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]: dispatch 2024-09-16T05:23:22.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:22.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:22.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:22.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: pgmap v1437: 127 pgs: 127 active+clean; 119 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-16T05:23:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: osdmap e852: 12 total, 12 up, 12 in 2024-09-16T05:23:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]: dispatch 2024-09-16T05:23:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]: dispatch 2024-09-16T05:23:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]: dispatch 2024-09-16T05:23:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]: dispatch 2024-09-16T05:23:22.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:22 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]: dispatch 2024-09-16T05:23:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:22.424 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:22.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: pgmap v1437: 127 pgs: 127 active+clean; 119 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-16T05:23:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: osdmap e852: 12 total, 12 up, 12 in 2024-09-16T05:23:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]: dispatch 2024-09-16T05:23:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]: dispatch 2024-09-16T05:23:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]: dispatch 2024-09-16T05:23:22.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]: dispatch 2024-09-16T05:23:22.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]: dispatch 2024-09-16T05:23:22.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]: dispatch 2024-09-16T05:23:22.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]: dispatch 2024-09-16T05:23:22.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]: dispatch 2024-09-16T05:23:22.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]: dispatch 2024-09-16T05:23:22.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]: dispatch 2024-09-16T05:23:22.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]: dispatch 2024-09-16T05:23:22.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:22 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]: dispatch 2024-09-16T05:23:22.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:23:22.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:22.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:22.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:22.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:22.676 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:22.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:22.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:22.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:22.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:22.850 INFO:tasks.rados.rados.2.smithi204.stdout:3385: done (2 left) 2024-09-16T05:23:22.850 INFO:tasks.rados.rados.2.smithi204.stdout:3387: done (1 left) 2024-09-16T05:23:22.850 INFO:tasks.rados.rados.2.smithi204.stdout:3388: done (0 left) 2024-09-16T05:23:22.850 INFO:tasks.rados.rados.2.smithi204.stdout:3389: read oid 10 snap 354 2024-09-16T05:23:22.850 INFO:tasks.rados.rados.2.smithi204.stdout:3389: expect deleted 2024-09-16T05:23:22.850 INFO:tasks.rados.rados.2.smithi204.stdout:3390: write (excl) oid 14 current snap is 359 2024-09-16T05:23:22.850 INFO:tasks.rados.rados.2.smithi204.stdout:3390: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2024-09-16T05:23:22.856 INFO:tasks.rados.rados.2.smithi204.stdout:3390: writing smithi20457400-14 from 531418 to 1192106 tid 1 2024-09-16T05:23:22.863 INFO:tasks.rados.rados.2.smithi204.stdout:3390: writing smithi20457400-14 from 1982919 to 2435325 tid 2 2024-09-16T05:23:22.870 INFO:tasks.rados.rados.2.smithi204.stdout:3390: writing smithi20457400-14 from 3039770 to 3731418 tid 3 2024-09-16T05:23:22.870 INFO:tasks.rados.rados.2.smithi204.stdout:3391: read oid 27 snap -1 2024-09-16T05:23:22.870 INFO:tasks.rados.rados.2.smithi204.stdout:3391: expect (ObjNum 1168 snap 354 seq_num 1168) 2024-09-16T05:23:22.871 INFO:tasks.rados.rados.2.smithi204.stdout:3392: snap_remove snap 357 2024-09-16T05:23:22.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:22.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:23.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:23.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:23.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:23.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:23.183 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:23.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:23.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:23.309 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:23.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:23.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:23.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:23.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:23.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:23.549 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:23.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:23.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:23.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:23.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:23.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:23.812 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:23.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:23.846 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:23:23.850 INFO:tasks.rados.rados.2.smithi204.stdout:3392: done (3 left) 2024-09-16T05:23:23.850 INFO:tasks.rados.rados.2.smithi204.stdout:3393: copy_from oid 3 from oid 48 current snap is 359 2024-09-16T05:23:23.850 INFO:tasks.rados.rados.2.smithi204.stdout:3390: finishing write tid 1 to smithi20457400-14 2024-09-16T05:23:23.851 INFO:tasks.rados.rados.2.smithi204.stdout:3390: finishing write tid 2 to smithi20457400-14 2024-09-16T05:23:23.851 INFO:tasks.rados.rados.2.smithi204.stdout:3390: finishing write tid 3 to smithi20457400-14 2024-09-16T05:23:23.851 INFO:tasks.rados.rados.2.smithi204.stdout:3390: finishing write tid 4 to smithi20457400-14 2024-09-16T05:23:23.851 INFO:tasks.rados.rados.2.smithi204.stdout:3390: finishing write tid 5 to smithi20457400-14 2024-09-16T05:23:23.851 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1472 (ObjNum 1174 snap 359 seq_num 1174) dirty exists 2024-09-16T05:23:23.851 INFO:tasks.rados.rados.2.smithi204.stdout:3390: left oid 14 (ObjNum 1174 snap 359 seq_num 1174) 2024-09-16T05:23:23.851 INFO:tasks.rados.rados.2.smithi204.stdout:3391: expect (ObjNum 1168 snap 354 seq_num 1168) 2024-09-16T05:23:23.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:23:23.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:23.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: pgmap v1440: 127 pgs: 127 active+clean; 134 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-16T05:23:23.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]': finished 2024-09-16T05:23:23.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]': finished 2024-09-16T05:23:23.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]': finished 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: osdmap e853: 12 total, 12 up, 12 in 2024-09-16T05:23:23.925 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1136076008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:23.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:23.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:23.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:24.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:24.062 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:24.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:24.085 INFO:tasks.rados.rados.2.smithi204.stdout:3393: finishing copy_from to smithi20457400-3 2024-09-16T05:23:24.085 INFO:tasks.rados.rados.2.smithi204.stdout:3393: got expected ENOENT (src dne) 2024-09-16T05:23:24.086 INFO:tasks.rados.rados.2.smithi204.stdout:3393: finishing copy_from racing read to smithi20457400-3 2024-09-16T05:23:24.086 INFO:tasks.rados.rados.2.smithi204.stdout:3389: done (3 left) 2024-09-16T05:23:24.086 INFO:tasks.rados.rados.2.smithi204.stdout:3390: done (2 left) 2024-09-16T05:23:24.086 INFO:tasks.rados.rados.2.smithi204.stdout:3391: done (1 left) 2024-09-16T05:23:24.086 INFO:tasks.rados.rados.2.smithi204.stdout:3393: done (0 left) 2024-09-16T05:23:24.086 INFO:tasks.rados.rados.2.smithi204.stdout:3394: snap_remove snap 351 2024-09-16T05:23:24.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:24.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: pgmap v1440: 127 pgs: 127 active+clean; 134 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]': finished 2024-09-16T05:23:24.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]': finished 2024-09-16T05:23:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: osdmap e853: 12 total, 12 up, 12 in 2024-09-16T05:23:24.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1136076008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:24.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:24.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: pgmap v1440: 127 pgs: 127 active+clean; 134 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714115646917095]}]': finished 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07800222499201677]}]': finished 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.1268011032477532]}]': finished 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09456908415270396]}]': finished 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08969874631535538]}]': finished 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.088053400862521]}]': finished 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08076993473063149]}]': finished 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.059543757417121486]}]': finished 2024-09-16T05:23:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959190537881893]}]': finished 2024-09-16T05:23:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506986199652171]}]': finished 2024-09-16T05:23:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.0708498624967084]}]': finished 2024-09-16T05:23:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745534865942664]}]': finished 2024-09-16T05:23:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: osdmap e853: 12 total, 12 up, 12 in 2024-09-16T05:23:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1136076008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:24.271 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:23:24.271 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:23:24.271 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:23:24.272 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:23:24.272 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:23:24.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:24.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:24.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:24.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:24.441 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:24.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:24.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:24.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:24.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:24.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:24.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:24.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:24.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:24.821 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:24.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:24.868 INFO:tasks.rados.rados.2.smithi204.stdout:3394: done (0 left) 2024-09-16T05:23:24.869 INFO:tasks.rados.rados.2.smithi204.stdout:3395: read oid 6 snap -1 2024-09-16T05:23:24.869 INFO:tasks.rados.rados.2.smithi204.stdout:3395: expect (ObjNum 1148 snap 342 seq_num 1148) 2024-09-16T05:23:24.869 INFO:tasks.rados.rados.2.smithi204.stdout:3396: delete oid 10 current snap is 359 2024-09-16T05:23:24.875 INFO:tasks.rados.rados.2.smithi204.stdout:3396: done (1 left) 2024-09-16T05:23:24.875 INFO:tasks.rados.rados.2.smithi204.stdout:3397: read oid 10 snap -1 2024-09-16T05:23:24.875 INFO:tasks.rados.rados.2.smithi204.stdout:3397: expect deleted 2024-09-16T05:23:24.875 INFO:tasks.rados.rados.2.smithi204.stdout:3398: rollback oid 45 current snap is 359 2024-09-16T05:23:24.875 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 354 2024-09-16T05:23:24.876 INFO:tasks.rados.rados.2.smithi204.stdout:3399: snap_create 2024-09-16T05:23:24.879 INFO:tasks.rados.rados.2.smithi204.stdout:3395: expect (ObjNum 1148 snap 342 seq_num 1148) 2024-09-16T05:23:24.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:24.944 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:24.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:24.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:23:25.020 INFO:tasks.rados.rados.2.smithi204.stdout:3398: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:23:25.020 INFO:tasks.rados.rados.2.smithi204.stdout:3398: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:23:25.021 INFO:tasks.rados.rados.2.smithi204.stdout:3398: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:23:25.021 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1269 (ObjNum 1139 snap 341 seq_num 1139) dirty exists 2024-09-16T05:23:25.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:25.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:25.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:25.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:24 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:25.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1136076008' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:25.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:24 smithi032 ceph-mon[27388]: osdmap e854: 12 total, 12 up, 12 in 2024-09-16T05:23:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:24 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1136076008' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:25.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:24 smithi204 ceph-mon[30933]: osdmap e854: 12 total, 12 up, 12 in 2024-09-16T05:23:25.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:25.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:25.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:24 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1136076008' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:24 smithi196 ceph-mon[30802]: osdmap e854: 12 total, 12 up, 12 in 2024-09-16T05:23:25.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:25.324 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:25.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:25.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:25.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:25.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:25.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:25.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:25.575 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:25.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:25.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:25.703 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:25.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:25.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:25.827 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:25.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:25.848 INFO:tasks.rados.rados.2.smithi204.stdout:3395: done (3 left) 2024-09-16T05:23:25.848 INFO:tasks.rados.rados.2.smithi204.stdout:3397: done (2 left) 2024-09-16T05:23:25.848 INFO:tasks.rados.rados.2.smithi204.stdout:3398: done (1 left) 2024-09-16T05:23:25.848 INFO:tasks.rados.rados.2.smithi204.stdout:3399: done (0 left) 2024-09-16T05:23:25.848 INFO:tasks.rados.rados.2.smithi204.stdout:3400: write (excl) oid 30 current snap is 360 2024-09-16T05:23:25.849 INFO:tasks.rados.rados.2.smithi204.stdout:3400: seq_num 1176 ranges {565032=411186,1583530=456075,2578263=758593} 2024-09-16T05:23:25.853 INFO:tasks.rados.rados.2.smithi204.stdout:3400: writing smithi20457400-30 from 565032 to 976218 tid 1 2024-09-16T05:23:25.860 INFO:tasks.rados.rados.2.smithi204.stdout:3400: writing smithi20457400-30 from 1583530 to 2039605 tid 2 2024-09-16T05:23:25.868 INFO:tasks.rados.rados.2.smithi204.stdout:3400: writing smithi20457400-30 from 2578263 to 3336856 tid 3 2024-09-16T05:23:25.868 INFO:tasks.rados.rados.2.smithi204.stdout:3401: write (excl) oid 46 current snap is 360 2024-09-16T05:23:25.868 INFO:tasks.rados.rados.2.smithi204.stdout:3400: finishing write tid 1 to smithi20457400-30 2024-09-16T05:23:25.868 INFO:tasks.rados.rados.2.smithi204.stdout:3400: finishing write tid 2 to smithi20457400-30 2024-09-16T05:23:25.868 INFO:tasks.rados.rados.2.smithi204.stdout:3401: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2024-09-16T05:23:25.873 INFO:tasks.rados.rados.2.smithi204.stdout:3401: writing smithi20457400-46 from 581839 to 1068274 tid 1 2024-09-16T05:23:25.879 INFO:tasks.rados.rados.2.smithi204.stdout:3401: writing smithi20457400-46 from 1642012 to 2241745 tid 2 2024-09-16T05:23:25.887 INFO:tasks.rados.rados.2.smithi204.stdout:3401: writing smithi20457400-46 from 3005686 to 3575551 tid 3 2024-09-16T05:23:25.887 INFO:tasks.rados.rados.2.smithi204.stdout:3402: rollback oid 42 current snap is 360 2024-09-16T05:23:25.887 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 42 to 359 2024-09-16T05:23:25.887 INFO:tasks.rados.rados.2.smithi204.stdout:3400: finishing write tid 3 to smithi20457400-30 2024-09-16T05:23:25.887 INFO:tasks.rados.rados.2.smithi204.stdout:3400: finishing write tid 4 to smithi20457400-30 2024-09-16T05:23:25.887 INFO:tasks.rados.rados.2.smithi204.stdout:3403: rollback oid 49 current snap is 360 2024-09-16T05:23:25.887 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 354 2024-09-16T05:23:25.888 INFO:tasks.rados.rados.2.smithi204.stdout:3400: finishing write tid 5 to smithi20457400-30 2024-09-16T05:23:25.888 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 30 v 1476 (ObjNum 1176 snap 360 seq_num 1176) dirty exists 2024-09-16T05:23:25.888 INFO:tasks.rados.rados.2.smithi204.stdout:3400: left oid 30 (ObjNum 1176 snap 360 seq_num 1176) 2024-09-16T05:23:25.888 INFO:tasks.rados.rados.2.smithi204.stdout:3401: finishing write tid 1 to smithi20457400-46 2024-09-16T05:23:25.888 INFO:tasks.rados.rados.2.smithi204.stdout:3401: finishing write tid 2 to smithi20457400-46 2024-09-16T05:23:25.888 INFO:tasks.rados.rados.2.smithi204.stdout:3400: done (3 left) 2024-09-16T05:23:25.888 INFO:tasks.rados.rados.2.smithi204.stdout:3404: write (excl) oid 29 current snap is 360 2024-09-16T05:23:25.888 INFO:tasks.rados.rados.2.smithi204.stdout:3404: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2024-09-16T05:23:25.892 INFO:tasks.rados.rados.2.smithi204.stdout:3404: writing smithi20457400-29 from 598646 to 1160330 tid 1 2024-09-16T05:23:25.898 INFO:tasks.rados.rados.2.smithi204.stdout:3404: writing smithi20457400-29 from 1700494 to 2160238 tid 2 2024-09-16T05:23:25.906 INFO:tasks.rados.rados.2.smithi204.stdout:3404: writing smithi20457400-29 from 2633109 to 3414246 tid 3 2024-09-16T05:23:25.906 INFO:tasks.rados.rados.2.smithi204.stdout:3405: delete oid 6 current snap is 360 2024-09-16T05:23:25.906 INFO:tasks.rados.rados.2.smithi204.stdout:3402: finishing rollback tid 1 to smithi20457400-42 2024-09-16T05:23:25.906 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 81 snap 0 seq_num 2671656119) dirty dne 2024-09-16T05:23:25.906 INFO:tasks.rados.rados.2.smithi204.stdout:3403: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:23:25.906 INFO:tasks.rados.rados.2.smithi204.stdout:3403: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:23:25.906 INFO:tasks.rados.rados.2.smithi204.stdout:3401: finishing write tid 3 to smithi20457400-46 2024-09-16T05:23:25.907 INFO:tasks.rados.rados.2.smithi204.stdout:3401: finishing write tid 4 to smithi20457400-46 2024-09-16T05:23:25.907 INFO:tasks.rados.rados.2.smithi204.stdout:3401: finishing write tid 5 to smithi20457400-46 2024-09-16T05:23:25.907 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1032 (ObjNum 1177 snap 360 seq_num 1177) dirty exists 2024-09-16T05:23:25.907 INFO:tasks.rados.rados.2.smithi204.stdout:3401: left oid 46 (ObjNum 1177 snap 360 seq_num 1177) 2024-09-16T05:23:25.907 INFO:tasks.rados.rados.2.smithi204.stdout:3404: finishing write tid 1 to smithi20457400-29 2024-09-16T05:23:25.907 INFO:tasks.rados.rados.2.smithi204.stdout:3403: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:23:25.907 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 674 (ObjNum 921 snap 273 seq_num 921) dirty exists 2024-09-16T05:23:25.907 INFO:tasks.rados.rados.2.smithi204.stdout:3404: finishing write tid 2 to smithi20457400-29 2024-09-16T05:23:25.908 INFO:tasks.rados.rados.2.smithi204.stdout:3401: done (4 left) 2024-09-16T05:23:25.908 INFO:tasks.rados.rados.2.smithi204.stdout:3402: done (3 left) 2024-09-16T05:23:25.908 INFO:tasks.rados.rados.2.smithi204.stdout:3403: done (2 left) 2024-09-16T05:23:25.908 INFO:tasks.rados.rados.2.smithi204.stdout:3405: done (1 left) 2024-09-16T05:23:25.908 INFO:tasks.rados.rados.2.smithi204.stdout:3406: write oid 20 current snap is 360 2024-09-16T05:23:25.908 INFO:tasks.rados.rados.2.smithi204.stdout:3406: seq_num 1180 ranges {632260=428535,1817458=463413,2687955=520034,3709324=122936} 2024-09-16T05:23:25.913 INFO:tasks.rados.rados.2.smithi204.stdout:3406: writing smithi20457400-20 from 632260 to 1060795 tid 1 2024-09-16T05:23:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:25 smithi032 ceph-mon[27388]: pgmap v1443: 127 pgs: 127 active+clean; 130 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.7 MiB/s wr, 9 op/s 2024-09-16T05:23:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:25 smithi032 ceph-mon[27388]: osdmap e855: 12 total, 12 up, 12 in 2024-09-16T05:23:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:25 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:23:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:25 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:23:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:25 smithi032 ceph-mon[27388]: osdmap e856: 12 total, 12 up, 12 in 2024-09-16T05:23:25.921 INFO:tasks.rados.rados.2.smithi204.stdout:3406: writing smithi20457400-20 from 1817458 to 2280871 tid 2 2024-09-16T05:23:25.924 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:25.925 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:25.925 INFO:tasks.rados.rados.2.smithi204.stdout:3406: writing smithi20457400-20 from 2687955 to 3207989 tid 3 2024-09-16T05:23:25.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3406: writing smithi20457400-20 from 3709324 to 3832260 tid 4 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3407: delete oid 40 current snap is 360 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3404: finishing write tid 3 to smithi20457400-29 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3404: finishing write tid 4 to smithi20457400-29 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3404: finishing write tid 5 to smithi20457400-29 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1239 (ObjNum 1178 snap 360 seq_num 1178) dirty exists 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3404: left oid 29 (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3406: finishing write tid 1 to smithi20457400-20 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3406: finishing write tid 2 to smithi20457400-20 2024-09-16T05:23:25.930 INFO:tasks.rados.rados.2.smithi204.stdout:3406: finishing write tid 3 to smithi20457400-20 2024-09-16T05:23:25.931 INFO:tasks.rados.rados.2.smithi204.stdout:3404: done (2 left) 2024-09-16T05:23:25.931 INFO:tasks.rados.rados.2.smithi204.stdout:3407: done (1 left) 2024-09-16T05:23:25.931 INFO:tasks.rados.rados.2.smithi204.stdout:3408: write (excl) oid 26 current snap is 360 2024-09-16T05:23:25.931 INFO:tasks.rados.rados.2.smithi204.stdout:3406: finishing write tid 4 to smithi20457400-20 2024-09-16T05:23:25.932 INFO:tasks.rados.rados.2.smithi204.stdout:3408: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2024-09-16T05:23:25.938 INFO:tasks.rados.rados.2.smithi204.stdout:3408: writing smithi20457400-26 from 665874 to 1244907 tid 1 2024-09-16T05:23:25.944 INFO:tasks.rados.rados.2.smithi204.stdout:3408: writing smithi20457400-26 from 1934422 to 2401504 tid 2 2024-09-16T05:23:25.947 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:23:25.948 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:23:25.951 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:25.951 INFO:tasks.rados.rados.2.smithi204.stdout:3408: writing smithi20457400-26 from 3142801 to 3685379 tid 3 2024-09-16T05:23:25.952 INFO:tasks.rados.rados.2.smithi204.stdout:3409: write oid 8 current snap is 360 2024-09-16T05:23:25.952 INFO:tasks.rados.rados.2.smithi204.stdout:3409: seq_num 1183 ranges {682681=654282,1876551=610740,3053871=470203} 2024-09-16T05:23:25.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:25.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:25.958 INFO:tasks.rados.rados.2.smithi204.stdout:3409: writing smithi20457400-8 from 682681 to 1336963 tid 1 2024-09-16T05:23:25.965 INFO:tasks.rados.rados.2.smithi204.stdout:3409: writing smithi20457400-8 from 1876551 to 2487291 tid 2 2024-09-16T05:23:25.971 INFO:tasks.rados.rados.2.smithi204.stdout:3409: writing smithi20457400-8 from 3053871 to 3524074 tid 3 2024-09-16T05:23:25.971 INFO:tasks.rados.rados.2.smithi204.stdout:3410: snap_create 2024-09-16T05:23:25.971 INFO:tasks.rados.rados.2.smithi204.stdout:3406: finishing write tid 5 to smithi20457400-20 2024-09-16T05:23:25.971 INFO:tasks.rados.rados.2.smithi204.stdout:3406: finishing write tid 6 to smithi20457400-20 2024-09-16T05:23:25.971 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1357 (ObjNum 1180 snap 360 seq_num 1180) dirty exists 2024-09-16T05:23:25.971 INFO:tasks.rados.rados.2.smithi204.stdout:3406: left oid 20 (ObjNum 1180 snap 360 seq_num 1180) 2024-09-16T05:23:25.971 INFO:tasks.rados.rados.2.smithi204.stdout:3408: finishing write tid 1 to smithi20457400-26 2024-09-16T05:23:25.971 INFO:tasks.rados.rados.2.smithi204.stdout:3408: finishing write tid 2 to smithi20457400-26 2024-09-16T05:23:25.972 INFO:tasks.rados.rados.2.smithi204.stdout:3408: finishing write tid 3 to smithi20457400-26 2024-09-16T05:23:25.972 INFO:tasks.rados.rados.2.smithi204.stdout:3408: finishing write tid 4 to smithi20457400-26 2024-09-16T05:23:25.972 INFO:tasks.rados.rados.2.smithi204.stdout:3408: finishing write tid 5 to smithi20457400-26 2024-09-16T05:23:25.972 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1481 (ObjNum 1182 snap 360 seq_num 1182) dirty exists 2024-09-16T05:23:25.972 INFO:tasks.rados.rados.2.smithi204.stdout:3408: left oid 26 (ObjNum 1182 snap 360 seq_num 1182) 2024-09-16T05:23:25.972 INFO:tasks.rados.rados.2.smithi204.stdout:3409: finishing write tid 1 to smithi20457400-8 2024-09-16T05:23:25.972 INFO:tasks.rados.rados.2.smithi204.stdout:3409: finishing write tid 2 to smithi20457400-8 2024-09-16T05:23:25.972 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:25.974 INFO:tasks.rados.rados.2.smithi204.stdout:3409: finishing write tid 3 to smithi20457400-8 2024-09-16T05:23:25.974 INFO:tasks.rados.rados.2.smithi204.stdout:3409: finishing write tid 4 to smithi20457400-8 2024-09-16T05:23:25.974 INFO:tasks.rados.rados.2.smithi204.stdout:3409: finishing write tid 5 to smithi20457400-8 2024-09-16T05:23:25.974 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1421 (ObjNum 1183 snap 360 seq_num 1183) dirty exists 2024-09-16T05:23:25.974 INFO:tasks.rados.rados.2.smithi204.stdout:3409: left oid 8 (ObjNum 1183 snap 360 seq_num 1183) 2024-09-16T05:23:25.982 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:25.993 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:25.993 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:26.016 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:26.038 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:26.038 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:26.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:26.059 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:26.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:26.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:26.081 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:26.081 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:26.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:26.124 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:26.124 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:26.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:26.166 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:26.166 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:26.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:25 smithi204 ceph-mon[30933]: pgmap v1443: 127 pgs: 127 active+clean; 130 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.7 MiB/s wr, 9 op/s 2024-09-16T05:23:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:25 smithi204 ceph-mon[30933]: osdmap e855: 12 total, 12 up, 12 in 2024-09-16T05:23:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:25 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:23:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:25 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:23:26.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:25 smithi204 ceph-mon[30933]: osdmap e856: 12 total, 12 up, 12 in 2024-09-16T05:23:26.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:26.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:26.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:25 smithi196 ceph-mon[30802]: pgmap v1443: 127 pgs: 127 active+clean; 130 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.7 MiB/s wr, 9 op/s 2024-09-16T05:23:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:25 smithi196 ceph-mon[30802]: osdmap e855: 12 total, 12 up, 12 in 2024-09-16T05:23:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:25 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:23:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:25 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:23:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:25 smithi196 ceph-mon[30802]: osdmap e856: 12 total, 12 up, 12 in 2024-09-16T05:23:26.306 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:26.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:26.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:26.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:26.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:26.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:26.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:26.583 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:26.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:26.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:26.708 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:26.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:26.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:26.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:26.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:3406: done (3 left) 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:3408: done (2 left) 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:3409: done (1 left) 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:3410: done (0 left) 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:3411: copy_from oid 43 from oid 30 current snap is 361 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:3412: rollback oid 20 current snap is 361 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 20 to 354 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:3413: write (excl) oid 47 current snap is 361 2024-09-16T05:23:26.847 INFO:tasks.rados.rados.2.smithi204.stdout:3413: seq_num 1184 ranges {699488=729531,1935033=470751,3081294=681475} 2024-09-16T05:23:26.855 INFO:tasks.rados.rados.2.smithi204.stdout:3413: writing smithi20457400-47 from 699488 to 1429019 tid 1 2024-09-16T05:23:26.860 INFO:tasks.rados.rados.2.smithi204.stdout:3413: writing smithi20457400-47 from 1935033 to 2405784 tid 2 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:3413: writing smithi20457400-47 from 3081294 to 3762769 tid 3 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:3414: rollback oid 36 current snap is 361 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 36 to 354 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:3412: finishing rollback tid 0 to smithi20457400-20 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:3412: finishing rollback tid 1 to smithi20457400-20 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:3412: finishing rollback tid 2 to smithi20457400-20 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1361 (ObjNum 1087 snap 330 seq_num 1087) dirty exists 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:3411: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:23:26.869 INFO:tasks.rados.rados.2.smithi204.stdout:3413: finishing write tid 1 to smithi20457400-47 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3411: finishing copy_from to smithi20457400-43 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1476 (ObjNum 1176 snap 360 seq_num 1176) dirty exists 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3413: finishing write tid 2 to smithi20457400-47 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3411: done (3 left) 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3412: done (2 left) 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3415: read oid 35 snap 359 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3415: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3416: read oid 38 snap 358 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3416: expect deleted 2024-09-16T05:23:26.870 INFO:tasks.rados.rados.2.smithi204.stdout:3417: snap_create 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:3414: finishing rollback tid 0 to smithi20457400-36 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:3413: finishing write tid 3 to smithi20457400-47 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:3413: finishing write tid 4 to smithi20457400-47 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:3414: finishing rollback tid 1 to smithi20457400-36 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:3414: finishing rollback tid 2 to smithi20457400-36 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 797 (ObjNum 1158 snap 348 seq_num 1158) dirty exists 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:3413: finishing write tid 5 to smithi20457400-47 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1074 (ObjNum 1184 snap 361 seq_num 1184) dirty exists 2024-09-16T05:23:26.872 INFO:tasks.rados.rados.2.smithi204.stdout:3413: left oid 47 (ObjNum 1184 snap 361 seq_num 1184) 2024-09-16T05:23:26.877 INFO:tasks.rados.rados.2.smithi204.stdout:3415: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:23:26.938 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:26.960 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:26.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:27.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:27.075 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:27.076 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:27.083 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:27.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3811724985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:26 smithi032 ceph-mon[27388]: osdmap e857: 12 total, 12 up, 12 in 2024-09-16T05:23:27.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3811724985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:27.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:26 smithi204 ceph-mon[30933]: osdmap e857: 12 total, 12 up, 12 in 2024-09-16T05:23:27.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:27.211 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:27.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:27.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3811724985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:27.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:26 smithi196 ceph-mon[30802]: osdmap e857: 12 total, 12 up, 12 in 2024-09-16T05:23:27.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:27.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:27.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:27.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:23:27.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:27.495 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:27.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:27.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:27.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:27.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:27.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:23:27.721 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:27.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:27.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:27.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:27.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:27.875 INFO:tasks.rados.rados.2.smithi204.stdout:3413: done (4 left) 2024-09-16T05:23:27.876 INFO:tasks.rados.rados.2.smithi204.stdout:3414: done (3 left) 2024-09-16T05:23:27.876 INFO:tasks.rados.rados.2.smithi204.stdout:3415: done (2 left) 2024-09-16T05:23:27.876 INFO:tasks.rados.rados.2.smithi204.stdout:3416: done (1 left) 2024-09-16T05:23:27.876 INFO:tasks.rados.rados.2.smithi204.stdout:3417: done (0 left) 2024-09-16T05:23:27.876 INFO:tasks.rados.rados.2.smithi204.stdout:3418: snap_remove snap 354 2024-09-16T05:23:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:27.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:27.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:27 smithi032 ceph-mon[27388]: pgmap v1446: 127 pgs: 127 active+clean; 127 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 441 KiB/s wr, 5 op/s 2024-09-16T05:23:27.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:27.994 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:23:27.994 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:23:27.994 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T05:23:27.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:23:28.000 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:28.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:28.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:28.126 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:28.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:28.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:27 smithi204 ceph-mon[30933]: pgmap v1446: 127 pgs: 127 active+clean; 127 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 441 KiB/s wr, 5 op/s 2024-09-16T05:23:28.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:28.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:27 smithi196 ceph-mon[30802]: pgmap v1446: 127 pgs: 127 active+clean; 127 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 441 KiB/s wr, 5 op/s 2024-09-16T05:23:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:28.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:28.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:28.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:28.357 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:28.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:28.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:28.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:28.393 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:28.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:28.501 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:28.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:28.602 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:28.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:28.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:28.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:28.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:28.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:28.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:23:28.882 INFO:tasks.rados.rados.2.smithi204.stdout:3418: done (0 left) 2024-09-16T05:23:28.882 INFO:tasks.rados.rados.2.smithi204.stdout:3419: write (excl) oid 26 current snap is 362 2024-09-16T05:23:28.882 INFO:tasks.rados.rados.2.smithi204.stdout:3419: seq_num 1185 ranges {716295=404780,1593515=614409,2708717=492747,3640519=275776} 2024-09-16T05:23:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:28.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:28.889 INFO:tasks.rados.rados.2.smithi204.stdout:3419: writing smithi20457400-26 from 716295 to 1121075 tid 1 2024-09-16T05:23:28.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:23:28.896 INFO:tasks.rados.rados.2.smithi204.stdout:3419: writing smithi20457400-26 from 1593515 to 2207924 tid 2 2024-09-16T05:23:28.901 INFO:tasks.rados.rados.2.smithi204.stdout:3419: writing smithi20457400-26 from 2708717 to 3201464 tid 3 2024-09-16T05:23:28.905 INFO:tasks.rados.rados.2.smithi204.stdout:3419: writing smithi20457400-26 from 3640519 to 3916295 tid 4 2024-09-16T05:23:28.906 INFO:tasks.rados.rados.2.smithi204.stdout:3420: write (excl) oid 7 current snap is 362 2024-09-16T05:23:28.906 INFO:tasks.rados.rados.2.smithi204.stdout:3420: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2024-09-16T05:23:28.912 INFO:tasks.rados.rados.2.smithi204.stdout:3420: writing smithi20457400-7 from 733102 to 1329484 tid 1 2024-09-16T05:23:28.919 INFO:tasks.rados.rados.2.smithi204.stdout:3420: writing smithi20457400-7 from 1768350 to 2526417 tid 2 2024-09-16T05:23:28.926 INFO:tasks.rados.rados.2.smithi204.stdout:3420: writing smithi20457400-7 from 3136140 to 3840159 tid 3 2024-09-16T05:23:28.926 INFO:tasks.rados.rados.2.smithi204.stdout:3421: write oid 39 current snap is 362 2024-09-16T05:23:28.926 INFO:tasks.rados.rados.2.smithi204.stdout:3421: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-09-16T05:23:28.934 INFO:tasks.rados.rados.2.smithi204.stdout:3421: writing smithi20457400-39 from 749909 to 1421540 tid 1 2024-09-16T05:23:28.941 INFO:tasks.rados.rados.2.smithi204.stdout:3421: writing smithi20457400-39 from 2110479 to 2728557 tid 2 2024-09-16T05:23:28.946 INFO:tasks.rados.rados.2.smithi204.stdout:3421: writing smithi20457400-39 from 3163563 to 3678854 tid 3 2024-09-16T05:23:28.946 INFO:tasks.rados.rados.2.smithi204.stdout:3422: snap_remove snap 360 2024-09-16T05:23:28.988 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:29.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:29.104 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:29.104 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:29.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:29.137 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:29.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:28 smithi032 ceph-mon[27388]: osdmap e858: 12 total, 12 up, 12 in 2024-09-16T05:23:29.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4059178321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:29.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:28 smithi204 ceph-mon[30933]: osdmap e858: 12 total, 12 up, 12 in 2024-09-16T05:23:29.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4059178321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:29.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:28 smithi196 ceph-mon[30802]: osdmap e858: 12 total, 12 up, 12 in 2024-09-16T05:23:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4059178321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:29.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:29.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:29.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:29.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:29.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:29.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:29.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:29.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:29.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:29.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:29.641 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:29.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:29.649 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:23:29.649 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:23:29.649 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 859, 2024-09-16T05:23:29.649 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:23:29.649 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:23:29.649 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:23:28.872780+0000", 2024-09-16T05:23:29.649 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:14:38.892129+0000", 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:21:53.914532+0000", 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5804032, 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 39, 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:23:29.650 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:23:29.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:23:29.652 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 31, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 31, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 31, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 31, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 31, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:23:29.653 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "687", 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "685", 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:23:29.654 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:23:29.655 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:23:29.656 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.3199999332427979, 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.3199999332427979, 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.3199999332427979, 2024-09-16T05:23:29.657 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.3199999332427979, 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:23:29.658 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:23:29.659 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 96, 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 96, 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 96, 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 96, 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 96, 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:23:29.660 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "859", 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "737", 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 365, 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 859, 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:23:29.661 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:23:29.662 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:23:29.663 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 1.6299999952316284, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 1.6299999952316284, 2024-09-16T05:23:29.664 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 1, 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 1.6299999952316284, 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 1.6299999952316284, 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.665 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 794, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:23:29.666 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.667 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.668 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.669 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:23:29.670 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.671 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.671 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.671 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.671 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:23:29.671 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 853, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.675 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.676 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.677 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.678 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.679 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.680 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 21, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 295, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 329, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 792, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 296, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:23:29.681 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.682 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.685 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:23:29.685 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:29.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.685 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3973436581", 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3973436581", 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3973436581", 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3973436581", 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 792, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.687 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.688 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:23:29.689 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:29.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.690 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.690 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.690 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.690 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:23:29.692 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.693 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 795, 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:29.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.695 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.695 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:23:29.695 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:29.695 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.695 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.695 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:29.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:23:29.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.699 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:23:29.700 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 855, 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:23:29.701 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:23:29.702 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:23:29.705 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.706 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 767, 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:23:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:23:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:23:29.712 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 764, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.713 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.714 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.715 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.716 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.716 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:23:29.716 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:29.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.716 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.716 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:23:29.719 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 794, 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:23:29.720 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.722 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.724 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.725 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:23:29.725 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:23:29.725 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 856, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.726 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.727 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:23:29.731 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 764, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:23:29.736 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:23:29.737 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 853, 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.740 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:29.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:23:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:38.130007+0000", 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:53.256817+0000", 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.744 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:23:25.375559+0000", 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 295 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.745 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.747 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:18.313133+0000", 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:57.887383+0000", 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:49.885060+0000", 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:23:29.749 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:44.205006+0000", 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.750 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:36.577250+0000", 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:14.707362+0000", 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.751 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:16.519346+0000", 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.752 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:59.519632+0000", 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:23:29.753 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:26.588661+0000", 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:29.754 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:23:29.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 352, 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 355, 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:23:29.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 358, 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 361, 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [ 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.757 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 358, 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:23:29.758 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:23:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:23:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:23:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:23:29.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:29.759 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:23:29.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:29.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:29.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:29.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:29.893 INFO:tasks.rados.rados.2.smithi204.stdout:3422: done (3 left) 2024-09-16T05:23:29.894 INFO:tasks.rados.rados.2.smithi204.stdout:3423: rollback oid 46 current snap is 362 2024-09-16T05:23:29.894 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 361 2024-09-16T05:23:29.894 INFO:tasks.rados.rados.2.smithi204.stdout:3424: snap_remove snap 359 2024-09-16T05:23:29.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:29.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:29 smithi032 ceph-mon[27388]: pgmap v1449: 127 pgs: 127 active+clean; 127 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-16T05:23:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:29 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4059178321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:29 smithi032 ceph-mon[27388]: osdmap e859: 12 total, 12 up, 12 in 2024-09-16T05:23:29.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3982815501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:23:30.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:30.039 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:30.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:30.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:30.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:30.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:29 smithi204 ceph-mon[30933]: pgmap v1449: 127 pgs: 127 active+clean; 127 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-16T05:23:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:29 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4059178321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:30.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:29 smithi204 ceph-mon[30933]: osdmap e859: 12 total, 12 up, 12 in 2024-09-16T05:23:30.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3982815501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:23:30.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:29 smithi196 ceph-mon[30802]: pgmap v1449: 127 pgs: 127 active+clean; 127 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2024-09-16T05:23:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:29 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4059178321' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:29 smithi196 ceph-mon[30802]: osdmap e859: 12 total, 12 up, 12 in 2024-09-16T05:23:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3982815501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:23:30.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:30.291 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:30.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:30.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:30.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:30.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:30.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:30.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:30.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:30.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:30.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:30.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:30.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:30.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:30.904 INFO:tasks.rados.rados.2.smithi204.stdout:3424: done (4 left) 2024-09-16T05:23:30.904 INFO:tasks.rados.rados.2.smithi204.stdout:3425: read oid 36 snap 358 2024-09-16T05:23:30.904 INFO:tasks.rados.rados.2.smithi204.stdout:3425: expect (ObjNum 1158 snap 348 seq_num 1158) 2024-09-16T05:23:30.904 INFO:tasks.rados.rados.2.smithi204.stdout:3426: delete oid 42 current snap is 362 2024-09-16T05:23:30.904 INFO:tasks.rados.rados.2.smithi204.stdout:3419: finishing write tid 1 to smithi20457400-26 2024-09-16T05:23:30.904 INFO:tasks.rados.rados.2.smithi204.stdout:3419: finishing write tid 2 to smithi20457400-26 2024-09-16T05:23:30.904 INFO:tasks.rados.rados.2.smithi204.stdout:3419: finishing write tid 3 to smithi20457400-26 2024-09-16T05:23:30.904 INFO:tasks.rados.rados.2.smithi204.stdout:3419: finishing write tid 4 to smithi20457400-26 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3419: finishing write tid 5 to smithi20457400-26 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3419: finishing write tid 6 to smithi20457400-26 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1491 (ObjNum 1185 snap 362 seq_num 1185) dirty exists 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3419: left oid 26 (ObjNum 1185 snap 362 seq_num 1185) 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3420: finishing write tid 1 to smithi20457400-7 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3420: finishing write tid 2 to smithi20457400-7 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3420: finishing write tid 3 to smithi20457400-7 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3420: finishing write tid 4 to smithi20457400-7 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3420: finishing write tid 5 to smithi20457400-7 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1157 (ObjNum 1186 snap 362 seq_num 1186) dirty exists 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3420: left oid 7 (ObjNum 1186 snap 362 seq_num 1186) 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3421: finishing write tid 1 to smithi20457400-39 2024-09-16T05:23:30.905 INFO:tasks.rados.rados.2.smithi204.stdout:3421: finishing write tid 2 to smithi20457400-39 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:3421: finishing write tid 3 to smithi20457400-39 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:3421: finishing write tid 4 to smithi20457400-39 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:3421: finishing write tid 5 to smithi20457400-39 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 1367 (ObjNum 1187 snap 362 seq_num 1187) dirty exists 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:3421: left oid 39 (ObjNum 1187 snap 362 seq_num 1187) 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:3423: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:3423: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:3423: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:23:30.906 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1035 (ObjNum 1177 snap 360 seq_num 1177) dirty exists 2024-09-16T05:23:30.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:30.912 INFO:tasks.rados.rados.2.smithi204.stdout:3419: done (5 left) 2024-09-16T05:23:30.912 INFO:tasks.rados.rados.2.smithi204.stdout:3420: done (4 left) 2024-09-16T05:23:30.912 INFO:tasks.rados.rados.2.smithi204.stdout:3421: done (3 left) 2024-09-16T05:23:30.912 INFO:tasks.rados.rados.2.smithi204.stdout:3423: done (2 left) 2024-09-16T05:23:30.912 INFO:tasks.rados.rados.2.smithi204.stdout:3426: done (1 left) 2024-09-16T05:23:30.912 INFO:tasks.rados.rados.2.smithi204.stdout:3427: delete oid 27 current snap is 362 2024-09-16T05:23:30.915 INFO:tasks.rados.rados.2.smithi204.stdout:3425: expect (ObjNum 1158 snap 348 seq_num 1158) 2024-09-16T05:23:30.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:30.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:30.940 INFO:tasks.thrashosds.thrasher:j is {'epoch': 859, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:23:28.872780+0000', 'last_up_change': '2024-09-16T05:14:38.892129+0000', 'last_in_change': '2024-09-16T05:21:53.914532+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 39, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '687', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '685', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 1, 'raw_score_acting': 2.319999933242798, 'raw_score_stable': 2.319999933242798, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 96, 'pg_placement_num': 96, 'pg_placement_num_target': 96, 'pg_num_target': 96, 'pg_num_pending': 96, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '859', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '737', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 365, 'snap_epoch': 859, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 794, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 853, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 21, 'last_clean_end': 295, 'up_from': 329, 'up_thru': 792, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3973436581}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3973436581}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3973436581}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3973436581}]}, 'public_addr': '172.21.15.32:6819/3973436581', 'cluster_addr': '172.21.15.32:6821/3973436581', 'heartbeat_back_addr': '172.21.15.32:6825/3973436581', 'heartbeat_front_addr': '172.21.15.32:6823/3973436581', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 795, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 855, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 767, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 794, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 856, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 853, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:38.130007+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:53.256817+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:23:25.375559+0000', 'dead_epoch': 295}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:18.313133+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:57.887383+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:49.885060+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:44.205006+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:36.577250+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:14.707362+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:16.519346+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:59.519632+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:26.588661+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 352, 'length': 1}, {'begin': 355, 'length': 1}, {'begin': 358, 'length': 1}, {'begin': 361, 'length': 1}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 358, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:23:30.940 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T05:23:31.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:31.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:31.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:31.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:30 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:23:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:30 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:23:31.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:30 smithi032 ceph-mon[27388]: osdmap e860: 12 total, 12 up, 12 in 2024-09-16T05:23:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:30 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:23:31.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:30 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:23:31.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:30 smithi204 ceph-mon[30933]: osdmap e860: 12 total, 12 up, 12 in 2024-09-16T05:23:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:31.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:31.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:31.189 INFO:tasks.rados.rados.2.smithi204.stdout:3425: done (1 left) 2024-09-16T05:23:31.189 INFO:tasks.rados.rados.2.smithi204.stdout:3427: done (0 left) 2024-09-16T05:23:31.189 INFO:tasks.rados.rados.2.smithi204.stdout:3428: snap_create 2024-09-16T05:23:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:30 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:23:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:30 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:23:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:30 smithi196 ceph-mon[30802]: osdmap e860: 12 total, 12 up, 12 in 2024-09-16T05:23:31.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:31.307 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:31.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:31.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:31.432 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:31.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:31.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:23:31.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:31.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:31.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:31.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:31.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:31.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:31.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:31.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:31.872 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:31.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:31.911 INFO:tasks.rados.rados.2.smithi204.stdout:3428: done (0 left) 2024-09-16T05:23:31.911 INFO:tasks.rados.rados.2.smithi204.stdout:3429: snap_create 2024-09-16T05:23:31.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:31.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:32.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:32.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:32.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:32.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:31 smithi032 ceph-mon[27388]: pgmap v1452: 127 pgs: 127 active+clean; 139 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-16T05:23:32.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:31 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:23:32.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:31 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:23:32.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:31 smithi032 ceph-mon[27388]: osdmap e861: 12 total, 12 up, 12 in 2024-09-16T05:23:32.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:31 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:23:32.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:31 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:23:32.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:31 smithi032 ceph-mon[27388]: 2.44 scrub starts 2024-09-16T05:23:32.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:31 smithi032 ceph-mon[27388]: 2.44 scrub ok 2024-09-16T05:23:32.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:32.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:31 smithi204 ceph-mon[30933]: pgmap v1452: 127 pgs: 127 active+clean; 139 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-16T05:23:32.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:31 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:23:32.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:31 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:23:32.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:31 smithi204 ceph-mon[30933]: osdmap e861: 12 total, 12 up, 12 in 2024-09-16T05:23:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:31 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:23:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:31 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:23:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:31 smithi204 ceph-mon[30933]: 2.44 scrub starts 2024-09-16T05:23:32.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:31 smithi204 ceph-mon[30933]: 2.44 scrub ok 2024-09-16T05:23:32.183 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:32.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:31 smithi196 ceph-mon[30802]: pgmap v1452: 127 pgs: 127 active+clean; 139 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.7 MiB/s wr, 18 op/s 2024-09-16T05:23:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:31 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:23:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:31 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:23:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:31 smithi196 ceph-mon[30802]: osdmap e861: 12 total, 12 up, 12 in 2024-09-16T05:23:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:31 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:23:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:31 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:23:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:31 smithi196 ceph-mon[30802]: 2.44 scrub starts 2024-09-16T05:23:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:31 smithi196 ceph-mon[30802]: 2.44 scrub ok 2024-09-16T05:23:32.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:32.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:32.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:32.346 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:32.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:32.435 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:32.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:32.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:32.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:32.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:32.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:32.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:32.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:32.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:32.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:32.815 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:32.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:32.913 INFO:tasks.rados.rados.2.smithi204.stdout:3429: done (0 left) 2024-09-16T05:23:32.913 INFO:tasks.rados.rados.2.smithi204.stdout:3430: snap_create 2024-09-16T05:23:32.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:32.918 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:32.918 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:32.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:32.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:33.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:33.068 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:33.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:33.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:32 smithi032 ceph-mon[27388]: osdmap e862: 12 total, 12 up, 12 in 2024-09-16T05:23:33.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:32 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:23:33.168 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:32 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/822430314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:33.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:32 smithi204 ceph-mon[30933]: osdmap e862: 12 total, 12 up, 12 in 2024-09-16T05:23:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:32 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:23:33.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:32 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/822430314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:33.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:33.195 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:33.195 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:33.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:33.216 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:23:33.217 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:23:33.237 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:33.237 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:33.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:32 smithi196 ceph-mon[30802]: osdmap e862: 12 total, 12 up, 12 in 2024-09-16T05:23:33.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:32 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:23:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:32 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/822430314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:33.257 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:33.258 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:33.280 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:33.280 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:33.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:33.301 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:33.301 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:33.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:33.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:33.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:33.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:33.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:33.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:33.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:33.578 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:33.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:33.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:23:33.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:23:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:33.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:23:33.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:33.827 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:33.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:33.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:23:33.916 INFO:tasks.rados.rados.2.smithi204.stdout:3430: done (0 left) 2024-09-16T05:23:33.917 INFO:tasks.rados.rados.2.smithi204.stdout:3431: copy_from oid 42 from oid 29 current snap is 365 2024-09-16T05:23:33.917 INFO:tasks.rados.rados.2.smithi204.stdout:3432: write (excl) oid 47 current snap is 365 2024-09-16T05:23:33.917 INFO:tasks.rados.rados.2.smithi204.stdout:3432: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-16T05:23:33.921 INFO:tasks.rados.rados.2.smithi204.stdout:3432: writing smithi20457400-47 from 400330 to 897708 tid 1 2024-09-16T05:23:33.928 INFO:tasks.rados.rados.2.smithi204.stdout:3432: writing smithi20457400-47 from 1485925 to 2000330 tid 2 2024-09-16T05:23:33.928 INFO:tasks.rados.rados.2.smithi204.stdout:3433: write oid 3 current snap is 365 2024-09-16T05:23:33.928 INFO:tasks.rados.rados.2.smithi204.stdout:3433: seq_num 1191 ranges {417137=572627,1428054=589083} 2024-09-16T05:23:33.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:33.933 INFO:tasks.rados.rados.2.smithi204.stdout:3433: writing smithi20457400-3 from 417137 to 989764 tid 1 2024-09-16T05:23:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:3433: writing smithi20457400-3 from 1428054 to 2017137 tid 2 2024-09-16T05:23:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:3434: delete oid 39 current snap is 365 2024-09-16T05:23:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:3432: finishing write tid 1 to smithi20457400-47 2024-09-16T05:23:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:3432: finishing write tid 2 to smithi20457400-47 2024-09-16T05:23:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:3432: finishing write tid 3 to smithi20457400-47 2024-09-16T05:23:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:3432: finishing write tid 4 to smithi20457400-47 2024-09-16T05:23:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1077 (ObjNum 1190 snap 365 seq_num 1190) dirty exists 2024-09-16T05:23:33.940 INFO:tasks.rados.rados.2.smithi204.stdout:3432: left oid 47 (ObjNum 1190 snap 365 seq_num 1190) 2024-09-16T05:23:33.941 INFO:tasks.rados.rados.2.smithi204.stdout:3431: finishing copy_from racing read to smithi20457400-42 2024-09-16T05:23:33.941 INFO:tasks.rados.rados.2.smithi204.stdout:3431: finishing copy_from to smithi20457400-42 2024-09-16T05:23:33.941 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 1239 (ObjNum 1178 snap 360 seq_num 1178) dirty exists 2024-09-16T05:23:33.941 INFO:tasks.rados.rados.2.smithi204.stdout:3433: finishing write tid 1 to smithi20457400-3 2024-09-16T05:23:33.943 INFO:tasks.rados.rados.2.smithi204.stdout:3433: finishing write tid 2 to smithi20457400-3 2024-09-16T05:23:33.943 INFO:tasks.rados.rados.2.smithi204.stdout:3433: finishing write tid 3 to smithi20457400-3 2024-09-16T05:23:33.943 INFO:tasks.rados.rados.2.smithi204.stdout:3433: finishing write tid 4 to smithi20457400-3 2024-09-16T05:23:33.943 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1099 (ObjNum 1191 snap 365 seq_num 1191) dirty exists 2024-09-16T05:23:33.943 INFO:tasks.rados.rados.2.smithi204.stdout:3433: left oid 3 (ObjNum 1191 snap 365 seq_num 1191) 2024-09-16T05:23:33.944 INFO:tasks.rados.rados.2.smithi204.stdout:3431: done (3 left) 2024-09-16T05:23:33.944 INFO:tasks.rados.rados.2.smithi204.stdout:3432: done (2 left) 2024-09-16T05:23:33.944 INFO:tasks.rados.rados.2.smithi204.stdout:3433: done (1 left) 2024-09-16T05:23:33.944 INFO:tasks.rados.rados.2.smithi204.stdout:3434: done (0 left) 2024-09-16T05:23:33.945 INFO:tasks.rados.rados.2.smithi204.stdout:3435: snap_create 2024-09-16T05:23:33.952 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:33.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:34.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:34.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:34.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:34.126 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:34.127 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:34.148 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:34.169 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:34.170 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: pgmap v1455: 127 pgs: 127 active+clean; 132 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 19 op/s 2024-09-16T05:23:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:23:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:23:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:23:34.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: 1.1a scrub starts 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: 1.1a scrub ok 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: osdmap e863: 12 total, 12 up, 12 in 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: 2.54 scrub starts 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:33 smithi032 ceph-mon[27388]: 2.54 scrub ok 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: pgmap v1455: 127 pgs: 127 active+clean; 132 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 19 op/s 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:23:34.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:23:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:23:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:23:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:23:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: 1.1a scrub starts 2024-09-16T05:23:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: 1.1a scrub ok 2024-09-16T05:23:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: osdmap e863: 12 total, 12 up, 12 in 2024-09-16T05:23:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: 2.54 scrub starts 2024-09-16T05:23:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:33 smithi204 ceph-mon[30933]: 2.54 scrub ok 2024-09-16T05:23:34.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:34.191 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:34.192 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:34.204 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:34.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:34.214 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:34.214 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: pgmap v1455: 127 pgs: 127 active+clean; 132 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 19 op/s 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: 1.1a scrub starts 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: 1.1a scrub ok 2024-09-16T05:23:34.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: osdmap e863: 12 total, 12 up, 12 in 2024-09-16T05:23:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: 2.54 scrub starts 2024-09-16T05:23:34.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:33 smithi196 ceph-mon[30802]: 2.54 scrub ok 2024-09-16T05:23:34.239 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:34.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:34.331 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:34.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:34.337 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:23:34.337 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:23:34.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:23:34.337 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:23:34.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:611 2.15 e366/273 2:a94680a3:::smithi20457400-13:66 v 366'446, mlcod=362'442)", 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 498.61859426000001, 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119602217, 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716831+0000", 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8099999999999999e-07 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.716837+0000", 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4909999999999996e-06 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835914+0000", 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.119077768 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835922+0000", 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.994e-06 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835962+0000", 2024-09-16T05:23:34.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9230000000000002e-05 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.835979+0000", 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.787e-05 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.836402+0000", 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00042242199999999999 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.836433+0000", 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1260999999999998e-05 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4658 2.35 2:ad4a88eb:::smithi20457400-16:head [copy-from ver 686] snapc c1=[c1,bf,be] ondisk+write+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.782375+0000", 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 349.55305027399999, 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019484680000000001, 2024-09-16T05:23:34.342 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4658 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782375+0000", 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782375+0000", 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782377+0000", 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2579999999999998e-06 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782399+0000", 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1585000000000001e-05 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782417+0000", 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8658e-05 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782426+0000", 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2029999999999992e-06 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.345 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782493+0000", 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7094000000000005e-05 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782498+0000", 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2349999999999999e-06 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.784105+0000", 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016068079999999999 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.784171+0000", 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6023000000000001e-05 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.796572+0000", 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012401392000000001 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.799172+0000", 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025995240000000002 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.801780+0000", 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026087430000000002 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.801815+0000", 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4950000000000002e-05 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.801860+0000", 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4206999999999999e-05 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.348 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4659 2.35 2:ad4a88eb:::smithi20457400-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.782448+0000", 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 349.55297758699999, 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.016349405000000001, 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.349 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4659 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782448+0000", 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782448+0000", 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.350 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782448+0000", 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4300000000000003e-07 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782451+0000", 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.6570000000000001e-06 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782459+0000", 2024-09-16T05:23:34.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1610000000000003e-06 2024-09-16T05:23:34.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782465+0000", 2024-09-16T05:23:34.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7790000000000001e-06 2024-09-16T05:23:34.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782548+0000", 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3323000000000002e-05 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.782552+0000", 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6710000000000001e-06 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.784265+0000", 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017134850000000001 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.784292+0000", 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7180000000000001e-05 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.798667+0000", 2024-09-16T05:23:34.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.014374408 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.798731+0000", 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4429000000000003e-05 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.798797+0000", 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5769e-05 2024-09-16T05:23:34.356 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5589 2.54 2:2ace617f:::smithi20457400-21:head [copy-from ver 833] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.804999+0000", 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 281.53042602900001, 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013042259, 2024-09-16T05:23:34.357 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5589 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.804999+0000", 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.804999+0000", 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805000+0000", 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3090000000000001e-06 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805006+0000", 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5550000000000002e-06 2024-09-16T05:23:34.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805018+0000", 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1995e-05 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805024+0000", 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8279999999999999e-06 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805088+0000", 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4295999999999996e-05 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.805198+0000", 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011024299999999999 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.811590+0000", 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063921100000000003 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.361 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.815801+0000", 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042101969999999997 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.817919+0000", 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002118071 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.817988+0000", 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9666999999999996e-05 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.818041+0000", 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2988000000000002e-05 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5603 2.35 2:ad4a88eb:::smithi20457400-16:head [copy-from ver 369] snapc e7=[e7,e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e584)", 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:53.850711+0000", 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 280.48471365799998, 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015532102000000001, 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.363 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5603 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850711+0000", 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850711+0000", 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850713+0000", 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3289999999999999e-06 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850718+0000", 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7339999999999999e-06 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850730+0000", 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2695e-05 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850737+0000", 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2010000000000002e-06 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850792+0000", 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4817e-05 2024-09-16T05:23:34.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850868+0000", 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5765000000000004e-05 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.858838+0000", 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079695340000000003 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.863812+0000", 2024-09-16T05:23:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049744610000000003 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.866156+0000", 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0023442889999999998 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.866206+0000", 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0120000000000001e-05 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.866244+0000", 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7156999999999997e-05 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5604 2.35 2:ad4a88eb:::smithi20457400-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e584)", 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:53.850750+0000", 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 280.48467474099999, 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.011376676, 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:23:34.369 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5604 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850750+0000", 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850750+0000", 2024-09-16T05:23:34.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850751+0000", 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850756+0000", 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6630000000000001e-06 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850763+0000", 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4680000000000003e-06 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850769+0000", 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7699999999999998e-06 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.850987+0000", 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021844799999999999 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.851020+0000", 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3238999999999998e-05 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.861934+0000", 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010913983 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.862047+0000", 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000112487 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:53.862127+0000", 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0131999999999998e-05 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.7.0:974 2.3c e595/293 2:3d4bedc8:::smithi20457400-22:d8 v 595'589, mlcod=592'583)", 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:04.831453+0000", 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 269.50397233500001, 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.549161342, 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.374 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:04.831453+0000", 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:04.831453+0000", 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.375 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:04.831454+0000", 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8899999999999998e-07 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:04.831458+0000", 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5879999999999999e-06 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.379926+0000", 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.548467912 2024-09-16T05:23:34.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.379935+0000", 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6459999999999999e-06 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.380009+0000", 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3879000000000002e-05 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.380036+0000", 2024-09-16T05:23:34.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7574999999999999e-05 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.380573+0000", 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00053675900000000004 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:05.380614+0000", 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1094000000000003e-05 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.378 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7273 2.35 2:ad4a88eb:::smithi20457400-16:head [append 0~0] snapc 133=[133,132,131,12f,12c,12a,129,126,125,123,119,116] ondisk+write+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:26.823738+0000", 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 127.51168743700001, 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01193862, 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.379 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7273 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823738+0000", 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823738+0000", 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823744+0000", 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5679999999999996e-06 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823752+0000", 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1450000000000008e-06 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823777+0000", 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4995e-05 2024-09-16T05:23:34.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823790+0000", 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2223e-05 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.824144+0000", 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000354039 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.824272+0000", 2024-09-16T05:23:34.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012827499999999999 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.824558+0000", 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00028587699999999997 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.832146+0000", 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00758846 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.835455+0000", 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003308797 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.835594+0000", 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013877299999999999 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.835676+0000", 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2467999999999997e-05 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.384 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7274 2.35 2:ad4a88eb:::smithi20457400-16:head [rollback 116] snapc 133=[133,132,131,12f,12c,12a,129,126,125,123,119,116] ondisk+write+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:26.823818+0000", 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 127.51160758100001, 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012147374000000001, 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.385 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7274 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823818+0000", 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823818+0000", 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.386 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823818+0000", 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5999999999999991e-07 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823823+0000", 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1030000000000004e-06 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823833+0000", 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9340000000000001e-06 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823840+0000", 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8099999999999998e-06 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.825444+0000", 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001603824 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.826515+0000", 2024-09-16T05:23:34.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001070779 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.826839+0000", 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00032387399999999998 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.832617+0000", 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0057784220000000001 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.835776+0000", 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0031586050000000001 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.835910+0000", 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000134549 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.835965+0000", 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4514e-05 2024-09-16T05:23:34.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7275 2.35 2:ad4a88eb:::smithi20457400-16:head [append 0~0] snapc 133=[133,132,131,12f,12c,12a,129,126,125,123,119,116] ondisk+write+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:26.823852+0000", 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 127.511572904, 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013927182, 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.391 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7275 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823852+0000", 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823852+0000", 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823853+0000", 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8500000000000001e-07 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823856+0000", 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1980000000000001e-06 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823864+0000", 2024-09-16T05:23:34.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6229999999999997e-06 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.823870+0000", 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9200000000000001e-06 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.830884+0000", 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.007014068 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.831019+0000", 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013579900000000001 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.831266+0000", 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00024654700000000001 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.832626+0000", 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013596299999999999 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.837650+0000", 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050246459999999998 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.837722+0000", 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2038000000000004e-05 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:26.837779+0000", 2024-09-16T05:23:34.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7127999999999999e-05 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.1.0:601 2.35 e743/720 ondisk, result = 0)", 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:33.847939+0000", 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 120.48748629799999, 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.58106756999999998, 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:23:34.397 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:33.847939+0000", 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:33.847939+0000", 2024-09-16T05:23:34.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:33.847940+0000", 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3689999999999999e-06 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:33.847947+0000", 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3589999999999996e-06 2024-09-16T05:23:34.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:34.428692+0000", 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.58074561000000002 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:34.428715+0000", 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2646e-05 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:34.428861+0000", 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014647799999999999 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:34.428871+0000", 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7010000000000008e-06 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:34.429006+0000", 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013540700000000001 2024-09-16T05:23:34.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7348 2.5d 2:ba7e58cf:::smithi20457400-26:head [delete] snapc 139=[139,137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+full_force+supports_pool_eio e746)", 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:35.842276+0000", 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 118.493149249, 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0096951050000000007, 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.402 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7348 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842276+0000", 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842276+0000", 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842278+0000", 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8279999999999999e-06 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842285+0000", 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9700000000000002e-06 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842304+0000", 2024-09-16T05:23:34.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9040999999999999e-05 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842315+0000", 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1783000000000001e-05 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842383+0000", 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7834000000000001e-05 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842498+0000", 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000114881 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.842711+0000", 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000213038 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.843712+0000", 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0010002909999999999 2024-09-16T05:23:34.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.851856+0000", 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0081442870000000001 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.851911+0000", 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5294000000000003e-05 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:35.851971+0000", 2024-09-16T05:23:34.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9858000000000002e-05 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7421 2.5d 2:ba754fcb:::smithi20457400-30:head [copy-from ver 326] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.839743+0000", 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 109.495682562, 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010730801999999999, 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.408 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7421 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.839743+0000", 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.839743+0000", 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.839744+0000", 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6869999999999999e-06 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.839751+0000", 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7580000000000003e-06 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.839766+0000", 2024-09-16T05:23:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5002e-05 2024-09-16T05:23:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.839775+0000", 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0599999999999997e-06 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.839835+0000", 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0124999999999998e-05 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:23:34.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.839840+0000", 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8790000000000002e-06 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.841607+0000", 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001766887 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.841660+0000", 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2942999999999998e-05 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.845555+0000", 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0038949879999999998 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.848779+0000", 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0032239870000000002 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.850408+0000", 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001629129 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.850445+0000", 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6511999999999999e-05 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.850473+0000", 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8844999999999999e-05 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7528 2.5d 2:ba21f1a9:::smithi20457400-14:head [copy-from ver 1051] snapc 13f=[13f,13d,13b,134,133,12c,12a,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:51.936680+0000", 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 102.398744842, 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013974477000000001, 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.416 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7528 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936680+0000", 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936680+0000", 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936682+0000", 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.911e-06 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936690+0000", 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1079999999999993e-06 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936711+0000", 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0295000000000001e-05 2024-09-16T05:23:34.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936720+0000", 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2450000000000004e-06 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936789+0000", 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9251999999999994e-05 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936908+0000", 2024-09-16T05:23:34.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118507 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.943030+0000", 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061226350000000004 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.946650+0000", 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036202629999999999 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.950551+0000", 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0039001180000000002 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.950618+0000", 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7798000000000004e-05 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.950655+0000", 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6344999999999998e-05 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.421 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7529 2.5d 2:ba21f1a9:::smithi20457400-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:51.936741+0000", 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 102.398684308, 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094191710000000005, 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.422 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7529 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936741+0000", 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936741+0000", 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936741+0000", 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8599999999999999e-07 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936744+0000", 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4080000000000001e-06 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936750+0000", 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2020000000000001e-06 2024-09-16T05:23:34.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.936754+0000", 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7330000000000001e-06 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.937014+0000", 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026087300000000001 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.937050+0000", 2024-09-16T05:23:34.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5762000000000001e-05 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.945922+0000", 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088719490000000005 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.946039+0000", 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000117009 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.946160+0000", 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012084899999999999 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7547 2.35 2:ad4a88eb:::smithi20457400-16:head [copy-from ver 1012] snapc 13f=[13f,13d,13b,134,133,12c,12a,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:51.956876+0000", 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 102.378549562, 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012755326000000001, 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.427 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7547 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.956876+0000", 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.956876+0000", 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.956878+0000", 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1629999999999999e-06 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.956888+0000", 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0737000000000001e-05 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.956941+0000", 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2292e-05 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.956958+0000", 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7557000000000001e-05 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.957092+0000", 2024-09-16T05:23:34.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013368700000000001 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.957177+0000", 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4921999999999998e-05 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 11", 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.963588+0000", 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0064111469999999999 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.966078+0000", 2024-09-16T05:23:34.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002490108 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.969557+0000", 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0034792019999999998 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.969599+0000", 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1950999999999999e-05 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.969631+0000", 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1560000000000003e-05 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7698 2.5d 2:ba7e58cf:::smithi20457400-26:head [rollback 12c] snapc 147=[147,146,145,144,143,142,141,13b,133,12c,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e776)", 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:05.824335+0000", 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 88.511089788000007, 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010110862, 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.433 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7698 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824335+0000", 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824335+0000", 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824337+0000", 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1739999999999999e-06 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824345+0000", 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8590000000000005e-06 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824369+0000", 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3255999999999999e-05 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824379+0000", 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0562e-05 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824502+0000", 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012308100000000001 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824511+0000", 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.4009999999999997e-06 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.829390+0000", 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0048796530000000003 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.829494+0000", 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103387 2024-09-16T05:23:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.829707+0000", 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021322600000000001 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.834179+0000", 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0044718689999999998 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.834336+0000", 2024-09-16T05:23:34.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000157498 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.834399+0000", 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2662000000000004e-05 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.834446+0000", 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7234000000000002e-05 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7699 2.5d 2:ba7e58cf:::smithi20457400-26:head [append 0~0] snapc 147=[147,146,145,144,143,142,141,13b,133,12c,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e776)", 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:05.824405+0000", 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 88.511020113000001, 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010129825, 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7699 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.440 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824405+0000", 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824405+0000", 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824406+0000", 2024-09-16T05:23:34.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8400000000000004e-07 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824409+0000", 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.433e-06 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824418+0000", 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0270000000000001e-06 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.442 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824424+0000", 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8690000000000001e-06 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824570+0000", 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014613899999999999 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.824576+0000", 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0639999999999999e-06 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.830421+0000", 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058445499999999996 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.830508+0000", 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7749000000000003e-05 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.830657+0000", 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014882799999999999 2024-09-16T05:23:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.834322+0000", 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0036643190000000001 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.834454+0000", 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132235 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:23:34.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.834498+0000", 2024-09-16T05:23:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.409e-05 2024-09-16T05:23:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:05.834535+0000", 2024-09-16T05:23:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6937999999999998e-05 2024-09-16T05:23:34.446 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.1.0:691 2.5d e799/742 ondisk, result = 0)", 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:28.879062+0000", 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 65.456363468000006, 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.85765913800000004, 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.448 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.879062+0000", 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.879062+0000", 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.449 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.879062+0000", 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4600000000000004e-07 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.879066+0000", 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6720000000000002e-06 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:29.736544+0000", 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.85747783 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:29.736553+0000", 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8689999999999993e-06 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:29.736610+0000", 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6956000000000002e-05 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:29.736620+0000", 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0437e-05 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:29.736721+0000", 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100728 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.15,QREQUEST,e:862,from:9,reservation_nonce:3)", 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:33.297933+0000", 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.0374925960000001, 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.58120057199999997, 2024-09-16T05:23:34.452 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.297933+0000", 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.297933+0000", 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.297934+0000", 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.237e-06 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.297940+0000", 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6059999999999999e-06 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.878956+0000", 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.58101562200000001 2024-09-16T05:23:34.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.878965+0000", 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2220000000000008e-06 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.879063+0000", 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8256000000000003e-05 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:23:34.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.879087+0000", 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3334999999999999e-05 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:33.879133+0000", 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6294000000000001e-05 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:23:34.456 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:23:34.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:34.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:34.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:34.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:34.610 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:34.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:34.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:34.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:34.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:34.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:34.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:34.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:34.926 INFO:tasks.rados.rados.2.smithi204.stdout:3435: done (0 left) 2024-09-16T05:23:34.926 INFO:tasks.rados.rados.2.smithi204.stdout:3436: read oid 29 snap -1 2024-09-16T05:23:34.926 INFO:tasks.rados.rados.2.smithi204.stdout:3436: expect (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:23:34.926 INFO:tasks.rados.rados.2.smithi204.stdout:3437: write oid 47 current snap is 366 2024-09-16T05:23:34.927 INFO:tasks.rados.rados.2.smithi204.stdout:3437: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-09-16T05:23:34.932 INFO:tasks.rados.rados.2.smithi204.stdout:3437: writing smithi20457400-47 from 450751 to 1173876 tid 1 2024-09-16T05:23:34.938 INFO:tasks.rados.rados.2.smithi204.stdout:3437: writing smithi20457400-47 from 1945018 to 2050751 tid 2 2024-09-16T05:23:34.938 INFO:tasks.rados.rados.2.smithi204.stdout:3438: write oid 28 current snap is 366 2024-09-16T05:23:34.938 INFO:tasks.rados.rados.2.smithi204.stdout:3438: seq_num 1194 ranges {467558=514727,1719853=347705} 2024-09-16T05:23:34.943 INFO:tasks.rados.rados.2.smithi204.stdout:3438: writing smithi20457400-28 from 467558 to 982285 tid 1 2024-09-16T05:23:34.949 INFO:tasks.rados.rados.2.smithi204.stdout:3438: writing smithi20457400-28 from 1719853 to 2067558 tid 2 2024-09-16T05:23:34.949 INFO:tasks.rados.rados.2.smithi204.stdout:3439: write (excl) oid 17 current snap is 366 2024-09-16T05:23:34.949 INFO:tasks.rados.rados.2.smithi204.stdout:3439: seq_num 1195 ranges {484365=589976,1661982=422383} 2024-09-16T05:23:34.954 INFO:tasks.rados.rados.2.smithi204.stdout:3439: writing smithi20457400-17 from 484365 to 1074341 tid 1 2024-09-16T05:23:34.960 INFO:tasks.rados.rados.2.smithi204.stdout:3439: writing smithi20457400-17 from 1661982 to 2084365 tid 2 2024-09-16T05:23:34.960 INFO:tasks.rados.rados.2.smithi204.stdout:3440: write oid 8 current snap is 366 2024-09-16T05:23:34.960 INFO:tasks.rados.rados.2.smithi204.stdout:3440: seq_num 1196 ranges {501172=665225,1720464=380708} 2024-09-16T05:23:34.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:34.966 INFO:tasks.rados.rados.2.smithi204.stdout:3440: writing smithi20457400-8 from 501172 to 1166397 tid 1 2024-09-16T05:23:34.970 INFO:tasks.rados.rados.2.smithi204.stdout:3440: writing smithi20457400-8 from 1720464 to 2101172 tid 2 2024-09-16T05:23:34.971 INFO:tasks.rados.rados.2.smithi204.stdout:3441: rollback oid 32 current snap is 366 2024-09-16T05:23:34.971 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 365 2024-09-16T05:23:34.971 INFO:tasks.rados.rados.2.smithi204.stdout:3436: expect (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:23:34.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:34.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:35.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:35.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:35.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:23:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:23:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:23:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:23:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: osdmap e864: 12 total, 12 up, 12 in 2024-09-16T05:23:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:23:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:23:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 2.4c scrub starts 2024-09-16T05:23:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 2.4c scrub ok 2024-09-16T05:23:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 1.16 scrub starts 2024-09-16T05:23:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 1.16 scrub ok 2024-09-16T05:23:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 2.35 scrub starts 2024-09-16T05:23:35.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:34 smithi032 ceph-mon[27388]: 2.35 scrub ok 2024-09-16T05:23:35.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: osdmap e864: 12 total, 12 up, 12 in 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 2.4c scrub starts 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 2.4c scrub ok 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 1.16 scrub starts 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 1.16 scrub ok 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 2.35 scrub starts 2024-09-16T05:23:35.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:34 smithi204 ceph-mon[30933]: 2.35 scrub ok 2024-09-16T05:23:35.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: osdmap e864: 12 total, 12 up, 12 in 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 2.4c scrub starts 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 2.4c scrub ok 2024-09-16T05:23:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 1.16 scrub starts 2024-09-16T05:23:35.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 1.16 scrub ok 2024-09-16T05:23:35.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 2.35 scrub starts 2024-09-16T05:23:35.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:34 smithi196 ceph-mon[30802]: 2.35 scrub ok 2024-09-16T05:23:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:35.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:35.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:35.242 INFO:tasks.rados.rados.2.smithi204.stdout:3437: finishing write tid 1 to smithi20457400-47 2024-09-16T05:23:35.242 INFO:tasks.rados.rados.2.smithi204.stdout:3437: finishing write tid 2 to smithi20457400-47 2024-09-16T05:23:35.242 INFO:tasks.rados.rados.2.smithi204.stdout:3437: finishing write tid 3 to smithi20457400-47 2024-09-16T05:23:35.242 INFO:tasks.rados.rados.2.smithi204.stdout:3437: finishing write tid 4 to smithi20457400-47 2024-09-16T05:23:35.242 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1080 (ObjNum 1193 snap 366 seq_num 1193) dirty exists 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3437: left oid 47 (ObjNum 1193 snap 366 seq_num 1193) 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3438: finishing write tid 1 to smithi20457400-28 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3438: finishing write tid 2 to smithi20457400-28 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3438: finishing write tid 3 to smithi20457400-28 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3436: done (5 left) 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3437: done (4 left) 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3442: rollback oid 24 current snap is 366 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 24 to 358 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3438: finishing write tid 4 to smithi20457400-28 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1135 (ObjNum 1194 snap 366 seq_num 1194) dirty exists 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3438: left oid 28 (ObjNum 1194 snap 366 seq_num 1194) 2024-09-16T05:23:35.243 INFO:tasks.rados.rados.2.smithi204.stdout:3439: finishing write tid 1 to smithi20457400-17 2024-09-16T05:23:35.244 INFO:tasks.rados.rados.2.smithi204.stdout:3439: finishing write tid 2 to smithi20457400-17 2024-09-16T05:23:35.244 INFO:tasks.rados.rados.2.smithi204.stdout:3439: finishing write tid 3 to smithi20457400-17 2024-09-16T05:23:35.244 INFO:tasks.rados.rados.2.smithi204.stdout:3439: finishing write tid 4 to smithi20457400-17 2024-09-16T05:23:35.244 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1138 (ObjNum 1195 snap 366 seq_num 1195) dirty exists 2024-09-16T05:23:35.244 INFO:tasks.rados.rados.2.smithi204.stdout:3439: left oid 17 (ObjNum 1195 snap 366 seq_num 1195) 2024-09-16T05:23:35.244 INFO:tasks.rados.rados.2.smithi204.stdout:3441: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:23:35.244 INFO:tasks.rados.rados.2.smithi204.stdout:3441: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:23:35.244 INFO:tasks.rados.rados.2.smithi204.stdout:3441: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1238 (ObjNum 1109 snap 336 seq_num 1109) dirty exists 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3440: finishing write tid 1 to smithi20457400-8 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3438: done (4 left) 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3439: done (3 left) 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3441: done (2 left) 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3443: write (excl) oid 47 current snap is 366 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3440: finishing write tid 2 to smithi20457400-8 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3440: finishing write tid 3 to smithi20457400-8 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3440: finishing write tid 4 to smithi20457400-8 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1429 (ObjNum 1196 snap 366 seq_num 1196) dirty exists 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3440: left oid 8 (ObjNum 1196 snap 366 seq_num 1196) 2024-09-16T05:23:35.245 INFO:tasks.rados.rados.2.smithi204.stdout:3443: seq_num 1197 ranges {517979=740474,1778946=339033} 2024-09-16T05:23:35.247 INFO:tasks.rados.rados.2.smithi204.stdout:3443: writing smithi20457400-47 from 517979 to 1258453 tid 1 2024-09-16T05:23:35.252 INFO:tasks.rados.rados.2.smithi204.stdout:3443: writing smithi20457400-47 from 1778946 to 2117979 tid 2 2024-09-16T05:23:35.252 INFO:tasks.rados.rados.2.smithi204.stdout:3440: done (2 left) 2024-09-16T05:23:35.252 INFO:tasks.rados.rados.2.smithi204.stdout:3444: write oid 8 current snap is 366 2024-09-16T05:23:35.252 INFO:tasks.rados.rados.2.smithi204.stdout:3444: seq_num 1198 ranges {534786=415723,1437428=697358} 2024-09-16T05:23:35.256 INFO:tasks.rados.rados.2.smithi204.stdout:3444: writing smithi20457400-8 from 534786 to 950509 tid 1 2024-09-16T05:23:35.265 INFO:tasks.rados.rados.2.smithi204.stdout:3444: writing smithi20457400-8 from 1437428 to 2134786 tid 2 2024-09-16T05:23:35.265 INFO:tasks.rados.rados.2.smithi204.stdout:3445: read oid 4 snap -1 2024-09-16T05:23:35.266 INFO:tasks.rados.rados.2.smithi204.stdout:3445: expect deleted 2024-09-16T05:23:35.266 INFO:tasks.rados.rados.2.smithi204.stdout:3446: read oid 25 snap 363 2024-09-16T05:23:35.266 INFO:tasks.rados.rados.2.smithi204.stdout:3446: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:23:35.266 INFO:tasks.rados.rados.2.smithi204.stdout:3447: rollback oid 27 current snap is 366 2024-09-16T05:23:35.266 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 27 to 365 2024-09-16T05:23:35.266 INFO:tasks.rados.rados.2.smithi204.stdout:3442: finishing rollback tid 1 to smithi20457400-24 2024-09-16T05:23:35.266 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:23:35.267 INFO:tasks.rados.rados.2.smithi204.stdout:3443: finishing write tid 1 to smithi20457400-47 2024-09-16T05:23:35.267 INFO:tasks.rados.rados.2.smithi204.stdout:3443: finishing write tid 2 to smithi20457400-47 2024-09-16T05:23:35.267 INFO:tasks.rados.rados.2.smithi204.stdout:3444: finishing write tid 1 to smithi20457400-8 2024-09-16T05:23:35.267 INFO:tasks.rados.rados.2.smithi204.stdout:3442: done (5 left) 2024-09-16T05:23:35.267 INFO:tasks.rados.rados.2.smithi204.stdout:3448: read oid 9 snap -1 2024-09-16T05:23:35.267 INFO:tasks.rados.rados.2.smithi204.stdout:3448: expect deleted 2024-09-16T05:23:35.267 INFO:tasks.rados.rados.2.smithi204.stdout:3449: delete oid 14 current snap is 366 2024-09-16T05:23:35.268 INFO:tasks.rados.rados.2.smithi204.stdout:3447: finishing rollback tid 1 to smithi20457400-27 2024-09-16T05:23:35.268 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 1266 (ObjNum 33 snap 0 seq_num 9) dirty dne 2024-09-16T05:23:35.268 INFO:tasks.rados.rados.2.smithi204.stdout:3445: done (6 left) 2024-09-16T05:23:35.268 INFO:tasks.rados.rados.2.smithi204.stdout:3447: done (5 left) 2024-09-16T05:23:35.268 INFO:tasks.rados.rados.2.smithi204.stdout:3449: done (4 left) 2024-09-16T05:23:35.268 INFO:tasks.rados.rados.2.smithi204.stdout:3450: delete oid 21 current snap is 366 2024-09-16T05:23:35.269 INFO:tasks.rados.rados.2.smithi204.stdout:3444: finishing write tid 2 to smithi20457400-8 2024-09-16T05:23:35.269 INFO:tasks.rados.rados.2.smithi204.stdout:3444: finishing write tid 3 to smithi20457400-8 2024-09-16T05:23:35.269 INFO:tasks.rados.rados.2.smithi204.stdout:3444: finishing write tid 4 to smithi20457400-8 2024-09-16T05:23:35.269 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1432 (ObjNum 1198 snap 366 seq_num 1198) dirty exists 2024-09-16T05:23:35.269 INFO:tasks.rados.rados.2.smithi204.stdout:3444: left oid 8 (ObjNum 1198 snap 366 seq_num 1198) 2024-09-16T05:23:35.272 INFO:tasks.rados.rados.2.smithi204.stdout:3446: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:23:35.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:35.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:35.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:35.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:35.469 INFO:tasks.rados.rados.2.smithi204.stdout:3444: done (4 left) 2024-09-16T05:23:35.470 INFO:tasks.rados.rados.2.smithi204.stdout:3446: done (3 left) 2024-09-16T05:23:35.470 INFO:tasks.rados.rados.2.smithi204.stdout:3448: done (2 left) 2024-09-16T05:23:35.470 INFO:tasks.rados.rados.2.smithi204.stdout:3450: done (1 left) 2024-09-16T05:23:35.470 INFO:tasks.rados.rados.2.smithi204.stdout:3451: snap_remove snap 364 2024-09-16T05:23:35.487 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:35.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:35.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:35.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:35.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:35.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:23:35.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:35.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:35.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:35.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:35.853 INFO:tasks.rados.rados.2.smithi204.stdout:3451: done (1 left) 2024-09-16T05:23:35.854 INFO:tasks.rados.rados.2.smithi204.stdout:3452: snap_remove snap 363 2024-09-16T05:23:35.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:35.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:35.941 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:23:35.941 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:23:35.941 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:23:35.942 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-16T05:23:35.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.1 2024-09-16T05:23:35.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:35.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:35.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:36.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:36.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:36.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:36.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: pgmap v1458: 127 pgs: 127 active+clean; 123 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-16T05:23:36.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:23:36.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:23:36.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: 1.15 scrub starts 2024-09-16T05:23:36.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: 1.15 scrub ok 2024-09-16T05:23:36.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: osdmap e865: 12 total, 12 up, 12 in 2024-09-16T05:23:36.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: 2.26 scrub starts 2024-09-16T05:23:36.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: 2.26 scrub ok 2024-09-16T05:23:36.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:35 smithi032 ceph-mon[27388]: osdmap e866: 12 total, 12 up, 12 in 2024-09-16T05:23:36.181 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:35 smithi032 systemd[1]: Stopping Ceph osd.1 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:23:36.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: pgmap v1458: 127 pgs: 127 active+clean; 123 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: 1.15 scrub starts 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: 1.15 scrub ok 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: osdmap e865: 12 total, 12 up, 12 in 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: 2.26 scrub starts 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: 2.26 scrub ok 2024-09-16T05:23:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:35 smithi196 ceph-mon[30802]: osdmap e866: 12 total, 12 up, 12 in 2024-09-16T05:23:36.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:36.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:36.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:36.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:36.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:36.376 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:36.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: pgmap v1458: 127 pgs: 127 active+clean; 123 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-16T05:23:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:23:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:23:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: 1.15 scrub starts 2024-09-16T05:23:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: 1.15 scrub ok 2024-09-16T05:23:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: osdmap e865: 12 total, 12 up, 12 in 2024-09-16T05:23:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: 2.26 scrub starts 2024-09-16T05:23:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: 2.26 scrub ok 2024-09-16T05:23:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:35 smithi204 ceph-mon[30933]: osdmap e866: 12 total, 12 up, 12 in 2024-09-16T05:23:36.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:36.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:36.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:36.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:36.625 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:36.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:36.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:36.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:36.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:36.772 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[46290]: 2024-09-16T05:23:36.515+0000 7f11166c2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:23:36.772 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[46290]: 2024-09-16T05:23:36.515+0000 7f11166c2640 -1 osd.1 866 *** Got signal Terminated *** 2024-09-16T05:23:36.772 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[46290]: 2024-09-16T05:23:36.515+0000 7f11166c2640 -1 osd.1 866 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:23:36.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:36.853 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:23:36.856 INFO:tasks.rados.rados.2.smithi204.stdout:3452: done (1 left) 2024-09-16T05:23:36.856 INFO:tasks.rados.rados.2.smithi204.stdout:3453: write (excl) oid 8 current snap is 366 2024-09-16T05:23:36.856 INFO:tasks.rados.rados.2.smithi204.stdout:3443: finishing write tid 3 to smithi20457400-47 2024-09-16T05:23:36.856 INFO:tasks.rados.rados.2.smithi204.stdout:3443: finishing write tid 4 to smithi20457400-47 2024-09-16T05:23:36.856 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1083 (ObjNum 1197 snap 366 seq_num 1197) dirty exists 2024-09-16T05:23:36.857 INFO:tasks.rados.rados.2.smithi204.stdout:3443: left oid 47 (ObjNum 1197 snap 366 seq_num 1197) 2024-09-16T05:23:36.857 INFO:tasks.rados.rados.2.smithi204.stdout:3453: seq_num 1201 ranges {585207=641470,1896521=288686} 2024-09-16T05:23:36.863 INFO:tasks.rados.rados.2.smithi204.stdout:3453: writing smithi20457400-8 from 585207 to 1226677 tid 1 2024-09-16T05:23:36.868 INFO:tasks.rados.rados.2.smithi204.stdout:3453: writing smithi20457400-8 from 1896521 to 2185207 tid 2 2024-09-16T05:23:36.868 INFO:tasks.rados.rados.2.smithi204.stdout:3443: done (1 left) 2024-09-16T05:23:36.869 INFO:tasks.rados.rados.2.smithi204.stdout:3454: copy_from oid 10 from oid 30 current snap is 366 2024-09-16T05:23:36.869 INFO:tasks.rados.rados.2.smithi204.stdout:3455: write (excl) oid 25 current snap is 366 2024-09-16T05:23:36.869 INFO:tasks.rados.rados.2.smithi204.stdout:3455: seq_num 1202 ranges {602014=433072,1671356=530658} 2024-09-16T05:23:36.874 INFO:tasks.rados.rados.2.smithi204.stdout:3455: writing smithi20457400-25 from 602014 to 1035086 tid 1 2024-09-16T05:23:36.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:36.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:36.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:36.881 INFO:tasks.rados.rados.2.smithi204.stdout:3455: writing smithi20457400-25 from 1671356 to 2202014 tid 2 2024-09-16T05:23:36.881 INFO:tasks.rados.rados.2.smithi204.stdout:3456: snap_remove snap 365 2024-09-16T05:23:36.977 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:23:36.977 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:23:36.977 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:23:36.978 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:23:36.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:37.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:37.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:37.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:37.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:37.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: 1.18 scrub starts 2024-09-16T05:23:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: 1.18 scrub ok 2024-09-16T05:23:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: 2.51 scrub starts 2024-09-16T05:23:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: 2.51 scrub ok 2024-09-16T05:23:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1045417783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: osd.1 marked itself down and dead 2024-09-16T05:23:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T05:23:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:23:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1045417783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:37.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:36 smithi032 ceph-mon[27388]: osdmap e867: 12 total, 11 up, 12 in 2024-09-16T05:23:37.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:37.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: 1.18 scrub starts 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: 1.18 scrub ok 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: 2.51 scrub starts 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: 2.51 scrub ok 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1045417783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: osd.1 marked itself down and dead 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1045417783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:36 smithi196 ceph-mon[30802]: osdmap e867: 12 total, 11 up, 12 in 2024-09-16T05:23:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:37.251 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:37.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:37.352 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:37.375 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:37.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:37.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: 1.18 scrub starts 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: 1.18 scrub ok 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: 2.51 scrub starts 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: 2.51 scrub ok 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1045417783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: osd.1 marked itself down and dead 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1045417783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:36 smithi204 ceph-mon[30933]: osdmap e867: 12 total, 11 up, 12 in 2024-09-16T05:23:37.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:37.498 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:37.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:37.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:37.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:37.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:37.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:37.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:37.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:37.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:37.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:3456: done (3 left) 2024-09-16T05:23:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:3457: write (excl) oid 31 current snap is 366 2024-09-16T05:23:37.857 INFO:tasks.rados.rados.2.smithi204.stdout:3457: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-09-16T05:23:37.864 INFO:tasks.rados.rados.2.smithi204.stdout:3457: writing smithi20457400-31 from 618821 to 1127142 tid 1 2024-09-16T05:23:37.870 INFO:tasks.rados.rados.2.smithi204.stdout:3457: writing smithi20457400-31 from 1613485 to 2144562 tid 2 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3458: delete oid 6 current snap is 366 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3453: finishing write tid 1 to smithi20457400-8 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3453: finishing write tid 2 to smithi20457400-8 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3453: finishing write tid 3 to smithi20457400-8 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3453: finishing write tid 4 to smithi20457400-8 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1439 (ObjNum 1201 snap 366 seq_num 1201) dirty exists 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3453: left oid 8 (ObjNum 1201 snap 366 seq_num 1201) 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3455: finishing write tid 1 to smithi20457400-25 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3455: finishing write tid 2 to smithi20457400-25 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3455: finishing write tid 3 to smithi20457400-25 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3455: finishing write tid 4 to smithi20457400-25 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1274 (ObjNum 1202 snap 366 seq_num 1202) dirty exists 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3455: left oid 25 (ObjNum 1202 snap 366 seq_num 1202) 2024-09-16T05:23:37.871 INFO:tasks.rados.rados.2.smithi204.stdout:3457: finishing write tid 1 to smithi20457400-31 2024-09-16T05:23:37.872 INFO:tasks.rados.rados.2.smithi204.stdout:3453: done (4 left) 2024-09-16T05:23:37.872 INFO:tasks.rados.rados.2.smithi204.stdout:3455: done (3 left) 2024-09-16T05:23:37.872 INFO:tasks.rados.rados.2.smithi204.stdout:3458: done (2 left) 2024-09-16T05:23:37.872 INFO:tasks.rados.rados.2.smithi204.stdout:3459: write oid 2 current snap is 366 2024-09-16T05:23:37.872 INFO:tasks.rados.rados.2.smithi204.stdout:3459: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-09-16T05:23:37.879 INFO:tasks.rados.rados.2.smithi204.stdout:3459: writing smithi20457400-2 from 652435 to 1311254 tid 1 2024-09-16T05:23:37.882 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:37.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:37.885 INFO:tasks.rados.rados.2.smithi204.stdout:3459: writing smithi20457400-2 from 1730449 to 2252435 tid 2 2024-09-16T05:23:37.885 INFO:tasks.rados.rados.2.smithi204.stdout:3460: delete oid 5 current snap is 366 2024-09-16T05:23:37.885 INFO:tasks.rados.rados.2.smithi204.stdout:3457: finishing write tid 2 to smithi20457400-31 2024-09-16T05:23:37.885 INFO:tasks.rados.rados.2.smithi204.stdout:3457: finishing write tid 3 to smithi20457400-31 2024-09-16T05:23:37.885 INFO:tasks.rados.rados.2.smithi204.stdout:3457: finishing write tid 4 to smithi20457400-31 2024-09-16T05:23:37.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 1433 (ObjNum 1203 snap 366 seq_num 1203) dirty exists 2024-09-16T05:23:37.885 INFO:tasks.rados.rados.2.smithi204.stdout:3457: left oid 31 (ObjNum 1203 snap 366 seq_num 1203) 2024-09-16T05:23:37.885 INFO:tasks.rados.rados.2.smithi204.stdout:3459: finishing write tid 1 to smithi20457400-2 2024-09-16T05:23:37.886 INFO:tasks.rados.rados.2.smithi204.stdout:3457: done (3 left) 2024-09-16T05:23:37.886 INFO:tasks.rados.rados.2.smithi204.stdout:3460: done (2 left) 2024-09-16T05:23:37.886 INFO:tasks.rados.rados.2.smithi204.stdout:3461: rollback oid 14 current snap is 366 2024-09-16T05:23:37.886 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 14 to 358 2024-09-16T05:23:37.886 INFO:tasks.rados.rados.2.smithi204.stdout:3462: write oid 36 current snap is 366 2024-09-16T05:23:37.886 INFO:tasks.rados.rados.2.smithi204.stdout:3462: seq_num 1207 ranges {686049=409317,1847413=438636} 2024-09-16T05:23:37.892 INFO:tasks.rados.rados.2.smithi204.stdout:3462: writing smithi20457400-36 from 686049 to 1095366 tid 1 2024-09-16T05:23:37.899 INFO:tasks.rados.rados.2.smithi204.stdout:3462: writing smithi20457400-36 from 1847413 to 2286049 tid 2 2024-09-16T05:23:37.899 INFO:tasks.rados.rados.2.smithi204.stdout:3459: finishing write tid 2 to smithi20457400-2 2024-09-16T05:23:37.899 INFO:tasks.rados.rados.2.smithi204.stdout:3459: finishing write tid 3 to smithi20457400-2 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3459: finishing write tid 4 to smithi20457400-2 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 552 (ObjNum 1205 snap 366 seq_num 1205) dirty exists 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3459: left oid 2 (ObjNum 1205 snap 366 seq_num 1205) 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3462: finishing write tid 1 to smithi20457400-36 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3459: done (3 left) 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3463: copy_from oid 41 from oid 17 current snap is 366 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3464: read oid 21 snap -1 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3464: expect deleted 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3465: write (excl) oid 29 current snap is 366 2024-09-16T05:23:37.900 INFO:tasks.rados.rados.2.smithi204.stdout:3465: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-09-16T05:23:37.906 INFO:tasks.rados.rados.2.smithi204.stdout:3465: writing smithi20457400-29 from 702856 to 1187422 tid 1 2024-09-16T05:23:37.912 INFO:tasks.rados.rados.2.smithi204.stdout:3465: writing smithi20457400-29 from 1789542 to 2302856 tid 2 2024-09-16T05:23:37.912 INFO:tasks.rados.rados.2.smithi204.stdout:3466: write oid 45 current snap is 366 2024-09-16T05:23:37.912 INFO:tasks.rados.rados.2.smithi204.stdout:3466: seq_num 1209 ranges {719663=676168,1964377=355286} 2024-09-16T05:23:37.919 INFO:tasks.rados.rados.2.smithi204.stdout:3466: writing smithi20457400-45 from 719663 to 1395831 tid 1 2024-09-16T05:23:37.925 INFO:tasks.rados.rados.2.smithi204.stdout:3466: writing smithi20457400-45 from 1964377 to 2319663 tid 2 2024-09-16T05:23:37.925 INFO:tasks.rados.rados.2.smithi204.stdout:3467: rollback oid 12 current snap is 366 2024-09-16T05:23:37.925 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 361 2024-09-16T05:23:37.925 INFO:tasks.rados.rados.2.smithi204.stdout:3462: finishing write tid 2 to smithi20457400-36 2024-09-16T05:23:37.926 INFO:tasks.rados.rados.2.smithi204.stdout:3463: finishing copy_from racing read to smithi20457400-41 2024-09-16T05:23:37.926 INFO:tasks.rados.rados.2.smithi204.stdout:3465: finishing write tid 1 to smithi20457400-29 2024-09-16T05:23:37.926 INFO:tasks.rados.rados.2.smithi204.stdout:3463: finishing copy_from to smithi20457400-41 2024-09-16T05:23:37.926 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1138 (ObjNum 1195 snap 366 seq_num 1195) dirty exists 2024-09-16T05:23:37.926 INFO:tasks.rados.rados.2.smithi204.stdout:3465: finishing write tid 2 to smithi20457400-29 2024-09-16T05:23:37.926 INFO:tasks.rados.rados.2.smithi204.stdout:3465: finishing write tid 3 to smithi20457400-29 2024-09-16T05:23:37.926 INFO:tasks.rados.rados.2.smithi204.stdout:3465: finishing write tid 4 to smithi20457400-29 2024-09-16T05:23:37.927 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1247 (ObjNum 1208 snap 366 seq_num 1208) dirty exists 2024-09-16T05:23:37.927 INFO:tasks.rados.rados.2.smithi204.stdout:3465: left oid 29 (ObjNum 1208 snap 366 seq_num 1208) 2024-09-16T05:23:37.927 INFO:tasks.rados.rados.2.smithi204.stdout:3466: finishing write tid 1 to smithi20457400-45 2024-09-16T05:23:37.927 INFO:tasks.rados.rados.2.smithi204.stdout:3463: done (7 left) 2024-09-16T05:23:37.927 INFO:tasks.rados.rados.2.smithi204.stdout:3464: done (6 left) 2024-09-16T05:23:37.927 INFO:tasks.rados.rados.2.smithi204.stdout:3465: done (5 left) 2024-09-16T05:23:37.927 INFO:tasks.rados.rados.2.smithi204.stdout:3468: write oid 5 current snap is 366 2024-09-16T05:23:37.927 INFO:tasks.rados.rados.2.smithi204.stdout:3468: seq_num 1210 ranges {736470=751417,2022859=313611} 2024-09-16T05:23:37.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:23:37.932 INFO:tasks.rados.rados.2.smithi204.stdout:3468: writing smithi20457400-5 from 736470 to 1487887 tid 1 2024-09-16T05:23:37.937 INFO:tasks.rados.rados.2.smithi204.stdout:3468: writing smithi20457400-5 from 2022859 to 2336470 tid 2 2024-09-16T05:23:37.937 INFO:tasks.rados.rados.2.smithi204.stdout:3469: write (excl) oid 25 current snap is 366 2024-09-16T05:23:37.937 INFO:tasks.rados.rados.2.smithi204.stdout:3469: seq_num 1211 ranges {753277=426666,1681341=545753} 2024-09-16T05:23:37.944 INFO:tasks.rados.rados.2.smithi204.stdout:3469: writing smithi20457400-25 from 753277 to 1179943 tid 1 2024-09-16T05:23:37.951 INFO:tasks.rados.rados.2.smithi204.stdout:3469: writing smithi20457400-25 from 1681341 to 2227094 tid 2 2024-09-16T05:23:37.951 INFO:tasks.rados.rados.2.smithi204.stdout:3470: read oid 41 snap -1 2024-09-16T05:23:37.951 INFO:tasks.rados.rados.2.smithi204.stdout:3470: expect (ObjNum 1195 snap 366 seq_num 1195) 2024-09-16T05:23:37.951 INFO:tasks.rados.rados.2.smithi204.stdout:3471: rollback oid 50 current snap is 366 2024-09-16T05:23:37.951 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 362 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:3467: finishing rollback tid 0 to smithi20457400-12 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:3467: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:3467: finishing rollback tid 2 to smithi20457400-12 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1251 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:3466: finishing write tid 2 to smithi20457400-45 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:3466: finishing write tid 3 to smithi20457400-45 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:3466: finishing write tid 4 to smithi20457400-45 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1272 (ObjNum 1209 snap 366 seq_num 1209) dirty exists 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:3466: left oid 45 (ObjNum 1209 snap 366 seq_num 1209) 2024-09-16T05:23:37.952 INFO:tasks.rados.rados.2.smithi204.stdout:3468: finishing write tid 1 to smithi20457400-5 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3468: finishing write tid 2 to smithi20457400-5 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3468: finishing write tid 3 to smithi20457400-5 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3468: finishing write tid 4 to smithi20457400-5 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 633 (ObjNum 1210 snap 366 seq_num 1210) dirty exists 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3468: left oid 5 (ObjNum 1210 snap 366 seq_num 1210) 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3469: finishing write tid 1 to smithi20457400-25 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3466: done (8 left) 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3467: done (7 left) 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3468: done (6 left) 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3472: copy_from oid 17 from oid 42 current snap is 366 2024-09-16T05:23:37.953 INFO:tasks.rados.rados.2.smithi204.stdout:3473: read oid 27 snap -1 2024-09-16T05:23:37.954 INFO:tasks.rados.rados.2.smithi204.stdout:3473: expect deleted 2024-09-16T05:23:37.954 INFO:tasks.rados.rados.2.smithi204.stdout:3474: rollback oid 9 current snap is 366 2024-09-16T05:23:37.954 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 9 to 361 2024-09-16T05:23:37.954 INFO:tasks.rados.rados.2.smithi204.stdout:3475: snap_create 2024-09-16T05:23:37.955 INFO:tasks.rados.rados.2.smithi204.stdout:3469: finishing write tid 2 to smithi20457400-25 2024-09-16T05:23:37.955 INFO:tasks.rados.rados.2.smithi204.stdout:3469: finishing write tid 3 to smithi20457400-25 2024-09-16T05:23:37.955 INFO:tasks.rados.rados.2.smithi204.stdout:3469: finishing write tid 4 to smithi20457400-25 2024-09-16T05:23:37.955 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1277 (ObjNum 1211 snap 366 seq_num 1211) dirty exists 2024-09-16T05:23:37.955 INFO:tasks.rados.rados.2.smithi204.stdout:3469: left oid 25 (ObjNum 1211 snap 366 seq_num 1211) 2024-09-16T05:23:37.955 INFO:tasks.rados.rados.2.smithi204.stdout:3471: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:23:37.955 INFO:tasks.rados.rados.2.smithi204.stdout:3471: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:23:37.955 INFO:tasks.rados.rados.2.smithi204.stdout:3471: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:23:37.956 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1160 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:23:37.956 INFO:tasks.rados.rados.2.smithi204.stdout:3470: expect (ObjNum 1195 snap 366 seq_num 1195) 2024-09-16T05:23:37.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:38.009 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:38.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:38.024 INFO:tasks.rados.rados.2.smithi204.stdout:3474: finishing rollback tid 1 to smithi20457400-9 2024-09-16T05:23:38.024 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T05:23:38.024 INFO:tasks.rados.rados.2.smithi204.stdout:3472: finishing copy_from racing read to smithi20457400-17 2024-09-16T05:23:38.024 INFO:tasks.rados.rados.2.smithi204.stdout:3472: finishing copy_from to smithi20457400-17 2024-09-16T05:23:38.024 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1239 (ObjNum 1178 snap 360 seq_num 1178) dirty exists 2024-09-16T05:23:38.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:38.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:38.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:38.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:37 smithi032 podman[174657]: 2024-09-16 05:23:37.960886553 +0000 UTC m=+1.579639177 container died d620229b0db0409a8061fe9c492c86fd5e91fb0aadeeb4bf4ff23cf4ca646a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-09-16T05:23:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: pgmap v1461: 127 pgs: 127 active+clean; 125 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 998 KiB/s wr, 7 op/s 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: 2.50 scrub starts 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: 2.50 scrub ok 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: 1.5 deep-scrub starts 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: 1.5 deep-scrub ok 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:23:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:37 smithi032 ceph-mon[27388]: osdmap e868: 12 total, 11 up, 12 in 2024-09-16T05:23:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: pgmap v1461: 127 pgs: 127 active+clean; 125 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 998 KiB/s wr, 7 op/s 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: 2.50 scrub starts 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: 2.50 scrub ok 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: 1.5 deep-scrub starts 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: 1.5 deep-scrub ok 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:23:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:37 smithi196 ceph-mon[30802]: osdmap e868: 12 total, 11 up, 12 in 2024-09-16T05:23:38.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:38.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:38.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:38.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:23:38.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:38.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:38.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:38.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: pgmap v1461: 127 pgs: 127 active+clean; 125 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 998 KiB/s wr, 7 op/s 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: 2.50 scrub starts 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: 2.50 scrub ok 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: 1.5 deep-scrub starts 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: 1.5 deep-scrub ok 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:23:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:37 smithi204 ceph-mon[30933]: osdmap e868: 12 total, 11 up, 12 in 2024-09-16T05:23:38.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:38.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:38.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:38.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:38.572 INFO:tasks.rados.rados.2.smithi204.stdout:3462: finishing write tid 3 to smithi20457400-36 2024-09-16T05:23:38.572 INFO:tasks.rados.rados.2.smithi204.stdout:3462: finishing write tid 4 to smithi20457400-36 2024-09-16T05:23:38.572 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 800 (ObjNum 1207 snap 366 seq_num 1207) dirty exists 2024-09-16T05:23:38.572 INFO:tasks.rados.rados.2.smithi204.stdout:3462: left oid 36 (ObjNum 1207 snap 366 seq_num 1207) 2024-09-16T05:23:38.581 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:38 smithi032 podman[174657]: 2024-09-16 05:23:38.47703853 +0000 UTC m=+2.095791154 container cleanup d620229b0db0409a8061fe9c492c86fd5e91fb0aadeeb4bf4ff23cf4ca646a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e) 2024-09-16T05:23:38.581 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:38 smithi032 bash[174657]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1 2024-09-16T05:23:38.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:38.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:38.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:38.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:38.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:38.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:38.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:38.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:38.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:38.943 INFO:tasks.rados.rados.2.smithi204.stdout:3462: done (9 left) 2024-09-16T05:23:38.943 INFO:tasks.rados.rados.2.smithi204.stdout:3469: done (8 left) 2024-09-16T05:23:38.943 INFO:tasks.rados.rados.2.smithi204.stdout:3470: done (7 left) 2024-09-16T05:23:38.943 INFO:tasks.rados.rados.2.smithi204.stdout:3471: done (6 left) 2024-09-16T05:23:38.943 INFO:tasks.rados.rados.2.smithi204.stdout:3472: done (5 left) 2024-09-16T05:23:38.943 INFO:tasks.rados.rados.2.smithi204.stdout:3473: done (4 left) 2024-09-16T05:23:38.944 INFO:tasks.rados.rados.2.smithi204.stdout:3474: done (3 left) 2024-09-16T05:23:38.944 INFO:tasks.rados.rados.2.smithi204.stdout:3475: done (2 left) 2024-09-16T05:23:38.944 INFO:tasks.rados.rados.2.smithi204.stdout:3476: rollback oid 49 current snap is 367 2024-09-16T05:23:38.944 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 49 to 362 2024-09-16T05:23:38.944 INFO:tasks.rados.rados.2.smithi204.stdout:3477: read oid 2 snap -1 2024-09-16T05:23:38.944 INFO:tasks.rados.rados.2.smithi204.stdout:3477: expect (ObjNum 1205 snap 366 seq_num 1205) 2024-09-16T05:23:38.944 INFO:tasks.rados.rados.2.smithi204.stdout:3478: write (excl) oid 12 current snap is 367 2024-09-16T05:23:38.944 INFO:tasks.rados.rados.2.smithi204.stdout:3478: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-09-16T05:23:38.954 INFO:tasks.rados.rados.2.smithi204.stdout:3478: writing smithi20457400-12 from 770084 to 1271999 tid 1 2024-09-16T05:23:38.956 INFO:tasks.rados.rados.2.smithi204.stdout:3478: writing smithi20457400-12 from 2023470 to 2370084 tid 2 2024-09-16T05:23:38.956 INFO:tasks.rados.rados.2.smithi204.stdout:3479: snap_create 2024-09-16T05:23:38.956 INFO:tasks.rados.rados.2.smithi204.stdout:3478: finishing write tid 1 to smithi20457400-12 2024-09-16T05:23:38.956 INFO:tasks.rados.rados.2.smithi204.stdout:3477: expect (ObjNum 1205 snap 366 seq_num 1205) 2024-09-16T05:23:39.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:39.045 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:39.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:39.079 INFO:tasks.rados.rados.2.smithi204.stdout:3478: finishing write tid 2 to smithi20457400-12 2024-09-16T05:23:39.079 INFO:tasks.rados.rados.2.smithi204.stdout:3478: finishing write tid 3 to smithi20457400-12 2024-09-16T05:23:39.079 INFO:tasks.rados.rados.2.smithi204.stdout:3478: finishing write tid 4 to smithi20457400-12 2024-09-16T05:23:39.079 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1255 (ObjNum 1212 snap 367 seq_num 1212) dirty exists 2024-09-16T05:23:39.079 INFO:tasks.rados.rados.2.smithi204.stdout:3478: left oid 12 (ObjNum 1212 snap 367 seq_num 1212) 2024-09-16T05:23:39.116 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:39.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:39.169 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:39.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:39.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:39 smithi032 podman[174769]: 2024-09-16 05:23:39.05650214 +0000 UTC m=+1.725905351 container remove d620229b0db0409a8061fe9c492c86fd5e91fb0aadeeb4bf4ff23cf4ca646a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-16T05:23:39.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:38 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:23:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:38 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:23:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:38 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:23:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:38 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:23:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:38 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:23:39.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:38 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:23:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:38 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:23:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:38 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:23:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:38 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:23:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:38 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:23:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:38 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:23:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:38 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:23:39.271 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:39.293 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:39.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:39.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:39.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:39.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:38 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:23:39.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:38 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:23:39.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:38 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:23:39.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:38 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:23:39.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:38 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:23:39.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:38 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:23:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:39.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:39.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:39.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:39.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:39.639 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:23:39.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:39.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:39.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:39.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:39.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:39.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:39.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:39.868 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:39.868 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:39.888 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:39.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:39.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:39.910 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:39.926 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:39.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:39.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:39.931 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:39.944 INFO:tasks.rados.rados.2.smithi204.stdout:3477: done (5 left) 2024-09-16T05:23:39.945 INFO:tasks.rados.rados.2.smithi204.stdout:3478: done (4 left) 2024-09-16T05:23:39.945 INFO:tasks.rados.rados.2.smithi204.stdout:3479: done (3 left) 2024-09-16T05:23:39.945 INFO:tasks.rados.rados.2.smithi204.stdout:3480: write (excl) oid 1 current snap is 368 2024-09-16T05:23:39.945 INFO:tasks.rados.rados.2.smithi204.stdout:3480: seq_num 1213 ranges {786891=577164,2081952=304939} 2024-09-16T05:23:39.951 INFO:tasks.rados.rados.2.smithi204.stdout:3480: writing smithi20457400-1 from 786891 to 1364055 tid 1 2024-09-16T05:23:39.954 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:39.954 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:39.957 INFO:tasks.rados.rados.2.smithi204.stdout:3480: writing smithi20457400-1 from 2081952 to 2386891 tid 2 2024-09-16T05:23:39.957 INFO:tasks.rados.rados.2.smithi204.stdout:3480: finishing write tid 1 to smithi20457400-1 2024-09-16T05:23:39.957 INFO:tasks.rados.rados.2.smithi204.stdout:3481: snap_create 2024-09-16T05:23:39.959 INFO:tasks.rados.rados.2.smithi204.stdout:3480: finishing write tid 2 to smithi20457400-1 2024-09-16T05:23:39.959 INFO:tasks.rados.rados.2.smithi204.stdout:3480: finishing write tid 3 to smithi20457400-1 2024-09-16T05:23:39.959 INFO:tasks.rados.rados.2.smithi204.stdout:3480: finishing write tid 4 to smithi20457400-1 2024-09-16T05:23:39.959 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 1144 (ObjNum 1213 snap 368 seq_num 1213) dirty exists 2024-09-16T05:23:39.959 INFO:tasks.rados.rados.2.smithi204.stdout:3480: left oid 1 (ObjNum 1213 snap 368 seq_num 1213) 2024-09-16T05:23:39.976 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:39.997 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:39.998 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:40.018 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:40.019 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:40.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:40.040 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:40.040 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:40.040 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: pgmap v1464: 127 pgs: 2 undersized+peered, 5 peering, 9 stale+active+clean, 111 active+clean; 137 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 18 op/s 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: osdmap e869: 12 total, 11 up, 12 in 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1817585449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: 1.1c scrub starts 2024-09-16T05:23:40.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:39 smithi032 ceph-mon[27388]: 1.1c scrub ok 2024-09-16T05:23:40.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:40.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:40.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:40.061 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:40.086 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:40.086 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:40.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:40.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:40.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: pgmap v1464: 127 pgs: 2 undersized+peered, 5 peering, 9 stale+active+clean, 111 active+clean; 137 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 18 op/s 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: osdmap e869: 12 total, 11 up, 12 in 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1817585449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: 1.1c scrub starts 2024-09-16T05:23:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:39 smithi196 ceph-mon[30802]: 1.1c scrub ok 2024-09-16T05:23:40.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:40.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:40.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:40.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:40.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: pgmap v1464: 127 pgs: 2 undersized+peered, 5 peering, 9 stale+active+clean, 111 active+clean; 137 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 18 op/s 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: osdmap e869: 12 total, 11 up, 12 in 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1817585449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: 1.1c scrub starts 2024-09-16T05:23:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:39 smithi204 ceph-mon[30933]: 1.1c scrub ok 2024-09-16T05:23:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:40.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:40.529 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:40.551 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:40.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:40.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:40.676 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:40.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:40.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:40.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:40.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:3480: done (4 left) 2024-09-16T05:23:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:3481: done (3 left) 2024-09-16T05:23:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:3482: read oid 40 snap 361 2024-09-16T05:23:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:3482: expect deleted 2024-09-16T05:23:40.863 INFO:tasks.rados.rados.2.smithi204.stdout:3483: snap_remove snap 366 2024-09-16T05:23:40.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:23:40.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:40.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:40.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:23:40.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:23:41.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:41.058 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:41.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:41.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:40 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:23:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:40 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:23:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:40 smithi032 ceph-mon[27388]: osdmap e870: 12 total, 11 up, 12 in 2024-09-16T05:23:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:40 smithi032 ceph-mon[27388]: 2.29 scrub starts 2024-09-16T05:23:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:40 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:23:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:40 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:23:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:40 smithi032 ceph-mon[27388]: osdmap e871: 12 total, 11 up, 12 in 2024-09-16T05:23:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:41.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:41.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:40 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:23:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:40 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:23:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:40 smithi196 ceph-mon[30802]: osdmap e870: 12 total, 11 up, 12 in 2024-09-16T05:23:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:40 smithi196 ceph-mon[30802]: 2.29 scrub starts 2024-09-16T05:23:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:40 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:23:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:40 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:23:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:40 smithi196 ceph-mon[30802]: osdmap e871: 12 total, 11 up, 12 in 2024-09-16T05:23:41.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:41.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:41.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:41.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:23:41.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:40 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:23:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:40 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:23:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:40 smithi204 ceph-mon[30933]: osdmap e870: 12 total, 11 up, 12 in 2024-09-16T05:23:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:40 smithi204 ceph-mon[30933]: 2.29 scrub starts 2024-09-16T05:23:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:40 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:23:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:40 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:23:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:40 smithi204 ceph-mon[30933]: osdmap e871: 12 total, 11 up, 12 in 2024-09-16T05:23:41.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:41.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:41.517 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:41.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:41.559 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:41.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:41.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:41.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:41.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:41.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:41.812 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:41.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:41.864 INFO:tasks.rados.rados.2.smithi204.stdout:3483: done (4 left) 2024-09-16T05:23:41.864 INFO:tasks.rados.rados.2.smithi204.stdout:3484: copy_from oid 43 from oid 15 current snap is 369 2024-09-16T05:23:41.864 INFO:tasks.rados.rados.2.smithi204.stdout:3482: done (4 left) 2024-09-16T05:23:41.865 INFO:tasks.rados.rados.2.smithi204.stdout:3485: snap_remove snap 367 2024-09-16T05:23:41.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:41.920 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:41 smithi032 podman[175258]: 2024-09-16 05:23:41.563140982 +0000 UTC m=+0.027752170 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:23:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:41.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:41.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:42.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:42.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:42.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:42.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:42.191 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:42.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 1.d scrub starts 2024-09-16T05:23:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 1.d scrub ok 2024-09-16T05:23:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: pgmap v1467: 127 pgs: 4 undersized+peered, 5 peering, 8 stale+active+clean, 110 active+clean; 139 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2024-09-16T05:23:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:23:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:23:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:23:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:23:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:23:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:23:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:23:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:23:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:23:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:23:42.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:41 smithi196 ceph-mon[30802]: osdmap e872: 12 total, 11 up, 12 in 2024-09-16T05:23:42.258 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:41 smithi032 podman[175258]: 2024-09-16 05:23:41.951351325 +0000 UTC m=+0.415962493 container create 88b59f4de46a2b77433b98d80b4d0a00bed3888049e3633853b10bc2969f0254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:23:42.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 1.d scrub starts 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 1.d scrub ok 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: pgmap v1467: 127 pgs: 4 undersized+peered, 5 peering, 8 stale+active+clean, 110 active+clean; 139 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:23:42.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:41 smithi032 ceph-mon[27388]: osdmap e872: 12 total, 11 up, 12 in 2024-09-16T05:23:42.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:42.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:42.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:42.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 1.d scrub starts 2024-09-16T05:23:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 1.d scrub ok 2024-09-16T05:23:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: pgmap v1467: 127 pgs: 4 undersized+peered, 5 peering, 8 stale+active+clean, 110 active+clean; 139 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2024-09-16T05:23:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:23:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:23:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:23:42.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:23:42.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:23:42.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:23:42.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:23:42.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:23:42.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:23:42.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:23:42.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:41 smithi204 ceph-mon[30933]: osdmap e872: 12 total, 11 up, 12 in 2024-09-16T05:23:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:42.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:42.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:42.517 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:42.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:42.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:42.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:42.600 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:42 smithi032 podman[175258]: 2024-09-16 05:23:42.396983732 +0000 UTC m=+0.861594897 container init 88b59f4de46a2b77433b98d80b4d0a00bed3888049e3633853b10bc2969f0254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-16T05:23:42.600 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:42 smithi032 podman[175258]: 2024-09-16 05:23:42.400546865 +0000 UTC m=+0.865158030 container start 88b59f4de46a2b77433b98d80b4d0a00bed3888049e3633853b10bc2969f0254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-09-16T05:23:42.600 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:42 smithi032 podman[175258]: 2024-09-16 05:23:42.480212992 +0000 UTC m=+0.944824170 container attach 88b59f4de46a2b77433b98d80b4d0a00bed3888049e3633853b10bc2969f0254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:23:42.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:42.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:42.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:42.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:42.826 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:42.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:42.863 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:23:42.864 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:23:42 smithi032 podman[175258]: 2024-09-16 05:23:42.779161509 +0000 UTC m=+1.243772691 container died 88b59f4de46a2b77433b98d80b4d0a00bed3888049e3633853b10bc2969f0254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-16T05:23:42.866 INFO:tasks.rados.rados.2.smithi204.stdout:3485: done (4 left) 2024-09-16T05:23:42.867 INFO:tasks.rados.rados.2.smithi204.stdout:3486: write (excl) oid 48 current snap is 369 2024-09-16T05:23:42.867 INFO:tasks.rados.rados.2.smithi204.stdout:3484: finishing copy_from to smithi20457400-43 2024-09-16T05:23:42.867 INFO:tasks.rados.rados.2.smithi204.stdout:3484: got expected ENOENT (src dne) 2024-09-16T05:23:42.867 INFO:tasks.rados.rados.2.smithi204.stdout:3486: seq_num 1214 ranges {403698=652413,1740434=663264} 2024-09-16T05:23:42.872 INFO:tasks.rados.rados.2.smithi204.stdout:3486: writing smithi20457400-48 from 403698 to 1056111 tid 1 2024-09-16T05:23:42.880 INFO:tasks.rados.rados.2.smithi204.stdout:3486: writing smithi20457400-48 from 1740434 to 2403698 tid 2 2024-09-16T05:23:42.880 INFO:tasks.rados.rados.2.smithi204.stdout:3487: delete oid 2 current snap is 369 2024-09-16T05:23:42.881 INFO:tasks.rados.rados.2.smithi204.stdout:3484: finishing copy_from racing read to smithi20457400-43 2024-09-16T05:23:42.881 INFO:tasks.rados.rados.2.smithi204.stdout:3486: finishing write tid 1 to smithi20457400-48 2024-09-16T05:23:42.883 INFO:tasks.rados.rados.2.smithi204.stdout:3486: finishing write tid 2 to smithi20457400-48 2024-09-16T05:23:42.883 INFO:tasks.rados.rados.2.smithi204.stdout:3486: finishing write tid 3 to smithi20457400-48 2024-09-16T05:23:42.884 INFO:tasks.rados.rados.2.smithi204.stdout:3484: done (5 left) 2024-09-16T05:23:42.884 INFO:tasks.rados.rados.2.smithi204.stdout:3487: done (4 left) 2024-09-16T05:23:42.884 INFO:tasks.rados.rados.2.smithi204.stdout:3488: snap_remove snap 361 2024-09-16T05:23:42.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:42.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:23:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:42.955 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:42.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:43.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:43.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:43.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:42 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:23:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:42 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:23:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4224603333' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:23:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:42 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4224603333' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:23:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:42 smithi032 ceph-mon[27388]: osdmap e873: 12 total, 11 up, 12 in 2024-09-16T05:23:43.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:43.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:43.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:43.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:42 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:23:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:42 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:23:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4224603333' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:23:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:42 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4224603333' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:23:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:42 smithi196 ceph-mon[30802]: osdmap e873: 12 total, 11 up, 12 in 2024-09-16T05:23:43.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:43.335 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:43.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:42 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:23:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:42 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:23:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4224603333' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:23:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:42 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4224603333' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:23:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:42 smithi204 ceph-mon[30933]: osdmap e873: 12 total, 11 up, 12 in 2024-09-16T05:23:43.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:43.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:43.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:43.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:43.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:43.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:43.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:43.719 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:43.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:43.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:43.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:43.868 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:43.869 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:43.871 INFO:tasks.rados.rados.2.smithi204.stdout:3486: finishing write tid 4 to smithi20457400-48 2024-09-16T05:23:43.871 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 783 (ObjNum 1214 snap 369 seq_num 1214) dirty exists 2024-09-16T05:23:43.871 INFO:tasks.rados.rados.2.smithi204.stdout:3486: left oid 48 (ObjNum 1214 snap 369 seq_num 1214) 2024-09-16T05:23:43.871 INFO:tasks.rados.rados.2.smithi204.stdout:3486: done (4 left) 2024-09-16T05:23:43.871 INFO:tasks.rados.rados.2.smithi204.stdout:3488: done (3 left) 2024-09-16T05:23:43.871 INFO:tasks.rados.rados.2.smithi204.stdout:3489: snap_remove snap 368 2024-09-16T05:23:43.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:43.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:43.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:44.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:44.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:44.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:44.201 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:44.224 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:44.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:44.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:43 smithi196 ceph-mon[30802]: pgmap v1470: 127 pgs: 13 undersized+peered, 5 peering, 3 undersized+degraded+peered, 106 active+clean; 144 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 9/140 objects degraded (6.429%) 2024-09-16T05:23:44.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:43 smithi196 ceph-mon[30802]: Health check failed: Degraded data redundancy: 9/140 objects degraded (6.429%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:43 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:23:44.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:43 smithi196 ceph-mon[30802]: osdmap e874: 12 total, 11 up, 12 in 2024-09-16T05:23:44.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:44.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:44.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:44.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:43 smithi032 ceph-mon[27388]: pgmap v1470: 127 pgs: 13 undersized+peered, 5 peering, 3 undersized+degraded+peered, 106 active+clean; 144 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 9/140 objects degraded (6.429%) 2024-09-16T05:23:44.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:43 smithi032 ceph-mon[27388]: Health check failed: Degraded data redundancy: 9/140 objects degraded (6.429%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:44.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:43 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:23:44.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:43 smithi032 ceph-mon[27388]: osdmap e874: 12 total, 11 up, 12 in 2024-09-16T05:23:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:43 smithi204 ceph-mon[30933]: pgmap v1470: 127 pgs: 13 undersized+peered, 5 peering, 3 undersized+degraded+peered, 106 active+clean; 144 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 9/140 objects degraded (6.429%) 2024-09-16T05:23:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:43 smithi204 ceph-mon[30933]: Health check failed: Degraded data redundancy: 9/140 objects degraded (6.429%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:43 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:23:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:43 smithi204 ceph-mon[30933]: osdmap e874: 12 total, 11 up, 12 in 2024-09-16T05:23:44.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:44.474 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:44.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:44.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:44.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:44.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:44.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:23:44.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:44.728 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:44.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:44.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:44.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:44.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:44.871 INFO:tasks.rados.rados.2.smithi204.stdout:3489: done (3 left) 2024-09-16T05:23:44.871 INFO:tasks.rados.rados.2.smithi204.stdout:3490: read oid 9 snap -1 2024-09-16T05:23:44.872 INFO:tasks.rados.rados.2.smithi204.stdout:3490: expect deleted 2024-09-16T05:23:44.872 INFO:tasks.rados.rados.2.smithi204.stdout:3491: write (excl) oid 24 current snap is 369 2024-09-16T05:23:44.872 INFO:tasks.rados.rados.2.smithi204.stdout:3491: seq_num 1216 ranges {437312=402911,1341045=696749} 2024-09-16T05:23:44.876 INFO:tasks.rados.rados.2.smithi204.stdout:3491: writing smithi20457400-24 from 437312 to 840223 tid 1 2024-09-16T05:23:44.883 INFO:tasks.rados.rados.2.smithi204.stdout:3491: writing smithi20457400-24 from 1341045 to 2037794 tid 2 2024-09-16T05:23:44.883 INFO:tasks.rados.rados.2.smithi204.stdout:3492: rollback oid 34 current snap is 369 2024-09-16T05:23:44.883 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 34 to 362 2024-09-16T05:23:44.883 INFO:tasks.rados.rados.2.smithi204.stdout:3490: done (5 left) 2024-09-16T05:23:44.883 INFO:tasks.rados.rados.2.smithi204.stdout:3493: rollback oid 17 current snap is 369 2024-09-16T05:23:44.883 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 17 to 362 2024-09-16T05:23:44.883 INFO:tasks.rados.rados.2.smithi204.stdout:3494: read oid 4 snap -1 2024-09-16T05:23:44.883 INFO:tasks.rados.rados.2.smithi204.stdout:3494: expect deleted 2024-09-16T05:23:44.884 INFO:tasks.rados.rados.2.smithi204.stdout:3495: copy_from oid 38 from oid 39 current snap is 369 2024-09-16T05:23:44.884 INFO:tasks.rados.rados.2.smithi204.stdout:3496: delete oid 28 current snap is 369 2024-09-16T05:23:44.884 INFO:tasks.rados.rados.2.smithi204.stdout:3493: finishing rollback tid 0 to smithi20457400-17 2024-09-16T05:23:44.885 INFO:tasks.rados.rados.2.smithi204.stdout:3493: finishing rollback tid 1 to smithi20457400-17 2024-09-16T05:23:44.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1241 (ObjNum 145 snap 0 seq_num 4282256688) dirty dne 2024-09-16T05:23:44.885 INFO:tasks.rados.rados.2.smithi204.stdout:3493: done (8 left) 2024-09-16T05:23:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:3496: done (7 left) 2024-09-16T05:23:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:3497: write oid 2 current snap is 369 2024-09-16T05:23:44.886 INFO:tasks.rados.rados.2.smithi204.stdout:3497: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-16T05:23:44.892 INFO:tasks.rados.rados.2.smithi204.stdout:3497: writing smithi20457400-2 from 470926 to 1140688 tid 1 2024-09-16T05:23:44.899 INFO:tasks.rados.rados.2.smithi204.stdout:3497: writing smithi20457400-2 from 1574362 to 2274780 tid 2 2024-09-16T05:23:44.899 INFO:tasks.rados.rados.2.smithi204.stdout:3498: write (excl) oid 8 current snap is 369 2024-09-16T05:23:44.899 INFO:tasks.rados.rados.2.smithi204.stdout:3498: seq_num 1219 ranges {487733=745011,1632844=560429} 2024-09-16T05:23:44.903 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:44.905 INFO:tasks.rados.rados.2.smithi204.stdout:3498: writing smithi20457400-8 from 487733 to 1232744 tid 1 2024-09-16T05:23:44.911 INFO:tasks.rados.rados.2.smithi204.stdout:3498: writing smithi20457400-8 from 1632844 to 2193273 tid 2 2024-09-16T05:23:44.911 INFO:tasks.rados.rados.2.smithi204.stdout:3499: snap_create 2024-09-16T05:23:44.911 INFO:tasks.rados.rados.2.smithi204.stdout:3492: finishing rollback tid 0 to smithi20457400-34 2024-09-16T05:23:44.911 INFO:tasks.rados.rados.2.smithi204.stdout:3492: finishing rollback tid 1 to smithi20457400-34 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:3492: finishing rollback tid 2 to smithi20457400-34 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1269 (ObjNum 1139 snap 341 seq_num 1139) dirty exists 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:3497: finishing write tid 1 to smithi20457400-2 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:3497: finishing write tid 2 to smithi20457400-2 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:3497: finishing write tid 3 to smithi20457400-2 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:3497: finishing write tid 4 to smithi20457400-2 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 556 (ObjNum 1218 snap 369 seq_num 1218) dirty exists 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:3497: left oid 2 (ObjNum 1218 snap 369 seq_num 1218) 2024-09-16T05:23:44.912 INFO:tasks.rados.rados.2.smithi204.stdout:3498: finishing write tid 1 to smithi20457400-8 2024-09-16T05:23:44.914 INFO:tasks.rados.rados.2.smithi204.stdout:3498: finishing write tid 2 to smithi20457400-8 2024-09-16T05:23:44.919 INFO:tasks.rados.rados.2.smithi204.stdout:3498: finishing write tid 3 to smithi20457400-8 2024-09-16T05:23:44.919 INFO:tasks.rados.rados.2.smithi204.stdout:3498: finishing write tid 4 to smithi20457400-8 2024-09-16T05:23:44.920 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1446 (ObjNum 1219 snap 369 seq_num 1219) dirty exists 2024-09-16T05:23:44.920 INFO:tasks.rados.rados.2.smithi204.stdout:3498: left oid 8 (ObjNum 1219 snap 369 seq_num 1219) 2024-09-16T05:23:44.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:44.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:44.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:45.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.1.service' 2024-09-16T05:23:45.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:45.102 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:23:45.102 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-16T05:23:45.102 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-09-16T05:23:45.102 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi032.front.sepia.ceph.com 2024-09-16T05:23:45.103 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.1 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-16T05:23:45.107 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:45.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:45.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:45.118 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:45.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:45.231 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:45.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:45.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:45.350 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.1/config 2024-09-16T05:23:45.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:45.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:45.388 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:45.389 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:45.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:45.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:45.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:45.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:45.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:45.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:45.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:23:45.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:45.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:45.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:45.803 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:45.803 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:45.803 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:23:45.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:45.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:45.846 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:45.857 INFO:tasks.rados.rados.2.smithi204.stdout:3492: done (9 left) 2024-09-16T05:23:45.857 INFO:tasks.rados.rados.2.smithi204.stdout:3494: done (8 left) 2024-09-16T05:23:45.857 INFO:tasks.rados.rados.2.smithi204.stdout:3497: done (7 left) 2024-09-16T05:23:45.857 INFO:tasks.rados.rados.2.smithi204.stdout:3498: done (6 left) 2024-09-16T05:23:45.857 INFO:tasks.rados.rados.2.smithi204.stdout:3499: done (5 left) 2024-09-16T05:23:45.857 INFO:tasks.rados.rados.2.smithi204.stdout:3500: snap_remove snap 369 2024-09-16T05:23:45.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:45.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:45.869 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:45.869 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:45.891 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:45.891 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:45.912 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:45.934 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:45.934 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:45 smithi032 ceph-mon[27388]: pgmap v1473: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 147 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 13 op/s; 9/142 objects degraded (6.338%) 2024-09-16T05:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:45 smithi032 ceph-mon[27388]: osdmap e875: 12 total, 11 up, 12 in 2024-09-16T05:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:45 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1777483349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:45 smithi032 ceph-mon[27388]: osdmap e876: 12 total, 11 up, 12 in 2024-09-16T05:23:45.957 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:45.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:45.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:45.979 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:45.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:45.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:46.001 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:46.002 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:46.029 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:46.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:46.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:23:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:46.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:46.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:45 smithi204 ceph-mon[30933]: pgmap v1473: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 147 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 13 op/s; 9/142 objects degraded (6.338%) 2024-09-16T05:23:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:45 smithi204 ceph-mon[30933]: osdmap e875: 12 total, 11 up, 12 in 2024-09-16T05:23:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:45 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:23:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1777483349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:46.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:45 smithi204 ceph-mon[30933]: osdmap e876: 12 total, 11 up, 12 in 2024-09-16T05:23:46.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:46.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:45 smithi196 ceph-mon[30802]: pgmap v1473: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 147 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 13 op/s; 9/142 objects degraded (6.338%) 2024-09-16T05:23:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:45 smithi196 ceph-mon[30802]: osdmap e875: 12 total, 11 up, 12 in 2024-09-16T05:23:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:45 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-16T05:23:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1777483349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:45 smithi196 ceph-mon[30802]: osdmap e876: 12 total, 11 up, 12 in 2024-09-16T05:23:46.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:46.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:46.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:46.337 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:46.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:46.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:46.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:46.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:46.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:46.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:46.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:46.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:46.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:46.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:46.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:46.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:46.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:46.795 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:23:46.795 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:23:46.796 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:23:46.796 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:23:46.796 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:23:46.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:46.860 INFO:tasks.rados.rados.2.smithi204.stdout:3500: done (5 left) 2024-09-16T05:23:46.860 INFO:tasks.rados.rados.2.smithi204.stdout:3501: copy_from oid 40 from oid 18 current snap is 370 2024-09-16T05:23:46.860 INFO:tasks.rados.rados.2.smithi204.stdout:3502: write (excl) oid 26 current snap is 370 2024-09-16T05:23:46.860 INFO:tasks.rados.rados.2.smithi204.stdout:3502: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-09-16T05:23:46.865 INFO:tasks.rados.rados.2.smithi204.stdout:3502: writing smithi20457400-26 from 504540 to 924800 tid 1 2024-09-16T05:23:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:46.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:46.874 INFO:tasks.rados.rados.2.smithi204.stdout:3502: writing smithi20457400-26 from 1574973 to 2279060 tid 2 2024-09-16T05:23:46.874 INFO:tasks.rados.rados.2.smithi204.stdout:3503: read oid 16 snap 362 2024-09-16T05:23:46.874 INFO:tasks.rados.rados.2.smithi204.stdout:3503: expect deleted 2024-09-16T05:23:46.874 INFO:tasks.rados.rados.2.smithi204.stdout:3504: rollback oid 11 current snap is 370 2024-09-16T05:23:46.874 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 11 to 358 2024-09-16T05:23:46.874 INFO:tasks.rados.rados.2.smithi204.stdout:3505: delete oid 37 current snap is 370 2024-09-16T05:23:46.879 INFO:tasks.rados.rados.2.smithi204.stdout:3504: finishing rollback tid 0 to smithi20457400-11 2024-09-16T05:23:46.879 INFO:tasks.rados.rados.2.smithi204.stdout:3504: finishing rollback tid 1 to smithi20457400-11 2024-09-16T05:23:46.880 INFO:tasks.rados.rados.2.smithi204.stdout:3504: finishing rollback tid 2 to smithi20457400-11 2024-09-16T05:23:46.880 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1381 (ObjNum 1149 snap 342 seq_num 1149) dirty exists 2024-09-16T05:23:46.880 INFO:tasks.rados.rados.2.smithi204.stdout:3501: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:23:46.881 INFO:tasks.rados.rados.2.smithi204.stdout:3504: done (9 left) 2024-09-16T05:23:46.881 INFO:tasks.rados.rados.2.smithi204.stdout:3505: done (8 left) 2024-09-16T05:23:46.881 INFO:tasks.rados.rados.2.smithi204.stdout:3506: snap_create 2024-09-16T05:23:46.885 INFO:tasks.rados.rados.2.smithi204.stdout:3501: finishing copy_from to smithi20457400-40 2024-09-16T05:23:46.885 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 549 (ObjNum 1168 snap 354 seq_num 1168) dirty exists 2024-09-16T05:23:46.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:23:46.975 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:23:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:46.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:47.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:47.121 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:47.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:47.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:47.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:47.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:47.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:47.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:47.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:47.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:47.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:47.407 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:23:47.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:47.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:23:47.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:47.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:47.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:47.617 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:47.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:47.714 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:47.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:47.747 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:47.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:47.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:47.871 INFO:tasks.rados.rados.2.smithi204.stdout:3501: done (8 left) 2024-09-16T05:23:47.872 INFO:tasks.rados.rados.2.smithi204.stdout:3506: done (7 left) 2024-09-16T05:23:47.872 INFO:tasks.rados.rados.2.smithi204.stdout:3507: snap_create 2024-09-16T05:23:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:47.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:47.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:47.925 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:47.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:48.000 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:48.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:48.047 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:47 smithi032 ceph-mon[27388]: pgmap v1476: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 150 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 7 op/s; 9/144 objects degraded (6.250%) 2024-09-16T05:23:48.047 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:47 smithi032 ceph-mon[27388]: osdmap e877: 12 total, 11 up, 12 in 2024-09-16T05:23:48.094 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:47 smithi196 ceph-mon[30802]: pgmap v1476: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 150 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 7 op/s; 9/144 objects degraded (6.250%) 2024-09-16T05:23:48.094 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:47 smithi196 ceph-mon[30802]: osdmap e877: 12 total, 11 up, 12 in 2024-09-16T05:23:48.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:48.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:48.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:47 smithi204 ceph-mon[30933]: pgmap v1476: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 150 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 7 op/s; 9/144 objects degraded (6.250%) 2024-09-16T05:23:48.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:47 smithi204 ceph-mon[30933]: osdmap e877: 12 total, 11 up, 12 in 2024-09-16T05:23:48.202 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:23:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:23:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:23:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:23:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:2511 2.3 2.e1e51803 (undecoded) ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e320)", 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 563.36921311900005, 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99828526200000001, 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 2511 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823072+0000", 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823073+0000", 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.04e-06 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823078+0000", 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6519999999999997e-06 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823091+0000", 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3277e-05 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823100+0000", 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4909999999999994e-06 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823141+0000", 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0187e-05 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-09-16T05:23:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:24.823147+0000", 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3929999999999996e-06 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821329+0000", 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99818146900000004 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:14:25.821357+0000", 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8753000000000001e-05 2024-09-16T05:23:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:136 2.34 e355/278 ondisk, result = 0)", 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 522.49051139300002, 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83858643600000005, 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701774+0000", 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701775+0000", 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6200000000000006e-07 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:05.701786+0000", 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0815000000000001e-05 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540112+0000", 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83832645699999997 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540147+0000", 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5416000000000001e-05 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540246+0000", 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8171999999999995e-05 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540254+0000", 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.540360+0000", 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010627400000000001 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3573 2.3 2:c018a787:::smithi20457400-18:head [write 1692526~473837] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:23:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 442.32654325599998, 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022224677000000002, 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3573 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865743+0000", 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0100000000000001e-06 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866150+0000", 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040748299999999999 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866163+0000", 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2187999999999999e-05 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866168+0000", 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4689999999999996e-06 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866251+0000", 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2911000000000006e-05 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866320+0000", 2024-09-16T05:23:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8596000000000005e-05 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866403+0000", 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3796999999999997e-05 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867984+0000", 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015806399999999999 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887897+0000", 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019913157000000001 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887934+0000", 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6482000000000001e-05 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887967+0000", 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2944000000000002e-05 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3574 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 2403500] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 442.326103703, 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039246627999999999, 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3574 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:23:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866182+0000", 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866205+0000", 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2592e-05 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866222+0000", 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7422e-05 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866231+0000", 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5220000000000007e-06 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867407+0000", 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001176014 2024-09-16T05:23:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867514+0000", 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106947 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867661+0000", 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000147396 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867987+0000", 2024-09-16T05:23:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032622400000000001 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905316+0000", 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037329222000000002 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905388+0000", 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1896000000000004e-05 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905428+0000", 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9830000000000003e-05 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3575 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 442.326045706, 2024-09-16T05:23:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039343647000000002, 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3575 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866240+0000", 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7999999999999995e-07 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866243+0000", 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5770000000000001e-06 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866249+0000", 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866254+0000", 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-06 2024-09-16T05:23:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867919+0000", 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016648609999999999 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867974+0000", 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4478000000000003e-05 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905464+0000", 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037490460000000003 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905516+0000", 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1894999999999997e-05 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905583+0000", 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3580 2.34 2:2cffa4d6:::smithi20457400-50:head [write 1351008~617495] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 442.315744962, 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030886027999999999, 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3580 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876542+0000", 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877114+0000", 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057156899999999996 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877126+0000", 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2751999999999999e-05 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877134+0000", 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.072e-06 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0603000000000003e-05 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877240+0000", 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4561000000000002e-05 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877317+0000", 2024-09-16T05:23:48.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7002999999999994e-05 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878479+0000", 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011619130000000001 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907362+0000", 2024-09-16T05:23:48.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028883039999999999 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907396+0000", 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4807000000000001e-05 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907426+0000", 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9821000000000001e-05 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3581 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2420307] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 442.31513416799999, 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030393973000000001, 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.238 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3581 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877152+0000", 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3099999999999996e-07 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877156+0000", 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7589999999999999e-06 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877169+0000", 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3129000000000001e-05 2024-09-16T05:23:48.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877175+0000", 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6259999999999997e-06 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877997+0000", 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082123199999999997 2024-09-16T05:23:48.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878081+0000", 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4914999999999996e-05 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878202+0000", 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120207 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878482+0000", 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027995500000000003 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907462+0000", 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028980381 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907500+0000", 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907545+0000", 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5120999999999997e-05 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3582 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 442.31509976199999, 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030495267999999999, 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.244 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3582 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877186+0000", 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-07 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877189+0000", 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6800000000000002e-06 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877196+0000", 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0650000000000001e-06 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877202+0000", 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.28e-06 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878412+0000", 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001210205 2024-09-16T05:23:48.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878470+0000", 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7759e-05 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907572+0000", 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029102086999999999 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907620+0000", 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8384000000000003e-05 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907681+0000", 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0244000000000001e-05 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:288 2.16 e446/273 ondisk, result = 0)", 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:23:48.249 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 431.30856840199999, 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65473014699999998, 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:23:48.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883718+0000", 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3799999999999996e-07 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883722+0000", 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.284e-06 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538184+0000", 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.654462031 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538201+0000", 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6634e-05 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538327+0000", 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012605799999999999 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538338+0000", 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1188e-05 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538447+0000", 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109014 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4460 2.3 2:c018a787:::smithi20457400-18:head [copy-from ver 511] snapc bc=[bc,bb,ba,b8] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 375.13298447900002, 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019695292, 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4460 2024-09-16T05:23:48.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059301+0000", 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9799999999999999e-07 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059304+0000", 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5490000000000001e-06 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059308+0000", 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3610000000000003e-06 2024-09-16T05:23:48.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059311+0000", 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9179999999999998e-06 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059364+0000", 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2790000000000001e-05 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.059422+0000", 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7929000000000001e-05 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.073174+0000", 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013752234 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.077028+0000", 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038540319999999999 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078924+0000", 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001896179 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078961+0000", 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7311000000000002e-05 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:33.078996+0000", 2024-09-16T05:23:48.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4591000000000001e-05 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5587 2.34 2:2d3f13f8:::smithi20457400-7:head [setxattr _header (57) in=7b,truncate 3260723] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 295.38644681, 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.147062895, 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.260 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5587 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805839+0000", 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8299999999999997e-07 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805842+0000", 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6790000000000001e-06 2024-09-16T05:23:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:48.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805846+0000", 2024-09-16T05:23:48.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4089999999999996e-06 2024-09-16T05:23:48.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805849+0000", 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.191e-06 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806887+0000", 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010375440000000001 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806953+0000", 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5833000000000005e-05 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807079+0000", 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012666500000000001 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.809689+0000", 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026092860000000002 2024-09-16T05:23:48.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952747+0000", 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14305815 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952826+0000", 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9773999999999999e-05 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952901+0000", 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4781000000000001e-05 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5588 2.34 2:2d3f13f8:::smithi20457400-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 295.386420337, 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14722005799999999, 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.268 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5588 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.65e-07 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805867+0000", 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.86e-06 2024-09-16T05:23:48.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805872+0000", 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805876+0000", 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0380000000000001e-06 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807396+0000", 2024-09-16T05:23:48.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001520448 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807470+0000", 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3695000000000005e-05 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952937+0000", 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14546690300000001 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953007+0000", 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9801999999999993e-05 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953085+0000", 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8121999999999995e-05 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:534 2.34 e617/278 ondisk, result = 0)", 2024-09-16T05:23:48.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 261.38013020099999, 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93956844500000003, 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812156+0000", 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0599999999999999e-07 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812160+0000", 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.437e-06 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751436+0000", 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93927549399999999 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751483+0000", 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7117999999999998e-05 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751598+0000", 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115448 2024-09-16T05:23:48.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751609+0000", 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1156e-05 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751723+0000", 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113986 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7360 2.3 2:c018a787:::smithi20457400-18:head [write 443982~796776,stat] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 131.29533154200001, 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.046574619999999997, 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7360 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.278 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896958+0000", 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3869999999999997e-06 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897803+0000", 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084549400000000002 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897817+0000", 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3172e-05 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897822+0000", 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7400000000000001e-06 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897880+0000", 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7275999999999998e-05 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897887+0000", 2024-09-16T05:23:48.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0450000000000003e-06 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899506+0000", 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016191300000000001 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899612+0000", 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010592200000000001 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899807+0000", 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019501699999999999 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.900940+0000", 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011327760000000001 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943442+0000", 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.042502861000000003 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943488+0000", 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5266000000000003e-05 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943528+0000", 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0534000000000002e-05 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7361 2.3 2:c018a787:::smithi20457400-18:head [write 1989231~479018] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 131.288666662, 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040030532000000001, 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7361 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.285 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903620+0000", 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.885e-06 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904044+0000", 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042330699999999999 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904056+0000", 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2088e-05 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904064+0000", 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3820000000000004e-06 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904117+0000", 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2352000000000003e-05 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904226+0000", 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109895 2024-09-16T05:23:48.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904403+0000", 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017676899999999999 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.905359+0000", 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00095604999999999996 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943572+0000", 2024-09-16T05:23:48.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038213241000000002 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943611+0000", 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8668999999999999e-05 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943649+0000", 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7894e-05 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7362 2.3 2:c018a787:::smithi20457400-18:head [write 2959167~284815] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 131.284416292, 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035905712999999999, 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7362 2024-09-16T05:23:48.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907870+0000", 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4200000000000005e-07 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908124+0000", 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025446300000000002 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908135+0000", 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0964999999999999e-05 2024-09-16T05:23:48.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908143+0000", 2024-09-16T05:23:48.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5190000000000003e-06 2024-09-16T05:23:48.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:23:48.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9714e-05 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908239+0000", 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6212999999999999e-05 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908346+0000", 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010734999999999999 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909154+0000", 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080830699999999997 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943689+0000", 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034534978000000001 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943732+0000", 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2558999999999997e-05 2024-09-16T05:23:48.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943775+0000", 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2803000000000003e-05 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7363 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 3243982] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 131.284127993, 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035723752999999997, 2024-09-16T05:23:48.299 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7363 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:23:48.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908158+0000", 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908161+0000", 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1410000000000001e-06 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908167+0000", 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.674e-06 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908173+0000", 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4149999999999998e-06 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908684+0000", 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051148000000000003 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908742+0000", 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7358999999999997e-05 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908839+0000", 2024-09-16T05:23:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6930999999999997e-05 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909158+0000", 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031886399999999997 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943805+0000", 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034647491000000002 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:23:48.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943842+0000", 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7004e-05 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943881+0000", 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8868999999999997e-05 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7364 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:23:48.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 131.284103133, 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035867990000000002, 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7364 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5999999999999999e-07 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908185+0000", 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1600000000000001e-06 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908191+0000", 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6869999999999997e-06 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908196+0000", 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9989999999999999e-06 2024-09-16T05:23:48.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909088+0000", 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089121600000000001 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909143+0000", 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5887000000000002e-05 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943913+0000", 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034769683000000003 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943981+0000", 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.792e-05 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.944050+0000", 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8978000000000003e-05 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.310 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:830 2.3 e799/329 ondisk, result = 0)", 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 79.322020398999996, 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.330002407, 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870266+0000", 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.223e-06 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870277+0000", 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0560999999999999e-05 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.199989+0000", 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32971226399999998 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200004+0000", 2024-09-16T05:23:48.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5366000000000002e-05 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200139+0000", 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013504000000000001 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200150+0000", 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0305999999999999e-05 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200267+0000", 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011764699999999999 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(osd.10.83:28 2.3 2:c018a787:::smithi20457400-18:head [assert-version v549,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e877)", 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:46.862470+0000", 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 1.329814955, 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067878130000000002, 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:48.315 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.862470+0000", 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.862470+0000", 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.862472+0000", 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.658e-06 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.862479+0000", 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1720000000000001e-06 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.862496+0000", 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.702e-05 2024-09-16T05:23:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.862504+0000", 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8550000000000002e-06 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.862561+0000", 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7442000000000003e-05 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.862570+0000", 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7749999999999998e-06 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.864377+0000", 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001806637 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.864473+0000", 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5960999999999998e-05 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:48.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:46.869258+0000", 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047852930000000004 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:48.320 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:23:48.346 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:48.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:48.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:48.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:48.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:48.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:48.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:48.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:48.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:48.706 INFO:teuthology.orchestra.run.smithi032.stdout:2.9 2024-09-16T05:23:48.706 INFO:teuthology.orchestra.run.smithi032.stdout:2.5d 2024-09-16T05:23:48.706 INFO:teuthology.orchestra.run.smithi032.stdout:2.4f 2024-09-16T05:23:48.706 INFO:teuthology.orchestra.run.smithi032.stdout:2.4 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.17 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.5b 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:1.1d 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.10 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.15 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.44 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.23 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.28 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:1.13 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.2 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.45 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.35 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.3c 2024-09-16T05:23:48.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.33 2024-09-16T05:23:48.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.3d 2024-09-16T05:23:48.708 INFO:teuthology.orchestra.run.smithi032.stdout:1.b 2024-09-16T05:23:48.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.36 2024-09-16T05:23:48.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:48.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:48.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:48.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:48.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:48.877 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:23:48.882 INFO:tasks.rados.rados.2.smithi204.stdout:3507: done (7 left) 2024-09-16T05:23:48.883 INFO:tasks.rados.rados.2.smithi204.stdout:3508: read oid 3 snap -1 2024-09-16T05:23:48.883 INFO:tasks.rados.rados.2.smithi204.stdout:3508: expect (ObjNum 1191 snap 365 seq_num 1191) 2024-09-16T05:23:48.883 INFO:tasks.rados.rados.2.smithi204.stdout:3509: read oid 4 snap -1 2024-09-16T05:23:48.883 INFO:tasks.rados.rados.2.smithi204.stdout:3509: expect deleted 2024-09-16T05:23:48.883 INFO:tasks.rados.rados.2.smithi204.stdout:3510: write oid 32 current snap is 372 2024-09-16T05:23:48.883 INFO:tasks.rados.rados.2.smithi204.stdout:3510: seq_num 1222 ranges {538154=570758,1691937=707756} 2024-09-16T05:23:48.889 INFO:tasks.rados.rados.2.smithi204.stdout:3510: writing smithi20457400-32 from 538154 to 1108912 tid 1 2024-09-16T05:23:48.896 INFO:tasks.rados.rados.2.smithi204.stdout:3510: writing smithi20457400-32 from 1691937 to 2399693 tid 2 2024-09-16T05:23:48.896 INFO:tasks.rados.rados.2.smithi204.stdout:3511: read oid 25 snap -1 2024-09-16T05:23:48.896 INFO:tasks.rados.rados.2.smithi204.stdout:3511: expect (ObjNum 1211 snap 366 seq_num 1211) 2024-09-16T05:23:48.896 INFO:tasks.rados.rados.2.smithi204.stdout:3512: read oid 5 snap -1 2024-09-16T05:23:48.896 INFO:tasks.rados.rados.2.smithi204.stdout:3512: expect (ObjNum 1210 snap 366 seq_num 1210) 2024-09-16T05:23:48.897 INFO:tasks.rados.rados.2.smithi204.stdout:3513: delete oid 36 current snap is 372 2024-09-16T05:23:48.897 INFO:tasks.rados.rados.2.smithi204.stdout:3508: expect (ObjNum 1191 snap 365 seq_num 1191) 2024-09-16T05:23:48.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:48.975 INFO:tasks.rados.rados.2.smithi204.stdout:3510: finishing write tid 1 to smithi20457400-32 2024-09-16T05:23:48.975 INFO:tasks.rados.rados.2.smithi204.stdout:3508: done (12 left) 2024-09-16T05:23:48.975 INFO:tasks.rados.rados.2.smithi204.stdout:3509: done (11 left) 2024-09-16T05:23:48.975 INFO:tasks.rados.rados.2.smithi204.stdout:3513: done (10 left) 2024-09-16T05:23:48.975 INFO:tasks.rados.rados.2.smithi204.stdout:3514: rollback oid 1 current snap is 372 2024-09-16T05:23:48.975 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 1 to 370 2024-09-16T05:23:48.976 INFO:tasks.rados.rados.2.smithi204.stdout:3510: finishing write tid 2 to smithi20457400-32 2024-09-16T05:23:48.976 INFO:tasks.rados.rados.2.smithi204.stdout:3510: finishing write tid 3 to smithi20457400-32 2024-09-16T05:23:48.976 INFO:tasks.rados.rados.2.smithi204.stdout:3510: finishing write tid 4 to smithi20457400-32 2024-09-16T05:23:48.976 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1250 (ObjNum 1222 snap 372 seq_num 1222) dirty exists 2024-09-16T05:23:48.976 INFO:tasks.rados.rados.2.smithi204.stdout:3510: left oid 32 (ObjNum 1222 snap 372 seq_num 1222) 2024-09-16T05:23:48.976 INFO:tasks.rados.rados.2.smithi204.stdout:3512: expect (ObjNum 1210 snap 366 seq_num 1210) 2024-09-16T05:23:48.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:23:48.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:48.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:49.000 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:48 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:23:49.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:48 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:23:49.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:48 smithi032 ceph-mon[27388]: osdmap e878: 12 total, 11 up, 12 in 2024-09-16T05:23:49.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:48 smithi032 ceph-mon[27388]: from='client.17400 172.21.15.32:0/2317661604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:49.051 INFO:tasks.rados.rados.2.smithi204.stdout:3511: expect (ObjNum 1211 snap 366 seq_num 1211) 2024-09-16T05:23:49.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:23:49.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:49.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:49.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:48 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:23:49.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:48 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:23:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:48 smithi204 ceph-mon[30933]: osdmap e878: 12 total, 11 up, 12 in 2024-09-16T05:23:49.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:48 smithi204 ceph-mon[30933]: from='client.17400 172.21.15.32:0/2317661604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:49.231 INFO:tasks.rados.rados.2.smithi204.stdout:3514: finishing rollback tid 0 to smithi20457400-1 2024-09-16T05:23:49.231 INFO:tasks.rados.rados.2.smithi204.stdout:3514: finishing rollback tid 1 to smithi20457400-1 2024-09-16T05:23:49.231 INFO:tasks.rados.rados.2.smithi204.stdout:3514: finishing rollback tid 2 to smithi20457400-1 2024-09-16T05:23:49.231 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 1147 (ObjNum 1213 snap 368 seq_num 1213) dirty exists 2024-09-16T05:23:49.231 INFO:tasks.rados.rados.2.smithi204.stdout:3510: done (10 left) 2024-09-16T05:23:49.231 INFO:tasks.rados.rados.2.smithi204.stdout:3511: done (9 left) 2024-09-16T05:23:49.231 INFO:tasks.rados.rados.2.smithi204.stdout:3512: done (8 left) 2024-09-16T05:23:49.231 INFO:tasks.rados.rados.2.smithi204.stdout:3514: done (7 left) 2024-09-16T05:23:49.232 INFO:tasks.rados.rados.2.smithi204.stdout:3515: snap_create 2024-09-16T05:23:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:48 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:23:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:48 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:23:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:48 smithi196 ceph-mon[30802]: osdmap e878: 12 total, 11 up, 12 in 2024-09-16T05:23:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:48 smithi196 ceph-mon[30802]: from='client.17400 172.21.15.32:0/2317661604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:23:49.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:49.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:49.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:49.316 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:49.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:49.383 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:49.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:49.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:49.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:49.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:49.633 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:49.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:49.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:49.757 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:49.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:36.869235+0000", 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 12.897268958, 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 12.897295399000001, 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:49.786 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8703 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869235+0000", 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869235+0000", 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869236+0000", 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7000000000000005e-07 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:49.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869239+0000", 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.208e-06 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869247+0000", 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1829999999999994e-06 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869254+0000", 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0740000000000004e-06 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.872495+0000", 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032404550000000002 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.872573+0000", 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8412999999999996e-05 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8704 2.29 2:96dc17ef:::smithi20457400-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e867)", 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:36.869264+0000", 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 12.897240406, 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 12.89730861, 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for blocked object", 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:23:49.791 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:23:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8704 2024-09-16T05:23:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:23:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:23:49.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869264+0000", 2024-09-16T05:23:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:23:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869264+0000", 2024-09-16T05:23:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:23:49.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869264+0000", 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6400000000000003e-07 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869266+0000", 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.153e-06 2024-09-16T05:23:49.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869273+0000", 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4930000000000003e-06 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869281+0000", 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.605e-06 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.872743+0000", 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034624510000000001 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.872803+0000", 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9923999999999998e-05 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 2 2024-09-16T05:23:49.796 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:23:49.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:49.879 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:49 smithi032 ceph-mon[27388]: pgmap v1479: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 145 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 975 KiB/s wr, 5 op/s; 9/140 objects degraded (6.429%) 2024-09-16T05:23:49.879 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:49 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/140 objects degraded (6.429%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:49.879 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:49 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:49.879 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:49 smithi032 ceph-mon[27388]: from='client.17400 172.21.15.32:0/2317661604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:49.879 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:49 smithi032 ceph-mon[27388]: osdmap e879: 12 total, 11 up, 12 in 2024-09-16T05:23:49.881 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:49.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:49.890 INFO:tasks.rados.rados.2.smithi204.stdout:3515: done (7 left) 2024-09-16T05:23:49.890 INFO:tasks.rados.rados.2.smithi204.stdout:3516: write oid 37 current snap is 373 2024-09-16T05:23:49.890 INFO:tasks.rados.rados.2.smithi204.stdout:3516: seq_num 1224 ranges {571768=437609,1808901=711425} 2024-09-16T05:23:49.896 INFO:tasks.rados.rados.2.smithi204.stdout:3516: writing smithi20457400-37 from 571768 to 1009377 tid 1 2024-09-16T05:23:49.905 INFO:tasks.rados.rados.2.smithi204.stdout:3516: writing smithi20457400-37 from 1808901 to 2520326 tid 2 2024-09-16T05:23:49.905 INFO:tasks.rados.rados.2.smithi204.stdout:3517: delete oid 36 current snap is 373 2024-09-16T05:23:49.905 INFO:tasks.rados.rados.2.smithi204.stdout:3516: finishing write tid 1 to smithi20457400-37 2024-09-16T05:23:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:3517: done (8 left) 2024-09-16T05:23:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:3518: read oid 47 snap -1 2024-09-16T05:23:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:3518: expect (ObjNum 1197 snap 366 seq_num 1197) 2024-09-16T05:23:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:3519: write (excl) oid 13 current snap is 373 2024-09-16T05:23:49.906 INFO:tasks.rados.rados.2.smithi204.stdout:3519: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-09-16T05:23:49.913 INFO:tasks.rados.rados.2.smithi204.stdout:3519: writing smithi20457400-13 from 605382 to 1193489 tid 1 2024-09-16T05:23:49.921 INFO:tasks.rados.rados.2.smithi204.stdout:3519: writing smithi20457400-13 from 1925865 to 2605382 tid 2 2024-09-16T05:23:49.921 INFO:tasks.rados.rados.2.smithi204.stdout:3520: delete oid 1 current snap is 373 2024-09-16T05:23:49.921 INFO:tasks.rados.rados.2.smithi204.stdout:3516: finishing write tid 2 to smithi20457400-37 2024-09-16T05:23:49.921 INFO:tasks.rados.rados.2.smithi204.stdout:3516: finishing write tid 3 to smithi20457400-37 2024-09-16T05:23:49.921 INFO:tasks.rados.rados.2.smithi204.stdout:3516: finishing write tid 4 to smithi20457400-37 2024-09-16T05:23:49.921 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1087 (ObjNum 1224 snap 373 seq_num 1224) dirty exists 2024-09-16T05:23:49.921 INFO:tasks.rados.rados.2.smithi204.stdout:3516: left oid 37 (ObjNum 1224 snap 373 seq_num 1224) 2024-09-16T05:23:49.922 INFO:tasks.rados.rados.2.smithi204.stdout:3518: expect (ObjNum 1197 snap 366 seq_num 1197) 2024-09-16T05:23:49.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:50.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:50.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:50.037 INFO:tasks.rados.rados.2.smithi204.stdout:3516: done (10 left) 2024-09-16T05:23:50.037 INFO:tasks.rados.rados.2.smithi204.stdout:3518: done (9 left) 2024-09-16T05:23:50.037 INFO:tasks.rados.rados.2.smithi204.stdout:3520: done (8 left) 2024-09-16T05:23:50.037 INFO:tasks.rados.rados.2.smithi204.stdout:3521: snap_remove snap 372 2024-09-16T05:23:50.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:50.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:50.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:50.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:49 smithi204 ceph-mon[30933]: pgmap v1479: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 145 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 975 KiB/s wr, 5 op/s; 9/140 objects degraded (6.429%) 2024-09-16T05:23:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:49 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/140 objects degraded (6.429%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:49 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:49 smithi204 ceph-mon[30933]: from='client.17400 172.21.15.32:0/2317661604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:50.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:49 smithi204 ceph-mon[30933]: osdmap e879: 12 total, 11 up, 12 in 2024-09-16T05:23:50.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:49 smithi196 ceph-mon[30802]: pgmap v1479: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 145 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 975 KiB/s wr, 5 op/s; 9/140 objects degraded (6.429%) 2024-09-16T05:23:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:49 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/140 objects degraded (6.429%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:49 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:50.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:49 smithi196 ceph-mon[30802]: from='client.17400 172.21.15.32:0/2317661604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:23:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:49 smithi196 ceph-mon[30802]: osdmap e879: 12 total, 11 up, 12 in 2024-09-16T05:23:50.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:50.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:50.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:50.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:50.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:50.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:23:50.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.1 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 2.36 --file /var/log/ceph/exp.2.36.1 2024-09-16T05:23:50.401 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:50.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:50.504 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:50.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:50.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:50.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:50.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:50.630 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.1/config 2024-09-16T05:23:50.675 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:50.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:50.752 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:50.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:50.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:50.874 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:50.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:50.896 INFO:tasks.rados.rados.2.smithi204.stdout:3521: done (8 left) 2024-09-16T05:23:50.896 INFO:tasks.rados.rados.2.smithi204.stdout:3522: write oid 12 current snap is 373 2024-09-16T05:23:50.896 INFO:tasks.rados.rados.2.smithi204.stdout:3522: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-09-16T05:23:50.903 INFO:tasks.rados.rados.2.smithi204.stdout:3522: writing smithi20457400-12 from 638996 to 1377601 tid 1 2024-09-16T05:23:50.910 INFO:tasks.rados.rados.2.smithi204.stdout:3522: writing smithi20457400-12 from 1926476 to 2638996 tid 2 2024-09-16T05:23:50.911 INFO:tasks.rados.rados.2.smithi204.stdout:3523: snap_remove snap 371 2024-09-16T05:23:50.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:50 smithi032 ceph-mon[27388]: osdmap e880: 12 total, 11 up, 12 in 2024-09-16T05:23:50.917 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:50 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:23:50.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:50.999 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:51.084 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:50 smithi196 ceph-mon[30802]: osdmap e880: 12 total, 11 up, 12 in 2024-09-16T05:23:51.084 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:50 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:23:51.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:51.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:50 smithi204 ceph-mon[30933]: osdmap e880: 12 total, 11 up, 12 in 2024-09-16T05:23:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:50 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:23:51.201 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:23:51.201 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:23:51.201 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:23:51.201 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:23:51.202 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:23:51.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:51.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:51.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:51.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:51.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:51.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:51.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:51.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:51.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:23:51.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:51.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:51.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:51.852 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:23:51.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:3523: done (9 left) 2024-09-16T05:23:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:3524: snap_create 2024-09-16T05:23:51.900 INFO:tasks.rados.rados.2.smithi204.stdout:3522: finishing write tid 1 to smithi20457400-12 2024-09-16T05:23:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:3522: finishing write tid 2 to smithi20457400-12 2024-09-16T05:23:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:3522: finishing write tid 3 to smithi20457400-12 2024-09-16T05:23:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:3522: finishing write tid 4 to smithi20457400-12 2024-09-16T05:23:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1265 (ObjNum 1228 snap 373 seq_num 1228) dirty exists 2024-09-16T05:23:51.901 INFO:tasks.rados.rados.2.smithi204.stdout:3522: left oid 12 (ObjNum 1228 snap 373 seq_num 1228) 2024-09-16T05:23:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:51 smithi032 ceph-mon[27388]: pgmap v1482: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 145 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 319 KiB/s wr, 5 op/s; 9/142 objects degraded (6.338%) 2024-09-16T05:23:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:51 smithi032 ceph-mon[27388]: osdmap e881: 12 total, 11 up, 12 in 2024-09-16T05:23:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:51 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:51 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:51 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:51 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:51.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/144470010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:51.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:51.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:23:51.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:23:51.985 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:51.985 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:51.985 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:51.985 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:51.986 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:51.986 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:51.986 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:51.986 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:51.986 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:51.986 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:51.986 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:51.987 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:51.992 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:51.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:52.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:52.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:52.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:51 smithi204 ceph-mon[30933]: pgmap v1482: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 145 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 319 KiB/s wr, 5 op/s; 9/142 objects degraded (6.338%) 2024-09-16T05:23:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:51 smithi204 ceph-mon[30933]: osdmap e881: 12 total, 11 up, 12 in 2024-09-16T05:23:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:51 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:51 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:51 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:51 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/144470010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:52.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:52.231 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:52.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:51 smithi196 ceph-mon[30802]: pgmap v1482: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 145 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 319 KiB/s wr, 5 op/s; 9/142 objects degraded (6.338%) 2024-09-16T05:23:52.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:51 smithi196 ceph-mon[30802]: osdmap e881: 12 total, 11 up, 12 in 2024-09-16T05:23:52.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:51 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:51 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:51 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:51 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:52.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/144470010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:52.240 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:52.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:52.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:52.365 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:52.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:52.466 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:52.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:52.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:52.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:52.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:52.683 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:52.684 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:52.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:23:52.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:23:52.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:52.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:52.739 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:52.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:52.861 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:52.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:52.904 INFO:tasks.rados.rados.2.smithi204.stdout:3522: done (9 left) 2024-09-16T05:23:52.904 INFO:tasks.rados.rados.2.smithi204.stdout:3524: done (8 left) 2024-09-16T05:23:52.905 INFO:tasks.rados.rados.2.smithi204.stdout:3525: snap_remove snap 370 2024-09-16T05:23:52.956 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:52 smithi032 ceph-mon[27388]: osdmap e882: 12 total, 11 up, 12 in 2024-09-16T05:23:52.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:52.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:52.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:52.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:52 smithi196 ceph-mon[30802]: osdmap e882: 12 total, 11 up, 12 in 2024-09-16T05:23:53.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:53.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:53.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:52 smithi204 ceph-mon[30933]: osdmap e882: 12 total, 11 up, 12 in 2024-09-16T05:23:53.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:53.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:53.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:53.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:53.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:53.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:53.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:23:53.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:53.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:53.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:53.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:53.607 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:53.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:53.659 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:53.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:53.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:53.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:53.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:53.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:53.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:53.898 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:23:53.905 INFO:tasks.rados.rados.2.smithi204.stdout:3525: done (8 left) 2024-09-16T05:23:53.906 INFO:tasks.rados.rados.2.smithi204.stdout:3526: snap_create 2024-09-16T05:23:53.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:53.986 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:53.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:54.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:53 smithi196 ceph-mon[30802]: pgmap v1485: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 7 op/s; 9/150 objects degraded (6.000%) 2024-09-16T05:23:54.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:53 smithi196 ceph-mon[30802]: osdmap e883: 12 total, 11 up, 12 in 2024-09-16T05:23:54.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:23:54.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:23:54.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:53 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:54.083 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3666269389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:54.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:54.107 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:54.107 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:54.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:54.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:53 smithi032 ceph-mon[27388]: pgmap v1485: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 7 op/s; 9/150 objects degraded (6.000%) 2024-09-16T05:23:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:53 smithi032 ceph-mon[27388]: osdmap e883: 12 total, 11 up, 12 in 2024-09-16T05:23:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:23:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:23:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:53 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3666269389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:53 smithi204 ceph-mon[30933]: pgmap v1485: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 7 op/s; 9/150 objects degraded (6.000%) 2024-09-16T05:23:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:53 smithi204 ceph-mon[30933]: osdmap e883: 12 total, 11 up, 12 in 2024-09-16T05:23:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:23:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:23:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:53 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:23:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3666269389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:23:54.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:54.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:54.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:54.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:54.363 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:54.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:54.450 INFO:teuthology.orchestra.run.smithi032.stderr:Exporting 2.36 info 2.36( v 272'60 (0'0,272'60] local-lis/les=717/718 n=0 ec=273/107 lis/c=717/717 les/c/f=718/718/0 sis=717) 2024-09-16T05:23:54.450 INFO:teuthology.orchestra.run.smithi032.stderr:Export successful 2024-09-16T05:23:54.465 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:54.491 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:54.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:54.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:54.615 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:54.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:54.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:23:54.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:23:54.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:23:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:54.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:23:54.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:54.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:54.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:54.914 INFO:tasks.rados.rados.2.smithi204.stdout:3526: done (8 left) 2024-09-16T05:23:54.914 INFO:tasks.rados.rados.2.smithi204.stdout:3527: snap_create 2024-09-16T05:23:54.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:54.977 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:54.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:54 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3666269389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:54 smithi196 ceph-mon[30802]: osdmap e884: 12 total, 11 up, 12 in 2024-09-16T05:23:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:54 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:23:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:54 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:23:54.995 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:54.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:55.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:55.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:55.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:54 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3666269389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:54 smithi032 ceph-mon[27388]: osdmap e884: 12 total, 11 up, 12 in 2024-09-16T05:23:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:54 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:23:55.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:54 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:23:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:54 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:23:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3666269389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:23:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:54 smithi204 ceph-mon[30933]: osdmap e884: 12 total, 11 up, 12 in 2024-09-16T05:23:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:54 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:23:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:54 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:23:55.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:55.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:55.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:55.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:55.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:55.403 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:55.403 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:55.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:55.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:55.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:55.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:55.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:55.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:55.749 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:55.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:55.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.1 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 2.36 2024-09-16T05:23:55.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:55.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:55.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:55.922 INFO:tasks.rados.rados.2.smithi204.stdout:3527: done (8 left) 2024-09-16T05:23:55.922 INFO:tasks.rados.rados.2.smithi204.stdout:3528: read oid 3 snap -1 2024-09-16T05:23:55.922 INFO:tasks.rados.rados.2.smithi204.stdout:3528: expect (ObjNum 1191 snap 365 seq_num 1191) 2024-09-16T05:23:55.923 INFO:tasks.rados.rados.2.smithi204.stdout:3529: copy_from oid 5 from oid 4 current snap is 376 2024-09-16T05:23:55.923 INFO:tasks.rados.rados.2.smithi204.stdout:3530: read oid 15 snap -1 2024-09-16T05:23:55.923 INFO:tasks.rados.rados.2.smithi204.stdout:3530: expect deleted 2024-09-16T05:23:55.923 INFO:tasks.rados.rados.2.smithi204.stdout:3531: snap_create 2024-09-16T05:23:55.923 INFO:tasks.rados.rados.2.smithi204.stdout:3529: finishing copy_from to smithi20457400-5 2024-09-16T05:23:55.923 INFO:tasks.rados.rados.2.smithi204.stdout:3529: got expected ENOENT (src dne) 2024-09-16T05:23:55.923 INFO:tasks.rados.rados.2.smithi204.stdout:3529: finishing copy_from racing read to smithi20457400-5 2024-09-16T05:23:55.926 INFO:tasks.rados.rados.2.smithi204.stdout:3528: expect (ObjNum 1191 snap 365 seq_num 1191) 2024-09-16T05:23:55.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:56.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:56.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:56.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:23:56.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: pgmap v1488: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 153 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 9/154 objects degraded (5.844%) 2024-09-16T05:23:56.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:23:56.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:23:56.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:23:56.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:23:56.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/154 objects degraded (5.844%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:56.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: osdmap e885: 12 total, 11 up, 12 in 2024-09-16T05:23:56.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:23:56.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:23:56.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:23:56.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:55 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:23:56.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:56.107 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.1/config 2024-09-16T05:23:56.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:56.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:56.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: pgmap v1488: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 153 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 9/154 objects degraded (5.844%) 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/154 objects degraded (5.844%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: osdmap e885: 12 total, 11 up, 12 in 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:23:56.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:55 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:23:56.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:56.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: pgmap v1488: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 153 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 9/154 objects degraded (5.844%) 2024-09-16T05:23:56.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/154 objects degraded (5.844%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: osdmap e885: 12 total, 11 up, 12 in 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:23:56.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:55 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:23:56.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:56.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:56.303 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:23:56.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:56.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:56.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:56.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:23:56.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:56.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:23:56.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:23:56.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:56.628 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:56.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:56.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:56.736 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:23:56.736 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:23:56.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:56.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:56.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:56.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:56.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:56.927 INFO:tasks.rados.rados.2.smithi204.stdout:3528: done (11 left) 2024-09-16T05:23:56.927 INFO:tasks.rados.rados.2.smithi204.stdout:3529: done (10 left) 2024-09-16T05:23:56.928 INFO:tasks.rados.rados.2.smithi204.stdout:3530: done (9 left) 2024-09-16T05:23:56.928 INFO:tasks.rados.rados.2.smithi204.stdout:3531: done (8 left) 2024-09-16T05:23:56.928 INFO:tasks.rados.rados.2.smithi204.stdout:3532: snap_remove snap 373 2024-09-16T05:23:56.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:56.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:56 smithi196 ceph-mon[30802]: osdmap e886: 12 total, 11 up, 12 in 2024-09-16T05:23:56.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1300951621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:57.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:57.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:57.041 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:23:57.041 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:23:57.042 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:23:57.063 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:23:57.063 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:23:57.083 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:23:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:23:57.105 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:23:57.105 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:23:57.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:57.126 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:23:57.127 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:23:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:57.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:57.149 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:23:57.149 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:23:57.170 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:23:57.171 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:23:57.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:23:57.197 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:23:57.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:56 smithi032 ceph-mon[27388]: osdmap e886: 12 total, 11 up, 12 in 2024-09-16T05:23:57.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1300951621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:57.215 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:23:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:23:57.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:57.238 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:23:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:23:57.255 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:57.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:57.260 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:23:57.260 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:23:57.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:23:57.358 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:57.378 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:56 smithi204 ceph-mon[30933]: osdmap e886: 12 total, 11 up, 12 in 2024-09-16T05:23:57.378 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1300951621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:23:57.392 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:23:57.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:57.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:57.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:57.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:57.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:57.610 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:57.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:57.649 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:57.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:57.751 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:23:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:57.773 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:57.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:23:57.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:57.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:57.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:57.930 INFO:tasks.rados.rados.2.smithi204.stdout:3532: done (8 left) 2024-09-16T05:23:57.930 INFO:tasks.rados.rados.2.smithi204.stdout:3533: snap_remove snap 375 2024-09-16T05:23:57.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:58.007 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:57 smithi204 ceph-mon[30933]: pgmap v1491: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 152 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 667 KiB/s wr, 6 op/s; 9/150 objects degraded (6.000%) 2024-09-16T05:23:58.007 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:57 smithi204 ceph-mon[30933]: osdmap e887: 12 total, 11 up, 12 in 2024-09-16T05:23:58.007 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:57 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:23:58.007 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:57 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:23:58.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:58.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:58.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:23:58.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:23:58.108 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:23:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:23:58.108 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:23:58.108 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:23:58.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:58.145 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:58.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:58.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:57 smithi032 ceph-mon[27388]: pgmap v1491: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 152 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 667 KiB/s wr, 6 op/s; 9/150 objects degraded (6.000%) 2024-09-16T05:23:58.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:57 smithi032 ceph-mon[27388]: osdmap e887: 12 total, 11 up, 12 in 2024-09-16T05:23:58.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:57 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:23:58.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:57 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:23:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:57 smithi196 ceph-mon[30802]: pgmap v1491: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 152 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 667 KiB/s wr, 6 op/s; 9/150 objects degraded (6.000%) 2024-09-16T05:23:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:57 smithi196 ceph-mon[30802]: osdmap e887: 12 total, 11 up, 12 in 2024-09-16T05:23:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:57 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:23:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:57 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:23:58.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:23:58.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:58.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:23:58.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:58.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:58.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:58.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:23:58.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:58.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:23:58.522 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:23:58.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:58.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:58.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:58.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:23:58.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:58.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:23:58.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:23:58.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:58.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:23:58.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:23:58.938 INFO:tasks.rados.rados.2.smithi204.stdout:3533: done (8 left) 2024-09-16T05:23:58.939 INFO:tasks.rados.rados.2.smithi204.stdout:3534: delete oid 15 current snap is 377 2024-09-16T05:23:58.942 INFO:tasks.rados.rados.2.smithi204.stdout:3534: done (8 left) 2024-09-16T05:23:58.943 INFO:tasks.rados.rados.2.smithi204.stdout:3535: read oid 2 snap -1 2024-09-16T05:23:58.943 INFO:tasks.rados.rados.2.smithi204.stdout:3535: expect (ObjNum 1218 snap 369 seq_num 1218) 2024-09-16T05:23:58.943 INFO:tasks.rados.rados.2.smithi204.stdout:3536: rollback oid 50 current snap is 377 2024-09-16T05:23:58.943 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 358 2024-09-16T05:23:58.943 INFO:tasks.rados.rados.2.smithi204.stdout:3537: read oid 43 snap 358 2024-09-16T05:23:58.943 INFO:tasks.rados.rados.2.smithi204.stdout:3537: expect (ObjNum 1110 snap 336 seq_num 1110) 2024-09-16T05:23:58.944 INFO:tasks.rados.rados.2.smithi204.stdout:3538: write (excl) oid 21 current snap is 377 2024-09-16T05:23:58.944 INFO:tasks.rados.rados.2.smithi204.stdout:3538: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-09-16T05:23:58.949 INFO:tasks.rados.rados.2.smithi204.stdout:3538: writing smithi20457400-21 from 672610 to 1161713 tid 1 2024-09-16T05:23:58.957 INFO:tasks.rados.rados.2.smithi204.stdout:3538: writing smithi20457400-21 from 1643440 to 2365872 tid 2 2024-09-16T05:23:58.957 INFO:tasks.rados.rados.2.smithi204.stdout:3539: rollback oid 25 current snap is 377 2024-09-16T05:23:58.957 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 25 to 376 2024-09-16T05:23:58.957 INFO:tasks.rados.rados.2.smithi204.stdout:3536: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:23:58.958 INFO:tasks.rados.rados.2.smithi204.stdout:3536: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:23:58.958 INFO:tasks.rados.rados.2.smithi204.stdout:3536: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:23:58.958 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1163 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:23:58.958 INFO:tasks.rados.rados.2.smithi204.stdout:3537: expect (ObjNum 1110 snap 336 seq_num 1110) 2024-09-16T05:23:58.961 INFO:teuthology.orchestra.run.smithi032.stdout: marking collection for removal 2024-09-16T05:23:58.961 INFO:teuthology.orchestra.run.smithi032.stdout:setting '_remove' omap key 2024-09-16T05:23:58.961 INFO:teuthology.orchestra.run.smithi032.stdout:finish_remove_pgs 2.36_head removing 2.36 2024-09-16T05:23:58.961 INFO:teuthology.orchestra.run.smithi032.stdout:Remove successful 2024-09-16T05:23:58.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:59.014 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:59.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:59.047 INFO:tasks.rados.rados.2.smithi204.stdout:3538: finishing write tid 1 to smithi20457400-21 2024-09-16T05:23:59.047 INFO:tasks.rados.rados.2.smithi204.stdout:3535: expect (ObjNum 1218 snap 369 seq_num 1218) 2024-09-16T05:23:59.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:59.128 INFO:tasks.rados.rados.2.smithi204.stdout:3539: finishing rollback tid 0 to smithi20457400-25 2024-09-16T05:23:59.128 INFO:tasks.rados.rados.2.smithi204.stdout:3538: finishing write tid 2 to smithi20457400-21 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3539: finishing rollback tid 1 to smithi20457400-25 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3535: done (12 left) 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3536: done (11 left) 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3537: done (10 left) 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3540: snap_create 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3539: finishing rollback tid 2 to smithi20457400-25 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1280 (ObjNum 1211 snap 366 seq_num 1211) dirty exists 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3538: finishing write tid 3 to smithi20457400-21 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3538: finishing write tid 4 to smithi20457400-21 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 1179 (ObjNum 1230 snap 377 seq_num 1230) dirty exists 2024-09-16T05:23:59.129 INFO:tasks.rados.rados.2.smithi204.stdout:3538: left oid 21 (ObjNum 1230 snap 377 seq_num 1230) 2024-09-16T05:23:59.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:59.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:58 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:23:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:58 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:23:59.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:58 smithi204 ceph-mon[30933]: osdmap e888: 12 total, 11 up, 12 in 2024-09-16T05:23:59.184 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:23:59.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:58 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:23:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:58 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:23:59.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:58 smithi196 ceph-mon[30802]: osdmap e888: 12 total, 11 up, 12 in 2024-09-16T05:23:59.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:59.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:59.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:59.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:59.384 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:59.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:58 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:23:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:58 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:23:59.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:58 smithi032 ceph-mon[27388]: osdmap e888: 12 total, 11 up, 12 in 2024-09-16T05:23:59.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:23:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:23:59.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:23:59.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:23:59.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:59.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:23:59.664 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:23:59.664 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:23:59.664 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:23:59.664 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:23:59.665 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:23:59.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:23:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:59.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:23:59.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:23:59.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:23:59.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:23:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:23:59.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:23:59.936 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:23:59.939 INFO:tasks.rados.rados.2.smithi204.stdout:3538: done (10 left) 2024-09-16T05:23:59.939 INFO:tasks.rados.rados.2.smithi204.stdout:3539: done (9 left) 2024-09-16T05:23:59.939 INFO:tasks.rados.rados.2.smithi204.stdout:3540: done (8 left) 2024-09-16T05:23:59.939 INFO:tasks.rados.rados.2.smithi204.stdout:3541: snap_create 2024-09-16T05:23:59.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:00.016 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:00.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:00.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: pgmap v1494: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 0 op/s; 9/146 objects degraded (6.164%) 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: osdmap e889: 12 total, 11 up, 12 in 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3751697685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:24:00.080 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:23:59 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:24:00.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:00.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.1 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.36.1 2024-09-16T05:24:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:00.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:00.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: pgmap v1494: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 0 op/s; 9/146 objects degraded (6.164%) 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: osdmap e889: 12 total, 11 up, 12 in 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3751697685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:24:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:23:59 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:24:00.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:00.269 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:00.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: pgmap v1494: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 0 op/s; 9/146 objects degraded (6.164%) 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: osdmap e889: 12 total, 11 up, 12 in 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3751697685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:24:00.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:23:59 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:24:00.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:24:00.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:00.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:00.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:00.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:00.522 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:00.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:00.611 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:00.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:00.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:00.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:00.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:00.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:00.771 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:00.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:00.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:00.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:00.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:00.910 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.1/config 2024-09-16T05:24:00.931 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:00.955 INFO:tasks.rados.rados.2.smithi204.stdout:3541: done (8 left) 2024-09-16T05:24:00.955 INFO:tasks.rados.rados.2.smithi204.stdout:3542: copy_from oid 15 from oid 1 current snap is 379 2024-09-16T05:24:00.955 INFO:tasks.rados.rados.2.smithi204.stdout:3543: snap_remove snap 374 2024-09-16T05:24:00.982 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:00 smithi204 ceph-mon[30933]: 2.5c scrub starts 2024-09-16T05:24:00.982 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:00 smithi204 ceph-mon[30933]: 2.5c scrub ok 2024-09-16T05:24:00.982 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:00 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:00.982 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3751697685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:00.982 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:00 smithi204 ceph-mon[30933]: osdmap e890: 12 total, 11 up, 12 in 2024-09-16T05:24:00.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:00 smithi204 ceph-mon[30933]: 1.19 deep-scrub starts 2024-09-16T05:24:00.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:00 smithi204 ceph-mon[30933]: 1.19 deep-scrub ok 2024-09-16T05:24:00.983 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:00 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/146 objects degraded (6.164%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:00.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:01.021 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:01.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:01.083 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:24:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:24:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:24:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:24:01.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:839 2.1a e438/291 ondisk, result = 0)", 2024-09-16T05:24:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:24:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 453.35031570500001, 2024-09-16T05:24:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51227693699999999, 2024-09-16T05:24:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.084 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730490+0000", 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730494+0000", 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0810000000000004e-06 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242338+0000", 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51184441999999997 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242353+0000", 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.447e-05 2024-09-16T05:24:01.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242501+0000", 2024-09-16T05:24:01.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014851799999999999 2024-09-16T05:24:01.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242532+0000", 2024-09-16T05:24:01.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0406999999999999e-05 2024-09-16T05:24:01.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242766+0000", 2024-09-16T05:24:01.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000234587 2024-09-16T05:24:01.092 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGRecoveryDelete(2.4c e439,438 2:33100188:::smithi20457400-19:2f,230'190)", 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 452.247404598, 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.584288378, 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.093 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833401+0000", 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833406+0000", 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8940000000000001e-06 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417262+0000", 2024-09-16T05:24:01.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.58385600999999998 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417279+0000", 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6880000000000001e-05 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417458+0000", 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000178751 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417489+0000", 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1192999999999998e-05 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417689+0000", 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019989399999999999 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5460 2.25 2:a5bc1608:::smithi20457400-11:head [setxattr _header (58) in=7b,truncate 2975004] snapc e1=[e1,e0,df,dd,dc,da,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:24:01.097 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.08637828000002, 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69151579299999999, 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5460 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:24:01.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994430+0000", 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2729999999999998e-06 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994437+0000", 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9670000000000004e-06 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994443+0000", 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7780000000000004e-06 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995089+0000", 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064609900000000002 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995143+0000", 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4042000000000002e-05 2024-09-16T05:24:01.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995231+0000", 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7798000000000002e-05 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995551+0000", 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000319847 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:01.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685849+0000", 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.690298254 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685893+0000", 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3368000000000002e-05 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685942+0000", 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9811000000000001e-05 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.103 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5461 2.25 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.08635113499997, 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69165347799999999, 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.104 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5461 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994457+0000", 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.429e-06 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994463+0000", 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994469+0000", 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4380000000000003e-06 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995446+0000", 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00097678900000000004 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995491+0000", 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5153e-05 2024-09-16T05:24:01.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685982+0000", 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69049136499999997 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686042+0000", 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9896999999999999e-05 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686107+0000", 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5270000000000004e-05 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6242 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 913] snapc 103=[103,102,101,100,fe,fd,fc,fb,fa,f9,f7,f6,f2,f0,e7] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 253.267477528, 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47407041599999999, 2024-09-16T05:24:01.110 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6242 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813328+0000", 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9399999999999993e-07 2024-09-16T05:24:01.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813334+0000", 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1619999999999996e-06 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813347+0000", 2024-09-16T05:24:01.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2145e-05 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813354+0000", 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1230000000000002e-06 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813414+0000", 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0489999999999999e-05 2024-09-16T05:24:01.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813421+0000", 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1010000000000003e-06 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816226+0000", 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002804647 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816349+0000", 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012330699999999999 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.281344+0000", 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.464994937 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284684+0000", 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033402309999999999 2024-09-16T05:24:01.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287216+0000", 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002531298 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287319+0000", 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102865 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287398+0000", 2024-09-16T05:24:01.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9116000000000006e-05 2024-09-16T05:24:01.118 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.118 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.118 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6243 2.5 2:a1d8c6d9:::smithi20457400-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 253.26743215100001, 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47072968100000001, 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.119 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6243 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:24:01.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1800000000000005e-07 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813377+0000", 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3699999999999999e-06 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813385+0000", 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5280000000000003e-06 2024-09-16T05:24:01.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813392+0000", 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0600000000000002e-06 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813515+0000", 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012306 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813521+0000", 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.203e-06 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816658+0000", 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003137365 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816689+0000", 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0592000000000001e-05 2024-09-16T05:24:01.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.283968+0000", 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46727931 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284030+0000", 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2316000000000004e-05 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284102+0000", 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2158999999999998e-05 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7270 2.1a 2:5827f14d:::smithi20457400-12:head [sparse-read 0~1919657,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 154.25730149899999, 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.003199682, 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.127 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7270 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823506+0000", 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1099999999999999e-06 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823515+0000", 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6219999999999998e-06 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823531+0000", 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.632e-05 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823544+0000", 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2508e-05 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832369+0000", 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088252939999999992 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832385+0000", 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5534999999999999e-05 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.822672+0000", 2024-09-16T05:24:01.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99028730499999995 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.822833+0000", 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016117400000000001 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.826703+0000", 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038698140000000001 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.132 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7271 2.1a 2:5827f14d:::smithi20457400-12:head [sparse-read 0~1919657] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 154.25724508900001, 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0044004120000001, 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7271 2024-09-16T05:24:01.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6699999999999999e-07 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823563+0000", 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.824e-06 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823569+0000", 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.319e-06 2024-09-16T05:24:01.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823574+0000", 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0180000000000001e-06 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832415+0000", 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088402089999999999 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:24:01.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832423+0000", 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8089999999999999e-06 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827702+0000", 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.995278938 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827810+0000", 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107433 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827960+0000", 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150395 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7272 2.1a 2:5827f14d:::smithi20457400-12:head [read 0~3025491,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 154.25722447999999, 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0059436770000001, 2024-09-16T05:24:01.138 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7272 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823581+0000", 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1800000000000002e-07 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.140 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823600+0000", 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9296999999999998e-05 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823607+0000", 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.063e-06 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823612+0000", 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0699999999999997e-06 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832476+0000", 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088636289999999996 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832485+0000", 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5359999999999998e-06 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827992+0000", 2024-09-16T05:24:01.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99550709199999998 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.828085+0000", 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3086999999999998e-05 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.829524+0000", 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001438585 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.143 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPush(2.29 746/745 [PushOp(2:9662c29c:::smithi20457400-9:120, version: 737'1148, data_included: [2490368~196608,3108095~707292], data_size: 903900, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9662c29c:::smithi20457400-9:120@737'1148, size: 3815387, copy_subset: [(0, 3815387)], clone_subset: {2:9662c29c:::smithi20457400-9:116: []}, snapset: 130=[]:{116: [116], 120: [119], 125: [125], 130: [12f, 12c, 12a, 126]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3815387, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2490368, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 145.091129841, 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86202506199999995, 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989676+0000", 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8999999999999997e-07 2024-09-16T05:24:01.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.990550+0000", 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087471100000000002 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850633+0000", 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86008260599999997 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850658+0000", 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5537999999999998e-05 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850763+0000", 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104546 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850778+0000", 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4603999999999999e-05 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.851700+0000", 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00092256700000000001 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7554 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 768] snapc 13f=[13f,13d,13b,134,133,12c,12a,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 128.861023695, 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010270486000000001, 2024-09-16T05:24:01.148 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7554 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219783+0000", 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1430000000000001e-06 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219795+0000", 2024-09-16T05:24:01.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2174999999999999e-05 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219821+0000", 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5364e-05 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219831+0000", 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0069e-05 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219911+0000", 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0131999999999998e-05 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.220032+0000", 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000121305 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.224076+0000", 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040437390000000002 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.227596+0000", 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035194359999999999 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.229981+0000", 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023851279999999998 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.230018+0000", 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7030999999999998e-05 2024-09-16T05:24:01.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.230052+0000", 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3964000000000002e-05 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7717 2.f 2:f08ec412:::smithi20457400-44:head [copy-from ver 1121] snapc 149=[149,148,147,146,145,144,143,142,141,13b,133,12c,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e778)", 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 112.98665658500001, 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010866597, 2024-09-16T05:24:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7717 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094150+0000", 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4899999999999999e-06 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094158+0000", 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9980000000000003e-06 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094174+0000", 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6291e-05 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094184+0000", 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6369999999999994e-06 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094245+0000", 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1261999999999997e-05 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094313+0000", 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7793999999999995e-05 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9", 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.100828+0000", 2024-09-16T05:24:01.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0065157000000000001 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104619+0000", 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037903139999999999 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104944+0000", 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032508400000000001 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:01.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104979+0000", 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5567e-05 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.105015+0000", 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5460000000000003e-05 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:1707 2.1a e799/764 ondisk, result = 0)", 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:24:01.160 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.218934477999994, 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69806231100000005, 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861871+0000", 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7800000000000001e-07 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861876+0000", 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4889999999999997e-06 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559739+0000", 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69786377399999999 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559746+0000", 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3409999999999999e-06 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559833+0000", 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7454000000000007e-05 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.163 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559840+0000", 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559933+0000", 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2936000000000006e-05 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.164 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8288 2.25 2:a7ee7933:::smithi20457400-20:head [copy-from ver 750] snapc 15a=[15a] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.162738915000006, 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01619603, 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.165 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8288 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918068+0000", 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4499999999999998e-06 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918095+0000", 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6695000000000001e-05 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918117+0000", 2024-09-16T05:24:01.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2120000000000002e-05 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918127+0000", 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0074e-05 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918253+0000", 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125339 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:24:01.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918260+0000", 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9530000000000002e-06 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.921653+0000", 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033933230000000002 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.921758+0000", 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104949 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.927615+0000", 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058567039999999999 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.933643+0000", 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006028233 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934133+0000", 2024-09-16T05:24:01.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00049045800000000004 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934202+0000", 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8817000000000002e-05 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934262+0000", 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9914999999999998e-05 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8289 2.25 2:a7ee7933:::smithi20457400-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.162640030999995, 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012771173, 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.172 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8289 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:24:01.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2500000000000005e-07 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918187+0000", 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1101999999999999e-05 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918198+0000", 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1139e-05 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918203+0000", 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5589999999999996e-06 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918361+0000", 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015787099999999999 2024-09-16T05:24:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918367+0000", 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2389999999999999e-06 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.922539+0000", 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041721240000000001 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:01.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.922594+0000", 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4969000000000002e-05 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930701+0000", 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081063300000000001 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930817+0000", 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116116 2024-09-16T05:24:01.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930936+0000", 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011909900000000001 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8525 2.1a 2:580e4d0e:::smithi20457400-29:head [copy-from ver 785] snapc 169=[169,166,163] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:21.855145+0000", 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 39.225659698000001, 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063705000000000003, 2024-09-16T05:24:01.178 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8525 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855145+0000", 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855145+0000", 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855148+0000", 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4600000000000002e-06 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855157+0000", 2024-09-16T05:24:01.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6199999999999994e-06 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855172+0000", 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4695000000000001e-05 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855181+0000", 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4800000000000007e-06 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855260+0000", 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8174000000000001e-05 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855382+0000", 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012251800000000001 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.861516+0000", 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061335529999999999 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.182 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8526 2.1a 2:580e4d0e:::smithi20457400-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e852)", 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:21.855208+0000", 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 39.225596664999998, 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0065074269999999997, 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.183 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8526 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855208+0000", 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855208+0000", 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.184 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855209+0000", 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7899999999999999e-07 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855234+0000", 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5678e-05 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855244+0000", 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4609999999999997e-06 2024-09-16T05:24:01.185 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855250+0000", 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3840000000000002e-06 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855574+0000", 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032338000000000001 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.855651+0000", 2024-09-16T05:24:01.186 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7322000000000003e-05 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.861579+0000", 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059283460000000001 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.861647+0000", 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7846000000000001e-05 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:21.861716+0000", 2024-09-16T05:24:01.187 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8530999999999998e-05 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8541 2.5 2:a1d8c6d9:::smithi20457400-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e854)", 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:23.850591+0000", 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 37.230214085999997, 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073552019999999999, 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.188 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8541 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.850591+0000", 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.850591+0000", 2024-09-16T05:24:01.189 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.850591+0000", 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6000000000000003e-07 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.850595+0000", 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7110000000000001e-06 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.850614+0000", 2024-09-16T05:24:01.190 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8876e-05 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.850622+0000", 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5120000000000002e-06 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.850690+0000", 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8048999999999998e-05 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.850693+0000", 2024-09-16T05:24:01.191 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1879999999999998e-06 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.855579+0000", 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048866680000000003 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.855621+0000", 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.163e-05 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.192 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.857777+0000", 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002155442 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.857854+0000", 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7257999999999998e-05 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:23.857946+0000", 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2207999999999995e-05 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.193 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8830 2.25 2:a5bc1608:::smithi20457400-11:head [rollback 16a] snapc 175=[16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:46.873386+0000", 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.207418995999999, 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061460899999999999, 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.194 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8830 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873386+0000", 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873386+0000", 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.195 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873386+0000", 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4400000000000001e-07 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873389+0000", 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3099999999999999e-06 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873395+0000", 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.567e-06 2024-09-16T05:24:01.196 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873401+0000", 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4580000000000001e-06 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.874880+0000", 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001479705 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.197 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.874982+0000", 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010131 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.875175+0000", 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000192995 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.878673+0000", 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003498176 2024-09-16T05:24:01.198 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.879442+0000", 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076919800000000002 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.879492+0000", 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9756999999999998e-05 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.199 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.879532+0000", 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0269999999999999e-05 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8831 2.25 2:a5bc1608:::smithi20457400-11:head [append 0~0] snapc 175=[16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:46.873441+0000", 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.207363774999999, 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006603116, 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:01.200 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8831 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873441+0000", 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.201 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873441+0000", 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873442+0000", 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873447+0000", 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.322e-06 2024-09-16T05:24:01.202 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873462+0000", 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4476e-05 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873470+0000", 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3080000000000008e-06 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.876334+0000", 2024-09-16T05:24:01.203 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028632660000000002 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.876435+0000", 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101798 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.876591+0000", 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000156167 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.204 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.879539+0000", 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029476810000000002 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.879934+0000", 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00039438599999999999 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.879991+0000", 2024-09-16T05:24:01.205 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7000000000000003e-05 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.880044+0000", 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3671000000000002e-05 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:01.206 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:24:01.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:01.207 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:00 smithi032 ceph-mon[27388]: 2.5c scrub starts 2024-09-16T05:24:01.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:00 smithi032 ceph-mon[27388]: 2.5c scrub ok 2024-09-16T05:24:01.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:00 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:01.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3751697685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:01.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:00 smithi032 ceph-mon[27388]: osdmap e890: 12 total, 11 up, 12 in 2024-09-16T05:24:01.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:00 smithi032 ceph-mon[27388]: 1.19 deep-scrub starts 2024-09-16T05:24:01.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:00 smithi032 ceph-mon[27388]: 1.19 deep-scrub ok 2024-09-16T05:24:01.208 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:00 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/146 objects degraded (6.164%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:01.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:01.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:01.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:00 smithi196 ceph-mon[30802]: 2.5c scrub starts 2024-09-16T05:24:01.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:00 smithi196 ceph-mon[30802]: 2.5c scrub ok 2024-09-16T05:24:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:00 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3751697685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:00 smithi196 ceph-mon[30802]: osdmap e890: 12 total, 11 up, 12 in 2024-09-16T05:24:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:00 smithi196 ceph-mon[30802]: 1.19 deep-scrub starts 2024-09-16T05:24:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:00 smithi196 ceph-mon[30802]: 1.19 deep-scrub ok 2024-09-16T05:24:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:00 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/146 objects degraded (6.164%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:01.335 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:01.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:01.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:01.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:01.481 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:01.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:01.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:01.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:01.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:01.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:01.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:01.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:01.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:01.857 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:01.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:01.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:24:01.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:01.968 INFO:tasks.rados.rados.2.smithi204.stdout:3543: done (9 left) 2024-09-16T05:24:01.968 INFO:tasks.rados.rados.2.smithi204.stdout:3544: snap_create 2024-09-16T05:24:01.968 INFO:tasks.rados.rados.2.smithi204.stdout:3542: finishing copy_from to smithi20457400-15 2024-09-16T05:24:01.968 INFO:tasks.rados.rados.2.smithi204.stdout:3542: got expected ENOENT (src dne) 2024-09-16T05:24:01.968 INFO:tasks.rados.rados.2.smithi204.stdout:3542: finishing copy_from racing read to smithi20457400-15 2024-09-16T05:24:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:01.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:02.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:02.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:02.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:02.198 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:02.199 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:02.199 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:02.199 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:02.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:02.220 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:02.220 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 2.2e scrub starts 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 2.2e scrub ok 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: pgmap v1497: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2 op/s; 9/146 objects degraded (6.164%) 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 1.1e deep-scrub starts 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 1.1e deep-scrub ok 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: osdmap e891: 12 total, 11 up, 12 in 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 2.3f scrub starts 2024-09-16T05:24:02.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:01 smithi032 ceph-mon[27388]: 2.3f scrub ok 2024-09-16T05:24:02.232 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:02.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:02.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 2.2e scrub starts 2024-09-16T05:24:02.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 2.2e scrub ok 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: pgmap v1497: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2 op/s; 9/146 objects degraded (6.164%) 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 1.1e deep-scrub starts 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 1.1e deep-scrub ok 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: osdmap e891: 12 total, 11 up, 12 in 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 2.3f scrub starts 2024-09-16T05:24:02.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:01 smithi196 ceph-mon[30802]: 2.3f scrub ok 2024-09-16T05:24:02.241 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:02.241 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:02.263 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:02.264 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:02.285 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:02.286 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:02.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:02.308 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 2.2e scrub starts 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 2.2e scrub ok 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: pgmap v1497: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2 op/s; 9/146 objects degraded (6.164%) 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 1.1e deep-scrub starts 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 1.1e deep-scrub ok 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: osdmap e891: 12 total, 11 up, 12 in 2024-09-16T05:24:02.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 2.3f scrub starts 2024-09-16T05:24:02.316 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:01 smithi204 ceph-mon[30933]: 2.3f scrub ok 2024-09-16T05:24:02.329 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:02.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:02.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:02.351 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:02.351 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:02.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:02.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:02.372 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:02.372 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:02.394 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:02.395 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:02.417 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:02.418 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:02.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:02.487 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:02.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:02.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:02.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:02.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:02.650 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:02.650 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:02.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:02.742 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:02.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:02.841 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:24:02.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:02.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:02.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:02.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:24:02.970 INFO:tasks.rados.rados.2.smithi204.stdout:3542: done (9 left) 2024-09-16T05:24:02.970 INFO:tasks.rados.rados.2.smithi204.stdout:3544: done (8 left) 2024-09-16T05:24:02.970 INFO:tasks.rados.rados.2.smithi204.stdout:3545: write (excl) oid 22 current snap is 380 2024-09-16T05:24:02.970 INFO:tasks.rados.rados.2.smithi204.stdout:3545: seq_num 1231 ranges {689417=564352,1701922=466090} 2024-09-16T05:24:02.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:02.977 INFO:tasks.rados.rados.2.smithi204.stdout:3545: writing smithi20457400-22 from 689417 to 1253769 tid 1 2024-09-16T05:24:02.982 INFO:tasks.rados.rados.2.smithi204.stdout:3545: writing smithi20457400-22 from 1701922 to 2168012 tid 2 2024-09-16T05:24:02.982 INFO:tasks.rados.rados.2.smithi204.stdout:3546: read oid 15 snap -1 2024-09-16T05:24:02.982 INFO:tasks.rados.rados.2.smithi204.stdout:3546: expect deleted 2024-09-16T05:24:02.983 INFO:tasks.rados.rados.2.smithi204.stdout:3547: rollback oid 29 current snap is 380 2024-09-16T05:24:02.983 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 377 2024-09-16T05:24:02.983 INFO:tasks.rados.rados.2.smithi204.stdout:3548: read oid 3 snap -1 2024-09-16T05:24:02.983 INFO:tasks.rados.rados.2.smithi204.stdout:3548: expect (ObjNum 1191 snap 365 seq_num 1191) 2024-09-16T05:24:02.983 INFO:tasks.rados.rados.2.smithi204.stdout:3549: write oid 27 current snap is 380 2024-09-16T05:24:02.983 INFO:tasks.rados.rados.2.smithi204.stdout:3549: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-09-16T05:24:02.990 INFO:tasks.rados.rados.2.smithi204.stdout:3549: writing smithi20457400-27 from 706224 to 1462178 tid 1 2024-09-16T05:24:02.993 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:02.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:02.997 INFO:tasks.rados.rados.2.smithi204.stdout:3549: writing smithi20457400-27 from 2160404 to 2706224 tid 2 2024-09-16T05:24:02.998 INFO:tasks.rados.rados.2.smithi204.stdout:3550: copy_from oid 18 from oid 32 current snap is 380 2024-09-16T05:24:02.998 INFO:tasks.rados.rados.2.smithi204.stdout:3548: expect (ObjNum 1191 snap 365 seq_num 1191) 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3546: done (13 left) 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3548: done (12 left) 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3551: snap_create 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3547: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3547: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3547: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1273 (ObjNum 1208 snap 366 seq_num 1208) dirty exists 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3549: finishing write tid 1 to smithi20457400-27 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3549: finishing write tid 2 to smithi20457400-27 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3549: finishing write tid 3 to smithi20457400-27 2024-09-16T05:24:03.079 INFO:tasks.rados.rados.2.smithi204.stdout:3549: finishing write tid 4 to smithi20457400-27 2024-09-16T05:24:03.080 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 1269 (ObjNum 1232 snap 380 seq_num 1232) dirty exists 2024-09-16T05:24:03.080 INFO:tasks.rados.rados.2.smithi204.stdout:3549: left oid 27 (ObjNum 1232 snap 380 seq_num 1232) 2024-09-16T05:24:03.080 INFO:tasks.rados.rados.2.smithi204.stdout:3550: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:24:03.080 INFO:tasks.rados.rados.2.smithi204.stdout:3550: finishing copy_from to smithi20457400-18 2024-09-16T05:24:03.080 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 1250 (ObjNum 1222 snap 372 seq_num 1222) dirty exists 2024-09-16T05:24:03.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:03.117 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:03.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:02 smithi032 ceph-mon[27388]: 2.59 scrub starts 2024-09-16T05:24:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:02 smithi032 ceph-mon[27388]: 2.59 scrub ok 2024-09-16T05:24:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:02 smithi032 ceph-mon[27388]: osdmap e892: 12 total, 11 up, 12 in 2024-09-16T05:24:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3673374337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:02 smithi032 ceph-mon[27388]: 1.1b scrub starts 2024-09-16T05:24:03.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:02 smithi032 ceph-mon[27388]: 1.1b scrub ok 2024-09-16T05:24:03.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:03.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:02 smithi196 ceph-mon[30802]: 2.59 scrub starts 2024-09-16T05:24:03.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:02 smithi196 ceph-mon[30802]: 2.59 scrub ok 2024-09-16T05:24:03.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:02 smithi196 ceph-mon[30802]: osdmap e892: 12 total, 11 up, 12 in 2024-09-16T05:24:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3673374337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:02 smithi196 ceph-mon[30802]: 1.1b scrub starts 2024-09-16T05:24:03.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:02 smithi196 ceph-mon[30802]: 1.1b scrub ok 2024-09-16T05:24:03.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:03.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:03.271 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:02 smithi204 ceph-mon[30933]: 2.59 scrub starts 2024-09-16T05:24:03.272 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:02 smithi204 ceph-mon[30933]: 2.59 scrub ok 2024-09-16T05:24:03.272 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:02 smithi204 ceph-mon[30933]: osdmap e892: 12 total, 11 up, 12 in 2024-09-16T05:24:03.272 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3673374337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:03.272 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:02 smithi204 ceph-mon[30933]: 1.1b scrub starts 2024-09-16T05:24:03.272 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:02 smithi204 ceph-mon[30933]: 1.1b scrub ok 2024-09-16T05:24:03.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:24:03.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:03.369 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:03.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:03.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:03.496 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:03.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:03.538 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:03.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:03.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:03.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:03.660 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:03.673 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:03.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:03.746 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:03.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:03.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:03.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:03.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:03.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:04.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:04.008 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:04.008 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:04.029 INFO:tasks.rados.rados.2.smithi204.stdout:3547: done (12 left) 2024-09-16T05:24:04.029 INFO:tasks.rados.rados.2.smithi204.stdout:3549: done (11 left) 2024-09-16T05:24:04.029 INFO:tasks.rados.rados.2.smithi204.stdout:3550: done (10 left) 2024-09-16T05:24:04.029 INFO:tasks.rados.rados.2.smithi204.stdout:3551: done (9 left) 2024-09-16T05:24:04.029 INFO:tasks.rados.rados.2.smithi204.stdout:3552: copy_from oid 50 from oid 46 current snap is 381 2024-09-16T05:24:04.030 INFO:tasks.rados.rados.2.smithi204.stdout:3553: delete oid 28 current snap is 381 2024-09-16T05:24:04.036 INFO:tasks.rados.rados.2.smithi204.stdout:3553: done (10 left) 2024-09-16T05:24:04.036 INFO:tasks.rados.rados.2.smithi204.stdout:3554: write (excl) oid 48 current snap is 381 2024-09-16T05:24:04.036 INFO:tasks.rados.rados.2.smithi204.stdout:3554: seq_num 1234 ranges {739838=506452,1877368=613417} 2024-09-16T05:24:04.043 INFO:tasks.rados.rados.2.smithi204.stdout:3554: writing smithi20457400-48 from 739838 to 1246290 tid 1 2024-09-16T05:24:04.050 INFO:tasks.rados.rados.2.smithi204.stdout:3554: writing smithi20457400-48 from 1877368 to 2490785 tid 2 2024-09-16T05:24:04.050 INFO:tasks.rados.rados.2.smithi204.stdout:3555: snap_create 2024-09-16T05:24:04.050 INFO:tasks.rados.rados.2.smithi204.stdout:3552: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:24:04.050 INFO:tasks.rados.rados.2.smithi204.stdout:3554: finishing write tid 1 to smithi20457400-48 2024-09-16T05:24:04.050 INFO:tasks.rados.rados.2.smithi204.stdout:3552: finishing copy_from to smithi20457400-50 2024-09-16T05:24:04.050 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1035 (ObjNum 1177 snap 360 seq_num 1177) dirty exists 2024-09-16T05:24:04.052 INFO:tasks.rados.rados.2.smithi204.stdout:3554: finishing write tid 2 to smithi20457400-48 2024-09-16T05:24:04.052 INFO:tasks.rados.rados.2.smithi204.stdout:3554: finishing write tid 3 to smithi20457400-48 2024-09-16T05:24:04.053 INFO:tasks.rados.rados.2.smithi204.stdout:3554: finishing write tid 4 to smithi20457400-48 2024-09-16T05:24:04.053 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 786 (ObjNum 1234 snap 381 seq_num 1234) dirty exists 2024-09-16T05:24:04.053 INFO:tasks.rados.rados.2.smithi204.stdout:3554: left oid 48 (ObjNum 1234 snap 381 seq_num 1234) 2024-09-16T05:24:04.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:04.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:04.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:03 smithi204 ceph-mon[30933]: 2.2a scrub starts 2024-09-16T05:24:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:03 smithi204 ceph-mon[30933]: 2.2a scrub ok 2024-09-16T05:24:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:03 smithi204 ceph-mon[30933]: pgmap v1500: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 0 B/s wr, 4 op/s; 9/148 objects degraded (6.081%) 2024-09-16T05:24:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:03 smithi204 ceph-mon[30933]: osdmap e893: 12 total, 11 up, 12 in 2024-09-16T05:24:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:03 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:24:04.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:03 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:24:04.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:03 smithi032 ceph-mon[27388]: 2.2a scrub starts 2024-09-16T05:24:04.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:03 smithi032 ceph-mon[27388]: 2.2a scrub ok 2024-09-16T05:24:04.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:03 smithi032 ceph-mon[27388]: pgmap v1500: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 0 B/s wr, 4 op/s; 9/148 objects degraded (6.081%) 2024-09-16T05:24:04.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:03 smithi032 ceph-mon[27388]: osdmap e893: 12 total, 11 up, 12 in 2024-09-16T05:24:04.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:03 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:24:04.199 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:03 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:24:04.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:04.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:03 smithi196 ceph-mon[30802]: 2.2a scrub starts 2024-09-16T05:24:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:03 smithi196 ceph-mon[30802]: 2.2a scrub ok 2024-09-16T05:24:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:03 smithi196 ceph-mon[30802]: pgmap v1500: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 151 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 0 B/s wr, 4 op/s; 9/148 objects degraded (6.081%) 2024-09-16T05:24:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:03 smithi196 ceph-mon[30802]: osdmap e893: 12 total, 11 up, 12 in 2024-09-16T05:24:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:03 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:24:04.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:03 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:24:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:04.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:04.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:04.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:04.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:04.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:04.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:04.502 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:04.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:04.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:04.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:24:04.626 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:04.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:04.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:04.750 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:04.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:04.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:04.863 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:04.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:04.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:04.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:05.003 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:05.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:05.031 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:24:05.034 INFO:tasks.rados.rados.2.smithi204.stdout:3552: done (11 left) 2024-09-16T05:24:05.034 INFO:tasks.rados.rados.2.smithi204.stdout:3554: done (10 left) 2024-09-16T05:24:05.034 INFO:tasks.rados.rados.2.smithi204.stdout:3555: done (9 left) 2024-09-16T05:24:05.034 INFO:tasks.rados.rados.2.smithi204.stdout:3556: snap_create 2024-09-16T05:24:05.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:05.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:05.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:05.141 INFO:teuthology.orchestra.run.smithi032.stdout:get_pg_num_history pg_num_history pg_num_history(e866 pg_nums {1={15=1,104=11,401=21,685=31},2={107=16,183=26,209=36,247=46,273=56,291=66,320=76,435=86,737=96}} deleted_pools ) 2024-09-16T05:24:05.142 INFO:teuthology.orchestra.run.smithi032.stdout:Importing pgid 2.36 2024-09-16T05:24:05.142 INFO:teuthology.orchestra.run.smithi032.stdout:write_pg epoch 866 info 2.36( v 272'60 (0'0,272'60] local-lis/les=717/718 n=0 ec=273/107 lis/c=717/717 les/c/f=718/718/0 sis=717) 2024-09-16T05:24:05.142 INFO:teuthology.orchestra.run.smithi032.stdout:Import successful 2024-09-16T05:24:05.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:05.238 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:05 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:24:05.238 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:05 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:24:05.238 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:05 smithi204 ceph-mon[30933]: osdmap e894: 12 total, 11 up, 12 in 2024-09-16T05:24:05.238 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:05 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:24:05.238 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:05 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:24:05.238 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1671193477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:05.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:05.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:05.324 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:05 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:24:05.324 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:05 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:24:05.324 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:05 smithi032 ceph-mon[27388]: osdmap e894: 12 total, 11 up, 12 in 2024-09-16T05:24:05.324 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:05 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:24:05.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:05 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:24:05.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1671193477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:05.341 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:05.342 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:05.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:05.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:05.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:05.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:05.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:05 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:24:05.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:05 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:24:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:05 smithi196 ceph-mon[30802]: osdmap e894: 12 total, 11 up, 12 in 2024-09-16T05:24:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:05 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:24:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:05 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:24:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1671193477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:05.504 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:05.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:05.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:05.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:05.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:24:05.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:05.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:05.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:05.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:05.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:05.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:05.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:05.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:06.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:06.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:06.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:24:06.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:06 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:24:06.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:06 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:24:06.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:06 smithi204 ceph-mon[30933]: pgmap v1503: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 153 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 296 KiB/s wr, 9 op/s; 9/152 objects degraded (5.921%) 2024-09-16T05:24:06.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:06 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:06.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1671193477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:06.037 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:06 smithi204 ceph-mon[30933]: osdmap e895: 12 total, 11 up, 12 in 2024-09-16T05:24:06.038 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:06 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/152 objects degraded (5.921%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:06.050 INFO:tasks.rados.rados.2.smithi204.stdout:3556: done (9 left) 2024-09-16T05:24:06.050 INFO:tasks.rados.rados.2.smithi204.stdout:3557: read oid 6 snap -1 2024-09-16T05:24:06.050 INFO:tasks.rados.rados.2.smithi204.stdout:3557: expect deleted 2024-09-16T05:24:06.051 INFO:tasks.rados.rados.2.smithi204.stdout:3558: snap_create 2024-09-16T05:24:06.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:06.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:06.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:06.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:06.260 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:06.260 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:06.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:06.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:06.385 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:06.386 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:06.413 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:06 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:24:06.416 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:06 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:24:06.416 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:06 smithi032 ceph-mon[27388]: pgmap v1503: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 153 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 296 KiB/s wr, 9 op/s; 9/152 objects degraded (5.921%) 2024-09-16T05:24:06.416 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:06 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:06.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1671193477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:06.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:06 smithi032 ceph-mon[27388]: osdmap e895: 12 total, 11 up, 12 in 2024-09-16T05:24:06.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:06 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/152 objects degraded (5.921%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:06.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/exp.2.36.1 2024-09-16T05:24:06.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:06 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:24:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:06 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:24:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:06 smithi196 ceph-mon[30802]: pgmap v1503: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 153 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 296 KiB/s wr, 9 op/s; 9/152 objects degraded (5.921%) 2024-09-16T05:24:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:06 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1671193477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:06 smithi196 ceph-mon[30802]: osdmap e895: 12 total, 11 up, 12 in 2024-09-16T05:24:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:06 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/152 objects degraded (5.921%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:06.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:06.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:06.538 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:06.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:06.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:06.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:06.715 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:06.715 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:06.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:06.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:06.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:06.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:06.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:06.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:06.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:07.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:07.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:3557: done (10 left) 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:3558: done (9 left) 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:3559: read oid 41 snap 379 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:3559: expect (ObjNum 1195 snap 366 seq_num 1195) 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:3560: rollback oid 50 current snap is 384 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 376 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:3561: read oid 35 snap -1 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:3561: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:24:07.057 INFO:tasks.rados.rados.2.smithi204.stdout:3562: snap_remove snap 382 2024-09-16T05:24:07.094 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:07.095 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:07.095 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:07.116 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:07.116 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:07.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:07.136 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:07.136 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:07.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:07.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:07.158 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:07.180 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:07.180 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:07.202 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:07.202 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:07.224 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:07.224 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:07.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:07.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:07.342 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:07.343 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:07.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:07 smithi032 ceph-mon[27388]: 2.18 deep-scrub starts 2024-09-16T05:24:07.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:07 smithi032 ceph-mon[27388]: 2.18 deep-scrub ok 2024-09-16T05:24:07.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:07 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:24:07.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:07 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:24:07.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:07 smithi032 ceph-mon[27388]: osdmap e896: 12 total, 11 up, 12 in 2024-09-16T05:24:07.361 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:07 smithi204 ceph-mon[30933]: 2.18 deep-scrub starts 2024-09-16T05:24:07.361 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:07 smithi204 ceph-mon[30933]: 2.18 deep-scrub ok 2024-09-16T05:24:07.361 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:07 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:24:07.362 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:07 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:24:07.362 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:07 smithi204 ceph-mon[30933]: osdmap e896: 12 total, 11 up, 12 in 2024-09-16T05:24:07.364 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:07.364 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:07.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:07.386 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:07.386 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:07.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:24:07.394 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:07.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:07.407 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:07.407 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:07.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:07 smithi196 ceph-mon[30802]: 2.18 deep-scrub starts 2024-09-16T05:24:07.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:07 smithi196 ceph-mon[30802]: 2.18 deep-scrub ok 2024-09-16T05:24:07.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:07 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:24:07.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:07 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:24:07.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:07 smithi196 ceph-mon[30802]: osdmap e896: 12 total, 11 up, 12 in 2024-09-16T05:24:07.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:07.517 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:07.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:07.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:07.634 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:07.641 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:07.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:07.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:07.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:24:07.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:07.771 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:07.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:07.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:07.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:07.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:07.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:08.016 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:08.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:08.057 INFO:tasks.rados.rados.2.smithi204.stdout:3562: done (12 left) 2024-09-16T05:24:08.057 INFO:tasks.rados.rados.2.smithi204.stdout:3563: snap_remove snap 377 2024-09-16T05:24:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:08.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:08.103 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:24:08.104 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:24:08.104 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:24:08.104 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:24:08.104 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:24:08.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:08.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:08.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:08 smithi196 ceph-mon[30802]: pgmap v1506: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 161 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.1 MiB/s wr, 9 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:08 smithi196 ceph-mon[30802]: osdmap e897: 12 total, 11 up, 12 in 2024-09-16T05:24:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3182955789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:08.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:07 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:07.928+0000 7f6af1ad6640 -1 osd.6 897 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:08.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:08.268 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:08.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:08.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:08.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:08 smithi032 ceph-mon[27388]: pgmap v1506: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 161 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.1 MiB/s wr, 9 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:08.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:08 smithi032 ceph-mon[27388]: osdmap e897: 12 total, 11 up, 12 in 2024-09-16T05:24:08.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3182955789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:08.395 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:08.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:08.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:08 smithi204 ceph-mon[30933]: pgmap v1506: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 161 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.1 MiB/s wr, 9 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:08 smithi204 ceph-mon[30933]: osdmap e897: 12 total, 11 up, 12 in 2024-09-16T05:24:08.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3182955789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:08.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:08.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:08.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:08.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:08.648 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:08.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:08.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:24:08.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:08.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:08.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:08.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:08.901 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:08.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:08.969 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:09.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:09.029 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:09.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:09.062 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:24:09.066 INFO:tasks.rados.rados.2.smithi204.stdout:3563: done (12 left) 2024-09-16T05:24:09.067 INFO:tasks.rados.rados.2.smithi204.stdout:3564: read oid 44 snap -1 2024-09-16T05:24:09.067 INFO:tasks.rados.rados.2.smithi204.stdout:3564: expect (ObjNum 1145 snap 342 seq_num 1145) 2024-09-16T05:24:09.067 INFO:tasks.rados.rados.2.smithi204.stdout:3565: delete oid 8 current snap is 384 2024-09-16T05:24:09.067 INFO:tasks.rados.rados.2.smithi204.stdout:3561: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:24:09.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:09.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:09.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:09.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:09 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:09 smithi196 ceph-mon[30802]: osdmap e898: 12 total, 11 up, 12 in 2024-09-16T05:24:09.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2269563980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:09.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:08 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:08.902+0000 7f6af1ad6640 -1 osd.6 898 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:09.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:09.273 INFO:tasks.rados.rados.2.smithi204.stdout:3560: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:24:09.273 INFO:tasks.rados.rados.2.smithi204.stdout:3560: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:24:09.273 INFO:tasks.rados.rados.2.smithi204.stdout:3560: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:24:09.273 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1166 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:24:09.273 INFO:tasks.rados.rados.2.smithi204.stdout:3559: expect (ObjNum 1195 snap 366 seq_num 1195) 2024-09-16T05:24:09.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:09.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:09.342 INFO:tasks.rados.rados.2.smithi204.stdout:3564: expect (ObjNum 1145 snap 342 seq_num 1145) 2024-09-16T05:24:09.365 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:09 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:09.365 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:09 smithi204 ceph-mon[30933]: osdmap e898: 12 total, 11 up, 12 in 2024-09-16T05:24:09.365 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2269563980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:09.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:09.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:09 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:09.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:09 smithi032 ceph-mon[27388]: osdmap e898: 12 total, 11 up, 12 in 2024-09-16T05:24:09.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2269563980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:09.404 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:09.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:09.482 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:24:09.482 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:24:09.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:24:09.482 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2559 2.1d 2:ba754fcb:::smithi20457400-30:head [copy-from ver 324] snapc 6d=[6c,6b,6a,69,67,66,64,63,62,60,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 564.50872893999997, 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.74015230600000004, 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.483 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2559 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956645+0000", 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956647+0000", 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2919999999999998e-06 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956655+0000", 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.729e-06 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956672+0000", 2024-09-16T05:24:09.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6775000000000001e-05 2024-09-16T05:24:09.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956685+0000", 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2832e-05 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956782+0000", 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6862999999999997e-05 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956790+0000", 2024-09-16T05:24:09.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4020000000000002e-06 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958620+0000", 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001830415 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958718+0000", 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7708999999999996e-05 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.688352+0000", 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.72963416800000003 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.695488+0000", 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071356659999999997 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696731+0000", 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00124332 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696767+0000", 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6171000000000003e-05 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.696797+0000", 2024-09-16T05:24:09.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9964e-05 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:2560 2.1d 2:ba754fcb:::smithi20457400-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e335)", 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 564.50866306, 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73643418699999996, 2024-09-16T05:24:09.492 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2560 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956711+0000", 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956712+0000", 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.82e-07 2024-09-16T05:24:09.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956714+0000", 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6419999999999999e-06 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956737+0000", 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2668999999999999e-05 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956744+0000", 2024-09-16T05:24:09.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.289e-06 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956847+0000", 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102729 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:09.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.956855+0000", 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4009999999999997e-06 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958884+0000", 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020286409999999999 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:44.958928+0000", 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4458e-05 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693004+0000", 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73407526999999995 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693071+0000", 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7176999999999997e-05 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:14:45.693145+0000", 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4328999999999999e-05 2024-09-16T05:24:09.498 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.0.0:67 2.31 e356/279 2:8c1ba486:::smithi20457400-2:75 v 356'145, mlcod=350'141)", 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 543.75947244500003, 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.929470409, 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705902+0000", 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4499999999999998e-07 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:05.705907+0000", 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8799999999999999e-06 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634756+0000", 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.92884895999999995 2024-09-16T05:24:09.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634764+0000", 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9589999999999995e-06 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634827+0000", 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3280000000000004e-05 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.634853+0000", 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6231999999999999e-05 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635342+0000", 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048899100000000004 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:15:06.635372+0000", 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9762000000000001e-05 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3597 2.4c 2.118008cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 462.719256465, 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98230618599999997, 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.504 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3597 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746119+0000", 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9999999999999995e-07 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746124+0000", 2024-09-16T05:24:09.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3349999999999997e-06 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746136+0000", 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2339e-05 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746143+0000", 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3940000000000001e-06 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746201+0000", 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8529000000000002e-05 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746206+0000", 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0250000000000002e-06 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728413+0000", 2024-09-16T05:24:09.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98220633700000004 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728424+0000", 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1226999999999999e-05 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.509 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3598 2.4c 2.118008cc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 462.71921490099999, 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.982592205, 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3598 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746160+0000", 2024-09-16T05:24:09.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6899999999999997e-07 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746162+0000", 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.323e-06 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746170+0000", 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4339999999999992e-06 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746176+0000", 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746227+0000", 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1848999999999998e-05 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746232+0000", 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5619999999999997e-06 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728741+0000", 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98250897100000001 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728751+0000", 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.10.0:319 2.4a e458/325 ondisk, result = 0)", 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 440.648902373, 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.733792216, 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:24:09.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816473+0000", 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.206e-06 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816480+0000", 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4789999999999999e-06 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550063+0000", 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73358232899999998 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550069+0000", 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3489999999999997e-06 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550155+0000", 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5490999999999999e-05 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550163+0000", 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1859999999999992e-06 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550264+0000", 2024-09-16T05:24:09.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010117599999999999 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3966 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 645534~783707] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 431.57637561000001, 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.228671191, 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.519 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3966 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889000+0000", 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.745e-06 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889398+0000", 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000397644 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889415+0000", 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7368e-05 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889440+0000", 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4358999999999998e-05 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889496+0000", 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6168999999999997e-05 2024-09-16T05:24:09.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889500+0000", 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0579999999999999e-06 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107487+0000", 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21798717200000001 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107565+0000", 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7715999999999999e-05 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107734+0000", 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016950099999999999 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111344+0000", 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036100160000000002 2024-09-16T05:24:09.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117576+0000", 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062321929999999996 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117628+0000", 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.189e-05 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117670+0000", 2024-09-16T05:24:09.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1359999999999997e-05 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3967 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 2103939~638517] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 431.56945058000002, 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.221880365, 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3967 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895924+0000", 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.85e-07 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896200+0000", 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027563199999999999 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896212+0000", 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1671999999999999e-05 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896217+0000", 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3970000000000002e-06 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896274+0000", 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6740000000000002e-05 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896280+0000", 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109156+0000", 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.212875278 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109242+0000", 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6163999999999997e-05 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109378+0000", 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013637400000000001 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111350+0000", 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019716429999999999 2024-09-16T05:24:09.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117716+0000", 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063661830000000001 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117776+0000", 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9753999999999997e-05 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117804+0000", 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.817e-05 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3970 2.1d 2:ba7e58cf:::smithi20457400-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 431.56552293499999, 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218352989, 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3970 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899852+0000", 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8299999999999997e-07 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899854+0000", 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.463e-06 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899861+0000", 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0079999999999996e-06 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899866+0000", 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899949+0000", 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2713000000000006e-05 2024-09-16T05:24:09.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899954+0000", 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7400000000000004e-06 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111261+0000", 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.211307514 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111332+0000", 2024-09-16T05:24:09.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1198999999999994e-05 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118056+0000", 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067238530000000001 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118121+0000", 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4770000000000005e-05 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118204+0000", 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3051000000000002e-05 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:4971 2.1d 2:ba7e58cf:::smithi20457400-26:head [stat,delete] snapc cc=[cb,ca,c9,c7,c5,c4,c3,c2,c1] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 366.26694149600002, 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.70425506000000004, 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.539 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4971 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198434+0000", 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8300000000000001e-06 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198443+0000", 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-16T05:24:09.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198460+0000", 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7229999999999999e-05 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198467+0000", 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1409999999999999e-06 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198531+0000", 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3371e-05 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198539+0000", 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8930000000000005e-06 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891869+0000", 2024-09-16T05:24:09.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69333044700000002 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891964+0000", 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4990999999999999e-05 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892092+0000", 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012792599999999999 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892842+0000", 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075013 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902619+0000", 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0097773840000000001 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902658+0000", 2024-09-16T05:24:09.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8167999999999998e-05 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902688+0000", 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9991000000000001e-05 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5379 2.1c 2:3a3554ba:::smithi20457400-48:head [setxattr _header (58) in=7b,truncate 2823741] snapc e0=[e0,df,dd,dc,db,da,d9,d8,d7,d4,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.67060733099999, 2024-09-16T05:24:09.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46119776499999998, 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5379 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9500000000000002e-07 2024-09-16T05:24:09.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794770+0000", 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9890000000000001e-06 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794783+0000", 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2281e-05 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794788+0000", 2024-09-16T05:24:09.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.011e-06 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795397+0000", 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060981000000000002 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795462+0000", 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4869000000000005e-05 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795576+0000", 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001141 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795836+0000", 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026010999999999997 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255870+0000", 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46003327399999999 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255915+0000", 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5064000000000001e-05 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255964+0000", 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9662e-05 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.552 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5380 2.1c 2:3a3554ba:::smithi20457400-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.67057277700002, 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46133110599999999, 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5380 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794802+0000", 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7599999999999998e-07 2024-09-16T05:24:09.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794804+0000", 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0590000000000001e-06 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794810+0000", 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794815+0000", 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6700000000000002e-06 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795747+0000", 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093155799999999997 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795821+0000", 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3975999999999998e-05 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256004+0000", 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46018367199999999 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256065+0000", 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0299e-05 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256132+0000", 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7776999999999998e-05 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:5379 2.1c e571/532 ondisk, result = 0)", 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 328.66853909100001, 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45914334200000001, 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0800000000000001e-07 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:09.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796837+0000", 2024-09-16T05:24:09.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9850000000000001e-06 2024-09-16T05:24:09.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255760+0000", 2024-09-16T05:24:09.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45892265399999999 2024-09-16T05:24:09.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255769+0000", 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3449999999999995e-06 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255857+0000", 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7547999999999996e-05 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255865+0000", 2024-09-16T05:24:09.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6810000000000004e-06 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255978+0000", 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112821 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5901 2.1d 2:ba754fcb:::smithi20457400-30:head [stat,delete] snapc f3=[f3,f2,f0,ef,ee,ed,ec,e8,e7,e5,e0,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 291.562066856, 2024-09-16T05:24:09.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79482897200000002, 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5901 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:24:09.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903309+0000", 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4699999999999999e-06 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903314+0000", 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903330+0000", 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5770999999999999e-05 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903340+0000", 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0498e-05 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903402+0000", 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1385999999999998e-05 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903513+0000", 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111113 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903707+0000", 2024-09-16T05:24:09.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019455500000000001 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.904477+0000", 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076932900000000002 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:09.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698051+0000", 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79357449800000002 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698095+0000", 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3452999999999999e-05 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698136+0000", 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1482e-05 2024-09-16T05:24:09.569 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.7.0:1084 2.3c e626/625 2:3d4bedc8:::smithi20457400-22:ee v 626'652, mlcod=624'646)", 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 271.991113348, 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45487590700000002, 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.570 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8800000000000002e-07 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474267+0000", 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928275+0000", 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45400790800000002 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928291+0000", 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5852999999999999e-05 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928455+0000", 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164026 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928479+0000", 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4258000000000001e-05 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929099+0000", 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000620117 2024-09-16T05:24:09.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929137+0000", 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7639999999999999e-05 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6377 2.1d 2:ba21f1a9:::smithi20457400-14:head [setxattr _header (58) in=7b,truncate 3059072] snapc 107=[107,106,104,102,101,fb,fa,f9,f7,f2,e7] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 249.578338022, 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46418828400000001, 2024-09-16T05:24:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6377 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887037+0000", 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887040+0000", 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5039999999999998e-06 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887047+0000", 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0330000000000003e-06 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887053+0000", 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1430000000000003e-06 2024-09-16T05:24:09.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888098+0000", 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001044866 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888199+0000", 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101143 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888364+0000", 2024-09-16T05:24:09.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164403 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888873+0000", 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050974699999999996 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351123+0000", 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46224975499999998 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351169+0000", 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6316000000000002e-05 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351224+0000", 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4837999999999999e-05 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6378 2.1d 2:ba21f1a9:::smithi20457400-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:24:09.581 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 249.57831109400001, 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46436558900000002, 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6378 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887064+0000", 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3799999999999997e-07 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887071+0000", 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9670000000000001e-06 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887077+0000", 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9739999999999999e-06 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887081+0000", 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.405e-06 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888760+0000", 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016788319999999999 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888855+0000", 2024-09-16T05:24:09.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5670999999999999e-05 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351265+0000", 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46241011799999998 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351353+0000", 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7416000000000005e-05 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351429+0000", 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5667999999999994e-05 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(1.7,GRANT,e:689,from:6,reservation_nonce:0)", 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 210.00806225700001, 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388634586, 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.587 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457313+0000", 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0020000000000001e-06 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457317+0000", 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.279e-06 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845738+0000", 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388420879 2024-09-16T05:24:09.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845745+0000", 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1849999999999998e-06 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845793+0000", 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8300000000000002e-05 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845814+0000", 2024-09-16T05:24:09.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0353999999999999e-05 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845946+0000", 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013258700000000001 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.6.49:26 2.5d 2.d3f2ae5d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e867)", 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:24:09.591 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 32.592537706000002, 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99584287000000005, 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872838+0000", 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-06 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872846+0000", 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0790000000000001e-06 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872866+0000", 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9624999999999999e-05 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872875+0000", 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4790000000000002e-06 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872926+0000", 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0260000000000003e-05 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872935+0000", 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.346e-06 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:37.868662+0000", 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99572677300000001 2024-09-16T05:24:09.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:37.868679+0000", 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7554000000000002e-05 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8820 2.5e 2:7b642727:::smithi20457400-40:head [copy-from ver 549] snapc 175=[16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.605611019000001, 2024-09-16T05:24:09.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025559208, 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8820 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:24:09.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859765+0000", 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1409999999999999e-06 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859774+0000", 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6430000000000001e-06 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859789+0000", 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5441999999999999e-05 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859798+0000", 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9239999999999996e-06 2024-09-16T05:24:09.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859875+0000", 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6508000000000007e-05 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859883+0000", 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2190000000000004e-06 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.862172+0000", 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022892210000000001 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.862279+0000", 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106606 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873916+0000", 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011637412999999999 2024-09-16T05:24:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.881734+0000", 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078177669999999998 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885232+0000", 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034977849999999998 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885271+0000", 2024-09-16T05:24:09.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9079000000000003e-05 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885322+0000", 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1459999999999999e-05 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:24:09.603 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:24:09.605 INFO:tasks.rados.rados.2.smithi204.stdout:3559: done (13 left) 2024-09-16T05:24:09.605 INFO:tasks.rados.rados.2.smithi204.stdout:3560: done (12 left) 2024-09-16T05:24:09.605 INFO:tasks.rados.rados.2.smithi204.stdout:3561: done (11 left) 2024-09-16T05:24:09.605 INFO:tasks.rados.rados.2.smithi204.stdout:3564: done (10 left) 2024-09-16T05:24:09.605 INFO:tasks.rados.rados.2.smithi204.stdout:3565: done (9 left) 2024-09-16T05:24:09.605 INFO:tasks.rados.rados.2.smithi204.stdout:3566: snap_remove snap 379 2024-09-16T05:24:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:09.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:09.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:09.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:09.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:09.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:09.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:09.854 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:09.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:09.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:09.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:09.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:10.076 INFO:tasks.rados.rados.2.smithi204.stdout:3566: done (9 left) 2024-09-16T05:24:10.076 INFO:tasks.rados.rados.2.smithi204.stdout:3567: write (excl) oid 3 current snap is 384 2024-09-16T05:24:10.076 INFO:tasks.rados.rados.2.smithi204.stdout:3567: seq_num 1236 ranges {773452=656950,1877979=617086} 2024-09-16T05:24:10.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:10.083 INFO:tasks.rados.rados.2.smithi204.stdout:3567: writing smithi20457400-3 from 773452 to 1430402 tid 1 2024-09-16T05:24:10.090 INFO:tasks.rados.rados.2.smithi204.stdout:3567: writing smithi20457400-3 from 1877979 to 2495065 tid 2 2024-09-16T05:24:10.090 INFO:tasks.rados.rados.2.smithi204.stdout:3568: read oid 28 snap 380 2024-09-16T05:24:10.090 INFO:tasks.rados.rados.2.smithi204.stdout:3568: expect deleted 2024-09-16T05:24:10.090 INFO:tasks.rados.rados.2.smithi204.stdout:3569: rollback oid 32 current snap is 384 2024-09-16T05:24:10.090 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 32 to 380 2024-09-16T05:24:10.090 INFO:tasks.rados.rados.2.smithi204.stdout:3567: finishing write tid 1 to smithi20457400-3 2024-09-16T05:24:10.090 INFO:tasks.rados.rados.2.smithi204.stdout:3570: copy_from oid 33 from oid 15 current snap is 384 2024-09-16T05:24:10.090 INFO:tasks.rados.rados.2.smithi204.stdout:3571: snap_remove snap 376 2024-09-16T05:24:10.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:24:10.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:10.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:10.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:10.230 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:10.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:10.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:09 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:09.858+0000 7f6af1ad6640 -1 osd.6 899 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:10 smithi196 ceph-mon[30802]: pgmap v1509: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 163 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 6 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:10 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2269563980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:10.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:10 smithi196 ceph-mon[30802]: osdmap e899: 12 total, 11 up, 12 in 2024-09-16T05:24:10.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:10 smithi204 ceph-mon[30933]: pgmap v1509: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 163 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 6 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:10.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:10 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:10.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2269563980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:10.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:10 smithi204 ceph-mon[30933]: osdmap e899: 12 total, 11 up, 12 in 2024-09-16T05:24:10.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:24:10.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:10.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:10.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:10 smithi032 ceph-mon[27388]: pgmap v1509: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 163 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 6 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:10 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2269563980' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:10 smithi032 ceph-mon[27388]: osdmap e899: 12 total, 11 up, 12 in 2024-09-16T05:24:10.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:10.478 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:10.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:10.543 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:10.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:10.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:10.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:10.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:10.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:10.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:10.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:10.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:10.825 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:10.825 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:10.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:10.846 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:10.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:10.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:10.867 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:10.867 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:10.889 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:10.911 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:10.932 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:10.954 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:10.954 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:10.962 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:10.975 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:10.975 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:10.985 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:10.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:10.996 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:10.996 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:11.017 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:11.017 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:11.038 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:11.038 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:11.074 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:11.075 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:11.081 INFO:tasks.rados.rados.2.smithi204.stdout:3571: done (13 left) 2024-09-16T05:24:11.081 INFO:tasks.rados.rados.2.smithi204.stdout:3572: snap_remove snap 383 2024-09-16T05:24:11.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:11.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:11.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:11.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:11.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:11.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:11.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:10 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:10.893+0000 7f6af1ad6640 -1 osd.6 900 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:11 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:11 smithi196 ceph-mon[30802]: osdmap e900: 12 total, 11 up, 12 in 2024-09-16T05:24:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4139924776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:11 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:11.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:11 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/158 objects degraded (5.696%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:11.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:11.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:11.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:11.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:11 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:11.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:11 smithi204 ceph-mon[30933]: osdmap e900: 12 total, 11 up, 12 in 2024-09-16T05:24:11.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4139924776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:11.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:11 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:11.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:11 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/158 objects degraded (5.696%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:11.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:11 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:11.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:11 smithi032 ceph-mon[27388]: osdmap e900: 12 total, 11 up, 12 in 2024-09-16T05:24:11.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4139924776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:11.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:11 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:11.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:11 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/158 objects degraded (5.696%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:11.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:11.469 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [1] live_osds: [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:24:11.469 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:24:11.469 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T05:24:11.469 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-16T05:24:11.484 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:11.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:11.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:11.610 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:11.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:11.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:11.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:11.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:11.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:24:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:11.772 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:11.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:24:11.805 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:11.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:11.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:11.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:11.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:11.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:11.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:12.053 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:12.082 INFO:tasks.rados.rados.2.smithi204.stdout:3572: done (13 left) 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3573: delete oid 4 current snap is 384 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3567: finishing write tid 2 to smithi20457400-3 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3570: finishing copy_from to smithi20457400-33 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3570: got expected ENOENT (src dne) 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3569: finishing rollback tid 0 to smithi20457400-32 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3570: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3569: finishing rollback tid 1 to smithi20457400-32 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3569: finishing rollback tid 2 to smithi20457400-32 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1258 (ObjNum 1222 snap 372 seq_num 1222) dirty exists 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3567: finishing write tid 3 to smithi20457400-3 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3567: finishing write tid 4 to smithi20457400-3 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1102 (ObjNum 1236 snap 384 seq_num 1236) dirty exists 2024-09-16T05:24:12.083 INFO:tasks.rados.rados.2.smithi204.stdout:3567: left oid 3 (ObjNum 1236 snap 384 seq_num 1236) 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3567: done (13 left) 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3568: done (12 left) 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3569: done (11 left) 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3570: done (10 left) 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3573: done (9 left) 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3574: read oid 8 snap -1 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3574: expect deleted 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3575: read oid 18 snap -1 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3575: expect (ObjNum 1222 snap 372 seq_num 1222) 2024-09-16T05:24:12.087 INFO:tasks.rados.rados.2.smithi204.stdout:3576: write oid 29 current snap is 384 2024-09-16T05:24:12.088 INFO:tasks.rados.rados.2.smithi204.stdout:3576: seq_num 1238 ranges {407066=407448,1594943=620755} 2024-09-16T05:24:12.091 INFO:tasks.rados.rados.2.smithi204.stdout:3576: writing smithi20457400-29 from 407066 to 814514 tid 1 2024-09-16T05:24:12.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:12.100 INFO:tasks.rados.rados.2.smithi204.stdout:3576: writing smithi20457400-29 from 1594943 to 2215698 tid 2 2024-09-16T05:24:12.100 INFO:tasks.rados.rados.2.smithi204.stdout:3577: read oid 48 snap -1 2024-09-16T05:24:12.100 INFO:tasks.rados.rados.2.smithi204.stdout:3577: expect (ObjNum 1234 snap 381 seq_num 1234) 2024-09-16T05:24:12.100 INFO:tasks.rados.rados.2.smithi204.stdout:3578: write (excl) oid 7 current snap is 384 2024-09-16T05:24:12.100 INFO:tasks.rados.rados.2.smithi204.stdout:3576: finishing write tid 1 to smithi20457400-29 2024-09-16T05:24:12.100 INFO:tasks.rados.rados.2.smithi204.stdout:3578: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2024-09-16T05:24:12.105 INFO:tasks.rados.rados.2.smithi204.stdout:3578: writing smithi20457400-7 from 423873 to 906570 tid 1 2024-09-16T05:24:12.112 INFO:tasks.rados.rados.2.smithi204.stdout:3578: writing smithi20457400-7 from 1653425 to 2134191 tid 2 2024-09-16T05:24:12.116 INFO:tasks.rados.rados.2.smithi204.stdout:3578: writing smithi20457400-7 from 2691441 to 2823873 tid 3 2024-09-16T05:24:12.116 INFO:tasks.rados.rados.2.smithi204.stdout:3574: done (13 left) 2024-09-16T05:24:12.116 INFO:tasks.rados.rados.2.smithi204.stdout:3579: rollback oid 50 current snap is 384 2024-09-16T05:24:12.116 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 362 2024-09-16T05:24:12.117 INFO:tasks.rados.rados.2.smithi204.stdout:3575: expect (ObjNum 1222 snap 372 seq_num 1222) 2024-09-16T05:24:12.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:12.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:12.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:12.249 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:12.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:12.286 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:12.303 INFO:tasks.rados.rados.2.smithi204.stdout:3576: finishing write tid 2 to smithi20457400-29 2024-09-16T05:24:12.303 INFO:tasks.rados.rados.2.smithi204.stdout:3576: finishing write tid 3 to smithi20457400-29 2024-09-16T05:24:12.303 INFO:tasks.rados.rados.2.smithi204.stdout:3576: finishing write tid 4 to smithi20457400-29 2024-09-16T05:24:12.303 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1283 (ObjNum 1238 snap 384 seq_num 1238) dirty exists 2024-09-16T05:24:12.304 INFO:tasks.rados.rados.2.smithi204.stdout:3576: left oid 29 (ObjNum 1238 snap 384 seq_num 1238) 2024-09-16T05:24:12.304 INFO:tasks.rados.rados.2.smithi204.stdout:3577: expect (ObjNum 1234 snap 381 seq_num 1234) 2024-09-16T05:24:12.349 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:12 smithi204 ceph-mon[30933]: pgmap v1512: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 163 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 318 KiB/s wr, 5 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:12.349 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:12 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:12.349 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:12 smithi204 ceph-mon[30933]: osdmap e901: 12 total, 11 up, 12 in 2024-09-16T05:24:12.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:12.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:12.412 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:12 smithi032 ceph-mon[27388]: pgmap v1512: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 163 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 318 KiB/s wr, 5 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:12.412 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:12 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:12.412 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:12 smithi032 ceph-mon[27388]: osdmap e901: 12 total, 11 up, 12 in 2024-09-16T05:24:12.437 INFO:tasks.rados.rados.2.smithi204.stdout:3578: finishing write tid 1 to smithi20457400-7 2024-09-16T05:24:12.437 INFO:tasks.rados.rados.2.smithi204.stdout:3578: finishing write tid 2 to smithi20457400-7 2024-09-16T05:24:12.437 INFO:tasks.rados.rados.2.smithi204.stdout:3578: finishing write tid 3 to smithi20457400-7 2024-09-16T05:24:12.437 INFO:tasks.rados.rados.2.smithi204.stdout:3578: finishing write tid 4 to smithi20457400-7 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3578: finishing write tid 5 to smithi20457400-7 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1170 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3578: left oid 7 (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3579: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3579: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3575: done (13 left) 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3576: done (12 left) 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3577: done (11 left) 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3578: done (10 left) 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:3580: rollback oid 31 current snap is 384 2024-09-16T05:24:12.438 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 31 to 378 2024-09-16T05:24:12.439 INFO:tasks.rados.rados.2.smithi204.stdout:3579: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:24:12.439 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1173 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:24:12.439 INFO:tasks.rados.rados.2.smithi204.stdout:3579: done (10 left) 2024-09-16T05:24:12.439 INFO:tasks.rados.rados.2.smithi204.stdout:3581: snap_create 2024-09-16T05:24:12.440 INFO:tasks.rados.rados.2.smithi204.stdout:3580: finishing rollback tid 0 to smithi20457400-31 2024-09-16T05:24:12.440 INFO:tasks.rados.rados.2.smithi204.stdout:3580: finishing rollback tid 1 to smithi20457400-31 2024-09-16T05:24:12.440 INFO:tasks.rados.rados.2.smithi204.stdout:3580: finishing rollback tid 2 to smithi20457400-31 2024-09-16T05:24:12.440 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 1436 (ObjNum 1203 snap 366 seq_num 1203) dirty exists 2024-09-16T05:24:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:24:12.456 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:24:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 902, 2024-09-16T05:24:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "c1b4ea6a-73e8-11ef-bceb-c7b262605968", 2024-09-16T05:24:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2024-09-16T05:02:35.583426+0000", 2024-09-16T05:24:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2024-09-16T05:24:12.076861+0000", 2024-09-16T05:24:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2024-09-16T05:23:36.850362+0000", 2024-09-16T05:24:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2024-09-16T05:21:53.914532+0000", 2024-09-16T05:24:12.456 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T05:24:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5806080, 2024-09-16T05:24:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2024-09-16T05:24:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "nodeep-scrub", 2024-09-16T05:24:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "noscrub", 2024-09-16T05:24:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2024-09-16T05:24:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2024-09-16T05:24:12.457 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 39, 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 2, 2024-09-16T05:24:12.458 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 12, 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "luminous", 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "squid", 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_crimson": false, 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": ".mgr", 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:05:16.445728+0000", 2024-09-16T05:24:12.459 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:24:12.460 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 31, 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 31, 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 31, 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 31, 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 31, 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:24:12.461 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "687", 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "685", 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:24:12.462 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:24:12.463 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:24:12.464 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:24:12.465 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_max": 32, 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 1 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": {} 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.466 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 2.3199999332427979, 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 2.3199999332427979, 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.92000001668930054, 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 2.130000114440918, 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 2.130000114440918, 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:24:12.467 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "unique_pool_0", 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2024-09-16T05:10:35.539280+0000", 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 8193, 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 2, 2024-09-16T05:24:12.468 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2024-09-16T05:24:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_count": 0, 2024-09-16T05:24:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_target": 0, 2024-09-16T05:24:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T05:24:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T05:24:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "is_stretch_pool": false, 2024-09-16T05:24:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2024-09-16T05:24:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "off", 2024-09-16T05:24:12.469 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 96, 2024-09-16T05:24:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 96, 2024-09-16T05:24:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 96, 2024-09-16T05:24:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 96, 2024-09-16T05:24:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 96, 2024-09-16T05:24:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2024-09-16T05:24:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2024-09-16T05:24:12.470 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "902", 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "737", 2024-09-16T05:24:12.471 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 387, 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 902, 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2024-09-16T05:24:12.472 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2024-09-16T05:24:12.473 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2024-09-16T05:24:12.474 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "rados": {} 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "read_balance": { 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "score_type": "Fair distribution", 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "score_acting": 1.7400000095367432, 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "score_stable": 1.7400000095367432, 2024-09-16T05:24:12.475 INFO:teuthology.orchestra.run.smithi032.stdout: "optimal_score": 0.92000001668930054, 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_acting": 1.6000000238418579, 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "raw_score_stable": 1.6000000238418579, 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity_weighted": 0.93785977363586426, 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity": 0.93999999761581421, 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.476 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "f3cfb49b-8223-4649-b211-a2e254248d88", 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 8, 2024-09-16T05:24:12.477 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.479 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2024-09-16T05:24:12.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:24:12.480 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.482 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.483 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2024-09-16T05:24:12.484 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 406159384 2024-09-16T05:24:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6803/406159384", 2024-09-16T05:24:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6805/406159384", 2024-09-16T05:24:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6809/406159384", 2024-09-16T05:24:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6807/406159384", 2024-09-16T05:24:12.485 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "cfb43840-55b6-4d39-a325-2c5416ed97b4", 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 0, 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.486 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 853, 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 867, 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2024-09-16T05:24:12.487 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2024-09-16T05:24:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:24:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.491 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 976643951 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.492 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6811/976643951", 2024-09-16T05:24:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6813/976643951", 2024-09-16T05:24:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6817/976643951", 2024-09-16T05:24:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6815/976643951", 2024-09-16T05:24:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: "exists" 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "3158b0cc-849c-4383-8d4d-585cf0723e70", 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 21, 2024-09-16T05:24:12.494 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 295, 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 329, 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 296, 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2024-09-16T05:24:12.496 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.498 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3973436581 2024-09-16T05:24:12.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6819/3973436581", 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6821/3973436581", 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6825/3973436581", 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6823/3973436581", 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "71705a06-9102-4b97-be6a-4989306666d2", 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 27, 2024-09-16T05:24:12.501 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.502 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:24:12.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6832", 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6833", 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3040098358 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6827/3040098358", 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6829/3040098358", 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6833/3040098358", 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6831/3040098358", 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.507 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "ca8f47c4-c13f-4bde-b162-0e3b57bad1b4", 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 33, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.508 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6800", 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6801", 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:24:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6802", 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6803", 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6806", 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:24:12.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6807", 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6804", 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6805", 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3219469655 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6801/3219469655", 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6803/3219469655", 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3219469655", 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3219469655", 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "b5b76271-0539-42dd-a9e7-a95afded2d59", 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 38, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6808", 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6809", 2024-09-16T05:24:12.516 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6810", 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6811", 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6814", 2024-09-16T05:24:12.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6815", 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6812", 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6813", 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3274041128 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.520 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6809/3274041128", 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6811/3274041128", 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6815/3274041128", 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6813/3274041128", 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "d3f2312b-cc04-48e4-993a-f08a948ba3ca", 2024-09-16T05:24:12.521 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 0.2543182373046875, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 49, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 767, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6816", 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6817", 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.523 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6818", 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6819", 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:24:12.524 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6822", 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.525 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6823", 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6820", 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6821", 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2509099861 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6817/2509099861", 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6819/2509099861", 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6823/2509099861", 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6821/2509099861", 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.528 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.528 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.528 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:24:12.528 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7515ee-0a55-4b10-825e-8a4efa96d0e2", 2024-09-16T05:24:12.528 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 58, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.529 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6824", 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6825", 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6826", 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6827", 2024-09-16T05:24:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6830", 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6831", 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6828", 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.196:6829", 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3304789846 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.196:6825/3304789846", 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.196:6827/3304789846", 2024-09-16T05:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.196:6831/3304789846", 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.196:6829/3304789846", 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "e116a9c9-3c07-4b81-8b23-f2ba84916090", 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.535 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 66, 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6800", 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6801", 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6802", 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6803", 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6806", 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6807", 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:24:12.539 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6804", 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6805", 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3463685812 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6801/3463685812", 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6803/3463685812", 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6807/3463685812", 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6805/3463685812", 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.541 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "06573a93-d7be-426b-97c4-4e088b2bfa1a", 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.542 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 76, 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6808", 2024-09-16T05:24:12.543 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6809", 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6810", 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6811", 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6814", 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6815", 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6812", 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:24:12.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6813", 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1139381345 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6809/1139381345", 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6811/1139381345", 2024-09-16T05:24:12.548 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6815/1139381345", 2024-09-16T05:24:12.550 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:12.550 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:12.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:12 smithi196 ceph-mon[30802]: pgmap v1512: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 163 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 318 KiB/s wr, 5 op/s; 9/158 objects degraded (5.696%) 2024-09-16T05:24:12.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:12 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:12.550 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:12 smithi196 ceph-mon[30802]: osdmap e901: 12 total, 11 up, 12 in 2024-09-16T05:24:12.550 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6813/1139381345", 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "388cc928-3439-4f37-a2dc-1f2a494f9846", 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.551 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 83, 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6816", 2024-09-16T05:24:12.553 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:24:12.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6817", 2024-09-16T05:24:12.553 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6818", 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6819", 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.555 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6822", 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6823", 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.556 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6820", 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6821", 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 726811436 2024-09-16T05:24:12.557 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6817/726811436", 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6819/726811436", 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6823/726811436", 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6821/726811436", 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "49911d7e-fe03-41a6-a25a-11a11c08f8b3", 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2024-09-16T05:24:12.559 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 89, 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 867, 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6824", 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6825", 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6826", 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6827", 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6830", 2024-09-16T05:24:12.563 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6831", 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2024-09-16T05:24:12.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-09-16T05:24:12.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6828", 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.204:6829", 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1668426191 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.568 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.204:6825/1668426191", 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.204:6827/1668426191", 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.204:6831/1668426191", 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.204:6829/1668426191", 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.569 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:23:41.243289+0000", 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.570 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:53.256817+0000", 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 866 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2024-09-16T05:24:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:23:25.375559+0000", 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 295 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2024-09-16T05:24:12.572 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:23:47.401560+0000", 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.573 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:57.887383+0000", 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.574 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:23:54.478109+0000", 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.575 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:44.205006+0000", 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.576 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:24:04.026963+0000", 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 8, 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.577 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:23:41.042137+0000", 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 9, 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.578 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:23:32.510829+0000", 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 10, 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:22:59.519632+0000", 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 11, 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4540701547738038271, 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0, 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "last_purged_snaps_scrub": "2024-09-16T05:23:30.319158+0000", 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "dead_epoch": 0 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:24:12.580 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_primaries": [], 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklist": { 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2865075910": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/1180926382": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/194694811": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1290699598": "2024-09-17T05:03:25.948027+0000", 2024-09-16T05:24:12.581 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6800/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/2769196199": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/4153012034": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6801/178109651": "2024-09-17T05:03:00.800739+0000", 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:0/1722832924": "2024-09-17T05:03:00.800739+0000" 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "range_blocklist": {}, 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2024-09-16T05:24:12.582 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [ 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 365, 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:24:12.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 367, 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 369, 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 13 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 383, 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:24:12.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 386, 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 2 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [ 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: "snaps": [ 2024-09-16T05:24:12.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "begin": 387, 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "length": 1 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {}, 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode": { 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_enabled": false, 2024-09-16T05:24:12.586 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_bucket_count": 0, 2024-09-16T05:24:12.587 INFO:teuthology.orchestra.run.smithi032.stdout: "degraded_stretch_mode": 0, 2024-09-16T05:24:12.587 INFO:teuthology.orchestra.run.smithi032.stdout: "recovering_stretch_mode": 0, 2024-09-16T05:24:12.587 INFO:teuthology.orchestra.run.smithi032.stdout: "stretch_mode_bucket": 0 2024-09-16T05:24:12.587 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:12.587 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:24:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:12.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:12.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:12.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:12.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:12.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:12.838 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:12.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:12.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:12.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:12.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:13.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:13.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:13.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:13.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:13.106 INFO:tasks.rados.rados.2.smithi204.stdout:3580: done (10 left) 2024-09-16T05:24:13.106 INFO:tasks.rados.rados.2.smithi204.stdout:3581: done (9 left) 2024-09-16T05:24:13.106 INFO:tasks.rados.rados.2.smithi204.stdout:3582: snap_create 2024-09-16T05:24:13.198 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:13.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:13.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:13.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:24:13.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:13.350 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:13.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:13 smithi032 ceph-mon[27388]: osdmap e902: 12 total, 11 up, 12 in 2024-09-16T05:24:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4066995919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:24:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:13 smithi204 ceph-mon[30933]: osdmap e902: 12 total, 11 up, 12 in 2024-09-16T05:24:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4066995919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:24:13.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:13.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:13.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:13.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:13 smithi196 ceph-mon[30802]: osdmap e902: 12 total, 11 up, 12 in 2024-09-16T05:24:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4066995919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-16T05:24:13.524 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:24:13.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:13.593 INFO:tasks.thrashosds.thrasher:j is {'epoch': 902, 'fsid': 'c1b4ea6a-73e8-11ef-bceb-c7b262605968', 'created': '2024-09-16T05:02:35.583426+0000', 'modified': '2024-09-16T05:24:12.076861+0000', 'last_up_change': '2024-09-16T05:23:36.850362+0000', 'last_in_change': '2024-09-16T05:21:53.914532+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 39, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:05:16.445728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '687', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '685', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T05:10:35.539280+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 96, 'pg_placement_num': 96, 'pg_placement_num_target': 96, 'pg_num_target': 96, 'pg_num_pending': 96, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '902', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '737', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 387, 'snap_epoch': 902, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.600000023841858, 'raw_score_stable': 1.600000023841858, 'primary_affinity_weighted': 0.9378597736358643, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': 'f3cfb49b-8223-4649-b211-a2e254248d88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6802', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6803', 'nonce': 406159384}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6804', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6805', 'nonce': 406159384}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6808', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6809', 'nonce': 406159384}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6806', 'nonce': 406159384}, {'type': 'v1', 'addr': '172.21.15.32:6807', 'nonce': 406159384}]}, 'public_addr': '172.21.15.32:6803/406159384', 'cluster_addr': '172.21.15.32:6805/406159384', 'heartbeat_back_addr': '172.21.15.32:6809/406159384', 'heartbeat_front_addr': '172.21.15.32:6807/406159384', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'cfb43840-55b6-4d39-a325-2c5416ed97b4', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 853, 'down_at': 867, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6810', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6811', 'nonce': 976643951}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6812', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6813', 'nonce': 976643951}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6816', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6817', 'nonce': 976643951}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6814', 'nonce': 976643951}, {'type': 'v1', 'addr': '172.21.15.32:6815', 'nonce': 976643951}]}, 'public_addr': '172.21.15.32:6811/976643951', 'cluster_addr': '172.21.15.32:6813/976643951', 'heartbeat_back_addr': '172.21.15.32:6817/976643951', 'heartbeat_front_addr': '172.21.15.32:6815/976643951', 'state': ['exists']}, {'osd': 2, 'uuid': '3158b0cc-849c-4383-8d4d-585cf0723e70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 21, 'last_clean_end': 295, 'up_from': 329, 'up_thru': 867, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6818', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6819', 'nonce': 3973436581}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6820', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6821', 'nonce': 3973436581}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6824', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6825', 'nonce': 3973436581}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6822', 'nonce': 3973436581}, {'type': 'v1', 'addr': '172.21.15.32:6823', 'nonce': 3973436581}]}, 'public_addr': '172.21.15.32:6819/3973436581', 'cluster_addr': '172.21.15.32:6821/3973436581', 'heartbeat_back_addr': '172.21.15.32:6825/3973436581', 'heartbeat_front_addr': '172.21.15.32:6823/3973436581', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '71705a06-9102-4b97-be6a-4989306666d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6826', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6827', 'nonce': 3040098358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6828', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6829', 'nonce': 3040098358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3040098358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6830', 'nonce': 3040098358}, {'type': 'v1', 'addr': '172.21.15.32:6831', 'nonce': 3040098358}]}, 'public_addr': '172.21.15.32:6827/3040098358', 'cluster_addr': '172.21.15.32:6829/3040098358', 'heartbeat_back_addr': '172.21.15.32:6833/3040098358', 'heartbeat_front_addr': '172.21.15.32:6831/3040098358', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ca8f47c4-c13f-4bde-b162-0e3b57bad1b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3219469655}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3219469655}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3219469655}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3219469655}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3219469655}]}, 'public_addr': '172.21.15.196:6801/3219469655', 'cluster_addr': '172.21.15.196:6803/3219469655', 'heartbeat_back_addr': '172.21.15.196:6807/3219469655', 'heartbeat_front_addr': '172.21.15.196:6805/3219469655', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5b76271-0539-42dd-a9e7-a95afded2d59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 3274041128}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 3274041128}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 3274041128}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 3274041128}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 3274041128}]}, 'public_addr': '172.21.15.196:6809/3274041128', 'cluster_addr': '172.21.15.196:6811/3274041128', 'heartbeat_back_addr': '172.21.15.196:6815/3274041128', 'heartbeat_front_addr': '172.21.15.196:6813/3274041128', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd3f2312b-cc04-48e4-993a-f08a948ba3ca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2543182373046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 767, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 2509099861}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 2509099861}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 2509099861}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 2509099861}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 2509099861}]}, 'public_addr': '172.21.15.196:6817/2509099861', 'cluster_addr': '172.21.15.196:6819/2509099861', 'heartbeat_back_addr': '172.21.15.196:6823/2509099861', 'heartbeat_front_addr': '172.21.15.196:6821/2509099861', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fc7515ee-0a55-4b10-825e-8a4efa96d0e2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 3304789846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 3304789846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 3304789846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 3304789846}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 3304789846}]}, 'public_addr': '172.21.15.196:6825/3304789846', 'cluster_addr': '172.21.15.196:6827/3304789846', 'heartbeat_back_addr': '172.21.15.196:6831/3304789846', 'heartbeat_front_addr': '172.21.15.196:6829/3304789846', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'e116a9c9-3c07-4b81-8b23-f2ba84916090', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6801', 'nonce': 3463685812}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6803', 'nonce': 3463685812}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6807', 'nonce': 3463685812}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 3463685812}, {'type': 'v1', 'addr': '172.21.15.204:6805', 'nonce': 3463685812}]}, 'public_addr': '172.21.15.204:6801/3463685812', 'cluster_addr': '172.21.15.204:6803/3463685812', 'heartbeat_back_addr': '172.21.15.204:6807/3463685812', 'heartbeat_front_addr': '172.21.15.204:6805/3463685812', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '06573a93-d7be-426b-97c4-4e088b2bfa1a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6809', 'nonce': 1139381345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6811', 'nonce': 1139381345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6815', 'nonce': 1139381345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 1139381345}, {'type': 'v1', 'addr': '172.21.15.204:6813', 'nonce': 1139381345}]}, 'public_addr': '172.21.15.204:6809/1139381345', 'cluster_addr': '172.21.15.204:6811/1139381345', 'heartbeat_back_addr': '172.21.15.204:6815/1139381345', 'heartbeat_front_addr': '172.21.15.204:6813/1139381345', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '388cc928-3439-4f37-a2dc-1f2a494f9846', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6816', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6817', 'nonce': 726811436}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6818', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6819', 'nonce': 726811436}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6822', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6823', 'nonce': 726811436}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6820', 'nonce': 726811436}, {'type': 'v1', 'addr': '172.21.15.204:6821', 'nonce': 726811436}]}, 'public_addr': '172.21.15.204:6817/726811436', 'cluster_addr': '172.21.15.204:6819/726811436', 'heartbeat_back_addr': '172.21.15.204:6823/726811436', 'heartbeat_front_addr': '172.21.15.204:6821/726811436', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '49911d7e-fe03-41a6-a25a-11a11c08f8b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 867, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6824', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6825', 'nonce': 1668426191}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6826', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6827', 'nonce': 1668426191}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6830', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6831', 'nonce': 1668426191}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6828', 'nonce': 1668426191}, {'type': 'v1', 'addr': '172.21.15.204:6829', 'nonce': 1668426191}]}, 'public_addr': '172.21.15.204:6825/1668426191', 'cluster_addr': '172.21.15.204:6827/1668426191', 'heartbeat_back_addr': '172.21.15.204:6831/1668426191', 'heartbeat_front_addr': '172.21.15.204:6829/1668426191', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:23:41.243289+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:53.256817+0000', 'dead_epoch': 866}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:23:25.375559+0000', 'dead_epoch': 295}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:23:47.401560+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:57.887383+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:23:54.478109+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:44.205006+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:24:04.026963+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:23:41.042137+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:23:32.510829+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:22:59.519632+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T05:23:30.319158+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.32:0/2865075910': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6801/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/1180926382': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/194694811': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:0/1290699598': '2024-09-17T05:03:25.948027+0000', '172.21.15.32:6800/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/2769196199': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/4153012034': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:6801/178109651': '2024-09-17T05:03:00.800739+0000', '172.21.15.32:0/1722832924': '2024-09-17T05:03:00.800739+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 365, 'length': 1}, {'begin': 367, 'length': 1}, {'begin': 369, 'length': 13}, {'begin': 383, 'length': 1}, {'begin': 386, 'length': 2}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 387, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T05:24:13.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:24:13.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:13.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:13.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:13.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:13.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:13.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:24:13.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:13.853 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:13.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:13.890 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:13.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:13.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:13.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:14.014 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:14.014 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:14.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:14.103 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:24:14.107 INFO:tasks.rados.rados.2.smithi204.stdout:3582: done (9 left) 2024-09-16T05:24:14.107 INFO:tasks.rados.rados.2.smithi204.stdout:3583: snap_create 2024-09-16T05:24:14.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:14.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:14 smithi204 ceph-mon[30933]: pgmap v1515: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 166 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 0 B/s wr, 4 op/s; 9/160 objects degraded (5.625%) 2024-09-16T05:24:14.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:14 smithi204 ceph-mon[30933]: osdmap e903: 12 total, 11 up, 12 in 2024-09-16T05:24:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4073904244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:24:14.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:14.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:14.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:14.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:14.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:14.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:14 smithi032 ceph-mon[27388]: pgmap v1515: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 166 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 0 B/s wr, 4 op/s; 9/160 objects degraded (5.625%) 2024-09-16T05:24:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:14 smithi032 ceph-mon[27388]: osdmap e903: 12 total, 11 up, 12 in 2024-09-16T05:24:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4073904244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:24:14.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:14.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:14 smithi196 ceph-mon[30802]: pgmap v1515: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 166 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 0 B/s wr, 4 op/s; 9/160 objects degraded (5.625%) 2024-09-16T05:24:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:14 smithi196 ceph-mon[30802]: osdmap e903: 12 total, 11 up, 12 in 2024-09-16T05:24:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4073904244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:24:14.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:14.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:14.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:14.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:24:14.614 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:14.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:14.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:14.739 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:24:14.742 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:24:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:14.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:14.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:14.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:14.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:14.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:14.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:15.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:15.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:15.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:15.117 INFO:tasks.rados.rados.2.smithi204.stdout:3583: done (9 left) 2024-09-16T05:24:15.117 INFO:tasks.rados.rados.2.smithi204.stdout:3584: delete oid 40 current snap is 387 2024-09-16T05:24:15.122 INFO:tasks.rados.rados.2.smithi204.stdout:3584: done (9 left) 2024-09-16T05:24:15.122 INFO:tasks.rados.rados.2.smithi204.stdout:3585: read oid 40 snap -1 2024-09-16T05:24:15.122 INFO:tasks.rados.rados.2.smithi204.stdout:3585: expect deleted 2024-09-16T05:24:15.122 INFO:tasks.rados.rados.2.smithi204.stdout:3586: read oid 45 snap -1 2024-09-16T05:24:15.122 INFO:tasks.rados.rados.2.smithi204.stdout:3586: expect (ObjNum 1209 snap 366 seq_num 1209) 2024-09-16T05:24:15.122 INFO:tasks.rados.rados.2.smithi204.stdout:3587: snap_create 2024-09-16T05:24:15.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:15.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:15.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:15.132 INFO:tasks.rados.rados.2.smithi204.stdout:3586: expect (ObjNum 1209 snap 366 seq_num 1209) 2024-09-16T05:24:15.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:15.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:15.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:15.296 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:15.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:15.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:15 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:24:15.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:15 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:24:15.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4073904244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:24:15.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:15 smithi032 ceph-mon[27388]: osdmap e904: 12 total, 11 up, 12 in 2024-09-16T05:24:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:15.381 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:15.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:15 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:24:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:15 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:24:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4073904244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:24:15.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:15 smithi204 ceph-mon[30933]: osdmap e904: 12 total, 11 up, 12 in 2024-09-16T05:24:15.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:15.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:15 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:24:15.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:15 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:24:15.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4073904244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:24:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:15 smithi196 ceph-mon[30802]: osdmap e904: 12 total, 11 up, 12 in 2024-09-16T05:24:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:15.505 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:15.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:15.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:15.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:15.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:15.685 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1518,"stamp":"2024-09-16T05:24:14.234855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":177668535,"num_objects":83,"num_object_clones":42,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":6,"num_read":3998,"num_read_kb":3957498,"num_write":2853,"num_write_kb":1175241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38213,"ondisk_log_size":38213,"up":233,"acting":233,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":254,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":864376,"kb_used_data":309564,"kb_used_omap":27,"kb_used_meta":554148,"kb_avail":1124077448,"statfs":{"total":1151940427776,"available":1151055306752,"internally_reserved":0,"allocated":316993536,"data_stored":354496402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":567447719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6328851,"num_objects":4,"num_object_clones":4,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":11,"num_read_kb":8912,"num_write":13,"num_write_kb":2250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000795"},"pg_stats":[{"pgid":"2.5f","version":"284'221","reported_seq":4203,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.077232+0000","last_change":"2024-09-16T05:24:10.078189+0000","last_active":"2024-09-16T05:24:10.077232+0000","last_peered":"2024-09-16T05:24:10.077232+0000","last_clean":"2024-09-16T05:24:10.077232+0000","last_became_active":"2024-09-16T05:21:54.939189+0000","last_became_peered":"2024-09-16T05:21:54.939189+0000","last_unstale":"2024-09-16T05:24:10.077232+0000","last_undegraded":"2024-09-16T05:24:10.077232+0000","last_fullsized":"2024-09-16T05:24:10.077232+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:23:08.674041+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:23:08.674041+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:25.554768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011716459999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.5e","version":"877'564","reported_seq":4679,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200665+0000","last_change":"2024-09-16T05:24:10.200833+0000","last_active":"2024-09-16T05:24:10.200665+0000","last_peered":"2024-09-16T05:24:10.200665+0000","last_clean":"2024-09-16T05:24:10.200665+0000","last_became_active":"2024-09-16T05:21:54.937455+0000","last_became_peered":"2024-09-16T05:21:54.937455+0000","last_unstale":"2024-09-16T05:24:10.200665+0000","last_undegraded":"2024-09-16T05:24:10.200665+0000","last_fullsized":"2024-09-16T05:24:10.200665+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"823'562","last_scrub_stamp":"2024-09-16T05:23:16.364326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:16.364326+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:27.603924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001077508,"stat_sum":{"num_bytes":3630576,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":13007,"num_write":8,"num_write_kb":4535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4819607,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.5d","version":"866'1499","reported_seq":5614,"reported_epoch":900,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:24:10.199854+0000","last_change":"2024-09-16T05:23:37.866923+0000","last_active":"2024-09-16T05:23:36.872091+0000","last_peered":"2024-09-16T05:24:10.199854+0000","last_clean":"2024-09-16T05:23:35.855325+0000","last_became_active":"2024-09-16T05:21:32.830157+0000","last_became_peered":"2024-09-16T05:23:37.866923+0000","last_unstale":"2024-09-16T05:24:10.199854+0000","last_undegraded":"2024-09-16T05:23:37.861087+0000","last_fullsized":"2024-09-16T05:23:37.858832+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'1436","last_scrub_stamp":"2024-09-16T05:22:51.640147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.640147+0000","objects_scrubbed":6,"log_size":1499,"log_dups_size":0,"ondisk_log_size":1499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060484479999999997,"stat_sum":{"num_bytes":14532663,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":89,"num_read_kb":77940,"num_write":59,"num_write_kb":23832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[{"shards":"10","objects":6}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5c","version":"900'495","reported_seq":4099,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.091167+0000","last_change":"2024-09-16T05:24:10.091616+0000","last_active":"2024-09-16T05:24:10.091167+0000","last_peered":"2024-09-16T05:24:10.091167+0000","last_clean":"2024-09-16T05:24:10.091167+0000","last_became_active":"2024-09-16T05:21:32.828660+0000","last_became_peered":"2024-09-16T05:21:32.828660+0000","last_unstale":"2024-09-16T05:24:10.091167+0000","last_undegraded":"2024-09-16T05:24:10.091167+0000","last_fullsized":"2024-09-16T05:24:10.091167+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"873'487","last_scrub_stamp":"2024-09-16T05:23:59.754690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:59.754690+0000","objects_scrubbed":1,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:14.298999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053630769999999999,"stat_sum":{"num_bytes":3772620,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":7422,"num_write":9,"num_write_kb":2381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.5b","version":"861'678","reported_seq":4529,"reported_epoch":900,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:24:10.199452+0000","last_change":"2024-09-16T05:23:37.864464+0000","last_active":"2024-09-16T05:23:36.871791+0000","last_peered":"2024-09-16T05:24:10.199452+0000","last_clean":"2024-09-16T05:23:30.902132+0000","last_became_active":"2024-09-16T05:21:32.829681+0000","last_became_peered":"2024-09-16T05:23:37.864464+0000","last_unstale":"2024-09-16T05:24:10.199452+0000","last_undegraded":"2024-09-16T05:23:37.861042+0000","last_fullsized":"2024-09-16T05:23:37.860945+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'660","last_scrub_stamp":"2024-09-16T05:22:52.654103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:52.654103+0000","objects_scrubbed":2,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002530137,"stat_sum":{"num_bytes":3479247,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":52562,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[{"shards":"10","objects":2}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5a","version":"902'1262","reported_seq":5190,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.088443+0000","last_change":"2024-09-16T05:24:12.088654+0000","last_active":"2024-09-16T05:24:12.088443+0000","last_peered":"2024-09-16T05:24:12.088443+0000","last_clean":"2024-09-16T05:24:12.088443+0000","last_became_active":"2024-09-16T05:21:32.826884+0000","last_became_peered":"2024-09-16T05:21:32.826884+0000","last_unstale":"2024-09-16T05:24:12.088443+0000","last_undegraded":"2024-09-16T05:24:12.088443+0000","last_fullsized":"2024-09-16T05:24:12.088443+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"823'1224","last_scrub_stamp":"2024-09-16T05:22:54.387969+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.387969+0000","objects_scrubbed":1,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001795495,"stat_sum":{"num_bytes":3917477,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":21682,"num_write":34,"num_write_kb":7640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.59","version":"182'60","reported_seq":860,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.085748+0000","last_change":"2024-09-16T05:24:12.085965+0000","last_active":"2024-09-16T05:24:12.085748+0000","last_peered":"2024-09-16T05:24:12.085748+0000","last_clean":"2024-09-16T05:24:12.085748+0000","last_became_active":"2024-09-16T05:21:54.931981+0000","last_became_peered":"2024-09-16T05:21:54.931981+0000","last_unstale":"2024-09-16T05:24:12.085748+0000","last_undegraded":"2024-09-16T05:24:12.085748+0000","last_fullsized":"2024-09-16T05:24:12.085748+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:24:01.689923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:01.689923+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:18.048174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001223935,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.58","version":"181'57","reported_seq":3979,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.091535+0000","last_change":"2024-09-16T05:24:10.091904+0000","last_active":"2024-09-16T05:24:10.091535+0000","last_peered":"2024-09-16T05:24:10.091535+0000","last_clean":"2024-09-16T05:24:10.091535+0000","last_became_active":"2024-09-16T05:21:29.826747+0000","last_became_peered":"2024-09-16T05:21:29.826747+0000","last_unstale":"2024-09-16T05:24:10.091535+0000","last_undegraded":"2024-09-16T05:24:10.091535+0000","last_fullsized":"2024-09-16T05:24:10.091535+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:22:56.040114+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:22:56.040114+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00054953499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.57","version":"0'0","reported_seq":1513,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200761+0000","last_change":"2024-09-16T05:24:10.200946+0000","last_active":"2024-09-16T05:24:10.200761+0000","last_peered":"2024-09-16T05:24:10.200761+0000","last_clean":"2024-09-16T05:24:10.200761+0000","last_became_active":"2024-09-16T05:21:29.827377+0000","last_became_peered":"2024-09-16T05:21:29.827377+0000","last_unstale":"2024-09-16T05:24:10.200761+0000","last_undegraded":"2024-09-16T05:24:10.200761+0000","last_fullsized":"2024-09-16T05:24:10.200761+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:48.915248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.915248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":557,"objects_trimmed":0,"snaptrim_duration":0.001083618,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.56","version":"874'405","reported_seq":4021,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.087416+0000","last_change":"2024-09-16T05:24:10.087801+0000","last_active":"2024-09-16T05:24:10.087416+0000","last_peered":"2024-09-16T05:24:10.087416+0000","last_clean":"2024-09-16T05:24:10.087416+0000","last_became_active":"2024-09-16T05:21:32.829699+0000","last_became_peered":"2024-09-16T05:21:32.829699+0000","last_unstale":"2024-09-16T05:24:10.087416+0000","last_undegraded":"2024-09-16T05:24:10.087416+0000","last_fullsized":"2024-09-16T05:24:10.087416+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"834'390","last_scrub_stamp":"2024-09-16T05:23:07.704997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.704997+0000","objects_scrubbed":1,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:21.345556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00141943,"stat_sum":{"num_bytes":5358441,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":18486,"num_write":25,"num_write_kb":6689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.55","version":"433'507","reported_seq":2987,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.087583+0000","last_change":"2024-09-16T05:24:12.087835+0000","last_active":"2024-09-16T05:24:12.087583+0000","last_peered":"2024-09-16T05:24:12.087583+0000","last_clean":"2024-09-16T05:24:12.087583+0000","last_became_active":"2024-09-16T05:21:24.815957+0000","last_became_peered":"2024-09-16T05:21:24.815957+0000","last_unstale":"2024-09-16T05:24:12.087583+0000","last_undegraded":"2024-09-16T05:24:12.087583+0000","last_fullsized":"2024-09-16T05:24:12.087583+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:22:53.432495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:53.432495+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066649600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.54","version":"889'464","reported_seq":4080,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.080277+0000","last_change":"2024-09-16T05:24:12.080508+0000","last_active":"2024-09-16T05:24:12.080277+0000","last_peered":"2024-09-16T05:24:12.080277+0000","last_clean":"2024-09-16T05:24:12.080277+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:24:12.080277+0000","last_undegraded":"2024-09-16T05:24:12.080277+0000","last_fullsized":"2024-09-16T05:24:12.080277+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"852'451","last_scrub_stamp":"2024-09-16T05:23:33.467546+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:23:33.467546+0000","objects_scrubbed":1,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:58.182605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080612700000000002,"stat_sum":{"num_bytes":5916625,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":63623,"num_write":59,"num_write_kb":26798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.53","version":"181'15","reported_seq":2343,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.078121+0000","last_change":"2024-09-16T05:24:10.078907+0000","last_active":"2024-09-16T05:24:10.078121+0000","last_peered":"2024-09-16T05:24:10.078121+0000","last_clean":"2024-09-16T05:24:10.078121+0000","last_became_active":"2024-09-16T05:19:19.899501+0000","last_became_peered":"2024-09-16T05:19:19.899501+0000","last_unstale":"2024-09-16T05:24:10.078121+0000","last_undegraded":"2024-09-16T05:24:10.078121+0000","last_fullsized":"2024-09-16T05:24:10.078121+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:23:12.688624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:12.688624+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:23.848573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099500900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":2346,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.087846+0000","last_change":"2024-09-16T05:24:10.088063+0000","last_active":"2024-09-16T05:24:10.087846+0000","last_peered":"2024-09-16T05:24:10.087846+0000","last_clean":"2024-09-16T05:24:10.087846+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:24:10.087846+0000","last_undegraded":"2024-09-16T05:24:10.087846+0000","last_fullsized":"2024-09-16T05:24:10.087846+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:24:05.769924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:05.769924+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:20.568185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00165726,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":2406,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200219+0000","last_change":"2024-09-16T05:24:10.200402+0000","last_active":"2024-09-16T05:24:10.200219+0000","last_peered":"2024-09-16T05:24:10.200219+0000","last_clean":"2024-09-16T05:24:10.200219+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:24:10.200219+0000","last_undegraded":"2024-09-16T05:24:10.200219+0000","last_fullsized":"2024-09-16T05:24:10.200219+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:23:36.254009+0000","last_deep_scrub":"272'115","last_deep_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_clean_scrub_stamp":"2024-09-16T05:23:36.254009+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:59.094170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084690000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":2348,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.087148+0000","last_change":"2024-09-16T05:24:12.087454+0000","last_active":"2024-09-16T05:24:12.087148+0000","last_peered":"2024-09-16T05:24:12.087148+0000","last_clean":"2024-09-16T05:24:12.087148+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:24:12.087148+0000","last_undegraded":"2024-09-16T05:24:12.087148+0000","last_fullsized":"2024-09-16T05:24:12.087148+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:23:36.455890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:36.455890+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:40.478920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049244800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2450,"reported_epoch":901,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:11.107951+0000","last_change":"2024-09-16T05:23:37.862596+0000","last_active":"2024-09-16T05:23:36.874931+0000","last_peered":"2024-09-16T05:24:11.107951+0000","last_clean":"2024-09-16T05:21:05.836464+0000","last_became_active":"2024-09-16T05:19:35.789805+0000","last_became_peered":"2024-09-16T05:23:37.862596+0000","last_unstale":"2024-09-16T05:24:11.107951+0000","last_undegraded":"2024-09-16T05:24:11.107951+0000","last_fullsized":"2024-09-16T05:23:37.860001+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:22:33.493608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:33.493608+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021468759999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4e","version":"272'232","reported_seq":4099,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200122+0000","last_change":"2024-09-16T05:24:10.200328+0000","last_active":"2024-09-16T05:24:10.200122+0000","last_peered":"2024-09-16T05:24:10.200122+0000","last_clean":"2024-09-16T05:24:10.200122+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:24:10.200122+0000","last_undegraded":"2024-09-16T05:24:10.200122+0000","last_fullsized":"2024-09-16T05:24:10.200122+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:22:49.748809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.748809+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:11.798148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072092399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.4d","version":"901'751","reported_seq":4802,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.079858+0000","last_change":"2024-09-16T05:24:12.080133+0000","last_active":"2024-09-16T05:24:12.079858+0000","last_peered":"2024-09-16T05:24:12.079858+0000","last_clean":"2024-09-16T05:24:12.079858+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:24:12.079858+0000","last_undegraded":"2024-09-16T05:24:12.079858+0000","last_fullsized":"2024-09-16T05:24:12.079858+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"807'708","last_scrub_stamp":"2024-09-16T05:22:37.651060+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.651060+0000","objects_scrubbed":4,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00048425699999999998,"stat_sum":{"num_bytes":4050076,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":56213,"num_write":98,"num_write_kb":31496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.4c","version":"859'436","reported_seq":4419,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200526+0000","last_change":"2024-09-16T05:24:10.200694+0000","last_active":"2024-09-16T05:24:10.200526+0000","last_peered":"2024-09-16T05:24:10.200526+0000","last_clean":"2024-09-16T05:24:10.200526+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:24:10.200526+0000","last_undegraded":"2024-09-16T05:24:10.200526+0000","last_fullsized":"2024-09-16T05:24:10.200526+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"859'436","last_scrub_stamp":"2024-09-16T05:23:34.224426+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:23:34.224426+0000","objects_scrubbed":1,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:50.488597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010289909999999999,"stat_sum":{"num_bytes":2638963,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":99,"num_read_kb":121159,"num_write":46,"num_write_kb":17401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":4162,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.080672+0000","last_change":"2024-09-16T05:24:12.080995+0000","last_active":"2024-09-16T05:24:12.080672+0000","last_peered":"2024-09-16T05:24:12.080672+0000","last_clean":"2024-09-16T05:24:12.080672+0000","last_became_active":"2024-09-16T05:22:22.832492+0000","last_became_peered":"2024-09-16T05:22:22.832492+0000","last_unstale":"2024-09-16T05:24:12.080672+0000","last_undegraded":"2024-09-16T05:24:12.080672+0000","last_fullsized":"2024-09-16T05:24:12.080672+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:23:31.411299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:31.411299+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:44.000637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00095025899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.4a","version":"859'671","reported_seq":4624,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200891+0000","last_change":"2024-09-16T05:24:10.201062+0000","last_active":"2024-09-16T05:24:10.200891+0000","last_peered":"2024-09-16T05:24:10.200891+0000","last_clean":"2024-09-16T05:24:10.200891+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:24:10.200891+0000","last_undegraded":"2024-09-16T05:24:10.200891+0000","last_fullsized":"2024-09-16T05:24:10.200891+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"859'671","last_scrub_stamp":"2024-09-16T05:24:04.550256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:04.550256+0000","objects_scrubbed":1,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:30.510471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001190303,"stat_sum":{"num_bytes":2790226,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":72834,"num_write":60,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":4176,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199793+0000","last_change":"2024-09-16T05:24:10.200056+0000","last_active":"2024-09-16T05:24:10.199793+0000","last_peered":"2024-09-16T05:24:10.199793+0000","last_clean":"2024-09-16T05:24:10.199793+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:24:10.199793+0000","last_undegraded":"2024-09-16T05:24:10.199793+0000","last_fullsized":"2024-09-16T05:24:10.199793+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:23:07.713041+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.713041+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:36.632722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055725000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.48","version":"884'450","reported_seq":4389,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.091614+0000","last_change":"2024-09-16T05:24:10.092191+0000","last_active":"2024-09-16T05:24:10.091614+0000","last_peered":"2024-09-16T05:24:10.091614+0000","last_clean":"2024-09-16T05:24:10.091614+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:24:10.091614+0000","last_undegraded":"2024-09-16T05:24:10.091614+0000","last_fullsized":"2024-09-16T05:24:10.091614+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"823'428","last_scrub_stamp":"2024-09-16T05:23:36.279162+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:23:36.279162+0000","objects_scrubbed":1,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:54.067160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081900400000000002,"stat_sum":{"num_bytes":2487700,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":86,"num_read_kb":75762,"num_write":74,"num_write_kb":33570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":2964,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.086803+0000","last_change":"2024-09-16T05:24:10.087357+0000","last_active":"2024-09-16T05:24:10.086803+0000","last_peered":"2024-09-16T05:24:10.086803+0000","last_clean":"2024-09-16T05:24:10.086803+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:24:10.086803+0000","last_undegraded":"2024-09-16T05:24:10.086803+0000","last_fullsized":"2024-09-16T05:24:10.086803+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:23:34.586085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_clean_scrub_stamp":"2024-09-16T05:23:34.586085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:53.488008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011309930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":1201,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.080942+0000","last_change":"2024-09-16T05:24:12.081165+0000","last_active":"2024-09-16T05:24:12.080942+0000","last_peered":"2024-09-16T05:24:12.080942+0000","last_clean":"2024-09-16T05:24:12.080942+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:20:20.797682+0000","last_unstale":"2024-09-16T05:24:12.080942+0000","last_undegraded":"2024-09-16T05:24:12.080942+0000","last_fullsized":"2024-09-16T05:24:12.080942+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:23:37.496136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:37.496136+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:40.913774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094927900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.45","version":"305'464","reported_seq":3310,"reported_epoch":900,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:10.086007+0000","last_change":"2024-09-16T05:23:37.860738+0000","last_active":"2024-09-16T05:23:22.845158+0000","last_peered":"2024-09-16T05:24:10.086007+0000","last_clean":"2024-09-16T05:23:21.851499+0000","last_became_active":"2024-09-16T05:21:10.818112+0000","last_became_peered":"2024-09-16T05:23:37.860738+0000","last_unstale":"2024-09-16T05:24:10.086007+0000","last_undegraded":"2024-09-16T05:24:10.086007+0000","last_fullsized":"2024-09-16T05:23:37.858122+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:23:09.592714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.592714+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:27.524955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.006563284,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.44","version":"182'89","reported_seq":1674,"reported_epoch":902,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:12.084804+0000","last_change":"2024-09-16T05:23:37.862679+0000","last_active":"2024-09-16T05:23:36.858406+0000","last_peered":"2024-09-16T05:24:12.084804+0000","last_clean":"2024-09-16T05:21:06.121649+0000","last_became_active":"2024-09-16T05:19:35.788828+0000","last_became_peered":"2024-09-16T05:23:37.862679+0000","last_unstale":"2024-09-16T05:24:12.084804+0000","last_undegraded":"2024-09-16T05:24:12.084804+0000","last_fullsized":"2024-09-16T05:23:37.858165+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:23:31.555842+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_clean_scrub_stamp":"2024-09-16T05:23:31.555842+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:49.762059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744239999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.43","version":"281'58","reported_seq":2956,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.108353+0000","last_change":"2024-09-16T05:24:11.108572+0000","last_active":"2024-09-16T05:24:11.108353+0000","last_peered":"2024-09-16T05:24:11.108353+0000","last_clean":"2024-09-16T05:24:11.108353+0000","last_became_active":"2024-09-16T05:14:27.837701+0000","last_became_peered":"2024-09-16T05:14:27.837701+0000","last_unstale":"2024-09-16T05:24:11.108353+0000","last_undegraded":"2024-09-16T05:24:11.108353+0000","last_fullsized":"2024-09-16T05:24:11.108353+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:23:16.132504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:16.132504+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:21.817281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086974800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.42","version":"182'20","reported_seq":2950,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104406+0000","last_change":"2024-09-16T05:24:12.082428+0000","last_active":"2024-09-16T05:24:13.104406+0000","last_peered":"2024-09-16T05:24:13.104406+0000","last_clean":"2024-09-16T05:24:13.104406+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:24:13.104406+0000","last_undegraded":"2024-09-16T05:24:13.104406+0000","last_fullsized":"2024-09-16T05:24:13.104406+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:23:55.704187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:55.704187+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:14.603941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043242399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.41","version":"178'50","reported_seq":1809,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.086068+0000","last_change":"2024-09-16T05:24:10.086358+0000","last_active":"2024-09-16T05:24:10.086068+0000","last_peered":"2024-09-16T05:24:10.086068+0000","last_clean":"2024-09-16T05:24:10.086068+0000","last_became_active":"2024-09-16T05:21:54.930175+0000","last_became_peered":"2024-09-16T05:21:54.930175+0000","last_unstale":"2024-09-16T05:24:10.086068+0000","last_undegraded":"2024-09-16T05:24:10.086068+0000","last_fullsized":"2024-09-16T05:24:10.086068+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:23:39.661994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:39.661994+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:55.004686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049362699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":1495,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.202267+0000","last_change":"2024-09-16T05:24:10.202423+0000","last_active":"2024-09-16T05:24:10.202267+0000","last_peered":"2024-09-16T05:24:10.202267+0000","last_clean":"2024-09-16T05:24:10.202267+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:19:19.898993+0000","last_unstale":"2024-09-16T05:24:10.202267+0000","last_undegraded":"2024-09-16T05:24:10.202267+0000","last_fullsized":"2024-09-16T05:24:10.202267+0000","mapping_epoch":609,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:23:40.639251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:40.639251+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:54.106626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024562479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.3f","version":"874'563","reported_seq":4660,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.109296+0000","last_change":"2024-09-16T05:24:11.109296+0000","last_active":"2024-09-16T05:24:11.109296+0000","last_peered":"2024-09-16T05:24:11.109296+0000","last_clean":"2024-09-16T05:24:11.109296+0000","last_became_active":"2024-09-16T05:21:54.939233+0000","last_became_peered":"2024-09-16T05:21:54.939233+0000","last_unstale":"2024-09-16T05:24:11.109296+0000","last_undegraded":"2024-09-16T05:24:11.109296+0000","last_fullsized":"2024-09-16T05:24:11.109296+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"874'563","last_scrub_stamp":"2024-09-16T05:24:01.125030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:01.125030+0000","objects_scrubbed":3,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:03.965191+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010417639999999999,"stat_sum":{"num_bytes":4174627,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":104033,"num_write":77,"num_write_kb":26795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":11726959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.3e","version":"902'587","reported_seq":4543,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.086260+0000","last_change":"2024-09-16T05:24:12.086260+0000","last_active":"2024-09-16T05:24:12.086260+0000","last_peered":"2024-09-16T05:24:12.086260+0000","last_clean":"2024-09-16T05:24:12.086260+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:24:12.086260+0000","last_undegraded":"2024-09-16T05:24:12.086260+0000","last_fullsized":"2024-09-16T05:24:12.086260+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"820'580","last_scrub_stamp":"2024-09-16T05:22:51.944900+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.944900+0000","objects_scrubbed":1,"log_size":587,"log_dups_size":0,"ondisk_log_size":587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010387319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":61220,"num_write":68,"num_write_kb":35144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2007,"reported_epoch":903,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:13.104469+0000","last_change":"2024-09-16T05:23:37.856541+0000","last_active":"2024-09-16T05:23:36.855029+0000","last_peered":"2024-09-16T05:24:13.104469+0000","last_clean":"2024-09-16T05:21:05.805509+0000","last_became_active":"2024-09-16T05:19:35.784967+0000","last_became_peered":"2024-09-16T05:23:37.856541+0000","last_unstale":"2024-09-16T05:24:13.104469+0000","last_undegraded":"2024-09-16T05:24:13.104469+0000","last_fullsized":"2024-09-16T05:23:37.855660+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.497749+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:14.736192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053549299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[8],"avail_no_missing":["8"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3c","version":"819'826","reported_seq":4942,"reported_epoch":901,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:11.107618+0000","last_change":"2024-09-16T05:23:37.862640+0000","last_active":"2024-09-16T05:23:36.873896+0000","last_peered":"2024-09-16T05:24:11.107618+0000","last_clean":"2024-09-16T05:23:36.060434+0000","last_became_active":"2024-09-16T05:21:07.828778+0000","last_became_peered":"2024-09-16T05:23:37.862640+0000","last_unstale":"2024-09-16T05:24:11.107618+0000","last_undegraded":"2024-09-16T05:24:11.107618+0000","last_fullsized":"2024-09-16T05:23:37.859543+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'807","last_scrub_stamp":"2024-09-16T05:22:31.305689+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:22:31.305689+0000","objects_scrubbed":4,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00083692700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":194,"num_read_kb":258104,"num_write":95,"num_write_kb":34389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":14713263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3b","version":"852'522","reported_seq":4492,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.202025+0000","last_change":"2024-09-16T05:24:10.202304+0000","last_active":"2024-09-16T05:24:10.202025+0000","last_peered":"2024-09-16T05:24:10.202025+0000","last_clean":"2024-09-16T05:24:10.202025+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:24:10.202025+0000","last_undegraded":"2024-09-16T05:24:10.202025+0000","last_fullsized":"2024-09-16T05:24:10.202025+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"823'516","last_scrub_stamp":"2024-09-16T05:22:56.618547+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:22:56.618547+0000","objects_scrubbed":1,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024558269999999998,"stat_sum":{"num_bytes":2941489,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":103,"num_read_kb":103894,"num_write":70,"num_write_kb":32800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.3a","version":"875'686","reported_seq":4613,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.092494+0000","last_change":"2024-09-16T05:24:10.092837+0000","last_active":"2024-09-16T05:24:10.092494+0000","last_peered":"2024-09-16T05:24:10.092494+0000","last_clean":"2024-09-16T05:24:10.092494+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:24:10.092494+0000","last_undegraded":"2024-09-16T05:24:10.092494+0000","last_fullsized":"2024-09-16T05:24:10.092494+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"823'682","last_scrub_stamp":"2024-09-16T05:23:10.127124+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:23:10.127124+0000","objects_scrubbed":1,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:38.500207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012511130000000001,"stat_sum":{"num_bytes":3143173,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":66016,"num_write":80,"num_write_kb":43920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":3032,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.092246+0000","last_change":"2024-09-16T05:24:10.092647+0000","last_active":"2024-09-16T05:24:10.092246+0000","last_peered":"2024-09-16T05:24:10.092246+0000","last_clean":"2024-09-16T05:24:10.092246+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:24:10.092246+0000","last_undegraded":"2024-09-16T05:24:10.092246+0000","last_fullsized":"2024-09-16T05:24:10.092246+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:22:50.962024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:50.962024+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012092559999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":3063,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104159+0000","last_change":"2024-09-16T05:24:12.080922+0000","last_active":"2024-09-16T05:24:13.104159+0000","last_peered":"2024-09-16T05:24:13.104159+0000","last_clean":"2024-09-16T05:24:13.104159+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:24:13.104159+0000","last_undegraded":"2024-09-16T05:24:13.104159+0000","last_fullsized":"2024-09-16T05:24:13.104159+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:22:54.645961+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.645961+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:19.132485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090680600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.37","version":"0'0","reported_seq":1901,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.201129+0000","last_change":"2024-09-16T05:24:10.201271+0000","last_active":"2024-09-16T05:24:10.201129+0000","last_peered":"2024-09-16T05:24:10.201129+0000","last_clean":"2024-09-16T05:24:10.201129+0000","last_became_active":"2024-09-16T05:21:54.938965+0000","last_became_peered":"2024-09-16T05:21:54.938965+0000","last_unstale":"2024-09-16T05:24:10.201129+0000","last_undegraded":"2024-09-16T05:24:10.201129+0000","last_fullsized":"2024-09-16T05:24:10.201129+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:23:57.540847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:23:57.540847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:23.496917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010640319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":535,"reported_epoch":902,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:12.085173+0000","last_change":"2024-09-16T05:23:37.863558+0000","last_active":"2024-09-16T05:23:36.858320+0000","last_peered":"2024-09-16T05:24:12.085173+0000","last_clean":"2024-09-16T05:21:06.122490+0000","last_became_active":"2024-09-16T05:19:35.795592+0000","last_became_peered":"2024-09-16T05:23:37.863558+0000","last_unstale":"2024-09-16T05:24:12.085173+0000","last_undegraded":"2024-09-16T05:24:12.085173+0000","last_fullsized":"2024-09-16T05:23:37.859570+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:22:49.686720+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.686720+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026213400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.35","version":"843'947","reported_seq":4703,"reported_epoch":902,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:12.085257+0000","last_change":"2024-09-16T05:23:37.863020+0000","last_active":"2024-09-16T05:23:36.858871+0000","last_peered":"2024-09-16T05:24:12.085257+0000","last_clean":"2024-09-16T05:22:58.839384+0000","last_became_active":"2024-09-16T05:21:10.815197+0000","last_became_peered":"2024-09-16T05:23:37.863020+0000","last_unstale":"2024-09-16T05:24:12.085257+0000","last_undegraded":"2024-09-16T05:24:12.085257+0000","last_fullsized":"2024-09-16T05:23:37.858837+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"843'947","last_scrub_stamp":"2024-09-16T05:23:34.604753+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:23:34.604753+0000","objects_scrubbed":9,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:41.981950+0000","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0.0054130100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":191,"num_read_kb":188287,"num_write":131,"num_write_kb":57425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":17054243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.34","version":"900'864","reported_seq":4415,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.201559+0000","last_change":"2024-09-16T05:24:10.201981+0000","last_active":"2024-09-16T05:24:10.201559+0000","last_peered":"2024-09-16T05:24:10.201559+0000","last_clean":"2024-09-16T05:24:10.201559+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:24:10.201559+0000","last_undegraded":"2024-09-16T05:24:10.201559+0000","last_fullsized":"2024-09-16T05:24:10.201559+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"859'834","last_scrub_stamp":"2024-09-16T05:23:29.647792+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:23:29.647792+0000","objects_scrubbed":3,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:35.612969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023862330000000002,"stat_sum":{"num_bytes":15557420,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":221,"num_read_kb":245668,"num_write":157,"num_write_kb":57830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":3040,"reported_epoch":901,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:11.107729+0000","last_change":"2024-09-16T05:23:37.862595+0000","last_active":"2024-09-16T05:23:36.872961+0000","last_peered":"2024-09-16T05:24:11.107729+0000","last_clean":"2024-09-16T05:23:36.060365+0000","last_became_active":"2024-09-16T05:21:07.818018+0000","last_became_peered":"2024-09-16T05:23:37.862595+0000","last_unstale":"2024-09-16T05:24:11.107729+0000","last_undegraded":"2024-09-16T05:24:11.107729+0000","last_fullsized":"2024-09-16T05:23:37.859660+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:23:10.097585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:10.097585+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:35.119635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000766243,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.32","version":"889'320","reported_seq":3457,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.085314+0000","last_change":"2024-09-16T05:24:12.085559+0000","last_active":"2024-09-16T05:24:12.085314+0000","last_peered":"2024-09-16T05:24:12.085314+0000","last_clean":"2024-09-16T05:24:12.085314+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:24:12.085314+0000","last_undegraded":"2024-09-16T05:24:12.085314+0000","last_fullsized":"2024-09-16T05:24:12.085314+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"820'309","last_scrub_stamp":"2024-09-16T05:23:14.810426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:14.810426+0000","objects_scrubbed":1,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:31.807482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00083577900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":123,"num_read_kb":122722,"num_write":55,"num_write_kb":22810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.31","version":"875'402","reported_seq":4110,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.086180+0000","last_change":"2024-09-16T05:24:10.086751+0000","last_active":"2024-09-16T05:24:10.086180+0000","last_peered":"2024-09-16T05:24:10.086180+0000","last_clean":"2024-09-16T05:24:10.086180+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:24:10.086180+0000","last_undegraded":"2024-09-16T05:24:10.086180+0000","last_fullsized":"2024-09-16T05:24:10.086180+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"834'373","last_scrub_stamp":"2024-09-16T05:23:09.677247+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.677247+0000","objects_scrubbed":1,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:21.186701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000749565,"stat_sum":{"num_bytes":6101502,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":134688,"num_write":80,"num_write_kb":30708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":1875,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199831+0000","last_change":"2024-09-16T05:24:10.200159+0000","last_active":"2024-09-16T05:24:10.199831+0000","last_peered":"2024-09-16T05:24:10.199831+0000","last_clean":"2024-09-16T05:24:10.199831+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:24:10.199831+0000","last_undegraded":"2024-09-16T05:24:10.199831+0000","last_fullsized":"2024-09-16T05:24:10.199831+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:23:09.435437+0000","last_deep_scrub":"182'16","last_deep_scrub_stamp":"2024-09-16T05:21:49.039822+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.435437+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:13.819405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065041899999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":1450,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199396+0000","last_change":"2024-09-16T05:23:15.783311+0000","last_active":"2024-09-16T05:24:10.199396+0000","last_peered":"2024-09-16T05:24:10.199396+0000","last_clean":"2024-09-16T05:24:10.199396+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:24:10.199396+0000","last_undegraded":"2024-09-16T05:24:10.199396+0000","last_fullsized":"2024-09-16T05:24:10.199396+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:15.783191+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:23:15.783191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:38.927996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1244,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.089616+0000","last_change":"2024-09-16T05:24:12.090187+0000","last_active":"2024-09-16T05:24:12.089616+0000","last_peered":"2024-09-16T05:24:12.089616+0000","last_clean":"2024-09-16T05:24:12.089616+0000","last_became_active":"2024-09-16T05:21:57.821735+0000","last_became_peered":"2024-09-16T05:21:57.821735+0000","last_unstale":"2024-09-16T05:24:12.089616+0000","last_undegraded":"2024-09-16T05:24:12.089616+0000","last_fullsized":"2024-09-16T05:24:12.089616+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:23:06.972379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:06.972379+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00075949600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.13","version":"35'139","reported_seq":1316,"reported_epoch":902,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:12.084713+0000","last_change":"2024-09-16T05:23:37.862671+0000","last_active":"2024-09-16T05:23:36.857882+0000","last_peered":"2024-09-16T05:24:12.084713+0000","last_clean":"2024-09-16T05:21:06.121181+0000","last_became_active":"2024-09-16T05:19:35.788516+0000","last_became_peered":"2024-09-16T05:23:37.862671+0000","last_unstale":"2024-09-16T05:24:12.084713+0000","last_undegraded":"2024-09-16T05:24:12.084713+0000","last_fullsized":"2024-09-16T05:23:37.858101+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:13.594789+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:13.594789+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:37.050335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":3510,"reported_epoch":903,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:13.104277+0000","last_change":"2024-09-16T05:23:37.855979+0000","last_active":"2024-09-16T05:23:36.854647+0000","last_peered":"2024-09-16T05:24:13.104277+0000","last_clean":"2024-09-16T05:23:36.143409+0000","last_became_active":"2024-09-16T05:21:07.815584+0000","last_became_peered":"2024-09-16T05:23:37.855979+0000","last_unstale":"2024-09-16T05:24:13.104277+0000","last_undegraded":"2024-09-16T05:24:13.104277+0000","last_fullsized":"2024-09-16T05:23:37.855410+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:22:11.576311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:11.576311+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:14.736179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086469800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[8],"avail_no_missing":["8"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1414,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.087105+0000","last_change":"2024-09-16T05:23:17.423556+0000","last_active":"2024-09-16T05:24:12.087105+0000","last_peered":"2024-09-16T05:24:12.087105+0000","last_clean":"2024-09-16T05:24:12.087105+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:24:12.087105+0000","last_undegraded":"2024-09-16T05:24:12.087105+0000","last_fullsized":"2024-09-16T05:24:12.087105+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:17.423461+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:17.423461+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:40.550116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"823'461","reported_seq":4471,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104533+0000","last_change":"2024-09-16T05:24:12.082873+0000","last_active":"2024-09-16T05:24:13.104533+0000","last_peered":"2024-09-16T05:24:13.104533+0000","last_clean":"2024-09-16T05:24:13.104533+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:24:13.104533+0000","last_undegraded":"2024-09-16T05:24:13.104533+0000","last_fullsized":"2024-09-16T05:24:13.104533+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'437","last_scrub_stamp":"2024-09-16T05:22:27.742914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:27.742914+0000","objects_scrubbed":3,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00043977300000000002,"stat_sum":{"num_bytes":3244015,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":79204,"num_write":57,"num_write_kb":24396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.d","version":"35'139","reported_seq":1570,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.107599+0000","last_change":"2024-09-16T05:23:40.169449+0000","last_active":"2024-09-16T05:24:11.107599+0000","last_peered":"2024-09-16T05:24:11.107599+0000","last_clean":"2024-09-16T05:24:11.107599+0000","last_became_active":"2024-09-16T05:20:35.804924+0000","last_became_peered":"2024-09-16T05:20:35.804924+0000","last_unstale":"2024-09-16T05:24:11.107599+0000","last_undegraded":"2024-09-16T05:24:11.107599+0000","last_fullsized":"2024-09-16T05:24:11.107599+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:40.169358+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:40.169358+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:04.340430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":4117,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.081200+0000","last_change":"2024-09-16T05:24:12.081382+0000","last_active":"2024-09-16T05:24:12.081200+0000","last_peered":"2024-09-16T05:24:12.081200+0000","last_clean":"2024-09-16T05:24:12.081200+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:24:12.081200+0000","last_undegraded":"2024-09-16T05:24:12.081200+0000","last_fullsized":"2024-09-16T05:24:12.081200+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:23:40.603834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:40.603834+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:08.510114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093065900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.e","version":"35'139","reported_seq":1567,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.076891+0000","last_change":"2024-09-16T05:22:51.495227+0000","last_active":"2024-09-16T05:24:10.076891+0000","last_peered":"2024-09-16T05:24:10.076891+0000","last_clean":"2024-09-16T05:24:10.076891+0000","last_became_active":"2024-09-16T05:20:35.802192+0000","last_became_peered":"2024-09-16T05:20:35.802192+0000","last_unstale":"2024-09-16T05:24:10.076891+0000","last_undegraded":"2024-09-16T05:24:10.076891+0000","last_fullsized":"2024-09-16T05:24:10.076891+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:51.495148+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.495148+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:11.863592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"850'671","reported_seq":4653,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.093714+0000","last_change":"2024-09-16T05:24:10.093714+0000","last_active":"2024-09-16T05:24:10.093714+0000","last_peered":"2024-09-16T05:24:10.093714+0000","last_clean":"2024-09-16T05:24:10.093714+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:24:10.093714+0000","last_undegraded":"2024-09-16T05:24:10.093714+0000","last_fullsized":"2024-09-16T05:24:10.093714+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'671","last_scrub_stamp":"2024-09-16T05:23:40.265022+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:23:40.265022+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:07.621641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001619431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":52780,"num_write":55,"num_write_kb":30221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":1555,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199534+0000","last_change":"2024-09-16T05:23:32.647942+0000","last_active":"2024-09-16T05:24:10.199534+0000","last_peered":"2024-09-16T05:24:10.199534+0000","last_clean":"2024-09-16T05:24:10.199534+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:24:10.199534+0000","last_undegraded":"2024-09-16T05:24:10.199534+0000","last_fullsized":"2024-09-16T05:24:10.199534+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:32.647850+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:23:32.647850+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:02.015167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":4249,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.085387+0000","last_change":"2024-09-16T05:24:12.085631+0000","last_active":"2024-09-16T05:24:12.085387+0000","last_peered":"2024-09-16T05:24:12.085387+0000","last_clean":"2024-09-16T05:24:12.085387+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:24:12.085387+0000","last_undegraded":"2024-09-16T05:24:12.085387+0000","last_fullsized":"2024-09-16T05:24:12.085387+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:23:07.822545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.822545+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:34.908366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090864600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.b","version":"35'139","reported_seq":1112,"reported_epoch":900,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:10.085774+0000","last_change":"2024-09-16T05:23:37.860671+0000","last_active":"2024-09-16T05:23:36.858161+0000","last_peered":"2024-09-16T05:24:10.085774+0000","last_clean":"2024-09-16T05:22:23.838536+0000","last_became_active":"2024-09-16T05:22:23.836724+0000","last_became_peered":"2024-09-16T05:23:37.860671+0000","last_unstale":"2024-09-16T05:24:10.085774+0000","last_undegraded":"2024-09-16T05:24:10.085774+0000","last_fullsized":"2024-09-16T05:23:37.857985+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.487937+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.487937+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:14.784272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"893'385","reported_seq":4378,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.107987+0000","last_change":"2024-09-16T05:24:11.108271+0000","last_active":"2024-09-16T05:24:11.107987+0000","last_peered":"2024-09-16T05:24:11.107987+0000","last_clean":"2024-09-16T05:24:11.107987+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:14:25.821279+0000","last_unstale":"2024-09-16T05:24:11.107987+0000","last_undegraded":"2024-09-16T05:24:11.107987+0000","last_fullsized":"2024-09-16T05:24:11.107987+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"823'365","last_scrub_stamp":"2024-09-16T05:23:15.182994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:15.182994+0000","objects_scrubbed":1,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:20.566098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073290200000000001,"stat_sum":{"num_bytes":6336800,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":109729,"num_write":60,"num_write_kb":20355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.3","version":"900'311","reported_seq":4296,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.202083+0000","last_change":"2024-09-16T05:24:10.202220+0000","last_active":"2024-09-16T05:24:10.202083+0000","last_peered":"2024-09-16T05:24:10.202083+0000","last_clean":"2024-09-16T05:24:10.202083+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:24:10.202083+0000","last_undegraded":"2024-09-16T05:24:10.202083+0000","last_fullsized":"2024-09-16T05:24:10.202083+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'293","last_scrub_stamp":"2024-09-16T05:23:14.786482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:14.786482+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:17.334349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024749889999999999,"stat_sum":{"num_bytes":6168730,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":128645,"num_write":66,"num_write_kb":31951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":1105,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.084601+0000","last_change":"2024-09-16T05:22:57.013492+0000","last_active":"2024-09-16T05:24:12.084601+0000","last_peered":"2024-09-16T05:24:12.084601+0000","last_clean":"2024-09-16T05:24:12.084601+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:24:12.084601+0000","last_undegraded":"2024-09-16T05:24:12.084601+0000","last_fullsized":"2024-09-16T05:24:12.084601+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:57.013376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:57.013376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:25.634739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":3874,"reported_epoch":900,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:10.199518+0000","last_change":"2024-09-16T05:23:37.862002+0000","last_active":"2024-09-16T05:23:36.872784+0000","last_peered":"2024-09-16T05:24:10.199518+0000","last_clean":"2024-09-16T05:23:36.142641+0000","last_became_active":"2024-09-16T05:21:07.817459+0000","last_became_peered":"2024-09-16T05:23:37.862002+0000","last_unstale":"2024-09-16T05:24:10.199518+0000","last_undegraded":"2024-09-16T05:24:10.199518+0000","last_fullsized":"2024-09-16T05:23:37.860959+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:22:55.657664+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:22:55.657664+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088962700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1326,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199327+0000","last_change":"2024-09-16T05:22:38.207552+0000","last_active":"2024-09-16T05:24:10.199327+0000","last_peered":"2024-09-16T05:24:10.199327+0000","last_clean":"2024-09-16T05:24:10.199327+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:24:10.199327+0000","last_undegraded":"2024-09-16T05:24:10.199327+0000","last_fullsized":"2024-09-16T05:24:10.199327+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:22:38.207426+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":632,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.076910+0000","last_change":"2024-09-16T05:23:37.725107+0000","last_active":"2024-09-16T05:24:10.076910+0000","last_peered":"2024-09-16T05:24:10.076910+0000","last_clean":"2024-09-16T05:24:10.076910+0000","last_became_active":"2024-09-16T05:22:25.830349+0000","last_became_peered":"2024-09-16T05:22:25.830349+0000","last_unstale":"2024-09-16T05:24:10.076910+0000","last_undegraded":"2024-09-16T05:24:10.076910+0000","last_fullsized":"2024-09-16T05:24:10.076910+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:37.725010+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:23:37.725010+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:00.210217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":3926,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.079771+0000","last_change":"2024-09-16T05:24:12.080076+0000","last_active":"2024-09-16T05:24:12.079771+0000","last_peered":"2024-09-16T05:24:12.079771+0000","last_clean":"2024-09-16T05:24:12.079771+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:24:12.079771+0000","last_undegraded":"2024-09-16T05:24:12.079771+0000","last_fullsized":"2024-09-16T05:24:12.079771+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:23:12.563996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:12.563996+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:20.233778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050671600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.3","version":"35'139","reported_seq":2093,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.086876+0000","last_change":"2024-09-16T05:23:33.460077+0000","last_active":"2024-09-16T05:24:12.086876+0000","last_peered":"2024-09-16T05:24:12.086876+0000","last_clean":"2024-09-16T05:24:12.086876+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:24:12.086876+0000","last_undegraded":"2024-09-16T05:24:12.086876+0000","last_fullsized":"2024-09-16T05:24:12.086876+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:33.459993+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:23:33.459993+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:43.677011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":4007,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.108629+0000","last_change":"2024-09-16T05:24:11.108860+0000","last_active":"2024-09-16T05:24:11.108629+0000","last_peered":"2024-09-16T05:24:11.108629+0000","last_clean":"2024-09-16T05:24:11.108629+0000","last_became_active":"2024-09-16T05:17:59.868015+0000","last_became_peered":"2024-09-16T05:17:59.868015+0000","last_unstale":"2024-09-16T05:24:11.108629+0000","last_undegraded":"2024-09-16T05:24:11.108629+0000","last_fullsized":"2024-09-16T05:24:11.108629+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:23:41.134965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:41.134965+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:43.641429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010185940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":3209,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.093554+0000","last_change":"2024-09-16T05:24:10.093774+0000","last_active":"2024-09-16T05:24:10.093554+0000","last_peered":"2024-09-16T05:24:10.093554+0000","last_clean":"2024-09-16T05:24:10.093554+0000","last_became_active":"2024-09-16T05:21:54.939117+0000","last_became_peered":"2024-09-16T05:21:54.939117+0000","last_unstale":"2024-09-16T05:24:10.093554+0000","last_undegraded":"2024-09-16T05:24:10.093554+0000","last_fullsized":"2024-09-16T05:24:10.093554+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:58.088776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:58.088776+0000","last_clean_scrub_stamp":"2024-09-16T05:22:58.088776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:24.458656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001434556,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":2045,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.084928+0000","last_change":"2024-09-16T05:23:23.863728+0000","last_active":"2024-09-16T05:24:12.084928+0000","last_peered":"2024-09-16T05:24:12.084928+0000","last_clean":"2024-09-16T05:24:12.084928+0000","last_became_active":"2024-09-16T05:23:23.863394+0000","last_became_peered":"2024-09-16T05:23:23.863394+0000","last_unstale":"2024-09-16T05:24:12.084928+0000","last_undegraded":"2024-09-16T05:24:12.084928+0000","last_fullsized":"2024-09-16T05:24:12.084928+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:48.935924+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.935924+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":2411,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.081045+0000","last_change":"2024-09-16T05:24:12.081236+0000","last_active":"2024-09-16T05:24:12.081045+0000","last_peered":"2024-09-16T05:24:12.081045+0000","last_clean":"2024-09-16T05:24:12.081045+0000","last_became_active":"2024-09-16T05:21:21.816054+0000","last_became_peered":"2024-09-16T05:21:21.816054+0000","last_unstale":"2024-09-16T05:24:12.081045+0000","last_undegraded":"2024-09-16T05:24:12.081045+0000","last_fullsized":"2024-09-16T05:24:12.081045+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:23:54.550284+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_clean_scrub_stamp":"2024-09-16T05:23:54.550284+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:16.244823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090641699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.5","version":"35'139","reported_seq":1354,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200084+0000","last_change":"2024-09-16T05:23:37.275429+0000","last_active":"2024-09-16T05:24:10.200084+0000","last_peered":"2024-09-16T05:24:10.200084+0000","last_clean":"2024-09-16T05:24:10.200084+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:24:10.200084+0000","last_undegraded":"2024-09-16T05:24:10.200084+0000","last_fullsized":"2024-09-16T05:24:10.200084+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:23:37.275354+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:50.003034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"902'694","reported_seq":4708,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104444+0000","last_change":"2024-09-16T05:24:12.084212+0000","last_active":"2024-09-16T05:24:13.104444+0000","last_peered":"2024-09-16T05:24:13.104444+0000","last_clean":"2024-09-16T05:24:13.104444+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:24:13.104444+0000","last_undegraded":"2024-09-16T05:24:13.104444+0000","last_fullsized":"2024-09-16T05:24:13.104444+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"823'676","last_scrub_stamp":"2024-09-16T05:23:07.681237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.681237+0000","objects_scrubbed":1,"log_size":694,"log_dups_size":0,"ondisk_log_size":694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:21.341081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0035773440000000001,"stat_sum":{"num_bytes":5874113,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":172809,"num_write":62,"num_write_kb":23912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.6","version":"35'139","reported_seq":1298,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199700+0000","last_change":"2024-09-16T05:23:55.459224+0000","last_active":"2024-09-16T05:24:10.199700+0000","last_peered":"2024-09-16T05:24:10.199700+0000","last_clean":"2024-09-16T05:24:10.199700+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:24:10.199700+0000","last_undegraded":"2024-09-16T05:24:10.199700+0000","last_fullsized":"2024-09-16T05:24:10.199700+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:55.459140+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:55.459140+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:13.183578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":977,"reported_epoch":900,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:10.086270+0000","last_change":"2024-09-16T05:23:37.860731+0000","last_active":"2024-09-16T05:23:36.860060+0000","last_peered":"2024-09-16T05:24:10.086270+0000","last_clean":"2024-09-16T05:21:05.880843+0000","last_became_active":"2024-09-16T05:19:35.793136+0000","last_became_peered":"2024-09-16T05:23:37.860731+0000","last_unstale":"2024-09-16T05:24:10.086270+0000","last_undegraded":"2024-09-16T05:24:10.086270+0000","last_fullsized":"2024-09-16T05:23:37.858591+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:23:17.624562+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:23:17.624562+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:44.626527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010972359999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1344,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200034+0000","last_change":"2024-09-16T05:23:39.246543+0000","last_active":"2024-09-16T05:24:10.200034+0000","last_peered":"2024-09-16T05:24:10.200034+0000","last_clean":"2024-09-16T05:24:10.200034+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:24:10.200034+0000","last_undegraded":"2024-09-16T05:24:10.200034+0000","last_fullsized":"2024-09-16T05:24:10.200034+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:39.246469+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:23:39.246469+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:42.767761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":4169,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104294+0000","last_change":"2024-09-16T05:24:12.080881+0000","last_active":"2024-09-16T05:24:13.104294+0000","last_peered":"2024-09-16T05:24:13.104294+0000","last_clean":"2024-09-16T05:24:13.104294+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:24:13.104294+0000","last_undegraded":"2024-09-16T05:24:13.104294+0000","last_fullsized":"2024-09-16T05:24:13.104294+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:24:03.635537+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:24:03.635537+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:26.380267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043558300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.8","version":"35'139","reported_seq":2081,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200151+0000","last_change":"2024-09-16T05:23:37.620309+0000","last_active":"2024-09-16T05:24:10.200151+0000","last_peered":"2024-09-16T05:24:10.200151+0000","last_clean":"2024-09-16T05:24:10.200151+0000","last_became_active":"2024-09-16T05:20:35.807550+0000","last_became_peered":"2024-09-16T05:20:35.807550+0000","last_unstale":"2024-09-16T05:24:10.200151+0000","last_undegraded":"2024-09-16T05:24:10.200151+0000","last_fullsized":"2024-09-16T05:24:10.200151+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:37.620222+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:37.620222+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:45.364854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"884'766","reported_seq":4797,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.085818+0000","last_change":"2024-09-16T05:24:12.086519+0000","last_active":"2024-09-16T05:24:12.085818+0000","last_peered":"2024-09-16T05:24:12.085818+0000","last_clean":"2024-09-16T05:24:12.085818+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:24:12.085818+0000","last_undegraded":"2024-09-16T05:24:12.085818+0000","last_fullsized":"2024-09-16T05:24:12.085818+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"868'757","last_scrub_stamp":"2024-09-16T05:23:38.706205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:38.706205+0000","objects_scrubbed":3,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:03.355372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014028879999999999,"stat_sum":{"num_bytes":3462506,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":144,"num_read_kb":161043,"num_write":96,"num_write_kb":44068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.9","version":"35'139","reported_seq":2081,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.107472+0000","last_change":"2024-09-16T05:23:34.099216+0000","last_active":"2024-09-16T05:24:11.107472+0000","last_peered":"2024-09-16T05:24:11.107472+0000","last_clean":"2024-09-16T05:24:11.107472+0000","last_became_active":"2024-09-16T05:20:35.804221+0000","last_became_peered":"2024-09-16T05:20:35.804221+0000","last_unstale":"2024-09-16T05:24:11.107472+0000","last_undegraded":"2024-09-16T05:24:11.107472+0000","last_fullsized":"2024-09-16T05:24:11.107472+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:34.099127+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:34.099127+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:47.618240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2008,"reported_epoch":900,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:10.077540+0000","last_change":"2024-09-16T05:23:37.857332+0000","last_active":"2024-09-16T05:23:36.858356+0000","last_peered":"2024-09-16T05:24:10.077540+0000","last_clean":"2024-09-16T05:21:06.115367+0000","last_became_active":"2024-09-16T05:19:35.786308+0000","last_became_peered":"2024-09-16T05:23:37.857332+0000","last_unstale":"2024-09-16T05:24:10.077540+0000","last_undegraded":"2024-09-16T05:24:10.077540+0000","last_fullsized":"2024-09-16T05:23:37.856797+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:23:07.627317+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.627317+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:36.208497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053873599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2087,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.091672+0000","last_change":"2024-09-16T05:23:16.175068+0000","last_active":"2024-09-16T05:24:10.091672+0000","last_peered":"2024-09-16T05:24:10.091672+0000","last_clean":"2024-09-16T05:24:10.091672+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:24:10.091672+0000","last_undegraded":"2024-09-16T05:24:10.091672+0000","last_fullsized":"2024-09-16T05:24:10.091672+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:16.174965+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:23:16.174965+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:45.145736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1565,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.085763+0000","last_change":"2024-09-16T05:23:40.660852+0000","last_active":"2024-09-16T05:24:10.085763+0000","last_peered":"2024-09-16T05:24:10.085763+0000","last_clean":"2024-09-16T05:24:10.085763+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:24:10.085763+0000","last_undegraded":"2024-09-16T05:24:10.085763+0000","last_fullsized":"2024-09-16T05:24:10.085763+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:40.660777+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:23:40.660777+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:44.752307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":1875,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199895+0000","last_change":"2024-09-16T05:24:10.200149+0000","last_active":"2024-09-16T05:24:10.199895+0000","last_peered":"2024-09-16T05:24:10.199895+0000","last_clean":"2024-09-16T05:24:10.199895+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:24:10.199895+0000","last_undegraded":"2024-09-16T05:24:10.199895+0000","last_fullsized":"2024-09-16T05:24:10.199895+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:22:56.409531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:56.409531+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:18.285559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060466399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":2069,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200002+0000","last_change":"2024-09-16T05:22:50.729358+0000","last_active":"2024-09-16T05:24:10.200002+0000","last_peered":"2024-09-16T05:24:10.200002+0000","last_clean":"2024-09-16T05:24:10.200002+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:24:10.200002+0000","last_undegraded":"2024-09-16T05:24:10.200002+0000","last_fullsized":"2024-09-16T05:24:10.200002+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:50.729283+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:50.729283+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:18.895673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":3592,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.093066+0000","last_change":"2024-09-16T05:24:10.093366+0000","last_active":"2024-09-16T05:24:10.093066+0000","last_peered":"2024-09-16T05:24:10.093066+0000","last_clean":"2024-09-16T05:24:10.093066+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:24:10.093066+0000","last_undegraded":"2024-09-16T05:24:10.093066+0000","last_fullsized":"2024-09-16T05:24:10.093066+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:23:58.198238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:58.198238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:00.547604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001469269,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.17","version":"35'139","reported_seq":968,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104329+0000","last_change":"2024-09-16T05:23:15.626020+0000","last_active":"2024-09-16T05:24:13.104329+0000","last_peered":"2024-09-16T05:24:13.104329+0000","last_clean":"2024-09-16T05:24:13.104329+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:24:13.104329+0000","last_undegraded":"2024-09-16T05:24:13.104329+0000","last_fullsized":"2024-09-16T05:24:13.104329+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:15.625945+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:23:15.625945+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:28.396721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":3870,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.093216+0000","last_change":"2024-09-16T05:24:10.093502+0000","last_active":"2024-09-16T05:24:10.093216+0000","last_peered":"2024-09-16T05:24:10.093216+0000","last_clean":"2024-09-16T05:24:10.093216+0000","last_became_active":"2024-09-16T05:21:54.931954+0000","last_became_peered":"2024-09-16T05:21:54.931954+0000","last_unstale":"2024-09-16T05:24:10.093216+0000","last_undegraded":"2024-09-16T05:24:10.093216+0000","last_fullsized":"2024-09-16T05:24:10.093216+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:23:11.097094+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:23:11.097094+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:11.687931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001475163,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.16","version":"35'139","reported_seq":1021,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.079998+0000","last_change":"2024-09-16T05:23:34.506617+0000","last_active":"2024-09-16T05:24:12.079998+0000","last_peered":"2024-09-16T05:24:12.079998+0000","last_clean":"2024-09-16T05:24:12.079998+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:24:12.079998+0000","last_undegraded":"2024-09-16T05:24:12.079998+0000","last_fullsized":"2024-09-16T05:24:12.079998+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:34.506534+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:23:34.506534+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:42.772369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"848'659","reported_seq":4668,"reported_epoch":900,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:24:10.091820+0000","last_change":"2024-09-16T05:23:37.863036+0000","last_active":"2024-09-16T05:23:37.015531+0000","last_peered":"2024-09-16T05:24:10.091820+0000","last_clean":"2024-09-16T05:23:36.145867+0000","last_became_active":"2024-09-16T05:21:07.826042+0000","last_became_peered":"2024-09-16T05:23:37.863036+0000","last_unstale":"2024-09-16T05:24:10.091820+0000","last_undegraded":"2024-09-16T05:23:37.861512+0000","last_fullsized":"2024-09-16T05:23:37.861414+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"848'659","last_scrub_stamp":"2024-09-16T05:23:33.881707+0000","last_deep_scrub":"760'658","last_deep_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_clean_scrub_stamp":"2024-09-16T05:23:33.881707+0000","objects_scrubbed":1,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:58.895066+0000","scrub_duration":582,"objects_trimmed":0,"snaptrim_duration":0.003249806,"stat_sum":{"num_bytes":3008717,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":54,"num_read_kb":47657,"num_write":26,"num_write_kb":17519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":7118573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":1}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":1033,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199473+0000","last_change":"2024-09-16T05:23:34.643043+0000","last_active":"2024-09-16T05:24:10.199473+0000","last_peered":"2024-09-16T05:24:10.199473+0000","last_clean":"2024-09-16T05:24:10.199473+0000","last_became_active":"2024-09-16T05:20:40.802090+0000","last_became_peered":"2024-09-16T05:20:40.802090+0000","last_unstale":"2024-09-16T05:24:10.199473+0000","last_undegraded":"2024-09-16T05:24:10.199473+0000","last_fullsized":"2024-09-16T05:24:10.199473+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:34.642901+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:23:34.642901+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:52.097380+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"733'334","reported_seq":3954,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200509+0000","last_change":"2024-09-16T05:24:10.200769+0000","last_active":"2024-09-16T05:24:10.200509+0000","last_peered":"2024-09-16T05:24:10.200509+0000","last_clean":"2024-09-16T05:24:10.200509+0000","last_became_active":"2024-09-16T05:21:54.937285+0000","last_became_peered":"2024-09-16T05:21:54.937285+0000","last_unstale":"2024-09-16T05:24:10.200509+0000","last_undegraded":"2024-09-16T05:24:10.200509+0000","last_fullsized":"2024-09-16T05:24:10.200509+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'334","last_scrub_stamp":"2024-09-16T05:24:00.907413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:00.907413+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:09.839167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001066043,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":1403,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.079746+0000","last_change":"2024-09-16T05:23:13.614007+0000","last_active":"2024-09-16T05:24:12.079746+0000","last_peered":"2024-09-16T05:24:12.079746+0000","last_clean":"2024-09-16T05:24:12.079746+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:24:12.079746+0000","last_undegraded":"2024-09-16T05:24:12.079746+0000","last_fullsized":"2024-09-16T05:24:12.079746+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:13.613928+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:13.613928+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:32.652535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":753,"reported_epoch":900,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:10.199919+0000","last_change":"2024-09-16T05:23:37.866082+0000","last_active":"2024-09-16T05:21:26.818410+0000","last_peered":"2024-09-16T05:24:10.199919+0000","last_clean":"2024-09-16T05:21:26.020444+0000","last_became_active":"2024-09-16T05:21:07.822615+0000","last_became_peered":"2024-09-16T05:23:37.866082+0000","last_unstale":"2024-09-16T05:24:10.199919+0000","last_undegraded":"2024-09-16T05:24:10.199919+0000","last_fullsized":"2024-09-16T05:23:37.858903+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.831827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:14.572429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068139299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1493,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104178+0000","last_change":"2024-09-16T05:24:02.673934+0000","last_active":"2024-09-16T05:24:13.104178+0000","last_peered":"2024-09-16T05:24:13.104178+0000","last_clean":"2024-09-16T05:24:13.104178+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:24:13.104178+0000","last_undegraded":"2024-09-16T05:24:13.104178+0000","last_fullsized":"2024-09-16T05:24:13.104178+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:02.673862+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:24:02.673862+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:18.985051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":3985,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.092703+0000","last_change":"2024-09-16T05:24:10.092998+0000","last_active":"2024-09-16T05:24:10.092703+0000","last_peered":"2024-09-16T05:24:10.092703+0000","last_clean":"2024-09-16T05:24:10.092703+0000","last_became_active":"2024-09-16T05:21:27.832312+0000","last_became_peered":"2024-09-16T05:21:27.832312+0000","last_unstale":"2024-09-16T05:24:10.092703+0000","last_undegraded":"2024-09-16T05:24:10.092703+0000","last_fullsized":"2024-09-16T05:24:10.092703+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:24:05.187021+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:24:05.187021+0000","last_clean_scrub_stamp":"2024-09-16T05:24:05.187021+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:22.770336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001366728,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.1a","version":"35'139","reported_seq":851,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.077386+0000","last_change":"2024-09-16T05:23:32.769040+0000","last_active":"2024-09-16T05:24:10.077386+0000","last_peered":"2024-09-16T05:24:10.077386+0000","last_clean":"2024-09-16T05:24:10.077386+0000","last_became_active":"2024-09-16T05:20:40.806260+0000","last_became_peered":"2024-09-16T05:20:40.806260+0000","last_unstale":"2024-09-16T05:24:10.077386+0000","last_undegraded":"2024-09-16T05:24:10.077386+0000","last_fullsized":"2024-09-16T05:24:10.077386+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:32.768956+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:23:32.768956+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:01.489226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":3616,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.087949+0000","last_change":"2024-09-16T05:24:12.088159+0000","last_active":"2024-09-16T05:24:12.087949+0000","last_peered":"2024-09-16T05:24:12.087949+0000","last_clean":"2024-09-16T05:24:12.087949+0000","last_became_active":"2024-09-16T05:21:54.930126+0000","last_became_peered":"2024-09-16T05:21:54.930126+0000","last_unstale":"2024-09-16T05:24:12.087949+0000","last_undegraded":"2024-09-16T05:24:12.087949+0000","last_fullsized":"2024-09-16T05:24:12.087949+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:24:00.341680+0000","last_deep_scrub":"182'60","last_deep_scrub_stamp":"2024-09-16T05:21:29.356695+0000","last_clean_scrub_stamp":"2024-09-16T05:24:00.341680+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:02.469724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068435900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.19","version":"35'139","reported_seq":2079,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199416+0000","last_change":"2024-09-16T05:24:00.579767+0000","last_active":"2024-09-16T05:24:10.199416+0000","last_peered":"2024-09-16T05:24:10.199416+0000","last_clean":"2024-09-16T05:24:10.199416+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:24:10.199416+0000","last_undegraded":"2024-09-16T05:24:10.199416+0000","last_fullsized":"2024-09-16T05:24:10.199416+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:24:00.579637+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:03.398614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"902'1283","reported_seq":5407,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104593+0000","last_change":"2024-09-16T05:24:12.082176+0000","last_active":"2024-09-16T05:24:13.104593+0000","last_peered":"2024-09-16T05:24:13.104593+0000","last_clean":"2024-09-16T05:24:13.104593+0000","last_became_active":"2024-09-16T05:21:54.943140+0000","last_became_peered":"2024-09-16T05:21:54.943140+0000","last_unstale":"2024-09-16T05:24:13.104593+0000","last_undegraded":"2024-09-16T05:24:13.104593+0000","last_fullsized":"2024-09-16T05:24:13.104593+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"819'1229","last_scrub_stamp":"2024-09-16T05:22:49.730316+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.730316+0000","objects_scrubbed":4,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00045379200000000001,"stat_sum":{"num_bytes":12826644,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":46,"num_read_kb":54733,"num_write":39,"num_write_kb":14056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":19652756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.18","version":"35'139","reported_seq":2081,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.199896+0000","last_change":"2024-09-16T05:23:35.605262+0000","last_active":"2024-09-16T05:24:10.199896+0000","last_peered":"2024-09-16T05:24:10.199896+0000","last_clean":"2024-09-16T05:24:10.199896+0000","last_became_active":"2024-09-16T05:20:35.809442+0000","last_became_peered":"2024-09-16T05:20:35.809442+0000","last_unstale":"2024-09-16T05:24:10.199896+0000","last_undegraded":"2024-09-16T05:24:10.199896+0000","last_fullsized":"2024-09-16T05:24:10.199896+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:35.605162+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:35.605162+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:50.747365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"861'742","reported_seq":4769,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.086332+0000","last_change":"2024-09-16T05:24:12.086550+0000","last_active":"2024-09-16T05:24:12.086332+0000","last_peered":"2024-09-16T05:24:12.086332+0000","last_clean":"2024-09-16T05:24:12.086332+0000","last_became_active":"2024-09-16T05:21:27.832003+0000","last_became_peered":"2024-09-16T05:21:27.832003+0000","last_unstale":"2024-09-16T05:24:12.086332+0000","last_undegraded":"2024-09-16T05:24:12.086332+0000","last_fullsized":"2024-09-16T05:24:12.086332+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"823'734","last_scrub_stamp":"2024-09-16T05:22:54.026063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.026063+0000","objects_scrubbed":1,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:19.427162+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001150114,"stat_sum":{"num_bytes":6420802,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":38120,"num_write":20,"num_write_kb":10654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.1c","version":"723'472","reported_seq":4146,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200466+0000","last_change":"2024-09-16T05:24:10.200609+0000","last_active":"2024-09-16T05:24:10.200466+0000","last_peered":"2024-09-16T05:24:10.200466+0000","last_clean":"2024-09-16T05:24:10.200466+0000","last_became_active":"2024-09-16T05:21:27.831591+0000","last_became_peered":"2024-09-16T05:21:27.831591+0000","last_unstale":"2024-09-16T05:24:10.200466+0000","last_undegraded":"2024-09-16T05:24:10.200466+0000","last_fullsized":"2024-09-16T05:24:10.200466+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"723'472","last_scrub_stamp":"2024-09-16T05:22:51.347894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.347894+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096977299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1572,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.085005+0000","last_change":"2024-09-16T05:24:00.648131+0000","last_active":"2024-09-16T05:24:12.085005+0000","last_peered":"2024-09-16T05:24:12.085005+0000","last_clean":"2024-09-16T05:24:12.085005+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:24:12.085005+0000","last_undegraded":"2024-09-16T05:24:12.085005+0000","last_fullsized":"2024-09-16T05:24:12.085005+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:24:00.648056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:26.421151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"874'1314","reported_seq":5523,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200987+0000","last_change":"2024-09-16T05:24:10.201154+0000","last_active":"2024-09-16T05:24:10.200987+0000","last_peered":"2024-09-16T05:24:10.200987+0000","last_clean":"2024-09-16T05:24:10.200987+0000","last_became_active":"2024-09-16T05:21:27.833962+0000","last_became_peered":"2024-09-16T05:21:27.833962+0000","last_unstale":"2024-09-16T05:24:10.200987+0000","last_undegraded":"2024-09-16T05:24:10.200987+0000","last_fullsized":"2024-09-16T05:24:10.200987+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"823'1307","last_scrub_stamp":"2024-09-16T05:22:53.289164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:53.289164+0000","objects_scrubbed":0,"log_size":1314,"log_dups_size":0,"ondisk_log_size":1314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010917769999999999,"stat_sum":{"num_bytes":3258219,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":7,"num_write_kb":5679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"1.1d","version":"35'139","reported_seq":661,"reported_epoch":902,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:12.084921+0000","last_change":"2024-09-16T05:23:37.862749+0000","last_active":"2024-09-16T05:23:36.858349+0000","last_peered":"2024-09-16T05:24:12.084921+0000","last_clean":"2024-09-16T05:21:06.122751+0000","last_became_active":"2024-09-16T05:20:40.803676+0000","last_became_peered":"2024-09-16T05:23:37.862749+0000","last_unstale":"2024-09-16T05:24:12.084921+0000","last_undegraded":"2024-09-16T05:24:12.084921+0000","last_fullsized":"2024-09-16T05:23:37.858541+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:37.530632+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.530632+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1e","version":"736'512","reported_seq":4540,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.088225+0000","last_change":"2024-09-16T05:24:12.088395+0000","last_active":"2024-09-16T05:24:12.088225+0000","last_peered":"2024-09-16T05:24:12.088225+0000","last_clean":"2024-09-16T05:24:12.088225+0000","last_became_active":"2024-09-16T05:21:27.826212+0000","last_became_peered":"2024-09-16T05:21:27.826212+0000","last_unstale":"2024-09-16T05:24:12.088225+0000","last_undegraded":"2024-09-16T05:24:12.088225+0000","last_fullsized":"2024-09-16T05:24:12.088225+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'512","last_scrub_stamp":"2024-09-16T05:23:54.269322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:54.269322+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:05.235773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090722000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"1.1c","version":"35'139","reported_seq":823,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104325+0000","last_change":"2024-09-16T05:23:39.720611+0000","last_active":"2024-09-16T05:24:13.104325+0000","last_peered":"2024-09-16T05:24:13.104325+0000","last_clean":"2024-09-16T05:24:13.104325+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:24:13.104325+0000","last_undegraded":"2024-09-16T05:24:13.104325+0000","last_fullsized":"2024-09-16T05:24:13.104325+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:39.720513+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:39.720513+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:04.660636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":4206,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.088129+0000","last_change":"2024-09-16T05:24:10.088331+0000","last_active":"2024-09-16T05:24:10.088129+0000","last_peered":"2024-09-16T05:24:10.088129+0000","last_clean":"2024-09-16T05:24:10.088129+0000","last_became_active":"2024-09-16T05:21:27.831337+0000","last_became_peered":"2024-09-16T05:21:27.831337+0000","last_unstale":"2024-09-16T05:24:10.088129+0000","last_undegraded":"2024-09-16T05:24:10.088129+0000","last_fullsized":"2024-09-16T05:24:10.088129+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:23:57.788027+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:23:57.788027+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:02.901820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018674589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.20","version":"868'385","reported_seq":4412,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.108064+0000","last_change":"2024-09-16T05:24:11.108326+0000","last_active":"2024-09-16T05:24:11.108064+0000","last_peered":"2024-09-16T05:24:11.108064+0000","last_clean":"2024-09-16T05:24:11.108064+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:12:23.818804+0000","last_unstale":"2024-09-16T05:24:11.108064+0000","last_undegraded":"2024-09-16T05:24:11.108064+0000","last_fullsized":"2024-09-16T05:24:11.108064+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"823'378","last_scrub_stamp":"2024-09-16T05:23:09.047313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.047313+0000","objects_scrubbed":1,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:33.922269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00077312699999999997,"stat_sum":{"num_bytes":3294436,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":182,"num_read_kb":215625,"num_write":76,"num_write_kb":29696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.21","version":"178'50","reported_seq":3933,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.080200+0000","last_change":"2024-09-16T05:24:12.080457+0000","last_active":"2024-09-16T05:24:12.080200+0000","last_peered":"2024-09-16T05:24:12.080200+0000","last_clean":"2024-09-16T05:24:12.080200+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:24:12.080200+0000","last_undegraded":"2024-09-16T05:24:12.080200+0000","last_fullsized":"2024-09-16T05:24:12.080200+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:23:59.475854+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:23:59.475854+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:29.440988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079271699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.22","version":"182'20","reported_seq":1881,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.200298+0000","last_change":"2024-09-16T05:24:10.200452+0000","last_active":"2024-09-16T05:24:10.200298+0000","last_peered":"2024-09-16T05:24:10.200298+0000","last_clean":"2024-09-16T05:24:10.200298+0000","last_became_active":"2024-09-16T05:21:54.942985+0000","last_became_peered":"2024-09-16T05:21:54.942985+0000","last_unstale":"2024-09-16T05:24:10.200298+0000","last_undegraded":"2024-09-16T05:24:10.200298+0000","last_fullsized":"2024-09-16T05:24:10.200298+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:23:32.664470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:32.664470+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:46.022345+0000","scrub_duration":419,"objects_trimmed":0,"snaptrim_duration":0.00094468500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":3362,"reported_epoch":902,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:12.087025+0000","last_change":"2024-09-16T05:23:37.859916+0000","last_active":"2024-09-16T05:23:36.869822+0000","last_peered":"2024-09-16T05:24:12.087025+0000","last_clean":"2024-09-16T05:23:35.997430+0000","last_became_active":"2024-09-16T05:22:22.838582+0000","last_became_peered":"2024-09-16T05:23:37.859916+0000","last_unstale":"2024-09-16T05:24:12.087025+0000","last_undegraded":"2024-09-16T05:24:12.087025+0000","last_fullsized":"2024-09-16T05:23:37.858901+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:22:26.319237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.319237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001072094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"182'89","reported_seq":3266,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.092881+0000","last_change":"2024-09-16T05:24:10.093166+0000","last_active":"2024-09-16T05:24:10.092881+0000","last_peered":"2024-09-16T05:24:10.092881+0000","last_clean":"2024-09-16T05:24:10.092881+0000","last_became_active":"2024-09-16T05:21:54.932630+0000","last_became_peered":"2024-09-16T05:21:54.932630+0000","last_unstale":"2024-09-16T05:24:10.092881+0000","last_undegraded":"2024-09-16T05:24:10.092881+0000","last_fullsized":"2024-09-16T05:24:10.092881+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:23:08.095941+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:23:08.095941+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:16.396719+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001308144,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.25","version":"877'1381","reported_seq":5282,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104721+0000","last_change":"2024-09-16T05:24:12.081359+0000","last_active":"2024-09-16T05:24:13.104721+0000","last_peered":"2024-09-16T05:24:13.104721+0000","last_clean":"2024-09-16T05:24:13.104721+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:24:13.104721+0000","last_undegraded":"2024-09-16T05:24:13.104721+0000","last_fullsized":"2024-09-16T05:24:13.104721+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"867'1377","last_scrub_stamp":"2024-09-16T05:23:38.683553+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:23:38.683553+0000","objects_scrubbed":5,"log_size":1381,"log_dups_size":0,"ondisk_log_size":1381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:41.896930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00032111500000000002,"stat_sum":{"num_bytes":10410647,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":271,"num_read_kb":267964,"num_write":238,"num_write_kb":92646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.26","version":"116'9","reported_seq":3264,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.108946+0000","last_change":"2024-09-16T05:24:11.109101+0000","last_active":"2024-09-16T05:24:11.108946+0000","last_peered":"2024-09-16T05:24:11.108946+0000","last_clean":"2024-09-16T05:24:11.108946+0000","last_became_active":"2024-09-16T05:13:01.676093+0000","last_became_peered":"2024-09-16T05:13:01.676093+0000","last_unstale":"2024-09-16T05:24:11.108946+0000","last_undegraded":"2024-09-16T05:24:11.108946+0000","last_fullsized":"2024-09-16T05:24:11.108946+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:23:35.108726+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_clean_scrub_stamp":"2024-09-16T05:23:35.108726+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:43.758362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091979699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":3279,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.109003+0000","last_change":"2024-09-16T05:24:11.109179+0000","last_active":"2024-09-16T05:24:11.109003+0000","last_peered":"2024-09-16T05:24:11.109003+0000","last_clean":"2024-09-16T05:24:11.109003+0000","last_became_active":"2024-09-16T05:13:01.675511+0000","last_became_peered":"2024-09-16T05:13:01.675511+0000","last_unstale":"2024-09-16T05:24:11.109003+0000","last_undegraded":"2024-09-16T05:24:11.109003+0000","last_fullsized":"2024-09-16T05:24:11.109003+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:23:58.148133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:58.148133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:06.411953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093050000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.28","version":"245'105","reported_seq":2186,"reported_epoch":902,"state":"undersized+peered","last_fresh":"2024-09-16T05:24:12.079411+0000","last_change":"2024-09-16T05:23:37.855933+0000","last_active":"2024-09-16T05:23:36.854671+0000","last_peered":"2024-09-16T05:24:12.079411+0000","last_clean":"2024-09-16T05:23:36.167119+0000","last_became_active":"2024-09-16T05:21:07.818030+0000","last_became_peered":"2024-09-16T05:23:37.855933+0000","last_unstale":"2024-09-16T05:24:12.079411+0000","last_undegraded":"2024-09-16T05:24:12.079411+0000","last_fullsized":"2024-09-16T05:23:37.855095+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:22:34.641066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.641066+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016227399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.29","version":"899'1448","reported_seq":5693,"reported_epoch":902,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T05:24:12.089254+0000","last_change":"2024-09-16T05:23:40.031990+0000","last_active":"2024-09-16T05:24:12.089254+0000","last_peered":"2024-09-16T05:24:12.089254+0000","last_clean":"2024-09-16T05:24:12.089254+0000","last_became_active":"2024-09-16T05:21:57.823393+0000","last_became_peered":"2024-09-16T05:21:57.823393+0000","last_unstale":"2024-09-16T05:24:12.089254+0000","last_undegraded":"2024-09-16T05:24:12.089254+0000","last_fullsized":"2024-09-16T05:24:12.089254+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":6,"last_scrub":"795'1297","last_scrub_stamp":"2024-09-16T05:22:26.617228+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.617228+0000","objects_scrubbed":0,"log_size":1448,"log_dups_size":0,"ondisk_log_size":1448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 32s","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.002515939,"stat_sum":{"num_bytes":10955956,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":407,"num_read_kb":383264,"num_write":302,"num_write_kb":115597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":45326140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16f","length":"1"},{"start":"171","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1558,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.092127+0000","last_change":"2024-09-16T05:24:10.092438+0000","last_active":"2024-09-16T05:24:10.092127+0000","last_peered":"2024-09-16T05:24:10.092127+0000","last_clean":"2024-09-16T05:24:10.092127+0000","last_became_active":"2024-09-16T05:23:26.848398+0000","last_became_peered":"2024-09-16T05:23:26.848398+0000","last_unstale":"2024-09-16T05:24:10.092127+0000","last_undegraded":"2024-09-16T05:24:10.092127+0000","last_fullsized":"2024-09-16T05:24:10.092127+0000","mapping_epoch":856,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":857,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:24:02.175009+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_clean_scrub_stamp":"2024-09-16T05:24:02.175009+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:16.163240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001031784,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.2b","version":"868'449","reported_seq":4363,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-16T05:24:12.080612+0000","last_change":"2024-09-16T05:24:12.080872+0000","last_active":"2024-09-16T05:24:12.080612+0000","last_peered":"2024-09-16T05:24:12.080612+0000","last_clean":"2024-09-16T05:24:12.080612+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:24:12.080612+0000","last_undegraded":"2024-09-16T05:24:12.080612+0000","last_fullsized":"2024-09-16T05:24:12.080612+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"825'443","last_scrub_stamp":"2024-09-16T05:22:57.584272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:57.584272+0000","objects_scrubbed":2,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00084516300000000001,"stat_sum":{"num_bytes":2336470,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":109,"num_read_kb":92227,"num_write":79,"num_write_kb":29907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.2c","version":"241'194","reported_seq":4107,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-16T05:24:13.104740+0000","last_change":"2024-09-16T05:24:12.081785+0000","last_active":"2024-09-16T05:24:13.104740+0000","last_peered":"2024-09-16T05:24:13.104740+0000","last_clean":"2024-09-16T05:24:13.104740+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:24:13.104740+0000","last_undegraded":"2024-09-16T05:24:13.104740+0000","last_fullsized":"2024-09-16T05:24:13.104740+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:22:48.737646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:48.737646+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:16.016796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000357902,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"2"}]},{"pgid":"2.2d","version":"864'525","reported_seq":4584,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.108427+0000","last_change":"2024-09-16T05:24:11.108593+0000","last_active":"2024-09-16T05:24:11.108427+0000","last_peered":"2024-09-16T05:24:11.108427+0000","last_clean":"2024-09-16T05:24:11.108427+0000","last_became_active":"2024-09-16T05:21:54.933266+0000","last_became_peered":"2024-09-16T05:21:54.933266+0000","last_unstale":"2024-09-16T05:24:11.108427+0000","last_undegraded":"2024-09-16T05:24:11.108427+0000","last_fullsized":"2024-09-16T05:24:11.108427+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"819'522","last_scrub_stamp":"2024-09-16T05:22:54.200053+0000","last_deep_scrub":"819'522","last_deep_scrub_stamp":"2024-09-16T05:22:54.200053+0000","last_clean_scrub_stamp":"2024-09-16T05:22:54.200053+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:21.448599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000814644,"stat_sum":{"num_bytes":3798646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":95,"num_read_kb":88584,"num_write":75,"num_write_kb":37510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":10583843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.2e","version":"894'822","reported_seq":4747,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-16T05:24:11.108715+0000","last_change":"2024-09-16T05:24:11.108890+0000","last_active":"2024-09-16T05:24:11.108715+0000","last_peered":"2024-09-16T05:24:11.108715+0000","last_clean":"2024-09-16T05:24:11.108715+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:13:30.746458+0000","last_unstale":"2024-09-16T05:24:11.108715+0000","last_undegraded":"2024-09-16T05:24:11.108715+0000","last_fullsized":"2024-09-16T05:24:11.108715+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"875'821","last_scrub_stamp":"2024-09-16T05:24:00.151377+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:24:00.151377+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:14.479705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00071570399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":80564,"num_write":131,"num_write_kb":62507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"d"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]},{"pgid":"2.2f","version":"880'767","reported_seq":4793,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-16T05:24:10.093794+0000","last_change":"2024-09-16T05:24:10.094074+0000","last_active":"2024-09-16T05:24:10.093794+0000","last_peered":"2024-09-16T05:24:10.093794+0000","last_clean":"2024-09-16T05:24:10.093794+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:24:10.093794+0000","last_undegraded":"2024-09-16T05:24:10.093794+0000","last_fullsized":"2024-09-16T05:24:10.093794+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"880'767","last_scrub_stamp":"2024-09-16T05:24:04.184519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:04.184519+0000","objects_scrubbed":4,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:29.631187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017046380000000001,"stat_sum":{"num_bytes":11188641,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":166,"num_read_kb":119227,"num_write":173,"num_write_kb":60241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"b"},{"start":"17d","length":"1"},{"start":"17f","length":"1"},{"start":"182","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":96,"stat_sum":{"num_bytes":177078167,"num_objects":81,"num_object_clones":42,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":6,"num_read":3998,"num_read_kb":3957498,"num_write":2853,"num_write_kb":1175241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":234983424,"data_stored":301129540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33904,"ondisk_log_size":33904,"up":174,"acting":174,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":59,"acting":59,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089526,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69236,"kb_used_data":24152,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93675916,"statfs":{"total":95995035648,"available":95924137984,"internally_reserved":0,"allocated":24731648,"data_stored":28810919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":46135748},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285754,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79656,"kb_used_data":34580,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93665496,"statfs":{"total":95995035648,"available":95913467904,"internally_reserved":0,"allocated":35409920,"data_stored":41391014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514692,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78128,"kb_used_data":28892,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93667024,"statfs":{"total":95995035648,"available":95915032576,"internally_reserved":0,"allocated":29585408,"data_stored":29784820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841732,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81712,"kb_used_data":36632,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93663440,"statfs":{"total":95995035648,"available":95911362560,"internally_reserved":0,"allocated":37511168,"data_stored":43708117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103369,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69624,"kb_used_data":20384,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93675528,"statfs":{"total":95995035648,"available":95923740672,"internally_reserved":0,"allocated":20873216,"data_stored":21823357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397712,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78140,"kb_used_data":28900,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93667012,"statfs":{"total":95995035648,"available":95915020288,"internally_reserved":0,"allocated":29593600,"data_stored":33090788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757459,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61060,"kb_used_data":25704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684092,"statfs":{"total":95995035648,"available":95932510208,"internally_reserved":0,"allocated":26320896,"data_stored":32472336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920984,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49300,"kb_used_data":9788,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93695852,"statfs":{"total":95995035648,"available":95944552448,"internally_reserved":0,"allocated":10022912,"data_stored":7965488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117213,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78512,"kb_used_data":29268,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93666640,"statfs":{"total":95995035648,"available":95914639360,"internally_reserved":0,"allocated":29970432,"data_stored":39378702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240500,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56896,"kb_used_data":24612,"kb_used_omap":10,"kb_used_meta":32245,"kb_avail":93688256,"statfs":{"total":95995035648,"available":95936774144,"internally_reserved":0,"allocated":25202688,"data_stored":26018055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":33019187},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81388,"kb_used_data":23492,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93663764,"statfs":{"total":95995035648,"available":95911694336,"internally_reserved":0,"allocated":24055808,"data_stored":26781009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738604,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80724,"kb_used_data":23160,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93664428,"statfs":{"total":95995035648,"available":95912374272,"internally_reserved":0,"allocated":23715840,"data_stored":23271797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18030592,"data_stored":22204895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":19995522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18255872,"data_stored":21124060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22962176,"data_stored":34419715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3366912,"data_stored":3351157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19460096,"data_stored":27656757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23957504,"data_stored":29964758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14217216,"data_stored":17209026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":30818304,"data_stored":39068378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":22929408,"data_stored":25170489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":28753920,"data_stored":36776683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18063360,"data_stored":24188100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:24:15.692 INFO:tasks.thrashosds.thrasher:Setting 2.40 to [3, 5, 2, 0] 2024-09-16T05:24:15.692 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.40', '3', '5', '2', '0'] 2024-09-16T05:24:15.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 2.40 3 5 2 0 2024-09-16T05:24:15.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:15.755 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:15.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:15.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:15.880 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:15.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:15.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:16.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:16.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:16.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:16.117 INFO:tasks.rados.rados.2.smithi204.stdout:3585: done (11 left) 2024-09-16T05:24:16.117 INFO:tasks.rados.rados.2.smithi204.stdout:3586: done (10 left) 2024-09-16T05:24:16.118 INFO:tasks.rados.rados.2.smithi204.stdout:3587: done (9 left) 2024-09-16T05:24:16.118 INFO:tasks.rados.rados.2.smithi204.stdout:3588: snap_remove snap 380 2024-09-16T05:24:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:16.135 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:16.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:16.230 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:16.230 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:16.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: pgmap v1518: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 169 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 562 KiB/s wr, 5 op/s; 9/166 objects degraded (5.422%) 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: from='client.17475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: 2.58 scrub starts 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: osdmap e905: 12 total, 11 up, 12 in 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: 2.58 scrub ok 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:24:16.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:24:16.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:16.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:16 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/166 objects degraded (5.422%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:16.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:16.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:16.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:16.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: pgmap v1518: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 169 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 562 KiB/s wr, 5 op/s; 9/166 objects degraded (5.422%) 2024-09-16T05:24:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:24:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:24:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: from='client.17475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:24:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: 2.58 scrub starts 2024-09-16T05:24:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: osdmap e905: 12 total, 11 up, 12 in 2024-09-16T05:24:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: 2.58 scrub ok 2024-09-16T05:24:16.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:24:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:24:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:16.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:16 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/166 objects degraded (5.422%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: pgmap v1518: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 169 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 562 KiB/s wr, 5 op/s; 9/166 objects degraded (5.422%) 2024-09-16T05:24:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:24:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:24:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: from='client.17475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:24:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: 2.58 scrub starts 2024-09-16T05:24:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: osdmap e905: 12 total, 11 up, 12 in 2024-09-16T05:24:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: 2.58 scrub ok 2024-09-16T05:24:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:24:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:24:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:16.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:16 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/166 objects degraded (5.422%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:16.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:16.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:16.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:16.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:16.640 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:16.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:16.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:16.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:16.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:16.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:16.893 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:16.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:16.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:16.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:17.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:24:17.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:17.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:17.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:24:17.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:17.135 INFO:tasks.rados.rados.2.smithi204.stdout:3588: done (9 left) 2024-09-16T05:24:17.135 INFO:tasks.rados.rados.2.smithi204.stdout:3589: read oid 23 snap -1 2024-09-16T05:24:17.135 INFO:tasks.rados.rados.2.smithi204.stdout:3589: expect (ObjNum 1127 snap 338 seq_num 1127) 2024-09-16T05:24:17.135 INFO:tasks.rados.rados.2.smithi204.stdout:3590: read oid 6 snap -1 2024-09-16T05:24:17.135 INFO:tasks.rados.rados.2.smithi204.stdout:3590: expect deleted 2024-09-16T05:24:17.135 INFO:tasks.rados.rados.2.smithi204.stdout:3591: snap_create 2024-09-16T05:24:17.150 INFO:tasks.rados.rados.2.smithi204.stdout:3589: expect (ObjNum 1127 snap 338 seq_num 1127) 2024-09-16T05:24:17.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:17.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:17.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:17.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:17.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:17.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:17.397 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: 2.20 scrub starts 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: 2.20 scrub ok 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: osdmap e906: 12 total, 11 up, 12 in 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4214285291,config=1549860826,health=313164883,kv=3469612755}) 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=4091229503}) 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3292764098}) 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {kv=35,logm=65} crc {kv=2958136907,logm=3570422597}) 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069279514}) 2024-09-16T05:24:17.398 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737462664}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826588394}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293221009}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2577214628}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190728615}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013944445}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=550875084}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=645137360}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358674015}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1276691859}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919067420}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575810008}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746125598}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265114355}) 2024-09-16T05:24:17.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077188411}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2801117152}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3797997261}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1059271961}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107305700}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926078961}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875612869}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287595075}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485965889}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006734193}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575852771}) 2024-09-16T05:24:17.400 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078080412}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731504275}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3275633595}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1670607148}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1940029818}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3461414934}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461765482}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746743939}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=176922894}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559727245}) 2024-09-16T05:24:17.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3701121714}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264978355}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664455184}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982269345}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1582802485}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373971873}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103499395}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2334916101}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3157056760}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1601509581}) 2024-09-16T05:24:17.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822337467}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867892628}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419874711}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648300081}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867102214}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503672508}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066589031}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3928621446}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1849185574}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591175041}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1292368408}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959555761}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=382339750}) 2024-09-16T05:24:17.403 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371977215}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284580923}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1678866005}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=212869024}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186329187}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=813428198}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=140092798}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1202523798}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=8583296}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=518623206}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125804185}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964825883}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432284051}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4184300709}) 2024-09-16T05:24:17.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796374473}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584828361}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1630494770}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3681138041}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479839224}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=944341976}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=76,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2} crc {logm=1498022748,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=1828394773}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=8,mon_config_key=92} crc {mgrstat=1913296672,mon_config_key=3519465424}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2900874311}) 2024-09-16T05:24:17.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=1} crc {mon_config_key=2052115435,monmap=2256977890,osd_metadata=3570072447,osd_pg_creating=2565371551,osd_snap=950980780}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3747573299}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=42,osdmap=58} crc {osd_snap=2071035347,osdmap=2397234803}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4191861204}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2571156600}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3212452493}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3938650895}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4131055616}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2546093849}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3405861829}) 2024-09-16T05:24:17.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4207387719}) 2024-09-16T05:24:17.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3930693272}) 2024-09-16T05:24:17.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2313572337}) 2024-09-16T05:24:17.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2669169375}) 2024-09-16T05:24:17.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:24:17.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:24:17.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: 2.20 scrub starts 2024-09-16T05:24:17.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: 2.20 scrub ok 2024-09-16T05:24:17.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:24:17.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:24:17.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: osdmap e906: 12 total, 11 up, 12 in 2024-09-16T05:24:17.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4214285291,config=1549860826,health=313164883,kv=3469612755}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=4091229503}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3292764098}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {kv=35,logm=65} crc {kv=2958136907,logm=3570422597}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069279514}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737462664}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826588394}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293221009}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2577214628}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190728615}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013944445}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=550875084}) 2024-09-16T05:24:17.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=645137360}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358674015}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1276691859}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919067420}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575810008}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746125598}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265114355}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077188411}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2801117152}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3797997261}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1059271961}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107305700}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926078961}) 2024-09-16T05:24:17.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875612869}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287595075}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485965889}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006734193}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575852771}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078080412}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731504275}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3275633595}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1670607148}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1940029818}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3461414934}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461765482}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746743939}) 2024-09-16T05:24:17.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=176922894}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559727245}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3701121714}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264978355}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664455184}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982269345}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1582802485}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373971873}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103499395}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2334916101}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3157056760}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1601509581}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822337467}) 2024-09-16T05:24:17.662 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867892628}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419874711}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648300081}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867102214}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503672508}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066589031}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3928621446}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1849185574}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591175041}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1292368408}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959555761}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=382339750}) 2024-09-16T05:24:17.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371977215}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284580923}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1678866005}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=212869024}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186329187}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=813428198}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=140092798}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1202523798}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=8583296}) 2024-09-16T05:24:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=518623206}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125804185}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964825883}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432284051}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4184300709}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796374473}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584828361}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1630494770}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3681138041}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479839224}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=944341976}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=76,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2} crc {logm=1498022748,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=1828394773}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=8,mon_config_key=92} crc {mgrstat=1913296672,mon_config_key=3519465424}) 2024-09-16T05:24:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2900874311}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=1} crc {mon_config_key=2052115435,monmap=2256977890,osd_metadata=3570072447,osd_pg_creating=2565371551,osd_snap=950980780}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3747573299}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=42,osdmap=58} crc {osd_snap=2071035347,osdmap=2397234803}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4191861204}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2571156600}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3212452493}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3938650895}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4131055616}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2546093849}) 2024-09-16T05:24:17.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3405861829}) 2024-09-16T05:24:17.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4207387719}) 2024-09-16T05:24:17.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3930693272}) 2024-09-16T05:24:17.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2313572337}) 2024-09-16T05:24:17.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2669169375}) 2024-09-16T05:24:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:17.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:17.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:17.668 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:24:17.668 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: 2.20 scrub starts 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: 2.20 scrub ok 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: osdmap e906: 12 total, 11 up, 12 in 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4214285291,config=1549860826,health=313164883,kv=3469612755}) 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=4091229503}) 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=100} crc {kv=3292764098}) 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {kv=35,logm=65} crc {kv=2958136907,logm=3570422597}) 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069279514}) 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=737462664}) 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826588394}) 2024-09-16T05:24:17.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293221009}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2577214628}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190728615}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013944445}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=550875084}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=645137360}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358674015}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1276691859}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919067420}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575810008}) 2024-09-16T05:24:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746125598}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265114355}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4077188411}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2801117152}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3797997261}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1059271961}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2107305700}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926078961}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875612869}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287595075}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485965889}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006734193}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575852771}) 2024-09-16T05:24:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078080412}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731504275}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3275633595}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1670607148}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1940029818}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3461414934}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461765482}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746743939}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=176922894}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559727245}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3701121714}) 2024-09-16T05:24:17.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264978355}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664455184}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982269345}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1582802485}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373971873}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103499395}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2334916101}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3157056760}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1601509581}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822337467}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867892628}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419874711}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648300081}) 2024-09-16T05:24:17.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867102214}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503672508}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066589031}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3928621446}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1849185574}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591175041}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1292368408}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959555761}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=382339750}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371977215}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284580923}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1678866005}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=212869024}) 2024-09-16T05:24:17.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186329187}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=813428198}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=140092798}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1202523798}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=8583296}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=518623206}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125804185}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964825883}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432284051}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4184300709}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796374473}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584828361}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1630494770}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3681138041}) 2024-09-16T05:24:17.675 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3479839224}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=944341976}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=76,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2} crc {logm=1498022748,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=1828394773}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=8,mon_config_key=92} crc {mgrstat=1913296672,mon_config_key=3519465424}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2900874311}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=1} crc {mon_config_key=2052115435,monmap=2256977890,osd_metadata=3570072447,osd_pg_creating=2565371551,osd_snap=950980780}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=3747573299}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=42,osdmap=58} crc {osd_snap=2071035347,osdmap=2397234803}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4191861204}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2571156600}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3212452493}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3938650895}) 2024-09-16T05:24:17.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4131055616}) 2024-09-16T05:24:17.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2546093849}) 2024-09-16T05:24:17.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3405861829}) 2024-09-16T05:24:17.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4207387719}) 2024-09-16T05:24:17.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3930693272}) 2024-09-16T05:24:17.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2313572337}) 2024-09-16T05:24:17.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=2669169375}) 2024-09-16T05:24:17.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:17.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:17.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:17.826 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:17.848 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:17.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:17.869 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:17.870 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:17.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:17.893 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:17.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:17.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:17.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:17.916 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:17.916 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:17.937 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:17.937 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:17.959 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:17.959 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:17.986 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:17.986 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:18.007 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:18.008 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:18.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:18.029 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:18.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:18.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:18.055 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:18.056 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:18.136 INFO:tasks.rados.rados.2.smithi204.stdout:3589: done (11 left) 2024-09-16T05:24:18.136 INFO:tasks.rados.rados.2.smithi204.stdout:3590: done (10 left) 2024-09-16T05:24:18.136 INFO:tasks.rados.rados.2.smithi204.stdout:3591: done (9 left) 2024-09-16T05:24:18.136 INFO:tasks.rados.rados.2.smithi204.stdout:3592: write oid 4 current snap is 389 2024-09-16T05:24:18.136 INFO:tasks.rados.rados.2.smithi204.stdout:3592: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2024-09-16T05:24:18.142 INFO:tasks.rados.rados.2.smithi204.stdout:3592: writing smithi20457400-4 from 457487 to 1207035 tid 1 2024-09-16T05:24:18.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:18.148 INFO:tasks.rados.rados.2.smithi204.stdout:3592: writing smithi20457400-4 from 1770389 to 2254824 tid 2 2024-09-16T05:24:18.152 INFO:tasks.rados.rados.2.smithi204.stdout:3592: writing smithi20457400-4 from 2746287 to 2857487 tid 3 2024-09-16T05:24:18.152 INFO:tasks.rados.rados.2.smithi204.stdout:3593: read oid 15 snap -1 2024-09-16T05:24:18.152 INFO:tasks.rados.rados.2.smithi204.stdout:3593: expect deleted 2024-09-16T05:24:18.152 INFO:tasks.rados.rados.2.smithi204.stdout:3594: read oid 9 snap -1 2024-09-16T05:24:18.152 INFO:tasks.rados.rados.2.smithi204.stdout:3594: expect deleted 2024-09-16T05:24:18.152 INFO:tasks.rados.rados.2.smithi204.stdout:3595: delete oid 19 current snap is 389 2024-09-16T05:24:18.152 INFO:tasks.rados.rados.2.smithi204.stdout:3592: finishing write tid 1 to smithi20457400-4 2024-09-16T05:24:18.152 INFO:tasks.rados.rados.2.smithi204.stdout:3592: finishing write tid 2 to smithi20457400-4 2024-09-16T05:24:18.154 INFO:tasks.rados.rados.2.smithi204.stdout:3592: finishing write tid 3 to smithi20457400-4 2024-09-16T05:24:18.154 INFO:tasks.rados.rados.2.smithi204.stdout:3592: finishing write tid 4 to smithi20457400-4 2024-09-16T05:24:18.154 INFO:tasks.rados.rados.2.smithi204.stdout:3593: done (12 left) 2024-09-16T05:24:18.154 INFO:tasks.rados.rados.2.smithi204.stdout:3594: done (11 left) 2024-09-16T05:24:18.154 INFO:tasks.rados.rados.2.smithi204.stdout:3595: done (10 left) 2024-09-16T05:24:18.154 INFO:tasks.rados.rados.2.smithi204.stdout:3596: write (excl) oid 17 current snap is 389 2024-09-16T05:24:18.154 INFO:tasks.rados.rados.2.smithi204.stdout:3596: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2024-09-16T05:24:18.159 INFO:tasks.rados.rados.2.smithi204.stdout:3596: writing smithi20457400-17 from 491101 to 991147 tid 1 2024-09-16T05:24:18.165 INFO:tasks.rados.rados.2.smithi204.stdout:3596: writing smithi20457400-17 from 1487353 to 1975457 tid 2 2024-09-16T05:24:18.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:18.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3596: writing smithi20457400-17 from 2401133 to 2891101 tid 3 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3597: rollback oid 15 current snap is 389 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 15 to 358 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3592: finishing write tid 5 to smithi20457400-4 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1361 (ObjNum 1241 snap 389 seq_num 1241) dirty exists 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3592: left oid 4 (ObjNum 1241 snap 389 seq_num 1241) 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3596: finishing write tid 1 to smithi20457400-17 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3592: done (11 left) 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3598: rollback oid 35 current snap is 389 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 35 to 388 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3596: finishing write tid 2 to smithi20457400-17 2024-09-16T05:24:18.171 INFO:tasks.rados.rados.2.smithi204.stdout:3599: snap_create 2024-09-16T05:24:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:3597: finishing rollback tid 1 to smithi20457400-15 2024-09-16T05:24:18.172 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 49 snap 0 seq_num 10) dirty dne 2024-09-16T05:24:18.173 INFO:tasks.rados.rados.2.smithi204.stdout:3596: finishing write tid 3 to smithi20457400-17 2024-09-16T05:24:18.173 INFO:tasks.rados.rados.2.smithi204.stdout:3596: finishing write tid 4 to smithi20457400-17 2024-09-16T05:24:18.174 INFO:tasks.rados.rados.2.smithi204.stdout:3598: finishing rollback tid 0 to smithi20457400-35 2024-09-16T05:24:18.174 INFO:tasks.rados.rados.2.smithi204.stdout:3596: finishing write tid 5 to smithi20457400-17 2024-09-16T05:24:18.174 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1246 (ObjNum 1243 snap 389 seq_num 1243) dirty exists 2024-09-16T05:24:18.174 INFO:tasks.rados.rados.2.smithi204.stdout:3596: left oid 17 (ObjNum 1243 snap 389 seq_num 1243) 2024-09-16T05:24:18.174 INFO:tasks.rados.rados.2.smithi204.stdout:3598: finishing rollback tid 1 to smithi20457400-35 2024-09-16T05:24:18.174 INFO:tasks.rados.rados.2.smithi204.stdout:3598: finishing rollback tid 2 to smithi20457400-35 2024-09-16T05:24:18.174 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 1274 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-09-16T05:24:18.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:18.268 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:18.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:18.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: pgmap v1521: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 170 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 830 KiB/s wr, 10 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.4e scrub starts 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.4e scrub ok 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:24:18.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:24:18.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:24:18.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:24:18.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: osdmap e907: 12 total, 11 up, 12 in 2024-09-16T05:24:18.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.4d scrub starts 2024-09-16T05:24:18.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: 2.4d scrub ok 2024-09-16T05:24:18.393 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/486567764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:18.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: pgmap v1521: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 170 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 830 KiB/s wr, 10 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.4e scrub starts 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.4e scrub ok 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:24:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: osdmap e907: 12 total, 11 up, 12 in 2024-09-16T05:24:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.4d scrub starts 2024-09-16T05:24:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: 2.4d scrub ok 2024-09-16T05:24:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/486567764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:18.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:18.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:18.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: pgmap v1521: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 170 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 830 KiB/s wr, 10 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.4e scrub starts 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.4e scrub ok 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:24:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:24:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: osdmap e907: 12 total, 11 up, 12 in 2024-09-16T05:24:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.4d scrub starts 2024-09-16T05:24:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: 2.4d scrub ok 2024-09-16T05:24:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/486567764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:18.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:18.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:18.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:18.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:18.672 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:18.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:18.775 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:18.796 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:18.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:18.898 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:18.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:18.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:19.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:19.042 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:19.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:19.078 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:19.144 INFO:teuthology.orchestra.run.smithi032.stderr:set 2.40 pg_upmap_items mapping to [3->5,2->0] 2024-09-16T05:24:19.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:19.148 INFO:tasks.rados.rados.2.smithi204.stdout:3596: done (12 left) 2024-09-16T05:24:19.149 INFO:tasks.rados.rados.2.smithi204.stdout:3597: done (11 left) 2024-09-16T05:24:19.149 INFO:tasks.rados.rados.2.smithi204.stdout:3598: done (10 left) 2024-09-16T05:24:19.149 INFO:tasks.rados.rados.2.smithi204.stdout:3599: done (9 left) 2024-09-16T05:24:19.149 INFO:tasks.rados.rados.2.smithi204.stdout:3600: write oid 44 current snap is 390 2024-09-16T05:24:19.149 INFO:tasks.rados.rados.2.smithi204.stdout:3600: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-09-16T05:24:19.154 INFO:tasks.rados.rados.2.smithi204.stdout:3600: writing smithi20457400-44 from 507908 to 1083203 tid 1 2024-09-16T05:24:19.162 INFO:tasks.rados.rados.2.smithi204.stdout:3600: writing smithi20457400-44 from 1829482 to 2461244 tid 2 2024-09-16T05:24:19.162 INFO:tasks.rados.rados.2.smithi204.stdout:3601: rollback oid 50 current snap is 390 2024-09-16T05:24:19.162 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 50 to 389 2024-09-16T05:24:19.162 INFO:tasks.rados.rados.2.smithi204.stdout:3600: finishing write tid 1 to smithi20457400-44 2024-09-16T05:24:19.163 INFO:tasks.rados.rados.2.smithi204.stdout:3602: rollback oid 18 current snap is 390 2024-09-16T05:24:19.163 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 18 to 386 2024-09-16T05:24:19.163 INFO:tasks.rados.rados.2.smithi204.stdout:3603: read oid 35 snap 389 2024-09-16T05:24:19.163 INFO:tasks.rados.rados.2.smithi204.stdout:3603: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:24:19.163 INFO:tasks.rados.rados.2.smithi204.stdout:3604: read oid 25 snap -1 2024-09-16T05:24:19.163 INFO:tasks.rados.rados.2.smithi204.stdout:3604: expect (ObjNum 1211 snap 366 seq_num 1211) 2024-09-16T05:24:19.163 INFO:tasks.rados.rados.2.smithi204.stdout:3605: write (excl) oid 28 current snap is 390 2024-09-16T05:24:19.163 INFO:tasks.rados.rados.2.smithi204.stdout:3605: seq_num 1245 ranges {524715=650544,1887964=491773} 2024-09-16T05:24:19.169 INFO:tasks.rados.rados.2.smithi204.stdout:3605: writing smithi20457400-28 from 524715 to 1175259 tid 1 2024-09-16T05:24:19.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:19.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:19.177 INFO:tasks.rados.rados.2.smithi204.stdout:3605: writing smithi20457400-28 from 1887964 to 2379737 tid 2 2024-09-16T05:24:19.177 INFO:tasks.rados.rados.2.smithi204.stdout:3606: read oid 36 snap 388 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3606: expect deleted 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3600: finishing write tid 2 to smithi20457400-44 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3600: finishing write tid 3 to smithi20457400-44 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3600: finishing write tid 4 to smithi20457400-44 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1148 (ObjNum 1244 snap 390 seq_num 1244) dirty exists 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3600: left oid 44 (ObjNum 1244 snap 390 seq_num 1244) 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3602: finishing rollback tid 0 to smithi20457400-18 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3600: done (15 left) 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3607: snap_create 2024-09-16T05:24:19.178 INFO:tasks.rados.rados.2.smithi204.stdout:3601: finishing rollback tid 0 to smithi20457400-50 2024-09-16T05:24:19.179 INFO:tasks.rados.rados.2.smithi204.stdout:3602: finishing rollback tid 1 to smithi20457400-18 2024-09-16T05:24:19.179 INFO:tasks.rados.rados.2.smithi204.stdout:3603: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-09-16T05:24:19.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:18 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:18.897+0000 7f6af1ad6640 -1 osd.6 908 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: 2.3b scrub starts 2024-09-16T05:24:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: 2.3b scrub ok 2024-09-16T05:24:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:24:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:24:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:24:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:24:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: osdmap e908: 12 total, 11 up, 12 in 2024-09-16T05:24:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3569912574' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.40", "id": [3, 5, 2, 0]}]: dispatch 2024-09-16T05:24:19.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:24:19.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:24:19.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:19.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:19.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:19.378 INFO:tasks.rados.rados.2.smithi204.stdout:3601: finishing rollback tid 1 to smithi20457400-50 2024-09-16T05:24:19.379 INFO:tasks.rados.rados.2.smithi204.stdout:3601: finishing rollback tid 2 to smithi20457400-50 2024-09-16T05:24:19.379 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1176 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:24:19.379 INFO:tasks.rados.rados.2.smithi204.stdout:3604: expect (ObjNum 1211 snap 366 seq_num 1211) 2024-09-16T05:24:19.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:19.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: 2.3b scrub starts 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: 2.3b scrub ok 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: osdmap e908: 12 total, 11 up, 12 in 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3569912574' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.40", "id": [3, 5, 2, 0]}]: dispatch 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:24:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:19 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: 2.3b scrub starts 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: 2.3b scrub ok 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: osdmap e908: 12 total, 11 up, 12 in 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3569912574' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.40", "id": [3, 5, 2, 0]}]: dispatch 2024-09-16T05:24:19.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:24:19.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:19 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:24:19.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:19.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:19.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:19.559 INFO:tasks.rados.rados.2.smithi204.stdout:3605: finishing write tid 1 to smithi20457400-28 2024-09-16T05:24:19.559 INFO:tasks.rados.rados.2.smithi204.stdout:3605: finishing write tid 2 to smithi20457400-28 2024-09-16T05:24:19.559 INFO:tasks.rados.rados.2.smithi204.stdout:3605: finishing write tid 3 to smithi20457400-28 2024-09-16T05:24:19.559 INFO:tasks.rados.rados.2.smithi204.stdout:3605: finishing write tid 4 to smithi20457400-28 2024-09-16T05:24:19.559 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1249 (ObjNum 1245 snap 390 seq_num 1245) dirty exists 2024-09-16T05:24:19.560 INFO:tasks.rados.rados.2.smithi204.stdout:3605: left oid 28 (ObjNum 1245 snap 390 seq_num 1245) 2024-09-16T05:24:19.560 INFO:tasks.rados.rados.2.smithi204.stdout:3602: finishing rollback tid 2 to smithi20457400-18 2024-09-16T05:24:19.560 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 1253 (ObjNum 1222 snap 372 seq_num 1222) dirty exists 2024-09-16T05:24:19.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:19.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:19.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:19.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:19.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:19.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:19.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:19.920 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:19.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:20.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:20.077 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:20.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:20.150 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:24:20.154 INFO:tasks.rados.rados.2.smithi204.stdout:3601: done (15 left) 2024-09-16T05:24:20.154 INFO:tasks.rados.rados.2.smithi204.stdout:3602: done (14 left) 2024-09-16T05:24:20.154 INFO:tasks.rados.rados.2.smithi204.stdout:3603: done (13 left) 2024-09-16T05:24:20.154 INFO:tasks.rados.rados.2.smithi204.stdout:3604: done (12 left) 2024-09-16T05:24:20.154 INFO:tasks.rados.rados.2.smithi204.stdout:3605: done (11 left) 2024-09-16T05:24:20.154 INFO:tasks.rados.rados.2.smithi204.stdout:3606: done (10 left) 2024-09-16T05:24:20.155 INFO:tasks.rados.rados.2.smithi204.stdout:3607: done (9 left) 2024-09-16T05:24:20.155 INFO:tasks.rados.rados.2.smithi204.stdout:3608: snap_remove snap 381 2024-09-16T05:24:20.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:20.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:20.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:20.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:19 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:19.919+0000 7f6af1ad6640 -1 osd.6 909 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: pgmap v1524: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 170 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 268 KiB/s wr, 9 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:20.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 2.43 scrub starts 2024-09-16T05:24:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: 2.43 scrub ok 2024-09-16T05:24:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3569912574' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.40", "id": [3, 5, 2, 0]}]': finished 2024-09-16T05:24:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: osdmap e909: 12 total, 11 up, 12 in 2024-09-16T05:24:20.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1112784437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:20.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: pgmap v1524: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 170 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 268 KiB/s wr, 9 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 2.43 scrub starts 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: 2.43 scrub ok 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3569912574' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.40", "id": [3, 5, 2, 0]}]': finished 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: osdmap e909: 12 total, 11 up, 12 in 2024-09-16T05:24:20.245 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1112784437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:20.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:20.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:20.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:20.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: pgmap v1524: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 170 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 268 KiB/s wr, 9 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 2.43 scrub starts 2024-09-16T05:24:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: 2.43 scrub ok 2024-09-16T05:24:20.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3569912574' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.40", "id": [3, 5, 2, 0]}]': finished 2024-09-16T05:24:20.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: osdmap e909: 12 total, 11 up, 12 in 2024-09-16T05:24:20.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1112784437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:20.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:20.458 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:20.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:20.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:20.582 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:20.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:20.663 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:24:20.663 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:24:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:24:20.664 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:24:20.664 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:24:20.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:20.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:20.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:20.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:20.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:20.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:20.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:20.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:20.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:21.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:21.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:24:21.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:21.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:21.158 INFO:tasks.rados.rados.2.smithi204.stdout:3608: done (9 left) 2024-09-16T05:24:21.158 INFO:tasks.rados.rados.2.smithi204.stdout:3609: read oid 15 snap -1 2024-09-16T05:24:21.158 INFO:tasks.rados.rados.2.smithi204.stdout:3609: expect deleted 2024-09-16T05:24:21.158 INFO:tasks.rados.rados.2.smithi204.stdout:3610: copy_from oid 50 from oid 9 current snap is 391 2024-09-16T05:24:21.158 INFO:tasks.rados.rados.2.smithi204.stdout:3611: read oid 32 snap -1 2024-09-16T05:24:21.159 INFO:tasks.rados.rados.2.smithi204.stdout:3611: expect (ObjNum 1222 snap 372 seq_num 1222) 2024-09-16T05:24:21.159 INFO:tasks.rados.rados.2.smithi204.stdout:3612: delete oid 1 current snap is 391 2024-09-16T05:24:21.161 INFO:tasks.rados.rados.2.smithi204.stdout:3610: finishing copy_from to smithi20457400-50 2024-09-16T05:24:21.162 INFO:tasks.rados.rados.2.smithi204.stdout:3610: got expected ENOENT (src dne) 2024-09-16T05:24:21.162 INFO:tasks.rados.rados.2.smithi204.stdout:3610: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:24:21.163 INFO:tasks.rados.rados.2.smithi204.stdout:3609: done (12 left) 2024-09-16T05:24:21.163 INFO:tasks.rados.rados.2.smithi204.stdout:3610: done (11 left) 2024-09-16T05:24:21.163 INFO:tasks.rados.rados.2.smithi204.stdout:3612: done (10 left) 2024-09-16T05:24:21.163 INFO:tasks.rados.rados.2.smithi204.stdout:3613: write oid 19 current snap is 391 2024-09-16T05:24:21.164 INFO:tasks.rados.rados.2.smithi204.stdout:3613: seq_num 1247 ranges {558329=517395,1721281=779089} 2024-09-16T05:24:21.169 INFO:tasks.rados.rados.2.smithi204.stdout:3613: writing smithi20457400-19 from 558329 to 1075724 tid 1 2024-09-16T05:24:21.179 INFO:tasks.rados.rados.2.smithi204.stdout:3613: writing smithi20457400-19 from 1721281 to 2500370 tid 2 2024-09-16T05:24:21.179 INFO:tasks.rados.rados.2.smithi204.stdout:3614: delete oid 8 current snap is 391 2024-09-16T05:24:21.179 INFO:tasks.rados.rados.2.smithi204.stdout:3611: expect (ObjNum 1222 snap 372 seq_num 1222) 2024-09-16T05:24:21.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:21.213 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:21.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:21.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:20 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:20.948+0000 7f6af1ad6640 -1 osd.6 910 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:21 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:21 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1112784437' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:21 smithi196 ceph-mon[30802]: osdmap e910: 12 total, 11 up, 12 in 2024-09-16T05:24:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:21 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/170 objects degraded (5.294%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:21.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:21.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:21.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:21.361 INFO:tasks.rados.rados.2.smithi204.stdout:3613: finishing write tid 1 to smithi20457400-19 2024-09-16T05:24:21.361 INFO:tasks.rados.rados.2.smithi204.stdout:3613: finishing write tid 2 to smithi20457400-19 2024-09-16T05:24:21.361 INFO:tasks.rados.rados.2.smithi204.stdout:3613: finishing write tid 3 to smithi20457400-19 2024-09-16T05:24:21.361 INFO:tasks.rados.rados.2.smithi204.stdout:3611: done (11 left) 2024-09-16T05:24:21.361 INFO:tasks.rados.rados.2.smithi204.stdout:3614: done (10 left) 2024-09-16T05:24:21.361 INFO:tasks.rados.rados.2.smithi204.stdout:3615: write (excl) oid 28 current snap is 391 2024-09-16T05:24:21.362 INFO:tasks.rados.rados.2.smithi204.stdout:3613: finishing write tid 4 to smithi20457400-19 2024-09-16T05:24:21.362 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 1210 (ObjNum 1247 snap 391 seq_num 1247) dirty exists 2024-09-16T05:24:21.362 INFO:tasks.rados.rados.2.smithi204.stdout:3613: left oid 19 (ObjNum 1247 snap 391 seq_num 1247) 2024-09-16T05:24:21.362 INFO:tasks.rados.rados.2.smithi204.stdout:3615: seq_num 1249 ranges {591943=667893,1721892=782758} 2024-09-16T05:24:21.367 INFO:tasks.rados.rados.2.smithi204.stdout:3615: writing smithi20457400-28 from 591943 to 1259836 tid 1 2024-09-16T05:24:21.374 INFO:tasks.rados.rados.2.smithi204.stdout:3615: writing smithi20457400-28 from 1721892 to 2504650 tid 2 2024-09-16T05:24:21.375 INFO:tasks.rados.rados.2.smithi204.stdout:3613: done (10 left) 2024-09-16T05:24:21.375 INFO:tasks.rados.rados.2.smithi204.stdout:3616: write oid 35 current snap is 391 2024-09-16T05:24:21.375 INFO:tasks.rados.rados.2.smithi204.stdout:3616: seq_num 1250 ranges {608750=743142,1780374=642769} 2024-09-16T05:24:21.382 INFO:tasks.rados.rados.2.smithi204.stdout:3616: writing smithi20457400-35 from 608750 to 1351892 tid 1 2024-09-16T05:24:21.388 INFO:tasks.rados.rados.2.smithi204.stdout:3616: writing smithi20457400-35 from 1780374 to 2423143 tid 2 2024-09-16T05:24:21.388 INFO:tasks.rados.rados.2.smithi204.stdout:3617: copy_from oid 29 from oid 3 current snap is 391 2024-09-16T05:24:21.388 INFO:tasks.rados.rados.2.smithi204.stdout:3615: finishing write tid 1 to smithi20457400-28 2024-09-16T05:24:21.389 INFO:tasks.rados.rados.2.smithi204.stdout:3615: finishing write tid 2 to smithi20457400-28 2024-09-16T05:24:21.389 INFO:tasks.rados.rados.2.smithi204.stdout:3615: finishing write tid 3 to smithi20457400-28 2024-09-16T05:24:21.389 INFO:tasks.rados.rados.2.smithi204.stdout:3615: finishing write tid 4 to smithi20457400-28 2024-09-16T05:24:21.389 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1252 (ObjNum 1249 snap 391 seq_num 1249) dirty exists 2024-09-16T05:24:21.389 INFO:tasks.rados.rados.2.smithi204.stdout:3615: left oid 28 (ObjNum 1249 snap 391 seq_num 1249) 2024-09-16T05:24:21.389 INFO:tasks.rados.rados.2.smithi204.stdout:3616: finishing write tid 1 to smithi20457400-35 2024-09-16T05:24:21.389 INFO:tasks.rados.rados.2.smithi204.stdout:3615: done (11 left) 2024-09-16T05:24:21.389 INFO:tasks.rados.rados.2.smithi204.stdout:3618: snap_remove snap 389 2024-09-16T05:24:21.390 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:21 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:21.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:21 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:21.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1112784437' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:21.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:21 smithi032 ceph-mon[27388]: osdmap e910: 12 total, 11 up, 12 in 2024-09-16T05:24:21.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:21 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/170 objects degraded (5.294%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:21.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:24:21.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:21 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:21 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1112784437' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:21 smithi204 ceph-mon[30933]: osdmap e910: 12 total, 11 up, 12 in 2024-09-16T05:24:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:21 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/170 objects degraded (5.294%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:21.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:21.487 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:21.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:21.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:21.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:21.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:21.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:21.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:21.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:21.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:21.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:21.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:21.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:21.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:22.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:22.113 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:22.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:22.163 INFO:tasks.rados.rados.2.smithi204.stdout:3618: done (11 left) 2024-09-16T05:24:22.163 INFO:tasks.rados.rados.2.smithi204.stdout:3619: snap_create 2024-09-16T05:24:22.163 INFO:tasks.rados.rados.2.smithi204.stdout:3616: finishing write tid 2 to smithi20457400-35 2024-09-16T05:24:22.163 INFO:tasks.rados.rados.2.smithi204.stdout:3616: finishing write tid 3 to smithi20457400-35 2024-09-16T05:24:22.163 INFO:tasks.rados.rados.2.smithi204.stdout:3616: finishing write tid 4 to smithi20457400-35 2024-09-16T05:24:22.164 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 1277 (ObjNum 1250 snap 391 seq_num 1250) dirty exists 2024-09-16T05:24:22.164 INFO:tasks.rados.rados.2.smithi204.stdout:3616: left oid 35 (ObjNum 1250 snap 391 seq_num 1250) 2024-09-16T05:24:22.164 INFO:tasks.rados.rados.2.smithi204.stdout:3617: finishing copy_from racing read to smithi20457400-29 2024-09-16T05:24:22.164 INFO:tasks.rados.rados.2.smithi204.stdout:3617: finishing copy_from to smithi20457400-29 2024-09-16T05:24:22.164 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1102 (ObjNum 1236 snap 384 seq_num 1236) dirty exists 2024-09-16T05:24:22.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:22.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:21 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:21.947+0000 7f6af1ad6640 -1 osd.6 911 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: pgmap v1527: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 170 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: 2.57 scrub starts 2024-09-16T05:24:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: 2.57 scrub ok 2024-09-16T05:24:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: osdmap e911: 12 total, 11 up, 12 in 2024-09-16T05:24:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:24:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:24:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:24:22.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:24:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:22.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:22.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:22.343 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:22.366 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:22.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: pgmap v1527: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 170 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: 2.57 scrub starts 2024-09-16T05:24:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: 2.57 scrub ok 2024-09-16T05:24:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: osdmap e911: 12 total, 11 up, 12 in 2024-09-16T05:24:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:24:22.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:22 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: pgmap v1527: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 170 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: 2.57 scrub starts 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: 2.57 scrub ok 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: osdmap e911: 12 total, 11 up, 12 in 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:24:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:24:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:24:22.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:22 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:24:22.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:22.491 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:22.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:22.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:22.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:22.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2318 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 583.700058958, 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98387304600000003, 2024-09-16T05:24:22.627 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2318 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924655+0000", 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924656+0000", 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.341e-06 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924663+0000", 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6050000000000003e-06 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.630 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924673+0000", 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8830000000000001e-06 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924680+0000", 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2230000000000001e-06 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924730+0000", 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0256999999999997e-05 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.631 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924737+0000", 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1550000000000001e-06 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.908327+0000", 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98359007499999995 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.908528+0000", 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000200507 2024-09-16T05:24:22.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2319 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 583.70001823899997, 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98447908799999995, 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.633 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2319 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:24:22.634 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924696+0000", 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5900000000000003e-07 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924698+0000", 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9309999999999998e-06 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924703+0000", 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9609999999999996e-06 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924707+0000", 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8990000000000003e-06 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924769+0000", 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1797999999999993e-05 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924776+0000", 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.99e-06 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909060+0000", 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98428400299999996 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909175+0000", 2024-09-16T05:24:22.637 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000115147 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2320 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~0,checksum] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e329)", 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 583.69999484899995, 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98482718000000002, 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "reached pg", 2024-09-16T05:24:22.638 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2320 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:24:22.639 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924719+0000", 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4200000000000002e-07 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.640 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924722+0000", 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4380000000000002e-06 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924726+0000", 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0709999999999996e-06 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924731+0000", 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0069999999999997e-06 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.641 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924792+0000", 2024-09-16T05:24:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1561000000000002e-05 2024-09-16T05:24:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for peered", 2024-09-16T05:24:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:38.924796+0000", 2024-09-16T05:24:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0400000000000003e-06 2024-09-16T05:24:22.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909278+0000", 2024-09-16T05:24:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.98448187499999995 2024-09-16T05:24:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:39.909546+0000", 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026784600000000002 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2898 2.2b 2:d7c71ae7:::smithi20457400-5:head [copy-from ver 322] snapc 7c=[7c,7b,7a,76,6c,67,64] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:24:22.644 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 540.91801812100005, 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.026725529000000001, 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2898 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.645 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706696+0000", 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706697+0000", 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4789999999999999e-06 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.646 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706705+0000", 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0490000000000004e-06 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706719+0000", 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4231e-05 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706728+0000", 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9339999999999995e-06 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706819+0000", 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.0402000000000006e-05 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706891+0000", 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2605e-05 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.724332+0000", 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017440699 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.728315+0000", 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.003982546 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733300+0000", 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049854369999999997 2024-09-16T05:24:22.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733368+0000", 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8133000000000006e-05 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.733421+0000", 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3013999999999998e-05 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2899 2.2b 2:d7c71ae7:::smithi20457400-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e372)", 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 540.91796855400003, 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020779419, 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2899 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.651 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706745+0000", 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.652 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706746+0000", 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1600000000000002e-07 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706748+0000", 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3140000000000002e-06 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706754+0000", 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2609999999999999e-06 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.706768+0000", 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3788000000000001e-05 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.707033+0000", 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00026479299999999998 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.707061+0000", 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8050000000000001e-05 2024-09-16T05:24:22.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727362+0000", 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.020301534 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727439+0000", 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6749000000000007e-05 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:21.727525+0000", 2024-09-16T05:24:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5513999999999995e-05 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:3573 2.3 e436/329 2:c018a787:::smithi20457400-18:head v 436'117, mlcod=436'115)", 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 476.75824352500001, 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021202584, 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866470+0000", 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.866471+0000", 2024-09-16T05:24:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.119e-06 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886258+0000", 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019786313 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886318+0000", 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0550000000000001e-05 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.658 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886330+0000", 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2126999999999999e-05 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886419+0000", 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8892000000000002e-05 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.886447+0000", 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7959000000000001e-05 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.887619+0000", 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001171526 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:16:25.887673+0000", 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4098000000000003e-05 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4678 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~2974971,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 397.81856421499998, 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26066084699999997, 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4678 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806150+0000", 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806152+0000", 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0360000000000001e-06 2024-09-16T05:24:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806162+0000", 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0380000000000001e-05 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806180+0000", 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8442999999999999e-05 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806193+0000", 2024-09-16T05:24:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914999999999999e-05 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806258+0000", 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4830000000000001e-05 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806263+0000", 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6959999999999996e-06 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.064487+0000", 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.25822445799999999 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.064549+0000", 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1278999999999994e-05 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.665 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066810+0000", 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022618099999999999 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4679 2.2b 2:d7c71ae7:::smithi20457400-5:head [sparse-read 0~1225823] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 397.81849587200003, 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26074116000000003, 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:22.666 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4679 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.667 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806218+0000", 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806219+0000", 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806221+0000", 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7159999999999999e-06 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.668 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806228+0000", 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4799999999999998e-06 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806234+0000", 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0939999999999996e-06 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806309+0000", 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5340999999999996e-05 2024-09-16T05:24:22.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806312+0000", 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0620000000000001e-06 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066832+0000", 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26051971200000001 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066880+0000", 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7602000000000003e-05 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066959+0000", 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9597000000000004e-05 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.671 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4680 2.2b 2:d7c71ae7:::smithi20457400-5:head [read 0~2974971,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e515)", 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 397.81845214999998, 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.26141080300000002, 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4680 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806262+0000", 2024-09-16T05:24:22.673 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.99e-07 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806267+0000", 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3050000000000003e-06 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806275+0000", 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2930000000000001e-06 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.674 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806281+0000", 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2779999999999999e-06 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806346+0000", 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5061999999999995e-05 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:44.806354+0000", 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0669999999999992e-06 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.066986+0000", 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.260631273 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.067038+0000", 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2138999999999997e-05 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:45.067672+0000", 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063478700000000005 2024-09-16T05:24:22.676 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:5712 2.2b 2:d7c71ae7:::smithi20457400-5:head [copy-from ver 601] snapc ee=[ee,ed,ec,eb,ea,e9,e8,e7,e6,e5,e0,d9,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e597)", 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:06.981347+0000", 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 315.643366609, 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.019455719, 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.677 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 5712 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981347+0000", 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981347+0000", 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981349+0000", 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5e-06 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981358+0000", 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1200000000000008e-06 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981379+0000", 2024-09-16T05:24:22.679 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1044999999999999e-05 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981388+0000", 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.577e-06 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981441+0000", 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3003000000000001e-05 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.680 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.981508+0000", 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6804999999999995e-05 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 4", 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.995338+0000", 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.013829641 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:06.999508+0000", 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0041704999999999997 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:07.000738+0000", 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00122931 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:07.000774+0000", 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6146999999999998e-05 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:07.000803+0000", 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9071e-05 2024-09-16T05:24:22.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:7360 2.3 e747/329 2:c018a787:::smithi20457400-18:head v 747'249, mlcod=742'246)", 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:36.902349+0000", 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 165.722364699, 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.040589499000000001, 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.683 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.902349+0000", 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.902349+0000", 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.902349+0000", 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.03e-07 2024-09-16T05:24:22.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.941573+0000", 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.039223181000000003 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.941635+0000", 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2502000000000006e-05 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.941642+0000", 2024-09-16T05:24:22.685 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1810000000000004e-06 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.941688+0000", 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6088999999999999e-05 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.941704+0000", 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5656999999999999e-05 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.686 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.942909+0000", 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0012048790000000001 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.942939+0000", 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9807000000000001e-05 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.687 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7595 2.4d 2:b21a57e7:::smithi20457400-25:head [rollback 116] snapc 141=[141,140,13f,13d,13b,134,133,12c,12a,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e764)", 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:53.928542+0000", 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 148.696171593, 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.17219789999999999, 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7595 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.928542+0000", 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.928542+0000", 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.928543+0000", 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1300000000000001e-07 2024-09-16T05:24:22.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.928546+0000", 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.18e-06 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.928560+0000", 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3806e-05 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.928572+0000", 2024-09-16T05:24:22.690 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2713e-05 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.929473+0000", 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00090085000000000002 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.929535+0000", 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1511000000000001e-05 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.929663+0000", 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012813599999999999 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:53.930028+0000", 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00036519599999999998 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:54.100650+0000", 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.170621779 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:54.100699+0000", 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9218000000000002e-05 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:54.100740+0000", 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1097999999999999e-05 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8001 2.4d 2:b21a57e7:::smithi20457400-25:head [write 706191~739675] snapc 153=[153,152,151,150,14e,14b,149,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:32.901627+0000", 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 109.723086655, 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.050918560000000002, 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8001 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.694 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.901627+0000", 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.901627+0000", 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.901629+0000", 2024-09-16T05:24:22.695 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.12e-06 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.902430+0000", 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080023900000000001 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.902453+0000", 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3777999999999998e-05 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.902466+0000", 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2721000000000001e-05 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.902538+0000", 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2064000000000007e-05 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.902615+0000", 2024-09-16T05:24:22.697 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6644999999999995e-05 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.902829+0000", 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00021462099999999999 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.904730+0000", 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001900807 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:22.698 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952474+0000", 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.047743974000000002 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952515+0000", 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.057e-05 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952546+0000", 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1021e-05 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.699 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8002 2.4d 2:b21a57e7:::smithi20457400-25:head [write 1913635~431384] snapc 153=[153,152,151,150,14e,14b,149,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:32.906617+0000", 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 109.71809688499999, 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.046040197999999997, 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.700 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8002 2024-09-16T05:24:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.906617+0000", 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.906617+0000", 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.702 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.906619+0000", 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.699e-06 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907012+0000", 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00039346500000000001 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907028+0000", 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5828000000000002e-05 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.703 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907037+0000", 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5809999999999997e-06 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907104+0000", 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7304000000000005e-05 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907193+0000", 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9682e-05 2024-09-16T05:24:22.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907354+0000", 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016083499999999999 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.908878+0000", 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00152356 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:22.705 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952589+0000", 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.043710986 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952628+0000", 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8776999999999998e-05 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952657+0000", 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9481000000000001e-05 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8003 2.4d 2:b21a57e7:::smithi20457400-25:head [setxattr _header (58) in=7b,truncate 2706191] snapc 153=[153,152,151,150,14e,14b,149,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:32.907060+0000", 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 109.71765349499999, 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.045695711, 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8003 2024-09-16T05:24:22.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907060+0000", 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907060+0000", 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.708 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907061+0000", 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6300000000000004e-07 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907065+0000", 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.619e-06 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907073+0000", 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8660000000000006e-06 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907079+0000", 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3300000000000004e-06 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907926+0000", 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00084742400000000005 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.908019+0000", 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3032e-05 2024-09-16T05:24:22.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.908179+0000", 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000159877 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.908883+0000", 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00070398200000000002 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:22.711 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952686+0000", 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.043802189999999998 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952721+0000", 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5071999999999999e-05 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952756+0000", 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5456000000000001e-05 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.712 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8004 2.4d 2:b21a57e7:::smithi20457400-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e803)", 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:32.907089+0000", 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 109.71762525600001, 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.04582162, 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8004 2024-09-16T05:24:22.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907089+0000", 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907089+0000", 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907089+0000", 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0399999999999996e-07 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907092+0000", 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.587e-06 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907100+0000", 2024-09-16T05:24:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0679999999999997e-06 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.907105+0000", 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2129999999999999e-06 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.908776+0000", 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001671477 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.716 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.908863+0000", 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6720999999999999e-05 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952781+0000", 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.043918319999999997 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952837+0000", 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5355000000000001e-05 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.952910+0000", 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3374999999999995e-05 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8159 2.4d 2:b21a57e7:::smithi20457400-25:head [copy-from ver 1265] snapc 158=[158,155,153,14e,149,119,116] ondisk+write+known_if_redirected+supports_pool_eio e814)", 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:43.836539+0000", 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 98.788175050000007, 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.021906421999999998, 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8159 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836539+0000", 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836539+0000", 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836541+0000", 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.283e-06 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836552+0000", 2024-09-16T05:24:22.720 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0869e-05 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836577+0000", 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4748000000000001e-05 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836587+0000", 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0431999999999999e-05 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.721 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836733+0000", 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000145453 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836740+0000", 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0600000000000002e-06 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.839027+0000", 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022876760000000002 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.722 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.839142+0000", 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011458100000000001 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.845193+0000", 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006051079 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.850417+0000", 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0052238099999999997 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.858303+0000", 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0078862829999999991 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.858386+0000", 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2682999999999994e-05 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.858445+0000", 2024-09-16T05:24:22.724 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9465000000000001e-05 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8312 2.4d 2:b21a57e7:::smithi20457400-25:head [append 0~0] snapc 15b=[15b,15a] ondisk+write+known_if_redirected+supports_pool_eio e825)", 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:54.932483+0000", 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.692231140999993, 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.57080769899999995, 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.725 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8312 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.932483+0000", 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.932483+0000", 2024-09-16T05:24:22.726 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.932483+0000", 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5900000000000002e-07 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.932485+0000", 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.232e-06 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.727 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.932492+0000", 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3280000000000002e-06 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.932497+0000", 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0189999999999997e-06 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.932590+0000", 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3175000000000002e-05 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.932594+0000", 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9210000000000002e-06 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.935123+0000", 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0025296020000000002 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.935180+0000", 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6900000000000001e-05 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.935276+0000", 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.5659e-05 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.935610+0000", 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00033413600000000001 2024-09-16T05:24:22.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:55.503198+0000", 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.56758829899999996 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:55.503248+0000", 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9129999999999999e-05 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:55.503290+0000", 2024-09-16T05:24:22.731 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2839000000000001e-05 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:8312 2.4d e825/529 ondisk, result = 0)", 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:54.936284+0000", 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.688429745999997, 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.56701196399999998, 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:22.732 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.936284+0000", 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.936284+0000", 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.936284+0000", 2024-09-16T05:24:22.733 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1899999999999998e-07 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.936286+0000", 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.03e-06 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:55.502992+0000", 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.56670523399999995 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.734 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:55.503014+0000", 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2342999999999999e-05 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:55.503183+0000", 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016917200000000001 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:55.503192+0000", 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6470000000000004e-06 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.735 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:55.503296+0000", 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000104219 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:9051 2.3 e909/329 2:c018a787:::smithi20457400-18:head v 909'317, mlcod=900'311)", 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:19.166232+0000", 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 3.4584816009999999, 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34533785900000002, 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:22.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.166232+0000", 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.166232+0000", 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.737 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.166233+0000", 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6700000000000003e-07 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.166239+0000", 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1480000000000003e-06 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.510923+0000", 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.34468411700000001 2024-09-16T05:24:22.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.510932+0000", 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.619e-06 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.511007+0000", 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4938000000000007e-05 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.511038+0000", 2024-09-16T05:24:22.739 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0908000000000003e-05 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.511522+0000", 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048419100000000003 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:19.511570+0000", 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8170999999999997e-05 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:22.741 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:22.741 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:24:22.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:22.764 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:22.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:22.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:22.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:22.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:22.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:23.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:23.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:23.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:23.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:23.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:23.171 INFO:tasks.rados.rados.2.smithi204.stdout:3616: done (11 left) 2024-09-16T05:24:23.171 INFO:tasks.rados.rados.2.smithi204.stdout:3617: done (10 left) 2024-09-16T05:24:23.171 INFO:tasks.rados.rados.2.smithi204.stdout:3619: done (9 left) 2024-09-16T05:24:23.171 INFO:tasks.rados.rados.2.smithi204.stdout:3620: read oid 35 snap -1 2024-09-16T05:24:23.171 INFO:tasks.rados.rados.2.smithi204.stdout:3620: expect (ObjNum 1250 snap 391 seq_num 1250) 2024-09-16T05:24:23.171 INFO:tasks.rados.rados.2.smithi204.stdout:3621: read oid 45 snap 390 2024-09-16T05:24:23.171 INFO:tasks.rados.rados.2.smithi204.stdout:3621: expect (ObjNum 1209 snap 366 seq_num 1209) 2024-09-16T05:24:23.171 INFO:tasks.rados.rados.2.smithi204.stdout:3622: snap_remove snap 388 2024-09-16T05:24:23.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:22 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:22.930+0000 7f6af1ad6640 -1 osd.6 912 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2.12 deep-scrub starts 2024-09-16T05:24:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2.12 deep-scrub ok 2024-09-16T05:24:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:24:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:24:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: osdmap e912: 12 total, 11 up, 12 in 2024-09-16T05:24:23.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:24:23.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:24:23.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:24:23.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:24:23.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3572767108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:23.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:23.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:23.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:23.397 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:23.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2.12 deep-scrub starts 2024-09-16T05:24:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2.12 deep-scrub ok 2024-09-16T05:24:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:24:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:24:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:23.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: osdmap e912: 12 total, 11 up, 12 in 2024-09-16T05:24:23.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:24:23.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:24:23.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:24:23.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:24:23.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3572767108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2.12 deep-scrub starts 2024-09-16T05:24:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2.12 deep-scrub ok 2024-09-16T05:24:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:24:23.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:24:23.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:23.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: osdmap e912: 12 total, 11 up, 12 in 2024-09-16T05:24:23.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:24:23.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:24:23.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:24:23.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:24:23.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3572767108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:23.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:23.522 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:23.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:23.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:23.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:23.647 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:23.648 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:23.651 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:23.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:23.671 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:23.671 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:23.692 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:23.715 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:23.715 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:23.737 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:23.737 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:23.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:23.759 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:23.759 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:23.779 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:23.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:23.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:23.781 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:23.781 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:23.803 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:23.804 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:23.825 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:23.847 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:23.847 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:23.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:24:23.869 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:23.870 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:23.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:24:23.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:23.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:23.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:24.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:24.031 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:24.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:24.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:24.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:24.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:24.171 INFO:tasks.rados.rados.2.smithi204.stdout:3622: done (11 left) 2024-09-16T05:24:24.171 INFO:tasks.rados.rados.2.smithi204.stdout:3623: read oid 12 snap -1 2024-09-16T05:24:24.171 INFO:tasks.rados.rados.2.smithi204.stdout:3623: expect (ObjNum 1228 snap 373 seq_num 1228) 2024-09-16T05:24:24.171 INFO:tasks.rados.rados.2.smithi204.stdout:3624: write (excl) oid 20 current snap is 392 2024-09-16T05:24:24.171 INFO:tasks.rados.rados.2.smithi204.stdout:3624: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-09-16T05:24:24.177 INFO:tasks.rados.rados.2.smithi204.stdout:3624: writing smithi20457400-20 from 625557 to 1043948 tid 1 2024-09-16T05:24:24.186 INFO:tasks.rados.rados.2.smithi204.stdout:3624: writing smithi20457400-20 from 1838856 to 2625283 tid 2 2024-09-16T05:24:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:3625: rollback oid 3 current snap is 392 2024-09-16T05:24:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 391 2024-09-16T05:24:24.187 INFO:tasks.rados.rados.2.smithi204.stdout:3620: expect (ObjNum 1250 snap 391 seq_num 1250) 2024-09-16T05:24:24.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:24 smithi032 ceph-mon[27388]: pgmap v1530: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 177 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 995 KiB/s wr, 10 op/s; 9/184 objects degraded (4.891%) 2024-09-16T05:24:24.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:24 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:24:24.220 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:24 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:24:24.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:24 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:24.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:24 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:24:24.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:24 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:24:24.221 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:24 smithi032 ceph-mon[27388]: osdmap e913: 12 total, 11 up, 12 in 2024-09-16T05:24:24.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:24.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:24.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:24.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:24.409 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:24.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:24.415 INFO:tasks.rados.rados.2.smithi204.stdout:3621: expect (ObjNum 1209 snap 366 seq_num 1209) 2024-09-16T05:24:24.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:24 smithi196 ceph-mon[30802]: pgmap v1530: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 177 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 995 KiB/s wr, 10 op/s; 9/184 objects degraded (4.891%) 2024-09-16T05:24:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:24 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:24:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:24 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:24:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:24 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:24 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:24:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:24 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:24:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:24 smithi196 ceph-mon[30802]: osdmap e913: 12 total, 11 up, 12 in 2024-09-16T05:24:24.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:24.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:24.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:24.542 INFO:tasks.rados.rados.2.smithi204.stdout:3623: expect (ObjNum 1228 snap 373 seq_num 1228) 2024-09-16T05:24:24.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:24.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:24.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:24.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:24.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:24 smithi204 ceph-mon[30933]: pgmap v1530: 127 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 104 active+clean; 177 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 995 KiB/s wr, 10 op/s; 9/184 objects degraded (4.891%) 2024-09-16T05:24:24.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:24 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:24:24.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:24 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:24:24.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:24 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:24 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:24:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:24 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:24:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:24 smithi204 ceph-mon[30933]: osdmap e913: 12 total, 11 up, 12 in 2024-09-16T05:24:24.690 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:24.696 INFO:tasks.rados.rados.2.smithi204.stdout:3624: finishing write tid 1 to smithi20457400-20 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:3624: finishing write tid 2 to smithi20457400-20 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:3624: finishing write tid 3 to smithi20457400-20 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:3625: finishing rollback tid 0 to smithi20457400-3 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:3625: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:3624: finishing write tid 4 to smithi20457400-20 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1385 (ObjNum 1251 snap 392 seq_num 1251) dirty exists 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:3624: left oid 20 (ObjNum 1251 snap 392 seq_num 1251) 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:3620: done (13 left) 2024-09-16T05:24:24.697 INFO:tasks.rados.rados.2.smithi204.stdout:3621: done (12 left) 2024-09-16T05:24:24.698 INFO:tasks.rados.rados.2.smithi204.stdout:3623: done (11 left) 2024-09-16T05:24:24.698 INFO:tasks.rados.rados.2.smithi204.stdout:3624: done (10 left) 2024-09-16T05:24:24.698 INFO:tasks.rados.rados.2.smithi204.stdout:3626: copy_from oid 18 from oid 50 current snap is 392 2024-09-16T05:24:24.698 INFO:tasks.rados.rados.2.smithi204.stdout:3627: delete oid 37 current snap is 392 2024-09-16T05:24:24.698 INFO:tasks.rados.rados.2.smithi204.stdout:3627: done (11 left) 2024-09-16T05:24:24.698 INFO:tasks.rados.rados.2.smithi204.stdout:3628: snap_remove snap 384 2024-09-16T05:24:24.715 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:24.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:24.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:24.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:24.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:24.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:24.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:25.014 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [1] live_osds: [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:24:25.014 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:24:25.014 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-16T05:24:25.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-16T05:24:25.018 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:25.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:25.180 INFO:tasks.rados.rados.2.smithi204.stdout:3628: done (11 left) 2024-09-16T05:24:25.180 INFO:tasks.rados.rados.2.smithi204.stdout:3629: write (excl) oid 33 current snap is 392 2024-09-16T05:24:25.180 INFO:tasks.rados.rados.2.smithi204.stdout:3626: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:24:25.181 INFO:tasks.rados.rados.2.smithi204.stdout:3629: seq_num 1253 ranges {659171=568889,1839467=790096} 2024-09-16T05:24:25.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:25.187 INFO:tasks.rados.rados.2.smithi204.stdout:3629: writing smithi20457400-33 from 659171 to 1228060 tid 1 2024-09-16T05:24:25.196 INFO:tasks.rados.rados.2.smithi204.stdout:3629: writing smithi20457400-33 from 1839467 to 2629563 tid 2 2024-09-16T05:24:25.196 INFO:tasks.rados.rados.2.smithi204.stdout:3630: delete oid 17 current snap is 392 2024-09-16T05:24:25.196 INFO:tasks.rados.rados.2.smithi204.stdout:3626: finishing copy_from to smithi20457400-18 2024-09-16T05:24:25.196 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 1176 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:24:25.196 INFO:tasks.rados.rados.2.smithi204.stdout:3625: finishing rollback tid 2 to smithi20457400-3 2024-09-16T05:24:25.196 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1105 (ObjNum 1236 snap 384 seq_num 1236) dirty exists 2024-09-16T05:24:25.196 INFO:tasks.rados.rados.2.smithi204.stdout:3629: finishing write tid 1 to smithi20457400-33 2024-09-16T05:24:25.199 INFO:tasks.rados.rados.2.smithi204.stdout:3629: finishing write tid 2 to smithi20457400-33 2024-09-16T05:24:25.199 INFO:tasks.rados.rados.2.smithi204.stdout:3629: finishing write tid 3 to smithi20457400-33 2024-09-16T05:24:25.199 INFO:tasks.rados.rados.2.smithi204.stdout:3629: finishing write tid 4 to smithi20457400-33 2024-09-16T05:24:25.199 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 1117 (ObjNum 1253 snap 392 seq_num 1253) dirty exists 2024-09-16T05:24:25.199 INFO:tasks.rados.rados.2.smithi204.stdout:3629: left oid 33 (ObjNum 1253 snap 392 seq_num 1253) 2024-09-16T05:24:25.202 INFO:tasks.rados.rados.2.smithi204.stdout:3625: done (12 left) 2024-09-16T05:24:25.202 INFO:tasks.rados.rados.2.smithi204.stdout:3626: done (11 left) 2024-09-16T05:24:25.202 INFO:tasks.rados.rados.2.smithi204.stdout:3629: done (10 left) 2024-09-16T05:24:25.202 INFO:tasks.rados.rados.2.smithi204.stdout:3630: done (9 left) 2024-09-16T05:24:25.202 INFO:tasks.rados.rados.2.smithi204.stdout:3631: delete oid 15 current snap is 392 2024-09-16T05:24:25.203 INFO:tasks.rados.rados.2.smithi204.stdout:3631: done (9 left) 2024-09-16T05:24:25.203 INFO:tasks.rados.rados.2.smithi204.stdout:3632: delete oid 41 current snap is 392 2024-09-16T05:24:25.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:25.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:25.207 INFO:tasks.rados.rados.2.smithi204.stdout:3632: done (9 left) 2024-09-16T05:24:25.207 INFO:tasks.rados.rados.2.smithi204.stdout:3633: write oid 18 current snap is 392 2024-09-16T05:24:25.207 INFO:tasks.rados.rados.2.smithi204.stdout:3633: seq_num 1257 ranges {726399=586238,2073395=797434} 2024-09-16T05:24:25.213 INFO:tasks.rados.rados.2.smithi204.stdout:3633: writing smithi20457400-18 from 726399 to 1312637 tid 1 2024-09-16T05:24:25.222 INFO:tasks.rados.rados.2.smithi204.stdout:3633: writing smithi20457400-18 from 2073395 to 2870829 tid 2 2024-09-16T05:24:25.223 INFO:tasks.rados.rados.2.smithi204.stdout:3634: rollback oid 23 current snap is 392 2024-09-16T05:24:25.223 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 23 to 391 2024-09-16T05:24:25.223 INFO:tasks.rados.rados.2.smithi204.stdout:3633: finishing write tid 1 to smithi20457400-18 2024-09-16T05:24:25.223 INFO:tasks.rados.rados.2.smithi204.stdout:3635: delete oid 1 current snap is 392 2024-09-16T05:24:25.224 INFO:tasks.rados.rados.2.smithi204.stdout:3635: done (11 left) 2024-09-16T05:24:25.224 INFO:tasks.rados.rados.2.smithi204.stdout:3636: snap_create 2024-09-16T05:24:25.226 INFO:tasks.rados.rados.2.smithi204.stdout:3633: finishing write tid 2 to smithi20457400-18 2024-09-16T05:24:25.227 INFO:tasks.rados.rados.2.smithi204.stdout:3633: finishing write tid 3 to smithi20457400-18 2024-09-16T05:24:25.227 INFO:tasks.rados.rados.2.smithi204.stdout:3634: finishing rollback tid 0 to smithi20457400-23 2024-09-16T05:24:25.228 INFO:tasks.rados.rados.2.smithi204.stdout:3633: finishing write tid 4 to smithi20457400-18 2024-09-16T05:24:25.228 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 1256 (ObjNum 1257 snap 392 seq_num 1257) dirty exists 2024-09-16T05:24:25.228 INFO:tasks.rados.rados.2.smithi204.stdout:3633: left oid 18 (ObjNum 1257 snap 392 seq_num 1257) 2024-09-16T05:24:25.228 INFO:tasks.rados.rados.2.smithi204.stdout:3634: finishing rollback tid 1 to smithi20457400-23 2024-09-16T05:24:25.228 INFO:tasks.rados.rados.2.smithi204.stdout:3634: finishing rollback tid 2 to smithi20457400-23 2024-09-16T05:24:25.228 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 865 (ObjNum 1127 snap 338 seq_num 1127) dirty exists 2024-09-16T05:24:25.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:25 smithi032 ceph-mon[27388]: 2.5e deep-scrub starts 2024-09-16T05:24:25.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:25 smithi032 ceph-mon[27388]: 2.5e deep-scrub ok 2024-09-16T05:24:25.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:25 smithi032 ceph-mon[27388]: osdmap e914: 12 total, 11 up, 12 in 2024-09-16T05:24:25.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:25 smithi032 ceph-mon[27388]: 2.3a scrub starts 2024-09-16T05:24:25.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:25 smithi032 ceph-mon[27388]: 2.3a scrub ok 2024-09-16T05:24:25.269 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:25.270 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:25.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:25.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:25.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:25.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:25.452 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:25.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:25.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:25 smithi196 ceph-mon[30802]: 2.5e deep-scrub starts 2024-09-16T05:24:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:25 smithi196 ceph-mon[30802]: 2.5e deep-scrub ok 2024-09-16T05:24:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:25 smithi196 ceph-mon[30802]: osdmap e914: 12 total, 11 up, 12 in 2024-09-16T05:24:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:25 smithi196 ceph-mon[30802]: 2.3a scrub starts 2024-09-16T05:24:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:25 smithi196 ceph-mon[30802]: 2.3a scrub ok 2024-09-16T05:24:25.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:25.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:25.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:25.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:25 smithi204 ceph-mon[30933]: 2.5e deep-scrub starts 2024-09-16T05:24:25.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:25 smithi204 ceph-mon[30933]: 2.5e deep-scrub ok 2024-09-16T05:24:25.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:25 smithi204 ceph-mon[30933]: osdmap e914: 12 total, 11 up, 12 in 2024-09-16T05:24:25.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:25 smithi204 ceph-mon[30933]: 2.3a scrub starts 2024-09-16T05:24:25.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:25 smithi204 ceph-mon[30933]: 2.3a scrub ok 2024-09-16T05:24:25.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:25.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:25.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:25.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:25.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:25.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:25.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:25.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:25.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:26.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:26.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:24:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:26.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:26.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:26.184 INFO:tasks.rados.rados.2.smithi204.stdout:3633: done (11 left) 2024-09-16T05:24:26.184 INFO:tasks.rados.rados.2.smithi204.stdout:3634: done (10 left) 2024-09-16T05:24:26.184 INFO:tasks.rados.rados.2.smithi204.stdout:3636: done (9 left) 2024-09-16T05:24:26.184 INFO:tasks.rados.rados.2.smithi204.stdout:3637: rollback oid 3 current snap is 393 2024-09-16T05:24:26.185 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 386 2024-09-16T05:24:26.185 INFO:tasks.rados.rados.2.smithi204.stdout:3638: write oid 32 current snap is 393 2024-09-16T05:24:26.185 INFO:tasks.rados.rados.2.smithi204.stdout:3638: seq_num 1259 ranges {760013=736736,2190359=401103} 2024-09-16T05:24:26.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:26.192 INFO:tasks.rados.rados.2.smithi204.stdout:3638: writing smithi20457400-32 from 760013 to 1496749 tid 1 2024-09-16T05:24:26.198 INFO:tasks.rados.rados.2.smithi204.stdout:3638: writing smithi20457400-32 from 2190359 to 2591462 tid 2 2024-09-16T05:24:26.198 INFO:tasks.rados.rados.2.smithi204.stdout:3639: read oid 37 snap -1 2024-09-16T05:24:26.198 INFO:tasks.rados.rados.2.smithi204.stdout:3639: expect deleted 2024-09-16T05:24:26.198 INFO:tasks.rados.rados.2.smithi204.stdout:3640: write (excl) oid 46 current snap is 393 2024-09-16T05:24:26.198 INFO:tasks.rados.rados.2.smithi204.stdout:3640: seq_num 1260 ranges {776820=411985,1848841=544761,2918265=258555} 2024-09-16T05:24:26.204 INFO:tasks.rados.rados.2.smithi204.stdout:3640: writing smithi20457400-46 from 776820 to 1188805 tid 1 2024-09-16T05:24:26.211 INFO:tasks.rados.rados.2.smithi204.stdout:3640: writing smithi20457400-46 from 1848841 to 2393602 tid 2 2024-09-16T05:24:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:26.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:26.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:26.217 INFO:tasks.rados.rados.2.smithi204.stdout:3640: writing smithi20457400-46 from 2918265 to 3176820 tid 3 2024-09-16T05:24:26.217 INFO:tasks.rados.rados.2.smithi204.stdout:3641: read oid 50 snap 378 2024-09-16T05:24:26.217 INFO:tasks.rados.rados.2.smithi204.stdout:3641: expect (ObjNum 1145 snap 342 seq_num 1145) 2024-09-16T05:24:26.217 INFO:tasks.rados.rados.2.smithi204.stdout:3642: rollback oid 45 current snap is 393 2024-09-16T05:24:26.217 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 362 2024-09-16T05:24:26.217 INFO:tasks.rados.rados.2.smithi204.stdout:3637: finishing rollback tid 0 to smithi20457400-3 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3637: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3637: finishing rollback tid 2 to smithi20457400-3 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1108 (ObjNum 1236 snap 384 seq_num 1236) dirty exists 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3638: finishing write tid 1 to smithi20457400-32 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3638: finishing write tid 2 to smithi20457400-32 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3638: finishing write tid 3 to smithi20457400-32 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3638: finishing write tid 4 to smithi20457400-32 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 32 v 1270 (ObjNum 1259 snap 393 seq_num 1259) dirty exists 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3638: left oid 32 (ObjNum 1259 snap 393 seq_num 1259) 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3640: finishing write tid 1 to smithi20457400-46 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3640: finishing write tid 2 to smithi20457400-46 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3637: done (14 left) 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3638: done (13 left) 2024-09-16T05:24:26.218 INFO:tasks.rados.rados.2.smithi204.stdout:3639: done (12 left) 2024-09-16T05:24:26.219 INFO:tasks.rados.rados.2.smithi204.stdout:3643: snap_remove snap 385 2024-09-16T05:24:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:26.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:26.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:26.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:26.351 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: pgmap v1533: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 184 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.2 MiB/s wr, 18 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:24:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:24:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:24:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:24:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:24:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:24:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 45 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: osdmap e915: 12 total, 11 up, 12 in 2024-09-16T05:24:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: 2.56 scrub starts 2024-09-16T05:24:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: 2.56 scrub ok 2024-09-16T05:24:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:24:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:26 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/188 objects degraded (4.787%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:26.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:26.469 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:26.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: pgmap v1533: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 184 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.2 MiB/s wr, 18 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:24:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:24:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:24:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:24:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:24:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:24:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 45 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: osdmap e915: 12 total, 11 up, 12 in 2024-09-16T05:24:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: 2.56 scrub starts 2024-09-16T05:24:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: 2.56 scrub ok 2024-09-16T05:24:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:24:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:26 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/188 objects degraded (4.787%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:26.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:26.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:26.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:24:26.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: pgmap v1533: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 184 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.2 MiB/s wr, 18 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:24:26.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 45 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: osdmap e915: 12 total, 11 up, 12 in 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: 2.56 scrub starts 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: 2.56 scrub ok 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:24:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:26 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/188 objects degraded (4.787%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:26.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:26.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:26.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:26.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:26.868 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:26.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:26.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:26.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:26.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:27.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:27.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:27.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:27.187 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:24:27.191 INFO:tasks.rados.rados.2.smithi204.stdout:3643: done (12 left) 2024-09-16T05:24:27.191 INFO:tasks.rados.rados.2.smithi204.stdout:3644: read oid 2 snap 378 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3644: expect (ObjNum 1218 snap 369 seq_num 1218) 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3645: write (excl) oid 9 current snap is 393 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3640: finishing write tid 3 to smithi20457400-46 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3640: finishing write tid 4 to smithi20457400-46 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3640: finishing write tid 5 to smithi20457400-46 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1039 (ObjNum 1260 snap 393 seq_num 1260) dirty exists 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3640: left oid 46 (ObjNum 1260 snap 393 seq_num 1260) 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3642: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3642: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3642: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1275 (ObjNum 1139 snap 341 seq_num 1139) dirty exists 2024-09-16T05:24:27.192 INFO:tasks.rados.rados.2.smithi204.stdout:3641: expect (ObjNum 1145 snap 342 seq_num 1145) 2024-09-16T05:24:27.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:27.245 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:27.245 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:27.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:27 smithi032 ceph-mon[27388]: osdmap e916: 12 total, 11 up, 12 in 2024-09-16T05:24:27.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3478677238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:27.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:27 smithi032 ceph-mon[27388]: 2.24 scrub starts 2024-09-16T05:24:27.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:27 smithi032 ceph-mon[27388]: 2.24 scrub ok 2024-09-16T05:24:27.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:27.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:27.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:27.378 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:27.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:27.410 INFO:tasks.rados.rados.2.smithi204.stdout:3644: expect (ObjNum 1218 snap 369 seq_num 1218) 2024-09-16T05:24:27.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:27.486 INFO:tasks.rados.rados.2.smithi204.stdout:3645: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-16T05:24:27.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:27 smithi196 ceph-mon[30802]: osdmap e916: 12 total, 11 up, 12 in 2024-09-16T05:24:27.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3478677238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:27 smithi196 ceph-mon[30802]: 2.24 scrub starts 2024-09-16T05:24:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:27 smithi196 ceph-mon[30802]: 2.24 scrub ok 2024-09-16T05:24:27.493 INFO:tasks.rados.rados.2.smithi204.stdout:3645: writing smithi20457400-9 from 793627 to 1280861 tid 1 2024-09-16T05:24:27.498 INFO:tasks.rados.rados.2.smithi204.stdout:3645: writing smithi20457400-9 from 1790970 to 2195742 tid 2 2024-09-16T05:24:27.504 INFO:tasks.rados.rados.2.smithi204.stdout:3645: writing smithi20457400-9 from 2829335 to 3193627 tid 3 2024-09-16T05:24:27.504 INFO:tasks.rados.rados.2.smithi204.stdout:3640: done (13 left) 2024-09-16T05:24:27.504 INFO:tasks.rados.rados.2.smithi204.stdout:3641: done (12 left) 2024-09-16T05:24:27.504 INFO:tasks.rados.rados.2.smithi204.stdout:3642: done (11 left) 2024-09-16T05:24:27.504 INFO:tasks.rados.rados.2.smithi204.stdout:3644: done (10 left) 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3646: read oid 12 snap -1 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3646: expect (ObjNum 1228 snap 373 seq_num 1228) 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3647: copy_from oid 11 from oid 29 current snap is 393 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3645: finishing write tid 1 to smithi20457400-9 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3645: finishing write tid 2 to smithi20457400-9 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3648: copy_from oid 15 from oid 40 current snap is 393 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3649: rollback oid 4 current snap is 393 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 4 to 387 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3650: write oid 45 current snap is 393 2024-09-16T05:24:27.505 INFO:tasks.rados.rados.2.smithi204.stdout:3650: seq_num 1262 ranges {410434=678836,1565805=548430,2573111=637323} 2024-09-16T05:24:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:27.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:27.509 INFO:tasks.rados.rados.2.smithi204.stdout:3650: writing smithi20457400-45 from 410434 to 1089270 tid 1 2024-09-16T05:24:27.515 INFO:tasks.rados.rados.2.smithi204.stdout:3650: writing smithi20457400-45 from 1565805 to 2114235 tid 2 2024-09-16T05:24:27.522 INFO:tasks.rados.rados.2.smithi204.stdout:3650: writing smithi20457400-45 from 2573111 to 3210434 tid 3 2024-09-16T05:24:27.522 INFO:tasks.rados.rados.2.smithi204.stdout:3651: snap_remove snap 391 2024-09-16T05:24:27.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:27.632 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:27.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:27.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:27 smithi204 ceph-mon[30933]: osdmap e916: 12 total, 11 up, 12 in 2024-09-16T05:24:27.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3478677238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:27.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:27 smithi204 ceph-mon[30933]: 2.24 scrub starts 2024-09-16T05:24:27.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:27 smithi204 ceph-mon[30933]: 2.24 scrub ok 2024-09-16T05:24:27.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:27.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:27.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:27.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:27.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:27.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:27.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:28.009 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:28.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:28.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:24:28.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:28.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:28.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:28.201 INFO:tasks.rados.rados.2.smithi204.stdout:3651: done (15 left) 2024-09-16T05:24:28.201 INFO:tasks.rados.rados.2.smithi204.stdout:3652: write (excl) oid 5 current snap is 393 2024-09-16T05:24:28.201 INFO:tasks.rados.rados.2.smithi204.stdout:3648: finishing copy_from to smithi20457400-15 2024-09-16T05:24:28.201 INFO:tasks.rados.rados.2.smithi204.stdout:3648: got expected ENOENT (src dne) 2024-09-16T05:24:28.201 INFO:tasks.rados.rados.2.smithi204.stdout:3648: finishing copy_from racing read to smithi20457400-15 2024-09-16T05:24:28.202 INFO:tasks.rados.rados.2.smithi204.stdout:3645: finishing write tid 3 to smithi20457400-9 2024-09-16T05:24:28.202 INFO:tasks.rados.rados.2.smithi204.stdout:3645: finishing write tid 4 to smithi20457400-9 2024-09-16T05:24:28.202 INFO:tasks.rados.rados.2.smithi204.stdout:3652: seq_num 1263 ranges {427241=754085,1624287=408441,2600534=549964} 2024-09-16T05:24:28.207 INFO:tasks.rados.rados.2.smithi204.stdout:3652: writing smithi20457400-5 from 427241 to 1181326 tid 1 2024-09-16T05:24:28.212 INFO:tasks.rados.rados.2.smithi204.stdout:3652: writing smithi20457400-5 from 1624287 to 2032728 tid 2 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:3652: writing smithi20457400-5 from 2600534 to 3150498 tid 3 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:3648: done (15 left) 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:3653: copy_from oid 37 from oid 20 current snap is 393 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:3649: finishing rollback tid 0 to smithi20457400-4 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:3649: finishing rollback tid 1 to smithi20457400-4 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1363 (ObjNum 129 snap 0 seq_num 4282958032) dirty dne 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:3645: finishing write tid 5 to smithi20457400-9 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1455 (ObjNum 1261 snap 393 seq_num 1261) dirty exists 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:3645: left oid 9 (ObjNum 1261 snap 393 seq_num 1261) 2024-09-16T05:24:28.219 INFO:tasks.rados.rados.2.smithi204.stdout:3646: expect (ObjNum 1228 snap 373 seq_num 1228) 2024-09-16T05:24:28.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:28.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:28.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:28.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:3650: finishing write tid 1 to smithi20457400-45 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:3647: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:3650: finishing write tid 2 to smithi20457400-45 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:3645: done (15 left) 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:3646: done (14 left) 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:3649: done (13 left) 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:3654: delete oid 47 current snap is 393 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:3647: finishing copy_from to smithi20457400-11 2024-09-16T05:24:28.374 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1102 (ObjNum 1236 snap 384 seq_num 1236) dirty exists 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3650: finishing write tid 3 to smithi20457400-45 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3650: finishing write tid 4 to smithi20457400-45 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3650: finishing write tid 5 to smithi20457400-45 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1279 (ObjNum 1262 snap 393 seq_num 1262) dirty exists 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3650: left oid 45 (ObjNum 1262 snap 393 seq_num 1262) 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3652: finishing write tid 1 to smithi20457400-5 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3652: finishing write tid 2 to smithi20457400-5 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3652: finishing write tid 3 to smithi20457400-5 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3652: finishing write tid 4 to smithi20457400-5 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3652: finishing write tid 5 to smithi20457400-5 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 637 (ObjNum 1263 snap 393 seq_num 1263) dirty exists 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3652: left oid 5 (ObjNum 1263 snap 393 seq_num 1263) 2024-09-16T05:24:28.375 INFO:tasks.rados.rados.2.smithi204.stdout:3653: finishing copy_from racing read to smithi20457400-37 2024-09-16T05:24:28.376 INFO:tasks.rados.rados.2.smithi204.stdout:3653: finishing copy_from to smithi20457400-37 2024-09-16T05:24:28.376 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1385 (ObjNum 1251 snap 392 seq_num 1251) dirty exists 2024-09-16T05:24:28.376 INFO:tasks.rados.rados.2.smithi204.stdout:3647: done (13 left) 2024-09-16T05:24:28.376 INFO:tasks.rados.rados.2.smithi204.stdout:3650: done (12 left) 2024-09-16T05:24:28.377 INFO:tasks.rados.rados.2.smithi204.stdout:3652: done (11 left) 2024-09-16T05:24:28.377 INFO:tasks.rados.rados.2.smithi204.stdout:3653: done (10 left) 2024-09-16T05:24:28.377 INFO:tasks.rados.rados.2.smithi204.stdout:3654: done (9 left) 2024-09-16T05:24:28.377 INFO:tasks.rados.rados.2.smithi204.stdout:3655: read oid 47 snap -1 2024-09-16T05:24:28.377 INFO:tasks.rados.rados.2.smithi204.stdout:3655: expect deleted 2024-09-16T05:24:28.377 INFO:tasks.rados.rados.2.smithi204.stdout:3656: snap_create 2024-09-16T05:24:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:28.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:28.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:28.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: pgmap v1536: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 191 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.3 MiB/s wr, 16 op/s; 9/196 objects degraded (4.592%) 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: 2.49 deep-scrub starts 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: 2.49 deep-scrub ok 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: 2.5f scrub starts 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: 2.5f scrub ok 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3478677238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:28.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:28 smithi032 ceph-mon[27388]: osdmap e917: 12 total, 11 up, 12 in 2024-09-16T05:24:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: pgmap v1536: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 191 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.3 MiB/s wr, 16 op/s; 9/196 objects degraded (4.592%) 2024-09-16T05:24:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: 2.49 deep-scrub starts 2024-09-16T05:24:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: 2.49 deep-scrub ok 2024-09-16T05:24:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:24:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:24:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: 2.5f scrub starts 2024-09-16T05:24:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: 2.5f scrub ok 2024-09-16T05:24:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3478677238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:28.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:28 smithi196 ceph-mon[30802]: osdmap e917: 12 total, 11 up, 12 in 2024-09-16T05:24:28.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:28.513 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:28.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:28.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:28.638 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:28.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: pgmap v1536: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 191 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.3 MiB/s wr, 16 op/s; 9/196 objects degraded (4.592%) 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: 2.49 deep-scrub starts 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: 2.49 deep-scrub ok 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: 2.5f scrub starts 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: 2.5f scrub ok 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3478677238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:28 smithi204 ceph-mon[30933]: osdmap e917: 12 total, 11 up, 12 in 2024-09-16T05:24:28.740 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:28.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-16T05:24:28.762 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:28.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:28.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:28.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:28.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:28.988 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:29.011 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:29.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:29.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:29.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:29.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:29.203 INFO:tasks.rados.rados.2.smithi204.stdout:3655: done (10 left) 2024-09-16T05:24:29.204 INFO:tasks.rados.rados.2.smithi204.stdout:3656: done (9 left) 2024-09-16T05:24:29.204 INFO:tasks.rados.rados.2.smithi204.stdout:3657: snap_remove snap 393 2024-09-16T05:24:29.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:29.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:29.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:29.350 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:29.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:29.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:29.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:29.471 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:29 smithi032 ceph-mon[27388]: osdmap e918: 12 total, 11 up, 12 in 2024-09-16T05:24:29.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:29 smithi196 ceph-mon[30802]: osdmap e918: 12 total, 11 up, 12 in 2024-09-16T05:24:29.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:29.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:29.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:29.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:29.641 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:29.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:29.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:29 smithi204 ceph-mon[30933]: osdmap e918: 12 total, 11 up, 12 in 2024-09-16T05:24:29.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:29.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:29.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:29.829 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:29.829 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:29.829 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:29.839 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:29.851 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:29.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:29.856 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:29.871 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:29.884 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:29.885 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:29.894 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:29.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:29.907 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:29.907 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:29.929 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:29.951 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:29.951 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:29.973 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:29.973 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:29.995 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:29.995 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:29.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:30.017 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:30.017 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:30.018 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:30.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:30.039 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:30.039 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:30.060 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:30.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:30.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:30.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:30.211 INFO:tasks.rados.rados.2.smithi204.stdout:3657: done (9 left) 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3658: read oid 9 snap -1 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3658: expect (ObjNum 1261 snap 393 seq_num 1261) 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3659: read oid 40 snap -1 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3659: expect deleted 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3660: copy_from oid 7 from oid 36 current snap is 394 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3661: read oid 8 snap -1 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3661: expect deleted 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3662: write (excl) oid 48 current snap is 394 2024-09-16T05:24:30.212 INFO:tasks.rados.rados.2.smithi204.stdout:3662: seq_num 1265 ranges {460855=504583,1624898=412110,2539027=572508} 2024-09-16T05:24:30.217 INFO:tasks.rados.rados.2.smithi204.stdout:3662: writing smithi20457400-48 from 460855 to 965438 tid 1 2024-09-16T05:24:30.223 INFO:tasks.rados.rados.2.smithi204.stdout:3662: writing smithi20457400-48 from 1624898 to 2037008 tid 2 2024-09-16T05:24:30.230 INFO:tasks.rados.rados.2.smithi204.stdout:3662: writing smithi20457400-48 from 2539027 to 3111535 tid 3 2024-09-16T05:24:30.230 INFO:tasks.rados.rados.2.smithi204.stdout:3663: copy_from oid 28 from oid 21 current snap is 394 2024-09-16T05:24:30.230 INFO:tasks.rados.rados.2.smithi204.stdout:3660: finishing copy_from to smithi20457400-7 2024-09-16T05:24:30.230 INFO:tasks.rados.rados.2.smithi204.stdout:3660: got expected ENOENT (src dne) 2024-09-16T05:24:30.231 INFO:tasks.rados.rados.2.smithi204.stdout:3660: finishing copy_from racing read to smithi20457400-7 2024-09-16T05:24:30.231 INFO:tasks.rados.rados.2.smithi204.stdout:3658: expect (ObjNum 1261 snap 393 seq_num 1261) 2024-09-16T05:24:30.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:30.271 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:30.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3662: finishing write tid 1 to smithi20457400-48 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3662: finishing write tid 2 to smithi20457400-48 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3662: finishing write tid 3 to smithi20457400-48 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3662: finishing write tid 4 to smithi20457400-48 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3662: finishing write tid 5 to smithi20457400-48 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 790 (ObjNum 1265 snap 394 seq_num 1265) dirty exists 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3662: left oid 48 (ObjNum 1265 snap 394 seq_num 1265) 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3663: finishing copy_from racing read to smithi20457400-28 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3663: finishing copy_from to smithi20457400-28 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1179 (ObjNum 1230 snap 377 seq_num 1230) dirty exists 2024-09-16T05:24:30.341 INFO:tasks.rados.rados.2.smithi204.stdout:3658: done (14 left) 2024-09-16T05:24:30.342 INFO:tasks.rados.rados.2.smithi204.stdout:3659: done (13 left) 2024-09-16T05:24:30.342 INFO:tasks.rados.rados.2.smithi204.stdout:3660: done (12 left) 2024-09-16T05:24:30.342 INFO:tasks.rados.rados.2.smithi204.stdout:3661: done (11 left) 2024-09-16T05:24:30.342 INFO:tasks.rados.rados.2.smithi204.stdout:3662: done (10 left) 2024-09-16T05:24:30.342 INFO:tasks.rados.rados.2.smithi204.stdout:3663: done (9 left) 2024-09-16T05:24:30.342 INFO:tasks.rados.rados.2.smithi204.stdout:3664: write (excl) oid 2 current snap is 394 2024-09-16T05:24:30.342 INFO:tasks.rados.rados.2.smithi204.stdout:3664: seq_num 1266 ranges {477662=579832,1683380=555768,2966450=311212} 2024-09-16T05:24:30.345 INFO:tasks.rados.rados.2.smithi204.stdout:3664: writing smithi20457400-2 from 477662 to 1057494 tid 1 2024-09-16T05:24:30.352 INFO:tasks.rados.rados.2.smithi204.stdout:3664: writing smithi20457400-2 from 1683380 to 2239148 tid 2 2024-09-16T05:24:30.358 INFO:tasks.rados.rados.2.smithi204.stdout:3664: writing smithi20457400-2 from 2966450 to 3277662 tid 3 2024-09-16T05:24:30.358 INFO:tasks.rados.rados.2.smithi204.stdout:3665: read oid 34 snap -1 2024-09-16T05:24:30.358 INFO:tasks.rados.rados.2.smithi204.stdout:3665: expect (ObjNum 1139 snap 341 seq_num 1139) 2024-09-16T05:24:30.358 INFO:tasks.rados.rados.2.smithi204.stdout:3666: snap_create 2024-09-16T05:24:30.358 INFO:tasks.rados.rados.2.smithi204.stdout:3664: finishing write tid 1 to smithi20457400-2 2024-09-16T05:24:30.358 INFO:tasks.rados.rados.2.smithi204.stdout:3664: finishing write tid 2 to smithi20457400-2 2024-09-16T05:24:30.361 INFO:tasks.rados.rados.2.smithi204.stdout:3664: finishing write tid 3 to smithi20457400-2 2024-09-16T05:24:30.361 INFO:tasks.rados.rados.2.smithi204.stdout:3664: finishing write tid 4 to smithi20457400-2 2024-09-16T05:24:30.361 INFO:tasks.rados.rados.2.smithi204.stdout:3664: finishing write tid 5 to smithi20457400-2 2024-09-16T05:24:30.361 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 560 (ObjNum 1266 snap 394 seq_num 1266) dirty exists 2024-09-16T05:24:30.361 INFO:tasks.rados.rados.2.smithi204.stdout:3664: left oid 2 (ObjNum 1266 snap 394 seq_num 1266) 2024-09-16T05:24:30.363 INFO:tasks.rados.rados.2.smithi204.stdout:3665: expect (ObjNum 1139 snap 341 seq_num 1139) 2024-09-16T05:24:30.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:30.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:30.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:30.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:30 smithi032 ceph-mon[27388]: pgmap v1539: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 197 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 9/208 objects degraded (4.327%) 2024-09-16T05:24:30.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:30 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 45 sec, osd.6 has slow ops) 2024-09-16T05:24:30.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:30 smithi032 ceph-mon[27388]: osdmap e919: 12 total, 11 up, 12 in 2024-09-16T05:24:30.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2133797848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:30.488 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:30 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:30.021+0000 7f6af1ad6640 -1 osd.6 919 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:30 smithi196 ceph-mon[30802]: pgmap v1539: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 197 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 9/208 objects degraded (4.327%) 2024-09-16T05:24:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:30 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 45 sec, osd.6 has slow ops) 2024-09-16T05:24:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:30 smithi196 ceph-mon[30802]: osdmap e919: 12 total, 11 up, 12 in 2024-09-16T05:24:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2133797848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:30.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:30.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:30.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:30.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:30.651 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:30.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:30 smithi204 ceph-mon[30933]: pgmap v1539: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 197 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 9/208 objects degraded (4.327%) 2024-09-16T05:24:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:30 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 45 sec, osd.6 has slow ops) 2024-09-16T05:24:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:30 smithi204 ceph-mon[30933]: osdmap e919: 12 total, 11 up, 12 in 2024-09-16T05:24:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2133797848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:30.742 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:30.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:30.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:30.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:30.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:30.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:31.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:31.027 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:31.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:31.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:31.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:31.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:31.219 INFO:tasks.rados.rados.2.smithi204.stdout:3664: done (11 left) 2024-09-16T05:24:31.219 INFO:tasks.rados.rados.2.smithi204.stdout:3665: done (10 left) 2024-09-16T05:24:31.219 INFO:tasks.rados.rados.2.smithi204.stdout:3666: done (9 left) 2024-09-16T05:24:31.220 INFO:tasks.rados.rados.2.smithi204.stdout:3667: snap_create 2024-09-16T05:24:31.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:31.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:31.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:31.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:31 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:31.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:31 smithi032 ceph-mon[27388]: osdmap e920: 12 total, 11 up, 12 in 2024-09-16T05:24:31.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:31 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/208 objects degraded (4.327%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:2536 2.31 2:8c1ba486:::smithi20457400-2:head [write 551362~726078] snapc 6d=[6d,6c,6b,6a,69,67,66,64,63,62,60,5d,5a,58,52,49] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 588.35150990800003, 2024-09-16T05:24:31.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055342209999999998, 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 2536 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:24:31.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941184+0000", 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.941188+0000", 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3469999999999999e-06 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942118+0000", 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00092990300000000004 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942150+0000", 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2888000000000001e-05 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942164+0000", 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3920000000000001e-05 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942258+0000", 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3839999999999996e-05 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.942266+0000", 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4139999999999997e-06 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.943927+0000", 2024-09-16T05:24:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016610450000000001 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.944033+0000", 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105979 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.944207+0000", 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000174105 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.945786+0000", 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001579352 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946595+0000", 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00080861300000000002 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946660+0000", 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4906000000000005e-05 2024-09-16T05:24:31.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:14:42.946719+0000", 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8909000000000001e-05 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.11.0:569 2.c e357/247 2:33100188:::smithi20457400-19:6a v 357'229, mlcod=351'225)", 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 564.58708058499997, 2024-09-16T05:24:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14687246600000001, 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705614+0000", 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.9599999999999999e-07 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.705621+0000", 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7150000000000001e-06 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851933+0000", 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14631169599999999 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851940+0000", 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.221e-06 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.851995+0000", 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5473999999999998e-05 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852016+0000", 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0965999999999999e-05 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852452+0000", 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043602700000000002 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:06.852486+0000", 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3871000000000001e-05 2024-09-16T05:24:31.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.31,GRANT,e:542,from:10,reservation_nonce:0)", 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 378.27645136199999, 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.69991632400000003, 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.307 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.308 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016244+0000", 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7400000000000002e-07 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016249+0000", 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0259999999999998e-06 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.715937+0000", 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.69968885199999997 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.310 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.715950+0000", 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914999999999999e-05 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716018+0000", 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7972000000000006e-05 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716036+0000", 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7302000000000001e-05 2024-09-16T05:24:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716159+0000", 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000123483 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7279 2.31 2:8c1ba486:::smithi20457400-2:head [stat,delete] snapc 133=[133,132,131,12f,12c,12a,126,125,123,119,116] ondisk+write+known_if_redirected+supports_pool_eio e738)", 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:27.821755+0000", 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 183.47093936600001, 2024-09-16T05:24:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063770049999999998, 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7279 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821755+0000", 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821755+0000", 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821756+0000", 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5600000000000001e-06 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821762+0000", 2024-09-16T05:24:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7599999999999999e-06 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821774+0000", 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1837e-05 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821784+0000", 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0141e-05 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.315 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.822598+0000", 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00081351800000000005 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.822702+0000", 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103905 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.822974+0000", 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00027219499999999998 2024-09-16T05:24:31.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.825144+0000", 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002170216 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.827867+0000", 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027226379999999999 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.828043+0000", 2024-09-16T05:24:31.317 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017595100000000001 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.828132+0000", 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9283999999999994e-05 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.1f,GRANT,e:738,from:7,reservation_nonce:0)", 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:27.877847+0000", 2024-09-16T05:24:31.318 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 183.41484691100001, 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.247356669, 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.877847+0000", 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.319 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.877847+0000", 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.877848+0000", 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6000000000000003e-07 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.877850+0000", 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4889999999999998e-06 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.124951+0000", 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24710046999999999 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.124962+0000", 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1044e-05 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.125031+0000", 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9036999999999996e-05 2024-09-16T05:24:31.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.125049+0000", 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7813e-05 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.125204+0000", 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015515599999999999 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7356 2.31 2:8c1ba486:::smithi20457400-2:head [write 427175~721527,stat] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:36.881000+0000", 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 174.411693808, 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071041089999999999, 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7356 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881000+0000", 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881000+0000", 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881005+0000", 2024-09-16T05:24:31.324 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1099999999999996e-06 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881920+0000", 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091555300000000005 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881948+0000", 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7817000000000002e-05 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.325 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881961+0000", 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3616e-05 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.882034+0000", 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3005999999999999e-05 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.882041+0000", 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9700000000000002e-06 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.326 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.884292+0000", 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022501690000000002 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.884403+0000", 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111054 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.884593+0000", 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019044900000000001 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.886037+0000", 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001443571 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.888030+0000", 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001992794 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.888072+0000", 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2719000000000002e-05 2024-09-16T05:24:31.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.888105+0000", 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2280999999999998e-05 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.56,GRANT,e:751,from:2,reservation_nonce:0)", 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:41.909362+0000", 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 169.38333184999999, 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.82786359200000004, 2024-09-16T05:24:31.329 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:41.909362+0000", 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:41.909362+0000", 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:41.909363+0000", 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:41.909366+0000", 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.084e-06 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.736932+0000", 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.82756565000000004 2024-09-16T05:24:31.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.736943+0000", 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1231e-05 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.737058+0000", 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011534799999999999 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.737073+0000", 2024-09-16T05:24:31.332 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4447000000000001e-05 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.737226+0000", 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015337799999999999 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:7554 2.5 e762/576 2:a1d8c6d9:::smithi20457400-3:head v 762'634, mlcod=761'631)", 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:52.224172+0000", 2024-09-16T05:24:31.333 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 159.068522667, 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056041479999999998, 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.224172+0000", 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.224172+0000", 2024-09-16T05:24:31.334 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.224174+0000", 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2709999999999999e-06 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.226944+0000", 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00277005 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.335 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.226987+0000", 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3625999999999999e-05 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.226997+0000", 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8900000000000002e-06 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.227070+0000", 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2711000000000002e-05 2024-09-16T05:24:31.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.227089+0000", 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8450000000000001e-05 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.229712+0000", 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026229999999999999 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.229776+0000", 2024-09-16T05:24:31.337 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4150000000000001e-05 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7997 2.56 2:6be67188:::smithi20457400-46:head [write 689384~664426,stat] snapc 153=[153,152,151,150,14e,14b,149,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:32.886758+0000", 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 118.405936662, 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058758949999999999, 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.338 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7997 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.886758+0000", 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.339 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.886758+0000", 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.886760+0000", 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5069999999999999e-06 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887721+0000", 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096062100000000004 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.340 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887746+0000", 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5806999999999999e-05 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887758+0000", 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1141999999999999e-05 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887826+0000", 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8781000000000004e-05 2024-09-16T05:24:31.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887834+0000", 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7389999999999999e-06 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.889443+0000", 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016089660000000001 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.889548+0000", 2024-09-16T05:24:31.342 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105035 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.889781+0000", 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000232585 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.891868+0000", 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020870419999999999 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.343 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.892550+0000", 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068241000000000005 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.892592+0000", 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1808999999999999e-05 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.892633+0000", 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1451e-05 2024-09-16T05:24:31.344 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8154 2.56 2:6be67188:::smithi20457400-46:116 [read 0~2369985,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e814)", 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:43.836540+0000", 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 107.4561542, 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0062454349999999997, 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.345 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8154 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836540+0000", 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836540+0000", 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836541+0000", 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9200000000000001e-07 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836545+0000", 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0729999999999998e-06 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836558+0000", 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3045e-05 2024-09-16T05:24:31.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836565+0000", 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1080000000000004e-06 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836736+0000", 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017138799999999999 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836742+0000", 2024-09-16T05:24:31.348 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1800000000000001e-06 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.841347+0000", 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004605011 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.841461+0000", 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000113952 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.349 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.842785+0000", 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013240859999999999 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8155 2.56 2:6be67188:::smithi20457400-46:116 [read 0~2369985,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e814)", 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:43.836581+0000", 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 107.456113609, 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074930639999999998, 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.350 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8155 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836581+0000", 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836581+0000", 2024-09-16T05:24:31.351 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836581+0000", 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3900000000000004e-07 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836585+0000", 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4249999999999998e-06 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836592+0000", 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7200000000000006e-06 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836598+0000", 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6389999999999997e-06 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836776+0000", 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017738300000000001 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.353 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836781+0000", 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5049999999999998e-06 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.842827+0000", 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006046147 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.842953+0000", 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012547 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.844074+0000", 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001121036 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8298 2.31 2:8c1ba486:::smithi20457400-2:head [setxattr _header (57) in=7b,truncate 3361664] snapc 15a=[15a] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:54.178108+0000", 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 97.114585841999997, 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095921700000000006, 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.355 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8298 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178108+0000", 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.356 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178108+0000", 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178109+0000", 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8000000000000004e-07 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178114+0000", 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5750000000000002e-06 2024-09-16T05:24:31.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178120+0000", 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.409e-06 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178126+0000", 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2700000000000001e-06 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179248+0000", 2024-09-16T05:24:31.358 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00112155 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179305+0000", 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6554000000000001e-05 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179413+0000", 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108739 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179850+0000", 2024-09-16T05:24:31.359 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043647400000000001 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187615+0000", 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077655429999999998 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187655+0000", 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9270999999999997e-05 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.360 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187701+0000", 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5905000000000002e-05 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8299 2.31 2:8c1ba486:::smithi20457400-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:54.178139+0000", 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 97.114555404000001, 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097353619999999991, 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.361 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8299 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178139+0000", 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.362 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178139+0000", 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178139+0000", 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3200000000000005e-07 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178142+0000", 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7310000000000002e-06 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.363 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178150+0000", 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.481e-06 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178156+0000", 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0399999999999998e-06 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179775+0000", 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016190169999999999 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179834+0000", 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8878000000000001e-05 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187742+0000", 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079080120000000007 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187797+0000", 2024-09-16T05:24:31.365 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5648e-05 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187874+0000", 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6922999999999995e-05 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8377 2.56 2:6be67188:::smithi20457400-46:head [write 696120~574496,stat] snapc 15f=[15f] ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:03.967471+0000", 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 87.325223152000007, 2024-09-16T05:24:31.366 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0073863979999999997, 2024-09-16T05:24:31.367 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.367 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.367 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.367 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.367 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.367 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:24:31.368 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8377 2024-09-16T05:24:31.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.967471+0000", 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.967471+0000", 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.369 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.967477+0000", 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6749999999999999e-06 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968251+0000", 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077443200000000003 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968274+0000", 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2738000000000002e-05 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968285+0000", 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1296e-05 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968363+0000", 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8243000000000003e-05 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968371+0000", 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.306e-06 2024-09-16T05:24:31.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.971425+0000", 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030540440000000001 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.971524+0000", 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9303999999999997e-05 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-09-16T05:24:31.372 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.971691+0000", 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000167009 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.974327+0000", 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026363990000000002 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.974756+0000", 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000428458 2024-09-16T05:24:31.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.974813+0000", 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7216999999999998e-05 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.974857+0000", 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4277000000000003e-05 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8891 2.31 2:8c1ba486:::smithi20457400-2:head [sparse-read 0~1370180,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e889)", 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:58.942478+0000", 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.350216516000003, 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0047342920000000002, 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8891 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942478+0000", 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942478+0000", 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942480+0000", 2024-09-16T05:24:31.376 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0619999999999999e-06 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942488+0000", 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.157e-06 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942504+0000", 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6475000000000001e-05 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.377 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942515+0000", 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0219e-05 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942571+0000", 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6477999999999997e-05 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942577+0000", 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2770000000000002e-06 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.945032+0000", 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024546189999999999 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.945125+0000", 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3036999999999997e-05 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.947212+0000", 2024-09-16T05:24:31.379 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020869679999999998 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8892 2.31 2:8c1ba486:::smithi20457400-2:head [sparse-read 0~1370180] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e889)", 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:58.942537+0000", 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.350157135000003, 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0049265070000000001, 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.380 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8892 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942537+0000", 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942537+0000", 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942538+0000", 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8400000000000004e-07 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942541+0000", 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8739999999999999e-06 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942548+0000", 2024-09-16T05:24:31.382 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8150000000000006e-06 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942555+0000", 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2099999999999998e-06 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942617+0000", 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2255999999999995e-05 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.383 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942620+0000", 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.9009999999999998e-06 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.947249+0000", 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0046288340000000001 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.947341+0000", 2024-09-16T05:24:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2763000000000006e-05 2024-09-16T05:24:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.947464+0000", 2024-09-16T05:24:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012217 2024-09-16T05:24:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8893 2.31 2:8c1ba486:::smithi20457400-2:head [read 0~2470926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e889)", 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:58.942568+0000", 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 32.350126087, 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006711463, 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.386 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8893 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942568+0000", 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942568+0000", 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.387 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942569+0000", 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7999999999999996e-07 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942572+0000", 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6780000000000002e-06 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942579+0000", 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.725e-06 2024-09-16T05:24:31.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942586+0000", 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7680000000000003e-06 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942656+0000", 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0528e-05 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:31.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942665+0000", 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7679999999999997e-06 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.947735+0000", 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050697249999999998 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.947822+0000", 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7022999999999997e-05 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.949280+0000", 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014577679999999999 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:8964 2.5 e900/576 2:a1d8c6d9:::smithi20457400-3:head v 900'690, mlcod=874'686)", 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:10.091185+0000", 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.201508979, 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.089365204000000004, 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.391 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.091185+0000", 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.091185+0000", 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.091186+0000", 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6700000000000002e-07 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.091193+0000", 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0700000000000001e-06 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.179929+0000", 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.088736310999999998 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.179942+0000", 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2269999999999999e-05 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.180011+0000", 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9436000000000004e-05 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.180032+0000", 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1319999999999999e-05 2024-09-16T05:24:31.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.180514+0000", 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048135300000000001 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.180550+0000", 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6576999999999999e-05 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.3.0:322 2.5a e907/742 2:5ae26c04:::smithi20457400-32:183 v 907'1264, mlcod=901'1260)", 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:17.143478+0000", 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 14.149216116, 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97020481599999997, 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:17.143478+0000", 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:17.143478+0000", 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:17.143478+0000", 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:17.143486+0000", 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6009999999999997e-06 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.112924+0000", 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.969437781 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.112932+0000", 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0050000000000004e-06 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.112997+0000", 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5691000000000005e-05 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.113028+0000", 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0852999999999998e-05 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.113637+0000", 2024-09-16T05:24:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060868800000000002 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.113683+0000", 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5945999999999997e-05 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:9199 2.5c 2:3a3554ba:::smithi20457400-48:head [write 460855~504583,stat] snapc 18d=[18c,18a,187,186,17e,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e920)", 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:30.218130+0000", 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 1.0745641990000001, 2024-09-16T05:24:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0042614619999999997, 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 9199 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.218130+0000", 2024-09-16T05:24:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.218130+0000", 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.218131+0000", 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3319999999999999e-06 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.218849+0000", 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071799400000000001 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.218861+0000", 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1731e-05 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.218869+0000", 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6279999999999996e-06 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.218936+0000", 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6854999999999996e-05 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.219007+0000", 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1390000000000006e-05 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.219140+0000", 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000132917 2024-09-16T05:24:31.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.220514+0000", 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001373785 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.222280+0000", 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001766728 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:31.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.222323+0000", 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2314000000000002e-05 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:30.222391+0000", 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8788000000000006e-05 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:31.406 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:24:31.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:31.428 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:31.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:31.488 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:31.003+0000 7f6af1ad6640 -1 osd.6 920 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:31 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:31 smithi196 ceph-mon[30802]: osdmap e920: 12 total, 11 up, 12 in 2024-09-16T05:24:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:31 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/208 objects degraded (4.327%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:31.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:31.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:31.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:31.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:31.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:31 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:31.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:31 smithi204 ceph-mon[30933]: osdmap e920: 12 total, 11 up, 12 in 2024-09-16T05:24:31.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:31 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/208 objects degraded (4.327%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:31.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:31.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:31.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:31.813 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:31.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:31.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:31.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:31.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:32.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:32.069 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:32.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:32.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:32.196 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:32.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:32.226 INFO:tasks.rados.rados.2.smithi204.stdout:3667: done (9 left) 2024-09-16T05:24:32.226 INFO:tasks.rados.rados.2.smithi204.stdout:3668: snap_remove snap 394 2024-09-16T05:24:32.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:32.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:24:32.323 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:32.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:32.426 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:32.451 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:32.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:32.488 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:31.989+0000 7f6af1ad6640 -1 osd.6 921 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:32 smithi196 ceph-mon[30802]: pgmap v1542: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 9/216 objects degraded (4.167%) 2024-09-16T05:24:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:32 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:32 smithi196 ceph-mon[30802]: osdmap e921: 12 total, 11 up, 12 in 2024-09-16T05:24:32.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:32.575 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:32.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:32.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:32 smithi032 ceph-mon[27388]: pgmap v1542: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 9/216 objects degraded (4.167%) 2024-09-16T05:24:32.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:32 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:32.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:32 smithi032 ceph-mon[27388]: osdmap e921: 12 total, 11 up, 12 in 2024-09-16T05:24:32.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:32 smithi204 ceph-mon[30933]: pgmap v1542: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 9/216 objects degraded (4.167%) 2024-09-16T05:24:32.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:32 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:32.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:32 smithi204 ceph-mon[30933]: osdmap e921: 12 total, 11 up, 12 in 2024-09-16T05:24:32.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:32.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:32.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:32.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:24:32.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:32.825 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:32.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:32.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:32.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:32.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:33.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:33.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:33.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:33.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:33.207 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:33.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:33.223 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:32 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:32.945+0000 7f6af1ad6640 -1 osd.6 921 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:33.227 INFO:tasks.rados.rados.2.smithi204.stdout:3668: done (9 left) 2024-09-16T05:24:33.227 INFO:tasks.rados.rados.2.smithi204.stdout:3669: read oid 9 snap -1 2024-09-16T05:24:33.228 INFO:tasks.rados.rados.2.smithi204.stdout:3669: expect (ObjNum 1261 snap 393 seq_num 1261) 2024-09-16T05:24:33.228 INFO:tasks.rados.rados.2.smithi204.stdout:3670: write oid 44 current snap is 396 2024-09-16T05:24:33.228 INFO:tasks.rados.rados.2.smithi204.stdout:3670: seq_num 1267 ranges {494469=655081,1741862=415779,2710226=584243} 2024-09-16T05:24:33.233 INFO:tasks.rados.rados.2.smithi204.stdout:3670: writing smithi20457400-44 from 494469 to 1149550 tid 1 2024-09-16T05:24:33.239 INFO:tasks.rados.rados.2.smithi204.stdout:3670: writing smithi20457400-44 from 1741862 to 2157641 tid 2 2024-09-16T05:24:33.246 INFO:tasks.rados.rados.2.smithi204.stdout:3670: writing smithi20457400-44 from 2710226 to 3294469 tid 3 2024-09-16T05:24:33.247 INFO:tasks.rados.rados.2.smithi204.stdout:3671: write oid 15 current snap is 396 2024-09-16T05:24:33.247 INFO:tasks.rados.rados.2.smithi204.stdout:3671: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2024-09-16T05:24:33.253 INFO:tasks.rados.rados.2.smithi204.stdout:3671: writing smithi20457400-15 from 511276 to 1241606 tid 1 2024-09-16T05:24:33.259 INFO:tasks.rados.rados.2.smithi204.stdout:3671: writing smithi20457400-15 from 1800344 to 2359781 tid 2 2024-09-16T05:24:33.265 INFO:tasks.rados.rados.2.smithi204.stdout:3671: writing smithi20457400-15 from 3021296 to 3311276 tid 3 2024-09-16T05:24:33.265 INFO:tasks.rados.rados.2.smithi204.stdout:3672: read oid 11 snap -1 2024-09-16T05:24:33.265 INFO:tasks.rados.rados.2.smithi204.stdout:3672: expect (ObjNum 1236 snap 384 seq_num 1236) 2024-09-16T05:24:33.266 INFO:tasks.rados.rados.2.smithi204.stdout:3673: read oid 7 snap -1 2024-09-16T05:24:33.266 INFO:tasks.rados.rados.2.smithi204.stdout:3673: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:24:33.266 INFO:tasks.rados.rados.2.smithi204.stdout:3674: read oid 47 snap -1 2024-09-16T05:24:33.266 INFO:tasks.rados.rados.2.smithi204.stdout:3674: expect deleted 2024-09-16T05:24:33.266 INFO:tasks.rados.rados.2.smithi204.stdout:3675: delete oid 6 current snap is 396 2024-09-16T05:24:33.266 INFO:tasks.rados.rados.2.smithi204.stdout:3669: expect (ObjNum 1261 snap 393 seq_num 1261) 2024-09-16T05:24:33.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:33.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-16T05:24:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:33.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:33.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:33.373 INFO:tasks.rados.rados.2.smithi204.stdout:3670: finishing write tid 1 to smithi20457400-44 2024-09-16T05:24:33.373 INFO:tasks.rados.rados.2.smithi204.stdout:3670: finishing write tid 2 to smithi20457400-44 2024-09-16T05:24:33.373 INFO:tasks.rados.rados.2.smithi204.stdout:3670: finishing write tid 3 to smithi20457400-44 2024-09-16T05:24:33.373 INFO:tasks.rados.rados.2.smithi204.stdout:3669: done (15 left) 2024-09-16T05:24:33.373 INFO:tasks.rados.rados.2.smithi204.stdout:3675: done (14 left) 2024-09-16T05:24:33.373 INFO:tasks.rados.rados.2.smithi204.stdout:3676: delete oid 20 current snap is 396 2024-09-16T05:24:33.373 INFO:tasks.rados.rados.2.smithi204.stdout:3670: finishing write tid 4 to smithi20457400-44 2024-09-16T05:24:33.373 INFO:tasks.rados.rados.2.smithi204.stdout:3670: finishing write tid 5 to smithi20457400-44 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1152 (ObjNum 1267 snap 396 seq_num 1267) dirty exists 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:3670: left oid 44 (ObjNum 1267 snap 396 seq_num 1267) 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:3671: finishing write tid 1 to smithi20457400-15 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:3671: finishing write tid 2 to smithi20457400-15 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:3671: finishing write tid 3 to smithi20457400-15 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:3671: finishing write tid 4 to smithi20457400-15 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:3671: finishing write tid 5 to smithi20457400-15 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 755 (ObjNum 1268 snap 396 seq_num 1268) dirty exists 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:3671: left oid 15 (ObjNum 1268 snap 396 seq_num 1268) 2024-09-16T05:24:33.374 INFO:tasks.rados.rados.2.smithi204.stdout:3672: expect (ObjNum 1236 snap 384 seq_num 1236) 2024-09-16T05:24:33.383 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:33.409 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:33.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:33.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:33.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:33.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:33 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:33.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:33 smithi196 ceph-mon[30802]: osdmap e922: 12 total, 11 up, 12 in 2024-09-16T05:24:33.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:33 smithi196 ceph-mon[30802]: pgmap v1545: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 212 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 21 op/s; 9/226 objects degraded (3.982%) 2024-09-16T05:24:33.492 INFO:tasks.rados.rados.2.smithi204.stdout:3673: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:24:33.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:33 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:33.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:33 smithi032 ceph-mon[27388]: osdmap e922: 12 total, 11 up, 12 in 2024-09-16T05:24:33.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:33 smithi032 ceph-mon[27388]: pgmap v1545: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 212 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 21 op/s; 9/226 objects degraded (3.982%) 2024-09-16T05:24:33.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:33.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:33.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:33.653 INFO:tasks.rados.rados.2.smithi204.stdout:3670: done (14 left) 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3671: done (13 left) 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3672: done (12 left) 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3673: done (11 left) 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3674: done (10 left) 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3676: done (9 left) 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3677: read oid 5 snap 386 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3677: expect (ObjNum 1210 snap 366 seq_num 1210) 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3678: rollback oid 47 current snap is 396 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 47 to 392 2024-09-16T05:24:33.654 INFO:tasks.rados.rados.2.smithi204.stdout:3679: write (excl) oid 18 current snap is 396 2024-09-16T05:24:33.655 INFO:tasks.rados.rados.2.smithi204.stdout:3679: seq_num 1271 ranges {561697=672430,1975790=706764,3103565=258132} 2024-09-16T05:24:33.660 INFO:tasks.rados.rados.2.smithi204.stdout:3679: writing smithi20457400-18 from 561697 to 1234127 tid 1 2024-09-16T05:24:33.668 INFO:tasks.rados.rados.2.smithi204.stdout:3679: writing smithi20457400-18 from 1975790 to 2682554 tid 2 2024-09-16T05:24:33.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:33 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:33.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:33 smithi204 ceph-mon[30933]: osdmap e922: 12 total, 11 up, 12 in 2024-09-16T05:24:33.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:33 smithi204 ceph-mon[30933]: pgmap v1545: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 212 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 21 op/s; 9/226 objects degraded (3.982%) 2024-09-16T05:24:33.671 INFO:tasks.rados.rados.2.smithi204.stdout:3679: writing smithi20457400-18 from 3103565 to 3361697 tid 3 2024-09-16T05:24:33.671 INFO:tasks.rados.rados.2.smithi204.stdout:3680: snap_remove snap 387 2024-09-16T05:24:33.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:33.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:33.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:33.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:33.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:33.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:33.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:33.882 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:33.936 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:33.958 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:33.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:34.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:34.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:34.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:34.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:34.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:34.224 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:33 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:33.969+0000 7f6af1ad6640 -1 osd.6 923 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:24:34.232 INFO:tasks.rados.rados.2.smithi204.stdout:3680: done (12 left) 2024-09-16T05:24:34.232 INFO:tasks.rados.rados.2.smithi204.stdout:3681: rollback oid 29 current snap is 396 2024-09-16T05:24:34.232 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 362 2024-09-16T05:24:34.232 INFO:tasks.rados.rados.2.smithi204.stdout:3682: snap_remove snap 390 2024-09-16T05:24:34.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:34.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:34.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:34.367 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:24:34.367 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:24:34.368 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:24:34.368 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:24:34.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:34 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:34.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:34 smithi032 ceph-mon[27388]: osdmap e923: 12 total, 11 up, 12 in 2024-09-16T05:24:34.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1067723746' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:34.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:34.460 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:34.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:34.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:34 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:34 smithi196 ceph-mon[30802]: osdmap e923: 12 total, 11 up, 12 in 2024-09-16T05:24:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1067723746' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:34.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:34.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:34.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:34.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:34 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:34.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:34 smithi204 ceph-mon[30933]: osdmap e923: 12 total, 11 up, 12 in 2024-09-16T05:24:34.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1067723746' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:34.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:34.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:34.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:34.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:34.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:34.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:34.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:34.964 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:34.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:35.054 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:35.054 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:35.054 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:35.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:35.077 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:35.077 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:35.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:35.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:35.096 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:35.118 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:35.118 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:35.140 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:35.161 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:35.161 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:35.183 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:35.183 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:35.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:35.205 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:35.205 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:35.219 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:35.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:35.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:35.226 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:35.227 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:35.236 INFO:tasks.rados.rados.2.smithi204.stdout:3682: done (13 left) 2024-09-16T05:24:35.236 INFO:tasks.rados.rados.2.smithi204.stdout:3683: delete oid 28 current snap is 396 2024-09-16T05:24:35.236 INFO:tasks.rados.rados.2.smithi204.stdout:3678: finishing rollback tid 1 to smithi20457400-47 2024-09-16T05:24:35.236 INFO:tasks.rados.rados.2.smithi204.stdout:3678: finishing rollback tid 2 to smithi20457400-47 2024-09-16T05:24:35.236 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1388 (ObjNum 1197 snap 366 seq_num 1197) dirty exists 2024-09-16T05:24:35.236 INFO:tasks.rados.rados.2.smithi204.stdout:3677: expect (ObjNum 1210 snap 366 seq_num 1210) 2024-09-16T05:24:35.253 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:35.254 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:35.274 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:35.275 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:35.317 INFO:tasks.rados.rados.2.smithi204.stdout:3679: finishing write tid 1 to smithi20457400-18 2024-09-16T05:24:35.317 INFO:tasks.rados.rados.2.smithi204.stdout:3679: finishing write tid 2 to smithi20457400-18 2024-09-16T05:24:35.317 INFO:tasks.rados.rados.2.smithi204.stdout:3677: done (13 left) 2024-09-16T05:24:35.317 INFO:tasks.rados.rados.2.smithi204.stdout:3678: done (12 left) 2024-09-16T05:24:35.317 INFO:tasks.rados.rados.2.smithi204.stdout:3683: done (11 left) 2024-09-16T05:24:35.317 INFO:tasks.rados.rados.2.smithi204.stdout:3684: snap_create 2024-09-16T05:24:35.317 INFO:tasks.rados.rados.2.smithi204.stdout:3679: finishing write tid 3 to smithi20457400-18 2024-09-16T05:24:35.317 INFO:tasks.rados.rados.2.smithi204.stdout:3679: finishing write tid 4 to smithi20457400-18 2024-09-16T05:24:35.318 INFO:tasks.rados.rados.2.smithi204.stdout:3679: finishing write tid 5 to smithi20457400-18 2024-09-16T05:24:35.318 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 1260 (ObjNum 1271 snap 396 seq_num 1271) dirty exists 2024-09-16T05:24:35.318 INFO:tasks.rados.rados.2.smithi204.stdout:3679: left oid 18 (ObjNum 1271 snap 396 seq_num 1271) 2024-09-16T05:24:35.318 INFO:tasks.rados.rados.2.smithi204.stdout:3681: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:24:35.318 INFO:tasks.rados.rados.2.smithi204.stdout:3681: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:24:35.318 INFO:tasks.rados.rados.2.smithi204.stdout:3681: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:24:35.318 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1303 (ObjNum 1178 snap 360 seq_num 1178) dirty exists 2024-09-16T05:24:35.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:35.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:35.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:35 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:35 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1067723746' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:35 smithi032 ceph-mon[27388]: osdmap e924: 12 total, 11 up, 12 in 2024-09-16T05:24:35.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:35 smithi032 ceph-mon[27388]: pgmap v1548: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 216 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 21 op/s; 9/232 objects degraded (3.879%) 2024-09-16T05:24:35.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4149845602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:35.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:35.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:35.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:35.487 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:35 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:35.004+0000 7f6af1ad6640 -1 osd.6 924 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:35 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:35 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1067723746' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:35 smithi196 ceph-mon[30802]: osdmap e924: 12 total, 11 up, 12 in 2024-09-16T05:24:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:35 smithi196 ceph-mon[30802]: pgmap v1548: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 216 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 21 op/s; 9/232 objects degraded (3.879%) 2024-09-16T05:24:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4149845602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:35.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:35.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:35.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:35.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:35 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:35.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:35 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:35.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1067723746' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:35.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:35 smithi204 ceph-mon[30933]: osdmap e924: 12 total, 11 up, 12 in 2024-09-16T05:24:35.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:35 smithi204 ceph-mon[30933]: pgmap v1548: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 216 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.6 MiB/s wr, 21 op/s; 9/232 objects degraded (3.879%) 2024-09-16T05:24:35.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4149845602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:35.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:35.729 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:35.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:35.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:35.857 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:35.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:35.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:35.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:35.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:36.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:24:36.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:36.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:36.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:36.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:36.237 INFO:tasks.rados.rados.2.smithi204.stdout:3679: done (11 left) 2024-09-16T05:24:36.238 INFO:tasks.rados.rados.2.smithi204.stdout:3681: done (10 left) 2024-09-16T05:24:36.238 INFO:tasks.rados.rados.2.smithi204.stdout:3684: done (9 left) 2024-09-16T05:24:36.238 INFO:tasks.rados.rados.2.smithi204.stdout:3685: read oid 43 snap -1 2024-09-16T05:24:36.238 INFO:tasks.rados.rados.2.smithi204.stdout:3685: expect (ObjNum 1176 snap 360 seq_num 1176) 2024-09-16T05:24:36.238 INFO:tasks.rados.rados.2.smithi204.stdout:3686: write (excl) oid 25 current snap is 397 2024-09-16T05:24:36.238 INFO:tasks.rados.rados.2.smithi204.stdout:3686: seq_num 1273 ranges {595311=422928,1576401=710433,3042058=353253} 2024-09-16T05:24:36.241 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:36.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:36.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:36.243 INFO:tasks.rados.rados.2.smithi204.stdout:3686: writing smithi20457400-25 from 595311 to 1018239 tid 1 2024-09-16T05:24:36.250 INFO:tasks.rados.rados.2.smithi204.stdout:3686: writing smithi20457400-25 from 1576401 to 2286834 tid 2 2024-09-16T05:24:36.257 INFO:tasks.rados.rados.2.smithi204.stdout:3686: writing smithi20457400-25 from 3042058 to 3395311 tid 3 2024-09-16T05:24:36.257 INFO:tasks.rados.rados.2.smithi204.stdout:3687: read oid 15 snap -1 2024-09-16T05:24:36.257 INFO:tasks.rados.rados.2.smithi204.stdout:3687: expect (ObjNum 1268 snap 396 seq_num 1268) 2024-09-16T05:24:36.257 INFO:tasks.rados.rados.2.smithi204.stdout:3688: copy_from oid 47 from oid 46 current snap is 397 2024-09-16T05:24:36.257 INFO:tasks.rados.rados.2.smithi204.stdout:3685: expect (ObjNum 1176 snap 360 seq_num 1176) 2024-09-16T05:24:36.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:36.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:36.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:36.382 INFO:tasks.rados.rados.2.smithi204.stdout:3686: finishing write tid 1 to smithi20457400-25 2024-09-16T05:24:36.382 INFO:tasks.rados.rados.2.smithi204.stdout:3686: finishing write tid 2 to smithi20457400-25 2024-09-16T05:24:36.382 INFO:tasks.rados.rados.2.smithi204.stdout:3686: finishing write tid 3 to smithi20457400-25 2024-09-16T05:24:36.382 INFO:tasks.rados.rados.2.smithi204.stdout:3686: finishing write tid 4 to smithi20457400-25 2024-09-16T05:24:36.382 INFO:tasks.rados.rados.2.smithi204.stdout:3686: finishing write tid 5 to smithi20457400-25 2024-09-16T05:24:36.382 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1284 (ObjNum 1273 snap 397 seq_num 1273) dirty exists 2024-09-16T05:24:36.383 INFO:tasks.rados.rados.2.smithi204.stdout:3686: left oid 25 (ObjNum 1273 snap 397 seq_num 1273) 2024-09-16T05:24:36.383 INFO:tasks.rados.rados.2.smithi204.stdout:3685: done (12 left) 2024-09-16T05:24:36.383 INFO:tasks.rados.rados.2.smithi204.stdout:3686: done (11 left) 2024-09-16T05:24:36.383 INFO:tasks.rados.rados.2.smithi204.stdout:3689: snap_create 2024-09-16T05:24:36.383 INFO:tasks.rados.rados.2.smithi204.stdout:3687: expect (ObjNum 1268 snap 396 seq_num 1268) 2024-09-16T05:24:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:36 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:36 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 55 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:36 smithi032 ceph-mon[27388]: osdmap e925: 12 total, 11 up, 12 in 2024-09-16T05:24:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:36 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/232 objects degraded (3.879%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:36.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:36.483 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:36.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:36 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:36.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:36 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 55 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:36 smithi196 ceph-mon[30802]: osdmap e925: 12 total, 11 up, 12 in 2024-09-16T05:24:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:36 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/232 objects degraded (3.879%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:36.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:36.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:36.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:36.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:36.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:36.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:36.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:36 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:36.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:36 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 55 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:36.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:36 smithi204 ceph-mon[30933]: osdmap e925: 12 total, 11 up, 12 in 2024-09-16T05:24:36.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:36 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/232 objects degraded (3.879%), 3 pgs degraded (PG_DEGRADED) 2024-09-16T05:24:36.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:36.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:36.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:36.866 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:36.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:36.969 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:36.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:36.991 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:36.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:37.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:37.115 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:37.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:24:37.119+0000 7f1d66d9d640 0 --2- 172.21.15.32:0/2059448785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d68106f10 0x7f1d6819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:24:37.133 INFO:tasks.rados.rados.2.smithi204.stdout:3688: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:24:37.136 INFO:tasks.rados.rados.2.smithi204.stdout:3688: finishing copy_from to smithi20457400-47 2024-09-16T05:24:37.136 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1039 (ObjNum 1260 snap 393 seq_num 1260) dirty exists 2024-09-16T05:24:37.217 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:37.240 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:37.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:37.243 INFO:tasks.rados.rados.2.smithi204.stdout:3687: done (11 left) 2024-09-16T05:24:37.243 INFO:tasks.rados.rados.2.smithi204.stdout:3688: done (10 left) 2024-09-16T05:24:37.243 INFO:tasks.rados.rados.2.smithi204.stdout:3689: done (9 left) 2024-09-16T05:24:37.243 INFO:tasks.rados.rados.2.smithi204.stdout:3690: read oid 37 snap -1 2024-09-16T05:24:37.243 INFO:tasks.rados.rados.2.smithi204.stdout:3690: expect (ObjNum 1251 snap 392 seq_num 1251) 2024-09-16T05:24:37.243 INFO:tasks.rados.rados.2.smithi204.stdout:3691: copy_from oid 47 from oid 25 current snap is 398 2024-09-16T05:24:37.243 INFO:tasks.rados.rados.2.smithi204.stdout:3692: write oid 18 current snap is 398 2024-09-16T05:24:37.243 INFO:tasks.rados.rados.2.smithi204.stdout:3692: seq_num 1274 ranges {612118=498177,1634883=570444,2669481=706662} 2024-09-16T05:24:37.249 INFO:tasks.rados.rados.2.smithi204.stdout:3692: writing smithi20457400-18 from 612118 to 1110295 tid 1 2024-09-16T05:24:37.255 INFO:tasks.rados.rados.2.smithi204.stdout:3692: writing smithi20457400-18 from 1634883 to 2205327 tid 2 2024-09-16T05:24:37.261 INFO:tasks.rados.rados.2.smithi204.stdout:3692: writing smithi20457400-18 from 2669481 to 3376143 tid 3 2024-09-16T05:24:37.261 INFO:tasks.rados.rados.2.smithi204.stdout:3693: write (excl) oid 40 current snap is 398 2024-09-16T05:24:37.261 INFO:tasks.rados.rados.2.smithi204.stdout:3693: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2024-09-16T05:24:37.268 INFO:tasks.rados.rados.2.smithi204.stdout:3693: writing smithi20457400-40 from 628925 to 1202351 tid 1 2024-09-16T05:24:37.275 INFO:tasks.rados.rados.2.smithi204.stdout:3693: writing smithi20457400-40 from 1693365 to 2407467 tid 2 2024-09-16T05:24:37.280 INFO:tasks.rados.rados.2.smithi204.stdout:3693: writing smithi20457400-40 from 3096904 to 3428925 tid 3 2024-09-16T05:24:37.280 INFO:tasks.rados.rados.2.smithi204.stdout:3694: read oid 6 snap -1 2024-09-16T05:24:37.281 INFO:tasks.rados.rados.2.smithi204.stdout:3694: expect deleted 2024-09-16T05:24:37.281 INFO:tasks.rados.rados.2.smithi204.stdout:3695: snap_remove snap 395 2024-09-16T05:24:37.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:37.364 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:37.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:37.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:37 smithi032 ceph-mon[27388]: osdmap e926: 12 total, 11 up, 12 in 2024-09-16T05:24:37.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:37 smithi032 ceph-mon[27388]: pgmap v1551: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 219 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:37.379 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:37.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:37.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:37.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:37.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:37.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:37.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:37 smithi204 ceph-mon[30933]: osdmap e926: 12 total, 11 up, 12 in 2024-09-16T05:24:37.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:37 smithi204 ceph-mon[30933]: pgmap v1551: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 219 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:37.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:37.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:37 smithi196 ceph-mon[30802]: osdmap e926: 12 total, 11 up, 12 in 2024-09-16T05:24:37.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:37 smithi196 ceph-mon[30802]: pgmap v1551: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 219 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:37.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:37.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:37.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:37.869 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:37.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:37.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-16T05:24:37.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:37.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:37.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:38.037 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:38.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:38.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:38.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:38.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:38.247 INFO:tasks.rados.rados.2.smithi204.stdout:3695: done (14 left) 2024-09-16T05:24:38.247 INFO:tasks.rados.rados.2.smithi204.stdout:3696: delete oid 21 current snap is 398 2024-09-16T05:24:38.247 INFO:tasks.rados.rados.2.smithi204.stdout:3690: expect (ObjNum 1251 snap 392 seq_num 1251) 2024-09-16T05:24:38.249 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:38.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:38.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:38.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:38.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:38.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:38 smithi032 ceph-mon[27388]: osdmap e927: 12 total, 11 up, 12 in 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:3692: finishing write tid 1 to smithi20457400-18 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:3692: finishing write tid 2 to smithi20457400-18 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:3692: finishing write tid 3 to smithi20457400-18 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:3692: finishing write tid 4 to smithi20457400-18 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:3692: finishing write tid 5 to smithi20457400-18 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 1264 (ObjNum 1274 snap 398 seq_num 1274) dirty exists 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:3692: left oid 18 (ObjNum 1274 snap 398 seq_num 1274) 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:3693: finishing write tid 1 to smithi20457400-40 2024-09-16T05:24:38.427 INFO:tasks.rados.rados.2.smithi204.stdout:3693: finishing write tid 2 to smithi20457400-40 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3690: done (14 left) 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3692: done (13 left) 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3696: done (12 left) 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3697: snap_create 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3693: finishing write tid 3 to smithi20457400-40 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3693: finishing write tid 4 to smithi20457400-40 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3693: finishing write tid 5 to smithi20457400-40 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1063 (ObjNum 1275 snap 398 seq_num 1275) dirty exists 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3693: left oid 40 (ObjNum 1275 snap 398 seq_num 1275) 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3691: finishing copy_from racing read to smithi20457400-47 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:3691: finishing copy_from to smithi20457400-47 2024-09-16T05:24:38.428 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1284 (ObjNum 1273 snap 397 seq_num 1273) dirty exists 2024-09-16T05:24:38.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:38.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:38.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:38.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:24:38.527 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:38.531 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:38.531 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:38.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:38.627 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:38.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:38.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:38 smithi204 ceph-mon[30933]: osdmap e927: 12 total, 11 up, 12 in 2024-09-16T05:24:38.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:38.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:38 smithi196 ceph-mon[30802]: osdmap e927: 12 total, 11 up, 12 in 2024-09-16T05:24:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:38.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:38.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:38.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:38.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:38.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:38.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:39.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:39.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:39.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:39.126 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:39.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:39.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:39.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:39.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:39.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:39.255 INFO:tasks.rados.rados.2.smithi204.stdout:3691: done (12 left) 2024-09-16T05:24:39.255 INFO:tasks.rados.rados.2.smithi204.stdout:3693: done (11 left) 2024-09-16T05:24:39.255 INFO:tasks.rados.rados.2.smithi204.stdout:3694: done (10 left) 2024-09-16T05:24:39.255 INFO:tasks.rados.rados.2.smithi204.stdout:3697: done (9 left) 2024-09-16T05:24:39.256 INFO:tasks.rados.rados.2.smithi204.stdout:3698: read oid 32 snap -1 2024-09-16T05:24:39.256 INFO:tasks.rados.rados.2.smithi204.stdout:3698: expect (ObjNum 1259 snap 393 seq_num 1259) 2024-09-16T05:24:39.256 INFO:tasks.rados.rados.2.smithi204.stdout:3699: rollback oid 29 current snap is 399 2024-09-16T05:24:39.256 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 392 2024-09-16T05:24:39.256 INFO:tasks.rados.rados.2.smithi204.stdout:3700: write oid 25 current snap is 399 2024-09-16T05:24:39.256 INFO:tasks.rados.rados.2.smithi204.stdout:3700: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-09-16T05:24:39.263 INFO:tasks.rados.rados.2.smithi204.stdout:3700: writing smithi20457400-25 from 662539 to 1386463 tid 1 2024-09-16T05:24:39.270 INFO:tasks.rados.rados.2.smithi204.stdout:3700: writing smithi20457400-25 from 2093976 to 2811747 tid 2 2024-09-16T05:24:39.274 INFO:tasks.rados.rados.2.smithi204.stdout:3700: writing smithi20457400-25 from 3435397 to 3462539 tid 3 2024-09-16T05:24:39.274 INFO:tasks.rados.rados.2.smithi204.stdout:3701: write (excl) oid 31 current snap is 399 2024-09-16T05:24:39.275 INFO:tasks.rados.rados.2.smithi204.stdout:3699: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:24:39.275 INFO:tasks.rados.rados.2.smithi204.stdout:3699: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:24:39.275 INFO:tasks.rados.rados.2.smithi204.stdout:3699: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:24:39.275 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1311 (ObjNum 1236 snap 384 seq_num 1236) dirty exists 2024-09-16T05:24:39.275 INFO:tasks.rados.rados.2.smithi204.stdout:3698: expect (ObjNum 1259 snap 393 seq_num 1259) 2024-09-16T05:24:39.337 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:39.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:39.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:39 smithi032 ceph-mon[27388]: pgmap v1553: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 220 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.1 MiB/s wr, 16 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:39.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:39 smithi032 ceph-mon[27388]: osdmap e928: 12 total, 11 up, 12 in 2024-09-16T05:24:39.375 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:39.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:39.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:24:39.440 INFO:tasks.rados.rados.2.smithi204.stdout:3700: finishing write tid 1 to smithi20457400-25 2024-09-16T05:24:39.440 INFO:tasks.rados.rados.2.smithi204.stdout:3700: finishing write tid 2 to smithi20457400-25 2024-09-16T05:24:39.440 INFO:tasks.rados.rados.2.smithi204.stdout:3700: finishing write tid 3 to smithi20457400-25 2024-09-16T05:24:39.440 INFO:tasks.rados.rados.2.smithi204.stdout:3700: finishing write tid 4 to smithi20457400-25 2024-09-16T05:24:39.440 INFO:tasks.rados.rados.2.smithi204.stdout:3700: finishing write tid 5 to smithi20457400-25 2024-09-16T05:24:39.440 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1288 (ObjNum 1277 snap 399 seq_num 1277) dirty exists 2024-09-16T05:24:39.441 INFO:tasks.rados.rados.2.smithi204.stdout:3700: left oid 25 (ObjNum 1277 snap 399 seq_num 1277) 2024-09-16T05:24:39.441 INFO:tasks.rados.rados.2.smithi204.stdout:3701: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2024-09-16T05:24:39.445 INFO:tasks.rados.rados.2.smithi204.stdout:3701: writing smithi20457400-31 from 679346 to 1194872 tid 1 2024-09-16T05:24:39.452 INFO:tasks.rados.rados.2.smithi204.stdout:3701: writing smithi20457400-31 from 1868811 to 2446593 tid 2 2024-09-16T05:24:39.458 INFO:tasks.rados.rados.2.smithi204.stdout:3701: writing smithi20457400-31 from 3179173 to 3479346 tid 3 2024-09-16T05:24:39.459 INFO:tasks.rados.rados.2.smithi204.stdout:3698: done (12 left) 2024-09-16T05:24:39.459 INFO:tasks.rados.rados.2.smithi204.stdout:3699: done (11 left) 2024-09-16T05:24:39.459 INFO:tasks.rados.rados.2.smithi204.stdout:3700: done (10 left) 2024-09-16T05:24:39.459 INFO:tasks.rados.rados.2.smithi204.stdout:3702: copy_from oid 33 from oid 46 current snap is 399 2024-09-16T05:24:39.459 INFO:tasks.rados.rados.2.smithi204.stdout:3701: finishing write tid 1 to smithi20457400-31 2024-09-16T05:24:39.459 INFO:tasks.rados.rados.2.smithi204.stdout:3701: finishing write tid 2 to smithi20457400-31 2024-09-16T05:24:39.459 INFO:tasks.rados.rados.2.smithi204.stdout:3703: write oid 17 current snap is 399 2024-09-16T05:24:39.459 INFO:tasks.rados.rados.2.smithi204.stdout:3703: seq_num 1279 ranges {696153=590775,1927293=721440,3206596=289557} 2024-09-16T05:24:39.465 INFO:tasks.rados.rados.2.smithi204.stdout:3703: writing smithi20457400-17 from 696153 to 1286928 tid 1 2024-09-16T05:24:39.473 INFO:tasks.rados.rados.2.smithi204.stdout:3703: writing smithi20457400-17 from 1927293 to 2648733 tid 2 2024-09-16T05:24:39.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:39.478 INFO:tasks.rados.rados.2.smithi204.stdout:3703: writing smithi20457400-17 from 3206596 to 3496153 tid 3 2024-09-16T05:24:39.478 INFO:tasks.rados.rados.2.smithi204.stdout:3704: copy_from oid 28 from oid 43 current snap is 399 2024-09-16T05:24:39.478 INFO:tasks.rados.rados.2.smithi204.stdout:3701: finishing write tid 3 to smithi20457400-31 2024-09-16T05:24:39.478 INFO:tasks.rados.rados.2.smithi204.stdout:3705: snap_remove snap 396 2024-09-16T05:24:39.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:39.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:39.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:39.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:39.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:39.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:39 smithi204 ceph-mon[30933]: pgmap v1553: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 220 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.1 MiB/s wr, 16 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:39.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:39 smithi204 ceph-mon[30933]: osdmap e928: 12 total, 11 up, 12 in 2024-09-16T05:24:39.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:39.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:39 smithi196 ceph-mon[30802]: pgmap v1553: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 220 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.1 MiB/s wr, 16 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:39.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:39 smithi196 ceph-mon[30802]: osdmap e928: 12 total, 11 up, 12 in 2024-09-16T05:24:39.752 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:39.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:39.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:24:39.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:39.876 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:39.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:39.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:40.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:40.103 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:40.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:40.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:40.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:40.259 INFO:tasks.rados.rados.2.smithi204.stdout:3705: done (13 left) 2024-09-16T05:24:40.259 INFO:tasks.rados.rados.2.smithi204.stdout:3706: delete oid 34 current snap is 399 2024-09-16T05:24:40.259 INFO:tasks.rados.rados.2.smithi204.stdout:3701: finishing write tid 4 to smithi20457400-31 2024-09-16T05:24:40.259 INFO:tasks.rados.rados.2.smithi204.stdout:3701: finishing write tid 5 to smithi20457400-31 2024-09-16T05:24:40.259 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 1440 (ObjNum 1278 snap 399 seq_num 1278) dirty exists 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3701: left oid 31 (ObjNum 1278 snap 399 seq_num 1278) 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3703: finishing write tid 1 to smithi20457400-17 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3702: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3702: finishing copy_from to smithi20457400-33 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 1039 (ObjNum 1260 snap 393 seq_num 1260) dirty exists 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3703: finishing write tid 2 to smithi20457400-17 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3703: finishing write tid 3 to smithi20457400-17 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3703: finishing write tid 4 to smithi20457400-17 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3703: finishing write tid 5 to smithi20457400-17 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1258 (ObjNum 1279 snap 399 seq_num 1279) dirty exists 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3703: left oid 17 (ObjNum 1279 snap 399 seq_num 1279) 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3704: finishing copy_from racing read to smithi20457400-28 2024-09-16T05:24:40.260 INFO:tasks.rados.rados.2.smithi204.stdout:3704: finishing copy_from to smithi20457400-28 2024-09-16T05:24:40.261 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1476 (ObjNum 1176 snap 360 seq_num 1176) dirty exists 2024-09-16T05:24:40.263 INFO:tasks.rados.rados.2.smithi204.stdout:3701: done (13 left) 2024-09-16T05:24:40.263 INFO:tasks.rados.rados.2.smithi204.stdout:3702: done (12 left) 2024-09-16T05:24:40.263 INFO:tasks.rados.rados.2.smithi204.stdout:3703: done (11 left) 2024-09-16T05:24:40.263 INFO:tasks.rados.rados.2.smithi204.stdout:3704: done (10 left) 2024-09-16T05:24:40.263 INFO:tasks.rados.rados.2.smithi204.stdout:3706: done (9 left) 2024-09-16T05:24:40.263 INFO:tasks.rados.rados.2.smithi204.stdout:3707: write (excl) oid 35 current snap is 399 2024-09-16T05:24:40.264 INFO:tasks.rados.rados.2.smithi204.stdout:3707: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-09-16T05:24:40.271 INFO:tasks.rados.rados.2.smithi204.stdout:3707: writing smithi20457400-35 from 729767 to 1471040 tid 1 2024-09-16T05:24:40.278 INFO:tasks.rados.rados.2.smithi204.stdout:3707: writing smithi20457400-35 from 1927904 to 2653013 tid 2 2024-09-16T05:24:40.282 INFO:tasks.rados.rados.2.smithi204.stdout:3707: writing smithi20457400-35 from 3145089 to 3529767 tid 3 2024-09-16T05:24:40.282 INFO:tasks.rados.rados.2.smithi204.stdout:3708: snap_remove snap 398 2024-09-16T05:24:40.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:40.376 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:40.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:40.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:40 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 21 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:24:40.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:40 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 55 sec, osd.6 has slow ops) 2024-09-16T05:24:40.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:40 smithi032 ceph-mon[27388]: osdmap e929: 12 total, 11 up, 12 in 2024-09-16T05:24:40.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:40.499 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:40.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:40.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:40.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-16T05:24:40.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:40.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:40.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:40 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 21 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:24:40.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:40 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 55 sec, osd.6 has slow ops) 2024-09-16T05:24:40.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:40 smithi204 ceph-mon[30933]: osdmap e929: 12 total, 11 up, 12 in 2024-09-16T05:24:40.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:40.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:40 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 21 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:24:40.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:40 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 55 sec, osd.6 has slow ops) 2024-09-16T05:24:40.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:40 smithi196 ceph-mon[30802]: osdmap e929: 12 total, 11 up, 12 in 2024-09-16T05:24:40.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:40.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:40.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:40.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:40.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:40.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:40.890 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:40.927 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:40.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:41.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:41.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:41.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:41.134 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:41.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:41.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:41.260 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:24:41.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:41.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:41.265 INFO:tasks.rados.rados.2.smithi204.stdout:3708: done (10 left) 2024-09-16T05:24:41.265 INFO:tasks.rados.rados.2.smithi204.stdout:3709: read oid 31 snap 362 2024-09-16T05:24:41.265 INFO:tasks.rados.rados.2.smithi204.stdout:3709: expect (ObjNum 1080 snap 329 seq_num 1080) 2024-09-16T05:24:41.265 INFO:tasks.rados.rados.2.smithi204.stdout:3710: snap_remove snap 386 2024-09-16T05:24:41.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:41.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:41.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:41.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:41 smithi032 ceph-mon[27388]: pgmap v1556: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 224 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 748 KiB/s wr, 12 op/s; 9/232 objects degraded (3.879%) 2024-09-16T05:24:41.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:41 smithi032 ceph-mon[27388]: osdmap e930: 12 total, 11 up, 12 in 2024-09-16T05:24:41.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:41 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/232 objects degraded (3.879%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:41.399 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4067884086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:24:41.490 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:41.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:41.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:41.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:41.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:41.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:41.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:41 smithi204 ceph-mon[30933]: pgmap v1556: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 224 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 748 KiB/s wr, 12 op/s; 9/232 objects degraded (3.879%) 2024-09-16T05:24:41.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:41 smithi204 ceph-mon[30933]: osdmap e930: 12 total, 11 up, 12 in 2024-09-16T05:24:41.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:41 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/232 objects degraded (3.879%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:41.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4067884086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:24:41.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:41 smithi196 ceph-mon[30802]: pgmap v1556: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 224 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 748 KiB/s wr, 12 op/s; 9/232 objects degraded (3.879%) 2024-09-16T05:24:41.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:41 smithi196 ceph-mon[30802]: osdmap e930: 12 total, 11 up, 12 in 2024-09-16T05:24:41.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:41 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/232 objects degraded (3.879%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:41.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4067884086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:24:41.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:41.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:41.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:41.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:41.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:41.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:41.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:41.998 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:42.024 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:42.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:42.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:42.151 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:42.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:42.219 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:42.242 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:42.242 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:42.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:42.264 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:42.264 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:42.269 INFO:tasks.rados.rados.2.smithi204.stdout:3710: done (11 left) 2024-09-16T05:24:42.270 INFO:tasks.rados.rados.2.smithi204.stdout:3711: read oid 19 snap -1 2024-09-16T05:24:42.270 INFO:tasks.rados.rados.2.smithi204.stdout:3711: expect (ObjNum 1247 snap 391 seq_num 1247) 2024-09-16T05:24:42.270 INFO:tasks.rados.rados.2.smithi204.stdout:3712: write (excl) oid 43 current snap is 399 2024-09-16T05:24:42.270 INFO:tasks.rados.rados.2.smithi204.stdout:3712: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-09-16T05:24:42.276 INFO:tasks.rados.rados.2.smithi204.stdout:3712: writing smithi20457400-43 from 746574 to 1163096 tid 1 2024-09-16T05:24:42.282 INFO:tasks.rados.rados.2.smithi204.stdout:3712: writing smithi20457400-43 from 1586386 to 2055153 tid 2 2024-09-16T05:24:42.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:42.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:42.290 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:42.290 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:42.291 INFO:tasks.rados.rados.2.smithi204.stdout:3712: writing smithi20457400-43 from 2772512 to 3402056 tid 3 2024-09-16T05:24:42.291 INFO:tasks.rados.rados.2.smithi204.stdout:3713: read oid 18 snap -1 2024-09-16T05:24:42.291 INFO:tasks.rados.rados.2.smithi204.stdout:3713: expect (ObjNum 1274 snap 398 seq_num 1274) 2024-09-16T05:24:42.291 INFO:tasks.rados.rados.2.smithi204.stdout:3714: copy_from oid 5 from oid 36 current snap is 399 2024-09-16T05:24:42.291 INFO:tasks.rados.rados.2.smithi204.stdout:3707: finishing write tid 1 to smithi20457400-35 2024-09-16T05:24:42.291 INFO:tasks.rados.rados.2.smithi204.stdout:3707: finishing write tid 2 to smithi20457400-35 2024-09-16T05:24:42.291 INFO:tasks.rados.rados.2.smithi204.stdout:3707: finishing write tid 3 to smithi20457400-35 2024-09-16T05:24:42.292 INFO:tasks.rados.rados.2.smithi204.stdout:3707: finishing write tid 4 to smithi20457400-35 2024-09-16T05:24:42.292 INFO:tasks.rados.rados.2.smithi204.stdout:3707: finishing write tid 5 to smithi20457400-35 2024-09-16T05:24:42.292 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 1281 (ObjNum 1281 snap 399 seq_num 1281) dirty exists 2024-09-16T05:24:42.292 INFO:tasks.rados.rados.2.smithi204.stdout:3707: left oid 35 (ObjNum 1281 snap 399 seq_num 1281) 2024-09-16T05:24:42.292 INFO:tasks.rados.rados.2.smithi204.stdout:3709: expect (ObjNum 1080 snap 329 seq_num 1080) 2024-09-16T05:24:42.312 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:42.312 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:42.334 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:42.334 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:42.357 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:42.358 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:42.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:42.386 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:42.390 INFO:tasks.rados.rados.2.smithi204.stdout:3711: expect (ObjNum 1247 snap 391 seq_num 1247) 2024-09-16T05:24:42.405 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:42.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:42.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:42.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:42.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:42 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:42.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4067884086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:24:42.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:42 smithi032 ceph-mon[27388]: osdmap e931: 12 total, 11 up, 12 in 2024-09-16T05:24:42.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:42 smithi032 ceph-mon[27388]: 2.55 scrub starts 2024-09-16T05:24:42.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:42 smithi032 ceph-mon[27388]: 2.55 scrub ok 2024-09-16T05:24:42.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/915113340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:42.449 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:42.449 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3712: finishing write tid 1 to smithi20457400-43 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3712: finishing write tid 2 to smithi20457400-43 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3707: done (14 left) 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3709: done (13 left) 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3711: done (12 left) 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3715: snap_create 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3714: finishing copy_from to smithi20457400-5 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3714: got expected ENOENT (src dne) 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3714: finishing copy_from racing read to smithi20457400-5 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3712: finishing write tid 3 to smithi20457400-43 2024-09-16T05:24:42.489 INFO:tasks.rados.rados.2.smithi204.stdout:3712: finishing write tid 4 to smithi20457400-43 2024-09-16T05:24:42.490 INFO:tasks.rados.rados.2.smithi204.stdout:3712: finishing write tid 5 to smithi20457400-43 2024-09-16T05:24:42.490 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1480 (ObjNum 1282 snap 399 seq_num 1282) dirty exists 2024-09-16T05:24:42.490 INFO:tasks.rados.rados.2.smithi204.stdout:3712: left oid 43 (ObjNum 1282 snap 399 seq_num 1282) 2024-09-16T05:24:42.490 INFO:tasks.rados.rados.2.smithi204.stdout:3713: expect (ObjNum 1274 snap 398 seq_num 1274) 2024-09-16T05:24:42.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:42.534 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:42.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:42.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:42.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:42.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:42 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:42.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4067884086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:24:42.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:42 smithi204 ceph-mon[30933]: osdmap e931: 12 total, 11 up, 12 in 2024-09-16T05:24:42.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:42 smithi204 ceph-mon[30933]: 2.55 scrub starts 2024-09-16T05:24:42.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:42 smithi204 ceph-mon[30933]: 2.55 scrub ok 2024-09-16T05:24:42.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/915113340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:42.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:42 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:42.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4067884086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:24:42.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:42 smithi196 ceph-mon[30802]: osdmap e931: 12 total, 11 up, 12 in 2024-09-16T05:24:42.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:42 smithi196 ceph-mon[30802]: 2.55 scrub starts 2024-09-16T05:24:42.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:42 smithi196 ceph-mon[30802]: 2.55 scrub ok 2024-09-16T05:24:42.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/915113340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:42.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:42.785 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:42.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:42.849 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:24:42.849 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:24:42.849 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:24:42.849 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:24:42.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.849 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:606 2.48 e365/322 2:12e92f6c:::smithi20457400-1:79 v 365'161, mlcod=362'157)", 2024-09-16T05:24:42.849 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:24:42.849 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 567.369211812, 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64187130800000003, 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.851 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:24:42.851 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.851 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071136+0000", 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0200000000000001e-07 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.071139+0000", 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5609999999999999e-06 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712578+0000", 2024-09-16T05:24:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.64143852199999996 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712582+0000", 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4000000000000002e-06 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712651+0000", 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9102000000000003e-05 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712670+0000", 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9165999999999999e-05 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.712968+0000", 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00029722899999999998 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.713007+0000", 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9125999999999998e-05 2024-09-16T05:24:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.9.0:614 2.48 e366/322 2:12e92f6c:::smithi20457400-1:79 v 366'163, mlcod=364'159)", 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 566.71978458199999, 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11945663500000001, 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720563+0000", 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9699999999999996e-07 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.720568+0000", 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6870000000000002e-06 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839371+0000", 2024-09-16T05:24:42.857 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.11880265399999999 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839384+0000", 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3156000000000001e-05 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839531+0000", 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00014728399999999999 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.858 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839575+0000", 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4223e-05 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.839981+0000", 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040524499999999999 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:15:15.840019+0000", 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8788999999999997e-05 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.859 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4629 2.1e 2:7b642727:::smithi20457400-40:head [copy-from ver 511] snapc bf=[bf,be] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 420.68782306399999, 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01647931, 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:42.860 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:42.861 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4629 2024-09-16T05:24:42.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.861 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.866 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.866 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:24:42.866 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.867 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.867 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:24:42.867 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.868 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.868 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752526+0000", 2024-09-16T05:24:42.868 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.894e-06 2024-09-16T05:24:42.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.869 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752536+0000", 2024-09-16T05:24:42.869 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:24:42.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.869 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752555+0000", 2024-09-16T05:24:42.870 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8907999999999999e-05 2024-09-16T05:24:42.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.870 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.870 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752564+0000", 2024-09-16T05:24:42.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:24:42.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.871 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.871 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752638+0000", 2024-09-16T05:24:42.871 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3343999999999994e-05 2024-09-16T05:24:42.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.872 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:24:42.872 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752646+0000", 2024-09-16T05:24:42.872 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7690000000000002e-06 2024-09-16T05:24:42.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.873 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.873 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754646+0000", 2024-09-16T05:24:42.873 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001999331 2024-09-16T05:24:42.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.874 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.874 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754740+0000", 2024-09-16T05:24:42.874 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3778999999999998e-05 2024-09-16T05:24:42.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.875 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:24:42.875 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.764207+0000", 2024-09-16T05:24:42.875 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094672899999999997 2024-09-16T05:24:42.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.876 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:42.876 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768754+0000", 2024-09-16T05:24:42.876 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045470199999999997 2024-09-16T05:24:42.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.877 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:42.877 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768860+0000", 2024-09-16T05:24:42.877 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010586600000000001 2024-09-16T05:24:42.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.878 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768948+0000", 2024-09-16T05:24:42.878 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8503000000000003e-05 2024-09-16T05:24:42.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.878 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.879 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.769004+0000", 2024-09-16T05:24:42.879 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5362000000000003e-05 2024-09-16T05:24:42.879 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.879 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.879 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.880 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4630 2.1e 2:7b642727:::smithi20457400-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:24:42.880 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:24:42.880 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 420.68776531399999, 2024-09-16T05:24:42.880 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015483190000000001, 2024-09-16T05:24:42.880 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4630 2024-09-16T05:24:42.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.882 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.882 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.882 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:24:42.882 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.883 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.883 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:24:42.883 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.884 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.884 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:24:42.884 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.34e-07 2024-09-16T05:24:42.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.885 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.885 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752584+0000", 2024-09-16T05:24:42.885 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.956e-06 2024-09-16T05:24:42.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.886 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752591+0000", 2024-09-16T05:24:42.886 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2580000000000001e-06 2024-09-16T05:24:42.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.886 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752594+0000", 2024-09-16T05:24:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6830000000000001e-06 2024-09-16T05:24:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.887 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752703+0000", 2024-09-16T05:24:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108818 2024-09-16T05:24:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:24:42.888 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752708+0000", 2024-09-16T05:24:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.561e-06 2024-09-16T05:24:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754913+0000", 2024-09-16T05:24:42.889 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022055680000000002 2024-09-16T05:24:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754965+0000", 2024-09-16T05:24:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2222000000000002e-05 2024-09-16T05:24:42.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767827+0000", 2024-09-16T05:24:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012861097 2024-09-16T05:24:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767929+0000", 2024-09-16T05:24:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010275799999999999 2024-09-16T05:24:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768065+0000", 2024-09-16T05:24:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013593500000000001 2024-09-16T05:24:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:142 2.1e e571/291 ondisk, result = 0)", 2024-09-16T05:24:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 360.67618915899999, 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55541789799999997, 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764159+0000", 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764162+0000", 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599999999999998e-06 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319278+0000", 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55511595899999999 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319296+0000", 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7900000000000001e-05 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319440+0000", 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143175 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319450+0000", 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0705e-05 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.898 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319576+0000", 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012584299999999999 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5587 2.34 e583/278 2:2d3f13f8:::smithi20457400-7:head v 583'466, mlcod=583'463)", 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 349.63324280799998, 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14534661199999999, 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:24:42.900 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807109+0000", 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2030000000000002e-06 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951917+0000", 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14480800699999999 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.901 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951927+0000", 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0115000000000001e-05 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951990+0000", 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2689999999999998e-05 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952011+0000", 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.084e-05 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952413+0000", 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040190199999999999 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.903 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952451+0000", 2024-09-16T05:24:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8302e-05 2024-09-16T05:24:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.904 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:383 2.29 e617/253 2:9662c29c:::smithi20457400-9:d8 v 617'830, mlcod=616'824)", 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 315.65755462700002, 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038031031999999999, 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782794+0000", 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3500000000000005e-07 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782801+0000", 2024-09-16T05:24:42.906 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1210000000000001e-06 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819928+0000", 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037126740999999998 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819938+0000", 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0845e-05 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820001+0000", 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2786999999999994e-05 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820023+0000", 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.213e-05 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820771+0000", 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074818699999999998 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820824+0000", 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2286e-05 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.909 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:199 2.1e e714/291 ondisk, result = 0)", 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 218.62813851199999, 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222874399, 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:42.910 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:24:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.911 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2099999999999997e-07 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812213+0000", 2024-09-16T05:24:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3859999999999999e-06 2024-09-16T05:24:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.913 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034914+0000", 2024-09-16T05:24:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222700804 2024-09-16T05:24:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.914 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034920+0000", 2024-09-16T05:24:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6579999999999996e-06 2024-09-16T05:24:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034968+0000", 2024-09-16T05:24:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7833000000000001e-05 2024-09-16T05:24:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034975+0000", 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0400000000000004e-06 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.035083+0000", 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108157 2024-09-16T05:24:42.916 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7351 2.29 2:94d33c56:::smithi20457400-8:head [write 410368~646278] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:36.863973+0000", 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 185.57637388699999, 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084857950000000008, 2024-09-16T05:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7351 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.863973+0000", 2024-09-16T05:24:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.863973+0000", 2024-09-16T05:24:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.863975+0000", 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9630000000000001e-06 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.864689+0000", 2024-09-16T05:24:42.920 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00071403299999999999 2024-09-16T05:24:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.864704+0000", 2024-09-16T05:24:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4307999999999999e-05 2024-09-16T05:24:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.864712+0000", 2024-09-16T05:24:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.67e-06 2024-09-16T05:24:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.922 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.865089+0000", 2024-09-16T05:24:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00037642600000000002 2024-09-16T05:24:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.923 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.865172+0000", 2024-09-16T05:24:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.3288000000000006e-05 2024-09-16T05:24:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:24:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.865331+0000", 2024-09-16T05:24:42.924 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000158765 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.867114+0000", 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017832880000000001 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:42.925 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.872161+0000", 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050464459999999996 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.872261+0000", 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010050799999999999 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.926 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.872459+0000", 2024-09-16T05:24:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019809999999999999 2024-09-16T05:24:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.927 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7352 2.29 2:94d33c56:::smithi20457400-8:head [write 1472267~475349] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:36.868856+0000", 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 185.57149127700001, 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0087654470000000009, 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:42.928 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7352 2024-09-16T05:24:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.868856+0000", 2024-09-16T05:24:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.868856+0000", 2024-09-16T05:24:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.930 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.868858+0000", 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.221e-06 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869362+0000", 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00050372899999999996 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.931 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869384+0000", 2024-09-16T05:24:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1503000000000001e-05 2024-09-16T05:24:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869396+0000", 2024-09-16T05:24:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2016e-05 2024-09-16T05:24:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869477+0000", 2024-09-16T05:24:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.1207999999999999e-05 2024-09-16T05:24:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869610+0000", 2024-09-16T05:24:42.933 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013292600000000001 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.869778+0000", 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016856900000000001 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.871275+0000", 2024-09-16T05:24:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001496961 2024-09-16T05:24:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.877452+0000", 2024-09-16T05:24:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0061771819999999998 2024-09-16T05:24:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.877510+0000", 2024-09-16T05:24:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7181e-05 2024-09-16T05:24:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.877622+0000", 2024-09-16T05:24:42.936 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111951 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7414 2.5a 2:5ae26c04:::smithi20457400-32:head [write 1423159~486356] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 177.60816634, 2024-09-16T05:24:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010834233, 2024-09-16T05:24:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7414 2024-09-16T05:24:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:24:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832183+0000", 2024-09-16T05:24:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6589999999999999e-06 2024-09-16T05:24:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832584+0000", 2024-09-16T05:24:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040119599999999999 2024-09-16T05:24:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832598+0000", 2024-09-16T05:24:42.941 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3736e-05 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832607+0000", 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1870000000000003e-06 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832655+0000", 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8043000000000001e-05 2024-09-16T05:24:42.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832725+0000", 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0136999999999996e-05 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832832+0000", 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010700399999999999 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.943 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834013+0000", 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011807479999999999 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842930+0000", 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0089172439999999995 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842977+0000", 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7547999999999999e-05 2024-09-16T05:24:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.843015+0000", 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7731000000000002e-05 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:7414 2.5a e755/742 ondisk, result = 0)", 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 177.60613670000001, 2024-09-16T05:24:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088119329999999992, 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.946 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834212+0000", 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2699999999999998e-07 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834216+0000", 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2139999999999998e-06 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.947 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842854+0000", 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086381679999999999 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842859+0000", 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842913+0000", 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3235000000000001e-05 2024-09-16T05:24:42.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842922+0000", 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.431e-06 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.843023+0000", 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100485 2024-09-16T05:24:42.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7416 2.5a 2:5ae26c04:::smithi20457400-32:head [setxattr _header (58) in=7b,truncate 3311210] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 177.59972639599999, 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.075538587000000004, 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7416 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.951 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840622+0000", 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6400000000000001e-07 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840626+0000", 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8589999999999998e-06 2024-09-16T05:24:42.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840639+0000", 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2982999999999999e-05 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840644+0000", 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.57e-06 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.841987+0000", 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013431599999999999 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842059+0000", 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1450000000000002e-05 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842166+0000", 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107421 2024-09-16T05:24:42.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842788+0000", 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062151399999999999 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916025+0000", 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073237675000000002 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916092+0000", 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6631999999999995e-05 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916160+0000", 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7459000000000006e-05 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7417 2.5a 2:5ae26c04:::smithi20457400-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 177.59967011500001, 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.075671839000000005, 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:42.957 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7417 2024-09-16T05:24:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:24:42.958 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:24:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840678+0000", 2024-09-16T05:24:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9800000000000003e-07 2024-09-16T05:24:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840682+0000", 2024-09-16T05:24:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9310000000000001e-06 2024-09-16T05:24:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840695+0000", 2024-09-16T05:24:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2672e-05 2024-09-16T05:24:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840705+0000", 2024-09-16T05:24:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0498e-05 2024-09-16T05:24:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.962 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842425+0000", 2024-09-16T05:24:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001719497 2024-09-16T05:24:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842530+0000", 2024-09-16T05:24:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105824 2024-09-16T05:24:42.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916202+0000", 2024-09-16T05:24:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073671367000000001 2024-09-16T05:24:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916273+0000", 2024-09-16T05:24:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1316000000000006e-05 2024-09-16T05:24:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916349+0000", 2024-09-16T05:24:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5935999999999999e-05 2024-09-16T05:24:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.965 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:7416 2.5a e755/742 ondisk, result = 0)", 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 177.597395073, 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073216748999999998, 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.966 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:24:42.967 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842953+0000", 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4000000000000002e-07 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842956+0000", 2024-09-16T05:24:42.968 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.915928+0000", 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.072972180999999997 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.915937+0000", 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3689999999999996e-06 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916004+0000", 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6836999999999997e-05 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916017+0000", 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3200000000000001e-05 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.970 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916169+0000", 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000151512 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:251 2.29 e762/745 ondisk, result = 0)", 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 170.51787285200001, 2024-09-16T05:24:42.971 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.502823558, 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:24:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.973 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922475+0000", 2024-09-16T05:24:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3000000000000001e-07 2024-09-16T05:24:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922478+0000", 2024-09-16T05:24:42.974 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.18e-06 2024-09-16T05:24:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425018+0000", 2024-09-16T05:24:42.975 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.50254011200000004 2024-09-16T05:24:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425039+0000", 2024-09-16T05:24:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0832999999999999e-05 2024-09-16T05:24:42.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425166+0000", 2024-09-16T05:24:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012726 2024-09-16T05:24:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425177+0000", 2024-09-16T05:24:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0217999999999999e-05 2024-09-16T05:24:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.978 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425298+0000", 2024-09-16T05:24:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121425 2024-09-16T05:24:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.7.0:1425 2.2e e786/278 2:75dc7feb:::smithi20457400-28:13f v 786'738, mlcod=781'732)", 2024-09-16T05:24:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:24:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 146.58986052500001, 2024-09-16T05:24:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72979714900000003, 2024-09-16T05:24:42.980 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.981 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:24:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:24:42.982 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:24:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:24:42.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850493+0000", 2024-09-16T05:24:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1120000000000001e-06 2024-09-16T05:24:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579433+0000", 2024-09-16T05:24:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72893962899999998 2024-09-16T05:24:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579449+0000", 2024-09-16T05:24:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6059999999999999e-05 2024-09-16T05:24:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579568+0000", 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118911 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579618+0000", 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0436000000000003e-05 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.580252+0000", 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063313400000000004 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.580284+0000", 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2381e-05 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.2e,QREQUEST,e:797,from:7,reservation_nonce:8)", 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 134.09794570700001, 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27875738500000002, 2024-09-16T05:24:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:24:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342404+0000", 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9300000000000002e-06 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342412+0000", 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6319999999999997e-06 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.620932+0000", 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27851996800000001 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.620943+0000", 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0892e-05 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621083+0000", 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013946999999999999 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621110+0000", 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7934000000000001e-05 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621159+0000", 2024-09-16T05:24:42.993 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8559e-05 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:8050 2.2e e804/278 2:75dc7feb:::smithi20457400-28:head v 804'753, mlcod=804'750)", 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 128.493802687, 2024-09-16T05:24:42.994 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32122192599999999, 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:42.995 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:24:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:24:42.996 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9999999999999997e-07 2024-09-16T05:24:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946550+0000", 2024-09-16T05:24:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4739999999999999e-06 2024-09-16T05:24:42.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.266951+0000", 2024-09-16T05:24:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32040173 2024-09-16T05:24:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.266959+0000", 2024-09-16T05:24:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0199999999999994e-06 2024-09-16T05:24:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267034+0000", 2024-09-16T05:24:42.999 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4408999999999999e-05 2024-09-16T05:24:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267065+0000", 2024-09-16T05:24:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1538000000000002e-05 2024-09-16T05:24:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267727+0000", 2024-09-16T05:24:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066189099999999996 2024-09-16T05:24:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:43.001 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267767+0000", 2024-09-16T05:24:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9264000000000002e-05 2024-09-16T05:24:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:690 2.29 e855/767 2:96dc17ef:::smithi20457400-10:165 v 855'1417, mlcod=852'1410)", 2024-09-16T05:24:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:24:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.565652826000004, 2024-09-16T05:24:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72853946700000005, 2024-09-16T05:24:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:24:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:24:43.003 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:24:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:24:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:24:43.004 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:24:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:24:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:24:43.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2500000000000005e-07 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874701+0000", 2024-09-16T05:24:43.006 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8100000000000003e-06 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.601981+0000", 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72728016900000003 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.601998+0000", 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6753e-05 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.007 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:24:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.602123+0000", 2024-09-16T05:24:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012546000000000001 2024-09-16T05:24:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:24:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.602147+0000", 2024-09-16T05:24:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3597000000000001e-05 2024-09-16T05:24:43.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.603190+0000", 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001043228 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.603234+0000", 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3724999999999999e-05 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:43.009 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:24:43.010 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:24:43.010 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:24:43.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:43.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:43.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:43.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:43.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:43.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:43.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:42 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:42.855+0000 7f6af1ad6640 -1 osd.6 932 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:43.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:43.277 INFO:tasks.rados.rados.2.smithi204.stdout:3712: done (12 left) 2024-09-16T05:24:43.277 INFO:tasks.rados.rados.2.smithi204.stdout:3713: done (11 left) 2024-09-16T05:24:43.277 INFO:tasks.rados.rados.2.smithi204.stdout:3714: done (10 left) 2024-09-16T05:24:43.277 INFO:tasks.rados.rados.2.smithi204.stdout:3715: done (9 left) 2024-09-16T05:24:43.277 INFO:tasks.rados.rados.2.smithi204.stdout:3716: read oid 3 snap -1 2024-09-16T05:24:43.277 INFO:tasks.rados.rados.2.smithi204.stdout:3716: expect (ObjNum 1236 snap 384 seq_num 1236) 2024-09-16T05:24:43.277 INFO:tasks.rados.rados.2.smithi204.stdout:3717: delete oid 44 current snap is 400 2024-09-16T05:24:43.279 INFO:tasks.rados.rados.2.smithi204.stdout:3717: done (10 left) 2024-09-16T05:24:43.279 INFO:tasks.rados.rados.2.smithi204.stdout:3718: read oid 27 snap 399 2024-09-16T05:24:43.279 INFO:tasks.rados.rados.2.smithi204.stdout:3718: expect (ObjNum 1232 snap 380 seq_num 1232) 2024-09-16T05:24:43.279 INFO:tasks.rados.rados.2.smithi204.stdout:3719: write (excl) oid 23 current snap is 400 2024-09-16T05:24:43.279 INFO:tasks.rados.rados.2.smithi204.stdout:3719: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2024-09-16T05:24:43.286 INFO:tasks.rados.rados.2.smithi204.stdout:3719: writing smithi20457400-23 from 780188 to 1347208 tid 1 2024-09-16T05:24:43.292 INFO:tasks.rados.rados.2.smithi204.stdout:3719: writing smithi20457400-23 from 2103350 to 2575786 tid 2 2024-09-16T05:24:43.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:43.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:3719: writing smithi20457400-23 from 3227358 to 3580188 tid 3 2024-09-16T05:24:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:3720: write (excl) oid 5 current snap is 400 2024-09-16T05:24:43.298 INFO:tasks.rados.rados.2.smithi204.stdout:3720: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-16T05:24:43.306 INFO:tasks.rados.rados.2.smithi204.stdout:3720: writing smithi20457400-5 from 796995 to 1555617 tid 1 2024-09-16T05:24:43.314 INFO:tasks.rados.rados.2.smithi204.stdout:3720: writing smithi20457400-5 from 2161832 to 2894279 tid 2 2024-09-16T05:24:43.314 INFO:tasks.rados.rados.2.smithi204.stdout:3721: read oid 11 snap -1 2024-09-16T05:24:43.314 INFO:tasks.rados.rados.2.smithi204.stdout:3721: expect (ObjNum 1236 snap 384 seq_num 1236) 2024-09-16T05:24:43.314 INFO:tasks.rados.rados.2.smithi204.stdout:3722: write (excl) oid 37 current snap is 400 2024-09-16T05:24:43.314 INFO:tasks.rados.rados.2.smithi204.stdout:3722: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2024-09-16T05:24:43.318 INFO:tasks.rados.rados.2.smithi204.stdout:3722: writing smithi20457400-37 from 413802 to 847673 tid 1 2024-09-16T05:24:43.324 INFO:tasks.rados.rados.2.smithi204.stdout:3722: writing smithi20457400-37 from 1420314 to 1896419 tid 2 2024-09-16T05:24:43.331 INFO:tasks.rados.rados.2.smithi204.stdout:3722: writing smithi20457400-37 from 2482204 to 3273189 tid 3 2024-09-16T05:24:43.332 INFO:tasks.rados.rados.2.smithi204.stdout:3723: write oid 31 current snap is 400 2024-09-16T05:24:43.332 INFO:tasks.rados.rados.2.smithi204.stdout:3723: seq_num 1287 ranges {430609=509120,1478796=736116,2909627=602257} 2024-09-16T05:24:43.336 INFO:tasks.rados.rados.2.smithi204.stdout:3723: writing smithi20457400-31 from 430609 to 939729 tid 1 2024-09-16T05:24:43.343 INFO:tasks.rados.rados.2.smithi204.stdout:3723: writing smithi20457400-31 from 1478796 to 2214912 tid 2 2024-09-16T05:24:43.351 INFO:tasks.rados.rados.2.smithi204.stdout:3723: writing smithi20457400-31 from 2909627 to 3511884 tid 3 2024-09-16T05:24:43.351 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:24:43.351 INFO:tasks.rados.rados.2.smithi204.stdout:3716: expect (ObjNum 1236 snap 384 seq_num 1236) 2024-09-16T05:24:43.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:43.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:43.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:43.470 INFO:tasks.rados.rados.2.smithi204.stdout:3718: expect (ObjNum 1232 snap 380 seq_num 1232) 2024-09-16T05:24:43.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:43.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:43.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:43.556 INFO:tasks.rados.rados.2.smithi204.stdout:3719: finishing write tid 1 to smithi20457400-23 2024-09-16T05:24:43.556 INFO:tasks.rados.rados.2.smithi204.stdout:3719: finishing write tid 2 to smithi20457400-23 2024-09-16T05:24:43.556 INFO:tasks.rados.rados.2.smithi204.stdout:3719: finishing write tid 3 to smithi20457400-23 2024-09-16T05:24:43.556 INFO:tasks.rados.rados.2.smithi204.stdout:3719: finishing write tid 4 to smithi20457400-23 2024-09-16T05:24:43.556 INFO:tasks.rados.rados.2.smithi204.stdout:3719: finishing write tid 5 to smithi20457400-23 2024-09-16T05:24:43.556 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 869 (ObjNum 1284 snap 400 seq_num 1284) dirty exists 2024-09-16T05:24:43.556 INFO:tasks.rados.rados.2.smithi204.stdout:3719: left oid 23 (ObjNum 1284 snap 400 seq_num 1284) 2024-09-16T05:24:43.557 INFO:tasks.rados.rados.2.smithi204.stdout:3720: finishing write tid 1 to smithi20457400-5 2024-09-16T05:24:43.557 INFO:tasks.rados.rados.2.smithi204.stdout:3716: done (15 left) 2024-09-16T05:24:43.557 INFO:tasks.rados.rados.2.smithi204.stdout:3718: done (14 left) 2024-09-16T05:24:43.557 INFO:tasks.rados.rados.2.smithi204.stdout:3719: done (13 left) 2024-09-16T05:24:43.557 INFO:tasks.rados.rados.2.smithi204.stdout:3724: snap_remove snap 399 2024-09-16T05:24:43.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:43.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: 2.51 scrub starts 2024-09-16T05:24:43.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: 2.51 scrub ok 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: 2.8 scrub starts 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: 2.8 scrub ok 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: pgmap v1559: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 244 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.2 MiB/s wr, 13 op/s; 9/244 objects degraded (3.689%) 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: osdmap e932: 12 total, 11 up, 12 in 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: 2.5a scrub starts 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: 2.5a scrub ok 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:24:43.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:43 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:24:43.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: 2.51 scrub starts 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: 2.51 scrub ok 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: 2.8 scrub starts 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: 2.8 scrub ok 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: pgmap v1559: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 244 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.2 MiB/s wr, 13 op/s; 9/244 objects degraded (3.689%) 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: osdmap e932: 12 total, 11 up, 12 in 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: 2.5a scrub starts 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: 2.5a scrub ok 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:24:43.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:43 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:24:43.672 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:43.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:43.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:24:43.713 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: 2.51 scrub starts 2024-09-16T05:24:43.713 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: 2.51 scrub ok 2024-09-16T05:24:43.713 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:24:43.713 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:24:43.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: 2.8 scrub starts 2024-09-16T05:24:43.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: 2.8 scrub ok 2024-09-16T05:24:43.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: pgmap v1559: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 244 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.2 MiB/s wr, 13 op/s; 9/244 objects degraded (3.689%) 2024-09-16T05:24:43.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: osdmap e932: 12 total, 11 up, 12 in 2024-09-16T05:24:43.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: 2.5a scrub starts 2024-09-16T05:24:43.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: 2.5a scrub ok 2024-09-16T05:24:43.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:24:43.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:24:43.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:43.796 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:43.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:43.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:43.919 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:43.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:43.941 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:43.988 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:43 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:43.897+0000 7f6af1ad6640 -1 osd.6 933 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:43.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:24:44.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:44.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:44.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:44.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:44.166 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:44.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:44.268 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:44.282 INFO:tasks.rados.rados.2.smithi204.stdout:3724: done (13 left) 2024-09-16T05:24:44.282 INFO:tasks.rados.rados.2.smithi204.stdout:3725: write (excl) oid 29 current snap is 400 2024-09-16T05:24:44.282 INFO:tasks.rados.rados.2.smithi204.stdout:3725: seq_num 1288 ranges {447416=584369,1537278=479774,2537050=529882,3524784=122632} 2024-09-16T05:24:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:44.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:44.288 INFO:tasks.rados.rados.2.smithi204.stdout:3725: writing smithi20457400-29 from 447416 to 1031785 tid 1 2024-09-16T05:24:44.293 INFO:tasks.rados.rados.2.smithi204.stdout:3725: writing smithi20457400-29 from 1537278 to 2017052 tid 2 2024-09-16T05:24:44.294 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:44.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:44.299 INFO:tasks.rados.rados.2.smithi204.stdout:3725: writing smithi20457400-29 from 2537050 to 3066932 tid 3 2024-09-16T05:24:44.305 INFO:tasks.rados.rados.2.smithi204.stdout:3725: writing smithi20457400-29 from 3524784 to 3647416 tid 4 2024-09-16T05:24:44.305 INFO:tasks.rados.rados.2.smithi204.stdout:3726: snap_remove snap 378 2024-09-16T05:24:44.396 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:44.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:44.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:24:44.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:24:44.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-mon[30802]: 2.53 scrub starts 2024-09-16T05:24:44.400 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-mon[30802]: 2.53 scrub ok 2024-09-16T05:24:44.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-mon[30802]: osdmap e933: 12 total, 11 up, 12 in 2024-09-16T05:24:44.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:24:44.401 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:24:44.401 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:24:44.401 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:24:44.401 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:24:44.401 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:24:44.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:44.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:44.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:44.533 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:44.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:44 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:44.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:44 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:24:44.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:44 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:24:44.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:44 smithi032 ceph-mon[27388]: 2.53 scrub starts 2024-09-16T05:24:44.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:44 smithi032 ceph-mon[27388]: 2.53 scrub ok 2024-09-16T05:24:44.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:44 smithi032 ceph-mon[27388]: osdmap e933: 12 total, 11 up, 12 in 2024-09-16T05:24:44.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:44 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:24:44.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:44 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:24:44.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:44.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:44.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:44.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:44.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:44 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:44.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:44 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:24:44.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:44 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:24:44.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:44 smithi204 ceph-mon[30933]: 2.53 scrub starts 2024-09-16T05:24:44.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:44 smithi204 ceph-mon[30933]: 2.53 scrub ok 2024-09-16T05:24:44.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:44 smithi204 ceph-mon[30933]: osdmap e933: 12 total, 11 up, 12 in 2024-09-16T05:24:44.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:44 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:24:44.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:44 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:24:44.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:44.791 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:44.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:44.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:44.917 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:44.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:45.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:45.041 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:45.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:45.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:45.137 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:44 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:44.860+0000 7f6af1ad6640 -1 osd.6 934 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:45.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:45.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:24:45.166 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:45.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:45.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:45.288 INFO:tasks.rados.rados.2.smithi204.stdout:3726: done (14 left) 2024-09-16T05:24:45.288 INFO:tasks.rados.rados.2.smithi204.stdout:3727: write (excl) oid 41 current snap is 400 2024-09-16T05:24:45.289 INFO:tasks.rados.rados.2.smithi204.stdout:3727: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2024-09-16T05:24:45.295 INFO:tasks.rados.rados.2.smithi204.stdout:3727: writing smithi20457400-41 from 464223 to 1123841 tid 1 2024-09-16T05:24:45.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:45.296 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:45.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:45.303 INFO:tasks.rados.rados.2.smithi204.stdout:3727: writing smithi20457400-41 from 1879407 to 2619192 tid 2 2024-09-16T05:24:45.309 INFO:tasks.rados.rados.2.smithi204.stdout:3727: writing smithi20457400-41 from 3248120 to 3664223 tid 3 2024-09-16T05:24:45.309 INFO:tasks.rados.rados.2.smithi204.stdout:3728: delete oid 12 current snap is 400 2024-09-16T05:24:45.309 INFO:tasks.rados.rados.2.smithi204.stdout:3721: expect (ObjNum 1236 snap 384 seq_num 1236) 2024-09-16T05:24:45.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-16T05:24:45.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:45.413 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:45.428 INFO:tasks.rados.rados.2.smithi204.stdout:3720: finishing write tid 2 to smithi20457400-5 2024-09-16T05:24:45.428 INFO:tasks.rados.rados.2.smithi204.stdout:3720: finishing write tid 3 to smithi20457400-5 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3720: finishing write tid 4 to smithi20457400-5 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 640 (ObjNum 1285 snap 400 seq_num 1285) dirty exists 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3720: left oid 5 (ObjNum 1285 snap 400 seq_num 1285) 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3722: finishing write tid 1 to smithi20457400-37 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3720: done (15 left) 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3721: done (14 left) 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3728: done (13 left) 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3729: read oid 20 snap -1 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3729: expect deleted 2024-09-16T05:24:45.429 INFO:tasks.rados.rados.2.smithi204.stdout:3730: snap_remove snap 392 2024-09-16T05:24:45.437 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:45.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:45.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: pgmap v1562: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 248 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.9 MiB/s wr, 23 op/s; 9/250 objects degraded (3.600%) 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: osdmap e934: 12 total, 11 up, 12 in 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2.54 scrub starts 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2.54 scrub ok 2024-09-16T05:24:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:24:45.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:24:45.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:45.555 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:24:45.555 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:24:45.555 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:45.555 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: pgmap v1562: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 248 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.9 MiB/s wr, 23 op/s; 9/250 objects degraded (3.600%) 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: osdmap e934: 12 total, 11 up, 12 in 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2.54 scrub starts 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2.54 scrub ok 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:24:45.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:45 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:24:45.560 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:45.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:45.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:45.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:24:45.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:24:45.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: pgmap v1562: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 248 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.9 MiB/s wr, 23 op/s; 9/250 objects degraded (3.600%) 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: osdmap e934: 12 total, 11 up, 12 in 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2.54 scrub starts 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2.54 scrub ok 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:24:45.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:45 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:24:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:45.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:45.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:45.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:45.836 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:45.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:45.858 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:45.890 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:24:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:24:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:24:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:24:45.891 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:24:45.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:45.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:45.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:46.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:46.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:46.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:46.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:46.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:46.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:46.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:45 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:45.856+0000 7f6af1ad6640 -1 osd.6 935 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:46.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:24:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:24:46.303 INFO:tasks.rados.rados.2.smithi204.stdout:3730: done (14 left) 2024-09-16T05:24:46.303 INFO:tasks.rados.rados.2.smithi204.stdout:3731: read oid 21 snap -1 2024-09-16T05:24:46.304 INFO:tasks.rados.rados.2.smithi204.stdout:3731: expect deleted 2024-09-16T05:24:46.304 INFO:tasks.rados.rados.2.smithi204.stdout:3732: write (excl) oid 34 current snap is 400 2024-09-16T05:24:46.304 INFO:tasks.rados.rados.2.smithi204.stdout:3722: finishing write tid 2 to smithi20457400-37 2024-09-16T05:24:46.304 INFO:tasks.rados.rados.2.smithi204.stdout:3732: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-09-16T05:24:46.309 INFO:tasks.rados.rados.2.smithi204.stdout:3732: writing smithi20457400-34 from 497837 to 907953 tid 1 2024-09-16T05:24:46.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:46.317 INFO:tasks.rados.rados.2.smithi204.stdout:3732: writing smithi20457400-34 from 1596371 to 2339825 tid 2 2024-09-16T05:24:46.325 INFO:tasks.rados.rados.2.smithi204.stdout:3732: writing smithi20457400-34 from 2902966 to 3666664 tid 3 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3722: finishing write tid 3 to smithi20457400-37 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3723: finishing write tid 1 to smithi20457400-31 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3723: finishing write tid 2 to smithi20457400-31 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3723: finishing write tid 3 to smithi20457400-31 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3723: finishing write tid 4 to smithi20457400-31 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3723: finishing write tid 5 to smithi20457400-31 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 31 v 1444 (ObjNum 1287 snap 400 seq_num 1287) dirty exists 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3723: left oid 31 (ObjNum 1287 snap 400 seq_num 1287) 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3722: finishing write tid 4 to smithi20457400-37 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3722: finishing write tid 5 to smithi20457400-37 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1392 (ObjNum 1286 snap 400 seq_num 1286) dirty exists 2024-09-16T05:24:46.326 INFO:tasks.rados.rados.2.smithi204.stdout:3722: left oid 37 (ObjNum 1286 snap 400 seq_num 1286) 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3725: finishing write tid 1 to smithi20457400-29 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3725: finishing write tid 2 to smithi20457400-29 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3725: finishing write tid 3 to smithi20457400-29 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3722: done (15 left) 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3723: done (14 left) 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3733: delete oid 46 current snap is 400 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3725: finishing write tid 4 to smithi20457400-29 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3725: finishing write tid 5 to smithi20457400-29 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3725: finishing write tid 6 to smithi20457400-29 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1322 (ObjNum 1288 snap 400 seq_num 1288) dirty exists 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3725: left oid 29 (ObjNum 1288 snap 400 seq_num 1288) 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3727: finishing write tid 1 to smithi20457400-41 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3727: finishing write tid 2 to smithi20457400-41 2024-09-16T05:24:46.327 INFO:tasks.rados.rados.2.smithi204.stdout:3727: finishing write tid 3 to smithi20457400-41 2024-09-16T05:24:46.328 INFO:tasks.rados.rados.2.smithi204.stdout:3727: finishing write tid 4 to smithi20457400-41 2024-09-16T05:24:46.328 INFO:tasks.rados.rados.2.smithi204.stdout:3727: finishing write tid 5 to smithi20457400-41 2024-09-16T05:24:46.328 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1459 (ObjNum 1289 snap 400 seq_num 1289) dirty exists 2024-09-16T05:24:46.328 INFO:tasks.rados.rados.2.smithi204.stdout:3727: left oid 41 (ObjNum 1289 snap 400 seq_num 1289) 2024-09-16T05:24:46.328 INFO:tasks.rados.rados.2.smithi204.stdout:3732: finishing write tid 1 to smithi20457400-34 2024-09-16T05:24:46.328 INFO:tasks.rados.rados.2.smithi204.stdout:3732: finishing write tid 2 to smithi20457400-34 2024-09-16T05:24:46.329 INFO:tasks.rados.rados.2.smithi204.stdout:3725: done (14 left) 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3727: done (13 left) 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3729: done (12 left) 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3731: done (11 left) 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3733: done (10 left) 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3734: read oid 29 snap 362 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3734: expect (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3735: rollback oid 15 current snap is 400 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 15 to 362 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3736: copy_from oid 11 from oid 33 current snap is 400 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3737: rollback oid 27 current snap is 400 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 27 to 362 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3738: delete oid 37 current snap is 400 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3732: finishing write tid 3 to smithi20457400-34 2024-09-16T05:24:46.330 INFO:tasks.rados.rados.2.smithi204.stdout:3732: finishing write tid 4 to smithi20457400-34 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:3732: finishing write tid 5 to smithi20457400-34 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1274 (ObjNum 1291 snap 400 seq_num 1291) dirty exists 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:3732: left oid 34 (ObjNum 1291 snap 400 seq_num 1291) 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:3735: finishing rollback tid 0 to smithi20457400-15 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:3735: finishing rollback tid 1 to smithi20457400-15 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 15 v 757 (ObjNum 177 snap 0 seq_num 4282117808) dirty dne 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:3732: done (14 left) 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:3735: done (13 left) 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:3738: done (12 left) 2024-09-16T05:24:46.331 INFO:tasks.rados.rados.2.smithi204.stdout:3739: snap_create 2024-09-16T05:24:46.332 INFO:tasks.rados.rados.2.smithi204.stdout:3737: finishing rollback tid 0 to smithi20457400-27 2024-09-16T05:24:46.332 INFO:tasks.rados.rados.2.smithi204.stdout:3737: finishing rollback tid 1 to smithi20457400-27 2024-09-16T05:24:46.332 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 1271 (ObjNum 33 snap 0 seq_num 4282949424) dirty dne 2024-09-16T05:24:46.335 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:46.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:46.337 INFO:tasks.rados.rados.2.smithi204.stdout:3734: expect (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:24:46.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:46.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:46.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:46.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:46.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: 2.4c scrub starts 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: 2.4c scrub ok 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: osdmap e935: 12 total, 11 up, 12 in 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: 1.17 scrub starts 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: 1.17 scrub ok 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/250 objects degraded (3.600%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2983468510' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:46.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:24:46.588 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:46.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:46.612 INFO:tasks.rados.rados.2.smithi204.stdout:3736: finishing copy_from racing read to smithi20457400-11 2024-09-16T05:24:46.612 INFO:tasks.rados.rados.2.smithi204.stdout:3736: finishing copy_from to smithi20457400-11 2024-09-16T05:24:46.612 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1039 (ObjNum 1260 snap 393 seq_num 1260) dirty exists 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: 2.4c scrub starts 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: 2.4c scrub ok 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: osdmap e935: 12 total, 11 up, 12 in 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: 1.17 scrub starts 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: 1.17 scrub ok 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/250 objects degraded (3.600%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:46.662 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2983468510' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:46.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: 2.4c scrub starts 2024-09-16T05:24:46.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: 2.4c scrub ok 2024-09-16T05:24:46.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:46.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: osdmap e935: 12 total, 11 up, 12 in 2024-09-16T05:24:46.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:24:46.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:24:46.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: 1.17 scrub starts 2024-09-16T05:24:46.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: 1.17 scrub ok 2024-09-16T05:24:46.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/250 objects degraded (3.600%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:46.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2983468510' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:46.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:46.714 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:46.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:46.807 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:46.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:46.839 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:46.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:46.940 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:46.962 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:46.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:47.061 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:47.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:47.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:47.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:47.172 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:46 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:46.883+0000 7f6af1ad6640 -1 osd.6 936 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:47.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:47.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:47.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:47.272 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:24:47.273 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:24:47.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:24:47.273 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:24:47.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.273 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2749 2.d e357/247 2:b21a57e7:::smithi20457400-25:head v 357'413, mlcod=356'409)", 2024-09-16T05:24:47.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:24:47.273 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 580.55343155200001, 2024-09-16T05:24:47.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079488969999999999, 2024-09-16T05:24:47.274 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.274 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.274 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.274 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:24:47.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717233+0000", 2024-09-16T05:24:47.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.717234+0000", 2024-09-16T05:24:47.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0210000000000001e-06 2024-09-16T05:24:47.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722533+0000", 2024-09-16T05:24:47.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052989389999999999 2024-09-16T05:24:47.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722583+0000", 2024-09-16T05:24:47.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9592000000000002e-05 2024-09-16T05:24:47.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722593+0000", 2024-09-16T05:24:47.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0567999999999999e-05 2024-09-16T05:24:47.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722673+0000", 2024-09-16T05:24:47.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.983e-05 2024-09-16T05:24:47.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.722700+0000", 2024-09-16T05:24:47.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6908999999999999e-05 2024-09-16T05:24:47.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.725139+0000", 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024389920000000001 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:06.725182+0000", 2024-09-16T05:24:47.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3046000000000001e-05 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:2898 2.2b e372/329 2:d7c71ae7:::smithi20457400-5:head v 372'201, mlcod=330'198)", 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 565.54608571899996, 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0085128900000000004, 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.281 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724579+0000", 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.724580+0000", 2024-09-16T05:24:47.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2199999999999999e-07 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729164+0000", 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045841340000000001 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729217+0000", 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3152999999999997e-05 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729225+0000", 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9850000000000006e-06 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729320+0000", 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5067000000000003e-05 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.729381+0000", 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1297999999999995e-05 2024-09-16T05:24:47.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.733031+0000", 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036494600000000002 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:21.733092+0000", 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1371000000000006e-05 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:3975 2.d e468/435 2:b1061e13:::smithi20457400-33:head v 468'472, mlcod=459'469)", 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:57.903941+0000", 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 469.36672408999999, 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067494490000000002, 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.903941+0000", 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.903941+0000", 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.903941+0000", 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9000000000000001e-07 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908253+0000", 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043118399999999999 2024-09-16T05:24:47.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908300+0000", 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6953999999999998e-05 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908334+0000", 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3995000000000002e-05 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908424+0000", 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0288e-05 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.908443+0000", 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9184000000000001e-05 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.910651+0000", 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022077669999999998 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:57.910690+0000", 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9230999999999998e-05 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4670 2.3b e515/296 2:de669526:::smithi20457400-23:head v 515'332, mlcod=511'328)", 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:44.810292+0000", 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 422.46037321699998, 2024-09-16T05:24:47.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073315560000000004, 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.810292+0000", 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.810292+0000", 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.810292+0000", 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.03e-07 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812464+0000", 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021723290000000002 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812594+0000", 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013015999999999999 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812612+0000", 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7696000000000001e-05 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812717+0000", 2024-09-16T05:24:47.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010487100000000001 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.812740+0000", 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2943e-05 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.817579+0000", 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048388800000000003 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.817623+0000", 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4474000000000001e-05 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4665 2.d e515/435 2:b1061e13:::smithi20457400-33:head v 515'512, mlcod=512'509)", 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:44.814949+0000", 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 422.455716064, 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0071594459999999999, 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.296 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.814949+0000", 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.814949+0000", 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.814949+0000", 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.36e-07 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820286+0000", 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053367509999999998 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820312+0000", 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6325999999999998e-05 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820318+0000", 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0610000000000001e-06 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820407+0000", 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8327999999999999e-05 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.820425+0000", 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8437999999999999e-05 2024-09-16T05:24:47.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.822073+0000", 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016484220000000001 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:44.822108+0000", 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4884e-05 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.11.0:825 2.32 e535/275 2:4e7d172c:::smithi20457400-15:ca v 535'161, mlcod=529'158)", 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:04.901216+0000", 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 402.36944851300001, 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.44222377000000002, 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:04.901216+0000", 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:04.901216+0000", 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:04.901217+0000", 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.65e-07 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:04.901221+0000", 2024-09-16T05:24:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6909999999999996e-06 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:05.342781+0000", 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.44155928300000002 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:05.342790+0000", 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4839999999999993e-06 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:05.342860+0000", 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9467000000000004e-05 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:05.342880+0000", 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0423000000000001e-05 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:05.343396+0000", 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051584600000000003 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:05.343440+0000", 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4311000000000003e-05 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:5086 2.32 e544/275 2:4e7d172c:::smithi20457400-15:head v 544'165, mlcod=535'161)", 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:13.964429+0000", 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 393.30623622100001, 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0062415420000000001, 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.305 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.964429+0000", 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.964429+0000", 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.964429+0000", 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7700000000000004e-07 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.968010+0000", 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003580401 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.968042+0000", 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2604000000000002e-05 2024-09-16T05:24:47.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.968068+0000", 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5820999999999999e-05 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.968155+0000", 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7396000000000001e-05 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.968177+0000", 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1058999999999999e-05 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.970630+0000", 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024532059999999999 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:13.970670+0000", 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0478000000000001e-05 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.309 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.9.0:1031 2.d e561/435 2:b1061e13:::smithi20457400-33:d4 v 561'535, mlcod=557'531)", 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:30.763495+0000", 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 376.50717009499999, 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14551234499999999, 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.763495+0000", 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.763495+0000", 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.763495+0000", 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.72e-07 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.763502+0000", 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7669999999999998e-06 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.908424+0000", 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14492200599999999 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.908433+0000", 2024-09-16T05:24:47.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4390000000000006e-06 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.908485+0000", 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1975999999999998e-05 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.908502+0000", 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7136e-05 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.908971+0000", 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046843900000000002 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:30.909007+0000", 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6409999999999999e-05 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.9.0:1231 2.d e660/435 2:b1061e13:::smithi20457400-33:107 v 660'598, mlcod=648'595)", 2024-09-16T05:24:47.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:09.798973+0000", 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 277.47169196499999, 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.700095625, 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:09.798973+0000", 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:09.798973+0000", 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:09.798973+0000", 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.15e-07 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:09.798982+0000", 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9709999999999993e-06 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:10.498305+0000", 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.69932320100000001 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:10.498317+0000", 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1202e-05 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:10.498413+0000", 2024-09-16T05:24:47.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6872000000000004e-05 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:10.498435+0000", 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1801999999999999e-05 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:10.499028+0000", 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00059307299999999995 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:10.499069+0000", 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0188999999999998e-05 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:6567 2.32 e670/275 2:4e7d172c:::smithi20457400-15:head v 670'225, mlcod=654'221)", 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:19.847807+0000", 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 267.422857937, 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0072182710000000001, 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.319 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.847807+0000", 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.847807+0000", 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.847807+0000", 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.48e-07 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.851248+0000", 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003441108 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.851263+0000", 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4319e-05 2024-09-16T05:24:47.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.851269+0000", 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7000000000000002e-06 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.851316+0000", 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6520000000000002e-05 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.851332+0000", 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6172e-05 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.854985+0000", 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036529599999999998 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:19.855025+0000", 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0244000000000003e-05 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.323 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:6961 2.d e698/435 2:b1061e13:::smithi20457400-33:head v 698'618, mlcod=695'614)", 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:47.997479+0000", 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 239.27318603699999, 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0064949719999999999, 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:47.997479+0000", 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:47.997479+0000", 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:47.997480+0000", 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5000000000000002e-07 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:48.001003+0000", 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003523736 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:48.001033+0000", 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9434999999999999e-05 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:48.001057+0000", 2024-09-16T05:24:47.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4609000000000001e-05 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:48.001149+0000", 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2201999999999996e-05 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:48.001175+0000", 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5686000000000001e-05 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:48.003932+0000", 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027567999999999998 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:48.003974+0000", 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1854000000000003e-05 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.328 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.27,QREQUEST,e:711,from:7,reservation_nonce:7)", 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:01.833042+0000", 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 225.43762339700001, 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18896977500000001, 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:01.833042+0000", 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:01.833042+0000", 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:01.833043+0000", 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.424e-06 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:01.833051+0000", 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0320000000000003e-06 2024-09-16T05:24:47.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:02.021772+0000", 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18872114800000001 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:02.021783+0000", 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0564000000000001e-05 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:02.021945+0000", 2024-09-16T05:24:47.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000161879 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:02.021966+0000", 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1685e-05 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:02.022011+0000", 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5043000000000002e-05 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(1.e,GRANT,e:755,from:7,reservation_nonce:0)", 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:45.476798+0000", 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 181.793866522, 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017194013000000001, 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.476798+0000", 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.476798+0000", 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.476799+0000", 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5799999999999998e-07 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.476804+0000", 2024-09-16T05:24:47.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3939999999999998e-06 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.493769+0000", 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016965760999999999 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.493775+0000", 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9950000000000002e-06 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.493841+0000", 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5149999999999998e-05 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.493855+0000", 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3898e-05 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:45.493992+0000", 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013785699999999999 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.336 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:7592 2.3b e764/296 2:de669526:::smithi20457400-23:head v 764'470, mlcod=763'467)", 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:53.937274+0000", 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 173.33339082099999, 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0079363119999999992, 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.937274+0000", 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.937274+0000", 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.937275+0000", 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2759999999999999e-06 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.939038+0000", 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001763026 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.939094+0000", 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5347999999999999e-05 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.939105+0000", 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.111e-05 2024-09-16T05:24:47.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.939185+0000", 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9826000000000004e-05 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.939214+0000", 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9280000000000001e-05 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.945150+0000", 2024-09-16T05:24:47.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059362820000000002 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:53.945210+0000", 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0164000000000002e-05 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.341 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.2.0:313 2.2b e813/329 2:d7c71ae7:::smithi20457400-5:157 v 813'425, mlcod=808'421)", 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:22:43.064672+0000", 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 124.205992674, 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.44479925199999998, 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.064672+0000", 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.064672+0000", 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.064673+0000", 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3099999999999998e-07 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.064678+0000", 2024-09-16T05:24:47.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3020000000000002e-06 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.508790+0000", 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.44411214500000001 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.508797+0000", 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7259999999999996e-06 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.508893+0000", 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5768999999999997e-05 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.508914+0000", 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1596000000000001e-05 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.509440+0000", 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00052546299999999999 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:43.509471+0000", 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1819999999999997e-05 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.2.0:326 2.2b e823/329 2:d7c71ae7:::smithi20457400-5:153 v 823'437, mlcod=819'433)", 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:22:52.842244+0000", 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 114.428421357, 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.76053622899999995, 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.346 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:52.842244+0000", 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:52.842244+0000", 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:52.842244+0000", 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4900000000000001e-07 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:52.842249+0000", 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4049999999999999e-06 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:53.601801+0000", 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.75955117800000005 2024-09-16T05:24:47.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:53.601822+0000", 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1769000000000001e-05 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:53.601921+0000", 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8686999999999999e-05 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:53.601942+0000", 2024-09-16T05:24:47.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0489999999999999e-05 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:53.602726+0000", 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00078416399999999995 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:53.602780+0000", 2024-09-16T05:24:47.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4086999999999999e-05 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:8387 2.3b e835/296 2:de669526:::smithi20457400-23:head v 835'518, mlcod=820'514)", 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:04.973395+0000", 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 102.297269844, 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078968600000000003, 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.351 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.973395+0000", 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.973395+0000", 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.973395+0000", 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.975774+0000", 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023783189999999998 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.975800+0000", 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6597999999999998e-05 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.975810+0000", 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.696e-06 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.975857+0000", 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7046000000000003e-05 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.975875+0000", 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8601e-05 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.981227+0000", 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053518680000000001 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:04.981292+0000", 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4555999999999997e-05 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:9351 2.3b e933/296 2:de669526:::smithi20457400-23:head v 933'538, mlcod=925'534)", 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:43.290689+0000", 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.979975708, 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020267919999999999, 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.290689+0000", 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.290689+0000", 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.290690+0000", 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0890000000000001e-06 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.291068+0000", 2024-09-16T05:24:47.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00037780700000000002 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.291120+0000", 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2312000000000003e-05 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.291130+0000", 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1619999999999997e-06 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.291195+0000", 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5407999999999994e-05 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.291216+0000", 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0874999999999999e-05 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.292662+0000", 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001445881 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.292716+0000", 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4258000000000001e-05 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:9356 2.2b e933/329 2:d7c71ae7:::smithi20457400-5:head v 933'462, mlcod=925'458)", 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:43.308296+0000", 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 3.9623691590000001, 2024-09-16T05:24:47.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022195259999999999, 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.308296+0000", 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.308296+0000", 2024-09-16T05:24:47.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.308296+0000", 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7100000000000001e-07 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.309156+0000", 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00086045100000000003 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.309181+0000", 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4287e-05 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.309187+0000", 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2940000000000002e-06 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.309234+0000", 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7345000000000002e-05 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.309251+0000", 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7153999999999999e-05 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.310482+0000", 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012309300000000001 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:43.310515+0000", 2024-09-16T05:24:47.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2894e-05 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.2.0:397 2.2b e935/329 2:d7c71ae7:::smithi20457400-5:18c v 935'468, mlcod=933'464)", 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:45.285155+0000", 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 1.9855099169999999, 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023976330000000001, 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:47.365 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:47.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:47.367 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:47.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:47.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.285155+0000", 2024-09-16T05:24:47.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.285155+0000", 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.285155+0000", 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7599999999999999e-07 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.285160+0000", 2024-09-16T05:24:47.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8300000000000003e-06 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.285192+0000", 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2366999999999997e-05 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.285201+0000", 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1499999999999999e-06 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:47.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.285258+0000", 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7009999999999998e-05 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.285275+0000", 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7649e-05 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.287500+0000", 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022250690000000001 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:47.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:47.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:45.287553+0000", 2024-09-16T05:24:47.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2382000000000001e-05 2024-09-16T05:24:47.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.371 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:47.371 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:47.371 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:24:47.373 INFO:tasks.rados.rados.2.smithi204.stdout:3734: done (12 left) 2024-09-16T05:24:47.373 INFO:tasks.rados.rados.2.smithi204.stdout:3736: done (11 left) 2024-09-16T05:24:47.373 INFO:tasks.rados.rados.2.smithi204.stdout:3737: done (10 left) 2024-09-16T05:24:47.373 INFO:tasks.rados.rados.2.smithi204.stdout:3739: done (9 left) 2024-09-16T05:24:47.373 INFO:tasks.rados.rados.2.smithi204.stdout:3740: copy_from oid 33 from oid 21 current snap is 401 2024-09-16T05:24:47.373 INFO:tasks.rados.rados.2.smithi204.stdout:3741: rollback oid 19 current snap is 401 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 19 to 362 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3742: read oid 25 snap -1 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3742: expect (ObjNum 1277 snap 399 seq_num 1277) 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3743: rollback oid 29 current snap is 401 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 400 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3744: delete oid 6 current snap is 401 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3744: done (13 left) 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3745: copy_from oid 27 from oid 7 current snap is 401 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3743: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3746: delete oid 48 current snap is 401 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3740: finishing copy_from to smithi20457400-33 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3740: got expected ENOENT (src dne) 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3740: finishing copy_from racing read to smithi20457400-33 2024-09-16T05:24:47.374 INFO:tasks.rados.rados.2.smithi204.stdout:3741: finishing rollback tid 0 to smithi20457400-19 2024-09-16T05:24:47.375 INFO:tasks.rados.rados.2.smithi204.stdout:3741: finishing rollback tid 1 to smithi20457400-19 2024-09-16T05:24:47.375 INFO:tasks.rados.rados.2.smithi204.stdout:3741: finishing rollback tid 2 to smithi20457400-19 2024-09-16T05:24:47.375 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 1213 (ObjNum 1109 snap 336 seq_num 1109) dirty exists 2024-09-16T05:24:47.375 INFO:tasks.rados.rados.2.smithi204.stdout:3743: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:24:47.375 INFO:tasks.rados.rados.2.smithi204.stdout:3743: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:24:47.375 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1334 (ObjNum 1288 snap 400 seq_num 1288) dirty exists 2024-09-16T05:24:47.375 INFO:tasks.rados.rados.2.smithi204.stdout:3742: expect (ObjNum 1277 snap 399 seq_num 1277) 2024-09-16T05:24:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:47.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:47.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:47.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:24:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:24:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: pgmap v1565: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 243 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.9 MiB/s wr, 22 op/s; 9/240 objects degraded (3.750%) 2024-09-16T05:24:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2983468510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: osdmap e936: 12 total, 11 up, 12 in 2024-09-16T05:24:47.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:24:47.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:24:47.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:47.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:47.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:47.520 INFO:tasks.rados.rados.2.smithi204.stdout:3745: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:24:47.520 INFO:tasks.rados.rados.2.smithi204.stdout:3745: finishing copy_from to smithi20457400-27 2024-09-16T05:24:47.521 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 1170 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-09-16T05:24:47.521 INFO:tasks.rados.rados.2.smithi204.stdout:3740: done (14 left) 2024-09-16T05:24:47.521 INFO:tasks.rados.rados.2.smithi204.stdout:3741: done (13 left) 2024-09-16T05:24:47.521 INFO:tasks.rados.rados.2.smithi204.stdout:3742: done (12 left) 2024-09-16T05:24:47.521 INFO:tasks.rados.rados.2.smithi204.stdout:3743: done (11 left) 2024-09-16T05:24:47.521 INFO:tasks.rados.rados.2.smithi204.stdout:3745: done (10 left) 2024-09-16T05:24:47.522 INFO:tasks.rados.rados.2.smithi204.stdout:3746: done (9 left) 2024-09-16T05:24:47.522 INFO:tasks.rados.rados.2.smithi204.stdout:3747: write (excl) oid 19 current snap is 401 2024-09-16T05:24:47.522 INFO:tasks.rados.rados.2.smithi204.stdout:3747: seq_num 1296 ranges {581872=502714,1488781=494450,2640081=736411} 2024-09-16T05:24:47.525 INFO:tasks.rados.rados.2.smithi204.stdout:3747: writing smithi20457400-19 from 581872 to 1084586 tid 1 2024-09-16T05:24:47.530 INFO:tasks.rados.rados.2.smithi204.stdout:3747: writing smithi20457400-19 from 1488781 to 1983231 tid 2 2024-09-16T05:24:47.538 INFO:tasks.rados.rados.2.smithi204.stdout:3747: writing smithi20457400-19 from 2640081 to 3376492 tid 3 2024-09-16T05:24:47.538 INFO:tasks.rados.rados.2.smithi204.stdout:3748: write (excl) oid 12 current snap is 401 2024-09-16T05:24:47.538 INFO:tasks.rados.rados.2.smithi204.stdout:3747: finishing write tid 1 to smithi20457400-19 2024-09-16T05:24:47.539 INFO:tasks.rados.rados.2.smithi204.stdout:3747: finishing write tid 2 to smithi20457400-19 2024-09-16T05:24:47.539 INFO:tasks.rados.rados.2.smithi204.stdout:3748: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-09-16T05:24:47.544 INFO:tasks.rados.rados.2.smithi204.stdout:3748: writing smithi20457400-12 from 598679 to 1176642 tid 1 2024-09-16T05:24:47.552 INFO:tasks.rados.rados.2.smithi204.stdout:3748: writing smithi20457400-12 from 1947263 to 2585371 tid 2 2024-09-16T05:24:47.559 INFO:tasks.rados.rados.2.smithi204.stdout:3748: writing smithi20457400-12 from 3351151 to 3798679 tid 3 2024-09-16T05:24:47.559 INFO:tasks.rados.rados.2.smithi204.stdout:3749: write (excl) oid 6 current snap is 401 2024-09-16T05:24:47.559 INFO:tasks.rados.rados.2.smithi204.stdout:3749: seq_num 1298 ranges {615486=653212,1889392=498119,2978574=475308} 2024-09-16T05:24:47.565 INFO:tasks.rados.rados.2.smithi204.stdout:3749: writing smithi20457400-6 from 615486 to 1268698 tid 1 2024-09-16T05:24:47.571 INFO:tasks.rados.rados.2.smithi204.stdout:3749: writing smithi20457400-6 from 1889392 to 2387511 tid 2 2024-09-16T05:24:47.578 INFO:tasks.rados.rados.2.smithi204.stdout:3749: writing smithi20457400-6 from 2978574 to 3453882 tid 3 2024-09-16T05:24:47.578 INFO:tasks.rados.rados.2.smithi204.stdout:3750: rollback oid 36 current snap is 401 2024-09-16T05:24:47.578 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 36 to 358 2024-09-16T05:24:47.578 INFO:tasks.rados.rados.2.smithi204.stdout:3747: finishing write tid 3 to smithi20457400-19 2024-09-16T05:24:47.579 INFO:tasks.rados.rados.2.smithi204.stdout:3747: finishing write tid 4 to smithi20457400-19 2024-09-16T05:24:47.579 INFO:tasks.rados.rados.2.smithi204.stdout:3747: finishing write tid 5 to smithi20457400-19 2024-09-16T05:24:47.579 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 1217 (ObjNum 1296 snap 401 seq_num 1296) dirty exists 2024-09-16T05:24:47.579 INFO:tasks.rados.rados.2.smithi204.stdout:3747: left oid 19 (ObjNum 1296 snap 401 seq_num 1296) 2024-09-16T05:24:47.579 INFO:tasks.rados.rados.2.smithi204.stdout:3748: finishing write tid 1 to smithi20457400-12 2024-09-16T05:24:47.579 INFO:tasks.rados.rados.2.smithi204.stdout:3747: done (12 left) 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:3751: delete oid 44 current snap is 401 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:3748: finishing write tid 2 to smithi20457400-12 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:3748: finishing write tid 3 to smithi20457400-12 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:3748: finishing write tid 4 to smithi20457400-12 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:3748: finishing write tid 5 to smithi20457400-12 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1338 (ObjNum 1297 snap 401 seq_num 1297) dirty exists 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:3748: left oid 12 (ObjNum 1297 snap 401 seq_num 1297) 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:3749: finishing write tid 1 to smithi20457400-6 2024-09-16T05:24:47.580 INFO:tasks.rados.rados.2.smithi204.stdout:3749: finishing write tid 2 to smithi20457400-6 2024-09-16T05:24:47.581 INFO:tasks.rados.rados.2.smithi204.stdout:3748: done (12 left) 2024-09-16T05:24:47.581 INFO:tasks.rados.rados.2.smithi204.stdout:3751: done (11 left) 2024-09-16T05:24:47.581 INFO:tasks.rados.rados.2.smithi204.stdout:3752: read oid 28 snap -1 2024-09-16T05:24:47.581 INFO:tasks.rados.rados.2.smithi204.stdout:3752: expect (ObjNum 1176 snap 360 seq_num 1176) 2024-09-16T05:24:47.581 INFO:tasks.rados.rados.2.smithi204.stdout:3753: snap_remove snap 400 2024-09-16T05:24:47.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:47.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:47.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:24:47.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:24:47.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:47.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: pgmap v1565: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 243 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.9 MiB/s wr, 22 op/s; 9/240 objects degraded (3.750%) 2024-09-16T05:24:47.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2983468510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: osdmap e936: 12 total, 11 up, 12 in 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:47 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: pgmap v1565: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 243 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.9 MiB/s wr, 22 op/s; 9/240 objects degraded (3.750%) 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:24:47.672 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2983468510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:47.673 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: osdmap e936: 12 total, 11 up, 12 in 2024-09-16T05:24:47.673 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:24:47.673 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:47 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:24:47.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:47.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:47.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:47.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:47.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:47.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:47.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:48.016 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:48.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:48.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-09-16T05:24:48.118 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:48.140 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:47 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:47.836+0000 7f6af1ad6640 -1 osd.6 937 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:48.531 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:48.532 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:48.532 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:48.533 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:48.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:48.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:24:48.534 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:24:48.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:24:48.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:24:48.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:24:48.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:48.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: osdmap e937: 12 total, 11 up, 12 in 2024-09-16T05:24:48.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:24:48.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:24:48.535 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4243361721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:48.535 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:48.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:48.536 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:48.537 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:48.538 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:48.538 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:48.539 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:48.539 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:48.541 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:48.541 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:48.542 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:48.543 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:48.543 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:48.544 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:48.544 INFO:tasks.rados.rados.2.smithi204.stdout:3753: done (12 left) 2024-09-16T05:24:48.544 INFO:tasks.rados.rados.2.smithi204.stdout:3754: read oid 8 snap -1 2024-09-16T05:24:48.544 INFO:tasks.rados.rados.2.smithi204.stdout:3754: expect deleted 2024-09-16T05:24:48.544 INFO:tasks.rados.rados.2.smithi204.stdout:3755: delete oid 35 current snap is 401 2024-09-16T05:24:48.544 INFO:tasks.rados.rados.2.smithi204.stdout:3750: finishing rollback tid 1 to smithi20457400-36 2024-09-16T05:24:48.544 INFO:tasks.rados.rados.2.smithi204.stdout:3750: finishing rollback tid 2 to smithi20457400-36 2024-09-16T05:24:48.544 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 803 (ObjNum 1158 snap 348 seq_num 1158) dirty exists 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3749: finishing write tid 3 to smithi20457400-6 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3749: finishing write tid 4 to smithi20457400-6 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3749: finishing write tid 5 to smithi20457400-6 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 1249 (ObjNum 1298 snap 401 seq_num 1298) dirty exists 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3749: left oid 6 (ObjNum 1298 snap 401 seq_num 1298) 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3752: expect (ObjNum 1176 snap 360 seq_num 1176) 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3749: done (13 left) 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3750: done (12 left) 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3752: done (11 left) 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3754: done (10 left) 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3755: done (9 left) 2024-09-16T05:24:48.545 INFO:tasks.rados.rados.2.smithi204.stdout:3756: copy_from oid 41 from oid 32 current snap is 401 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3757: read oid 4 snap -1 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3757: expect deleted 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3758: read oid 5 snap 358 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3758: expect deleted 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3759: copy_from oid 46 from oid 18 current snap is 401 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3760: read oid 23 snap -1 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3760: expect (ObjNum 1284 snap 400 seq_num 1284) 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3761: write oid 17 current snap is 401 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3761: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3761: writing smithi20457400-17 from 665907 to 1261219 tid 1 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3761: writing smithi20457400-17 from 1781191 to 2426637 tid 2 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3761: writing smithi20457400-17 from 3177196 to 3865907 tid 3 2024-09-16T05:24:48.546 INFO:tasks.rados.rados.2.smithi204.stdout:3762: read oid 48 snap -1 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3762: expect deleted 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3757: done (15 left) 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3758: done (14 left) 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3763: read oid 11 snap -1 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3763: expect (ObjNum 1260 snap 393 seq_num 1260) 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3764: write (excl) oid 40 current snap is 401 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3756: finishing copy_from racing read to smithi20457400-41 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3761: finishing write tid 1 to smithi20457400-17 2024-09-16T05:24:48.547 INFO:tasks.rados.rados.2.smithi204.stdout:3756: finishing copy_from to smithi20457400-41 2024-09-16T05:24:48.548 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1270 (ObjNum 1259 snap 393 seq_num 1259) dirty exists 2024-09-16T05:24:48.548 INFO:tasks.rados.rados.2.smithi204.stdout:3761: finishing write tid 2 to smithi20457400-17 2024-09-16T05:24:48.548 INFO:tasks.rados.rados.2.smithi204.stdout:3760: expect (ObjNum 1284 snap 400 seq_num 1284) 2024-09-16T05:24:48.548 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:48.548 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:48.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:48.549 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:48.550 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:48.551 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:48.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:48.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:24:48.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:24:48.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:24:48.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:24:48.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:48.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: osdmap e937: 12 total, 11 up, 12 in 2024-09-16T05:24:48.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:24:48.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:24:48.642 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4243361721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:48.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:48.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:48.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:24:48.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:24:48.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:24:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:24:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: osdmap e937: 12 total, 11 up, 12 in 2024-09-16T05:24:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:24:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:24:48.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4243361721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:3759: finishing copy_from racing read to smithi20457400-46 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:3759: finishing copy_from to smithi20457400-46 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1264 (ObjNum 1274 snap 398 seq_num 1274) dirty exists 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:3761: finishing write tid 3 to smithi20457400-17 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:3761: finishing write tid 4 to smithi20457400-17 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:3761: finishing write tid 5 to smithi20457400-17 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1480 (ObjNum 1301 snap 401 seq_num 1301) dirty exists 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:3761: left oid 17 (ObjNum 1301 snap 401 seq_num 1301) 2024-09-16T05:24:48.723 INFO:tasks.rados.rados.2.smithi204.stdout:3763: expect (ObjNum 1260 snap 393 seq_num 1260) 2024-09-16T05:24:48.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:48.780 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:48.781 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:48.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:48.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:48.897 INFO:tasks.rados.rados.2.smithi204.stdout:3764: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-09-16T05:24:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:48.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:48.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:48.906 INFO:tasks.rados.rados.2.smithi204.stdout:3764: writing smithi20457400-40 from 682714 to 1353275 tid 1 2024-09-16T05:24:48.911 INFO:tasks.rados.rados.2.smithi204.stdout:3764: writing smithi20457400-40 from 2123320 to 2628777 tid 2 2024-09-16T05:24:48.918 INFO:tasks.rados.rados.2.smithi204.stdout:3764: writing smithi20457400-40 from 3088266 to 3725015 tid 3 2024-09-16T05:24:48.918 INFO:tasks.rados.rados.2.smithi204.stdout:3756: done (15 left) 2024-09-16T05:24:48.918 INFO:tasks.rados.rados.2.smithi204.stdout:3759: done (14 left) 2024-09-16T05:24:48.918 INFO:tasks.rados.rados.2.smithi204.stdout:3760: done (13 left) 2024-09-16T05:24:48.918 INFO:tasks.rados.rados.2.smithi204.stdout:3761: done (12 left) 2024-09-16T05:24:48.918 INFO:tasks.rados.rados.2.smithi204.stdout:3762: done (11 left) 2024-09-16T05:24:48.918 INFO:tasks.rados.rados.2.smithi204.stdout:3763: done (10 left) 2024-09-16T05:24:48.919 INFO:tasks.rados.rados.2.smithi204.stdout:3765: write oid 12 current snap is 401 2024-09-16T05:24:48.919 INFO:tasks.rados.rados.2.smithi204.stdout:3764: finishing write tid 1 to smithi20457400-40 2024-09-16T05:24:48.919 INFO:tasks.rados.rados.2.smithi204.stdout:3764: finishing write tid 2 to smithi20457400-40 2024-09-16T05:24:48.919 INFO:tasks.rados.rados.2.smithi204.stdout:3765: seq_num 1303 ranges {699521=745810,2181802=649115,3515689=383832} 2024-09-16T05:24:48.925 INFO:tasks.rados.rados.2.smithi204.stdout:3765: writing smithi20457400-12 from 699521 to 1445331 tid 1 2024-09-16T05:24:48.933 INFO:tasks.rados.rados.2.smithi204.stdout:3765: writing smithi20457400-12 from 2181802 to 2830917 tid 2 2024-09-16T05:24:48.939 INFO:tasks.rados.rados.2.smithi204.stdout:3765: writing smithi20457400-12 from 3515689 to 3899521 tid 3 2024-09-16T05:24:48.940 INFO:tasks.rados.rados.2.smithi204.stdout:3766: rollback oid 46 current snap is 401 2024-09-16T05:24:48.940 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 46 to 397 2024-09-16T05:24:48.940 INFO:tasks.rados.rados.2.smithi204.stdout:3764: finishing write tid 3 to smithi20457400-40 2024-09-16T05:24:48.940 INFO:tasks.rados.rados.2.smithi204.stdout:3764: finishing write tid 4 to smithi20457400-40 2024-09-16T05:24:48.940 INFO:tasks.rados.rados.2.smithi204.stdout:3764: finishing write tid 5 to smithi20457400-40 2024-09-16T05:24:48.940 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1067 (ObjNum 1302 snap 401 seq_num 1302) dirty exists 2024-09-16T05:24:48.940 INFO:tasks.rados.rados.2.smithi204.stdout:3764: left oid 40 (ObjNum 1302 snap 401 seq_num 1302) 2024-09-16T05:24:48.940 INFO:tasks.rados.rados.2.smithi204.stdout:3765: finishing write tid 1 to smithi20457400-12 2024-09-16T05:24:48.941 INFO:tasks.rados.rados.2.smithi204.stdout:3765: finishing write tid 2 to smithi20457400-12 2024-09-16T05:24:48.941 INFO:tasks.rados.rados.2.smithi204.stdout:3764: done (11 left) 2024-09-16T05:24:48.941 INFO:tasks.rados.rados.2.smithi204.stdout:3767: write oid 21 current snap is 401 2024-09-16T05:24:48.941 INFO:tasks.rados.rados.2.smithi204.stdout:3767: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2024-09-16T05:24:48.945 INFO:tasks.rados.rados.2.smithi204.stdout:3767: writing smithi20457400-21 from 716328 to 1137387 tid 1 2024-09-16T05:24:48.952 INFO:tasks.rados.rados.2.smithi204.stdout:3767: writing smithi20457400-21 from 1840284 to 2349410 tid 2 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3767: writing smithi20457400-21 from 3143112 to 3802405 tid 3 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3768: snap_create 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3765: finishing write tid 3 to smithi20457400-12 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3766: finishing rollback tid 0 to smithi20457400-46 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3765: finishing write tid 4 to smithi20457400-12 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3765: finishing write tid 5 to smithi20457400-12 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1344 (ObjNum 1303 snap 401 seq_num 1303) dirty exists 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3765: left oid 12 (ObjNum 1303 snap 401 seq_num 1303) 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3766: finishing rollback tid 1 to smithi20457400-46 2024-09-16T05:24:48.960 INFO:tasks.rados.rados.2.smithi204.stdout:3766: finishing rollback tid 2 to smithi20457400-46 2024-09-16T05:24:48.961 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1267 (ObjNum 1260 snap 393 seq_num 1260) dirty exists 2024-09-16T05:24:48.961 INFO:tasks.rados.rados.2.smithi204.stdout:3767: finishing write tid 1 to smithi20457400-21 2024-09-16T05:24:48.961 INFO:tasks.rados.rados.2.smithi204.stdout:3767: finishing write tid 2 to smithi20457400-21 2024-09-16T05:24:48.962 INFO:tasks.rados.rados.2.smithi204.stdout:3767: finishing write tid 3 to smithi20457400-21 2024-09-16T05:24:48.962 INFO:tasks.rados.rados.2.smithi204.stdout:3767: finishing write tid 4 to smithi20457400-21 2024-09-16T05:24:48.962 INFO:tasks.rados.rados.2.smithi204.stdout:3767: finishing write tid 5 to smithi20457400-21 2024-09-16T05:24:48.962 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 1184 (ObjNum 1304 snap 401 seq_num 1304) dirty exists 2024-09-16T05:24:48.962 INFO:tasks.rados.rados.2.smithi204.stdout:3767: left oid 21 (ObjNum 1304 snap 401 seq_num 1304) 2024-09-16T05:24:48.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:24:49.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:49.029 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:49.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:49.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:49.152 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:49.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:49.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:49.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:49.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:49.313 INFO:tasks.rados.rados.2.smithi204.stdout:3765: done (12 left) 2024-09-16T05:24:49.313 INFO:tasks.rados.rados.2.smithi204.stdout:3766: done (11 left) 2024-09-16T05:24:49.313 INFO:tasks.rados.rados.2.smithi204.stdout:3767: done (10 left) 2024-09-16T05:24:49.313 INFO:tasks.rados.rados.2.smithi204.stdout:3768: done (9 left) 2024-09-16T05:24:49.313 INFO:tasks.rados.rados.2.smithi204.stdout:3769: read oid 23 snap 401 2024-09-16T05:24:49.314 INFO:tasks.rados.rados.2.smithi204.stdout:3769: expect (ObjNum 1284 snap 400 seq_num 1284) 2024-09-16T05:24:49.314 INFO:tasks.rados.rados.2.smithi204.stdout:3770: delete oid 33 current snap is 402 2024-09-16T05:24:49.315 INFO:tasks.rados.rados.2.smithi204.stdout:3770: done (10 left) 2024-09-16T05:24:49.315 INFO:tasks.rados.rados.2.smithi204.stdout:3771: snap_remove snap 397 2024-09-16T05:24:49.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:49.391 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:49.391 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:49.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:49.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:49.466 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:49.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:49.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:49.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:49.589 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:49 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:49.589 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:49 smithi032 ceph-mon[27388]: pgmap v1568: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 243 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.0 MiB/s wr, 19 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:49.589 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:49 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 70 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:49.589 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:49 smithi032 ceph-mon[27388]: osdmap e938: 12 total, 11 up, 12 in 2024-09-16T05:24:49.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:49.647 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:49.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:49.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:49 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:49.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:49 smithi204 ceph-mon[30933]: pgmap v1568: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 243 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.0 MiB/s wr, 19 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:49.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:49 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 70 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:49.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:49 smithi204 ceph-mon[30933]: osdmap e938: 12 total, 11 up, 12 in 2024-09-16T05:24:49.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:49 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:49.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:49 smithi196 ceph-mon[30802]: pgmap v1568: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 243 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.0 MiB/s wr, 19 op/s; 9/230 objects degraded (3.913%) 2024-09-16T05:24:49.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:49 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 70 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:49.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:49 smithi196 ceph-mon[30802]: osdmap e938: 12 total, 11 up, 12 in 2024-09-16T05:24:49.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:49.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:49.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:49.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:49.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:49.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:49.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:50.018 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:50.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:50.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:24:50.120 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:50.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:50.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:50.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:50.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:50.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:50.281 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:50.314 INFO:tasks.rados.rados.2.smithi204.stdout:3771: done (10 left) 2024-09-16T05:24:50.315 INFO:tasks.rados.rados.2.smithi204.stdout:3772: delete oid 46 current snap is 402 2024-09-16T05:24:50.315 INFO:tasks.rados.rados.2.smithi204.stdout:3769: expect (ObjNum 1284 snap 400 seq_num 1284) 2024-09-16T05:24:50.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:50.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:50.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:50.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:50.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-09-16T05:24:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:50.517 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:50.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:50.518 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:50.586 INFO:tasks.rados.rados.2.smithi204.stdout:3769: done (10 left) 2024-09-16T05:24:50.586 INFO:tasks.rados.rados.2.smithi204.stdout:3772: done (9 left) 2024-09-16T05:24:50.587 INFO:tasks.rados.rados.2.smithi204.stdout:3773: write oid 37 current snap is 402 2024-09-16T05:24:50.587 INFO:tasks.rados.rados.2.smithi204.stdout:3773: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2024-09-16T05:24:50.594 INFO:tasks.rados.rados.2.smithi204.stdout:3773: writing smithi20457400-37 from 766749 to 1413555 tid 1 2024-09-16T05:24:50.600 INFO:tasks.rados.rados.2.smithi204.stdout:3773: writing smithi20457400-37 from 1899377 to 2555830 tid 2 2024-09-16T05:24:50.606 INFO:tasks.rados.rados.2.smithi204.stdout:3773: writing smithi20457400-37 from 3109028 to 3718490 tid 3 2024-09-16T05:24:50.607 INFO:tasks.rados.rados.2.smithi204.stdout:3774: snap_remove snap 401 2024-09-16T05:24:50.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:50.640 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:50.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:50 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:24:50.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:50 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:24:50.641 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:50 smithi032 ceph-mon[27388]: osdmap e939: 12 total, 11 up, 12 in 2024-09-16T05:24:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:50.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:50.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:50 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:24:50.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:50 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:24:50.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:50 smithi196 ceph-mon[30802]: osdmap e939: 12 total, 11 up, 12 in 2024-09-16T05:24:50.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:50 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:24:50.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:50 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:24:50.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:50 smithi204 ceph-mon[30933]: osdmap e939: 12 total, 11 up, 12 in 2024-09-16T05:24:50.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:50.765 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:50.766 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:50.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:50.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:50.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:50.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:50.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:50.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:50.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:51.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:51.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:51.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:51.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:51.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:51.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:51.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:51.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:51.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:51.315 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:24:51.318 INFO:tasks.rados.rados.2.smithi204.stdout:3774: done (10 left) 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3775: rollback oid 41 current snap is 402 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 41 to 358 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3773: finishing write tid 1 to smithi20457400-37 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3773: finishing write tid 2 to smithi20457400-37 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3773: finishing write tid 3 to smithi20457400-37 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3773: finishing write tid 4 to smithi20457400-37 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3773: finishing write tid 5 to smithi20457400-37 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1397 (ObjNum 1307 snap 402 seq_num 1307) dirty exists 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3773: left oid 37 (ObjNum 1307 snap 402 seq_num 1307) 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3773: done (10 left) 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3776: read oid 9 snap -1 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3776: expect (ObjNum 1261 snap 393 seq_num 1261) 2024-09-16T05:24:51.319 INFO:tasks.rados.rados.2.smithi204.stdout:3777: read oid 18 snap -1 2024-09-16T05:24:51.320 INFO:tasks.rados.rados.2.smithi204.stdout:3777: expect (ObjNum 1274 snap 398 seq_num 1274) 2024-09-16T05:24:51.320 INFO:tasks.rados.rados.2.smithi204.stdout:3778: read oid 42 snap -1 2024-09-16T05:24:51.320 INFO:tasks.rados.rados.2.smithi204.stdout:3778: expect (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:24:51.320 INFO:tasks.rados.rados.2.smithi204.stdout:3779: write oid 36 current snap is 402 2024-09-16T05:24:51.320 INFO:tasks.rados.rados.2.smithi204.stdout:3779: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-09-16T05:24:51.325 INFO:tasks.rados.rados.2.smithi204.stdout:3779: writing smithi20457400-36 from 783556 to 1221964 tid 1 2024-09-16T05:24:51.330 INFO:tasks.rados.rados.2.smithi204.stdout:3779: writing smithi20457400-36 from 1674212 to 2074323 tid 2 2024-09-16T05:24:51.336 INFO:tasks.rados.rados.2.smithi204.stdout:3779: writing smithi20457400-36 from 2736451 to 3157185 tid 3 2024-09-16T05:24:51.341 INFO:tasks.rados.rados.2.smithi204.stdout:3779: writing smithi20457400-36 from 3882270 to 3983556 tid 4 2024-09-16T05:24:51.341 INFO:tasks.rados.rados.2.smithi204.stdout:3780: delete oid 3 current snap is 402 2024-09-16T05:24:51.341 INFO:tasks.rados.rados.2.smithi204.stdout:3779: finishing write tid 1 to smithi20457400-36 2024-09-16T05:24:51.341 INFO:tasks.rados.rados.2.smithi204.stdout:3779: finishing write tid 2 to smithi20457400-36 2024-09-16T05:24:51.341 INFO:tasks.rados.rados.2.smithi204.stdout:3775: finishing rollback tid 0 to smithi20457400-41 2024-09-16T05:24:51.342 INFO:tasks.rados.rados.2.smithi204.stdout:3779: finishing write tid 3 to smithi20457400-36 2024-09-16T05:24:51.342 INFO:tasks.rados.rados.2.smithi204.stdout:3776: expect (ObjNum 1261 snap 393 seq_num 1261) 2024-09-16T05:24:51.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:51.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:51.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:51.402 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: 2.26 scrub starts 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: 2.26 scrub ok 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: pgmap v1571: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 244 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 19 op/s; 9/220 objects degraded (4.091%) 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: osdmap e940: 12 total, 11 up, 12 in 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: 2.25 scrub starts 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/220 objects degraded (4.091%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:51.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3507498135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:51.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:24:51.449 INFO:tasks.rados.rados.2.smithi204.stdout:3775: finishing rollback tid 1 to smithi20457400-41 2024-09-16T05:24:51.449 INFO:tasks.rados.rados.2.smithi204.stdout:3775: finishing rollback tid 2 to smithi20457400-41 2024-09-16T05:24:51.449 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1463 (ObjNum 1142 snap 341 seq_num 1142) dirty exists 2024-09-16T05:24:51.449 INFO:tasks.rados.rados.2.smithi204.stdout:3775: done (14 left) 2024-09-16T05:24:51.450 INFO:tasks.rados.rados.2.smithi204.stdout:3776: done (13 left) 2024-09-16T05:24:51.450 INFO:tasks.rados.rados.2.smithi204.stdout:3780: done (12 left) 2024-09-16T05:24:51.450 INFO:tasks.rados.rados.2.smithi204.stdout:3781: rollback oid 11 current snap is 402 2024-09-16T05:24:51.450 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 11 to 358 2024-09-16T05:24:51.450 INFO:tasks.rados.rados.2.smithi204.stdout:3778: expect (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:24:51.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:24:51.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:51.490 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:24:51.490 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:24:51.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: 2.26 scrub starts 2024-09-16T05:24:51.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: 2.26 scrub ok 2024-09-16T05:24:51.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: pgmap v1571: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 244 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 19 op/s; 9/220 objects degraded (4.091%) 2024-09-16T05:24:51.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: osdmap e940: 12 total, 11 up, 12 in 2024-09-16T05:24:51.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: 2.25 scrub starts 2024-09-16T05:24:51.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:24:51.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/220 objects degraded (4.091%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:51.491 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3507498135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:51.511 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:51.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:51.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:51.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:51.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:51.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:24:51.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:24:51.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: 2.26 scrub starts 2024-09-16T05:24:51.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: 2.26 scrub ok 2024-09-16T05:24:51.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: pgmap v1571: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 244 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 19 op/s; 9/220 objects degraded (4.091%) 2024-09-16T05:24:51.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: osdmap e940: 12 total, 11 up, 12 in 2024-09-16T05:24:51.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: 2.25 scrub starts 2024-09-16T05:24:51.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:24:51.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/220 objects degraded (4.091%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:51.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3507498135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:24:51.711 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:51.721 INFO:tasks.rados.rados.2.smithi204.stdout:3779: finishing write tid 4 to smithi20457400-36 2024-09-16T05:24:51.722 INFO:tasks.rados.rados.2.smithi204.stdout:3779: finishing write tid 5 to smithi20457400-36 2024-09-16T05:24:51.722 INFO:tasks.rados.rados.2.smithi204.stdout:3778: done (12 left) 2024-09-16T05:24:51.722 INFO:tasks.rados.rados.2.smithi204.stdout:3782: write oid 2 current snap is 402 2024-09-16T05:24:51.722 INFO:tasks.rados.rados.2.smithi204.stdout:3779: finishing write tid 6 to smithi20457400-36 2024-09-16T05:24:51.722 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 808 (ObjNum 1308 snap 402 seq_num 1308) dirty exists 2024-09-16T05:24:51.722 INFO:tasks.rados.rados.2.smithi204.stdout:3779: left oid 36 (ObjNum 1308 snap 402 seq_num 1308) 2024-09-16T05:24:51.723 INFO:tasks.rados.rados.2.smithi204.stdout:3777: expect (ObjNum 1274 snap 398 seq_num 1274) 2024-09-16T05:24:51.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:51.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:51.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:51.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:51.883 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:51.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:51.963 INFO:tasks.rados.rados.2.smithi204.stdout:3781: finishing rollback tid 0 to smithi20457400-11 2024-09-16T05:24:51.963 INFO:tasks.rados.rados.2.smithi204.stdout:3781: finishing rollback tid 1 to smithi20457400-11 2024-09-16T05:24:51.964 INFO:tasks.rados.rados.2.smithi204.stdout:3781: finishing rollback tid 2 to smithi20457400-11 2024-09-16T05:24:51.964 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1426 (ObjNum 1149 snap 342 seq_num 1149) dirty exists 2024-09-16T05:24:51.964 INFO:tasks.rados.rados.2.smithi204.stdout:3782: seq_num 1310 ranges {417170=588906,1674823=342347} 2024-09-16T05:24:51.968 INFO:tasks.rados.rados.2.smithi204.stdout:3782: writing smithi20457400-2 from 417170 to 1006076 tid 1 2024-09-16T05:24:51.974 INFO:tasks.rados.rados.2.smithi204.stdout:3782: writing smithi20457400-2 from 1674823 to 2017170 tid 2 2024-09-16T05:24:51.974 INFO:tasks.rados.rados.2.smithi204.stdout:3777: done (12 left) 2024-09-16T05:24:51.975 INFO:tasks.rados.rados.2.smithi204.stdout:3779: done (11 left) 2024-09-16T05:24:51.975 INFO:tasks.rados.rados.2.smithi204.stdout:3781: done (10 left) 2024-09-16T05:24:51.975 INFO:tasks.rados.rados.2.smithi204.stdout:3783: write oid 23 current snap is 402 2024-09-16T05:24:51.975 INFO:tasks.rados.rados.2.smithi204.stdout:3783: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-09-16T05:24:51.980 INFO:tasks.rados.rados.2.smithi204.stdout:3783: writing smithi20457400-23 from 433977 to 1098132 tid 1 2024-09-16T05:24:51.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:51.986 INFO:tasks.rados.rados.2.smithi204.stdout:3783: writing smithi20457400-23 from 1733305 to 2033977 tid 2 2024-09-16T05:24:51.986 INFO:tasks.rados.rados.2.smithi204.stdout:3784: read oid 40 snap -1 2024-09-16T05:24:51.986 INFO:tasks.rados.rados.2.smithi204.stdout:3784: expect (ObjNum 1302 snap 401 seq_num 1302) 2024-09-16T05:24:51.986 INFO:tasks.rados.rados.2.smithi204.stdout:3785: read oid 5 snap 362 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3785: expect deleted 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3786: copy_from oid 9 from oid 35 current snap is 402 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3782: finishing write tid 1 to smithi20457400-2 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3782: finishing write tid 2 to smithi20457400-2 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3782: finishing write tid 3 to smithi20457400-2 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3782: finishing write tid 4 to smithi20457400-2 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 563 (ObjNum 1310 snap 402 seq_num 1310) dirty exists 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3782: left oid 2 (ObjNum 1310 snap 402 seq_num 1310) 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3783: finishing write tid 1 to smithi20457400-23 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3782: done (13 left) 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3787: copy_from oid 50 from oid 42 current snap is 402 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3788: write oid 2 current snap is 402 2024-09-16T05:24:51.987 INFO:tasks.rados.rados.2.smithi204.stdout:3788: seq_num 1312 ranges {450784=739404,1791787=258997} 2024-09-16T05:24:51.991 INFO:tasks.rados.rados.2.smithi204.stdout:3788: writing smithi20457400-2 from 450784 to 1190188 tid 1 2024-09-16T05:24:51.997 INFO:tasks.rados.rados.2.smithi204.stdout:3788: writing smithi20457400-2 from 1791787 to 2050784 tid 2 2024-09-16T05:24:51.998 INFO:tasks.rados.rados.2.smithi204.stdout:3789: read oid 45 snap -1 2024-09-16T05:24:51.998 INFO:tasks.rados.rados.2.smithi204.stdout:3789: expect (ObjNum 1262 snap 393 seq_num 1262) 2024-09-16T05:24:51.999 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:24:51.999 INFO:tasks.rados.rados.2.smithi204.stdout:3786: finishing copy_from to smithi20457400-9 2024-09-16T05:24:51.999 INFO:tasks.rados.rados.2.smithi204.stdout:3786: got expected ENOENT (src dne) 2024-09-16T05:24:52.000 INFO:tasks.rados.rados.2.smithi204.stdout:3786: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:24:52.000 INFO:tasks.rados.rados.2.smithi204.stdout:3785: done (15 left) 2024-09-16T05:24:52.001 INFO:tasks.rados.rados.2.smithi204.stdout:3786: done (14 left) 2024-09-16T05:24:52.001 INFO:tasks.rados.rados.2.smithi204.stdout:3790: delete oid 11 current snap is 402 2024-09-16T05:24:52.001 INFO:tasks.rados.rados.2.smithi204.stdout:3783: finishing write tid 2 to smithi20457400-23 2024-09-16T05:24:52.001 INFO:tasks.rados.rados.2.smithi204.stdout:3783: finishing write tid 3 to smithi20457400-23 2024-09-16T05:24:52.001 INFO:tasks.rados.rados.2.smithi204.stdout:3783: finishing write tid 4 to smithi20457400-23 2024-09-16T05:24:52.001 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 872 (ObjNum 1311 snap 402 seq_num 1311) dirty exists 2024-09-16T05:24:52.002 INFO:tasks.rados.rados.2.smithi204.stdout:3783: left oid 23 (ObjNum 1311 snap 402 seq_num 1311) 2024-09-16T05:24:52.002 INFO:tasks.rados.rados.2.smithi204.stdout:3784: expect (ObjNum 1302 snap 401 seq_num 1302) 2024-09-16T05:24:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:52.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:52.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:52.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:52.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:52.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:3787: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:3788: finishing write tid 1 to smithi20457400-2 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:3783: done (14 left) 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:3784: done (13 left) 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:3790: done (12 left) 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:3791: write (excl) oid 46 current snap is 402 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:3787: finishing copy_from to smithi20457400-50 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1239 (ObjNum 1178 snap 360 seq_num 1178) dirty exists 2024-09-16T05:24:52.173 INFO:tasks.rados.rados.2.smithi204.stdout:3788: finishing write tid 2 to smithi20457400-2 2024-09-16T05:24:52.174 INFO:tasks.rados.rados.2.smithi204.stdout:3788: finishing write tid 3 to smithi20457400-2 2024-09-16T05:24:52.174 INFO:tasks.rados.rados.2.smithi204.stdout:3788: finishing write tid 4 to smithi20457400-2 2024-09-16T05:24:52.174 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 566 (ObjNum 1312 snap 402 seq_num 1312) dirty exists 2024-09-16T05:24:52.174 INFO:tasks.rados.rados.2.smithi204.stdout:3788: left oid 2 (ObjNum 1312 snap 402 seq_num 1312) 2024-09-16T05:24:52.174 INFO:tasks.rados.rados.2.smithi204.stdout:3789: expect (ObjNum 1262 snap 393 seq_num 1262) 2024-09-16T05:24:52.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:52.238 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:52.239 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:52.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:52.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:52.300 INFO:tasks.rados.rados.2.smithi204.stdout:3791: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-09-16T05:24:52.305 INFO:tasks.rados.rados.2.smithi204.stdout:3791: writing smithi20457400-46 from 484398 to 974300 tid 1 2024-09-16T05:24:52.309 INFO:tasks.rados.rados.2.smithi204.stdout:3791: writing smithi20457400-46 from 1392398 to 1803516 tid 2 2024-09-16T05:24:52.310 INFO:tasks.rados.rados.2.smithi204.stdout:3787: done (12 left) 2024-09-16T05:24:52.310 INFO:tasks.rados.rados.2.smithi204.stdout:3788: done (11 left) 2024-09-16T05:24:52.310 INFO:tasks.rados.rados.2.smithi204.stdout:3789: done (10 left) 2024-09-16T05:24:52.310 INFO:tasks.rados.rados.2.smithi204.stdout:3792: delete oid 36 current snap is 402 2024-09-16T05:24:52.310 INFO:tasks.rados.rados.2.smithi204.stdout:3791: finishing write tid 1 to smithi20457400-46 2024-09-16T05:24:52.311 INFO:tasks.rados.rados.2.smithi204.stdout:3792: done (10 left) 2024-09-16T05:24:52.311 INFO:tasks.rados.rados.2.smithi204.stdout:3793: delete oid 33 current snap is 402 2024-09-16T05:24:52.312 INFO:tasks.rados.rados.2.smithi204.stdout:3791: finishing write tid 2 to smithi20457400-46 2024-09-16T05:24:52.312 INFO:tasks.rados.rados.2.smithi204.stdout:3791: finishing write tid 3 to smithi20457400-46 2024-09-16T05:24:52.312 INFO:tasks.rados.rados.2.smithi204.stdout:3791: finishing write tid 4 to smithi20457400-46 2024-09-16T05:24:52.312 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1271 (ObjNum 1314 snap 402 seq_num 1314) dirty exists 2024-09-16T05:24:52.312 INFO:tasks.rados.rados.2.smithi204.stdout:3791: left oid 46 (ObjNum 1314 snap 402 seq_num 1314) 2024-09-16T05:24:52.313 INFO:tasks.rados.rados.2.smithi204.stdout:3791: done (10 left) 2024-09-16T05:24:52.313 INFO:tasks.rados.rados.2.smithi204.stdout:3793: done (9 left) 2024-09-16T05:24:52.313 INFO:tasks.rados.rados.2.smithi204.stdout:3794: copy_from oid 9 from oid 46 current snap is 402 2024-09-16T05:24:52.313 INFO:tasks.rados.rados.2.smithi204.stdout:3795: delete oid 17 current snap is 402 2024-09-16T05:24:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:52 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:24:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:52 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:24:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:52 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:24:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:52 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:24:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:52 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:52.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3507498135' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:52.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:52 smithi196 ceph-mon[30802]: osdmap e941: 12 total, 11 up, 12 in 2024-09-16T05:24:52.315 INFO:tasks.rados.rados.2.smithi204.stdout:3795: done (10 left) 2024-09-16T05:24:52.315 INFO:tasks.rados.rados.2.smithi204.stdout:3796: write (excl) oid 7 current snap is 402 2024-09-16T05:24:52.316 INFO:tasks.rados.rados.2.smithi204.stdout:3796: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-09-16T05:24:52.320 INFO:tasks.rados.rados.2.smithi204.stdout:3796: writing smithi20457400-7 from 551626 to 1058877 tid 1 2024-09-16T05:24:52.326 INFO:tasks.rados.rados.2.smithi204.stdout:3796: writing smithi20457400-7 from 1626326 to 2044782 tid 2 2024-09-16T05:24:52.326 INFO:tasks.rados.rados.2.smithi204.stdout:3797: delete oid 47 current snap is 402 2024-09-16T05:24:52.326 INFO:tasks.rados.rados.2.smithi204.stdout:3794: finishing copy_from racing read to smithi20457400-9 2024-09-16T05:24:52.326 INFO:tasks.rados.rados.2.smithi204.stdout:3796: finishing write tid 1 to smithi20457400-7 2024-09-16T05:24:52.326 INFO:tasks.rados.rados.2.smithi204.stdout:3794: finishing copy_from to smithi20457400-9 2024-09-16T05:24:52.327 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1271 (ObjNum 1314 snap 402 seq_num 1314) dirty exists 2024-09-16T05:24:52.327 INFO:tasks.rados.rados.2.smithi204.stdout:3794: done (11 left) 2024-09-16T05:24:52.327 INFO:tasks.rados.rados.2.smithi204.stdout:3797: done (10 left) 2024-09-16T05:24:52.327 INFO:tasks.rados.rados.2.smithi204.stdout:3798: write (excl) oid 17 current snap is 402 2024-09-16T05:24:52.327 INFO:tasks.rados.rados.2.smithi204.stdout:3798: seq_num 1320 ranges {585240=657749,1743290=422125} 2024-09-16T05:24:52.334 INFO:tasks.rados.rados.2.smithi204.stdout:3798: writing smithi20457400-17 from 585240 to 1242989 tid 1 2024-09-16T05:24:52.339 INFO:tasks.rados.rados.2.smithi204.stdout:3798: writing smithi20457400-17 from 1743290 to 2165415 tid 2 2024-09-16T05:24:52.340 INFO:tasks.rados.rados.2.smithi204.stdout:3799: read oid 11 snap -1 2024-09-16T05:24:52.340 INFO:tasks.rados.rados.2.smithi204.stdout:3799: expect deleted 2024-09-16T05:24:52.340 INFO:tasks.rados.rados.2.smithi204.stdout:3800: write oid 43 current snap is 402 2024-09-16T05:24:52.340 INFO:tasks.rados.rados.2.smithi204.stdout:3796: finishing write tid 2 to smithi20457400-7 2024-09-16T05:24:52.340 INFO:tasks.rados.rados.2.smithi204.stdout:3796: finishing write tid 3 to smithi20457400-7 2024-09-16T05:24:52.340 INFO:tasks.rados.rados.2.smithi204.stdout:3796: finishing write tid 4 to smithi20457400-7 2024-09-16T05:24:52.340 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 7 v 1242 (ObjNum 1318 snap 402 seq_num 1318) dirty exists 2024-09-16T05:24:52.340 INFO:tasks.rados.rados.2.smithi204.stdout:3796: left oid 7 (ObjNum 1318 snap 402 seq_num 1318) 2024-09-16T05:24:52.341 INFO:tasks.rados.rados.2.smithi204.stdout:3798: finishing write tid 1 to smithi20457400-17 2024-09-16T05:24:52.341 INFO:tasks.rados.rados.2.smithi204.stdout:3800: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-09-16T05:24:52.346 INFO:tasks.rados.rados.2.smithi204.stdout:3800: writing smithi20457400-43 from 602047 to 1335045 tid 1 2024-09-16T05:24:52.351 INFO:tasks.rados.rados.2.smithi204.stdout:3800: writing smithi20457400-43 from 1801772 to 2202047 tid 2 2024-09-16T05:24:52.351 INFO:tasks.rados.rados.2.smithi204.stdout:3796: done (12 left) 2024-09-16T05:24:52.351 INFO:tasks.rados.rados.2.smithi204.stdout:3801: read oid 23 snap -1 2024-09-16T05:24:52.352 INFO:tasks.rados.rados.2.smithi204.stdout:3801: expect (ObjNum 1311 snap 402 seq_num 1311) 2024-09-16T05:24:52.352 INFO:tasks.rados.rados.2.smithi204.stdout:3802: read oid 31 snap 358 2024-09-16T05:24:52.352 INFO:tasks.rados.rados.2.smithi204.stdout:3802: expect (ObjNum 1080 snap 329 seq_num 1080) 2024-09-16T05:24:52.352 INFO:tasks.rados.rados.2.smithi204.stdout:3803: write (excl) oid 37 current snap is 402 2024-09-16T05:24:52.352 INFO:tasks.rados.rados.2.smithi204.stdout:3803: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-09-16T05:24:52.357 INFO:tasks.rados.rados.2.smithi204.stdout:3803: writing smithi20457400-37 from 618854 to 1027101 tid 1 2024-09-16T05:24:52.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:52.363 INFO:tasks.rados.rados.2.smithi204.stdout:3803: writing smithi20457400-37 from 1743901 to 2169695 tid 2 2024-09-16T05:24:52.363 INFO:tasks.rados.rados.2.smithi204.stdout:3804: snap_create 2024-09-16T05:24:52.363 INFO:tasks.rados.rados.2.smithi204.stdout:3798: finishing write tid 2 to smithi20457400-17 2024-09-16T05:24:52.363 INFO:tasks.rados.rados.2.smithi204.stdout:3798: finishing write tid 3 to smithi20457400-17 2024-09-16T05:24:52.364 INFO:tasks.rados.rados.2.smithi204.stdout:3798: finishing write tid 4 to smithi20457400-17 2024-09-16T05:24:52.364 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1484 (ObjNum 1320 snap 402 seq_num 1320) dirty exists 2024-09-16T05:24:52.364 INFO:tasks.rados.rados.2.smithi204.stdout:3798: left oid 17 (ObjNum 1320 snap 402 seq_num 1320) 2024-09-16T05:24:52.364 INFO:tasks.rados.rados.2.smithi204.stdout:3800: finishing write tid 1 to smithi20457400-43 2024-09-16T05:24:52.364 INFO:tasks.rados.rados.2.smithi204.stdout:3802: expect (ObjNum 1080 snap 329 seq_num 1080) 2024-09-16T05:24:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:52.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:52.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:52.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:52 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:24:52.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:52 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:24:52.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:52 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:24:52.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:52 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:24:52.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:52 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:52.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3507498135' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:52.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:52 smithi032 ceph-mon[27388]: osdmap e941: 12 total, 11 up, 12 in 2024-09-16T05:24:52.460 INFO:tasks.rados.rados.2.smithi204.stdout:3800: finishing write tid 2 to smithi20457400-43 2024-09-16T05:24:52.460 INFO:tasks.rados.rados.2.smithi204.stdout:3800: finishing write tid 3 to smithi20457400-43 2024-09-16T05:24:52.460 INFO:tasks.rados.rados.2.smithi204.stdout:3800: finishing write tid 4 to smithi20457400-43 2024-09-16T05:24:52.460 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1483 (ObjNum 1321 snap 402 seq_num 1321) dirty exists 2024-09-16T05:24:52.460 INFO:tasks.rados.rados.2.smithi204.stdout:3800: left oid 43 (ObjNum 1321 snap 402 seq_num 1321) 2024-09-16T05:24:52.460 INFO:tasks.rados.rados.2.smithi204.stdout:3801: expect (ObjNum 1311 snap 402 seq_num 1311) 2024-09-16T05:24:52.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:52.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:52.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:52.587 INFO:tasks.rados.rados.2.smithi204.stdout:3803: finishing write tid 1 to smithi20457400-37 2024-09-16T05:24:52.587 INFO:tasks.rados.rados.2.smithi204.stdout:3803: finishing write tid 2 to smithi20457400-37 2024-09-16T05:24:52.587 INFO:tasks.rados.rados.2.smithi204.stdout:3803: finishing write tid 3 to smithi20457400-37 2024-09-16T05:24:52.587 INFO:tasks.rados.rados.2.smithi204.stdout:3803: finishing write tid 4 to smithi20457400-37 2024-09-16T05:24:52.587 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1401 (ObjNum 1322 snap 402 seq_num 1322) dirty exists 2024-09-16T05:24:52.587 INFO:tasks.rados.rados.2.smithi204.stdout:3803: left oid 37 (ObjNum 1322 snap 402 seq_num 1322) 2024-09-16T05:24:52.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:52.631 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:52.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:52.657 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:52.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:52 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:24:52.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:52 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:24:52.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:52 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:24:52.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:52 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:24:52.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:52 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:52.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3507498135' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:24:52.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:52 smithi204 ceph-mon[30933]: osdmap e941: 12 total, 11 up, 12 in 2024-09-16T05:24:52.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:52.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:52.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:52.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:24:52.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:52.877 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:52.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:52.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:53.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:53.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:53.082 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:53.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:53.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:53.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:53.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-09-16T05:24:53.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:53.292 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:53.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:53.325 INFO:tasks.rados.rados.2.smithi204.stdout:3798: done (15 left) 2024-09-16T05:24:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:3799: done (14 left) 2024-09-16T05:24:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:3800: done (13 left) 2024-09-16T05:24:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:3801: done (12 left) 2024-09-16T05:24:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:3802: done (11 left) 2024-09-16T05:24:53.326 INFO:tasks.rados.rados.2.smithi204.stdout:3803: done (10 left) 2024-09-16T05:24:53.327 INFO:tasks.rados.rados.2.smithi204.stdout:3804: done (9 left) 2024-09-16T05:24:53.327 INFO:tasks.rados.rados.2.smithi204.stdout:3805: snap_remove snap 402 2024-09-16T05:24:53.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:53.415 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:53.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:53.456 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:53 smithi032 ceph-mon[27388]: pgmap v1574: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 230 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 5.6 MiB/s wr, 32 op/s; 9/196 objects degraded (4.592%) 2024-09-16T05:24:53.456 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:53 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:24:53.456 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:53 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:24:53.456 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:24:53.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:53 smithi196 ceph-mon[30802]: pgmap v1574: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 230 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 5.6 MiB/s wr, 32 op/s; 9/196 objects degraded (4.592%) 2024-09-16T05:24:53.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:53 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:24:53.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:53 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:24:53.470 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:24:53.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:53.538 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:53.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:53.585 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:53.585 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:53.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:53.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:53.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:53.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:53 smithi204 ceph-mon[30933]: pgmap v1574: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 230 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 5.6 MiB/s wr, 32 op/s; 9/196 objects degraded (4.592%) 2024-09-16T05:24:53.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:53 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:24:53.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:53 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:24:53.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:24:53.696 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:53.696 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:53.696 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:53.719 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:53.719 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:53.743 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:53.743 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:53.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:53.765 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:53.785 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:53.785 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:53.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:53.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:53.806 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:53.806 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:53.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:53.827 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:53.848 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:53.849 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:53.869 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:53.870 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:53.887 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:53.890 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:53.890 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:53.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:53.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:53.911 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:53.912 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:54.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:54.032 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:54.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:54.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:54.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:54.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:54.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:24:54.212 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:53 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:53.903+0000 7f6af1ad6640 -1 osd.6 941 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:54.242 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:24:54.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:54.279 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:54.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:54.327 INFO:tasks.rados.rados.2.smithi204.stdout:3805: done (9 left) 2024-09-16T05:24:54.327 INFO:tasks.rados.rados.2.smithi204.stdout:3806: delete oid 7 current snap is 403 2024-09-16T05:24:54.335 INFO:tasks.rados.rados.2.smithi204.stdout:3806: done (9 left) 2024-09-16T05:24:54.336 INFO:tasks.rados.rados.2.smithi204.stdout:3807: write (excl) oid 37 current snap is 403 2024-09-16T05:24:54.336 INFO:tasks.rados.rados.2.smithi204.stdout:3807: seq_num 1324 ranges {652468=675098,1977218=275250} 2024-09-16T05:24:54.343 INFO:tasks.rados.rados.2.smithi204.stdout:3807: writing smithi20457400-37 from 652468 to 1327566 tid 1 2024-09-16T05:24:54.348 INFO:tasks.rados.rados.2.smithi204.stdout:3807: writing smithi20457400-37 from 1977218 to 2252468 tid 2 2024-09-16T05:24:54.349 INFO:tasks.rados.rados.2.smithi204.stdout:3808: write (excl) oid 28 current snap is 403 2024-09-16T05:24:54.349 INFO:tasks.rados.rados.2.smithi204.stdout:3808: seq_num 1325 ranges {669275=750347,2035700=233575} 2024-09-16T05:24:54.357 INFO:tasks.rados.rados.2.smithi204.stdout:3808: writing smithi20457400-28 from 669275 to 1419622 tid 1 2024-09-16T05:24:54.362 INFO:tasks.rados.rados.2.smithi204.stdout:3808: writing smithi20457400-28 from 2035700 to 2269275 tid 2 2024-09-16T05:24:54.362 INFO:tasks.rados.rados.2.smithi204.stdout:3809: copy_from oid 6 from oid 27 current snap is 403 2024-09-16T05:24:54.362 INFO:tasks.rados.rados.2.smithi204.stdout:3807: finishing write tid 1 to smithi20457400-37 2024-09-16T05:24:54.362 INFO:tasks.rados.rados.2.smithi204.stdout:3810: snap_create 2024-09-16T05:24:54.362 INFO:tasks.rados.rados.2.smithi204.stdout:3807: finishing write tid 2 to smithi20457400-37 2024-09-16T05:24:54.362 INFO:tasks.rados.rados.2.smithi204.stdout:3807: finishing write tid 3 to smithi20457400-37 2024-09-16T05:24:54.362 INFO:tasks.rados.rados.2.smithi204.stdout:3807: finishing write tid 4 to smithi20457400-37 2024-09-16T05:24:54.362 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 37 v 1404 (ObjNum 1324 snap 403 seq_num 1324) dirty exists 2024-09-16T05:24:54.363 INFO:tasks.rados.rados.2.smithi204.stdout:3807: left oid 37 (ObjNum 1324 snap 403 seq_num 1324) 2024-09-16T05:24:54.364 INFO:tasks.rados.rados.2.smithi204.stdout:3808: finishing write tid 1 to smithi20457400-28 2024-09-16T05:24:54.364 INFO:tasks.rados.rados.2.smithi204.stdout:3808: finishing write tid 2 to smithi20457400-28 2024-09-16T05:24:54.364 INFO:tasks.rados.rados.2.smithi204.stdout:3808: finishing write tid 3 to smithi20457400-28 2024-09-16T05:24:54.364 INFO:tasks.rados.rados.2.smithi204.stdout:3808: finishing write tid 4 to smithi20457400-28 2024-09-16T05:24:54.364 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1487 (ObjNum 1325 snap 403 seq_num 1325) dirty exists 2024-09-16T05:24:54.364 INFO:tasks.rados.rados.2.smithi204.stdout:3808: left oid 28 (ObjNum 1325 snap 403 seq_num 1325) 2024-09-16T05:24:54.370 INFO:tasks.rados.rados.2.smithi204.stdout:3809: finishing copy_from racing read to smithi20457400-6 2024-09-16T05:24:54.373 INFO:tasks.rados.rados.2.smithi204.stdout:3809: finishing copy_from to smithi20457400-6 2024-09-16T05:24:54.374 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 1170 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-09-16T05:24:54.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:54.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:54.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:54.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:54 smithi196 ceph-mon[30802]: osdmap e942: 12 total, 11 up, 12 in 2024-09-16T05:24:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:54 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:54 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3874182521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:54 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:24:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:54 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:24:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:54 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.494 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:54.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:54.526 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:54.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:54.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:54.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:54 smithi032 ceph-mon[27388]: osdmap e942: 12 total, 11 up, 12 in 2024-09-16T05:24:54.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:54 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:54 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3874182521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:54.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:54 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:24:54.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:54 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:24:54.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:54 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:54.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:54.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:24:54.656 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:24:54.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:54 smithi204 ceph-mon[30933]: osdmap e942: 12 total, 11 up, 12 in 2024-09-16T05:24:54.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:54 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:54 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3874182521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:54.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:54 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:24:54.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:54 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:24:54.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:54 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:24:54.752 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:54.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:54.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:54.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:54.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:54.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:54.969 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:36.869235+0000", 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 78.098270635999995, 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 78.098287567, 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:54.970 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8703 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869235+0000", 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869235+0000", 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869236+0000", 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7000000000000005e-07 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869239+0000", 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.208e-06 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:54.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869247+0000", 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1829999999999994e-06 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869254+0000", 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0740000000000004e-06 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.872495+0000", 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032404550000000002 2024-09-16T05:24:54.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.872573+0000", 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8412999999999996e-05 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8704 2.29 2:96dc17ef:::smithi20457400-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e867)", 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:36.869264+0000", 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 78.098242084000006, 2024-09-16T05:24:54.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 78.098298985, 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for blocked object", 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8704 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869264+0000", 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:54.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:54.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869264+0000", 2024-09-16T05:24:54.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:54.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:54.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869264+0000", 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6400000000000003e-07 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869266+0000", 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.153e-06 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869273+0000", 2024-09-16T05:24:54.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4930000000000003e-06 2024-09-16T05:24:54.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:54.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.869281+0000", 2024-09-16T05:24:54.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.605e-06 2024-09-16T05:24:54.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:54.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.872743+0000", 2024-09-16T05:24:54.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034624510000000001 2024-09-16T05:24:54.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:54.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:36.872803+0000", 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9923999999999998e-05 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:24:54.981 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 2 2024-09-16T05:24:54.982 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:24:55.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:55.023 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:55.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:55.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:55.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:55.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:55.150 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:55.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:54 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:54.885+0000 7f6af1ad6640 -1 osd.6 943 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:55.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:55.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:55.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:55.340 INFO:tasks.rados.rados.2.smithi204.stdout:3807: done (12 left) 2024-09-16T05:24:55.340 INFO:tasks.rados.rados.2.smithi204.stdout:3808: done (11 left) 2024-09-16T05:24:55.340 INFO:tasks.rados.rados.2.smithi204.stdout:3809: done (10 left) 2024-09-16T05:24:55.340 INFO:tasks.rados.rados.2.smithi204.stdout:3810: done (9 left) 2024-09-16T05:24:55.340 INFO:tasks.rados.rados.2.smithi204.stdout:3811: snap_remove snap 403 2024-09-16T05:24:55.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:55.394 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:55.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:55.496 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:55.519 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:55.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:55.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:55 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:55.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:55 smithi196 ceph-mon[30802]: pgmap v1576: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 206 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.8 MiB/s wr, 25 op/s; 9/178 objects degraded (5.056%) 2024-09-16T05:24:55.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:55 smithi196 ceph-mon[30802]: osdmap e943: 12 total, 11 up, 12 in 2024-09-16T05:24:55.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:55 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 70 sec, osd.6 has slow ops) 2024-09-16T05:24:55.567 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:55 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:55.567 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:55 smithi032 ceph-mon[27388]: pgmap v1576: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 206 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.8 MiB/s wr, 25 op/s; 9/178 objects degraded (5.056%) 2024-09-16T05:24:55.567 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:55 smithi032 ceph-mon[27388]: osdmap e943: 12 total, 11 up, 12 in 2024-09-16T05:24:55.567 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:55 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 70 sec, osd.6 has slow ops) 2024-09-16T05:24:55.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:24:55.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-09-16T05:24:55.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:55.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:55.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:55.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:55 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:55.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:55 smithi204 ceph-mon[30933]: pgmap v1576: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 206 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.8 MiB/s wr, 25 op/s; 9/178 objects degraded (5.056%) 2024-09-16T05:24:55.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:55 smithi204 ceph-mon[30933]: osdmap e943: 12 total, 11 up, 12 in 2024-09-16T05:24:55.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:55 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 70 sec, osd.6 has slow ops) 2024-09-16T05:24:55.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:55.783 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:55.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:55.833 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:55.884 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:55.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:55.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:56.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:56.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:56.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:56.076 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:56.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:56.155 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:56.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:56.218 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:55 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:55.934+0000 7f6af1ad6640 -1 osd.6 944 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:56.256 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:56.282 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:56.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:56.327 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:24:56.327 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:24:56.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:24:56.327 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:24:56.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.327 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.6.0:147 2.29 e354/253 ondisk, result = 0)", 2024-09-16T05:24:56.327 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 592.568513951, 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92583168000000005, 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748951+0000", 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748952+0000", 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2099999999999996e-07 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:03.748956+0000", 2024-09-16T05:24:56.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4839999999999998e-06 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674453+0000", 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92549724300000002 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674462+0000", 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8019999999999998e-06 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674569+0000", 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107186 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674584+0000", 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.502e-05 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:04.674783+0000", 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019832399999999999 2024-09-16T05:24:56.331 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3235 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 450] snapc 8e=[8e,8d,8c,8b,8a,87,85,83,7c] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:15:55.718646+0000", 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 540.59881850099998, 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020234347, 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.332 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3235 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718646+0000", 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718646+0000", 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718648+0000", 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8649999999999999e-06 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718657+0000", 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7499999999999992e-06 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718671+0000", 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4085e-05 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.718680+0000", 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1090000000000004e-06 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.719983+0000", 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001302625 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.720146+0000", 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016294500000000001 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.730144+0000", 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0099980529999999998 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.734998+0000", 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048545769999999997 2024-09-16T05:24:56.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.738811+0000", 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038123609999999998 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.738849+0000", 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7799999999999997e-05 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:15:55.738881+0000", 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2177e-05 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3544 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 351] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.725348+0000", 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 510.59211662600001, 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021240347999999999, 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.338 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3544 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725348+0000", 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725348+0000", 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725350+0000", 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8679999999999999e-06 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725357+0000", 2024-09-16T05:24:56.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0060000000000003e-06 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725371+0000", 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3981e-05 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725381+0000", 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4199999999999996e-06 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.726955+0000", 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015744610000000001 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.727068+0000", 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112963 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.740216+0000", 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013147808 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.743799+0000", 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0035831589999999998 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.746481+0000", 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026815620000000002 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.746542+0000", 2024-09-16T05:24:56.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1815000000000004e-05 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.746589+0000", 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6304999999999998e-05 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3545 2.29 2:9662c29c:::smithi20457400-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:24:56.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.725398+0000", 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 510.59206658400001, 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018036086999999999, 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3545 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725398+0000", 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725398+0000", 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725399+0000", 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3600000000000003e-07 2024-09-16T05:24:56.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725402+0000", 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.19e-06 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725410+0000", 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9209999999999992e-06 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.725417+0000", 2024-09-16T05:24:56.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.426e-06 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.727335+0000", 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019184009999999999 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.727395+0000", 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9466000000000003e-05 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.743208+0000", 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015813105000000001 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.743325+0000", 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117784 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.743434+0000", 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109058 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.11,GRANT,e:436,from:4,reservation_nonce:0)", 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:26.377804+0000", 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 509.93966099800002, 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.244246716, 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.377804+0000", 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.377804+0000", 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.377805+0000", 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3710000000000001e-06 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.377810+0000", 2024-09-16T05:24:56.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3370000000000001e-06 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.621781+0000", 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.24397083999999999 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.621791+0000", 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0142e-05 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.621882+0000", 2024-09-16T05:24:56.352 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1050000000000004e-05 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.621900+0000", 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8349e-05 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:26.622051+0000", 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015062699999999999 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4414 2.3f e499/293 2:fc733d2f:::smithi20457400-45:head v 499'316, mlcod=472'312)", 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 447.56747832299999, 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94419809600000004, 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8500000000000002e-07 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.751332+0000", 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001345001 2024-09-16T05:24:56.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691571+0000", 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94023934399999998 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691580+0000", 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691653+0000", 2024-09-16T05:24:56.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3137000000000001e-05 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691676+0000", 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3349e-05 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694130+0000", 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024536900000000001 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694185+0000", 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4750000000000003e-05 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4417 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 329] snapc ba=[ba,b9,b8] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 446.57335113800002, 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017204743000000002, 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4417 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744116+0000", 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4099999999999998e-06 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744129+0000", 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2717e-05 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744150+0000", 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1246e-05 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744160+0000", 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.815e-06 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744237+0000", 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6860000000000006e-05 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744246+0000", 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1220000000000001e-06 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.745711+0000", 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014654679999999999 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.745823+0000", 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011192300000000001 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.755847+0000", 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010023119 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.759255+0000", 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034084950000000001 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761242+0000", 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019874369999999999 2024-09-16T05:24:56.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761284+0000", 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1984000000000003e-05 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761319+0000", 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4146999999999997e-05 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.7.0:741 2.2d e509/252 2:b7234079:::smithi20457400-42:bc v 509'382, mlcod=507'378)", 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 437.28901668999998, 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72794596499999997, 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028449+0000", 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5400000000000001e-07 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028455+0000", 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4339999999999997e-06 2024-09-16T05:24:56.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755790+0000", 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72733470600000005 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755802+0000", 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1771e-05 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755868+0000", 2024-09-16T05:24:56.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6512000000000003e-05 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755889+0000", 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0245e-05 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.756344+0000", 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045536699999999998 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.756394+0000", 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0275999999999998e-05 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4547 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 303] snapc be=[be] ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 436.54974404799998, 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01820447, 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.369 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4547 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:24:56.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767723+0000", 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0549999999999998e-06 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767732+0000", 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9930000000000001e-06 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767748+0000", 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5715000000000001e-05 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767764+0000", 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5811000000000002e-05 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767896+0000", 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013272800000000001 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767969+0000", 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3120999999999993e-05 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.780323+0000", 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012353563 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:24:56.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.782979+0000", 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026563440000000002 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785858+0000", 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028785719999999998 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785889+0000", 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1569000000000002e-05 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785925+0000", 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5998999999999999e-05 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.8.0:932 2.1a e526/291 2:5827f14d:::smithi20457400-12:c3 v 526'716, mlcod=518'711)", 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 420.55909444600002, 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.24009196499999999, 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.375 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3300000000000001e-07 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758373+0000", 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1909999999999999e-06 2024-09-16T05:24:56.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997774+0000", 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.23940144999999999 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997782+0000", 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7379999999999994e-06 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997836+0000", 2024-09-16T05:24:56.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3557000000000002e-05 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997855+0000", 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9491999999999999e-05 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.998416+0000", 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056088899999999996 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.998462+0000", 2024-09-16T05:24:56.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6514999999999998e-05 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6720 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 624] snapc 116=[116,115,113,112,10c,10b,10a,109,108,106,101,f2] ondisk+write+known_if_redirected+supports_pool_eio e679)", 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:29.354895+0000", 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 266.96256953, 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019406367000000001, 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.380 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6720 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354895+0000", 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354895+0000", 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354897+0000", 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5540000000000001e-06 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354906+0000", 2024-09-16T05:24:56.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2439999999999999e-06 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354923+0000", 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6796e-05 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354930+0000", 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9700000000000002e-06 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.355017+0000", 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7447000000000005e-05 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.355127+0000", 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010949 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.367256+0000", 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012129457999999999 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:24:56.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.371746+0000", 2024-09-16T05:24:56.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044893679999999997 2024-09-16T05:24:56.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:56.386 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:24:56.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:56.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.374189+0000", 2024-09-16T05:24:56.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024428330000000002 2024-09-16T05:24:56.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.374254+0000", 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5578000000000001e-05 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.374302+0000", 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7629000000000001e-05 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6721 2.29 2:9662c29c:::smithi20457400-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e679)", 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:29.354949+0000", 2024-09-16T05:24:56.388 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 266.96251595199999, 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016035351999999999, 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6721 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354949+0000", 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354949+0000", 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354949+0000", 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.34e-07 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354978+0000", 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8634000000000001e-05 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354983+0000", 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3260000000000003e-06 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354988+0000", 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8080000000000004e-06 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.355284+0000", 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029562799999999998 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.355316+0000", 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2675000000000001e-05 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.370849+0000", 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015532750999999999 2024-09-16T05:24:56.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.370913+0000", 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3986999999999996e-05 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.370984+0000", 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1209000000000002e-05 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7200 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 1175] snapc 130=[12f,12e,12c,12b,12a,129,128,126,125,124,123,120,119,116,108] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:18.170705+0000", 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 218.14676024799999, 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022304378, 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7200 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170705+0000", 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170705+0000", 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170708+0000", 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8959999999999999e-06 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170720+0000", 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.203e-05 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170769+0000", 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8912000000000002e-05 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170785+0000", 2024-09-16T05:24:56.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.668e-05 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170955+0000", 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000169443 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.171085+0000", 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013044599999999999 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:24:56.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.186857+0000", 2024-09-16T05:24:56.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015772083999999999 2024-09-16T05:24:56.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.191857+0000", 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004999603 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.192929+0000", 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010723009999999999 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.192971+0000", 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1384000000000002e-05 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.193009+0000", 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8599000000000001e-05 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.401 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7201 2.29 2:9662c29c:::smithi20457400-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e728)", 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:18.170827+0000", 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 218.146638317, 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020617390999999999, 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.402 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7201 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170827+0000", 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170827+0000", 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170828+0000", 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9999999999999995e-07 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170833+0000", 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.87e-06 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170845+0000", 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2326e-05 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170854+0000", 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0159999999999997e-06 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.171306+0000", 2024-09-16T05:24:56.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045261100000000001 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.171346+0000", 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9038999999999997e-05 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.191222+0000", 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019876441000000002 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.191328+0000", 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106251 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.191444+0000", 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115837 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8547 2.29 2:96dc17ef:::smithi20457400-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e855)", 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:24.875781+0000", 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 91.441684338000002, 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72819094100000004, 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.408 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8547 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875781+0000", 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875781+0000", 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875782+0000", 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8220000000000001e-06 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875787+0000", 2024-09-16T05:24:56.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7190000000000001e-06 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875800+0000", 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2928999999999999e-05 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875808+0000", 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6750000000000002e-06 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875975+0000", 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016753300000000001 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876072+0000", 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.692e-05 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.603869+0000", 2024-09-16T05:24:56.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72779689700000005 2024-09-16T05:24:56.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.603972+0000", 2024-09-16T05:24:56.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102446 2024-09-16T05:24:56.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8548 2.29 2:96dc17ef:::smithi20457400-10:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e855)", 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:24.875858+0000", 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 91.441606657999998, 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72821206299999997, 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.414 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8548 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875858+0000", 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875858+0000", 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875859+0000", 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0299999999999998e-07 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875864+0000", 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5959999999999997e-06 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875874+0000", 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0178e-05 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875881+0000", 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0550000000000002e-06 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876143+0000", 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026236400000000001 2024-09-16T05:24:56.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876228+0000", 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4697999999999994e-05 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.603993+0000", 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72776453200000002 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.604070+0000", 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7836999999999994e-05 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8549 2.29 2:96dc17ef:::smithi20457400-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e855)", 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:24.875893+0000", 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 91.441571572000001, 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72825555500000005, 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:24:56.419 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8549 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875893+0000", 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875893+0000", 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875894+0000", 2024-09-16T05:24:56.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7400000000000003e-07 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875908+0000", 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4518999999999999e-05 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875916+0000", 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9440000000000005e-06 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:24:56.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875923+0000", 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1859999999999997e-06 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876470+0000", 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000546906 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876617+0000", 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014755700000000001 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.604090+0000", 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72747304700000004 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.604149+0000", 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8822e-05 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:24:56.424 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:24:56.425 INFO:tasks.rados.rados.2.smithi204.stdout:3811: done (9 left) 2024-09-16T05:24:56.425 INFO:tasks.rados.rados.2.smithi204.stdout:3812: read oid 5 snap -1 2024-09-16T05:24:56.425 INFO:tasks.rados.rados.2.smithi204.stdout:3812: expect (ObjNum 1285 snap 400 seq_num 1285) 2024-09-16T05:24:56.425 INFO:tasks.rados.rados.2.smithi204.stdout:3813: delete oid 43 current snap is 404 2024-09-16T05:24:56.426 INFO:tasks.rados.rados.2.smithi204.stdout:3813: done (10 left) 2024-09-16T05:24:56.426 INFO:tasks.rados.rados.2.smithi204.stdout:3814: write (excl) oid 34 current snap is 404 2024-09-16T05:24:56.426 INFO:tasks.rados.rados.2.smithi204.stdout:3814: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-09-16T05:24:56.426 INFO:tasks.rados.rados.2.smithi204.stdout:3814: writing smithi20457400-34 from 702889 to 1203734 tid 1 2024-09-16T05:24:56.426 INFO:tasks.rados.rados.2.smithi204.stdout:3814: writing smithi20457400-34 from 1636311 to 2213101 tid 2 2024-09-16T05:24:56.426 INFO:tasks.rados.rados.2.smithi204.stdout:3815: rollback oid 33 current snap is 404 2024-09-16T05:24:56.426 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 33 to 362 2024-09-16T05:24:56.426 INFO:tasks.rados.rados.2.smithi204.stdout:3812: expect (ObjNum 1285 snap 400 seq_num 1285) 2024-09-16T05:24:56.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:56.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:56.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:56 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:56.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:56 smithi196 ceph-mon[30802]: osdmap e944: 12 total, 11 up, 12 in 2024-09-16T05:24:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:56 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:24:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:56 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:24:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:56 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/178 objects degraded (5.056%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4244393723' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:56.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:56.567 INFO:tasks.rados.rados.2.smithi204.stdout:3814: finishing write tid 1 to smithi20457400-34 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3815: finishing rollback tid 1 to smithi20457400-33 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3814: finishing write tid 2 to smithi20457400-34 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3814: finishing write tid 3 to smithi20457400-34 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3812: done (11 left) 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3815: done (10 left) 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3816: write (excl) oid 35 current snap is 404 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3814: finishing write tid 4 to smithi20457400-34 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 34 v 1277 (ObjNum 1327 snap 404 seq_num 1327) dirty exists 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3814: left oid 34 (ObjNum 1327 snap 404 seq_num 1327) 2024-09-16T05:24:56.568 INFO:tasks.rados.rados.2.smithi204.stdout:3816: seq_num 1328 ranges {719696=576094,2094793=224903} 2024-09-16T05:24:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:56.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:56.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:56.574 INFO:tasks.rados.rados.2.smithi204.stdout:3816: writing smithi20457400-35 from 719696 to 1295790 tid 1 2024-09-16T05:24:56.580 INFO:tasks.rados.rados.2.smithi204.stdout:3816: writing smithi20457400-35 from 2094793 to 2319696 tid 2 2024-09-16T05:24:56.580 INFO:tasks.rados.rados.2.smithi204.stdout:3814: done (10 left) 2024-09-16T05:24:56.580 INFO:tasks.rados.rados.2.smithi204.stdout:3817: delete oid 40 current snap is 404 2024-09-16T05:24:56.580 INFO:tasks.rados.rados.2.smithi204.stdout:3816: finishing write tid 1 to smithi20457400-35 2024-09-16T05:24:56.581 INFO:tasks.rados.rados.2.smithi204.stdout:3817: done (10 left) 2024-09-16T05:24:56.581 INFO:tasks.rados.rados.2.smithi204.stdout:3818: copy_from oid 2 from oid 32 current snap is 404 2024-09-16T05:24:56.581 INFO:tasks.rados.rados.2.smithi204.stdout:3819: read oid 42 snap -1 2024-09-16T05:24:56.582 INFO:tasks.rados.rados.2.smithi204.stdout:3819: expect (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:24:56.582 INFO:tasks.rados.rados.2.smithi204.stdout:3820: write oid 20 current snap is 404 2024-09-16T05:24:56.582 INFO:tasks.rados.rados.2.smithi204.stdout:3820: seq_num 1330 ranges {753310=726592,2095404=257906} 2024-09-16T05:24:56.589 INFO:tasks.rados.rados.2.smithi204.stdout:3820: writing smithi20457400-20 from 753310 to 1479902 tid 1 2024-09-16T05:24:56.595 INFO:tasks.rados.rados.2.smithi204.stdout:3820: writing smithi20457400-20 from 2095404 to 2353310 tid 2 2024-09-16T05:24:56.595 INFO:tasks.rados.rados.2.smithi204.stdout:3821: copy_from oid 27 from oid 40 current snap is 404 2024-09-16T05:24:56.595 INFO:tasks.rados.rados.2.smithi204.stdout:3816: finishing write tid 2 to smithi20457400-35 2024-09-16T05:24:56.595 INFO:tasks.rados.rados.2.smithi204.stdout:3816: finishing write tid 3 to smithi20457400-35 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:3816: finishing write tid 4 to smithi20457400-35 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 1285 (ObjNum 1328 snap 404 seq_num 1328) dirty exists 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:3816: left oid 35 (ObjNum 1328 snap 404 seq_num 1328) 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:3818: finishing copy_from racing read to smithi20457400-2 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:3816: done (13 left) 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:3822: read oid 8 snap -1 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:3822: expect deleted 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:3823: delete oid 43 current snap is 404 2024-09-16T05:24:56.596 INFO:tasks.rados.rados.2.smithi204.stdout:3819: expect (ObjNum 1178 snap 360 seq_num 1178) 2024-09-16T05:24:56.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:24:56.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:56 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:56.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:56 smithi032 ceph-mon[27388]: osdmap e944: 12 total, 11 up, 12 in 2024-09-16T05:24:56.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:56 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:24:56.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:56 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:24:56.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:56 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/178 objects degraded (5.056%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:56.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4244393723' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:56.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:56.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:56 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:56.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:56 smithi204 ceph-mon[30933]: osdmap e944: 12 total, 11 up, 12 in 2024-09-16T05:24:56.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:56 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:24:56.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:56 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:24:56.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:56 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/178 objects degraded (5.056%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:24:56.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4244393723' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:24:56.691 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:56.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:56.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:56.814 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:56.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:56.867 INFO:tasks.rados.rados.2.smithi204.stdout:3820: finishing write tid 1 to smithi20457400-20 2024-09-16T05:24:56.867 INFO:tasks.rados.rados.2.smithi204.stdout:3821: finishing copy_from to smithi20457400-27 2024-09-16T05:24:56.867 INFO:tasks.rados.rados.2.smithi204.stdout:3821: got expected ENOENT (src dne) 2024-09-16T05:24:56.867 INFO:tasks.rados.rados.2.smithi204.stdout:3821: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:24:56.867 INFO:tasks.rados.rados.2.smithi204.stdout:3820: finishing write tid 2 to smithi20457400-20 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3820: finishing write tid 3 to smithi20457400-20 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3820: finishing write tid 4 to smithi20457400-20 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1430 (ObjNum 1330 snap 404 seq_num 1330) dirty exists 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3820: left oid 20 (ObjNum 1330 snap 404 seq_num 1330) 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3818: finishing copy_from to smithi20457400-2 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 1270 (ObjNum 1259 snap 393 seq_num 1259) dirty exists 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3818: done (14 left) 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3819: done (13 left) 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3820: done (12 left) 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3821: done (11 left) 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3822: done (10 left) 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3823: done (9 left) 2024-09-16T05:24:56.868 INFO:tasks.rados.rados.2.smithi204.stdout:3824: snap_create 2024-09-16T05:24:56.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:24:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:56.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:56.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:24:56.987 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:56 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:56.890+0000 7f6af1ad6640 -1 osd.6 945 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:57.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:57.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:57.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:57.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:57.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:24:57.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:57.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:57.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:57.310 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:57.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:24:57.329 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:57.350 INFO:tasks.rados.rados.2.smithi204.stdout:3824: done (9 left) 2024-09-16T05:24:57.350 INFO:tasks.rados.rados.2.smithi204.stdout:3825: snap_remove snap 404 2024-09-16T05:24:57.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:57.425 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:57.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:57.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:57.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:57 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:57 smithi196 ceph-mon[30802]: pgmap v1579: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 198 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 5.4 MiB/s wr, 32 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:24:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:57 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4244393723' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:57 smithi196 ceph-mon[30802]: osdmap e945: 12 total, 11 up, 12 in 2024-09-16T05:24:57.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:57.560 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:57.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:57.606 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:57 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:57.606 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:57 smithi032 ceph-mon[27388]: pgmap v1579: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 198 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 5.4 MiB/s wr, 32 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:24:57.606 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:57 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:57.606 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4244393723' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:57.606 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:57 smithi032 ceph-mon[27388]: osdmap e945: 12 total, 11 up, 12 in 2024-09-16T05:24:57.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:24:57.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:57 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:57.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:57 smithi204 ceph-mon[30933]: pgmap v1579: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 198 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 5.4 MiB/s wr, 32 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:24:57.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:57 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:24:57.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4244393723' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:24:57.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:57 smithi204 ceph-mon[30933]: osdmap e945: 12 total, 11 up, 12 in 2024-09-16T05:24:57.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:57.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:24:57.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:57.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:57.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:57.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:57.879 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:24:57.879 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:24:57.879 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:24:57.879 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:24:57.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:57.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:57.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:58.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:24:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:58.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:24:58.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:58.184 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:58.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:58.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:24:57 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:24:57.860+0000 7f6af1ad6640 -1 osd.6 946 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:24:58.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:58.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-16T05:24:58.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:58.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:58.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:58.353 INFO:tasks.rados.rados.2.smithi204.stdout:3825: done (9 left) 2024-09-16T05:24:58.353 INFO:tasks.rados.rados.2.smithi204.stdout:3826: write (excl) oid 19 current snap is 405 2024-09-16T05:24:58.354 INFO:tasks.rados.rados.2.smithi204.stdout:3826: seq_num 1332 ranges {786924=593443,1928721=458203} 2024-09-16T05:24:58.361 INFO:tasks.rados.rados.2.smithi204.stdout:3826: writing smithi20457400-19 from 786924 to 1380367 tid 1 2024-09-16T05:24:58.367 INFO:tasks.rados.rados.2.smithi204.stdout:3826: writing smithi20457400-19 from 1928721 to 2386924 tid 2 2024-09-16T05:24:58.367 INFO:tasks.rados.rados.2.smithi204.stdout:3827: write (excl) oid 50 current snap is 405 2024-09-16T05:24:58.367 INFO:tasks.rados.rados.2.smithi204.stdout:3826: finishing write tid 1 to smithi20457400-19 2024-09-16T05:24:58.367 INFO:tasks.rados.rados.2.smithi204.stdout:3827: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-09-16T05:24:58.373 INFO:tasks.rados.rados.2.smithi204.stdout:3827: writing smithi20457400-50 from 403731 to 1072423 tid 1 2024-09-16T05:24:58.379 INFO:tasks.rados.rados.2.smithi204.stdout:3827: writing smithi20457400-50 from 1587203 to 2175000 tid 2 2024-09-16T05:24:58.379 INFO:tasks.rados.rados.2.smithi204.stdout:3828: write (excl) oid 23 current snap is 405 2024-09-16T05:24:58.379 INFO:tasks.rados.rados.2.smithi204.stdout:3828: seq_num 1334 ranges {420538=743941,1929332=491206} 2024-09-16T05:24:58.385 INFO:tasks.rados.rados.2.smithi204.stdout:3828: writing smithi20457400-23 from 420538 to 1164479 tid 1 2024-09-16T05:24:58.392 INFO:tasks.rados.rados.2.smithi204.stdout:3828: writing smithi20457400-23 from 1929332 to 2420538 tid 2 2024-09-16T05:24:58.392 INFO:tasks.rados.rados.2.smithi204.stdout:3829: delete oid 31 current snap is 405 2024-09-16T05:24:58.392 INFO:tasks.rados.rados.2.smithi204.stdout:3826: finishing write tid 2 to smithi20457400-19 2024-09-16T05:24:58.393 INFO:tasks.rados.rados.2.smithi204.stdout:3826: finishing write tid 3 to smithi20457400-19 2024-09-16T05:24:58.393 INFO:tasks.rados.rados.2.smithi204.stdout:3827: finishing write tid 1 to smithi20457400-50 2024-09-16T05:24:58.393 INFO:tasks.rados.rados.2.smithi204.stdout:3827: finishing write tid 2 to smithi20457400-50 2024-09-16T05:24:58.393 INFO:tasks.rados.rados.2.smithi204.stdout:3828: finishing write tid 1 to smithi20457400-23 2024-09-16T05:24:58.394 INFO:tasks.rados.rados.2.smithi204.stdout:3826: finishing write tid 4 to smithi20457400-19 2024-09-16T05:24:58.394 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 1220 (ObjNum 1332 snap 405 seq_num 1332) dirty exists 2024-09-16T05:24:58.394 INFO:tasks.rados.rados.2.smithi204.stdout:3826: left oid 19 (ObjNum 1332 snap 405 seq_num 1332) 2024-09-16T05:24:58.395 INFO:tasks.rados.rados.2.smithi204.stdout:3828: finishing write tid 2 to smithi20457400-23 2024-09-16T05:24:58.395 INFO:tasks.rados.rados.2.smithi204.stdout:3828: finishing write tid 3 to smithi20457400-23 2024-09-16T05:24:58.395 INFO:tasks.rados.rados.2.smithi204.stdout:3828: finishing write tid 4 to smithi20457400-23 2024-09-16T05:24:58.395 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 875 (ObjNum 1334 snap 405 seq_num 1334) dirty exists 2024-09-16T05:24:58.395 INFO:tasks.rados.rados.2.smithi204.stdout:3828: left oid 23 (ObjNum 1334 snap 405 seq_num 1334) 2024-09-16T05:24:58.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:24:58.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:58.434 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:58.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:24:58.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:24:58.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:58.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:58 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:58.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:58 smithi032 ceph-mon[27388]: osdmap e946: 12 total, 11 up, 12 in 2024-09-16T05:24:58.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:58 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:24:58.553 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:58 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:24:58.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:58.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:58.633 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:24:58.642 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:58 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:58 smithi196 ceph-mon[30802]: osdmap e946: 12 total, 11 up, 12 in 2024-09-16T05:24:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:58 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:24:58.643 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:58 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:24:58.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:58.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:58 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:58.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:58 smithi204 ceph-mon[30933]: osdmap e946: 12 total, 11 up, 12 in 2024-09-16T05:24:58.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:58 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:24:58.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:58 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:24:58.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:58.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:58.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:24:58.713+0000 7f887987d640 0 --2- 172.21.15.32:0/3992393370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8874101380 0x7f887419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:24:58.740 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:24:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:24:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:58.753 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:24:58.762 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:24:58.762 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:24:58.768 INFO:tasks.rados.rados.2.smithi204.stdout:3827: finishing write tid 3 to smithi20457400-50 2024-09-16T05:24:58.769 INFO:tasks.rados.rados.2.smithi204.stdout:3827: finishing write tid 4 to smithi20457400-50 2024-09-16T05:24:58.769 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1246 (ObjNum 1333 snap 405 seq_num 1333) dirty exists 2024-09-16T05:24:58.769 INFO:tasks.rados.rados.2.smithi204.stdout:3827: left oid 50 (ObjNum 1333 snap 405 seq_num 1333) 2024-09-16T05:24:58.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:58.784 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:24:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:24:58.805 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:58.806 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:24:58.806 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:24:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:58.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:58.827 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:24:58.827 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:24:58.846 INFO:tasks.rados.rados.2.smithi204.stdout:3826: done (12 left) 2024-09-16T05:24:58.846 INFO:tasks.rados.rados.2.smithi204.stdout:3827: done (11 left) 2024-09-16T05:24:58.846 INFO:tasks.rados.rados.2.smithi204.stdout:3828: done (10 left) 2024-09-16T05:24:58.846 INFO:tasks.rados.rados.2.smithi204.stdout:3829: done (9 left) 2024-09-16T05:24:58.846 INFO:tasks.rados.rados.2.smithi204.stdout:3830: rollback oid 9 current snap is 405 2024-09-16T05:24:58.846 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 9 to 358 2024-09-16T05:24:58.846 INFO:tasks.rados.rados.2.smithi204.stdout:3831: read oid 47 snap -1 2024-09-16T05:24:58.846 INFO:tasks.rados.rados.2.smithi204.stdout:3831: expect deleted 2024-09-16T05:24:58.847 INFO:tasks.rados.rados.2.smithi204.stdout:3832: read oid 29 snap -1 2024-09-16T05:24:58.847 INFO:tasks.rados.rados.2.smithi204.stdout:3832: expect (ObjNum 1288 snap 400 seq_num 1288) 2024-09-16T05:24:58.847 INFO:tasks.rados.rados.2.smithi204.stdout:3833: snap_create 2024-09-16T05:24:58.848 INFO:tasks.rados.rados.2.smithi204.stdout:3830: finishing rollback tid 0 to smithi20457400-9 2024-09-16T05:24:58.848 INFO:tasks.rados.rados.2.smithi204.stdout:3830: finishing rollback tid 1 to smithi20457400-9 2024-09-16T05:24:58.848 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1466 (ObjNum 657 snap 0 seq_num 2944376672) dirty dne 2024-09-16T05:24:58.848 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:24:58.848 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:24:58.851 INFO:tasks.rados.rados.2.smithi204.stdout:3832: expect (ObjNum 1288 snap 400 seq_num 1288) 2024-09-16T05:24:58.870 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:24:58.870 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:24:58.893 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:24:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:24:58.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:24:58.914 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:24:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:24:58.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:58.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:58.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:24:58.936 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:24:58.936 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:24:58.957 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:24:58.957 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:24:59.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:24:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:59.052 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:59.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:24:59.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:24:59.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:24:59.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:59.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:24:59.260 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:24:59.260 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:24:59.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:24:59.298 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:59.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:24:59.363 INFO:tasks.rados.rados.2.smithi204.stdout:3830: done (12 left) 2024-09-16T05:24:59.363 INFO:tasks.rados.rados.2.smithi204.stdout:3831: done (11 left) 2024-09-16T05:24:59.363 INFO:tasks.rados.rados.2.smithi204.stdout:3832: done (10 left) 2024-09-16T05:24:59.363 INFO:tasks.rados.rados.2.smithi204.stdout:3833: done (9 left) 2024-09-16T05:24:59.363 INFO:tasks.rados.rados.2.smithi204.stdout:3834: snap_remove snap 405 2024-09-16T05:24:59.387 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:24:59.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:59.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:59.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:59.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:59 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:59.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:59 smithi196 ceph-mon[30802]: pgmap v1582: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 193 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:24:59.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:59 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 80 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:59.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:59 smithi196 ceph-mon[30802]: osdmap e947: 12 total, 11 up, 12 in 2024-09-16T05:24:59.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:24:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3992393370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:59.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:59.546 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:59.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:59.643 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:59 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:59.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:59 smithi032 ceph-mon[27388]: pgmap v1582: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 193 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:24:59.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:59 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 80 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:59.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:59 smithi032 ceph-mon[27388]: osdmap e947: 12 total, 11 up, 12 in 2024-09-16T05:24:59.644 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:24:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3992393370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:59.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:24:59.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:24:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:59.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:24:59.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:59 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:24:59.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:59 smithi204 ceph-mon[30933]: pgmap v1582: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 193 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:24:59.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:59 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 80 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:24:59.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:59 smithi204 ceph-mon[30933]: osdmap e947: 12 total, 11 up, 12 in 2024-09-16T05:24:59.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:24:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3992393370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:24:59.771 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:24:59.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:24:59.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:59.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:24:59.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:24:59.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:24:59.921 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:24:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:24:59.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:00.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:00.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:00.117 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:00.148 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-09-16T05:25:00.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-16T05:25:00.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:00.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:00.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:00.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:00.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:00.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:00.368 INFO:tasks.rados.rados.2.smithi204.stdout:3834: done (9 left) 2024-09-16T05:25:00.369 INFO:tasks.rados.rados.2.smithi204.stdout:3835: read oid 45 snap 362 2024-09-16T05:25:00.369 INFO:tasks.rados.rados.2.smithi204.stdout:3835: expect (ObjNum 1139 snap 341 seq_num 1139) 2024-09-16T05:25:00.369 INFO:tasks.rados.rados.2.smithi204.stdout:3836: snap_create 2024-09-16T05:25:00.382 INFO:tasks.rados.rados.2.smithi204.stdout:3835: expect (ObjNum 1139 snap 341 seq_num 1139) 2024-09-16T05:25:00.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:00.420 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:00.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:00.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:00.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:00.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:00.559 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:00 smithi196 ceph-mon[30802]: osdmap e948: 12 total, 11 up, 12 in 2024-09-16T05:25:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:00.584 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:00.635 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:00.642 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:00.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:00.658 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:00.658 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:00.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:00.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:00.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:00 smithi032 ceph-mon[27388]: osdmap e948: 12 total, 11 up, 12 in 2024-09-16T05:25:00.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:00 smithi204 ceph-mon[30933]: osdmap e948: 12 total, 11 up, 12 in 2024-09-16T05:25:00.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:00.792 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:00.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:00.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:00.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:00.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:01.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:01.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:01.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:01.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:01.162 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:01.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:01.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:25:01.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:01.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:01.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:01.370 INFO:tasks.rados.rados.2.smithi204.stdout:3835: done (10 left) 2024-09-16T05:25:01.370 INFO:tasks.rados.rados.2.smithi204.stdout:3836: done (9 left) 2024-09-16T05:25:01.370 INFO:tasks.rados.rados.2.smithi204.stdout:3837: copy_from oid 5 from oid 28 current snap is 407 2024-09-16T05:25:01.370 INFO:tasks.rados.rados.2.smithi204.stdout:3838: read oid 44 snap -1 2024-09-16T05:25:01.370 INFO:tasks.rados.rados.2.smithi204.stdout:3838: expect deleted 2024-09-16T05:25:01.371 INFO:tasks.rados.rados.2.smithi204.stdout:3839: delete oid 12 current snap is 407 2024-09-16T05:25:01.372 INFO:tasks.rados.rados.2.smithi204.stdout:3838: done (11 left) 2024-09-16T05:25:01.372 INFO:tasks.rados.rados.2.smithi204.stdout:3839: done (10 left) 2024-09-16T05:25:01.372 INFO:tasks.rados.rados.2.smithi204.stdout:3840: snap_create 2024-09-16T05:25:01.382 INFO:tasks.rados.rados.2.smithi204.stdout:3837: finishing copy_from racing read to smithi20457400-5 2024-09-16T05:25:01.384 INFO:tasks.rados.rados.2.smithi204.stdout:3837: finishing copy_from to smithi20457400-5 2024-09-16T05:25:01.385 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 5 v 1487 (ObjNum 1325 snap 403 seq_num 1325) dirty exists 2024-09-16T05:25:01.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:01.410 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:01.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:01.486 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:01.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:01 smithi196 ceph-mon[30802]: pgmap v1585: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 195 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:25:01.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:01 smithi196 ceph-mon[30802]: osdmap e949: 12 total, 11 up, 12 in 2024-09-16T05:25:01.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:01 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/172 objects degraded (5.233%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:01.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:01.534 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:01.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:01.595 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:01 smithi032 ceph-mon[27388]: pgmap v1585: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 195 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:25:01.595 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:01 smithi032 ceph-mon[27388]: osdmap e949: 12 total, 11 up, 12 in 2024-09-16T05:25:01.595 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:01 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/172 objects degraded (5.233%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:01.606 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:01 smithi204 ceph-mon[30933]: pgmap v1585: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 195 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 9/172 objects degraded (5.233%) 2024-09-16T05:25:01.606 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:01 smithi204 ceph-mon[30933]: osdmap e949: 12 total, 11 up, 12 in 2024-09-16T05:25:01.607 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:01 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/172 objects degraded (5.233%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:01.635 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:01.656 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:01.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:01.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:01.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:01.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:01.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:01.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:02.007 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:02.008 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:02.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:02.034 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:02.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:02.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:02.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:02.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:02.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:02.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:02.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:02.376 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:25:02.379 INFO:tasks.rados.rados.2.smithi204.stdout:3837: done (10 left) 2024-09-16T05:25:02.380 INFO:tasks.rados.rados.2.smithi204.stdout:3840: done (9 left) 2024-09-16T05:25:02.380 INFO:tasks.rados.rados.2.smithi204.stdout:3841: copy_from oid 46 from oid 6 current snap is 408 2024-09-16T05:25:02.380 INFO:tasks.rados.rados.2.smithi204.stdout:3842: read oid 37 snap -1 2024-09-16T05:25:02.380 INFO:tasks.rados.rados.2.smithi204.stdout:3842: expect (ObjNum 1324 snap 403 seq_num 1324) 2024-09-16T05:25:02.380 INFO:tasks.rados.rados.2.smithi204.stdout:3843: read oid 29 snap -1 2024-09-16T05:25:02.380 INFO:tasks.rados.rados.2.smithi204.stdout:3843: expect (ObjNum 1288 snap 400 seq_num 1288) 2024-09-16T05:25:02.380 INFO:tasks.rados.rados.2.smithi204.stdout:3844: write (excl) oid 42 current snap is 408 2024-09-16T05:25:02.380 INFO:tasks.rados.rados.2.smithi204.stdout:3844: seq_num 1337 ranges {470959=569688,1704778=595135} 2024-09-16T05:25:02.385 INFO:tasks.rados.rados.2.smithi204.stdout:3844: writing smithi20457400-42 from 470959 to 1040647 tid 1 2024-09-16T05:25:02.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:02.392 INFO:tasks.rados.rados.2.smithi204.stdout:3844: writing smithi20457400-42 from 1704778 to 2299913 tid 2 2024-09-16T05:25:02.392 INFO:tasks.rados.rados.2.smithi204.stdout:3845: copy_from oid 23 from oid 47 current snap is 408 2024-09-16T05:25:02.392 INFO:tasks.rados.rados.2.smithi204.stdout:3843: expect (ObjNum 1288 snap 400 seq_num 1288) 2024-09-16T05:25:02.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:02.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:02.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:02 smithi032 ceph-mon[27388]: osdmap e950: 12 total, 11 up, 12 in 2024-09-16T05:25:02.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3114992244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:02.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:02.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:02.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:02.583 INFO:tasks.rados.rados.2.smithi204.stdout:3842: expect (ObjNum 1324 snap 403 seq_num 1324) 2024-09-16T05:25:02.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:25:02.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:02.648 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:02 smithi204 ceph-mon[30933]: osdmap e950: 12 total, 11 up, 12 in 2024-09-16T05:25:02.649 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3114992244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:02.660 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:02.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:02.709 INFO:tasks.rados.rados.2.smithi204.stdout:3844: finishing write tid 1 to smithi20457400-42 2024-09-16T05:25:02.709 INFO:tasks.rados.rados.2.smithi204.stdout:3842: done (13 left) 2024-09-16T05:25:02.709 INFO:tasks.rados.rados.2.smithi204.stdout:3843: done (12 left) 2024-09-16T05:25:02.709 INFO:tasks.rados.rados.2.smithi204.stdout:3846: write (excl) oid 41 current snap is 408 2024-09-16T05:25:02.709 INFO:tasks.rados.rados.2.smithi204.stdout:3841: finishing copy_from racing read to smithi20457400-46 2024-09-16T05:25:02.709 INFO:tasks.rados.rados.2.smithi204.stdout:3841: finishing copy_from to smithi20457400-46 2024-09-16T05:25:02.709 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 46 v 1170 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-09-16T05:25:02.709 INFO:tasks.rados.rados.2.smithi204.stdout:3845: finishing copy_from to smithi20457400-23 2024-09-16T05:25:02.710 INFO:tasks.rados.rados.2.smithi204.stdout:3845: got expected ENOENT (src dne) 2024-09-16T05:25:02.710 INFO:tasks.rados.rados.2.smithi204.stdout:3845: finishing copy_from racing read to smithi20457400-23 2024-09-16T05:25:02.710 INFO:tasks.rados.rados.2.smithi204.stdout:3844: finishing write tid 2 to smithi20457400-42 2024-09-16T05:25:02.710 INFO:tasks.rados.rados.2.smithi204.stdout:3844: finishing write tid 3 to smithi20457400-42 2024-09-16T05:25:02.710 INFO:tasks.rados.rados.2.smithi204.stdout:3844: finishing write tid 4 to smithi20457400-42 2024-09-16T05:25:02.710 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 42 v 1242 (ObjNum 1337 snap 408 seq_num 1337) dirty exists 2024-09-16T05:25:02.710 INFO:tasks.rados.rados.2.smithi204.stdout:3844: left oid 42 (ObjNum 1337 snap 408 seq_num 1337) 2024-09-16T05:25:02.710 INFO:tasks.rados.rados.2.smithi204.stdout:3846: seq_num 1338 ranges {487766=644937,1646907=738793} 2024-09-16T05:25:02.714 INFO:tasks.rados.rados.2.smithi204.stdout:3846: writing smithi20457400-41 from 487766 to 1132703 tid 1 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3846: writing smithi20457400-41 from 1646907 to 2385700 tid 2 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3841: done (12 left) 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3844: done (11 left) 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3845: done (10 left) 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3847: read oid 15 snap -1 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3847: expect deleted 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3848: rollback oid 29 current snap is 408 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 362 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3846: finishing write tid 1 to smithi20457400-41 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3849: read oid 6 snap -1 2024-09-16T05:25:02.722 INFO:tasks.rados.rados.2.smithi204.stdout:3849: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:25:02.723 INFO:tasks.rados.rados.2.smithi204.stdout:3850: copy_from oid 21 from oid 23 current snap is 408 2024-09-16T05:25:02.723 INFO:tasks.rados.rados.2.smithi204.stdout:3851: snap_remove snap 406 2024-09-16T05:25:02.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:02 smithi196 ceph-mon[30802]: osdmap e950: 12 total, 11 up, 12 in 2024-09-16T05:25:02.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3114992244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:02.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:02.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:25:02.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:02.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:02.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:02.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:02.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:03.010 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:03.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:03.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:03.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:03.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:03.074 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:03.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:03.160 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:03.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:03.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:03.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:03.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:03.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:03.390 INFO:tasks.rados.rados.2.smithi204.stdout:3851: done (14 left) 2024-09-16T05:25:03.390 INFO:tasks.rados.rados.2.smithi204.stdout:3852: copy_from oid 20 from oid 37 current snap is 408 2024-09-16T05:25:03.390 INFO:tasks.rados.rados.2.smithi204.stdout:3848: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:25:03.390 INFO:tasks.rados.rados.2.smithi204.stdout:3848: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:25:03.390 INFO:tasks.rados.rados.2.smithi204.stdout:3848: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:25:03.391 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1354 (ObjNum 1178 snap 360 seq_num 1178) dirty exists 2024-09-16T05:25:03.391 INFO:tasks.rados.rados.2.smithi204.stdout:3846: finishing write tid 2 to smithi20457400-41 2024-09-16T05:25:03.391 INFO:tasks.rados.rados.2.smithi204.stdout:3846: finishing write tid 3 to smithi20457400-41 2024-09-16T05:25:03.391 INFO:tasks.rados.rados.2.smithi204.stdout:3846: finishing write tid 4 to smithi20457400-41 2024-09-16T05:25:03.391 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1470 (ObjNum 1338 snap 408 seq_num 1338) dirty exists 2024-09-16T05:25:03.391 INFO:tasks.rados.rados.2.smithi204.stdout:3846: left oid 41 (ObjNum 1338 snap 408 seq_num 1338) 2024-09-16T05:25:03.391 INFO:tasks.rados.rados.2.smithi204.stdout:3849: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:25:03.397 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:03 smithi204 ceph-mon[30933]: pgmap v1588: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 188 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:25:03.398 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:03 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:03.398 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3114992244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:03.398 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:03 smithi204 ceph-mon[30933]: osdmap e951: 12 total, 11 up, 12 in 2024-09-16T05:25:03.398 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:03 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:25:03.398 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:03 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:25:03.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:03.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:03.497 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:03.497 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:03.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:03.537 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:03.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:03.560 INFO:tasks.rados.rados.2.smithi204.stdout:3850: finishing copy_from racing read to smithi20457400-21 2024-09-16T05:25:03.560 INFO:tasks.rados.rados.2.smithi204.stdout:3850: finishing copy_from to smithi20457400-21 2024-09-16T05:25:03.560 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 875 (ObjNum 1334 snap 405 seq_num 1334) dirty exists 2024-09-16T05:25:03.560 INFO:tasks.rados.rados.2.smithi204.stdout:3846: done (14 left) 2024-09-16T05:25:03.560 INFO:tasks.rados.rados.2.smithi204.stdout:3847: done (13 left) 2024-09-16T05:25:03.560 INFO:tasks.rados.rados.2.smithi204.stdout:3848: done (12 left) 2024-09-16T05:25:03.560 INFO:tasks.rados.rados.2.smithi204.stdout:3849: done (11 left) 2024-09-16T05:25:03.560 INFO:tasks.rados.rados.2.smithi204.stdout:3850: done (10 left) 2024-09-16T05:25:03.561 INFO:tasks.rados.rados.2.smithi204.stdout:3853: read oid 1 snap -1 2024-09-16T05:25:03.561 INFO:tasks.rados.rados.2.smithi204.stdout:3853: expect deleted 2024-09-16T05:25:03.561 INFO:tasks.rados.rados.2.smithi204.stdout:3854: write (excl) oid 48 current snap is 408 2024-09-16T05:25:03.561 INFO:tasks.rados.rados.2.smithi204.stdout:3854: seq_num 1339 ranges {504573=436539,1421742=598804,2437505=67068} 2024-09-16T05:25:03.564 INFO:tasks.rados.rados.2.smithi204.stdout:3854: writing smithi20457400-48 from 504573 to 941112 tid 1 2024-09-16T05:25:03.570 INFO:tasks.rados.rados.2.smithi204.stdout:3854: writing smithi20457400-48 from 1421742 to 2020546 tid 2 2024-09-16T05:25:03.573 INFO:tasks.rados.rados.2.smithi204.stdout:3854: writing smithi20457400-48 from 2437505 to 2504573 tid 3 2024-09-16T05:25:03.573 INFO:tasks.rados.rados.2.smithi204.stdout:3855: write oid 2 current snap is 408 2024-09-16T05:25:03.573 INFO:tasks.rados.rados.2.smithi204.stdout:3854: finishing write tid 1 to smithi20457400-48 2024-09-16T05:25:03.573 INFO:tasks.rados.rados.2.smithi204.stdout:3855: seq_num 1340 ranges {521380=511788,1480224=742462} 2024-09-16T05:25:03.578 INFO:tasks.rados.rados.2.smithi204.stdout:3855: writing smithi20457400-2 from 521380 to 1033168 tid 1 2024-09-16T05:25:03.585 INFO:tasks.rados.rados.2.smithi204.stdout:3855: writing smithi20457400-2 from 1480224 to 2222686 tid 2 2024-09-16T05:25:03.586 INFO:tasks.rados.rados.2.smithi204.stdout:3856: snap_create 2024-09-16T05:25:03.586 INFO:tasks.rados.rados.2.smithi204.stdout:3854: finishing write tid 2 to smithi20457400-48 2024-09-16T05:25:03.586 INFO:tasks.rados.rados.2.smithi204.stdout:3854: finishing write tid 3 to smithi20457400-48 2024-09-16T05:25:03.586 INFO:tasks.rados.rados.2.smithi204.stdout:3854: finishing write tid 4 to smithi20457400-48 2024-09-16T05:25:03.586 INFO:tasks.rados.rados.2.smithi204.stdout:3854: finishing write tid 5 to smithi20457400-48 2024-09-16T05:25:03.586 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 795 (ObjNum 1339 snap 408 seq_num 1339) dirty exists 2024-09-16T05:25:03.586 INFO:tasks.rados.rados.2.smithi204.stdout:3854: left oid 48 (ObjNum 1339 snap 408 seq_num 1339) 2024-09-16T05:25:03.586 INFO:tasks.rados.rados.2.smithi204.stdout:3855: finishing write tid 1 to smithi20457400-2 2024-09-16T05:25:03.587 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:03.587 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:03.587 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:03 smithi032 ceph-mon[27388]: pgmap v1588: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 188 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:25:03.587 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:03 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:03.587 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3114992244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:03.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:03 smithi032 ceph-mon[27388]: osdmap e951: 12 total, 11 up, 12 in 2024-09-16T05:25:03.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:03 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:25:03.588 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:03 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:25:03.588 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:03.589 INFO:tasks.rados.rados.2.smithi204.stdout:3855: finishing write tid 2 to smithi20457400-2 2024-09-16T05:25:03.590 INFO:tasks.rados.rados.2.smithi204.stdout:3855: finishing write tid 3 to smithi20457400-2 2024-09-16T05:25:03.590 INFO:tasks.rados.rados.2.smithi204.stdout:3855: finishing write tid 4 to smithi20457400-2 2024-09-16T05:25:03.590 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 1273 (ObjNum 1340 snap 408 seq_num 1340) dirty exists 2024-09-16T05:25:03.590 INFO:tasks.rados.rados.2.smithi204.stdout:3855: left oid 2 (ObjNum 1340 snap 408 seq_num 1340) 2024-09-16T05:25:03.608 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:03.608 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:03.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:03.629 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:03.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:03.651 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:03.651 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:03.661 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:03.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:03.672 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:03.673 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:03.686 INFO:tasks.rados.rados.2.smithi204.stdout:3852: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:25:03.687 INFO:tasks.rados.rados.2.smithi204.stdout:3852: finishing copy_from to smithi20457400-20 2024-09-16T05:25:03.687 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1404 (ObjNum 1324 snap 403 seq_num 1324) dirty exists 2024-09-16T05:25:03.694 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:03.694 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:03.717 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:25:03.717 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:25:03.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:03 smithi196 ceph-mon[30802]: pgmap v1588: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 188 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 9/170 objects degraded (5.294%) 2024-09-16T05:25:03.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:03 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:03.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3114992244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:03.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:03 smithi196 ceph-mon[30802]: osdmap e951: 12 total, 11 up, 12 in 2024-09-16T05:25:03.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:03 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:25:03.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:03 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:25:03.738 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:03.739 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:03.759 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:03.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:03.781 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:03.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:03.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:03.802 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:03.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:03.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:03.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:04.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:04.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:04.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:04.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:25:04.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:04.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:04.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:04.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:03 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:03.957+0000 7f6af1ad6640 -1 osd.6 952 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:04.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:04.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:04.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:04.369 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:04.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:04 smithi204 ceph-mon[30933]: 2.0 scrub starts 2024-09-16T05:25:04.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:04 smithi204 ceph-mon[30933]: 2.0 scrub ok 2024-09-16T05:25:04.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:04 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:25:04.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:04 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:25:04.390 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:04 smithi204 ceph-mon[30933]: osdmap e952: 12 total, 11 up, 12 in 2024-09-16T05:25:04.390 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2358884292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:04.390 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:04 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:25:04.390 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:04 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:25:04.391 INFO:tasks.rados.rados.2.smithi204.stdout:3852: done (13 left) 2024-09-16T05:25:04.391 INFO:tasks.rados.rados.2.smithi204.stdout:3853: done (12 left) 2024-09-16T05:25:04.392 INFO:tasks.rados.rados.2.smithi204.stdout:3854: done (11 left) 2024-09-16T05:25:04.392 INFO:tasks.rados.rados.2.smithi204.stdout:3855: done (10 left) 2024-09-16T05:25:04.392 INFO:tasks.rados.rados.2.smithi204.stdout:3856: done (9 left) 2024-09-16T05:25:04.392 INFO:tasks.rados.rados.2.smithi204.stdout:3857: delete oid 48 current snap is 409 2024-09-16T05:25:04.394 INFO:tasks.rados.rados.2.smithi204.stdout:3857: done (9 left) 2024-09-16T05:25:04.395 INFO:tasks.rados.rados.2.smithi204.stdout:3858: snap_create 2024-09-16T05:25:04.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:04.419 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:04.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:04 smithi032 ceph-mon[27388]: 2.0 scrub starts 2024-09-16T05:25:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:04 smithi032 ceph-mon[27388]: 2.0 scrub ok 2024-09-16T05:25:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:04 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:25:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:04 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:25:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:04 smithi032 ceph-mon[27388]: osdmap e952: 12 total, 11 up, 12 in 2024-09-16T05:25:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2358884292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:04.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:04 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:25:04.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:04 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:25:04.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:04.541 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:04.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:04.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:04.651 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:04.683 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:04.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:04.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-mon[30802]: 2.0 scrub starts 2024-09-16T05:25:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-mon[30802]: 2.0 scrub ok 2024-09-16T05:25:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:25:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:25:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-mon[30802]: osdmap e952: 12 total, 11 up, 12 in 2024-09-16T05:25:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2358884292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:04.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:25:04.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:25:04.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:04.810 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:04.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:04.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:04.864 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:839 2.1a e438/291 ondisk, result = 0)", 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 517.13231435900002, 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51227693699999999, 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.865 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730490+0000", 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730494+0000", 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0810000000000004e-06 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242338+0000", 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51184441999999997 2024-09-16T05:25:04.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242353+0000", 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.447e-05 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242501+0000", 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014851799999999999 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242532+0000", 2024-09-16T05:25:04.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0406999999999999e-05 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242766+0000", 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000234587 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGRecoveryDelete(2.4c e439,438 2:33100188:::smithi20457400-19:2f,230'190)", 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:25:04.869 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 516.02940325199995, 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.584288378, 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833401+0000", 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833406+0000", 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8940000000000001e-06 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417262+0000", 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.58385600999999998 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417279+0000", 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6880000000000001e-05 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417458+0000", 2024-09-16T05:25:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000178751 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417489+0000", 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1192999999999998e-05 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417689+0000", 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019989399999999999 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5460 2.25 2:a5bc1608:::smithi20457400-11:head [setxattr _header (58) in=7b,truncate 2975004] snapc e1=[e1,e0,df,dd,dc,da,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 380.86837693400003, 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69151579299999999, 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.874 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5460 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994430+0000", 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2729999999999998e-06 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994437+0000", 2024-09-16T05:25:04.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9670000000000004e-06 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994443+0000", 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7780000000000004e-06 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995089+0000", 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064609900000000002 2024-09-16T05:25:04.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995143+0000", 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4042000000000002e-05 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995231+0000", 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7798000000000002e-05 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995551+0000", 2024-09-16T05:25:04.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000319847 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685849+0000", 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.690298254 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685893+0000", 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3368000000000002e-05 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685942+0000", 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9811000000000001e-05 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5461 2.25 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 380.86834978899998, 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69165347799999999, 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.880 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5461 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994457+0000", 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.429e-06 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994463+0000", 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994469+0000", 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4380000000000003e-06 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995446+0000", 2024-09-16T05:25:04.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00097678900000000004 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995491+0000", 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5153e-05 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685982+0000", 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69049136499999997 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686042+0000", 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9896999999999999e-05 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686107+0000", 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5270000000000004e-05 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.885 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6242 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 913] snapc 103=[103,102,101,100,fe,fd,fc,fb,fa,f9,f7,f6,f2,f0,e7] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.04947618199998, 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47407041599999999, 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6242 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813328+0000", 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9399999999999993e-07 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813334+0000", 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1619999999999996e-06 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813347+0000", 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2145e-05 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813354+0000", 2024-09-16T05:25:04.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1230000000000002e-06 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813414+0000", 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0489999999999999e-05 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813421+0000", 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1010000000000003e-06 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816226+0000", 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002804647 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816349+0000", 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012330699999999999 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.281344+0000", 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.464994937 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284684+0000", 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033402309999999999 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287216+0000", 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002531298 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287319+0000", 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102865 2024-09-16T05:25:04.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287398+0000", 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9116000000000006e-05 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6243 2.5 2:a1d8c6d9:::smithi20457400-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 317.04943080499999, 2024-09-16T05:25:04.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47072968100000001, 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6243 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:25:04.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:25:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:25:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1800000000000005e-07 2024-09-16T05:25:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813377+0000", 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3699999999999999e-06 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813385+0000", 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5280000000000003e-06 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813392+0000", 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0600000000000002e-06 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813515+0000", 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012306 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813521+0000", 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.203e-06 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816658+0000", 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003137365 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816689+0000", 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0592000000000001e-05 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.283968+0000", 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46727931 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284030+0000", 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2316000000000004e-05 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284102+0000", 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2158999999999998e-05 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7270 2.1a 2:5827f14d:::smithi20457400-12:head [sparse-read 0~1919657,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:25:04.899 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 218.039300153, 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.003199682, 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7270 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823506+0000", 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1099999999999999e-06 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823515+0000", 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6219999999999998e-06 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823531+0000", 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.632e-05 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823544+0000", 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2508e-05 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832369+0000", 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088252939999999992 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832385+0000", 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5534999999999999e-05 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.822672+0000", 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99028730499999995 2024-09-16T05:25:04.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.822833+0000", 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016117400000000001 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.826703+0000", 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038698140000000001 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7271 2.1a 2:5827f14d:::smithi20457400-12:head [sparse-read 0~1919657] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 218.03924374299999, 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0044004120000001, 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7271 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.905 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:25:04.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6699999999999999e-07 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823563+0000", 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.824e-06 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823569+0000", 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.319e-06 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823574+0000", 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0180000000000001e-06 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832415+0000", 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088402089999999999 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832423+0000", 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8089999999999999e-06 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827702+0000", 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.995278938 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827810+0000", 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107433 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827960+0000", 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150395 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.909 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7272 2.1a 2:5827f14d:::smithi20457400-12:head [read 0~3025491,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 218.039223134, 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0059436770000001, 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.910 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7272 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823581+0000", 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1800000000000002e-07 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823600+0000", 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9296999999999998e-05 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823607+0000", 2024-09-16T05:25:04.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.063e-06 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823612+0000", 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0699999999999997e-06 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832476+0000", 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088636289999999996 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832485+0000", 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5359999999999998e-06 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827992+0000", 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99550709199999998 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.828085+0000", 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3086999999999998e-05 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.829524+0000", 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001438585 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPush(2.29 746/745 [PushOp(2:9662c29c:::smithi20457400-9:120, version: 737'1148, data_included: [2490368~196608,3108095~707292], data_size: 903900, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9662c29c:::smithi20457400-9:120@737'1148, size: 3815387, copy_subset: [(0, 3815387)], clone_subset: {2:9662c29c:::smithi20457400-9:116: []}, snapset: 130=[]:{116: [116], 120: [119], 125: [125], 130: [12f, 12c, 12a, 126]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3815387, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2490368, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 208.873128495, 2024-09-16T05:25:04.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86202506199999995, 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989676+0000", 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8999999999999997e-07 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.990550+0000", 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087471100000000002 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850633+0000", 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86008260599999997 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850658+0000", 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5537999999999998e-05 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850763+0000", 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104546 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850778+0000", 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4603999999999999e-05 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.851700+0000", 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00092256700000000001 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.919 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7554 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 768] snapc 13f=[13f,13d,13b,134,133,12c,12a,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 192.64302234900001, 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010270486000000001, 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7554 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219781+0000", 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219783+0000", 2024-09-16T05:25:04.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1430000000000001e-06 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219795+0000", 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2174999999999999e-05 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219821+0000", 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5364e-05 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219831+0000", 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0069e-05 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.219911+0000", 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0131999999999998e-05 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.220032+0000", 2024-09-16T05:25:04.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000121305 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.224076+0000", 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040437390000000002 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.227596+0000", 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035194359999999999 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.229981+0000", 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023851279999999998 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.230018+0000", 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7030999999999998e-05 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.230052+0000", 2024-09-16T05:25:04.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3964000000000002e-05 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7717 2.f 2:f08ec412:::smithi20457400-44:head [copy-from ver 1121] snapc 149=[149,148,147,146,145,144,143,142,141,13b,133,12c,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e778)", 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 176.768655239, 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010866597, 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.926 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7717 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094148+0000", 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094150+0000", 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4899999999999999e-06 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094158+0000", 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9980000000000003e-06 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094174+0000", 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6291e-05 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094184+0000", 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6369999999999994e-06 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094245+0000", 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1261999999999997e-05 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.094313+0000", 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7793999999999995e-05 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9", 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.100828+0000", 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0065157000000000001 2024-09-16T05:25:04.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104619+0000", 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037903139999999999 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104944+0000", 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032508400000000001 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:04.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.104979+0000", 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5567e-05 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:08.105015+0000", 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5460000000000003e-05 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:1707 2.1a e799/764 ondisk, result = 0)", 2024-09-16T05:25:04.932 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 156.000933132, 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69806231100000005, 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:25:04.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861871+0000", 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7800000000000001e-07 2024-09-16T05:25:04.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861876+0000", 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4889999999999997e-06 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559739+0000", 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69786377399999999 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559746+0000", 2024-09-16T05:25:04.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3409999999999999e-06 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559833+0000", 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7454000000000007e-05 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559840+0000", 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559933+0000", 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2936000000000006e-05 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8288 2.25 2:a7ee7933:::smithi20457400-20:head [copy-from ver 750] snapc 15a=[15a] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.94473756900001, 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01619603, 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.937 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8288 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918068+0000", 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4499999999999998e-06 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918095+0000", 2024-09-16T05:25:04.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6695000000000001e-05 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918117+0000", 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2120000000000002e-05 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918127+0000", 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0074e-05 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918253+0000", 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125339 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918260+0000", 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9530000000000002e-06 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.921653+0000", 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033933230000000002 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.921758+0000", 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104949 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.927615+0000", 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058567039999999999 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.933643+0000", 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006028233 2024-09-16T05:25:04.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934133+0000", 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00049045800000000004 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934202+0000", 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8817000000000002e-05 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934262+0000", 2024-09-16T05:25:04.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9914999999999998e-05 2024-09-16T05:25:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8289 2.25 2:a7ee7933:::smithi20457400-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:25:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:25:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 130.944638685, 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012771173, 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8289 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2500000000000005e-07 2024-09-16T05:25:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918187+0000", 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1101999999999999e-05 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918198+0000", 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1139e-05 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918203+0000", 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5589999999999996e-06 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918361+0000", 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015787099999999999 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918367+0000", 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2389999999999999e-06 2024-09-16T05:25:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.922539+0000", 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041721240000000001 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.922594+0000", 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4969000000000002e-05 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930701+0000", 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081063300000000001 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930817+0000", 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116116 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930936+0000", 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011909900000000001 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8964 2.5 2:a1d8c6d9:::smithi20457400-3:head [setxattr _header (57) in=7b,truncate 2773452] snapc 183=[183,181,180,17e,17c,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e900)", 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:10.090030+0000", 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.772773505000004, 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.090973712999999998, 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8964 2024-09-16T05:25:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090030+0000", 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090030+0000", 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090031+0000", 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3499999999999996e-07 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090034+0000", 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3979999999999999e-06 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090041+0000", 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2230000000000001e-06 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090047+0000", 2024-09-16T05:25:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7359999999999998e-06 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090919+0000", 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087230900000000004 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090980+0000", 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0677000000000002e-05 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.091076+0000", 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5878999999999995e-05 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.091952+0000", 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087587299999999999 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.180909+0000", 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088957350000000004 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.180954+0000", 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5303000000000003e-05 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.181004+0000", 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9429999999999999e-05 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8965 2.5 2:a1d8c6d9:::smithi20457400-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e900)", 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:10.090055+0000", 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.772748591999999, 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.091122063000000003, 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8965 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090055+0000", 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090055+0000", 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090055+0000", 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.51e-07 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090058+0000", 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2910000000000002e-06 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090064+0000", 2024-09-16T05:25:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2040000000000002e-06 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090069+0000", 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4609999999999999e-06 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.091527+0000", 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014578320000000001 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.091608+0000", 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1213999999999999e-05 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.181042+0000", 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.089433828000000007 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.181105+0000", 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2632000000000006e-05 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.181177+0000", 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2150000000000005e-05 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9089 2.1a 2:580e4d0e:::smithi20457400-29:head [copy-from ver 1102] snapc 18a=[18a,189,188,187,186,185,184,17e,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e911)", 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:21.388513+0000", 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 43.474291000000001, 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010229436, 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9089 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.388513+0000", 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.388513+0000", 2024-09-16T05:25:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.388515+0000", 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.018e-06 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.388524+0000", 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.388541+0000", 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7087000000000001e-05 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.388551+0000", 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6290000000000005e-06 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.388625+0000", 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4617999999999996e-05 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.388699+0000", 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3800999999999994e-05 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.392619+0000", 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039202869999999997 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.396826+0000", 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042069660000000004 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:04.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.398668+0000", 2024-09-16T05:25:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018419409999999999 2024-09-16T05:25:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.398706+0000", 2024-09-16T05:25:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7934e-05 2024-09-16T05:25:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:21.398742+0000", 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5685000000000001e-05 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9405 2.25 2:a5bc1608:::smithi20457400-11:head [copy-from ver 1039] snapc 193=[191,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:46.327516+0000", 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.535287601, 2024-09-16T05:25:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016089552999999999, 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9405 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327516+0000", 2024-09-16T05:25:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327516+0000", 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327516+0000", 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999996e-07 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327521+0000", 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.206e-06 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327528+0000", 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1130000000000003e-06 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327533+0000", 2024-09-16T05:25:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8740000000000003e-06 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327589+0000", 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6150999999999998e-05 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327672+0000", 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3502000000000001e-05 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:25:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.336126+0000", 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084542309999999996 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.341290+0000", 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051636030000000001 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.343539+0000", 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022492419999999998 2024-09-16T05:25:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.343575+0000", 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5695999999999998e-05 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.343605+0000", 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0437999999999999e-05 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9406 2.25 2:a5bc1608:::smithi20457400-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e936)", 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:46.327549+0000", 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.535254627, 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012786812999999999, 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9406 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327549+0000", 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327549+0000", 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327549+0000", 2024-09-16T05:25:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1600000000000001e-07 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327552+0000", 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4940000000000002e-06 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327559+0000", 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9909999999999996e-06 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327564+0000", 2024-09-16T05:25:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3890000000000004e-06 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327836+0000", 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027136599999999999 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327893+0000", 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7389999999999998e-05 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.340178+0000", 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012284669 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.340245+0000", 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6937e-05 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.340336+0000", 2024-09-16T05:25:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1061000000000001e-05 2024-09-16T05:25:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:04.982 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:25:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:05.004 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:05.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:05.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:25:05.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:05.131 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:05.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:05.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:05.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:04 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:04.976+0000 7f6af1ad6640 -1 osd.6 952 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:05.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:05.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:05.361 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:05.385 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:05.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:05.392 INFO:tasks.rados.rados.2.smithi204.stdout:3858: done (9 left) 2024-09-16T05:25:05.392 INFO:tasks.rados.rados.2.smithi204.stdout:3859: snap_remove snap 408 2024-09-16T05:25:05.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:05.507 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:05.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:05.610 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:05.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:05.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:05.636 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:05 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:25:05.637 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:05 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:25:05.637 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:05 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:05.637 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:05 smithi204 ceph-mon[30933]: pgmap v1591: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 193 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.7 MiB/s wr, 15 op/s; 9/176 objects degraded (5.114%) 2024-09-16T05:25:05.637 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:05 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 80 sec, osd.6 has slow ops) 2024-09-16T05:25:05.637 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:05 smithi204 ceph-mon[30933]: osdmap e953: 12 total, 11 up, 12 in 2024-09-16T05:25:05.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:25:05.678 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:05 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:25:05.678 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:05 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:25:05.678 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:05 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:05.678 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:05 smithi032 ceph-mon[27388]: pgmap v1591: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 193 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.7 MiB/s wr, 15 op/s; 9/176 objects degraded (5.114%) 2024-09-16T05:25:05.678 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:05 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 80 sec, osd.6 has slow ops) 2024-09-16T05:25:05.678 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:05 smithi032 ceph-mon[27388]: osdmap e953: 12 total, 11 up, 12 in 2024-09-16T05:25:05.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:05.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:05 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:25:05.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:05 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:25:05.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:05 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:05.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:05 smithi196 ceph-mon[30802]: pgmap v1591: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 193 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.7 MiB/s wr, 15 op/s; 9/176 objects degraded (5.114%) 2024-09-16T05:25:05.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:05 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 80 sec, osd.6 has slow ops) 2024-09-16T05:25:05.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:05 smithi196 ceph-mon[30802]: osdmap e953: 12 total, 11 up, 12 in 2024-09-16T05:25:05.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:05.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:05.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:05.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:05.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:05.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:05.944 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:05.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:06.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:06.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:06.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:06.138 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:06.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:06.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:06.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:05 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:05.940+0000 7f6af1ad6640 -1 osd.6 954 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:06.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:06.262 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:06.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:06.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:06.390 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:06.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:06.402 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:25:06.408 INFO:tasks.rados.rados.2.smithi204.stdout:3859: done (9 left) 2024-09-16T05:25:06.408 INFO:tasks.rados.rados.2.smithi204.stdout:3860: write (excl) oid 4 current snap is 410 2024-09-16T05:25:06.408 INFO:tasks.rados.rados.2.smithi204.stdout:3860: seq_num 1342 ranges {554994=662286,1997188=557806} 2024-09-16T05:25:06.415 INFO:tasks.rados.rados.2.smithi204.stdout:3860: writing smithi20457400-4 from 554994 to 1217280 tid 1 2024-09-16T05:25:06.423 INFO:tasks.rados.rados.2.smithi204.stdout:3860: writing smithi20457400-4 from 1997188 to 2554994 tid 2 2024-09-16T05:25:06.423 INFO:tasks.rados.rados.2.smithi204.stdout:3861: snap_remove snap 409 2024-09-16T05:25:06.434 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:25:06.434 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:25:06.434 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:25:06.434 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:25:06.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:06.514 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:06.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:06.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:06.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:06.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:06.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:06.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:25:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:06 smithi032 ceph-mon[27388]: 2.40 scrub starts 2024-09-16T05:25:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:06 smithi032 ceph-mon[27388]: 2.40 scrub ok 2024-09-16T05:25:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:06 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:06 smithi032 ceph-mon[27388]: 2.50 scrub starts 2024-09-16T05:25:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:06 smithi032 ceph-mon[27388]: 2.50 scrub ok 2024-09-16T05:25:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:06 smithi032 ceph-mon[27388]: osdmap e954: 12 total, 11 up, 12 in 2024-09-16T05:25:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:06 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/176 objects degraded (5.114%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:06.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2120087403' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:06 smithi204 ceph-mon[30933]: 2.40 scrub starts 2024-09-16T05:25:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:06 smithi204 ceph-mon[30933]: 2.40 scrub ok 2024-09-16T05:25:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:06 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:06 smithi204 ceph-mon[30933]: 2.50 scrub starts 2024-09-16T05:25:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:06 smithi204 ceph-mon[30933]: 2.50 scrub ok 2024-09-16T05:25:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:06 smithi204 ceph-mon[30933]: osdmap e954: 12 total, 11 up, 12 in 2024-09-16T05:25:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:06 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/176 objects degraded (5.114%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:06.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2120087403' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-mon[30802]: 2.40 scrub starts 2024-09-16T05:25:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-mon[30802]: 2.40 scrub ok 2024-09-16T05:25:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-mon[30802]: 2.50 scrub starts 2024-09-16T05:25:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-mon[30802]: 2.50 scrub ok 2024-09-16T05:25:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-mon[30802]: osdmap e954: 12 total, 11 up, 12 in 2024-09-16T05:25:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/176 objects degraded (5.114%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:06.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2120087403' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:06.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:06.767 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:06.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:06.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:06.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:06.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:06.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:07.020 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:07.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:07.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:25:07.085 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:07.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:07.143 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:07.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:07.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:06 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:06.893+0000 7f6af1ad6640 -1 osd.6 955 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:07.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:07.267 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:07.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:07.288 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:07.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:07.391 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:07.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:07.417 INFO:tasks.rados.rados.2.smithi204.stdout:3861: done (10 left) 2024-09-16T05:25:07.417 INFO:tasks.rados.rados.2.smithi204.stdout:3862: read oid 41 snap -1 2024-09-16T05:25:07.417 INFO:tasks.rados.rados.2.smithi204.stdout:3862: expect (ObjNum 1338 snap 408 seq_num 1338) 2024-09-16T05:25:07.417 INFO:tasks.rados.rados.2.smithi204.stdout:3863: read oid 27 snap -1 2024-09-16T05:25:07.417 INFO:tasks.rados.rados.2.smithi204.stdout:3863: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:25:07.417 INFO:tasks.rados.rados.2.smithi204.stdout:3864: read oid 11 snap 362 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:3864: expect (ObjNum 1149 snap 342 seq_num 1149) 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:3860: finishing write tid 1 to smithi20457400-4 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:3860: finishing write tid 2 to smithi20457400-4 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:3860: finishing write tid 3 to smithi20457400-4 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:3860: finishing write tid 4 to smithi20457400-4 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 4 v 1366 (ObjNum 1342 snap 410 seq_num 1342) dirty exists 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:3860: left oid 4 (ObjNum 1342 snap 410 seq_num 1342) 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:3860: done (12 left) 2024-09-16T05:25:07.418 INFO:tasks.rados.rados.2.smithi204.stdout:3865: delete oid 47 current snap is 410 2024-09-16T05:25:07.422 INFO:tasks.rados.rados.2.smithi204.stdout:3865: done (12 left) 2024-09-16T05:25:07.422 INFO:tasks.rados.rados.2.smithi204.stdout:3866: snap_create 2024-09-16T05:25:07.426 INFO:tasks.rados.rados.2.smithi204.stdout:3864: expect (ObjNum 1149 snap 342 seq_num 1149) 2024-09-16T05:25:07.491 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [1] live_osds: [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:25:07.491 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:25:07.491 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T05:25:07.491 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-09-16T05:25:07.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 0 2024-09-16T05:25:07.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:07.515 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:07.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:07.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:07.575 INFO:tasks.rados.rados.2.smithi204.stdout:3862: expect (ObjNum 1338 snap 408 seq_num 1338) 2024-09-16T05:25:07.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.19 scrub starts 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.19 scrub ok 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.18 scrub starts 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.18 scrub ok 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: pgmap v1594: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 199 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 11 op/s; 9/184 objects degraded (4.891%) 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2120087403' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:07.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: osdmap e955: 12 total, 11 up, 12 in 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.16 scrub starts 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.16 scrub ok 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.1c scrub starts 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.1c scrub ok 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.d deep-scrub starts 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.d deep-scrub ok 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:25:07.627 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:07 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:25:07.629 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:07.630 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:07.630 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:07.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:07.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:07.652 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:07.652 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.19 scrub starts 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.19 scrub ok 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.18 scrub starts 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.18 scrub ok 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: pgmap v1594: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 199 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 11 op/s; 9/184 objects degraded (4.891%) 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:25:07.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2120087403' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: osdmap e955: 12 total, 11 up, 12 in 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.16 scrub starts 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.16 scrub ok 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.1c scrub starts 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.1c scrub ok 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.d deep-scrub starts 2024-09-16T05:25:07.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.d deep-scrub ok 2024-09-16T05:25:07.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:25:07.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:07 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:25:07.673 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:07.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:07.691 INFO:tasks.rados.rados.2.smithi204.stdout:3863: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:25:07.696 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:07.697 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:07.718 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:07.719 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:07.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.19 scrub starts 2024-09-16T05:25:07.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.19 scrub ok 2024-09-16T05:25:07.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.18 scrub starts 2024-09-16T05:25:07.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.18 scrub ok 2024-09-16T05:25:07.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:07.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:25:07.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:25:07.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: pgmap v1594: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 199 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 11 op/s; 9/184 objects degraded (4.891%) 2024-09-16T05:25:07.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:25:07.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:25:07.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:07.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2120087403' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:07.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: osdmap e955: 12 total, 11 up, 12 in 2024-09-16T05:25:07.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:25:07.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:25:07.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.16 scrub starts 2024-09-16T05:25:07.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.16 scrub ok 2024-09-16T05:25:07.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.1c scrub starts 2024-09-16T05:25:07.741 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.1c scrub ok 2024-09-16T05:25:07.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.d deep-scrub starts 2024-09-16T05:25:07.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.d deep-scrub ok 2024-09-16T05:25:07.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:25:07.742 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:07 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:25:07.743 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:07.743 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:07.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:07.751 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:07.763 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:25:07.763 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:25:07.766 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:07.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:07.786 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:07.787 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:07.808 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:07.808 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:07.829 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:07.829 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:07.833 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:07.833 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:07.850 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:07.850 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:07.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:07.890 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:07.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:07.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:08.014 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:08.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:08.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:08.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:08.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:08.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:08.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:08.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:08.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:08.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:08.418 INFO:tasks.rados.rados.2.smithi204.stdout:3862: done (12 left) 2024-09-16T05:25:08.418 INFO:tasks.rados.rados.2.smithi204.stdout:3863: done (11 left) 2024-09-16T05:25:08.418 INFO:tasks.rados.rados.2.smithi204.stdout:3864: done (10 left) 2024-09-16T05:25:08.418 INFO:tasks.rados.rados.2.smithi204.stdout:3866: done (9 left) 2024-09-16T05:25:08.418 INFO:tasks.rados.rados.2.smithi204.stdout:3867: write oid 27 current snap is 411 2024-09-16T05:25:08.419 INFO:tasks.rados.rados.2.smithi204.stdout:3867: seq_num 1344 ranges {588608=412784,1597799=749800} 2024-09-16T05:25:08.423 INFO:tasks.rados.rados.2.smithi204.stdout:3867: writing smithi20457400-27 from 588608 to 1001392 tid 1 2024-09-16T05:25:08.431 INFO:tasks.rados.rados.2.smithi204.stdout:3867: writing smithi20457400-27 from 1597799 to 2347599 tid 2 2024-09-16T05:25:08.432 INFO:tasks.rados.rados.2.smithi204.stdout:3868: read oid 21 snap -1 2024-09-16T05:25:08.432 INFO:tasks.rados.rados.2.smithi204.stdout:3868: expect (ObjNum 1334 snap 405 seq_num 1334) 2024-09-16T05:25:08.432 INFO:tasks.rados.rados.2.smithi204.stdout:3869: write oid 44 current snap is 411 2024-09-16T05:25:08.432 INFO:tasks.rados.rados.2.smithi204.stdout:3867: finishing write tid 1 to smithi20457400-27 2024-09-16T05:25:08.432 INFO:tasks.rados.rados.2.smithi204.stdout:3869: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-09-16T05:25:08.437 INFO:tasks.rados.rados.2.smithi204.stdout:3869: writing smithi20457400-44 from 605415 to 1093448 tid 1 2024-09-16T05:25:08.444 INFO:tasks.rados.rados.2.smithi204.stdout:3869: writing smithi20457400-44 from 1656281 to 2149739 tid 2 2024-09-16T05:25:08.444 INFO:tasks.rados.rados.2.smithi204.stdout:3870: write (excl) oid 40 current snap is 411 2024-09-16T05:25:08.444 INFO:tasks.rados.rados.2.smithi204.stdout:3867: finishing write tid 2 to smithi20457400-27 2024-09-16T05:25:08.444 INFO:tasks.rados.rados.2.smithi204.stdout:3867: finishing write tid 3 to smithi20457400-27 2024-09-16T05:25:08.445 INFO:tasks.rados.rados.2.smithi204.stdout:3867: finishing write tid 4 to smithi20457400-27 2024-09-16T05:25:08.445 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 1274 (ObjNum 1344 snap 411 seq_num 1344) dirty exists 2024-09-16T05:25:08.445 INFO:tasks.rados.rados.2.smithi204.stdout:3867: left oid 27 (ObjNum 1344 snap 411 seq_num 1344) 2024-09-16T05:25:08.445 INFO:tasks.rados.rados.2.smithi204.stdout:3869: finishing write tid 1 to smithi20457400-44 2024-09-16T05:25:08.445 INFO:tasks.rados.rados.2.smithi204.stdout:3868: expect (ObjNum 1334 snap 405 seq_num 1334) 2024-09-16T05:25:08.496 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:08.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:08.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:08.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:25:08.616 INFO:tasks.rados.rados.2.smithi204.stdout:3869: finishing write tid 2 to smithi20457400-44 2024-09-16T05:25:08.616 INFO:tasks.rados.rados.2.smithi204.stdout:3869: finishing write tid 3 to smithi20457400-44 2024-09-16T05:25:08.616 INFO:tasks.rados.rados.2.smithi204.stdout:3869: finishing write tid 4 to smithi20457400-44 2024-09-16T05:25:08.616 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1156 (ObjNum 1345 snap 411 seq_num 1345) dirty exists 2024-09-16T05:25:08.616 INFO:tasks.rados.rados.2.smithi204.stdout:3869: left oid 44 (ObjNum 1345 snap 411 seq_num 1345) 2024-09-16T05:25:08.616 INFO:tasks.rados.rados.2.smithi204.stdout:3870: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-09-16T05:25:08.622 INFO:tasks.rados.rados.2.smithi204.stdout:3870: writing smithi20457400-40 from 622222 to 1301857 tid 1 2024-09-16T05:25:08.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:08.630 INFO:tasks.rados.rados.2.smithi204.stdout:3870: writing smithi20457400-40 from 1831116 to 2584585 tid 2 2024-09-16T05:25:08.630 INFO:tasks.rados.rados.2.smithi204.stdout:3867: done (12 left) 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:3868: done (11 left) 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:3869: done (10 left) 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:3871: rollback oid 36 current snap is 411 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 36 to 362 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:3870: finishing write tid 1 to smithi20457400-40 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:3872: read oid 41 snap 358 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:3872: expect (ObjNum 1142 snap 341 seq_num 1142) 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:3873: read oid 5 snap 410 2024-09-16T05:25:08.631 INFO:tasks.rados.rados.2.smithi204.stdout:3873: expect (ObjNum 1325 snap 403 seq_num 1325) 2024-09-16T05:25:08.632 INFO:tasks.rados.rados.2.smithi204.stdout:3874: snap_remove snap 407 2024-09-16T05:25:08.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:08.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:08.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:08.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:25:08.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:25:08.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: osdmap e956: 12 total, 11 up, 12 in 2024-09-16T05:25:08.656 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3275445999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:08.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: 2.4b deep-scrub starts 2024-09-16T05:25:08.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: 2.4b deep-scrub ok 2024-09-16T05:25:08.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: 1.1a scrub starts 2024-09-16T05:25:08.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:08 smithi032 ceph-mon[27388]: 1.1a scrub ok 2024-09-16T05:25:08.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:08.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:25:08.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:25:08.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: osdmap e956: 12 total, 11 up, 12 in 2024-09-16T05:25:08.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3275445999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:08.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: 2.4b deep-scrub starts 2024-09-16T05:25:08.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: 2.4b deep-scrub ok 2024-09-16T05:25:08.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: 1.1a scrub starts 2024-09-16T05:25:08.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:08 smithi204 ceph-mon[30933]: 1.1a scrub ok 2024-09-16T05:25:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:08.686 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:08.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:08.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:25:08.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:25:08.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: osdmap e956: 12 total, 11 up, 12 in 2024-09-16T05:25:08.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3275445999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:08.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: 2.4b deep-scrub starts 2024-09-16T05:25:08.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: 2.4b deep-scrub ok 2024-09-16T05:25:08.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: 1.1a scrub starts 2024-09-16T05:25:08.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:08 smithi196 ceph-mon[30802]: 1.1a scrub ok 2024-09-16T05:25:08.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:08.772 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:08.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:08.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:08.807 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:08.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:08.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:08.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:09.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:09.024 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:09.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:09.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:25:09.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:09.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:09.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:09.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:09.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:09.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:09.317 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:09.317 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:09.318 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:09.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:09.399 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:09.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:09.420 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.4 primary-affinity to 0 (802) 2024-09-16T05:25:09.426 INFO:tasks.rados.rados.2.smithi204.stdout:3874: done (13 left) 2024-09-16T05:25:09.426 INFO:tasks.rados.rados.2.smithi204.stdout:3875: rollback oid 3 current snap is 411 2024-09-16T05:25:09.426 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 358 2024-09-16T05:25:09.427 INFO:tasks.rados.rados.2.smithi204.stdout:3870: finishing write tid 2 to smithi20457400-40 2024-09-16T05:25:09.427 INFO:tasks.rados.rados.2.smithi204.stdout:3870: finishing write tid 3 to smithi20457400-40 2024-09-16T05:25:09.427 INFO:tasks.rados.rados.2.smithi204.stdout:3870: finishing write tid 4 to smithi20457400-40 2024-09-16T05:25:09.427 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1071 (ObjNum 1346 snap 411 seq_num 1346) dirty exists 2024-09-16T05:25:09.427 INFO:tasks.rados.rados.2.smithi204.stdout:3870: left oid 40 (ObjNum 1346 snap 411 seq_num 1346) 2024-09-16T05:25:09.427 INFO:tasks.rados.rados.2.smithi204.stdout:3871: finishing rollback tid 1 to smithi20457400-36 2024-09-16T05:25:09.427 INFO:tasks.rados.rados.2.smithi204.stdout:3873: expect (ObjNum 1325 snap 403 seq_num 1325) 2024-09-16T05:25:09.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:09.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:09.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:09.528 INFO:tasks.rados.rados.2.smithi204.stdout:3872: expect (ObjNum 1142 snap 341 seq_num 1142) 2024-09-16T05:25:09.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:09.639 INFO:tasks.rados.rados.2.smithi204.stdout:3871: finishing rollback tid 2 to smithi20457400-36 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 811 (ObjNum 1158 snap 348 seq_num 1158) dirty exists 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:3875: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:3875: finishing rollback tid 2 to smithi20457400-3 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1111 (ObjNum 1140 snap 341 seq_num 1140) dirty exists 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:3870: done (13 left) 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:3871: done (12 left) 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:3872: done (11 left) 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:3873: done (10 left) 2024-09-16T05:25:09.640 INFO:tasks.rados.rados.2.smithi204.stdout:3875: done (9 left) 2024-09-16T05:25:09.641 INFO:tasks.rados.rados.2.smithi204.stdout:3876: copy_from oid 28 from oid 50 current snap is 411 2024-09-16T05:25:09.641 INFO:tasks.rados.rados.2.smithi204.stdout:3877: copy_from oid 8 from oid 20 current snap is 411 2024-09-16T05:25:09.641 INFO:tasks.rados.rados.2.smithi204.stdout:3878: copy_from oid 7 from oid 12 current snap is 411 2024-09-16T05:25:09.641 INFO:tasks.rados.rados.2.smithi204.stdout:3879: snap_remove snap 410 2024-09-16T05:25:09.654 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:09.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:09.660 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:09 smithi204 ceph-mon[30933]: pgmap v1597: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:25:09.660 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:09 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 90 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:09.660 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:09 smithi204 ceph-mon[30933]: osdmap e957: 12 total, 11 up, 12 in 2024-09-16T05:25:09.660 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1782245962' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-16T05:25:09.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:09 smithi032 ceph-mon[27388]: pgmap v1597: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:25:09.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:09 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 90 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:09.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:09 smithi032 ceph-mon[27388]: osdmap e957: 12 total, 11 up, 12 in 2024-09-16T05:25:09.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1782245962' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-16T05:25:09.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:09 smithi196 ceph-mon[30802]: pgmap v1597: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.1 MiB/s wr, 14 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:25:09.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:09 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 90 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:09.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:09 smithi196 ceph-mon[30802]: osdmap e957: 12 total, 11 up, 12 in 2024-09-16T05:25:09.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1782245962' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-16T05:25:09.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:09.780 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:09.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:09.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:09.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:09.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:25:10.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:10.031 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:10.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:10.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:10.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:10.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:10.211 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:10.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:10.282 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:10.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:10.383 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:10.405 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:10.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3879: done (12 left) 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3880: read oid 3 snap -1 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3880: expect (ObjNum 1140 snap 341 seq_num 1140) 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3881: snap_create 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3878: finishing copy_from to smithi20457400-7 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3878: got expected ENOENT (src dne) 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3878: finishing copy_from racing read to smithi20457400-7 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3876: finishing copy_from racing read to smithi20457400-28 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3877: finishing copy_from racing read to smithi20457400-8 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:3877: finishing copy_from to smithi20457400-8 2024-09-16T05:25:10.426 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1404 (ObjNum 1324 snap 403 seq_num 1324) dirty exists 2024-09-16T05:25:10.427 INFO:tasks.rados.rados.2.smithi204.stdout:3876: finishing copy_from to smithi20457400-28 2024-09-16T05:25:10.427 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1246 (ObjNum 1333 snap 405 seq_num 1333) dirty exists 2024-09-16T05:25:10.432 INFO:tasks.rados.rados.2.smithi204.stdout:3880: expect (ObjNum 1140 snap 341 seq_num 1140) 2024-09-16T05:25:10.482 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:10 smithi032 ceph-mon[27388]: 1.15 scrub starts 2024-09-16T05:25:10.482 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:10 smithi032 ceph-mon[27388]: 1.15 scrub ok 2024-09-16T05:25:10.482 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1782245962' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-16T05:25:10.482 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:10 smithi032 ceph-mon[27388]: osdmap e958: 12 total, 11 up, 12 in 2024-09-16T05:25:10.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:10.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:10.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:10.589 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:10 smithi204 ceph-mon[30933]: 1.15 scrub starts 2024-09-16T05:25:10.589 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:10 smithi204 ceph-mon[30933]: 1.15 scrub ok 2024-09-16T05:25:10.589 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1782245962' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-16T05:25:10.590 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:10 smithi204 ceph-mon[30933]: osdmap e958: 12 total, 11 up, 12 in 2024-09-16T05:25:10.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:10.655 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:10.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:10.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:25:10.697 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:10.698 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:10.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:10 smithi196 ceph-mon[30802]: 1.15 scrub starts 2024-09-16T05:25:10.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:10 smithi196 ceph-mon[30802]: 1.15 scrub ok 2024-09-16T05:25:10.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1782245962' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-16T05:25:10.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:10 smithi196 ceph-mon[30802]: osdmap e958: 12 total, 11 up, 12 in 2024-09-16T05:25:10.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:10.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:10.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:10.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:10.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:10.938 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:11.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:11.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:11.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:11.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:11.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:11.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:11.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:25:11.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:11.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:11.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:11.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:11.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:11.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:11.432 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:25:11.437 INFO:tasks.rados.rados.2.smithi204.stdout:3876: done (13 left) 2024-09-16T05:25:11.437 INFO:tasks.rados.rados.2.smithi204.stdout:3877: done (12 left) 2024-09-16T05:25:11.437 INFO:tasks.rados.rados.2.smithi204.stdout:3878: done (11 left) 2024-09-16T05:25:11.437 INFO:tasks.rados.rados.2.smithi204.stdout:3880: done (10 left) 2024-09-16T05:25:11.437 INFO:tasks.rados.rados.2.smithi204.stdout:3881: done (9 left) 2024-09-16T05:25:11.437 INFO:tasks.rados.rados.2.smithi204.stdout:3882: snap_create 2024-09-16T05:25:11.497 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:11.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:11.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:11.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:11.618 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:11 smithi204 ceph-mon[30933]: pgmap v1600: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 203 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:25:11.618 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:11 smithi204 ceph-mon[30933]: osdmap e959: 12 total, 11 up, 12 in 2024-09-16T05:25:11.618 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2701499562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:11.618 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:11 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/188 objects degraded (4.787%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:11.618 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:11 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:25:11.618 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:11 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:25:11.618 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:11 smithi204 ceph-mon[30933]: 1.e deep-scrub starts 2024-09-16T05:25:11.619 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:11 smithi204 ceph-mon[30933]: 1.e deep-scrub ok 2024-09-16T05:25:11.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:11.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:11.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:11.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:11 smithi032 ceph-mon[27388]: pgmap v1600: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 203 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:25:11.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:11 smithi032 ceph-mon[27388]: osdmap e959: 12 total, 11 up, 12 in 2024-09-16T05:25:11.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2701499562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:11.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:11 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/188 objects degraded (4.787%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:11.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:11 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:25:11.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:11 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:25:11.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:11 smithi032 ceph-mon[27388]: 1.e deep-scrub starts 2024-09-16T05:25:11.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:11 smithi032 ceph-mon[27388]: 1.e deep-scrub ok 2024-09-16T05:25:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:11 smithi196 ceph-mon[30802]: pgmap v1600: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 203 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 9/188 objects degraded (4.787%) 2024-09-16T05:25:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:11 smithi196 ceph-mon[30802]: osdmap e959: 12 total, 11 up, 12 in 2024-09-16T05:25:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2701499562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:11 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/188 objects degraded (4.787%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:11 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:25:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:11 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:25:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:11 smithi196 ceph-mon[30802]: 1.e deep-scrub starts 2024-09-16T05:25:11.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:11 smithi196 ceph-mon[30802]: 1.e deep-scrub ok 2024-09-16T05:25:11.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:11.790 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:11.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:11.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:11.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:11.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:11.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:12.005 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:12.006 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:12.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:12.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:12.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:12.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:12.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:12.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:12.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:25:12.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:12.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:12.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:25:12.318 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:12.318 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:12.339 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:12.340 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:12.362 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:12.362 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:12.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:12.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:12.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:12.445 INFO:tasks.rados.rados.2.smithi204.stdout:3882: done (9 left) 2024-09-16T05:25:12.445 INFO:tasks.rados.rados.2.smithi204.stdout:3883: read oid 32 snap -1 2024-09-16T05:25:12.445 INFO:tasks.rados.rados.2.smithi204.stdout:3883: expect (ObjNum 1259 snap 393 seq_num 1259) 2024-09-16T05:25:12.445 INFO:tasks.rados.rados.2.smithi204.stdout:3884: read oid 7 snap -1 2024-09-16T05:25:12.446 INFO:tasks.rados.rados.2.smithi204.stdout:3884: expect deleted 2024-09-16T05:25:12.446 INFO:tasks.rados.rados.2.smithi204.stdout:3885: write oid 1 current snap is 413 2024-09-16T05:25:12.446 INFO:tasks.rados.rados.2.smithi204.stdout:3885: seq_num 1347 ranges {639029=754884,2173245=465784} 2024-09-16T05:25:12.454 INFO:tasks.rados.rados.2.smithi204.stdout:3885: writing smithi20457400-1 from 639029 to 1393913 tid 1 2024-09-16T05:25:12.462 INFO:tasks.rados.rados.2.smithi204.stdout:3885: writing smithi20457400-1 from 2173245 to 2639029 tid 2 2024-09-16T05:25:12.462 INFO:tasks.rados.rados.2.smithi204.stdout:3886: delete oid 40 current snap is 413 2024-09-16T05:25:12.462 INFO:tasks.rados.rados.2.smithi204.stdout:3883: expect (ObjNum 1259 snap 393 seq_num 1259) 2024-09-16T05:25:12.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:12.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:12.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:12.588 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:12.589 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:12.608 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:12.608 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:12.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:12 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:12.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2701499562' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:12.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:12 smithi032 ceph-mon[27388]: osdmap e960: 12 total, 11 up, 12 in 2024-09-16T05:25:12.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/472527799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:12.610 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:12 smithi032 ceph-mon[27388]: 1.1a scrub starts 2024-09-16T05:25:12.610 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:12 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:25:12.628 INFO:tasks.rados.rados.2.smithi204.stdout:3885: finishing write tid 1 to smithi20457400-1 2024-09-16T05:25:12.628 INFO:tasks.rados.rados.2.smithi204.stdout:3885: finishing write tid 2 to smithi20457400-1 2024-09-16T05:25:12.628 INFO:tasks.rados.rados.2.smithi204.stdout:3885: finishing write tid 3 to smithi20457400-1 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:3885: finishing write tid 4 to smithi20457400-1 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 1151 (ObjNum 1347 snap 413 seq_num 1347) dirty exists 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:3885: left oid 1 (ObjNum 1347 snap 413 seq_num 1347) 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:3883: done (12 left) 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:3884: done (11 left) 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:3885: done (10 left) 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:3886: done (9 left) 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:3887: write (excl) oid 45 current snap is 413 2024-09-16T05:25:12.629 INFO:tasks.rados.rados.2.smithi204.stdout:3887: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-09-16T05:25:12.634 INFO:tasks.rados.rados.2.smithi204.stdout:3887: writing smithi20457400-45 from 672643 to 1178025 tid 1 2024-09-16T05:25:12.641 INFO:tasks.rados.rados.2.smithi204.stdout:3887: writing smithi20457400-45 from 1890209 to 2391005 tid 2 2024-09-16T05:25:12.641 INFO:tasks.rados.rados.2.smithi204.stdout:3888: write oid 2 current snap is 413 2024-09-16T05:25:12.641 INFO:tasks.rados.rados.2.smithi204.stdout:3888: seq_num 1350 ranges {689450=580631,1948691=740759} 2024-09-16T05:25:12.648 INFO:tasks.rados.rados.2.smithi204.stdout:3888: writing smithi20457400-2 from 689450 to 1270081 tid 1 2024-09-16T05:25:12.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:12.656 INFO:tasks.rados.rados.2.smithi204.stdout:3888: writing smithi20457400-2 from 1948691 to 2689450 tid 2 2024-09-16T05:25:12.656 INFO:tasks.rados.rados.2.smithi204.stdout:3889: snap_create 2024-09-16T05:25:12.656 INFO:tasks.rados.rados.2.smithi204.stdout:3887: finishing write tid 1 to smithi20457400-45 2024-09-16T05:25:12.656 INFO:tasks.rados.rados.2.smithi204.stdout:3887: finishing write tid 2 to smithi20457400-45 2024-09-16T05:25:12.657 INFO:tasks.rados.rados.2.smithi204.stdout:3887: finishing write tid 3 to smithi20457400-45 2024-09-16T05:25:12.657 INFO:tasks.rados.rados.2.smithi204.stdout:3887: finishing write tid 4 to smithi20457400-45 2024-09-16T05:25:12.657 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1282 (ObjNum 1349 snap 413 seq_num 1349) dirty exists 2024-09-16T05:25:12.657 INFO:tasks.rados.rados.2.smithi204.stdout:3887: left oid 45 (ObjNum 1349 snap 413 seq_num 1349) 2024-09-16T05:25:12.657 INFO:tasks.rados.rados.2.smithi204.stdout:3888: finishing write tid 1 to smithi20457400-2 2024-09-16T05:25:12.659 INFO:tasks.rados.rados.2.smithi204.stdout:3888: finishing write tid 2 to smithi20457400-2 2024-09-16T05:25:12.659 INFO:tasks.rados.rados.2.smithi204.stdout:3888: finishing write tid 3 to smithi20457400-2 2024-09-16T05:25:12.659 INFO:tasks.rados.rados.2.smithi204.stdout:3888: finishing write tid 4 to smithi20457400-2 2024-09-16T05:25:12.659 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 2 v 1276 (ObjNum 1350 snap 413 seq_num 1350) dirty exists 2024-09-16T05:25:12.659 INFO:tasks.rados.rados.2.smithi204.stdout:3888: left oid 2 (ObjNum 1350 snap 413 seq_num 1350) 2024-09-16T05:25:12.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:12.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:12.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:12 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:12.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2701499562' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:12.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:12 smithi196 ceph-mon[30802]: osdmap e960: 12 total, 11 up, 12 in 2024-09-16T05:25:12.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/472527799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:12.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:12 smithi196 ceph-mon[30802]: 1.1a scrub starts 2024-09-16T05:25:12.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:12 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:25:12.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:12 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:12.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2701499562' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:12.760 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:12 smithi204 ceph-mon[30933]: osdmap e960: 12 total, 11 up, 12 in 2024-09-16T05:25:12.760 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/472527799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:12.760 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:12 smithi204 ceph-mon[30933]: 1.1a scrub starts 2024-09-16T05:25:12.760 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:12 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:25:12.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:12.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:25:12.803 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:12.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:12.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:12.932 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:12.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:13.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:13.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:13.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:13.066 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:13.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:13.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:13.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:13.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:12 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:12.843+0000 7f6af1ad6640 -1 osd.6 961 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:13.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:13.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:13.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:13.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:13.433 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:25:13.433 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:25:13.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:13.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:13.452 INFO:tasks.rados.rados.2.smithi204.stdout:3887: done (11 left) 2024-09-16T05:25:13.452 INFO:tasks.rados.rados.2.smithi204.stdout:3888: done (10 left) 2024-09-16T05:25:13.452 INFO:tasks.rados.rados.2.smithi204.stdout:3889: done (9 left) 2024-09-16T05:25:13.453 INFO:tasks.rados.rados.2.smithi204.stdout:3890: snap_create 2024-09-16T05:25:13.461 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:13.462 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:13.484 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:13.485 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:13.507 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:13.507 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:13.530 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:13.530 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:13.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:13.563 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:13.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:13.572 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:25:13.572 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:25:13.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:25:13.573 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3597 2.4c 2.118008cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 526.82380702199998, 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98230618599999997, 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.574 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3597 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746119+0000", 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9999999999999995e-07 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746124+0000", 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3349999999999997e-06 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746136+0000", 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2339e-05 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746143+0000", 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3940000000000001e-06 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746201+0000", 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8529000000000002e-05 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:25:13.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746206+0000", 2024-09-16T05:25:13.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0250000000000002e-06 2024-09-16T05:25:13.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728413+0000", 2024-09-16T05:25:13.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98220633700000004 2024-09-16T05:25:13.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728424+0000", 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1226999999999999e-05 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.580 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3598 2.4c 2.118008cc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 526.82376545800003, 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.982592205, 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3598 2024-09-16T05:25:13.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.582 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:25:13.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:25:13.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746160+0000", 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6899999999999997e-07 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746162+0000", 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.323e-06 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746170+0000", 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4339999999999992e-06 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746176+0000", 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746227+0000", 2024-09-16T05:25:13.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1848999999999998e-05 2024-09-16T05:25:13.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:25:13.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746232+0000", 2024-09-16T05:25:13.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5619999999999997e-06 2024-09-16T05:25:13.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728741+0000", 2024-09-16T05:25:13.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98250897100000001 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728751+0000", 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.10.0:319 2.4a e458/325 ondisk, result = 0)", 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 504.75345292999998, 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.733792216, 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:25:13.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:25:13.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816473+0000", 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.206e-06 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816480+0000", 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4789999999999999e-06 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550063+0000", 2024-09-16T05:25:13.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73358232899999998 2024-09-16T05:25:13.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550069+0000", 2024-09-16T05:25:13.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3489999999999997e-06 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550155+0000", 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5490999999999999e-05 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550163+0000", 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1859999999999992e-06 2024-09-16T05:25:13.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550264+0000", 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010117599999999999 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.595 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3966 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 645534~783707] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:25:13.595 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:25:13.595 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 495.680926167, 2024-09-16T05:25:13.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.228671191, 2024-09-16T05:25:13.595 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.595 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.595 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.595 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3966 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889000+0000", 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.745e-06 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889398+0000", 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000397644 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889415+0000", 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7368e-05 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889440+0000", 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4358999999999998e-05 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889496+0000", 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6168999999999997e-05 2024-09-16T05:25:13.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889500+0000", 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0579999999999999e-06 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107487+0000", 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21798717200000001 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107565+0000", 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7715999999999999e-05 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107734+0000", 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016950099999999999 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111344+0000", 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036100160000000002 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117576+0000", 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062321929999999996 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117628+0000", 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.189e-05 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117670+0000", 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1359999999999997e-05 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3967 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 2103939~638517] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 495.674001137, 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.221880365, 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.602 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3967 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895924+0000", 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.85e-07 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896200+0000", 2024-09-16T05:25:13.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027563199999999999 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896212+0000", 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1671999999999999e-05 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896217+0000", 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3970000000000002e-06 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896274+0000", 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6740000000000002e-05 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896280+0000", 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109156+0000", 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.212875278 2024-09-16T05:25:13.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109242+0000", 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6163999999999997e-05 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109378+0000", 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013637400000000001 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111350+0000", 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019716429999999999 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117716+0000", 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063661830000000001 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117776+0000", 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9753999999999997e-05 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117804+0000", 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.817e-05 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3970 2.1d 2:ba7e58cf:::smithi20457400-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:25:13.609 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 495.67007349199997, 2024-09-16T05:25:13.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218352989, 2024-09-16T05:25:13.610 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.610 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.610 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:13.610 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.610 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.610 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.611 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3970 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899852+0000", 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8299999999999997e-07 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899854+0000", 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.463e-06 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899861+0000", 2024-09-16T05:25:13.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0079999999999996e-06 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899866+0000", 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899949+0000", 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2713000000000006e-05 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899954+0000", 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7400000000000004e-06 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111261+0000", 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.211307514 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111332+0000", 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1198999999999994e-05 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118056+0000", 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067238530000000001 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118121+0000", 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4770000000000005e-05 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118204+0000", 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3051000000000002e-05 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:4971 2.1d 2:ba7e58cf:::smithi20457400-26:head [stat,delete] snapc cc=[cb,ca,c9,c7,c5,c4,c3,c2,c1] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 430.371492053, 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.70425506000000004, 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.617 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4971 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198434+0000", 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8300000000000001e-06 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198443+0000", 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-16T05:25:13.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198460+0000", 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7229999999999999e-05 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198467+0000", 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1409999999999999e-06 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198531+0000", 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3371e-05 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198539+0000", 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8930000000000005e-06 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891869+0000", 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69333044700000002 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891964+0000", 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4990999999999999e-05 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892092+0000", 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012792599999999999 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892842+0000", 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075013 2024-09-16T05:25:13.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902619+0000", 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0097773840000000001 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902658+0000", 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8167999999999998e-05 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902688+0000", 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9991000000000001e-05 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5379 2.1c 2:3a3554ba:::smithi20457400-48:head [setxattr _header (58) in=7b,truncate 2823741] snapc e0=[e0,df,dd,dc,db,da,d9,d8,d7,d4,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:25:13.624 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 392.77515788800002, 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46119776499999998, 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5379 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:25:13.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9500000000000002e-07 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794770+0000", 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9890000000000001e-06 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794783+0000", 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2281e-05 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794788+0000", 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.011e-06 2024-09-16T05:25:13.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795397+0000", 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060981000000000002 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795462+0000", 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4869000000000005e-05 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795576+0000", 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001141 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795836+0000", 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026010999999999997 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255870+0000", 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46003327399999999 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255915+0000", 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5064000000000001e-05 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255964+0000", 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9662e-05 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.630 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5380 2.1c 2:3a3554ba:::smithi20457400-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 392.775123334, 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46133110599999999, 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5380 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794802+0000", 2024-09-16T05:25:13.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7599999999999998e-07 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794804+0000", 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0590000000000001e-06 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794810+0000", 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794815+0000", 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6700000000000002e-06 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795747+0000", 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093155799999999997 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795821+0000", 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3975999999999998e-05 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256004+0000", 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46018367199999999 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256065+0000", 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0299e-05 2024-09-16T05:25:13.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256132+0000", 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7776999999999998e-05 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.636 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:5379 2.1c e571/532 ondisk, result = 0)", 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 392.773089648, 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45914334200000001, 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0800000000000001e-07 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796837+0000", 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9850000000000001e-06 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255760+0000", 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45892265399999999 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255769+0000", 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3449999999999995e-06 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255857+0000", 2024-09-16T05:25:13.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7547999999999996e-05 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255865+0000", 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6810000000000004e-06 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255978+0000", 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112821 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5901 2.1d 2:ba754fcb:::smithi20457400-30:head [stat,delete] snapc f3=[f3,f2,f0,ef,ee,ed,ec,e8,e7,e5,e0,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 355.66661741299998, 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79482897200000002, 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5901 2024-09-16T05:25:13.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903309+0000", 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4699999999999999e-06 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903314+0000", 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903330+0000", 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5770999999999999e-05 2024-09-16T05:25:13.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903340+0000", 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0498e-05 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903402+0000", 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1385999999999998e-05 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903513+0000", 2024-09-16T05:25:13.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111113 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903707+0000", 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019455500000000001 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.904477+0000", 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076932900000000002 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698051+0000", 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79357449800000002 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698095+0000", 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3452999999999999e-05 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698136+0000", 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1482e-05 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.646 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.7.0:1084 2.3c e626/625 2:3d4bedc8:::smithi20457400-22:ee v 626'652, mlcod=624'646)", 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 336.09566390499998, 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45487590700000002, 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8800000000000002e-07 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474267+0000", 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928275+0000", 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45400790800000002 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928291+0000", 2024-09-16T05:25:13.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5852999999999999e-05 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928455+0000", 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164026 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928479+0000", 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4258000000000001e-05 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929099+0000", 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000620117 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929137+0000", 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7639999999999999e-05 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6377 2.1d 2:ba21f1a9:::smithi20457400-14:head [setxattr _header (58) in=7b,truncate 3059072] snapc 107=[107,106,104,102,101,fb,fa,f9,f7,f2,e7] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:25:13.651 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 313.68288857900001, 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46418828400000001, 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6377 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887037+0000", 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887040+0000", 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5039999999999998e-06 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887047+0000", 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0330000000000003e-06 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887053+0000", 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1430000000000003e-06 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888098+0000", 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001044866 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888199+0000", 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101143 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888364+0000", 2024-09-16T05:25:13.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164403 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888873+0000", 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050974699999999996 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351123+0000", 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46224975499999998 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351169+0000", 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6316000000000002e-05 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351224+0000", 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4837999999999999e-05 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.657 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6378 2.1d 2:ba21f1a9:::smithi20457400-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 313.682861651, 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46436558900000002, 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6378 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887064+0000", 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3799999999999997e-07 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887071+0000", 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9670000000000001e-06 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887077+0000", 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9739999999999999e-06 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887081+0000", 2024-09-16T05:25:13.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.405e-06 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888760+0000", 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016788319999999999 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888855+0000", 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5670999999999999e-05 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351265+0000", 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46241011799999998 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351353+0000", 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7416000000000005e-05 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351429+0000", 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5667999999999994e-05 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.662 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(1.7,GRANT,e:689,from:6,reservation_nonce:0)", 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 274.11261281399999, 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388634586, 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457313+0000", 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0020000000000001e-06 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457317+0000", 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.279e-06 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845738+0000", 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388420879 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845745+0000", 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1849999999999998e-06 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845793+0000", 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8300000000000002e-05 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845814+0000", 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0353999999999999e-05 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845946+0000", 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013258700000000001 2024-09-16T05:25:13.666 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.6.49:26 2.5d 2.d3f2ae5d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e867)", 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 96.697088262999998, 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99584287000000005, 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872838+0000", 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-06 2024-09-16T05:25:13.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872846+0000", 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0790000000000001e-06 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872866+0000", 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9624999999999999e-05 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872875+0000", 2024-09-16T05:25:13.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4790000000000002e-06 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872926+0000", 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0260000000000003e-05 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872935+0000", 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.346e-06 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:37.868662+0000", 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99572677300000001 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:37.868679+0000", 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7554000000000002e-05 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8820 2.5e 2:7b642727:::smithi20457400-40:head [copy-from ver 549] snapc 175=[16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-09-16T05:25:13.671 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 86.710161576000004, 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025559208, 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8820 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859765+0000", 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1409999999999999e-06 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859774+0000", 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6430000000000001e-06 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859789+0000", 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5441999999999999e-05 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859798+0000", 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9239999999999996e-06 2024-09-16T05:25:13.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859875+0000", 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6508000000000007e-05 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859883+0000", 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2190000000000004e-06 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.862172+0000", 2024-09-16T05:25:13.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022892210000000001 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.862279+0000", 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106606 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873916+0000", 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011637412999999999 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.881734+0000", 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078177669999999998 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885232+0000", 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034977849999999998 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885271+0000", 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9079000000000003e-05 2024-09-16T05:25:13.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885322+0000", 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1459999999999999e-05 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.678 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9415 2.4c 2:33100188:::smithi20457400-19:head [append 0~0] snapc 194=[194,191,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e937)", 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:47.301227+0000", 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 26.268698117, 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046627700000000001, 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9415 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301227+0000", 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301227+0000", 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301227+0000", 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8599999999999998e-07 2024-09-16T05:25:13.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301230+0000", 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7520000000000001e-06 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301236+0000", 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4799999999999998e-06 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301242+0000", 2024-09-16T05:25:13.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3430000000000003e-06 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301402+0000", 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016081900000000001 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301408+0000", 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9769999999999997e-06 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.304617+0000", 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032087980000000001 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.304701+0000", 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3588e-05 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8", 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.304861+0000", 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016004499999999999 2024-09-16T05:25:13.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.305239+0000", 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037855799999999998 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.305797+0000", 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00055756399999999998 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.305846+0000", 2024-09-16T05:25:13.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8949999999999997e-05 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.305889+0000", 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3309999999999997e-05 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:9619 2.31 e945/529 2:8c1ba486:::smithi20457400-2:head v 945'424, mlcod=941'422)", 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:56.594305+0000", 2024-09-16T05:25:13.685 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 16.975619988999998, 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0052787659999999998, 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.594305+0000", 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.594305+0000", 2024-09-16T05:25:13.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.594305+0000", 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9400000000000004e-07 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597567+0000", 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032611010000000002 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597607+0000", 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0361000000000002e-05 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597616+0000", 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2199999999999998e-06 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597685+0000", 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9208000000000006e-05 2024-09-16T05:25:13.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597709+0000", 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4142999999999999e-05 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.599544+0000", 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001834565 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.599583+0000", 2024-09-16T05:25:13.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9373999999999999e-05 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(1.19,GRANT,e:954,from:0,reservation_nonce:0)", 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:25:05.893852+0000", 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 7.6760726589999999, 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021196799999999999, 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:25:13.690 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.893852+0000", 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.893852+0000", 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.893853+0000", 2024-09-16T05:25:13.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.17e-06 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.893859+0000", 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1e-06 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.893885+0000", 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5709999999999999e-05 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:25:13.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.893895+0000", 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0199e-05 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.893939+0000", 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4267999999999997e-05 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.893953+0000", 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3858e-05 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:25:13.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:05.894064+0000", 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011066299999999999 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:25:13.694 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:25:13.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:13.717 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:13.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:13.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:13 smithi196 ceph-mon[30802]: pgmap v1603: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 197 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.5 MiB/s wr, 15 op/s; 9/180 objects degraded (5.000%); 47 KiB/s, 0 objects/s recovering 2024-09-16T05:25:13.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:13 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:25:13.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:13 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:25:13.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:13 smithi196 ceph-mon[30802]: osdmap e961: 12 total, 11 up, 12 in 2024-09-16T05:25:13.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:13.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:13.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:13.854 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:13 smithi032 ceph-mon[27388]: pgmap v1603: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 197 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.5 MiB/s wr, 15 op/s; 9/180 objects degraded (5.000%); 47 KiB/s, 0 objects/s recovering 2024-09-16T05:25:13.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:13 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:25:13.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:13 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:25:13.855 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:13 smithi032 ceph-mon[27388]: osdmap e961: 12 total, 11 up, 12 in 2024-09-16T05:25:13.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:13 smithi204 ceph-mon[30933]: pgmap v1603: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 197 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.5 MiB/s wr, 15 op/s; 9/180 objects degraded (5.000%); 47 KiB/s, 0 objects/s recovering 2024-09-16T05:25:13.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:13 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:25:13.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:13 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:25:13.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:13 smithi204 ceph-mon[30933]: osdmap e961: 12 total, 11 up, 12 in 2024-09-16T05:25:13.943 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:13.965 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:13.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:13.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:14.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:14.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:14.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:14.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:14.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:25:14.217 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:14.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:14.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:13 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:13.798+0000 7f6af1ad6640 -1 osd.6 961 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:14.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:14.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:14.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:14.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:14.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:14.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:25:14.454 INFO:tasks.rados.rados.2.smithi204.stdout:3890: done (9 left) 2024-09-16T05:25:14.455 INFO:tasks.rados.rados.2.smithi204.stdout:3891: rollback oid 20 current snap is 415 2024-09-16T05:25:14.455 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 20 to 414 2024-09-16T05:25:14.455 INFO:tasks.rados.rados.2.smithi204.stdout:3892: read oid 27 snap -1 2024-09-16T05:25:14.455 INFO:tasks.rados.rados.2.smithi204.stdout:3892: expect (ObjNum 1344 snap 411 seq_num 1344) 2024-09-16T05:25:14.455 INFO:tasks.rados.rados.2.smithi204.stdout:3893: snap_remove snap 411 2024-09-16T05:25:14.472 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:14.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:14.476 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:14.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:14.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:14 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:14.598 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:14 smithi032 ceph-mon[27388]: osdmap e962: 12 total, 11 up, 12 in 2024-09-16T05:25:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:14.599 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:14.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:14.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:14 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:14.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:14 smithi204 ceph-mon[30933]: osdmap e962: 12 total, 11 up, 12 in 2024-09-16T05:25:14.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:14.701 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:14.723 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:14.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:14.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:14 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:14.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:14 smithi196 ceph-mon[30802]: osdmap e962: 12 total, 11 up, 12 in 2024-09-16T05:25:14.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:14.846 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:14.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:14.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:14.969 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:14.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:15.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:15.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:15.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:15.171 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:25:15.171 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:25:15.171 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:25:15.171 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:25:15.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:15.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:15.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:15.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:14 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:14.839+0000 7f6af1ad6640 -1 osd.6 963 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:15.243 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [1] live_osds: [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:25:15.243 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:25:15.243 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-09-16T05:25:15.243 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-16T05:25:15.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.7 2024-09-16T05:25:15.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:15.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:15.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:15.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:15.459 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:25:15.473 INFO:tasks.rados.rados.2.smithi204.stdout:3893: done (11 left) 2024-09-16T05:25:15.473 INFO:tasks.rados.rados.2.smithi204.stdout:3894: rollback oid 8 current snap is 415 2024-09-16T05:25:15.473 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 8 to 358 2024-09-16T05:25:15.473 INFO:tasks.rados.rados.2.smithi204.stdout:3891: finishing rollback tid 0 to smithi20457400-20 2024-09-16T05:25:15.473 INFO:tasks.rados.rados.2.smithi204.stdout:3891: finishing rollback tid 1 to smithi20457400-20 2024-09-16T05:25:15.473 INFO:tasks.rados.rados.2.smithi204.stdout:3891: finishing rollback tid 2 to smithi20457400-20 2024-09-16T05:25:15.473 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1438 (ObjNum 1324 snap 403 seq_num 1324) dirty exists 2024-09-16T05:25:15.473 INFO:tasks.rados.rados.2.smithi204.stdout:3892: expect (ObjNum 1344 snap 411 seq_num 1344) 2024-09-16T05:25:15.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:15.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:15.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:15.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-mon[30802]: pgmap v1606: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 196 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.8 MiB/s wr, 24 op/s; 9/174 objects degraded (5.172%); 47 KiB/s, 0 objects/s recovering 2024-09-16T05:25:15.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 90 sec, osd.6 has slow ops) 2024-09-16T05:25:15.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-mon[30802]: osdmap e963: 12 total, 11 up, 12 in 2024-09-16T05:25:15.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/413595694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:15.567 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:15 smithi196 systemd[1]: Stopping Ceph osd.7 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:25:15.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:15.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:15.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:15.639 INFO:tasks.rados.rados.2.smithi204.stdout:3894: finishing rollback tid 0 to smithi20457400-8 2024-09-16T05:25:15.639 INFO:tasks.rados.rados.2.smithi204.stdout:3894: finishing rollback tid 1 to smithi20457400-8 2024-09-16T05:25:15.639 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 8 v 1474 (ObjNum 49 snap 0 seq_num 4282336032) dirty dne 2024-09-16T05:25:15.639 INFO:tasks.rados.rados.2.smithi204.stdout:3891: done (11 left) 2024-09-16T05:25:15.639 INFO:tasks.rados.rados.2.smithi204.stdout:3892: done (10 left) 2024-09-16T05:25:15.639 INFO:tasks.rados.rados.2.smithi204.stdout:3894: done (9 left) 2024-09-16T05:25:15.639 INFO:tasks.rados.rados.2.smithi204.stdout:3895: rollback oid 9 current snap is 415 2024-09-16T05:25:15.640 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 9 to 412 2024-09-16T05:25:15.640 INFO:tasks.rados.rados.2.smithi204.stdout:3896: delete oid 28 current snap is 415 2024-09-16T05:25:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:3896: done (10 left) 2024-09-16T05:25:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:3897: copy_from oid 27 from oid 5 current snap is 415 2024-09-16T05:25:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:3895: finishing rollback tid 1 to smithi20457400-9 2024-09-16T05:25:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 353 snap 0 seq_num 4287854720) dirty dne 2024-09-16T05:25:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:3895: done (10 left) 2024-09-16T05:25:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:3898: write oid 23 current snap is 415 2024-09-16T05:25:15.642 INFO:tasks.rados.rados.2.smithi204.stdout:3898: seq_num 1352 ranges {723064=731129,1949302=764476} 2024-09-16T05:25:15.650 INFO:tasks.rados.rados.2.smithi204.stdout:3898: writing smithi20457400-23 from 723064 to 1454193 tid 1 2024-09-16T05:25:15.657 INFO:tasks.rados.rados.2.smithi204.stdout:3898: writing smithi20457400-23 from 1949302 to 2713778 tid 2 2024-09-16T05:25:15.657 INFO:tasks.rados.rados.2.smithi204.stdout:3897: finishing copy_from racing read to smithi20457400-27 2024-09-16T05:25:15.657 INFO:tasks.rados.rados.2.smithi204.stdout:3898: finishing write tid 1 to smithi20457400-23 2024-09-16T05:25:15.658 INFO:tasks.rados.rados.2.smithi204.stdout:3899: write oid 45 current snap is 415 2024-09-16T05:25:15.658 INFO:tasks.rados.rados.2.smithi204.stdout:3899: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-09-16T05:25:15.664 INFO:tasks.rados.rados.2.smithi204.stdout:3899: writing smithi20457400-45 from 739871 to 1146249 tid 1 2024-09-16T05:25:15.673 INFO:tasks.rados.rados.2.smithi204.stdout:3899: writing smithi20457400-45 from 1607784 to 2115918 tid 2 2024-09-16T05:25:15.673 INFO:tasks.rados.rados.2.smithi204.stdout:3900: rollback oid 3 current snap is 415 2024-09-16T05:25:15.673 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 362 2024-09-16T05:25:15.673 INFO:tasks.rados.rados.2.smithi204.stdout:3898: finishing write tid 2 to smithi20457400-23 2024-09-16T05:25:15.673 INFO:tasks.rados.rados.2.smithi204.stdout:3898: finishing write tid 3 to smithi20457400-23 2024-09-16T05:25:15.673 INFO:tasks.rados.rados.2.smithi204.stdout:3898: finishing write tid 4 to smithi20457400-23 2024-09-16T05:25:15.673 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 878 (ObjNum 1352 snap 415 seq_num 1352) dirty exists 2024-09-16T05:25:15.673 INFO:tasks.rados.rados.2.smithi204.stdout:3898: left oid 23 (ObjNum 1352 snap 415 seq_num 1352) 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3897: finishing copy_from to smithi20457400-27 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 27 v 1487 (ObjNum 1325 snap 403 seq_num 1325) dirty exists 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3897: done (12 left) 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3898: done (11 left) 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3901: delete oid 19 current snap is 415 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3900: finishing rollback tid 0 to smithi20457400-3 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3899: finishing write tid 1 to smithi20457400-45 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3900: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3899: finishing write tid 2 to smithi20457400-45 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3900: finishing rollback tid 2 to smithi20457400-3 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1114 (ObjNum 1140 snap 341 seq_num 1140) dirty exists 2024-09-16T05:25:15.674 INFO:tasks.rados.rados.2.smithi204.stdout:3899: finishing write tid 3 to smithi20457400-45 2024-09-16T05:25:15.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:15.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:15.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:15 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:15.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:15 smithi204 ceph-mon[30933]: pgmap v1606: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 196 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.8 MiB/s wr, 24 op/s; 9/174 objects degraded (5.172%); 47 KiB/s, 0 objects/s recovering 2024-09-16T05:25:15.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:15 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 90 sec, osd.6 has slow ops) 2024-09-16T05:25:15.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:15 smithi204 ceph-mon[30933]: osdmap e963: 12 total, 11 up, 12 in 2024-09-16T05:25:15.768 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/413595694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:15.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:25:15.801 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:15 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:15.801 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:15 smithi032 ceph-mon[27388]: pgmap v1606: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 196 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 1.8 MiB/s wr, 24 op/s; 9/174 objects degraded (5.172%); 47 KiB/s, 0 objects/s recovering 2024-09-16T05:25:15.801 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:15 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 90 sec, osd.6 has slow ops) 2024-09-16T05:25:15.801 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:15 smithi032 ceph-mon[27388]: osdmap e963: 12 total, 11 up, 12 in 2024-09-16T05:25:15.801 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/413595694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:15.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:15.834 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[52338]: 2024-09-16T05:25:15.565+0000 7fc269e1c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:25:15.834 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[52338]: 2024-09-16T05:25:15.565+0000 7fc269e1c640 -1 osd.7 964 *** Got signal Terminated *** 2024-09-16T05:25:15.835 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[52338]: 2024-09-16T05:25:15.565+0000 7fc269e1c640 -1 osd.7 964 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:25:15.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:15.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:15.952 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:15.975 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:15.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:16.009 INFO:tasks.rados.rados.2.smithi204.stdout:3900: done (11 left) 2024-09-16T05:25:16.010 INFO:tasks.rados.rados.2.smithi204.stdout:3901: done (10 left) 2024-09-16T05:25:16.010 INFO:tasks.rados.rados.2.smithi204.stdout:3902: read oid 33 snap -1 2024-09-16T05:25:16.010 INFO:tasks.rados.rados.2.smithi204.stdout:3902: expect deleted 2024-09-16T05:25:16.010 INFO:tasks.rados.rados.2.smithi204.stdout:3903: write oid 25 current snap is 415 2024-09-16T05:25:16.010 INFO:tasks.rados.rados.2.smithi204.stdout:3903: seq_num 1355 ranges {773485=673229,2124748=511803} 2024-09-16T05:25:16.017 INFO:tasks.rados.rados.2.smithi204.stdout:3903: writing smithi20457400-25 from 773485 to 1446714 tid 1 2024-09-16T05:25:16.024 INFO:tasks.rados.rados.2.smithi204.stdout:3903: writing smithi20457400-25 from 2124748 to 2636551 tid 2 2024-09-16T05:25:16.024 INFO:tasks.rados.rados.2.smithi204.stdout:3904: write (excl) oid 12 current snap is 415 2024-09-16T05:25:16.024 INFO:tasks.rados.rados.2.smithi204.stdout:3904: seq_num 1356 ranges {790292=748478,2183230=607062} 2024-09-16T05:25:16.034 INFO:tasks.rados.rados.2.smithi204.stdout:3904: writing smithi20457400-12 from 790292 to 1538770 tid 1 2024-09-16T05:25:16.038 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:16.042 INFO:tasks.rados.rados.2.smithi204.stdout:3904: writing smithi20457400-12 from 2183230 to 2790292 tid 2 2024-09-16T05:25:16.042 INFO:tasks.rados.rados.2.smithi204.stdout:3905: snap_create 2024-09-16T05:25:16.042 INFO:tasks.rados.rados.2.smithi204.stdout:3899: finishing write tid 4 to smithi20457400-45 2024-09-16T05:25:16.042 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1285 (ObjNum 1353 snap 415 seq_num 1353) dirty exists 2024-09-16T05:25:16.042 INFO:tasks.rados.rados.2.smithi204.stdout:3899: left oid 45 (ObjNum 1353 snap 415 seq_num 1353) 2024-09-16T05:25:16.042 INFO:tasks.rados.rados.2.smithi204.stdout:3903: finishing write tid 1 to smithi20457400-25 2024-09-16T05:25:16.042 INFO:tasks.rados.rados.2.smithi204.stdout:3903: finishing write tid 2 to smithi20457400-25 2024-09-16T05:25:16.043 INFO:tasks.rados.rados.2.smithi204.stdout:3903: finishing write tid 3 to smithi20457400-25 2024-09-16T05:25:16.043 INFO:tasks.rados.rados.2.smithi204.stdout:3903: finishing write tid 4 to smithi20457400-25 2024-09-16T05:25:16.043 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1291 (ObjNum 1355 snap 415 seq_num 1355) dirty exists 2024-09-16T05:25:16.043 INFO:tasks.rados.rados.2.smithi204.stdout:3903: left oid 25 (ObjNum 1355 snap 415 seq_num 1355) 2024-09-16T05:25:16.043 INFO:tasks.rados.rados.2.smithi204.stdout:3904: finishing write tid 1 to smithi20457400-12 2024-09-16T05:25:16.044 INFO:tasks.rados.rados.2.smithi204.stdout:3904: finishing write tid 2 to smithi20457400-12 2024-09-16T05:25:16.045 INFO:tasks.rados.rados.2.smithi204.stdout:3904: finishing write tid 3 to smithi20457400-12 2024-09-16T05:25:16.045 INFO:tasks.rados.rados.2.smithi204.stdout:3904: finishing write tid 4 to smithi20457400-12 2024-09-16T05:25:16.045 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1363 (ObjNum 1356 snap 415 seq_num 1356) dirty exists 2024-09-16T05:25:16.045 INFO:tasks.rados.rados.2.smithi204.stdout:3904: left oid 12 (ObjNum 1356 snap 415 seq_num 1356) 2024-09-16T05:25:16.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:16.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:16.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:16.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:16.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:16.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:16.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:15 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:15.834+0000 7f6af1ad6640 -1 osd.6 964 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:16.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:16.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:25:16.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:16.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:16.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:16.469 INFO:tasks.rados.rados.2.smithi204.stdout:3899: done (13 left) 2024-09-16T05:25:16.470 INFO:tasks.rados.rados.2.smithi204.stdout:3902: done (12 left) 2024-09-16T05:25:16.470 INFO:tasks.rados.rados.2.smithi204.stdout:3903: done (11 left) 2024-09-16T05:25:16.470 INFO:tasks.rados.rados.2.smithi204.stdout:3904: done (10 left) 2024-09-16T05:25:16.470 INFO:tasks.rados.rados.2.smithi204.stdout:3905: done (9 left) 2024-09-16T05:25:16.470 INFO:tasks.rados.rados.2.smithi204.stdout:3906: delete oid 44 current snap is 416 2024-09-16T05:25:16.476 INFO:tasks.rados.rados.2.smithi204.stdout:3906: done (9 left) 2024-09-16T05:25:16.476 INFO:tasks.rados.rados.2.smithi204.stdout:3907: write oid 47 current snap is 416 2024-09-16T05:25:16.476 INFO:tasks.rados.rados.2.smithi204.stdout:3907: seq_num 1358 ranges {423906=498976,1500194=659130,2609483=214423} 2024-09-16T05:25:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:16.477 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:16.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:16.481 INFO:tasks.rados.rados.2.smithi204.stdout:3907: writing smithi20457400-47 from 423906 to 922882 tid 1 2024-09-16T05:25:16.488 INFO:tasks.rados.rados.2.smithi204.stdout:3907: writing smithi20457400-47 from 1500194 to 2159324 tid 2 2024-09-16T05:25:16.492 INFO:tasks.rados.rados.2.smithi204.stdout:3907: writing smithi20457400-47 from 2609483 to 2823906 tid 3 2024-09-16T05:25:16.492 INFO:tasks.rados.rados.2.smithi204.stdout:3908: read oid 15 snap 358 2024-09-16T05:25:16.492 INFO:tasks.rados.rados.2.smithi204.stdout:3908: expect deleted 2024-09-16T05:25:16.492 INFO:tasks.rados.rados.2.smithi204.stdout:3909: snap_create 2024-09-16T05:25:16.492 INFO:tasks.rados.rados.2.smithi204.stdout:3907: finishing write tid 1 to smithi20457400-47 2024-09-16T05:25:16.492 INFO:tasks.rados.rados.2.smithi204.stdout:3907: finishing write tid 2 to smithi20457400-47 2024-09-16T05:25:16.493 INFO:tasks.rados.rados.2.smithi204.stdout:3907: finishing write tid 3 to smithi20457400-47 2024-09-16T05:25:16.494 INFO:tasks.rados.rados.2.smithi204.stdout:3907: finishing write tid 4 to smithi20457400-47 2024-09-16T05:25:16.494 INFO:tasks.rados.rados.2.smithi204.stdout:3907: finishing write tid 5 to smithi20457400-47 2024-09-16T05:25:16.494 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1408 (ObjNum 1358 snap 416 seq_num 1358) dirty exists 2024-09-16T05:25:16.494 INFO:tasks.rados.rados.2.smithi204.stdout:3907: left oid 47 (ObjNum 1358 snap 416 seq_num 1358) 2024-09-16T05:25:16.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:16.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:16.608 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:16.637 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:25:16.637 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:25:16.638 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:25:16.638 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:25:16.638 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:25:16.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:16 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:16.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:16 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:25:16.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:16 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:25:16.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/413595694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:16.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:16 smithi204 ceph-mon[30933]: osdmap e964: 12 total, 11 up, 12 in 2024-09-16T05:25:16.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:16 smithi204 ceph-mon[30933]: osd.7 marked itself down and dead 2024-09-16T05:25:16.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:16 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 9/174 objects degraded (5.172%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:16.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:16.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:16.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:16.729 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:16 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:16.730 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:16 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:25:16.730 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:16 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:25:16.730 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/413595694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:16.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:16 smithi032 ceph-mon[27388]: osdmap e964: 12 total, 11 up, 12 in 2024-09-16T05:25:16.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:16 smithi032 ceph-mon[27388]: osd.7 marked itself down and dead 2024-09-16T05:25:16.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:16 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 9/174 objects degraded (5.172%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:16.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:16 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:16.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:16 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:25:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:16 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:25:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/413595694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:16 smithi196 ceph-mon[30802]: osdmap e964: 12 total, 11 up, 12 in 2024-09-16T05:25:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:16 smithi196 ceph-mon[30802]: osd.7 marked itself down and dead 2024-09-16T05:25:16.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:16 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 9/174 objects degraded (5.172%), 3 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:16.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:16.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:16.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:16.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:16.976 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:16.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:17.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:17.082 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:17.083 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:17.083 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:17.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:17.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:17.105 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:17.105 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:17.126 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:17.126 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:17.148 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:17.170 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:17.170 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:17.194 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:17.194 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:17.194 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:25:17.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:17.216 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:17.216 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:17.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:17.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:17.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:16 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:16.830+0000 7f6af1ad6640 -1 osd.6 965 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:17.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:17 smithi196 podman[114152]: 2024-09-16 05:25:17.003358488 +0000 UTC m=+1.571554666 container died da4ca952f32a13b2bb7c0f87e7ccde78e807c559a4d2bc8ce70a11661ac9e837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-16T05:25:17.238 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:17.238 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:17.259 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:17.281 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:17.281 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:17.323 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:25:17.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:17.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:17.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:17.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:17.471 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:17.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:17.473 INFO:tasks.rados.rados.2.smithi204.stdout:3907: done (11 left) 2024-09-16T05:25:17.473 INFO:tasks.rados.rados.2.smithi204.stdout:3908: done (10 left) 2024-09-16T05:25:17.473 INFO:tasks.rados.rados.2.smithi204.stdout:3909: done (9 left) 2024-09-16T05:25:17.473 INFO:tasks.rados.rados.2.smithi204.stdout:3910: rollback oid 3 current snap is 417 2024-09-16T05:25:17.473 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 3 to 412 2024-09-16T05:25:17.473 INFO:tasks.rados.rados.2.smithi204.stdout:3911: read oid 1 snap -1 2024-09-16T05:25:17.473 INFO:tasks.rados.rados.2.smithi204.stdout:3911: expect (ObjNum 1347 snap 413 seq_num 1347) 2024-09-16T05:25:17.473 INFO:tasks.rados.rados.2.smithi204.stdout:3912: snap_remove snap 413 2024-09-16T05:25:17.568 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:17.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:17.594 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:17.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:17.617 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:17 smithi196 podman[114152]: 2024-09-16 05:25:17.42450266 +0000 UTC m=+1.992698884 container cleanup da4ca952f32a13b2bb7c0f87e7ccde78e807c559a4d2bc8ce70a11661ac9e837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-16T05:25:17.617 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:17 smithi196 bash[114152]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7 2024-09-16T05:25:17.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:17 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:17.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:17 smithi196 ceph-mon[30802]: pgmap v1609: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 9/182 objects degraded (4.945%) 2024-09-16T05:25:17.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:17 smithi196 ceph-mon[30802]: Health check update: 2 osds down (OSD_DOWN) 2024-09-16T05:25:17.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:17 smithi196 ceph-mon[30802]: osdmap e965: 12 total, 10 up, 12 in 2024-09-16T05:25:17.618 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2426848213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:17.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:17 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:17 smithi204 ceph-mon[30933]: pgmap v1609: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 9/182 objects degraded (4.945%) 2024-09-16T05:25:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:17 smithi204 ceph-mon[30933]: Health check update: 2 osds down (OSD_DOWN) 2024-09-16T05:25:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:17 smithi204 ceph-mon[30933]: osdmap e965: 12 total, 10 up, 12 in 2024-09-16T05:25:17.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2426848213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:17.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:17.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:17.739 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:17 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:17.739 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:17 smithi032 ceph-mon[27388]: pgmap v1609: 127 pgs: 1 active+clean+scrubbing, 18 undersized+peered, 3 undersized+degraded+peered, 105 active+clean; 201 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 9/182 objects degraded (4.945%) 2024-09-16T05:25:17.739 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:17 smithi032 ceph-mon[27388]: Health check update: 2 osds down (OSD_DOWN) 2024-09-16T05:25:17.739 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:17 smithi032 ceph-mon[27388]: osdmap e965: 12 total, 10 up, 12 in 2024-09-16T05:25:17.739 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2426848213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:17.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:17.842 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:17.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:17.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:17.968 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:17.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:17.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:17 smithi196 podman[114252]: 2024-09-16 05:25:17.617372828 +0000 UTC m=+0.610580109 container remove da4ca952f32a13b2bb7c0f87e7ccde78e807c559a4d2bc8ce70a11661ac9e837 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-16T05:25:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:18.064 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:18.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:18.091 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:18.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:18.179 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:18.179 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:18.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:18.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:18.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:18.226 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:25:18.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:18.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:18.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:18.370 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:18 smithi196 podman[114377]: 2024-09-16 05:25:17.937426318 +0000 UTC m=+0.015856453 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:25:18.370 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:18 smithi196 podman[114377]: 2024-09-16 05:25:18.034732415 +0000 UTC m=+0.113162541 container create 82e0161aea20891435d3231befc8a627784ed5f357af16c072743889c19ecb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-16T05:25:18.370 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:18 smithi196 podman[114377]: 2024-09-16 05:25:18.270483535 +0000 UTC m=+0.348913661 container init 82e0161aea20891435d3231befc8a627784ed5f357af16c072743889c19ecb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:25:18.370 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:18 smithi196 podman[114377]: 2024-09-16 05:25:18.273831353 +0000 UTC m=+0.352261484 container start 82e0161aea20891435d3231befc8a627784ed5f357af16c072743889c19ecb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-16T05:25:18.370 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:18 smithi196 podman[114377]: 2024-09-16 05:25:18.370317317 +0000 UTC m=+0.448747452 container attach 82e0161aea20891435d3231befc8a627784ed5f357af16c072743889c19ecb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:25:18.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:18.464 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:18.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:18.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:18.479 INFO:tasks.rados.rados.2.smithi204.stdout:3912: done (11 left) 2024-09-16T05:25:18.479 INFO:tasks.rados.rados.2.smithi204.stdout:3913: snap_create 2024-09-16T05:25:18.479 INFO:tasks.rados.rados.2.smithi204.stdout:3910: finishing rollback tid 0 to smithi20457400-3 2024-09-16T05:25:18.479 INFO:tasks.rados.rados.2.smithi204.stdout:3910: finishing rollback tid 1 to smithi20457400-3 2024-09-16T05:25:18.479 INFO:tasks.rados.rados.2.smithi204.stdout:3910: finishing rollback tid 2 to smithi20457400-3 2024-09-16T05:25:18.479 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 3 v 1117 (ObjNum 1140 snap 341 seq_num 1140) dirty exists 2024-09-16T05:25:18.480 INFO:tasks.rados.rados.2.smithi204.stdout:3911: expect (ObjNum 1347 snap 413 seq_num 1347) 2024-09-16T05:25:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:18.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:18.588 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:18.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:18.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:18.714 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:18.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:18.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:18 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:18.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:18 smithi196 ceph-mon[30802]: osdmap e966: 12 total, 10 up, 12 in 2024-09-16T05:25:18.738 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:25:18 smithi196 podman[114377]: 2024-09-16 05:25:18.470435113 +0000 UTC m=+0.548865250 container died 82e0161aea20891435d3231befc8a627784ed5f357af16c072743889c19ecb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-16T05:25:18.744 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:18 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:18.744 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:18 smithi032 ceph-mon[27388]: osdmap e966: 12 total, 10 up, 12 in 2024-09-16T05:25:18.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:18 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:18.756 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:18 smithi204 ceph-mon[30933]: osdmap e966: 12 total, 10 up, 12 in 2024-09-16T05:25:18.783 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:25:19.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:19.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.7.service' 2024-09-16T05:25:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:19.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:19.289 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:25:19.289 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-16T05:25:19.290 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-16T05:25:19.290 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi196.front.sepia.ceph.com 2024-09-16T05:25:19.290 INFO:tasks.thrashosds.thrasher:remote for exp osd 1 is ubuntu@smithi032.front.sepia.ceph.com 2024-09-16T05:25:19.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.1 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-16T05:25:19.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:19.336 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:19.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:19.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:19.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:19.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:19.480 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:25:19.483 INFO:tasks.rados.rados.2.smithi204.stdout:3910: done (11 left) 2024-09-16T05:25:19.483 INFO:tasks.rados.rados.2.smithi204.stdout:3911: done (10 left) 2024-09-16T05:25:19.484 INFO:tasks.rados.rados.2.smithi204.stdout:3913: done (9 left) 2024-09-16T05:25:19.484 INFO:tasks.rados.rados.2.smithi204.stdout:3914: write (excl) oid 35 current snap is 418 2024-09-16T05:25:19.484 INFO:tasks.rados.rados.2.smithi204.stdout:3914: seq_num 1359 ranges {440713=574225,1442323=519141,2636906=203807} 2024-09-16T05:25:19.489 INFO:tasks.rados.rados.2.smithi204.stdout:3914: writing smithi20457400-35 from 440713 to 1014938 tid 1 2024-09-16T05:25:19.494 INFO:tasks.rados.rados.2.smithi204.stdout:3914: writing smithi20457400-35 from 1442323 to 1961464 tid 2 2024-09-16T05:25:19.499 INFO:tasks.rados.rados.2.smithi204.stdout:3914: writing smithi20457400-35 from 2636906 to 2840713 tid 3 2024-09-16T05:25:19.499 INFO:tasks.rados.rados.2.smithi204.stdout:3915: delete oid 47 current snap is 418 2024-09-16T05:25:19.499 INFO:tasks.rados.rados.2.smithi204.stdout:3914: finishing write tid 1 to smithi20457400-35 2024-09-16T05:25:19.499 INFO:tasks.rados.rados.2.smithi204.stdout:3914: finishing write tid 2 to smithi20457400-35 2024-09-16T05:25:19.501 INFO:tasks.rados.rados.2.smithi204.stdout:3914: finishing write tid 3 to smithi20457400-35 2024-09-16T05:25:19.501 INFO:tasks.rados.rados.2.smithi204.stdout:3914: finishing write tid 4 to smithi20457400-35 2024-09-16T05:25:19.501 INFO:tasks.rados.rados.2.smithi204.stdout:3914: finishing write tid 5 to smithi20457400-35 2024-09-16T05:25:19.501 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 1289 (ObjNum 1359 snap 418 seq_num 1359) dirty exists 2024-09-16T05:25:19.501 INFO:tasks.rados.rados.2.smithi204.stdout:3914: left oid 35 (ObjNum 1359 snap 418 seq_num 1359) 2024-09-16T05:25:19.503 INFO:tasks.rados.rados.2.smithi204.stdout:3914: done (10 left) 2024-09-16T05:25:19.503 INFO:tasks.rados.rados.2.smithi204.stdout:3915: done (9 left) 2024-09-16T05:25:19.503 INFO:tasks.rados.rados.2.smithi204.stdout:3916: write (excl) oid 41 current snap is 418 2024-09-16T05:25:19.503 INFO:tasks.rados.rados.2.smithi204.stdout:3916: seq_num 1361 ranges {474327=441076,1675640=522810,2808105=66222} 2024-09-16T05:25:19.507 INFO:tasks.rados.rados.2.smithi204.stdout:3916: writing smithi20457400-41 from 474327 to 915403 tid 1 2024-09-16T05:25:19.514 INFO:tasks.rados.rados.2.smithi204.stdout:3916: writing smithi20457400-41 from 1675640 to 2198450 tid 2 2024-09-16T05:25:19.518 INFO:tasks.rados.rados.2.smithi204.stdout:3916: writing smithi20457400-41 from 2808105 to 2874327 tid 3 2024-09-16T05:25:19.518 INFO:tasks.rados.rados.2.smithi204.stdout:3917: read oid 21 snap -1 2024-09-16T05:25:19.518 INFO:tasks.rados.rados.2.smithi204.stdout:3917: expect (ObjNum 1334 snap 405 seq_num 1334) 2024-09-16T05:25:19.518 INFO:tasks.rados.rados.2.smithi204.stdout:3918: write oid 11 current snap is 418 2024-09-16T05:25:19.519 INFO:tasks.rados.rados.2.smithi204.stdout:3916: finishing write tid 1 to smithi20457400-41 2024-09-16T05:25:19.519 INFO:tasks.rados.rados.2.smithi204.stdout:3916: finishing write tid 2 to smithi20457400-41 2024-09-16T05:25:19.519 INFO:tasks.rados.rados.2.smithi204.stdout:3918: seq_num 1362 ranges {491134=516325,1734122=666468} 2024-09-16T05:25:19.524 INFO:tasks.rados.rados.2.smithi204.stdout:3918: writing smithi20457400-11 from 491134 to 1007459 tid 1 2024-09-16T05:25:19.532 INFO:tasks.rados.rados.2.smithi204.stdout:3918: writing smithi20457400-11 from 1734122 to 2400590 tid 2 2024-09-16T05:25:19.532 INFO:tasks.rados.rados.2.smithi204.stdout:3919: copy_from oid 12 from oid 29 current snap is 418 2024-09-16T05:25:19.532 INFO:tasks.rados.rados.2.smithi204.stdout:3916: finishing write tid 3 to smithi20457400-41 2024-09-16T05:25:19.532 INFO:tasks.rados.rados.2.smithi204.stdout:3916: finishing write tid 4 to smithi20457400-41 2024-09-16T05:25:19.532 INFO:tasks.rados.rados.2.smithi204.stdout:3916: finishing write tid 5 to smithi20457400-41 2024-09-16T05:25:19.533 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 41 v 1480 (ObjNum 1361 snap 418 seq_num 1361) dirty exists 2024-09-16T05:25:19.533 INFO:tasks.rados.rados.2.smithi204.stdout:3916: left oid 41 (ObjNum 1361 snap 418 seq_num 1361) 2024-09-16T05:25:19.533 INFO:tasks.rados.rados.2.smithi204.stdout:3918: finishing write tid 1 to smithi20457400-11 2024-09-16T05:25:19.533 INFO:tasks.rados.rados.2.smithi204.stdout:3916: done (12 left) 2024-09-16T05:25:19.533 INFO:tasks.rados.rados.2.smithi204.stdout:3920: write (excl) oid 48 current snap is 418 2024-09-16T05:25:19.533 INFO:tasks.rados.rados.2.smithi204.stdout:3917: expect (ObjNum 1334 snap 405 seq_num 1334) 2024-09-16T05:25:19.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:19.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:19.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:19.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:19.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:19.699 INFO:tasks.rados.rados.2.smithi204.stdout:3918: finishing write tid 2 to smithi20457400-11 2024-09-16T05:25:19.700 INFO:tasks.rados.rados.2.smithi204.stdout:3918: finishing write tid 3 to smithi20457400-11 2024-09-16T05:25:19.700 INFO:tasks.rados.rados.2.smithi204.stdout:3920: seq_num 1363 ranges {507941=591574,1676251=526479,2746598=161343} 2024-09-16T05:25:19.705 INFO:tasks.rados.rados.2.smithi204.stdout:3920: writing smithi20457400-48 from 507941 to 1099515 tid 1 2024-09-16T05:25:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:19.709 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:19.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:19.712 INFO:tasks.rados.rados.2.smithi204.stdout:3920: writing smithi20457400-48 from 1676251 to 2202730 tid 2 2024-09-16T05:25:19.716 INFO:tasks.rados.rados.2.smithi204.stdout:3920: writing smithi20457400-48 from 2746598 to 2907941 tid 3 2024-09-16T05:25:19.716 INFO:tasks.rados.rados.2.smithi204.stdout:3917: done (12 left) 2024-09-16T05:25:19.716 INFO:tasks.rados.rados.2.smithi204.stdout:3921: copy_from oid 40 from oid 23 current snap is 418 2024-09-16T05:25:19.717 INFO:tasks.rados.rados.2.smithi204.stdout:3918: finishing write tid 4 to smithi20457400-11 2024-09-16T05:25:19.717 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1445 (ObjNum 1362 snap 418 seq_num 1362) dirty exists 2024-09-16T05:25:19.717 INFO:tasks.rados.rados.2.smithi204.stdout:3918: left oid 11 (ObjNum 1362 snap 418 seq_num 1362) 2024-09-16T05:25:19.717 INFO:tasks.rados.rados.2.smithi204.stdout:3919: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:25:19.717 INFO:tasks.rados.rados.2.smithi204.stdout:3918: done (12 left) 2024-09-16T05:25:19.717 INFO:tasks.rados.rados.2.smithi204.stdout:3922: write (excl) oid 36 current snap is 418 2024-09-16T05:25:19.717 INFO:tasks.rados.rados.2.smithi204.stdout:3919: finishing copy_from to smithi20457400-12 2024-09-16T05:25:19.717 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1354 (ObjNum 1178 snap 360 seq_num 1178) dirty exists 2024-09-16T05:25:19.718 INFO:tasks.rados.rados.2.smithi204.stdout:3922: seq_num 1364 ranges {524748=666823,1734733=670137} 2024-09-16T05:25:19.722 INFO:tasks.rados.rados.2.smithi204.stdout:3922: writing smithi20457400-36 from 524748 to 1191571 tid 1 2024-09-16T05:25:19.726 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.1/config 2024-09-16T05:25:19.727 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:19 smithi032 ceph-mon[27388]: pgmap v1612: 127 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 3 stale+undersized+peered, 17 undersized+peered, 6 undersized+degraded+peered, 90 active+clean; 204 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 1.7 MiB/s wr, 5 op/s; 18/192 objects degraded (9.375%) 2024-09-16T05:25:19.727 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:19 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 99 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:19.727 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:19 smithi032 ceph-mon[27388]: osdmap e967: 12 total, 10 up, 12 in 2024-09-16T05:25:19.727 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/395087640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3922: writing smithi20457400-36 from 1734733 to 2404870 tid 2 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3919: done (12 left) 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3923: read oid 33 snap -1 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3923: expect deleted 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3924: read oid 15 snap -1 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3924: expect deleted 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3925: read oid 3 snap 415 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3925: expect (ObjNum 1140 snap 341 seq_num 1140) 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3926: write oid 35 current snap is 418 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3920: finishing write tid 1 to smithi20457400-48 2024-09-16T05:25:19.730 INFO:tasks.rados.rados.2.smithi204.stdout:3926: seq_num 1365 ranges {541555=742072,1793215=530148,2801444=140111} 2024-09-16T05:25:19.736 INFO:tasks.rados.rados.2.smithi204.stdout:3926: writing smithi20457400-35 from 541555 to 1283627 tid 1 2024-09-16T05:25:19.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:19 smithi196 ceph-mon[30802]: pgmap v1612: 127 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 3 stale+undersized+peered, 17 undersized+peered, 6 undersized+degraded+peered, 90 active+clean; 204 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 1.7 MiB/s wr, 5 op/s; 18/192 objects degraded (9.375%) 2024-09-16T05:25:19.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:19 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 99 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:19.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:19 smithi196 ceph-mon[30802]: osdmap e967: 12 total, 10 up, 12 in 2024-09-16T05:25:19.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/395087640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:19.742 INFO:tasks.rados.rados.2.smithi204.stdout:3926: writing smithi20457400-35 from 1793215 to 2323363 tid 2 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3926: writing smithi20457400-35 from 2801444 to 2941555 tid 3 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 16 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3920: finishing write tid 2 to smithi20457400-48 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3920: finishing write tid 3 to smithi20457400-48 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3920: finishing write tid 4 to smithi20457400-48 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3920: finishing write tid 5 to smithi20457400-48 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 800 (ObjNum 1363 snap 418 seq_num 1363) dirty exists 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3920: left oid 48 (ObjNum 1363 snap 418 seq_num 1363) 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3921: finishing copy_from racing read to smithi20457400-40 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3922: finishing write tid 1 to smithi20457400-36 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3921: finishing copy_from to smithi20457400-40 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 878 (ObjNum 1352 snap 415 seq_num 1352) dirty exists 2024-09-16T05:25:19.747 INFO:tasks.rados.rados.2.smithi204.stdout:3920: done (15 left) 2024-09-16T05:25:19.748 INFO:tasks.rados.rados.2.smithi204.stdout:3921: done (14 left) 2024-09-16T05:25:19.748 INFO:tasks.rados.rados.2.smithi204.stdout:3927: copy_from oid 1 from oid 8 current snap is 418 2024-09-16T05:25:19.748 INFO:tasks.rados.rados.2.smithi204.stdout:3922: finishing write tid 2 to smithi20457400-36 2024-09-16T05:25:19.748 INFO:tasks.rados.rados.2.smithi204.stdout:3922: finishing write tid 3 to smithi20457400-36 2024-09-16T05:25:19.748 INFO:tasks.rados.rados.2.smithi204.stdout:3922: finishing write tid 4 to smithi20457400-36 2024-09-16T05:25:19.748 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 36 v 814 (ObjNum 1364 snap 418 seq_num 1364) dirty exists 2024-09-16T05:25:19.748 INFO:tasks.rados.rados.2.smithi204.stdout:3922: left oid 36 (ObjNum 1364 snap 418 seq_num 1364) 2024-09-16T05:25:19.748 INFO:tasks.rados.rados.2.smithi204.stdout:3925: expect (ObjNum 1140 snap 341 seq_num 1140) 2024-09-16T05:25:19.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:19.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:19.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:19 smithi204 ceph-mon[30933]: pgmap v1612: 127 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 3 stale+undersized+peered, 17 undersized+peered, 6 undersized+degraded+peered, 90 active+clean; 204 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 1.7 MiB/s wr, 5 op/s; 18/192 objects degraded (9.375%) 2024-09-16T05:25:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:19 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 99 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:19 smithi204 ceph-mon[30933]: osdmap e967: 12 total, 10 up, 12 in 2024-09-16T05:25:19.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/395087640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:19.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:19.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:19.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:19.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:19.963 INFO:tasks.rados.rados.2.smithi204.stdout:3926: finishing write tid 1 to smithi20457400-35 2024-09-16T05:25:19.963 INFO:tasks.rados.rados.2.smithi204.stdout:3926: finishing write tid 2 to smithi20457400-35 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3927: finishing copy_from to smithi20457400-1 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3927: got expected ENOENT (src dne) 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3927: finishing copy_from racing read to smithi20457400-1 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3926: finishing write tid 3 to smithi20457400-35 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3926: finishing write tid 4 to smithi20457400-35 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3926: finishing write tid 5 to smithi20457400-35 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 35 v 1293 (ObjNum 1365 snap 418 seq_num 1365) dirty exists 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3926: left oid 35 (ObjNum 1365 snap 418 seq_num 1365) 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3922: done (14 left) 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3923: done (13 left) 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3924: done (12 left) 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3925: done (11 left) 2024-09-16T05:25:19.964 INFO:tasks.rados.rados.2.smithi204.stdout:3926: done (10 left) 2024-09-16T05:25:19.965 INFO:tasks.rados.rados.2.smithi204.stdout:3927: done (9 left) 2024-09-16T05:25:19.965 INFO:tasks.rados.rados.2.smithi204.stdout:3928: rollback oid 48 current snap is 418 2024-09-16T05:25:19.965 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 48 to 414 2024-09-16T05:25:19.965 INFO:tasks.rados.rados.2.smithi204.stdout:3929: write oid 23 current snap is 418 2024-09-16T05:25:19.965 INFO:tasks.rados.rados.2.smithi204.stdout:3929: seq_num 1366 ranges {558362=417321,1451697=673806,2712514=245848} 2024-09-16T05:25:19.968 INFO:tasks.rados.rados.2.smithi204.stdout:3929: writing smithi20457400-23 from 558362 to 975683 tid 1 2024-09-16T05:25:19.974 INFO:tasks.rados.rados.2.smithi204.stdout:3929: writing smithi20457400-23 from 1451697 to 2125503 tid 2 2024-09-16T05:25:19.980 INFO:tasks.rados.rados.2.smithi204.stdout:3929: writing smithi20457400-23 from 2712514 to 2958362 tid 3 2024-09-16T05:25:19.980 INFO:tasks.rados.rados.2.smithi204.stdout:3930: read oid 37 snap -1 2024-09-16T05:25:19.980 INFO:tasks.rados.rados.2.smithi204.stdout:3930: expect (ObjNum 1324 snap 403 seq_num 1324) 2024-09-16T05:25:19.980 INFO:tasks.rados.rados.2.smithi204.stdout:3931: rollback oid 19 current snap is 418 2024-09-16T05:25:19.980 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 19 to 415 2024-09-16T05:25:19.980 INFO:tasks.rados.rados.2.smithi204.stdout:3928: finishing rollback tid 0 to smithi20457400-48 2024-09-16T05:25:19.980 INFO:tasks.rados.rados.2.smithi204.stdout:3928: finishing rollback tid 1 to smithi20457400-48 2024-09-16T05:25:19.980 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 802 (ObjNum 49 snap 0 seq_num 4283158896) dirty dne 2024-09-16T05:25:19.981 INFO:tasks.rados.rados.2.smithi204.stdout:3929: finishing write tid 1 to smithi20457400-23 2024-09-16T05:25:19.981 INFO:tasks.rados.rados.2.smithi204.stdout:3929: finishing write tid 2 to smithi20457400-23 2024-09-16T05:25:19.981 INFO:tasks.rados.rados.2.smithi204.stdout:3928: done (12 left) 2024-09-16T05:25:19.981 INFO:tasks.rados.rados.2.smithi204.stdout:3932: write (excl) oid 9 current snap is 418 2024-09-16T05:25:19.981 INFO:tasks.rados.rados.2.smithi204.stdout:3932: seq_num 1367 ranges {575169=492570,1793826=417464,2623584=351585} 2024-09-16T05:25:19.985 INFO:tasks.rados.rados.2.smithi204.stdout:3932: writing smithi20457400-9 from 575169 to 1067739 tid 1 2024-09-16T05:25:19.986 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:19.986 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:19.992 INFO:tasks.rados.rados.2.smithi204.stdout:3932: writing smithi20457400-9 from 1793826 to 2211290 tid 2 2024-09-16T05:25:19.996 INFO:tasks.rados.rados.2.smithi204.stdout:3932: writing smithi20457400-9 from 2623584 to 2975169 tid 3 2024-09-16T05:25:19.996 INFO:tasks.rados.rados.2.smithi204.stdout:3933: copy_from oid 18 from oid 5 current snap is 418 2024-09-16T05:25:19.996 INFO:tasks.rados.rados.2.smithi204.stdout:3931: finishing rollback tid 1 to smithi20457400-19 2024-09-16T05:25:19.996 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 19 v 1221 (ObjNum 33 snap 0 seq_num 4283220864) dirty dne 2024-09-16T05:25:19.996 INFO:tasks.rados.rados.2.smithi204.stdout:3929: finishing write tid 3 to smithi20457400-23 2024-09-16T05:25:19.996 INFO:tasks.rados.rados.2.smithi204.stdout:3929: finishing write tid 4 to smithi20457400-23 2024-09-16T05:25:19.996 INFO:tasks.rados.rados.2.smithi204.stdout:3929: finishing write tid 5 to smithi20457400-23 2024-09-16T05:25:19.996 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 882 (ObjNum 1366 snap 418 seq_num 1366) dirty exists 2024-09-16T05:25:19.997 INFO:tasks.rados.rados.2.smithi204.stdout:3929: left oid 23 (ObjNum 1366 snap 418 seq_num 1366) 2024-09-16T05:25:19.997 INFO:tasks.rados.rados.2.smithi204.stdout:3929: done (13 left) 2024-09-16T05:25:19.997 INFO:tasks.rados.rados.2.smithi204.stdout:3931: done (12 left) 2024-09-16T05:25:19.997 INFO:tasks.rados.rados.2.smithi204.stdout:3934: rollback oid 40 current snap is 418 2024-09-16T05:25:19.997 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 40 to 414 2024-09-16T05:25:19.997 INFO:tasks.rados.rados.2.smithi204.stdout:3930: expect (ObjNum 1324 snap 403 seq_num 1324) 2024-09-16T05:25:20.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:20.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:25:20.079 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:20.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:20.120 INFO:tasks.rados.rados.2.smithi204.stdout:3932: finishing write tid 1 to smithi20457400-9 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3932: finishing write tid 2 to smithi20457400-9 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3934: finishing rollback tid 0 to smithi20457400-40 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3934: finishing rollback tid 1 to smithi20457400-40 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1074 (ObjNum 241 snap 0 seq_num 4282974176) dirty dne 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3932: finishing write tid 3 to smithi20457400-9 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3932: finishing write tid 4 to smithi20457400-9 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3932: finishing write tid 5 to smithi20457400-9 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 9 v 1484 (ObjNum 1367 snap 418 seq_num 1367) dirty exists 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3932: left oid 9 (ObjNum 1367 snap 418 seq_num 1367) 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3933: finishing copy_from racing read to smithi20457400-18 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3933: finishing copy_from to smithi20457400-18 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 18 v 1487 (ObjNum 1325 snap 403 seq_num 1325) dirty exists 2024-09-16T05:25:20.121 INFO:tasks.rados.rados.2.smithi204.stdout:3930: done (12 left) 2024-09-16T05:25:20.122 INFO:tasks.rados.rados.2.smithi204.stdout:3932: done (11 left) 2024-09-16T05:25:20.122 INFO:tasks.rados.rados.2.smithi204.stdout:3933: done (10 left) 2024-09-16T05:25:20.122 INFO:tasks.rados.rados.2.smithi204.stdout:3934: done (9 left) 2024-09-16T05:25:20.122 INFO:tasks.rados.rados.2.smithi204.stdout:3935: snap_create 2024-09-16T05:25:20.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:20.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:20.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:20.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:20.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:20.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:20.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:20.450 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:20.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:20.492 INFO:tasks.rados.rados.2.smithi204.stdout:3935: done (9 left) 2024-09-16T05:25:20.492 INFO:tasks.rados.rados.2.smithi204.stdout:3936: write (excl) oid 47 current snap is 419 2024-09-16T05:25:20.492 INFO:tasks.rados.rados.2.smithi204.stdout:3936: seq_num 1368 ranges {591976=567819,1852308=677475} 2024-09-16T05:25:20.497 INFO:tasks.rados.rados.2.smithi204.stdout:3936: writing smithi20457400-47 from 591976 to 1159795 tid 1 2024-09-16T05:25:20.505 INFO:tasks.rados.rados.2.smithi204.stdout:3936: writing smithi20457400-47 from 1852308 to 2529783 tid 2 2024-09-16T05:25:20.506 INFO:tasks.rados.rados.2.smithi204.stdout:3936: finishing write tid 1 to smithi20457400-47 2024-09-16T05:25:20.506 INFO:tasks.rados.rados.2.smithi204.stdout:3937: delete oid 42 current snap is 419 2024-09-16T05:25:20.508 INFO:tasks.rados.rados.2.smithi204.stdout:3936: finishing write tid 2 to smithi20457400-47 2024-09-16T05:25:20.508 INFO:tasks.rados.rados.2.smithi204.stdout:3936: finishing write tid 3 to smithi20457400-47 2024-09-16T05:25:20.508 INFO:tasks.rados.rados.2.smithi204.stdout:3936: finishing write tid 4 to smithi20457400-47 2024-09-16T05:25:20.508 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 47 v 1412 (ObjNum 1368 snap 419 seq_num 1368) dirty exists 2024-09-16T05:25:20.509 INFO:tasks.rados.rados.2.smithi204.stdout:3936: left oid 47 (ObjNum 1368 snap 419 seq_num 1368) 2024-09-16T05:25:20.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:20.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:20.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:20.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:20 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:20.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/395087640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:20.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:20 smithi032 ceph-mon[27388]: osdmap e968: 12 total, 10 up, 12 in 2024-09-16T05:25:20.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:20.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:20.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:20.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:20 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/395087640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:20.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:20 smithi196 ceph-mon[30802]: osdmap e968: 12 total, 10 up, 12 in 2024-09-16T05:25:20.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:20.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:20.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:20.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:20 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/395087640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:20.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:20 smithi204 ceph-mon[30933]: osdmap e968: 12 total, 10 up, 12 in 2024-09-16T05:25:20.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:20.947 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:20.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:21.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:21.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:21.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:21.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:21.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:21.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:21.321 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:21.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:21.423 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:21.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:21.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:21.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:21.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:21.654 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:25:21.654 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:25:21.654 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: 2.2a scrub starts 2024-09-16T05:25:21.654 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: 2.2a scrub ok 2024-09-16T05:25:21.655 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: pgmap v1615: 127 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 3 stale+undersized+peered, 17 undersized+peered, 6 undersized+degraded+peered, 90 active+clean; 213 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1.1 MiB/s wr, 7 op/s; 18/200 objects degraded (9.000%) 2024-09-16T05:25:21.655 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: osdmap e969: 12 total, 10 up, 12 in 2024-09-16T05:25:21.655 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 18/200 objects degraded (9.000%), 6 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:21.655 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: 2.59 scrub starts 2024-09-16T05:25:21.655 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:21 smithi032 ceph-mon[27388]: 2.59 scrub ok 2024-09-16T05:25:21.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:21.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:25:21.700 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:21.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:21.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:21.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:21.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:21.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:25:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:25:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: 2.2a scrub starts 2024-09-16T05:25:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: 2.2a scrub ok 2024-09-16T05:25:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: pgmap v1615: 127 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 3 stale+undersized+peered, 17 undersized+peered, 6 undersized+degraded+peered, 90 active+clean; 213 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1.1 MiB/s wr, 7 op/s; 18/200 objects degraded (9.000%) 2024-09-16T05:25:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: osdmap e969: 12 total, 10 up, 12 in 2024-09-16T05:25:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 18/200 objects degraded (9.000%), 6 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: 2.59 scrub starts 2024-09-16T05:25:21.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:21 smithi204 ceph-mon[30933]: 2.59 scrub ok 2024-09-16T05:25:21.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:21.955 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:21.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: 2.2a scrub starts 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: 2.2a scrub ok 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: pgmap v1615: 127 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 3 stale+undersized+peered, 17 undersized+peered, 6 undersized+degraded+peered, 90 active+clean; 213 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1.1 MiB/s wr, 7 op/s; 18/200 objects degraded (9.000%) 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: osdmap e969: 12 total, 10 up, 12 in 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 18/200 objects degraded (9.000%), 6 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: 2.59 scrub starts 2024-09-16T05:25:21.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:21 smithi196 ceph-mon[30802]: 2.59 scrub ok 2024-09-16T05:25:22.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:22.082 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:22.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:22.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:22.185 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:22.209 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:22.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:22.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:22.337 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:22.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:22.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:22.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:22.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:22.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:22.591 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:22.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:22.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:22.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:22.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:22.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:22.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:22.822 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:22.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:22.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:22 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 27 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:25:22.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4092837680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:22.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:22 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 27 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:25:22.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4092837680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:22.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:22.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:22.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:22.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:22 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 27 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:25:22.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4092837680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:22.995 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:25:23.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:25:23.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:23.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:23.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:23.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:23.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:23.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:23.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:23.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:23.359 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:23.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:23.390 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:23.412 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:23.413 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:23.433 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:23.434 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:23.456 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:23.456 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:23.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:23.478 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:23.478 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:23.486 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:23.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:23.500 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:23.500 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:25:23.524 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:23.524 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:23.546 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:23.546 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:23.567 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:23.568 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:23.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:23.589 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:23.589 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:23.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:23.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:23.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:23.740 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:23.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:23.823 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:23 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:25:23.824 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:23 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:25:23.824 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:23 smithi196 ceph-mon[30802]: pgmap v1617: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 229 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 4.2 MiB/s wr, 22 op/s; 27/214 objects degraded (12.617%) 2024-09-16T05:25:23.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:23.852 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:23 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:25:23.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:23 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:25:23.853 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:23 smithi032 ceph-mon[27388]: pgmap v1617: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 229 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 4.2 MiB/s wr, 22 op/s; 27/214 objects degraded (12.617%) 2024-09-16T05:25:23.868 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:23.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:23.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:23 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:25:23.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:23 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:25:23.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:23 smithi204 ceph-mon[30933]: pgmap v1617: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 229 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 4.2 MiB/s wr, 22 op/s; 27/214 objects degraded (12.617%) 2024-09-16T05:25:23.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:23.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:23.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:24.045 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:25:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:25:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:25:24.045 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:25:24.045 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:25:24.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:24.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:24.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:24.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:23.821+0000 7f6af1ad6640 -1 osd.6 969 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:24.251 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:24.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:24.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:24.376 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:24.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:24.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:24.503 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:24.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:24.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:24.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:24.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:24.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:24 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 99 sec, osd.6 has slow ops) 2024-09-16T05:25:24.706 INFO:teuthology.orchestra.run.smithi032.stdout:2.9 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.5d 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.4f 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.4 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.17 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.5b 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:1.1d 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.10 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.15 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.44 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.23 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:2.28 2024-09-16T05:25:24.707 INFO:teuthology.orchestra.run.smithi032.stdout:1.13 2024-09-16T05:25:24.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.2 2024-09-16T05:25:24.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.45 2024-09-16T05:25:24.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.35 2024-09-16T05:25:24.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.3c 2024-09-16T05:25:24.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.33 2024-09-16T05:25:24.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.3d 2024-09-16T05:25:24.708 INFO:teuthology.orchestra.run.smithi032.stdout:1.b 2024-09-16T05:25:24.708 INFO:teuthology.orchestra.run.smithi032.stdout:2.36 2024-09-16T05:25:24.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:24.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:24.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:24.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:25:24.858 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:24.872 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:24 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 99 sec, osd.6 has slow ops) 2024-09-16T05:25:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:24.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:24.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:24.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:24 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 99 sec, osd.6 has slow ops) 2024-09-16T05:25:24.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:25.012 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:25.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:25.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:25.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:25.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:25.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:24 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:24.871+0000 7f6af1ad6640 -1 osd.6 969 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:25.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:25.265 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:25.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:25.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:25.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:25.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:25.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:25.521 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:25.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:25.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:25.646 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:25.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:25.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:25.774 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:25.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:25.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:25.900 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:25.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:25:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:25:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:25:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:25:25.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 2.2f scrub starts 2024-09-16T05:25:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 2.2f scrub ok 2024-09-16T05:25:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: pgmap v1618: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 21 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:25:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:25:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:25:25.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:25 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:25:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:25.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 2.2f scrub starts 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 2.2f scrub ok 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: pgmap v1618: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 21 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:25:25.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:25 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:25:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:25.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.1 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 1.b --file /var/log/ceph/exp.1.b.1 2024-09-16T05:25:25.959 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:25.988 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:25.832+0000 7f6af1ad6640 -1 osd.6 969 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:25:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:25:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:25:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:25:25.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 2.2f scrub starts 2024-09-16T05:25:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 2.2f scrub ok 2024-09-16T05:25:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: pgmap v1618: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 21 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:25:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:25:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:25:25.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:25 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:25:26.002 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:26.030 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:26.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:26.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:26.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:26.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:26.197 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.1/config 2024-09-16T05:25:26.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:26.286 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:26.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:26.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:26.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:26.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:26.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:26.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:26.642 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:26.667 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:26.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:26.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:26.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:26.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:26.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:26 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:26.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:26 smithi032 ceph-mon[27388]: 2.5c scrub starts 2024-09-16T05:25:26.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:26 smithi032 ceph-mon[27388]: 2.5c scrub ok 2024-09-16T05:25:26.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:26 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:25:26.864 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:26 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:25:26.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:26 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:25:26.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:26 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:25:26.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:26 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 27/222 objects degraded (12.162%), 10 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:26.878 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:26.878 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-mon[30802]: 2.5c scrub starts 2024-09-16T05:25:26.878 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-mon[30802]: 2.5c scrub ok 2024-09-16T05:25:26.878 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:25:26.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:25:26.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:25:26.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:25:26.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 27/222 objects degraded (12.162%), 10 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:26.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:26.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:26 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:26 smithi204 ceph-mon[30933]: 2.5c scrub starts 2024-09-16T05:25:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:26 smithi204 ceph-mon[30933]: 2.5c scrub ok 2024-09-16T05:25:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:26 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:25:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:26 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:25:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:26 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:25:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:26 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:25:26.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:26 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 27/222 objects degraded (12.162%), 10 pgs degraded, 21 pgs undersized (PG_DEGRADED) 2024-09-16T05:25:26.922 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:26.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:27.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:27.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:27.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:27.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:27.177 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:27.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:27.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:25:27.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:26.877+0000 7f6af1ad6640 -1 osd.6 969 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:27.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:27.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:27.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.31,GRANT,e:542,from:10,reservation_nonce:0)", 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 434.36814347500001, 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.69991632400000003, 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.387 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016243+0000", 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.388 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016244+0000", 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7400000000000002e-07 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.016249+0000", 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0259999999999998e-06 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.715937+0000", 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.69968885199999997 2024-09-16T05:25:27.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.715950+0000", 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2914999999999999e-05 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716018+0000", 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7972000000000006e-05 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716036+0000", 2024-09-16T05:25:27.390 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7302000000000001e-05 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:13.716159+0000", 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000123483 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7279 2.31 2:8c1ba486:::smithi20457400-2:head [stat,delete] snapc 133=[133,132,131,12f,12c,12a,126,125,123,119,116] ondisk+write+known_if_redirected+supports_pool_eio e738)", 2024-09-16T05:25:27.391 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:27.821755+0000", 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 239.562631479, 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0063770049999999998, 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7279 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821755+0000", 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821755+0000", 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821756+0000", 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5600000000000001e-06 2024-09-16T05:25:27.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821762+0000", 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7599999999999999e-06 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821774+0000", 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1837e-05 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.394 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.821784+0000", 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0141e-05 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.822598+0000", 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00081351800000000005 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.822702+0000", 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000103905 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.822974+0000", 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00027219499999999998 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.825144+0000", 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.002170216 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.827867+0000", 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0027226379999999999 2024-09-16T05:25:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.828043+0000", 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017595100000000001 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.828132+0000", 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9283999999999994e-05 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.1f,GRANT,e:738,from:7,reservation_nonce:0)", 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:27.877847+0000", 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 239.50653902400001, 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.247356669, 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.877847+0000", 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.877847+0000", 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.877848+0000", 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6000000000000003e-07 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:27.877850+0000", 2024-09-16T05:25:27.399 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4889999999999998e-06 2024-09-16T05:25:27.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.400 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.124951+0000", 2024-09-16T05:25:27.400 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.24710046999999999 2024-09-16T05:25:27.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.400 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.124962+0000", 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1044e-05 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.125031+0000", 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9036999999999996e-05 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.125049+0000", 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7813e-05 2024-09-16T05:25:27.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.402 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.402 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:28.125204+0000", 2024-09-16T05:25:27.402 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015515599999999999 2024-09-16T05:25:27.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.402 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.403 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7356 2.31 2:8c1ba486:::smithi20457400-2:head [write 427175~721527,stat] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:25:27.403 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:36.881000+0000", 2024-09-16T05:25:27.403 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 230.50338592099999, 2024-09-16T05:25:27.403 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0071041089999999999, 2024-09-16T05:25:27.403 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.403 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7356 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881000+0000", 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.404 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881000+0000", 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881005+0000", 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1099999999999996e-06 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881920+0000", 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00091555300000000005 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.405 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881948+0000", 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7817000000000002e-05 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.881961+0000", 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3616e-05 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.882034+0000", 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3005999999999999e-05 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.882041+0000", 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9700000000000002e-06 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.884292+0000", 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022501690000000002 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.884403+0000", 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000111054 2024-09-16T05:25:27.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.884593+0000", 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00019044900000000001 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.886037+0000", 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001443571 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:27.408 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.888030+0000", 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001992794 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.888072+0000", 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2719000000000002e-05 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:36.888105+0000", 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2280999999999998e-05 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.56,GRANT,e:751,from:2,reservation_nonce:0)", 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:41.909362+0000", 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 225.47502396300001, 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.82786359200000004, 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:41.909362+0000", 2024-09-16T05:25:27.410 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:41.909362+0000", 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:41.909363+0000", 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.411 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:41.909366+0000", 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.084e-06 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.736932+0000", 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.82756565000000004 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.736943+0000", 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1231e-05 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.737058+0000", 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00011534799999999999 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.737073+0000", 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.4447000000000001e-05 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:42.737226+0000", 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015337799999999999 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:7554 2.5 e762/576 2:a1d8c6d9:::smithi20457400-3:head v 762'634, mlcod=761'631)", 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:52.224172+0000", 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 215.16021477999999, 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0056041479999999998, 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.224172+0000", 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.224172+0000", 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.415 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.224174+0000", 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2709999999999999e-06 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.226944+0000", 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00277005 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.226987+0000", 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3625999999999999e-05 2024-09-16T05:25:27.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.226997+0000", 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.8900000000000002e-06 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.227070+0000", 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2711000000000002e-05 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.227089+0000", 2024-09-16T05:25:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8450000000000001e-05 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.229712+0000", 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026229999999999999 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.229776+0000", 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.4150000000000001e-05 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7997 2.56 2:6be67188:::smithi20457400-46:head [write 689384~664426,stat] snapc 153=[153,152,151,150,14e,14b,149,126,119,116] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:32.886758+0000", 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 174.49762877500001, 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0058758949999999999, 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7997 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.886758+0000", 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.886758+0000", 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.886760+0000", 2024-09-16T05:25:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5069999999999999e-06 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887721+0000", 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00096062100000000004 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887746+0000", 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5806999999999999e-05 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887758+0000", 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1141999999999999e-05 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887826+0000", 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8781000000000004e-05 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.887834+0000", 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7389999999999999e-06 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.889443+0000", 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016089660000000001 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.889548+0000", 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105035 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.889781+0000", 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000232585 2024-09-16T05:25:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.891868+0000", 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0020870419999999999 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.892550+0000", 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00068241000000000005 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.424 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.892592+0000", 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1808999999999999e-05 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:32.892633+0000", 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1451e-05 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8154 2.56 2:6be67188:::smithi20457400-46:116 [read 0~2369985,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e814)", 2024-09-16T05:25:27.425 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:43.836540+0000", 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 163.54784631300001, 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0062454349999999997, 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8154 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.426 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836540+0000", 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836540+0000", 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836541+0000", 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9200000000000001e-07 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.427 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836545+0000", 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0729999999999998e-06 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836558+0000", 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3045e-05 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836565+0000", 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1080000000000004e-06 2024-09-16T05:25:27.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836736+0000", 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017138799999999999 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836742+0000", 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1800000000000001e-06 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.841347+0000", 2024-09-16T05:25:27.429 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.004605011 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.841461+0000", 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000113952 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.842785+0000", 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013240859999999999 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8155 2.56 2:6be67188:::smithi20457400-46:116 [read 0~2369985,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e814)", 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:43.836581+0000", 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 163.54780572199999, 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0074930639999999998, 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8155 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.431 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836581+0000", 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836581+0000", 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.432 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836581+0000", 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3900000000000004e-07 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836585+0000", 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4249999999999998e-06 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836592+0000", 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7200000000000006e-06 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.433 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836598+0000", 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6389999999999997e-06 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836776+0000", 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00017738300000000001 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.836781+0000", 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5049999999999998e-06 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.842827+0000", 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006046147 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.842953+0000", 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012547 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:43.844074+0000", 2024-09-16T05:25:27.435 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001121036 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8298 2.31 2:8c1ba486:::smithi20457400-2:head [setxattr _header (57) in=7b,truncate 3361664] snapc 15a=[15a] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:54.178108+0000", 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 153.20627795499999, 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0095921700000000006, 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.436 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8298 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178108+0000", 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178108+0000", 2024-09-16T05:25:27.437 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178109+0000", 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8000000000000004e-07 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178114+0000", 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5750000000000002e-06 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.438 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178120+0000", 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.409e-06 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178126+0000", 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2700000000000001e-06 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179248+0000", 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00112155 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.439 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179305+0000", 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6554000000000001e-05 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179413+0000", 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108739 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179850+0000", 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00043647400000000001 2024-09-16T05:25:27.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187615+0000", 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0077655429999999998 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187655+0000", 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9270999999999997e-05 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187701+0000", 2024-09-16T05:25:27.441 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5905000000000002e-05 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8299 2.31 2:8c1ba486:::smithi20457400-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:54.178139+0000", 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 153.20624751700001, 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0097353619999999991, 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.442 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8299 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178139+0000", 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.443 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178139+0000", 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178139+0000", 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3200000000000005e-07 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178142+0000", 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7310000000000002e-06 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178150+0000", 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.481e-06 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.178156+0000", 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.0399999999999998e-06 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.445 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179775+0000", 2024-09-16T05:25:27.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:27.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0016190169999999999 2024-09-16T05:25:27.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.447 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:27.447 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.179834+0000", 2024-09-16T05:25:27.447 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8878000000000001e-05 2024-09-16T05:25:27.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187742+0000", 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0079080120000000007 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187797+0000", 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5648e-05 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.448 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:54.187874+0000", 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6922999999999995e-05 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8377 2.56 2:6be67188:::smithi20457400-46:head [write 696120~574496,stat] snapc 15f=[15f] ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:03.967471+0000", 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 143.416915265, 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0073863979999999997, 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.449 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8377 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.967471+0000", 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.967471+0000", 2024-09-16T05:25:27.450 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.967477+0000", 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6749999999999999e-06 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968251+0000", 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00077443200000000003 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.451 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968274+0000", 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.2738000000000002e-05 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968285+0000", 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.1296e-05 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968363+0000", 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8243000000000003e-05 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.968371+0000", 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.306e-06 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.971425+0000", 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030540440000000001 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.971524+0000", 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.9303999999999997e-05 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.971691+0000", 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000167009 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:27.454 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.974327+0000", 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026363990000000002 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.974756+0000", 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000428458 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.974813+0000", 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7216999999999998e-05 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.455 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:03.974857+0000", 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4277000000000003e-05 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:8893 2.31 2:8c1ba486:::smithi20457400-2:head [read 0~2470926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e889)", 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:58.942568+0000", 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 88.4418182, 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.006711463, 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 8893 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942568+0000", 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942568+0000", 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942569+0000", 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7999999999999996e-07 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942572+0000", 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6780000000000002e-06 2024-09-16T05:25:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942579+0000", 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.725e-06 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942586+0000", 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7680000000000003e-06 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942656+0000", 2024-09-16T05:25:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0528e-05 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.942665+0000", 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7679999999999997e-06 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.947735+0000", 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0050697249999999998 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.947822+0000", 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7022999999999997e-05 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:58.949280+0000", 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0014577679999999999 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:8964 2.5 e900/576 2:a1d8c6d9:::smithi20457400-3:head v 900'690, mlcod=874'686)", 2024-09-16T05:25:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:10.091185+0000", 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 77.293201092000004, 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.089365204000000004, 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.091185+0000", 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.462 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.091185+0000", 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.091186+0000", 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6700000000000002e-07 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.091193+0000", 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0700000000000001e-06 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.463 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.179929+0000", 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.088736310999999998 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.179942+0000", 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2269999999999999e-05 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.180011+0000", 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9436000000000004e-05 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.180032+0000", 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.1319999999999999e-05 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.180514+0000", 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048135300000000001 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:10.180550+0000", 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6576999999999999e-05 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.465 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.3.0:322 2.5a e907/742 2:5ae26c04:::smithi20457400-32:183 v 907'1264, mlcod=901'1260)", 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:17.143478+0000", 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 70.240908228999999, 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.97020481599999997, 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.466 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:17.143478+0000", 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:17.143478+0000", 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:17.143478+0000", 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.5100000000000001e-07 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:17.143486+0000", 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6009999999999997e-06 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.112924+0000", 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.969437781 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.112932+0000", 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0050000000000004e-06 2024-09-16T05:25:27.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.112997+0000", 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5691000000000005e-05 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.113028+0000", 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0852999999999998e-05 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.113637+0000", 2024-09-16T05:25:27.469 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00060868800000000002 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:18.113683+0000", 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5945999999999997e-05 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:9425 2.5c 2:3a3554ba:::smithi20457400-48:head [delete] snapc 194=[194,191,16e,16a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e937)", 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:47.305046+0000", 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 40.079339881999999, 2024-09-16T05:25:27.470 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010243557, 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 9425 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305046+0000", 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305046+0000", 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305049+0000", 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.4169999999999999e-06 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305057+0000", 2024-09-16T05:25:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0299999999999994e-06 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305077+0000", 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0069e-05 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305087+0000", 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0142e-05 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305183+0000", 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.6175999999999995e-05 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305425+0000", 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000242122 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 8", 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.305586+0000", 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016091200000000001 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.306766+0000", 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001179317 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.315202+0000", 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0084363419999999995 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.315244+0000", 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2496000000000001e-05 2024-09-16T05:25:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:47.315290+0000", 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.5534000000000001e-05 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:9619 2.31 2:8c1ba486:::smithi20457400-2:head [copy-from ver 1270] snapc 197=[16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:56.581942+0000", 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.802443901, 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.017899093000000001, 2024-09-16T05:25:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 9619 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581942+0000", 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581942+0000", 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581944+0000", 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.322e-06 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581950+0000", 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.823e-06 2024-09-16T05:25:27.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581962+0000", 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2384999999999999e-05 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581970+0000", 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6229999999999997e-06 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.582036+0000", 2024-09-16T05:25:27.479 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6654000000000003e-05 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.582044+0000", 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6850000000000001e-06 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.583645+0000", 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00160103 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.583715+0000", 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9565000000000002e-05 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.586751+0000", 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0030361619999999998 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.590530+0000", 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0037790850000000002 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.599752+0000", 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0092226840000000001 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.599794+0000", 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.1621e-05 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.599842+0000", 2024-09-16T05:25:27.482 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7453999999999997e-05 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:9620 2.31 2:8c1ba486:::smithi20457400-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e945)", 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:56.581987+0000", 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 30.802399732000001, 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0076722880000000002, 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.483 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 9620 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581987+0000", 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581987+0000", 2024-09-16T05:25:27.484 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581987+0000", 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6899999999999998e-07 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581989+0000", 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6419999999999999e-06 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.485 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581993+0000", 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.6500000000000004e-06 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.581997+0000", 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3639999999999999e-06 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.582088+0000", 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1621999999999998e-05 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.486 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.582091+0000", 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0929999999999999e-06 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.583873+0000", 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0017812349999999999 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.583930+0000", 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.7781000000000001e-05 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.487 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.589525+0000", 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0055946870000000001 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.589588+0000", 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.3077000000000006e-05 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:56.589659+0000", 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0767999999999997e-05 2024-09-16T05:25:27.488 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(1.16,QREQUEST,e:954,from:2,reservation_nonce:5)", 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:25:05.715825+0000", 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 21.668561551, 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90022242699999999, 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.489 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:05.715825+0000", 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:05.715825+0000", 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:05.715826+0000", 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3099999999999996e-07 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.491 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.491 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:05.715831+0000", 2024-09-16T05:25:27.491 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3689999999999998e-06 2024-09-16T05:25:27.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.492 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:06.615935+0000", 2024-09-16T05:25:27.492 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.90010355600000003 2024-09-16T05:25:27.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.492 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:06.615943+0000", 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.5620000000000004e-06 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:06.615999+0000", 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6203999999999999e-05 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:06.616017+0000", 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7316000000000002e-05 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:06.616047+0000", 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0488999999999999e-05 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:9787 2.31 2:8c1ba486:::smithi20457400-2:head [write 689450~580631] snapc 1a0=[1a0,19f,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e961)", 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:25:12.648662+0000", 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 14.735723833, 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0035877650000000001, 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 9787 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.648662+0000", 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.648662+0000", 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.648666+0000", 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.247e-06 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.649513+0000", 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00084688899999999998 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.649531+0000", 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7997999999999999e-05 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.649540+0000", 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.9509999999999995e-06 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.649608+0000", 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8250000000000006e-05 2024-09-16T05:25:27.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.649686+0000", 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.8165999999999997e-05 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 10", 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.649818+0000", 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013192100000000001 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.651361+0000", 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0015429529999999999 2024-09-16T05:25:27.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.652125+0000", 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00076369399999999998 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.652190+0000", 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.5196000000000004e-05 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:12.652250+0000", 2024-09-16T05:25:27.499 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.05e-05 2024-09-16T05:25:27.500 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.500 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.500 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.500 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:27.500 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:27.500 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:25:27.525 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:27.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:27.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:27.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:27.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:27.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:27.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:27.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:27.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:27.905 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:27.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:27.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:27 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:27 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:25:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:27 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:25:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:27 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:25:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:27 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:25:27.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:27 smithi032 ceph-mon[27388]: pgmap v1619: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.2 MiB/s wr, 16 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:27 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:27 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:25:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:27 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:25:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:27 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:25:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:27 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:25:27.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:27 smithi204 ceph-mon[30933]: pgmap v1619: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.2 MiB/s wr, 16 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:27.989 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:27.836+0000 7f6af1ad6640 -1 osd.6 969 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:27 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:27 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:25:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:27 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:25:27.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:27 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:25:27.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:27 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:25:27.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:27 smithi196 ceph-mon[30802]: pgmap v1619: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.2 MiB/s wr, 16 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:28.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:28.029 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:28.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:28.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:28.154 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:28.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:28.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:28.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:28.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:28.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:28.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:28.411 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:28.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:28.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:28.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:28.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:25:28.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:28.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:25:28.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:28.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:28.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:28.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:28.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:28.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:28 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:28.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:28 smithi032 ceph-mon[27388]: 2.42 scrub starts 2024-09-16T05:25:28.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:28 smithi032 ceph-mon[27388]: 2.42 scrub ok 2024-09-16T05:25:28.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2521355396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:28.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:28 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:28.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:28 smithi204 ceph-mon[30933]: 2.42 scrub starts 2024-09-16T05:25:28.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:28 smithi204 ceph-mon[30933]: 2.42 scrub ok 2024-09-16T05:25:28.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2521355396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:28.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:28.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:28.988 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:28 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:28.789+0000 7f6af1ad6640 -1 osd.6 970 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:28 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:28 smithi196 ceph-mon[30802]: 2.42 scrub starts 2024-09-16T05:25:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:28 smithi196 ceph-mon[30802]: 2.42 scrub ok 2024-09-16T05:25:28.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2521355396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:29.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:29.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:29.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:29.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:29.180 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:29.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:29.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:29.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:29.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:29.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:29.434 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:29.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:29.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:29.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:29.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:29.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:29.691 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:29.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:29.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:29.819 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:29.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:29.826 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:25:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:25:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: pgmap v1620: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.7 MiB/s wr, 13 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: 1.1b scrub starts 2024-09-16T05:25:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: 1.1b scrub ok 2024-09-16T05:25:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 110 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:29.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2521355396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: osdmap e970: 12 total, 10 up, 12 in 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: 2.16 scrub starts 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:29 smithi204 ceph-mon[30933]: 2.16 scrub ok 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: pgmap v1620: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.7 MiB/s wr, 13 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: 1.1b scrub starts 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: 1.1b scrub ok 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 110 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:29.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2521355396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:29.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: osdmap e970: 12 total, 10 up, 12 in 2024-09-16T05:25:29.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: 2.16 scrub starts 2024-09-16T05:25:29.923 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:29 smithi032 ceph-mon[27388]: 2.16 scrub ok 2024-09-16T05:25:29.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:29.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:29.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:29.987 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:29.825+0000 7f6af1ad6640 -1 osd.6 970 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:25:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:25:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: pgmap v1620: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.7 MiB/s wr, 13 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: 1.1b scrub starts 2024-09-16T05:25:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: 1.1b scrub ok 2024-09-16T05:25:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 110 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:29.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:29.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2521355396' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:29.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: osdmap e970: 12 total, 10 up, 12 in 2024-09-16T05:25:29.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: 2.16 scrub starts 2024-09-16T05:25:29.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:29 smithi196 ceph-mon[30802]: 2.16 scrub ok 2024-09-16T05:25:30.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:30.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:30.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:30.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:30.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:30.205 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:30.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:Exporting 1.b info 1.b( v 35'139 (0'0,35'139] local-lis/les=795/796 n=0 ec=401/15 lis/c=795/795 les/c/f=796/796/0 sis=795) 2024-09-16T05:25:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:Export successful 2024-09-16T05:25:30.307 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:30.332 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:30.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:30.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:25:30.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:30.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:30.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:30.585 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:30.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:30.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:30.711 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:30.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:30.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:30.839 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:30.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:30.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:30.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:30 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:30.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:30 smithi032 ceph-mon[27388]: 2.58 scrub starts 2024-09-16T05:25:30.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:30 smithi032 ceph-mon[27388]: 2.58 scrub ok 2024-09-16T05:25:30.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:30 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:25:30.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:30 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:25:30.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:30 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:25:30.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:30 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:25:30.919 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:30.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:30 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:30.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:30 smithi204 ceph-mon[30933]: 2.58 scrub starts 2024-09-16T05:25:30.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:30 smithi204 ceph-mon[30933]: 2.58 scrub ok 2024-09-16T05:25:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:30 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:25:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:30 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:25:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:30 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:25:30.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:30 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:25:30.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:30.942 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:30.942 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:30.962 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:30.967 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:30.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:30.984 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:30.984 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:30.987 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:30 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:30.849+0000 7f6af1ad6640 -1 osd.6 970 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:30.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:30 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:30 smithi196 ceph-mon[30802]: 2.58 scrub starts 2024-09-16T05:25:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:30 smithi196 ceph-mon[30802]: 2.58 scrub ok 2024-09-16T05:25:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:30 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:25:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:30 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:25:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:30 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:25:30.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:30 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:25:31.006 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:31.006 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:31.028 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:31.028 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:31.029 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:25:31.050 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:31.050 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:31.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:31.072 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:31.072 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:31.092 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:31.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:31.095 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:31.096 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:31.118 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:31.118 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:31.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:31.217 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:31.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:31.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:31.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:25:31.345 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:31.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:31.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:31.474 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:31.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:31.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:31.597 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:31.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:31.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.1 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 1.b 2024-09-16T05:25:31.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:31.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:31.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:31.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:31.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:31.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:31 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:31.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:31 smithi032 ceph-mon[27388]: pgmap v1622: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:31.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:31 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:31 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:31 smithi032 ceph-mon[27388]: 2.4e scrub starts 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:31 smithi032 ceph-mon[27388]: 2.4e scrub ok 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1232479134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:31 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:31 smithi204 ceph-mon[30933]: pgmap v1622: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:31 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:31 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:25:31.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:31 smithi204 ceph-mon[30933]: 2.4e scrub starts 2024-09-16T05:25:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:31 smithi204 ceph-mon[30933]: 2.4e scrub ok 2024-09-16T05:25:31.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1232479134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:31.953 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:31.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:31 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:31 smithi196 ceph-mon[30802]: pgmap v1622: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.4 MiB/s wr, 12 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:31 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:25:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:31 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:25:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:31 smithi196 ceph-mon[30802]: 2.4e scrub starts 2024-09-16T05:25:31.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:31 smithi196 ceph-mon[30802]: 2.4e scrub ok 2024-09-16T05:25:31.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1232479134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:32.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:32.107 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:32.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:32.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:32.232 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:32.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:32.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:32.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:32.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:32.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:32.427 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.1/config 2024-09-16T05:25:32.431 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:32.433 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:32.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:32.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:32.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:32.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:32.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:32.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:32.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:32 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:32.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:32 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:25:32.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:32 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:25:32.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:32.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:32.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:32.871 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:32.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:32.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:32 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:32.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:32 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:25:32.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:32 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:25:32.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:32.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:32 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:32.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:32 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:25:32.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:32 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:25:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:32.997 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:32.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:33.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:33.125 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:33.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:33.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:33.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:33.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:33.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:33.379 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:33.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:33.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:33.507 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:33.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:33.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:33.635 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:33.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:33.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:33.763 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:33.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:33.788 INFO:teuthology.orchestra.run.smithi032.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:25:33.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:33.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:33.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:33.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:33 smithi032 ceph-mon[27388]: pgmap v1623: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 841 KiB/s wr, 4 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:33.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:33 smithi032 ceph-mon[27388]: 2.1a scrub starts 2024-09-16T05:25:33.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:33 smithi032 ceph-mon[27388]: 2.1a scrub ok 2024-09-16T05:25:33.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:33 smithi032 ceph-mon[27388]: 1.1e scrub starts 2024-09-16T05:25:33.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:33 smithi032 ceph-mon[27388]: 1.1e scrub ok 2024-09-16T05:25:33.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:33 smithi204 ceph-mon[30933]: pgmap v1623: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 841 KiB/s wr, 4 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:33.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:33 smithi204 ceph-mon[30933]: 2.1a scrub starts 2024-09-16T05:25:33.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:33 smithi204 ceph-mon[30933]: 2.1a scrub ok 2024-09-16T05:25:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:33 smithi204 ceph-mon[30933]: 1.1e scrub starts 2024-09-16T05:25:33.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:33 smithi204 ceph-mon[30933]: 1.1e scrub ok 2024-09-16T05:25:33.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:33 smithi196 ceph-mon[30802]: pgmap v1623: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 841 KiB/s wr, 4 op/s; 27/222 objects degraded (12.162%) 2024-09-16T05:25:33.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:33 smithi196 ceph-mon[30802]: 2.1a scrub starts 2024-09-16T05:25:33.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:33 smithi196 ceph-mon[30802]: 2.1a scrub ok 2024-09-16T05:25:33.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:33 smithi196 ceph-mon[30802]: 1.1e scrub starts 2024-09-16T05:25:33.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:33 smithi196 ceph-mon[30802]: 1.1e scrub ok 2024-09-16T05:25:33.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:34.017 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:34.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:34.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:34.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:34.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:34.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:34.273 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:34.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:34.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:34.401 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:34.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:34.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:25:34.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:34.526 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:34.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:34.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:34.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:34.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:34.742 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:34.742 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:34.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:34.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:34.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:34.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:34.908 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:34.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:34 smithi032 ceph-mon[27388]: 2.56 scrub starts 2024-09-16T05:25:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:34 smithi032 ceph-mon[27388]: 2.56 scrub ok 2024-09-16T05:25:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:34 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:25:34.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:34 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:25:34.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:34 smithi204 ceph-mon[30933]: 2.56 scrub starts 2024-09-16T05:25:34.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:34 smithi204 ceph-mon[30933]: 2.56 scrub ok 2024-09-16T05:25:34.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:34 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:25:34.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:34 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:25:34.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:34 smithi196 ceph-mon[30802]: 2.56 scrub starts 2024-09-16T05:25:34.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:34 smithi196 ceph-mon[30802]: 2.56 scrub ok 2024-09-16T05:25:34.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:34 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:25:34.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:34 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:25:35.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:35.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:35.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:35.137 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:35.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:35.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:35.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:35.290 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:35.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:35.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:35.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:35.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:35.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:35.543 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:35.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:35.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:35.671 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:35.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:35.699 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:35.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:35.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:35.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:35.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:35 smithi032 ceph-mon[27388]: pgmap v1624: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:35.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:35 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 110 sec, osd.6 has slow ops) 2024-09-16T05:25:35.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:35 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:25:35.830 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:35 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:25:35.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:35.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:35 smithi204 ceph-mon[30933]: pgmap v1624: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:35.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:35 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 110 sec, osd.6 has slow ops) 2024-09-16T05:25:35.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:35 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:25:35.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:35 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:25:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:35.929 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:35.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:35.966 INFO:teuthology.orchestra.run.smithi032.stdout: marking collection for removal 2024-09-16T05:25:35.966 INFO:teuthology.orchestra.run.smithi032.stdout:setting '_remove' omap key 2024-09-16T05:25:35.967 INFO:teuthology.orchestra.run.smithi032.stdout:finish_remove_pgs 1.b_head removing 1.b 2024-09-16T05:25:35.967 INFO:teuthology.orchestra.run.smithi032.stdout:Remove successful 2024-09-16T05:25:35.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:35 smithi196 ceph-mon[30802]: pgmap v1624: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:35.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:35 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 110 sec, osd.6 has slow ops) 2024-09-16T05:25:35.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:35 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:25:35.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:35 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:25:36.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:36.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:36.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:36.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:36.182 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:36.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:36.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:36.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:36.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:36.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:36.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:36.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:36.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:25:36.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:36.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:36.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:36.637 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:25:36.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:36.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:36.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:36.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:36.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:36.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3262114626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:36.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3262114626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:36.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:36.964 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:36.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:36.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3262114626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:37.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:37.087 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:37.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:37.131 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:37.132 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:37.132 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:37.154 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:37.175 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:37.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:37.196 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:37.216 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:37.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:37.218 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:37.240 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:37.241 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:37.241 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:25:37.263 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:37.263 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:37.285 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:37.307 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:37.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:37.328 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:37.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:37.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:37.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:37.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:37.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:37.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:37.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:37.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:37.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:37.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:37.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:37.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:37.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:37 smithi032 ceph-mon[27388]: pgmap v1625: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:37 smithi032 ceph-mon[27388]: 2.2b scrub starts 2024-09-16T05:25:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:37 smithi032 ceph-mon[27388]: 2.2b scrub ok 2024-09-16T05:25:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:37 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3262114626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:37.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:37 smithi032 ceph-mon[27388]: osdmap e971: 12 total, 10 up, 12 in 2024-09-16T05:25:37.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1955061563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:37.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:37 smithi204 ceph-mon[30933]: pgmap v1625: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:37.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:37 smithi204 ceph-mon[30933]: 2.2b scrub starts 2024-09-16T05:25:37.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:37 smithi204 ceph-mon[30933]: 2.2b scrub ok 2024-09-16T05:25:37.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:37 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:37.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3262114626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:37.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:37 smithi204 ceph-mon[30933]: osdmap e971: 12 total, 10 up, 12 in 2024-09-16T05:25:37.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1955061563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:37.954 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:37 smithi196 ceph-mon[30802]: pgmap v1625: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:37.954 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:37 smithi196 ceph-mon[30802]: 2.2b scrub starts 2024-09-16T05:25:37.954 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:37 smithi196 ceph-mon[30802]: 2.2b scrub ok 2024-09-16T05:25:37.954 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:37 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:37.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3262114626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:37.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:37 smithi196 ceph-mon[30802]: osdmap e971: 12 total, 10 up, 12 in 2024-09-16T05:25:37.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1955061563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:37.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:37.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:37.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:38.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.7 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-16T05:25:38.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:38.106 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:38.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:38.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:38.230 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:38.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:38.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:37 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:37.953+0000 7f6af1ad6640 -1 osd.6 971 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:38.248 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.7/config 2024-09-16T05:25:38.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:38.358 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:38.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:38.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:38.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:38.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:38.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:38.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:38.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:38.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:38.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:38.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:38.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:38.747 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:38.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:38.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:38.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:38.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:38.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:38.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:39.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:39.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:39.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:39.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:39.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:39.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:38 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:38.980+0000 7f6af1ad6640 -1 osd.6 971 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:39.286 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:25:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:25:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:25:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:25:39.286 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:25:39.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:39.353 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:39.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:39.402 INFO:teuthology.orchestra.run.smithi196.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:25:39.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:39.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:39.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:39.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:39.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:25:39.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:39.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:39.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:39.734 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:39.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:39.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:39.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:39.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:39.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:39 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:39.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:39 smithi032 ceph-mon[27388]: pgmap v1627: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:39.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:39 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:39.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:39 smithi204 ceph-mon[30933]: pgmap v1627: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:39.954 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:39 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:39.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:39 smithi196 ceph-mon[30802]: pgmap v1627: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:39.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:39.983 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:39.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:40.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:25:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:40.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:40.109 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:40.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:40.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:40.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:40.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:40.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:39 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:39.953+0000 7f6af1ad6640 -1 osd.6 972 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:40.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:40.362 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:40.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:40.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:40.490 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:40.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:40.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:40.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:40.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:40.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:2.8 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:2.4f 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:2.40 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:2.3c 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:2.2e 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:1.9 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:2.33 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:1.4 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:2.43 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:1.d 2024-09-16T05:25:40.710 INFO:teuthology.orchestra.run.smithi196.stdout:2.20 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.26 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.2d 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:1.e 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.1c 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.3f 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:1.1a 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.0 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.1d 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.46 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.1f 2024-09-16T05:25:40.711 INFO:teuthology.orchestra.run.smithi196.stdout:2.27 2024-09-16T05:25:40.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:40.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:40.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:40.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:40.865 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:40.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:40 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:25:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:40 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:25:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:40 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:40.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:40 smithi032 ceph-mon[27388]: osdmap e972: 12 total, 10 up, 12 in 2024-09-16T05:25:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:40 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:25:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:40 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:25:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:40 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:40.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:40 smithi204 ceph-mon[30933]: osdmap e972: 12 total, 10 up, 12 in 2024-09-16T05:25:40.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:40.979 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:40 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:25:40.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:40 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:25:40.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:40 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:40.980 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:40 smithi196 ceph-mon[30802]: osdmap e972: 12 total, 10 up, 12 in 2024-09-16T05:25:40.994 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:40.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4629 2.1e 2:7b642727:::smithi20457400-40:head [copy-from ver 511] snapc bf=[bf,be] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 479.30288676399999, 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01647931, 2024-09-16T05:25:41.058 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4629 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752526+0000", 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.894e-06 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.060 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752536+0000", 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752555+0000", 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8907999999999999e-05 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752564+0000", 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:25:41.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752638+0000", 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3343999999999994e-05 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752646+0000", 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7690000000000002e-06 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.062 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.063 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754646+0000", 2024-09-16T05:25:41.063 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001999331 2024-09-16T05:25:41.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.063 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754740+0000", 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3778999999999998e-05 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.764207+0000", 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094672899999999997 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768754+0000", 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045470199999999997 2024-09-16T05:25:41.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768860+0000", 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010586600000000001 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768948+0000", 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8503000000000003e-05 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.065 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.066 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.769004+0000", 2024-09-16T05:25:41.066 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5362000000000003e-05 2024-09-16T05:25:41.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4630 2.1e 2:7b642727:::smithi20457400-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 479.302829014, 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015483190000000001, 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4630 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:25:41.068 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.34e-07 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752584+0000", 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.956e-06 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752591+0000", 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2580000000000001e-06 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752594+0000", 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6830000000000001e-06 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752703+0000", 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108818 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752708+0000", 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.561e-06 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.070 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754913+0000", 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022055680000000002 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754965+0000", 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2222000000000002e-05 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767827+0000", 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012861097 2024-09-16T05:25:41.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767929+0000", 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010275799999999999 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768065+0000", 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013593500000000001 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:142 2.1e e571/291 ondisk, result = 0)", 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 419.291252859, 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55541789799999997, 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.074 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.074 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764159+0000", 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764162+0000", 2024-09-16T05:25:41.075 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599999999999998e-06 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319278+0000", 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55511595899999999 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319296+0000", 2024-09-16T05:25:41.076 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7900000000000001e-05 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319440+0000", 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143175 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319450+0000", 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0705e-05 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.077 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319576+0000", 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012584299999999999 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5587 2.34 e583/278 2:2d3f13f8:::smithi20457400-7:head v 583'466, mlcod=583'463)", 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 408.24830650799998, 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14534661199999999, 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.078 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:25:41.079 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807109+0000", 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2030000000000002e-06 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951917+0000", 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14480800699999999 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.080 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951927+0000", 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0115000000000001e-05 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951990+0000", 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2689999999999998e-05 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952011+0000", 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.084e-05 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.081 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952413+0000", 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040190199999999999 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952451+0000", 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8302e-05 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:383 2.29 e617/253 2:9662c29c:::smithi20457400-9:d8 v 617'830, mlcod=616'824)", 2024-09-16T05:25:41.082 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 374.27261832699998, 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038031031999999999, 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.083 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782794+0000", 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3500000000000005e-07 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782801+0000", 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1210000000000001e-06 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.084 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819928+0000", 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037126740999999998 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819938+0000", 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0845e-05 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820001+0000", 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2786999999999994e-05 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820023+0000", 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.213e-05 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820771+0000", 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074818699999999998 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820824+0000", 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2286e-05 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.086 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:199 2.1e e714/291 ondisk, result = 0)", 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 277.24320221200003, 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222874399, 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.087 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2099999999999997e-07 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.088 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812213+0000", 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3859999999999999e-06 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034914+0000", 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222700804 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034920+0000", 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6579999999999996e-06 2024-09-16T05:25:41.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034968+0000", 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7833000000000001e-05 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034975+0000", 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0400000000000004e-06 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.035083+0000", 2024-09-16T05:25:41.090 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108157 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7414 2.5a 2:5ae26c04:::smithi20457400-32:head [write 1423159~486356] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 236.22323004, 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010834233, 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.091 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7414 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:25:41.092 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832183+0000", 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6589999999999999e-06 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832584+0000", 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040119599999999999 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.093 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832598+0000", 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3736e-05 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832607+0000", 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1870000000000003e-06 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832655+0000", 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8043000000000001e-05 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832725+0000", 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0136999999999996e-05 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832832+0000", 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010700399999999999 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834013+0000", 2024-09-16T05:25:41.095 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011807479999999999 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842930+0000", 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0089172439999999995 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842977+0000", 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7547999999999999e-05 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.096 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.843015+0000", 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7731000000000002e-05 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:7414 2.5a e755/742 ondisk, result = 0)", 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 236.22120039999999, 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088119329999999992, 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.097 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.098 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834212+0000", 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2699999999999998e-07 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834216+0000", 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2139999999999998e-06 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842854+0000", 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086381679999999999 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.099 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842859+0000", 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842913+0000", 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3235000000000001e-05 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842922+0000", 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.431e-06 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.843023+0000", 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100485 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7416 2.5a 2:5ae26c04:::smithi20457400-32:head [setxattr _header (58) in=7b,truncate 3311210] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 236.214790096, 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.075538587000000004, 2024-09-16T05:25:41.101 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7416 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840622+0000", 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6400000000000001e-07 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840626+0000", 2024-09-16T05:25:41.103 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8589999999999998e-06 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840639+0000", 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2982999999999999e-05 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840644+0000", 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.57e-06 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.104 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.841987+0000", 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013431599999999999 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842059+0000", 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1450000000000002e-05 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842166+0000", 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107421 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:25:41.105 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842788+0000", 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062151399999999999 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916025+0000", 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073237675000000002 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916092+0000", 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6631999999999995e-05 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.106 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916160+0000", 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7459000000000006e-05 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7417 2.5a 2:5ae26c04:::smithi20457400-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:25:41.107 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 236.21473381499999, 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.075671839000000005, 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7417 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.108 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840678+0000", 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9800000000000003e-07 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840682+0000", 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9310000000000001e-06 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840695+0000", 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2672e-05 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840705+0000", 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0498e-05 2024-09-16T05:25:41.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842425+0000", 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001719497 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842530+0000", 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105824 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916202+0000", 2024-09-16T05:25:41.111 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073671367000000001 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916273+0000", 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1316000000000006e-05 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916349+0000", 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5935999999999999e-05 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.112 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:7416 2.5a e755/742 ondisk, result = 0)", 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 236.21245877300001, 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073216748999999998, 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:25:41.113 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842953+0000", 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4000000000000002e-07 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842956+0000", 2024-09-16T05:25:41.114 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-09-16T05:25:41.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.115 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.915928+0000", 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.072972180999999997 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.915937+0000", 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3689999999999996e-06 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916004+0000", 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6836999999999997e-05 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916017+0000", 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3200000000000001e-05 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916169+0000", 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000151512 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:251 2.29 e762/745 ondisk, result = 0)", 2024-09-16T05:25:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 229.13293655199999, 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.502823558, 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922475+0000", 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3000000000000001e-07 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922478+0000", 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.18e-06 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425018+0000", 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.50254011200000004 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425039+0000", 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0832999999999999e-05 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425166+0000", 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012726 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425177+0000", 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0217999999999999e-05 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425298+0000", 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121425 2024-09-16T05:25:41.121 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.7.0:1425 2.2e e786/278 2:75dc7feb:::smithi20457400-28:13f v 786'738, mlcod=781'732)", 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 205.20492422500001, 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72979714900000003, 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.122 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850493+0000", 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1120000000000001e-06 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579433+0000", 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72893962899999998 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579449+0000", 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6059999999999999e-05 2024-09-16T05:25:41.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579568+0000", 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118911 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579618+0000", 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0436000000000003e-05 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.580252+0000", 2024-09-16T05:25:41.125 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063313400000000004 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.580284+0000", 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2381e-05 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.2e,QREQUEST,e:797,from:7,reservation_nonce:8)", 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 192.71300940699999, 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27875738500000002, 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342404+0000", 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9300000000000002e-06 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342412+0000", 2024-09-16T05:25:41.128 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6319999999999997e-06 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.620932+0000", 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27851996800000001 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.620943+0000", 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0892e-05 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.129 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621083+0000", 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013946999999999999 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621110+0000", 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7934000000000001e-05 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621159+0000", 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8559e-05 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:8050 2.2e e804/278 2:75dc7feb:::smithi20457400-28:head v 804'753, mlcod=804'750)", 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 187.10886638700001, 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32122192599999999, 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9999999999999997e-07 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946550+0000", 2024-09-16T05:25:41.132 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4739999999999999e-06 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.266951+0000", 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32040173 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.266959+0000", 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0199999999999994e-06 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267034+0000", 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4408999999999999e-05 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267065+0000", 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1538000000000002e-05 2024-09-16T05:25:41.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267727+0000", 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066189099999999996 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267767+0000", 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9264000000000002e-05 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:690 2.29 e855/767 2:96dc17ef:::smithi20457400-10:165 v 855'1417, mlcod=852'1410)", 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 136.180716526, 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72853946700000005, 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.137 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2500000000000005e-07 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874701+0000", 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8100000000000003e-06 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.601981+0000", 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72728016900000003 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.601998+0000", 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6753e-05 2024-09-16T05:25:41.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.602123+0000", 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012546000000000001 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.602147+0000", 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3597000000000001e-05 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.140 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.603190+0000", 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001043228 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.603234+0000", 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3724999999999999e-05 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.7.0:1618 2.2e e936/278 2:75dc7feb:::smithi20457400-28:18d v 936'854, mlcod=929'851)", 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:46.301199+0000", 2024-09-16T05:25:41.141 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 54.754212451000001, 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094076890000000003, 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301199+0000", 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301199+0000", 2024-09-16T05:25:41.142 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301199+0000", 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8400000000000001e-07 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301205+0000", 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8100000000000003e-06 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.143 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301245+0000", 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0435999999999997e-05 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301261+0000", 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5943999999999998e-05 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301336+0000", 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4931999999999993e-05 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301366+0000", 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0236e-05 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.310549+0000", 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0091832730000000005 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.310606+0000", 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6873999999999998e-05 2024-09-16T05:25:41.145 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:9642 2.34 e947/278 2:2cffa4d6:::smithi20457400-50:head v 947'912, mlcod=941'908)", 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:58.381234+0000", 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 42.674177401999998, 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.38724071399999999, 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.146 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.381234+0000", 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.381234+0000", 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.147 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.381234+0000", 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.381239+0000", 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.767638+0000", 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.38639962799999999 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.767646+0000", 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7619999999999995e-06 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.767715+0000", 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8368999999999995e-05 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.767743+0000", 2024-09-16T05:25:41.149 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8884999999999999e-05 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.768441+0000", 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069796699999999997 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.768474+0000", 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2907999999999997e-05 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:9800 2.29 e964/767 2:94d33c56:::smithi20457400-8:head v 964'1473, mlcod=951'1470)", 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:25:15.473346+0000", 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 25.582065415999999, 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026575290000000001, 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473346+0000", 2024-09-16T05:25:41.151 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473346+0000", 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473346+0000", 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8400000000000001e-07 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.152 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473351+0000", 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2689999999999999e-06 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473385+0000", 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4236000000000002e-05 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473396+0000", 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0644e-05 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.153 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473472+0000", 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6334000000000005e-05 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473501+0000", 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8909999999999999e-05 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.475958+0000", 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024570640000000001 2024-09-16T05:25:41.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.476003+0000", 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4888000000000001e-05 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.58,RELEASE,e:970,from:9,reservation_nonce:0)", 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:25:29.257128+0000", 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 11.798283356000001, 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00027914299999999998, 2024-09-16T05:25:41.155 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257128+0000", 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257128+0000", 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257128+0000", 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.35e-07 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257130+0000", 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9319999999999999e-06 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257199+0000", 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.9552000000000001e-05 2024-09-16T05:25:41.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257205+0000", 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3090000000000003e-06 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257372+0000", 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00016751400000000001 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257391+0000", 2024-09-16T05:25:41.158 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8508999999999998e-05 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:29.257407+0000", 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6192e-05 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:25:41.159 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:25:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:41.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:41.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:41.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:40 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:40.978+0000 7f6af1ad6640 -1 osd.6 972 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:41.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:41.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:41.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:41.332 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:41.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:41.414 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:41.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:41.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:41.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:41.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:41.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:41.650 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:25:41.663 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:41.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:41.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:41.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:41.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:41.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:41 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:41.879 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:41 smithi196 ceph-mon[30802]: pgmap v1629: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:41.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2603797236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:41.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:41.908 INFO:tasks.thrashosds.thrasher:Moving pg 1.b from osd.1 to osd.7 2024-09-16T05:25:41.908 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi032.front.sepia.ceph.com to ubuntu@smithi196.front.sepia.ceph.com 2024-09-16T05:25:41.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-09-16T05:25:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:41.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:41.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:41 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:41 smithi032 ceph-mon[27388]: pgmap v1629: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:41.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2603797236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:41.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:41 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:41.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:41 smithi204 ceph-mon[30933]: pgmap v1629: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:41.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2603797236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:41.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2024-09-16T05:25:41.979 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-09-16T05:25:41.997 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.cG9EQ7SRA0 2024-09-16T05:25:41.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/exp.1.b.1 /tmp/tmp.cG9EQ7SRA0 2024-09-16T05:25:42.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:25:42.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:42.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:42.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:42.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.cG9EQ7SRA0 2024-09-16T05:25:42.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:42.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:42.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:42.205 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.cG9EQ7SRA0 is 9KB 2024-09-16T05:25:42.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:41 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:41.941+0000 7f6af1ad6640 -1 osd.6 973 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:42.254 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.cG9EQ7SRA0 2024-09-16T05:25:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:42.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:42.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv /var/log/ceph/exp.1.b.1 /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/exp.1.b.1 2024-09-16T05:25:42.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -n osd.7 --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.b.1 2024-09-16T05:25:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:42.332 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:42.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:42.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:42.457 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:42.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:42.542 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/osd.7/config 2024-09-16T05:25:42.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:42.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:42.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:42.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:42.708 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:42.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:42.768 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:25:42.768 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:25:42.768 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:25:42.768 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:25:42.810 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:42.836 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:42.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:42 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:42 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:25:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:42 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:25:42.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:42 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:42.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2603797236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:42.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:42 smithi032 ceph-mon[27388]: osdmap e973: 12 total, 10 up, 12 in 2024-09-16T05:25:42.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:42 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:42.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:42 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:25:42.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:42 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:25:42.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:42 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:42.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2603797236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:42.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:42 smithi204 ceph-mon[30933]: osdmap e973: 12 total, 10 up, 12 in 2024-09-16T05:25:42.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:42.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:42 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:42.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:42 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:25:42.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:42 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:25:42.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:42 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:42.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2603797236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:42.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:42 smithi196 ceph-mon[30802]: osdmap e973: 12 total, 10 up, 12 in 2024-09-16T05:25:42.961 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:42.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:43.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:43.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:43.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:43.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:43.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:43.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:43.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:42 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:42.989+0000 7f6af1ad6640 -1 osd.6 973 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:43.301 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:43.302 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:43.302 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:43.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:43.323 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:43.323 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:43.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:43.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:43.344 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:43.344 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:43.365 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:43.366 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:43.386 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:43.387 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:43.407 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:25:43.430 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:43.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:43.452 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:43.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:43.474 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:43.474 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:43.496 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:43.496 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:43.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:43.583 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:43.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:43.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:43.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:43.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:43.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:43.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:43.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:43 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:43 smithi032 ceph-mon[27388]: pgmap v1631: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:43.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2532602543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:43 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:43 smithi204 ceph-mon[30933]: pgmap v1631: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:43.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2532602543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:43.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:43.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:43 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:43.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:43 smithi196 ceph-mon[30802]: pgmap v1631: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:43.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2532602543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:43.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:43.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:44.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:44.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:44.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:44.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:44.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:44.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:44.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:44.271 INFO:teuthology.orchestra.run.smithi196.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-16T05:25:44.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:44.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:44.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:44.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:44.462 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:44.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:44.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:44.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:25:44.590 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:44.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:44.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:44.717 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:44.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:44.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:44.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:44.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:44.880 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:44 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:44.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:44 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:44.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:44 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:44.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:44.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:44.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:45.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:45.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:25:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:45.094 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:45.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:45.152 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:45.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:45.223 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:45.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:45.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:45.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:45.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:45.350 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:45.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:45.470 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:45.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:45.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:45.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:45.669 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:45.669 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:45.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:45.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:45.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:45.730 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:45 smithi032 ceph-mon[27388]: pgmap v1632: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:45.730 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:45 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 125 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:45.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:45.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:45.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:45.885 INFO:teuthology.orchestra.run.smithi196.stdout:get_pg_num_history pg_num_history pg_num_history(e964 pg_nums {1={104=11,401=21,685=31},2={320=76,435=86,737=96}} deleted_pools ) 2024-09-16T05:25:45.885 INFO:teuthology.orchestra.run.smithi196.stdout:Importing pgid 1.b 2024-09-16T05:25:45.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:45 smithi196 ceph-mon[30802]: pgmap v1632: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:45.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:45 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 125 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:45.886 INFO:teuthology.orchestra.run.smithi196.stdout:write_pg epoch 837 info 1.b( v 35'139 (0'0,35'139] local-lis/les=795/796 n=0 ec=401/15 lis/c=795/795 les/c/f=796/796/0 sis=795) 2024-09-16T05:25:45.886 INFO:teuthology.orchestra.run.smithi196.stdout:Import successful 2024-09-16T05:25:45.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:45 smithi204 ceph-mon[30933]: pgmap v1632: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:45.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:45 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 125 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:45.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:45.973 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:45.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:46.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:46.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:46.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:46.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:46.225 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:46.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:46.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:25:46.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:25:46.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:46.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:46.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:46.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:46.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:46.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:46.482 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:46.578 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:46.602 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:46.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:46.666 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:25:46.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:46.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:46.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:46.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:46.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:46.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:46.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/717426083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:46.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/717426083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:46.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:46.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:46.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:46.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/717426083' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:25:47.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/exp.1.b.1 2024-09-16T05:25:47.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:47.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/exp.1.b.1 2024-09-16T05:25:47.109 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:47.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:47.146 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:47.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:47.236 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:47.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:47.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:47.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:47.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:47.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:47.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:47.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:47.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:47.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:47.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:47.644 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:47.644 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:47.682 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:47.683 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:47.683 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:47.703 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:47.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:47.724 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:47.724 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:47.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:47.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:47.746 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:47.746 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:47.767 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:47.767 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:47.788 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:47.789 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:47.789 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:25:47.810 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:47.811 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:47.833 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:47.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:47.855 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:47.855 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:47.856 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:47.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:47.877 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:47.877 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:47 smithi032 ceph-mon[27388]: pgmap v1633: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:47 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/717426083' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:47.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:47 smithi032 ceph-mon[27388]: osdmap e974: 12 total, 10 up, 12 in 2024-09-16T05:25:47.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:47 smithi204 ceph-mon[30933]: pgmap v1633: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:47.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:47 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:47.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/717426083' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:47.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:47 smithi204 ceph-mon[30933]: osdmap e974: 12 total, 10 up, 12 in 2024-09-16T05:25:47.957 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:47 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:47.908+0000 7f6af1ad6640 -1 osd.6 974 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:47.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:47 smithi196 ceph-mon[30802]: pgmap v1633: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:47.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:47 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:47.957 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/717426083' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:25:47.958 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:47 smithi196 ceph-mon[30802]: osdmap e974: 12 total, 10 up, 12 in 2024-09-16T05:25:47.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:47.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:47.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:48.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:48.105 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:48.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:48.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:48.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:48.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:48.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:48.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:25:48.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:48.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:48.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:48.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:48.487 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:48.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:48.496 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:48.538 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:48.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:48.614 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:48.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:48.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:48.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:48.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:48.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:48.840 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:48.866 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:48.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:48.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2266922826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:48.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2266922826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:48.953 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2266922826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:48.953 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:48 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:48.905+0000 7f6af1ad6640 -1 osd.6 974 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:48.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:48.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:48.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:49.062 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:25:49.062 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:25:49.063 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:25:49.063 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:25:49.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:49.117 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:49.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:49.219 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:49.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:49.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:49.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:49.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:49.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:49.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:25:49.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:49.496 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:49.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:49.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:49.879 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:49.879 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:49 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:49.879 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:49 smithi032 ceph-mon[27388]: 2.39 scrub starts 2024-09-16T05:25:49.879 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:49 smithi032 ceph-mon[27388]: 2.39 scrub ok 2024-09-16T05:25:49.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:49 smithi032 ceph-mon[27388]: pgmap v1635: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:49.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:49 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 125 sec, osd.6 has slow ops) 2024-09-16T05:25:49.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:25:49.906 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:49.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:49.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:49 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:49.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:49 smithi204 ceph-mon[30933]: 2.39 scrub starts 2024-09-16T05:25:49.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:49 smithi204 ceph-mon[30933]: 2.39 scrub ok 2024-09-16T05:25:49.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:49 smithi204 ceph-mon[30933]: pgmap v1635: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:49.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:49 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 125 sec, osd.6 has slow ops) 2024-09-16T05:25:49.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:49 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:49.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:49 smithi196 ceph-mon[30802]: 2.39 scrub starts 2024-09-16T05:25:49.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:49 smithi196 ceph-mon[30802]: 2.39 scrub ok 2024-09-16T05:25:49.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:49 smithi196 ceph-mon[30802]: pgmap v1635: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:49.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:49 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 125 sec, osd.6 has slow ops) 2024-09-16T05:25:50.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:50.029 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:50.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:50.125 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:50.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:50.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:50.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:50.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:49 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:49.949+0000 7f6af1ad6640 -1 osd.6 974 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:50.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:50.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:50.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:50.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:50.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:50.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:50.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:25:50.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:50.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:50.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:50.628 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:25:50.628 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:25:50.628 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-09-16T05:25:50.628 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-09-16T05:25:50.628 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:25:50.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:50.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:50.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:50.672 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:25:50.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:50.782 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:50.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:50.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:50.908 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:50.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:50 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:50 smithi032 ceph-mon[27388]: 2.5e scrub starts 2024-09-16T05:25:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:50 smithi032 ceph-mon[27388]: 2.5e scrub ok 2024-09-16T05:25:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:50 smithi032 ceph-mon[27388]: 2.32 scrub starts 2024-09-16T05:25:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:50 smithi032 ceph-mon[27388]: 2.32 scrub ok 2024-09-16T05:25:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:50 smithi032 ceph-mon[27388]: 2.3b scrub starts 2024-09-16T05:25:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:50 smithi032 ceph-mon[27388]: 2.3b scrub ok 2024-09-16T05:25:50.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2769123193' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:50.948 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:50.949 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-mon[30802]: 2.5e scrub starts 2024-09-16T05:25:50.949 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-mon[30802]: 2.5e scrub ok 2024-09-16T05:25:50.949 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-mon[30802]: 2.32 scrub starts 2024-09-16T05:25:50.949 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-mon[30802]: 2.32 scrub ok 2024-09-16T05:25:50.949 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-mon[30802]: 2.3b scrub starts 2024-09-16T05:25:50.949 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-mon[30802]: 2.3b scrub ok 2024-09-16T05:25:50.949 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2769123193' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:51.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:51.035 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:51.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:51.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:51.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:51.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:51.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:50 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:50 smithi204 ceph-mon[30933]: 2.5e scrub starts 2024-09-16T05:25:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:50 smithi204 ceph-mon[30933]: 2.5e scrub ok 2024-09-16T05:25:51.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:50 smithi204 ceph-mon[30933]: 2.32 scrub starts 2024-09-16T05:25:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:50 smithi204 ceph-mon[30933]: 2.32 scrub ok 2024-09-16T05:25:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:50 smithi204 ceph-mon[30933]: 2.3b scrub starts 2024-09-16T05:25:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:50 smithi204 ceph-mon[30933]: 2.3b scrub ok 2024-09-16T05:25:51.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2769123193' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:51.231 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:50 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:50.947+0000 7f6af1ad6640 -1 osd.6 975 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:51.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:25:51.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:51.287 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:51.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:51.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:51.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:51.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:51.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:51.502 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:51.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:51.516 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:51.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:51.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:51.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:51.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:51.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:51.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:51.674 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:51 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:25:51.674 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:51 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:25:51.674 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:51 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:51.674 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:51 smithi032 ceph-mon[27388]: pgmap v1636: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:51.674 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:51 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:51.674 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2769123193' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:51.675 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:51 smithi032 ceph-mon[27388]: osdmap e975: 12 total, 10 up, 12 in 2024-09-16T05:25:51.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:51.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:51.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:51.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:51.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:51 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:25:51.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:51 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:25:51.910 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:51 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:51.910 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:51 smithi196 ceph-mon[30802]: pgmap v1636: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:51.910 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:51 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:51.910 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2769123193' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:51.910 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:51 smithi196 ceph-mon[30802]: osdmap e975: 12 total, 10 up, 12 in 2024-09-16T05:25:51.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:51.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:51.977 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:25:51.977 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:25:51.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:25:51.977 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:25:51.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.977 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3573 2.3 2:c018a787:::smithi20457400-18:head [write 1692526~473837] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 566.10174871900006, 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022224677000000002, 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3573 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.978 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865742+0000", 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:51.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.865743+0000", 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0100000000000001e-06 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866150+0000", 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040748299999999999 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866163+0000", 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2187999999999999e-05 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866168+0000", 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4689999999999996e-06 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866251+0000", 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2911000000000006e-05 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866320+0000", 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8596000000000005e-05 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866403+0000", 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3796999999999997e-05 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867984+0000", 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015806399999999999 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887897+0000", 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019913157000000001 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887934+0000", 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6482000000000001e-05 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:51.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.887967+0000", 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2944000000000002e-05 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3574 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 2403500] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 566.10130916599996, 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039246627999999999, 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:51.984 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3574 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866181+0000", 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866182+0000", 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6300000000000005e-07 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866205+0000", 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2592e-05 2024-09-16T05:25:51.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866222+0000", 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7422e-05 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866231+0000", 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5220000000000007e-06 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867407+0000", 2024-09-16T05:25:51.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001176014 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867514+0000", 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106947 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867661+0000", 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000147396 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867987+0000", 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00032622400000000001 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905316+0000", 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037329222000000002 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905388+0000", 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1896000000000004e-05 2024-09-16T05:25:51.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905428+0000", 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9830000000000003e-05 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3575 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 566.10125116899997, 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039343647000000002, 2024-09-16T05:25:51.990 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3575 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:51.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866239+0000", 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866240+0000", 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7999999999999995e-07 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:51.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866243+0000", 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5770000000000001e-06 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866249+0000", 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.866254+0000", 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-06 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.993 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867919+0000", 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016648609999999999 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.867974+0000", 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4478000000000003e-05 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905464+0000", 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037490460000000003 2024-09-16T05:25:51.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905516+0000", 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1894999999999997e-05 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.905583+0000", 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6946999999999994e-05 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.995 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3580 2.34 2:2cffa4d6:::smithi20457400-50:head [write 1351008~617495] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 566.09095042499996, 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030886027999999999, 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3580 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.996 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876540+0000", 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:51.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.876542+0000", 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8870000000000001e-06 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877114+0000", 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00057156899999999996 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877126+0000", 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2751999999999999e-05 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.998 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877134+0000", 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.072e-06 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0603000000000003e-05 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877240+0000", 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4561000000000002e-05 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:51.999 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877317+0000", 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7002999999999994e-05 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878479+0000", 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011619130000000001 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907362+0000", 2024-09-16T05:25:52.000 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028883039999999999 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907396+0000", 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4807000000000001e-05 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907426+0000", 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9821000000000001e-05 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3581 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2420307] snapc 99=[99] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 566.09033963100001, 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030393973000000001, 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.002 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3581 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877151+0000", 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877152+0000", 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3099999999999996e-07 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877156+0000", 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7589999999999999e-06 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877169+0000", 2024-09-16T05:25:52.004 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3129000000000001e-05 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877175+0000", 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6259999999999997e-06 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877997+0000", 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082123199999999997 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.005 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878081+0000", 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4914999999999996e-05 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878202+0000", 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120207 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878482+0000", 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027995500000000003 2024-09-16T05:25:52.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907462+0000", 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.028980381 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907500+0000", 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7817000000000001e-05 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907545+0000", 2024-09-16T05:25:52.007 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5120999999999997e-05 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:3582 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e436)", 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 566.09030522499995, 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030495267999999999, 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.008 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3582 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877185+0000", 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877186+0000", 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-07 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877189+0000", 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6800000000000002e-06 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877196+0000", 2024-09-16T05:25:52.010 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0650000000000001e-06 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.877202+0000", 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.28e-06 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878412+0000", 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001210205 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.011 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.878470+0000", 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7759e-05 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907572+0000", 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.029102086999999999 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907620+0000", 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8384000000000003e-05 2024-09-16T05:25:52.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:25.907681+0000", 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0244000000000001e-05 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:288 2.16 e446/273 ondisk, result = 0)", 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 555.08377386500001, 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.65473014699999998, 2024-09-16T05:25:52.013 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883717+0000", 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883718+0000", 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3799999999999996e-07 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:36.883722+0000", 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.284e-06 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538184+0000", 2024-09-16T05:25:52.015 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.654462031 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538201+0000", 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6634e-05 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538327+0000", 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012605799999999999 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538338+0000", 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1188e-05 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:16:37.538447+0000", 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109014 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5587 2.34 2:2d3f13f8:::smithi20457400-7:head [setxattr _header (57) in=7b,truncate 3260723] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:25:52.017 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 419.16165227300002, 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.147062895, 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5587 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805839+0000", 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8299999999999997e-07 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.019 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805842+0000", 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6790000000000001e-06 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805846+0000", 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4089999999999996e-06 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805849+0000", 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.191e-06 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.020 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806887+0000", 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010375440000000001 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806953+0000", 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5833000000000005e-05 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807079+0000", 2024-09-16T05:25:52.021 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012666500000000001 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.809689+0000", 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026092860000000002 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952747+0000", 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14305815 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952826+0000", 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9773999999999999e-05 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952901+0000", 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4781000000000001e-05 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5588 2.34 2:2d3f13f8:::smithi20457400-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:25:52.023 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 419.16162580000002, 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14722005799999999, 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5588 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:25:52.024 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.65e-07 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.025 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805867+0000", 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.86e-06 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805872+0000", 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805876+0000", 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0380000000000001e-06 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.026 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807396+0000", 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001520448 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807470+0000", 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3695000000000005e-05 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952937+0000", 2024-09-16T05:25:52.027 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14546690300000001 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953007+0000", 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9801999999999993e-05 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953085+0000", 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8121999999999995e-05 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:534 2.34 e617/278 ondisk, result = 0)", 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 385.15533566400001, 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93956844500000003, 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812156+0000", 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0599999999999999e-07 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812160+0000", 2024-09-16T05:25:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.437e-06 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751436+0000", 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93927549399999999 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751483+0000", 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7117999999999998e-05 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751598+0000", 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115448 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751609+0000", 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1156e-05 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751723+0000", 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113986 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.032 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7360 2.3 2:c018a787:::smithi20457400-18:head [write 443982~796776,stat] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.07053700500001, 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.046574619999999997, 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7360 2024-09-16T05:25:52.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896958+0000", 2024-09-16T05:25:52.034 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3869999999999997e-06 2024-09-16T05:25:52.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.035 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:52.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:52.036 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:52.037 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:52.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897803+0000", 2024-09-16T05:25:52.037 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084549400000000002 2024-09-16T05:25:52.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.037 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.037 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897817+0000", 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3172e-05 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897822+0000", 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7400000000000001e-06 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897880+0000", 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7275999999999998e-05 2024-09-16T05:25:52.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897887+0000", 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0450000000000003e-06 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899506+0000", 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016191300000000001 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899612+0000", 2024-09-16T05:25:52.039 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010592200000000001 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899807+0000", 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019501699999999999 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.900940+0000", 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011327760000000001 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.040 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943442+0000", 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.042502861000000003 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943488+0000", 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5266000000000003e-05 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943528+0000", 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0534000000000002e-05 2024-09-16T05:25:52.041 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7361 2.3 2:c018a787:::smithi20457400-18:head [write 1989231~479018] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.06387212499999, 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040030532000000001, 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.042 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7361 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.043 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903620+0000", 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.885e-06 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904044+0000", 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042330699999999999 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904056+0000", 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2088e-05 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904064+0000", 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3820000000000004e-06 2024-09-16T05:25:52.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904117+0000", 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2352000000000003e-05 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904226+0000", 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109895 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:52.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904403+0000", 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017676899999999999 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.905359+0000", 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00095604999999999996 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943572+0000", 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038213241000000002 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.047 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943611+0000", 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8668999999999999e-05 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943649+0000", 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7894e-05 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.048 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7362 2.3 2:c018a787:::smithi20457400-18:head [write 2959167~284815] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.05962175499999, 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035905712999999999, 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7362 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.049 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907870+0000", 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4200000000000005e-07 2024-09-16T05:25:52.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908124+0000", 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025446300000000002 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908135+0000", 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0964999999999999e-05 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908143+0000", 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5190000000000003e-06 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9714e-05 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.052 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908239+0000", 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6212999999999999e-05 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908346+0000", 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010734999999999999 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909154+0000", 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080830699999999997 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.053 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943689+0000", 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034534978000000001 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943732+0000", 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2558999999999997e-05 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943775+0000", 2024-09-16T05:25:52.054 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2803000000000003e-05 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7363 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 3243982] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.05933345599999, 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035723752999999997, 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.055 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7363 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908158+0000", 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908161+0000", 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1410000000000001e-06 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.057 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908167+0000", 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.674e-06 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908173+0000", 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4149999999999998e-06 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908684+0000", 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051148000000000003 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.058 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908742+0000", 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7358999999999997e-05 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908839+0000", 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6930999999999997e-05 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909158+0000", 2024-09-16T05:25:52.059 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031886399999999997 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943805+0000", 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034647491000000002 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943842+0000", 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7004e-05 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943881+0000", 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8868999999999997e-05 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7364 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 255.05930859599999, 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035867990000000002, 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.061 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7364 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5999999999999999e-07 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908185+0000", 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1600000000000001e-06 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908191+0000", 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6869999999999997e-06 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908196+0000", 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9989999999999999e-06 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909088+0000", 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089121600000000001 2024-09-16T05:25:52.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909143+0000", 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5887000000000002e-05 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943913+0000", 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034769683000000003 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943981+0000", 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.792e-05 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.944050+0000", 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8978000000000003e-05 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:830 2.3 e799/329 ondisk, result = 0)", 2024-09-16T05:25:52.066 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 203.09722586199999, 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.330002407, 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870266+0000", 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.223e-06 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870277+0000", 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0560999999999999e-05 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.199989+0000", 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32971226399999998 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200004+0000", 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5366000000000002e-05 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200139+0000", 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013504000000000001 2024-09-16T05:25:52.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200150+0000", 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0305999999999999e-05 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200267+0000", 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011764699999999999 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9051 2.3 2:c018a787:::smithi20457400-18:head [append 0~0] snapc 189=[189,188,187,186,185,184,181,17e,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e909)", 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:19.162722+0000", 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 92.804768566999996, 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34937753599999999, 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.071 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9051 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162722+0000", 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162722+0000", 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162722+0000", 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.65e-07 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162725+0000", 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1940000000000001e-06 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162735+0000", 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0592e-05 2024-09-16T05:25:52.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162739+0000", 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.591e-06 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162836+0000", 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6895e-05 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:25:52.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162842+0000", 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3589999999999996e-06 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.165934+0000", 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030923370000000001 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.166031+0000", 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6910000000000006e-05 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.166203+0000", 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017222799999999999 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.166680+0000", 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00047637599999999998 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.511963+0000", 2024-09-16T05:25:52.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.345283436 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.512039+0000", 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5852999999999994e-05 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.512100+0000", 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.05e-05 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.077 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9642 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2403731] snapc 198=[16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e947)", 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:58.380051+0000", 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 53.587440074, 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.388745954, 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.078 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9642 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380051+0000", 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380051+0000", 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380051+0000", 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9299999999999998e-07 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380062+0000", 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0917999999999999e-05 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380067+0000", 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7310000000000001e-06 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380070+0000", 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7780000000000001e-06 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381091+0000", 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010207219999999999 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381152+0000", 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1257e-05 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381243+0000", 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0611000000000004e-05 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381707+0000", 2024-09-16T05:25:52.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046380999999999999 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768681+0000", 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.38697416299999998 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768722+0000", 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1124000000000001e-05 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.083 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768796+0000", 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4346999999999998e-05 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9643 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e947)", 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:58.380076+0000", 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 53.587414535999997, 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.38888499500000001, 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.084 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9643 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380076+0000", 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380076+0000", 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380076+0000", 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7099999999999997e-07 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380078+0000", 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.361e-06 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380083+0000", 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2839999999999997e-06 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380087+0000", 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.427e-06 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381622+0000", 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015349999999999999 2024-09-16T05:25:52.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381671+0000", 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9447000000000003e-05 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768830+0000", 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.38715916900000003 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768895+0000", 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4467000000000005e-05 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768961+0000", 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6469000000000003e-05 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9806 2.3b 2:de669526:::smithi20457400-23:head [write 723064~731129] snapc 1a2=[1a2,1a1,1a0,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-09-16T05:25:52.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:25:15.650973+0000", 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 36.31651798, 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034474850000000001, 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9806 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:25:52.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.650973+0000", 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.650973+0000", 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.650974+0000", 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.804e-06 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.091 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651800+0000", 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082599400000000003 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651816+0000", 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6044000000000001e-05 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651824+0000", 2024-09-16T05:25:52.092 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4630000000000004e-06 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651888+0000", 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4407999999999997e-05 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651981+0000", 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3122999999999996e-05 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4", 2024-09-16T05:25:52.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.652156+0000", 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017429300000000001 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.654012+0000", 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001855953 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.654312+0000", 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030017500000000002 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.654371+0000", 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8795000000000002e-05 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.654420+0000", 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9432999999999999e-05 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:25:52.095 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:25:52.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:52.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:52.097 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:52.100 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:52.100 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:52.122 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:52.122 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:52.122 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:25:52.123 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:52.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:52.131 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [1, 7] live_osds: [0, 3, 4, 5, 6, 8, 9, 10, 11, 2] 2024-09-16T05:25:52.131 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:25:52.131 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-16T05:25:52.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:25:52.144 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:52.144 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:52.166 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:52.166 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:51 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:25:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:51 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:25:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:51 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:52.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:51 smithi204 ceph-mon[30933]: pgmap v1636: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:51 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:25:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2769123193' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:25:52.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:51 smithi204 ceph-mon[30933]: osdmap e975: 12 total, 10 up, 12 in 2024-09-16T05:25:52.187 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:52.187 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:52.209 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:52.209 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:52.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:52.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:51 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:51.908+0000 7f6af1ad6640 -1 osd.6 975 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:52.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:52.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:52.348 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:52.374 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:52.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:52.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:52.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:52.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:52.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:52.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:52.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:52.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:25:52.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:52.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:52.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:52.823 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:52.823 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2.48 scrub starts 2024-09-16T05:25:52.823 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2.48 scrub ok 2024-09-16T05:25:52.823 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2.5a scrub starts 2024-09-16T05:25:52.824 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2.5a scrub ok 2024-09-16T05:25:52.824 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:25:52.824 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:25:52.824 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2.3 scrub starts 2024-09-16T05:25:52.824 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: 2.3 scrub ok 2024-09-16T05:25:52.824 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1042075184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:52.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:52.862 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:52.915 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:52.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:52.918 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:52.937 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2.48 scrub starts 2024-09-16T05:25:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2.48 scrub ok 2024-09-16T05:25:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2.5a scrub starts 2024-09-16T05:25:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2.5a scrub ok 2024-09-16T05:25:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:25:52.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:25:52.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2.3 scrub starts 2024-09-16T05:25:52.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: 2.3 scrub ok 2024-09-16T05:25:52.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1042075184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:53.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:53.040 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:53.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:53.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:53.167 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:53.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:53.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2.48 scrub starts 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2.48 scrub ok 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2.5a scrub starts 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2.5a scrub ok 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2.3 scrub starts 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: 2.3 scrub ok 2024-09-16T05:25:53.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1042075184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:53.269 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:53.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:53.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:53.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:53.422 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:53.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:53.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:25:53.507 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:53.507 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:53.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:53.550 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:53.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:53.567 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:25:53.569 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:25:53.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:53.677 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:53.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:53.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:53.801 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:53.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:53.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:53.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:53.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:53 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:53.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:53 smithi032 ceph-mon[27388]: 2.5f scrub starts 2024-09-16T05:25:53.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:53 smithi032 ceph-mon[27388]: 2.5f scrub ok 2024-09-16T05:25:53.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:53 smithi032 ceph-mon[27388]: pgmap v1638: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:53.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:53 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:25:53.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:53 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:25:53.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:53.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:53 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:53 smithi196 ceph-mon[30802]: 2.5f scrub starts 2024-09-16T05:25:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:53 smithi196 ceph-mon[30802]: 2.5f scrub ok 2024-09-16T05:25:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:53 smithi196 ceph-mon[30802]: pgmap v1638: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:53 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:25:53.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:53 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:25:54.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:54.055 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:54.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:54.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:25:54.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:53 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:53 smithi204 ceph-mon[30933]: 2.5f scrub starts 2024-09-16T05:25:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:53 smithi204 ceph-mon[30933]: 2.5f scrub ok 2024-09-16T05:25:54.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:53 smithi204 ceph-mon[30933]: pgmap v1638: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:53 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:25:54.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:53 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:25:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:54.181 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:54.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:54.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:54.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:54.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:54.318 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:54.378 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:54.398 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1638,"stamp":"2024-09-16T05:25:52.265668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":252785218,"num_objects":111,"num_object_clones":66,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":10,"num_read":4724,"num_read_kb":4707814,"num_write":3366,"num_write_kb":1366450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39275,"ondisk_log_size":39275,"up":214,"acting":214,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":254,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1004728,"kb_used_data":388916,"kb_used_omap":27,"kb_used_meta":615204,"kb_avail":1123937096,"statfs":{"total":1151940427776,"available":1150911586304,"internally_reserved":0,"allocated":398249984,"data_stored":442300328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28505,"internal_metadata":629969063},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002789"},"pg_stats":[{"pgid":"2.5f","version":"284'221","reported_seq":3569,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.672898+0000","last_change":"2024-09-16T05:25:18.477192+0000","last_active":"2024-09-16T05:25:46.672898+0000","last_peered":"2024-09-16T05:25:46.672898+0000","last_clean":"2024-09-16T05:25:46.672898+0000","last_became_active":"2024-09-16T05:25:10.429050+0000","last_became_peered":"2024-09-16T05:25:10.429050+0000","last_unstale":"2024-09-16T05:25:46.672898+0000","last_undegraded":"2024-09-16T05:25:46.672898+0000","last_fullsized":"2024-09-16T05:25:46.672898+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":7,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:24:26.972354+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:24:26.972354+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.979141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056402700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.5e","version":"968'598","reported_seq":5086,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678685+0000","last_change":"2024-09-16T05:25:48.994649+0000","last_active":"2024-09-16T05:25:50.678685+0000","last_peered":"2024-09-16T05:25:50.678685+0000","last_clean":"2024-09-16T05:25:50.678685+0000","last_became_active":"2024-09-16T05:21:54.937455+0000","last_became_peered":"2024-09-16T05:21:54.937455+0000","last_unstale":"2024-09-16T05:25:50.678685+0000","last_undegraded":"2024-09-16T05:25:50.678685+0000","last_fullsized":"2024-09-16T05:25:50.678685+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"968'598","last_scrub_stamp":"2024-09-16T05:25:48.994532+0000","last_deep_scrub":"913'571","last_deep_scrub_stamp":"2024-09-16T05:24:23.716116+0000","last_clean_scrub_stamp":"2024-09-16T05:25:48.994532+0000","objects_scrubbed":2,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:09.917458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001338164,"stat_sum":{"num_bytes":2622222,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":33,"num_read_kb":35766,"num_write":28,"num_write_kb":11949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4819607,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.5d","version":"866'1499","reported_seq":5763,"reported_epoch":975,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:50.678686+0000","last_change":"2024-09-16T05:23:37.866923+0000","last_active":"2024-09-16T05:23:36.872091+0000","last_peered":"2024-09-16T05:25:50.678686+0000","last_clean":"2024-09-16T05:23:35.855325+0000","last_became_active":"2024-09-16T05:21:32.830157+0000","last_became_peered":"2024-09-16T05:23:37.866923+0000","last_unstale":"2024-09-16T05:25:50.678686+0000","last_undegraded":"2024-09-16T05:23:37.861087+0000","last_fullsized":"2024-09-16T05:23:37.858832+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'1436","last_scrub_stamp":"2024-09-16T05:22:51.640147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:51.640147+0000","objects_scrubbed":6,"log_size":1499,"log_dups_size":0,"ondisk_log_size":1499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:53.991967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060484479999999997,"stat_sum":{"num_bytes":14532663,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":89,"num_read_kb":77940,"num_write":59,"num_write_kb":23832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[{"shards":"10","objects":6}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5c","version":"968'534","reported_seq":4501,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.273421+0000","last_change":"2024-09-16T05:25:25.192598+0000","last_active":"2024-09-16T05:25:47.273421+0000","last_peered":"2024-09-16T05:25:47.273421+0000","last_clean":"2024-09-16T05:25:47.273421+0000","last_became_active":"2024-09-16T05:21:32.828660+0000","last_became_peered":"2024-09-16T05:21:32.828660+0000","last_unstale":"2024-09-16T05:25:47.273421+0000","last_undegraded":"2024-09-16T05:25:47.273421+0000","last_fullsized":"2024-09-16T05:25:47.273421+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"968'534","last_scrub_stamp":"2024-09-16T05:25:25.192459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:25.192459+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:41.845000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012314089999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":15893,"num_write":28,"num_write_kb":6166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.5b","version":"861'678","reported_seq":4678,"reported_epoch":975,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:50.678272+0000","last_change":"2024-09-16T05:23:37.864464+0000","last_active":"2024-09-16T05:23:36.871791+0000","last_peered":"2024-09-16T05:25:50.678272+0000","last_clean":"2024-09-16T05:23:30.902132+0000","last_became_active":"2024-09-16T05:21:32.829681+0000","last_became_peered":"2024-09-16T05:23:37.864464+0000","last_unstale":"2024-09-16T05:25:50.678272+0000","last_undegraded":"2024-09-16T05:23:37.861042+0000","last_fullsized":"2024-09-16T05:23:37.860945+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"820'660","last_scrub_stamp":"2024-09-16T05:22:52.654103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:52.654103+0000","objects_scrubbed":2,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:53.991951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002530137,"stat_sum":{"num_bytes":3479247,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":52562,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[{"shards":"10","objects":2}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5a","version":"936'1284","reported_seq":5567,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667079+0000","last_change":"2024-09-16T05:25:18.477246+0000","last_active":"2024-09-16T05:25:46.667079+0000","last_peered":"2024-09-16T05:25:46.667079+0000","last_clean":"2024-09-16T05:25:46.667079+0000","last_became_active":"2024-09-16T05:21:32.826884+0000","last_became_peered":"2024-09-16T05:21:32.826884+0000","last_unstale":"2024-09-16T05:25:46.667079+0000","last_undegraded":"2024-09-16T05:25:46.667079+0000","last_fullsized":"2024-09-16T05:25:46.667079+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"932'1280","last_scrub_stamp":"2024-09-16T05:24:42.377144+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:24:42.377144+0000","objects_scrubbed":4,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.885585+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00081264100000000001,"stat_sum":{"num_bytes":5065634,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":61907,"num_write":38,"num_write_kb":8752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.59","version":"182'60","reported_seq":1207,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667708+0000","last_change":"2024-09-16T05:25:20.980153+0000","last_active":"2024-09-16T05:25:46.667708+0000","last_peered":"2024-09-16T05:25:46.667708+0000","last_clean":"2024-09-16T05:25:46.667708+0000","last_became_active":"2024-09-16T05:21:54.931981+0000","last_became_peered":"2024-09-16T05:21:54.931981+0000","last_unstale":"2024-09-16T05:25:46.667708+0000","last_undegraded":"2024-09-16T05:25:46.667708+0000","last_fullsized":"2024-09-16T05:25:46.667708+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:25:20.980001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:20.980001+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:37.556279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056945699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.58","version":"181'57","reported_seq":4348,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192144+0000","last_change":"2024-09-16T05:25:29.257034+0000","last_active":"2024-09-16T05:25:47.192144+0000","last_peered":"2024-09-16T05:25:47.192144+0000","last_clean":"2024-09-16T05:25:47.192144+0000","last_became_active":"2024-09-16T05:21:29.826747+0000","last_became_peered":"2024-09-16T05:21:29.826747+0000","last_unstale":"2024-09-16T05:25:47.192144+0000","last_undegraded":"2024-09-16T05:25:47.192144+0000","last_fullsized":"2024-09-16T05:25:47.192144+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:25:29.256936+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:25:29.256936+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:52.047439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081423900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.57","version":"0'0","reported_seq":1872,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678636+0000","last_change":"2024-09-16T05:25:18.533342+0000","last_active":"2024-09-16T05:25:50.678636+0000","last_peered":"2024-09-16T05:25:50.678636+0000","last_clean":"2024-09-16T05:25:50.678636+0000","last_became_active":"2024-09-16T05:21:29.827377+0000","last_became_peered":"2024-09-16T05:21:29.827377+0000","last_unstale":"2024-09-16T05:25:50.678636+0000","last_undegraded":"2024-09-16T05:25:50.678636+0000","last_fullsized":"2024-09-16T05:25:50.678636+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:24:20.747066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:20.747066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002001409,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.56","version":"958'445","reported_seq":4431,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.273717+0000","last_change":"2024-09-16T05:25:33.225925+0000","last_active":"2024-09-16T05:25:47.273717+0000","last_peered":"2024-09-16T05:25:47.273717+0000","last_clean":"2024-09-16T05:25:47.273717+0000","last_became_active":"2024-09-16T05:21:32.829699+0000","last_became_peered":"2024-09-16T05:21:32.829699+0000","last_unstale":"2024-09-16T05:25:47.273717+0000","last_undegraded":"2024-09-16T05:25:47.273717+0000","last_fullsized":"2024-09-16T05:25:47.273717+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"958'445","last_scrub_stamp":"2024-09-16T05:25:33.225813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:33.225813+0000","objects_scrubbed":3,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:55.031728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013276659999999999,"stat_sum":{"num_bytes":8182314,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":18488,"num_write":40,"num_write_kb":14849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.55","version":"433'507","reported_seq":3330,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667200+0000","last_change":"2024-09-16T05:25:18.477417+0000","last_active":"2024-09-16T05:25:46.667200+0000","last_peered":"2024-09-16T05:25:46.667200+0000","last_clean":"2024-09-16T05:25:46.667200+0000","last_became_active":"2024-09-16T05:21:24.815957+0000","last_became_peered":"2024-09-16T05:21:24.815957+0000","last_unstale":"2024-09-16T05:25:46.667200+0000","last_undegraded":"2024-09-16T05:25:46.667200+0000","last_fullsized":"2024-09-16T05:25:46.667200+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:24:41.350393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:41.350393+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:57.022173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000373099,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.54","version":"958'486","reported_seq":4453,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635270+0000","last_change":"2024-09-16T05:25:18.531442+0000","last_active":"2024-09-16T05:25:47.635270+0000","last_peered":"2024-09-16T05:25:47.635270+0000","last_clean":"2024-09-16T05:25:47.635270+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:25:47.635270+0000","last_undegraded":"2024-09-16T05:25:47.635270+0000","last_fullsized":"2024-09-16T05:25:47.635270+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"932'470","last_scrub_stamp":"2024-09-16T05:24:44.837416+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:24:44.837416+0000","objects_scrubbed":3,"log_size":486,"log_dups_size":0,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:09.297269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00076292799999999996,"stat_sum":{"num_bytes":5664553,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":86504,"num_write":66,"num_write_kb":30716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":911,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.666829+0000","last_change":"2024-09-16T05:25:18.477146+0000","last_active":"2024-09-16T05:25:46.666829+0000","last_peered":"2024-09-16T05:25:46.666829+0000","last_clean":"2024-09-16T05:25:46.666829+0000","last_became_active":"2024-09-16T05:25:10.429082+0000","last_became_peered":"2024-09-16T05:25:10.429082+0000","last_unstale":"2024-09-16T05:25:46.666829+0000","last_undegraded":"2024-09-16T05:25:46.666829+0000","last_fullsized":"2024-09-16T05:25:46.666829+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:24:43.021928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:43.021928+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:57.398868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068708099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":2705,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.273375+0000","last_change":"2024-09-16T05:25:20.148245+0000","last_active":"2024-09-16T05:25:47.273375+0000","last_peered":"2024-09-16T05:25:47.273375+0000","last_clean":"2024-09-16T05:25:47.273375+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:25:47.273375+0000","last_undegraded":"2024-09-16T05:25:47.273375+0000","last_fullsized":"2024-09-16T05:25:47.273375+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:25:20.148158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:20.148158+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:44.828844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000958022,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":2766,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678532+0000","last_change":"2024-09-16T05:25:18.536278+0000","last_active":"2024-09-16T05:25:50.678532+0000","last_peered":"2024-09-16T05:25:50.678532+0000","last_clean":"2024-09-16T05:25:50.678532+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:25:50.678532+0000","last_undegraded":"2024-09-16T05:25:50.678532+0000","last_fullsized":"2024-09-16T05:25:50.678532+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:24:41.782482+0000","last_deep_scrub":"272'115","last_deep_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_clean_scrub_stamp":"2024-09-16T05:24:41.782482+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002780856,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":2692,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.666784+0000","last_change":"2024-09-16T05:25:18.476675+0000","last_active":"2024-09-16T05:25:46.666784+0000","last_peered":"2024-09-16T05:25:46.666784+0000","last_clean":"2024-09-16T05:25:46.666784+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:25:46.666784+0000","last_undegraded":"2024-09-16T05:25:46.666784+0000","last_fullsized":"2024-09-16T05:25:46.666784+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:25:05.365430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:05.365430+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:29.184468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052661799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":2566,"reported_epoch":959,"state":"stale+undersized+peered","last_fresh":"2024-09-16T05:25:10.425570+0000","last_change":"2024-09-16T05:23:37.862596+0000","last_active":"2024-09-16T05:23:36.874931+0000","last_peered":"2024-09-16T05:25:10.425570+0000","last_clean":"2024-09-16T05:21:05.836464+0000","last_became_active":"2024-09-16T05:19:35.789805+0000","last_became_peered":"2024-09-16T05:23:37.862596+0000","last_unstale":"2024-09-16T05:25:16.470786+0000","last_undegraded":"2024-09-16T05:25:10.425570+0000","last_fullsized":"2024-09-16T05:23:37.860001+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:22:33.493608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:33.493608+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:12.153396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021468759999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4e","version":"272'232","reported_seq":4469,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.676167+0000","last_change":"2024-09-16T05:25:30.774404+0000","last_active":"2024-09-16T05:25:50.676167+0000","last_peered":"2024-09-16T05:25:50.676167+0000","last_clean":"2024-09-16T05:25:50.676167+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:25:50.676167+0000","last_undegraded":"2024-09-16T05:25:50.676167+0000","last_fullsized":"2024-09-16T05:25:50.676167+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:25:30.774286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:30.774286+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:34.092281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000666852,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.4d","version":"967'781","reported_seq":5182,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635481+0000","last_change":"2024-09-16T05:25:18.534263+0000","last_active":"2024-09-16T05:25:47.635481+0000","last_peered":"2024-09-16T05:25:47.635481+0000","last_clean":"2024-09-16T05:25:47.635481+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:25:47.635481+0000","last_undegraded":"2024-09-16T05:25:47.635481+0000","last_fullsized":"2024-09-16T05:25:47.635481+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"901'751","last_scrub_stamp":"2024-09-16T05:24:17.609804+0000","last_deep_scrub":"509'500","last_deep_scrub_stamp":"2024-09-16T05:17:50.121236+0000","last_clean_scrub_stamp":"2024-09-16T05:24:17.609804+0000","objects_scrubbed":2,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0035391390000000002,"stat_sum":{"num_bytes":6652548,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":78695,"num_write":112,"num_write_kb":35542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.4c","version":"968'475","reported_seq":4821,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678472+0000","last_change":"2024-09-16T05:25:18.540381+0000","last_active":"2024-09-16T05:25:50.678472+0000","last_peered":"2024-09-16T05:25:50.678472+0000","last_clean":"2024-09-16T05:25:50.678472+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:25:50.678472+0000","last_undegraded":"2024-09-16T05:25:50.678472+0000","last_fullsized":"2024-09-16T05:25:50.678472+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"932'453","last_scrub_stamp":"2024-09-16T05:24:44.796331+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:24:44.796331+0000","objects_scrubbed":2,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:51.515895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.008708538,"stat_sum":{"num_bytes":5025887,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":112,"num_read_kb":129478,"num_write":63,"num_write_kb":21390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":4505,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635294+0000","last_change":"2024-09-16T05:25:18.532786+0000","last_active":"2024-09-16T05:25:47.635294+0000","last_peered":"2024-09-16T05:25:47.635294+0000","last_clean":"2024-09-16T05:25:47.635294+0000","last_became_active":"2024-09-16T05:22:22.832492+0000","last_became_peered":"2024-09-16T05:22:22.832492+0000","last_unstale":"2024-09-16T05:25:47.635294+0000","last_undegraded":"2024-09-16T05:25:47.635294+0000","last_fullsized":"2024-09-16T05:25:47.635294+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:25:07.644216+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:25:07.644216+0000","last_clean_scrub_stamp":"2024-09-16T05:25:07.644216+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:23.318503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001704812,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.4a","version":"968'721","reported_seq":5041,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678483+0000","last_change":"2024-09-16T05:25:27.939801+0000","last_active":"2024-09-16T05:25:50.678483+0000","last_peered":"2024-09-16T05:25:50.678483+0000","last_clean":"2024-09-16T05:25:50.678483+0000","last_became_active":"2024-09-16T05:18:02.890434+0000","last_became_peered":"2024-09-16T05:18:02.890434+0000","last_unstale":"2024-09-16T05:25:50.678483+0000","last_undegraded":"2024-09-16T05:25:50.678483+0000","last_fullsized":"2024-09-16T05:25:50.678483+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"968'721","last_scrub_stamp":"2024-09-16T05:25:27.939688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:27.939688+0000","objects_scrubbed":3,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:31.280600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003041195,"stat_sum":{"num_bytes":7452068,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":102681,"num_write":86,"num_write_kb":34757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":4538,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.675913+0000","last_change":"2024-09-16T05:25:18.531481+0000","last_active":"2024-09-16T05:25:50.675913+0000","last_peered":"2024-09-16T05:25:50.675913+0000","last_clean":"2024-09-16T05:25:50.675913+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:25:50.675913+0000","last_undegraded":"2024-09-16T05:25:50.675913+0000","last_fullsized":"2024-09-16T05:25:50.675913+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:24:26.763617+0000","last_deep_scrub":"245'220","last_deep_scrub_stamp":"2024-09-16T05:24:26.763617+0000","last_clean_scrub_stamp":"2024-09-16T05:24:26.763617+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:52.823521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050694800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.48","version":"961'455","reported_seq":4761,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192183+0000","last_change":"2024-09-16T05:25:18.477406+0000","last_active":"2024-09-16T05:25:47.192183+0000","last_peered":"2024-09-16T05:25:47.192183+0000","last_clean":"2024-09-16T05:25:47.192183+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:25:47.192183+0000","last_undegraded":"2024-09-16T05:25:47.192183+0000","last_fullsized":"2024-09-16T05:25:47.192183+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"915'452","last_scrub_stamp":"2024-09-16T05:24:49.145628+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:24:49.145628+0000","objects_scrubbed":2,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.543232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011004319999999999,"stat_sum":{"num_bytes":5126729,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":95,"num_read_kb":79346,"num_write":78,"num_write_kb":34763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":3324,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.273634+0000","last_change":"2024-09-16T05:25:18.532112+0000","last_active":"2024-09-16T05:25:47.273634+0000","last_peered":"2024-09-16T05:25:47.273634+0000","last_clean":"2024-09-16T05:25:47.273634+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:25:47.273634+0000","last_undegraded":"2024-09-16T05:25:47.273634+0000","last_fullsized":"2024-09-16T05:25:47.273634+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:24:47.954522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_clean_scrub_stamp":"2024-09-16T05:24:47.954522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:59.926289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011606920000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":1538,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.635464+0000","last_change":"2024-09-16T05:25:17.472406+0000","last_active":"2024-09-16T05:25:16.468342+0000","last_peered":"2024-09-16T05:25:47.635464+0000","last_clean":"2024-09-16T05:25:15.462223+0000","last_became_active":"2024-09-16T05:20:20.797682+0000","last_became_peered":"2024-09-16T05:25:17.472406+0000","last_unstale":"2024-09-16T05:25:47.635464+0000","last_undegraded":"2024-09-16T05:25:47.635464+0000","last_fullsized":"2024-09-16T05:25:17.471912+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:25:02.710151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:02.710151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:28.961100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010214530000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.45","version":"305'464","reported_seq":3458,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.273510+0000","last_change":"2024-09-16T05:23:37.860738+0000","last_active":"2024-09-16T05:23:22.845158+0000","last_peered":"2024-09-16T05:25:47.273510+0000","last_clean":"2024-09-16T05:23:21.851499+0000","last_became_active":"2024-09-16T05:21:10.818112+0000","last_became_peered":"2024-09-16T05:23:37.860738+0000","last_unstale":"2024-09-16T05:25:47.273510+0000","last_undegraded":"2024-09-16T05:25:47.273510+0000","last_fullsized":"2024-09-16T05:23:37.858122+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:23:09.592714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:09.592714+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:52.272507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.006563284,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.44","version":"182'89","reported_seq":1819,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.667995+0000","last_change":"2024-09-16T05:23:37.862679+0000","last_active":"2024-09-16T05:23:36.858406+0000","last_peered":"2024-09-16T05:25:46.667995+0000","last_clean":"2024-09-16T05:21:06.121649+0000","last_became_active":"2024-09-16T05:19:35.788828+0000","last_became_peered":"2024-09-16T05:23:37.862679+0000","last_unstale":"2024-09-16T05:25:46.667995+0000","last_undegraded":"2024-09-16T05:25:46.667995+0000","last_fullsized":"2024-09-16T05:23:37.858165+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:23:31.555842+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_clean_scrub_stamp":"2024-09-16T05:23:31.555842+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744239999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.43","version":"281'58","reported_seq":25,"reported_epoch":974,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:47.192436+0000","last_change":"2024-09-16T05:25:17.479922+0000","last_active":"2024-09-16T05:13:47.683286+0000","last_peered":"2024-09-16T05:25:47.192436+0000","last_clean":"2024-09-16T05:13:46.730800+0000","last_became_active":"2024-09-16T05:12:21.663580+0000","last_became_peered":"2024-09-16T05:25:17.479922+0000","last_unstale":"2024-09-16T05:25:47.192436+0000","last_undegraded":"2024-09-16T05:25:17.478776+0000","last_fullsized":"2024-09-16T05:25:17.478660+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:24:19.083933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:19.083933+0000","objects_scrubbed":3,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:51.187015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3149099,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":25349,"num_write":5,"num_write_kb":1312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.42","version":"182'20","reported_seq":3295,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635539+0000","last_change":"2024-09-16T05:25:27.473803+0000","last_active":"2024-09-16T05:25:47.635539+0000","last_peered":"2024-09-16T05:25:47.635539+0000","last_clean":"2024-09-16T05:25:47.635539+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:25:47.635539+0000","last_undegraded":"2024-09-16T05:25:47.635539+0000","last_fullsized":"2024-09-16T05:25:47.635539+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:25:27.473695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:27.473695+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:55.292132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059670399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":2169,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.273242+0000","last_change":"2024-09-16T05:25:18.532515+0000","last_active":"2024-09-16T05:25:47.273242+0000","last_peered":"2024-09-16T05:25:47.273242+0000","last_clean":"2024-09-16T05:25:47.273242+0000","last_became_active":"2024-09-16T05:21:54.930175+0000","last_became_peered":"2024-09-16T05:21:54.930175+0000","last_unstale":"2024-09-16T05:25:47.273242+0000","last_undegraded":"2024-09-16T05:25:47.273242+0000","last_fullsized":"2024-09-16T05:25:47.273242+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:24:44.904499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:44.904499+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:52.995010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001305386,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":1850,"reported_epoch":975,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:50.675838+0000","last_change":"2024-09-16T05:25:17.475977+0000","last_active":"2024-09-16T05:25:16.469296+0000","last_peered":"2024-09-16T05:25:50.675838+0000","last_clean":"2024-09-16T05:25:15.510062+0000","last_became_active":"2024-09-16T05:19:19.898993+0000","last_became_peered":"2024-09-16T05:25:17.475977+0000","last_unstale":"2024-09-16T05:25:50.675838+0000","last_undegraded":"2024-09-16T05:25:50.675838+0000","last_fullsized":"2024-09-16T05:25:17.475342+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:25:04.895079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:04.895079+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:33.561906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00061814599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.3f","version":"964'595","reported_seq":5039,"reported_epoch":974,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:46.673047+0000","last_change":"2024-09-16T05:25:17.473317+0000","last_active":"2024-09-16T05:25:16.469263+0000","last_peered":"2024-09-16T05:25:46.673047+0000","last_clean":"2024-09-16T05:25:15.666322+0000","last_became_active":"2024-09-16T05:21:54.939233+0000","last_became_peered":"2024-09-16T05:25:17.473317+0000","last_unstale":"2024-09-16T05:25:46.673047+0000","last_undegraded":"2024-09-16T05:25:17.472953+0000","last_fullsized":"2024-09-16T05:25:17.472893+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"874'563","last_scrub_stamp":"2024-09-16T05:24:01.125030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:01.125030+0000","objects_scrubbed":3,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:46.942101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030368080000000002,"stat_sum":{"num_bytes":7761117,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":154,"num_read_kb":147209,"num_write":92,"num_write_kb":30495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":11726959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.3e","version":"955'603","reported_seq":4908,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.668271+0000","last_change":"2024-09-16T05:25:18.532299+0000","last_active":"2024-09-16T05:25:46.668271+0000","last_peered":"2024-09-16T05:25:46.668271+0000","last_clean":"2024-09-16T05:25:46.668271+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:25:46.668271+0000","last_undegraded":"2024-09-16T05:25:46.668271+0000","last_fullsized":"2024-09-16T05:25:46.668271+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"902'587","last_scrub_stamp":"2024-09-16T05:24:16.791425+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:24:16.791425+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080115500000000001,"stat_sum":{"num_bytes":2554994,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":61222,"num_write":79,"num_write_kb":37651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2150,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.635828+0000","last_change":"2024-09-16T05:23:37.856541+0000","last_active":"2024-09-16T05:23:36.855029+0000","last_peered":"2024-09-16T05:25:47.635828+0000","last_clean":"2024-09-16T05:21:05.805509+0000","last_became_active":"2024-09-16T05:19:35.784967+0000","last_became_peered":"2024-09-16T05:23:37.856541+0000","last_unstale":"2024-09-16T05:25:47.635828+0000","last_undegraded":"2024-09-16T05:25:47.635828+0000","last_fullsized":"2024-09-16T05:23:37.855660+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.497749+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:47.724313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053549299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[8],"avail_no_missing":["8"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3c","version":"819'826","reported_seq":5058,"reported_epoch":959,"state":"stale+undersized+peered","last_fresh":"2024-09-16T05:25:10.425915+0000","last_change":"2024-09-16T05:23:37.862640+0000","last_active":"2024-09-16T05:23:36.873896+0000","last_peered":"2024-09-16T05:25:10.425915+0000","last_clean":"2024-09-16T05:23:36.060434+0000","last_became_active":"2024-09-16T05:21:07.828778+0000","last_became_peered":"2024-09-16T05:23:37.862640+0000","last_unstale":"2024-09-16T05:25:16.470792+0000","last_undegraded":"2024-09-16T05:25:10.425915+0000","last_fullsized":"2024-09-16T05:23:37.859543+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"799'807","last_scrub_stamp":"2024-09-16T05:22:31.305689+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:22:31.305689+0000","objects_scrubbed":4,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:12.153426+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00083692700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":194,"num_read_kb":258104,"num_write":95,"num_write_kb":34389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":14713263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3b","version":"968'566","reported_seq":4921,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.676117+0000","last_change":"2024-09-16T05:25:49.860076+0000","last_active":"2024-09-16T05:25:50.676117+0000","last_peered":"2024-09-16T05:25:50.676117+0000","last_clean":"2024-09-16T05:25:50.676117+0000","last_became_active":"2024-09-16T05:13:48.688926+0000","last_became_peered":"2024-09-16T05:13:48.688926+0000","last_unstale":"2024-09-16T05:25:50.676117+0000","last_undegraded":"2024-09-16T05:25:50.676117+0000","last_fullsized":"2024-09-16T05:25:50.676117+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"968'566","last_scrub_stamp":"2024-09-16T05:25:49.859987+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:25:49.859987+0000","objects_scrubbed":4,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:59.161638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016977990000000001,"stat_sum":{"num_bytes":7397258,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":143,"num_read_kb":147006,"num_write":94,"num_write_kb":39080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.3a","version":"945'705","reported_seq":4997,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192391+0000","last_change":"2024-09-16T05:25:18.478019+0000","last_active":"2024-09-16T05:25:47.192391+0000","last_peered":"2024-09-16T05:25:47.192391+0000","last_clean":"2024-09-16T05:25:47.192391+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:25:47.192391+0000","last_undegraded":"2024-09-16T05:25:47.192391+0000","last_fullsized":"2024-09-16T05:25:47.192391+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"875'686","last_scrub_stamp":"2024-09-16T05:24:25.046109+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:24:25.046109+0000","objects_scrubbed":2,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:52.265910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016957350000000001,"stat_sum":{"num_bytes":5446062,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":76386,"num_write":90,"num_write_kb":46848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":3401,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:48.204098+0000","last_change":"2024-09-16T05:25:48.204130+0000","last_active":"2024-09-16T05:25:48.204098+0000","last_peered":"2024-09-16T05:25:48.204098+0000","last_clean":"2024-09-16T05:25:48.204098+0000","last_became_active":"2024-09-16T05:13:45.684555+0000","last_became_peered":"2024-09-16T05:13:45.684555+0000","last_unstale":"2024-09-16T05:25:48.204098+0000","last_undegraded":"2024-09-16T05:25:48.204098+0000","last_fullsized":"2024-09-16T05:25:48.204098+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:25:48.204026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:48.204026+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:56.275908+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001329893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":3407,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635314+0000","last_change":"2024-09-16T05:25:18.531252+0000","last_active":"2024-09-16T05:25:47.635314+0000","last_peered":"2024-09-16T05:25:47.635314+0000","last_clean":"2024-09-16T05:25:47.635314+0000","last_became_active":"2024-09-16T05:13:45.685329+0000","last_became_peered":"2024-09-16T05:13:45.685329+0000","last_unstale":"2024-09-16T05:25:47.635314+0000","last_undegraded":"2024-09-16T05:25:47.635314+0000","last_fullsized":"2024-09-16T05:25:47.635314+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:24:25.752866+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:24:25.752866+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:51.712967+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00053179699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":2261,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.679045+0000","last_change":"2024-09-16T05:25:18.534145+0000","last_active":"2024-09-16T05:25:50.679045+0000","last_peered":"2024-09-16T05:25:50.679045+0000","last_clean":"2024-09-16T05:25:50.679045+0000","last_became_active":"2024-09-16T05:21:54.938965+0000","last_became_peered":"2024-09-16T05:21:54.938965+0000","last_unstale":"2024-09-16T05:25:50.679045+0000","last_undegraded":"2024-09-16T05:25:50.679045+0000","last_fullsized":"2024-09-16T05:25:50.679045+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:25:03.860895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:25:03.860895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:25.704102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022621120000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":679,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.668407+0000","last_change":"2024-09-16T05:23:37.863558+0000","last_active":"2024-09-16T05:23:36.858320+0000","last_peered":"2024-09-16T05:25:46.668407+0000","last_clean":"2024-09-16T05:21:06.122490+0000","last_became_active":"2024-09-16T05:19:35.795592+0000","last_became_peered":"2024-09-16T05:23:37.863558+0000","last_unstale":"2024-09-16T05:25:46.668407+0000","last_undegraded":"2024-09-16T05:25:46.668407+0000","last_fullsized":"2024-09-16T05:23:37.859570+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:22:49.686720+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:22:49.686720+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026213400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.35","version":"843'947","reported_seq":4847,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.667996+0000","last_change":"2024-09-16T05:23:37.863020+0000","last_active":"2024-09-16T05:23:36.858871+0000","last_peered":"2024-09-16T05:25:46.667996+0000","last_clean":"2024-09-16T05:22:58.839384+0000","last_became_active":"2024-09-16T05:21:10.815197+0000","last_became_peered":"2024-09-16T05:23:37.863020+0000","last_unstale":"2024-09-16T05:25:46.667996+0000","last_undegraded":"2024-09-16T05:25:46.667996+0000","last_fullsized":"2024-09-16T05:23:37.858837+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":6,"last_scrub":"843'947","last_scrub_stamp":"2024-09-16T05:23:34.604753+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:23:34.604753+0000","objects_scrubbed":9,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924088+0000","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0.0054130100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":191,"num_read_kb":188287,"num_write":131,"num_write_kb":57425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":17054243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.34","version":"947'912","reported_seq":4835,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.676083+0000","last_change":"2024-09-16T05:25:18.532165+0000","last_active":"2024-09-16T05:25:50.676083+0000","last_peered":"2024-09-16T05:25:50.676083+0000","last_clean":"2024-09-16T05:25:50.676083+0000","last_became_active":"2024-09-16T05:13:30.744272+0000","last_became_peered":"2024-09-16T05:13:30.744272+0000","last_unstale":"2024-09-16T05:25:50.676083+0000","last_undegraded":"2024-09-16T05:25:50.676083+0000","last_fullsized":"2024-09-16T05:25:50.676083+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"932'900","last_scrub_stamp":"2024-09-16T05:24:42.860563+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:24:42.860563+0000","objects_scrubbed":6,"log_size":912,"log_dups_size":0,"ondisk_log_size":912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00099818600000000008,"stat_sum":{"num_bytes":10935297,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":248,"num_read_kb":273999,"num_write":176,"num_write_kb":64746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":3156,"reported_epoch":959,"state":"stale+undersized+peered","last_fresh":"2024-09-16T05:25:10.425742+0000","last_change":"2024-09-16T05:23:37.862595+0000","last_active":"2024-09-16T05:23:36.872961+0000","last_peered":"2024-09-16T05:25:10.425742+0000","last_clean":"2024-09-16T05:23:36.060365+0000","last_became_active":"2024-09-16T05:21:07.818018+0000","last_became_peered":"2024-09-16T05:23:37.862595+0000","last_unstale":"2024-09-16T05:25:16.470794+0000","last_undegraded":"2024-09-16T05:25:10.425742+0000","last_fullsized":"2024-09-16T05:23:37.859660+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:23:10.097585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:10.097585+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:12.153450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000766243,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.32","version":"940'330","reported_seq":3819,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667773+0000","last_change":"2024-09-16T05:25:18.533571+0000","last_active":"2024-09-16T05:25:46.667773+0000","last_peered":"2024-09-16T05:25:46.667773+0000","last_clean":"2024-09-16T05:25:46.667773+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:25:46.667773+0000","last_undegraded":"2024-09-16T05:25:46.667773+0000","last_fullsized":"2024-09-16T05:25:46.667773+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"889'320","last_scrub_stamp":"2024-09-16T05:24:24.818788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:24.818788+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:48.695374+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00099320500000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":137206,"num_write":62,"num_write_kb":24355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.31","version":"964'436","reported_seq":4510,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.273535+0000","last_change":"2024-09-16T05:25:18.532625+0000","last_active":"2024-09-16T05:25:47.273535+0000","last_peered":"2024-09-16T05:25:47.273535+0000","last_clean":"2024-09-16T05:25:47.273535+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:25:47.273535+0000","last_undegraded":"2024-09-16T05:25:47.273535+0000","last_fullsized":"2024-09-16T05:25:47.273535+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"875'402","last_scrub_stamp":"2024-09-16T05:24:22.789621+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:24:22.789621+0000","objects_scrubbed":2,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:52.272573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00118114,"stat_sum":{"num_bytes":7473346,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":138714,"num_write":102,"num_write_kb":39613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":2243,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678193+0000","last_change":"2024-09-16T05:25:49.947013+0000","last_active":"2024-09-16T05:25:50.678193+0000","last_peered":"2024-09-16T05:25:50.678193+0000","last_clean":"2024-09-16T05:25:50.678193+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:25:50.678193+0000","last_undegraded":"2024-09-16T05:25:50.678193+0000","last_fullsized":"2024-09-16T05:25:50.678193+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:25:49.946916+0000","last_deep_scrub":"182'16","last_deep_scrub_stamp":"2024-09-16T05:21:49.039822+0000","last_clean_scrub_stamp":"2024-09-16T05:25:49.946916+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:59.484210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027794859999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":1608,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.675945+0000","last_change":"2024-09-16T05:24:46.793611+0000","last_active":"2024-09-16T05:25:50.675945+0000","last_peered":"2024-09-16T05:25:50.675945+0000","last_clean":"2024-09-16T05:25:50.675945+0000","last_became_active":"2024-09-16T05:17:18.778404+0000","last_became_peered":"2024-09-16T05:17:18.778404+0000","last_unstale":"2024-09-16T05:25:50.675945+0000","last_undegraded":"2024-09-16T05:25:50.675945+0000","last_fullsized":"2024-09-16T05:25:50.675945+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:46.793537+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:24:46.793537+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:11.756827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1596,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.672237+0000","last_change":"2024-09-16T05:25:23.824586+0000","last_active":"2024-09-16T05:25:46.672237+0000","last_peered":"2024-09-16T05:25:46.672237+0000","last_clean":"2024-09-16T05:25:46.672237+0000","last_became_active":"2024-09-16T05:21:57.821735+0000","last_became_peered":"2024-09-16T05:21:57.821735+0000","last_unstale":"2024-09-16T05:25:46.672237+0000","last_undegraded":"2024-09-16T05:25:46.672237+0000","last_fullsized":"2024-09-16T05:25:46.672237+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:25:23.824482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:23.824482+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:34.940098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067703200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1460,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.667842+0000","last_change":"2024-09-16T05:23:37.862671+0000","last_active":"2024-09-16T05:23:36.857882+0000","last_peered":"2024-09-16T05:25:46.667842+0000","last_clean":"2024-09-16T05:21:06.121181+0000","last_became_active":"2024-09-16T05:19:35.788516+0000","last_became_peered":"2024-09-16T05:23:37.862671+0000","last_unstale":"2024-09-16T05:25:46.667842+0000","last_undegraded":"2024-09-16T05:25:46.667842+0000","last_fullsized":"2024-09-16T05:23:37.858101+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:13.594789+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:13.594789+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":3652,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.635424+0000","last_change":"2024-09-16T05:23:37.855979+0000","last_active":"2024-09-16T05:23:36.854647+0000","last_peered":"2024-09-16T05:25:47.635424+0000","last_clean":"2024-09-16T05:23:36.143409+0000","last_became_active":"2024-09-16T05:21:07.815584+0000","last_became_peered":"2024-09-16T05:23:37.855979+0000","last_unstale":"2024-09-16T05:25:47.635424+0000","last_undegraded":"2024-09-16T05:25:47.635424+0000","last_fullsized":"2024-09-16T05:23:37.855410+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:22:11.576311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:11.576311+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:47.724300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086469800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[8],"avail_no_missing":["8"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1565,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667165+0000","last_change":"2024-09-16T05:24:44.355530+0000","last_active":"2024-09-16T05:25:46.667165+0000","last_peered":"2024-09-16T05:25:46.667165+0000","last_clean":"2024-09-16T05:25:46.667165+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:25:46.667165+0000","last_undegraded":"2024-09-16T05:25:46.667165+0000","last_fullsized":"2024-09-16T05:25:46.667165+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:44.355418+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:24:44.355418+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:55.946937+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"967'506","reported_seq":4867,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635837+0000","last_change":"2024-09-16T05:25:29.510288+0000","last_active":"2024-09-16T05:25:47.635837+0000","last_peered":"2024-09-16T05:25:47.635837+0000","last_clean":"2024-09-16T05:25:47.635837+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:25:47.635837+0000","last_undegraded":"2024-09-16T05:25:47.635837+0000","last_fullsized":"2024-09-16T05:25:47.635837+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'506","last_scrub_stamp":"2024-09-16T05:25:29.510184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:29.510184+0000","objects_scrubbed":3,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:59.106050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0047348340000000003,"stat_sum":{"num_bytes":5849430,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":69,"num_read_kb":79207,"num_write":72,"num_write_kb":28152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.d","version":"35'139","reported_seq":540,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.672664+0000","last_change":"2024-09-16T05:25:17.476321+0000","last_active":"2024-09-16T05:15:53.835354+0000","last_peered":"2024-09-16T05:25:46.672664+0000","last_clean":"2024-09-16T05:13:32.766417+0000","last_became_active":"2024-09-16T05:13:32.765809+0000","last_became_peered":"2024-09-16T05:25:17.476321+0000","last_unstale":"2024-09-16T05:25:46.672664+0000","last_undegraded":"2024-09-16T05:25:46.672664+0000","last_fullsized":"2024-09-16T05:25:17.475118+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:07.155306+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:07.155306+0000","last_clean_scrub_stamp":"2024-09-16T05:25:07.155306+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.212236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":4460,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635621+0000","last_change":"2024-09-16T05:25:29.507154+0000","last_active":"2024-09-16T05:25:47.635621+0000","last_peered":"2024-09-16T05:25:47.635621+0000","last_clean":"2024-09-16T05:25:47.635621+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:25:47.635621+0000","last_undegraded":"2024-09-16T05:25:47.635621+0000","last_fullsized":"2024-09-16T05:25:47.635621+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:25:29.507059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:29.507059+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:42.094740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025755439999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":1712,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.672932+0000","last_change":"2024-09-16T05:25:17.476321+0000","last_active":"2024-09-16T05:25:16.742373+0000","last_peered":"2024-09-16T05:25:46.672932+0000","last_clean":"2024-09-16T05:25:08.420992+0000","last_became_active":"2024-09-16T05:20:35.802192+0000","last_became_peered":"2024-09-16T05:25:17.476321+0000","last_unstale":"2024-09-16T05:25:46.672932+0000","last_undegraded":"2024-09-16T05:25:46.672932+0000","last_fullsized":"2024-09-16T05:25:17.475840+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:25:11.119111+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:15.593568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"945'690","reported_seq":5035,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192530+0000","last_change":"2024-09-16T05:25:18.479133+0000","last_active":"2024-09-16T05:25:47.192530+0000","last_peered":"2024-09-16T05:25:47.192530+0000","last_clean":"2024-09-16T05:25:47.192530+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:25:47.192530+0000","last_undegraded":"2024-09-16T05:25:47.192530+0000","last_fullsized":"2024-09-16T05:25:47.192530+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'690","last_scrub_stamp":"2024-09-16T05:25:06.225141+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:25:06.225141+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:18.229123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002207269,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":52781,"num_write":61,"num_write_kb":34652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":1713,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.676297+0000","last_change":"2024-09-16T05:25:24.770120+0000","last_active":"2024-09-16T05:25:50.676297+0000","last_peered":"2024-09-16T05:25:50.676297+0000","last_clean":"2024-09-16T05:25:50.676297+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:25:50.676297+0000","last_undegraded":"2024-09-16T05:25:50.676297+0000","last_fullsized":"2024-09-16T05:25:50.676297+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:24.770025+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:15:59.140738+0000","last_clean_scrub_stamp":"2024-09-16T05:25:24.770025+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:28.472402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":4596,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667845+0000","last_change":"2024-09-16T05:25:18.533114+0000","last_active":"2024-09-16T05:25:46.667845+0000","last_peered":"2024-09-16T05:25:46.667845+0000","last_clean":"2024-09-16T05:25:46.667845+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:25:46.667845+0000","last_undegraded":"2024-09-16T05:25:46.667845+0000","last_fullsized":"2024-09-16T05:25:46.667845+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:24:21.764745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:21.764745+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015044220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1259,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.273334+0000","last_change":"2024-09-16T05:23:37.860671+0000","last_active":"2024-09-16T05:23:36.858161+0000","last_peered":"2024-09-16T05:25:47.273334+0000","last_clean":"2024-09-16T05:22:23.838536+0000","last_became_active":"2024-09-16T05:22:23.836724+0000","last_became_peered":"2024-09-16T05:23:37.860671+0000","last_unstale":"2024-09-16T05:25:47.273334+0000","last_undegraded":"2024-09-16T05:25:47.273334+0000","last_fullsized":"2024-09-16T05:23:37.857985+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:26.487937+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.487937+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.241660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"964'401","reported_seq":4749,"reported_epoch":974,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:47.192358+0000","last_change":"2024-09-16T05:25:17.482592+0000","last_active":"2024-09-16T05:25:16.476228+0000","last_peered":"2024-09-16T05:25:47.192358+0000","last_clean":"2024-09-16T05:25:15.463087+0000","last_became_active":"2024-09-16T05:14:25.821279+0000","last_became_peered":"2024-09-16T05:25:17.482592+0000","last_unstale":"2024-09-16T05:25:47.192358+0000","last_undegraded":"2024-09-16T05:25:17.480992+0000","last_fullsized":"2024-09-16T05:25:17.480895+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"893'385","last_scrub_stamp":"2024-09-16T05:24:42.153717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:42.153717+0000","objects_scrubbed":2,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:51.644358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067776499999999999,"stat_sum":{"num_bytes":8488459,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":134,"num_read_kb":146776,"num_write":68,"num_write_kb":26467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":3}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.3","version":"968'362","reported_seq":4719,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.676232+0000","last_change":"2024-09-16T05:25:18.532589+0000","last_active":"2024-09-16T05:25:50.676232+0000","last_peered":"2024-09-16T05:25:50.676232+0000","last_clean":"2024-09-16T05:25:50.676232+0000","last_became_active":"2024-09-16T05:14:39.909633+0000","last_became_peered":"2024-09-16T05:14:39.909633+0000","last_unstale":"2024-09-16T05:25:50.676232+0000","last_undegraded":"2024-09-16T05:25:50.676232+0000","last_fullsized":"2024-09-16T05:25:50.676232+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'352","last_scrub_stamp":"2024-09-16T05:24:41.844068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:41.844068+0000","objects_scrubbed":5,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00061232099999999998,"stat_sum":{"num_bytes":9311969,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":148,"num_read_kb":165400,"num_write":84,"num_write_kb":42025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":1258,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667708+0000","last_change":"2024-09-16T05:24:26.855969+0000","last_active":"2024-09-16T05:25:46.667708+0000","last_peered":"2024-09-16T05:25:46.667708+0000","last_clean":"2024-09-16T05:25:46.667708+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:25:46.667708+0000","last_undegraded":"2024-09-16T05:25:46.667708+0000","last_fullsized":"2024-09-16T05:25:46.667708+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:26.855873+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:24:26.855873+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924127+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":4024,"reported_epoch":975,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:50.676042+0000","last_change":"2024-09-16T05:23:37.862002+0000","last_active":"2024-09-16T05:23:36.872784+0000","last_peered":"2024-09-16T05:25:50.676042+0000","last_clean":"2024-09-16T05:23:36.142641+0000","last_became_active":"2024-09-16T05:21:07.817459+0000","last_became_peered":"2024-09-16T05:23:37.862002+0000","last_unstale":"2024-09-16T05:25:50.676042+0000","last_undegraded":"2024-09-16T05:25:50.676042+0000","last_fullsized":"2024-09-16T05:23:37.860959+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:22:55.657664+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:22:55.657664+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088962700000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1490,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678160+0000","last_change":"2024-09-16T05:25:25.923962+0000","last_active":"2024-09-16T05:25:50.678160+0000","last_peered":"2024-09-16T05:25:50.678160+0000","last_clean":"2024-09-16T05:25:50.678160+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:25:50.678160+0000","last_undegraded":"2024-09-16T05:25:50.678160+0000","last_fullsized":"2024-09-16T05:25:50.678160+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:25.923872+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:25:25.923872+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:47.768611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":453,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.273467+0000","last_change":"2024-09-16T05:25:11.092764+0000","last_active":"2024-09-16T05:25:47.273467+0000","last_peered":"2024-09-16T05:25:47.273467+0000","last_clean":"2024-09-16T05:25:47.273467+0000","last_became_active":"2024-09-16T05:25:10.425314+0000","last_became_peered":"2024-09-16T05:25:10.425314+0000","last_unstale":"2024-09-16T05:25:47.273467+0000","last_undegraded":"2024-09-16T05:25:47.273467+0000","last_fullsized":"2024-09-16T05:25:47.273467+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:11.092652+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:56.353243+0000","last_clean_scrub_stamp":"2024-09-16T05:25:11.092652+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:35.601765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":4270,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635203+0000","last_change":"2024-09-16T05:25:18.531817+0000","last_active":"2024-09-16T05:25:47.635203+0000","last_peered":"2024-09-16T05:25:47.635203+0000","last_clean":"2024-09-16T05:25:47.635203+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:25:47.635203+0000","last_undegraded":"2024-09-16T05:25:47.635203+0000","last_fullsized":"2024-09-16T05:25:47.635203+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:24:21.560842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:21.560842+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012013690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":2244,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667041+0000","last_change":"2024-09-16T05:24:45.398395+0000","last_active":"2024-09-16T05:25:46.667041+0000","last_peered":"2024-09-16T05:25:46.667041+0000","last_clean":"2024-09-16T05:25:46.667041+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:25:46.667041+0000","last_undegraded":"2024-09-16T05:25:46.667041+0000","last_fullsized":"2024-09-16T05:25:46.667041+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:45.398306+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:24:45.398306+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:10.726932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":2142,"reported_epoch":975,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:50.678160+0000","last_change":"2024-09-16T05:25:17.480262+0000","last_active":"2024-09-16T05:17:58.860357+0000","last_peered":"2024-09-16T05:25:50.678160+0000","last_clean":"2024-09-16T05:17:58.231552+0000","last_became_active":"2024-09-16T05:17:37.762842+0000","last_became_peered":"2024-09-16T05:25:17.480262+0000","last_unstale":"2024-09-16T05:25:50.678160+0000","last_undegraded":"2024-09-16T05:25:50.678160+0000","last_fullsized":"2024-09-16T05:25:17.478475+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:25:03.194302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:03.194302+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:08.311862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0039516209999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.7","version":"0'0","reported_seq":3578,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192079+0000","last_change":"2024-09-16T05:25:31.180605+0000","last_active":"2024-09-16T05:25:47.192079+0000","last_peered":"2024-09-16T05:25:47.192079+0000","last_clean":"2024-09-16T05:25:47.192079+0000","last_became_active":"2024-09-16T05:21:54.939117+0000","last_became_peered":"2024-09-16T05:21:54.939117+0000","last_unstale":"2024-09-16T05:25:47.192079+0000","last_undegraded":"2024-09-16T05:25:47.192079+0000","last_fullsized":"2024-09-16T05:25:47.192079+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:25:31.180506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:58.088776+0000","last_clean_scrub_stamp":"2024-09-16T05:25:31.180506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:57.118944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022527609999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":2200,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.667995+0000","last_change":"2024-09-16T05:25:17.481502+0000","last_active":"2024-09-16T05:25:16.484132+0000","last_peered":"2024-09-16T05:25:46.667995+0000","last_clean":"2024-09-16T05:25:15.468916+0000","last_became_active":"2024-09-16T05:23:23.863394+0000","last_became_peered":"2024-09-16T05:25:17.481502+0000","last_unstale":"2024-09-16T05:25:46.667995+0000","last_undegraded":"2024-09-16T05:25:46.667995+0000","last_fullsized":"2024-09-16T05:25:17.480110+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:18.781014+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:24:18.781014+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":2755,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635510+0000","last_change":"2024-09-16T05:25:24.449430+0000","last_active":"2024-09-16T05:25:47.635510+0000","last_peered":"2024-09-16T05:25:47.635510+0000","last_clean":"2024-09-16T05:25:47.635510+0000","last_became_active":"2024-09-16T05:21:21.816054+0000","last_became_peered":"2024-09-16T05:21:21.816054+0000","last_unstale":"2024-09-16T05:25:47.635510+0000","last_undegraded":"2024-09-16T05:25:47.635510+0000","last_fullsized":"2024-09-16T05:25:47.635510+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:25:24.449347+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_clean_scrub_stamp":"2024-09-16T05:25:24.449347+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:31.725615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020005790000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":1511,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678791+0000","last_change":"2024-09-16T05:24:50.808648+0000","last_active":"2024-09-16T05:25:50.678791+0000","last_peered":"2024-09-16T05:25:50.678791+0000","last_clean":"2024-09-16T05:25:50.678791+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:25:50.678791+0000","last_undegraded":"2024-09-16T05:25:50.678791+0000","last_fullsized":"2024-09-16T05:25:50.678791+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:50.808580+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:24:50.808580+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:03.470669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"967'735","reported_seq":5100,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635745+0000","last_change":"2024-09-16T05:25:18.536526+0000","last_active":"2024-09-16T05:25:47.635745+0000","last_peered":"2024-09-16T05:25:47.635745+0000","last_clean":"2024-09-16T05:25:47.635745+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:25:47.635745+0000","last_undegraded":"2024-09-16T05:25:47.635745+0000","last_fullsized":"2024-09-16T05:25:47.635745+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"932'718","last_scrub_stamp":"2024-09-16T05:24:44.713602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:44.713602+0000","objects_scrubbed":4,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:08.329949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0055821930000000001,"stat_sum":{"num_bytes":12639920,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":187,"num_read_kb":207594,"num_write":72,"num_write_kb":23912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":1455,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678754+0000","last_change":"2024-09-16T05:25:23.951041+0000","last_active":"2024-09-16T05:25:50.678754+0000","last_peered":"2024-09-16T05:25:50.678754+0000","last_clean":"2024-09-16T05:25:50.678754+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:25:50.678754+0000","last_undegraded":"2024-09-16T05:25:50.678754+0000","last_fullsized":"2024-09-16T05:25:50.678754+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:23.950949+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:25:23.950949+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:50.048592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1125,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.273650+0000","last_change":"2024-09-16T05:23:37.860731+0000","last_active":"2024-09-16T05:23:36.860060+0000","last_peered":"2024-09-16T05:25:47.273650+0000","last_clean":"2024-09-16T05:21:05.880843+0000","last_became_active":"2024-09-16T05:19:35.793136+0000","last_became_peered":"2024-09-16T05:23:37.860731+0000","last_unstale":"2024-09-16T05:25:47.273650+0000","last_undegraded":"2024-09-16T05:25:47.273650+0000","last_fullsized":"2024-09-16T05:23:37.858591+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:23:17.624562+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:23:17.624562+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:52.272522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010972359999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1501,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678924+0000","last_change":"2024-09-16T05:24:46.734250+0000","last_active":"2024-09-16T05:25:50.678924+0000","last_peered":"2024-09-16T05:25:50.678924+0000","last_clean":"2024-09-16T05:25:50.678924+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:25:50.678924+0000","last_undegraded":"2024-09-16T05:25:50.678924+0000","last_fullsized":"2024-09-16T05:25:50.678924+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:46.734177+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:24:46.734177+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:01.078076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":4513,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635495+0000","last_change":"2024-09-16T05:25:30.511249+0000","last_active":"2024-09-16T05:25:47.635495+0000","last_peered":"2024-09-16T05:25:47.635495+0000","last_clean":"2024-09-16T05:25:47.635495+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:25:47.635495+0000","last_undegraded":"2024-09-16T05:25:47.635495+0000","last_fullsized":"2024-09-16T05:25:47.635495+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:25:30.511147+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:25:30.511147+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:59.224523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066392100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":2239,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.676320+0000","last_change":"2024-09-16T05:24:49.905563+0000","last_active":"2024-09-16T05:25:50.676320+0000","last_peered":"2024-09-16T05:25:50.676320+0000","last_clean":"2024-09-16T05:25:50.676320+0000","last_became_active":"2024-09-16T05:20:35.807550+0000","last_became_peered":"2024-09-16T05:20:35.807550+0000","last_unstale":"2024-09-16T05:25:50.676320+0000","last_undegraded":"2024-09-16T05:25:50.676320+0000","last_fullsized":"2024-09-16T05:25:50.676320+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:49.905487+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:24:49.905487+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:58.436941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"968'780","reported_seq":5160,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667832+0000","last_change":"2024-09-16T05:25:18.533843+0000","last_active":"2024-09-16T05:25:46.667832+0000","last_peered":"2024-09-16T05:25:46.667832+0000","last_clean":"2024-09-16T05:25:46.667832+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:25:46.667832+0000","last_undegraded":"2024-09-16T05:25:46.667832+0000","last_fullsized":"2024-09-16T05:25:46.667832+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"884'766","last_scrub_stamp":"2024-09-16T05:24:45.819841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:45.819841+0000","objects_scrubbed":3,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:06.456443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012537500000000001,"stat_sum":{"num_bytes":8261972,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":148,"num_read_kb":161045,"num_write":109,"num_write_kb":46705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":1607,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.192234+0000","last_change":"2024-09-16T05:25:17.480737+0000","last_active":"2024-09-16T05:20:19.830796+0000","last_peered":"2024-09-16T05:25:47.192234+0000","last_clean":"2024-09-16T05:20:19.109008+0000","last_became_active":"2024-09-16T05:19:19.900294+0000","last_became_peered":"2024-09-16T05:25:17.480737+0000","last_unstale":"2024-09-16T05:25:47.192234+0000","last_undegraded":"2024-09-16T05:25:47.192234+0000","last_fullsized":"2024-09-16T05:25:17.479787+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:51.176377+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:24:51.176377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:51.187027+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2157,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.672210+0000","last_change":"2024-09-16T05:23:37.857332+0000","last_active":"2024-09-16T05:23:36.858356+0000","last_peered":"2024-09-16T05:25:46.672210+0000","last_clean":"2024-09-16T05:21:06.115367+0000","last_became_active":"2024-09-16T05:19:35.786308+0000","last_became_peered":"2024-09-16T05:23:37.857332+0000","last_unstale":"2024-09-16T05:25:46.672210+0000","last_undegraded":"2024-09-16T05:25:46.672210+0000","last_fullsized":"2024-09-16T05:23:37.856797+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:23:07.627317+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.627317+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:51.212324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053873599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2244,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192115+0000","last_change":"2024-09-16T05:25:03.243827+0000","last_active":"2024-09-16T05:25:47.192115+0000","last_peered":"2024-09-16T05:25:47.192115+0000","last_clean":"2024-09-16T05:25:47.192115+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:25:47.192115+0000","last_undegraded":"2024-09-16T05:25:47.192115+0000","last_fullsized":"2024-09-16T05:25:47.192115+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:03.243753+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:25:03.243753+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:15.535286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1721,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.273405+0000","last_change":"2024-09-16T05:25:04.001570+0000","last_active":"2024-09-16T05:25:47.273405+0000","last_peered":"2024-09-16T05:25:47.273405+0000","last_clean":"2024-09-16T05:25:47.273405+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:25:47.273405+0000","last_undegraded":"2024-09-16T05:25:47.273405+0000","last_fullsized":"2024-09-16T05:25:47.273405+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:04.001500+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:25:04.001500+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:09.918650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":2235,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678753+0000","last_change":"2024-09-16T05:25:18.534895+0000","last_active":"2024-09-16T05:25:50.678753+0000","last_peered":"2024-09-16T05:25:50.678753+0000","last_clean":"2024-09-16T05:25:50.678753+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:25:50.678753+0000","last_undegraded":"2024-09-16T05:25:50.678753+0000","last_fullsized":"2024-09-16T05:25:50.678753+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:24:21.697507+0000","last_deep_scrub":"272'69","last_deep_scrub_stamp":"2024-09-16T05:24:21.697507+0000","last_clean_scrub_stamp":"2024-09-16T05:24:21.697507+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024176979999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":2235,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.675916+0000","last_change":"2024-09-16T05:25:25.794885+0000","last_active":"2024-09-16T05:25:50.675916+0000","last_peered":"2024-09-16T05:25:50.675916+0000","last_clean":"2024-09-16T05:25:50.675916+0000","last_became_active":"2024-09-16T05:17:18.742576+0000","last_became_peered":"2024-09-16T05:17:18.742576+0000","last_unstale":"2024-09-16T05:25:50.675916+0000","last_undegraded":"2024-09-16T05:25:50.675916+0000","last_fullsized":"2024-09-16T05:25:50.675916+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:25.794787+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:25:25.794787+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:53.752729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":3953,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192559+0000","last_change":"2024-09-16T05:25:26.201702+0000","last_active":"2024-09-16T05:25:47.192559+0000","last_peered":"2024-09-16T05:25:47.192559+0000","last_clean":"2024-09-16T05:25:47.192559+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:25:47.192559+0000","last_undegraded":"2024-09-16T05:25:47.192559+0000","last_fullsized":"2024-09-16T05:25:47.192559+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:25:26.201601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:26.201601+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:30.719969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001901762,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.17","version":"35'139","reported_seq":1119,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635816+0000","last_change":"2024-09-16T05:24:45.748287+0000","last_active":"2024-09-16T05:25:47.635816+0000","last_peered":"2024-09-16T05:25:47.635816+0000","last_clean":"2024-09-16T05:25:47.635816+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:25:47.635816+0000","last_undegraded":"2024-09-16T05:25:47.635816+0000","last_fullsized":"2024-09-16T05:25:47.635816+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:45.748212+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:24:45.748212+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:55.502819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":4231,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192058+0000","last_change":"2024-09-16T05:25:18.479106+0000","last_active":"2024-09-16T05:25:47.192058+0000","last_peered":"2024-09-16T05:25:47.192058+0000","last_clean":"2024-09-16T05:25:47.192058+0000","last_became_active":"2024-09-16T05:21:54.931954+0000","last_became_peered":"2024-09-16T05:21:54.931954+0000","last_unstale":"2024-09-16T05:25:47.192058+0000","last_undegraded":"2024-09-16T05:25:47.192058+0000","last_fullsized":"2024-09-16T05:25:47.192058+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:24:44.054707+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:24:44.054707+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:03.459951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002182144,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.16","version":"35'139","reported_seq":1172,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635395+0000","last_change":"2024-09-16T05:25:06.617759+0000","last_active":"2024-09-16T05:25:47.635395+0000","last_peered":"2024-09-16T05:25:47.635395+0000","last_clean":"2024-09-16T05:25:47.635395+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:25:47.635395+0000","last_undegraded":"2024-09-16T05:25:47.635395+0000","last_fullsized":"2024-09-16T05:25:47.635395+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:06.617640+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:25:06.617640+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:11.023421+0000","scrub_duration":901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"848'659","reported_seq":4816,"reported_epoch":974,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:47.192428+0000","last_change":"2024-09-16T05:23:37.863036+0000","last_active":"2024-09-16T05:23:37.015531+0000","last_peered":"2024-09-16T05:25:47.192428+0000","last_clean":"2024-09-16T05:23:36.145867+0000","last_became_active":"2024-09-16T05:21:07.826042+0000","last_became_peered":"2024-09-16T05:23:37.863036+0000","last_unstale":"2024-09-16T05:25:47.192428+0000","last_undegraded":"2024-09-16T05:23:37.861512+0000","last_fullsized":"2024-09-16T05:23:37.861414+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":5,"last_scrub":"848'659","last_scrub_stamp":"2024-09-16T05:23:33.881707+0000","last_deep_scrub":"760'658","last_deep_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_clean_scrub_stamp":"2024-09-16T05:23:33.881707+0000","objects_scrubbed":1,"log_size":659,"log_dups_size":0,"ondisk_log_size":659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.178630+0000","scrub_duration":582,"objects_trimmed":0,"snaptrim_duration":0.003249806,"stat_sum":{"num_bytes":3008717,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":54,"num_read_kb":47657,"num_write":26,"num_write_kb":17519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":7118573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":1}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":1191,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.676128+0000","last_change":"2024-09-16T05:25:08.815407+0000","last_active":"2024-09-16T05:25:50.676128+0000","last_peered":"2024-09-16T05:25:50.676128+0000","last_clean":"2024-09-16T05:25:50.676128+0000","last_became_active":"2024-09-16T05:20:40.802090+0000","last_became_peered":"2024-09-16T05:20:40.802090+0000","last_unstale":"2024-09-16T05:25:50.676128+0000","last_undegraded":"2024-09-16T05:25:50.676128+0000","last_fullsized":"2024-09-16T05:25:50.676128+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:08.815330+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:25:08.815330+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:37.535988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"733'334","reported_seq":4315,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.676240+0000","last_change":"2024-09-16T05:25:28.832505+0000","last_active":"2024-09-16T05:25:50.676240+0000","last_peered":"2024-09-16T05:25:50.676240+0000","last_clean":"2024-09-16T05:25:50.676240+0000","last_became_active":"2024-09-16T05:21:54.937285+0000","last_became_peered":"2024-09-16T05:21:54.937285+0000","last_unstale":"2024-09-16T05:25:50.676240+0000","last_undegraded":"2024-09-16T05:25:50.676240+0000","last_fullsized":"2024-09-16T05:25:50.676240+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'334","last_scrub_stamp":"2024-09-16T05:25:28.832412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:28.832412+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:53.980867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00093678000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":1554,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635360+0000","last_change":"2024-09-16T05:24:46.812844+0000","last_active":"2024-09-16T05:25:47.635360+0000","last_peered":"2024-09-16T05:25:47.635360+0000","last_clean":"2024-09-16T05:25:47.635360+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:25:47.635360+0000","last_undegraded":"2024-09-16T05:25:47.635360+0000","last_fullsized":"2024-09-16T05:25:47.635360+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:46.812767+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:24:46.812767+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:03.774468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":901,"reported_epoch":975,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:50.678897+0000","last_change":"2024-09-16T05:23:37.866082+0000","last_active":"2024-09-16T05:21:26.818410+0000","last_peered":"2024-09-16T05:25:50.678897+0000","last_clean":"2024-09-16T05:21:26.020444+0000","last_became_active":"2024-09-16T05:21:07.822615+0000","last_became_peered":"2024-09-16T05:23:37.866082+0000","last_unstale":"2024-09-16T05:25:50.678897+0000","last_undegraded":"2024-09-16T05:25:50.678897+0000","last_fullsized":"2024-09-16T05:23:37.858903+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_clean_scrub_stamp":"2024-09-16T05:22:36.831827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.917548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068139299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1644,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635444+0000","last_change":"2024-09-16T05:25:28.473418+0000","last_active":"2024-09-16T05:25:47.635444+0000","last_peered":"2024-09-16T05:25:47.635444+0000","last_clean":"2024-09-16T05:25:47.635444+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:25:47.635444+0000","last_undegraded":"2024-09-16T05:25:47.635444+0000","last_fullsized":"2024-09-16T05:25:47.635444+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:28.473325+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:25:28.473325+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:32.767175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":4346,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192325+0000","last_change":"2024-09-16T05:25:22.255173+0000","last_active":"2024-09-16T05:25:47.192325+0000","last_peered":"2024-09-16T05:25:47.192325+0000","last_clean":"2024-09-16T05:25:47.192325+0000","last_became_active":"2024-09-16T05:21:27.832312+0000","last_became_peered":"2024-09-16T05:21:27.832312+0000","last_unstale":"2024-09-16T05:25:47.192325+0000","last_undegraded":"2024-09-16T05:25:47.192325+0000","last_fullsized":"2024-09-16T05:25:47.192325+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:25:22.255047+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:24:05.187021+0000","last_clean_scrub_stamp":"2024-09-16T05:25:22.255047+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:36.131358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019053760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.1a","version":"35'139","reported_seq":996,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.672401+0000","last_change":"2024-09-16T05:25:17.476314+0000","last_active":"2024-09-16T05:25:16.742341+0000","last_peered":"2024-09-16T05:25:46.672401+0000","last_clean":"2024-09-16T05:25:08.420864+0000","last_became_active":"2024-09-16T05:20:40.806260+0000","last_became_peered":"2024-09-16T05:25:17.476314+0000","last_unstale":"2024-09-16T05:25:46.672401+0000","last_undegraded":"2024-09-16T05:25:46.672401+0000","last_fullsized":"2024-09-16T05:25:17.475144+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:08.177962+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:25:08.177962+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:22.616156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":3959,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.666772+0000","last_change":"2024-09-16T05:25:25.499643+0000","last_active":"2024-09-16T05:25:46.666772+0000","last_peered":"2024-09-16T05:25:46.666772+0000","last_clean":"2024-09-16T05:25:46.666772+0000","last_became_active":"2024-09-16T05:21:54.930126+0000","last_became_peered":"2024-09-16T05:21:54.930126+0000","last_unstale":"2024-09-16T05:25:46.666772+0000","last_undegraded":"2024-09-16T05:25:46.666772+0000","last_fullsized":"2024-09-16T05:25:46.666772+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:25:25.499551+0000","last_deep_scrub":"182'60","last_deep_scrub_stamp":"2024-09-16T05:21:29.356695+0000","last_clean_scrub_stamp":"2024-09-16T05:25:25.499551+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:38.921807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046652400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.19","version":"35'139","reported_seq":2236,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678160+0000","last_change":"2024-09-16T05:25:05.894986+0000","last_active":"2024-09-16T05:25:50.678160+0000","last_peered":"2024-09-16T05:25:50.678160+0000","last_clean":"2024-09-16T05:25:50.678160+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:25:50.678160+0000","last_undegraded":"2024-09-16T05:25:50.678160+0000","last_fullsized":"2024-09-16T05:25:50.678160+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:05.894911+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:25:05.894911+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:25.569731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"968'1365","reported_seq":5854,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635232+0000","last_change":"2024-09-16T05:25:32.575586+0000","last_active":"2024-09-16T05:25:47.635232+0000","last_peered":"2024-09-16T05:25:47.635232+0000","last_clean":"2024-09-16T05:25:47.635232+0000","last_became_active":"2024-09-16T05:21:54.943140+0000","last_became_peered":"2024-09-16T05:21:54.943140+0000","last_unstale":"2024-09-16T05:25:47.635232+0000","last_undegraded":"2024-09-16T05:25:47.635232+0000","last_fullsized":"2024-09-16T05:25:47.635232+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"968'1365","last_scrub_stamp":"2024-09-16T05:25:32.575481+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:25:32.575481+0000","objects_scrubbed":6,"log_size":1365,"log_dups_size":0,"ondisk_log_size":1365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:50.934588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0033265740000000001,"stat_sum":{"num_bytes":19232810,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":121277,"num_write":71,"num_write_kb":26842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":19652756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.18","version":"35'139","reported_seq":2239,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.675826+0000","last_change":"2024-09-16T05:25:05.903998+0000","last_active":"2024-09-16T05:25:50.675826+0000","last_peered":"2024-09-16T05:25:50.675826+0000","last_clean":"2024-09-16T05:25:50.675826+0000","last_became_active":"2024-09-16T05:20:35.809442+0000","last_became_peered":"2024-09-16T05:20:35.809442+0000","last_unstale":"2024-09-16T05:25:50.675826+0000","last_undegraded":"2024-09-16T05:25:50.675826+0000","last_fullsized":"2024-09-16T05:25:50.675826+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:05.903922+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:25:05.903922+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:31.950832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"945'758","reported_seq":5146,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667907+0000","last_change":"2024-09-16T05:25:35.058682+0000","last_active":"2024-09-16T05:25:46.667907+0000","last_peered":"2024-09-16T05:25:46.667907+0000","last_clean":"2024-09-16T05:25:46.667907+0000","last_became_active":"2024-09-16T05:21:27.832003+0000","last_became_peered":"2024-09-16T05:21:27.832003+0000","last_unstale":"2024-09-16T05:25:46.667907+0000","last_undegraded":"2024-09-16T05:25:46.667907+0000","last_fullsized":"2024-09-16T05:25:46.667907+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"945'758","last_scrub_stamp":"2024-09-16T05:25:35.058573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:35.058573+0000","objects_scrubbed":3,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:55.437677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001327485,"stat_sum":{"num_bytes":6420802,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":51,"num_read_kb":48936,"num_write":30,"num_write_kb":13241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.1c","version":"723'472","reported_seq":4499,"reported_epoch":975,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:50.678307+0000","last_change":"2024-09-16T05:25:17.481508+0000","last_active":"2024-09-16T05:25:16.742314+0000","last_peered":"2024-09-16T05:25:50.678307+0000","last_clean":"2024-09-16T05:25:15.513071+0000","last_became_active":"2024-09-16T05:21:27.831591+0000","last_became_peered":"2024-09-16T05:25:17.481508+0000","last_unstale":"2024-09-16T05:25:50.678307+0000","last_undegraded":"2024-09-16T05:25:50.678307+0000","last_fullsized":"2024-09-16T05:25:17.479528+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"723'472","last_scrub_stamp":"2024-09-16T05:24:16.709079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:16.709079+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:54.944906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023948839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1725,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.668294+0000","last_change":"2024-09-16T05:25:33.066067+0000","last_active":"2024-09-16T05:25:46.668294+0000","last_peered":"2024-09-16T05:25:46.668294+0000","last_clean":"2024-09-16T05:25:46.668294+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:25:46.668294+0000","last_undegraded":"2024-09-16T05:25:46.668294+0000","last_fullsized":"2024-09-16T05:25:46.668294+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:33.065967+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:25:33.065967+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:49.041058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"947'1347","reported_seq":5914,"reported_epoch":975,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:50.678539+0000","last_change":"2024-09-16T05:25:17.481501+0000","last_active":"2024-09-16T05:25:16.742501+0000","last_peered":"2024-09-16T05:25:50.678539+0000","last_clean":"2024-09-16T05:25:15.511742+0000","last_became_active":"2024-09-16T05:21:27.833962+0000","last_became_peered":"2024-09-16T05:25:17.481501+0000","last_unstale":"2024-09-16T05:25:50.678539+0000","last_undegraded":"2024-09-16T05:25:17.480500+0000","last_fullsized":"2024-09-16T05:25:17.480410+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"902'1318","last_scrub_stamp":"2024-09-16T05:24:14.712332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:14.712332+0000","objects_scrubbed":3,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:53.991981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022112569999999999,"stat_sum":{"num_bytes":2151560,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":27,"num_read_kb":19367,"num_write":20,"num_write_kb":8848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10],"avail_no_missing":["10"],"object_location_counts":[{"shards":"10","objects":2}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"1.1d","version":"35'139","reported_seq":805,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.667930+0000","last_change":"2024-09-16T05:23:37.862749+0000","last_active":"2024-09-16T05:23:36.858349+0000","last_peered":"2024-09-16T05:25:46.667930+0000","last_clean":"2024-09-16T05:21:06.122751+0000","last_became_active":"2024-09-16T05:20:40.803676+0000","last_became_peered":"2024-09-16T05:23:37.862749+0000","last_unstale":"2024-09-16T05:25:46.667930+0000","last_undegraded":"2024-09-16T05:25:46.667930+0000","last_fullsized":"2024-09-16T05:23:37.858541+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":721,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:22:37.530632+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:22:37.530632+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:49.924065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1e","version":"736'512","reported_seq":4884,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:46.667357+0000","last_change":"2024-09-16T05:25:18.477807+0000","last_active":"2024-09-16T05:25:46.667357+0000","last_peered":"2024-09-16T05:25:46.667357+0000","last_clean":"2024-09-16T05:25:46.667357+0000","last_became_active":"2024-09-16T05:21:27.826212+0000","last_became_peered":"2024-09-16T05:21:27.826212+0000","last_unstale":"2024-09-16T05:25:46.667357+0000","last_undegraded":"2024-09-16T05:25:46.667357+0000","last_fullsized":"2024-09-16T05:25:46.667357+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'512","last_scrub_stamp":"2024-09-16T05:25:06.385344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:06.385344+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:31.248937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044007300000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"1.1c","version":"35'139","reported_seq":974,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635269+0000","last_change":"2024-09-16T05:25:06.651098+0000","last_active":"2024-09-16T05:25:47.635269+0000","last_peered":"2024-09-16T05:25:47.635269+0000","last_clean":"2024-09-16T05:25:47.635269+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:25:47.635269+0000","last_undegraded":"2024-09-16T05:25:47.635269+0000","last_fullsized":"2024-09-16T05:25:47.635269+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:06.650988+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:25:06.650988+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:26.008335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":4554,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.273780+0000","last_change":"2024-09-16T05:25:17.474567+0000","last_active":"2024-09-16T05:25:16.742728+0000","last_peered":"2024-09-16T05:25:47.273780+0000","last_clean":"2024-09-16T05:25:15.510359+0000","last_became_active":"2024-09-16T05:21:27.831337+0000","last_became_peered":"2024-09-16T05:25:17.474567+0000","last_unstale":"2024-09-16T05:25:47.273780+0000","last_undegraded":"2024-09-16T05:25:47.273780+0000","last_fullsized":"2024-09-16T05:25:17.474083+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:23:57.788027+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:23:57.788027+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:52.272532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000681326,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.20","version":"943'392","reported_seq":4671,"reported_epoch":975,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:50.676140+0000","last_change":"2024-09-16T05:25:17.476260+0000","last_active":"2024-09-16T05:25:16.470051+0000","last_peered":"2024-09-16T05:25:50.676140+0000","last_clean":"2024-09-16T05:24:54.359911+0000","last_became_active":"2024-09-16T05:12:23.818804+0000","last_became_peered":"2024-09-16T05:25:17.476260+0000","last_unstale":"2024-09-16T05:25:50.676140+0000","last_undegraded":"2024-09-16T05:25:17.475736+0000","last_fullsized":"2024-09-16T05:25:17.475641+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":6,"last_scrub":"868'385","last_scrub_stamp":"2024-09-16T05:24:16.080130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:16.080130+0000","objects_scrubbed":2,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068396400000000001,"stat_sum":{"num_bytes":6118309,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":184,"num_read_kb":215626,"num_write":82,"num_write_kb":34044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.21","version":"178'50","reported_seq":4277,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635278+0000","last_change":"2024-09-16T05:25:33.497173+0000","last_active":"2024-09-16T05:25:47.635278+0000","last_peered":"2024-09-16T05:25:47.635278+0000","last_clean":"2024-09-16T05:25:47.635278+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:25:47.635278+0000","last_undegraded":"2024-09-16T05:25:47.635278+0000","last_fullsized":"2024-09-16T05:25:47.635278+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:25:33.497085+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:25:33.497085+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:50.982271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001629887,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":2241,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-09-16T05:25:50.678328+0000","last_change":"2024-09-16T05:25:18.535496+0000","last_active":"2024-09-16T05:25:50.678328+0000","last_peered":"2024-09-16T05:25:50.678328+0000","last_clean":"2024-09-16T05:25:50.678328+0000","last_became_active":"2024-09-16T05:21:54.942985+0000","last_became_peered":"2024-09-16T05:21:54.942985+0000","last_unstale":"2024-09-16T05:25:50.678328+0000","last_undegraded":"2024-09-16T05:25:50.678328+0000","last_fullsized":"2024-09-16T05:25:50.678328+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:24:43.831327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:43.831327+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025786619999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":3505,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.667136+0000","last_change":"2024-09-16T05:23:37.859916+0000","last_active":"2024-09-16T05:23:36.869822+0000","last_peered":"2024-09-16T05:25:46.667136+0000","last_clean":"2024-09-16T05:23:35.997430+0000","last_became_active":"2024-09-16T05:22:22.838582+0000","last_became_peered":"2024-09-16T05:23:37.859916+0000","last_unstale":"2024-09-16T05:25:46.667136+0000","last_undegraded":"2024-09-16T05:25:46.667136+0000","last_fullsized":"2024-09-16T05:23:37.858901+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:22:26.319237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.319237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001072094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"182'89","reported_seq":3627,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192088+0000","last_change":"2024-09-16T05:25:18.478670+0000","last_active":"2024-09-16T05:25:47.192088+0000","last_peered":"2024-09-16T05:25:47.192088+0000","last_clean":"2024-09-16T05:25:47.192088+0000","last_became_active":"2024-09-16T05:21:54.932630+0000","last_became_peered":"2024-09-16T05:21:54.932630+0000","last_unstale":"2024-09-16T05:25:47.192088+0000","last_undegraded":"2024-09-16T05:25:47.192088+0000","last_fullsized":"2024-09-16T05:25:47.192088+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:24:27.024089+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:24:27.024089+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.274359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019785509999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.25","version":"968'1445","reported_seq":5698,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635867+0000","last_change":"2024-09-16T05:25:18.537646+0000","last_active":"2024-09-16T05:25:47.635867+0000","last_peered":"2024-09-16T05:25:47.635867+0000","last_clean":"2024-09-16T05:25:47.635867+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:25:47.635867+0000","last_undegraded":"2024-09-16T05:25:47.635867+0000","last_fullsized":"2024-09-16T05:25:47.635867+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"940'1423","last_scrub_stamp":"2024-09-16T05:24:50.794492+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:24:50.794492+0000","objects_scrubbed":7,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:02.997213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0058727320000000003,"stat_sum":{"num_bytes":15554249,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":312,"num_read_kb":311018,"num_write":259,"num_write_kb":103953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":25,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.673016+0000","last_change":"2024-09-16T05:25:17.476761+0000","last_active":"2024-09-16T05:13:01.672366+0000","last_peered":"2024-09-16T05:25:46.673016+0000","last_clean":"2024-09-16T05:13:00.129694+0000","last_became_active":"2024-09-16T05:12:59.778178+0000","last_became_peered":"2024-09-16T05:25:17.476761+0000","last_unstale":"2024-09-16T05:25:46.673016+0000","last_undegraded":"2024-09-16T05:25:46.673016+0000","last_fullsized":"2024-09-16T05:25:17.475979+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:24:50.200774+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_clean_scrub_stamp":"2024-09-16T05:24:50.200774+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.212246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.27","version":"0'0","reported_seq":25,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:46.673222+0000","last_change":"2024-09-16T05:25:17.476320+0000","last_active":"2024-09-16T05:13:01.672108+0000","last_peered":"2024-09-16T05:25:46.673222+0000","last_clean":"2024-09-16T05:12:59.748533+0000","last_became_active":"2024-09-16T05:12:59.747969+0000","last_became_peered":"2024-09-16T05:25:17.476320+0000","last_unstale":"2024-09-16T05:25:46.673222+0000","last_undegraded":"2024-09-16T05:25:46.673222+0000","last_fullsized":"2024-09-16T05:25:17.475556+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:23:58.148133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:23:58.148133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.212225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.28","version":"245'105","reported_seq":2328,"reported_epoch":974,"state":"undersized+peered","last_fresh":"2024-09-16T05:25:47.635146+0000","last_change":"2024-09-16T05:23:37.855933+0000","last_active":"2024-09-16T05:23:36.854671+0000","last_peered":"2024-09-16T05:25:47.635146+0000","last_clean":"2024-09-16T05:23:36.167119+0000","last_became_active":"2024-09-16T05:21:07.818030+0000","last_became_peered":"2024-09-16T05:23:37.855933+0000","last_unstale":"2024-09-16T05:25:47.635146+0000","last_undegraded":"2024-09-16T05:25:47.635146+0000","last_fullsized":"2024-09-16T05:23:37.855095+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:22:34.641066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:22:34.641066+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016227399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.29","version":"968'1484","reported_seq":5962,"reported_epoch":974,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T05:25:46.672002+0000","last_change":"2024-09-16T05:23:40.031990+0000","last_active":"2024-09-16T05:25:46.672002+0000","last_peered":"2024-09-16T05:25:46.672002+0000","last_clean":"2024-09-16T05:25:46.672002+0000","last_became_active":"2024-09-16T05:21:57.823393+0000","last_became_peered":"2024-09-16T05:21:57.823393+0000","last_unstale":"2024-09-16T05:25:46.672002+0000","last_undegraded":"2024-09-16T05:25:46.672002+0000","last_fullsized":"2024-09-16T05:25:46.672002+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":6,"last_scrub":"795'1297","last_scrub_stamp":"2024-09-16T05:22:26.617228+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:22:26.617228+0000","objects_scrubbed":0,"log_size":1484,"log_dups_size":0,"ondisk_log_size":1484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 126s","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.002515939,"stat_sum":{"num_bytes":22111364,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":466,"num_read_kb":444552,"num_write":336,"num_write_kb":129515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":45326140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16f","length":"1"},{"start":"171","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":1920,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192279+0000","last_change":"2024-09-16T05:25:20.249888+0000","last_active":"2024-09-16T05:25:47.192279+0000","last_peered":"2024-09-16T05:25:47.192279+0000","last_clean":"2024-09-16T05:25:47.192279+0000","last_became_active":"2024-09-16T05:23:26.848398+0000","last_became_peered":"2024-09-16T05:23:26.848398+0000","last_unstale":"2024-09-16T05:25:47.192279+0000","last_undegraded":"2024-09-16T05:25:47.192279+0000","last_fullsized":"2024-09-16T05:25:47.192279+0000","mapping_epoch":856,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":857,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:25:20.249806+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_clean_scrub_stamp":"2024-09-16T05:25:20.249806+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:22.986235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017622569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.2b","version":"952'476","reported_seq":4753,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635428+0000","last_change":"2024-09-16T05:25:36.488631+0000","last_active":"2024-09-16T05:25:47.635428+0000","last_peered":"2024-09-16T05:25:47.635428+0000","last_clean":"2024-09-16T05:25:47.635428+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:25:47.635428+0000","last_undegraded":"2024-09-16T05:25:47.635428+0000","last_fullsized":"2024-09-16T05:25:47.635428+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"952'476","last_scrub_stamp":"2024-09-16T05:25:36.488467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:36.488467+0000","objects_scrubbed":1,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:01.533265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002420672,"stat_sum":{"num_bytes":2269275,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":143,"num_read_kb":133267,"num_write":89,"num_write_kb":35255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":4452,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.635711+0000","last_change":"2024-09-16T05:25:18.536839+0000","last_active":"2024-09-16T05:25:47.635711+0000","last_peered":"2024-09-16T05:25:47.635711+0000","last_clean":"2024-09-16T05:25:47.635711+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:25:47.635711+0000","last_undegraded":"2024-09-16T05:25:47.635711+0000","last_fullsized":"2024-09-16T05:25:47.635711+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:24:21.664766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:21.664766+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:51.712993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053415629999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]},{"pgid":"2.2d","version":"958'533","reported_seq":4919,"reported_epoch":974,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:46.672450+0000","last_change":"2024-09-16T05:25:17.473409+0000","last_active":"2024-09-16T05:25:16.469470+0000","last_peered":"2024-09-16T05:25:46.672450+0000","last_clean":"2024-09-16T05:25:09.426907+0000","last_became_active":"2024-09-16T05:21:54.933266+0000","last_became_peered":"2024-09-16T05:25:17.473409+0000","last_unstale":"2024-09-16T05:25:46.672450+0000","last_undegraded":"2024-09-16T05:25:17.472956+0000","last_fullsized":"2024-09-16T05:25:17.472823+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"864'525","last_scrub_stamp":"2024-09-16T05:24:18.122638+0000","last_deep_scrub":"819'522","last_deep_scrub_stamp":"2024-09-16T05:22:54.200053+0000","last_clean_scrub_stamp":"2024-09-16T05:24:18.122638+0000","objects_scrubbed":1,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:25:50.979118+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.00045017000000000001,"stat_sum":{"num_bytes":2470959,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":113,"num_read_kb":112905,"num_write":79,"num_write_kb":38649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":10583843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.2e","version":"964'871","reported_seq":5143,"reported_epoch":974,"state":"undersized+degraded+peered","last_fresh":"2024-09-16T05:25:46.667272+0000","last_change":"2024-09-16T05:25:17.475227+0000","last_active":"2024-09-16T05:25:16.742206+0000","last_peered":"2024-09-16T05:25:46.667272+0000","last_clean":"2024-09-16T05:25:15.464896+0000","last_became_active":"2024-09-16T05:13:30.746458+0000","last_became_peered":"2024-09-16T05:25:17.475227+0000","last_unstale":"2024-09-16T05:25:46.667272+0000","last_undegraded":"2024-09-16T05:25:17.474869+0000","last_fullsized":"2024-09-16T05:25:17.474756+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":6,"last_scrub":"875'821","last_scrub_stamp":"2024-09-16T05:24:00.151377+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:24:00.151377+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011990939999999999,"stat_sum":{"num_bytes":4588971,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":152,"num_read_kb":100467,"num_write":169,"num_write_kb":80591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"}]},{"pgid":"2.2f","version":"969'837","reported_seq":5233,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-09-16T05:25:47.192599+0000","last_change":"2024-09-16T05:25:24.189682+0000","last_active":"2024-09-16T05:25:47.192599+0000","last_peered":"2024-09-16T05:25:47.192599+0000","last_clean":"2024-09-16T05:25:47.192599+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:25:47.192599+0000","last_undegraded":"2024-09-16T05:25:47.192599+0000","last_fullsized":"2024-09-16T05:25:47.192599+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"969'837","last_scrub_stamp":"2024-09-16T05:25:24.189547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:24.189547+0000","objects_scrubbed":5,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:26:40.720699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023789660000000002,"stat_sum":{"num_bytes":15195046,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":206,"num_read_kb":147197,"num_write":209,"num_write_kb":77450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"16d","length":"1"},{"start":"16f","length":"1"},{"start":"171","length":"2f"},{"start":"1a1","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":96,"stat_sum":{"num_bytes":252194850,"num_objects":109,"num_object_clones":66,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":10,"num_read":4724,"num_read_kb":4707814,"num_write":3366,"num_write_kb":1366450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":154679829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":304189440,"data_stored":376882825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34966,"ondisk_log_size":34966,"up":160,"acting":160,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":54,"acting":54,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089545,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76396,"kb_used_data":31320,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93668756,"statfs":{"total":95995035648,"available":95916806144,"internally_reserved":0,"allocated":32071680,"data_stored":39163364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":46135748},"hb_peers":[0,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285774,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90928,"kb_used_data":41692,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93654224,"statfs":{"total":95995035648,"available":95901925376,"internally_reserved":0,"allocated":42692608,"data_stored":48102581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514712,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90220,"kb_used_data":36824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93654932,"statfs":{"total":95995035648,"available":95902650368,"internally_reserved":0,"allocated":37707776,"data_stored":35258762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[0,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841751,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88160,"kb_used_data":38924,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93656992,"statfs":{"total":95995035648,"available":95904759808,"internally_reserved":0,"allocated":39858176,"data_stored":50616651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":58,"seq":249108103381,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97004,"kb_used_data":25556,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93648148,"statfs":{"total":95995035648,"available":95895703552,"internally_reserved":0,"allocated":26169344,"data_stored":26664957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397731,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85608,"kb_used_data":36372,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93659544,"statfs":{"total":95995035648,"available":95907373056,"internally_reserved":0,"allocated":37244928,"data_stored":45058114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":262144},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757479,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95636,"kb_used_data":38080,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93649516,"statfs":{"total":95995035648,"available":95897104384,"internally_reserved":0,"allocated":38993920,"data_stored":39196002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921004,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55076,"kb_used_data":15572,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93690076,"statfs":{"total":95995035648,"available":95938637824,"internally_reserved":0,"allocated":15945728,"data_stored":15970191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117232,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89776,"kb_used_data":40540,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93655376,"statfs":{"total":95995035648,"available":95903105024,"internally_reserved":0,"allocated":41512960,"data_stored":48320213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240519,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67684,"kb_used_data":35408,"kb_used_omap":10,"kb_used_meta":32245,"kb_avail":93677468,"statfs":{"total":95995035648,"available":95925727232,"internally_reserved":0,"allocated":36257792,"data_stored":35281285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":33019187},"hb_peers":[0,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81388,"kb_used_data":23492,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93663764,"statfs":{"total":95995035648,"available":95911694336,"internally_reserved":0,"allocated":24055808,"data_stored":26781009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738624,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86852,"kb_used_data":25136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93658300,"statfs":{"total":95995035648,"available":95906099200,"internally_reserved":0,"allocated":25739264,"data_stored":31887199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17846272,"data_stored":26320436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":19995522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28438528,"data_stored":29787590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33619968,"data_stored":42753450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8380416,"data_stored":10731108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31211520,"data_stored":33747595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":29679616,"data_stored":39819031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":18788352,"data_stored":21551143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32292864,"data_stored":45377568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":30142464,"data_stored":30019679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":35115008,"data_stored":42855422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24506368,"data_stored":33924281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:25:54.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 41 2024-09-16T05:25:54.406 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:54.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:54.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:54.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:54.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:54.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:54.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:54.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:54.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:54.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:54.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:54.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:54.806 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:54.806 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:54.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:25:54.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:25:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: 2.4d deep-scrub starts 2024-09-16T05:25:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:25:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: 2.38 scrub starts 2024-09-16T05:25:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: 2.38 scrub ok 2024-09-16T05:25:54.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: osdmap e976: 12 total, 10 up, 12 in 2024-09-16T05:25:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:25:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: 1.0 deep-scrub starts 2024-09-16T05:25:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: 1.0 deep-scrub ok 2024-09-16T05:25:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: 2.24 scrub starts 2024-09-16T05:25:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: 2.24 scrub ok 2024-09-16T05:25:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:25:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:25:54.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:54 smithi032 ceph-mon[27388]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:25:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:54.925 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:54.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:25:54.932 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:54.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: 2.4d deep-scrub starts 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: 2.38 scrub starts 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: 2.38 scrub ok 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: osdmap e976: 12 total, 10 up, 12 in 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: 1.0 deep-scrub starts 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: 1.0 deep-scrub ok 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: 2.24 scrub starts 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: 2.24 scrub ok 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:25:54.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:54 smithi196 ceph-mon[30802]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:25:55.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:55.050 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:55.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:55.151 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: 2.4d deep-scrub starts 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: 2.38 scrub starts 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: 2.38 scrub ok 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: osdmap e976: 12 total, 10 up, 12 in 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: 1.0 deep-scrub starts 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: 1.0 deep-scrub ok 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: 2.24 scrub starts 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: 2.24 scrub ok 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:25:55.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:54 smithi204 ceph-mon[30933]: from='mgr.14149 ' entity='mgr.y' 2024-09-16T05:25:55.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:55.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:55.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:55.297 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:55.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:55.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:55.421 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:55.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:55.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:25:55.479 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:55.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:55.544 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:55.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:Error EINVAL: specified pgp_num 41 > pg_num 31 2024-09-16T05:25:55.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:55.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:55.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:55.691 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: 2.4d deep-scrub ok 2024-09-16T05:25:55.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: pgmap v1640: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:55.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:25:55.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:25:55.692 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 135 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:25:55.724 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:25:55.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:55.793 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:55.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:55.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:55.918 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:55.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: 2.4d deep-scrub ok 2024-09-16T05:25:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: pgmap v1640: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:25:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:25:55.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 135 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:55.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3007636417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:55.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: 2.c scrub starts 2024-09-16T05:25:55.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: 2.c scrub ok 2024-09-16T05:25:55.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:55 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/630989352' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-09-16T05:25:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3007636417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: 2.c scrub starts 2024-09-16T05:25:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: 2.c scrub ok 2024-09-16T05:25:55.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:55 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/630989352' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-09-16T05:25:56.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:56.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:56.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:56.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:56.166 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:56.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: 2.4d deep-scrub ok 2024-09-16T05:25:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: pgmap v1640: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:56.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:25:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:25:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 135 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:25:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3007636417' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:25:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: 2.c scrub starts 2024-09-16T05:25:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: 2.c scrub ok 2024-09-16T05:25:56.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:55 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/630989352' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-09-16T05:25:56.212 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:25:56.213 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:25:56.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:56.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:56.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:56.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:56.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:56.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:56.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:56.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:56.546 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:56.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:56.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:56.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:56.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:56.708 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:25:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:25:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:25:56.730 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:25:56.730 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:25:56.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:56 smithi032 ceph-mon[27388]: 2.2c scrub starts 2024-09-16T05:25:56.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:56 smithi032 ceph-mon[27388]: 2.2c scrub ok 2024-09-16T05:25:56.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:56 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:56.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3007636417' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:56.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:56 smithi032 ceph-mon[27388]: osdmap e977: 12 total, 10 up, 12 in 2024-09-16T05:25:56.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:56 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:25:56.731 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3227311630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:56.750 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:25:56.751 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:25:56.772 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:25:56.772 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:25:56.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:56 smithi196 ceph-mon[30802]: 2.2c scrub starts 2024-09-16T05:25:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:56 smithi196 ceph-mon[30802]: 2.2c scrub ok 2024-09-16T05:25:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:56 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3007636417' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:56 smithi196 ceph-mon[30802]: osdmap e977: 12 total, 10 up, 12 in 2024-09-16T05:25:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:56 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:25:56.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3227311630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:56.793 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:25:56.793 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:25:56.798 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:56.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:56.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:25:56.814 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:25:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:25:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:25:56.836 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:25:56.837 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:25:56.858 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:25:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:25:56.880 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:25:56.880 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:25:56.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:56.902 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:25:56.902 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:25:56.923 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:56.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:57.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:57.047 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:57.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:57.090 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:57.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:56 smithi204 ceph-mon[30933]: 2.2c scrub starts 2024-09-16T05:25:57.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:56 smithi204 ceph-mon[30933]: 2.2c scrub ok 2024-09-16T05:25:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:56 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:25:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3007636417' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:25:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:56 smithi204 ceph-mon[30933]: osdmap e977: 12 total, 10 up, 12 in 2024-09-16T05:25:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:56 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:25:57.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3227311630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:25:57.173 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:57.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:57.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:56 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:56.998+0000 7f6af1ad6640 -1 osd.6 977 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:57.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:57.298 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:57.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:57.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:57.424 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:57.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T05:25:57.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:57.528 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:25:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:25:57.528 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:25:57.529 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:25:57.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:57.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:57.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:57.672 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:57.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:57.774 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:57.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:57.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:57.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:57.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:57 smithi032 ceph-mon[27388]: pgmap v1642: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:57.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:57 smithi204 ceph-mon[30933]: pgmap v1642: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:57.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:57.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:57.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:57 smithi196 ceph-mon[30802]: pgmap v1642: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:25:57.997 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:25:58.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:58.049 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:58.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:58.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:25:58.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:58.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:25:58.279 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:25:58.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:58.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:25:58.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:25:58.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:25:58.426 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:58.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:25:58.487 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:58 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:58.018+0000 7f6af1ad6640 -1 osd.6 977 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:58.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:25:58.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:58.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:58.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:58.625 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:58.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:58.679 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:58.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:58.761 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:25:58.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:58.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:58.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:58.882 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:58 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:58.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:58.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:58.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:58.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:58 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:58.997 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:58 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:59.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:59.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:59.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:59.097 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:25:59.097 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:25:59.097 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:25:59.097 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:25:59.097 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:25:59.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:25:59.182 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:59.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:59.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:25:59.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:59.308 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:59.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:59.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:25:59.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:59.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:25:59.487 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:58 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:58.998+0000 7f6af1ad6640 -1 osd.6 977 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:25:59.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:25:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:59.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:25:59.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:25:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:59.687 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:25:59.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:25:59.704 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:25:59.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:25:59.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:25:59.811 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:25:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:59.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:25:59.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:25:59.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:25:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:25:59.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:25:59.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:59 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:25:59.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:59 smithi196 ceph-mon[30802]: pgmap v1643: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:25:59.961 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:25:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3750642141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:25:59.987 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:25:59.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:26:00.017 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:59 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:00.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:59 smithi032 ceph-mon[27388]: pgmap v1643: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:00.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:25:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3750642141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:00.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:00.064 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:00.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:00.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:59 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:00.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:59 smithi204 ceph-mon[30933]: pgmap v1643: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:00.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:25:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3750642141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:00.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:00.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:00.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:00.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:25:59 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:25:59.960+0000 7f6af1ad6640 -1 osd.6 978 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:00.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:00.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:00.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:00.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:00.432 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:00.440 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:00.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:00.471 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:839 2.1a e438/291 ondisk, result = 0)", 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 572.73885394199999, 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51227693699999999, 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.472 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730489+0000", 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730490+0000", 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5400000000000002e-07 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.730494+0000", 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0810000000000004e-06 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242338+0000", 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.51184441999999997 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242353+0000", 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.447e-05 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242501+0000", 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014851799999999999 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242532+0000", 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0406999999999999e-05 2024-09-16T05:26:00.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.242766+0000", 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000234587 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGRecoveryDelete(2.4c e439,438 2:33100188:::smithi20457400-19:2f,230'190)", 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 571.63594283500004, 2024-09-16T05:26:00.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.584288378, 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833400+0000", 2024-09-16T05:26:00.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833401+0000", 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:28.833406+0000", 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8940000000000001e-06 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417262+0000", 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.58385600999999998 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417279+0000", 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6880000000000001e-05 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417458+0000", 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000178751 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417489+0000", 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1192999999999998e-05 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:29.417689+0000", 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019989399999999999 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5460 2.25 2:a5bc1608:::smithi20457400-11:head [setxattr _header (58) in=7b,truncate 2975004] snapc e1=[e1,e0,df,dd,dc,da,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 436.474916517, 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69151579299999999, 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5460 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994427+0000", 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:26:00.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994430+0000", 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2729999999999998e-06 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994437+0000", 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9670000000000004e-06 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994443+0000", 2024-09-16T05:26:00.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7780000000000004e-06 2024-09-16T05:26:00.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995089+0000", 2024-09-16T05:26:00.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064609900000000002 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995143+0000", 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4042000000000002e-05 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995231+0000", 2024-09-16T05:26:00.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7798000000000002e-05 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995551+0000", 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000319847 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685849+0000", 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.690298254 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:00.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685893+0000", 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3368000000000002e-05 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685942+0000", 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9811000000000001e-05 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5461 2.25 2:a5bc1608:::smithi20457400-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e574)", 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:26:00.487 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 436.47488937200001, 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69165347799999999, 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5461 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994454+0000", 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994457+0000", 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.429e-06 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994463+0000", 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.994469+0000", 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4380000000000003e-06 2024-09-16T05:26:00.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995446+0000", 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00097678900000000004 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:43.995491+0000", 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5153e-05 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.685982+0000", 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69049136499999997 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686042+0000", 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9896999999999999e-05 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:44.686107+0000", 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5270000000000004e-05 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.492 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6242 2.5 2:a1d8c6d9:::smithi20457400-3:head [copy-from ver 913] snapc 103=[103,102,101,100,fe,fd,fc,fb,fa,f9,f7,f6,f2,f0,e7] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 372.65601576500001, 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47407041599999999, 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.493 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6242 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813327+0000", 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813328+0000", 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9399999999999993e-07 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813334+0000", 2024-09-16T05:26:00.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1619999999999996e-06 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813347+0000", 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2145e-05 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813354+0000", 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1230000000000002e-06 2024-09-16T05:26:00.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813414+0000", 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0489999999999999e-05 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813421+0000", 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1010000000000003e-06 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816226+0000", 2024-09-16T05:26:00.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002804647 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816349+0000", 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012330699999999999 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.281344+0000", 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.464994937 2024-09-16T05:26:00.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284684+0000", 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033402309999999999 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287216+0000", 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002531298 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287319+0000", 2024-09-16T05:26:00.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000102865 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.287398+0000", 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9116000000000006e-05 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6243 2.5 2:a1d8c6d9:::smithi20457400-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e638)", 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:26:00.500 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 372.65597038800001, 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.47072968100000001, 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6243 2024-09-16T05:26:00.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813373+0000", 2024-09-16T05:26:00.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1800000000000005e-07 2024-09-16T05:26:00.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813377+0000", 2024-09-16T05:26:00.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3699999999999999e-06 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813385+0000", 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5280000000000003e-06 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813392+0000", 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0600000000000002e-06 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813515+0000", 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012306 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.813521+0000", 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.203e-06 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816658+0000", 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003137365 2024-09-16T05:26:00.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:26:00.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:47.816689+0000", 2024-09-16T05:26:00.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0592000000000001e-05 2024-09-16T05:26:00.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.283968+0000", 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46727931 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284030+0000", 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2316000000000004e-05 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:48.284102+0000", 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2158999999999998e-05 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7270 2.1a 2:5827f14d:::smithi20457400-12:head [sparse-read 0~1919657,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 273.64583973600003, 2024-09-16T05:26:00.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.003199682, 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7270 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823503+0000", 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823506+0000", 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1099999999999999e-06 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823515+0000", 2024-09-16T05:26:00.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6219999999999998e-06 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823531+0000", 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.632e-05 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823544+0000", 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2508e-05 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832369+0000", 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088252939999999992 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832385+0000", 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5534999999999999e-05 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.822672+0000", 2024-09-16T05:26:00.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99028730499999995 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.822833+0000", 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016117400000000001 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.826703+0000", 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038698140000000001 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.514 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7271 2.1a 2:5827f14d:::smithi20457400-12:head [sparse-read 0~1919657] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 273.64578332600001, 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0044004120000001, 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.515 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7271 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823560+0000", 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6699999999999999e-07 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823563+0000", 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.824e-06 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823569+0000", 2024-09-16T05:26:00.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.319e-06 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823574+0000", 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0180000000000001e-06 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832415+0000", 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088402089999999999 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:26:00.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832423+0000", 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8089999999999999e-06 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827702+0000", 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.995278938 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827810+0000", 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107433 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827960+0000", 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150395 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7272 2.1a 2:5827f14d:::smithi20457400-12:head [read 0~3025491,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e737)", 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 273.64576271700003, 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0059436770000001, 2024-09-16T05:26:00.520 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7272 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823580+0000", 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823581+0000", 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1800000000000002e-07 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823600+0000", 2024-09-16T05:26:00.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9296999999999998e-05 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823607+0000", 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.063e-06 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.823612+0000", 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0699999999999997e-06 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832476+0000", 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088636289999999996 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:26.832485+0000", 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5359999999999998e-06 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.827992+0000", 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99550709199999998 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.828085+0000", 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3086999999999998e-05 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:27.829524+0000", 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001438585 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDPGPush(2.29 746/745 [PushOp(2:9662c29c:::smithi20457400-9:120, version: 737'1148, data_included: [2490368~196608,3108095~707292], data_size: 903900, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9662c29c:::smithi20457400-9:120@737'1148, size: 3815387, copy_subset: [(0, 3815387)], clone_subset: {2:9662c29c:::smithi20457400-9:116: []}, snapset: 130=[]:{116: [116], 120: [119], 125: [125], 130: [12f, 12c, 12a, 126]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3815387, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2490368, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 264.47966807799997, 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86202506199999995, 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989675+0000", 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.989676+0000", 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8999999999999997e-07 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:35.990550+0000", 2024-09-16T05:26:00.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087471100000000002 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850633+0000", 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.86008260599999997 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850658+0000", 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5537999999999998e-05 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850763+0000", 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104546 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.850778+0000", 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4603999999999999e-05 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:36.851700+0000", 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00092256700000000001 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.529 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.8.0:1707 2.1a e799/764 ondisk, result = 0)", 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 211.607472715, 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69806231100000005, 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861870+0000", 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861871+0000", 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7800000000000001e-07 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:28.861876+0000", 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4889999999999997e-06 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559739+0000", 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69786377399999999 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559746+0000", 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3409999999999999e-06 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559833+0000", 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7454000000000007e-05 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559840+0000", 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5390000000000004e-06 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:29.559933+0000", 2024-09-16T05:26:00.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2936000000000006e-05 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8288 2.25 2:a7ee7933:::smithi20457400-20:head [copy-from ver 750] snapc 15a=[15a] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 186.55127715200001, 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01619603, 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8288 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918066+0000", 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918068+0000", 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4499999999999998e-06 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918095+0000", 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6695000000000001e-05 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918117+0000", 2024-09-16T05:26:00.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2120000000000002e-05 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918127+0000", 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0074e-05 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918253+0000", 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125339 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918260+0000", 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9530000000000002e-06 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.921653+0000", 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033933230000000002 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.921758+0000", 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104949 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.927615+0000", 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058567039999999999 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.933643+0000", 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006028233 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934133+0000", 2024-09-16T05:26:00.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00049045800000000004 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934202+0000", 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8817000000000002e-05 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.934262+0000", 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9914999999999998e-05 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8289 2.25 2:a7ee7933:::smithi20457400-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e824)", 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 186.551178268, 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012771173, 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8289 2024-09-16T05:26:00.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918165+0000", 2024-09-16T05:26:00.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2500000000000005e-07 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918187+0000", 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1101999999999999e-05 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918198+0000", 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1139e-05 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.543 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918203+0000", 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5589999999999996e-06 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918361+0000", 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015787099999999999 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.918367+0000", 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2389999999999999e-06 2024-09-16T05:26:00.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.922539+0000", 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041721240000000001 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.922594+0000", 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4969000000000002e-05 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930701+0000", 2024-09-16T05:26:00.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081063300000000001 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930817+0000", 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116116 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:22:53.930936+0000", 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011909900000000001 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8964 2.5 2:a1d8c6d9:::smithi20457400-3:head [setxattr _header (57) in=7b,truncate 2773452] snapc 183=[183,181,180,17e,17c,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e900)", 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:10.090030+0000", 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.379313088, 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.090973712999999998, 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8964 2024-09-16T05:26:00.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090030+0000", 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090030+0000", 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090031+0000", 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3499999999999996e-07 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090034+0000", 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3979999999999999e-06 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090041+0000", 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2230000000000001e-06 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090047+0000", 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7359999999999998e-06 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090919+0000", 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087230900000000004 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090980+0000", 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0677000000000002e-05 2024-09-16T05:26:00.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0", 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.091076+0000", 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5878999999999995e-05 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.091952+0000", 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00087587299999999999 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.180909+0000", 2024-09-16T05:26:00.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.088957350000000004 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.180954+0000", 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5303000000000003e-05 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.181004+0000", 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9429999999999999e-05 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.552 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8965 2.5 2:a1d8c6d9:::smithi20457400-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e900)", 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:10.090055+0000", 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 110.379288175, 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.091122063000000003, 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.553 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8965 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090055+0000", 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090055+0000", 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090055+0000", 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.51e-07 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090058+0000", 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2910000000000002e-06 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090064+0000", 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2040000000000002e-06 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.090069+0000", 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4609999999999999e-06 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.091527+0000", 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014578320000000001 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.091608+0000", 2024-09-16T05:26:00.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1213999999999999e-05 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.181042+0000", 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.089433828000000007 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.181105+0000", 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2632000000000006e-05 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:10.181177+0000", 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2150000000000005e-05 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9405 2.25 2:a5bc1608:::smithi20457400-11:head [copy-from ver 1039] snapc 193=[191,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e936)", 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:46.327516+0000", 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 74.141827183999993, 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016089552999999999, 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.558 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9405 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327516+0000", 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327516+0000", 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327516+0000", 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999996e-07 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327521+0000", 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.206e-06 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327528+0000", 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1130000000000003e-06 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327533+0000", 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8740000000000003e-06 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327589+0000", 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6150999999999998e-05 2024-09-16T05:26:00.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327672+0000", 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3502000000000001e-05 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.336126+0000", 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084542309999999996 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.341290+0000", 2024-09-16T05:26:00.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051636030000000001 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.343539+0000", 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022492419999999998 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.343575+0000", 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5695999999999998e-05 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.343605+0000", 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0437999999999999e-05 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9406 2.25 2:a5bc1608:::smithi20457400-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e936)", 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:46.327549+0000", 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 74.14179421, 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012786812999999999, 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.564 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9406 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327549+0000", 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327549+0000", 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327549+0000", 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1600000000000001e-07 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327552+0000", 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4940000000000002e-06 2024-09-16T05:26:00.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327559+0000", 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9909999999999996e-06 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327564+0000", 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3890000000000004e-06 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327836+0000", 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027136599999999999 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for blocked object", 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.327893+0000", 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7389999999999998e-05 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.340178+0000", 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012284669 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.340245+0000", 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6937e-05 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:46.340336+0000", 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1061000000000001e-05 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.569 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9793 2.25 2:a7ee7933:::smithi20457400-20:head [append 0~0] snapc 1a2=[1a2,1a1,1a0,19f,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e963)", 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:25:14.454393+0000", 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 46.014949635999997, 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01158406, 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9793 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454393+0000", 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454393+0000", 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454396+0000", 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5280000000000002e-06 2024-09-16T05:26:00.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454409+0000", 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2543e-05 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454434+0000", 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5213000000000001e-05 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454447+0000", 2024-09-16T05:26:00.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.329e-05 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454567+0000", 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000120287 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454575+0000", 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0749999999999998e-06 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.461642+0000", 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0070669940000000001 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.461758+0000", 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115149 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.461979+0000", 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022116799999999999 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.465811+0000", 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038327349999999999 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.465860+0000", 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8081000000000003e-05 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.465908+0000", 2024-09-16T05:26:00.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8744e-05 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.465978+0000", 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9252999999999996e-05 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9794 2.25 2:a7ee7933:::smithi20457400-20:head [rollback 1a2] snapc 1a2=[1a2,1a1,1a0,19f,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e963)", 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:25:14.454481+0000", 2024-09-16T05:26:00.576 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 46.014861951999997, 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011586773999999999, 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9794 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454481+0000", 2024-09-16T05:26:00.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454481+0000", 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454482+0000", 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454485+0000", 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5829999999999998e-06 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454495+0000", 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3689999999999996e-06 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454501+0000", 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9560000000000002e-06 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454663+0000", 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016199000000000001 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454669+0000", 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6170000000000003e-06 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.463752+0000", 2024-09-16T05:26:00.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0090827149999999999 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.463866+0000", 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000113619 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.464056+0000", 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019013499999999999 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.465846+0000", 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001790217 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.465987+0000", 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014084300000000001 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.466031+0000", 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4116999999999998e-05 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.466068+0000", 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7076999999999998e-05 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9795 2.25 2:a7ee7933:::smithi20457400-20:head [append 0~0] snapc 1a2=[1a2,1a1,1a0,19f,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e963)", 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:25:14.454509+0000", 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 46.014834276999999, 2024-09-16T05:26:00.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012056186999999999, 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9795 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:00.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454509+0000", 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454509+0000", 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454509+0000", 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8000000000000001e-07 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454512+0000", 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.553e-06 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454523+0000", 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1127999999999999e-05 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454529+0000", 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6300000000000003e-06 2024-09-16T05:26:00.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454706+0000", 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017737699999999999 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.454712+0000", 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1789999999999996e-06 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.464987+0000", 2024-09-16T05:26:00.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010275358 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.465085+0000", 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7572999999999996e-05 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.465228+0000", 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014319199999999999 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.466078+0000", 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00084988100000000003 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.466480+0000", 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00040220300000000001 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.466519+0000", 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.875e-05 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:00.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:14.466565+0000", 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5982999999999997e-05 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:00.590 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:26:00.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:00.612 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:00.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:00.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:00.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:00.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:00.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:00.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:00.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:00.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:00.947 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:00.947 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:00.948 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:26:00.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:00 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:00.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:00 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3750642141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:00.948 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:00 smithi032 ceph-mon[27388]: osdmap e978: 12 total, 10 up, 12 in 2024-09-16T05:26:00.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:00.968 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:00 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:00.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:00 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3750642141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:00.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:00 smithi196 ceph-mon[30802]: osdmap e978: 12 total, 10 up, 12 in 2024-09-16T05:26:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:00.990 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:00.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:00.991 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:00.991 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:01.010 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:01.010 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:01.032 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:01.053 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:01.054 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:01.054 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:01.074 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:01.075 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:01.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:01.096 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:01.096 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:01.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:01.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:01.117 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:01.138 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:01.139 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:01.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:00 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:01.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:00 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3750642141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:01.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:00 smithi204 ceph-mon[30933]: osdmap e978: 12 total, 10 up, 12 in 2024-09-16T05:26:01.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:01.237 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:01.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:01.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:01.361 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:26:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:01.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:01.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:01.487 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:00 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:00.991+0000 7f6af1ad6640 -1 osd.6 978 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:01.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:01.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:01.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:01.602 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:01.613 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:01.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:01.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:01.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:01.736 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:01.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:01.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:01.860 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:01.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:01.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:01.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:01.986 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:01.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:01 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:01 smithi196 ceph-mon[30802]: pgmap v1645: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:01 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:01 smithi196 ceph-mon[30802]: Health check update: 2 slow ops, oldest one blocked for 140 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:01.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1754227590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:02.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:01 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:02.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:01 smithi204 ceph-mon[30933]: pgmap v1645: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:02.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:01 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:02.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:01 smithi204 ceph-mon[30933]: Health check update: 2 slow ops, oldest one blocked for 140 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:02.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1754227590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:02.057 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:02.057 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:02.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:02.111 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:02.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:02.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:01 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:01 smithi032 ceph-mon[27388]: pgmap v1645: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:01 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:01 smithi032 ceph-mon[27388]: Health check update: 2 slow ops, oldest one blocked for 140 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1754227590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:02.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:02.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:02.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:02.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:02.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:02.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:02.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:26:02.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:02.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:02.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:02.496 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [1, 7] live_osds: [0, 3, 4, 5, 6, 8, 9, 10, 11, 2] 2024-09-16T05:26:02.496 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:26:02.496 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T05:26:02.496 INFO:tasks.thrashosds.thrasher:Setting osd 10 primary_affinity to 0.991936 2024-09-16T05:26:02.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 10 0.991935961092332 2024-09-16T05:26:02.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:02.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:02.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:02.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:26:02.669 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:02.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:02.746 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:02.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:02.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:02.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:02.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:02.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:02 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:02.941 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:02.975 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:02.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:02 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:02.996 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:02.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:03.069 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:02 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:03.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:03.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:03.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:03.211 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:03.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:03.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:03.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:03.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:03.373 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:03.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:03.439 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:03.440 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:03.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:03.497 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:03.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:03.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:03.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:03.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:03.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:26:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:26:03.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:03.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:03.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:03.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:03.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:03.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:03 smithi032 ceph-mon[27388]: pgmap v1646: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:03 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:03 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:03.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/684475001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:03.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:03.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:03 smithi196 ceph-mon[30802]: pgmap v1646: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:03.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:03 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:03.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:03 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:03.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/684475001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:04.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:04.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:04.044 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:03 smithi204 ceph-mon[30933]: pgmap v1646: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:04.044 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:03 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:04.044 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:03 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:04.044 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/684475001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:04.072 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:26:04.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:04.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:04.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:04.234 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:04.257 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:04.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:04.311 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:04.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:04.383 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:04.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:04.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:04.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:04.507 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:04.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:04.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:04.631 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:04.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:04.660 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:04.728 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.10 primary-affinity to 0.991936 (8650072) 2024-09-16T05:26:04.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:04.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:04.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:04.831 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:04.831 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:04.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:04.878 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:04.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:04.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/684475001' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:04.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:04 smithi032 ceph-mon[27388]: osdmap e979: 12 total, 10 up, 12 in 2024-09-16T05:26:04.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/977559343' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.991935961092332}]: dispatch 2024-09-16T05:26:04.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:04.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/684475001' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:04.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:04 smithi196 ceph-mon[30802]: osdmap e979: 12 total, 10 up, 12 in 2024-09-16T05:26:04.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/977559343' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.991935961092332}]: dispatch 2024-09-16T05:26:05.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:05.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:05.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:05.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/684475001' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:05.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:04 smithi204 ceph-mon[30933]: osdmap e979: 12 total, 10 up, 12 in 2024-09-16T05:26:05.128 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/977559343' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.991935961092332}]: dispatch 2024-09-16T05:26:05.130 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:05.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:05.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:05.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:05.257 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:05.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:05.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:05.358 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:05.358 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:05.358 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:26:05.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:05.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:05.379 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:05.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:05.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:05.400 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:05.400 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:05.422 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:05.422 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:05.444 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:05.444 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:05.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:26:05.465 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:05.466 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:05.466 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:05.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:05.488 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:05.489 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:05.504 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:05.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:05.510 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:05.510 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:05.531 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:05.532 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:05.553 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:05.553 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:05.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:05.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:05.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:05.686 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:05.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:05.758 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:05.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:05.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:05.883 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:05.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:05.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: pgmap v1648: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:26:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:26:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: 2.25 scrub ok 2024-09-16T05:26:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:05.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 140 sec, osd.6 has slow ops) 2024-09-16T05:26:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/977559343' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.991935961092332}]': finished 2024-09-16T05:26:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: osdmap e980: 12 total, 10 up, 12 in 2024-09-16T05:26:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: 2.14 scrub starts 2024-09-16T05:26:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: 2.14 scrub ok 2024-09-16T05:26:05.922 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3635377553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:05.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: pgmap v1648: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:26:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:26:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: 2.25 scrub ok 2024-09-16T05:26:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 140 sec, osd.6 has slow ops) 2024-09-16T05:26:05.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/977559343' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.991935961092332}]': finished 2024-09-16T05:26:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: osdmap e980: 12 total, 10 up, 12 in 2024-09-16T05:26:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: 2.14 scrub starts 2024-09-16T05:26:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: 2.14 scrub ok 2024-09-16T05:26:05.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3635377553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:05.989 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:05 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:05.924+0000 7f6af1ad6640 -1 osd.6 980 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:06.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:06.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:06.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: pgmap v1648: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:06.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:26:06.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:26:06.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: 2.25 scrub ok 2024-09-16T05:26:06.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:06.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:06.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 140 sec, osd.6 has slow ops) 2024-09-16T05:26:06.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/977559343' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 0.991935961092332}]': finished 2024-09-16T05:26:06.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: osdmap e980: 12 total, 10 up, 12 in 2024-09-16T05:26:06.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: 2.14 scrub starts 2024-09-16T05:26:06.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: 2.14 scrub ok 2024-09-16T05:26:06.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3635377553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:06.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:06.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:06.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:06.198 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:26:06.198 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:26:06.198 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:26:06.198 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:26:06.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:06.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:06.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:06.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:06.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:06.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:06.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:06.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:06.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:26:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:06.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:06.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:06.765 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:06.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:06.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:06 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:26:06.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:06 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:26:06.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:06 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:26:06.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:06 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:26:06.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:06 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:06.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:06 smithi204 ceph-mon[30933]: 2.22 scrub starts 2024-09-16T05:26:06.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:06 smithi204 ceph-mon[30933]: 2.22 scrub ok 2024-09-16T05:26:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:06.838 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:06.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:26:06.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:06.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:06.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:06 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:26:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:06 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:26:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:06 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:26:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:06 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:26:06.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:06 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:06.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:06 smithi032 ceph-mon[27388]: 2.22 scrub starts 2024-09-16T05:26:06.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:06 smithi032 ceph-mon[27388]: 2.22 scrub ok 2024-09-16T05:26:06.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:07.014 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:07.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:07.094 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:07.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:07.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:07.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:07.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:06 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:06.874+0000 7f6af1ad6640 -1 osd.6 981 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:06 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:26:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:06 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:26:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:06 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:26:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:06 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:26:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:06 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:06 smithi196 ceph-mon[30802]: 2.22 scrub starts 2024-09-16T05:26:07.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:06 smithi196 ceph-mon[30802]: 2.22 scrub ok 2024-09-16T05:26:07.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:07.266 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:07.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:07.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:07.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:07.390 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:07.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:07.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:07.514 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:07.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:07.588 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:26:07.588 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:26:07.589 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:26:07.589 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:26:07.589 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:26:07.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:07.636 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:07.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:07.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:07.746 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:26:07.760 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:07.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:07.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:07.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:07.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:07.987 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:08.010 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:08.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:08.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:08.121 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:08.122 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:08.122 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:08.122 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: pgmap v1650: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:08.122 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:26:08.122 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:26:08.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: 1.17 scrub starts 2024-09-16T05:26:08.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: 1.17 scrub ok 2024-09-16T05:26:08.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: osdmap e981: 12 total, 10 up, 12 in 2024-09-16T05:26:08.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: 2.3e scrub starts 2024-09-16T05:26:08.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: 2.3e scrub ok 2024-09-16T05:26:08.123 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:07 smithi032 ceph-mon[27388]: from='client.17826 172.21.15.32:0/3838169854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:08.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:08.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: pgmap v1650: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: 1.17 scrub starts 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: 1.17 scrub ok 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: osdmap e981: 12 total, 10 up, 12 in 2024-09-16T05:26:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: 2.3e scrub starts 2024-09-16T05:26:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: 2.3e scrub ok 2024-09-16T05:26:08.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:07 smithi204 ceph-mon[30933]: from='client.17826 172.21.15.32:0/3838169854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:08.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:26:08.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:07.894+0000 7f6af1ad6640 -1 osd.6 981 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: pgmap v1650: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: 1.17 scrub starts 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: 1.17 scrub ok 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: osdmap e981: 12 total, 10 up, 12 in 2024-09-16T05:26:08.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: 2.3e scrub starts 2024-09-16T05:26:08.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: 2.3e scrub ok 2024-09-16T05:26:08.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:07 smithi196 ceph-mon[30802]: from='client.17826 172.21.15.32:0/3838169854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:08.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:08.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:08.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:08.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:08.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:08.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:08.469 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:08.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:08.517 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:08.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:08.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:08.644 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:08.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:08.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:08.768 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:08.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:08.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:26:08.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:08.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: 2.55 scrub starts 2024-09-16T05:26:08.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: 2.55 scrub ok 2024-09-16T05:26:08.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:08.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: from='client.17826 172.21.15.32:0/3838169854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:08.865 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: osdmap e982: 12 total, 10 up, 12 in 2024-09-16T05:26:08.866 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: 2.a scrub starts 2024-09-16T05:26:08.866 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: 2.a scrub ok 2024-09-16T05:26:08.866 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: 2.51 scrub starts 2024-09-16T05:26:08.866 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:08 smithi032 ceph-mon[27388]: 2.51 scrub ok 2024-09-16T05:26:08.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:08.891 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:08.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:08.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:08.905 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: 2.55 scrub starts 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: 2.55 scrub ok 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: from='client.17826 172.21.15.32:0/3838169854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: osdmap e982: 12 total, 10 up, 12 in 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: 2.a scrub starts 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: 2.a scrub ok 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: 2.51 scrub starts 2024-09-16T05:26:08.906 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:08 smithi204 ceph-mon[30933]: 2.51 scrub ok 2024-09-16T05:26:08.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:09.009 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:26:09.009 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:26:09.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:26:09.009 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:26:09.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.009 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3597 2.4c 2.118008cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:26:09.009 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:26:09.009 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 582.26144365899995, 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98230618599999997, 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3597 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746118+0000", 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746119+0000", 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9999999999999995e-07 2024-09-16T05:26:09.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746124+0000", 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3349999999999997e-06 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746136+0000", 2024-09-16T05:26:09.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2339e-05 2024-09-16T05:26:09.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746143+0000", 2024-09-16T05:26:09.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3940000000000001e-06 2024-09-16T05:26:09.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746201+0000", 2024-09-16T05:26:09.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8529000000000002e-05 2024-09-16T05:26:09.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:26:09.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746206+0000", 2024-09-16T05:26:09.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0250000000000002e-06 2024-09-16T05:26:09.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728413+0000", 2024-09-16T05:26:09.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98220633700000004 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728424+0000", 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1226999999999999e-05 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.016 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3598 2.4c 2.118008cc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e437)", 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 582.26140209499999, 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.982592205, 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3598 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746159+0000", 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746160+0000", 2024-09-16T05:26:09.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6899999999999997e-07 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746162+0000", 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.323e-06 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746170+0000", 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4339999999999992e-06 2024-09-16T05:26:09.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746176+0000", 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1649999999999997e-06 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746227+0000", 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1848999999999998e-05 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:26.746232+0000", 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5619999999999997e-06 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728741+0000", 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.98250897100000001 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:27.728751+0000", 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0332e-05 2024-09-16T05:26:09.021 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.10.0:319 2.4a e458/325 ondisk, result = 0)", 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 560.19108956699995, 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.733792216, 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.022 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816472+0000", 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816473+0000", 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.206e-06 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:48.816480+0000", 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4789999999999999e-06 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550063+0000", 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.73358232899999998 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550069+0000", 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3489999999999997e-06 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550155+0000", 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5490999999999999e-05 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550163+0000", 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1859999999999992e-06 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:49.550264+0000", 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010117599999999999 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.026 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3966 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 645534~783707] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:26:09.027 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:26:09.027 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 551.11856280400002, 2024-09-16T05:26:09.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.228671191, 2024-09-16T05:26:09.027 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.027 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.027 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3966 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.888998+0000", 2024-09-16T05:26:09.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889000+0000", 2024-09-16T05:26:09.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.745e-06 2024-09-16T05:26:09.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889398+0000", 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000397644 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889415+0000", 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7368e-05 2024-09-16T05:26:09.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889440+0000", 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4358999999999998e-05 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889496+0000", 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6168999999999997e-05 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.889500+0000", 2024-09-16T05:26:09.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0579999999999999e-06 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107487+0000", 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21798717200000001 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107565+0000", 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7715999999999999e-05 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.107734+0000", 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016950099999999999 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111344+0000", 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036100160000000002 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117576+0000", 2024-09-16T05:26:09.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062321929999999996 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117628+0000", 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.189e-05 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117670+0000", 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1359999999999997e-05 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3967 2.1d 2:ba7e58cf:::smithi20457400-26:head [write 2103939~638517] snapc a9=[a9] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 551.11163777399997, 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.221880365, 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3967 2024-09-16T05:26:09.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895923+0000", 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.895924+0000", 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.85e-07 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896200+0000", 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027563199999999999 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896212+0000", 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1671999999999999e-05 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896217+0000", 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3970000000000002e-06 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896274+0000", 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6740000000000002e-05 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.896280+0000", 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109156+0000", 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.212875278 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109242+0000", 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6163999999999997e-05 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:26:09.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.109378+0000", 2024-09-16T05:26:09.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013637400000000001 2024-09-16T05:26:09.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:09.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111350+0000", 2024-09-16T05:26:09.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019716429999999999 2024-09-16T05:26:09.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117716+0000", 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063661830000000001 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117776+0000", 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9753999999999997e-05 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.117804+0000", 2024-09-16T05:26:09.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.817e-05 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:3970 2.1d 2:ba7e58cf:::smithi20457400-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e468)", 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 551.107710129, 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.218352989, 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.042 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3970 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899851+0000", 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899852+0000", 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8299999999999997e-07 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899854+0000", 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.463e-06 2024-09-16T05:26:09.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899861+0000", 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0079999999999996e-06 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899866+0000", 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899949+0000", 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2713000000000006e-05 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:57.899954+0000", 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7400000000000004e-06 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111261+0000", 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.211307514 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.111332+0000", 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1198999999999994e-05 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118056+0000", 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067238530000000001 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118121+0000", 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4770000000000005e-05 2024-09-16T05:26:09.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:16:58.118204+0000", 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3051000000000002e-05 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:4971 2.1d 2:ba7e58cf:::smithi20457400-26:head [stat,delete] snapc cc=[cb,ca,c9,c7,c5,c4,c3,c2,c1] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:26:09.048 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 485.80912869000002, 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.70425506000000004, 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4971 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198433+0000", 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198434+0000", 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8300000000000001e-06 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198443+0000", 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198460+0000", 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7229999999999999e-05 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198467+0000", 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1409999999999999e-06 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198531+0000", 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3371e-05 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.198539+0000", 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8930000000000005e-06 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891869+0000", 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.69333044700000002 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.891964+0000", 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4990999999999999e-05 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892092+0000", 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012792599999999999 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.892842+0000", 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00075013 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902619+0000", 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0097773840000000001 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902658+0000", 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8167999999999998e-05 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:03.902688+0000", 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9991000000000001e-05 2024-09-16T05:26:09.055 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5379 2.1c 2:3a3554ba:::smithi20457400-48:head [setxattr _header (58) in=7b,truncate 2823741] snapc e0=[e0,df,dd,dc,db,da,d9,d8,d7,d4,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 448.21279452499999, 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46119776499999998, 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.056 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5379 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794767+0000", 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9500000000000002e-07 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794770+0000", 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9890000000000001e-06 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794783+0000", 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2281e-05 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794788+0000", 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.011e-06 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795397+0000", 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060981000000000002 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795462+0000", 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4869000000000005e-05 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795576+0000", 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0001141 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795836+0000", 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00026010999999999997 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255870+0000", 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46003327399999999 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255915+0000", 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5064000000000001e-05 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255964+0000", 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9662e-05 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5380 2.1c 2:3a3554ba:::smithi20457400-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e571)", 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:26:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 448.21275997100003, 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46133110599999999, 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5380 2024-09-16T05:26:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794801+0000", 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794802+0000", 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7599999999999998e-07 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794804+0000", 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0590000000000001e-06 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794810+0000", 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7190000000000004e-06 2024-09-16T05:26:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.794815+0000", 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6700000000000002e-06 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795747+0000", 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00093155799999999997 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.795821+0000", 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3975999999999998e-05 2024-09-16T05:26:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256004+0000", 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46018367199999999 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256065+0000", 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0299e-05 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.256132+0000", 2024-09-16T05:26:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7776999999999998e-05 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:5379 2.1c e571/532 ondisk, result = 0)", 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 448.21072628500002, 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45914334200000001, 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:09.069 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796835+0000", 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0800000000000001e-07 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:40.796837+0000", 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9850000000000001e-06 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255760+0000", 2024-09-16T05:26:09.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45892265399999999 2024-09-16T05:26:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255769+0000", 2024-09-16T05:26:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3449999999999995e-06 2024-09-16T05:26:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255857+0000", 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7547999999999996e-05 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255865+0000", 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6810000000000004e-06 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:41.255978+0000", 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112821 2024-09-16T05:26:09.073 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:5901 2.1d 2:ba754fcb:::smithi20457400-30:head [stat,delete] snapc f3=[f3,f2,f0,ef,ee,ed,ec,e8,e7,e5,e0,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 411.10425405000001, 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79482897200000002, 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.074 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.075 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:09.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:09.076 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.076 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 5901 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903307+0000", 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903309+0000", 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4699999999999999e-06 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903314+0000", 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903330+0000", 2024-09-16T05:26:09.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5770999999999999e-05 2024-09-16T05:26:09.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903340+0000", 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0498e-05 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903402+0000", 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1385999999999998e-05 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903513+0000", 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111113 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.903707+0000", 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019455500000000001 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:09.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:17.904477+0000", 2024-09-16T05:26:09.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076932900000000002 2024-09-16T05:26:09.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:09.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698051+0000", 2024-09-16T05:26:09.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.79357449800000002 2024-09-16T05:26:09.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698095+0000", 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3452999999999999e-05 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:18.698136+0000", 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1482e-05 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.7.0:1084 2.3c e626/625 2:3d4bedc8:::smithi20457400-22:ee v 626'652, mlcod=624'646)", 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 391.53330054200001, 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45487590700000002, 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:26:09.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:26:09.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474261+0000", 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8800000000000002e-07 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.474267+0000", 2024-09-16T05:26:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.417e-06 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928275+0000", 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45400790800000002 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928291+0000", 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5852999999999999e-05 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928455+0000", 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164026 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.928479+0000", 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4258000000000001e-05 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929099+0000", 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000620117 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:37.929137+0000", 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7639999999999999e-05 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6377 2.1d 2:ba21f1a9:::smithi20457400-14:head [setxattr _header (58) in=7b,truncate 3059072] snapc 107=[107,106,104,102,101,fb,fa,f9,f7,f2,e7] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 369.12052521599998, 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46418828400000001, 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.089 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6377 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887036+0000", 2024-09-16T05:26:09.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887037+0000", 2024-09-16T05:26:09.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3600000000000004e-07 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887040+0000", 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5039999999999998e-06 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887047+0000", 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0330000000000003e-06 2024-09-16T05:26:09.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887053+0000", 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1430000000000003e-06 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888098+0000", 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001044866 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888199+0000", 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101143 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 7", 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888364+0000", 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000164403 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888873+0000", 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00050974699999999996 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351123+0000", 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46224975499999998 2024-09-16T05:26:09.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351169+0000", 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6316000000000002e-05 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351224+0000", 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4837999999999999e-05 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.096 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:6378 2.1d 2:ba21f1a9:::smithi20457400-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e650)", 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 369.12049828800002, 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46436558900000002, 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:09.097 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 6378 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887063+0000", 2024-09-16T05:26:09.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887064+0000", 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3799999999999997e-07 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887071+0000", 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9670000000000001e-06 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887077+0000", 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9739999999999999e-06 2024-09-16T05:26:09.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.887081+0000", 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.405e-06 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888760+0000", 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016788319999999999 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:59.888855+0000", 2024-09-16T05:26:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5670999999999999e-05 2024-09-16T05:26:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351265+0000", 2024-09-16T05:26:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.46241011799999998 2024-09-16T05:26:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351353+0000", 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7416000000000005e-05 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:00.351429+0000", 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5667999999999994e-05 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(1.7,GRANT,e:689,from:6,reservation_nonce:0)", 2024-09-16T05:26:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 329.55024945100001, 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388634586, 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457312+0000", 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457313+0000", 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0020000000000001e-06 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.457317+0000", 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.279e-06 2024-09-16T05:26:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845738+0000", 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.388420879 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845745+0000", 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1849999999999998e-06 2024-09-16T05:26:09.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845793+0000", 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8300000000000002e-05 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845814+0000", 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0353999999999999e-05 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:20:39.845946+0000", 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013258700000000001 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.6.49:26 2.5d 2.d3f2ae5d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e867)", 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 152.13472490000001, 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99584287000000005, 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-16T05:26:09.107 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872836+0000", 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872838+0000", 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7540000000000001e-06 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872846+0000", 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0790000000000001e-06 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872866+0000", 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9624999999999999e-05 2024-09-16T05:26:09.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872875+0000", 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4790000000000002e-06 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872926+0000", 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0260000000000003e-05 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:36.872935+0000", 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.346e-06 2024-09-16T05:26:09.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:37.868662+0000", 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99572677300000001 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:37.868679+0000", 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7554000000000002e-05 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.111 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8820 2.5e 2:7b642727:::smithi20457400-40:head [copy-from ver 549] snapc 175=[16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e877)", 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 142.14779821299999, 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025559208, 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8820 2024-09-16T05:26:09.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859763+0000", 2024-09-16T05:26:09.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859765+0000", 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1409999999999999e-06 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859774+0000", 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6430000000000001e-06 2024-09-16T05:26:09.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859789+0000", 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5441999999999999e-05 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859798+0000", 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9239999999999996e-06 2024-09-16T05:26:09.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859875+0000", 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6508000000000007e-05 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.859883+0000", 2024-09-16T05:26:09.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2190000000000004e-06 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.862172+0000", 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0022892210000000001 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.862279+0000", 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106606 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 6", 2024-09-16T05:26:09.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.873916+0000", 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011637412999999999 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.881734+0000", 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078177669999999998 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885232+0000", 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034977849999999998 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885271+0000", 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9079000000000003e-05 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:46.885322+0000", 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1459999999999999e-05 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9415 2.4c 2:33100188:::smithi20457400-19:head [append 0~0] snapc 194=[194,191,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e937)", 2024-09-16T05:26:09.119 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:47.301227+0000", 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 81.706334753999997, 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046627700000000001, 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9415 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301227+0000", 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301227+0000", 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301227+0000", 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8599999999999998e-07 2024-09-16T05:26:09.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301230+0000", 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7520000000000001e-06 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301236+0000", 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4799999999999998e-06 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301242+0000", 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3430000000000003e-06 2024-09-16T05:26:09.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301402+0000", 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016081900000000001 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.301408+0000", 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9769999999999997e-06 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.304617+0000", 2024-09-16T05:26:09.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032087980000000001 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.304701+0000", 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3588e-05 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8", 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.304861+0000", 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016004499999999999 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:26:09.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.305239+0000", 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037855799999999998 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.305797+0000", 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00055756399999999998 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.305846+0000", 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8949999999999997e-05 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:47.305889+0000", 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3309999999999997e-05 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:9619 2.31 e945/529 2:8c1ba486:::smithi20457400-2:head v 945'424, mlcod=941'422)", 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:56.594305+0000", 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.413256626000006, 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0052787659999999998, 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.594305+0000", 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.594305+0000", 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.594305+0000", 2024-09-16T05:26:09.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9400000000000004e-07 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597567+0000", 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032611010000000002 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597607+0000", 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0361000000000002e-05 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597616+0000", 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2199999999999998e-06 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597685+0000", 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9208000000000006e-05 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.597709+0000", 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4142999999999999e-05 2024-09-16T05:26:09.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.599544+0000", 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001834565 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:56.599583+0000", 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9373999999999999e-05 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:9821 2.4d e964/529 2:b21a57e7:::smithi20457400-25:head v 964'777, mlcod=936'773)", 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:25:16.019219+0000", 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.988341833, 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023843089999999998, 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.019219+0000", 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.019219+0000", 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.019222+0000", 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0810000000000001e-06 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.020006+0000", 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000784091 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.020059+0000", 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3063999999999999e-05 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.020078+0000", 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9584000000000001e-05 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.020146+0000", 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7997999999999995e-05 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.020166+0000", 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9318999999999999e-05 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:26:09.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.021553+0000", 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013876019999999999 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:16.021604+0000", 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0569999999999999e-05 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.135 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.136 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:26:09.136 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:26:09.136 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:26:09.136 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:09.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:08.873+0000 7f6af1ad6640 -1 osd.6 981 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: 2.55 scrub starts 2024-09-16T05:26:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: 2.55 scrub ok 2024-09-16T05:26:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:09.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: from='client.17826 172.21.15.32:0/3838169854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: osdmap e982: 12 total, 10 up, 12 in 2024-09-16T05:26:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: 2.a scrub starts 2024-09-16T05:26:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: 2.a scrub ok 2024-09-16T05:26:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: 2.51 scrub starts 2024-09-16T05:26:09.140 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:08 smithi196 ceph-mon[30802]: 2.51 scrub ok 2024-09-16T05:26:09.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:09.199 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:09.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:09.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:09.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:09.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:09.448 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:09.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:09.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:09.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:09.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:09.656 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:09.656 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:26:09.678 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:09.678 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:09.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:09.699 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:09.699 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:09.702 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:09.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:09.720 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:09.741 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:09.742 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:09.780 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:09.780 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:09.780 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:09.801 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:09.801 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:09.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:09.825 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:09.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:09.839 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:09.840 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:09.863 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:09.864 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:09.884 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:09.885 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:09.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:09.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:09.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:09.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:26:09.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:09 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:09 smithi196 ceph-mon[30802]: 2.3a scrub starts 2024-09-16T05:26:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:09 smithi196 ceph-mon[30802]: 2.3a scrub ok 2024-09-16T05:26:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:09 smithi196 ceph-mon[30802]: pgmap v1653: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:09 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 150 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:09 smithi196 ceph-mon[30802]: 2.34 scrub starts 2024-09-16T05:26:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:09 smithi196 ceph-mon[30802]: 2.34 scrub ok 2024-09-16T05:26:09.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2092375728' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:10.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:10.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:10.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:09 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:09 smithi032 ceph-mon[27388]: 2.3a scrub starts 2024-09-16T05:26:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:09 smithi032 ceph-mon[27388]: 2.3a scrub ok 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:09 smithi032 ceph-mon[27388]: pgmap v1653: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:09 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 150 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:09 smithi032 ceph-mon[27388]: 2.34 scrub starts 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:09 smithi032 ceph-mon[27388]: 2.34 scrub ok 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2092375728' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:09 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:09 smithi204 ceph-mon[30933]: 2.3a scrub starts 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:09 smithi204 ceph-mon[30933]: 2.3a scrub ok 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:09 smithi204 ceph-mon[30933]: pgmap v1653: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:09 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 150 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:09 smithi204 ceph-mon[30933]: 2.34 scrub starts 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:09 smithi204 ceph-mon[30933]: 2.34 scrub ok 2024-09-16T05:26:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2092375728' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:10.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:10.194 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:10.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:10.205 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:10.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:10.320 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:10.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:10.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:10.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:10.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:10.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:10.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:26:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:10.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:10.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:10.607 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:10.607 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:10.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:10.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:10.689 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:10.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:10.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:10.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:10.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:10.816 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:10.914 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:10 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:10 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:26:10.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:10 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:26:10.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:10 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:10.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:10 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:26:10.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:10 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:26:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:10.936 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:10.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:11.014 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [1, 7] live_osds: [0, 3, 4, 5, 6, 8, 9, 10, 11, 2] 2024-09-16T05:26:11.014 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:26:11.014 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-16T05:26:11.014 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-09-16T05:26:11.015 DEBUG:teuthology.orchestra.run.smithi032:osd.1> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.1.service 2024-09-16T05:26:11.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl start ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.1 2024-09-16T05:26:11.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:11.061 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:11.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:11.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:11.185 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:11.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:11.217 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:11 smithi032 systemd[1]: Starting Ceph osd.1 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:26:11.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:10 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:11.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:10 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:26:11.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:10 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:26:11.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:11.310 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:11.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:11.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:26:11.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:11.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:11.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:11.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:11.561 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:11.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:11.612 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:11.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:11.686 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:11.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:11.751 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:26:11.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:11.809 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:11.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:11.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:11.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-mon[27388]: pgmap v1654: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:11.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-mon[27388]: 2.5 scrub starts 2024-09-16T05:26:11.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-mon[27388]: 2.5 scrub ok 2024-09-16T05:26:11.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-mon[27388]: 2.12 scrub starts 2024-09-16T05:26:11.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-mon[27388]: 2.12 scrub ok 2024-09-16T05:26:11.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-mon[27388]: 1.8 deep-scrub starts 2024-09-16T05:26:11.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-mon[27388]: 1.8 deep-scrub ok 2024-09-16T05:26:11.913 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1427546495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:11.914 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:11 smithi032 podman[195363]: 2024-09-16 05:26:11.447634405 +0000 UTC m=+0.017005095 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:26:11.914 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:11 smithi032 podman[195363]: 2024-09-16 05:26:11.567216514 +0000 UTC m=+0.136587195 container create f24a350bfa502cb001b5bed83f52fbd9af72532a7df198283ae0b1c55a8e453d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:26:11.914 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:11 smithi032 podman[195363]: 2024-09-16 05:26:11.769573899 +0000 UTC m=+0.338944580 container init f24a350bfa502cb001b5bed83f52fbd9af72532a7df198283ae0b1c55a8e453d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-16T05:26:11.914 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:11 smithi032 podman[195363]: 2024-09-16 05:26:11.774884433 +0000 UTC m=+0.344255125 container start f24a350bfa502cb001b5bed83f52fbd9af72532a7df198283ae0b1c55a8e453d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-09-16T05:26:11.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:11.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:12.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:12.043 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:11 smithi204 ceph-mon[30933]: pgmap v1654: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:12.043 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:11 smithi204 ceph-mon[30933]: 2.5 scrub starts 2024-09-16T05:26:12.043 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:11 smithi204 ceph-mon[30933]: 2.5 scrub ok 2024-09-16T05:26:12.043 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:11 smithi204 ceph-mon[30933]: 2.12 scrub starts 2024-09-16T05:26:12.043 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:11 smithi204 ceph-mon[30933]: 2.12 scrub ok 2024-09-16T05:26:12.043 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:11 smithi204 ceph-mon[30933]: 1.8 deep-scrub starts 2024-09-16T05:26:12.043 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:11 smithi204 ceph-mon[30933]: 1.8 deep-scrub ok 2024-09-16T05:26:12.043 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1427546495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:12.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:12.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:12.143 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:12.144 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:12.160 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:12.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:26:12.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:11 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:26:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:12.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:12.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:11 smithi196 ceph-mon[30802]: pgmap v1654: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:11 smithi196 ceph-mon[30802]: 2.5 scrub starts 2024-09-16T05:26:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:11 smithi196 ceph-mon[30802]: 2.5 scrub ok 2024-09-16T05:26:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:11 smithi196 ceph-mon[30802]: 2.12 scrub starts 2024-09-16T05:26:12.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:11 smithi196 ceph-mon[30802]: 2.12 scrub ok 2024-09-16T05:26:12.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:11 smithi196 ceph-mon[30802]: 1.8 deep-scrub starts 2024-09-16T05:26:12.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:11 smithi196 ceph-mon[30802]: 1.8 deep-scrub ok 2024-09-16T05:26:12.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1427546495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:12.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:12.306 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:12.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:12.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:12.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:12.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:12.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:12.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:12.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:12.570 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 podman[195363]: 2024-09-16 05:26:12.195720056 +0000 UTC m=+0.765090753 container attach f24a350bfa502cb001b5bed83f52fbd9af72532a7df198283ae0b1c55a8e453d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-8921c8e) 2024-09-16T05:26:12.571 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:26:12.571 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:26:12.571 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T05:26:12.571 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T05:26:12.571 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-16T05:26:12.571 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-16T05:26:12.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:26:12.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:12.684 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:12.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:12.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:26:12.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:12.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:12.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:12.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:26:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:26:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1427546495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:12.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-mon[27388]: osdmap e983: 12 total, 10 up, 12 in 2024-09-16T05:26:12.920 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate[195396]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 bash[195363]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-16T05:26:12.921 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:12 smithi032 podman[195363]: 2024-09-16 05:26:12.719683709 +0000 UTC m=+1.289054392 container died f24a350bfa502cb001b5bed83f52fbd9af72532a7df198283ae0b1c55a8e453d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-09-16T05:26:12.934 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:12.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:13.019 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:13.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:13.058 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:13.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:13.139 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:12 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:26:13.139 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:12 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:26:13.139 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:12 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:13.139 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1427546495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:13.139 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:12 smithi204 ceph-mon[30933]: osdmap e983: 12 total, 10 up, 12 in 2024-09-16T05:26:13.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:13.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:13.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:13.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:13.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:12 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:26:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:12 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:26:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:12 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1427546495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:13.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:12 smithi196 ceph-mon[30802]: osdmap e983: 12 total, 10 up, 12 in 2024-09-16T05:26:13.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:13.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:13.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:13.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:13.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:13.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:13.492 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:13.493 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:13.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:13.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:13.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:13.670 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:13 smithi032 podman[195566]: 2024-09-16 05:26:13.361146491 +0000 UTC m=+0.635489055 container remove f24a350bfa502cb001b5bed83f52fbd9af72532a7df198283ae0b1c55a8e453d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-16T05:26:13.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:13.694 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:13.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:13.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:13.821 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:13.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:13.902 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:13.904 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:26:13.923 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:13.923 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:13.924 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:13.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:13 smithi032 ceph-mon[27388]: pgmap v1656: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:13.945 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:13.946 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:13.947 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:13.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:13.970 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:13.992 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:13.993 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:14.014 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:14.015 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:14.015 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:14.037 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:14.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:14.058 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:14.058 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:14.074 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:14.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:14.080 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:14.081 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:14.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:14.102 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:14.155 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:13 smithi204 ceph-mon[30933]: pgmap v1656: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:14.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:14.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:26:14.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:14.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:14.205 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:14 smithi032 podman[195731]: 2024-09-16 05:26:13.958188488 +0000 UTC m=+0.018246077 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:26:14.205 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:14 smithi032 podman[195731]: 2024-09-16 05:26:14.090295694 +0000 UTC m=+0.150353272 container create c450081619965088f49743edc739264129d3c1e0693e6c783c407c17a3cb2891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:26:14.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:13 smithi196 ceph-mon[30802]: pgmap v1656: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:14.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:14.326 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:14.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:14.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:14.451 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:14.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:14.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:14.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:14.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:14.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:14.682 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:14.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:26:14.705 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:14.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:14.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:14.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:14.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3422736375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:14.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:14.957 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:14.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:14.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:15.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:15.082 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:15.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:15.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:26:15.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3422736375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:15.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:15.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:15.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:15.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3422736375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:15.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:14 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:14.877+0000 7f6af1ad6640 -1 osd.6 983 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:15.302 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:15.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:15.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:15.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:15.386 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:15 smithi032 podman[195731]: 2024-09-16 05:26:15.047205976 +0000 UTC m=+1.107263571 container init c450081619965088f49743edc739264129d3c1e0693e6c783c407c17a3cb2891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e) 2024-09-16T05:26:15.387 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:15 smithi032 podman[195731]: 2024-09-16 05:26:15.051303796 +0000 UTC m=+1.111361376 container start c450081619965088f49743edc739264129d3c1e0693e6c783c407c17a3cb2891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:26:15.387 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:15 smithi032 ceph-osd[195887]: -- 172.21.15.32:0/3023947800 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ab234da40 con 0x564ab3114000 2024-09-16T05:26:15.387 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:15 smithi032 bash[195731]: c450081619965088f49743edc739264129d3c1e0693e6c783c407c17a3cb2891 2024-09-16T05:26:15.387 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:15 smithi032 systemd[1]: Started Ceph osd.1 for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:26:15.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:15.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:15.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:15.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:15.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:15.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:15.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:15.711 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:15.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:15.813 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:15.837 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:15.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:15.838 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:15 smithi032 ceph-mon[27388]: pgmap v1657: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:15 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:15 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:15 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 150 sec, osd.6 has slow ops) 2024-09-16T05:26:15.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:15 smithi032 ceph-mon[27388]: osdmap e984: 12 total, 10 up, 12 in 2024-09-16T05:26:15.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:15.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:15.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:16.060 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:26:16.061 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:26:16.061 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:26:16.061 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:26:16.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:16.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:16.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:16.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:15 smithi204 ceph-mon[30933]: pgmap v1657: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:15 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:15 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:15 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 150 sec, osd.6 has slow ops) 2024-09-16T05:26:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:15 smithi204 ceph-mon[30933]: osdmap e984: 12 total, 10 up, 12 in 2024-09-16T05:26:16.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:16.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:16.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:16.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:15 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:15.904+0000 7f6af1ad6640 -1 osd.6 984 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:15 smithi196 ceph-mon[30802]: pgmap v1657: 127 pgs: 2 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 85 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:15 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:15 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:15 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 150 sec, osd.6 has slow ops) 2024-09-16T05:26:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:15 smithi196 ceph-mon[30802]: osdmap e984: 12 total, 10 up, 12 in 2024-09-16T05:26:16.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:16.348 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:16.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:16.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:16.473 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:16.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:16.524 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:16.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:16.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:16.663 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:16 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[195790]: 2024-09-16T05:26:16.446+0000 7fb66e1b5640 -1 Falling back to public interface 2024-09-16T05:26:16.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:16.725 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:16.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:16.776 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:26:16.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:16.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:16.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:16.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:16 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:16.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2143674946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:16.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:16.977 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:16.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:16.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:26:17.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:17.105 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:17.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:16 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:17.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2143674946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:17.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:17.233 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:17.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:17.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:16 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:17.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2143674946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:17.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:16 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:16.896+0000 7f6af1ad6640 -1 osd.6 985 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:17.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:26:17.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:17.374 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:17.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:17.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:17.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:17.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:17.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:no valid command found; 10 closest matches: 2024-09-16T05:26:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:0 2024-09-16T05:26:17.534 INFO:teuthology.orchestra.run.smithi032.stderr:1 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:2 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:abort 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:assert 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:config diff 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:config diff get 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:config get 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:config help [] 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:config set ... 2024-09-16T05:26:17.535 INFO:teuthology.orchestra.run.smithi032.stderr:admin_socket: invalid command 2024-09-16T05:26:17.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:17.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:17.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:17.754 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:17.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:17.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:17.881 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:17.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:17.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:18.007 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:18.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:18.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:17 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:18.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:17 smithi032 ceph-mon[27388]: pgmap v1659: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:18.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:17 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:18.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2143674946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:18.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:17 smithi032 ceph-mon[27388]: osdmap e985: 12 total, 10 up, 12 in 2024-09-16T05:26:18.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:17 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:18.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:17 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:18.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T05:26:18.133 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-16T05:26:18.134 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:18.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:18.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:17 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:17 smithi204 ceph-mon[30933]: pgmap v1659: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:17 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2143674946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:17 smithi204 ceph-mon[30933]: osdmap e985: 12 total, 10 up, 12 in 2024-09-16T05:26:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:17 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:18.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:17 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:18.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:18.237 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:17 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:17.917+0000 7f6af1ad6640 -1 osd.6 985 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:17 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:17 smithi196 ceph-mon[30802]: pgmap v1659: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:17 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2143674946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:17 smithi196 ceph-mon[30802]: osdmap e985: 12 total, 10 up, 12 in 2024-09-16T05:26:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:17 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:18.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:17 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:18.258 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:18.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:18.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:18.380 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:18.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:18.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:18.509 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:18.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:18.611 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:18.637 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:18.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:18.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:18.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:18.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:18.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:18.894 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:18.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:18.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:19.020 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:19.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:19.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:19.147 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:19.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:19.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:18 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:19.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:18 smithi032 ceph-mon[27388]: osdmap e986: 12 total, 10 up, 12 in 2024-09-16T05:26:19.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:18 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:19.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:18 smithi204 ceph-mon[30933]: osdmap e986: 12 total, 10 up, 12 in 2024-09-16T05:26:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:18 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:19.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:18 smithi196 ceph-mon[30802]: osdmap e986: 12 total, 10 up, 12 in 2024-09-16T05:26:19.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:18 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:18.964+0000 7f6af1ad6640 -1 osd.6 986 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:19.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:19.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:19.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:19.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:19.377 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:19.404 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:19.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:19.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:19.532 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:19.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:19.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:19.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:19.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:19.663 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:26:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:26:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:26:19.664 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:26:19.664 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:26:19.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:19.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:19.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:19.890 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:19.915 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:19.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:19.920 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:19 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[195790]: 2024-09-16T05:26:19.777+0000 7fb66e1b5640 -1 osd.1 866 log_to_monitors true 2024-09-16T05:26:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:19 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:19 smithi032 ceph-mon[27388]: pgmap v1662: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:19 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 40 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-16T05:26:19.920 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:19 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 27/222 objects degraded (12.162%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:19 smithi032 ceph-mon[27388]: Health check failed: 2 slow ops, oldest one blocked for 160 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:19.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:19 smithi032 ceph-mon[27388]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T05:26:20.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:20.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:20.144 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:20.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:20.171 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:20.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:20.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:19 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:19 smithi204 ceph-mon[30933]: pgmap v1662: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:19 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 40 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-16T05:26:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:19 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 27/222 objects degraded (12.162%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:19 smithi204 ceph-mon[30933]: Health check failed: 2 slow ops, oldest one blocked for 160 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:20.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:19 smithi204 ceph-mon[30933]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T05:26:20.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:26:19 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:26:19.921+0000 7f6af1ad6640 -1 osd.6 987 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24571.0:8703 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 1476 in=55b] snapc 171=[171,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e867) 2024-09-16T05:26:20.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:19 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:20.383 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:19 smithi196 ceph-mon[30802]: pgmap v1662: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:20.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:19 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 40 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-16T05:26:20.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:19 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 27/222 objects degraded (12.162%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:20.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:19 smithi196 ceph-mon[30802]: Health check failed: 2 slow ops, oldest one blocked for 160 sec, osd.6 has slow ops (SLOW_OPS) 2024-09-16T05:26:20.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:19 smithi196 ceph-mon[30802]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T05:26:20.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:20.404 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[195790]: 2024-09-16T05:26:20.140+0000 7fb661f58640 -1 osd.1 987 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:26:20.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:20.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:20.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:26:20.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:20.537 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:20.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:20.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:20.666 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:20.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:20.700 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:20.700 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:20.700 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T05:26:20.722 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:20.723 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:20.745 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:20.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:20.767 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:20.768 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:20.784 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:20.790 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:20.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:20.794 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:20.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:20.813 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:20.813 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:20.813 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:20.836 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:20.836 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:20.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:20.880 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:20.880 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:20.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:20.901 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:20.901 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:20.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:21.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:21.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:21.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:21.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:21.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T05:26:21.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: osdmap e987: 12 total, 10 up, 12 in 2024-09-16T05:26:21.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:26:21.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:21.034 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' 2024-09-16T05:26:21.035 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/29640726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:21.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:21.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:21.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:21.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:21.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T05:26:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: osdmap e987: 12 total, 10 up, 12 in 2024-09-16T05:26:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:26:21.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' 2024-09-16T05:26:21.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/29640726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:21.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T05:26:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: osdmap e987: 12 total, 10 up, 12 in 2024-09-16T05:26:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-09-16T05:26:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'unique_pool_0' : 2 ]) 2024-09-16T05:26:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: from='osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074]' entity='osd.1' 2024-09-16T05:26:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/29640726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:21.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:21.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:21.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:21.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:21.419 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:21.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:21.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:26:21.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:21.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:21.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:21.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:21.670 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:21.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:21.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:21.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:21.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:21.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:21.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:21.917 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:21.918 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:21.926 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:21.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:21.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:21 smithi032 ceph-mon[27388]: pgmap v1664: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:21.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:21 smithi032 ceph-mon[27388]: Health check update: 1 osds down (OSD_DOWN) 2024-09-16T05:26:21.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:21 smithi032 ceph-mon[27388]: osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074] boot 2024-09-16T05:26:21.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:21 smithi032 ceph-mon[27388]: osdmap e988: 12 total, 11 up, 12 in 2024-09-16T05:26:21.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:21 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:26:21.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:21 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:21.928 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:21 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:22.028 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:22.054 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:22.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:22.115 INFO:tasks.rados.rados.2.smithi204.stdout:3519: finishing write tid 1 to smithi20457400-13 2024-09-16T05:26:22.115 INFO:tasks.rados.rados.2.smithi204.stdout:3476: finishing rollback tid 0 to smithi20457400-49 2024-09-16T05:26:22.115 INFO:tasks.rados.rados.2.smithi204.stdout:3519: finishing write tid 2 to smithi20457400-13 2024-09-16T05:26:22.115 INFO:tasks.rados.rados.2.smithi204.stdout:3519: finishing write tid 3 to smithi20457400-13 2024-09-16T05:26:22.116 INFO:tasks.rados.rados.2.smithi204.stdout:3476: finishing rollback tid 1 to smithi20457400-49 2024-09-16T05:26:22.116 INFO:tasks.rados.rados.2.smithi204.stdout:3519: finishing write tid 4 to smithi20457400-13 2024-09-16T05:26:22.116 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 1153 (ObjNum 1226 snap 373 seq_num 1226) dirty exists 2024-09-16T05:26:22.116 INFO:tasks.rados.rados.2.smithi204.stdout:3519: left oid 13 (ObjNum 1226 snap 373 seq_num 1226) 2024-09-16T05:26:22.119 INFO:tasks.rados.rados.2.smithi204.stdout:3476: finishing rollback tid 2 to smithi20457400-49 2024-09-16T05:26:22.119 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 682 (ObjNum 921 snap 273 seq_num 921) dirty exists 2024-09-16T05:26:22.120 INFO:tasks.rados.rados.2.smithi204.stdout:3461: finishing rollback tid 1 to smithi20457400-14 2024-09-16T05:26:22.120 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1497 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T05:26:22.120 INFO:tasks.rados.rados.2.smithi204.stdout:3502: finishing write tid 1 to smithi20457400-26 2024-09-16T05:26:22.123 INFO:tasks.rados.rados.2.smithi204.stdout:3502: finishing write tid 2 to smithi20457400-26 2024-09-16T05:26:22.123 INFO:tasks.rados.rados.2.smithi204.stdout:3502: finishing write tid 3 to smithi20457400-26 2024-09-16T05:26:22.130 INFO:tasks.rados.rados.2.smithi204.stdout:3454: finishing copy_from racing read to smithi20457400-10 2024-09-16T05:26:22.130 INFO:tasks.rados.rados.2.smithi204.stdout:3454: finishing copy_from to smithi20457400-10 2024-09-16T05:26:22.130 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1476 (ObjNum 1176 snap 360 seq_num 1176) dirty exists 2024-09-16T05:26:22.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:21 smithi204 ceph-mon[30933]: pgmap v1664: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:21 smithi204 ceph-mon[30933]: Health check update: 1 osds down (OSD_DOWN) 2024-09-16T05:26:22.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:21 smithi204 ceph-mon[30933]: osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074] boot 2024-09-16T05:26:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:21 smithi204 ceph-mon[30933]: osdmap e988: 12 total, 11 up, 12 in 2024-09-16T05:26:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:21 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:26:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:21 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:22.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:21 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:22.179 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:22.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:22.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:21 smithi196 ceph-mon[30802]: pgmap v1664: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:21 smithi196 ceph-mon[30802]: Health check update: 1 osds down (OSD_DOWN) 2024-09-16T05:26:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:21 smithi196 ceph-mon[30802]: osd.1 [v2:172.21.15.32:6810/2976159074,v1:172.21.15.32:6811/2976159074] boot 2024-09-16T05:26:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:21 smithi196 ceph-mon[30802]: osdmap e988: 12 total, 11 up, 12 in 2024-09-16T05:26:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:21 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T05:26:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:21 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:21 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:22.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:22.305 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:22.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:22.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:22.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:22.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:22.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:22.556 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:22.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:22.657 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:22.679 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:22.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:3502: finishing write tid 4 to smithi20457400-26 2024-09-16T05:26:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1504 (ObjNum 1220 snap 370 seq_num 1220) dirty exists 2024-09-16T05:26:22.742 INFO:tasks.rados.rados.2.smithi204.stdout:3502: left oid 26 (ObjNum 1220 snap 370 seq_num 1220) 2024-09-16T05:26:22.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:22.805 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:22.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:22.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:22.914 INFO:tasks.rados.rados.2.smithi204.stdout:3495: finishing copy_from to smithi20457400-38 2024-09-16T05:26:22.914 INFO:tasks.rados.rados.2.smithi204.stdout:3495: got expected ENOENT (src dne) 2024-09-16T05:26:22.914 INFO:tasks.rados.rados.2.smithi204.stdout:3495: finishing copy_from racing read to smithi20457400-38 2024-09-16T05:26:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:22.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:22.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-16T05:26:23.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:23.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:26:23.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:26:23.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:23.200 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:23.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:23.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:23 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:23.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:23 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:23.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:23 smithi032 ceph-mon[27388]: osdmap e989: 12 total, 11 up, 12 in 2024-09-16T05:26:23.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3169683736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:23.302 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:23.329 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:23.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:23 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:23 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:23 smithi204 ceph-mon[30933]: osdmap e989: 12 total, 11 up, 12 in 2024-09-16T05:26:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3169683736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:23.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:23.456 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:23.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:23.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:23 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:23.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:23 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:23.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:23 smithi196 ceph-mon[30802]: osdmap e989: 12 total, 11 up, 12 in 2024-09-16T05:26:23.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3169683736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:23.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:23.582 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:23.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:23.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:23.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:23.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:26:23.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:23.826 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:23.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:23.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:23.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:23.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:24.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:24.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:24.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:24.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:24.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:24.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:24.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:24.349 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:24.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:24.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:24 smithi032 ceph-mon[27388]: pgmap v1667: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:24.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:24 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:24.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3169683736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:24.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:24 smithi032 ceph-mon[27388]: osdmap e990: 12 total, 11 up, 12 in 2024-09-16T05:26:24.401 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:26:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:26:24.401 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:26:24.401 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:26:24.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:24 smithi204 ceph-mon[30933]: pgmap v1667: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:24 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3169683736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:24 smithi204 ceph-mon[30933]: osdmap e990: 12 total, 11 up, 12 in 2024-09-16T05:26:24.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:24.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:24.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:24 smithi196 ceph-mon[30802]: pgmap v1667: 127 pgs: 1 active+clean+scrubbing, 3 stale+undersized+peered, 27 undersized+peered, 10 undersized+degraded+peered, 86 active+clean; 241 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 27/222 objects degraded (12.162%) 2024-09-16T05:26:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:24 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3169683736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:24 smithi196 ceph-mon[30802]: osdmap e990: 12 total, 11 up, 12 in 2024-09-16T05:26:24.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:24.605 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:24.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:24.643 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:24.654 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:24.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:24.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:24.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:24.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:24.858 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:24.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:24.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:24.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:24.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:25.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:25.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:25.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:25.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:25.244 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:25.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:25.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:25 smithi032 ceph-mon[27388]: osdmap e991: 12 total, 11 up, 12 in 2024-09-16T05:26:25.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:25.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:25.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:25.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:25 smithi204 ceph-mon[30933]: osdmap e991: 12 total, 11 up, 12 in 2024-09-16T05:26:25.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:25.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:25 smithi196 ceph-mon[30802]: osdmap e991: 12 total, 11 up, 12 in 2024-09-16T05:26:25.502 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:25.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:25.564 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:25.565 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:25.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:25.625 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:25.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:25.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:25.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:25.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:25.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:26:25.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:25.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:26.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:26.038 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:26.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:26.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:26.125 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:26.125 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:26.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:26.147 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:26:26.147 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:26:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:26.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:26.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:26.171 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:26.172 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:26.201 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:26.201 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:26.215 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:26.228 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:26.228 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:26.250 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:26.251 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:26.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:26.272 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:26.273 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: pgmap v1670: 127 pgs: 3 stale+undersized+peered, 24 undersized+peered, 10 undersized+degraded+peered, 90 active+clean; 239 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 919 KiB/s wr, 0 op/s; 27/218 objects degraded (12.385%) 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: 2.2a scrub starts 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: 2.2a scrub ok 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: 2.2 scrub starts 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: 2.2 scrub ok 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 37 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 27/218 objects degraded (12.385%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 160 sec, osd.6 has slow ops) 2024-09-16T05:26:26.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: osdmap e992: 12 total, 11 up, 12 in 2024-09-16T05:26:26.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: 2.28 scrub starts 2024-09-16T05:26:26.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: 2.28 scrub ok 2024-09-16T05:26:26.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:26:26.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: 1.b scrub ok 2024-09-16T05:26:26.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2434531027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:26.274 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:26.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:26.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:26.294 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:26.295 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:26.317 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:26.317 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:26.339 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:26.362 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:26.362 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:26.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:26.419 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:26.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: pgmap v1670: 127 pgs: 3 stale+undersized+peered, 24 undersized+peered, 10 undersized+degraded+peered, 90 active+clean; 239 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 919 KiB/s wr, 0 op/s; 27/218 objects degraded (12.385%) 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: 2.2a scrub starts 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: 2.2a scrub ok 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: 2.2 scrub starts 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: 2.2 scrub ok 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 37 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 27/218 objects degraded (12.385%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 160 sec, osd.6 has slow ops) 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: osdmap e992: 12 total, 11 up, 12 in 2024-09-16T05:26:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: 2.28 scrub starts 2024-09-16T05:26:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: 2.28 scrub ok 2024-09-16T05:26:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:26:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: 1.b scrub ok 2024-09-16T05:26:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2434531027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: pgmap v1670: 127 pgs: 3 stale+undersized+peered, 24 undersized+peered, 10 undersized+degraded+peered, 90 active+clean; 239 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 919 KiB/s wr, 0 op/s; 27/218 objects degraded (12.385%) 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: 2.2a scrub starts 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: 2.2a scrub ok 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: 2.2 scrub starts 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: 2.2 scrub ok 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 37 pgs inactive, 3 pgs stale (PG_AVAILABILITY) 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 27/218 objects degraded (12.385%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 160 sec, osd.6 has slow ops) 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: osdmap e992: 12 total, 11 up, 12 in 2024-09-16T05:26:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: 2.28 scrub starts 2024-09-16T05:26:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: 2.28 scrub ok 2024-09-16T05:26:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:26:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: 1.b scrub ok 2024-09-16T05:26:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2434531027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:26.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:26.547 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:26.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:26.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:26.673 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:26.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:26.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:26.798 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:26.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:26.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-16T05:26:26.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:26.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:26.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:27.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:27.050 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:27.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:27.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:27.177 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:27.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:27.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:27.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:27.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:27.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:27.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:27 smithi204 ceph-mon[30933]: 2.23 scrub starts 2024-09-16T05:26:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:27 smithi204 ceph-mon[30933]: 2.23 scrub ok 2024-09-16T05:26:27.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:27 smithi032 ceph-mon[27388]: 2.23 scrub starts 2024-09-16T05:26:27.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:27 smithi032 ceph-mon[27388]: 2.23 scrub ok 2024-09-16T05:26:27.428 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:27.429 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:27.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:27 smithi196 ceph-mon[30802]: 2.23 scrub starts 2024-09-16T05:26:27.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:27 smithi196 ceph-mon[30802]: 2.23 scrub ok 2024-09-16T05:26:27.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:27.535 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:27.554 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:27.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:27.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:27.681 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:27.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:27.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:27.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:27.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:27.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:27.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:28.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:28.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:28.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:28.103 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:28.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:28.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:28.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:28.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:28.318 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:28.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:28.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: 2.53 scrub starts 2024-09-16T05:26:28.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: 2.53 scrub ok 2024-09-16T05:26:28.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:26:28.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:26:28.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:26:28.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:26:28.347 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:26:28.347 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:26:28.347 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:26:28.347 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: 1.1c scrub starts 2024-09-16T05:26:28.347 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:28 smithi032 ceph-mon[27388]: 1.1c scrub ok 2024-09-16T05:26:28.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:28.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: 2.53 scrub starts 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: 2.53 scrub ok 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: 1.1c scrub starts 2024-09-16T05:26:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:28 smithi204 ceph-mon[30933]: 1.1c scrub ok 2024-09-16T05:26:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:28.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:28.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: 2.53 scrub starts 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: 2.53 scrub ok 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: 1.1c scrub starts 2024-09-16T05:26:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:28 smithi196 ceph-mon[30802]: 1.1c scrub ok 2024-09-16T05:26:28.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:28.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:28.598 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:26:28.598 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:26:28.598 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:26:28.598 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:26:28.598 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:26:28.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:28.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:28.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:28.797 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:28.823 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:28.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:28.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:28.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:28.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:29.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:29.078 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:29.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:29.123 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:26:29.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: 2.17 scrub starts 2024-09-16T05:26:29.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: 2.17 scrub ok 2024-09-16T05:26:29.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:26:29.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:26:29.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: 2.54 scrub starts 2024-09-16T05:26:29.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: 2.54 scrub ok 2024-09-16T05:26:29.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2106423906' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:29.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: 2.57 scrub starts 2024-09-16T05:26:29.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:29 smithi032 ceph-mon[27388]: 2.57 scrub ok 2024-09-16T05:26:29.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:29.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-16T05:26:29.206 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:29.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:29.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:29.335 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:29.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:29.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: 2.17 scrub starts 2024-09-16T05:26:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: 2.17 scrub ok 2024-09-16T05:26:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:26:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:26:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: 2.54 scrub starts 2024-09-16T05:26:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: 2.54 scrub ok 2024-09-16T05:26:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2106423906' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: 2.57 scrub starts 2024-09-16T05:26:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:29 smithi204 ceph-mon[30933]: 2.57 scrub ok 2024-09-16T05:26:29.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:29.462 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:29.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: 2.17 scrub starts 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: 2.17 scrub ok 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: 2.54 scrub starts 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: 2.54 scrub ok 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2106423906' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: 2.57 scrub starts 2024-09-16T05:26:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:29 smithi196 ceph-mon[30802]: 2.57 scrub ok 2024-09-16T05:26:29.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:26:29.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:29.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:29.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:29.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:29.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:29.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:29.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:29.845 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:29.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:29.892 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:29.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:29.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:29.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:30.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:30.101 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:30.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:30.105 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [7] live_osds: [0, 3, 4, 5, 6, 8, 9, 10, 11, 2, 1] 2024-09-16T05:26:30.105 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:26:30.105 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.2819734267292591 2024-09-16T05:26:30.105 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 5 0.2819734267292591 2024-09-16T05:26:30.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:30.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:30.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:30.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:30.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:30.356 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:30.356 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:30.378 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:30.381 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:26:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: pgmap v1673: 127 pgs: 1 peering, 15 undersized+peered, 7 undersized+degraded+peered, 104 active+clean; 240 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1002 KiB/s wr, 2 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: 2.15 scrub starts 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: 2.15 scrub ok 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2106423906' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: osdmap e993: 12 total, 11 up, 12 in 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: 1.16 scrub starts 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: 1.16 scrub ok 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: 2.4c scrub starts 2024-09-16T05:26:30.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:30 smithi032 ceph-mon[27388]: 2.4c scrub ok 2024-09-16T05:26:30.404 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:30.404 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:30.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: pgmap v1673: 127 pgs: 1 peering, 15 undersized+peered, 7 undersized+degraded+peered, 104 active+clean; 240 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1002 KiB/s wr, 2 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:30.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: 2.15 scrub starts 2024-09-16T05:26:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: 2.15 scrub ok 2024-09-16T05:26:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2106423906' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: osdmap e993: 12 total, 11 up, 12 in 2024-09-16T05:26:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: 1.16 scrub starts 2024-09-16T05:26:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: 1.16 scrub ok 2024-09-16T05:26:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: 2.4c scrub starts 2024-09-16T05:26:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:30 smithi204 ceph-mon[30933]: 2.4c scrub ok 2024-09-16T05:26:30.426 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:30.448 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:30.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:30.470 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:30.471 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:30.482 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:30.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: pgmap v1673: 127 pgs: 1 peering, 15 undersized+peered, 7 undersized+degraded+peered, 104 active+clean; 240 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1002 KiB/s wr, 2 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: 2.15 scrub starts 2024-09-16T05:26:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: 2.15 scrub ok 2024-09-16T05:26:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2106423906' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: osdmap e993: 12 total, 11 up, 12 in 2024-09-16T05:26:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: 1.16 scrub starts 2024-09-16T05:26:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: 1.16 scrub ok 2024-09-16T05:26:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: 2.4c scrub starts 2024-09-16T05:26:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:30 smithi196 ceph-mon[30802]: 2.4c scrub ok 2024-09-16T05:26:30.492 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:30.492 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:30.493 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:30.514 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:30.515 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:30.537 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:30.537 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:30.558 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:30.558 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:30.580 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:30.581 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:30.584 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:30.610 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:30.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:30.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:30.737 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:30.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:30.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:30.862 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:30.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:30.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:30.882 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:30.964 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:30.988 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:30.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:30.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:31.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:31.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:31.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:31.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:31.244 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:31.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:31.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:31.370 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:31.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:31.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:31.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:26:31.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:26:31.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/618611089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:31.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: 2.4b scrub starts 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: 2.4b scrub ok 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: 2.3d scrub starts 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: 2.3d scrub ok 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 21 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:31 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 18/222 objects degraded (8.108%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/618611089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: 2.4b scrub starts 2024-09-16T05:26:31.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: 2.4b scrub ok 2024-09-16T05:26:31.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: 2.3d scrub starts 2024-09-16T05:26:31.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: 2.3d scrub ok 2024-09-16T05:26:31.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 21 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:26:31.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:31 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 18/222 objects degraded (8.108%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:31.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:31.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:26:31.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:26:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/618611089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: 2.4b scrub starts 2024-09-16T05:26:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: 2.4b scrub ok 2024-09-16T05:26:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: 2.3d scrub starts 2024-09-16T05:26:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: 2.3d scrub ok 2024-09-16T05:26:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 21 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:26:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:31 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 18/222 objects degraded (8.108%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:31.497 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:31.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:31.505 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:31.591 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:31.591 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:31.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:31.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:31.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:31.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:31.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:31.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:31.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:26:31.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:31.879 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:31.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:31.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:32.007 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:32.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:32.109 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:32.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:32.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:32.264 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:32.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:32.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:32.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:32.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:26:32.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:26:32.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: pgmap v1675: 127 pgs: 15 undersized+peered, 7 undersized+degraded+peered, 105 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 331 B/s rd, 379 KiB/s wr, 2 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:32.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: 2.29 deep-scrub starts 2024-09-16T05:26:32.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: 1.f deep-scrub starts 2024-09-16T05:26:32.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: 1.f deep-scrub ok 2024-09-16T05:26:32.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: 2.29 deep-scrub ok 2024-09-16T05:26:32.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: 2.10 scrub starts 2024-09-16T05:26:32.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:32 smithi032 ceph-mon[27388]: 2.10 scrub ok 2024-09-16T05:26:32.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:26:32.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:26:32.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: pgmap v1675: 127 pgs: 15 undersized+peered, 7 undersized+degraded+peered, 105 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 331 B/s rd, 379 KiB/s wr, 2 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:32.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: 2.29 deep-scrub starts 2024-09-16T05:26:32.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: 1.f deep-scrub starts 2024-09-16T05:26:32.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: 1.f deep-scrub ok 2024-09-16T05:26:32.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: 2.29 deep-scrub ok 2024-09-16T05:26:32.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: 2.10 scrub starts 2024-09-16T05:26:32.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:32 smithi204 ceph-mon[30933]: 2.10 scrub ok 2024-09-16T05:26:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:26:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:26:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: pgmap v1675: 127 pgs: 15 undersized+peered, 7 undersized+degraded+peered, 105 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 331 B/s rd, 379 KiB/s wr, 2 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: 2.29 deep-scrub starts 2024-09-16T05:26:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: 1.f deep-scrub starts 2024-09-16T05:26:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: 1.f deep-scrub ok 2024-09-16T05:26:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: 2.29 deep-scrub ok 2024-09-16T05:26:32.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: 2.10 scrub starts 2024-09-16T05:26:32.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:32 smithi196 ceph-mon[30802]: 2.10 scrub ok 2024-09-16T05:26:32.493 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:32.518 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:32.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:32.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:32.646 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:32.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:32.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:32.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-16T05:26:32.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:32.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:32.800 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:32.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:32.902 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:32.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:33.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:33.029 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:33.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:33.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:33.132 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 0.281973 (482f) 2024-09-16T05:26:33.158 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:33.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:33.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:33.283 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:33.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:33.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:33.407 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:33.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:33.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:33 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:26:33.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:33 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:26:33.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/575750670' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 0.2819734267292591}]: dispatch 2024-09-16T05:26:33.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:33 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:26:33.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:33 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:26:33.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/575750670' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 0.2819734267292591}]: dispatch 2024-09-16T05:26:33.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:33 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:26:33.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:33 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:26:33.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/575750670' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 0.2819734267292591}]: dispatch 2024-09-16T05:26:33.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:26:33.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:33.535 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:33.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:33.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:33.659 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:33.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:33.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:33.786 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:33.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:33.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:33.914 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:33.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:34.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:34.041 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:34.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:34.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:26:34.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:34.170 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:34.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:34.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:34.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:34.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:34.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:34.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:34.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: pgmap v1676: 127 pgs: 15 undersized+peered, 7 undersized+degraded+peered, 105 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 292 KiB/s wr, 1 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:34.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: 2.50 scrub starts 2024-09-16T05:26:34.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:26:34.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: 2.50 scrub ok 2024-09-16T05:26:34.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:26:34.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/575750670' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.2819734267292591}]': finished 2024-09-16T05:26:34.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: osdmap e994: 12 total, 11 up, 12 in 2024-09-16T05:26:34.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: 1.1b scrub starts 2024-09-16T05:26:34.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: 1.1b scrub ok 2024-09-16T05:26:34.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4283078748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:34.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: pgmap v1676: 127 pgs: 15 undersized+peered, 7 undersized+degraded+peered, 105 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 292 KiB/s wr, 1 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:34.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: 2.50 scrub starts 2024-09-16T05:26:34.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:26:34.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: 2.50 scrub ok 2024-09-16T05:26:34.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:26:34.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/575750670' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.2819734267292591}]': finished 2024-09-16T05:26:34.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: osdmap e994: 12 total, 11 up, 12 in 2024-09-16T05:26:34.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: 1.1b scrub starts 2024-09-16T05:26:34.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: 1.1b scrub ok 2024-09-16T05:26:34.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4283078748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:34.423 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:34.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: pgmap v1676: 127 pgs: 15 undersized+peered, 7 undersized+degraded+peered, 105 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 292 KiB/s wr, 1 op/s; 18/222 objects degraded (8.108%) 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: 2.50 scrub starts 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: 2.50 scrub ok 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/575750670' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.2819734267292591}]': finished 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: osdmap e994: 12 total, 11 up, 12 in 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: 1.1b scrub starts 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: 1.1b scrub ok 2024-09-16T05:26:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4283078748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:34.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:34.547 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:34.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:34.616 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:34.616 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:34.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:34.672 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:34.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:34.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:34.797 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:34.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:34.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:34.922 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:34.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:35.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:35.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:35.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:35.175 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:35.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:35.264 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-16T05:26:35.271 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:35.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:35.301 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:35.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:35.370 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:35 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:35.371 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4283078748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:35.371 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:35 smithi032 ceph-mon[27388]: osdmap e995: 12 total, 11 up, 12 in 2024-09-16T05:26:35.371 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:35 smithi032 ceph-mon[27388]: osdmap e996: 12 total, 11 up, 12 in 2024-09-16T05:26:35.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:35.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:35 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:35.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4283078748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:35.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:35 smithi204 ceph-mon[30933]: osdmap e995: 12 total, 11 up, 12 in 2024-09-16T05:26:35.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:35 smithi204 ceph-mon[30933]: osdmap e996: 12 total, 11 up, 12 in 2024-09-16T05:26:35.426 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:35.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:35.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:35 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4283078748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:35 smithi196 ceph-mon[30802]: osdmap e995: 12 total, 11 up, 12 in 2024-09-16T05:26:35.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:35 smithi196 ceph-mon[30802]: osdmap e996: 12 total, 11 up, 12 in 2024-09-16T05:26:35.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:35.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:35.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:35.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:35.679 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:35.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:35.734 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:35.734 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:35.755 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:26:35.756 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:26:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:35.776 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:35.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:35.797 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:35.797 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:35.807 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:35.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:35.819 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:35.819 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:35.840 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:35.841 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:35.862 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:35.863 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:35.863 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:35.884 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:35.884 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:35.906 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:35.907 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:35.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:35.929 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:35.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:35.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:35.951 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:35.951 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:36.037 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:36.063 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:36.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:36.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:36.188 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:36.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:36.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:36.316 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:36.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:36.384 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:36.418 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:36 smithi032 ceph-mon[27388]: pgmap v1679: 127 pgs: 2 peering, 15 undersized+peered, 7 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 18/222 objects degraded (8.108%) 2024-09-16T05:26:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/271899567' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:36 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 21 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:26:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:36 smithi032 ceph-mon[27388]: osdmap e997: 12 total, 11 up, 12 in 2024-09-16T05:26:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:36 smithi204 ceph-mon[30933]: pgmap v1679: 127 pgs: 2 peering, 15 undersized+peered, 7 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 18/222 objects degraded (8.108%) 2024-09-16T05:26:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/271899567' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:36 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 21 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:26:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:36 smithi204 ceph-mon[30933]: osdmap e997: 12 total, 11 up, 12 in 2024-09-16T05:26:36.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:36.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:36.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:36 smithi196 ceph-mon[30802]: pgmap v1679: 127 pgs: 2 peering, 15 undersized+peered, 7 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 18/222 objects degraded (8.108%) 2024-09-16T05:26:36.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/271899567' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:36 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 21 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-16T05:26:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:36 smithi196 ceph-mon[30802]: osdmap e997: 12 total, 11 up, 12 in 2024-09-16T05:26:36.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:36.570 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:36.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:36.672 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:36.695 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:36.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:36.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:36.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:36.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:36.862 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:36.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:36.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:36.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:36.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:37.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:37.075 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:37.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:37.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:37.203 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:37.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:37.303 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:26:37.304 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:26:37.304 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:26:37.304 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:26:37.304 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:26:37.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:37.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:37.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:37.454 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:37.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:37.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:37.581 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:37.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:37.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:37.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:37.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:37.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:37.835 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:37.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:37.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:37.959 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:37.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:37.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:38.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:38.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:38.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:37 smithi032 ceph-mon[27388]: pgmap v1682: 127 pgs: 1 active+clean+remapped, 3 peering, 14 undersized+peered, 7 undersized+degraded+peered, 102 active+clean; 240 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 18/222 objects degraded (8.108%); 1.1 MiB/s, 6 objects/s recovering 2024-09-16T05:26:38.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:37 smithi032 ceph-mon[27388]: osdmap e998: 12 total, 11 up, 12 in 2024-09-16T05:26:38.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:37 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 18/222 objects degraded (8.108%), 7 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:38.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-16T05:26:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:37 smithi204 ceph-mon[30933]: pgmap v1682: 127 pgs: 1 active+clean+remapped, 3 peering, 14 undersized+peered, 7 undersized+degraded+peered, 102 active+clean; 240 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 18/222 objects degraded (8.108%); 1.1 MiB/s, 6 objects/s recovering 2024-09-16T05:26:38.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:37 smithi204 ceph-mon[30933]: osdmap e998: 12 total, 11 up, 12 in 2024-09-16T05:26:38.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:37 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 18/222 objects degraded (8.108%), 7 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:38.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:26:38.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:38.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:38.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:37 smithi196 ceph-mon[30802]: pgmap v1682: 127 pgs: 1 active+clean+remapped, 3 peering, 14 undersized+peered, 7 undersized+degraded+peered, 102 active+clean; 240 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 18/222 objects degraded (8.108%); 1.1 MiB/s, 6 objects/s recovering 2024-09-16T05:26:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:37 smithi196 ceph-mon[30802]: osdmap e998: 12 total, 11 up, 12 in 2024-09-16T05:26:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:37 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 18/222 objects degraded (8.108%), 7 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:38.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:38.336 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:38.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:38.435 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:38.436 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:38.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:38.464 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:38.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:38.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:38.566 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:38.592 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:38.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:38.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:38.716 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:38.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:38.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:38.842 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:38.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:38.862 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [7] live_osds: [0, 3, 4, 5, 6, 8, 9, 10, 11, 2, 1] 2024-09-16T05:26:38.862 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:26:38.862 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T05:26:38.862 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.876975 2024-09-16T05:26:38.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 0.87697512780021 2024-09-16T05:26:38.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:26:38.945 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:38.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:38.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:39.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:39.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:39.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:39.167 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:26:39.167 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 20, 2024-09-16T05:26:39.167 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 600, 2024-09-16T05:26:39.167 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [ 2024-09-16T05:26:39.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.168 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4629 2.1e 2:7b642727:::smithi20457400-40:head [copy-from ver 511] snapc bf=[bf,be] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:26:39.168 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:26:39.168 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 537.41213413900005, 2024-09-16T05:26:39.168 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.01647931, 2024-09-16T05:26:39.169 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.169 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.169 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:26:39.169 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:26:39.170 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:39.170 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4629 2024-09-16T05:26:39.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.170 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.170 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:26:39.171 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.171 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.171 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752524+0000", 2024-09-16T05:26:39.172 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.172 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.172 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752526+0000", 2024-09-16T05:26:39.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.894e-06 2024-09-16T05:26:39.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.173 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.173 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752536+0000", 2024-09-16T05:26:39.173 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.4700000000000008e-06 2024-09-16T05:26:39.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.174 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.174 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752555+0000", 2024-09-16T05:26:39.174 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8907999999999999e-05 2024-09-16T05:26:39.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.175 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.175 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752564+0000", 2024-09-16T05:26:39.175 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.7739999999999999e-06 2024-09-16T05:26:39.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.176 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.176 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752638+0000", 2024-09-16T05:26:39.176 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.3343999999999994e-05 2024-09-16T05:26:39.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.177 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op must wait for map", 2024-09-16T05:26:39.177 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752646+0000", 2024-09-16T05:26:39.177 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.7690000000000002e-06 2024-09-16T05:26:39.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.178 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.178 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754646+0000", 2024-09-16T05:26:39.178 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001999331 2024-09-16T05:26:39.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754740+0000", 2024-09-16T05:26:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3778999999999998e-05 2024-09-16T05:26:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:26:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.764207+0000", 2024-09-16T05:26:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094672899999999997 2024-09-16T05:26:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.180 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:26:39.181 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768754+0000", 2024-09-16T05:26:39.181 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0045470199999999997 2024-09-16T05:26:39.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.181 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:39.182 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768860+0000", 2024-09-16T05:26:39.182 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010586600000000001 2024-09-16T05:26:39.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.182 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.182 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768948+0000", 2024-09-16T05:26:39.183 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.8503000000000003e-05 2024-09-16T05:26:39.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.183 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.183 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.769004+0000", 2024-09-16T05:26:39.184 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5362000000000003e-05 2024-09-16T05:26:39.184 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.184 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.184 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.185 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:4630 2.1e 2:7b642727:::smithi20457400-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e512)", 2024-09-16T05:26:39.185 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:26:39.185 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 537.41207638900005, 2024-09-16T05:26:39.185 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.015483190000000001, 2024-09-16T05:26:39.185 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.185 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.186 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.186 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:26:39.186 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:26:39.186 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:39.186 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 4630 2024-09-16T05:26:39.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.187 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.187 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.187 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:26:39.187 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.188 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.188 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:26:39.188 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.189 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.189 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752582+0000", 2024-09-16T05:26:39.189 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.34e-07 2024-09-16T05:26:39.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.190 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.190 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752584+0000", 2024-09-16T05:26:39.190 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.956e-06 2024-09-16T05:26:39.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.191 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752591+0000", 2024-09-16T05:26:39.191 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2580000000000001e-06 2024-09-16T05:26:39.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.191 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.192 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752594+0000", 2024-09-16T05:26:39.192 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.6830000000000001e-06 2024-09-16T05:26:39.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.192 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752703+0000", 2024-09-16T05:26:39.193 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108818 2024-09-16T05:26:39.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.193 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:39.193 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.752708+0000", 2024-09-16T05:26:39.194 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.561e-06 2024-09-16T05:26:39.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.194 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.194 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754913+0000", 2024-09-16T05:26:39.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0022055680000000002 2024-09-16T05:26:39.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.195 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for blocked object", 2024-09-16T05:26:39.195 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.754965+0000", 2024-09-16T05:26:39.195 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2222000000000002e-05 2024-09-16T05:26:39.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.196 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.196 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767827+0000", 2024-09-16T05:26:39.196 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.012861097 2024-09-16T05:26:39.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.197 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.197 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.767929+0000", 2024-09-16T05:26:39.197 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010275799999999999 2024-09-16T05:26:39.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.198 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.198 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:17:41.768065+0000", 2024-09-16T05:26:39.198 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013593500000000001 2024-09-16T05:26:39.198 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.198 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.199 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.199 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:142 2.1e e571/291 ondisk, result = 0)", 2024-09-16T05:26:39.199 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:26:39.199 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 477.40050023399999, 2024-09-16T05:26:39.200 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55541789799999997, 2024-09-16T05:26:39.200 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.200 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.200 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.200 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.201 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.201 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:26:39.201 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.202 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.202 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764158+0000", 2024-09-16T05:26:39.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:39.206 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.206 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.206 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764159+0000", 2024-09-16T05:26:39.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5600000000000005e-07 2024-09-16T05:26:39.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.207 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.207 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:41.764162+0000", 2024-09-16T05:26:39.207 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.5599999999999998e-06 2024-09-16T05:26:39.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.208 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.208 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319278+0000", 2024-09-16T05:26:39.208 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.55511595899999999 2024-09-16T05:26:39.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.209 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.209 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319296+0000", 2024-09-16T05:26:39.210 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.7900000000000001e-05 2024-09-16T05:26:39.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.210 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.210 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319440+0000", 2024-09-16T05:26:39.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000143175 2024-09-16T05:26:39.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.211 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.211 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319450+0000", 2024-09-16T05:26:39.211 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0705e-05 2024-09-16T05:26:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:42.319576+0000", 2024-09-16T05:26:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012584299999999999 2024-09-16T05:26:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:5587 2.34 e583/278 2:2d3f13f8:::smithi20457400-7:head v 583'466, mlcod=583'463)", 2024-09-16T05:26:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:26:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 466.35755388299998, 2024-09-16T05:26:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14534661199999999, 2024-09-16T05:26:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.215 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.215 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.215 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:26:39.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.216 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.216 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:26:39.216 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.217 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.217 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807105+0000", 2024-09-16T05:26:39.217 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5300000000000004e-07 2024-09-16T05:26:39.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.218 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.218 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.807109+0000", 2024-09-16T05:26:39.218 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2030000000000002e-06 2024-09-16T05:26:39.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.219 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.219 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951917+0000", 2024-09-16T05:26:39.219 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.14480800699999999 2024-09-16T05:26:39.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.220 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.220 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951927+0000", 2024-09-16T05:26:39.220 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0115000000000001e-05 2024-09-16T05:26:39.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.221 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.951990+0000", 2024-09-16T05:26:39.221 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2689999999999998e-05 2024-09-16T05:26:39.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.221 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952011+0000", 2024-09-16T05:26:39.222 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.084e-05 2024-09-16T05:26:39.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.222 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.222 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952413+0000", 2024-09-16T05:26:39.223 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040190199999999999 2024-09-16T05:26:39.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.223 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.223 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:18:52.952451+0000", 2024-09-16T05:26:39.224 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8302e-05 2024-09-16T05:26:39.224 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.224 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.224 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.225 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:383 2.29 e617/253 2:9662c29c:::smithi20457400-9:d8 v 617'830, mlcod=616'824)", 2024-09-16T05:26:39.225 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:26:39.225 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 432.38186570200003, 2024-09-16T05:26:39.225 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.038031031999999999, 2024-09-16T05:26:39.225 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.226 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.226 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.226 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.226 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:26:39.227 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.227 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.227 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782793+0000", 2024-09-16T05:26:39.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.228 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.228 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782794+0000", 2024-09-16T05:26:39.228 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3500000000000005e-07 2024-09-16T05:26:39.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.229 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.229 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.782801+0000", 2024-09-16T05:26:39.229 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1210000000000001e-06 2024-09-16T05:26:39.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.230 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.230 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819928+0000", 2024-09-16T05:26:39.230 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.037126740999999998 2024-09-16T05:26:39.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.819938+0000", 2024-09-16T05:26:39.231 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0845e-05 2024-09-16T05:26:39.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.231 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.231 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820001+0000", 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.2786999999999994e-05 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820023+0000", 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.213e-05 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820771+0000", 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00074818699999999998 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:19:26.820824+0000", 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2286e-05 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:199 2.1e e714/291 ondisk, result = 0)", 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 335.35244958700002, 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222874399, 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.234 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812209+0000", 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2099999999999997e-07 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:03.812213+0000", 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.3859999999999999e-06 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034914+0000", 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.222700804 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034920+0000", 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6579999999999996e-06 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034968+0000", 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7833000000000001e-05 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.034975+0000", 2024-09-16T05:26:39.236 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0400000000000004e-06 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:04.035083+0000", 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000108157 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7414 2.5a 2:5ae26c04:::smithi20457400-32:head [write 1423159~486356] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 294.33247741500003, 2024-09-16T05:26:39.237 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.010834233, 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7414 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832181+0000", 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832183+0000", 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6589999999999999e-06 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.239 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832584+0000", 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00040119599999999999 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832598+0000", 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3736e-05 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832607+0000", 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.1870000000000003e-06 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.240 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832655+0000", 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8043000000000001e-05 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832725+0000", 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.0136999999999996e-05 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.832832+0000", 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00010700399999999999 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834013+0000", 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0011807479999999999 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842930+0000", 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0089172439999999995 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842977+0000", 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.7547999999999999e-05 2024-09-16T05:26:39.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.843015+0000", 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.7731000000000002e-05 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:7414 2.5a e755/742 ondisk, result = 0)", 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 294.33044777499998, 2024-09-16T05:26:39.243 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0088119329999999992, 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834211+0000", 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834212+0000", 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.2699999999999998e-07 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.834216+0000", 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2139999999999998e-06 2024-09-16T05:26:39.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842854+0000", 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0086381679999999999 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842859+0000", 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4729999999999999e-06 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842913+0000", 2024-09-16T05:26:39.246 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3235000000000001e-05 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842922+0000", 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.431e-06 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.843023+0000", 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000100485 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7416 2.5a 2:5ae26c04:::smithi20457400-32:head [setxattr _header (58) in=7b,truncate 3311210] snapc 13e=[13e,13d,13c,13b,136,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 294.324037471, 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.075538587000000004, 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7416 2024-09-16T05:26:39.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840621+0000", 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840622+0000", 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6400000000000001e-07 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840626+0000", 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.8589999999999998e-06 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840639+0000", 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2982999999999999e-05 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840644+0000", 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.57e-06 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.841987+0000", 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0013431599999999999 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842059+0000", 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1450000000000002e-05 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for subops from 0", 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842166+0000", 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000107421 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "op_commit", 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842788+0000", 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00062151399999999999 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916025+0000", 2024-09-16T05:26:39.252 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073237675000000002 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916092+0000", 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6631999999999995e-05 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916160+0000", 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.7459000000000006e-05 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_op(client.24571.0:7417 2.5a 2:5ae26c04:::smithi20457400-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e755)", 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 294.32398118999998, 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.075671839000000005, 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "client_info": { 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "client": "client.24571", 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:39.254 INFO:teuthology.orchestra.run.smithi032.stdout: "tid": 7417 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840677+0000", 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840678+0000", 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.9800000000000003e-07 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840682+0000", 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9310000000000001e-06 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840695+0000", 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.2672e-05 2024-09-16T05:26:39.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.840705+0000", 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0498e-05 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842425+0000", 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001719497 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:39.257 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842530+0000", 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000105824 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916202+0000", 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073671367000000001 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916273+0000", 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.1316000000000006e-05 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.258 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916349+0000", 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.5935999999999999e-05 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(client.24571.0:7416 2.5a e755/742 ondisk, result = 0)", 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 294.32170614799998, 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.073216748999999998, 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.259 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842952+0000", 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.260 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842953+0000", 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.4000000000000002e-07 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.842956+0000", 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1099999999999999e-06 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.915928+0000", 2024-09-16T05:26:39.261 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.072972180999999997 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.915937+0000", 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 9.3689999999999996e-06 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916004+0000", 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.6836999999999997e-05 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916017+0000", 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.3200000000000001e-05 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:44.916169+0000", 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000151512 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.263 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop_reply(osd.3.0:251 2.29 e762/745 ondisk, result = 0)", 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 287.24218392699999, 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.502823558, 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922474+0000", 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922475+0000", 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.3000000000000001e-07 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:51.922478+0000", 2024-09-16T05:26:39.265 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.18e-06 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425018+0000", 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.50254011200000004 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425039+0000", 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0832999999999999e-05 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.266 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425166+0000", 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012726 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425177+0000", 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0217999999999999e-05 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:21:52.425298+0000", 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000121425 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.267 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.7.0:1425 2.2e e786/278 2:75dc7feb:::smithi20457400-28:13f v 786'738, mlcod=781'732)", 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 263.31417160000001, 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72979714900000003, 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.268 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850487+0000", 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8599999999999998e-07 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:15.850493+0000", 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.1120000000000001e-06 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579433+0000", 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72893962899999998 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579449+0000", 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6059999999999999e-05 2024-09-16T05:26:39.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579568+0000", 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.000118911 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.579618+0000", 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.0436000000000003e-05 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.580252+0000", 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00063313400000000004 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:16.580284+0000", 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2381e-05 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.2e,QREQUEST,e:797,from:7,reservation_nonce:8)", 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:26:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 250.82225678200001, 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27875738500000002, 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342402+0000", 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342404+0000", 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9300000000000002e-06 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.342412+0000", 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.6319999999999997e-06 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.274 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.620932+0000", 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.27851996800000001 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.620943+0000", 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0892e-05 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621083+0000", 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00013946999999999999 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621110+0000", 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.7934000000000001e-05 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:28.621159+0000", 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.8559e-05 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:8050 2.2e e804/278 2:75dc7feb:::smithi20457400-28:head v 804'753, mlcod=804'750)", 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 245.218113762, 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32122192599999999, 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946545+0000", 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.9999999999999997e-07 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:33.946550+0000", 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4739999999999999e-06 2024-09-16T05:26:39.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.266951+0000", 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.32040173 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.266959+0000", 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.0199999999999994e-06 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.279 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267034+0000", 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4408999999999999e-05 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267065+0000", 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.1538000000000002e-05 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267727+0000", 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00066189099999999996 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:22:34.267767+0000", 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.9264000000000002e-05 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.6.0:690 2.29 e855/767 2:96dc17ef:::smithi20457400-10:165 v 855'1417, mlcod=852'1410)", 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 194.28996390099999, 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72853946700000005, 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.281 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.282 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874695+0000", 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.2500000000000005e-07 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:24.874701+0000", 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8100000000000003e-06 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.601981+0000", 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.72728016900000003 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.601998+0000", 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.6753e-05 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.602123+0000", 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00012546000000000001 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.602147+0000", 2024-09-16T05:26:39.284 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.3597000000000001e-05 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.603190+0000", 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.001043228 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:23:25.603234+0000", 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.3724999999999999e-05 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(osd.7.0:1618 2.2e e936/278 2:75dc7feb:::smithi20457400-28:18d v 936'854, mlcod=929'851)", 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:46.301199+0000", 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 112.863459826, 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0094076890000000003, 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301199+0000", 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301199+0000", 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301199+0000", 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8400000000000001e-07 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.287 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301205+0000", 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.8100000000000003e-06 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301245+0000", 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.0435999999999997e-05 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301261+0000", 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.5943999999999998e-05 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.288 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301336+0000", 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.4931999999999993e-05 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.301366+0000", 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.0236e-05 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.310549+0000", 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0091832730000000005 2024-09-16T05:26:39.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:46.310606+0000", 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6873999999999998e-05 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:9642 2.34 e947/278 2:2cffa4d6:::smithi20457400-50:head v 947'912, mlcod=941'908)", 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:24:58.381234+0000", 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 100.78342477699999, 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.38724071399999999, 2024-09-16T05:26:39.290 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.381234+0000", 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.381234+0000", 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.381234+0000", 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.5599999999999995e-07 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.381239+0000", 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.639e-06 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.767638+0000", 2024-09-16T05:26:39.292 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.38639962799999999 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.767646+0000", 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.7619999999999995e-06 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.767715+0000", 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 6.8368999999999995e-05 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.767743+0000", 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8884999999999999e-05 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.768441+0000", 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00069796699999999997 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:24:58.768474+0000", 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.2907999999999997e-05 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "osd_repop(client.24571.0:9800 2.29 e964/767 2:94d33c56:::smithi20457400-8:head v 964'1473, mlcod=951'1470)", 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:25:15.473346+0000", 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 83.691312791000001, 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0026575290000000001, 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.295 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473346+0000", 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473346+0000", 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473346+0000", 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.8400000000000001e-07 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.296 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473351+0000", 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.2689999999999999e-06 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473385+0000", 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 3.4236000000000002e-05 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473396+0000", 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.0644e-05 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473472+0000", 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 7.6334000000000005e-05 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.473501+0000", 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.8909999999999999e-05 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "commit_sent", 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.475958+0000", 2024-09-16T05:26:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.0024570640000000001 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:25:15.476003+0000", 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.4888000000000001e-05 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "description": "MOSDScrubReserve(2.53,GRANT,e:992,from:4,reservation_nonce:0)", 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "initiated_at": "2024-09-16T05:26:26.299744+0000", 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "age": 12.864914025000001, 2024-09-16T05:26:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00048737200000000002, 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "continuous": false, 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "type_data": { 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "flag_point": "started", 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "events": [ 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "initiated", 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.299744+0000", 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "header_read", 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.299744+0000", 2024-09-16T05:26:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "throttled", 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.299746+0000", 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 1.9360000000000002e-06 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "all_read", 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.299755+0000", 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 8.2269999999999993e-06 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "dispatched", 2024-09-16T05:26:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.299811+0000", 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 5.6168999999999997e-05 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "queued_for_pg", 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.299832+0000", 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 2.0973000000000001e-05 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "reached_pg", 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.299986+0000", 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00015445699999999999 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "started", 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.300028+0000", 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 4.2126000000000003e-05 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: "event": "done", 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: "time": "2024-09-16T05:26:26.300232+0000", 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: "duration": 0.00020348400000000001 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-09-16T05:26:39.303 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:26:39.326 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:39.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:39.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:39.427 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:39.453 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:39.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:39.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:39.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:39.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:39.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:39.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:39.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:39.833 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:39.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:39.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:39.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:39.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:40.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:40.088 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:40.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:39 smithi032 ceph-mon[27388]: pgmap v1684: 127 pgs: 4 active+clean+remapped, 3 peering, 14 undersized+peered, 6 undersized+degraded+peered, 100 active+clean; 240 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 64 KiB/s wr, 5 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 2.9 MiB/s, 20 objects/s recovering 2024-09-16T05:26:40.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3044550600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:40.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:39 smithi204 ceph-mon[30933]: pgmap v1684: 127 pgs: 4 active+clean+remapped, 3 peering, 14 undersized+peered, 6 undersized+degraded+peered, 100 active+clean; 240 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 64 KiB/s wr, 5 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 2.9 MiB/s, 20 objects/s recovering 2024-09-16T05:26:40.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3044550600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:40.190 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:40.212 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:40.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:39 smithi196 ceph-mon[30802]: pgmap v1684: 127 pgs: 4 active+clean+remapped, 3 peering, 14 undersized+peered, 6 undersized+degraded+peered, 100 active+clean; 240 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 64 KiB/s wr, 5 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 2.9 MiB/s, 20 objects/s recovering 2024-09-16T05:26:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3044550600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:40.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:40.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:40.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-16T05:26:40.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:40.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:40.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:40.514 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:26:40.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:40.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:40.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:40.668 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:40.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:40.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:40.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:40.850 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:40.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:40.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:40.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:40.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:41.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:40 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:41.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3044550600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:41.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:40 smithi196 ceph-mon[30802]: osdmap e999: 12 total, 11 up, 12 in 2024-09-16T05:26:41.053 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:40 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 19 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:26:41.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:41.101 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:41.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:41.154 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:26:41.154 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:26:41.154 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:26:41.154 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:26:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:40 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3044550600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:40 smithi032 ceph-mon[27388]: osdmap e999: 12 total, 11 up, 12 in 2024-09-16T05:26:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:40 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 19 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:26:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:40 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3044550600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:40 smithi204 ceph-mon[30933]: osdmap e999: 12 total, 11 up, 12 in 2024-09-16T05:26:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:40 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 19 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-16T05:26:41.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:41.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:41.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:41.324 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:41.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:41.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:41.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:41.355 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:41.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:41.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:41.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:41.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:41.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:41.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:41.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:41.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:41.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:41.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:41.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:41.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-16T05:26:41.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:41.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:42.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:42.093 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:42.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:42.135 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:42.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:41 smithi032 ceph-mon[27388]: pgmap v1686: 127 pgs: 4 active+clean+remapped, 1 peering, 14 undersized+peered, 6 undersized+degraded+peered, 102 active+clean; 240 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 52 KiB/s wr, 4 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 2.4 MiB/s, 16 objects/s recovering 2024-09-16T05:26:42.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:41 smithi204 ceph-mon[30933]: pgmap v1686: 127 pgs: 4 active+clean+remapped, 1 peering, 14 undersized+peered, 6 undersized+degraded+peered, 102 active+clean; 240 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 52 KiB/s wr, 4 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 2.4 MiB/s, 16 objects/s recovering 2024-09-16T05:26:42.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:42.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:42.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:42.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:41 smithi196 ceph-mon[30802]: pgmap v1686: 127 pgs: 4 active+clean+remapped, 1 peering, 14 undersized+peered, 6 undersized+degraded+peered, 102 active+clean; 240 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 52 KiB/s wr, 4 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 2.4 MiB/s, 16 objects/s recovering 2024-09-16T05:26:42.242 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:42.242 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:26:42.242 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:42.242 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:42.243 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:42.243 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:26:42.243 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:42.243 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:42.243 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:42.243 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:42.243 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:42.243 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:42.244 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:42.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:42.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:42.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:42.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:42.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:42.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:42.571 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:42.595 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:42.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:42.666 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:42.666 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:42.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:42.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:42.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:42.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:42.845 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:42.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:42.906 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.5 primary-affinity to 0.876975 (8574732) 2024-09-16T05:26:42.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:42.957 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:42.970 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:42.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:43.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:43.098 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:43.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1109613814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:43.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/770080096' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.87697512780021}]: dispatch 2024-09-16T05:26:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1109613814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:43.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/770080096' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.87697512780021}]: dispatch 2024-09-16T05:26:43.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:43.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:43.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1109613814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/770080096' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.87697512780021}]: dispatch 2024-09-16T05:26:43.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:26:43.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-16T05:26:43.325 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:43.352 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:43.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:43.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:43.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:43.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:43.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:43.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:43.602 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:43.605 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:43.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:43.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:43.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:43.849 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:43.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:43.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:43.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:43.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:43.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:43 smithi196 ceph-mon[30802]: pgmap v1687: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 42 KiB/s wr, 3 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 1.9 MiB/s, 13 objects/s recovering 2024-09-16T05:26:43.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/770080096' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.87697512780021}]': finished 2024-09-16T05:26:43.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:43 smithi196 ceph-mon[30802]: osdmap e1000: 12 total, 11 up, 12 in 2024-09-16T05:26:44.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:44.099 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:44.099 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:44.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:44.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:43 smithi032 ceph-mon[27388]: pgmap v1687: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 42 KiB/s wr, 3 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 1.9 MiB/s, 13 objects/s recovering 2024-09-16T05:26:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/770080096' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.87697512780021}]': finished 2024-09-16T05:26:44.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:43 smithi032 ceph-mon[27388]: osdmap e1000: 12 total, 11 up, 12 in 2024-09-16T05:26:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:43 smithi204 ceph-mon[30933]: pgmap v1687: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 42 KiB/s wr, 3 op/s; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%); 1.9 MiB/s, 13 objects/s recovering 2024-09-16T05:26:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/770080096' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.87697512780021}]': finished 2024-09-16T05:26:44.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:43 smithi204 ceph-mon[30933]: osdmap e1000: 12 total, 11 up, 12 in 2024-09-16T05:26:44.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:44.226 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:44.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:44.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:44.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:44.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:44.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:44.474 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:44.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:44.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:44.602 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:44.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:44.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-16T05:26:44.704 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:44.730 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:44.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:44.832 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:44.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:44.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:44.918 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:26:44.930 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:44.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:26:44.960 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:44.984 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:44.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:45.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:45.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:45.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:44 smithi032 ceph-mon[27388]: osdmap e1001: 12 total, 11 up, 12 in 2024-09-16T05:26:45.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2555447685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:45.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:44 smithi196 ceph-mon[30802]: osdmap e1001: 12 total, 11 up, 12 in 2024-09-16T05:26:45.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2555447685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:45.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:45.237 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:45.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:45.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:45.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:45.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:45.381 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:45.381 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:45.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:44 smithi204 ceph-mon[30933]: osdmap e1001: 12 total, 11 up, 12 in 2024-09-16T05:26:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2555447685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:26:45.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:45.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:45.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:45.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:45.611 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:45.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:45.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:45.739 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:45.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:45.744 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:45.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:45.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:45.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:45.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:45.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:45 smithi032 ceph-mon[27388]: pgmap v1690: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%) 2024-09-16T05:26:45.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:45 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:45.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2555447685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:45.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:45 smithi032 ceph-mon[27388]: osdmap e1002: 12 total, 11 up, 12 in 2024-09-16T05:26:45.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:45 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:26:45.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:45 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:26:45.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:45 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 18 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:26:45.991 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:45 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 16/220 objects degraded (7.273%), 6 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:45.992 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:45.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:46.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-16T05:26:46.095 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:46.119 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:46.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:46.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:46.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:45 smithi196 ceph-mon[30802]: pgmap v1690: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%) 2024-09-16T05:26:46.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:45 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:46.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2555447685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:45 smithi196 ceph-mon[30802]: osdmap e1002: 12 total, 11 up, 12 in 2024-09-16T05:26:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:45 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:26:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:45 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:26:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:45 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 18 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:26:46.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:45 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 16/220 objects degraded (7.273%), 6 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:46.243 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:46.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:46.294 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:46.345 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:46.368 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:46.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:46.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:45 smithi204 ceph-mon[30933]: pgmap v1690: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%) 2024-09-16T05:26:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:45 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2555447685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:26:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:45 smithi204 ceph-mon[30933]: osdmap e1002: 12 total, 11 up, 12 in 2024-09-16T05:26:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:45 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:26:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:45 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:26:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:45 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 18 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:26:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:45 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 16/220 objects degraded (7.273%), 6 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:46.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:46.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:46.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:46.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:46.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:46.704 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:46.705 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:46.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:46.742 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:46.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:46.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:46.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:46.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:46.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:46.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:46 smithi196 ceph-mon[30802]: 2.52 scrub starts 2024-09-16T05:26:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:46 smithi196 ceph-mon[30802]: 2.52 scrub ok 2024-09-16T05:26:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:46 smithi196 ceph-mon[30802]: 1.19 scrub starts 2024-09-16T05:26:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:46 smithi196 ceph-mon[30802]: 1.19 scrub ok 2024-09-16T05:26:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/367303105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:46 smithi196 ceph-mon[30802]: 1.18 scrub starts 2024-09-16T05:26:46.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:46 smithi196 ceph-mon[30802]: 1.18 scrub ok 2024-09-16T05:26:46.995 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:46.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:47.079 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:47.079 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:47.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:47.102 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:26:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:26:47.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:47.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:47.124 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:47.124 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:47.145 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:47.145 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:47.166 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:47.167 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:47.188 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:47.189 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:47.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:46 smithi032 ceph-mon[27388]: 2.52 scrub starts 2024-09-16T05:26:47.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:46 smithi032 ceph-mon[27388]: 2.52 scrub ok 2024-09-16T05:26:47.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:46 smithi032 ceph-mon[27388]: 1.19 scrub starts 2024-09-16T05:26:47.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:46 smithi032 ceph-mon[27388]: 1.19 scrub ok 2024-09-16T05:26:47.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/367303105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:47.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:46 smithi032 ceph-mon[27388]: 1.18 scrub starts 2024-09-16T05:26:47.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:46 smithi032 ceph-mon[27388]: 1.18 scrub ok 2024-09-16T05:26:47.210 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:47.210 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:47.210 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:47.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:47.232 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:47.232 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:47.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:47.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:47.255 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:47.255 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:47.277 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:47.277 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:47.299 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:47.299 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:47.345 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:47.367 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:47.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:47.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-16T05:26:47.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:46 smithi204 ceph-mon[30933]: 2.52 scrub starts 2024-09-16T05:26:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:46 smithi204 ceph-mon[30933]: 2.52 scrub ok 2024-09-16T05:26:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:46 smithi204 ceph-mon[30933]: 1.19 scrub starts 2024-09-16T05:26:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:46 smithi204 ceph-mon[30933]: 1.19 scrub ok 2024-09-16T05:26:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/367303105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:46 smithi204 ceph-mon[30933]: 1.18 scrub starts 2024-09-16T05:26:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:46 smithi204 ceph-mon[30933]: 1.18 scrub ok 2024-09-16T05:26:47.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:47.490 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:47.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:47.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:47.616 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:47.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:47.623 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:47.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:47.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:26:47.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:47.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:47.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:47.862 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:47.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:47.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:47.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:47.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:48.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:47 smithi196 ceph-mon[30802]: pgmap v1692: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%) 2024-09-16T05:26:48.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:47 smithi196 ceph-mon[30802]: 2.37 scrub starts 2024-09-16T05:26:48.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:47 smithi196 ceph-mon[30802]: 2.37 scrub ok 2024-09-16T05:26:48.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:47 smithi196 ceph-mon[30802]: 2.6 scrub starts 2024-09-16T05:26:48.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:47 smithi196 ceph-mon[30802]: 2.6 scrub ok 2024-09-16T05:26:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:48.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:48.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:48.103 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:26:48.103 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5587 2.34 2:2d3f13f8:::smithi20457400-7:head [setxattr _header (57) in=7b,truncate 3260723] snapc e6=[e6,e5,e2,e1,e0,dd,dc,d9,d8,d7,ca,c3] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 475.29598112000002, 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.147062895, 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.104 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5587 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805838+0000", 2024-09-16T05:26:48.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805839+0000", 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8299999999999997e-07 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805842+0000", 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6790000000000001e-06 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805846+0000", 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4089999999999996e-06 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805849+0000", 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.191e-06 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806887+0000", 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010375440000000001 2024-09-16T05:26:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.806953+0000", 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5833000000000005e-05 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807079+0000", 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012666500000000001 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.809689+0000", 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026092860000000002 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952747+0000", 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14305815 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952826+0000", 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9773999999999999e-05 2024-09-16T05:26:48.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952901+0000", 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4781000000000001e-05 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:5588 2.34 2:2d3f13f8:::smithi20457400-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e583)", 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 475.29595464699997, 2024-09-16T05:26:48.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14722005799999999, 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 5588 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:26:48.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805865+0000", 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.65e-07 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805867+0000", 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.86e-06 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805872+0000", 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8250000000000004e-06 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.805876+0000", 2024-09-16T05:26:48.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0380000000000001e-06 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807396+0000", 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001520448 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.807470+0000", 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3695000000000005e-05 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.952937+0000", 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14546690300000001 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953007+0000", 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9801999999999993e-05 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:18:52.953085+0000", 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8121999999999995e-05 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.115 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:534 2.34 e617/278 ondisk, result = 0)", 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 441.28966451100001, 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93956844500000003, 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812155+0000", 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812156+0000", 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0599999999999999e-07 2024-09-16T05:26:48.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:26.812160+0000", 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.437e-06 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751436+0000", 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93927549399999999 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751483+0000", 2024-09-16T05:26:48.119 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7117999999999998e-05 2024-09-16T05:26:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751598+0000", 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115448 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751609+0000", 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1156e-05 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:19:27.751723+0000", 2024-09-16T05:26:48.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113986 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7360 2.3 2:c018a787:::smithi20457400-18:head [write 443982~796776,stat] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 311.20486585200001, 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.046574619999999997, 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.122 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7360 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896954+0000", 2024-09-16T05:26:48.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.896958+0000", 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3869999999999997e-06 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897803+0000", 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084549400000000002 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897817+0000", 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3172e-05 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897822+0000", 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7400000000000001e-06 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897880+0000", 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7275999999999998e-05 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.897887+0000", 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0450000000000003e-06 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899506+0000", 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016191300000000001 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899612+0000", 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010592200000000001 2024-09-16T05:26:48.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.899807+0000", 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019501699999999999 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.900940+0000", 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011327760000000001 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:48.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943442+0000", 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.042502861000000003 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943488+0000", 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5266000000000003e-05 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943528+0000", 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0534000000000002e-05 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.128 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7361 2.3 2:c018a787:::smithi20457400-18:head [write 1989231~479018] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 311.198200972, 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040030532000000001, 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.129 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7361 2024-09-16T05:26:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:26:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903618+0000", 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.903620+0000", 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.885e-06 2024-09-16T05:26:48.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904044+0000", 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00042330699999999999 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904056+0000", 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2088e-05 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904064+0000", 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3820000000000004e-06 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904117+0000", 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2352000000000003e-05 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904226+0000", 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109895 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.904403+0000", 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017676899999999999 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.905359+0000", 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00095604999999999996 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943572+0000", 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038213241000000002 2024-09-16T05:26:48.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943611+0000", 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8668999999999999e-05 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943649+0000", 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7894e-05 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7362 2.3 2:c018a787:::smithi20457400-18:head [write 2959167~284815] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 311.19395060199997, 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035905712999999999, 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7362 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.137 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907869+0000", 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.907870+0000", 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4200000000000005e-07 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908124+0000", 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025446300000000002 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908135+0000", 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0964999999999999e-05 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908143+0000", 2024-09-16T05:26:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5190000000000003e-06 2024-09-16T05:26:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9714e-05 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908239+0000", 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6212999999999999e-05 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:26:48.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908346+0000", 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010734999999999999 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909154+0000", 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080830699999999997 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943689+0000", 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034534978000000001 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943732+0000", 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2558999999999997e-05 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943775+0000", 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2803000000000003e-05 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7363 2.3 2:c018a787:::smithi20457400-18:head [setxattr _header (58) in=7b,truncate 3243982] snapc 139=[137,136,135,134,133,132,12f,12c,12a,126,125,119,116] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 311.193662303, 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035723752999999997, 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7363 2024-09-16T05:26:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:26:48.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908157+0000", 2024-09-16T05:26:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908158+0000", 2024-09-16T05:26:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2499999999999995e-07 2024-09-16T05:26:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908161+0000", 2024-09-16T05:26:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1410000000000001e-06 2024-09-16T05:26:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908167+0000", 2024-09-16T05:26:48.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.674e-06 2024-09-16T05:26:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908173+0000", 2024-09-16T05:26:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4149999999999998e-06 2024-09-16T05:26:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908684+0000", 2024-09-16T05:26:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051148000000000003 2024-09-16T05:26:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908742+0000", 2024-09-16T05:26:48.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7358999999999997e-05 2024-09-16T05:26:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:26:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908839+0000", 2024-09-16T05:26:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6930999999999997e-05 2024-09-16T05:26:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909158+0000", 2024-09-16T05:26:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031886399999999997 2024-09-16T05:26:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.152 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943805+0000", 2024-09-16T05:26:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034647491000000002 2024-09-16T05:26:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943842+0000", 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7004e-05 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943881+0000", 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8868999999999997e-05 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.154 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7364 2.3 2:c018a787:::smithi20457400-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e747)", 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 311.193637443, 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035867990000000002, 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.155 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7364 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908182+0000", 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5999999999999999e-07 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908185+0000", 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1600000000000001e-06 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908191+0000", 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6869999999999997e-06 2024-09-16T05:26:48.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.908196+0000", 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9989999999999999e-06 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909088+0000", 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00089121600000000001 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.909143+0000", 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5887000000000002e-05 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943913+0000", 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034769683000000003 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.943981+0000", 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.792e-05 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:36.944050+0000", 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8978000000000003e-05 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.160 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(osd.5.0:830 2.3 e799/329 ondisk, result = 0)", 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 259.23155470900002, 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.330002407, 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870265+0000", 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870266+0000", 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.223e-06 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:28.870277+0000", 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0560999999999999e-05 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.199989+0000", 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.32971226399999998 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200004+0000", 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5366000000000002e-05 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200139+0000", 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013504000000000001 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200150+0000", 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0305999999999999e-05 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:22:29.200267+0000", 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011764699999999999 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9051 2.3 2:c018a787:::smithi20457400-18:head [append 0~0] snapc 189=[189,188,187,186,185,184,181,17e,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e909)", 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:19.162722+0000", 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 148.939097414, 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.34937753599999999, 2024-09-16T05:26:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9051 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162722+0000", 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162722+0000", 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162722+0000", 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.65e-07 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162725+0000", 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1940000000000001e-06 2024-09-16T05:26:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162735+0000", 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0592e-05 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162739+0000", 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.591e-06 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162836+0000", 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6895e-05 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.162842+0000", 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3589999999999996e-06 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.165934+0000", 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030923370000000001 2024-09-16T05:26:48.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.166031+0000", 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6910000000000006e-05 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2", 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.166203+0000", 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017222799999999999 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.166680+0000", 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00047637599999999998 2024-09-16T05:26:48.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.511963+0000", 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.345283436 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.512039+0000", 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5852999999999994e-05 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:19.512100+0000", 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.05e-05 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9642 2.34 2:2cffa4d6:::smithi20457400-50:head [setxattr _header (58) in=7b,truncate 2403731] snapc 198=[16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e947)", 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:58.380051+0000", 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.72176892100001, 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.388745954, 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:48.172 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9642 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380051+0000", 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380051+0000", 2024-09-16T05:26:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380051+0000", 2024-09-16T05:26:48.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9299999999999998e-07 2024-09-16T05:26:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380062+0000", 2024-09-16T05:26:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0917999999999999e-05 2024-09-16T05:26:48.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380067+0000", 2024-09-16T05:26:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7310000000000001e-06 2024-09-16T05:26:48.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380070+0000", 2024-09-16T05:26:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7780000000000001e-06 2024-09-16T05:26:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381091+0000", 2024-09-16T05:26:48.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010207219999999999 2024-09-16T05:26:48.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381152+0000", 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1257e-05 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381243+0000", 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0611000000000004e-05 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381707+0000", 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046380999999999999 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768681+0000", 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.38697416299999998 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:48.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768722+0000", 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1124000000000001e-05 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768796+0000", 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4346999999999998e-05 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9643 2.34 2:2cffa4d6:::smithi20457400-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e947)", 2024-09-16T05:26:48.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:24:58.380076+0000", 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.721743383, 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.38888499500000001, 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9643 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.182 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380076+0000", 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380076+0000", 2024-09-16T05:26:48.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380076+0000", 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7099999999999997e-07 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380078+0000", 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.361e-06 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380083+0000", 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2839999999999997e-06 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.380087+0000", 2024-09-16T05:26:48.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.427e-06 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381622+0000", 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015349999999999999 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.381671+0000", 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9447000000000003e-05 2024-09-16T05:26:48.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768830+0000", 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.38715916900000003 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768895+0000", 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4467000000000005e-05 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:24:58.768961+0000", 2024-09-16T05:26:48.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6469000000000003e-05 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9806 2.3b 2:de669526:::smithi20457400-23:head [write 723064~731129] snapc 1a2=[1a2,1a1,1a0,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:25:15.650973+0000", 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 92.450846827000007, 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034474850000000001, 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:48.188 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9806 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.650973+0000", 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.650973+0000", 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.650974+0000", 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.804e-06 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651800+0000", 2024-09-16T05:26:48.191 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:48.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:48.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082599400000000003 2024-09-16T05:26:48.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651816+0000", 2024-09-16T05:26:48.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6044000000000001e-05 2024-09-16T05:26:48.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651824+0000", 2024-09-16T05:26:48.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4630000000000004e-06 2024-09-16T05:26:48.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651888+0000", 2024-09-16T05:26:48.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4407999999999997e-05 2024-09-16T05:26:48.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.651981+0000", 2024-09-16T05:26:48.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3122999999999996e-05 2024-09-16T05:26:48.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4", 2024-09-16T05:26:48.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.652156+0000", 2024-09-16T05:26:48.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017429300000000001 2024-09-16T05:26:48.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:48.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.654012+0000", 2024-09-16T05:26:48.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001855953 2024-09-16T05:26:48.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:48.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.654312+0000", 2024-09-16T05:26:48.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00030017500000000002 2024-09-16T05:26:48.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:48.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.654371+0000", 2024-09-16T05:26:48.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8795000000000002e-05 2024-09-16T05:26:48.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.654420+0000", 2024-09-16T05:26:48.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9432999999999999e-05 2024-09-16T05:26:48.199 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.199 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.199 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.200 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(1.12,GRANT,e:992,from:3,reservation_nonce:0)", 2024-09-16T05:26:48.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:28.028804+0000", 2024-09-16T05:26:48.200 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.073015588000001, 2024-09-16T05:26:48.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027941400000000002, 2024-09-16T05:26:48.200 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.028804+0000", 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.028804+0000", 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.028805+0000", 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4300000000000001e-07 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.028814+0000", 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7900000000000005e-06 2024-09-16T05:26:48.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.028847+0000", 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2969999999999998e-05 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.028853+0000", 2024-09-16T05:26:48.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6810000000000001e-06 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.028914+0000", 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0745999999999997e-05 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.028933+0000", 2024-09-16T05:26:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8510999999999999e-05 2024-09-16T05:26:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029083+0000", 2024-09-16T05:26:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015077299999999999 2024-09-16T05:26:48.205 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(1.12 e488 from shard 3)", 2024-09-16T05:26:48.206 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:28.029735+0000", 2024-09-16T05:26:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.072084189000002, 2024-09-16T05:26:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016563299999999999, 2024-09-16T05:26:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.207 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029735+0000", 2024-09-16T05:26:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029735+0000", 2024-09-16T05:26:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029736+0000", 2024-09-16T05:26:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1500000000000004e-07 2024-09-16T05:26:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029740+0000", 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0099999999999997e-06 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029796+0000", 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5550000000000002e-05 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029801+0000", 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7019999999999998e-06 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029863+0000", 2024-09-16T05:26:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1513000000000005e-05 2024-09-16T05:26:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029875+0000", 2024-09-16T05:26:48.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1922999999999999e-05 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.029901+0000", 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.622e-05 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.57,QREQUEST,e:992,from:10,reservation_nonce:5)", 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:28.247490+0000", 2024-09-16T05:26:48.213 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.854329942, 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027933, 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247490+0000", 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247490+0000", 2024-09-16T05:26:48.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247492+0000", 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.526e-06 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247523+0000", 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1084000000000003e-05 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247573+0000", 2024-09-16T05:26:48.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9613000000000001e-05 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247600+0000", 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7058000000000001e-05 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247701+0000", 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101264 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247720+0000", 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.878e-05 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.247769+0000", 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9005000000000003e-05 2024-09-16T05:26:48.217 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "replica_scrub(pg: 2.57,from:0'0,to:0'0,epoch:992/739,start:2:ea000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:28.248566+0000", 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.853253910999999, 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024372200000000001, 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248566+0000", 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248566+0000", 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248575+0000", 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2029999999999998e-06 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248597+0000", 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1846000000000001e-05 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248627+0000", 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0448999999999999e-05 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248651+0000", 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3649e-05 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248719+0000", 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8546000000000004e-05 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248733+0000", 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4201e-05 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.248809+0000", 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5828000000000007e-05 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.57,RELEASE,e:992,from:10,reservation_nonce:0)", 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:28.249611+0000", 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 19.852208940000001, 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00023601500000000001, 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249611+0000", 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249611+0000", 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249611+0000", 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2700000000000001e-07 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249612+0000", 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.587e-06 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249677+0000", 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4817e-05 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249683+0000", 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1850000000000001e-06 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249821+0000", 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013776399999999999 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249836+0000", 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4742999999999999e-05 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:28.249847+0000", 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0791999999999999e-05 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.226 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.4b,QREQUEST,e:993,from:2,reservation_nonce:4)", 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:30.202458+0000", 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 17.899361669000001, 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00039521700000000001, 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:48.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202458+0000", 2024-09-16T05:26:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202458+0000", 2024-09-16T05:26:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202460+0000", 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1660000000000001e-06 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202469+0000", 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1800000000000002e-06 2024-09-16T05:26:48.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202538+0000", 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9098999999999997e-05 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202553+0000", 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5238999999999999e-05 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202706+0000", 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000152447 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202747+0000", 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1353000000000002e-05 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:48.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.202853+0000", 2024-09-16T05:26:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105733 2024-09-16T05:26:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:48.232 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:48.232 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:26:48.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:47 smithi032 ceph-mon[27388]: pgmap v1692: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%) 2024-09-16T05:26:48.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:47 smithi032 ceph-mon[27388]: 2.37 scrub starts 2024-09-16T05:26:48.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:47 smithi032 ceph-mon[27388]: 2.37 scrub ok 2024-09-16T05:26:48.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:47 smithi032 ceph-mon[27388]: 2.6 scrub starts 2024-09-16T05:26:48.233 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:47 smithi032 ceph-mon[27388]: 2.6 scrub ok 2024-09-16T05:26:48.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:48.315 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:48.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:48.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:48.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:47 smithi204 ceph-mon[30933]: pgmap v1692: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/220 objects degraded (7.273%); 5/220 objects misplaced (2.273%) 2024-09-16T05:26:48.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:47 smithi204 ceph-mon[30933]: 2.37 scrub starts 2024-09-16T05:26:48.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:47 smithi204 ceph-mon[30933]: 2.37 scrub ok 2024-09-16T05:26:48.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:47 smithi204 ceph-mon[30933]: 2.6 scrub starts 2024-09-16T05:26:48.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:47 smithi204 ceph-mon[30933]: 2.6 scrub ok 2024-09-16T05:26:48.443 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:48.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:48.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:48.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:48.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:48.569 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 8, 9, 11, 10, 1, 6] out_osds: [] dead_osds: [7] live_osds: [0, 3, 4, 5, 6, 8, 9, 10, 11, 2, 1] 2024-09-16T05:26:48.569 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-16T05:26:48.572 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-09-16T05:26:48.572 DEBUG:teuthology.orchestra.run.smithi196:osd.7> sudo journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.7.service 2024-09-16T05:26:48.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl start ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.7 2024-09-16T05:26:48.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:48.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:48.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:48.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:48.844 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:48.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:26:48.935 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:48 smithi196 systemd[1]: Starting Ceph osd.7 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:26:48.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:48.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-16T05:26:49.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:49.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:49.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:49.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:49.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:49.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:49.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:48 smithi196 ceph-mon[30802]: 2.4a scrub starts 2024-09-16T05:26:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:48 smithi196 ceph-mon[30802]: 2.4a scrub ok 2024-09-16T05:26:49.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4206440688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:49.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 podman[120941]: 2024-09-16 05:26:49.049491083 +0000 UTC m=+0.017155638 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:26:49.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 podman[120941]: 2024-09-16 05:26:49.175905883 +0000 UTC m=+0.143570433 container create 987c2d2d7a60aecd8afe47a5405b0db25426ca3428de823e93cc8b5f81305e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:26:49.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:48 smithi032 ceph-mon[27388]: 2.4a scrub starts 2024-09-16T05:26:49.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:48 smithi032 ceph-mon[27388]: 2.4a scrub ok 2024-09-16T05:26:49.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4206440688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:49.261 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:49.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:49.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:49.388 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:49.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:48 smithi204 ceph-mon[30933]: 2.4a scrub starts 2024-09-16T05:26:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:48 smithi204 ceph-mon[30933]: 2.4a scrub ok 2024-09-16T05:26:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4206440688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:49.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:49.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:49.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:49.547 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 podman[120941]: 2024-09-16 05:26:49.4034639 +0000 UTC m=+0.371128447 container init 987c2d2d7a60aecd8afe47a5405b0db25426ca3428de823e93cc8b5f81305e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-16T05:26:49.548 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 podman[120941]: 2024-09-16 05:26:49.407574122 +0000 UTC m=+0.375238676 container start 987c2d2d7a60aecd8afe47a5405b0db25426ca3428de823e93cc8b5f81305e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True) 2024-09-16T05:26:49.548 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 podman[120941]: 2024-09-16 05:26:49.494877259 +0000 UTC m=+0.462541809 container attach 987c2d2d7a60aecd8afe47a5405b0db25426ca3428de823e93cc8b5f81305e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True) 2024-09-16T05:26:49.556 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:49.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:49.641 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:49.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:49.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:49.769 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:49.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:49.799 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:26:49.799 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 bash[120941]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:26:49.799 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:26:49.799 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:49 smithi196 bash[120941]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-16T05:26:49.871 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:49.898 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:49.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:49.999 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:50.025 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:50.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:50.045 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:26:50.045 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-09-16T05:26:50.045 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-09-16T05:26:50.045 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:26:50.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:26:50.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:50.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:50.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:50.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T05:26:50.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 bash[120941]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T05:26:50.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-16T05:26:50.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 bash[120941]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-16T05:26:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: pgmap v1693: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4206440688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: osdmap e1003: 12 total, 11 up, 12 in 2024-09-16T05:26:50.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: 2.20 deep-scrub starts 2024-09-16T05:26:50.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: 2.20 deep-scrub ok 2024-09-16T05:26:50.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:26:50.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:26:50.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: 1.15 scrub starts 2024-09-16T05:26:50.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:49 smithi196 ceph-mon[30802]: 1.15 scrub ok 2024-09-16T05:26:50.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:50.280 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:50.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:50.326 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:50.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:50.408 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:50.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:50.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: pgmap v1693: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:50.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:50.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4206440688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:50.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: osdmap e1003: 12 total, 11 up, 12 in 2024-09-16T05:26:50.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: 2.20 deep-scrub starts 2024-09-16T05:26:50.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: 2.20 deep-scrub ok 2024-09-16T05:26:50.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:26:50.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:26:50.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: 1.15 scrub starts 2024-09-16T05:26:50.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:49 smithi032 ceph-mon[27388]: 1.15 scrub ok 2024-09-16T05:26:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: pgmap v1693: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4206440688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: osdmap e1003: 12 total, 11 up, 12 in 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: 2.20 deep-scrub starts 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: 2.20 deep-scrub ok 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: 1.15 scrub starts 2024-09-16T05:26:50.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:49 smithi204 ceph-mon[30933]: 1.15 scrub ok 2024-09-16T05:26:50.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:50.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:50.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:50.611 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-16T05:26:50.611 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 bash[120941]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-16T05:26:50.611 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-16T05:26:50.611 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 bash[120941]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-16T05:26:50.611 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-16T05:26:50.612 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 bash[120941]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-16T05:26:50.612 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T05:26:50.612 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 bash[120941]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T05:26:50.612 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate[120985]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-16T05:26:50.612 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 bash[120941]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-16T05:26:50.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-16T05:26:50.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:50.661 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:50.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:50.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:50.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:50.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:50.865 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:50.866 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:50.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:50.887 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:26:50.887 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:26:50.908 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:50.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:50.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:50.927 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:50.941 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:50 smithi196 podman[120941]: 2024-09-16 05:26:50.658105792 +0000 UTC m=+1.625770334 container died 987c2d2d7a60aecd8afe47a5405b0db25426ca3428de823e93cc8b5f81305e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-16T05:26:50.952 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:51.253 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:51.253 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:51.253 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:51.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:51.254 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:51.254 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:51.254 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:51.255 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: 2.4e scrub starts 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: 2.4e scrub ok 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: 2.59 scrub starts 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: 2.59 scrub ok 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: 2.11 scrub starts 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: 2.11 scrub ok 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2524653630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:51.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:50 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 16/222 objects degraded (7.207%), 6 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:51.256 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:51.256 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:51.256 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:51.257 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:51.257 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:51.257 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:51.258 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:51.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: 2.4e scrub starts 2024-09-16T05:26:51.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: 2.4e scrub ok 2024-09-16T05:26:51.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: 2.59 scrub starts 2024-09-16T05:26:51.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: 2.59 scrub ok 2024-09-16T05:26:51.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:26:51.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:26:51.258 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: 2.11 scrub starts 2024-09-16T05:26:51.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: 2.11 scrub ok 2024-09-16T05:26:51.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2524653630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:51.259 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:50 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 16/222 objects degraded (7.207%), 6 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:51.283 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:51.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:51.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:51.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:51.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:51.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:51.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: 2.4e scrub starts 2024-09-16T05:26:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: 2.4e scrub ok 2024-09-16T05:26:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: 2.59 scrub starts 2024-09-16T05:26:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: 2.59 scrub ok 2024-09-16T05:26:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:26:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:26:51.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: 2.11 scrub starts 2024-09-16T05:26:51.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: 2.11 scrub ok 2024-09-16T05:26:51.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2524653630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:51.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:50 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 16/222 objects degraded (7.207%), 6 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2024-09-16T05:26:51.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:51.533 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:51.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:51.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:51.657 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:51.658 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:51.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:51.671 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:51 smithi196 podman[121184]: 2024-09-16 05:26:51.348058493 +0000 UTC m=+1.012597468 container remove 987c2d2d7a60aecd8afe47a5405b0db25426ca3428de823e93cc8b5f81305e5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-09-16T05:26:51.672 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:51 smithi196 podman[121339]: 2024-09-16 05:26:51.46025507 +0000 UTC m=+0.018869263 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:26:51.672 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:51 smithi196 podman[121339]: 2024-09-16 05:26:51.557061733 +0000 UTC m=+0.115675912 container create cb10ac9c829f709d3fed489f74ee31a7e50d6063c3630bfe4af320f47d64a9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-16T05:26:51.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:26:51.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:51.775 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:51.782 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:51.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:51.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:51.910 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:51.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:51.969 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:52.012 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:52.036 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:52.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: pgmap v1695: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: 2.d scrub starts 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: 2.d scrub ok 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: 1.2 deep-scrub starts 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: 1.2 deep-scrub ok 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: 1.1e scrub starts 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: 1.1e scrub ok 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:26:52.091 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:51 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:26:52.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:26:52.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:52.160 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:52.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:52.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: pgmap v1695: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:52.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: 2.d scrub starts 2024-09-16T05:26:52.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: 2.d scrub ok 2024-09-16T05:26:52.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: 1.2 deep-scrub starts 2024-09-16T05:26:52.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: 1.2 deep-scrub ok 2024-09-16T05:26:52.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: 1.1e scrub starts 2024-09-16T05:26:52.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: 1.1e scrub ok 2024-09-16T05:26:52.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:26:52.197 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:51 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:26:52.197 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:52 smithi196 podman[121339]: 2024-09-16 05:26:52.012170833 +0000 UTC m=+0.570785019 container init cb10ac9c829f709d3fed489f74ee31a7e50d6063c3630bfe4af320f47d64a9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-16T05:26:52.197 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:52 smithi196 podman[121339]: 2024-09-16 05:26:52.015920733 +0000 UTC m=+0.574534911 container start cb10ac9c829f709d3fed489f74ee31a7e50d6063c3630bfe4af320f47d64a9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8921c8e) 2024-09-16T05:26:52.197 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:52 smithi196 ceph-osd[121391]: -- 172.21.15.196:0/412128921 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a9d1217a40 con 0x55a9d1fde400 2024-09-16T05:26:52.197 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:52 smithi196 bash[121339]: cb10ac9c829f709d3fed489f74ee31a7e50d6063c3630bfe4af320f47d64a9b0 2024-09-16T05:26:52.198 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:52 smithi196 systemd[1]: Started Ceph osd.7 for c1b4ea6a-73e8-11ef-bceb-c7b262605968. 2024-09-16T05:26:52.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:52.286 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:52.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:52.300 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:52.300 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:52.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:52.414 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:52.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:52.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: pgmap v1695: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:52.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: 2.d scrub starts 2024-09-16T05:26:52.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: 2.d scrub ok 2024-09-16T05:26:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: 1.2 deep-scrub starts 2024-09-16T05:26:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: 1.2 deep-scrub ok 2024-09-16T05:26:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: 1.1e scrub starts 2024-09-16T05:26:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: 1.1e scrub ok 2024-09-16T05:26:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:26:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:51 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:26:52.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:52.540 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:52.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:52.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:52.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:52.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:52.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:52.789 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:52.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:52.891 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:52.914 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:52.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:52.953 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:26:53.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:53.038 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:53.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:53.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-16T05:26:53.079 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:53.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:53.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:53.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:53.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:52 smithi196 ceph-mon[30802]: osdmap e1004: 12 total, 11 up, 12 in 2024-09-16T05:26:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:52 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:26:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:52 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:26:53.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1797618447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:53.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:53.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:52 smithi032 ceph-mon[27388]: osdmap e1004: 12 total, 11 up, 12 in 2024-09-16T05:26:53.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:52 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:26:53.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:52 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:26:53.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1797618447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:53.289 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:53.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:53.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:53.417 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:53.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:53.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:52 smithi204 ceph-mon[30933]: osdmap e1004: 12 total, 11 up, 12 in 2024-09-16T05:26:53.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:52 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:26:53.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:52 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:26:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1797618447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:26:53.519 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:53.544 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:53.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:53.574 INFO:teuthology.orchestra.run.smithi196.stderr:no valid command found; 10 closest matches: 2024-09-16T05:26:53.574 INFO:teuthology.orchestra.run.smithi196.stderr:0 2024-09-16T05:26:53.574 INFO:teuthology.orchestra.run.smithi196.stderr:1 2024-09-16T05:26:53.574 INFO:teuthology.orchestra.run.smithi196.stderr:2 2024-09-16T05:26:53.574 INFO:teuthology.orchestra.run.smithi196.stderr:abort 2024-09-16T05:26:53.574 INFO:teuthology.orchestra.run.smithi196.stderr:assert 2024-09-16T05:26:53.574 INFO:teuthology.orchestra.run.smithi196.stderr:config diff 2024-09-16T05:26:53.574 INFO:teuthology.orchestra.run.smithi196.stderr:config diff get 2024-09-16T05:26:53.577 INFO:teuthology.orchestra.run.smithi196.stderr:config get 2024-09-16T05:26:53.577 INFO:teuthology.orchestra.run.smithi196.stderr:config help [] 2024-09-16T05:26:53.577 INFO:teuthology.orchestra.run.smithi196.stderr:config set ... 2024-09-16T05:26:53.577 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: invalid command 2024-09-16T05:26:53.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:53.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:53.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:53.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:53.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:26:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:53.794 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:53.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:53.819 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:53.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:53.917 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:53.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:53.954 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:53 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:26:53.675+0000 7fd6f34d7640 -1 Falling back to public interface 2024-09-16T05:26:54.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:54.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:54.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:54.045 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:54.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:54.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:54.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:54.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:53 smithi196 ceph-mon[30802]: pgmap v1697: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:53 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1797618447' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:54.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:53 smithi196 ceph-mon[30802]: osdmap e1005: 12 total, 11 up, 12 in 2024-09-16T05:26:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T05:26:54.268 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-16T05:26:54.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:54.295 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:54.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:54.305 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:53 smithi032 ceph-mon[27388]: pgmap v1697: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:54.305 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:53 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:54.305 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1797618447' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:54.305 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:53 smithi032 ceph-mon[27388]: osdmap e1005: 12 total, 11 up, 12 in 2024-09-16T05:26:54.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:54.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:53 smithi204 ceph-mon[30933]: pgmap v1697: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:53 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:26:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1797618447' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:26:54.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:53 smithi204 ceph-mon[30933]: osdmap e1005: 12 total, 11 up, 12 in 2024-09-16T05:26:54.422 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:54.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:54.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:54.548 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:54.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:54.549 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:54.550 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:54.570 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:26:54.570 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:26:54.591 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:54.591 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:54.611 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:54.611 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:54.632 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:54.633 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:54.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:54.653 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:54.653 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:54.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:54.674 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:54.674 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:54.674 INFO:teuthology.orchestra.run.smithi032.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T05:26:54.697 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:54.697 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:54.718 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:54.719 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.24571.0:4414 2.3f e499/293 2:fc733d2f:::smithi20457400-45:head v 499'316, mlcod=472'312)", 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 565.97631140099998, 2024-09-16T05:26:54.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94419809600000004, 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:26:54.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.749987+0000", 2024-09-16T05:26:54.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8500000000000002e-07 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:28.751332+0000", 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001345001 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691571+0000", 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94023934399999998 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691580+0000", 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-06 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691653+0000", 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3137000000000001e-05 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.691676+0000", 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3349e-05 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694130+0000", 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024536900000000001 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.694185+0000", 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4750000000000003e-05 2024-09-16T05:26:54.733 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4417 2.29 2:96dc17ef:::smithi20457400-10:head [copy-from ver 329] snapc ba=[ba,b9,b8] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 564.98218421599995, 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017204743000000002, 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.734 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4417 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744114+0000", 2024-09-16T05:26:54.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744116+0000", 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4099999999999998e-06 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744129+0000", 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2717e-05 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744150+0000", 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1246e-05 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744160+0000", 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.815e-06 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744237+0000", 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6860000000000006e-05 2024-09-16T05:26:54.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.744246+0000", 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1220000000000001e-06 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.745711+0000", 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014654679999999999 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.745823+0000", 2024-09-16T05:26:54.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011192300000000001 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.755847+0000", 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010023119 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.759255+0000", 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034084950000000001 2024-09-16T05:26:54.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761242+0000", 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019874369999999999 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761284+0000", 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1984000000000003e-05 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:29.761319+0000", 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4146999999999997e-05 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.7.0:741 2.2d e509/252 2:b7234079:::smithi20457400-42:bc v 509'382, mlcod=507'378)", 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 555.69784976799997, 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72794596499999997, 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:54.741 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028448+0000", 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028449+0000", 2024-09-16T05:26:54.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5400000000000001e-07 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.028455+0000", 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4339999999999997e-06 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755790+0000", 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72733470600000005 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.744 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:54.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755802+0000", 2024-09-16T05:26:54.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1771e-05 2024-09-16T05:26:54.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755868+0000", 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6512000000000003e-05 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.755889+0000", 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0245e-05 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.756344+0000", 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045536699999999998 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.756394+0000", 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0275999999999998e-05 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:4547 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 303] snapc be=[be] ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 554.95857712600002, 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.01820447, 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.747 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4547 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767721+0000", 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767723+0000", 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0549999999999998e-06 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767732+0000", 2024-09-16T05:26:54.749 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9930000000000001e-06 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767748+0000", 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5715000000000001e-05 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767764+0000", 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5811000000000002e-05 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767896+0000", 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013272800000000001 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.767969+0000", 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3120999999999993e-05 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.780323+0000", 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012353563 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.782979+0000", 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026563440000000002 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785858+0000", 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028785719999999998 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785889+0000", 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1569000000000002e-05 2024-09-16T05:26:54.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:39.785925+0000", 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5998999999999999e-05 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(osd.8.0:932 2.1a e526/291 2:5827f14d:::smithi20457400-12:c3 v 526'716, mlcod=518'711)", 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 538.96792752399995, 2024-09-16T05:26:54.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.24009196499999999, 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758371+0000", 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3300000000000001e-07 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.758373+0000", 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1909999999999999e-06 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997774+0000", 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.23940144999999999 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997782+0000", 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7379999999999994e-06 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997836+0000", 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3557000000000002e-05 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.997855+0000", 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9491999999999999e-05 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.998416+0000", 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056088899999999996 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:17:55.998462+0000", 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6514999999999998e-05 2024-09-16T05:26:54.757 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6720 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 624] snapc 116=[116,115,113,112,10c,10b,10a,109,108,106,101,f2] ondisk+write+known_if_redirected+supports_pool_eio e679)", 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:29.354895+0000", 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 385.37140260799998, 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019406367000000001, 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6720 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354895+0000", 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354895+0000", 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354897+0000", 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5540000000000001e-06 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354906+0000", 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2439999999999999e-06 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354923+0000", 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6796e-05 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354930+0000", 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9700000000000002e-06 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.355017+0000", 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7447000000000005e-05 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.355127+0000", 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010949 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.367256+0000", 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012129457999999999 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.371746+0000", 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0044893679999999997 2024-09-16T05:26:54.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.374189+0000", 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024428330000000002 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.374254+0000", 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5578000000000001e-05 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.374302+0000", 2024-09-16T05:26:54.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7629000000000001e-05 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:6721 2.29 2:9662c29c:::smithi20457400-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e679)", 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:20:29.354949+0000", 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 385.37134902999998, 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016035351999999999, 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.764 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6721 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354949+0000", 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354949+0000", 2024-09-16T05:26:54.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354949+0000", 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.34e-07 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354978+0000", 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8634000000000001e-05 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354983+0000", 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3260000000000003e-06 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.354988+0000", 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8080000000000004e-06 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.355284+0000", 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029562799999999998 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.355316+0000", 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2675000000000001e-05 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.370849+0000", 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015532750999999999 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.370913+0000", 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3986999999999996e-05 2024-09-16T05:26:54.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:20:29.370984+0000", 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1209000000000002e-05 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7200 2.29 2:9662c29c:::smithi20457400-9:head [copy-from ver 1175] snapc 130=[12f,12e,12c,12b,12a,129,128,126,125,124,123,120,119,116,108] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:18.170705+0000", 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 336.55559332600001, 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.022304378, 2024-09-16T05:26:54.769 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7200 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170705+0000", 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170705+0000", 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170708+0000", 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8959999999999999e-06 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170720+0000", 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.203e-05 2024-09-16T05:26:54.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170769+0000", 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8912000000000002e-05 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170785+0000", 2024-09-16T05:26:54.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.668e-05 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170955+0000", 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000169443 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.171085+0000", 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013044599999999999 2024-09-16T05:26:54.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.186857+0000", 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015772083999999999 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.191857+0000", 2024-09-16T05:26:54.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004999603 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.192929+0000", 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010723009999999999 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.192971+0000", 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1384000000000002e-05 2024-09-16T05:26:54.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.193009+0000", 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8599000000000001e-05 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:7201 2.29 2:9662c29c:::smithi20457400-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e728)", 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:21:18.170827+0000", 2024-09-16T05:26:54.776 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 336.55547139499998, 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020617390999999999, 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 7201 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170827+0000", 2024-09-16T05:26:54.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170827+0000", 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170828+0000", 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9999999999999995e-07 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170833+0000", 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.87e-06 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170845+0000", 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2326e-05 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.170854+0000", 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0159999999999997e-06 2024-09-16T05:26:54.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.171306+0000", 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045261100000000001 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for blocked object", 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.171346+0000", 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9038999999999997e-05 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.191222+0000", 2024-09-16T05:26:54.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019876441000000002 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.191328+0000", 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106251 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:21:18.191444+0000", 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115837 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.781 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8547 2.29 2:96dc17ef:::smithi20457400-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e855)", 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:24.875781+0000", 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 209.850517416, 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72819094100000004, 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.782 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8547 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875781+0000", 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875781+0000", 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875782+0000", 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8220000000000001e-06 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875787+0000", 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7190000000000001e-06 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875800+0000", 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2928999999999999e-05 2024-09-16T05:26:54.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875808+0000", 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6750000000000002e-06 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875975+0000", 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016753300000000001 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876072+0000", 2024-09-16T05:26:54.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.692e-05 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.603869+0000", 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72779689700000005 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.603972+0000", 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102446 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8548 2.29 2:96dc17ef:::smithi20457400-10:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e855)", 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:24.875858+0000", 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 209.850439736, 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72821206299999997, 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8548 2024-09-16T05:26:54.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875858+0000", 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875858+0000", 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875859+0000", 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0299999999999998e-07 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875864+0000", 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5959999999999997e-06 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875874+0000", 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0178e-05 2024-09-16T05:26:54.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875881+0000", 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0550000000000002e-06 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876143+0000", 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026236400000000001 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876228+0000", 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4697999999999994e-05 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.603993+0000", 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72776453200000002 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.604070+0000", 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7836999999999994e-05 2024-09-16T05:26:54.791 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:8549 2.29 2:96dc17ef:::smithi20457400-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e855)", 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:23:24.875893+0000", 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 209.85040465, 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72825555500000005, 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 8549 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875893+0000", 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875893+0000", 2024-09-16T05:26:54.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875894+0000", 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7400000000000003e-07 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875908+0000", 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4518999999999999e-05 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875916+0000", 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9440000000000005e-06 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.875923+0000", 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1859999999999997e-06 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876470+0000", 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000546906 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for rw locks", 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:24.876617+0000", 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014755700000000001 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.604090+0000", 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.72747304700000004 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:23:25.604149+0000", 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8822e-05 2024-09-16T05:26:54.796 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9723 2.29 2:9423b7dc:::smithi20457400-41:head [read 0~2487766,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e956)", 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:25:07.417102+0000", 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 107.309195927, 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057144810000000004, 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.797 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9723 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417102+0000", 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417102+0000", 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417104+0000", 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.277e-06 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417126+0000", 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1552999999999998e-05 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417141+0000", 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5213e-05 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417150+0000", 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8959999999999992e-06 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417228+0000", 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8223e-05 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417237+0000", 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3000000000000002e-06 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.419517+0000", 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022801560000000002 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.419642+0000", 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012501799999999999 2024-09-16T05:26:54.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.422817+0000", 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031748449999999999 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9724 2.29 2:9423b7dc:::smithi20457400-41:head [sparse-read 0~1794885] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e956)", 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:25:07.417171+0000", 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 107.309127249, 2024-09-16T05:26:54.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059954429999999996, 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9724 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417171+0000", 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.804 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:54.805 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:54.805 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:54.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417171+0000", 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417172+0000", 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3600000000000003e-07 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417175+0000", 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5379999999999998e-06 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417183+0000", 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.977e-06 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417189+0000", 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4339999999999997e-06 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417298+0000", 2024-09-16T05:26:54.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010897599999999999 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417304+0000", 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.716e-06 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.422859+0000", 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055546830000000004 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.422970+0000", 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111513 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.423166+0000", 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019587000000000001 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.809 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9725 2.29 2:9423b7dc:::smithi20457400-41:head [read 0~2487766,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e956)", 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:25:07.417201+0000", 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 107.30909700399999, 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068888819999999998, 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9725 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417201+0000", 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417201+0000", 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417202+0000", 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3200000000000005e-07 2024-09-16T05:26:54.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417206+0000", 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5959999999999997e-06 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417213+0000", 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7830000000000001e-06 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417219+0000", 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9279999999999998e-06 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417356+0000", 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013708899999999999 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting_for_map not empty", 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.417363+0000", 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8789999999999997e-06 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.423196+0000", 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058330639999999998 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.423292+0000", 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5727e-05 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:07.424090+0000", 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079828400000000002 2024-09-16T05:26:54.814 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24571.0:9801 2.29 2:94d33c56:::smithi20457400-8:head [rollback 16a] snapc 1a2=[1a2,1a1,1a0,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:25:15.473019+0000", 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 99.253279043000006, 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036096700000000002, 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-09-16T05:26:54.815 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24571", 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 9801 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.473019+0000", 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.473019+0000", 2024-09-16T05:26:54.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.473020+0000", 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3000000000000001e-07 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.473022+0000", 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6589999999999999e-06 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.473028+0000", 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.891e-06 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.473034+0000", 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6080000000000003e-06 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.474035+0000", 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010012650000000001 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.474092+0000", 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6988999999999999e-05 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 3", 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.474228+0000", 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000135778 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.474748+0000", 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00051989500000000004 2024-09-16T05:26:54.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.476494+0000", 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017460220000000001 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.476557+0000", 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3115999999999997e-05 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:25:15.476629+0000", 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1916999999999996e-05 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.22,QREQUEST,e:980,from:10,reservation_nonce:4)", 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:06.021481+0000", 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 48.704816762, 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000299141, 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.821 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021481+0000", 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021481+0000", 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021484+0000", 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8899999999999999e-06 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021498+0000", 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3563e-05 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021543+0000", 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5705999999999999e-05 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021559+0000", 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5860000000000001e-05 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021695+0000", 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013608700000000001 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021731+0000", 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6010000000000003e-05 2024-09-16T05:26:54.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:06.021780+0000", 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9024999999999999e-05 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "replica_scrub(pg: 1.3,from:0'0,to:0'0,epoch:993/764,start:1:c0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:30.268855+0000", 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.457442816, 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034143599999999997, 2024-09-16T05:26:54.825 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.268855+0000", 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.268855+0000", 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.268864+0000", 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4460000000000001e-06 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.268876+0000", 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2547e-05 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.268945+0000", 2024-09-16T05:26:54.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8788000000000006e-05 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.268960+0000", 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.535e-05 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.269112+0000", 2024-09-16T05:26:54.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000152087 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.269136+0000", 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3609e-05 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:30.269197+0000", 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0609000000000002e-05 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.829 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDScrubReserve(2.20,GRANT,e:1003,from:5,reservation_nonce:0)", 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:49.041233+0000", 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.6850650859999998, 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027401299999999998, 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041233+0000", 2024-09-16T05:26:54.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041233+0000", 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041234+0000", 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6099999999999999e-07 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041238+0000", 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2549999999999999e-06 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041254+0000", 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5964000000000002e-05 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041262+0000", 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2400000000000007e-06 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041314+0000", 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1236000000000002e-05 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041329+0000", 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5628999999999999e-05 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.041507+0000", 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017782799999999999 2024-09-16T05:26:54.833 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(2.20 e997 from shard 5)", 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-16T05:26:49.051255+0000", 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 5.6750435079999999, 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065927599999999998, 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "continuous": false, 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051255+0000", 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051255+0000", 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051256+0000", 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.243e-06 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051263+0000", 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1080000000000004e-06 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051281+0000", 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7608000000000001e-05 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051289+0000", 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0509999999999997e-06 2024-09-16T05:26:54.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051850+0000", 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00056192099999999995 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051867+0000", 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6997999999999999e-05 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-09-16T05:26:54.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-09-16T05:26:49.051914+0000", 2024-09-16T05:26:54.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6347000000000002e-05 2024-09-16T05:26:54.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.838 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-16T05:26:54.838 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-16T05:26:54.838 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-16T05:26:54.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:54.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:54.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:54.990 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:54.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:55.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:55.117 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:55.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:55.218 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:55.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/544155551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:55.245 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:55.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:55.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:55.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:55.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/544155551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:55.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/544155551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T05:26:55.462 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T05:26:55.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:55.495 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:55.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:55.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-16T05:26:55.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:55.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:55.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:55.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:26:55.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:55.741 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:55.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:55.769 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:55.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:55.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:55.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:55.944 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:55.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:55.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:55.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:56.065 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:55 smithi032 ceph-mon[27388]: pgmap v1699: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:56.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:56.111 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:56.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:56.168 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:55 smithi204 ceph-mon[30933]: pgmap v1699: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:56.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:26:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:56.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:56.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:26:56.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:55 smithi196 ceph-mon[30802]: pgmap v1699: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:56.268 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:26:56.269 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:26:56.269 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:26:56.269 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:26:56.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:56.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:56.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:56.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:56.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:56.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:56.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:56.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:56.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:56.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:26:56.731 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:56.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:26:56.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:56.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:56.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:56.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:56.965 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:26:56.980 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:56.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:56.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-16T05:26:57.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/499631903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:57.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:56 smithi196 ceph-mon[30802]: from='osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:26:57.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:56 smithi196 ceph-mon[30802]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:26:57.026 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:56 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:26:56.742+0000 7fd6f34d7640 -1 osd.7 964 log_to_monitors true 2024-09-16T05:26:57.027 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:26:57 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:26:57.025+0000 7fd6eb282640 -1 osd.7 1006 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:26:57.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:57.103 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:57.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:57.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:57.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:57.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:57.273 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:57.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/499631903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:57.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:56 smithi032 ceph-mon[27388]: from='osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:26:57.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:56 smithi032 ceph-mon[27388]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:26:57.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:57.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:57.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:57.394 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/499631903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:26:57.394 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:56 smithi204 ceph-mon[30933]: from='osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:26:57.394 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:56 smithi204 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T05:26:57.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:57.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:26:57.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:57.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:57.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:57.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:57.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:57.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:57.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:57.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:57.731 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:57.770 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:26:57.770 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:26:57.771 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:26:57.771 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:26:57.771 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:26:57.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:57.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:57.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:26:57.974 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:57.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:57.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:26:58.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:26:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:58.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:58.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:26:58.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:58.205 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:26:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:26:58.226 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:26:58.226 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:26:58.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:57 smithi032 ceph-mon[27388]: pgmap v1700: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:58.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:57 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/499631903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:58.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:57 smithi032 ceph-mon[27388]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T05:26:58.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:57 smithi032 ceph-mon[27388]: from='osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:26:58.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:57 smithi032 ceph-mon[27388]: osdmap e1006: 12 total, 11 up, 12 in 2024-09-16T05:26:58.227 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:57 smithi032 ceph-mon[27388]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:26:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:58.228 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:58.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:57 smithi196 ceph-mon[30802]: pgmap v1700: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:57 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/499631903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:58.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:57 smithi196 ceph-mon[30802]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T05:26:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:57 smithi196 ceph-mon[30802]: from='osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:26:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:57 smithi196 ceph-mon[30802]: osdmap e1006: 12 total, 11 up, 12 in 2024-09-16T05:26:58.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:57 smithi196 ceph-mon[30802]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:26:58.247 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:26:58.247 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:26:58.267 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:26:58.267 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:26:58.289 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:26:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:26:58.310 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:26:58.311 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:26:58.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:58.332 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:26:58.332 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:26:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:58.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:58.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:58.356 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:26:58.356 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:26:58.366 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:57 smithi204 ceph-mon[30933]: pgmap v1700: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:26:58.366 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:57 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/499631903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:26:58.367 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:57 smithi204 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T05:26:58.367 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:57 smithi204 ceph-mon[30933]: from='osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:26:58.367 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:57 smithi204 ceph-mon[30933]: osdmap e1006: 12 total, 11 up, 12 in 2024-09-16T05:26:58.367 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:57 smithi204 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-09-16T05:26:58.379 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:26:58.380 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:26:58.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-16T05:26:58.397 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:26:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:26:58.419 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:26:58.420 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:26:58.440 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:26:58.441 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:26:58.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:58.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:58.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:58.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:58.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:26:58.676 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:26:58.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:26:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:58.732 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:58.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:26:58.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:58.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:58.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:58.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:26:58.981 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:58.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:26:58.985 INFO:tasks.rados.rados.2.smithi204.stdout:3491: finishing write tid 1 to smithi20457400-24 2024-09-16T05:26:58.986 INFO:tasks.rados.rados.2.smithi204.stdout:3454: done (10 left) 2024-09-16T05:26:58.986 INFO:tasks.rados.rados.2.smithi204.stdout:3461: done (9 left) 2024-09-16T05:26:58.986 INFO:tasks.rados.rados.2.smithi204.stdout:3476: done (8 left) 2024-09-16T05:26:58.986 INFO:tasks.rados.rados.2.smithi204.stdout:3495: done (7 left) 2024-09-16T05:26:58.986 INFO:tasks.rados.rados.2.smithi204.stdout:3502: done (6 left) 2024-09-16T05:26:58.987 INFO:tasks.rados.rados.2.smithi204.stdout:3503: done (5 left) 2024-09-16T05:26:58.987 INFO:tasks.rados.rados.2.smithi204.stdout:3519: done (4 left) 2024-09-16T05:26:58.987 INFO:tasks.rados.rados.2.smithi204.stdout:3936: done (3 left) 2024-09-16T05:26:58.987 INFO:tasks.rados.rados.2.smithi204.stdout:3937: done (2 left) 2024-09-16T05:26:58.987 INFO:tasks.rados.rados.2.smithi204.stdout:3938: snap_create 2024-09-16T05:26:58.988 INFO:tasks.rados.rados.2.smithi204.stdout:3491: finishing write tid 2 to smithi20457400-24 2024-09-16T05:26:58.988 INFO:tasks.rados.rados.2.smithi204.stdout:3491: finishing write tid 3 to smithi20457400-24 2024-09-16T05:26:58.989 INFO:tasks.rados.rados.2.smithi204.stdout:3491: finishing write tid 4 to smithi20457400-24 2024-09-16T05:26:58.989 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 24 v 829 (ObjNum 1216 snap 369 seq_num 1216) dirty exists 2024-09-16T05:26:58.989 INFO:tasks.rados.rados.2.smithi204.stdout:3491: left oid 24 (ObjNum 1216 snap 369 seq_num 1216) 2024-09-16T05:26:58.989 INFO:tasks.rados.rados.2.smithi204.stdout:3545: finishing write tid 1 to smithi20457400-22 2024-09-16T05:26:58.989 INFO:tasks.rados.rados.2.smithi204.stdout:3545: finishing write tid 2 to smithi20457400-22 2024-09-16T05:26:58.989 INFO:tasks.rados.rados.2.smithi204.stdout:3545: finishing write tid 3 to smithi20457400-22 2024-09-16T05:26:59.001 INFO:tasks.rados.rados.2.smithi204.stdout:3545: finishing write tid 4 to smithi20457400-22 2024-09-16T05:26:59.001 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 22 v 832 (ObjNum 1231 snap 380 seq_num 1231) dirty exists 2024-09-16T05:26:59.001 INFO:tasks.rados.rados.2.smithi204.stdout:3545: left oid 22 (ObjNum 1231 snap 380 seq_num 1231) 2024-09-16T05:26:59.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:59 smithi204 ceph-mon[30933]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T05:26:59.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:59 smithi204 ceph-mon[30933]: osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909] boot 2024-09-16T05:26:59.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:59 smithi204 ceph-mon[30933]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-16T05:26:59.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:59 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:26:59.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:59 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/197443203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:59.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:26:59 smithi204 ceph-mon[30933]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-16T05:26:59.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:59.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:59.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:59.187 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:26:59.187 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:26:59.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:59.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:59.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:59.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:26:59.296 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:59 smithi196 ceph-mon[30802]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T05:26:59.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:59 smithi196 ceph-mon[30802]: osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909] boot 2024-09-16T05:26:59.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:59 smithi196 ceph-mon[30802]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-16T05:26:59.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:59 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:26:59.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:59 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/197443203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:59.297 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:26:59 smithi196 ceph-mon[30802]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-16T05:26:59.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:59.355 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:59.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:59 smithi032 ceph-mon[27388]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-16T05:26:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:59 smithi032 ceph-mon[27388]: osd.7 [v2:172.21.15.196:6824/1346226909,v1:172.21.15.196:6825/1346226909] boot 2024-09-16T05:26:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:59 smithi032 ceph-mon[27388]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-16T05:26:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:59 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T05:26:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:59 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/197443203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:26:59.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:26:59 smithi032 ceph-mon[27388]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-16T05:26:59.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:26:59.483 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:26:59.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:59.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:26:59.522 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:26:59.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:26:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:59.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:59.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:26:59.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:26:59.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:26:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:59.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:26:59.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:26:59.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-16T05:26:59.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:26:59.885 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:26:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:26:59.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:26:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:26:59.973 INFO:tasks.rados.rados.2.smithi204.stdout:3491: done (2 left) 2024-09-16T05:26:59.973 INFO:tasks.rados.rados.2.smithi204.stdout:3545: done (1 left) 2024-09-16T05:26:59.973 INFO:tasks.rados.rados.2.smithi204.stdout:3938: done (0 left) 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3939: rollback oid 29 current snap is 420 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 29 to 418 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3940: read oid 42 snap -1 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3940: expect deleted 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3941: copy_from oid 12 from oid 46 current snap is 420 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3942: copy_from oid 50 from oid 26 current snap is 420 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3943: read oid 11 snap -1 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3943: expect (ObjNum 1362 snap 418 seq_num 1362) 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3944: write (excl) oid 44 current snap is 420 2024-09-16T05:26:59.974 INFO:tasks.rados.rados.2.smithi204.stdout:3944: seq_num 1370 ranges {625590=434670,1685625=681144,2822206=203384} 2024-09-16T05:26:59.979 INFO:tasks.rados.rados.2.smithi204.stdout:3944: writing smithi20457400-44 from 625590 to 1060260 tid 1 2024-09-16T05:26:59.986 INFO:tasks.rados.rados.2.smithi204.stdout:3944: writing smithi20457400-44 from 1685625 to 2366769 tid 2 2024-09-16T05:26:59.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:26:59.990 INFO:tasks.rados.rados.2.smithi204.stdout:3944: writing smithi20457400-44 from 2822206 to 3025590 tid 3 2024-09-16T05:26:59.990 INFO:tasks.rados.rados.2.smithi204.stdout:3945: write oid 21 current snap is 420 2024-09-16T05:26:59.990 INFO:tasks.rados.rados.2.smithi204.stdout:3945: seq_num 1371 ranges {642397=509919,1627754=424802,2733276=309121} 2024-09-16T05:26:59.996 INFO:tasks.rados.rados.2.smithi204.stdout:3945: writing smithi20457400-21 from 642397 to 1152316 tid 1 2024-09-16T05:27:00.001 INFO:tasks.rados.rados.2.smithi204.stdout:3945: writing smithi20457400-21 from 1627754 to 2052556 tid 2 2024-09-16T05:27:00.003 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:00.003 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:00.007 INFO:tasks.rados.rados.2.smithi204.stdout:3945: writing smithi20457400-21 from 2733276 to 3042397 tid 3 2024-09-16T05:27:00.007 INFO:tasks.rados.rados.2.smithi204.stdout:3946: delete oid 1 current snap is 420 2024-09-16T05:27:00.007 INFO:tasks.rados.rados.2.smithi204.stdout:3943: expect (ObjNum 1362 snap 418 seq_num 1362) 2024-09-16T05:27:00.011 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:00.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:00.080 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:00.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:00.124 INFO:tasks.rados.rados.2.smithi204.stdout:3939: finishing rollback tid 0 to smithi20457400-29 2024-09-16T05:27:00.124 INFO:tasks.rados.rados.2.smithi204.stdout:3939: finishing rollback tid 1 to smithi20457400-29 2024-09-16T05:27:00.124 INFO:tasks.rados.rados.2.smithi204.stdout:3940: done (7 left) 2024-09-16T05:27:00.124 INFO:tasks.rados.rados.2.smithi204.stdout:3943: done (6 left) 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3946: done (5 left) 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3947: read oid 33 snap -1 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3947: expect deleted 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3948: read oid 48 snap -1 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3948: expect deleted 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3949: write oid 23 current snap is 420 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3939: finishing rollback tid 2 to smithi20457400-29 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 29 v 1369 (ObjNum 1178 snap 360 seq_num 1178) dirty exists 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3944: finishing write tid 1 to smithi20457400-44 2024-09-16T05:27:00.125 INFO:tasks.rados.rados.2.smithi204.stdout:3949: seq_num 1373 ranges {676011=660417,1744718=428471,2788122=287889} 2024-09-16T05:27:00.131 INFO:tasks.rados.rados.2.smithi204.stdout:3949: writing smithi20457400-23 from 676011 to 1336428 tid 1 2024-09-16T05:27:00.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:00.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:00.136 INFO:tasks.rados.rados.2.smithi204.stdout:3949: writing smithi20457400-23 from 1744718 to 2173189 tid 2 2024-09-16T05:27:00.141 INFO:tasks.rados.rados.2.smithi204.stdout:3949: writing smithi20457400-23 from 2788122 to 3076011 tid 3 2024-09-16T05:27:00.141 INFO:tasks.rados.rados.2.smithi204.stdout:3939: done (7 left) 2024-09-16T05:27:00.141 INFO:tasks.rados.rados.2.smithi204.stdout:3950: snap_remove snap 417 2024-09-16T05:27:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:00 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:00 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:00 smithi204 ceph-mon[30933]: pgmap v1703: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:27:00.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:00 smithi204 ceph-mon[30933]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-16T05:27:00.235 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:00.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:00 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:00.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:00 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:00.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:00 smithi196 ceph-mon[30802]: pgmap v1703: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:27:00.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:00 smithi196 ceph-mon[30802]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-16T05:27:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:00.259 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:00.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:00.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:00 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:00.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:00 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:00.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:00 smithi032 ceph-mon[27388]: pgmap v1703: 127 pgs: 4 active+clean+remapped, 14 undersized+peered, 6 undersized+degraded+peered, 103 active+clean; 240 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:27:00.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:00 smithi032 ceph-mon[27388]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-16T05:27:00.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:00.382 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:00.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:00.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:00.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:00.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:00.586 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:00.587 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:00.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:00.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T05:27:00.628 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:00.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:00.730 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:00.753 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:00.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:00.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:00.868 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:00.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:00.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:00.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:01.001 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:01.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:01.012 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:27:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:3950: done (7 left) 2024-09-16T05:27:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:3951: snap_create 2024-09-16T05:27:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:3942: finishing copy_from racing read to smithi20457400-50 2024-09-16T05:27:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:3941: finishing copy_from racing read to smithi20457400-12 2024-09-16T05:27:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:3941: finishing copy_from to smithi20457400-12 2024-09-16T05:27:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1170 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-09-16T05:27:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:3944: finishing write tid 2 to smithi20457400-44 2024-09-16T05:27:01.017 INFO:tasks.rados.rados.2.smithi204.stdout:3944: finishing write tid 3 to smithi20457400-44 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3944: finishing write tid 4 to smithi20457400-44 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3944: finishing write tid 5 to smithi20457400-44 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1161 (ObjNum 1370 snap 420 seq_num 1370) dirty exists 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3944: left oid 44 (ObjNum 1370 snap 420 seq_num 1370) 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3942: finishing copy_from to smithi20457400-50 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 50 v 1504 (ObjNum 1220 snap 370 seq_num 1220) dirty exists 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3945: finishing write tid 1 to smithi20457400-21 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3945: finishing write tid 2 to smithi20457400-21 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3945: finishing write tid 3 to smithi20457400-21 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3945: finishing write tid 4 to smithi20457400-21 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3945: finishing write tid 5 to smithi20457400-21 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 21 v 1188 (ObjNum 1371 snap 420 seq_num 1371) dirty exists 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3945: left oid 21 (ObjNum 1371 snap 420 seq_num 1371) 2024-09-16T05:27:01.018 INFO:tasks.rados.rados.2.smithi204.stdout:3949: finishing write tid 1 to smithi20457400-23 2024-09-16T05:27:01.019 INFO:tasks.rados.rados.2.smithi204.stdout:3949: finishing write tid 2 to smithi20457400-23 2024-09-16T05:27:01.019 INFO:tasks.rados.rados.2.smithi204.stdout:3949: finishing write tid 3 to smithi20457400-23 2024-09-16T05:27:01.019 INFO:tasks.rados.rados.2.smithi204.stdout:3949: finishing write tid 4 to smithi20457400-23 2024-09-16T05:27:01.019 INFO:tasks.rados.rados.2.smithi204.stdout:3949: finishing write tid 5 to smithi20457400-23 2024-09-16T05:27:01.019 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 23 v 886 (ObjNum 1373 snap 420 seq_num 1373) dirty exists 2024-09-16T05:27:01.019 INFO:tasks.rados.rados.2.smithi204.stdout:3949: left oid 23 (ObjNum 1373 snap 420 seq_num 1373) 2024-09-16T05:27:01.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:01.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:01.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:01.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:27:01.224 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/514726696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:01.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:01 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:01.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:01 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:01.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:01 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:01.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:01.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:01.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-16T05:27:01.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:01.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:01.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/514726696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:01 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:01.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:01 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:01.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:01 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:01.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:01.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/514726696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:01.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:01 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:01 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:01 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:01.493 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:01.498 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:01.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:01.600 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:01.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:01.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:01.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:01.750 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:01.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:01.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:01.877 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:01.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:01.960 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:01.961 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:01.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.002 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:02.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:02.017 INFO:tasks.rados.rados.2.smithi204.stdout:3941: done (7 left) 2024-09-16T05:27:02.017 INFO:tasks.rados.rados.2.smithi204.stdout:3942: done (6 left) 2024-09-16T05:27:02.017 INFO:tasks.rados.rados.2.smithi204.stdout:3944: done (5 left) 2024-09-16T05:27:02.018 INFO:tasks.rados.rados.2.smithi204.stdout:3945: done (4 left) 2024-09-16T05:27:02.018 INFO:tasks.rados.rados.2.smithi204.stdout:3947: done (3 left) 2024-09-16T05:27:02.018 INFO:tasks.rados.rados.2.smithi204.stdout:3948: done (2 left) 2024-09-16T05:27:02.018 INFO:tasks.rados.rados.2.smithi204.stdout:3949: done (1 left) 2024-09-16T05:27:02.018 INFO:tasks.rados.rados.2.smithi204.stdout:3951: done (0 left) 2024-09-16T05:27:02.018 INFO:tasks.rados.rados.2.smithi204.stdout:3952: write (excl) oid 6 current snap is 421 2024-09-16T05:27:02.018 INFO:tasks.rados.rados.2.smithi204.stdout:3952: seq_num 1374 ranges {692818=735666,2203200=688482} 2024-09-16T05:27:02.025 INFO:tasks.rados.rados.2.smithi204.stdout:3952: writing smithi20457400-6 from 692818 to 1428484 tid 1 2024-09-16T05:27:02.033 INFO:tasks.rados.rados.2.smithi204.stdout:3952: writing smithi20457400-6 from 2203200 to 2891682 tid 2 2024-09-16T05:27:02.033 INFO:tasks.rados.rados.2.smithi204.stdout:3953: rollback oid 44 current snap is 421 2024-09-16T05:27:02.033 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 44 to 412 2024-09-16T05:27:02.034 INFO:tasks.rados.rados.2.smithi204.stdout:3952: finishing write tid 1 to smithi20457400-6 2024-09-16T05:27:02.034 INFO:tasks.rados.rados.2.smithi204.stdout:3954: rollback oid 43 current snap is 421 2024-09-16T05:27:02.034 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 43 to 415 2024-09-16T05:27:02.034 INFO:tasks.rados.rados.2.smithi204.stdout:3955: snap_remove snap 416 2024-09-16T05:27:02.104 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.127 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:02.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:02.153 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:02 smithi032 ceph-mon[27388]: pgmap v1706: 127 pgs: 4 active+clean+remapped, 13 undersized+peered, 6 undersized+degraded+peered, 104 active+clean; 240 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:27:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:02 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/514726696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:02 smithi032 ceph-mon[27388]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-16T05:27:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:02 smithi204 ceph-mon[30933]: pgmap v1706: 127 pgs: 4 active+clean+remapped, 13 undersized+peered, 6 undersized+degraded+peered, 104 active+clean; 240 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:27:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:02 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/514726696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:02.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:02 smithi204 ceph-mon[30933]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-16T05:27:02.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.250 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:02.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:02.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:02.373 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:02.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:02.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:02 smithi196 ceph-mon[30802]: pgmap v1706: 127 pgs: 4 active+clean+remapped, 13 undersized+peered, 6 undersized+degraded+peered, 104 active+clean; 240 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16/222 objects degraded (7.207%); 6/222 objects misplaced (2.703%) 2024-09-16T05:27:02.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:02 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/514726696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:02 smithi196 ceph-mon[30802]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-16T05:27:02.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:02.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:02.626 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:02.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-16T05:27:02.716 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:27:02.717 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:27:02.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:02.737 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:27:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:27:02.751 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:02.759 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:27:02.759 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:27:02.780 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:27:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:27:02.801 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:27:02.801 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:27:02.823 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:27:02.823 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:27:02.845 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:27:02.845 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:27:02.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:02.867 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:27:02.867 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:27:02.875 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:02.888 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:27:02.888 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:27:02.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:27:02.909 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:27:02.913 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:02.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:27:02.931 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:27:02.952 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:27:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:27:02.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:02.998 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:02.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:03.023 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3955: done (3 left) 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3956: delete oid 37 current snap is 421 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3952: finishing write tid 2 to smithi20457400-6 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3952: finishing write tid 3 to smithi20457400-6 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3954: finishing rollback tid 1 to smithi20457400-43 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 43 v 1484 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3952: finishing write tid 4 to smithi20457400-6 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 1252 (ObjNum 1374 snap 421 seq_num 1374) dirty exists 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3952: left oid 6 (ObjNum 1374 snap 421 seq_num 1374) 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3953: finishing rollback tid 0 to smithi20457400-44 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3953: finishing rollback tid 1 to smithi20457400-44 2024-09-16T05:27:03.028 INFO:tasks.rados.rados.2.smithi204.stdout:3953: finishing rollback tid 2 to smithi20457400-44 2024-09-16T05:27:03.029 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1164 (ObjNum 1345 snap 411 seq_num 1345) dirty exists 2024-09-16T05:27:03.032 INFO:tasks.rados.rados.2.smithi204.stdout:3952: done (3 left) 2024-09-16T05:27:03.032 INFO:tasks.rados.rados.2.smithi204.stdout:3953: done (2 left) 2024-09-16T05:27:03.032 INFO:tasks.rados.rados.2.smithi204.stdout:3954: done (1 left) 2024-09-16T05:27:03.032 INFO:tasks.rados.rados.2.smithi204.stdout:3956: done (0 left) 2024-09-16T05:27:03.032 INFO:tasks.rados.rados.2.smithi204.stdout:3957: snap_remove snap 412 2024-09-16T05:27:03.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:03.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:27:03.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 2.1e scrub starts 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 2.1e scrub ok 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3820846336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 2.b scrub starts 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 2.b scrub ok 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 2.3c scrub starts 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4068470144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 2.3c scrub ok 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 2.4f scrub starts 2024-09-16T05:27:03.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:03 smithi032 ceph-mon[27388]: 2.4f scrub ok 2024-09-16T05:27:03.122 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:03.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:03.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:03.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 2.1e scrub starts 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 2.1e scrub ok 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3820846336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 2.b scrub starts 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 2.b scrub ok 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 2.3c scrub starts 2024-09-16T05:27:03.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4068470144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:03.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 2.3c scrub ok 2024-09-16T05:27:03.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 2.4f scrub starts 2024-09-16T05:27:03.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:03 smithi204 ceph-mon[30933]: 2.4f scrub ok 2024-09-16T05:27:03.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:03.362 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:27:03.362 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-16T05:27:03.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-16T05:27:03.362 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.8.0:1100 2.f e588/435 2:f08ec412:::smithi20457400-44:e0 v 588'362, mlcod=585'359)", 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 485.58731401799997, 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.771307407, 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:27:03.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773468+0000", 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773469+0000", 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0100000000000004e-07 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:57.773478+0000", 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1139999999999995e-06 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544068+0000", 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.770589827 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544083+0000", 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5213e-05 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544231+0000", 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014854000000000001 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544252+0000", 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0568e-05 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544741+0000", 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048860799999999997 2024-09-16T05:27:03.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:18:58.544775+0000", 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4836000000000003e-05 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.24571.0:5976 2.8 e617/320 2:1030fb5b:::smithi20457400-27:head v 617'305, mlcod=602'299)", 2024-09-16T05:27:03.367 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:19:26.797538+0000", 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 456.56324371599999, 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.75484651300000005, 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:26.797538+0000", 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:26.797538+0000", 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:26.797539+0000", 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0580000000000001e-06 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:26.797545+0000", 2024-09-16T05:27:03.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9309999999999996e-06 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.551777+0000", 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.75423199100000005 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.551785+0000", 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2200000000000003e-06 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.551874+0000", 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8968000000000006e-05 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.551893+0000", 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.95e-05 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.552352+0000", 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045916300000000001 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:19:27.552385+0000", 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2682000000000003e-05 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:7574 2.2f 2:f6b419a1:::smithi20457400-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e762)", 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:21:52.279172+0000", 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 311.08161017700002, 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.22072831600000001, 2024-09-16T05:27:03.372 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7574 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.279172+0000", 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.279172+0000", 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.279172+0000", 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4299999999999999e-07 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.279174+0000", 2024-09-16T05:27:03.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.533e-06 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.279178+0000", 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4399999999999998e-06 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.279186+0000", 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8890000000000002e-06 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280457+0000", 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001270455 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.280524+0000", 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7539000000000005e-05 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.499768+0000", 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.21924368899999999 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.499833+0000", 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5345000000000005e-05 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:21:52.499900+0000", 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7082999999999995e-05 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.377 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8679 2.2f 2:f4b938a4:::smithi20457400-47:head [setxattr _header (58) in=7b,truncate 2117979] snapc 171=[171,170,16f,16e,16d,16a] ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:35.252078+0000", 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 208.108703744, 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45294416300000001, 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8679 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252078+0000", 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252078+0000", 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252079+0000", 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5199999999999997e-07 2024-09-16T05:27:03.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252082+0000", 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9610000000000001e-06 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252088+0000", 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4130000000000003e-06 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252093+0000", 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1050000000000003e-06 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252862+0000", 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00076839900000000001 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252970+0000", 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010796 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.253148+0000", 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017859399999999999 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.253799+0000", 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00065068400000000001 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.704898+0000", 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45109872400000001 2024-09-16T05:27:03.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.704941+0000", 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3711000000000002e-05 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.705022+0000", 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.106e-05 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.383 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:8680 2.2f 2:f4b938a4:::smithi20457400-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e865)", 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:35.252102+0000", 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 208.10868034699999, 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45310305400000001, 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 8680 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.384 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252102+0000", 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252102+0000", 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252102+0000", 2024-09-16T05:27:03.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4200000000000001e-07 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252104+0000", 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0370000000000001e-06 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252111+0000", 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9410000000000001e-06 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.252116+0000", 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5789999999999997e-06 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.253523+0000", 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001407284 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.253610+0000", 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.666e-05 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.705052+0000", 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45144189000000001 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.705134+0000", 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2157000000000006e-05 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.705205+0000", 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1063999999999996e-05 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.388 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:8679 2.2f e865/275 ondisk, result = 0)", 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:23:35.253680+0000", 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 208.10710224499999, 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.45135062799999998, 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.253680+0000", 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.253680+0000", 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.253680+0000", 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0999999999999998e-07 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.253685+0000", 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0980000000000004e-06 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.704838+0000", 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.451152989 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.704844+0000", 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4069999999999998e-06 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.704885+0000", 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0837000000000002e-05 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.704893+0000", 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1529999999999997e-06 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:23:35.705030+0000", 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000137534 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9366 2.2f 2:f6b419a1:::smithi20457400-37:head [setxattr _header (58) in=7b,truncate 3613802] snapc 193=[193,191,18c,17e,16e,16a] ondisk+write+known_if_redirected+supports_pool_eio e933)", 2024-09-16T05:27:03.394 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:43.332318+0000", 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 140.02846449699999, 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.66554730200000001, 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9366 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332318+0000", 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332318+0000", 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332318+0000", 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7099999999999997e-07 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332320+0000", 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.531e-06 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332325+0000", 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0420000000000002e-06 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332329+0000", 2024-09-16T05:27:03.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0240000000000002e-06 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.333713+0000", 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001384195 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.333785+0000", 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2041999999999999e-05 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:27:03.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.333883+0000", 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8568999999999998e-05 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.334423+0000", 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000539793 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.997764+0000", 2024-09-16T05:27:03.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.66334049699999997 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.997811+0000", 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7416999999999997e-05 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.997865+0000", 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3820999999999999e-05 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.400 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9367 2.2f 2:f6b419a1:::smithi20457400-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e933)", 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:24:43.332355+0000", 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 140.02842701700001, 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.66568662999999995, 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9367 2024-09-16T05:27:03.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332355+0000", 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332355+0000", 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332355+0000", 2024-09-16T05:27:03.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.03e-07 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332357+0000", 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7060000000000001e-06 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332362+0000", 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6700000000000002e-06 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.332365+0000", 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3220000000000001e-06 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.334327+0000", 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019620879999999999 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.334408+0000", 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0675999999999998e-05 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.997900+0000", 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.66349266200000001 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.997967+0000", 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6635000000000002e-05 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:24:43.998042+0000", 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4567999999999995e-05 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.405 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(2.39,GRANT,e:974,from:5,reservation_nonce:0)", 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:25:48.202405+0000", 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 75.158377454999993, 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00049190900000000001, 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202405+0000", 2024-09-16T05:27:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202405+0000", 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202407+0000", 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9190000000000002e-06 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202416+0000", 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8430000000000005e-06 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202455+0000", 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9079999999999999e-05 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202470+0000", 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4513e-05 2024-09-16T05:27:03.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202615+0000", 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014496499999999999 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202654+0000", 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9053e-05 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:25:48.202897+0000", 2024-09-16T05:27:03.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024253600000000001 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(2.15,GRANT,e:992,from:1,reservation_nonce:0)", 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:26:28.294693+0000", 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.066088952999998, 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033510799999999998, 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.410 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.294693+0000", 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.294693+0000", 2024-09-16T05:27:03.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.294694+0000", 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.294697+0000", 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.241e-06 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.294752+0000", 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4421e-05 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.294761+0000", 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3729999999999999e-06 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.294822+0000", 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0406999999999999e-05 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.294847+0000", 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5038e-05 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.295028+0000", 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018153200000000001 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.414 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "rep_scrubmap(2.15 e988 from shard 1)", 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:26:28.296076+0000", 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.064705791999998, 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011620699999999999, 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296076+0000", 2024-09-16T05:27:03.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296076+0000", 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296077+0000", 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2299999999999999e-07 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296083+0000", 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5060000000000003e-06 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296117+0000", 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3924000000000002e-05 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296122+0000", 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8669999999999999e-06 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296157+0000", 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4542999999999998e-05 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296165+0000", 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9300000000000003e-06 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:28.296192+0000", 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7514000000000001e-05 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.418 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "MOSDScrubReserve(2.d,GRANT,e:1003,from:4,reservation_nonce:0)", 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:26:50.453238+0000", 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 12.907543970000001, 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027582300000000002, 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453238+0000", 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453238+0000", 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453240+0000", 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4389999999999999e-06 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453249+0000", 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8300000000000008e-06 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453284+0000", 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4718000000000002e-05 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453293+0000", 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9099999999999994e-06 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453341+0000", 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8046e-05 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453359+0000", 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7890999999999999e-05 2024-09-16T05:27:03.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.453514+0000", 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015498899999999999 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.423 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "rep_scrubmap(2.d e435 from shard 4)", 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:26:50.454456+0000", 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 12.906326091, 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017378000000000001, 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454456+0000", 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454456+0000", 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454457+0000", 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9500000000000001e-07 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454465+0000", 2024-09-16T05:27:03.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6430000000000001e-06 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454496+0000", 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0704999999999998e-05 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454520+0000", 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3731e-05 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454577+0000", 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6894000000000001e-05 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454592+0000", 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5534000000000001e-05 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:26:50.454630+0000", 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7478000000000003e-05 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.427 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.24571.0:9968 2.2f 2:f6b419a1:::smithi20457400-37:head [delete] snapc 1a8=[1a8,1a7,1a6,1a3,1a2,1a0,16e,16a] ondisk+write+known_if_redirected+full_force+supports_pool_eio e1012)", 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:27:03.028106+0000", 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.33267582699999998, 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004263518, 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.24571", 2024-09-16T05:27:03.428 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/662043117", 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 9968 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.028106+0000", 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.028106+0000", 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.028108+0000", 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6190000000000001e-06 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.028114+0000", 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4250000000000003e-06 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.028127+0000", 2024-09-16T05:27:03.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2785999999999999e-05 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.028134+0000", 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2180000000000002e-06 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.028196+0000", 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1804000000000007e-05 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-16T05:27:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.028205+0000", 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4309999999999994e-06 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.029863+0000", 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016583870000000001 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.029964+0000", 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101166 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 11", 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.030174+0000", 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000209581 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.031989+0000", 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0018152769999999999 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032219+0000", 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00022984700000000001 2024-09-16T05:27:03.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032277+0000", 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8727000000000002e-05 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032370+0000", 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2250000000000006e-05 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.9.0:1637 2.48 e1012/322 ondisk, result = 0)", 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:27:03.031416+0000", 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.32936563699999999, 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00074312300000000005, 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.031416+0000", 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.031416+0000", 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.031417+0000", 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9800000000000003e-07 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.031421+0000", 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0899999999999998e-06 2024-09-16T05:27:03.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.031436+0000", 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4562000000000001e-05 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.031456+0000", 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.022e-05 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032046+0000", 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00058945099999999995 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032052+0000", 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4119999999999998e-06 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032160+0000", 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010759 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.438 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(client.24571.0:9968 2.2f e1011/275 ondisk, result = 0)", 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:27:03.032057+0000", 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.32872468599999999, 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031806599999999999, 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032057+0000", 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032057+0000", 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032058+0000", 2024-09-16T05:27:03.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1699999999999998e-07 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032062+0000", 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2610000000000004e-06 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032102+0000", 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0241000000000003e-05 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032109+0000", 2024-09-16T05:27:03.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5509999999999996e-06 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032204+0000", 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5379999999999998e-05 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032213+0000", 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4859999999999997e-06 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.032375+0000", 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016253 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop_reply(osd.9.0:1639 2.2f e1012/275 ondisk, result = 0)", 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-16T05:27:03.035112+0000", 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.32566967499999999, 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000155511, 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-16T05:27:03.443 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035112+0000", 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035112+0000", 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035113+0000", 2024-09-16T05:27:03.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0299999999999999e-07 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035116+0000", 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4039999999999999e-06 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035140+0000", 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3862e-05 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-16T05:27:03.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035146+0000", 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5199999999999997e-06 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035179+0000", 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3513000000000003e-05 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035185+0000", 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5230000000000003e-06 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-16T05:27:03.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-16T05:27:03.035268+0000", 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3085999999999997e-05 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:27:03.447 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:03.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 2.1e scrub starts 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 2.1e scrub ok 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3820846336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 2.b scrub starts 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 2.b scrub ok 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 2.3c scrub starts 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4068470144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 2.3c scrub ok 2024-09-16T05:27:03.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 2.4f scrub starts 2024-09-16T05:27:03.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:03 smithi196 ceph-mon[30802]: 2.4f scrub ok 2024-09-16T05:27:03.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:03.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:03.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:03.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:03.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:03.698 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:03.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:03.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T05:27:03.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:03.828 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:03.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:03.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:27:03.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:03.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:03.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:04.032 INFO:tasks.rados.rados.2.smithi204.stdout:3957: done (0 left) 2024-09-16T05:27:04.032 INFO:tasks.rados.rados.2.smithi204.stdout:3958: write oid 17 current snap is 421 2024-09-16T05:27:04.032 INFO:tasks.rados.rados.2.smithi204.stdout:3958: seq_num 1376 ranges {726432=486164,1803811=692151} 2024-09-16T05:27:04.038 INFO:tasks.rados.rados.2.smithi204.stdout:3958: writing smithi20457400-17 from 726432 to 1212596 tid 1 2024-09-16T05:27:04.046 INFO:tasks.rados.rados.2.smithi204.stdout:3958: writing smithi20457400-17 from 1803811 to 2495962 tid 2 2024-09-16T05:27:04.046 INFO:tasks.rados.rados.2.smithi204.stdout:3959: snap_create 2024-09-16T05:27:04.047 INFO:tasks.rados.rados.2.smithi204.stdout:3958: finishing write tid 1 to smithi20457400-17 2024-09-16T05:27:04.050 INFO:tasks.rados.rados.2.smithi204.stdout:3958: finishing write tid 2 to smithi20457400-17 2024-09-16T05:27:04.050 INFO:tasks.rados.rados.2.smithi204.stdout:3958: finishing write tid 3 to smithi20457400-17 2024-09-16T05:27:04.051 INFO:tasks.rados.rados.2.smithi204.stdout:3958: finishing write tid 4 to smithi20457400-17 2024-09-16T05:27:04.051 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1491 (ObjNum 1376 snap 421 seq_num 1376) dirty exists 2024-09-16T05:27:04.051 INFO:tasks.rados.rados.2.smithi204.stdout:3958: left oid 17 (ObjNum 1376 snap 421 seq_num 1376) 2024-09-16T05:27:04.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:04.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:04.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:04.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:04.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:04.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:04.318 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:04.337 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:04.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:04.357 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:04.371 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:04 smithi204 ceph-mon[30933]: pgmap v1709: 127 pgs: 2 active+clean+remapped, 2 undersized+peered, 123 active+clean; 244 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s wr, 4 op/s; 4/228 objects misplaced (1.754%); 500 KiB/s, 0 objects/s recovering 2024-09-16T05:27:04.371 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:04 smithi204 ceph-mon[30933]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:27:04.371 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:04 smithi204 ceph-mon[30933]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-16T05:27:04.371 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:04 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3820846336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:27:04.371 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:04 smithi204 ceph-mon[30933]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-16T05:27:04.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-16T05:27:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:04 smithi032 ceph-mon[27388]: pgmap v1709: 127 pgs: 2 active+clean+remapped, 2 undersized+peered, 123 active+clean; 244 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s wr, 4 op/s; 4/228 objects misplaced (1.754%); 500 KiB/s, 0 objects/s recovering 2024-09-16T05:27:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:04 smithi032 ceph-mon[27388]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:27:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:04 smithi032 ceph-mon[27388]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-16T05:27:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:04 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3820846336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:27:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:04 smithi032 ceph-mon[27388]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-16T05:27:04.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:04.460 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:04.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:04.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:04 smithi196 ceph-mon[30802]: pgmap v1709: 127 pgs: 2 active+clean+remapped, 2 undersized+peered, 123 active+clean; 244 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s wr, 4 op/s; 4/228 objects misplaced (1.754%); 500 KiB/s, 0 objects/s recovering 2024-09-16T05:27:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:04 smithi196 ceph-mon[30802]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-16T05:27:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:04 smithi196 ceph-mon[30802]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-16T05:27:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:04 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3820846336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:27:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:04 smithi196 ceph-mon[30802]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-16T05:27:04.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:04.586 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:04.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:04.639 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:04.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:04.710 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:04.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:04.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:04.836 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:04.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:04.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:04.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:04.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:05.036 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:27:05.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.18 scrub starts 2024-09-16T05:27:05.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.18 scrub ok 2024-09-16T05:27:05.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-16T05:27:05.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.5a scrub starts 2024-09-16T05:27:05.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.5a scrub ok 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.1d scrub starts 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.1d scrub ok 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.46 scrub starts 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.46 scrub ok 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.f scrub starts 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.f scrub ok 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.33 scrub starts 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.33 scrub ok 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.36 scrub starts 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: 2.36 scrub ok 2024-09-16T05:27:05.037 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:05 smithi032 ceph-mon[27388]: from='client.17994 172.21.15.32:0/2124447766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:05.039 INFO:tasks.rados.rados.2.smithi204.stdout:3958: done (1 left) 2024-09-16T05:27:05.040 INFO:tasks.rados.rados.2.smithi204.stdout:3959: done (0 left) 2024-09-16T05:27:05.040 INFO:tasks.rados.rados.2.smithi204.stdout:3960: write (excl) oid 49 current snap is 422 2024-09-16T05:27:05.040 INFO:tasks.rados.rados.2.smithi204.stdout:3960: seq_num 1377 ranges {743239=677766,1978646=435809,2897814=245425} 2024-09-16T05:27:05.047 INFO:tasks.rados.rados.2.smithi204.stdout:3960: writing smithi20457400-49 from 743239 to 1421005 tid 1 2024-09-16T05:27:05.053 INFO:tasks.rados.rados.2.smithi204.stdout:3960: writing smithi20457400-49 from 1978646 to 2414455 tid 2 2024-09-16T05:27:05.057 INFO:tasks.rados.rados.2.smithi204.stdout:3960: writing smithi20457400-49 from 2897814 to 3143239 tid 3 2024-09-16T05:27:05.057 INFO:tasks.rados.rados.2.smithi204.stdout:3961: write oid 40 current snap is 422 2024-09-16T05:27:05.057 INFO:tasks.rados.rados.2.smithi204.stdout:3961: seq_num 1378 ranges {760046=753015,2037128=695820} 2024-09-16T05:27:05.065 INFO:tasks.rados.rados.2.smithi204.stdout:3961: writing smithi20457400-40 from 760046 to 1513061 tid 1 2024-09-16T05:27:05.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:05.072 INFO:tasks.rados.rados.2.smithi204.stdout:3961: writing smithi20457400-40 from 2037128 to 2732948 tid 2 2024-09-16T05:27:05.072 INFO:tasks.rados.rados.2.smithi204.stdout:3962: snap_remove snap 421 2024-09-16T05:27:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:05.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:05.093 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:27:05.093 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:27:05.093 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-16T05:27:05.093 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:27:05.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:05.214 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:05.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:05.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:05.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:05.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:05.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.18 scrub starts 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.18 scrub ok 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.5a scrub starts 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.5a scrub ok 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.1d scrub starts 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.1d scrub ok 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.46 scrub starts 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.46 scrub ok 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.f scrub starts 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.f scrub ok 2024-09-16T05:27:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.33 scrub starts 2024-09-16T05:27:05.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.33 scrub ok 2024-09-16T05:27:05.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.36 scrub starts 2024-09-16T05:27:05.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: 2.36 scrub ok 2024-09-16T05:27:05.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:05 smithi204 ceph-mon[30933]: from='client.17994 172.21.15.32:0/2124447766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:05.444 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:05.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:05.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.18 scrub starts 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.18 scrub ok 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.5a scrub starts 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.5a scrub ok 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.1d scrub starts 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.1d scrub ok 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.46 scrub starts 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.46 scrub ok 2024-09-16T05:27:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.f scrub starts 2024-09-16T05:27:05.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.f scrub ok 2024-09-16T05:27:05.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.33 scrub starts 2024-09-16T05:27:05.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.33 scrub ok 2024-09-16T05:27:05.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.36 scrub starts 2024-09-16T05:27:05.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: 2.36 scrub ok 2024-09-16T05:27:05.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:05 smithi196 ceph-mon[30802]: from='client.17994 172.21.15.32:0/2124447766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:05.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:05.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:05.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:05.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:05.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:05.722 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:05.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-16T05:27:05.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:05.844 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:05.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:05.898 INFO:tasks.rados.rados.2.smithi204.stdout:3962: done (2 left) 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3963: write oid 1 current snap is 422 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3960: finishing write tid 1 to smithi20457400-49 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3960: finishing write tid 2 to smithi20457400-49 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3960: finishing write tid 3 to smithi20457400-49 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3960: finishing write tid 4 to smithi20457400-49 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3960: finishing write tid 5 to smithi20457400-49 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 49 v 687 (ObjNum 1377 snap 422 seq_num 1377) dirty exists 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3960: left oid 49 (ObjNum 1377 snap 422 seq_num 1377) 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3961: finishing write tid 1 to smithi20457400-40 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3961: finishing write tid 2 to smithi20457400-40 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3961: finishing write tid 3 to smithi20457400-40 2024-09-16T05:27:05.899 INFO:tasks.rados.rados.2.smithi204.stdout:3961: finishing write tid 4 to smithi20457400-40 2024-09-16T05:27:05.900 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 40 v 1077 (ObjNum 1378 snap 422 seq_num 1378) dirty exists 2024-09-16T05:27:05.900 INFO:tasks.rados.rados.2.smithi204.stdout:3961: left oid 40 (ObjNum 1378 snap 422 seq_num 1378) 2024-09-16T05:27:05.900 INFO:tasks.rados.rados.2.smithi204.stdout:3963: seq_num 1379 ranges {776853=428264,1979257=439478,2836307=340546} 2024-09-16T05:27:05.906 INFO:tasks.rados.rados.2.smithi204.stdout:3963: writing smithi20457400-1 from 776853 to 1205117 tid 1 2024-09-16T05:27:05.913 INFO:tasks.rados.rados.2.smithi204.stdout:3963: writing smithi20457400-1 from 1979257 to 2418735 tid 2 2024-09-16T05:27:05.917 INFO:tasks.rados.rados.2.smithi204.stdout:3963: writing smithi20457400-1 from 2836307 to 3176853 tid 3 2024-09-16T05:27:05.918 INFO:tasks.rados.rados.2.smithi204.stdout:3960: done (2 left) 2024-09-16T05:27:05.918 INFO:tasks.rados.rados.2.smithi204.stdout:3961: done (1 left) 2024-09-16T05:27:05.918 INFO:tasks.rados.rados.2.smithi204.stdout:3964: copy_from oid 14 from oid 18 current snap is 422 2024-09-16T05:27:05.918 INFO:tasks.rados.rados.2.smithi204.stdout:3963: finishing write tid 1 to smithi20457400-1 2024-09-16T05:27:05.918 INFO:tasks.rados.rados.2.smithi204.stdout:3963: finishing write tid 2 to smithi20457400-1 2024-09-16T05:27:05.918 INFO:tasks.rados.rados.2.smithi204.stdout:3965: copy_from oid 39 from oid 24 current snap is 422 2024-09-16T05:27:05.918 INFO:tasks.rados.rados.2.smithi204.stdout:3966: delete oid 21 current snap is 422 2024-09-16T05:27:05.920 INFO:tasks.rados.rados.2.smithi204.stdout:3963: finishing write tid 3 to smithi20457400-1 2024-09-16T05:27:05.920 INFO:tasks.rados.rados.2.smithi204.stdout:3963: finishing write tid 4 to smithi20457400-1 2024-09-16T05:27:05.920 INFO:tasks.rados.rados.2.smithi204.stdout:3963: finishing write tid 5 to smithi20457400-1 2024-09-16T05:27:05.920 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 1 v 1156 (ObjNum 1379 snap 422 seq_num 1379) dirty exists 2024-09-16T05:27:05.920 INFO:tasks.rados.rados.2.smithi204.stdout:3963: left oid 1 (ObjNum 1379 snap 422 seq_num 1379) 2024-09-16T05:27:05.921 INFO:tasks.rados.rados.2.smithi204.stdout:3963: done (3 left) 2024-09-16T05:27:05.922 INFO:tasks.rados.rados.2.smithi204.stdout:3966: done (2 left) 2024-09-16T05:27:05.922 INFO:tasks.rados.rados.2.smithi204.stdout:3967: write (excl) oid 17 current snap is 422 2024-09-16T05:27:05.922 INFO:tasks.rados.rados.2.smithi204.stdout:3967: seq_num 1381 ranges {410467=578762,1696221=443147,2891153=319314} 2024-09-16T05:27:05.927 INFO:tasks.rados.rados.2.smithi204.stdout:3967: writing smithi20457400-17 from 410467 to 989229 tid 1 2024-09-16T05:27:05.933 INFO:tasks.rados.rados.2.smithi204.stdout:3967: writing smithi20457400-17 from 1696221 to 2139368 tid 2 2024-09-16T05:27:05.940 INFO:tasks.rados.rados.2.smithi204.stdout:3967: writing smithi20457400-17 from 2891153 to 3210467 tid 3 2024-09-16T05:27:05.940 INFO:tasks.rados.rados.2.smithi204.stdout:3968: read oid 46 snap 414 2024-09-16T05:27:05.940 INFO:tasks.rados.rados.2.smithi204.stdout:3968: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:27:05.940 INFO:tasks.rados.rados.2.smithi204.stdout:3969: write oid 44 current snap is 422 2024-09-16T05:27:05.940 INFO:tasks.rados.rados.2.smithi204.stdout:3969: seq_num 1382 ranges {427274=654011,1754703=586805,2918576=308698} 2024-09-16T05:27:05.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:05.946 INFO:tasks.rados.rados.2.smithi204.stdout:3969: writing smithi20457400-44 from 427274 to 1081285 tid 1 2024-09-16T05:27:05.956 INFO:tasks.rados.rados.2.smithi204.stdout:3969: writing smithi20457400-44 from 1754703 to 2341508 tid 2 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3969: writing smithi20457400-44 from 2918576 to 3227274 tid 3 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3970: copy_from oid 20 from oid 25 current snap is 422 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3964: finishing copy_from racing read to smithi20457400-14 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3971: rollback oid 10 current snap is 422 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 10 to 414 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3964: finishing copy_from to smithi20457400-14 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 14 v 1487 (ObjNum 1325 snap 403 seq_num 1325) dirty exists 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3967: finishing write tid 1 to smithi20457400-17 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3967: finishing write tid 2 to smithi20457400-17 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3965: finishing copy_from racing read to smithi20457400-39 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3967: finishing write tid 3 to smithi20457400-17 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:3965: finishing copy_from to smithi20457400-39 2024-09-16T05:27:05.959 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 39 v 829 (ObjNum 1216 snap 369 seq_num 1216) dirty exists 2024-09-16T05:27:05.960 INFO:tasks.rados.rados.2.smithi204.stdout:3967: finishing write tid 4 to smithi20457400-17 2024-09-16T05:27:05.960 INFO:tasks.rados.rados.2.smithi204.stdout:3967: finishing write tid 5 to smithi20457400-17 2024-09-16T05:27:05.960 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 17 v 1495 (ObjNum 1381 snap 422 seq_num 1381) dirty exists 2024-09-16T05:27:05.960 INFO:tasks.rados.rados.2.smithi204.stdout:3967: left oid 17 (ObjNum 1381 snap 422 seq_num 1381) 2024-09-16T05:27:05.960 INFO:tasks.rados.rados.2.smithi204.stdout:3969: finishing write tid 1 to smithi20457400-44 2024-09-16T05:27:05.960 INFO:tasks.rados.rados.2.smithi204.stdout:3968: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-09-16T05:27:05.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:05.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:06.032 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:06.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:06.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:06.130 INFO:tasks.rados.rados.2.smithi204.stdout:3969: finishing write tid 2 to smithi20457400-44 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3969: finishing write tid 3 to smithi20457400-44 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3969: finishing write tid 4 to smithi20457400-44 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3969: finishing write tid 5 to smithi20457400-44 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 44 v 1168 (ObjNum 1382 snap 422 seq_num 1382) dirty exists 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3969: left oid 44 (ObjNum 1382 snap 422 seq_num 1382) 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3971: finishing rollback tid 0 to smithi20457400-10 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3971: finishing rollback tid 1 to smithi20457400-10 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3971: finishing rollback tid 2 to smithi20457400-10 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 10 v 1517 (ObjNum 1176 snap 360 seq_num 1176) dirty exists 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3970: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3970: finishing copy_from to smithi20457400-20 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 20 v 1291 (ObjNum 1355 snap 415 seq_num 1355) dirty exists 2024-09-16T05:27:06.131 INFO:tasks.rados.rados.2.smithi204.stdout:3964: done (6 left) 2024-09-16T05:27:06.132 INFO:tasks.rados.rados.2.smithi204.stdout:3965: done (5 left) 2024-09-16T05:27:06.132 INFO:tasks.rados.rados.2.smithi204.stdout:3967: done (4 left) 2024-09-16T05:27:06.132 INFO:tasks.rados.rados.2.smithi204.stdout:3968: done (3 left) 2024-09-16T05:27:06.132 INFO:tasks.rados.rados.2.smithi204.stdout:3969: done (2 left) 2024-09-16T05:27:06.132 INFO:tasks.rados.rados.2.smithi204.stdout:3970: done (1 left) 2024-09-16T05:27:06.132 INFO:tasks.rados.rados.2.smithi204.stdout:3971: done (0 left) 2024-09-16T05:27:06.132 INFO:tasks.rados.rados.2.smithi204.stdout:3972: write oid 25 current snap is 422 2024-09-16T05:27:06.132 INFO:tasks.rados.rados.2.smithi204.stdout:3972: seq_num 1383 ranges {444081=729260,1813185=446816,2945999=298082} 2024-09-16T05:27:06.136 INFO:tasks.rados.rados.2.smithi204.stdout:3972: writing smithi20457400-25 from 444081 to 1173341 tid 1 2024-09-16T05:27:06.142 INFO:tasks.rados.rados.2.smithi204.stdout:3972: writing smithi20457400-25 from 1813185 to 2260001 tid 2 2024-09-16T05:27:06.148 INFO:tasks.rados.rados.2.smithi204.stdout:3972: writing smithi20457400-25 from 2945999 to 3244081 tid 3 2024-09-16T05:27:06.148 INFO:tasks.rados.rados.2.smithi204.stdout:3973: rollback oid 45 current snap is 422 2024-09-16T05:27:06.148 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 45 to 362 2024-09-16T05:27:06.148 INFO:tasks.rados.rados.2.smithi204.stdout:3972: finishing write tid 1 to smithi20457400-25 2024-09-16T05:27:06.148 INFO:tasks.rados.rados.2.smithi204.stdout:3972: finishing write tid 2 to smithi20457400-25 2024-09-16T05:27:06.148 INFO:tasks.rados.rados.2.smithi204.stdout:3974: rollback oid 13 current snap is 422 2024-09-16T05:27:06.148 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 13 to 418 2024-09-16T05:27:06.148 INFO:tasks.rados.rados.2.smithi204.stdout:3975: write oid 28 current snap is 422 2024-09-16T05:27:06.149 INFO:tasks.rados.rados.2.smithi204.stdout:3975: seq_num 1384 ranges {460888=520862,1471667=590474,2573422=455407} 2024-09-16T05:27:06.153 INFO:tasks.rados.rados.2.smithi204.stdout:3975: writing smithi20457400-28 from 460888 to 981750 tid 1 2024-09-16T05:27:06.159 INFO:tasks.rados.rados.2.smithi204.stdout:3975: writing smithi20457400-28 from 1471667 to 2062141 tid 2 2024-09-16T05:27:06.164 INFO:tasks.rados.rados.2.smithi204.stdout:3975: writing smithi20457400-28 from 2573422 to 3028829 tid 3 2024-09-16T05:27:06.164 INFO:tasks.rados.rados.2.smithi204.stdout:3976: snap_remove snap 415 2024-09-16T05:27:06.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:06.219 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:06.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:06.276 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:27:06.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: pgmap v1712: 127 pgs: 127 active+clean; 255 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 2.4 MiB/s wr, 9 op/s; 2.4 MiB/s, 7 objects/s recovering 2024-09-16T05:27:06.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: 2.1b scrub starts 2024-09-16T05:27:06.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: 2.1b scrub ok 2024-09-16T05:27:06.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:06.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:06.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-09-16T05:27:06.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-16T05:27:06.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:27:06.319 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: from='client.17994 172.21.15.32:0/2124447766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:06.319 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-16T05:27:06.319 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: 1.1a scrub starts 2024-09-16T05:27:06.319 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: 1.1a scrub ok 2024-09-16T05:27:06.319 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: 2.5d scrub starts 2024-09-16T05:27:06.319 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: 2.5d scrub ok 2024-09-16T05:27:06.319 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-16T05:27:06.319 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3869266862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:27:06.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: pgmap v1712: 127 pgs: 127 active+clean; 255 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 2.4 MiB/s wr, 9 op/s; 2.4 MiB/s, 7 objects/s recovering 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: 2.1b scrub starts 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: 2.1b scrub ok 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: from='client.17994 172.21.15.32:0/2124447766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: 1.1a scrub starts 2024-09-16T05:27:06.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: 1.1a scrub ok 2024-09-16T05:27:06.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: 2.5d scrub starts 2024-09-16T05:27:06.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: 2.5d scrub ok 2024-09-16T05:27:06.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-16T05:27:06.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3869266862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:27:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:06.344 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:06.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:06.445 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:06.467 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:06.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: pgmap v1712: 127 pgs: 127 active+clean; 255 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 2.4 MiB/s wr, 9 op/s; 2.4 MiB/s, 7 objects/s recovering 2024-09-16T05:27:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: 2.1b scrub starts 2024-09-16T05:27:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: 2.1b scrub ok 2024-09-16T05:27:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-09-16T05:27:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-16T05:27:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:27:06.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: from='client.17994 172.21.15.32:0/2124447766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:06.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-16T05:27:06.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: 1.1a scrub starts 2024-09-16T05:27:06.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: 1.1a scrub ok 2024-09-16T05:27:06.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: 2.5d scrub starts 2024-09-16T05:27:06.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: 2.5d scrub ok 2024-09-16T05:27:06.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-16T05:27:06.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3869266862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:27:06.529 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T05:27:06.530 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-16T05:27:06.530 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-16T05:27:06.530 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-16T05:27:06.530 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T05:27:06.549 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:06.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:06.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:06.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:06.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:06.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:06.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:06.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:06.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:06.896 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:3976: done (4 left) 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:3977: snap_create 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:3974: finishing rollback tid 0 to smithi20457400-13 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:3972: finishing write tid 3 to smithi20457400-25 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:3974: finishing rollback tid 1 to smithi20457400-13 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:3972: finishing write tid 4 to smithi20457400-25 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:3974: finishing rollback tid 2 to smithi20457400-13 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 13 v 1156 (ObjNum 1226 snap 373 seq_num 1226) dirty exists 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:3972: finishing write tid 5 to smithi20457400-25 2024-09-16T05:27:06.900 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 25 v 1295 (ObjNum 1383 snap 422 seq_num 1383) dirty exists 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3972: left oid 25 (ObjNum 1383 snap 422 seq_num 1383) 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3973: finishing rollback tid 0 to smithi20457400-45 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3973: finishing rollback tid 1 to smithi20457400-45 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3973: finishing rollback tid 2 to smithi20457400-45 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 45 v 1288 (ObjNum 1139 snap 341 seq_num 1139) dirty exists 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3975: finishing write tid 1 to smithi20457400-28 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3975: finishing write tid 2 to smithi20457400-28 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3975: finishing write tid 3 to smithi20457400-28 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3975: finishing write tid 4 to smithi20457400-28 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3975: finishing write tid 5 to smithi20457400-28 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 28 v 1499 (ObjNum 1384 snap 422 seq_num 1384) dirty exists 2024-09-16T05:27:06.901 INFO:tasks.rados.rados.2.smithi204.stdout:3975: left oid 28 (ObjNum 1384 snap 422 seq_num 1384) 2024-09-16T05:27:06.948 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:06.969 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:06.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:07.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:07.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:07.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:07.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:07 smithi204 ceph-mon[30933]: 2.1c scrub starts 2024-09-16T05:27:07.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:07 smithi204 ceph-mon[30933]: 2.1c scrub ok 2024-09-16T05:27:07.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:07 smithi204 ceph-mon[30933]: 1.1d scrub starts 2024-09-16T05:27:07.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3869266862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:27:07.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:07 smithi204 ceph-mon[30933]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-16T05:27:07.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4067912394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:07.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-16T05:27:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:07 smithi032 ceph-mon[27388]: 2.1c scrub starts 2024-09-16T05:27:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:07 smithi032 ceph-mon[27388]: 2.1c scrub ok 2024-09-16T05:27:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:07 smithi032 ceph-mon[27388]: 1.1d scrub starts 2024-09-16T05:27:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3869266862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:27:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:07 smithi032 ceph-mon[27388]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-16T05:27:07.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4067912394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:07.198 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:07.221 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:07.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:07.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:07.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:07.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:07.368 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:27:07.368 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:27:07.368 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:27:07.368 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:27:07.368 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:27:07.369 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:27:07.369 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:27:07.369 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:27:07.369 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:27:07.369 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:27:07.369 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:27:07.369 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:27:07.369 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:27:07.370 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:27:07.371 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:27:07.371 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:27:07.401 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:07.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:07.472 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:07.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:07.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:07 smithi196 ceph-mon[30802]: 2.1c scrub starts 2024-09-16T05:27:07.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:07 smithi196 ceph-mon[30802]: 2.1c scrub ok 2024-09-16T05:27:07.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:07 smithi196 ceph-mon[30802]: 1.1d scrub starts 2024-09-16T05:27:07.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3869266862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:27:07.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:07 smithi196 ceph-mon[30802]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-16T05:27:07.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4067912394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:07.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:07.597 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:07.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:07.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:07.722 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:07.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:07.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:07.848 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:07.848 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:07.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:07.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:07.901 INFO:tasks.rados.rados.2.smithi204.stdout:3972: done (4 left) 2024-09-16T05:27:07.901 INFO:tasks.rados.rados.2.smithi204.stdout:3973: done (3 left) 2024-09-16T05:27:07.901 INFO:tasks.rados.rados.2.smithi204.stdout:3974: done (2 left) 2024-09-16T05:27:07.901 INFO:tasks.rados.rados.2.smithi204.stdout:3975: done (1 left) 2024-09-16T05:27:07.901 INFO:tasks.rados.rados.2.smithi204.stdout:3977: done (0 left) 2024-09-16T05:27:07.901 INFO:tasks.rados.rados.2.smithi204.stdout:3978: delete oid 24 current snap is 423 2024-09-16T05:27:07.905 INFO:tasks.rados.rados.2.smithi204.stdout:3978: done (0 left) 2024-09-16T05:27:07.905 INFO:tasks.rados.rados.2.smithi204.stdout:3979: read oid 23 snap -1 2024-09-16T05:27:07.905 INFO:tasks.rados.rados.2.smithi204.stdout:3979: expect (ObjNum 1373 snap 420 seq_num 1373) 2024-09-16T05:27:07.905 INFO:tasks.rados.rados.2.smithi204.stdout:3980: read oid 13 snap 362 2024-09-16T05:27:07.906 INFO:tasks.rados.rados.2.smithi204.stdout:3980: expect (ObjNum 1131 snap 338 seq_num 1131) 2024-09-16T05:27:07.906 INFO:tasks.rados.rados.2.smithi204.stdout:3981: write oid 26 current snap is 423 2024-09-16T05:27:07.906 INFO:tasks.rados.rados.2.smithi204.stdout:3981: seq_num 1386 ranges {494502=671360,1588631=594143,2628268=477951} 2024-09-16T05:27:07.912 INFO:tasks.rados.rados.2.smithi204.stdout:3981: writing smithi20457400-26 from 494502 to 1165862 tid 1 2024-09-16T05:27:07.918 INFO:tasks.rados.rados.2.smithi204.stdout:3981: writing smithi20457400-26 from 1588631 to 2182774 tid 2 2024-09-16T05:27:07.923 INFO:tasks.rados.rados.2.smithi204.stdout:3981: writing smithi20457400-26 from 2628268 to 3106219 tid 3 2024-09-16T05:27:07.923 INFO:tasks.rados.rados.2.smithi204.stdout:3982: write (excl) oid 11 current snap is 423 2024-09-16T05:27:07.923 INFO:tasks.rados.rados.2.smithi204.stdout:3982: seq_num 1387 ranges {511309=746609,2047113=454154,3055691=255618} 2024-09-16T05:27:07.930 INFO:tasks.rados.rados.2.smithi204.stdout:3982: writing smithi20457400-11 from 511309 to 1257918 tid 1 2024-09-16T05:27:07.937 INFO:tasks.rados.rados.2.smithi204.stdout:3982: writing smithi20457400-11 from 2047113 to 2501267 tid 2 2024-09-16T05:27:07.942 INFO:tasks.rados.rados.2.smithi204.stdout:3982: writing smithi20457400-11 from 3055691 to 3311309 tid 3 2024-09-16T05:27:07.942 INFO:tasks.rados.rados.2.smithi204.stdout:3983: snap_remove snap 358 2024-09-16T05:27:07.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:07.971 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:07.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:08.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:08.094 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:08.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:08.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:08 smithi204 ceph-mon[30933]: pgmap v1715: 127 pgs: 127 active+clean; 252 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 8 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-16T05:27:08.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:08 smithi204 ceph-mon[30933]: 1.1d scrub ok 2024-09-16T05:27:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:08 smithi204 ceph-mon[30933]: 2.5b scrub starts 2024-09-16T05:27:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:08 smithi204 ceph-mon[30933]: 2.5b scrub ok 2024-09-16T05:27:08.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:08 smithi204 ceph-mon[30933]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-16T05:27:08.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:08.220 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:08.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:08.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:08.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-16T05:27:08.322 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:08.344 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:08.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:08.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:08 smithi032 ceph-mon[27388]: pgmap v1715: 127 pgs: 127 active+clean; 252 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 8 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-16T05:27:08.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:08 smithi032 ceph-mon[27388]: 1.1d scrub ok 2024-09-16T05:27:08.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:08 smithi032 ceph-mon[27388]: 2.5b scrub starts 2024-09-16T05:27:08.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:08 smithi032 ceph-mon[27388]: 2.5b scrub ok 2024-09-16T05:27:08.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:08 smithi032 ceph-mon[27388]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-16T05:27:08.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:08.468 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:08.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:08.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:08 smithi196 ceph-mon[30802]: pgmap v1715: 127 pgs: 127 active+clean; 252 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 8 op/s; 1.9 MiB/s, 7 objects/s recovering 2024-09-16T05:27:08.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:08 smithi196 ceph-mon[30802]: 1.1d scrub ok 2024-09-16T05:27:08.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:08 smithi196 ceph-mon[30802]: 2.5b scrub starts 2024-09-16T05:27:08.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:08 smithi196 ceph-mon[30802]: 2.5b scrub ok 2024-09-16T05:27:08.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:08 smithi196 ceph-mon[30802]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-16T05:27:08.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-16T05:27:08.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:08.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T05:27:08.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:08.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:08.624 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:08.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:08.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:08.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:08.779 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:08.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:08.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:08.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:08.903 INFO:tasks.rados.rados.2.smithi204.stdout:3983: done (4 left) 2024-09-16T05:27:08.903 INFO:tasks.rados.rados.2.smithi204.stdout:3984: read oid 31 snap -1 2024-09-16T05:27:08.903 INFO:tasks.rados.rados.2.smithi204.stdout:3984: expect deleted 2024-09-16T05:27:08.903 INFO:tasks.rados.rados.2.smithi204.stdout:3985: read oid 15 snap 419 2024-09-16T05:27:08.903 INFO:tasks.rados.rados.2.smithi204.stdout:3985: expect deleted 2024-09-16T05:27:08.904 INFO:tasks.rados.rados.2.smithi204.stdout:3986: copy_from oid 20 from oid 8 current snap is 423 2024-09-16T05:27:08.904 INFO:tasks.rados.rados.2.smithi204.stdout:3987: snap_create 2024-09-16T05:27:08.904 INFO:tasks.rados.rados.2.smithi204.stdout:3979: expect (ObjNum 1373 snap 420 seq_num 1373) 2024-09-16T05:27:08.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:08.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:08.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:09.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:09.093 INFO:tasks.rados.rados.2.smithi204.stdout:3980: expect (ObjNum 1131 snap 338 seq_num 1131) 2024-09-16T05:27:09.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:09.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:09.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:09 smithi204 ceph-mon[30933]: 2.42 deep-scrub starts 2024-09-16T05:27:09.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:09 smithi204 ceph-mon[30933]: 2.42 deep-scrub ok 2024-09-16T05:27:09.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:09 smithi204 ceph-mon[30933]: 2.45 scrub starts 2024-09-16T05:27:09.155 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:09 smithi204 ceph-mon[30933]: 2.45 scrub ok 2024-09-16T05:27:09.155 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:09 smithi204 ceph-mon[30933]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-16T05:27:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:09 smithi032 ceph-mon[27388]: 2.42 deep-scrub starts 2024-09-16T05:27:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:09 smithi032 ceph-mon[27388]: 2.42 deep-scrub ok 2024-09-16T05:27:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:09 smithi032 ceph-mon[27388]: 2.45 scrub starts 2024-09-16T05:27:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:09 smithi032 ceph-mon[27388]: 2.45 scrub ok 2024-09-16T05:27:09.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:09 smithi032 ceph-mon[27388]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-16T05:27:09.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:09.223 INFO:tasks.rados.rados.2.smithi204.stdout:3981: finishing write tid 1 to smithi20457400-26 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3981: finishing write tid 2 to smithi20457400-26 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3981: finishing write tid 3 to smithi20457400-26 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3981: finishing write tid 4 to smithi20457400-26 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3981: finishing write tid 5 to smithi20457400-26 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 26 v 1514 (ObjNum 1386 snap 423 seq_num 1386) dirty exists 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3981: left oid 26 (ObjNum 1386 snap 423 seq_num 1386) 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3982: finishing write tid 1 to smithi20457400-11 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3982: finishing write tid 2 to smithi20457400-11 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3982: finishing write tid 3 to smithi20457400-11 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3982: finishing write tid 4 to smithi20457400-11 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3982: finishing write tid 5 to smithi20457400-11 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 11 v 1457 (ObjNum 1387 snap 423 seq_num 1387) dirty exists 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3982: left oid 11 (ObjNum 1387 snap 423 seq_num 1387) 2024-09-16T05:27:09.224 INFO:tasks.rados.rados.2.smithi204.stdout:3986: finishing copy_from to smithi20457400-20 2024-09-16T05:27:09.225 INFO:tasks.rados.rados.2.smithi204.stdout:3986: got expected ENOENT (src dne) 2024-09-16T05:27:09.225 INFO:tasks.rados.rados.2.smithi204.stdout:3986: finishing copy_from racing read to smithi20457400-20 2024-09-16T05:27:09.229 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:09.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:09.271 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:09.271 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:09.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:09.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:09.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:09.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:09.484 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:09.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:09.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:09 smithi196 ceph-mon[30802]: 2.42 deep-scrub starts 2024-09-16T05:27:09.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:09 smithi196 ceph-mon[30802]: 2.42 deep-scrub ok 2024-09-16T05:27:09.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:09 smithi196 ceph-mon[30802]: 2.45 scrub starts 2024-09-16T05:27:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:09 smithi196 ceph-mon[30802]: 2.45 scrub ok 2024-09-16T05:27:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:09 smithi196 ceph-mon[30802]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-16T05:27:09.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:09.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:09.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:09.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:09.735 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:09.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:09.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:09.858 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:09.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:09.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-16T05:27:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T05:27:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is set 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3979: done (7 left) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3980: done (6 left) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3981: done (5 left) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3982: done (4 left) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3984: done (3 left) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3985: done (2 left) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3986: done (1 left) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3987: done (0 left) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3988: read oid 40 snap -1 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3988: expect (ObjNum 1378 snap 422 seq_num 1378) 2024-09-16T05:27:09.906 INFO:tasks.rados.rados.2.smithi204.stdout:3989: snap_create 2024-09-16T05:27:09.910 INFO:tasks.rados.rados.2.smithi204.stdout:3988: expect (ObjNum 1378 snap 422 seq_num 1378) 2024-09-16T05:27:09.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:09.985 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:09.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:10.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:27:10.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:10.112 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:10.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:10.153 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: pgmap v1718: 127 pgs: 127 active+clean; 262 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:27:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: 2.30 scrub starts 2024-09-16T05:27:10.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: 2.30 scrub ok 2024-09-16T05:27:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/42992831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:27:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2527853507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:27:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/42992831' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:27:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2527853507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:27:10.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:10 smithi204 ceph-mon[30933]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-16T05:27:10.213 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:10.236 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:10.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:10.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:10.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:10.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: pgmap v1718: 127 pgs: 127 active+clean; 262 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:27:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: 2.30 scrub starts 2024-09-16T05:27:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: 2.30 scrub ok 2024-09-16T05:27:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/42992831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:27:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2527853507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:27:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/42992831' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:27:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2527853507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:27:10.390 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:10 smithi196 ceph-mon[30802]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-16T05:27:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: pgmap v1718: 127 pgs: 127 active+clean; 262 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 2.0 MiB/s wr, 6 op/s 2024-09-16T05:27:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: 2.30 scrub starts 2024-09-16T05:27:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: 2.30 scrub ok 2024-09-16T05:27:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/42992831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-16T05:27:10.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2527853507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:27:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/42992831' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-16T05:27:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2527853507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:27:10.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:10 smithi032 ceph-mon[27388]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-16T05:27:10.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:10.491 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:10.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:10.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:10.614 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:10.614 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:10.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:10.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:10.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:10.743 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:10.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:10.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:10.869 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:10.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:10.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T05:27:10.899 INFO:tasks.rados.rados.2.smithi204.stdout:3988: done (1 left) 2024-09-16T05:27:10.899 INFO:tasks.rados.rados.2.smithi204.stdout:3989: done (0 left) 2024-09-16T05:27:10.899 INFO:tasks.rados.rados.2.smithi204.stdout:3990: write (excl) oid 6 current snap is 425 2024-09-16T05:27:10.899 INFO:tasks.rados.rados.2.smithi204.stdout:3990: seq_num 1388 ranges {528116=421858,1589242=597812,2966761=361355} 2024-09-16T05:27:10.904 INFO:tasks.rados.rados.2.smithi204.stdout:3990: writing smithi20457400-6 from 528116 to 949974 tid 1 2024-09-16T05:27:10.911 INFO:tasks.rados.rados.2.smithi204.stdout:3990: writing smithi20457400-6 from 1589242 to 2187054 tid 2 2024-09-16T05:27:10.918 INFO:tasks.rados.rados.2.smithi204.stdout:3990: writing smithi20457400-6 from 2966761 to 3328116 tid 3 2024-09-16T05:27:10.918 INFO:tasks.rados.rados.2.smithi204.stdout:3991: rollback oid 12 current snap is 425 2024-09-16T05:27:10.918 INFO:tasks.rados.rados.2.smithi204.stdout:rollback oid 12 to 362 2024-09-16T05:27:10.918 INFO:tasks.rados.rados.2.smithi204.stdout:3990: finishing write tid 1 to smithi20457400-6 2024-09-16T05:27:10.918 INFO:tasks.rados.rados.2.smithi204.stdout:3990: finishing write tid 2 to smithi20457400-6 2024-09-16T05:27:10.919 INFO:tasks.rados.rados.2.smithi204.stdout:3992: snap_create 2024-09-16T05:27:10.920 INFO:tasks.rados.rados.2.smithi204.stdout:3990: finishing write tid 3 to smithi20457400-6 2024-09-16T05:27:10.920 INFO:tasks.rados.rados.2.smithi204.stdout:3990: finishing write tid 4 to smithi20457400-6 2024-09-16T05:27:10.920 INFO:tasks.rados.rados.2.smithi204.stdout:3990: finishing write tid 5 to smithi20457400-6 2024-09-16T05:27:10.920 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 6 v 1256 (ObjNum 1388 snap 425 seq_num 1388) dirty exists 2024-09-16T05:27:10.921 INFO:tasks.rados.rados.2.smithi204.stdout:3990: left oid 6 (ObjNum 1388 snap 425 seq_num 1388) 2024-09-16T05:27:10.921 INFO:tasks.rados.rados.2.smithi204.stdout:3991: finishing rollback tid 0 to smithi20457400-12 2024-09-16T05:27:10.921 INFO:tasks.rados.rados.2.smithi204.stdout:3991: finishing rollback tid 1 to smithi20457400-12 2024-09-16T05:27:10.921 INFO:tasks.rados.rados.2.smithi204.stdout:3991: finishing rollback tid 2 to smithi20457400-12 2024-09-16T05:27:10.921 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 12 v 1393 (ObjNum 1145 snap 342 seq_num 1145) dirty exists 2024-09-16T05:27:10.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:10.994 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:10.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:11.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:11.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:11.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:11.201 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-16T05:27:11.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:11.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:11.304 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:11.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:11.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:11.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:11.445 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T05:27:11.467 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:11.488 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:11.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:11.590 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:11.611 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:11.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:11.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:11.735 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:11.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:11.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:11.857 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:27:11.857 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:27:11.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:11.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:11.863 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:11.880 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:27:11.880 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:27:11.900 INFO:tasks.rados.rados.2.smithi204.stdout:3990: done (2 left) 2024-09-16T05:27:11.900 INFO:tasks.rados.rados.2.smithi204.stdout:3991: done (1 left) 2024-09-16T05:27:11.901 INFO:tasks.rados.rados.2.smithi204.stdout:3992: done (0 left) 2024-09-16T05:27:11.901 INFO:tasks.rados.rados.2.smithi204.stdout:3993: delete oid 33 current snap is 426 2024-09-16T05:27:11.907 INFO:tasks.rados.rados.2.smithi204.stdout:3993: done (0 left) 2024-09-16T05:27:11.907 INFO:tasks.rados.rados.2.smithi204.stdout:3994: snap_remove snap 414 2024-09-16T05:27:11.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:27:11.909 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:27:11.929 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:27:11.929 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:27:11.936 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:11.937 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:11.951 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:27:11.951 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:27:11.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:11.972 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:27:11.972 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:27:11.982 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:11.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:11.994 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:27:11.994 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:27:12.015 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:27:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:27:12.036 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:27:12.036 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:27:12.058 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:27:12.058 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:27:12.079 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:27:12.080 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:27:12.083 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:12.100 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:27:12.101 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:27:12.104 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:12.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:12.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:11 smithi032 ceph-mon[27388]: pgmap v1721: 127 pgs: 127 active+clean; 264 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 4.9 MiB/s wr, 13 op/s 2024-09-16T05:27:12.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:11 smithi032 ceph-mon[27388]: 2.1f scrub starts 2024-09-16T05:27:12.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:11 smithi032 ceph-mon[27388]: 2.1f scrub ok 2024-09-16T05:27:12.110 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:11 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:27:12.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:11 smithi032 ceph-mon[27388]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-16T05:27:12.111 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2512827513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:12.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:11 smithi204 ceph-mon[30933]: pgmap v1721: 127 pgs: 127 active+clean; 264 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 4.9 MiB/s wr, 13 op/s 2024-09-16T05:27:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:11 smithi204 ceph-mon[30933]: 2.1f scrub starts 2024-09-16T05:27:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:11 smithi204 ceph-mon[30933]: 2.1f scrub ok 2024-09-16T05:27:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:11 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:27:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:11 smithi204 ceph-mon[30933]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-16T05:27:12.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2512827513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:12.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:12.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:12.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:11 smithi196 ceph-mon[30802]: pgmap v1721: 127 pgs: 127 active+clean; 264 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 4.9 MiB/s wr, 13 op/s 2024-09-16T05:27:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:11 smithi196 ceph-mon[30802]: 2.1f scrub starts 2024-09-16T05:27:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:11 smithi196 ceph-mon[30802]: 2.1f scrub ok 2024-09-16T05:27:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:11 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:27:12.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:11 smithi196 ceph-mon[30802]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-16T05:27:12.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2512827513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:12.333 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:12.357 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:12.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:12.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:12.483 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:12.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:12.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-16T05:27:12.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:12.606 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:12.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:12.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:12.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:12.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:12.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:12.855 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:12.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:12.912 INFO:tasks.rados.rados.2.smithi204.stdout:3994: done (0 left) 2024-09-16T05:27:12.912 INFO:tasks.rados.rados.2.smithi204.stdout:3995: snap_create 2024-09-16T05:27:12.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:12.978 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:12.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:13.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:13.102 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:13.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:13.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:12 smithi032 ceph-mon[27388]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-16T05:27:13.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:12 smithi204 ceph-mon[30933]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-16T05:27:13.204 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:13.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:13.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:13.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:12 smithi196 ceph-mon[30802]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-16T05:27:13.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:13.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-16T05:27:13.349 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:13.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:13.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:13.475 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:13.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:13.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:13.598 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:13.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:13.694 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:27:13.694 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:27:13.694 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:27:13.694 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:27:13.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:13.721 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:13.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:13.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:13.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:13.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:13.909 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:13.910 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T05:27:13.913 INFO:tasks.rados.rados.2.smithi204.stdout:3995: done (0 left) 2024-09-16T05:27:13.913 INFO:tasks.rados.rados.2.smithi204.stdout:3996: snap_create 2024-09-16T05:27:13.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:13.973 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:13.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:14.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:14.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:14.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:14.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:13 smithi032 ceph-mon[27388]: pgmap v1724: 127 pgs: 127 active+clean; 260 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-16T05:27:14.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:13 smithi032 ceph-mon[27388]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-16T05:27:14.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3017970331' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:27:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:13 smithi204 ceph-mon[30933]: pgmap v1724: 127 pgs: 127 active+clean; 260 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-16T05:27:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:13 smithi204 ceph-mon[30933]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-16T05:27:14.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3017970331' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:27:14.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:14.226 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:14.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:14.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:14.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:13 smithi196 ceph-mon[30802]: pgmap v1724: 127 pgs: 127 active+clean; 260 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-16T05:27:14.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:13 smithi196 ceph-mon[30802]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-16T05:27:14.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3017970331' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:27:14.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:14.351 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:14.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:14.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:14.476 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:14.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:14.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:14.599 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:14.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:14.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:14.723 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:14.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:14.825 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:14.848 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:14.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:14.911 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is set 2024-09-16T05:27:14.915 INFO:tasks.rados.rados.2.smithi204.stdout:3996: done (0 left) 2024-09-16T05:27:14.915 INFO:tasks.rados.rados.2.smithi204.stdout:3997: write oid 48 current snap is 428 2024-09-16T05:27:14.915 INFO:tasks.rados.rados.2.smithi204.stdout:3997: seq_num 1390 ranges {561730=572356,1706206=601481,3021607=340123} 2024-09-16T05:27:14.921 INFO:tasks.rados.rados.2.smithi204.stdout:3997: writing smithi20457400-48 from 561730 to 1134086 tid 1 2024-09-16T05:27:14.927 INFO:tasks.rados.rados.2.smithi204.stdout:3997: writing smithi20457400-48 from 1706206 to 2307687 tid 2 2024-09-16T05:27:14.934 INFO:tasks.rados.rados.2.smithi204.stdout:3997: writing smithi20457400-48 from 3021607 to 3361730 tid 3 2024-09-16T05:27:14.934 INFO:tasks.rados.rados.2.smithi204.stdout:3998: read oid 44 snap -1 2024-09-16T05:27:14.934 INFO:tasks.rados.rados.2.smithi204.stdout:3998: expect (ObjNum 1382 snap 422 seq_num 1382) 2024-09-16T05:27:14.934 INFO:tasks.rados.rados.2.smithi204.stdout:3999: read oid 15 snap -1 2024-09-16T05:27:14.934 INFO:tasks.rados.rados.2.smithi204.stdout:3999: expect deleted 2024-09-16T05:27:14.934 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 3 2024-09-16T05:27:14.934 INFO:tasks.rados.rados.2.smithi204.stdout:3997: finishing write tid 1 to smithi20457400-48 2024-09-16T05:27:14.934 INFO:tasks.rados.rados.2.smithi204.stdout:3997: finishing write tid 2 to smithi20457400-48 2024-09-16T05:27:14.935 INFO:tasks.rados.rados.2.smithi204.stdout:3999: done (2 left) 2024-09-16T05:27:14.936 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 2 2024-09-16T05:27:14.936 INFO:tasks.rados.rados.2.smithi204.stdout:3997: finishing write tid 3 to smithi20457400-48 2024-09-16T05:27:14.936 INFO:tasks.rados.rados.2.smithi204.stdout:3997: finishing write tid 4 to smithi20457400-48 2024-09-16T05:27:14.936 INFO:tasks.rados.rados.2.smithi204.stdout:3997: finishing write tid 5 to smithi20457400-48 2024-09-16T05:27:14.936 INFO:tasks.rados.rados.2.smithi204.stdout:update_object_version oid 48 v 806 (ObjNum 1390 snap 428 seq_num 1390) dirty exists 2024-09-16T05:27:14.937 INFO:tasks.rados.rados.2.smithi204.stdout:3997: left oid 48 (ObjNum 1390 snap 428 seq_num 1390) 2024-09-16T05:27:14.937 INFO:tasks.rados.rados.2.smithi204.stdout:3997: done (1 left) 2024-09-16T05:27:14.937 INFO:tasks.rados.rados.2.smithi204.stdout: waiting on 1 2024-09-16T05:27:14.937 INFO:tasks.rados.rados.2.smithi204.stdout:3998: expect (ObjNum 1382 snap 422 seq_num 1382) 2024-09-16T05:27:14.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:14.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:27:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:14.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:14.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:15.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-16T05:27:15.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:15.080 INFO:tasks.rados.rados.2.smithi204.stdout:3998: done (0 left) 2024-09-16T05:27:15.081 INFO:tasks.rados.rados.2.smithi204.stderr:0 errors. 2024-09-16T05:27:15.081 INFO:tasks.rados.rados.2.smithi204.stderr: 2024-09-16T05:27:15.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T05:27:15.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:15.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3017970331' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:27:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:14 smithi032 ceph-mon[27388]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-16T05:27:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2202666011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:14 smithi032 ceph-mon[27388]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:15.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2202666011' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3017970331' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:27:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:14 smithi204 ceph-mon[30933]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-16T05:27:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2202666011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:14 smithi204 ceph-mon[30933]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:15.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2202666011' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:15.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:15.233 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:15.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:15.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3017970331' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:27:15.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:14 smithi196 ceph-mon[30802]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-16T05:27:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2202666011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:14 smithi196 ceph-mon[30802]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:15.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2202666011' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:15.283 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:15.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:15.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:15.359 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:15.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:15.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:15.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:15.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:15.607 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:15.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:15.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:15.732 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:15.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:15.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:15.856 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:15.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:15.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:15.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:15.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:16.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:16.109 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:16.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:15 smithi032 ceph-mon[27388]: pgmap v1727: 127 pgs: 127 active+clean; 253 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 633 KiB/s wr, 12 op/s 2024-09-16T05:27:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:15 smithi032 ceph-mon[27388]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-16T05:27:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:15 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:16.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:15 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:16.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:15 smithi032 ceph-mon[27388]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-16T05:27:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:15 smithi204 ceph-mon[30933]: pgmap v1727: 127 pgs: 127 active+clean; 253 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 633 KiB/s wr, 12 op/s 2024-09-16T05:27:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:15 smithi204 ceph-mon[30933]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-16T05:27:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:15 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:15 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:16.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:15 smithi204 ceph-mon[30933]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-16T05:27:16.212 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:16.235 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:16.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:16.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:15 smithi196 ceph-mon[30802]: pgmap v1727: 127 pgs: 127 active+clean; 253 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 633 KiB/s wr, 12 op/s 2024-09-16T05:27:16.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:15 smithi196 ceph-mon[30802]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-16T05:27:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:15 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:15 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:16.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:15 smithi196 ceph-mon[30802]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-16T05:27:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:16.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T05:27:16.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:16.360 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:16.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:16.462 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:16.485 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:16.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:16.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:16.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:16.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:16.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:16.739 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:16.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:16.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:16.857 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:27:16.857 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:27:16.857 INFO:teuthology.orchestra.run.smithi032.stdout: "complaint_time": 30, 2024-09-16T05:27:16.857 INFO:teuthology.orchestra.run.smithi032.stdout: "num_blocked_ops": 0 2024-09-16T05:27:16.857 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:27:16.864 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:16.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:16.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:16.988 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:16.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:17.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:17.108 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:16 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:17.108 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:16 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:17.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:17.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:17.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:17.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:16 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:17.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:16 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:17.241 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:17.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:17.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:17.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:17.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:16 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:16 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:17.467 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:17.493 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:17.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:17.572 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:17.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:17.621 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:17.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:17.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:17.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:17.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:27:17.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:17.873 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:17.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:17.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:18.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:18.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:18.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-16T05:27:18.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:18.106 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:27:18.106 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1025,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","created":"2024-09-16T05:02:35.583426+0000","modified":"2024-09-16T05:27:15.896152+0000","last_up_change":"2024-09-16T05:26:57.965143+0000","last_in_change":"2024-09-16T05:21:53.914532+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":41,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:05:16.445728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"687","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"685","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.7799999713897705,"score_stable":1.7799999713897705,"optimal_score":1.0900000333786011,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.84360116720199585,"average_primary_affinity":0.92000001668930054,"average_primary_affinity_weighted":0.8399999737739563}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2024-09-16T05:10:35.539280+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":96,"pg_placement_num":96,"pg_placement_num_target":96,"pg_num_target":96,"pg_num_pending":96,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1024","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"737","auid":0,"snap_mode":"selfmanaged","snap_seq":431,"snap_epoch":1024,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1.0900000333786011,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":0.84360116720199585,"average_primary_affinity":0.92000001668930054,"average_primary_affinity_weighted":0.8399999737739563}}],"osds":[{"osd":0,"uuid":"f3cfb49b-8223-4649-b211-a2e254248d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6803","nonce":406159384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6805","nonce":406159384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6809","nonce":406159384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6807","nonce":406159384}]},"public_addr":"172.21.15.32:6803/406159384","cluster_addr":"172.21.15.32:6805/406159384","heartbeat_back_addr":"172.21.15.32:6809/406159384","heartbeat_front_addr":"172.21.15.32:6807/406159384","state":["exists","up"]},{"osd":1,"uuid":"cfb43840-55b6-4d39-a325-2c5416ed97b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":866,"up_from":988,"up_thru":1007,"down_at":867,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6811","nonce":2976159074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6813","nonce":2976159074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6817","nonce":2976159074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6815","nonce":2976159074}]},"public_addr":"172.21.15.32:6811/2976159074","cluster_addr":"172.21.15.32:6813/2976159074","heartbeat_back_addr":"172.21.15.32:6817/2976159074","heartbeat_front_addr":"172.21.15.32:6815/2976159074","state":["exists","up"]},{"osd":2,"uuid":"3158b0cc-849c-4383-8d4d-585cf0723e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":295,"up_from":329,"up_thru":1008,"down_at":296,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6819","nonce":3973436581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6821","nonce":3973436581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6825","nonce":3973436581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6823","nonce":3973436581}]},"public_addr":"172.21.15.32:6819/3973436581","cluster_addr":"172.21.15.32:6821/3973436581","heartbeat_back_addr":"172.21.15.32:6825/3973436581","heartbeat_front_addr":"172.21.15.32:6823/3973436581","state":["exists","up"]},{"osd":3,"uuid":"71705a06-9102-4b97-be6a-4989306666d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":994,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6827","nonce":3040098358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6829","nonce":3040098358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6833","nonce":3040098358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6831","nonce":3040098358}]},"public_addr":"172.21.15.32:6827/3040098358","cluster_addr":"172.21.15.32:6829/3040098358","heartbeat_back_addr":"172.21.15.32:6833/3040098358","heartbeat_front_addr":"172.21.15.32:6831/3040098358","state":["exists","up"]},{"osd":4,"uuid":"ca8f47c4-c13f-4bde-b162-0e3b57bad1b4","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6801","nonce":3219469655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6803","nonce":3219469655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6807","nonce":3219469655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6805","nonce":3219469655}]},"public_addr":"172.21.15.196:6801/3219469655","cluster_addr":"172.21.15.196:6803/3219469655","heartbeat_back_addr":"172.21.15.196:6807/3219469655","heartbeat_front_addr":"172.21.15.196:6805/3219469655","state":["exists","up"]},{"osd":5,"uuid":"b5b76271-0539-42dd-a9e7-a95afded2d59","up":1,"in":1,"weight":0.2819671630859375,"primary_affinity":0.8769683837890625,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":996,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6809","nonce":3274041128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6811","nonce":3274041128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6815","nonce":3274041128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6813","nonce":3274041128}]},"public_addr":"172.21.15.196:6809/3274041128","cluster_addr":"172.21.15.196:6811/3274041128","heartbeat_back_addr":"172.21.15.196:6815/3274041128","heartbeat_front_addr":"172.21.15.196:6813/3274041128","state":["exists","up"]},{"osd":6,"uuid":"d3f2312b-cc04-48e4-993a-f08a948ba3ca","up":1,"in":1,"weight":1,"primary_affinity":0.2543182373046875,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6817","nonce":2509099861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6819","nonce":2509099861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6823","nonce":2509099861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6821","nonce":2509099861}]},"public_addr":"172.21.15.196:6817/2509099861","cluster_addr":"172.21.15.196:6819/2509099861","heartbeat_back_addr":"172.21.15.196:6823/2509099861","heartbeat_front_addr":"172.21.15.196:6821/2509099861","state":["exists","up"]},{"osd":7,"uuid":"fc7515ee-0a55-4b10-825e-8a4efa96d0e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":58,"last_clean_end":964,"up_from":1007,"up_thru":1008,"down_at":965,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6825","nonce":1346226909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6827","nonce":1346226909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6831","nonce":1346226909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6829","nonce":1346226909}]},"public_addr":"172.21.15.196:6825/1346226909","cluster_addr":"172.21.15.196:6827/1346226909","heartbeat_back_addr":"172.21.15.196:6831/1346226909","heartbeat_front_addr":"172.21.15.196:6829/1346226909","state":["exists","up"]},{"osd":8,"uuid":"e116a9c9-3c07-4b81-8b23-f2ba84916090","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":994,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6801","nonce":3463685812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6803","nonce":3463685812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6807","nonce":3463685812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6805","nonce":3463685812}]},"public_addr":"172.21.15.204:6801/3463685812","cluster_addr":"172.21.15.204:6803/3463685812","heartbeat_back_addr":"172.21.15.204:6807/3463685812","heartbeat_front_addr":"172.21.15.204:6805/3463685812","state":["exists","up"]},{"osd":9,"uuid":"06573a93-d7be-426b-97c4-4e088b2bfa1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":994,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6809","nonce":1139381345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6811","nonce":1139381345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6815","nonce":1139381345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6813","nonce":1139381345}]},"public_addr":"172.21.15.204:6809/1139381345","cluster_addr":"172.21.15.204:6811/1139381345","heartbeat_back_addr":"172.21.15.204:6815/1139381345","heartbeat_front_addr":"172.21.15.204:6813/1139381345","state":["exists","up"]},{"osd":10,"uuid":"388cc928-3439-4f37-a2dc-1f2a494f9846","up":1,"in":1,"weight":1,"primary_affinity":0.9919281005859375,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6817","nonce":726811436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6819","nonce":726811436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6823","nonce":726811436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6821","nonce":726811436}]},"public_addr":"172.21.15.204:6817/726811436","cluster_addr":"172.21.15.204:6819/726811436","heartbeat_back_addr":"172.21.15.204:6823/726811436","heartbeat_front_addr":"172.21.15.204:6821/726811436","state":["exists","up"]},{"osd":11,"uuid":"49911d7e-fe03-41a6-a25a-11a11c08f8b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6825","nonce":1668426191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6827","nonce":1668426191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6831","nonce":1668426191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6829","nonce":1668426191}]},"public_addr":"172.21.15.204:6825/1668426191","cluster_addr":"172.21.15.204:6827/1668426191","heartbeat_back_addr":"172.21.15.204:6831/1668426191","heartbeat_front_addr":"172.21.15.204:6829/1668426191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:15.587075+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:26:20.736447+0000","dead_epoch":866},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:15.196774+0000","dead_epoch":295},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:26:19.578424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:26:51.321361+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:26:17.144066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:00.711410+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:26:57.777730+0000","dead_epoch":964},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:26:21.783137+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:26:04.640467+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:04.952716+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:26:02.730537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2865075910":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6801/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/194694811":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/1290699598":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/2769196199":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/4153012034":"2024-09-17T05:03:00.800739+0000","172.21.15.32:6801/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/1722832924":"2024-09-17T05:03:00.800739+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":418,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":362,"length":1},{"begin":419,"length":1},{"begin":425,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:27:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:18.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:18.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:18.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:17 smithi196 ceph-mon[30802]: pgmap v1730: 127 pgs: 127 active+clean; 251 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 370 KiB/s wr, 1 op/s 2024-09-16T05:27:18.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3203048858' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:27:18.250 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:18.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:18.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:18.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:18.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:18.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:17 smithi032 ceph-mon[27388]: pgmap v1730: 127 pgs: 127 active+clean; 251 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 370 KiB/s wr, 1 op/s 2024-09-16T05:27:18.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3203048858' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:27:18.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:17 smithi204 ceph-mon[30933]: pgmap v1730: 127 pgs: 127 active+clean; 251 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 370 KiB/s wr, 1 op/s 2024-09-16T05:27:18.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3203048858' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:27:18.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:18.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:18.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:18.594 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:27:18.594 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:27:18.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:18.615 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:27:18.616 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:27:18.632 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:18.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:18.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:18.636 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:27:18.636 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:27:18.657 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:27:18.657 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:27:18.679 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:27:18.680 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:27:18.701 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:27:18.701 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:27:18.723 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:27:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:27:18.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:18.744 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:27:18.745 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:27:18.757 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:18.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:18.766 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:27:18.767 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:27:18.790 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:27:18.790 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:27:18.812 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:27:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:27:18.834 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:27:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:27:18.860 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:18.886 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:18.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:18.988 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:19.015 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:19.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:19.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:19.139 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:19.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:19.203 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/808443086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:19.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:27:19.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/808443086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:19.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:19.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:19.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:19.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/808443086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:19.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:19.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:19.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:19.552 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:19.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:19.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:19.658 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:19.659 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:19.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:19.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:19.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:19.805 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:19.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:19.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:19.932 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:19.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:20.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:20.057 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:20.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:20.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:20.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:19 smithi032 ceph-mon[27388]: pgmap v1731: 127 pgs: 127 active+clean; 236 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 190 B/s rd, 275 KiB/s wr, 1 op/s 2024-09-16T05:27:20.182 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:20.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:20.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:19 smithi196 ceph-mon[30802]: pgmap v1731: 127 pgs: 127 active+clean; 236 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 190 B/s rd, 275 KiB/s wr, 1 op/s 2024-09-16T05:27:20.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:20.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:20.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:20.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:20.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:20.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:19 smithi204 ceph-mon[30933]: pgmap v1731: 127 pgs: 127 active+clean; 236 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 190 B/s rd, 275 KiB/s wr, 1 op/s 2024-09-16T05:27:20.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:20.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:20.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:20.561 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:20.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:20.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:20.687 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:20.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:20.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:20.811 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:20.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:20.899 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:27:20.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:20.938 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:20.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:21.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:21.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:21.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:21.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2563685400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:20 smithi032 ceph-mon[27388]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:20 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2563685400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:20 smithi032 ceph-mon[27388]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-16T05:27:21.192 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:21.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:21.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2563685400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:20 smithi196 ceph-mon[30802]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:20 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2563685400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:21.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:20 smithi196 ceph-mon[30802]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-16T05:27:21.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:21.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:21.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:21.372 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:21.372 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:21.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2563685400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:21.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:20 smithi204 ceph-mon[30933]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-16T05:27:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:20 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2563685400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:20 smithi204 ceph-mon[30933]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-16T05:27:21.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:21.444 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:21.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:21.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:21.572 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:21.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:21.674 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:21.719 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:21.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:21.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:21.847 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:21.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:21.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:21.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:21.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:22.039 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:21 smithi032 ceph-mon[27388]: pgmap v1732: 127 pgs: 127 active+clean; 236 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 232 KiB/s wr, 2 op/s 2024-09-16T05:27:22.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-16T05:27:22.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:22.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:22.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:22.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:21 smithi196 ceph-mon[30802]: pgmap v1732: 127 pgs: 127 active+clean; 236 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 232 KiB/s wr, 2 op/s 2024-09-16T05:27:22.263 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:22.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:22.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:22.387 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:22.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:22.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:21 smithi204 ceph-mon[30933]: pgmap v1732: 127 pgs: 127 active+clean; 236 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 232 KiB/s wr, 2 op/s 2024-09-16T05:27:22.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T05:27:22.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:22.515 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:22.516 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:22.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:22.642 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:22.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:22.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:22.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:22.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:22.860 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:22.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:22.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:23.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:23.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:22 smithi032 ceph-mon[27388]: 2.27 scrub starts 2024-09-16T05:27:23.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:22 smithi032 ceph-mon[27388]: 2.27 scrub ok 2024-09-16T05:27:23.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:22 smithi032 ceph-mon[27388]: 2.5f scrub starts 2024-09-16T05:27:23.018 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:22 smithi032 ceph-mon[27388]: 2.5f scrub ok 2024-09-16T05:27:23.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:23.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:23.128 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:23.152 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:23.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:23.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:22 smithi196 ceph-mon[30802]: 2.27 scrub starts 2024-09-16T05:27:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:22 smithi196 ceph-mon[30802]: 2.27 scrub ok 2024-09-16T05:27:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:22 smithi196 ceph-mon[30802]: 2.5f scrub starts 2024-09-16T05:27:23.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:22 smithi196 ceph-mon[30802]: 2.5f scrub ok 2024-09-16T05:27:23.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:23.275 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:23.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:23.343 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:23.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:23.394 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:23.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:23.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:23.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:23.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:22 smithi204 ceph-mon[30933]: 2.27 scrub starts 2024-09-16T05:27:23.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:22 smithi204 ceph-mon[30933]: 2.27 scrub ok 2024-09-16T05:27:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:22 smithi204 ceph-mon[30933]: 2.5f scrub starts 2024-09-16T05:27:23.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:22 smithi204 ceph-mon[30933]: 2.5f scrub ok 2024-09-16T05:27:23.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:23.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:23.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:23.593 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:27:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:27:23.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:23.651 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:23.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:23.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:23.775 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:23.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:23.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:23.900 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:23.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:23.909 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:27:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:27:23.931 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:27:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:27:23.953 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:27:23.953 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:27:23.954 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T05:27:23.980 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:27:23.980 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:27:24.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:24.006 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:27:24.006 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:27:24.026 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:24.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:24.029 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:27:24.030 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:27:24.053 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:27:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:27:24.074 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:27:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:27:24.095 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:27:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:27:24.117 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:27:24.117 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:27:24.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:24.137 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:27:24.138 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:27:24.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:24.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:24.158 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:27:24.159 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:27:24.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: 2.2e scrub starts 2024-09-16T05:27:24.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: 2.2e scrub ok 2024-09-16T05:27:24.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:24.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:24.159 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]: dispatch 2024-09-16T05:27:24.160 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:27:24.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:27:24.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:27:24.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1676486216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:27:24.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: 2.21 scrub starts 2024-09-16T05:27:24.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: 2.21 scrub ok 2024-09-16T05:27:24.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: 2.9 scrub starts 2024-09-16T05:27:24.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: 2.9 scrub ok 2024-09-16T05:27:24.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='client.18072 172.21.15.32:0/4180502839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:24.162 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:23 smithi032 ceph-mon[27388]: from='client.18072 172.21.15.32:0/4180502839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:24.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: 2.2e scrub starts 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: 2.2e scrub ok 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]: dispatch 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]: dispatch 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]: dispatch 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]: dispatch 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:24.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]: dispatch 2024-09-16T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1676486216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: 2.21 scrub starts 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: 2.21 scrub ok 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: 2.9 scrub starts 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: 2.9 scrub ok 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='client.18072 172.21.15.32:0/4180502839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:24.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:23 smithi196 ceph-mon[30802]: from='client.18072 172.21.15.32:0/4180502839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:24.253 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:24.276 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:24.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:24.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:24.400 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:24.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: 2.2e scrub starts 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: 2.2e scrub ok 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]: dispatch 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]: dispatch 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]: dispatch 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]: dispatch 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:27:24.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1676486216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: 2.21 scrub starts 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: 2.21 scrub ok 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: 2.9 scrub starts 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: 2.9 scrub ok 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='client.18072 172.21.15.32:0/4180502839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:24.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:23 smithi204 ceph-mon[30933]: from='client.18072 172.21.15.32:0/4180502839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:24.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:24.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:24.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:24.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:24.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:27:24.648 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:24.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:24.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:24.773 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:24.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:24.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:24.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:25.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:25.025 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:25.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:25.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:25.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:25.152 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:25.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: 2.41 scrub starts 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: 2.41 scrub ok 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='client.26792 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]': finished 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]': finished 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]': finished 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]': finished 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:25.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1676486216' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:27:25.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: 2.e scrub starts 2024-09-16T05:27:25.173 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: 2.e scrub ok 2024-09-16T05:27:25.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: 2.49 scrub starts 2024-09-16T05:27:25.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: 2.49 scrub ok 2024-09-16T05:27:25.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:27:25.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:24 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:27:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: 2.41 scrub starts 2024-09-16T05:27:25.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: 2.41 scrub ok 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='client.26792 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]': finished 2024-09-16T05:27:25.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]': finished 2024-09-16T05:27:25.240 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]': finished 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1676486216' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]: dispatch 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]: dispatch 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]: dispatch 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]: dispatch 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]: dispatch 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]: dispatch 2024-09-16T05:27:25.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]: dispatch 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: 2.e scrub starts 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: 2.e scrub ok 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: 2.49 scrub starts 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: 2.49 scrub ok 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:27:25.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:24 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:27:25.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:25.277 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:25.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:25.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:25.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:25.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: 2.41 scrub starts 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: 2.41 scrub ok 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='client.26792 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]': finished 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]': finished 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]': finished 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]': finished 2024-09-16T05:27:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.07959763293494065]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07506665412385059]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07085128990422736]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09745784015417468]}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]': finished 2024-09-16T05:27:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1676486216' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:27:25.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: 2.e scrub starts 2024-09-16T05:27:25.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: 2.e scrub ok 2024-09-16T05:27:25.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: 2.49 scrub starts 2024-09-16T05:27:25.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: 2.49 scrub ok 2024-09-16T05:27:25.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:27:25.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:24 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:27:25.433 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1734,"stamp":"2024-09-16T05:27:22.293501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247293656,"num_objects":103,"num_object_clones":55,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":9,"num_read":4790,"num_read_kb":4736528,"num_write":3479,"num_write_kb":1407823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":802,"num_bytes_recovered":177106522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39700,"ondisk_log_size":39700,"up":254,"acting":254,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":254,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":986884,"kb_used_data":395352,"kb_used_omap":84,"kb_used_meta":591211,"kb_avail":1123954940,"statfs":{"total":1151940427776,"available":1150929858560,"internally_reserved":0,"allocated":404840448,"data_stored":430341416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86930,"internal_metadata":605400174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18445436,"num_objects":-10,"num_object_clones":-11,"num_object_copies":-20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":8,"num_read_kb":4871,"num_write":5,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.381745"},"pg_stats":[{"pgid":"2.5f","version":"284'221","reported_seq":3724,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.026631+0000","last_change":"2024-09-16T05:27:12.911070+0000","last_active":"2024-09-16T05:27:16.026631+0000","last_peered":"2024-09-16T05:27:16.026631+0000","last_clean":"2024-09-16T05:27:16.026631+0000","last_became_active":"2024-09-16T05:25:10.429050+0000","last_became_peered":"2024-09-16T05:25:10.429050+0000","last_unstale":"2024-09-16T05:27:16.026631+0000","last_undegraded":"2024-09-16T05:27:16.026631+0000","last_fullsized":"2024-09-16T05:27:16.026631+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":7,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:25:51.910938+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:25:51.910938+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:20.040728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000495544,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.5e","version":"1014'603","reported_seq":5239,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156360+0000","last_change":"2024-09-16T05:27:12.927261+0000","last_active":"2024-09-16T05:27:16.156360+0000","last_peered":"2024-09-16T05:27:16.156360+0000","last_clean":"2024-09-16T05:27:16.156360+0000","last_became_active":"2024-09-16T05:21:54.937455+0000","last_became_peered":"2024-09-16T05:21:54.937455+0000","last_unstale":"2024-09-16T05:27:16.156360+0000","last_undegraded":"2024-09-16T05:27:16.156360+0000","last_fullsized":"2024-09-16T05:27:16.156360+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"968'598","last_scrub_stamp":"2024-09-16T05:25:48.994532+0000","last_deep_scrub":"913'571","last_deep_scrub_stamp":"2024-09-16T05:24:23.716116+0000","last_clean_scrub_stamp":"2024-09-16T05:25:48.994532+0000","objects_scrubbed":2,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002623209,"stat_sum":{"num_bytes":3160046,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":41,"num_read_kb":40016,"num_write":32,"num_write_kb":13365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4819607,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.5d","version":"1022'1518","reported_seq":5903,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.489270+0000","last_change":"2024-09-16T05:27:12.926037+0000","last_active":"2024-09-16T05:27:16.489270+0000","last_peered":"2024-09-16T05:27:16.489270+0000","last_clean":"2024-09-16T05:27:16.489270+0000","last_became_active":"2024-09-16T05:26:22.103654+0000","last_became_peered":"2024-09-16T05:26:22.103654+0000","last_unstale":"2024-09-16T05:27:16.489270+0000","last_undegraded":"2024-09-16T05:27:16.489270+0000","last_fullsized":"2024-09-16T05:27:16.489270+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":7,"last_scrub":"989'1508","last_scrub_stamp":"2024-09-16T05:27:05.752112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.752112+0000","objects_scrubbed":6,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:08.404140+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.003510385,"stat_sum":{"num_bytes":16861832,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":77942,"num_write":69,"num_write_kb":28852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.5c","version":"1024'538","reported_seq":4650,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.898912+0000","last_change":"2024-09-16T05:27:12.917265+0000","last_active":"2024-09-16T05:27:15.898912+0000","last_peered":"2024-09-16T05:27:15.898912+0000","last_clean":"2024-09-16T05:27:15.898912+0000","last_became_active":"2024-09-16T05:21:32.828660+0000","last_became_peered":"2024-09-16T05:21:32.828660+0000","last_unstale":"2024-09-16T05:27:15.898912+0000","last_undegraded":"2024-09-16T05:27:15.898912+0000","last_fullsized":"2024-09-16T05:27:15.898912+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"968'534","last_scrub_stamp":"2024-09-16T05:25:25.192459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:25.192459+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:20.421368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054332800000000004,"stat_sum":{"num_bytes":3361730,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":15894,"num_write":33,"num_write_kb":7646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.5b","version":"1022'695","reported_seq":4832,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488920+0000","last_change":"2024-09-16T05:27:12.925150+0000","last_active":"2024-09-16T05:27:16.488920+0000","last_peered":"2024-09-16T05:27:16.488920+0000","last_clean":"2024-09-16T05:27:16.488920+0000","last_became_active":"2024-09-16T05:26:22.102783+0000","last_became_peered":"2024-09-16T05:26:22.102783+0000","last_unstale":"2024-09-16T05:27:16.488920+0000","last_undegraded":"2024-09-16T05:27:16.488920+0000","last_fullsized":"2024-09-16T05:27:16.488920+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":7,"last_scrub":"1016'691","last_scrub_stamp":"2024-09-16T05:27:07.797068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:07.797068+0000","objects_scrubbed":4,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:25.454355+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.003293495,"stat_sum":{"num_bytes":4838247,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":52563,"num_write":13,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.5a","version":"1018'1286","reported_seq":5730,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119390+0000","last_change":"2024-09-16T05:27:12.911154+0000","last_active":"2024-09-16T05:27:16.119390+0000","last_peered":"2024-09-16T05:27:16.119390+0000","last_clean":"2024-09-16T05:27:16.119390+0000","last_became_active":"2024-09-16T05:21:32.826884+0000","last_became_peered":"2024-09-16T05:21:32.826884+0000","last_unstale":"2024-09-16T05:27:16.119390+0000","last_undegraded":"2024-09-16T05:27:16.119390+0000","last_fullsized":"2024-09-16T05:27:16.119390+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"936'1284","last_scrub_stamp":"2024-09-16T05:27:04.123197+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.123197+0000","objects_scrubbed":2,"log_size":1286,"log_dups_size":0,"ondisk_log_size":1286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:04.506207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001141214,"stat_sum":{"num_bytes":5065634,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":61907,"num_write":38,"num_write_kb":8752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.59","version":"182'60","reported_seq":1362,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899540+0000","last_change":"2024-09-16T05:27:12.913321+0000","last_active":"2024-09-16T05:27:15.899540+0000","last_peered":"2024-09-16T05:27:15.899540+0000","last_clean":"2024-09-16T05:27:15.899540+0000","last_became_active":"2024-09-16T05:21:54.931981+0000","last_became_peered":"2024-09-16T05:21:54.931981+0000","last_unstale":"2024-09-16T05:27:15.899540+0000","last_undegraded":"2024-09-16T05:27:15.899540+0000","last_fullsized":"2024-09-16T05:27:15.899540+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:26:49.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.813088+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:03.760749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053935300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.58","version":"181'57","reported_seq":4494,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.287878+0000","last_change":"2024-09-16T05:27:12.919068+0000","last_active":"2024-09-16T05:27:16.287878+0000","last_peered":"2024-09-16T05:27:16.287878+0000","last_clean":"2024-09-16T05:27:16.287878+0000","last_became_active":"2024-09-16T05:21:29.826747+0000","last_became_peered":"2024-09-16T05:21:29.826747+0000","last_unstale":"2024-09-16T05:27:16.287878+0000","last_undegraded":"2024-09-16T05:27:16.287878+0000","last_fullsized":"2024-09-16T05:27:16.287878+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:25:29.256936+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:25:29.256936+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076311900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.57","version":"0'0","reported_seq":2025,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156392+0000","last_change":"2024-09-16T05:27:12.926760+0000","last_active":"2024-09-16T05:27:16.156392+0000","last_peered":"2024-09-16T05:27:16.156392+0000","last_clean":"2024-09-16T05:27:16.156392+0000","last_became_active":"2024-09-16T05:21:29.827377+0000","last_became_peered":"2024-09-16T05:21:29.827377+0000","last_unstale":"2024-09-16T05:27:16.156392+0000","last_undegraded":"2024-09-16T05:27:16.156392+0000","last_fullsized":"2024-09-16T05:27:16.156392+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:26:28.249327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:28.249327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:49.447557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025192769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.56","version":"1018'447","reported_seq":4582,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899085+0000","last_change":"2024-09-16T05:27:12.917330+0000","last_active":"2024-09-16T05:27:15.899085+0000","last_peered":"2024-09-16T05:27:15.899085+0000","last_clean":"2024-09-16T05:27:15.899085+0000","last_became_active":"2024-09-16T05:21:32.829699+0000","last_became_peered":"2024-09-16T05:21:32.829699+0000","last_unstale":"2024-09-16T05:27:15.899085+0000","last_undegraded":"2024-09-16T05:27:15.899085+0000","last_fullsized":"2024-09-16T05:27:15.899085+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":737,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":7,"last_scrub":"958'445","last_scrub_stamp":"2024-09-16T05:25:33.225813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:33.225813+0000","objects_scrubbed":3,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:20.421391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00035786800000000001,"stat_sum":{"num_bytes":6605712,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":35040,"num_write":40,"num_write_kb":14849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.55","version":"433'507","reported_seq":3484,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119577+0000","last_change":"2024-09-16T05:27:12.910821+0000","last_active":"2024-09-16T05:27:16.119577+0000","last_peered":"2024-09-16T05:27:16.119577+0000","last_clean":"2024-09-16T05:27:16.119577+0000","last_became_active":"2024-09-16T05:21:24.815957+0000","last_became_peered":"2024-09-16T05:21:24.815957+0000","last_unstale":"2024-09-16T05:27:16.119577+0000","last_undegraded":"2024-09-16T05:27:16.119577+0000","last_fullsized":"2024-09-16T05:27:16.119577+0000","mapping_epoch":734,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":7,"last_scrub":"433'507","last_scrub_stamp":"2024-09-16T05:26:07.420292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:07.420292+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.118205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086801000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.54","version":"1022'505","reported_seq":4621,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988330+0000","last_change":"2024-09-16T05:27:13.272167+0000","last_active":"2024-09-16T05:27:15.988330+0000","last_peered":"2024-09-16T05:27:15.988330+0000","last_clean":"2024-09-16T05:27:15.988330+0000","last_became_active":"2024-09-16T05:20:23.798799+0000","last_became_peered":"2024-09-16T05:20:23.798799+0000","last_unstale":"2024-09-16T05:27:15.988330+0000","last_undegraded":"2024-09-16T05:27:15.988330+0000","last_fullsized":"2024-09-16T05:27:15.988330+0000","mapping_epoch":673,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":674,"parent":"0.0","parent_split_bits":7,"last_scrub":"958'486","last_scrub_stamp":"2024-09-16T05:26:28.217650+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:26:28.217650+0000","objects_scrubbed":2,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:30.259978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025439009999999999,"stat_sum":{"num_bytes":7221133,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":99,"num_read_kb":86505,"num_write":72,"num_write_kb":31931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3977735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.53","version":"181'15","reported_seq":1064,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119468+0000","last_change":"2024-09-16T05:27:12.911282+0000","last_active":"2024-09-16T05:27:16.119468+0000","last_peered":"2024-09-16T05:27:16.119468+0000","last_clean":"2024-09-16T05:27:16.119468+0000","last_became_active":"2024-09-16T05:25:10.429082+0000","last_became_peered":"2024-09-16T05:25:10.429082+0000","last_unstale":"2024-09-16T05:27:16.119468+0000","last_undegraded":"2024-09-16T05:27:16.119468+0000","last_fullsized":"2024-09-16T05:27:16.119468+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:26:26.301353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:26.301353+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:43.998802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001180623,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.52","version":"272'69","reported_seq":2858,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.898768+0000","last_change":"2024-09-16T05:27:12.916961+0000","last_active":"2024-09-16T05:27:15.898768+0000","last_peered":"2024-09-16T05:27:15.898768+0000","last_clean":"2024-09-16T05:27:15.898768+0000","last_became_active":"2024-09-16T05:16:27.734325+0000","last_became_peered":"2024-09-16T05:16:27.734325+0000","last_unstale":"2024-09-16T05:27:15.898768+0000","last_undegraded":"2024-09-16T05:27:15.898768+0000","last_fullsized":"2024-09-16T05:27:15.898768+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:26:45.512785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:45.512785+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:14.830983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047082,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.51","version":"272'115","reported_seq":2918,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.155919+0000","last_change":"2024-09-16T05:27:12.925179+0000","last_active":"2024-09-16T05:27:16.155919+0000","last_peered":"2024-09-16T05:27:16.155919+0000","last_clean":"2024-09-16T05:27:16.155919+0000","last_became_active":"2024-09-16T05:18:02.890143+0000","last_became_peered":"2024-09-16T05:18:02.890143+0000","last_unstale":"2024-09-16T05:27:16.155919+0000","last_undegraded":"2024-09-16T05:27:16.155919+0000","last_fullsized":"2024-09-16T05:27:16.155919+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:26:08.006890+0000","last_deep_scrub":"272'115","last_deep_scrub_stamp":"2024-09-16T05:22:11.104571+0000","last_clean_scrub_stamp":"2024-09-16T05:26:08.006890+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:32.932447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00094823900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.50","version":"182'16","reported_seq":2845,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119201+0000","last_change":"2024-09-16T05:27:12.910869+0000","last_active":"2024-09-16T05:27:16.119201+0000","last_peered":"2024-09-16T05:27:16.119201+0000","last_clean":"2024-09-16T05:27:16.119201+0000","last_became_active":"2024-09-16T05:16:28.649179+0000","last_became_peered":"2024-09-16T05:16:28.649179+0000","last_unstale":"2024-09-16T05:27:16.119201+0000","last_undegraded":"2024-09-16T05:27:16.119201+0000","last_fullsized":"2024-09-16T05:27:16.119201+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:26:32.293246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.293246+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:49.505878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00099143100000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.4f","version":"360'281","reported_seq":3036,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.489018+0000","last_change":"2024-09-16T05:27:12.923792+0000","last_active":"2024-09-16T05:27:16.489018+0000","last_peered":"2024-09-16T05:27:16.489018+0000","last_clean":"2024-09-16T05:27:16.489018+0000","last_became_active":"2024-09-16T05:26:58.981300+0000","last_became_peered":"2024-09-16T05:26:58.981300+0000","last_unstale":"2024-09-16T05:27:16.489018+0000","last_undegraded":"2024-09-16T05:27:16.489018+0000","last_fullsized":"2024-09-16T05:27:16.489018+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":7,"last_scrub":"360'281","last_scrub_stamp":"2024-09-16T05:27:02.747937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:02.747937+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:30.552757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015534100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.4e","version":"272'232","reported_seq":4621,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:20.994910+0000","last_change":"2024-09-16T05:27:13.347980+0000","last_active":"2024-09-16T05:27:20.994910+0000","last_peered":"2024-09-16T05:27:20.994910+0000","last_clean":"2024-09-16T05:27:20.994910+0000","last_became_active":"2024-09-16T05:18:19.769089+0000","last_became_peered":"2024-09-16T05:18:19.769089+0000","last_unstale":"2024-09-16T05:27:20.994910+0000","last_undegraded":"2024-09-16T05:27:20.994910+0000","last_fullsized":"2024-09-16T05:27:20.994910+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":7,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:26:49.799970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.799970+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:14.359751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00099633100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4d","version":"1022'797","reported_seq":5350,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988495+0000","last_change":"2024-09-16T05:27:13.272204+0000","last_active":"2024-09-16T05:27:15.988495+0000","last_peered":"2024-09-16T05:27:15.988495+0000","last_clean":"2024-09-16T05:27:15.988495+0000","last_became_active":"2024-09-16T05:17:59.870914+0000","last_became_peered":"2024-09-16T05:17:59.870914+0000","last_unstale":"2024-09-16T05:27:15.988495+0000","last_undegraded":"2024-09-16T05:27:15.988495+0000","last_fullsized":"2024-09-16T05:27:15.988495+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":7,"last_scrub":"975'784","last_scrub_stamp":"2024-09-16T05:25:54.222878+0000","last_deep_scrub":"975'784","last_deep_scrub_stamp":"2024-09-16T05:25:54.222878+0000","last_clean_scrub_stamp":"2024-09-16T05:25:54.222878+0000","objects_scrubbed":3,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:17.220721+0000","scrub_duration":816,"objects_trimmed":1,"snaptrim_duration":0.0026449120000000001,"stat_sum":{"num_bytes":6485385,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":96,"num_read_kb":78696,"num_write":117,"num_write_kb":36984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8199946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.4c","version":"1022'481","reported_seq":4979,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.155957+0000","last_change":"2024-09-16T05:27:12.926522+0000","last_active":"2024-09-16T05:27:16.155957+0000","last_peered":"2024-09-16T05:27:16.155957+0000","last_clean":"2024-09-16T05:27:16.155957+0000","last_became_active":"2024-09-16T05:18:02.888895+0000","last_became_peered":"2024-09-16T05:18:02.888895+0000","last_unstale":"2024-09-16T05:27:16.155957+0000","last_undegraded":"2024-09-16T05:27:16.155957+0000","last_fullsized":"2024-09-16T05:27:16.155957+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":435,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":7,"last_scrub":"968'475","last_scrub_stamp":"2024-09-16T05:26:29.285007+0000","last_deep_scrub":"517'288","last_deep_scrub_stamp":"2024-09-16T05:17:48.155777+0000","last_clean_scrub_stamp":"2024-09-16T05:26:29.285007+0000","objects_scrubbed":3,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.643612+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0022536610000000001,"stat_sum":{"num_bytes":2638963,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":112,"num_read_kb":129478,"num_write":63,"num_write_kb":21390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2914314,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.4b","version":"244'177","reported_seq":4661,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988200+0000","last_change":"2024-09-16T05:27:13.272752+0000","last_active":"2024-09-16T05:27:15.988200+0000","last_peered":"2024-09-16T05:27:15.988200+0000","last_clean":"2024-09-16T05:27:15.988200+0000","last_became_active":"2024-09-16T05:26:34.243526+0000","last_became_peered":"2024-09-16T05:26:34.243526+0000","last_unstale":"2024-09-16T05:27:15.988200+0000","last_undegraded":"2024-09-16T05:27:15.988200+0000","last_fullsized":"2024-09-16T05:27:15.988200+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:26:30.518503+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:25:07.644216+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.518503+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:32.125742+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0.0027925290000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.4a","version":"1022'733","reported_seq":5269,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156214+0000","last_change":"2024-09-16T05:27:12.928082+0000","last_active":"2024-09-16T05:27:16.156214+0000","last_peered":"2024-09-16T05:27:16.156214+0000","last_clean":"2024-09-16T05:27:16.156214+0000","last_became_active":"2024-09-16T05:26:34.148371+0000","last_became_peered":"2024-09-16T05:26:34.148371+0000","last_unstale":"2024-09-16T05:27:16.156214+0000","last_undegraded":"2024-09-16T05:27:16.156214+0000","last_fullsized":"2024-09-16T05:27:16.156214+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":7,"last_scrub":"968'721","last_scrub_stamp":"2024-09-16T05:26:48.151827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:48.151827+0000","objects_scrubbed":3,"log_size":733,"log_dups_size":0,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:58.885149+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0036299309999999999,"stat_sum":{"num_bytes":5731781,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":102681,"num_write":86,"num_write_kb":34757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":6106400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.49","version":"245'220","reported_seq":4690,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:20.994959+0000","last_change":"2024-09-16T05:27:13.347982+0000","last_active":"2024-09-16T05:27:20.994959+0000","last_peered":"2024-09-16T05:27:20.994959+0000","last_clean":"2024-09-16T05:27:20.994959+0000","last_became_active":"2024-09-16T05:14:30.852410+0000","last_became_peered":"2024-09-16T05:14:30.852410+0000","last_unstale":"2024-09-16T05:27:20.994959+0000","last_undegraded":"2024-09-16T05:27:20.994959+0000","last_fullsized":"2024-09-16T05:27:20.994959+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:26:10.104210+0000","last_deep_scrub":"245'220","last_deep_scrub_stamp":"2024-09-16T05:24:26.763617+0000","last_clean_scrub_stamp":"2024-09-16T05:26:10.104210+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:22.409247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090971200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.48","version":"1022'471","reported_seq":4930,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288218+0000","last_change":"2024-09-16T05:27:12.920295+0000","last_active":"2024-09-16T05:27:16.288218+0000","last_peered":"2024-09-16T05:27:16.288218+0000","last_clean":"2024-09-16T05:27:16.288218+0000","last_became_active":"2024-09-16T05:14:27.837984+0000","last_became_peered":"2024-09-16T05:14:27.837984+0000","last_unstale":"2024-09-16T05:27:16.288218+0000","last_undegraded":"2024-09-16T05:27:16.288218+0000","last_fullsized":"2024-09-16T05:27:16.288218+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"961'455","last_scrub_stamp":"2024-09-16T05:25:51.181455+0000","last_deep_scrub":"628'287","last_deep_scrub_stamp":"2024-09-16T05:19:38.000359+0000","last_clean_scrub_stamp":"2024-09-16T05:25:51.181455+0000","objects_scrubbed":2,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020653440000000002,"stat_sum":{"num_bytes":8303582,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":96,"num_read_kb":79347,"num_write":84,"num_write_kb":35945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.47","version":"0'0","reported_seq":3476,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.898876+0000","last_change":"2024-09-16T05:27:12.917526+0000","last_active":"2024-09-16T05:27:15.898876+0000","last_peered":"2024-09-16T05:27:15.898876+0000","last_clean":"2024-09-16T05:27:15.898876+0000","last_became_active":"2024-09-16T05:16:17.734037+0000","last_became_peered":"2024-09-16T05:16:17.734037+0000","last_unstale":"2024-09-16T05:27:15.898876+0000","last_undegraded":"2024-09-16T05:27:15.898876+0000","last_fullsized":"2024-09-16T05:27:15.898876+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:26:05.465374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:10.838189+0000","last_clean_scrub_stamp":"2024-09-16T05:26:05.465374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:16.879876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036797499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.46","version":"116'9","reported_seq":1698,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988600+0000","last_change":"2024-09-16T05:27:13.272488+0000","last_active":"2024-09-16T05:27:15.988600+0000","last_peered":"2024-09-16T05:27:15.988600+0000","last_clean":"2024-09-16T05:27:15.988600+0000","last_became_active":"2024-09-16T05:26:58.981495+0000","last_became_peered":"2024-09-16T05:26:58.981495+0000","last_unstale":"2024-09-16T05:27:15.988600+0000","last_undegraded":"2024-09-16T05:27:15.988600+0000","last_fullsized":"2024-09-16T05:27:15.988600+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:27:04.323408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.323408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:19.578130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025322529999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.45","version":"305'464","reported_seq":3640,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.489041+0000","last_change":"2024-09-16T05:27:12.924050+0000","last_active":"2024-09-16T05:27:16.489041+0000","last_peered":"2024-09-16T05:27:16.489041+0000","last_clean":"2024-09-16T05:27:16.489041+0000","last_became_active":"2024-09-16T05:26:22.109974+0000","last_became_peered":"2024-09-16T05:26:22.109974+0000","last_unstale":"2024-09-16T05:27:16.489041+0000","last_undegraded":"2024-09-16T05:27:16.489041+0000","last_fullsized":"2024-09-16T05:27:16.489041+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'464","last_scrub_stamp":"2024-09-16T05:27:08.794841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:08.794841+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:38.544699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017091789999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.44","version":"182'89","reported_seq":1115,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488862+0000","last_change":"2024-09-16T05:27:12.922646+0000","last_active":"2024-09-16T05:27:16.488862+0000","last_peered":"2024-09-16T05:27:16.488862+0000","last_clean":"2024-09-16T05:27:16.488862+0000","last_became_active":"2024-09-16T05:26:22.907117+0000","last_became_peered":"2024-09-16T05:26:22.907117+0000","last_unstale":"2024-09-16T05:27:16.488862+0000","last_undegraded":"2024-09-16T05:27:16.488862+0000","last_fullsized":"2024-09-16T05:27:16.488862+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:23:31.555842+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:21:08.436254+0000","last_clean_scrub_stamp":"2024-09-16T05:23:31.555842+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090961600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.43","version":"281'58","reported_seq":3370,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.004773+0000","last_change":"2024-09-16T05:27:13.276014+0000","last_active":"2024-09-16T05:27:21.004773+0000","last_peered":"2024-09-16T05:27:21.004773+0000","last_clean":"2024-09-16T05:27:21.004773+0000","last_became_active":"2024-09-16T05:26:59.168240+0000","last_became_peered":"2024-09-16T05:26:59.168240+0000","last_unstale":"2024-09-16T05:27:21.004773+0000","last_undegraded":"2024-09-16T05:27:21.004773+0000","last_fullsized":"2024-09-16T05:27:21.004773+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":7,"last_scrub":"281'58","last_scrub_stamp":"2024-09-16T05:24:19.083933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:19.083933+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.757802+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011764430000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.42","version":"182'20","reported_seq":3447,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395731+0000","last_change":"2024-09-16T05:27:12.919301+0000","last_active":"2024-09-16T05:27:16.395731+0000","last_peered":"2024-09-16T05:27:16.395731+0000","last_clean":"2024-09-16T05:27:16.395731+0000","last_became_active":"2024-09-16T05:14:27.833792+0000","last_became_peered":"2024-09-16T05:14:27.833792+0000","last_unstale":"2024-09-16T05:27:16.395731+0000","last_undegraded":"2024-09-16T05:27:16.395731+0000","last_fullsized":"2024-09-16T05:27:16.395731+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":320,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":7,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:27:08.505063+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:27:08.505063+0000","last_clean_scrub_stamp":"2024-09-16T05:27:08.505063+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:24.419868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0043701679999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.41","version":"178'50","reported_seq":2322,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.898651+0000","last_change":"2024-09-16T05:27:12.916745+0000","last_active":"2024-09-16T05:27:15.898651+0000","last_peered":"2024-09-16T05:27:15.898651+0000","last_clean":"2024-09-16T05:27:15.898651+0000","last_became_active":"2024-09-16T05:21:54.930175+0000","last_became_peered":"2024-09-16T05:21:54.930175+0000","last_unstale":"2024-09-16T05:27:15.898651+0000","last_undegraded":"2024-09-16T05:27:15.898651+0000","last_fullsized":"2024-09-16T05:27:15.898651+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":7,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:26:04.510520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:04.510520+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:22.752695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037597900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.40","version":"205'33","reported_seq":1989,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.005018+0000","last_change":"2024-09-16T05:27:13.280981+0000","last_active":"2024-09-16T05:27:21.005018+0000","last_peered":"2024-09-16T05:27:21.005018+0000","last_clean":"2024-09-16T05:27:21.005018+0000","last_became_active":"2024-09-16T05:26:59.976988+0000","last_became_peered":"2024-09-16T05:26:59.976988+0000","last_unstale":"2024-09-16T05:27:21.005018+0000","last_undegraded":"2024-09-16T05:27:21.005018+0000","last_fullsized":"2024-09-16T05:27:21.005018+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":7,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:25:04.895079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:04.895079+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.757883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005523778,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3f","version":"1022'609","reported_seq":5121,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.005136+0000","last_change":"2024-09-16T05:27:13.279449+0000","last_active":"2024-09-16T05:27:21.005136+0000","last_peered":"2024-09-16T05:27:21.005136+0000","last_clean":"2024-09-16T05:27:21.005136+0000","last_became_active":"2024-09-16T05:26:58.981732+0000","last_became_peered":"2024-09-16T05:26:58.981732+0000","last_unstale":"2024-09-16T05:27:21.005136+0000","last_undegraded":"2024-09-16T05:27:21.005136+0000","last_fullsized":"2024-09-16T05:27:21.005136+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":6,"last_scrub":"874'563","last_scrub_stamp":"2024-09-16T05:24:01.125030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:01.125030+0000","objects_scrubbed":3,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.757743+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0043926440000000002,"stat_sum":{"num_bytes":7865401,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":154,"num_read_kb":147209,"num_write":94,"num_write_kb":30495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":11726959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3e","version":"955'603","reported_seq":5063,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.900080+0000","last_change":"2024-09-16T05:27:12.915793+0000","last_active":"2024-09-16T05:27:15.900080+0000","last_peered":"2024-09-16T05:27:15.900080+0000","last_clean":"2024-09-16T05:27:15.900080+0000","last_became_active":"2024-09-16T05:13:48.685792+0000","last_became_peered":"2024-09-16T05:13:48.685792+0000","last_unstale":"2024-09-16T05:27:15.900080+0000","last_undegraded":"2024-09-16T05:27:15.900080+0000","last_fullsized":"2024-09-16T05:27:15.900080+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":6,"last_scrub":"955'603","last_scrub_stamp":"2024-09-16T05:26:06.867173+0000","last_deep_scrub":"431'368","last_deep_scrub_stamp":"2024-09-16T05:16:20.868538+0000","last_clean_scrub_stamp":"2024-09-16T05:26:06.867173+0000","objects_scrubbed":1,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:16.384359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022899140000000001,"stat_sum":{"num_bytes":2554994,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":61222,"num_write":79,"num_write_kb":37651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.3d","version":"290'477","reported_seq":2575,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488852+0000","last_change":"2024-09-16T05:27:12.923139+0000","last_active":"2024-09-16T05:27:16.488852+0000","last_peered":"2024-09-16T05:27:16.488852+0000","last_clean":"2024-09-16T05:27:16.488852+0000","last_became_active":"2024-09-16T05:26:22.101654+0000","last_became_peered":"2024-09-16T05:26:22.101654+0000","last_unstale":"2024-09-16T05:27:16.488852+0000","last_undegraded":"2024-09-16T05:27:16.488852+0000","last_fullsized":"2024-09-16T05:27:16.488852+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":6,"last_scrub":"290'477","last_scrub_stamp":"2024-09-16T05:26:30.749976+0000","last_deep_scrub":"290'477","last_deep_scrub_stamp":"2024-09-16T05:22:34.497749+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.749976+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:46.816370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001067143,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.3c","version":"1022'836","reported_seq":5370,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.005126+0000","last_change":"2024-09-16T05:27:13.277411+0000","last_active":"2024-09-16T05:27:21.005126+0000","last_peered":"2024-09-16T05:27:21.005126+0000","last_clean":"2024-09-16T05:27:21.005126+0000","last_became_active":"2024-09-16T05:26:58.978797+0000","last_became_peered":"2024-09-16T05:26:58.978797+0000","last_unstale":"2024-09-16T05:27:21.005126+0000","last_undegraded":"2024-09-16T05:27:21.005126+0000","last_fullsized":"2024-09-16T05:27:21.005126+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":6,"last_scrub":"1008'832","last_scrub_stamp":"2024-09-16T05:27:02.696145+0000","last_deep_scrub":"342'400","last_deep_scrub_stamp":"2024-09-16T05:14:51.845721+0000","last_clean_scrub_stamp":"2024-09-16T05:27:02.696145+0000","objects_scrubbed":2,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:31.459247+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024788929999999998,"stat_sum":{"num_bytes":5126729,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":199,"num_read_kb":258106,"num_write":104,"num_write_kb":36472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":14713263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3b","version":"1022'583","reported_seq":5025,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.155870+0000","last_change":"2024-09-16T05:27:12.926675+0000","last_active":"2024-09-16T05:27:16.155870+0000","last_peered":"2024-09-16T05:27:16.155870+0000","last_clean":"2024-09-16T05:27:16.155870+0000","last_became_active":"2024-09-16T05:26:37.067134+0000","last_became_peered":"2024-09-16T05:26:37.067134+0000","last_unstale":"2024-09-16T05:27:16.155870+0000","last_undegraded":"2024-09-16T05:27:16.155870+0000","last_fullsized":"2024-09-16T05:27:16.155870+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":6,"last_scrub":"968'566","last_scrub_stamp":"2024-09-16T05:25:49.859987+0000","last_deep_scrub":"520'342","last_deep_scrub_stamp":"2024-09-16T05:17:50.304398+0000","last_clean_scrub_stamp":"2024-09-16T05:25:49.859987+0000","objects_scrubbed":1,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.00244751,"stat_sum":{"num_bytes":6856090,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":151,"num_read_kb":158206,"num_write":99,"num_write_kb":40426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.3a","version":"1018'707","reported_seq":5153,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.287969+0000","last_change":"2024-09-16T05:27:12.920294+0000","last_active":"2024-09-16T05:27:16.287969+0000","last_peered":"2024-09-16T05:27:16.287969+0000","last_clean":"2024-09-16T05:27:16.287969+0000","last_became_active":"2024-09-16T05:14:41.828635+0000","last_became_peered":"2024-09-16T05:14:41.828635+0000","last_unstale":"2024-09-16T05:27:16.287969+0000","last_undegraded":"2024-09-16T05:27:16.287969+0000","last_fullsized":"2024-09-16T05:27:16.287969+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":6,"last_scrub":"945'705","last_scrub_stamp":"2024-09-16T05:26:08.171116+0000","last_deep_scrub":"298'411","last_deep_scrub_stamp":"2024-09-16T05:13:50.352903+0000","last_clean_scrub_stamp":"2024-09-16T05:26:08.171116+0000","objects_scrubbed":2,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017920189999999999,"stat_sum":{"num_bytes":5446062,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":76386,"num_write":90,"num_write_kb":46848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2563138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.39","version":"182'60","reported_seq":3552,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.287978+0000","last_change":"2024-09-16T05:27:12.919643+0000","last_active":"2024-09-16T05:27:16.287978+0000","last_peered":"2024-09-16T05:27:16.287978+0000","last_clean":"2024-09-16T05:27:16.287978+0000","last_became_active":"2024-09-16T05:26:34.141355+0000","last_became_peered":"2024-09-16T05:26:34.141355+0000","last_unstale":"2024-09-16T05:27:16.287978+0000","last_undegraded":"2024-09-16T05:27:16.287978+0000","last_fullsized":"2024-09-16T05:27:16.287978+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:25:48.204026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:48.204026+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013424229999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.38","version":"181'57","reported_seq":3565,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395715+0000","last_change":"2024-09-16T05:27:12.915034+0000","last_active":"2024-09-16T05:27:16.395715+0000","last_peered":"2024-09-16T05:27:16.395715+0000","last_clean":"2024-09-16T05:27:16.395715+0000","last_became_active":"2024-09-16T05:26:34.143911+0000","last_became_peered":"2024-09-16T05:26:34.143911+0000","last_unstale":"2024-09-16T05:27:16.395715+0000","last_undegraded":"2024-09-16T05:27:16.395715+0000","last_fullsized":"2024-09-16T05:27:16.395715+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:25:53.640546+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:13:12.039899+0000","last_clean_scrub_stamp":"2024-09-16T05:25:53.640546+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001372294,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.37","version":"0'0","reported_seq":2413,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156659+0000","last_change":"2024-09-16T05:27:12.927890+0000","last_active":"2024-09-16T05:27:16.156659+0000","last_peered":"2024-09-16T05:27:16.156659+0000","last_clean":"2024-09-16T05:27:16.156659+0000","last_became_active":"2024-09-16T05:21:54.938965+0000","last_became_peered":"2024-09-16T05:21:54.938965+0000","last_unstale":"2024-09-16T05:27:16.156659+0000","last_undegraded":"2024-09-16T05:27:16.156659+0000","last_fullsized":"2024-09-16T05:27:16.156659+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:26:47.128875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:14:53.650972+0000","last_clean_scrub_stamp":"2024-09-16T05:26:47.128875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:12.095586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030012699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.36","version":"272'60","reported_seq":1115,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488856+0000","last_change":"2024-09-16T05:27:12.922393+0000","last_active":"2024-09-16T05:27:16.488856+0000","last_peered":"2024-09-16T05:27:16.488856+0000","last_clean":"2024-09-16T05:27:16.488856+0000","last_became_active":"2024-09-16T05:26:22.907174+0000","last_became_peered":"2024-09-16T05:26:22.907174+0000","last_unstale":"2024-09-16T05:27:16.488856+0000","last_undegraded":"2024-09-16T05:27:16.488856+0000","last_fullsized":"2024-09-16T05:27:16.488856+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":6,"last_scrub":"272'60","last_scrub_stamp":"2024-09-16T05:27:04.765847+0000","last_deep_scrub":"272'60","last_deep_scrub_stamp":"2024-09-16T05:16:02.904155+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.765847+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:32.195765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078336699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.35","version":"843'947","reported_seq":5140,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488991+0000","last_change":"2024-09-16T05:27:12.923570+0000","last_active":"2024-09-16T05:27:16.488991+0000","last_peered":"2024-09-16T05:27:16.488991+0000","last_clean":"2024-09-16T05:27:16.488991+0000","last_became_active":"2024-09-16T05:26:22.907282+0000","last_became_peered":"2024-09-16T05:26:22.907282+0000","last_unstale":"2024-09-16T05:27:16.488991+0000","last_undegraded":"2024-09-16T05:27:16.488991+0000","last_fullsized":"2024-09-16T05:27:16.488991+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":6,"last_scrub":"843'947","last_scrub_stamp":"2024-09-16T05:23:34.604753+0000","last_deep_scrub":"276'397","last_deep_scrub_stamp":"2024-09-16T05:13:28.279556+0000","last_clean_scrub_stamp":"2024-09-16T05:23:34.604753+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001383893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":191,"num_read_kb":188287,"num_write":131,"num_write_kb":57425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":17054243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.34","version":"1022'928","reported_seq":4959,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119732+0000","last_change":"2024-09-16T05:27:12.915300+0000","last_active":"2024-09-16T05:27:16.119732+0000","last_peered":"2024-09-16T05:27:16.119732+0000","last_clean":"2024-09-16T05:27:16.119732+0000","last_became_active":"2024-09-16T05:26:34.157542+0000","last_became_peered":"2024-09-16T05:26:34.157542+0000","last_unstale":"2024-09-16T05:27:16.119732+0000","last_undegraded":"2024-09-16T05:27:16.119732+0000","last_fullsized":"2024-09-16T05:27:16.119732+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":6,"last_scrub":"947'912","last_scrub_stamp":"2024-09-16T05:26:09.069495+0000","last_deep_scrub":"495'378","last_deep_scrub_stamp":"2024-09-16T05:17:29.256208+0000","last_clean_scrub_stamp":"2024-09-16T05:26:09.069495+0000","objects_scrubbed":6,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:31.800119+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0048136029999999996,"stat_sum":{"num_bytes":12085388,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":248,"num_read_kb":273998,"num_write":177,"num_write_kb":67192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":7974543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.33","version":"181'15","reported_seq":3455,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.004992+0000","last_change":"2024-09-16T05:27:13.275672+0000","last_active":"2024-09-16T05:27:21.004992+0000","last_peered":"2024-09-16T05:27:21.004992+0000","last_clean":"2024-09-16T05:27:21.004992+0000","last_became_active":"2024-09-16T05:26:58.979046+0000","last_became_peered":"2024-09-16T05:26:58.979046+0000","last_unstale":"2024-09-16T05:27:21.004992+0000","last_undegraded":"2024-09-16T05:27:21.004992+0000","last_fullsized":"2024-09-16T05:27:21.004992+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:27:04.740773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.740773+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:10.576407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00085837100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.32","version":"940'330","reported_seq":3973,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899566+0000","last_change":"2024-09-16T05:27:12.914606+0000","last_active":"2024-09-16T05:27:15.899566+0000","last_peered":"2024-09-16T05:27:15.899566+0000","last_clean":"2024-09-16T05:27:15.899566+0000","last_became_active":"2024-09-16T05:13:27.687406+0000","last_became_peered":"2024-09-16T05:13:27.687406+0000","last_unstale":"2024-09-16T05:27:15.899566+0000","last_undegraded":"2024-09-16T05:27:15.899566+0000","last_fullsized":"2024-09-16T05:27:15.899566+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"940'330","last_scrub_stamp":"2024-09-16T05:25:49.472822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:49.472822+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:16.558698+0000","scrub_duration":459,"objects_trimmed":0,"snaptrim_duration":0.00148277,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":137206,"num_write":62,"num_write_kb":24355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.31","version":"1018'440","reported_seq":4666,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.898841+0000","last_change":"2024-09-16T05:27:12.916977+0000","last_active":"2024-09-16T05:27:15.898841+0000","last_peered":"2024-09-16T05:27:15.898841+0000","last_clean":"2024-09-16T05:27:15.898841+0000","last_became_active":"2024-09-16T05:17:59.868049+0000","last_became_peered":"2024-09-16T05:17:59.868049+0000","last_unstale":"2024-09-16T05:27:15.898841+0000","last_undegraded":"2024-09-16T05:27:15.898841+0000","last_fullsized":"2024-09-16T05:27:15.898841+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"964'436","last_scrub_stamp":"2024-09-16T05:25:54.463609+0000","last_deep_scrub":"280'123","last_deep_scrub_stamp":"2024-09-16T05:13:31.958196+0000","last_clean_scrub_stamp":"2024-09-16T05:25:54.463609+0000","objects_scrubbed":3,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:20.421410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044523499999999999,"stat_sum":{"num_bytes":6320026,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":160,"num_read_kb":138714,"num_write":102,"num_write_kb":39613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.30","version":"182'16","reported_seq":2396,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.155920+0000","last_change":"2024-09-16T05:27:12.924618+0000","last_active":"2024-09-16T05:27:16.155920+0000","last_peered":"2024-09-16T05:27:16.155920+0000","last_clean":"2024-09-16T05:27:16.155920+0000","last_became_active":"2024-09-16T05:17:59.866138+0000","last_became_peered":"2024-09-16T05:17:59.866138+0000","last_unstale":"2024-09-16T05:27:16.155920+0000","last_undegraded":"2024-09-16T05:27:16.155920+0000","last_fullsized":"2024-09-16T05:27:16.155920+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:27:09.218880+0000","last_deep_scrub":"182'16","last_deep_scrub_stamp":"2024-09-16T05:21:49.039822+0000","last_clean_scrub_stamp":"2024-09-16T05:27:09.218880+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:19.289696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050573399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.12","version":"35'139","reported_seq":649,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119374+0000","last_change":"2024-09-16T05:26:34.142174+0000","last_active":"2024-09-16T05:27:16.119374+0000","last_peered":"2024-09-16T05:27:16.119374+0000","last_clean":"2024-09-16T05:27:16.119374+0000","last_became_active":"2024-09-16T05:26:34.141618+0000","last_became_peered":"2024-09-16T05:26:34.141618+0000","last_unstale":"2024-09-16T05:27:16.119374+0000","last_undegraded":"2024-09-16T05:27:16.119374+0000","last_fullsized":"2024-09-16T05:27:16.119374+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:28.030079+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:26:28.030079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:38.138428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"272'115","reported_seq":1751,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.026465+0000","last_change":"2024-09-16T05:27:12.910791+0000","last_active":"2024-09-16T05:27:16.026465+0000","last_peered":"2024-09-16T05:27:16.026465+0000","last_clean":"2024-09-16T05:27:16.026465+0000","last_became_active":"2024-09-16T05:21:57.821735+0000","last_became_peered":"2024-09-16T05:21:57.821735+0000","last_unstale":"2024-09-16T05:27:16.026465+0000","last_undegraded":"2024-09-16T05:27:16.026465+0000","last_fullsized":"2024-09-16T05:27:16.026465+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'115","last_scrub_stamp":"2024-09-16T05:26:50.627076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.627076+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:03.851381+0000","scrub_duration":636,"objects_trimmed":0,"snaptrim_duration":0.00033693299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.13","version":"35'139","reported_seq":1387,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488868+0000","last_change":"2024-09-16T05:26:22.907471+0000","last_active":"2024-09-16T05:27:16.488868+0000","last_peered":"2024-09-16T05:27:16.488868+0000","last_clean":"2024-09-16T05:27:16.488868+0000","last_became_active":"2024-09-16T05:26:22.907168+0000","last_became_peered":"2024-09-16T05:26:22.907168+0000","last_unstale":"2024-09-16T05:27:16.488868+0000","last_undegraded":"2024-09-16T05:27:16.488868+0000","last_fullsized":"2024-09-16T05:27:16.488868+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:23:13.594789+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:23:13.594789+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:17.856115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"182'16","reported_seq":4010,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395893+0000","last_change":"2024-09-16T05:27:12.914682+0000","last_active":"2024-09-16T05:27:16.395893+0000","last_peered":"2024-09-16T05:27:16.395893+0000","last_clean":"2024-09-16T05:27:16.395893+0000","last_became_active":"2024-09-16T05:26:22.097579+0000","last_became_peered":"2024-09-16T05:26:22.097579+0000","last_unstale":"2024-09-16T05:27:16.395893+0000","last_undegraded":"2024-09-16T05:27:16.395893+0000","last_fullsized":"2024-09-16T05:27:16.395893+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'16","last_scrub_stamp":"2024-09-16T05:26:31.354872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:31.354872+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:55.611598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078719599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.c","version":"35'139","reported_seq":1675,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119683+0000","last_change":"2024-09-16T05:26:06.379620+0000","last_active":"2024-09-16T05:27:16.119683+0000","last_peered":"2024-09-16T05:27:16.119683+0000","last_clean":"2024-09-16T05:27:16.119683+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:27:16.119683+0000","last_undegraded":"2024-09-16T05:27:16.119683+0000","last_fullsized":"2024-09-16T05:27:16.119683+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:06.379486+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:06.379486+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:34.674699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"1022'526","reported_seq":5044,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395983+0000","last_change":"2024-09-16T05:27:12.919300+0000","last_active":"2024-09-16T05:27:16.395983+0000","last_peered":"2024-09-16T05:27:16.395983+0000","last_clean":"2024-09-16T05:27:16.395983+0000","last_became_active":"2024-09-16T05:16:25.735133+0000","last_became_peered":"2024-09-16T05:16:25.735133+0000","last_unstale":"2024-09-16T05:27:16.395983+0000","last_undegraded":"2024-09-16T05:27:16.395983+0000","last_fullsized":"2024-09-16T05:27:16.395983+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"1013'516","last_scrub_stamp":"2024-09-16T05:27:04.455659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.455659+0000","objects_scrubbed":4,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:13.050387+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.004700287,"stat_sum":{"num_bytes":9496879,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":79,"num_read_kb":84079,"num_write":84,"num_write_kb":30957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.d","version":"35'139","reported_seq":1731,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.004856+0000","last_change":"2024-09-16T05:26:58.976962+0000","last_active":"2024-09-16T05:27:21.004856+0000","last_peered":"2024-09-16T05:27:21.004856+0000","last_clean":"2024-09-16T05:27:21.004856+0000","last_became_active":"2024-09-16T05:26:58.976632+0000","last_became_peered":"2024-09-16T05:26:58.976632+0000","last_unstale":"2024-09-16T05:27:21.004856+0000","last_undegraded":"2024-09-16T05:27:21.004856+0000","last_fullsized":"2024-09-16T05:27:21.004856+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:07.155306+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:07.155306+0000","last_clean_scrub_stamp":"2024-09-16T05:25:07.155306+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"272'232","reported_seq":4603,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988769+0000","last_change":"2024-09-16T05:27:13.273377+0000","last_active":"2024-09-16T05:27:15.988769+0000","last_peered":"2024-09-16T05:27:15.988769+0000","last_clean":"2024-09-16T05:27:15.988769+0000","last_became_active":"2024-09-16T05:16:25.724474+0000","last_became_peered":"2024-09-16T05:16:25.724474+0000","last_unstale":"2024-09-16T05:27:15.988769+0000","last_undegraded":"2024-09-16T05:27:15.988769+0000","last_fullsized":"2024-09-16T05:27:15.988769+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'232","last_scrub_stamp":"2024-09-16T05:25:29.507059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:29.507059+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029619490000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.e","version":"35'139","reported_seq":569,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.005042+0000","last_change":"2024-09-16T05:26:58.979390+0000","last_active":"2024-09-16T05:27:21.005042+0000","last_peered":"2024-09-16T05:27:21.005042+0000","last_clean":"2024-09-16T05:27:21.005042+0000","last_became_active":"2024-09-16T05:26:58.979090+0000","last_became_peered":"2024-09-16T05:26:58.979090+0000","last_unstale":"2024-09-16T05:27:21.005042+0000","last_undegraded":"2024-09-16T05:27:21.005042+0000","last_fullsized":"2024-09-16T05:27:21.005042+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:25:11.119111+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"1021'691","reported_seq":5190,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288331+0000","last_change":"2024-09-16T05:27:12.921798+0000","last_active":"2024-09-16T05:27:16.288331+0000","last_peered":"2024-09-16T05:27:16.288331+0000","last_clean":"2024-09-16T05:27:16.288331+0000","last_became_active":"2024-09-16T05:16:25.729933+0000","last_became_peered":"2024-09-16T05:16:25.729933+0000","last_unstale":"2024-09-16T05:27:16.288331+0000","last_undegraded":"2024-09-16T05:27:16.288331+0000","last_fullsized":"2024-09-16T05:27:16.288331+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'690","last_scrub_stamp":"2024-09-16T05:26:50.454814+0000","last_deep_scrub":"412'459","last_deep_scrub_stamp":"2024-09-16T05:16:02.081612+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.454814+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:53.601583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029533950000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":52781,"num_write":61,"num_write_kb":34652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.f","version":"35'139","reported_seq":1823,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:20.994921+0000","last_change":"2024-09-16T05:26:30.986289+0000","last_active":"2024-09-16T05:27:20.994921+0000","last_peered":"2024-09-16T05:27:20.994921+0000","last_clean":"2024-09-16T05:27:20.994921+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:27:20.994921+0000","last_undegraded":"2024-09-16T05:27:20.994921+0000","last_fullsized":"2024-09-16T05:27:20.994921+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.986186+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:44.208372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"433'272","reported_seq":4751,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899650+0000","last_change":"2024-09-16T05:27:12.913740+0000","last_active":"2024-09-16T05:27:15.899650+0000","last_peered":"2024-09-16T05:27:15.899650+0000","last_clean":"2024-09-16T05:27:15.899650+0000","last_became_active":"2024-09-16T05:16:25.734700+0000","last_became_peered":"2024-09-16T05:16:25.734700+0000","last_unstale":"2024-09-16T05:27:15.899650+0000","last_undegraded":"2024-09-16T05:27:15.899650+0000","last_fullsized":"2024-09-16T05:27:15.899650+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'272","last_scrub_stamp":"2024-09-16T05:25:55.102326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:55.102326+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:20.650626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081007900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.b","version":"35'139","reported_seq":1372,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488715+0000","last_change":"2024-09-16T05:26:25.745603+0000","last_active":"2024-09-16T05:27:16.488715+0000","last_peered":"2024-09-16T05:27:16.488715+0000","last_clean":"2024-09-16T05:27:16.488715+0000","last_became_active":"2024-09-16T05:26:25.117472+0000","last_became_peered":"2024-09-16T05:26:25.117472+0000","last_unstale":"2024-09-16T05:27:16.488715+0000","last_undegraded":"2024-09-16T05:27:16.488715+0000","last_fullsized":"2024-09-16T05:27:16.488715+0000","mapping_epoch":991,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:25.745532+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:26:25.745532+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:46.993488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"1022'409","reported_seq":4826,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.005129+0000","last_change":"2024-09-16T05:27:13.282925+0000","last_active":"2024-09-16T05:27:21.005129+0000","last_peered":"2024-09-16T05:27:21.005129+0000","last_clean":"2024-09-16T05:27:21.005129+0000","last_became_active":"2024-09-16T05:26:59.168243+0000","last_became_peered":"2024-09-16T05:26:59.168243+0000","last_unstale":"2024-09-16T05:27:21.005129+0000","last_undegraded":"2024-09-16T05:27:21.005129+0000","last_fullsized":"2024-09-16T05:27:21.005129+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"893'385","last_scrub_stamp":"2024-09-16T05:24:42.153717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:24:42.153717+0000","objects_scrubbed":2,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.757823+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0074435079999999997,"stat_sum":{"num_bytes":2269275,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":134,"num_read_kb":146776,"num_write":68,"num_write_kb":26467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"1022'374","reported_seq":4885,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988779+0000","last_change":"2024-09-16T05:27:13.274646+0000","last_active":"2024-09-16T05:27:15.988779+0000","last_peered":"2024-09-16T05:27:15.988779+0000","last_clean":"2024-09-16T05:27:15.988779+0000","last_became_active":"2024-09-16T05:26:59.983205+0000","last_became_peered":"2024-09-16T05:26:59.983205+0000","last_unstale":"2024-09-16T05:27:15.988779+0000","last_undegraded":"2024-09-16T05:27:15.988779+0000","last_fullsized":"2024-09-16T05:27:15.988779+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'362","last_scrub_stamp":"2024-09-16T05:25:51.868913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:51.868913+0000","objects_scrubbed":5,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0044430880000000004,"stat_sum":{"num_bytes":5899851,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":165400,"num_write":84,"num_write_kb":42025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":7637594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.0","version":"35'139","reported_seq":1369,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899493+0000","last_change":"2024-09-16T05:25:54.082323+0000","last_active":"2024-09-16T05:27:15.899493+0000","last_peered":"2024-09-16T05:27:15.899493+0000","last_clean":"2024-09-16T05:27:15.899493+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:27:15.899493+0000","last_undegraded":"2024-09-16T05:27:15.899493+0000","last_fullsized":"2024-09-16T05:27:15.899493+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_clean_scrub_stamp":"2024-09-16T05:25:54.082196+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:19.651659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"182'20","reported_seq":3128,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488680+0000","last_change":"2024-09-16T05:27:12.922073+0000","last_active":"2024-09-16T05:27:16.488680+0000","last_peered":"2024-09-16T05:27:16.488680+0000","last_clean":"2024-09-16T05:27:16.488680+0000","last_became_active":"2024-09-16T05:26:43.921685+0000","last_became_peered":"2024-09-16T05:26:43.921685+0000","last_unstale":"2024-09-16T05:27:16.488680+0000","last_undegraded":"2024-09-16T05:27:16.488680+0000","last_fullsized":"2024-09-16T05:27:16.488680+0000","mapping_epoch":1000,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":1001,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:26:25.041922+0000","last_deep_scrub":"182'20","last_deep_scrub_stamp":"2024-09-16T05:17:06.386786+0000","last_clean_scrub_stamp":"2024-09-16T05:26:25.041922+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.524680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056043900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.1","version":"35'139","reported_seq":1599,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.155693+0000","last_change":"2024-09-16T05:26:50.173116+0000","last_active":"2024-09-16T05:27:16.155693+0000","last_peered":"2024-09-16T05:27:16.155693+0000","last_clean":"2024-09-16T05:27:16.155693+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:27:16.155693+0000","last_undegraded":"2024-09-16T05:27:16.155693+0000","last_fullsized":"2024-09-16T05:27:16.155693+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.173026+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.173026+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.105143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":563,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.898813+0000","last_change":"2024-09-16T05:26:50.546912+0000","last_active":"2024-09-16T05:27:15.898813+0000","last_peered":"2024-09-16T05:27:15.898813+0000","last_clean":"2024-09-16T05:27:15.898813+0000","last_became_active":"2024-09-16T05:25:10.425314+0000","last_became_peered":"2024-09-16T05:25:10.425314+0000","last_unstale":"2024-09-16T05:27:15.898813+0000","last_undegraded":"2024-09-16T05:27:15.898813+0000","last_fullsized":"2024-09-16T05:27:15.898813+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.546823+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:18.515049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"178'50","reported_seq":4421,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988129+0000","last_change":"2024-09-16T05:27:13.270197+0000","last_active":"2024-09-16T05:27:15.988129+0000","last_peered":"2024-09-16T05:27:15.988129+0000","last_clean":"2024-09-16T05:27:15.988129+0000","last_became_active":"2024-09-16T05:14:40.261174+0000","last_became_peered":"2024-09-16T05:14:40.261174+0000","last_unstale":"2024-09-16T05:27:15.988129+0000","last_undegraded":"2024-09-16T05:27:15.988129+0000","last_fullsized":"2024-09-16T05:27:15.988129+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:25:51.423864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:51.423864+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064073899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.3","version":"35'139","reported_seq":2353,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119436+0000","last_change":"2024-09-16T05:26:30.269822+0000","last_active":"2024-09-16T05:27:16.119436+0000","last_peered":"2024-09-16T05:27:16.119436+0000","last_clean":"2024-09-16T05:27:16.119436+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:27:16.119436+0000","last_undegraded":"2024-09-16T05:27:16.119436+0000","last_fullsized":"2024-09-16T05:27:16.119436+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.269705+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:27.403439+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.269705+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:37.394250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"205'33","reported_seq":4422,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.004692+0000","last_change":"2024-09-16T05:27:13.275701+0000","last_active":"2024-09-16T05:27:21.004692+0000","last_peered":"2024-09-16T05:27:21.004692+0000","last_clean":"2024-09-16T05:27:21.004692+0000","last_became_active":"2024-09-16T05:26:58.977558+0000","last_became_peered":"2024-09-16T05:26:58.977558+0000","last_unstale":"2024-09-16T05:27:21.004692+0000","last_undegraded":"2024-09-16T05:27:21.004692+0000","last_fullsized":"2024-09-16T05:27:21.004692+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'33","last_scrub_stamp":"2024-09-16T05:25:03.194302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:03.194302+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.757863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000923283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3724,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288500+0000","last_change":"2024-09-16T05:27:12.922093+0000","last_active":"2024-09-16T05:27:16.288500+0000","last_peered":"2024-09-16T05:27:16.288500+0000","last_clean":"2024-09-16T05:27:16.288500+0000","last_became_active":"2024-09-16T05:21:54.939117+0000","last_became_peered":"2024-09-16T05:21:54.939117+0000","last_unstale":"2024-09-16T05:27:16.288500+0000","last_undegraded":"2024-09-16T05:27:16.288500+0000","last_fullsized":"2024-09-16T05:27:16.288500+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:25:31.180506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:58.088776+0000","last_clean_scrub_stamp":"2024-09-16T05:25:31.180506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029498179999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.4","version":"35'139","reported_seq":2316,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899774+0000","last_change":"2024-09-16T05:27:01.775171+0000","last_active":"2024-09-16T05:27:15.899774+0000","last_peered":"2024-09-16T05:27:15.899774+0000","last_clean":"2024-09-16T05:27:15.899774+0000","last_became_active":"2024-09-16T05:26:58.978042+0000","last_became_peered":"2024-09-16T05:26:58.978042+0000","last_unstale":"2024-09-16T05:27:15.899774+0000","last_undegraded":"2024-09-16T05:27:15.899774+0000","last_fullsized":"2024-09-16T05:27:15.899774+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:01.775049+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:01.775049+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:28.973729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"116'9","reported_seq":2906,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988737+0000","last_change":"2024-09-16T05:27:13.273135+0000","last_active":"2024-09-16T05:27:15.988737+0000","last_peered":"2024-09-16T05:27:15.988737+0000","last_clean":"2024-09-16T05:27:15.988737+0000","last_became_active":"2024-09-16T05:21:21.816054+0000","last_became_peered":"2024-09-16T05:21:21.816054+0000","last_unstale":"2024-09-16T05:27:15.988737+0000","last_undegraded":"2024-09-16T05:27:15.988737+0000","last_fullsized":"2024-09-16T05:27:15.988737+0000","mapping_epoch":731,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:26:47.249023+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:21:08.532148+0000","last_clean_scrub_stamp":"2024-09-16T05:26:47.249023+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:51.057203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029035340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.5","version":"35'139","reported_seq":1620,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156339+0000","last_change":"2024-09-16T05:26:32.280532+0000","last_active":"2024-09-16T05:27:16.156339+0000","last_peered":"2024-09-16T05:27:16.156339+0000","last_clean":"2024-09-16T05:27:16.156339+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:27:16.156339+0000","last_undegraded":"2024-09-16T05:27:16.156339+0000","last_fullsized":"2024-09-16T05:27:16.156339+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:32.280436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.280436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:45.243320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"1022'745","reported_seq":5263,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.396244+0000","last_change":"2024-09-16T05:27:12.916859+0000","last_active":"2024-09-16T05:27:16.396244+0000","last_peered":"2024-09-16T05:27:16.396244+0000","last_clean":"2024-09-16T05:27:16.396244+0000","last_became_active":"2024-09-16T05:18:46.769028+0000","last_became_peered":"2024-09-16T05:18:46.769028+0000","last_unstale":"2024-09-16T05:27:16.396244+0000","last_undegraded":"2024-09-16T05:27:16.396244+0000","last_fullsized":"2024-09-16T05:27:16.396244+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'735","last_scrub_stamp":"2024-09-16T05:26:10.675384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:10.675384+0000","objects_scrubbed":4,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:18.738306+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.002668279,"stat_sum":{"num_bytes":6319960,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":187,"num_read_kb":207594,"num_write":72,"num_write_kb":23912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.6","version":"35'139","reported_seq":1564,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156417+0000","last_change":"2024-09-16T05:26:52.226627+0000","last_active":"2024-09-16T05:27:16.156417+0000","last_peered":"2024-09-16T05:27:16.156417+0000","last_clean":"2024-09-16T05:27:16.156417+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:27:16.156417+0000","last_undegraded":"2024-09-16T05:27:16.156417+0000","last_fullsized":"2024-09-16T05:27:16.156417+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:52.226532+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:52.226532+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:08.001945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"182'89","reported_seq":1107,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.489110+0000","last_change":"2024-09-16T05:27:12.922758+0000","last_active":"2024-09-16T05:27:16.489110+0000","last_peered":"2024-09-16T05:27:16.489110+0000","last_clean":"2024-09-16T05:27:16.489110+0000","last_became_active":"2024-09-16T05:26:22.104516+0000","last_became_peered":"2024-09-16T05:26:22.104516+0000","last_unstale":"2024-09-16T05:27:16.489110+0000","last_undegraded":"2024-09-16T05:27:16.489110+0000","last_fullsized":"2024-09-16T05:27:16.489110+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:23:17.624562+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:14:16.554008+0000","last_clean_scrub_stamp":"2024-09-16T05:23:17.624562+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094812599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.7","version":"35'139","reported_seq":1610,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156580+0000","last_change":"2024-09-16T05:26:30.310092+0000","last_active":"2024-09-16T05:27:16.156580+0000","last_peered":"2024-09-16T05:27:16.156580+0000","last_clean":"2024-09-16T05:27:16.156580+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:27:16.156580+0000","last_undegraded":"2024-09-16T05:27:16.156580+0000","last_fullsized":"2024-09-16T05:27:16.156580+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.309950+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.309950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.005293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.b","version":"244'177","reported_seq":4666,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395871+0000","last_change":"2024-09-16T05:27:12.913679+0000","last_active":"2024-09-16T05:27:16.395871+0000","last_peered":"2024-09-16T05:27:16.395871+0000","last_clean":"2024-09-16T05:27:16.395871+0000","last_became_active":"2024-09-16T05:14:25.831177+0000","last_became_peered":"2024-09-16T05:14:25.831177+0000","last_unstale":"2024-09-16T05:27:16.395871+0000","last_undegraded":"2024-09-16T05:27:16.395871+0000","last_fullsized":"2024-09-16T05:27:16.395871+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'177","last_scrub_stamp":"2024-09-16T05:27:02.460776+0000","last_deep_scrub":"244'177","last_deep_scrub_stamp":"2024-09-16T05:15:41.633260+0000","last_clean_scrub_stamp":"2024-09-16T05:27:02.460776+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:04.137384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044981900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.8","version":"35'139","reported_seq":474,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988272+0000","last_change":"2024-09-16T05:26:45.218035+0000","last_active":"2024-09-16T05:27:15.988272+0000","last_peered":"2024-09-16T05:27:15.988272+0000","last_clean":"2024-09-16T05:27:15.988272+0000","last_became_active":"2024-09-16T05:26:34.139432+0000","last_became_peered":"2024-09-16T05:26:34.139432+0000","last_unstale":"2024-09-16T05:27:15.988272+0000","last_undegraded":"2024-09-16T05:27:15.988272+0000","last_fullsized":"2024-09-16T05:27:15.988272+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:45.217959+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:11.129971+0000","last_clean_scrub_stamp":"2024-09-16T05:26:45.217959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:56.052590+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"1022'792","reported_seq":5326,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899777+0000","last_change":"2024-09-16T05:27:12.916058+0000","last_active":"2024-09-16T05:27:15.899777+0000","last_peered":"2024-09-16T05:27:15.899777+0000","last_clean":"2024-09-16T05:27:15.899777+0000","last_became_active":"2024-09-16T05:14:25.825200+0000","last_became_peered":"2024-09-16T05:14:25.825200+0000","last_unstale":"2024-09-16T05:27:15.899777+0000","last_undegraded":"2024-09-16T05:27:15.899777+0000","last_fullsized":"2024-09-16T05:27:15.899777+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'780","last_scrub_stamp":"2024-09-16T05:26:07.856681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:07.856681+0000","objects_scrubbed":4,"log_size":792,"log_dups_size":0,"ondisk_log_size":792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:20.650604+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0030568869999999999,"stat_sum":{"num_bytes":6387254,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":161045,"num_write":109,"num_write_kb":46705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.9","version":"35'139","reported_seq":2211,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.004671+0000","last_change":"2024-09-16T05:26:59.168309+0000","last_active":"2024-09-16T05:27:21.004671+0000","last_peered":"2024-09-16T05:27:21.004671+0000","last_clean":"2024-09-16T05:27:21.004671+0000","last_became_active":"2024-09-16T05:26:59.168051+0000","last_became_peered":"2024-09-16T05:26:59.168051+0000","last_unstale":"2024-09-16T05:27:21.004671+0000","last_undegraded":"2024-09-16T05:27:21.004671+0000","last_fullsized":"2024-09-16T05:27:21.004671+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:24:51.176377+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:24:51.176377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"245'220","reported_seq":2584,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488683+0000","last_change":"2024-09-16T05:27:12.922058+0000","last_active":"2024-09-16T05:27:16.488683+0000","last_peered":"2024-09-16T05:27:16.488683+0000","last_clean":"2024-09-16T05:27:16.488683+0000","last_became_active":"2024-09-16T05:26:22.099840+0000","last_became_peered":"2024-09-16T05:26:22.099840+0000","last_unstale":"2024-09-16T05:27:16.488683+0000","last_undegraded":"2024-09-16T05:27:16.488683+0000","last_fullsized":"2024-09-16T05:27:16.488683+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'220","last_scrub_stamp":"2024-09-16T05:23:07.627317+0000","last_deep_scrub":"184'70","last_deep_scrub_stamp":"2024-09-16T05:11:56.116608+0000","last_clean_scrub_stamp":"2024-09-16T05:23:07.627317+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000530139,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.a","version":"35'139","reported_seq":2353,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288096+0000","last_change":"2024-09-16T05:26:32.294013+0000","last_active":"2024-09-16T05:27:16.288096+0000","last_peered":"2024-09-16T05:27:16.288096+0000","last_clean":"2024-09-16T05:27:16.288096+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:27:16.288096+0000","last_undegraded":"2024-09-16T05:27:16.288096+0000","last_fullsized":"2024-09-16T05:27:16.288096+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:32.293931+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.293931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:49.584816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1830,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.898672+0000","last_change":"2024-09-16T05:26:11.511232+0000","last_active":"2024-09-16T05:27:15.898672+0000","last_peered":"2024-09-16T05:27:15.898672+0000","last_clean":"2024-09-16T05:27:15.898672+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:27:15.898672+0000","last_undegraded":"2024-09-16T05:27:15.898672+0000","last_fullsized":"2024-09-16T05:27:15.898672+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:11.511127+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:26:11.511127+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:20.421428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"272'69","reported_seq":2388,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.155898+0000","last_change":"2024-09-16T05:27:12.927619+0000","last_active":"2024-09-16T05:27:16.155898+0000","last_peered":"2024-09-16T05:27:16.155898+0000","last_clean":"2024-09-16T05:27:16.155898+0000","last_became_active":"2024-09-16T05:17:59.867036+0000","last_became_peered":"2024-09-16T05:17:59.867036+0000","last_unstale":"2024-09-16T05:27:16.155898+0000","last_undegraded":"2024-09-16T05:27:16.155898+0000","last_fullsized":"2024-09-16T05:27:16.155898+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'69","last_scrub_stamp":"2024-09-16T05:26:11.051655+0000","last_deep_scrub":"272'69","last_deep_scrub_stamp":"2024-09-16T05:24:21.697507+0000","last_clean_scrub_stamp":"2024-09-16T05:26:11.051655+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:23.161132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029102989999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.10","version":"35'139","reported_seq":1283,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988227+0000","last_change":"2024-09-16T05:26:49.297470+0000","last_active":"2024-09-16T05:27:15.988227+0000","last_peered":"2024-09-16T05:27:15.988227+0000","last_clean":"2024-09-16T05:27:15.988227+0000","last_became_active":"2024-09-16T05:26:34.137623+0000","last_became_peered":"2024-09-16T05:26:34.137623+0000","last_unstale":"2024-09-16T05:27:15.988227+0000","last_undegraded":"2024-09-16T05:27:15.988227+0000","last_fullsized":"2024-09-16T05:27:15.988227+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:49.297389+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.297389+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:51.481299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"181'15","reported_seq":4099,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288338+0000","last_change":"2024-09-16T05:27:12.921521+0000","last_active":"2024-09-16T05:27:16.288338+0000","last_peered":"2024-09-16T05:27:16.288338+0000","last_clean":"2024-09-16T05:27:16.288338+0000","last_became_active":"2024-09-16T05:16:25.730077+0000","last_became_peered":"2024-09-16T05:16:25.730077+0000","last_unstale":"2024-09-16T05:27:16.288338+0000","last_undegraded":"2024-09-16T05:27:16.288338+0000","last_fullsized":"2024-09-16T05:27:16.288338+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'15","last_scrub_stamp":"2024-09-16T05:25:26.201601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:26.201601+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002776864,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.17","version":"35'139","reported_seq":1228,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395845+0000","last_change":"2024-09-16T05:26:06.657362+0000","last_active":"2024-09-16T05:27:16.395845+0000","last_peered":"2024-09-16T05:27:16.395845+0000","last_clean":"2024-09-16T05:27:16.395845+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:27:16.395845+0000","last_undegraded":"2024-09-16T05:27:16.395845+0000","last_fullsized":"2024-09-16T05:27:16.395845+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:06.657290+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:26:06.657290+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.783373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"433'289","reported_seq":4385,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288306+0000","last_change":"2024-09-16T05:27:12.921272+0000","last_active":"2024-09-16T05:27:16.288306+0000","last_peered":"2024-09-16T05:27:16.288306+0000","last_clean":"2024-09-16T05:27:16.288306+0000","last_became_active":"2024-09-16T05:21:54.931954+0000","last_became_peered":"2024-09-16T05:21:54.931954+0000","last_unstale":"2024-09-16T05:27:16.288306+0000","last_undegraded":"2024-09-16T05:27:16.288306+0000","last_fullsized":"2024-09-16T05:27:16.288306+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"433'289","last_scrub_stamp":"2024-09-16T05:26:05.083517+0000","last_deep_scrub":"342'239","last_deep_scrub_stamp":"2024-09-16T05:14:52.110587+0000","last_clean_scrub_stamp":"2024-09-16T05:26:05.083517+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:26.340055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025913619999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.16","version":"35'139","reported_seq":1280,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988519+0000","last_change":"2024-09-16T05:26:29.217364+0000","last_active":"2024-09-16T05:27:15.988519+0000","last_peered":"2024-09-16T05:27:15.988519+0000","last_clean":"2024-09-16T05:27:15.988519+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:27:15.988519+0000","last_undegraded":"2024-09-16T05:27:15.988519+0000","last_fullsized":"2024-09-16T05:27:15.988519+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:29.217292+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:26:29.217292+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.502835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.15","version":"1022'673","reported_seq":5191,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288504+0000","last_change":"2024-09-16T05:27:12.921530+0000","last_active":"2024-09-16T05:27:16.288504+0000","last_peered":"2024-09-16T05:27:16.288504+0000","last_clean":"2024-09-16T05:27:16.288504+0000","last_became_active":"2024-09-16T05:26:22.105598+0000","last_became_peered":"2024-09-16T05:26:22.105598+0000","last_unstale":"2024-09-16T05:27:16.288504+0000","last_undegraded":"2024-09-16T05:27:16.288504+0000","last_fullsized":"2024-09-16T05:27:16.288504+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"989'663","last_scrub_stamp":"2024-09-16T05:26:28.296623+0000","last_deep_scrub":"760'658","last_deep_scrub_stamp":"2024-09-16T05:22:05.847540+0000","last_clean_scrub_stamp":"2024-09-16T05:26:28.296623+0000","objects_scrubbed":2,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:29.249888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029162720000000001,"stat_sum":{"num_bytes":4276341,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":64,"num_read_kb":56868,"num_write":32,"num_write_kb":18758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":7118573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.15","version":"35'139","reported_seq":662,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395896+0000","last_change":"2024-09-16T05:26:49.474733+0000","last_active":"2024-09-16T05:27:16.395896+0000","last_peered":"2024-09-16T05:27:16.395896+0000","last_clean":"2024-09-16T05:27:16.395896+0000","last_became_active":"2024-09-16T05:26:34.243844+0000","last_became_peered":"2024-09-16T05:26:34.243844+0000","last_unstale":"2024-09-16T05:27:16.395896+0000","last_undegraded":"2024-09-16T05:27:16.395896+0000","last_fullsized":"2024-09-16T05:27:16.395896+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:49.474636+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.474636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:51.292318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"733'334","reported_seq":4463,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899825+0000","last_change":"2024-09-16T05:27:12.914119+0000","last_active":"2024-09-16T05:27:15.899825+0000","last_peered":"2024-09-16T05:27:15.899825+0000","last_clean":"2024-09-16T05:27:15.899825+0000","last_became_active":"2024-09-16T05:26:36.901050+0000","last_became_peered":"2024-09-16T05:26:36.901050+0000","last_unstale":"2024-09-16T05:27:15.899825+0000","last_undegraded":"2024-09-16T05:27:15.899825+0000","last_fullsized":"2024-09-16T05:27:15.899825+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'334","last_scrub_stamp":"2024-09-16T05:25:28.832412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:28.832412+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:20.650580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001038857,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.14","version":"35'139","reported_seq":1662,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988398+0000","last_change":"2024-09-16T05:26:05.401014+0000","last_active":"2024-09-16T05:27:15.988398+0000","last_peered":"2024-09-16T05:27:15.988398+0000","last_clean":"2024-09-16T05:27:15.988398+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:27:15.988398+0000","last_undegraded":"2024-09-16T05:27:15.988398+0000","last_fullsized":"2024-09-16T05:27:15.988398+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:05.400920+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:05.400920+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1126,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.488882+0000","last_change":"2024-09-16T05:27:12.922529+0000","last_active":"2024-09-16T05:27:16.488882+0000","last_peered":"2024-09-16T05:27:16.488882+0000","last_clean":"2024-09-16T05:27:16.488882+0000","last_became_active":"2024-09-16T05:26:22.102874+0000","last_became_peered":"2024-09-16T05:26:22.102874+0000","last_unstale":"2024-09-16T05:27:16.488882+0000","last_undegraded":"2024-09-16T05:27:16.488882+0000","last_fullsized":"2024-09-16T05:27:16.488882+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:26:27.800711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:22:36.831827+0000","last_clean_scrub_stamp":"2024-09-16T05:26:27.800711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:52.275017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000907714,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.1b","version":"35'139","reported_seq":1753,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395685+0000","last_change":"2024-09-16T05:26:33.343104+0000","last_active":"2024-09-16T05:27:16.395685+0000","last_peered":"2024-09-16T05:27:16.395685+0000","last_clean":"2024-09-16T05:27:16.395685+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:27:16.395685+0000","last_undegraded":"2024-09-16T05:27:16.395685+0000","last_fullsized":"2024-09-16T05:27:16.395685+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:33.343016+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:26:33.343016+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:59.079256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.18","version":"181'57","reported_seq":4500,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288082+0000","last_change":"2024-09-16T05:27:12.920526+0000","last_active":"2024-09-16T05:27:16.288082+0000","last_peered":"2024-09-16T05:27:16.288082+0000","last_clean":"2024-09-16T05:27:16.288082+0000","last_became_active":"2024-09-16T05:21:27.832312+0000","last_became_peered":"2024-09-16T05:21:27.832312+0000","last_unstale":"2024-09-16T05:27:16.288082+0000","last_undegraded":"2024-09-16T05:27:16.288082+0000","last_fullsized":"2024-09-16T05:27:16.288082+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'57","last_scrub_stamp":"2024-09-16T05:27:03.357209+0000","last_deep_scrub":"181'57","last_deep_scrub_stamp":"2024-09-16T05:24:05.187021+0000","last_clean_scrub_stamp":"2024-09-16T05:27:03.357209+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:06.202777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001913908,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.1a","version":"35'139","reported_seq":462,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.004683+0000","last_change":"2024-09-16T05:27:05.725386+0000","last_active":"2024-09-16T05:27:21.004683+0000","last_peered":"2024-09-16T05:27:21.004683+0000","last_clean":"2024-09-16T05:27:21.004683+0000","last_became_active":"2024-09-16T05:26:58.979664+0000","last_became_peered":"2024-09-16T05:26:58.979664+0000","last_unstale":"2024-09-16T05:27:21.004683+0000","last_undegraded":"2024-09-16T05:27:21.004683+0000","last_fullsized":"2024-09-16T05:27:21.004683+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:05.725288+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.725288+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:11.443926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"182'60","reported_seq":4105,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119169+0000","last_change":"2024-09-16T05:27:12.910283+0000","last_active":"2024-09-16T05:27:16.119169+0000","last_peered":"2024-09-16T05:27:16.119169+0000","last_clean":"2024-09-16T05:27:16.119169+0000","last_became_active":"2024-09-16T05:21:54.930126+0000","last_became_peered":"2024-09-16T05:21:54.930126+0000","last_unstale":"2024-09-16T05:27:16.119169+0000","last_undegraded":"2024-09-16T05:27:16.119169+0000","last_fullsized":"2024-09-16T05:27:16.119169+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"182'60","last_scrub_stamp":"2024-09-16T05:25:25.499551+0000","last_deep_scrub":"182'60","last_deep_scrub_stamp":"2024-09-16T05:21:29.356695+0000","last_clean_scrub_stamp":"2024-09-16T05:25:25.499551+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.118180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047423499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.19","version":"35'139","reported_seq":2344,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.155756+0000","last_change":"2024-09-16T05:26:46.145954+0000","last_active":"2024-09-16T05:27:16.155756+0000","last_peered":"2024-09-16T05:27:16.155756+0000","last_clean":"2024-09-16T05:27:16.155756+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:27:16.155756+0000","last_undegraded":"2024-09-16T05:27:16.155756+0000","last_fullsized":"2024-09-16T05:27:16.155756+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:46.145861+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:26:46.145861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.153358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1a","version":"1022'1395","reported_seq":6022,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.396180+0000","last_change":"2024-09-16T05:27:12.919066+0000","last_active":"2024-09-16T05:27:16.396180+0000","last_peered":"2024-09-16T05:27:16.396180+0000","last_clean":"2024-09-16T05:27:16.396180+0000","last_became_active":"2024-09-16T05:21:54.943140+0000","last_became_peered":"2024-09-16T05:21:54.943140+0000","last_unstale":"2024-09-16T05:27:16.396180+0000","last_undegraded":"2024-09-16T05:27:16.396180+0000","last_fullsized":"2024-09-16T05:27:16.396180+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"968'1365","last_scrub_stamp":"2024-09-16T05:25:32.575481+0000","last_deep_scrub":"582'782","last_deep_scrub_stamp":"2024-09-16T05:18:52.154075+0000","last_clean_scrub_stamp":"2024-09-16T05:25:32.575481+0000","objects_scrubbed":6,"log_size":1395,"log_dups_size":0,"ondisk_log_size":1395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0041454660000000004,"stat_sum":{"num_bytes":20707841,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":99,"num_read_kb":121277,"num_write":76,"num_write_kb":29600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":19652756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.18","version":"35'139","reported_seq":481,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988004+0000","last_change":"2024-09-16T05:26:59.981831+0000","last_active":"2024-09-16T05:27:15.988004+0000","last_peered":"2024-09-16T05:27:15.988004+0000","last_clean":"2024-09-16T05:27:15.988004+0000","last_became_active":"2024-09-16T05:26:59.978354+0000","last_became_peered":"2024-09-16T05:26:59.978354+0000","last_unstale":"2024-09-16T05:27:15.988004+0000","last_undegraded":"2024-09-16T05:27:15.988004+0000","last_fullsized":"2024-09-16T05:27:15.988004+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:46.234400+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:46.234400+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.950250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"1018'761","reported_seq":5303,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.899678+0000","last_change":"2024-09-16T05:27:12.915232+0000","last_active":"2024-09-16T05:27:15.899678+0000","last_peered":"2024-09-16T05:27:15.899678+0000","last_clean":"2024-09-16T05:27:15.899678+0000","last_became_active":"2024-09-16T05:21:27.832003+0000","last_became_peered":"2024-09-16T05:21:27.832003+0000","last_unstale":"2024-09-16T05:27:15.899678+0000","last_undegraded":"2024-09-16T05:27:15.899678+0000","last_fullsized":"2024-09-16T05:27:15.899678+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"1011'759","last_scrub_stamp":"2024-09-16T05:27:04.680556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.680556+0000","objects_scrubbed":3,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:18.107200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019496050000000001,"stat_sum":{"num_bytes":3765032,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":51,"num_read_kb":48936,"num_write":30,"num_write_kb":13241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.1c","version":"723'472","reported_seq":4660,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.155862+0000","last_change":"2024-09-16T05:27:12.924598+0000","last_active":"2024-09-16T05:27:16.155862+0000","last_peered":"2024-09-16T05:27:16.155862+0000","last_clean":"2024-09-16T05:27:16.155862+0000","last_became_active":"2024-09-16T05:26:58.979257+0000","last_became_peered":"2024-09-16T05:26:58.979257+0000","last_unstale":"2024-09-16T05:27:16.155862+0000","last_undegraded":"2024-09-16T05:27:16.155862+0000","last_fullsized":"2024-09-16T05:27:16.155862+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":5,"last_scrub":"723'472","last_scrub_stamp":"2024-09-16T05:27:06.199657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.199657+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:07.842039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043025300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.1e","version":"35'139","reported_seq":1836,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.900033+0000","last_change":"2024-09-16T05:26:50.779260+0000","last_active":"2024-09-16T05:27:15.900033+0000","last_peered":"2024-09-16T05:27:15.900033+0000","last_clean":"2024-09-16T05:27:15.900033+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:27:15.900033+0000","last_undegraded":"2024-09-16T05:27:15.900033+0000","last_fullsized":"2024-09-16T05:27:15.900033+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.779169+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.779169+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:20.159044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1d","version":"1018'1349","reported_seq":6077,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156298+0000","last_change":"2024-09-16T05:27:12.926977+0000","last_active":"2024-09-16T05:27:16.156298+0000","last_peered":"2024-09-16T05:27:16.156298+0000","last_clean":"2024-09-16T05:27:16.156298+0000","last_became_active":"2024-09-16T05:26:58.979187+0000","last_became_peered":"2024-09-16T05:26:58.979187+0000","last_unstale":"2024-09-16T05:27:16.156298+0000","last_undegraded":"2024-09-16T05:27:16.156298+0000","last_fullsized":"2024-09-16T05:27:16.156298+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":5,"last_scrub":"947'1347","last_scrub_stamp":"2024-09-16T05:27:04.182763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:04.182763+0000","objects_scrubbed":2,"log_size":1349,"log_dups_size":0,"ondisk_log_size":1349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:20.428173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024181369999999999,"stat_sum":{"num_bytes":2151560,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":27,"num_read_kb":19367,"num_write":20,"num_write_kb":8848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.1d","version":"35'139","reported_seq":891,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.489223+0000","last_change":"2024-09-16T05:27:06.911321+0000","last_active":"2024-09-16T05:27:16.489223+0000","last_peered":"2024-09-16T05:27:16.489223+0000","last_clean":"2024-09-16T05:27:16.489223+0000","last_became_active":"2024-09-16T05:26:22.907190+0000","last_became_peered":"2024-09-16T05:26:22.907190+0000","last_unstale":"2024-09-16T05:27:16.489223+0000","last_undegraded":"2024-09-16T05:27:16.489223+0000","last_fullsized":"2024-09-16T05:27:16.489223+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:06.911213+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.911213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:31.723471+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"736'512","reported_seq":5038,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119859+0000","last_change":"2024-09-16T05:27:12.911934+0000","last_active":"2024-09-16T05:27:16.119859+0000","last_peered":"2024-09-16T05:27:16.119859+0000","last_clean":"2024-09-16T05:27:16.119859+0000","last_became_active":"2024-09-16T05:21:27.826212+0000","last_became_peered":"2024-09-16T05:21:27.826212+0000","last_unstale":"2024-09-16T05:27:16.119859+0000","last_undegraded":"2024-09-16T05:27:16.119859+0000","last_fullsized":"2024-09-16T05:27:16.119859+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'512","last_scrub_stamp":"2024-09-16T05:27:02.147723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:02.147723+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:13.120747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001363244,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"1.1c","version":"35'139","reported_seq":1083,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.395969+0000","last_change":"2024-09-16T05:26:27.406964+0000","last_active":"2024-09-16T05:27:16.395969+0000","last_peered":"2024-09-16T05:27:16.395969+0000","last_clean":"2024-09-16T05:27:16.395969+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:27:16.395969+0000","last_undegraded":"2024-09-16T05:27:16.395969+0000","last_fullsized":"2024-09-16T05:27:16.395969+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:27.406853+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:27.406853+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:38.132020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1f","version":"284'221","reported_seq":4716,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.898992+0000","last_change":"2024-09-16T05:27:12.917492+0000","last_active":"2024-09-16T05:27:15.898992+0000","last_peered":"2024-09-16T05:27:15.898992+0000","last_clean":"2024-09-16T05:27:15.898992+0000","last_became_active":"2024-09-16T05:26:58.979173+0000","last_became_peered":"2024-09-16T05:26:58.979173+0000","last_unstale":"2024-09-16T05:27:15.898992+0000","last_undegraded":"2024-09-16T05:27:15.898992+0000","last_fullsized":"2024-09-16T05:27:15.898992+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'221","last_scrub_stamp":"2024-09-16T05:27:10.425611+0000","last_deep_scrub":"284'221","last_deep_scrub_stamp":"2024-09-16T05:16:21.634458+0000","last_clean_scrub_stamp":"2024-09-16T05:27:10.425611+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:26.690957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00038351200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.20","version":"1022'413","reported_seq":4859,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.004803+0000","last_change":"2024-09-16T05:27:13.279749+0000","last_active":"2024-09-16T05:27:21.004803+0000","last_peered":"2024-09-16T05:27:21.004803+0000","last_clean":"2024-09-16T05:27:21.004803+0000","last_became_active":"2024-09-16T05:26:59.977588+0000","last_became_peered":"2024-09-16T05:26:59.977588+0000","last_unstale":"2024-09-16T05:27:21.004803+0000","last_undegraded":"2024-09-16T05:27:21.004803+0000","last_fullsized":"2024-09-16T05:27:21.004803+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":6,"last_scrub":"1003'394","last_scrub_stamp":"2024-09-16T05:26:49.054224+0000","last_deep_scrub":"1003'394","last_deep_scrub_stamp":"2024-09-16T05:26:49.054224+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.054224+0000","objects_scrubbed":2,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:07.001455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0049802379999999997,"stat_sum":{"num_bytes":5830182,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":188,"num_read_kb":215628,"num_write":91,"num_write_kb":36785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1952796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"178'50","reported_seq":4420,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988168+0000","last_change":"2024-09-16T05:27:13.272511+0000","last_active":"2024-09-16T05:27:15.988168+0000","last_peered":"2024-09-16T05:27:15.988168+0000","last_clean":"2024-09-16T05:27:15.988168+0000","last_became_active":"2024-09-16T05:14:39.917856+0000","last_became_peered":"2024-09-16T05:14:39.917856+0000","last_unstale":"2024-09-16T05:27:15.988168+0000","last_undegraded":"2024-09-16T05:27:15.988168+0000","last_fullsized":"2024-09-16T05:27:15.988168+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"178'50","last_scrub_stamp":"2024-09-16T05:25:33.497085+0000","last_deep_scrub":"178'50","last_deep_scrub_stamp":"2024-09-16T05:14:52.848788+0000","last_clean_scrub_stamp":"2024-09-16T05:25:33.497085+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027002020000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.22","version":"182'20","reported_seq":2394,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.156031+0000","last_change":"2024-09-16T05:27:12.924986+0000","last_active":"2024-09-16T05:27:16.156031+0000","last_peered":"2024-09-16T05:27:16.156031+0000","last_clean":"2024-09-16T05:27:16.156031+0000","last_became_active":"2024-09-16T05:21:54.942985+0000","last_became_peered":"2024-09-16T05:21:54.942985+0000","last_unstale":"2024-09-16T05:27:16.156031+0000","last_undegraded":"2024-09-16T05:27:16.156031+0000","last_fullsized":"2024-09-16T05:27:16.156031+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'20","last_scrub_stamp":"2024-09-16T05:26:06.022962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:06.022962+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078223799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.23","version":"184'17","reported_seq":3864,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.119594+0000","last_change":"2024-09-16T05:27:12.911531+0000","last_active":"2024-09-16T05:27:16.119594+0000","last_peered":"2024-09-16T05:27:16.119594+0000","last_clean":"2024-09-16T05:27:16.119594+0000","last_became_active":"2024-09-16T05:26:22.099313+0000","last_became_peered":"2024-09-16T05:26:22.099313+0000","last_unstale":"2024-09-16T05:27:16.119594+0000","last_undegraded":"2024-09-16T05:27:16.119594+0000","last_fullsized":"2024-09-16T05:27:16.119594+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":6,"last_scrub":"184'17","last_scrub_stamp":"2024-09-16T05:26:25.314478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:25.314478+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:36.138941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011818309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.24","version":"182'89","reported_seq":3781,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288170+0000","last_change":"2024-09-16T05:27:12.920962+0000","last_active":"2024-09-16T05:27:16.288170+0000","last_peered":"2024-09-16T05:27:16.288170+0000","last_clean":"2024-09-16T05:27:16.288170+0000","last_became_active":"2024-09-16T05:21:54.932630+0000","last_became_peered":"2024-09-16T05:21:54.932630+0000","last_unstale":"2024-09-16T05:27:16.288170+0000","last_undegraded":"2024-09-16T05:27:16.288170+0000","last_fullsized":"2024-09-16T05:27:16.288170+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":6,"last_scrub":"182'89","last_scrub_stamp":"2024-09-16T05:25:54.100521+0000","last_deep_scrub":"182'89","last_deep_scrub_stamp":"2024-09-16T05:15:59.030307+0000","last_clean_scrub_stamp":"2024-09-16T05:25:54.100521+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:22.883036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023293630000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.25","version":"1022'1463","reported_seq":5873,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.396216+0000","last_change":"2024-09-16T05:27:12.917129+0000","last_active":"2024-09-16T05:27:16.396216+0000","last_peered":"2024-09-16T05:27:16.396216+0000","last_clean":"2024-09-16T05:27:16.396216+0000","last_became_active":"2024-09-16T05:13:04.678406+0000","last_became_peered":"2024-09-16T05:13:04.678406+0000","last_unstale":"2024-09-16T05:27:16.396216+0000","last_undegraded":"2024-09-16T05:27:16.396216+0000","last_fullsized":"2024-09-16T05:27:16.396216+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"968'1445","last_scrub_stamp":"2024-09-16T05:26:04.609570+0000","last_deep_scrub":"622'977","last_deep_scrub_stamp":"2024-09-16T05:19:32.332491+0000","last_clean_scrub_stamp":"2024-09-16T05:26:04.609570+0000","objects_scrubbed":8,"log_size":1463,"log_dups_size":0,"ondisk_log_size":1463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0026095409999999999,"stat_sum":{"num_bytes":21505698,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":325,"num_read_kb":318983,"num_write":266,"num_write_kb":110467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.26","version":"116'9","reported_seq":3678,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.005317+0000","last_change":"2024-09-16T05:27:13.280584+0000","last_active":"2024-09-16T05:27:21.005317+0000","last_peered":"2024-09-16T05:27:21.005317+0000","last_clean":"2024-09-16T05:27:21.005317+0000","last_became_active":"2024-09-16T05:26:58.979695+0000","last_became_peered":"2024-09-16T05:26:58.979695+0000","last_unstale":"2024-09-16T05:27:21.005317+0000","last_undegraded":"2024-09-16T05:27:21.005317+0000","last_fullsized":"2024-09-16T05:27:21.005317+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'9","last_scrub_stamp":"2024-09-16T05:24:50.200774+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-09-16T05:22:06.590423+0000","last_clean_scrub_stamp":"2024-09-16T05:24:50.200774+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.757843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053661280000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":3693,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.767405+0000","last_change":"2024-09-16T05:27:21.767439+0000","last_active":"2024-09-16T05:27:21.767405+0000","last_peered":"2024-09-16T05:27:21.767405+0000","last_clean":"2024-09-16T05:27:21.767405+0000","last_became_active":"2024-09-16T05:26:58.978126+0000","last_became_peered":"2024-09-16T05:26:58.978126+0000","last_unstale":"2024-09-16T05:27:21.767405+0000","last_undegraded":"2024-09-16T05:27:21.767405+0000","last_fullsized":"2024-09-16T05:27:21.767405+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:27:21.767342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:27:21.767342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:35.295090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049565989999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"245'105","reported_seq":2684,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988050+0000","last_change":"2024-09-16T05:27:13.270296+0000","last_active":"2024-09-16T05:27:15.988050+0000","last_peered":"2024-09-16T05:27:15.988050+0000","last_clean":"2024-09-16T05:27:15.988050+0000","last_became_active":"2024-09-16T05:26:22.097775+0000","last_became_peered":"2024-09-16T05:26:22.097775+0000","last_unstale":"2024-09-16T05:27:15.988050+0000","last_undegraded":"2024-09-16T05:27:15.988050+0000","last_fullsized":"2024-09-16T05:27:15.988050+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'105","last_scrub_stamp":"2024-09-16T05:26:25.255348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:26:25.255348+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:30.913526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00081833399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.29","version":"1022'1529","reported_seq":6351,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.026381+0000","last_change":"2024-09-16T05:27:12.914299+0000","last_active":"2024-09-16T05:27:16.026381+0000","last_peered":"2024-09-16T05:27:16.026381+0000","last_clean":"2024-09-16T05:27:16.026381+0000","last_became_active":"2024-09-16T05:21:57.823393+0000","last_became_peered":"2024-09-16T05:21:57.823393+0000","last_unstale":"2024-09-16T05:27:16.026381+0000","last_undegraded":"2024-09-16T05:27:16.026381+0000","last_fullsized":"2024-09-16T05:27:16.026381+0000","mapping_epoch":767,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":768,"parent":"0.0","parent_split_bits":6,"last_scrub":"993'1505","last_scrub_stamp":"2024-09-16T05:26:30.998200+0000","last_deep_scrub":"993'1505","last_deep_scrub_stamp":"2024-09-16T05:26:30.998200+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.998200+0000","objects_scrubbed":8,"log_size":1529,"log_dups_size":0,"ondisk_log_size":1529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:56.201724+0000","scrub_duration":30,"objects_trimmed":2,"snaptrim_duration":0.004065656,"stat_sum":{"num_bytes":16690803,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":467,"num_read_kb":444552,"num_write":339,"num_write_kb":133192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":45326140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.2a","version":"245'325","reported_seq":2073,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.287939+0000","last_change":"2024-09-16T05:27:12.919933+0000","last_active":"2024-09-16T05:27:16.287939+0000","last_peered":"2024-09-16T05:27:16.287939+0000","last_clean":"2024-09-16T05:27:16.287939+0000","last_became_active":"2024-09-16T05:23:26.848398+0000","last_became_peered":"2024-09-16T05:23:26.848398+0000","last_unstale":"2024-09-16T05:27:16.287939+0000","last_undegraded":"2024-09-16T05:27:16.287939+0000","last_fullsized":"2024-09-16T05:27:16.287939+0000","mapping_epoch":856,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":857,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'325","last_scrub_stamp":"2024-09-16T05:26:24.408846+0000","last_deep_scrub":"245'325","last_deep_scrub_stamp":"2024-09-16T05:21:03.783094+0000","last_clean_scrub_stamp":"2024-09-16T05:26:24.408846+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:36.241065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001552026,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.2b","version":"952'476","reported_seq":4896,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:15.988523+0000","last_change":"2024-09-16T05:27:13.273013+0000","last_active":"2024-09-16T05:27:15.988523+0000","last_peered":"2024-09-16T05:27:15.988523+0000","last_clean":"2024-09-16T05:27:15.988523+0000","last_became_active":"2024-09-16T05:14:39.907679+0000","last_became_peered":"2024-09-16T05:14:39.907679+0000","last_unstale":"2024-09-16T05:27:15.988523+0000","last_undegraded":"2024-09-16T05:27:15.988523+0000","last_fullsized":"2024-09-16T05:27:15.988523+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":6,"last_scrub":"952'476","last_scrub_stamp":"2024-09-16T05:25:36.488467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:36.488467+0000","objects_scrubbed":1,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030417159999999999,"stat_sum":{"num_bytes":2269275,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":143,"num_read_kb":133267,"num_write":89,"num_write_kb":35255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.2c","version":"241'194","reported_seq":4605,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.396039+0000","last_change":"2024-09-16T05:27:12.915376+0000","last_active":"2024-09-16T05:27:16.396039+0000","last_peered":"2024-09-16T05:27:16.396039+0000","last_clean":"2024-09-16T05:27:16.396039+0000","last_became_active":"2024-09-16T05:13:04.679811+0000","last_became_peered":"2024-09-16T05:13:04.679811+0000","last_unstale":"2024-09-16T05:27:16.396039+0000","last_undegraded":"2024-09-16T05:27:16.396039+0000","last_fullsized":"2024-09-16T05:27:16.396039+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":6,"last_scrub":"241'194","last_scrub_stamp":"2024-09-16T05:25:55.657706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:55.657706+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011310809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1a2","length":"1"}]},{"pgid":"2.2d","version":"1022'545","reported_seq":5025,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.004962+0000","last_change":"2024-09-16T05:27:13.277641+0000","last_active":"2024-09-16T05:27:21.004962+0000","last_peered":"2024-09-16T05:27:21.004962+0000","last_clean":"2024-09-16T05:27:21.004962+0000","last_became_active":"2024-09-16T05:26:58.981035+0000","last_became_peered":"2024-09-16T05:26:58.981035+0000","last_unstale":"2024-09-16T05:27:21.004962+0000","last_undegraded":"2024-09-16T05:27:21.004962+0000","last_fullsized":"2024-09-16T05:27:21.004962+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":6,"last_scrub":"864'525","last_scrub_stamp":"2024-09-16T05:24:18.122638+0000","last_deep_scrub":"819'522","last_deep_scrub_stamp":"2024-09-16T05:22:54.200053+0000","last_clean_scrub_stamp":"2024-09-16T05:24:18.122638+0000","objects_scrubbed":1,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.757701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026306849999999998,"stat_sum":{"num_bytes":2470959,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":114,"num_read_kb":112905,"num_write":80,"num_write_kb":38649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":10583843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2e","version":"1022'893","reported_seq":5235,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-09-16T05:27:21.005433+0000","last_change":"2024-09-16T05:27:13.281828+0000","last_active":"2024-09-16T05:27:21.005433+0000","last_peered":"2024-09-16T05:27:21.005433+0000","last_clean":"2024-09-16T05:27:21.005433+0000","last_became_active":"2024-09-16T05:26:58.976847+0000","last_became_peered":"2024-09-16T05:26:58.976847+0000","last_unstale":"2024-09-16T05:27:21.005433+0000","last_undegraded":"2024-09-16T05:27:21.005433+0000","last_fullsized":"2024-09-16T05:27:21.005433+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":6,"last_scrub":"875'821","last_scrub_stamp":"2024-09-16T05:24:00.151377+0000","last_deep_scrub":"519'444","last_deep_scrub_stamp":"2024-09-16T05:17:49.581405+0000","last_clean_scrub_stamp":"2024-09-16T05:24:00.151377+0000","objects_scrubbed":0,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:21.757651+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0066521890000000002,"stat_sum":{"num_bytes":7762503,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":156,"num_read_kb":100470,"num_write":183,"num_write_kb":84584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2f","version":"1022'851","reported_seq":5391,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-09-16T05:27:16.288279+0000","last_change":"2024-09-16T05:27:12.923105+0000","last_active":"2024-09-16T05:27:16.288279+0000","last_peered":"2024-09-16T05:27:16.288279+0000","last_clean":"2024-09-16T05:27:16.288279+0000","last_became_active":"2024-09-16T05:13:27.687646+0000","last_became_peered":"2024-09-16T05:13:27.687646+0000","last_unstale":"2024-09-16T05:27:16.288279+0000","last_undegraded":"2024-09-16T05:27:16.288279+0000","last_fullsized":"2024-09-16T05:27:16.288279+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":6,"last_scrub":"969'837","last_scrub_stamp":"2024-09-16T05:25:24.189547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:10:35.939787+0000","last_clean_scrub_stamp":"2024-09-16T05:25:24.189547+0000","objects_scrubbed":5,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0041242859999999996,"stat_sum":{"num_bytes":12371140,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":206,"num_read_kb":147197,"num_write":210,"num_write_kb":77450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"1a2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":96,"stat_sum":{"num_bytes":246703288,"num_objects":101,"num_object_clones":55,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":9,"num_read":4790,"num_read_kb":4736528,"num_write":3479,"num_write_kb":1407823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":176843834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":305336320,"data_stored":361182956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35391,"ondisk_log_size":35391,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089563,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83492,"kb_used_data":34256,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93661660,"statfs":{"total":95995035648,"available":95909539840,"internally_reserved":0,"allocated":35078144,"data_stored":33691881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285792,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96896,"kb_used_data":47664,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93648256,"statfs":{"total":95995035648,"available":95895814144,"internally_reserved":0,"allocated":48807936,"data_stored":53972820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514730,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105848,"kb_used_data":34404,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93639304,"statfs":{"total":95995035648,"available":95886647296,"internally_reserved":0,"allocated":35229696,"data_stored":34549481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841769,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108380,"kb_used_data":59140,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93636772,"statfs":{"total":95995035648,"available":95884054528,"internally_reserved":0,"allocated":60559360,"data_stored":68691375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1007,"seq":4325032067078,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60588,"kb_used_data":32408,"kb_used_omap":38,"kb_used_meta":28121,"kb_avail":93684564,"statfs":{"total":95995035648,"available":95932993536,"internally_reserved":0,"allocated":33185792,"data_stored":35202620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38947,"internal_metadata":28796893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397749,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89536,"kb_used_data":36140,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93655616,"statfs":{"total":95995035648,"available":95903350784,"internally_reserved":0,"allocated":37007360,"data_stored":48158938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65560,"kb_used_data":8000,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93679592,"statfs":{"total":95995035648,"available":95927902208,"internally_reserved":0,"allocated":8192000,"data_stored":5664139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921022,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77392,"kb_used_data":15676,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93667760,"statfs":{"total":95995035648,"available":95915786240,"internally_reserved":0,"allocated":16052224,"data_stored":14757480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117250,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95192,"kb_used_data":41796,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93649960,"statfs":{"total":95995035648,"available":95897559040,"internally_reserved":0,"allocated":42799104,"data_stored":43354978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240537,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69404,"kb_used_data":32972,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93675748,"statfs":{"total":95995035648,"available":95923965952,"internally_reserved":0,"allocated":33763328,"data_stored":33391879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":988,"seq":4243427688461,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64704,"kb_used_data":28076,"kb_used_omap":22,"kb_used_meta":36585,"kb_avail":93680448,"statfs":{"total":95995035648,"available":95928778752,"internally_reserved":0,"allocated":28749824,"data_stored":33593421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22663,"internal_metadata":37463929},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738642,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69892,"kb_used_data":24820,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93675260,"statfs":{"total":95995035648,"available":95923466240,"internally_reserved":0,"allocated":25415680,"data_stored":25312404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16896000,"data_stored":19320585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20557824,"data_stored":27929282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25305088,"data_stored":27465052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":34279424,"data_stored":37363159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7860224,"data_stored":9093341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28815360,"data_stored":42494799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24715264,"data_stored":29267737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":52367360,"data_stored":63027236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":27037696,"data_stored":28885342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":40615936,"data_stored":48308681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26886144,"data_stored":28027742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:25.439 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-16T05:27:25.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-16T05:27:25.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:25.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:25.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:25.658 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:25.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:25.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:25.782 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:25.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:25.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T05:27:25.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:25.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:25.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:26.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:26.036 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:26.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:26.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:26.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: pgmap v1736: 127 pgs: 2 peering, 125 active+clean; 236 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 0 op/s 2024-09-16T05:27:26.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: 2.13 scrub starts 2024-09-16T05:27:26.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: 2.13 scrub ok 2024-09-16T05:27:26.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:26.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:26.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]': finished 2024-09-16T05:27:26.147 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]': finished 2024-09-16T05:27:26.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]': finished 2024-09-16T05:27:26.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]': finished 2024-09-16T05:27:26.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]': finished 2024-09-16T05:27:26.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]': finished 2024-09-16T05:27:26.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]': finished 2024-09-16T05:27:26.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-16T05:27:26.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:25 smithi032 ceph-mon[27388]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-16T05:27:26.164 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:26.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: pgmap v1736: 127 pgs: 2 peering, 125 active+clean; 236 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 0 op/s 2024-09-16T05:27:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: 2.13 scrub starts 2024-09-16T05:27:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: 2.13 scrub ok 2024-09-16T05:27:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]': finished 2024-09-16T05:27:26.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]': finished 2024-09-16T05:27:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]': finished 2024-09-16T05:27:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]': finished 2024-09-16T05:27:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]': finished 2024-09-16T05:27:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]': finished 2024-09-16T05:27:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]': finished 2024-09-16T05:27:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-16T05:27:26.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:25 smithi196 ceph-mon[30802]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-16T05:27:26.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:26.292 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:26.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:26.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: pgmap v1736: 127 pgs: 2 peering, 125 active+clean; 236 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 0 op/s 2024-09-16T05:27:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: 2.13 scrub starts 2024-09-16T05:27:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: 2.13 scrub ok 2024-09-16T05:27:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10714070898196588]}]': finished 2024-09-16T05:27:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07799668006241113]}]': finished 2024-09-16T05:27:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12682033380538546]}]': finished 2024-09-16T05:27:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.0945696392773942]}]': finished 2024-09-16T05:27:26.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08970152326488998]}]': finished 2024-09-16T05:27:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0880280581321455]}]': finished 2024-09-16T05:27:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08077136770282659]}]': finished 2024-09-16T05:27:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05954465837453793]}]': finished 2024-09-16T05:27:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"5\": \"18806\"}"}]': finished 2024-09-16T05:27:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-16T05:27:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:25 smithi204 ceph-mon[30933]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-16T05:27:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:26.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:26.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:26.471 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:26.490 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:26.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:26.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:26.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:26.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:26.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:26.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:26.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:26.799 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:26.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:26.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:26.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:26.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:26.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T05:27:26.995 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-09-16T05:27:26.995 INFO:teuthology.orchestra.run.smithi032.stdout: "ops": [], 2024-09-16T05:27:26.996 INFO:teuthology.orchestra.run.smithi032.stdout: "num_ops": 0 2024-09-16T05:27:26.996 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-09-16T05:27:27.030 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:27.056 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:27.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:27.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: 2.47 scrub starts 2024-09-16T05:27:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: 2.47 scrub ok 2024-09-16T05:27:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: 2.19 scrub starts 2024-09-16T05:27:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: 2.19 scrub ok 2024-09-16T05:27:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: 2.1 scrub starts 2024-09-16T05:27:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: 2.1 scrub ok 2024-09-16T05:27:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: 2.4 scrub starts 2024-09-16T05:27:27.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: 2.4 scrub ok 2024-09-16T05:27:27.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:26 smithi032 ceph-mon[27388]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-16T05:27:27.183 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:27.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: 2.47 scrub starts 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: 2.47 scrub ok 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: 2.19 scrub starts 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: 2.19 scrub ok 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: 2.1 scrub starts 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: 2.1 scrub ok 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: 2.4 scrub starts 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: 2.4 scrub ok 2024-09-16T05:27:27.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:26 smithi196 ceph-mon[30802]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-16T05:27:27.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:27.308 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:27.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:27.410 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:27.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: 2.47 scrub starts 2024-09-16T05:27:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: 2.47 scrub ok 2024-09-16T05:27:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: 2.19 scrub starts 2024-09-16T05:27:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: 2.19 scrub ok 2024-09-16T05:27:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: 2.1 scrub starts 2024-09-16T05:27:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: 2.1 scrub ok 2024-09-16T05:27:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: 2.4 scrub starts 2024-09-16T05:27:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: 2.4 scrub ok 2024-09-16T05:27:27.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:26 smithi204 ceph-mon[30933]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-16T05:27:27.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:27.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:27.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:27.559 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:27.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:27.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:27.683 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:27.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:27.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:27.811 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:27.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:27.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:27.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:27.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:28.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-16T05:27:28.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:28.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:28.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:28.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:28.192 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:28.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:27 smithi196 ceph-mon[30802]: pgmap v1739: 127 pgs: 4 peering, 123 active+clean; 236 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:27 smithi196 ceph-mon[30802]: 2.5c scrub starts 2024-09-16T05:27:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:27 smithi196 ceph-mon[30802]: 2.5c scrub ok 2024-09-16T05:27:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:27 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:27:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:27 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:27:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:27 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:28.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:27 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:28.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:27 smithi032 ceph-mon[27388]: pgmap v1739: 127 pgs: 4 peering, 123 active+clean; 236 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:28.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:27 smithi032 ceph-mon[27388]: 2.5c scrub starts 2024-09-16T05:27:28.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:27 smithi032 ceph-mon[27388]: 2.5c scrub ok 2024-09-16T05:27:28.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:27 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:27:28.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:27 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:27:28.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:27 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:28.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:27 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:28.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:28.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:28.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:28.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:27 smithi204 ceph-mon[30933]: pgmap v1739: 127 pgs: 4 peering, 123 active+clean; 236 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:28.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:27 smithi204 ceph-mon[30933]: 2.5c scrub starts 2024-09-16T05:27:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:27 smithi204 ceph-mon[30933]: 2.5c scrub ok 2024-09-16T05:27:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:27 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:27:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:27 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:27:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:27 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:28.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:27 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:28.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:28.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:28.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:28.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:28.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:28.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:28.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:28.705 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:28.746 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:28.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:28.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:28.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:28.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:28.957 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:28.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:29.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3 2024-09-16T05:27:29.081 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:29.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.3' 2024-09-16T05:27:29.108 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:28 smithi032 ceph-mon[27388]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-16T05:27:29.108 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.108 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1332252757' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T05:27:29.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:27:29.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:27:29.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:28 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1053469301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:29.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:29.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:29.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:29.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:28 smithi196 ceph-mon[30802]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-16T05:27:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1332252757' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T05:27:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:27:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:27:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:28 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1053469301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:29.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:29.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:29.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:28 smithi204 ceph-mon[30933]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-16T05:27:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1332252757' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T05:27:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:27:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:27:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:28 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:29.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1053469301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:29.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:29.445 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:27:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T05:27:29.446 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'unique_pool_0' removed 2024-09-16T05:27:29.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:29.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:29.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:29.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:29.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:29.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:29.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:29.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:29.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:29.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:29.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:29.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:29.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:29.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:30.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1 2024-09-16T05:27:30.091 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:30.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.1' 2024-09-16T05:27:30.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: pgmap v1742: 127 pgs: 4 peering, 123 active+clean; 236 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: 2.7 scrub starts 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: 2.7 scrub ok 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: 2.56 scrub starts 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: 2.56 scrub ok 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: 2.2d scrub starts 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: 2.2d scrub ok 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:30.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3286961151' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:27:30.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:27:30.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:27:30.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1332252757' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T05:27:30.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1053469301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:30.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3286961151' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:27:30.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:29 smithi032 ceph-mon[27388]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-16T05:27:30.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:30.216 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:30.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: pgmap v1742: 127 pgs: 4 peering, 123 active+clean; 236 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: 2.7 scrub starts 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: 2.7 scrub ok 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: 2.56 scrub starts 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: 2.56 scrub ok 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: 2.2d scrub starts 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: 2.2d scrub ok 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:27:30.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:27:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3286961151' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:27:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:27:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:27:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1332252757' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T05:27:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1053469301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3286961151' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:27:30.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:29 smithi196 ceph-mon[30802]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-16T05:27:30.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5 2024-09-16T05:27:30.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-16T05:27:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:30.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:30.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.5' 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: pgmap v1742: 127 pgs: 4 peering, 123 active+clean; 236 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: 2.7 scrub starts 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: 2.7 scrub ok 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: 2.56 scrub starts 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: 2.56 scrub ok 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: 2.2d scrub starts 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: 2.2d scrub ok 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:27:30.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3286961151' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:27:30.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-16T05:27:30.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:27:30.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1332252757' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T05:27:30.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1053469301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:30.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3286961151' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:27:30.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:29 smithi204 ceph-mon[30933]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-16T05:27:30.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:30.473 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:30.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:30.561 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0: {} 2024-09-16T05:27:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T05:27:30.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:30.583 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1: {} 2024-09-16T05:27:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T05:27:30.600 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:30.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:30.604 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2: {} 2024-09-16T05:27:30.604 INFO:teuthology.orchestra.run.smithi032.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T05:27:30.625 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3: {} 2024-09-16T05:27:30.625 INFO:teuthology.orchestra.run.smithi032.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T05:27:30.647 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4: {} 2024-09-16T05:27:30.647 INFO:teuthology.orchestra.run.smithi032.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T05:27:30.669 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5: {} 2024-09-16T05:27:30.669 INFO:teuthology.orchestra.run.smithi032.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T05:27:30.691 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6: {} 2024-09-16T05:27:30.691 INFO:teuthology.orchestra.run.smithi032.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T05:27:30.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:30.713 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7: {} 2024-09-16T05:27:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T05:27:30.728 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:30.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:30.736 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8: {} 2024-09-16T05:27:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T05:27:30.760 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9: {} 2024-09-16T05:27:30.760 INFO:teuthology.orchestra.run.smithi032.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T05:27:30.782 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10: {} 2024-09-16T05:27:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T05:27:30.804 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11: {} 2024-09-16T05:27:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T05:27:30.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:30.831 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:30.832 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:30.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:30.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6 2024-09-16T05:27:30.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:30.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.6' 2024-09-16T05:27:31.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:30 smithi032 ceph-mon[27388]: 2.31 scrub starts 2024-09-16T05:27:31.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:30 smithi032 ceph-mon[27388]: 2.31 scrub ok 2024-09-16T05:27:31.070 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:30 smithi032 ceph-mon[27388]: 1.17 scrub starts 2024-09-16T05:27:31.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:30 smithi032 ceph-mon[27388]: 1.17 scrub ok 2024-09-16T05:27:31.071 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3228442640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:31.087 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:31.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:31.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:31.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:30 smithi196 ceph-mon[30802]: 2.31 scrub starts 2024-09-16T05:27:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:30 smithi196 ceph-mon[30802]: 2.31 scrub ok 2024-09-16T05:27:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:30 smithi196 ceph-mon[30802]: 1.17 scrub starts 2024-09-16T05:27:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:30 smithi196 ceph-mon[30802]: 1.17 scrub ok 2024-09-16T05:27:31.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3228442640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:31.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:31.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:31.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:31.369 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:31.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:31.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:30 smithi204 ceph-mon[30933]: 2.31 scrub starts 2024-09-16T05:27:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:30 smithi204 ceph-mon[30933]: 2.31 scrub ok 2024-09-16T05:27:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:30 smithi204 ceph-mon[30933]: 1.17 scrub starts 2024-09-16T05:27:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:30 smithi204 ceph-mon[30933]: 1.17 scrub ok 2024-09-16T05:27:31.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3228442640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-16T05:27:31.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7 2024-09-16T05:27:31.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:31.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.7' 2024-09-16T05:27:31.597 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:31.623 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:31.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:31.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:31.751 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:31.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:31.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:31.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:31.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:32.005 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:32.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:32.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11 2024-09-16T05:27:32.133 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:32.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.11' 2024-09-16T05:27:32.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4 2024-09-16T05:27:32.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:31 smithi196 ceph-mon[30802]: pgmap v1744: 31 pgs: 31 active+clean; 577 KiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:31 smithi196 ceph-mon[30802]: 1.9 deep-scrub starts 2024-09-16T05:27:32.239 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:31 smithi196 ceph-mon[30802]: 1.9 deep-scrub ok 2024-09-16T05:27:32.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-09-16T05:27:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:32.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.4' 2024-09-16T05:27:32.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:31 smithi032 ceph-mon[27388]: pgmap v1744: 31 pgs: 31 active+clean; 577 KiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:32.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:31 smithi032 ceph-mon[27388]: 1.9 deep-scrub starts 2024-09-16T05:27:32.301 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:31 smithi032 ceph-mon[27388]: 1.9 deep-scrub ok 2024-09-16T05:27:32.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8 2024-09-16T05:27:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:32.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.8' 2024-09-16T05:27:32.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:31 smithi204 ceph-mon[30933]: pgmap v1744: 31 pgs: 31 active+clean; 577 KiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:32.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:31 smithi204 ceph-mon[30933]: 1.9 deep-scrub starts 2024-09-16T05:27:32.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:31 smithi204 ceph-mon[30933]: 1.9 deep-scrub ok 2024-09-16T05:27:32.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0 2024-09-16T05:27:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:32.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.0' 2024-09-16T05:27:32.620 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10 2024-09-16T05:27:32.646 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:32.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.10' 2024-09-16T05:27:32.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2 2024-09-16T05:27:32.775 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: docker: command not found 2024-09-16T05:27:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:32.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi032 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.2' 2024-09-16T05:27:32.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9 2024-09-16T05:27:32.881 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T05:27:32.892 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T05:27:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:27:32.904 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: docker: command not found 2024-09-16T05:27:32.904 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi204 with status 1: 'sudo docker kill -s 1 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd.9' 2024-09-16T05:27:33.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-16T05:27:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:34.142 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:34.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:33 smithi032 ceph-mon[27388]: pgmap v1745: 31 pgs: 31 active+clean; 577 KiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:34.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:33 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:34.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:33 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:34.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:33 smithi032 ceph-mon[27388]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-16T05:27:34.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:33 smithi204 ceph-mon[30933]: pgmap v1745: 31 pgs: 31 active+clean; 577 KiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:34.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:33 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:34.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:33 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:34.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:33 smithi204 ceph-mon[30933]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-16T05:27:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:33 smithi196 ceph-mon[30802]: pgmap v1745: 31 pgs: 31 active+clean; 577 KiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:33 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:33 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:33 smithi196 ceph-mon[30802]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-16T05:27:34.550 INFO:teuthology.orchestra.run.smithi032.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:34.550 INFO:teuthology.orchestra.run.smithi032.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T05:27:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:34 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:27:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:34 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:27:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:34 smithi032 ceph-mon[27388]: 1.d deep-scrub starts 2024-09-16T05:27:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:34 smithi032 ceph-mon[27388]: 1.d deep-scrub ok 2024-09-16T05:27:35.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2263211337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:27:35.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-16T05:27:35.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:34 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:27:35.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:34 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:27:35.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:34 smithi196 ceph-mon[30802]: 1.d deep-scrub starts 2024-09-16T05:27:35.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:34 smithi196 ceph-mon[30802]: 1.d deep-scrub ok 2024-09-16T05:27:35.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2263211337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:27:35.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-16T05:27:35.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:34 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:27:35.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:34 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:27:35.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:34 smithi204 ceph-mon[30933]: 1.d deep-scrub starts 2024-09-16T05:27:35.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:34 smithi204 ceph-mon[30933]: 1.d deep-scrub ok 2024-09-16T05:27:35.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2263211337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:27:35.480 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:27:35.961 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:35.961 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:36.027 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-16T05:27:36.097 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:36.233 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:35 smithi196 ceph-mon[30802]: pgmap v1747: 31 pgs: 31 active+clean; 577 KiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:36.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:35 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:36.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:35 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:36.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:35 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:27:36.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:35 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:27:36.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2263211337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:27:36.234 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:35 smithi196 ceph-mon[30802]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-16T05:27:36.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:35 smithi032 ceph-mon[27388]: pgmap v1747: 31 pgs: 31 active+clean; 577 KiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:36.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:35 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:36.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:35 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:36.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:35 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:27:36.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:35 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:27:36.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2263211337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:27:36.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:35 smithi032 ceph-mon[27388]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-16T05:27:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:35 smithi204 ceph-mon[30933]: pgmap v1747: 31 pgs: 31 active+clean; 577 KiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:35 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:35 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:35 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:27:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:35 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:27:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2263211337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:27:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:35 smithi204 ceph-mon[30933]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-16T05:27:36.643 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:36.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-16T05:27:36.960 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:27:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:noscrub is unset 2024-09-16T05:27:37.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1526037660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:37.375 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1526037660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:37.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1526037660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-16T05:27:37.492 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:37.492 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:37.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-16T05:27:38.005 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T05:27:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:38 smithi032 ceph-mon[27388]: pgmap v1749: 31 pgs: 31 active+clean; 577 KiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1526037660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:38 smithi032 ceph-mon[27388]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-16T05:27:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3995479578' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:27:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:38 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:38 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:38 smithi196 ceph-mon[30802]: pgmap v1749: 31 pgs: 31 active+clean; 577 KiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1526037660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:38 smithi196 ceph-mon[30802]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-16T05:27:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3995479578' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:27:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:38 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:38 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:38.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-16T05:27:38.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:38 smithi204 ceph-mon[30933]: pgmap v1749: 31 pgs: 31 active+clean; 577 KiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1526037660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-16T05:27:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:38 smithi204 ceph-mon[30933]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-16T05:27:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3995479578' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:27:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:38 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:38 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:38.577 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T05:27:38.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-16T05:27:38.851 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:39.032 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-09-16T05:27:39.033 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T05:27:39.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3995479578' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:27:39.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:39 smithi032 ceph-mon[27388]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-16T05:27:39.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:39 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:27:39.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:39 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:27:39.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3995479578' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:27:39.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:39 smithi204 ceph-mon[30933]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-16T05:27:39.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:39 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:27:39.321 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:39 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:27:39.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3995479578' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:27:39.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:39 smithi196 ceph-mon[30802]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-16T05:27:39.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:39 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:27:39.327 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:39 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:27:39.328 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:40.016 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T05:27:40.017 INFO:teuthology.orchestra.run.smithi032.stderr:nodeep-scrub is unset 2024-09-16T05:27:40.294 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:40 smithi032 ceph-mon[27388]: pgmap v1752: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:40.295 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3441611754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:40.295 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2998422514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:27:40.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:40 smithi204 ceph-mon[30933]: pgmap v1752: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:40.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3441611754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:40.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2998422514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:27:40.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:40 smithi196 ceph-mon[30802]: pgmap v1752: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:40.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3441611754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-16T05:27:40.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2998422514' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:27:40.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-16T05:27:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:41 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:41 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3441611754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2998422514' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:27:41.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:41 smithi032 ceph-mon[27388]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-16T05:27:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:41 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:41 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3441611754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2998422514' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:27:41.171 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:41 smithi204 ceph-mon[30933]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-16T05:27:41.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:41 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:41.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:41 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:41.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3441611754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-16T05:27:41.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2998422514' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:27:41.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:41 smithi196 ceph-mon[30802]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-16T05:27:41.632 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:42.109 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:42 smithi032 ceph-mon[27388]: pgmap v1754: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:42.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:42 smithi204 ceph-mon[30933]: pgmap v1754: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:42.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:42 smithi196 ceph-mon[30802]: pgmap v1754: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T05:27:43.296 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:43 smithi032 ceph-mon[27388]: 1.3 deep-scrub starts 2024-09-16T05:27:43.296 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:43 smithi032 ceph-mon[27388]: 1.3 deep-scrub ok 2024-09-16T05:27:43.296 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:43 smithi032 ceph-mon[27388]: 1.1c scrub starts 2024-09-16T05:27:43.296 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:43 smithi032 ceph-mon[27388]: 1.1c scrub ok 2024-09-16T05:27:43.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1887621669' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:27:43.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:43 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:27:43.297 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:43 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:27:43.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:43 smithi204 ceph-mon[30933]: 1.3 deep-scrub starts 2024-09-16T05:27:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:43 smithi204 ceph-mon[30933]: 1.3 deep-scrub ok 2024-09-16T05:27:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:43 smithi204 ceph-mon[30933]: 1.1c scrub starts 2024-09-16T05:27:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:43 smithi204 ceph-mon[30933]: 1.1c scrub ok 2024-09-16T05:27:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1887621669' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:27:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:43 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:27:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:43 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:27:43.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:43 smithi196 ceph-mon[30802]: 1.3 deep-scrub starts 2024-09-16T05:27:43.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:43 smithi196 ceph-mon[30802]: 1.3 deep-scrub ok 2024-09-16T05:27:43.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:43 smithi196 ceph-mon[30802]: 1.1c scrub starts 2024-09-16T05:27:43.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:43 smithi196 ceph-mon[30802]: 1.1c scrub ok 2024-09-16T05:27:43.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1887621669' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:27:43.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:43 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:27:43.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:43 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:27:43.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T05:27:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:44.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:44 smithi032 ceph-mon[27388]: pgmap v1755: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:44.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1887621669' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:27:44.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:44 smithi032 ceph-mon[27388]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-16T05:27:44.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:44 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:44.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:44 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:44 smithi204 ceph-mon[30933]: pgmap v1755: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1887621669' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:27:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:44 smithi204 ceph-mon[30933]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-16T05:27:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:44 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:44.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:44 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:44.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:44 smithi196 ceph-mon[30802]: pgmap v1755: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:44.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1887621669' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:27:44.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:44 smithi196 ceph-mon[30802]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-16T05:27:44.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:44 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:44.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:44 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:45.039 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T05:27:45.395 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:45 smithi032 ceph-mon[27388]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-16T05:27:45.396 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:45 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2742943367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:27:45.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:45 smithi204 ceph-mon[30933]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-16T05:27:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:45 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2742943367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:27:45.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:45 smithi196 ceph-mon[30802]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-16T05:27:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:45 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2742943367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:27:45.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T05:27:46.099 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:46 smithi032 ceph-mon[27388]: pgmap v1758: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2742943367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:27:46.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:46 smithi032 ceph-mon[27388]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-16T05:27:46.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:46 smithi204 ceph-mon[30933]: pgmap v1758: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2742943367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:27:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:46 smithi204 ceph-mon[30933]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-16T05:27:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:46 smithi196 ceph-mon[30802]: pgmap v1758: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2742943367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:27:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:46 smithi196 ceph-mon[30802]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-16T05:27:47.048 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T05:27:47.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:47 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:27:47.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:47 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:27:47.405 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1209029886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:27:47.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:47 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:27:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:47 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:27:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1209029886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:27:47.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:47 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:27:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:47 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:27:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1209029886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:27:47.859 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T05:27:47.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:27:48.120 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:48 smithi032 ceph-mon[27388]: pgmap v1760: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1209029886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:27:48.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:48 smithi032 ceph-mon[27388]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-16T05:27:48.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:48 smithi204 ceph-mon[30933]: pgmap v1760: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:48.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1209029886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:27:48.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:48 smithi204 ceph-mon[30933]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-16T05:27:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:48 smithi196 ceph-mon[30802]: pgmap v1760: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1209029886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:27:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:48 smithi196 ceph-mon[30802]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-16T05:27:48.729 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:27:48.729 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:27:49.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:49 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:27:49.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:49 smithi032 ceph-mon[27388]: 1.b scrub ok 2024-09-16T05:27:49.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:49 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:27:49.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:49 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:27:49.321 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1762,"stamp":"2024-09-16T05:27:48.299288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":674228,"kb_used_data":99308,"kb_used_omap":84,"kb_used_meta":574571,"kb_avail":1124267596,"statfs":{"total":1151940427776,"available":1151250018304,"internally_reserved":0,"allocated":101691392,"data_stored":70570945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86930,"internal_metadata":588360814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.267763"},"pg_stats":[{"pgid":"1.12","version":"35'139","reported_seq":683,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048911+0000","last_change":"2024-09-16T05:27:26.907634+0000","last_active":"2024-09-16T05:27:47.048911+0000","last_peered":"2024-09-16T05:27:47.048911+0000","last_clean":"2024-09-16T05:27:47.048911+0000","last_became_active":"2024-09-16T05:27:26.907217+0000","last_became_peered":"2024-09-16T05:27:26.907217+0000","last_unstale":"2024-09-16T05:27:47.048911+0000","last_undegraded":"2024-09-16T05:27:47.048911+0000","last_fullsized":"2024-09-16T05:27:47.048911+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:28.030079+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:26:28.030079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:48.116060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"35'139","reported_seq":1423,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.094462+0000","last_change":"2024-09-16T05:27:24.731478+0000","last_active":"2024-09-16T05:27:44.094462+0000","last_peered":"2024-09-16T05:27:44.094462+0000","last_clean":"2024-09-16T05:27:44.094462+0000","last_became_active":"2024-09-16T05:26:22.907168+0000","last_became_peered":"2024-09-16T05:26:22.907168+0000","last_unstale":"2024-09-16T05:27:44.094462+0000","last_undegraded":"2024-09-16T05:27:44.094462+0000","last_fullsized":"2024-09-16T05:27:44.094462+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:24.731376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:24.731376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:27.331684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1715,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048928+0000","last_change":"2024-09-16T05:27:38.109553+0000","last_active":"2024-09-16T05:27:47.048928+0000","last_peered":"2024-09-16T05:27:47.048928+0000","last_clean":"2024-09-16T05:27:47.048928+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:27:47.048928+0000","last_undegraded":"2024-09-16T05:27:47.048928+0000","last_fullsized":"2024-09-16T05:27:47.048928+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:38.109468+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:38.109468+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:46.874057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"35'139","reported_seq":1767,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403505+0000","last_change":"2024-09-16T05:27:33.791163+0000","last_active":"2024-09-16T05:27:45.403505+0000","last_peered":"2024-09-16T05:27:45.403505+0000","last_clean":"2024-09-16T05:27:45.403505+0000","last_became_active":"2024-09-16T05:26:58.976632+0000","last_became_peered":"2024-09-16T05:26:58.976632+0000","last_unstale":"2024-09-16T05:27:45.403505+0000","last_undegraded":"2024-09-16T05:27:45.403505+0000","last_fullsized":"2024-09-16T05:27:45.403505+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_clean_scrub_stamp":"2024-09-16T05:27:33.791043+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:01.742303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"35'139","reported_seq":605,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403594+0000","last_change":"2024-09-16T05:27:34.839730+0000","last_active":"2024-09-16T05:27:45.403594+0000","last_peered":"2024-09-16T05:27:45.403594+0000","last_clean":"2024-09-16T05:27:45.403594+0000","last_became_active":"2024-09-16T05:26:58.979090+0000","last_became_peered":"2024-09-16T05:26:58.979090+0000","last_unstale":"2024-09-16T05:27:45.403594+0000","last_undegraded":"2024-09-16T05:27:45.403594+0000","last_fullsized":"2024-09-16T05:27:45.403594+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:34.839637+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:27:34.839637+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:59.564317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"35'139","reported_seq":1851,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403495+0000","last_change":"2024-09-16T05:26:30.986289+0000","last_active":"2024-09-16T05:27:45.403495+0000","last_peered":"2024-09-16T05:27:45.403495+0000","last_clean":"2024-09-16T05:27:45.403495+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:27:45.403495+0000","last_undegraded":"2024-09-16T05:27:45.403495+0000","last_fullsized":"2024-09-16T05:27:45.403495+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.986186+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"35'139","reported_seq":1400,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.094373+0000","last_change":"2024-09-16T05:26:25.745603+0000","last_active":"2024-09-16T05:27:44.094373+0000","last_peered":"2024-09-16T05:27:44.094373+0000","last_clean":"2024-09-16T05:27:44.094373+0000","last_became_active":"2024-09-16T05:26:25.117472+0000","last_became_peered":"2024-09-16T05:26:25.117472+0000","last_unstale":"2024-09-16T05:27:44.094373+0000","last_undegraded":"2024-09-16T05:27:44.094373+0000","last_fullsized":"2024-09-16T05:27:44.094373+0000","mapping_epoch":991,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:25.745532+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:26:25.745532+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:46.993488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"35'139","reported_seq":1409,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173800+0000","last_change":"2024-09-16T05:27:26.597568+0000","last_active":"2024-09-16T05:27:47.173800+0000","last_peered":"2024-09-16T05:27:47.173800+0000","last_clean":"2024-09-16T05:27:47.173800+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:27:47.173800+0000","last_undegraded":"2024-09-16T05:27:47.173800+0000","last_fullsized":"2024-09-16T05:27:47.173800+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:26.597439+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_clean_scrub_stamp":"2024-09-16T05:27:26.597439+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:36.839187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1629,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099245+0000","last_change":"2024-09-16T05:26:50.173116+0000","last_active":"2024-09-16T05:27:45.099245+0000","last_peered":"2024-09-16T05:27:45.099245+0000","last_clean":"2024-09-16T05:27:45.099245+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:27:45.099245+0000","last_undegraded":"2024-09-16T05:27:45.099245+0000","last_fullsized":"2024-09-16T05:27:45.099245+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.173026+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.173026+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.105143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":593,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.155223+0000","last_change":"2024-09-16T05:26:50.546912+0000","last_active":"2024-09-16T05:27:45.155223+0000","last_peered":"2024-09-16T05:27:45.155223+0000","last_clean":"2024-09-16T05:27:45.155223+0000","last_became_active":"2024-09-16T05:25:10.425314+0000","last_became_peered":"2024-09-16T05:25:10.425314+0000","last_unstale":"2024-09-16T05:27:45.155223+0000","last_undegraded":"2024-09-16T05:27:45.155223+0000","last_fullsized":"2024-09-16T05:27:45.155223+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.546823+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:18.515049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'139","reported_seq":2393,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048911+0000","last_change":"2024-09-16T05:27:42.143188+0000","last_active":"2024-09-16T05:27:47.048911+0000","last_peered":"2024-09-16T05:27:47.048911+0000","last_clean":"2024-09-16T05:27:47.048911+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:27:47.048911+0000","last_undegraded":"2024-09-16T05:27:47.048911+0000","last_fullsized":"2024-09-16T05:27:47.048911+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.143106+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:57.339231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"35'139","reported_seq":2348,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173850+0000","last_change":"2024-09-16T05:27:01.775171+0000","last_active":"2024-09-16T05:27:47.173850+0000","last_peered":"2024-09-16T05:27:47.173850+0000","last_clean":"2024-09-16T05:27:47.173850+0000","last_became_active":"2024-09-16T05:26:58.978042+0000","last_became_peered":"2024-09-16T05:26:58.978042+0000","last_unstale":"2024-09-16T05:27:47.173850+0000","last_undegraded":"2024-09-16T05:27:47.173850+0000","last_fullsized":"2024-09-16T05:27:47.173850+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:01.775049+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:01.775049+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:28.973729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"35'139","reported_seq":1650,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099248+0000","last_change":"2024-09-16T05:26:32.280532+0000","last_active":"2024-09-16T05:27:45.099248+0000","last_peered":"2024-09-16T05:27:45.099248+0000","last_clean":"2024-09-16T05:27:45.099248+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:27:45.099248+0000","last_undegraded":"2024-09-16T05:27:45.099248+0000","last_fullsized":"2024-09-16T05:27:45.099248+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:32.280436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.280436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:45.243320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"35'139","reported_seq":1594,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099244+0000","last_change":"2024-09-16T05:26:52.226627+0000","last_active":"2024-09-16T05:27:45.099244+0000","last_peered":"2024-09-16T05:27:45.099244+0000","last_clean":"2024-09-16T05:27:45.099244+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:27:45.099244+0000","last_undegraded":"2024-09-16T05:27:45.099244+0000","last_fullsized":"2024-09-16T05:27:45.099244+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:52.226532+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:52.226532+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:08.001945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1640,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099307+0000","last_change":"2024-09-16T05:26:30.310092+0000","last_active":"2024-09-16T05:27:45.099307+0000","last_peered":"2024-09-16T05:27:45.099307+0000","last_clean":"2024-09-16T05:27:45.099307+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:27:45.099307+0000","last_undegraded":"2024-09-16T05:27:45.099307+0000","last_fullsized":"2024-09-16T05:27:45.099307+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.309950+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.309950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.005293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"35'139","reported_seq":510,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251203+0000","last_change":"2024-09-16T05:27:25.909160+0000","last_active":"2024-09-16T05:27:47.251203+0000","last_peered":"2024-09-16T05:27:47.251203+0000","last_clean":"2024-09-16T05:27:47.251203+0000","last_became_active":"2024-09-16T05:27:25.908795+0000","last_became_peered":"2024-09-16T05:27:25.908795+0000","last_unstale":"2024-09-16T05:27:47.251203+0000","last_undegraded":"2024-09-16T05:27:47.251203+0000","last_fullsized":"2024-09-16T05:27:47.251203+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:45.217959+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:11.129971+0000","last_clean_scrub_stamp":"2024-09-16T05:26:45.217959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:52.673282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'139","reported_seq":2247,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403473+0000","last_change":"2024-09-16T05:27:30.832857+0000","last_active":"2024-09-16T05:27:45.403473+0000","last_peered":"2024-09-16T05:27:45.403473+0000","last_clean":"2024-09-16T05:27:45.403473+0000","last_became_active":"2024-09-16T05:26:59.168051+0000","last_became_peered":"2024-09-16T05:26:59.168051+0000","last_unstale":"2024-09-16T05:27:45.403473+0000","last_undegraded":"2024-09-16T05:27:45.403473+0000","last_fullsized":"2024-09-16T05:27:45.403473+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_clean_scrub_stamp":"2024-09-16T05:27:30.832744+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:40.846360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2383,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403540+0000","last_change":"2024-09-16T05:26:32.294013+0000","last_active":"2024-09-16T05:27:45.403540+0000","last_peered":"2024-09-16T05:27:45.403540+0000","last_clean":"2024-09-16T05:27:45.403540+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:27:45.403540+0000","last_undegraded":"2024-09-16T05:27:45.403540+0000","last_fullsized":"2024-09-16T05:27:45.403540+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:32.293931+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.293931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:49.584816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1869,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.155221+0000","last_change":"2024-09-16T05:27:33.314807+0000","last_active":"2024-09-16T05:27:45.155221+0000","last_peered":"2024-09-16T05:27:45.155221+0000","last_clean":"2024-09-16T05:27:45.155221+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:27:45.155221+0000","last_undegraded":"2024-09-16T05:27:45.155221+0000","last_fullsized":"2024-09-16T05:27:45.155221+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:33.314672+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:27:33.314672+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:33.888940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"35'139","reported_seq":1317,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251096+0000","last_change":"2024-09-16T05:27:26.906821+0000","last_active":"2024-09-16T05:27:47.251096+0000","last_peered":"2024-09-16T05:27:47.251096+0000","last_clean":"2024-09-16T05:27:47.251096+0000","last_became_active":"2024-09-16T05:27:26.906530+0000","last_became_peered":"2024-09-16T05:27:26.906530+0000","last_unstale":"2024-09-16T05:27:47.251096+0000","last_undegraded":"2024-09-16T05:27:47.251096+0000","last_fullsized":"2024-09-16T05:27:47.251096+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:49.297389+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.297389+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:03.261298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"35'139","reported_seq":1264,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.095085+0000","last_change":"2024-09-16T05:27:30.199187+0000","last_active":"2024-09-16T05:27:44.095085+0000","last_peered":"2024-09-16T05:27:44.095085+0000","last_clean":"2024-09-16T05:27:44.095085+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:27:44.095085+0000","last_undegraded":"2024-09-16T05:27:44.095085+0000","last_fullsized":"2024-09-16T05:27:44.095085+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:30.199092+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:27:30.199092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:44.350797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"35'139","reported_seq":1312,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251140+0000","last_change":"2024-09-16T05:26:29.217364+0000","last_active":"2024-09-16T05:27:47.251140+0000","last_peered":"2024-09-16T05:27:47.251140+0000","last_clean":"2024-09-16T05:27:47.251140+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:27:47.251140+0000","last_undegraded":"2024-09-16T05:27:47.251140+0000","last_fullsized":"2024-09-16T05:27:47.251140+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:29.217292+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:26:29.217292+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.502835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":694,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.095116+0000","last_change":"2024-09-16T05:27:25.909318+0000","last_active":"2024-09-16T05:27:44.095116+0000","last_peered":"2024-09-16T05:27:44.095116+0000","last_clean":"2024-09-16T05:27:44.095116+0000","last_became_active":"2024-09-16T05:27:25.908930+0000","last_became_peered":"2024-09-16T05:27:25.908930+0000","last_unstale":"2024-09-16T05:27:44.095116+0000","last_undegraded":"2024-09-16T05:27:44.095116+0000","last_fullsized":"2024-09-16T05:27:44.095116+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:49.474636+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.474636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:55.990716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"35'139","reported_seq":1703,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251099+0000","last_change":"2024-09-16T05:27:42.336327+0000","last_active":"2024-09-16T05:27:47.251099+0000","last_peered":"2024-09-16T05:27:47.251099+0000","last_clean":"2024-09-16T05:27:47.251099+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:27:47.251099+0000","last_undegraded":"2024-09-16T05:27:47.251099+0000","last_fullsized":"2024-09-16T05:27:47.251099+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.336247+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.336247+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:59.548749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1781,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.095127+0000","last_change":"2024-09-16T05:26:33.343104+0000","last_active":"2024-09-16T05:27:44.095127+0000","last_peered":"2024-09-16T05:27:44.095127+0000","last_clean":"2024-09-16T05:27:44.095127+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:27:44.095127+0000","last_undegraded":"2024-09-16T05:27:44.095127+0000","last_fullsized":"2024-09-16T05:27:44.095127+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:33.343016+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:26:33.343016+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:59.079256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"35'139","reported_seq":490,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403472+0000","last_change":"2024-09-16T05:27:05.725386+0000","last_active":"2024-09-16T05:27:45.403472+0000","last_peered":"2024-09-16T05:27:45.403472+0000","last_clean":"2024-09-16T05:27:45.403472+0000","last_became_active":"2024-09-16T05:26:58.979664+0000","last_became_peered":"2024-09-16T05:26:58.979664+0000","last_unstale":"2024-09-16T05:27:45.403472+0000","last_undegraded":"2024-09-16T05:27:45.403472+0000","last_fullsized":"2024-09-16T05:27:45.403472+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:05.725288+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.725288+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:11.443926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"35'139","reported_seq":2374,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099253+0000","last_change":"2024-09-16T05:26:46.145954+0000","last_active":"2024-09-16T05:27:45.099253+0000","last_peered":"2024-09-16T05:27:45.099253+0000","last_clean":"2024-09-16T05:27:45.099253+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:27:45.099253+0000","last_undegraded":"2024-09-16T05:27:45.099253+0000","last_fullsized":"2024-09-16T05:27:45.099253+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:46.145861+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:26:46.145861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.153358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"35'139","reported_seq":515,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251082+0000","last_change":"2024-09-16T05:27:26.909313+0000","last_active":"2024-09-16T05:27:47.251082+0000","last_peered":"2024-09-16T05:27:47.251082+0000","last_clean":"2024-09-16T05:27:47.251082+0000","last_became_active":"2024-09-16T05:27:26.908930+0000","last_became_peered":"2024-09-16T05:27:26.908930+0000","last_unstale":"2024-09-16T05:27:47.251082+0000","last_undegraded":"2024-09-16T05:27:47.251082+0000","last_fullsized":"2024-09-16T05:27:47.251082+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:46.234400+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:46.234400+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:12.024106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1e","version":"35'139","reported_seq":1868,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173850+0000","last_change":"2024-09-16T05:26:50.779260+0000","last_active":"2024-09-16T05:27:47.173850+0000","last_peered":"2024-09-16T05:27:47.173850+0000","last_clean":"2024-09-16T05:27:47.173850+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:27:47.173850+0000","last_undegraded":"2024-09-16T05:27:47.173850+0000","last_fullsized":"2024-09-16T05:27:47.173850+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.779169+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.779169+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:20.159044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"35'139","reported_seq":919,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.094556+0000","last_change":"2024-09-16T05:27:06.911321+0000","last_active":"2024-09-16T05:27:44.094556+0000","last_peered":"2024-09-16T05:27:44.094556+0000","last_clean":"2024-09-16T05:27:44.094556+0000","last_became_active":"2024-09-16T05:26:22.907190+0000","last_became_peered":"2024-09-16T05:26:22.907190+0000","last_unstale":"2024-09-16T05:27:44.094556+0000","last_undegraded":"2024-09-16T05:27:44.094556+0000","last_fullsized":"2024-09-16T05:27:44.094556+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:06.911213+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.911213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:31.723471+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"35'139","reported_seq":1119,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.095158+0000","last_change":"2024-09-16T05:27:42.211155+0000","last_active":"2024-09-16T05:27:44.095158+0000","last_peered":"2024-09-16T05:27:44.095158+0000","last_clean":"2024-09-16T05:27:44.095158+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:27:44.095158+0000","last_undegraded":"2024-09-16T05:27:44.095158+0000","last_fullsized":"2024-09-16T05:27:44.095158+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.211073+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.211073+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:50.447456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57420,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687732,"statfs":{"total":95995035648,"available":95936237568,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285797,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57408,"kb_used_data":8172,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687744,"statfs":{"total":95995035648,"available":95936249856,"internally_reserved":0,"allocated":8368128,"data_stored":5778066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514735,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79620,"kb_used_data":8172,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665532,"statfs":{"total":95995035648,"available":95913504768,"internally_reserved":0,"allocated":8368128,"data_stored":5778066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841774,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57396,"kb_used_data":8160,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687756,"statfs":{"total":95995035648,"available":95936262144,"internally_reserved":0,"allocated":8355840,"data_stored":5770381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1007,"seq":4325032067083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36624,"kb_used_data":8444,"kb_used_omap":38,"kb_used_meta":28121,"kb_avail":93708528,"statfs":{"total":95995035648,"available":95957532672,"internally_reserved":0,"allocated":8646656,"data_stored":6048397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38947,"internal_metadata":28796893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397755,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61588,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683564,"statfs":{"total":95995035648,"available":95931969536,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757502,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65740,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93679412,"statfs":{"total":95995035648,"available":95927717888,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921027,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53268,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691884,"statfs":{"total":95995035648,"available":95940489216,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117256,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61916,"kb_used_data":8516,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683236,"statfs":{"total":95995035648,"available":95931633664,"internally_reserved":0,"allocated":8720384,"data_stored":6121032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240543,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44892,"kb_used_data":8456,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93700260,"statfs":{"total":95995035648,"available":95949066240,"internally_reserved":0,"allocated":8658944,"data_stored":6056040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":988,"seq":4243427688466,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44788,"kb_used_data":8160,"kb_used_omap":22,"kb_used_meta":36585,"kb_avail":93700364,"statfs":{"total":95995035648,"available":95949172736,"internally_reserved":0,"allocated":8355840,"data_stored":5770381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22663,"internal_metadata":37463929},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738647,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53568,"kb_used_data":8492,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691584,"statfs":{"total":95995035648,"available":95940182016,"internally_reserved":0,"allocated":8695808,"data_stored":6105746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:49.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:27:49.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:49 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:27:49.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:49 smithi204 ceph-mon[30933]: 1.b scrub ok 2024-09-16T05:27:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:49 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:27:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:49 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:27:49.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:49 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:27:49.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:49 smithi196 ceph-mon[30802]: 1.b scrub ok 2024-09-16T05:27:49.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:49 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:27:49.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:49 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:27:49.575 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:50.238 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:27:50.239 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:27:50.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:50 smithi032 ceph-mon[27388]: pgmap v1762: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:50.239 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:50 smithi032 ceph-mon[27388]: from='client.18150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:50.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:50 smithi204 ceph-mon[30933]: pgmap v1762: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:50 smithi204 ceph-mon[30933]: from='client.18150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:50.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:50 smithi196 ceph-mon[30802]: pgmap v1762: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:50.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:50 smithi196 ceph-mon[30802]: from='client.18150 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:50.927 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1762,"stamp":"2024-09-16T05:27:48.299288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":674228,"kb_used_data":99308,"kb_used_omap":84,"kb_used_meta":574571,"kb_avail":1124267596,"statfs":{"total":1151940427776,"available":1151250018304,"internally_reserved":0,"allocated":101691392,"data_stored":70570945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86930,"internal_metadata":588360814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.267763"},"pg_stats":[{"pgid":"1.12","version":"35'139","reported_seq":683,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048911+0000","last_change":"2024-09-16T05:27:26.907634+0000","last_active":"2024-09-16T05:27:47.048911+0000","last_peered":"2024-09-16T05:27:47.048911+0000","last_clean":"2024-09-16T05:27:47.048911+0000","last_became_active":"2024-09-16T05:27:26.907217+0000","last_became_peered":"2024-09-16T05:27:26.907217+0000","last_unstale":"2024-09-16T05:27:47.048911+0000","last_undegraded":"2024-09-16T05:27:47.048911+0000","last_fullsized":"2024-09-16T05:27:47.048911+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:28.030079+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:26:28.030079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:48.116060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"35'139","reported_seq":1423,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.094462+0000","last_change":"2024-09-16T05:27:24.731478+0000","last_active":"2024-09-16T05:27:44.094462+0000","last_peered":"2024-09-16T05:27:44.094462+0000","last_clean":"2024-09-16T05:27:44.094462+0000","last_became_active":"2024-09-16T05:26:22.907168+0000","last_became_peered":"2024-09-16T05:26:22.907168+0000","last_unstale":"2024-09-16T05:27:44.094462+0000","last_undegraded":"2024-09-16T05:27:44.094462+0000","last_fullsized":"2024-09-16T05:27:44.094462+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:24.731376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:24.731376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:27.331684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1715,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048928+0000","last_change":"2024-09-16T05:27:38.109553+0000","last_active":"2024-09-16T05:27:47.048928+0000","last_peered":"2024-09-16T05:27:47.048928+0000","last_clean":"2024-09-16T05:27:47.048928+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:27:47.048928+0000","last_undegraded":"2024-09-16T05:27:47.048928+0000","last_fullsized":"2024-09-16T05:27:47.048928+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:38.109468+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:38.109468+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:46.874057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"35'139","reported_seq":1767,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403505+0000","last_change":"2024-09-16T05:27:33.791163+0000","last_active":"2024-09-16T05:27:45.403505+0000","last_peered":"2024-09-16T05:27:45.403505+0000","last_clean":"2024-09-16T05:27:45.403505+0000","last_became_active":"2024-09-16T05:26:58.976632+0000","last_became_peered":"2024-09-16T05:26:58.976632+0000","last_unstale":"2024-09-16T05:27:45.403505+0000","last_undegraded":"2024-09-16T05:27:45.403505+0000","last_fullsized":"2024-09-16T05:27:45.403505+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_clean_scrub_stamp":"2024-09-16T05:27:33.791043+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:01.742303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"35'139","reported_seq":605,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403594+0000","last_change":"2024-09-16T05:27:34.839730+0000","last_active":"2024-09-16T05:27:45.403594+0000","last_peered":"2024-09-16T05:27:45.403594+0000","last_clean":"2024-09-16T05:27:45.403594+0000","last_became_active":"2024-09-16T05:26:58.979090+0000","last_became_peered":"2024-09-16T05:26:58.979090+0000","last_unstale":"2024-09-16T05:27:45.403594+0000","last_undegraded":"2024-09-16T05:27:45.403594+0000","last_fullsized":"2024-09-16T05:27:45.403594+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:34.839637+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:27:34.839637+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:59.564317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"35'139","reported_seq":1851,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403495+0000","last_change":"2024-09-16T05:26:30.986289+0000","last_active":"2024-09-16T05:27:45.403495+0000","last_peered":"2024-09-16T05:27:45.403495+0000","last_clean":"2024-09-16T05:27:45.403495+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:27:45.403495+0000","last_undegraded":"2024-09-16T05:27:45.403495+0000","last_fullsized":"2024-09-16T05:27:45.403495+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.986186+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"35'139","reported_seq":1400,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.094373+0000","last_change":"2024-09-16T05:26:25.745603+0000","last_active":"2024-09-16T05:27:44.094373+0000","last_peered":"2024-09-16T05:27:44.094373+0000","last_clean":"2024-09-16T05:27:44.094373+0000","last_became_active":"2024-09-16T05:26:25.117472+0000","last_became_peered":"2024-09-16T05:26:25.117472+0000","last_unstale":"2024-09-16T05:27:44.094373+0000","last_undegraded":"2024-09-16T05:27:44.094373+0000","last_fullsized":"2024-09-16T05:27:44.094373+0000","mapping_epoch":991,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:25.745532+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:26:25.745532+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:46.993488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"35'139","reported_seq":1409,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173800+0000","last_change":"2024-09-16T05:27:26.597568+0000","last_active":"2024-09-16T05:27:47.173800+0000","last_peered":"2024-09-16T05:27:47.173800+0000","last_clean":"2024-09-16T05:27:47.173800+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:27:47.173800+0000","last_undegraded":"2024-09-16T05:27:47.173800+0000","last_fullsized":"2024-09-16T05:27:47.173800+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:26.597439+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_clean_scrub_stamp":"2024-09-16T05:27:26.597439+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:36.839187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1629,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099245+0000","last_change":"2024-09-16T05:26:50.173116+0000","last_active":"2024-09-16T05:27:45.099245+0000","last_peered":"2024-09-16T05:27:45.099245+0000","last_clean":"2024-09-16T05:27:45.099245+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:27:45.099245+0000","last_undegraded":"2024-09-16T05:27:45.099245+0000","last_fullsized":"2024-09-16T05:27:45.099245+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.173026+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.173026+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.105143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":593,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.155223+0000","last_change":"2024-09-16T05:26:50.546912+0000","last_active":"2024-09-16T05:27:45.155223+0000","last_peered":"2024-09-16T05:27:45.155223+0000","last_clean":"2024-09-16T05:27:45.155223+0000","last_became_active":"2024-09-16T05:25:10.425314+0000","last_became_peered":"2024-09-16T05:25:10.425314+0000","last_unstale":"2024-09-16T05:27:45.155223+0000","last_undegraded":"2024-09-16T05:27:45.155223+0000","last_fullsized":"2024-09-16T05:27:45.155223+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.546823+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:18.515049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'139","reported_seq":2393,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048911+0000","last_change":"2024-09-16T05:27:42.143188+0000","last_active":"2024-09-16T05:27:47.048911+0000","last_peered":"2024-09-16T05:27:47.048911+0000","last_clean":"2024-09-16T05:27:47.048911+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:27:47.048911+0000","last_undegraded":"2024-09-16T05:27:47.048911+0000","last_fullsized":"2024-09-16T05:27:47.048911+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.143106+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:57.339231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"35'139","reported_seq":2348,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173850+0000","last_change":"2024-09-16T05:27:01.775171+0000","last_active":"2024-09-16T05:27:47.173850+0000","last_peered":"2024-09-16T05:27:47.173850+0000","last_clean":"2024-09-16T05:27:47.173850+0000","last_became_active":"2024-09-16T05:26:58.978042+0000","last_became_peered":"2024-09-16T05:26:58.978042+0000","last_unstale":"2024-09-16T05:27:47.173850+0000","last_undegraded":"2024-09-16T05:27:47.173850+0000","last_fullsized":"2024-09-16T05:27:47.173850+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:01.775049+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:01.775049+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:28.973729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"35'139","reported_seq":1650,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099248+0000","last_change":"2024-09-16T05:26:32.280532+0000","last_active":"2024-09-16T05:27:45.099248+0000","last_peered":"2024-09-16T05:27:45.099248+0000","last_clean":"2024-09-16T05:27:45.099248+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:27:45.099248+0000","last_undegraded":"2024-09-16T05:27:45.099248+0000","last_fullsized":"2024-09-16T05:27:45.099248+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:32.280436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.280436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:45.243320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"35'139","reported_seq":1594,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099244+0000","last_change":"2024-09-16T05:26:52.226627+0000","last_active":"2024-09-16T05:27:45.099244+0000","last_peered":"2024-09-16T05:27:45.099244+0000","last_clean":"2024-09-16T05:27:45.099244+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:27:45.099244+0000","last_undegraded":"2024-09-16T05:27:45.099244+0000","last_fullsized":"2024-09-16T05:27:45.099244+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:52.226532+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:52.226532+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:08.001945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1640,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099307+0000","last_change":"2024-09-16T05:26:30.310092+0000","last_active":"2024-09-16T05:27:45.099307+0000","last_peered":"2024-09-16T05:27:45.099307+0000","last_clean":"2024-09-16T05:27:45.099307+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:27:45.099307+0000","last_undegraded":"2024-09-16T05:27:45.099307+0000","last_fullsized":"2024-09-16T05:27:45.099307+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.309950+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.309950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.005293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"35'139","reported_seq":510,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251203+0000","last_change":"2024-09-16T05:27:25.909160+0000","last_active":"2024-09-16T05:27:47.251203+0000","last_peered":"2024-09-16T05:27:47.251203+0000","last_clean":"2024-09-16T05:27:47.251203+0000","last_became_active":"2024-09-16T05:27:25.908795+0000","last_became_peered":"2024-09-16T05:27:25.908795+0000","last_unstale":"2024-09-16T05:27:47.251203+0000","last_undegraded":"2024-09-16T05:27:47.251203+0000","last_fullsized":"2024-09-16T05:27:47.251203+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:45.217959+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:11.129971+0000","last_clean_scrub_stamp":"2024-09-16T05:26:45.217959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:52.673282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'139","reported_seq":2247,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403473+0000","last_change":"2024-09-16T05:27:30.832857+0000","last_active":"2024-09-16T05:27:45.403473+0000","last_peered":"2024-09-16T05:27:45.403473+0000","last_clean":"2024-09-16T05:27:45.403473+0000","last_became_active":"2024-09-16T05:26:59.168051+0000","last_became_peered":"2024-09-16T05:26:59.168051+0000","last_unstale":"2024-09-16T05:27:45.403473+0000","last_undegraded":"2024-09-16T05:27:45.403473+0000","last_fullsized":"2024-09-16T05:27:45.403473+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_clean_scrub_stamp":"2024-09-16T05:27:30.832744+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:40.846360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2383,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403540+0000","last_change":"2024-09-16T05:26:32.294013+0000","last_active":"2024-09-16T05:27:45.403540+0000","last_peered":"2024-09-16T05:27:45.403540+0000","last_clean":"2024-09-16T05:27:45.403540+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:27:45.403540+0000","last_undegraded":"2024-09-16T05:27:45.403540+0000","last_fullsized":"2024-09-16T05:27:45.403540+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:32.293931+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.293931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:49.584816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1869,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.155221+0000","last_change":"2024-09-16T05:27:33.314807+0000","last_active":"2024-09-16T05:27:45.155221+0000","last_peered":"2024-09-16T05:27:45.155221+0000","last_clean":"2024-09-16T05:27:45.155221+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:27:45.155221+0000","last_undegraded":"2024-09-16T05:27:45.155221+0000","last_fullsized":"2024-09-16T05:27:45.155221+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:33.314672+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:27:33.314672+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:33.888940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"35'139","reported_seq":1317,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251096+0000","last_change":"2024-09-16T05:27:26.906821+0000","last_active":"2024-09-16T05:27:47.251096+0000","last_peered":"2024-09-16T05:27:47.251096+0000","last_clean":"2024-09-16T05:27:47.251096+0000","last_became_active":"2024-09-16T05:27:26.906530+0000","last_became_peered":"2024-09-16T05:27:26.906530+0000","last_unstale":"2024-09-16T05:27:47.251096+0000","last_undegraded":"2024-09-16T05:27:47.251096+0000","last_fullsized":"2024-09-16T05:27:47.251096+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:49.297389+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.297389+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:03.261298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"35'139","reported_seq":1264,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.095085+0000","last_change":"2024-09-16T05:27:30.199187+0000","last_active":"2024-09-16T05:27:44.095085+0000","last_peered":"2024-09-16T05:27:44.095085+0000","last_clean":"2024-09-16T05:27:44.095085+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:27:44.095085+0000","last_undegraded":"2024-09-16T05:27:44.095085+0000","last_fullsized":"2024-09-16T05:27:44.095085+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:30.199092+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:27:30.199092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:44.350797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"35'139","reported_seq":1312,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251140+0000","last_change":"2024-09-16T05:26:29.217364+0000","last_active":"2024-09-16T05:27:47.251140+0000","last_peered":"2024-09-16T05:27:47.251140+0000","last_clean":"2024-09-16T05:27:47.251140+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:27:47.251140+0000","last_undegraded":"2024-09-16T05:27:47.251140+0000","last_fullsized":"2024-09-16T05:27:47.251140+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:29.217292+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:26:29.217292+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.502835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":694,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.095116+0000","last_change":"2024-09-16T05:27:25.909318+0000","last_active":"2024-09-16T05:27:44.095116+0000","last_peered":"2024-09-16T05:27:44.095116+0000","last_clean":"2024-09-16T05:27:44.095116+0000","last_became_active":"2024-09-16T05:27:25.908930+0000","last_became_peered":"2024-09-16T05:27:25.908930+0000","last_unstale":"2024-09-16T05:27:44.095116+0000","last_undegraded":"2024-09-16T05:27:44.095116+0000","last_fullsized":"2024-09-16T05:27:44.095116+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:49.474636+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.474636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:55.990716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"35'139","reported_seq":1703,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251099+0000","last_change":"2024-09-16T05:27:42.336327+0000","last_active":"2024-09-16T05:27:47.251099+0000","last_peered":"2024-09-16T05:27:47.251099+0000","last_clean":"2024-09-16T05:27:47.251099+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:27:47.251099+0000","last_undegraded":"2024-09-16T05:27:47.251099+0000","last_fullsized":"2024-09-16T05:27:47.251099+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.336247+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.336247+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:59.548749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1781,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.095127+0000","last_change":"2024-09-16T05:26:33.343104+0000","last_active":"2024-09-16T05:27:44.095127+0000","last_peered":"2024-09-16T05:27:44.095127+0000","last_clean":"2024-09-16T05:27:44.095127+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:27:44.095127+0000","last_undegraded":"2024-09-16T05:27:44.095127+0000","last_fullsized":"2024-09-16T05:27:44.095127+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:33.343016+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:26:33.343016+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:59.079256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"35'139","reported_seq":490,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403472+0000","last_change":"2024-09-16T05:27:05.725386+0000","last_active":"2024-09-16T05:27:45.403472+0000","last_peered":"2024-09-16T05:27:45.403472+0000","last_clean":"2024-09-16T05:27:45.403472+0000","last_became_active":"2024-09-16T05:26:58.979664+0000","last_became_peered":"2024-09-16T05:26:58.979664+0000","last_unstale":"2024-09-16T05:27:45.403472+0000","last_undegraded":"2024-09-16T05:27:45.403472+0000","last_fullsized":"2024-09-16T05:27:45.403472+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:05.725288+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.725288+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:11.443926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"35'139","reported_seq":2374,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099253+0000","last_change":"2024-09-16T05:26:46.145954+0000","last_active":"2024-09-16T05:27:45.099253+0000","last_peered":"2024-09-16T05:27:45.099253+0000","last_clean":"2024-09-16T05:27:45.099253+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:27:45.099253+0000","last_undegraded":"2024-09-16T05:27:45.099253+0000","last_fullsized":"2024-09-16T05:27:45.099253+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:46.145861+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:26:46.145861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.153358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"35'139","reported_seq":515,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251082+0000","last_change":"2024-09-16T05:27:26.909313+0000","last_active":"2024-09-16T05:27:47.251082+0000","last_peered":"2024-09-16T05:27:47.251082+0000","last_clean":"2024-09-16T05:27:47.251082+0000","last_became_active":"2024-09-16T05:27:26.908930+0000","last_became_peered":"2024-09-16T05:27:26.908930+0000","last_unstale":"2024-09-16T05:27:47.251082+0000","last_undegraded":"2024-09-16T05:27:47.251082+0000","last_fullsized":"2024-09-16T05:27:47.251082+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:46.234400+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:46.234400+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:12.024106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1e","version":"35'139","reported_seq":1868,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173850+0000","last_change":"2024-09-16T05:26:50.779260+0000","last_active":"2024-09-16T05:27:47.173850+0000","last_peered":"2024-09-16T05:27:47.173850+0000","last_clean":"2024-09-16T05:27:47.173850+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:27:47.173850+0000","last_undegraded":"2024-09-16T05:27:47.173850+0000","last_fullsized":"2024-09-16T05:27:47.173850+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.779169+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.779169+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:20.159044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"35'139","reported_seq":919,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.094556+0000","last_change":"2024-09-16T05:27:06.911321+0000","last_active":"2024-09-16T05:27:44.094556+0000","last_peered":"2024-09-16T05:27:44.094556+0000","last_clean":"2024-09-16T05:27:44.094556+0000","last_became_active":"2024-09-16T05:26:22.907190+0000","last_became_peered":"2024-09-16T05:26:22.907190+0000","last_unstale":"2024-09-16T05:27:44.094556+0000","last_undegraded":"2024-09-16T05:27:44.094556+0000","last_fullsized":"2024-09-16T05:27:44.094556+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:06.911213+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.911213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:31.723471+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"35'139","reported_seq":1119,"reported_epoch":1039,"state":"active+clean","last_fresh":"2024-09-16T05:27:44.095158+0000","last_change":"2024-09-16T05:27:42.211155+0000","last_active":"2024-09-16T05:27:44.095158+0000","last_peered":"2024-09-16T05:27:44.095158+0000","last_clean":"2024-09-16T05:27:44.095158+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:27:44.095158+0000","last_undegraded":"2024-09-16T05:27:44.095158+0000","last_fullsized":"2024-09-16T05:27:44.095158+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.211073+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.211073+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:50.447456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57420,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687732,"statfs":{"total":95995035648,"available":95936237568,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285797,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57408,"kb_used_data":8172,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687744,"statfs":{"total":95995035648,"available":95936249856,"internally_reserved":0,"allocated":8368128,"data_stored":5778066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514735,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79620,"kb_used_data":8172,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665532,"statfs":{"total":95995035648,"available":95913504768,"internally_reserved":0,"allocated":8368128,"data_stored":5778066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841774,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57396,"kb_used_data":8160,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687756,"statfs":{"total":95995035648,"available":95936262144,"internally_reserved":0,"allocated":8355840,"data_stored":5770381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1007,"seq":4325032067083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36624,"kb_used_data":8444,"kb_used_omap":38,"kb_used_meta":28121,"kb_avail":93708528,"statfs":{"total":95995035648,"available":95957532672,"internally_reserved":0,"allocated":8646656,"data_stored":6048397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38947,"internal_metadata":28796893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397755,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61588,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683564,"statfs":{"total":95995035648,"available":95931969536,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757502,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65740,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93679412,"statfs":{"total":95995035648,"available":95927717888,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921027,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53268,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691884,"statfs":{"total":95995035648,"available":95940489216,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117256,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61916,"kb_used_data":8516,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683236,"statfs":{"total":95995035648,"available":95931633664,"internally_reserved":0,"allocated":8720384,"data_stored":6121032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240543,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44892,"kb_used_data":8456,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93700260,"statfs":{"total":95995035648,"available":95949066240,"internally_reserved":0,"allocated":8658944,"data_stored":6056040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":988,"seq":4243427688466,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44788,"kb_used_data":8160,"kb_used_omap":22,"kb_used_meta":36585,"kb_avail":93700364,"statfs":{"total":95995035648,"available":95949172736,"internally_reserved":0,"allocated":8355840,"data_stored":5770381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22663,"internal_metadata":37463929},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738647,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53568,"kb_used_data":8492,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691584,"statfs":{"total":95995035648,"available":95940182016,"internally_reserved":0,"allocated":8695808,"data_stored":6105746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:50.929 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T05:27:50.930 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T05:27:50.930 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-16T05:27:50.930 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:27:51.187 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:51.339 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:51 smithi032 ceph-mon[27388]: from='client.18156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:51.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:51 smithi204 ceph-mon[30933]: from='client.18156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:51.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:51 smithi196 ceph-mon[30802]: from='client.18156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:27:51.845 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:27:52.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:52 smithi032 ceph-mon[27388]: pgmap v1763: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:52.432 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:52 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:27:52.432 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:52 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:27:52.432 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:52 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:27:52.432 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:52 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:27:52.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:52 smithi204 ceph-mon[30933]: pgmap v1763: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:52.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:52 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:27:52.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:52 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:27:52.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:52 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:27:52.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:52 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:27:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:52 smithi196 ceph-mon[30802]: pgmap v1763: 31 pgs: 31 active+clean; 577 KiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:52 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:27:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:52 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:27:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:52 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:27:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:52 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:27:52.602 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1763,"stamp":"2024-09-16T05:27:50.299673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":674304,"kb_used_data":99380,"kb_used_omap":84,"kb_used_meta":574571,"kb_avail":1124267520,"statfs":{"total":1151940427776,"available":1151249940480,"internally_reserved":0,"allocated":101765120,"data_stored":70616887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86930,"internal_metadata":588360814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.262817"},"pg_stats":[{"pgid":"1.12","version":"35'139","reported_seq":683,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048911+0000","last_change":"2024-09-16T05:27:26.907634+0000","last_active":"2024-09-16T05:27:47.048911+0000","last_peered":"2024-09-16T05:27:47.048911+0000","last_clean":"2024-09-16T05:27:47.048911+0000","last_became_active":"2024-09-16T05:27:26.907217+0000","last_became_peered":"2024-09-16T05:27:26.907217+0000","last_unstale":"2024-09-16T05:27:47.048911+0000","last_undegraded":"2024-09-16T05:27:47.048911+0000","last_fullsized":"2024-09-16T05:27:47.048911+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:28.030079+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:26:28.030079+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:48.116060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"35'139","reported_seq":1427,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173459+0000","last_change":"2024-09-16T05:27:24.731478+0000","last_active":"2024-09-16T05:27:47.173459+0000","last_peered":"2024-09-16T05:27:47.173459+0000","last_clean":"2024-09-16T05:27:47.173459+0000","last_became_active":"2024-09-16T05:26:22.907168+0000","last_became_peered":"2024-09-16T05:26:22.907168+0000","last_unstale":"2024-09-16T05:27:47.173459+0000","last_undegraded":"2024-09-16T05:27:47.173459+0000","last_fullsized":"2024-09-16T05:27:47.173459+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:24.731376+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:24.731376+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:27.331684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1715,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048928+0000","last_change":"2024-09-16T05:27:38.109553+0000","last_active":"2024-09-16T05:27:47.048928+0000","last_peered":"2024-09-16T05:27:47.048928+0000","last_clean":"2024-09-16T05:27:47.048928+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:27:47.048928+0000","last_undegraded":"2024-09-16T05:27:47.048928+0000","last_fullsized":"2024-09-16T05:27:47.048928+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:38.109468+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:38.109468+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:46.874057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"35'139","reported_seq":1767,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403505+0000","last_change":"2024-09-16T05:27:33.791163+0000","last_active":"2024-09-16T05:27:45.403505+0000","last_peered":"2024-09-16T05:27:45.403505+0000","last_clean":"2024-09-16T05:27:45.403505+0000","last_became_active":"2024-09-16T05:26:58.976632+0000","last_became_peered":"2024-09-16T05:26:58.976632+0000","last_unstale":"2024-09-16T05:27:45.403505+0000","last_undegraded":"2024-09-16T05:27:45.403505+0000","last_fullsized":"2024-09-16T05:27:45.403505+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_clean_scrub_stamp":"2024-09-16T05:27:33.791043+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:01.742303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"35'139","reported_seq":605,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403594+0000","last_change":"2024-09-16T05:27:34.839730+0000","last_active":"2024-09-16T05:27:45.403594+0000","last_peered":"2024-09-16T05:27:45.403594+0000","last_clean":"2024-09-16T05:27:45.403594+0000","last_became_active":"2024-09-16T05:26:58.979090+0000","last_became_peered":"2024-09-16T05:26:58.979090+0000","last_unstale":"2024-09-16T05:27:45.403594+0000","last_undegraded":"2024-09-16T05:27:45.403594+0000","last_fullsized":"2024-09-16T05:27:45.403594+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:34.839637+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:27:34.839637+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:59.564317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"35'139","reported_seq":1851,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403495+0000","last_change":"2024-09-16T05:26:30.986289+0000","last_active":"2024-09-16T05:27:45.403495+0000","last_peered":"2024-09-16T05:27:45.403495+0000","last_clean":"2024-09-16T05:27:45.403495+0000","last_became_active":"2024-09-16T05:21:54.938961+0000","last_became_peered":"2024-09-16T05:21:54.938961+0000","last_unstale":"2024-09-16T05:27:45.403495+0000","last_undegraded":"2024-09-16T05:27:45.403495+0000","last_fullsized":"2024-09-16T05:27:45.403495+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.986186+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"35'139","reported_seq":1412,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.927307+0000","last_change":"2024-09-16T05:27:47.927330+0000","last_active":"2024-09-16T05:27:47.927307+0000","last_peered":"2024-09-16T05:27:47.927307+0000","last_clean":"2024-09-16T05:27:47.927307+0000","last_became_active":"2024-09-16T05:26:25.117472+0000","last_became_peered":"2024-09-16T05:26:25.117472+0000","last_unstale":"2024-09-16T05:27:47.927307+0000","last_undegraded":"2024-09-16T05:27:47.927307+0000","last_fullsized":"2024-09-16T05:27:47.927307+0000","mapping_epoch":991,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:47.927225+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:27:47.927225+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:11.188590+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"35'139","reported_seq":1409,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173800+0000","last_change":"2024-09-16T05:27:26.597568+0000","last_active":"2024-09-16T05:27:47.173800+0000","last_peered":"2024-09-16T05:27:47.173800+0000","last_clean":"2024-09-16T05:27:47.173800+0000","last_became_active":"2024-09-16T05:21:54.931983+0000","last_became_peered":"2024-09-16T05:21:54.931983+0000","last_unstale":"2024-09-16T05:27:47.173800+0000","last_undegraded":"2024-09-16T05:27:47.173800+0000","last_fullsized":"2024-09-16T05:27:47.173800+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:26.597439+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_clean_scrub_stamp":"2024-09-16T05:27:26.597439+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:36.839187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1629,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099245+0000","last_change":"2024-09-16T05:26:50.173116+0000","last_active":"2024-09-16T05:27:45.099245+0000","last_peered":"2024-09-16T05:27:45.099245+0000","last_clean":"2024-09-16T05:27:45.099245+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:27:45.099245+0000","last_undegraded":"2024-09-16T05:27:45.099245+0000","last_fullsized":"2024-09-16T05:27:45.099245+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.173026+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.173026+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.105143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":593,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.155223+0000","last_change":"2024-09-16T05:26:50.546912+0000","last_active":"2024-09-16T05:27:45.155223+0000","last_peered":"2024-09-16T05:27:45.155223+0000","last_clean":"2024-09-16T05:27:45.155223+0000","last_became_active":"2024-09-16T05:25:10.425314+0000","last_became_peered":"2024-09-16T05:25:10.425314+0000","last_unstale":"2024-09-16T05:27:45.155223+0000","last_undegraded":"2024-09-16T05:27:45.155223+0000","last_fullsized":"2024-09-16T05:27:45.155223+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.546823+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:18.515049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'139","reported_seq":2393,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.048911+0000","last_change":"2024-09-16T05:27:42.143188+0000","last_active":"2024-09-16T05:27:47.048911+0000","last_peered":"2024-09-16T05:27:47.048911+0000","last_clean":"2024-09-16T05:27:47.048911+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:27:47.048911+0000","last_undegraded":"2024-09-16T05:27:47.048911+0000","last_fullsized":"2024-09-16T05:27:47.048911+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.143106+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:57.339231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"35'139","reported_seq":2348,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173850+0000","last_change":"2024-09-16T05:27:01.775171+0000","last_active":"2024-09-16T05:27:47.173850+0000","last_peered":"2024-09-16T05:27:47.173850+0000","last_clean":"2024-09-16T05:27:47.173850+0000","last_became_active":"2024-09-16T05:26:58.978042+0000","last_became_peered":"2024-09-16T05:26:58.978042+0000","last_unstale":"2024-09-16T05:27:47.173850+0000","last_undegraded":"2024-09-16T05:27:47.173850+0000","last_fullsized":"2024-09-16T05:27:47.173850+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:01.775049+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:01.775049+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:28.973729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"35'139","reported_seq":1650,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099248+0000","last_change":"2024-09-16T05:26:32.280532+0000","last_active":"2024-09-16T05:27:45.099248+0000","last_peered":"2024-09-16T05:27:45.099248+0000","last_clean":"2024-09-16T05:27:45.099248+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:27:45.099248+0000","last_undegraded":"2024-09-16T05:27:45.099248+0000","last_fullsized":"2024-09-16T05:27:45.099248+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:32.280436+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.280436+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:45.243320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"35'139","reported_seq":1594,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099244+0000","last_change":"2024-09-16T05:26:52.226627+0000","last_active":"2024-09-16T05:27:45.099244+0000","last_peered":"2024-09-16T05:27:45.099244+0000","last_clean":"2024-09-16T05:27:45.099244+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:27:45.099244+0000","last_undegraded":"2024-09-16T05:27:45.099244+0000","last_fullsized":"2024-09-16T05:27:45.099244+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:52.226532+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:52.226532+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:08.001945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1640,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099307+0000","last_change":"2024-09-16T05:26:30.310092+0000","last_active":"2024-09-16T05:27:45.099307+0000","last_peered":"2024-09-16T05:27:45.099307+0000","last_clean":"2024-09-16T05:27:45.099307+0000","last_became_active":"2024-09-16T05:21:54.938950+0000","last_became_peered":"2024-09-16T05:21:54.938950+0000","last_unstale":"2024-09-16T05:27:45.099307+0000","last_undegraded":"2024-09-16T05:27:45.099307+0000","last_fullsized":"2024-09-16T05:27:45.099307+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:30.309950+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:26:30.309950+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.005293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"35'139","reported_seq":510,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251203+0000","last_change":"2024-09-16T05:27:25.909160+0000","last_active":"2024-09-16T05:27:47.251203+0000","last_peered":"2024-09-16T05:27:47.251203+0000","last_clean":"2024-09-16T05:27:47.251203+0000","last_became_active":"2024-09-16T05:27:25.908795+0000","last_became_peered":"2024-09-16T05:27:25.908795+0000","last_unstale":"2024-09-16T05:27:47.251203+0000","last_undegraded":"2024-09-16T05:27:47.251203+0000","last_fullsized":"2024-09-16T05:27:47.251203+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:45.217959+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:11.129971+0000","last_clean_scrub_stamp":"2024-09-16T05:26:45.217959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:52.673282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'139","reported_seq":2247,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403473+0000","last_change":"2024-09-16T05:27:30.832857+0000","last_active":"2024-09-16T05:27:45.403473+0000","last_peered":"2024-09-16T05:27:45.403473+0000","last_clean":"2024-09-16T05:27:45.403473+0000","last_became_active":"2024-09-16T05:26:59.168051+0000","last_became_peered":"2024-09-16T05:26:59.168051+0000","last_unstale":"2024-09-16T05:27:45.403473+0000","last_undegraded":"2024-09-16T05:27:45.403473+0000","last_fullsized":"2024-09-16T05:27:45.403473+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_clean_scrub_stamp":"2024-09-16T05:27:30.832744+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:40.846360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2383,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403540+0000","last_change":"2024-09-16T05:26:32.294013+0000","last_active":"2024-09-16T05:27:45.403540+0000","last_peered":"2024-09-16T05:27:45.403540+0000","last_clean":"2024-09-16T05:27:45.403540+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:27:45.403540+0000","last_undegraded":"2024-09-16T05:27:45.403540+0000","last_fullsized":"2024-09-16T05:27:45.403540+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:32.293931+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:26:32.293931+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:49.584816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1869,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.155221+0000","last_change":"2024-09-16T05:27:33.314807+0000","last_active":"2024-09-16T05:27:45.155221+0000","last_peered":"2024-09-16T05:27:45.155221+0000","last_clean":"2024-09-16T05:27:45.155221+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:27:45.155221+0000","last_undegraded":"2024-09-16T05:27:45.155221+0000","last_fullsized":"2024-09-16T05:27:45.155221+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:33.314672+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:27:33.314672+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:33.888940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"35'139","reported_seq":1317,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251096+0000","last_change":"2024-09-16T05:27:26.906821+0000","last_active":"2024-09-16T05:27:47.251096+0000","last_peered":"2024-09-16T05:27:47.251096+0000","last_clean":"2024-09-16T05:27:47.251096+0000","last_became_active":"2024-09-16T05:27:26.906530+0000","last_became_peered":"2024-09-16T05:27:26.906530+0000","last_unstale":"2024-09-16T05:27:47.251096+0000","last_undegraded":"2024-09-16T05:27:47.251096+0000","last_fullsized":"2024-09-16T05:27:47.251096+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:49.297389+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.297389+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:03.261298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"35'139","reported_seq":1268,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.050220+0000","last_change":"2024-09-16T05:27:30.199187+0000","last_active":"2024-09-16T05:27:47.050220+0000","last_peered":"2024-09-16T05:27:47.050220+0000","last_clean":"2024-09-16T05:27:47.050220+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:27:47.050220+0000","last_undegraded":"2024-09-16T05:27:47.050220+0000","last_fullsized":"2024-09-16T05:27:47.050220+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:30.199092+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:27:30.199092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:44.350797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"35'139","reported_seq":1312,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251140+0000","last_change":"2024-09-16T05:26:29.217364+0000","last_active":"2024-09-16T05:27:47.251140+0000","last_peered":"2024-09-16T05:27:47.251140+0000","last_clean":"2024-09-16T05:27:47.251140+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:27:47.251140+0000","last_undegraded":"2024-09-16T05:27:47.251140+0000","last_fullsized":"2024-09-16T05:27:47.251140+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:29.217292+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:26:29.217292+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:57.502835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":698,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.050187+0000","last_change":"2024-09-16T05:27:25.909318+0000","last_active":"2024-09-16T05:27:47.050187+0000","last_peered":"2024-09-16T05:27:47.050187+0000","last_clean":"2024-09-16T05:27:47.050187+0000","last_became_active":"2024-09-16T05:27:25.908930+0000","last_became_peered":"2024-09-16T05:27:25.908930+0000","last_unstale":"2024-09-16T05:27:47.050187+0000","last_undegraded":"2024-09-16T05:27:47.050187+0000","last_fullsized":"2024-09-16T05:27:47.050187+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:49.474636+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:26:49.474636+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:55.990716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"35'139","reported_seq":1703,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251099+0000","last_change":"2024-09-16T05:27:42.336327+0000","last_active":"2024-09-16T05:27:47.251099+0000","last_peered":"2024-09-16T05:27:47.251099+0000","last_clean":"2024-09-16T05:27:47.251099+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:27:47.251099+0000","last_undegraded":"2024-09-16T05:27:47.251099+0000","last_fullsized":"2024-09-16T05:27:47.251099+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.336247+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.336247+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:59.548749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1785,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.050177+0000","last_change":"2024-09-16T05:26:33.343104+0000","last_active":"2024-09-16T05:27:47.050177+0000","last_peered":"2024-09-16T05:27:47.050177+0000","last_clean":"2024-09-16T05:27:47.050177+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:27:47.050177+0000","last_undegraded":"2024-09-16T05:27:47.050177+0000","last_fullsized":"2024-09-16T05:27:47.050177+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:33.343016+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:26:33.343016+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:59.079256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"35'139","reported_seq":490,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.403472+0000","last_change":"2024-09-16T05:27:05.725386+0000","last_active":"2024-09-16T05:27:45.403472+0000","last_peered":"2024-09-16T05:27:45.403472+0000","last_clean":"2024-09-16T05:27:45.403472+0000","last_became_active":"2024-09-16T05:26:58.979664+0000","last_became_peered":"2024-09-16T05:26:58.979664+0000","last_unstale":"2024-09-16T05:27:45.403472+0000","last_undegraded":"2024-09-16T05:27:45.403472+0000","last_fullsized":"2024-09-16T05:27:45.403472+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:05.725288+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:27:05.725288+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:11.443926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"35'139","reported_seq":2374,"reported_epoch":1040,"state":"active+clean","last_fresh":"2024-09-16T05:27:45.099253+0000","last_change":"2024-09-16T05:26:46.145954+0000","last_active":"2024-09-16T05:27:45.099253+0000","last_peered":"2024-09-16T05:27:45.099253+0000","last_clean":"2024-09-16T05:27:45.099253+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:27:45.099253+0000","last_undegraded":"2024-09-16T05:27:45.099253+0000","last_fullsized":"2024-09-16T05:27:45.099253+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:46.145861+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:26:46.145861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:27:54.153358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"35'139","reported_seq":515,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.251082+0000","last_change":"2024-09-16T05:27:26.909313+0000","last_active":"2024-09-16T05:27:47.251082+0000","last_peered":"2024-09-16T05:27:47.251082+0000","last_clean":"2024-09-16T05:27:47.251082+0000","last_became_active":"2024-09-16T05:27:26.908930+0000","last_became_peered":"2024-09-16T05:27:26.908930+0000","last_unstale":"2024-09-16T05:27:47.251082+0000","last_undegraded":"2024-09-16T05:27:47.251082+0000","last_fullsized":"2024-09-16T05:27:47.251082+0000","mapping_epoch":1029,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:46.234400+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:26:46.234400+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:12.024106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1e","version":"35'139","reported_seq":1868,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173850+0000","last_change":"2024-09-16T05:26:50.779260+0000","last_active":"2024-09-16T05:27:47.173850+0000","last_peered":"2024-09-16T05:27:47.173850+0000","last_clean":"2024-09-16T05:27:47.173850+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:27:47.173850+0000","last_undegraded":"2024-09-16T05:27:47.173850+0000","last_fullsized":"2024-09-16T05:27:47.173850+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:26:50.779169+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:26:50.779169+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:20.159044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"35'139","reported_seq":923,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.173365+0000","last_change":"2024-09-16T05:27:06.911321+0000","last_active":"2024-09-16T05:27:47.173365+0000","last_peered":"2024-09-16T05:27:47.173365+0000","last_clean":"2024-09-16T05:27:47.173365+0000","last_became_active":"2024-09-16T05:26:22.907190+0000","last_became_peered":"2024-09-16T05:26:22.907190+0000","last_unstale":"2024-09-16T05:27:47.173365+0000","last_undegraded":"2024-09-16T05:27:47.173365+0000","last_fullsized":"2024-09-16T05:27:47.173365+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:06.911213+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:06.911213+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:31.723471+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"35'139","reported_seq":1123,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-16T05:27:47.050276+0000","last_change":"2024-09-16T05:27:42.211155+0000","last_active":"2024-09-16T05:27:47.050276+0000","last_peered":"2024-09-16T05:27:47.050276+0000","last_clean":"2024-09-16T05:27:47.050276+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:27:47.050276+0000","last_undegraded":"2024-09-16T05:27:47.050276+0000","last_fullsized":"2024-09-16T05:27:47.050276+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:42.211073+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:27:42.211073+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:28:50.447456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57420,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687732,"statfs":{"total":95995035648,"available":95936237568,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285797,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57408,"kb_used_data":8172,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687744,"statfs":{"total":95995035648,"available":95936249856,"internally_reserved":0,"allocated":8368128,"data_stored":5778066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514735,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79620,"kb_used_data":8172,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665532,"statfs":{"total":95995035648,"available":95913504768,"internally_reserved":0,"allocated":8368128,"data_stored":5778066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841775,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57436,"kb_used_data":8196,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687716,"statfs":{"total":95995035648,"available":95936221184,"internally_reserved":0,"allocated":8392704,"data_stored":5793352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1007,"seq":4325032067083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36624,"kb_used_data":8444,"kb_used_omap":38,"kb_used_meta":28121,"kb_avail":93708528,"statfs":{"total":95995035648,"available":95957532672,"internally_reserved":0,"allocated":8646656,"data_stored":6048397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38947,"internal_metadata":28796893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397755,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61588,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683564,"statfs":{"total":95995035648,"available":95931969536,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757502,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65740,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93679412,"statfs":{"total":95995035648,"available":95927717888,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921027,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53268,"kb_used_data":8184,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691884,"statfs":{"total":95995035648,"available":95940489216,"internally_reserved":0,"allocated":8380416,"data_stored":5785709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117256,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61916,"kb_used_data":8516,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683236,"statfs":{"total":95995035648,"available":95931633664,"internally_reserved":0,"allocated":8720384,"data_stored":6121032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240543,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44892,"kb_used_data":8456,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93700260,"statfs":{"total":95995035648,"available":95949066240,"internally_reserved":0,"allocated":8658944,"data_stored":6056040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":988,"seq":4243427688467,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44824,"kb_used_data":8196,"kb_used_omap":22,"kb_used_meta":36585,"kb_avail":93700328,"statfs":{"total":95995035648,"available":95949135872,"internally_reserved":0,"allocated":8392704,"data_stored":5793352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22663,"internal_metadata":37463929},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738647,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53568,"kb_used_data":8492,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691584,"statfs":{"total":95995035648,"available":95940182016,"internally_reserved":0,"allocated":8695808,"data_stored":6105746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:27:52.604 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-16T05:27:52.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 51 2024-09-16T05:27:52.861 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:27:53.241 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:53 smithi032 ceph-mon[27388]: from='client.18162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:53.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:53 smithi204 ceph-mon[30933]: from='client.18162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:53.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:53 smithi196 ceph-mon[30802]: from='client.18162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:27:53.513 INFO:teuthology.orchestra.run.smithi032.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-09-16T05:27:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:54 smithi032 ceph-mon[27388]: pgmap v1764: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:54.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1691905880' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-16T05:27:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T05:27:54.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:54 smithi204 ceph-mon[30933]: pgmap v1764: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:54.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1691905880' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-16T05:27:54.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:54 smithi196 ceph-mon[30802]: pgmap v1764: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:54.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1691905880' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-09-16T05:27:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:56 smithi032 ceph-mon[27388]: pgmap v1765: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:56 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:27:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:56 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:27:56.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:56 smithi032 ceph-mon[27388]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-16T05:27:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:56 smithi204 ceph-mon[30933]: pgmap v1765: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:56 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:27:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:56 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:27:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:56 smithi204 ceph-mon[30933]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-16T05:27:56.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:56 smithi196 ceph-mon[30802]: pgmap v1765: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:56 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:27:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:56 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:27:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:56 smithi196 ceph-mon[30802]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-16T05:27:57.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:57 smithi032 ceph-mon[27388]: 1.15 scrub starts 2024-09-16T05:27:57.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:57 smithi032 ceph-mon[27388]: 1.15 scrub ok 2024-09-16T05:27:57.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:57 smithi204 ceph-mon[30933]: 1.15 scrub starts 2024-09-16T05:27:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:57 smithi204 ceph-mon[30933]: 1.15 scrub ok 2024-09-16T05:27:57.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:57 smithi196 ceph-mon[30802]: 1.15 scrub starts 2024-09-16T05:27:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:57 smithi196 ceph-mon[30802]: 1.15 scrub ok 2024-09-16T05:27:58.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:27:58 smithi032 ceph-mon[27388]: pgmap v1767: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:58.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:27:58 smithi204 ceph-mon[30933]: pgmap v1767: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:58.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:27:58 smithi196 ceph-mon[30802]: pgmap v1767: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:27:59.246 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-16T05:27:59.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 51 2024-09-16T05:27:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:00 smithi032 ceph-mon[27388]: pgmap v1768: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:00 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:28:00.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:00 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:28:00.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:00 smithi204 ceph-mon[30933]: pgmap v1768: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:00.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:00 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:28:00.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:00 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:28:00.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:00 smithi196 ceph-mon[30802]: pgmap v1768: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:00.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:00 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:28:00.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:00 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:28:01.047 INFO:teuthology.orchestra.run.smithi032.stderr:Error EINVAL: specified pgp_num 51 > pg_num 31 2024-09-16T05:28:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:01 smithi032 ceph-mon[27388]: 1.19 scrub starts 2024-09-16T05:28:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:01 smithi032 ceph-mon[27388]: 1.19 scrub ok 2024-09-16T05:28:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:01 smithi032 ceph-mon[27388]: 1.1b scrub starts 2024-09-16T05:28:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:01 smithi032 ceph-mon[27388]: 1.1b scrub ok 2024-09-16T05:28:01.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:01 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3688547599' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-09-16T05:28:01.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:01 smithi204 ceph-mon[30933]: 1.19 scrub starts 2024-09-16T05:28:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:01 smithi204 ceph-mon[30933]: 1.19 scrub ok 2024-09-16T05:28:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:01 smithi204 ceph-mon[30933]: 1.1b scrub starts 2024-09-16T05:28:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:01 smithi204 ceph-mon[30933]: 1.1b scrub ok 2024-09-16T05:28:01.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:01 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3688547599' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-09-16T05:28:01.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:01 smithi196 ceph-mon[30802]: 1.19 scrub starts 2024-09-16T05:28:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:01 smithi196 ceph-mon[30802]: 1.19 scrub ok 2024-09-16T05:28:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:01 smithi196 ceph-mon[30802]: 1.1b scrub starts 2024-09-16T05:28:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:01 smithi196 ceph-mon[30802]: 1.1b scrub ok 2024-09-16T05:28:01.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:01 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3688547599' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-09-16T05:28:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T05:28:01.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T05:28:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:02.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:02 smithi032 ceph-mon[27388]: pgmap v1769: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:02.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:02 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:28:02.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:02 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:28:02.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:02 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:28:02.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:02 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:28:02.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:02 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:02.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:02 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:02.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:02 smithi204 ceph-mon[30933]: pgmap v1769: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:02.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:02 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:28:02.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:02 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:28:02.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:02 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:28:02.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:02 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:28:02.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:02 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:02.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:02 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:02.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:02 smithi196 ceph-mon[30802]: pgmap v1769: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:02.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:02 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:28:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:02 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:28:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:02 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:28:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:02 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:28:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:02 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:02 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:02.549 INFO:teuthology.orchestra.run.smithi032.stdout:mon.a: {} 2024-09-16T05:28:02.550 INFO:teuthology.orchestra.run.smithi032.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T05:28:02.564 INFO:teuthology.orchestra.run.smithi032.stdout:mon.c: {} 2024-09-16T05:28:02.564 INFO:teuthology.orchestra.run.smithi032.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T05:28:02.579 INFO:teuthology.orchestra.run.smithi032.stdout:mon.b: {} 2024-09-16T05:28:02.579 INFO:teuthology.orchestra.run.smithi032.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T05:28:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:03 smithi032 ceph-mon[27388]: osdmap e1043: 12 total, 12 up, 12 in 2024-09-16T05:28:03.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:03 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2458853521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:28:03.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-16T05:28:03.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:03 smithi204 ceph-mon[30933]: osdmap e1043: 12 total, 12 up, 12 in 2024-09-16T05:28:03.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:03 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2458853521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:28:03.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:03 smithi196 ceph-mon[30802]: osdmap e1043: 12 total, 12 up, 12 in 2024-09-16T05:28:03.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:03 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2458853521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T05:28:03.556 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:04 smithi032 ceph-mon[27388]: pgmap v1771: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:04 smithi032 ceph-mon[27388]: 1.16 scrub starts 2024-09-16T05:28:04.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:04 smithi032 ceph-mon[27388]: 1.16 scrub ok 2024-09-16T05:28:04.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:04 smithi204 ceph-mon[30933]: pgmap v1771: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:04 smithi204 ceph-mon[30933]: 1.16 scrub starts 2024-09-16T05:28:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:04 smithi204 ceph-mon[30933]: 1.16 scrub ok 2024-09-16T05:28:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:04 smithi196 ceph-mon[30802]: pgmap v1771: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:04 smithi196 ceph-mon[30802]: 1.16 scrub starts 2024-09-16T05:28:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:04 smithi196 ceph-mon[30802]: 1.16 scrub ok 2024-09-16T05:28:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T05:28:05.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:05 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3014050055' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:28:05.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:05 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3014050055' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:28:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:05 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3014050055' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:28:05.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-16T05:28:06.154 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:06.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:06 smithi032 ceph-mon[27388]: pgmap v1772: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:06.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:06 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3014050055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:28:06.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:06 smithi032 ceph-mon[27388]: osdmap e1044: 12 total, 12 up, 12 in 2024-09-16T05:28:06.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:06 smithi032 ceph-mon[27388]: 1.10 scrub starts 2024-09-16T05:28:06.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:06 smithi032 ceph-mon[27388]: 1.10 scrub ok 2024-09-16T05:28:06.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:06 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:06.412 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:06 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:06.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:06 smithi204 ceph-mon[30933]: pgmap v1772: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:06.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:06 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3014050055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:28:06.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:06 smithi204 ceph-mon[30933]: osdmap e1044: 12 total, 12 up, 12 in 2024-09-16T05:28:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:06 smithi204 ceph-mon[30933]: 1.10 scrub starts 2024-09-16T05:28:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:06 smithi204 ceph-mon[30933]: 1.10 scrub ok 2024-09-16T05:28:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:06 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:06.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:06 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:06.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:06 smithi196 ceph-mon[30802]: pgmap v1772: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:06.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:06 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3014050055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:28:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:06 smithi196 ceph-mon[30802]: osdmap e1044: 12 total, 12 up, 12 in 2024-09-16T05:28:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:06 smithi196 ceph-mon[30802]: 1.10 scrub starts 2024-09-16T05:28:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:06 smithi196 ceph-mon[30802]: 1.10 scrub ok 2024-09-16T05:28:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:06 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:06.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:06 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:07.124 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T05:28:07.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:07 smithi032 ceph-mon[27388]: osdmap e1045: 12 total, 12 up, 12 in 2024-09-16T05:28:07.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:07 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2724483651' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:28:07.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:07 smithi204 ceph-mon[30933]: osdmap e1045: 12 total, 12 up, 12 in 2024-09-16T05:28:07.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:07 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2724483651' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:28:07.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:07 smithi196 ceph-mon[30802]: osdmap e1045: 12 total, 12 up, 12 in 2024-09-16T05:28:07.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:07 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2724483651' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-16T05:28:07.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-16T05:28:08.237 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:08.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:08 smithi032 ceph-mon[27388]: pgmap v1775: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:08.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:08 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2724483651' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:28:08.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:08 smithi032 ceph-mon[27388]: osdmap e1046: 12 total, 12 up, 12 in 2024-09-16T05:28:08.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:08 smithi204 ceph-mon[30933]: pgmap v1775: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:08.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:08 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2724483651' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:28:08.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:08 smithi204 ceph-mon[30933]: osdmap e1046: 12 total, 12 up, 12 in 2024-09-16T05:28:08.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:08 smithi196 ceph-mon[30802]: pgmap v1775: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:08.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:08 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2724483651' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-16T05:28:08.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:08 smithi196 ceph-mon[30802]: osdmap e1046: 12 total, 12 up, 12 in 2024-09-16T05:28:09.131 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T05:28:09.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:09 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:28:09.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:09 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:28:09.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:09 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3262129668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:28:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:09 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:28:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:09 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:28:09.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:09 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3262129668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:28:09.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:09 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:28:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:09 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:28:09.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:09 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3262129668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:28:09.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-16T05:28:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:10 smithi032 ceph-mon[27388]: pgmap v1777: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3262129668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:28:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:10 smithi032 ceph-mon[27388]: osdmap e1047: 12 total, 12 up, 12 in 2024-09-16T05:28:10.205 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:10 smithi196 ceph-mon[30802]: pgmap v1777: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3262129668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:28:10.389 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:10 smithi196 ceph-mon[30802]: osdmap e1047: 12 total, 12 up, 12 in 2024-09-16T05:28:10.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:10 smithi204 ceph-mon[30933]: pgmap v1777: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:10.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3262129668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:28:10.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:10 smithi204 ceph-mon[30933]: osdmap e1047: 12 total, 12 up, 12 in 2024-09-16T05:28:11.141 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T05:28:11.419 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:11 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/39038053' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:28:11.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:11 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/39038053' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:28:11.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:11 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/39038053' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-16T05:28:11.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-16T05:28:12.155 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:12 smithi032 ceph-mon[27388]: pgmap v1779: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/39038053' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:28:12.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:12 smithi032 ceph-mon[27388]: osdmap e1048: 12 total, 12 up, 12 in 2024-09-16T05:28:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:12 smithi204 ceph-mon[30933]: pgmap v1779: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/39038053' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:28:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:12 smithi204 ceph-mon[30933]: osdmap e1048: 12 total, 12 up, 12 in 2024-09-16T05:28:12.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:12 smithi196 ceph-mon[30802]: pgmap v1779: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/39038053' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-16T05:28:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:12 smithi196 ceph-mon[30802]: osdmap e1048: 12 total, 12 up, 12 in 2024-09-16T05:28:13.144 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T05:28:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:13 smithi032 ceph-mon[27388]: 1.1a scrub starts 2024-09-16T05:28:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:13 smithi032 ceph-mon[27388]: 1.1a scrub ok 2024-09-16T05:28:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:13 smithi032 ceph-mon[27388]: 1.18 deep-scrub starts 2024-09-16T05:28:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:13 smithi032 ceph-mon[27388]: 1.18 deep-scrub ok 2024-09-16T05:28:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2073352108' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:28:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:13 smithi204 ceph-mon[30933]: 1.1a scrub starts 2024-09-16T05:28:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:13 smithi204 ceph-mon[30933]: 1.1a scrub ok 2024-09-16T05:28:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:13 smithi204 ceph-mon[30933]: 1.18 deep-scrub starts 2024-09-16T05:28:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:13 smithi204 ceph-mon[30933]: 1.18 deep-scrub ok 2024-09-16T05:28:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2073352108' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:28:13.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:13 smithi196 ceph-mon[30802]: 1.1a scrub starts 2024-09-16T05:28:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:13 smithi196 ceph-mon[30802]: 1.1a scrub ok 2024-09-16T05:28:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:13 smithi196 ceph-mon[30802]: 1.18 deep-scrub starts 2024-09-16T05:28:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:13 smithi196 ceph-mon[30802]: 1.18 deep-scrub ok 2024-09-16T05:28:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2073352108' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:28:13.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-16T05:28:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:14 smithi032 ceph-mon[27388]: pgmap v1781: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:14 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2073352108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:28:14.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:14 smithi032 ceph-mon[27388]: osdmap e1049: 12 total, 12 up, 12 in 2024-09-16T05:28:14.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:14 smithi204 ceph-mon[30933]: pgmap v1781: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:14 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2073352108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:28:14.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:14 smithi204 ceph-mon[30933]: osdmap e1049: 12 total, 12 up, 12 in 2024-09-16T05:28:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:14 smithi196 ceph-mon[30802]: pgmap v1781: 31 pgs: 31 active+clean; 577 KiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:14 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2073352108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:28:14.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:14 smithi196 ceph-mon[30802]: osdmap e1049: 12 total, 12 up, 12 in 2024-09-16T05:28:15.155 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T05:28:15.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1390445871' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:28:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1390445871' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:28:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1390445871' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-16T05:28:15.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-16T05:28:16.162 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:16.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:16 smithi032 ceph-mon[27388]: pgmap v1783: 31 pgs: 31 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:16.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:16 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1390445871' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:28:16.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:16 smithi032 ceph-mon[27388]: osdmap e1050: 12 total, 12 up, 12 in 2024-09-16T05:28:16.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:16 smithi204 ceph-mon[30933]: pgmap v1783: 31 pgs: 31 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:16.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:16 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1390445871' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:28:16.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:16 smithi204 ceph-mon[30933]: osdmap e1050: 12 total, 12 up, 12 in 2024-09-16T05:28:16.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:16 smithi196 ceph-mon[30802]: pgmap v1783: 31 pgs: 31 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:16 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1390445871' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-16T05:28:16.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:16 smithi196 ceph-mon[30802]: osdmap e1050: 12 total, 12 up, 12 in 2024-09-16T05:28:17.165 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T05:28:17.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:17 smithi032 ceph-mon[27388]: osdmap e1051: 12 total, 12 up, 12 in 2024-09-16T05:28:17.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:17 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:17.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:17 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:17.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1687764862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:28:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:17 smithi204 ceph-mon[30933]: osdmap e1051: 12 total, 12 up, 12 in 2024-09-16T05:28:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:17 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:17 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1687764862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:28:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:17 smithi196 ceph-mon[30802]: osdmap e1051: 12 total, 12 up, 12 in 2024-09-16T05:28:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:17 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:17 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:17.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1687764862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:28:17.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-16T05:28:18.212 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:18.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:18 smithi032 ceph-mon[27388]: pgmap v1786: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:18.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:18 smithi032 ceph-mon[27388]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-16T05:28:18.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:18 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1687764862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:28:18.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:18 smithi032 ceph-mon[27388]: osdmap e1052: 12 total, 12 up, 12 in 2024-09-16T05:28:18.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:18 smithi196 ceph-mon[30802]: pgmap v1786: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:18 smithi196 ceph-mon[30802]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-16T05:28:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:18 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1687764862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:28:18.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:18 smithi196 ceph-mon[30802]: osdmap e1052: 12 total, 12 up, 12 in 2024-09-16T05:28:18.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:18 smithi204 ceph-mon[30933]: pgmap v1786: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:18.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:18 smithi204 ceph-mon[30933]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-16T05:28:18.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:18 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1687764862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:28:18.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:18 smithi204 ceph-mon[30933]: osdmap e1052: 12 total, 12 up, 12 in 2024-09-16T05:28:19.168 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T05:28:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:19 smithi032 ceph-mon[27388]: 1.2 scrub starts 2024-09-16T05:28:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:19 smithi032 ceph-mon[27388]: 1.2 scrub ok 2024-09-16T05:28:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:19 smithi032 ceph-mon[27388]: osdmap e1053: 12 total, 12 up, 12 in 2024-09-16T05:28:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/926652860' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:28:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:19 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/926652860' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:28:19.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:19 smithi032 ceph-mon[27388]: osdmap e1054: 12 total, 12 up, 12 in 2024-09-16T05:28:19.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:19 smithi196 ceph-mon[30802]: 1.2 scrub starts 2024-09-16T05:28:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:19 smithi196 ceph-mon[30802]: 1.2 scrub ok 2024-09-16T05:28:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:19 smithi196 ceph-mon[30802]: osdmap e1053: 12 total, 12 up, 12 in 2024-09-16T05:28:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/926652860' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:28:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:19 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/926652860' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:28:19.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:19 smithi196 ceph-mon[30802]: osdmap e1054: 12 total, 12 up, 12 in 2024-09-16T05:28:19.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:19 smithi204 ceph-mon[30933]: 1.2 scrub starts 2024-09-16T05:28:19.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:19 smithi204 ceph-mon[30933]: 1.2 scrub ok 2024-09-16T05:28:19.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:19 smithi204 ceph-mon[30933]: osdmap e1053: 12 total, 12 up, 12 in 2024-09-16T05:28:19.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/926652860' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-16T05:28:19.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:19 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/926652860' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-16T05:28:19.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:19 smithi204 ceph-mon[30933]: osdmap e1054: 12 total, 12 up, 12 in 2024-09-16T05:28:19.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-16T05:28:20.175 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:20.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:20 smithi204 ceph-mon[30933]: pgmap v1789: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:20.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:20 smithi204 ceph-mon[30933]: osdmap e1055: 12 total, 12 up, 12 in 2024-09-16T05:28:20.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:20 smithi032 ceph-mon[27388]: pgmap v1789: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:20.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:20 smithi032 ceph-mon[27388]: osdmap e1055: 12 total, 12 up, 12 in 2024-09-16T05:28:20.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:20 smithi196 ceph-mon[30802]: pgmap v1789: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:20.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:20 smithi196 ceph-mon[30802]: osdmap e1055: 12 total, 12 up, 12 in 2024-09-16T05:28:20.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:28:20.863+0000 7f5d777fe640 0 --2- 172.21.15.32:0/2380381423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d78102da0 0x7f5d78103180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:28:21.188 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T05:28:21.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1110106152' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:28:21.492 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1110106152' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:28:21.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1110106152' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:28:21.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-16T05:28:22.163 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:22.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:22 smithi032 ceph-mon[27388]: pgmap v1792: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:22.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:22 smithi032 ceph-mon[27388]: 1.1e scrub starts 2024-09-16T05:28:22.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:22 smithi032 ceph-mon[27388]: 1.1e scrub ok 2024-09-16T05:28:22.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1110106152' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:28:22.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:22 smithi032 ceph-mon[27388]: osdmap e1056: 12 total, 12 up, 12 in 2024-09-16T05:28:22.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:22 smithi196 ceph-mon[30802]: pgmap v1792: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:22.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:22 smithi196 ceph-mon[30802]: 1.1e scrub starts 2024-09-16T05:28:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:22 smithi196 ceph-mon[30802]: 1.1e scrub ok 2024-09-16T05:28:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1110106152' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:28:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:22 smithi196 ceph-mon[30802]: osdmap e1056: 12 total, 12 up, 12 in 2024-09-16T05:28:22.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:22 smithi204 ceph-mon[30933]: pgmap v1792: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:22.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:22 smithi204 ceph-mon[30933]: 1.1e scrub starts 2024-09-16T05:28:22.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:22 smithi204 ceph-mon[30933]: 1.1e scrub ok 2024-09-16T05:28:22.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1110106152' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:28:22.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:22 smithi204 ceph-mon[30933]: osdmap e1056: 12 total, 12 up, 12 in 2024-09-16T05:28:23.197 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T05:28:23.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:23 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2263408466' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:28:23.500 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:23 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2263408466' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:28:23.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:23 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2263408466' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-16T05:28:23.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-16T05:28:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:24.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:24 smithi196 ceph-mon[30802]: pgmap v1794: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:24 smithi196 ceph-mon[30802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-16T05:28:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:24 smithi196 ceph-mon[30802]: Cluster is now healthy 2024-09-16T05:28:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2263408466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:28:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:24 smithi196 ceph-mon[30802]: osdmap e1057: 12 total, 12 up, 12 in 2024-09-16T05:28:24.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:24 smithi032 ceph-mon[27388]: pgmap v1794: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:24.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:24 smithi032 ceph-mon[27388]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-16T05:28:24.505 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:24 smithi032 ceph-mon[27388]: Cluster is now healthy 2024-09-16T05:28:24.505 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2263408466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:28:24.505 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:24 smithi032 ceph-mon[27388]: osdmap e1057: 12 total, 12 up, 12 in 2024-09-16T05:28:24.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:24 smithi204 ceph-mon[30933]: pgmap v1794: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:24 smithi204 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-16T05:28:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:24 smithi204 ceph-mon[30933]: Cluster is now healthy 2024-09-16T05:28:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2263408466' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-16T05:28:24.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:24 smithi204 ceph-mon[30933]: osdmap e1057: 12 total, 12 up, 12 in 2024-09-16T05:28:25.201 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T05:28:25.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:25 smithi196 ceph-mon[30802]: osdmap e1058: 12 total, 12 up, 12 in 2024-09-16T05:28:25.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:25 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2763766118' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:28:25.524 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:25 smithi032 ceph-mon[27388]: osdmap e1058: 12 total, 12 up, 12 in 2024-09-16T05:28:25.525 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:25 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2763766118' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:28:25.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:25 smithi204 ceph-mon[30933]: osdmap e1058: 12 total, 12 up, 12 in 2024-09-16T05:28:25.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:25 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2763766118' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:28:25.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 8 1 2024-09-16T05:28:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:26 smithi196 ceph-mon[30802]: pgmap v1797: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:26 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:26 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:26 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2763766118' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:28:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:26 smithi196 ceph-mon[30802]: osdmap e1059: 12 total, 12 up, 12 in 2024-09-16T05:28:26.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:26 smithi032 ceph-mon[27388]: pgmap v1797: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:26.538 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:26 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:26.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:26 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:26.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:26 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2763766118' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:28:26.539 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:26 smithi032 ceph-mon[27388]: osdmap e1059: 12 total, 12 up, 12 in 2024-09-16T05:28:26.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:26 smithi204 ceph-mon[30933]: pgmap v1797: 31 pgs: 3 peering, 28 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:26.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:26 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:26 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:26 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2763766118' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:28:26.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:26 smithi204 ceph-mon[30933]: osdmap e1059: 12 total, 12 up, 12 in 2024-09-16T05:28:27.214 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-16T05:28:27.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:27 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2475593775' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:28:27.583 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:27 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2475593775' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:28:27.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:27 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2475593775' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-09-16T05:28:28.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-16T05:28:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:28.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:28 smithi032 ceph-mon[27388]: pgmap v1799: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:28.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:28 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:28.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:28 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:28.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:28 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2475593775' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:28:28.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:28 smithi032 ceph-mon[27388]: osdmap e1060: 12 total, 12 up, 12 in 2024-09-16T05:28:28.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:28 smithi196 ceph-mon[30802]: pgmap v1799: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:28 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:28 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:28 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2475593775' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:28:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:28 smithi196 ceph-mon[30802]: osdmap e1060: 12 total, 12 up, 12 in 2024-09-16T05:28:28.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:28 smithi204 ceph-mon[30933]: pgmap v1799: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:28.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:28 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:28 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:28 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2475593775' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-16T05:28:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:28 smithi204 ceph-mon[30933]: osdmap e1060: 12 total, 12 up, 12 in 2024-09-16T05:28:29.220 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T05:28:29.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/280421507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:28:29.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:29 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:28:29.582 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/280421507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:28:29.583 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:29 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:28:29.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/280421507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:28:29.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:29 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:28:30.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 9 1 2024-09-16T05:28:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:30.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:30 smithi032 ceph-mon[27388]: pgmap v1801: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:30.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:30 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/280421507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:28:30.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:30 smithi032 ceph-mon[27388]: osdmap e1061: 12 total, 12 up, 12 in 2024-09-16T05:28:30.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:30 smithi032 ceph-mon[27388]: 1.13 scrub starts 2024-09-16T05:28:30.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:30 smithi032 ceph-mon[27388]: 1.13 scrub ok 2024-09-16T05:28:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:30 smithi196 ceph-mon[30802]: pgmap v1801: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:30 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/280421507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:28:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:30 smithi196 ceph-mon[30802]: osdmap e1061: 12 total, 12 up, 12 in 2024-09-16T05:28:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:30 smithi196 ceph-mon[30802]: 1.13 scrub starts 2024-09-16T05:28:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:30 smithi196 ceph-mon[30802]: 1.13 scrub ok 2024-09-16T05:28:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:30 smithi204 ceph-mon[30933]: pgmap v1801: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:30 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/280421507' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:28:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:30 smithi204 ceph-mon[30933]: osdmap e1061: 12 total, 12 up, 12 in 2024-09-16T05:28:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:30 smithi204 ceph-mon[30933]: 1.13 scrub starts 2024-09-16T05:28:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:30 smithi204 ceph-mon[30933]: 1.13 scrub ok 2024-09-16T05:28:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: 1.0 scrub starts 2024-09-16T05:28:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: 1.0 scrub ok 2024-09-16T05:28:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: pgmap v1803: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:28:31.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:28:31.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:31 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3800550806' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:28:31.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: 1.0 scrub starts 2024-09-16T05:28:31.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: 1.0 scrub ok 2024-09-16T05:28:31.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: pgmap v1803: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:31.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:28:31.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:28:31.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:31 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3800550806' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:28:31.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: 1.0 scrub starts 2024-09-16T05:28:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: 1.0 scrub ok 2024-09-16T05:28:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: pgmap v1803: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:28:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:28:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:31.738 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:31 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3800550806' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-09-16T05:28:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-16T05:28:32.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-16T05:28:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:32 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3800550806' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:28:33.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:32 smithi196 ceph-mon[30802]: osdmap e1062: 12 total, 12 up, 12 in 2024-09-16T05:28:33.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:32 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3800550806' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:28:33.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:32 smithi032 ceph-mon[27388]: osdmap e1062: 12 total, 12 up, 12 in 2024-09-16T05:28:33.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:32 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3800550806' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-16T05:28:33.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:32 smithi204 ceph-mon[30933]: osdmap e1062: 12 total, 12 up, 12 in 2024-09-16T05:28:33.922 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T05:28:34.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:33 smithi032 ceph-mon[27388]: pgmap v1805: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:34.230 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:33 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3184265862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:28:34.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:33 smithi196 ceph-mon[30802]: pgmap v1805: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:34.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:33 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3184265862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:28:34.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:33 smithi204 ceph-mon[30933]: pgmap v1805: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:34.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:33 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3184265862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:28:34.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 10 1 2024-09-16T05:28:34.918 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:35.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:34 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3184265862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:28:35.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:34 smithi032 ceph-mon[27388]: osdmap e1063: 12 total, 12 up, 12 in 2024-09-16T05:28:35.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:34 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3184265862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:28:35.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:34 smithi196 ceph-mon[30802]: osdmap e1063: 12 total, 12 up, 12 in 2024-09-16T05:28:35.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:34 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3184265862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:28:35.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:34 smithi204 ceph-mon[30933]: osdmap e1063: 12 total, 12 up, 12 in 2024-09-16T05:28:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-16T05:28:36.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:35 smithi196 ceph-mon[30802]: 1.11 scrub starts 2024-09-16T05:28:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:35 smithi196 ceph-mon[30802]: 1.11 scrub ok 2024-09-16T05:28:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:35 smithi196 ceph-mon[30802]: pgmap v1807: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:35 smithi196 ceph-mon[30802]: 1.4 scrub starts 2024-09-16T05:28:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:35 smithi196 ceph-mon[30802]: 1.4 scrub ok 2024-09-16T05:28:36.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:35 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3385760443' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:28:36.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:35 smithi032 ceph-mon[27388]: 1.11 scrub starts 2024-09-16T05:28:36.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:35 smithi032 ceph-mon[27388]: 1.11 scrub ok 2024-09-16T05:28:36.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:35 smithi032 ceph-mon[27388]: pgmap v1807: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:36.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:35 smithi032 ceph-mon[27388]: 1.4 scrub starts 2024-09-16T05:28:36.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:35 smithi032 ceph-mon[27388]: 1.4 scrub ok 2024-09-16T05:28:36.325 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:35 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3385760443' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:28:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:35 smithi204 ceph-mon[30933]: 1.11 scrub starts 2024-09-16T05:28:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:35 smithi204 ceph-mon[30933]: 1.11 scrub ok 2024-09-16T05:28:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:35 smithi204 ceph-mon[30933]: pgmap v1807: 31 pgs: 31 active+clean; 577 KiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:35 smithi204 ceph-mon[30933]: 1.4 scrub starts 2024-09-16T05:28:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:35 smithi204 ceph-mon[30933]: 1.4 scrub ok 2024-09-16T05:28:36.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:35 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3385760443' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-09-16T05:28:36.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-16T05:28:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:37.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:36 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3385760443' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:28:37.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:36 smithi032 ceph-mon[27388]: osdmap e1064: 12 total, 12 up, 12 in 2024-09-16T05:28:37.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:36 smithi032 ceph-mon[27388]: 1.1d scrub starts 2024-09-16T05:28:37.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:36 smithi032 ceph-mon[27388]: 1.1d scrub ok 2024-09-16T05:28:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:36 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3385760443' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:28:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:36 smithi196 ceph-mon[30802]: osdmap e1064: 12 total, 12 up, 12 in 2024-09-16T05:28:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:36 smithi196 ceph-mon[30802]: 1.1d scrub starts 2024-09-16T05:28:37.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:36 smithi196 ceph-mon[30802]: 1.1d scrub ok 2024-09-16T05:28:37.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:36 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3385760443' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-16T05:28:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:36 smithi204 ceph-mon[30933]: osdmap e1064: 12 total, 12 up, 12 in 2024-09-16T05:28:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:36 smithi204 ceph-mon[30933]: 1.1d scrub starts 2024-09-16T05:28:37.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:36 smithi204 ceph-mon[30933]: 1.1d scrub ok 2024-09-16T05:28:37.944 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T05:28:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:37 smithi196 ceph-mon[30802]: pgmap v1809: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:38.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3721971302' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:28:38.280 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:37 smithi032 ceph-mon[27388]: pgmap v1809: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:38.280 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3721971302' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:28:38.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:37 smithi204 ceph-mon[30933]: pgmap v1809: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:38.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3721971302' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:28:38.628 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 11 1 2024-09-16T05:28:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:39.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:38 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3721971302' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:28:39.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:38 smithi196 ceph-mon[30802]: osdmap e1065: 12 total, 12 up, 12 in 2024-09-16T05:28:39.284 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:38 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3721971302' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:28:39.285 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:38 smithi032 ceph-mon[27388]: osdmap e1065: 12 total, 12 up, 12 in 2024-09-16T05:28:39.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:38 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3721971302' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:28:39.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:38 smithi204 ceph-mon[30933]: osdmap e1065: 12 total, 12 up, 12 in 2024-09-16T05:28:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-16T05:28:40.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:39 smithi196 ceph-mon[30802]: pgmap v1811: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:40.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:39 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3411604548' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:28:40.305 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:39 smithi032 ceph-mon[27388]: pgmap v1811: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:40.305 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:39 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3411604548' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:28:40.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:39 smithi204 ceph-mon[30933]: pgmap v1811: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:40.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:39 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3411604548' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-09-16T05:28:40.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-16T05:28:41.017 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:41.138 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:40 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3411604548' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:28:41.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:40 smithi032 ceph-mon[27388]: osdmap e1066: 12 total, 12 up, 12 in 2024-09-16T05:28:41.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:40 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3411604548' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:28:41.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:40 smithi196 ceph-mon[30802]: osdmap e1066: 12 total, 12 up, 12 in 2024-09-16T05:28:41.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:40 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3411604548' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-16T05:28:41.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:40 smithi204 ceph-mon[30933]: osdmap e1066: 12 total, 12 up, 12 in 2024-09-16T05:28:41.977 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T05:28:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:41 smithi196 ceph-mon[30802]: pgmap v1813: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:42.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:41 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2025104235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:28:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:41 smithi032 ceph-mon[27388]: pgmap v1813: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:42.293 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:41 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2025104235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:28:42.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:41 smithi204 ceph-mon[30933]: pgmap v1813: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:42.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:41 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2025104235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:28:42.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-16T05:28:43.011 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:42 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2025104235' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:28:43.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:42 smithi196 ceph-mon[30802]: osdmap e1067: 12 total, 12 up, 12 in 2024-09-16T05:28:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:42 smithi196 ceph-mon[30802]: 1.9 scrub starts 2024-09-16T05:28:43.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:42 smithi196 ceph-mon[30802]: 1.9 scrub ok 2024-09-16T05:28:43.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:42 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2025104235' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:28:43.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:42 smithi032 ceph-mon[27388]: osdmap e1067: 12 total, 12 up, 12 in 2024-09-16T05:28:43.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:42 smithi032 ceph-mon[27388]: 1.9 scrub starts 2024-09-16T05:28:43.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:42 smithi032 ceph-mon[27388]: 1.9 scrub ok 2024-09-16T05:28:43.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:42 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2025104235' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:28:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:42 smithi204 ceph-mon[30933]: osdmap e1067: 12 total, 12 up, 12 in 2024-09-16T05:28:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:42 smithi204 ceph-mon[30933]: 1.9 scrub starts 2024-09-16T05:28:43.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:42 smithi204 ceph-mon[30933]: 1.9 scrub ok 2024-09-16T05:28:43.987 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T05:28:44.317 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:43 smithi032 ceph-mon[27388]: pgmap v1815: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:44.317 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:43 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1579473060' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:28:44.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:43 smithi204 ceph-mon[30933]: pgmap v1815: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:44.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:43 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1579473060' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:28:44.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:43 smithi196 ceph-mon[30802]: pgmap v1815: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:44.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:43 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1579473060' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-16T05:28:44.729 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-16T05:28:44.992 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:45.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:44 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1579473060' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:28:45.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:44 smithi032 ceph-mon[27388]: osdmap e1068: 12 total, 12 up, 12 in 2024-09-16T05:28:45.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:44 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1579473060' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:28:45.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:44 smithi204 ceph-mon[30933]: osdmap e1068: 12 total, 12 up, 12 in 2024-09-16T05:28:45.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:44 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1579473060' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-16T05:28:45.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:44 smithi196 ceph-mon[30802]: osdmap e1068: 12 total, 12 up, 12 in 2024-09-16T05:28:45.998 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T05:28:46.384 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:46 smithi032 ceph-mon[27388]: pgmap v1817: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:46.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:46 smithi032 ceph-mon[27388]: 1.17 scrub starts 2024-09-16T05:28:46.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:46 smithi032 ceph-mon[27388]: 1.17 scrub ok 2024-09-16T05:28:46.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:46 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3462452025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:28:46.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:46 smithi204 ceph-mon[30933]: pgmap v1817: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:46 smithi204 ceph-mon[30933]: 1.17 scrub starts 2024-09-16T05:28:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:46 smithi204 ceph-mon[30933]: 1.17 scrub ok 2024-09-16T05:28:46.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:46 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3462452025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:28:46.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:46 smithi196 ceph-mon[30802]: pgmap v1817: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:46 smithi196 ceph-mon[30802]: 1.17 scrub starts 2024-09-16T05:28:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:46 smithi196 ceph-mon[30802]: 1.17 scrub ok 2024-09-16T05:28:46.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:46 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3462452025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:28:46.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-16T05:28:47.007 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:47 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3462452025' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:28:47.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:47 smithi032 ceph-mon[27388]: osdmap e1069: 12 total, 12 up, 12 in 2024-09-16T05:28:47.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:47 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:47.008 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:47 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:47.031 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:47.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:47 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3462452025' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:28:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:47 smithi204 ceph-mon[30933]: osdmap e1069: 12 total, 12 up, 12 in 2024-09-16T05:28:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:47 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:47.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:47 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:47.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:47 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3462452025' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:28:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:47 smithi196 ceph-mon[30802]: osdmap e1069: 12 total, 12 up, 12 in 2024-09-16T05:28:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:47 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:47.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:47 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T05:28:48.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:48 smithi032 ceph-mon[27388]: pgmap v1819: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:48.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:48 smithi032 ceph-mon[27388]: osdmap e1070: 12 total, 12 up, 12 in 2024-09-16T05:28:48.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:48 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1026587505' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:28:48.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:48 smithi204 ceph-mon[30933]: pgmap v1819: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:48.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:48 smithi204 ceph-mon[30933]: osdmap e1070: 12 total, 12 up, 12 in 2024-09-16T05:28:48.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:48 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1026587505' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:28:48.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:48 smithi196 ceph-mon[30802]: pgmap v1819: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:48 smithi196 ceph-mon[30802]: osdmap e1070: 12 total, 12 up, 12 in 2024-09-16T05:28:48.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:48 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1026587505' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-16T05:28:48.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-16T05:28:48.948 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:49.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:49 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1026587505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:28:49.356 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:49 smithi032 ceph-mon[27388]: osdmap e1071: 12 total, 12 up, 12 in 2024-09-16T05:28:49.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:49 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1026587505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:28:49.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:49 smithi204 ceph-mon[30933]: osdmap e1071: 12 total, 12 up, 12 in 2024-09-16T05:28:49.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:49 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1026587505' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-16T05:28:49.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:49 smithi196 ceph-mon[30802]: osdmap e1071: 12 total, 12 up, 12 in 2024-09-16T05:28:50.024 INFO:teuthology.orchestra.run.smithi032.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T05:28:50.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:50 smithi032 ceph-mon[27388]: pgmap v1822: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:50.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:50 smithi032 ceph-mon[27388]: 1.c scrub starts 2024-09-16T05:28:50.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:50 smithi032 ceph-mon[27388]: 1.c scrub ok 2024-09-16T05:28:50.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:50 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/355482085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:28:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:50 smithi204 ceph-mon[30933]: pgmap v1822: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:50 smithi204 ceph-mon[30933]: 1.c scrub starts 2024-09-16T05:28:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:50 smithi204 ceph-mon[30933]: 1.c scrub ok 2024-09-16T05:28:50.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:50 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/355482085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:28:50.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:50 smithi196 ceph-mon[30802]: pgmap v1822: 31 pgs: 31 active+clean; 577 KiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:50.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:50 smithi196 ceph-mon[30802]: 1.c scrub starts 2024-09-16T05:28:50.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:50 smithi196 ceph-mon[30802]: 1.c scrub ok 2024-09-16T05:28:50.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:50 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/355482085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:28:50.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-16T05:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:51 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:51 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:51 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/355482085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:28:51.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:51 smithi032 ceph-mon[27388]: osdmap e1072: 12 total, 12 up, 12 in 2024-09-16T05:28:51.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:51 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:51 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:51 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/355482085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:28:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:51 smithi204 ceph-mon[30933]: osdmap e1072: 12 total, 12 up, 12 in 2024-09-16T05:28:51.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:51 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:51.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:51 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:51.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:51 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/355482085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:28:51.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:51 smithi196 ceph-mon[30802]: osdmap e1072: 12 total, 12 up, 12 in 2024-09-16T05:28:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T05:28:52.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:52 smithi032 ceph-mon[27388]: pgmap v1824: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:52.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:52 smithi032 ceph-mon[27388]: 1.e scrub starts 2024-09-16T05:28:52.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:52 smithi032 ceph-mon[27388]: 1.e scrub ok 2024-09-16T05:28:52.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:52 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3019662605' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:28:52.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:52 smithi204 ceph-mon[30933]: pgmap v1824: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:52 smithi204 ceph-mon[30933]: 1.e scrub starts 2024-09-16T05:28:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:52 smithi204 ceph-mon[30933]: 1.e scrub ok 2024-09-16T05:28:52.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:52 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3019662605' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:28:52.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:52 smithi196 ceph-mon[30802]: pgmap v1824: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:52.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:52 smithi196 ceph-mon[30802]: 1.e scrub starts 2024-09-16T05:28:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:52 smithi196 ceph-mon[30802]: 1.e scrub ok 2024-09-16T05:28:52.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:52 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3019662605' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-16T05:28:52.749 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T05:28:52.749 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T05:28:52.749 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T05:28:52.749 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T05:28:52.750 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-16T05:28:52.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T05:28:53.003 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:53.307 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:53.307 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:53.307 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: 1.12 scrub starts 2024-09-16T05:28:53.307 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: 1.12 scrub ok 2024-09-16T05:28:53.307 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/3019662605' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: osdmap e1073: 12 total, 12 up, 12 in 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:28:53.308 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:53 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: 1.12 scrub starts 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: 1.12 scrub ok 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/3019662605' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: osdmap e1073: 12 total, 12 up, 12 in 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:53.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:28:53.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:28:53.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:28:53.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:28:53.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:28:53.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:28:53.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:53 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: 1.12 scrub starts 2024-09-16T05:28:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: 1.12 scrub ok 2024-09-16T05:28:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/3019662605' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-16T05:28:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: osdmap e1073: 12 total, 12 up, 12 in 2024-09-16T05:28:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:53.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:53.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T05:28:53.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T05:28:53.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T05:28:53.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:28:53.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T05:28:53.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T05:28:53.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:53 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' 2024-09-16T05:28:53.712 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:28:53.712 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1074,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","created":"2024-09-16T05:02:35.583426+0000","modified":"2024-09-16T05:28:53.045863+0000","last_up_change":"2024-09-16T05:26:57.965143+0000","last_in_change":"2024-09-16T05:21:53.914532+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":43,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:05:16.445728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"687","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"685","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f3cfb49b-8223-4649-b211-a2e254248d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6803","nonce":406159384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6805","nonce":406159384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6809","nonce":406159384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6807","nonce":406159384}]},"public_addr":"172.21.15.32:6803/406159384","cluster_addr":"172.21.15.32:6805/406159384","heartbeat_back_addr":"172.21.15.32:6809/406159384","heartbeat_front_addr":"172.21.15.32:6807/406159384","state":["exists","up"]},{"osd":1,"uuid":"cfb43840-55b6-4d39-a325-2c5416ed97b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":866,"up_from":988,"up_thru":1007,"down_at":867,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6811","nonce":2976159074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6813","nonce":2976159074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6817","nonce":2976159074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6815","nonce":2976159074}]},"public_addr":"172.21.15.32:6811/2976159074","cluster_addr":"172.21.15.32:6813/2976159074","heartbeat_back_addr":"172.21.15.32:6817/2976159074","heartbeat_front_addr":"172.21.15.32:6815/2976159074","state":["exists","up"]},{"osd":2,"uuid":"3158b0cc-849c-4383-8d4d-585cf0723e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":295,"up_from":329,"up_thru":1054,"down_at":296,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6819","nonce":3973436581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6821","nonce":3973436581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6825","nonce":3973436581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6823","nonce":3973436581}]},"public_addr":"172.21.15.32:6819/3973436581","cluster_addr":"172.21.15.32:6821/3973436581","heartbeat_back_addr":"172.21.15.32:6825/3973436581","heartbeat_front_addr":"172.21.15.32:6823/3973436581","state":["exists","up"]},{"osd":3,"uuid":"71705a06-9102-4b97-be6a-4989306666d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":1054,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6827","nonce":3040098358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6829","nonce":3040098358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6833","nonce":3040098358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6831","nonce":3040098358}]},"public_addr":"172.21.15.32:6827/3040098358","cluster_addr":"172.21.15.32:6829/3040098358","heartbeat_back_addr":"172.21.15.32:6833/3040098358","heartbeat_front_addr":"172.21.15.32:6831/3040098358","state":["exists","up"]},{"osd":4,"uuid":"ca8f47c4-c13f-4bde-b162-0e3b57bad1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":1050,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6801","nonce":3219469655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6803","nonce":3219469655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6807","nonce":3219469655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6805","nonce":3219469655}]},"public_addr":"172.21.15.196:6801/3219469655","cluster_addr":"172.21.15.196:6803/3219469655","heartbeat_back_addr":"172.21.15.196:6807/3219469655","heartbeat_front_addr":"172.21.15.196:6805/3219469655","state":["exists","up"]},{"osd":5,"uuid":"b5b76271-0539-42dd-a9e7-a95afded2d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1055,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6809","nonce":3274041128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6811","nonce":3274041128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6815","nonce":3274041128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6813","nonce":3274041128}]},"public_addr":"172.21.15.196:6809/3274041128","cluster_addr":"172.21.15.196:6811/3274041128","heartbeat_back_addr":"172.21.15.196:6815/3274041128","heartbeat_front_addr":"172.21.15.196:6813/3274041128","state":["exists","up"]},{"osd":6,"uuid":"d3f2312b-cc04-48e4-993a-f08a948ba3ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":1057,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6817","nonce":2509099861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6819","nonce":2509099861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6823","nonce":2509099861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6821","nonce":2509099861}]},"public_addr":"172.21.15.196:6817/2509099861","cluster_addr":"172.21.15.196:6819/2509099861","heartbeat_back_addr":"172.21.15.196:6823/2509099861","heartbeat_front_addr":"172.21.15.196:6821/2509099861","state":["exists","up"]},{"osd":7,"uuid":"fc7515ee-0a55-4b10-825e-8a4efa96d0e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":58,"last_clean_end":964,"up_from":1007,"up_thru":1028,"down_at":965,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6825","nonce":1346226909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6827","nonce":1346226909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6831","nonce":1346226909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6829","nonce":1346226909}]},"public_addr":"172.21.15.196:6825/1346226909","cluster_addr":"172.21.15.196:6827/1346226909","heartbeat_back_addr":"172.21.15.196:6831/1346226909","heartbeat_front_addr":"172.21.15.196:6829/1346226909","state":["exists","up"]},{"osd":8,"uuid":"e116a9c9-3c07-4b81-8b23-f2ba84916090","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":1052,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6801","nonce":3463685812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6803","nonce":3463685812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6807","nonce":3463685812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6805","nonce":3463685812}]},"public_addr":"172.21.15.204:6801/3463685812","cluster_addr":"172.21.15.204:6803/3463685812","heartbeat_back_addr":"172.21.15.204:6807/3463685812","heartbeat_front_addr":"172.21.15.204:6805/3463685812","state":["exists","up"]},{"osd":9,"uuid":"06573a93-d7be-426b-97c4-4e088b2bfa1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":1028,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6809","nonce":1139381345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6811","nonce":1139381345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6815","nonce":1139381345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6813","nonce":1139381345}]},"public_addr":"172.21.15.204:6809/1139381345","cluster_addr":"172.21.15.204:6811/1139381345","heartbeat_back_addr":"172.21.15.204:6815/1139381345","heartbeat_front_addr":"172.21.15.204:6813/1139381345","state":["exists","up"]},{"osd":10,"uuid":"388cc928-3439-4f37-a2dc-1f2a494f9846","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":1029,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6817","nonce":726811436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6819","nonce":726811436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6823","nonce":726811436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6821","nonce":726811436}]},"public_addr":"172.21.15.204:6817/726811436","cluster_addr":"172.21.15.204:6819/726811436","heartbeat_back_addr":"172.21.15.204:6823/726811436","heartbeat_front_addr":"172.21.15.204:6821/726811436","state":["exists","up"]},{"osd":11,"uuid":"49911d7e-fe03-41a6-a25a-11a11c08f8b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":1029,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6825","nonce":1668426191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6827","nonce":1668426191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6831","nonce":1668426191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6829","nonce":1668426191}]},"public_addr":"172.21.15.204:6825/1668426191","cluster_addr":"172.21.15.204:6827/1668426191","heartbeat_back_addr":"172.21.15.204:6831/1668426191","heartbeat_front_addr":"172.21.15.204:6829/1668426191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:16.585524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:46.530846+0000","dead_epoch":866},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:25.113515+0000","dead_epoch":295},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:39.536504+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:54.462189+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:52.262516+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:05.620119+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:01.627875+0000","dead_epoch":964},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:27.453362+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:51.824274+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:26.783838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:49.448299+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2865075910":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6801/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/194694811":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/1290699598":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/2769196199":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/4153012034":"2024-09-17T05:03:00.800739+0000","172.21.15.32:6801/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/1722832924":"2024-09-17T05:03:00.800739+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:28:54.305 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:54 smithi032 ceph-mon[27388]: osdmap e1074: 12 total, 12 up, 12 in 2024-09-16T05:28:54.305 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:54 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/775903952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:28:54.335 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-16T05:28:54.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T05:28:54.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:54 smithi204 ceph-mon[30933]: osdmap e1074: 12 total, 12 up, 12 in 2024-09-16T05:28:54.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:54 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/775903952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:28:54.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:54 smithi196 ceph-mon[30802]: osdmap e1074: 12 total, 12 up, 12 in 2024-09-16T05:28:54.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:54 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/775903952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:28:54.592 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:55.278 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:28:55.278 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1075,"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","created":"2024-09-16T05:02:35.583426+0000","modified":"2024-09-16T05:28:55.053322+0000","last_up_change":"2024-09-16T05:26:57.965143+0000","last_in_change":"2024-09-16T05:21:53.914532+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":43,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:05:16.445728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"687","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"685","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f3cfb49b-8223-4649-b211-a2e254248d88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6803","nonce":406159384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6805","nonce":406159384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6809","nonce":406159384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":406159384},{"type":"v1","addr":"172.21.15.32:6807","nonce":406159384}]},"public_addr":"172.21.15.32:6803/406159384","cluster_addr":"172.21.15.32:6805/406159384","heartbeat_back_addr":"172.21.15.32:6809/406159384","heartbeat_front_addr":"172.21.15.32:6807/406159384","state":["exists","up"]},{"osd":1,"uuid":"cfb43840-55b6-4d39-a325-2c5416ed97b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":866,"up_from":988,"up_thru":1007,"down_at":867,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6811","nonce":2976159074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6813","nonce":2976159074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6817","nonce":2976159074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":2976159074},{"type":"v1","addr":"172.21.15.32:6815","nonce":2976159074}]},"public_addr":"172.21.15.32:6811/2976159074","cluster_addr":"172.21.15.32:6813/2976159074","heartbeat_back_addr":"172.21.15.32:6817/2976159074","heartbeat_front_addr":"172.21.15.32:6815/2976159074","state":["exists","up"]},{"osd":2,"uuid":"3158b0cc-849c-4383-8d4d-585cf0723e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":295,"up_from":329,"up_thru":1054,"down_at":296,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6819","nonce":3973436581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6821","nonce":3973436581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6825","nonce":3973436581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3973436581},{"type":"v1","addr":"172.21.15.32:6823","nonce":3973436581}]},"public_addr":"172.21.15.32:6819/3973436581","cluster_addr":"172.21.15.32:6821/3973436581","heartbeat_back_addr":"172.21.15.32:6825/3973436581","heartbeat_front_addr":"172.21.15.32:6823/3973436581","state":["exists","up"]},{"osd":3,"uuid":"71705a06-9102-4b97-be6a-4989306666d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":1054,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6827","nonce":3040098358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6829","nonce":3040098358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6833","nonce":3040098358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":3040098358},{"type":"v1","addr":"172.21.15.32:6831","nonce":3040098358}]},"public_addr":"172.21.15.32:6827/3040098358","cluster_addr":"172.21.15.32:6829/3040098358","heartbeat_back_addr":"172.21.15.32:6833/3040098358","heartbeat_front_addr":"172.21.15.32:6831/3040098358","state":["exists","up"]},{"osd":4,"uuid":"ca8f47c4-c13f-4bde-b162-0e3b57bad1b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":1050,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6801","nonce":3219469655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6803","nonce":3219469655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6807","nonce":3219469655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3219469655},{"type":"v1","addr":"172.21.15.196:6805","nonce":3219469655}]},"public_addr":"172.21.15.196:6801/3219469655","cluster_addr":"172.21.15.196:6803/3219469655","heartbeat_back_addr":"172.21.15.196:6807/3219469655","heartbeat_front_addr":"172.21.15.196:6805/3219469655","state":["exists","up"]},{"osd":5,"uuid":"b5b76271-0539-42dd-a9e7-a95afded2d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1055,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6809","nonce":3274041128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6811","nonce":3274041128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6815","nonce":3274041128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3274041128},{"type":"v1","addr":"172.21.15.196:6813","nonce":3274041128}]},"public_addr":"172.21.15.196:6809/3274041128","cluster_addr":"172.21.15.196:6811/3274041128","heartbeat_back_addr":"172.21.15.196:6815/3274041128","heartbeat_front_addr":"172.21.15.196:6813/3274041128","state":["exists","up"]},{"osd":6,"uuid":"d3f2312b-cc04-48e4-993a-f08a948ba3ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":1057,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6817","nonce":2509099861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6819","nonce":2509099861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6823","nonce":2509099861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2509099861},{"type":"v1","addr":"172.21.15.196:6821","nonce":2509099861}]},"public_addr":"172.21.15.196:6817/2509099861","cluster_addr":"172.21.15.196:6819/2509099861","heartbeat_back_addr":"172.21.15.196:6823/2509099861","heartbeat_front_addr":"172.21.15.196:6821/2509099861","state":["exists","up"]},{"osd":7,"uuid":"fc7515ee-0a55-4b10-825e-8a4efa96d0e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":58,"last_clean_end":964,"up_from":1007,"up_thru":1028,"down_at":965,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6825","nonce":1346226909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6827","nonce":1346226909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6831","nonce":1346226909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":1346226909},{"type":"v1","addr":"172.21.15.196:6829","nonce":1346226909}]},"public_addr":"172.21.15.196:6825/1346226909","cluster_addr":"172.21.15.196:6827/1346226909","heartbeat_back_addr":"172.21.15.196:6831/1346226909","heartbeat_front_addr":"172.21.15.196:6829/1346226909","state":["exists","up"]},{"osd":8,"uuid":"e116a9c9-3c07-4b81-8b23-f2ba84916090","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":1052,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6801","nonce":3463685812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6803","nonce":3463685812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6807","nonce":3463685812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3463685812},{"type":"v1","addr":"172.21.15.204:6805","nonce":3463685812}]},"public_addr":"172.21.15.204:6801/3463685812","cluster_addr":"172.21.15.204:6803/3463685812","heartbeat_back_addr":"172.21.15.204:6807/3463685812","heartbeat_front_addr":"172.21.15.204:6805/3463685812","state":["exists","up"]},{"osd":9,"uuid":"06573a93-d7be-426b-97c4-4e088b2bfa1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":1028,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6809","nonce":1139381345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6811","nonce":1139381345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6815","nonce":1139381345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1139381345},{"type":"v1","addr":"172.21.15.204:6813","nonce":1139381345}]},"public_addr":"172.21.15.204:6809/1139381345","cluster_addr":"172.21.15.204:6811/1139381345","heartbeat_back_addr":"172.21.15.204:6815/1139381345","heartbeat_front_addr":"172.21.15.204:6813/1139381345","state":["exists","up"]},{"osd":10,"uuid":"388cc928-3439-4f37-a2dc-1f2a494f9846","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":1029,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6817","nonce":726811436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6819","nonce":726811436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6823","nonce":726811436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":726811436},{"type":"v1","addr":"172.21.15.204:6821","nonce":726811436}]},"public_addr":"172.21.15.204:6817/726811436","cluster_addr":"172.21.15.204:6819/726811436","heartbeat_back_addr":"172.21.15.204:6823/726811436","heartbeat_front_addr":"172.21.15.204:6821/726811436","state":["exists","up"]},{"osd":11,"uuid":"49911d7e-fe03-41a6-a25a-11a11c08f8b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":1029,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6825","nonce":1668426191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6827","nonce":1668426191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6831","nonce":1668426191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1668426191},{"type":"v1","addr":"172.21.15.204:6829","nonce":1668426191}]},"public_addr":"172.21.15.204:6825/1668426191","cluster_addr":"172.21.15.204:6827/1668426191","heartbeat_back_addr":"172.21.15.204:6831/1668426191","heartbeat_front_addr":"172.21.15.204:6829/1668426191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:16.585524+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:46.530846+0000","dead_epoch":866},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:25.113515+0000","dead_epoch":295},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:54.494066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:54.462189+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:52.262516+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:05.620119+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:01.627875+0000","dead_epoch":964},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:27:27.453362+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:51.824274+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:26.783838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:28:49.448299+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/2865075910":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6801/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/1180926382":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/194694811":"2024-09-17T05:03:25.948027+0000","172.21.15.32:0/1290699598":"2024-09-17T05:03:25.948027+0000","172.21.15.32:6800/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/2769196199":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/4153012034":"2024-09-17T05:03:00.800739+0000","172.21.15.32:6801/178109651":"2024-09-17T05:03:00.800739+0000","172.21.15.32:0/1722832924":"2024-09-17T05:03:00.800739+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:28:55.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-16T05:28:55.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-16T05:28:55.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-16T05:28:55.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-16T05:28:55.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-16T05:28:55.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-16T05:28:55.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-16T05:28:55.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-16T05:28:55.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-16T05:28:55.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-16T05:28:55.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-16T05:28:55.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-16T05:28:56.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: pgmap v1828: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:56.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:56.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:56.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: 1.1c scrub starts 2024-09-16T05:28:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: 1.1c scrub ok 2024-09-16T05:28:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: osdmap e1075: 12 total, 12 up, 12 in 2024-09-16T05:28:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/385315012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:28:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:28:56.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:56 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:28:56.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: pgmap v1828: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: 1.1c scrub starts 2024-09-16T05:28:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: 1.1c scrub ok 2024-09-16T05:28:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: osdmap e1075: 12 total, 12 up, 12 in 2024-09-16T05:28:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/385315012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:28:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:28:56.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:56 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: pgmap v1828: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: 1.1c scrub starts 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: 1.1c scrub ok 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: osdmap e1075: 12 total, 12 up, 12 in 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/385315012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:28:56.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:56 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:28:56.643 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:56.643 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:56.652 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:56.652 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:56.653 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:56.653 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:56.659 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:56.659 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:56.707 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:57.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:57 smithi032 ceph-mon[27388]: 1.a scrub starts 2024-09-16T05:28:57.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:57 smithi032 ceph-mon[27388]: 1.a scrub ok 2024-09-16T05:28:57.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:57 smithi032 ceph-mon[27388]: osdmap e1076: 12 total, 12 up, 12 in 2024-09-16T05:28:57.348 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:57.348 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:28:57.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:57 smithi204 ceph-mon[30933]: 1.a scrub starts 2024-09-16T05:28:57.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:57 smithi204 ceph-mon[30933]: 1.a scrub ok 2024-09-16T05:28:57.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:57 smithi204 ceph-mon[30933]: osdmap e1076: 12 total, 12 up, 12 in 2024-09-16T05:28:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:57 smithi196 ceph-mon[30802]: 1.a scrub starts 2024-09-16T05:28:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:57 smithi196 ceph-mon[30802]: 1.a scrub ok 2024-09-16T05:28:57.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:57 smithi196 ceph-mon[30802]: osdmap e1076: 12 total, 12 up, 12 in 2024-09-16T05:28:58.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:58 smithi032 ceph-mon[27388]: pgmap v1831: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:58.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:58 smithi204 ceph-mon[30933]: pgmap v1831: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:58.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:58 smithi196 ceph-mon[30802]: pgmap v1831: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:28:58.570 INFO:teuthology.orchestra.run.smithi032.stdout:141733921042 2024-09-16T05:28:58.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-16T05:28:59.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:59 smithi032 ceph-mon[27388]: 1.5 scrub starts 2024-09-16T05:28:59.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:28:59 smithi032 ceph-mon[27388]: 1.5 scrub ok 2024-09-16T05:28:59.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:59 smithi204 ceph-mon[30933]: 1.5 scrub starts 2024-09-16T05:28:59.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:28:59 smithi204 ceph-mon[30933]: 1.5 scrub ok 2024-09-16T05:28:59.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:59 smithi196 ceph-mon[30802]: 1.5 scrub starts 2024-09-16T05:28:59.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:28:59 smithi196 ceph-mon[30802]: 1.5 scrub ok 2024-09-16T05:29:00.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:00 smithi032 ceph-mon[27388]: pgmap v1832: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:00.307 INFO:teuthology.orchestra.run.smithi032.stdout:210453397770 2024-09-16T05:29:00.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-16T05:29:00.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:00 smithi204 ceph-mon[30933]: pgmap v1832: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:00.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:00 smithi196 ceph-mon[30802]: pgmap v1832: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:02 smithi032 ceph-mon[27388]: pgmap v1833: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:02 smithi032 ceph-mon[27388]: 1.14 scrub starts 2024-09-16T05:29:02.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:02 smithi032 ceph-mon[27388]: 1.14 scrub ok 2024-09-16T05:29:02.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:02 smithi204 ceph-mon[30933]: pgmap v1833: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:02.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:02 smithi204 ceph-mon[30933]: 1.14 scrub starts 2024-09-16T05:29:02.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:02 smithi204 ceph-mon[30933]: 1.14 scrub ok 2024-09-16T05:29:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:02 smithi196 ceph-mon[30802]: pgmap v1833: 31 pgs: 31 active+clean; 577 KiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:02 smithi196 ceph-mon[30802]: 1.14 scrub starts 2024-09-16T05:29:02.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:02 smithi196 ceph-mon[30802]: 1.14 scrub ok 2024-09-16T05:29:03.888 INFO:teuthology.orchestra.run.smithi032.stdout:283467841791 2024-09-16T05:29:03.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-16T05:29:04.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-09-16T05:29:03.516+0000 7fe553fff640 0 --2- 172.21.15.32:0/2608188587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe554101df0 0x7fe554197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T05:29:04.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:04 smithi032 ceph-mon[27388]: pgmap v1834: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:04.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:04 smithi032 ceph-mon[27388]: 1.d scrub starts 2024-09-16T05:29:04.155 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:04 smithi032 ceph-mon[27388]: 1.d scrub ok 2024-09-16T05:29:04.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:04 smithi204 ceph-mon[30933]: pgmap v1834: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:04.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:04 smithi204 ceph-mon[30933]: 1.d scrub starts 2024-09-16T05:29:04.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:04 smithi204 ceph-mon[30933]: 1.d scrub ok 2024-09-16T05:29:04.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:04 smithi196 ceph-mon[30802]: pgmap v1834: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:04 smithi196 ceph-mon[30802]: 1.d scrub starts 2024-09-16T05:29:04.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:04 smithi196 ceph-mon[30802]: 1.d scrub ok 2024-09-16T05:29:04.761 INFO:teuthology.orchestra.run.smithi032.stdout:382252089584 2024-09-16T05:29:04.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-16T05:29:05.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:05 smithi032 ceph-mon[27388]: 1.3 scrub starts 2024-09-16T05:29:05.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:05 smithi032 ceph-mon[27388]: 1.3 scrub ok 2024-09-16T05:29:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:05 smithi204 ceph-mon[30933]: 1.3 scrub starts 2024-09-16T05:29:05.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:05 smithi204 ceph-mon[30933]: 1.3 scrub ok 2024-09-16T05:29:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:05 smithi196 ceph-mon[30802]: 1.3 scrub starts 2024-09-16T05:29:05.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:05 smithi196 ceph-mon[30802]: 1.3 scrub ok 2024-09-16T05:29:05.554 INFO:teuthology.orchestra.run.smithi032.stdout:326417514751 2024-09-16T05:29:05.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-16T05:29:06.046 INFO:teuthology.orchestra.run.smithi032.stdout:1413044240558 2024-09-16T05:29:06.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-16T05:29:06.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:06 smithi032 ceph-mon[27388]: pgmap v1835: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:06.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:06 smithi204 ceph-mon[30933]: pgmap v1835: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:06.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:06 smithi196 ceph-mon[30802]: pgmap v1835: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:06.817 INFO:teuthology.orchestra.run.smithi032.stdout:163208757518 2024-09-16T05:29:06.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-16T05:29:07.151 INFO:teuthology.orchestra.run.smithi032.stdout:34359738663 2024-09-16T05:29:07.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-16T05:29:07.729 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:07.740 INFO:teuthology.orchestra.run.smithi032.stdout:4325032067098 2024-09-16T05:29:07.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-16T05:29:08.326 INFO:teuthology.orchestra.run.smithi032.stdout:4243427688482 2024-09-16T05:29:08.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-16T05:29:08.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:08 smithi032 ceph-mon[27388]: pgmap v1836: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:08.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:08 smithi204 ceph-mon[30933]: pgmap v1836: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:08.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:08 smithi196 ceph-mon[30802]: pgmap v1836: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:08.791 INFO:teuthology.orchestra.run.smithi032.stdout:115964117272 2024-09-16T05:29:08.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-16T05:29:08.807 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:08.808 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:08.810 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:08.891 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:08.892 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:09.816 INFO:teuthology.orchestra.run.smithi032.stdout:210453397772 2024-09-16T05:29:09.868 INFO:teuthology.orchestra.run.smithi032.stdout:163208757519 2024-09-16T05:29:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:10 smithi032 ceph-mon[27388]: pgmap v1837: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/4282566202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T05:29:10.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:10 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2815394034' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T05:29:10.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:10 smithi204 ceph-mon[30933]: pgmap v1837: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:10.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/4282566202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T05:29:10.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:10 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2815394034' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T05:29:10.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:10 smithi196 ceph-mon[30802]: pgmap v1837: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:10.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/4282566202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T05:29:10.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:10 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2815394034' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T05:29:10.644 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:10.644 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:10.644 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:10.655 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757518 got 163208757519 for osd.5 2024-09-16T05:29:10.655 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:11.790 INFO:teuthology.orchestra.run.smithi032.stdout:4325032067100 2024-09-16T05:29:12.131 INFO:teuthology.orchestra.run.smithi032.stdout:4243427688484 2024-09-16T05:29:12.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:12 smithi032 ceph-mon[27388]: pgmap v1838: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:12.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:12 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:29:12.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:12 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:29:12.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:12 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/816073187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T05:29:12.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:12 smithi204 ceph-mon[30933]: pgmap v1838: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:12 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:29:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:12 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:29:12.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:12 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/816073187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T05:29:12.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:12 smithi196 ceph-mon[30802]: pgmap v1838: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:12 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:29:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:12 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:29:12.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:12 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/816073187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T05:29:12.584 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397770 got 210453397772 for osd.6 2024-09-16T05:29:12.584 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:13.234 INFO:tasks.cephadm.ceph_manager.ceph:need seq 4243427688482 got 4243427688484 for osd.1 2024-09-16T05:29:13.235 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:13.349 INFO:teuthology.orchestra.run.smithi032.stdout:382252089586 2024-09-16T05:29:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:13 smithi032 ceph-mon[27388]: 1.f scrub starts 2024-09-16T05:29:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:13 smithi032 ceph-mon[27388]: 1.f scrub ok 2024-09-16T05:29:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:13 smithi032 ceph-mon[27388]: osdmap e1077: 12 total, 12 up, 12 in 2024-09-16T05:29:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1460851747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T05:29:13.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:13 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1430743476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T05:29:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:13 smithi204 ceph-mon[30933]: 1.f scrub starts 2024-09-16T05:29:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:13 smithi204 ceph-mon[30933]: 1.f scrub ok 2024-09-16T05:29:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:13 smithi204 ceph-mon[30933]: osdmap e1077: 12 total, 12 up, 12 in 2024-09-16T05:29:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1460851747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T05:29:13.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:13 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1430743476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T05:29:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:13 smithi196 ceph-mon[30802]: 1.f scrub starts 2024-09-16T05:29:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:13 smithi196 ceph-mon[30802]: 1.f scrub ok 2024-09-16T05:29:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:13 smithi196 ceph-mon[30802]: osdmap e1077: 12 total, 12 up, 12 in 2024-09-16T05:29:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1460851747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T05:29:13.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:13 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1430743476' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T05:29:14.035 INFO:tasks.cephadm.ceph_manager.ceph:need seq 382252089584 got 382252089586 for osd.11 2024-09-16T05:29:14.035 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:14.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:14 smithi032 ceph-mon[27388]: pgmap v1840: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:14.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:14 smithi204 ceph-mon[30933]: pgmap v1840: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:14.851 INFO:teuthology.orchestra.run.smithi032.stdout:1413044240560 2024-09-16T05:29:14.852 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:14 smithi196 ceph-mon[30802]: pgmap v1840: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:15.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 4325032067098 got 4325032067100 for osd.7 2024-09-16T05:29:15.139 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:15.417 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:15 smithi032 ceph-mon[27388]: 1.b scrub starts 2024-09-16T05:29:15.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:15 smithi032 ceph-mon[27388]: 1.b scrub ok 2024-09-16T05:29:15.418 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:15 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1296695387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T05:29:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:15 smithi204 ceph-mon[30933]: 1.b scrub starts 2024-09-16T05:29:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:15 smithi204 ceph-mon[30933]: 1.b scrub ok 2024-09-16T05:29:15.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:15 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1296695387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T05:29:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:15 smithi196 ceph-mon[30802]: 1.b scrub starts 2024-09-16T05:29:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:15 smithi196 ceph-mon[30802]: 1.b scrub ok 2024-09-16T05:29:15.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:15 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1296695387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T05:29:16.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:16 smithi032 ceph-mon[27388]: pgmap v1841: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:16.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:16 smithi204 ceph-mon[30933]: pgmap v1841: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:16.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:16 smithi196 ceph-mon[30802]: pgmap v1841: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:16.503 INFO:teuthology.orchestra.run.smithi032.stdout:283467841793 2024-09-16T05:29:17.264 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841791 got 283467841793 for osd.8 2024-09-16T05:29:17.265 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:17.368 INFO:teuthology.orchestra.run.smithi032.stdout:141733921045 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/260165986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2167420267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2871038031,config=2267314355,health=3897446751,kv=2969127351}) 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3244693409}) 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776424690}) 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1544587}) 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488084889}) 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1842471446}) 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2328045314}) 2024-09-16T05:29:17.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974961044}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=489670356}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=738426157}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336274641}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3557272958}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021887702}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798018627}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=107462204}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415052967}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993996716}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284943846}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77766921}) 2024-09-16T05:29:17.422 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826463679}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2730712536}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3788157}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761618442}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489229320}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858503014}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2261920525}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554895470}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2021224254}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1154937047}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878269809}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1008147775}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=907935211}) 2024-09-16T05:29:17.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558652174}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87480739}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878707412}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1732190488}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001873462}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990390592}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1203158456}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2141813344}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2793066868}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691981032}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646288090}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059385152}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451876107}) 2024-09-16T05:29:17.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887700537}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3948999374}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3499047649}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745391173}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882328687}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885075636}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219365453}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426280147}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2886850086}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2581529673}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699924641}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901137814}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069532514}) 2024-09-16T05:29:17.425 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1916641214}) 2024-09-16T05:29:17.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912651509}) 2024-09-16T05:29:17.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4029351523}) 2024-09-16T05:29:17.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856637483}) 2024-09-16T05:29:17.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788771260}) 2024-09-16T05:29:17.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980792198}) 2024-09-16T05:29:17.426 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:17 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751241201}) 2024-09-16T05:29:17.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/260165986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2167420267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2871038031,config=2267314355,health=3897446751,kv=2969127351}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3244693409}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776424690}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1544587}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488084889}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1842471446}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2328045314}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974961044}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=489670356}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=738426157}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336274641}) 2024-09-16T05:29:17.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3557272958}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021887702}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798018627}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=107462204}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415052967}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993996716}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284943846}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77766921}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826463679}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2730712536}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3788157}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761618442}) 2024-09-16T05:29:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489229320}) 2024-09-16T05:29:17.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858503014}) 2024-09-16T05:29:17.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2261920525}) 2024-09-16T05:29:17.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554895470}) 2024-09-16T05:29:17.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2021224254}) 2024-09-16T05:29:17.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1154937047}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878269809}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1008147775}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=907935211}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558652174}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87480739}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878707412}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1732190488}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001873462}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990390592}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1203158456}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2141813344}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2793066868}) 2024-09-16T05:29:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691981032}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646288090}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059385152}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451876107}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887700537}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3948999374}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3499047649}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745391173}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882328687}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885075636}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219365453}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426280147}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2886850086}) 2024-09-16T05:29:17.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2581529673}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699924641}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901137814}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069532514}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1916641214}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912651509}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4029351523}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856637483}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788771260}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980792198}) 2024-09-16T05:29:17.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:17 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751241201}) 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/260165986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2167420267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2871038031,config=2267314355,health=3897446751,kv=2969127351}) 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3244693409}) 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776424690}) 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1544587}) 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488084889}) 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1842471446}) 2024-09-16T05:29:17.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2328045314}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3974961044}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=489670356}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=738426157}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336274641}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3557272958}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021887702}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798018627}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=107462204}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=415052967}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993996716}) 2024-09-16T05:29:17.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284943846}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77766921}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826463679}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2730712536}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3788157}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761618442}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489229320}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=858503014}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2261920525}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554895470}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2021224254}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1154937047}) 2024-09-16T05:29:17.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878269809}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1008147775}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=907935211}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558652174}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87480739}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3878707412}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1732190488}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001873462}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990390592}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1203158456}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2141813344}) 2024-09-16T05:29:17.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2793066868}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691981032}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646288090}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059385152}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451876107}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887700537}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3948999374}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3499047649}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745391173}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882328687}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885075636}) 2024-09-16T05:29:17.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219365453}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426280147}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2886850086}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2581529673}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699924641}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=901137814}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1069532514}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1916641214}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912651509}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4029351523}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856637483}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788771260}) 2024-09-16T05:29:17.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980792198}) 2024-09-16T05:29:17.495 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:17 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1751241201}) 2024-09-16T05:29:18.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: pgmap v1842: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:18.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: 1.7 scrub starts 2024-09-16T05:29:18.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873158983}) 2024-09-16T05:29:18.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: 1.7 scrub ok 2024-09-16T05:29:18.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755135071}) 2024-09-16T05:29:18.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772958094}) 2024-09-16T05:29:18.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1141290377}) 2024-09-16T05:29:18.327 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199099419}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224039357}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=813324789}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3649768050}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=166439397}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=712523745}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2472265779}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3134322413}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228302528}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2928492626}) 2024-09-16T05:29:18.328 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559066213}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685714876}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454792225}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050428184}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667489013}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926160948}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=100064483}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1865071384}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421198142}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1223148417}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3562454535}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1162209194}) 2024-09-16T05:29:18.329 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2419892520}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=14284694}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272589062}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1487207478}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161504726}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=543429042}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1190652585}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=71} crc {logm=1720030585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=997238633}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3471162631}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=87,mon_config_key=13} crc {mgrstat=1449186469,mon_config_key=2721527092}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3432925488}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=397309196}) 2024-09-16T05:29:18.330 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=19} crc {mon_config_key=2541617054,monmap=2256977890,osd_metadata=319107580,osd_pg_creating=3056396280,osd_snap=1894409994}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=2772317942}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=4000849064,osdmap=2200681364}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3516627285}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3074042723}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1399624434}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2968972546}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1583661362}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=958227611}) 2024-09-16T05:29:18.331 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2796535025}) 2024-09-16T05:29:18.332 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1749565539}) 2024-09-16T05:29:18.332 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=875318985}) 2024-09-16T05:29:18.332 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1594790294}) 2024-09-16T05:29:18.332 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1714943609}) 2024-09-16T05:29:18.332 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3009011558}) 2024-09-16T05:29:18.332 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=418595556}) 2024-09-16T05:29:18.332 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:18 smithi032 ceph-mon[27388]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=67} crc {osdmap=2681006146}) 2024-09-16T05:29:18.362 INFO:teuthology.orchestra.run.smithi032.stdout:356482285815 2024-09-16T05:29:18.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-16T05:29:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: pgmap v1842: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: 1.7 scrub starts 2024-09-16T05:29:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873158983}) 2024-09-16T05:29:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: 1.7 scrub ok 2024-09-16T05:29:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755135071}) 2024-09-16T05:29:18.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772958094}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1141290377}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199099419}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224039357}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=813324789}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3649768050}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=166439397}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=712523745}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2472265779}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3134322413}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228302528}) 2024-09-16T05:29:18.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2928492626}) 2024-09-16T05:29:18.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559066213}) 2024-09-16T05:29:18.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685714876}) 2024-09-16T05:29:18.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454792225}) 2024-09-16T05:29:18.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050428184}) 2024-09-16T05:29:18.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667489013}) 2024-09-16T05:29:18.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926160948}) 2024-09-16T05:29:18.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=100064483}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1865071384}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421198142}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1223148417}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3562454535}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1162209194}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2419892520}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=14284694}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272589062}) 2024-09-16T05:29:18.424 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1487207478}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161504726}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=543429042}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1190652585}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=71} crc {logm=1720030585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=997238633}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3471162631}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=87,mon_config_key=13} crc {mgrstat=1449186469,mon_config_key=2721527092}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3432925488}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=397309196}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=19} crc {mon_config_key=2541617054,monmap=2256977890,osd_metadata=319107580,osd_pg_creating=3056396280,osd_snap=1894409994}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=2772317942}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=4000849064,osdmap=2200681364}) 2024-09-16T05:29:18.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3516627285}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3074042723}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1399624434}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2968972546}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1583661362}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=958227611}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2796535025}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1749565539}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=875318985}) 2024-09-16T05:29:18.426 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1594790294}) 2024-09-16T05:29:18.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1714943609}) 2024-09-16T05:29:18.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3009011558}) 2024-09-16T05:29:18.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=418595556}) 2024-09-16T05:29:18.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:18 smithi204 ceph-mon[30933]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=67} crc {osdmap=2681006146}) 2024-09-16T05:29:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: pgmap v1842: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: 1.7 scrub starts 2024-09-16T05:29:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873158983}) 2024-09-16T05:29:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: 1.7 scrub ok 2024-09-16T05:29:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755135071}) 2024-09-16T05:29:18.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=772958094}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1141290377}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199099419}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224039357}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=813324789}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3649768050}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=166439397}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=712523745}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2472265779}) 2024-09-16T05:29:18.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3134322413}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228302528}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2928492626}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559066213}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3685714876}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454792225}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050428184}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667489013}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926160948}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=100064483}) 2024-09-16T05:29:18.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1865071384}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421198142}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1223148417}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3562454535}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1162209194}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2419892520}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=14284694}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272589062}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1487207478}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161504726}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=543429042}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1190652585}) 2024-09-16T05:29:18.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=71} crc {logm=1720030585,mdsmap=3929276158,mgr=2258745391,mgr_command_descs=1300230762,mgr_metadata=1989862148,mgrstat=997238633}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=100} crc {mgrstat=3471162631}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=87,mon_config_key=13} crc {mgrstat=1449186469,mon_config_key=2721527092}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3432925488}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=397309196}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=63,monmap=5,osd_metadata=12,osd_pg_creating=1,osd_snap=19} crc {mon_config_key=2541617054,monmap=2256977890,osd_metadata=319107580,osd_pg_creating=3056396280,osd_snap=1894409994}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=100} crc {osd_snap=2772317942}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=4000849064,osdmap=2200681364}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3516627285}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3074042723}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1399624434}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2968972546}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1583661362}) 2024-09-16T05:29:18.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=958227611}) 2024-09-16T05:29:18.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2796535025}) 2024-09-16T05:29:18.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1749565539}) 2024-09-16T05:29:18.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=875318985}) 2024-09-16T05:29:18.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1594790294}) 2024-09-16T05:29:18.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1714943609}) 2024-09-16T05:29:18.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3009011558}) 2024-09-16T05:29:18.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=418595556}) 2024-09-16T05:29:18.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:18 smithi196 ceph-mon[30802]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=67} crc {osdmap=2681006146}) 2024-09-16T05:29:18.769 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:18.769 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:18.770 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:18.804 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733921042 got 141733921045 for osd.4 2024-09-16T05:29:18.804 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:19.149 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1413044240558 got 1413044240560 for osd.2 2024-09-16T05:29:19.149 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:20.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:20 smithi032 ceph-mon[27388]: pgmap v1843: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:20.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:20 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:29:20.231 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:20 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:29:20.312 INFO:teuthology.orchestra.run.smithi032.stdout:115964117275 2024-09-16T05:29:20.315 INFO:teuthology.orchestra.run.smithi032.stdout:326417514754 2024-09-16T05:29:20.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:20 smithi204 ceph-mon[30933]: pgmap v1843: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:20.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:20 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:29:20.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:20 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:29:20.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:20 smithi196 ceph-mon[30802]: pgmap v1843: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:20.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:20 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:29:20.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:20 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:29:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:21 smithi032 ceph-mon[27388]: osdmap e1078: 12 total, 12 up, 12 in 2024-09-16T05:29:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/2600060040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T05:29:21.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:21 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1836964657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T05:29:21.340 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117272 got 115964117275 for osd.3 2024-09-16T05:29:21.340 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:21.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:21 smithi204 ceph-mon[30933]: osdmap e1078: 12 total, 12 up, 12 in 2024-09-16T05:29:21.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/2600060040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T05:29:21.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:21 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1836964657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T05:29:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:21 smithi196 ceph-mon[30802]: osdmap e1078: 12 total, 12 up, 12 in 2024-09-16T05:29:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/2600060040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T05:29:21.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:21 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1836964657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T05:29:21.728 INFO:teuthology.orchestra.run.smithi032.stdout:34359738666 2024-09-16T05:29:21.978 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514751 got 326417514754 for osd.9 2024-09-16T05:29:21.978 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:22.286 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:22 smithi032 ceph-mon[27388]: pgmap v1845: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:22.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:22 smithi032 ceph-mon[27388]: 1.15 scrub starts 2024-09-16T05:29:22.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:22 smithi032 ceph-mon[27388]: 1.15 scrub ok 2024-09-16T05:29:22.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:22 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/780504769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T05:29:22.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:22 smithi204 ceph-mon[30933]: pgmap v1845: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:22 smithi204 ceph-mon[30933]: 1.15 scrub starts 2024-09-16T05:29:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:22 smithi204 ceph-mon[30933]: 1.15 scrub ok 2024-09-16T05:29:22.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:22 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/780504769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T05:29:22.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:22 smithi196 ceph-mon[30802]: pgmap v1845: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:22 smithi196 ceph-mon[30802]: 1.15 scrub starts 2024-09-16T05:29:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:22 smithi196 ceph-mon[30802]: 1.15 scrub ok 2024-09-16T05:29:22.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:22 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/780504769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T05:29:22.714 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738663 got 34359738666 for osd.0 2024-09-16T05:29:22.715 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:22.751 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:23.356 INFO:teuthology.orchestra.run.smithi032.stdout:356482285817 2024-09-16T05:29:24.111 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285815 got 356482285817 for osd.10 2024-09-16T05:29:24.111 DEBUG:teuthology.parallel:result is None 2024-09-16T05:29:24.111 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T05:29:24.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:29:24.409 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:24.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:24 smithi032 ceph-mon[27388]: pgmap v1846: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:24.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:24 smithi032 ceph-mon[27388]: 1.1 scrub starts 2024-09-16T05:29:24.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:24 smithi032 ceph-mon[27388]: 1.1 scrub ok 2024-09-16T05:29:24.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:24 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1585038269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T05:29:24.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:24 smithi204 ceph-mon[30933]: pgmap v1846: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:24.420 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:24 smithi204 ceph-mon[30933]: 1.1 scrub starts 2024-09-16T05:29:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:24 smithi204 ceph-mon[30933]: 1.1 scrub ok 2024-09-16T05:29:24.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:24 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1585038269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T05:29:24.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:24 smithi196 ceph-mon[30802]: pgmap v1846: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:24 smithi196 ceph-mon[30802]: 1.1 scrub starts 2024-09-16T05:29:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:24 smithi196 ceph-mon[30802]: 1.1 scrub ok 2024-09-16T05:29:24.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:24 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1585038269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T05:29:25.054 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:29:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:29:25.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10697072452145957]}]: dispatch 2024-09-16T05:29:25.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07839473561616196]}]: dispatch 2024-09-16T05:29:25.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12662493825635246]}]: dispatch 2024-09-16T05:29:25.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09505227482163094]}]: dispatch 2024-09-16T05:29:25.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08967680448438438]}]: dispatch 2024-09-16T05:29:25.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08789547212139301]}]: dispatch 2024-09-16T05:29:25.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08064321961146355]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05953026388194416]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0793936025375686]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07519738875996121]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07073993362101365]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:25 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09742702848541658]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10697072452145957]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07839473561616196]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12662493825635246]}]: dispatch 2024-09-16T05:29:25.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09505227482163094]}]: dispatch 2024-09-16T05:29:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08967680448438438]}]: dispatch 2024-09-16T05:29:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08789547212139301]}]: dispatch 2024-09-16T05:29:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08064321961146355]}]: dispatch 2024-09-16T05:29:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05953026388194416]}]: dispatch 2024-09-16T05:29:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0793936025375686]}]: dispatch 2024-09-16T05:29:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07519738875996121]}]: dispatch 2024-09-16T05:29:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07073993362101365]}]: dispatch 2024-09-16T05:29:25.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:25 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09742702848541658]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10697072452145957]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07839473561616196]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12662493825635246]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09505227482163094]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08967680448438438]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08789547212139301]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08064321961146355]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05953026388194416]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0793936025375686]}]: dispatch 2024-09-16T05:29:25.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07519738875996121]}]: dispatch 2024-09-16T05:29:25.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07073993362101365]}]: dispatch 2024-09-16T05:29:25.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:25 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09742702848541658]}]: dispatch 2024-09-16T05:29:25.906 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1847,"stamp":"2024-09-16T05:29:24.318958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":679684,"kb_used_data":104792,"kb_used_omap":84,"kb_used_meta":574571,"kb_avail":1124262140,"statfs":{"total":1151940427776,"available":1151244431360,"internally_reserved":0,"allocated":107307008,"data_stored":74077068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86930,"internal_metadata":588360814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001913"},"pg_stats":[{"pgid":"1.12","version":"35'139","reported_seq":775,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517443+0000","last_change":"2024-09-16T05:28:51.890924+0000","last_active":"2024-09-16T05:29:20.517443+0000","last_peered":"2024-09-16T05:29:20.517443+0000","last_clean":"2024-09-16T05:29:20.517443+0000","last_became_active":"2024-09-16T05:28:21.191459+0000","last_became_peered":"2024-09-16T05:28:21.191459+0000","last_unstale":"2024-09-16T05:29:20.517443+0000","last_undegraded":"2024-09-16T05:29:20.517443+0000","last_fullsized":"2024-09-16T05:29:20.517443+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:51.890834+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:28:51.890834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:19.285120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"35'139","reported_seq":1508,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-16T05:29:12.409186+0000","last_change":"2024-09-16T05:28:29.769978+0000","last_active":"2024-09-16T05:29:12.409186+0000","last_peered":"2024-09-16T05:29:12.409186+0000","last_clean":"2024-09-16T05:29:12.409186+0000","last_became_active":"2024-09-16T05:26:22.907168+0000","last_became_peered":"2024-09-16T05:26:22.907168+0000","last_unstale":"2024-09-16T05:29:12.409186+0000","last_undegraded":"2024-09-16T05:29:12.409186+0000","last_fullsized":"2024-09-16T05:29:12.409186+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:29.769890+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:29.769890+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:46.455950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1798,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.989156+0000","last_change":"2024-09-16T05:28:48.441113+0000","last_active":"2024-09-16T05:29:20.989156+0000","last_peered":"2024-09-16T05:29:20.989156+0000","last_clean":"2024-09-16T05:29:20.989156+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:29:20.989156+0000","last_undegraded":"2024-09-16T05:29:20.989156+0000","last_fullsized":"2024-09-16T05:29:20.989156+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:48.441014+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:48.441014+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:18.091526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"35'139","reported_seq":1852,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517447+0000","last_change":"2024-09-16T05:29:02.856802+0000","last_active":"2024-09-16T05:29:20.517447+0000","last_peered":"2024-09-16T05:29:20.517447+0000","last_clean":"2024-09-16T05:29:20.517447+0000","last_became_active":"2024-09-16T05:26:58.976632+0000","last_became_peered":"2024-09-16T05:26:58.976632+0000","last_unstale":"2024-09-16T05:29:20.517447+0000","last_undegraded":"2024-09-16T05:29:20.517447+0000","last_fullsized":"2024-09-16T05:29:20.517447+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:02.856695+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_clean_scrub_stamp":"2024-09-16T05:29:02.856695+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:31.504867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"35'139","reported_seq":1848,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.150389+0000","last_change":"2024-09-16T05:28:51.322038+0000","last_active":"2024-09-16T05:29:20.150389+0000","last_peered":"2024-09-16T05:29:20.150389+0000","last_clean":"2024-09-16T05:29:20.150389+0000","last_became_active":"2024-09-16T05:28:16.165227+0000","last_became_peered":"2024-09-16T05:28:16.165227+0000","last_unstale":"2024-09-16T05:29:20.150389+0000","last_undegraded":"2024-09-16T05:29:20.150389+0000","last_fullsized":"2024-09-16T05:29:20.150389+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:51.321905+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:28:51.321905+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:16.800529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"35'139","reported_seq":1322,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517417+0000","last_change":"2024-09-16T05:29:12.087578+0000","last_active":"2024-09-16T05:29:20.517417+0000","last_peered":"2024-09-16T05:29:20.517417+0000","last_clean":"2024-09-16T05:29:20.517417+0000","last_became_active":"2024-09-16T05:28:24.204528+0000","last_became_peered":"2024-09-16T05:28:24.204528+0000","last_unstale":"2024-09-16T05:29:20.517417+0000","last_undegraded":"2024-09-16T05:29:20.517417+0000","last_fullsized":"2024-09-16T05:29:20.517417+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:12.087489+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_clean_scrub_stamp":"2024-09-16T05:29:12.087489+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:38.977304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"35'139","reported_seq":1493,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-16T05:29:14.497531+0000","last_change":"2024-09-16T05:29:14.497553+0000","last_active":"2024-09-16T05:29:14.497531+0000","last_peered":"2024-09-16T05:29:14.497531+0000","last_clean":"2024-09-16T05:29:14.497531+0000","last_became_active":"2024-09-16T05:26:25.117472+0000","last_became_peered":"2024-09-16T05:26:25.117472+0000","last_unstale":"2024-09-16T05:29:14.497531+0000","last_undegraded":"2024-09-16T05:29:14.497531+0000","last_fullsized":"2024-09-16T05:29:14.497531+0000","mapping_epoch":991,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:14.497468+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:29:14.497468+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:44.280746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"35'139","reported_seq":874,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517208+0000","last_change":"2024-09-16T05:28:30.036111+0000","last_active":"2024-09-16T05:29:20.517208+0000","last_peered":"2024-09-16T05:29:20.517208+0000","last_clean":"2024-09-16T05:29:20.517208+0000","last_became_active":"2024-09-16T05:28:24.205007+0000","last_became_peered":"2024-09-16T05:28:24.205007+0000","last_unstale":"2024-09-16T05:29:20.517208+0000","last_undegraded":"2024-09-16T05:29:20.517208+0000","last_fullsized":"2024-09-16T05:29:20.517208+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:30.036037+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_clean_scrub_stamp":"2024-09-16T05:28:30.036037+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:42.097121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1714,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517456+0000","last_change":"2024-09-16T05:27:58.979091+0000","last_active":"2024-09-16T05:29:20.517456+0000","last_peered":"2024-09-16T05:29:20.517456+0000","last_clean":"2024-09-16T05:29:20.517456+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:29:20.517456+0000","last_undegraded":"2024-09-16T05:29:20.517456+0000","last_fullsized":"2024-09-16T05:29:20.517456+0000","mapping_epoch":759,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:27:58.979002+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:27:58.979002+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":827,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.150389+0000","last_change":"2024-09-16T05:28:17.407013+0000","last_active":"2024-09-16T05:29:20.150389+0000","last_peered":"2024-09-16T05:29:20.150389+0000","last_clean":"2024-09-16T05:29:20.150389+0000","last_became_active":"2024-09-16T05:28:16.164987+0000","last_became_peered":"2024-09-16T05:28:16.164987+0000","last_unstale":"2024-09-16T05:29:20.150389+0000","last_undegraded":"2024-09-16T05:29:20.150389+0000","last_fullsized":"2024-09-16T05:29:20.150389+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:17.406923+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_clean_scrub_stamp":"2024-09-16T05:28:17.406923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.464418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"35'139","reported_seq":2476,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.989112+0000","last_change":"2024-09-16T05:29:03.445323+0000","last_active":"2024-09-16T05:29:20.989112+0000","last_peered":"2024-09-16T05:29:20.989112+0000","last_clean":"2024-09-16T05:29:20.989112+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:29:20.989112+0000","last_undegraded":"2024-09-16T05:29:20.989112+0000","last_fullsized":"2024-09-16T05:29:20.989112+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:03.445237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_clean_scrub_stamp":"2024-09-16T05:29:03.445237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:20.619991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"35'139","reported_seq":2431,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.461539+0000","last_change":"2024-09-16T05:28:35.161241+0000","last_active":"2024-09-16T05:29:20.461539+0000","last_peered":"2024-09-16T05:29:20.461539+0000","last_clean":"2024-09-16T05:29:20.461539+0000","last_became_active":"2024-09-16T05:26:58.978042+0000","last_became_peered":"2024-09-16T05:26:58.978042+0000","last_unstale":"2024-09-16T05:29:20.461539+0000","last_undegraded":"2024-09-16T05:29:20.461539+0000","last_fullsized":"2024-09-16T05:29:20.461539+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:35.161125+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:35.161125+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:05.105469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"35'139","reported_seq":1743,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517454+0000","last_change":"2024-09-16T05:28:58.118422+0000","last_active":"2024-09-16T05:29:20.517454+0000","last_peered":"2024-09-16T05:29:20.517454+0000","last_clean":"2024-09-16T05:29:20.517454+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:29:20.517454+0000","last_undegraded":"2024-09-16T05:29:20.517454+0000","last_fullsized":"2024-09-16T05:29:20.517454+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:58.118326+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:28:58.118326+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.058327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"35'139","reported_seq":1679,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517457+0000","last_change":"2024-09-16T05:28:08.091090+0000","last_active":"2024-09-16T05:29:20.517457+0000","last_peered":"2024-09-16T05:29:20.517457+0000","last_clean":"2024-09-16T05:29:20.517457+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:29:20.517457+0000","last_undegraded":"2024-09-16T05:29:20.517457+0000","last_fullsized":"2024-09-16T05:29:20.517457+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:08.091000+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:08.091000+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:28.574231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1111,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517364+0000","last_change":"2024-09-16T05:29:17.128120+0000","last_active":"2024-09-16T05:29:20.517364+0000","last_peered":"2024-09-16T05:29:20.517364+0000","last_clean":"2024-09-16T05:29:20.517364+0000","last_became_active":"2024-09-16T05:28:24.204654+0000","last_became_peered":"2024-09-16T05:28:24.204654+0000","last_unstale":"2024-09-16T05:29:20.517364+0000","last_undegraded":"2024-09-16T05:29:20.517364+0000","last_fullsized":"2024-09-16T05:29:20.517364+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:17.128029+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:29:17.128029+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:22.966252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"35'139","reported_seq":596,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517484+0000","last_change":"2024-09-16T05:28:20.181305+0000","last_active":"2024-09-16T05:29:20.517484+0000","last_peered":"2024-09-16T05:29:20.517484+0000","last_clean":"2024-09-16T05:29:20.517484+0000","last_became_active":"2024-09-16T05:28:20.181056+0000","last_became_peered":"2024-09-16T05:28:20.181056+0000","last_unstale":"2024-09-16T05:29:20.517484+0000","last_undegraded":"2024-09-16T05:29:20.517484+0000","last_fullsized":"2024-09-16T05:29:20.517484+0000","mapping_epoch":1054,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1055,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:01.145545+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:11.129971+0000","last_clean_scrub_stamp":"2024-09-16T05:28:01.145545+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:25.214675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"35'139","reported_seq":2332,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517143+0000","last_change":"2024-09-16T05:28:42.092940+0000","last_active":"2024-09-16T05:29:20.517143+0000","last_peered":"2024-09-16T05:29:20.517143+0000","last_clean":"2024-09-16T05:29:20.517143+0000","last_became_active":"2024-09-16T05:26:59.168051+0000","last_became_peered":"2024-09-16T05:26:59.168051+0000","last_unstale":"2024-09-16T05:29:20.517143+0000","last_undegraded":"2024-09-16T05:29:20.517143+0000","last_fullsized":"2024-09-16T05:29:20.517143+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:42.092847+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_clean_scrub_stamp":"2024-09-16T05:28:42.092847+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.609055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2476,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517097+0000","last_change":"2024-09-16T05:28:55.235326+0000","last_active":"2024-09-16T05:29:20.517097+0000","last_peered":"2024-09-16T05:29:20.517097+0000","last_clean":"2024-09-16T05:29:20.517097+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:29:20.517097+0000","last_undegraded":"2024-09-16T05:29:20.517097+0000","last_fullsized":"2024-09-16T05:29:20.517097+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:55.235220+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:28:55.235220+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:23.996312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1953,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.145367+0000","last_change":"2024-09-16T05:28:34.191267+0000","last_active":"2024-09-16T05:29:20.145367+0000","last_peered":"2024-09-16T05:29:20.145367+0000","last_clean":"2024-09-16T05:29:20.145367+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:29:20.145367+0000","last_undegraded":"2024-09-16T05:29:20.145367+0000","last_fullsized":"2024-09-16T05:29:20.145367+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:34.191184+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:28:34.191184+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:56.005005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"35'139","reported_seq":1401,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517353+0000","last_change":"2024-09-16T05:28:21.192606+0000","last_active":"2024-09-16T05:29:20.517353+0000","last_peered":"2024-09-16T05:29:20.517353+0000","last_clean":"2024-09-16T05:29:20.517353+0000","last_became_active":"2024-09-16T05:28:21.192346+0000","last_became_peered":"2024-09-16T05:28:21.192346+0000","last_unstale":"2024-09-16T05:29:20.517353+0000","last_undegraded":"2024-09-16T05:29:20.517353+0000","last_fullsized":"2024-09-16T05:29:20.517353+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:05.149072+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:05.149072+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:31.074543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"35'139","reported_seq":1349,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-16T05:29:12.109808+0000","last_change":"2024-09-16T05:28:45.005820+0000","last_active":"2024-09-16T05:29:12.109808+0000","last_peered":"2024-09-16T05:29:12.109808+0000","last_clean":"2024-09-16T05:29:12.109808+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:29:12.109808+0000","last_undegraded":"2024-09-16T05:29:12.109808+0000","last_fullsized":"2024-09-16T05:29:12.109808+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:45.005717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:28:45.005717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.847310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"35'139","reported_seq":1395,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516960+0000","last_change":"2024-09-16T05:28:03.168727+0000","last_active":"2024-09-16T05:29:20.516960+0000","last_peered":"2024-09-16T05:29:20.516960+0000","last_clean":"2024-09-16T05:29:20.516960+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:29:20.516960+0000","last_undegraded":"2024-09-16T05:29:20.516960+0000","last_fullsized":"2024-09-16T05:29:20.516960+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:03.168644+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:28:03.168644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:31.721465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":792,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.856513+0000","last_change":"2024-09-16T05:29:20.856560+0000","last_active":"2024-09-16T05:29:20.856513+0000","last_peered":"2024-09-16T05:29:20.856513+0000","last_clean":"2024-09-16T05:29:20.856513+0000","last_became_active":"2024-09-16T05:28:20.181291+0000","last_became_peered":"2024-09-16T05:28:20.181291+0000","last_unstale":"2024-09-16T05:29:20.856513+0000","last_undegraded":"2024-09-16T05:29:20.856513+0000","last_fullsized":"2024-09-16T05:29:20.856513+0000","mapping_epoch":1054,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1055,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:20.856433+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:29:20.856433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:23.619644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"35'139","reported_seq":1785,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516913+0000","last_change":"2024-09-16T05:29:01.283505+0000","last_active":"2024-09-16T05:29:20.516913+0000","last_peered":"2024-09-16T05:29:20.516913+0000","last_clean":"2024-09-16T05:29:20.516913+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:29:20.516913+0000","last_undegraded":"2024-09-16T05:29:20.516913+0000","last_fullsized":"2024-09-16T05:29:20.516913+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:01.283431+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:29:01.283431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:04.945286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1866,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-16T05:29:12.109772+0000","last_change":"2024-09-16T05:28:00.199933+0000","last_active":"2024-09-16T05:29:12.109772+0000","last_peered":"2024-09-16T05:29:12.109772+0000","last_clean":"2024-09-16T05:29:12.109772+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:29:12.109772+0000","last_undegraded":"2024-09-16T05:29:12.109772+0000","last_fullsized":"2024-09-16T05:29:12.109772+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:00.199841+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:28:00.199841+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.295519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"35'139","reported_seq":1124,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.150389+0000","last_change":"2024-09-16T05:28:16.165544+0000","last_active":"2024-09-16T05:29:20.150389+0000","last_peered":"2024-09-16T05:29:20.150389+0000","last_clean":"2024-09-16T05:29:20.150389+0000","last_became_active":"2024-09-16T05:28:16.165219+0000","last_became_peered":"2024-09-16T05:28:16.165219+0000","last_unstale":"2024-09-16T05:29:20.150389+0000","last_undegraded":"2024-09-16T05:29:20.150389+0000","last_fullsized":"2024-09-16T05:29:20.150389+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:11.963926+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:28:11.963926+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:25.930017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"35'139","reported_seq":2459,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517481+0000","last_change":"2024-09-16T05:28:00.024127+0000","last_active":"2024-09-16T05:29:20.517481+0000","last_peered":"2024-09-16T05:29:20.517481+0000","last_clean":"2024-09-16T05:29:20.517481+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:29:20.517481+0000","last_undegraded":"2024-09-16T05:29:20.517481+0000","last_fullsized":"2024-09-16T05:29:20.517481+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:00.023996+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:28:00.023996+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.810769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"35'139","reported_seq":599,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517634+0000","last_change":"2024-09-16T05:28:21.192959+0000","last_active":"2024-09-16T05:29:20.517634+0000","last_peered":"2024-09-16T05:29:20.517634+0000","last_clean":"2024-09-16T05:29:20.517634+0000","last_became_active":"2024-09-16T05:28:21.192702+0000","last_became_peered":"2024-09-16T05:28:21.192702+0000","last_unstale":"2024-09-16T05:29:20.517634+0000","last_undegraded":"2024-09-16T05:29:20.517634+0000","last_fullsized":"2024-09-16T05:29:20.517634+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:12.248317+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:28:12.248317+0000","last_clean_scrub_stamp":"2024-09-16T05:28:12.248317+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:22.470338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1e","version":"35'139","reported_seq":1951,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.461414+0000","last_change":"2024-09-16T05:28:21.156066+0000","last_active":"2024-09-16T05:29:20.461414+0000","last_peered":"2024-09-16T05:29:20.461414+0000","last_clean":"2024-09-16T05:29:20.461414+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:29:20.461414+0000","last_undegraded":"2024-09-16T05:29:20.461414+0000","last_fullsized":"2024-09-16T05:29:20.461414+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:21.155973+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:28:21.155973+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:30.066462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"35'139","reported_seq":1004,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-16T05:29:12.409182+0000","last_change":"2024-09-16T05:28:36.722915+0000","last_active":"2024-09-16T05:29:12.409182+0000","last_peered":"2024-09-16T05:29:12.409182+0000","last_clean":"2024-09-16T05:29:12.409182+0000","last_became_active":"2024-09-16T05:26:22.907190+0000","last_became_peered":"2024-09-16T05:26:22.907190+0000","last_unstale":"2024-09-16T05:29:12.409182+0000","last_undegraded":"2024-09-16T05:29:12.409182+0000","last_fullsized":"2024-09-16T05:29:12.409182+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:36.722758+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:36.722758+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:57.798323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"35'139","reported_seq":1204,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-09-16T05:29:12.109772+0000","last_change":"2024-09-16T05:28:54.951923+0000","last_active":"2024-09-16T05:29:12.109772+0000","last_peered":"2024-09-16T05:29:12.109772+0000","last_clean":"2024-09-16T05:29:12.109772+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:29:12.109772+0000","last_undegraded":"2024-09-16T05:29:12.109772+0000","last_fullsized":"2024-09-16T05:29:12.109772+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:54.951829+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:54.951829+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:02.103935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57876,"kb_used_data":8640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687276,"statfs":{"total":95995035648,"available":95935770624,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285817,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57864,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687288,"statfs":{"total":95995035648,"available":95935782912,"internally_reserved":0,"allocated":8835072,"data_stored":6069571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514755,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80072,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665080,"statfs":{"total":95995035648,"available":95913041920,"internally_reserved":0,"allocated":8835072,"data_stored":6069571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841795,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57864,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687288,"statfs":{"total":95995035648,"available":95935782912,"internally_reserved":0,"allocated":8835072,"data_stored":6069571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1007,"seq":4325032067103,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36820,"kb_used_data":8640,"kb_used_omap":38,"kb_used_meta":28121,"kb_avail":93708332,"statfs":{"total":95995035648,"available":95957331968,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38947,"internal_metadata":28796893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397775,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62036,"kb_used_data":8640,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683116,"statfs":{"total":95995035648,"available":95931510784,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66456,"kb_used_data":8900,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93678696,"statfs":{"total":95995035648,"available":95926984704,"internally_reserved":0,"allocated":9113600,"data_stored":6339944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921047,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53716,"kb_used_data":8640,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691436,"statfs":{"total":95995035648,"available":95940030464,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117276,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62356,"kb_used_data":8960,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93682796,"statfs":{"total":95995035648,"available":95931183104,"internally_reserved":0,"allocated":9175040,"data_stored":6404936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240563,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45336,"kb_used_data":8900,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93699816,"statfs":{"total":95995035648,"available":95948611584,"internally_reserved":0,"allocated":9113600,"data_stored":6339944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":988,"seq":4243427688486,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45252,"kb_used_data":8628,"kb_used_omap":22,"kb_used_meta":36585,"kb_avail":93699900,"statfs":{"total":95995035648,"available":95948697600,"internally_reserved":0,"allocated":8835072,"data_stored":6069571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22663,"internal_metadata":37463929},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54036,"kb_used_data":8960,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691116,"statfs":{"total":95995035648,"available":95939702784,"internally_reserved":0,"allocated":9175040,"data_stored":6404936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:29:25.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:29:26.169 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: pgmap v1847: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: 1.18 scrub starts 2024-09-16T05:29:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: 1.18 scrub ok 2024-09-16T05:29:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='client.18492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10697072452145957]}]': finished 2024-09-16T05:29:26.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07839473561616196]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12662493825635246]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09505227482163094]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08967680448438438]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08789547212139301]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08064321961146355]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05953026388194416]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0793936025375686]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07519738875996121]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07073993362101365]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09742702848541658]}]': finished 2024-09-16T05:29:26.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:26 smithi032 ceph-mon[27388]: osdmap e1079: 12 total, 12 up, 12 in 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: pgmap v1847: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: 1.18 scrub starts 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: 1.18 scrub ok 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='client.18492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10697072452145957]}]': finished 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07839473561616196]}]': finished 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12662493825635246]}]': finished 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09505227482163094]}]': finished 2024-09-16T05:29:26.422 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08967680448438438]}]': finished 2024-09-16T05:29:26.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08789547212139301]}]': finished 2024-09-16T05:29:26.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08064321961146355]}]': finished 2024-09-16T05:29:26.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05953026388194416]}]': finished 2024-09-16T05:29:26.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0793936025375686]}]': finished 2024-09-16T05:29:26.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07519738875996121]}]': finished 2024-09-16T05:29:26.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07073993362101365]}]': finished 2024-09-16T05:29:26.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09742702848541658]}]': finished 2024-09-16T05:29:26.423 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:26 smithi204 ceph-mon[30933]: osdmap e1079: 12 total, 12 up, 12 in 2024-09-16T05:29:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: pgmap v1847: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: 1.18 scrub starts 2024-09-16T05:29:26.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: 1.18 scrub ok 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='client.18492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.10697072452145957]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.07839473561616196]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.12662493825635246]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09505227482163094]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08967680448438438]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08789547212139301]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08064321961146355]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.05953026388194416]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0793936025375686]}]': finished 2024-09-16T05:29:26.489 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.07519738875996121]}]': finished 2024-09-16T05:29:26.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07073993362101365]}]': finished 2024-09-16T05:29:26.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.09742702848541658]}]': finished 2024-09-16T05:29:26.490 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:26 smithi196 ceph-mon[30802]: osdmap e1079: 12 total, 12 up, 12 in 2024-09-16T05:29:26.818 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:29:26.818 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:29:27.477 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:27 smithi032 ceph-mon[27388]: osdmap e1080: 12 total, 12 up, 12 in 2024-09-16T05:29:27.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:27 smithi196 ceph-mon[30802]: osdmap e1080: 12 total, 12 up, 12 in 2024-09-16T05:29:27.506 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1850,"stamp":"2024-09-16T05:29:26.319330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":679796,"kb_used_data":104900,"kb_used_omap":84,"kb_used_meta":574571,"kb_avail":1124262028,"statfs":{"total":1151940427776,"available":1151244316672,"internally_reserved":0,"allocated":107417600,"data_stored":74153418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86930,"internal_metadata":588360814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.174645"},"pg_stats":[{"pgid":"1.12","version":"35'139","reported_seq":777,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590646+0000","last_change":"2024-09-16T05:28:51.890924+0000","last_active":"2024-09-16T05:29:25.590646+0000","last_peered":"2024-09-16T05:29:25.590646+0000","last_clean":"2024-09-16T05:29:25.590646+0000","last_became_active":"2024-09-16T05:28:21.191459+0000","last_became_peered":"2024-09-16T05:28:21.191459+0000","last_unstale":"2024-09-16T05:29:25.590646+0000","last_undegraded":"2024-09-16T05:29:25.590646+0000","last_fullsized":"2024-09-16T05:29:25.590646+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:51.890834+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:28:51.890834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:19.285120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"35'139","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516797+0000","last_change":"2024-09-16T05:28:29.769978+0000","last_active":"2024-09-16T05:29:20.516797+0000","last_peered":"2024-09-16T05:29:20.516797+0000","last_clean":"2024-09-16T05:29:20.516797+0000","last_became_active":"2024-09-16T05:26:22.907168+0000","last_became_peered":"2024-09-16T05:26:22.907168+0000","last_unstale":"2024-09-16T05:29:20.516797+0000","last_undegraded":"2024-09-16T05:29:20.516797+0000","last_fullsized":"2024-09-16T05:29:20.516797+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:29.769890+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:29.769890+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:46.455950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1798,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.989156+0000","last_change":"2024-09-16T05:28:48.441113+0000","last_active":"2024-09-16T05:29:20.989156+0000","last_peered":"2024-09-16T05:29:20.989156+0000","last_clean":"2024-09-16T05:29:20.989156+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:29:20.989156+0000","last_undegraded":"2024-09-16T05:29:20.989156+0000","last_fullsized":"2024-09-16T05:29:20.989156+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:48.441014+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:48.441014+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:18.091526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"35'139","reported_seq":1852,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517447+0000","last_change":"2024-09-16T05:29:02.856802+0000","last_active":"2024-09-16T05:29:20.517447+0000","last_peered":"2024-09-16T05:29:20.517447+0000","last_clean":"2024-09-16T05:29:20.517447+0000","last_became_active":"2024-09-16T05:26:58.976632+0000","last_became_peered":"2024-09-16T05:26:58.976632+0000","last_unstale":"2024-09-16T05:29:20.517447+0000","last_undegraded":"2024-09-16T05:29:20.517447+0000","last_fullsized":"2024-09-16T05:29:20.517447+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:02.856695+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_clean_scrub_stamp":"2024-09-16T05:29:02.856695+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:31.504867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"35'139","reported_seq":1850,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168820+0000","last_change":"2024-09-16T05:28:51.322038+0000","last_active":"2024-09-16T05:29:25.168820+0000","last_peered":"2024-09-16T05:29:25.168820+0000","last_clean":"2024-09-16T05:29:25.168820+0000","last_became_active":"2024-09-16T05:28:16.165227+0000","last_became_peered":"2024-09-16T05:28:16.165227+0000","last_unstale":"2024-09-16T05:29:25.168820+0000","last_undegraded":"2024-09-16T05:29:25.168820+0000","last_fullsized":"2024-09-16T05:29:25.168820+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:51.321905+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:28:51.321905+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:16.800529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"35'139","reported_seq":1322,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517417+0000","last_change":"2024-09-16T05:29:12.087578+0000","last_active":"2024-09-16T05:29:20.517417+0000","last_peered":"2024-09-16T05:29:20.517417+0000","last_clean":"2024-09-16T05:29:20.517417+0000","last_became_active":"2024-09-16T05:28:24.204528+0000","last_became_peered":"2024-09-16T05:28:24.204528+0000","last_unstale":"2024-09-16T05:29:20.517417+0000","last_undegraded":"2024-09-16T05:29:20.517417+0000","last_fullsized":"2024-09-16T05:29:20.517417+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:12.087489+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_clean_scrub_stamp":"2024-09-16T05:29:12.087489+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:38.977304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"35'139","reported_seq":1495,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516797+0000","last_change":"2024-09-16T05:29:14.497553+0000","last_active":"2024-09-16T05:29:20.516797+0000","last_peered":"2024-09-16T05:29:20.516797+0000","last_clean":"2024-09-16T05:29:20.516797+0000","last_became_active":"2024-09-16T05:26:25.117472+0000","last_became_peered":"2024-09-16T05:26:25.117472+0000","last_unstale":"2024-09-16T05:29:20.516797+0000","last_undegraded":"2024-09-16T05:29:20.516797+0000","last_fullsized":"2024-09-16T05:29:20.516797+0000","mapping_epoch":991,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:14.497468+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:29:14.497468+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:44.280746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"35'139","reported_seq":874,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517208+0000","last_change":"2024-09-16T05:28:30.036111+0000","last_active":"2024-09-16T05:29:20.517208+0000","last_peered":"2024-09-16T05:29:20.517208+0000","last_clean":"2024-09-16T05:29:20.517208+0000","last_became_active":"2024-09-16T05:28:24.205007+0000","last_became_peered":"2024-09-16T05:28:24.205007+0000","last_unstale":"2024-09-16T05:29:20.517208+0000","last_undegraded":"2024-09-16T05:29:20.517208+0000","last_fullsized":"2024-09-16T05:29:20.517208+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:30.036037+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_clean_scrub_stamp":"2024-09-16T05:28:30.036037+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:42.097121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1723,"reported_epoch":1078,"state":"peering","last_fresh":"2024-09-16T05:29:23.046392+0000","last_change":"2024-09-16T05:29:25.169164+0000","last_active":"2024-09-16T05:29:25.169164+0000","last_peered":"2024-09-16T05:29:23.046392+0000","last_clean":"2024-09-16T05:29:23.046392+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:29:23.046392+0000","last_undegraded":"2024-09-16T05:29:23.046392+0000","last_fullsized":"2024-09-16T05:29:23.046392+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:23.046301+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:29:23.046301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[8,9],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":829,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168845+0000","last_change":"2024-09-16T05:28:17.407013+0000","last_active":"2024-09-16T05:29:25.168845+0000","last_peered":"2024-09-16T05:29:25.168845+0000","last_clean":"2024-09-16T05:29:25.168845+0000","last_became_active":"2024-09-16T05:28:16.164987+0000","last_became_peered":"2024-09-16T05:28:16.164987+0000","last_unstale":"2024-09-16T05:29:25.168845+0000","last_undegraded":"2024-09-16T05:29:25.168845+0000","last_fullsized":"2024-09-16T05:29:25.168845+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:17.406923+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_clean_scrub_stamp":"2024-09-16T05:28:17.406923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.464418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"35'139","reported_seq":2476,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.989112+0000","last_change":"2024-09-16T05:29:03.445323+0000","last_active":"2024-09-16T05:29:20.989112+0000","last_peered":"2024-09-16T05:29:20.989112+0000","last_clean":"2024-09-16T05:29:20.989112+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:29:20.989112+0000","last_undegraded":"2024-09-16T05:29:20.989112+0000","last_fullsized":"2024-09-16T05:29:20.989112+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:03.445237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_clean_scrub_stamp":"2024-09-16T05:29:03.445237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:20.619991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"35'139","reported_seq":2431,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.461539+0000","last_change":"2024-09-16T05:28:35.161241+0000","last_active":"2024-09-16T05:29:20.461539+0000","last_peered":"2024-09-16T05:29:20.461539+0000","last_clean":"2024-09-16T05:29:20.461539+0000","last_became_active":"2024-09-16T05:26:58.978042+0000","last_became_peered":"2024-09-16T05:26:58.978042+0000","last_unstale":"2024-09-16T05:29:20.461539+0000","last_undegraded":"2024-09-16T05:29:20.461539+0000","last_fullsized":"2024-09-16T05:29:20.461539+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:35.161125+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:35.161125+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:05.105469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"35'139","reported_seq":1745,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168657+0000","last_change":"2024-09-16T05:28:58.118422+0000","last_active":"2024-09-16T05:29:25.168657+0000","last_peered":"2024-09-16T05:29:25.168657+0000","last_clean":"2024-09-16T05:29:25.168657+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:29:25.168657+0000","last_undegraded":"2024-09-16T05:29:25.168657+0000","last_fullsized":"2024-09-16T05:29:25.168657+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:58.118326+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:28:58.118326+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.058327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"35'139","reported_seq":1681,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168585+0000","last_change":"2024-09-16T05:28:08.091090+0000","last_active":"2024-09-16T05:29:25.168585+0000","last_peered":"2024-09-16T05:29:25.168585+0000","last_clean":"2024-09-16T05:29:25.168585+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:29:25.168585+0000","last_undegraded":"2024-09-16T05:29:25.168585+0000","last_fullsized":"2024-09-16T05:29:25.168585+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:08.091000+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:08.091000+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:28.574231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1111,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517364+0000","last_change":"2024-09-16T05:29:17.128120+0000","last_active":"2024-09-16T05:29:20.517364+0000","last_peered":"2024-09-16T05:29:20.517364+0000","last_clean":"2024-09-16T05:29:20.517364+0000","last_became_active":"2024-09-16T05:28:24.204654+0000","last_became_peered":"2024-09-16T05:28:24.204654+0000","last_unstale":"2024-09-16T05:29:20.517364+0000","last_undegraded":"2024-09-16T05:29:20.517364+0000","last_fullsized":"2024-09-16T05:29:20.517364+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:17.128029+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:29:17.128029+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:22.966252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"35'139","reported_seq":598,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590411+0000","last_change":"2024-09-16T05:28:20.181305+0000","last_active":"2024-09-16T05:29:25.590411+0000","last_peered":"2024-09-16T05:29:25.590411+0000","last_clean":"2024-09-16T05:29:25.590411+0000","last_became_active":"2024-09-16T05:28:20.181056+0000","last_became_peered":"2024-09-16T05:28:20.181056+0000","last_unstale":"2024-09-16T05:29:25.590411+0000","last_undegraded":"2024-09-16T05:29:25.590411+0000","last_fullsized":"2024-09-16T05:29:25.590411+0000","mapping_epoch":1054,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1055,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:01.145545+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:11.129971+0000","last_clean_scrub_stamp":"2024-09-16T05:28:01.145545+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"35'139","reported_seq":2332,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.517143+0000","last_change":"2024-09-16T05:28:42.092940+0000","last_active":"2024-09-16T05:29:20.517143+0000","last_peered":"2024-09-16T05:29:20.517143+0000","last_clean":"2024-09-16T05:29:20.517143+0000","last_became_active":"2024-09-16T05:26:59.168051+0000","last_became_peered":"2024-09-16T05:26:59.168051+0000","last_unstale":"2024-09-16T05:29:20.517143+0000","last_undegraded":"2024-09-16T05:29:20.517143+0000","last_fullsized":"2024-09-16T05:29:20.517143+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:42.092847+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_clean_scrub_stamp":"2024-09-16T05:28:42.092847+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.609055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2478,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.170644+0000","last_change":"2024-09-16T05:28:55.235326+0000","last_active":"2024-09-16T05:29:25.170644+0000","last_peered":"2024-09-16T05:29:25.170644+0000","last_clean":"2024-09-16T05:29:25.170644+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:29:25.170644+0000","last_undegraded":"2024-09-16T05:29:25.170644+0000","last_fullsized":"2024-09-16T05:29:25.170644+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:55.235220+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:28:55.235220+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:23.996312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1955,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168644+0000","last_change":"2024-09-16T05:28:34.191267+0000","last_active":"2024-09-16T05:29:25.168644+0000","last_peered":"2024-09-16T05:29:25.168644+0000","last_clean":"2024-09-16T05:29:25.168644+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:29:25.168644+0000","last_undegraded":"2024-09-16T05:29:25.168644+0000","last_fullsized":"2024-09-16T05:29:25.168644+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:34.191184+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:28:34.191184+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:56.005005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"35'139","reported_seq":1403,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590418+0000","last_change":"2024-09-16T05:28:21.192606+0000","last_active":"2024-09-16T05:29:25.590418+0000","last_peered":"2024-09-16T05:29:25.590418+0000","last_clean":"2024-09-16T05:29:25.590418+0000","last_became_active":"2024-09-16T05:28:21.192346+0000","last_became_peered":"2024-09-16T05:28:21.192346+0000","last_unstale":"2024-09-16T05:29:25.590418+0000","last_undegraded":"2024-09-16T05:29:25.590418+0000","last_fullsized":"2024-09-16T05:29:25.590418+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:05.149072+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:05.149072+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:31.074543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"35'139","reported_seq":1351,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.147897+0000","last_change":"2024-09-16T05:28:45.005820+0000","last_active":"2024-09-16T05:29:20.147897+0000","last_peered":"2024-09-16T05:29:20.147897+0000","last_clean":"2024-09-16T05:29:20.147897+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:29:20.147897+0000","last_undegraded":"2024-09-16T05:29:20.147897+0000","last_fullsized":"2024-09-16T05:29:20.147897+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:45.005717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:28:45.005717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.847310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"35'139","reported_seq":1395,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516960+0000","last_change":"2024-09-16T05:28:03.168727+0000","last_active":"2024-09-16T05:29:20.516960+0000","last_peered":"2024-09-16T05:29:20.516960+0000","last_clean":"2024-09-16T05:29:20.516960+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:29:20.516960+0000","last_undegraded":"2024-09-16T05:29:20.516960+0000","last_fullsized":"2024-09-16T05:29:20.516960+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:03.168644+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:28:03.168644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:31.721465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":794,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590674+0000","last_change":"2024-09-16T05:29:20.856560+0000","last_active":"2024-09-16T05:29:25.590674+0000","last_peered":"2024-09-16T05:29:25.590674+0000","last_clean":"2024-09-16T05:29:25.590674+0000","last_became_active":"2024-09-16T05:28:20.181291+0000","last_became_peered":"2024-09-16T05:28:20.181291+0000","last_unstale":"2024-09-16T05:29:25.590674+0000","last_undegraded":"2024-09-16T05:29:25.590674+0000","last_fullsized":"2024-09-16T05:29:25.590674+0000","mapping_epoch":1054,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1055,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:20.856433+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:29:20.856433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:23.619644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"35'139","reported_seq":1785,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516913+0000","last_change":"2024-09-16T05:29:01.283505+0000","last_active":"2024-09-16T05:29:20.516913+0000","last_peered":"2024-09-16T05:29:20.516913+0000","last_clean":"2024-09-16T05:29:20.516913+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:29:20.516913+0000","last_undegraded":"2024-09-16T05:29:20.516913+0000","last_fullsized":"2024-09-16T05:29:20.516913+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:01.283431+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:29:01.283431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:04.945286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1868,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.147800+0000","last_change":"2024-09-16T05:28:00.199933+0000","last_active":"2024-09-16T05:29:20.147800+0000","last_peered":"2024-09-16T05:29:20.147800+0000","last_clean":"2024-09-16T05:29:20.147800+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:29:20.147800+0000","last_undegraded":"2024-09-16T05:29:20.147800+0000","last_fullsized":"2024-09-16T05:29:20.147800+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:00.199841+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:28:00.199841+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.295519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"35'139","reported_seq":1126,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168837+0000","last_change":"2024-09-16T05:28:16.165544+0000","last_active":"2024-09-16T05:29:25.168837+0000","last_peered":"2024-09-16T05:29:25.168837+0000","last_clean":"2024-09-16T05:29:25.168837+0000","last_became_active":"2024-09-16T05:28:16.165219+0000","last_became_peered":"2024-09-16T05:28:16.165219+0000","last_unstale":"2024-09-16T05:29:25.168837+0000","last_undegraded":"2024-09-16T05:29:25.168837+0000","last_fullsized":"2024-09-16T05:29:25.168837+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:11.963926+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:28:11.963926+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:25.930017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"35'139","reported_seq":2461,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168561+0000","last_change":"2024-09-16T05:28:00.024127+0000","last_active":"2024-09-16T05:29:25.168561+0000","last_peered":"2024-09-16T05:29:25.168561+0000","last_clean":"2024-09-16T05:29:25.168561+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:29:25.168561+0000","last_undegraded":"2024-09-16T05:29:25.168561+0000","last_fullsized":"2024-09-16T05:29:25.168561+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:00.023996+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:28:00.023996+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.810769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"35'139","reported_seq":609,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590616+0000","last_change":"2024-09-16T05:29:24.891712+0000","last_active":"2024-09-16T05:29:25.590616+0000","last_peered":"2024-09-16T05:29:25.590616+0000","last_clean":"2024-09-16T05:29:25.590616+0000","last_became_active":"2024-09-16T05:28:21.192702+0000","last_became_peered":"2024-09-16T05:28:21.192702+0000","last_unstale":"2024-09-16T05:29:25.590616+0000","last_undegraded":"2024-09-16T05:29:25.590616+0000","last_fullsized":"2024-09-16T05:29:25.590616+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:24.891609+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:28:12.248317+0000","last_clean_scrub_stamp":"2024-09-16T05:29:24.891609+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:46.196747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1e","version":"35'139","reported_seq":1951,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.461414+0000","last_change":"2024-09-16T05:28:21.156066+0000","last_active":"2024-09-16T05:29:20.461414+0000","last_peered":"2024-09-16T05:29:20.461414+0000","last_clean":"2024-09-16T05:29:20.461414+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:29:20.461414+0000","last_undegraded":"2024-09-16T05:29:20.461414+0000","last_fullsized":"2024-09-16T05:29:20.461414+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:21.155973+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:28:21.155973+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:30.066462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"35'139","reported_seq":1006,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516797+0000","last_change":"2024-09-16T05:28:36.722915+0000","last_active":"2024-09-16T05:29:20.516797+0000","last_peered":"2024-09-16T05:29:20.516797+0000","last_clean":"2024-09-16T05:29:20.516797+0000","last_became_active":"2024-09-16T05:26:22.907190+0000","last_became_peered":"2024-09-16T05:26:22.907190+0000","last_unstale":"2024-09-16T05:29:20.516797+0000","last_undegraded":"2024-09-16T05:29:20.516797+0000","last_fullsized":"2024-09-16T05:29:20.516797+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:36.722758+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:36.722758+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:57.798323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"35'139","reported_seq":1206,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.147803+0000","last_change":"2024-09-16T05:28:54.951923+0000","last_active":"2024-09-16T05:29:20.147803+0000","last_peered":"2024-09-16T05:29:20.147803+0000","last_clean":"2024-09-16T05:29:20.147803+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:29:20.147803+0000","last_undegraded":"2024-09-16T05:29:20.147803+0000","last_fullsized":"2024-09-16T05:29:20.147803+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:54.951829+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:54.951829+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:02.103935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57876,"kb_used_data":8640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687276,"statfs":{"total":95995035648,"available":95935770624,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285818,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57888,"kb_used_data":8652,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687264,"statfs":{"total":95995035648,"available":95935758336,"internally_reserved":0,"allocated":8859648,"data_stored":6086378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514756,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80096,"kb_used_data":8652,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665056,"statfs":{"total":95995035648,"available":95913017344,"internally_reserved":0,"allocated":8859648,"data_stored":6086378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841796,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57876,"kb_used_data":8640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687276,"statfs":{"total":95995035648,"available":95935770624,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1007,"seq":4325032067103,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36820,"kb_used_data":8640,"kb_used_omap":38,"kb_used_meta":28121,"kb_avail":93708332,"statfs":{"total":95995035648,"available":95957331968,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38947,"internal_metadata":28796893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397775,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62036,"kb_used_data":8640,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683116,"statfs":{"total":95995035648,"available":95931510784,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757523,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66468,"kb_used_data":8912,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93678684,"statfs":{"total":95995035648,"available":95926972416,"internally_reserved":0,"allocated":9125888,"data_stored":6349066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921048,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53728,"kb_used_data":8652,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691424,"statfs":{"total":95995035648,"available":95940018176,"internally_reserved":0,"allocated":8859648,"data_stored":6086378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117276,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62356,"kb_used_data":8960,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93682796,"statfs":{"total":95995035648,"available":95931183104,"internally_reserved":0,"allocated":9175040,"data_stored":6404936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240563,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45336,"kb_used_data":8900,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93699816,"statfs":{"total":95995035648,"available":95948611584,"internally_reserved":0,"allocated":9113600,"data_stored":6339944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":988,"seq":4243427688487,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45268,"kb_used_data":8640,"kb_used_omap":22,"kb_used_meta":36585,"kb_avail":93699884,"statfs":{"total":95995035648,"available":95948681216,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22663,"internal_metadata":37463929},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54048,"kb_used_data":8972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691104,"statfs":{"total":95995035648,"available":95939690496,"internally_reserved":0,"allocated":9187328,"data_stored":6414058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:29:27.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-16T05:29:27.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:27 smithi204 ceph-mon[30933]: osdmap e1080: 12 total, 12 up, 12 in 2024-09-16T05:29:27.763 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:28.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:28 smithi032 ceph-mon[27388]: pgmap v1850: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:28.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:28 smithi032 ceph-mon[27388]: from='client.18498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:28.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:28 smithi032 ceph-mon[27388]: 1.1b scrub starts 2024-09-16T05:29:28.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:28 smithi032 ceph-mon[27388]: 1.1b scrub ok 2024-09-16T05:29:28.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:28 smithi196 ceph-mon[30802]: pgmap v1850: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:28 smithi196 ceph-mon[30802]: from='client.18498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:28 smithi196 ceph-mon[30802]: 1.1b scrub starts 2024-09-16T05:29:28.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:28 smithi196 ceph-mon[30802]: 1.1b scrub ok 2024-09-16T05:29:28.497 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:29:28.497 INFO:teuthology.orchestra.run.smithi032.stdout:{"fsid":"c1b4ea6a-73e8-11ef-bceb-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1512,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1080,"num_osds":12,"num_up_osds":12,"osd_up_since":1726464417,"num_in_osds":12,"osd_in_since":1726464113,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":30},{"state_name":"peering","count":1}],"num_pgs":31,"num_pools":1,"num_objects":2,"data_bytes":590368,"bytes_used":696111104,"bytes_avail":1151244316672,"bytes_total":1151940427776,"inactive_pgs_ratio":0.032258063554763794},"fsmap":{"epoch":1,"btime":"2024-09-16T05:02:35:583250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.32:8443/"}},"servicemap":{"epoch":41,"modified":"2024-09-16T05:28:06.303186+0000","services":{}},"progress_events":{}} 2024-09-16T05:29:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:28 smithi204 ceph-mon[30933]: pgmap v1850: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:28 smithi204 ceph-mon[30933]: from='client.18498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:28 smithi204 ceph-mon[30933]: 1.1b scrub starts 2024-09-16T05:29:28.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:28 smithi204 ceph-mon[30933]: 1.1b scrub ok 2024-09-16T05:29:29.165 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T05:29:29.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:29:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:29 smithi032 ceph-mon[27388]: 1.8 scrub starts 2024-09-16T05:29:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:29 smithi032 ceph-mon[27388]: 1.8 scrub ok 2024-09-16T05:29:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:29 smithi032 ceph-mon[27388]: 1.19 scrub starts 2024-09-16T05:29:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:29 smithi032 ceph-mon[27388]: 1.19 scrub ok 2024-09-16T05:29:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:29 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:29:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:29 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:29:29.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:29 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/22254620' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-16T05:29:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:29.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:29 smithi196 ceph-mon[30802]: 1.8 scrub starts 2024-09-16T05:29:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:29 smithi196 ceph-mon[30802]: 1.8 scrub ok 2024-09-16T05:29:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:29 smithi196 ceph-mon[30802]: 1.19 scrub starts 2024-09-16T05:29:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:29 smithi196 ceph-mon[30802]: 1.19 scrub ok 2024-09-16T05:29:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:29 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:29:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:29 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:29:29.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:29 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/22254620' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-16T05:29:29.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:29 smithi204 ceph-mon[30933]: 1.8 scrub starts 2024-09-16T05:29:29.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:29 smithi204 ceph-mon[30933]: 1.8 scrub ok 2024-09-16T05:29:29.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:29 smithi204 ceph-mon[30933]: 1.19 scrub starts 2024-09-16T05:29:29.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:29 smithi204 ceph-mon[30933]: 1.19 scrub ok 2024-09-16T05:29:29.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:29 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:29:29.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:29 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:29:29.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:29 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/22254620' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-16T05:29:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:29:30.041 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:29:30.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:30 smithi196 ceph-mon[30802]: pgmap v1851: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:30 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:29:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:30 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:29:30.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:30 smithi196 ceph-mon[30802]: osdmap e1081: 12 total, 12 up, 12 in 2024-09-16T05:29:30.668 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:30 smithi032 ceph-mon[27388]: pgmap v1851: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:30.668 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:30 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:29:30.668 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:30 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:29:30.668 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:30 smithi032 ceph-mon[27388]: osdmap e1081: 12 total, 12 up, 12 in 2024-09-16T05:29:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:30 smithi204 ceph-mon[30933]: pgmap v1851: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:30 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:29:30.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:30 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:29:30.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:30 smithi204 ceph-mon[30933]: osdmap e1081: 12 total, 12 up, 12 in 2024-09-16T05:29:30.698 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1852,"stamp":"2024-09-16T05:29:29.198957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":679928,"kb_used_data":105020,"kb_used_omap":84,"kb_used_meta":574571,"kb_avail":1124261896,"statfs":{"total":1151940427776,"available":1151244181504,"internally_reserved":0,"allocated":107540480,"data_stored":74237243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86930,"internal_metadata":588360814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.880349"},"pg_stats":[{"pgid":"1.12","version":"35'139","reported_seq":777,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590646+0000","last_change":"2024-09-16T05:28:51.890924+0000","last_active":"2024-09-16T05:29:25.590646+0000","last_peered":"2024-09-16T05:29:25.590646+0000","last_clean":"2024-09-16T05:29:25.590646+0000","last_became_active":"2024-09-16T05:28:21.191459+0000","last_became_peered":"2024-09-16T05:28:21.191459+0000","last_unstale":"2024-09-16T05:29:25.590646+0000","last_undegraded":"2024-09-16T05:29:25.590646+0000","last_fullsized":"2024-09-16T05:29:25.590646+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:51.890834+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:28:51.890834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:19.285120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"35'139","reported_seq":1510,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516797+0000","last_change":"2024-09-16T05:28:29.769978+0000","last_active":"2024-09-16T05:29:20.516797+0000","last_peered":"2024-09-16T05:29:20.516797+0000","last_clean":"2024-09-16T05:29:20.516797+0000","last_became_active":"2024-09-16T05:26:22.907168+0000","last_became_peered":"2024-09-16T05:26:22.907168+0000","last_unstale":"2024-09-16T05:29:20.516797+0000","last_undegraded":"2024-09-16T05:29:20.516797+0000","last_fullsized":"2024-09-16T05:29:20.516797+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:29.769890+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:29.769890+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:46.455950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1802,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:26.415029+0000","last_change":"2024-09-16T05:28:48.441113+0000","last_active":"2024-09-16T05:29:26.415029+0000","last_peered":"2024-09-16T05:29:26.415029+0000","last_clean":"2024-09-16T05:29:26.415029+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:29:26.415029+0000","last_undegraded":"2024-09-16T05:29:26.415029+0000","last_fullsized":"2024-09-16T05:29:26.415029+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:48.441014+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:48.441014+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:18.091526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"35'139","reported_seq":1856,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:26.415520+0000","last_change":"2024-09-16T05:29:02.856802+0000","last_active":"2024-09-16T05:29:26.415520+0000","last_peered":"2024-09-16T05:29:26.415520+0000","last_clean":"2024-09-16T05:29:26.415520+0000","last_became_active":"2024-09-16T05:26:58.976632+0000","last_became_peered":"2024-09-16T05:26:58.976632+0000","last_unstale":"2024-09-16T05:29:26.415520+0000","last_undegraded":"2024-09-16T05:29:26.415520+0000","last_fullsized":"2024-09-16T05:29:26.415520+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:02.856695+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_clean_scrub_stamp":"2024-09-16T05:29:02.856695+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:31.504867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"35'139","reported_seq":1850,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168820+0000","last_change":"2024-09-16T05:28:51.322038+0000","last_active":"2024-09-16T05:29:25.168820+0000","last_peered":"2024-09-16T05:29:25.168820+0000","last_clean":"2024-09-16T05:29:25.168820+0000","last_became_active":"2024-09-16T05:28:16.165227+0000","last_became_peered":"2024-09-16T05:28:16.165227+0000","last_unstale":"2024-09-16T05:29:25.168820+0000","last_undegraded":"2024-09-16T05:29:25.168820+0000","last_fullsized":"2024-09-16T05:29:25.168820+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:51.321905+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:28:51.321905+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:16.800529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"35'139","reported_seq":1326,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:27.011368+0000","last_change":"2024-09-16T05:29:12.087578+0000","last_active":"2024-09-16T05:29:27.011368+0000","last_peered":"2024-09-16T05:29:27.011368+0000","last_clean":"2024-09-16T05:29:27.011368+0000","last_became_active":"2024-09-16T05:28:24.204528+0000","last_became_peered":"2024-09-16T05:28:24.204528+0000","last_unstale":"2024-09-16T05:29:27.011368+0000","last_undegraded":"2024-09-16T05:29:27.011368+0000","last_fullsized":"2024-09-16T05:29:27.011368+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:12.087489+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_clean_scrub_stamp":"2024-09-16T05:29:12.087489+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:38.977304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"35'139","reported_seq":1495,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516797+0000","last_change":"2024-09-16T05:29:14.497553+0000","last_active":"2024-09-16T05:29:20.516797+0000","last_peered":"2024-09-16T05:29:20.516797+0000","last_clean":"2024-09-16T05:29:20.516797+0000","last_became_active":"2024-09-16T05:26:25.117472+0000","last_became_peered":"2024-09-16T05:26:25.117472+0000","last_unstale":"2024-09-16T05:29:20.516797+0000","last_undegraded":"2024-09-16T05:29:20.516797+0000","last_fullsized":"2024-09-16T05:29:20.516797+0000","mapping_epoch":991,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:14.497468+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:29:14.497468+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:44.280746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"35'139","reported_seq":878,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:27.011331+0000","last_change":"2024-09-16T05:28:30.036111+0000","last_active":"2024-09-16T05:29:27.011331+0000","last_peered":"2024-09-16T05:29:27.011331+0000","last_clean":"2024-09-16T05:29:27.011331+0000","last_became_active":"2024-09-16T05:28:24.205007+0000","last_became_peered":"2024-09-16T05:28:24.205007+0000","last_unstale":"2024-09-16T05:29:27.011331+0000","last_undegraded":"2024-09-16T05:29:27.011331+0000","last_fullsized":"2024-09-16T05:29:27.011331+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:30.036037+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_clean_scrub_stamp":"2024-09-16T05:28:30.036037+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:42.097121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1723,"reported_epoch":1078,"state":"peering","last_fresh":"2024-09-16T05:29:23.046392+0000","last_change":"2024-09-16T05:29:25.169164+0000","last_active":"2024-09-16T05:29:25.169164+0000","last_peered":"2024-09-16T05:29:23.046392+0000","last_clean":"2024-09-16T05:29:23.046392+0000","last_became_active":"2024-09-16T05:21:49.835694+0000","last_became_peered":"2024-09-16T05:21:49.835694+0000","last_unstale":"2024-09-16T05:29:23.046392+0000","last_undegraded":"2024-09-16T05:29:23.046392+0000","last_fullsized":"2024-09-16T05:29:23.046392+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":760,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:23.046301+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:29:23.046301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[8,9],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":829,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168845+0000","last_change":"2024-09-16T05:28:17.407013+0000","last_active":"2024-09-16T05:29:25.168845+0000","last_peered":"2024-09-16T05:29:25.168845+0000","last_clean":"2024-09-16T05:29:25.168845+0000","last_became_active":"2024-09-16T05:28:16.164987+0000","last_became_peered":"2024-09-16T05:28:16.164987+0000","last_unstale":"2024-09-16T05:29:25.168845+0000","last_undegraded":"2024-09-16T05:29:25.168845+0000","last_fullsized":"2024-09-16T05:29:25.168845+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:17.406923+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_clean_scrub_stamp":"2024-09-16T05:28:17.406923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.464418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"35'139","reported_seq":2480,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:26.415030+0000","last_change":"2024-09-16T05:29:03.445323+0000","last_active":"2024-09-16T05:29:26.415030+0000","last_peered":"2024-09-16T05:29:26.415030+0000","last_clean":"2024-09-16T05:29:26.415030+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:29:26.415030+0000","last_undegraded":"2024-09-16T05:29:26.415030+0000","last_fullsized":"2024-09-16T05:29:26.415030+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:03.445237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_clean_scrub_stamp":"2024-09-16T05:29:03.445237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:20.619991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"35'139","reported_seq":2435,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:26.415429+0000","last_change":"2024-09-16T05:28:35.161241+0000","last_active":"2024-09-16T05:29:26.415429+0000","last_peered":"2024-09-16T05:29:26.415429+0000","last_clean":"2024-09-16T05:29:26.415429+0000","last_became_active":"2024-09-16T05:26:58.978042+0000","last_became_peered":"2024-09-16T05:26:58.978042+0000","last_unstale":"2024-09-16T05:29:26.415429+0000","last_undegraded":"2024-09-16T05:29:26.415429+0000","last_fullsized":"2024-09-16T05:29:26.415429+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:35.161125+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:35.161125+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:05.105469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"35'139","reported_seq":1745,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168657+0000","last_change":"2024-09-16T05:28:58.118422+0000","last_active":"2024-09-16T05:29:25.168657+0000","last_peered":"2024-09-16T05:29:25.168657+0000","last_clean":"2024-09-16T05:29:25.168657+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:29:25.168657+0000","last_undegraded":"2024-09-16T05:29:25.168657+0000","last_fullsized":"2024-09-16T05:29:25.168657+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:58.118326+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:28:58.118326+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.058327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"35'139","reported_seq":1681,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168585+0000","last_change":"2024-09-16T05:28:08.091090+0000","last_active":"2024-09-16T05:29:25.168585+0000","last_peered":"2024-09-16T05:29:25.168585+0000","last_clean":"2024-09-16T05:29:25.168585+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:29:25.168585+0000","last_undegraded":"2024-09-16T05:29:25.168585+0000","last_fullsized":"2024-09-16T05:29:25.168585+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:08.091000+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:08.091000+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:28.574231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1115,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:27.011352+0000","last_change":"2024-09-16T05:29:17.128120+0000","last_active":"2024-09-16T05:29:27.011352+0000","last_peered":"2024-09-16T05:29:27.011352+0000","last_clean":"2024-09-16T05:29:27.011352+0000","last_became_active":"2024-09-16T05:28:24.204654+0000","last_became_peered":"2024-09-16T05:28:24.204654+0000","last_unstale":"2024-09-16T05:29:27.011352+0000","last_undegraded":"2024-09-16T05:29:27.011352+0000","last_fullsized":"2024-09-16T05:29:27.011352+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:17.128029+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:29:17.128029+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:22.966252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"35'139","reported_seq":598,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590411+0000","last_change":"2024-09-16T05:28:20.181305+0000","last_active":"2024-09-16T05:29:25.590411+0000","last_peered":"2024-09-16T05:29:25.590411+0000","last_clean":"2024-09-16T05:29:25.590411+0000","last_became_active":"2024-09-16T05:28:20.181056+0000","last_became_peered":"2024-09-16T05:28:20.181056+0000","last_unstale":"2024-09-16T05:29:25.590411+0000","last_undegraded":"2024-09-16T05:29:25.590411+0000","last_fullsized":"2024-09-16T05:29:25.590411+0000","mapping_epoch":1054,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1055,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:01.145545+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:11.129971+0000","last_clean_scrub_stamp":"2024-09-16T05:28:01.145545+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"35'139","reported_seq":2336,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:26.415506+0000","last_change":"2024-09-16T05:28:42.092940+0000","last_active":"2024-09-16T05:29:26.415506+0000","last_peered":"2024-09-16T05:29:26.415506+0000","last_clean":"2024-09-16T05:29:26.415506+0000","last_became_active":"2024-09-16T05:26:59.168051+0000","last_became_peered":"2024-09-16T05:26:59.168051+0000","last_unstale":"2024-09-16T05:29:26.415506+0000","last_undegraded":"2024-09-16T05:29:26.415506+0000","last_fullsized":"2024-09-16T05:29:26.415506+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:42.092847+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_clean_scrub_stamp":"2024-09-16T05:28:42.092847+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.609055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2478,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.170644+0000","last_change":"2024-09-16T05:28:55.235326+0000","last_active":"2024-09-16T05:29:25.170644+0000","last_peered":"2024-09-16T05:29:25.170644+0000","last_clean":"2024-09-16T05:29:25.170644+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:29:25.170644+0000","last_undegraded":"2024-09-16T05:29:25.170644+0000","last_fullsized":"2024-09-16T05:29:25.170644+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:55.235220+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:28:55.235220+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:23.996312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1955,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168644+0000","last_change":"2024-09-16T05:28:34.191267+0000","last_active":"2024-09-16T05:29:25.168644+0000","last_peered":"2024-09-16T05:29:25.168644+0000","last_clean":"2024-09-16T05:29:25.168644+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:29:25.168644+0000","last_undegraded":"2024-09-16T05:29:25.168644+0000","last_fullsized":"2024-09-16T05:29:25.168644+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:34.191184+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:28:34.191184+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:56.005005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"35'139","reported_seq":1403,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590418+0000","last_change":"2024-09-16T05:28:21.192606+0000","last_active":"2024-09-16T05:29:25.590418+0000","last_peered":"2024-09-16T05:29:25.590418+0000","last_clean":"2024-09-16T05:29:25.590418+0000","last_became_active":"2024-09-16T05:28:21.192346+0000","last_became_peered":"2024-09-16T05:28:21.192346+0000","last_unstale":"2024-09-16T05:29:25.590418+0000","last_undegraded":"2024-09-16T05:29:25.590418+0000","last_fullsized":"2024-09-16T05:29:25.590418+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:05.149072+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:05.149072+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:31.074543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"35'139","reported_seq":1351,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.147897+0000","last_change":"2024-09-16T05:28:45.005820+0000","last_active":"2024-09-16T05:29:20.147897+0000","last_peered":"2024-09-16T05:29:20.147897+0000","last_clean":"2024-09-16T05:29:20.147897+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:29:20.147897+0000","last_undegraded":"2024-09-16T05:29:20.147897+0000","last_fullsized":"2024-09-16T05:29:20.147897+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:45.005717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:28:45.005717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.847310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"35'139","reported_seq":1399,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:26.415101+0000","last_change":"2024-09-16T05:28:03.168727+0000","last_active":"2024-09-16T05:29:26.415101+0000","last_peered":"2024-09-16T05:29:26.415101+0000","last_clean":"2024-09-16T05:29:26.415101+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:29:26.415101+0000","last_undegraded":"2024-09-16T05:29:26.415101+0000","last_fullsized":"2024-09-16T05:29:26.415101+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:03.168644+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:28:03.168644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:31.721465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":794,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590674+0000","last_change":"2024-09-16T05:29:20.856560+0000","last_active":"2024-09-16T05:29:25.590674+0000","last_peered":"2024-09-16T05:29:25.590674+0000","last_clean":"2024-09-16T05:29:25.590674+0000","last_became_active":"2024-09-16T05:28:20.181291+0000","last_became_peered":"2024-09-16T05:28:20.181291+0000","last_unstale":"2024-09-16T05:29:25.590674+0000","last_undegraded":"2024-09-16T05:29:25.590674+0000","last_fullsized":"2024-09-16T05:29:25.590674+0000","mapping_epoch":1054,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1055,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:20.856433+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:29:20.856433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:23.619644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"35'139","reported_seq":1789,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:26.415066+0000","last_change":"2024-09-16T05:29:01.283505+0000","last_active":"2024-09-16T05:29:26.415066+0000","last_peered":"2024-09-16T05:29:26.415066+0000","last_clean":"2024-09-16T05:29:26.415066+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:29:26.415066+0000","last_undegraded":"2024-09-16T05:29:26.415066+0000","last_fullsized":"2024-09-16T05:29:26.415066+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:01.283431+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:29:01.283431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:04.945286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1868,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.147800+0000","last_change":"2024-09-16T05:28:00.199933+0000","last_active":"2024-09-16T05:29:20.147800+0000","last_peered":"2024-09-16T05:29:20.147800+0000","last_clean":"2024-09-16T05:29:20.147800+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:29:20.147800+0000","last_undegraded":"2024-09-16T05:29:20.147800+0000","last_fullsized":"2024-09-16T05:29:20.147800+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:00.199841+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:28:00.199841+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.295519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"35'139","reported_seq":1126,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168837+0000","last_change":"2024-09-16T05:28:16.165544+0000","last_active":"2024-09-16T05:29:25.168837+0000","last_peered":"2024-09-16T05:29:25.168837+0000","last_clean":"2024-09-16T05:29:25.168837+0000","last_became_active":"2024-09-16T05:28:16.165219+0000","last_became_peered":"2024-09-16T05:28:16.165219+0000","last_unstale":"2024-09-16T05:29:25.168837+0000","last_undegraded":"2024-09-16T05:29:25.168837+0000","last_fullsized":"2024-09-16T05:29:25.168837+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:11.963926+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:28:11.963926+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:25.930017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"35'139","reported_seq":2461,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.168561+0000","last_change":"2024-09-16T05:28:00.024127+0000","last_active":"2024-09-16T05:29:25.168561+0000","last_peered":"2024-09-16T05:29:25.168561+0000","last_clean":"2024-09-16T05:29:25.168561+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:29:25.168561+0000","last_undegraded":"2024-09-16T05:29:25.168561+0000","last_fullsized":"2024-09-16T05:29:25.168561+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:00.023996+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:28:00.023996+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:26.810769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"35'139","reported_seq":609,"reported_epoch":1079,"state":"active+clean","last_fresh":"2024-09-16T05:29:25.590616+0000","last_change":"2024-09-16T05:29:24.891712+0000","last_active":"2024-09-16T05:29:25.590616+0000","last_peered":"2024-09-16T05:29:25.590616+0000","last_clean":"2024-09-16T05:29:25.590616+0000","last_became_active":"2024-09-16T05:28:21.192702+0000","last_became_peered":"2024-09-16T05:28:21.192702+0000","last_unstale":"2024-09-16T05:29:25.590616+0000","last_undegraded":"2024-09-16T05:29:25.590616+0000","last_fullsized":"2024-09-16T05:29:25.590616+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:24.891609+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:28:12.248317+0000","last_clean_scrub_stamp":"2024-09-16T05:29:24.891609+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:46.196747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1e","version":"35'139","reported_seq":1955,"reported_epoch":1080,"state":"active+clean","last_fresh":"2024-09-16T05:29:26.415473+0000","last_change":"2024-09-16T05:28:21.156066+0000","last_active":"2024-09-16T05:29:26.415473+0000","last_peered":"2024-09-16T05:29:26.415473+0000","last_clean":"2024-09-16T05:29:26.415473+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:29:26.415473+0000","last_undegraded":"2024-09-16T05:29:26.415473+0000","last_fullsized":"2024-09-16T05:29:26.415473+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:21.155973+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:28:21.155973+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:30.066462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"35'139","reported_seq":1006,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.516797+0000","last_change":"2024-09-16T05:28:36.722915+0000","last_active":"2024-09-16T05:29:20.516797+0000","last_peered":"2024-09-16T05:29:20.516797+0000","last_clean":"2024-09-16T05:29:20.516797+0000","last_became_active":"2024-09-16T05:26:22.907190+0000","last_became_peered":"2024-09-16T05:26:22.907190+0000","last_unstale":"2024-09-16T05:29:20.516797+0000","last_undegraded":"2024-09-16T05:29:20.516797+0000","last_fullsized":"2024-09-16T05:29:20.516797+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:36.722758+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:36.722758+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:57.798323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"35'139","reported_seq":1206,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-09-16T05:29:20.147803+0000","last_change":"2024-09-16T05:28:54.951923+0000","last_active":"2024-09-16T05:29:20.147803+0000","last_peered":"2024-09-16T05:29:20.147803+0000","last_clean":"2024-09-16T05:29:20.147803+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:29:20.147803+0000","last_undegraded":"2024-09-16T05:29:20.147803+0000","last_fullsized":"2024-09-16T05:29:20.147803+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:54.951829+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:54.951829+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:02.103935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089590,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57904,"kb_used_data":8664,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687248,"statfs":{"total":95995035648,"available":95935741952,"internally_reserved":0,"allocated":8871936,"data_stored":6094021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285818,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57888,"kb_used_data":8652,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687264,"statfs":{"total":95995035648,"available":95935758336,"internally_reserved":0,"allocated":8859648,"data_stored":6086378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514756,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80096,"kb_used_data":8652,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665056,"statfs":{"total":95995035648,"available":95913017344,"internally_reserved":0,"allocated":8859648,"data_stored":6086378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841796,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57876,"kb_used_data":8640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687276,"statfs":{"total":95995035648,"available":95935770624,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1007,"seq":4325032067104,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36848,"kb_used_data":8664,"kb_used_omap":38,"kb_used_meta":28121,"kb_avail":93708304,"statfs":{"total":95995035648,"available":95957303296,"internally_reserved":0,"allocated":8871936,"data_stored":6094021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38947,"internal_metadata":28796893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397776,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62060,"kb_used_data":8664,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683092,"statfs":{"total":95995035648,"available":95931486208,"internally_reserved":0,"allocated":8871936,"data_stored":6094021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757523,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66468,"kb_used_data":8912,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93678684,"statfs":{"total":95995035648,"available":95926972416,"internally_reserved":0,"allocated":9125888,"data_stored":6349066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921048,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53728,"kb_used_data":8652,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691424,"statfs":{"total":95995035648,"available":95940018176,"internally_reserved":0,"allocated":8859648,"data_stored":6086378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117277,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62384,"kb_used_data":8984,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93682768,"statfs":{"total":95995035648,"available":95931154432,"internally_reserved":0,"allocated":9199616,"data_stored":6421701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240564,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45360,"kb_used_data":8924,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93699792,"statfs":{"total":95995035648,"available":95948587008,"internally_reserved":0,"allocated":9138176,"data_stored":6356709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":988,"seq":4243427688487,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45268,"kb_used_data":8640,"kb_used_omap":22,"kb_used_meta":36585,"kb_avail":93699884,"statfs":{"total":95995035648,"available":95948681216,"internally_reserved":0,"allocated":8847360,"data_stored":6077256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22663,"internal_metadata":37463929},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54048,"kb_used_data":8972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691104,"statfs":{"total":95995035648,"available":95939690496,"internally_reserved":0,"allocated":9187328,"data_stored":6414058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:29:31.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:31 smithi196 ceph-mon[30802]: 1.6 scrub starts 2024-09-16T05:29:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:31 smithi196 ceph-mon[30802]: 1.6 scrub ok 2024-09-16T05:29:31.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:31 smithi196 ceph-mon[30802]: from='client.18510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:31 smithi032 ceph-mon[27388]: 1.6 scrub starts 2024-09-16T05:29:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:31 smithi032 ceph-mon[27388]: 1.6 scrub ok 2024-09-16T05:29:31.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:31 smithi032 ceph-mon[27388]: from='client.18510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:31.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:31 smithi204 ceph-mon[30933]: 1.6 scrub starts 2024-09-16T05:29:31.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:31 smithi204 ceph-mon[30933]: 1.6 scrub ok 2024-09-16T05:29:31.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:31 smithi204 ceph-mon[30933]: from='client.18510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:32.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:32 smithi196 ceph-mon[30802]: pgmap v1853: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:32.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:32 smithi032 ceph-mon[27388]: pgmap v1853: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:32.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:32 smithi204 ceph-mon[30933]: pgmap v1853: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:33.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:33 smithi196 ceph-mon[30802]: 1.1e scrub starts 2024-09-16T05:29:33.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:33 smithi196 ceph-mon[30802]: 1.1e scrub ok 2024-09-16T05:29:33.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:33 smithi032 ceph-mon[27388]: 1.1e scrub starts 2024-09-16T05:29:33.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:33 smithi032 ceph-mon[27388]: 1.1e scrub ok 2024-09-16T05:29:33.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:33 smithi204 ceph-mon[30933]: 1.1e scrub starts 2024-09-16T05:29:33.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:33 smithi204 ceph-mon[30933]: 1.1e scrub ok 2024-09-16T05:29:33.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T05:29:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:34.371 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:34 smithi032 ceph-mon[27388]: pgmap v1854: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:34.371 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:34 smithi032 ceph-mon[27388]: 1.1a deep-scrub starts 2024-09-16T05:29:34.371 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:34 smithi032 ceph-mon[27388]: 1.1a deep-scrub ok 2024-09-16T05:29:34.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:34 smithi196 ceph-mon[30802]: pgmap v1854: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:34 smithi196 ceph-mon[30802]: 1.1a deep-scrub starts 2024-09-16T05:29:34.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:34 smithi196 ceph-mon[30802]: 1.1a deep-scrub ok 2024-09-16T05:29:34.639 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-09-16T05:29:34.639 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-09-16T05:29:34.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:34 smithi204 ceph-mon[30933]: pgmap v1854: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:34 smithi204 ceph-mon[30933]: 1.1a deep-scrub starts 2024-09-16T05:29:34.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:34 smithi204 ceph-mon[30933]: 1.1a deep-scrub ok 2024-09-16T05:29:35.400 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":1855,"stamp":"2024-09-16T05:29:34.320989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":680160,"kb_used_data":105260,"kb_used_omap":84,"kb_used_meta":574571,"kb_avail":1124261664,"statfs":{"total":1151940427776,"available":1151243943936,"internally_reserved":0,"allocated":107786240,"data_stored":74394073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86930,"internal_metadata":588360814},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.139550"},"pg_stats":[{"pgid":"1.12","version":"35'139","reported_seq":781,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492097+0000","last_change":"2024-09-16T05:28:51.890924+0000","last_active":"2024-09-16T05:29:29.492097+0000","last_peered":"2024-09-16T05:29:29.492097+0000","last_clean":"2024-09-16T05:29:29.492097+0000","last_became_active":"2024-09-16T05:28:21.191459+0000","last_became_peered":"2024-09-16T05:28:21.191459+0000","last_unstale":"2024-09-16T05:29:29.492097+0000","last_undegraded":"2024-09-16T05:29:29.492097+0000","last_fullsized":"2024-09-16T05:29:29.492097+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:51.890834+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:50.372923+0000","last_clean_scrub_stamp":"2024-09-16T05:28:51.890834+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:19.285120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"35'139","reported_seq":1516,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492112+0000","last_change":"2024-09-16T05:28:29.769978+0000","last_active":"2024-09-16T05:29:29.492112+0000","last_peered":"2024-09-16T05:29:29.492112+0000","last_clean":"2024-09-16T05:29:29.492112+0000","last_became_active":"2024-09-16T05:26:22.907168+0000","last_became_peered":"2024-09-16T05:26:22.907168+0000","last_unstale":"2024-09-16T05:29:29.492112+0000","last_undegraded":"2024-09-16T05:29:29.492112+0000","last_fullsized":"2024-09-16T05:29:29.492112+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:29.769890+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:29.769890+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:46.455950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"35'139","reported_seq":1804,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.199716+0000","last_change":"2024-09-16T05:28:48.441113+0000","last_active":"2024-09-16T05:29:29.199716+0000","last_peered":"2024-09-16T05:29:29.199716+0000","last_clean":"2024-09-16T05:29:29.199716+0000","last_became_active":"2024-09-16T05:20:35.799747+0000","last_became_peered":"2024-09-16T05:20:35.799747+0000","last_unstale":"2024-09-16T05:29:29.199716+0000","last_undegraded":"2024-09-16T05:29:29.199716+0000","last_fullsized":"2024-09-16T05:29:29.199716+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:48.441014+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:48.441014+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:18.091526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"35'139","reported_seq":1858,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492293+0000","last_change":"2024-09-16T05:29:02.856802+0000","last_active":"2024-09-16T05:29:29.492293+0000","last_peered":"2024-09-16T05:29:29.492293+0000","last_clean":"2024-09-16T05:29:29.492293+0000","last_became_active":"2024-09-16T05:26:58.976632+0000","last_became_peered":"2024-09-16T05:26:58.976632+0000","last_unstale":"2024-09-16T05:29:29.492293+0000","last_undegraded":"2024-09-16T05:29:29.492293+0000","last_fullsized":"2024-09-16T05:29:29.492293+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:02.856695+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:33.791043+0000","last_clean_scrub_stamp":"2024-09-16T05:29:02.856695+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:31.504867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"35'139","reported_seq":1854,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.200267+0000","last_change":"2024-09-16T05:28:51.322038+0000","last_active":"2024-09-16T05:29:29.200267+0000","last_peered":"2024-09-16T05:29:29.200267+0000","last_clean":"2024-09-16T05:29:29.200267+0000","last_became_active":"2024-09-16T05:28:16.165227+0000","last_became_peered":"2024-09-16T05:28:16.165227+0000","last_unstale":"2024-09-16T05:29:29.200267+0000","last_undegraded":"2024-09-16T05:29:29.200267+0000","last_fullsized":"2024-09-16T05:29:29.200267+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:51.321905+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:11.119111+0000","last_clean_scrub_stamp":"2024-09-16T05:28:51.321905+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:16.800529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"35'139","reported_seq":1328,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492643+0000","last_change":"2024-09-16T05:29:12.087578+0000","last_active":"2024-09-16T05:29:29.492643+0000","last_peered":"2024-09-16T05:29:29.492643+0000","last_clean":"2024-09-16T05:29:29.492643+0000","last_became_active":"2024-09-16T05:28:24.204528+0000","last_became_peered":"2024-09-16T05:28:24.204528+0000","last_unstale":"2024-09-16T05:29:29.492643+0000","last_undegraded":"2024-09-16T05:29:29.492643+0000","last_fullsized":"2024-09-16T05:29:29.492643+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:12.087489+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:30.986186+0000","last_clean_scrub_stamp":"2024-09-16T05:29:12.087489+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:38.977304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"35'139","reported_seq":1501,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492111+0000","last_change":"2024-09-16T05:29:14.497553+0000","last_active":"2024-09-16T05:29:29.492111+0000","last_peered":"2024-09-16T05:29:29.492111+0000","last_clean":"2024-09-16T05:29:29.492111+0000","last_became_active":"2024-09-16T05:26:25.117472+0000","last_became_peered":"2024-09-16T05:26:25.117472+0000","last_unstale":"2024-09-16T05:29:29.492111+0000","last_undegraded":"2024-09-16T05:29:29.492111+0000","last_fullsized":"2024-09-16T05:29:29.492111+0000","mapping_epoch":991,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:14.497468+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:07.872208+0000","last_clean_scrub_stamp":"2024-09-16T05:29:14.497468+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:44.280746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"35'139","reported_seq":880,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492511+0000","last_change":"2024-09-16T05:28:30.036111+0000","last_active":"2024-09-16T05:29:29.492511+0000","last_peered":"2024-09-16T05:29:29.492511+0000","last_clean":"2024-09-16T05:29:29.492511+0000","last_became_active":"2024-09-16T05:28:24.205007+0000","last_became_peered":"2024-09-16T05:28:24.205007+0000","last_unstale":"2024-09-16T05:29:29.492511+0000","last_undegraded":"2024-09-16T05:29:29.492511+0000","last_fullsized":"2024-09-16T05:29:29.492511+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:30.036037+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:25:54.082196+0000","last_clean_scrub_stamp":"2024-09-16T05:28:30.036037+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:42.097121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"35'139","reported_seq":1733,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492503+0000","last_change":"2024-09-16T05:29:26.257268+0000","last_active":"2024-09-16T05:29:29.492503+0000","last_peered":"2024-09-16T05:29:29.492503+0000","last_clean":"2024-09-16T05:29:29.492503+0000","last_became_active":"2024-09-16T05:29:26.256980+0000","last_became_peered":"2024-09-16T05:29:26.256980+0000","last_unstale":"2024-09-16T05:29:29.492503+0000","last_undegraded":"2024-09-16T05:29:29.492503+0000","last_fullsized":"2024-09-16T05:29:29.492503+0000","mapping_epoch":1079,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1080,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:23.046301+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:22:38.207426+0000","last_clean_scrub_stamp":"2024-09-16T05:29:23.046301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:49.960889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"35'139","reported_seq":833,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.200266+0000","last_change":"2024-09-16T05:28:17.407013+0000","last_active":"2024-09-16T05:29:29.200266+0000","last_peered":"2024-09-16T05:29:29.200266+0000","last_clean":"2024-09-16T05:29:29.200266+0000","last_became_active":"2024-09-16T05:28:16.164987+0000","last_became_peered":"2024-09-16T05:28:16.164987+0000","last_unstale":"2024-09-16T05:29:29.200266+0000","last_undegraded":"2024-09-16T05:29:29.200266+0000","last_fullsized":"2024-09-16T05:29:29.200266+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:17.406923+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:50.546823+0000","last_clean_scrub_stamp":"2024-09-16T05:28:17.406923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"35'139","reported_seq":2482,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.199734+0000","last_change":"2024-09-16T05:29:03.445323+0000","last_active":"2024-09-16T05:29:29.199734+0000","last_peered":"2024-09-16T05:29:29.199734+0000","last_clean":"2024-09-16T05:29:29.199734+0000","last_became_active":"2024-09-16T05:21:54.929163+0000","last_became_peered":"2024-09-16T05:21:54.929163+0000","last_unstale":"2024-09-16T05:29:29.199734+0000","last_undegraded":"2024-09-16T05:29:29.199734+0000","last_fullsized":"2024-09-16T05:29:29.199734+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:03.445237+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:42.143106+0000","last_clean_scrub_stamp":"2024-09-16T05:29:03.445237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:20.619991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"35'139","reported_seq":2437,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492831+0000","last_change":"2024-09-16T05:28:35.161241+0000","last_active":"2024-09-16T05:29:29.492831+0000","last_peered":"2024-09-16T05:29:29.492831+0000","last_clean":"2024-09-16T05:29:29.492831+0000","last_became_active":"2024-09-16T05:26:58.978042+0000","last_became_peered":"2024-09-16T05:26:58.978042+0000","last_unstale":"2024-09-16T05:29:29.492831+0000","last_undegraded":"2024-09-16T05:29:29.492831+0000","last_fullsized":"2024-09-16T05:29:29.492831+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:35.161125+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:35.161125+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:05.105469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"35'139","reported_seq":1749,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492518+0000","last_change":"2024-09-16T05:28:58.118422+0000","last_active":"2024-09-16T05:29:29.492518+0000","last_peered":"2024-09-16T05:29:29.492518+0000","last_clean":"2024-09-16T05:29:29.492518+0000","last_became_active":"2024-09-16T05:20:35.810896+0000","last_became_peered":"2024-09-16T05:20:35.810896+0000","last_unstale":"2024-09-16T05:29:29.492518+0000","last_undegraded":"2024-09-16T05:29:29.492518+0000","last_fullsized":"2024-09-16T05:29:29.492518+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:58.118326+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:23:37.275354+0000","last_clean_scrub_stamp":"2024-09-16T05:28:58.118326+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.058327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"35'139","reported_seq":1693,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.839406+0000","last_change":"2024-09-16T05:29:29.839429+0000","last_active":"2024-09-16T05:29:29.839406+0000","last_peered":"2024-09-16T05:29:29.839406+0000","last_clean":"2024-09-16T05:29:29.839406+0000","last_became_active":"2024-09-16T05:20:35.812021+0000","last_became_peered":"2024-09-16T05:20:35.812021+0000","last_unstale":"2024-09-16T05:29:29.839406+0000","last_undegraded":"2024-09-16T05:29:29.839406+0000","last_fullsized":"2024-09-16T05:29:29.839406+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:29.839340+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:29:29.839340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:39.983667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"35'139","reported_seq":1117,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492574+0000","last_change":"2024-09-16T05:29:17.128120+0000","last_active":"2024-09-16T05:29:29.492574+0000","last_peered":"2024-09-16T05:29:29.492574+0000","last_clean":"2024-09-16T05:29:29.492574+0000","last_became_active":"2024-09-16T05:28:24.204654+0000","last_became_peered":"2024-09-16T05:28:24.204654+0000","last_unstale":"2024-09-16T05:29:29.492574+0000","last_undegraded":"2024-09-16T05:29:29.492574+0000","last_fullsized":"2024-09-16T05:29:29.492574+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":3,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:17.128029+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:29:17.128029+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:22.966252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"35'139","reported_seq":610,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492156+0000","last_change":"2024-09-16T05:29:27.888626+0000","last_active":"2024-09-16T05:29:29.492156+0000","last_peered":"2024-09-16T05:29:29.492156+0000","last_clean":"2024-09-16T05:29:29.492156+0000","last_became_active":"2024-09-16T05:28:20.181056+0000","last_became_peered":"2024-09-16T05:28:20.181056+0000","last_unstale":"2024-09-16T05:29:29.492156+0000","last_undegraded":"2024-09-16T05:29:29.492156+0000","last_fullsized":"2024-09-16T05:29:29.492156+0000","mapping_epoch":1054,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1055,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:27.888503+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:26:11.129971+0000","last_clean_scrub_stamp":"2024-09-16T05:29:27.888503+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:41.132758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"35'139","reported_seq":2338,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492263+0000","last_change":"2024-09-16T05:28:42.092940+0000","last_active":"2024-09-16T05:29:29.492263+0000","last_peered":"2024-09-16T05:29:29.492263+0000","last_clean":"2024-09-16T05:29:29.492263+0000","last_became_active":"2024-09-16T05:26:59.168051+0000","last_became_peered":"2024-09-16T05:26:59.168051+0000","last_unstale":"2024-09-16T05:29:29.492263+0000","last_undegraded":"2024-09-16T05:29:29.492263+0000","last_fullsized":"2024-09-16T05:29:29.492263+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:42.092847+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:27:30.832744+0000","last_clean_scrub_stamp":"2024-09-16T05:28:42.092847+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.609055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"35'139","reported_seq":2482,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.202840+0000","last_change":"2024-09-16T05:28:55.235326+0000","last_active":"2024-09-16T05:29:29.202840+0000","last_peered":"2024-09-16T05:29:29.202840+0000","last_clean":"2024-09-16T05:29:29.202840+0000","last_became_active":"2024-09-16T05:21:54.942967+0000","last_became_peered":"2024-09-16T05:21:54.942967+0000","last_unstale":"2024-09-16T05:29:29.202840+0000","last_undegraded":"2024-09-16T05:29:29.202840+0000","last_fullsized":"2024-09-16T05:29:29.202840+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":4,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:55.235220+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:28:55.235220+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:23.996312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"35'139","reported_seq":1959,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492282+0000","last_change":"2024-09-16T05:28:34.191267+0000","last_active":"2024-09-16T05:29:29.492282+0000","last_peered":"2024-09-16T05:29:29.492282+0000","last_clean":"2024-09-16T05:29:29.492282+0000","last_became_active":"2024-09-16T05:15:56.714931+0000","last_became_peered":"2024-09-16T05:15:56.714931+0000","last_unstale":"2024-09-16T05:29:29.492282+0000","last_undegraded":"2024-09-16T05:29:29.492282+0000","last_fullsized":"2024-09-16T05:29:29.492282+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:34.191184+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:18:26.950417+0000","last_clean_scrub_stamp":"2024-09-16T05:28:34.191184+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:56.005005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"35'139","reported_seq":1407,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492091+0000","last_change":"2024-09-16T05:28:21.192606+0000","last_active":"2024-09-16T05:29:29.492091+0000","last_peered":"2024-09-16T05:29:29.492091+0000","last_clean":"2024-09-16T05:29:29.492091+0000","last_became_active":"2024-09-16T05:28:21.192346+0000","last_became_peered":"2024-09-16T05:28:21.192346+0000","last_unstale":"2024-09-16T05:29:29.492091+0000","last_undegraded":"2024-09-16T05:29:29.492091+0000","last_fullsized":"2024-09-16T05:29:29.492091+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:05.149072+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:05.149072+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:31.074543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"35'139","reported_seq":1357,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492282+0000","last_change":"2024-09-16T05:28:45.005820+0000","last_active":"2024-09-16T05:29:29.492282+0000","last_peered":"2024-09-16T05:29:29.492282+0000","last_clean":"2024-09-16T05:29:29.492282+0000","last_became_active":"2024-09-16T05:20:40.808938+0000","last_became_peered":"2024-09-16T05:20:40.808938+0000","last_unstale":"2024-09-16T05:29:29.492282+0000","last_undegraded":"2024-09-16T05:29:29.492282+0000","last_fullsized":"2024-09-16T05:29:29.492282+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:45.005717+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:29.881423+0000","last_clean_scrub_stamp":"2024-09-16T05:28:45.005717+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:06.847310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"35'139","reported_seq":1401,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.663577+0000","last_change":"2024-09-16T05:28:03.168727+0000","last_active":"2024-09-16T05:29:29.663577+0000","last_peered":"2024-09-16T05:29:29.663577+0000","last_clean":"2024-09-16T05:29:29.663577+0000","last_became_active":"2024-09-16T05:20:40.800841+0000","last_became_peered":"2024-09-16T05:20:40.800841+0000","last_unstale":"2024-09-16T05:29:29.663577+0000","last_undegraded":"2024-09-16T05:29:29.663577+0000","last_fullsized":"2024-09-16T05:29:29.663577+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:03.168644+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.564525+0000","last_clean_scrub_stamp":"2024-09-16T05:28:03.168644+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:31.721465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"35'139","reported_seq":798,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492098+0000","last_change":"2024-09-16T05:29:20.856560+0000","last_active":"2024-09-16T05:29:29.492098+0000","last_peered":"2024-09-16T05:29:29.492098+0000","last_clean":"2024-09-16T05:29:29.492098+0000","last_became_active":"2024-09-16T05:28:20.181291+0000","last_became_peered":"2024-09-16T05:28:20.181291+0000","last_unstale":"2024-09-16T05:29:29.492098+0000","last_undegraded":"2024-09-16T05:29:29.492098+0000","last_fullsized":"2024-09-16T05:29:29.492098+0000","mapping_epoch":1054,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1055,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:20.856433+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:19:13.533798+0000","last_clean_scrub_stamp":"2024-09-16T05:29:20.856433+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:23.619644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"35'139","reported_seq":1791,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.663589+0000","last_change":"2024-09-16T05:29:01.283505+0000","last_active":"2024-09-16T05:29:29.663589+0000","last_peered":"2024-09-16T05:29:29.663589+0000","last_clean":"2024-09-16T05:29:29.663589+0000","last_became_active":"2024-09-16T05:16:17.723435+0000","last_became_peered":"2024-09-16T05:16:17.723435+0000","last_unstale":"2024-09-16T05:29:29.663589+0000","last_undegraded":"2024-09-16T05:29:29.663589+0000","last_fullsized":"2024-09-16T05:29:29.663589+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":401,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:01.283431+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:29:01.283431+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:04.945286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"35'139","reported_seq":1882,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492245+0000","last_change":"2024-09-16T05:29:26.893482+0000","last_active":"2024-09-16T05:29:29.492245+0000","last_peered":"2024-09-16T05:29:29.492245+0000","last_clean":"2024-09-16T05:29:29.492245+0000","last_became_active":"2024-09-16T05:20:40.804892+0000","last_became_peered":"2024-09-16T05:20:40.804892+0000","last_unstale":"2024-09-16T05:29:29.492245+0000","last_undegraded":"2024-09-16T05:29:29.492245+0000","last_fullsized":"2024-09-16T05:29:29.492245+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:26.893396+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:20:41.173808+0000","last_clean_scrub_stamp":"2024-09-16T05:29:26.893396+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:27.408482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"35'139","reported_seq":1130,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.200264+0000","last_change":"2024-09-16T05:28:16.165544+0000","last_active":"2024-09-16T05:29:29.200264+0000","last_peered":"2024-09-16T05:29:29.200264+0000","last_clean":"2024-09-16T05:29:29.200264+0000","last_became_active":"2024-09-16T05:28:16.165219+0000","last_became_peered":"2024-09-16T05:28:16.165219+0000","last_unstale":"2024-09-16T05:29:29.200264+0000","last_undegraded":"2024-09-16T05:29:29.200264+0000","last_fullsized":"2024-09-16T05:29:29.200264+0000","mapping_epoch":1050,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1051,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:11.963926+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:17:47.986643+0000","last_clean_scrub_stamp":"2024-09-16T05:28:11.963926+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"35'139","reported_seq":2473,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492637+0000","last_change":"2024-09-16T05:29:27.924998+0000","last_active":"2024-09-16T05:29:29.492637+0000","last_peered":"2024-09-16T05:29:29.492637+0000","last_clean":"2024-09-16T05:29:29.492637+0000","last_became_active":"2024-09-16T05:20:40.804655+0000","last_became_peered":"2024-09-16T05:20:40.804655+0000","last_unstale":"2024-09-16T05:29:29.492637+0000","last_undegraded":"2024-09-16T05:29:29.492637+0000","last_fullsized":"2024-09-16T05:29:29.492637+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:27.924901+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.579637+0000","last_clean_scrub_stamp":"2024-09-16T05:29:27.924901+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:32.450448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"35'139","reported_seq":613,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492270+0000","last_change":"2024-09-16T05:29:24.891712+0000","last_active":"2024-09-16T05:29:29.492270+0000","last_peered":"2024-09-16T05:29:29.492270+0000","last_clean":"2024-09-16T05:29:29.492270+0000","last_became_active":"2024-09-16T05:28:21.192702+0000","last_became_peered":"2024-09-16T05:28:21.192702+0000","last_unstale":"2024-09-16T05:29:29.492270+0000","last_undegraded":"2024-09-16T05:29:29.492270+0000","last_fullsized":"2024-09-16T05:29:29.492270+0000","mapping_epoch":1055,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":1056,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:24.891609+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:28:12.248317+0000","last_clean_scrub_stamp":"2024-09-16T05:29:24.891609+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:46.196747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1e","version":"35'139","reported_seq":1965,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:31.784119+0000","last_change":"2024-09-16T05:29:31.784147+0000","last_active":"2024-09-16T05:29:31.784119+0000","last_peered":"2024-09-16T05:29:31.784119+0000","last_clean":"2024-09-16T05:29:31.784119+0000","last_became_active":"2024-09-16T05:20:40.804416+0000","last_became_peered":"2024-09-16T05:20:40.804416+0000","last_unstale":"2024-09-16T05:29:31.784119+0000","last_undegraded":"2024-09-16T05:29:31.784119+0000","last_fullsized":"2024-09-16T05:29:31.784119+0000","mapping_epoch":690,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:29:31.784051+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:24:00.648056+0000","last_clean_scrub_stamp":"2024-09-16T05:29:31.784051+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:40.211519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"35'139","reported_seq":1012,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492111+0000","last_change":"2024-09-16T05:28:36.722915+0000","last_active":"2024-09-16T05:29:29.492111+0000","last_peered":"2024-09-16T05:29:29.492111+0000","last_clean":"2024-09-16T05:29:29.492111+0000","last_became_active":"2024-09-16T05:26:22.907190+0000","last_became_peered":"2024-09-16T05:26:22.907190+0000","last_unstale":"2024-09-16T05:29:29.492111+0000","last_undegraded":"2024-09-16T05:29:29.492111+0000","last_fullsized":"2024-09-16T05:29:29.492111+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:36.722758+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:36.722758+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:29:57.798323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"35'139","reported_seq":1212,"reported_epoch":1081,"state":"active+clean","last_fresh":"2024-09-16T05:29:29.492246+0000","last_change":"2024-09-16T05:28:54.951923+0000","last_active":"2024-09-16T05:29:29.492246+0000","last_peered":"2024-09-16T05:29:29.492246+0000","last_clean":"2024-09-16T05:29:29.492246+0000","last_became_active":"2024-09-16T05:21:54.929888+0000","last_became_peered":"2024-09-16T05:21:54.929888+0000","last_unstale":"2024-09-16T05:29:29.492246+0000","last_undegraded":"2024-09-16T05:29:29.492246+0000","last_fullsized":"2024-09-16T05:29:29.492246+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":685,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":5,"last_scrub":"35'139","last_scrub_stamp":"2024-09-16T05:28:54.951829+0000","last_deep_scrub":"35'139","last_deep_scrub_stamp":"2024-09-16T05:07:53.915693+0000","last_clean_scrub_stamp":"2024-09-16T05:28:54.951829+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:30:02.103935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":89,"seq":382252089591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57912,"kb_used_data":8676,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687240,"statfs":{"total":95995035648,"available":95935733760,"internally_reserved":0,"allocated":8884224,"data_stored":6101756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":50395588},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":83,"seq":356482285819,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57912,"kb_used_data":8676,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687240,"statfs":{"total":95995035648,"available":95935733760,"internally_reserved":0,"allocated":8884224,"data_stored":6101756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514757,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80120,"kb_used_data":8676,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665032,"statfs":{"total":95995035648,"available":95912992768,"internally_reserved":0,"allocated":8884224,"data_stored":6101756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":66,"seq":283467841797,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57904,"kb_used_data":8664,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93687248,"statfs":{"total":95995035648,"available":95935741952,"internally_reserved":0,"allocated":8871936,"data_stored":6094021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1007,"seq":4325032067105,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36856,"kb_used_data":8676,"kb_used_omap":38,"kb_used_meta":28121,"kb_avail":93708296,"statfs":{"total":95995035648,"available":95957295104,"internally_reserved":0,"allocated":8884224,"data_stored":6101756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38947,"internal_metadata":28796893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397777,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62072,"kb_used_data":8676,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93683080,"statfs":{"total":95995035648,"available":95931473920,"internally_reserved":0,"allocated":8884224,"data_stored":6101756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757524,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66492,"kb_used_data":8936,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93678660,"statfs":{"total":95995035648,"available":95926947840,"internally_reserved":0,"allocated":9150464,"data_stored":6364444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921049,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53752,"kb_used_data":8676,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691400,"statfs":{"total":95995035648,"available":95939993600,"internally_reserved":0,"allocated":8884224,"data_stored":6101756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117278,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62392,"kb_used_data":8996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93682760,"statfs":{"total":95995035648,"available":95931146240,"internally_reserved":0,"allocated":9211904,"data_stored":6429436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":329,"seq":1413044240565,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45372,"kb_used_data":8936,"kb_used_omap":10,"kb_used_meta":36405,"kb_avail":93699780,"statfs":{"total":95995035648,"available":95948574720,"internally_reserved":0,"allocated":9150464,"data_stored":6364444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10957,"internal_metadata":37279027},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":988,"seq":4243427688488,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45304,"kb_used_data":8676,"kb_used_omap":22,"kb_used_meta":36585,"kb_avail":93699848,"statfs":{"total":95995035648,"available":95948644352,"internally_reserved":0,"allocated":8884224,"data_stored":6101756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22663,"internal_metadata":37463929},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54072,"kb_used_data":8996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93691080,"statfs":{"total":95995035648,"available":95939665920,"internally_reserved":0,"allocated":9211904,"data_stored":6429436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1175,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:29:35.402 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-16T05:29:35.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T05:29:35.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:35 smithi196 ceph-mon[30802]: 1.16 scrub starts 2024-09-16T05:29:35.661 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T05:29:35.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:35 smithi032 ceph-mon[27388]: 1.16 scrub starts 2024-09-16T05:29:35.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:35 smithi204 ceph-mon[30933]: 1.16 scrub starts 2024-09-16T05:29:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:36 smithi032 ceph-mon[27388]: pgmap v1855: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:36 smithi032 ceph-mon[27388]: from='client.18516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:36 smithi032 ceph-mon[27388]: 1.16 scrub ok 2024-09-16T05:29:36.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:36 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:29:36.421 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:36 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:29:36.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:36 smithi196 ceph-mon[30802]: pgmap v1855: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:36 smithi196 ceph-mon[30802]: from='client.18516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:36 smithi196 ceph-mon[30802]: 1.16 scrub ok 2024-09-16T05:29:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:36 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:29:36.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:36 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:29:36.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:36 smithi204 ceph-mon[30933]: pgmap v1855: 31 pgs: 31 active+clean; 577 KiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T05:29:36.670 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:36 smithi204 ceph-mon[30933]: from='client.18516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T05:29:36.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:36 smithi204 ceph-mon[30933]: 1.16 scrub ok 2024-09-16T05:29:36.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:36 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:29:36.671 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:36 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:29:36.928 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-16T05:29:36.937 INFO:tasks.cephadm:Teardown begin 2024-09-16T05:29:36.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:29:36.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:29:36.993 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:29:37.022 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-16T05:29:37.022 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T05:29:37.038 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T05:29:37.054 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T05:29:37.078 INFO:tasks.cephadm:Stopping all daemons... 2024-09-16T05:29:37.078 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-16T05:29:37.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.a 2024-09-16T05:29:37.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 systemd[1]: Stopping Ceph mon.a for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:29:37.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:37 smithi196 ceph-mon[30802]: osdmap e1082: 12 total, 12 up, 12 in 2024-09-16T05:29:37.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1982279063' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-16T05:29:37.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:37 smithi196 ceph-mon[30802]: from='client.? 172.21.15.32:0/1982279063' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-16T05:29:37.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:37 smithi196 ceph-mon[30802]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:29:37.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:37 smithi196 ceph-mon[30802]: purged_snaps scrub starts 2024-09-16T05:29:37.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:37 smithi196 ceph-mon[30802]: purged_snaps scrub ok 2024-09-16T05:29:37.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 ceph-mon[27388]: osdmap e1082: 12 total, 12 up, 12 in 2024-09-16T05:29:37.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1982279063' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-16T05:29:37.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 ceph-mon[27388]: from='client.? 172.21.15.32:0/1982279063' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-16T05:29:37.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 ceph-mon[27388]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:29:37.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 ceph-mon[27388]: purged_snaps scrub starts 2024-09-16T05:29:37.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 ceph-mon[27388]: purged_snaps scrub ok 2024-09-16T05:29:37.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:29:37.385+0000 7f7c4519d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:29:37.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a[27384]: 2024-09-16T05:29:37.385+0000 7f7c4519d640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-16T05:29:37.671 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 podman[215036]: 2024-09-16 05:29:37.457690129 +0000 UTC m=+0.187068309 container died d9caed178edf300992f3eec70594794e53439b7571d4cc9a28ad07a27b8c35a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:29:37.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:37 smithi204 ceph-mon[30933]: osdmap e1082: 12 total, 12 up, 12 in 2024-09-16T05:29:37.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1982279063' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-16T05:29:37.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:37 smithi204 ceph-mon[30933]: from='client.? 172.21.15.32:0/1982279063' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-16T05:29:37.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:37 smithi204 ceph-mon[30933]: from='mgr.14149 172.21.15.32:0/2114299874' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T05:29:37.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:37 smithi204 ceph-mon[30933]: purged_snaps scrub starts 2024-09-16T05:29:37.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:37 smithi204 ceph-mon[30933]: purged_snaps scrub ok 2024-09-16T05:29:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 podman[215036]: 2024-09-16 05:29:37.961241282 +0000 UTC m=+0.690619460 container cleanup d9caed178edf300992f3eec70594794e53439b7571d4cc9a28ad07a27b8c35a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e) 2024-09-16T05:29:38.170 INFO:journalctl@ceph.mon.a.smithi032.stdout:Sep 16 05:29:37 smithi032 bash[215036]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-a 2024-09-16T05:29:38.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.a.service' 2024-09-16T05:29:38.435 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:29:38.435 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-16T05:29:38.435 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-16T05:29:38.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.b 2024-09-16T05:29:38.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:38 smithi196 systemd[1]: Stopping Ceph mon.b for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:29:39.116 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:38 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-b[30798]: 2024-09-16T05:29:38.756+0000 7f70bb09f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:29:39.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:38 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-b[30798]: 2024-09-16T05:29:38.756+0000 7f70bb09f640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-16T05:29:39.117 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:38 smithi196 podman[124742]: 2024-09-16 05:29:38.797647412 +0000 UTC m=+0.167604668 container died cc968130f8a228108a3d6b5dbbbd35e3b08741fa386b639cbfe4f34f36a8cd1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-16T05:29:39.487 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:39 smithi196 podman[124742]: 2024-09-16 05:29:39.290430193 +0000 UTC m=+0.660387491 container cleanup cc968130f8a228108a3d6b5dbbbd35e3b08741fa386b639cbfe4f34f36a8cd1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:29:39.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Sep 16 05:29:39 smithi196 bash[124742]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-b 2024-09-16T05:29:39.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.b.service' 2024-09-16T05:29:39.700 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:29:39.700 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-16T05:29:39.701 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-16T05:29:39.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.c 2024-09-16T05:29:40.021 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:39 smithi204 systemd[1]: Stopping Ceph mon.c for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:29:40.283 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-c[30915]: 2024-09-16T05:29:40.019+0000 7f42a72ed640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:29:40.283 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-c[30915]: 2024-09-16T05:29:40.019+0000 7f42a72ed640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-16T05:29:40.283 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:40 smithi204 podman[125091]: 2024-09-16 05:29:40.142349397 +0000 UTC m=+0.246966475 container died 66b7fb61ff0fa0e3a38eb8e65afeaf6a8848d009fba38cbc5cb17d88ce625111 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-16T05:29:40.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:40 smithi204 podman[125091]: 2024-09-16 05:29:40.612384961 +0000 UTC m=+0.717002040 container cleanup 66b7fb61ff0fa0e3a38eb8e65afeaf6a8848d009fba38cbc5cb17d88ce625111 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-c, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:29:40.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Sep 16 05:29:40 smithi204 bash[125091]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mon-c 2024-09-16T05:29:41.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mon.c.service' 2024-09-16T05:29:41.168 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:29:41.169 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-16T05:29:41.169 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-16T05:29:41.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.y 2024-09-16T05:29:41.467 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:29:41 smithi032 systemd[1]: Stopping Ceph mgr.y for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:29:41.776 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:29:41 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y[27614]: 2024-09-16T05:29:41.465+0000 7fe0f59c9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:29:41.776 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:29:41 smithi032 podman[215156]: 2024-09-16 05:29:41.497169637 +0000 UTC m=+0.130589476 container died 2ac258de4f2a37583f9abb2b352e44a61993c23b52422454cb103077a92edd35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:29:42.170 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:29:41 smithi032 podman[215156]: 2024-09-16 05:29:41.97469982 +0000 UTC m=+0.608119662 container cleanup 2ac258de4f2a37583f9abb2b352e44a61993c23b52422454cb103077a92edd35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:29:42.170 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Sep 16 05:29:41 smithi032 bash[215156]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-y 2024-09-16T05:29:42.478 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-16T05:29:42.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:29:42.517 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-16T05:29:42.517 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-16T05:29:42.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.x 2024-09-16T05:29:42.808 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:29:42 smithi196 systemd[1]: Stopping Ceph mgr.x for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:29:43.237 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:29:42 smithi196 podman[124864]: 2024-09-16 05:29:42.874297446 +0000 UTC m=+0.148352429 container died 935883dfbbc79b26b857ffc55c73e28c20c41edd93222bebe7d969bf25e59ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-16T05:29:43.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:29:43 smithi196 podman[124864]: 2024-09-16 05:29:43.36166976 +0000 UTC m=+0.635724753 container cleanup 935883dfbbc79b26b857ffc55c73e28c20c41edd93222bebe7d969bf25e59ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-16T05:29:43.613 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Sep 16 05:29:43 smithi196 bash[124864]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-mgr-x 2024-09-16T05:29:43.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-16T05:29:43.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:29:43.836 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-16T05:29:43.836 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-16T05:29:43.837 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.0 2024-09-16T05:29:44.155 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:43 smithi032 systemd[1]: Stopping Ceph osd.0 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:29:44.420 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:44 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0[40713]: 2024-09-16T05:29:44.153+0000 7f6038588640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:29:44.420 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:44 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0[40713]: 2024-09-16T05:29:44.153+0000 7f6038588640 -1 osd.0 1083 *** Got signal Terminated *** 2024-09-16T05:29:44.421 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:44 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0[40713]: 2024-09-16T05:29:44.153+0000 7f6038588640 -1 osd.0 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:29:49.851 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:49 smithi032 podman[215285]: 2024-09-16 05:29:49.547319562 +0000 UTC m=+5.501712493 container died f37d8c3b1d95e9ca7af36b287e3a79cf72c02ab570a29e4a43c5a918ba36286e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-16T05:29:50.170 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:50 smithi032 podman[215285]: 2024-09-16 05:29:50.012869448 +0000 UTC m=+5.967262382 container cleanup f37d8c3b1d95e9ca7af36b287e3a79cf72c02ab570a29e4a43c5a918ba36286e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:29:50.170 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:50 smithi032 bash[215285]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0 2024-09-16T05:29:50.644 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:50 smithi032 podman[215299]: 2024-09-16 05:29:50.272462659 +0000 UTC m=+0.720696302 container remove f37d8c3b1d95e9ca7af36b287e3a79cf72c02ab570a29e4a43c5a918ba36286e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:29:50.896 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:50 smithi032 podman[215386]: 2024-09-16 05:29:50.54724041 +0000 UTC m=+0.016618961 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:29:50.896 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:50 smithi032 podman[215386]: 2024-09-16 05:29:50.665120177 +0000 UTC m=+0.134498772 container create 90f2f2e3076c9b36c9a50b5615dc413a490dc8e1a1c07180327e849ab514afa4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:29:50.897 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:50 smithi032 podman[215386]: 2024-09-16 05:29:50.892411017 +0000 UTC m=+0.361789568 container init 90f2f2e3076c9b36c9a50b5615dc413a490dc8e1a1c07180327e849ab514afa4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:29:50.897 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:50 smithi032 podman[215386]: 2024-09-16 05:29:50.896024512 +0000 UTC m=+0.365403071 container start 90f2f2e3076c9b36c9a50b5615dc413a490dc8e1a1c07180327e849ab514afa4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:29:51.170 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:50 smithi032 podman[215386]: 2024-09-16 05:29:50.992493356 +0000 UTC m=+0.461871921 container attach 90f2f2e3076c9b36c9a50b5615dc413a490dc8e1a1c07180327e849ab514afa4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_CLEAN=True) 2024-09-16T05:29:51.171 INFO:journalctl@ceph.osd.0.smithi032.stdout:Sep 16 05:29:51 smithi032 podman[215386]: 2024-09-16 05:29:51.091020291 +0000 UTC m=+0.560398842 container died 90f2f2e3076c9b36c9a50b5615dc413a490dc8e1a1c07180327e849ab514afa4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e) 2024-09-16T05:29:51.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.0.service' 2024-09-16T05:29:51.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:29:51.784 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-16T05:29:51.784 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-16T05:29:51.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.1 2024-09-16T05:29:52.126 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:51 smithi032 systemd[1]: Stopping Ceph osd.1 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:29:52.420 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:52 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[195790]: 2024-09-16T05:29:52.124+0000 7fb66b14d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:29:52.420 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:52 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[195790]: 2024-09-16T05:29:52.124+0000 7fb66b14d640 -1 osd.1 1083 *** Got signal Terminated *** 2024-09-16T05:29:52.420 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:52 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1[195790]: 2024-09-16T05:29:52.124+0000 7fb66b14d640 -1 osd.1 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:29:57.901 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:57 smithi032 podman[215490]: 2024-09-16 05:29:57.589918677 +0000 UTC m=+5.577748568 container died c450081619965088f49743edc739264129d3c1e0693e6c783c407c17a3cb2891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:29:58.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:58 smithi032 podman[215490]: 2024-09-16 05:29:58.082806527 +0000 UTC m=+6.070636415 container cleanup c450081619965088f49743edc739264129d3c1e0693e6c783c407c17a3cb2891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T05:29:58.170 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:58 smithi032 bash[215490]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1 2024-09-16T05:29:58.652 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:58 smithi032 podman[215506]: 2024-09-16 05:29:58.334410186 +0000 UTC m=+0.740998336 container remove c450081619965088f49743edc739264129d3c1e0693e6c783c407c17a3cb2891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-16T05:29:58.920 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:58 smithi032 podman[215595]: 2024-09-16 05:29:58.651759452 +0000 UTC m=+0.104178881 container create 1919937e8d41e95b5608c87f0424a952b80362129c0574c00a79ddf175b2f147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-09-16T05:29:58.920 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:58 smithi032 podman[215595]: 2024-09-16 05:29:58.564372465 +0000 UTC m=+0.016791905 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:29:58.920 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:58 smithi032 podman[215595]: 2024-09-16 05:29:58.854533468 +0000 UTC m=+0.306952893 container init 1919937e8d41e95b5608c87f0424a952b80362129c0574c00a79ddf175b2f147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-16T05:29:58.920 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:58 smithi032 podman[215595]: 2024-09-16 05:29:58.857833073 +0000 UTC m=+0.310252507 container start 1919937e8d41e95b5608c87f0424a952b80362129c0574c00a79ddf175b2f147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-16T05:29:59.350 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:58 smithi032 podman[215595]: 2024-09-16 05:29:58.949005994 +0000 UTC m=+0.401425429 container attach 1919937e8d41e95b5608c87f0424a952b80362129c0574c00a79ddf175b2f147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-8921c8e, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:29:59.350 INFO:journalctl@ceph.osd.1.smithi032.stdout:Sep 16 05:29:59 smithi032 podman[215595]: 2024-09-16 05:29:59.050478034 +0000 UTC m=+0.502897472 container died 1919937e8d41e95b5608c87f0424a952b80362129c0574c00a79ddf175b2f147 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True) 2024-09-16T05:29:59.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.1.service' 2024-09-16T05:29:59.815 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:29:59.815 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-16T05:29:59.815 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-16T05:29:59.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.2 2024-09-16T05:30:00.162 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:29:59 smithi032 systemd[1]: Stopping Ceph osd.2 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:30:00.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:00 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[102067]: 2024-09-16T05:30:00.160+0000 7f19be73b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:30:00.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:00 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[102067]: 2024-09-16T05:30:00.160+0000 7f19be73b640 -1 osd.2 1083 *** Got signal Terminated *** 2024-09-16T05:30:00.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:00 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2[102067]: 2024-09-16T05:30:00.160+0000 7f19be73b640 -1 osd.2 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:30:06.012 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:05 smithi032 podman[215698]: 2024-09-16 05:30:05.69911696 +0000 UTC m=+5.651236924 container died 7b309f5042a491e074273c38642ff74261fd1db2537574e78f165e8a111104ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-16T05:30:06.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:06 smithi032 podman[215698]: 2024-09-16 05:30:06.152694762 +0000 UTC m=+6.104814768 container cleanup 7b309f5042a491e074273c38642ff74261fd1db2537574e78f165e8a111104ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-16T05:30:06.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:06 smithi032 bash[215698]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2 2024-09-16T05:30:06.754 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:06 smithi032 podman[215715]: 2024-09-16 05:30:06.420515521 +0000 UTC m=+0.717662757 container remove 7b309f5042a491e074273c38642ff74261fd1db2537574e78f165e8a111104ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:30:07.049 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:06 smithi032 podman[215801]: 2024-09-16 05:30:06.754682099 +0000 UTC m=+0.108435960 container create 646b35b562f427778362e07ee4c0482d48ad3be5a31c21792eeff5f8e6e00f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-16T05:30:07.049 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:06 smithi032 podman[215801]: 2024-09-16 05:30:06.662526182 +0000 UTC m=+0.016280055 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:30:07.049 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:06 smithi032 podman[215801]: 2024-09-16 05:30:06.982254709 +0000 UTC m=+0.336008570 container init 646b35b562f427778362e07ee4c0482d48ad3be5a31c21792eeff5f8e6e00f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:07.049 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:06 smithi032 podman[215801]: 2024-09-16 05:30:06.985649396 +0000 UTC m=+0.339403257 container start 646b35b562f427778362e07ee4c0482d48ad3be5a31c21792eeff5f8e6e00f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:30:07.049 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:07 smithi032 podman[215801]: 2024-09-16 05:30:07.048726993 +0000 UTC m=+0.402480870 container attach 646b35b562f427778362e07ee4c0482d48ad3be5a31c21792eeff5f8e6e00f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-16T05:30:07.420 INFO:journalctl@ceph.osd.2.smithi032.stdout:Sep 16 05:30:07 smithi032 podman[215801]: 2024-09-16 05:30:07.178504407 +0000 UTC m=+0.532258269 container died 646b35b562f427778362e07ee4c0482d48ad3be5a31c21792eeff5f8e6e00f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-16T05:30:07.825 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.2.service' 2024-09-16T05:30:07.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:30:07.866 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-16T05:30:07.867 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-16T05:30:07.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.3 2024-09-16T05:30:08.170 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:07 smithi032 systemd[1]: Stopping Ceph osd.3 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:30:08.670 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:08 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:30:08.206+0000 7f5a30935640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:30:08.670 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:08 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:30:08.206+0000 7f5a30935640 -1 osd.3 1083 *** Got signal Terminated *** 2024-09-16T05:30:08.670 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:08 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:30:08.206+0000 7f5a30935640 -1 osd.3 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:30:10.645 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:10.175+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:10.920 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:10 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:30:10.511+0000 7f5a2cf4b640 -1 osd.3 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.935053+0000 front 2024-09-16T05:29:44.934951+0000 (oldest deadline 2024-09-16T05:30:10.235230+0000) 2024-09-16T05:30:10.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:10.643+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:10.921 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:10.828+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:11.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:11.158+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:11.822 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:11 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:30:11.480+0000 7f5a2cf4b640 -1 osd.3 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.935053+0000 front 2024-09-16T05:29:44.934951+0000 (oldest deadline 2024-09-16T05:30:10.235230+0000) 2024-09-16T05:30:12.126 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:11.863+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:12.127 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:11.674+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:12.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:12.124+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:12.920 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:12 smithi032 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3[58043]: 2024-09-16T05:30:12.468+0000 7f5a2cf4b640 -1 osd.3 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.935053+0000 front 2024-09-16T05:29:44.934951+0000 (oldest deadline 2024-09-16T05:30:10.235230+0000) 2024-09-16T05:30:12.988 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:12 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:12.530+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:13.105 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:12.908+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:13.105 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:12.677+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:13.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:13.138+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:13.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:13.103+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:13.737 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:13 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:13.495+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:13.831 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:13 smithi032 podman[215902]: 2024-09-16 05:30:13.574671052 +0000 UTC m=+5.477968753 container died deff6dc8ef08ae31ecb9bc478753643ef8a75039cd1c5e274bcfef9d7a3d007c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-09-16T05:30:14.061 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:13.859+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:14.062 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:13.715+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:14.170 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:14 smithi032 podman[215902]: 2024-09-16 05:30:14.013728186 +0000 UTC m=+5.917025887 container cleanup deff6dc8ef08ae31ecb9bc478753643ef8a75039cd1c5e274bcfef9d7a3d007c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-16T05:30:14.170 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:14 smithi032 bash[215902]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3 2024-09-16T05:30:14.420 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:14.060+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:14.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:14.166+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:14.582 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:14 smithi032 podman[215932]: 2024-09-16 05:30:14.173017739 +0000 UTC m=+0.592882575 container remove deff6dc8ef08ae31ecb9bc478753643ef8a75039cd1c5e274bcfef9d7a3d007c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS) 2024-09-16T05:30:14.835 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:14 smithi032 podman[216017]: 2024-09-16 05:30:14.485311259 +0000 UTC m=+0.016645054 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:30:14.835 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:14 smithi032 podman[216017]: 2024-09-16 05:30:14.624243218 +0000 UTC m=+0.155577059 container create 1dabb70eb4eeae6d0f2035c5455396a779a614470b1cf5b5e198c8eff3a510b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-16T05:30:14.835 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:14 smithi032 podman[216017]: 2024-09-16 05:30:14.834705513 +0000 UTC m=+0.366039306 container init 1dabb70eb4eeae6d0f2035c5455396a779a614470b1cf5b5e198c8eff3a510b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, ceph=True) 2024-09-16T05:30:14.921 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:14.818+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:14.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:14.704+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:14.987 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:14 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:14.534+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:15.171 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:14 smithi032 podman[216017]: 2024-09-16 05:30:14.838119462 +0000 UTC m=+0.369453257 container start 1dabb70eb4eeae6d0f2035c5455396a779a614470b1cf5b5e198c8eff3a510b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731) 2024-09-16T05:30:15.171 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:14 smithi032 podman[216017]: 2024-09-16 05:30:14.909590974 +0000 UTC m=+0.440924768 container attach 1dabb70eb4eeae6d0f2035c5455396a779a614470b1cf5b5e198c8eff3a510b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-16T05:30:15.171 INFO:journalctl@ceph.osd.3.smithi032.stdout:Sep 16 05:30:15 smithi032 podman[216017]: 2024-09-16 05:30:15.034363566 +0000 UTC m=+0.565697359 container died 1dabb70eb4eeae6d0f2035c5455396a779a614470b1cf5b5e198c8eff3a510b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:15.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:15.096+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:15.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:15.215+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:15.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.3.service' 2024-09-16T05:30:15.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:30:15.688 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-16T05:30:15.688 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-16T05:30:15.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.4 2024-09-16T05:30:15.987 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:15 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:15.558+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:15.988 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:15 smithi196 systemd[1]: Stopping Ceph osd.4 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:30:16.094 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:15.868+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:16.094 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:15.694+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:16.420 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:16 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:16.093+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:16.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:16 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:16.172+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:16.487 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:16 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:16.070+0000 7fe42c885640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:30:16.487 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:16 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:16.070+0000 7fe42c885640 -1 osd.4 1083 *** Got signal Terminated *** 2024-09-16T05:30:16.488 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:16 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:16.070+0000 7fe42c885640 -1 osd.4 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:30:16.987 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:16 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:16.569+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:17.107 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:16 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:16.881+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:17.107 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:16 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:16.726+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:17.420 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:17 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:17.106+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:17.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:17 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:17.152+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:17.987 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:17 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:17.549+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:18.144 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:17 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:17.912+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:18.145 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:17 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:17.912+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:18.145 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:17 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:17.761+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:18.420 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:18 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:18.143+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:18.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:18 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:18.158+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:18.987 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:18 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:18.586+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:18.988 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:18 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:18.586+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:51.693699+0000 front 2024-09-16T05:29:51.693797+0000 (oldest deadline 2024-09-16T05:30:17.593721+0000) 2024-09-16T05:30:19.166 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:18 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:18.790+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:19.166 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:18 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:18.910+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:19.166 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:18 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:18.910+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:19.420 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:19 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:19.165+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:19.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:19 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:19.194+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:19.987 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:19 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:19.551+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:19.988 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:19 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:19.551+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:51.693699+0000 front 2024-09-16T05:29:51.693797+0000 (oldest deadline 2024-09-16T05:30:17.593721+0000) 2024-09-16T05:30:20.170 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:19 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:19.789+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:20.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:19 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:19.953+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:20.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:19 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:19.953+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:20.670 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:20 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:20.192+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:20.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:20 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:20.236+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:20.988 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:20 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:20.551+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.993228+0000 front 2024-09-16T05:29:46.993352+0000 (oldest deadline 2024-09-16T05:30:11.693129+0000) 2024-09-16T05:30:20.988 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:20 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4[35378]: 2024-09-16T05:30:20.551+0000 7fe428e9b640 -1 osd.4 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:51.693699+0000 front 2024-09-16T05:29:51.693797+0000 (oldest deadline 2024-09-16T05:30:17.593721+0000) 2024-09-16T05:30:21.148 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:20 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:20.818+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:21.149 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:20 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:20.924+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:21.149 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:20 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:20.924+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:21.420 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:21 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:21.147+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:21.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:21 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:21.282+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:21.488 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:21 smithi196 podman[125013]: 2024-09-16 05:30:21.38303422 +0000 UTC m=+5.500783716 container died cec86eaeb2989064a838e66d30691429dfc806d5e59d08066d3a91b2c24f6a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e) 2024-09-16T05:30:21.488 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:21 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:21.228+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:21.989 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:21 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:21.886+0000 7f7269c68640 -1 osd.5 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.987097+0000 front 2024-09-16T05:29:56.987024+0000 (oldest deadline 2024-09-16T05:30:21.687023+0000) 2024-09-16T05:30:21.989 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:21 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:21.793+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:21.990 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:21 smithi196 podman[125013]: 2024-09-16 05:30:21.7788502 +0000 UTC m=+5.896599694 container cleanup cec86eaeb2989064a838e66d30691429dfc806d5e59d08066d3a91b2c24f6a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:21.990 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:21 smithi196 bash[125013]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4 2024-09-16T05:30:22.145 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:21 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:21.781+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:22.145 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:21 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:21.962+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:22.146 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:21 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:21.962+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:22.269 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:22 smithi196 podman[125028]: 2024-09-16 05:30:22.005091378 +0000 UTC m=+0.615224690 container remove cec86eaeb2989064a838e66d30691429dfc806d5e59d08066d3a91b2c24f6a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-16T05:30:22.420 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:22 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:22.144+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:22.420 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:22 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:22.144+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:22.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:22 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:22.253+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:22.520 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:22 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:22.269+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:22.521 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:22 smithi196 podman[125118]: 2024-09-16 05:30:22.301485526 +0000 UTC m=+0.095225090 container create c69a49f5711a5f474c8c313e702ac1ea954206f1daaa8795dadb32159ac2ca60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-16T05:30:22.521 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:22 smithi196 podman[125118]: 2024-09-16 05:30:22.222352474 +0000 UTC m=+0.016092038 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:30:22.521 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:22 smithi196 podman[125118]: 2024-09-16 05:30:22.516842918 +0000 UTC m=+0.310582491 container init c69a49f5711a5f474c8c313e702ac1ea954206f1daaa8795dadb32159ac2ca60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:30:22.521 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:22 smithi196 podman[125118]: 2024-09-16 05:30:22.520175403 +0000 UTC m=+0.313914968 container start c69a49f5711a5f474c8c313e702ac1ea954206f1daaa8795dadb32159ac2ca60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:22.913 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:22 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:22.743+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:22.913 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:22 smithi196 podman[125118]: 2024-09-16 05:30:22.616560958 +0000 UTC m=+0.410300537 container attach c69a49f5711a5f474c8c313e702ac1ea954206f1daaa8795dadb32159ac2ca60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-16T05:30:22.913 INFO:journalctl@ceph.osd.4.smithi196.stdout:Sep 16 05:30:22 smithi196 podman[125118]: 2024-09-16 05:30:22.702998245 +0000 UTC m=+0.496737812 container died c69a49f5711a5f474c8c313e702ac1ea954206f1daaa8795dadb32159ac2ca60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731) 2024-09-16T05:30:23.107 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:22 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:22.811+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:23.108 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:23 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:22.999+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:23.108 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:23 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:22.999+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:23.237 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:22 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:22.912+0000 7f7269c68640 -1 osd.5 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.987097+0000 front 2024-09-16T05:29:56.987024+0000 (oldest deadline 2024-09-16T05:30:21.687023+0000) 2024-09-16T05:30:23.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.4.service' 2024-09-16T05:30:23.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:30:23.351 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-16T05:30:23.351 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-16T05:30:23.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.5 2024-09-16T05:30:23.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:23 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:23.272+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:23.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:23 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:23.106+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:23.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:23 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:23.106+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:23.699 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:23.267+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:23.699 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:23 smithi196 systemd[1]: Stopping Ceph osd.5 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:30:23.699 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:23.698+0000 7f726d652640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:30:23.699 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:23.698+0000 7f726d652640 -1 osd.5 1083 *** Got signal Terminated *** 2024-09-16T05:30:23.700 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:23.698+0000 7f726d652640 -1 osd.5 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:30:23.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:23.724+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:23.988 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:23 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:23.924+0000 7f7269c68640 -1 osd.5 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.987097+0000 front 2024-09-16T05:29:56.987024+0000 (oldest deadline 2024-09-16T05:30:21.687023+0000) 2024-09-16T05:30:24.026 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:23 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:23.768+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:24.287 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:24 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:24.108+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:24.287 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:24 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:24.108+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:24.288 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:24 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:24.025+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:24.288 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:24 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:24.025+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:24.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:24 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:24.286+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:24.737 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:24 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:24.290+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:25.052 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:24 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:24.737+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:25.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:24 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:24.744+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:25.238 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:24 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:24.963+0000 7f7269c68640 -1 osd.5 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.987097+0000 front 2024-09-16T05:29:56.987024+0000 (oldest deadline 2024-09-16T05:30:21.687023+0000) 2024-09-16T05:30:25.312 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:25 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:25.069+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:25.312 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:25 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:25.069+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:25.312 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:25 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:25.051+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:25.312 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:25 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:25.051+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:25.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:25 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:25.311+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:25.737 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:25 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:25.334+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:26.019 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:25 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:25.754+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:26.238 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:25 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:25.768+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:26.238 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:25 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:25.951+0000 7f7269c68640 -1 osd.5 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.987097+0000 front 2024-09-16T05:29:56.987024+0000 (oldest deadline 2024-09-16T05:30:21.687023+0000) 2024-09-16T05:30:26.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:26.267+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:26.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:26.267+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:26.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:26.066+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:26.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:26.066+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:26.421 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:26.018+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:26.421 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:26.018+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:26.738 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:26.382+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:27.170 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:26.796+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:27.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:26.970+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:27.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:26 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:26.970+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:27.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:27 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:27.039+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:27.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:27 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:27.039+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:27.237 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:26.770+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:27.238 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:26 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:26.974+0000 7f7269c68640 -1 osd.5 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.987097+0000 front 2024-09-16T05:29:56.987024+0000 (oldest deadline 2024-09-16T05:30:21.687023+0000) 2024-09-16T05:30:27.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:27 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:27.242+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:27.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:27 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:27.242+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:27.738 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:27.382+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:27.994 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:27.738+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:28.170 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:27 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:27.819+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:28.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:28.019+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:28.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:28.019+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:28.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:27 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:27.960+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:28.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:27 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:27.960+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:28.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:27 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:27.960+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:28.401 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:27 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5[41226]: 2024-09-16T05:30:27.993+0000 7f7269c68640 -1 osd.5 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.987097+0000 front 2024-09-16T05:29:56.987024+0000 (oldest deadline 2024-09-16T05:30:21.687023+0000) 2024-09-16T05:30:28.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:28.243+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:28.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:28.243+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:28.691 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:28 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:28.400+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:28.691 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:28 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:28.400+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:28.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:28 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:28.690+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:28.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:28 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:28.690+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:29.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:28.835+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:29.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:28.835+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:29.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:28.995+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:29.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:28.995+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:29.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:28.992+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:29.172 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:28.992+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:29.172 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:28 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:28.992+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:29.352 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:29 smithi196 podman[125220]: 2024-09-16 05:30:29.08212427 +0000 UTC m=+5.503257776 container died fa0fc233182deeb8fe84f96b03ae114ece2cd8a04b3491b42af9da3f93b49520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:29.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:29.246+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:29.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:29.246+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:29.686 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:29 smithi196 podman[125220]: 2024-09-16 05:30:29.447585638 +0000 UTC m=+5.868719141 container cleanup fa0fc233182deeb8fe84f96b03ae114ece2cd8a04b3491b42af9da3f93b49520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-16T05:30:29.686 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:29 smithi196 bash[125220]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5 2024-09-16T05:30:29.686 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:29 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:29.423+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:29.686 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:29 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:29.423+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:29.987 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:29 smithi196 podman[125240]: 2024-09-16 05:30:29.740449523 +0000 UTC m=+0.650424575 container remove fa0fc233182deeb8fe84f96b03ae114ece2cd8a04b3491b42af9da3f93b49520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-16T05:30:29.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:29 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:29.684+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:29.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:29 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:29.684+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:30.170 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:29.986+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:30.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:29.986+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:30.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:29.799+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:30.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:29.799+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:30.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:29.968+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:30.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:29.968+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:30.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:29 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:29.968+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:30.327 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:30 smithi196 podman[125327]: 2024-09-16 05:30:29.967622206 +0000 UTC m=+0.016050283 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:30:30.327 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:30 smithi196 podman[125327]: 2024-09-16 05:30:30.099687643 +0000 UTC m=+0.148115752 container create 1fb47497032dffbf7657facf8729ce57205de67d9793485f863e5a6cee3a838d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-16T05:30:30.327 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:30 smithi196 podman[125327]: 2024-09-16 05:30:30.327055618 +0000 UTC m=+0.375483682 container init 1fb47497032dffbf7657facf8729ce57205de67d9793485f863e5a6cee3a838d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-16T05:30:30.641 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:30 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:30.381+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:30.641 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:30 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:30.381+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:30.641 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:30 smithi196 podman[125327]: 2024-09-16 05:30:30.330420676 +0000 UTC m=+0.378848741 container start 1fb47497032dffbf7657facf8729ce57205de67d9793485f863e5a6cee3a838d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:30.642 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:30 smithi196 podman[125327]: 2024-09-16 05:30:30.401979161 +0000 UTC m=+0.450407226 container attach 1fb47497032dffbf7657facf8729ce57205de67d9793485f863e5a6cee3a838d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:30:30.642 INFO:journalctl@ceph.osd.5.smithi196.stdout:Sep 16 05:30:30 smithi196 podman[125327]: 2024-09-16 05:30:30.520644341 +0000 UTC m=+0.569072416 container died 1fb47497032dffbf7657facf8729ce57205de67d9793485f863e5a6cee3a838d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2) 2024-09-16T05:30:30.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:30 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:30.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:30.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:30 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:30.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:30.989 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:30 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:30.640+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:30.989 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:30 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:30.640+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:31.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.5.service' 2024-09-16T05:30:31.170 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:30 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:30.993+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:31.170 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:30 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:30.993+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:31.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:30 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:30.778+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:31.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:30 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:30.778+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:31.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:31 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:31.012+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:31.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:31 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:31.012+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:31.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:31 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:31.012+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:31.177 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:30:31.178 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-16T05:30:31.178 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-16T05:30:31.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.6 2024-09-16T05:30:31.488 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:31 smithi196 systemd[1]: Stopping Ceph osd.6 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:30:31.488 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:31.344+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:31.488 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:31.344+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:31.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:31 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:31.258+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:31.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:31 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:31.258+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:31.987 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:31.506+0000 7f6af94c8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:30:31.988 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:31.506+0000 7f6af94c8640 -1 osd.6 1083 *** Got signal Terminated *** 2024-09-16T05:30:31.988 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:31.506+0000 7f6af94c8640 -1 osd.6 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:30:31.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:31.685+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:31.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:31 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:31.685+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:32.027 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:31 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:31.757+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:32.027 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:31 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:31.757+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:32.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:32.235+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:32.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:32.235+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:32.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:32.026+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:32.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:32.026+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:32.421 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:32.047+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:32.421 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:32.047+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:32.422 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:32.047+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:32.668 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:32 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:32.299+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:32.668 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:32 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:32.299+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:32.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:32 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:32.666+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:32.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:32 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:32.666+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:32.999 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:32.722+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:32.999 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:32.722+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:33.251 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:33.002+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:33.251 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:33.002+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:33.252 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:32.998+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:33.252 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:32.998+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:33.252 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:32 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:32.998+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:33.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:33.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:33.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:33.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:33.709 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:33 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:33.268+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:33.709 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:33 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:33.268+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:33.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:33 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:33.708+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:33.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:33 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:33.708+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:33.997 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:33.744+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:33.998 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:33.744+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:34.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:34.214+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:34.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:34.214+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:34.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:34.039+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:34.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:34.039+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:34.421 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:33.996+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:34.421 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:33.996+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:34.421 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:33 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:33.996+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:34.669 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:34 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:34.241+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:34.669 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:34 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:34.241+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:34.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:34 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:34.667+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:34.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:34 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:34.668+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:34.997 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:34.733+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:34.998 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:34.733+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:34.998 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:34.981+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:34.998 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:34.981+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:34.998 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:34.981+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:34.998 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:34.981+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:35.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:35 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:35.224+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:35.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:35 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:35.224+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:35.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:34.997+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:35.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:34.997+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:35.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:34 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:34.997+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:35.487 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:35 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:35.228+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:35.488 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:35 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:35.228+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:35.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:35 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:35.641+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:35.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:35 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:35.641+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:36.034 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:35 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:35.772+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:36.034 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:35 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:35.772+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:36.034 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:35 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:35.996+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:36.034 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:35 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:35.996+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:36.034 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:35 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:35.996+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:36.034 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:35 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:35.996+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:36.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:36.266+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:36.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:36.266+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:36.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:36.266+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:36.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:36.033+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:36.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:36.033+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:36.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:36.033+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:36.655 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:36 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:36.257+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.254895+0000 front 2024-09-16T05:29:56.254780+0000 (oldest deadline 2024-09-16T05:30:20.954787+0000) 2024-09-16T05:30:36.655 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:36 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6[46706]: 2024-09-16T05:30:36.257+0000 7f6af1ad6640 -1 osd.6 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:00.955260+0000 front 2024-09-16T05:30:00.955364+0000 (oldest deadline 2024-09-16T05:30:27.955712+0000) 2024-09-16T05:30:36.987 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:36 smithi196 podman[125427]: 2024-09-16 05:30:36.839586627 +0000 UTC m=+5.443585827 container died 4a23d33e37101d666a2d61b6c46feb9dee99d031264f4fded9a88bfd6e16d769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T05:30:36.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:36 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:36.654+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:36.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:36 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:36.654+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:36.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:36 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:36.654+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:11.876833+0000 front 2024-09-16T05:30:11.876699+0000 (oldest deadline 2024-09-16T05:30:36.576866+0000) 2024-09-16T05:30:37.047 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:36.763+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:37.047 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:36.763+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:37.047 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:36.975+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:37.047 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:36.975+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:37.047 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:36.975+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:37.047 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:36 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:36.975+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:37.401 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:37 smithi196 podman[125427]: 2024-09-16 05:30:37.115856091 +0000 UTC m=+5.719855290 container cleanup 4a23d33e37101d666a2d61b6c46feb9dee99d031264f4fded9a88bfd6e16d769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e) 2024-09-16T05:30:37.401 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:37 smithi196 bash[125427]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6 2024-09-16T05:30:37.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:37.242+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:37.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:37.242+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:37.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:37.242+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:37.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:37.046+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:37.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:37.046+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:37.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:37.046+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:37.693 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:37 smithi196 podman[125444]: 2024-09-16 05:30:37.400579075 +0000 UTC m=+0.555510593 container remove 4a23d33e37101d666a2d61b6c46feb9dee99d031264f4fded9a88bfd6e16d769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-16T05:30:37.978 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:37 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:37.692+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:37.978 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:37 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:37.692+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:37.979 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:37 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:37.692+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:11.876833+0000 front 2024-09-16T05:30:11.876699+0000 (oldest deadline 2024-09-16T05:30:36.576866+0000) 2024-09-16T05:30:37.979 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:37 smithi196 podman[125529]: 2024-09-16 05:30:37.612309889 +0000 UTC m=+0.016025780 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:30:37.979 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:37 smithi196 podman[125529]: 2024-09-16 05:30:37.717852654 +0000 UTC m=+0.121568544 container create a20288ad7e8d850e5cb621659db310b2c0fed3c9897f80196b5653e94f2279f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-16T05:30:37.979 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:37 smithi196 podman[125529]: 2024-09-16 05:30:37.903460098 +0000 UTC m=+0.307175988 container init a20288ad7e8d850e5cb621659db310b2c0fed3c9897f80196b5653e94f2279f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-16T05:30:37.979 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:37 smithi196 podman[125529]: 2024-09-16 05:30:37.906669075 +0000 UTC m=+0.310384974 container start a20288ad7e8d850e5cb621659db310b2c0fed3c9897f80196b5653e94f2279f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:30:37.979 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:37 smithi196 podman[125529]: 2024-09-16 05:30:37.978375069 +0000 UTC m=+0.382090962 container attach a20288ad7e8d850e5cb621659db310b2c0fed3c9897f80196b5653e94f2279f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-16T05:30:38.040 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:37.985+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:38.040 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:37.985+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:38.040 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:37.985+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:38.040 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:37.985+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:38.040 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:37.781+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:38.040 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:37.781+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:38.041 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:37 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:37.781+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:38.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Sep 16 05:30:38 smithi196 podman[125529]: 2024-09-16 05:30:38.096989267 +0000 UTC m=+0.500705158 container died a20288ad7e8d850e5cb621659db310b2c0fed3c9897f80196b5653e94f2279f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-09-16T05:30:38.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:38.280+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:38.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:38.280+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:38.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:38.280+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:38.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:38.039+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:38.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:38.039+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:38.421 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:38.039+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:38.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.6.service' 2024-09-16T05:30:38.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:30:38.762 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-16T05:30:38.763 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-16T05:30:38.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.7 2024-09-16T05:30:38.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:38 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:38.642+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:38.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:38 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:38.642+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:38.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:38 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:38.642+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:11.876833+0000 front 2024-09-16T05:30:11.876699+0000 (oldest deadline 2024-09-16T05:30:36.576866+0000) 2024-09-16T05:30:38.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:38 smithi196 systemd[1]: Stopping Ceph osd.7 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:30:39.034 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:38.938+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:39.034 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:38.938+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:39.035 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:38.938+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:39.035 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:38.938+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:39.035 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:38.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:39.035 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:38.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:39.035 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:38 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:38.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:39.314 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:39.034+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:39.314 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:39.034+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:39.314 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:39.034+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:39.487 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:39 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:39.084+0000 7fd6f046f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:30:39.488 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:39 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:39.084+0000 7fd6f046f640 -1 osd.7 1083 *** Got signal Terminated *** 2024-09-16T05:30:39.488 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:39 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:39.084+0000 7fd6f046f640 -1 osd.7 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:30:39.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:39.313+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:39.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:39.313+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:39.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:39.313+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:39.970 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:39.713+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:39.970 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:39.713+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:39.970 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:39.713+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:39.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:39 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:39.617+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:39.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:39 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:39.617+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:39.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:39 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:39.617+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:11.876833+0000 front 2024-09-16T05:30:11.876699+0000 (oldest deadline 2024-09-16T05:30:36.576866+0000) 2024-09-16T05:30:40.359 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:39.986+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:40.359 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:39.986+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:40.359 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:39.986+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:40.360 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:39.969+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:40.360 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:39.969+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:40.360 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:39.969+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:40.360 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:39 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:39.969+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:40.620 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:40.358+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:40.620 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:40.358+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:40.620 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:40.358+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:40.920 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:40.671+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:40.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:40.671+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:40.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:40.671+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:40.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:40 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:40.597+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:40.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:40 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:40.597+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:40.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:40 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:40.597+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:11.876833+0000 front 2024-09-16T05:30:11.876699+0000 (oldest deadline 2024-09-16T05:30:36.576866+0000) 2024-09-16T05:30:41.318 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:40.971+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:41.318 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:40.971+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:41.318 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:40.971+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:41.318 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:40.997+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:41.318 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:40.997+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:41.318 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:40.997+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:41.318 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:40 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:40.997+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:41.570 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:41.316+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:41.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:41.316+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:41.571 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:41.316+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:41.920 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:41.692+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:41.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:41.692+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:41.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:41.692+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:41.989 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:41 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:41.563+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:41.989 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:41 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:41.563+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:41.989 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:41 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:41.563+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:11.876833+0000 front 2024-09-16T05:30:11.876699+0000 (oldest deadline 2024-09-16T05:30:36.576866+0000) 2024-09-16T05:30:42.344 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:41.984+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:42.345 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:41.984+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:42.345 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:41.984+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:42.345 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:41.971+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:42.345 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:41.971+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:42.345 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:41.971+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:42.345 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:41 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:41.971+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:42.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:42.344+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:42.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:42.344+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:42.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:42.344+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:42.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:42 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:42.552+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:42.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:42 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:42.552+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:42.889 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:42 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:42.552+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:11.876833+0000 front 2024-09-16T05:30:11.876699+0000 (oldest deadline 2024-09-16T05:30:36.576866+0000) 2024-09-16T05:30:42.971 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:42.717+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:42.972 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:42.717+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:42.972 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:42.717+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:43.317 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:43.008+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:43.317 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:43.008+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:43.317 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:43.008+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:43.317 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:42.971+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:43.317 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:42.971+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:43.317 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:42.971+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:43.317 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:42 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:42.971+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:43.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:43.316+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:43.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:43.316+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:43.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:43.316+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:43.985 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:43.723+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:43.985 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:43.723+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:43.985 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:43.723+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:43.985 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:43.723+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:43.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:43 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:43.519+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:53.173762+0000 front 2024-09-16T05:29:53.173648+0000 (oldest deadline 2024-09-16T05:30:21.374310+0000) 2024-09-16T05:30:43.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:43 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:43.519+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:03.075671+0000 front 2024-09-16T05:30:03.075557+0000 (oldest deadline 2024-09-16T05:30:28.375576+0000) 2024-09-16T05:30:43.987 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:43 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:43.519+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:11.876833+0000 front 2024-09-16T05:30:11.876699+0000 (oldest deadline 2024-09-16T05:30:36.576866+0000) 2024-09-16T05:30:43.988 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:43 smithi196 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7[121373]: 2024-09-16T05:30:43.519+0000 7fd6eca85640 -1 osd.7 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.477777+0000 front 2024-09-16T05:30:19.477966+0000 (oldest deadline 2024-09-16T05:30:42.977840+0000) 2024-09-16T05:30:44.339 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:44.036+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:44.339 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:44.036+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:44.339 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:44.036+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:44.339 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:43.984+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:44.339 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:43.984+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:44.340 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:43.984+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:44.340 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:43 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:43.984+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:44.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:44.338+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:44.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:44.338+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:44.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:44.338+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:44.785 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:44 smithi196 podman[125631]: 2024-09-16 05:30:44.523314393 +0000 UTC m=+5.538629685 container died cb10ac9c829f709d3fed489f74ee31a7e50d6063c3630bfe4af320f47d64a9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:30:44.988 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:44.688+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:44.988 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:44.688+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:44.988 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:44.688+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:44.989 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:44.688+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:45.153 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:44 smithi196 podman[125631]: 2024-09-16 05:30:44.934699935 +0000 UTC m=+5.950015268 container cleanup cb10ac9c829f709d3fed489f74ee31a7e50d6063c3630bfe4af320f47d64a9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-16T05:30:45.153 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:44 smithi196 bash[125631]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7 2024-09-16T05:30:45.357 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:45.062+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:45.358 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:45.062+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:45.358 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:45.062+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:45.358 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:44.987+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:45.358 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:44.987+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:45.359 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:44.987+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:45.359 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:44 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:44.987+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:45.488 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:45 smithi196 podman[125662]: 2024-09-16 05:30:45.15225416 +0000 UTC m=+0.626084548 container remove cb10ac9c829f709d3fed489f74ee31a7e50d6063c3630bfe4af320f47d64a9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-16T05:30:45.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:45.357+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:45.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:45.357+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:45.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:45.357+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:45.813 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:45 smithi196 podman[125750]: 2024-09-16 05:30:45.54457655 +0000 UTC m=+0.107690814 container create f1780950dfb4e240553eb9561e05195057ab1b3b5228d329fb267365f1448c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e) 2024-09-16T05:30:45.813 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:45 smithi196 podman[125750]: 2024-09-16 05:30:45.452948003 +0000 UTC m=+0.016062276 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:30:45.813 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:45 smithi196 podman[125750]: 2024-09-16 05:30:45.746802058 +0000 UTC m=+0.309916324 container init f1780950dfb4e240553eb9561e05195057ab1b3b5228d329fb267365f1448c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:30:45.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:45 smithi196 podman[125750]: 2024-09-16 05:30:45.750068002 +0000 UTC m=+0.313182269 container start f1780950dfb4e240553eb9561e05195057ab1b3b5228d329fb267365f1448c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-16T05:30:45.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:45 smithi196 podman[125750]: 2024-09-16 05:30:45.813315798 +0000 UTC m=+0.376430082 container attach f1780950dfb4e240553eb9561e05195057ab1b3b5228d329fb267365f1448c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T05:30:45.989 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:45.721+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:45.990 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:45.721+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:45.990 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:45.721+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:45.990 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:45.721+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:46.233 INFO:journalctl@ceph.osd.7.smithi196.stdout:Sep 16 05:30:45 smithi196 podman[125750]: 2024-09-16 05:30:45.938572815 +0000 UTC m=+0.501687088 container died f1780950dfb4e240553eb9561e05195057ab1b3b5228d329fb267365f1448c9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:46.330 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:46.089+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:46.330 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:46.089+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:46.330 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:46.089+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:46.330 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:46.089+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.807529+0000 front 2024-09-16T05:30:20.807382+0000 (oldest deadline 2024-09-16T05:30:45.507326+0000) 2024-09-16T05:30:46.330 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:45.988+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:46.330 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:45.988+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:46.330 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:45.988+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:46.330 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:45.988+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:46.331 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:45 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:45.988+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:46.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.7.service' 2024-09-16T05:30:46.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:30:46.630 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-16T05:30:46.630 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-16T05:30:46.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.8 2024-09-16T05:30:46.658 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:46.329+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:46.658 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:46.329+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:46.659 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:46.329+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:46.659 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:46.329+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:46.920 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:46 smithi204 systemd[1]: Stopping Ceph osd.8 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:30:46.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:46.732+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:46.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:46.732+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:46.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:46.732+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:46.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:46.732+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:47.299 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:46.964+0000 7f7351b6d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:30:47.299 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:46.964+0000 7f7351b6d640 -1 osd.8 1083 *** Got signal Terminated *** 2024-09-16T05:30:47.299 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:46.964+0000 7f7351b6d640 -1 osd.8 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:30:47.299 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:47.081+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:47.299 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:47.081+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:47.299 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:47.081+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:47.299 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:47.081+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.807529+0000 front 2024-09-16T05:30:20.807382+0000 (oldest deadline 2024-09-16T05:30:45.507326+0000) 2024-09-16T05:30:47.300 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:46.962+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:47.300 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:46.962+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:47.300 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:46.962+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:47.300 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:46.962+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:47.300 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:46 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:46.962+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:47.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:47.298+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:47.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:47.298+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:47.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:47.298+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:47.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:47.298+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:48.085 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:47.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:48.085 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:47.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:48.086 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:47.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:48.086 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:47.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:48.086 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:47.958+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:48.086 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:47.958+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:48.086 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:47.958+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:48.086 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:47.958+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:48.086 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:47 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:47.958+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:48.344 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:48.084+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:48.344 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:48.084+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:48.344 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:48.084+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:48.344 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:48.084+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.807529+0000 front 2024-09-16T05:30:20.807382+0000 (oldest deadline 2024-09-16T05:30:45.507326+0000) 2024-09-16T05:30:48.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:48.342+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:48.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:48.342+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:48.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:48.342+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:48.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:48.342+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:49.113 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:48.729+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:49.113 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:48.729+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:49.113 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:48.729+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:49.113 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:48.729+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:49.114 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:48.941+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:49.114 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:48.941+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:49.114 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:48.941+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:49.114 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:48.941+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:49.114 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:48 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:48.941+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:49.387 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:49.112+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:49.387 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:49.112+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:49.387 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:49.112+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:49.387 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:49.112+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.807529+0000 front 2024-09-16T05:30:20.807382+0000 (oldest deadline 2024-09-16T05:30:45.507326+0000) 2024-09-16T05:30:49.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:49.386+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:49.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:49.386+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:49.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:49.386+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:49.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:49.386+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:50.080 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:49.773+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:50.080 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:49.773+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:50.080 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:49.773+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:50.080 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:49.773+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:50.081 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:49.891+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:50.081 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:49.891+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:50.081 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:49.891+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:50.081 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:49.891+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:50.081 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:49 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:49.891+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:50.349 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:50.079+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:50.350 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:50.079+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:50.350 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:50.079+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:50.350 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:50.079+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.807529+0000 front 2024-09-16T05:30:20.807382+0000 (oldest deadline 2024-09-16T05:30:45.507326+0000) 2024-09-16T05:30:50.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:50.348+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:50.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:50.348+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:50.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:50.348+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:50.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:50.348+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:51.084 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:50.791+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:51.084 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:50.791+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:51.084 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:50.791+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:51.084 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:50.791+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:51.084 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:50.791+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:51.084 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:50.867+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:51.085 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:50.867+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:51.085 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:50.867+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:51.085 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:50.867+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:51.085 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:50.867+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:51.085 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:50 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:50.867+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:51.338 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:51.115+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:47.001789+0000 front 2024-09-16T05:29:47.001834+0000 (oldest deadline 2024-09-16T05:30:12.301857+0000) 2024-09-16T05:30:51.338 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:51.115+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.403142+0000 front 2024-09-16T05:29:56.403100+0000 (oldest deadline 2024-09-16T05:30:21.703203+0000) 2024-09-16T05:30:51.338 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:51.115+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.504683+0000 front 2024-09-16T05:30:10.769297+0000 (oldest deadline 2024-09-16T05:30:34.604809+0000) 2024-09-16T05:30:51.338 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:51.115+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.807529+0000 front 2024-09-16T05:30:20.807382+0000 (oldest deadline 2024-09-16T05:30:45.507326+0000) 2024-09-16T05:30:51.338 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8[35773]: 2024-09-16T05:30:51.115+0000 7f734e183640 -1 osd.8 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.408536+0000 front 2024-09-16T05:30:28.408629+0000 (oldest deadline 2024-09-16T05:30:50.708541+0000) 2024-09-16T05:30:51.644 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:51.337+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:51.644 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:51.337+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:51.645 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:51.337+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:51.645 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:51.337+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:51.902 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:51.830+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:51.902 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:51.830+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:51.902 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:51.830+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:51.902 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:51.830+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:51.903 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:51.830+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:52.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:51.901+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:52.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:51.901+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:52.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:51.901+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:52.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:51.901+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:52.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:51.901+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:52.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:51 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:51.901+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:52.670 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:52 smithi204 podman[125242]: 2024-09-16 05:30:52.428342619 +0000 UTC m=+5.597678038 container died 3033be91d6e51cfd961f921d48d3475722f226d2cc8ae3c4abc9a6515a461a93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-16T05:30:52.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:52.318+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:52.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:52.318+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:52.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:52.318+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:52.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:52.318+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:53.042 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:52 smithi204 podman[125242]: 2024-09-16 05:30:52.856928388 +0000 UTC m=+6.026263849 container cleanup 3033be91d6e51cfd961f921d48d3475722f226d2cc8ae3c4abc9a6515a461a93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:53.042 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:52 smithi204 bash[125242]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:52.829+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:52.829+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:52.829+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:52.829+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:52.829+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:52.859+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:52.859+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:52.859+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:52.859+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:53.043 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:52.859+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:53.044 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:52 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:52.859+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:53.356 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:53 smithi204 podman[125258]: 2024-09-16 05:30:53.042009478 +0000 UTC m=+0.615452716 container remove 3033be91d6e51cfd961f921d48d3475722f226d2cc8ae3c4abc9a6515a461a93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-16T05:30:53.670 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:53 smithi204 podman[125345]: 2024-09-16 05:30:53.353154903 +0000 UTC m=+0.016123402 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:30:53.670 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:53 smithi204 podman[125345]: 2024-09-16 05:30:53.451140094 +0000 UTC m=+0.114108588 container create 86d8fadd3ad6cb906481f69baeb05721c63c74b3b0dcaf5185005f1b5581276e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-16T05:30:53.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:53.355+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:53.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:53.355+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:53.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:53.355+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:53.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:53.355+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:53.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:53.355+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:30:53.921 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:53.827+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:53.921 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:53.827+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:53.921 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:53.827+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:53.921 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:53.827+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:53.921 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:53.827+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:53.921 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:53.827+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:53.921 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:53 smithi204 podman[125345]: 2024-09-16 05:30:53.670179867 +0000 UTC m=+0.333148361 container init 86d8fadd3ad6cb906481f69baeb05721c63c74b3b0dcaf5185005f1b5581276e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-16T05:30:53.922 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:53 smithi204 podman[125345]: 2024-09-16 05:30:53.674510843 +0000 UTC m=+0.337479340 container start 86d8fadd3ad6cb906481f69baeb05721c63c74b3b0dcaf5185005f1b5581276e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-16T05:30:53.922 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:53 smithi204 podman[125345]: 2024-09-16 05:30:53.7784567 +0000 UTC m=+0.441425208 container attach 86d8fadd3ad6cb906481f69baeb05721c63c74b3b0dcaf5185005f1b5581276e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-16T05:30:53.922 INFO:journalctl@ceph.osd.8.smithi204.stdout:Sep 16 05:30:53 smithi204 podman[125345]: 2024-09-16 05:30:53.875879913 +0000 UTC m=+0.538848411 container died 86d8fadd3ad6cb906481f69baeb05721c63c74b3b0dcaf5185005f1b5581276e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-16T05:30:53.922 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:53.817+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:53.922 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:53.817+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:53.922 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:53.817+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:53.922 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:53.817+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:53.922 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:53 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:53.817+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:54.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:54.377+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:54.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:54.377+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:54.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:54.377+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:54.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:54.377+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:54.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:54.377+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:30:54.458 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.8.service' 2024-09-16T05:30:54.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:30:54.538 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-16T05:30:54.538 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-16T05:30:54.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.9 2024-09-16T05:30:54.784 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 systemd[1]: Stopping Ceph osd.9 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:30:54.784 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.783+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:54.784 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.783+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:54.784 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.783+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:54.784 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.783+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:54.785 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.783+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:54.785 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.783+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:55.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.885+0000 7f9fda5d4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:30:55.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.885+0000 7f9fda5d4640 -1 osd.9 1083 *** Got signal Terminated *** 2024-09-16T05:30:55.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:54.885+0000 7f9fda5d4640 -1 osd.9 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:30:55.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:54.823+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:55.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:54.823+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:55.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:54.823+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:55.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:54.823+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:55.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:54 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:54.823+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:55.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:55.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:55.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:55.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:55.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:55.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:55.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:55.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:55.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:55.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:30:56.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:55.739+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:55.739+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:55.739+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:55.739+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:55.739+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:55.739+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:55.778+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:55.778+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:55.778+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:56.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:55.778+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:56.172 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:55 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:55.778+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:56.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:56.290+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:56.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:56.290+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:56.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:56.290+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:56.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:56.290+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:56.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:56.290+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:30:57.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:56.775+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:56.775+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:56.775+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:56.775+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:56.775+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:56.775+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:56.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:56.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:56.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:57.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:56.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:57.172 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:56 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:56.735+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:57.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:57.326+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:57.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:57.326+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:57.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:57.326+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:57.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:57.326+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:57.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:57.326+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:30:58.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:57.770+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:57.770+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:57.770+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:57.770+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:57.770+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:57.770+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:57.780+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:57.780+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:57.780+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:58.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:57.780+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:58.172 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:57 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:57.780+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:58.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:58.327+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:58.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:58.327+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:58.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:58.327+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:58.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:58.327+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:58.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:58.327+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:30:59.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:58.721+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:30:59.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:58.721+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:58.721+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:58.721+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:58.721+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:58.721+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:58.771+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:58.771+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:58.771+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:58.771+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:30:59.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:58 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:58.771+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:30:59.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:59.315+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:30:59.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:59.315+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:30:59.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:59.315+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:30:59.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:59.315+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:30:59.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:59.315+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:30:59.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:30:59.315+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:00.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:59.734+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.172167+0000 front 2024-09-16T05:29:46.172263+0000 (oldest deadline 2024-09-16T05:30:10.271924+0000) 2024-09-16T05:31:00.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:59.734+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6814 osd.1 since back 2024-09-16T05:29:56.172821+0000 front 2024-09-16T05:29:56.173134+0000 (oldest deadline 2024-09-16T05:30:17.872729+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:59.734+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.874184+0000 front 2024-09-16T05:30:04.874210+0000 (oldest deadline 2024-09-16T05:30:27.773907+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:59.734+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.075165+0000 front 2024-09-16T05:30:10.075134+0000 (oldest deadline 2024-09-16T05:30:34.774985+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:59.734+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:19.976321+0000 front 2024-09-16T05:30:19.976306+0000 (oldest deadline 2024-09-16T05:30:45.876081+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9[41206]: 2024-09-16T05:30:59.734+0000 7f9fd2be2640 -1 osd.9 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:25.876793+0000 front 2024-09-16T05:30:25.876827+0000 (oldest deadline 2024-09-16T05:30:49.976555+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:59.763+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:59.763+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:59.763+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:59.763+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:59.763+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:31:00.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:30:59 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:30:59.763+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:35.927028+0000 front 2024-09-16T05:30:35.927140+0000 (oldest deadline 2024-09-16T05:30:59.427032+0000) 2024-09-16T05:31:00.528 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:00 smithi204 podman[125449]: 2024-09-16 05:31:00.277508677 +0000 UTC m=+5.521140704 container died f6762616097051f5f8495ec4c78e0ec9d9f527ac8d5e3c0d4b4581f9abbd1fa4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-16T05:31:00.528 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:00.304+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:00.528 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:00.304+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:00.528 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:00.304+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:00.529 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:00.304+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:00.529 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:00.304+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:00.529 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:00.304+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:00.782 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:00 smithi204 podman[125449]: 2024-09-16 05:31:00.702439443 +0000 UTC m=+5.946071470 container cleanup f6762616097051f5f8495ec4c78e0ec9d9f527ac8d5e3c0d4b4581f9abbd1fa4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-16T05:31:00.782 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:00 smithi204 bash[125449]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9 2024-09-16T05:31:01.170 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:00 smithi204 podman[125465]: 2024-09-16 05:31:00.912319709 +0000 UTC m=+0.637249152 container remove f6762616097051f5f8495ec4c78e0ec9d9f527ac8d5e3c0d4b4581f9abbd1fa4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2) 2024-09-16T05:31:01.170 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:00.782+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:31:01.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:00.782+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:31:01.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:00.782+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:31:01.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:00.782+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:31:01.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:00.782+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:31:01.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:00 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:00.782+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:35.927028+0000 front 2024-09-16T05:30:35.927140+0000 (oldest deadline 2024-09-16T05:30:59.427032+0000) 2024-09-16T05:31:01.582 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:01.270+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:01.582 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:01.270+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:01.582 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:01.270+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:01.582 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:01.270+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:01.582 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:01.270+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:01.582 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:01.270+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:01.583 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:01 smithi204 podman[125551]: 2024-09-16 05:31:01.313219952 +0000 UTC m=+0.111960162 container create 7264a97d4e980a2226e36b1a651a2a0fae67a06476598626f3295af5e7ead34a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9-deactivate, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-16T05:31:01.583 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:01 smithi204 podman[125551]: 2024-09-16 05:31:01.217393729 +0000 UTC m=+0.016133944 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:31:01.583 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:01 smithi204 podman[125551]: 2024-09-16 05:31:01.507074689 +0000 UTC m=+0.305814909 container init 7264a97d4e980a2226e36b1a651a2a0fae67a06476598626f3295af5e7ead34a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-16T05:31:01.583 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:01 smithi204 podman[125551]: 2024-09-16 05:31:01.510405093 +0000 UTC m=+0.309145296 container start 7264a97d4e980a2226e36b1a651a2a0fae67a06476598626f3295af5e7ead34a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-16T05:31:01.583 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:01 smithi204 podman[125551]: 2024-09-16 05:31:01.581923926 +0000 UTC m=+0.380664142 container attach 7264a97d4e980a2226e36b1a651a2a0fae67a06476598626f3295af5e7ead34a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-09-16T05:31:01.920 INFO:journalctl@ceph.osd.9.smithi204.stdout:Sep 16 05:31:01 smithi204 podman[125551]: 2024-09-16 05:31:01.69786115 +0000 UTC m=+0.496601367 container died 7264a97d4e980a2226e36b1a651a2a0fae67a06476598626f3295af5e7ead34a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-9-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-16T05:31:01.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:01.785+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:31:01.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:01.785+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:31:01.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:01.785+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:31:01.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:01.785+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:31:01.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:01.785+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:31:01.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:01 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:01.785+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:35.927028+0000 front 2024-09-16T05:30:35.927140+0000 (oldest deadline 2024-09-16T05:30:59.427032+0000) 2024-09-16T05:31:02.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.9.service' 2024-09-16T05:31:02.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:31:02.422 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-16T05:31:02.422 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-16T05:31:02.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.10 2024-09-16T05:31:02.670 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 systemd[1]: Stopping Ceph osd.10 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:31:02.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:02.292+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:02.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:02.292+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:02.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:02.292+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:02.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:02.292+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:02.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:02.292+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:02.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:02.292+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.764+0000 7f6c76b29640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.764+0000 7f6c76b29640 -1 osd.10 1083 *** Got signal Terminated *** 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.764+0000 7f6c76b29640 -1 osd.10 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.787+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.787+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.787+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.787+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.787+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:31:03.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:02 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:02.787+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:35.927028+0000 front 2024-09-16T05:30:35.927140+0000 (oldest deadline 2024-09-16T05:30:59.427032+0000) 2024-09-16T05:31:03.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:03.325+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:03.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:03.325+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:03.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:03.325+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:03.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:03.325+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:03.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:03.325+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:03.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:03.325+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:04.170 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:03.837+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:31:04.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:03.837+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:31:04.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:03.837+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:31:04.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:03.837+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:31:04.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:03.837+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:31:04.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:03 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:03.837+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:35.927028+0000 front 2024-09-16T05:30:35.927140+0000 (oldest deadline 2024-09-16T05:30:59.427032+0000) 2024-09-16T05:31:04.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:04.308+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:04.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:04.308+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:04.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:04.308+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:04.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:04.308+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:04.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:04.308+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:04.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:04.308+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:05.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:04.828+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:31:05.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:04.828+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:31:05.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:04.828+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:31:05.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:04.828+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:31:05.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:04.828+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:31:05.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:04 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:04.828+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:35.927028+0000 front 2024-09-16T05:30:35.927140+0000 (oldest deadline 2024-09-16T05:30:59.427032+0000) 2024-09-16T05:31:05.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:05.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:05.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:05.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:05.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:05.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:05.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:05.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:05.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:05.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:05.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:05.336+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:06.170 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:05.816+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:31:06.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:05.816+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:31:06.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:05.816+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:31:06.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:05.816+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:31:06.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:05.816+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:31:06.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:05 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:05.816+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:35.927028+0000 front 2024-09-16T05:30:35.927140+0000 (oldest deadline 2024-09-16T05:30:59.427032+0000) 2024-09-16T05:31:06.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:06.371+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:06.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:06.371+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:06.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:06.371+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:06.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:06.371+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:06.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:06.371+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:06.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:06.371+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:06.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:06.371+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:07.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:06.824+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:46.518984+0000 front 2024-09-16T05:29:46.518754+0000 (oldest deadline 2024-09-16T05:30:10.018692+0000) 2024-09-16T05:31:07.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:06.824+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.422893+0000 front 2024-09-16T05:30:04.422674+0000 (oldest deadline 2024-09-16T05:30:27.922609+0000) 2024-09-16T05:31:07.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:06.824+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:12.023532+0000 front 2024-09-16T05:30:12.023555+0000 (oldest deadline 2024-09-16T05:30:37.323628+0000) 2024-09-16T05:31:07.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:06.824+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:17.324556+0000 front 2024-09-16T05:30:17.324432+0000 (oldest deadline 2024-09-16T05:30:43.224316+0000) 2024-09-16T05:31:07.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:06.824+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:27.925544+0000 front 2024-09-16T05:30:27.925660+0000 (oldest deadline 2024-09-16T05:30:50.225452+0000) 2024-09-16T05:31:07.171 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:06 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10[46926]: 2024-09-16T05:31:06.824+0000 7f6c6f137640 -1 osd.10 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:35.927028+0000 front 2024-09-16T05:30:35.927140+0000 (oldest deadline 2024-09-16T05:30:59.427032+0000) 2024-09-16T05:31:07.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:07.356+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:07.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:07.356+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:07.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:07.356+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:07.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:07.356+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:07.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:07.356+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:07.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:07.356+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:07.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:07 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:07.356+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:08.565 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:08 smithi204 podman[125654]: 2024-09-16 05:31:08.186522065 +0000 UTC m=+5.544626853 container died 819f929cd263b158cea849525b6e661fb4562a754b001ffb3d5a5a58bba858ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-16T05:31:08.566 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:08 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:08.331+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:08.566 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:08 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:08.331+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:08.566 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:08 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:08.331+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:08.566 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:08 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:08.331+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:08.566 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:08 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:08.331+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:08.566 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:08 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:08.331+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:08.566 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:08 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:08.331+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:08.920 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:08 smithi204 podman[125654]: 2024-09-16 05:31:08.748168044 +0000 UTC m=+6.106272867 container cleanup 819f929cd263b158cea849525b6e661fb4562a754b001ffb3d5a5a58bba858ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2) 2024-09-16T05:31:08.921 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:08 smithi204 bash[125654]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10 2024-09-16T05:31:09.415 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:09 smithi204 podman[125670]: 2024-09-16 05:31:09.066693578 +0000 UTC m=+0.877661799 container remove 819f929cd263b158cea849525b6e661fb4562a754b001ffb3d5a5a58bba858ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:31:09.415 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:09 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:09.284+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:09.415 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:09 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:09.284+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:09.415 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:09 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:09.284+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:09.415 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:09 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:09.284+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:09.415 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:09 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:09.284+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:09.415 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:09 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:09.284+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:09.416 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:09 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:09.284+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:09.670 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:09 smithi204 podman[125760]: 2024-09-16 05:31:09.317104416 +0000 UTC m=+0.015632773 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:31:09.671 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:09 smithi204 podman[125760]: 2024-09-16 05:31:09.434152326 +0000 UTC m=+0.132680668 container create 83c262102401adf633040f7cbdaf3960665b95c4ac5992c5a9a5cbfa0f49c5a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-16T05:31:09.671 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:09 smithi204 podman[125760]: 2024-09-16 05:31:09.603421401 +0000 UTC m=+0.301949750 container init 83c262102401adf633040f7cbdaf3960665b95c4ac5992c5a9a5cbfa0f49c5a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-16T05:31:09.671 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:09 smithi204 podman[125760]: 2024-09-16 05:31:09.606801755 +0000 UTC m=+0.305330101 container start 83c262102401adf633040f7cbdaf3960665b95c4ac5992c5a9a5cbfa0f49c5a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-16T05:31:09.671 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:09 smithi204 podman[125760]: 2024-09-16 05:31:09.661542584 +0000 UTC m=+0.360070935 container attach 83c262102401adf633040f7cbdaf3960665b95c4ac5992c5a9a5cbfa0f49c5a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T05:31:10.155 INFO:journalctl@ceph.osd.10.smithi204.stdout:Sep 16 05:31:09 smithi204 podman[125760]: 2024-09-16 05:31:09.803569534 +0000 UTC m=+0.502097879 container died 83c262102401adf633040f7cbdaf3960665b95c4ac5992c5a9a5cbfa0f49c5a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-16T05:31:10.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:10.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:10.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:10.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:10.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:10.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:10.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.288+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:10.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.10.service' 2024-09-16T05:31:10.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:31:10.595 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-16T05:31:10.595 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-16T05:31:10.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.11 2024-09-16T05:31:10.920 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 systemd[1]: Stopping Ceph osd.11 for c1b4ea6a-73e8-11ef-bceb-c7b262605968... 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.925+0000 7fed16641640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.925+0000 7fed16641640 -1 osd.11 1083 *** Got signal Terminated *** 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:10 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:10.925+0000 7fed16641640 -1 osd.11 1083 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:11.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:11.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:11.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:11.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:11.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:11.252 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:11.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:11.253 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:11 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:11.250+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:12.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:12.239+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:12.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:12.239+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:12.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:12.239+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:12.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:12.239+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:12.422 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:12.239+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:12.422 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:12.239+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:12.422 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:12 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:12.239+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:13.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:13.210+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:13.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:13.210+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:13.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:13.210+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:13.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:13.210+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:13.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:13.210+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:13.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:13.210+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:13.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:13.210+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:13.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:13 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:13.210+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.204:6804 osd.8 since back 2024-09-16T05:30:51.373838+0000 front 2024-09-16T05:30:51.373895+0000 (oldest deadline 2024-09-16T05:31:13.073833+0000) 2024-09-16T05:31:14.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:14.196+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:14.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:14.196+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:14.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:14.196+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:14.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:14.196+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:14.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:14.196+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:14.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:14.196+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:14.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:14.196+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:14.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:14 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:14.196+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.204:6804 osd.8 since back 2024-09-16T05:30:51.373838+0000 front 2024-09-16T05:30:51.373895+0000 (oldest deadline 2024-09-16T05:31:13.073833+0000) 2024-09-16T05:31:15.420 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:15.163+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-09-16T05:29:44.964565+0000 front 2024-09-16T05:29:44.964387+0000 (oldest deadline 2024-09-16T05:30:09.664263+0000) 2024-09-16T05:31:15.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:15.163+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6822 osd.2 since back 2024-09-16T05:30:04.266959+0000 front 2024-09-16T05:30:04.266951+0000 (oldest deadline 2024-09-16T05:30:25.966913+0000) 2024-09-16T05:31:15.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:15.163+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.32:6830 osd.3 since back 2024-09-16T05:30:10.568356+0000 front 2024-09-16T05:30:10.568220+0000 (oldest deadline 2024-09-16T05:30:35.268283+0000) 2024-09-16T05:31:15.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:15.163+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-09-16T05:30:20.470053+0000 front 2024-09-16T05:30:20.470131+0000 (oldest deadline 2024-09-16T05:30:45.769879+0000) 2024-09-16T05:31:15.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:15.163+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-09-16T05:30:28.071003+0000 front 2024-09-16T05:30:28.070975+0000 (oldest deadline 2024-09-16T05:30:52.770778+0000) 2024-09-16T05:31:15.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:15.163+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6820 osd.6 since back 2024-09-16T05:30:33.271821+0000 front 2024-09-16T05:30:33.271751+0000 (oldest deadline 2024-09-16T05:30:59.171634+0000) 2024-09-16T05:31:15.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:15.163+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.196:6828 osd.7 since back 2024-09-16T05:30:40.872413+0000 front 2024-09-16T05:30:40.872475+0000 (oldest deadline 2024-09-16T05:31:06.172343+0000) 2024-09-16T05:31:15.421 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:15 smithi204 ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11[52782]: 2024-09-16T05:31:15.163+0000 7fed12c57640 -1 osd.11 1083 heartbeat_check: no reply from 172.21.15.204:6804 osd.8 since back 2024-09-16T05:30:51.373838+0000 front 2024-09-16T05:30:51.373895+0000 (oldest deadline 2024-09-16T05:31:13.073833+0000) 2024-09-16T05:31:16.644 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:16 smithi204 podman[125858]: 2024-09-16 05:31:16.372924752 +0000 UTC m=+5.550231542 container died 091e471eb8151e365591e478fb0ac95199750cf605bdc7d06c5e139bf837ef41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-16T05:31:16.920 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:16 smithi204 podman[125858]: 2024-09-16 05:31:16.818207321 +0000 UTC m=+5.995514108 container cleanup 091e471eb8151e365591e478fb0ac95199750cf605bdc7d06c5e139bf837ef41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11, RELEASE=squid-8921c8e, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-16T05:31:16.920 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:16 smithi204 bash[125858]: ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11 2024-09-16T05:31:17.378 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:17 smithi204 podman[125885]: 2024-09-16 05:31:17.077777526 +0000 UTC m=+0.706828192 container remove 091e471eb8151e365591e478fb0ac95199750cf605bdc7d06c5e139bf837ef41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-16T05:31:17.670 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:17 smithi204 podman[125970]: 2024-09-16 05:31:17.37864427 +0000 UTC m=+0.111246218 container create e20cdc4127eee34ccc6077746a15d1f2d7e3c6af23985ed00e5b2dd9f6e6e32f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-16T05:31:17.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:17 smithi204 podman[125970]: 2024-09-16 05:31:17.2830808 +0000 UTC m=+0.015682705 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T05:31:17.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:17 smithi204 podman[125970]: 2024-09-16 05:31:17.614310757 +0000 UTC m=+0.346912649 container init e20cdc4127eee34ccc6077746a15d1f2d7e3c6af23985ed00e5b2dd9f6e6e32f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-09-16T05:31:17.671 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:17 smithi204 podman[125970]: 2024-09-16 05:31:17.617602209 +0000 UTC m=+0.350204108 container start e20cdc4127eee34ccc6077746a15d1f2d7e3c6af23985ed00e5b2dd9f6e6e32f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:31:18.101 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:17 smithi204 podman[125970]: 2024-09-16 05:31:17.69751601 +0000 UTC m=+0.430117917 container attach e20cdc4127eee34ccc6077746a15d1f2d7e3c6af23985ed00e5b2dd9f6e6e32f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T05:31:18.101 INFO:journalctl@ceph.osd.11.smithi204.stdout:Sep 16 05:31:17 smithi204 podman[125970]: 2024-09-16 05:31:17.803785849 +0000 UTC m=+0.536387751 container died e20cdc4127eee34ccc6077746a15d1f2d7e3c6af23985ed00e5b2dd9f6e6e32f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968-osd-11-deactivate, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-16T05:31:18.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c1b4ea6a-73e8-11ef-bceb-c7b262605968@osd.11.service' 2024-09-16T05:31:18.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T05:31:18.441 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-16T05:31:18.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T05:31:18.653 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:31:20.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T05:31:20.612 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:31:22.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T05:31:22.537 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:31:23.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:31:24.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:31:24.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T05:31:24.067 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-16T05:31:24.069 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi032/crash 2024-09-16T05:31:24.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash -- . 2024-09-16T05:31:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T05:31:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-09-16T05:31:24.110 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi196/crash 2024-09-16T05:31:24.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash -- . 2024-09-16T05:31:24.160 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T05:31:24.161 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-09-16T05:31:24.163 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi204/crash 2024-09-16T05:31:24.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash -- . 2024-09-16T05:31:24.230 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T05:31:24.230 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-09-16T05:31:24.231 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-16T05:31:24.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-16T05:31:24.274 INFO:tasks.cephadm:Compressing logs... 2024-09-16T05:31:24.274 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T05:31:24.316 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T05:31:24.321 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T05:31:24.343 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’: No such file or directorygzip 2024-09-16T05:31:24.343 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-16T05:31:24.343 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-16T05:31:24.344 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/objectstore_tool.2.log 2024-09-16T05:31:24.344 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T05:31:24.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T05:31:24.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T05:31:24.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-16T05:31:24.345 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-16T05:31:24.345 INFO:teuthology.orchestra.run.smithi032.stderr: 82.4%/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.a.log: -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T05:31:24.350 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.10.log: No such file or directory 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.9.log 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log: /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.11.log 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log 2024-09-16T05:31:24.351 INFO:teuthology.orchestra.run.smithi204.stderr: 81.9% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T05:31:24.354 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.11.log: /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.8.log 2024-09-16T05:31:24.354 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T05:31:24.354 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.8.log: /var/log/ceph/cephadm.log: 90.3% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T05:31:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.2.log: 90.3% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T05:31:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-16T05:31:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log 2024-09-16T05:31:24.357 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-16T05:31:24.358 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-16T05:31:24.358 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/objectstore_tool.2.log 2024-09-16T05:31:24.358 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-16T05:31:24.358 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T05:31:24.358 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-16T05:31:24.358 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr: 82.0% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log: 94.7% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/objectstore_tool.2.log.gz 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr: 90.3% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T05:31:24.359 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose --/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mgr.x.log: /var/log/ceph/cephadm.log 2024-09-16T05:31:24.361 INFO:teuthology.orchestra.run.smithi204.stderr: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T05:31:24.361 INFO:teuthology.orchestra.run.smithi204.stderr: 91.2% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T05:31:24.361 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.b.log: 91.2% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T05:31:24.362 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log: 91.1% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T05:31:24.362 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T05:31:24.364 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/cephadm.log: 92.9% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-16T05:31:24.365 INFO:teuthology.orchestra.run.smithi196.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T05:31:24.365 INFO:teuthology.orchestra.run.smithi196.stderr: 86.6% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T05:31:24.367 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T05:31:24.370 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.3.log: 86.6% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T05:31:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cephadm.log: 94.8% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/objectstore_tool.2.log.gz 2024-09-16T05:31:24.389 INFO:teuthology.orchestra.run.smithi032.stderr: 95.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T05:31:24.827 INFO:teuthology.orchestra.run.smithi032.stderr: 91.4% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-16T05:31:24.882 INFO:teuthology.orchestra.run.smithi196.stderr: 91.6% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-16T05:31:25.202 INFO:teuthology.orchestra.run.smithi204.stderr: 91.5% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-16T05:31:25.851 INFO:teuthology.orchestra.run.smithi032.stderr: 90.6% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-16T05:31:26.230 INFO:teuthology.orchestra.run.smithi196.stderr: 93.2% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-16T05:31:26.495 INFO:teuthology.orchestra.run.smithi032.stderr: 93.4% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-16T05:31:26.723 INFO:teuthology.orchestra.run.smithi196.stderr: 93.4% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-16T05:31:26.725 INFO:teuthology.orchestra.run.smithi196.stderr: 93.2% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-16T05:31:26.779 INFO:teuthology.orchestra.run.smithi032.stderr: 93.8% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-16T05:31:26.917 INFO:teuthology.orchestra.run.smithi032.stderr: 93.2% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-16T05:31:26.928 INFO:teuthology.orchestra.run.smithi204.stderr: 93.3% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.11.log.gz 2024-09-16T05:31:27.085 INFO:teuthology.orchestra.run.smithi204.stderr: 93.6% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.10.log.gz 2024-09-16T05:31:27.277 INFO:teuthology.orchestra.run.smithi032.stderr: 93.3% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-16T05:31:27.278 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-09-16T05:31:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m2.948s 2024-09-16T05:31:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m11.759s 2024-09-16T05:31:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m0.403s 2024-09-16T05:31:27.447 INFO:teuthology.orchestra.run.smithi204.stderr: 93.6% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.9.log.gz 2024-09-16T05:31:27.504 INFO:teuthology.orchestra.run.smithi204.stderr: 93.5% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.8.log.gz 2024-09-16T05:31:27.505 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T05:31:27.505 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m3.170s 2024-09-16T05:31:27.505 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m11.984s 2024-09-16T05:31:27.506 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.488s 2024-09-16T05:31:27.655 INFO:teuthology.orchestra.run.smithi196.stderr: 93.7% -- replaced with /var/log/ceph/c1b4ea6a-73e8-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-16T05:31:27.657 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-09-16T05:31:27.657 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m3.323s 2024-09-16T05:31:27.657 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m10.126s 2024-09-16T05:31:27.657 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m0.412s 2024-09-16T05:31:27.658 INFO:tasks.cephadm:Archiving logs... 2024-09-16T05:31:27.658 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi032/log 2024-09-16T05:31:27.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T05:31:29.330 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi196/log 2024-09-16T05:31:29.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T05:31:30.860 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi204/log 2024-09-16T05:31:30.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T05:31:32.616 INFO:tasks.cephadm:Removing cluster... 2024-09-16T05:31:32.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 --force 2024-09-16T05:31:32.830 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:31:33.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 --force 2024-09-16T05:31:33.298 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:31:33.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c1b4ea6a-73e8-11ef-bceb-c7b262605968 --force 2024-09-16T05:31:33.767 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: c1b4ea6a-73e8-11ef-bceb-c7b262605968 2024-09-16T05:31:34.040 INFO:tasks.cephadm:Removing cephadm ... 2024-09-16T05:31:34.040 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T05:31:34.060 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T05:31:34.076 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T05:31:34.095 INFO:tasks.cephadm:Teardown complete 2024-09-16T05:31:34.096 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T05:31:34.117 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T05:31:34.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T05:31:34.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T05:31:34.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T05:31:34.232 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T05:31:34.241 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T05:31:34.242 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:31:34.245 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:31:34.248 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:31:34.260 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-16T05:31:34.261 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-09-16T05:31:34.262 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-16T05:31:34.264 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T05:31:34.264 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-16T05:31:34.264 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv01.front.sepia.ceph.com 3 6 377 16 +505us[ +507us] +/- 66ms 2024-09-16T05:31:34.264 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 78 +730us[ +732us] +/- 82ms 2024-09-16T05:31:34.264 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -1711us[-1708us] +/- 92ms 2024-09-16T05:31:34.264 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T05:31:34.266 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv01.front.sepia.ceph.com 3 6 377 15 +549us[ +582us] +/- 66ms 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 +740us[ +740us] +/- 55ms 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 -1686us[-1652us] +/- 92ms 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv01.front.sepia.ceph.com 3 6 377 15 +484us[ +568us] +/- 66ms 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 +675us[ +758us] +/- 55ms 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 -1767us[-1767us] +/- 64ms 2024-09-16T05:31:34.267 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T05:31:34.268 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T05:31:34.280 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T05:31:34.281 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T05:31:34.290 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T05:31:34.365 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T05:31:34.467 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T05:31:34.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T05:31:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:31:34.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T05:31:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:31:34.634 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T05:31:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:31:34.665 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T05:31:34.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T05:31:34.687 INFO:teuthology.task.internal:Duration was 2127.899510 seconds 2024-09-16T05:31:34.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T05:31:34.697 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T05:31:34.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T05:31:34.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T05:31:34.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T05:31:34.738 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T05:31:34.743 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T05:31:34.744 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T05:31:35.025 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T05:31:35.025 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-09-16T05:31:35.026 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T05:31:35.047 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-09-16T05:31:35.048 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T05:31:35.099 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T05:31:35.099 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T05:31:35.127 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T05:31:35.127 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T05:31:35.130 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T05:31:35.142 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T05:31:35.194 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T05:31:35.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T05:31:35.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T05:31:35.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T05:31:37.220 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T05:31:37.230 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T05:31:37.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T05:31:37.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T05:31:37.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T05:31:37.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T05:31:37.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:37.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:37.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:37.348 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-09-16T05:31:37.354 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-09-16T05:31:37.380 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-09-16T05:31:37.409 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:31:37.441 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:31:37.457 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:31:37.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T05:31:37.483 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T05:31:37.484 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi032 2024-09-16T05:31:37.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T05:31:37.553 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi196 2024-09-16T05:31:37.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T05:31:37.606 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906095/remote/smithi204 2024-09-16T05:31:37.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T05:31:37.653 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T05:31:37.654 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T05:31:37.656 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T05:31:37.659 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T05:31:37.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T05:31:37.716 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T05:31:37.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T05:31:37.725 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T05:31:37.725 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T05:31:37.728 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T05:31:37.730 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T05:31:37.741 INFO:teuthology.orchestra.run.smithi032.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 05:31 /home/ubuntu/cephtest 2024-09-16T05:31:37.744 INFO:teuthology.orchestra.run.smithi196.stdout: 265329 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 05:31 /home/ubuntu/cephtest 2024-09-16T05:31:37.757 INFO:teuthology.orchestra.run.smithi204.stdout: 266064 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 05:31 /home/ubuntu/cephtest 2024-09-16T05:31:37.758 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T05:31:37.768 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T05:31:37.795 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} duration: 2127.899510383606 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T05:31:37.796 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T05:31:37.872 INFO:teuthology.run:pass